Python Read Files

Python Read Files: Streaming Lines vs Loading Whole Files

Python gives you several ways to read a file: .read() loads the entire file into one string, .readline() grabs one line, and looping directly over the file object streams one line at a time using minimal memory.

Choosing the right method keeps small config reads simple while allowing scripts to scan multi-gigabyte server logs without running out of RAM.

Comparing File Reading Methods

For small files under a few megabytes, .read() or Path.read_text() is quick and convenient. For large logs or CSV exports, iterate directly over the file object inside with open(...).

from pathlib import Path

sample = Path("events.txt")
sample.write_text("INFO started
ERROR timeout
INFO recovered
", encoding="utf-8")

# Memory-safe streaming: reads one line at a time
error_lines = []
with sample.open("r", encoding="utf-8") as f:
    for line_no, raw_line in enumerate(f, start=1):
        clean_line = raw_line.rstrip("
")
        if "ERROR" in clean_line:
            error_lines.append((line_no, clean_line))

print(error_lines)  # [(2, 'ERROR timeout')]

Reading Fixed Chunks for Binary Files

When hashing or copying huge binary files, pass a byte size to .read(chunk_size) inside a loop (for example, 65536 bytes per iteration).

Common Mistakes to Avoid

• Calling .read() twice on the same open file: The first .read() moves the cursor to the end of the file, so the second .read() returns "" unless you call f.seek(0).
• Using .readlines() on huge files: .readlines() loads every line into a Python list in memory; for line in f: uses almost zero RAM.
• Forgetting stays at the end of each line: Strip trailing newlines with .rstrip(" ") before splitting or comparing values.

Frequently Asked Questions

How do I read only the first 5 lines of a file?

Use itertools.islice(f, 5) inside with open(...) so Python stops reading right after line 5.

What happens if I open a missing file in "r" mode?

Python raises FileNotFoundError, which you can handle cleanly with try ... except FileNotFoundError:.

How do I read structured CSV files properly?

Use Python's built-in csv.DictReader(f) with newline="" inside open() instead of manually calling line.split(","), which breaks on quoted commas.