Python Read Files

Python Read Files: Streaming, Chunking, & Memory Efficiency

Reading small files into memory at once is straightforward using .read(), but large files (gigabytes) must be streamed line-by-line or processed in fixed chunk buffers to avoid MemoryError crashes.

File Reading Strategies

flowchart TD
    A["File Read Requirement"] --> B{"File Size vs Available RAM"}
    B -- "Small (< 100MB)" --> C["file.read() or file.readlines()"]
    B -- "Large (> 1GB)" --> D["Line-by-Line Iteration: for line in file:"]
    B -- "Binary / Large Chunks" --> E["file.read(chunk_size) Loop"]

Practical Code Example

from pathlib import Path
from typing import Iterator

def read_file_line_by_line(file_path: Path) -> Iterator[str]:
    """Stream file line-by-line lazily with memory safety."""
    with open(file_path, mode='r', encoding='utf-8') as f:
        for line in f:
            yield line.strip()

def read_binary_chunks(file_path: Path, chunk_size: int = 1024) -> Iterator[bytes]:
    """Stream binary file in fixed-size memory chunks."""
    with open(file_path, mode='rb') as f:
        while chunk := f.read(chunk_size):
            yield chunk

if __name__ == "__main__":
    demo_file = Path("demo_read.txt")
    demo_file.write_text("Alpha
Beta
Gamma
Delta
", encoding="utf-8")

    print("--- Streamed Lines ---")
    for line in read_file_line_by_line(demo_file):
        print(f"Read: {line}")

    demo_file.unlink()

Best Practices & Gotchas

• Iterate File Objects Directly: Use for line in f: to iterate over lines lazily. Do NOT use f.readlines() on multi-gigabyte files.
• Strip Trailing Newlines: Remember that for line in f: includes trailing newline characters ( ); use `line.rstrip('

')`.

• Use pathlib.Path.read_text() for Quick Reads: For tiny text files, Path("file.txt").read_text(encoding="utf-8") is concise and clean.

Self-Check Challenge

Write a function count_lines(filepath: str) -> int that counts lines in a file by iterating line-by-line.