Advanced Python Iterators & Generators

Advanced Python Iterators: Generator Pipelines & itertools

The itertools standard library module provides memory-efficient, fast building blocks for creating complex iterator pipelines without memory overhead.

Itertools Functional Categories

flowchart TD
    A["itertools Module"] --> B["Infinite Iterators (count, cycle, repeat)"]
    A --> C["Terminating Iterators (chain, islice, zip_longest)"]
    A --> D["Combinatoric Iterators (product, permutations, combinations)"]

Essential itertools Functions

• itertools.chain(*iterables): Flattens multiple iterables into a single continuous stream.
• itertools.islice(iterable, stop): Slices an iterator lazily without converting to a list.
• itertools.cycle(iterable): Cycles indefinitely through an iterable sequence.
• itertools.combinations(iterable, r): Yields unique r-length combinations.

Practical Code Example

import itertools
from typing import List, Tuple

def stream_processing_demo() -> None:
    # Combining multiple data streams lazily
    batch1 = [10, 20, 30]
    batch2 = [40, 50]
    combined_stream = itertools.chain(batch1, batch2)
    
    # Lazy slicing first 4 items
    sliced_stream = list(itertools.islice(combined_stream, 4))
    print("Combined & Sliced Stream:", sliced_stream)

    # Combinatorics: Generate 2-item pairs
    team_members = ["Alice", "Bob", "Charlie"]
    pairs: List[Tuple[str, str]] = list(itertools.combinations(team_members, 2))
    print("Team Pair Combinations:", pairs)

if __name__ == "__main__":
    stream_processing_demo()

Best Practices & Gotchas

• Do Not Convert Large Generators to Lists: Calling list(generator) loads all generated items into memory at once, destroying the lazy memory advantage of iterators.
• Infinite Iterators Need Guards: itertools.cycle() or count() will loop forever unless bounded by islice() or an explicit break.
• Use Generator Expressions: (x*2 for x in range(1000)) is a memory-light generator, whereas [x*2 for x in range(1000)] creates a full list in heap memory.

Self-Check Challenge

Write a generator pipeline using itertools.chain() that merges two lists [1, 2] and [3, 4], squares each element lazily, and prints the result.