Python Iterators

Python Iterators: Protocol, Generators, and itertools

A Python iterator is an object that produces values one at a time via next() and remembers its position between calls. Behind every for loop, Python requests an iterator with iter() and reads items until StopIteration signals the end.

An iterable (like a list, str, or dict) implements __iter__() to return a fresh iterator, while an iterator implements both __iter__() and __next__().

How the Iterator Protocol Works

flowchart LR
    A["Iterable (list, file)"] -->|"iter(obj)"| B["Iterator"]
    B -->|"next(it)"| C{"Item left?"}
    C -- "Yes" --> D["Return value"]
    C -- "No" --> E["Raise StopIteration"]

You can build a custom iterator class, or write a generator function using yield, which implements the iterator protocol automatically in a fraction of the code.

def batch_numbers(limit: int):
    current = 1
    while current <= limit:
        yield current
        current += 1

gen = batch_numbers(3)
print(next(gen), next(gen), list(gen))  # 1 2 [3]

Memory-Efficient Pipelines with itertools

The standard library itertools module provides fast C-backed building blocks such as islice (slice an iterator), chain (combine iterables), and count (infinite counter) without loading datasets into RAM.

Common Mistakes to Avoid

• Exhausting a generator and looping twice: Iterators can only be consumed once; a second for x in gen: loop produces zero items.
• Returning None instead of raising StopIteration in __next__: Returning None creates an infinite loop of None values.
• Calling list(huge_generator) unnecessarily: Converting a million-row stream to a list throws away the memory benefit of streaming.

Frequently Asked Questions

What is the difference between return and yield?

return ends a function permanently and sends back one value. yield pauses the function, hands back one value, and resumes from the exact same line on the next next() call.

Are lists iterators in Python?

No. A list is an iterable, not an iterator; calling next([1, 2]) raises TypeError unless you wrap it with iter([1, 2]).

What is a generator expression?

Writing (x * x for x in range(1000)) with parentheses creates a lazy generator instead of allocating a full list in memory.