Iterators and Generators

Big idea

An iterator hands out values one at a time. A generator is the easiest way to make one: a function that uses yield to produce each value only when it's asked for.

How a for loop really works

Anything you can loop over is iterable. Behind the scenes, for asks it for an iterator, then calls next() on that until it runs out:

letters = iter("abc")
print(next(letters))
print(next(letters))
print(next(letters))

output
a
b
c

One more next(letters) would raise StopIteration — the signal a for loop uses to stop.

yield

A function with yield in it doesn't run when you call it. It returns a generator, which runs the body only as far as the next yield, hands out that value, and pauses there:

def countdown(n):
    print("starting")
    while n > 0:
        yield n
        n -= 1
    print("finished")

gen = countdown(3)
print("made the generator")
for value in gen:
    print(value)

made the generator
starting
3
2
1
finished

"starting" only appears once the loop asks for the first value. Each yield pauses the function with all its variables intact; the next request resumes right after it.

Why bother? Laziness

A generator never builds the whole sequence. It can even be infinite, because only the values you ask for are ever made:

def evens():
    n = 0
    while True:
        yield n
        n += 2

for i, value in enumerate(evens()):
    if i == 5:
        break
    print(value)

output
0
2
4
6
8

A list of every even number would never finish building. The generator makes exactly five.

Generator expressions

Swap a comprehension's square brackets for round ones and you get a generator instead of a list:

total = sum(n * n for n in range(1_000_000))   # no million-item list in memory

Used up after one pass

squares = (n * n for n in range(4))
print(list(squares))
print(sum(squares))

output
[0, 1, 4, 9]
14

A generator is used up after one pass — the second use finds it empty, so sum is 0. Make a list once (squares = [n * n for n in range(4)]) if you need the values twice.

Watch it

Think of it like

A Pez dispenser versus a bag of sweets. The bag (a list) holds everything at once. The dispenser (a generator) gives you one sweet each time you tilt it — and it never had to make them all up front.

Key terms

• Iterator — An object that gives the next value each time next() is called.
• Generator — A function using yield; produces values lazily.
• Lazy evaluation — Computing values only when they're needed.