hayaliv / Hi! Python / Level 2

Generators and lazy iteration

A generator function uses yield to hand back one item at a time, so you can stream records without building a full list.

Hi! Python · Level 2: Level 2: Writing reliable code · Checked against the Python 3.12 documentation · Updated 2026-10-17

Think of it like this

A generator is like reading a book aloud with a bookmark. You read one page, then stop. Next time you pick up exactly where the bookmark is. When you reach the last page, the book is finished.

Where the analogy stops being true. You can reopen a real book at page one, but a used-up generator cannot be rewound. You call the generator function again to get a new one.

What to know

A function that contains yield is a generator function.
Calling a generator function returns a generator and does not run the body yet.
Each time next() is called, the generator resumes where it left off.
When the generator finishes, it raises StopIteration.
In the example, the second list(g) is empty. A generator is an iterator, and the docs say a used-up iterator looks like an empty container.
An io.StringIO can be looped over to give its lines, so stream() can loop over it.
A good habit for big datasets: a generator hands over one record at a time, so you do not need to build one big list first.

Example

import io

def stream(lines):
    for line in lines:
        text, label = line.rstrip("\n").split("|")
        yield {"input": text, "label": label}

data = io.StringIO("call me|meeting_request\nstop|not_interested\nhi|unknown\n")
g = stream(data)
print(next(g))
print(list(g))
print(list(g))

It prints:

{'input': 'call me', 'label': 'meeting_request'}
[{'input': 'stop', 'label': 'not_interested'}, {'input': 'hi', 'label': 'unknown'}]
[]

Predict, then check

1. What does this print?

def count():
    print("start")
    yield 1
    yield 2

g = count()
print("made")
print(next(g))

Show the answer

made
start
1

Calling count() only builds the generator. The body starts at the first next(), so made prints before start.

2. What does this print?

def count():
    yield 1
    yield 2

g = count()
print(next(g), next(g))
try:
    next(g)
except StopIteration:
    print("done")

Show the answer

1 2
done

The two next() calls give 1 and 2. A third call finds the generator finished and raises StopIteration.

3. What does this print?

def evens(nums):
    for n in nums:
        if n % 2 == 0:
            yield n

g = evens([1, 2, 3, 4])
print(list(g))
print(list(g))

Show the answer

[2, 4]
[]

The first list(g) uses the generator up. The second one finds nothing left, so it gives an empty list.

Try it

Write a generator function long_replies(texts, min_words) that yields each text with at least min_words words (use len(text.split())), one at a time.

def long_replies(texts, min_words):
    return []

Show a solution

def long_replies(texts, min_words):
    for text in texts:
        if len(text.split()) >= min_words:
            yield text

Common mistake

A generator can be used only once. If you loop over it twice, the second loop gets nothing (the example's second list(g) prints []). Call the generator function again, or build a list, when you need a second pass.

In a small-language-model project

A training file can be too large to hold comfortably as one list. A good habit is to write a generator that reads and cleans one record at a time, then feed those records to your tokenizer or training loop.

Sources

Each quote below appears word for word on the cited page of the official Python documentation (version 3.12).

Using a yield expression in a function's body causes that function to be a generator function
docs.python.org/3.12/reference/expressions.html — A function whose body uses yield is a generator function.
When a generator function is called, it returns an iterator known as a generator.
docs.python.org/3.12/reference/expressions.html — Calling a generator function returns a generator.
Each time next() is called on it, the generator resumes where it left off
docs.python.org/3.12/tutorial/classes.html — A generator resumes where it left off each time next() is called.
when generators terminate, they automatically raise StopIteration .
docs.python.org/3.12/tutorial/classes.html — A finished generator raises StopIteration.
Attempting this with an iterator will just return the same exhausted iterator object used in the previous iteration pass, making it appear like an empty container.
docs.python.org/3.12/glossary.html — An exhausted iterator looks like an empty container on a second pass.
an IOBase object can be iterated over yielding the lines in a stream
docs.python.org/3.12/library/io.html — IOBase (and so StringIO) can be iterated over, yielding lines.

Step through this lesson in the Hi! Python desktop