Python Generators and Yield Explained: Save Memory With Real Examples

Imagine loading a 10 GB log file into a list just to find the errors. Your program may run out of memory before it finds one. Python generators solve that by producing values on demand, one at a time, so you can work through huge or even endless data with almost no memory.

This guide shows what really happens when a generator runs, step by step. We ran each example on Python 3.13, and the output comes from that run. It helps to already be comfortable with functions and loops, which our Python tutorial for beginners covers.

What is a generator?

A generator is a special kind of iterator that you create by writing a function using yield. Any function with yield in it automatically becomes a generator function, and calling it gives you a generator object rather than running the code.

def count_up(limit):
    print("starting")
    n = 1
    while n <= limit:
        yield n
        n += 1
    print("finished")

gen = count_up(2)
print(gen)

# <generator object count_up at 0x...>

Notice that “starting” did not print. Calling the function only created the generator. Nothing inside has run yet.

How yield works, step by step

Each call to next() runs the function until it reaches a yield, hands back that value, and freezes in place. The next call resumes right after that line, with every local variable intact.

print(next(gen))
# starting
# 1

print(next(gen))
# 2

next(gen)
# finished
# StopIteration

Follow the output carefully. “starting” appears only on the first next(). On the second, execution continues from where it stopped, so n is still available and becomes 2. On the third, the loop ends, “finished” prints, the function falls off the end, and Python raises StopIteration to signal that there is nothing left.

You rarely call next() yourself. A for loop does it for you and handles StopIteration quietly, and so do list(), sum() and similar functions:

print(list(count_up(3)))

# starting
# finished
# [1, 2, 3]

yield vs return

This delay is known as lazy evaluation: values are worked out only when needed. A plain return inside a generator simply ends it, which triggers StopIteration.

The memory difference, measured

We compared a list of one million squares with an equivalent generator, using sys.getsizeof:

import sys

squares_list = [n * n for n in range(1_000_000)]
squares_gen = (n * n for n in range(1_000_000))

print(sys.getsizeof(squares_list), sys.getsizeof(squares_gen))

# 8448728 200

The list’s own structure took about 8.4 MB, while the generator took just 200 bytes on our system. The real gap is even bigger, as getsizeof does not count the integer objects inside the list. The generator’s size also stays the same for a billion items, since it stores only its current state and not its results.

There is a trade-off, though. A list can be reused, indexed and measured with len(). A generator offers none of these. If you need random access, or you will loop several times over a small amount of data, a list is simpler. See our guide to Python list vs tuple vs set for more on those types.

Generator expressions

For simple cases, you do not need a function. A generator expression looks like a list comprehension with round brackets in place of square ones:

squares = [n * n for n in range(5)]   # builds a list now
squares = (n * n for n in range(5))   # produces values on demand

print(sum(n * n for n in range(5)))

# 30

When a generator expression is the only argument to a function, as in sum(), you can leave out the extra brackets. This is a tidy habit for totals over large data, because no temporary list is created.

Build a data pipeline

This is where generators really shine. Small generators can be chained so that each item flows through every step, one at a time:

def read_lines(lines):
    for line in lines:
        yield line.strip()

def only_errors(lines):
    for line in lines:
        if line.startswith("ERROR"):
            yield line

def messages(lines):
    for line in lines:
        yield line.split(" ", 1)[1]

log = ["INFO started\n", "ERROR db down\n",
       "ERROR disk full\n", "INFO stopped\n"]

print(list(messages(only_errors(read_lines(log)))))

# ['db down', 'disk full']

Each step does one small job and can be tested on its own. In real use, swap the sample list for an open file, because file objects can be read line by line, as our guide to Python file handling shows. Even a multi-gigabyte log would then pass through with very little memory.

Infinite sequences

A list cannot be infinite, but a generator can, because it only ever computes the next value. Use itertools.islice to take as many as you need:

import itertools

def fibonacci():
    a, b = 0, 1
    while True:
        yield a
        a, b = b, a + b

print(list(itertools.islice(fibonacci(), 8)))

# [0, 1, 1, 2, 3, 5, 8, 13]

Be careful, though: calling list(fibonacci()) without a limit would run until your memory is gone. The same pattern suits ID generators, polling loops and paginated API results, where you fetch the next page only when the current one runs out.

yield from and batches

yield from hands off to another iterable, saving you a nested loop:

def chain_all(*groups):
    for group in groups:
        yield from group

print(list(chain_all([1, 2], (3, 4))))

# [1, 2, 3, 4]

A very practical pattern is splitting data into fixed-size batches, for example to insert rows into a database a thousand at a time:

def batches(items, size):
    batch = []
    for item in items:
        batch.append(item)
        if len(batch) == size:
            yield batch
            batch = []
    if batch:
        yield batch

print(list(batches(range(7), 3)))

# [[0, 1, 2], [3, 4, 5], [6]]

The final if batch makes sure that a smaller leftover group is not lost. Python 3.12 and later also include itertools.batched, which does a similar job.

send, close and cleanup

Generators can also receive values. send() resumes the generator, and the value becomes the result of the yield expression. You must call next() once first to reach the first yield:

def running_total():
    total = 0
    while True:
        value = yield total
        total += value

t = running_total()
next(t)
print(t.send(10), t.send(5))

# 10 15

Most people seldom need this. A more common concern is cleanup. If you stop early, a finally block still runs when the generator is closed:

def tracked():
    try:
        yield 1
        yield 2
    finally:
        print("cleanup ran")

tr = tracked()
next(tr)
tr.close()

# cleanup ran

Pytest uses this very idea: a fixture with yield runs its cleanup afterwards, as our pytest tutorial demonstrates. Likewise, the standard library’s contextmanager decorator turns a generator into a with block.

Generators in classes

Writing an __iter__ method as a generator is the easiest way to make your own class iterable. A bonus is that each loop gets a fresh generator, so the object can be looped over again and again:

class Countdown:
    def __init__(self, start):
        self.start = start

    def __iter__(self):
        n = self.start
        while n > 0:
            yield n
            n -= 1

c = Countdown(3)
print(list(c), list(c))

# [3, 2, 1] [3, 2, 1]

When to use a generator

A generator is not always faster overall. If you end up using every value, the total work is similar, and a list comprehension can even be quicker for small data because it has less per-item overhead. The main benefits are memory use, and getting the first results sooner.

Common mistakes

Reusing an exhausted generator

This catches nearly everyone at some point. A generator remembers that it is finished:

g = count_up(2)
print(list(g), list(g))

# [1, 2] []

The second call gets an empty list with no error at all. Call the function again to get a new generator, or use the class approach shown above.

Expecting code to run immediately

def noisy():
    print("body runs")
    yield 1

n = noisy()        # nothing is printed
print(next(n))     # body runs, then 1

Argument checks at the top of a generator do not run until the first next(), so errors can appear far from the call that caused them.

Other traps

• Calling len() or indexing. Generators support neither, and raise a TypeError.
• Closing a file too soon. Returning a generator from inside a with block can mean the file closes before any line is read. Put the yield inside the with block.
• Wrapping everything in list(). That throws away the memory benefit.
• Using one for tiny data. A plain list is often clearer.

Final thoughts

A generator is a function that can pause. That one idea gives you low memory use, endless sequences and clean pipelines. To put it into practice, find a place in your code that builds a large list only to loop over it once, and turn that into a generator. Then try chaining two small generators together on a real log file.

Frequently asked questions

Keep learning