Generators and yield in Python: Process Large Data Without Wasting RAM

1. Overview

When processing multi-gigabyte log files or streaming millions of records from a database, writing code that “reads everything into a list then processes” will consume RAM and can crash your program. This article helps you understand how generator and yield work, compare them with regular lists, and apply them to real problems like reading large files, data processing pipelines. After reading, you’ll be able to write generators correctly and know when not to use them.

2. The Problem with the Traditional Approach

Suppose you need to read a 5GB log file and count the lines containing the word “ERROR”:

def count_errors_bad(filepath: str) -> int:
    with open(filepath, "r") as f:
        lines = f.readlines()  # load ENTIRE file into RAM
    return sum(1 for line in lines if "ERROR" in line)

f.readlines() reads the entire file contents into a list of strings in memory. With a 5GB file, Python will allocate ~5GB+ of RAM (plus object overhead) before it even starts counting. If your machine only has 4GB of RAM, the program will hit MemoryError or freeze the system with swapping.

The core problem: you only need to process one line at a time, but your code forces the machine to keep all lines in memory at once.

3. What Is a Generator?

A generator is a special type of iterable: it produces values on demand (lazy evaluation) instead of computing and storing everything upfront. Instead of returning a complete list, a generator yields one value at a time and only computes the next value when asked.

def count_errors_good(filepath: str) -> int:
    with open(filepath, "r") as f:
        return sum(1 for line in f if "ERROR" in line)

Here, f (the file object) is itself an iterator — it reads one line at a time when the loop requests it, without loading the entire file into RAM. Memory usage is just one line at a time, regardless of whether the file is 5GB or 50GB.

4. The yield Keyword: Creating Generator Functions

You can write your own generators using yield inside a function. When a function contains yield, Python automatically turns it into a generator function — calling it doesn’t run the code immediately but returns a generator object.

def read_large_file(filepath: str):
    with open(filepath, "r") as f:
        for line in f:
            yield line.strip()

• yield line.strip() pauses the function here and yields the value line.strip() to the caller.
• On the next next() call, the function resumes execution from where it paused, not starting over from the beginning.
• The function’s state (local variables, file pointer position) is preserved between yield calls.

Let’s try it:

gen = read_large_file("access.log")
print(type(gen))
print(next(gen))
print(next(gen))

<class 'generator'>
127.0.0.1 - GET /index.html 200
127.0.0.1 - GET /style.css 200

Note: the line gen = read_large_file(...) doesn’t open the file or read anything. The code inside only runs when you call next(gen) for the first time.

5. Comparing RAM: Generators vs Lists

To see the difference clearly, use the sys module to measure object sizes:

import sys

def squares_list(n: int) -> list[int]:
    return [i * i for i in range(n)]

def squares_gen(n: int):
    for i in range(n):
        yield i * i

big_list = squares_list(1_000_000)
big_gen = squares_gen(1_000_000)

print(f"List size: {sys.getsizeof(big_list):,} bytes")
print(f"Generator size: {sys.getsizeof(big_gen):,} bytes")

List size: 8,448,728 bytes
Generator size: 208 bytes

A list containing 1 million elements takes up nearly 8.5MB, while a generator is just a tiny 208-byte object — because it stores no values, only the state needed to compute the next value when required.

Note: A generator can only be traversed once. After it’s exhausted, calling next() again raises StopIteration. If you need to traverse multiple times, you must create a new generator by calling the function again.

6. Generator Expressions: More Concise Syntax

Similar to list comprehension but using parentheses instead of square brackets, a generator expression lets you create a generator without defining a separate function:

numbers = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

even_squares_gen = (n * n for n in numbers if n % 2 == 0)
print(sum(even_squares_gen))

220

Using parentheses () instead of square brackets [] makes Python create a generator instead of a list immediately. This is handy when you only need the values once, such as passing them directly to sum(), max(), or any().

7. Building Data Processing Pipelines with Generators

The power of generators is that you can chain multiple processing steps without creating any intermediate lists. Suppose you need to: read an order CSV file → filter for status = "PAID" → sum the amount.

import csv

def read_orders(filepath: str):
    with open(filepath, newline="") as f:
        reader = csv.DictReader(f)
        for row in reader:
            yield row

def filter_paid(orders):
    for order in orders:
        if order["status"] == "PAID":
            yield order

def extract_amount(orders):
    for order in orders:
        yield float(order["amount"])

def total_paid_amount(filepath: str) -> float:
    orders = read_orders(filepath)
    paid_orders = filter_paid(orders)
    amounts = extract_amount(paid_orders)
    return sum(amounts)

Each function read_orders, filter_paid, and extract_amount is a generator — they don’t process all data at once, but “chain” together into a pipeline. When sum(amounts) calls next(), values are pulled through each step: read 1 CSV row → check status → convert to float → accumulate. At any moment, only one record exists in memory, whether the CSV file has 10 rows or 10 million.

7.1. Testing the Pipeline with pytest

import pytest
from pathlib import Path

@pytest.fixture
def orders_csv(tmp_path: Path) -> str:
    content = (
        "order_id,amount,status\n"
        "1,100.0,PAID\n"
        "2,50.0,PENDING\n"
        "3,200.0,PAID\n"
    )
    file_path = tmp_path / "orders.csv"
    file_path.write_text(content)
    return str(file_path)

def test_total_paid_amount(orders_csv):
    result = total_paid_amount(orders_csv)
    assert result == pytest.approx(300.0)

tmp_path is a built-in pytest fixture that creates a temporary directory for each test. This test creates a mock CSV file with 3 orders, only 2 marked PAID, and verifies the sum equals 300.0.

8. yield from: Delegating to Another Generator

When a generator needs to call and embed results from another generator, yield from makes code cleaner than writing loops manually.

def read_file_lines(filepath: str):
    with open(filepath) as f:
        for line in f:
            yield line.strip()

def read_multiple_files(filepaths: list[str]):
    for filepath in filepaths:
        yield from read_file_lines(filepath)

yield from read_file_lines(filepath) is equivalent to writing for line in read_file_lines(filepath): yield line, but more concise and correctly handles cases where the inner generator has a return value. The read_multiple_files function lets you iterate through multiple log files as if they were one continuous stream, without merging lists.

9. Sending Values Into a Generator with send()

Generators don’t just produce values — you can also send values into them using send(), turning them into simple coroutines.

def running_average():
    total = 0.0
    count = 0
    average = None
    while True:
        value = yield average
        total += value
        count += 1
        average = total / count

avg_gen = running_average()
next(avg_gen)              # prime the generator, run until first yield
print(avg_gen.send(10))    # send 10, average = 10/1
print(avg_gen.send(20))    # send 20, average = 30/2
print(avg_gen.send(30))    # send 30, average = 60/3

10.0
15.0
20.0

The line value = yield average does two things: yields average to the caller, and waits to receive a value via send() to assign to value. You must call next(avg_gen) once first to advance the generator to its first yield point — if you call send() on a fresh generator, Python raises TypeError.

Note: Using send() makes code harder to read than plain generators. In practice, frameworks like asyncio abstract this mechanism; you rarely need to write send() yourself unless building your own library.

10. When NOT to Use Generators

Generators aren’t always better than lists:

• Need random access or multiple iterations: generators don’t support gen[5] or re-iterating from the start. If you need len(), slicing, or traversing data 2-3 times, a list is better.
• Small data, no RAM concerns: with a few thousand elements, generator state management overhead can sometimes be slower than list comprehension because lists are optimized at the C level.
• Need parallelism or distributing work to multiple workers: lists are easy to chunk and send to multiprocessing; generators must be partially “materialized” before splitting.
• Debugging is harder: printing a list shows all values at once; printing a generator only shows <generator object ... at 0x...>. To see values, you must iterate (and afterward the generator is exhausted).

If in doubt, here’s the simple rule: use a list when data fits in RAM and you need to reuse it many times; use a generator when data is large, you only process it sequentially once, or you’re building a streaming pipeline.

11. Conclusion

Generators and yield solve the right problem: processing large data without loading everything into RAM, thanks to lazy evaluation — computing values only when requested. You’ve learned how to write generator functions, generator expressions, chain multiple generators into pipelines using yield from, and even send data back into a generator with send(). Remember the trade-off: generators save memory but in exchange you lose the ability to re-iterate, random access, and sometimes code becomes harder to read. Choose the right tool for the right problem — don’t use generators everywhere just because they save RAM.