Multiprocessing in Python for CPU-bound Tasks

1. Overview

When your Python code needs to do heavy calculations—for example, image processing, checksum computation, or numerical simulation—you’ll notice that using threads makes little to no difference. The culprit is the GIL (Global Interpreter Lock). This article explains the issue concisely. Then, it shows you how to use the multiprocessing module and ProcessPoolExecutor to distribute work across multiple processes. After reading, you’ll know how to parallelize a CPU-bound task, write tests for it, and avoid common pitfalls.

All examples use Python 3.12+ and only the standard library—no additional packages needed.

2. Why threading doesn’t help

2.1. A CPU-intensive task

We’ll use a prime-counting function written in pure Python. This is a typical CPU-intensive task:

# cpu_task.py
import math

def count_primes(start: int, end: int) -> int:
    count = 0
    for n in range(max(start, 2), end):
        if all(n % d for d in range(2, math.isqrt(n) + 1)):
            count += 1
    return count

The function iterates through each number in [start, end) and checks divisibility by all numbers from 2 to its square root. It reads no files, makes no network calls. It just computes.

2.2. Benchmarking: Sequential vs. Threads

# bench_threads.py
import time
from concurrent.futures import ThreadPoolExecutor
from cpu_task import count_primes

RANGES = [(i, i + 125_000) for i in range(0, 1_000_000, 125_000)]

def run_sequential() -> int:
    return sum(count_primes(a, b) for a, b in RANGES)

def run_threads() -> int:
    starts, ends = zip(*RANGES)
    with ThreadPoolExecutor(max_workers=8) as executor:
        return sum(executor.map(count_primes, starts, ends))

if __name__ == "__main__":
    for fn in (run_sequential, run_threads):
        t0 = time.perf_counter()
        result = fn()
        print(f"{fn.__name__}: {result} primes, {time.perf_counter() - t0:.2f}s")

RANGES splits the range [0, 1_000_000) into 8 parts. run_threads assigns each part to a thread. Results on an 8-core machine (your numbers will vary):

run_sequential: 78498 primes, 6.41s
run_threads: 78498 primes, 6.58s

The threaded version is actually slightly slower. In standard CPython, only one thread can execute Python bytecode at a time because it holds the GIL. Eight threads merely take turns using a single core, and context switching adds overhead.

Note: Starting with Python 3.13, there’s a free-threaded build (python3.13t) that removes the GIL. This build is still experimental, and many C extensions don’t support it yet. For most projects today, using processes remains the most reliable approach.

3. Basic multiprocessing

Each process has its own interpreter and its own GIL. Therefore, N processes can truly run in parallel on N cores.

3.1. Creating Process objects manually

# basic_process.py
from multiprocessing import Process, Queue
from cpu_task import count_primes

def worker(start: int, end: int, results: Queue) -> None:
    results.put(count_primes(start, end))

if __name__ == "__main__":
    results = Queue()
    ranges = [(0, 500_000), (500_000, 1_000_000)]
    processes = [Process(target=worker, args=(a, b, results)) for a, b in ranges]

    for p in processes:
        p.start()
    total = sum(results.get() for _ in processes)
    for p in processes:
        p.join()

    print("Total primes:", total)

Here’s what’s happening:

• Process(target=..., args=...) describes a child process. It doesn’t run until you call start().
• The child process doesn’t share memory with the parent. To send results back, you must use a communication channel—here, a Queue.
• The code calls results.get() before join(). If you join() before reading from the queue while it still holds data, your program may deadlock.

Total primes: 78498

This approach is flexible but verbose. In practice, you rarely manage Process objects directly.

3.2. Using Pool

Pool creates a group of workers upfront and automatically distributes tasks:

# pool_example.py
from multiprocessing import Pool
from cpu_task import count_primes

RANGES = [(i, i + 125_000) for i in range(0, 1_000_000, 125_000)]

if __name__ == "__main__":
    with Pool(processes=8) as pool:
        counts = pool.starmap(count_primes, RANGES)
    print(counts)
    print("Total:", sum(counts))

starmap unpacks each tuple in RANGES into start, end arguments. Results are returned in the same order as the input:

[10746, 9895, 9622, 9480, 9637, 9554, 9454, 9610]
Total: 78498

If you don’t specify processes, Pool uses os.cpu_count() workers. The with block cleans up workers on exit.

4. ProcessPoolExecutor: A more modern API

concurrent.futures.ProcessPoolExecutor has the same interface as ThreadPoolExecutor. This means you can switch from threads to processes simply by changing the class name:

# parallel.py
import os
from concurrent.futures import ProcessPoolExecutor
from cpu_task import count_primes

def split_range(start: int, end: int, parts: int) -> list[tuple[int, int]]:
    step = (end - start + parts - 1) // parts
    return [(s, min(s + step, end)) for s in range(start, end, step)]

def parallel_count_primes(start: int, end: int, workers: int | None = None) -> int:
    workers = workers or os.cpu_count() or 1
    chunks = split_range(start, end, workers)
    starts, ends = zip(*chunks)
    with ProcessPoolExecutor(max_workers=workers) as executor:
        return sum(executor.map(count_primes, starts, ends))

if __name__ == "__main__":
    import time
    t0 = time.perf_counter()
    print(parallel_count_primes(0, 1_000_000, workers=8))
    print(f"{time.perf_counter() - t0:.2f}s")

78498
1.27s

Time drops from about 6.4 seconds to 1.3 seconds. Speed doesn’t increase by exactly 8× for two reasons. First, the workload isn’t evenly distributed across chunks—larger numbers take longer to check. Second, spawning processes and serializing data takes time.

If you want to process results as soon as they’re ready, use submit with as_completed:

from concurrent.futures import ProcessPoolExecutor, as_completed
from cpu_task import count_primes

if __name__ == "__main__":
    with ProcessPoolExecutor() as executor:
        futures = {executor.submit(count_primes, a, a + 250_000): a
                   for a in range(0, 1_000_000, 250_000)}
        for future in as_completed(futures):
            print(f"Range starting {futures[future]}: {future.result()} primes")

If a worker raises an exception, future.result() re-raises it in the parent process. You can catch errors with try/except as usual.

5. Many small tasks: chunksize and imap_unordered

Each time you send a task to a worker, Python must pickle the arguments, send them through a pipe, and unpickle the result. With 100,000 tiny tasks, this overhead can exceed the actual computation. You can handle this by batching tasks with chunksize:

# chunks.py
from multiprocessing import Pool

def is_prime(n: int) -> bool:
    if n < 2:
        return False
    return all(n % d for d in range(2, int(n ** 0.5) + 1))

if __name__ == "__main__":
    numbers = range(1_000_000)
    with Pool() as pool:
        total = sum(pool.imap_unordered(is_prime, numbers, chunksize=10_000))
    print(total)

• chunksize=10_000 causes each batch to contain 10,000 numbers instead of 1. This drastically reduces inter-process communication.
• imap_unordered returns results lazily and out of order. This is ideal when you only need aggregation (like summing) and want to save memory.
• ProcessPoolExecutor.map also accepts chunksize. Its default is 1, so adjust it for many small tasks.

6. Initializing resources once per worker

Suppose each task needs a lookup table that’s expensive to create. Rebuilding it for every function call would be wasteful. The initializer parameter lets you run a function once when a worker starts:

# initializer_example.py
from concurrent.futures import ProcessPoolExecutor

_price_table: dict[str, float] = {}

def load_prices(currency: str) -> None:
    global _price_table
    rate = 25_000 if currency == "VND" else 1
    _price_table = {f"SKU-{i}": i * 1.5 * rate for i in range(100_000)}

def order_total(skus: list[str]) -> float:
    return sum(_price_table[s] for s in skus)

if __name__ == "__main__":
    orders = [["SKU-1", "SKU-2"], ["SKU-10", "SKU-20", "SKU-30"]]
    with ProcessPoolExecutor(max_workers=2, initializer=load_prices,
                             initargs=("VND",)) as executor:
        print(list(executor.map(order_total, orders)))

[112500.0, 2250000.0]

The _price_table variable is global within each worker. The parent process doesn’t see those changes. This is a safe way for each worker to have its own resources without sending the lookup table with every task.

7. Testing parallel code

You can test code using processes with pytest like any other code. The requirement is that worker functions must be defined at module level:

# test_parallel.py
from cpu_task import count_primes
from parallel import parallel_count_primes, split_range

def test_count_primes_known_value():
    assert count_primes(0, 100) == 25

def test_split_range_covers_whole_interval():
    chunks = split_range(0, 10, 3)
    assert chunks == [(0, 4), (4, 8), (8, 10)]

def test_parallel_matches_sequential():
    expected = count_primes(0, 50_000)
    assert parallel_count_primes(0, 50_000, workers=4) == expected

pip install pytest
pytest -q test_parallel.py

...                                                    [100%]
3 passed in 0.62s

The most important test is test_parallel_matches_sequential. It ensures your partitioning doesn’t skip or double-count elements. This is a common bug when writing split_range.

8. Common pitfalls and when not to use multiprocessing

Forgetting if __name__ == "__main__": With the spawn start method (default on Windows and macOS) or forkserver (default on Linux from Python 3.14), child processes re-import the main module. If pool creation code is at module level, each child spawns a new pool, leading to RuntimeError or infinite child creation.

Passing lambdas or nested functions. Everything sent to a worker must be pickleable:

from concurrent.futures import ProcessPoolExecutor

if __name__ == "__main__":
    with ProcessPoolExecutor() as executor:
        print(list(executor.map(lambda x: x * x, range(5))))

_pickle.PicklingError: Can't pickle <function <lambda> at 0x...>: attribute lookup <lambda> on __main__ failed

Define functions with def at module level. Use functools.partial to fix arguments if needed.

Passing huge data. A list of millions of elements sent with each task gets pickled repeatedly. Copy time can dwarf computation. Workarounds include: send only indices or file paths, use initializer, or use multiprocessing.shared_memory for arrays.

Relying on fork behavior. Code that works on older Linux via fork (children inherit globals) may fail on macOS, Windows, or Python 3.14. For consistent behavior, explicitly set the start method:

import multiprocessing as mp

if __name__ == "__main__":
    mp.set_start_method("spawn")

Note: Don’t use multiprocessing for I/O-bound tasks like HTTP calls or database queries. While I/O waits, threads release the GIL, so ThreadPoolExecutor or asyncio is much lighter. Also avoid multiprocessing when each task takes only microseconds without batching, or when computation already lives in a GIL-releasing C library (NumPy, hash functions on large data, etc.). Threads or vectorization usually suffice.

Creating too many workers. Setting workers greater than CPU cores won’t speed up CPU-bound code. It only wastes RAM, since each process is a full interpreter. Start with os.cpu_count().

9. Conclusion

The GIL prevents threads from speeding up CPU-intensive pure Python code. Since each process has its own GIL, distributing work across processes is the simple way to use all CPU cores. For most cases, ProcessPoolExecutor is the cleanest choice. Pool is useful when you need imap_unordered or starmap. Manual Process objects are for special situations.

When using multiprocessing, remember these points:

• Always include if __name__ == "__main__".
• Pass only pickleable functions and data.
• Increase chunksize for many small tasks.
• Use initializer for heavy resources.
• Write a test comparing parallel and sequential results to verify your partitioning is correct.