Python Multiprocessing: Pool, Processes, Queues and Real Timings
Python multiprocessing runs code in separate processes, each with its own interpreter and GIL, so CPU-heavy work can finally use every core. We timed a Pool against a plain loop, showed why tiny tasks can be 37 times slower without the right chunksize, lost updates on a shared counter, and hit the main-guard error that trips up every Windows user.
- Python multiprocessing cut four CPU-heavy jobs from 4.16 seconds to 2.74 on a 4-core laptop, a 1.52x speed-up.
- Starting a pool cost about 0.36 seconds, so tiny jobs run far faster in a plain loop.
- 200,000 tiny tasks took 11.17 seconds with
chunksize=1and 0.30 seconds withchunksize=1000. - Four processes adding to a shared counter without a lock got 31,254 instead of 80,000; with the lock, 80,000.
Python threads cannot run pure Python calculations in parallel because of the Global Interpreter Lock, as our Python threading guide measured. Python multiprocessing sidesteps the GIL by starting separate worker processes. Each one is a full Python interpreter with its own memory and its own GIL, so four processes really can keep four cores busy.
All timings come from Python 3.13.6 on a 4-core Windows laptop, where the default start method is spawn. Timings on Linux, which uses a different default, will differ, particularly start-up costs.
Why multiprocessing?
Use it for CPU-bound work written in Python: image processing, simulations, parsing huge files, feature engineering, brute-force searches. If your program mostly waits for networks or disks, threads or asyncio are lighter and simpler. If the heavy work already runs inside NumPy or another C library that releases the GIL, threads may be enough.
The main guard: required, not optional
import multiprocessing as mp
def sq(x): return x * x
with mp.Pool(2) as p: # no if __name__ == "__main__" guard
print(p.map(sq, [1, 2]))
# RuntimeError:
# An attempt has been made to start a new process before the
# current process has finished its bootstrapping phase.
With the spawn start method (the default on Windows and macOS), each worker starts a fresh interpreter and imports your script. Without a guard, that import tries to create another pool, and Python stops it with this error. Always put the code that starts processes under if __name__ == "__main__":, and keep worker functions at module level so they can be imported.
Pool.map, timed
def cpu(n):
s = 0
for i in range(n): s += i * i
return s
if __name__ == "__main__":
with mp.Pool(4) as pool:
pool.map(cpu, [20_000_000] * 4)
# 4 x cpu(20M): sequential 4.16s pool(4) 2.74s speedup 1.52x
Pool.map works like the built-in map: it splits the input across workers and returns results in input order. We got a 1.52x speed-up rather than 4x. Part of that is the 0.36-second start-up cost, and part is the laptop itself: background processes, thermal limits and shared caches mean four busy cores rarely deliver four times the throughput. Expect real-world gains between 1.5x and 3.5x on four cores, and measure on your hardware.
Start-up overhead: when a loop wins
pool start + 10 tiny jobs 0.359 s
plain loop, same jobs 0.000012 s
Creating four worker processes with spawn means four interpreters starting and importing your module. For tiny workloads that overhead dwarfs the work. Create one pool and reuse it for many tasks, and only parallelise work that takes at least a good fraction of a second in total.
map, starmap, apply_async and imap
with mp.Pool(4) as pool:
pool.map(square, range(8)) # [0, 1, 4, 9, 16, 25, 36, 49]
pool.starmap(add, [(1, 2), (3, 4)]) # [3, 7]
pool.apply_async(square, (12,)).get(timeout=5) # 144
sorted(pool.imap_unordered(square, range(5))) # [0, 1, 4, 9, 16]
map: one argument per call, results in order, waits for all.starmap: unpacks tuples into several arguments.apply_async: submit one call, get anAsyncResultand collect it later.imap/imap_unordered: return results lazily as they finish, which saves memory for huge inputs and lets you show progress. Results from the unordered version arrive in completion order.
chunksize: the 37x setting
200k tiny tasks chunksize=1: 11.17 s
200k tiny tasks chunksize=1000: 0.30 s
Every task sent to a worker is pickled, passed through a pipe, unpickled, run, and its result sent back. With chunksize=1, that round trip happened 200,000 times and dominated everything. Sending 1,000 items per message made the same work 37 times faster. map picks a chunk size automatically when you do not pass one, but imap and imap_unordered default to 1, so set it explicitly when tasks are small.
Chunking has a side effect worth knowing. When we mapped a function returning os.getpid() over 40 tiny items, all 40 ran in the same worker, because the automatic chunks were large enough for one process to grab everything before the others started. That is fine for throughput, but surprising if you expect even distribution.
Processes do not share memory
g = []
def append_global(x):
g.append(x); return len(g)
pool.map(append_global, range(4)) # [1, 1, 1, 1]
g # [] in the main process
Each worker has its own copy of module globals. Appending in a worker changed only that worker’s list, and the main process saw nothing. Pass data in as arguments and get results back as return values. Arguments and results must be picklable, so lambdas, open files and database connections cannot be sent; open such resources inside the worker instead.
Exceptions in workers
def boom(x):
if x == 2: raise ValueError(f"bad input {x}")
return x
try:
pool.map(boom, range(4))
except ValueError as e:
print("exception propagated:", e) # bad input 2
An exception in a worker is pickled and re-raised in the parent when you collect results, so normal exception handling works. With map, one failure loses the whole batch’s results, so for long jobs catch errors inside the worker function and return an error marker instead.
Shared Value and the missing lock
def unsafe(counter, n):
for _ in range(n): counter.value += 1
def safe(counter, n):
for _ in range(n):
with counter.get_lock(): counter.value += 1
# 4 processes x 20,000 increments
# shared Value without lock 31254 expected 80000
# shared Value with lock 80000
multiprocessing.Value places a number in shared memory that all processes can see, but += is a read followed by a write, so processes overwrote each other’s updates and lost 61 percent of them. Holding the value’s built-in lock made the count exact. Shared state is slow and error-prone; returning results and combining them in the parent is almost always simpler.
Queues between processes
q = mp.Queue()
p = mp.Process(target=q.put, args=("hello from child",))
p.start()
print(q.get()) # hello from child
p.join()
p.exitcode # 0
multiprocessing.Queue is a process-safe pipe for producer and consumer designs, just like queue.Queue for threads. Call get() before join() when sending large objects, or the child can block forever waiting for the queue to drain. Check exitcode: 0 means success, a negative number means the process was killed by a signal.
ProcessPoolExecutor: the modern interface
from concurrent.futures import ProcessPoolExecutor
with ProcessPoolExecutor(4) as ex:
list(ex.map(square, range(5))) # [0, 1, 4, 9, 16]
concurrent.futures.ProcessPoolExecutor has the same API as the thread pool executor, so switching between threads and processes is a one-word change. It returns futures, works with as_completed, and is the recommended choice for new code unless you need Pool-only features such as imap_unordered or maxtasksperchild. Both pools are context managers that shut workers down on exit.
Start methods: spawn, fork and forkserver
- spawn starts a fresh interpreter and imports your module. Safe and predictable, but slower to start. Default on Windows and macOS.
- fork copies the parent process instantly, including its memory. Fast, but unsafe if the parent has threads or locks; only available on Unix.
- forkserver forks workers from a clean helper process. Recent Python versions use it as the default on Linux.
We only ran spawn, since that is what Windows supports. Choose a method explicitly with mp.get_context("spawn") if your code must behave the same on every platform.
When to use multiprocessing
| Situation | Recommendation |
|---|---|
| CPU-heavy pure Python, seconds or more of work | ProcessPoolExecutor or Pool |
| Many tiny tasks | Batch them, set a large chunksize |
| Network or disk waits | Threads or asyncio |
| Large NumPy arrays | Vectorise first; share with shared_memory if needed |
| Less than a second of total work | A plain loop |
Add logging with the process name in the format string, so you can tell which worker wrote each line.
Final thoughts
Python multiprocessing is the standard way to use every core for Python code. Guard the entry point with if __name__ == "__main__", reuse one pool, batch small tasks with a sensible chunksize, pass data in and out rather than sharing it, and protect any shared value with a lock. Measure on your own machine: our four-core laptop gave 1.5x, not 4x, and for small jobs the plain loop won outright.
Frequently asked questions
What is the difference between multiprocessing and threading in Python?
Threads share one process and one GIL, so they help with waiting but not with CPU-heavy Python code. Multiprocessing starts separate processes, each with its own GIL, so CPU work runs in parallel.
Why do I need if name equals main for multiprocessing?
With the spawn start method, each worker imports your script. Without the guard, that import tries to start new processes again and Python raises a RuntimeError about the bootstrapping phase.
How many processes should a Pool use?
Start with the number of CPU cores, which is the default. More processes than cores rarely helps CPU-bound work and adds memory and start-up cost.
Why is my multiprocessing code slower than a loop?
Usually because the tasks are tiny. Process start-up and sending each task between processes cost more than the work itself. Batch tasks or raise the chunksize.
Can processes share variables?
Not normal Python objects. Use multiprocessing Value, Array or a Manager for shared state, protect updates with a lock, or better, return results and combine them in the parent.
Keep learning
Need help with testing, WordPress or SEO?
We help teams ship better software and grow their sites. Tell us what you are working on and we will suggest a clear next step.