1. Home
  2. Blog
  3. Software Development

Software Development

Python Multiprocessing: Pool, Processes, Queues and Real Timings

Python multiprocessing runs code in separate processes, each with its own interpreter and GIL, so CPU-heavy work can finally use every core. We timed a Pool against a plain loop, showed why tiny tasks can be 37 times slower without the right chunksize, lost updates on a shared counter, and hit the main-guard error that trips up every Windows user.

9 min read
Python multiprocessing: Pool, processes, queues and real timings

Key takeaways

  • Python multiprocessing cut four CPU-heavy jobs from 4.16 seconds to 2.74 on a 4-core laptop, a 1.52x speed-up.
  • Starting a pool cost about 0.36 seconds, so tiny jobs run far faster in a plain loop.
  • 200,000 tiny tasks took 11.17 seconds with chunksize=1 and 0.30 seconds with chunksize=1000.
  • Four processes adding to a shared counter without a lock got 31,254 instead of 80,000; with the lock, 80,000.

Python threads cannot run pure Python calculations in parallel because of the Global Interpreter Lock, as our Python threading guide measured. Python multiprocessing sidesteps the GIL by starting separate worker processes. Each one is a full Python interpreter with its own memory and its own GIL, so four processes really can keep four cores busy.

All timings come from Python 3.13.6 on a 4-core Windows laptop, where the default start method is spawn. Timings on Linux, which uses a different default, will differ, particularly start-up costs.

Why multiprocessing?

Use it for CPU-bound work written in Python: image processing, simulations, parsing huge files, feature engineering, brute-force searches. If your program mostly waits for networks or disks, threads or asyncio are lighter and simpler. If the heavy work already runs inside NumPy or another C library that releases the GIL, threads may be enough.

The main guard: required, not optional

import multiprocessing as mp
def sq(x): return x * x
with mp.Pool(2) as p:          # no if __name__ == "__main__" guard
    print(p.map(sq, [1, 2]))

# RuntimeError:
#   An attempt has been made to start a new process before the
#   current process has finished its bootstrapping phase.

With the spawn start method (the default on Windows and macOS), each worker starts a fresh interpreter and imports your script. Without a guard, that import tries to create another pool, and Python stops it with this error. Always put the code that starts processes under if __name__ == "__main__":, and keep worker functions at module level so they can be imported.

Pool.map, timed

def cpu(n):
    s = 0
    for i in range(n): s += i * i
    return s

if __name__ == "__main__":
    with mp.Pool(4) as pool:
        pool.map(cpu, [20_000_000] * 4)

# 4 x cpu(20M): sequential 4.16s   pool(4) 2.74s   speedup 1.52x

Pool.map works like the built-in map: it splits the input across workers and returns results in input order. We got a 1.52x speed-up rather than 4x. Part of that is the 0.36-second start-up cost, and part is the laptop itself: background processes, thermal limits and shared caches mean four busy cores rarely deliver four times the throughput. Expect real-world gains between 1.5x and 3.5x on four cores, and measure on your hardware.

Start-up overhead: when a loop wins

pool start + 10 tiny jobs   0.359 s
plain loop, same jobs       0.000012 s

Creating four worker processes with spawn means four interpreters starting and importing your module. For tiny workloads that overhead dwarfs the work. Create one pool and reuse it for many tasks, and only parallelise work that takes at least a good fraction of a second in total.

map, starmap, apply_async and imap

with mp.Pool(4) as pool:
    pool.map(square, range(8))                  # [0, 1, 4, 9, 16, 25, 36, 49]
    pool.starmap(add, [(1, 2), (3, 4)])          # [3, 7]
    pool.apply_async(square, (12,)).get(timeout=5)   # 144
    sorted(pool.imap_unordered(square, range(5)))     # [0, 1, 4, 9, 16]
  • map: one argument per call, results in order, waits for all.
  • starmap: unpacks tuples into several arguments.
  • apply_async: submit one call, get an AsyncResult and collect it later.
  • imap / imap_unordered: return results lazily as they finish, which saves memory for huge inputs and lets you show progress. Results from the unordered version arrive in completion order.

chunksize: the 37x setting

200k tiny tasks chunksize=1:    11.17 s
200k tiny tasks chunksize=1000:  0.30 s

Every task sent to a worker is pickled, passed through a pipe, unpickled, run, and its result sent back. With chunksize=1, that round trip happened 200,000 times and dominated everything. Sending 1,000 items per message made the same work 37 times faster. map picks a chunk size automatically when you do not pass one, but imap and imap_unordered default to 1, so set it explicitly when tasks are small.

Chunking has a side effect worth knowing. When we mapped a function returning os.getpid() over 40 tiny items, all 40 ran in the same worker, because the automatic chunks were large enough for one process to grab everything before the others started. That is fine for throughput, but surprising if you expect even distribution.

Processes do not share memory

g = []
def append_global(x):
    g.append(x); return len(g)

pool.map(append_global, range(4))   # [1, 1, 1, 1]
g                                   # [] in the main process

Each worker has its own copy of module globals. Appending in a worker changed only that worker’s list, and the main process saw nothing. Pass data in as arguments and get results back as return values. Arguments and results must be picklable, so lambdas, open files and database connections cannot be sent; open such resources inside the worker instead.

Exceptions in workers

def boom(x):
    if x == 2: raise ValueError(f"bad input {x}")
    return x

try:
    pool.map(boom, range(4))
except ValueError as e:
    print("exception propagated:", e)    # bad input 2

An exception in a worker is pickled and re-raised in the parent when you collect results, so normal exception handling works. With map, one failure loses the whole batch’s results, so for long jobs catch errors inside the worker function and return an error marker instead.

Shared Value and the missing lock

def unsafe(counter, n):
    for _ in range(n): counter.value += 1

def safe(counter, n):
    for _ in range(n):
        with counter.get_lock(): counter.value += 1

# 4 processes x 20,000 increments
# shared Value without lock 31254   expected 80000
# shared Value with lock    80000

multiprocessing.Value places a number in shared memory that all processes can see, but += is a read followed by a write, so processes overwrote each other’s updates and lost 61 percent of them. Holding the value’s built-in lock made the count exact. Shared state is slow and error-prone; returning results and combining them in the parent is almost always simpler.

Queues between processes

q = mp.Queue()
p = mp.Process(target=q.put, args=("hello from child",))
p.start()
print(q.get())     # hello from child
p.join()
p.exitcode         # 0

multiprocessing.Queue is a process-safe pipe for producer and consumer designs, just like queue.Queue for threads. Call get() before join() when sending large objects, or the child can block forever waiting for the queue to drain. Check exitcode: 0 means success, a negative number means the process was killed by a signal.

ProcessPoolExecutor: the modern interface

from concurrent.futures import ProcessPoolExecutor
with ProcessPoolExecutor(4) as ex:
    list(ex.map(square, range(5)))     # [0, 1, 4, 9, 16]

concurrent.futures.ProcessPoolExecutor has the same API as the thread pool executor, so switching between threads and processes is a one-word change. It returns futures, works with as_completed, and is the recommended choice for new code unless you need Pool-only features such as imap_unordered or maxtasksperchild. Both pools are context managers that shut workers down on exit.

Start methods: spawn, fork and forkserver

  • spawn starts a fresh interpreter and imports your module. Safe and predictable, but slower to start. Default on Windows and macOS.
  • fork copies the parent process instantly, including its memory. Fast, but unsafe if the parent has threads or locks; only available on Unix.
  • forkserver forks workers from a clean helper process. Recent Python versions use it as the default on Linux.

We only ran spawn, since that is what Windows supports. Choose a method explicitly with mp.get_context("spawn") if your code must behave the same on every platform.

When to use multiprocessing

Situation Recommendation
CPU-heavy pure Python, seconds or more of work ProcessPoolExecutor or Pool
Many tiny tasks Batch them, set a large chunksize
Network or disk waits Threads or asyncio
Large NumPy arrays Vectorise first; share with shared_memory if needed
Less than a second of total work A plain loop

Add logging with the process name in the format string, so you can tell which worker wrote each line.

Final thoughts

Python multiprocessing is the standard way to use every core for Python code. Guard the entry point with if __name__ == "__main__", reuse one pool, batch small tasks with a sensible chunksize, pass data in and out rather than sharing it, and protect any shared value with a lock. Measure on your own machine: our four-core laptop gave 1.5x, not 4x, and for small jobs the plain loop won outright.

Frequently asked questions

What is the difference between multiprocessing and threading in Python?

Threads share one process and one GIL, so they help with waiting but not with CPU-heavy Python code. Multiprocessing starts separate processes, each with its own GIL, so CPU work runs in parallel.

Why do I need if name equals main for multiprocessing?

With the spawn start method, each worker imports your script. Without the guard, that import tries to start new processes again and Python raises a RuntimeError about the bootstrapping phase.

How many processes should a Pool use?

Start with the number of CPU cores, which is the default. More processes than cores rarely helps CPU-bound work and adds memory and start-up cost.

Why is my multiprocessing code slower than a loop?

Usually because the tasks are tiny. Process start-up and sending each task between processes cost more than the work itself. Batch tasks or raise the chunksize.

Can processes share variables?

Not normal Python objects. Use multiprocessing Value, Array or a Manager for shared state, protect updates with a lock, or better, return results and combine them in the parent.

Keep learning

Work with Stack Quora

Need help with testing, WordPress or SEO?

We help teams ship better software and grow their sites. Tell us what you are working on and we will suggest a clear next step.

Keep reading

More guides on testing, AI & modern development

Enjoyed this article? Pick a topic and jump straight into our newest tutorials.

Artificial Intelligence

Confusion Matrix Explained: Precision, Recall, F1 and Thresholds in Python

Confusion matrix explained with Python: TP, FP, FN, TN, precision, recall, F1, the accuracy trap, threshold choice by cost and…

Read article →
Software Development

API Pagination: Offset vs Cursor vs Keyset, Tested on 1 Million Rows

API pagination explained: offset vs keyset and cursor pagination timed on 1 million rows, duplicates when data changes, the ties…

Read article →
DevOps

Git Submodule Explained: Add, Clone, Update and Remove Safely

Git submodule explained: add, clone, update –init, update –remote, the detached HEAD trap, foreach and clean removal, tested step by…

Read article →
Test Automation

Pytest tmp path Explained: tmp_path, tmp_path_factory and Temp Files

Pytest tmp_path explained: per-test temporary folders, tmp_path_factory for shared files, monkeypatch.chdir, legacy tmpdir, retention and basetemp.

Read article →
Software Testing

Test Case Prioritization: 6 Techniques Compared With APFD

Test case prioritization compared: risk, coverage, cost, history and random orders scored with APFD and time to find all faults…

Read article →
Databases & SQL

Full Text Search in SQL: FTS5, Ranking, Phrases and Speed

Full text search in SQL with SQLite FTS5: MATCH queries, phrases, prefixes, NEAR, stemming, BM25 ranking, snippets, two gotchas and…

Read article →
Databases & SQL

Generated Columns in SQL: Virtual vs Stored, Indexes and Limits

Generated columns in SQL explained: virtual vs stored, automatic updates, write errors, storage cost, indexing, JSON fields and support in…

Read article →
Software Development

Object Freeze in JavaScript: Shallow vs Deep, Seal and const

Object freeze in JavaScript explained: freeze vs seal vs const, strict mode errors, shallow vs deep freeze, frozen arrays, Map…

Read article →
Software Development

Python hashlib: SHA-256, File Hashes, HMAC and Password Hashing

Python hashlib explained: SHA-256 for strings and files, algorithm speeds, why passwords need PBKDF2 or scrypt, safe verification and HMAC…

Read article →
Software Development

Python UUID Explained: uuid1, uuid4, uuid5, uuid7 and Database Keys

Python UUID explained: uuid4, uuid5, uuid1 privacy, time-ordered uuid7, parsing, storage formats and UUIDs as database keys, tested on Python…

Read article →
Artificial Intelligence

Random Forest in Python: Bagging, Feature Sampling and OOB From Scratch

Random forest in Python from scratch: bootstrap sampling, feature sampling, number of trees, out-of-bag evaluation and impurity vs permutation importance.

Read article →
Software Development

MIME Types Explained: Content-Type, nosniff and a Common Types List

MIME types explained and tested in a browser: Content-Type, nosniff, scripts and stylesheets with wrong types, a common MIME types…

Read article →
DevOps

Git Clean Explained: Remove Untracked Files Without Regret

Git clean explained with dry runs: -n, -f, -d, -x, -X, exclusions, paths, nested repositories and interactive mode, plus a…

Read article →
Test Automation

Pytest approx: Comparing Floats With rel, abs and Collections

Pytest approx explained: default tolerances, the near-zero trap, rel and abs, lists and dicts, NaN, NumPy, and how it differs…

Read article →
Software Testing

Accessibility Testing: WCAG Checklist, Automated Tools and Manual Checks

Accessibility testing explained: WCAG basics, an axe-core scan of a broken and a fixed page, what automation missed, contrast checks…

Read article →
Databases & SQL

SQL UNPIVOT: Turn Columns Into Rows, 4 Ways Tested

SQL UNPIVOT explained: turn columns into rows with UNION ALL, CROSS JOIN with CASE, JSON functions or native UNPIVOT, with…

Read article →
Software Development

Intl NumberFormat in JavaScript: Currency, Compact, Percent and Units

Intl.NumberFormat in JavaScript: locales, currency, compact notation, percent, units, rounding, formatToParts, the non-breaking space trap and formatter reuse.

Read article →
Databases & SQL

SQL EXPLAIN Plan: How to Read Query Plans and Fix Slow Queries

SQL EXPLAIN plan explained: read scans, index searches, covering indexes and temporary sorts, with before and after timings for five…

Read article →
Artificial Intelligence

Decision Tree in Python: Gini, Entropy and Overfitting From Scratch

Decision tree in Python from scratch: Gini impurity, entropy, information gain, best splits, readable rules, and how depth and leaf…

Read article →
DevOps

Git Blame Explained: Find Who Changed a Line, and Why

Git blame explained with a real repository: ignore whitespace, line ranges and functions, ignore-revs files for formatting commits, moved code…

Read article →
Test Automation

Pytest Benchmark: Measure Speed and Catch Performance Regressions

Pytest benchmark tutorial: the benchmark fixture, reading results, parametrized and pedantic benchmarks, saving baselines and failing the build on regressions.

Read article →
Software Testing

Defect Density: Formula, Worked Examples and How to Use It

Defect density explained: the formula, a five-module worked example, per KLOC vs per function point, severity weighting, defect removal efficiency…

Read article →
Databases & SQL

Case Insensitive Search in SQL: LOWER, LIKE, COLLATE and Indexes

Case insensitive search in SQL: LOWER, LIKE, COLLATE NOCASE and expression indexes compared with query plans, plus accented letters and…

Read article →
Databases & SQL

ON DELETE CASCADE in SQL: CASCADE vs SET NULL vs RESTRICT

ON DELETE CASCADE in SQL tested: CASCADE vs SET NULL vs RESTRICT, multi-level cascades, the SQLite foreign key pragma, row…

Read article →
Artificial Intelligence

Logistic Regression in Python: Sigmoid, Log Loss and Gradient Descent From Scratch

Logistic regression in Python from scratch: sigmoid, log loss, gradient descent, odds ratios, thresholds, sigmoid overflow and separable data, tested…

Read article →
DevOps

Git Worktree Explained: Work on Two Branches at Once

Git worktree explained: add, list, remove, prune, lock and detached worktrees, a hotfix workflow, and the errors you will hit,…

Read article →
Test Automation

Pytest ini Explained: addopts, markers, testpaths and Precedence

Pytest ini explained: testpaths, addopts, strict markers, xfail_strict, filterwarnings, minversion, config file precedence and command-line overrides, tested.

Read article →
Software Testing

Test Estimation Techniques: WBS, Three Point PERT and Monte Carlo

Test estimation techniques worked through: WBS, three point PERT, total uncertainty, a Monte Carlo check, percentage rules and Wideband Delphi.

Read article →
Artificial Intelligence

Linear Regression in Python: Least Squares From Scratch, With R Squared

Linear regression in Python from scratch: closed form, NumPy least squares and gradient descent, R squared, RMSE, outliers and curved…

Read article →
DevOps

Git Remote Explained: Add, Set-URL, Rename, Remove and Push URLs

Git remote explained: add, set-url, separate push URLs, rename, remove and prune, plus how to avoid leaking tokens in remote…

Read article →
Test Automation

Pytest Mock: mocker.patch, spy, side_effect and autospec Explained

Pytest mock explained: mocker.patch, return_value, side_effect, spy, autospec and PropertyMock, with tested examples for APIs, clocks and retries.

Read article →
Software Testing

System Testing Explained: Types, Process and a Real End-to-End Example

System testing explained: types, entry and exit criteria, the process, and a real end-to-end test suite run against a small…

Read article →
Artificial Intelligence

Naive Bayes Classifier Explained: Build a Spam Filter in Python

Naive Bayes classifier explained by building a spam filter in Python: Bayes rule, the naive assumption, Laplace smoothing, log probabilities…

Read article →
DevOps

Git Branch Explained: Create, Switch, Rename, Delete and Recover

Git branch explained with tested commands: list, create, switch, rename, delete, merged checks, recovering deleted branches, remote tracking and naming.

Read article →
Test Automation

Pytest raises: Testing Exceptions, match and excinfo

Pytest raises explained with tested examples: match regexes, excinfo, custom and chained exceptions, parametrized error cases, pytest.warns and common mistakes.

Read article →
Software Testing

Non Functional Testing: Types, Metrics and a Real Load Test

Non functional testing explained: 10 key types, measurable requirements, latency percentiles and what a small real load test revealed about…

Read article →