Pytest Benchmark: Measure Speed and Catch Performance Regressions
Pytest benchmark turns speed into something your test suite can check. The pytest-benchmark plugin times functions over many rounds, reports minimum, mean and spread, saves results, and can fail the build when code gets slower. We benchmarked real functions, caught a deliberate 70 percent slowdown with one flag, and noted a privacy detail in the saved files.
- Pytest benchmark adds a
benchmarkfixture: callbenchmark(func, *args)and it times the function over many rounds and returns its result. - Building a string with
"".jointook 733 microseconds (minimum) versus 1,246 with+=in our run. - After saving a baseline,
--benchmark-compare-fail=min:20%failed the run when a slower version was 69.95 percent worse. - Saved JSON files include machine details such as the computer’s host name, so review them before publishing.
Unit tests prove that code gives the right answer. They say nothing about whether it still gives that answer quickly. A refactor that turns a linear loop into a quadratic one passes every test and quietly ships. Pytest benchmark, the pytest-benchmark plugin, closes that gap by measuring speed inside your normal test suite and comparing it with earlier runs.
We used pytest-benchmark 5.3.1 with pytest 9.2.0 on Python 3.13.6 on a Windows laptop. Timings will differ on your machine; the relative numbers and workflow are what matter.
What is pytest-benchmark?
pytest-benchmark is a pytest plugin that provides a benchmark fixture. It calibrates how many times to call your function so each round is long enough to time accurately, repeats rounds for about a second by default, discards warm-up effects, and reports statistics. It is a micro-benchmarking tool, for functions and code paths. For whole-system throughput under many users, see our guide on load testing vs stress testing.
Install
pip install pytest-benchmark
Once installed, the benchmark fixture is available in every test with no imports or configuration, like any other fixture from our pytest fixtures guide.
Your first benchmark
import json
data = [{"id": i, "name": f"user{i}", "tags": ["a", "b"]} for i in range(1000)]
text = json.dumps(data)
def test_json_loads(benchmark):
result = benchmark(json.loads, text)
assert len(result) == 1000
Pass the function and its arguments, not the result of calling it: benchmark(json.loads, text), not benchmark(json.loads(text)). The fixture returns whatever the function returns, so you can still assert on correctness in the same test. A benchmark that returns a wrong answer quickly is worse than useless.
Reading the results table
Name (time in us) Min Mean StdDev Rounds
test_concat_join 733.4000 (349.24) 771.9119 (328.36) 88.1791 (73.14) 1177
test_concat_plus 1,246.0000 (593.34) 1,383.0772 (588.34) 241.5366 (200.34) 734
test_json_loads 441.7000 (210.33) 470.2646 (200.04) 89.0248 (73.84) 1450
test_pedantic 21.0000 (10.00) 27.6900 (11.78) 17.1542 (14.23) 50
test_sorted[10000] 890.1000 (423.86) 980.9698 (417.29) 226.6633 (188.01) 1034
test_sorted[100] 2.1000 (1.0) 2.3508 (1.0) 1.2056 (1.0) 129871
- Min is the fastest round, the best estimate of the code’s real cost because noise only ever adds time.
- Mean and StdDev show typical time and spread; a large StdDev means the machine was noisy.
- Rounds is how many timed rounds ran: 129,871 for the 2-microsecond sort, 50 for the pedantic test.
- Numbers in brackets are ratios to the fastest test in the table.
We chose these columns with --benchmark-columns=min,mean,stddev,rounds and sorted with --benchmark-sort=name; the default table also shows max, median, IQR, outliers and operations per second.
Comparing two implementations
def concat_plus(n):
s = ""
for i in range(n): s += str(i)
return s
def concat_join(n):
return "".join(str(i) for i in range(n))
def test_concat_plus(benchmark):
assert len(benchmark(concat_plus, 10000)) == 38890
def test_concat_join(benchmark):
assert len(benchmark(concat_join, 10000)) == 38890
Writing two tests side by side settles arguments with data. Here join was about 1.7 times faster than repeated += (733 versus 1,246 microseconds minimum), and both asserted the same 38,890-character output. CPython has an optimisation that makes += on strings faster than it used to be, which is exactly why measuring beats folklore.
Parametrized benchmarks: how cost grows
@pytest.mark.parametrize("n", [100, 10000])
def test_sorted(benchmark, n):
xs = [random.random() for _ in range(n)]
benchmark(sorted, xs)
# test_sorted[100] 2.1 us
# test_sorted[10000] 890.1 us
Parametrizing over input sizes shows scaling, not just a single number. 100 times more data took about 420 times longer, consistent with sorting’s n log n cost. A sudden jump at one size is how you spot an accidentally quadratic algorithm.
Pedantic mode: full control
def test_pedantic(benchmark):
def setup():
return (list(range(5000, 0, -1)),), {}
benchmark.pedantic(sorted, setup=setup, rounds=50, iterations=1)
# 21.0 us min over 50 rounds
Normal mode calls the function repeatedly with the same arguments, which is wrong when the function changes its input (sorting in place, consuming an iterator) or when setup is expensive. benchmark.pedantic lets you set the exact number of rounds and iterations and a setup function that builds fresh arguments before every round, excluded from the timing. Sorting an already reversed 5,000-item list took only 21 microseconds because Python’s sort detects descending runs.
Saving a baseline
pytest test_reg.py --benchmark-save=baseline
# Saved benchmark data in: .benchmarks/Windows-CPython-3.13-64bit/0001_baseline.json
# test_build min 224.3 us mean 246.0 us
--benchmark-save=NAME writes the results to a numbered JSON file under .benchmarks, in a folder named after the platform and Python version so that results from different machines are not mixed. --benchmark-autosave saves every run with an automatic name.
Failing the build on a regression
SLOW=1 pytest test_reg.py --benchmark-compare --benchmark-compare-fail=min:20%
test_build (0001_baselin) 224.3000 (1.0) 246.0390 (1.0)
test_build (NOW) 381.2000 (1.70) 413.1476 (1.68)
Performance has regressed:
test_build (0001_baselin) - Field 'min' has failed PercentageRegressionCheck:
69.950940003 > 20.000000000
We made build() deliberately slower with an environment variable, then compared against the saved baseline. --benchmark-compare printed both runs side by side, and --benchmark-compare-fail=min:20% raised a PerformanceRegression because the minimum time grew by 69.95 percent, over the 20 percent limit. The expression can use other fields such as mean, and absolute limits in seconds such as mean:0.001. Use a generous threshold: 5 percent will fail constantly on shared machines because of noise.
Disabling benchmarks in normal runs
pytest test_reg.py --benchmark-disable
# 1 passed
Benchmarks make a suite slower, because each one runs for about a second. --benchmark-disable runs each benchmarked function once as an ordinary test, so correctness assertions still execute. --benchmark-skip skips benchmark tests entirely, and --benchmark-only runs just them. Put the choice in your pytest ini addopts for local runs, and enable benchmarks in a dedicated CI job.
JSON output, and a privacy note
pytest test_bench.py -k json --benchmark-json=out.json
# out.json: benchmarks[0] test_json_loads, min 452.7 us, 1442 rounds
# machine_info keys include: cpu, machine, node, processor, python_build, python_compiler
--benchmark-json writes full statistics for dashboards or custom checks. The file also records machine_info and commit information. The node field is the computer’s network host name, and file paths may appear too. That is useful for comparing machines, but review or strip it before committing results to a public repository or attaching them to a public bug report.
Benchmarks in CI
Shared CI runners vary in speed from run to run, so comparisons there are noisy. Practical approaches: run benchmarks on a dedicated or self-hosted runner; compare against a baseline produced on the same runner type in the same job; use relative thresholds around 15 to 25 percent; or use a service built for CI benchmarking. Our GitHub Actions guide shows how to add a separate job. Treat an occasional failure like a flaky test: rerun, and investigate if it repeats.
Tips for reliable benchmarks
- Benchmark the smallest unit that matters and keep setup outside the timed function.
- Compare minimums for regressions; mean and max reflect machine noise more than code.
- Close other heavy programs and keep laptops plugged in; power saving changes CPU speed.
- Use realistic inputs. Our JSON test parsed 1,000 realistic records, not a tiny string, so the number means something. Real payloads can come from your JSON fixtures.
- Keep the assertion. Every benchmark should also check its result.
A benchmark checklist for code review
Benchmarks are code too, and a misleading one is worse than none. Before merging a new benchmark, check these points:
- Does it measure the right thing? Data creation, file reads and network calls belong in module-level setup or a pedantic
setupfunction, not inside the timed call. - Is the input realistic in size and shape? A function that is fast on ten items may be quadratic on ten thousand; parametrize over at least two sizes.
- Does it assert the result? An optimisation that returns the wrong answer should fail the benchmark run.
- Is it deterministic? Seed random data, as our sorting test should have done, so that two runs compare the same work.
- Is the threshold sensible? Tie
--benchmark-compare-faillimits to how noisy the machine is, and record why a value was chosen. - Is the baseline fresh? Re-save the baseline after an intentional performance change, such as a new feature that legitimately costs time, so later comparisons stay meaningful.
Applied consistently, these checks keep the benchmark suite trustworthy enough that a red result is treated as a real problem rather than noise to be retried.
Final thoughts
Pytest benchmark makes performance a tested property instead of a hope. Wrap a function call with the benchmark fixture, read the minimum time, save a baseline, and fail the build with --benchmark-compare-fail when a change makes things meaningfully slower. In our test, one flag caught a 70 percent slowdown that every correctness test would have passed. Keep thresholds realistic, run on stable hardware, and check the saved files before sharing them.
Frequently asked questions
How do I use pytest-benchmark?
Install the plugin, add the benchmark fixture to a test, and call benchmark with the function and its arguments. It times many rounds and returns the function result so you can assert on it.
How do I compare benchmark results between runs?
Save a run with the benchmark save option, then run again with the benchmark compare option. Add the compare fail option with a rule such as min:20% to fail on regressions.
Should I compare min or mean times?
For detecting regressions, the minimum is usually more stable, because noise from other processes only ever adds time. The mean is useful to see typical behaviour.
How do I skip benchmarks during normal test runs?
Use the benchmark disable option to run each benchmarked function once as a normal test, or the benchmark skip option to skip them entirely.
What is pedantic mode in pytest-benchmark?
A mode where you set rounds, iterations and a setup function yourself. Use it when the function changes its input or needs fresh, untimed setup before each round.
Keep learning
Need help with testing, WordPress or SEO?
We help teams ship better software and grow their sites. Tell us what you are working on and we will suggest a clear next step.