1. Home
  2. Blog
  3. Test Automation

Test Automation

Pytest Benchmark: Measure Speed and Catch Performance Regressions

Pytest benchmark turns speed into something your test suite can check. The pytest-benchmark plugin times functions over many rounds, reports minimum, mean and spread, saves results, and can fail the build when code gets slower. We benchmarked real functions, caught a deliberate 70 percent slowdown with one flag, and noted a privacy detail in the saved files.

9 min read
Pytest benchmark: measure speed and catch performance regressions

Key takeaways

  • Pytest benchmark adds a benchmark fixture: call benchmark(func, *args) and it times the function over many rounds and returns its result.
  • Building a string with "".join took 733 microseconds (minimum) versus 1,246 with += in our run.
  • After saving a baseline, --benchmark-compare-fail=min:20% failed the run when a slower version was 69.95 percent worse.
  • Saved JSON files include machine details such as the computer’s host name, so review them before publishing.

Unit tests prove that code gives the right answer. They say nothing about whether it still gives that answer quickly. A refactor that turns a linear loop into a quadratic one passes every test and quietly ships. Pytest benchmark, the pytest-benchmark plugin, closes that gap by measuring speed inside your normal test suite and comparing it with earlier runs.

We used pytest-benchmark 5.3.1 with pytest 9.2.0 on Python 3.13.6 on a Windows laptop. Timings will differ on your machine; the relative numbers and workflow are what matter.

What is pytest-benchmark?

pytest-benchmark is a pytest plugin that provides a benchmark fixture. It calibrates how many times to call your function so each round is long enough to time accurately, repeats rounds for about a second by default, discards warm-up effects, and reports statistics. It is a micro-benchmarking tool, for functions and code paths. For whole-system throughput under many users, see our guide on load testing vs stress testing.

Install

pip install pytest-benchmark

Once installed, the benchmark fixture is available in every test with no imports or configuration, like any other fixture from our pytest fixtures guide.

Your first benchmark

import json
data = [{"id": i, "name": f"user{i}", "tags": ["a", "b"]} for i in range(1000)]
text = json.dumps(data)

def test_json_loads(benchmark):
    result = benchmark(json.loads, text)
    assert len(result) == 1000

Pass the function and its arguments, not the result of calling it: benchmark(json.loads, text), not benchmark(json.loads(text)). The fixture returns whatever the function returns, so you can still assert on correctness in the same test. A benchmark that returns a wrong answer quickly is worse than useless.

Reading the results table

Name (time in us)             Min                  Mean              StdDev            Rounds
test_concat_join         733.4000 (349.24)     771.9119 (328.36)    88.1791 (73.14)      1177
test_concat_plus       1,246.0000 (593.34)   1,383.0772 (588.34)   241.5366 (200.34)      734
test_json_loads          441.7000 (210.33)     470.2646 (200.04)    89.0248 (73.84)      1450
test_pedantic             21.0000 (10.00)       27.6900 (11.78)     17.1542 (14.23)        50
test_sorted[10000]       890.1000 (423.86)     980.9698 (417.29)   226.6633 (188.01)     1034
test_sorted[100]           2.1000 (1.0)          2.3508 (1.0)        1.2056 (1.0)      129871
  • Min is the fastest round, the best estimate of the code’s real cost because noise only ever adds time.
  • Mean and StdDev show typical time and spread; a large StdDev means the machine was noisy.
  • Rounds is how many timed rounds ran: 129,871 for the 2-microsecond sort, 50 for the pedantic test.
  • Numbers in brackets are ratios to the fastest test in the table.

We chose these columns with --benchmark-columns=min,mean,stddev,rounds and sorted with --benchmark-sort=name; the default table also shows max, median, IQR, outliers and operations per second.

Comparing two implementations

def concat_plus(n):
    s = ""
    for i in range(n): s += str(i)
    return s

def concat_join(n):
    return "".join(str(i) for i in range(n))

def test_concat_plus(benchmark):
    assert len(benchmark(concat_plus, 10000)) == 38890
def test_concat_join(benchmark):
    assert len(benchmark(concat_join, 10000)) == 38890

Writing two tests side by side settles arguments with data. Here join was about 1.7 times faster than repeated += (733 versus 1,246 microseconds minimum), and both asserted the same 38,890-character output. CPython has an optimisation that makes += on strings faster than it used to be, which is exactly why measuring beats folklore.

Parametrized benchmarks: how cost grows

@pytest.mark.parametrize("n", [100, 10000])
def test_sorted(benchmark, n):
    xs = [random.random() for _ in range(n)]
    benchmark(sorted, xs)
# test_sorted[100]     2.1 us
# test_sorted[10000] 890.1 us

Parametrizing over input sizes shows scaling, not just a single number. 100 times more data took about 420 times longer, consistent with sorting’s n log n cost. A sudden jump at one size is how you spot an accidentally quadratic algorithm.

Pedantic mode: full control

def test_pedantic(benchmark):
    def setup():
        return (list(range(5000, 0, -1)),), {}
    benchmark.pedantic(sorted, setup=setup, rounds=50, iterations=1)
# 21.0 us min over 50 rounds

Normal mode calls the function repeatedly with the same arguments, which is wrong when the function changes its input (sorting in place, consuming an iterator) or when setup is expensive. benchmark.pedantic lets you set the exact number of rounds and iterations and a setup function that builds fresh arguments before every round, excluded from the timing. Sorting an already reversed 5,000-item list took only 21 microseconds because Python’s sort detects descending runs.

Saving a baseline

pytest test_reg.py --benchmark-save=baseline
# Saved benchmark data in: .benchmarks/Windows-CPython-3.13-64bit/0001_baseline.json
# test_build   min 224.3 us   mean 246.0 us

--benchmark-save=NAME writes the results to a numbered JSON file under .benchmarks, in a folder named after the platform and Python version so that results from different machines are not mixed. --benchmark-autosave saves every run with an automatic name.

Failing the build on a regression

SLOW=1 pytest test_reg.py --benchmark-compare --benchmark-compare-fail=min:20%

test_build (0001_baselin)     224.3000 (1.0)      246.0390 (1.0)
test_build (NOW)              381.2000 (1.70)     413.1476 (1.68)

Performance has regressed:
  test_build (0001_baselin) - Field 'min' has failed PercentageRegressionCheck:
  69.950940003 > 20.000000000

We made build() deliberately slower with an environment variable, then compared against the saved baseline. --benchmark-compare printed both runs side by side, and --benchmark-compare-fail=min:20% raised a PerformanceRegression because the minimum time grew by 69.95 percent, over the 20 percent limit. The expression can use other fields such as mean, and absolute limits in seconds such as mean:0.001. Use a generous threshold: 5 percent will fail constantly on shared machines because of noise.

Disabling benchmarks in normal runs

pytest test_reg.py --benchmark-disable
# 1 passed

Benchmarks make a suite slower, because each one runs for about a second. --benchmark-disable runs each benchmarked function once as an ordinary test, so correctness assertions still execute. --benchmark-skip skips benchmark tests entirely, and --benchmark-only runs just them. Put the choice in your pytest ini addopts for local runs, and enable benchmarks in a dedicated CI job.

JSON output, and a privacy note

pytest test_bench.py -k json --benchmark-json=out.json
# out.json: benchmarks[0] test_json_loads, min 452.7 us, 1442 rounds
# machine_info keys include: cpu, machine, node, processor, python_build, python_compiler

--benchmark-json writes full statistics for dashboards or custom checks. The file also records machine_info and commit information. The node field is the computer’s network host name, and file paths may appear too. That is useful for comparing machines, but review or strip it before committing results to a public repository or attaching them to a public bug report.

Benchmarks in CI

Shared CI runners vary in speed from run to run, so comparisons there are noisy. Practical approaches: run benchmarks on a dedicated or self-hosted runner; compare against a baseline produced on the same runner type in the same job; use relative thresholds around 15 to 25 percent; or use a service built for CI benchmarking. Our GitHub Actions guide shows how to add a separate job. Treat an occasional failure like a flaky test: rerun, and investigate if it repeats.

Tips for reliable benchmarks

  • Benchmark the smallest unit that matters and keep setup outside the timed function.
  • Compare minimums for regressions; mean and max reflect machine noise more than code.
  • Close other heavy programs and keep laptops plugged in; power saving changes CPU speed.
  • Use realistic inputs. Our JSON test parsed 1,000 realistic records, not a tiny string, so the number means something. Real payloads can come from your JSON fixtures.
  • Keep the assertion. Every benchmark should also check its result.

A benchmark checklist for code review

Benchmarks are code too, and a misleading one is worse than none. Before merging a new benchmark, check these points:

  • Does it measure the right thing? Data creation, file reads and network calls belong in module-level setup or a pedantic setup function, not inside the timed call.
  • Is the input realistic in size and shape? A function that is fast on ten items may be quadratic on ten thousand; parametrize over at least two sizes.
  • Does it assert the result? An optimisation that returns the wrong answer should fail the benchmark run.
  • Is it deterministic? Seed random data, as our sorting test should have done, so that two runs compare the same work.
  • Is the threshold sensible? Tie --benchmark-compare-fail limits to how noisy the machine is, and record why a value was chosen.
  • Is the baseline fresh? Re-save the baseline after an intentional performance change, such as a new feature that legitimately costs time, so later comparisons stay meaningful.

Applied consistently, these checks keep the benchmark suite trustworthy enough that a red result is treated as a real problem rather than noise to be retried.

Final thoughts

Pytest benchmark makes performance a tested property instead of a hope. Wrap a function call with the benchmark fixture, read the minimum time, save a baseline, and fail the build with --benchmark-compare-fail when a change makes things meaningfully slower. In our test, one flag caught a 70 percent slowdown that every correctness test would have passed. Keep thresholds realistic, run on stable hardware, and check the saved files before sharing them.

Frequently asked questions

How do I use pytest-benchmark?

Install the plugin, add the benchmark fixture to a test, and call benchmark with the function and its arguments. It times many rounds and returns the function result so you can assert on it.

How do I compare benchmark results between runs?

Save a run with the benchmark save option, then run again with the benchmark compare option. Add the compare fail option with a rule such as min:20% to fail on regressions.

Should I compare min or mean times?

For detecting regressions, the minimum is usually more stable, because noise from other processes only ever adds time. The mean is useful to see typical behaviour.

How do I skip benchmarks during normal test runs?

Use the benchmark disable option to run each benchmarked function once as a normal test, or the benchmark skip option to skip them entirely.

What is pedantic mode in pytest-benchmark?

A mode where you set rounds, iterations and a setup function yourself. Use it when the function changes its input or needs fresh, untimed setup before each round.

Keep learning

Work with Stack Quora

Need help with testing, WordPress or SEO?

We help teams ship better software and grow their sites. Tell us what you are working on and we will suggest a clear next step.

Keep reading

More guides on testing, AI & modern development

Enjoyed this article? Pick a topic and jump straight into our newest tutorials.

Artificial Intelligence

Confusion Matrix Explained: Precision, Recall, F1 and Thresholds in Python

Confusion matrix explained with Python: TP, FP, FN, TN, precision, recall, F1, the accuracy trap, threshold choice by cost and…

Read article →
Software Development

API Pagination: Offset vs Cursor vs Keyset, Tested on 1 Million Rows

API pagination explained: offset vs keyset and cursor pagination timed on 1 million rows, duplicates when data changes, the ties…

Read article →
DevOps

Git Submodule Explained: Add, Clone, Update and Remove Safely

Git submodule explained: add, clone, update –init, update –remote, the detached HEAD trap, foreach and clean removal, tested step by…

Read article →
Test Automation

Pytest tmp path Explained: tmp_path, tmp_path_factory and Temp Files

Pytest tmp_path explained: per-test temporary folders, tmp_path_factory for shared files, monkeypatch.chdir, legacy tmpdir, retention and basetemp.

Read article →
Software Testing

Test Case Prioritization: 6 Techniques Compared With APFD

Test case prioritization compared: risk, coverage, cost, history and random orders scored with APFD and time to find all faults…

Read article →
Databases & SQL

Full Text Search in SQL: FTS5, Ranking, Phrases and Speed

Full text search in SQL with SQLite FTS5: MATCH queries, phrases, prefixes, NEAR, stemming, BM25 ranking, snippets, two gotchas and…

Read article →
Databases & SQL

Generated Columns in SQL: Virtual vs Stored, Indexes and Limits

Generated columns in SQL explained: virtual vs stored, automatic updates, write errors, storage cost, indexing, JSON fields and support in…

Read article →
Software Development

Object Freeze in JavaScript: Shallow vs Deep, Seal and const

Object freeze in JavaScript explained: freeze vs seal vs const, strict mode errors, shallow vs deep freeze, frozen arrays, Map…

Read article →
Software Development

Python hashlib: SHA-256, File Hashes, HMAC and Password Hashing

Python hashlib explained: SHA-256 for strings and files, algorithm speeds, why passwords need PBKDF2 or scrypt, safe verification and HMAC…

Read article →
Software Development

Python UUID Explained: uuid1, uuid4, uuid5, uuid7 and Database Keys

Python UUID explained: uuid4, uuid5, uuid1 privacy, time-ordered uuid7, parsing, storage formats and UUIDs as database keys, tested on Python…

Read article →
Artificial Intelligence

Random Forest in Python: Bagging, Feature Sampling and OOB From Scratch

Random forest in Python from scratch: bootstrap sampling, feature sampling, number of trees, out-of-bag evaluation and impurity vs permutation importance.

Read article →
Software Development

MIME Types Explained: Content-Type, nosniff and a Common Types List

MIME types explained and tested in a browser: Content-Type, nosniff, scripts and stylesheets with wrong types, a common MIME types…

Read article →
DevOps

Git Clean Explained: Remove Untracked Files Without Regret

Git clean explained with dry runs: -n, -f, -d, -x, -X, exclusions, paths, nested repositories and interactive mode, plus a…

Read article →
Test Automation

Pytest approx: Comparing Floats With rel, abs and Collections

Pytest approx explained: default tolerances, the near-zero trap, rel and abs, lists and dicts, NaN, NumPy, and how it differs…

Read article →
Software Testing

Accessibility Testing: WCAG Checklist, Automated Tools and Manual Checks

Accessibility testing explained: WCAG basics, an axe-core scan of a broken and a fixed page, what automation missed, contrast checks…

Read article →
Databases & SQL

SQL UNPIVOT: Turn Columns Into Rows, 4 Ways Tested

SQL UNPIVOT explained: turn columns into rows with UNION ALL, CROSS JOIN with CASE, JSON functions or native UNPIVOT, with…

Read article →
Software Development

Intl NumberFormat in JavaScript: Currency, Compact, Percent and Units

Intl.NumberFormat in JavaScript: locales, currency, compact notation, percent, units, rounding, formatToParts, the non-breaking space trap and formatter reuse.

Read article →
Databases & SQL

SQL EXPLAIN Plan: How to Read Query Plans and Fix Slow Queries

SQL EXPLAIN plan explained: read scans, index searches, covering indexes and temporary sorts, with before and after timings for five…

Read article →
Artificial Intelligence

Decision Tree in Python: Gini, Entropy and Overfitting From Scratch

Decision tree in Python from scratch: Gini impurity, entropy, information gain, best splits, readable rules, and how depth and leaf…

Read article →
DevOps

Git Blame Explained: Find Who Changed a Line, and Why

Git blame explained with a real repository: ignore whitespace, line ranges and functions, ignore-revs files for formatting commits, moved code…

Read article →
Test Automation

Pytest Benchmark: Measure Speed and Catch Performance Regressions

Pytest benchmark tutorial: the benchmark fixture, reading results, parametrized and pedantic benchmarks, saving baselines and failing the build on regressions.

Read article →
Software Testing

Defect Density: Formula, Worked Examples and How to Use It

Defect density explained: the formula, a five-module worked example, per KLOC vs per function point, severity weighting, defect removal efficiency…

Read article →
Databases & SQL

Case Insensitive Search in SQL: LOWER, LIKE, COLLATE and Indexes

Case insensitive search in SQL: LOWER, LIKE, COLLATE NOCASE and expression indexes compared with query plans, plus accented letters and…

Read article →
Databases & SQL

ON DELETE CASCADE in SQL: CASCADE vs SET NULL vs RESTRICT

ON DELETE CASCADE in SQL tested: CASCADE vs SET NULL vs RESTRICT, multi-level cascades, the SQLite foreign key pragma, row…

Read article →
Artificial Intelligence

Logistic Regression in Python: Sigmoid, Log Loss and Gradient Descent From Scratch

Logistic regression in Python from scratch: sigmoid, log loss, gradient descent, odds ratios, thresholds, sigmoid overflow and separable data, tested…

Read article →
DevOps

Git Worktree Explained: Work on Two Branches at Once

Git worktree explained: add, list, remove, prune, lock and detached worktrees, a hotfix workflow, and the errors you will hit,…

Read article →
Test Automation

Pytest ini Explained: addopts, markers, testpaths and Precedence

Pytest ini explained: testpaths, addopts, strict markers, xfail_strict, filterwarnings, minversion, config file precedence and command-line overrides, tested.

Read article →
Software Testing

Test Estimation Techniques: WBS, Three Point PERT and Monte Carlo

Test estimation techniques worked through: WBS, three point PERT, total uncertainty, a Monte Carlo check, percentage rules and Wideband Delphi.

Read article →
Artificial Intelligence

Linear Regression in Python: Least Squares From Scratch, With R Squared

Linear regression in Python from scratch: closed form, NumPy least squares and gradient descent, R squared, RMSE, outliers and curved…

Read article →
DevOps

Git Remote Explained: Add, Set-URL, Rename, Remove and Push URLs

Git remote explained: add, set-url, separate push URLs, rename, remove and prune, plus how to avoid leaking tokens in remote…

Read article →
Test Automation

Pytest Mock: mocker.patch, spy, side_effect and autospec Explained

Pytest mock explained: mocker.patch, return_value, side_effect, spy, autospec and PropertyMock, with tested examples for APIs, clocks and retries.

Read article →
Software Testing

System Testing Explained: Types, Process and a Real End-to-End Example

System testing explained: types, entry and exit criteria, the process, and a real end-to-end test suite run against a small…

Read article →
Artificial Intelligence

Naive Bayes Classifier Explained: Build a Spam Filter in Python

Naive Bayes classifier explained by building a spam filter in Python: Bayes rule, the naive assumption, Laplace smoothing, log probabilities…

Read article →
DevOps

Git Branch Explained: Create, Switch, Rename, Delete and Recover

Git branch explained with tested commands: list, create, switch, rename, delete, merged checks, recovering deleted branches, remote tracking and naming.

Read article →
Test Automation

Pytest raises: Testing Exceptions, match and excinfo

Pytest raises explained with tested examples: match regexes, excinfo, custom and chained exceptions, parametrized error cases, pytest.warns and common mistakes.

Read article →
Software Testing

Non Functional Testing: Types, Metrics and a Real Load Test

Non functional testing explained: 10 key types, measurable requirements, latency percentiles and what a small real load test revealed about…

Read article →