1. Home
  2. Blog
  3. Test Automation

Test Automation

Pytest Xdist: Run Tests in Parallel and Cut Test Time

pytest-xdist runs your pytest suite on several CPU cores at once. Learn how to install it, choose the number of workers, pick a distribution mode, and fix the shared-state bugs that parallel runs expose, with timings and failures from real runs.

9 min read
Pytest xdist: run tests in parallel and cut test time

Key takeaways

  • pytest xdist adds -n: pytest -n auto starts one worker process per CPU core.
  • Our 16-test suite dropped from 8.05 s to 3.41 s with 4 workers, about 2.4 times faster.
  • Tests that share files, databases or globals can fail in parallel. Our shared-file test failed on every parallel run.
  • For tiny, fast suites, worker start-up costs more than it saves: 0.11 s became 1.43 s.

A test suite that takes 20 minutes changes how a team works. People stop running it locally, pull requests wait in CI, and bugs slip through. If your tests spend most of their time waiting on I/O, browsers or slow computations, running them in parallel is often the biggest speed-up available, and for pytest users the standard tool is pytest-xdist.

We ran every example on Python 3.13.6, pytest 9.2.0 and pytest-xdist 3.8.0, on a machine with 4 CPU cores. New to pytest? Read our pytest tutorial first.

What is pytest-xdist?

pytest-xdist is the official pytest plugin for distributed testing. It starts several worker processes, collects the tests in each, and hands tests out to the workers so they run at the same time. Each worker is a separate Python process with its own memory, so CPU-bound tests get real parallelism, not just threads waiting on the global interpreter lock.

Installing pytest-xdist

pip install pytest-xdist
pytest --version
python -c "import xdist; print(xdist.__version__)"   # 3.8.0

There is no configuration file to edit. Once installed, the plugin registers itself, and pytest gains new command-line options. Without -n, your tests run exactly as before.

Running tests in parallel with -n

pytest -n 4          # four workers
pytest -n auto       # one worker per CPU core
pytest -n logical    # one per logical CPU (hyper-threads)
pytest -n 0          # xdist installed but disabled

-n auto is the usual choice. It uses the number of physical CPU cores when it can detect them and falls back to logical CPUs otherwise. You can also put it in your configuration so everyone gets it by default:

# pytest.ini
[pytest]
addopts = -n auto

Our timings: serial vs parallel

Our test file had 16 parametrized tests, each sleeping for half a second to simulate a slow API call:

import time, pytest

@pytest.mark.parametrize("n", range(16))
def test_slow_api_call(n):
    time.sleep(0.5)
    assert n * 2 == n + n
Command Result Speed-up
pytest 16 passed in 8.05s 1.0x
pytest -n 2 16 passed in 4.94s 1.6x
pytest -n 4 16 passed in 3.41s 2.4x
pytest -n auto 16 passed in 3.45s 2.3x

The ideal with 4 workers would be 2 seconds (8 seconds of sleep split four ways). The rest is the fixed cost of starting workers and collecting tests in each one, roughly a second or more here. That fixed cost is why larger suites see results closer to the ideal, and why tiny suites can get slower, as we show below.

How pytest-xdist works

  1. The main process, called the controller, starts N workers named gw0, gw1 and so on.
  2. Every worker collects the full test suite independently. The controller checks that all workers found the same tests in the same order.
  3. The controller sends test IDs to workers according to the distribution mode.
  4. Workers run their tests and report results back, and the controller prints one combined report.

Because each worker collects independently, test collection must be deterministic. Generating parametrize values from an unseeded random source, or from a set with unpredictable order, can make collection differ between workers and abort the run.

Distribution modes with –dist

Mode How tests are shared Use when
load (default) Pending tests go to any free worker Independent tests
loadscope Tests in the same module or class stay together Expensive module or class fixtures
loadfile Tests in the same file stay together Files that share setup or state
loadgroup Tests marked xdist_group(name) stay together Hand-picked groups
worksteal Idle workers take tests from busy ones Tests with very uneven durations

Keeping related tests on one worker lets module and class fixtures be created once instead of once per worker, and keeps tests that touch the same resource from running at the same moment.

The real problem: shared state

Parallel runs assume your tests are independent. Many suites are not, and nobody notices until xdist runs them at the same time. We wrote three tests that all use one JSON file as a fake database:

DB = pathlib.Path("shared_db.json")

def test_create_user():
    _save({"users": ["asha"]})
    time.sleep(0.3)
    assert _load()["users"] == ["asha"]

def test_reset_db():
    _save({"users": []})
    time.sleep(0.3)
    assert _load()["users"] == []

Serially, all three passed. With -n 3 we ran them six times. Five runs ended with 2 failed, 1 passed and one run with all 3 failed, with errors like:

E   json.decoder.JSONDecodeError: Extra data: line 1 column 14 (char 13)

Two workers wrote the file at the same moment and left it half overwritten. In a real suite the same race shows up as intermittent failures that disappear when rerun, which is exactly what makes flaky tests so expensive.

Fixing shared state

Option 1: isolate each test. Give every test its own resource. pytest’s built-in tmp_path fixture creates a unique directory per test:

def test_create_user_isolated(tmp_path):
    db = tmp_path / "db.json"
    db.write_text(json.dumps({"users": ["asha"]}))
    assert json.loads(db.read_text())["users"] == ["asha"]

This version passed on three out of three runs with -n 2. The same idea applies to databases (a schema or database per worker), ports (bind to port 0), and test accounts (unique names per test).

Option 2: keep conflicting tests together. Running the original file with -n 3 --dist loadfile passed on three out of three runs, because all three tests went to one worker and ran one after another. This is a quick fix that sacrifices parallelism within the file. Isolation is better long-term.

Tip: before adding -n auto to CI, run the suite in parallel several times locally. A single green run proves little when races are timing-dependent.

Session fixtures run once per worker

A scope="session" fixture normally runs once per test run. Under xdist, every worker has its own session, so the fixture runs once per worker. If it creates a database, seeds data or starts a server, you will get N copies. Either design it to be per-worker, for example using the worker ID in names, or follow the xdist documentation pattern of using a file lock so only the first worker does the shared setup. We did not run the file-lock pattern here. Our pytest fixtures guide covers scopes in detail.

Knowing which worker you are on

def test_which_worker(worker_id):
    assert worker_id in ("master", "gw0", "gw1", "gw2", "gw3")

The worker_id fixture returns gw0, gw1 and so on under xdist, and master when tests run without workers. We confirmed it printed running on master with -n 0. With -p no:xdist the plugin is not loaded at all, so a test that requests worker_id errors out; we saw exactly that. Workers also get a PYTEST_XDIST_WORKER environment variable. Use them to build per-worker database names, ports or output folders.

When xdist makes tests slower

# 200 trivial assertions
pytest fast/          # 200 passed in 0.11s
pytest -n 4 fast/     # 200 passed in 1.43s

Starting four Python processes and collecting the suite four times took longer than simply running 200 tiny tests. xdist pays off when tests are slow: network calls, browsers, databases or heavy computation. For a fast unit suite, keep it serial, or enable xdist only in CI for the slow integration suite. Use --durations=10 to find where time actually goes before parallelising.

Output, -s and debugging

  • print output: workers capture stdout, so -s output did not appear in our parallel run. Debug with -n 0 or -p no:xdist.
  • Order: tests finish out of order, so do not rely on one test running before another. If a test needs another to run first, fix the test.
  • -x: stopping on the first failure still works, but tests already running on other workers may finish first.
  • Coverage: pytest-cov combines data from workers automatically; see pytest coverage.
  • Markers: keep slow or serial-only tests separate with pytest markers and run them in a second, non-parallel command.

Using pytest-xdist in CI

CI runners often have 2 to 4 cores, so -n auto gives modest gains per job, but they add up across hundreds of runs a day. Combine xdist inside a job with splitting the suite across several jobs for big suites. Watch memory: each worker imports your whole project, so four workers use roughly four times the memory. Our GitHub Actions CI/CD guide shows a typical pipeline.

Pytest xdist checklist

  • Measure first with --durations; parallelise slow suites, not fast ones.
  • Make tests independent: no shared files, rows, globals or ports.
  • Use tmp_path and per-worker names built from worker_id.
  • Remember session fixtures run once per worker.
  • Keep collection deterministic, especially parametrize values.
  • Use --dist loadfile or loadgroup as a stopgap for tests that cannot be isolated yet.
  • Run the parallel suite several times before trusting it.

Final thoughts

pytest-xdist is one install and one flag away from a faster suite: our slow tests ran 2.4 times faster on 4 cores. The flag is easy; the real work is making tests independent, which parallel runs expose immediately. Isolate state with tmp_path and per-worker resources, group what cannot be isolated yet, and keep tiny suites serial. Done that way, pytest-xdist turns long waits into quick feedback.

Frequently asked questions

How do I run pytest tests in parallel?

Install the plugin with pip install pytest-xdist, then run pytest -n auto to start one worker per CPU core, or pytest -n 4 for a fixed number of workers.

Why do my tests fail only with pytest xdist?

Usually the tests share state such as a file, database rows, a global variable or a port, and workers now touch it at the same time. Isolate each test, for example with tmp_path, or keep related tests together with the loadfile mode.

What does n auto mean in pytest?

It tells pytest-xdist to start as many workers as there are CPU cores, using physical cores when they can be detected and logical CPUs otherwise.

Does pytest xdist always make tests faster?

No. Starting workers has a fixed cost. In our run, 200 tiny tests took 0.11 seconds serially and 1.43 seconds with four workers. It helps most with slow tests.

Do session scoped fixtures work with pytest xdist?

They work, but each worker has its own session, so a session fixture runs once per worker rather than once overall. Design it to be safe to run several times, or use a file lock for shared setup.

Keep learning

Work with Stack Quora

Need help with testing, WordPress or SEO?

We help teams ship better software and grow their sites. Tell us what you are working on and we will suggest a clear next step.

Keep reading

More guides on testing, AI & modern development

Enjoyed this article? Pick a topic and jump straight into our newest tutorials.

Artificial Intelligence

Softmax Function Explained: Formula, Python Code and Stable Version

The softmax function explained: formula, worked example, NumPy code, the stable version that avoids NaN, temperature, cross-entropy and softmax vs…

Read article →
Software Development

CORS Explained: Preflight Requests, Headers and How to Fix CORS Errors

CORS explained: origins, simple and preflight requests, the response headers your server must send, credentials, and how to fix common…

Read article →
DevOps

Git Diff Explained: Staged Changes, Branches and Commits

Git diff explained: read the output, compare unstaged and staged changes, commits and branches, two dots vs three dots, and…

Read article →
Test Automation

Pytest Xdist: Run Tests in Parallel and Cut Test Time

Run pytest tests in parallel with pytest-xdist: install, the -n option, distribution modes, real timings and how to fix shared-state…

Read article →
Software Testing

Verification vs Validation in Software Testing: Example and Table

Verification vs validation in software testing explained with a comparison table and a tested coupon example where every spec test…

Read article →
Databases & SQL

SQL CAST Explained: Convert Data Types With Tested Examples

SQL CAST explained with tested examples: text to number, truncation, integer division, numbers stored as text, dates, bad input and…

Read article →
Databases & SQL

SQL DISTINCT: How to Remove Duplicate Rows (and Its Traps)

SQL DISTINCT explained with tested queries: one and many columns, NULLs, COUNT DISTINCT, DISTINCT vs GROUP BY, the SUM DISTINCT…

Read article →
Software Development

JavaScript Reduce Explained: 12 Practical Examples

JavaScript reduce explained step by step with 12 practical examples: sums, tallies, grouping, Maps, pipelines, sequential promises and the empty…

Read article →
Software Development

Python Random Module: randint, choice, seed and More

The Python random module explained: randint, randrange, choice, choices, sample, shuffle, seed, weights and when to use secrets instead, with…

Read article →
Software Development

Python Slicing Explained: Start, Stop and Step With Examples

Python slicing explained with start, stop and step, negative indexes, reversing, copying and slice assignment, with tested examples and a…

Read article →
Artificial Intelligence

Levenshtein Distance: Formula, Worked Table and Python Code

Levenshtein distance explained: the edit operations, the dynamic programming table for kitten and sitting, Python code with two-row memory, normalised…

Read article →
Software Development

URL Encoding Explained: Spaces, Plus Signs and UTF-8

URL encoding explained with tested Python and JavaScript code: percent encoding, reserved characters, %20 vs plus, UTF-8 bytes, encodeURI vs…

Read article →
DevOps

Gitignore: Patterns, Negation and Fixing Tracked Files

A tested gitignore guide: pattern rules, leading and trailing slashes, negation with !, the ignored parent folder trap, git check-ignore…

Read article →
Software Testing

Acceptance Criteria: Given When Then Examples and Checklist

How to write acceptance criteria testers can run: rule-based and Given When Then formats, a worked discount code example turned…

Read article →
Software Testing

State Transition Testing: Diagram, Table and Example

State transition testing explained with a login lockout example: states, events, the diagram, the state table, valid and invalid transitions,…

Read article →
Databases & SQL

Second Highest Salary in SQL: 5 Tested Ways and the Tie Trap

Find the second highest salary in SQL five ways: MAX subquery, LIMIT OFFSET, DENSE_RANK, a correlated subquery and per department,…

Read article →
Databases & SQL

SQL ROW_NUMBER: Top N per Group, Dedupe and Pagination

SQL ROW_NUMBER explained with tested queries: PARTITION BY, top N per group, deleting duplicates, pagination, the tie trap and how…

Read article →
Software Development

JavaScript Set: Union, Intersection and Removing Duplicates

JavaScript Set explained with tested examples: add, has, delete, deduplicating arrays, the new union and intersection methods, equality rules and…

Read article →
Test Automation

Python Unittest: Assert Methods, setUp and Mocks With Examples

A practical Python unittest tutorial: TestCase, assert methods, setUp and tearDown, subTest, skips, expected failures and mock, with a test…

Read article →
Artificial Intelligence

Jaccard Similarity: Formula, Python, MinHash

Jaccard similarity measures overlap between two sets as intersection over union. Work the formula, compute it in Python for words…

Read article →
DevOps

Git Hooks: Pre-commit and Commit-msg Examples

Git hooks run scripts at key moments such as commit and push. See tested pre-commit, commit-msg and prepare-commit-msg hooks, sharing…

Read article →
Test Automation

API Testing in Python: Pytest + Requests Suite

Build an API testing suite in Python with pytest and requests: fixtures, status codes, headers, JSON schema validation, CRUD flows,…

Read article →
Software Testing

Test Design Techniques: 9 Methods + Examples

Test design techniques help you write fewer tests that catch more bugs. Learn 9 black-box, white-box and experience-based methods with…

Read article →
Databases & SQL

SQL Temporary Table: Create, Use and Drop It

SQL temporary tables store interim results for one session. Learn syntax for SQL Server, PostgreSQL, MySQL, Oracle and SQLite, session…

Read article →
Databases & SQL

SQL CROSS JOIN: Every Combination + Examples

SQL CROSS JOIN returns every combination of rows from two tables. See the row math, product variants, calendar grids that…

Read article →
Artificial Intelligence

TF-IDF Explained: Formula and Worked Example

TF-IDF scores a word by how often it appears in a document and how rare it is across all documents….

Read article →
DevOps

Git Tag: Create, Push and Delete Release Tags

Git tag explained with tested commands: annotated vs lightweight tags, listing and sorting, pushing with follow-tags, deleting local and remote…

Read article →
Test Automation

Pytest Coverage: pytest-cov and Branch Gaps

Measure pytest coverage with pytest-cov: term-missing reports, branch coverage, HTML reports, pyproject settings and a fail-under gate for CI, with…

Read article →
Software Testing

Error Guessing in Testing: Checklist + Demo

Error guessing uses tester experience to predict where code breaks. Get a ready checklist of risky inputs, a step by…

Read article →
Artificial Intelligence

BM25 Explained: How Keyword Search Ranks Results, With Python Code and Hybrid Search

BM25 explained with a pure Python implementation we ran: the formula, IDF, term saturation, the k1 and b parameters, why…

Read article →
DevOps

Git Bisect Explained: Find the Commit That Broke Your Code, Manually or Automatically

Git bisect explained with a tested repository: find the one bad commit among 41 in 5 test runs, automate it…

Read article →
Test Automation

pytest Markers Explained: Custom Marks, skip, skipif, xfail and -m Filters

pytest markers explained with a tested suite: register custom marks, run only smoke tests with -m, combine filters with and,…

Read article →
Software Testing

Positive vs Negative Testing: Differences, 24 Real Test Cases and a Checklist

Positive vs negative testing explained with a real signup form: 6 positive and 18 negative test cases run in pytest,…

Read article →
Artificial Intelligence

Cosine Similarity Explained: Formula, Worked Example and Python Code

Cosine similarity explained simply: the formula, a worked example by hand, pure Python and NumPy code, cosine vs Euclidean distance…

Read article →
DevOps

Git Squash Commits: 4 Easy Ways, Tested Step by Step (rebase -i, reset, merge)

Git squash commits explained with 4 tested methods: interactive rebase, reset –soft, merge –squash and fixup with autosquash. Before-and-after logs,…

Read article →
Test Automation

pytest parametrize Explained: 7 Patterns with Tested Examples

pytest parametrize explained with 7 tested patterns: basic cases, readable ids, stacked decorators, pytest.param with xfail and skip, indirect fixtures…

Read article →