Pytest Xdist: Run Tests in Parallel and Cut Test Time
pytest-xdist runs your pytest suite on several CPU cores at once. Learn how to install it, choose the number of workers, pick a distribution mode, and fix the shared-state bugs that parallel runs expose, with timings and failures from real runs.
- pytest xdist adds
-n:pytest -n autostarts one worker process per CPU core. - Our 16-test suite dropped from 8.05 s to 3.41 s with 4 workers, about 2.4 times faster.
- Tests that share files, databases or globals can fail in parallel. Our shared-file test failed on every parallel run.
- For tiny, fast suites, worker start-up costs more than it saves: 0.11 s became 1.43 s.
A test suite that takes 20 minutes changes how a team works. People stop running it locally, pull requests wait in CI, and bugs slip through. If your tests spend most of their time waiting on I/O, browsers or slow computations, running them in parallel is often the biggest speed-up available, and for pytest users the standard tool is pytest-xdist.
We ran every example on Python 3.13.6, pytest 9.2.0 and pytest-xdist 3.8.0, on a machine with 4 CPU cores. New to pytest? Read our pytest tutorial first.
What is pytest-xdist?
pytest-xdist is the official pytest plugin for distributed testing. It starts several worker processes, collects the tests in each, and hands tests out to the workers so they run at the same time. Each worker is a separate Python process with its own memory, so CPU-bound tests get real parallelism, not just threads waiting on the global interpreter lock.
Installing pytest-xdist
pip install pytest-xdist
pytest --version
python -c "import xdist; print(xdist.__version__)" # 3.8.0
There is no configuration file to edit. Once installed, the plugin registers itself, and pytest gains new command-line options. Without -n, your tests run exactly as before.
Running tests in parallel with -n
pytest -n 4 # four workers
pytest -n auto # one worker per CPU core
pytest -n logical # one per logical CPU (hyper-threads)
pytest -n 0 # xdist installed but disabled
-n auto is the usual choice. It uses the number of physical CPU cores when it can detect them and falls back to logical CPUs otherwise. You can also put it in your configuration so everyone gets it by default:
# pytest.ini
[pytest]
addopts = -n auto
Our timings: serial vs parallel
Our test file had 16 parametrized tests, each sleeping for half a second to simulate a slow API call:
import time, pytest
@pytest.mark.parametrize("n", range(16))
def test_slow_api_call(n):
time.sleep(0.5)
assert n * 2 == n + n
| Command | Result | Speed-up |
|---|---|---|
pytest |
16 passed in 8.05s | 1.0x |
pytest -n 2 |
16 passed in 4.94s | 1.6x |
pytest -n 4 |
16 passed in 3.41s | 2.4x |
pytest -n auto |
16 passed in 3.45s | 2.3x |
The ideal with 4 workers would be 2 seconds (8 seconds of sleep split four ways). The rest is the fixed cost of starting workers and collecting tests in each one, roughly a second or more here. That fixed cost is why larger suites see results closer to the ideal, and why tiny suites can get slower, as we show below.
How pytest-xdist works
- The main process, called the controller, starts N workers named
gw0,gw1and so on. - Every worker collects the full test suite independently. The controller checks that all workers found the same tests in the same order.
- The controller sends test IDs to workers according to the distribution mode.
- Workers run their tests and report results back, and the controller prints one combined report.
Because each worker collects independently, test collection must be deterministic. Generating parametrize values from an unseeded random source, or from a set with unpredictable order, can make collection differ between workers and abort the run.
Distribution modes with –dist
| Mode | How tests are shared | Use when |
|---|---|---|
load (default) |
Pending tests go to any free worker | Independent tests |
loadscope |
Tests in the same module or class stay together | Expensive module or class fixtures |
loadfile |
Tests in the same file stay together | Files that share setup or state |
loadgroup |
Tests marked xdist_group(name) stay together |
Hand-picked groups |
worksteal |
Idle workers take tests from busy ones | Tests with very uneven durations |
Keeping related tests on one worker lets module and class fixtures be created once instead of once per worker, and keeps tests that touch the same resource from running at the same moment.
The real problem: shared state
Parallel runs assume your tests are independent. Many suites are not, and nobody notices until xdist runs them at the same time. We wrote three tests that all use one JSON file as a fake database:
DB = pathlib.Path("shared_db.json")
def test_create_user():
_save({"users": ["asha"]})
time.sleep(0.3)
assert _load()["users"] == ["asha"]
def test_reset_db():
_save({"users": []})
time.sleep(0.3)
assert _load()["users"] == []
Serially, all three passed. With -n 3 we ran them six times. Five runs ended with 2 failed, 1 passed and one run with all 3 failed, with errors like:
E json.decoder.JSONDecodeError: Extra data: line 1 column 14 (char 13)
Two workers wrote the file at the same moment and left it half overwritten. In a real suite the same race shows up as intermittent failures that disappear when rerun, which is exactly what makes flaky tests so expensive.
Fixing shared state
Option 1: isolate each test. Give every test its own resource. pytest’s built-in tmp_path fixture creates a unique directory per test:
def test_create_user_isolated(tmp_path):
db = tmp_path / "db.json"
db.write_text(json.dumps({"users": ["asha"]}))
assert json.loads(db.read_text())["users"] == ["asha"]
This version passed on three out of three runs with -n 2. The same idea applies to databases (a schema or database per worker), ports (bind to port 0), and test accounts (unique names per test).
Option 2: keep conflicting tests together. Running the original file with -n 3 --dist loadfile passed on three out of three runs, because all three tests went to one worker and ran one after another. This is a quick fix that sacrifices parallelism within the file. Isolation is better long-term.
Tip: before adding -n auto to CI, run the suite in parallel several times locally. A single green run proves little when races are timing-dependent.
Session fixtures run once per worker
A scope="session" fixture normally runs once per test run. Under xdist, every worker has its own session, so the fixture runs once per worker. If it creates a database, seeds data or starts a server, you will get N copies. Either design it to be per-worker, for example using the worker ID in names, or follow the xdist documentation pattern of using a file lock so only the first worker does the shared setup. We did not run the file-lock pattern here. Our pytest fixtures guide covers scopes in detail.
Knowing which worker you are on
def test_which_worker(worker_id):
assert worker_id in ("master", "gw0", "gw1", "gw2", "gw3")
The worker_id fixture returns gw0, gw1 and so on under xdist, and master when tests run without workers. We confirmed it printed running on master with -n 0. With -p no:xdist the plugin is not loaded at all, so a test that requests worker_id errors out; we saw exactly that. Workers also get a PYTEST_XDIST_WORKER environment variable. Use them to build per-worker database names, ports or output folders.
When xdist makes tests slower
# 200 trivial assertions
pytest fast/ # 200 passed in 0.11s
pytest -n 4 fast/ # 200 passed in 1.43s
Starting four Python processes and collecting the suite four times took longer than simply running 200 tiny tests. xdist pays off when tests are slow: network calls, browsers, databases or heavy computation. For a fast unit suite, keep it serial, or enable xdist only in CI for the slow integration suite. Use --durations=10 to find where time actually goes before parallelising.
Output, -s and debugging
- print output: workers capture stdout, so
-soutput did not appear in our parallel run. Debug with-n 0or-p no:xdist. - Order: tests finish out of order, so do not rely on one test running before another. If a test needs another to run first, fix the test.
- -x: stopping on the first failure still works, but tests already running on other workers may finish first.
- Coverage: pytest-cov combines data from workers automatically; see pytest coverage.
- Markers: keep slow or serial-only tests separate with pytest markers and run them in a second, non-parallel command.
Using pytest-xdist in CI
CI runners often have 2 to 4 cores, so -n auto gives modest gains per job, but they add up across hundreds of runs a day. Combine xdist inside a job with splitting the suite across several jobs for big suites. Watch memory: each worker imports your whole project, so four workers use roughly four times the memory. Our GitHub Actions CI/CD guide shows a typical pipeline.
Pytest xdist checklist
- Measure first with
--durations; parallelise slow suites, not fast ones. - Make tests independent: no shared files, rows, globals or ports.
- Use
tmp_pathand per-worker names built fromworker_id. - Remember session fixtures run once per worker.
- Keep collection deterministic, especially parametrize values.
- Use
--dist loadfileorloadgroupas a stopgap for tests that cannot be isolated yet. - Run the parallel suite several times before trusting it.
Final thoughts
pytest-xdist is one install and one flag away from a faster suite: our slow tests ran 2.4 times faster on 4 cores. The flag is easy; the real work is making tests independent, which parallel runs expose immediately. Isolate state with tmp_path and per-worker resources, group what cannot be isolated yet, and keep tiny suites serial. Done that way, pytest-xdist turns long waits into quick feedback.
Frequently asked questions
How do I run pytest tests in parallel?
Install the plugin with pip install pytest-xdist, then run pytest -n auto to start one worker per CPU core, or pytest -n 4 for a fixed number of workers.
Why do my tests fail only with pytest xdist?
Usually the tests share state such as a file, database rows, a global variable or a port, and workers now touch it at the same time. Isolate each test, for example with tmp_path, or keep related tests together with the loadfile mode.
What does n auto mean in pytest?
It tells pytest-xdist to start as many workers as there are CPU cores, using physical cores when they can be detected and logical CPUs otherwise.
Does pytest xdist always make tests faster?
No. Starting workers has a fixed cost. In our run, 200 tiny tests took 0.11 seconds serially and 1.43 seconds with four workers. It helps most with slow tests.
Do session scoped fixtures work with pytest xdist?
They work, but each worker has its own session, so a session fixture runs once per worker rather than once overall. Design it to be safe to run several times, or use a file lock for shared setup.
Keep learning
Need help with testing, WordPress or SEO?
We help teams ship better software and grow their sites. Tell us what you are working on and we will suggest a clear next step.