Flaky Tests: What Causes Them and How to Fix Them for Good

The build goes red, someone clicks “re-run”, it passes, and everybody moves on. It seems harmless, but flaky tests slowly destroy the most valuable thing a test suite has: people’s trust in it. When failures are usually noise, real bugs get waved through too.

This guide covers why tests become flaky and how to fix each cause properly. We ran the timing and shared-state demos in Python 3.13, and the numbers shown come from those runs. They are small demonstrations designed to make flakiness easy to see, so your own failure rates will differ.

What is a flaky test?

A flaky test gives different results on the same code. It passes on one run and fails on the next, even though nothing in the product changed. The test may be at fault, or the code may contain a real intermittent bug. That second possibility is exactly why you should not simply dismiss them.

Why flaky tests are so costly

• Lost trust. Once people assume red means “probably flaky”, they stop investigating real failures.
• Wasted time. Developers spend hours rerunning pipelines and examining failures that have nothing to do with their change.
• Slower delivery. Merges and releases wait on reruns, undermining the point of CI/CD.
• Hidden bugs. Race conditions in the product can look just like test noise, and those are the ones that hurt in production.

Common causes at a glance

Cause 1: Timing and fixed sleeps

This is the most common cause. A test assumes that background work will be finished after a fixed delay. To see the effect, we wrote a small test in which the work takes up to 2 ms, while the test waits for just 1 ms before checking:

def flaky_timing():
    result = {}
    def work():
        time.sleep(random.uniform(0, 0.002))
        result["done"] = True
    threading.Thread(target=work).start()
    time.sleep(0.001)        # hope it has finished
    assert result.get("done")

The fix is to wait for the actual condition, with a generous upper limit:

def fixed_timing():
    result = threading.Event()
    def work():
        time.sleep(random.uniform(0, 0.002))
        result.set()
    threading.Thread(target=work).start()
    assert result.wait(timeout=1)   # continues as soon as it is ready

We ran each version 200 times:

The result is more than just reliability. The fixed version moves on the moment the work is done, so it is usually faster too. A longer sleep would make the first version fail less often, but every run would then be slower, and it could still fail on a busy CI machine.

In browser tests, the same principle applies. Use the auto-waiting of Playwright, or explicit Selenium waits, instead of time.sleep:

# Playwright: retries until true or a timeout is reached
expect(page.get_by_text("Order placed")).to_be_visible()

# Selenium: an explicit wait for a condition
WebDriverWait(driver, 10).until(
    EC.visibility_of_element_located((By.ID, "confirmation"))
)

Cause 2: Shared state and test order

When tests share data, one can affect another, and the result depends on the order they run in. These two tests use one shared list:

store = []

def test_add():
    store.append("x")
    assert len(store) == 1

def test_empty():
    assert store == []

Our run confirmed this, and it is why a suite can pass for months and then break when someone adds parallel runs or a random ordering plugin. The cure is isolation:

• Give each test fresh data, using a fixture that sets it up and cleans it up. Our pytest tutorial shows how.
• Create unique records, such as an email with a random suffix, in place of a shared “test user”.
• Wrap database tests in a transaction and roll it back, where your setup allows it.
• Run tests in random order regularly, so that these problems show up early.

Cause 3: Dates, time zones and randomness

A test that passes all week can fail on a Saturday, at midnight or on the last day of the month. Make time an input, rather than reading the clock inside your logic:

def is_weekend(today=None):
    today = today or datetime.date.today()
    return today.weekday() >= 5

is_weekend(datetime.date(2026, 11, 7))   # True  (a Saturday)
is_weekend(datetime.date(2026, 11, 9))   # False (a Monday)

Those tests give the same answer on any day. Likewise, set a fixed seed for random values in tests, and log it when you use random data so that you can reproduce a failure. Do not assume a particular order from sets, dictionaries from external sources or database queries with no ORDER BY.

Cause 4: Brittle selectors

A test that finds a button with div:nth-child(3) > span breaks with any layout tweak, and it may fail only for some screen sizes or variants. Prefer locators based on what users see, or dedicated test attributes. Our guide to Playwright locators explains the order to try them in. Animations are another frequent source of trouble, so consider disabling them in test environments.

Cause 5: External services and networks

If a test calls a real payment provider or a third-party API, then its result depends on someone else’s uptime and rate limits. Mock those calls for most tests, and keep a small number of separate contract checks against the real service, so that you still learn if it changes. Our guide to unit vs integration testing explains how to split these up.

How to diagnose a flaky test

1. Confirm it. Run the test many times without changes. With pytest, a simple shell loop works, and a plugin such as pytest-repeat gives you --count=100.
2. Record evidence. Save logs, screenshots and traces for each failure. Playwright’s trace viewer is particularly useful here.
3. Stress the conditions. Run it with limited CPU, in parallel or in random order. Making it fail more often makes it easier to study.
4. Test it alone. If it always passes alone but fails in the suite, suspect shared state.
5. Look for patterns. Check which machines, times of day and neighbouring tests are involved.
6. Check the product. Ask honestly whether this could be a real race condition in your application.

Are automatic retries a fix?

Retries are a useful short-term safety net, and they are not a cure. A single automatic retry in CI can keep work moving while you investigate, but a test that needs a retry to pass is still telling you something. Worse, a real intermittent product bug that fails 20% of the time will usually get through a retry unnoticed.

Use retries with these rules: limit them to one or two, record every test that needed one, and treat that list as your to-do list. A test reported as “passed on retry” should count as a problem to investigate, not a success.

A team process for flaky tests

1. Measure. Track which tests fail and then pass on the same commit. Many CI tools and test reporters can show this.
2. Quarantine. Move a confirmed flaky test out of the blocking run, so that it stops disrupting everybody, but keep running it and tracking it.
3. Assign an owner and a deadline. A quarantine with no deadline just becomes a graveyard.
4. Fix or delete. If a test is not worth fixing, remove it, and check that its coverage still exists elsewhere.
5. Prevent. Run new tests several times before merging. A test that fails once in fifty runs is better caught now than in six months.

Treat a flaky test as a bug with priority, and not as background noise. Severity and priority apply here as well, as covered in severity vs priority. If flakiness is widespread, read about broader test automation mistakes, and consider whether you have too many tests at the browser level.

Final thoughts

Flaky tests feel random, but they rarely are. Nearly every one comes down to an assumption about time, order, data or environment that is not always true. Pick the test that fails most often in your pipeline this week. Run it fifty times, find the assumption, and fix that for good. One real fix teaches a team more than a hundred reruns ever will.

Frequently asked questions

Keep learning