Property-Based Testing in Python: Hypothesis Finds the Bugs Your Examples Miss

You write a test with three carefully chosen examples. They pass, and the code ships. A week later, an input you never thought of breaks it. The problem is not laziness; nobody can imagine every edge case. Property-based testing hands that job to a computer.

We wrote three small functions, each with passing example tests, then added property tests using Hypothesis 6.170 with pytest 9.2 on Python 3.13. The failures shown are copied from that run.

What is property-based testing?

In an example-based test, you choose the input and the expected output: split_bill(1000, 4) == [250, 250, 250, 250]. In a property-based test, you describe what kind of inputs are valid and a property that must hold for all of them. The library then generates many inputs, by default 100 per test in Hypothesis, and checks the property each time.

The idea comes from QuickCheck, a Haskell library from 1999. Hypothesis brought it to Python, and similar tools exist for most languages, such as fast-check for JavaScript, jqwik for Java and proptest for Rust.

Our functions and the example tests that passed

def split_bill(total_paise, people):
    share = total_paise // people
    return [share] * people

def encode(s):   # run-length encoding: "aaab" -> "3a1b"
    ...

def decode(s):   # "3a1b" -> "aaab"
    ...

def test_example_based():
    assert split_bill(1000, 4) == [250, 250, 250, 250]
    assert encode("aaab") == "3a1b"

The example test passed. Both functions look correct for the obvious inputs, and that is exactly the trap.

A first property

What should always be true about splitting a bill? The shares must add up to the total, whatever the amount and number of people:

from hypothesis import given, strategies as st

@given(st.integers(min_value=0, max_value=10**9),
       st.integers(min_value=1, max_value=50))
def test_split_bill_adds_up(total, people):
    assert sum(split_bill(total, people)) == total

Hypothesis reported:

E       assert 0 == 1
E        +  where 0 = sum([0, 0])
E        +    where [0, 0] = split_bill(1, 2)
E       Failing test case: test_split_bill_adds_up(
E           total=1,
E           people=2,
E       )

Splitting 1 paisa between 2 people gave each person 0, and the paisa disappeared. In general, whenever the total does not divide evenly, the remainder is lost. The fix is to hand out the remainder, for example one extra paisa to the first few people. Money rounding is a classic source of real bugs, and our example of 1000 between 4 was simply too neat to reveal it.

Shrinking: the simplest failing case

While searching, Hypothesis probably first failed on something like total=839201, people=17. Rather than report that, it shrinks the input: it tries smaller and simpler values that still fail, until it cannot simplify further. total=1, people=2 is the smallest possible case of the bug, and it makes the cause obvious at a glance. Shrinking is what turns random testing from noisy to genuinely useful.

Hypothesis also saves failing examples in a local database and tries them first on the next run, so a bug, once found, is reproduced immediately until it is fixed.

Round-trip properties

For any pair of functions that convert data and convert it back, such as encode and decode, serialise and parse, or compress and decompress, there is a powerful property: the round trip should return the original.

@given(st.text())
def test_roundtrip(s):
    assert decode(encode(s)) == s

Hypothesis found two distinct failures in one run:

Failing test case: test_roundtrip(s='0')
AssertionError: assert '' == '0'

The input “0” encodes to “10”, meaning one zero, but the decoder sees two digits and reads “10” as a count with no character to repeat. Our format cannot tell digits in the data apart from digits in the counts. No example test we wrote included a digit, because run-length encoding examples are always letters like “aaab”.

The Unicode surprise

The second failure was stranger:

Failing test case: test_roundtrip(s='\xb9:')
ValueError: invalid literal for int() with base 10: '1\xb91'

The character \xb9 is “¹”, a superscript one. Python’s str.isdigit() returns True for it, so our decoder treated it as part of a number, but int() cannot convert it. We would never have written this test case by hand. Hypothesis’s text strategy deliberately includes unusual Unicode, because that is where real software breaks. Using isdecimal(), or better, a format that cannot confuse data with counts, would fix both bugs.

Strategies: describing valid inputs

A strategy tells Hypothesis how to generate data. Common ones:

Strategies can be combined and filtered, and st.from_type() can even build values from type hints or dataclasses. Prefer constraining a strategy, such as min_value=1, over filtering out invalid values afterwards, which wastes generated examples.

Property patterns that work

The hardest part is thinking of properties. These patterns cover most real cases:

• Round trip: decode(encode(x)) == x. Ideal for parsers, serialisers and converters.
• Invariants: something that never changes, such as the total in our bill split, or the length of a list after sorting.
• Idempotence: doing it twice equals doing it once, as in normalise(normalise(x)) == normalise(x).
• Oracle: compare against a simpler, slower implementation that is obviously correct.
• Metamorphic: related inputs give related outputs, such as adding a filter never increasing the number of search results.
• No crash: the function never raises an unexpected exception for valid input. Simple, and surprisingly effective.

Properties that passed

Not every property fails, and a passing property is still valuable evidence. These passed in our run:

@given(st.text())
def test_slug_has_no_spaces(t):
    assert " " not in slugify(t)

@given(st.text())
def test_slug_idempotent(t):
    assert slugify(slugify(t)) == slugify(t)

@given(st.lists(st.integers()))
def test_sorted_properties(xs):
    ys = sorted(xs)
    assert len(ys) == len(xs)
    assert all(a <= b for a, b in zip(ys, ys[1:]))
    assert sorted(ys) == ys

The final result was 2 failed, 4 passed in 5.27s. Notice the cost: property tests run many examples, so they take longer than ordinary unit tests. Five seconds for six tests is fine locally, but it shapes how you run them in CI.

Using it in practice

• Start with one function that parses, converts or calculates, where edge cases matter most.
• Use settings profiles: fewer examples on every commit, many more in a nightly job, using @settings(max_examples=...) or registered profiles.
• Combine with fixtures: Hypothesis works inside pytest, although function-scoped fixtures are not reset between generated examples, so keep shared state out. See our pytest fixtures guide.
• Keep the examples: property tests complement, rather than replace, readable example tests that document expected behaviour.
• Try stateful testing for APIs and data structures: Hypothesis can generate whole sequences of operations and check the system stays consistent.

When to use it

Property-based testing shines for pure functions with clear rules: parsing, validation, maths, money, encoding, sorting and data transformations. It is less useful for UI flows or code whose correct output is hard to describe. It fits naturally into the base of the test pyramid, and it complements techniques like boundary value analysis by exploring boundaries you did not think to list.

Fixing the bill split, and strengthening the property

We fixed split_bill by handing the remainder out one paisa at a time, then made the property stricter at the same time:

def split_bill(total_paise, people):
    share, rem = divmod(total_paise, people)
    return [share + 1 if i < rem else share for i in range(people)]

@given(st.integers(min_value=0, max_value=10**9),
       st.integers(min_value=1, max_value=50))
def test_split_bill_adds_up(total, people):
    parts = split_bill(total, people)
    assert sum(parts) == total
    assert max(parts) - min(parts) <= 1     # nobody pays more than 1 paisa extra

def test_examples():
    assert split_bill(1, 2) == [1, 0]
    assert split_bill(1000, 3) == [334, 333, 333]

All tests passed. The second assertion matters: without it, a “fix” that gave the whole remainder to one person would also have passed. Good properties describe what “correct” means as completely as you can, and the shrunk failing case, split_bill(1, 2), now lives on as a plain example test.

Final thoughts

Example tests check the cases you imagined; property tests search for the ones you did not. In a few lines, Hypothesis found a lost-paisa rounding bug, a digit-handling bug and a Unicode bug in code that passed its examples. Pick one function with tricky inputs, write a round-trip or invariant property, and see what it finds.

Frequently asked questions

Keep learning