1. Home
  2. Blog
  3. Software Testing

Software Testing

Mutation Testing Explained: Why 100% Coverage Still Missed Our Bug

Mutation testing checks your tests by planting small bugs in the code and seeing whether the tests catch them. We built a tiny mutation tool, ran it on a suite with full line coverage, and it still found three gaps.

9 min read
Mutation testing explained: why 100 percent coverage still missed our bug

Key takeaways

  • Mutation testing makes many small changes to the code, called mutants, and runs your tests against each one.
  • A test suite kills a mutant when at least one test fails. The share killed is the mutation score.
  • Our suite had 100% line coverage but scored only 79%, missing three boundary bugs. Two boundary tests raised it to 100%.
  • It is slow, so run it on critical modules or changed files rather than the whole codebase on every commit.

Code coverage tells you which lines your tests executed. It does not tell you whether the tests would notice if those lines were wrong. A test can run a line and assert almost nothing about it. Mutation testing answers the harder question: if a developer introduced a small bug here, would any test fail?

To show how it works without hiding behind a tool, we wrote a small mutation tester in about 60 lines of Python using the ast module, then ran it with Python 3.13.6. Every number below comes from that run. Real tools such as PIT, Stryker and mutmut do the same job at scale; we describe them later but did not run them here.

What is mutation testing?

The process has four steps:

  1. Run your tests against the original code. They must pass.
  2. Create a mutant: a copy of the code with one small change, such as >= turned into >.
  3. Run the tests against the mutant. If any test fails, the mutant is killed. If all pass, it survived.
  4. Repeat for many mutants and report which survived.

Each surviving mutant is a concrete bug that your tests would let through. That makes the output far more actionable than a coverage percentage.

Key terms

Term Meaning
Mutant A version of the code with one deliberate change
Mutation operator A rule for making changes, such as “swap comparison operators”
Killed At least one test failed against the mutant
Survived Every test passed, so the change went unnoticed
Mutation score Killed mutants divided by total mutants
Equivalent mutant A change that does not alter behaviour, so no test can kill it

The code under test

def shipping_fee(total, is_member):
    if total >= 500:
        return 0
    if is_member:
        return 20
    return 50

Orders of 500 or more ship free, members pay 20 and everyone else pays 50. The boundary at exactly 500 is where bugs like to hide, as our boundary value analysis guide explains.

A suite with full coverage and weak checks

def weak_tests(f):
    assert f(600, False) == 0
    assert f(100, True) == 20
    assert f(100, False) == 50

These three tests look reasonable, one per outcome. We traced execution and confirmed the suite ran all five body lines, which is 100% line coverage. Any coverage report would show this function as fully tested. Our code coverage guide explains why that number alone can mislead, and this example shows it.

The 14 mutants we generated

Our tool walked the syntax tree and applied three operators:

  • Relational operator replacement: >= became >, <, <= and == (4 mutants).
  • Constant change: each number, 500, 0, 20 and 50, moved up and down by one (8 mutants).
  • Condition negation: each if condition was wrapped in not (2 mutants).

The core of the tool is short. It deep-copies the tree, changes one node, compiles the result and runs the test suite against it:

def build(tree, idx, kind, val):
    t = copy.deepcopy(tree)
    for i, node in enumerate(ast.walk(t)):
        if i == idx:
            if kind == "op":    node.ops = [val()]
            if kind == "const": node.value = val
            if kind == "neg":   node.test = ast.UnaryOp(op=ast.Not(), operand=node.test)
    ast.fix_missing_locations(t)
    ns = {}
    exec(compile(t, "<mutant>", "exec"), ns)
    return ns["shipping_fee"]

Tip: running mutated code with exec is fine for a demo on your own code. Real tools also isolate each mutant run and add timeouts, because some mutants create infinite loops.

Results: 79% with full coverage

original passes both suites
weak suite executes body lines: [3, 4, 5, 6, 7]
mutants generated: 14
weak: killed 11/14 = 79%; survivors: ['line 3: >= -> Gt', 'line 3: 500 -> 501', 'line 3: 500 -> 499']

Eleven mutants died, which shows the tests did check outcomes. The three survivors all sit on the free-shipping boundary:

  • >= to >: an order of exactly 500 would wrongly pay a fee. No test used 500.
  • 500 to 501: the same bug, written differently.
  • 500 to 499: an order of 499 would wrongly ship free. No test used 499.

Each survivor is a believable real-world bug, the kind a hurried refactor introduces. Coverage said “done”; mutation testing listed three bugs that would ship.

Killing the survivors

def strong_tests(f):
    weak_tests(f)
    assert f(500, False) == 0     # exactly on the boundary
    assert f(499, False) == 50    # just below it
    assert f(499, True) == 20     # just below it, member
strong: killed 14/14 = 100%; survivors: []

The boundary tests killed all three survivors. Notice what changed: not more lines covered, but sharper assertions at the exact values where behaviour flips. That is the real lesson. Mutation testing does not just grade your suite; it points to the specific test you forgot. In pytest, these would become separate parametrised cases, as shown in our pytest tutorial.

Common mutation operators

Operator Example change Bug it imitates
Relational >= to > Off-by-one at a boundary
Arithmetic + to - Wrong formula
Logical and to or Wrong condition logic
Constant 500 to 501 Wrong threshold or limit
Return value return x to return None Result ignored by tests
Statement removal Delete a method call Missing side effect, such as a save

The return-value and statement-removal operators are especially good at finding tests that call code but never check what it did.

Equivalent mutants

Some mutants cannot be killed because they behave exactly like the original. For example, if a loop index starts at 0 and the code checks i != len(items), changing it to i < len(items) may produce identical results. These equivalent mutants inflate the survivor list and must be reviewed by a person. That is one reason teams rarely chase a 100% mutation score on large codebases; a high score on critical code is a better target.

Mutation testing tools

  • PIT (Pitest) for Java and the JVM, widely used with Maven and Gradle.
  • Stryker for JavaScript and TypeScript, C# (Stryker.NET) and Scala.
  • mutmut and Cosmic Ray for Python.
  • cargo-mutants for Rust and Infection for PHP.

We did not run these tools for this article. They add what our demo lacks: many more operators, parallel runs, timeouts, caching, and HTML reports that show each survivor in your source.

Cost and running it in CI

Every mutant means another test run. Our 14 mutants were instant, but a real module can produce thousands, so a full run can take hours. Practical ways to keep it affordable:

  • Run it on changed files only in pull requests, and on the full codebase nightly or weekly.
  • Point it at critical logic such as pricing, permissions and calculations, not generated code or simple getters.
  • Keep unit tests fast and deterministic; a flaky test makes mutants look killed at random, which our flaky tests guide covers.
  • Track the score over time instead of forcing a hard gate on day one.

When mutation testing is worth it

It fits best where a wrong answer is expensive: billing, tax, discounts, access control and data validation. It also helps when a team trusts coverage numbers too much, or when you inherit a large test suite and want to know whether it really protects you. For quick prototypes, UI layout or code with few unit tests, start with the basics in our test pyramid guide first. Pairing it with property-based testing is powerful, because generated inputs often kill survivors that hand-picked examples miss.

Final thoughts

Mutation testing tests your tests. In our small example, a suite with perfect line coverage still let three boundary bugs through, and the tool named each one. Adding two precise boundary checks fixed it. You do not need to mutate everything; run it where mistakes cost the most, and treat every survivor as a test you have not written yet.

Frequently asked questions

What is mutation testing in simple terms?

It deliberately inserts small bugs, called mutants, into the code and runs your tests. If a test fails, the bug was caught. Bugs that pass every test reveal weak spots in the suite.

What is a good mutation score?

There is no universal number. Aim high on critical logic and review survivors rather than chasing 100 percent, because equivalent mutants cannot be killed. Our example went from 79 to 100 percent.

How is mutation testing different from code coverage?

Coverage shows which lines ran during tests. Mutation testing shows whether tests notice wrong behaviour. Our suite had 100 percent line coverage but missed three boundary mutants.

Which tools are used for mutation testing?

Popular tools include PIT for Java, Stryker for JavaScript, TypeScript and C#, mutmut and Cosmic Ray for Python, cargo-mutants for Rust and Infection for PHP.

Why is mutation testing slow?

Each mutant needs its own test run, and real projects generate thousands of mutants. Limit runs to changed files or critical modules and keep unit tests fast.

Keep learning

Work with Stack Quora

Need help with testing, WordPress or SEO?

We help teams ship better software and grow their sites. Tell us what you are working on and we will suggest a clear next step.

Keep reading

More guides on testing, AI & modern development

Enjoyed this article? Pick a topic and jump straight into our newest tutorials.

Artificial Intelligence

BM25 Explained: How Keyword Search Ranks Results, With Python Code and Hybrid Search

BM25 explained with a pure Python implementation we ran: the formula, IDF, term saturation, the k1 and b parameters, why…

Read article →
Software Development

HTTP Methods Explained: GET, POST, PUT, PATCH, DELETE, Tested on a Live Server

HTTP methods explained with a small server we ran: GET, POST, PUT, PATCH, DELETE, HEAD and OPTIONS, safe vs idempotent,…

Read article →
DevOps

Git Bisect Explained: Find the Commit That Broke Your Code, Manually or Automatically

Git bisect explained with a tested repository: find the one bad commit among 41 in 5 test runs, automate it…

Read article →
Test Automation

pytest Markers Explained: Custom Marks, skip, skipif, xfail and -m Filters

pytest markers explained with a tested suite: register custom marks, run only smoke tests with -m, combine filters with and,…

Read article →
Software Testing

Positive vs Negative Testing: Differences, 24 Real Test Cases and a Checklist

Positive vs negative testing explained with a real signup form: 6 positive and 18 negative test cases run in pytest,…

Read article →
Databases & SQL

SQL ORDER BY Multiple Columns: ASC, DESC, NULLS LAST, Case and Custom Order

SQL ORDER BY with multiple columns explained with tested queries: mixed ASC and DESC, tie-breakers, NULLS FIRST and LAST, case-insensitive…

Read article →
Databases & SQL

SQL Running Total: SUM OVER, Per-Group Totals, Moving Sums and the RANGE Trap

How to calculate a SQL running total with SUM() OVER: per-group totals, daily cumulative sums, moving sums, percent of total,…

Read article →
Software Development

JavaScript Sort Array: Numbers, Objects, Strings and Why 10 Comes Before 5

How to sort an array in JavaScript correctly: numbers with a compare function, objects by one or more fields, strings…

Read article →
Software Development

Python String Methods: Cheat Sheet, Real Examples and the strip() Trap

Python string methods with tested examples: split, join, strip, replace, find, startswith, is-methods, casefold, partition and translate, plus the strip()…

Read article →
Software Development

Python zip() Explained: Loop Two Lists at Once, Unzip, strict=True and Traps

Python zip() pairs items from several lists in one loop. Tested examples for dicts, unzipping with *, strict=True, zip_longest, sliding…

Read article →
Artificial Intelligence

Cosine Similarity Explained: Formula, Worked Example and Python Code

Cosine similarity explained simply: the formula, a worked example by hand, pure Python and NumPy code, cosine vs Euclidean distance…

Read article →
Software Development

UTF-8 Encoding Explained: How Text Becomes Bytes and How to Fix Garbled Text

UTF-8 encoding explained with byte-level examples we ran in Python: how characters become 1 to 4 bytes, why café turns…

Read article →
DevOps

Git Squash Commits: 4 Easy Ways, Tested Step by Step (rebase -i, reset, merge)

Git squash commits explained with 4 tested methods: interactive rebase, reset –soft, merge –squash and fixup with autosquash. Before-and-after logs,…

Read article →
Test Automation

pytest parametrize Explained: 7 Patterns with Tested Examples

pytest parametrize explained with 7 tested patterns: basic cases, readable ids, stacked decorators, pytest.param with xfail and skip, indirect fixtures…

Read article →
Software Testing

Retesting vs Regression Testing: 8 Key Differences with a Real Bug Fix Example

Retesting vs regression testing explained with a real bug fix run in pytest: retesting confirmed the fix, the regression run…

Read article →
Databases & SQL

Delete Duplicate Rows in SQL: 5 Safe Methods, Tested Step by Step

How to delete duplicate rows in SQL safely: find them, preview them, then remove them with ROW_NUMBER, MIN(id), EXISTS, rowid…

Read article →
Databases & SQL

SQL RANK vs DENSE_RANK vs ROW_NUMBER: Differences with Tested Examples

SQL RANK vs DENSE_RANK vs ROW_NUMBER in one table: how each handles ties, top-N per group, the second-highest salary trap…

Read article →
Software Development

JavaScript Optional Chaining (?.) Explained: 8 Examples, ?? Defaults and Traps

JavaScript optional chaining stops “Cannot read properties of undefined” errors. 8 tested examples for properties, arrays and methods, ?? vs…

Read article →
Artificial Intelligence

RAG Chunking Strategies Explained: Fixed, Overlap, Sentence, Recursive and Headings, Measured

RAG chunking strategies compared on one document with tiktoken 0.14: fixed-size, overlap, sentence packing, recursive splitting and heading-based chunks, measuring…

Read article →
DevOps

Git Stash Explained: Save, Apply, Pop and Untracked Files, Tested

Git stash explained with commands we ran on Git 2.56: push with a message, untracked files, list and show, apply…

Read article →
Software Testing

Pairwise Testing Explained: How All-Pairs Cuts 216 Tests to 13, Tested

Pairwise testing explained with a generator we ran in Python 3.13: a checkout with 216 combinations covered by 13 tests,…

Read article →
Test Automation

Test Doubles Explained: Dummy, Stub, Spy, Mock and Fake, with Tested Python Code

Test doubles explained with one Python example run in pytest 9.2: dummy, stub, spy, mock and fake, when to use…

Read article →
Databases & SQL

SQL COALESCE Explained: Defaults, NULLIF, IFNULL vs ISNULL and Traps, Tested

SQL COALESCE explained with queries we ran in SQLite 3.50: first non-null values, empty strings, NULL in maths, NULLIF for…

Read article →
Databases & SQL

SQL Self Join Explained: Employee-Manager, Pairs and Row Comparisons, Tested

SQL self join explained with queries we ran in SQLite 3.50: employee and manager, earning more than the manager, unique…

Read article →
Artificial Intelligence

LLM Context Window Explained: Tokens, Limits and How to Fit More, Measured

The LLM context window explained with token counts we measured using tiktoken: words vs tokens for prose, code and Hindi,…

Read article →
DevOps

Git Cherry-Pick Explained: Copy the Commit You Need, Safely, Tested

Git cherry-pick explained with commands we ran in Git 2.56: copying a hotfix to a release branch, new hashes and…

Read article →
Test Automation

Snapshot Testing Explained: How It Works, When It Helps and When It Hurts

Snapshot testing explained with a pytest and syrupy demo we ran: creating snapshots, reading the diff after a code change,…

Read article →
Software Testing

Mutation Testing Explained: Why 100% Coverage Still Missed Our Bug

Mutation testing explained with a small mutation tool we built and ran: 14 mutants, a test suite with 100% line…

Read article →
Artificial Intelligence

Prompt Injection Explained: How AI Apps Get Tricked and How to Defend Them

Prompt injection explained for builders: direct and indirect attacks, why keyword filters fail (our test blocked 1 of 4), and…

Read article →
DevOps

DNS Records Explained: A, AAAA, CNAME, MX, TXT and TTL With Real Lookups

DNS records explained with lookups we ran: how resolution works, A and AAAA, CNAME rules, MX and null MX, TXT…

Read article →
DevOps

Feature Flags Explained: Safe Releases, Gradual Rollouts and Flag Hygiene

Feature flags explained with a rollout simulation we ran: flag types, percentage rollouts with stable hashing, kill switches, allow lists,…

Read article →
Test Automation

Test Data Generation With Python Faker: Realistic, Repeatable and Safe Data

Test data generation with Python Faker, run and measured: seeding for repeatable data, locales, unique values, domain-shaped records, edge cases…

Read article →
Software Testing

Shift-Left Testing Explained: Find Bugs Earlier Without Slowing the Team

Shift-left testing explained: what it means, why early defects are cheaper, practices from requirement reviews to CI checks, the tester's…

Read article →
Artificial Intelligence

LLM Structured Output Explained: Getting Reliable JSON From AI Models

LLM structured output explained with validation we ran in Pydantic: JSON mode and schemas, constrained decoding, the ways model output…

Read article →
Test Automation

API Contract Testing Explained: Catch Breaking Changes Before They Ship

API contract testing explained with a schema check we ran: what a contract is, consumer-driven contracts, Pact, OpenAPI, which changes…

Read article →
Software Testing

Risk-Based Testing Explained: Score Risks, Build a Matrix and Test What Matters

Risk-based testing explained step by step: identifying risks, scoring likelihood and impact, building a risk matrix, mapping risk to test…

Read article →