1. Home
  2. Blog
  3. Software Testing

Software Testing

Test Case Prioritization: 6 Techniques Compared With APFD

Test case prioritization orders a test suite so the tests most likely to find important bugs run first. That matters when time is short, builds are slow or a release is tomorrow. We took a ten-test regression suite with six known faults, ordered it six different ways, scored every order with APFD, and measured how many minutes each took to find all the faults.

9 min read
Test case prioritization: 6 techniques compared with APFD

Key takeaways

  • Test case prioritization reorders tests so faults are found earlier; it does not remove any tests.
  • Ordering by faults found per minute in the last release found all six faults in 24 minutes, versus 42 for the reverse order.
  • Three orders scored the same APFD, 0.683, yet took 24, 27 and 35 minutes to find every fault, because APFD ignores test duration.
  • Random orders averaged an APFD of 0.577, ranging from 0.333 to 0.783.

Full regression suites keep growing while the time available to run them does not. When a run takes hours, or a hotfix must ship today, the order of tests decides how soon you learn about a problem. Test case prioritization is the practice of running the most valuable tests first, so that failures surface in minutes rather than at the end of the run, and so that a cut-short run still covers what matters most.

We built a realistic ten-test suite with known faults from a past release, then computed every ordering and score in Python 3.13.6. The numbers are illustrative but the method is exactly what you would apply to your own suite.

What is test case prioritization?

Test case prioritization (TCP) arranges existing test cases in an order that maximises some goal, most often the rate of fault detection. Other goals include covering high-risk features early, covering code faster, or getting the fastest possible feedback on a pull request. Every test still runs if time allows; only the order changes.

Prioritization vs selection vs minimisation

  • Prioritization: run everything, but in a better order.
  • Selection: run only the tests relevant to a change, for example those touching modified files.
  • Minimisation: permanently remove redundant tests from the suite.

They combine well: select the relevant tests for a pull request, prioritise them, and run the full prioritised suite nightly as part of the regression process.

Our test suite

Test Faults it detects Minutes Risk (1 to 5) Coverage %
T1 login 1 2 5 30
T2 checkout 2, 3 6 5 45
T3 search none 3 2 25
T4 profile 4 2 2 15
T5 payments 2, 5 8 5 40
T6 reports none 10 1 35
T7 coupons 3 3 3 20
T8 email 6 1 3 10
T9 admin none 5 1 30
T10 cart 2 2 4 25

Ten tests take 42 minutes in total. Six faults are known: some are caught by one test only (fault 6 only by the email test), others by several (fault 2 by checkout, payments and cart). In real life you never know the faults in advance, which is why techniques rely on proxies such as risk, coverage and history.

How APFD scores an order

APFD = 1 - (TF1 + TF2 + ... + TFm) / (n * m) + 1 / (2n)

n   = number of tests (10)
m   = number of faults (6)
TFi = position of the first test that reveals fault i

APFD (Average Percentage of Faults Detected) measures how early faults are found. It is the area under the curve of “percentage of faults found” against “percentage of tests run”. A value near 1 means faults are found almost immediately; lower values mean they appear late. It is the most widely used metric in prioritization research.

Results for six techniques

Order APFD Minutes to find all 6 faults First three tests
History: faults per minute 0.683 24 T8, T1, T4
Risk (highest first) 0.683 27 T1, T2, T5
Risk per minute 0.667 27 T8, T1, T10
Shortest first 0.633 32 T8, T1, T4
Coverage (highest first) 0.600 42 T2, T5, T6
Random (mean of 10,000) 0.577 varies random
Baseline: as written 0.683 35 T1, T2, T3
Baseline: reverse 0.533 42 T10, T9, T8

Risk-based prioritization

Ranking tests by the business risk of the feature they cover (login, checkout and payments first) found all faults in 27 minutes and matched the best APFD. Risk-based ordering is the most common approach in practice because it needs no tooling, only a shared view of what would hurt most if it broke; our risk-based testing guide explains how to score likelihood and impact. Its weakness is subjectivity: risk scores drift out of date unless they are reviewed.

Coverage-based prioritization

Ordering by total code coverage put long, broad tests such as checkout, payments and reports first. It scored only 0.600 and needed the full 42 minutes to find every fault, because the reports test covered a lot of code but found nothing, and the one-minute email test that alone catches fault 6 ran near the end. Coverage-based ordering is popular in research and in tools, often in an “additional coverage” variant that picks the test adding the most new coverage at each step. Coverage says which code ran, not whether a test would notice a bug there, which mutation testing measures better.

Cost-aware orders

Running the shortest tests first finds cheap wins early: the one-minute email test and two-minute login and profile tests found three faults in five minutes. Dividing risk by duration combined both ideas and matched the risk order’s 27 minutes while starting with faster feedback. Cost matters most in CI, where a developer waiting for a pull request result cares about the first failure, not about running the whole suite.

History-based prioritization

Ordering by how many faults each test found in the previous release, per minute of runtime, was the best on every measure: APFD 0.683 and all faults in 24 minutes. Tests that failed recently are statistically more likely to fail again, because the code they cover is volatile or fragile. CI systems make this easy to automate from stored test results. Watch out for flaky tests, which fail often for reasons unrelated to real faults and would otherwise climb to the top of a history-based order.

The random baseline

10,000 random orders: mean APFD 0.577, minimum 0.333, maximum 0.783

Random ordering is the baseline every technique should beat. Most of our techniques did, but the coverage order (0.600) barely improved on chance, and the reverse of the written order (0.533) was worse than average. The spread from 0.333 to 0.783 shows how much order matters even when exactly the same tests run.

Why APFD is not enough

Three orders tied at an APFD of 0.683, yet they needed 24, 27 and 35 minutes to find all faults. APFD counts positions in the order, treating a one-minute test and a ten-minute test as equal steps, and it treats a typo and a payment failure as equally important faults. Research variants address this: APFDc weights by test cost and fault severity. In practice, also report “time to first failure” and “time to find all known faults”, which are what developers and release managers actually feel.

With a fixed time budget

15-minute budget:
as written       5 of 6 faults in 14 min
risk per minute  5 of 6 faults in 13 min
history-based    5 of 6 faults in 13 min

When only 15 minutes were available, as with a quick smoke run before a hotfix, three orders each caught five of the six faults. With a suite this small the differences shrink; with hundreds of tests and a budget covering only a fraction of them, prioritization decides which faults you see at all. Pair this with a separate smoke suite for the absolute essentials.

Putting it into practice

  1. Tag each test with the feature it covers and a risk score agreed with product owners.
  2. Record duration and pass/fail history for every run in CI.
  3. Order by a simple combined score, such as (risk + recent failures) divided by duration.
  4. Run in that order and fail fast, reporting the first failure immediately.
  5. Review the order each release, and compare its APFD and time-to-first-failure against the previous release.

Prioritization also feeds planning: knowing which tests matter most helps when building a test estimate under a fixed deadline, and modules with high defect density deserve higher risk scores.

Final thoughts

Test case prioritization costs nothing in test coverage and can halve the time to find faults. In our suite, history-based ordering found every known fault in 24 minutes, risk-based in 27, while pure coverage ordering needed all 42 and barely beat random. Score orders with APFD, but also measure time to first failure, because APFD alone could not tell apart orders that differed by 11 minutes. Start with risk, add history from CI, account for test duration, and review the order every release.

Frequently asked questions

What is test case prioritization?

Ordering the tests in a suite so that the ones most likely to find important faults run first. All tests still run if time allows; only the order changes.

What is APFD in software testing?

Average Percentage of Faults Detected, a score between 0 and 1 that measures how early a test order finds faults. Higher values mean faults are found earlier in the run.

Which test case prioritization technique is best?

It depends on the data you have. History-based ordering using recent failures and durations did best in our example, with risk-based ordering a close second and the easiest to start with.

What is the difference between test prioritization and test selection?

Prioritization changes the order of all tests. Selection chooses a subset to run, usually the tests related to a code change. Teams often use both together.

Why is coverage-based prioritization sometimes poor?

Broad tests cover a lot of code but may not detect faults there. In our suite the highest-coverage order needed the full run time to find every fault.

Keep learning

Work with Stack Quora

Need help with testing, WordPress or SEO?

We help teams ship better software and grow their sites. Tell us what you are working on and we will suggest a clear next step.

Keep reading

More guides on testing, AI & modern development

Enjoyed this article? Pick a topic and jump straight into our newest tutorials.

Artificial Intelligence

Confusion Matrix Explained: Precision, Recall, F1 and Thresholds in Python

Confusion matrix explained with Python: TP, FP, FN, TN, precision, recall, F1, the accuracy trap, threshold choice by cost and…

Read article →
Software Development

API Pagination: Offset vs Cursor vs Keyset, Tested on 1 Million Rows

API pagination explained: offset vs keyset and cursor pagination timed on 1 million rows, duplicates when data changes, the ties…

Read article →
DevOps

Git Submodule Explained: Add, Clone, Update and Remove Safely

Git submodule explained: add, clone, update –init, update –remote, the detached HEAD trap, foreach and clean removal, tested step by…

Read article →
Test Automation

Pytest tmp path Explained: tmp_path, tmp_path_factory and Temp Files

Pytest tmp_path explained: per-test temporary folders, tmp_path_factory for shared files, monkeypatch.chdir, legacy tmpdir, retention and basetemp.

Read article →
Software Testing

Test Case Prioritization: 6 Techniques Compared With APFD

Test case prioritization compared: risk, coverage, cost, history and random orders scored with APFD and time to find all faults…

Read article →
Databases & SQL

Full Text Search in SQL: FTS5, Ranking, Phrases and Speed

Full text search in SQL with SQLite FTS5: MATCH queries, phrases, prefixes, NEAR, stemming, BM25 ranking, snippets, two gotchas and…

Read article →
Databases & SQL

Generated Columns in SQL: Virtual vs Stored, Indexes and Limits

Generated columns in SQL explained: virtual vs stored, automatic updates, write errors, storage cost, indexing, JSON fields and support in…

Read article →
Software Development

Object Freeze in JavaScript: Shallow vs Deep, Seal and const

Object freeze in JavaScript explained: freeze vs seal vs const, strict mode errors, shallow vs deep freeze, frozen arrays, Map…

Read article →
Software Development

Python hashlib: SHA-256, File Hashes, HMAC and Password Hashing

Python hashlib explained: SHA-256 for strings and files, algorithm speeds, why passwords need PBKDF2 or scrypt, safe verification and HMAC…

Read article →
Software Development

Python UUID Explained: uuid1, uuid4, uuid5, uuid7 and Database Keys

Python UUID explained: uuid4, uuid5, uuid1 privacy, time-ordered uuid7, parsing, storage formats and UUIDs as database keys, tested on Python…

Read article →
Artificial Intelligence

Random Forest in Python: Bagging, Feature Sampling and OOB From Scratch

Random forest in Python from scratch: bootstrap sampling, feature sampling, number of trees, out-of-bag evaluation and impurity vs permutation importance.

Read article →
Software Development

MIME Types Explained: Content-Type, nosniff and a Common Types List

MIME types explained and tested in a browser: Content-Type, nosniff, scripts and stylesheets with wrong types, a common MIME types…

Read article →
DevOps

Git Clean Explained: Remove Untracked Files Without Regret

Git clean explained with dry runs: -n, -f, -d, -x, -X, exclusions, paths, nested repositories and interactive mode, plus a…

Read article →
Test Automation

Pytest approx: Comparing Floats With rel, abs and Collections

Pytest approx explained: default tolerances, the near-zero trap, rel and abs, lists and dicts, NaN, NumPy, and how it differs…

Read article →
Software Testing

Accessibility Testing: WCAG Checklist, Automated Tools and Manual Checks

Accessibility testing explained: WCAG basics, an axe-core scan of a broken and a fixed page, what automation missed, contrast checks…

Read article →
Databases & SQL

SQL UNPIVOT: Turn Columns Into Rows, 4 Ways Tested

SQL UNPIVOT explained: turn columns into rows with UNION ALL, CROSS JOIN with CASE, JSON functions or native UNPIVOT, with…

Read article →
Software Development

Intl NumberFormat in JavaScript: Currency, Compact, Percent and Units

Intl.NumberFormat in JavaScript: locales, currency, compact notation, percent, units, rounding, formatToParts, the non-breaking space trap and formatter reuse.

Read article →
Databases & SQL

SQL EXPLAIN Plan: How to Read Query Plans and Fix Slow Queries

SQL EXPLAIN plan explained: read scans, index searches, covering indexes and temporary sorts, with before and after timings for five…

Read article →
Artificial Intelligence

Decision Tree in Python: Gini, Entropy and Overfitting From Scratch

Decision tree in Python from scratch: Gini impurity, entropy, information gain, best splits, readable rules, and how depth and leaf…

Read article →
DevOps

Git Blame Explained: Find Who Changed a Line, and Why

Git blame explained with a real repository: ignore whitespace, line ranges and functions, ignore-revs files for formatting commits, moved code…

Read article →
Test Automation

Pytest Benchmark: Measure Speed and Catch Performance Regressions

Pytest benchmark tutorial: the benchmark fixture, reading results, parametrized and pedantic benchmarks, saving baselines and failing the build on regressions.

Read article →
Software Testing

Defect Density: Formula, Worked Examples and How to Use It

Defect density explained: the formula, a five-module worked example, per KLOC vs per function point, severity weighting, defect removal efficiency…

Read article →
Databases & SQL

Case Insensitive Search in SQL: LOWER, LIKE, COLLATE and Indexes

Case insensitive search in SQL: LOWER, LIKE, COLLATE NOCASE and expression indexes compared with query plans, plus accented letters and…

Read article →
Databases & SQL

ON DELETE CASCADE in SQL: CASCADE vs SET NULL vs RESTRICT

ON DELETE CASCADE in SQL tested: CASCADE vs SET NULL vs RESTRICT, multi-level cascades, the SQLite foreign key pragma, row…

Read article →
Artificial Intelligence

Logistic Regression in Python: Sigmoid, Log Loss and Gradient Descent From Scratch

Logistic regression in Python from scratch: sigmoid, log loss, gradient descent, odds ratios, thresholds, sigmoid overflow and separable data, tested…

Read article →
DevOps

Git Worktree Explained: Work on Two Branches at Once

Git worktree explained: add, list, remove, prune, lock and detached worktrees, a hotfix workflow, and the errors you will hit,…

Read article →
Test Automation

Pytest ini Explained: addopts, markers, testpaths and Precedence

Pytest ini explained: testpaths, addopts, strict markers, xfail_strict, filterwarnings, minversion, config file precedence and command-line overrides, tested.

Read article →
Software Testing

Test Estimation Techniques: WBS, Three Point PERT and Monte Carlo

Test estimation techniques worked through: WBS, three point PERT, total uncertainty, a Monte Carlo check, percentage rules and Wideband Delphi.

Read article →
Artificial Intelligence

Linear Regression in Python: Least Squares From Scratch, With R Squared

Linear regression in Python from scratch: closed form, NumPy least squares and gradient descent, R squared, RMSE, outliers and curved…

Read article →
DevOps

Git Remote Explained: Add, Set-URL, Rename, Remove and Push URLs

Git remote explained: add, set-url, separate push URLs, rename, remove and prune, plus how to avoid leaking tokens in remote…

Read article →
Test Automation

Pytest Mock: mocker.patch, spy, side_effect and autospec Explained

Pytest mock explained: mocker.patch, return_value, side_effect, spy, autospec and PropertyMock, with tested examples for APIs, clocks and retries.

Read article →
Software Testing

System Testing Explained: Types, Process and a Real End-to-End Example

System testing explained: types, entry and exit criteria, the process, and a real end-to-end test suite run against a small…

Read article →
Artificial Intelligence

Naive Bayes Classifier Explained: Build a Spam Filter in Python

Naive Bayes classifier explained by building a spam filter in Python: Bayes rule, the naive assumption, Laplace smoothing, log probabilities…

Read article →
DevOps

Git Branch Explained: Create, Switch, Rename, Delete and Recover

Git branch explained with tested commands: list, create, switch, rename, delete, merged checks, recovering deleted branches, remote tracking and naming.

Read article →
Test Automation

Pytest raises: Testing Exceptions, match and excinfo

Pytest raises explained with tested examples: match regexes, excinfo, custom and chained exceptions, parametrized error cases, pytest.warns and common mistakes.

Read article →
Software Testing

Non Functional Testing: Types, Metrics and a Real Load Test

Non functional testing explained: 10 key types, measurable requirements, latency percentiles and what a small real load test revealed about…

Read article →