Test Case Prioritization: 6 Techniques Compared With APFD
Test case prioritization orders a test suite so the tests most likely to find important bugs run first. That matters when time is short, builds are slow or a release is tomorrow. We took a ten-test regression suite with six known faults, ordered it six different ways, scored every order with APFD, and measured how many minutes each took to find all the faults.
- Test case prioritization reorders tests so faults are found earlier; it does not remove any tests.
- Ordering by faults found per minute in the last release found all six faults in 24 minutes, versus 42 for the reverse order.
- Three orders scored the same APFD, 0.683, yet took 24, 27 and 35 minutes to find every fault, because APFD ignores test duration.
- Random orders averaged an APFD of 0.577, ranging from 0.333 to 0.783.
Full regression suites keep growing while the time available to run them does not. When a run takes hours, or a hotfix must ship today, the order of tests decides how soon you learn about a problem. Test case prioritization is the practice of running the most valuable tests first, so that failures surface in minutes rather than at the end of the run, and so that a cut-short run still covers what matters most.
We built a realistic ten-test suite with known faults from a past release, then computed every ordering and score in Python 3.13.6. The numbers are illustrative but the method is exactly what you would apply to your own suite.
What is test case prioritization?
Test case prioritization (TCP) arranges existing test cases in an order that maximises some goal, most often the rate of fault detection. Other goals include covering high-risk features early, covering code faster, or getting the fastest possible feedback on a pull request. Every test still runs if time allows; only the order changes.
Prioritization vs selection vs minimisation
- Prioritization: run everything, but in a better order.
- Selection: run only the tests relevant to a change, for example those touching modified files.
- Minimisation: permanently remove redundant tests from the suite.
They combine well: select the relevant tests for a pull request, prioritise them, and run the full prioritised suite nightly as part of the regression process.
Our test suite
| Test | Faults it detects | Minutes | Risk (1 to 5) | Coverage % |
|---|---|---|---|---|
| T1 login | 1 | 2 | 5 | 30 |
| T2 checkout | 2, 3 | 6 | 5 | 45 |
| T3 search | none | 3 | 2 | 25 |
| T4 profile | 4 | 2 | 2 | 15 |
| T5 payments | 2, 5 | 8 | 5 | 40 |
| T6 reports | none | 10 | 1 | 35 |
| T7 coupons | 3 | 3 | 3 | 20 |
| T8 email | 6 | 1 | 3 | 10 |
| T9 admin | none | 5 | 1 | 30 |
| T10 cart | 2 | 2 | 4 | 25 |
Ten tests take 42 minutes in total. Six faults are known: some are caught by one test only (fault 6 only by the email test), others by several (fault 2 by checkout, payments and cart). In real life you never know the faults in advance, which is why techniques rely on proxies such as risk, coverage and history.
How APFD scores an order
APFD = 1 - (TF1 + TF2 + ... + TFm) / (n * m) + 1 / (2n)
n = number of tests (10)
m = number of faults (6)
TFi = position of the first test that reveals fault i
APFD (Average Percentage of Faults Detected) measures how early faults are found. It is the area under the curve of “percentage of faults found” against “percentage of tests run”. A value near 1 means faults are found almost immediately; lower values mean they appear late. It is the most widely used metric in prioritization research.
Results for six techniques
| Order | APFD | Minutes to find all 6 faults | First three tests |
|---|---|---|---|
| History: faults per minute | 0.683 | 24 | T8, T1, T4 |
| Risk (highest first) | 0.683 | 27 | T1, T2, T5 |
| Risk per minute | 0.667 | 27 | T8, T1, T10 |
| Shortest first | 0.633 | 32 | T8, T1, T4 |
| Coverage (highest first) | 0.600 | 42 | T2, T5, T6 |
| Random (mean of 10,000) | 0.577 | varies | random |
| Baseline: as written | 0.683 | 35 | T1, T2, T3 |
| Baseline: reverse | 0.533 | 42 | T10, T9, T8 |
Risk-based prioritization
Ranking tests by the business risk of the feature they cover (login, checkout and payments first) found all faults in 27 minutes and matched the best APFD. Risk-based ordering is the most common approach in practice because it needs no tooling, only a shared view of what would hurt most if it broke; our risk-based testing guide explains how to score likelihood and impact. Its weakness is subjectivity: risk scores drift out of date unless they are reviewed.
Coverage-based prioritization
Ordering by total code coverage put long, broad tests such as checkout, payments and reports first. It scored only 0.600 and needed the full 42 minutes to find every fault, because the reports test covered a lot of code but found nothing, and the one-minute email test that alone catches fault 6 ran near the end. Coverage-based ordering is popular in research and in tools, often in an “additional coverage” variant that picks the test adding the most new coverage at each step. Coverage says which code ran, not whether a test would notice a bug there, which mutation testing measures better.
Cost-aware orders
Running the shortest tests first finds cheap wins early: the one-minute email test and two-minute login and profile tests found three faults in five minutes. Dividing risk by duration combined both ideas and matched the risk order’s 27 minutes while starting with faster feedback. Cost matters most in CI, where a developer waiting for a pull request result cares about the first failure, not about running the whole suite.
History-based prioritization
Ordering by how many faults each test found in the previous release, per minute of runtime, was the best on every measure: APFD 0.683 and all faults in 24 minutes. Tests that failed recently are statistically more likely to fail again, because the code they cover is volatile or fragile. CI systems make this easy to automate from stored test results. Watch out for flaky tests, which fail often for reasons unrelated to real faults and would otherwise climb to the top of a history-based order.
The random baseline
10,000 random orders: mean APFD 0.577, minimum 0.333, maximum 0.783
Random ordering is the baseline every technique should beat. Most of our techniques did, but the coverage order (0.600) barely improved on chance, and the reverse of the written order (0.533) was worse than average. The spread from 0.333 to 0.783 shows how much order matters even when exactly the same tests run.
Why APFD is not enough
Three orders tied at an APFD of 0.683, yet they needed 24, 27 and 35 minutes to find all faults. APFD counts positions in the order, treating a one-minute test and a ten-minute test as equal steps, and it treats a typo and a payment failure as equally important faults. Research variants address this: APFDc weights by test cost and fault severity. In practice, also report “time to first failure” and “time to find all known faults”, which are what developers and release managers actually feel.
With a fixed time budget
15-minute budget:
as written 5 of 6 faults in 14 min
risk per minute 5 of 6 faults in 13 min
history-based 5 of 6 faults in 13 min
When only 15 minutes were available, as with a quick smoke run before a hotfix, three orders each caught five of the six faults. With a suite this small the differences shrink; with hundreds of tests and a budget covering only a fraction of them, prioritization decides which faults you see at all. Pair this with a separate smoke suite for the absolute essentials.
Putting it into practice
- Tag each test with the feature it covers and a risk score agreed with product owners.
- Record duration and pass/fail history for every run in CI.
- Order by a simple combined score, such as (risk + recent failures) divided by duration.
- Run in that order and fail fast, reporting the first failure immediately.
- Review the order each release, and compare its APFD and time-to-first-failure against the previous release.
Prioritization also feeds planning: knowing which tests matter most helps when building a test estimate under a fixed deadline, and modules with high defect density deserve higher risk scores.
Final thoughts
Test case prioritization costs nothing in test coverage and can halve the time to find faults. In our suite, history-based ordering found every known fault in 24 minutes, risk-based in 27, while pure coverage ordering needed all 42 and barely beat random. Score orders with APFD, but also measure time to first failure, because APFD alone could not tell apart orders that differed by 11 minutes. Start with risk, add history from CI, account for test duration, and review the order every release.
Frequently asked questions
What is test case prioritization?
Ordering the tests in a suite so that the ones most likely to find important faults run first. All tests still run if time allows; only the order changes.
What is APFD in software testing?
Average Percentage of Faults Detected, a score between 0 and 1 that measures how early a test order finds faults. Higher values mean faults are found earlier in the run.
Which test case prioritization technique is best?
It depends on the data you have. History-based ordering using recent failures and durations did best in our example, with risk-based ordering a close second and the easiest to start with.
What is the difference between test prioritization and test selection?
Prioritization changes the order of all tests. Selection chooses a subset to run, usually the tests related to a code change. Teams often use both together.
Why is coverage-based prioritization sometimes poor?
Broad tests cover a lot of code but may not detect faults there. In our suite the highest-coverage order needed the full run time to find every fault.
Keep learning
Need help with testing, WordPress or SEO?
We help teams ship better software and grow their sites. Tell us what you are working on and we will suggest a clear next step.