Pairwise Testing Explained: How All-Pairs Cuts 216 Tests to 13, Tested
Pairwise testing is a test design technique that covers every combination of values for every pair of inputs, without testing every full combination. See how it shrank a checkout matrix from 216 cases to 13, how we verified the coverage, and where it falls short.
- Pairwise testing (all-pairs testing) makes sure every value of every input appears with every value of every other input at least once.
- Our generator covered all 89 value pairs of a five-input checkout in 13 tests, against 216 exhaustive combinations.
- With 13 inputs of 3 values each, 22 tests covered every pair of a space of 1,594,323 combinations.
- It finds two-factor bugs reliably but can miss three-factor ones: our set had no Safari + UPI + French test.
Modern software runs on a mess of combinations: browsers, operating systems, languages, payment methods, account types, feature flags. Testing every combination quickly becomes impossible. Five settings with a handful of values each already produce hundreds of cases, and each new setting multiplies the total again. Pairwise testing is a systematic way to cut that number down while keeping the combinations most likely to reveal bugs.
We built a small pairwise generator in Python 3.13, ran it on a realistic checkout example, and wrote a separate check that every pair really is covered. The numbers below come from that run. If you are new to test design techniques, our guides to equivalence partitioning and boundary value analysis are good companions.
What is pairwise testing?
Pairwise testing is a black-box, combinatorial test design technique. For every pair of input parameters, it makes sure each possible combination of their values appears in at least one test. It does not try every combination of all parameters together.
Suppose you test browser and operating system. Pairwise testing guarantees that Chrome on Windows, Chrome on macOS, Safari on Android and every other browser-OS pair appear somewhere. Because one test case holds many pairs at once (browser-OS, browser-language, OS-payment and so on), a small set of tests can cover them all.
You may also hear it called all-pairs testing, or seen as a special case of combinatorial testing with strength 2. Orthogonal array testing is a related, more mathematical way of building such sets.
Why pairs are enough most of the time
The idea rests on an observation from field studies: most defects are triggered by a single input value or by the interaction of two inputs. Research by the US National Institute of Standards and Technology (NIST) on real failures found that the share of bugs needing three or more interacting factors falls off quickly. Pairwise testing aims at the large middle: bugs that appear only when two settings meet, such as a layout that breaks only in Safari when the language is French.
Single-factor bugs are covered automatically, because every value appears at least once. The technique is a bet on probability, not a guarantee, and that matters later.
A checkout example
We modelled the checkout page of an online shop with five parameters:
| Parameter | Values | Count |
|---|---|---|
| Browser | Chrome, Firefox, Safari | 3 |
| Operating system | Windows, macOS, Android | 3 |
| Payment | Card, UPI, Wallet | 3 |
| User type | Guest, Member | 2 |
| Language | en, hi, fr, es | 4 |
Testing every combination means 3 × 3 × 3 × 2 × 4 = 216 tests. Across the ten pairs of parameters there are 89 value pairs to cover. The theoretical minimum is 12 tests, because the two largest parameters, language (4) and browser (3), alone form 12 pairs that each need their own row.
The 13 pairwise tests
Our greedy generator repeatedly picks the test case that covers the most still-uncovered pairs, until none remain. It produced 13 tests:
| # | Browser | OS | Payment | User | Language |
|---|---|---|---|---|---|
| 1 | Chrome | Windows | Card | Member | en |
| 2 | Firefox | Windows | UPI | Guest | es |
| 3 | Firefox | macOS | Wallet | Member | hi |
| 4 | Safari | macOS | Card | Guest | fr |
| 5 | Chrome | Android | UPI | Member | fr |
| 6 | Safari | Android | Wallet | Guest | en |
| 7 | Safari | Windows | UPI | Guest | hi |
| 8 | Chrome | macOS | Wallet | Guest | es |
| 9 | Safari | Android | Card | Member | es |
| 10 | Firefox | Windows | Wallet | Guest | fr |
| 11 | Firefox | macOS | UPI | Guest | en |
| 12 | Firefox | Android | Card | Guest | hi |
| 13 | Chrome | Windows | Card | Guest | hi |
That is a 94% reduction, one test above the theoretical minimum. Some combinations are unrealistic, such as Safari on Windows or Android in row 6. In real projects you add constraints to exclude impossible pairs; dedicated tools support this directly.
Verifying the coverage
Never trust a generated set blindly. We checked it with an independent function that looks for every value pair in the tests:
import itertools
def covered(names, values, tests):
for i, j in itertools.combinations(range(len(names)), 2):
for a in values[i]:
for b in values[j]:
if not any(t[i] == a and t[j] == b for t in tests):
return False
return True
exhaustive: 216 pairwise: 13 all pairs covered: True pairs: 89
A checker like this is also useful when someone hand-edits a pairwise table later, for example to swap an unrealistic row. One careless change can silently drop a pair.
How pairwise testing scales
The real win shows up as the number of inputs grows. Exhaustive testing grows exponentially, while pairwise sets grow roughly with the logarithm of the number of parameters. We ran our generator on three larger configurations:
| Configuration | Exhaustive tests | Pairwise tests |
|---|---|---|
| 10 parameters × 2 values | 1,024 | 11 |
| 20 parameters × 2 values | 1,048,576 | 16 |
| 13 parameters × 3 values | 1,594,323 | 22 |
All three sets passed the coverage check. Twenty on/off feature flags have over a million combinations, yet 16 tests touched every pair of flag states. Mature tools often produce slightly smaller sets than our simple greedy approach, but the order of magnitude is the same.
What pairwise testing misses
Pairwise testing guarantees pairs, not triples. We simulated two bugs against our 13 tests:
- A bug that needs Safari + UPI (two factors) was caught, by test 7.
- A bug that needs Safari + UPI + French (three factors) was missed. No test combined all three.
A third simulated bug, Safari + UPI + Hindi, happened to be caught because test 7 includes Hindi, but that was luck rather than design. If interactions between three settings are risky in your system, raise the strength to 3-way testing. Tools support any strength, at the cost of more tests.
Tip: pairwise testing chooses which combinations to run, not which values. Pick values with equivalence partitioning and boundary analysis first, otherwise you will cover every pair of the wrong inputs.
Pairwise testing step by step
- List the parameters that change behaviour: environment, inputs, settings, user roles.
- Choose representative values for each, using partitions and boundaries rather than every possible value.
- Add constraints for impossible combinations, such as Safari only on macOS and iOS.
- Generate the set with a tool, and note its strength (2-way, 3-way).
- Verify coverage with the tool’s report or a checker like ours.
- Add must-have cases by hand: the most common real-world configuration and any combination that failed before.
- Define expected results for each row, then automate where possible, for example as a parameterised test.
Pairwise testing tools
You rarely need to write your own generator. Microsoft’s open-source PICT is a widely used command-line tool with constraint support. In Python, the allpairspy package generates pairs, and NIST’s ACTS tool supports higher strengths. We did not run these tools for this article; our numbers come from our own generator. Whatever you use, feed the result into data-driven tests, as shown in our test data generation guide.
When to use pairwise testing
Use it when you have several independent parameters with a few values each: browser and device matrices, configuration options, form fields with dropdowns, API query parameters and feature flag combinations. Skip it when parameters depend heavily on each other, when a single input has a huge range (use boundary analysis instead), or when the business rules are explicit conditions, which decision table testing handles better. For safety-critical systems, pairwise is a floor, not a ceiling.
Final thoughts
Pairwise testing turns an impossible matrix into a short, defensible list. In our checkout example, 13 tests covered every pair that 216 exhaustive tests would have covered, and the savings grew dramatically with more inputs. Choose good values first, add constraints and must-have cases, verify the coverage, and raise the strength where three-way interactions are a real risk. Then spend the time you saved on the riskiest areas, as described in our risk-based testing guide.
Frequently asked questions
What is pairwise testing in software testing?
It is a combinatorial test design technique that covers every combination of values for every pair of input parameters at least once, using far fewer tests than exhaustive testing.
Is pairwise testing the same as all-pairs testing?
Yes. All-pairs testing is another name for it. Both mean 2-way combinatorial testing, where every pair of parameter values appears in at least one test.
How many test cases does pairwise testing need?
At least the product of the two largest value counts. In our five-parameter checkout that minimum was 12, and our generator produced 13 tests instead of 216.
Can pairwise testing miss bugs?
Yes. It guarantees pairs, so bugs that need three or more specific values together can be missed. Use 3-way testing or targeted cases for those risks.
Which tools generate pairwise test cases?
Common options include Microsoft PICT, NIST ACTS and the Python package allpairspy. Many test management tools also include a pairwise generator.
Keep learning
Need help with testing, WordPress or SEO?
We help teams ship better software and grow their sites. Tell us what you are working on and we will suggest a clear next step.