TEST DESIGN
Combinatorial Testing.
Black-Box The family of techniques that systematically reduce the combinatorial explosion of multi-parameter test spaces by covering all interactions up to a chosen strength — with Pairwise Testing (2-way) as the most common strength in practice.
What it is
Combinatorial Testing is the overarching discipline that covers all strength-based interaction testing: 1-way (each value at least once), 2-way (pairwise — every pair of values), 3-way (every triple), and so on. In practice 'combinatorial testing' almost always means t-way Covering Array testing, where t is the interaction strength. Pairwise Testing is the special case where t=2 and is by far the most used strength because NIST research found that most software defects involve only 1 or 2 variables (1-way and 2-way defects account for approximately 97% of defects in the studied systems). Combinatorial testing is the theoretical foundation that pairwise, decision tables, and all-pairs tools are built on.
When to use it
When to use
- When you need to reason about WHAT interaction strength is appropriate for a test suite (2-way, 3-way, n-way) — CT gives you the vocabulary and the maths
- High-reliability or safety-critical systems where 3-way or 4-way interaction coverage is mandated
- When you have defect-history data suggesting that 3+ variable interactions are causing failures in a specific subsystem
- Tool evaluation: when comparing pairwise generators, CT metrics (covering array number, CAN) tell you which is more efficient
- Academic or process documentation contexts where the relationship between exhaustive and strength-n testing needs to be stated precisely
When NOT to use
- As a substitute for Pairwise Testing when you just need a practical 2-way test set — use the Pairwise Test Case Generator directly
- When the test system doesn't have well-defined independent parameters to combine — CT requires discrete, independent input parameters
How it works
The key metric is the Covering Array Number (CAN): the minimum number of test cases needed to achieve t-way coverage for n parameters with v values each. The worked example below shows how test count grows with strength and why 2-way (pairwise) hits the practical sweet spot. For 3 parameters × 3 values (27 exhaustive), pairwise produces 9 tests; 3-way requires 27 — identical to exhaustive for this small case. The payoff from 3-way coverage only becomes apparent with more parameters.
3 parameters × 3 values each (Browser × Auth × Remember-me) — comparing exhaustive vs. t-way
| Coverage strength | Guarantee | Test count | Reduction vs. exhaustive | Result |
|---|---|---|---|---|
| Exhaustive (all combos) | Every combination of all 3 params | 27 | — | Reject |
| 3-way (CT at t=3) | Every triple of values from all 3 params | 27 | 0% (same as exhaustive for 3 params) | Reject |
| 2-way / Pairwise (CT at t=2) | Every pair of values across any 2 params | 9 | 67% reduction ✓ | Accept |
| 1-way (each value once) | Each parameter value appears at least once | 3 | 89% reduction (insufficient coverage) | Reject |
⚠ For 10 parameters × 3 values each: exhaustive = 59,049 tests; pairwise ≈ 15 tests; 3-way ≈ 54 tests. The payoff from pairwise increases dramatically with parameter count.
Step by step
Determine the required interaction strength (t)
Default to t=2 (pairwise) unless defect history or risk analysis points to known 3-way interactions. Higher strength means more tests — justify t=3 with evidence, not instinct.
List parameters and discretise continuous ones
Identify all independent parameters. Convert continuous ranges to discrete partitions using Equivalence Partitioning before applying combinatorial testing.
Calculate expected test count and compare to exhaustive
For a given t, n, and v, estimate the covering array size. If the estimate is close to exhaustive (as it can be for small parameter counts), reconsider whether exhaustive is feasible — combinatorial testing pays off most for large parameter spaces.
Generate the covering array
Use a tool (the Pairwise Test Case Generator for t=2). Verify the output covers the required interaction strength by spot-checking: pick any two parameter columns and confirm every pair of their values appears at least once.
Add expected results and execute
Annotate each test case with its expected result. Execute and record. A failure in a t-way test identifies an interaction among exactly t parameter values as the failure region.
Pitfalls & what it misses
Conflating Combinatorial Testing with Pairwise Testing
Pairwise is one specific strength (t=2) of combinatorial testing. If you need 3-way coverage, a pairwise generator is insufficient. The distinction matters in regulated contexts where a specific t-way strength may be mandated.
Applying CT to dependent parameters
Combinatorial testing assumes parameters are mutually independent. Feeding correlated parameters into a CT tool produces invalid test cases. Pre-process or model correlated parameters as a single combined parameter.
Underestimating how fast test counts grow with t
For 6 parameters × 3 values, pairwise (t=2) ≈ 15 tests; 3-way (t=3) ≈ 30 tests; 4-way ≈ 81 tests. Higher-order CT has rapidly diminishing returns for typical commercial software and should be reserved for safety-critical subsystems.
Paired utility
// Related resources