Exhaustive Testing &
Pairwise Testing Calculator
See why testing every combination is impossible, then build a pairwise set ordered by coverage for the time you have.
Picture an online shop's checkout. Customers can use five different browsers, three screen sizes, five ways to pay, four delivery options and three types of account. Add three kinds of basket and three discount code situations, and testing every possible combination would take 8,100 tests. At 10 minutes per test, that's about 56 days of testing, non-stop. Most testers have days, not months, and sometimes only a few hours.
This calculator shows how many tests exhaustive testing would need, then builds a much smaller set using pairwise testing. It puts the tests in order of coverage, so if you only have 3 hours, you know which tests to run first.
What Is Exhaustive Testing?
Exhaustive testing means testing every possible combination of inputs and conditions. The catch is that combinations multiply. Two settings with 3 options each give 9 combinations. Add a third setting with 3 options and it's 27. Add a few more and the numbers quickly run into thousands, then millions.
For almost all real software, exhaustive testing can't be done. That's why exhaustive testing is impossible. Testers don't try to test everything. They choose which tests to run, and pairwise testing is one way to make that choice.
What Is Pairwise Testing?
Leaving out most combinations can seem risky, and it's a fair concern. The reason it works is that most defects caused by settings interacting involve only one or two settings. A payment button that breaks in Safari on a mobile screen depends on two settings, not seven.
Pairwise testing, also called all-pairs testing, makes sure every pair of options is tested together at least once. Chrome gets tested with PayPal, PayPal with next-day delivery, next-day delivery with a guest account and so on. For the checkout example, that takes the number of tests from 8,100 down to around 25 to 30.
Who Is It For?
- Students and anyone new to software testing can see why testers can't test everything and how they decide what to test instead.
- Working testers get a pairwise test set for configuration testing, ordered by coverage and ready to download as a CSV file.
- Test leads and managers can use the numbers to explain why a test plan covers a chosen set of tests rather than every combination.
How To Use The Calculator
- Choose one of the examples, or start from scratch.
- Add each setting and its options, separated by commas. For example, a setting called Browser could have the options Chrome, Safari, Firefox.
- Enter how much time you have and roughly how long one test takes.
- Compare the results: the exhaustive testing total, the pairwise testing total and the share of pairs you can cover in your time.
- Work down the test list from the top. The line across the table shows where your time runs out.
- The page shows the top 10 tests. Download the CSV file for the full list to use in a spreadsheet or test management tool.
Build Your Test Plan
Compare exhaustive testing with pairwise testing, then see which tests to run in the time you have.
Things to know
- "Best coverage" means each test covers as many new pairs of options as possible. Every pair is treated as equally important, so if some combinations matter more, such as the browser most of your users have, make sure they're tested early.
- The calculator doesn't know which combinations are impossible, such as Apple Pay in a browser that doesn't support it. If one comes up, swap a value in that test and note why.
- Most defects caused by settings interacting involve just one or two settings, which is why pairwise testing works. It won't catch every defect that needs three or more settings to line up.
Understanding The Results
The exhaustive testing figure is the number of tests needed to try every combination, along with how long they'd take. The pairwise testing figure is the number needed to cover every pair at least once. The time available figure shows how many of those tests fit in your time and what percentage of pairs they cover.
In the test list, new pairs shows how many pairs each test covers for the first time. The first few tests cover the most, and later tests pick up the last few pairs. Pairs covered is the running total. That's why a short test session still covers a large share of pairs when the tests are run in order.
Key Points
- Exhaustive testing means testing every possible combination of inputs and conditions.
- The number of combinations multiplies with every setting added, so exhaustive testing is impossible for almost all real software.
- Pairwise testing covers every pair of options at least once, using a fraction of the tests.
- Running tests in order of coverage means the most ground is covered first when time is short.
Frequently Asked Questions
For a small example like the checkout, they could. But real software has far more settings and inputs, and the numbers grow faster than any automated tests can keep up with. A username field that accepts any 8 letters has over 208 billion possible values on its own. Automated tests also have to be written and maintained, so every test still has a cost.
No. It won't catch a defect that only appears when three or more settings line up in a particular way. That's why testers combine it with other techniques and spend more time on the areas where a defect would cause the most harm.
Pairwise testing is the most common type of combinatorial testing. Combinatorial testing can also cover every set of three settings together, called three-way testing, which needs more tests but catches more interaction defects.
The calculator builds the list one test at a time, choosing whichever test covers the most new pairs. Changing a setting changes the pairs, so the list changes too. The same settings always give the same list.
No. Software testing is about working out how software should behave and checking that it does. Coding helps in some testing roles, but you don't need it to get started.