Theory: The Seven Principles of Testing
These seven principles come from ISTQB (the International Software Testing Qualifications Board), and they are classic. The professor asked the class many questions while teaching them, so this lesson keeps the reasoning the professor wanted, not just the headlines.
| # | Principle | Picture to remember |
|---|
| 1 | Testing shows the presence of defects | A metal detector on a beach |
| 2 | Exhaustive testing is impossible | Five inputs are easy; a hundred are not |
| 3 | Early testing | The ×10 cost ladder from 1.3 |
| 4 | Defects cluster together | One cockroach means more behind the fridge |
| 5 | The pesticide paradox | The same spray stops working |
| 6 | Testing is context-dependent | A heart monitor is not a mobile game |
| 7 | The absence-of-errors fallacy | Silence from the metal detector proves nothing |
Principle 1: Testing Shows the Presence of Defects
Imagine sweeping a metal detector across a beach. If it beeps, you know there is metal underneath. If it stays silent, you cannot say there is no metal, only that this sweep did not find any. Testing works like that: it can show that bugs exist, but it can never prove that the software is correct.
A principle about mindset
The professor spent time on this, because it is really about mindset.
- If you test to prove the software works, you will unconsciously write gentle tests, choosing inputs the software is likely to handle.
- If you test to break it, you will write nasty tests and expose many more defects.
The same idea explains why developers should not test their own code: they know how it is meant to work and will subconsciously steer around the ways it could fail. There is one exception, unit testing, which the owner of the code does best because that person understands the unit (Lecture 8).
Principle 2: Exhaustive Testing Is Impossible
Suppose a function takes five inputs. If each is a yes/no value, there are
25=32
combinations, which is easy. But if each is a whole number, or a piece of text, the number of combinations explodes. Safety-critical aviation software may have a hundred parameters. On top of the inputs there are the paths the program can take from start to finish, and their number is even worse. In Lecture 2 (2.1) you will calculate how long it would take to test a simple calculator exhaustively.
The trick question
The professor asked this, and waited a long time for the right answer: since exhaustive testing is impossible, how can you, as a tester, eliminate the risk that comes from not testing everything?
Exam radar: you cannot eliminate the risk. You can only reduce it, by prioritising what matters most and by using techniques such as equivalence partitioning and boundary value analysis. No matter how clever you are, some risk always remains. That is why testing is a risk-based exercise.
The question is a trap because it smuggles in a false assumption, the word eliminate. The easy way to go wrong is to answer a different question, for example by explaining at length why testing everything is impossible.
Common mistake: writing something that is true in general but does not answer the question that was asked. The professor warned that such an answer earns no credit. Here, explaining why exhaustive testing is impossible does not answer “how do you eliminate the risk?”; the answer must start with “you cannot”.
Principle 3: Early Testing
This is the cost rule from 1.3 in one sentence: the sooner you find a bug, the cheaper it is, because each stage it travels through multiplies the cost by roughly ten.
Principle 4: Defects Cluster Together
When you find one cockroach in the kitchen, you do not squash it and assume you are finished, because one usually means more behind the fridge. Bugs behave the same way: they gather in the same places.
The professor linked this to Murphy’s Law: when something goes wrong, more things go wrong. Picture a developer having a terrible day, tired and distracted, who writes several bugs into the same module. So when bugs arrive more densely than usual in one area, look harder there. The slides also express this as the Pareto idea: roughly 80% of the bugs live in about 20% of the code.
Principle 5: The Pesticide Paradox
A farmer sprays the same pesticide every week. At first it kills a lot of insects, but the survivors are the ones that resist it, and they multiply, until the spray does almost nothing.
A test suite behaves the same way. The first time you run it, it catches the bugs it was designed for. You fix those, and running the same tests again finds nothing new, because the bugs those tests can reach are gone. New bugs hide in places the old tests never look. So you must keep writing new and different test cases.
Principle 6: Testing Is Context-Dependent
There is no one-size-fits-all way to test. Testing the software inside a heart monitor is nothing like testing a mobile game. Each project has its own risks, its own users and its own consequences of failure, so the design of the tests must fit the context.
Principle 7: The Absence-of-Errors Fallacy
This is the principle that confused the class, so read it carefully. When a student said that software with no bugs is still useless if nobody needs it, the professor called it a good thought, but placed in the wrong spot. There are two readings:
- The professor’s reading. The phrase says “absence of errors” followed by “fallacy”, meaning a false belief. If you do not see any bugs, that does not mean there are none. It is the mirror image of Principle 1, back to the metal detector: silence proves nothing, especially since exhaustive testing is impossible.
- The ISTQB textbook reading. Finding and fixing every bug you found is pointless if the system is unusable and does not meet the users’ needs.
Both readings are true and both are worth knowing.
How to write the answer: for a question on Principle 7, lead with the professor’s reading (no visible bugs does not mean no bugs), then add the textbook reading (a bug-free system is still useless if it does not meet the users’ needs). That way you are covered either way.
Worked Examples
Exercise 1
Since exhaustive testing is impossible, how can a tester eliminate the risk that remains?
Show solution ↓Hide solution ↑
Solution
The tester cannot eliminate it. The risk can only be reduced, by prioritising what matters most and using techniques such as equivalence partitioning and boundary value analysis. Some risk always remains, which is why testing is described as a risk-based exercise.
Note the shape of the answer: the question hides a false assumption (“eliminate”), so the first words reject it. Explaining why exhaustive testing is impossible, however correct, would not answer the question and would earn no credit.
Exercise 2
Which principle explains why rerunning the same test suite for years gives diminishing returns?
Show solution ↓Hide solution ↑
Solution
The pesticide paradox (Principle 5). The same tests catch only the bugs they were designed to catch. Once those are fixed, rerunning the suite finds nothing new, while new bugs hide where the old tests never look. So the suite must be renewed with new and different test cases, just as a farmer must change a pesticide the surviving insects have become resistant to.
Exercise 3
A lot of bugs are found in the payment module. What should the testers do, and which principle tells you so?
Show solution ↓Hide solution ↑
Solution
They should spend more testing effort on the payment module, with new and different test cases. This is defect clustering (Principle 4): bugs gather in the same places (one cockroach means more behind the fridge), so a module that has already shown many bugs probably hides more.
Payment is also a high-impact area, which fits the risk-based thinking of Principle 2: where failure costs most, testing effort should be greatest.
Exercise 4
A function takes five yes/no inputs. How many input combinations are there? How many if it took ten? What does the growth tell you about real software?
Show solution ↓Hide solution ↑
Solution
Each yes/no input doubles the number of combinations:
25=32210=1,024Doubling the inputs from 5 to 10 multiplied the combinations by 25=32. Real software has inputs that are whole numbers or text (far more than two values each), safety-critical systems may have a hundred parameters, and the number of paths through the code is worse still. So exhaustive testing is impossible (Principle 2), and testers must sample wisely and prioritise by risk.
Exercise 5
(Discussion) Explain why developers should not normally test their own code, and name the exception.
Show solution ↓Hide solution ↑
Solution
Principle. Testing shows the presence of defects, not their absence (Principle 1), and the results depend on mindset: a tester trying to prove the software works writes gentle tests, while a tester trying to break it writes nasty tests and finds more defects.
What it means here. Developers know how their code is meant to work, so they subconsciously choose inputs that follow that intention and steer around the ways it could fail. An independent tester has no such bias. The exception is unit testing: the owner of the code does it best, because that person understands the unit in detail.
Exercise 6
(Discussion) Explain the absence-of-errors fallacy.
Show solution ↓Hide solution ↑
Solution
The professor’s reading. “Absence of errors” followed by “fallacy” names a false belief: if you do not see any bugs, that does not mean there are none. It mirrors Principle 1. Like a silent metal detector on a beach, a test run that finds nothing proves nothing, especially because exhaustive testing is impossible, so some untested input could still fail.
The ISTQB textbook reading. Even if every bug found has been fixed, the effort is pointless if the system is unusable or does not meet the users’ needs. A defect-free product that solves the wrong problem is still a failure, which is why validation matters (see 1.2).
Exercise 7
A company has a well-proven test process for its mobile games and proposes to reuse it, unchanged, for the software inside a heart monitor. Which principle does this break, and why?
Show solution ↓Hide solution ↑
Solution
It breaks Principle 6: testing is context-dependent. There is no one-size-fits-all way to test. A game failure means a restart; a heart monitor failure can kill someone, so it is safety-critical (see 1.4) and needs far more rigorous, risk-driven testing. Its users, risks and consequences of failure are completely different, so the tests must be designed for that context.
Exercise 8
Match each situation to the principle it illustrates:
- (a) A requirements review finds a contradiction before any code is written.
- (b) After 500 passing tests, a manager announces the release is bug-free.
- (c) 80% of last year’s defects came from 2 of the 10 modules.
Show solution ↓Hide solution ↑
Solution
- (a) Principle 3, early testing: the defect is caught at the cheapest point on the ×10 cost ladder.
- (b) Principle 1 (testing shows presence, not absence) and Principle 7 in the professor’s reading: not seeing bugs does not mean there are none. The manager’s claim is the fallacy.
- (c) Principle 4, defect clustering, in its Pareto form: about 80% of bugs live in about 20% of the code, so those two modules deserve extra testing.