Lecture 2 taught that testing can never find every bug. That leaves an awkward practical question that every real project has to answer: when do we stop? This lesson explains why a project needs a stopping rule decided in advance, why the two most popular rules are bad ones, and the three better categories of rule the professor taught.
Why We Need a Stopping Rule
Think of searching a large house for a lost earring. The first rooms you search turn up all sorts of lost things. After a while you find less and less, and eventually you are lifting the same cushions for the third time. You will never be certain the house holds nothing else, so at some point you must decide that the search is finished.
Testing behaves the same way. Plot the number of bugs found against the time spent testing: the curve rises steeply at first and then flattens out, and it never reaches the top, because there is no point at which you can prove that no bugs remain.
The bug curve flattens but never reaches the top, so “keep going until everything is found” never ends.
So every project needs a test completion criterion, also called a stopping criterion: a rule, decided in advance, that says “now testing is complete”. Deciding in advance matters. A rule invented at the end, when everyone is tired and the deadline is close, is not a criterion; it is an excuse.
Test completion criterion (stopping criterion): a rule, fixed before testing starts, that defines when a testing activity is finished.
Two Bad Rules
From the slides (Myers): the two stopping rules used most often in practice are also the two worst.
“Stop when the scheduled time runs out.” This measures the calendar, not the testing. Under this rule you could do nothing at all and still “finish” on time.
“Stop when all the test cases pass.” This one sounds responsible, but it rewards the wrong behaviour. If passing is what ends the work, testers are pushed, consciously or not, to write easy tests that do not find bugs.
The second rule contradicts the purpose of testing you met in Lecture 3 (Myers’ principles, see 3.1): a test is successful when it shows a bug, not when it passes. A stopping rule should therefore be about how well we have searched, not about how quiet the results have become.
The Three Categories of Completion Criteria
The professor taught three better ways to decide when to stop. Each relies on a different kind of evidence.
Category 1
Category 2
Category 3
Based on
Principles, theory & coverage
Statistics & past experience
The bug-finding trend of this project
Rule looks like
“Stop the module test when all MC/DC test cases have been run.”
“Stop the module test after finding 130 errors.”
“Stop at the greatest drop in the slope of the bug curve.”
Evidence needed
The code (to measure coverage)
Data from earlier, comparable projects
Bug counts collected now, at regular intervals
Strength
Precise and measurable at low level
Gives a concrete target for each test level
Needs no past data
Weakness
Usually not applicable to higher-order tests
Prior work must be done first
Must monitor the current project closely
Category 1: principles and coverage
You stop when the coverage you planned has been reached, using a test design method you have learned: statement or decision coverage, MC/DC (3.5), boundary values (5.3), and so on. This works well for low-level tests (unit and module tests), because there you can see the code and measure exactly what has been covered.
It is usually not applicable to higher-order tests (function, system, acceptance testing). At those levels there are only guidelines, not exact methods for measuring coverage. Think of a car factory: you can measure precisely that every bolt on one engine was torque-tested, but “has the whole car been driven in every way a customer could drive it?” has no exact percentage.
Category 2: statistics and past experience
From old projects you know roughly how many errors a program of a given size contains, and what share of them each test level finds. From these numbers you calculate how many errors each level should find before it stops. A bakery that has made the same cake a thousand times knows it should bake for 40 minutes; a new bakery does not.
Its weakness is exactly that: prior work must be done first. Without data from comparable projects you cannot calculate a trustworthy number. The full calculation is in 7.2.
Category 3: the bug-finding trend
You watch this project. You count the bugs found in each time period and stop when finding new bugs becomes much slower, at the greatest drop in the slope of the bug curve. This is like making popcorn: you do not need last year’s records, you listen to the pops, and when they suddenly thin out you take the pan off the heat. It needs no past data, but you must monitor the current project closely. The method is in 7.3.
Exam radar: know the three categories and one key point for each.
Category 1 (principles/coverage): good for low-level tests (unit/module), usually not for higher-order tests.
Category 2 (statistics/past experience): needs prior data to derive the stopping number.
Category 3 (bug-finding trend/slope): monitors the current project; stop at the greatest drop in the slope.
The categories are not rivals. A sensible project often combines them: coverage targets for its modules (Category 1), and a trend or statistical target for system testing (Category 3 or 2), so that one kind of evidence covers the blind spot of another.
Worked Examples
Exercise 1
What is a test completion criterion, and why must it be decided before testing starts?
Show solution ↓Hide solution ↑
Solution
A test completion criterion (stopping criterion) is a rule that says when a testing activity is finished.
It must be decided in advance because testing can never find every bug: the bug curve flattens but never reaches the top, so there is no natural end point. If the rule is chosen at the end, it will be chosen under deadline pressure and will simply describe whatever has already happened. A rule fixed in advance gives an objective target that the team works towards.
Exercise 2
A manager proposes: “We stop system testing when all our test cases pass.” Explain why this is a poor completion criterion.
Show solution ↓Hide solution ↑
Solution
It is a poor criterion because it rewards tests that do not find bugs. The purpose of testing is to show bugs; a test that finds a bug is the successful one (Myers, see 3.1). If passing is what ends the work, testers are pushed to write easy test cases that pass, and the weakest test suite finishes first.
The rule also says nothing about how thoroughly the software was searched: a tiny, shallow suite can “all pass” on day one. A good criterion measures the quality of the search (coverage, an error target, or the bug-finding trend), not the silence of the results.
Exercise 3
Name the three categories of test completion criteria. For each one, give an example rule and its main limitation.
Show solution ↓Hide solution ↑
Solution
Category
Example rule
Main limitation
1. Principles & coverage
“Stop the module test when all MC/DC test cases have been run.”
Usually not applicable to higher-order tests (system, acceptance), where only guidelines exist.
2. Statistics & past experience
“Stop the module test after finding 130 errors.”
Needs prior data from comparable projects; without it the number is not trustworthy.
3. Bug-finding trend
“Stop system testing at the greatest drop in the slope of the bug curve.”
Must monitor the current project closely; a falling rate shows diminishing returns, not the absence of bugs.
Exercise 4
Why is Category 1 a good fit for module testing but usually not for system testing?
Show solution ↓Hide solution ↑
Solution
Because coverage can only be measured exactly where you can see the code. In a module test the tester has the source, so it is possible to say precisely “every MC/DC pair has been exercised” or “every branch has been taken”.
At the system or acceptance level, the software is tested as a whole against its objectives, and there is no exact method for measuring how much of that whole has been covered, only guidelines. So a coverage-based stopping rule has nothing precise to count, and Category 1 is usually not applicable to higher-order tests.
Exercise 5
A company is testing the first product it has ever built, so it has no records from earlier projects. Which category of completion criterion can it use for system testing, and which can it not? Explain.
Show solution ↓Hide solution ↑
Solution
It cannot rely on Category 2, because Category 2 calculates its stopping numbers from statistics of past projects (error rates per 100 statements, detection rates per test level). With no history, those numbers would be guesses.
It can use Category 3, because the bug-finding trend needs no past data: the team records the cumulative number of bugs found at regular intervals during this project and stops at the greatest drop in the slope. (Category 1 does not suit system testing, as explained in Exercise 4, though it can still be used for the modules.)
Exercise 6
(Evaluate and recommend.) A new organization has no historical defect database and is deciding when to stop system testing. Recommend a completion approach and justify it.
Show solution ↓Hide solution ↑
Solution
How to write the answer: name the evidence that is available, pick the category that fits it, state that category’s limitation, recommend a second category to cover it, and reject the weak rules explicitly.
Category 2 is weak as the primary approach, because the organization has no reliable historical rates from comparable projects. Any predetermined error target would rest on an unsupported estimate and could be impossible to reach, or too low to give any confidence.
The organization should record the cumulative number of defects found at equal time intervals during this project and apply Category 3: calculate the slope for each interval and the change between neighbouring slopes, and stop at the greatest drop, which signals diminishing returns. Because a falling discovery rate can also be caused by weak tests, this trend should be combined with Category 1 evidence where suitable methods exist, for example required coverage criteria for the individual modules.
The team should not declare completion merely because the schedule has run out or because the existing tests have stopped finding errors. Recommendation: Category 3 for system testing, backed by Category 1 coverage at module level, so that the decision combines current trend evidence with evidence that the planned techniques were applied thoroughly.