7.2 Category 2: Past Statistics (the 10,000-Statement Example)

Theory: Category 2: Past Statistics (the 10,000-Statement Example)

Category 2 turns experience into numbers. If old projects tell you roughly how many errors a program contains and which test level tends to find them, you can calculate in advance how many errors each level should find before it stops. The professor solved the classic example from Myers’ book on the board, and it is a likely exam question, so this lesson goes through it slowly and asks “why?” at every step.

From the slides: the problem. You are testing a program of 10,000 statements. After code inspection, about 5 errors per 100 statements are expected to remain. The goal is to detect 98% of the coding and logic-design (CLD) errors and 95% of the design errors. Calculate the stopping criterion for the module, function and system tests.

Two Kinds of Errors

An industry study in the slides found that errors come in two kinds:

• CLD errors (coding and logic design), about 40% of all errors: small mistakes inside the code and its small-scale logic, such as a wrong condition or an off-by-one loop. Low-level tests catch most of them.
• Design errors, about 60%: bigger mistakes in how the system is put together, such as two components that make incompatible assumptions. Higher-level tests catch most of them.

A spelling mistake in one sentence of a book is like a CLD error: a proofreader working page by page will find it. A plot hole that only appears when chapter 2 is compared with chapter 19 is like a design error: you only see it when you read the whole book.

Exam radar: the 40% / 60% split is in the slides, but the professor admitted it was “tricky” because it was not said aloud in class. If an exam question does not give the split, state your assumption: “Assume 40% CLD errors and 60% design errors (industry study).”

The Calculation, Step by Step

Step 1: How many errors are there in total?

If every 100 statements contain about 5 errors, then 10,000 statements contain 100 groups of 100:

$$\frac{10{,}000 \times 5}{100} = 500 \text{ errors}$$

Step 2: How many of each kind?

Apply the 40/60 split to the total:

$$\text{CLD} = 0.40 \times 500 = 200 \qquad \text{Design} = 0.60 \times 500 = 300$$

Step 3: What are the goals?

We do not aim for 100%. Finding every error is the spiral of unattainable perfection from 2.5: the last few errors would cost more than everything before them, and you could never prove you had found them anyway. So the goals are a little below 100%:

$$\text{CLD goal} = 0.98 \times 200 = 196 \qquad \text{Design goal} = 0.95 \times 300 = 285$$

Step 4: Which test level finds what?

Past projects give the share of each kind of error that each test level detects:

Test level	CLD errors found	Design errors found
Module test	65%	0%
Function test	30%	60%
System test	3%	35%
Total	98%	95%

Two things to notice:

• Why does the module test find 0% of design errors? Because a module test looks at one small unit on its own. It is too low-level to see a mistake in how the units fit together, just as the page-by-page proofreader never notices the plot hole.
• The columns add up to the goals. $65 + 30 + 3 = 98$ and $0 + 60 + 35 = 95$. The table is simply the two goals shared out between the three levels.

Step 5: The stopping number for each level

Multiply each detection rate by the total of its own kind of error, then add the two parts:

$$\text{Module} = 0.65 \times 200 + 0 \times 300 = 130$$ $$\text{Function} = 0.30 \times 200 + 0.60 \times 300 = 60 + 180 = 240$$ $$\text{System} = 0.03 \times 200 + 0.35 \times 300 = 6 + 105 = 111$$

Check: $130 + 240 + 111 = 481$ and $196 + 285 = 481$ ✓. The three levels together reach exactly both goals.

In practice this means: stop the module test when 130 errors have been found, the function test at 240, and the system test at 111.

Diagram loads as you scroll.

The Trap: “65% of What?”

The professor stopped the class at Step 5 and asked: “65% of what?”

Common mistake: taking the percentage of the wrong number. The 65% is of the total CLD errors (200), not of all 500 errors ($0.65 \times 500 = 325$, which mixes in design errors a module test cannot see) and not of the goal 196 ($0.65 \times 196 = 127.4$, which applies the 98% twice). Likewise every design percentage is of the total design errors (300).

The Procedure to Reproduce in the Exam

1. Total errors = program size × error rate.
2. Split the total into CLD and design errors using the given percentages (or the stated 40/60 assumption).
3. Goals = goal percentage × each kind’s total.
4. For each test level, multiply each detection rate by its own kind’s total, then add the two contributions.
5. Check that the column percentages add up to the goals and that the stage targets add up to the sum of the goals.
6. State that the targets concern errors found and repaired, not test cases executed. “Stop the module test after 130” means 130 errors found and fixed, not 130 tests run.

How to write the answer: write the assumption first, show every multiplication, and finish with the check $\text{module} + \text{function} + \text{system} = \text{CLD goal} + \text{design goal}$. A marker who sees the check knows you understood where each number came from.

---

Worked Examples

Exercise 1

In the 10,000-statement example, why are the goals set at 98% and 95% instead of 100%? And why does the module test find 0% of the design errors?

Show solution ↓Hide solution ↑

Solution

The goals are below 100% because 100% is unattainable. Testing can never prove that every error has been found, and the last few errors cost far more to find than all the earlier ones (the spiral of unattainable perfection, 2.5). A goal of 100% would be a stopping rule that never stops.

The module test finds 0% of design errors because it is too low-level to see them. A module test examines one small unit on its own. A design error is a mistake in how the units are put together, and it only shows when units interact, which happens in function and system testing.

Exercise 2

A student computes the module-test target as $0.65 \times 500 = 325$. Another computes $0.65 \times 196 = 127.4$. Explain what each did wrong and give the correct target.

Show solution ↓Hide solution ↑

Solution

The 65% detection rate is a percentage of the total CLD errors, so it must multiply 200.

• $0.65 \times 500$ applies the CLD rate to all errors, including the 300 design errors, which a module test cannot detect at all.
• $0.65 \times 196$ applies it to the goal, so the 98% is counted twice (the 65% is already one share of the 98%).

Correct: $0.65 \times 200 + 0 \times 300 =$ 130 errors.

Exercise 3

A program has 30,000 statements and about 5 errors per 100 statements are expected to remain. Assume 40% CLD and 60% design errors. The goal is 98% of CLD and 95% of design errors, and the detection rates are the same as in the lesson (module 65%/0%, function 30%/60%, system 3%/35%). Calculate the stopping criterion for each test level.

Show solution ↓Hide solution ↑

Solution

Assumption: CLD = 40%, design = 60% of all errors (industry study).

Step 1, total errors:

$$\frac{30{,}000 \times 5}{100} = 1{,}500$$

Step 2, split: CLD $= 0.40 \times 1{,}500 = 600$; design $= 0.60 \times 1{,}500 = 900$.

Step 3, goals: CLD goal $= 0.98 \times 600 = 588$; design goal $= 0.95 \times 900 = 855$.

Step 4, each level (rates times totals 600 and 900):

Test level	Calculation	Criterion
Module	$0.65 \times 600 + 0 \times 900$	390
Function	$0.30 \times 600 + 0.60 \times 900 = 180 + 540$	720
System	$0.03 \times 600 + 0.35 \times 900 = 18 + 315$	333

Check: $390 + 720 + 333 = 1{,}443 = 588 + 855$ ✓

Exercise 4

Repeat the calculation for a program of 50,000 statements with 4 errors per 100 statements, using the same assumptions, goals and detection rates.

Show solution ↓Hide solution ↑

Solution

Assumption: 40% CLD, 60% design.

1. Total: $50{,}000 \times 4 / 100 = 2{,}000$ errors.
2. CLD $= 0.40 \times 2{,}000 = 800$; design $= 0.60 \times 2{,}000 = 1{,}200$.
3. Goals: CLD $0.98 \times 800 = 784$; design $0.95 \times 1{,}200 = 1{,}140$.
4. Levels:
   • Module: $0.65 \times 800 + 0 \times 1{,}200 =$ 520
   • Function: $0.30 \times 800 + 0.60 \times 1{,}200 = 240 + 720 =$ 960
   • System: $0.03 \times 800 + 0.35 \times 1{,}200 = 24 + 420 =$ 444

Check: $520 + 960 + 444 = 1{,}924 = 784 + 1{,}140$ ✓

Exercise 5

The team in the lesson’s example finishes the module test after running 130 test cases, of which 40 found errors. Has the module-test completion criterion been met?

Show solution ↓Hide solution ↑

Solution

No. The Category 2 target is about errors found and repaired, not about test cases executed. The criterion “stop the module test at 130” means 130 errors detected (and fixed). Running 130 test cases that found only 40 errors means the module test has reached $40 / 130 \approx 31\%$ of its target, so testing should continue (or the team should ask why its tests are finding so few errors).

Exercise 6

Why can a brand-new company not use this calculation reliably, and which category should it use instead?

Show solution ↓Hide solution ↑

Solution

Because every number in the calculation comes from past projects: the error rate per 100 statements, the 40/60 split and the detection rate of each test level. A brand-new company has no comparable history, so its targets would be guesses and could be unreachable or far too low.

It should use Category 3 (the bug-finding trend, 7.3), which needs no past data: it monitors the current project’s bug curve and stops at the greatest drop in the slope, ideally combined with Category 1 coverage targets for its modules.