3.4 The Coverage Ladder

Theory: The Coverage Ladder

Coverage measures how much of the code your tests actually used. There is not one kind of coverage but six, and the professor presented them as a ladder: each step asks for more than the one below and catches bugs the lower step lets slip. For every step the professor asked the same two questions: what does it ask? and what bug can still escape? Keep those two questions in mind; they are also the best way to answer exam questions about coverage.

The coverage ladder. Decision and condition coverage share a step because neither guarantees the other. MC/DC is the best balance.

All the examples use foo from 3.3, with each test written as (a, b, x):

public void foo(int a, int b, int x) {
    if (a > 1 && b == 0) {      // decision 1
        x = x / a;
    }
    if (a == 2 || x > 1) {      // decision 2
        x = x + 1;
    }
}

Step 1: Statement Coverage

What it asks: every statement runs at least once.

Example: one test is enough: (2, 0, 3). Decision 1 is true (x = 3 / 2 = 1), decision 2 is true because $a = 2$ , so both assignments run (path ace).

What can escape: we never see a decision being false. A bug that only shows up on a false branch is never found. The professor called this the weakest coverage: “baby stuff”.

Step 2: Decision Coverage (Branch Coverage)

What it asks: every decision is true at least once and false at least once. Because every branch is taken, every statement also runs, so decision coverage includes statement coverage.

Example:

(3, 0, 3): decision 1 true (x = 1), decision 2 false (path acd).
(2, 1, 1): decision 1 false, decision 2 true (path abe).

What can escape: a wrong condition inside a decision. If the programmer typed x < 1 instead of x > 1, both tests still pass: in (3, 0, 3), x = 1 makes both $1 > 1$ and $1 < 1$ false; in (2, 1, 1), $a = 2$ already makes decision 2 true. Decision coverage looks at each decision only as a whole.

Step 3: Condition Coverage

What it asks: every condition is true at least once and false at least once.

Example: (1, 1, 2) and (2, 0, 3).

Test	`a > 1`	`b == 0`	`a == 2`	`x > 1`	Decision 1	Decision 2
(1, 1, 2)	F	F	F	T	F	T
(2, 0, 3)	T	T	T	F	T	T

(In the second test x > 1 is checked after x = 3 / 2 = 1, so it is false.) Each of the four conditions is true once and false once.

What can escape: a whole branch! Decision 2 is true in both tests, so its false branch is never taken. Condition coverage does not even guarantee that every statement runs: for if (A && B), the tests (T, F) and (F, T) make each condition true and false once, yet the decision is false both times and the body of the if never executes.

Exam radar: decision coverage and condition coverage are independent. One does not guarantee the other. The decision coverage tests above miss a wrong condition; the condition coverage tests above miss a whole branch.

Step 4: Decision/Condition Coverage

What it asks: both of the above at the same time: every decision and every condition is true once and false once.

Example: (2, 0, 4) makes everything true (x = 4 / 2 = 2, then $a = 2$ and $2 > 1$ ). (1, 1, 1) makes everything false.

What can escape: conditions hidden by a short circuit. If || were typed instead of && in decision 1, both tests would still give exactly the same results: T || T is true like T && T, and F || F is false like F && F.

Short circuit and masking: the club guard

Think of a club guard with two rules: you must be over 18, and your name must be on the list. If you are 15, the guard does not bother to check the list. The answer is already no.

Java evaluates && and || the same way:

Expression	If the first condition is…	Result is already…	Second condition
`A && B`	false	false	never checked
`A \|\| B`	true	true	never checked

The first condition already decides the answer, so the computer skips the rest. This is a short circuit. We say A masks B: it hides B, so B can no longer change the result, and a bug in B can hide too. That is exactly what happens in Step 4: in (1, 1, 1), a > 1 is false and masks b == 0; in (2, 0, 4), a == 2 is true and masks x > 1.

Step 5: Multiple Condition Coverage

What it asks: every combination of true and false for the conditions of each decision.

Example: (2, 0, 4), (2, 1, 1), (1, 0, 2), (1, 1, 1).

Test	Decision 1: `a > 1`, `b == 0`	Decision 2: `a == 2`, `x > 1`
(2, 0, 4)	T, T	T, T
(2, 1, 1)	T, F	T, F
(1, 0, 2)	F, T	F, T
(1, 1, 1)	F, F	F, F

Each decision sees all four combinations TT, TF, FT, FF.

What can escape: nothing important, but the cost explodes. A decision with $n$ conditions has $2^n$ combinations: 10 conditions need $2^{10} = 1{,}024$ tests, and 20 conditions need more than a million. Too expensive.

Step 6: MC/DC

What it asks: each condition is shown to change the decision on its own. For foo, the three tests (1, 0, 0), (2, 1, 0) and (3, 0, 6) are enough. MC/DC is the best balance: nearly as strong as multiple condition coverage, with only about $n + 1$ tests per decision. It is important enough to get its own lesson (3.5) and a whole lecture of practice (Lecture 4).

The Whole Ladder on One Page

Coverage	What it asks	Tests (a, b, x)	What can still escape
Statement	Every statement runs at least once.	(2, 0, 3)	Mistakes on the false side of a decision.
Decision (branch)	Every decision is true once and false once.	(3, 0, 3), (2, 1, 1)	A wrong condition inside a decision, e.g. `x < 1` instead of `x > 1`.
Condition	Every condition is true once and false once.	(1, 1, 2), (2, 0, 3)	A whole branch: here decision 2 is never false.
Decision/Condition	Both of the above at the same time.	(2, 0, 4), (1, 1, 1)	Conditions hidden by a short circuit, e.g. `\|\|` typed instead of `&&`.
Multiple condition	Every combination of T/F in each decision.	(2, 0, 4), (2, 1, 1), (1, 0, 2), (1, 1, 1)	Nothing important, but it needs $2^n$ tests.
MC/DC	Each condition shown to change the decision on its own.	(1, 0, 0), (2, 1, 0), (3, 0, 6)	Best balance: strong, with about $n + 1$ tests per decision.

A Procedure for Ordinary Coverage Questions

Exam questions on the lower steps of the ladder follow a repeatable procedure:

Draw (or picture) the control-flow graph.
List the executable statements, the decision outcomes and the atomic conditions.
Translate the requested criterion literally: statements, decision outcomes, condition outcomes, or combinations.
Choose the smallest clear set of inputs that reaches the required items.
Trace every test and mark what it covers.
State which criterion is satisfied, and which stronger criteria are not proved.

Example: the decision A AND B

The program runs x = 1 if A AND B is true and x = 2 otherwise, then prints x. Three candidate tests:

Test	A	B	Decision	Branch executed
T1	T	T	T	`x = 1`
T2	F	T	F	`x = 2`
T3	T	F	F	`x = 2`

Statement coverage: T1 plus either T2 or T3 runs both assignments and the print.
Decision coverage: T1 gives true, T2 or T3 gives false; so the same two tests suffice.
Condition coverage: A must be T and F, and B must be T and F. T2 and T3 alone already do that, but both make the decision false, so x = 1 never runs. Using T1, T2 and T3 gives a clear condition-coverage set, {TT, FT, TF}, that also achieves decision and statement coverage.

None of this proves that every path of a larger program was tested, or that the software is correct.

How to write the answer: for coverage questions, always show the trace: a small table with each test, the value of every condition, and the outcome of every decision. Then state the conclusion explicitly (“all four conditions are true once and false once, so condition coverage is achieved; decision 2 is never false, so decision coverage is not”). A correct-looking test set with no trace is not proof of coverage.

Worked Examples

Exercise 1

Explain short circuit and masking using if (a > 1 && b == 0) when $a = 1$ .

Show solution ↓

Solution

With $a = 1$ , the condition a > 1 is false. With &&, one false condition makes the whole decision false, so Java does not evaluate b == 0 at all. That is a short circuit.

We say a > 1 masks b == 0: it hides it, so the value of b == 0 cannot change the result. If b == 0 contained a bug (say, b == 1 was typed), this test could not reveal it, because that condition is never even checked.

Exercise 2

Why is decision coverage not enough? Illustrate with the decision coverage tests for foo.

Show solution ↓

Solution

Decision coverage checks each decision only as a whole: true once, false once. A wrong condition inside the decision can still produce the expected decisions, so the bug is not found.

For foo, the tests (3, 0, 3) and (2, 1, 1) give decision coverage. Suppose the programmer typed x < 1 instead of x > 1 in decision 2:

(3, 0, 3): after x = 3 / 3 = 1, both $1 > 1$ and $1 < 1$ are false, and $a = 2$ is false, so decision 2 is false either way.
(2, 1, 1): $a = 2$ is true, so decision 2 is true either way; the x condition is masked.

Both tests pass on the buggy code, so decision coverage did not detect the wrong condition.

Exercise 3

For the program “if A AND B then x = 1 else x = 2; print x”, select test sets for statement coverage, decision coverage and condition coverage. Trace each one.

Show solution ↓

Solution

The control flow has one decision with two outcomes and two conditions, A and B.

Test	A	B	Decision	Executed
T1	T	T	T	`x = 1`, print
T2	F	T	F	`x = 2`, print
T3	T	F	F	`x = 2`, print

Statement coverage: {T1, T2} (or {T1, T3}) runs x = 1, x = 2 and the print.
Decision coverage: {T1, T2}: the decision is T in T1 and F in T2.
Condition coverage: A is T in T1/T3 and F in T2; B is T in T1/T2 and F in T3. So {TT, FT, TF} gives condition coverage, and since the decision is both T and F it also gives decision and statement coverage.

Check: every assignment executed ✓, decision T and F ✓, each condition T and F ✓.

Exercise 4

A tester runs only (1, 1, 2) and (2, 0, 3) on foo. Which of statement, decision and condition coverage are achieved?

Show solution ↓

Solution

Trace both tests (integer division):

Test	`a > 1`	`b == 0`	`a == 2`	`x > 1`	D1	D2	Path
(1, 1, 2)	F	F	F	T	F	T	abe
(2, 0, 3)	T	T	T	F (x = 1)	T	T	ace

Statement coverage: yes. Path ace runs both assignments.
Decision coverage: no. Decision 1 is F and T, but decision 2 is true in both tests; its false branch is never taken.
Condition coverage: yes. Each of the four conditions is true once and false once.

This is the standard example showing that condition coverage does not imply decision coverage.

Exercise 5

Show that the decision/condition coverage tests (2, 0, 4) and (1, 1, 1) cannot tell whether decision 1 was typed as a > 1 && b == 0 or as a > 1 || b == 0. What is the underlying reason?

Show solution ↓

Solution

(2, 0, 4): both conditions of decision 1 are true. T && T = T and T || T = T, so x = 4 / 2 = 2 either way; decision 2 is then true and x = 3.
(1, 1, 1): both conditions are false. F && F = F and F || F = F, so x is unchanged; decision 2 is false and x = 1.

Both versions give identical results, so the || bug escapes.

The reason is that the conditions only ever move together (all true, or all false). No test shows a single condition changing the decision on its own, and with short circuit evaluation the first condition masks the second. Showing each condition’s independent effect is exactly what MC/DC adds (3.5).

Exercise 6

A decision has 10 conditions. How many tests does multiple condition coverage need for it? Why does the professor call MC/DC the best balance?

Show solution ↓

Solution

Multiple condition coverage needs every combination of true and false:

2^{10} = 1{,}024 \text{ tests}

MC/DC needs only about $n + 1 = 11$ tests for the same decision, yet still shows that every condition can change the decision on its own, which catches the short circuit and masking problems of the cheaper coverages. It is much stronger than decision/condition coverage and much cheaper than multiple condition coverage: the best balance.

Exercise 7

Rank these from weakest to strongest and mark the pair that cannot be ranked against each other: MC/DC, statement, multiple condition, decision/condition, decision, condition.

Show solution ↓

Solution

Statement → {decision, condition} → decision/condition → MC/DC → multiple condition.

Decision and condition coverage share a step: they are independent, so neither guarantees the other (decision coverage can miss a wrong condition; condition coverage can miss a whole branch). Decision/condition coverage combines them, MC/DC adds the independent effect of every condition, and multiple condition coverage tries every combination at a cost of $2^n$ .