Theory: Unit Testing, Failures vs. Errors & Independence
Lecture 8 moves from when to stop testing to how testing is organised at the lowest level. This lesson defines the unit, explains the one distinction the professor insisted you must know (failure vs. error), and gives the three rules that keep unit tests trustworthy.
What Is a Unit?
Before a car is assembled, each part is checked on its own: the brakes on a brake rig, the battery on a battery tester. Only parts that pass are bolted together. Software is built the same way.
A unit is the smallest piece of software that can be tested on its own, for example one method or one class. Unit testing means testing these small pieces one by one. The same idea applies to hardware too: a single chip or circuit board can be unit tested before it goes into a device.
Unit tests are usually written by the developer, because the person who wrote the unit knows it best. This is one of the exceptions to Myers’ principle that programmers should avoid testing their own programs (see 3.1): at the unit level, the owner’s detailed knowledge is worth more than an outsider’s independence.
Failure or Error?
Unit-testing tools such as JUnit always report two different kinds of problem, and they mean very different things.
Think of a driving test. If the candidate drives through a red light, the test ran properly and showed a problem with the driver: that is a failure. If the examiner’s car will not start, the test could not be carried out at all, and it tells you nothing about the driver: that is an error.
| Failure | Error |
|---|
| What happened | The test ran normally, but the answer was wrong (actual ≠ expected). | The test could not finish: something crashed unexpectedly. |
| Typical sign | assertEquals(150, balance) reports expected 150 but was 140. | A NullPointerException, a missing file, a broken set-up method. |
| Cause | The code under test does the wrong thing. | Something not caused by the test case’s expected result. |
| Meaning | The test case found a bug. | The test itself could not run properly; investigate the test or its environment. |
Exam radar: the professor said, know the difference. A failure = the test ran and the actual result differed from the expected result (the test found a bug). An error = the test could not run properly because something crashed for a reason not caused by the test case (for example a NullPointerException).
Three Rules for Unit Tests
- Each unit test must run independently of all other unit tests.
- Therefore the tests can run in any order.
- A test must not depend on a side effect of a previous test.
What is a side effect?
When a test runs a unit, the unit might change something that other code can see: a shared variable x, a record in a file or database, a message sent to another unit. Such a change is a side effect.
Imagine a shared kitchen. If the first cook leaves the oven at 250°C and the second cook’s recipe silently assumes “the oven is already hot”, the second dish only works when it is cooked after the first. Cook it first, or alone, and it fails. Tests that share state behave exactly like that:
Diagram loads as you scroll.
If test B is run alone, or before A, x has a different value and B breaks, even though nothing in the code under test has changed. The professor’s point: if you cannot control side effects, you cannot know the expected output. So every test must start fresh, with values that the test sets itself. In JUnit this fresh start is the job of the fixture (see 8.5).
Gray box testing. The professor also mentioned gray box testing, which mixes black box and white box ideas. It is not standardized (IEEE and ACM do not define it clearly), so the course concentrates on black box and white box testing (see 3.2).
Worked Examples
Exercise 1
What is a unit, and who usually writes unit tests? Why is this an exception to one of Myers’ principles?
Show solution ↓Hide solution ↑
Solution
A unit is the smallest piece of software that can be tested on its own, such as one method or one class.
Unit tests are usually written by the developer who wrote the unit. Myers’ principle says programmers should avoid testing their own programs, because they are biased towards believing their code works. Unit testing is an exception because the unit is small and detailed, and the owner knows it best: knowing exactly how the method should behave on every input is more useful at this level than an outsider’s independence. Independent testers take over at the higher levels.
Exercise 2
Classify each JUnit result as a failure or an error:
assertEquals(150, acc.getBalance()) reports expected 150 but was 140.- A test stops with a
NullPointerException because the set-up method forgot to create the object.
- A test cannot open its input file because the file was deleted.
assertTrue(list.isEmpty()) reports that the assertion was false.
Show solution ↓Hide solution ↑
Solution
- Failure. The test ran normally and the actual result (140) differed from the expected result (150): the test found a bug.
- Error. The test could not run properly; it crashed for a reason not caused by the expected result.
- Error. The environment broke the test; it says nothing about whether the code is correct.
- Failure. The assertion was evaluated and was wrong: the code did not behave as expected.
Exercise 3
State the three rules for unit tests and explain what a side effect is.
Show solution ↓Hide solution ↑
Solution
- Each unit test must run independently of all other unit tests.
- Therefore the tests can run in any order.
- A test must not depend on a side effect of a previous test.
A side effect is a change a unit makes, while being tested, to something other code can see: a shared variable, a file, a database record, or a message sent to another unit. If a later test relies on that change, its result depends on which tests ran before it.
Exercise 4
Test A deposits 50 into a shared account object created once for the whole test class. Test B asserts that the balance is 150, assuming the account started at 100. Both pass when run in the order A, B. What is wrong, and how do you fix it?
Show solution ↓Hide solution ↑
Solution
Test B depends on a side effect of test A. It only passes because A has already changed the shared account. If B runs alone or before A, the balance is 100, B fails, and the failure says nothing about the code. The tests are not independent, so they cannot run in any order.
Fix: make every test start fresh. Create a new account with balance 100 before each test (a fixture, @Before in JUnit 4), and let B perform its own deposit of 50 before asserting 150. Then each test sets every value it relies on and can run alone, first or last.
Exercise 5
Why did the professor say “if you cannot control side effects, you cannot know the expected output”?
Show solution ↓Hide solution ↑
Solution
Because the expected output of a test is only defined for a known starting state. A test case is an input plus an expected result, and the expected result is worked out assuming certain initial values. If another test can change those values through a side effect, the real starting state is unknown, so the result the test should expect is unknown too. A difference between actual and expected output could then come from the earlier test instead of a bug. That is why each test must set its own starting values.
Exercise 6
What is gray box testing, and why does the course not focus on it?
Show solution ↓Hide solution ↑
Solution
Gray box testing mixes black box ideas (testing from the specification) with white box ideas (using knowledge of the code).
The course does not focus on it because it is not standardized: bodies such as IEEE and ACM do not define it clearly, so there is no agreed method to learn and apply. The course concentrates on black box and white box testing, which are well defined.