Theory: Expected Results, Verification & Validation
Running a test is easy. Knowing whether the result is right is the hard part. This lesson starts with the question the professor put to the class, how do you know a test result is correct?, and follows the answer to two of the most important words in the course: verification and validation.
How Do You Know a Test Result Is Right?
When the professor asked the room this question, some students answered, “check it against the requirements.” It sounds sensible, and the professor pushed back gently, because it hides a problem.
Think of a teacher marking a pile of maths exams with no answer key. She can read every answer, and she can even solve a few of them herself, but she has nothing fixed to compare against, so she can never be sure of any mark. Testing is exactly the same. To say that a result is right or wrong, you need a reference: something decided in advance and independent of the run itself. In testing that reference is called the expected result.
Decided before you run anything
The expected result is written at the same moment as the test case, before you run anything. You never run the program first, look at the answer, and then decide whether it seems fine; that would be like marking the exam by asking whether each answer looks plausible. Instead:
- write down the input;
- write down what the output must be (the expected result);
- only then run the code and compare the actual output with the expected one.
Where the expected result comes from
The professor asked whether you would really trace each of ten thousand test results all the way back to the customer’s original requirements. Of course not; it would take forever. In practice, testers derive the expected result from the specification (SRS), which is precise, written for the development team, and close at hand. Keep that in mind, because it is the seed of the next idea: checking against the specification is not the same as checking against what the customer really needs.
Exam radar: a test case is not a test case until it contains both the input and the expected output. The professor repeated this several times and expects it in every test case you write, including those in the exam.
Verification and Validation
Return to the house from 1.1. The family said, “We want a big kitchen,” and the architect turned that into a blueprint. The builder builds the house exactly as the blueprint says.
- Walk through the finished house with the blueprint in your hand, ticking off every wall and every window. If everything matches, you have done verification. The question is: did we build the thing right?
- Now suppose the architect forgot to draw a kitchen. The builder built precisely what the blueprint showed, so the house passes your check with flying colours, and yet the family cannot cook. To notice the problem you must put the blueprint down, go back to the family and ask, “Is this what you actually needed?” That is validation: did we build the right thing?
Diagram loads as you scroll.
Verification compares the finished work with the specification, something already derived from the needs. Validation jumps further back and compares the finished work with the needs themselves.
| Verification | Validation |
|---|
| Informal question | Did we build the thing right? | Did we build the right thing? |
| Reference point | The specification (SRS, design) | The user’s real needs / intended use |
| Catches | Builders who did not follow the plan | Plans that were wrong from the beginning |
Formal definitions (CMMI).
- Verification confirms that a work product properly reflects the requirements specified for it.
- Validation confirms that the product, as delivered, fulfils its intended use in its real environment.
Why Both Matter
The two checks catch different kinds of trouble. Verification catches builders who did not follow the plan; validation catches plans that were wrong from the beginning. A product can conform perfectly to a faulty specification, pass verification, and still fail the user. Testing as a whole is the larger umbrella: one activity that serves both.
Exam radar: the exam will not use the neat phrases “build the thing right” and “build the right thing”; they will be paraphrased. Learn the ideas: verification = checked against the specification; validation = checked against the user’s real needs or intended use. Also watch the context: in casual speech people say “verification and validation” to mean nothing more than “testing”. Read the sentence to see whether the technical meaning is intended.
Worked Examples
Exercise 1
A student writes this test case: “Test 7: enter a quantity of 0 into the order form and see what happens.” What is wrong with it? Rewrite it properly, assuming the SRS says quantities must be between 1 and 99 and invalid quantities are rejected with the message “Invalid quantity”.
Show solution ↓Hide solution ↑
Solution
The test case has an input but no expected result. “See what happens” means the tester will run the program first and then judge whether the answer looks fine, which is marking an exam without an answer key.
A complete test case states both, decided before running:
| Test | Input | Expected result |
|---|
| 7 | quantity = 0 | Order is rejected with the message “Invalid quantity” |
The expected result is taken from the SRS (valid range 1 to 99). Now the actual output can be compared with something fixed. ✓
Exercise 2
Why must the expected result be decided before the test is run? Use the professor’s exam-marking comparison.
Show solution ↓Hide solution ↑
Solution
To call a result right or wrong you need a reference that is independent of the run itself. A teacher marking exams with no answer key can only ask whether each answer looks plausible, so she can never be sure of any mark.
If a tester runs the program first and then decides whether the output “seems fine”, the program’s own output has quietly become the reference, and a wrong answer that looks reasonable will pass. Writing the expected result first, at the same time as the test case, gives a fixed answer key, so the comparison of actual vs. expected is meaningful.
Exercise 3
In practice, where do testers take the expected result from, and why not directly from the customer’s original requirements?
Show solution ↓Hide solution ↑
Solution
From the specification (SRS). Tracing each of thousands of test results back to the customer’s original, everyday-language requirements would take forever. The SRS is precise, written for the development team, and close at hand, so it is the practical source of expected results.
The consequence is important: tests built this way check the software against the specification, which is verification. Whether the specification itself matches the real need is a separate question (validation).
Exercise 4
A team finds that their software passes every check against the SRS, but the customer says it is useless. Is this a verification failure or a validation failure?
Show solution ↓Hide solution ↑
Solution
It is a validation failure. The software matched the specification, so verification succeeded. The trouble is that the specification did not match what the user actually needed, and only validation checks the product against the real need. In the house analogy, the house matches the blueprint perfectly, but the blueprint forgot the kitchen.
Exercise 5
(Discussion) Distinguish verification from validation and explain why both are necessary.
Show solution ↓Hide solution ↑
Solution
Definitions. Verification asks whether the team built the product correctly according to its specification: the tester compares the software, or another development artifact, with the SRS or the design document. Validation asks whether the team built the correct product for the user’s actual needs, so its reference point is the original requirement or the intended use in the real environment.
Why both. Both are necessary because the specification can itself be incomplete or wrong. A program may conform perfectly to a faulty SRS and therefore pass verification while still failing the user. Bug definition 4 (see 1.6) illustrates this: the product omits something the specification does not mention but the user needs, and verification cannot expose the omission when the faulty specification is its reference; validation can. Therefore, verification establishes conformance to the specified solution, while validation establishes fitness for the user’s real problem.
Exercise 6
Classify each activity as verification or validation, or say if “V&V” is being used casually.
- (a) A reviewer checks that every screen in the design document implements a requirement listed in the SRS.
- (b) A group of real nurses uses the new ward-scheduling app for a week on the hospital floor and reports whether it fits their work.
- (c) A tester confirms that a work product properly reflects the requirements specified for it.
- (d) A job advert says: “Join our V&V team and help us find bugs before release.”
Show solution ↓Hide solution ↑
Solution
- (a) Verification: the design is checked against the specification.
- (b) Validation: the product is checked against the users’ real needs, in its real environment.
- (c) Verification: this is the CMMI definition, paraphrased (the reference is what was specified).
- (d) Casual use: here “V&V” simply means testing in general, not the technical distinction.
Exercise 7
An exam question says: “Which activity confirms that the delivered product fulfils its intended use when placed in its operational environment?” Answer, and explain how you recognised it even though the usual slogan was not used.
Show solution ↓Hide solution ↑
Solution
Validation. The wording is a paraphrase of the CMMI definition of validation. The give-away is the reference point: “intended use” and “operational environment” point to the user’s real needs, not to a specification. Had the question said “conforms to what was specified for it”, the answer would be verification.