Theory: Software Artifacts & the Four Kinds of Review
Many people think testing means running code. But most of a software project is not code at all, and, as 1.3 showed, most bugs are born in the parts you cannot run. This lesson widens the picture: first what can be tested, then how to test something that cannot be executed, using the four kinds of review.
What Can We Test? More Than Code
Think about a mobile app project. It has a requirements document, screen designs, the code, a test plan, a user guide and a database. Each of these parts is a software artifact: any product of a software project, whether a document, a design, code or data. All of them can be tested.
| Artifact | How it is tested |
|---|
| Requirements document (SRS) | By reading: review |
| Design documents, screen designs | By reading: review |
| Test plan and test cases | By reading: review |
| User guide and other documents | By reading: review |
| Program code | Both: review it and run it |
| Links to hardware and other apps | Both ways |
| Stored data (database) | Both ways |
| The running program | By running it |
Why this matters
Remember where bugs are born (1.3): about 55% in the requirements, about 25% in the design, and only about 15% in the code. So about 80% of bugs live in artifacts that cannot be run. If we wait for runnable code, we meet those bugs late, and every stage a bug survives makes it about ten times more expensive. So testing must start early, on the documents themselves.
Reviews: Testing What Cannot Be Run
How do you test a document? You cannot execute it. You use a review: people read the artifact carefully and look for problems. It is like checking your group report with a friend before you submit it.
Reviews are very powerful, yet they are used less than they should be. The reason is simple: reviews need people, and people need time. The CalcEngine exercise (2.3) showed how slow careful reading is, even for under 100 lines. And we cannot review everything, so once again risk comes first: review the most important artifacts.
Four Kinds of Review, from Casual to Strict
Think about checking an assignment before you submit it. You can read it quietly by yourself. You can ask a friend to glance at it. You can meet your group and discuss it. Or, like a big company, you can use a full checklist and a timer. The stricter the review, the more preparation it needs and the more it costs.
| Kind | Formality | How it works | Goals |
|---|
| Walkthrough | Most informal | You explain the work step by step; with code, you follow it on paper line by line | Understanding the work; slow, but it makes you understand the code |
| Ad hoc review | Quick and informal | A colleague takes a fast look | A fast result, and training for developers |
| Technical review | More technical, more formal | The author presents the work and the team discusses it | Find missing requirements, remove unclear parts, train people |
| Formal inspection | Most strict, most expensive | Formal process, checklist, measurement, and a follow-up step to confirm the problems were fixed | Find and remove bugs as completely as possible |
Reading down the table, each kind is more formal, needs more preparation and costs more than the one before. Because the formal inspection is so expensive, use it only for the most important work: the high-risk artifacts.
From the slides: inspection roles. A formal inspection team has trained roles: a moderator, who plans and leads the inspection; a reader, who presents the artifact piece by piece; and a recorder, who writes down every problem found. These roles are the normal shape of an inspection team even though they were not discussed at length in class.
The Checklist Question
In a formal inspection, reviewers work from a checklist: a list of items to check, one by one. Think of a pilot before take-off, reading the checklist and confirming every item. The professor asked the class: what is the problem with a checklist? This is a likely exam question, and a good answer weighs both sides.
The good points.
- A checklist is complete for the items on it: nobody forgets a standard item.
- A formal process must be repeatable, and a checklist lets you repeat the same review many times with the same quality. That is exactly why pilots use one.
The problems.
- You miss anything that is not on the list. If a new kind of problem is not on the checklist, nobody looks for it.
- A checklist can kill creativity. Reviewers tick boxes instead of thinking.
So a good checklist is very important, but it cannot replace a thinking person.
Exam radar: for “what is the problem with a checklist?”, give both sides. Good: complete for the listed items, and repeatable. Bad: misses anything not on the list, and reduces creative thinking.
Do Reviews Pay Off?
Reviews look expensive: they take people away from other work. But studies show that when each phase is reviewed, the output of each phase carries fewer bugs into the next, and the total saving is bigger than the cost of the reviews. It is much cheaper than waiting until the end to test, which is the ×10-per-stage lesson of 1.3 once more.
Worked Examples
Exercise 1
What is the main good point and the main weak point of a checklist in a review?
Show solution ↓Hide solution ↑
Solution
Good point: the review is complete for the listed items (nobody forgets a standard check), and it is repeatable, so the same review can be done many times with the same quality.
Weak point: reviewers can miss any problem that is not on the list, and a checklist reduces creative thinking, because people tick boxes instead of thinking. A checklist is valuable but cannot replace a thinking reviewer.
Exercise 2
What is a software artifact? Give four examples and say how each one can be tested.
Show solution ↓Hide solution ↑
Solution
A software artifact is any product of a software project: a document, a design, code or data. Testing covers all of them, not only the code. For example:
- Requirements document: tested by reading it, in a review.
- Design documents: review.
- Program code: both: review it and run it.
- Database (stored data): both ways: inspect its structure and contents, and exercise it through the running program.
(Others: test plan and test cases, user guide, links to hardware and other apps, the running program.)
Exercise 3
About what percentage of bugs are found in artifacts that cannot be run? Show where the number comes from, and explain what it means for when testing should start.
Show solution ↓Hide solution ↑
Solution
Defects are born in requirements (about 55%), design (about 25%) and code (about 15%). Requirements and design documents cannot be executed, so
55%+25%=80%of bugs are in artifacts that cannot be run. If testing waits for runnable code, those bugs are found late, and each stage a bug survives makes it about ten times more expensive. So testing must start early, by reviewing the requirements and design documents.
Exercise 4
Reviews are very powerful. Why are they used less than they should be, and how should a team decide what to review?
Show solution ↓Hide solution ↑
Solution
Reviews are underused because they need people, and people need time: careful reading is slow (the CalcEngine exercise took about 20 minutes for under 100 lines), and a team cannot review everything.
The team should therefore use risk first: review the artifacts where a defect would have the highest impact, and reserve the most formal, expensive kind of review for the most important work.
Exercise 5
Order the four kinds of review from most informal to most formal, giving the goal of each. Then choose a suitable kind for:
(a) the braking-control module of a car;
(b) a junior developer’s two-line fix, needed in an hour.
Show solution ↓Hide solution ↑
Solution
From informal to formal:
- Walkthrough: the author explains the work step by step (code traced on paper); slow, but it builds understanding.
- Ad hoc review: quick and informal; goals are a fast result and training developers.
- Technical review: the author presents, the team discusses; goals are to find missing requirements, remove unclear parts and train people.
- Formal inspection: formal process, checklist, measurement and follow-up; goal is to find and remove bugs as completely as possible.
(a) The braking module is safety-critical, so the highest impact: formal inspection, despite its cost.
(b) A tiny, urgent change: an ad hoc review, which gives a fast result and trains the junior developer.
Exercise 6
Name the three roles in a formal inspection team and say what each does.
Show solution ↓Hide solution ↑
Solution
- Moderator: plans and leads the inspection and makes sure the process is followed.
- Reader: presents the artifact to the team, piece by piece.
- Recorder: writes down every problem found.
The roles are trained and fixed, which is part of what makes a formal inspection formal and repeatable.
Exercise 7
A manager says: “Reviews take people away from development, so we will skip them and test at the end.” Do you agree? Explain.
Show solution ↓Hide solution ↑
Solution
No. About 80% of bugs are created in requirements and design, before any code exists, and a bug costs about ten times more for every stage it survives. Testing only at the end finds those bugs at the most expensive point. Studies show that when each phase is reviewed, each phase passes on fewer bugs, and the total saving is bigger than the cost of the reviews. The manager is right that reviews cost people’s time, so the answer is to review by risk (heavy reviews only for the most important artifacts), not to skip them.