Theory: Risk-Based Testing & Testability
2.1 ended with an uncomfortable fact: we can never test everything. So two practical questions follow. What do we test first? The answer is risk. And why was the calculator so painful to test? The answer is testability: how easily we can put an input in and see the result come out.
If We Cannot Test Everything, What Comes First?
Think about a food delivery app. Two things could go wrong:
- Problem A: the app charges the customer twice. This may not happen often, but when it does, customers lose money, get angry and leave.
- Problem B: a button has the wrong colour. This can happen easily, but it costs almost nothing.
Nobody needs a formula to see that the payment deserves testing first and the button colour can wait. Testers make that instinct systematic with risk: the chance that something bad happens, together with how bad it is. The professor asks two questions about every possible problem:
- How often can it happen? Use old data from earlier projects and your own experience.
- How much does it cost? Count the cost of fixing it, and the cost if we do not fix it (lost customers, refunds, legal trouble).
Putting the two answers together gives the impact. We then sort every problem into high, medium or low impact and test the high-impact parts first.
| How often ↓ / Cost if it happens → | Small | Medium | Big |
|---|
| Often | Medium | High | High |
| Sometimes | Low | Medium | High |
| Rare | Low | Low | Medium |
Read the grid like a map: impact grows as you move up (more often) and to the right (more expensive). The double charge sits on the right edge (big cost) and comes out high or medium even if it is rare; the button colour sits in the left column and comes out low even though it happens often. High-impact cells are tested first; low-impact cells are tested last, or skipped.
This work is called risk management, and it belongs to the planning stage of the STLC (1.3). When planning ends, the team knows what to test first, what can wait, and what it is consciously choosing to skip.
Risk = how often a problem can happen, together with how much it costs. Combined, they give the impact (high / medium / low). Because exhaustive testing is impossible, testing is risk-based: the high-impact parts are tested first, and some risk always remains.
The calculator exercise exposed a second problem. Typing every number by hand was absurdly slow, and the reason was that we had no easy way to control how input gets into the software.
The payment screen
Back to the food delivery app. You want to test only the payment screen. But to reach it you must log in, choose a restaurant, choose food, add it to the cart, and only then pay. That is slow. Worse, the app may change your data along the way: it adds a delivery fee, applies a discount. By the time you reach the payment screen, the amount it receives is not the input you wanted to test.
Then there is a second problem. After you pay, the app shows only “Success.” You cannot see the exact amount the app stored inside. But a test is the comparison of the actual result with the expected result. If we cannot see the actual result, we cannot test.
These two problems have names:
- Controllability: you can control the input that goes into the software.
- Observability: you can see the result that comes out.
Testability = controllability + observability. Software that lets you set its input precisely and see its output precisely is easy to test. Lose either one and testing becomes slow or impossible.
Take the part out and test it alone
How do we make the payment easier to test? Take the payment part out of the app and test it alone. A small piece of software like this, for example a single function, is called a unit. Tested on its own, you type the exact amount you want as input and see the exact output it produces. Both controllability and observability become good.
| Payment inside the whole app | Payment tested alone (a unit) |
|---|
| Control the input? | No: you must pass Login and Menu first, and fees change the amount on the way | Yes: you pass in the exact amount |
| See the output? | No: the app only says “Success” | Yes: you read the exact result |
| Testability | Poor | Good |
The professor added two more remarks.
- Some software is very hard to test. A typical example is embedded software, the software inside a machine such as a washing machine or a car. You cannot easily feed it chosen inputs or look at what happens inside, so both controllability and observability are poor.
- Testing each unit alone is not enough. A unit can work perfectly on its own and still fail when combined with others.
Units That Work Alone but Fail Together
Two examples from class:
- The app sends the price as
"1,000", with a comma. The bank’s part expects "1000", without one. Each part is correct alone. Together they fail.
- Two parts each round the money slightly. Alone, the error is too small to matter. Passed from one to the other, the error grows.
The professor named three kinds of problem that live between units:
| Problem | Meaning | Example |
|---|
| Precision error | A small numeric error that grows as values pass between units | Two parts each round the money, and the errors add up |
| Communication mismatch | One part does not understand the message another part sends | "1,000" sent where "1000" is expected |
| Protocol mismatch | Two parts follow different rules for how to talk to each other | One part expects a request and then a confirmation, the other sends them in the opposite order |
To find these problems we add the units together step by step and test after each step. That is integration testing, which Lecture 8 (8.3) covers in detail.
Common mistake: concluding “every unit passed, so the system works.” Unit tests cannot see precision, communication or protocol problems, because those only exist once units talk to each other. Integration testing is a separate job.
Worked Examples
Exercise 1
What is controllability? What is observability? Why do we need both?
Show solution ↓Hide solution ↑
Solution
Controllability means we can control the input that goes into the software. Observability means we can see the result that comes out.
We need both because a test compares the actual result with the expected result. Without control, we cannot prepare the test: we cannot be sure the software received the input we intended. Without observation, we cannot see what happened, so there is nothing to compare. Together they make up testability.
Exercise 2
Using the impact grid, classify each problem in a banking app as high, medium or low impact, and say which you would test first:
(a) a transfer occasionally sends money to the wrong account;
(b) the logo is slightly blurry on some screens;
(c) the balance screen sometimes takes 3 seconds instead of 1 to load;
(d) the login rejects valid passwords for many users after each update.
Show solution ↓Hide solution ↑
Solution
For each, ask how often and how much it costs, then read the grid.
- (a) Sometimes (or rare) × big cost (lost money, legal trouble) → High (Medium at worst if truly rare).
- (b) Often × small cost → Medium at most; arguably Low, since a blurry logo costs almost nothing.
- (c) Sometimes × medium cost (annoyed users) → Medium.
- (d) Often × big cost (nobody can use the app) → High.
Test (a) and (d) first, then (c), and (b) last or not at all. The ranking comes from combining both questions, not from either one alone.
Exercise 3
Why is the payment screen of a food delivery app hard to test through the full app? How can its testability be improved?
Show solution ↓Hide solution ↑
Solution
It is hard to test because both parts of testability are poor:
- Controllability is poor. To reach the payment screen you must log in, choose a restaurant and food, and fill the cart. On the way the app adds fees or discounts, so the amount that reaches the payment is not the input you wanted.
- Observability is poor. After paying, the app only shows “Success”, so you cannot see the exact amount it stored and cannot compare it with the expected result.
To improve it, take the payment part out and test it alone as a unit: pass in the exact amount you choose and read the exact result it produces. Both control and observation become good.
Exercise 4
Name the kind of integration problem in each case:
(a) The shop module sends dates as DD/MM/YYYY; the delivery module reads them as MM/DD/YYYY.
(b) A currency converter rounds to 2 decimals, then the tax module rounds again; across thousands of orders the totals drift by several baht.
(c) The payment service expects a “start” message before the amount; the cart sends the amount first.
Show solution ↓Hide solution ↑
Solution
- (a) One part does not understand the message the other sends: communication mismatch (the same kind as
"1,000" versus "1000").
- (b) Small numeric errors that grow as values pass between units: precision error.
- (c) The two parts follow different rules about the order of their conversation: protocol mismatch.
In all three cases each unit can pass its own unit tests. The problems only appear when the units are combined, which is why integration testing adds units step by step and tests after each step.
Exercise 5
Why is embedded software, such as the controller in a washing machine, usually hard to test?
Show solution ↓Hide solution ↑
Solution
Embedded software runs inside a machine, not on a screen in front of the tester. Controllability is poor because you cannot easily feed it chosen inputs (you would have to produce real water levels, temperatures or button sequences). Observability is poor because you cannot easily see its internal results; you only see what the machine physically does. Since testability is controllability plus observability, both halves are weak, so testing is hard.
Exercise 6
Every unit of a system has passed its unit tests. A manager says integration testing can be skipped. Do you agree? Explain.
Show solution ↓Hide solution ↑
Solution
No. Unit tests check each part alone, but some bugs exist only between parts: a precision error that grows when values pass between units, a communication mismatch such as "1,000" sent where "1000" is expected, or a protocol mismatch where the parts follow different rules for talking to each other. Each unit is correct alone, yet together they fail. Only integration testing, which combines the units step by step and tests after each step, can reveal these problems.