1.4 Software Disasters & Functional vs. Non-Functional Requirements

Theory: Software Disasters & Functional vs. Non-Functional Requirements

The professor spent real time on famous software failures, not to scare anyone but to show what a small mistake can cost. Each story carries a lesson, and together they open the door to risk, an idea that grows in importance every week of this course. The second half of the lesson looks at what a specification asks for, and why some requirements are far harder to test than others.

Logic Slips: One Missing Branch, One Missing Character

The F-18 and the if with no else

A fighter jet, the F-18, crashed because a piece of code had an if with no else. The programmers believed one situation could never happen, and it happened. The lesson: always ask, “what if the impossible case happens?”

The F-16 simulator and the equator

A flight simulator for the F-16 flipped upside down every time it crossed the equator, because the code lacked a minus sign for south latitude. One tiny character.

The Year Bugs

These are funny and sad at once.

• Mary Bandar, aged 104, received an invitation to kindergarten. The software stored ages in two digits, so 104 became 04 and the machine saw a four-year-old.
• Mr. Blodgett’s car insurance tripled when he turned 101, because the system read his age as 01, and a one-year-old driver looked like a disaster waiting to happen.

The lesson is that the developers simply never imagined that people could live past 100. A boundary nobody thought of is a boundary nobody tested.

Overflow: When the Boxes Run Out

The overflow bugs come from limited storage. A 16-bit whole number is like a row of sixteen little boxes (bits). If one box is used to remember the sign (positive or negative), only fifteen are left for the size:

$$2^{15} - 1 = 32{,}767$$

is the biggest number a signed 16-bit integer can hold. An unsigned 16-bit integer uses all sixteen boxes for the size, so its maximum is $2^{16} - 1 = 65{,}535$. Count past the limit and the number wraps around and misbehaves: one more than 32,767 in a signed 16-bit counter becomes $-32{,}768$.

• A hospital computer in Washington, D.C. crashed when its day counter passed 32,767, and staff had to work manually for a long time.
• In 1993 the Linux term program died worldwide for the same reason: it used a signed integer where an unsigned one, with all sixteen boxes for the size and twice the range, was needed.

Safety-Critical Software

The darkest examples are the radiation therapy machines. A prescribed dose of 86 rads was entered into the system and wrongly interpreted as thousands of rads, and patients died.

Software like this, where a bug can kill someone, is called safety-critical, and it demands far more testing effort than ordinary software. The professor asked whether the apps on your phone are safety-critical. The answer is mostly no: when a phone app hangs, you restart it, and nobody is harmed.

A break in the Wrong Place: AT&T

The AT&T outage blocked about five million calls over nine hours. A programmer had placed a break inside a switch that sat inside a loop. The break only escaped the switch, not the loop, so the program carried on in a place the programmer never intended. The whole thing came from a small change made to improve readability.

while (messagesWaiting()) {          // the loop
    switch (message.type()) {
        case STATUS:
            if (alreadyHandled) {
                break;               // leaves only the switch, NOT the loop
            }
            update();
            break;
        // ...
    }
    process(message);                // still runs after the "early exit"
}

The lesson: even a “harmless” tidy-up changes behaviour, and a change anywhere deserves testing.

The Money Bugs

• A Norwegian ATM paid out ten times the requested cash.
• A UK bank duplicated every transfer for half an hour.
• An Australian man bought Sri Lankan rupees at a wrongly displayed exchange rate, sold them the next day for about four times as much, and a judge let him keep the profit because he had not acted fraudulently.

What Ties the Stories Together

Story	What went wrong	Lesson
F-18	if without else	Ask what happens if the “impossible” case happens.
F-16 simulator	Missing minus sign for south latitude	One character can flip the world.
Mary Bandar, 104	Two-digit age: 104 → 04	Nobody imagined ages past 100.
Mr. Blodgett, 101	Age read as 1, insurance tripled	Same: an unimagined boundary.
Hospital counter, Linux term	Signed 16-bit overflow past 32,767	Storage is limited; values wrap around.
Radiation therapy	86 rads interpreted as thousands	Safety-critical software needs far more testing.
AT&T	break in a switch inside a loop	Small “readability” changes can break everything.
ATM, UK bank, rupees	Wrong payouts, duplicated transfers, wrong rate	Bugs cost real money.

The bigger the possible damage, the more testing effort a project must invest to prevent it. This is the doorway to risk, which drives how testers decide what to test first (Principle 2 in 1.5, and 2.2).

Functional and Non-Functional Requirements

A specification contains two families of requirements, and testers treat them very differently.

	Functional requirements	Non-functional requirements
Describe	What the software does	How well it does it
Examples	Save a file, calculate a tax, print a report	Performance, security, reliability, usability
How hard to test	Relatively easy: each one is in the specification and can be checked	Far harder: quality is slippery (“what does fast mean?”)
Tools and techniques	Many well-studied techniques and frameworks	No single standard framework

Why non-functional testing is hard

To test performance you must first pick a metric, and the standard metric is response time. Then you need a tool and a set-up that can simulate the load. The professor’s example: “the system must support one million users at once, each receiving a response within one millisecond.” You cannot test that by clicking around; you need special tools and a special environment.

For security, one very popular technique is penetration testing: attacking the system the way an intruder would, to find the weaknesses first.

The deeper difficulty is that non-functional testing has no single standard framework, so very often you have to invent your own metrics, techniques and tools each time.

Load vs. stress testing. Two performance tests are easy to mix up. Load testing gives the system a lot of work (many users, many requests) and checks it still performs. Stress testing restricts the resources available to do the work (less memory, CPU, bandwidth) and checks how it copes.

---

Worked Examples

Exercise 1

The F-18 crash came from an if with no else. What question should a tester always ask because of this story, and what kind of test case does it lead to?

Show solution ↓Hide solution ↑

Solution

The programmers believed one situation could never happen, so they wrote no branch for it, and it happened. The tester’s question is: “What if the impossible case happens?”

This leads to test cases that deliberately feed the software the situations its authors assumed could not occur (unexpected values, unusual combinations, the missing else path), each with an expected result saying how the system must behave safely.

Exercise 2

Mary Bandar, aged 104, was invited to kindergarten. Explain exactly how the software reached that conclusion, and suggest three ages a tester could have used to catch the bug.

Show solution ↓Hide solution ↑

Solution

The software stored ages in two digits. Keeping only the last two digits of 104 gives 04, so the system believed she was 4 years old, the right age for kindergarten.

The root cause is that nobody imagined anyone living past 100, so the boundary at 100 was never tested. Good test ages sit at and around that boundary: 99, 100 and 101 (and an extreme such as 104). For 100 and above, the expected result is that the age is stored and used correctly, not truncated.

Exercise 3

(a) What is the largest value a signed 16-bit integer can hold, and why? (b) What is the largest unsigned 16-bit value? (c) A hospital’s signed 16-bit day counter is at its maximum. What happens the next day?

Show solution ↓Hide solution ↑

Solution

(a) One of the 16 bits stores the sign, leaving 15 bits for the size:

$$2^{15} - 1 = 32{,}768 - 1 = \mathbf{32{,}767}$$

(b) Unsigned uses all 16 bits for the size:

$$2^{16} - 1 = 65{,}536 - 1 = \mathbf{65{,}535}$$

which is roughly twice the range. ✓

(c) Adding 1 to 32,767 overflows: the value wraps around to the most negative signed value, $-32{,}768$, and the program misbehaves. This is what crashed the Washington, D.C. hospital computer, and the same signed-vs-unsigned mistake killed the Linux term program in 1993.

Exercise 4

What is safety-critical software? Is a typical mobile game safety-critical? Explain what the difference means for testing effort.

Show solution ↓Hide solution ↑

Solution

Safety-critical software is software where a bug can kill or seriously harm someone, for example the radiation therapy machines where an entered dose of 86 rads was interpreted as thousands of rads and patients died.

A typical mobile game is not safety-critical: if it hangs, you restart it and nobody is harmed.

Because the possible damage is so much larger, safety-critical software demands far more testing effort. This is the general rule behind risk: the bigger the possible damage, the more effort a project must invest to prevent it.

Exercise 5

In the AT&T outage, a break was placed inside a switch that sat inside a loop. Explain why this caused the program to behave differently from what the programmer intended.

Show solution ↓Hide solution ↑

Solution

In languages such as C and Java, break exits only the innermost enclosing switch or loop. The programmer expected the break to leave the surrounding processing, but it only escaped the switch; the loop kept running, so code after the switch executed in a situation where it should not have. The change had been made to improve readability, which shows that even a “harmless” tidy-up changes behaviour and must be tested. The result was about five million blocked calls over nine hours.

Exercise 6

Classify each requirement as functional or non-functional:

• (a) The system shall calculate VAT at 7% on every invoice.
• (b) 95% of search requests shall receive a response within 2 seconds.
• (c) The system shall export the monthly report as a PDF.
• (d) The system shall resist the attacks listed in the company’s penetration-testing checklist.
• (e) A first-time user shall be able to place an order without help.

Show solution ↓Hide solution ↑

Solution

A functional requirement says what the software does; a non-functional one says how well it does it.

• (a) Functional: a calculation the software performs.
• (b) Non-functional: performance, measured by response time.
• (c) Functional: an action (export a report).
• (d) Non-functional: security.
• (e) Non-functional: usability.

Exercise 7

(Discussion) Explain why testing the requirement “the system must support one million users at once, each receiving a response within one millisecond” is much harder than testing “the system prints the monthly report”.

Show solution ↓Hide solution ↑

Solution

Principle. The first is a non-functional requirement (how well: performance), the second a functional one (what the software does). Functional requirements are relatively easy to test: each is in the specification and can be checked directly, and many well-studied techniques exist. Non-functional requirements are harder because quality is slippery and there is no single standard framework for testing them.

What it means here. For the performance requirement the tester must first fix a metric (response time), then obtain special tools and a special environment that can simulate a million simultaneous users. This is load testing: giving the system a lot of work, as opposed to stress testing, which restricts the resources it has. Clicking around cannot test it, and the tools and metrics often have to be built for the project. Printing a report, by contrast, can be checked by running the function once and comparing the output with the expected report.