Case Study: Backup Design & Ransomware Drill

Theory: How to Approach These

This case uses everything from Week 9: RAID (9.2), RPO/RTO and backup layers (9.3), and how ransomware attacks backups (9.1). Work through it in this order:

Critique first. For every sentence of the current setup, ask: which incident would this fail against? Check each layer against the “RAID is not backup” table (9.2) and the “Not all are backup” table (9.3). Look for things an attacker with admin rights could reach (9.1).
RAID calculation (see 9.2): usable capacity with $N$ disks of size $S$ is RAID 5 $= (N-1)S$ , RAID 6 $= (N-2)S$ , RAID 10 $= NS/2$ . Discard any level that misses the capacity requirement, then choose on protection. Large disks mean long rebuilds, so think about RAID 6 and a hot spare.
Protection design (see 9.3):
1. RPO → how often: the most frequent layer that survives ransomware (log backup) must run at least once per RPO.
2. RTO → how to restore: prefer full + differential + logs (short chain) over long incremental chains.
3. Where: real backups go to immutable, off-domain, offsite storage; snapshots and replication are only speed layers.
Ransomware drill (see 9.3, “Ransomware: what can we restore?”):
1. Which layers survive? Anything a Domain Admin can reach does not.
2. The latest restore point is the last backup that ran before the jobs were stopped, not the time of encryption.
3. Data lost = restore point → encryption; compare with the RPO.
4. Downtime = encryption → service restored; compare with the RTO.
5. Server images are only safe from before the attacker got in; otherwise rebuild clean and restore data only.

The Case

MediCare Plus’s IT manager describes how systems are protected today:

“The file server uses RAID 5 with 6 disks of 16 TB each. The server has 8 disk bays. There is no hot spare. The file server needs at least 50 TB of usable space.

All systems are backed up every night at 23:00 to a NAS in the same server room. The NAS is on the hospital network and joined to the domain, so we can manage it with our normal admin accounts. Full backup on Sunday, incremental Monday to Saturday.

The EMR is also replicated live to a second server in the same building. Honestly, replication is our real backup.

The file server uses Windows Shadow Copies so users can restore their own files.

Backup failure alerts are sent to the email of Khun Anan. (Khun Anan left the hospital last year.)

We have never tested a full restore. The backup software says the jobs are successful.”

System information

System	What it does	If it stops
EMR	All patient records	Patient care slows; risk of wrong treatment. The medical director has set RPO 15 minutes and RTO 4 hours.
Lab system	Receives results from lab instruments and sends them to the EMR	Samples must be sent to an outside lab (expensive). Results are delayed. Lab instruments can store the last 6 hours of results and send them again later.
Appointment booking	Online and phone bookings, about 500 per day	Patients must call. Staff take bookings on paper. No direct clinical risk.

Ransomware drill timeline (assume the protection design from 1.2 is in place):

When	What happened
Tue 10:20	An attacker logs in to the VPN with a stolen password.
Tue 22:40	The attacker becomes Domain Admin.
Thu 03:30	The attacker disables all backup jobs, and tries to delete every snapshot and backup it can reach. Nobody notices.
Thu 14:00	The EMR and the Lab system are encrypted. Replication copies the encryption to the standby server.
Thu 14:30	Staff report a ransom note. IT starts the response.

Assume rebuilding clean servers and restoring data takes 9 hours after IT starts the response.

Case Questions & Model Answers

Exercise Critique

Identify every weakness in the current setup and explain what incident each one would fail against.

Show solution ↓

Solution

Go through the setup sentence by sentence.

#	Current setup	Weakness	Fails against
1	RAID 5, 6 × 16 TB	Survives only one disk failure. Rebuilding a 16 TB disk takes about two days or more, under heavy load; a second failure (or unreadable block) during that time loses the whole array.	A second disk failure during rebuild
2	No hot spare, although 2 of the 8 bays are empty	The rebuild only starts after someone notices, buys and fits a disk; with a 3-person IT team and no night shift the array can stay degraded for days.	Long degraded period
3	Backups on a NAS in the same server room	Production and backups share one room: no offsite copy.	Fire, flood, theft, power surge
4	NAS on the hospital network, domain-joined, managed with normal admin accounts	One stolen domain admin account can delete or encrypt the backups (the attacker’s stage 5). Backups are not immutable.	Ransomware
5	Nightly backup only (23:00) for all systems	Worst-case data loss is almost 24 h; the EMR’s 15-minute RPO cannot be met. No log backups.	Any EMR incident
6	Full Sunday + incremental Mon–Sat	By Saturday a restore needs the full + 6 incrementals: slow (threatens the 4-hour RTO), and one broken incremental breaks the chain.	Restores late in the week
7	“Replication is our real backup”	Replication copies the damage (deletion, corruption, encryption) within seconds. The standby is also in the same building. Replication is not backup.	Ransomware, corruption, building disaster
8	Shadow Copies for users	Snapshots on the same storage, controlled by the same admins; ransomware routinely deletes them. Good only for user mistakes, not backup.	Ransomware, disk/array loss
9	Alerts go to a departed employee	Failed or stopped backups are never noticed; the gap is discovered only during a disaster. (His mailbox should also have been closed when he left.)	Silent backup failure
10	Never tested a full restore; “jobs are successful”	A successful job does not prove the data can be restored. Nobody knows the real recovery time, so the RTO is unproven, and there is no practised procedure.	Every incident
11	Same schedule for every system	Protection is not set by business impact: the EMR gets the same treatment as low-risk systems.	EMR RPO/RTO

Answer: the setup protects only against a single disk failure. Against ransomware, a building disaster, silent backup failure or a demanding RPO/RTO it fails, because the only backup copy is reachable by domain admins, sits in the same room, runs only nightly, is unmonitored and has never been restore-tested.

Exercise 1.1

The file server will get new disks: 6 disks of 16 TB. Calculate the usable capacity for RAID 5, RAID 6 and RAID 10. Choose one that meets the 50 TB requirement, explain your choice, and say whether you need a hot spare.

Show solution ↓

Solution

$N = 6$ , $S = 16$ TB (raw $= 96$ TB).

Level	Working	Usable	Meets 50 TB?	Failures survived
RAID 5	$(6 - 1) \times 16$	80 TB	✓	1
RAID 6	$(6 - 2) \times 16$	64 TB	✓	2
RAID 10	$(6 \times 16) / 2$	48 TB	✗ (2 TB short)	1 guaranteed

RAID 10 is out: 48 TB is below the 50 TB requirement.

RAID 5 vs RAID 6: both have enough space. But each disk is 16 TB, so a rebuild takes a long time (around two days at 100 MB/s, see 9.2), with heavy load on the remaining disks. RAID 5 has no protection left during that rebuild; a second failure destroys the file server. RAID 6 still survives one more failure while rebuilding. RAID 6 gives 64 TB, which leaves 14 TB of headroom above 50 TB.

Choice: RAID 6 (64 TB usable, survives 2 disk failures).

Hot spare: yes. The server has 8 bays and the array uses 6, so put a 7th 16 TB disk in one free bay as a hot spare. When a disk fails, the rebuild starts automatically, instead of waiting for a 3-person team with no night shift to replace it. The hot spare does not add usable capacity (still 64 TB), and one bay stays free.

Answer: RAID 5 = 80 TB, RAID 6 = 64 TB, RAID 10 = 48 TB (fails 50 TB) → choose RAID 6 with one hot spare.

Exercise 1.2

For each system (EMR, Lab system, Appointment booking), choose the protection methods and how often they run (full, incremental or differential, log backup, snapshot, replication), and say where the backups are stored. The design must meet the RPO and RTO. Explain in 3 to 5 sentences why it meets them.

Show solution ↓

Solution

Step 1: set targets. Only the EMR’s targets are given, so set the others from the impact:

Lab system: RPO 1 h, RTO 4 h. Its results feed the EMR and an outage costs money (outside lab) and delays care. The instruments keep the last 6 hours of results and can resend them. If a failure loses at most 1 h of data and the system is back within 4 h, every lost result is at most $1 + 4 = 5$ h old when the system returns, which is still inside the 6-hour buffer. So the instruments can resend everything, and the effective data loss is zero.
Appointment booking: RPO 24 h, RTO 24 h. No direct clinical risk; staff can take bookings on paper, and lost bookings can be rebuilt by phone. Cheap protection is enough.

Step 2: design.

Method	EMR (RPO 15 min · RTO 4 h)	Lab system (RPO 1 h · RTO 4 h)	Appointment booking (RPO 24 h · RTO 24 h)
Full backup	Weekly, Sun 23:00	Weekly, Sun 23:00	Weekly, Sun 23:00
Incremental or differential	Differential nightly 23:00 (Mon–Sat)	Differential nightly 23:00	Differential nightly 23:00
Log backup	Every 15 min	Every hour	None
Snapshot	Hourly, kept 48 h (quick rollback of mistakes)	Every 4 h, kept 48 h	None
Replication	Live to a standby server, preferably in another building (hardware failure)	None (restore from backup meets 4 h for a server failure)	None
Where backups are stored	All full, differential and log backups go to an immutable (object-lock/WORM) backup repository that is not domain-joined, on its own network segment, managed with separate accounts with MFA, in a different building from the servers. A second immutable copy goes offsite (immutable cloud vault). Retention at least 30 days, longer than an attacker’s likely dwell time.	Same repository and offsite copy	Same repository and offsite copy

Operational controls that are part of the design (each fixes a critique point):

Backup alerts go to a monitored IT group mailbox / on-call phone, including an alert when no log backup has succeeded for 30 minutes.
Full restore tests at least quarterly, timing each one against the RTO.
The file server uses RAID 6 + hot spare (1.1); its Shadow Copies stay as a user convenience only.
The old NAS is either retired or used only as a fast local cache; it never counts as “the backup”.

Why it meets the targets: The EMR’s 15-minute log backups go straight to immutable storage, so even ransomware can cost at most 15 minutes of data, which meets RPO 15 min; replication covers the most common outage (server failure) in minutes. The full + one differential + logs chain is short (never more than one differential), which keeps the restore within the 4-hour RTO and avoids broken incremental chains. The Lab’s hourly log backups meet its 1-hour RPO, and because RPO + RTO ( $1 + 4 = 5$ h) fits within the instruments’ 6-hour buffer, any lost results can be resent. Appointment booking needs only a nightly differential to meet a 24-hour RPO, and a restore of a small system easily fits in 24 hours.

Exercise 2a

Ransomware drill: which of your protection layers survive the attack, and which do not? Explain why.

Show solution ↓

Solution

The attacker became Domain Admin on Tue 22:40, so the test for each layer is: can a Domain Admin change or delete it?

Layer	Survives?	Why
Full, differential and log backups in the immutable, off-domain repository	✓	The repository does not accept domain accounts, and object lock means nobody, not even an admin, can delete or change a backup before its retention ends. The 03:30 deletion attempts fail.
Offsite immutable cloud copy	✓	Same reasons, and it is in another place.
Snapshots (EMR hourly, Lab 4-hourly)	✗	They live on the production storage, managed by domain admins; deleted at 03:30 (anything left is encrypted with the system at 14:00).
Replication (EMR standby)	✗	Replication copied the encryption to the standby at 14:00.
New backups after 03:30	✗ (never made)	The attacker disabled the jobs. Disabling a job does not delete old backups, but nothing is backed up after 03:30: no more log backups, and the Thu 23:00 differential would never run.

Note that the “no log backup for 30 minutes” alert would fire at about 04:00, but the drill says nobody noticed (a 3-person team with no night shift). An alert only helps if someone is on call to act on it.

Answer: only the immutable, off-domain backups (local repository and offsite copy) survive; snapshots and the replica do not, and no new backups exist after Thu 03:30.

Exercise 2b

EMR: write the restore chain (which backups, in which order). What is the latest point you can restore to?

Show solution ↓

Solution

Find the last backup of each kind that ran before the jobs were disabled at Thu 03:30:

Full: Sun 23:00.
Differential: Wed 23:00 (Thursday’s 23:00 differential never ran).
Log backups: every 15 minutes after Wed 23:00, until the jobs stopped: 23:15, 23:30, …, Thu 03:15, 03:30.

The 03:30 log backup is the borderline case: the jobs were disabled at 03:30, so assume it completed just before; if it did not, the chain ends at 03:15.

Restore chain:

Full backup, Sun 23:00
Differential, Wed 23:00 (contains all changes Mon–Wed)
Log backups Wed 23:15 → Thu 03:30, replayed in order (18 files: 3 before midnight + 4 each for 00:xx, 01:xx and 02:xx + 03:00, 03:15, 03:30)

Latest restore point: Thu 03:30 (Thu 03:15 if the 03:30 log backup did not run).

The snapshots and the replica are not used: they are gone or encrypted, and they are the attacker’s territory.

Exercise 2c

How much EMR data is lost? How much Lab data is lost? (Remember the lab instruments.) Compare each with its RPO.

Show solution ↓

Solution

EMR. Data lost = restore point → encryption:

\text{Thu } 03{:}30 \to \text{Thu } 14{:}00 = 10\ \text{h}\ 30\ \text{min}

(10 h 45 min if the 03:30 log backup did not run.) The RPO is 15 min, so the RPO is badly missed. The design was correct, but the backups silently stopped 10.5 hours before the encryption; this is exactly the “nobody noticed when they stopped” lesson from 9.3.

Lab system. Its log backups run hourly, so the last one before 03:30 was Thu 03:00.

\text{Thu } 03{:}00 \to \text{Thu } 14{:}00 = 11\ \text{h}

From backups alone, 11 h is lost against an RPO of 1 h: missed.

The instruments’ 6-hour buffer (treat it as a rolling window: at any moment the instruments hold the previous 6 hours of results):

At 14:30, when the response starts, the buffer holds results from about 08:30 to 14:30. If IT preserves it right away (export the buffered results, or stop sending new runs to in-house instruments), the results from 08:30 to 14:00 can be resent once the Lab system is back.
If nobody acts, the window keeps moving: by the time the Lab system returns at 23:30 the buffer holds only 17:30–23:30, so none of the lost 03:00–14:00 results can be resent.
Even in the best case, results from 03:00 to 08:30 (5 h 30 min) are beyond the buffer and are lost; they must be recovered from printouts or instrument records, or the samples re-run if still usable. That still misses the 1-hour RPO.

Answer: EMR loses about 10 h 30 min (RPO 15 min missed); Lab loses 11 h from backups, reduced to about 5 h 30 min if the instrument buffers are preserved at 14:30 (RPO 1 h still missed).

Exercise 2d

How long is the downtime? Compare it with the RTO of the EMR and the Lab system. How is data from the downtime period handled?

Show solution ↓

Solution

Downtime. Service stops at encryption and returns 9 h after the response starts:

14{:}30 + 9\ \text{h} = 23{:}30 \text{ Thu}

\text{Downtime} = 14{:}00 \to 23{:}30 = 9\ \text{h}\ 30\ \text{min}

System	RTO	Downtime	Result
EMR	4 h	9 h 30 min	Missed by 5 h 30 min
Lab system	4 h (our target)	9 h 30 min	Missed by 5 h 30 min

Even counted from the start of the response (14:30), 9 h is more than twice the 4-hour RTO. The design’s backups survived, but a clean rebuild after ransomware takes longer than a normal restore, so the RTO is not met.

Data from the downtime period (Thu 14:00–23:30):

Staff switch to paper downtime procedures: paper orders, medication checks and notes; lab samples go to the outside lab.
After the restore is verified, the paper records are re-entered into the EMR (step 7, go live), checked by a second person for accuracy, and outside-lab results are entered into the Lab system/EMR.
The lost window before the outage (EMR 03:30–14:00, Lab 03:00–14:00) is harder: it was entered electronically, so there is no paper. It must be rebuilt from other sources (pharmacy dispensing records, printouts, resent instrument results, staff and patients) and the affected records flagged for review.

How to bring the RTO down to 4 h:

Keep a pre-built clean recovery environment (golden server images, documented build scripts) so “rebuild clean servers” is not on the critical path.
Restore in priority order: the EMR first, the Lab second, booking last.
Rehearse the full ransomware restore and time it; fix whatever is slow.
Shorten the time to detect: MFA on the VPN (blocks the Tue 10:20 login), alerts on new Domain Admin rights (Tue 22:40) and on stopped backup jobs (Thu 03:30), with a 24/7 on-call responder (for example an external monitoring service, since there is no night shift). Acting at 03:30 would also have cut the data loss to minutes.
Limit admin power: separate, tightly controlled privileged accounts (PAM, just-in-time admin) so one stolen account cannot disable every backup job.

Answer: 9 h 30 min of downtime against a 4-hour RTO for both systems: not met. Work continues on paper and is re-entered after go-live; the RTO needs pre-built clean infrastructure, rehearsed restores and faster detection.

Exercise 2e

Can you restore the servers from your backups? Which backups are safe to use for servers, and why?

Show solution ↓

Solution

The problem. The attacker was inside from Tue 10:20 and had Domain Admin from Tue 22:40. Any server or system-state backup taken after Tue 10:20 may contain the attacker’s persistence: new admin accounts, scheduled tasks, backdoor software, changed Active Directory settings. Restoring such a server image would let the attacker straight back in, and the encryption could start again.

Which backups are safe for servers:

Backup	Taken	Safe for servers?
Full, Sun 23:00	Before Tue 10:20	✓
Differential, Mon 23:00	Before Tue 10:20	✓
Differential, Tue 23:00	After Tue 10:20 (and after Domain Admin at 22:40)	✗
Differential, Wed 23:00, logs up to Thu 03:30	After the compromise	✗ for servers (use for data only)

So the newest server state you can trust is Mon 23:00 (Sunday full + Monday differential). The 30-day immutable retention makes these older copies available.

Better practice: rebuild clean, restore data only.

Build new servers from trusted installation media or golden images, fully patched.
Reset all admin and service passwords (the attacker had Domain Admin), and close the entry point: MFA on the VPN, disable the stolen account.
Restore only the data: the EMR database to Thu 03:30 and the Lab database to Thu 03:00, using the chains above.
Scan and verify the restored data with users before going live.

Answer: server images are safe only from before the Tue 10:20 compromise (Sunday full + Monday differential); the safest approach is to rebuild servers clean and restore only the data from the later backups.