The drive passed every SMART attribute for three weeks while it was dying. Sequential read settled at 24 MB/s on a disk that should triple that cold. No reallocated sectors, no pending sectors, no warnings from the tool built to warn. The throughput was the only honest signal, and throughput isn’t a SMART attribute.
That’s the first lesson and it cost a month of intermittent wedges. The box would lock up under I/O bursts — a docker build, a backup job, anything sustained — recover when the load dropped, and report perfect health the whole way through. SMART is a dashboard of self-reported claims. The drive was vouching for itself.
Stopgap, then decision
First pass was triage: park the heaviest services, kill the timers that triggered bursts, move CI elsewhere. That bought a week. It didn’t fix anything, and every day on a dying disk is a day the disk picks the failure time instead of you.
Two ways off:
| Option | What it buys | What it risks |
|---|---|---|
| Clone to new disk | Everything identical, fast cutover | Copies the rot; inherits months of config drift; one bad block poisons the copy |
| Swap-then-build | Fresh OS, clean slate, old disk preserved read-only as the archive | Everything not on the runbook’s copy list is gone until you notice |
I went swap-then-build. The old disk went into a USB enclosure, mounted read-only, demoted from system drive to reference archive. That inversion is the whole trick: the dying disk stops being a liability the moment it stops being load-bearing. Fresh install on a spare drive that had been sitting in a drawer since a decommission, SMART-clean and actually behaving like it.
The detail that made this viable: Tailscale identity is a directory. Copy /var/lib/tailscale to the new install before first boot and the box rejoins the tailnet as itself — same address, no re-enrollment, no re-auth of anything that pointed at it. Without that, swap-then-build means re-keying a dozen clients. With it, the box’s identity survives a brain transplant.
What the runbook forgot
The rebuild runbook’s copy list covered home directories, docker volumes, systemd units. It did not cover /opt — which is where every agent workspace in the fleet lives. Five hundred megabytes of working state, configured by a defaults key nobody thinks of as a path, missed by an rsync list that was correct the last time anyone checked.
Runbooks rot in the direction of the last disaster. The list was written to recover what had been lost before, not what had been added since. The old drive in its enclosure saved this from being a data loss event instead of a gotcha — mounted read-only, still had everything.
Fleet rule now: the copy list is generated by diffing the live filesystem against it, not maintained by memory.
The fleet wouldn’t boot
Fresh install, new OpenClaw version, restored workspaces. Every agent that restarted hit the same wall: exit 78, gateway.maintenance_required. Six root causes, stacked, each one hiding the next. The three that generalize:
A file-existence boot gate. The new version refuses to start while certain legacy files exist in the workspace — a pure existence check, no database consulted. The restore re-created all of them. The nasty part: agents that booted before the restore kept running, so the gate armed itself as a time bomb that detonated one agent at a time on their next restart, over days. Half the fleet was down before the pattern was legible.
A doctor that no-ops without a terminal. doctor --fix run non-interactively exits 0, prints that the database “awaits its schema repair,” and does nothing. The real migrations only run under a PTY. Success theater: the exit code says fixed, the state says otherwise. The check for “did the doctor actually do anything” has to be the database state, never the exit code.
A lease held at the last prompt. The doctor finishes its work, then parks at a final confirmation prompt indefinitely — holding the database maintenance lease. The gateway crash-loops on offline maintenance and the journal names the offending PID. The fix is killing the doctor, which is safe because the work commits before the prompt. A tool that finishes the job and then blocks the system it repaired, waiting for a keypress nobody warned you about.
Stacked causes like this are why “turning it off and on again” fails after restores. Each restart surfaces one layer, the fix for that layer reveals the next, and the whole time the box looks broken in a new way each cycle.
Standing after
The fleet came back over an evening, agent by agent, each restart now boring. The old drive stays in its enclosure until the last backup gap closes — it’s the only copy of things the runbook forgot, and I don’t delete the teacher until the exam is actually rescheduled.
The next drive won’t get three weeks. Throughput gets trended now, because the one gauge that told the truth wasn’t the one built for the job.

