Every agent left Discord on the same morning
A fleet-wide Discord outage with two root causes, both planted by a rebuild days earlier, and a sanctioned repair tool that hangs — so the fix that worked was moving one file aside.
16 posts · tagged infra
A fleet-wide Discord outage with two root causes, both planted by a rebuild days earlier, and a sanctioned repair tool that hangs — so the fix that worked was moving one file aside.
A swap-then-build rebuild that ended green, a fleet that ran on crons for a day because nobody checked enablement state, and the dependency cascade that surfaced the moment the timers came back.
Mid-degradation is a bad time to learn how your failover actually works. A Gitea migration executed under disk-failure pressure, the four-door verification that proved it, and the backup gap the migration surfaced that three months of green dashboards never mentioned.
Three weeks of I/O wedges on a disk that passed every SMART attribute, a rebuild I chose to do as swap-then-build instead of clone, and the six stacked root causes that surfaced when the agent fleet came back up on the new install.
Gail's gateway churned for two hours while its watchdog restarted a process that was already running and missed the one that mattered. Three stacked failures, one netstat line that settled it, and a memory-offload config that only worked after the docs stopped being trusted.
Standing up dev.jordannewell.com on 2026-10-03 produced three deploy failures in 82 minutes — a PATH-shadowed .env, a security-hardened umask that left the web root unreadable to nginx, and gitleaks flagging an analytics token that ships in every page's HTML. Each failure came from something built to be on my side.
The fleet ran unattended for two months while the site sat on a splash page. Between 2026-09-25 and 2026-09-28 the deferred maintenance billed all at once — OOM loops at half the memory cap, ~800 wasted model runs a day from a default-ON heartbeat, and a billing suspension every liveness check called alive. One shared root cause: defaults and alerts that assumed someone was watching.
A Hetzner billing suspension left the VM answering pings while every real port was dark, and the first-ever mailcow restore drill that followed turned cleanup into proof the backups work.
A five-minute cron wiped the note that kept my agent fleet quiet, the default-ON heartbeat took over, and the fix came down to one config line and a rule: anything that spends money per run defaults to off.
Node 22 derives its V8 heap from the cgroup limit — heap ≈ MemoryHigh/2, measured twice. A lower cap silently shrank every heap on the fleet, and the busy agents OOM-looped.
A routine container restart on my Matrix homeserver silently broke encrypted replies across every agent on the tailnet. Agents looked healthy. None of them could talk. Here's the mechanism and the lesson.
A proactive security audit on my Matrix homeserver turned up four P0 findings. All closed. Two P1 items stay blocked on bigger migrations. The specifics stay private; the lesson doesn't.
My media pipeline silently died when a VPN exit node dropped off the tailnet. Six cascading root causes hid under each other. Five hours to find the first one.
Three days off Tailscale. Cause: dead PSU in a small-form-factor host with a proprietary power supply. Disks recovered clean. Now I decide what to build next.
Mail-auth, transport posture, web hardening, DNS integrity. The category list — not the receipt. If you're about to open a host that does mail and web on the same IP, this is the floor.
When status.jordannewell.com's upstream went down, my first two fixes cut latency from 60 seconds to 1.2 seconds. The third cut it to 170 milliseconds. I should have started with the third one.