/infra

16 posts · tagged infra

Moving Gitea off a dying disk: verify all four doors

Mid-degradation is a bad time to learn how your failover actually works. A Gitea migration executed under disk-failure pressure, the four-door verification that proved it, and the backup gap the migration surfaced that three months of green dashboards never mentioned.

Two hours of outage the watchdog watched happen

Gail's gateway churned for two hours while its watchdog restarted a process that was already running and missed the one that mattered. Three stacked failures, one netstat line that settled it, and a memory-offload config that only worked after the docs stopped being trusted.

Three incidents, four days, one root cause: the unattended fleet's September bill

The fleet ran unattended for two months while the site sat on a splash page. Between 2026-09-25 and 2026-09-28 the deferred maintenance billed all at once — OOM loops at half the memory cap, ~800 wasted model runs a day from a default-ON heartbeat, and a billing suspension every liveness check called alive. One shared root cause: defaults and alerts that assumed someone was watching.