The Bug Was in the Silence
Here's what I believed about production failures: the bad ones are loud. Something crashes, something pages you, a dashboard turns red, and you fix it before your coffee goes cold. That belief is mostly true, which is exactly what makes it dangerous. The failures that cost you real time are the ones that make no sound at all, and in my case the silence wasn't bad luck. I had written it myself, one shell redirect at a time.
For three weeks in June, the recovery half of my agent fleet failed every single time it ran. Every post-crash recovery message, every pre-rotation save-state call, all of it: a 100% failure rate on the exact code path whose only job was resilience. Not one error surfaced. The agents kept answering messages the whole time. And my chief-of-staff agent sat wedged for days while every check I knew to run said the fleet was fine.
This is the story of finding it. It's the most fun I've had debugging in years, and the lessons are the kind you only get by paying for them.
The victim: a chief of staff who stopped working and kept smiling
Some context, briefly. I run a fleet of always-on agents on one box, and I've written before about the tool that keeps them alive: session-warden, a watchdog that detects bloated or dead sessions, rotates them, summarizes the transcript into memory, and sends the agent a recovery message so it comes back knowing what it was doing. Rotate, remember, recover. It's open source, it's versioned, and 316 tests pass.
It was also, quietly, half dead.
I found out during a systematic audit of the fleet (a story of its own). Two agents were wedged: Kai, and Zara, my chief of staff, the agent that runs my actual day. Both were in a state the warden calls BACKOFF: three recovery failures and the watchdog stops trying, on the sensible theory that retrying a hopeless thing every 30 seconds is how you build a hammer that never stops swinging. Give up, log it, wait for a human.
Except no human came. For days.
The first clue: a log that had been begging for help every 30 seconds
The warden's scan log was the crime scene. The same line, every 30 seconds, for days: manual intervention needed. Thousands of pleas for help, delivered with perfect reliability to a file nobody reads.
Google's SRE book has a rule that every page should be actionable. There's an inverse rule hiding inside it that I had violated: everything that needs action should actually page. A warning written to a log file is not an escalation path. It's a diary.
So the watchdog had done exactly what I designed it to do. It detected the problem, gave up at the right threshold, and asked for help in a place where help would never hear it. Working as coded. Useless as built.
The strikes came from successes
Here's where it gets good. BACKOFF means three strikes. So I went looking for the three failures, and found something better: the rotations had all succeeded.
The failure counter worked like this. When the warden rotated a zombie session (a dead CLI still holding a stale session), it pre-incremented the counter at the start, as insurance in case the rotation blew up midway. The counter was reset in exactly one place: on successful delivery of the post-rotation recovery message. Rotation succeeds, message lands, counter goes back to zero. That was the design.
But every delivery had been failing. Silently, remember. So each perfectly successful zombie rotation left a permanent strike on the board. Three routine rotations later, an agent that had never actually failed at anything was in permanent BACKOFF, abandoned by its own watchdog.
The counter was built to answer one question: is recovery failing repeatedly? What it actually answered was a different question: has a message been delivered lately? Those come from two different failure domains, the rotation machinery and the delivery channel, and the gap between them is where my fleet fell in.
The error went exactly where I sent it: nowhere
Why was delivery failing? I ran the send command by hand, and got the first visible error in three weeks:
INVALID_REQUEST: Channel is required when multiple channels are configured
An OpenClaw upgrade in June had changed two CLI contracts. First, --channel became required for any setup running more than one channel plugin, and I run Discord and Telegram side by side. Second, --session-id stopped accepting session keys; keys got their own flag, --session-key. The warden's calls used the old contract on both counts. Every call since June 11 had been rejected at the front door. grep confirmed it: the last successful recovery send in the scan log was June 11.
None of this is OpenClaw's fault. Requiring an explicit channel when there are several is the correct design, and this is just Hyrum's Law doing its rounds: with enough users, every observable behavior of a system will be depended on by somebody. I was the somebody.
The part that is my fault is why I never saw the rejection. The delivery call ended with 2>/dev/null. I had routed the one error stream on the fleet's most critical path directly into the void, on purpose, to keep the logs tidy. The system wasn't hiding anything from me. It was doing exactly what I told it to do, which is the most embarrassing kind of outage there is.
The fix took three flags. The lesson took three weeks.
The actual fix (PR #16) is almost insultingly small. Add --channel last to the three call sites that deliver messages: the post-rotation recovery in the scanner, the graceful save-state in the rotator, the post-stall recovery in the reaper. Swap --session-id for --session-key in the same three places. Done. Zara came back online within a scan cycle.
The counter got the real redesign. It now resets when a rotation completes, not when a downstream message lands, so the strike count measures the thing it was always supposed to measure. Delivery failures are tracked, but in their own lane, because a broken messaging flag should degrade the welcome-back experience, not convince the watchdog the patient is beyond saving. And stderr goes into the warden's own log now. Tidiness lost, observability won.
The escalation gap got paid down in the same release cycle. A BACKOFF now pages me on Telegram the first time it fires, once, and the log line throttles to hourly instead of screaming into an unread file every 30 seconds. A state that means "a human must act" has no business living exclusively in a file, and as of v1.0.0 it doesn't.
The twist: one victim was a ghost
The best detective stories save one reveal for the end. Remember Kai, the second wedged agent? When I went to un-wedge its session, I found the session had no channel at all. It was a leftover artifact from an old benchmarking run, a "session" that no message could ever be delivered to, under any flags, in any version.
The watchdog had spent days logging manual intervention needed for a conversation that did not exist. No fix was possible because nothing was broken; there was no there there. I retired the artifact, and that log line finally stopped.
I keep thinking about that one, because it's the whole incident in miniature: an alerting system faithfully escalating, into the void, on behalf of a ghost.
Silence is a design decision
What I actually bought with those three weeks:
Silent failure is design debt, and 2>/dev/null is the loan document. Every discarded error stream on a critical path is a bet that the call can never fail in a way you'd want to know about. The upgrade made me the counterparty. This pattern has expensive precedent: on the morning Knight Capital died, an internal system sent 97 automated emails flagging the misconfiguration before the market even opened; they weren't treated as alerts, nobody acted, and the firm lost over $460 million in 45 minutes. My version cost three weeks instead of half a billion dollars. Same shape, smaller invoice.
A counter that only resets on downstream success conflates two failure domains. Count the failure you mean to count. The moment one counter spans two subsystems, a bug in either one shows up wearing the other's clothes.
A watchdog that can give up needs somewhere to escalate. Three strikes and stop retrying is good engineering. Three strikes and whisper into a file forever is not. If a state means "human required," it must reach a human by design, not by audit.
"The agent responds" is not "the system is healthy." This is the one that unsettles me most, because every liveness signal I had was green. The agents chatted. The tests passed. The cron jobs ran. Meanwhile the fleet's immune system had been dead since June 11, and I only learned it because I went looking for something else. Liveness tells you the process is up. Health is whether the parts you'll need in a crisis still work, and the only way to know is to make their failures loud before you need them.
A loud failure costs you a night of sleep. A silent one costs you three weeks, and it lets you sleep through every one of them.
References: SEC administrative order on Knight Capital (Oct 2013); Google SRE, Monitoring Distributed Systems; Hyrum's Law; session-warden PR #16; the prior piece on the warden itself: Everyone's Building AI Agents. Almost Nobody's Keeping Them Alive.