The most dangerous failure mode in a production system is not noise. It is silence. A component that crashes pages someone at 3am. A component that was never scheduled to run in the first place produces no error, no alert, and no log line to debug. It simply is not there, and every dashboard reports green.
We shipped one. A customer deployment ran for several weeks with most of its scheduled quality checks absent: the code was present in the build, the schedules that should have fired it were not. There was no failure to investigate because nothing failed. The work just never happened.
This is a post about what made the recovery cheap, because the recovery is the part that generalises.
What actually happened
The original specification called for a deterministic chain of background quality passes, coordinated by a scheduler, each running on a defined cadence. Weeks later the architecture moved on: at our own site that chain was replaced by a different orchestration model, and the original specification was formally superseded. The supersession was logged, dated and cross-referenced. Everyone moved on, correctly.
The replacement orchestration was built for our own environment and assumes things only our environment has. A customer deployment has a gateway, its knowledge stores, and a small set of services. When we shipped, we shipped the quality passes themselves but no component to fire them on a schedule. One of them carried its own internal timer and therefore ran. The rest waited for an instruction that was never going to arrive.
The gap was not hidden and it was not subtle. It was written down, in documents we had authored ourselves, months before the symptom appeared. Nobody read them, because nothing had gone wrong.
The temptation
Two fixes were available inside an hour of the report landing.
The first was to bolt schedulers into the gateway runtime, copying the pattern from the one component that already had a timer. It would have worked by the end of the afternoon. It would also have welded orchestration to the gateway runtime, which the architecture specification explicitly told us not to do. The supersession we had logged existed precisely to keep those concerns apart.
The second was to extend the orchestration we already ran at our own site to cover the missing passes. It would have closed the gap where we could see it and widened the gap where it mattered, because no customer runs our internal environment.
Both were fast. Both came from wanting to fix it for the person waiting on the other end. That instinct is not the wrong one, and we will feel it again. It was simply the wrong instinct in a moment where we had already written down the answer.
We did neither. We read the documents.
What the documents said
The architecture specification was precise about the cadences: which pass runs, how often, and at what time. Each was a scheduled call to an authenticated endpoint. The passes were already implemented. What was missing was the thing that fires them, in a place that exists at every deployment.
The decision wrote itself. Build a small, separate orchestrator. Ship it everywhere, ours and every customer's. Make the schedules configuration rather than code, so changing a cadence is one command and no rebuild. Match the cadences the specification already prescribed.
It was built the same afternoon: Alpine, bash and curl, because the orchestrator does exactly one thing and should be impossible to misconfigure. One scheduled job per pass, each with its own cron expression. The schedule lives on the job. The endpoint lives in an environment variable. The image carries no business logic, only the discipline to make one authenticated call and exit cleanly.
Deployed to our own environment first. The first pass completed end to end in 218 seconds, classified 84 entries, labelled 20 with confidence grades, put two forward for promotion and rejected 17. Deployed to the customer tenant next: same image, same shape, different endpoint and different secret path. Every pass ran end to end the same evening, and the quality layer was doing its job for the first time since that deployment went live.
When the symptom arrived we did not have to invent a response. We had to find one. Reading replaced thinking from scratch, and the cheat code is not a hack: it is the cost of documentation, paid in advance.
Why the recovery was cheap
The fix took hours. Not because of heroics, and not because anyone worked through the night, but because every step it required had been decided and written down before it became urgent. The cadences were specified. The supersession was logged with its date and its reason. The shape of the missing component, what it should and should not contain, was already implicit in the same documents.
There is a common pattern in how organisations adopt AI. Move fast, wire something together, ship it. Then a problem arrives that the wiring never anticipated and there is no canonical artefact to consult. The team reasons from scratch, under pressure, in real time, often via whoever is available rather than whoever knows. The fix contradicts a decision nobody remembered making. Six months later the contradictions have compounded and the system is unmaintainable in ways nobody chose.
Boyd called the second phase of the loop Orient: placing new information in the context of everything you already know. It is the phase organisations starve, because it produces no visible output. An organisation that has documented its decisions has a fast Orient phase and short Decide and Act phases behind it. An organisation that has not pays the difference at every surprise, and pays it in the worst currency, which is time under pressure.
Documentation is not bureaucracy and it is not cleanup. It is the substrate that lets a team move fast when something unexpected arrives, and it has to exist before the surprise, because that is the only time it can be written calmly.
What we changed
Two things came out of this, and only one of them is the orchestrator.
The first is the component itself, now a standard part of every deployment rather than something our own environment happened to provide.
The second is a verification step at the deployment boundary. Scheduled-work telemetry now has to appear within 24 hours of an install for that install to count as complete. A system that reports nothing is no longer read as a system with nothing to report. That step is written down too, which is the only reason it will still be true in six months.
The general lesson is about what monitoring is for. Most monitoring is built to catch things that go wrong. This failure went undetected because nothing went wrong: the absence of work generates no signal at all. Designing for silence means asserting that expected work actually happened, not waiting to be told that it failed. If you run scheduled work of any kind, the question worth asking this week is not whether it is erroring. It is how you would know if it had simply stopped.
The side that loops fastest wins. The side that documents every loop catches itself when it stumbles.