TLDR: When you batch-process a registry of inputs, one malformed item will eventually reach production. If you don't isolate it, it aborts everything after it — silently. Isolating without surfacing just trades a loud crash for a quiet rot. You need both.

What I Built

Apollo Watches is a lightweight dispatch system I run locally — an hourly launchd (macOS's background-job scheduler) daemon called com.apollo.watches that reads a YAML registry of small self-retiring checks and runs the ones that are due.

Think of each "watch" as a tiny standing question: "Has the Apple Watch Ultra 4 event date been announced yet?" The dispatcher wakes up every hour, checks what's due, runs the relevant checks, and if a condition resolves, it notifies me and retires that watch. Clean, minimal, no babysitting.

It passed every test I wrote for it.

That was the problem.

The Wall I Hit

I brought in Grok — xAI's model, which I can invoke headless from the terminal via grok -p — to do an adversarial code review.

Grok found it immediately: if any single watch entry in the registry was malformed — bad YAML, missing required field, wrong type — the tick threw an exception and stopped cold. Every watch that came after it in the loop? Never ran. No error message surfaced to me. The dispatcher just exited cleanly with a non-zero code that nobody was watching.

My tests all passed because I wrote them with well-formed fixtures. I never once threw a bad entry at the loop.

I gave myself false confidence by only testing the happy path.

The Two Fixes (In Order — Order Matters)

Fix one: per-item isolation. Wrap each watch's run in its own try/except. A malformed or crashing watch logs its failure and moves on. The rest of the tick runs. This is the obvious half.

Fix two: surface it — don't let it rot silently. This is the part I almost skipped.

Isolation alone just trades a loud abort for a quiet one. If a watch fails on every tick and I never know, I've just buried the problem deeper. So: after three consecutive errors on the same watch, the dispatcher latches an alert_pending flag and fires me a notification. It stays red on my Active Watches dashboard panel until a clean success comes through. The failure can't accumulate silently anymore.

Isolation without surfacing is just hiding the wreckage.

Why This Generalizes

I've hit the same shape elsewhere. In a batch LLM pipeline over 65 transcript files, one outlier — a 278KB blob file that hit an API rate limit — would crash the whole run. The lesson there was the same: re-spawn just the failed slice, split smaller. Don't redo the whole run. The healthy items shouldn't pay for the sick one.

One bad input in a batch is not an edge case. It's a guarantee, eventually. Design for it from the start.

The Part I Haven't Closed Yet

Worth being honest: isolation + surfacing covers check failures. What it doesn't cover yet is a chronic notify failure — where a watch resolves correctly but can't deliver the notification (say, iMessage is down for days). That watch would retry forever, silently. Different shape, same rot. I know it's there. It's on the list.

Build the isolation first. Then build the surface. Then go find what you still haven't closed.