TLDR: If your background daemon alerts when an error counter crosses a threshold, it'll oscillate silently. Latch a flag the moment you cross it — and don't clear that flag until you get a clean success.

The Setup

I built Apollo (my personal AI agent, runs hourly via launchd on my Mac) a "watches" system — little self-managing checks I can register in one CLI call.

Each watch has its own cadence, a term date it auto-retires on, and one job: run a check, and if the condition resolves, send me an iMessage and quietly disappear.

Simple enough. Until I handed it to Grok 4.5 for an adversarial review.

The Problem With Naive Alerting

The original logic was: once consecutive_errors hits three, fire an alert.

Sounds right. But watch what actually happens with a flaky check.

Tick 1: fails → errors=1. Tick 2: fails → errors=2. Tick 3: coincidental success → errors=0. Tick 4: fails → errors=1 again.

The counter bounces around the threshold and never stays at three. No alert fires. The watch is chronically broken, and I hear nothing.

This isn't a theoretical concern — any real-world check against an external system is going to have the occasional pass in the middle of a longer failure run. That's exactly when silent oscillation bites you.

What Grok Found (And What I Shipped)

Grok 4.5 flagged this as one of four defects in the watches framework. All four were real. All four shipped the same day as atomic commits + regression tests in test_watches.py.

The other three were nasty in their own right — one malformed watch was aborting the entire hourly tick (a KeyError that escaped before save_state()), and the Things delivery channel was completely broken due to a raw newline inside an AppleScript string literal. But the error-alerting oscillation was the kind of bug that just quietly costs you.

The Fix: Latch It

Once consecutive_errors >= ERROR_ALERT_THRESHOLD (3), set alert_pending = True.

Then leave it set until a genuine clean success clears it.

Alert fires exactly once. The "Active Watches" dashboard card stays red for the duration. No re-alerting on every bounce. No false-clear when a coincidental tick succeeds. If even the iMessage alert send fails (say, iMessage is down), the dashboard is the backstop.

One flag. One direction of travel until you earn your way clear.

Where It Still Hides

I have to be honest here — the latch fixed chronic check failures, which is the common case.

But there's a subtler oscillation on the notify path that I left alone.

When a watch resolves but the notification delivery fails, _process_watch zeroes out consecutive_errors (clean execution) before _register_error has a chance to bump it back up. So that error counter runs 0→1→0 every tick and never reaches three. A watch that can't deliver — iMessage down for days — retries forever, silently.

I flagged it, marked it narrow + pre-existing, and consciously left it. Candidate fix when I get back to it: track a separate notify-failure streak, or don't zero the counter when a resolve-notify fails.

Why This Matters to Me

The latch is obvious in hindsight. What wasn't obvious is the constraint underneath it: a latch is only as good as the counter feeding it. If anything zeroes your error count before the failure is registered, the threshold becomes unreachable — and the latch never engages.

Order-of-operations around your error state matters as much as the alerting logic on top of it. Check the sequence, not just the rule.

P.S. Grok 4.5 running as an adversarial advisor — grok -p headless — is increasingly earning its place in my review stack. Finding four real defects in a framework I'd already put through two rounds of advisor + Gemini is not nothing.