TLDR: Any daemon that does side-effect-then-mark-done has a crash window. Ordering alone can't save you. Write a durable marker the moment the side effect confirms, and check it before you ever re-attempt.

The Setup

I built something I'm calling Apollo Watches — a framework that lets me register a simple one-liner check ("watch") for any condition I care about, and have an hourly launchd daemon (macOS's job scheduler) run it, then notify me the instant it resolves and retire itself.

First watch: is there an Apple Watch Ultra 4 event date yet?

The core dispatcher loop felt clean. Run the due watches, evaluate, and — critically — notify first, retire only if the send confirmed. That "notify-then-retire" ordering was deliberate. If iMessage goes down, the watch stays alive and retries next tick. You get at least once delivery.

That felt solid. It wasn't.

The Wall

I was thinking about crash paths when it hit me.

The notifier shells out, gets a real return code back, and only then does the dispatcher flip the watch to retired in watches_state.json. Between those two moments — send confirmed, file written — there is a window.

Crash there, and launchd relaunches the daemon next hour. The watch reads as still-active. The check re-runs, resolves, and... it notifies again.

Notify-then-retire ordering got me guaranteed delivery. But it did nothing to prevent double delivery on a crash-restart. Those are two different problems, and I'd only solved one.

The Fix That Worked

A durable delivered-marker, written to watches_state.json immediately after the notify rc confirms — before anything else. The retire step reads it. A crash anywhere after that marker lands means the next tick sees "already delivered" and skips.

Sequence is now: perform → confirm (real rc) → persist marker → retire.

On restart: check marker first, always, before re-evaluating.

The detail that makes this work is that the notifier returns a real return code — unlike some of the other send paths in Apollo that return None and make you guess. The marker is only written when there's actual proof the notification landed.

(I caught two other defects the same session — a malformed watch that could abort the whole tick, and a Things 3 newline bug silently killing that notification channel. Different lessons, different fixes. But this one I found myself, thinking through crash paths.)

Why This Matters to Me

It's easy to feel like you've solved the reliability problem once you've got the right ordering. Notify before retire. That's the right answer to "what if the notify fails."

But ordering doesn't protect you against the crash window between a confirmed side effect and a persisted acknowledgment. And a user-visible side effect — a notification, a charge, an email — is never idempotent just because it happened once.

Persist the durable marker the instant you have confirmation. Check it before you ever re-attempt. That's the move.

P.S. The Apollo Watches framework itself is in a personal RAG pipeline/src/apollo_rag/watches/. The CLI is apollo-watch add | list | show | retire | run <id>. If you're building something similar: the design decision to split watches.yaml (human-authored defs) from watches_state.json (dispatcher-only state) is worth stealing — software state-churn should never race a hand edit.