TLDR: Two layers, one store. Deterministic for instant. Batch LLM for generalization. Never re-run the pipeline on every thumb.

The Setup

Apollo (my personal AI assistant dashboard, lives in my Dock) surfaces email and Slack threads every morning.

The volume is manageable. The noise is not.

Too many senders I always skip. Too many threads I've seen a dozen times. I needed the dashboard to learn — not just fetch.

So I wired in a feedback loop: thumbs up/down on every surface item.

In theory: one tap teaches it. In practice, I had to get the architecture right before it was worth anything.

The Trap I Almost Fell Into

The obvious instinct is: user presses 👍 → re-run the LLM with the new signal → surface updates immediately.

Don't do that.

Re-running the whole ingestion pipeline on every click is slow, expensive (the dashboard uses metered Haiku after June 15 — every re-ingest costs REAL MONEY), and spammy as hell.

I wanted instant feedback. I almost paid for it the dumbest possible way.

The Fix That Actually Works

Two layers. One persisted store.

  • Instant deterministic layer — applies the signal at render time with plain code. No model call. No cost. The item hides, demotes, or floats immediately because the read path checks the store and acts on it.
  • Slow generalization layer — the LLM reads the SAME store on its next batch run and generalizes to new/similar cases it hasn't seen yet.

Immediacy comes from determinism, not from model calls. That's the whole insight.

One more thing: write-on-click atomically. I use tmp + os.replace() so a crash or a closed lid never corrupts the signal. Idempotent, reversible — undo retracts the lesson, not just the UI state.

The Wall I Hit After Shipping

A feedback loop you can't inspect is just noise you're generating blind.

So I built /rules — a page that shows what the system has actually learned. Inspect it, understand it, delete individual rules before the next batch run.

First load: email rules showed readable sender addresses. Slack rules showed u07p1ern533.

Completely opaque. No idea whose feedback I was reading.

The fix: a _name_map() call in slack_feedback that resolves each from_id → display name via the ingest's existing state/slack_users.json user cache.

One gotcha I did not see coming: the cache keys are UPPERCASE. Feedback keys are lowercase. Case-insensitive lookup is not optional here — it's required, or the map comes back empty.

I also surfaced the example message text alongside each rule (Slack prefs already store the snippet in examples). The rules page now shows name primary, raw ID muted secondary, and a reason-or-example context line. Email got the same treatment for parity.

The Reset Button

I added a "Clear all rules" button — confirm-gated red button, no accidental wipes.

Then immediately used it.

Cleared both stores back to zero: threads, rows, prefs, sender rules, topics — all gone. Starting the training over from scratch with eyes open this time.

You can only trust a feedback loop you can inspect AND reset.

Why This Matters

Every AI surface eventually becomes a trust problem.

The default instinct is "re-run the model" because it feels smart. But smart and fast and cheap aren't the same thing. The deterministic layer is the move — instant behavior for free, model budget only where it compounds: generalization at batch time.

If you're building anything with a feedback signal, wire the inspect surface before you wire the model call. A loop you can't see into isn't training data. It's entropy with a UI on top.

P.S. When you ship a two-layer loop, tell your user plainly which effects are instant vs. which need the next batch run. An unchanged surface that's "waiting for the batch" reads as broken — unless you say so.