TLDR: I added AI-generated action plans to a webinar analytics dashboard. They looked polished and confident. They were confidently wrong — because the data under them was wrong. Fix the data layer first. Always.

The Setup

I've been building a deep analytics dashboard for an ecommerce business's webinar program — attendance curves, cohort analysis, conversion funnels, a report card with letter grades, the works.

The last big feature on my list was an AI-generated action plan: look at this webinar's data, generate a ranked set of recommendations, tell the team what to do differently next time.

I shipped it on March 24th and it looked incredible.

The Problem I Didn't See Coming

Here's the thing about AI recommendations in a dashboard…

They're a confidence multiplier.

Not just for what's right — for what's wrong too.

The AI doesn't second-guess the numbers you hand it. It just synthesizes them, formats them beautifully, and delivers them with the tone of someone who has Seen Things. So when the underlying metrics are off, the output doesn't look uncertain or hedged. It looks authoritative.

That's exactly what happened.

What Was Actually Broken

By March 27th I was chasing a cluster of bugs that had been quietly poisoning the data:

  • Watch-through was measured against total webinar duration, not pitch time. So someone who watched through the entire offer presentation looked like a 40% viewer. The AI was recommending we "improve mid-session engagement" for people who'd stayed start-to-finish.

  • Sales updates were returning HTTP 200 but silently dropping rows. The save looked successful. The data was gone. (This one stings — I have a rule about HTTP 200 ≠ success and I still got caught by it.)

  • A timezone offset was shifting the viewer timeline, so content timestamps didn't line up with real viewer behavior. I had to add diagnostics just to find it.

  • The RSC cache (Next.js's React Server Component cache) was serving stale numbers after an update. I had to force a full page reload to bypass it.

Each of these on their own would be annoying. Combined? The AI was generating recommendations from metrics that had nothing to do with reality.

What Fixed It

In order:

  1. Switched to createMany explicitly — no more silent data loss on sales updates
  2. Derived contentStart from the viewer timeline labels instead of assuming a fixed offset
  3. Measured watch-through against pitch duration, not total session length
  4. Full page reload after data updates to bypass the RSC cache

Once those four things were clean, the AI recommendations actually made sense. Same prompt, same model, completely different output.

Why This Matters to Me

AI gets you 65-80% of the way there on almost anything — and that last 20% is where your judgment lives.

But in a dashboard, that 20% isn't the prose. It's the metric definitions. It's the write path. It's what happens after the API returns 200.

The AI will sound equally confident whether it's working with clean data or garbage. That's not a flaw — it's just how it works. Which means the discipline has to live in you, before the AI ever touches it.

Build the data layer right first. Then let the AI narrate.

P.S. — The action plan is genuinely useful now. Prioritized, specific, and actually tied to what happened in the webinar. Worth every debugging session to get there.