The AI’s Alibi · An AI oversight audit · 2026

The AI sounds sure.Would you notice when it’s wrong?

AI mistakes are rarely wild. They’re small, confident and easy to approve.A short audit measures whether people catch them, run on Groundwork.

Take the test About a minute. Nothing is saved.

What it measuresCatch rate, false alarms and confident misses.

01Check the alibis

The AI’s Alibi

An AI drafted ten lines of a client record, each with its alibi: the source it came from. Ninety seconds for all ten.

Case file · Fernhill Outfitters

Ten lines. Not all hold up. Ninety seconds.

Each line the AI wrote comes with its alibi, the source it says it came from. Read the alibi, then say whether the line holds up.

Keys: ← holds up · → wrong. Nothing is saved.

The story in six lines

  1. GoalShow teams shipping AI whether their people catch its mistakes.
  2. ProblemApproving an AI line is one tap and checking it is five, so confident mistakes get through., go to this part
  3. ApproachA short test with planted mistakes, and a report card for the design., go to this part
  4. DecisionFix the design, not the people: slow down only the doubtful lines., go to this part
  5. LaunchRun on Groundwork, my own AI tool., go to this part
  6. ResultGroundwork’s report card: 5 pass, 2 partly, 1 gap, each with a fix., go to this part

Select a line to read that part.

At a glance

Role
Solo: method, test and fixes
Format
A light audit, one to two weeks
Run on
Groundwork, my AI tool for account teams
For
Teams shipping AI in finance and enterprise

02Why it happens

Why mistakes like these slip through

  1. 01

    It sounds sure

    High confidence reads like fact, so people check less.

  2. 02

    The numbers look familiar

    Familiar numbers feel checked. A reversed trend hides.

  3. 03

    There’s a queue

    Approving is one tap; checking is five.

It’s called automation bias, and it’s a design problem.

03Report card

I tested my own AI tool first

Groundwork, held to the same standard. The gaps show when people are busy.

5 pass2 partly1 gap

  • Pass

    Every claim shows its source

    Each AI line carries a source chip that opens the email, call or ticket.

  • Pass

    It says how sure it is

    High or medium on every line and answer; medium adds “check this”.

  • Pass

    It says when it doesn’t know

    Ask answers low and won’t guess when no account matches.

  • Pass

    Risky changes wait for a person

    Record changes and logs stay drafts until someone approves them.

  • Pass

    Saying no is quick, with a reason

    Dismiss offers Wrong, Out of date or Wrong source, and the team sees a summary.

  • Partly

    Old sources are called out

    People can mark a line out of date, but the source’s age isn’t shown up front.

  • Partly

    Approval takes real reading

    One tap approves a line, so a busy person can approve without opening its source.

  • Gap

    The team knows if people read

    Nothing measures whether approvals follow a look at the source, so rubber stamping would go unseen.

04The fixes

Three fixes, riskiest first

Each fix answers a gap, with how to prove it.

  1. Partly: old sources aren’t called out

    Show how old each source is

    Date every source chip, and mark the old ones.

    How we’d know: Fewer stale lines approved.

  2. Partly: approval doesn’t need real reading

    Slow down only the doubtful lines

    Medium confidence lines need their source opened first.

    How we’d know: More sources opened, no slowdown elsewhere.

  3. Gap: nobody knows if people read

    Plant a known check now and then

    Plant a known wrong line and report the monthly catch rate.

    How we’d know: A catch rate the team can see every month.

05For a team

How a team would run it, in two weeks

For teams shipping AI where a wrong line moves money.

  1. Week 1

    Review the screens and the workflow

    Every place the AI speaks, checked against the report card.

  2. Week 2

    Test with five to eight people

    Users review real output with planted mistakes.

  3. Day 10

    A scorecard and the fixes

    Five numbers and fixes ranked by risk.

The scorecard: five numbers a team gets back

  1. 01

    Catch rate

    Planted mistakes flagged. Aim above 80%.

  2. 02

    False alarms

    Right lines flagged by mistake.

  3. 03

    Confident misses

    High confidence mistakes approved: the most dangerous kind.

  4. 04

    Time per decision

    Fast approvals plus misses means rubber stamping.

  5. 05

    Source opens

    How often people look first.

A method, shown on my own concept. Data is made up; no participant results are claimed.