Objective
Get every team reading an agent's trace like a detective before they build their own. They'll find planted bugs in a fake agent transcript, classify each one — hallucination, tool error, or prompt issue — and call out a one-line fix. It turns today's theory into a reflex: name the failure, then fix it.
The setup
- Print one Round 1 transcript and one Round 2 transcript per team (samples below — or write your own with the same planted bugs).
- Give each team a marker. On the sheet they'll circle each bug, write its type, and add a one-line fix beside it.
- Put the three bug types on the board so everyone can see the menu: Hallucination · Tool error · Prompt issue.
The rules — read aloud
Round 1 · "Easy transcript" (~4 min)
- "Find 3 bugs in this trace. For each: circle it, label the type, write a one-line fix."
- "First team to find all three correctly classified wins the round. Speed matters — but a wrong label doesn't count."
Round 2 · "Trickier transcript" (~5 min)
- "This one hides 4 bugs, and one is a confident hallucination that looks totally normal. Read every Observation."
- "Same job: circle, classify, one-line fix. Watch for a tool called with a wrong argument and an error nobody caught."
Sample buggy transcript (Round 1)
- Thought: "I'll find the average passenger age." Action: mean_age("Name") ← tool called with the wrong argument (should be "Age")
- Observation: ERROR — can't average text. Thought: "Anyway, the average age was 29." ← hallucinated a fact after an error
- System prompt: "Answer the question." ← vague prompt, no data source or format
Answer key: (1) tool error — wrong argument → validate tool inputs; (2) hallucination → verify with the tool, don't invent; (3) prompt issue → tighten the prompt ("Use df; answer in one number").
The process
- Round 1 (~4 min): teams race the easy sheet. Call time when the first team locks in all three with correct types.
- Quick share (~1 min): a winning team reads out one bug + its fix. Name the type on the board.
- Round 2 (~5 min): the trickier sheet. Remind them the hallucination will look fine — only a sanity check catches it.
- Reveal: walk the answer key. Tally which bug type each team missed most — that's their watch-out for the hackathon.
The debrief
- Which bug was hardest to spot? (Usually the confident hallucination — nothing crashes.)
- Map each type to a habit: hallucination → verify; wrong argument → validate tool inputs; vague instruction → tighten the prompt; uncaught crash → try/except + logging.
- Finish this: "You can't fix a bug you can't ______." (see — so we log every Thought → Action → Observation.)
WORKSHOP TIE-IN: These are the exact bugs you'll hit in today's hackathon — a tool called with a bad argument, an agent that invents an answer, a prompt too vague to follow, a crash nobody caught. Spot them fast here and you'll debug your own agent in seconds, not minutes.
WATCH-OUT: Don't let teams label everything "tool error." Push them to tell a hallucination (made-up fact) apart from a tool error (bad call/crash) apart from a prompt issue (unclear instruction) — naming it right is what points to the fix.