Chapter 19 of 36 · ~1 min

Observability

When an agent fails, "the AI gave a bad answer" is not a diagnosis. Was the prompt wrong? Did retrieval return the wrong passage? Did the model call the wrong tool, or the right tool with wrong arguments? Did a tool return an error the model then paraphrased as a fact? Each has a different fix, and you can only tell which happened if you recorded the whole execution.

Trace everything: the prompt as sent, the context retrieved, the model's decision, each tool call and its result, each subsequent decision, and the final output. This is ordinary logging applied to an unusual component, and it is the difference between debugging and guessing.

A run tracePromptRetrievalDecisionTool callTool resultDecisionOutputthe tool returned an errorand the next decision paraphrased it as a factprompt as sent · context retrieved · each decision · each call and result · final output
A full trace turns "the AI gave a bad answer" into a question with an answer.

Exercise

Do it yourself

Inspect the full trace of a run that produced a wrong answer and locate the step where it went wrong.

Big question

If you could see only one thing from a failed run, what would it be?