Chapter 18 of 36 · ~1 min
Evaluation
Unit tests assert that a function returns a value. A model returns a distribution. Testing it means building a set of representative inputs, defining what a good output looks like for each, running the system many times, and measuring. That set is your evaluation dataset, and it is the most valuable artefact your project will produce.
Without it you are judging by the last three examples you happened to try. With it you can say whether a prompt change, a model swap, or a new tool made things better, worse, or just different. Evaluation is how "it seems to work" becomes "it works".
Exercise
Do it yourselfAssemble twenty test inputs for a small task, define pass criteria, and score two versions of the same agent against them.
Big question
How would you know if your agent got worse last week?
