Chapter 18 of 36 · ~1 min

Evaluation

Unit tests assert that a function returns a value. A model returns a distribution. Testing it means building a set of representative inputs, defining what a good output looks like for each, running the system many times, and measuring. That set is your evaluation dataset, and it is the most valuable artefact your project will produce.

Without it you are judging by the last three examples you happened to try. With it you can say whether a prompt change, a model swap, or a new tool made things better, worse, or just different. Evaluation is how "it seems to work" becomes "it works".

Exercise

Do it yourself

Assemble twenty test inputs for a small task, define pass criteria, and score two versions of the same agent against them.

Big question

How would you know if your agent got worse last week?