Chapter 34 of 36 · ~1 min

Break It

Now attack your own system. Give it ambiguous instructions. Feed it bad data. Plant an instruction inside a document it reads. Make a tool unavailable. Provoke a hallucination. Behave like a user who did not read the instructions. Run the most expensive path you can find. Record what happened each time.

The goal is to discover, on your own terms, why a demo and a production system are very different things. Every break you find here is one a user, or an attacker, will not.

Exercise

Do it yourself

Run the attack list against your prototype. Log each failure with its cause and a proposed fix.

Big question

Which break surprised you, and what does that say about your mental model?