Cost, Latency, and Scale

A prototype that costs a few cents per run and takes twenty seconds is fine for one user. At ten thousand runs a day it is a budget line and a queue. Every model call has a price in tokens and a delay in seconds, and agents make many calls. Multi-step loops, long contexts, and retries multiply both.

The levers are familiar: choose a smaller model for steps that do not need a large one, cache what repeats, keep contexts short, run independent steps in parallel, cap tool calls and iterations, and set a latency budget per task before you design it. Cost per task, not cost per call, is the number to watch.