What it costs, and how you know it is working

At the end of the first month Nadia possessed two figures she had never had before. The first was the cost of a completed quotation. The ledger kept alongside every run had priced each model call according to the model that served it and the tokens it consumed, and had totalled them by workflow. In model time a quotation cost less than a cup of coffee, and the figure came from a table rather than from an estimate. The second was the proportion of the month's enquiries that had required a person at all. It was about one in five, almost all of them requests for discounts above policy.

A ceiling and a ledger

The cost figure is worth having only because it cannot run away. Every model step in the workflow is subject to a ceiling on the number of times it may repeat its loop, the number of tools it may call, the number of tokens it may consume and the time it may take. A step that reaches its ceiling stops cleanly, and the process decides what follows, which is usually to refer the matter to a person. A conversation about budget therefore begins from a ledger and a set of limits rather than from apprehension, and it reduces to two questions: which steps account for the spending, and whether a cheaper model at one of them would serve. The second question is answered by changing a single setting and reading the following month's ledger.

Runs, not demos

The harder question is whether the process is working, and the honest answer is not to be found in a demo but in the runs themselves. Nadia read them, not all forty-one but those the desk had flagged, and kept the enquiries that had surprised the workflow: the one containing the planted instruction, the one from a country the rule did not recognise, and the one in which the buyer wrote back with a question for which the workflow had no step. These became the test set against which every subsequent change was run, and the rule governing changes became a simple one: one step at a time, on the evidence of the runs, with each change recorded against the version of the workflow that made it.

Three numbers

Three figures proved sufficient to manage the desk thereafter: the number of quotations completed, the cost of each, and the share that had required a representative. Taken together they indicate whether the process is doing more work, whether it is doing so economically, and whether the line between rules, models and people has been drawn in the right place. When the board later asked about adoption, the relevant measure was not the number of workflows Kettle Co. possessed but the number of quotations that had gone out on the day the enquiry arrived.