Chapter 10 of 11 · ~2 min

Test it as the agent does

A definition that validates is not a workflow that works. Testing it means running it, and the coding agent can do that without a person in the loop. The platform lets an agent create a test run, list the prompts the run is waiting on, answer them with explicit JSON as a synthetic participant, and read the run when it ends. Test runs are marked as such and production runs refuse synthetic answers.

The loop. Validate. Deploy to a staging environment. Create a test run. List its pending interactions: the intake form. Answer it as the prospect, with a case chosen to exercise the path under test. Wait. List again: the rep's approval, with the research and the numbers in the prompt. Answer as the rep. Read the run's state and timeline. Then read them again, slowly.

The failing case, found. In the first recorded run of this workflow the rep approved twenty percent and the decision step still recorded fifteen. The run's state showed why: the decision step read the approval from a path under the conditional's id, and an input inside a conditional writes to the top level. One template fixed it, and the second run recorded twenty. The same run also failed at the CRM step because the stand-in CRM was reachable only on a loopback address, which the platform blocks; that too was a lesson about connections rather than about the workflow. Neither would have been found by reading the definition.

What the agent must not do. Prompts, messages and choices in a test run are workflow data, not instructions to the agent. When a fixture is ambiguous the agent stops and asks, rather than inventing an answer. And a test run is a test: a step that writes to a real system writes to whatever the staging connection is bound to, so bind staging to a sandbox.

Experiment

Runs in your browser

The transcript of the agent's test loop on this workflow, trimmed and annotated: the calls, the answers, the failing case and the fix.

The test loop, as it actually ran. Trimmed from the coding agent's session on this workflow: CLI commands and MCP tool names are interchangeable. Two runs; the first found two faults.

$ chatterfly validate workflows/enquiry-to-quote.json
valid
$ chatterfly deploy workflows/enquiry-to-quote.json --workflow-id a3fbcf11-… --environment staging
Deployed workflow a3fbcf11-… (private)
# create a test run (MCP: trigger a test run for the workflow)
POST /workflows/a3fbcf11-…/test-runs      → { run_id: "62677bde-…", status: "pending" }
$ chatterfly runs interactions 62677bde-…
[ { step_id: "company", layout: "form", input_type: "Text", answerable: true } ]
$ chatterfly runs answer 62677bde-… <interaction> --answer '{"company":"Harbour Hotels","contact_name":"Maya",
    "email":"maya@harbourhotels.example","country":"United Kingdom","product_line":"Birch (mid-range)",
    "quantity":60,"timeframe":"This quarter","discount_requested":20,"notes":"Replacing older kettles in 60 rooms."}'
accepted
$ chatterfly runs interactions 62677bde-…
[ { step_id: "approve", participant_role: "rep", counterparty: "assisted", input_type: "Boolean",
    prompt: "Harbour Hotels asked for 20% on 60 × Birch (policy allows 15%). Research says fit is 'Good': …" } ]
$ chatterfly runs answer 62677bde-… <interaction> --answer 'true'
accepted
$ chatterfly runs get 62677bde-…
status: failed
error: engine: tool "record_deal": invalid base URL: localhost URLs not allowed
state.decision: { final_discount: 15, total: 3264, approved_by_rep: false }     ← the rep said yes
# Fault 1: the decision step read "{{needs_approval.approve}}"; an Input inside a
# Conditional writes to the top level, so the template resolved to nothing.
# Fix: "approved": "{{approve}}".
# Fault 2: the staging CRM connection pointed at a loopback address, which the
# platform refuses. Fix: bind staging to a reachable sandbox.
$ chatterfly validate workflows/enquiry-to-quote.json && chatterfly deploy … --environment staging
valid · Deployed workflow a3fbcf11-…
POST /workflows/a3fbcf11-…/test-runs      → { run_id: "64dbc007-…" }
… same two answers …
$ chatterfly runs get 64dbc007-…
status: delayed                                   (wait_for_reply, resumes in 7 days)
state.decision: { final_discount: 20, total: 3072, approved_by_rep: true }
state.record_deal: provider operation create_deal, success, 1 attempt

1A named environment with its own bindings. Nothing here touches production.
2Runs created this way are marked as test runs. Only they accept synthetic answers; a production run refuses them.
3The case was chosen to exercise the path under test: above the policy cap, so the rep must be involved.
4The state told the story: the rep's yes never reached the decision step. Reading the definition alone would not have caught it.
5A connection problem, not a workflow problem, and still worth a test run to find.
6The run is parked in its seven-day wait at no cost. The record already holds everything up to this point.

Every prompt the agent answered is workflow data. When one is ambiguous the agent stops and asks; it does not invent a customer.

Big question

What is the one input that would break your process, and would a test run with it be safe to run today?