Multiply 0.95 by itself ten times and you get 0.599. An agent that gets each step right 95 times in 100 finishes a ten-step task about 60 times in 100, and 95 percent per step is a number most teams would be pleased with.
I think this explains what happened to the autonomous agents of spring 2023 better than any complaint about the models. Auto-GPT and the projects around it let a model plan its own steps and run them in a ReAct-style loop until it declared itself done. Any single step looked reasonable. Thirty in a row rarely landed.
The multiplication, written out
Assume steps fail independently. That’s generous, since a wrong step also hands bad input to the next one. End-to-end success is then per-step accuracy raised to the number of steps.
| Per-step accuracy | 5 steps | 10 steps | 20 steps |
|---|---|---|---|
| 90% | 59% | 35% | 12% |
| 95% | 77% | 60% | 36% |
| 98% | 90% | 82% | 67% |
| 99% | 95% | 90% | 82% |
Read along the 95 row and doubling the steps takes you from 60 to 36. Read down the ten-step column and the jump you need, from 60 to 90, means going from 95 to 99 per step. That’s a five-fold cut in the error rate, on every step at once.
Prompt edits run out before 99
The first rewrite of a weak prompt fixes the common failure. What’s left afterward is the odd input: the scanned invoice with a rotated table, the vendor whose name shows up twice with different spellings. Each needs its own fix, and a new model version reshuffles the set.
So I’d stop treating a 60 percent run as a wording problem. There are two numbers in that formula, and the exponent is the one an engineer controls.
Four changes that aren’t in the prompt
Start by taking steps away from the model. Anything code can do exactly, code should do. Function calling made it easy to hand the model a fetch tool and then let it decide when to fetch, which turns a call that never fails into one that fails 3 percent of the time. Every step moved to code drops out of the exponent.
Then check every step the model still owns, with code. I mean a deterministic check. A second model’s opinion doesn’t count. Do the line totals sum to the invoice total? Does the purchase order number exist? When a check fails, retry once and include the failure message. It’s the idea I described for tool arguments, applied to a whole step’s output.
Save state after each verified step, so a failure at step eight resumes at step eight. Rerunning one to seven means rolling the dice on them again. LangGraph’s checkpointers do this, and the library reached a stable 0.1 in June. A table keyed by run ID and step number does it too, if you own the loop.
The last change is a person in front of anything that can’t be undone. LangGraph calls this an interrupt: the graph pauses before a named node, the state waits in the checkpoint store, and the run continues when someone approves. The gate depends on the checkpoint. Without saved state, an approval that arrives three hours later has nothing to resume.
Laid end to end, a run goes from planner to executor to verifier, into the checkpoint store, and through an approval gate before the irreversible action. A run log sits at the far end.
An invoice agent stuck at 58 percent
Wexcombe Procurement is fictional. I picked its success rates so the multiplication stays readable.
Wexcombe runs a ten-step agent that matches supplier invoices to purchase orders and posts them to the ledger. The model decides every step. Per-step success sits between 94 and 99 percent, and over 300 runs the agent finishes correctly 174 times. That’s 58 percent.
The team fills in a step table and finds four steps that never needed a model: fetching the invoice, looking up the purchase order, the tolerance arithmetic, and the posting call. Those go to code. The six that remain each get a verifier and one retry.
# One entry per step. Fill it in before touching a prompt.
run: invoice-match
steps:
- step: fetch_invoice
decided_by: code
verifier: file opens and has at least one page
reversible: true
checkpoint: true
approval: none
- step: extract_header
decided_by: model
verifier: invoice number, date and total present and well formed
reversible: true
checkpoint: true
approval: none
- step: extract_lines
decided_by: model
verifier: line totals sum to the invoice total
reversible: true
checkpoint: true
approval: none
- step: resolve_vendor
decided_by: model
verifier: vendor ID exists and is active
reversible: true
checkpoint: true
approval: none
- step: lookup_purchase_order
decided_by: code
verifier: purchase order exists for that vendor
reversible: true
checkpoint: true
approval: none
- step: match_lines
decided_by: model
verifier: every invoice line maps to one open order line
reversible: true
checkpoint: true
approval: none
- step: check_tolerance
decided_by: code
verifier: none needed, plain arithmetic with unit tests
reversible: true
checkpoint: true
approval: none
- step: classify_exception
decided_by: model
verifier: reason code is in the allowed list
reversible: true
checkpoint: true
approval: none
- step: draft_posting
decided_by: model
verifier: debits equal credits and accounts exist
reversible: true
checkpoint: true
approval: none
- step: post_to_ledger
decided_by: code
verifier: ledger returns a posting ID
reversible: false
checkpoint: true
approval: required above 10000 USD
With verifiers and a retry, the six model steps come out at about 98.5 percent each. Six steps at 98.5 is 91 percent, so 273 good runs in 300 where there were 174. Postings above $10,000 wait for a person.
Where the retry stops helping
A verifier only catches what it can check. If a 94 percent step had a perfect verifier, one retry would lift it past 99.6. Wexcombe gets 98.5 because some wrong answers pass: line items that sum correctly but are matched to the wrong order line get through the total check.
That’s what the run log is for. Every verifier failure and every retry goes into it with the step name, and after a few hundred runs it shows which step is eating the retries. That step gets a stricter check or a move to code.
In Wexcombe’s final table, no step pairs decided_by: model with reversible: false. I wouldn’t ship an agent whose table has that pairing anywhere.