The agent’s pull request goes red again, and someone in the thread asks whether it’s time to try the bigger model.
It’s a fair question with an expensive answer. A coding agent is a model running in a loop, and you rent both. What you own is everything they touch in your repo, which I’ll call the harness: the instructions file at the root, the commands the agent may run, the context it’s handed up front, the checks between its output and a pull request, and the rule for when it gives up. That last one is the easiest to forget. You can change any of them this afternoon.
The trouble is that teams change the harness by feel. It’s an easy pattern to fall into: after a bad PR, someone adds a paragraph to AGENTS.md, and a few months later the file runs to hundreds of lines and nobody knows which edits helped. Switching models starts to look like the only real decision, because it’s the only one with a price attached.
The harness deserves what code gets: a test suite, and one lever changed at a time.
A test suite made of merged PRs
The suite is a replay set: 20 to 30 pull requests from your own history, turned back into tasks. Each task is the issue text plus the commit on main just before the PR merged. The agent works as usual and passes if the repo’s tests pass, plus the tests the PR added or changed, which it never sees. They’re the answer key.
It’s a golden set, really, with pull requests as the inputs and your own test suite as the grader. A public benchmark grades an agent on someone else’s code. Your merged PRs grade it on your build and the conventions nobody wrote down, which is where agents trip.
Pick PRs a capable new hire could have done from the issue alone, not ones whose real spec lived in a meeting or that need a live third-party service. Read their tests, too. If one calls a helper by a name nothing in the issue gives away, put the name in the prompt or drop the task. The hidden tests should fail on the old commit and pass on the merged one. Hold back eight or so more as a holdout set, and leave it alone while you tune.
Run every task three times and average, because agents rarely take the same path twice. Before the first change, run the unchanged harness twice. The gap between those runs is your noise floor, and on 25 tasks it can be a whole task.
Five levers and a changelog
So the harness has five levers: instructions, commands, context, checks, and stopping rules. The model is a sixth, and the only one outside your repo. In the diagram, commands are the loop’s tools, and a replay task joins the line where a real issue does, so it passes all five.
Change one lever at a time, because they interact. A clearer test command makes the agent run tests more often, which changes how hard a rule about repeated failures bites. Change two together and you learn something worked, but not what.
Each change gets a row in a changelog kept next to AGENTS.md: what changed, which lever, the resulting pass rate, cost per task and wall time, and what the holdout said. Each row is judged against the one above, so four changes that each quietly cost half a task can all pass for noise and still leave you two tasks behind. Every few rows, rerun the starting harness beside the current one.
An AGENTS.md short enough to test
This one is for a hypothetical Python service. Keep a line only if the replay set shows it helps.
# AGENTS.md
## Build and test
- `make test` installs dependencies and runs every test. Run it from the repo root.
- `make test T=tests/core/test_billing.py` runs one file. Use it while you work.
- `make check` runs the linter and the type checker.
- Don't call pytest, pip or docker directly. If a target is missing, say so.
## Where things live
- `app/api/` HTTP handlers, one module per resource. No business rules here.
- `app/core/` Business rules. Never imports from `app/api/`.
- `app/db/` Queries and models. `app/db/migrations/` is append-only.
- `app/jobs/` Scheduled jobs. Every job must be safe to rerun.
- `tests/` Mirrors `app/`. `app/core/billing.py` is tested in `tests/core/test_billing.py`.
- `scripts/` One-off tools, not part of the build. Ignore them.
## Conventions
- Raise errors defined in `app/core/errors.py`, never a bare `Exception`.
- Money is an integer number of cents. No floats for amounts.
- Every new function in `app/core/` gets a test.
- Match the style of the file you're editing.
## Never
- Edit a migration that has been merged. Add a new one.
- Touch CI config or any `.env` file.
- Skip, delete, or loosen a failing test to get a green run.
- Add a dependency, or edit the lockfile, without saying why in the PR description.
## Done means
- `make test` and `make check` both pass.
- The PR description says what changed, why, and what you chose not to do.
- If the same failure comes back after three attempts, stop and write up
what you tried instead of opening a PR.
That last bullet is a stopping rule. Without one, a stuck agent spends until the timeout and then opens a confident PR built on a wrong guess.
One replay task and the runner
A task file, for an invented PR:
# replay/tasks/pr-1482.yaml
id: pr-1482
set: replay # or: holdout
base_commit: 4e8a1c9d2f7b6a3e5c0d1f2a9b8e7c6d5a4f3b2e # main just before the merge
merge_commit: b71f0e3c9a2d8e6f4b5c1a7d3e9f2b8c6a0d4e1f # hidden tests come from here
prompt: |
Invoices issued before a customer changes their billing address now
show the new address when the PDF is regenerated. They should keep
the address that was on file on the issue date.
hidden_tests: # every test file the PR added or changed
- tests/core/test_invoice_address.py
- tests/api/test_invoices.py
budget_usd: 5.00
timeout_min: 20
The runner is a short script:
run-replay --harness <dir> --set replay|holdout
for each task file in replay/tasks/ whose set matches:
repeat 3 times:
1. Copy the tree at base_commit into a fresh, throwaway container,
with no history after it (a shallow clone of base_commit, or no
.git at all) and no network route to the code host.
2. Copy the harness under test on top: AGENTS.md, agent config,
and the Makefile targets it relies on.
3. Run the agent on the prompt. Stop it at budget_usd or timeout_min.
Hitting a limit, or stopping under its own rule, is a fail.
4. Put tests/ and the test config back as they were at base_commit,
delete any new files under tests/, then copy in hidden_tests,
fetched from merge_commit outside the container. The agent can't
pass by editing a test, or by reading the answer.
5. Run the full suite. The attempt passes only if everything is green.
6. Record pass or fail, dollars, minutes, and the transcript.
print: pass rate, cost per task, wall time, passes per dollar,
and the difference from the last changelog row
Steps 1 and 2 are easy to get wrong. A full clone carries the merged fix in its history, and an agent with git will find it. An old commit has an old AGENTS.md, or none, so the harness under test goes on top. Keep the PRs recent, too, since today’s Makefile may not build last year’s tree.
Nine tasks out of twenty-five
To show how the changelog reads, imagine a team whose agent starts at 9 of 25 replay tasks. Each row is one change:
| Change | Lever | Pass rate | Cost per task | Wall time | Passes per $ | Holdout |
|---|---|---|---|---|---|---|
| Starting point | none | 36% | $1.10 | 11 min | 0.33 | 38% |
One make test target |
commands | 36% → 52% | $1.05 | 9 min | 0.50 | +17 pts, holds |
| 30-line “where things live” map | instructions | 52% → 60% | $1.10 | 8 min | 0.55 | +8 pts, holds |
| Type check before done | checks | 60% → 68% | $1.25 | 10 min | 0.54 | +4 pts, about half |
| Model two tiers larger | model | 68% → 72% | $3.75 | 13 min | 0.19 | +4 pts |
The make test target alone was worth four tasks, mostly because the agent stopped guessing at test commands. The three harness changes added eight tasks for fifteen cents more per task. The bigger model added one, inside their noise floor, at three times the cost. Invented numbers can show the bookkeeping, not prove that harness work always wins.
The holdout backs the first two gains. The type check gained about half as much there, so its honest value is nearer one task than two. They kept it, with the smaller number written down.
What the type check really cost
More context and more checks usually raise the pass rate, and they always cost something. Whole modules in the context pack burn tokens on every task, and the full integration suite adds minutes to every attempt.
The type check row is the one I’d point at. Pass rate went up, every attempt got slower and pricier, and passes per dollar dipped. That can still be worth it, if whoever keeps it has seen both columns.
Mining traces without closing the loop
You could hand the tuning itself to a script: read the traces for where the agent struggled, edit the instructions, rerun, keep whatever scores higher. I like the first half. If the agent spends a third of its budget hunting for the config loader, that’s a line for the map.
I don’t want the loop closing itself. Twenty-five tasks is a small target, and an automatic loop will hit it with lines like “when editing invoices, check the address history table,” which fix one replay task and mean nothing anywhere else. A holdout should catch that, but a loop that checks it every round turns it into training data too.
Instead, the trace job opens a pull request against the harness, with traces and replay numbers attached, for a person to review like any other code. Run the holdout only when you’re about to keep something, and swap in fresh PRs every few months.
I’m not against the bigger model. Once harness changes stop moving the number, it may be the next lever worth paying for, and I’d want a replay set big enough to show what it bought.