A coding agent can’t tell that it’s wrong until something tells it. In most repositories that something is a test suite that takes a coffee break to run, prints a few thousand lines when it fails, and only works on the laptop of whoever set it up two years ago.
I think most of the gap between a good agent demo and a disappointing rollout comes from here. People swap models and rewrite prompts, or compare Claude Code against Codex CLI against Cursor’s agent mode, when all of them are stuck behind the same slow, noisy check. Asynchronous agents such as Devin work in their own cloud environment and report back through a pull request. They have the same dependency, with nobody at the keyboard to say “ignore that warning, it always does that.”
So before tuning anything on the agent side, I’d measure two things about the repository: how long it takes to say whether a change is right, and how much it prints when the answer is no.
What the agent does with your test output
Every one of these tools works the same loop: edit files, run a command, read what came back, decide what to do next. The command’s output goes straight into the model’s context, and that context is finite.
A human skims 4,000 lines of log and jumps to the red part. An agent reads all of it, or the tool truncates it and the agent reads whichever end survived. Either way, the one line that says which assertion failed is competing with deprecation warnings and a stack trace from a test that’s been flaky for a year.
Slow checks do a different kind of damage. If a full run takes twenty minutes, the agent gets three attempts an hour. Worse, it starts skipping the check. It runs one test it guessed was relevant, sees green, and declares the work finished.
Garnet Health’s 19-minute check
To show the mechanism, I’ve made up a company and made up numbers to go with it.
Garnet Health keeps everything in one monorepo. A full test run takes 19 minutes and prints about 4,000 lines when anything fails. The team hands an agent 20 small backlog tickets, the kind a new hire gets in week two. It finishes 6. On the other 14 it mostly either gives up after a couple of 19-minute rounds or “fixes” a failure that belonged to an unrelated flaky test.
Then someone spends a few days on the repository and leaves the agent alone. Each package gets a check target that runs lint, types and that package’s unit tests in 70 seconds and prints failures only. Same agent, same model, same 20 tickets: 14 finished.
The six it still misses are honest misses, tickets that needed a design decision or touched four packages. That’s the split I’d want: the agent losing to hard tickets, and none lost to a check it couldn’t use.
Six tests for the repository
Each row is a test you can run this afternoon. Pass or fail, no scoring.
| Check | How to test it | Pass when |
|---|---|---|
| One command | Fresh clone in a clean container, then make check |
Exits 0 with nothing typed by hand in between |
| Inner-loop time | time make check PKG=<one package> |
Under 2 minutes on a CI-sized machine |
| Flaky tests | Run make check 10 times on an unchanged main |
10 of 10 green |
| Output on failure | Break one assertion on purpose, run the check | Under 100 lines, with the failing test and assertion in them |
| Same answer everywhere | Compare what CI runs with what the check runs | CI calls the same target, then adds the slow suites |
| Task size | Pick five open tickets | Each names one test file that decides if it’s done |
The flaky row is the one I’d be strictest about. A person learns which tests lie. An agent tends to treat every red line as its own fault and start editing code that was fine.
The last row is about the ticket, but it depends on the repository. “Make exports faster” can’t be judged by anything. “Exports of 10,000 rows finish in under 5 seconds, see tests/export/test_large_export.py” can, and the agent can run that one file in seconds, as often as it likes.
A check target that only speaks when something breaks
This is the Makefile I’d start from for a Python package. Swap the three commands for your own stack; the wrapper is what you’re copying.
SHELL := bash
PKG ?= .
MAX_LINES ?= 60
BIN := .venv/bin
# $(call step,name,command)
# Runs the command quietly. Prints its last MAX_LINES lines only if it fails.
# The command must not contain a comma.
define step
log=$$(mktemp); \
if $(2) >"$$log" 2>&1; then echo "ok $(1)"; \
else echo "FAIL $(1)"; tail -n $(MAX_LINES) "$$log"; fail=1; fi; \
rm -f "$$log";
endef
.PHONY: check
check: .venv
@fail=0; \
$(call step,lint,$(BIN)/ruff check --quiet --output-format=concise $(PKG)) \
$(call step,types,$(BIN)/mypy --no-error-summary $(PKG)) \
$(call step,tests,$(BIN)/pytest $(PKG) -q --no-header --tb=short) \
exit $$fail
.venv: requirements-dev.txt
@python3 -m venv .venv
@$(BIN)/pip install --quiet -r requirements-dev.txt
@touch .venv
A green run prints three lines. A red run prints the name of the step that failed and the tail of its output, which for pytest is the short tracebacks and the list of failed tests. All three steps run even when the first one fails, so the agent sees a lint error and a broken test in the same round.
make check PKG=services/billing is the inner loop. Plain make check covers the whole repository, and CI calls that same target before it goes on to integration tests and anything else too slow for the loop.
Two trade-offs I’d accept knowingly. Cutting output to 60 lines will sometimes hide the line that explains a failure, so MAX_LINES stays overridable and the agent can rerun with MAX_LINES=300. And a fast check is a partial check: something will pass the loop and fail CI. That’s fine as long as it happens after the cheap mistakes are out of the way.
Where the instruction file fits
Claude Code reads a CLAUDE.md at the repository root, and Cursor reads project rules from .cursor/rules, though it still honors an older .cursorrules file. Once the checklist passes, that file gets very short: run make check PKG=<package> after every change, and don’t open a pull request until it’s green. If yours needs a page of caveats about which tests to ignore and which environment variables to export first, read the page as a list of repository bugs.
Two numbers before any model comparison
Garnet’s two numbers went from 19 minutes and 4,000 lines to 70 seconds and a few dozen. Nothing on the agent side changed at all, and I’d want both figures in any write-up that compares one agent with another.