A right answer by the wrong path still fails

A YAML trajectory spec and a 61-line Python checker that grade an agent's recorded tool calls with cheap rules before any judge model gets involved.

Scenario set Agent under test Trace recorder Path rules Answer check Judge model Scorecard deterministic unclear cases only
Recorded traces split into path rules and an answer check with a judge model, then meet at one scorecard.
On this page
  1. What the recorder keeps
  2. One spec per scenario
  3. The checker
  4. Pebblebank Books, scored both ways
  5. What I left for the judge model
  6. What the checker doesn’t look at yet

An eval that reads only the final message will pass an agent that issued a refund first and checked the return window second. The customer was owed the money, the reply says so, and the score goes up. I wouldn’t accept that order of work from a person, and I don’t see why an agent should get a pass for it.

What follows is a small trajectory check: one YAML spec per scenario, and a Python script that scores a recorded tool-call trace against it. It runs beside the answer check and reports a second score.

What the recorder keeps

OpenAI function calling and Claude tool use both hand you the path for free. Each tool call arrives as a structured block with a name and JSON arguments, and the loop you own is what executes it. So the recorder is a wrapper around that loop. For every step it appends the tool name, the arguments, and the dollar cost of the model call that produced the step, worked out from the token usage on the response.

I write one JSON file per scenario run. If your traces already go to a store such as LangSmith, export from there and skip the wrapper. The checker needs only this much:

{
  "scenario": "refund-inside-window",
  "steps": [
    {"tool": "get_order", "args": {"order_id": "Q-1042"}, "cost_usd": 0.004},
    {"tool": "issue_refund", "args": {"order_id": "Q-1042"}, "cost_usd": 0.005},
    {"tool": "check_return_window", "args": {"order_id": "Q-1042"}, "cost_usd": 0.004}
  ],
  "answer": "Your refund of $18.50 is on its way."
}

That trace is the failure from the first paragraph. Nothing in the answer field gives it away.

One spec per scenario

scenario: refund-inside-window
must_call:
  - get_order
  - check_return_window
  - issue_refund
must_not_call:
  - escalate_to_human
order:
  - before: get_order
    after: check_return_window
  - before: check_return_window
    after: issue_refund
max_steps: 5
max_cost: 0.03

The order rules are precedence rules. The second one reads: every issue_refund needs a check_return_window somewhere before it. It says nothing about the two being adjacent. My first draft of this spec was the full expected sequence. I dropped it, because an exact-match golden path fails every time the agent does something harmless, like looking up the customer before the order. Precedence pairs let me pin the two or three orderings I care about and leave the rest alone.

must_not_call changes per scenario, which is why the spec lives with the scenario and not in one global file. For the order that’s outside its return window, issue_refund moves into that list.

The ceilings catch the agent that makes six calls where two would do. I set max_steps at about twice the path I’d take by hand. Set it tighter and the check fails on one extra lookup, and people learn to ignore it.

The checker

It needs PyYAML and nothing else. Run it as python check_path.py specs traces.

import json
import pathlib
import sys

import yaml


def check(spec, trace):
    """Return a list of failures. An empty list means the path passed."""
    steps = trace["steps"]
    calls = [step["tool"] for step in steps]
    failures = []

    for tool in spec.get("must_call", []):
        if tool not in calls:
            failures.append(f"missing required call: {tool}")

    for tool in spec.get("must_not_call", []):
        if tool in calls:
            failures.append(f"forbidden call: {tool}")

    for rule in spec.get("order", []):
        first, later = rule["before"], rule["after"]
        for i, tool in enumerate(calls):
            if tool == later and first not in calls[:i]:
                failures.append(f"{later} at step {i + 1} ran before any {first}")
                break

    max_steps = spec.get("max_steps")
    if max_steps is not None and len(calls) > max_steps:
        failures.append(f"{len(calls)} steps, ceiling is {max_steps}")

    cost = sum(step.get("cost_usd", 0.0) for step in steps)
    max_cost = spec.get("max_cost")
    if max_cost is not None and cost > max_cost:
        failures.append(f"cost ${cost:.4f}, ceiling is ${max_cost:.4f}")

    return failures


def main(spec_dir, trace_dir):
    spec_paths = sorted(pathlib.Path(spec_dir).glob("*.yaml"))
    passed = 0
    for spec_path in spec_paths:
        spec = yaml.safe_load(spec_path.read_text(encoding="utf-8"))
        trace_path = pathlib.Path(trace_dir) / f"{spec['scenario']}.json"
        if not trace_path.exists():
            print(f"FAIL {spec['scenario']}: no trace recorded")
            continue
        failures = check(spec, json.loads(trace_path.read_text(encoding="utf-8")))
        if failures:
            print(f"FAIL {spec['scenario']}: " + "; ".join(failures))
        else:
            passed += 1
            print(f"PASS {spec['scenario']}")
    print(f"{passed} of {len(spec_paths)} paths passed")
    return 0 if spec_paths and passed == len(spec_paths) else 1


if __name__ == "__main__":
    sys.exit(main(sys.argv[1], sys.argv[2]))

Against the trace above it prints FAIL refund-inside-window: issue_refund at step 2 ran before any check_return_window and exits non-zero, so it can gate a build. A missing trace counts as a failure. An agent that crashed before recording anything shouldn’t raise the pass rate by being absent.

Every failure is a plain sentence, and the ordering ones carry a step number. I care about that more than the score, because whoever reads the output is usually about to open the trace and wants to know where to look.

Pebblebank Books, scored both ways

Pebblebank Books is a shop I’m imagining, refund agent and all, and so are its numbers.

Say its refund agent runs 40 scenarios. The answer check, the kind of assertion on the final reply that promptfoo or OpenAI Evals handles well, passes 36 of them. Then the path check runs over the same traces. In 9 of those 36, the agent issued the refund before it checked the return window.

Scored on Passing Score
Answer only 36 of 40 90%
Answer and path 27 of 40 67.5%

Those nine came out right only because their scenarios put the order inside the window. Give the same habit an order from five months ago and money leaves that shouldn’t have. The 90% was partly a measure of how forgiving the scenario set was.

It also changes what you fix. Four wrong answers send you to the prompt wording. Nine refunds issued before the check send you to the tool design: maybe issue_refund should refuse to run without a window result passed in.

What I left for the judge model

Rules can’t tell me whether the reply explained a denied refund kindly, or whether a retry after a tool error was a reasonable thing to try. That’s judge-model work, and in the diagram the judge sits on the answer line, after the plain assertion, for the cases the assertion can’t settle.

I keep it off the path line on purpose. A rule costs nothing to run and gives the same verdict next month. A judge costs a model call per trace, and its verdicts can shift when the judge model changes underneath you. So the rules run first, and a trace that already failed on its path never reaches the judge. When the judge does run, it gets one narrow question and has to answer pass or fail.

What the checker doesn’t look at yet

Arguments are the obvious gap. The checker confirms issue_refund was called after the window check, and has no opinion on whether the amount matched the order. A small matcher on args is the next thing I’d add.

Per-tool call limits would help too. max_steps: 5 allows four order searches in a row.

And one run per scenario isn’t enough. The same agent on the same input can take a different path on the next run, so I’d run each scenario five times and report “path passed 3 of 5” per scenario. A single green tick hides the scenarios that need the most attention.

If you’re adding this to an existing eval set, write the first spec for a scenario that ends in a write. That’s where a bad path costs something.