Somewhere in every agent there’s a line that decides whether to call the model again or stop. In an agent built on a framework, that line usually sits in someone else’s package.
It’s the line I’d want open when the agent quits a step early, or spends 40,000 tokens on one refund question. So I want it in my repository, in a file I can read top to bottom. That file is shorter than most people expect: a hundred-odd lines for an agent in production, with a core of under forty.
Function calling took the parsing out of the loop
The loop goes back to the ReAct paper from late 2022. The model picks an action and reads the result, then goes around again. Back then the action arrived as prose, and pulling a tool name and its input out of a paragraph was finicky work. LangChain did that parsing for you, which is a big part of why so many teams started there.
With function calling, now in every large provider’s API, the model returns the tool name and its arguments as JSON, so the parsing is the provider’s job. What’s left for an orchestration layer is the eight stops in the diagram above, and most of them come to a line or two.
The whole loop in 39 lines
This runs against OpenAI’s chat completions API, with the tools defined in another file.
import json
from openai import OpenAI
client = OpenAI(max_retries=3) # backoff on rate limits and 5xx stays in the SDK
def run_agent(question, system_prompt, tool_specs, tool_fns, trace,
max_steps=8, token_budget=20_000):
messages = [
{"role": "system", "content": system_prompt},
{"role": "user", "content": question},
]
tokens_used = 0
for step in range(1, max_steps + 1):
response = client.chat.completions.create(
model="gpt-4-turbo", messages=messages, tools=tool_specs
)
message = response.choices[0].message
tokens_used += response.usage.total_tokens
record = {"step": step, "tokens_used": tokens_used, "tool_calls": []}
if not message.tool_calls: # no tool requested, so this is the answer
trace({**record, "stop": "answer"})
return message.content
messages.append(message.model_dump(exclude_none=True))
for call in message.tool_calls:
fn = call.function
try:
result = tool_fns[fn.name](**json.loads(fn.arguments))
except Exception as err: # sent back so the model can correct itself
result = {"error": f"{type(err).__name__}: {err}"}
messages.append({"role": "tool", "tool_call_id": call.id,
"content": json.dumps(result, default=str)})
record["tool_calls"].append(
{"name": fn.name, "arguments": fn.arguments, "result": result})
if tokens_used >= token_budget:
record["stop"] = "token_budget"
elif step == max_steps:
record["stop"] = "step_limit"
trace(record)
if "stop" in record:
return None # no answer: the caller hands the case to a person
Read it against the diagram. The prompt assembler is the messages list at the top, and system_prompt comes from a file in the repo, so it has a history. The step router is a single if. The tool runner catches whatever goes wrong, an unknown tool name or arguments that don’t parse included, and sends the error back as a tool result. State is that same messages list. It only grows in this version, and the day it needs trimming I know which file to open.
Both limits are arguments you can see: eight steps and 20,000 tokens. Tokens are summed across steps because every step sends the whole conversation again, and that sum is what you pay for. Each step writes one record, and the last record of a run says why it stopped.
trace is any function that takes a dict. Mine would append a line to a JSON Lines file and forward the same record to a tracing service.
Four things to keep and four to write
| Piece | Verdict | Why |
|---|---|---|
| Provider SDK | Keep | The provider maintains it. Across several providers, LiteLLM gives you one call format. |
| Retry helper | Keep | max_retries on the client, or Tenacity around a flaky tool. Backoff is solved. |
| Tracing | Keep | LangSmith accepts traces from plain Python. You still decide what goes in the record. |
| Document loaders | Keep | Parsing PDFs and HTML is real work, and it happens outside the loop. |
| Prompts | Write | They’re the agent’s behavior, so they belong in files with a history. |
| The loop | Write | It’s the code you’ll debug most. |
| State | Write | What the model sees on step six decides what it does on step six. |
| Tool registry | Write | Tool names and schemas are instructions to the model. |
The line I draw is between plumbing and behavior. A library at the edge can be swapped without the agent acting any differently. Whatever the model reads, and whatever decides the next step, changes the agent when it changes, so I want it where a reviewer will see it. Tool definitions are on that side of the line, and I wrote about getting them right last month.
Tamarind’s two days looking for a prompt nobody had read
I invented Tamarind Travel for this post, along with its scores and the upgrade that breaks them. Its refund agent reads a booking and the fare rules, then decides what the customer gets back. It’s built on an agent framework, and before each release the team runs it against 200 labeled cases. It scores 93 percent.
A routine dependency upgrade goes in, and the next run scores 81. Twenty-four cases that used to pass now fail.
Nothing the team wrote has changed. Their prompt is the same and so is the model. Day one goes to suspecting the provider. Day two goes to rerunning cases one at a time, until someone prints the raw request from the old version and the new one and diffs them. The framework’s default agent prompt, the template wrapped around Tamarind’s own instructions, was reworded in the upgrade. Nobody on the team had read it. It shipped inside the package, and it had always worked.
I wouldn’t blame the framework. Changing defaults is part of what a release does. The trouble is where Tamarind could see the change: as a version number in a lockfile.
So the team rewrites the agent as a 140-line loop, with every prompt in a file in the repository. The lockfile drops from 61 packages to 9. The next reworded prompt arrives as a pull request, where a reviewer reads the sentence that changed and the 200-case score is attached before it merges.
What the 140 lines cost
Owning the loop means owning everything the framework used to hand over. Streaming partial answers and trimming a long conversation are now mine to write and test. That’s how 39 lines became 140 at Tamarind, and the count keeps climbing unless someone guards it.
There’s a cost for people as well. A new engineer can look up a framework’s documentation. Ours is the file itself, so it has to stay plain enough to read in one sitting.
For a prototype I’d still start with a framework. In the first week, trying five ideas quickly is worth more than knowing where the stop condition lives.
LangChain’s own LangGraph, announced in January, comes at the same problem from the framework side: you define the loop as an explicit graph of your own steps. It hasn’t reached a stable release yet, so I’d experiment with it and keep it away from refunds for now.
Pick yesterday’s worst run and print the exact request the model received on step three. Then say which file each line of it came from. If that takes ten minutes, it doesn’t much matter whose loop you’re running. If it takes two days, write the loop.