A model fine-tuned in March doesn’t know the rate that changed in April. A model handed every document you own can still write a friendly paragraph where the form wanted one fixed line.
Those are two kinds of wrong, and each has its own fix. Retrieval changes what the model knows at the moment of one call. Fine-tuning changes how it behaves on every call. The retrieve-or-tune argument tends to start from whichever technique the team read about last. I’d start by reading a pile of wrong answers.
What each one is able to change
Retrieval-augmented generation (RAG) looks text up when the question arrives and puts it in the prompt. The model stays as it was. Update the index and the next call has the new fact, plus a passage the answer can cite. You pay in prompt tokens on every call.
Fine-tuning trains the model on pairs: an input, and the output you wanted for it. These days that usually means LoRA, which freezes the weights and trains a small add-on beside them, called an adapter. QLoRA does it against a 4-bit copy of the model, which is how Mistral 7B or the smallest Llama 2 trains on one GPU.
What comes out is a habit: the model does the thing without being asked. It has learned nothing about last week, and it can’t say which document a claim came from.
Sixty-four, twenty-seven and nine
To have a pile to sort, I’ve given a freight forwarder a name, Pennant Freight, and a support assistant. Neither is real, and I picked every number below.
The assistant answers shippers’ questions about rates and customs paperwork, on a Mistral 7B the company hosts itself. An engineer pulls 100 answers that human agents flagged as wrong and asks one thing of each: did the model lack a fact, or did it have the fact and handle it badly?
Sixty-four lacked a fact or had a stale one. Rate tables change every week, and the assistant’s rates came from a summary pasted into its system prompt over the winter. Twenty-seven had the right customs code in the wrong layout. The brokers’ intake system takes one fixed line, HTS 8471.30.0100; origin CN; value USD 18,250, and the assistant wrote a helpful sentence, despite two examples in its prompt. Nine were neither, mostly questions too vague to answer.
That afternoon of reading is the gap classifier in the diagram. A hundred answers won’t tell 27 from 31. They will show which pile is big.
From symptom to approach
| Symptom in the failed answer | Gap | Approach | Why not the other |
|---|---|---|---|
| A rate or policy that was right once | Missing fact | Retrieval | Tuned in, it’s stale at the next change |
| A guess about something only your records hold | Missing fact | Retrieval or a direct lookup | Weights recall specifics badly and can’t cite |
| Right content, wrong layout | Wrong behavior | Prompt examples, then tuning | More passages don’t change a habit |
| Your shorthand read as plain English | Wrong behavior | Prompt glossary, then tuning | An index only helps terms that get looked up |
| Wrong with the right passage in the prompt | Neither | A reworked prompt or a stronger model | The fact was already there |
Two rows put the prompt ahead of tuning on purpose. A couple of examples cost an afternoon. If they work, the rest is a money question: pay for them on every call, or once for an adapter. If they don’t, as at Pennant, thousands of examples in the weights usually manage what two in the prompt couldn’t.
Each pile gets its own line
The 64 take the upper line. Pennant splits its rate sheets by shipping lane, the way they’re already organized, and stores them in pgvector with an HNSW index, inside the Postgres it already runs. A scheduled job reloads changed lanes when the weekly rate file lands. Each call now carries three passages, about 1,800 extra prompt tokens, and on a rerun the stale-rate pile clears. Training can’t keep up with a file that changes every week.
The 27 take the lower line. Pennant gathers 4,000 pairs from closed tickets where a human agent wrote the customs line correctly, with the retrieved passages in each input, because that’s the prompt the tuned model will see. Two people check the pairs, and the 27 stay out of training. On a rerun with the LoRA adapter loaded, 3 of the 27 still fail, and the two examples come out of the prompt.
Both reruns, and every later change to either fix, use one eval set: the 100 failures plus 300 answers that were already right. An adapter trained on customs lines can shift answers that have nothing to do with customs, and retrieved passages can distract the model on questions it used to get right.
Four lines per option
The last stop is the cost table. Each option gets the same four lines, and it pastes into a spreadsheet as is.
option,line,amount,unit,driver
retrieval,one_time_cost,6,person-days,index the rate sheets and wire the prompt
retrieval,added_tokens_per_call,1800,prompt tokens,three passages on every call
retrieval,refresh_cost,0,person-days per refresh,scheduled reload each week
retrieval,added_latency,200,ms,one search plus a longer prompt
adapter,one_time_cost,10,person-days,6 checking 4000 pairs and 4 engineering
adapter,added_tokens_per_call,-350,prompt tokens,layout examples leave the prompt
adapter,refresh_cost,3,person-days per refresh,new layout or new base model
adapter,added_latency,0,ms,adapter merged into the weights
Pennant handles 120,000 calls a month, so the retrieval line adds 216 million prompt tokens. Pennant pays for GPUs, not tokens, so I’ll borrow a hosted price as a yardstick. At the $0.50 per million input tokens GPT-3.5 Turbo costs as I write, that’s $108 a month, small next to anything counted in person-days. At a hundred times the traffic, it’s the line that decides.
The adapter’s token line is negative only because Pennant serves the model itself. OpenAI has sold fine-tuning for GPT-3.5 Turbo since last August, and it bills the tuned model at several times the base model’s rate.
Refresh is the line that keeps billing. The index refreshes on a schedule with nobody watching. The adapter has two triggers. A layout change means new pairs. A new base model means retraining on the old ones, since an adapter only fits the model it was trained on. Mistral 7B is seven months old, and I wouldn’t bet on it lasting another twelve.
Latency moved the decision least: 200 ms for retrieval, nothing for an adapter merged into the weights.
Pennant’s table says yes twice, so it ships both, each for its own pile. That isn’t a rule. Had the sort found six layout failures and not 27, ten person-days and a standing refresh bill would be a poor trade, and I’d have lived with the six.
Once both are live, this hundred is spent. What’s left is three layout misses and nine oddities that neither fix was aimed at. So pull a fresh hundred and sort again. The next fix may be neither of these.