A support assistant gets asked how to rotate an API key. It replies with neat numbered steps and a link to its source. The steps are for a version of the product that was retired two years ago.
The usual reaction is to open the prompt and add a line like “always use the latest documentation.” That rarely fixes anything. The model got an old page and summarized it faithfully. The mistake happened one step earlier, when search picked that page.
I think most wrong answers from retrieval-augmented generation (RAG) start there. The model takes the blame because it’s the part that talks.
At Okta I was tech lead for the customer-support GenAI agent on support.okta.com. I’ll leave its internals out, and every example here, the opening one included, is made up. I covered how I’d design a whole support agent in an earlier post. This one stays on the search step.
Read the passages before you touch the prompt
Start by pulling up the exact passages the model saw for that question.
Usually one of these happened:
- The right document was never indexed, or was indexed before its last update.
- Chunking cut the steps away from the heading that says which product they’re for.
- The question used an exact term, like an error code, and vector search returned pages about the general topic that never mention it.
- An older or broader page outranked the specific one.
- Near-copies of one article filled every slot and pushed the answer below the cut.
If the right passage was there and the answer was still wrong, then yes, work on the prompt or the model. I just think that’s the smaller share, and you can’t tell which share you’re in until you look.
A starting design
Most stops on the diagram up top exist to head off one of those failures.
Sources come first. The earlier post covered keeping internal pages out of the index. The other half is staleness: fix or retire old pages at the source, not in the index.
Cleaning and chunking deserve more time than feels reasonable. Strip the page furniture, split on the document’s own headings, and carry the page title and section path into every chunk, so a passage still says which product and page it came from.
The index is hybrid, with keyword search (BM25 or similar) and vector search over the same chunks. Vectors catch paraphrase. Keywords catch the exact strings people paste in, like error codes and product names. Support content is full of both, so I wouldn’t ship with only one.
Retrieval casts a wide net: around 50 candidates from the two sides, merged into one list. Reciprocal rank fusion, which scores each chunk by how high it placed in each list, is a fine way to start. Collapse near-duplicates here too, or one article copied into four places takes four of your five slots.
The reranker, usually a cross-encoder, reads each candidate together with the question and keeps the best five or so. It’s slow per passage, so it only sees the 50. Adding one is often the biggest gain for the least work. Hand the model all 50 and the useful passage gets buried in text that’s only nearly relevant.
The model answers from those passages alone and cites one for each claim. A sentence that can’t point at a passage doesn’t belong.
Then the exit. If the top rerank score is low, or the best passages are for different product versions, or the draft makes a claim no passage supports, the system says it isn’t sure and passes the conversation to a person. That’s a different outcome from a wrong answer, and I’d track it separately.
Chunking defaults
Expect to tune all of these. I’m least sure about chunk size, which depends a lot on how your documents are written. Kept as metadata, product_version lets retrieval filter to the version the customer is on, which is what would have kept the retired steps from the opening out of the answer. last_updated helps you find pages nobody has touched since a release.
chunking:
split_on: [h2, h3]
max_tokens: 400
overlap_tokens: 50 # only when a section has to be split
never_split: # even when they run past max_tokens
- tables
- code_blocks
- numbered_steps
prefix_each_chunk_with:
- page_title
- section_path # "Admin guide > API keys > Rotate a key"
drop:
- navigation
- footers
- cookie_banners
- feedback_widgets
keep_as_metadata:
- source_url
- last_updated
- product_version
- content_type # docs page or KB article
Test retrieval on its own
The support-agent post argued for scoring retrieval apart from the final answer. I’d keep a separate set of retrieval tests: a real question, the document that should come back, and whether it made the reranked top five. Each miss maps to a failure above.
| Question | Should return | In top 5? |
|---|---|---|
| How do I rotate an API key? | Admin guide > API keys > Rotate a key | Yes, rank 1 |
| What does ERR_SYNC_409 mean? | KB: Sync fails with ERR_SYNC_409 | No, only vector search was on |
| Add someone to a team | Docs: Manage team members | No, duplicates of one KB article filled the list |
| Change the billing contact | Docs: Manage billing contacts | Yes, rank 4 |
| Is SSO on the starter plan? | Plans and pricing | No, retired plan pages crowded it out |
| Webhook retries stopped after an outage | KB: Webhook retry schedule | No, the retry table lost its heading |
Start with 30 or 40 real questions from search logs or closed tickets. Add a row whenever someone reports a wrong answer, and rerun the set whenever chunking, the index, the reranker or the sources change. The score is recall at 5, and it takes a few lines:
def hit_rate(cases, search, k=5):
"""cases: (question, expected_doc_id) pairs. search(q) returns doc ids, best first."""
misses = [q for q, expected in cases if expected not in search(q)[:k]]
return 1 - len(misses) / len(cases), misses
I’d read the misses before the number. The number says whether you got better. The misses say what to fix next.
Setting the “not sure” threshold
The handoff cutoff is the hardest number to pick in the whole design. Rerank scores aren’t calibrated probabilities, and they don’t mean the same thing for every kind of question, so a cutoff that suits how-to questions might hand off too much, or too little, on pricing questions. I’d start strict and loosen it one topic at a time, only where someone reading the answers agrees they were fine.
That means a high handoff rate early on, and I’m comfortable with that. I’d rather explain why a lot of questions went to people than explain a confident answer built on a page nobody should have indexed.