Build a fixed-size index over a shelf of service manuals and somewhere in it there’ll be a chunk that is six rows of a table and nothing else. A fastener name and two numbers, six times over. The header row that said which number is newton-meters went to the chunk before, along with the caption. The heading that named the machine is further back still.
The splitter that made it counted to 512 tokens and cut. No embedding model repairs that afterwards. A model embeds the text it’s handed, and the words that would make those six rows findable aren’t in that text anymore.
So I’d split on the document’s own structure. Small pieces do the matching, and the model gets the whole section a match came from. Then I’d settle the chunk-size argument, which every retrieval project seems to have, with one table of hit rates on the team’s own questions.
What a token count can’t see
A fixed-size splitter fails in two ways, and they show up separately in the numbers later.
The first is the cut table. The second is quieter. “Replace the seal if the gap is over 2 mm” is true of one pump on one machine. In the manual that’s obvious, because the sentence sits under “Drain pump”, which sits under a model name. In a chunk that starts mid-section it’s a rule with no owner, and it embeds almost the same as the matching sentence in forty other manuals.
Recursive character splitting, the usual choice in LangChain tutorials, is more polite about where it cuts. It tries blank lines first, then line breaks, then spaces. It still works to a size, and it has no idea what a heading is.
Small leaves to match, whole sections to read
The diagram at the top is the full route. A parser reads each document into a tree of sections, using its headings, lists and tables. A splitter walks the tree and writes leaf chunks, each carrying the path of headings above it. That path is embedded together with the text, so a torque table’s vector knows which machine and which assembly it describes.
Leaves do the matching, because a small chunk about one thing makes a sharper vector than a large chunk about four. Sections do the answering, because the leaf that matched is rarely enough to answer from. When the top five leaves come back, each is swapped for its enclosing section, and leaves that share a parent collapse into one.
This is the record I’d store and the rules I’d give the splitter.
from dataclasses import dataclass
from typing import Optional
LEAF_MAX_TOKENS = 300 # leaves are embedded and matched
PARENT_MAX_TOKENS = 1500 # parents are what the model reads
@dataclass
class Chunk:
chunk_id: str
doc_id: str # one service manual
heading_path: list[str] # ["HX-40 washer", "Drum assembly", "Tightening torques"]
parent_id: Optional[str] # enclosing section; None when this row is a parent
text: str
token_count: int # tokens in text, by the embedding model's tokenizer
page_start: int # pages are only used for scoring
page_end: int
def embed_text(self) -> str:
# The heading path goes into the vector, so a table knows its machine.
return " > ".join(self.heading_path) + "\n\n" + self.text
# Splitter rules. Where two disagree, the lower number wins.
# 1. A leaf never crosses a heading.
# 2. A table is one leaf, with its caption and header row. Never split it.
# 3. A list stays in the same leaf as the sentence that introduces it.
# 4. Otherwise pack whole paragraphs into a leaf until the next one
# would pass LEAF_MAX_TOKENS.
# 5. A leaf's parent is the smallest section that contains it. A section
# over PARENT_MAX_TOKENS is cut at its subheadings, then at paragraphs.
# Only leaves get an embedding. Parents are fetched by parent_id.
Rules 2 and 3 can break the 300-token cap, and that’s deliberate. A 600-token table is no trouble for text-embedding-3-small, which accepts 8,191 tokens. A small sentence-transformers model is another matter: all-MiniLM-L6-v2 reads the first 256 word pieces and drops the rest without a warning. With a model like that I’d cut a long table between rows and repeat the caption and header on each piece.
LlamaIndex’s hierarchical node parser and LangChain’s parent document retriever will both store leaves that point at parents. In their documented examples the hierarchy comes from chunk sizes, though, so the structure-aware splitting is still yours to write.
Three splitters, 150 questions
To show the comparison filled in, picture Wychwood Appliances, a manufacturer I’ve made up, numbers included. Wychwood has 420 service manuals. Its support engineers write 150 questions a field technician would ask, and note the manual and page where each answer is printed.
Three indexes are built from the same manuals, and only the splitter changes. The vectors come from text-embedding-3-small and sit in pgvector behind an HNSW index, the setup I’d start any index this size on, and every query takes the top five. A question is a hit when its answer page is among the pages that reach the prompt.
| Splitter | Hit rate at 5, all 150 | 31 split-table questions | Other 119 |
|---|---|---|---|
| Fixed, 512 tokens | 0.71 | 0.29 | 0.82 |
| Recursive character, 512 tokens | 0.78 | 0.45 | 0.87 |
| Structure leaves, parent expansion | 0.90 | 0.94 | 0.89 |
The first column says structure wins by 19 points. The middle column says where. Thirty-one of the questions ask for a value from a torque table that the fixed-size splitter had cut across two chunks, and on those it reaches the right page 9 times. The other 22 times, the top five are torque tables from the wrong manuals, which all read alike once the model name is gone.
The recursive splitter keeps more of those tables whole, since it would rather cut at a blank line, and gets 14. A whole table that has lost its heading still looks like every other manual’s table. Structure leaves get 29.
Across all 150 questions the third row finds 28 more answers than the first, and 20 of the 28 are those table questions. On the other 119 the gain is 0.82 to 0.89. That’s real, but I wouldn’t rebuild an index for it. If Wychwood’s manuals were plain prose, I’d expect three close rows, and I’d keep the simplest splitter.
Keep the three rows and swap the middle column for whatever your documents have that a size-based cut would break. One overall number would have hidden where the gain came from.
What the third row costs
Five parents can run to 7,500 tokens where five fixed chunks are 2,560. Shared parents bring that down, but it’s a bigger prompt on every question. It’s also a fair objection to the table, since row three gets to show the model more pages. I’d answer that with a fourth row, fixed-size at 1,500 tokens, and let the numbers argue.
The parser is the larger cost. Wychwood’s manuals are PDFs, and most PDFs record where characters sit on a page and little about what’s a heading or a table cell. If the manuals still exist in whatever they were authored in, I’d parse that. If PDF is all there is, the parser is most of the project, and tables are the worst of it.
Which is why I’d write the 150 questions first and fill in the fixed-size row before touching a parser. If that row comes back at 0.90, the manuals don’t need one.