A graph index pays for itself on two kinds of question

Chunk retrieval handles lookups. A graph index is for multi-hop and whole-corpus questions, and it costs thousands to build, so sort your real questions before paying.

Documents Chunker Extractor Graph store Summaries Classifier Vector index Answer entities, relations per community graph or summaries
Documents become chunks, a graph and community summaries; a classifier sends each question to the vector index or to the graph.
On this page
  1. Chunks answer one kind of question
  2. Sort the questions before you price anything
  3. What the extraction bill looks like
  4. Changed documents are the cost that keeps coming
  5. Where I’d draw the line for Stannard

Some answers were never written down in one place. Ask a research archive which contract labs ran assays for compounds that later failed on liver toxicity, and a retrieval system built on chunks hands back five passages. One mentions a lab, two mention assays, two mention liver toxicity.

None of them holds the answer. It sits in the links between a lab, the compounds that lab touched, and what happened to those compounds two reports later.

That’s the case for a graph index. Since Microsoft open-sourced GraphRAG in July, I’ve seen it treated as the next version of retrieval, the thing you upgrade to. I’d describe it more narrowly: a second index that answers two kinds of question well and is priced like a small project.

Chunks answer one kind of question

I sort the questions people ask a document system into three kinds.

A lookup has its answer in one or two passages. “What was the Phase 1 dropout rate for ST-204?” Hybrid search with a reranker on top handles these, and when it doesn’t, the cause is usually how the documents were chunked.

A multi-hop question connects things that live in different documents, like the lab question above. Top-k retrieval finds one end of the chain and stops there.

A corpus-wide question asks about everything at once. “What are the recurring reasons our formulation studies get delayed?” The ten chunks nearest to that sentence are a sample of delays. The question wanted a tally.

A graph index is built for the second and third kinds. At indexing time a model reads every chunk and pulls out entities and the relations between them. Those merge into a graph, the graph gets clustered into communities, and a model writes a summary of each community. Multi-hop questions walk the edges. Corpus-wide questions get answered from the summaries. A classifier in front decides which index a question goes to, and lookups keep going to the vector index.

Sort the questions before you price anything

Pull a couple hundred real questions from your logs and put them in a sheet like this:

id,question,kind,vector_correct,graph_correct,notes
q017,"What was the Phase 1 dropout rate for ST-204?",lookup,yes,,one report holds it
q052,"Which contract labs ran assays for compounds that later failed on liver toxicity?",multi_hop,no,yes,lab to compound to outcome
q131,"What are the recurring reasons our formulation studies get delayed?",corpus_wide,no,yes,no single passage holds it

Two rules keep the sheet honest. A person labels the kind before anyone runs anything, so the label can’t drift toward the result. And the vector column has to come from your best chunk setup, with hybrid search and reranking switched on. Otherwise the graph gets credit for fixing retrieval you hadn’t finished.

The graph column needs a graph, which is awkward when the point is to decide whether to build one. Index a slice first, one product line or one year of documents, and score the questions that belong to it.

To show what the tally looks like, I’ve invented a company. Stannard Biotech has 12,000 research reports, split into 180,000 chunks, and 200 questions its analysts typed into the existing search.

Kind Questions Correct with vectors Correct with graph
Lookup 141 124 (88%) not routed
Multi-hop 38 13 (34%) 27 (71%)
Corpus-wide 21 2 (10%) 14 (67%)

Vectors alone get 139 of 200 right. Routing the two hard kinds to the graph gets 165. That’s 26 more answers, all of them from the 59 questions that weren’t lookups.

What the extraction bill looks like

Embedding is nearly free. 180,000 chunks at 600 tokens each is 108 million tokens, about $2 at two cents per million.

Extraction is where the money goes, because a chat model reads every chunk at least once, and usually twice: one pass to extract, one to ask what it missed. The prompt with its examples is bigger than the chunk.

def index_cost(chunks, in_tok, out_tok,
               communities, sum_in_tok, sum_out_tok,
               in_price, out_price, batch=False):
    """Prices are dollars per million tokens. in_tok and out_tok are per
    chunk, all extraction passes added up. sum_* are per community summary."""
    per_m = 1_000_000
    extraction = chunks * (in_tok * in_price + out_tok * out_price) / per_m
    summaries = communities * (sum_in_tok * in_price + sum_out_tok * out_price) / per_m
    total = extraction + summaries
    return total / 2 if batch else total


# Stannard, illustrative prices: $2.50 in, $10.00 out per million tokens
print(index_cost(180_000, 5_000, 1_500, 15_000, 6_000, 1_500, 2.50, 10.00))
# 5400.0

About $5,400, of which $4,950 is extraction and $450 is summaries. The prices are illustrative, so put in your own. Indexing isn’t urgent, which makes it a good fit for OpenAI’s Batch API: half price if you can wait up to a day for results. That brings Stannard to $2,700, still over a thousand times the embedding bill.

Changed documents are the cost that keeps coming

Suppose 6 percent of Stannard’s chunks change in a month. That’s 10,800 chunks to re-extract, roughly $300 at the same rates. The money is the easy part.

A changed chunk can add an entity, drop a relation, or shift which community a node belongs to. Each of those leaves a summary describing a graph that no longer exists. You can’t recluster on every edit, and you can’t ignore edits either, so write the policy down before the first build:

rebuild_policy:
  detect:
    key: chunk_content_hash
    unchanged: skip
  on_changed_chunk:
    - delete entities and relations sourced only from the old chunk
    - re-extract the new chunk
    - mark every community it touches as stale
  stale_summaries:
    refresh: nightly
    use: batch
    serve_meanwhile: old summary, flagged with its build date
  full_recluster:
    when_any:
      - changed_chunks_since_last_recluster_pct >= 15
      - days_since_last_recluster >= 90
  prompt_or_model_change:
    action: full re-extraction
    budget: same as the first build

The last entry is the expensive one. Changing the extraction prompt changes what counts as an entity, and old and new extractions don’t mix. Every prompt improvement costs another first build.

Where I’d draw the line for Stannard

I’d build it. Thirty percent of their questions are the kinds a graph is for, and those are the questions an analyst otherwise answers by reading reports for an afternoon.

I’d also fix the 17 lookups that failed first, because that work is cheaper and it helps the larger group. My line is about one question in ten. Under that, the hard ones can go to a person, or to a model with a long context window and the right twenty reports.