Chunking, retrieval, and reranking

Chunking decides what a "unit" of retrievable knowledge is, and it is one of the highest-leverage, most under-discussed decisions in a RAG system. Chunk too small and you lose surrounding context a passage needs to make sense; chunk too large and irrelevant text dilutes the embedding, making genuinely relevant chunks harder to find. Document structure matters more than a fixed token count: a policy document's clauses, a codebase's functions, and a legal contract's sections all have natural boundaries that a naive fixed-size splitter destroys.

Once candidates are retrieved, a first-pass method (often the cheap one -- BM25 or a lightweight embedding search) casts a wide net, and a reranker narrows it. Rerankers are typically cross-encoders: slower per-item than the initial retrieval, but far more accurate at judging true relevance, because they see the query and passage together instead of comparing two independently-computed vectors. Running a reranker over the top 50-100 candidates and keeping only the top 5-10 is a standard, high-value pattern once a system has enough traffic to justify the extra latency.

Evaluation has to be layered the same way retrieval is built: measure whether the right passage was retrieved at all (context relevance/recall), separately from whether the final answer is actually grounded in what was retrieved (faithfulness), separately from whether the answer is simply correct. Collapsing these into one end-to-end score hides which stage is actually broken when something goes wrong.