Why retrieval, not just a bigger model

A model's parametric knowledge is frozen at training time and baked into its weights at a fixed cost per token, whether that fact is used once a year or a million times a day. Retrieval separates knowledge from reasoning: the model stays general-purpose, and a datastore holds whatever facts actually need to be current, auditable, or specific to one client's data. That separation is what makes RAG cheaper to keep fresh (re-index a document instead of re-training a model) and easier to make trustworthy (you can point at the retrieved passage a claim came from).

The trade-off is real, though. A RAG system is only as good as what it retrieves: if the right passage never makes it into the candidate set, no amount of generation quality recovers it. That is why RAG failures are almost always retrieval failures wearing a generation-shaped costume, and why the next chapter starts at chunking and retrieval rather than at prompting.

Retrieval also is not free of judgment calls. Dense (embedding) search finds semantic similarity but can miss exact terms -- product codes, legal citations, acronyms -- that a keyword method like BM25 catches directly. Production systems increasingly run both and combine the results (hybrid search) rather than picking one.