What RAG actually is, and why it became the substrate under every serious LLM system

Nobody pitches “this app uses RAG” anymore. That is the whole signal. The moment a pattern stops being the headline, you know it crossed the line from feature to substrate. The same thing happened to gasoline, to TCP/IP, to the relational database. Once a piece of infrastructure is everywhere, the conversation moves on to what you build on top of it. Retrieval-augmented generation is at exactly that moment.

I noticed it in client calls a few months ago. In 2023 the question was “should we use RAG”, with a slide deck attached. In 2026 the question is “which retrieval pattern did you pick”, with an architecture diagram already drawn. The pattern itself is assumed. What gets argued about is the chunking strategy, the reranker, the citation rendering. RAG is the gasoline. The argument is about the engine.

So it is worth slowing down and saying what RAG actually is, where it came from, and what it changed about the thing we call a language model. The moment a substrate disappears, the people building on it forget why it is there.

The origin: Lewis 2020, a paper about an unsolved problem

In May 2020, Patrick Lewis and his colleagues at Facebook AI Research and University College London published a paper with the unassuming title “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks”. The problem was concrete: large language models could write fluent answers, but they could not reliably handle questions that required specific, verifiable facts. The model knew a lot, but it did not know what it knew, and it could not tell you where it learned it.

The proposal was simple in shape. Wrap the model in a retrieval step. Before the model answers, fetch a handful of relevant documents from a corpus you control, stuff them into the prompt, then ask the model to answer using that material as its source. Two systems: one that finds things, one that writes things.

The paper was modest about its ambition. It proposed RAG as one technique among several for knowledge-intensive natural-language tasks (open-domain question answering, fact verification, that sort of thing) and showed it beat the alternatives on several benchmarks. The authors closed by calling it a “general-purpose recipe” other teams could apply. That last sentence turned out to be the load-bearing one. Six years later, the recipe is the floor every production system is built on.

How the pattern moved from research to default

Three things turned a 2020 paper into a 2026 substrate, and none of them are inside the paper.

The first was cost. Fine-tuning a model on your company’s documents is expensive, slow, and breaks every time the documents change. Retrieval is cheap, fast, and updates the moment you re-index the corpus. The economics quietly settled the debate. By late 2023, “should we fine-tune or use RAG” had stopped being a real argument for most use cases.

The second was the maturation of vector databases. By 2022, purpose-built stores (Pinecone, Weaviate, Qdrant, the pgvector extension for Postgres) made similarity search over millions of document chunks a commodity operation. The infrastructure caught up with the idea.

The third was context-window growth. The 2020 paper assumed short prompts. Models in 2024 and 2025 routinely accept 100,000 to 2,000,000 tokens of input. The system could suddenly fetch dozens of passages instead of three, and the model had room to read all of them. The pattern got more powerful without any change to its shape.

By the time enterprise teams started asking how to ground LLM answers in their own data (contracts, knowledge bases, support tickets, policies, manuals), the answer was already there, packaged in frameworks like LangChain and LlamaIndex, with worked examples and community-tested defaults. The pattern moved from “novel research technique” to “the obvious thing to do” in roughly thirty months. That is fast, even by software standards.

The architecture shifted visibly across those thirty months. The 2020 paper described something academically minimal; the 2026 enterprise default added two boxes the research version did not need.

The 2020 version is the load-bearing pattern: retrieve, then generate. The 2026 version adds reranking (because raw vector similarity surfaces too many near-duplicates) and citations (because users need a verification path, not just an answer). The shape is the same. The accessories are what production taught the field to add.

The three components, named honestly

Strip RAG to its load-bearing parts and three pieces have to work together.

The retriever turns a user query into an embedding, searches a corpus of pre-embedded passages, and returns the top N by similarity. Most failures start here. If the retriever surfaces passages that look semantically similar but are about a different thing, the model produces a confident-sounding answer about the wrong topic. The chunking problem (how you split the source documents before embedding) is a deeper rabbit hole than most teams expect.

The reranker is the unglamorous step that earns its keep on day forty. The retriever is fast and approximate; the reranker is slow and accurate. It takes the top 50 or 100 passages and rescores them with a more expensive model, returning the top 5 or 10 that actually deserve a place in the context window. Skipping the reranker is the most common reason a production system feels worse than the prototype did.

The generator is the language model itself, now wearing a different hat. Instead of an oracle, it is a reader. Its job is to read the retrieved passages and synthesise an answer that stays inside them. The discipline is in the prompting: “answer only from the passages below; if the answer is not in them, say so.” A model held inside its retrieved context behaves very differently from one let loose on its training memory.

What changes about an LLM the moment you wrap it in retrieval

This is the part the substrate framing helps you see. A bare LLM is a frozen object. Whatever it learned in training is what it knows, and you cannot change that without retraining. The same model on Monday and Friday gives you the same answer to the same question.

Wrap it in retrieval and the object becomes a different kind of thing. The model is no longer the source of truth; the corpus is. Update the corpus and the answers update. Add a document and the system knows about it before lunch. Remove a document and the system forgets it. The model has become a reader of a corpus you control, not a vault of knowledge you cannot inspect.

That is the architectural shift most teams underestimate. The bare LLM is a black box you trust or do not trust. The retrieval-wrapped LLM is a system with a knowable input (the retrieved passages), a knowable output (the generated answer), and a clear failure path (you can show the user which passages were retrieved and let them check whether the answer follows from them). The model went from oracle to clerk, and the clerk has receipts. That receipt is the citation, and citations are the user-visible proof that the grounding is on.

It is also the point at which RAG becomes one side of the larger LLM harness. The other sides (guardrails, evaluation, hallucination mitigation, audit) all assume the grounding step is in place. Without it, they are wrapping nothing.

Where it still fails

Substrates fail too. RAG fails in three predictable ways, and naming them is what separates a serious team from a hopeful one.

If the corpus is wrong, the answer is wrong with full confidence. The grounding does not validate the data; it narrows the surface from which the wrong answer is generated. Garbage in, fluent garbage out.

If the retrieval misses the right passage, the model fills in from training memory or, worse, from adjacent passages that sound related. Fluency disguises the gap. This is the failure citations expose the moment a careful user clicks through.

If the chunking destroys the structure of the source (cuts a paragraph in half, splits a table from its caption, separates a definition from its example), retrieval recall collapses in ways invisible until production traffic hits patterns the development set never tested. The document retrieval deep dive covers this.

The substrate works. It also has edges. That is true of every piece of infrastructure that earned the name.

The one-sentence summary

RAG is not the trendy answer anymore; it is the assumed substrate. The work moved up a layer: which variant you pick, how you chunk, what your reranker does, whether your citations point at the passage that justified the answer. The Lewis 2020 paper is the foundation, and like any foundation, you notice it most when it is missing.