Document RAG for the Enterprise: Chunking, Retrieval Quality, and the Evaluation Problem

Document RAG looks easy in a demo and quietly fails eighteen months in. The demo pulls three clean paragraphs from a fifty-page PDF and the model writes a confident, cited answer. The production system, with a hundred thousand documents and a thousand daily queries, watches its quality erode for reasons nobody on the team can name. The model did not get worse. The corpus did not get worse. The chunking and the retrieval did, slowly, in ways nobody was measuring.

I think of document RAG the way I think of restoring an old building. Anyone can paint a room. The part that decides whether the building stands is the part nobody sees: how the bricks were cut, how the load travels through the walls. Chunking is the brickwork of retrieval. Get it wrong and no amount of paint on top will hide it for long.

This is the practitioner companion to the broader retrieval and grounding deep dive, narrowed to the variant most enterprise teams ship: passages chunked from documents, embedded into a vector store, retrieved by similarity, fed to a model.

Chunking, the discipline nobody calls a discipline

Chunking is the act of cutting a document into the units you will retrieve. It sounds like a one-line decision. It is not. Chunk size, overlap, and boundary strategy together cap the ceiling of every downstream metric. You can swap the model, swap the vector database, swap the reranker, and the system stays capped by the choices made at this layer.

Four strategies dominate in 2026, and the right answer depends on the shape of your documents.

Fixed-size chunking. Cut every document into chunks of N tokens with M overlap. Cheap, predictable, oblivious to meaning. Acceptable for homogeneous text (knowledge base articles, support tickets). Breaks for documents where structure carries the meaning.

Semantic chunking. Split where similarity between adjacent sentences drops below a threshold. Honours the shape of the argument. More expensive at indexing time, often the right default for narrative prose.

Structural chunking. Cut on the document’s own structure (headings, table rows, slide boundaries). The one most teams should try first and usually skip. A PDF with clear H1/H2/H3 already tells you where the chunks live; falling back to fixed-size throws away information the author already provided.

Parent-document and late-chunking patterns. Retrieve at one granularity, generate from another. Embed small chunks for precise retrieval, pass the larger parent passage to the model for context. Günther et al. published a related technique called late chunking in 2024, embedding the whole document first and chunking the embeddings afterwards, which preserves cross-chunk context. Anthropic shipped a sibling pattern called contextual retrieval later that year. Both answer the same underlying problem: a chunk read in isolation often loses the context that made it meaningful.

No single chunking strategy wins across all corpora. The work is to match the strategy to the document shape, then measure.

Retrieval, the boring decision that decides everything

Once the chunks exist, the question is how to find the right ones. Three flavours dominate, and the field has stopped pretending one is universally best.

Pure vector search was sold as the replacement for keyword search around 2022, and most teams that took the advice came back to hybrid within eighteen months. The reason is not subtle. Enterprise corpora are full of identifiers (SKUs, case numbers, regulatory citations, internal acronyms) that vector embeddings flatten into approximate neighbourhoods. A user typing “policy 7-A.3.iv” wants the document that says exactly that, and a semantic neighbour is not good enough.

Hybrid retrieval, with a fusion step combining BM25 and vector scores (Reciprocal Rank Fusion is the common one), is the 2026 production default for a reason. The other default is a reranker on top: a small cross-encoder that re-scores the top fifty or hundred candidates and picks the actual top five. Rerankers are cheap compared to the generator and they consistently move the precision needle. If retrieval is the bottleneck, the reranker is the cheapest place to relieve it.

The pipeline, with the failure modes labelled

Every box on the diagram below is a place the wheels can come off. The conversation usually goes faster when the failure surfaces are named on the same picture as the happy path.

When a production system regresses, walk the diagram left to right and ask which box stopped doing its job. The bug is almost never in the box you suspect. It is usually two boxes upstream of where the symptom shows up.

The evaluation problem, finally measurable

The thing that changed most in the last eighteen months is that retrieval quality is now measurable in production, not just in offline tests. The open-source framework RAGAS formalised the metrics that matter: faithfulness (does the answer stay inside what was retrieved), answer relevance (does it address the question), context precision (are the retrieved passages on-topic), context recall (did retrieval surface everything needed). Other frameworks exist; the convergence on roughly this set is the more important signal than which one you pick.

The eval gap most teams discover on day forty-five is the one between offline test-set accuracy (which looked great) and production answer quality (which is worse than the demo promised). The cause is almost always query-side drift. The test set was built from queries the team thought users would ask. Real users ask different questions, in different language, about different documents, often referencing things that have changed since indexing. A retrieval system optimised against a static test set is a clock that tells the right time only at noon.

The discipline is to evaluate on live traffic, sampled and labelled continuously, with the metrics above tracked over time. The broader practice of measuring the full system is the evaluation and benchmarking discipline; on the retrieval side specifically, the controls are: track context precision and recall weekly, alert when faithfulness drops, sample failed queries for manual review, fold the learnings back into chunking and retrieval.

Where document RAG genuinely breaks

Three failure modes are worth naming because every team hits them.

Long documents. A two-hundred-page report does not chunk gracefully. The answer to “what changed between the 2024 and 2025 versions” needs both held end-to-end; chunked retrieval gives you fragments. Hierarchical retrieval (summary chunks at the top, detail chunks below) helps, but the honest answer is that some queries want a whole-document reader, not a chunked one.

Tables. Tables encode meaning in two dimensions. Most chunkers flatten them to one and the column-row relationships dissolve. Specialised table extraction is its own sub-discipline; a chunker that treats a table the same as a paragraph is shipping a known bug.

Multi-modal documents. Diagrams, charts, scanned pages, handwritten annotations. Text-only retrieval misses everything that is not text. Multi-modal embedding is maturing fast in 2026; a serious system for the next eighteen months needs a retrieval path for non-text content, not a polite “currently supports text.”

What the discipline now looks like

Document RAG is no longer something you stand up. It is something you operate. Chunking is an engineering decision with measurable consequences. Retrieval method is a tradeoff you evaluate, not a setting copied from a tutorial. Hybrid retrieval with a reranker is the production default. RAGAS and its peers turned retrieval quality from a vibe into a number, and the teams that measure out-ship the teams that do not.

The deeper point, the one that connects back to the LLM harness, is that retrieval quality is upstream of trust. Whatever happens at the citation surface, credibility was decided here, in the chunking and the retrieval, weeks earlier and without ceremony. Get the brickwork right and the rest of the building has a chance.