Reducing hallucinations: what the response layer actually does

The first time I watched a team try to “fix hallucination” with a single technique, they spent three months tuning a retrieval pipeline and shipped a system that hallucinated almost as much as the day they started. The retrieval was excellent. The prompts were clean. The model still made things up, just in different places. The lesson they learned the hard way, and the one I want to put at the front of this piece, is that no single technique kills hallucination. The stack does. Each layer kills a different class of the defect, and a stack with a gap leaks at exactly that gap.

This is the response-layer view of the three classes of hallucination the parent piece names: intrinsic (output contradicts the input), extrinsic (output adds claims the input does not support), and systemic (the harness around the model fed it bad context or no constraints, and the model dutifully answered). Three classes; different originators; different mitigations. The point of this article is to draw the stack and show which layer reduces which class, because that mapping is the move most teams miss.

The stack, with each layer mapped to the class it catches

Read it left to right and the architectural shape becomes the argument. Retrieval narrows the universe of things the model is allowed to talk about. The model writes. Self-consistency and chain-of-verification stress-test the model’s own reasoning before anyone else sees it. A faithfulness evaluator compares the candidate output against the retrieved input. Guardrails enforce policy at the boundary. The answer leaves. Each layer is doing different work, and pulling any one of them out re-opens a specific failure mode the next layer was not built to catch.

Retrieval kills extrinsic hallucinations (mostly)

Extrinsic hallucinations are claims the model adds with no grounding in the input. The model invented a board seat, a citation, a feature. The single most effective response-layer mitigation against this class is retrieval-augmented generation, because the system stops asking the model “what do you know about X” and starts asking “answer X using only this material.” That shift is structural, not stylistic. The model can still extrapolate beyond the passages it was given, but the surface area for extrapolation collapses, and a downstream check (does the answer cite anything; do the cited spans actually contain the claim) becomes mechanical.

The deeper retrieval treatment is in the retrieval and grounding piece on the same harness. The point here is the mapping: retrieval is the layer that addresses the extrinsic class. It does not catch the other two. A perfectly retrieved answer can still contradict the retrieved passages (intrinsic) or get released because no policy gate stopped it (systemic).

Faithfulness evaluation kills intrinsic hallucinations

Intrinsic hallucinations contradict the input the model was given. The summary says 2027, the contract says 2025. The translation flips a negation. The structured field is wrong even though the right number is sitting in the prompt. Retrieval cannot help here, because the retrieval already happened, and the input is correct. The defect is the model’s own contradiction of it.

The mitigation that maps to this class is a faithfulness evaluator. The pattern is a separate pass that takes the candidate output, the retrieved input, and a rubric, and scores whether every claim in the output is supported by something in the input. Huang et al.’s 2023 survey on LLM hallucination is the cleanest cross-section of the research literature on this class; the survey catalogues both metric-based faithfulness scores (NLI-style entailment between output and source) and LLM-as-judge variants. Either way, the evaluator’s job is single-purpose: catch the contradiction. The full evaluation discipline lives in the evaluation and benchmarking piece; for the present mapping, what matters is that faithfulness is the layer that catches what retrieval cannot.

Guardrails kill systemic hallucinations

Systemic hallucinations are the failures that originate outside the model. The retrieval layer returned irrelevant documents and the model answered anyway. The system prompt was ambiguous and the model interpolated a default. The post-processing layer failed to enforce a refusal and a wrong answer reached the user. The model behaved as a probabilistic engine should; the harness around it had no policy to enforce.

The layer that addresses this class is guardrails: refusal patterns (“if your confidence is below X, return the no-answer token”), output schema validation (the response must parse as JSON, must contain a citation array, must not include certain phrases), policy gates (rate limits, content filters, role-based access), and the operational discipline of failing closed rather than failing open. OWASP’s Top 10 for LLM Applications is the closest thing the practitioner literature has to a checklist for this layer, and Anthropic’s writing on refusal patterns and character training describes the model-side preparation that makes the guardrail layer’s job easier. The deeper piece is the guardrails and runtime controls article in the same series. For the mapping: guardrails address what retrieval and faithfulness cannot, because the defect is not in the model’s content, it is in the system’s failure to constrain.

Self-consistency and chain-of-verification are cross-cutting

The two cross-cutting mitigations are self-consistency (sample the same question multiple times, take the answer the model converges on) and Dhuliawala et al.’s chain-of-verification, where the model drafts an answer, generates verification questions about its own draft, answers each verification question independently, and then revises. Both techniques are model-internal: no retrieval upgrade, no external evaluator, no policy layer. They work by exploiting the same property that produces hallucination in the first place (the model is a probabilistic generator) and using sampling or self-interrogation to wash the high-variance answers out.

I am calling these cross-cutting deliberately. They are not pinned to a single hallucination class. Self-consistency tends to wash out both intrinsic and extrinsic errors when the underlying reasoning is unstable; chain-of-verification is particularly strong on extrinsic claims with discrete facts (dates, names, citations). Their place in the stack is “between the model and the evaluator,” because they cheaply improve the candidate before any external check spends compute on it.

The mapping, on one card

The diagonal is what most stacks get wrong. A team reaches for retrieval to fix intrinsic contradictions (the wrong layer; faithfulness is what catches that). A team reaches for an LLM-as-judge faithfulness pass to fix systemic failures (the judge cannot enforce a refusal that the surrounding code does not implement). A team reaches for guardrails to fix extrinsic invention (you cannot policy-gate your way around an absent retrieval). The mapping is the discipline. Pick the layer that addresses the class, do not patch one layer to do another layer’s job.

When the stack is honest about what it does not catch

There is one more move worth naming. The mature stacks I see in 2026 publish, internally, a short statement of what the stack is engineered to catch and what it is not. Intrinsic contradictions, caught at a measured rate. Extrinsic invention, caught when retrieval coverage is good. Systemic failures, caught when the guardrail layer’s rules are kept current. Cross-cutting low-confidence reasoning, partially caught through self-consistency, never zero. The unmitigated residue is real, and an audit log of where it shows up is what feeds the next iteration. A stack that claims to catch everything is a stack that has not measured what it actually catches; that conversation is the bridge into the evaluation and benchmarking piece.

The takeaway is the line I started with: no single technique kills hallucination, because the defect is not a single thing. Three classes, four layers, one mapping. Build the stack to the mapping and the failure mode becomes addressable. Build it to a single technique and you are tuning one layer to do the work of three.

Resources

• The three classes of hallucination: intrinsic, extrinsic, systemic (deep dive)
• Hallucination as a design constraint, not a bug (deep dive)
• Wikidata and knowledge-graph grounding for output checks (deep dive)
• LLM-as-judge: where it works, where the verdict fails (deep dive)
• Retrieval and grounding (parent topic)
• Guardrails and runtime controls (parent topic)
• Evaluation and benchmarking (parent topic)
• Back to: Hallucination Taxonomies and Mitigations
• Primary source: Dhuliawala et al., “Chain-of-Verification Reduces Hallucination in Large Language Models” (2023)
• Survey: Huang et al., “A Survey on Hallucination in Large Language Models” (2023)
• Foundational: Lewis et al., “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks” (2020)
• Practitioner guidance: Anthropic on refusal patterns and character training
• Practitioner checklist: OWASP Top 10 for LLM Applications