When a Hallucination Is a Design Constraint, Not a Bug

A memory leak is a property of how programs are. A race condition is a property of how concurrent programs are. A cosmic-ray bit-flip is a property of how dense storage media are. None of those are bugs. They are facts about the substrate, and a mature engineering discipline grew up around each one by designing AROUND the property rather than pretending it could be eliminated. Hallucination is the same shape of fact about probabilistic generators, and the same intellectual move is finally happening in the LLM stack. The teams that have made the move ship responsibly. The teams that have not are still selling the next version that will “fix it.”

I am seeing this split clearly in the briefs I read. The vendor pitch decks promising “zero hallucinations” land at the same prospects as the architecture diagrams that build refusal patterns, confidence bands, and human-review thresholds into the UI. Both are addressing the same property. Only one is honest about what it is.

The three engineering analogies that have already been here

Software engineering has been through this exact maturation curve three times. Each one is worth holding in your head when someone on your team insists hallucination is a temporary defect awaiting a patch.

The memory leak. In the C era, freeing memory was the programmer’s job, and forgetting was so easy that “leak” became a class of defect with its own tooling (valgrind, address sanitizers, automated leak detectors). The mature response was not “we will write code that never leaks.” It was reference counting, garbage collection, RAII, ownership models. The substrate has a property; you build the property into your design. Today, a senior C++ engineer who claims their code “has zero possibility of a leak” tells you they are junior. A senior one tells you about their leak budget and their detection harness.

The race condition. Concurrent code has a property: independent threads observing shared state will sometimes interleave in ways the author did not anticipate. The mature response was not “we will eliminate races.” It was locks, lock-free data structures, actor models, transactional memory, formal verification of the dangerous bits. The honest engineer ships with a race budget, an instrumentation layer that catches the ones that escape, and a runbook for what to do when they do.

The cosmic-ray bit-flip. Dense storage has a property: at scale, cosmic radiation will flip bits in DRAM and on disk often enough to matter. The mature response was not “we will harden the silicon until flips stop.” It was ECC memory, RAID parity, checksumming filesystems, end-to-end checksums in distributed protocols, replication. Google operates at a scale where bit-flips happen continuously; they engineered the system to assume they will and to detect and correct them. The substrate has a property; the design absorbs it.

Hallucination sits cleanly alongside these three. It is a property of how transformer-based probabilistic generators work. You can reduce it (better data, RAG, constitutional training, post-hoc checks). You cannot make it zero, any more than you can make a long-running C++ program statistically immune to a single byte going stale. The mature posture is to design AROUND the property: instrument it, budget it, catch it, surface it. The immature posture is to keep promising the patch that ends it.

What designing-around looks like, in four concrete moves

Once you accept hallucination as a substrate property, the design surface opens up. Four moves do most of the work, and each one corresponds to a decision that has nothing to do with the model and everything to do with the harness around it. (The parent piece walks the whole harness; this one is about the design posture.)

Confidence calibration. The model has internal signals about how sure it is, and the well-known Kadavath et al. 2022 result is that those signals are surprisingly well-calibrated when you sample them properly (“Language Models (Mostly) Know What They Know”). Surface that calibration to the UI. A bare claim presented with 95% confidence is not the same artefact as a claim presented with 40%. Most production UIs throw the confidence away and present every output with the same visual weight, which is the design equivalent of returning every database query without telling the caller whether the result is from cache, a fresh read, or a stale replica.

Refusal patterns. The model can be trained, prompted, and gated to say “I do not know” or “I will not answer that without a source.” Refusal is a design choice, not a degradation. A legal-research assistant that refuses every claim it cannot ground against a citation is doing exactly what a competent paralegal does on day one of the job. Anthropic’s constitutional AI work is one explicit version of this: the model is shaped to refuse in classes of cases the harness defines as out of scope. The teams that resist refusal patterns (“our users want answers, not refusals”) are confusing helpfulness with confidence-theatre.

Human review thresholds. Not every output needs a human in the loop. A clinical decision-support tool with high stakes and low decision-frequency wants every output reviewed. A marketing copy generator with low stakes and high decision-frequency wants no reviews until the model flags itself uncertain. The threshold is a slider, not a switch, and the slider position is a design parameter tied to the workload’s cost of being wrong. The mistake is shipping the same threshold to every workload because “the model is good enough now.”

Decision-vs-information UIs. This is the one most product designers miss. There is a fundamental difference between a UI that gives the user information (“here are three options, here is what I found on each”) and a UI that gives the user a decision (“the answer is X”). The first puts the user in the loop; the second takes them out. Both are valid; they are not interchangeable. A high-stakes domain almost always wants the information UI even when the model is technically capable of the decision UI. NIST’s AI 600-1 Generative AI Profile formalises this as a risk-tier concern: the harm class shifts when the system moves from informing to deciding.

How confidence buckets map to UI treatment

The four moves combine into a decision tree the product team can actually implement. Pick a confidence bucket (the calibrated signal from the model), and the UI treatment falls out of it.

The shape of the tree is the point. Every output gets a treatment matched to its confidence, and the workload owner decides where the thresholds sit. A creative-writing tool slides every threshold left (more tolerance for low-confidence extrapolation); a clinical tool slides every threshold right (more refusals, more reviews). The model does not change; the harness does. That is what designing-around looks like.

The honest tension

This framing offends two kinds of team. The first is the vendor selling “zero hallucinations” as a feature; the framing reveals that line as marketing rather than engineering. The second is the buying team that wants to believe the line, because believing it lets them skip the harness work. Both are common, and both are temporary. The market is shifting the same way memory-safe languages shifted the C++ world: the responsible posture wins because the irresponsible one ships incidents.

The third group is the one this piece is for. Senior practitioners who have already accepted the property and are looking for the design vocabulary to talk about it with their teams. The four moves above are that vocabulary. The decision tree is a starter; the real version lives in your product’s risk profile and your users’ tolerance for each kind of wrong. Build the harness to the workload, not to the marketing promise. The model is a probabilistic generator. Engineer accordingly.

Resources

• Back to: Hallucination taxonomies and mitigations (parent)
• The three classes of hallucination: intrinsic, extrinsic, systemic
• Reducing hallucinations: the response layer as a stack
• Wikidata and knowledge-graph grounding for output checks
• LLM-as-judge: where it works, where the verdict fails
• Hallucinations, Guardrails, and the LLM Harness (the parent piece)
• Primary source: Anthropic, Claude’s Constitution (Constitutional AI)
• Primary source: Kadavath et al., “Language Models (Mostly) Know What They Know” (2022)
• Regulatory reference: NIST AI 600-1, Generative AI Profile