Wikidata and External Knowledge Graphs as Ground Truth for LLM Output QC

The first time I ran a model output through a Wikidata check I felt the same small joy I get from a well-tuned guitar. The model had cheerfully informed me that a French AI startup was founded in 2022. The check came back: QID Q117459408, founded 2023. Two seconds of SPARQL turned a confident-sounding paragraph into a corrected one. I am seeing teams reach for hand-rolled regex and citation-extraction pipelines for this job, when a public, curated graph with stable identifiers will do the work cleanly. The operational unlock is brutally simple: ask, for every entity claim in the output, would this survive a SPARQL query against Wikidata. If the answer is no, the model has produced an extrinsic hallucination, and you have caught it before the user did.

This is the second face of the symbolic-versus-semantic split the parent piece draws. The other face, LLM-as-judge, is probabilistic by construction: one model evaluates another, and the verdict is only ever as good as the judge. Wikidata is symbolic: the relationship either holds in the graph or it does not. When ground truth exists, hallucination becomes catchable, not just preventable. That difference is what makes the symbolic pattern the cheapest E-E-A-T win in the harness and the one most production stacks under-build.

What Wikidata actually is (and why a QID matters)

Wikidata is the structured-data sibling of Wikipedia, run by the same foundation, edited by the same kind of crowd. Every entity has a stable identifier called a QID (Q-number): Mistral AI is Q117459408, the European Union AI Act is Q117228499, the city of Antwerp is Q12892. Properties have P-numbers (P571 is “inception date”, P159 is “headquarters location”). Every claim in the graph is a (subject, property, object) triple anchored to those IDs, with a provenance reference attached. The whole thing is queryable in SPARQL at the public endpoint.

What the QID buys you is disambiguation without surface-form games. “Anthropic” the company is Q116867316. “Anthropic” the adjective is something else entirely. A model that writes “Anthropic was founded in San Francisco in 2021” makes two checkable claims, and the QID is the hinge that lets a deterministic check resolve them. Without the QID, you are matching strings, which is the road to false positives on every common name and false negatives on every translated one.

Entity linking is the bridge from prose to QID

A model’s output is text. Wikidata is a graph of IDs. The piece that connects them is entity linking, the task of taking a span of text (“Mistral AI”) and resolving it to a knowledge-base ID (Q117459408). For Wikidata-grounded fact-checking, the entity linker is the rate-limiting step. If you mis-link, every downstream check verifies the wrong entity and the whole pass returns a confident, wrong answer.

The state of the art is good enough to use in production. BLINK (Wu et al., 2020) introduced dense-retrieval entity linking that handles zero-shot cases (entities the linker never saw at training time) by encoding candidates with a bi-encoder. ReFinED (Ayoola et al., 2022) tightened the speed-quality trade-off enough to make end-to-end linking practical on commodity hardware. Both ship as libraries. Neither requires a research team. The honest read on entity-linking accuracy: high for common, well-documented entities (organizations, geographies, public figures), middling for niche or recent ones (a startup founded last quarter may not be in Wikidata at all yet), and brittle for ambiguous surface forms.

The pragmatic pattern is to score the link’s confidence and route low-confidence claims to a different path (human review, semantic judge, or refusal) rather than treat every link as gospel.

The post-generation symbolic check

The simplest, highest-leverage use of Wikidata in an LLM stack is a post-generation check: the model writes its answer, a downstream pass extracts the entity claims, links them to QIDs, queries Wikidata, and either confirms, contradicts, or marks unknown. The shape is a small pipeline that adds 200-800 milliseconds and catches a class of error the model itself cannot.

Three outcomes, three actions. Confirm is the boring case and the most common, which is the point of running it. Contradict is the gold: a probabilistic problem (did the model lie) turned into a deterministic one (the graph says 2023, the model said 2022). Unknown is the honest case the literature undersells. Wikidata does not know everything; when the graph cannot answer, you route the claim to a semantic mitigation or refuse. Confusing “not in the graph” with “the model is wrong” is the most common failure mode of teams new to this pattern.

Catching versus preventing: post-hoc check vs RAG over the KG

This article is about the post-hoc symbolic check, where you let the model generate and then verify. The same Wikidata graph can also be used the other way around: as a retrieval source the model reads from before generating. That is knowledge-graph-grounded retrieval, a cousin of standard RAG with the corpus being structured triples instead of unstructured passages. The deep dive on the retrieval side lives in Retrieval and grounding.

The two patterns are complementary, not redundant. Retrieval-grounding prevents the hallucination by giving the model the right facts up front; post-hoc checking catches the hallucination that slipped through because retrieval was incomplete, the model ignored its context (a “context hallucination”, per the three-class taxonomy), or the answer mixed retrieved and parametric knowledge. The mature stack uses both. The audit log loves the second one, because the check is deterministic and the result is binary.

Where Wikidata earns its keep, and where it does not

Wikidata is excellent for a specific shape of claim and a poor fit for others. The discipline is knowing which is which before you build.

Strong fit (use Wikidata): – Entities and their core attributes. Companies (founding date, headquarters, parent organization), people (birth date, nationality, role), products (manufacturer, release date), works (author, year, publisher). The graph is dense here, the provenance is good. – Geography. Cities, countries, regions, coordinates, administrative hierarchy. Effectively complete. – Dates and identifiers. ISBNs, DOIs, ISO country codes, ticker symbols. These cross-reference well. – Regulated facts. Named laws, regulations, treaties, standards bodies. The EU AI Act has its own QID; the GDPR has its own QID; this is the layer that pairs cleanly with the EU AI Act high-risk requirements.

Poor fit (use something else): – Subjective claims. “The best framework for X”, “the leading vendor in Y”. No symbolic ground truth exists; this is semantic-judge territory. – Recent events that have not been encoded yet. A funding round announced last week may not have a QID for a month. The graph lags; treat its silence as “unknown”, not “false”. – Niche or domain-specific entities below the Wikidata coverage threshold. Small open-source projects, specific contract clauses, internal product names. For these, a domain-specific knowledge graph (your own or a vendor’s) is the right tool, not the public graph. – Reasoning chains. Wikidata can verify that A is the parent of B and B is the parent of C; it does not opine on whether the model’s three-step argument from A to C is sound.

The maintenance question: staleness, gaps, and the cost of being wrong

The honest weakness of any external knowledge graph is freshness. Wikidata is updated continuously by a volunteer community; coverage is uneven, and the entities the model is most likely to hallucinate about (new startups, freshly named regulations, last week’s product launches) are exactly the entities most likely to be missing or out of date. Treating “no QID” as a failure produces too many false positives; treating it as a pass produces too many misses. The mitigation is the third outcome in the diagram above: when the graph is silent, route to a different check. Do not collapse the three-way decision into a binary one.

A second cost is provenance discipline. A Wikidata claim carries a reference; high-stakes pipelines should fetch that reference and prefer claims backed by primary sources over those backed by Wikipedia infoboxes, which are themselves derivative. The graph supports this natively; most teams using it do not bother. The bother is what separates a hobby check from an auditable one.

A third cost is the internal-KG question. For a regulated industry or a proprietary product line, Wikidata’s coverage will always be incomplete. The pattern then is to maintain a domain-specific knowledge graph alongside Wikidata, with the same QID-anchored shape, the same SPARQL surface, and the same provenance discipline. The post-generation check queries both: Wikidata first (free, public, broad), the internal graph second (curated, private, deep). The harness reads them as a layered ground truth, which is exactly what an audit log wants to record.

The framing the harness deserves

Symbolic checks against an external knowledge graph are the cheapest reliability win most production LLM stacks skip. The component is small, the latency is low, the result is deterministic, and the audit trail writes itself. When the symbolic ground truth exists for a domain, hallucination becomes a defect you can engineer against rather than a probability you can only hope to reduce. That is the difference between a system you ship and a system you trust.

Resources

• Parent: Hallucination taxonomies and mitigations
• Sibling: The three classes of hallucination (deep dive)
• Sibling: Reducing hallucinations: the response layer as a stack (deep dive)
• Sibling: When hallucination is a design constraint, not a bug (deep dive)
• Sibling: LLM-as-judge: where it works, where the verdict fails (deep dive)
• Upstream: Retrieval and grounding
• Primary source: Wikidata (Wikimedia Foundation)
• Primary source: Ayoola et al., “ReFinED: An Efficient Zero-shot-capable Approach to End-to-End Entity Linking” (2022)
• Primary source: Wu et al., “Scalable Zero-shot Entity Linking with Dense Entity Retrieval” / BLINK (2020)
• Production write-up: Galitsky et al., KG-grounded fact-checking in production