LLM Observability: The Production View of the Harness

LLM observability is just observability applied to a new kind of system. The signals come in the same three shapes a serious engineering team has shipped for two decades (traces, metrics, logs), they roll up into the same dashboards, they fire the same kind of pager at three in the morning. What is different is that an LLM system has three signal classes traditional application performance monitoring does not know how to read: faithfulness scores, guardrail trips, and refusal events. The whole production discipline turns on whether you instrument those three the same way you instrument latency and error rate, or whether you treat them as a side panel a data scientist looks at once a week.

I am seeing a lot of teams in 2026 still treating them as the side panel. That is the same mistake the early web ops teams made with response time before the SRE discipline named it. The metric that decides whether your system is trustworthy lives in a notebook somebody opens on Friday afternoon, and the on-call rotation does not get paged when it slips. By the time the customer notices, the regression has been live for a week.

This piece is the production view that sits underneath the rest of the observability and audit work for LLM systems. It assumes you have an audit trail writing structured records for every call and that data integrity holds at the corpus layer. With those two in place, observability is the live dashboard on top: the rolling reads, the alerts, the thing the engineer opens at 3 AM.

The three signal classes APM does not have

Traditional APM watches latency, error rate, throughput, and saturation. Those still matter for an LLM system. They are necessary and they are not sufficient. The three new classes:

None of these read like classic APM. All three behave like classic APM once you have them on a rolling time-series with thresholds, alerts, and a runbook for each. The discipline is the move from “we score faithfulness in the eval set” to “we score faithfulness on every production call and chart the rolling distribution.”

They map onto the primitives you already have

Trace, metric, log. Same shapes you have shipped since 2010, with new fields.

• Trace. A single end-to-end call. The span tree is what you already know: request span, retrieval span, model-inference span, guardrail span, tool-call span, response span. The new fields on the spans are gen_ai.system, gen_ai.request.model, gen_ai.prompt, gen_ai.completion, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens. The shape of the span is identical to a database call or an HTTP request, the attributes are LLM-specific.
• Metric. A rolled-up number over time. Faithfulness p50 and p95 by route. Guardrail trip rate by category. Refusal rate per endpoint. Tokens per request. Cost per request. Cache hit rate on the retrieval layer. Every one of these is a time-series the operations team can chart, threshold, and alert on with the same tooling they already run.
• Log. A structured event with context. The append-only audit trail already writes one per stop on the call path. Observability turns that stream into queries: “show me every refusal in the last hour for endpoint X with the prompt that triggered it.” Same query language, new attributes.

The lesson from a decade of SRE practice carries cleanly: do not invent new tooling where the old tooling fits. Run your LLM observability through the same OpenTelemetry collector, the same Prometheus or your equivalent, the same Grafana, the same on-call rotation. The page that wakes you at 3 AM should have the same shape whether the binding metric is latency or faithfulness; the engineer’s muscle memory matters more than the metric’s novelty.

OpenTelemetry GenAI semantic conventions: the cross-vendor anchor

The thing that nearly killed early LLM observability was every framework inventing its own schema. LangChain spans looked one way, LlamaIndex spans another, OpenAI’s native traces a third. The dashboards never composed.

The OpenTelemetry Semantic Conventions for Generative AI Systems are the fix. They specify the attribute names every framework and vendor should emit on a model-inference span (gen_ai.system, gen_ai.request.model, gen_ai.usage.input_tokens, and the rest), so a single backend can ingest spans from any framework and the queries do not care which library produced them. The convention is what you anchor on before you pick the backend. The backend is interchangeable; the convention is the contract.

This is the same pattern the W3C trace context took for distributed tracing in 2019. The convention is boring, and boring is the point. Boring conventions outlive every framework that tried to own the schema.

What NIST AI 600-1 expects you to be able to answer

The NIST AI 600-1 Generative AI Profile is the US reference frame for what the operator of a generative AI system is supposed to be able to demonstrate. The Measure function in the AI RMF, read against the Profile, expects an operator to answer four questions on demand:

1. How is the system performing right now, on the dimensions that matter for its risk profile (faithfulness, refusal rate, guardrail trips, harmful-content rate)?
2. How has it performed over the last 30 / 90 / 365 days, with the evidence to back the claim?
3. When the system deviates from its baseline, who is notified, and what is the runbook?
4. When the system is updated (a model version change, a prompt change, a corpus refresh), how do you know the deviation is from the change and not from something else?

Every one of those is an observability question, not a documentation question. You cannot answer them from a PDF; you answer them from a dashboard, an alert history, and an annotated time-series. The Profile is the regulator stating the bar; the observability layer is the operator’s mechanism for clearing it.

The observability stack, drawn

The stack reads top to bottom. The instrumented call path emits via the OpenTelemetry GenAI semantic conventions. Five signal classes feed the single pane of glass: the three classic ones plus the two LLM-specific ones. Production tooling like Arize Phoenix, LangSmith, Langfuse, and Helicone has moved in the last eighteen months from “developer trace viewer” to “production observability surface with retention, role-based access, and alert hooks,” which is the shape the discipline grew into.

The day-2 reality

Nothing about this is exotic once the instrumentation is in place. The faithfulness rolling p50 drops below 0.82 on the customer-support route, the on-call gets paged, the runbook says check the last corpus refresh and the last model version, the engineer looks at the trace and sees retrieval returning a stale chunk, the fix is the same shape as fixing a stale cache. The alert was the same shape as a latency alert. The fix was the same shape as a database fix. Only the metric was new.

That is the whole production view. Run the LLM system on the same observability discipline you would run any system that matters. The signals are slightly stranger. The runbook is the same.

Resources

• Observability and audit for LLM systems
• Data integrity for LLM systems
• Audit trails for LLM applications
• Evaluation and benchmarking
• Guardrails and runtime controls
• Regulatory reference: NIST AI 600-1, Generative AI Profile of the AI Risk Management Framework
• Cross-vendor signal standard: OpenTelemetry Semantic Conventions for Generative AI Systems
• Production tooling: Arize Phoenix open-source LLM observability platform