Continuous Evaluation: When the Benchmark Is Your Production Traffic
The held-out test set was a useful fiction. It got us through the first wave of putting language models into real workflows; everyone needed something to point at and say “we tested this.” But the longer I watch teams operate these systems in production, the clearer it becomes that the only honest evaluation surface is the queries your users actually send. Everything else is a rehearsal in a room your audience will never sit in.
I am not arguing the held-out set is worthless. It still catches regressions, and it still earns its keep as a release gate. The argument is narrower than that. The held-out set is a snapshot of what the team thought to ask the system, frozen at the moment the set was built. Production traffic is the thing that is actually happening. Two weeks after launch they have already diverged, and the gap only grows from there. If your only eval surface is the snapshot, your evidence base is going stale on a schedule you cannot control.
Continuous evaluation is the move from snapshot to stream. A small, sampled slice of live traffic flows back into the same evaluation framework that ran at release, scored on the same metrics, on a rolling window. The held-out set still has a job (regression). The stream has a different job (drift, novelty, the corner cases nobody anticipated). Both run; neither replaces the other.
The hot path and the cold path
The architectural shift that makes continuous evaluation tractable is separating the path that serves users from the path that scores what got served. The user-facing request is latency-sensitive and runs on the hot path. The evaluation runs asynchronously on the cold path, against a sample of the traffic that already shipped. The user never waits for an eval to complete; the team never has to choose between fast responses and good evidence.
The hot path is unchanged from what the team already runs. The cold path is the new investment, and it is the one that buys the evidence. Tools have crystallised around exactly this shape: LangSmith, Arize, Braintrust, and the open-source Helicone all sit on the cold path, ingest traces from the hot path, and score them on a rolling window. The naming differs; the topology is the same.
Trace capture: the prerequisite
Continuous evaluation is impossible without traces, and traces are not the same thing as logs. A log says “request 47 returned 200.” A trace says: this prompt, with this retrieved context, hit this model variant, tripped this guardrail at this step, produced this output, in this many milliseconds, costing this many tokens, with this confidence score on each retrieved chunk. The trace is the full record of one request as a structured artefact, captured at the moment it happened. You cannot reconstruct it later. Either the instrumentation was there or the evidence is gone.
The non-negotiables for trace capture:
| Captured per request | Why it earns its place |
|---|---|
| User query (full, after PII redaction) | The eval input |
| Retrieved context (chunks + scores + source IDs) | Lets faithfulness and context-precision scoring re-run offline |
| Final prompt sent to the model | Catches prompt drift after a silent template change |
| Model output (raw, pre-postprocessing) | The eval target |
| Guardrail events (which fired, what blocked, what passed) | Lets you score refusal accuracy and detect over-blocking |
| Latency, cost, model variant | Connects quality drift to infra changes |
| Trace ID, user session (anonymised), timestamp | Joins back to the audit log the regulator wants |
If a trace is missing any of the first four, that request is invisible to the eval suite. Teams that bolt observability on after launch always discover they forgot one, and the eval is biased on whatever the missing field correlates with. Capture it all from day one, then sample on the cold path. Sampling at trace-capture time is a mistake you discover six weeks too late.
You cannot evaluate every request: sample on purpose
A serious production system serves enough traffic that scoring every request is uneconomic. Arize‘s own guidance and the patterns in Braintrust converge on three sampling strategies, used together:
- Volume-weighted random. A flat percentage of all traffic (1 to 5 percent is typical) gives you the baseline. It catches average-case drift and gives you a denominator for everything else.
- Stratified by intent. Sample more aggressively from low-volume intents that matter (a refund-policy question is rarer than a balance check but the cost of getting it wrong is higher). Stratification stops the rare-but-important cases from being statistically invisible.
- Importance-weighted for high-stakes workflows. Any trace where a guardrail tripped, the user gave negative feedback, or the workflow touches a regulated decision (credit, hiring, clinical recommendation) gets sampled at 100 percent. These are the traces the auditor will ask about; they are also the traces where eval pays back the most per dollar.
The combination gives you statistical power on the bulk of traffic and full coverage on the slice that matters. The single biggest sampling mistake is pure random at low rates: you get a clean number on average behaviour and total blindness on the tail, which is where the failures actually live.
Drift detection: when production diverges from the eval set
A held-out eval set scores well at release. Six weeks later, faithfulness has dropped four points and nobody can tell you when. That is drift, and it has three usual sources: the input distribution shifted (users started asking different things), the retrieval corpus shifted (documents added, removed, re-chunked), or the model shifted (a provider pushed a silent update, common with hosted models). All three look identical from the metric alone; the trace data is what lets you tell them apart.
The mechanic is straightforward. Compute the same metrics (faithfulness, refusal accuracy, context precision) on a rolling 7-day window of sampled production traffic, compare against a 30-day baseline, alert when the delta exceeds a threshold calibrated against historical noise. The deeper theoretical work on this lives in the dataset-shift literature, notably Rabanser, Günnemann, and Lipton’s Failing Loudly study, which catalogues which shift-detection methods actually fire on real distribution changes versus which fail silently. The applied lesson is short: detect on the eval-metric output, not on raw input distributions, because the eval metric is what the user actually experiences.
The threshold matters more than the metric. Set it too tight and you page on noise; set it too loose and you discover the regression in next quarter’s review. Calibrate against the historical noise floor: two weeks of normal operation tells you the natural week-to-week variance, and the alert threshold sits at two or three standard deviations beyond that. Boring statistics; load-bearing in practice.
The feedback loop, made cheap on purpose
The point of the cold path is not the dashboard. The point is the loop. A drift alert fires, the on-call engineer pulls the offending traces, the eval suite explains which metric moved (faithfulness dropped, context precision held, retrieval source distribution shifted), and the fix lands in the right layer (a retrieval and grounding tune, a guardrail threshold change, a prompt rewrite). The same eval suite reruns on the next sample window and confirms the fix. Then it lands in production.
This is the difference between evaluation as a one-off proof and evaluation as a control. A one-off proof gets stored in a slide. A control gets wired into the deployment pipeline: no deploy ships unless the eval suite passes on the latest 7-day window, no rollback gets reverted until the same suite confirms the rollback worked. The same audit log the EU AI Act requires for high-risk systems is a near-free byproduct of running the cold path properly; you were going to capture the traces anyway.
A held-out test set proves the system worked once. Continuous evaluation proves it still works this morning, against the traffic it actually got, and produces the log that says so. That is the only evidence base that ages well. Everything else turns into a snapshot of a room nobody is sitting in anymore.
