Context Window Management: Truncation, Eviction, and the Failure Mode When the Model Runs Out of Room

The first time I caught this failure in the wild, it was in a long customer-support conversation. The agent answered correctly, walked the customer through three follow-ups, then on turn nine contradicted what it had said on turn one. The system prompt that pinned the product policy had quietly fallen out of the window twenty minutes earlier. The model did not warn anyone. It kept answering in the same composed register, against a context that no longer contained the rule.

The team called it a hallucination. It was not. The model was doing what models do, which is generate plausible continuation against whatever context it has. The bug was the eviction policy. Nobody had decided what the window should drop when it filled. The default did the dropping, the default dropped the wrong thing, and the audit log had no way to flag the missing rule because no one was watching the window’s contents as a state worth observing.

This is the guardrail nobody documents. When context exceeds the budget, the eviction strategy is the runtime control. What falls out, in what order, and what gets summarised instead of dropped is a policy decision with downstream effects on every answer that follows. Treat the window as plumbing and it will burn you the way every untreated runtime control burns its operators: silently, late, on the call you were not in the room for.

The window is a budget, and budgets need a policy

A language model’s context window is a fixed token budget. Every system prompt, retrieved passage, conversation turn, and tool result spends from it. The moment the cumulative spend exceeds the budget, the harness has to make a choice. Either the framework in front of the model truncates, or the API does, or both. The question is which tokens leave.

Three strategies dominate in production code, each with a different failure profile.

Earliest-first (FIFO). Drop the oldest tokens until the input fits. Easy to implement, almost always wrong for system prompts and long conversations, because the oldest tokens are usually the ones that pinned the policy, the role, the schema, or the user’s original intent. FIFO is the default in a lot of naive harness code, and it is the policy that bit the customer-support agent in the opening paragraph.

Least-relevant-first. A scoring pass ranks tokens by embedding similarity to the current turn and drops the lowest-scoring segments. Softer than FIFO in the average case, but with its own failure: a passage that scores low against the current turn’s surface vocabulary may still be the load-bearing constraint for the answer. A policy clause about regulated industries may not match a customer’s casual phrasing about “what can you actually book me for,” even though the clause is exactly what should determine the answer.

Summarise-and-replace. When the window approaches its budget, run a summarisation pass over the oldest N turns and replace them with a single compressed summary that stays pinned. The conversation gains a memory tier (recent turns at full fidelity, older turns at compressed fidelity, the system prompt always preserved). This is what serious chat products do, and it is what practitioner write-ups, including the DeepLearning.AI short course on building systems with ChatGPT, walk teams through as the production-grade pattern. The cost is real (an extra model call per summarisation, and the summary can distort a detail that matters), but the trade is usually worth it because the alternative is silent loss of constraint.

The right policy is rarely one of the three. It is layered: system prompt stays pinned, user’s stated goal stays pinned, retrieved passages get evicted in least-relevant order, and the long tail of conversation history gets summarise-and-replace. The harness designer who picks one strategy and applies it uniformly is the one whose pager goes off at 3am.

Why long-context models help and do not solve it

The obvious read is that as context windows have grown from 4K tokens to 200K and beyond, the eviction problem has gone away. It has not. It has changed shape.

The reference paper here is Liu et al., “Lost in the Middle: How Language Models Use Long Contexts” (2023), and its central finding has held up through three years of model changes. When relevant information sits at the very beginning or the very end of a long context, the model uses it well. When the same information sits in the middle, performance degrades, sometimes dramatically. The attention mechanism is not uniform across the window; the middle is where things go quiet.

That changes the calculus. Having room to fit everything is not the same as the model attending to everything. A 200K-token context that crams the load-bearing constraint into position 90,000 is, for practical purposes, no better than an 8K context that summarised its way past it. The mitigation is the same shape as the eviction problem: position the things you most need the model to honour at the edges (system prompt at the top, the most recent and most relevant retrieved passages at the bottom), and accept that the middle is a soft zone where attention dilutes.

Both Anthropic’s context-window guidance and OpenAI’s prompt-engineering guidance name the same habit: pin what must hold, treat the middle as fungible, and budget for compression, not raw growth. The vendors are not telling you long context solves eviction; they are telling you the problem moved.

Hallucination by amnesia

This is the failure pattern that deserves a name, because once you have the name you start seeing it everywhere.

Hallucination by amnesia is what happens when context the model was meant to honour falls out of the window, the model is not informed, and the answer keeps coming with the same fluency it had when the context was present. The answer is not wrong because the model invented a fact. It is wrong because the model is generating against a context that no longer contains the constraint, and there is no signal in the output that anything has changed. The voice is the same. The grounding evaporated and the speaker did not notice.

The reason this slips past every other guardrail is that the input gate, the output gate, and the policy gate all evaluate the surface of the call, not the contents of the window. The input gate sees a clean user turn. The output gate sees a fluent, schema-valid response. The policy guard sees a request that matches no forbidden pattern. None of them is positioned to notice that the constraint they were meant to enforce was never visible to the model on this turn. The amnesia is invisible to the rest of the harness by design.

The mitigations are mechanical: instrument the window itself, not just the inputs and outputs. Log which tokens are in the window on every call, log every eviction event with the displaced span, and run an audit-side check that fires when a previously-pinned span leaves the window without the harness having authorised it. If the system prompt has been evicted, the next call should not silently proceed. It should re-pin, summarise the missing context back in, or refuse. The model will not raise its hand. The harness has to.

The control surface, in summary

Pick the eviction strategy deliberately. Pin what must hold. Position the load-bearing context at the edges, not the middle. Summarise rather than drop, when the alternative is silent loss of constraint. Log the window’s state, not just the API’s input and output. And give your harness a way to notice that the amnesia has happened, because if you wait for the model to notice, you will be waiting a long time.

This is the control that gets treated as a configuration field and quietly bites. Treat it as a runtime control surface, the same way you treat input and output gates, and it stops being the one that wakes you up at 3am.

Resources

• The wider guardrail and runtime control layer
• What LLM guardrails are, and what they cannot do
• Prompt injection and the OWASP Top 10
• Output handling and structured response gates
• Context as a guardrail
• The whole LLM harness
• Observability and audit for LLM systems (deep dive)
• Research reference: Liu et al., “Lost in the Middle: How Language Models Use Long Contexts” (2023)
• Vendor guidance: Anthropic, Context windows guidance
• Vendor guidance: OpenAI, Prompt engineering guidance
• Practitioner reference: DeepLearning.AI, Building Systems with ChatGPT (conversation summarisation pattern)