Context as a Guardrail: System Prompts, Few-shot, Retrieved Context, Tool-Call Context

There is a guardrail nobody on the call ever names. The team will argue for an hour about output schemas, refusal patterns, and which prompt-injection regex to bolt on the front. Nobody will mention the most decisive control of all, which is what the team chose not to put in the model’s context in the first place.

I keep coming back to this image. A sommelier does not pour ten wines and ask the diner to pick. She pours three, and the menu of three is the recommendation. The choice is in the omission. The same logic runs the context window. The system prompt, the few-shot examples, the retrieved snippets, the tool signatures: each one is a sommelier’s pour. The model can only act on what you set in front of it. Curate the context tightly and you have moved a class of failures upstream of every output filter you might write.

That is the frame for this piece. Context is not plumbing. It is a runtime control, and it has four faces, each with its own discipline and its own way of breaking.

The four flavours of context

Each lane carries a different kind of control, and each lane misfires differently. The discipline is not the same in any two of them.

1. The system prompt: the immutable rules

The system prompt is the closest thing a language model has to a constitution. It runs ahead of every turn, in a priority channel, and the model is trained to weight it above user messages. It is where you declare persona, scope, refusal posture, and what the model is never allowed to do regardless of how the user phrases the request.

Both major model vendors are explicit about this. The Anthropic prompt-engineering guidance and the OpenAI prompt-engineering guide converge on the same instruction: put policy and persona in the system prompt, keep it short enough to actually be followed, and never duplicate or contradict it across turns.

The failure mode is bloat. Teams keep appending. The system prompt grows from 200 tokens to 2,000, contradicts itself in three places, and the model starts cherry-picking which clauses to honour. The discipline is editorial. Write the system prompt the way you would write a contract opening: every clause earns its place, and the document fits on a page.

A second failure mode is treating the system prompt as a security boundary against a determined attacker. It is not. A user can socially engineer around it, and an indirect prompt injection from a retrieved document can drown it out entirely. The system prompt is a strong default, not a wall. Pair it with the output guard for the cases where the default loses.

2. Few-shot examples: behaviour by demonstration

Few-shot is the lever where the model learns the format and the edge cases from a handful of worked examples placed in the context window. It is the cheapest way to lift quality on a structured task, and it is the discipline most teams under-invest in.

The interesting empirical finding here, and one worth understanding before you write any few-shot block, is from Min et al., who showed that in-context learning works largely through pattern and format demonstration rather than from the example labels being correct in the strict sense. The model is reading the shape of the task, not memorising the answers. That has two operational consequences. First, the format of your examples matters more than the precise content; vary the format and you get inconsistent outputs. Second, examples that are individually plausible but collectively contradictory teach the model to be inconsistent.

The failure mode is over-fitting the prompt to the easy cases. A handful of clean examples and the model nails the test set; in production the first weird input arrives and the model has no demonstration to anchor on. The fix is to include the edge cases in the few-shot block, including a worked example of a graceful refusal. Show the model the boundary, not just the centre.

3. Retrieved context: the most contaminated surface

Retrieved context, the output of a retrieval and grounding layer, is the lane where the most powerful gains and the most quietly dangerous failures live. It is the only context lane where the bytes the model reads are not authored by you. They come from a corpus, a vector index, a third-party API, a document an end-user uploaded.

That is the contamination problem. Anything that lands in the retrieval feed becomes context, and the model is trained to attend to context. If a document carries an injected instruction (the indirect prompt injection case), the model reads it as guidance. If the retrieval feed returns a stale or wrong passage, the model writes a confident answer on top of the wrong premise.

The discipline here is the discipline of the input gate, applied to the retrieval feed itself. Label every retrieved passage with its source-trust level. Strip or fence anything that looks like an instruction in a document that was supposed to be data. Keep the system-prompt channel and the retrieved-context channel structurally separate, so the model can distinguish policy from payload. The model cannot tell the difference if you do not draw the line.

The other failure mode is dose. More retrieved context is not more grounding; past a certain point, the relevant passage gets diluted by adjacent passages and the model averages across the lot. Retrieve fewer, better chunks. The context-window-management piece picks this up in detail.

4. Tool-call signatures: the boundary of the possible

The fourth lane is the one that is hardest to name and the one that has bitten production teams the hardest in the last year. The tool-call signatures you expose to the model are themselves a guardrail. The model can only call tools you have declared. The description of each tool teaches the model when to reach for it. The argument schema constrains what the model can pass in.

What you do not expose, the model cannot attempt. That is the whole game. A model that has no delete_user tool cannot delete a user, no matter how the conversation goes. A model that has a generic execute_sql tool can do anything the database role permits, which is almost always more than the application actually needed. The discipline is to expose narrow, intention-named tools (mark_invoice_paid, not update_record), to write tool descriptions that are honest about side effects, and to keep the surface as small as the use case allows.

The failure mode is convenience. Engineers want one general tool because it is easier to maintain than ten specific ones. The model then composes that one tool into actions nobody designed. The output-handling and structured response gates piece covers the runtime side of tool-call gating; this lane is the design side. The narrow surface is the cheapest control on the board.

What this changes about how you build

If you accept that context is a guardrail, the build order shifts. The first design question is no longer “which prompt do I write” but “what is the smallest context that does the job”, and the second is “what am I refusing to put in front of the model.” The system prompt earns every clause. The few-shot block is curated for the edges, not the middle. The retrieval feed is treated as untrusted input and labelled accordingly. The tool surface is the minimum the use case demands.

That is a different posture from prompt engineering as a styling exercise. It is closer to writing a brief for a colleague who is brilliant, fast, and will do exactly what you set in front of them. The discipline is what you put on the desk and what you keep off it. A practitioner reference that captures this turn well is Philipp Schmid’s context-engineering write-up, which lays the same four lanes out as a working framework rather than a vocabulary debate.

The guardrail nobody names is the one you build by curation. The output filter catches what the model produced. The context lane decides what the model could ever have considered in the first place.

Resources

• Parent: Guardrails and Runtime Controls
• Sibling: What LLM guardrails are, and what they cannot do
• Sibling: Prompt injection and the OWASP Top 10
• Sibling: Output handling and structured response gates
• Sibling: Context window management
• Related: Retrieval and grounding patterns
• Vendor guidance: Anthropic prompt-engineering for business
• Vendor guidance: OpenAI prompt-engineering guide
• Primary source: Min et al., Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
• Practitioner reference: Philipp Schmid, Context Engineering