The AI Attack Surface: How AI Systems Get Compromised

The first time I sat with a security team to threat-model an AI feature, they drew the same diagram they always drew. A box for the app, a box for the database, a box for the API gateway, arrows between them, trust boundaries on the arrows. Then someone asked where the model went. The lead pointed at the app box. “It is just another service we call.” Everybody nodded and moved on. They had just compressed the entire AI attack surface into a single rectangle, and they had no idea.

That meeting is in my head every time someone asks why AI security needs its own discipline. The answer is not that the attacks are exotic. The answer is that the surface is wider than the model, and the standard appsec diagram does not show it.

The surface is wider than the model

When I walk a client through their AI attack surface for the first time, I draw seven things, not one. The training pipeline that produced the model. The data sources that fed the pipeline. The model weights sitting at rest in a registry or a bucket. The inference endpoint that serves the model. The prompt interface where user content arrives. The tool integrations the model can call. The supply chain that delivered every piece of the above, the base model, the libraries, the embeddings, the vector store, the agent framework.

Each of those is a place an attacker can stand. Each one carries its own threat model. A program that only protects the inference endpoint is protecting one rectangle out of seven.

Two axes are useful for sorting what you see on that diagram. First, training time versus inference time: an attack that lands before the model is deployed (poisoned data, backdoored weights, a malicious dependency) versus one that lands while the model is serving traffic (a crafted prompt, an extraction query, a tool abuse). Second, model versus data versus infrastructure: is the attack going after the model behaviour, the data flowing through it, or the systems hosting it. Most security teams have controls for the infrastructure column. Almost none have controls for the model column. The data column is split: lineage and access exist, integrity for ML training sets usually does not.

The AI-specific threat classes

Five classes of attack do not exist in classical appsec, or exist in a form that the existing controls miss. Each maps cleanly to a place on the surface above.

Prompt injection lives at the prompt interface and on any content the model reads. An attacker hides instructions in content the model will process (a web page it summarises, an email it triages, a document it ingests) and the model follows them. The model cannot distinguish “data to reason about” from “instructions to obey”, because both arrive in the same channel. This is the threat class that the OWASP Top 10 for LLM Applications puts at number one, and it is the one I see most under-defended in production. We dedicate a whole article to it: Prompt Injection.

Training-data poisoning lives at the data and pipeline boundaries. An attacker contaminates the training set so the model learns a behaviour the operator did not intend: a backdoor triggered by a specific phrase, a bias toward a specific outcome, a vulnerability to a specific input. The attack is invisible after the fact because the weights look normal. The defence is upstream: provenance on every dataset, integrity checks on the pipeline, and a clean separation between data the world contributes to and data that trains your model.

Model extraction and theft live at the inference endpoint. An attacker queries the model enough times to reconstruct a functional copy, or exfiltrates the weights from where they sit. The first version is a query-pattern problem (high-volume, fingerprint-shaped traffic from a few accounts). The second is an access-control problem (who can read the bucket, who can call the registry, who held the laptop the weights were copied to). Both are solvable with controls that already exist in most security programs. They just have to be turned on for AI assets.

Adversarial inputs live at the inference endpoint, mostly for classifiers and computer-vision systems. An attacker crafts an input that looks normal to a human but flips the model’s decision. Stickers on a stop sign that fool a vision system. Pixels that look like noise but classify an image as something it is not. The threat class is older than generative AI and has lived in academic papers for a decade; what changed is that the models are now deployed in places where the failure has a real-world cost.

Supply-chain attacks thread through every box on the diagram. A base model downloaded from a hub that is not what its README claims. A library with a malicious post-install hook. An embedding model with a backdoor. An agent framework that ships a vulnerable default. The AI stack has the same supply-chain exposure as the rest of modern software, plus a few categories software did not have, because the dependencies now include weights and datasets, not only code. We go deeper in AI Supply Chain Security.

Why traditional threat modelling misses these

The standard appsec threat-modelling habit is STRIDE or its cousins: walk the data-flow diagram, ask spoofing, tampering, repudiation, information disclosure, denial of service, elevation of privilege at every trust boundary. The method is sound. The problem is the diagram. If your data-flow diagram collapses the model into one box, the threat model collapses with it, and the prompt interface, the training pipeline, and the supply chain disappear from the conversation.

The MITRE ATLAS project (the AI cousin of the ATT&CK framework) exists because the security community recognised that the existing tactics-and-techniques catalogue did not cover what was happening to ML systems. Reading ATLAS through alongside your usual threat-modelling artefact is the cheapest way to close the gap. You do not need a new methodology. You need a wider diagram.

The cheap controls that close eighty percent

Most of the heavy lifting on the AI attack surface is done by a small set of unglamorous controls that most teams skip because they look too simple to matter.

Input and output validation at the prompt interface. Treat every prompt as untrusted, including prompts assembled from your own systems if any of the content traces back to user input. Strip or escape known-bad patterns, but do not rely on filters alone; assume some injection will land and design the downstream behaviour to be safe under that assumption. On the output side, never let the model’s raw text reach a system that interprets it (a shell, a SQL engine, a tool dispatcher) without a parser in between.

Prompt and response logging, retained. Every call, every input, every output, every tool invocation, kept for the period your sector requires. This is the same audit-log control from the AI Governance pillar, applied to AI security. Without the log, you cannot investigate the incident; with it, most investigations are short.

Supply-chain provenance on weights and models. Pin model versions. Record where each model came from. Verify checksums against a trusted source. Treat a model weight file like a binary you are about to run in production, because that is exactly what it is. The data-security half of this lives in AI Data Security.

Rate limits and pattern detection on the inference endpoint. Extraction and adversarial-probing attacks both look like high-volume, narrowly shaped traffic. Standard API rate limits plus a basic anomaly detector on call patterns catch most of them long before they cost you a model.

A named owner for the AI security surface. A person whose job includes the seven boxes on the diagram. Not a committee. Not “the AI team” and “the security team” hoping the other one has it.

Do these five and you have closed most of the realistic threat. They cost less than one custom-built defence, and unlike clever mitigations they keep working when the attack class evolves.