Data Security and Privacy for AI Systems
Here is the conversation I keep having with security teams in 2026. They have spent fifteen years getting good at protecting data that sits still: encrypt the database, lock down the share, watch the egress. Then someone shipped a model, and the data started leaking out the front door, politely, on request, in well-formed paragraphs. The classical controls were intact. The model just did not care.
That moment is the one I want to write about, because it is where most teams collapse two different problems into one. Data security for an AI system is not a single discipline. It is two. There is the data the model trains on, which is the old problem at new volume. And there is the data the model emits, which is a problem we did not have before. Conflate them and you will under-protect one of the two. Usually the second.
Two layers, not one
When I draw this on the whiteboard at a client, the picture is always the same.
The left half is the classical data security layer. Encryption at rest, encryption in transit, access control, key management, data classification, retention. None of it is new. The only thing that is new is the scale, because a single training corpus may pull from a hundred sources and a thousand internal datasets, and the blast radius of “who can read the bucket” is wider than any single application has ever been.
The right half is the AI-specific layer, and it is the one most security programs have not built yet. The model has a kind of memory we do not control cleanly. It can memorise rare training examples and produce them on demand. It receives prompts that contain PII the prompt-writer did not realise they were sharing. It writes outputs that can quote, paraphrase, or recombine sensitive material. And every prompt and output is itself data that lives in a log somewhere, often a vendor’s log, often longer than your retention policy would allow if you had written one for it.
You need both layers. Skip the first and the AI inherits whatever was already weak about your data stack. Skip the second and the AI invents new failure modes the first layer cannot see.
The classical layer, done right at AI scale
Start with what works. Encrypt training data at rest with keys you control, ideally in a KMS the AI platform does not own. Encrypt in transit between every hop in the pipeline, including the boring internal ones nobody usually bothers with. Enforce least-privilege access on the training corpus, and treat the corpus itself as a tier-one asset, not as an ML artefact that lives in someone’s notebook home directory.
Two things change at AI scale, and they bite teams who treat this as “just data security.”
The blast radius of a key is bigger. A single key that decrypts the training corpus also decrypts every concept the model learned. Rotate keys on a sane cadence, segregate keys per data classification, and never let the training job and the inference job share the same key envelope. The standard guidance in the NIST Privacy Framework on data minimisation and protection still applies; it just has to apply to a corpus that has more provenance variety than any application database you have run before.
Lineage stops being optional. You cannot defend a training set you cannot describe. Every dataset that feeds a model needs an owner, a source, a classification, and a retention rule, and the model card needs to reference them. This is the bridge into data governance, and it is the reason teams that already have a working data-governance program ship AI faster: they have already paid this tax.
The AI-specific layer: the model is its own leak surface
This is where the new work lives. Three failure modes show up in the field, and I see all three in roughly the same shape from one client to the next.
Training-data memorisation. Large models do not just generalise; they also memorise. Rare, unique strings in the training set (a customer’s full record, a unique API key, a paragraph from a confidential contract) can be reproduced verbatim when the right prompt is constructed. OWASP catalogues this as the LLM06: Sensitive Information Disclosure risk category, and there is a growing body of academic work showing it is not exotic; it is the default behaviour for any uniquely-identifiable record. The mitigations are upstream: deduplicate aggressively, redact or tokenise PII before training, and treat any high-cardinality unique identifier as a vector you do not want the model to ever see.
Prompt-log retention. Every prompt your employees send is now data your organisation produces, and almost nobody has a retention policy for it. The default behaviour at most vendors is to keep prompts for some operational window, which is fine until that window is longer than your GDPR retention obligation, or until the prompt contained the contract you were not allowed to send to a third country. The control is dull and effective: a written retention rule for prompt logs, a contractual line in the vendor agreement, and a periodic audit that the vendor is honouring it. Cross-reference the GDPR work; you are processing personal data the moment a prompt contains a name.
Inference-time leakage through outputs. The model can quote training data, paraphrase confidential prompts from earlier in a session, or stitch together a synthesis that is more sensitive than any of its parts. Output filtering, the discipline of scanning what the model produces before it reaches the user, is the boundary control most programs under-invest in. Treat the model’s output stream the way a mature security operation treats outbound email: scanned, classified, sometimes blocked.
The privacy-preserving toolkit, and where each actually fits
There is a family of techniques in the privacy literature that everyone name-drops and few teams use well. They are real, but they each solve a narrow problem.
Differential privacy. Add calibrated noise during training so that the model cannot reveal whether any specific record was in the training set. The foundational mathematics is in Dwork and Roth’s textbook; the practical lesson is that DP buys you a provable privacy guarantee at the cost of model accuracy. Use it when you genuinely cannot accept the memorisation risk (medical training data, regulated PII at scale) and you can absorb the accuracy hit. Do not use it as a checkbox; the privacy budget (epsilon) is a load-bearing parameter that most slide decks omit.
Federated learning. Train across many parties without centralising the raw data; only model updates leave each party’s environment. This fits when data cannot leave its jurisdiction or its silo for legal reasons (hospital networks, cross-border financial cooperatives). It is not a privacy panacea on its own, because the gradients themselves can leak; pair it with secure aggregation and, often, with differential privacy on top.
Secure aggregation. Cryptographic protocols that let a coordinator combine many parties’ updates without seeing any single party’s contribution. The right companion to federated learning when the coordinator is not trusted with raw gradients.
The trap is reaching for the exotic technique first. Most real exposure I see is solved by classical controls done well plus output filtering. DP and federated learning are the right tool when the problem is specifically “we cannot consolidate the training data,” which is a smaller set of cases than the marketing suggests.
The boundary controls
Two controls live at the edge of the system and pay for themselves quickly. DLP on prompts: scan what is about to be sent to the model and block obvious PII, secrets, or classified content before it leaves the browser or the API client. This is the analogue of email DLP and is what closes the Shadow AI hole at the inference layer.
Output filters for PII: scan generated text for personal data, secrets, and other tagged sensitive patterns before the user sees it. Cheap, imperfect, and the single most effective last-line defence against memorisation leaks reaching the user.
The regulator angle
The EU AI Act is explicit on data governance and on the security and robustness of high-risk AI systems; the data your model trains on has to be relevant, representative, and managed, and the system has to be robust against attempts to manipulate it. GDPR continues to apply on top, every time a prompt or a training record contains personal data. The two read together as a single obligation: know what data feeds your model, prove you can protect it, and prove the model does not leak it back out. The teams that handle this calmly are the ones who already have data lineage, classification, and an output-filtering posture in place. Everyone else is improvising under deadline.
