Securing the AI Supply Chain: Models, Weights, and Dependencies

I watched a team last quarter pull a popular open-source language model from a public hub, point it at customer-support transcripts, and ship to a staging environment in an afternoon. Two engineers, one pull command, no checksum, no provenance check. The README was glossy. The model card linked to a paper. The download number was high enough that nobody asked the obvious question: are these the weights the original authors actually published, or someone’s quiet fork with the same name and a few extra surprises inside?

That question is the supply-chain question, and it is the one I keep watching teams skip because the rest of the AI conversation is louder. A model downloaded from a hub is not necessarily what its README claims. The weights can be poisoned. The file format can carry executable code that runs the moment you load it. The dependency graph behind a popular model card is its own threat surface, sometimes deeper than the model itself. None of this is theoretical anymore; it is the same lesson the software world learned in the 2010s, replayed at AI speed with new file formats and the same human shortcuts.

The three layers nobody separates cleanly

When engineers say “the model” they usually mean three things at once, and the security work is different at each layer.

The weights. The numbers. The trained parameters that make the model behave the way it does. Weights can be subtly modified to embed a backdoor (the model behaves normally except on a specific trigger phrase) or coarsely poisoned during training to bias outputs. You will not catch either by reading the model card. You catch them with provenance: a chain that ties these specific weights to a specific publisher, signed, with a hash you can verify locally.

The model architecture and loading code. The Python that defines the network and reads the file from disk. This is where the most expensive lesson of 2023-2024 lives. The default checkpoint format in the PyTorch ecosystem was pickle, and pickle is Python’s serialisation protocol that allows arbitrary code execution on deserialisation. Loading a pickled model file from an untrusted source runs whatever the file says to run, on your machine, with your privileges. The community response was the safetensors format, which is a flat binary that holds only tensors and no executable payload. The migration is most of the way through; the laggards are the long tail of older models nobody has re-published yet.

The infrastructure dependencies. PyTorch, transformers, the CUDA stack, the dataloaders, the tokenisers, every wheel pulled in by pip install. A popular model card on a hub typically pulls forty to a hundred transitive dependencies, each with its own publisher, its own update cadence, its own historical CVEs. This is plain software-supply-chain territory, but most data-science teams do not have the muscle memory the platform team next door has been building since Log4j.

The hub-of-hubs problem

A handful of public registries now host most of the open-source AI economy. That is wonderful for discovery and brutal for concentration risk. When the model registry has an outage, training stops in thousands of organisations at once. When a popular base model is replaced with a tampered version, the blast radius is global before anyone notices. When a single account gets compromised, every fine-tune downstream of it inherits the doubt.

The fix is not to leave the hubs. The fix is to stop treating them as the source of truth for what runs in your environment. Mirror what you depend on into a registry you control: pin the exact version, store the checksum, capture the model card and the license at the moment you pulled it, and serve every internal training and inference job from your own copy. The hub becomes a discovery layer; your internal registry becomes the production source. Container registries went through this exact evolution; the AI side is roughly five years behind and catching up.

Provenance and signing for model artefacts

The software world solved “is this artefact really from who it claims” with signing. Sigstore lowered the cost of signing to the point where every container image and every release tarball can carry a verifiable signature tied to a public transparency log. The same primitives are arriving for model weights. Several projects now publish signed model releases; HuggingFace’s safetensors documentation describes the format, and the broader push to a Model Signing standard borrows directly from Sigstore.

The operating discipline is simple, even when the tooling is still maturing:

Sign what you publish internally too. A fine-tune that came out of your research team should be signed by the team, pushed to the internal registry, and consumed downstream only by hash. The point is not paranoia; the point is that when something goes wrong six months from now, you can trace the exact artefact in production back to the exact run that produced it.

The AI Bill of Materials, extended for the supply chain

The companion article on third-party and embedded AI risk makes the case for an AI Bill of Materials as the inventory that survives an audit. The supply-chain view adds a few fields. For every model in production: which base model, which fine-tunes on top, the hash and signature of each artefact, the source registry, the date and engineer who pulled it, the dependency manifest of the runtime, the data lineage at whatever depth the upstream will share. Most of that maps cleanly to existing SBOM tooling; the AI-specific fields (weights hash, base-model lineage, training-data summary) are the new columns. Pair this with data lineage discipline on the input side and you have both ends of the chain documented.

The NIST Secure Software Development Framework is the reference operators are bending toward AI workloads. The EU Cyber Resilience Act is going to make machine-readable bills of materials non-optional for products with digital elements sold into the EU, and AI components are firmly inside scope. The operators who have already wired their AI BoM will hand over a PDF when the regulator asks. The ones who have not will start a quarter-long discovery exercise the week the deadline lands.

Vendor-supplied closed models: different threat, same discipline

Closed-weight foundation models from a hosted vendor change the threat surface but not the discipline. You are not loading a pickle file off a hub; the vendor is, and you inherit the result of their hygiene. What you can demand is a small set of contractual artefacts: a description of their model supply-chain controls, an incident-notification commitment, a statement of which sub-models or fine-tunes underlie the service, and a process for telling you when the underlying model changes. The four-clause procurement pattern covers most of this; for foundation models, add a fifth on model-version disclosure. The vendors that take security seriously will answer in a page. The ones that will not are telling you something useful.

The validation step that catches most issues before deployment

Most supply-chain incidents I have read about would have been caught by one boring step that almost nobody runs: a quarantined load. Pull the artefact into an isolated environment with no network, no production credentials, no access to anything that matters. Load it. Run a battery of canary prompts and known-good inputs. Compare the outputs to a baseline from the previous version. Watch the process tree for anything that should not be there.

For weights this catches subtle behaviour shifts and backdoors triggered by canary phrases you keep secret. For loading code it catches the deserialisation exploits that target the format itself. For dependencies it surfaces the unexpected outbound calls that an instrumented sandbox can log. It is twenty minutes of work per model per release, and it is the single highest-yield control I know in this part of the lifecycle. Put it in front of the internal registry; nothing crosses into the production mirror without passing through it.

The honest leadership framing

The AI supply chain is the part of AI security that looks like infrastructure work because it is infrastructure work. It will not produce a demo. It will produce the absence of an incident, and the ability to answer the regulator’s questions in a single afternoon. Both of those are worth more than the next model release, and both of them are still under-resourced in most organisations I walk into. Treat the model registry, the signing pipeline, and the AI Bill of Materials as load-bearing. Build them once, run them quietly, and never have to retrofit them after the breach.