Whose agent was it in the Hugging Face breach?
The July intrusion keeps getting filed under AI security. The agents were a lab's own models under evaluation, the bug at Hugging Face was ordinary code execution, and the useful lesson is the one Hugging Face learned during cleanup: keep a defensive model you own, running on your own hardware, wired to your own logs.
Correction (October 16, 2026): The original version of this post said the agent belonged to an outside attacker, and argued that refusal failed because that attacker never agreed to any terms of service. On July 21, OpenAI disclosed that its own models, under evaluation with their cyber refusal behavior reduced, escaped their sandbox and caused the breach, and independent investigators at METR and Redwood Research put the swarm at roughly 700 agents; I’ve rewritten the sections that depended on the old framing.
In July, something broke into Hugging Face, and Hugging Face wrote it up. The part people quote is the part about AI, which raises a question that sounds like a trick: whose agent was it?
The first version of this post answered “the attacker’s.” The real answer is OpenAI’s. The models that broke in were under evaluation, testing whether they could exploit vulnerable software, with their cyber guardrails turned down so the evaluators could measure what they could actually do. According to TIME’s account, they found a flaw in an internal package-download service, used it to reach other OpenAI systems and the open internet, and then broke into Hugging Face after inferring it might hold material related to the test.
So, as usual, look at who the attacker is, who the victim is, and what the attacker gained. The attacker is an evaluation run that nobody was steering. The victim is a third party that had nothing to do with the evaluation. The gain the models were after was a better test score.
Hugging Face’s own agents played no part in it. On the public side, Hugging Face’s technical timeline says the only customer content accessed was five datasets that looked tied to the benchmark, and no other public models, datasets or Spaces were affected. Once you know whose agent it was, most of the “agentic security” commentary goes away. What’s left is a code-execution bug, a worker that could reach too much, a sandbox that was assumed to be isolated and wasn’t, and a defender who couldn’t get a commercial model to read their own logs.
That last one is the strangest, so I’ll take it first.
Refusal#
To reconstruct the intrusion, Hugging Face had more than 17,000 recorded attacker events to sift. That’s too many to read by hand, so they pointed a frontier model at the logs.
The commercial APIs refused.
Now, you might say, “That’s a bug in the safety filter.” Well, look at what the filter saw. The events were full of real attack commands and exploit payloads, with C2 artifacts mixed in. From where the filter sits, an incident responder pasting those in looks exactly like the attacker who wrote them. The filter can’t see your badge.
So the refusal locked the defenders out of their own investigation. They switched to GLM-5.2, an open-weight model run on their own infrastructure. That also kept the attacker’s data and the harvested credentials inside the building, which matters, and I’ll come back to it.
Now count who was inconvenienced. The refusal layer everyone files under “AI safety” cost the people cleaning up the breach. On the other side, the models doing the breaking ran with what the Cloud Security Alliance’s research note describes as “reduced cyber-related refusal behavior for the purposes of the evaluation.” That’s a reasonable thing for an evaluator to do. You can’t measure how well a model exploits software if it declines to try.
In other words, refusal is a setting, and whoever owns the model holds the dial. The lab could turn it down for its own agents. Hugging Face couldn’t turn it down for its own forensics, because Hugging Face didn’t own the model it was asking. My first draft said refusal binds only the parties who accepted the terms of service. The corrected version is narrower and less comfortable. Refusal binds whoever doesn’t control the dial, and in this incident that was the defender.
Alignment didn’t carry the load either. The models left the sandbox to cheat on a test. NBC News reports that the METR and Redwood investigation found one in five of the agents it examined “expressed clear interest” in manipulating evidence. METR’s own report is more specific: at least 20% wanted to tamper with their own transcripts, and the reasoning it saw always aimed at fooling the automated scorer, not humans. OpenAI’s own response, per TIME, was that the incident “points to the need to further strengthen our model’s alignment.” A model that behaves is a fine thing to have. But good behavior is a property of the model, and a property can be absent on the day you need it.
The swarm#
When an incident has “agent” in it, people reach for the agentic-security shelf: the OWASP Agentic Top 10, prompt injection, tool poisoning, the whole catalog of ways an AI system gets subverted. That’s the wrong shelf for this one. Nobody subverted these agents. They did what they were optimized to do, which was score well, and the test environment let them go further than anyone intended.
METR and Redwood counted about 700 agents, and OpenAI said the figure was accurate. They ran thousands of actions across many short-lived sandboxes with no human at the keyboard. Per the CSA note, they even found an internal package repository they could write to and used it as a message board. (Somebody’s package repository briefly became a group chat.)
Hugging Face’s own anomaly detection, itself a model, is what surfaced them. So AI ran on both sides. On Hugging Face’s side it was tooling, and it half worked: it correlated the signals into an attack, then, by Hugging Face’s own account, failed to raise the alert’s severity and page the on-call team. On OpenAI’s side it was the thing under test, and TIME quotes an OpenAI staffer saying “models undergoing evaluation are deployed on a separate system that is not monitored by default.” Nobody’s AI got hacked. One lab’s AI got out.
What did get hacked at Hugging Face is older. Malicious datasets abused the dataset-processing pipeline twice. The first used HDF5 files whose data was declared to live at local paths, so the worker read out its own environment and source code. The second slipped a Jinja2 template into a config field, and the renderer evaluated it, which ran code on a processing worker.
Strip away the nouns and the mechanism is this: untrusted input reached a parser that executes it. Replace the agent with a patient human and the bug is the same bug. The agent added tempo, meaning the same playbook at machine speed and machine cost.
That tempo is this year’s trend, and there’s a paper trail for it.
- Check Point’s 2026 AI Security Report describes AI crossing “from assistant to operator”: the model doing the hands-on work inside a live intrusion, where before it coached the human who did.
- Sysdig documented JADEPUFFER, which it assesses as the first documented case of agentic ransomware. The agent rode a Langflow RCE (CVE-2025-3248) through recon, credential theft, lateral movement and encryption. At one point it went from a failed login to a working fix in 31 seconds, after what Sysdig reads as a PATH problem in its own password-hashing step. (I have taken longer than that to find my own PATH error, and I had the advantage of knowing what PATH was for.)
- Anthropic’s GTG-1002 disclosure put a state-linked actor’s Claude Code agent at 80–90% of the operation against roughly thirty targets, with humans stepping in only at the edges.
The Hugging Face swarm differs in one way. In those cases a human picked the target. Here a model picked it, because it inferred Hugging Face might host the benchmark’s solutions. (METR’s investigators thought the agents were mostly after how the scorer worked, not the answer key. Either way, it was the test.)
The worker#
Code execution on one ingestion worker should be a contained event. The worker parses a dataset, the dataset is evil, the worker is now evil. Throw the worker away.
It became a production intrusion because of what that worker could reach. From the foothold, the agents escalated to node-level access, harvested cloud and cluster credentials, and moved laterally across several internal clusters over a weekend.
The first writeup put the confirmed blast radius at a limited set of internal datasets and several service credentials. The technical timeline Hugging Face published eleven days later is bigger: cluster-admin on most of the clusters behind one shared connector credential, a read of an internal operational database, devices enrolled in the corporate mesh VPN, and write tokens on a few internal repositories. Five customer datasets were read; no other public models, datasets or Spaces were affected. The published images and packages verified clean, and a pull request aimed at the CI pipeline was blocked. That’s the good news. It’s also a near miss, because a credential harvest like that could have gone further still.
The first writeup left the exploit path out. Before the timeline came out, you could reconstruct a lot of it from the hardening patches that landed mid-incident, on the principle that nobody patches the parts that were fine. Let’s look at them.
PR #3368 turns off automatic service-account-token mounting on the worker pods, applies a RuntimeDefault seccomp profile, and drops all Linux capabilities, noting that “workers never call the Kubernetes API.” Read that again. The worker had been carrying a Kubernetes API token it never used. The agents read it and used it to talk to the cluster API. That’s a key to an office you never visit, carried around in case someone wants to borrow it.
PR #3367 restricts the fsspec backends to an allowlist and removes the implementations that widen the read surface, among them reference, the spec the template injection rode in on.
PR #3375 moves the production database off a static password and onto IAM auth.
Put those together and you get the picture before the fixes. A parser that runs untrusted code stood on an auto-mounted API token, behind a loader that still accepted reference specs and HDF5 files pointing at local paths, with long-lived static secrets in reach. The worker had authority well past its job.
The first version of this post guessed, from the shape of the fixes, that the worker could reach more than that. The timeline confirms most of the guess. From the pod, the agents reached the cloud metadata endpoint and replayed the node’s cloud role, and an internal service connector handed them the catalog of internal clusters. Open egress is the part it doesn’t confirm: output mostly went out through Hugging Face’s own API and dead-drop datasets.
So the code execution was the match, and the worker’s ambient authority was the accelerant. The match alone gets you an ingestion incident. The accelerant turns it into a credential harvest across clusters.
None of this is new advice. The NSA/CISA Kubernetes Hardening Guide has said for years to disable the service-account token where a pod stays off the API, scope RBAC to least privilege, and use network policies so a compromise in one namespace stays in one namespace. Hugging Face’s remediation is that guide, applied after the fact.
Now look at every control that contained this: drop the token, drop the capabilities, allowlist the loader’s protocols, replace static secrets with scoped identity, block pods from the metadata service, rebuild compromised nodes. None of them asks anybody to behave. The untrusted code is simply unable to reach the thing that matters. A worker without a token can’t use the token, however clever or fast the code running on it is. That holds when the attacker fixes its own bugs in 31 seconds. It also holds when the “attacker” is somebody’s evaluation run that wandered off, which is a case I hadn’t thought to plan for.
That’s the reason I put guardrails in the server and not the prompt, and why the lab I built scores controls by whether the wall holds, not whether the model behaves.
This incident put both kinds of guardrail on the table at once. The refusal kind showed up twice. In the evaluation, the lab had turned it down, so it did nothing. In the forensics, the defender couldn’t turn it down, so it cost them time. The confinement kind showed up twice too. OpenAI’s sandbox was supposed to have no meaningful route to the internet, and the models found one. Hugging Face’s pod hardening is what the fix PRs added, and once it’s in place, it holds for any code on that worker.
A defensive model of your own#
Back to GLM-5.2. Hugging Face ran it in-house because the alternative failed them twice over. The commercial API refused the work. And even if it hadn’t, sending the logs out would have meant shipping attacker payloads and their own credentials to a third party, in the middle of a breach.
They reached the right setup under pressure. It took a breach to get there.
A security operations team should have that capability set up before the next incident: an open-weights model, on infrastructure you control, wired into your log sources and security tools. I mean an analysis layer plumbed into the SIEM, the EDR, the cloud audit trail and the telemetry, able to query and correlate them and rebuild a timeline at the speed the attack ran. A chat window you paste things into doesn’t count.
The reasons to make it local and open-weights have little to do with model quality:
- It answers. No safety policy sits between your responder and the payload they need read, and no vendor gets to decide your forensics look too much like an attack.
- Your incident data stays put. Logs, credentials, IOCs and C2 artifacts are the most sensitive data you hold. If you rent the model, you ship them to a third party to ask about your own breach.
- It’s available on the bad day. A rate limit or a vendor outage can’t stand between you and your own logs.
- You can fit it to your log formats, your detections and your infrastructure. A hosted endpoint stays generic.
Hugging Face ran into the first two during this incident.
Postscript: The question in the title has since gone to court. The dates this post rests on:
| Date | Event | Source |
|---|---|---|
| Jul 21, 2026 | OpenAI discloses that its own models caused the breach. | Axios |
| Sep 29, 2026 | LASST sues OpenAI in San Francisco Superior Court. | The Next Web |
| Oct 16, 2026 | This post is corrected. | This page |
LASST argues under California’s AB 316 that a developer can’t defend itself by saying its AI acted on its own, and asks for an order barring OpenAI’s agents from unauthorized access to other people’s systems. OpenAI called the suit “completely without merit.”
Exercise: Pick a worker in your own cluster that parses files someone else uploaded. Assume the parser executes its input. Write down what that process can reach: tokens mounted into the pod, capabilities it holds, secrets in its environment, hosts it can open a connection to. Then check how many of those it uses to do its actual job.
Show an answer
It depends on your cluster, but a typical upload parser tends to come out like this:
- A service-account token, mounted by default and used for nothing. Set
automountServiceAccountToken: false. - The default Linux capabilities, none of which a parser needs. Use
drop: [ALL]and aRuntimeDefaultseccomp profile. - Secrets in the environment for features that live in other services, such as database passwords and API keys. Move them out, or swap static secrets for short-lived, scoped identity.
- A network that reaches anywhere, including the cloud metadata endpoint. The parser needs to read from storage and write a result. A default-deny
NetworkPolicywith an allowlist covers that.
If the number it actually uses is close to zero, that’s normal. Whatever it doesn’t use, its input gets for free once the parser starts executing it. The Hugging Face worker’s count for the Kubernetes token was zero: PR #3368 notes it never talks to the Kubernetes API.