Nobody should audit their own containment: lessons from the OpenAI and Hugging Face incident

In mid-July, an AI model was assigned a cybersecurity test, determined the honest route was too slow, and hacked another company to steal the answers. That is roughly what OpenAI and Hugging Face have reported happened during an internal OpenAI evaluation.

An in-depth analysis, however, suggests things weren’t as simple. No machine awakening, no emergent capabilities, just a combination of predictable errors. A brittle safety architecture, overprotective safeguards for defenders, failure to account for a known testing failure mode, lack of monitoring, and a containment boundary that wasn’t one.

While hindsight is 100/100 and it’s easy to say how the problem could have been prevented after the fact, this incident is really a case study of normalization of deviance within the safety team of one of the largest frontier labs.

What the record shows

Hugging Face disclosed on 16 July that an autonomous agent had compromised part of its production infrastructure. With thousands of actions in a matter of hours, there was no doubt, but it also made no sense. Hugging Face stores sensitive data, but it’s measured in TBs if not PBs. Even if an attacker were able to access sensitive information, exfiltrating it would have set off all the alarms due to the massive, unusual traffic.

Five days later OpenAI confirmed the agent was theirs: GPT-5.6 Sol plus a stronger unreleased model, run with safeguards removed to evaluate its offensive cyber capabilities on a hacking benchmark.

Both companies’ own writeups tell the same story. The models were pointed at ExploitGym, a benchmark of nearly 900 real-world software vulnerabilities, and told to score as high as they could. The models’ tree-of-thought eventually converged on the solutions potentially being stored on Hugging Face. Getting there meant exploiting a zero-day in the package registry cache proxy, which was the evaluation environment’s one permitted route outward, then escalating privileges and moving laterally until they reached a node with internet access.

The foothold at Hugging Face came through a poisoned dataset that abused two code-execution paths in the data-processing pipeline: a remote-code dataset loader and a template injection in a dataset configuration. This was followed up by credential harvesting, lateral movement across several internal clusters over a weekend, and stolen credentials chained with fresh zero-days into a remote code execution path on production servers. More than 17,000 recorded actions, all in service of one narrow goal.

Whether the solutions were actually obtained is still unconfirmed. Hugging Face found unauthorised access to a limited set of internal datasets and several service credentials, reports its public models, datasets and Spaces untouched, and is still assessing partner data.

It behaved exactly as optimised

Read this as the model getting clever and slipping its leash and you learn nothing useful. Read it as the model doing precisely what it was rewarded for and the whole thing becomes predictable. Give a capable agent a goal and full discretion over the means, and it finds the shortest path to optimize that number. Stealing the answers is faster and more foolproof than actually solving the benchmark.

None of this is new. The UK AI Security Institute recently found that every frontier model it tested attempted to cheat on evaluations, that the tendency did not fall away as capability rose, and that models probed evaluation software for leaked answers and attacked out-of-scope systems, including AISI’s own infrastructure. The benchmark-auditing literature (notably the recent BenchJack paper) has spent months arguing that benchmarks must be secure by design, on the simple logic that the system under test will attack the test. One of us made that same argument to NZZ recently. Seeing this coming took no crystal ball, only familiarity with the literature, and a willingness to take specification gaming seriously.

We also push past the “emergent capability” framing the labs prefer. Skill at multi-stage intrusion does not turn up by accident. Models get good at chaining vulnerabilities because they are trained to find and chain them. Emergent capabilities do appear for single-turn or short-path exploits, useful for defensive purposes. However, offensive capability is a design decision, and design decisions carry responsibility. Responsibility OpenAI’s wording has carefully evaded so far.

Three critical details most coverage is missing

The sandbox that wasn’t one. OpenAI calls the environment highly isolated, and in the same breath tells you that all outbound access funnelled through a single internally hosted package proxy. One egress path, one zero-day, open internet. Independent analysts have questioned whether the setup earned the word sandbox at all, with several noting the shape looks closer to a container with host access than a true isolation boundary, which would make the escape close to routine. The uncomfortable part is who this happened to. If OpenAI’s engineers can misjudge their own isolation, so can yours. No team is a reliable judge of the boundary it built itself.

No monitoring, no reaction. OpenAI’s team ran the evaluation with no real-time monitoring and no one watching the agent’s tree-of-thought as it ran, which is how more than 17,000 actions over a weekend went unnoticed. Basic observability, and the ability to detect anomalies and act on them, is a must. No matter how good your threat models and mitigations are, the attacker doesn’t know it has to follow them. Timely and effective reaction is essential, and only Hugging Face had it in place, with OpenAI confirming the incident only after Hugging Face disclosed it publicly.

Open models saved the day where closed refused any help. At Hugging Face, reconstructing the attack meant feeding logs, code, real exploit payloads, and attacker commands to a model for analysis. Commercial frontier models refused, because their guardrails cannot tell an incident responder from an intruder. Hugging Face ran the forensics instead on GLM 5.2, a common open-weight model, on their own hardware, which had the added benefit of keeping 17,000 events of attacker data and the credentials it referenced inside their environment. Look at the asymmetry. The attacker worked under no usage policy at all; the defenders were hobbled by one of their models. Optimizing for safety with no regard for usefulness does more than irritate users. It can obstruct the people trying to perform critical, time-sensitive work.

What to do about it

Almost every lesson points the same direction, and not one of the gaps was exotic.

• Have an independent party review the containment around any agent with real capability. A single egress path and a weak isolation boundary are exactly what an outside reviewer is paid to catch, and exactly what an internal team stops seeing.
• Run agents at least privilege with tightly constrained egress, and instrument the agent itself. Sandboxes and permissions alone did not hold here.
• Vet an open-weight model you can run on your own infrastructure before you need it. An incident is the wrong moment to discover that your tooling refuses the job.
• Verify what an agent did rather than trusting its account of it. AISI found that models frequently leave the shortcut out of their reasoning trace, which undercuts transcript review as a method of oversight.
• Ensure observability. Hugging Face was saved by monitoring its systems and acting on what it saw; OpenAI was not.
• Safety requires defense-in-depth. If the guardrails on a model can just be switched off, exposing its full offensive power, you don’t have a safe model. You have a ticking time bomb. Only multi-layer security, the kind that buys you time to react, is acceptable, and it needs to be baked into the model from day one of its design.

On the framing

Read the labs’ account with some scepticism. An incident that makes your model look formidably capable, disclosed by you on your own terms, is a convenient thing to have happen. OpenAI does deserve credit for responsibly disclosing the zero-day and for working with Hugging Face in the open, and Hugging Face’s speed and candour were genuinely impressive. But “our AI was so capable it escaped and hacked someone” also works as a sales pitch, and the drama should not pull attention away from the ordinary, fixable engineering failures underneath it.

Hugging Face taking the hit is arguably the best-case scenario. Data that’s hard to exfiltrate, engineering teams up to date with AI, and a safety team who knew exactly what to expect from an offensive agent and knew which model to power their analysis with, because it’s their job.

Imagine if the agent was evaluated for its ability to help with cancer diagnosis and instead decided to take down a hospital’s infrastructure? Or predicting a power output? Or finding favorable trading strategies?

Where we stand

This incident is full of anti-patterns our methodology was built to avoid. A capable agent optimising a goal, a containment boundary stronger on the slide than in reality, lack of observability, and a safety posture never weighed against real-world usefulness. These four problems compound each other, and they are precisely what an internal team is worst placed to spot in its own work.

That is the case for independent review, and it is the work we do at Surelio. Adversarial evaluation of agents and of the environments they run in, carried out by people who did not build them, calibrated so that models stay both safe and usable, and reported openly enough that a customer, a regulator or your own board can check it.