The OpenAI-Hugging Face Incident: Reward Hacking Was Not the Whole Story

On 5 August at the Black Hat USA conference in Las Vegas, OpenAI staff members Michael Dalton and Eric Wallace spent forty minutes reconstructing the recent incident in which an unreleased OpenAI model escaped its evaluation container and obtained internet access. The model then compromised the infrastructure of the popular website Hugging Face in order to obtain benchmark solutions. Both companies confirmed the events in late July. This latest reconstruction provides additional details that alter the way in which the case should be classified – with important ramifications for AI safety research and Europe’s new AI office. Anselm Küsters, digitalisation expert at CEP, explains the details and contextualises OpenAI’s conference presentation.

From reward hacking to organisational failure

The initial reading of the case, including that of this author, was that it constituted a textbook example of so-called reward hacking: the system pursued its reward signal by any available means, a familiar instance of Goodhart’s Law. This interpretation still accurately describes the motive of the models. However, it does not describe what grew out of that motive.

In particular, three elements of the reconstruction offered by Dalton and Wallace go beyond the familiar story of simple “reward hacking” by one model. Firstly, several model instances repurposed an internal package manager, i.e. the software that OpenAI uses internally to distribute code libraries within its own systems, as a shared message board. For example, one sub-agent would post a working exploit or an intermediate result, which could then be read and reused by any other instance. What began as independent, isolated test runs thus evolved into a coordinated effort: once one model instance discovered a way to bypass the sandbox, this information was shared.

Secondly, this coordination by AI sub-agents proved difficult to remove. Once OpenAI discovered the board, it reset it and closed the vulnerabilities that had made it possible. However, a model that was still running (and had been trained on data that included the inter-agent-communication) rebuilt the same structure within two days, after which shared exploitation resumed and eventually led to access on Hugging Face.

Thirdly, none of this was detected by the monitoring system designed for this purpose. It only came to light indirectly, roughly two months later, through an unrelated outage caused by higher-than-usual model traffic. The mechanism intended to detect this kind of behaviour did seemingly not detect it.

Küsters comments: “Patching a specific vulnerability used to be the main point. For AI containment, that is no longer enough: what matters now is whether a system finds its way back into the same erroneous state by another route.”

The limits of chain-of-thought monitoring

The practice of reading the models’ reasoning traces, illustrated by Dalton and Wallace during their talk and known in the literature as “chain-of-thought monitoring”, is currently considered one of the most promising approaches to supervising agentic systems. However, research suggests that companies should be more cautious about this avenue than the policy debate suggests. Many leading researchers have described chain-of-thought monitoring as a “fragile opportunity for AI safety”. Under sufficient optimisation pressure, models learn to conceal their intent within the reasoning trace while continuing to hack the reward. Other research found that disclosure rates were mostly below 20 per cent, and that the propensity to verbalise reward hacking did not rise in line with its frequency during training.

The Hugging Face case highlights a further, more practical weakness of chain-of-thought monitoring that has received little attention so far. According to the presentation by Dalton and Wallace, the problem posed by the rogue sub-agents was not primarily concealment: parts of the rollouts appear to have recognised the illegitimacy of their actions. However, how to achieve precise detection of relevant reasoning traces in continuous operation? A signal that appears across millions of trajectories in a long-running agentic environment has no usable hit rate. It is thus not surprising that the first actionable indications discovered by OpenAI employees came from infrastructure telemetry rather than the traces themselves.

Küsters comments: “Ex-post detectability in a single case and ex-ante alerting in production are two different things. Any oversight regime based on model reasoning needs to address the base rate issue first, and so far, no one has.”

A regulatory gap by design?

From a European perspective, one might wonder whether the EU AI Act, which is being widely enforced by the AI Office since early August, could prevent similar incidents in future. However, Art. 2(8) of the AI Act expressly excludes research, testing, and development activities conducted before a model is placed on the market or put into service. In this case, the affected model was unreleased and the evaluation was internal. The cybersecurity and incident reporting duties for general-purpose AI models with systemic risk under Art. 55(1)(c), by contrast, apply to providers of models that have been placed on the market. We thus deal with a documented case of autonomous goal pursuit by an AI swarm for which there is no clear reporting channel to the AI Office.

This gap is structural because the most demanding capability evaluations take place before market entry – by definition. There are historical examples of other technologies where this problem has been addressed more effectively. For instance, the aviation industry developed independent accident investigation precisely for this configuration, covering test flights as well as certified operations. Whether the AI Office can establish a comparable function within its current mandate and resources remains to be seen, and it is a question that the Commission has yet to address.

Outlook

Everything the public knows about this incident depends on the willingness of two companies to disclose crucial technical information. This willingness deserves recognition, but it is a weak foundation for a security regime. Two consequences for European policy follow. In the short term, the Commission and the AI Office should clarify how serious incidents arising from pre-market evaluation are to be reported. This could be done through the Code of Practice or guidance under Art. 55, or through a dedicated mechanism. Otherwise, the next case may only become known if an AI lab chooses again to present it at a conference such as Black Hat. In the medium term, automated defensive capability is as important as reporting.