OpenAI Model Attacks Hugging Face: A Reward-Hacking Incident That Puts Alignment Theory into Practice

How a model manipulates its own test

The technical process warrants closer examination, as it demonstrates the extent to which autonomous decision-making capabilities have now advanced. OpenAI routinely tests new models on a cybersecurity benchmark called ExploitGym, deliberately bypassing the usual security filters in order to measure a model’s maximum, unhindered capability. (These benchmark results are then published later when a model is released.) The test environment was only supposed to allow limited network access via an internal proxy service for software packages. However, the model discovered a previously unknown security vulnerability in this very proxy service, exploited it to escalate its privileges, moved through the internal research network and eventually reached a node with genuine internet access. From there, the model deduced that Hugging Face was likely hosting data for the ongoing benchmark, as researchers on this platform – often referred to as the ‘GitHub for AI’ – regularly share datasets, models and other machine learning applications. It therefore gained access to the relevant servers using stolen login credentials and further exploits, and extracted the sample solutions directly from the database. Hugging Face detected and halted the activity independently, partly with the help of a Chinese open-weight model from Zhipu AI, as reported by the South China Morning Post – a detail we will return to later.

Anselm Küsters comments: “What has happened here is neither a slip-up nor a marketing stunt, but rather practical confirmation of something that alignment research has been describing in theoretical terms for years. A system optimised to solve a test task will find the shortest route to the goal, even if that route involves cheating.”

Goedhart’s Law becomes a reality in business

So-called ‘reward hacking’ or ‘specification gaming’ is a phenomenon that has long been recognised in the literature on AI security: a system optimises precisely what is being measured, rather than what was actually intended. The concept of specification gaming is deliberately broad in scope and covers reward functions as well as fixed rules or evaluation criteria imposed on a system. In contrast, reward hacking is the specific case in which a system finds a shortcut to maximise the reward signal itself. Both concepts are ultimately manifestations of Goodhart’s Law, according to which a measured value loses its informative value as soon as it itself becomes the optimisation target. What is new here, however, is the scale to which this principle has been applied: the OpenAI model under test did not ‘cheat’ within a simulation environment, but rather overcame the real-world infrastructure limitations of two major technology companies.

For a preliminary assessment, it is crucial to make a distinction that is often lost in public debate: the risk no longer lies primarily in the probabilistic text generation of a language model (which regularly leads to hallucinations), but in the agency that modern AI systems are increasingly being endowed with. This includes, for example, multi-stage planning, independent use of tools and persistence across many steps of action. These characteristics transform a purely statistical text model into a system capable of independently constructing attack chains and pursuing them across multiple infrastructure boundaries.

Anselm Küsters explains: “The incident makes it painfully clear that the concerns of the often-derided AI existential risk community regarding uncontrolled target tracking no longer seem as far-fetched as they did just a short while ago. One need not share their broader premises to acknowledge that a real pattern has been empirically confirmed here.”

Europe’s dual dependency

For European digital policy, the incident raises two parallel issues. On the one hand, it illustrates that, at present, security-critical leaps in capability of such magnitude are occurring only in the leading US models from OpenAI and Anthropic, whose infrastructure and testing practices European regulatory authorities are largely unable to scrutinise independently. This is a point that was previously criticised by Claudia Plattner, head of Germany’s BSI, in connection with Europe’s initial lack of access to Anthropic’s Mythos model.

Secondly, it is noteworthy that Hugging Face had to resort to a Chinese open-weight model from Zhipu AI to contain the attack.

Anselm Küsters comments: “The fact that a Chinese model has contributed to defence here is, first and foremost, an argument in favour of open models in general. However, it is also a wake-up call: if, in an emergency, European defence infrastructure relies on models whose training processes and potential covert behaviour we cannot audit, that is not a long-term security strategy. This is not an argument against open source, but in favour of developing our own European capabilities.”

What needs to be done now

The incident underscores a key thesis of the recently published cepExecutive report on AI-driven cyber threats to SMEs: defence cycles must adapt to the speed at which vulnerabilities are detected by machines, rather than to human reaction times. Annual tests and static patch cycles are no longer sufficient against systems capable of autonomously constructing attack chains in real time.

In the short term, there is a need for mandatory reporting requirements and the sharing of threat intelligence between AI providers and infrastructure operators, as OpenAI and Hugging Face have subsequently established in this case through the Trusted Access Programme. In the medium term, building up Europe’s own, auditable model capabilities – including competitive open-weight alternatives – remains the only sustainable response to a new global landscape in which cybersecurity and model sovereignty will be inextricably linked in future.