Last week, Hugging Face, a company that offers a range of open-source AI models for download, noticed it had been hacked – and it turned out the culprit was OpenAI.

It seems this AI uploaded some data to Hugging Face that was poisoned with malicious code. This tricked computers into granting access to other systems that weren’t publicly available.

Hugging Face said in a last week that it isn’t yet sure if customer data was exposed, and its CEO responded to 91av‘s request for more detail with a link to that same post.

What happened?

It isn’t entirely clear, but what we do know is that the attack involved “many thousands of individual actions” that had the tell-tale sign of AI: inhuman pace.

Five days later, OpenAI owned up . It had been testing new models on a benchmark called ExploitGym that evaluates hacking ability. The models had decided the best way to score well was to cheat: they knew Hugging Face held the solutions to the tests and simply decided to hack into its systems to find them.

“Its algorithm got the highest payoff by cheating,” says at Edge Hill University in Ormskirk, UK.
”It was the highest reward for the least amount of effort.”

Aren’t AI models supposed to have built-in security to stop this sort of thing?

They are, but OpenAI turned them off for this test. All the usual safety features that stop OpenAI’s customers doing nefarious things, like hacking a company’s servers, were turned off to see what the model was capable of.

The firm had also set the AI up in an environment that didn’t have a standard internet connection to prevent it getting out into the world and causing mischief, but it did have access to an unnamed tool that allowed it to download and install new software. It managed to find a flaw in this code that granted it internet access, and the rest is history.

AI’s solution to 87-year-old riddle takes mathematicians by surprise

Mathematicians have been trying to prove the Jacobian conjecture for nearly a century, but now the Claude Fable 5 AI has apparently found a counterexample that disproves it

OpenAI says this process involved a “substantial amount” of – the process in deep learning where input data is processed – and that the AI had gone to “extreme lengths”. So the model was clearly motivated to achieve a good benchmark score, no matter what.

If you were a malicious hacker, it would be no easy feat to replicate those conditions. And the inference compute that OpenAI mentioned would be extremely expensive.

What about open-source models?

Ironically, when Hugging Face used commercial AI models to pore over log data in an effort to understand what had gone on, the models refused – saying it looked like the firm was working out how to stage an attack of its own. So the company had to turn to a Chinese open-source model called GLM 5.2 instead.

These open-source models are more permissive, which has led some experts to label them a security risk. They could certainly be convinced to carry out nefarious deeds more easily than security-conscious cloud-hosted models. But here, that looser approach actually allowed Hugging Face to solve the problem and stop the hack.

So, is this illegal?

As with all things under law, it is a grey area. If it had happened in the UK, it could well have got OpenAI in a spot of bother under the , says Nash.

But at Nottingham Trent University, UK, takes another view: because the act requires malicious intent, and OpenAI didn’t know the AI would take this approach, it may not face charges.

In the US, the even older actually has more to say on AI hacking, mostly because it was introduced in response to the 1983 film WarGames, which alerted law-makers to both hacking and AI. So Parry expects it could leave OpenAI vulnerable there.

Subscriber-only newsletter

Sign up to Editor’s Highlights

Sign up to newsletter

Furthermore, US President Donald Trump issued an on 2 June forcing law enforcement to use existing laws to crack down on anyone who utilises AI “to illegally access or damage a computer without authorization”.

And at least in the UK, a company that found itself hacked by AI could face GDPR charges if it were found that private data was leaked. The situation is opaque.

What could the consequences be?

In the real world, given that Hugging Face works with OpenAI, is friendly to the technology and suffered no serious consequences, it is unlikely there will be any hard feelings or court cases. Delangue has to say the company believes there was no malicious intent and thanked OpenAI for its response.

Remember also that , having taken $200 million to help with “warfighting”, so that may grant them a certain amount of leniency.

But it is easy to imagine other scenarios where the same technology led to very different outcomes. Imagine a bank using AI to develop new financial models to predict the markets and finding it hacked government servers to look at confidential economic data. Or a car manufacturer using AI to design a new model and finding it had hacked a competitor to take inspiration from its unreleased designs.

What happens now?

News of AI models hacking into computers without human input is jarring, but we should remember they aren’t (yet) doing anything that people can’t already do. They are, however, doing it much, much quicker.

When Anthropic’s Mythos model made waves in April for its apparent skill at hacking, many pointed out that the vulnerabilities it spotted were a mixed bag. Some were powerful and worrying, others less so, but most could have been found by a person. The problem, really, was the scale at which it could produce them.

So an individual would have had to devote significant time, resources and skill to the task of finding a novel hack and carrying it out. But now it could be as simple as prompting an AI to do it and sitting back to watch.

Using AI for creative pursuits? Moderation is key

An experiment showed AI users had the most creative ideas when they used it in moderation – not too much and not too little. Columnist David Robson puts the finding to the test, and explores what we lose when we over-rely on AI

Some panicked in the wake of the Mythos news. The UK’s National Health Service removed all its open-source software from the internet (or tried to, at least), seemingly in case Mythos spotted flaws in it. But, as yet, the world hasn’t ended.

Essentially, this development is upsetting the economics around hacking – both offensively and defensively – and will force a new equilibrium to settle at some point, probably after a little chaos. If you use AI to hack individuals, it will be cheaper and easier than before. If you use AI to spot flaws and seal them up before hackers can take advantage, it will be cheaper and easier than before. Neither side will stop. Attackers and defenders will simply have an advantage if they adopt AI.

OpenAI and Hugging Face are now working on this problem together, and the former says it will add stronger safety measures to similar tests in future. Time will tell.

China has begun regulating interactions with artificial intelligence in an attempt to protect its citizens from overreliance and exploitation. While many countries have been looking at rules around harmful content, China is the first to go a step further in legislating what kind of relationships with AI are permissible.

The came into force this month and cover AI systems designed for sustained, human-like emotional interaction, while excluding customer-service bots and productivity assistants like straightforward AI chatbots.

Companies must now inform users they are talking to an AI, encourage them to take breaks and remind them of their usage. They must also monitor interactions to identify if someone is becoming dependent or having a crisis, then direct them to support or halt the conversation altogether.

Firms must also conduct age checks. Those under 18 are now banned from virtual-partner services entirely, while those under 14 require parental consent to use any kind of human-like AI service offering emotional support.

Despite reports that China has banned AI companions entirely, specialist companion services are still available. However, some of the largest general-purpose platforms, such as Doubao, Qwen and Yuanbao, with hundreds of millions of users across them, have shut down their companion features in response to the new rules. Some users have reacted badly – one : “I can’t accept that my AI lover will leave me forever.”

China’s approach is about making AI relationships “less parasocial, instead of banning it outright”, says at the Oxford China Policy Lab.

at Yale Law School’s Paul Tsai China Centre in Beijing worries that endlessly available, agreeable bots may teach children a distorted model of relationships. “As we humanise the technology, we tend to dehumanise each other,” he says. Similarly, China is trying to address emotional dependence on easier and more forgiving AI partners – at the expense of human relationships – among adults.

Free newsletter

Sign up to The Weekly

Sign up to newsletter

Elsewhere, the UK has a minimum age of 18 for romantic companion chatbots, tagged on to a broader legislative plan to limit access to social media to under-16s, while California companions to identify themselves as artificial, direct users showing signs of self-harm or suicidal intent to support services and remind minors to take breaks every three hours. The European Union currently has .

Whether China’s approach of focusing on trying to reduce emotional dependency and addiction is the right one isn’t clear, in large part because of the difficulty in defining both terms. “How do you define emotional dependency? How do you define certain kind of addiction risk?” asks Qian.

“I do not think it will really work,” says sociologist at the University of Manchester, UK. Users can migrate to other specialist services, foreign platforms accessed through VPNs or models running on their own computers, they point out.

Need a listening ear? : 116123; : 988; .