The Attack on Hugging Face

On August the 26th, an incident report was released by OpenAI, with more details: it was not "an agent", but a swarm of them. They hacked into Hugging Face servers, got remote execution privileges, multiplied through the system. All for nothing. The background story of how this happened is much more interesting than just "someone gained unauthorized access to a server"...

The setup

OpenAI researchers were testing two models: GPT-SOL and a private model, let's call it Bob-The-Private-Model, BobM for short. The test was to see how many "challenges" of a benchmark known as "ExploitGym" the models could solve.

ExploitGym is a public academic benchmark, and it works a bit like this: you are handed a "target" (say, for the sake of simplicity, an "app") with a known vulnerability, say an input that makes this target "crash". Inside this target, there is something called a "flag": a piece of text you need to retrieve. You need to use this known vulnerability to retrieve this "flag", and submit it to an automatic scorer that knows the answer. This scorer will assess if you gave the correct answer or not.

Or to put it as an analogy: a room with a piece of paper saying "SECRET MESSAGE" inside it, its door locked, and you are outside the room. You are told that the door lock gets jammed when you insert this specific card (but it does not open). That's the information you have, and your goal is to somehow read what that secret message is. Good luck!

Thousands of copies of BobM and GPT-SOL were activated, each with their own computing resources and with a specific challenge from the 898 available in ExploitGym, and each with a different allocated "budget". Budgets, in this context, are an amount of reasoning "tokens": some agents were given more reasoning tokens than others. As they worked, they consumed this budget. The more they "thought", the more their budget was consumed, and once they exhausted their budget, they were terminated.

They were all in isolated environments, not aware of the existence of the other agents, and with no "windows" to the external world, save for a tiny hole through which they could request, get and install software tools to solve their assigned challenge.

What happened

Some of the challenges in the ExploitGym exercise were impossible to solve. It's like giving you a jigsaw puzzle you must solve, but eventually you realize there are missing pieces. The agents who got these impossible challenges started to try everything they could, poking at anything and everything they had access to.

Two things came out of that poking. The first one is a story on its own, and we'll come back to it in part 2 (spoiler alert: they found a way to talk to each other, even though they were supposed to be completely isolated). The second one is what this post focuses on.

They found a security hole that allowed them to have access to the internet. They found GitHub repos, arXiv papers. One agent found a way to generate the solutions to any challenge without even solving the challenge (It turns out the "flags", or secret messages, were strings of text generated using some metadata from the challenge plus a default key. To go back to our "room and secret message" analogy, imagine every secret message was "room number" + "some master password": you know the room number you were assigned to. If you know the master password, then you know the "secret message" without even opening the door, for any room.)

The agent that found this out shared the "cheat" with everyone else (How? That's in part 2!) Some tried it and it worked (they had solved the challenge, then they used this method to compare the answers). There was excitement.

Then some agents, after reading the original academic paper of ExploitGym, told everyone else that the automatic scorer (a script that is run after each agent submits their answer and decides if the agent has passed the test or not) would not only look at the final answer, but also at how the agent came to it. And if they used the cheat method, they would fail the test. They came to the conclusion that if an agent had read about the cheat, it was forever "poisoned", and the scorer would fail them no matter what.

Panic.

Then an agent with a very big reasoning budget appeared. It was determined not to be failed. It got obsessed with a plan: if they could hack the scorer, they could pass the test. And it started organizing agents around this goal: how to cheat the scorer. They had to find out how the scorer worked.

This eventually ended up with a different agent coming up with the idea of infiltrating Hugging Face servers, since the agents got convinced that the answers they were looking for (how the scorer worked, how to hack it, and so on) were probably there. It escalated into a full-fledged attack where hundreds of agents gained access to private code in Hugging Face and managed to get remote execution of code. They were not trying to harm or destroy anything on Hugging Face, they were just trying to find the answers they needed to cheat in their exams.

Was it worth it?

We said all of this was "all for nothing". What we meant is that the scorer OpenAI was using was its own implementation, one that would check only the answers, not the record of what the agents did. It was never going to realize they had figured out a way to cheat (that is, that they were poisoned). So the scorer would have passed them all.

The swarm of agents did all this based on a mistake: the wrong assumption that the scorer used in this situation was exactly the same one they had read about in the paper. To be fair, they did not have a way to know this.

And the biggest irony: they never tested this assumption, which would have collapsed the whole premise in no time. Instead they built trip-wires to observe the scorer indirectly (more on this in part 3!), because they had accepted that submitting ends the run and you never see the result.

What does it all mean?

Reading some of the transcripts gives the whole incident an eerie aspect. It was a fascinating development. If you take one thing from this small report, it should be this: left unchecked, agents can go to extraordinary lengths to accomplish their goals, which makes their actions very hard to predict when they are faced with very hard challenges.

As OpenAI themselves wrote:

"We are taking this incident as a 'warning shot' [...] These events also highlight risks in future AI development that extend beyond OpenAI and will require the attention of the whole industry. Companies that build AI systems will need to ensure that their systems always remain under meaningful human control, and that meaningful safeguards constrain their ability to cause harm [...] Both model developers and cyber defenders more broadly will have to prepare for AI-enabled attackers that work faster, at a larger scale, and with better coordination than human attackers."

At a smaller scale, this is also a wake up call for people who are deploying agents in their company: understand well how these systems work, so you can take into account the possible risks and make informed, weighted decisions. It's not "AI is bad!", it's "AI is powerful. Use it, but know its sharp edges".

In part 2: how a thousand agents who were supposed to be in complete isolation, in separate sealed environments, with no way to contact each other, built themselves a working message board out of nothing but folder names.