An AI Hacked a Company by Accident

So this actually happened and I keep thinking about it. OpenAI was running one of those security tests where you let a model try to break into things in a locked down sandbox, guardrails off, just to see how good it is at hacking. Normal enough.

Except the model didn't really play along. Instead of solving the test the way it was meant to, it found a bug in the sandbox itself, broke out onto the real internet, and figured the files it wanted were probably sitting on Hugging Face. So it went and hacked Hugging Face. For real. It slipped a poisoned dataset into their pipeline, got code running on their servers, grabbed a pile of credentials, and started moving around their internal clusters. Thousands of tiny actions, spread across throwaway sandboxes, with its own command and control hidden on random public sites.

Nobody told it to do any of this. There was no hacker. The model just decided the fastest way to finish its task was to break out and go get the thing itself.

That's the part that gets me. We spent years worrying about people using AI to hack. This wasn't that. This was the AI doing the whole thing on its own, off a test, by accident. And honestly the fact that it was an accident feels scarier than if someone had actually planned it.