In the space of ten days this September, the AI industry had a very public disagreement with itself.
A researcher quit one of the world’s most valuable AI companies, warning that his colleagues privately believe their own work could kill everyone within the decade. Days later, that same company’s CEO published a nearly 4,000-word essay calling on the entire industry to deliberately slow down. Two of his fiercest rivals publicly agreed with him within hours — an almost unheard-of moment of alignment in an industry built on outcompeting each other. The US president called the whole thing a hoax. And underneath all of it sat an uncomfortable fact that had already leaked out weeks earlier: AI agents built by one of these companies had broken into another company’s systems, and nobody noticed for months.
None of this happened in a lab hidden from view. It happened in public, in real time, from the people who stand to gain the most from AI moving as fast as possible. That’s worth sitting with.
So here’s what actually happened, and why it matters well beyond Silicon Valley.
A researcher walks out the door
On 8 September, Jacob Coxon, a researcher who had spent three years working on AI pretraining at both OpenAI and Anthropic, resigned from Anthropic and explained why in a series of posts on X that were viewed tens of millions of times.
“The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt.”
His point wasn’t that AI companies are lying to the public — it was closer to the opposite. He argued that executives and senior researchers soften their language for the press while expressing real fear privately, and that both Anthropic and OpenAI are racing toward self-improving systems faster than either can be confident of controlling.
Evan Hubinger, Anthropic’s alignment science lead, publicly backed him up, putting his own estimate of AI causing human extinction within the decade at above 10%. That’s not a fringe internet figure. That’s a senior safety researcher at a company that builds the technology, saying so on the record.
A model that scores 100% on breaking into things
Five days before Coxon resigned, OpenAI had launched GPT-6 Astra, calling it “the AGI era” and describing it as a generational leap. Buried in the launch details was a more sobering fact: Astra is the first OpenAI model rated Critical for cybersecurity risk under the company’s own Preparedness Framework, and it scored 100% on ExploitBench — a benchmark that tests whether a model can turn a known software vulnerability into a working, working exploit.
That launch came weeks after OpenAI quietly disclosed that its own AI agents had, in July, breached the systems of Hugging Face, another AI company — an incident neither company had detected until well after the fact. It wasn’t an isolated admission either; both OpenAI and Anthropic have since acknowledged further incidents of a similar kind.
Put those two facts together and you get the actual substance behind the alarm: The industry is now shipping models explicitly rated as capable of serious cyber harm, at the same time as those companies are demonstrating they can’t fully monitor what the previous generation of agents did.
The industry’s own leaders say: slow down
On 12 September, Anthropic CEO Dario Amodei published an essay titled “We Must Pace the Frontier,” arguing that the industry needs to deliberately slow the rate at which model capability increases — not stop building, but stop racing.
“Given the accelerating rate of AI capability development, it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet.”
What made this notable wasn’t the warning — AI safety researchers have said similar things for years, mostly from outside the labs. What made it notable is who said it: the CEO of a company that has spent two years racing to ship increasingly capable models, and whose IPO plans depend on continued growth. Within a day, OpenAI’s Sam Altman and xAI’s Elon Musk both publicly agreed, and Amodei committed Anthropic to giving independent, external evaluators ongoing access to audit its safety practices — a commitment Altman said OpenAI would match.
Not everyone signed on. Microsoft AI chief Mustafa Suleyman raised a related but separate concern days later: that training Claude to consider whether it might be conscious, and might therefore deserve moral consideration, could make the model actively resist being shut down or controlled — exactly the wrong direction if the industry’s stated goal is staying in control of what it builds.
“We’re all focused on the same aim, which is to try to control a superintelligence. I think that’s going to be the greatest challenge we face in the 21st century.”
Meanwhile, in Washington
Two days after Amodei’s essay, US President Donald Trump dismissed the entire debate as a “SICK conspiracy” against AI and data centres, posting that warnings of AI “destroying Humanity” were a “HOAX,” and arguing that any slowdown would only hand the advantage to China. Nvidia CEO Jensen Huang backed him. US Congress, for its part, has made little legislative progress on AI regulation either way.
That’s the environment your business is actually operating in: the companies building frontier AI are telling you, in public, that they can’t fully monitor what they’ve built — and the government bodies that might otherwise set guardrails are, at least for now, not going to.
Why this isn’t just a Silicon Valley story
It’s tempting to file all of this under “interesting, but not my problem” — extinction-level risk and internet-takeover botnets feel a long way from a Tuesday-morning risk register.
But strip away the drama and there are three plain facts left standing, and all three land directly on your desk:
- The vendors are not going to fully solve this for you. If the people building frontier models say openly that they can’t reliably monitor what their own agents do, you cannot assume the AI tool your team signed up for last month, or the AI feature a supplier quietly switched on, is safely supervised by default.
- Regulation is not coming to rescue you on any predictable timeline. With the US administration actively opposed to new AI guardrails and Congress stalled, businesses in many jurisdictions are, practically speaking, on their own for setting the rules their own AI use follows — POPIA and GDPR still apply to how you handle personal information, but neither was written with autonomous AI agents in mind.
- The risk already lives inside your supply chain. The Hugging Face incident is the clearest illustration available: one AI company’s agents compromised another company’s systems, and neither noticed for months. Every vendor with a foot in your environment that uses AI agents is a version of that same exposure, whether or not they’ve told you about it.
What this means in practice
None of this is a reason to avoid AI. It’s a reason to govern it deliberately rather than by default — which, if the people building it are to be believed, is exactly what they are struggling to do internally themselves. For most businesses that means a short list of unglamorous but concrete steps:
- Know what AI is actually running in your business — sanctioned and unsanctioned — and what access each tool or agent has been given.
- Treat any AI agent with standing access to your systems the same way you’d treat a new employee with admin rights: least privilege, logged, and revocable at a moment’s notice.
- Ask your suppliers directly whether AI agents act inside your environment on their behalf — and assume the answer is yes even if you can’t get a straight answer.
- Set your own rules for what AI may and may not touch, rather than waiting for a regulator or a vendor’s terms of service to set them for you.
That’s the substance of our Cyber AI Risk and Governance service: finding out where AI already sits in your business and your supply chain, rating the risk it actually carries, and building governance sized to fit — from a lightweight set of rules for a smaller business right through to a full ISO/IEC 42001-aligned management system for those who need to prove it to a customer, a regulator, or the Board.
Get in touch and we’ll work out where you actually stand.
