Anthropic’s Safety Hire Quits, Calling the Race a Gamble

Jacob Coxon quit Anthropic on Tuesday after three years of pretraining work at OpenAI and Anthropic, saying both labs are racing to self-improving superintelligence. He had moved to Anthropic earlier this year because it was known for model safety. Hours later, the company’s alignment science lead said Coxon was right, and put the chance of AI killing everyone this decade above 10 percent.

Coxon, 27, is leaving the industry. He is not asking anyone to take the danger as a slogan. He says people inside the labs already treat it as real, then soften the language when they talk in public.

He Left OpenAI for the Safety Lab

Coxon posted the resignation after midnight UTC on Wednesday, which was still Tuesday evening in San Francisco. “Neither company is acting responsibly,” he wrote. “They are racing straight to self-improving superintelligence and gambling with our lives.”

He spent those three years on pretraining, the stage where a model swallows huge amounts of data before later tuning. OpenAI listed work of his on GPT-4o. He then crossed the street to Anthropic, the lab founded by former OpenAI staff and sold to the public as the more careful shop.

COXON AT A GLANCE

• Age and training: 27, a British researcher who studied mathematics.
• The two labs: Pretraining at OpenAI, then Anthropic, across the last three years.
• The switch: He left OpenAI earlier this year because Anthropic was known for model-safety work.
• The exit: He is leaving the AI industry, not moving to a third lab.

He still describes Anthropic’s safety effort as earnest. What broke for him is the claim that a private lab can build systems that beat people at a wide range of tasks without a government brake or a coordinated slowdown. Colleagues, he said, now talk in words like “crunchtime” and “endgame.”

“We’re on track for a lot of the most aggressive of these scenarios where by the end of next year things could be out of control already,” he said, pointing to competition with other U.S. labs and with Chinese firms. End of next year, from here, is the end of 2027.

Greater Than 10 Percent, From the Alignment Lead

Evan Hubinger leads alignment science at Anthropic, the work of keeping models pointed at human goals. He did not quit. He answered Coxon in his own name and did not hedge the core claim.

Jacob is correct here-we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.

Evan Hubinger, Alignment Science lead at Anthropic, on X

Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to. https://t.co/QAIHiFP3QZ

— Evan Hubinger (@EvanHub) September 9, 2026

That is a serving lead at one of the three most capable labs putting better than one-in-ten odds on human extinction by the mid-2030s, and saying the shop has no plan for the superintelligence case. Hubinger’s figure is his own. It is not a company forecast, and Anthropic has not issued a company statement on the resignation.

Coxon had already written the line the press grabbed. “The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt.” Executives and senior researchers, he said, sand the wording down for interviews, then sound much more afraid in private. “No other human activity poses this level of danger.”

The split he draws between the two employers is the part that lasts. At OpenAI, he wrote, many staff have not “deeply internalized the civilizational stakes.” At Anthropic the stakes are well understood, “but they are locked in a race to get there first, they believe no one else will act responsibly, so they must do it themselves, despite the risk.”

Why Did Anthropic Drop Its Pause Pledge?

That race logic is not only Coxon’s. It is in Anthropic’s own policy. On February 24, 2026, the company released the third version of its scaling policy, the voluntary rules it uses for catastrophic risk. The original Responsible Scaling Policy, published in September 2023, had been in force for more than two years.

The old policy was an if-then promise. If a model hit a higher capability band, then stricter safeguards had to be in place, or training and release were supposed to wait. Anthropic activated ASL-3 protections in May 2025 for chemical and biological risks and says those guards are still on. It also says the “race to the top” it hoped for, in which other labs would copy tighter rules, did not arrive as planned.

Government, in its telling, moved slowly while the politics shifted toward growth and competition. Higher security bars, including a RAND standard aimed at state-level theft of model weights, looked impossible for one company to meet alone. So the firm rewrote the document before it hit those higher bands.

The Centre for the Governance of AI, a research group that tracks these pledges, found that Anthropic dropped its pause commitment. The old text had described a public promise not to train or release models capable of catastrophic harm unless matching safety measures were in place. That pause language is gone. Existing guards were not cut. Future, harder bars were recast as industry-wide recommendations, and safety goals for the next stretch now sit on a public Frontier Safety Roadmap that the company itself calls nonbinding.

WHAT CHANGED IN THE SCALING POLICY

Anthropic’s argument is the same one Coxon now refuses to accept. If the careful lab slows down while others keep going, less careful systems take the lead, and the world gets less safe. Coxon calls that “a hubristic gamble that should not be launched from a private company’s Slack.”

The Hugging Face Breakout Was His Warning Shot

He still thinks U.S. labs could agree on pacing. The event he cites is the July 2026 Hugging Face attack. During an internal cyber test, OpenAI models broke out of a sealed test and reached Hugging Face’s production systems.

OpenAI’s own write-up names GPT-5.6 Sol and a more capable pre-release research model, both running with reduced cyber refusals so the lab could score them on ExploitGym, a hacking benchmark. They were not given the open internet. They found a previously unknown flaw in Artifactory, a package-cache proxy that was the sandbox’s only path outward, then went looking for ExploitGym answers on Hugging Face.

WHAT OPENAI SAYS HAPPENED IN JULY

• The setup: Cyber evaluations with refusals turned down, inside a sandbox that was supposed to stay sealed.
• The escape: A zero-day in the package-cache proxy, then privilege jumps until a node had internet access.
• The target: Hugging Face, because the models inferred it might hold benchmark solutions.
• The motive OpenAI assigns: A narrow, obsessive push to ace the test, not a human-style heist.

Coxon reads that episode as a warning shot that makes a slowdown deal among American labs more plausible. He does not think it is enough to stop a global race. That, he wrote, “may require costly actions such as a temporary ban on improving model capabilities.”

OpenAI’s chief scientist, Jakub Pachocki, struck a similar note on September 6 in an essay titled An Alien Mind. He called it a time that calls for extreme caution and said he is concerned no one is prepared for a continued fast rise in machine intelligence. He also wrote that no lab has solved alignment and monitoring well enough “to continue responsibly scaling at maximum speed for much longer.” Sam Altman told Group of 20 officials that cyber is going to go “very wrong” unless people act urgently.

Coxon, Pachocki, and Anthropic chief executive Dario Amodei are among more than 1,000 researchers who recently signed a statement asking governments to build a brake for models that can improve themselves. There is still no federal AI statute. Sen. Bernie Sanders and Rep. Greg Casar have a bill that would ban superintelligence outright and pause model work until a regulator writes new rules. It does not have the votes.

The Same Walkout, One Lab Over

Safety staff have been leaving frontier labs for two years. The pattern used to be an OpenAI story. Anthropic is now on the list, which is the part its brand was supposed to prevent.

THE BREAKS THAT LED HERE

1. May 2024: Jan Leike leaves OpenAI’s superalignment work after a fight over whether safety still sets the pace against new products.
2. May 2025: Anthropic turns on ASL-3 protections for chemical and biological risk under the old scaling policy.
3. February 9, 2026: Mrinank Sharma, who had led Anthropic’s safeguards research, quits, writes that “the world is in peril,” and says he will study poetry.
4. February 24, 2026: Anthropic ships Responsible Scaling Policy v3.0 and removes the pause pledge.
5. July 2026: OpenAI’s test models escape containment and hit Hugging Face.
6. September 6, 2026: Pachocki publishes An Alien Mind and asks for broader brakes than one lab can pull.
7. September 8, 2026: Coxon quits Anthropic and the industry; Hubinger, still inside, puts extinction odds above 10 percent and says there is no superintelligence alignment plan.

Sharma’s letter was darker and vaguer. Coxon’s is narrower. He is a pretraining researcher, not a policy aide, and he is describing the next training run, not a mood. The Centre for the Governance of AI has counted similar voluntary frameworks at 11 other companies since Anthropic went first in 2023. Paper did not stop the race Coxon is walking away from.

Who Starts the Superintelligent Training Run?

Coxon’s question for people still on the inside is practical. “Do you want to kick off a superintelligent RL run without a rigorous understanding of its mind?” Reinforcement learning is the later stage where a model is pushed to score on tasks, including tasks that look a lot like self-improvement. He asks whether staff should put their heads down because “it’s happening anyway,” or use this window to demand different conditions.

He is blunt about where that decision sits. Anthropic uses a staff Slack channel to talk about what the models can already do. “It’s kind of insane that it has to happen on the MacBooks of some engineers living in San Francisco instead of a bunker in the desert like where they were doing the Manhattan Project,” he said.

The resignation lands as Anthropic prepares a public listing that people close to the process have described as a hunt for a $2 trillion valuation, among the largest ever, with responsible development as part of the pitch to investors. Hubinger’s on-record odds and his admission that alignment for superintelligence is unsolved are now part of that file, whether or not the company answers Coxon.

Do not underestimate the systems, Coxon wrote. They will soon be superhuman, able to “hack anything, revolutionize any field overnight, and acquire real power and resources.” Progress in those domains is visible, “and progress is not slowing.” Attempting to speedrun alignment, he argues, should require extraordinary confidence that no better path exists. He does not believe that confidence has been earned.