Super intelligent AI could kill us all

Jacob Coxon, ex Anthropic researcher quit his job and sends an alarming signal to the world.

Here is his post on X from SEP 9:

I resigned from Anthropic today. I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives. More thoughts below. I resigned from Anthropic today. I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives. More thoughts below

Do not underestimate the power of this technology. These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources. We have all witnessed the progress in each of these domains, and progress is not slowing. Do not underestimate the power of this technology. These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources. We have all witnessed the progress in each of these domains, and progress is not slowing. No other human activity poses this level of danger

A common response is “if they truly believe this, why are they still building it?” At OpenAI, many have not deeply internalized the civilizational stakes. At Anthropic, the stakes are well-understood, but they are locked in a race to get there first – they believe no one else will act responsibly, so they must do it themselves, despite the risk.

Accepting this race and entering the “endgame” is a hubristic gamble that should not be launched from a private company’s Slack. Attempting to speedrun alignment should require extraordinary confidence that there are no better trajectories available. Accepting this race and entering the “endgame” is a hubristic gamble that should not be launched from a private company’s SlacIf you are a lab researcher, I urge you to consider what the next few years will actually feel like. Do you want to kick off a superintelligent RL run without a rigorous understanding of its mind? Should you put your head down because “it’s happening anyway” – or take this moment to call for different conditions? If you are a lab researcher, I urge you to consider what the next few years will actually feel like. Do you want to kick off a superintelligent RL run without a rigorous understanding of its mind? Should you put your head down because “it’s happening anyway” – or take this moment to call for different conditions? k. Attempting to speedrun alignment should require extraordinary confidence that there are no better trajectories available.

I am optimistic about the potential for coordination. Warning shots like the Hugging Face attack have made pacing agreements between U.S. labs more viable. I don’t feel like we’re on track to prevent a global race, which may require costly actions such as a temporary ban on improving model capabilities.

If you are a lab researcher, I urge you to consider what the next few years will actually feel like. Do you want to kick off a superintelligent RL run without a rigorous understanding of its mind? Should you put your head down because “it’s happening anyway” – or take this moment to call for different conditions? I

mind? Should you put your head down because “it’s happening anyway” – or take this moment to call for different conditions?

Jacob Coxon

The 27-year-old’s former boss, Anthropic head Dario Amodei, recently called for AI development to slow down – though some have questioned the motivations behind this.

The bosses of two rival AI firms, Sam Altman of OpenAI and Elon Musk of xAI, have both said they agree with Amodei’s proposal for industry-wide deceleration and regulation, as well as independent monitoring of AI model development.

Amodei wrote in an essay on Saturday that developing the technology was not in question, but the risks associated with it were „serious“, and that companies and governments must be given time to address them.

Coxon – who worked at OpenAI before joining Anthropic – welcomed the suggestion of a slowdown but said it would need to be co-ordinated with China to avoid „a race at an international scale“.

„The people who work at these companies are completely serious when they ask for regulation because they find themselves trapped in a race. And they’re scared of the outcomes of that race,“ he told Sunday with Laura Kuenssberg.

The hardest question to answer, according to Coxon, was what an AI apocalypse would look like.

One of the risks outlined in Amodei’s comments was of a swarm of bots acting like a supercomputer that could take over the internet.

Coxon said this scenario could be realistic in six months to a year.

Coxon said his peers feared the danger could even arrive in the next two years.

He said staff at AI firms were „planning what to do with their lives and thinking about the impacts of their work“, while some were „considering buying land somewhere because they’re so scared of the instability as a result of rapid AI progress“.

„They all keep this in their head on a daily basis while working on the technology.“