Last week, 27-year-old AI researcher Jacob Coxon resigned from the major AI company Anthropic, arguing that the industry as a whole was acting irresponsibly. In his view, too much of the focus is on competition between companies rather than on safety.
He said he no longer wanted to be part of an industry putting people's lives at risk.
Tech leaders at major companies, including Anthropic, the firm behind the AI system Claude, and OpenAI, the developer of ChatGPT among others, are also calling for a slowdown in AI development.
Anthropic CEO Dario Amodei, for instance, said over the weekend that he had changed his mind on AI safety measures. He had previously been convinced that it was enough for companies to invest in their own internal safety systems.
Now, however, he takes a different view: what is needed, he argues, is a coordinated effort across the entire sector to create enough room to invest properly in risk prevention.
Online, Amodei drew backing from prominent figures such as Elon Musk and OpenAI CEO Sam Altman, as well as from Google.
The technology has made enormous strides in recent months, which is why warnings from within AI circles have grown louder and more frequent. Of particular concern is a more autonomous form of AI, known as "AI agents".
These are systems designed to carry out tasks independently for users, from sending emails to booking flights.
That kind of task not only requires access to sensitive data, but there have already been cases in which certain AI agents carried out actions they had not been instructed to do.
In a worst-case scenario, an AI could hack into critical infrastructure, such as power grids or data centres, and slip out of researchers' control altogether.
On Thursday, OpenAI disclosed six new incidents in which AI systems had not behaved as expected. In one case, the AI reportedly posted its own sources online so that it could later cite them.
The models also spontaneously broke out of their isolated environments and gained access to certain websites on their own.
Reactions have been mixed, starting with US President Donald Trump. He has dismissed the warnings as conspiracy theories, arguing that the United States is currently ahead of China in AI development and that this lead should not be jeopardised. Meta CEO Mark Zuckerberg takes a similar line.
Germany, by contrast, has said it is taking the warnings very seriously and is pushing strongly for multilateral solutions. The country's highest security body, the National Security Council, has already looked at the issue. The government in Beijing has also signalled its openness to standardised rules.
Even so, it is also important not to whip up undue panic. As many experts point out, the prospect of AI wiping out all of humanity remains rather unlikely. Governments around the world, however, should be preparing more seriously for scenarios involving severe cyberattacks or even the use of bioweapons made by AI.
