'Ultimate risk': AI pioneer warns of danger of humans losing control

Robert Booth UK technology editor

- 

Jared Kaplan, chief scientist and co-owner of the AI firm Anthropic, which is based in San Francisco, below, the centre of an intense race to build advanced AI

(PHOTOGRAPHS: ONCE CLIFFORD/ THE GUARDIAN; GETTY IMAGES)

Humanity will have to decide by 2030 whether to take the “ultimate risk” of letting artificial intelligence systems train themselves to become more powerful, one of the world’s leading AI scientists has said.

Jared Kaplan, the chief scientist and co-owner of the $180bn (£135bn) US startup Anthropic, said a choice was looming about how much autonomy the systems should be given to evolve. This could trigger a beneficial “intelligence explosion” - or be the moment humans end up losing control.

In an interview about the intensely competitive race to reach artificial general intelligence (AGI) - sometimes called superintelligence - Kaplan urged international governments and society to engage in what he called “the biggest decision”.

Anthropic is part of a pack of frontier Al companies including OpenAI, Google DeepMind, xAI, Meta and Chinese rivals led by DeepSeek, racing for Al dominance. Its widely used AI assistant, Claude, is particularly popular among business customers.

Kaplan said that while efforts to align the technology to human interests had to date been successful, freeing it to recursively self-improve “is in some ways the ultimate risk, because it's kind of like letting AI kind of go”. The decision could come between 2027 and 2030, he said.

Kaplan has gone from being a theoretical physicist to an AI billionaire in seven years working in the field. In a wide-ranging interview, he also said:

Kaplan met the Guardian at Anthropic’s headquarters in San Francisco, where the interior of knitted rugs and upbeat jazz music belies the existential concerns about the technology being developed.