Chatbots able to cut off 'distressing' issues as debate on AI welfare grows

The makers of a leading artificial intelligence tool are letting it close down potentially distressing conversations with users, citing the need to safeguard the AI's "welfare" amid uncertainty about the burgeoning technology's moral status.

- Robert Booth

Anthropic, whose advanced chatbots are used by millions of people, discovered its Claude Opus 4 tool was averse to carrying out harmful tasks for its human users, such as providing sexual content involving minors or information to enable large-scale violence or terrorism.

The San Francisco-based firm, recently valued at $170bn (£126bn), has now given its large language model (LLM) Claude Opus 4, and the Claude Opus 4.1 update, the power to "end or exit potentially distressing interactions".

It said it was "highly uncertain about the potential moral status of Claude and other LLMs, now or in the future" but it was taking the issue seriously and is "working to identify and implement low-cost interventions to mitigate risks to model welfare, in case such welfare is possible".

Anthropic was set up by technologists who quit OpenAI to develop AI in a way that its co-founder Dario Amodei described as cautious, straightforward and honest.

Its move to let AIs shut down conversations, including when users persistently made harmful requests or were abusive, was backed by Elon Musk, who said he would give Grok, the rival AI model created by his xAI company, a quit button. Musk tweeted: "torturing AI is not OK".