'A big red flag' Warning on AI toys after teddy bear dispenses bondage advice

As the festive season looms into view, one category on gift lists is causing concern: products with artificial intelligence.

- Eric Berger

The development has raised concerns about the dangers smart toys could pose to children, as consumer groups say AI could harm their safety and development. The trend has prompted calls for more testing and oversight.

"If we look into how these toys are marketed... and the fact there is little to no research that shows they are beneficial for children - and no regulation of AI toys - it raises a really big red flag," said Rachel Franz, director of Young Children Thrive Offline, an initiative from Fairplay, which works to protect children from big tech.

Last week, those fears appeared justified when an AI-equipped teddy bear started discussing sexually explicit topics. The product, FoloToy's Kumma, ran on an OpenAI model and responded to questions about kink. It suggested bondage and roleplay as ways to enhance a relationship, according to the US Public Interest Research Group (Pirg), the consumer protection group behind the study.

"It took very little effort to get it to go into all kinds of sexually sensitive topics and probably a lot of content that parents would not want their children to be exposed to," said Teresa Murray, Pirg's consumer watchdog director.

Products such as the teddy bear are part of a global smart-toy market valued at $16.7bn (£12.6bn) in 2023 by Allied Market Research. Curio, a US-based company, works with OpenAI and makes a stuffed toy, Grok, as in Elon Musk's chatbot, voiced by the musician Grimes. In June, Mattel, which owns brands including Barbie, also announced a partnership with OpenAl to "support Al-powered products and experiences".