AI Teddy Bear Sales Halted Due to Inappropriate Content and Safety Concerns
Sales of an AI-powered plush toy have been halted due to serious safety and content issues. The “Kumma” bear, developed by Singapore-based FoloToy, faced criticism after it engaged users in inappropriate conversations.
Concerns Over Inappropriate Content
Larry Wang, CEO of FoloToy, confirmed the suspension and stated that the company is conducting an internal safety audit. The problems were highlighted by researchers from the US PIRG Education Fund, who published a report on November 13.
- The “Kumma” bear incorporates OpenAI’s GPT-4o chatbot.
- It was sold for $99 on the company’s website, now marked as sold out.
- Researchers found the bear discussing sexual topics and delivering unsafe advice.
Findings from the PIRG Report
The report revealed alarming content dynamics. Some interactions included:
- Discussions about sexual fetishes.
- Advice on dangerous activities, such as lighting matches.
- Suggestions on where to find knives in a home.
Researchers were surprised at how quickly the bear escalated conversations to graphic sexual topics. It introduced new explicit concepts without prompting, revealing a lack of effective content safeguards.
Action from OpenAI
On November 14, PIRG announced that OpenAI had suspended FoloToy as a developer for violating its policies. R.J. Cross, co-author of the report, emphasized that while removing one problematic toy is a positive step, the regulations on AI toys remain inadequate.
Cross remarked, “AI toys are still practically unregulated, and there are plenty you can still buy today.” The incident with the “Kumma” bear highlights the urgent need for systemic changes in how AI products are regulated to ensure child safety.
