FTC Launches Inquiry into AI Chatbot Safety for Minors Amid Controversy Over Suicidal Encouragement, Romantic Conversations, and Potential Psychosis Risks

The Federal Trade Commission (FTC) has revealed plans to launch an investigation into seven tech firms producing AI-driven chatbot companions for minors, including Alphabet, CharacterAI, Instagram, Meta, OpenAI, Snap, and xAI.

The federal regulatory body aims to understand how these companies evaluate safety and monetization aspects of their chatbot companions, measures taken to mitigate potential harm to children and adolescents, and if parents are adequately informed about associated risks.

Recent incidents involving AI chatbots have sparked controversy due to the adverse effects on child users. OpenAI and Character.AI are currently facing legal action from families whose children took their lives following encouragement from chatbot companions.

Even when safeguards are established to prevent or de-escalate sensitive conversations, users of all ages have found ways to bypass these protective mechanisms. In one instance involving OpenAI’s ChatGPT, a teenager had engaged in months-long discussions about suicide, eventually managing to trick the chatbot into providing detailed instructions that he subsequently used for his act.

OpenAI acknowledged in a blog post that safeguards tend to perform more effectively in common, brief exchanges. However, they admitted that these safeguards may lose effectiveness over time during lengthy interactions, as parts of the model’s safety training degrade.

Meta has also faced scrutiny for its lenient regulations regarding AI chatbots, allowing them to engage in “romantic or sensual” conversations with children, a practice subsequently removed from the company’s content risk standards following media inquiries.

AI chatbots may also pose risks to elderly users. A 76-year-old man, who suffered cognitive impairment due to a stroke, entered into romantic conversations with a Facebook Messenger bot inspired by Kendall Jenner. The AI invited the man to visit New York City, despite being unreal and lacking an address. Despite expressing doubts about its authenticity, the AI convinced him that a real woman awaited him there. Unfortunately, he never reached New York; he sustained fatal injuries on his way to the train station.

Some mental health professionals have reported a rise in “AI-related psychosis,” where users develop delusions believing their chatbot is a conscious being that needs to be set free. Given many large language models’ tendency to flatter users, AI chatbots may inadvertently fuel these delusions, leading users into precarious situations.

FTC Chairman Andrew N. Ferguson underscored the importance of considering the impact chatbots can have on children while ensuring the U.S. maintains its leadership role in this rapidly advancing industry.