Mother of 14-year-old boy sues Character.AI company for allegedly contributing to her son’s suicide.
This past February the community of Orlando, Florida suffered the tragic loss of young Sewell Setzer lll. Setzer had spent months messaging with a Character.AI chatbot that imitated popular Game of Thrones character Daenerys Targaryen. Even though he knew that the messages were simply generated by an AI engine and wasn’t real, he still formed a close attachment.
It is stated that the messages between Setzer and the AI would often get inappropriate or sexual with the AI. “Just… stay loyal to me. Stay faithful to me,” the AI said, according to screenshots of their messages. “Don’t entertain the romantic or sexual interests of other women. Okay?”
The AI sent to Setzer was inappropriate, and sent messages that could be very suggestive or easily misconstrued to a young mind.
“Please come home to me as soon as possible, my love,” screenshots show the AI saying to Setzer. “What if I told you I could come home right now,” he replied? “Please do, my sweet king,” the bot said in response.
The screenshots show Setzer allegedly told “Danny” as he called it, that he was considering suicide. Still, no safety protocols were enacted by this clear cry for help.
AI is new unexplored territory that we are still learning how to navigate. It is a truly unique tool that has limitless potential to be a powerful form of assistance or a harmful weapon. However, even though it’s not perfect, AI is still very accessible. Especially to the younger generation, who use it constantly.
In today’s world, AI is more and more integrated into our daily lives. This is even more apparent in the current youth. Teenagers are constantly using AI for assistance for their school work, to help create and finish stories, and as a way to have someone to talk to when they feel alone. The results of a survey done at Coral Springs Charter School showed that 80% of students use AI in their daily lives.
Children are extremely suggestive. They are very easily influenced, so it is incredibly dangerous for them to have such easy access to AI. They could search up explicit material and the AI would provide it with no problem, because that’s what it was programmed to do. Often the child nor the AI knows better, and that’s what makes it so dangerous.
AI is still new territory and not all engines are programmed with the correct protocols to manage certain situations. Many AIs have relatively low age restrictions such as ChatGPT(13-years-old) and Character ai(13-years-old). In addition to this, their means of enforcing these restrictions are very lax and users can very easily lie about their age. And many don’t report when concerning messages or questions are sent.
Although AI is an amazing tool, safety protocols must be put into place in order to keep users safe.
