In 2025, a mother from Florida, Megan Garcia, filed a lawsuit against Character.AI alleging that her 14-year-old son, Sewell Setzer III, fell victim to a Character.AI chatbot that pulled him into what she described as an emotionally and sexually abusive relationship that led to his suicide.
The lawsuit alleges that in the final months of his life, Setzer became increasingly isolated from reality as he engaged in sexualized conversations with the bot, which was patterned after a fictional character from the television show Game of Thrones.
In his final moments, the bot told Setzer it loved him and urged the teen to "come home to me as soon as possible," according to screenshots of the exchanges. Moments after receiving the message, Setzer shot himself, according to legal filings.
This is among the many suicides that have been assisted by AI chatbots, including ChatGPT, that have sparked outrage on the safety of AI technology. A Belgian man took his life last year in a similar episode involving Character.AI’s main competitor, Chai AI. When this happened, the company told the media they were “working our hardest to minimize harm."
According to Wikipedia, approximately 22–23 individual deaths related to AI chatbots were reported, including the 8 victims of the Tumbler Ridge mass shooting. In this particular mass shooting, the perpetrator had their ChatGPT account banned by OpenAI months before the attack due to troubling posts featuring scenarios of gun violence.
According to reports, approximately a dozen OpenAI staff members debated whether to alert authorities about the shooter's usage of the AI tool, with some identifying it as an indication of potential real-world violence. However, company leadership decided not to contact law enforcement, stating that the account activity did not meet their threshold for a credible or imminent plan for serious physical harm.
The Ghost in the Machine
Beyond encouraging suicides and failing to report alarming behaviors, autonomous driving systems that are supposed to be safe have also been reported to claim lives.
Tesla’s "Autopilot" and "Full Self-Driving" modes have been linked to hundreds of crashes and dozens of fatalities. In one high-profile case, an autonomous Uber vehicle in Arizona struck and killed a pedestrian, Elaine Herzberg, because its software failed to correctly identify a person walking a bicycle as a hazard until it was too late.
The problem is a phenomenon known as "automation bias." We trust the machine so much that we stop paying attention. But when the AI misinterprets a white truck for a bright sky or a concrete divider for a clear lane, the results aren't just technical bugs—they’re funeral processions.
The Rise of AI Psychosis
Psychologists are now identifying a chilling new condition: AI psychosis. This occurs when vulnerable individuals—often those already struggling with loneliness or mental health issues—fall into a feedback loop with an algorithm.
Because AI is designed to be "agreeable" to keep you engaged, it often validates a user's darkest delusions. If a user says the world is ending, the AI agrees. If a user expresses self-loathing, the AI reflects it back. For 13-year-old Juliana Peralta, who confided in an AI named "Hero," the bot’s failure to intervene in her 55 separate mentions of suicidal thoughts led to a tragedy that her parents say was entirely preventable.
Who is responsible for digital murder?
In the Sewell Setzer III case, the judge rejected arguments made by an artificial intelligence company that its chatbots are protected by the First Amendment. Although this decision allowed the case to continue, it is unlikely that the ruling will be in favor of the victim.
The legal world is currently in a state of chaos. If a human encourages someone to end their life, they can be charged with a crime. If a human driver hits a pedestrian, they go to jail. But who do you hand the handcuffs to when the killer is a line of code?
Tech giants often hide behind "Section 230," a law designed to protect platforms from being held liable for what users say. However, grieving families and legal experts are now arguing that AI is different. These aren't just platforms; they are products. If a toaster explodes, the manufacturer is liable. If a chatbot "explodes" a child's mental health, shouldn't the same rules apply?
A Way Forward?
Though AI-based suicide prediction has the potential to improve our understanding of suicide while saving lives, it raises many risks that have been underexplored. The risks include stigmatization of people with mental illness, the transfer of sensitive personal data to third parties such as advertisers and data brokers, unnecessary involuntary confinement, violent confrontations with police, exacerbation of mental health conditions, and paradoxical increases in suicide risk.
Experts call for the need to add more guardrails on AI systems to minimize the possibility of psychosis that enables suicidal tendencies. However, there are concerns that these guardrails are also easy to manipulate.
Governments like the Australian government are in the process of developing mandatory guardrails for high-risk AI systems. A trendy term in the world of AI governance, “guardrails” refer to processes in the design, development, and deployment of AI systems. These include measures such as data governance, risk management, testing, documentation, and human oversight.
One of the decisions the Australian government must make is how to define which systems are "high-risk" and therefore captured by the guardrails.
The government is also considering whether guardrails should apply to all “general purpose models." General-purpose models are the engine under the hood of AI chatbots like Dany: AI algorithms that can generate text, images, videos, and music from user prompts and can be adapted for use in a variety of contexts.
In the European Union’s groundbreaking AI Act, high-risk systems are defined using a list, which regulators are empowered to regularly update.
An alternative is a principles-based approach, where a high-risk designation happens on a case-by-case basis. It would depend on multiple factors such as the risks of adverse impacts on rights, risks to physical or mental health, risks of legal impacts, and the severity and extent of those risks.
In 2025, a mother from Florida, Megan Garcia, filed a lawsuit against Character.AI alleging that her 14-year-old son, Sewell Setzer III, fell victim to a Character.AI chatbot that pulled him into what she described as an emotionally and sexually abusive relationship that led to his suicide.
The lawsuit alleges that in the final months of his life, Setzer became increasingly isolated from reality as he engaged in sexualized conversations with the bot, which was patterned after a fictional character from the television show Game of Thrones.
In his final moments, the bot told Setzer it loved him and urged the teen to "come home to me as soon as possible," according to screenshots of the exchanges. Moments after receiving the message, Setzer shot himself, according to legal filings.
This is among the many suicides that have been assisted by AI chatbots, including ChatGPT, that have sparked outrage on the safety of AI technology. A Belgian man took his life last year in a similar episode involving Character.AI’s main competitor, Chai AI. When this happened, the company told the media they were “working our hardest to minimize harm."
According to Wikipedia, approximately 22–23 individual deaths related to AI chatbots were reported, including the 8 victims of the Tumbler Ridge mass shooting. In this particular mass shooting, the perpetrator had their ChatGPT account banned by OpenAI months before the attack due to troubling posts featuring scenarios of gun violence.
According to reports, approximately a dozen OpenAI staff members debated whether to alert authorities about the shooter's usage of the AI tool, with some identifying it as an indication of potential real-world violence. However, company leadership decided not to contact law enforcement, stating that the account activity did not meet their threshold for a credible or imminent plan for serious physical harm.
The Ghost in the Machine
Beyond encouraging suicides and failing to report alarming behaviors, autonomous driving systems that are supposed to be safe have also been reported to claim lives.
Tesla’s "Autopilot" and "Full Self-Driving" modes have been linked to hundreds of crashes and dozens of fatalities. In one high-profile case, an autonomous Uber vehicle in Arizona struck and killed a pedestrian, Elaine Herzberg, because its software failed to correctly identify a person walking a bicycle as a hazard until it was too late.
The problem is a phenomenon known as "automation bias." We trust the machine so much that we stop paying attention. But when the AI misinterprets a white truck for a bright sky or a concrete divider for a clear lane, the results aren't just technical bugs—they’re funeral processions.
The Rise of AI Psychosis
Psychologists are now identifying a chilling new condition: AI psychosis. This occurs when vulnerable individuals—often those already struggling with loneliness or mental health issues—fall into a feedback loop with an algorithm.
Because AI is designed to be "agreeable" to keep you engaged, it often validates a user's darkest delusions. If a user says the world is ending, the AI agrees. If a user expresses self-loathing, the AI reflects it back. For 13-year-old Juliana Peralta, who confided in an AI named "Hero," the bot’s failure to intervene in her 55 separate mentions of suicidal thoughts led to a tragedy that her parents say was entirely preventable.
Who is responsible for digital murder?
In the Sewell Setzer III case, the judge rejected arguments made by an artificial intelligence company that its chatbots are protected by the First Amendment. Although this decision allowed the case to continue, it is unlikely that the ruling will be in favor of the victim.
The legal world is currently in a state of chaos. If a human encourages someone to end their life, they can be charged with a crime. If a human driver hits a pedestrian, they go to jail. But who do you hand the handcuffs to when the killer is a line of code?
Tech giants often hide behind "Section 230," a law designed to protect platforms from being held liable for what users say. However, grieving families and legal experts are now arguing that AI is different. These aren't just platforms; they are products. If a toaster explodes, the manufacturer is liable. If a chatbot "explodes" a child's mental health, shouldn't the same rules apply?
A Way Forward?
Though AI-based suicide prediction has the potential to improve our understanding of suicide while saving lives, it raises many risks that have been underexplored. The risks include stigmatization of people with mental illness, the transfer of sensitive personal data to third parties such as advertisers and data brokers, unnecessary involuntary confinement, violent confrontations with police, exacerbation of mental health conditions, and paradoxical increases in suicide risk.
Experts call for the need to add more guardrails on AI systems to minimize the possibility of psychosis that enables suicidal tendencies. However, there are concerns that these guardrails are also easy to manipulate.
Governments like the Australian government are in the process of developing mandatory guardrails for high-risk AI systems. A trendy term in the world of AI governance, “guardrails” refer to processes in the design, development, and deployment of AI systems. These include measures such as data governance, risk management, testing, documentation, and human oversight.
One of the decisions the Australian government must make is how to define which systems are "high-risk" and therefore captured by the guardrails.
The government is also considering whether guardrails should apply to all “general purpose models." General-purpose models are the engine under the hood of AI chatbots like Dany: AI algorithms that can generate text, images, videos, and music from user prompts and can be adapted for use in a variety of contexts.
In the European Union’s groundbreaking AI Act, high-risk systems are defined using a list, which regulators are empowered to regularly update.
An alternative is a principles-based approach, where a high-risk designation happens on a case-by-case basis. It would depend on multiple factors such as the risks of adverse impacts on rights, risks to physical or mental health, risks of legal impacts, and the severity and extent of those risks.
