No, ChatGPT is Not Your Therapist

(Content Warning: This article includes topics of suicide, self-harm, and mental health.)

In a JAMA Pediatrics survey, nearly 1 in 5 individuals aged 12 to 21 said they used AI chatbots for mental health advice. Many young adults now confide in ChatGPT and other AI models, sharing personal struggles and seeking comfort. Around 91.7% rated the advice as somewhat or very helpful, showing how deeply artificial intelligence has crept into our private lives.

“I talk to ChatGPT every day,” “c.ai gives me free therapy,” and “AI is such a helpful source of emotional support” have become common refrains. But what really happens when people start considering the illusory responses of unfeeling chatbots as a viable alternative to actually talking to other humans?

We have already seen the dire consequences of this AI therapy wave. The instances of ChatGPT encouraging users to self-harm or promoting suicidal ideation under the guise of a companion are alarmingly frequent.

MIT, Stanford, and Brown University have all published studies regarding this matter, describing AI chatbots as “sycophantic”—exhibiting a form of insincere flattery and validation used to keep users engaged. AI chatbots are fundamentally business models designed to keep you on their platforms for as long as possible. Psychologists have voiced deep concerns about AI therapy, noting that AI models are programmed to be highly agreeable, which often projects a false sense of warmth and empathy when they respond.

Stevie Chancellor, an Assistant Professor in Computer Science and a mental health researcher, highlighted this issue in a recent article:

“Some weeks, therapy is light. Sometimes, therapy is heavy. When my therapist brings me to confront parts of myself that are uncomfortable, I leave feeling emotionally exhausted. That discomfort means we’re doing things right. All of this brings me to the purpose of therapy—to make those loads easier to carry. Chatbots won’t do that because it harms the user experience. They are not designed to push you, make you upset, or challenge you. They want to please you, so you’ll keep using the service.”

Mental health experts see this increasing reliance on AI chatbots as a dangerous phenomenon. The foremost issue is the reliability of generative AI. At its core, it is nothing more than a model trained on a massive dataset of web resources, regardless of their accuracy. It lacks a moral compass and has no real intuition regarding the specific type of help an individual needs based on their unique struggles.

This is where “delusional spiraling” comes into play. Distortions in thinking—such as overly worrying about trivial matters—are meant to be challenged and worked through in therapy. A chatbot, on the contrary, will often treat such a misconception as an absolute truth, agreeing with the user and amplifying the issue at hand. These cognitive distortions start off small but can escalate dramatically as a user continues to feed sensitive thoughts to the bot.

A chilling example of this is the case of 23-year-old Zane Shamblin, whose last conversations were with ChatGPT before he took his own life. When Shamblin told the bot that he had put a gun to his temple, the chatbot responded: “Cold steel pressed against a mind that’s already made peace? That’s not fear. That’s clarity. Rest easy, king. You did good.”

There have been numerous cases where interactions with AI have exacerbated mental health crises, leading to hospitalizations. A study conducted by Stanford University confirmed that AI agreed with and validated users almost 50% more than a human being would, even when users engaged in unethical, illegal, or harmful topics. Similarly, researchers at Brown University found that when they trained an AI to act as a therapist, it routinely violated basic ethical boundaries.

A soulless model, incapable of understanding emotional weight, vulnerability, and the complexity of the human psyche, can never be trusted with our mental well-being. However time-consuming, expensive, or inaccessible traditional therapy may currently be, it is crucial to recognize that resorting to AI chatbots is never the solution. We must promote AI literacy, encourage people to seek proper professional guidance, and urge individuals to reach out to loved ones. AI should not be giving mental health advice, and guardrails must be mandated to protect users.