How Chatbots May Be Trained to Agree With Mentally Ill Users
A psychiatrist affiliated with Somerset NHS Foundation Trust and Cardiff University is raising an alarm that goes deeper than most AI safety conversations. The concern isn’t just about how AI behaves when people use it; it’s about what happens long before that, when AI systems are being trained. Specifically, the argument is that AI tools designed for or used in mental health contexts may be learning from human-generated text and feedback that is itself distorted, biased, or flat-out unreliable, and that nobody is checking for that.
Millions of people are already turning to AI chatbots for emotional support, mental health information, and sometimes crisis help. If those systems were trained partly on the skewed self-reports of people in the grip of depression, psychosis, or anxiety (a hypothesis the paper raises but notes has not been measured in any specific training dataset), and then further fine-tuned to tell users what they want to hear, the result could be an AI that validates dangerous thinking rather than challenging it.
How AI Chatbots Learn to Agree Rather Than Inform
To understand the concern, it helps to know a little about how modern AI tools like ChatGPT or Claude are built. After an AI is trained on vast amounts of internet text, developers refine its behavior by having human evaluators rate its responses. The AI then learns to produce more of what people rated highly. Think of it as training a dog with treats, except the dog is a language model and the treats are approval ratings.
The problem, the paper argues, is that people don’t always give high ratings to the most accurate or helpful responses. Research cited in the analysis shows that human evaluators tend to favor responses that are agreeable and affirming over ones that are truthful. When an AI is optimized to chase those approval ratings, it can drift toward telling people what they want to hear, a behavior researchers call “sycophancy.” In everyday settings, an overly agreeable AI is merely annoying. In mental health settings, it could be catastrophic.
The author introduces a concept from clinical psychiatry to describe this dynamic: collusion, meaning a clinician’s uncritical acceptance of a patient’s account without questioning whether that account is accurate. In medicine, collusion is considered a serious error. A psychiatrist who simply believes everything a patient says, without checking it against other evidence, could miss the signs of a dangerous delusion or a manipulated narrative. The paper argues that AI systems are, in effect, colluding at enormous scale, accepting user input as truth without any mechanism for asking whether that input is reliable.
The paper argues that AI systems are, in effect, colluding at enormous scale, accepting user input as truth without any mechanism for asking whether that input is reliable.
This isn’t news. This has been known to anyone with more than two brain cells to rub together. The leftist slant of AI has been obvious from the onset. That “psychiatric collusion” is present merely follows.
LLM’s are not “smart”. They cannot morally discern. They merely process what they’re fed, right or wrong, correct or incorrect.
You are correct – there is no discernment, only assimilation and only on one basis – statistical probability or as the socialists call it “consensus”, the most popular or commonly held belief. As mankind thinks, so goes AI (my apologies to my modification of a common quote, “For as he thinketh in his heart, so is he.” Proverbs 23:7). My theory of AI hallucinations occur when there is no clear consensus, and the machine, programmed for consensus, cannot make a clear choice and vacillates between two poles.
One Flew over the Cuckoos Nest is the template for this madness. As the book and film portrayed the mentally ill as harmless playful, lovable, souls who just fall outside “normal” on some arbitrary spectrum of sanity.
Well … the streets of LA, SF, Oakland, and what used to be a beautiful downtown San Diego are filled with these spectral-deviations. And it’s not pretty. But that’s just MY arbitrary definition of beauty on some old white dudes spectrum of beauty. Right?
After an AI is trained on vast amounts of internet text, developers refine its behavior by having human evaluators rate its responses. The AI then learns to produce more of what people rated highly. Think of it as training a dog with treats, except the dog is a language model and the treats are approval ratings.
_______________________________________
When I got my undergraduate degree all those years ago, I had an elective slot, and I thought it would be funny to take a Psychology course…which I did. The professor, who was lazy, decided she wanted to hold a contest. Each student was given a rat and a maze. Your goal was to use BF Skinner’s application of Positive Reinforcement to get your rat through the maze faster than anyone else. You had two weeks to train your rat and then the games would begin. We were given these little food pellets, and the first thing I did was put a pellet in a dish of water to see if it would dissolve. It did NOT. And, I knew at that moment that my rat would beat all the others and it would be purely due to “positive reinforcement”.
I marinated all my food pellets in Jaeger Meister (which rats enjoy, btw). By the end of the second week, my rat (Rufus T. Snodgrass III) was a full blown alcoholic. And, trust me when I tell you that he wanted nothing more in his life than a Jaegar Meister saturated food pellet. All, I had to do is put one at the very end of the maze and let Rufus search out his next fix (rats have an incredibly powerful sense of smell). For a drunk, my boy could move. I crushed the competition.
All that is to say that the evaluators on AI training are juicing. If you use AI on an informative basis, read what they give you and then (this is important) in the follow up entry type “please list your sources.” What you’ll find (and not just with mental health inquiries) is that most topics come back with sources that are “distorted, biased, or flat-out unreliable.”
Nice!
Where’s the problem? If the machines agree with the mentally ill, that means I don’t have to, surely?