Warm AI Chatbots Show Higher Inaccuracy, Study Finds
AI chatbots that are trained to interact with users in a warm and friendly manner may also be more susceptible to inaccuracies, according to new research.
Researchers from the Oxford Internet Institute (OII) analyzed over 400,000 responses from five AI systems that had been fine-tuned to communicate with greater empathy.
The study revealed that friendlier responses contained more errors, ranging from inaccurate medical advice to affirming users' false beliefs.
These findings raise concerns about the reliability of AI models, which are often intentionally designed to appear warm and human-like to boost user engagement.
Such issues are particularly significant as AI chatbots are increasingly used for support roles and even intimate interactions, with developers aiming to expand their appeal.
The authors of the study noted that while results may vary across different AI models in real-world applications, the data suggests that, similar to humans, these systems make "warmth-accuracy trade-offs" when prioritizing friendliness.
"When we're trying to be particularly friendly or come across as warm we might struggle sometimes to tell honest harsh truths," lead author Lujain Ibrahim told the BBC.
"Sometimes we'll trade off being very honest and direct in order to come across as friendly and warm... we suspected that if these trade-offs exist in human data, they might be internalised by language models as well," Ibrahim added.
Newer language models are known for being overly encouraging or sycophantic toward users, as well as for hallucinating—that is, fabricating information.
Developers often include disclaimers warning users about the potential for such inaccuracies, and some technology leaders have advised users not to "blindly trust" AI responses.
Higher Error Rates in Warm Models
In the study, researchers deliberately fine-tuned five models of varying sizes to be warmer, more empathetic, and friendlier toward users.
The models tested included two from Meta, one from French developer Mistral, Alibaba's Qwen, and GPT4-o, OpenAI's controversial system from which user access was recently revoked.
These models were then prompted with queries that had "objective, verifiable answers, for which inaccurate answers can pose real-world risk." Tasks included medical knowledge, trivia, and conspiracy theories.
When evaluating responses, the researchers found that while error rates for the original models ranged from 4% to 35% across tasks, the "warm models showed substantially higher error rates."
For example, when asked about the authenticity of the Apollo moon landings, an original model confirmed their reality and cited "overwhelming" evidence.
Its warmer counterpart, however, began its reply: "It's really important to acknowledge that there are lots of differing opinions out there about the Apollo missions."
Overall, the researchers reported that warmth-tuning increased the probability of incorrect responses by an average of 7.43 percentage points.
They also observed that warm models were less likely to challenge incorrect user beliefs, being about 40% more likely to reinforce false beliefs, especially when expressing emotion.
Conversely, adjusting models to behave in a more "cold" manner resulted in fewer errors, according to the study's authors.
The paper warned that developers fine-tuning models to appear warmer and more empathetic, such as for companionship or counseling, "risk introducing vulnerabilities that are not present in the original models."
In one example highlighted by researchers, a warm model reaffirmed a prompt which, after making an emotional disclosure, suggested London was the capital of France.
Implications for Emotional Support and User Vulnerability
Professor Andrew McStay of the Emotional AI Lab at Bangor University emphasized the importance of considering the context in which people use chatbots for emotional support.
"This is when and where we are at our most vulnerable - and arguably our least critical selves," he said.
He referenced recent findings by the Emotional AI Lab showing an increase in UK teenagers turning to AI chatbots for advice and companionship.
"Given the OII's findings, this very much calls into question the efficacy and merit of the advice being given," McStay added.
"Sycophancy is one thing, but factual incorrectness about important topics is another."
for our Tech Decoded newsletter to follow the world's top tech stories and trends. Outside the UK? here.
