BMJ Open Study Reveals Half of Chatbot Medical Answers Problematic: Which Chatbots Are Affected?

The recent BMJ Open chatbot study has sent ripples through the healthcare and AI communities by revealing that approximately half of medical answers provided by popular generative AI chatbots were problematic. This eye-opening finding highlights significant concerns about the reliability and safety of AI health advice, especially as patients and clinicians increasingly turn away from traditional search engines in favor of conversational AI interfaces driven by large language models (LLMs).

From Search Engines to AI Chatbots: A Paradigm Shift in Seeking Health Information

For decades, online health information seekers have relied primarily on search engines like Google to sift through vast amounts of content. The process required users to identify credible sources—often from academic journals, trusted websites, or systematic reviews housed within resources like the Cochrane Library, maintained by The Cochrane Collaboration. Cochrane is renowned for its rigorous synthesis of evidence, helping clinicians and patients make informed decisions.

However, with the rise of generative AI chatbots such as ChatGPT, Bard, and others, users now receive conversational, seemingly personalized answers at their fingertips. These chatbots utilize large language models (LLMs) which predict the next word in a sentence based on patterns learned from massive datasets. The result is a smooth and interactive experience, often supplanting traditional search engine queries.

Why are people making this shift?

• Convenience: Chatbots provide quick, human-like interactions without browsing multiple sites.
• Perceived personalization: Responses can be tailored to user input, creating an illusion of expertise.
• Accessibility: For non-experts, chatbots simplify complex medical language.

But this shift brings critical questions: can these chatbots be trusted for accurate, safe medical advice? And if not, what does that mean for patients who increasingly rely on them?

Trust is the Limiting Factor in AI-Driven Health Information

Trust in health information is foundational—especially when users make decisions impacting their wellbeing. The BMJ Open chatbot study directly addresses this by examining how often AI chatbot-generated medical answers were accurate, complete, and devoid of misleading information.

The findings were sobering: nearly half of the chatbot answers considered had significant issues ranging from factual inaccuracies to dangerous omissions. This fuels concerns about health misinformation spreading under the guise of AI-generated content.

Why is trust so fragile in this new AI landscape? Because LLMs operate fundamentally differently from traditional databases or curated resources. They do not “understand” in any human or clinical sense; instead, they predict plausible text sequences based on statistical patterns learned during training.

LLMs Predict Text – They Do Not Understand Truth

Large language models generate answers word by word, estimating what is likely to come next based on prior context. This makes their outputs fluent and plausible-sounding, but not necessarily factual or evidence-based. They lack direct access to the latest medical research, guidelines, or systematic reviews unless specifically integrated into their architecture.

For example, Cochrane and The Cochrane Collaboration provide gold-standard systematic reviews summarizing the best available evidence. These are painstakingly peer-reviewed and updated, representing some of the most reliable health knowledge globally. Chatbots, unless linked live to such databases, cannot reliably replicate this depth or accuracy.

Hallucinations and Invented Citations: The Trouble with AI Outputs

One of the most worrying phenomena seen cochrane.org in AI-generated medical content is “hallucination.” Hallucination occurs when an LLM produces confident-sounding but entirely fabricated information. This can include invented studies, false statistics, or bogus treatment claims.

Some chatbots also generate citations that look real at first glance—complete with plausible DOIs or journal names—but upon closer inspection do not exist. This kind of misinformation can be dangerously misleading, especially in healthcare, where clinical decisions require trustworthy evidence.

Examples of hallucination-induced risks in AI health advice:

1. Recommending ineffective or harmful medications based on non-existent trial data.
2. Claiming cures or diagnostic procedures unsupported by clinical guidelines.
3. Creating false reassurance or undue alarm based on fabricated findings.

Which Chatbots Were Assessed in the BMJ Open Study?

The BMJ Open study evaluated several leading generative AI chatbots popular among the public for health queries. These included, but were not limited to, ChatGPT by OpenAI, Google Bard, and other emerging LLM-powered systems accessible at the time of the research.

While the study did not name every product explicitly to avoid unfairly singling out providers (and recognizing models change frequently), it underscored a systemic issue rooted in how LLMs operate rather than flaws unique to any one chatbot.

What this means for developers and healthcare stakeholders:

• The need for integrating verified evidence sources like the Cochrane Library directly into AI systems to improve trustworthiness.
• Developing mechanisms to detect and correct hallucinations and invented citations dynamically.
• Enhancing transparency so users know the basis—or limits—of chatbot advice.

Standing on the Shoulders of Evidence: The Role of The Cochrane Collaboration

Across this landscape, institutions like The Cochrane Collaboration remain vital. By generating comprehensive systematic reviews on treatments and interventions through meticulous methods, Cochrane sets the benchmark for reliable evidence.

As AI chatbots evolve, partnerships integrating these trusted databases into conversational agents could bridge the gap between polished AI language generation and credible health knowledge. This would help mitigate the risk of unchecked misinformation and empower users to make informed health decisions.

What Would Change Our Mind About Using Chatbots for Medical Advice?

Given these concerns, it is natural to ask: what would convince us that AI chatbots could reliably deliver medical advice? Likely, it would require:

• A transparent audit trail showing every claim can be traced to a peer-reviewed source.
• Regular validation studies confirming low rates of errors and hallucinations.
• Clear guidance on chatbot limitations, encouraging professional consultation for significant health issues.
• Active collaboration between AI developers and healthcare evidence synthesis organizations like Cochrane.

Until these safeguards are widespread and robust, the cautious approach remains to use AI chatbots as supplementary tools—not definitive sources—for medical information.

Conclusion

The BMJ Open chatbot study serves as a critical reminder of the challenges we face as AI-powered tools transform healthcare information access. While generative AI chatbots powered by large language models offer unprecedented convenience and engagement, their limitations in understanding truth and tendency to hallucinate mean that health misinformation remains a real threat.

Trust is, unequivocally, the limiting factor. Users, clinicians, and developers alike must push for AI tools grounded firmly in verified evidence, such as the systematic reviews curated by The Cochrane Collaboration and accessible through the Cochrane Library. This will help ensure that the promise of AI in medicine does not become undone by the perils of inaccurate advice.

In the meantime, anyone seeking medical advice should remain vigilant, question AI-generated information critically, and consult healthcare professionals for decisions affecting their health.