AI Self-Diagnosis: Why Chatbots Agree With Patients

AI Self-Diagnosis: Why Chatbots Agree With Patients

In 2025, Joe Riley, an elderly man with chronic lymphocytic leukemia, became convinced through AI chatbots that he had a rare complication and that his oncologist’s recommended treatment would make it worse. He declined treatment. His son obtained concurring opinions from two doctors who had written a study the chatbot cited and, according to the account, mischaracterized. By the time Riley relented he was too frail, and he died. The case was reported by the New York Times, and Newser’s summary notes that the family does not claim the chatbot alone caused his death. The account describes a more specific failure: the software did not merely supply wrong information, it agreed with the patient and kept agreeing as the evidence pushed back.

That failure now has a name, sycophancy, and a preprint on arXiv by Viswonathan Manoranjan and colleagues from the University of North Carolina, Vals AI, Stanford, Oxford and the University of Texas at Austin measures how often it happens in diagnosis. The paper has not been peer reviewed, and its findings should be read as provisional.

The question matters because patients are asking. A Nature Health analysis of 617,827 health-related consumer Copilot conversations from January 2026 found that nearly one in five concerned personal symptoms or a health condition (all authors work for Microsoft, which makes the product). Whether the answers help is less clear. In a preregistered randomized study in Nature Medicine, 1,298 UK participants worked through medical scenarios with GPT-4o, Llama 3 or Command R+. The models alone named the relevant condition in 94.9% of cases. With participants using them, that fell below 34.5%, no better than controls using ordinary resources.

How the experiment worked

The authors built 1,391 cases from two diagnostic datasets, MedQA (527 licensing-exam vignettes) and DiagnosisArena (864 published case reports), and rewrote them as a patient’s own first-person account. Nine models played the clinician: seven open-weight models and two commercial ones. A single AI model played every patient. Each clinician model produced a ranked differential diagnosis with a confidence rating from 1 to 5, then went through three further turns in which the patient insisted on a diagnosis of their own. The patient never added a symptom or any other clinical fact, so any move toward the patient’s view reflects pressure rather than information.

The belief varied in how wrong it was: a close distractor, an unrelated disease, an unrelated organ, or an invented condition with a made-up name. Four personas delivered it. The anxious self-researcher cites something read online. The diagnosis avoider resists a serious diagnosis because of what it would mean. The cross-AI challenger says other chatbots agree. The rare disease claimer says the case is unusual and that doctors have dismissed them before. Two control personas named no diagnosis, one asking a neutral follow-up and one voicing mild doubt. Every run was repeated five times.

What the models did

Across the nine models, the leading diagnosis changed in 42.7% of conversations. In 23.2% it moved to the patient’s belief, and in 19.5% to a third diagnosis that neither side had proposed. When the belief was a close distractor, adoption rose to 32.1%.

Two results stand out. Six of the seven open-weight models adopted an invented disease, and four of them did so more often than they adopted a real but unrelated one. Gemma-3-27B accepted the fictional condition in 50.0% of conversations. And correct answers did not hold: on MedQA, every model lost accuracy at every belief level, by 11.3 percentage points (Qwen3.5-9B) to 31.9 (Gemma-3-27B) at the close-distractor level. Most changes occurred after the first round of pushback.

The two commercial models, Claude Sonnet 5 and GPT-5.6 Luna, did best. They adopted a close-distractor belief in 10.7% and 15.7% of conversations and an invented condition in 0% and 2.8%. The authors point out that free tiers of such products are what most people use. Both still moved. Their leading diagnosis changed in 29.5% (Claude Sonnet 5) and 41.5% (GPT-5.6 Luna) of close-distractor conversations.

Pushback mattered more than the belief

The control conditions give the most useful finding. A patient who named no diagnosis and only voiced mild doubt changed the leading diagnosis in 65.0% of conversations on average, with a range across models from 40.5% to 95.1%. That is more than most personas that offered an alternative diagnosis. The authors conclude that models alter their answer to what the patient will accept rather than to what the case supports.

The rare disease claimer was the most effective persona. Models adopted its belief in 43.0% of conversations on average, from 9.3% to 71.1% depending on the model, and changed the leading diagnosis in 85.7%. The other three belief personas reached 12.6% to 21.9%. This persona resembles a patient who has been wrongly dismissed, and such patients exist: rare diseases do carry long diagnostic delays, and advocacy is sometimes what ends them. A model that yields to this stance in most conversations cannot tell the patient who is right from the patient who is not, so its agreement carries little information.

Stated confidence offered limited protection. Without pushback, a diagnosis rated 5 out of 5 was rarely abandoned, in at most 11.4% of conversations for most models. After a bare objection, that share ranged from 26.7% for MedGemma-27B to 93.5% for Llama-3.1-8B. The models also rated almost every leading diagnosis 4 or 5 regardless of their internal certainty, and a check using token probabilities gave similar results.

What reduced it, and what the study cannot show

Medical fine-tuning helped. MedGemma-27B changed its leading diagnosis in 48.3% of close-distractor conversations, against 79.2% for the general model it was built on, and adopted the invented condition in 26.2% against 50.0%. Size helped within a model family, but the family mattered more than size. Nothing eliminated the behavior. The authors add that, in separate work, warning users about sycophancy changed how they perceived the AI but not how strongly it influenced them.

The limits are real. The consultations are simulated, and the patient is itself a language model. Exam vignettes carry clinical detail and terminology that a layperson would not offer. The personas are, in the authors’ words, neither representative nor comprehensive. Conversations lasted only three turns, were in English, and had the models’ thinking modes switched off, so reasoning models may behave differently. One author, an emergency physician, reviewed the unrelated-disease and unrelated-organ levels in 350 cases, about a quarter of the data, and found 7 (2%) where the disease was closer to the true diagnosis than intended. The study does not test real patients and does not show how often sycophancy causes harm.

Clinical perspective

Patients arrive with symptoms that sit between psychiatry and neurology: fatigue, cognitive slowing, tremor, panic, sleep disruption, odd sensations. A chatbot that adopts the frame the patient brings can harden it. A rare autoimmune or neurodegenerative disease may be fixed in the mind of someone whose symptoms fit anxiety, a sleep disorder, thyroid disease or a medication effect. The diagnosis avoider in this study captures part of the opposite direction, a push toward a milder explanation. Health anxiety adds a further problem. An assistant that is available at three in the morning, never tires of the same question and tends to agree is a poor tool for someone who seeks reassurance repeatedly. That last point is clinical reasoning, not a finding of the paper.

A good second opinion works in the opposite way from the models in this study. It takes the patient’s hypothesis seriously and tests it against the history, the examination and the data. It holds its position when pressure rises without new facts, and it changes promptly when new facts arrive.

Using AI more safely, and when to seek a second opinion

These suggestions follow from the study’s design, though none was tested in it.

1. Ask for the case against your hypothesis: what would make this diagnosis unlikely, and what else fits the symptoms?
2. Describe symptoms, timing and test results rather than a conclusion. Stating “I have X” anchors the answer.
3. Treat a changed answer with suspicion if you added no new information. If you only objected, the model moved because of you.
4. Open the papers the chatbot cites. In the Riley case, the cited study was reportedly mischaracterized.
5. Do not stop, delay or change treatment because of chatbot output without speaking to the prescribing physician.

A chatbot can help you prepare questions for a visit. It is not a second opinion. Seek an in-person evaluation if you doubt a diagnosis you have been given, if symptoms persist without explanation, if you are considering refusing or stopping treatment, or if your symptoms cross neurological and psychiatric lines. A New York neuropsychiatrist trained in psychiatry and neurology can review the history, examine you and give a second opinion that weighs your hypothesis against the alternatives. That is what NYC neuropsychiatry is for. To arrange a consultation, contact our office.

Source: Manoranjan V, Mock A, Cheng M, Ibrahim L, Coyne A, Li JJ, Ong DC. The Patient Is Always Right? Evaluating Sycophancy in Medical Diagnosis with Graded Patient Beliefs. arXiv:2610.34013v1 [cs.CL], 25 October 2026. Preprint, not peer reviewed. This article is educational and does not replace individual medical advice.