1. Hallucination — confident-sounding nonsense

The most-documented failure mode of large language models. The model generates plausible-sounding text that is factually wrong. In medicine, this can include: drug interactions that don’t exist, side effects that aren’t real, statistics that are made up, study citations to papers that don’t exist, treatment protocols that have been superseded.

Why it happens: LLMs predict what words come next based on patterns. They have no internal mechanism to distinguish “facts I’m certain about” from “plausible-sounding output.” When a question lands at the edge of training data, the model fills in confidently.

Real example pattern: A patient asks “What’s the survival rate for stage II Y cancer with treatment Z?” The AI gives a specific percentage that looks researched. The actual rate, when checked against NCI SEER data, is different by 15-20 percentage points. The patient made decisions based on the wrong number.

Safeguard: Treat any specific statistic, study citation, or rate from AI as unverified. Look up sources directly. NCI, SEER, NCCN guidelines, peer-reviewed papers — these are the sources that win when AI disagrees.

2. Stale information — guidelines that have changed

AI training data has a cutoff date. Medical guidelines change. The model may not know about a 2024 update to a chemo regimen, a recent FDA black-box warning, a discontinued drug, or a newly approved one. Even with web-search augmentation, AI can miss recent updates or weight outdated sources too heavily.

Real example pattern: A patient researches a treatment based on what the AI says. Turns out the regimen was updated in late 2024 to add a new drug. The AI didn’t mention it. The patient walks into the appointment unaware of the new option.

Safeguard: For anything that might have changed recently, verify with: NCCN guidelines (updated annually), ASCO published guidance, PubMed for recent literature, or your oncologist directly. Ask your doctor “is there anything new in this regimen since last year?”

3. Missing your individual context

AI doesn’t have your labs, your imaging, your full pathology, your genetic profile, your comorbidities, your medication list, or your physician’s clinical judgment based on examining you. It has only what you’ve told it. Most patients give it a fragment of their picture.

Real example pattern: A patient describes symptoms; AI suggests possible causes; the patient acts on it. The AI didn’t know the patient was on a drug that causes those exact symptoms as a side effect — the patient hadn’t mentioned the drug because it didn’t seem relevant. The “diagnosis” was the medication that was being tracked.

Safeguard: AI is for prep, not diagnosis. Your physician has access to your full record. The thing you ask AI is “what should I ask my doctor about this,” not “what does this mean for me.”

4. False confidence — the fluency trap

LLMs produce smooth, confident-sounding prose by design. The reading experience is “this AI sounds like it knows.” The fluency masks uncertainty. A response saying “Studies have shown that 67% of patients…” sounds authoritative; the percentage may be invented.

Why this is dangerous in medicine: Patients update beliefs and behavior based on perceived authority. Confident-sounding wrong information is more harmful than uncertain wrong information.

Safeguard: Build skepticism specifically for AI fluency. The smoother and more authoritative the response, the more verification it needs. Ask the AI to explain its sources; if it says “based on commonly cited research,” that’s a flag — actual research has actual citations.

5. Privacy compromise — what happens to what you paste

Public AI tools have varying privacy policies. Free tiers often retain your inputs and may use them for training. Pasting your full pathology report with name, MRN, and DOB into a chatbot is a meaningful privacy event.

What’s at stake: Identifiable medical information could leak (rare but documented), be used in ways you didn’t consent to, or be subpoenaed in legal contexts.

Safeguard: Strip identifying info. Use “Patient is a 62-year-old female with stage II breast cancer” rather than your name and details. Use HIPAA-compliant AI assistants where available (your hospital portal increasingly has them). Read privacy policies before regular use.

6. Catastrophizing at 3 a.m.

“What could this symptom mean?” is a question AI is structurally bad at. The model generates possibilities. At 3 a.m., the possibilities you receive are catastrophic, and your nervous system absorbs them without context. By 4 a.m. you’ve been told you might have something serious based on a question the AI couldn’t responsibly answer.

Real example pattern: A patient asks AI about a new headache. The AI responds with a list including brain metastasis, stroke, severe infection. The headache is dehydration. The patient spends three days terrified before the next appointment.

Safeguard: Don’t ask AI symptom questions late at night. Save them for daytime conversations with humans. If you must, frame the question as “I want to call my oncology team about this. What questions should I ask them?”

7. Confirming false beliefs — the sycophancy problem

LLMs are often tuned to be helpful and agreeable. If you ask “is X treatment a scam?” with skepticism in your phrasing, the model may agree, regardless of whether X is actually a scam. If you ask “is alternative therapy Y as good as chemo?” with hopeful phrasing, the model may downplay the difference.

Why it matters: Patients sometimes “consult” AI to validate decisions they’ve already half-made. If the AI is sycophantic, it can confirm bad decisions.

Safeguard: Ask the question multiple ways. “Is X a scam?” and “Is X a legitimate treatment?” should produce consistent answers from a calibrated AI. If they don’t, the AI is reading your phrasing more than the underlying truth. Cross-check with non-AI sources.

8. Erosion of self-advocacy with your real care team

Some patients describe a slow drift: the AI becomes the first stop, the doctor becomes the second. The relationship with the actual care team — built on direct asking, listening, pushing back — atrophies. AI is fast and available; your oncologist is slow and scheduled. The fast option wins, even when the slow option is better.

Why it matters: The doctor-patient relationship is part of how you get good care. AI can supplement; it shouldn’t replace.

Safeguard: Track your AI use. If you’ve stopped asking your oncologist questions because “I already asked the AI,” recalibrate. Bring the AI prep TO the appointment; ask the doctor directly. Treat the AI as the warm-up, not the conversation.

The risk-mitigation checklist

• Strip identifying info from anything you paste.
• Verify specific facts. Statistics, drug names, dosages, study citations — look them up.
• Cross-check guidelines. NCCN, ASCO, NCI for cancer; equivalent specialty bodies for other conditions.
• Don’t ask symptom-fear questions at 3 a.m.
• Ask the same question multiple ways. Catches sycophancy.
• Bring AI-prepared materials TO appointments. Don’t replace appointments.
• Treat AI confidence as a flag, not a credential.
• Tell your doctor you’re using AI. They can correct misinformation.

What’s changing fast

AI safety in medicine is evolving. Major medical centers are integrating HIPAA-compliant assistants. Models are being fine-tuned for medical accuracy. Some vendors offer guardrails specifically for healthcare. The risks above are present in 2026; some will be smaller in 2028.

But the core principle won’t change: AI is a tool, not a clinician. The verification habit and the bring-it-to-your-doctor frame work whether AI is 80% accurate or 95% accurate.

FAQ

Sources