Are AI Tools Ready to Answer Your Patients' Medical Questions?
A JAMA Medical News report published March 6, 2026, examines the clinical accuracy, privacy implications, and evolving regulatory landscape of patient-facing generative AI tools—with findings that demand physician attention.
The Rise of Patient-Facing AI
Generative artificial intelligence (AI) is no longer a technology confined to clinician workflows or administrative back offices. It is now arriving directly in patients' hands. In January 2026, OpenAI launched ChatGPT Health, a specialized iteration of its widely used chatbot engineered to ingest personal health data—including laboratory results, imaging reports, and wearable device outputs—and provide individualized health information to users. The launch represents one of the most significant deployments of consumer-facing health AI to date.
According to Nate Gross, MD, MBA, who leads healthcare strategy at OpenAI, of the 800 million weekly users of ChatGPT, one in four seek health-related information. That figure alone underscores the scale at which patients are already turning to AI for medical guidance—independent of, and in some cases instead of, their physicians.
The therapeutic and educational applications of these tools are expansive. Researchers and clinicians are exploring their utility across a spectrum of use cases: educating patients on women's sexual health and hip replacement surgery, generating postoperative instructions, and digitizing informed consent processes. Yet as adoption accelerates, critical questions about accuracy, privacy, and appropriate use boundaries are coming into sharper focus.
A Clinician Extender—Not a Clinician Replacer
Experts are careful to frame the appropriate role of these technologies within existing care paradigms. Cardiologist Haider Warraich, MD, a program manager at the US government's Advanced Research Projects Agency for Health (ARPA-H) and a former digital health and AI policy architect at the US Food and Drug Administration (FDA), offered a pointed perspective on the distinction.
"I hate the term AI doctor," Warraich said. "There's a lot more to me than what these technologies can do."
At their most sophisticated, Warraich and others argue, AI tools should serve as a "clinician extender," augmenting patients' preparation for clinical encounters rather than supplanting the diagnostic and therapeutic judgment of trained physicians. Danielle Bitterman, MD, clinical lead for data science and AI at Mass General Brigham and a radiation oncologist, acknowledged the structural drivers pushing patients toward these tools: "There's a reason patients want to use these models. It's so hard to access health care right now."
Accuracy Gaps: What the Evidence Shows
Despite the intuitive appeal of AI-assisted health information, accumulating evidence reveals significant performance limitations when these tools are deployed by real patients in real-world conditions.
Perhaps the most consequential finding comes from the first published study to assess ChatGPT Health's clinical triage capabilities. Using physician-written vignettes, researchers found that the tool failed to properly triage both the most and the least serious cases. The authors cautioned that under-triage of emergent conditions risks delaying or precluding lifesaving treatment, while over-triage of nonurgent presentations may drive unnecessary healthcare utilization.
A separate large-scale study led by the Oxford Internet Institute reinforces these concerns in a broader context. Researchers randomly assigned 1,300 participants to receive assistance from one of three LLMs—ChatGPT-4o, Meta's Llama 3, or Cohere's Command R+—or from a control source (typically Google) in navigating 10 physician-drafted health scenarios. The results were striking. When researchers fed vignettes directly to the LLMs, bypassing human interaction, the models correctly identified the relevant condition 95% of the time and the appropriate course of action 56% of the time. However, when actual participants interacted with the same models, correct condition identification dropped to approximately one-third, and appropriate course-of-action accuracy fell below 44%—performing no better than Google.
Coauthor Rebecca Payne, MBBS, PhD, MPH, a general practitioner at Bangor University's North Wales Medical School, identified the crux of the failure: "The limiting factor wasn't just the model's medical knowledge. It was the human-AI communication loop: people providing incomplete information, the model misinterpreting key details, and, importantly, people failing to carry forward a relevant diagnostic suggestion that the model did raise during the exchange."
Payne also noted that participants "tended to underestimate the severity in the vignettes we tested," raising the risk of false reassurance and delayed care-seeking.
The Privacy Question: HIPAA Does Not Apply
A dimension of this landscape that should concern clinicians and healthcare administrators alike is the regulatory vacuum surrounding data privacy. ChatGPT Health invites users to upload comprehensive personal health information—yet neither ChatGPT Health nor competing tools like Elon Musk's Grok are covered entities under the Health Insurance Portability and Accountability Act (HIPAA), nor are they business associates of covered entities.
Gross openly acknowledged that ChatGPT Health is not HIPAA compliant, though he emphasized that this reflects legal classification rather than negligence. OpenAI has implemented voluntary protections: ChatGPT Health conversations will not be used to train the underlying language model, and users may enable temporary chat functionality to prevent conversation storage. Yet experts remain cautious.
Bitterman was direct: "They are not held to the same legal requirements that doctors and health care institutions are." David Liebovitz, MD, co-director of the Institute for Artificial Intelligence in Medicine's Center for Medical Education in Data Science and Digital Health at Northwestern University Feinberg School of Medicine, raised an additional long-term concern: "Those assurances may not be worth that much if companies get sold."
Appropriate Clinical Applications: Lower-Stakes Support
Given the current evidence base, experts recommend a deliberate, scope-limited approach to patient use of AI health tools. Payne's guidance to patients is instructive and worth sharing: LLMs perform best as "assistants/secretaries" that help organize known information rather than generate high-stakes clinical interpretations. She advises limiting AI chatbot use to explaining medical terminology, preparing questions ahead of clinical visits, and summarizing information already conveyed by physicians.
For clinicians developing more focused tools, retrieval-augmented generation (RAG)—a technique that grounds the LLM in a medically verified knowledge base—has shown promise. Gio Cacciamani, MD, director of the Artificial Intelligence Center for Surgical and Clinical Applications in Urology at USC's Keck School of Medicine, applied this approach to develop Pub2Post, a tool that translates and summarizes peer-reviewed abstracts into patient-accessible language. More than 6,000 users have used the platform to date, with several medical journals adopting it for social media content.
Similarly, Antonio Forte, MD, a plastic surgeon at the Mayo Clinic, used RAG to create a virtual assistant for postoperative instructions—addressing the clinical reality that patients discharged under residual anesthesia or analgesic effects frequently cannot retain verbal or printed discharge guidance.
Federal Regulatory Developments
Institutional momentum is building at the federal level. In January 2026, the FDA and the Center for Medicare & Medicaid Innovation jointly launched the Technology-Enabled Meaningful Patient Outcomes (TEMPO) for Digital Health Devices Pilot—a voluntary initiative designed to evaluate a novel regulatory pathway for digital health tools targeting cardio-kidney-metabolic, musculoskeletal, and behavioral health conditions.
Concurrently, ARPA-H launched the Agentic AI-Enabled Cardiovascular Care Transformation (ADVOCATE) program, led by Warraich. Its goal is the development of LLM systems ready for FDA submission as authorized medical devices within two years. ADVOCATE's initial application targets congestive heart failure—envisioning agentic AI capable of advising patients on emergency department utilization, prescription adjustments, and medication titration, monitored by a supervisory AI "overseer" for post-deployment safety surveillance.
As Bitterman observed in assessing just how far generative AI has advanced since the release of ChatGPT three years ago: "This is so far beyond what I would have predicted 5 years ago."
For physicians in private practice, the imperative is clear: patients are already engaging with these tools. Understanding their capabilities, limitations, and privacy implications is no longer optional—it is a core component of contemporary patient care.
.png)
.png)
.png)
.png)
.png)
.png)