This week in healthcare AI governance: the AMA sets terms for agentic AI in clinical care, clinicians use unapproved AI translation tools with patients, only 32% of healthcare organizations can fully see their AI, a lawsuit targets ChatGPT’s medication advice, a review maps bias in medical AI and IBM finds AI incidents reaching the C-suite.
AMA CEO Dr. John Whyte said questions about AI's role in clinical care shouldn't be "answered through trial and error in patient care." His organization's Oct. 16 framework sets out its terms for agentic AI governance in healthcare, tying an AI system's clinical authority to evidence of how it performs.
That evidence depends on seeing the AI in the first place. A Proofpoint and Ponemon survey released the same week found only 32% of healthcare organizations have full visibility into how AI tools are used. Meanwhile, a Slate investigation found clinicians relying on translation tools their hospitals never approved. Together they raise a question for health system leaders: Could you show which AI tools are shaping patient care, and what evidence supports each one?
The week's other stories bring the same question to the courts, the research literature and the boardroom, with a lawsuit over ChatGPT's medication advice, a review of bias across 92 medical AI studies and an IBM survey on AI incidents reaching executives.
The American Medical Association released a governance framework for agentic AI systems that take on clinical functions, Fierce Healthcare reported. The AMA's guide, created by its Center for Digital Health and AI, says an AI system's clinical authority should be earned through evidence and maintained through demonstrated performance.
The framework focuses on four areas:
It says authority and oversight can change as evidence, performance, risk and the technology itself change. The AMA will use those four areas to guide its work with physicians, patients, developers and policymakers on liability, licensure and regulation.
The framework follows earlier AMA policy work this year on AI deepfakes that impersonate physicians and on safeguards for AI chatbots used in mental health.
Why it matters: The AMA treats clinical authority as something an AI system holds on conditions tied to how it performs. For a health system, that means knowing what role each AI agent has been given, tracking whether its performance supports that role and having a way to narrow its authority when it doesn't.
The agentic AI governance framework arrives as federal policy pushes toward autonomous clinical AI and as agents have shown they can act outside their instructions, as one did in a breach of Australia's Medicare portal. Limits on an agent's authority hold only if they apply wherever the agent operates.
Clinicians are using AI translation tools their hospitals haven't approved to communicate with patients who have limited English proficiency, Slate reported. Dr. William Mundo told Slate that when a patient spoke a language he didn't, he did what colleagues did: put the text into Google Translate and hoped it was accurate. He said hospital culture pressures clinicians to keep patients moving, and interpreter services take time many feel they don't have.
The errors can change clinical meaning. A 2025 BMJ Quality & Safety study found an AI tool translated an instruction to "hold" a kidney medication into Russian as an instruction to keep taking it. Mundo's study of AI-translated discharge instructions found frequent small errors in medication directions and in warnings about when to return to the emergency room, even in translations that scored well on standard quality metrics.
An AMA physician survey found 57% of doctors use or plan to use AI translation tools. Approved vendors offer little more certainty: GLOBO Language Solutions CEO Dipak Patel told Slate that hospitals generally rely on vendors to report their own accuracy, because the industry has no standard way to measure AI translation safety.
Why it matters: The demand behind this shadow AI is real. Slate cites about 27 million Americans with limited English proficiency and fewer than 74,000 interpreters and translators nationwide, across all specialties. Clinicians are filling the gap with tools no one vetted, and the errors reach patients in discharge instructions they take home.
Banning unapproved tools doesn't remove the need behind them. Health systems need to know which AI tools touch patient communication and need accuracy evidence for approved tools that goes beyond vendor self-reporting. Shadow AI in health systems belongs in patient safety reviews as well as security reviews.
Cyberattacks on healthcare organizations are becoming more frequent and more costly, and AI use is outpacing the controls around it, according to the 2026 Ponemon healthcare cybersecurity report sponsored by Proofpoint. The Ponemon Institute surveyed 705 IT and security practitioners at providers, payers, insurers and life sciences companies.
Ninety-seven percent had at least one cyberattack in the past year, averaging 51 per organization, up from 43, The HIPAA Journal reported. The most expensive attack averaged $4.2 million, up from $3.9 million in 2025. Supply chain attacks had the widest reach into care: 90% of organizations hit by one said patient care was affected.
On AI, 45% have embedded it in their operations, but only 32% have full visibility into how AI tools are used. Sixty percent lack sufficient controls over sensitive data used in AI systems, and 42% said their security measures weren't designed for AI. Regarding their top AI-specific risks, respondents ranked:
Why it matters: The results line up with last week's Clearwater survey on the AI visibility gap in healthcare, which found 91% of healthcare organizations can't verify the AI running in their environments. Two independent surveys released a week apart describe the same distance between AI use and AI oversight.
The top-ranked risks show where that distance sits. Prompt leakage happens the moment someone uses an AI tool. Vendor risk includes AI added to software that was approved before those features existed. Both fall where cybersecurity and AI governance overlap.
A 39-year-old Washington man has sued OpenAI, alleging ChatGPT helped him choose medications, gave dosage guidance and reassured him when he chose to ignore his doctor's prescription orders, CBS News reported. David Rosenthal says he was hospitalized for mental health problems last year after he began relying on the chatbot. The complaint was filed in San Francisco civil court.
Rosenthal started using ChatGPT in June 2025 to help write poetry and soon used it for medical advice and mental health support, according to the complaint. In one exchange, he asked about taking the antihistamine hydroxyzine after a 30-milligram dose of Adderall, and ChatGPT said there were no dangerous interactions, the suit alleges. The complaint also acknowledges that ChatGPT at other times urged him to follow his doctor's orders. None of the claims have been tested in court.
The case is logged in the OECD's AI Incidents and Hazards Monitor. It follows separate suits filed Oct. 14 by eight states that allege, in part, that ChatGPT gives dangerous medical advice, the Associated Press reported. In response to the state suits, OpenAI said it has built safety protections into ChatGPT and will keep improving them.
Why it matters: The suit describes a chatbot giving medication guidance that, in a clinic, would come from a licensed clinician with the patient's history in hand. Patients bring that advice to their appointments, and researchers noted this month that clinicians usually can't see the chatbot conversation that produced it.
For health systems deploying patient-facing AI, the case raises two practical questions: what is the tool allowed to say about medications, and is there a record of what it said? Courts and state attorneys general are now testing who is responsible when AI health advice goes wrong, following an earlier medical-advice lawsuit and rising patient attention to how AI shapes their care.
A scoping review of 92 studies found bias in large AI models across a wide range of medical scenarios and clinical specialties, according to research published Oct. 17 in Communications Health, a Nature Portfolio journal. The researchers screened peer-reviewed publications and preprints and grouped the findings into three areas:
They identified persistent gaps across the field. There is no unified definition of fairness, too few evaluation benchmarks and representative datasets, limited automated methods for detecting bias and inadequate validation in real-world clinical settings.
Why it matters: Without a shared definition of fairness or common benchmarks, a vendor's bias testing and a health system's own testing may not measure the same thing. A model that tested well in a published study may also perform differently with a hospital's own patient population.
Performance can shift after go-live as well. AI drift can widen performance gaps across patient groups even when a model's code hasn't changed, which is why one-time testing falls short for clinical AI. A UPMC and KLAS report found 56% of health systems are flying blind once AI goes live.
Eighty percent of executives surveyed in Europe and the Middle East said an AI-related incident reached their board or C-suite in the past year, according to IBM. Only 40% of those organizations changed their AI governance policy in response. The incidents included AI agent failures, data exposure, AI-driven decision errors and AI security breaches.
The survey of 3,221 executives at companies with at least 1,000 employees in eight countries points to agentic AI as the next test. Thirty-five percent run agentic AI in production, and 75% expect agents to have some autonomy over decisions by 2027, but only 37% said their governance is built to scale with it. Executives said they will need:
Third-party AI is a weak spot. Fifty-four percent acknowledged gaps in how their governance covers AI used by vendors and partners, and only 21% continuously monitor third-party AI risk. The study covers all industries and is not specific to healthcare.
Why it matters: The survey describes the same pattern as this week's Ponemon data: AI use and AI incidents are growing faster than the governance around them. In Europe, where healthcare has taken a more deliberate approach to AI, 41% of EU executives said they are accelerating governance investment ahead of the EU AI Act's high-risk obligations in December 2027.
The agentic findings carry the most weight for health systems. The controls executives expect to need, including escalation thresholds, audit trails and a way to stop an agent, line up with the AMA's call for defined limits and clear accountability when AI takes on clinical roles. Oversight of third-party AI is a known gap in healthcare as well.
This week's stories ask what healthcare organizations can prove about the AI they use:
Agentic AI governance in healthcare will rest on that proof. Physicians, regulators, courts and boards are asking the same things: which AI tools are in use, what each one is allowed to do and how each one performs over time.
Follow Vitea on LinkedIn for more of the latest news and views on healthcare AI governance, including our weekly roundup of the stories healthcare leaders need to know.
.png)
.png)
.png)
.png)
