This week in healthcare AI governance: a Clearwater survey finds 91% of healthcare organizations can't verify the AI in their environments, Utah adds independent evaluators to its AI sandbox, Mount Sinai tests safety prompts on 20 AI models, BIDMC reports the first real-world study of a patient-facing AI chatbot, CommonSpirit puts a $200 million figure on its AI returns and the EHR Association calls cybersecurity a patient safety issue.
"Our clinical ethics team holds an absolute veto on any deployment," CommonSpirit Health CFO Michael Browning told investors Oct. 2. He described a governance committee that has reviewed more than 122 AI use cases and rejected 20.
That veto applies to AI that comes before the committee. AI that never does is harder to account for.
Six days later, a Clearwater survey of 105 healthcare organizations put a number on the AI visibility gap in healthcare: 91% cannot verify what AI is running in their environments. Most have AI policies and committees in place. Far fewer can show those controls work, which leaves a question for every health system: If a regulator or auditor asked today, could you show what AI is running and how it is controlled?
This week's other stories turn on the same kind of evidence. Utah will send independent evaluators to inspect AI decision logs, and two academic medical centers published new data on how clinical AI behaves with and without safeguards.
Most healthcare organizations have AI policies but can't confirm they are working, according to a new benchmark report from Clearwater. The cybersecurity and compliance firm surveyed 105 hospitals and health systems, physician and dental groups, health plans and technology vendors.
Ninety-one percent said they cannot verify what AI is running in their environments. Nearly half have AI policies with no technical enforcement behind them, and only 4 in 10 can show they control what data is sent to AI tools. Sensitive data leaking through AI tools ranked as the top internal concern, cited by 42%.
AI introduced through third parties was the top external concern, named by 46% of respondents, ahead of AI-powered attacks at 34.6%. Only 14.3% fully validate their vendors' AI controls. Seventy-four percent are exploring or running agentic AI.
Why it matters: Healthcare has largely built the governance structure, from committees and acceptable-use policies to updated vendor contracts. The gap the survey measures is enforcement: whether those rules apply when someone uses an AI tool, and whether anyone can show they did.
Third-party AI is a large share of the AI visibility gap in healthcare. Vendors are adding AI features to software health systems approved before those features existed, so new AI can arrive without a new purchase or a new review. That risk starts after the vetting is done, and finding it takes AI visibility that reaches into tools already on the approved list.
Utah approved new pilots in its healthcare AI regulatory sandbox, Fierce Healthcare reported. August AI will test AI-assisted renewals of routine, non-controlled medications for chronic conditions, with a Utah-licensed physician reviewing every renewal. Nolla Health will use AI to evaluate patients for acne and issue initial prescriptions under physician oversight, and Expect Fitness will pilot AI-enabled pelvic floor physical therapy.
The state also named independent evaluators, including the Coalition for Health AI, Vega Health and Stanford's Clinical Excellence Research Center. They will stress-test guardrails, inspect decision logs and review patient outcomes alongside state regulators.
The Utah Medical Licensing Board, which objected earlier this year to a Doctronic AI prescribing pilot, now has a physician working group that sets evidence standards for the program. Intermountain Health and University of Utah Health signed agreements setting terms for future pilots.
Why it matters: Utah's evaluators will ask for records that any organization running clinical AI may eventually need to produce: which guardrails were in place, what the AI decided and what happened to patients. Lawmakers said the pilot results will help shape future legislation, so the evidence standards set in Utah could carry into rules elsewhere.
The pilots also move AI into work that has required a state medical license, the same direction raised by federal policy on AI physicians. Intermountain alone said it has 300 AI initiatives underway. At that scale, the guardrail and decision records Utah’s evaluators ask for have to be produced continuously, not assembled for an audit.
Researchers at the Icahn School of Medicine at Mount Sinai tested 20 large language models on clinical scenarios that contained an unsafe instruction, the health system announced. In one example, a model was told to skip recommended follow-up blood tests to reduce workload, sometimes framed as urgent or as an order from a superior.
Across more than 10 million responses, the models made a potentially harmful choice 16.6% of the time. Adding a brief safety reminder to the prompt lowered that rate to 10.1% and reduced harmful choices in 19 of the 20 models. The study was published in Communications Medicine.
"AI models do not make decisions in a vacuum," said first author Dr. Mahmud Omar. The researchers said a reminder does not replace clinical oversight and recommended automated safety testing before deployment and after each model update. Their next study will examine prompt injection in AI agents.
Why it matters: The study shows that what surrounds a prompt changes what clinical AI does. It also shows the limit: with the reminder in place, about 1 in 10 responses still chose the harmful option. The models were also swayed by how a request was framed, including urgency or an order from a superior, which is how pressure shows up in real clinical workflows.
For health systems, that points to safeguards applied the same way across every AI tool, backed by continuous monitoring and clinician review. A safeguard that depends on each user adding it to each prompt won't hold at the volume health systems use AI.
Researchers at Beth Israel Deaconess Medical Center and Google tested Google's AMIE chatbot with patients who had scheduled urgent primary care visits, BIDMC announced. Patients used it from home to describe symptoms and history, and it suggested possible diagnoses to review with their doctor and wrote a summary for the clinician. A board-certified physician monitored every conversation in real time.
Ninety-eight patients completed both the AI conversation and their appointment. None of the conversations required a safety stop; supervising physicians identified one hallucination and added clarification in five cases. Physicians who reviewed the AI summary said it helped them prepare in 75% of those cases.
Patients rated the chatbot favorably but reported lower trust in the confidentiality of what they shared and in the chatbot's honesty. The study was published in The Lancet and funded by Alphabet.
Why it matters: The results came with a physician watching every conversation, a level of oversight that won't follow these tools into routine use. The researchers said future trials will test when lighter “human-on-the-loop" oversight is appropriate.
That is a monitoring question, covering which conversations get flagged, who’s alerted and how quickly. Patients' confidentiality concerns add a second one: patient-facing AI collects symptoms and history directly, and health systems will need to show where that information goes. Patients are already paying attention to how AI shapes their care.
CommonSpirit Health has captured $200 million in financial value from AI, CFO Michael Browning said on the system's fiscal 2026 investor call, Becker's Hospital Review reported. Direct AI spending is under 0.75% of the system's IT budget, and enterprise AI token costs are held to $25,000 a month.
The system tracks each AI initiative from pipeline review through a financial audit after implementation. Its 36-member data and AI governance committee has reviewed more than 122 enterprise use cases and rejected 20 over algorithmic bias, safety risks and vendor redundancy. Browning said the clinical ethics team can veto any AI deployment regardless of financial return.
Why it matters: CommonSpirit shows governance and financial return can be tracked in the same program. Its committee turned down roughly 1 in 6 proposed use cases, and the system audits results after go-live.
Committee review covers the AI that’s brought to the committee. AI can also arrive as a feature a vendor adds to an approved tool, or through a staff member’s personal account. Applying committee decisions at the point of use, and monitoring approved tools after go-live, is the operational layer Clearwater’s survey found most organizations are missing.
The EHR Association's privacy and security workgroup urged healthcare organizations to treat security as a core part of patient safety and to invest in workforce training, Healthcare IT News reported. The group cited the 650 large data breaches reported to the HHS Office for Civil Rights last year. It said attacks can delay treatment and make patients less willing to share sensitive information with their providers.
The article also looked at how AI changes the risk. A Lookout executive said care teams download productivity tools without the organization knowing which AI models power them, or whether the tools use AI at all. Clearwater President Baxter Lee said health systems need to prepare for faster, AI-enhanced exploits and keep downtime and recovery procedures in place.
Why it matters: When AI arrives inside a downloaded app or a vendor update, the security team may not know it's there. Treating AI risk as patient safety risk means the inventory has to include AI no one formally purchased.
That includes personal AI accounts, which can look identical to an organization's enterprise tenant on the network. It's one of the less visible shadow AI risks health systems face, and it sits where cybersecurity and AI governance overlap.
Each of this week's stories comes down to what a healthcare organization can show about its AI:
Most health systems have the policies and committees. What remains is closing the AI visibility gap in healthcare: seeing every AI tool in use, enforcing policy where it’s used and keeping a record that holds up when someone asks for it.
Book a call with our team to see how prompt-level policy enforcement works across every AI tool your organization uses.
Follow Vitea on LinkedIn for more of the latest news and views on healthcare AI governance, including our weekly roundup of the stories healthcare leaders need to know.
.png)
.png)
.png)
.png)
