.png)
It's the question behind every morbidity and mortality conference, every incident report, every root cause analysis: What could we have done to prevent a bad patient outcome? For nearly three decades, it has driven the patient safety movement. I've come to believe that same question is now one of the most important ones we should be asking about artificial intelligence. Many health systems don't yet have a way to answer it.
Consider a tool that is deployed across hundreds of U.S. hospitals to flag patients at risk of sepsis. When Dr. Wong et al. at the University of Michigan independently validated it against more than 38,000 hospitalizations, they found it missed roughly two-thirds of sepsis cases while generating a heavy burden of false alarms. Their criteria suggest that the tool performs well below what was advertised.
Sepsis is a condition where early treatment can dramatically improve a patient’s chance of survival. A model that misses most of the patients it was meant to catch isn't a technical footnote, but rather a patient safety event that can repeat, silently, thousands of times before anyone notices.
That's the part that should give us pause. This wasn't a rogue app. It was a sanctioned clinical tool, running inside the electronic health record, trusted by the people using it.
And it isn't an isolated story. A national eating-disorder organization replaced its human helpline with a chatbot, then pulled it within days after it recommended calorie restriction and weight loss to the very people it was meant to protect.
In controlled testing, AI chatbots asked about the most commonly prescribed medications in the U.S. have produced guidance that expert reviewers judged capable of causing serious harm if a patient or clinician acted on it, and research shows ordinary people often can't tell an unsafe AI answer from a sound one. To date, no patient death has been publicly attributed to an AI-generated dose. But the fact that we're leaning on that distinction — harm shown in testing, not yet counted at the bedside — shouldn't be reassuring.
Other cases are already in court. Families have brought wrongful-death suits alleging that consumer AI companions failed to recognize a person in crisis and never pointed them toward help; several have moved toward settlement. Insurers have been sued over algorithms alleged to have cut off care for elderly patients against their physicians' recommendations, with the large majority of those denials later reversed on appeal.
Whatever each case ultimately decides, the direction is clear: AI is already making decisions that reach patients, and it's already failing in ways we would recognize, in any other context, as harm.
None of this is an argument against AI in medicine. Used well, these tools catch things we miss, ease the documentation burden that's burning out our workforce, and extend scarce expertise into places that have too little of it. I've seen that upside firsthand. The argument is narrower and, I think, harder to dismiss: capability without control is exposure. The same model that can flag a deteriorating patient earlier can also miss that patient silently, and only one of those outcomes shows up in a product demo.
We have, in a sense, been here before. In 1999, the landmark report To Err Is Human estimated that as many as 98,000 people were dying each year in American hospitals from preventable error. The response, of course, wasn't to stop practicing medicine, but rather was to build infrastructure: incident reporting systems, near-miss tracking, root cause analysis (RCA), the Joint Commission's sentinel event process, the National Patient Safety Goals.
We built a culture and the machinery for surfacing error, investigating it, and preventing the next one. When a clinician makes a mistake today, a system exists to catch it and learn from it.
Here's the problem. Every one of those systems was built for human error.
There is not yet a standard sentinel event process for a model that misses a significant share of sepsis cases, or a consistent RCA pathway when a chatbot provides potentially harmful advice. Many organizations also lack an incident workflow for cases in which an algorithm influences care at scale in ways clinicians did not intend.
AI error behaves differently from the kind we designed our safety systems around; it's often invisible, it moves at machine speed, and a single flawed model can touch thousands of patients before anyone senses that something's off. The safety net we spent a quarter century building has a hole in it, precisely where AI now sits.
And AI now sits nearly everywhere. In a recent survey of health system leaders by KLAS Research and UPMC's Center for Connected Medicine, more than 90% reported deploying third-party AI, while less than half actually had the testing environments or governance frameworks to validate those tools before or after go-live.
Adoption has outpaced oversight. That gap is, at its core, a patient safety problem.
This is why I've stopped thinking of AI governance as paperwork or legal cover, and started thinking of it as the patient safety infrastructure of the AI era.
Real governance does for AI what incident reporting and RCA did for human error, except it has to operate at the speed AI operates. It means:
We built modern patient safety on a simple conviction: harm isn't inevitable, and systems can be designed to prevent it. AI deserves the same conviction. The health systems that thrive with AI won't be the ones that adopt it fastest, or the ones that avoid it out of fear. They'll be the ones that can see it, govern it, and trust it — because they built the safety layer underneath it first.
That's the work we do at Vitea. We give healthcare organizations visibility into every place AI is being used, the ability to set and enforce policy on AI behavior in real time, and continuous assurance that those tools keep performing safely long after they go live. Because in healthcare, AI governance IS patient safety.
Murali Naidu, MD, is Chief Medical Officer at Vitea, where he leads clinical strategy for AI governance in healthcare. A former hospital CEO and Chief Clinical Officer, he brings 25+ years of patient safety and clinical leadership experience to the challenge of making AI safe at the bedside.
.png)
.png)
.png)
