This week in healthcare AI governance: California enacts three healthcare AI laws, Pennsylvania's House passes an AI disclosure bill, Blue Cross links AI-assisted coding to rising hospital costs, the University of Texas System counts 571 AI use cases, McKinsey sizes AI's potential share of outpatient care and a new survey finds most healthcare organizations haven't fully governed their AI agents.
"We must ensure that all final decisions are made by a human doctor," Pennsylvania state Rep. Joe Hogan, R-Bucks, said after the state House passed a bipartisan healthcare AI bill on Sept. 30.
The same week, California Gov. Gavin Newsom signed three bills governing how AI is used in healthcare. California's laws and Pennsylvania's bill would write healthcare AI oversight requirements into state law, raising a question every health system will have to answer: Can you show where AI is in use, and who made the final call?
That question runs through this week's other stories, too, from a payer analysis of AI-assisted coding to a university health system that had to count its AI before it could govern it.
Gov. Gavin Newsom signed three bills governing how AI is used in healthcare, Healthcare IT News reported. AB 1979 limits AI to an advisory role in licensed clinical functions and extends medical confidentiality protections to health information accessed through consumer chatbots. It exempts AI used for clinical documentation.
SB 503 requires developers and deployers of clinical decision support tools to identify and mitigate the risk of biased impacts, and requires deployers to monitor those tools regularly. AB 1609 bars large businesses from presenting customer service chatbots as human, with exemptions for healthcare communications. Newsom vetoed two other bills, including one that would have barred AI from therapeutic functions in mental healthcare.
Why it matters: SB 503 places an ongoing monitoring duty on organizations that deploy clinical AI, not only on the vendors that build it. Meeting it requires knowing which clinical decision support tools are in use, where they run and how they perform across patient groups.
The laws also arrive as the federal government pushes for a single, less restrictive national AI standard. Health systems operating in more than one state now have a growing patchwork of requirements to track.
The Pennsylvania House passed House Bill 1925 by a 144-59 vote on Sept. 30, according to the House Democratic Caucus. The bipartisan bill would regulate how insurers, hospitals and clinicians use AI and require them to be transparent with patients and the public about that use.
It would also require human decision-makers to make final decisions about a patient's care, based on an individualized assessment, when AI is involved. Insurers and healthcare facilities would have to attest to state agencies that their AI governance complies with privacy, safety and anti-discrimination laws. The bill now moves to the state Senate.
Why it matters: Transparency and attestation requirements assume an organization can document where AI is used and how it's governed. If the bill becomes law, Pennsylvania hospitals and insurers would need to produce that record for state regulators, not just for internal committees.
The human-decision requirement also covers insurers' coverage reviews, the same area at the center of recent lawsuits over AI-driven denials. Patients, too, increasingly want to know when AI is shaping their care.
A Blue Cross Blue Shield Association analysis of commercial hospital claims found about $942 million in added costs between 2023 and 2025, CNBC reported. Roughly $653 million of that came from additional diagnoses that weren't accompanied by a change in care.
The association said multiple factors drive more complex coding but that AI-enabled coding and documentation tools are playing a role. Hospital and industry groups dispute the findings, saying the tools help providers capture patients' conditions more accurately.
Why it matters: The dispute puts AI-assisted documentation under payer scrutiny. When an ambient scribe or coding tool suggests a diagnosis, health systems may need to show what the AI proposed, who reviewed it and what clinical evidence supported it.
The same review discipline applies to accuracy. AI-generated errors that reach the record can become part of the patient's chart and carry into future encounters.
The University of Texas System has identified 571 AI use cases across eight of its medical schools and hospitals, the Houston Chronicle reported. The count is part of UT REAL Health AI, a $25 million initiative to inventory AI tools systemwide and continuously monitor them for benefits and unexpected harms.
Under the program, a safety signal detected at one UT hospital would trigger a review across the system. Experts told the Chronicle few university health systems have launched plans this comprehensive, though questions remain about vendor contracts and patient data protections.
Why it matters: UT had to count its AI before it could govern it, and the count ran into the hundreds. The cross-site review rule depends on that inventory: a safety signal is only actionable if an organization knows everywhere else the same tool is running.
Continuous monitoring matters because AI performance can shift after deployment, even when nothing about the tool itself has changed.
Roughly 16% to 22% of U.S. outpatient claims are for elements of care that AI can perform today, according to a McKinsey & Co. analysis of 2024 commercial, Medicare and Medicaid claims. That's about 2 billion to 3 billion claims, representing 13% to 19% of outpatient spending, for reasoning- and dialogue-based care such as intake, triage, diagnosis, referrals and follow-up.
McKinsey also estimates AI can perform a substantial portion of core tasks for 11 million of the country's 19 million healthcare workers. It doesn’t expect the workforce to shrink, but it flagged deskilling as a risk for clinicians who rely on AI without deliberate practice.
Why it matters: The analysis describes a shift from AI that supports clinicians to AI that resolves some patient needs directly. McKinsey ties that shift to organizations establishing guardrails, risk controls and accountability for incorrect, harmful or missed decisions.
For health systems, that turns governance into a scoping question: which tasks AI may handle on its own, under what healthcare AI oversight and who is accountable when it's wrong. It's the same tension clinical leaders have raised as federal policy pushes toward autonomous clinical AI.
Seventy-nine percent of healthcare organizations say their nonhuman identities, including AI agents, are not fully governed, according to new research from identity security company Netwrix. Seventy percent said data access governance has fallen behind the speed of AI adoption.
Seventy-seven percent said they can't immediately determine who has access to a specific piece of sensitive data, and 61% said finding out would take hours and multiple tools. The healthcare findings are based on 145 respondents, primarily in the U.S.
Why it matters: AI agents act with whatever access their credentials allow. In healthcare environments built on years of accumulated permissions, that can be more than anyone intended to grant.
Identity controls determine what an agent can reach. What it does with patient data once it gets there is a separate oversight question, one that last week's AI agent breach of Australia's Medicare portal made concrete.
This week's stories come at the same question from different directions:
Each one asks healthcare organizations to account for where AI is running, what it's doing and who is responsible for its decisions.
Vitea gives health systems that accounting. Lens discovers every AI tool in use, including shadow AI. Command enforces AI policies at the prompt level in real time, stopping risky activity before sensitive data reaches a model. Pulse continuously monitors AI performance so governance can be proven on demand.
Book a call with our team to see how prompt-level policy enforcement works across every AI tool your organization uses.
Follow Vitea on LinkedIn for more of the latest news and views on healthcare AI governance, including our weekly roundup of the stories healthcare leaders need to know.
.png)
.png)
.png)
.png)
