A physician uses a free, web-based AI tool to summarize a complex patient history. It saves fifteen minutes of charting. However, the system hallucinates a detail or omits a critical medication allergy. When an adverse event occurs, the error is traced back to an artificial intelligence tool the hospital or clinic never knew was being used.
This scenario represents the growing problem of “shadow AI” in healthcare. Just as patients are bringing AI into the exam room, clinical teams are increasingly adopting AI tools to ease administrative burdens or assist with clinical decision-making. Recent industry analyses have highlighted the widespread use of unregistered AI as a major surprise in recent years, noting that clinical teams frequently adopt these tools without IT or compliance involvement.
The liability implications of shadow AI are severe. In standard medical malpractice claims, the focus is on whether the provider breached the accepted standard of care. When a hospital-vetted AI tool is used, liability may involve the developers of the software. But when a clinician relies on an unauthorized AI platform, the institution and the provider assume the entirety of the risk. Further, placing patient information into public AI models immediately triggers HIPAA violations, as standard Business Associate Agreements do not cover these consumer-grade tools.
Relying on the excuse that “the AI said so” is never a defensible legal posture. Regulators are actively demanding transparency. The Office of the National Coordinator for Health IT (“ONC”) recently implemented the HTI-1 algorithm transparency rules, which require certified health IT developers to ensure their predictive algorithms are fair, appropriate, valid, effective, and safe. These regulations underscore a clear expectation: institutions must know exactly what algorithms are influencing patient care and how those algorithms make decisions.
To address this vulnerability, healthcare organizations must shift from passive policies to active governance. Institutions need to recognize that outright bans on AI are rarely effective. Instead, proactive management is required.
Healthcare organizations should consider implementing the following practices:
Conducting routine network audits to identify unauthorized AI applications operating on company devices.
Updating acceptable use policies to clearly define which AI tools are approved for clinical use and which are strictly prohibited.
Providing secure, compliant, and institutionally approved AI alternatives to deter the use of public models.
Educating staff on the specific privacy and liability risks associated with consumer-grade artificial intelligence.
Healthcare providers cannot afford to ignore the use of unregistered AI. While AI holds the promise of reducing clinician burnout, an unvetted algorithm is a massive compliance and liability blind spot. By acknowledging and actively managing shadow AI, organizations can protect their patients, their providers, and their practice.
