When Sutter Health Chief Medical Information Officer Dr. Vina Jones considered whether she should take her coughing child to the doctor, in addition to consulting her own experience as a pediatrician, she also turned to Sutter’s patient chatbot Emmy to help make her decision.

“Being a pediatrician doesn’t make me the expert on everything about my kid’s health,” Dr. Jones said. “It pulled up some [clinical] history to help inform my decision. Should I reach out to my clinician? Should I take them in? It can provide a lot of well-rounded reassurance and advice.”

It’s one example of how AI has joined other tools like X-rays in the clinical toolkit to speed proper care and important insight. In fact, it’s likely already in the exam room with you in the form of “ambient AI scribes.”

Looking at Patients Instead of Screens

AI scribes listen to a patient-doctor conversation and automatically generate clinical notes summarizing the visit, including the patient’s history, exam findings, and treatment plan. Scribes doesn’t simply transcribe. It organizes conversations into structured notes that doctors review and approve before finalizing medical records.

Kaiser Permanente’s ambient AI scribes have recorded more than 19 million patient interactions systemwide, the healthcare giant said in a statement provided for this story, while Sutter Health has run ambient documentation for over two years, widely across the system, said Jones.

These tools liberate doctors from mountains of paperwork. Before scribes became mainstream, record-keeping consumed twice as much physician time as seeing patients, per a 2016 study published in the Annals of Internal Medicine.

“It’s time clinicians spend, not just while they’re in the room with the patient typing on the computer,” said Jones, “but also at the end of the day: thinking about closing the charts, finishing notes and documentation.

“I’ve heard from clinicians about how it’s really saved them time,” she said, restoring focus to “the reason they chose medicine as a career.”

Better Diagnosis Saves Lives

AI now plays a role in diagnostics both through earlier disease detection and predictive patient monitoring.

“At Sutter Health, we’ve implemented an AI tool that is specific to lung nodule detection,” said Jones. “This has enabled us to diagnose lung cancer at earlier stages.”

Kaiser Permanente’s Advanced Alert Monitor (AAM) predicts the likelihood that hospitalized patients will decline. In a New England Journal of Medicine evaluation of nearly 44,000 alert-triggering hospitalizations, patients cared for under AAM had a 16% lower mortality rate — about 520 deaths prevented annually, Kaiser estimates.

“Predictive analytics and machine learning are unlocking new frontiers in the use of complex patient data to improve our care in real time,” said study coauthor Vincent Liu, intensive care physician and regional director of Kaiser Permanente Northern California hospital advanced analytics in a 2020 Kaiser Permanente Research report. “They augment our clinicians’ practice by finding signals hidden within the electronic health record.”

Oversight not Autonomy

Both Sutter and Kaiser Permanente stress that AI isn’t running on autopilot.

At Sutter, every tool passes through an AI governance committee of legal, administrative, clinical and digital experts that assesses the performance of underlying AI models before tools are deployed. Dr. Jones is emphatic that nothing is operating independently (agentic). Clinicians sign off on all AI-generated outputs before they enter patients’ records.

Kaiser’s approach is the same, perhaps reflecting an emerging industry consensus. Tools are validated with representative patient data, tested locally, and monitored on an ongoing basis. Doctors retain full decision-making autonomy.

The Accountability Gap

However, when lives depend on it, emerging consensus isn’t enough. Stanford law professor Michelle Mello has studied AI liability in healthcare for years and says the legal system has barely started looking at these tools.

“Very few cases have been brought,” she said. “But it’s impossible to know for sure — there’s no centralized repository. We don’t learn about case outcomes unless they went to a judge and resulted in the written opinion. Liability insurers tell me the number of claims they have received involving AI is one or zero, usually.”

Right now there’s little regulatory review confirming AI tool safety before they’re deployed.

“Developers compete on factors like user interface and how well a product integrates with existing records — not necessarily on clinical performance,” said Mello. “We hope developers are producing high-quality products and being candid about limitations, but there are not a lot of things that ensure that.”

Ask Before Trusting

Mello’s advice is simple.

“The key question to ask is how the healthcare organization ensures it’s [AI] used safely and responsibly,” she said.

“In many organizations, AI is used so pervasively that it’s not productive to have a conversation about all the places where it might come into contact with patients’ records or care,” said Mello. “But patients should be concerned if the organization can’t answer the question, ‘How do you vet AI tools to make sure that they perform well? How do you monitor them to make sure that people are staying safe?’”

Dr. Jones’s own experience is a good model. She asked her patient portal chatbot for advice on her child’s cough. BUT she made the decision.

Related Posts:
Santa Clara University, Sutter Health Partner for New 21st Century Medical School
Because Of, Not In Spite Of
Sutter Health Opens Doors at New Facility on Mission College Boulevard in Santa Clara