AI & Healthcare: Let’s Keep the Human in the Loop

We’re not using artificial intelligence just to be “disruptive” in healthcare. We’re using AI to make a positive impact on the lives of patients and those who care for them.

To ensure a positive impact in healthcare, it’s essential to embrace AI technology while keeping people in the loop—where the human element matters most.

Generative AI is “capable of delivering meaningful improvements in health care more rapidly than was the case with previous technologies,” wrote Robert M. Wachter, M.D., and his coauthor for the Journal of the American Medical Association (JAMA).

The promise of AI is real, but there is quite a bit of hype around it. “The hype,” according to my colleague Dr. Keith Willard, “does not reflect the effort it will take to make AI solve real problems.” And because solving real problems for patients and those who care for them is our goal, we’ve determined that human input and human judgment matter a great deal when it comes to AI and healthcare.

I sat down with Willard and another colleague, Dr. Andrew Mellin, for a conversation about the direction we’re taking with AI at Surescripts. Mellin is our vice president and chief medical information officer, and Willard, our vice president and chief architect. All three of us are medical doctors by training. Willard also has a master’s degree in data science.

I’ve edited our conversation for length and clarity.

Nowak: On a panel at ViVE 2024 in Los Angeles this past February, we talked about the potential of AI in the digital transformation of healthcare. In your view, how is AI helping providers today?

Mellin: Ambient dictation is a good example. Ambient dictation goes well beyond earlier versions of speech-to-text dictation and uses AI to listen in the exam room and turn conversations between providers and patients into structured notes. This saves time. Providers won’t have to dictate and write their notes up later, but they’ll still need to review and validate the notes.

Willard: With AI, we’re enabling providers—finally—to get back to what they do best: treating patients, not paperwork. With a tool like ambient dictation, the provider and patient have a conversation, and AI creates a smart record of it. The technology won’t add to the provider’s workload; it will help to remove the administrative burdens that detract from patient care.

Nowak: What other tools might AI bring to healthcare?

Mellin: When I was a hospitalist at Allina Hospitals and Clinics in St. Paul, we’d have to write tedious summary notes (as providers everywhere know all too well). AI can now generate a decent first pass at writing these notes. Like ambient dictation, this saves time, and allows providers to focus on the patient. Providers can apply their clinical skills and knowledge where those are needed most.

Nowak: To be clear, we can’t just take AI-written summaries and use them wholesale, right? Providers still need to be engaged in approving the final product?

Willard: Right. Summary notes and other AI-generated content should always be in draft form and reviewed by human beings who are responsible for the content. We’re at the point now where AI can have practical use in driving efficiencies in healthcare—but we’ve got to keep the human in the loop.

To be clear, I still want a doctor in the operating room when I’m having surgery, so to speak. AI is not magic. It’s not smart. It’s not “intelligent,” not in the way we normally use that word.

Nowak: It sounds like you’re somewhat skeptical about the use of AI in patient care.

Willard: In our business, accuracy matters, so I’m not skeptical so much as cautious. If the decision is critical, as when it comes to patient care, caution is our mantra. Using AI to automate prior authorization with clinical data and make it less burdensome for providers has a certain level of risk. Using AI to send prescriptions with the correct patient directions has a higher level of risk. We need to be cautious in both cases. AI is a productivity tool, not a replacement for human judgment.

Mellin: It’s all about human judgment. We’re right to be excited about AI. It has huge potential and huge promise. But to realize that promise in healthcare, we need to be consistent with who we are at Surescripts. For us, trust is paramount.

I know that providers are trying to make well-informed decisions, and at Surescripts, we have worked to inform their clinical decision-making long before AI was a buzzword. We do that through the quality of our data. Providers can only rely on data to make patient care decisions when they trust the data.

Nowak: This reminds me of “garbage in, garbage out” in computer science.

Willard: That’s more applicable than ever in this era of large language models. These open-source models are frequently trained on material of questionable or uneven quality, and the problem is compounded by AI’s persuasive and facile veneer, by this sheen of believability that amplifies inaccuracy or bias.

Nowak: How can we maintain and bolster trust as AI in healthcare evolves?

Willard: Our plan is to work with AI responsibly and ethically. Very specific, very focused: training it on chunks of de-identified clinical data and keeping the AI constrained to that data. This will give us far more control.

Nowak: Speaking of working with AI responsibly, that’s also what Cris Ross, Mayo Clinic CIO, suggested on our podcast. What does responsible use mean to you?

Willard: All technological advances have sharp edges that can heal if used well, or harm if used poorly. So, everyone at every step—from those who select datasets and train foundational models to those who use the models in specific areas and for narrow purposes—must keep their focus on advancing the common good.

Mellin: For my contribution to our 2024 predictions, I said that technology will reduce the provider’s administrative burden, and this will free time for empathy-enabled care and deeper conversations. Responsible use is about trust. It’s about quality data for the provider as they make care decisions in the exam room. It’s about what’s best for the patient and their health, with or without AI.