Can AI chatbots save doctors time while avoiding bias? : Short Wave A doctor's job is to help patients. With that help, often comes lots and lots of paperwork. That's where some startups are betting artificial intelligence may come in. The hope is that chatbots could generate data like treatment plans that would let doctors spend less time on paperwork and more time with their patients. But some academics warn biases and errors could hurt patients.

Have a lead on AI in innovative spaces? Email us at!

Will artificial intelligence help — or hurt — medicine?

  • Download
  • <iframe src="" width="100%" height="290" frameborder="0" scrolling="no" title="NPR embedded audio player">
  • Transcript


You're listening to SHORT WAVE...


KWONG: ...From NPR.

Hey, hey, SHORT WAVErs. Emily Kwong here. Our quest to dissect artificial intelligence continues with my friend Geoff Brumfiel from the NPR Science Desk. Hi, Geoff.


KWONG: Hi. OK. It honestly seems like everyone is trying out ChatGPT these days, including medical professionals. You told me about this guy, Cliff Stermer, a rheumatologist on TikTok...


CLIFF STERMER: Save time. Save efforts. Use these programs, ChatGPT, to help out in your medical practice.

KWONG: ...Which is wild to me because he's not alone. Cliff, like a lot of doctors, has been playing around with ChatGPT to see if it could help save time on paperwork. And I've learned from you that some tech companies and startups are offering to build systems for this very thing, for medical paperwork.

BRUMFIEL: That's right. Yeah. It's really interesting. There are a number of ways these AI programs that can generate things like stories and poems could potentially help out with all the stacks of notes and things that doctors have to do. And that includes, you know, things like listening in on a conversation between a patient and a doctor and taking notes about what they're talking about.

KWONG: Is that OK? Is that ethical?

BRUMFIEL: I mean, they make very clear, the doctors, that they are using this system when they do.

KWONG: Yeah.

BRUMFIEL: But there are also real worries, too. You know, as we've talked about in the show, these models and AI programs can make mistakes. They can sometimes just make things up. And then there's this extra problem that you get when it comes to medicine, and that's the problem of bias.

KWONG: So today on the show - is AI about to revolutionize medicine?

BRUMFIEL: And what could it mean for both doctors and patients?

KWONG: You're listening to SHORT WAVE from NPR.


KWONG: So, Geoff, I guess the first question on my mind is why are doctors so interested in AI? Aside from it being kind of inherently cool, why are so many people latching on to it?

BRUMFIEL: Yeah, I called up Pearse Keane, a professor of artificial medical intelligence at University College London, and he really laid out why doctors are so interested.

PEARSE KEANE: The brutal truth is that we are drowning in the number of patients that we need to see and treat.

BRUMFIEL: So AI has been making quiet inroads in medicine for a couple of years now in fields like radiology and ophthalmology. In fact, Pearse is an ophthalmologist. And, you know, he pointed out to me one of the first algorithms was one that screens for diabetic retinopathy - basically, when blood vessels in the back of your eye become damaged from diabetes, and it can eventually lead to blindness.

KWONG: And how does AI help detect that?

BRUMFIEL: Yeah. So the way the AI works, as I understand it, is there are systems that can take an image of the retina, and the algorithms can make this determination whether the patient should be referred for further screening.

KWONG: Right. It's, like, a way to quickly sort for who might be having this condition and treat them accordingly.

BRUMFIEL: That's right. And so there are a number of algorithms now that can look at things like - you know, screen for retinopathy or look at scans and X-rays maybe for radiology, but they tend to be quite narrow. They can look at one type of scan. They can look for one kind of problem. What makes this new technology based on ChatGPT and other chatbots so exciting is it's actually much broader. It can be applied to many medical specialties, at least potentially. So that's where I caught up with another doctor named Dereck Paul. He's the founder of a company called Glass Health in San Francisco. When he first saw ChatGPT, he wasn't really all that impressed with it.

DERECK PAUL: I looked at it, and I thought, man, this is going to write some bad blog posts. Who cares? That was really my initial - I was like, who cares?

KWONG: I did the same thing. I downplayed it. I was like, oh, it's just another toy for the internet.

BRUMFIEL: Yeah, yeah.

KWONG: Whatever.

BRUMFIEL: He kept hearing about it from medical students who were younger than him, who were trying it out for answering questions and learning about things. And then eventually his customers started asking him about it, too. And he's sort of been converted. He says that, you know, ChatGPT could produce halfway decent answers. And now he's got this product called Glass AI. It looks kind of like a Google search bar, except where you'd normally enter what you were looking for, you just enter a quick description of your patient.

PAUL: We're working on doctors being able to put in that what we call, like, a one-liner, that patient summary, and for us to be able to generate the first draft of a clinical plan for that doctor - so what tests they would order and what treatments they would order.

BRUMFIEL: This is paperwork that doctors have to fill out all the time for patients. So you can imagine...

KWONG: Yeah.

BRUMFIEL: ...If doctors could just do a little copypasta and then put it in the hospital record system, that would cut down on a lot of time.

KWONG: But Geoff, we have done reporting on the potential for racial bias in AI systems - for all kinds of bias, really. And because of that, I'm kind of nervous about how we're continuing to hand over the clinical keys to a computer with these matters.

BRUMFIEL: Yeah, no, I think you're absolutely right. I mean, before we even get to racial biases, we've talked about the issues on this program before. You know, this new AI is designed to generate new things, but it doesn't always know whether it's right or not.

KWONG: Yeah.

BRUMFIEL: You know, these systems can hallucinate. They can fabricate. They can sometimes just mash together outdated information. Maybe you're giving the, you know, treatment for a heart attack from 1984 instead of from, you know, 2022. So there's all sorts of stuff that can happen when you have these generative systems. But just focusing for a minute on bias, that is a really serious problem. There's a lot of bias in our health care system already, and AI can make it even worse. So Marzyeh Ghassemi is a computer scientist studying AI and health care at MIT.

MARZYEH GHASSEMI: When you take state-of-the-art machine-learning methods and systems and then evaluate them on different patient groups, they do not perform equally.

KWONG: OK. Unpack this a little for me because I know, about AI, that these systems - they're trained on big data sets put together by humans. And whether that data is from the internet or a medical study, it contains, naturally, all the human biases that already exist in our society. Those biases just tend to show up in the data. Is that kind of what's happening with medical AI sometimes?

BRUMFIEL: That's exactly it. I mean, it's that simple. Basically, you know, if there's bias in the data - and there definitely is - then these systems will reflect those biases back onto the doctors using them.

KWONG: Yeah.

BRUMFIEL: So, for example, Marzyeh used an AI chatbot trained only on scientific papers and medical notes...


BRUMFIEL: ...To complete a sentence from a patient's medical record. And here's what it did.

GHASSEMI: When we said white or Caucasian patient was belligerent or violent, the model filled in the blank was, you know, patient was sent to hospital. If we said Black, African American or African patient was belligerent and violent, the model completed the note was - patient was sent to prison.

KWONG: I know that was just for a study, but that's pretty disturbing, Geoff.

BRUMFIEL: Yeah. I mean, this is a fairly extreme example, but, I mean...

KWONG: Yeah.

BRUMFIEL: ...It's also important to remember, like, this system wasn't trained on some chat board somewhere. This was trained on scientific texts only. So even using supposedly more impartial data, it still turned out this really biased result.

KWONG: Yeah.

BRUMFIEL: And Marzyeh says other studies have turned up similar things. She worries that these medical chatbots are basically going to turn into bias parrots that are just going to regurgitate bad decisions to doctors, and the doctors will just go along with it.

GHASSEMI: It has the sheen of objectivity. ChatGPT said that you shouldn't have this medication. It's not me, right? Like, a model, an algorithm made this choice.

KWONG: Yeah, I can understand her fear 'cause it shifts blame for bad decisions onto the computer, wrongfully so since people made computers.

BRUMFIEL: Right. And, I mean, there's a whole separate question of who is to blame if one of these computers makes mistakes, whether it's the doctor. Dereck Paul of Glass AI makes a real point of saying doctors need to be cautious when they use AI. They need to understand it could get things wrong.

KWONG: Yeah.

BRUMFIEL: And that's not something they're typically that used to.

PAUL: You have to supervise it the way we supervise medical students and residents, which means that you can't be lazy about it.

BRUMFIEL: You know, but I have to ask the question, if you give a busy doctor a powerful tool like this and it's right 99% of the time, how carefully will they be checking it, both for factuality and, you know, for what can be very subtle biases? It's not just going to jail or not. I mean, it's - it can be much more subtle stuff that really affects patient outcomes.

KWONG: So bottom line, Geoff, where do you think we're headed with this? And do the enormous net positives - to be clear, with efficiency, with being able to see patients who need to be seen - outweigh these also pretty serious negatives?

BRUMFIEL: Well, you know, I hate to say this 'cause it's what journalists always say, but time will tell, Emily. Time will tell. No, really...

KWONG: It's the truth.

BRUMFIEL: ...Look, it really comes down to your perspective. You know, Paul says that there are real problems out here that need to be solved. Doctors are overwhelmed with paperwork.

PAUL: We need these folks, not just in burnt-out states, trying to complete documentation. We need them thinking and working on all these different things. And then patients need more than 10 minutes with their doctors.

BRUMFIEL: And, you know, I mean, he's also working on ways to get around these problems. He obviously is aware of the problem of bias and, you know, of hallucination or making stuff up. And so they're developing this sort of medical - almost like a virtual textbook for Glass AI. The computer program is supposed to reference specifically only information in that textbook to try and keep it from fabricating or relying on sources that are somehow biased or otherwise unworthy. But Marzyeh is still pretty skeptical. She says that there are real underlying problems - right? - that this technology is supposedly kind of trying to work around. Hospitals don't have enough staff. Maybe insurance companies are requiring too much paperwork.

GHASSEMI: When people say technology is a solution for this, I often think, no, you made this problem. Technology will not fix it. Just fix the problem.

BRUMFIEL: I think, regardless of where you come down on this, the bottom line is it's happening, right? I mean, Glass Health has seen a lot of strong interest from doctors, and the tech is coming, you know, whether we're ready for it or not. I think we're going to find out just how good it is.

KWONG: Geoff, thank you so much for coming on to talk about this aspect of AI entering our lives and our medical systems as well.

BRUMFIEL: Yeah, it's been a pleasure. Thank you.

KWONG: This episode was produced by Berly McCoy. It was edited by our managing producer, Rebecca Ramirez, and fact-checked by Nicolette Khan. Robert Rodriguez was our audio engineer. Beth Donovan is our senior director and Anya Grundmann is our senior vice president of programming. I'm Emily Kwong. Thanks, as always, for listening to SHORT WAVE from NPR.

Copyright © 2023 NPR. All rights reserved. Visit our website terms of use and permissions pages at for further information.

NPR transcripts are created on a rush deadline by an NPR contractor. This text may not be in its final form and may be updated or revised in the future. Accuracy and availability may vary. The authoritative record of NPR’s programming is the audio record.