If Science Takes A Wrong Turn, Who Rights It?
IRA FLATOW, host: This is SCIENCE FRIDAY. I'm Ira Flatow. The physicist Max Planck once quipped that science advances one funeral at a time. And by that he meant that old theories die hard, only when their supporters die off. Think for example how many funerals it took for Copernicus's idea about the Earth orbiting the sun to catch on.
Of course, scientific debates didn't stop there. New contradictory research surfaces all the time. And then others come out saying no, the original theory was the correct one. And then some studies in the top journals are retracted, as happened just a few weeks ago with a study in Science that claimed to have found the genetic signature of longevity, only to be retracted due to technical errors with a flawed chip.
With all that flip-flopping, how do you know when you can actually trust the science, not to mention cases of scientific misconduct, fraud, when researchers fudge the numbers or even invent patients. If they get caught, their papers are retracted from journals ideally, but then what? Many times that bad science is never really truly forgotten. Scientists continue to cite those studies, and busy physicians continue to rely on them, perhaps putting patients at risk even after those studies have been debunked.
So up next, the scientific process. Is science really as good at correcting its mistakes as we'd like it to be? Our number, 1-800-989-8255. You can tweet us, @scifri, S-C-I-F-R-I. What do you think? You can also join a discussion on our website at sciencefriday.com. Leave your opinion.
Dr. Ivan Oransky is an executive editor at Reuters Health here in New York, and he writes for the blog Retraction Watch. He's here in our New York studios. Welcome to SCIENCE FRIDAY, Dr. Oransky.
Dr. IVAN ORANSKY: Thanks very much, good to be here, Ira.
FLATOW: You're welcome. Dr. R. Grant Steen is president of Medical Communications Consultants. He's also a former professor of psychiatry at the University of North Carolina in Chapel Hill. He joins us from WUNC. Welcome to SCIENCE FRIDAY, Dr. Steen.
Dr. R. GRANT STEEN: Thank you for having me, Ira.
FLATOW: Ivan, scientific opinion seems to flip-flop a lot and must confuse a lot of we laypeople. How do you know when you can trust the science?
ORANSKY: I think that's an excellent question. You want to look at some things. Was this published in a journal is the first thing to look at, and we'll get to whether or not you can trust anything in a journal, which I think your intro kind of hinted at in a good way.
FLATOW: But that's got to be the minority parts, right? Most things in journals are reliable.
ORANSKY: I think that's fair to say. In fact, I'd even go as far as to say the vast majority of things in journals are reliable. Whether or not they represent, if you want to be grandiose about this, and we were quoting Planck, so why not, whether they represents truth with a capital T I think is more about what science does and what science represents than about whether they're true at the time they were published.
What I mean by that is that science should be - it is, and it should be - an ever-evolving process.
FLATOW: Self-correcting. Grant, Dr. Steen, do you believe that it is self-correcting?
STEEN: I do with reservations. I think it corrects slower than we would like it to and that there are some things that are sort of accepted in the canon of knowledge that are probably not true but no longer relevant either.
FLATOW: Such as?
STEEN: Well, for example, if you have a treatment for hepatitis C that you claim works, and someone else comes out with a vaccine for hepatitis C, it doesn't really matter whether your treatment works. It's not relevant anymore. So the field has moved on.
But I wonder whether there are papers that should be retracted that haven't been left behind like landmines in an abandoned field.
FLATOW: You mean for doctors to rely on.
STEEN: Well, if the field is no longer relevant, I guess doctors aren't relying on it anymore, but you do wish that the literature had a purity that it probably doesn't have.
ORANSKY: I would agree. I think it's an interesting question whether or not say a body of work on a treatment, a medication for hepatitis C, just using Grant's example, should that be retracted once a vaccine that we have sort of demonstrated is effective comes out and is proven.
On the other hand, you know, maybe that's a question about just again the progress of science. I guess the one thing that we've been looking at, at Retraction Watch, Adam Marcus and I, he's my co-blogger there, should something that can't be reproduced be retracted?
In other words, if another group of scientists try and do something, according to what it says was the protocol in that old paper, and they can't do it, or they do it, and they get a different result a number of times, there's actually some debate over whether that should be retracted.
So it's - I think that's analogous in terms of the clinical science that has moved on from what you know. It doesn't mean something was untrue. It just means you can't really rely on it anymore.
FLATOW: Is it - I mean, you are involved in this website called Retraction Watch. It would seem to me that - and I know the website - there are so many journals on there that so many things need to be retracted that you have discovered.
ORANSKY: We have - Adam and I have, covered, we estimate about 200. We just celebrated our first anniversary on Wednesday.
FLATOW: Mazel tov.
ORANSKY: Thank you.
(SOUNDBITE OF LAUGHTER)
ORANSKY: I think if the first anniversary is paper, which I'm not sure it is, then I guess it's a retracted paper in our case. We've covered something like 200 retractions, but 90 of them came from one person.
ORANSKY: Ninety. Whether it's 89 or 91, I should be careful, but it's about...
FLATOW: You might have to retract that later.
ORANSKY: I might, I might, and I'd be happy to, of course. I want to correct the record. That's - he was an anesthesiologist, and we can talk about that case if that's interesting. But I do think there's a tremendous - the number of papers has jumped a great deal.
There are older journals that are publishing more and more online. Some of that gets to print, or it gets into print later on, and there are just far more journals.
If you look at the proliferation of journals online, some of which are open-access journals, you know, that's a subject you've covered here, you just - there's a lot more journals.
FLATOW: And how to know whether to trust them, the proliferation of online...
ORANSKY: Exactly. It's not as though there are a sort of - an appropriate, more, larger number, proportionally, of good peer reviewers to necessarily look at them. You have the same number of people essentially doing more peer review. I think we all make more mistakes when we do more work.
FLATOW: 1-800-989-8255 is our number. Dr. Steen, are there any instances where bad science has actually put patients at risk?
STEEN: Oh, I think so. I definitely think so. But before we move on, I just wanted to point out that Science and Nature, the two highest profile journals in all of science, have more retractions than virtually any other journal. So relying on the journal, the quality of the journal to tell you what's good and what's not, can be fraught with problems.
FLATOW: They will print a word retraction on it, right? They'll leave the article there.
STEEN: Yes, yes.
FLATOW: A lot of times, you don't see the word retraction, Ivan, do you on some of these retracted papers?
ORANSKY: No, in at least a third, you basically - again, it depends on when you look at it. If you look at it the moment it was quote-unquote "retracted" and it's not there, I don't know if that counts. But if you look a certain amount of time later - and granted I think you've actually, this is part of one of your studies, you found that about a third of papers just simply weren't noted.
ORANSKY: You could get to a PDF of the paper without knowing that that was retracted. And if I may go right back to what Grant was saying, I think that's an excellent point about Nature and Science, again the world's sort of leading journals, certainly in the basic sciences.
Proceedings of the National Academy of Sciences has also a very high number, in fact I think higher than either of them but as per capita, in other words, per paper, I don't know what the appropriate, you know, Greek or Latin is there, it's not necessarily higher.
STEEN: Yeah, both Science and Nature publish a lot more papers than most other journals do.
FLATOW: Grant, I was - I interrupted you. I interrupted my own question when I asked you have patients actually been put at risk.
STEEN: Yeah, I asked the question whether patients had been put at risk in any papers that had been retracted between about 2000 and 2010. And I specifically looked at those, each of the 180 clinical papers that had been retracted and tallied up the number of people enrolled and patients treated. And the numbers were rather large.
But then if you look at how many patients were enrolled in studies that were influenced by those retracted papers, the numbers are astronomical. So to complete the circle, the number of patients enrolled in a retracted study was over 28,000. So this is 28,000 people spread among 180 different retracted studies.
And of those 28,000, something over 9,000 actually received treatment of some sort. Now if you look at the secondary papers, the papers that were influenced by those retracted papers, over 400,000 patients were enrolled, and then roughly about 70,000 were treated. Those are gigantic numbers.
FLATOW: And did the patients receive bad treatments as a result?
STEEN: Well, in some cases yes, and in some cases no, and in some cases the treatment may have been quite trivial. It's hard to, after the fact, go back and figure out how much risk a patient actually was in. In most cases, probably not very much because the institutional review board at the institution where the patients were treated should be looking out for the welfare of the patients.
So you wouldn't want to ever have patients enrolled in the study where they were really at risk. On the other hand, you wouldn't want to have any patients at all enrolled in a study that's built on false premises.
ORANSKY: You know, what Grant is saying is absolutely right, and I would just sort of focus on what he just said about institutional review boards, IRBs, which again are overworked. Grant used the word should, and I think he's being careful there.
They should have looked at the protocols to see whether or not the data were, you know, reasonable, looked at things throughout the process, in fact, the Data Safety Monitoring Board. But if you look at a couple recent cases, this situation I mentioned with 90 retractions, that was a case where the anesthesiologist named Joachim Boldt, in Germany, had not even obtained IRB approval. And in fact, the journals are retracting - and in a very proactive way that Adam and I think we would completely applaud. -are retracting because of that.
ORANSKY: If you look at Neal Pody(ph), OK, the - not far from you, Grant, down at - he was formerly at Duke. There's a lot of investigations continuing so we should be careful here. But certainly, this was an oncologist who - no one can reproduce his results. There's someone named Keith Baggerly at M.D. Anderson, who's done a terrific job of trying to reproduce those and looking at what's wrong and actually was the one who prompted a lot of the retractions. Those wouldn't necessarily have shown up in Grant's work yet, because they have been retracted in the last 12 months. So...
STEEN: Yeah. It didn't show up.
ORANSKY: Exactly. And so there were patients and trials that were, at the very least, being subjected to - they were being stratified. In other words, they were deciding - doctors were deciding what they got in the trial based on these protocols, based on those data that turned out to be, at best, mistaken.
FLATOW: Couldn't they - wouldn't they be maybe thoughtful about being sued or something, if they would...
ORANSKY: I think and I - certainly, I'm not speaking for Duke here, and Grant you may have more insight here, but Duke is, I think, quite aware that they might get sued and the FDA - I can tell you that the Institute of Medicine - excuse me - the National Cancer Institute was really not happy with the way all this has been handled. I - there are actually some - I know that there's a firm that has been sort of approaching patients. Whether there's an actual lawsuit, I don't know.
FLATOW: Well, if doctors are fraudulent, wouldn't it be fair to say that scientists or doctors, they're just like the general population. There are frauds everywhere. Everybody has their own, right?
ORANSKY: That's what Adam and I think.
STEEN: Yeah. I agree.
FLATOW: I mean, they're not any different than, you know, your car repairman or your plumber or somebody who puts the wrong stuff on your bill or...
ORANSKY: I think like many professions, they would like us to think that they are. I trained as a physician. And without, you know, sort of criticizing my med school classmates, we all had a little bit of a God complex. I think that happens in science, too. It happens any time you have a lot of training, a lot of knowledge. It's understandable. But professionalism requires that you're constantly testing whether or not what you believe is true. And that's among a number of things it requires, and the ability to self-police. And that's what, you know, Adam and I are really looking at Retraction Watch, whether or not scientists and journals are willing to actually say when they're wrong.
FLATOW: Grant, do you agree - that they're out there?
STEEN: Oh, I do agree. When you're working in science, you're working in the dark. You're trying to feel yourself along through a very complex landscape, and it's pitch dark. And there's a very good chance that you're going to make a mistake and fall down a hole. There's a very good chance that you're going to grab an elephant's tail and think that it's a snake. It's a complex environment, and we don't understand very much of it.
FLATOW: 1-800-989-8255 is our number. We're talking about science and fraud this hour on SCIENCE FRIDAY from NPR.
STEEN: One thing that I wanted to chip in is the idea of if science is true it's self-correcting, then why would a scientist engage in fraud? Well, it may be as simple as the fact that they think they can get away with it. Bank robbers are caught more than half the time. So why would a bank robber rob a bank? Well, that's because that's where the money is. And fraud and science can lead to, really, advancement in career and success and money.
FLATOW: But if there's more - is it different this, you know, in this era that we live in?
ORANSKY: I think...
STEEN: I think it is.
ORANSKY: Oh, yeah.
STEEN: I agree. It's so much more competitive to get a grant these days. Med schools have sort of relied upon the federal government to finance a large part of their running costs by grants. And now that grant rate - grant success rate has gone down so dramatically, it went from 20 percent a decade ago to less than 10 percent, maybe five percent at some institutes now. It's just so much more competitive, and these people, you know, they have careers. They have families. They have mortgages that they're tying to pay. And so many people are put on in an untenable situation.
ORANSKY: I think the other thing that's changed, though, not to be Pollyannaish here, but is that there are more eyeballs on all of this work. You see a lot more whether - again, even if they're not open access journals, there are simply more people looking at studies online et cetera. If Science and Nature are the ones that Grant cited before, part of the reason they have more retractions is because those papers tend to be looked at more. It's - they're high-profile journals. They tend to be cited more.
And so if you look at some recent cases, I think that the longevity study that you mentioned in the beginning of the segment that was retracted in Science just a couple of weeks ago, that's a good example. There were just more eyeballs on that. There was actually a reporter, Mary Carmichael, at Newsweek who did a great job, you know, looking into that and talking to people. Arsenic bacteria, there's some questions about that paper from last year in Science because Carl Zimmer had interviewed so many people who have questioned it. And whether or not science is ready for that, and I mean, that with a lower case and capital S, by the way, the journal as well science itself, whether or not it's ready for that kind of scrutiny, that many eyeballs, remains to be seen.
FLATOW: What about the journals themselves? I mean, they're competing with each other...
FLATOW: ...maybe not be more open to flashier research.
STEEN: Yeah. An arsenic-based life form is a very, very splashy paper. So it's not surprising that it got accepted. It's not surprising that it got played up in the media. It's perhaps not surprising that it turns out maybe that it's not true.
FLATOW: So they deserve some of the blame here.
STEEN: Oh, I agree.
ORANSKY: Well, what we've seen at Retraction Watch is that there so - there are so many variations in - or I should say there's so much variation between how different journals approach retractions and how transparent they are. There are journals that will literally only say this journal - this paper was retracted. It was withdrawn because the author asked us to. And there are ones that will say several paragraphs. Hey, this one went wrong. We think the latter is a far better approach.
FLATOW: Let me see if I can get a quick call in here before we go to the break. John in Crestview, Florida. Hi, John.
JOHN: Yes. I'm related to Dr. Crawford Long in Atlanta, Georgia. And prior to the Civil War, he had done work using ether as an anesthesia. And another doctor that had studied under him took the information that he had been studying for a long time and submitted a paper. I think it was The New England Journal of Medicine and claimed credit. Dr. Long had to submit all his evidence...
FLATOW: Please (unintelligible) question.
JOHN: ...and I'm just wondering if retracting doctors that take credit for work that they didn't actually do.
FLATOW: From other people, Ivan?
ORANSKY: Oh, that's happened a lot. Not necessarily in that sort of historical sense where someone else actually did something and you're taking credit. Well, what we've noticed a fair amount of is authorship where someone is either taking honorary credit, in other words, honorary authorship, which is, I think, a problem a lot of journals care about, but also forged authorship, where people put other people's names on papers.
FLATOW: We'll be back after this break, talking with Ivan Oransky and R. Grant Steen. Our number, 1-800-989-8255. You can tweet us @scifri. Science and research and fraudulent paper. Stay with us. We'll be right back after this break.
(SOUNDBITE OF MUSIC)
FLATOW: ..TEXT: You're listening to SCIENCE FRIDAY. I'm Ira Flatow. We're talking this hour about scientific process and how science corrects itself with my guests Dr. Ivan Oransky is executive editor at Reuters Health here in New York. He writes for the blog Retraction Watch. R. Grant Steen is president of Medical Communications Consultants. He's also a former professor of psychiatry at the University of North Carolina, Chapel Hill. Our number, 1-800-989-8255. Grant, you've written that in your mind just about every scientific paper contains misinformation. What do you mean by that?
STEEN: Well, I think - I stand by that. I know that looking back on my own papers there are things that I wish I had done differently. But I would say that most of those errors are quite trivial, like a reference number that is supposed to lead to one citation actually leads to another, something like that, a typo that you didn't catch in the galleys. I think that there are very, very few papers that are completely free of overt error. And then, if you throw in the mix whether or not those papers stand the test of time, I think nearly no paper is perfect. I'll be daring and say no paper is perfect.
FLATOW: Well, that's why is just a snapshot, one theory at a time, yeah.
FLATOW: Let's go to the phones. Michael in Portland. Hi, Michael.
MICHAEL: Hi there. My question is it doesn't seem to be that there are many disincentives to scientists, aside from - perhaps their reputation suffering or potential suits that they might face or the organization. If there have been any desire to look at whether or not their credentials might be removed.
ORANSKY: You know...
MICHAEL: When - I'm talking about a scientist - I'm sorry - a scientist who intentionally misrepresent data.
FLATOW: Good question. Thanks for calling.
ORANSKY: Yeah. That's a good question. We've seen some cases at the junior level, in other words, people who seemed to have published papers based on what turns out to be a fraudulent Ph.D. thesis. They'll lose their Ph.D. We had a case that we were following pretty closely in London of someone who had gotten - basically kicked out of Cambridge for his graduate school and then got kicked out of a post-doc as well, all for fraudulent stuff. Nobody knew about it. They weren't talking et cetera.
He has - well, he's left his institution. We don't know exactly - we know the circumstances of that. They won't say. But he did get kicked out of Cambridge for the fraudulent stuff. Whether senior scientists do, I think there's obviously a number of cases that we can point to where people's careers have been affected, whether they lose their credentials...
FLATOW: Don't they put in jail or anything?
ORANSKY: Well, (unintelligible).
STEEN: People have been put in jail.
ORANSKY: People have been. Scott Reuben who's - a case that Grant is certainly familiar with. I think he's out now, by the way, Grant?
ORANSKY: But he's certainly served time very briefly. He's someone who made up enough data for about 20 studies. I mean, maybe it's 22. Adam actually - again, my co-blogger at Retraction Watch - broke that story at Anesthesiology News. This was before we had started Retraction Watch. Scott committed fraud. He's an, again, anesthesiologist and served time because the circumstances were so great.
FLATOW: I hear some questions, some tweets coming in from - Dan Mclane(ph) writes I work with autistic kids, and nearly 75 percent of the parents continue to believe vaccinations are to blame. Mis-info dies slowly.
STEEN: Yes. I definitely agree with that. The Wakefield paper is kind of an embarrassment to the whole field. It came out in 1999 and was only retracted this year.
STEEN: So it was out there for 12 years.
STEEN: I'm sorry.
ORANSKY: Yeah. It was actually '98 and then retracted in 2010, but same distance between them.
STEEN: Yeah. So it was out there for 12 years, and that's a long time. And it led to vaccine rejection by a lot of parents. And if you look at the rate of infection with mumps and measles, it's spiked in both the United States and in England.
FLATOW: Let's go to Nancy...
STEEN: So that's...
FLATOW: I'm sorry.
STEEN: No, I was going to say that there are probably children who died of mumps or measles infections that could have been prevented by vaccination.
FLATOW: Yeah. We've talked about that a lot here. One last quick question from Nancy in North Kansas. Hi, Nancy.
NANCY: Hi. Thank you for taking my call. I still GAO report not too long ago that was titled "IRB Review is Subject to Unethical Manipulation." So they concluded that IRB review is not reliable of protecting human subjects because they did a sting operation where they looked at - they organized a fake IRB, and they got it registered with the Department of Health and Human Services. Companies called them up to ask for their services to review research.
They also did another sting operation where they had their own fake protocol, where they hired out three IRBs, and it was an out-of-textbook unethical (unintelligible).
FLATOW: Nancy (unintelligible).
NANCY: ...the IRBs approved it. So, anyway, it's not reliable.
FLATOW: Thanks for the call. Let me get a reaction from Ivan on that one.
ORANSKY: Yeah. That actually it's something...
FLATOW: What's an IRB?
ORANSKY: An IRB is an institutional review board. So that's sort of supposed to be - it's not supposed to be rubber stamped. It's supposed to be what's between you, if you want it to be in a trial, and in other words, the protocol, whether the protocol was kosher, so to speak. And IRBs are under a lot of pressure.
In fact, what I think Nancy is referring is the FDA did a sting of, quote, unquote, "for-profit IRBs." Most of the time these IRBs are within institutions, so NYU, my alma mater, Columbia will have an IRB, again, overworked. You want to get your stuff through the pipeline, you're going to go to a for-profit IRB. It's a relatively new development. So they were looking at that, and they did find - they created this sort of fake protocol and fake company and they did find some serious gaping holes.
FLATOW: Well, gentlemen, we've run out of time. I want to thank you both for taking time to be with us.
ORANSKY: Thanks very much. It was great.
STEEN: Thank you.
FLATOW: R. Grant Steen is president of Medical Communications Consultants, also a former professor of psychiatry at the University of North Carolina, Chapel Hill. Dr. Ivan Oransky is the executive director at Reuters Health here in New York, and he writes for the blog Retraction Watch. Interesting website. You should all go over and take a look. Thank you, gentlemen.
ORANSKY: Thank you.
NPR transcripts are created on a rush deadline by a contractor for NPR, and accuracy and availability may vary. This text may not be in its final form and may be updated or revised in the future. Please be aware that the authoritative record of NPR's programming is the audio.