Episode 452: How Much Should We Trust Economics? : Planet Money This week, a grad student found a simple error in one of the most famous economics studies of the past several years. Should this change the way we think about economic research?
NPR logo

Listen

  • Download
  • <iframe src="https://www.npr.org/player/embed/177999020/178036141" width="100%" height="290" frameborder="0" scrolling="no" title="NPR embedded audio player">
  • Transcript
Episode 452: How Much Should We Trust Economics?

Listen

  • Download
  • <iframe src="https://www.npr.org/player/embed/177999020/178036141" width="100%" height="290" frameborder="0" scrolling="no" title="NPR embedded audio player">
  • Transcript

JACOB GOLDSTEIN, HOST:

Three years ago, Ken Rogoff and Carmen Reinhart published a paper that quickly became one of the most famous, most talked about economics papers since the financial crisis. It got so much attention because it answered a basic question everybody was asking - how much debt is too much?

CHANA JOFFE-WALT, HOST:

And this was a question that was being talked about constantly. The European debt crisis was in the news all the time. In the U.S., we were debating - as we still are - what to do about the deficit. Did we need more stimulus money? And these two professors, Ken Rogoff and Carmen Reinhart, offered a very specific way to think about government debt. They had a number, 90 percent. A debt-to-GDP ratio of 90 percent. They said this was an important line.

Countries with debt above that threshold, their economies grew much, much slower than countries that stayed below it, which was an interesting idea, that there is this threshold that matters a lot. And that idea was all over the news after the study came out. And we here at PLANET MONEY, we had Ken Rogoff on to talk about it. It was on news shows and newspapers. Lots of people cited it.

GOLDSTEIN: Policymakers started citing this, too. In the U.S., you heard Paul Ryan talking about this study when he argued for the U.S. to cut its deficit. In Europe, policymakers there cited it in arguing for austerity for cutting deficits in Europe.

JOFFE-WALT: And then a couple months ago, a 28-year-old graduate student in economics at UMass, Amherst, got what seemed like a totally routine assignment for one of his classes. He was supposed to pick a published study and replicate it - basically get all the data the authors got, do the math that they did and show how they got their results.

GOLDSTEIN: His name was Thomas Herndon, and he chose this Rogoff and Reinhart paper. He got the basic data on growth and debt, a lot of which the authors themselves had posted online, and he started crunching the numbers.

THOMAS HERNDON: It was a few weeks, maybe a month into the semester, and I run just the first basic thing of taking the average - and this is the basic statistics - and they didn't match at all. I was like, oh, no, I must have made some gross error. That doesn't look at all like what they cited.

GOLDSTEIN: Herndon went to his professors, and they were, like, come on, this is not even supposed to be the hard part, right? You're just taking an average. You're doing something wrong. Just figure it out. So Herndon goes back. He keeps checking everything, he keeps checking everything. And he keeps getting a really different answer than what Reinhart and Rogoff got. Eventually he emails Reinhart and Rogoff 'cause he wants to get the spreadsheet where they actually did their work, right? He wants to know if there's some kind of difference of opinion, or did Reinhart and Rogoff actually do the math wrong or something?

JOFFE-WALT: And Carmen Reinhart wrote him back and said, here you go. Here is an attached spreadsheet of our original math.

HERNDON: I was in the middle of writing a different paper, and I just immediately stopped, opened up the spreadsheet, and that's when I saw the error in the average. I almost didn't believe my eyes. I had to ask my girlfriend to look at it for me to see if I wasn't just, you know, just wrong.

GOLDSTEIN: He wasn't wrong. The authors of one of the most influential economics papers of recent years had made a simple Excel error. Herndon, with the help of his professors, published his findings this week. And as soon as they hit the Internet, it seemed like everybody in America who pays attention to economics was talking about what this 28-year-old grad student had discovered.

(SOUNDBITE OF ARCHIVED RECORDINGS)

UNIDENTIFIED MAN #1: All right. A hot topic in the world of economics. Carmen Reinhart and Kenneth Rogoff, their famous study claims countries see their debt-to-GDP ratio climb above 90 percent and growth then slows dramatically.

UNIDENTIFIED MAN #2: A team of researchers from the University of Massachusetts at Amherst has found basic errors in Reinhart and Rogoff's work.

UNIDENTIFIED WOMAN: What do you make of the Reinhart-Rogoff controversy?

UNIDENTIFIED MAN #3: Look; I haven't audited the now-famous spreadsheet, but others have, apparently.

UNIDENTIFIED MAN #4: So do we have a Harvard mistake here, and has that mistake led us down the wrong road to a Harvard austerity?

(SOUNDBITE OF SONG, "BRILLIANT MISTAKE")

ELVIS COSTELLO: (Singing) He thought he was the king of America, where they bought Coca-Cola just like vintage wine...

GOLDSTEIN: Hello, and welcome to PLANET MONEY. I'm Jacob Goldstein.

JOFFE-WALT: And I'm Chana Joffe-Walt. On the show today, we've got a famous economic study with a basic spreadsheet error, which raises some questions, like, just what are we supposed to make of that? How much should we trust the economic research that we look to and cite and talk about? And how much should we trust economists?

(SOUNDBITE OF SONG, "BRILLIANT MISTAKE")

COSTELLO: (Singing) It was a fine idea at the time. Now it's a brilliant mistake.

GOLDSTEIN: Before we get to those big questions, let's just spend a couple minutes talking about the details of what Thomas Herndon found. Besides that error in the spreadsheet, he also noticed that Reinhart and Rogoff left out some data and weighted the countries in a way that seemed weird to him. And Herndon says, if you don't do those things, you find that, yeah, average growth is slower in countries where debt is over 90 percent, but you do not see the big, scary drop-off that Reinhart and Rogoff found. The threshold just is not there. And that threshold, the idea that there was this line you hit at 90 percent, that was largely why the Reinhart and Rogoff paper was such a big deal in the first place.

JOFFE-WALT: We did reach out to Reinhart and Rogoff. They declined our request for an interview. But they put out a statement this week. And in the statement they defended the way they weighted the countries. They said they were constantly updating their database. And Thomas Herndon was able to use some data that wasn't ready when they wrote their paper back in 2010.

GOLDSTEIN: But Reinhart and Rogoff said they did make that simple error in Excel. They wrote, quote, "it is sobering that such an error slipped into one of our papers despite our best efforts to be consistently careful. We will redouble our efforts to avoid such errors in the future."

JOFFE-WALT: So the very first simple question we wanted to ask was just, how often does this happen? How often are economic studies just wrong?

JUSTIN WOLFERS: You find mistakes in many, many papers.

JOFFE-WALT: This is Justin Wolfers. He's an economist at the University of Michigan.

WOLFERS: Finding errors is not rare. And...

JOFFE-WALT: Finding errors three years later for a very important paper, that's not rare?

WOLFERS: I can tell you I've been through the - I've replicated every single paper in the death penalty literature, and I found errors in 90 percent of them anywhere between one month and 10 years later.

JOFFE-WALT: Wow, really?

GOLDSTEIN: Yeah, that seems like a big deal.

JOFFE-WALT: So, I mean, just in - in the death penalty case those were certainly papers that were written about probably by journalists, by policy people who would cite a study to bolster some claim that they were...

WOLFERS: They were written about by the Supreme Court and they were cited in congressional testimony.

GOLDSTEIN: Now, some of what Wolfers is calling errors here are questions about methodology, things reasonable people might disagree on. But in other cases, he says, people just screwed up the math.

WOLFERS: Yeah. In one case it changed an author's estimate from a claim that each execution deters 18 homicides to instead each execution causes another 18 homicides.

GOLDSTEIN: He got the sign wrong.

WOLFERS: Completely wrong.

JOFFE-WALT: Now, the authors of that study have responded to Wolfers. But essentially, what Justin Wolfers is telling us here is our question - should we be worried about economic studies? - yes. He says there may be other famous papers where people just got the math wrong. And even if they got the math right, that doesn't mean they got the right answer.

EMILY OSTER: It felt like [expletive].

JOFFE-WALT: This is Emily Oster. She's an economist at the Booth School at the University of Chicago. And in 2005, Emily Oster wrote a paper about why some parts of Asia have more boys than girls. And Oster was looking at medical research that people with hepatitis B were more likely to have boys, and the rates of Hepatitis B were higher in these countries in Asia.

OSTER: Maybe it's this hepatitis thing and, in fact, that more boys are being born. And, you know, I had a bunch of what I felt at the time was quite good evidence to support that looking across countries, looking at within some particular populations.

JOFFE-WALT: But then some researchers in Taiwan tried to repeat Oster's study, and they showed pretty clearly she was wrong. It wasn't that she had made some mistake in some Excel spreadsheet, just that they had much better data. And the data they had showed there was no link between hepatitis B and the ratio of boys to girls in these countries.

OSTER: I remember just at the time it just felt terrible because you put so much of yourself into this work and, like, you really want it - you really want it to be right. And you feel like you've worked so hard. And then, you know, they just - it just wasn't.

GOLDSTEIN: And, I mean, I understand why this must have sucked for you. But it is good for the world, right?

OSTER: Yes.

GOLDSTEIN: I mean, you put out this splashy paper advancing this interesting idea that seems plausible and it turns out it's wrong. And thank God that somebody...

OSTER: Yes.

GOLDSTEIN: ...Figured out it was wrong and that you had the whatever - courage and integrity - to say, oh, yeah, I was wrong.

OSTER: Yeah. I mean, I think it's right that, like, really it was very good that this happened. And I really do, like, genuinely think that. But there's just sort of the moment when somebody says - and I can sort of envision how this must have been for Rogoff and Reinhart when you just sort of look and you realize, yeah. Yeah. That does not seem so good for my theory and, like, ah (ph), really, you know?

JOFFE-WALT: Emily Oster says when she heard someone had replicated Rogoff and Reinhart's work and was challenging their findings she thought, this is a good sign for economics. A mistake has been caught. We now have a much fuller picture of how to think about their findings.

OSTER: I think that you should feel, like, pretty good about most of the stuff that we publish. And at least you should feel that, like, people are out there and trying to test it.

JOFFE-WALT: So when I say that when I read about this and I feel like, oh, man, like, economics is even less of a science than I thought it was, you have exactly the opposite reaction.

OSTER: Yes. Right. I think that's right. I'm sorry you have that reaction. I don't think that's the right reaction.

GOLDSTEIN: Well, you wouldn't, would you?

OSTER: No. I guess that's right. I devoted my life to this.

GOLDSTEIN: So this thing that happened to Emily Oster with her hepatitis B paper, it was awful for her at the time. But it had a happy ending. Economists were checking up on each other's work. Over time, they got better data. And that got them a better answer. It got them closer to the truth. This is how science is supposed to work.

JOFFE-WALT: But there are parts of economics that just do not work that way. And unfortunately, they're the parts of economics that often try to address some of the biggest, most pressing questions like, how much should a government spend or how high or how low should taxes be? Macroeconomists tend to look at these questions. And they have a hard time finding clear answers because macroeconomist often have a data problem.

GOLDSTEIN: It's not like macroeconomists can run scientific experiments. If you want to understand, say, inflation, you can't randomly assign half of Americans to live in a world with low inflation and the other half to live in an identical world except with high inflation.

JOFFE-WALT: And macroeconomists are acutely aware of this problem. When Mark Thoma, University of Oregon economist, was in grad school in the 1980s, he thought a lot about how limiting it was to his work that he only had 30 years of data. There were only 30 years of consistent data to study. But then he had this experience that changed the way he thought about that problem.

MARK THOMA: So I was at a conference in - shortly after I had gotten out of graduate school. I was a young assistant professor. And Milton Friedman gave a paper that he had written 25 years before.

JOFFE-WALT: Milton Friedman, famous economist.

THOMA: Yes, Nobel Prize-winning economist Milton Friedman from the University of Chicago. And he had written a paper 25 years earlier, waited all of that time for new data to accumulate, and then used that new data to test his model. And he found out that it did very well.

JOFFE-WALT: This was exciting because it made Thoma think this problem that macroeconomist always have about data, maybe it won't be such a big problem in the future the older I get.

THOMA: Oh, I thought it was great. It made me think of two things. Twenty-five years from now I'm going have twice as much data as I have now, and I can go back and test a bunch of these papers that are important now. And so I was trying to think, you know, 25 years from now what should I test? What will still be important? And go back and look at these really important papers and see if they hold up after we have data we didn't know about when the models were built.

GOLDSTEIN: And given that - so you're writing these papers at that time based on 30 years of data. And you're thinking essentially 25, 30 years from now I'll have twice as much data.

THOMA: Yeah. Yeah. It'll be great. We'll have twice as much precision. One of our big problems, lack of data, will be resolved, you know? And my kids and their kids, boy, they'll be much, much better for them.

GOLDSTEIN: And things are just going to get better and better over time. Economics is going to get closer and closer to the truth.

THOMA: Yes. Theory will get better. We'll have way more data. It slowly but surely will converge on the truth about the economy. I was very optimistic as a young economist.

GOLDSTEIN: Mark Thoma, he studies monetary policy, which basically means he studies what the Fed does and how that affects the world. And just as he was getting out of grad school, the Fed made some really fundamental changes in how it operated. Those changes meant that that 30 years of data Thoma already had couldn't really be compared to all the data that came after it. And other things in the economy were changing in really big ways, too. All that meant that Thoma was not going to get more and more data as time passed. Macroeconomics was not necessarily going to get closer and closer to finding clear answers to these big questions.

JOFFE-WALT: You had to start all over again.

THOMA: Essentially, yeah. The earlier data you just have to throw away.

JOFFE-WALT: So did that make you feel like this discipline that I'm in, that I've chosen for my career is less of a science than I thought it was or than I thought it could be?

THOMA: I was one of these gung-ho young people - I mean, they come out even now - who just thought that economics was going to be able to find the truth and find the answers and maybe even in my lifetime. Boy, that was exciting.

JOFFE-WALT: What do you think now?

THOMA: I think it's going to be a struggle.

GOLDSTEIN: So just to recap, how much should we believe economic research? Well, some of it. And we actually do not know how much has some basic math errors, basic spreadsheet errors in it. And then we know the data available to people trying to answer big macroeconomic questions about government policy will always be limited. It'll always be imperfect. And then Mark Thoma told us he's not even sure macroeconomists are getting closer to the truth.

JOFFE-WALT: You know, a lot of economists we talked to this week heard the latest news about the Reinhart-Rogoff study and it being questioned, and it seems like it sort of plunged many of them into what for economists at least sort of passes as existential doubt about the discipline, about the research that it produces.

GOLDSTEIN: Tyler Cowen, this economist who - we read his blog, he wrote this question that we loved. He said, does this mean I should read fewer empirical economic studies and just read more narrative history?

JOFFE-WALT: Right. That might be better. I might get more knowledge that way. But Justin Wolfers who we talked to earlier, he did offer one defense of economics. And it was not that economists always get the math right or that they've solved the data problem or can really clearly prove things. It was a much simpler argument.

WOLFERS: Right. The best difference of economics is not that we're good. It's that everything else is terrible.

GOLDSTEIN: Economists think about really hard questions, and they're questions that policymakers have to answer one way or another.

WOLFERS: You can have a lot of criticisms about how we do empirical economics and we're still in almost every case better than the alternative, which is listening to politicians make it up as they go or ideologues spouting nonsense.

GOLDSTEIN: Wolfers is saying essentially the best thing you can hope for is that economists will keep arguing over these questions. And that's basically what was happening this week with all the drama over Reinhart and Rogoff's paper.

(SOUNDBITE OF SONG, "BRILLIANT MISTAKE")

COSTELLO: (Singing) He thought he was the king of America.

GOLDSTEIN: And finally, that Rogoff and Reinhart paper, it was presented at this very prestigious conference, but it actually was not published in a peer-reviewed journal.

JOFFE-WALT: So it was never peer-reviewed, and we will never know if peer reviewers would have actually caught the math mistakes. Anyway...

GOLDSTEIN: As always, we want to hear from you. You can email us at planetmoney@npr.org.

JOFFE-WALT: You can find us at npr.org/money on the Internet. I'm Chana Joffe-Walt.

GOLDSTEIN: And I'm Jacob Goldstein. Thanks for listening.

(SOUNDBITE OF SONG, "BRILLIANT MISTAKE")

COSTELLO: (Singing) It was a fine idea at the time. Now it's a brilliant mistake.

Copyright © 2013 NPR. All rights reserved. Visit our website terms of use and permissions pages at www.npr.org for further information.

NPR transcripts are created on a rush deadline by Verb8tm, Inc., an NPR contractor, and produced using a proprietary transcription process developed with NPR. This text may not be in its final form and may be updated or revised in the future. Accuracy and availability may vary. The authoritative record of NPR’s programming is the audio record.