What's Changed Since Polls Were Wrong About 2016's Presidential Election Many people were surprised when Donald Trump won the presidency. NPR's Rachel Martin talks to database journalist Dhrumil Mehta and polling director Patrick Murray about how pollsters were wrong.
NPR logo

What's Changed Since Polls Were Wrong About 2016's Presidential Election

  • Download
  • <iframe src="https://www.npr.org/player/embed/928556186/928556187" width="100%" height="290" frameborder="0" scrolling="no" title="NPR embedded audio player">
  • Transcript
What's Changed Since Polls Were Wrong About 2016's Presidential Election

What's Changed Since Polls Were Wrong About 2016's Presidential Election

What's Changed Since Polls Were Wrong About 2016's Presidential Election

  • Download
  • <iframe src="https://www.npr.org/player/embed/928556186/928556187" width="100%" height="290" frameborder="0" scrolling="no" title="NPR embedded audio player">
  • Transcript

Many people were surprised when Donald Trump won the presidency. NPR's Rachel Martin talks to database journalist Dhrumil Mehta and polling director Patrick Murray about how pollsters were wrong.

RACHEL MARTIN, HOST:

The Biden and Trump campaigns have at least one thing in common - they're not sure you should trust the polling data. They don't want the numbers to keep potential voters home.

DHRUMIL MEHTA: A poll is not a crystal ball. It doesn't tell you who's going to win. No one can predict the future.

MARTIN: That's FiveThirtyEight database journalist Dhrumil Mehta. It's hard to forget how many polls were wrong in 2016. So what has changed since then? I spoke with Mehta about that, along with Patrick Murray, the polling director at Monmouth University.

PATRICK MURRAY: The thing that we didn't see coming was how the amount of volatility that we were measuring in the polls could actually translate on Election Day. And there were a couple of things that pollsters didn't do as well as they could have, such as understanding the impact of a college education on how people would vote in 2016, which was different than we saw before. But even with a correction for that error, we still would have had Hillary Clinton in the lead in all these polls, by a smaller amount but consistently.

But it was the volatility, something that we're not seeing this year that we saw then, where we had 20% or more of the electorate going into the final week who said they were undecided, were going to vote for a third party or were leaning towards one of two candidates but could easily change their mind or could simply not show up to vote at all. And all those things happened. That was a huge number. And there was enough of that shift in those key states where you were moving the needle about, you know, two or three points. And that was enough for Donald Trump to win those key states.

MARTIN: Dhrumil, the final forecast from FiveThirtyEight in the 2016 election gave Donald Trump, I think, a 29% chance of winning the Electoral College. And you got a lot of grief for that, especially from Clinton supporters who accused you of underplaying Trump's likelihood of winning. But those were actually far better odds than most other outlets gave him at that point, right?

MEHTA: Yeah, I think one of the big challenges that we face at FiveThirtyEight, as an aggregator of polls, is explaining to people uncertainty - right? - this idea that there is a percent chance that Trump will win. So, for example, a 3 in 10 chance is not an impossibility; it's a 3 in 10 chance. So if you were to buy a house, for example, and I was to tell you that the roof of this house might cave in - there's a 3 in 10 chance that it might cave in - you would probably think twice about buying that house.

MARTIN: Right. Because the three can happen (laughter).

MEHTA: Exactly. And with something so important as the roof of your house, you know, you'll take a 3 in 10 chance pretty seriously. Looking at FiveThirtyEight, it doesn't tell you this will definitely happen or this won't definitely happen; it gives you a chance of that event happening. And in this case, it was roughly 3 in 10.

MARTIN: So, Patrick, what changed in polling after 2016? Because there was a sort of autopsy about what pollsters did right and what they did wrong.

MURRAY: Yeah. I mean, the one thing I mentioned was the issue of education now becoming a factor in how people voted, when it wasn't before. And...

MARTIN: Explain what that means exactly. Education, college education - this wasn't something that was used as a defining criteria in the samples before?

MURRAY: Prior to 2016 - although we did see inklings of this in 2012 - white voters with a college degree voted in about the same way, Democrats versus Republicans, as white voters without a college degree. So we really didn't have to pay a lot of attention to what proportion of our sample had a college degree and which didn't.

But by 2016, that got blown out of the water - white voters without a college degree preferring Donald Trump by a lot and white voters with a college degree preferring Hillary Clinton. Because we weren't paying close enough attention to that, we had too many college-educated voters in our sample, enough to inflate Hillary Clinton's overall share of the vote by one or two points. But the other things that we have to pay attention to is what Dhrumil was talking about, was conveying uncertainty.

MARTIN: So, ideally, that meant that you should have been, in retrospect, putting an asterisk or something, some kind of footnote, to the polling that says this might not hold true tomorrow, that this is a snapshot in time and things are super erratic.

MURRAY: FiveThirtyEight, as you mentioned, had Hillary Clinton at 71%, but other aggregators had her at 90% to 95%. And there was just - nobody was talking about the uncertainty, and it was very hard to break through. One of the things that I'm actually doing this year - and I did it in 2018's midterms - is actually present a range of likely voter outcomes because we don't know exactly what that likely voter electorate is going to do. There's still enough room in these close states right now that the outcome could be different from where the polls say right now.

MARTIN: Dhrumil, on the FiveThirtyEight site right now, it says the following - quote, "President Trump needs a big polling error in his favor if he's going to win." So I don't get this. An error? I mean, don't the polls and related odds still show a path for President Trump to win? So why would it need to be an error?

MEHTA: On average, national polls will be off in one direction than the other, and state polls tend to be a little bit more off than national polls. But the state polls tend to be more consistently off in the direction of one candidate or another. So that's not to say that they always favor Democrats or they always favor Republicans more. It could be either way. But when there is a little bit of a polling error, it tends to be correlated across a bunch of different states.

And so I think what that sentence is trying to convey is that Trump has a much better chance of winning if the polls are off in his favor, if they are undercounting him this year in the same way that they did in 2016. However, it's worth noting that in 2012, the polls on average undercounted Obama's chances. So the polling error can go either way.

MARTIN: Patrick, do you have thoughts on that? And in addition, what are you looking for in the next week?

MURRAY: You have to step back, and you have to accept that polling can only get you so far. So right now we see Joe Biden with a very large lead in the national polls, and we're pretty confident that that means that he's going to win the national popular vote within about four points. But the state polls, they are close enough that you should sit back and prepare yourself for any eventuality from a very close Donald Trump win to the potential - there is a potential for a Joe Biden blowout. And it's anywhere in between. And you just got to accept that polling - what's polling telling you is that there is that range.

And, in fact, what you should focus on are all the other questions that we ask in the polls, other than the horse race, which is 98% of our poll content, which is - why are voters thinking this way? What's the impact of these different events that are happening? What are the issues that are top of mind? And, most importantly - and one of the things that we're tracking right now - is what is the potential for voters to accept the legitimacy of the outcome, regardless of what it is? That's something that we're trying to figure out to prepare ourselves for what might happen after November 3.

MARTIN: Patrick Murray, the director of Monmouth University's Polling Institute, and Dhrumil Mehta, a database journalist with FiveThirtyEight. Thanks to you both. We appreciate it.

MURRAY: My pleasure.

MEHTA: Thank you very much.

(SOUNDBITE OF THE BUDOS BAND'S "GUN METAL GREY")

Copyright © 2020 NPR. All rights reserved. Visit our website terms of use and permissions pages at www.npr.org for further information.

NPR transcripts are created on a rush deadline by Verb8tm, Inc., an NPR contractor, and produced using a proprietary transcription process developed with NPR. This text may not be in its final form and may be updated or revised in the future. Accuracy and availability may vary. The authoritative record of NPR’s programming is the audio record.