Everything You Need To Know About Polls Not all polls are created equal. The team breaks down what makes a good poll, how much they should be relied on and what they're saying right now about the 2020 election. This episode: Congressional correspondent Susan Davis, political reporter Danielle Kurtzleben and political editor Domenico Montanaro. Email the show at nprpolitics@npr.org. Find and support your local public radio station at npr.org/stations.
NPR logo

Everything You Need To Know About Polls

  • Download
  • <iframe src="https://www.npr.org/player/embed/722819724/722930317" width="100%" height="290" frameborder="0" scrolling="no" title="NPR embedded audio player">
  • Transcript
Everything You Need To Know About Polls

SARAH: Hi, this is Sarah (ph) calling from the bottom of the Grand Canyon, where I just backpacked nine miles and 5,000 vertical feet. This podcast was recorded at...

SUSAN DAVIS, HOST:

12:40 p.m., on Monday, May 13.

SARAH: Things may have changed by the time you hear this. Keep up with all of NPR's political coverage on npr.org, on the NPR One app and on your local public radio station. All right, here's the show.

(SOUNDBITE OF THE BIGTOP ORCHESTRA'S "TEETER BOARD: FOLIES BERGERE (MARCH AND TWO-STEP)")

DAVIS: I'm really impressed that she climbed to the bottom of the Grand Canyon. That's a huge achievement.

DOMENICO MONTANARO, BYLINE: It's a lot, yeah.

DAVIS: Congrats to you, Sarah.

DANIELLE KURTZLEBEN, BYLINE: Yeah.

DAVIS: Hey there. It's the NPR POLITICS PODCAST. I'm Susan Davis. I cover Congress.

KURTZLEBEN: I'm Danielle Kurtzleben, political reporter.

MONTANARO: And I'm Domenico Montanaro, political editor.

DAVIS: Today we're going to talk all about polls - what makes a good poll, how much should we be relying on what the polls are telling us, and what are the polls telling us right now about 2020. One reason why I think it's a good idea to have this conversation as we go into 2020 is, remember, coming out of 2016, there was this sense that the polls got it all wrong, and did we overread the polls? So I think it's worth starting with just saying, how are we smartly reading the polls in 2020? And Danielle, as someone who is both out on the trail and very much into the data, what do you look for when we get news that a new poll is out?

KURTZLEBEN: I'd say, first things first, I look at how the poll was conducted. So was it done by live callers, people actually calling people up on their phones, for example, as opposed to robo-polling, when you get a call on your phone and it's like, press one for so-and-so, and you get the robotic voice and all? There is some dispute out there over whether online polls are good or bad. I mean, if they are scientifically conducted, they aren't necessarily a bad thing; they can be good. But live callers are - sort of remain the gold-ish standard as far as polling.

A reasonably large sample size - I mean, a sort of shorthand you can look for with that is check the margin of errors, see how big the margin of error is. You want a margin of error that is not absolutely gaping because, then, the results are much more meaningless. One other thing I look at - and this is not a measure of goodness or badness of poll - but it's, look at the audience the poll that is measuring. So for example, is it looking at all adults? Is it looking at registered voters? Is it looking at likely voters? The more that it drills down into that, it's still not going to be perfect, but at least you're getting closer to the sample of people that you really want to poll and get their opinions.

MONTANARO: You know, not all polls are obviously created equal. We should take them with a grain of salt. But first of all, I wanted to backup for a second and say that the polls were not wrong in 2016.

DAVIS: Right, right.

KURTZLEBEN: Yeah.

DAVIS: Let's say that off the bat.

MONTANARO: You know, for as much as that narrative and that myth got created, when the pollsters went back and looked it over, there were some state polls that did have some problems, obviously. National polls were much better. And I think the state versus national polling is a real issue here because when we talk about primaries, what really matters are those state polls.

DAVIS: Yeah.

MONTANARO: And if you're having fewer and fewer polls that are conducted well, conducted rigorously, by good polling organizations, it does present a problem for those of us who are trying to analyze where the field is.

DAVIS: You know, I love when the new polls come out. I love poll data. I like them as part of our campaign coverage. But I always think you have to take, like, a deep breath every time a poll comes out. I mean, these are just literal snapshots in time of what is still now an event 18 months from now, right?

KURTZLEBEN: Totally correct.

DAVIS: Like, what the polls are telling us today could be different the next day, could be different the next day. So putting too much stock in any one poll...

MONTANARO: It's really a bad idea. And...

DAVIS: It's just not - you have to remember that these are very ephemeral things.

KURTZLEBEN: Well, yeah.

MONTANARO: You got to take a step back from them. And I think, month to month, seeing what the trend lines are and not putting a lot of stock, especially in outliers, in a poll that shows your candidate doing particularly well or not particularly well, you want to look at sort of where the middle is and how things move altogether in aggregate.

DAVIS: The other thing I'd say about polling, too, and when we talk about which polls are better or worse - good polling is really expensive.

MONTANARO: Really.

KURTZLEBEN: Yes.

DAVIS: And that's not something I'm sure that people who - you know, you just see a new poll out - I'm not sure people understand what it takes. And that's why a lot of news organizations partner with other news organizations. That's why TV networks, like The Wall Street Journal and NBC, partner for polls. A lot of that is, in part, to pay for them because it's so expensive to do these live caller polls.

KURTZLEBEN: Yeah.

DAVIS: So I think I always look to polls like that; these big news organizations that spend a lot of money into their polling, that hire professional pollsters and really get the data as best as possible.

KURTZLEBEN: Right.

DAVIS: Versus a robocall, versus a quick and dirty poll.

KURTZLEBEN: Right. And one thing I do want to add that I neglected to mention earlier - cellphones. You want to see if the poll...

DAVIS: Yeah.

KURTZLEBEN: ...Called just landlines or if it called cellphone because more and more - we all know this - people do not have landlines. I don't.

DAVIS: You're not answering your landline to get your - to talk to the pollsters (laughter)?

KURTZLEBEN: My what now? No (laughter). Yeah.

MONTANARO: It's illegal for robo-pollsters to call cellphones.

DAVIS: Yeah.

KURTZLEBEN: Right.

MONTANARO: So because of that, that's why the live caller issue is something that, you know, makes them a more of the standard in polling because they're able to call everyone. And you know what happens if you get rid of only people who have cellphones. I mean, how many people only have cellphones now or have a landline at all?

KURTZLEBEN: Yeah.

MONTANARO: I don't have a landline. I'm only on my cellphone.

KURTZLEBEN: Right.

MONTANARO: And I'm not that young anymore.

DAVIS: The other thing I would say, too, as you look at polling at this point the race, national polling really doesn't matter all that much. It does tell us something about where the country is, but in terms of things like the Democratic primary fight, I'm so much more interested in polls coming out of Iowa, New Hampshire, South Carolina and Nevada because that's closer to where the Democratic primary voter is that's going to elect the nominee.

KURTZLEBEN: Right. And the other thing that I constantly find myself thinking about and a little tortured by this early in the game is news coverage drives polling drives news coverage.

DAVIS: Yeah.

KURTZLEBEN: Especially on that national level. And I believe it's FiveThirtyEight that has been doing some tracking of cable news coverage, who is getting a whole bunch of the cable news coverage. Well, as it turns out, Joe Biden and Bernie Sanders get a fair amount of it, especially Joe Biden. Now, Joe Biden...

DAVIS: Do the polls create reality, or does the reality create the polls, kind of thing (laughter)?

KURTZLEBEN: Most definitely, yeah. On top of that, polls also measure name recognition. Well, Joe Biden was our vice president (laughter), so a heck of a lot of people know who he is. So polls right now are measuring a heck of a lot more than just - which candidate do you think would be best to run the country out of all of these? It also measures just, like, who do you happen to know who you think would be a good president?

MONTANARO: And this is usually where someone jumps in and says, you know, remember President Giuliani and Clinton?

KURTZLEBEN: Yeah (laughter).

MONTANARO: You know, because in 2008, they both led in the polls by quite a bit.

DAVIS: Right.

MONTANARO: And they didn't become president. Now, that said, there are plenty of other examples of people who've led early in the polls who then go on to become the nominee, and it's happened repeatedly. You think about Mitt Romney; he had sort of a back and forth with Mike Huckabee for a while in 2012 and then wound up, you know, sailing to the nomination and being ahead most of the time. Al Gore in 2000. George H.W. Bush in 1988. So there are plenty examples of people who have led from beginning to end.

I just think when you look at them, though, it's not that necessary to get yourself all wrapped up in those national polls right now. I'm looking more at favorability ratings. These horse race numbers are far more ephemeral, but how you feel about somebody doesn't usually change quite as much, unless there's an, you know, interrupting factor.

DAVIS: The other thing I'd say about polls, I always feel like, is more is more. And the good thing about this election cycle is I think we're going to have a ton of data. And the harder part is when you have fewer polls in a state or fewer national polls. I don't know if that's going to be a problem this time around. We're going to have a whole lot of data. And that makes it a little bit - again, not to be predictive, but if you have 10 polls in the state of New Hampshire and 10 different pollsters in 10 different partnerships all telling you Joe Biden or Bernie Sanders or Kamala Harris is winning...

KURTZLEBEN: Totally.

DAVIS: ...That does lead your reporting and your thinking to say. I mean, I do - you brought up the really interesting point, too, and where I think we get criticism from the public about polling is that they think that does the coverage of the polling kind of guide voters in a certain direction, right?

KURTZLEBEN: Right.

DAVIS: If all you're ever hearing as a New Hampshire voter is, Biden's going to win, does that influence voter behavior? And that's where I think the criticism of the culture of polling comes from.

MONTANARO: I will just say, it doesn't drive our coverage at NPR.

DAVIS: Yeah. Sure.

MONTANARO: We're as leery of the polls as other people are. We have lots of different themes that we want our reporters to go out and cover, lots of different ways that we want to get to each of those candidates to make sure that we're giving them their fairest shake. And you know, for our podcast listeners, obviously they've heard our Opening Arguments policy podcasts, where we're trying to lay out everybody's best case. So I think it's really important to not try to lead that coverage or lead those voters to make those decisions.

At the same time, there are obvious groups of tiers of what voters are telling us and what we're seeing from the polls of who they're thinking about as top candidates.

DAVIS: One thing I want to get to before we take a break, too, is the idea of issue polling. We talk so much about people, but I think we're also going to hear a lot about issue polling in this election when we hear about things like "Medicare for All," Green New Deal, socialism - how do you read those?

KURTZLEBEN: Issue polling is a really tough, nuanced thing to read, I personally think, because it matters so much how you ask the person the question. You know, if I ask Joe Blow (ph) out on the street, do you support Medicare for All, I might get a different answer than if I say, do you support single-payer, than if I say - inarguably, a much better-worded poll question than either of those - do you support a single-payer health system in which every single person in America is on a government-run - socialized would be a bad word there, but you know what I mean.

DAVIS: Yeah, wording matters.

KURTZLEBEN: Yes, exactly. So wording matters, and also, people's understanding of things matters. If you ask someone, do you support Medicare for All, they may not fully understand that, as it tends to mean in, say, the Bernie Sanders' usage, Medicare for All means, yes, single-payer health care. Some people might think it means public option.

DAVIS: Yeah.

KURTZLEBEN: So there are all sorts of different ways. And aside from that, you might get an answer from people that, yes, a majority of Americans support Medicare for All, but that doesn't tell you about the softness of that opinion. You know, the Kaiser Family Foundation, for example, put out polls where they say, OK, now, you say you like Medicare for All, but what if I told you that that would mean a lower on insurance rate? Then people support it more. OK, what if I told you that it means higher taxes? Then people go, oh, gosh, never mind, you know.

DAVIS: Domenico, do you think, though, that - should we be more skeptical of issue polling because there is much more ambiguity in the - what the numbers tell us, or is there a better way to read them?

MONTANARO: I think that it depends on the way not just the question's asked because there can be a top-line thing that I think is so much more valuable to the American political fabric than horse race polling; something like, do you support legalization of marriage for same-sex couples, right?

DAVIS: Yeah.

MONTANARO: That's a pretty straightforward question. And you can see the trend lines on that, and how they've reversed themselves over the past 10 to 20 years has been so dramatic. We wouldn't know that about American sentiment without that kind of polling. So to me, that is the real gold in polling, when you're able to be able to really test culturally where the American public is, how it changes, what it believes.

DAVIS: OK, so now that we know how to read the polls really smart...

KURTZLEBEN: Uh-huh (laughter).

DAVIS: We're going to take a quick break, and when we come back, we're going to talk exactly about what the polls are telling us right now.

OK, we're back. Again, some caveats - we're 18 months from an election. A lot of things can happen. But Domenico, as we sit here today, what are the polls telling us about the state of the presidential race?

MONTANARO: They're telling us nothing.

(LAUGHTER)

MONTANARO: Nothing matters.

KURTZLEBEN: It's over.

DAVIS: And thank you for listening to the NPR POLITICS PODCAST.

MONTANARO: Nothing matters. It's all - I mean, overall - right? And we've stayed away from horse race polling in the polling we've done so far because we are so far away. But when you look at the aggregation polls of what everyone else has done, Joe Biden has come into this field, has been a giant boulder in this lake or river - someone was upset that I said lake and said, you should say river because it's all flowing downstream and affects the currents going in whatever direction.

(LAUGHTER)

MONTANARO: Thank you for the help on my extended metaphor. He's been a big blanking deal coming into this election, in his words.

DAVIS: He's the front-runner.

MONTANARO: He's the front-runner. And he has gotten even stronger as his campaign has gone on in those head-to-head horse race polls. Bernie Sanders has basically been somewhere in second place, for the most part; the only other candidate, most of the time, who's in double digits. And then you have a cascade of others, with Pete Buttigieg, Kamala Harris, whole host of others who are there but haven't quite broken through to that next tier. There's a lot of time to go, lot of debates to come between now and the Iowa, New Hampshire primaries, and those two contests are going to really be a very sharp filter in this campaign.

DAVIS: One of the things that a lot of these early polls do is do head-to-heads. Donald Trump is likely to be the nominee. We don't know who he's going to run against. And it does seem like Biden is also running the strongest against Trump in a general election, not just for the Democratic primary.

KURTZLEBEN: Ignore it - boom.

DAVIS: Yeah, OK.

KURTZLEBEN: Yeah (laughter). No, I'm being flip. But I will honestly say that I tend to blow right past head-to-heads when I'm looking at a poll - right? - especially right now.

DAVIS: Yeah.

KURTZLEBEN: Like, because we're so far out. I mean, if you poll - I'm just going pull someone out of the ether - if you poll Pete Buttigieg versus Donald Trump right now, you get an answer that tells you something, but it tells you something so nebulous.

DAVIS: Yeah.

KURTZLEBEN: It tells you, as voters conceive of Pete Buttigieg right now in this field of nearly two dozen people, compared to Donald Trump right now, here's how he might run. Well, think about Pete Buttigieg now versus Pete Buttigieg in November of 2020, hypothetically. Like, by then, even leaving aside any sort of whatevers (ph) might happen by then, also by then, he won't have 20-some people he's running against, which means he won't have attacks from other Democrats. He will also have probably sustained some attacks from Donald Trump by then.

He - like, his competition will be totally changed. That's just one of many things that will have changed by then. So it's one good reason to just kind of take those head-to-heads with a bajillion (ph) grains of salt.

MONTANARO: You know, I don't think it's completely meaningless.

KURTZLEBEN: Yeah.

MONTANARO: I mean, what I wind up doing with it is I kind of put it in my back pocket. It's like, I would take that as a point in the line of figuring out the trend of how Pete Buttigieg did against Donald Trump in comparison from, you know, May of 2019 until July of 2020. If he were to get the nomination...

DAVIS: Sure.

MONTANARO: ...Then you're looking at, hey, this is where he was back then; here's where he is now.

KURTZLEBEN: Right.

DAVIS: On the other side of this, one thing I am very interested in watching with the president is his approval rating, right? I mean, that is something we get in polling, and that is something that really matters to a president seeking reelection. And one of the things I have found so fascinating about the Trump presidency is how stable his poll numbers have been throughout the course of his presidency.

You know, if you look at past presidents in recent years, people like George W. Bush or Barack Obama, they had 30-, 40-, 50-point swings in their popularity over the course of their presidencies. Trump's has stayed within about a nine-point range. It is the most narrow range for any modern president since they have started polling. And one of the things that tells us is that this country's minds' kind of made up when it comes to Donald Trump. You know, that's one of the narratives of this election. And whether we see any movement in that number is really something to watch as the race intensifies.

KURTZLEBEN: Right. And one thing I'd add to that is people's minds are made up in the current economic environment, right?

DAVIS: Yeah.

KURTZLEBEN: Because Donald Trump has been president during a pretty good economy. And so if next year, around the election, if somehow there is a downturn of some sort or, at the very least, some sort of a slowdown, it's possible that his approval rating will go down. Then again, maybe it won't because maybe his base just loves him that much. But that is definitely a variable we haven't gotten to see in action yet, is Donald Trump during a not-great economy.

MONTANARO: You know, aside from all the cultural issues that Republicans really care about and they deeply care about with this president, taxes and terrorism are two of the biggest other reasons that could pull voters away or keep his support there with them. And the fact of the matter, as Danielle noted, there's a pretty stable and strong economy right now, and there is not a hot war that's going on where you're seeing, you know, the death toll of troops go up day by day.

If you were to see either of those things suddenly drop off and there become major - a major world problem that's on the front pages of your newspapers and on your television every night, and you're seeing a major decline in the stock market and something that goes wrong with the economy, that's when you could see and usually do see a drop for a president. What I would posit, frankly, is that a president who's in this position with an economy that's this strong, with a foreign policy where there isn't anything necessarily, quote, "going wrong" right now, he should be much higher than where he is.

KURTZLEBEN: Most definitely.

MONTANARO: And that's sort of the precarious position that he's in.

DAVIS: I would just say that I think our fave-unfave (ph) is very high for the NPR POLITICS PODCAST.

MONTANARO: I like us.

DAVIS: (Laughter).

KURTZLEBEN: Am I partisan (ph)?

DAVIS: Three out of three.

MONTANARO: I strongly approve.

KURTZLEBEN: So does my mom.

DAVIS: All right, that is a...

KURTZLEBEN: She thinks we're a catch. OK, I'll stop. I'll stop.

(LAUGHTER)

DAVIS: That's a wrap for today. To keep up with up-to-the-minute news, follow us on Facebook, Twitter and Instagram. Just search for NPR Politics. I'm Susan Davis. I cover Congress.

KURTZLEBEN: I'm Danielle Kurtzleben, political reporter

MONTANARO: And I'm Domenico Montanaro, political editor.

DAVIS: And thanks for listening to the NPR POLITICS PODCAST.

(SOUNDBITE OF THE BIGTOP ORCHESTRA'S "TEETER BOARD: FOLIES BERGERE (MARCH AND TWO-STEP)")

Copyright © 2019 NPR. All rights reserved. Visit our website terms of use and permissions pages at www.npr.org for further information.

NPR transcripts are created on a rush deadline by Verb8tm, Inc., an NPR contractor, and produced using a proprietary transcription process developed with NPR. This text may not be in its final form and may be updated or revised in the future. Accuracy and availability may vary. The authoritative record of NPR’s programming is the audio record.