The Challenges Behind Accurate Opinion Polls

This week's stunning defeat of House Majority Leader Eric Cantor came after his internal polls showed him winning — instead he lost by double digits.

Copyright © 2014 NPR. For personal, noncommercial use only. See Terms of Use. For other uses, prior permission required.

RENEE MONTAGNE, HOST:

And now to another big reversal, this one in the world of American politics. The shocking defeat of House Majority Leader Eric Cantor came after his internal polls showed him winning by a safe margin. Instead, he lost by double digits. NPR's national political correspondent Mara Liasson reports on the challenges of figuring out how people are going to vote.

MARA LIASSON, BYLINE: America is having a big data moment. Not only is everything in our lives subject to measurement analysis, but public opinion has never been so closely monitored by so many.

UNIDENTIFIED REPORTER 1: Well, brand-new Fox News poll shows that Americans trust...

UNIDENTIFIED REPORTER 2: Brand-new numbers releasing right now...

UNIDENTIFIED REPORTER 3: Three out of 5 Americans are very or moderately worried about having enough money for retirement.

UNIDENTIFIED REPORTER 4: Twenty-seven percent of those surveyed have an unfavorable opinion of him.

UNIDENTIFIED REPORTER 5: And a CBS news poll out this morning reveals most Americans do not like...

LIASSON: Many horserace polls, the ones that try to determine the outcome of an election, are accurate, but there's always an exception that proves the rule - like this week in Virginia, where Eric Cantor's pollster thought he would win by 34 points. Instead, he lost by 11. Getting a representative sample in an electorate that small - only 6,500 people voted in Cantor's primary - is really difficult. But even in a high turnout general election, it's tricky to get a sample of voters that mirrors the actual electorate. Neil Newhouse famously struggled with this when he forecasted a Romney win in 2012. Newhouse had created a turnout model of the electorate that included fewer young and minority voters than ended up turning out.

NEIL NEWHOUSE: We all recognize the growing challenges of pulling in a demographically changing country and especially one where white voters are proportionally less of the population as each year progresses.

LIASSON: J. Ann Selzer is a nonpartisan pollster who says the challenges the campaign grassroots machines are changing the electorate faster than pollsters can model.

J ANN SELZER: Pollsters that had likely voter models based on past elections weren't recognizing that the present and the future of getting people out to vote has changed. So they look to past elections, and they said well, younger people vote less often, minorities vote less often, so we're going to count them as less likely to vote.

LIASSON: Polling has other challenges. And to be a savvy consumer of polls, like any other product, it's important to read the fine print. Every poll includes information about how the poll was taken. How many people were surveyed? Was it a random sample? Did the questions seem fair? And, says Pew Research Center pollster Scott Keeter, was the poll conducted by human beings or a computer?

SCOTT KEETER: Robot polls have been very popular in the last several election cycles in particular because they're cheap. They're a way of getting a lot of interviews in a short space of time. But they have a couple of problems that they've had to confront.

LIASSON: One problem with automated polls where respondents are asked to answer questions by punching buttons on their phone is that federal law allows robocalls on landline telephones only.

KEETER: We really don't know who's answering them. There's, by definition, no interviewer present. So children could be responding to them if they wanted to. People that are not in the target population, such as registered voters, could be answering them.

LIASSON: And then there are Internet polls - even more unreliable because they allow people to opt in voluntarily from their favorite websites. But Pew and other polls continue to experiment with new ways to find people because the old ways just aren't working. Democratic pollster Peter Hart started his career by going door-to-door to survey votes, but now Hart says fewer and fewer people want to talk to pollsters.

PETER HART: That is the big challenge facing this industry - is there's a declining response rate. And the second challenge is we now have at least 35 percent of our population that has cell phones only. So that's the only way that you can reach people, and that's much more expensive.

LIASSON: More expensive because it takes a lot more effort to find people with cell phones. There's no cell phone phonebook, and there's no way of knowing where people live - not like in the old days when you move to a new city and got a new phone number.

HART: People get a cell phone and they move, and therefore, the person who was in Cleveland may now be in Boston but using a Cleveland area code.

LIASSON: Pollsters also suggest that consumers reading polls look at a lot of them and look at the same ones over and over again so they can see trends. That's because, says Neil Newhouse, public opinion is a moving target.

NEWHOUSE: Public opinion is fluid, and polls try to pick that up. They're a snapshot in time, but it's where voters stand at a certain point in time, and that can change.

LIASSON: And that's why designing a survey that captures those changes is truly a combination of art and science. Mara Liasson, NPR News, Washington.

MONTAGNE: You're listening to MORNING EDITION from NPR News.

Copyright © 2014 NPR. All rights reserved. No quotes from the materials contained herein may be used in any media without attribution to NPR. This transcript is provided for personal, noncommercial use only, pursuant to our Terms of Use. Any other use requires NPR's prior permission. Visit our permissions page for further information.

NPR transcripts are created on a rush deadline by a contractor for NPR, and accuracy and availability may vary. This text may not be in its final form and may be updated or revised in the future. Please be aware that the authoritative record of NPR's programming is the audio.

Comments

 

Please keep your community civil. All comments must follow the NPR.org Community rules and terms of use, and will be moderated prior to posting. NPR reserves the right to use the comments we receive, in whole or in part, and to use the commenter's name and location, in any medium. See also the Terms of Use, Privacy Policy and Community FAQ.