SYLVIE DOUGLIS, BYLINE: This is PLANET MONEY from NPR.
(SOUNDBITE OF COIN SPINNING)
NICK FOUNTAIN, HOST:
A few weeks ago, Jeff, you and I headed to Poughkeepsie, N.Y., to visit the Marist College Polling Center. Polling is how we figure out what Americans think about anything.
JEFF GUO, HOST:
And for pollsters, this is the most exciting time of the year. Right here, they are making hundreds of thousands of calls to figure out who's likely to win in the upcoming midterm elections.
FOUNTAIN: And the person tasked with training up all the people who do the dialing is Daniela Charter. And Daniela has agreed not only to let us sit in on a training, but later tonight she's going to let us do the dialing for a real poll. But first, she needs to run us through the script we're going to use.
DANIELA CHARTER: So why don't we start? Jeff, read that introduction, my first victim.
GUO: Hello, my name is Jeff. I'm calling for Marist College. We're talking to people in your community and collecting opinions about issues facing residents.
CHARTER: What? Who are you? Why are you calling me?
GUO: Daniela's mission is to prepare us for the hundreds of calls we're going to make tonight and all the different ways people are going to try to avoid answering our survey.
CHARTER: Sometimes people will come back right away and be like, I'm not buying anything. Or they hear college - I'm not donating. No, no, no, no. We're not looking for money. We're not selling anything. We just want your opinion. We're doing important research.
FOUNTAIN: We came to Marist College because, well, full disclosure, Marist has a working relationship with NPR. They do some polls for us. And also, Marist has one of the top polling operations in the country.
GUO: And one reason they're so good, Daniela says, is the magic phrases they use to keep people on the phone.
CHARTER: My very favorite is, let's try a few and see how it goes.
FOUNTAIN: As in, let's try a few questions and see how it goes. She says it works for nearly everything. If a person says they're too busy, well, let's try a few and see how it goes. If they say they don't have any opinions, well, let's try a few, see how it goes.
CHARTER: And it works. It really does. Does it work every single time? No, but you'd be surprised.
FOUNTAIN: Do you use that term in your personal life now?
CHARTER: Definitely. Let's try a few and see how it goes. In fact, I think I just said it to my husband yesterday about something - I think about changing his diet. I really do. I really - I just remembered that. I really think that I said it.
FOUNTAIN: Daniela teaches us some other phrases we can use to get people to answer. Every opinion matters. There are no right or wrong answers. It'll just take a few minutes, I promise.
GUO: Though Daniela says her most important advice. It's not a phrase. No, no, no. It's more of a state of mind. She tells us, when you are out there dialing later, promise me this - you're going to throw a big smile on your face.
CHARTER: Smile while you dial. Absolutely.
GUO: Smile while you dial because if you smile...
CHARTER: ...Sound upbeat, you sound good, and you sound like someone that the respondent's going to want to talk to.
GUO: That's how you hook them. And that's really important because the biggest problem that Daniela and other pollsters are facing right now - not enough people are completing surveys. They're hanging up, or worse, they're not even picking up the phone.
FOUNTAIN: And when people don't respond, it is way harder to get accurate polls. It has become a full-on crisis for the pollsters and maybe for the rest of us. Hello and welcome to PLANET MONEY. I'm Nick Fountain.
GUO: And I'm Jeff Guo. A lot of important information, a lot of economic data like what the unemployment rate is, where inflation's at, all of that relies on polling.
FOUNTAIN: But how do you know the polls are right? In just a couple of days, Americans are going to vote in the midterms. And for all the pollsters of America, this is their final exam, their big chance to check their work and make a case for their very existence.
GUO: Because for years, the polls have been off. They've been getting elections wrong. People are saying you can't trust the polls anymore.
FOUNTAIN: Today on the show, is polling broken, and can we fix it? We start smiling and dialing to find out.
(SOUNDBITE OF MUSIC)
GUO: We got to Marist ready to dial. And they told us, actually, you got to wait because pollsters generally work at night. That's when more people are likely to be home to pick up the phone. And so because of this, pollsters tend to be night owls.
BARBARA CARVALHO: We're really not morning people.
GUO: This is the director of the Marist Poll, Barbara Carvalho.
CARVALHO: We start the day probably about noon, and then, we go to about midnight.
GUO: Have you looked at a mirror recently? Was there something there or...
CARVALHO: Well, you know, some people think we're even worse than vampires.
CARVALHO: You can get rid of a vampire a lot easier than you can get rid of a pollster.
GUO: Barbara is a big deal in the polling world. She's been a pollster her entire career. For decades, she's run every aspect of the Marist Poll - what questions they ask, the methodology they use, the math behind it. It's Barbara's job to make sure they get things right and, maybe more importantly, that they don't mess things up.
FOUNTAIN: And that's because the history of polling has been defined, unfortunately, by blunders. And so to show us how polling works today, Barbara took us through the three most infamous blunders.
GUO: The first major blunder was the election of 1936. It was the middle of the Great Depression. Franklin Delano Roosevelt was running for reelection. His opponent was Alf Landon, the governor of Kansas.
(SOUNDBITE OF ARCHIVED RECORDING)
UNIDENTIFIED PERSON #1: The Republicans think Alf Landon is the man who will win in November. He is our next president if he can beat Roosevelt.
GUO: People were glued to their radios and to their magazines.
FOUNTAIN: Yeah. One of the most popular magazines in America at the time was called The Literary Digest, and it was especially famous for its election polls. For years, The Literary Digest mailed out millions of surveys to their subscribers and to any other address they could get their hands on.
CARVALHO: They would survey people who owned cars because there was a list of people who owned cars. They would survey people who owned homes because that was a list.
GUO: The Literary Digest asked these millions of people who they were going to vote for. It was basically trying to do the vote before the vote. And for a long time, this worked. The Literary Digest had a 25-year track record of getting elections right.
FOUNTAIN: Enter George Gallup, a pioneer of modern polling. Gallup had a hunch based on math that there was a better way to predict election results. Rather than get the opinions of millions of people, Gallup said, I'll just interview a fraction of that amount - just 3% - but the right people. And they'll give me the more accurate answer.
GUO: So in one corner, there was The Literary Digest poll and their survey of 2 million people. They were saying, yeah, FDR is going down.
FOUNTAIN: They said it was going to be a Landon-slide (ph) for Landon, that Alf Landon was going to land on Washington. Those are his campaign slogans, not our bad puns.
GUO: And then, there was Gallup and his teeny, tiny survey. And he said, no, actually, FDR is going to win.
FOUNTAIN: If you know anything about American history, you know who won this battle. It wasn't even close. FDR won all but two states. Gallup was right, and The Literary Digest went out of business.
GUO: Gallup said the problem with The Literary Digest poll was that the respondents, even though there were a lot of them, they didn't look like America. They were wealthier than average. They owned cars. They subscribed to a magazine called The Literary Digest.
CARVALHO: Just from the name Literary Digest, you kind of get a sense this is not an Everyman's publication.
GUO: Are you saying that it's snooty to read?
CARVALHO: Well, I think for The Literary Digest, that was true.
FOUNTAIN: Gallup said the issue was snooty people with their fancy cars and homes and books, they were not what's called a representative sample of voters.
GUO: That was Gallup's big idea and the big lesson of polling blunder No. 1, that if you're going to conduct a poll, you have to talk to the right mix of people, get a representative sample.
CARVALHO: Math won. Math always wins.
FOUNTAIN: However, Gallup's victory lap, it wouldn't last long because the next big polling blunder was just around the corner - the famous Dewey-Defeats-Truman headline, where a newspaper got the 1948 election results so very wrong. That was, in part, because Gallup and a bunch of other pollsters had been saying that Harry Truman would lose the election.
How big of a moment was that for polling?
CARVALHO: Do we really have to talk about that? I mean, that is, like, one of the biggest, you know, polling debacles, you know, in the history of polling. And it had a significant impact.
FOUNTAIN: Yeah, the polling blunder here was although the pollsters tried to get a representative sample, it just wasn't representative enough.
GUO: Here's what happened. Gallup had been telling his surveyors, just get me a bunch of people that look like the census. Go poll this many housewives, this many farmers, this many businessmen of a certain income bracket, and so on and so forth.
FOUNTAIN: And as pollsters would hunt for these very specific people, grab them off the street and ask them, you voting Dewey or Truman, this was problematic because it was hard to fill these buckets. And so the pollsters, who weren't paid all that well, some of them started to cut corners. All right, sure. You're not a well-heeled businessman. But say you were. You voting Dewey or Truman?
GUO: So one of the big lessons from the Dewey-Defeats-Truman disaster, polling blunder No. 2, is that it's really hard to make that representative sample by hand.
FOUNTAIN: And the way modern polling fixed this problem was with more statistics. The math people said there's a better way of making a representative sample. And it's so simple that it almost sounds dumb. Here it is - just pick randos. Just close your eyes, and pick people at random.
CARVALHO: Generally, if you're looking at a simple random sampling, when you get to about a hundred individuals, you can make some pretty clear generalizations about that particular group of people.
GUO: This is the magic behind random polling. You don't have to go hunting around for the right people to make a representative sample. In fact, that can distort the poll by introducing bias. The math says a truly random sample won't be biased and that even a small random sample will be surprisingly accurate. This was one of the most important ideas of the 20th century. It changed how scientists do research, changed how we study economics. And for pollsters, this became their holy grail - getting that perfect random sample.
FOUNTAIN: Luckily, just around the corner, there was a revolutionary technology that would make it kind of easy to achieve that holy grail, the random sample - the telephone.
GUO: Did the telephone revolutionize polling?
CARVALHO: No question about it.
GUO: At a certain point, by the '70s or '80s, almost every household in America had a telephone. And telephones are great for random sampling because you can just pick random numbers out of a list.
CARVALHO: So that would be the golden age of polling.
GUO: And Barbara says the golden age lasted for decades pretty much. People had landlines, and they were actually excited to pick up their phones. Soon, pollsters could even dial with a computer - robo-dialing. And even though there were complications over the years, like the rise of caller ID and cellphones, pollsters were able to get pretty close to that holy grail, that fully random representative sample.
Was there a time in the golden age of polling that, like, you just, like, nailed it?
CARVALHO: Oh, absolutely. Absolutely. In fact, I think 2014.
(SOUNDBITE OF ARCHIVED RECORDING)
RACHEL MADDOW: Get your popcorn popped, plan your hydration needs accordingly because this is going to be a long night with lots of...
FOUNTAIN: Barbara and her team were at NBC that night. And she remembers the excitement, the delicious snacks, but mostly watching the election results roll in and how they totally matched her Marist polls.
CARVALHO: It was just so cool that everything just worked out perfectly, like, practically to the decimal point.
GUO: But Barbara had a bad feeling.
CARVALHO: My comment to my colleagues was, relish this moment. Remember it forever 'cause it doesn't always happen this way. And polling is always going to change, so you actually might not see this again.
GUO: Dun dun duh (ph).
FOUNTAIN: You jinxed it.
CARVALHO: I jinxed it. I totally jinxed it.
(SOUNDBITE OF MONTAGE)
SIMON SCOTT, BYLINE: Polls leading up to the 2016 presidential election were wrong, wrong, wrong.
KATY TUR: At 2:50 this morning, Donald Trump did what so many said could never happen.
RENEE MONTAGNE, BYLINE: Many people this morning are wondering, how could pollsters have gotten this so wrong?
FOUNTAIN: Yeah, 2016 was the third major blunder for pollsters. A lot of them and the people who make forecasts based on polls said Hillary Clinton was going to be the president. That did not happen. The polls missed. And a lot of the polls missed again in 2018 and again in 2020. What the past three elections have revealed is the golden age of polling is over.
GUO: And mostly, that's because it's gotten so much harder to get that random representative sample. These days, people just don't pick up the phone anymore. The response rate for telephone polls, that's plummeted to, like, less than 1 in 20. And what made it worse was in 2015, the FCC took away a really important tool. The FCC said, you pollsters, you can't robo-dial cellphones anymore.
It's been seven years. You still seem kind of pissed.
CARVALHO: Well, of course, because in order to connect with a cellphone, we have to hand-dial that cellphone.
FOUNTAIN: Literally hand-dial.
GUO: Wait, are you literally, like, boop, boop, booping the phone?
CARVALHO: We boop, boop, booped the phone for quite a while.
FOUNTAIN: Barbara says the FCC rule, it tripled their costs. They need a lot more people to dial and a lot more stacks to feed them.
GUO: What's really concerning is that it's become especially hard to reach certain kinds of people like rural voters and younger voters. They're not participating in these surveys as much anymore. So all the samples that the pollsters are getting, they might not be that representative of America anymore. Like, some analysts think that Trump supporters specifically are less likely to talk to pollsters. And if that's true, that could explain why so many polls seem to favor Democrats recently.
FOUNTAIN: There are a lot of other challenges for pollsters, of course. But most of them come back to the same age-old problem - getting that perfect representative and random sample.
GUO: And this has got a lot of pollsters wondering...
(SOUNDBITE OF JAMES TAYLOR QUARTET'S "THE DANZIG CONNECTION")
GUO: ...Is this the death of the telephone poll?
FOUNTAIN: After the break, just how bad has polling gotten? We try boop, boop, booping to find out.
(SOUNDBITE OF JAMES TAYLOR QUARTET'S "THE DANZIG CONNECTION")
FOUNTAIN: The Marist Survey Operations Center is a sight to behold. There are rows and rows of undergraduates with their AirPods and their sweatpants and in the center, a prize wheel. You get to spin the wheel when you complete interview.
GUO: Nick, are you ready?
FOUNTAIN: I think we got to be ready. Like, it's dialin' time. We got, like, 30 undergrads here. They're already dialing.
GUO: The goal of the evening is to get as many completed surveys as possible. And lately, this has been really hard, like, existential crisis for the polling industry hard. People just don't want to pick up their phones, or they screen their calls. And even when they do pick up, they don't want to talk to pollsters.
FOUNTAIN: We figured, let's try a few and see how it goes. Let's see how many calls we have to make before we can get someone to complete our poll.
GUO: Nick, look, you have a special nametag. It says Nick Fountain, training.
FOUNTAIN: Exclamation point.
GUO: Exclamation point - you got to smile while you dial.
FOUNTAIN: All right. Good luck out there.
GUO: OK. You got this.
Nick and I are dialing North Carolina tonight. There's a contentious Senate race going on.
FOUNTAIN: The Marist folks set us up with our own cubicles with these cute little headsets hooked up to phones. On the screens in front of us, there's a script. Jeff calls a few numbers, and one of them connects.
GUO: Hi, my name is Jeff Guo. I'm calling from Marist College.
FOUNTAIN: Immediately, he starts using the tricks that Daniela taught us earlier.
GUO: So we could try a few questions and see how it goes. Those magic phrases - they really work.
FOUNTAIN: Pretty soon, Jeff has finished his first interview.
CARVALHO: Got a complete.
UNIDENTIFIED POLLSTER #1: Woo.
GUO: I got it.
UNIDENTIFIED POLLSTERS: Yay.
GUO: My God, this is stressful. Nick.
CARVALHO: Now you got to spin the wheel.
GUO: Wait, really?
GUO: Wait, I get to spin the wheel?
They take me to the middle of the room, and they let me spin the prize wheel. Gosh, look at all these choices. OK.
(SOUNDBITE OF PRIZE WHEEL CLICKING)
GUO: Pick a prize. Pick a prize. I could pick any of these?
UNIDENTIFIED POLLSTER #2: Yes.
GUO: I pick the M&Ms.
FOUNTAIN: Must be nice. Must be nice - because while Jeff is enjoying his M&Ms, basking in his glory, I'm striking out.
Talking to people in your community and collecting opinions about issues facing residents. Oh, I got hung up on. I did not smile.
UNIDENTIFIED POLLSTER #3: And you know what? That's all right. That's all right.
FOUNTAIN: I did not smile.
GUO: For each call, Nick has to document why he failed at getting a completed survey - in this case, caller hung up. That's one of nearly 20 reasons you can put down. There's also the soft refusal...
FOUNTAIN: He was nice about it, though.
GUO: ...The hard refusal...
FOUNTAIN: He said opinions are like buttholes. Everyone's got them.
GUO: ...Their answering machines...
FOUNTAIN: Oh, thank you. Is this a real person? It was a voicemail.
So many answering machines
GUO: ...And fax machines.
FOUNTAIN: This is a fax machine. Sorry, it's blowing out my ear.
GUO: Remember, we are randomly dialing North Carolina numbers.
FOUNTAIN: Have you ever seen someone strike out 70 times in a row?
Things are not going well. I get so frustrated, I start rubbing my eyes.
Oh. I'm sorry.
UNIDENTIFIED PERSON #2: No, it's OK.
FOUNTAIN: I just lost a contact.
Starting at about call 150, I start to think, yes, polling is truly doomed. Phone polling is supposed to be the gold standard of polling. It gave us the golden age. But it is impossible to get people to complete these surveys.
GUO: And the people at Marist are trying to figure out how to do polls in this era when most people don't pick up the phones and don't want to answer surveys. Barbara says pollsters have come up with a couple ways to try to fix the problem.
FOUNTAIN: The first solution is maybe just abandoning the phone poll altogether. A lot of pollsters these days are starting to text people. It's still a random survey. It just doesn't rely on people answering the phone.
GUO: The second solution is to try to adjust the data that you have to make it more representative. For instance, if you can't get enough white rural voters in your sample, you could take the handful of white rural voters you did talk to and multiply them by, like, two or three or four, so you give their answers more weight. This solution is called weighting. It's how pollsters try to fix a bad sample by using math. And pollsters are getting more and more sophisticated about weighting. A lot weight by race and education. At Marist, after 2016, they've become a lot more careful to wait by geography.
FOUNTAIN: And they're also trying to weight by what people call psychographic factors. Jeff, who has actually gotten someone to pick up the phone - he got to ask these psychographic questions, and they're pretty weird.
GUO: And have you smoked at least 100 cigarettes in your entire life?
FOUNTAIN: The idea behind this question is that it might help you understand someone better than their race or age or gender. And pollsters are hoping that this extra information will help them make their samples more representative.
GUO: And there's a third solution, one that I personally think is brilliant. It involves this really interesting idea that economists love called the wisdom of crowds.
FOUNTAIN: Some researchers have been testing these ideas. And they told us, if you want to predict an election, do not ask people who they are going to vote for. Instead, there are these two other questions that will get you way closer to the truth.
GUO: The first question is, who do you think will win? And this is supposed to be better because you're asking people to make a prediction themselves based on everything they've heard and seen in the last few months.
FOUNTAIN: The other question, which the researchers really love, is, think of all the people in your life - your friends, your family, your co-workers. Who are they going to vote for? And the beauty of that question is you're basically asking people to share information from their social networks. So talking to one person is like talking to a dozen. Right now Barbara and the folks at Marist - they aren't using these wisdom-of-the-crowds techniques.
So, Barbara, we have a question for you.
FOUNTAIN: Would you be willing to let us test one of these methods in one of your polls?
CARVALHO: Oh, sure.
CARVALHO: Well, if they're - I mean, if they're appropriate, objective, you know, questions, that would be great.
GUO: So here's the exciting news. We worked with Barbara to draft two questions - one, where we ask people to predict the election and another asking them who the folks in their social circles are voting for.
FOUNTAIN: She said she would slip them into a poll Marist is doing before the midterms, run some analyses and let us know how useful they might be for the future of polling.
GUO: Back in the dialing room, it's, like, 8 p.m.
FOUNTAIN: I'm not even sad when they hang up anymore. I'm numb.
GUO: We've been dialing for 3 hours, and poor Nick is still striking out.
FOUNTAIN: Yeah. At this point, I'd pretty much given up. I'd called 236 numbers and hadn't gotten a single completed survey. The rejections on top of rejections - they started to get rather amusing in a dark-humored sort of way.
UNIDENTIFIED POLLSTER #4: This is another thing that's really cool about this. You get a lot of funny stories.
FOUNTAIN: Yeah? What's a funny story?
And then just as one of the students is telling me a hilarious story...
Hi there. My name's Nick Fountain. I'm calling from Marist College. We're talking to people in your community and collecting opinions about issues facing residents. Are you 18 years of age or older? It'll just take a few minutes. Oh, terrific.
GUO: Nick has hooked someone. And as he gets further and further along in the survey, the room starts to get a little quiet. A crowd starts to hover around his cubicle. We're all waiting to see if he will finally finish a survey.
FOUNTAIN: That's totally fine. Thank you. Have a wonderful night. I really appreciate your time. Take care.
How do I hang up?
UNIDENTIFIED POLLSTERS: Woo.
(SOUNDBITE OF BELL)
UNIDENTIFIED POLLSTER #4: Amazing.
CARVALHO: Spin that wheel.
UNIDENTIFIED POLLSTER #5: Woo.
GUO: You did it. You did it.
FOUNTAIN: I did it.
(SOUNDBITE OF CHEN BOI'S "SAKU")
FOUNTAIN: I think I figured out why we cracked it.
UNIDENTIFIED POLLSTER #4: OK.
FOUNTAIN: You were telling me a funny story. And so I was already smiling when I was dialing.
UNIDENTIFIED POLLSTER #4: See, that's the secret. Smile while you dial. That's it. That's it.
(SOUNDBITE OF CHEN BOI'S "SAKU")
GUO: Hi, Barbara.
FOUNTAIN: Hey, Barbara.
CARVALHO: Hey, guys.
FOUNTAIN: A couple days ago, we called Barbara to see how our wisdom-of-the-crowds type questions came out.
So I understand you were able to sneak a couple of our questions into one of your polls.
CARVALHO: We did. We did.
GUO: Barbara says they talked to over a thousand people all over the country and asked them about the race for the House of Representatives in their district. First, they asked the standard question, who are you going to vote for, the Democrat or the Republican? And for that question, the two parties are basically neck and neck. But when you look at the new PLANET MONEY questions, our wisdom-of-crowds questions...
CARVALHO: When we look at who people think that their social contacts are going to vote for, we actually found among registered voters a plus-10 Republican...
GUO: That's a big difference.
CARVALHO: ...And a plus-14 among those who say that they're definitely going to vote. When we look at who they think is most likely to win the election in their district, we see among registered voters a plus-12 Republican.
CARVALHO: ...And a definite vote of a plus-16.
FOUNTAIN: The wisdom-of-the-crowds type questions are heavily favoring the Republicans, much more than the standard question, though Barbara is careful to mention that all of this is just preliminary, and the real test is when she puts all this data together and does some analysis. And besides, it's a little hard to figure out which questions are working better on a national poll. But...
GUO: If the election goes more toward the Republicans, is that a little bit more evidence for maybe the usefulness of these questions?
CARVALHO: Absolutely. And so what we will take a look at - and mind you, we're not ending the experiment here.
CARVALHO: What we've done is we've also included your questions on the state polling...
GUO: No way.
CARVALHO: ...That we're doing through Monday. Oh, way - because this is too much fun for statistical geeks. So we will also have these questions to be able to compare the results in the Senate races in, let's see, Pennsylvania, Arizona and Georgia - so to be continued.
GUO: Oh, my gosh.
FOUNTAIN: That's awesome.
GUO: Oh, my gosh. That's so exciting.
FOUNTAIN: Well, Barbara, thank you for keeping the experiment alive. Like, we're so excited to check in with you in the future.
CARVALHO: Well, thank you so much. It really - we had a lot of fun, and hopefully we'll get some interesting results as well to help move the polling process forward.
(SOUNDBITE OF CHEN BOI'S "SAKU")
FOUNTAIN: This episode of PLANET MONEY was produced by Emma Peaslee and mastered by Gilly Moon. It was edited by the great Jess Jiang.
GUO: Special thanks to the whole team at Marist, including Stephanie Calvano, Mary Griffith, Lee Miringoff, Rachel Sandford and our student coach H. Ede (ph) Also, thanks to the researchers who helped us with our wisdom-of-crowds questions, Henrik Olsen and Drazen Prelec, and to our colleague Domenico Montanaro. I'm Jeff Guo.
FOUNTAIN: And I'm Nick Fountain. This is NPR. Thank you for listening.
(SOUNDBITE OF CHEN BOI'S "SAKU")
NPR transcripts are created on a rush deadline by an NPR contractor. This text may not be in its final form and may be updated or revised in the future. Accuracy and availability may vary. The authoritative record of NPR’s programming is the audio record.