Mona Chalabi: How Can We Tell The Good Statistics From The Bad Ones?
GUY RAZ, HOST:
It's the TED Radio Hour from NPR. I'm Guy Raz. And on the show today, Can We Trust The Numbers?
(SOUNDBITE OF MUSIC)
RAZ: And if you just watch like an hour of cable news...
(SOUNDBITE OF NEWS MONTAGE)
UNIDENTIFIED REPORTER #1: Important - 228,000 net new jobs were created.
UNIDENTIFIED REPORTER #2: That middle fifth gets 1.5 percent of the total...
RAZ: ...You get a lot of numbers thrown at you...
(SOUNDBITE OF ARCHIVED RECORDING)
UNIDENTIFIED REPORTER #3: Thirty thousand undocumented immigrants with criminal records released last year.
UNIDENTIFIED REPORTER #4: The uninsured rate dropped to below 10 percent, the lowest in history.
RAZ: ...Form a lot of different sources.
(SOUNDBITE OF ARCHIVED RECORDING)
UNIDENTIFIED REPORTER #5: Thirty-six percent of Americans think global warming is a serious threat.
UNIDENTIFIED REPORTER #6: The top 1 percent gets 8.5 percent of the benefit.
RAZ: And while some people might have too much faith in the numbers, this kind of barrage makes other people really skeptical.
MONA CHALABI: So I think the reason why people are more distrustful of statistics is because they feel more alienated by them than they do with other sources of information. They feel that it's beyond them. They feel that they can't do it. And very often, it's just tempting to say, oh, whatever, it's probably a lie.
RAZ: This is Mona Chalabi. She's a data editor for The Guardian newspaper.
CHALABI: Basically, my job is to take a story and to kind of zoom out and provide context for readers. And the thing that excites me about data is just the scale of it, right? Like, data gives you a scale. It gives you a new frame of understanding. But very often, the way that statistics can be misleading is by simply changing what that context is. And I think people don't necessarily feel equipped to say, I don't really understand where these numbers came from. But they also know that that plays a role. And I think that's part of where the skepticism with statistics comes from.
(SOUNDBITE OF MUSIC)
RAZ: And, Mona says, we should be skeptical because there are a lot of bad numbers out there. But that means it's kind of up to us to blur how to spot the good statistics from the bad ones.
CHALABI: So if you get to the bottom of a piece and you think, I don't really get it - ask questions. Ask how they gathered that data. What is the base number? How many people are we talking about here? For any claim.
(SOUNDBITE OF TED TALK)
CHALABI: Anyone can do this. You don't have to be a geek or a nerd. You can ignore those words. They're used by people who are trying to say they're smart or pretending they're humble. Absolutely anyone can do this.
RAZ: Mona Chalabi picks up the idea from the TED stage.
(SOUNDBITE OF TED TALK)
CHALABI: So I want to give you guys questions that will help you be able to spot some bad statistics. So can you see uncertainty? Now, one of the things that's really changed people's relationship with numbers - and, in fact, even their trust in the media - has been the use of political polls. Based on national elections in the U.K., Italy and, of course, the most recent U.S. presidential election, using polls to predict electoral outcomes is about as accurate as using the moon to predict hospital admissions. No, seriously, I used actual data from an academic study to draw this.
There are a lot of reasons why polling has become so inaccurate. Our societies have become really diverse, which makes it very difficult for pollsters to get a really nice representative sample of the population for their polls. People are really reluctant to answer their phones to pollsters. And also, shockingly enough, people might lie. But you wouldn't necessarily know that to look at the media.
For one thing, the probability of a Hillary Clinton win was communicated with decimal places. How on earth can predicting the behavior of 230 million voters in this country be that precise? And then there were those sleek charts. See, a lot of data visualizations will overstate certainty and it works. These charts can numb our brains to criticism. When you hear a statistic, you might feel skeptical. As soon as it's buried in a chart, it feels like some kind of objective science and it's not.
The second question that you guys should be asking yourselves to spot bad numbers is, can I see myself in the data? Because part of the reason why people are so frustrated with these national statistics is they don't really tell the story of who's winning and who's losing from national policy. It's easy to understand why people are frustrated with these global averages when they don't match up with their personal experiences. The point of this isn't necessarily that every single dataset has to relate specifically to you. The point of asking where you fit in is to get as much context as possible.
OK. So the final question I want you guys to think about when you're looking at statistics is, how was the data collected? And I know this is tough because methodologies can be opaque and actually kind of boring. But there are still some simple steps you can take to check this. One poll found that 41 percent of Muslims in this country support jihad, which is, obviously, pretty scary. And it was reported everywhere in 2015.
Now, when I want to check a number like that, I'll start off by finding the original questionnaire. And it turns out that journalists who reported on that statistic ignored a question lower down on the survey that asked respondents how they defined jihad. And most of them defined it as, quote, "Muslims' personal, peaceful struggle to be more religious." Only 16 percent defined it as violent holy war against unbelievers.
Now this is the really important point. Based on those numbers, it's totally possible but no one in the survey who defined it as violent holy war also said they support it. Those two groups might overlap at all. It's also worth asking how the survey was carried out. This was something called an opt-in poll, which means that anyone could have found it on the Internet and completed it. There's no way of knowing if those people really even identified as Muslim. And finally, there were 600 respondents in that poll. There are roughly 3 million Muslims in this country according to Pew Research Center. That means the poll spoke to roughly 1 in every 5,000 Muslims in this country.
(SOUNDBITE OF MUSIC)
CHALABI: That survey, it was conducted by a polling organization called Woman Trend. And Woman Trend was set up by a woman called Kellyanne Conway, who is now part of the Trump administration.
RAZ: Yeah.
CHALABI: And so understanding the incentives that people might have behind those data-gathering organizations to reach certain conclusions is really important too. And again, for any statistic it's really important to ask, how was it gathered?
RAZ: But, I mean, we're so inundated with so many statistics. I'm not sure if, you know, most of us have the tools or the patience or the context to differentiate between the good ones and the bad ones.
CHALABI: It's quite funny. I think people think of numbers as being a very intellectual thing, but there are a lot of feelings wrapped up in it. So, you know, people literally have nightmares about being back in math class. And I think that actually people do have the tools. I haven't met anyone who doesn't have the intelligence to be able to kind of ask these questions. A lot of it is about feeling intimidated by the numbers and feeling like you can't question them.
RAZ: So on the one hand, it seems like you're saying, hey, be skeptical. Like, you should not really trust statistics. But on the other hand, you're saying, well, don't count them out either.
CHALABI: Yeah, absolutely. Skepticism is an inherently healthy and positive thing to have, but don't use that skepticism to just write numbers off. Just channel that skepticism to ask questions and feel empowered about the numbers that are available you.
(SOUNDBITE OF MUSIC)
RAZ: That's Mona Chalabi. She's a data editor for The Guardian newspaper and host of the new podcast Strange Bird. You can hear her entire talk at ted.com.
Copyright © 2018 NPR. All rights reserved. Visit our website terms of use and permissions pages at www.npr.org for further information.
NPR transcripts are created on a rush deadline by an NPR contractor. This text may not be in its final form and may be updated or revised in the future. Accuracy and availability may vary. The authoritative record of NPR’s programming is the audio record.