Behind The Scenes: NPR Fact Checks First Debate In Near Real Time
NPR listeners and readers have said they want fact-checking during this political campaign season and NPR responded with what I found to be a very impressive new offering Monday night: a close to real-time annotation of a transcript of the first televised debate between Democratic presidential candidate Hillary Clinton and Republican presidential candidate Donald Trump.
As the debate transcript rolled by, supplied by an automated transcription service, a team of some 30 newsroom staffers, including reporters, editors and members of the visuals team, jumped in to fact-check and annotate what the candidates were saying and keep the process moving along. By the end of the process, the transcript contained 74 annotations, some adding context and others pointing out statements the candidates got right or wrong.
NPR's audience clearly appreciated the massive effort; as of Tuesday afternoon, more than 6 million people had checked it out.
My office heard great praise for and some criticism of the effort. Some of the critics were unhappy with the errors made by the automated transcription service. My take on that is that most of those looked to be hand corrected by NPR staff by the morning.
Others saw bias in which candidate statements got fact-checked and which didn't. That's always an inherent risk in fact-checking a lengthy candidate speech or 90-minute debate.
Here's how the totals broke out, according to our tally. NPR fact-checked nine Clinton statements, and 33 Trump statements. Another 32 annotations added context for what the candidates were saying (11 added context to Clinton, and 21 to Trump). The vast majority of fact-checks on Trump concluded that he was wrong; by our count just a handful of Clinton's statements were judged wrong.
Some emailers sent lists of Clinton statements that NPR did not fact-check. One example, sent by Grant Cates of Orlando, was this Clinton statement:
"Well, let's stop for a second and remember where we were eight years ago, we had the worst financial crisis — the great recession, the worst since the 1930s. That was, in large part, because of tax policies that slash taxes on the wealthy, failed to invest in the middle class, took their eyes off of Wall Street and created a perfect storm."
Cates wrote: "Seems like an obvious opportunity for NPR to summarize the causes of the great recession. What were the tax policies and failed investments that led to the recession? What about subprime loans for housing and how government policies promoting home ownership had unintended consequences?"
I'd agree with that, and I see other Clinton statements that could be checked, too. I also see Trump statements that went unchallenged, like this one: "Let me give you the example of Mexico. They have a VAT tax, we're on a different system. When we sell into Mexico, there is a tax when they sell, an automatic 16 percent, approximately. When they sell into us there's no tax. It's a defective agreement."
I asked the newsroom about the methodology behind the fact-checking. Domenico Montanaro, NPR's lead political editor, emailed:
"We wanted to try something new and comprehensive that focused on the facts and built on what we have done in recent months — annotations of convention speeches, economic addresses and more. In the planning stages, we reached out across desks to policy and beat experts and editors from business to foreign affairs to education and created a shared document that everyone who wanted to be involved could do so in an easy way. Our team of political and visuals editors behind the scenes filtered, managed, checked and updated. The editorial was organic. We told our folks essentially if you see something, say something. Like viewers at home, when we heard something that made us go, 'Is that true?' Or 'That's not true.' Or 'well that's not the whole context,' our reporters were on it. As the night evolved, more got added and updated. If something stood out to an editor that hadn't been fact checked, then we reached out to a relevant reporter. But honestly, this very sharp collaborative team left very few rocks unturned in about as fast a way as you can."
As for the emphasis on fact-checking Trump, he wrote:
"What's fair isn't always equal. Our reporters across multiple desks, across multiple beats with deep content knowledge did a stellar job of fact checking in real time. We certainly are always cognizant of providing balance and focusing on both candidates, but if the numbers show that one candidate was fact checked more, it might be because that candidate said more things that needed fact checking."
My take: It's not possible to fact-check every candidate statement. Some additional statements could have been checked but, overall, I think NPR's team offered a valuable service that met the goal of providing citizens with information they need to make their November decisions. For more on the impact of fact-checking in an election, I'd recommend reading Danielle Kurtzleben's piece at NPR.org. Meanwhile, NiemanLab had more on the technical gee-whiz behind the project.
Editorial researcher Annie Johnson and intern Anna Mazarakis contributed to this report.