NPR logo

Post A Survey On Mechanical Turk And Watch The Results Roll In

  • Download
  • <iframe src="" width="100%" height="290" frameborder="0" scrolling="no" title="NPR embedded audio player">
  • Transcript
Post A Survey On Mechanical Turk And Watch The Results Roll In

The Industry

Post A Survey On Mechanical Turk And Watch The Results Roll In

  • Download
  • <iframe src="" width="100%" height="290" frameborder="0" scrolling="no" title="NPR embedded audio player">
  • Transcript


Here's a new way college students and professors are doing research. It's called Mechanical Turk. It's a website run by Amazon, the same Amazon that delivers everything from books to toiletries. Let's say you need 200 smokers to take a survey on lung cancer or you have a moral dilemma to pose for a paper on Kierkegaard, you can login, offer a few pennies to each participant and watch the data roll in.

NPR's Gabrielle Emanuel explains.



EMANUEL: Go. OK, so the page that loads is full of these gray rectangles. Each one is a different task. Right here the top, there is one other UCLA. They're paying 25 cents for a five-minute survey on consumer decision-making. The next one is 10 cents and it's about dating preferences. Let's scroll down and see what else we can find.

OK, here is one from Ohio State paying 50 cents for workers to fill out a survey about their political beliefs.

ADAM BERINSKY: It's very efficient. It's very fast. It's very cheap.

EMANUEL: That's Adam Berinsky, a professor at MIT. He studies public opinion. And some of those gray rectangles are his studies and surveys.

BERINSKY: I mean, there's been a huge explosion. So about four years ago, not a lot of folks were doing it but now, everyone I talk to at conferences is doing that.

EMANUEL: It all started about a decade ago. Amazon had a problem. It has millions of Web pages each with a product for sale, but some were identical. The company wanted to weed out those duplicates. This is hard for computers to do but easy for people to do. So Amazon offered to pay people a few cents to do it for them. Soon, other companies were posting tasks too and Mechanical Turk was born.

Why the name? Well, it references a famous hoax from the 1700s. There was this chess playing machine that could beat everyone. But it actually was a chess master hiding inside. So, too, with Mechanical Turk. It seems like a computer is doing the work but it's actually a person.

ANA VASQUEZ: I do between six to eight hours and earn about $20 a day.

EMANUEL: Ana Vazquez is unemployed but between job interviews, she Turks from her Brooklyn apartment. She does the math.

VASQUEZ: Two fifty an hour, which is basically, I guess, waitressing without the tips.


EMANUEL: Mechanical Turk, or MTurk as it's called, has sometimes been criticized as a digital sweatshop. Vasquez knows the pay is peanuts but she really wants a job in social services.

VASQUEZ: I'm enjoying the flexibility to keep pursuing that and not have to like, say, choose something right now because I need the money.

EMANUEL: So she is strategic. There are lots of different tasks on MTurk; Googling websites to boost their ratings, transcribing, copy editing. But she says research studies pay the most.

VASQUEZ: Recently, I did one on memory and it just involved, you know, clicking different numbers on the screen. It's probably for someone's Ph.D. or something like that.

EMANUEL: Quite likely. Jonathan Phillips is a Ph.D. student at Yale. He uses MTurk all the time. Compared to what he did before, he says it's a dream. He used to print out questionnaires and hit the street.

JONATHAN PHILLIPS: You'd spend like two to three hours before you got too exhausted to continue. And you'd get 15, 20 responses. And it's just so painstaking.

EMANUEL: But now...

PHILLIPS: You come up with a study and then, like, an hour later, you have 200 responses, which is pretty...


PHILLIPS: know, fundamentally different.

EMANUEL: Berinsky, over at MIT, says researchers save...

BERINSKY: Not just weeks of effort but great amounts of money as well.

EMANUEL: He's paying a couple cents per participant, compared to the 10 or $15 he used to pay.

OK, so MTurk is fast and cheap. But is it good? Are the data they're getting reliable? The research shows that the population of Turkers is pretty representative, more so than signing up college students. There are other benefits, too.

BERINSKY: I think it's actually improved the extent to which we can really be sure about our results.

EMANUEL: From his tucked away office, Phillips gives me an example. Say you put up a survey but the wording isn't quite right. That could skew the findings.

PHILLIPS: People are much more willing to say: Well, I didn't really like that you used that one word there. So can you just change it and rerun it on Mechanical Turk, knowing that it's not going to take you that long.

EMANUEL: MIT's Berinsky has another upside.

BERINSKY: It's enabled people to take more chances, to do things that might seem crazy or sound crazy but are really clever and interesting and turn out to be right.

EMANUEL: So what about the downsides? Maybe it's made research too easy and too cheap. Berinsky says some scholars might be throwing up bad surveys.

BERINSKY: Maybe it would be better if people had to invest a little more of their own money before they ran with an idea.

EMANUEL: And not all research is suited to Mechanical Turk. If your study involves children, that's a no-go. Need to observe your subject's facial expressions, can't do that on MTurk. Those researchers can, of course, just do their research the old-fashioned way. But Jonathan Phillips worries some researchers are shying away from tough topics.

PHILLIPS: I don't think we have any good sense of exactly, you know, which questions are now being neglected because they're harder to study. But I think it would be crazy to deny that it's having an influence on which questions we ask and how we ask them.

EMANUEL: And there's another change underway. We think about academics toiling away in a cubicle or sterile laboratories. I spoke with one of Berinsky's colleagues. He surveys American voters on MTurk, but he does it from Indonesia. He might even be on the beach.

Gabrielle Emanuel, NPR News.

Copyright © 2014 NPR. All rights reserved. Visit our website terms of use and permissions pages at for further information.

NPR transcripts are created on a rush deadline by Verb8tm, Inc., an NPR contractor, and produced using a proprietary transcription process developed with NPR. This text may not be in its final form and may be updated or revised in the future. Accuracy and availability may vary. The authoritative record of NPR’s programming is the audio record.