#850: The Fake Review Hunter Fake product reviews are wrecking the internet. But help is on the way: From a bodybuilding fake review hunter.
NPR logo

#850: The Fake Review Hunter

  • Download
  • <iframe src="https://www.npr.org/player/embed/623988370/624037735" width="100%" height="290" frameborder="0" scrolling="no" title="NPR embedded audio player">
  • Transcript
#850: The Fake Review Hunter

#850: The Fake Review Hunter

  • Download
  • <iframe src="https://www.npr.org/player/embed/623988370/624037735" width="100%" height="290" frameborder="0" scrolling="no" title="NPR embedded audio player">
  • Transcript

KENNY MALONE, HOST:

Hello, and welcome to PLANET MONEY. I'm Kenny Malone.

HILKE SCHELLMANN, HOST:

And I'm Hilke Schellmann.

MALONE: Hilke Schellmann - journalism professor at NYU, documentary filmmaker. And we've asked you here to join us today because I feel like it's fair to say you've become mildly obsessed with the topic of today's show.

SCHELLMANN: There might be some truth to that. Look, I'm a journalism professor. I think a lot about - I worry a lot about fake news. And more generally, I'm really concerned about trust on the Internet.

MALONE: And Hilke came to us and said, before there was fake news, before there was the possibility of, like, fake video and fake audio, the world was dealing with a different fake problem.

SCHELLMANN: Yes. Fake online reviews. People have been posting dishonest online fake reviews for years.

MALONE: Yeah. And this is becoming such a huge problem because, like, you know, I go online to buy a hoverboard, and I don't know if this hoverboard has five stars because it is awesome, and I'm going to cruise around town looking cool, or if it has five stars because someone is making up those reviews and, in fact, when I take it home, it's going to set my house on fire.

But Hilke is here with a glimmer of hope for us.

SCHELLMANN: Yes. And the glimmer of hope is coming from a totally unexpected place - the online bodybuilding community.

MALONE: Today on the show, the story of one weightlifter battling fake reviews on his own website and what happened when he took on Amazon.com.

A couple of things to know about Tommy Noonan. He lives in Reno, Nev. He likes to juggle weird things.

TOMMY NOONAN: So just, you know, I'll be, like, in the grocery store, and I'll have, like, an avocado and a onion and a bottle of water.

MALONE: Whoa, that's high stakes. You drop an avocado, you're buying that thing.

NOONAN: Yeah.

MALONE: Yeah.

SCHELLMANN: The story of how Tommy started fighting fake reviews begins around 2006.

MALONE: Tommy was in college at Cal Poly, San Luis Obispo at the time. He was 19, and he was a bit of a gym rat. At his peak, he could bench press around 240 pounds.

SCHELLMANN: One day, Tommy was hanging out with a friend.

NOONAN: And he says, hey, do you want the rest of this pre-workout?

MALONE: Pre-workout - that's like a powdered stimulant you're supposed to take before you hit the gym.

NOONAN: And it just gives you crazy energy, and you can lift more and longer and harder. You know, at least those are the claims.

MALONE: Just to be clear, this is not a steroid, in case that's what you're thinking. Tommy is very against using those.

SCHELLMANN: Anyway, Tommy's friend is really not happy with the stuff that he purchased. And he says to Tommy...

NOONAN: I paid 60 bucks for it, and it gives me such bad stomach issues, it's ruining my workouts. It's, you know, counterproductive.

MALONE: If you're not familiar with the supplement world, it's not the most tightly regulated world. And on top of that, back in 2006, more and more folks were starting to buy things online, like supplements.

NOONAN: The gears in my head started turning, and I started looking up supplement reviews, and I couldn't find anything that was legit. I mean, it was all just written by the manufacturer. They could just...

SCHELLMANN: Tommy saw an opportunity. He could build an online review site where bodybuilders give honest feedback about these supplements they're putting into their bodies.

MALONE: And so Tommy got to work. He taught himself some computer coding, and he created what he would eventually call SupplementReviews.com. Before long, super-chiseled dudes were logging on to leave their super-honest reviews.

And, look, I know not all bodybuilders sound this way, but as I read through these reviews, I could not not hear them in the voice of The Situation from "Jersey Shore."

STACEY VANEK SMITH, BYLINE: (Imitating New Jersey accent) Just started using this product. Got way huge. Can't complain. Overall rating - 8.

MALONE: Yeah, you know, we couldn't get the real Situation. But filling in is our very own Stacey Vanek Smith.

VANEK SMITH: Very swoll Stacey Vanek Smith.

MALONE: (Laughter) Yeah.

VANEK SMITH: No one knows this. You can't tell from my voice, but I'm - you know, I'm huge.

MALONE: Stacey, how much can you bench press, would you say?

VANEK SMITH: Three-hundred pounds seems plausible. Like, I could probably bench press you, Kenny.

MALONE: Yeah, OK. Anyway, Tommy's site became a place where bodybuilders could share all kinds of supplement experiences, including great successes.

VANEK SMITH: This literally gives me skin-splitting pumps - no lie. I have stretch marks all over, and incredible gains. Overall rating - 10.

SCHELLMANN: Maybe more importantly, the site was also a place where bodybuilders could tell each other about problematic products.

VANEK SMITH: (Imitating New Jersey accent) I am sure that this will be the last day I take this [expletive]. I have heart palpitations, and I'm not sure if tomorrow I will wake up. Overall rating - 5.

MALONE: Overall rating - 5?

VANEK SMITH: (Imitating New Jersey accent) I don't know. I mean, it tasted pretty good.

MALONE: (Laughter) OK. Well, not everyone thought that same powder tasted good.

VANEK SMITH: (Imitating New Jersey accent) This stuff gave me a strong urge to vomit - some sort of strange circus going on in my stomach. Overall rating - 1.

MALONE: Yeah. There's a lot of talk of stomach issues.

SCHELLMANN: Yeah. And bloating.

MALONE: Is bloating a big problem?

NOONAN: Yeah, yeah. Yeah.

MALONE: Bodybuilders were writing reviews, and Tommy got a referral fee when customers found products they liked through his site.

SCHELLMANN: It was going so well for Tommy that he decided to just run the site full-time.

NOONAN: Then, as it grew, it started climbing the ranks in Google. You know, ranking number one for the term supplement reviews.

MALONE: It must've been a big day the day that happened.

NOONAN: Yeah, it was.

MALONE: But as the website became more and more prominent, supplement companies became more and more aware of Tommy's site and how their products were being reviewed on his site.

SCHELLMANN: Tommy says at the time, he was maybe getting, like, 20 reviews a day. Look, he wasn't reading every single one of them, but he was reading a lot of them. And he started to see these suspiciously positive reviews creeping in.

MALONE: This is one, for example.

VANEK SMITH: (Imitating New Jersey accent) University-proven studies - anyone can test a product, but to have proof that it works is key. I absolutely enjoy the taste, especially the grape. I also really enjoyed the unflavored and the tropical flavored...

NOONAN: And it almost reads like a sales pitch. And you kind of read it, and you think, all right. Was it a rep that wrote this, or was it someone that just drank the Kool-Aid of the marketing campaign?

VANEK SMITH: (Imitating New Jersey accent) I never get bloated from it, despite I sometimes take 2 times as much as the serving during and after my workout. Believe the hype, as this product truly delivers upon what it promises. The taste of it is very good as well, if mixed properly.

MALONE: So for Tommy, these kinds of reviews were sometimes hard to deal with because it sounded like a person from the company had just come in and written this. But what if it was someone who had tried the product and really did believe the hype?

SCHELLMANN: And that's the problem with online reviews in a nutshell. It's almost impossible to tell from the content alone if the review's fake or not.

MALONE: But Tommy knew if a company was coming in and pretending to be a regular bodybuilder who happened to love their product, the entire idea of his website would fall apart.

NOONAN: So I would really have to do a bunch of investigative work on each one to really see where it originated and whether it was from the company or not.

SCHELLMANN: So Tommy goes to work.

NOONAN: Basically, the first thing I check out is the email address that they used to register for the site.

MALONE: Did - were they dumb enough to use the company email address?

NOONAN: That happens sometimes (laughter).

SCHELLMANN: No.

NOONAN: Yes.

SCHELLMANN: It didn't take Tommy long to realize that these companies were coming in. And basically, they were just writing press releases.

MALONE: And it kept happening. These fake reviews were flooding in and starting to overwhelm the real reviews.

NOONAN: And I just thought, it's not fair that, you know, people that actually leave feedback that is, you know, real and honest are getting silenced.

SCHELLMANN: Tommy took all this time and created this place for honest feedback. And now, it was just being turned into the exact opposite.

NOONAN: Yeah, yeah. Well, it's my - you know, it was my work. It was my life's work. And it's just - it's painful to see somebody just disrespect it like that in the name of money.

MALONE: Tommy's site was a pretty small site in the grand scheme of things. But he started to wonder, like, if this is happening to SupplementReviews.com, I wonder how big of a problem this is at other sites.

SCHELLMANN: So he went into the belly of the beast of online shopping, Amazon.com.

NOONAN: And I looked at this testosterone booster that had 560 reviews.

MALONE: It was actually 580 reviews. Tommy starts to scan through a little bit, and his fake review Spidey senses are going bonkers.

SCHELLMANN: And he's like, OK, I'm going to inspect every single one of these reviews.

MALONE: It's going to be a lot of work, and I'm going to need my special Pandora station for this.

NOONAN: Based off of a song from - I call them deadmau5, but other people call them Dead Mouse (ph).

MALONE: Sure.

NOONAN: It's called - "Take Care Of The Proper Paperwork" is the name of the song.

(SOUNDBITE OF DEADMAU5'S "TAKE CARE OF THE PROPER PAPERWORK")

MALONE: This certainly must be in the Top 10 songs named after paperwork, don't you think?

SCHELLMANN: Yes. There's a mind-numbing sound of bureaucracy.

NOONAN: That is pretty accurate.

MALONE: So Tommy is going through every single review and essentially asking himself, how do I know I can trust this?

NOONAN: I've got an Excel spreadsheet open, and I got Amazon, and I've got, like, 500 tabs open on Amazon. And I'm sitting there, just look at the next review, tally, update it. Look at the next review, tally, update it.

MALONE: And as he does this, he starts to learn the tricks of online fake reviewers. Some of them are not so tricky. For example, one review, he noticed there was a before and an after picture. Before - super skinny guy, after - super ripped.

SCHELLMANN: Tommy does a reverse image search, and he finds the earliest example of the photo on Reddit.

NOONAN: And I message that user on Reddit, and I said, hey, did you also use this image for a review? And they said, no.

MALONE: Oh.

NOONAN: So...

SCHELLMANN: Busted.

NOONAN: Yeah.

SCHELLMANN: A stolen before-and-after picture - obviously fake.

MALONE: As he investigates, Tommy's also starting to notice that some suspicious reviewers have almost like an M.O. that he can identify. And he starts naming these different types of questionable reviewers.

SCHELLMANN: One category are the brand monogamists. So these are people that only review products from one company over and over again.

MALONE: Sure. Maybe that person really likes that company or maybe that person works for that company. Another fake reviewer profile - the one-hit wonders. These are people who review one thing and then never review anything again.

NOONAN: I mean, obviously, there are people that just review one thing. That's totally a normal thing that happens. But if it happens a lot and if those one-hit wonders are giving the product a five-star across the board, then, you know, again, maybe there is something unnatural about that.

MALONE: Yeah, like if this testosterone booster had a bunch of one-hit wonders that were all really positive, that is pretty unnatural. But the most unnatural pattern Tommy found was this suspicious flood of positive reviews around the exact same time.

NOONAN: You know, it was, like, three days of, you know, this super high volume where I think, like, 30 percent of the reviews were submitted.

MALONE: And that's fishy because it is way more reviews than any other three-a-day stretch, and they're really positive reviews. So it looks like maybe, Tommy says, somebody paid for a bunch of fake reviews around that time.

SCHELLMANN: The way this sometimes works is that a seller might want to boost their rankings. They want to sell more product, and they need more positive reviews. So there are sites where they can buy these fake reviews, but then those positive reviews all post around the same time.

MALONE: And so for Tommy, after reading through all those reviews, learning the tricks of the fake review trade, what percentage of those 580 reviews did he think were tainted or biased in some way?

NOONAN: Probably all of them (laughter).

MALONE: Like nearly 100 percent or maybe 100 percent.

NOONAN: Yeah, or something is wrong with them.

SCHELLMANN: So Tommy was on this mission to clean up the fake reviews on his own website. And then he looked at Amazon, and he finds out even Amazon has this gigantic problem of B.S. reviews.

MALONE: Not only that - if he ever wants to go on Amazon and buy something again, he's realizing he's going to have to spend hours going review by review to figure out what is even trustworthy. Or...

NOONAN: I could write a computer program to do all this in seconds - access these pages automatically, pull the data and give me, you know, a much better picture of the reviews. And I'm sitting there thinking, like, if I wrote that computer program to do all this in seconds, I could point it at any product.

MALONE: And so Tommy Noonan went back to work and, in essence, he coded a version of himself, an Internet review detective that can help any of us pull up an item on Amazon and cut through all the B.S. reviews to find just the good ones. He started a new website that he called reviewmeta.com.

SCHELLMANN: And after the break, we'll go shopping with it.

MALONE: All right, Hilke, show me how ReviewMeta can help me get over my fake review anxiety.

SCHELLMANN: Kenny, what do you need?

MALONE: I just was on vacation, and my phone died four times because I do not - I actually do not have any iPhone cables anymore.

SCHELLMANN: Got it. So you go to Amazon, and you put in iPhone cables. Yeah, so we have over 20,000 results for iPhone cables. That's a lot.

MALONE: Yes. This is where the anxiety starts.

SCHELLMANN: So we're going to click on the one that we like.

MALONE: All right. So this one has 177 reviews, four stars. Like, that looks great on the surface.

SCHELLMANN: And then you take the URL, copy it, and then you paste it into ReviewMeta.

MALONE: OK.

SCHELLMANN: And then I hit run report, and it runs a report.

MALONE: All right. And it says...

SCHELLMANN: Hit go.

MALONE: Computing, computing, bleep, bloop, bleep, bloop - whatever. It's going to take a while. It's reading, like, 177 reviews. It would take me, like, five hours. At any rate, eventually, it spits out a report, and we got Tommy Noonan on the line to walk us through a few of the things his website flagged about these iPhone cable reviews.

NOONAN: OK. So for this product, we see that over the course of five days, this product got 78 percent of their reviews. And on top of that, the reviews from those high-volume days gave the product a 4.9-star rating on average.

SCHELLMANN: These reviews over those five days - pretty suspicious, especially when you look at the reviews from those five days compared to the average reviews. They're way more positive.

MALONE: Tommy says the reviews for these cables raised a few other red flags as well. There was one issue with words that were being used over and over again in the reviews.

NOONAN: OK, so the phrase repetition - basically, what we do is we build a gigantic list of all of the phrases that are used in the reviews that are, you know, three words or more. And then we go...

MALONE: The algorithm is able to filter out normal phrases that would show up all the time. For example, it was the. And this way, the algorithm can look at weird phrases that you would not expect to be repeated. For example...

NOONAN: The cord is very tough and doesn't bend like other cords do, but that's probably a good thing. That's not, you know - that - you shouldn't see that in more than one review. But for here, we see it in three reviews.

MALONE: Yeah. And in seven reviews, someone wrote, it's easy to get it back onto the cord and use again. I don't know what it means, but seven people feel that way.

NOONAN: Well, seven people also say they performed their duty with honor and respect. And their is the incorrect version.

(LAUGHTER)

NOONAN: Yeah. So...

SCHELLMANN: Tommy's software ran about a dozen different tests on iPhone cable reviews. And his reports spit out an adjusted rating.

NOONAN: So it went from a 4 star out of 5, with 177 reviews on Amazon. And the adjusted rating on ReviewMeta is 1.7 stars after 10 reviews.

SCHELLMANN: Which is another way of saying Tommy's site only trusted 6 percent of the reviews. And the trustworthy reviews say, actually, these cables are crap.

MALONE: Tommy says, generally speaking, cables like this are exactly the kind of product that winds up cluttered with suspicious reviews.

NOONAN: Anything with, like, low manufacturing costs and high margins.

MALONE: So like what?

NOONAN: Yeah - iPhone cables, backup batteries, screen protectors, phone cases, weight loss pills, testosterone...

SCHELLMANN: We, of course, reached out to Amazon about this.

MALONE: And they declined to go on tape, but they are familiar with Tommy's website. They sent us an extensive statement about it.

VANEK SMITH: (Imitating New Jersey accent, reading) ReviewMeta itself admits that they do not have the data to...

MALONE: No, no, no. Stacey - Stacey, you could do this one as, like, a, you know, non-weightlifter.

VANEK SMITH: Oh, OK. (Reading) ReviewMeta itself admits that they do not have the data to know or even say with high confidence whether a review is fake. According to...

MALONE: What Amazon is saying is not necessarily different from what Tommy told us about his own website. He totally admits that his site is looking for patterns of unusual behavior, and there's a possibility that real reviews get flagged as unnatural.

SCHELLMANN: For Amazon, that is a problem.

VANEK SMITH: (Reading) We have hundreds of millions of reviews on Amazon. And of that total, less than 1 percent are inauthentic.

SCHELLMANN: Amazon also told us that it's investing in its own system to catch fake reviewers.

MALONE: In other words, Amazon is saying, there's not a problem, but we're working on it. And this makes me, like, wonder a little bit, how much should I trust that big sites like Amazon are actually going to go out of their way to fix this?

SCHELLMANN: So it's complicated - right? - because these e-commerce sites - they make money when they sell more products.

MALONE: Right. Like, it doesn't matter if it's a positive fake review. If someone buys a product, that's good for the site.

SCHELLMANN: But there might come a time where there is just too many fake reviews.

MALONE: Sure. Like, there's a tipping point where enough trust is lost that we all just give up...

SCHELLMANN: Yeah.

MALONE: ...On Internet shopping.

SCHELLMANN: Yeah. And we'll go back to France (ph) - to our friends.

MALONE: France?

SCHELLMANN: To our friends. We will go back to our friends...

MALONE: Oh, friends.

SCHELLMANN: ...For recommendations.

MALONE: I was like, you want to go to France? Great.

SCHELLMANN: (Laughter) Well, yes. There is a point where it just ruins it for all of us.

MALONE: And we have to go back to talking to our friends again. Bleh.

SCHELLMANN: Or you have to call your mom.

MALONE: For shopping advice?

SCHELLMANN: Yes. What toothpaste you should use.

MALONE: That is a mom question.

(SOUNDBITE OF CLAUDE PELOUSE AND LE FAT CLUB'S "CLUB SODA")

MALONE: Today's show was produced by Nick Fountain. Our editor is Bryant Urstadt. Our supervising producer is Alex Goldmark, who told me to tell you we could use reviews, too. PLANET MONEY likes reviews.

SCHELLMANN: But please don't fake it.

MALONE: No. Real reviews only.

SCHELLMANN: And special thanks to Fakespot, which is actually another similar software to ReviewMeta, which you can use to spot fake reviews. They have been really helpful.

MALONE: I'm Kenny Malone.

SCHELLMANN: And I'm Hilke Schellmann. Thanks for listening.

(SOUNDBITE OF CLAUDE PELOUSE AND LE FAT CLUB'S "CLUB SODA")

Copyright © 2018 NPR. All rights reserved. Visit our website terms of use and permissions pages at www.npr.org for further information.

NPR transcripts are created on a rush deadline by Verb8tm, Inc., an NPR contractor, and produced using a proprietary transcription process developed with NPR. This text may not be in its final form and may be updated or revised in the future. Accuracy and availability may vary. The authoritative record of NPR’s programming is the audio record.