Episode 642: The Big Red Button : Planet Money The big question surrounding automation isn't just about economics or technology. It's also about psychology. How do designers make us comfortable with something that can be really scary?

Episode 642: The Big Red Button

  • Download
  • <iframe src="https://www.npr.org/player/embed/427467598/427549279" width="100%" height="290" frameborder="0" scrolling="no" title="NPR embedded audio player">
  • Transcript


Earlier this summer, I took a ride in Google's self-driving car.


HENN: Pre-planned route, OK.

We were driving through a construction site, and we were going past a guy holding one of those two-sided stop signs. It says slow on one side and stop on the other. And right as we went by, the guy flipped the sign around to read stop.

COMPUTER-GENERATED VOICE: Temporary stop sign.

HENN: The Google car slammed on its brakes all by itself.



So that was quite interesting.

HENN: The car startled us it stopped so quickly. None of us had seen that stop sign flip around when it happened.

I think I've blown through manual stop signs like that a thousand times.

This car sees more than we do. It doesn't get distracted. And even though this is kind of a low bar, I'm convinced it is a better driver than I am - already. But if a little drone car pulled up to my house tomorrow and there was no driver inside, no steering wheel, no brakes, would I put my children in it? The statistics, the research say this thing is safe, but it's still scary. Automation is all around us - you know, self-opening doors at the supermarket, elevators, autopilots on airplanes. And most of time, we never think about it. It makes our lives easier. It keeps us safe. But there is this moment - this moment when something new is introduced, when it first becomes automatic that's disorienting. It freaks us out.

Hello, and welcome to PLANET MONEY. I'm Steve Henn. And with me today is Katie Mingle. Hey.

MINGLE: Hey, Steve.

HENN: So Katie is a producer with one of our favorite podcasts - 99% Invisible. And since we're talking about design today, we thought this would be a great chance to work together.

MINGLE: Right because the big question with automation isn't just about technology or economics. It's about how designers make us comfortable with something that's really scary, like screaming down the highway at 65 with a robot at the wheel.

HENN: And when do we humans decide just to quit worrying and trust something that we can't control?

UNIDENTIFIED WOMAN: This episode of PLANET MONEY and the following message are made possible by Scion - proud to support PLANET MONEY, where you can learn about things like the economics of haggling. At Scion, their pure price offer means you don't need to haggle for price when you shop for a Scion car. Learn more at scion.com.

HENN: Before we get to the challenges facing the Google car, we thought we'd take a closer look at some of the automated things we take for granted. I mean, basically there are two ways to make someone really comfortable with a machine. You can either give people the power to turn it on and turn it off, to override the automation and give humans a measure of control.

MINGLE: Or you can make the technology seem soft and fuzzy and harmless, like convince them that everything is going to be OK. Like elevators - you don't even think about them anymore. You just get in them.

HENN: But if you think about this, you are walking into a tiny box, and it's controlled by God knows what. You hurdle up through space 10 stories, a hundred stories. And we just get in it.

MINGLE: It's kind of crazy. And in fact, it took a long time and a lot of work from elevator designers to get us to this blissful ignorance we have today.

HENN: When elevators were first invented, they were like cars - big things that could kill you. And so like cars, they needed a driver. There was an operator who would open and close the doors and use ropes or levers to guide the elevator to the proper floor. And like any humans, these elevator operators occasionally made mistakes.

LEE GRAY: There are accounts in the 19th century of very bad things happening that way.

HENN: This is Lee Gray. He wrote the definitive history of the passenger elevator.

GRAY: People have always hurried in large cities, certainly American cities in business settings. There were lots of times when, as a businessman, you might say, ah, the elevator's leaving and you hurry up and you sort of jump the gap into the car and land and off you go just before the operator shuts the door. And if you're athletic, it's all good. If you trip and fall and you're lying half in and half out of the car, the operator's not going to have time to stop. And that ended rather badly.

HENN: This was not the image the elevator industry wanted for its new transportation device. So they tried to make it safer - they automated it. They created automatic doors with safety bumpers, automatic stopping and then a driverless elevator. It was like the Google car of its time.

MINGLE: And people hated it.

GRAY: People walked in and looked and walked right back out and tried to find someone to say, where's the operator?

MINGLE: They thought something had gone horribly wrong and, what the hell were those buttons with numbers on them?

HENN: The elevator industry decided they had to convince people to rethink what an elevator was. It wasn't a gritty, dangerous transportation device. It was an elegant room that magically did your bidding. They put out ads featuring this new driverless elevator.

GRAY: And in many of those early ads, they showed children pushing the buttons and grandma riding on the elevator car, showing, well, how safe this is to put one in your house.

HENN: Kids and grandmas - and for the nervous rider there was a soothing voice that would pipe out of speakers when you walked inside.

GRAY: This is an automatic elevator. Please press the button for the floor you desire.

MINGLE: And it had other messages, like door closing and please do not block the door. And it directed you to the biggest calming device ever invented - a red button that said stop.

GRAY: It would say, you have pulled the emergency stop. If this is not an emergency, please push it back in. If this is an emergency, please use the phone.

HENN: But here's what they didn't decide to do. They didn't give any real control to the people inside. There wasn't a special lever in an elevator that someone could grab and take control and change where you were going. There wasn't an emergency override or a way to open the doors and the elevator while you were still moving. It was safer to just let you listen to the music and stare at that red button.

MINGLE: And it took people years to get used to it. But now an elevator feels really safe. So he said there are two ways to get people used to automation, two ways forward for the Google car. The soft and fuzzy way, the elevator way - just keep reassuring people until they're comfortable.

HENN: And then there's a second way, let the humans take control when they need to. And this is how automation works on airplanes. Most of the time you're in the air on a commercial jet plane, no human is really flying it. It's mostly controlled by computers. But we don't think about this or freak out about it because there are men and women in uniforms sitting up there waiting to take over in an emergency.

MINGLE: The automation in planes is extremely safe. And almost all the time, the plane flies better with this combination of humans and computers. But designers have to think really carefully about how the two work together because there are rare times when you might allow humans to make a choice and they make the wrong one.

HENN: And that's what happened to Air France Flight 447. This flight crashed into the Atlantic Ocean in 2009. And it highlights the stakes in getting automation right. This flight was flying from Rio to Paris.

WILLIAM LANGEWIESCHE: There was a lot of thunderstorm activity - quite typical, nothing really unusual about that.

MINGLE: William Langewiesche is a former pilot who studied this crash and wrote about in Vanity Fair. He says the main pilot was asleep and the two copilots were at the controls of the plane. But in a real sense, the plane was flying itself.

LANGEWIESCHE: Suddenly, they lost airspeed indications.

HENN: A pressure probe on the outside of the plane had iced over. The automation could no longer tell how fast the plane was going.

MINGLE: And so at this point, the system told the pilots that something was wrong.

LANGEWIESCHE: And autopilot disconnected with a bell that went off.

HENN: But the automated system didn't completely turn itself off. The computer was still there, interpreting and limiting what the pilots did at the controls. But some of the built-in safety features dropped away. The computerized system handed more control back to the human pilots.

MINGLE: But that was no big deal, William says. The plane was still flying straight and steady. It just wasn't getting all the data it normally did. So it gave the human pilots a heads up then gave them a bit more control over the plane.

LANGEWIESCHE: It was what happened next that caused the problem.

MINGLE: The copilot in the right seat put his hand on the control stick, a little joystick-like thing to his right.

LANGEWIESCHE: And he pulled it back.

HENN: Not just a little - he pulled it three quarters of the way back.

LANGEWIESCHE: Why he did that is a major question.

MINGLE: Maybe he was startled or reacting to some turbulence.

HENN: Or perhaps he did it because he expected the automated system to smooth out his commands and keep him out of trouble.

LANGEWIESCHE: The airplane pulled its nose up and started to soar upward.

MINGLE: This is bad. This can cause a plane to lose speed, to lose lift. And when the automation is running normally, the plane physically will not let this happen. But remember, the plane had handed back control to the people in the cockpit. All the plane could do at this point was warn the pilots.

LANGEWIESCHE: It said stall, stall, stall, stall.

MINGLE: It would have sounded like this.


AUTOMATED VOICE: Stall, stall.


AUTOMATED VOICE: Stall, stall.


MINGLE: The plane knew what was wrong, but the crew didn't - or at least they never figured out how to correct it. Somewhere between the automation and the human pilots was this horrible gap. And in the final minute of the flight, the pilots knew they were going to crash. Four minutes and 20 seconds after the incident started...

LANGEWIESCHE: They pancaked into the water at a very high descent rate. And of course it killed everyone instantly.

MINGLE: Two-hundred-and-twenty-eight people.

HENN: And here's the thing in this case. If the crew had done just one simple thing when all of this had started, they would have been fine.

MINGLE: All the pilots needed to do to avoid this tragedy was nothing.

HENN: Not touch a thing.

LANGEWIESCHE: The airplane was not upset. Had they done nothing, they would've done exactly what they needed to do - nothing.

HENN: The plane would have flown on steadily. The automated system would have reengaged. And it most likely would've been another unremarkable, forgettable triumph of this machine. If you give people control, they'll use it. And that means sometimes they'll make mistakes, deadly mistakes. But if you take control away, it tends to make us all uneasy, irrationally anxious.

MINGLE: So this is the really high-stakes dilemma facing anyone who wants to make a driverless car. You have to make people comfortable enough to get in the car. You have to give them a sense of control. But then you have to keep them from doing something stupid.

HENN: Essentially, Google is asking should its self-driving car be more like an elevator or more like an airplane. And here's the guy who gets to decide.

CHRIS URMSON: I'm Chris Urmson. I lead the self-driving car project here at Google.

HENN: When Chris launched this project years ago, the assumption was that drivers would sit and let the Google car drive. But they'd be paying attention to the road, ready to take control when they needed to, just like airline pilots.

MINGLE: And in fact, they let Google employees borrow the self-driving cars to commute home from work.

HENN: And there is a little camera inside. So Chris could see if they were really paying attention. But he was amazed by what that camera recorded.

URMSON: We had one guy who decided to plug his laptop in by digging in the back, pulling out the power cord, pulling out the laptop, you know, all while moving down the freeway.

HENN: It was pretty easy to imagine the worst-case scenario - a driver taking a nap. The Google car is doing fine. The driver wakes up all of a sudden, sees something and panics, hits the brake or swerves. So Chris made a call. They had to make the Google car more like an elevator - no steering wheel at all.

URMSON: If you have a steering wheel there, there's kind of an implicit expectation that you're going to do something with it, right? Why put the steering wheel in the car if you don't - if you're not supposed to use it? And so, you know, you get to remove a part of the system that you can't design.

MINGLE: Chris is not an idiot. He knows that the idea of a car without a steering wheel is going to freak people out. And so like the elevator, Google is going the soft and fuzzy route. They're using design to make the car as nonthreatening as they can.

HENN: All right. It's a tiny, little car. You know, from the outside, it's like - it looks like someone stuck a Bug in a Shrinky Dink.

URMSON: (Laughter) I've never heard of it described like that before. But, you know, inside, it feels pretty spacious.

HENN: It's kind of adorable, all round and soft. It has a tiny, little, black radar mounted to the hood that looks like a button nose. And everything is designed to keep you calm. There's a rearview mirror, although there's no real reason for it. There's a little screen inside that shows you what the car can see. And just like in an elevator in the '50s, there's a reassuring automated voice.


MINGLE: No elevator music, though, which is probably good.

HENN: Now, you're not totally helpless when you're cruising around in this thing. You can use an app on your phone to tell it where to go. And there are a couple buttons inside. You know, there are two for the windows. There's one to pull over. And there is, under a Plexiglas shield, the big, red button that says stop.

URMSON: Just like an elevator. I've never pressed the red button in an elevator, but it's kind of comforting to know it's there.

HENN: And Chris's ideal world, that is all anyone would ever need. Now, we should say that this is the ideal prototype by Google - just a moving box with a big, red button. But this still makes some people nervous, specifically California regulators and politicians. So in order to test this car on the roads and highways around Silicon Valley, Google has had to add back a temporary steering wheel and a brake.

MINGLE: Chris hopes that by the time his oldest son turns 16, in four-and-a-half years, the steering wheel will be a distant memory - kind of like those old elevator levers.

HENN: For more about the lessons of Air France Flight 447 and the paradox of automation, you should really check out 99% Invisible. It's a great podcast. You can find it where you listen to podcasts or at 99pi.org. As always, you can email us at planetmoney@npr.org or tweet at us at @planetmoney.

MINGLE: Our episode today was produced by Nadia Wilson and Francis Harlow. Special thanks to William Langewiesche for answering my countless questions about Flight 447.

HENN: I also wanted to thank Sam Greenspan at 99pi for all of his help with reporting on the Google car. And if you're looking for more shows to listen to, check out Snap Judgment with Glynn Washington. It's an hour-long show about real people with real stories. You can find at npr.org/podcasts and on the NPR One app. I'm Steve Henn.

MINGLE: And I'm Katie Mingle.

HENN: Thanks for listening.


BROKEN BELLS: (Singing) Well, you got to give it up, give it up, give it up, oh, give it up, give it up, oh, give it up, give it up. You lost control 'cause nothing stays around too long. Nothing stays around too long. Nothing stays around too long. You lost control.

Copyright © 2015 NPR. All rights reserved. Visit our website terms of use and permissions pages at www.npr.org for further information.

NPR transcripts are created on a rush deadline by an NPR contractor. This text may not be in its final form and may be updated or revised in the future. Accuracy and availability may vary. The authoritative record of NPR’s programming is the audio record.