Abe Davis: How Can Hidden Sounds Be Captured By Everyday Objects Computer scientist Abe Davis explains how you can turn a plant or a bag of chips into a microphone, and capturing the hidden sound vibrations on a high-speed camera.
NPR logo

Abe Davis: How Can Hidden Sounds Be Captured By Everyday Objects

  • Download
  • <iframe src="https://www.npr.org/player/embed/470514319/643404353" width="100%" height="290" frameborder="0" scrolling="no" title="NPR embedded audio player">
  • Transcript
Abe Davis: How Can Hidden Sounds Be Captured By Everyday Objects

Abe Davis: How Can Hidden Sounds Be Captured By Everyday Objects

  • Download
  • <iframe src="https://www.npr.org/player/embed/470514319/643404353" width="100%" height="290" frameborder="0" scrolling="no" title="NPR embedded audio player">
  • Transcript

GUY RAZ, HOST:

It's the TED Radio Hour from NPR. I'm Guy Raz. And on the show today, ideas about the things that are hidden all around us and how technology and sometimes just old-fashioned sleuthing is helping us to detect more and more of our invisible world. So to start the show, a little experiment with...

(BAG RUSTLING)

RAZ: A bag of chips. These ones are kettle cooked. And what this bag of chips is about to reveal is just how much of our world is hidden.

Does it work with Doritos?

ABE DAVIS: You know, I don't think we've tried Doritos. I suspect so.

RAZ: Lays, I'm assuming, will work fine.

DAVIS: We have done Lays, yeah.

RAZ: OK, good. So this is the guy who did the experiment with chips.

DAVIS: I'm Abe Davis. I'm a Ph.D. candidate at MIT.

RAZ: As for what Abe's studying, let's just say it has to do with hidden information. And before we tell you about Abe's chip experiment, there's something you need to know. And it's a basic truth of our physical world. And it's a truth that enables Abe to do his work.

DAVIS: It turns out that most of things most of the time are moving a little bit.

RAZ: To put this another way, almost everything around us is constantly, almost imperceptibly, vibrating but in a way that is totally invisible to us.

DAVIS: You know, our visual system is sort of tuned to pay attention to certain things. And likewise, it's tuned to ignore certain things.

RAZ: Yeah.

DAVIS: For instance, really tiny motions or really fast motions, these are things that we don't typically see with our eyes.

RAZ: Yeah, like a hummingbird's wings. Like, we don't see the flap, flap, flap, flap. We just see, like, a hummingbird body and just, like, this weird kind of thing around it.

DAVIS: Yeah.

RAZ: But just like we can use cameras to capture the movement of a hummingbird's wings, Abe Davis uses cameras to capture teeny tiny vibrations, vibrations that are normally hidden from us but that surround us all the time.

DAVIS: Caused by sound.

RAZ: Sound. So I'm sitting here in the studio, and I have this mug of tea. It's not moving. You're telling me that the tea is vibrating just from the sound of my voice?

DAVIS: Oh, yeah.

RAZ: Really?

DAVIS: Always.

RAZ: So what happens? What do you do with that movement? You actually point a camera at it?

DAVIS: Yeah. So that image of that mug of tea to us, if we look at even the image, probably won't look like it's moving at all because the actual motion is going to be a lot smaller than even a single pixel. But if it moves by just a fraction of a fraction of a pixel, then some of these pixels that see the mug will get just a little bit brighter and some will get just a little bit darker. And we can analyze that very subtle change and we can actually recover this information from it.

RAZ: And then translate that information into what?

DAVIS: Well, in this case, into sound.

(SOUNDBITE OF MUSIC)

RAZ: OK, people, this is huge. So listen closely because what Abe is saying here is that almost any object in the world around you can become a microphone. And with a pretty ordinary video camera and the right lighting, you could shoot silent video of an object and then analyze the tiny, imperceptible motion in the video and translate that motion into sound, into words even. Now, until recently, this was all theoretical, which is how Abe wound up standing in his lab one day, screaming at a bag of potato chips in a video he shared on the TED stage.

(SOUNDBITE OF TED TALK, VIDEO PLAYING)

DAVIS: Three, two, one, go. Mary had a little lamb, little lamb, little lamb.

(LAUGHTER)

DAVIS: So this experiment looks completely ridiculous. I mean, I'm screaming at a bag of chips, and we're blasting it with so much light, we literally melted the first bag we tried this on. But ridiculous as this experiment looks, it was actually really important because with the right algorithms, we can take this silent, seemingly still video and we can recover this sound.

(VIDEO PLAYING)

DAVIS: (Audio muffled) Mary had a little lamb, little lamb, little lamb.

(APPLAUSE)

DAVIS: And this was really significant because it was the first time we recovered intelligible human speech from silent video of an object. And so it gave us this point of reference. And gradually, we could start to modify the experiment using different objects or less light or quieter sounds. And that led to experiments like this one, where again, I'm going to speak to a bag of chips. But this time, we've moved our camera about 15 feet away, outside, behind a soundproof window. And the whole thing is lit by only natural sunlight. And this is what things sound like from inside, next to the bag of chips.

(VIDEO PLAYING)

DAVIS: Mary had a little lamb whose fleece was white as snow. And everywhere that Mary went, that lamb was sure to go.

And here's what we were able to recover from our silent video captured outside, behind that window.

(VIDEO PLAYING)

DAVIS: (Audio muffled) Mary had a little lamb whose fleece was white as snow. And everywhere that Mary went, that lamb was sure to go.

(APPLAUSE)

RAZ: OK, Abe, we should explain this to people listening because what you've done here is you've taken silent video of the chips and then, using a computer algorithm, you analyze the tiny vibrations in the bag. You extract the sound that caused those tiny vibrations in the first place, all from a bag of chips.

DAVIS: Yep, just a bag of chips.

RAZ: This is incredible. I mean, it's...

DAVIS: Enticing.

RAZ: Yeah, right?

DAVIS: Yeah.

RAZ: Why do you think that is?

DAVIS: I mean, speaking personally, it's exciting to sort of discover this whole new world of information.

RAZ: I mean, you could imagine a future where nothing is hidden anymore, where somebody just sitting, having a quiet conversation next to an empty bag of chips, is being listened to.

DAVIS: You know, it's interesting. It's easy for the mind to go there, right? It's easy to imagine, oh, man, this gives me access to this information, so therefore this information, you know, can no longer hide.

RAZ: Yeah.

DAVIS: But I think what's going to happen now that we know the information is there, the information is everywhere. And it's in such abundance. And we just - nobody has the resources to look at all of it. It's like telling people, I know that there's a needle in this haystack. That doesn't mean you're going to find it.

RAZ: But now we can find it. That's the thing. Like, there's no way that, like, intelligence agencies aren't excited by this. Like, there's no way. This is technology that could disrupt the way they operate.

DAVIS: (Laughter) I mean, I think that if you invested enough, then this does potentially open up some new doors, yeah. Is that - is that a sufficiently vague answer?

RAZ: Yeah.

DAVIS: I think a lot of the potential is in other kinds of information.

(SOUNDBITE OF TED TALK)

DAVIS: It's not hard to imagine how you might use this technology to spy on someone. But keep in mind that there's already a lot of very mature technology out there for surveillance. In fact, people have been using lasers to eavesdrop on objects from a distance for decades. But what's really new here, what's really different, is that now we have a way to picture the vibrations of an object, which gives us a new lens through which to look at the world. And we can use that lens to learn not just about forces like sound that cause an object to vibrate but also about the object itself. And you could imagine, for instance, looking at an old bridge and wondering what would happen - how would that bridge hold up if I were to drive my car across it? And, you know, that's a question that you probably want to answer before you start driving across that bridge. We've just started to scratch the surface of what you can do with this kind of imaging because it gives us a new way to capture our surroundings with common, accessible technology. And so looking to the future, it's going to be really exciting to explore what this can tell us about the world. Thank you.

(APPLAUSE)

RAZ: Abe Davis, he's a Ph.D. candidate at MIT. You can see his entire TED Talk - and you really should 'cause it is super visual and really cool - at ted.com.

Copyright © 2016 NPR. All rights reserved. Visit our website terms of use and permissions pages at www.npr.org for further information.

NPR transcripts are created on a rush deadline by Verb8tm, Inc., an NPR contractor, and produced using a proprietary transcription process developed with NPR. This text may not be in its final form and may be updated or revised in the future. Accuracy and availability may vary. The authoritative record of NPR’s programming is the audio record.