RENEE MONTAGNE, Host:
Every time we wiggle a toe or form a thought, our brain is sending electrical signals. These signals literally broadcast what the brain is doing at any moment. Now scientists are figuring out how to eavesdrop on this broadcast, using a computer connected directly to the brain. As NPR's Jon Hamilton reports, the approach is revealing new information about how we control our muscles, listen to sounds, and communicate.
JON HAMILTON: Gerwin Schalk is a researcher at the Wadsworth Center in Albany, New York, who helped design that software. He says the system can decode brain signals so precisely that it can tell whether you're thinking of the word bat or cat.
GERWIN SCHALK: This is both very exciting and somewhat frightening at the same time because it really goes pretty close to what people used to call mind reading.
HAMILTON: About a decade ago, a small group of scientists wondered whether the technology might be able to do something a lot more sophisticated.
SCHALK: Corey, can you close the hand for me, please? That's great, thank you.
HAMILTON: Schalk shows me a video. In it he's working with a young man who is staring intently at a computer screen. The video was shot by the American Museum of Natural History as part of an exhibit on the brain.
SCHALK: Can you open it back up? OK. Keep it open.
HAMILTON: What's striking about this scene is that the young man's own hand isn't moving. What is moving is a virtual hand on the computer screen. The man is clenching and unclenching this virtual fist on command using only his thoughts.
SCHALK: Close it, close it, close it. Great job.
HAMILTON: Schalk says the experiment also shows how much of the brain gets involved in things we take for granted.
SCHALK: Even for simple functions such as opening and closing the hand, for example, there are many, many areas that contribute to the movement.
HAMILTON: Bionic arms are just one likely use for ECoG though. Eric Leuthardt is a brain surgeon at Washington University in St. Louis who has worked closely with Schalk. He says the technology has proved to be far more powerful and versatile than anyone expected.
ERIC LEUTHARDT: I'd say every couple weeks we find something that really kind of makes us scratch our head and say, wow, that's pretty neat.
HAMILTON: Another approach is to put electrodes on the scalp, but the signals are much less clear because they pass through skin and bone. Leuthardt says ECoG involves surgery but not going into the brain itself.
LEUTHARDT: We basically do what's called a craniotomy, where we make an incision in the scalp, make a large window of(ph) bone. We put the array over the brain. We close everything back up, and there's wires exciting through the skull, through the scalp, which then get directly connected to a computer.
HAMILTON: Schalk shows some of what ECoG can do in his lab. No animals or test tubes here...
(SOUNDBITE OF MUSIC)
HAMILTON: ...but there are plenty of computers, including one playing music.
(SOUNDBITE OF MUSIC)
SCHALK: The music is "The Wall" by Pink Floyd.
(SOUNDBITE OF MUSIC)
HAMILTON: Schalk is showing me the results of experiments he did using ECoG to monitor people as they listened. He points toward two waveforms on the computer screen. One shows the mountains and valleys that represent changes in the music volume. The second waveform looks almost the same, but it represents the electrical signals generated by the brain in response to the music.
SCHALK: This is the actual loudness in the music, the decoded loudness in the music. OK? A very close correlation between the actual loudness in the music - that is, just playing right now - and the music, the intensity of the music that we're decoding or inferring from the person's brain. I mean, isn't that pretty awesome?
HAMILTON: Schalk says the brain signal is so distinctive you could almost recognize the music from the waveform alone. What's really awesome, though, is the next part of the experiment.
SCHALK: So what we did was we played some music to the subjects and then we played the same music again, except that now every about 10 seconds it was a one-second silence period.
(SOUNDBITE OF SONG FOLLOWED BY SILENCE)
HAMILTON: Unidentified Man: (Singing) ...all in all it was just a brick in the wall...
(SOUNDBITE OF SONG)
HAMILTON: Schalk says what we're seeing is the brain's attempt to fill in the missing sounds.
SCHALK: The brain basically tells us a lot of information about the music in the times when there is really no music. It's not played.
HAMILTON: Schalk says ECoG is also revealing things about how the brain creates speech. He and other researchers are using the technology to watch the brains of people as they speak out loud and also as they say the words silently to themselves.
SCHALK: One of the surprising initial findings coming out of that research was that actual and imagined speech is very, very different.
HAMILTON: Schalk says when your brain wants you to say a word, it produces two sets of signals. One has to do with moving your muscles.
SCHALK: And that makes sense. First, you have to move your mouth around and your vocal tract around so as to produce a particular type of verbal output.
HAMILTON: And there are also signals coming from the auditory system. On the other hand, when a person simply thinks of a word instead of saying it, there are no muscle signals, just the activity in the parts of the brain involved in listening.
SCHALK: And that seems to suggest that what imagined speech actually really is, it's more like internally listening to your own voice.
HAMILTON: Jon Hamilton, NPR News.
MONTAGNE: And you can watch videos of these experiments, including a patient controlling a virtual hand with his mind, at our website, NPR.org.
NPR transcripts are created on a rush deadline by Verb8tm, Inc., an NPR contractor, and produced using a proprietary transcription process developed with NPR. This text may not be in its final form and may be updated or revised in the future. Accuracy and availability may vary. The authoritative record of NPR’s programming is the audio record.