RENEE MONTAGNE, host:
Every time we wiggle a toe or form a thought, our brain is sending electrical signals. These signals literally broadcast what the brain is doing at any moment. Now scientists are figuring out how to eavesdrop on this broadcast, using a computer connected directly to the brain. As NPR's Jon Hamilton reports, the approach is revealing new information about how we control our muscles, listen to sounds, and communicate.
JON HAMILTON: What's making all this possible is actually an old technology. It's called electrocorticography, or ECoG. Surgeons place electrodes on the surface of the brain. Those electrodes detect the brain's electrical signals and send them to a computer running special software - that's the new part.
Gerwin Schalk is a researcher at the Wadsworth Center in Albany, New York, who helped design that software. He says the system can decode brain signals so precisely that it can tell whether you're thinking of the word bat or cat.
Dr. GERWIN SCHALK (Researcher, Wadsworth Center): This is both very exciting and somewhat frightening at the same time because it really goes pretty close to what people used to call mind reading.
HAMILTON: ECoG, minus the software, has been around since the 1950s. Doctors use it to figure out which part of the brain is causing seizures in people with severe epilepsy.
About a decade ago, a small group of scientists wondered whether the technology might be able to do something a lot more sophisticated.
Mr. SCHALK: Corey, can you close the hand for me, please? That's great, thank you.
HAMILTON: Perhaps ECoG could pick out the brain signals we use to control our fingers, or even the signals our brain sends out when we just think about moving our fingers.
Schalk shows me a video. In it he's working with a young man who is staring intently at a computer screen. The video was shot by the American Museum of Natural History as part of an exhibit on the brain.
Mr. SCHALK: Can you open it back up? OK. Keep it open.
HAMILTON: What's striking about this scene is that the young man's own hand isn't moving. What is moving is a virtual hand on the computer screen. The man is clenching and unclenching this virtual fist on command using only his thoughts.
Mr. SCHALK: Close it, close it, close it. Great job.
HAMILTON: Like all volunteers in ECoG experiments so far, this patient has severe epilepsy. He took part in the experiment while doctors were using ECoG to find the source of his seizures. But the experiment shows how the technology could help a very different sort of patient: someone paralyzed by a spinal injury or Lou Gehrig's disease. ECoG could allow someone like that to operate a robotic arm with just their thoughts.
Schalk says the experiment also shows how much of the brain gets involved in things we take for granted.
Mr. SCHALK: Even for simple functions such as opening and closing the hand, for example, there are many, many areas that contribute to the movement.
HAMILTON: Bionic arms are just one likely use for ECoG though. Eric Leuthardt is a brain surgeon at Washington University in St. Louis who has worked closely with Schalk. He says the technology has proved to be far more powerful and versatile than anyone expected.
Dr. ERIC LEUTHARDT (Washington University): I'd say every couple weeks we find something that really kind of makes us scratch our head and say, wow, that's pretty neat.
HAMILTON: Leuthardt says ECoG hits a sweet spot between two competing approaches to detecting brain signals. One of these requires placing electrodes deep in the brain. That allows scientists to monitor individual brain cells with great precision. But they can't monitor very many brain cells at the same time.
Another approach is to put electrodes on the scalp, but the signals are much less clear because they pass through skin and bone. Leuthardt says ECoG involves surgery but not going into the brain itself.
Mr. LEUTHARDT: We basically do what's called a craniotomy, where we make an incision in the scalp, make a large window of(ph) bone. We put the array over the brain. We close everything back up, and there's wires exciting through the skull, through the scalp, which then get directly connected to a computer.
HAMILTON: In theory, researchers could receive signals from hundreds or even thousands of electrodes. So far they haven't gone beyond dozens, yet the results have been spectacular.
Schalk shows some of what ECoG can do in his lab. No animals or test tubes here...
(Soundbite of music)
HAMILTON: ...but there are plenty of computers, including one playing music.
(Soundbite of music)
Mr. SCHALK: The music is "The Wall" by Pink Floyd.
(Soundbite of music)
HAMILTON: Schalk is showing me the results of experiments he did using ECoG to monitor people as they listened. He points toward two waveforms on the computer screen. One shows the mountains and valleys that represent changes in the music volume. The second waveform looks almost the same, but it represents the electrical signals generated by the brain in response to the music.
Mr. SCHALK: This is the actual loudness in the music, the decoded loudness in the music. OK? A very close correlation between the actual loudness in the music - that is, just playing right now - and the music, the intensity of the music that we're decoding or inferring from the person's brain. I mean, isn't that pretty awesome?
HAMILTON: Schalk says the brain signal is so distinctive you could almost recognize the music from the waveform alone. What's really awesome, though, is the next part of the experiment.
Mr. SCHALK: So what we did was we played some music to the subjects and then we played the same music again, except that now every about 10 seconds it was a one-second silence period.
(Soundbite of song followed by silence)
HAMILTON: Silence in the room but not in the brain. Even when the music stops, the waveform from the brain continues.
(Soundbite of song)
Unidentified Man: (Singing) ...all in all it was just a brick in the wall...
HAMILTON: Schalk says what we're seeing is the brain's attempt to fill in the missing sounds.
Mr. SCHALK: The brain basically tells us a lot of information about the music in the times when there is really no music. It's not played.
HAMILTON: Whether it's musical phrases or strings of words or scenery we look at, our brains are always filling in missing information.
Schalk says ECoG is also revealing things about how the brain creates speech. He and other researchers are using the technology to watch the brains of people as they speak out loud and also as they say the words silently to themselves.
Mr. SCHALK: One of the surprising initial findings coming out of that research was that actual and imagined speech is very, very different.
HAMILTON: Schalk says when your brain wants you to say a word, it produces two sets of signals. One has to do with moving your muscles.
Mr. SCHALK: And that makes sense. First, you have to move your mouth around and your vocal tract around so as to produce a particular type of verbal output.
HAMILTON: And there are also signals coming from the auditory system. On the other hand, when a person simply thinks of a word instead of saying it, there are no muscle signals, just the activity in the parts of the brain involved in listening.
Mr. SCHALK: And that seems to suggest that what imagined speech actually really is, it's more like internally listening to your own voice.
HAMILTON: So it should be possible to use ECoG to decode what we're actually thinking. Schalk says he hasn't quite done that yet, but he's close. In one experiment, the system tried to recognize several dozen unspoken words in the minds of volunteers. It was right - about half the time.
Jon Hamilton, NPR News.
MONTAGNE: And you can watch videos of these experiments, including a patient controlling a virtual hand with his mind, at our website, NPR.org.
NPR transcripts are created on a rush deadline by Verb8tm, Inc., an NPR contractor, and produced using a proprietary transcription process developed with NPR. This text may not be in its final form and may be updated or revised in the future. Accuracy and availability may vary. The authoritative record of NPR’s programming is the audio record.