A Virtual Arm That Talks With The Brain
IRA FLATOW, host: You're listening to SCIENCE FRIDAY. I'm Ira Flatow.
Picking up a wineglass may seem like a simple task, but there's a lot of nervous processing going on between your fingers and your brain, you know. How do you know how gently to hold it so you don't crush the glass in your hand, or how tightly so it doesn't slip to the floor? Your fingers send signals of touch back to your brain so you can feel the glass, and the brain responds, and there's a lot of interesting coordination going on there. But what if you had a prosthetic hand that cannot feel? Life gets a little tougher. Scientists are working to solve this problem using mind control, and what they're working on, in this case, is a virtual arm for monkeys.
Miguel Nicolelis is a neuroscientist at Duke University in North Carolina and director of the Center for Neuroengineering there. He's also a co-author on a study published this week in the journal Nature about the new virtual limb he and his colleagues created. Welcome back to SCIENCE FRIDAY, Dr. Nicolelis.
Dr. MIGUEL NICOLELIS: Oh, thank you for having me. It's a great pleasure.
FLATOW: Tell us - you're welcome. Tell us about what you had these monkeys do.
NICOLELIS: Well, basically, we trained these animals to use their own brain activity to control an avatar arm and hand that could explore objects, virtual objects that we created for them. And to select a correct object that, you know, conduced them to receive a juice reward, they have to basically explore the texture of these objects by using this virtual hand, these virtual fingers and - because we were basically sending, every time they touch an object, an electrical message back directly to their brains that describe this texture. So they basically could do this entire tactile discrimination task just by using their brains, no interference of their bodies whatsoever. And they actually learned to do that.
FLATOW: So they used mind control to control something, a pointer on a computer that would touch the objects?
NICOLELIS: Well, it was a real virtual arm, you know...
FLATOW: It was a real arm.
NICOLELIS: ...like a monkey arm.
NICOLELIS: And, basically, we learned later that the monkeys very likely assumed that that was their own arm, because when we touched this virtual arm with, you know, virtual probes, the tactile part of their brain - brains responded to the touch, as if we had touched their own body. So it seems like they were assuming or assimilating the virtual arm as part of their own body. So they were able to use their brain activity to move it, and the signals that we send back to their brains to interpret what they have encountered in that virtual space.
FLATOW: And how much of a fine motor coordination could you get there, or was this just a proof of concept?
NICOLELIS: Well, it was pretty good. It was, you know, an evolution from what we have done when we started this field about 12 years ago. Now they could do, you know, moving in 2-D, and now we are working on 3-D. And they can actually explore the objects and touch them and press the correct object, assuming that they detected the correct texture. And every time they did that, they got a drop of fruit juice that they pretty much enjoy.
FLATOW: And so they had to be trained how to do this.
NICOLELIS: Yes, they had to be trained. But they - what we - made us really shocked was that in four to nine sessions, four to nine days, they actually learned the whole thing, to interpret these signals and to associate them with particular objects.
FLATOW: Hmm. And the wires that go from the brain to the virtual arm, do you need to hook these up to a specific part of the brain to get the right signals?
NICOLELIS: Well, not exactly as specific. You know, we used to think that each part of the brain does just one function, but we are discovering through, you know, a decade of this type of research, that messages, like motor messages, are distributed across multiple areas of the cortex of these animals. So in this case, we sampled one of these areas, the motor cortex, to basically generate the motor commands that we needed to move the avatar arm, and we delivered the tactile feedback message to the somatosensory cortex, one of the areas that is involved in processing normal touch.
And - but what we are really surprised is that we basically established this brain-machine-brain interface without any interference from the animal's body. And in a few sessions, the animal just basically took that for granted and performed the task just fine.
FLATOW: Wow. So they quickly picked this up.
NICOLELIS: They quickly picked it up, yes.
FLATOW: Can you explain why that might be?
NICOLELIS: Well, there's - one of the theories that we have is that as we use - as primates, as we use tools to basically increase our reach in the world, we propose that the brain actually incorporate these tools as an extension of the body representations that the brain has. So every brain of a primate creates a body image, this sense that we inhabit a particular body.
But, you know, we think that this is a flexible model, a flexible representation that can include or assimilate even the tools that we use in our daily routines. So our cars, our baseball bats, our soccer balls, everything that we use as a tool becomes part of us for the brain. So we think that part of this - the results that we got in this study support that vision, that that avatar arm was assimilated by the brain as if it were an extension of the animal's own body.
FLATOW: Of course, everyone hearing your report and listening to us now is going to say: Do you think people could learn to use a virtual arm like this?
NICOLELIS: Oh, absolutely. In fact, we want to go even further. We are hoping that this research can lead to people that suffer severe lesions of the spinal cord and - that make the patients severely paralyzed from the level of that lesion down, that they - one, they will be able to use this brain machine, brain interface to control a whole-body robotic vest, an exoskeleton that will restore full-body mobility to these patients, and also restore the ability to sense what they encounter as they move around in the world.
FLATOW: And what kind of prosthetics can you make, then, for them? Would they be incorporating the touch and the feel and the movement in - all into one?
NICOLELIS: Yes. We're already, actually, building a prototype of this exoskeleton, this vest. Our colleagues at University - Technical University of Munich, led by my good friend Gordon Cheng, is already building a prototype of this exoskeleton that we hope to test in clinical trials very soon.
FLATOW: Mm-hmm. Is that the walk-around - walking-in project?
NICOLELIS: Yes. That's part of this international consortium between Duke, the Technical University of Munich, the Polytechnic School in Lausanne and the Natal Institute of Neuroscience in Brazil, where we hope to basically put all this technology together to create a whole-body prosthetic device that may allow quadriplegic and paraplegic patients to regain mobility and the sense of touch as they interact with a brain machine, brain interface.
FLATOW: And so what - how many neurons do you need to measure for a full bodysuit?
NICOLELIS: Yeah, that's a very question. We are crossing, at this moment, the barrier of a thousand neurons, recorded simultaneously. So we are going very close to recorded activity, the electrical signals produced by 1,000 neurons. We expect that between 1,000 and 10,000 neurons will allow us to start controlling fundamental movements of this exoskeleton. So that's our goal for the next three years, to reach about 10,000 neurons simultaneously.
FLATOW: And it's possible. I imagine you're confident. Yeah.
NICOLELIS: Yeah. We have a new family of sensors that we are developing, 3-D sensors to record brain electrical activity. We hope to reach this level in the next few years.
FLATOW: If these, you know, I've heard, you know noise of neurons firing, you know, they sound like static electricity.
FLATOW: How do you make sense of all of that?
NICOLELIS: Well, there are messages embedded in the noise. In fact, my - one of my sons say that when he listened to these brain storms that we recorded from hundreds of neurons, that it sounds like popcorn in a microwave while you listen to an AM station that is not well-tuned.
(SOUNDBITE OF LAUGHTER)
NICOLELIS: But there is a message in there, and we can actually extract these motor commands very quickly using, actually, very simple mathematical models that run at the same time scale that our brains actually produce movements out of these electrical storms.
FLATOW: 1-800-989-8255. Let's get some phone calls in. Rick in Elizabethtown, Kentucky. Hi, Rick.
RICK: Hi. How are you today?
FLATOW: Hi, there.
RICK: Yeah. Listen, in 1973, while I was still in the Army, I was a part of a research program that the Army was doing on single-motor coordination skill. And they would actually inject a barbed wire into certain muscles of the body, and with the use of a computer, we were required to move that single muscle and isolate it with just using our brain. And we went through this for about a six-month period, and everybody was getting pretty well. And my comment is I'm really surprised, with the effort that was put forth back then, that we haven't been able to, if you will, finalize or make this thing just - I'm surprised it hasn't accelerated any greater than it is now.
FLATOW: At 40 years later almost, why haven't we gotten anywhere further? Miguel?
NICOLELIS: Well, that's interesting that you mentioned that, because one of the first things I did in grad school was - my thesis adviser actually made me do the same test, and I was amazed myself that I could control a single muscle and to do this kind of experiments. Well, the technology to record large-scale brain activity the way we do now only started to be developed really in the mid-'80s. And we only reached, you know, 100 neurons recorded simultaneously at the late '90s.
And so from the late '90s to now, we discovered that we could link brains to machines using, you know, computer devices that are, you know, available now cheap, and we could actually perform the kind of experiments that we reported this week. So it took a while to be able to actually get the kind of brain-drive signal that you need to run this type of experiments. It's much more difficult to do that than to record the electrical signals from muscles. So that was one of the major reasons that it took so long for us to get to this point.
FLATOW: It's one thing to probe a monkey's brain with wires. How do you expect to do that with people, where you can't do that?
NICOLELIS: Well, no. You can. In fact, you do already. There are several patients - there are thousands of patients, tens of thousands of patients that carry either a stimulator in the brain or in the periphery, in the inner ear, to restore neurological functions or to control diseases like Parkinson's disease. Basically, these electrodes, these sensors will be tested, you know, in patients. It's - the benefit that you get is pretty high compared to, you know, what the patient is suffering right now in terms of a lesion or a handicap.
So we think that the technology is safe, and at this level, we will be able to provide to patients a relief and a benefit that, you know, justifies a chronic implant on the surface of the brain.
FLATOW: I'm Ira Flatow. This is SCIENCE FRIDAY, from NPR.
Talking with Miguel Nicolelis about his experiments. Now, a lot of people are tweeting and asking whether you cut the limbs of these monkeys off or not. You didn't do that.
NICOLELIS: No. No. No, no. You don't need to do anything to the monkeys. The monkeys are actually pretty happy. And now that we have a wireless interface to broadcast these signals, these brain-drive signals, the brain - the monkeys really can be freely moving and actually enjoy playing with the video games. They actually have a lot of pleasure playing these games.
FLATOW: So even though you don't know what it feels like to them to get this stimulation back, they must like it.
NICOLELIS: Oh, they certainly like it. And besides, every time they got the correct object selected and they reach a very high level proficiency after a few weeks of training, they get a drop of fruit juice. And any monkey will do anything for you for that fruit juice.
(SOUNDBITE OF LAUGHTER)
FLATOW: I think you'd have to up the ante for people.
NICOLELIS: Yes, yes. We have to change the reward a little bit.
(SOUNDBITE OF LAUGHTER)
FLATOW: Let's go to the phones again. Harvey in Oakland. Hi, Harvey. Harvey, are you there? No. He's not. I guess one of the things he was going to ask is about possible military uses of these, and people are wondering all the time if you have - if you can control a robot by thinking about it, could you make, for example, a bomb-sniffing robot so that the humans don't have to go on there, and to control it with your own brainwaves?
NICOLELIS: Yeah. Well, I have no military application in my research. You know, we are all involved into rehabilitation medicine. But several scientists in the U.S. have proposed ideas, for instance, to have robots do exactly what he said, to go into environments that are too dangerous for people to go, like, for instance, as we had recently, the nuclear accidents in Japan. The idea would be that would send robots to do the job of fixing the place, and no need to send humans, because the risk was so severe, so dangerous.
So there are some ideas about that, but they are far from, you know, doing anything concrete because, of course, to get the best level of motor control, the ones that we get in our experiments, you have to implant a microchip in these sensors, a few millimeters inside the brain. And that will never be justified for an application like that in a regular person that doesn't have a severe disability.
FLATOW: Let's go to Maggie in Stanford, Connecticut. Hi, Maggie.
MAGGIE: Hi. I was wondering if or how you plan on incorporating proprioception into this robotic arm, so where it is in space.
FLATOW: Yeah, three-dimensional.
NICOLELIS: Yes. Well, we want to test this, the same technique that we use here, microstimulation. Electrical microstimulation of the cortex can be used to other sensory channels to restore other sensory mobilities like temperature or proprioception. That's part of the studies that we are starting to carry out at Duke right now. So this was the proof of concept that we could close this loop and could get these animals to feel something or to interpret these signals that are related to texture, to fine touch.
But now, you know, the door is open to test other possibilities. And, in fact, we are even testing in rodents, in rats, other modalities that are not even related to the regular modalities that we all share. So we are trying to see if with this, our rats can sense physical parameters like infrared light, for instance, that rats normally don't perceive. So it opens a lot of new venues for this type of research.
FLATOW: Thanks, Maggie.
FLATOW: That's about all the time we have. I want to thank you very much for taking time, because, as you say, there are a lot of things in the future where you might apply this to that we haven't even thought about it.
NICOLELIS: Well, I appreciate. Thank you very much for the invitation.
FLATOW: You're welcome. Miguel Nicolelis is a neuroscientist at Duke University in North Carolina and director of the Center for Neuroengineering there.
NPR transcripts are created on a rush deadline by an NPR contractor. This text may not be in its final form and may be updated or revised in the future. Accuracy and availability may vary. The authoritative record of NPR’s programming is the audio record.