Teaching Computers To Be More Empathetic High school juniors Matthew Fernandez and Akash Krishnan took the grand prize in the Siemens Competition in Math, Science & Technology for designing software that decodes emotions in human speech. They say the software could be used by call centers, to direct angry callers to a human.
NPR logo

Teaching Computers To Be More Empathetic

  • Download
  • <iframe src="https://www.npr.org/player/embed/131967508/131967493" width="100%" height="290" frameborder="0" scrolling="no" title="NPR embedded audio player">
  • Transcript
Teaching Computers To Be More Empathetic

Teaching Computers To Be More Empathetic

Teaching Computers To Be More Empathetic

  • Download
  • <iframe src="https://www.npr.org/player/embed/131967508/131967493" width="100%" height="290" frameborder="0" scrolling="no" title="NPR embedded audio player">
  • Transcript

High school juniors Matthew Fernandez and Akash Krishnan took the grand prize in the Siemens Competition in Math, Science & Technology for designing software that decodes emotions in human speech. They say the software could be used by call centers, to direct angry callers to a human.


You're listening to SCIENCE FRIDAY, from NPR. I'm Ira Flatow.

Now, I am sure that you have had one of these moments: You're driving down the highway. Your GPS is telling you get off of the next exit. But you know that's the wrong exit; the one you really want is right after that. But that little voice keeps insisting: Merge right 100 yards, turn right. And then comes the inevitable: recalibrating.

Now, I don't know about you, but it gets pretty aggravating to me, to the point where I've told my GPS to, let's put it mildly, to shut up, or I have threatened to swap it out for a compass.

Of course, it can't understand your insults. But what if your GPS could sense when you were losing your patience and react, maybe try to sweet talk you a little bit, like, since it seems like you know the way, I'll just sit back and let you drive. Gee, I'd like to hear that sometime. Wouldn't that be something?

Well, unfortunately, the technology is not quite there yet. It's hard for computers to crack the code of your human emotion, understanding whether you're angry or sad, or you're frustrated, or you're happy.

But my guests have designed a program that may get us a little bit closer to that goal, a way for the computer to analyze in real time, as you're talking, what sort of mood you might be in.

And my guests are - they're not just any computer scientists. They're two high school juniors who are splitting a $100,000 grand prize in the Siemens Competition in Math and Science Technology. Let me introduce them to you.

Matthew Fernandez is a junior at the Oregon Episcopal School in Portland. Welcome to SCIENCE FRIDAY, Matthew.

Mr. MATTHEW FERNANDEZ (Winner, Grand Prize, Siemens Competition in Math, Science & Technology; Junior, Oregon Episcopal School): Thank you, Ira.

FLATOW: You're welcome. Along with him at the school is Akash Krishnan. He's also a junior there. Welcome to SCIENCE FRIDAY.

Mr. AKASH KRISHNAN (Winner, Grand Prize Siemens Competition in Math, Science & Technology; Junior, Oregon Episcopal School): Thank you, Ira.

FLATOW: Let me ask you, Matthew: How did you guys ever get started on this project? What was the impetus for this idea?

Mr. FERNANDEZ: Let's see. We were actually inspired by a movie entitled "I, Robot." It was originally a collection of short stories by Isaac Asimov, and it was remade into a movie in the last 10 years. And we were taking a break from trying to brainstorm a science research project idea and watched this film, and we were ultimately inspired by a scene in the movie where a robot can recognize that its user is very afraid and then protects its user.

And we thought: That's a really useful and really handy thing. Is there any actual science behind this kind of science fiction story? And it turns out that there is. Emotion recognition is kind of a really close topic, and we've been really fixated and passionate about it for the last year.

FLATOW: And so how long did it take you to actually put all of this together?

Mr. FERNANDEZ: We started this project about one year ago now. We got the idea in about October, and then our first prototype for us was about a year ago.

FLATOW: And actually, if our listeners want to go to our website at sciencefriday.com, we have a video of you actually using this. But Akash, for listeners who are driving right now, describe to us how the system works.

Mr. KRISHNAN: So the way it works is we have to train our program to recognize emotions. So we use a database, and let's say we have 100 angry files. Using these files, we extract information, such as the frequency ranges, the averages of those frequencies and energy levels for angry. And then we do this for each of the emotions that we have.

So we extract it for happiness and then sadness, and then using these ranges of frequencies and energies, we can compare it to another audio signal that you're talking. And you can figure out those ranges of frequencies and energies and then compare them and try to figure out which is the best match for each of the emotions.

FLATOW: So did you actually use a lot of computer programming and math in making this design?

Mr. KRISHNAN: Yes, there's a lot of high-level calculus that was involved. Of course, we don't actually know most of it, but luckily, there's a lot of algorithms out there that are built in to a program that we have used, called Madlab.

So basically, we just use Madlab and its functions and put them to good use for our program.

FLATOW: Matthew, how does your algorithm compare to state-of-the-art stuff? How good is it versus what's already out there?

Mr. FERNANDEZ: Our system is definitely state of the art. Some of the results that we've achieved are much higher than some that other researchers have published, specifically for a contest that was actually put together for a conference in Brighton, U.K., called Interspeech. It was an emotion challenge and it was designed - and the goal was simply to create a highly accurate emotion-recognition system. And our system is well above the winners of that contest.

FLATOW: 1-800-989-8255. Let's go to Russell Kirsch in Portland. Hi, Russell.

Mr. RUSSELL KIRSCH (Computer Programmer): Hello. When we built the first computer in America, back at the National Bureau of Standards a half-century ago, we all felt that nothing would be withheld from us which we have conceived to do. And as a result, we have a whole generation of people who developed all kinds of wonderful things, which we now know of as computer technology.

However, today, as a result of one of the things that was constructed, namely the Internet, people feel a little differently. They now feel that nothing will be withheld from us which we decide to say or hear or read.

But your guests today still know what we knew a half-century ago, that anything they want to do, they can do with these powerful techniques. And so what we need is for schools to teach more students what your guests have learned, namely how to program computers, because there are magnificent things that can be done, as well as just communicating with others.

FLATOW: Matt...

MR. KIRSCH: And so you have an excellent example of something that schools should be encouraged to do today.

FLATOW: Let me get a response from Matthew or Akash. How encouraging was your school? I mean, did you have mentors there who helped you...

Mr. FERNANDEZ: We actually - yeah, we've been going to the same school. It's called Oregon Episcopal School. It's an independent school in Portland. And we've both been going since the lower grades. Akash joined in first grade, and I was there for kindergarten.

And we've done projects together in the past, and I think that the kind of holistic approach to learning has really helped us kind of seek out some of the resources that we needed to start our project. So we did a lot of time at the beginning of our research, coming up with books and Internet resources, trying to help us learn how to create an emotion-recognition system. And then using that information, we were kind of able to then innovate some really cool new ways to analyze emotions from human speech.

FLATOW: Russell, thanks for joining us on the program today.

MR. KIRSCH: Okay, sure.

FLATOW: Have a good weekend. 1-800-989-8255. What kinds of competition did you have? Was it tough competition in winning this prize?

Mr. KRISHNAN: Yeah, there was a lot of competition for - in the Siemens Competition, all of the other projects were in different categories, such as mathematics, or this computer science projects, as well, and biological projects.

And the interesting thing is they're all very different, and they're all top-notch. They're all very good. And most of the time, I couldn't understand what all the other projects were talking about, but for some of them, I could. So they're all pretty interesting.

FLATOW: I find it - I also find it interesting that two of the databases you use to train your algorithm were German speakers. So it doesn't matter what language that you use, you can still pick out the emotional parts.

Mr. KRISHNAN: Yeah, that's the awesome part of our program is that you can translate any language, and as long as it's similar to the English or German culture, then our program will be able to perform at pretty much the exact same level. So any language, English or German or any languages like that, should work.

FLATOW: So where do you guys go from here? Do you think you have a product that you might be able to market, Matthew or Akash?

Mr. FERNANDEZ: Yes. I think that we're working, in fact, on creating a specific device. We're working on a wristwatch which would use like a smiley face, a sad face, angry face, on the wristwatch, and that would show an autistic child what emotions are being spoken.

Autism is a disease which can often lead to children especially having a very difficult time recognizing the emotions that are being spoken to them. And so this wristwatch could have the potential to significantly improve their ability to interact with their peers or to communicate in general. And we're really fixated on creating this device.

FLATOW: Was this at the beginning - in your mind at the beginning of your project or something that you developed an idea for later on?

Mr. FERNANDEZ: This was developed later. We came up with this over the last summer. We spent the summer working at a local university called Oregon Health and Science University. And there is a lot of research there on autism. And so people there suggested that we consider applying our research towards helping autistic children.

FLATOW: And what kind of computing power? How big, you know, powerful a computer do you need to run your system?

Mr. FERNANDEZ: Our system actually can be run on just a simple PC. And so we used - we both used personal laptops for our research. And we haven't used really big supercomputers or anything too exciting. In fact, we're looking at creating perhaps something like an iPhone app, which uses less processing power, but we're have to redesign our system to accommodate that.

FLATOW: It looked like it might also be useful in video gaming, perhaps, if you thought about that.

Mr. FERNANDEZ: Yes, I think so. I think that gaming is moving towards a place where you can interact more and more, and now with the new Xbox Kinect system, where you can wave at your TV and it'll follow your movements, things like that, I think that if you can start talking and interacting with your TV or your games, I think that that just gives you another human connection. It could be really cool.

FLATOW: Yeah, it also gives you something else to learn how to be angry. Have you wave at your TV or your screen and decipher how much waving it takes to show anger or whatever kind of emotion.

Mr. FERNANDEZ: Exactly, exactly.

FLATOW: Have you guys started thinking about college yet, being juniors in high school, where you want to go? And are you thinking about following up on your careers in computing?

Mr. KRISHNAN: Yeah, I plan on learning computer science and electrical engineering. And we don't really have a list, or at least I don't have a list, of colleges I want to go to. But there are all the top schools that, of course, such as MIT or Stanford or Carnegie Mellon University. But it's not really a big list.

FLATOW: And you, Matthew?

Mr. FERNANDEZ: Yeah, I'm really excited about - we're continuing our research into the next year, trying to improve our results even further. And I think that potentially, where we go to college may be affected by the kind of interest that universities receive related to our research.

FLATOW: Well, you never know. There are a few people, a few professors listening, who feel, you know, we have over a million people listening. So maybe somebody's got an offer for you after they hear about your research. Have you been doing a lot of publicity on this? Have you found a lot of interest in your work?

Mr. FERNANDEZ: Yes, yes we have. We, in fact, earlier in May, got contacted by someone from the government, the military, to start perhaps using some of our stuff for military applications, specifically, like, teaching humans how to recognize the emotions in other languages, for language training.

FLATOW: Ah, right.

Mr. FERNANDEZ: But yeah, also from industry, there's been some interest, and I think that this kind of getting to appear of National Public Radio is, you know, another big step towards getting us more connections.

FLATOW: Well, good luck to you. Good luck in your college careers and your computing life.

Mr. FERNANDEZ: Thank you, thank you.

FLATOW: You're welcome. Thanks for taking time to be with us today. Matthew Fernandez and Akash Krishnan both of Oregon Episcopal School in Portland, who are the grand prize, $100,000 - gee, we didn't get to ask them what they're going to do - 100,000 bucks from the Siemens Competition in Math, Science and Technology

Stay with us. We're going to switch gears and bring on Isabella Rossellini. She'll be with us to talk about sex and animals. You'll have to stay by to see what I'm talking about. So stay with us. We'll be right back after this break.

(Soundbite of music)

FLATOW: I'm Ira Flatow. This is SCIENCE FRIDAY, from NPR.

Copyright © 2010 NPR. All rights reserved. Visit our website terms of use and permissions pages at www.npr.org for further information.

NPR transcripts are created on a rush deadline by Verb8tm, Inc., an NPR contractor, and produced using a proprietary transcription process developed with NPR. This text may not be in its final form and may be updated or revised in the future. Accuracy and availability may vary. The authoritative record of NPR’s programming is the audio record.