How Movie Makers Use Science To Make Magic
IRA FLATOW, HOST:
This is SCIENCE FRIDAY. I'm Ira Flatow. We're here in California, broadcasting from the California Academy of Sciences in San Francisco. And while you might think Silicon Valley or biotech when you think of Northern California, this part of the state is also home to some of the biggest names in the movie business.
Move over Hollywood, the movie stars here are the computers and the designers who create the animation and the visual effects that let actors do the impossible - electronically, of course. Engineers and mathematicians are developing technology and software that's transforming the movie business and the experiences of going to the movies.
Wondering how they made Brad Pitt age backwards in "Benjamin Button," how they created the squid-like Davy Jones character in "Pirates of the Caribbean," how they made those "Toy Story" toys seem so lifelike? Well, we'll be talking about the science behind what you see on the silver screen and ask: Will there come a time when technology gets so good we don't need those live-action folks, those human actors?
Have you ever sat in a theater saying and thinking to yourself how they heck did they do that? Well, we've got experts here who are going to tell you, and if you're going to make it to the movies, and you're a science geek, there's plenty of room to put those math and engineering skills to work in the movie business.
So we're going to be talking about that today, and let me introduce my guests. Steve Sullivan is senior technology officer at Industrial Light and Magic. He's also a three-time Academy Award winner. Welcome to SCIENCE FRIDAY, Dr. Sullivan.
DR. STEVE SULLIVAN: Thanks.
FLATOW: Also with us is Tony DeRose. He's a senior scientist and lead of the research group at Pixar Animation Studios. Welcome to SCIENCE FRIDAY, Dr. DeRose.
DR. TONY DEROSE: Thanks, it's great to be here.
FLATOW: Craig Barron is co-founder and visual effects supervisor for Matte World Digital. He won an Academy Award in 2009 for his work on "The Curious Case of Benjamin Button," and he joins us from NPR West. Welcome to SCIENCE FRIDAY.
CRAIG BARRON: Thanks, Ira.
FLATOW: Let's get going to define some terms, Steve. Help us set the scene here. When we're talking about visual effects in movies, there are physical effects, and then there are computer-generated effects, right? Are these two of the broad categories?
SULLIVAN: Yeah, you know, it's imprecise, the terminology, but we often refer to special effects as being the things we do in camera. So you shoot that footage on set, versus visual effects is usually referring to the computer-generated stuff that we do after the fact to enhance or alter the image to help the director tell a story.
FLATOW: Give us an example of a movie or two where that might - we might see that kind of thing.
SULLIVAN: Oh, all over the place, yeah.
(SOUNDBITE OF LAUGHTER)
FLATOW: Now it's all over the place.
SULLIVAN: Yeah, absolutely. You know, we have a movie coming out called "Avengers" that's going to have all these different computer-generated characters but also computer-generated backgrounds and cityscapes that you would assume have just been shot that way on camera, but actually we add it in to make the shooting easier, later.
FLATOW: Yeah, I'm thinking in particular to the HBO series "Boardwalk Empire," where they really didn't have a boardwalk or the whole city, right? They added all that stuff in.
SULLIVAN: A lot of our work is actually those so-called invisible effects, where the audience is not supposed to know anything's going on, but it makes the shooting a lot easier.
FLATOW: But there are cases like in "The Pirates of the Caribbean," Davy Jones' head, right? I mean, it was like he's got these squid-like tentacles coming out of him.
SULLIVAN: Yeah, actually, that's funny, people often ask us where did the makeup stop or what was real and what wasn't. The whole character was digital. There was no actor there onscreen ever.
FLATOW: There was never any actor?
SULLIVAN: No, never was. But we took great pains and invented some new technology so that Bill Nighy could be on set with the other actors and drive the performance. And you can tell that's Bill's performance there, but it's all computer graphics on the screen.
FLATOW: And the Benjamin Button, that was the character's head but nothing else, right?
SULLIVAN: Right, yeah, I think they had live actors there and replaced the head in most of the shots.
FLATOW: Craig, tell us about that.
BARRON: Well, Ira, you know, you mentioned in the opening about sort of replacing the pesky actors, but it's really more of a collaboration. The actors are driving a performance, and it's up to the special effects person to capture that performance and allow it to be coming through with the computer-generated character.
FLATOW: Tony, let's talk about animation itself. In the old days, I remember from, you know, the old Disney days, people would actually draw those cels one cel at a time. How has that changed now? What goes on now?
DEROSE: Well, in cel animation, often a senior animator would draw key frames, you know, maybe every 15 or 20 frames of film apart. And then the ink and paint department and in-betweeners would come along to fill in all the motion in between those key poses that the senior animators would provide.
Today we still have senior animators creating those key poses, those points in time where acting and emotion is really delivered, and then we use various mathematical methods to do the in-betweening automatically.
FLATOW: How do they know what to do? How does the computer know what to do? I mean, a person would know, I would think, logically, but it must be very complicated algorithms or something.
DEROSE: Well, the algorithms themselves really aren't that complicated. We use pieces of polynomials that algebra students learn about in class. We happen to use third-degree polynomials. So the mathematics isn't that complicated, but the skill of the artist and the animators are really what cause that mathematics to, you know, let the emotion come through.
FLATOW: In an animated film like "Toy Story," the characters were stylized, but what about the world they live in, the way their clothes move or the way the light looks? Do you try to mirror reality in that way, also?
DEROSE: Well, so much of an audience's expectations about motion are governed by, you know, physics in the real world. But we're not constrained by physics. We use physics kind of as a starting point, as a jumping-off point. And then we try to make our world and our motion even better than reality.
FLATOW: So what are the hardest things for an animator to do: skin color, tone? What are the hardest things to create?
DEROSE: Well, for the whole filmmaking process, capturing that emotion in a performance is very difficult, and that's what our animators bring to the table. Once the emotion is there, then other artists are responsible for making sure that, you know, the plastic looks like it should and that the lighting is going to reflect the desires of the director.
And then algorithms kick in to try to take the input from the artist and finally create the beautiful pixels you see in the theater.
FLATOW: Craig Barron, tell us about, has anybody actually made a totally digital character in a live-action movie?
BARRON: Well sure. You know, it's - that is actually one of the exciting things that's going on in visual effects today is that - and certainly one direction that this may lead to in the near future is a recreation of, say, an actor that may no longer be with us, like say a Marilyn Monroe.
So - like last year there was a film called "My Week with Marilyn," which was a fantastic movie that Michelle Williams portrayed Marilyn Monroe in the film, but it's possible in the near future to create Marilyn Monroe completely as a computer-generated character to make a more compelling, realistic story.
So that's in the near future. I don't think we're quite there yet, but that's sort of the holy grail of what we might see in the future.
FLATOW: So she would have to look good enough for people to believe it, though, right? I mean, you have to be...
BARRON: Well, the movie would not work unless it was undetectable as an illusion. You - there's also an example with this just recently is that the people at Digital Domain, a visual effects company in Los Angeles, did a concert film with slain rapper Tupac. And he performed onstage with Dr. Dre and Snoop Dogg in a musical number. You can Google that, and if you look at the video, it's really quite astounding.
FLATOW: Because I've seen - you've seen dancers and things. Fred Astaire was in a commercial, I think, recently. They brought him in from an old movie to do the commercial.
BARRON: Yeah, in that case it was a repurposing of film clips. In this particular case, this is a completely computer-generated person, I guess what Jim Cameron would call an avatar.
FLATOW: Wasn't the Gollum in "Lord of the Rings" a totally generated...
BARRON: Yes, absolutely, and again, but - the component of what's making that work is that there is an actor behind it, and it's again a collaboration between the visual effects staff, animators and an actor making a performance.
FLATOW: Tony, what are your feelings about this, animation stay away from people, or are we going to have to be very careful how we do this?
DEROSE: Well, I think there are a lot of different things that can be enabled by this technology, and it'll be interesting to see which directions things go. But as has been echoed here, actors bring, you know, such talent and warmth to the performances that, you know, I think one goal is to just, you know, help the performance and the final effect be better.
FLATOW: So you don't think that we're going to do away totally - all three of you, Steve, you're shaking your head - with actors?
SULLIVAN: Yeah, no, I don't think so. I think our challenge right now is to focus on how to get more of the actors driving performance into whatever character they're playing, whether it be a digital Marilyn Monroe or a totally fantastical creature. We want somebody there authoring that performance that the director is working with.
FLATOW: You know, people don't think about a director directing an animation. Now, what does a director do? You've got the people drawing this stuff, the computer's making it. What does a director do in an animation?
DEROSE: Well, our directors really have the vision of the film in their head, and they're trying to explain it to the rest of the crew in a way that the film comes together as a cohesive, creative whole.
FLATOW: So do they say, you know, that character smiles too much, don't make it smile so much, like you would say to a real, live-action actor? You're giving me too much energy. Give me less or more or something like that. Do you actually do that with the animation?
DEROSE: Over and over and over again.
(SOUNDBITE OF LAUGHTER)
BARRON: Too much.
FLATOW: It's usually too much? Usually they tell actors I need more. But so it's up to the - and I guess if you have an animator, you could start working with the animator and know who the animator, who gives you what you want, like if you're working with an actor who gives you - you know, you want to work over and over again with the same animator. Is it like that?
DEROSE: It is. At Pixar, we have two kinds of actors. We have the voice actors, you know, like Tom Hanks and Tim Allen, and then our animators are really actors, they're just acting through the characters rather than through their own bodies.
FLATOW: All right, we're going to take a break and come talk lots more about animation. If you're here at the California Academy, we have a couple of microphones. You can come down to the mic and ask questions. And we're going to be joined a little bit later by - James Cameron is going to come on and talk a little bit about animation. He's considered to be one of the gurus of the animation projects around the world.
And you can also tweet us, @scifri, @-S-C-I-F-R-I. We're taking tweets. So we're not taking calls today because we're right here at the California Academy of Sciences in San Francisco. So make use of the microphones. We'll be right back with our guests after this break. Stay with us.
(SOUNDBITE OF MUSIC)
FLATOW: I'm Ira Flatow. This is SCIENCE FRIDAY from NPR.
(SOUNDBITE OF MUSIC)
FLATOW: You're listening to SCIENCE FRIDAY. I'm Ira Flatow. We're talking this hour about the science of visual effects and animation in the movies with my guests: Tony DeRose is a senior scientist and lead of the Research Group at Pixar Animation Studios; Steve Sullivan, senior technology officer at Industrial Light and Magic. Craig Barron is co-founder and visual effects supervisor for Matte World Digital.
And you can tweet us @scifri, @-S-C-I-F-R-I. Joining us now, I want to bring on is James Cameron. You all know him. He is the director of "Avatar," "Titanic" and recently doing deep-sea exploring. Welcome to SCIENCE FRIDAY.
JAMES CAMERON: Hey, Ira, thanks for having me on the show, appreciate it.
FLATOW: It's our pleasure. Thank you for joining us. Where do you see the future of visual effects going? Is there any limitation to it?
CAMERON: No, I don't think there is a limitation. I think pretty much the only limitation is the human imagination. I've been saying this for years, you know, really for more than a decade, but I think it's really become, you know, eminently obviously to everyone in the last I'd say three to four years that if we can imagine it, we can do it.
FLATOW: The trend we see now is toward making 3-D movies. Do you feel that 3-D, we're going to making all of our movies in 3-D now?
CAMERON: Well, I sort of bet the farm on 3-D 10 years ago, you know, started a company to build 3-D cameras. So, you know, I'm - you know, my answer I think is fairly obvious. Everything I do is in 3-D, you know, whether it's documentary or feature film. And I think, you know, while there are stumbling blocks along the way, and there will continue to be stumbling blocks, I think we see a steady progression toward adoption.
Already at the cinema, I think we're, you know, over the major hurdles, and I see a steady progressive toward adoption in the home, as well, although there are different hurdles there. But I think really what we're doing is we're taking our entertainment, and we're, you know, calibrating it and aligning it to our sensory system. You know, our sensory is stereoscopic.
FLATOW: And we also have an old-fashioned frame rate, don't we, for films, at 30 frames per second. But there's - with the advent of computers, where we don't care about the frame rate, does that allow you to do things you couldn't do before?
CAMERON: Well, we've been stuck at 24 frames a second since the early '20s, when sound, you know, first was beginning to be adopted. And it's a barely adequate frame rate. It just barely does the job. But what happens is you, you know, when you increase the sense of visual acuity and physical presence that you get with 3-D, the motion artifacts of 24 frames per second become more obvious.
And some people notice it right away, and others don't. But I think a lot of the problem that people are having with the sense of, you know, call it motion-sickness when they watch 3-D is when the camera pans rapidly, or the subject moves rapidly, you're very aware of these edge artifacts caused by the 24-frame replacement rate.
So yeah, the answer is that now that we've gone to digital projection in order to embrace 3-D, the answer is right there because these digital projectors can run at much higher frame rates.
FLATOW: Tony DeRose, we were talking a little bit before about something that the human eye is capable of doing, but we haven't been able to do very much yet in the movies, and that's the high-dynamic range. Explain what that is to us.
DEROSE: Well, the visual system is capable of all kinds of things that we're not very good at reproducing with projectors and TV monitors. The range of colors that humans can see is much larger than a TV, but also the range of intensities that the human eye can pick up is very much greater than any of the projectors or television sets that we have now.
So, you know, I think we're going to be, you know, over the next couple of decades, pushing display technology more and more to the limits of visual acuity in every way that the visual system can work.
FLATOW: And just a matter of new technology, creating new technology to do that, or...?
DEROSE: Yeah, a lot of the technology exists. So it's a little bit of a chicken-and-egg problem, you know, the content and the technology has to come out together. And 3-D is something that theaters are adopting now, and I don't think there's a lot of appetite for them to adopt high-dynamic range anytime soon. But...
FLATOW: They have to do one thing at a time, get new projectors first before - Steve, you're shaking your head.
SULLIVAN: Yeah, absolutely. These are all very interesting technologies that really impact the way you sense the film, and they have an interesting impact on the way we make the movies, too, because visual effects relies a lot on cheats: just make the frame look good and move on.
But when you add 3-D into the mix, high dynamic range, higher frame rates, suddenly our palette of cheats gets much smaller, and we have to do things that are much more real, and it just ups the ante for us.
FLATOW: James Cameron, how do we - how do you take 3-D - or 2-D films and make them into 3-D? What do you have to do to make them look realistically 3-D?
CAMERON: Well, it's a daunting task because when you have a 2-D film, you have zero depth information from the actual moment of photography. There is literally nothing. There is no record of where objects were, you know, on the Z-axis. So really what you're left with is human interpretation of the image.
We have a lot of stereoscopic cues kind of built into our - or depth cues kind of built into our visual system. We know that, you know, a very small man in the picture is probably not a very small man, probably not a six-inch person, it's probably a full-sized person very far away. That's a fairly obvious depth cue. But we have lots of others.
And they're all running, you know, hundreds of times a second when we look at the world around us. So it turns out that human artists are actually pretty good, when you sit them down at a computer workstation, at starting to assign depth values to things in a two-dimensional picture.
So what you do is you take a bunch of artists, and you basically just kind of throw people at the problem. I think there is this myth out there that there is a kind of a magic box or a killer app, and you can stick a 2-D picture in one side of it and a good 3-D picture will come out the other side of it. That's just not possible.
It's a creative, human-in-the-loop, subjective process, and it involves a lot of people and a lot of man hours or person hours.
FLATOW: You know, back, I'm thinking back to when we started seeing big visual effects in the movies back in the '50s, and then we had VistaVision and Cinemascope and all these sort of ways to get people out of their TV sets and back into the theaters again. Are we having sort of a race like that now, where we're trying to get people away from their giant-screen TVs and back into the theaters by offering these new kinds of 3-D experiences? Craig Barron, what do you think about that?
BARRON: Well, I think you're absolutely right. You mentioned "The 10 Commandments." That was in the 1950s. And, you know, technology like VistaVision and optical effects was created in order to lure audiences back away from their TV sets to raise the bar for the theater-going experience. And I think we are definitely at that point now.
And the efforts of Jim Cameron to make movies that are more compelling, something that you can't download on your iPhone, is the name of the game, and increasing frame rates and improving projection and making more compelling experiences for audiences is what it's all about for the future.
FLATOW: Jim Cameron, you said that there - you know, there's no little magical box that you can stick a 2-D movie in, and 3-D comes out. What magical box, what technology would you like to have, and is it possible? What cutting-edge new technology would you like to wait - or help to develop?
CAMERON: Well, I think that what I see in the immediate future is not some revolutionary new thing. We've been through - you know, we've kind of - we've kind of been through the revolutions. We've added sound, we've added color, we've added depth. And there are variations on all of those themes.
For me, I see a consolidation in the near future of a lot of the things that have been just introduced in the last few years, the idea of virtual production and performance capture and so on. That needs to be consolidated. It's still cumbersome and expensive, although the end result I think speaks for itself.
I think 3-D, we need to consolidate. We need to improve the brightness in theaters. You can't have dynamic range if you don't have brightness. And right now, the - even the dynamic range of a 2-D picture is not realized in a 3-D movie. So I think 3-D right now is about an 80 or 90 percent success story, but that last 10 or 20 percent is going to be getting the light levels up so we can enjoy, you know, higher dynamic range.
The things that I'm personally involved in developing, aside from a lot of workflow stuff having to do with the things I just mentioned, is the higher frame rate technology and what that means to the visual effects process because it's fine to say that the digital cameras can run at high speed, and they can, and the projectors can, but there's that bit in the middle, where you're processing the image and you're adding elements to it and you're adding, you know, a lot of post-production visual effects that becomes more problematic at higher frame rates.
FLATOW: Let's go to the audience here. Yes, ma'am, step up to the mic.
UNIDENTIFIED WOMAN: If I wanted to be an animator, would I have to study computers or art?
CAMERON: I'd say both.
FLATOW: There you go. You said both, Jim?
CAMERON: Absolutely both. When I was CEO of Digital Domain, I had a lot of very smart computer artists working for me, and I sent them back to school on the basics of art, you know, linear perspective, aerial perspective, composition, all those things because they were very smart in the area of, you know, creating surfaces and objects and, you know, rendering and so on. But they didn't really know the basics of art, and it's still a visual medium.
FLATOW: And, Tony, do you agree, at Pixar that there's room for mathematicians who have a little bit of interest in art? Or what should you study if you want to get into this?
DEROSE: Yeah. At Pixar, we have a few different populations, so we have the technical crew. We have the artistic crew. And the main thing is to be really excellent at something. So if you're really a technologist, become an excellent technologist and then broaden so that you can learn to work effectively in teams, with people that think very differently.
FLATOW: Who wants to talk about the uncanny valley? Anybody? Steve?
CAMERON: Oh, boy. The uncanny valley, that was the terrain that we had to get across in order for "Avatar" to be a success. You know, I always said, you know, we're going to try to catapult ourselves across the uncanny valley and land with our fingertips embedded in the opposite cliff, try to pull ourselves over. That was the challenge for the whole team on "Avatar."
FLATOW: And what was the big challenge? Why was it so challenging?
CAMERON: Well, all the main characters, you know, or most of the main characters in the film were going to be done completely with CG. So we had to really feel like in order for us to have the correct emotional impact on the audience, the audience had to believe that the creatures that they were seeing, which were largely humanoid, were real. They couldn't be thinking that it was good CG, or we'd be doomed. And, you know, so we had the problem the so-called deadeye syndrome that we had to get over unfortunately.
You know, Weta Digital at the time was one of the leaders in creating a very kind of liquid beautifully refractive eye. And I always felt that the problem was around the area of the mouth, the kind of neuromuscular interaction around the mouth that forms the expression is so complex. And I always felt that that's where most CG characters broke down, not around the eyes. So I actually encouraged them to spend a lot of time and energy in that area as well.
FLATOW: Yeah. Because if you don't get stuff like that right, it sort of comes out creepy looking, doesn't it?
(SOUNDBITE OF LAUGHTER)
FLATOW: I mean, the audience has a little bit of problem with it. It's almost real but not really real.
CAMERON: Right. Well, the uncanny valley refers to a dip in positive response. It's actually meant to be, you know, kind of an X-Y graphing. On the Y-axis, you've got positive response to what you're seeing. And on the X-axis, it's from left to right. It's as you're approaching looking human. And so there's a point where kind of animated animals and a talking moose and so on at the left side of the graph. They're all fine because they're humanoid to the extent that they're doing human behaviors, but they don't look human.
Then as you start to approach looking human with things like androids and, you know, synthetic rubber flesh on Japanese robotics experiments and even things like corpses where there's no movement, we start to get a very negative effect, so the curve dips. And then it comes up again sharply as you exactly approach the look of a true living human.
FLATOW: One last question before we have to go. Is the story still matter? The story, you know, does the animation and visual effects overwhelm the story so much that the story doesn't really matter anymore? Well, I'll ask Jim. Jim, what do you think?
CAMERON: Well, look, I mean, I think we all know the answer to that question. People respond to stories. And, you know, they respond to human interaction and to emotion. And they want to have an emotional response themselves. And so I think we've seen the big hollow kind of clanking but quite beautiful and quite difficult visual effects extravaganzas that didn't work. And I think we've seen films that were much more modest in their effects goals that did work, you know?
So I think step one is you've got to cast it right. You've got to have a great script. And you've got to be - and you've got to be telling a well-modulated human story throughout. Then you can start to layer on the visuals and the imaginative elements, you know, to any extent and certainly my stuff has been, you know, with an awful lot of that...
FLATOW: All right.
CAMERON: ...but you still have to have the basics first.
FLATOW: All right. Thank you very much for joining us, James Cameron, Tony DeRose, Steve Sullivan and Craig Barron, thanks to all for taking time to be with us today.
BARRON: Thank you.
FLATOW: We're going to take a short break. We'll be right back after the break. Stay with us.
(SOUNDBITE OF APPLAUSE)
FLATOW: I'm Ira Flatow. This is SCIENCE FRIDAY from NPR.
NPR transcripts are created on a rush deadline by Verb8tm, Inc., an NPR contractor, and produced using a proprietary transcription process developed with NPR. This text may not be in its final form and may be updated or revised in the future. Accuracy and availability may vary. The authoritative record of NPR’s programming is the audio record.