Speaker 1: Welcome to tech Stuff, a production of I Heart Radios
Speaker 1: How Stuff Works. Hey there, and welcome to text Stuff.
Speaker 1: I'm your host, Jonathan Strickland. I'm an executive producer with
Speaker 1: I Heart Radio and I love all things tech and
Speaker 1: it's time for another classic episode of tech Stuff. This
Speaker 1: episode originally published on December nineteen, two thousand and twelve.
Speaker 1: It is titled how Motion Capture works and I love
Speaker 1: mo cap when it's done well. When mo cap has
Speaker 1: done well, you can get some phenomenal performances translated into
Speaker 1: different types of media, including video games and c g
Speaker 1: I films. It is an amazing tool and Chris Pullett
Speaker 1: and I break down how it works. So let's listen
Speaker 1: in on this classic episode. So today we thought we'd
Speaker 1: talk a bit about a type of performance that is
Speaker 1: relatively new as far as performance goes. Uh, something that uh,
Speaker 1: I guess this falls into our movie making category, but
Speaker 1: it's also something that's been used in things like video
Speaker 1: games and and other forms of media as well. Motion capture. Yeah,
Speaker 1: you'll even see it in UH in sports. They've been
Speaker 1: talking about this for a while now. And if you've
Speaker 1: ever seen the making of a a video or a game. Um,
Speaker 1: or you know even in sports rehabilitation, in medicine, Um,
Speaker 1: they where the people are wearing dots, little white dots
Speaker 1: all over their clothing and sometimes their faces and hands. Um,
Speaker 1: that's probably what they were doing. Either that or they
Speaker 1: just really like stickers. Yeah, I mean who doesn't. I
Speaker 1: remember being very competitive in elementary school in order to
Speaker 1: get a sticker. And also this is a tangent but
Speaker 1: a true story. I got a gold star sticker, uh
Speaker 1: just last month, as from Tracy, the head of our
Speaker 1: our site. So anyway, um, yeah, motion capture. Actually, there
Speaker 1: are a lot of different terms that you can use
Speaker 1: in this in this realm uh, motion capture am CAP
Speaker 1: is probably the one I hear the most frequently, but
Speaker 1: also things like performance animation, performance capture, digital puppetry, real
Speaker 1: time animation, motion scanning, which is really more of a
Speaker 1: proprietary thing, but these are The concept is pretty much
Speaker 1: the same across the board. The idea is to capture
Speaker 1: the physical representation of something and then converted into a
Speaker 1: virtual format. So usually it's something that's in motion, but
Speaker 1: it's not always that way. Uh. Since you know we're
Speaker 1: talking about motion capture, that makes sense, but you're trying
Speaker 1: to get uh, translates something that is moving through real
Speaker 1: space into a digital format. And Uh, there's different ways
Speaker 1: to do this. I mean you could do it the
Speaker 1: really hard way, which is where you study something and
Speaker 1: then you try to recreate it, uh, either by hand
Speaker 1: or by or digitally, you know, by by programming movements
Speaker 1: into an animated figure. But this is an idea that
Speaker 1: kind of takes that step out where you are directly
Speaker 1: porting the movements uh, something is making within physical space
Speaker 1: into virtual space. Yeah, there was an early technique, UM,
Speaker 1: And of course this is this is all an attempt
Speaker 1: to to get as real as you can with with animation. UM.
Speaker 1: And one of the earlier techniques that that was sort
Speaker 1: of a predecessor to this is called rotoscoping. Uh. Ralph
Speaker 1: Box She's Lord of the Rings had a lot of
Speaker 1: rotoscoping in it. Well, what happens is UM, in that
Speaker 1: case is that a real uh, a real human being
Speaker 1: goes through the motions and they act through the parts
Speaker 1: that are that you're going to see in the animation,
Speaker 1: and they shoot that on film. Yes, yes, and then
Speaker 1: the the animators basically are looking at that and are
Speaker 1: drawing more or less on top of that, they see
Speaker 1: a projection of that, and they are drawing, uh, the
Speaker 1: animation over that to capture the way that person's body looks.
Speaker 1: And this this was famous. You know, the Disney studios
Speaker 1: were famous for this. We're studying models and then they
Speaker 1: would do the rotoscoping technique to to try to make
Speaker 1: their uh, their characters look more realistic. Yeah, and there
Speaker 1: are some artists, like I said, like Bakshi who famously
Speaker 1: would leave the film image as part of the animation,
Speaker 1: so that you had this this weird effect where the
Speaker 1: thing you were looking at was part uh well quote
Speaker 1: unquote real image and part animated image, which was it
Speaker 1: was an artistic choice, uh, definitely something that was not
Speaker 1: meant to to necessarily fool you into thinking, oh, well,
Speaker 1: that animated character is moving very realistically. It was done
Speaker 1: on purpose, but it was. That's what I always think
Speaker 1: of when I think rotoscope as I just think of
Speaker 1: the different box sheet films, but in particular I think
Speaker 1: of his Lord of the Rings adaptation, um, which, as
Speaker 1: I recall, ended halfway through the Two Towers. So anyway,
Speaker 1: that's just bringing back memories. But yeah, that was that
Speaker 1: was sort of a precursor to motion capture motion capture itself.
Speaker 1: There are many different ways of achieving this. For example,
Speaker 1: there were it's not used very frequently now, but there
Speaker 1: were mechanical systems where you had sensors that would be
Speaker 1: attached to specific joints UH that would relay movement. And
Speaker 1: usually it was kind of like a like an actor
Speaker 1: would wear a physical metallic skeleton type device that would
Speaker 1: have the sensors attached to the various joints, and as
Speaker 1: the actor moved, the sensors would register the changes in
Speaker 1: motion in this metallic skeleton and UH, and that would
Speaker 1: be relayed through usually cables to a computer system that
Speaker 1: would measure these or take the measurements from the sensors
Speaker 1: and translated into movements for the virtual character. UH. It's
Speaker 1: very limiting on this particular system. There was another one
Speaker 1: that was a little more versatile, which was used electro magnets.
Speaker 1: And in this case you talked about sensors that would
Speaker 1: be attached by really thick cables that again would go
Speaker 1: to a computer and there'd be a magnetic field and
Speaker 1: by moving through this magnetic field, the sensors would pick
Speaker 1: up alterations. You know, it would you know, moving through
Speaker 1: magnetic field, you would get little electrical changes. We we've
Speaker 1: talked a lot about electricity magnetism in general, moving through
Speaker 1: uh fluctuating a magnetic field can induce electricity through a conductor,
Speaker 1: or putting electricity through a conductor can induce a magnetic field.
Speaker 1: So anyway, by moving these sensors through the magnetic field,
Speaker 1: it would create these electronic fluctuations that would then be
Speaker 1: measured and translated into movement. And again this was a
Speaker 1: fairly effective way of picking up movements. It actually didn't
Speaker 1: use as many points of contact as the optical systems
Speaker 1: that we mostly think about, that was the kind that
Speaker 1: Chris was referring to early with all the dots on
Speaker 1: the person. Those systems tend to have lots and lots
Speaker 1: and lots of points of reference. The electro magnet ones
Speaker 1: didn't tend to have as many points of reference because
Speaker 1: the software side of it, because you know, we do
Speaker 1: have a hardware and a software side to this. The
Speaker 1: software side would assume that the joints that these sensors
Speaker 1: were attached to behaved the way they normally would in humans,
Speaker 1: and that they don't have complete freedom of movement. Most
Speaker 1: of us are not multi jointed in every joint, so
Speaker 1: we can't you know that we have a imitation on
Speaker 1: how far we can move in certain directions with these
Speaker 1: various joints. So taking that into account, you didn't have
Speaker 1: to have sensors all over the body. You would just
Speaker 1: have in a few places, which was good considering that
Speaker 1: there were these thick cables attached to the sensors. And
Speaker 1: then once you were done moving, then the all that
Speaker 1: data would get be captured within the system and could
Speaker 1: then be rendered into animation. Although this was also a
Speaker 1: way that you could do real time animation or digital puppetry. Uh,
Speaker 1: it's not that different from controlling a video game character
Speaker 1: with a controller. It's sort of the same principle, except
Speaker 1: in this case the the video game controller instead of
Speaker 1: being something you hold in your hands, it's something you
Speaker 1: were actually wearing. And uh, I've seen plenty of instances
Speaker 1: of this. If you've ever seen Turtle Talk with Crush
Speaker 1: over at Disney, that's what they use. They use a
Speaker 1: digital you know, they use digital poetry, and it's awesome,
Speaker 1: by the way, I love that. Well, it it would
Speaker 1: all so seeing that, um, you would need to be
Speaker 1: aware of where those cables were going, and it would
Speaker 1: it would also affect the way that you would move.
Speaker 1: You wouldn't move as naturally if you were wearing something
Speaker 1: like that as if you were, you know, unencumbered by
Speaker 1: by that, which, um sort of I think would wind
Speaker 1: itself to an upgrade, which is I think why they
Speaker 1: were so keen on optical system Well, it's also also
Speaker 1: that's very true. It did limit what you could do,
Speaker 1: it could live, It would limit your movement. I mean,
Speaker 1: when you've got these big cables attached to you, you you
Speaker 1: obviously you can't just move freely within a space. Um.
Speaker 1: So it did put some limitations on you. Their limitations
Speaker 1: to the optical systems too, but we'll get into that.
Speaker 1: The The other problem was that the sampling rate for
Speaker 1: the magnetic systems was not as high as it is
Speaker 1: for optical systems. And by sampling rate, what I mean
Speaker 1: is that this the entire system as a whole is
Speaker 1: taking little measure sure mints of from the sensors of
Speaker 1: you know, the orientation of those sensors within the space,
Speaker 1: and it does that several times every second. But the
Speaker 1: sample rate of the magnetic motion capture systems was much
Speaker 1: lower than what it was for than what it would
Speaker 1: be if you were to use an optical system. So
Speaker 1: you're not getting data as frequently, I mean still several
Speaker 1: times a second, but it's not as precise as the
Speaker 1: optical system. So not only were you limited in the
Speaker 1: kind of movements you can make because you had these
Speaker 1: major cables attached to you, but also you couldn't get
Speaker 1: really minute precise measurements on every kind of movement. So
Speaker 1: it wasn't good for things like sports. So you know,
Speaker 1: something like throwing a pitch in baseball, there are a
Speaker 1: lot of movements, a little tiny motions that are involved
Speaker 1: in that. I mean, anyone who's watched slow motion footage
Speaker 1: of a professional baseball pitcher throwing a pitch, you can
Speaker 1: see that there are some incredibly subtle movements that are
Speaker 1: involved in that. And uh, and it takes place over
Speaker 1: a very short period of time. I mean, it's a
Speaker 1: very fast thing to to to measure. Using the magnetic
Speaker 1: motion capture system, you would probably one slow the person
Speaker 1: down because they have all these cables attached to them,
Speaker 1: and to not get enough data to give an accurate
Speaker 1: representation of what had happened in the virtual format. So
Speaker 1: if you were to say, create a video game, a
Speaker 1: baseball video game, the picture would not necessarily behave properly
Speaker 1: if all you did was directly port the data you
Speaker 1: got from the motion capture into the game. Yeah. Another
Speaker 1: drawback of the mechanical systems like that too, is that
Speaker 1: that it's UM. It's the kind of system that not
Speaker 1: only is cumbersome and inaccurate, but it has to be
Speaker 1: calibrated fairly frequently. UM. And you know, there there's there's
Speaker 1: some work that you can do with this. But the
Speaker 1: optical systems that they began to introduce UM, you know,
Speaker 1: generally became an upgrade UM. The only there is one
Speaker 1: big advantage that the mechanical systems do have, though, and
Speaker 1: that is that light. The lighting will not necessarily interfere
Speaker 1: with the different points of motion that are captured by
Speaker 1: the mechanical system UM. And that can be an issue
Speaker 1: with the optical systems UM. You know, because that's that's
Speaker 1: why UM. They will be wearing the people, the actors
Speaker 1: who will be UM having their motions captured by the
Speaker 1: system will be wearing you know, those bright dots so
Speaker 1: that the computer can pick up on that. And at
Speaker 1: the beginning, and these are these early systems, there were
Speaker 1: only so many points action points that they could capture. Um.
Speaker 1: They were very limited in what they could do at first,
Speaker 1: but still you know, somewhat of an upgrade over the
Speaker 1: mechanical Yeah. It also limited what you could have in
Speaker 1: the background, obviously, because you could not have anything that
Speaker 1: was going to be of a similar shade UH. You
Speaker 1: know usually where you're talking about reflective white substance used
Speaker 1: as the um the points of of UH articulations. So
Speaker 1: the little like white stickers is like what you were saying, Chris, Um,
Speaker 1: you couldn't have anything like that in the background because
Speaker 1: it would confuse the optical system. So that's why a
Speaker 1: lot of these motion capture scenes are shot against a
Speaker 1: blue screen or green screen. It's so that the background
Speaker 1: does not in any way interfere with the motion capture.
Speaker 1: So if you've ever seen behind the scenes footage of
Speaker 1: The Lord of the Rings movies is a great example
Speaker 1: with Andy Sarkis as Gollum or Sniegel if you prefer,
Speaker 1: But he's wearing you know, a tight like skin tight
Speaker 1: suit with these little white UH circles all over it.
Speaker 1: Those are the points that the camera track to create
Speaker 1: the the performance of gallam slash Sniegel. So the performance
Speaker 1: is something that's being created not only by the actor
Speaker 1: but also the animators because not we should also point
Speaker 1: out that the motion capture stuff rarely is motion capture
Speaker 1: uh completely. Uh there's there's rarely a moment where you
Speaker 1: don't have an animator step in and tweak it somehow,
Speaker 1: Like you don't normally have someone create a physical performance
Speaker 1: and that physical performance is completely without any tinkering represented
Speaker 1: in the final product. I mean it can happen, there
Speaker 1: are instances of it, but it's more frequently, uh, something
Speaker 1: where the motion capture performance goes to the animator who
Speaker 1: can then tweak things if the performance is not exactly
Speaker 1: what needs to be, which is kind of nice. You
Speaker 1: don't necessarily have that luxury with flesh and blood actors.
Speaker 1: I'm gonna stop motion ure in mid motion right now
Speaker 1: so that we can take a quick break to thank
Speaker 1: our sponsor. Well, especially with the earlier systems, especially the
Speaker 1: electromagnetic systems, Uh, those were really noisy, not literally noisy,
Speaker 1: but digital noise. They weren't they weren't really highly accurate. Um.
Speaker 1: The optical systems are are far cleaner and give them
Speaker 1: more accurate representation. But you know that that's it sort
Speaker 1: of falls in the realm of artistic license. I would think, um,
Speaker 1: where they need to go in and make subtle adjustments
Speaker 1: to make it look the way they wanted to look. Ye.
Speaker 1: I should also point out, now you just reminded me
Speaker 1: of something else another drawback to the electromagnetic systems, which
Speaker 1: was you couldn't have anything metal on the set because
Speaker 1: it would interfere with that magnetic field and give incorrect
Speaker 1: readings to the system. So you're you're virtual character would
Speaker 1: not move in the same way as the physical one
Speaker 1: because there would be some interference in that sense. So
Speaker 1: your set couldn't have anything metal in it. The props
Speaker 1: didn't shouldn't have anything metal in them, so that that
Speaker 1: limited to you as well. So each system has its
Speaker 1: own limitations. Getting back to the optical one, UM, one
Speaker 1: of the other things you have to remember is that
Speaker 1: in order to really capture a a physical object moving
Speaker 1: through three D space and to replicate that in virtual space,
Speaker 1: you need multiple cameras in that system because a single camera,
Speaker 1: assuming that's a regular video or film camera, something that
Speaker 1: does not have three D capability, pointing that at an
Speaker 1: object it's creating a two dimensional image of something that's
Speaker 1: moving in three dimensions. The camera can't necessarily tell where
Speaker 1: movements are happening within the depth frame of of that
Speaker 1: of that image, right, So if someone's moving in such
Speaker 1: a way where let's say they're moving their head where
Speaker 1: it would be bobbing closer to the camera. Uh, unless
Speaker 1: the size of the the sensors is such that something
Speaker 1: that's subtle could be picked up by the camera system,
Speaker 1: you would lose that information. So what you need are
Speaker 1: multiple cameras on the same object so that you can
Speaker 1: compare that data from the multiple angles to tell how
Speaker 1: this object is really moving through this three dimensional space.
Speaker 1: So it's kind of like the idea of having parallax
Speaker 1: with two eyes. You know, our eyes are offset, so
Speaker 1: by looking at an object, we can tell how far
Speaker 1: away it is in part because of parallax. Uh. We
Speaker 1: also have other visual cues that tell us about how
Speaker 1: far something is, you know, things like how tall it
Speaker 1: is in relation to where we are that kind of thing,
Speaker 1: or how tall it is in relation to other objects
Speaker 1: that are within our frame of vision. But parallax is
Speaker 1: very importan and same sort of thing with these optical systems,
Speaker 1: you would have multiple cameras set up to try and
Speaker 1: capture the information that's going on in the frame so
Speaker 1: that you could tell exactly how it's moving through that
Speaker 1: three dimensional space. Yeah, it seems like um, in order
Speaker 1: to capture the correct perspective, you need that additional information,
Speaker 1: even though you may not necessarily see it. UM. It
Speaker 1: helps the the animator do that, and the optical system
Speaker 1: to allows you to work with more than one actor um,
Speaker 1: which was not really an option with some of the
Speaker 1: earlier systems. So in other words, you can although it
Speaker 1: requires more equipment, you know, just simply out of necessity,
Speaker 1: the optical system is really affording the animators a an
Speaker 1: opportunity to use a greater amount of information um both
Speaker 1: you know, from the different the different points of data
Speaker 1: they're getting from a single actor, but from multiple actors
Speaker 1: on this set simultaneously, which enables them to to create
Speaker 1: more complex work. Right and uh. This also gives us
Speaker 1: a good example of how the optical motion capture systems
Speaker 1: are a passive system because you have these sensors you're
Speaker 1: wearing that are not necessarily or not even sensors, they're
Speaker 1: they're reflective markers that you're wearing. They aren't connected to
Speaker 1: any sort of electronic components at all, versus the active
Speaker 1: systems like the electromagnetic one where you are generating data
Speaker 1: by moving through a magnetic field and you have these
Speaker 1: big cables attached to it. Uh with the optical motion
Speaker 1: capture systems. And another thing that's kind of interesting I
Speaker 1: think is that a lot of at least the early ones,
Speaker 1: the cameras would have infrared l e ed s uh
Speaker 1: so emitters. Really that we're emitting infrared lights that's outside
Speaker 1: are our visible spectrum. We cannot see infrared light. But
Speaker 1: by putting an infrared filter on the camera, you could
Speaker 1: have the camera pick up reflections of infrared light. And
Speaker 1: that was a way of helping to identify the sensors
Speaker 1: that you had put on the actor. The actors, the
Speaker 1: sensors would be reflective specifically so that the infrared light
Speaker 1: would reflect back toward the camera and give the most
Speaker 1: accurate rendering of what's going on at any given moment
Speaker 1: within a scene. So, um, yeah, it's another way of
Speaker 1: making sure that the data being captured is as precise
Speaker 1: as possible. I mean, that is, of course, the goal
Speaker 1: is to try and recreate the physical movements as truthfully
Speaker 1: as you possibly can given all the limitations involved. Yeah,
Speaker 1: and if you're looking for a real life easy to
Speaker 1: find an example of this, you would look no farther
Speaker 1: than your local video game store. Um, because the Xbox
Speaker 1: connect uses very much that that exact form of technology
Speaker 1: is using an infrared emitter UM, and it has cameras
Speaker 1: that it uses to pick it up uh, the information,
Speaker 1: pick the information up that is coming back from what
Speaker 1: is being reflected around the room. And anybody who who
Speaker 1: has one is also aware that lighting is very much
Speaker 1: an issue. UM. The way that were room is let
Speaker 1: affects the information that the connect is able to refer
Speaker 1: to the Xbox. Now, it's not, while it is sophisticated,
Speaker 1: is not as sophisticated as the kind of equipment that
Speaker 1: they might use in making a movie or making a
Speaker 1: video game. But it is very very similar technology, and
Speaker 1: in some ways I would argue that it's more sophisticated
Speaker 1: than some of those early UH systems simply because it
Speaker 1: is able to capture a lot of information. UM. Whereas
Speaker 1: you know, the very early optical systems were only using
Speaker 1: a handful of data points, right, So UM, it's it's
Speaker 1: a pretty neat device. Um, you know, not only used
Speaker 1: for gaming. Now the hacker community has fallen in love
Speaker 1: with it too because it can do so much and
Speaker 1: can be used for so many things and is you know,
Speaker 1: fairly inexpensive. Yeah. The cool thing about the Connect is
Speaker 1: that rather than have to obviously, if you've if you've
Speaker 1: ever played an Xbox with a connect, you know you
Speaker 1: don't have to go out and buy a snug body
Speaker 1: suit covered in reflective markers in order to play. I mean,
Speaker 1: it doesn't hurt, but you know, if you're if you
Speaker 1: can pull that look off. There are very few of
Speaker 1: us who can. I count myself among them. But you
Speaker 1: don't have to do that because what it's doing is
Speaker 1: it's actually projecting essentially a grid uh in infrared light,
Speaker 1: so you can't see the grid, but it's being projected
Speaker 1: into the room, and then when you move uh within
Speaker 1: the space, you are deforming that grid. You know, the
Speaker 1: camera that's picking up the reflections of that infrared light
Speaker 1: can detect when the grid's being deformed by a physical
Speaker 1: object interrupting the grid. So as you move, you interrupt
Speaker 1: different parts of the grid, and it can start to
Speaker 1: interpret those as motions and commands. It's not uh, it's
Speaker 1: not as precise as what we're talking about with the
Speaker 1: optical systems that are used in movies and video games. Uh,
Speaker 1: to to create them, that is, not to to play them. Um,
Speaker 1: it's not as precise as those. But it also has
Speaker 1: other elements that help balance it out, Like it has
Speaker 1: regular optical cameras that can have some other software that
Speaker 1: aids it in recognizing things like facial recognition software, which
Speaker 1: does not necessarily rely upon that infrared grid. It relies
Speaker 1: more on the traditional camera functions, but has the software
Speaker 1: included that lets the the programs within recognize who is
Speaker 1: standing in front of it, so that combination increases the precision,
Speaker 1: which of course is very important whenever you're playing a game.
Speaker 1: I mean, anyone who's played any sort of game where
Speaker 1: you're using a faulty controller, or it's just a system
Speaker 1: that hasn't been fully uh it's not finished yet, it's
Speaker 1: just in prototypes, dage or whatever. You may have noticed
Speaker 1: that it could be very frustrating to try and control
Speaker 1: something where the actual controller is not as responsive as
Speaker 1: you would hope. It's um not a fun experience, but anyway,
Speaker 1: that is kind of related to this whole motion capture technology. UM.
Speaker 1: I'm sorry what you were You look like you have
Speaker 1: something to say, Well, no, I was, I was going
Speaker 1: to say that. UM. You know, we really hadn't other
Speaker 1: than my earlier statement about sports. UM. You know, we've
Speaker 1: we've been talking about it in an entertain amount entertainment
Speaker 1: about the the ability to capture motion to make characters
Speaker 1: more realistic. And UM that that is exactly what they
Speaker 1: want to do when they are using this in sports medicine. UM.
Speaker 1: Jonathan alluded to earlier the difficulty in uh in capturing
Speaker 1: all the little subtle motions that go into um, into
Speaker 1: a Major League Baseball players pitching. UM. And you know
Speaker 1: when somebody, when somebody gets hurt, UM, sometimes they go
Speaker 1: through uh extensive surgery. The Tommy John procedures is UH famous.
Speaker 1: You know, they do a ligament transplant to to help
Speaker 1: rebuild a picture's elbow, and that can really throw off, um,
Speaker 1: the mechanics of a pictures motion. So they use this
Speaker 1: motion capture technology to really get an idea of how, um,
Speaker 1: how that person is throwing going about the mechanics of
Speaker 1: their typical game play. And and that's exactly the same
Speaker 1: kind of thing that they're doing when they create these
Speaker 1: very realistic sports games. UM. But you know, in this case,
Speaker 1: they're using it for sports medicine to see if they can, uh,
Speaker 1: they can go back and recreate some of the motions
Speaker 1: that made them so successful before they were injured. Now. UM. Ironically,
Speaker 1: in in UH entertainment purposes, especially video, UM, you can
Speaker 1: get too realistic. UM. The Japanese professor massa Hiro Mori
Speaker 1: is famous for his Uncanny Valley UM, which has been
Speaker 1: used in uses a robotics term for a robot that
Speaker 1: looks so much and and moves so much like a
Speaker 1: human that it it creeps us out. It looks a
Speaker 1: little too realistic. And I can think of we're actually
Speaker 1: recording this in December of and UM. One of the
Speaker 1: movies that comes on about this time of year is
Speaker 1: The Polar Express, which is it's a nightmare known loved
Speaker 1: and reviled both for its story and it's um and
Speaker 1: the way that they use motion capture because the characters
Speaker 1: and they are so realistic, they're downright creepy. Yeah. It's
Speaker 1: it's one of those things where they are almost but
Speaker 1: not quite able to pass for a real person, so
Speaker 1: that there's just enough off about them to be unsettling. Now,
Speaker 1: this does bring up something else that's kind of interesting.
Speaker 1: We have an article on how stuff works dot com
Speaker 1: about motion scan technology, which is, as I said earlier,
Speaker 1: a proprietary technology. It's it's more specific than just motion captured.
Speaker 1: Specifically meant to capture facial motion activity. So when an
Speaker 1: actor is speaking, when they're delivering lines, the way that
Speaker 1: they furrow their brow or move their eyes or smile,
Speaker 1: or they give a facial tick, anything like that, this
Speaker 1: system is designed to pick that up so that it
Speaker 1: can be recreated virtually in a game, and it was
Speaker 1: used to great effect, in my opinion, in l A Noir.
Speaker 1: Le Noir was a video game that came out in
Speaker 1: two thousand eleven, and it was a game in which
Speaker 1: you played a well, you played a couple of different characters,
Speaker 1: but the one you played for most of the game
Speaker 1: spoiler alert was was a a police detective. And you're
Speaker 1: kind of rising through the ranks uh in L A
Speaker 1: h during the uh early part of the twentieth century,
Speaker 1: and it's it's um. It's notable in that you are uh,
Speaker 1: You're spending most of the game looking at people's reactions.
Speaker 1: You know, the idea behind l A Noir. It was
Speaker 1: a new type of video game where you would interrogate
Speaker 1: characters throughout your investigations, and as you interrogate them, you
Speaker 1: had to watch the characters facial reactions to kind of
Speaker 1: get an idea of whether the character was trying to
Speaker 1: be evasive or if they were telling the truth. And
Speaker 1: you would do things like watched for their eyes, and
Speaker 1: if they weren't able to maintain eye contact, that was
Speaker 1: an indication that perhaps they were being less than truthful.
Speaker 1: Or if they would, you know, twitch their mouth or
Speaker 1: clench their jaw, these would be little little pints that
Speaker 1: perhaps there's more going on than what they're letting onto.
Speaker 1: And obviously, if your gameplay depends upon trying to determine
Speaker 1: whether or not a virtual character is telling the truth,
Speaker 1: you have to be able to represent those facial expressions
Speaker 1: as closely to reality as possible, or else the game
Speaker 1: does not work. So they used this motion scan technology,
Speaker 1: and the way that they did this was that they
Speaker 1: had a very brightly lit studio that had lights trained
Speaker 1: on an actor from just about every angle, and the
Speaker 1: purpose of that was to try and eliminate shadows because
Speaker 1: any sort of shadows you would have there would of
Speaker 1: course affect the actual capture. It was really all about
Speaker 1: the light. And they used thirty two high definition cameras,
Speaker 1: So think about that, thirty two high definition cameras just
Speaker 1: to capture and actor's facial performance like that's it. There,
Speaker 1: there's no other move meant the actor is seated at
Speaker 1: the time and um and had to remain as still
Speaker 1: as possible and just do all the acting with their face,
Speaker 1: which for anyone out there who's done any sort of acting,
Speaker 1: you know, that's incredibly challenging because actors are trained to
Speaker 1: use their whole body when they are performance making a performance.
Speaker 1: They're trained to to really think about movement. I mean,
Speaker 1: if you're if you're really serious about acting, you've probably
Speaker 1: taken movement classes. And to suddenly have all of that
Speaker 1: taken away and all of your acting is restricted to
Speaker 1: just your face, it's pretty that's pretty dramatic. It's tough
Speaker 1: to do, but anyway, that's what the actors had to do.
Speaker 1: They had to sit down and and restrict their acting
Speaker 1: to just their facial expressions without it going like over
Speaker 1: the top crazy, because that would be just as distracting
Speaker 1: as not enough performance at all. And these thirty two
Speaker 1: cameras were paired up, so six pairs of cameras. There's
Speaker 1: technically there was a thirty third camera as well that
Speaker 1: the director used to watch the scene and give directions
Speaker 1: to the actors. Um. But these these pairs of cameras
Speaker 1: were trained on all these different angles of the face
Speaker 1: in order to capture that that performance so that in
Speaker 1: the virtual world they could recreate it accurately, which to
Speaker 1: me is phenomenal. And apparently the way the system works
Speaker 1: is you get that virtual version of the person's face
Speaker 1: and head almost instantly, which is kind of creepy but
Speaker 1: also awesome. Chris Bilette and I have a little bit
Speaker 1: more to say about how motion capture works, but first
Speaker 1: let's take another quick break. It's funny too that they
Speaker 1: used that many cameras in the creation of a video game, because, uh,
Speaker 1: as elsewhere in that article that notes that um Circus,
Speaker 1: who was playing Gollum Um only had only had cameras
Speaker 1: on on him, but in doing so, they were able
Speaker 1: to uh to create roughly, you know, ten thousand different
Speaker 1: kinds or identify ten thousand different kinds of facial movements
Speaker 1: that they could use in animating the character on screen.
Speaker 1: So um, clearly, uh, you know, this is very very
Speaker 1: high tech and painstaking procedure to do, but in doing
Speaker 1: so they can they can create very very realistic movements. Yeah,
Speaker 1: there's a lot of number crunching involved, and frankly, the
Speaker 1: the part that takes place after you've captured the data
Speaker 1: is can be dramatically different from one case to the next.
Speaker 1: In some cases, you may have already created uh, an
Speaker 1: animated figure any much from start to finish. You might
Speaker 1: not have completely put textures on it or or something,
Speaker 1: but you might have essentially the way the character is
Speaker 1: going to look in the finished product, uh, and then
Speaker 1: you just map it to the movements that you've captured
Speaker 1: and it's and there it goes. And in other cases
Speaker 1: you might see that what they do is they capture
Speaker 1: the motions and then you essentially have what looks like
Speaker 1: a very primitive stick figure skeleton that moves in the
Speaker 1: way that the actor moved, but there's no definition, there's
Speaker 1: no character there yet, And you may have animators who
Speaker 1: build the character somewhat based upon the way the actor
Speaker 1: moved through the space, so that perhaps the character's design
Speaker 1: is not finalized until you've captured that that performance, and
Speaker 1: the performance helps guide the design of the character. It
Speaker 1: all depends on the specific technology that's being used and
Speaker 1: the preference of the crew that's that's designing whatever it
Speaker 1: is that they're making. There's a video game or a movie,
Speaker 1: TV show, commercial, whatever it happens to be. Uh. In
Speaker 1: the case of digital puppetry, obviously you would already have
Speaker 1: the the full character realized, so that just by using
Speaker 1: whatever control mechanism happens to be there, you would be
Speaker 1: able to make the puppet move in real time. Otherwise
Speaker 1: it's not really puppetry. Um. And again that's sort of
Speaker 1: like the if you've been to that that turtle talk
Speaker 1: thing I talked about, the Disney World or Disneyland. UM.
Speaker 1: I'm sure there are other similar ones. I think Monsters Inc.
Speaker 1: Laugh Factory has a similar setup where you've got a
Speaker 1: digital character on a screen that can react in real
Speaker 1: time to things that are happening within the physical environment.
Speaker 1: So they interact with the audience like they'll specifically single
Speaker 1: people out and chat with people in the audience. And um,
Speaker 1: to two kids, this is amazing. I mean, it's a
Speaker 1: cartoon character acting in real time. It's a real person now, Uh.
Speaker 1: Two adults. It's fascinating because they're like, how the heck
Speaker 1: did that happen? Um? But yeah, that's it's all based
Speaker 1: on the same sort of technology. UM. And it's it's
Speaker 1: really interesting to me to see how the field is
Speaker 1: evolving over time because things like the connect show that
Speaker 1: we are adapting the same sort of technology in different ways.
Speaker 1: We're using different implementations to essentially do the same thing,
Speaker 1: and that perhaps we will get to a point where
Speaker 1: we won't have to worry about all the sensors so much. Um.
Speaker 1: You can maybe have an actor who's not completely coded
Speaker 1: and stickers perform and and you could capture all that
Speaker 1: data without having to worry about, you know, tracking these
Speaker 1: little dots that might be something that we've see in
Speaker 1: the future. I mean the motion scan is kind of
Speaker 1: like that, because before motion scan with that facial acting
Speaker 1: uh technology, Uh, whenever I saw anyone who was having
Speaker 1: their face tracked for a performance, they always were wearing
Speaker 1: those tiny little white stickers all over their face to track.
Speaker 1: I mean, we've got a lot of muscles in our face.
Speaker 1: There's something like nineteen muscles or something that you have
Speaker 1: to track, so um, you would have all these little
Speaker 1: dots on your face to track those motions. Well, with
Speaker 1: motion scan you don't need those anymore. So maybe we'll
Speaker 1: see something like that. Of course, so that would really
Speaker 1: depend upon perhaps the lighting, which could if you're shooting
Speaker 1: a virtual character that's next to real characters like in
Speaker 1: The Lord of the Rings real being. I guess you know,
Speaker 1: your mileage may vary. I mean, they're hobbits. But anyway,
Speaker 1: when you're next to real people, clearly you can't mess
Speaker 1: with the lighting too much or it'll just make the
Speaker 1: whole scene look strange. Speaking of strange, UM, while you
Speaker 1: might think that the techniques used in motion capture, um
Speaker 1: you know, bringing film into it, you know, adding a
Speaker 1: lot of advancement to to film, um basically uh, some
Speaker 1: people sort of regardless as cheating. Yeah. I I did
Speaker 1: some research that that indicated that although some other types
Speaker 1: of animation are considered you know, considered more artful, UM,
Speaker 1: motion capture is sort of not everyone. But some people say, well,
Speaker 1: you know, it's it's not Oscar worthy because you were
Speaker 1: using these computer add animation techniques that that really um
Speaker 1: are simulating human motion and it's just it's just not real.
Speaker 1: And the argument that I've seen used against it is, well,
Speaker 1: you consider rotoscoping, okay, why don't you consider motion capture,
Speaker 1: which is a kind of descendant from this technology. Why
Speaker 1: why isn't that okay to uh, you know, to consider
Speaker 1: for um quality and and and for rewards. But um,
Speaker 1: apparently it's a it's sort of a hot topic among
Speaker 1: um among movie makers. Yeah, I can see why animator
Speaker 1: a traditional animator or even a computer animator. I mean
Speaker 1: that's closer and closer to becoming traditional already, but either
Speaker 1: a hand drawn animation or computer animation. Someone who goes
Speaker 1: through the trouble of animating these things and doing a
Speaker 1: lot of this work. Uh by hand seems like it's
Speaker 1: the wrong term, but but personally going through and creating
Speaker 1: these performances, I can see where they might feel that way. Um.
Speaker 1: I have a completely different perspective on it. Of course,
Speaker 1: I'm not an animator, so that's part of it. But
Speaker 1: I think of it as creating a performance. And in
Speaker 1: the sense of creating a performance, I think it's a
Speaker 1: completely legitimate tool because you're still relying on an actor
Speaker 1: to create a performance. That that that people will relate
Speaker 1: to whether it's a character that you're supposed to love
Speaker 1: or hate or fear. That all is dependent upon the
Speaker 1: animator and the actor and several other people working to
Speaker 1: create this this performance. And uh, I don't see anything
Speaker 1: wrong with that. That to me is a completely legitimate
Speaker 1: form of creating the art of entertainment. So um, I mean,
Speaker 1: I do understand from an artistic perspective where some people
Speaker 1: could have a problem with it. But but if you
Speaker 1: take a bigger picture look and not not just you
Speaker 1: know what technique you're using, but the end goal of creating,
Speaker 1: whether you want to call it art or not, but
Speaker 1: creating something that has an impact to the viewer or
Speaker 1: player in the case of a video game, I think
Speaker 1: that's more important. But then again, I'm, like I said,
Speaker 1: I'm not an animator, so I don't have that kind
Speaker 1: of emotional attachment, you know, I'm not vested in it
Speaker 1: in that way. So um, I'd be curious to hear
Speaker 1: what our listeners think if they think that is motion capture?
Speaker 1: Is that cheating? Is it? Uh? Is it, as Red
Speaker 1: versus Blue would have you say, a legitimate strategy? What
Speaker 1: what do you think? What do you consider a motion
Speaker 1: capture you should less know. Yeah, I UM, I do
Speaker 1: see where UM it might make a traditional animator concerned,
Speaker 1: but I don't. I don't really think it diminishes their UM,
Speaker 1: their artistic value, to to UM, to a work whatever
Speaker 1: it may be that they are working on. UM. And
Speaker 1: there are certain times I'm sure where uh you would
Speaker 1: argue that using these techniques is completely inappropriate to what
Speaker 1: they might do. UM. But yeah, I mean it's it's
Speaker 1: always a concern when UM you start saying, well, the
Speaker 1: machine can do it, and we don't really need people
Speaker 1: to do it, so get out. Yeah, I don't think
Speaker 1: that's ever gonna be UM always fully the case, because
Speaker 1: you're going to have certain characters within movies that are
Speaker 1: going to be so different from the way humans are built,
Speaker 1: so to speak, that that, uh, that motion capture would
Speaker 1: not be practical. For example, like let's say that the
Speaker 1: character that you're creating has really super long arms, and
Speaker 1: you know, you've got an actor who's pretty lanky, but
Speaker 1: but their arms are not as long as the character's arms. Uh.
Speaker 1: If you were just to a direct translation of the
Speaker 1: actor's movements into the animation, it might not look right
Speaker 1: because the character has different dimensions, their body is built
Speaker 1: differently than the actor, and so without tweaking it, without
Speaker 1: having an animator go in there and adjust this and
Speaker 1: make it look correct compared to what the you know,
Speaker 1: the the vision is for the movie, it doesn't come
Speaker 1: out correctly, it doesn't look right. So I think there's
Speaker 1: very little risk of motion capture ever taking that away completely.
Speaker 1: Plus there is something too, you know, creating a performance
Speaker 1: through traditional animation that you know, it does feel differently
Speaker 1: the motion capture, but that's not a bad thing, like
Speaker 1: it just depends upon the vision of the director and
Speaker 1: what the tone of the piece needs to be. And
Speaker 1: that wraps up another classic episode of text Stuff. Hope
Speaker 1: you guys enjoyed it. If you have any suggestions for
Speaker 1: future episodes of tech Stuff, feel free to get in
Speaker 1: touch with me. You can send an email to tex
Speaker 1: Stuff at how stuff Works dot com, or you can
Speaker 1: drop me a line on Facebook or Twitter to handle it.
Speaker 1: Both of those is text Stuff HSW. You can pop
Speaker 1: on over to our website that's tech Stuff Podcast dot com.
Speaker 1: You're gonna find a link to our archive where we
Speaker 1: have every episode we've ever published right there searchable, so
Speaker 1: we can go check that out, and you can also
Speaker 1: find a link to our online store, where every purchase
Speaker 1: you make goes to help the show. We greatly appreciate it,
Speaker 1: and I'll talk to you again. Releasing text Stuff is
Speaker 1: a production of I Heart Radio's How Stuff Works. For
Speaker 1: more podcasts from I heart Radio, visit the i heart
Speaker 1: Radio app, Apple Podcasts, or wherever you listen to your
Speaker 1: favorite shows.