Speaker 1: Brought to you by the two thousand and twelve Toyota Camera.
Speaker 1: Get in touch with technology with tech Stuff from how
Speaker 1: stuff works dot com. Hello again, everyone, and welcome to
Speaker 1: tech stuff. My name is Chris Poulette, and I'm an
Speaker 1: editor at how stuff works dot com. Sitting across from
Speaker 1: me as always as senior writer Jonathan Strickland. Hey there,
Speaker 1: So today we thought we'd talk a bit about a
Speaker 1: type of performance that is relatively new as far as
Speaker 1: performance goes. Uh, something that uh, I guess this falls
Speaker 1: into our movie making category, but it's also something that's
Speaker 1: been used in things like video games and and other
Speaker 1: forms of media as well. Motion capture. Yeah, you'll even
Speaker 1: see it in uh, in sports, they've been talking about
Speaker 1: this for a while now. And if you've ever seen
Speaker 1: the making of a a video or a game, um,
Speaker 1: or you know, even in sports rehabilitation in medicine, um,
Speaker 1: they where the people are wearing dots, little white dots
Speaker 1: all over their clothing and sometimes their faces and hands. UM,
Speaker 1: that's probably what they were doing. Either that or they
Speaker 1: just really like stickers. Yeah, yeah, I mean, who doesn't.
Speaker 1: I remember being very competitive in elementary school in order
Speaker 1: to get a sticker. And also this is a tangent
Speaker 1: but a true story. I got a gold star sticker
Speaker 1: uh just last month awes from Tracy, the head of
Speaker 1: our our site. So anyway, um, yeah, motion capture. Actually,
Speaker 1: there are a lot of different terms that you can
Speaker 1: use in this in this realm Uh, motion capture or
Speaker 1: mo cap is probably the one I hear the most frequently,
Speaker 1: but also things like performance animation, performance capture, digital puppetry,
Speaker 1: real time animation, motion scanning, which is really more of
Speaker 1: a proprietary thing, but these are The concept is pretty
Speaker 1: much the same across the board. The idea is to
Speaker 1: capture the physical representation of something and then converted into
Speaker 1: a virtual format. So usually it's something that's in motion,
Speaker 1: but it's not always that way. Uh, since you know
Speaker 1: we're talking about motion capture, that makes sense. But you're
Speaker 1: trying to get uh, translate something that is moving through
Speaker 1: real space into a digital format. And uh, there's different
Speaker 1: ways to do this. I mean, you could do it
Speaker 1: the really hard way, which is where you study something
Speaker 1: and then you try to recreate it, uh, either by
Speaker 1: hand or by or digitally, you know, by by programming
Speaker 1: movements into an animated figure. But this is an idea
Speaker 1: that kind of takes that step out where you are
Speaker 1: directly porting the movements. Uh, something is making within physical
Speaker 1: space into virtual space. Yeah, there was an early technique, um.
Speaker 1: And of course this is this is all an attempt
Speaker 1: to get as real as you can with of animation. UM.
Speaker 1: And one of the earlier techniques that that was sort
Speaker 1: of a predecessor to this is called rotoscoping. Uh. Ralph
Speaker 1: Box She's Lord of the Rings had a lot of
Speaker 1: rotoscoping in it. Well, what happens is, um, in that
Speaker 1: case is that a real uh, real human being goes
Speaker 1: through the motions and they act through the parts that
Speaker 1: are that you're going to see in the animation. They
Speaker 1: shoot that on film, yes, yes, and then the the
Speaker 1: animators basically are looking at that and are drawing more
Speaker 1: or less on top of that. They see a projection
Speaker 1: of that, and they are drawing, uh, the animation over
Speaker 1: that to capture the way that person's body looks. And
Speaker 1: this this was famous. You know, the Disney studios were
Speaker 1: famous for this. We're studying models and then they would
Speaker 1: do the rotoscoping technique to to try to make their uh,
Speaker 1: their characters look more realistic. Yeah. And there are some artists,
Speaker 1: like I said, like box She who famously would leave
Speaker 1: the film image as part of the animation, so that
Speaker 1: you had this this weird effect where the thing you
Speaker 1: were looking at was part uh well quote unquote real
Speaker 1: image and part animated image, which was it was an
Speaker 1: artistic choice, uh, definitely something that was not meant to
Speaker 1: to necessarily fool you into thinking, oh, well, that animated
Speaker 1: character is moving very realistically. It was done on purpose,
Speaker 1: but it was. That's what I always think of when
Speaker 1: I think rhodoscoping, as I just think of the different
Speaker 1: box Sheet films, but in particular I think of his
Speaker 1: Lord of the Rings adaptation um, which, as I recall,
Speaker 1: ended halfway through the Two Towers. So anyway, that's just
Speaker 1: bringing back memories. But yeah, that was that was sort
Speaker 1: of a precursor to motion capture. Motion capture itself. There
Speaker 1: are many different ways of achieving this. For example, there
Speaker 1: were it's not used very frequently now, but there were
Speaker 1: mechanical systems where you had sensors that would be attached
Speaker 1: to specific joints uh that would relay movement. And usually
Speaker 1: it was kind of like a like an actor would
Speaker 1: wear a physical metallic skeleton type device that would have
Speaker 1: the sensors attached to the various joints and as the
Speaker 1: actor moved, the sensors would register the changes in motion
Speaker 1: in this metallic skeleton and UH, and that would be
Speaker 1: relayed through usually cables to a computer system that would
Speaker 1: measure these or take the measurements from the sensors and
Speaker 1: translated into movements for the virtual character. UH. It's very
Speaker 1: limiting this particular system. There was another one that was
Speaker 1: a little more versatile, which was used electro magnets. And
Speaker 1: in this case you talked about sensors that would be
Speaker 1: attached by really thin cables that again would go to
Speaker 1: a computer, and there'd be a magnetic field and by
Speaker 1: moving through this magnetic field, the sensors would pick up alterations.
Speaker 1: You know, it would you know, moving through magnetic field,
Speaker 1: you would get little electrical changes. We we've talked a
Speaker 1: lot about electricity magnetism in general, moving through UH, Fluctuating
Speaker 1: a magnetic field can induce electricity through a conductor, or
Speaker 1: putting electricity through a conductor can induce a magnetic field.
Speaker 1: So anyway, by moving these sensors through the magnetic field,
Speaker 1: it would create these electronic fluctuations that would then be
Speaker 1: measured and translated into movement, and again this was a
Speaker 1: fairly effective way of picking up movements. It actually didn't
Speaker 1: use as many points of contact as the optical systems
Speaker 1: that we mostly think about. That was the kind that
Speaker 1: Chris was referring to early with all the dots on
Speaker 1: the person. Those systems tend to have lots and lots
Speaker 1: and lots of points of ref. The electro magnet ones
Speaker 1: didn't tend to have as many points of reference because
Speaker 1: the the software side of it, because you know, we
Speaker 1: do have a hardware and a software side to this.
Speaker 1: The software side would assume that the joints that these
Speaker 1: sensors were attached to behaved the way they normally would
Speaker 1: in humans, and that they don't have complete freedom of movement.
Speaker 1: Most of us are not multi jointed in every joint,
Speaker 1: so we can't you know, that we have a limitation
Speaker 1: on how far we can move in certain directions with
Speaker 1: these various joints. So taking that into account, you didn't
Speaker 1: have to have sensors all over the body. You would
Speaker 1: just have them in a few places, which was good
Speaker 1: considering that there were these thick cables attached to the
Speaker 1: sensors and then once you were done moving, then the
Speaker 1: all that data would get be captured within the system
Speaker 1: and could then be rendered into animation. Although this was
Speaker 1: also a way that you could do real time animation
Speaker 1: or digital a tree. Uh, it's not that different from
Speaker 1: controlling a video game character with a controller. It's sort
Speaker 1: of the same principle, except in this case the the
Speaker 1: video game controller. Instead of it being something you hold
Speaker 1: in your hands, it's something you were actually wearing. And uh,
Speaker 1: I've seen plenty of instances of this. If you've ever
Speaker 1: seen Turtle Talk with Crush over at Disney, that's what
Speaker 1: they use. They use a digital you know, they use
Speaker 1: digital poetry, and it's awesome by the way, I love that.
Speaker 1: Well it uh, it would also seem that, um, you
Speaker 1: would need to be aware of where those cables were going,
Speaker 1: and it would it would also affect the way that
Speaker 1: you would move. You wouldn't move as naturally if you
Speaker 1: were wearing something like that as if you were, you know,
Speaker 1: unencumbered by by that, which, um, sort of I think
Speaker 1: would lend itself to to an upgrade, which is I
Speaker 1: think why they were so keen on Well, it's also
Speaker 1: also that's very true. It did limit what you could do,
Speaker 1: it could live, It would limit your movement. I mean,
Speaker 1: we've got these big cables attached to you. You You obviously
Speaker 1: you can't just move freely within a space. Um, So
Speaker 1: it did put some limitations on you. Their limitations to
Speaker 1: the optical systems too, but we'll get into that. The
Speaker 1: uh the other problem was that the sampling rate for
Speaker 1: the magnetic systems was not as high as it is
Speaker 1: for optical systems. And by sampling rate, what I mean
Speaker 1: is that this the entire system as a whole, is
Speaker 1: taking little measurements of from the sensors of you know,
Speaker 1: the orientation of those sensors within the space, and it
Speaker 1: does that several times every second. But the sample rate
Speaker 1: of the magnetic motion capture systems was much lower than
Speaker 1: what it was for than what it would be if
Speaker 1: you were to use an optical system. So you're not
Speaker 1: getting data as frequently. I mean still several times a second,
Speaker 1: but it's not as precise as the optical system. So
Speaker 1: not only where you limited in the kind of movements
Speaker 1: you can make because you had these major cables attached
Speaker 1: to you, but also you couldn't get really minute precise
Speaker 1: measurements on every kind of movement, So it wasn't good
Speaker 1: for things like sports. So, you know, something like throwing
Speaker 1: a pitch in baseball, there are a lot of movements,
Speaker 1: a little tiny motions that are involved in that. I mean,
Speaker 1: anyone who's watched slow motion footage of a professional baseball
Speaker 1: pitcher throwing a pitch, you can see that there are
Speaker 1: some incredibly subtle movements that are involved in that. And uh,
Speaker 1: and it takes place over a very short period of time.
Speaker 1: I mean, it's a very fast thing to to to measure.
Speaker 1: Using the magnetic motion capture system, you would probably one
Speaker 1: slow the person down because they have all these cables
Speaker 1: attached to them, and to not get enough data to
Speaker 1: give an accurate representation of what had happened in the
Speaker 1: virtual format. So if you were to say, create a
Speaker 1: video game, a baseball video game, the picture would not
Speaker 1: necessarily behave properly if all you did was directly port
Speaker 1: the data you got from the motion capture into the game. Yeah.
Speaker 1: Another drawback of the mechanical systems like that too, is
Speaker 1: that that it's um it's the kind of system that
Speaker 1: not only is cumbersome and inaccurate, but it has to
Speaker 1: be calibrated fairly frequently. UM. And you know, there there's
Speaker 1: a there's some work that you can do with this.
Speaker 1: With the optical systems that they began to introduce UM,
Speaker 1: you know, generally became an upgrade UM. The only there
Speaker 1: is one big advantage that the mechanical systems do have, though,
Speaker 1: and that is that light. The lighting will not necessarily
Speaker 1: interfere with the different points of motion that are captured
Speaker 1: by the mechanical system UM. And that can be an
Speaker 1: issue with the optical systems UM. You know, because that's
Speaker 1: that's why UM, they will be wearing the people, the
Speaker 1: actors who will be UM having their motions captured by
Speaker 1: the system, will be wearing you know, those bright dots
Speaker 1: so that the computer can pick up on that. And
Speaker 1: at the beginning, and these are these early systems, there
Speaker 1: were only so many points action points that they could capture. UM.
Speaker 1: They were very limited in what they could do at first,
Speaker 1: but still you know, somewhat of an upgrade over the
Speaker 1: mechanical Yeah. It also limited what you could have in
Speaker 1: the background, obviously, because you could not have anything that
Speaker 1: was going to be of a similar shade. Uh. You know,
Speaker 1: usually where you're talking about reflective white substance used as
Speaker 1: the um the points of of uh articulations. So the
Speaker 1: little like white stickers is like what you were saying,
Speaker 1: Chris Um, you couldn't have anything like that in the
Speaker 1: background because it would confuse the optical system. So that's
Speaker 1: why a lot of these motion capture scenes are shot
Speaker 1: against a blue screen or green screen. It's so that
Speaker 1: the background does not in any way interfere with the
Speaker 1: motion capture. So if you've ever seen behind the scenes
Speaker 1: footage of The Lord of the Rings movies is a
Speaker 1: great example with Andy Serkis as Gollum or Sniegel if
Speaker 1: you prefer, but he's wearing you know, a tight like
Speaker 1: skin tight suit with these little white uh circles all
Speaker 1: over it. Those are the points that the cameras track
Speaker 1: to create the the performance of Gollum slash Sniegel. So
Speaker 1: the performance is something that's being created not only by
Speaker 1: the actor but also the animators because not We should
Speaker 1: also point out that the motion capture stuff rarely is
Speaker 1: motion capture uh completely. Uh. There's there's rarely a moment
Speaker 1: where you don't have an animator step in and tweak
Speaker 1: it somehow, like Uh, you don't normally have someone create
Speaker 1: a physical performance and that physical performance is completely without
Speaker 1: any tinkering represented in the final product. I mean it
Speaker 1: can happen, there are instances of it, but it's more frequently, uh,
Speaker 1: something where the motion capture performance goes to the animator,
Speaker 1: who can then tweak things if the performance is not
Speaker 1: exactly what needs to be, which is kind of nice.
Speaker 1: You don't necessarily have that luxury with flesh and blood actors.
Speaker 1: That's that's true. That's true. Well, especially with the earlier systems,
Speaker 1: especially the electromagnetic systems. Uh, those were really noisy, not
Speaker 1: literally noisy, but but digital noise. They weren't They weren't
Speaker 1: really highly accurate. Um. The optical systems are are far
Speaker 1: cleaner and give them more accurate representation. But you know there,
Speaker 1: that's it's sort of falls in the realm of artistic license.
Speaker 1: I would think, um, where they need to go in
Speaker 1: and make subtle adjustments to make it look the way
Speaker 1: they want it to look. Up. I should also point out,
Speaker 1: now you just reminded me of something else another drawback
Speaker 1: to the electromagnetic systems, which was you couldn't have anything
Speaker 1: metal on the set because it would interfere with that
Speaker 1: magnetic field and give incorrect readings to the system. So
Speaker 1: you're you're virtual character would not move in the same
Speaker 1: way as the physical one because there would be some
Speaker 1: interference in that sense. So your set couldn't have anything
Speaker 1: metal in it. The props didn't shouldn't have anything metal
Speaker 1: in them, so that that limited to you as well.
Speaker 1: So each system has its own limitations. Getting back to
Speaker 1: the optical one, um, one of the other things you
Speaker 1: have to remember is that in order to really capture
Speaker 1: a a physical object moving through three D space and
Speaker 1: to replicate that in virtual space, you need multiple cameras
Speaker 1: in that system. Because a single camera, assuming that's a
Speaker 1: regular video or film camera, something that does not have
Speaker 1: three capability, pointing that at an object. It's creating a
Speaker 1: two dimensional image of something that's moving in three dimensions.
Speaker 1: The camera can't necessarily tell where movements are happening within
Speaker 1: the depth frame of of that of that image. Right, So,
Speaker 1: if someone's moving in such a way where let's say
Speaker 1: they're moving their head where it would be bobbing closer
Speaker 1: to the camera, Uh, unless the size of the the
Speaker 1: sensors is such that something that subtle could be picked
Speaker 1: up by the camera system, you would lose that information.
Speaker 1: So what you need are multiple cameras on the same
Speaker 1: object so that you can compare that data from the
Speaker 1: multiple angles to tell how this object is really moving
Speaker 1: through this three dimensional space. So it's kind of like
Speaker 1: the idea of having parallax with two eyes. You know,
Speaker 1: our eyes are offset, so by looking at an object,
Speaker 1: we can tell how far away it is in part
Speaker 1: because of parallax. Uh. We also have other visual cues
Speaker 1: that tell us about how far something is, you know,
Speaker 1: things like how tall it is in relation to where
Speaker 1: we are that kind of thing, or how tall it
Speaker 1: is in relation to other objects that are within our
Speaker 1: frame of vision. But parallax is very important. Same sort
Speaker 1: of thing. With these optical systems, you would have multiple
Speaker 1: cameras set up to try and capture the information that's
Speaker 1: going on in the frame so that you could tell
Speaker 1: exactly how it's moving through that three dimensional space. Yeah,
Speaker 1: it seems like um. In order to capture the correct perspective,
Speaker 1: you need that additional information, even though you may not
Speaker 1: necessarily see it. UM. It helps the the animator do that,
Speaker 1: and the optical system to allows you to work with
Speaker 1: more than one actor um, which was not really an
Speaker 1: option with some of the earlier systems. So in other words,
Speaker 1: you can, although it requires more equipment, you know, just
Speaker 1: simply out of necessity, the optical system is really affording
Speaker 1: the animators a an opportunity to use a greater amount
Speaker 1: of information um both you know, from the different the
Speaker 1: different points of data they're getting from a single actor,
Speaker 1: but from multiple actors on the set simultaneously, which enables
Speaker 1: them to to create more complex work right and UH.
Speaker 1: This also gives us a good example of how the
Speaker 1: optical motion capture systems are a passive system because you
Speaker 1: have these sensors you're wearing that are not necessarily or
Speaker 1: not even sensors. They're they're reflective markers that you're wearing.
Speaker 1: They aren't connected to any sort of electronic components at all,
Speaker 1: versus the active systems like the electromagnetic one, where you
Speaker 1: are generating data by moving through a magnetic field and
Speaker 1: you have these big cables attached to it. Uh. With
Speaker 1: the optical motion capture systems. Another thing that's kind of interesting,
Speaker 1: I think is that a lot of least the early ones,
Speaker 1: the cameras would have infrared l ed s uh so admitters,
Speaker 1: really that we're emitting infrared lights. That's outside our our
Speaker 1: visible spectrum. We cannot see infrared light. But by putting
Speaker 1: an infrared filter on the camera, you could have the
Speaker 1: camera pick up reflections of infrared light. And that was
Speaker 1: a way of helping to identify the sensors that you
Speaker 1: had put on the actor. The actors, the sensors would
Speaker 1: be reflective specifically so that the infrared light would reflect
Speaker 1: back toward the camera and give the most accurate rendering
Speaker 1: of what's going on at any given moment within a scene.
Speaker 1: So um, yeah, it's another way of making sure that
Speaker 1: the data being captured is as precise as possible. I
Speaker 1: mean that is, of course, the goal is to try
Speaker 1: and recreate the physical movements as truthfully as you possibly
Speaker 1: can given all the limitations involved. Yeah, and if you're
Speaker 1: looking for a real life easy to find an example
Speaker 1: of this, you would look no farther than your local
Speaker 1: video game store. Um, because the Xbox Connect h uses
Speaker 1: very much that that exact uh form of technology. It
Speaker 1: is using an infrared emitter, um, and it has cameras
Speaker 1: that it uses to pick it up uh, the information
Speaker 1: pick the information up that is coming back from what
Speaker 1: is being reflected around the room, and anybody who who
Speaker 1: has one is also aware that lighting is very much
Speaker 1: an issue. Um. The way that were room is let
Speaker 1: affects the information that the Connect is able to refer
Speaker 1: to the Xbox. Now, it's not, while it is sophisticated,
Speaker 1: is not as sophisticated as the kind of equipment that
Speaker 1: they might use in making a movie or making a
Speaker 1: video game. But it is very very similar technology, and
Speaker 1: in some ways I would argue that it's more sophisticated
Speaker 1: than some of those early UH systems simply because it
Speaker 1: is able to capture a lot of information uh, Whereas
Speaker 1: you know, the very early optical systems were only using
Speaker 1: a handful of data points. So um, it's it's a
Speaker 1: pretty neat device. Um, you know, not only used for gaming.
Speaker 1: Now the hacker community has fallen in love with it
Speaker 1: too because it can do so much and can be
Speaker 1: used for so many things and is you know, fairly inexpensive. Yeah.
Speaker 1: The cool thing about the Connect is that rather than
Speaker 1: have to obviously, if you've if you've ever played an
Speaker 1: Xbox with the connect, you know, you don't have to
Speaker 1: go out and buy a snug body suit covered in
Speaker 1: reflective markers in order to play. I mean, it doesn't hurt,
Speaker 1: but uh, you know, if you're if you can pull
Speaker 1: that look off. There a very few of us who can.
Speaker 1: I count myself among them. But you don't have to
Speaker 1: do that because what it's doing is it's actually projecting
Speaker 1: essentially a grid, uh in infrared light, so you can't
Speaker 1: see the grid, but it's being projected into the room.
Speaker 1: And then when you move uh within the space, you
Speaker 1: are deforming that grid. You know, the camera that's picking
Speaker 1: up the the reflections of that infrared light can detect
Speaker 1: when the grid's being deformed by a physical object interrupting
Speaker 1: the grid. So as you move, you interrupt different parts
Speaker 1: of the grid, and it can start to interpret those
Speaker 1: as motions and commands. It's not, uh, it's not as
Speaker 1: precise as what we're talking about with the optical systems
Speaker 1: that are used in movies and video games, uh, to
Speaker 1: to create them, that is, not to to play them. Um,
Speaker 1: it's not as precise as those, but it also has
Speaker 1: other elements that help balance it out, Like it has
Speaker 1: regular optical cameras that can have some other software that
Speaker 1: aids it in recognizing things like facial recognition software, which
Speaker 1: does not necessarily rely upon that infrared grid. It relies
Speaker 1: more on the traditional camera functions, but has the software
Speaker 1: included that, let's the the programs within recognize who is
Speaker 1: standing in front of it, so that combination, uh increases
Speaker 1: the precision, which of course is very important whenever you're
Speaker 1: playing a game. I mean, anyone who's played any sort
Speaker 1: of game where you're using a faulty controller, or it's
Speaker 1: just a system that hasn't been fully uh it's not
Speaker 1: finished yet, it's just in prototype stage or whatever, you
Speaker 1: may have noticed that it could be very frustrating to
Speaker 1: try and control something where the actual controller is not
Speaker 1: as responsive as you would hope. It's um not a
Speaker 1: fun experience. But anyway, that is kind of related to
Speaker 1: this whole motion capture technology. UM, I'm sorry what you
Speaker 1: were You look like you have something to say, Well, no,
Speaker 1: I was, I was going to say that. Um, you know,
Speaker 1: we really hadn't other than my earlier statement about sports. UM,
Speaker 1: you know, we've we've been talking about it in an
Speaker 1: entertain amount entertainment about the the ability to capture motion
Speaker 1: to make care act is more realistic. And um, that
Speaker 1: that is exactly what they want to do when they
Speaker 1: are using this in sports medicine. UM. Jonathan alluded to
Speaker 1: earlier the difficulty in UH and capturing all the little
Speaker 1: subtle motions that go into UM into a Major League
Speaker 1: baseball players pitching. UM. And you know when somebody, when
Speaker 1: somebody gets hurt, UM, sometimes they go through uh extensive surgery.
Speaker 1: The Tommy John procedures is UH famous. You know, they
Speaker 1: do a ligament transplant to to help rebuild a picture's elbow,
Speaker 1: and that can really throw off, um, the mechanics of
Speaker 1: a pictures motion. So they use this motion capture technology
Speaker 1: to really get an idea of how, UM, how that
Speaker 1: person is is throwing going about the mechanics of their
Speaker 1: typical game play. And and that's exactly the same kind
Speaker 1: of thing that they're doing when they create these very
Speaker 1: realistic sports games. UM. But you know, in this case,
Speaker 1: they're using it for sports medicine to see if they can, UH,
Speaker 1: they can go back and recreate some of the motions
Speaker 1: that made them so successful before they were injured. Now, UM, ironically,
Speaker 1: in in UH entertainment purposes, especially video, UM, you can
Speaker 1: get too realistic. UM. The Japanese professor massa Hiro Mori
Speaker 1: is famous for his Uncanny Valley UM, which has been
Speaker 1: used in uses a robotics term for a robot that
Speaker 1: looks so much and moves so much like a human
Speaker 1: that it it creeps us out. It looks a little
Speaker 1: too realistic. And I can think of we're actually recording
Speaker 1: this in December of and um. One of the movies
Speaker 1: that comes on about this time of year is The
Speaker 1: Polar Express, which is known, loved and reviled both for
Speaker 1: its story and it's um and the way that they
Speaker 1: use motion capture because the characters and there are so
Speaker 1: realistic they're downright creepy. Yeah, it's it's one of those
Speaker 1: things where they are almost but not quite able to
Speaker 1: pass for a real person, so that there's just enough
Speaker 1: off about them to be unsettling. Now, this does bring
Speaker 1: up something else that's kind of interesting. We have an
Speaker 1: article on how stuff works dot com about motion scan technology,
Speaker 1: which is, as I said earlier, a proprietary technology. It's
Speaker 1: it's more specific than just motion capture. It's specifically meant
Speaker 1: to capture facial motion activity. So when an actor is speaking,
Speaker 1: when they're delivering lines, the way that they furrow their
Speaker 1: brow or move their eyes or smile, or they give
Speaker 1: a facial take, anything like that. This system is designed
Speaker 1: to pick that up so that it can be recreated
Speaker 1: virtually in a game, and it was used too great effect,
Speaker 1: in my opinion, in L A Noir. Le Noir was
Speaker 1: a video game that came out in two thousand eleven,
Speaker 1: and it was a game in which you played a well,
Speaker 1: you played a couple of different characters, but the one
Speaker 1: you played for most of the game spoiler alert was
Speaker 1: was a a police detective. And you're kind of rising
Speaker 1: through the ranks uh in L A U during the
Speaker 1: uh early part of the twentieth century. And it's it's um,
Speaker 1: it's notable in that you are, uh, you're spending most
Speaker 1: of the game looking at people's reactions. You know. The
Speaker 1: idea behind L A Noir It was a new type
Speaker 1: of video game where you would interrogate characters throughout your investigations,
Speaker 1: and as you interrogate them, you had to watch the
Speaker 1: characters facial reactions to kind of get an idea of
Speaker 1: whether the character was trying to be evade se or
Speaker 1: if they were telling the truth. And you would do
Speaker 1: things like watch for their eyes and if they weren't
Speaker 1: able to maintain eye contact, that was an indication that
Speaker 1: perhaps they were being less than truthful. Or if they would,
Speaker 1: you know, twitch their mouth or clench their jaw, these
Speaker 1: would be little little hints that perhaps there's more going
Speaker 1: on than what they're letting onto. And obviously, if your
Speaker 1: gameplay depends upon trying to determine whether or not a
Speaker 1: virtual character is telling the truth, you have to be
Speaker 1: able to represent those facial expressions as closely to reality
Speaker 1: as possible, or else the game does not work. So
Speaker 1: they used this motion scan technology and the way that
Speaker 1: they did this was that they had a very brightly
Speaker 1: lit studio that had lights trained on an actor from
Speaker 1: just about every angle and the purpose of that was
Speaker 1: to try and eliminate shadows, because any sort of shadows
Speaker 1: you would have there would of course affect the actual capture.
Speaker 1: It was really all about the light. And they used
Speaker 1: thirty two high definition cameras. So think about that, thirty
Speaker 1: two high definition cameras just to capture and actor's facial
Speaker 1: performance like that's it. There, there's no other movement. The
Speaker 1: actor is seated at the time and um and had
Speaker 1: to remain as still as possible and just do all
Speaker 1: the acting with their face, which for anyone out there
Speaker 1: who's done any sort of acting, you know, that's incredibly
Speaker 1: challenging because actors are trained to use their whole body
Speaker 1: when they are performance making a performance. They're trained to
Speaker 1: to really think about movement. I mean, if you're if
Speaker 1: you're really serious about acting, you've probably taken movement classes.
Speaker 1: And to suddenly have all of that taken away and
Speaker 1: all of your acting is restricted to just your face,
Speaker 1: it's pretty that's pretty dramatic. It's tough to do, but anyway,
Speaker 1: that's what the actors had to do. They had to
Speaker 1: sit down and restrict their acting to just their facial
Speaker 1: expressions without it going like over the top crazy, because
Speaker 1: that would be just as distracting as not enough performance
Speaker 1: at all. And these thirty two cameras were paired up,
Speaker 1: so sixteen pairs of cameras. There's technically there was a
Speaker 1: thirty third camera as well that the director used to
Speaker 1: watch the scene and give directions to the actors um
Speaker 1: but these these pairs of cameras were trained on all
Speaker 1: these different angles of the face in order to capture
Speaker 1: that that performance so that in the virtual world they
Speaker 1: could recreate it accurately, which to me is phenomenal. And
Speaker 1: apparently the way the system works is you get that
Speaker 1: virtual version of the person's face and head almost instantly,
Speaker 1: which is kind of creepy but also awesome. It's it's
Speaker 1: funny too that uh they used that many cameras in
Speaker 1: the creation of a video game, because uh, elsewhere in
Speaker 1: that article that notes that um Circus who was playing
Speaker 1: Gollum um only had only had cameras on on him,
Speaker 1: but in doing so, they were able to, uh to
Speaker 1: create roughly, you know, ten thousand different kinds or identify
Speaker 1: ten thousand different kinds of facial movements that they could
Speaker 1: use in in animating the character on screen. So um, clearly, uh,
Speaker 1: you know, this is very very high tech and painstaking
Speaker 1: procedure to do, but in doing so they can they
Speaker 1: can create very very realistic movements. Yeah, there's a lot
Speaker 1: of number crunching involved, and frankly, the the part that
Speaker 1: takes place after you've captured the data is can be
Speaker 1: dramatically different from one case to the next. In some cases,
Speaker 1: may have already created uh an animated figure pretty much
Speaker 1: from start to finish, you might not have completely put
Speaker 1: textures on it or or something. But you might have
Speaker 1: essentially the way the character is going to look in
Speaker 1: the finished product, uh, and then you just map it
Speaker 1: to the movements that you've captured and it's and there
Speaker 1: it goes. And in other cases you might see that
Speaker 1: what they do is they capture the motions and then
Speaker 1: you essentially have what looks like a very primitive stick
Speaker 1: figure skeleton that moves in the way that the actor moved,
Speaker 1: but there's no definition, there's no character there yet. And
Speaker 1: you may have animators who build the character somewhat based
Speaker 1: upon the way the actor moved through the space, so
Speaker 1: that perhaps the character's design is not finalized until you've
Speaker 1: captured that that performance, and the performance helps guide the
Speaker 1: design of the character. It all depends on the specific
Speaker 1: technology that's being you and the preference of the crew
Speaker 1: that's that's designing whatever it is that they're making, whether
Speaker 1: it's a video game or movie, TV show, commercial, whatever
Speaker 1: it happens to be. UH. In the case of digital puppetry,
Speaker 1: obviously you would already have the the full character realized,
Speaker 1: so that just by using whatever control mechanism happens to
Speaker 1: be there, you would be able to make the puppet
Speaker 1: move in real time, otherwise it's not really puppetry. Um.
Speaker 1: And again that's sort of like the if you've been
Speaker 1: to that that turtle talk thing I talked about, the
Speaker 1: Disney World or Disneyland. Um. I'm sure there are other
Speaker 1: similar ones. I think Monsters Inc. Laugh Factory has a
Speaker 1: similar setup where you've got a digital character on a
Speaker 1: screen that can react in real time to things that
Speaker 1: are happening within the physical environment. So they interact with
Speaker 1: the audience like they'll specifically single people out and chat
Speaker 1: with people in the audience. And Um, to two kids,
Speaker 1: this is amazing. I means the cartoon character acting in
Speaker 1: real time, it's a real person. Now, Uh, two adults,
Speaker 1: it's fascinating because they're like, how the heck did that happen? Um,
Speaker 1: But yeah, that's it's all based on this same sort
Speaker 1: of technology. UM. And It's it's really interesting to me
Speaker 1: to see how the field is evolving over time, because
Speaker 1: things like the connect show that we are adapting the
Speaker 1: same sort of technology in different ways. We're using different
Speaker 1: implementations to essentially do the same thing, and that perhaps
Speaker 1: we will get to a point where we won't have
Speaker 1: to worry about all the sensors so much. Um, you
Speaker 1: can maybe have an actor who's not completely coded and
Speaker 1: stickers perform and and you could capture all that data
Speaker 1: without having to worry about, you know, tracking these little dots.
Speaker 1: That might be something that we've seen in the future.
Speaker 1: I mean the motion scan is kind of like that
Speaker 1: because before motion scan with that facial acting uh technology. Uh,
Speaker 1: whenever I saw anyone who was having their face tracked
Speaker 1: for a performance, they always were wearing those tiny little
Speaker 1: white stickers all over their face to track. I mean,
Speaker 1: we've got a lot of muscles in our face. There's
Speaker 1: something like nineteen muscles or something that you have to track,
Speaker 1: so um, you would have all these little dots on
Speaker 1: your face to track those motions. Well, with motion scan
Speaker 1: you don't need those anymore. So maybe we'll see something
Speaker 1: like that. Of course, that would really depend upon perhaps
Speaker 1: the lighting, which could if you're shooting a virtual character
Speaker 1: that's next to real characters like in The Lord of
Speaker 1: the Rings, real being I guess you know, your mileage
Speaker 1: may very I mean they're hobbits, but anyway, when you're
Speaker 1: next to real people, clearly you can't mess with the
Speaker 1: lighting too much or it'll just make the whole scene
Speaker 1: look strange. Speaking of strange, UM, well, you might think
Speaker 1: that the techniques used in motion capture uh, um, you know,
Speaker 1: bringing film into a uh you know, adding a lot
Speaker 1: of advancement to to film. Um basically uh, some people
Speaker 1: sort of regardless as cheating. Yeah. I did some research
Speaker 1: that that indicated that, um, although some other types of
Speaker 1: animation are considered you know, considered more artful, UM, motion
Speaker 1: capture is sort of not everyone. But some people say, well,
Speaker 1: you know it's it's not oscar worthy because you were
Speaker 1: using these computer add animation techniques that that really, um
Speaker 1: are simulating human motion, and it's just it's just not real.
Speaker 1: And uh. The argument that I've seen used against it is, well,
Speaker 1: you consider rotoscoping, okay, why don't you consider motion capture,
Speaker 1: which is a kind of descendant from this technology. Why
Speaker 1: why isn't that okay to uh, you know, to consider
Speaker 1: for um quality and and and for rewards. But um,
Speaker 1: apparently it's a it's sort of a hot topic among um,
Speaker 1: among movie makers. Yeah. I can see one animator a
Speaker 1: traditional animator, or even a computer animator. I mean that's
Speaker 1: closer and closer to becoming traditional already but either hand
Speaker 1: drawn animation or computer animation. Someone who goes through the
Speaker 1: trouble of animating these things and doing a lot of
Speaker 1: this work. Uh, by hand seems like it's the wrong term,
Speaker 1: but but personally going through and creating these performances, I
Speaker 1: can see where they might feel that way. Um, I
Speaker 1: have a completely different perspective on it. Of course, I'm
Speaker 1: not an animator, so that's part of it, but I
Speaker 1: think of it as creating a performance. And in the
Speaker 1: sense of creating a performance, I think it's a completely
Speaker 1: legitimate tool because you're still relying on an actor to
Speaker 1: create a performance that that that people will relate to,
Speaker 1: whether it's a character that you're supposed to love or
Speaker 1: hate or fear, that all is dependent upon the animator
Speaker 1: and the actor and several other people working to create
Speaker 1: this this performance. And uh, I don't see anything wrong
Speaker 1: with that. That to me is a completely legitimate form
Speaker 1: of creating the art of entertainment. So um, I mean,
Speaker 1: I do understand from an artistic perspective where some people
Speaker 1: could have a problem with it. But but if you
Speaker 1: take a bigger picture look and not not just you
Speaker 1: know what technique you're using, but the end goal of creating,
Speaker 1: whether you want to call it art or not, but
Speaker 1: creating something that has an impact to the viewer or
Speaker 1: player in the case of a video game, I think
Speaker 1: that's more important. But then again, I'm like I said,
Speaker 1: I'm not an animator, so I don't have that kind
Speaker 1: of emotional attachment, you know, I'm not vested in it
Speaker 1: in that way. So UM, I'll be curious to hear
Speaker 1: what our listeners think if they think that there is
Speaker 1: motion capture? Is that cheating? Is it? Uh? Is it
Speaker 1: as Red versus Blue would have you say, a legitimate strategy?
Speaker 1: What what do you think? What do you consider a
Speaker 1: motion capture? You should less know? Yeah, I UM, I
Speaker 1: do see where UM it might make a traditional animator concerned,
Speaker 1: but I don't. I don't really think it diminishes their UM,
Speaker 1: their artistic value, to to UM, to a work whatever
Speaker 1: it may be that they are working on. UM. And
Speaker 1: there are certain times I'm sure where uh you would
Speaker 1: argue that using these techniques is completely inappropriate to what
Speaker 1: they might do UM. But yeah, I mean it's it's
Speaker 1: always a concern when UM you start saying, well, the
Speaker 1: machine can do it, and we don't really need people
Speaker 1: to do it, so get out. Yeah, I don't think
Speaker 1: that's ever gonna be um always the fully the case,
Speaker 1: because you're going to have certain characters within movies that
Speaker 1: are going to be so different from the way humans
Speaker 1: are built, so to speak, that that, uh, that motion
Speaker 1: capture would not be practical. For example, like let's say
Speaker 1: that the character that you're creating has really super long arms,
Speaker 1: and you know, you've got an actor who's pretty lanky,
Speaker 1: but but their arms are not as long as the
Speaker 1: character's arms. Uh, if you were just to a direct
Speaker 1: translation of the actor's movements into the animation, it might
Speaker 1: not look right because the character has different dimensions, their
Speaker 1: body is built differently than the actor. And so without
Speaker 1: tweaking it, without having an animator go in there and
Speaker 1: adjust this and make it look correct compared to what
Speaker 1: the you know, the the vision is for the movie,
Speaker 1: it doesn't come out correctly, it doesn't look right. So
Speaker 1: I think there's very little risk of motion capture ever
Speaker 1: taking that away completely. Plus, there is something to you know,
Speaker 1: creating a performance through traditional animation that you know, it
Speaker 1: does feel differently the motion capture, but that's not a
Speaker 1: bad thing, Like it just depends upon the the vision
Speaker 1: of the director and what the tone of the piece
Speaker 1: needs to be. And in some cases motion capture is
Speaker 1: going to be the best way to achieve that. In
Speaker 1: other cases, motion capture would make it distracting. So um yeah,
Speaker 1: I think as long as we maintain this desire for
Speaker 1: different types of of entertainment and techniques, then there's not
Speaker 1: really any risk of making one disappear. I agree completely,
Speaker 1: I really do. It's it's just a different animals, yep, yep.
Speaker 1: But hey, guys, if you want to chime in on
Speaker 1: this motion capture discussion, please do. Let's know, send us
Speaker 1: an email, are address this tech stuff at Discovery dot com,
Speaker 1: or get in touch with us on Twitter or Facebook
Speaker 1: our handle at both of those. It's text stuff. H. S.
Speaker 1: W and Chris and I will talk to you again
Speaker 1: really soon for more on this and thousands of other topics.
Speaker 1: Is it how staff works dot com Brought to you
Speaker 1: by the two thousand twelve Toyota Camra