Speaker 1: Welcome to tech Stuff, a production from my Heart Radio.
Speaker 1: Hey there, and welcome to tech Stuff. I'm your host,
Speaker 1: Jonathan Strickland. I'm an executive producer with I Heart Radio,
Speaker 1: and I love all things tech. And Halloween is over
Speaker 1: by the time you hear this. I hope you had
Speaker 1: a happy one. But I still have something that falls
Speaker 1: into the kind of creepy category, at least in my opinion.
Speaker 1: And I discovered this after looking around at tech news
Speaker 1: in general, and I became fascinated by it and figured, hey,
Speaker 1: you know, I haven't done a really focused episode on
Speaker 1: a very specific implementation of technology in a long time,
Speaker 1: so why not do that now. Now, anyone who knows
Speaker 1: me can tell you that I am a sucker for
Speaker 1: Disney imagineering, which of course is the peculiar twist on
Speaker 1: engineering and innovation that Disney champions. Right. The inventiveness and
Speaker 1: the attention to detail impressed me a great deal. Those
Speaker 1: are hallmarks of Disney engineering or imagineering. And I've done
Speaker 1: episodes covering various elements that tie into this, from the
Speaker 1: history of upcot to how audio animatronics work. And it's
Speaker 1: that last topic I wish to revisit because not long
Speaker 1: ago I read a research paper from Disney Imagineers titled
Speaker 1: Realistic and Interactive Robot Gaze. That's g A Z E,
Speaker 1: you know, referring to where a person or in this
Speaker 1: case uh an object a robot appears to be looking.
Speaker 1: And the paper is fascinating and it's available for anyone
Speaker 1: to read for free. So if you find this subject
Speaker 1: matter neat, I really recommend you read it. Now. It
Speaker 1: does get a bit technical. There's some math in there too,
Speaker 1: but for the most part, I think it's a pretty
Speaker 1: accessible paper. The pictures and good gravy, y'all. The video
Speaker 1: that are connected to this project are the stuff of nightmares,
Speaker 1: but we'll get to that. The heart of the paper
Speaker 1: is all about designing systems so that an audio animatronic
Speaker 1: or or just an animatronic figure can make and maintain
Speaker 1: eye contact or at least appear to with someone who
Speaker 1: is looking at that figure and onlooker. So, in other words,
Speaker 1: imagine that there's a Disney attraction at a park, and
Speaker 1: in this attraction you can walk up to a robot.
Speaker 1: It's probably going to be behind like a rail or
Speaker 1: inside a booth or something, so that you can't you know,
Speaker 1: touch it, and the robot notices you looking at it,
Speaker 1: and it looks you in the eye. And then maybe
Speaker 1: you get to chat with the robot and it maintains
Speaker 1: eye contact with you, and occasionally maybe it's eyes dart
Speaker 1: around to glance at other stuff that's within its field
Speaker 1: of view, or maybe even indicating that the robot is
Speaker 1: appearing to like take a second to think of a response.
Speaker 1: That's kind of what we're talking about here. And here's
Speaker 1: the thing. This is surprisingly difficult to do, and it's
Speaker 1: extra hard to do without dipping into super unsettling territory.
Speaker 1: So today we're going to learn more about the technology
Speaker 1: and the psychology behind this project, as well as what
Speaker 1: makes it different from earlier audio animatronics, which is honestly
Speaker 1: a good place for us to start. The original audio
Speaker 1: animatronics were essentially puppets. In fact, you could argue that
Speaker 1: all animatronics are ultimately puppets. Each puppet has a certain
Speaker 1: number of degrees of freedom, and that refers to a
Speaker 1: number of independent directions of motion. So let's take a
Speaker 1: simple example. Let's say that a robots neck only has
Speaker 1: one degree of freedom. Well, that would mean the robot
Speaker 1: might be able to nod its head up and down.
Speaker 1: But if it could do that, it wouldn't be able
Speaker 1: to shake its head or tilt its head, because that
Speaker 1: would be an additional degree of freedom. Or maybe it's
Speaker 1: able to shake its head, but it's not able to
Speaker 1: nod or tilt because it only has that one degree
Speaker 1: of freedom. That one degree is really limiting, and it
Speaker 1: just tells us the full range of of direction emotions
Speaker 1: that any one joint can do, and we typically talk
Speaker 1: about degrees of freedom with joints to express the range
Speaker 1: of possible motions the you know, whatever it is can perform.
Speaker 1: The enchanted Tiki Room at Disneyland was an early example
Speaker 1: of audio animatronic ingenuity. It wasn't the very first use
Speaker 1: of audio animatronics, but it was an early one, and
Speaker 1: when you learned how it worked behind the scenes, it's
Speaker 1: pretty wacky. The various birds, flowers, and other elements in
Speaker 1: the attraction connected to a very complex system, including some
Speaker 1: pneumatic valves. A pneumatic system uses air under pressure to
Speaker 1: do work, so these valves in turn connected to a
Speaker 1: circuit that had thin metal reads as switches. Now, normally
Speaker 1: the switch would be open, meaning no electricity can flow
Speaker 1: through the circuit and thus provide electricity to open or
Speaker 1: close the valve. But when sounds of a certain frequency
Speaker 1: would play near these reads, it would cause those reads
Speaker 1: to vibrate, and you know, depending on the thickness and
Speaker 1: length of the read, that would determine what frequency of
Speaker 1: sound would most likely get it to start vibrating. Once
Speaker 1: it vibrated, it would close the circuit and thus allow
Speaker 1: power to go through to the respective valve. And every
Speaker 1: bird and flower in the attraction had this sort of
Speaker 1: system where the sounds playing through the sound system would
Speaker 1: actually cause the individual circuits for those birds and flowers
Speaker 1: to activate. So the chirping of the bird, that chirping
Speaker 1: sound was actually the sound that was opening and closing
Speaker 1: the the circuit and thus activating the valve that would
Speaker 1: control the bird's beak. And because the figures relied on
Speaker 1: the sound to close the circuit, they were audio animatronics.
Speaker 1: Over the years, Disney would improve on this design, sometimes
Speaker 1: by necessity. So for example, when the imagineers set out
Speaker 1: to create the attraction The Great Moments with Mr. Lincoln,
Speaker 1: they had to come up with new mechanisms to do
Speaker 1: that because pneumatics would not be a good solution. With pneumatics,
Speaker 1: you've got a couple of limitations that you're working with.
Speaker 1: One is that you can't move really heavy stuff effectively
Speaker 1: with pneumatics. Another is that pneumatic pistons tend to move
Speaker 1: really fast. It's hard to do controlled slow movements with pneumatics.
Speaker 1: So it might be okay for something like a bird
Speaker 1: flapping its wings or opening and closing its beak fairly quickly,
Speaker 1: but it's not so great for say, a revered US
Speaker 1: president lifting his hand. But I've covered that in other episodes.
Speaker 1: The really important thing I want to stress is that
Speaker 1: audio animatronic figures have historically been limited to a cific,
Speaker 1: pre programmed sequence of motions, so calling them puppets is
Speaker 1: fairly appropriate. These are figures that will do the exact
Speaker 1: same sequence of motions until something goes wrong or the
Speaker 1: attraction is shut off for some reason. The pirate and
Speaker 1: Pirates of the Caribbean that is precariously attempting to step
Speaker 1: onto a rowboat is never going to fall into the water.
Speaker 1: He's never going to get into the boat, and he's
Speaker 1: never gonna step back onto the shore. He will continue
Speaker 1: his balancing act until the end of time. And this
Speaker 1: is starting to sound like some sort of Greek myth
Speaker 1: about the afterlife at this point. Now, the reason I'm
Speaker 1: bringing this up, the reason it's important, is that creating
Speaker 1: an animatronic figure that can actually detect an onlookers gaze
Speaker 1: and return it making eye contact can't be totally dedicated
Speaker 1: to following the same set of motions on repeat. There
Speaker 1: has to be some room for variability within it. At
Speaker 1: the same time, Disney's whole gig is to create a show.
Speaker 1: The amusement parks are show business. If you are in
Speaker 1: a public space of one of those parks, like you're
Speaker 1: inside the confines of the park itself, walgging a down
Speaker 1: Main street or whatever, you are on stage. The employees
Speaker 1: are called cast members, and shows, while they can have
Speaker 1: some variation in them, are supposed to follow a general flow.
Speaker 1: They follow a script. And so the imagineers were working
Speaker 1: on creating a figure that would follow a scripted set
Speaker 1: of behaviors, but would have the freedom to throw in
Speaker 1: stuff like eye contact now and then the figure, in
Speaker 1: a way would be able to improvise. It's jazz Baby.
Speaker 1: The tune is more or less set, but how you
Speaker 1: go through it allows for a lot of variation. For
Speaker 1: the purposes of this work, the team relied on an
Speaker 1: animatronic bust. Now we've kind of dropped the audio at
Speaker 1: this point. Modern animatronic figures are not really driven by
Speaker 1: audio signals anymore. They're driven by circuitry and sophisticated computer
Speaker 1: systems and programs. Though to be fair, they still often
Speaker 1: are referred to as audio animatronic. But you really need
Speaker 1: to see a picture of this thing. I'll do my
Speaker 1: best to describe it, but really you should search this
Speaker 1: Disney uh interactive gaze animatronic because who boy, so imagine
Speaker 1: the V shaped torso of a bust sculpture, right, It's
Speaker 1: very narrow at the bottom, and it widens up to
Speaker 1: the shoulders. It's clad in a white button up shirt,
Speaker 1: you know, kind of like an Oxford shirt of business shirt.
Speaker 1: It does have shoulders, but does not have arms. It
Speaker 1: has a head, good golly, it has a head. The
Speaker 1: head of this figure has a sort of plastic skull,
Speaker 1: though it's kind of more like a plastic mask than
Speaker 1: a human skull. It doesn't look like a skeleton skull.
Speaker 1: It does have eyes, it's even got eyelids, and it's
Speaker 1: got teeth. And looking at this thing is a little unsettling.
Speaker 1: And that's before it even makes eye contact with you. Now,
Speaker 1: why would you want to make something like this be
Speaker 1: able to make eye contact in the first place. Well,
Speaker 1: eye contact is an important social signal. It shows mutual acknowledgement,
Speaker 1: and it can lead us to projecting certain things upon
Speaker 1: the person or animal that's making eye contact with us.
Speaker 1: We tend to perceive such creatures as possessing a certain
Speaker 1: amount of intelligence and sincerity. For example, when I make
Speaker 1: eye contact with my dog Ti Bolt, I perceive him
Speaker 1: to be intelligent and alert and loving. Now I have
Speaker 1: no way of knowing what is really going on in
Speaker 1: his doggy mind. I suspect it's probably more along the
Speaker 1: lines of is the bald man about to give me
Speaker 1: a treat? I should pay attention, But I like to
Speaker 1: think of it as sincere love. Now, as the paper states, quote,
Speaker 1: given the importance of gays in social interactions, as well
Speaker 1: as its ability to communicate states and shape perceptions, it
Speaker 1: is a parent that gays can function as a significant
Speaker 1: tool for an interactive robot character end quote. And I
Speaker 1: can totally grock that. I imagine what it might be
Speaker 1: like to a child who's going to Disney World or
Speaker 1: Disneyland for the very first time and going to a
Speaker 1: ride or an attraction where there's an animatronic figure, perhaps
Speaker 1: one that looks like a famous Disney character, and it
Speaker 1: makes eye contact with that child, maybe it even speaks
Speaker 1: to the child, and maybe it can respond to the
Speaker 1: child of the child speaks back. That sort of interaction
Speaker 1: would have been the kind of stuff that would have
Speaker 1: stuck with me as a kid well into adulthood, and
Speaker 1: I feel confident about that because I have a lot
Speaker 1: of memories of the seemingly magical moments I've experienced at
Speaker 1: Disney with far more primitive technology. Is that we're in
Speaker 1: the Disney parks when I first started visiting them in
Speaker 1: the nineteen seventies, so I can certainly see the show
Speaker 1: need for this sort of development. But there are numerous
Speaker 1: challenges that stand in the way of achieving this goal,
Speaker 1: and they fall into different broad categories. Perhaps the easiest
Speaker 1: set of challenges to conquer is actually the electro mechanical
Speaker 1: side of things. That is, the actual mechanisms that you're
Speaker 1: going to use to create these effects, the servos and
Speaker 1: the motors and the other components that will create the
Speaker 1: actual motions that will translate into the robot making eye
Speaker 1: contact or behaving in otherwise realistic ways. That's one of
Speaker 1: the set of challenges, but there are others. One is
Speaker 1: giving the robot the ability to detect the gaze of
Speaker 1: onlookers in the first place. There has to be some
Speaker 1: sort of face recognition and maybe even eye tracking technology
Speaker 1: so that the robot looks at the right spot. So
Speaker 1: the electro mechanical parts have to work correctly, but so
Speaker 1: does the robot vision or perception. Otherwise the robot is
Speaker 1: going to look in the wrong spot, perhaps staring off
Speaker 1: to one side or above or below and onlooker's eye
Speaker 1: contact or attempt at eye contact. Another challenge would be
Speaker 1: on the programming side. You have to figure out how
Speaker 1: to determine who the figure is going to look at.
Speaker 1: You also have to figure out how long the robot
Speaker 1: will look at somebody and what could distract the robot,
Speaker 1: and whether or not the robot would return to looking at,
Speaker 1: you know, the first person, or maybe look at a
Speaker 1: second person, or maybe look at something else Entirely, you
Speaker 1: have to solve the challenge of the program and prioritize
Speaker 1: the order of operations so that the robot behaves in
Speaker 1: a way that makes sense, as opposed to a robot
Speaker 1: that's just you know, reacting to all visual stimuli in
Speaker 1: a random way, which would be at the very least disconcerting.
Speaker 1: And then we get to something that's a bit harder
Speaker 1: to define than degrees of freedom or range of motion
Speaker 1: or the hierarchy of programming, and that's human psychology. Now,
Speaker 1: as the paper points out, eye contact is an important
Speaker 1: social cue for most of us, but there are a
Speaker 1: whole range of humans out there right For people who
Speaker 1: have autism, eye contact can be a really challenging task,
Speaker 1: and it tends to make people who have this type
Speaker 1: of autism. It makes their lives a little more difficult
Speaker 1: or complicated as a result. It's something that people some
Speaker 1: people anyway, have to consciously deal with. They have to
Speaker 1: remember to do this and work at it. It's not
Speaker 1: it's not a natural behavior for them. So this is
Speaker 1: something that can be tricky for human beings, let alone
Speaker 1: for robots. Now, while eye contact can help create a
Speaker 1: sense of sincerity and interest, it can also shift over
Speaker 1: into more unpleasant territory, such as a sense of predatory
Speaker 1: intent or as a comedian I once saw said there's
Speaker 1: a fine line between the casual eye contact of a
Speaker 1: friend and the cold stare of a serial killer. He
Speaker 1: was specifically taught king about trying to navigate the tricky
Speaker 1: territory of approaching people in order to get to know them.
Speaker 1: But I think the meaning could be used for lots
Speaker 1: of scenarios, including an encounter with a robotic figure. And
Speaker 1: along with that is the issue of the uncanny valley,
Speaker 1: which I have touched on in previous episodes. I'm not
Speaker 1: sure if I've ever actually talked about the origin of
Speaker 1: the phrase, however, a professor at the Tokyo Institute of
Speaker 1: Technology named massa Hiro Mori coined this phrase in the
Speaker 1: nineteen seventies to describe a pretty odd phenomenon. As robots
Speaker 1: become more human like or more lifelike in general, they
Speaker 1: become more appealing to us, but only up to a point,
Speaker 1: and once they get to that point and go beyond it,
Speaker 1: our reception of these robots plunges into the uncanny valley.
Speaker 1: The valley in this case is how humans react to
Speaker 1: the robot. This also applies to other stuff like c
Speaker 1: g I characters, for instance, and other words are a
Speaker 1: robot that might be a simple industrial arm is one
Speaker 1: we probably wouldn't feel very much affinity for, you know,
Speaker 1: it's obviously a machine. A robot that still looks really robotic,
Speaker 1: but has you know, arms and legs like a vaguely
Speaker 1: humanoid shape. We would probably feel a little more affinity
Speaker 1: towards that make it look a little bit more human,
Speaker 1: but you know, not to the point where anyone would
Speaker 1: mistake it for being human. We might like it even more.
Speaker 1: But once you start getting close to but not quite
Speaker 1: human in appearance and behavior, our response drops to a
Speaker 1: point where a lot of people feel unsettled, or even
Speaker 1: they might feel revulsion when looking at the figure. Something is,
Speaker 1: you know, not right. The cues that would normally help
Speaker 1: us identify with the synthetic figure now feel strange and
Speaker 1: maybe even scary. It's possible to get beyond the uncanny
Speaker 1: valley to create a robot or c g I character
Speaker 1: that doesn't initiate this kind of instant revulsion, but it
Speaker 1: is very hard to do so. A big challenge is
Speaker 1: building an animatronic that doesn't trigger the uncanny value response
Speaker 1: either by avoiding the trap of being almost but not
Speaker 1: quite human in behavior, you know, by keeping things a
Speaker 1: bit more obviously robotic, so there's that clear and distinct
Speaker 1: separation that kind of removes that that response we have,
Speaker 1: or creating something lifelike enough that we feel the same
Speaker 1: sort of reactions we would experience if that were a
Speaker 1: real human. So it's tough to do. It's easier to
Speaker 1: do the robot approach than it is to get something
Speaker 1: that seems human enough that we let our guard down.
Speaker 1: None of these challenges are trivial, but they all require
Speaker 1: distinct approaches that must ultimately converge into a single implementation.
Speaker 1: When we come back, I'll talk about some of the
Speaker 1: technologies in this animatronic figure and the engineering team's philosophy
Speaker 1: behind their design choices. But first let's take a quick break.
Speaker 1: The engineering team limited itself to parameters that related to
Speaker 1: creating a robot that could direct its gaze towards onlookers,
Speaker 1: which meant they didn't have to worry about it doing
Speaker 1: literally anything else. The audio animatronic bus they used has
Speaker 1: nineteen degrees of freedom total, but the team made no
Speaker 1: use of ten of those. They only used nine degrees
Speaker 1: of freedom. They focused on the neck, which has three
Speaker 1: degrees of freedom. The eyelids, which have two degrees of freedom,
Speaker 1: the eyes, which also have too, and the eyebrows, which
Speaker 1: have two degrees of freedom. The unused degrees of freedom
Speaker 1: are for moving the jaw and the lips of the figure,
Speaker 1: but since that's not necessary to make eye contact, the
Speaker 1: team just ignored those they didn't need to mess with them,
Speaker 1: which means we get the effect of a robotic skull
Speaker 1: with an unchanging rictus grin staring at us as its
Speaker 1: upper facial area remains animated it. I guess what I'm
Speaker 1: saying is I didn't find the overall effect particularly comforting.
Speaker 1: According to the paper, the commands going to these components
Speaker 1: come from a quote custom proprietary software stack operating on
Speaker 1: a one hurts real time loop end quote. Hurts is
Speaker 1: a cycle per second, so this means that the software
Speaker 1: is pulsing out operations one hundred times every second to
Speaker 1: control this animatronic bust. Many of those commands aren't only
Speaker 1: about making the bus do something specific, but to do
Speaker 1: it in a specific way. Let's get back to the
Speaker 1: Tiki birds as an example. The pneumatic valve that would
Speaker 1: control whether or not pressurized air could travel to a
Speaker 1: specific place like the mechanism that operates a bird's beak
Speaker 1: is a pretty simple on or off switch, meaning the
Speaker 1: valve is either open, in which case air can flow,
Speaker 1: or it's closed, in which case the air is blocked
Speaker 1: from flowing through. And a debating the mechanism, So the
Speaker 1: beak has a natural resting position, and for this example,
Speaker 1: will just assume that the rest position is a closed beak,
Speaker 1: and so that's what the beak will always return to
Speaker 1: when there's no air flowing. To the mechanism that opens
Speaker 1: the beak. If we open the valve, it lets air through,
Speaker 1: It rushes to the end point, forces the beak to
Speaker 1: open rapidly. Closing and opening the valve quickly forces the
Speaker 1: bird's beak to open and close quickly, and when matched
Speaker 1: with a soundtrack, it looks as though the bird is
Speaker 1: speaking or singing, or you know, whatever it's doing. But
Speaker 1: that movement is rapid and, just as I mentioned earlier,
Speaker 1: not suitable for all animatronic applications. Having life sized humanoids
Speaker 1: move with that kind of alarming speed would be scary
Speaker 1: and legitimately dangerous. The greater mass of the figures would
Speaker 1: mean you're dealing with larger amounts of inertia. I mean,
Speaker 1: I just imagine what it would look like if Mr Lincoln,
Speaker 1: in an effort to raise his hand in a gentle
Speaker 1: show of reserve determination, instead violently karate chopped his own
Speaker 1: head off. It would be, as the kids say, a
Speaker 1: bad look. To create the illusion of life, the animatronics
Speaker 1: that Disney designs follow certain general strategies. One is called
Speaker 1: slow in and slow out. Now. This refers to general
Speaker 1: movements and the ideas that any movement should start off
Speaker 1: slowly and then pick up speed as the movement continues,
Speaker 1: and then slow down again before coming to a stop.
Speaker 1: And it makes the motions appear more fluid, and it
Speaker 1: has the added benefit of not being quite so harsh
Speaker 1: on the figures themselves. So when a Disney figure raises
Speaker 1: its hand, the hand should start off moving upward with
Speaker 1: a nice, smooth slow motion, pick up a bit of
Speaker 1: speed as it's moving upward, and then slow down again
Speaker 1: as it's approaching its end point. And this means that
Speaker 1: the underlying motors and mechanical systems have to be capable
Speaker 1: of achieving the strategy. It's why you can't use pneumatic systems.
Speaker 1: They can't be those simple single speed devices that are
Speaker 1: either on or off, like the Tiki birds. Oh, and
Speaker 1: I guess I should specify I'm talking in this case
Speaker 1: about the original Tiki birds because the birds in the
Speaker 1: attractions today work on updated and more sophisticated computer systems
Speaker 1: that take up a fraction of a fraction of the
Speaker 1: space of the old attraction, which essentially required an entire
Speaker 1: room filled with cables and tubes to make everything work
Speaker 1: underneath the actual attraction itself. Now a few computers handled
Speaker 1: the whole shebang. Anyway, Let's get back to animatronics. Some
Speaker 1: of the other guiding principles in animatronic motion that in
Speaker 1: turn dictate the types of motors and joints and other
Speaker 1: mechanical elements that the team mustn't use to to make
Speaker 1: these happen include designing motions as arcs, meaning the motion
Speaker 1: should follow an arched trajectory. Another is that the motions
Speaker 1: should have overlap, meaning a robot shouldn't move a single
Speaker 1: element like an arm, stop, then go to move on
Speaker 1: the next element like the head position, and then stop
Speaker 1: and so on, because that would be well, really robotic. Instead,
Speaker 1: the robots motions should overlap with one another so that
Speaker 1: Let's say Mr. Lincoln is turning his head at the
Speaker 1: same time his arm is going up in determination. Now,
Speaker 1: another element that's connected to this concept is that of drag,
Speaker 1: which means that the different body parts are moving at
Speaker 1: different frequencies or timing. They're not moving all at the
Speaker 1: same speed. So, in other words, the speed at which Mr.
Speaker 1: Lincoln turns his head might be slightly faster or slower
Speaker 1: than the speed at which his arm goes up. This
Speaker 1: is all in an effort to create the illusion of life,
Speaker 1: but it also means that the programming in hardware underlying
Speaker 1: the figure has to support those strategies. For the purposes
Speaker 1: of this project, the engineers had certain motions they wanted
Speaker 1: to be included. One minimum set of motions needed were
Speaker 1: some that would imply that the bust was a breathing entity,
Speaker 1: So I need to move slightly as if it were
Speaker 1: drawing breath. Blinking was also an important motion to get down,
Speaker 1: as it would be more than a little unnerving to
Speaker 1: have an animatronic figure make eye contact with you and
Speaker 1: then never ever blink. And then there were the scads.
Speaker 1: Now I have to confess something to you, guys. When
Speaker 1: I first encountered the word scads, which is S A
Speaker 1: C C A D E S. I had no idea
Speaker 1: what that meant. It was a new word to me,
Speaker 1: and maybe it's a new word for some of you
Speaker 1: out there too. So if you happen to be like me,
Speaker 1: what the heck are scads? Well? That refers to the quick,
Speaker 1: simultaneous movement of both eyes from one point of focus
Speaker 1: to another. So think about how you might take in
Speaker 1: a scene that has a lot of stuff going on.
Speaker 1: Let's say you you walk up to a building that's
Speaker 1: that's that's burning. Well, your is are going to dart
Speaker 1: at different things that are going on in front of
Speaker 1: you that catch your attention as you focus on them,
Speaker 1: and then you file that information away. And perhaps you're
Speaker 1: even doing this subconsciously. Uh. It means our gaze is
Speaker 1: not always steady and unwavering. It it moves around a
Speaker 1: bit on occasion. And that's not the only way we
Speaker 1: move our eyes. Of course, we can actually track things
Speaker 1: that are moving and use our eyes to move in
Speaker 1: a more smooth and gradual motion. But the team knew
Speaker 1: that if they could incorporate the CODs, that would give
Speaker 1: the robot a more lifelike performance. But that decision meant
Speaker 1: the team needed to figure out something else, which was
Speaker 1: where to put the cameras. The animatronic needs its own
Speaker 1: vision to be able to detect onlookers and then direct
Speaker 1: its own gaze appropriately, and some robots do put cameras
Speaker 1: in the eyes of the robot so that the eyes
Speaker 1: are actually camera lenses, but that presents a challenge if
Speaker 1: you wish to incorporate rapid eye movement like the CODs,
Speaker 1: because that sort of movement introduces motion blur in the
Speaker 1: video imagery makes it more challenging for the robot to
Speaker 1: keep track of what's going on in front of it.
Speaker 1: For that reason, the team decided that the cameras would
Speaker 1: not be mounted in the eyes, but they rather were
Speaker 1: mounted on the animatronics chest. Presumably, should the gaze tracking
Speaker 1: technology find its way into full animatronic figures in the future,
Speaker 1: the camera will be you know, hidden within the body
Speaker 1: of the animatronic torso in order to avoid this problem,
Speaker 1: or otherwise maybe mounted in an obtrusive spot. One thing
Speaker 1: that interests me with this particular approach is that the
Speaker 1: system has to do some calculations as to where the
Speaker 1: eyes of the animatronic are in relation to the physical
Speaker 1: location of the cameras, you know, because for us, all
Speaker 1: our eyes are essentially the cameras, or at least the
Speaker 1: camera lenses, so we don't have to make any adjustments.
Speaker 1: Right where we're looking is like the point of our
Speaker 1: gaze is the point of where we're taking in visual information.
Speaker 1: For the animatronic, the eyes of the robot, the actual
Speaker 1: eyes that are in the skull, don't function as eyes.
Speaker 1: They aren't lenses. They're actually several inches above the actual camera.
Speaker 1: And yet the eyes in the robot's head need to
Speaker 1: point in the right direction. They need to be the
Speaker 1: part that's pointed at the person who's looking at it. Right,
Speaker 1: it doesn't make sense for the robot to just turn
Speaker 1: its sternom towards you. It needs to be looking at
Speaker 1: you with its robot eyes. And I think of this
Speaker 1: kind of like someone who's working a hand puppet and
Speaker 1: they've got the hand puppet up over their head, so
Speaker 1: maybe they're behind a little stage, you know, like like
Speaker 1: the muppets tend to be. You've got this hand puppet
Speaker 1: and it needs to make eye contact with a human being. Well,
Speaker 1: that just means the puppeteer has to take that into
Speaker 1: account and angle their hand so that the puppets eyes
Speaker 1: appear to be locking on the eyes of the real
Speaker 1: person that the muppet or puppet is interacting with. It's
Speaker 1: a little tricky. It requires some skill for the robot.
Speaker 1: It means that there's some you know, nifty geometry going
Speaker 1: on in the processor side to make this work out.
Speaker 1: Like the image recognition has to identify where the eyes
Speaker 1: are of the onlooker and then calculate where the robots
Speaker 1: eyes are in relation to that and direct them in
Speaker 1: the right way, which to me is really fascinating because again,
Speaker 1: the eyes of the robot are not where the visual
Speaker 1: information is actually coming in. We'll talk more about the
Speaker 1: behaviors of this robot in a second, but since we're
Speaker 1: already chatting about cameras, it's good to talk about what
Speaker 1: the team was actually using to give the robot it's vision.
Speaker 1: They went within off the shelf solution. They used a
Speaker 1: camera called the Mint Eye D one thousand and Mint
Speaker 1: is spelled m y nt. This particular camera has two
Speaker 1: lenses in it for stereoscopic vision, and so together they
Speaker 1: can create a stereo image that is an image with
Speaker 1: you know, kind of a depth like a three D
Speaker 1: image with a resolution of two thousand, five hundred sixty
Speaker 1: by seven twenty pixels at sixty frames per second, so
Speaker 1: it can do you know, this is video information. There's
Speaker 1: also a depth map mode which uses infrared light to
Speaker 1: help judge the depth of the things within its field
Speaker 1: of view, like how close is one thing versus another
Speaker 1: relative to the camera, and the depth maps resolution is
Speaker 1: at one thousand, two hundred eighty by seven twenty pixels
Speaker 1: at sixty frames per second. As I mentioned, these two
Speaker 1: lenses allow the camera to simulate human binocular vision. So
Speaker 1: just as we perceive depth in the world around us
Speaker 1: using two eyes, you know, most of us, uh, this
Speaker 1: camera can do the same thing and judge which things
Speaker 1: are in the foreground versus the background, what things are
Speaker 1: closest to it versus furthest away, and make a better
Speaker 1: determination of which things within its field of view are
Speaker 1: worthy of attention, which will become important in a little bit.
Speaker 1: The camera has a more limited field of view than
Speaker 1: a typical human. It has about half the horizontal field
Speaker 1: of view of persons, so it's periphery is more narrow,
Speaker 1: and it has a little more than a third the
Speaker 1: vertical field of view, so I can't see as much
Speaker 1: up and down as your typical person can. So any
Speaker 1: future animatronic figure might need a more expansive field of
Speaker 1: view to be able to interact with guests who could
Speaker 1: range and height from very small to quite tall. I mean,
Speaker 1: all sorts of people go to Disney. So I do
Speaker 1: see that as a potential limiting factor in the short run,
Speaker 1: that any stereoscopic kind of camera would need to have
Speaker 1: a pretty good field of view for a robot to
Speaker 1: be able to interact properly with guests of different heights. Now,
Speaker 1: I decided to see how much this camera would cost
Speaker 1: for some normal schlub like myself, and the answer is
Speaker 1: less than four hundred dollars. So this is actually a
Speaker 1: pretty inexpensive solution all things considered. And again so it's
Speaker 1: it's really more important for creating the basis for the
Speaker 1: work as opposed to saying this is a final product.
Speaker 1: And that's more or less the hardware side of things,
Speaker 1: or at least as specific as I can get based
Speaker 1: on the material available. Like I, I don't know what
Speaker 1: the power of their computer system was, you know, I
Speaker 1: don't know the specific types of motors they were using
Speaker 1: in the animatronic but from a high level we understand
Speaker 1: what's going on. However, the real magic happens with the
Speaker 1: system that gives this hardware it's orders, and the team
Speaker 1: made the conscious decision to create the illusion of life
Speaker 1: rather than attempt to replicate human behaviors perfectly, which is
Speaker 1: a bit of a challenging concept. You might think, well,
Speaker 1: what's the difference, But I think I have a pretty
Speaker 1: decent analogy. If you've ever gone to see a stage play,
Speaker 1: then you've seen sets. Maybe the sets were really detailed,
Speaker 1: maybe they were bare bones sets, But in any case,
Speaker 1: the sets are meant to create the illusion of a
Speaker 1: real place at a real moment of time. You know,
Speaker 1: it could be a room in the eighteenth century in
Speaker 1: a in a palatial estate, or it might be a
Speaker 1: modern day real estate sales office if it's a moment play,
Speaker 1: or maybe it's a campsite. In any case, the sets
Speaker 1: and props are meant to convey the illusion of that
Speaker 1: place and time, and if you were to actually get
Speaker 1: up on stage and walk around, that illusion would very
Speaker 1: quickly be broken. But when you're sitting in the audience,
Speaker 1: it's up to you to use your imagination to fill
Speaker 1: in some of the gaps and suspend disbelief it is
Speaker 1: a show. Likewise, the engineers who worked on this project
Speaker 1: talk about robot behaviors in terms of a show, and
Speaker 1: that means that the robot needs to react and move
Speaker 1: in ways that create the illusion of life, but it
Speaker 1: does not necessarily need to adhere completely to human behaviors.
Speaker 1: This makes things much more simple, particularly since it removes
Speaker 1: of tricky questions regarding what sets of behaviors are the
Speaker 1: most human, because I'm sure you've noticed human beings and
Speaker 1: human behavior occur in a really broad spectrum, and what
Speaker 1: might be a typical set of behaviors for one person
Speaker 1: could be completely alien to another person. So it's a
Speaker 1: good idea to not try and define what sets of
Speaker 1: behaviors are quintessentially human. When we come back, I'll talk
Speaker 1: about how the team determined how the robot would actually
Speaker 1: behave it's pretty cool, but first let's take another quick break.
Speaker 1: The team created an architecture to describe the relationship of
Speaker 1: various elements to create the behavior of an interactive robotic gaze.
Speaker 1: To create this robotic eye contact, the layers include the camera,
Speaker 1: which is you know, the point of perception from the robot,
Speaker 1: a perception engine h and an attention engine which determines
Speaker 1: which things within the robots perception are actually worthy of
Speaker 1: attention or focus. A behavior selection engine and a library
Speaker 1: of potential behaviors, and the audio animatronic figures systems. It's hardware,
Speaker 1: the motor commands and motor states go to that, and
Speaker 1: that's the layers in order from top to bottom. These
Speaker 1: layers explain the relationship of each element in sort of
Speaker 1: an abstract way, allowing us to understand how the robot
Speaker 1: processes and reacts to information. So the perception engine is
Speaker 1: designed to identify potential elements within the robotic vision, you know,
Speaker 1: separating things out from say just a static background, and
Speaker 1: the attention engine attempts to identify things within the robots
Speaker 1: vision that merit focus. The attention engine generates what the
Speaker 1: team calls a curiosity score. So if that curiosity score
Speaker 1: is below a certain threshold, the robot won't quota quote
Speaker 1: notice something within its field of view. It's it's not
Speaker 1: enough to capture its attention. Certain actions, such as you know,
Speaker 1: waving at the robot, merit a higher curiosity score. So
Speaker 1: if the score ends up being above the curiosity score threshold,
Speaker 1: the robot will look toward whatever it was that you know,
Speaker 1: quote unquote got its attention. The team decided it would
Speaker 1: be helpful to create a sort of scenario to work with,
Speaker 1: not just have you know, a robot randomly looking around,
Speaker 1: So their approach was to simulate an elderly man reading
Speaker 1: something like a newspaper or a book. Most of the time,
Speaker 1: the robot would be looking downward a bit, you know,
Speaker 1: it's head tilted down a little, as if it were
Speaker 1: reading something that was held more or less at torso level.
Speaker 1: If something moves into the robots field of you, the
Speaker 1: robot could glance up quickly, just as a human would
Speaker 1: to assess what's going on, and if whatever is within
Speaker 1: the field of view creates a curiosity score lower than
Speaker 1: what the threshold is, then the robot just goes back
Speaker 1: to reading. If whatever is going on is above that
Speaker 1: curiosity score threshold, the robot might look directly at whatever
Speaker 1: it is that's happening, and then things could progress from there.
Speaker 1: That's where the behavior selection engine and behavior library come
Speaker 1: into play. There are a few possible reactions, and the
Speaker 1: robot will choose one depending on several factors. For example,
Speaker 1: one such factor was familiarity. The robot would behave differently
Speaker 1: toward people it quote unquote recognized. It also wouldn't switch
Speaker 1: focus every time someone tried to wave it down, So
Speaker 1: if you were to distract the robot, it might look
Speaker 1: away from whatever it was looking at before and then
Speaker 1: look to you once. Then it might look back at
Speaker 1: someone quote unquote knows, and if you were to wave
Speaker 1: at it again, you wouldn't necessarily get a response. So
Speaker 1: kind of think about how adults can be with kids,
Speaker 1: where the adults tend to develop a highly attuned skill
Speaker 1: of ignoring the child after a bit, even if the
Speaker 1: child is saying, but look, look, look, hey, Look what's
Speaker 1: what I'm doing? Look? And so on. So the team
Speaker 1: created four basic states. The default state was called read,
Speaker 1: meaning it would appear as though the figure we're reading
Speaker 1: a book or newspaper at Torso level. The next state
Speaker 1: up is glance, where upon the robot would appear to
Speaker 1: glance away from the reading material to see what sort
Speaker 1: of ruckus is going on. This involved movement of not
Speaker 1: just the eyes but the head as well. So the
Speaker 1: head tilts up a bit and it looks for a moment,
Speaker 1: like the robot is looking away from the imaginary book
Speaker 1: or newspaper. If the curiosity threshold is met, then the
Speaker 1: next state engage would pop up. This means that whatever
Speaker 1: it was that got the robot's attention is worthy of
Speaker 1: further focus. In the robot will direct its gaze at
Speaker 1: that thing. With the engage stage, which has a nice
Speaker 1: rhyme to it, the robot will attempt to make eye contact,
Speaker 1: which involves the cameras detecting the face of the person
Speaker 1: of interest, and then the computer system commanding the robot's
Speaker 1: head and eyes to aim towards that detected face. The
Speaker 1: amount of time that the robot spends looking at a
Speaker 1: person is determined both by a minimum countdown clock saying
Speaker 1: you have to spend this amount at least looking at
Speaker 1: this person, and then there's the curiosity score that the
Speaker 1: robot has assigned to that person. So once that score
Speaker 1: decreases below the engaged threshold, the robot returns to read.
Speaker 1: So if you happen to be particularly interesting, the robot
Speaker 1: will look at you for longer, and when you stop
Speaker 1: being interesting, the robot eventually goes back to reading its
Speaker 1: pretend book or whatever. The final stage is called acknowledge,
Speaker 1: and that was the name that the team gave for
Speaker 1: those times when the robot is seeing a person that
Speaker 1: is familiar to the robot. For the purposes of the tests,
Speaker 1: the familiarity variable was actually randomized, so in other words,
Speaker 1: the robot wasn't necessary early familiar with people. It just
Speaker 1: was told it was familiar with somebody. So, in other words,
Speaker 1: that it could be a totally new person that walks
Speaker 1: up to the robot and the robot randomly assigns that
Speaker 1: person the familiar tag, and the robot will behave as
Speaker 1: if that's someone that the robot recognizes. Maybe they're just
Speaker 1: an old friend the robot just met. Is there a
Speaker 1: word for that? The robot system also had a sort
Speaker 1: of short term memory that the team called the guesthouse.
Speaker 1: As people would come into the robot's field of view
Speaker 1: or the scene as the team called it, the robot
Speaker 1: would analyze that person and assign that person a numerical
Speaker 1: value to keep track of that person, and it would
Speaker 1: also keep track of how many times that particular person
Speaker 1: had been within its field of view, and it would
Speaker 1: keep track of the curiosity score that was assigned to
Speaker 1: that person. In addition to the states, the team described
Speaker 1: lay years of show. Now this relates closely with the
Speaker 1: states I just mentioned, but it helps explain how the
Speaker 1: robot transitions from one set of behaviors to another, how
Speaker 1: does it make the determination to change from one thing
Speaker 1: to the next, and which behaviors will overwrite others versus
Speaker 1: behaviors that will always be present with the robot. All
Speaker 1: of this is necessary because of that variation I was
Speaker 1: talking about at the beginning of the show. If the
Speaker 1: robot were just following a scripted set of directions, it
Speaker 1: wouldn't have to make these determinations because it would just
Speaker 1: follow the same sequence over and over. But because we
Speaker 1: have this variability, we have to build in a system
Speaker 1: for the robot to follow in order to make decisions.
Speaker 1: So at the base level you have what the team
Speaker 1: calls zero show. This is essentially the robot in off mode.
Speaker 1: It is inanimate. But the next layer up is a
Speaker 1: live show, which has the baseline behaviors of simulated bree thing,
Speaker 1: eye blinking, and the scads. This level of show underlies
Speaker 1: all the other higher levels, so this is sort of
Speaker 1: always running in the background. You don't want the robot
Speaker 1: to suddenly stop breathing while it does other stuff. The
Speaker 1: next four show levels correspond with the four states of
Speaker 1: the robots. So you have read, glance, engage, and acknowledge,
Speaker 1: and an engage show will subsume the glance and read shows.
Speaker 1: It will take over the robots behaviors, So the robots
Speaker 1: not going to display the behaviors of read and glance
Speaker 1: when engage happens. So it's that hierarchy of operations, and
Speaker 1: I find it really interesting to look at robot behaviors
Speaker 1: in this way as that hierarchy of potential states. It's
Speaker 1: amazing when you break down those states and determine which
Speaker 1: should take priority given certain circumstances, and how long that
Speaker 1: state should remain active before it rever it's to a
Speaker 1: lower level state. Again, the team is trying to create
Speaker 1: the illusion of life. The robot doesn't have to actually
Speaker 1: lose interest or anything like that. It's just simulating it.
Speaker 1: This particular project was working within some pretty well defined
Speaker 1: parameters and restrictions. The team acknowledge that their work is
Speaker 1: really meant to be a starting point for further improvements.
Speaker 1: They point out that older audio animatronics might seem lifelike
Speaker 1: at greater distances and for shorter durations. So, for example,
Speaker 1: if you were to ride an attraction where you go
Speaker 1: by a scene of audio animatronic figures at a decent
Speaker 1: clip and and there you know, good, twenty feet away.
Speaker 1: The limited amount of time and the greater distance that
Speaker 1: are involved can help support that illusion of life. The
Speaker 1: animatronic figures don't have to be super convincing because you're
Speaker 1: not spending enough time and attention to see through the illusion,
Speaker 1: nor are you close enough to see it showed through.
Speaker 1: The more time you have and the less distance between
Speaker 1: you and the animatronic figure, the harder it is to
Speaker 1: create and maintain that illusion of life. Without an interactive gaze,
Speaker 1: Without eye contact, it becomes pretty clear that the animatronic
Speaker 1: figure has no real lifelike quality to it. If you
Speaker 1: were to stand close to one of these older animatronic figures,
Speaker 1: you would notice that it's not really looking at anything
Speaker 1: in particular, and that its movements are a matter of routine.
Speaker 1: It's not a demonstration of spontaneous or seemingly spontaneous decisions.
Speaker 1: The Interactive Gaze project takes this a step up. The
Speaker 1: robot can recognize and acknowledge someone that is in the
Speaker 1: robot's presence, it can direct its focus and attention at
Speaker 1: that person. This definitely is a step up in creating
Speaker 1: that illusion and works at much smaller distances of viewing
Speaker 1: than the older methods do, but the engineers admit it
Speaker 1: still has limitations. They point out that their approach as
Speaker 1: it stands, might serve as a way to reserve that
Speaker 1: illusion of life for a couple of minutes at the most,
Speaker 1: but beyond that the illusion would start to fade away.
Speaker 1: They point out that as the distance between the robot
Speaker 1: and the audience decreases, and as the time of observing
Speaker 1: the robot increases, you have to incorporate increasingly complex and
Speaker 1: natural behaviors to maintain that illusion of life, and interactive
Speaker 1: gaze is just one element. Others could include stuff like
Speaker 1: a display of emotion. The bust has sort of a
Speaker 1: little bit of this. It can it can imply a
Speaker 1: sense of emotion to some degree with the way it
Speaker 1: holds its eyes, but because it doesn't have any movement
Speaker 1: of its jaw or lips, and doesn't have any other
Speaker 1: means of really indicating emotion, this is pretty limited. So
Speaker 1: perhaps a robot that can hear and parse and respond
Speaker 1: to speech, you know, sort of like the voice activated
Speaker 1: digital assistance that are familiar to us, and you know,
Speaker 1: probably like the Amazon Echo or the iPhone or Android phones.
Speaker 1: That might be something that really pushes that illusion of life.
Speaker 1: And of course there's also the physical appearance aspect. Now,
Speaker 1: you would never mistake this animatronic bust for a human
Speaker 1: I mentioned before. It's pretty creepy looking. It's got a
Speaker 1: plastic and skeletal quality to it that prevents you from
Speaker 1: ever mistaking it as a person. But the team points
Speaker 1: out the physical appearance of the robot taps back into
Speaker 1: that problem of uncanny Valley. It might take a while
Speaker 1: to create something that's convincing enough and yet not repulsive
Speaker 1: to work as a robotic human animatronic. If you make
Speaker 1: it look too real, it's going to give people the creeps.
Speaker 1: I think, at least in the short term, we're more
Speaker 1: likely to see this technology used to create characters that
Speaker 1: are human like but still distinctly not human, in order
Speaker 1: to avoid that negative reaction when the uncanny Valley gets involved.
Speaker 1: In other words, using the US to create an animatronic
Speaker 1: figure that looks a lot like a cartoon character, even
Speaker 1: a human cartoon character because well, you recognize the cartoon
Speaker 1: character as representing a human. Cartoon characters don't really look
Speaker 1: like humans. Usually they look like they have human qualities
Speaker 1: to them, but they still have cartoonish qualities to them,
Speaker 1: so you wouldn't mistake them for actually being human. Or
Speaker 1: you just you know, go the robot route or some
Speaker 1: sort of animal career and you sidestep that problem. The
Speaker 1: engineers conclude their paper by talking about how the attention
Speaker 1: engine could, with some evolution, work for a lot of
Speaker 1: different applications. So imagine that you design an animatronic that
Speaker 1: represents someone who's really frightened, and that kind of character
Speaker 1: might have a very low threshold for stimuli to push
Speaker 1: it to a higher state of attentiveness, right like a
Speaker 1: little sound might cause that character to perk up and
Speaker 1: look around quickly because that that character is supposed to
Speaker 1: be frightened. Or you could create something like, you know,
Speaker 1: an absent minded book lover who only glances up from
Speaker 1: whatever book they're studying if something really exciting is happening,
Speaker 1: otherwise they just ignore it. They also talk about the
Speaker 1: bottom up approach to layering behaviors and deciding which behaviors
Speaker 1: will replace others that might inhabit a lower state. That
Speaker 1: is really fascinating to me. Now, we're still a far
Speaker 1: away off from seeing these sorts of technologies make their
Speaker 1: way into official attractions, but based on what I've seen
Speaker 1: and read, I wouldn't be surprised to find them making
Speaker 1: their way into Disney parks in the next say, five
Speaker 1: years or so, depending on how the company budgets stuff.
Speaker 1: Of course, the pandemic has created a particularly tricky situation
Speaker 1: for that branch of the Disney Company, even as other
Speaker 1: branches of that company continue it's global domination of all
Speaker 1: things entertainment. But the technology itself and the design philosophy
Speaker 1: of how to program a robot to behave as if
Speaker 1: it were doing so naturally, it's really neat to me.
Speaker 1: And as I said at the beginning, the paper is
Speaker 1: available for free to read, so if you want to
Speaker 1: check that out, I highly recommend it. I think it
Speaker 1: is a fascinating piece of work, and as I said,
Speaker 1: it's not that difficult to follow. There's some math stuff
Speaker 1: that will probably, you know, lose a lot of you,
Speaker 1: but it lost me. I'm not I'm not trying to
Speaker 1: shame you. I couldn't follow all of it, but it
Speaker 1: is otherwise pretty easy to understand. And like I said,
Speaker 1: it is titled Realistic and Interactive Robot Gaze g A
Speaker 1: Z E, so check that out. It is really a
Speaker 1: neat paper. Just I apologize for the pictures that are
Speaker 1: in there because they're creepy as all get out. That's
Speaker 1: it for me. I hope you guys enjoyed this episode.
Speaker 1: If you have suggestions for future topics I should tackle
Speaker 1: in tech stuff, let me know on Twitter. The handle
Speaker 1: is text stuff h s W and I'll talk to
Speaker 1: you again really soon. Text Stuff is an I Heart
Speaker 1: Radio production. For more podcasts from I Heart Radio, visit
Speaker 1: the I Heart Radio app, Apple Podcasts, or wherever you
Speaker 1: listen to your favorite shows,