Speaker 1: Pushkin.
Speaker 2: One of the things I look at in robotics as
Speaker 2: a big field. There are so many amazing demonstrations of mobility,
Speaker 2: robots doing backflips, robots running down hills, and that's really
Speaker 2: impressive to me because I can't do a backflip, or
Speaker 2: I might trip if I run down the hill. But
Speaker 2: where the really valuable parts of robotics are going to
Speaker 2: be are in manipulation. So my kid can take a
Speaker 2: blueberry out of her cereal bowl because she doesn't want
Speaker 2: to eat it, and that is an incredibly hard task
Speaker 2: for a robot. And you don't see any of those demos.
Speaker 2: And I think we're like kind of inherently programmed as
Speaker 2: people to like be biased towards the backflip being more
Speaker 2: impressive and in reality like the business value and the
Speaker 2: harder thing for the robot is to like take the
Speaker 2: blueberry out of the cereal bowl.
Speaker 1: I'm Jacob Goldstein and this is What's Your Problem, the
Speaker 1: show where I talk to people who are trying to
Speaker 1: make technological progress. My guest today is Aaron Parness. Aarin
Speaker 1: spent the earlier part of his career building space robots
Speaker 1: at NASA's Jet Propulsion Laboratory JPL. Six years ago, he
Speaker 1: went to work at Amazon. Now, Aaron is a director
Speaker 1: of Applied Science at Amazon Robotics. I wanted to talk
Speaker 1: to Erin about a robot arm called Vulcan. He and
Speaker 1: his team developed Vulcan to do a job that is
Speaker 1: surprisingly hard for robots to do, taking stuff that gets
Speaker 1: delivered to Amazon warehouses and putting it onto shelves. In
Speaker 1: order to solve this problem, Aaron and his team had
Speaker 1: to build a robot that had a sense of touch,
Speaker 1: that could deal with complicated, unpredictable situations, and that could
Speaker 1: look at a shelf and plan out a course of action.
Speaker 1: As you'll hear in the interview, all of those traits
Speaker 1: may someday be helpful, not just in stocking shelves in
Speaker 1: a warehouse, but in doing lots of boring sounding but
Speaker 1: complicated real world tasks like, for example, taking a blueberry
Speaker 1: out of a bowl of cereal. To start, I asked
Speaker 1: Aaron to tell me the problem that Vulcan was designed
Speaker 1: to solve at Amazon's warehouses.
Speaker 2: So new inventory comes into the building. You know, trucks
Speaker 2: pull up and they unload new stuff. We need to
Speaker 2: store that stuff while it's waiting for someone to click
Speaker 2: the buy button. We store it in these large fabric bookcases.
Speaker 2: It's about eight feet tall. It has about forty different
Speaker 2: shelves on it. It's four sided, so you can store
Speaker 2: stuff from any of the different faces of the case.
Speaker 2: What's really interesting is this stuff is randomly stowed, so
Speaker 2: it's not like all the iPhones are in one shelf.
Speaker 2: It'll be all different stuff, all mixed together.
Speaker 1: When you say random, do you mean random or do
Speaker 1: you mean it would look random to the untrained eye.
Speaker 2: I mean literally random. Really wherever there is space you
Speaker 2: can put the item, because.
Speaker 1: That's what's optimal. It turns out the optimal way to
Speaker 1: store stuff is random.
Speaker 2: That's right. Why the stems actually from Jeff Bezos is
Speaker 2: like original vision, I think, and it's it's incredible. So
Speaker 2: you want to have the most selection, and you want
Speaker 2: to have speed of delivery, and you want to have
Speaker 2: low cost, and that's what the customer wants, right. The
Speaker 2: customer is using Amazon dot Com because we have selection,
Speaker 2: we have speed, and we have low cost. In order
Speaker 2: to achieve that, you have to have these massive warehouses
Speaker 2: located really close to your customers, and you have a
Speaker 2: lot of customers in Tokyo, in New York City and
Speaker 2: San Francisco where real estate's really expensive, so you have
Speaker 2: to figure out a way to put all of this
Speaker 2: different stuff in like the densest packing area, you can
Speaker 2: and have access to it immediately so that you can
Speaker 2: you can deliver in hours instead of days. And what
Speaker 2: that means is that random is better than structured. So
Speaker 2: anywhere there's a space, you can add that item into
Speaker 2: the inventory, and that means it comes up for sale
Speaker 2: immediately on the website, and then when someone places in order,
Speaker 2: you don't have to wait for that iPhone bookcase to
Speaker 2: make its way all the way across the warehouse. You
Speaker 2: probably have a thousand iPhones in the warehouse, and whichever
Speaker 2: one is closest can go to whichever pickstation is eligible,
Speaker 2: and it ends up being actually substantially faster. So that last.
Speaker 1: Sentence seems to be the key. The idea is like, yes,
Speaker 1: given you have whatever a thousand iPhones in the warehouse
Speaker 1: in the universe where a human had to know where
Speaker 1: they all were, you'd put them all on one shelf.
Speaker 1: But you're saying at any given time that means that
Speaker 1: shelf is probably going to be pretty far away, whereas
Speaker 1: if you randomly distribute them throughout the shelves and warehouse
Speaker 1: at any given time, one of those thousand iPhones is
Speaker 1: probably going to be pretty close to where it needs
Speaker 1: to be. And because you have a whatever, a computerized
Speaker 1: system that can keep track of everything all the time,
Speaker 1: it makes sense to randomly distribute all the things.
Speaker 2: Yeah, that's exactly right. And it works on the flip
Speaker 2: side as well. So when you have a new item
Speaker 2: that's come in, rather than waiting for the shelf that
Speaker 2: has the right size thing to put the new dog
Speaker 2: toy in, you just put the dog toy anywhere you
Speaker 2: can find space for it.
Speaker 1: H It's like my house. We have a lot of
Speaker 1: dog toys in my house. Also, Yeah, that's really interesting.
Speaker 1: It's great for the customer, and that's optimal.
Speaker 2: It's optimal, and it creates an incredibly difficult environment for robotics. Huh,
Speaker 2: because now you have to deal with all this clutter.
Speaker 2: We can have more than a million unique items in
Speaker 2: one warehouse. Yeah, so it's not like you have a
Speaker 2: model of each of those items. And we sell more
Speaker 2: third party items than you know, Amazon owns themselves. Right.
Speaker 2: We are a platform for third party fulfillment, and so
Speaker 2: you don't have all the data about all those items,
Speaker 2: and so you have to handle all this uncertainty, all
Speaker 2: this clutter, and everything's tightly packed.
Speaker 1: And so still in most places as a result, when
Speaker 1: stuff comes into the warehouse every day off a truck
Speaker 1: to people take the things out of the truck and
Speaker 1: stick them randomly on shelves wherever they can find space.
Speaker 1: Is that the system?
Speaker 2: That is exactly the system, and it's in you know,
Speaker 2: hundreds of buildings around the world.
Speaker 1: And just to be clear, I mean it's pretty clear,
Speaker 1: but just to really put a point on it, why
Speaker 1: is this a hard environment for robots?
Speaker 2: Traditional industrial robots do not handle contact well, so like
Speaker 2: touching their environments, and they don't handle clutter or you know, uncertainty,
Speaker 2: and so it's hard because to put that last book
Speaker 2: onto the bookshelf, squeeze that teddy Bear into the just
Speaker 2: small enough space that it'll fit, you have to push
Speaker 2: the other stuff around that's already on that bookshelf. And
Speaker 2: a traditional robot doesn't have sensors, it doesn't even know
Speaker 2: how to do that. So if you think of like
Speaker 2: a car manufacturing line, you're like nineteen nineties two thousands,
Speaker 2: you know, welding robot or loading sheet metal into a press.
Speaker 2: It's doing all of that only knowing its position in space.
Speaker 2: So it has no force sensing. If it runs into something,
Speaker 2: it either is like an emergency stop because it's like broken,
Speaker 2: or it just smashes that thing and keeps going and
Speaker 2: it doesn't even know it's smashed anything. It literally has
Speaker 2: no sensing.
Speaker 1: That is an incredibly homogeneous environment. Right, It's doing like
Speaker 1: the exact same thing at a very high level of
Speaker 1: precision forever one thing.
Speaker 2: That's exactly right. And so this extension, the fundamental breakthrough
Speaker 2: for science for robotics manipulation that my team is trying
Speaker 2: to make is one giving the robot a sense of
Speaker 2: touch and using that along with site and along with
Speaker 2: like knowing where your robot is to do meaningfull tasks
Speaker 2: in like very high contact, high clutter environments. And then
Speaker 2: there's a brain part. It's also much more difficult to
Speaker 2: kind of predict how this random assortment of items is
Speaker 2: going to move or change as you push on it.
Speaker 2: And so there's an AI piece, there's a brain piece
Speaker 2: that's saying this item will fit in that bin. This
Speaker 2: is actually one of the most frustrating things when you
Speaker 2: try and do the job yourself. I'm like an optimist.
Speaker 2: I'm always oh, yeah, this will fit. And I go
Speaker 2: up there and I try and play Tetris and I
Speaker 2: try and rearrange the shelf and like, it clearly isn't
Speaker 2: going to fit. And then I've wasted thirty seconds or
Speaker 2: forty seconds and I have to try something else.
Speaker 1: That's a good statement of the problem. Well, like when
Speaker 1: did you come onto the scene?
Speaker 2: So I was working on some other stuff and there
Speaker 2: was a recent PhD that had joined our team. He was,
Speaker 2: you know, one year out of school something like this,
Speaker 2: and he says, I'm going to go try and solve
Speaker 2: stowing items into these bookshelves. And my thought was, Oh,
Speaker 2: how naive, Like the real world is going to teach
Speaker 2: this new grad. That's just way too hard a problem
Speaker 2: for robotics to solve. But I was helping him because
Speaker 2: it's fun, right, Like you like to work on hard
Speaker 2: problems when you're a researcher. And he was a very
Speaker 2: nice guy, and so I was, you know, helping him,
Speaker 2: but never thought it was going to work. And there
Speaker 2: were a couple of kind of moments where we made
Speaker 2: these simplifications that turn the problem from I have to
Speaker 2: try and do every possible game of Tetris that a
Speaker 2: person can do into a problem where you're like, oh,
Speaker 2: it's not that this is never going to work, it's
Speaker 2: that this is the future, Like this is robotics two
Speaker 2: point zero, Like this is I have to work on this.
Speaker 2: I can't do anything else anymore. I'm like, I'm all
Speaker 2: in on this problem.
Speaker 1: Tell me about one of those simplifications one of those moments.
Speaker 2: It was the gripper is one. The design, the mechanic
Speaker 2: design of the robotic hand was actually a big breakthrough.
Speaker 2: And when we started, we were trying to push items
Speaker 2: with the item we were gripping. So imagine you're pinching
Speaker 2: a book and you're trying to use that book to
Speaker 2: like push this dog toy over to the side.
Speaker 1: I see, So you want to put the book in
Speaker 1: a bin. Yeah, dog toys in the way. So you're like, okay,
Speaker 1: pick up the book and use the book kind of
Speaker 1: like a brush to sweep the dog toy out of
Speaker 1: the way.
Speaker 2: Okay, And I say, okay, like I understand, but it's
Speaker 2: never going to work. What if you don't have a book.
Speaker 2: What if you have a T shirt? Yeah, what if
Speaker 2: you have an iPhone and it's very expensive? Are you
Speaker 2: going to actually want to start pushing on stuff with
Speaker 2: the phone? And so we came up with this strategy
Speaker 2: to have like a spatula that would extend into the
Speaker 2: bin and you'd push everything with this spatula that was
Speaker 2: part of your hand. So imagine like you're like Wolverine
Speaker 2: and you can shoot out, you know, but instead of
Speaker 2: like the Adamantium claws, you're shooting out a spatula.
Speaker 1: So it's like a pincher grip. A little spatula shoots
Speaker 1: forward out of the pincher grip.
Speaker 2: Is the thing that's right.
Speaker 1: It's so simple when you put it that way. I mean,
Speaker 1: I'm sure making it was not low tech, but it
Speaker 1: sounds very like. It's not like some crazy AI thing.
Speaker 1: It's like just what if there was another little thing
Speaker 1: that came out and push stuff out of the way.
Speaker 2: But those ideas are like the really powerful ones when
Speaker 2: you have a simple, elegant solution and you're like, okay,
Speaker 2: that could work. That's different than like a five fingered
Speaker 2: hand that has twenty five motors embedded in it. Yeah,
Speaker 2: like Oh, it's just dispatulate.
Speaker 1: Fingers are famously difficult. Why didn't anybody think of it before?
Speaker 2: So we had been working on it as a company
Speaker 2: back to the Amazon Picking challenge, which was, you know,
Speaker 2: twenty fifteen. But I think a lot of robotics researchers
Speaker 2: like myself, were scared that this problem was just too hard.
Speaker 2: There was easier things to go try and work on,
Speaker 2: and there were a couple of simplifications, so using this
Speaker 2: spatulo as one, and then you watch people do the
Speaker 2: task and you realize they're kind of doing the same
Speaker 2: strategies over and over again. It's like insert dispatchela on
Speaker 2: the D and sweep to one side. For this kind
Speaker 2: of page turn mechanism. Something's fallen over and you need
Speaker 2: to sort of flip it back up to make space.
Speaker 1: So you put this spatula underneath it and flip the
Speaker 1: thing up ninety degreees basically, Yeah.
Speaker 2: And you realize that accounts for like ninety percent of
Speaker 2: the actions you do when you try and stow into
Speaker 2: these pins.
Speaker 1: And did you figure that out by watching people stow.
Speaker 2: We did and doing it yourself.
Speaker 1: How much stowing did you do couple of days? Okay,
Speaker 1: it's a hard job, thousands of items probably, I imagine.
Speaker 2: Yeah, exactly. And we tried to wear go pro cameras
Speaker 2: on our heads so we could look at the videos later,
Speaker 2: which turns out as a recipe for motion sickness. It's
Speaker 2: very difficult to watch those videos, but you go and
Speaker 2: you do it, and you build up this intuition. And
Speaker 2: I I think the other piece of the problem that
Speaker 2: made it tractable and made me this like huge believer
Speaker 2: was recognizing we didn't have to get to one hundred percent.
Speaker 2: So in some automation scenarios, you have to solve the
Speaker 2: whole problem, and if you don't, you have nothing, so
Speaker 2: like landing on the moon. And what we realized was
Speaker 2: there was a way to like make the business logic
Speaker 2: work that the robot could handle seventy five percent of
Speaker 2: the stoves and it just had to not make a
Speaker 2: mess and let like work alongside people to do the
Speaker 2: other twenty five percent, and the sum of the parts
Speaker 2: is actually much better than either all robots or all
Speaker 2: employees would be on their own. And making that realization
Speaker 2: all of a sudden meant that it could be a
Speaker 2: two or three year project instead of a twenty year
Speaker 2: project because chasing this long tail. You know, we have
Speaker 2: a million unique items in the building, but we also
Speaker 2: process a million items per day. So I have a
Speaker 2: phrase like, if something goes wrong one in a million,
Speaker 2: it happens every day in every Amazon building. And to
Speaker 2: try and solve all of those is it is a
Speaker 2: twenty year problem.
Speaker 1: I feel like that part of the solution generalizes in
Speaker 1: a really nice way, right, Like I mean, I guess
Speaker 1: the eighty twenty problem is a sort of cliche. But
Speaker 1: the idea that like, oh, if you think of the
Speaker 1: problem the right way, it's like, no, we don't have
Speaker 1: to build a robot that does it every time. Before
Speaker 1: we build a robot that does it seventy five percent
Speaker 1: of the time. That is a huge efficiency gain and
Speaker 1: maybe the optimal point on the curve. Right yep, if
Speaker 1: the robot is doing everything, you're working too hard to
Speaker 1: make the robot work.
Speaker 2: Probably exactly that.
Speaker 1: So, Okay, so you have these two big ideas. Do
Speaker 1: you want to tell me the sort of story of
Speaker 1: making it work? You want to tell me how it works.
Speaker 2: We've been running six of these robots at a warehouse
Speaker 2: in Spokane, Washington, Okay, since November of last year and
Speaker 2: so we've done over half a million stows it this point.
Speaker 2: We also have another product that's picking those items out
Speaker 2: of the bins, and so that's my team in Germany,
Speaker 2: and so we have a warehouse in Homburg where we've
Speaker 2: been picking items. And picking is a slightly harder problem
Speaker 2: in some ways because you have to identify the item.
Speaker 2: So for stow, you have to identify free space. It's
Speaker 2: either occupied or you can make space to put the
Speaker 2: next item in. For pick, you want to make sure
Speaker 2: I get you the red T shirt, not the red sweatpants,
Speaker 2: or I get you the Harry Potter volume two and
Speaker 2: not Sapiens or some other book.
Speaker 1: Tell me how it works. Let's do this stewing first,
Speaker 1: since that's what we've been talking about. So there's this
Speaker 1: warehouse in Spokane where this robot that you built is
Speaker 1: in use. Like what happens there? A truck pulls in
Speaker 1: and then what happens.
Speaker 2: The way the system works is one of these pods,
Speaker 2: one of these bookcases pulls up to the station, so
Speaker 2: it pulls in front of the robot. We have stereo
Speaker 2: camera towers and so we're looking with the eyes first,
Speaker 2: and we are creating a three D representation of the scene.
Speaker 2: So we're modeling, you know, all the items that are
Speaker 2: in the in the pod already. But the really interesting
Speaker 2: part is we're actually predicting on top of that, how
Speaker 2: we can move those items around to make more empty space,
Speaker 2: how can we squeeze more stuff in. So it's not
Speaker 2: just identifying vacant space. You have to predict where you
Speaker 2: can make that vacant space by pushing stuff with this spatula. Okay,
Speaker 2: then we do this matching algorithm. So we have about
Speaker 2: forty or fifty items waiting for us to stow, and
Speaker 2: so we have a variety of stuff, and we're matching
Speaker 2: those forty or fifty items to the thirty ish shelves
Speaker 2: that are in front of the robot. Which items should
Speaker 2: go where, and then how do we make that space?
Speaker 2: And so that's where a lot of the AI in
Speaker 2: the system is active and operating. It's predicting success, it's
Speaker 2: minimizing risk, it's trying to optimize for a bunch of
Speaker 2: different parameters. Once we've made that selection, we grasp the item,
Speaker 2: so that item we've selected for putting into the given
Speaker 2: shelf passes into our hand and our hand is two
Speaker 2: conveyor belt paddles, so you can think of it kind
Speaker 2: of like a Panini press, like a George Foreman grill.
Speaker 2: It is a George Foreman grill where each side has
Speaker 2: a conveyor built into it.
Speaker 1: Like a little belt, Like just a little belt going around.
Speaker 2: That's right. So each face of the grill, the top
Speaker 2: face and the bottom face have a conveyor belt. And
Speaker 2: that's important because you can control the pose of the
Speaker 2: item and you can feed it into the bin rather
Speaker 2: than like throwing it into the bin. One of the
Speaker 2: early versions we had kind of dropped it and tried
Speaker 2: to punch it to put the item into the bin,
Speaker 2: and that predictably failed in a lot.
Speaker 1: Of Well you say predictably now, but if you try
Speaker 1: to it wasn't predictable.
Speaker 2: Yeah. Yeah. I'm a huge believer in iterative design, and
Speaker 2: so we try and build early, build often build and
Speaker 2: learn from those builds. So it's actually really important to
Speaker 2: keep sixed off pose control of the item. So you
Speaker 2: want to make sure the item isn't rotating as you
Speaker 2: shoot it out. You want to make sure that you
Speaker 2: keep the orientation of the item because it's fitting tightly,
Speaker 2: so you don't want it to run into the bookshelf
Speaker 2: above it or below it, or the items thirty in there.
Speaker 2: Yeah yeah, yeah. We started by trying to shoot it out,
Speaker 2: and then we had all kinds of problems when it
Speaker 2: would like collide with stuff and fall on the floor.
Speaker 2: The worst case is, you know, you would shoot it out,
Speaker 2: it would bounce off the back of the back of
Speaker 2: the bookcase and then come back and hit you in
Speaker 2: the face or hit you in your.
Speaker 1: Did that happen?
Speaker 2: Yeah? Oh yeah, good.
Speaker 1: That's robot comedy, yesh, ro but physical comedy.
Speaker 2: Yeah.
Speaker 1: So, and that's it.
Speaker 2: That's the stow process. And we want to do that
Speaker 2: a few hundred times an hour, and we want to
Speaker 2: do it on the top shelves of those bookcases where. Yeah.
Speaker 2: That's one of the ways we are really complimentary to
Speaker 2: the employees is if the robots can do the top shelves,
Speaker 2: it saves a lot of ergonomic tasks. It allows the
Speaker 2: employees to work in their power zone, like you know,
Speaker 2: shoulder level. That makes them faster too. So if you
Speaker 2: put robots in, people get faster at the job.
Speaker 1: I mean, presumably as the robot gets better, it'll also
Speaker 1: be better at putting things on the middle shelf.
Speaker 2: Right, Well, there's this like sweet spot.
Speaker 1: The robot's going to get better faster than people will
Speaker 1: get better.
Speaker 2: Yeah, we want the robots to be as good as
Speaker 2: it can and not chase one hundred percent. We don't
Speaker 2: really believe in one hundred percent automation. We want to
Speaker 2: find that sweet spot where we're maximizing product.
Speaker 1: Mean, the sweet spot's going to keep moving, right, The
Speaker 1: robot's going to get better and better and be able
Speaker 1: to do more and more faster and faster.
Speaker 2: Presumably, and my science team's actually really excited about that.
Speaker 2: As you get more and more data. So we have
Speaker 2: five hundred thousand stows that we've done so far, but
Speaker 2: when we get to five hundred million STOs, you can
Speaker 2: leverage some of these techniques to start learning the motions
Speaker 2: and learning some of these strategies and refining them to
Speaker 2: be specific to the item that you are molding in
Speaker 2: your hand. There's a lot of opportunity as you get
Speaker 2: more and more data.
Speaker 1: Well, right, so we haven't really I mean, you mentioned
Speaker 1: the software side, the AI side, but we haven't really
Speaker 1: talked about it, and it is I mean, in talking
Speaker 1: to other people working on robotics. It's plainly a data
Speaker 1: game because there's no Internet of the physical world, right,
Speaker 1: because large language models work so well, because there's this
Speaker 1: huge data set and everybody is trying to get data
Speaker 1: from the physical world, and you seem very well positioned
Speaker 1: to get a lot of data from the physical world.
Speaker 2: I think that's true. So one of the joys of
Speaker 2: being a roboticist at Amazon is all the data that
Speaker 2: we have access to. But I will push back a
Speaker 2: little bit that it's just a data problem. It's a
Speaker 2: highly debated topic. Some people in the world believe that
Speaker 2: you can apply the same sort of transformer architectures that
Speaker 2: work so well for search and so well for natural
Speaker 2: language processing and apply those to robotics. If we only
Speaker 2: had the data. I would not put myself in that camp.
Speaker 2: I am not a believer that all we need is
Speaker 2: more torque data from robotic grippers and will solve it.
Speaker 2: Natural language is already tokenized in a way that's very
Speaker 2: amenable to those methods, and language and search are also
Speaker 2: very tolerant of sloppiness. So you and I can have
Speaker 2: a conversation. I don't have to get every single word correct,
Speaker 2: but if you mess up a torque on a gripper,
Speaker 2: you can crush your iPhone, or you can sort of
Speaker 2: smash something else that's there, or drop something, or just
Speaker 2: fail the task. And that's because you have physics and
Speaker 2: this nonlinear, very sort of difficult to model real world
Speaker 2: that these robots have to interact with. And so I
Speaker 2: think those techniques certainly accelerate us in a lot of places,
Speaker 2: but they don't just solve the problem. I think we
Speaker 2: need all of the rest of robotics, like hardware design
Speaker 2: and classical control theory to solve those problems.
Speaker 1: Compelling. Although you did start this part of the conversation,
Speaker 1: you brought it up by saying, the science team is
Speaker 1: really excited for what the model's going to learn once
Speaker 1: you have hundreds of millions of stoves.
Speaker 2: That's right, and that both things are true.
Speaker 1: I know, yes, plainly, we're just talking about sort of
Speaker 1: the margins, right, what is true? At what margins? I mean?
Speaker 1: I did wonder as I was reading about this, you know,
Speaker 1: I thought of AWS of Amazon Web Services, which of course,
Speaker 1: like was an internal Amazon thing that at some point
Speaker 1: Amazon was like, oh, maybe other people would find this
Speaker 1: service useful, And now it's a giant part of Amazon's business,
Speaker 1: and so I wondered, like, are you building Amazon robotics services.
Speaker 2: Yet not today? There's so much value that we can
Speaker 2: provide to our fulfillment business that we are one hundred
Speaker 2: percent focused on that. Certainly as a roboticist, though, I
Speaker 2: take great joy that the work we're doing is advancing
Speaker 2: the field of robots, and so it's definitely like in
Speaker 2: the makes my job better that we're advancing the state
Speaker 2: of the art. But from a business perspective, it's all
Speaker 2: hands on making the fulfillment process better for Amazon dot Com.
Speaker 1: We'll be back in just a minute. I think I
Speaker 1: read you say that you're building a foundation model of items.
Speaker 1: Is that right? And I sort of know what that means,
Speaker 1: But tell me what that means when you say that.
Speaker 2: So, when a robot handles an item, it would do
Speaker 2: better if it takes into account the properties of that item.
Speaker 2: So if you're trying to hand a bowling ball to someone,
Speaker 2: you should do that in a different way than if
Speaker 2: you're handing them a bouncy ball or a light bulb.
Speaker 2: At its core, a foundation model for items is simply
Speaker 2: a model that encodes all of those attributes of an item.
Speaker 2: And makes them available to the robotic systems that are
Speaker 2: going to use it. And one of the things that
Speaker 2: makes it a foundation model instead of just you know,
Speaker 2: some custom bespoke thing is that you can transfer it
Speaker 2: across lots of different applications. So if it's you know, stowing,
Speaker 2: you can use it. If you're packing it into a
Speaker 2: delivery box, you can use it. If you're putting it
Speaker 2: onto a shelf in a physical store like for grocery
Speaker 2: or whole foods or something, you can use it. And
Speaker 2: so that like commonality across applications is one of the
Speaker 2: things that's important.
Speaker 1: Is part of the notion there that like the model
Speaker 1: would allow a robot to sort of look at some
Speaker 1: novel item and make a reasonable inference about the properties
Speaker 1: of that item.
Speaker 2: Yeah, absolutely that. And the other thing that's a little
Speaker 2: non intuitive is that by understanding how to handle that
Speaker 2: item in all those different applications a grocery a you know,
Speaker 2: stowing picking, you get better at the individual application. So
Speaker 2: by training on all of this data across these different domains,
Speaker 2: you actually get better at the individual task that your
Speaker 2: specific robot is trying to do. Doesn't like it takes
Speaker 2: a while to like understand that it is not intuitive.
Speaker 1: SayMore, what do you mean, like, I don't know that
Speaker 1: I fully get it.
Speaker 2: Understanding how an item behaves when you gift wrap it,
Speaker 2: Uh huh shouldn't really inform how it's going to behave
Speaker 2: when you're picking it off of a bookshelf.
Speaker 1: Oh, I mean yes, it should, right, Like if you
Speaker 1: think of like a whatever, a stuffed animal versus a book. Yeah,
Speaker 1: maybe that's too easy of a case, but like if
Speaker 1: a thing is squishy or rigid, that seems like as
Speaker 1: a human being, I feel like we sort of port
Speaker 1: that knowledge from one use case to another, right.
Speaker 2: Yeah, it's a good point. And maybe that's because we
Speaker 2: are inherently sort of we think and manipulate items in
Speaker 2: the world more similarly to how these foundation models do.
Speaker 2: But ten years ago it was totally not the case.
Speaker 2: You would train your model in a very narrow domain,
Speaker 2: and if you gave it data from some other domain,
Speaker 2: it would kind of corrupt the results that you had,
Speaker 2: and so you were very careful to curate all the
Speaker 2: data that you were using to be very specific to
Speaker 2: the task that you wanted it to do, and that
Speaker 2: made the performance better. But it also meant that the
Speaker 2: model you had was only good at that one very
Speaker 2: narrow thing.
Speaker 1: It was why we were always so far from the
Speaker 1: general purpose robot. Yeah, because, as you're describing it, trying
Speaker 1: to make a robot do more than one thing just
Speaker 1: meant it couldn't even do.
Speaker 2: One We couldn't even do one thing, and so you're
Speaker 2: putting all your effort into making it do that one
Speaker 2: thing just a little better. I think there's another really
Speaker 2: interesting piece here, which is our team, the Vulcan team
Speaker 2: at Amazon, is trying to use touch and vision together,
Speaker 2: and that is how people interact with the world. That's
Speaker 2: how people manipulate the world. And so the example I
Speaker 2: like to give is picking a coin up off a
Speaker 2: table ten years ago, when a robot would try and
Speaker 2: do that, I mean it's impossible, Like, robot can't pick
Speaker 2: a coin up off table, it's too hard a task.
Speaker 2: My five year old can pick a coin up off
Speaker 2: the table in half a second without you noticing. Well,
Speaker 2: the reason is your strategy. So when you pick a
Speaker 2: coin up off the table, you actually don't grasp the coin.
Speaker 2: You go and you touch the table and then you
Speaker 2: slide your fingers along the surface of the table until
Speaker 2: you feel the coin, and when you feel the coin,
Speaker 2: that's your trigger to like rotate it up into a grasp.
Speaker 2: You're not going to some millimeter precision the way your
Speaker 2: grandfather's robot and the welding line would do. And you're
Speaker 2: not just watching with your eyes. You're using your eyes
Speaker 2: and your fingertips both your.
Speaker 1: Sense of touch. Yes, sense of touch is central to.
Speaker 2: Pick and we are trying to do those same kind
Speaker 2: of behaviors that are not only reacting to touch, but
Speaker 2: planning for touch. So the same way you plan to
Speaker 2: touch the table first, we plan to put our spatula
Speaker 2: against the side of the bookcase before we try to
Speaker 2: extend it in between this you know, small gap between
Speaker 2: the T shirt and the bag and the side of
Speaker 2: the bookcase. So we are building our plans and our
Speaker 2: controllers around having sight and touch.
Speaker 1: I mean when you say touch in the context of
Speaker 1: the robot, does that mean that it is getting feedback
Speaker 1: from the stuff it is coming into contact with? And
Speaker 1: is that novel? And how does that work?
Speaker 2: So the sensor is a force torque sensor. It looks
Speaker 2: like a hockey puck and a thousand times a second,
Speaker 2: it's telling you what it feels in the six degrees
Speaker 2: of freedom, So up, up and down is one, left
Speaker 2: and right is two, in and out as three, and
Speaker 2: then you've got roll pitch y'ah as the three torques.
Speaker 2: So a thousand times per second, you're sensing, you're feeling
Speaker 2: what the world is pushing on you with, and we
Speaker 2: use that to control the motion but also to plan
Speaker 2: the motion.
Speaker 1: When you say plan the motion, it's like, given the
Speaker 1: sense of touch that is happening right now, what should
Speaker 1: I do next? Yep.
Speaker 2: So in a like high level view, it's like touch
Speaker 2: the table first, slide along the table while keeping you know,
Speaker 2: sort of one pound of force pushing into the table
Speaker 2: until you touch the coin, and then you know, rotate.
Speaker 2: That's at a high level, but then even at a
Speaker 2: low level, the thousand times per second is so that
Speaker 2: as you slide your fingers along the table, you're sort
Speaker 2: of maintaining that accurate force.
Speaker 1: Yeah. Or like if you're putting a thing on the shelf,
Speaker 1: you can sort of tell if you've pushed it too
Speaker 1: far because the shelf is pushing back.
Speaker 2: At you exactly, or you can tell it slipping and
Speaker 2: you're about to like push over the top of it,
Speaker 2: so you can like, oh, it's about to fall over,
Speaker 2: so I can react. And those dynamics are happening at
Speaker 2: tens or hundreds of hurt since you need to sense
Speaker 2: them at a thousand hurts.
Speaker 1: What's the frontier right now for stewing? What are you
Speaker 1: trying to figure out?
Speaker 2: One of the things is getting the fullness of those
Speaker 2: bins all the way up to where they are today,
Speaker 2: so as a person you can pack those bins really,
Speaker 2: really densely, and so the robot's close but not quite
Speaker 2: as good as a person is today at getting as
Speaker 2: much stuff into the bookcase as it can. That's one frontier,
Speaker 2: and that is because one we're conservative, like our brain
Speaker 2: is telling us there's no space when really there is space.
Speaker 2: And two it's because those motions are not sophisticated enough yet.
Speaker 2: So we're trying to improve our video streaming. We're trying
Speaker 2: to get the eyes better to help as well as
Speaker 2: those low level touch centers to those behaviors to be better.
Speaker 2: So that's one of the major frontiers. The other one
Speaker 2: is the negative. The robot makes too many mistakes, so
Speaker 2: defects and exception handling are so important in robotic systems,
Speaker 2: and this is another thing I think the world on
Speaker 2: the Internet doesn't appreciate enough. Like you can do a
Speaker 2: demo and a happy path. Hey, it worked once. I
Speaker 2: can submit a paper to a conference, or I can
Speaker 2: put a cool video on YouTube. That's great. You have
Speaker 2: a demo. To have a product, you have to make
Speaker 2: sure it's working, you know, ninety nine percent of the time,
Speaker 2: or ninety nine and a half percent, or you know
Speaker 2: in some cases four nines or five nines. And a
Speaker 2: lot of the work you have to do is to
Speaker 2: recover and handle those rare exceptions or prevent or recover
Speaker 2: from those defects. And so the robot still drops too
Speaker 2: much stuff on the floor. One of our frontiers is
Speaker 2: not dropping crap on the floor, like, we need to
Speaker 2: get about three times better at that.
Speaker 1: Umly, the robot is already skipping some universe of items
Speaker 1: that the robot can't handle.
Speaker 2: Yeah, and so we need to get smarter about which
Speaker 2: items we skip and which items we take. We also
Speaker 2: need to get better at inserting those items in such
Speaker 2: a way that they're not going to fall back out?
Speaker 1: What items are particularly hard for the robot?
Speaker 2: So tight fitting items are the hardest.
Speaker 1: Uh huh. And so that's not the nature of the item,
Speaker 1: but the nature of the particular relationship between the item and.
Speaker 2: The shelf exactly.
Speaker 1: Yeah, Like, is there a kind of thing that the
Speaker 1: robot just can't do because of its shape or something.
Speaker 2: There is a particular rubber fish that we really hate.
Speaker 2: It's a dog toy.
Speaker 1: It's floppy. Is that what?
Speaker 2: Sticky?
Speaker 1: Oh? Sticky? Interesting? Yeah, And they don't put it in
Speaker 1: a bot, Nope, they just send you the sticky fish.
Speaker 2: Yeah, and it sort of gets hung up on whenever
Speaker 2: it makes contact. It doesn't slide, it like it wants
Speaker 2: to rotate about whatever it's made contact with. And so
Speaker 2: there's this particular dog toy and so we use it.
Speaker 2: We've bought like fifty of them and now we have
Speaker 2: them in the lab and this is like our diabolical
Speaker 2: item set.
Speaker 1: Is that a term of art that diabolical?
Speaker 2: I don't know, Yeah, it's our term of art. Yeah.
Speaker 2: Also bagged items where the bag is really loose. So
Speaker 2: imagine having like a T shirt in a bag, but
Speaker 2: the bag is like twice as big as the T shirt.
Speaker 1: Floppy? Is that the floppy problem?
Speaker 2: Floppy but also transparent, so sometimes you can see through
Speaker 2: the bag.
Speaker 1: Or so the robot gets confused about is the bag
Speaker 1: the item.
Speaker 2: Yeah or not? Sometimes you want one and sometimes you
Speaker 2: want the other. So like if it's just floppy plastic bag,
Speaker 2: it probably will fit. Like if I just push it
Speaker 2: into the bin, the bag is going to conform and
Speaker 2: slide in, but you can't be sure about that. You know,
Speaker 2: you get into a bunch of those edge cases that
Speaker 2: are in that long tail of being robust.
Speaker 1: I mean, it's interesting, right because the robot is dealing
Speaker 1: with this sort of human optimized world. Like it reminds
Speaker 1: me of the way I think is it. Ikea designs
Speaker 1: its furniture to fit optimally on a pallette, so you
Speaker 1: can fit the most of them, like not just the
Speaker 1: flat pack, but like in more subtle ways. And can
Speaker 1: you imagine that there is some shift in the world
Speaker 1: where I mean, obviously you're trying to make the robot better,
Speaker 1: but also people are trying to make things work better
Speaker 1: for the robot.
Speaker 2: Yes. Absolutely, And there is a different team within Amazon
Speaker 2: that's imagining a future world and future bookcases that are
Speaker 2: friendly for robots.
Speaker 1: Uh huh.
Speaker 2: However, there are currently five million of those bookshelves in
Speaker 2: warehouses holding inventory that's for sale on Amazon dot Com.
Speaker 2: And so it's a really really big lift to go
Speaker 2: replace all of those books.
Speaker 1: Shehlds interesting. So it's a whole other team that's just like,
Speaker 1: let's imagine the you know, a much more robot centric warehouse. Yeah,
Speaker 1: those guys like you don't even talk to them. They're
Speaker 1: just off in their own.
Speaker 2: I mean, they're friends, but yeah, we are spacing very
Speaker 2: different problems. And so we took a tenant very early on.
Speaker 2: It's like, the world exists, the robot needs to perform
Speaker 2: in the world as it exists. And this team they
Speaker 2: get their green field, so they get to think of
Speaker 2: like a new field. We are a brownfield, meaning we
Speaker 2: have to retrofit into these existing buildings. You know, we
Speaker 2: have like ten year leases on some of these building
Speaker 2: They're going to be there for a long long time.
Speaker 1: And then somebody else is out there. So they're building
Speaker 1: a whole other kind of robot. Your robot is optimized
Speaker 1: for the world today, and somebody else is building a
Speaker 1: robot for the robot world.
Speaker 2: That's right. I love that they have a building that
Speaker 2: they've built in Louisiana. It's in Shreveport, Louisiana. It has
Speaker 2: ten times the number of robots that traditional building has.
Speaker 2: It's a completely reimagined way of fulfilling your order. It
Speaker 2: also has a lot of people still working in those buildings,
Speaker 2: but they're working in maintenance and robotics quarterbacks jobs, and
Speaker 2: so they're higher skilled. And so we have a bunch
Speaker 2: of programs that are trying to transition our very talented
Speaker 2: workforce into the jobs of the future. One of the
Speaker 2: things I really like to say is, you don't need
Speaker 2: a college degree to work in robotics. At Amazon. It's
Speaker 2: about twenty twenty five percent of my team doesn't have
Speaker 2: a college degree but are enormously valuable. Like some of
Speaker 2: our top ten people on our team are those people.
Speaker 1: That facility in Treeport. Is it live? Like, is real
Speaker 1: stuff going in and real orders going out?
Speaker 2: Yeah, it's live. We could follow up with exactly the date,
Speaker 2: but it's been up for about a year. I think, so.
Speaker 1: Interesting thing like that, Well, I would be interested in
Speaker 1: talking to your counterpart there as well. That show would
Speaker 1: pair interestingly with this show. So okay, let's talk about
Speaker 1: the rest of the process. You know, the rest of
Speaker 1: what's going on in the warehouse and where else you're
Speaker 1: working on robots. So the piece we've been talking about
Speaker 1: this whole time is getting stuff as it comes in
Speaker 1: from the truck onto the shelf, which naively I wouldn't
Speaker 1: even think of that part, but it turns out to
Speaker 1: me this scrape big problem. What are the other pieces?
Speaker 2: What's interesting is the science we're building giving robots a
Speaker 2: sense of touch has applicability and lots and lots of
Speaker 2: places across that whole chain. Anytime the robots need to
Speaker 2: be physically interacting, like contacting, touching items is a good
Speaker 2: place for our core technology. So if we're packing four
Speaker 2: items into a box because we want to send you
Speaker 2: the four things you bought in one shipment, not in
Speaker 2: four separate packages, you need to touch the box. You
Speaker 2: need to touch the other items that are already in
Speaker 2: the box. You need to play that game of tetris.
Speaker 1: Yes, I mean it's a stowing problem again, right, I
Speaker 1: know it's called packing, but it's a version of that same.
Speaker 2: Problem, that's right, And those problems recur over and over again.
Speaker 2: So getting all of the packages, all of the cardboard
Speaker 2: boxes and paper mailers into a cart that can go
Speaker 2: onto the back of the truck, that is a stowing
Speaker 2: problem in the cart.
Speaker 1: Putting things in a thing, yeah, is a great, big
Speaker 1: problem in many ways.
Speaker 2: But you can also expand to think about grocery. So
Speaker 2: if you order produce, you don't want your grandfather's welding
Speaker 2: robot handling your peaches. It's gonna smash them, like, you
Speaker 2: need a robot with a sense of touch. If you
Speaker 2: think about household tasks, if you want a robot, you know,
Speaker 2: picking up your kid's toys or dealing with laundry, like,
Speaker 2: those robots need to have a sense of touch. They're
Speaker 2: physically interacting in a dexterous way with the world. And
Speaker 2: so one of the things that we're so excited about
Speaker 2: not only these big applications for stowing and picking off
Speaker 2: of you know, these bookcases, but everything that gets unlocked
Speaker 2: once the robot has that sense of touch.
Speaker 1: When you talk that way, it feels like a beyond
Speaker 1: what is typically considered Amazon kind of thing. It seems
Speaker 1: like a thing either Amazon's going to get into lots
Speaker 1: of other sort of non retail businesses or license the
Speaker 1: technology or sell you know, robotic touch as a service
Speaker 1: or whatever.
Speaker 2: Yeah, I think there are probably five or ten applications
Speaker 2: in how we process orders today that are all within
Speaker 2: the warehouses and delivery stations, and those are my first
Speaker 2: hill to climb. Then we do have a consumer robotics team.
Speaker 2: So there was a cool robot we released called Astro.
Speaker 2: It didn't have any manipulation capabilities, right, It would drive
Speaker 2: around your house. It had a camera on a mast
Speaker 2: that would extend up and down. You could talk to
Speaker 2: it the way you can talk to an Alexa device.
Speaker 2: The future versions of those robots are going to want
Speaker 2: to do more useful things, and so they're going to
Speaker 2: need this kind of underlying technology, and so that's a
Speaker 2: business opportunity in the long term. You know, that's not
Speaker 2: a thing my team is focused on now, but I
Speaker 2: get excited about it when I think about what we unlock.
Speaker 1: We'll be back in a minute with the lightning round.
Speaker 1: Let's do a lightning round. If you listen to the show,
Speaker 1: you have a sense of what this is. Tell me
Speaker 1: about the last time you were in zero gravity.
Speaker 2: I flew an experiment to try and drill into rocks,
Speaker 2: which was going to be applied to asteroids. And of course,
Speaker 2: if you're drilling into an asteroid, any amount you're pushing
Speaker 2: into the rock is pushing you back off into space
Speaker 2: because asteroids have almost zero gravity.
Speaker 1: Right, so you gotta have somebody push it on the
Speaker 1: other side. How do you solve that? What do you do?
Speaker 1: You grab it? How do you even do that?
Speaker 2: Says my passion for robot hands. We built a robot
Speaker 2: hand that would grab the rock with a bunch of claws.
Speaker 2: I think it had a thousand claws, and the claws
Speaker 2: were actually fish hooks. So imagine a bunch of fish
Speaker 2: hooks grabbing onto a rock to react the force of
Speaker 2: pushing a drill bit down the center.
Speaker 1: Did it work?
Speaker 2: It did work, but it only worked on rocks that
Speaker 2: were pretty rough, that had a lot of spots for
Speaker 2: the fish hooks to grab. But it turns out asteroids
Speaker 2: are really rough. Most of the smooth rocks you find
Speaker 2: on Earth have been processed by liquid water or ice,
Speaker 2: and that's not happening on asteroids. No liquid water.
Speaker 1: And so this was on the on the plane, on
Speaker 1: that NASA plane that flies what is it? Yeah, vomit
Speaker 1: combat fine curve basically, yeah, what was it like?
Speaker 2: The vomit comets actually very zen. So when you're in
Speaker 2: zero gravity, when you're floating, it's like very peaceful. It's
Speaker 2: when you're in double gravity, where you're the bottom of
Speaker 2: the parabola and you're like being glued and pushed against
Speaker 2: the floor. If you like turn your head very quickly,
Speaker 2: that's where you get like into serious trouble. And so
Speaker 2: the trick is just to like go into your zone
Speaker 2: for the bottom of the parabola and then you've become
Speaker 2: like very free and zen like in the zero G portion.
Speaker 1: You think you'll ever go to space?
Speaker 2: No, I think now that I have three kids, I
Speaker 2: think I'm landlocked.
Speaker 1: You seem a little bit sad about that. Does everybody
Speaker 1: who works at JPL kind of want to go to space?
Speaker 2: Yes, everybody that works at JPL, I think does think
Speaker 2: about going to space. I think what makes me sad
Speaker 2: is we could be doing so much more at building
Speaker 2: civilization out into space, at the scientific exploration of all
Speaker 2: of the interesting places in space, and I think we're
Speaker 2: kind of tripping ourselves up in a couple of places
Speaker 2: as a species. I wish we would get unblocked and
Speaker 2: get some of that eagerness you see some of the
Speaker 2: private investment, Like we're doing well in rockets, but we're
Speaker 2: not yet doing well in the spacecraft and the scientific
Speaker 2: instruments and the pieces that have to fly on top
Speaker 2: of the rockets.
Speaker 1: When you say we're tripping ourselves up in a couple places,
Speaker 1: in what places? Like, what do you mean?
Speaker 2: I think we became very conservative, like our risk posture
Speaker 2: about going to space. We stopped treating it as this
Speaker 2: like very dangerous activity and tried to make it extremely safe,
Speaker 2: and that slowed us down.
Speaker 1: To bring back to cowboys a little bit.
Speaker 2: Yeah. Interesting, Yeah, and then there's a lot of bureaucracy,
Speaker 2: of course, that built up over fifty years. I still
Speaker 2: have very optimistic there's a lot of smart people working
Speaker 2: in that area and a lot of exciting things happening,
Speaker 2: So we're going to get through it.
Speaker 1: Aaron Parness is a director of Applied Science at Amazon Robotics.
Speaker 1: Please email us at problem at Pushkin dot fm. We
Speaker 1: are always looking for new guests for the show. Today's
Speaker 1: show was produced by Trinomnino and Gabriel Hunter Chang. It
Speaker 1: was edited by Alexander Garretton and engineered by Sarah Bruguerrett.
Speaker 1: I'm Jacob Goldstein, and we'll be back next week with
Speaker 1: another episode of What's Your Problem.