WEBVTT - Why Amazon Built a Spatula-Wielding Robot 0:00:15.356 --> 0:00:15.796 Pushkin. 0:00:20.396 --> 0:00:22.596 One of the things I look at in robotics as 0:00:22.636 --> 0:00:27.076 a big field. There are so many amazing demonstrations of mobility, 0:00:27.636 --> 0:00:32.396 robots doing backflips, robots running down hills, and that's really 0:00:32.396 --> 0:00:34.916 impressive to me because I can't do a backflip, or 0:00:35.036 --> 0:00:37.756 I might trip if I run down the hill. But 0:00:37.876 --> 0:00:41.636 where the really valuable parts of robotics are going to 0:00:41.676 --> 0:00:46.036 be are in manipulation. So my kid can take a 0:00:46.036 --> 0:00:48.436 blueberry out of her cereal bowl because she doesn't want 0:00:48.476 --> 0:00:51.396 to eat it, and that is an incredibly hard task 0:00:51.436 --> 0:00:54.196 for a robot. And you don't see any of those demos. 0:00:54.276 --> 0:00:58.156 And I think we're like kind of inherently programmed as 0:00:58.236 --> 0:01:01.476 people to like be biased towards the backflip being more 0:01:01.516 --> 0:01:04.796 impressive and in reality like the business value and the 0:01:04.836 --> 0:01:07.516 harder thing for the robot is to like take the 0:01:07.516 --> 0:01:13.876 blueberry out of the cereal bowl. 0:01:15.076 --> 0:01:17.436 I'm Jacob Goldstein and this is What's Your Problem, the 0:01:17.476 --> 0:01:19.516 show where I talk to people who are trying to 0:01:19.596 --> 0:01:24.676 make technological progress. My guest today is Aaron Parness. Aarin 0:01:24.756 --> 0:01:27.876 spent the earlier part of his career building space robots 0:01:27.876 --> 0:01:32.236 at NASA's Jet Propulsion Laboratory JPL. Six years ago, he 0:01:32.276 --> 0:01:35.676 went to work at Amazon. Now, Aaron is a director 0:01:35.716 --> 0:01:39.636 of Applied Science at Amazon Robotics. I wanted to talk 0:01:39.676 --> 0:01:43.356 to Erin about a robot arm called Vulcan. He and 0:01:43.436 --> 0:01:45.996 his team developed Vulcan to do a job that is 0:01:46.076 --> 0:01:50.276 surprisingly hard for robots to do, taking stuff that gets 0:01:50.316 --> 0:01:55.236 delivered to Amazon warehouses and putting it onto shelves. In 0:01:55.316 --> 0:01:58.236 order to solve this problem, Aaron and his team had 0:01:58.236 --> 0:02:00.676 to build a robot that had a sense of touch, 0:02:01.156 --> 0:02:05.116 that could deal with complicated, unpredictable situations, and that could 0:02:05.196 --> 0:02:08.036 look at a shelf and plan out a course of action. 0:02:09.156 --> 0:02:11.796 As you'll hear in the interview, all of those traits 0:02:11.916 --> 0:02:14.836 may someday be helpful, not just in stocking shelves in 0:02:14.876 --> 0:02:18.036 a warehouse, but in doing lots of boring sounding but 0:02:18.196 --> 0:02:23.316 complicated real world tasks like, for example, taking a blueberry 0:02:23.396 --> 0:02:26.196 out of a bowl of cereal. To start, I asked 0:02:26.196 --> 0:02:28.716 Aaron to tell me the problem that Vulcan was designed 0:02:28.756 --> 0:02:30.716 to solve at Amazon's warehouses. 0:02:31.516 --> 0:02:35.356 So new inventory comes into the building. You know, trucks 0:02:35.356 --> 0:02:38.156 pull up and they unload new stuff. We need to 0:02:38.196 --> 0:02:40.796 store that stuff while it's waiting for someone to click 0:02:40.836 --> 0:02:45.236 the buy button. We store it in these large fabric bookcases. 0:02:46.116 --> 0:02:49.276 It's about eight feet tall. It has about forty different 0:02:49.316 --> 0:02:53.076 shelves on it. It's four sided, so you can store 0:02:53.156 --> 0:02:56.996 stuff from any of the different faces of the case. 0:02:57.756 --> 0:03:01.156 What's really interesting is this stuff is randomly stowed, so 0:03:01.196 --> 0:03:04.036 it's not like all the iPhones are in one shelf. 0:03:04.476 --> 0:03:08.076 It'll be all different stuff, all mixed together. 0:03:08.596 --> 0:03:11.356 When you say random, do you mean random or do 0:03:11.396 --> 0:03:13.876 you mean it would look random to the untrained eye. 0:03:14.116 --> 0:03:18.476 I mean literally random. Really wherever there is space you 0:03:18.556 --> 0:03:20.436 can put the item, because. 0:03:20.076 --> 0:03:22.956 That's what's optimal. It turns out the optimal way to 0:03:22.956 --> 0:03:24.516 store stuff is random. 0:03:24.236 --> 0:03:28.956 That's right. Why the stems actually from Jeff Bezos is 0:03:28.996 --> 0:03:32.516 like original vision, I think, and it's it's incredible. So 0:03:33.556 --> 0:03:36.596 you want to have the most selection, and you want 0:03:36.636 --> 0:03:39.156 to have speed of delivery, and you want to have 0:03:39.196 --> 0:03:41.556 low cost, and that's what the customer wants, right. The 0:03:42.276 --> 0:03:46.116 customer is using Amazon dot Com because we have selection, 0:03:46.236 --> 0:03:49.356 we have speed, and we have low cost. In order 0:03:49.396 --> 0:03:53.076 to achieve that, you have to have these massive warehouses 0:03:53.676 --> 0:03:56.996 located really close to your customers, and you have a 0:03:56.996 --> 0:03:59.316 lot of customers in Tokyo, in New York City and 0:03:59.356 --> 0:04:03.356 San Francisco where real estate's really expensive, so you have 0:04:03.396 --> 0:04:05.076 to figure out a way to put all of this 0:04:05.276 --> 0:04:10.116 different stuff in like the densest packing area, you can 0:04:10.596 --> 0:04:13.516 and have access to it immediately so that you can 0:04:13.676 --> 0:04:17.636 you can deliver in hours instead of days. And what 0:04:17.676 --> 0:04:22.556 that means is that random is better than structured. So 0:04:22.676 --> 0:04:25.476 anywhere there's a space, you can add that item into 0:04:25.516 --> 0:04:28.196 the inventory, and that means it comes up for sale 0:04:28.236 --> 0:04:31.916 immediately on the website, and then when someone places in order, 0:04:32.596 --> 0:04:34.956 you don't have to wait for that iPhone bookcase to 0:04:34.996 --> 0:04:37.436 make its way all the way across the warehouse. You 0:04:37.516 --> 0:04:40.076 probably have a thousand iPhones in the warehouse, and whichever 0:04:40.116 --> 0:04:43.676 one is closest can go to whichever pickstation is eligible, 0:04:44.316 --> 0:04:49.716 and it ends up being actually substantially faster. So that last. 0:04:49.676 --> 0:04:52.396 Sentence seems to be the key. The idea is like, yes, 0:04:52.436 --> 0:04:55.076 given you have whatever a thousand iPhones in the warehouse 0:04:56.076 --> 0:04:57.876 in the universe where a human had to know where 0:04:57.876 --> 0:04:59.676 they all were, you'd put them all on one shelf. 0:05:00.076 --> 0:05:02.516 But you're saying at any given time that means that 0:05:02.516 --> 0:05:05.836 shelf is probably going to be pretty far away, whereas 0:05:05.916 --> 0:05:09.756 if you randomly distribute them throughout the shelves and warehouse 0:05:10.236 --> 0:05:12.676 at any given time, one of those thousand iPhones is 0:05:12.676 --> 0:05:14.556 probably going to be pretty close to where it needs 0:05:14.596 --> 0:05:17.716 to be. And because you have a whatever, a computerized 0:05:17.716 --> 0:05:19.836 system that can keep track of everything all the time, 0:05:19.956 --> 0:05:22.276 it makes sense to randomly distribute all the things. 0:05:22.396 --> 0:05:25.236 Yeah, that's exactly right. And it works on the flip 0:05:25.276 --> 0:05:27.396 side as well. So when you have a new item 0:05:27.436 --> 0:05:30.676 that's come in, rather than waiting for the shelf that 0:05:30.716 --> 0:05:33.516 has the right size thing to put the new dog 0:05:33.596 --> 0:05:36.116 toy in, you just put the dog toy anywhere you 0:05:36.156 --> 0:05:37.076 can find space for it. 0:05:37.796 --> 0:05:41.636 H It's like my house. We have a lot of 0:05:41.676 --> 0:05:45.316 dog toys in my house. Also, Yeah, that's really interesting. 0:05:45.396 --> 0:05:47.996 It's great for the customer, and that's optimal. 0:05:48.036 --> 0:05:54.676 It's optimal, and it creates an incredibly difficult environment for robotics. Huh, 0:05:54.756 --> 0:05:57.076 because now you have to deal with all this clutter. 0:05:57.556 --> 0:06:00.316 We can have more than a million unique items in 0:06:00.436 --> 0:06:03.156 one warehouse. Yeah, so it's not like you have a 0:06:03.196 --> 0:06:06.756 model of each of those items. And we sell more 0:06:06.956 --> 0:06:10.596 third party items than you know, Amazon owns themselves. Right. 0:06:10.636 --> 0:06:14.036 We are a platform for third party fulfillment, and so 0:06:14.836 --> 0:06:17.116 you don't have all the data about all those items, 0:06:17.156 --> 0:06:19.236 and so you have to handle all this uncertainty, all 0:06:19.276 --> 0:06:22.516 this clutter, and everything's tightly packed. 0:06:22.596 --> 0:06:26.196 And so still in most places as a result, when 0:06:26.316 --> 0:06:28.836 stuff comes into the warehouse every day off a truck 0:06:28.876 --> 0:06:32.636 to people take the things out of the truck and 0:06:32.676 --> 0:06:35.156 stick them randomly on shelves wherever they can find space. 0:06:35.236 --> 0:06:36.116 Is that the system? 0:06:36.196 --> 0:06:38.876 That is exactly the system, and it's in you know, 0:06:39.036 --> 0:06:41.716 hundreds of buildings around the world. 0:06:42.556 --> 0:06:44.876 And just to be clear, I mean it's pretty clear, 0:06:44.916 --> 0:06:48.236 but just to really put a point on it, why 0:06:48.396 --> 0:06:50.756 is this a hard environment for robots? 0:06:51.396 --> 0:06:56.916 Traditional industrial robots do not handle contact well, so like 0:06:57.036 --> 0:07:03.836 touching their environments, and they don't handle clutter or you know, uncertainty, 0:07:03.956 --> 0:07:07.356 and so it's hard because to put that last book 0:07:07.356 --> 0:07:11.876 onto the bookshelf, squeeze that teddy Bear into the just 0:07:12.036 --> 0:07:15.036 small enough space that it'll fit, you have to push 0:07:15.076 --> 0:07:18.396 the other stuff around that's already on that bookshelf. And 0:07:18.916 --> 0:07:21.996 a traditional robot doesn't have sensors, it doesn't even know 0:07:22.036 --> 0:07:24.036 how to do that. So if you think of like 0:07:24.076 --> 0:07:28.156 a car manufacturing line, you're like nineteen nineties two thousands, 0:07:28.196 --> 0:07:31.516 you know, welding robot or loading sheet metal into a press. 0:07:32.196 --> 0:07:34.956 It's doing all of that only knowing its position in space. 0:07:35.356 --> 0:07:38.396 So it has no force sensing. If it runs into something, 0:07:38.556 --> 0:07:41.996 it either is like an emergency stop because it's like broken, 0:07:42.476 --> 0:07:44.756 or it just smashes that thing and keeps going and 0:07:44.836 --> 0:07:47.356 it doesn't even know it's smashed anything. It literally has 0:07:47.476 --> 0:07:48.236 no sensing. 0:07:48.956 --> 0:07:53.036 That is an incredibly homogeneous environment. Right, It's doing like 0:07:53.076 --> 0:07:55.196 the exact same thing at a very high level of 0:07:55.196 --> 0:07:57.996 precision forever one thing. 0:07:57.916 --> 0:08:01.876 That's exactly right. And so this extension, the fundamental breakthrough 0:08:02.516 --> 0:08:05.876 for science for robotics manipulation that my team is trying 0:08:05.876 --> 0:08:08.316 to make is one giving the robot a sense of 0:08:08.356 --> 0:08:11.796 touch and using that along with site and along with 0:08:11.996 --> 0:08:16.556 like knowing where your robot is to do meaningfull tasks 0:08:16.596 --> 0:08:20.276 in like very high contact, high clutter environments. And then 0:08:20.276 --> 0:08:23.316 there's a brain part. It's also much more difficult to 0:08:23.396 --> 0:08:27.476 kind of predict how this random assortment of items is 0:08:27.516 --> 0:08:30.436 going to move or change as you push on it. 0:08:30.756 --> 0:08:33.116 And so there's an AI piece, there's a brain piece 0:08:33.196 --> 0:08:36.196 that's saying this item will fit in that bin. This 0:08:36.276 --> 0:08:37.916 is actually one of the most frustrating things when you 0:08:37.916 --> 0:08:40.396 try and do the job yourself. I'm like an optimist. 0:08:40.396 --> 0:08:42.356 I'm always oh, yeah, this will fit. And I go 0:08:42.436 --> 0:08:44.236 up there and I try and play Tetris and I 0:08:44.276 --> 0:08:47.196 try and rearrange the shelf and like, it clearly isn't 0:08:47.196 --> 0:08:49.556 going to fit. And then I've wasted thirty seconds or 0:08:49.556 --> 0:08:52.076 forty seconds and I have to try something else. 0:08:53.036 --> 0:08:56.036 That's a good statement of the problem. Well, like when 0:08:56.036 --> 0:08:57.236 did you come onto the scene? 0:08:58.196 --> 0:09:01.276 So I was working on some other stuff and there 0:09:01.436 --> 0:09:06.476 was a recent PhD that had joined our team. He was, 0:09:06.516 --> 0:09:08.316 you know, one year out of school something like this, 0:09:08.956 --> 0:09:10.796 and he says, I'm going to go try and solve 0:09:10.996 --> 0:09:15.476 stowing items into these bookshelves. And my thought was, Oh, 0:09:15.476 --> 0:09:18.116 how naive, Like the real world is going to teach 0:09:18.556 --> 0:09:21.836 this new grad. That's just way too hard a problem 0:09:21.836 --> 0:09:24.716 for robotics to solve. But I was helping him because 0:09:24.756 --> 0:09:26.596 it's fun, right, Like you like to work on hard 0:09:26.596 --> 0:09:29.836 problems when you're a researcher. And he was a very 0:09:29.916 --> 0:09:32.196 nice guy, and so I was, you know, helping him, 0:09:32.236 --> 0:09:34.356 but never thought it was going to work. And there 0:09:34.356 --> 0:09:36.476 were a couple of kind of moments where we made 0:09:36.516 --> 0:09:41.556 these simplifications that turn the problem from I have to 0:09:41.596 --> 0:09:44.996 try and do every possible game of Tetris that a 0:09:45.036 --> 0:09:49.996 person can do into a problem where you're like, oh, 0:09:50.676 --> 0:09:52.436 it's not that this is never going to work, it's 0:09:52.436 --> 0:09:54.836 that this is the future, Like this is robotics two 0:09:54.836 --> 0:09:57.356 point zero, Like this is I have to work on this. 0:09:57.436 --> 0:10:00.116 I can't do anything else anymore. I'm like, I'm all 0:10:00.156 --> 0:10:01.236 in on this problem. 0:10:01.756 --> 0:10:05.596 Tell me about one of those simplifications one of those moments. 0:10:05.836 --> 0:10:09.276 It was the gripper is one. The design, the mechanic 0:10:09.476 --> 0:10:13.076 design of the robotic hand was actually a big breakthrough. 0:10:13.396 --> 0:10:17.036 And when we started, we were trying to push items 0:10:17.396 --> 0:10:19.876 with the item we were gripping. So imagine you're pinching 0:10:19.916 --> 0:10:21.996 a book and you're trying to use that book to 0:10:22.076 --> 0:10:25.236 like push this dog toy over to the side. 0:10:25.916 --> 0:10:27.516 I see, So you want to put the book in 0:10:27.556 --> 0:10:30.516 a bin. Yeah, dog toys in the way. So you're like, okay, 0:10:30.556 --> 0:10:32.236 pick up the book and use the book kind of 0:10:32.316 --> 0:10:34.516 like a brush to sweep the dog toy out of 0:10:34.556 --> 0:10:34.836 the way. 0:10:35.036 --> 0:10:37.916 Okay, And I say, okay, like I understand, but it's 0:10:37.996 --> 0:10:39.756 never going to work. What if you don't have a book. 0:10:39.756 --> 0:10:41.676 What if you have a T shirt? Yeah, what if 0:10:41.716 --> 0:10:44.836 you have an iPhone and it's very expensive? Are you 0:10:44.916 --> 0:10:46.756 going to actually want to start pushing on stuff with 0:10:46.836 --> 0:10:49.996 the phone? And so we came up with this strategy 0:10:50.156 --> 0:10:53.556 to have like a spatula that would extend into the 0:10:53.596 --> 0:10:56.436 bin and you'd push everything with this spatula that was 0:10:56.596 --> 0:11:00.116 part of your hand. So imagine like you're like Wolverine 0:11:00.316 --> 0:11:02.716 and you can shoot out, you know, but instead of 0:11:02.756 --> 0:11:05.676 like the Adamantium claws, you're shooting out a spatula. 0:11:07.516 --> 0:11:10.676 So it's like a pincher grip. A little spatula shoots 0:11:10.716 --> 0:11:12.116 forward out of the pincher grip. 0:11:12.236 --> 0:11:13.476 Is the thing that's right. 0:11:13.876 --> 0:11:16.796 It's so simple when you put it that way. I mean, 0:11:16.836 --> 0:11:18.756 I'm sure making it was not low tech, but it 0:11:18.796 --> 0:11:22.556 sounds very like. It's not like some crazy AI thing. 0:11:22.636 --> 0:11:24.556 It's like just what if there was another little thing 0:11:24.596 --> 0:11:26.756 that came out and push stuff out of the way. 0:11:26.916 --> 0:11:29.756 But those ideas are like the really powerful ones when 0:11:29.796 --> 0:11:33.956 you have a simple, elegant solution and you're like, okay, 0:11:34.396 --> 0:11:38.476 that could work. That's different than like a five fingered 0:11:38.556 --> 0:11:42.636 hand that has twenty five motors embedded in it. Yeah, 0:11:42.796 --> 0:11:44.596 like Oh, it's just dispatulate. 0:11:44.196 --> 0:11:48.716 Fingers are famously difficult. Why didn't anybody think of it before? 0:11:49.516 --> 0:11:51.836 So we had been working on it as a company 0:11:51.956 --> 0:11:54.396 back to the Amazon Picking challenge, which was, you know, 0:11:54.436 --> 0:11:59.916 twenty fifteen. But I think a lot of robotics researchers 0:11:59.956 --> 0:12:02.396 like myself, were scared that this problem was just too hard. 0:12:02.436 --> 0:12:04.476 There was easier things to go try and work on, 0:12:05.276 --> 0:12:08.276 and there were a couple of simplifications, so using this 0:12:08.316 --> 0:12:11.356 spatulo as one, and then you watch people do the 0:12:11.396 --> 0:12:13.716 task and you realize they're kind of doing the same 0:12:14.876 --> 0:12:19.276 strategies over and over again. It's like insert dispatchela on 0:12:19.316 --> 0:12:23.596 the D and sweep to one side. For this kind 0:12:23.596 --> 0:12:26.716 of page turn mechanism. Something's fallen over and you need 0:12:26.756 --> 0:12:29.396 to sort of flip it back up to make space. 0:12:30.316 --> 0:12:33.076 So you put this spatula underneath it and flip the 0:12:33.116 --> 0:12:35.276 thing up ninety degreees basically, Yeah. 0:12:35.156 --> 0:12:38.396 And you realize that accounts for like ninety percent of 0:12:38.436 --> 0:12:41.476 the actions you do when you try and stow into 0:12:41.516 --> 0:12:42.076 these pins. 0:12:42.796 --> 0:12:45.996 And did you figure that out by watching people stow. 0:12:45.956 --> 0:12:47.956 We did and doing it yourself. 0:12:48.356 --> 0:12:52.356 How much stowing did you do couple of days? Okay, 0:12:52.516 --> 0:12:56.156 it's a hard job, thousands of items probably, I imagine. 0:12:55.596 --> 0:12:59.116 Yeah, exactly. And we tried to wear go pro cameras 0:12:59.116 --> 0:13:00.796 on our heads so we could look at the videos later, 0:13:00.836 --> 0:13:03.716 which turns out as a recipe for motion sickness. It's 0:13:03.876 --> 0:13:06.516 very difficult to watch those videos, but you go and 0:13:06.556 --> 0:13:08.356 you do it, and you build up this intuition. And 0:13:08.916 --> 0:13:11.036 I I think the other piece of the problem that 0:13:11.556 --> 0:13:14.036 made it tractable and made me this like huge believer 0:13:14.636 --> 0:13:16.756 was recognizing we didn't have to get to one hundred percent. 0:13:17.356 --> 0:13:21.716 So in some automation scenarios, you have to solve the 0:13:21.756 --> 0:13:24.956 whole problem, and if you don't, you have nothing, so 0:13:24.996 --> 0:13:27.476 like landing on the moon. And what we realized was 0:13:27.516 --> 0:13:30.076 there was a way to like make the business logic 0:13:30.196 --> 0:13:34.196 work that the robot could handle seventy five percent of 0:13:34.236 --> 0:13:36.756 the stoves and it just had to not make a 0:13:36.796 --> 0:13:41.036 mess and let like work alongside people to do the 0:13:41.116 --> 0:13:44.276 other twenty five percent, and the sum of the parts 0:13:44.316 --> 0:13:47.836 is actually much better than either all robots or all 0:13:47.956 --> 0:13:51.876 employees would be on their own. And making that realization 0:13:52.916 --> 0:13:54.396 all of a sudden meant that it could be a 0:13:54.836 --> 0:13:57.316 two or three year project instead of a twenty year 0:13:57.356 --> 0:14:01.276 project because chasing this long tail. You know, we have 0:14:01.316 --> 0:14:04.156 a million unique items in the building, but we also 0:14:04.236 --> 0:14:06.636 process a million items per day. So I have a 0:14:06.676 --> 0:14:09.796 phrase like, if something goes wrong one in a million, 0:14:09.876 --> 0:14:13.636 it happens every day in every Amazon building. And to 0:14:13.676 --> 0:14:15.596 try and solve all of those is it is a 0:14:15.636 --> 0:14:16.596 twenty year problem. 0:14:17.076 --> 0:14:20.796 I feel like that part of the solution generalizes in 0:14:20.836 --> 0:14:23.436 a really nice way, right, Like I mean, I guess 0:14:23.436 --> 0:14:26.636 the eighty twenty problem is a sort of cliche. But 0:14:27.236 --> 0:14:30.516 the idea that like, oh, if you think of the 0:14:30.556 --> 0:14:32.636 problem the right way, it's like, no, we don't have 0:14:32.636 --> 0:14:34.156 to build a robot that does it every time. Before 0:14:34.156 --> 0:14:36.156 we build a robot that does it seventy five percent 0:14:36.196 --> 0:14:38.916 of the time. That is a huge efficiency gain and 0:14:38.956 --> 0:14:42.636 maybe the optimal point on the curve. Right yep, if 0:14:42.676 --> 0:14:45.956 the robot is doing everything, you're working too hard to 0:14:45.956 --> 0:14:46.876 make the robot work. 0:14:46.956 --> 0:14:48.556 Probably exactly that. 0:14:49.716 --> 0:14:52.956 So, Okay, so you have these two big ideas. Do 0:14:53.036 --> 0:14:54.916 you want to tell me the sort of story of 0:14:54.956 --> 0:14:56.876 making it work? You want to tell me how it works. 0:14:57.156 --> 0:15:00.516 We've been running six of these robots at a warehouse 0:15:00.636 --> 0:15:06.076 in Spokane, Washington, Okay, since November of last year and 0:15:06.116 --> 0:15:09.676 so we've done over half a million stows it this point. 0:15:10.116 --> 0:15:12.916 We also have another product that's picking those items out 0:15:12.916 --> 0:15:16.236 of the bins, and so that's my team in Germany, 0:15:16.316 --> 0:15:19.196 and so we have a warehouse in Homburg where we've 0:15:19.236 --> 0:15:22.716 been picking items. And picking is a slightly harder problem 0:15:22.756 --> 0:15:25.436 in some ways because you have to identify the item. 0:15:25.836 --> 0:15:28.196 So for stow, you have to identify free space. It's 0:15:28.236 --> 0:15:32.116 either occupied or you can make space to put the 0:15:32.156 --> 0:15:34.276 next item in. For pick, you want to make sure 0:15:34.276 --> 0:15:36.476 I get you the red T shirt, not the red sweatpants, 0:15:36.556 --> 0:15:39.276 or I get you the Harry Potter volume two and 0:15:39.356 --> 0:15:41.396 not Sapiens or some other book. 0:15:41.956 --> 0:15:44.156 Tell me how it works. Let's do this stewing first, 0:15:44.196 --> 0:15:46.356 since that's what we've been talking about. So there's this 0:15:46.476 --> 0:15:49.836 warehouse in Spokane where this robot that you built is 0:15:49.876 --> 0:15:52.916 in use. Like what happens there? A truck pulls in 0:15:53.076 --> 0:15:54.036 and then what happens. 0:15:54.476 --> 0:15:57.796 The way the system works is one of these pods, 0:15:57.876 --> 0:16:00.956 one of these bookcases pulls up to the station, so 0:16:00.996 --> 0:16:04.036 it pulls in front of the robot. We have stereo 0:16:04.116 --> 0:16:07.076 camera towers and so we're looking with the eyes first, 0:16:07.756 --> 0:16:12.716 and we are creating a three D representation of the scene. 0:16:12.796 --> 0:16:15.996 So we're modeling, you know, all the items that are 0:16:16.036 --> 0:16:20.716 in the in the pod already. But the really interesting 0:16:20.756 --> 0:16:23.796 part is we're actually predicting on top of that, how 0:16:23.836 --> 0:16:27.676 we can move those items around to make more empty space, 0:16:28.156 --> 0:16:30.996 how can we squeeze more stuff in. So it's not 0:16:31.276 --> 0:16:35.316 just identifying vacant space. You have to predict where you 0:16:35.356 --> 0:16:40.156 can make that vacant space by pushing stuff with this spatula. Okay, 0:16:40.356 --> 0:16:43.556 then we do this matching algorithm. So we have about 0:16:43.636 --> 0:16:46.916 forty or fifty items waiting for us to stow, and 0:16:46.956 --> 0:16:49.956 so we have a variety of stuff, and we're matching 0:16:49.996 --> 0:16:53.756 those forty or fifty items to the thirty ish shelves 0:16:53.836 --> 0:16:56.276 that are in front of the robot. Which items should 0:16:56.276 --> 0:16:58.876 go where, and then how do we make that space? 0:16:59.436 --> 0:17:01.636 And so that's where a lot of the AI in 0:17:01.716 --> 0:17:05.436 the system is active and operating. It's predicting success, it's 0:17:05.956 --> 0:17:08.756 minimizing risk, it's trying to optimize for a bunch of 0:17:08.756 --> 0:17:14.196 different parameters. Once we've made that selection, we grasp the item, 0:17:14.436 --> 0:17:18.036 so that item we've selected for putting into the given 0:17:18.076 --> 0:17:21.876 shelf passes into our hand and our hand is two 0:17:21.956 --> 0:17:24.196 conveyor belt paddles, so you can think of it kind 0:17:24.196 --> 0:17:27.036 of like a Panini press, like a George Foreman grill. 0:17:27.116 --> 0:17:30.596 It is a George Foreman grill where each side has 0:17:30.636 --> 0:17:32.316 a conveyor built into it. 0:17:32.596 --> 0:17:35.476 Like a little belt, Like just a little belt going around. 0:17:35.636 --> 0:17:37.796 That's right. So each face of the grill, the top 0:17:37.796 --> 0:17:40.876 face and the bottom face have a conveyor belt. And 0:17:40.916 --> 0:17:43.516 that's important because you can control the pose of the 0:17:43.556 --> 0:17:47.436 item and you can feed it into the bin rather 0:17:47.476 --> 0:17:50.676 than like throwing it into the bin. One of the 0:17:50.716 --> 0:17:53.076 early versions we had kind of dropped it and tried 0:17:53.116 --> 0:17:55.036 to punch it to put the item into the bin, 0:17:55.076 --> 0:17:57.396 and that predictably failed in a lot. 0:17:57.196 --> 0:18:00.876 Of Well you say predictably now, but if you try 0:18:00.916 --> 0:18:02.236 to it wasn't predictable. 0:18:02.396 --> 0:18:05.556 Yeah. Yeah. I'm a huge believer in iterative design, and 0:18:05.636 --> 0:18:09.716 so we try and build early, build often build and 0:18:09.796 --> 0:18:13.716 learn from those builds. So it's actually really important to 0:18:13.916 --> 0:18:17.956 keep sixed off pose control of the item. So you 0:18:17.996 --> 0:18:20.756 want to make sure the item isn't rotating as you 0:18:20.756 --> 0:18:22.556 shoot it out. You want to make sure that you 0:18:22.796 --> 0:18:25.996 keep the orientation of the item because it's fitting tightly, 0:18:26.636 --> 0:18:28.956 so you don't want it to run into the bookshelf 0:18:28.996 --> 0:18:31.796 above it or below it, or the items thirty in there. 0:18:31.876 --> 0:18:34.676 Yeah yeah, yeah. We started by trying to shoot it out, 0:18:34.676 --> 0:18:36.236 and then we had all kinds of problems when it 0:18:36.236 --> 0:18:38.756 would like collide with stuff and fall on the floor. 0:18:39.316 --> 0:18:42.236 The worst case is, you know, you would shoot it out, 0:18:42.236 --> 0:18:44.516 it would bounce off the back of the back of 0:18:44.556 --> 0:18:46.196 the bookcase and then come back and hit you in 0:18:46.196 --> 0:18:47.356 the face or hit you in your. 0:18:47.436 --> 0:18:48.076 Did that happen? 0:18:48.156 --> 0:18:48.996 Yeah? Oh yeah, good. 0:18:49.036 --> 0:18:52.636 That's robot comedy, yesh, ro but physical comedy. 0:18:52.916 --> 0:18:53.396 Yeah. 0:18:53.476 --> 0:18:54.556 So, and that's it. 0:18:54.916 --> 0:18:56.956 That's the stow process. And we want to do that 0:18:57.036 --> 0:18:59.516 a few hundred times an hour, and we want to 0:18:59.516 --> 0:19:03.476 do it on the top shelves of those bookcases where. Yeah. 0:19:03.516 --> 0:19:06.596 That's one of the ways we are really complimentary to 0:19:06.716 --> 0:19:10.116 the employees is if the robots can do the top shelves, 0:19:09.836 --> 0:19:13.436 it saves a lot of ergonomic tasks. It allows the 0:19:13.556 --> 0:19:16.436 employees to work in their power zone, like you know, 0:19:17.156 --> 0:19:20.236 shoulder level. That makes them faster too. So if you 0:19:20.236 --> 0:19:22.436 put robots in, people get faster at the job. 0:19:22.916 --> 0:19:25.996 I mean, presumably as the robot gets better, it'll also 0:19:26.036 --> 0:19:27.836 be better at putting things on the middle shelf. 0:19:28.036 --> 0:19:29.716 Right, Well, there's this like sweet spot. 0:19:29.756 --> 0:19:31.956 The robot's going to get better faster than people will 0:19:31.996 --> 0:19:32.436 get better. 0:19:33.036 --> 0:19:34.996 Yeah, we want the robots to be as good as 0:19:34.996 --> 0:19:37.396 it can and not chase one hundred percent. We don't 0:19:37.396 --> 0:19:39.756 really believe in one hundred percent automation. We want to 0:19:40.396 --> 0:19:43.236 find that sweet spot where we're maximizing product. 0:19:43.236 --> 0:19:45.476 Mean, the sweet spot's going to keep moving, right, The 0:19:45.556 --> 0:19:47.716 robot's going to get better and better and be able 0:19:47.716 --> 0:19:49.356 to do more and more faster and faster. 0:19:49.196 --> 0:19:52.516 Presumably, and my science team's actually really excited about that. 0:19:52.716 --> 0:19:54.996 As you get more and more data. So we have 0:19:55.036 --> 0:19:58.276 five hundred thousand stows that we've done so far, but 0:19:58.356 --> 0:20:00.876 when we get to five hundred million STOs, you can 0:20:00.956 --> 0:20:03.676 leverage some of these techniques to start learning the motions 0:20:03.716 --> 0:20:06.676 and learning some of these strategies and refining them to 0:20:06.716 --> 0:20:09.196 be specific to the item that you are molding in 0:20:09.196 --> 0:20:12.836 your hand. There's a lot of opportunity as you get 0:20:12.876 --> 0:20:13.916 more and more data. 0:20:14.156 --> 0:20:16.556 Well, right, so we haven't really I mean, you mentioned 0:20:16.636 --> 0:20:19.236 the software side, the AI side, but we haven't really 0:20:19.276 --> 0:20:22.316 talked about it, and it is I mean, in talking 0:20:22.316 --> 0:20:26.076 to other people working on robotics. It's plainly a data 0:20:27.036 --> 0:20:30.356 game because there's no Internet of the physical world, right, 0:20:30.356 --> 0:20:33.436 because large language models work so well, because there's this 0:20:33.556 --> 0:20:36.796 huge data set and everybody is trying to get data 0:20:36.836 --> 0:20:40.396 from the physical world, and you seem very well positioned 0:20:40.436 --> 0:20:42.756 to get a lot of data from the physical world. 0:20:43.516 --> 0:20:46.236 I think that's true. So one of the joys of 0:20:46.276 --> 0:20:48.756 being a roboticist at Amazon is all the data that 0:20:48.796 --> 0:20:51.796 we have access to. But I will push back a 0:20:51.796 --> 0:20:54.476