WEBVTT - Teaching Robots How to Do Everything 0:00:15.356 --> 0:00:23.476 Pushkin. In a metaphorical sense, AI is everywhere. It can 0:00:23.556 --> 0:00:26.356 write essays, it can do your texes, it can design drugs, 0:00:26.356 --> 0:00:30.516 it can make movies. But in a literal sense, AI 0:00:31.236 --> 0:00:35.356 is not everywhere. You know, a large language model can 0:00:35.396 --> 0:00:38.196 tell you whatever twenty seven ways to fold your shirts 0:00:38.196 --> 0:00:40.836 and put them in the drawer, but there's no robot 0:00:40.916 --> 0:00:44.076 that you can buy that can actually fold your shirts 0:00:44.156 --> 0:00:46.876 and put them in the drawer. At some point, though 0:00:47.596 --> 0:00:50.116 maybe at some point in the not that distant future, 0:00:50.756 --> 0:00:53.516 there will be a robot that can use AI to 0:00:53.596 --> 0:00:55.316 learn how to fold your shirts and put them in 0:00:55.316 --> 0:00:58.996 the drawer, or you know, cook lasagna, pack boxes, plug 0:00:58.996 --> 0:01:02.196 in cables. In other words, there will be a robot 0:01:02.316 --> 0:01:06.196 that can use AI to learn how to do basically anything. 0:01:12.276 --> 0:01:14.636 I'm Jacob Goldstein and this is What's Your Problem, the 0:01:14.676 --> 0:01:16.436 show where I talk to people who are trying to 0:01:16.436 --> 0:01:20.876 make technological progress. My guest today is Chelsea Finn. She's 0:01:20.916 --> 0:01:23.156 a professor at Stanford and the co founder of a 0:01:23.196 --> 0:01:28.556 company called Physical Intelligence aka PI. Chelsea's problem is this, 0:01:29.276 --> 0:01:32.316 can you build an AI model that will bring AI 0:01:32.676 --> 0:01:35.876 to robots, or, as she puts it, we're. 0:01:35.676 --> 0:01:39.356 Trying to develop a model that can control any robot 0:01:39.436 --> 0:01:41.036 to do any task anywhere. 0:01:41.756 --> 0:01:44.916 Physical Intelligence was founded just last year, but the company 0:01:44.916 --> 0:01:49.396 has already raised over four hundred million dollars. Investors include 0:01:49.516 --> 0:01:53.556 Jeff Bezos and OpenAI. The company has raised so much 0:01:53.596 --> 0:01:55.836 money in part because what they're trying to do is 0:01:55.876 --> 0:01:59.916 so hard. Motor skills, the ability to move and find 0:01:59.956 --> 0:02:02.596 ways to fold the shirt to plug in a cable, 0:02:02.996 --> 0:02:07.116 they feel simple to us, easy, basic, But Chelsea told 0:02:07.156 --> 0:02:10.756 me basic motor skills are in fact wildly complex. 0:02:11.476 --> 0:02:14.556 All of the motor control that we do with our body, 0:02:14.596 --> 0:02:18.196 with their hands, with our legs, our feet, a lot 0:02:18.236 --> 0:02:20.716 of it we don't think about when we do it. 0:02:20.716 --> 0:02:23.836 It actually is incredibly complicated what we do. This is 0:02:23.836 --> 0:02:26.876 actually like a really really hard problem to develop in 0:02:26.996 --> 0:02:30.476 aisystems into robots, despite it being so simple. And the 0:02:30.516 --> 0:02:33.516 reasons for that are because actually it is inherently very complex, 0:02:34.116 --> 0:02:37.316 and second that we don't have tons and tons of 0:02:37.356 --> 0:02:40.876 data of doing this, in part because it's so basic 0:02:40.956 --> 0:02:42.756 to humans as well. 0:02:42.836 --> 0:02:45.556 Right, let's talk about the data side, because that seems 0:02:45.636 --> 0:02:49.396 like really the story, right, the big challenge, and it's 0:02:49.436 --> 0:02:54.596 particularly interesting in the context of large language models and 0:02:54.636 --> 0:02:58.956 computer vision which really seem to have emerged in a 0:02:58.996 --> 0:03:01.876 weird way as a consequence of the Internet. Right, just 0:03:01.916 --> 0:03:06.436 because we happen to have this crazy amount of data 0:03:06.596 --> 0:03:09.276 of words and pictures on the Internet, we were able 0:03:09.316 --> 0:03:12.476 to train language models and computer vision models. But we 0:03:12.556 --> 0:03:16.756 don't have that for robots, right. There is no data 0:03:16.796 --> 0:03:19.876 set of training data for robots, which is like the 0:03:19.956 --> 0:03:22.756 big challenge for you and for robotics in general. 0:03:22.796 --> 0:03:25.636 It seems, Yeah, so we don't have an open internet 0:03:25.636 --> 0:03:29.316 of how to control motors to do like even really 0:03:29.356 --> 0:03:31.556 basic things. Maybe the closest thing we have is we 0:03:31.596 --> 0:03:34.596 have videos of people doing things, and perhaps that could 0:03:34.596 --> 0:03:37.076 be useful. But at the same time, if I watch 0:03:37.196 --> 0:03:40.036 like videos of like Roger Federer or playing tennis, you 0:03:40.076 --> 0:03:42.956 can't just become an amazing tennis player as a result 0:03:42.956 --> 0:03:45.476 of that. And likewise, just with videos of people doing things, 0:03:45.876 --> 0:03:48.716 it's very hard to actually extract the motor control behind that. 0:03:48.876 --> 0:03:51.476 And so that lack of data, that scarcity of data, 0:03:51.876 --> 0:03:56.316 makes it in some ways a very different problem than 0:03:56.636 --> 0:03:58.956 in language and computer vision. And I think that we 0:03:58.956 --> 0:04:00.796 should still learn a lot of things from language computer 0:04:00.876 --> 0:04:04.196 vision and collect large data sets like that. It opens 0:04:04.276 --> 0:04:07.596 up new new challenges new possibilities on that front, and 0:04:07.676 --> 0:04:08.996 I think that in the long run we should be 0:04:09.236 --> 0:04:11.876 to get large amounts of data, just like how in 0:04:11.916 --> 0:04:14.356 autonomous driving we have lots of data of cars driving 0:04:14.396 --> 0:04:18.076 around very effectively. Robots too, could be in the world 0:04:18.196 --> 0:04:21.316 collecting data learning about how to pick up mustard and 0:04:21.356 --> 0:04:23.516 put it on a hot dog fund, or learning how 0:04:23.556 --> 0:04:26.556 to open a cabinet to put some objects away. We 0:04:26.556 --> 0:04:29.356 can get that sort of data, but it's not given 0:04:29.436 --> 0:04:33.196 to us for free. 0:04:33.436 --> 0:04:36.596 You still have this core problem, which is there is 0:04:36.916 --> 0:04:41.956 no giant trove of physical reality data that you can 0:04:41.996 --> 0:04:44.996 train your model on. Right, That's the great big challenge, 0:04:45.036 --> 0:04:46.796 it seems, what do you do about that? How do 0:04:46.796 --> 0:04:47.996 you start to approach that? 0:04:49.196 --> 0:04:52.676 Yeah, so we're starting off by collecting data through telling 0:04:52.716 --> 0:04:57.436 operation where you are people are controlling the robot to 0:04:57.436 --> 0:05:00.116 do tasks, and then you don't just get video data. 0:05:00.196 --> 0:05:03.196 You get the videos alongside what are the actions or 0:05:03.196 --> 0:05:07.076 the motor commands needed to actually accomplish those tasks. We've 0:05:07.116 --> 0:05:10.636 collected data in our own office. We've also collected data 0:05:10.876 --> 0:05:14.956 in homes across San Francisco, and we also have a 0:05:15.076 --> 0:05:18.476 very modest warehouse. In some ways, it actually like our 0:05:18.516 --> 0:05:22.476 current operation is rather small, given that we're a little 0:05:22.476 --> 0:05:24.076 over a year old at this point. 0:05:24.356 --> 0:05:26.556 Like what's actually happening? Like if I went into your 0:05:26.556 --> 0:05:28.996 warehouse and somebody was doing teleoperation, what would I see? 0:05:29.036 --> 0:05:29.836 What would it look like? 0:05:30.676 --> 0:05:35.076 Yeah, so we it's a little bit like controlling a puppet. 0:05:35.276 --> 0:05:38.956 So the person who's operating at the robot, they are 0:05:38.996 --> 0:05:42.196 holding in some ways a set of robot arms, but 0:05:42.196 --> 0:05:44.596 they're very lightweight robot arms, and we use those to 0:05:44.676 --> 0:05:46.676 measure the positions of joints. 0:05:47.076 --> 0:05:49.516 It's almost like an elaborate control for a video game 0:05:49.636 --> 0:05:52.716 or something. It's like that, it's not actually a robot arm, right, 0:05:52.716 --> 0:05:54.796 It's a thing you control to sort of play the 0:05:54.956 --> 0:05:57.196 robot to the robot move. 0:05:57.076 --> 0:06:00.516 Yeah, exactly exactly, and then we record that and then 0:06:01.036 --> 0:06:04.956 directly translate those controls over to the robot. We have 0:06:04.996 --> 0:06:07.516 some robots that are just robot arms, where you're only 0:06:07.516 --> 0:06:09.636 just controlling the robot arm. It's mounted to a table 0:06:09.756 --> 0:06:11.996 or something like that. But we also have what we 0:06:12.036 --> 0:06:14.636 call mobile manipulators that have wheels and robot arms, and 0:06:14.676 --> 0:06:18.036 you can control both how the robot drives around as 0:06:18.116 --> 0:06:21.236 well as how the arms move and we're doing tasks 0:06:21.356 --> 0:06:26.956 like wiping down counters, folding laundry, putting dishes into dishwashers, 0:06:27.276 --> 0:06:32.716 plugging cables into data center racks, assembling cardboard boxes, lots 0:06:32.756 --> 0:06:35.556 and lots of different tasks that might be useful for 0:06:35.676 --> 0:06:38.636 robots to do, and recording all the data. So we 0:06:38.676 --> 0:06:40.996 have cameras on the robots. There are sensors on the 0:06:41.036 --> 0:06:44.636 joints on the motors of the robots as well, and 0:06:44.676 --> 0:06:47.596 we record that in like a synchronized way across time. 0:06:47.836 --> 0:06:50.596 So when you do it, it's like kind of like 0:06:50.756 --> 0:06:52.716 a real world video game, like you're moving your arms 0:06:52.716 --> 0:06:55.676 in these things, and in basically real time, the robot 0:06:55.796 --> 0:06:58.036 arm is moving and picking up the thing you wanted 0:06:58.076 --> 0:07:01.156 to pick up, And like, what's it like? Is there 0:07:01.236 --> 0:07:03.556 like a curve where like at the beginning it's really bad? 0:07:03.636 --> 0:07:06.036 Sort of tell me talk me through an instance. 0:07:06.956 --> 0:07:08.796 And it depends on the person. So some people can 0:07:08.836 --> 0:07:11.276 pay it really really quickly. Some people are a bit 0:07:11.276 --> 0:07:13.756 slower to pick it up. I've pride myself in being 0:07:13.756 --> 0:07:17.756 a pretty good operator, and so I have done tasks 0:07:17.756 --> 0:07:20.476 as complex as peeling a hard boiled egg with the robot, 0:07:21.196 --> 0:07:22.476 which is how are. 0:07:22.316 --> 0:07:24.916 You how are you at peeling a hardboard hard boiled 0:07:24.916 --> 0:07:25.796 egg with your hands. 0:07:27.276 --> 0:07:29.796 It's pretty hard with my own hands too, yeah, and 0:07:29.836 --> 0:07:31.076 with the robot is even harder. 0:07:31.156 --> 0:07:32.996 Tell me about the robot peeling a hard build egg 0:07:33.036 --> 0:07:35.276 because that sounds like a hard one. Yeah. 0:07:35.316 --> 0:07:37.796 So the robots, basically, all the robots that we're using 0:07:37.836 --> 0:07:40.716 are like kind of pincher grippers. They're called parallel drag rippers, 0:07:41.036 --> 0:07:44.756 where there's just one degree random like open clothes two pincers. 0:07:44.756 --> 0:07:46.556 It's basically two pincers, like two. 0:07:46.396 --> 0:07:50.676 Pinters, two arms. Yeah, exactly, and I've used that exact setup. 0:07:50.996 --> 0:07:52.956 There's six different joints on the arm, so it can 0:07:53.396 --> 0:07:56.556 move as kind of full basically full range of motion 0:07:56.676 --> 0:07:59.236 in three D space and three D rotation, and you 0:07:59.236 --> 0:08:01.396 can use that to peel a hard boiled egg. You 0:08:01.436 --> 0:08:04.156 don't have any tactile feedback, so you can't actually feel 0:08:04.556 --> 0:08:05.996 the egg, and that's actually one of the things that 0:08:06.116 --> 0:08:08.876 makes it more difficult. But you can actually you can 0:08:08.996 --> 0:08:13.036 use visual feedback to compensate for that. And so just 0:08:13.036 --> 0:08:15.516 by looking at the egg myself, I'm able to figure 0:08:15.516 --> 0:08:18.076 out if you're like in contact with something, and you just. 0:08:18.156 --> 0:08:21.156 Use one prong of the claw like what I could say, 0:08:21.156 --> 0:08:23.236 you squeeze it a little to crack it, and then 0:08:23.676 --> 0:08:25.836 use like one prong of the claw to get the 0:08:25.836 --> 0:08:26.316 shell off. 0:08:26.996 --> 0:08:28.956 Yeah, exactly, so you can. You want to crack it 0:08:28.996 --> 0:08:31.116 initially and then hold it with one gripper and then 0:08:31.236 --> 0:08:34.716 use basically one of the two fingers in the gripper 0:08:35.036 --> 0:08:38.076 to get pieces of shell off. When we did this, 0:08:38.116 --> 0:08:41.836 we heart boiled only two eggs and the moss egg. 0:08:42.556 --> 0:08:44.516 This is actually a Stanford The first egg and graduate 0:08:44.556 --> 0:08:46.956 student ended up breaking and so that I did the 0:08:46.996 --> 0:08:49.156 second egg, and I was able to successfully not break 0:08:49.196 --> 0:08:52.156 it and fully peel it. It took some patience, certainly, 0:08:52.156 --> 0:08:53.956 and I wasn't able to do it as quickly as 0:08:53.956 --> 0:08:56.556 with my own hands, But I guess goes to show 0:08:56.636 --> 0:09:00.276 the extent to which we're able to control robots to 0:09:00.356 --> 0:09:02.116 do pretty complicated things. 0:09:02.356 --> 0:09:05.596 Yeah, and so obviously, I mean that is a stunt 0:09:05.676 --> 0:09:07.876 or a game or something fun to do with the robot. 0:09:07.916 --> 0:09:11.956 But presumably in that instance, as in the other instances 0:09:11.996 --> 0:09:16.556 of folding clothes and vacuuming it like, there is learning, right. 0:09:16.596 --> 0:09:19.076 The idea is that you do it some number of 0:09:19.116 --> 0:09:21.476 times and then the robot can do it, and then 0:09:21.516 --> 0:09:24.516 presumably there's also generalization. But just to start with learning, 0:09:24.796 --> 0:09:29.036 like you know, reductively, how many times do you got 0:09:29.036 --> 0:09:30.356 to do it for the robot to learn it? 0:09:31.676 --> 0:09:35.796 Yeah, so it really depends on the extent to which 0:09:35.836 --> 0:09:38.636 you want the robot to handle different conditions. So in 0:09:38.676 --> 0:09:40.996 some of our research, we've been able to show the 0:09:41.116 --> 0:09:44.596 robot how to do something like thirty times or fifty times, 0:09:44.716 --> 0:09:47.716 and just with that maybe sounds like a bit, but 0:09:47.716 --> 0:09:49.476 you can do that in like typically less than an 0:09:49.476 --> 0:09:52.276 hour if it's a simple task, and from that the 0:09:52.356 --> 0:09:56.036 robot can under the circumstances. You only kind of demonstrate it. 0:09:56.036 --> 0:09:59.036 In a narrow set of circumstances, like a single environment, 0:09:59.396 --> 0:10:02.956 a single particular object, the robot can learn just from 0:10:03.036 --> 0:10:05.076 like less than hour of data. 0:10:05.156 --> 0:10:07.156 What is an example of a thing that the robot 0:10:07.196 --> 0:10:08.556 learned in less than an er of data? 0:10:09.316 --> 0:10:12.556 Oh yeah, we put a shoe on a foot, We 0:10:12.876 --> 0:10:14.156 tear it off a piece of tape and put it 0:10:14.196 --> 0:10:18.516 on a box. We've also hung up a shirt on 0:10:18.596 --> 0:10:19.036 a hangar. 0:10:19.676 --> 0:10:22.276 So that's not that much I mean, especially because you 0:10:22.316 --> 0:10:24.676 say the robot, but what you really mean is the model. 0:10:24.796 --> 0:10:29.116 So every robot, right, presumably or every robot that's built 0:10:29.156 --> 0:10:30.916 more or less like that one, right, Like that's one 0:10:30.956 --> 0:10:33.236 of the key things. It's like you're not teaching one robot, 0:10:33.276 --> 0:10:37.276 you're teaching every robot ever, because it's it's software fundamentally, 0:10:37.276 --> 0:10:38.836 it's an am model. It's not hardware. 0:10:39.356 --> 0:10:42.236 Yeah, yes, with the caveat that, if you want to 0:10:42.236 --> 0:10:44.796 be this data efficient, it works best if it's like 0:10:45.156 --> 0:10:47.356 in the same like the same color of the table, 0:10:47.756 --> 0:10:50.156 the same kind of rough initial conditions of where the 0:10:50.156 --> 0:10:52.636 objects are starting, right, and the same shirt for example. 0:10:52.676 --> 0:10:54.436 So this is just with like a single shirt and 0:10:54.476 --> 0:10:55.276 not like any shirt. 0:10:55.436 --> 0:10:59.556 So there's there's like concentric circles of generalizability, right, like 0:10:59.676 --> 0:11:02.836 exact same shirt, exact same spot, exact same table versus 0:11:02.876 --> 0:11:06.876 like fold a shirt versus fold clothes, right and versus. 0:11:07.676 --> 0:11:12.116 And so is that just infinitely harder, Like how does 0:11:12.156 --> 0:11:14.396 that work? That's your big that's your big challenge at 0:11:14.396 --> 0:11:16.396 some level, right, Yeah. 0:11:16.236 --> 0:11:18.396 So generalization is one of the big one of the 0:11:18.396 --> 0:11:20.076 big challenges, not the only one, but it's one of 0:11:20.076 --> 0:11:23.636 the big challenges. And in some ways, I mean the 0:11:23.956 --> 0:11:25.916 first unlock there is just to make sure that you're 0:11:25.916 --> 0:11:28.316 collecting data not just for one shirt, but collecting it 0:11:28.316 --> 0:11:30.036 for lots of shirts, or collecting it for lots of 0:11:30.036 --> 0:11:33.316 clothing items, and ideally also collecting data with lots of 0:11:33.356 --> 0:11:37.356 tables with different textures, and also like not just visual 0:11:37.596 --> 0:11:40.596 like appearances, but also like if you're folding on a 0:11:40.636 --> 0:11:43.716 surface that has very low friction, like it's very smooth, 0:11:43.796 --> 0:11:46.236 versus a surface that like maybe on top of carpet 0:11:46.316 --> 0:11:49.436 or something that's going to behave differently when you're trying 0:11:49.476 --> 0:11:53.916 to move the shirt across the table. So having variability 0:11:53.996 --> 0:11:57.236 in the scenarios in which the robot is experiencing in 0:11:57.276 --> 0:12:02.076 the data set is important, and we've seen evidence that 0:12:02.596 --> 0:12:04.716 you set things up correctly and collect data under lots 0:12:04.756 --> 0:12:08.276 of scenarios, you can actually generalize to completely new scenarios. 0:12:08.316 --> 0:12:11.556 And in like Pile five release, for example, we found 0:12:11.596 --> 0:12:15.356 that if we collected data in roughly like one hundred 0:12:15.396 --> 0:12:20.436 different rooms, then the robot is able to do some 0:12:20.636 --> 0:12:22.756 tasks in rooms that it's never been in before. 0:12:23.116 --> 0:12:26.516 So you mentioned Pile five, So PI zero point five 0:12:26.556 --> 0:12:31.716 that's your latest model that you've released, right, tell me 0:12:31.756 --> 0:12:35.676 about that, Like, what what does that model allow robots 0:12:35.716 --> 0:12:38.956 to do? Like what robots and what settings and what tasks. 0:12:39.436 --> 0:12:43.116 Yeah, yeah, definitely. So we were focusing on generalization. So 0:12:43.316 --> 0:12:46.196 the previous model, we were focusing on capability, and we 0:12:46.236 --> 0:12:49.756 did a really complicated task of laundry folding. From there, 0:12:49.796 --> 0:12:52.556 we wanted to answer, like, Okay, that model worked in 0:12:52.556 --> 0:12:54.596 one environment. It's fairly brittle. If you put it in 0:12:54.596 --> 0:12:56.556 a new environment, it wouldn't work. And we wanted to 0:12:56.556 --> 0:12:59.476 see if we put robots in new environments with new objects, 0:12:59.476 --> 0:13:03.476 new lighting conditions, new furniture, can the robot be successful. 0:13:03.636 --> 0:13:09.956 And to do that, we collected data on these manipulators, 0:13:10.076 --> 0:13:13.636 which feels like a terrible name, but robots with two 0:13:13.716 --> 0:13:16.036 arms and wheels that can drive around kind of like 0:13:16.036 --> 0:13:18.956 a humanoid, but we're using wheels instead of legs, a 0:13:18.956 --> 0:13:22.716 bit more practical in that regard, and we train the 0:13:22.796 --> 0:13:26.396 robot to do things like tidying a bed, or wiping 0:13:26.476 --> 0:13:29.556 spills off of a surface, or putting dishes into a sink, 0:13:29.676 --> 0:13:34.156 or putting away items into drawers, taking items of clothing, 0:13:34.156 --> 0:13:36.236 dirty clothing off the floor and putting them into a 0:13:36.276 --> 0:13:39.836 laundry basket, things like that, And then we tested whether 0:13:39.916 --> 0:13:42.036 or not after collecting data like that and lots of 0:13:42.116 --> 0:13:45.676 environments aggregated with other data, including data on the internet. 0:13:46.156 --> 0:13:49.876 Can the robot then do those things in a home 0:13:49.916 --> 0:13:53.076 that has never been in before. And in some ways 0:13:53.076 --> 0:13:57.916 that sounds kind of basic, like people have no problem 0:13:58.316 --> 0:14:01.236 with if you can do it something in like one home, 0:14:01.356 --> 0:14:03.236 probably could do the same thing in another home. It's 0:14:03.276 --> 0:14:05.796 not really doesn't seem like a complicated thing for humans, 0:14:05.956 --> 0:14:08.316 but for robots that are trained on data, if they're 0:14:08.316 --> 0:14:11.116 only trained on in one place there are whole universe, 0:14:11.196 --> 0:14:13.476 is that one place they haven't ever seen any other place? 0:14:13.836 --> 0:14:17.276 This is actually kind of a big challenge for existing methods. 0:14:17.276 --> 0:14:18.956 And yeah, it was a step four. We were able 0:14:18.996 --> 0:14:21.676 to see that it definitely isn't perfect by any means, 0:14:21.716 --> 0:14:25.916 and that kind of comes to another challenge, which is reliability. 0:14:26.036 --> 0:14:29.036 But we're able to see the robot do things in 0:14:29.076 --> 0:14:31.236 homes it's never been in before, where we set it up, 0:14:31.356 --> 0:14:33.156 ask it to do things, and it does some things 0:14:33.196 --> 0:14:33.756 that are useful. 0:14:33.876 --> 0:14:36.476 So like in the classical setting where a robot is 0:14:36.556 --> 0:14:38.356 changed in one room, like it doesn't even know that 0:14:38.436 --> 0:14:40.996 room is a room. That's just like the whole world 0:14:41.036 --> 0:14:43.196 to the robot, is that world right? And if you 0:14:43.236 --> 0:14:46.996 put it in another room, it's in a completely unfamiliar 0:14:47.036 --> 0:14:48.236 world exactly. 0:14:48.316 --> 0:14:50.316 And so for example, what we were talking about, like 0:14:50.556 --> 0:14:52.996 hanging up a shirt, its whole world was like that one, 0:14:53.156 --> 0:14:57.036 like like a black tabletop that smooth, that one blue shirt, 0:14:57.156 --> 0:14:59.436 that one coat hanger. And it doesn't know about this 0:14:59.916 --> 0:15:01.676 entire universe of other shirts and other. 0:15:01.716 --> 0:15:03.956 It doesn't know that there is a category called shirt. 0:15:04.156 --> 0:15:04.676 It only knows. 0:15:04.756 --> 0:15:05.876 Yeah, it doesn't even know what shirts are. 0:15:06.036 --> 0:15:08.356 Yeah, it doesn't even know what shirts are. For pie 0:15:08.436 --> 0:15:10.556 zero point five, Like, what did you ask the robot 0:15:10.596 --> 0:15:12.196 to do? And how well did it work? 0:15:13.316 --> 0:15:16.596 Yeah, So we trained the model. We took actually a 0:15:16.596 --> 0:15:19.956 pre trading language model with also like a vision component, 0:15:20.476 --> 0:15:23.156 and we fine tuned it on a lot of data, 0:15:23.196 --> 0:15:26.676 including data from different homes across San Francisco, but actually 0:15:26.676 --> 0:15:28.276 a lot of other data too. So actually only two 0:15:28.316 --> 0:15:31.796 percent of the data was on these like mobile robots 0:15:31.956 --> 0:15:35.196 with arms. So we can store how the motors were 0:15:35.196 --> 0:15:38.036 all moving in all of our previous data and then 0:15:38.356 --> 0:15:40.716 train the model to mimic that data that we've stored. 0:15:40.836 --> 0:15:43.476 It's like it's like predicting the next word, but instead 0:15:43.476 --> 0:15:45.716 of predicting the next word, it's like predicting the next movement. 0:15:45.876 --> 0:15:47.236 Or something like yes, exactly. 0:15:48.716 --> 0:15:50.956 We've kind of trained it to predict next actions or 0:15:51.036 --> 0:15:54.276 next motor commands instead of next words. We do an 0:15:54.316 --> 0:15:57.476 additional training process to have it focus on and be 0:15:57.596 --> 0:16:01.036 good at the mobile robot data and homes. Then we 0:16:01.036 --> 0:16:03.396 set up the robot in a new home and we 0:16:03.476 --> 0:16:06.516 give it language commands, so we can give it low 0:16:06.596 --> 0:16:09.516 level language commands, or we can actually all so give 0:16:09.516 --> 0:16:12.596 it higher level commands. So the highest level of command 0:16:12.676 --> 0:16:14.916 might be cleaned the bedroom. And one of the things 0:16:14.916 --> 0:16:16.556 that we've also been thinking about more recently is can 0:16:16.556 --> 0:16:18.916 you give it a more detailed description of how you 0:16:18.956 --> 0:16:20.916 want it to clean the bedroom? But we're not quite 0:16:20.916 --> 0:16:22.756 there yet, So we could say clean the bedroom. We'd 0:16:22.796 --> 0:16:25.316 also tell it put the dirty clothes in the laundry basket, 0:16:26.236 --> 0:16:29.476 so that would be kind of a subtask. Or we 0:16:29.516 --> 0:16:32.116 can tell it like commands like pick up the shirt, 0:16:32.556 --> 0:16:35.396 put the shirt in the laundry basket. Then after we 0:16:35.476 --> 0:16:39.996 tell it that command, then it will go off and 0:16:40.756 --> 0:16:44.476 follow that command and actually in most cases realize that 0:16:44.516 --> 0:16:46.636 command successfully in the real world. 0:16:47.156 --> 0:16:47.676 How did it do. 0:16:48.476 --> 0:16:51.556 So it depends on the task. The average success rate 0:16:51.596 --> 0:16:55.476 was around eighty percent, so definitely room for improvement, and 0:16:56.036 --> 0:16:58.436 in many snares it was able to be quite successful. 0:16:58.556 --> 0:17:01.756 We also saw some some failure modes where for example, 0:17:01.796 --> 0:17:04.956 if you're trying to put dishes into a sink, sometimes 0:17:05.076 --> 0:17:06.956 one of the dishes was a cutting board, and picking 0:17:06.996 --> 0:17:09.036 up a cutting board is actually pretty tricky for the 0:17:09.196 --> 0:17:11.516 robot because you either need to slide it to the 0:17:11.676 --> 0:17:14.236 edge of the counter and then grasp it or somehow 0:17:14.276 --> 0:17:17.916 get the kind of get the finger underneath the cutting board. 0:17:18.276 --> 0:17:20.396 And so sometimes it was able to do that successfully. 0:17:20.396 --> 0:17:24.076 Sometimes it struggled and got stuck. The exciting thing though, 0:17:24.116 --> 0:17:26.436 was that it was able to We were able to 0:17:26.476 --> 0:17:27.916 kind of drop it in place as it had never 0:17:27.956 --> 0:17:31.276 been before. And I was doing things that are quite reasonable. 0:17:32.036 --> 0:17:33.836 So what are you doing now, Like, what's the next 0:17:33.876 --> 0:17:35.996 thing you're trying to get to? Yeah? 0:17:35.996 --> 0:17:39.796 Absolutely, So the next thing we're focusing on is reliability 0:17:40.116 --> 0:17:44.036 and speed. So I mentioned like around eighty percent for 0:17:44.076 --> 0:17:46.956 these tasks. How do we get that to ninety nine percent? 0:17:47.116 --> 0:17:49.716 And I think that if we can get the reliability up, 0:17:49.916 --> 0:17:54.236 that's kind of, in my mind, the main missing ingredient 0:17:54.476 --> 0:17:57.596 before we can like really have these being like useful 0:17:58.236 --> 0:18:00.116 in real world scenarios. 0:18:00.716 --> 0:18:03.316 So getting to ninety nine percent is interesting. I mean, 0:18:03.396 --> 0:18:08.036 I think of self driving cars right where it seemed 0:18:08.516 --> 0:18:11.236 sometime go I don't know, ten years ago, fifteen years ago, 0:18:11.316 --> 0:18:14.116 like they were almost there, and I know they're more 0:18:14.156 --> 0:18:16.356 almost there now. I know in San Francisco there really 0:18:16.436 --> 0:18:18.676 are self driving cars, but they're still very much at 0:18:18.716 --> 0:18:22.036 the margin of cars in the world, right, And it 0:18:22.076 --> 0:18:26.236 does seem like almost there means different things in different settings, 0:18:26.276 --> 0:18:31.716 But I don't know. Is it super hard to get 0:18:31.716 --> 0:18:33.996 from eighty percent to ninety nine percent? Does the self 0:18:34.076 --> 0:18:38.716 driving car example teach us anything for your work? 0:18:39.796 --> 0:18:42.756 The self driving car analogy is pretty good. I do 0:18:42.836 --> 0:18:47.156 think that fortunately, we may not need There are scenarios 0:18:47.156 --> 0:18:48.916 where we may not need it to be quite as 0:18:48.956 --> 0:18:52.676 reliable as cars. Cars there is a much much higher 0:18:52.876 --> 0:18:56.956 safety risk. It's much easier to hurt people, and in 0:18:57.076 --> 0:18:59.036 robots there are safety risks because you are in the 0:18:59.036 --> 0:19:03.356 physical world. But it's easier to put in software precautions 0:19:03.396 --> 0:19:06.116 in place and even hardware precautions in place to prevent 0:19:06.156 --> 0:19:08.396 that as well, So that makes it a little bit easier. 0:19:08.396 --> 0:19:11.796 I mean, nine percent probably isn't good enough for cars, right, 0:19:11.796 --> 0:19:14.596 They probably need more nines than that, whereas it may 0:19:14.596 --> 0:19:16.356 well be good enough for a house. 0:19:16.156 --> 0:19:19.916 Cleaning robots, yeah, in certain circumstances. And yeah, like we're 0:19:19.916 --> 0:19:22.316 also thinking about scenarios where maybe even less than that 0:19:22.396 --> 0:19:26.076 is fine. And if we view humans and robots working together, 0:19:26.396 --> 0:19:29.436 it's more about kind of helping the person complete the 0:19:29.436 --> 0:19:33.436 task faster or complete the task more effectively. So I 0:19:33.436 --> 0:19:35.956 think there might be scenarios like that, but still we 0:19:35.996 --> 0:19:39.076 need the performance and reliability to be higher for the 0:19:39.156 --> 0:19:41.476 robots to be faster in order to accomplish that. 0:19:44.676 --> 0:19:59.156 We'll be back in just a minute. What do you 0:19:59.196 --> 0:20:02.436 imagine as the initial real world use cases? 0:20:05.076 --> 0:20:07.236 I don't know. There's a lot of examples of robotics 0:20:07.236 --> 0:20:11.196 companies that have a tempted to kind of start with 0:20:11.236 --> 0:20:16.156 an application and hone in on that, and I think 0:20:16.196 --> 0:20:20.156 the lesson from watching those companies is that you end 0:20:20.236 --> 0:20:23.596 up then spending a lot of time on the problems 0:20:23.596 --> 0:20:26.956 of that specific application and less on developing the sort 0:20:26.996 --> 0:20:28.796 of generalist systems that we think in the long run 0:20:28.836 --> 0:20:31.596 will be more effective. And so we're very focused on 0:20:32.276 --> 0:20:36.036 understanding what are the core bottlenecks and the core missing 0:20:36.076 --> 0:20:38.876 pieces for developing these generalist models, and we think that 0:20:38.916 --> 0:20:41.356 if we had picked an application now, we would kind 0:20:41.356 --> 0:20:43.156 of lose sight of that bigger problem because we need 0:20:43.156 --> 0:20:45.916 to solve things that are specific to that application. So 0:20:46.076 --> 0:20:48.636 we're very focused on what we think are like the 0:20:48.636 --> 0:20:53.876 core technological challenges. We have certain tasks that we're working on. 0:20:53.916 --> 0:20:56.556 Some of them have been home cleaning tasks. We've also 0:20:56.636 --> 0:20:59.716