Speaker 1: Pushkin. In a metaphorical sense, AI is everywhere. It can
Speaker 1: write essays, it can do your texes, it can design drugs,
Speaker 1: it can make movies. But in a literal sense, AI
Speaker 1: is not everywhere. You know, a large language model can
Speaker 1: tell you whatever twenty seven ways to fold your shirts
Speaker 1: and put them in the drawer, but there's no robot
Speaker 1: that you can buy that can actually fold your shirts
Speaker 1: and put them in the drawer. At some point, though
Speaker 1: maybe at some point in the not that distant future,
Speaker 1: there will be a robot that can use AI to
Speaker 1: learn how to fold your shirts and put them in
Speaker 1: the drawer, or you know, cook lasagna, pack boxes, plug
Speaker 1: in cables. In other words, there will be a robot
Speaker 1: that can use AI to learn how to do basically anything.
Speaker 1: I'm Jacob Goldstein and this is What's Your Problem, the
Speaker 1: show where I talk to people who are trying to
Speaker 1: make technological progress. My guest today is Chelsea Finn. She's
Speaker 1: a professor at Stanford and the co founder of a
Speaker 1: company called Physical Intelligence aka PI. Chelsea's problem is this,
Speaker 1: can you build an AI model that will bring AI
Speaker 1: to robots, or, as she puts it, we're.
Speaker 2: Trying to develop a model that can control any robot
Speaker 2: to do any task anywhere.
Speaker 1: Physical Intelligence was founded just last year, but the company
Speaker 1: has already raised over four hundred million dollars. Investors include
Speaker 1: Jeff Bezos and OpenAI. The company has raised so much
Speaker 1: money in part because what they're trying to do is
Speaker 1: so hard. Motor skills, the ability to move and find
Speaker 1: ways to fold the shirt to plug in a cable,
Speaker 1: they feel simple to us, easy, basic, But Chelsea told
Speaker 1: me basic motor skills are in fact wildly complex.
Speaker 2: All of the motor control that we do with our body,
Speaker 2: with their hands, with our legs, our feet, a lot
Speaker 2: of it we don't think about when we do it.
Speaker 2: It actually is incredibly complicated what we do. This is
Speaker 2: actually like a really really hard problem to develop in
Speaker 2: aisystems into robots, despite it being so simple. And the
Speaker 2: reasons for that are because actually it is inherently very complex,
Speaker 2: and second that we don't have tons and tons of
Speaker 2: data of doing this, in part because it's so basic
Speaker 2: to humans as well.
Speaker 1: Right, let's talk about the data side, because that seems
Speaker 1: like really the story, right, the big challenge, and it's
Speaker 1: particularly interesting in the context of large language models and
Speaker 1: computer vision which really seem to have emerged in a
Speaker 1: weird way as a consequence of the Internet. Right, just
Speaker 1: because we happen to have this crazy amount of data
Speaker 1: of words and pictures on the Internet, we were able
Speaker 1: to train language models and computer vision models. But we
Speaker 1: don't have that for robots, right. There is no data
Speaker 1: set of training data for robots, which is like the
Speaker 1: big challenge for you and for robotics in general.
Speaker 2: It seems, Yeah, so we don't have an open internet
Speaker 2: of how to control motors to do like even really
Speaker 2: basic things. Maybe the closest thing we have is we
Speaker 2: have videos of people doing things, and perhaps that could
Speaker 2: be useful. But at the same time, if I watch
Speaker 2: like videos of like Roger Federer or playing tennis, you
Speaker 2: can't just become an amazing tennis player as a result
Speaker 2: of that. And likewise, just with videos of people doing things,
Speaker 2: it's very hard to actually extract the motor control behind that.
Speaker 2: And so that lack of data, that scarcity of data,
Speaker 2: makes it in some ways a very different problem than
Speaker 2: in language and computer vision. And I think that we
Speaker 2: should still learn a lot of things from language computer
Speaker 2: vision and collect large data sets like that. It opens
Speaker 2: up new new challenges new possibilities on that front, and
Speaker 2: I think that in the long run we should be
Speaker 2: to get large amounts of data, just like how in
Speaker 2: autonomous driving we have lots of data of cars driving
Speaker 2: around very effectively. Robots too, could be in the world
Speaker 2: collecting data learning about how to pick up mustard and
Speaker 2: put it on a hot dog fund, or learning how
Speaker 2: to open a cabinet to put some objects away. We
Speaker 2: can get that sort of data, but it's not given
Speaker 2: to us for free.
Speaker 1: You still have this core problem, which is there is
Speaker 1: no giant trove of physical reality data that you can
Speaker 1: train your model on. Right, That's the great big challenge,
Speaker 1: it seems, what do you do about that? How do
Speaker 1: you start to approach that?
Speaker 2: Yeah, so we're starting off by collecting data through telling
Speaker 2: operation where you are people are controlling the robot to
Speaker 2: do tasks, and then you don't just get video data.
Speaker 2: You get the videos alongside what are the actions or
Speaker 2: the motor commands needed to actually accomplish those tasks. We've
Speaker 2: collected data in our own office. We've also collected data
Speaker 2: in homes across San Francisco, and we also have a
Speaker 2: very modest warehouse. In some ways, it actually like our
Speaker 2: current operation is rather small, given that we're a little
Speaker 2: over a year old at this point.
Speaker 1: Like what's actually happening? Like if I went into your
Speaker 1: warehouse and somebody was doing teleoperation, what would I see?
Speaker 1: What would it look like?
Speaker 2: Yeah, so we it's a little bit like controlling a puppet.
Speaker 2: So the person who's operating at the robot, they are
Speaker 2: holding in some ways a set of robot arms, but
Speaker 2: they're very lightweight robot arms, and we use those to
Speaker 2: measure the positions of joints.
Speaker 1: It's almost like an elaborate control for a video game
Speaker 1: or something. It's like that, it's not actually a robot arm, right,
Speaker 1: It's a thing you control to sort of play the
Speaker 1: robot to the robot move.
Speaker 2: Yeah, exactly exactly, and then we record that and then
Speaker 2: directly translate those controls over to the robot. We have
Speaker 2: some robots that are just robot arms, where you're only
Speaker 2: just controlling the robot arm. It's mounted to a table
Speaker 2: or something like that. But we also have what we
Speaker 2: call mobile manipulators that have wheels and robot arms, and
Speaker 2: you can control both how the robot drives around as
Speaker 2: well as how the arms move and we're doing tasks
Speaker 2: like wiping down counters, folding laundry, putting dishes into dishwashers,
Speaker 2: plugging cables into data center racks, assembling cardboard boxes, lots
Speaker 2: and lots of different tasks that might be useful for
Speaker 2: robots to do, and recording all the data. So we
Speaker 2: have cameras on the robots. There are sensors on the
Speaker 2: joints on the motors of the robots as well, and
Speaker 2: we record that in like a synchronized way across time.
Speaker 1: So when you do it, it's like kind of like
Speaker 1: a real world video game, like you're moving your arms
Speaker 1: in these things, and in basically real time, the robot
Speaker 1: arm is moving and picking up the thing you wanted
Speaker 1: to pick up, And like, what's it like? Is there
Speaker 1: like a curve where like at the beginning it's really bad?
Speaker 1: Sort of tell me talk me through an instance.
Speaker 2: And it depends on the person. So some people can
Speaker 2: pay it really really quickly. Some people are a bit
Speaker 2: slower to pick it up. I've pride myself in being
Speaker 2: a pretty good operator, and so I have done tasks
Speaker 2: as complex as peeling a hard boiled egg with the robot,
Speaker 2: which is how are.
Speaker 1: You how are you at peeling a hardboard hard boiled
Speaker 1: egg with your hands.
Speaker 2: It's pretty hard with my own hands too, yeah, and
Speaker 2: with the robot is even harder.
Speaker 1: Tell me about the robot peeling a hard build egg
Speaker 1: because that sounds like a hard one. Yeah.
Speaker 2: So the robots, basically, all the robots that we're using
Speaker 2: are like kind of pincher grippers. They're called parallel drag rippers,
Speaker 2: where there's just one degree random like open clothes two pincers.
Speaker 1: It's basically two pincers, like two.
Speaker 2: Pinters, two arms. Yeah, exactly, and I've used that exact setup.
Speaker 2: There's six different joints on the arm, so it can
Speaker 2: move as kind of full basically full range of motion
Speaker 2: in three D space and three D rotation, and you
Speaker 2: can use that to peel a hard boiled egg. You
Speaker 2: don't have any tactile feedback, so you can't actually feel
Speaker 2: the egg, and that's actually one of the things that
Speaker 2: makes it more difficult. But you can actually you can
Speaker 2: use visual feedback to compensate for that. And so just
Speaker 2: by looking at the egg myself, I'm able to figure
Speaker 2: out if you're like in contact with something, and you just.
Speaker 1: Use one prong of the claw like what I could say,
Speaker 1: you squeeze it a little to crack it, and then
Speaker 1: use like one prong of the claw to get the
Speaker 1: shell off.
Speaker 2: Yeah, exactly, so you can. You want to crack it
Speaker 2: initially and then hold it with one gripper and then
Speaker 2: use basically one of the two fingers in the gripper
Speaker 2: to get pieces of shell off. When we did this,
Speaker 2: we heart boiled only two eggs and the moss egg.
Speaker 2: This is actually a Stanford The first egg and graduate
Speaker 2: student ended up breaking and so that I did the
Speaker 2: second egg, and I was able to successfully not break
Speaker 2: it and fully peel it. It took some patience, certainly,
Speaker 2: and I wasn't able to do it as quickly as
Speaker 2: with my own hands, But I guess goes to show
Speaker 2: the extent to which we're able to control robots to
Speaker 2: do pretty complicated things.
Speaker 1: Yeah, and so obviously, I mean that is a stunt
Speaker 1: or a game or something fun to do with the robot.
Speaker 1: But presumably in that instance, as in the other instances
Speaker 1: of folding clothes and vacuuming it like, there is learning, right.
Speaker 1: The idea is that you do it some number of
Speaker 1: times and then the robot can do it, and then
Speaker 1: presumably there's also generalization. But just to start with learning,
Speaker 1: like you know, reductively, how many times do you got
Speaker 1: to do it for the robot to learn it?
Speaker 2: Yeah, so it really depends on the extent to which
Speaker 2: you want the robot to handle different conditions. So in
Speaker 2: some of our research, we've been able to show the
Speaker 2: robot how to do something like thirty times or fifty times,
Speaker 2: and just with that maybe sounds like a bit, but
Speaker 2: you can do that in like typically less than an
Speaker 2: hour if it's a simple task, and from that the
Speaker 2: robot can under the circumstances. You only kind of demonstrate it.
Speaker 2: In a narrow set of circumstances, like a single environment,
Speaker 2: a single particular object, the robot can learn just from
Speaker 2: like less than hour of data.
Speaker 1: What is an example of a thing that the robot
Speaker 1: learned in less than an er of data?
Speaker 2: Oh yeah, we put a shoe on a foot, We
Speaker 2: tear it off a piece of tape and put it
Speaker 2: on a box. We've also hung up a shirt on
Speaker 2: a hangar.
Speaker 1: So that's not that much I mean, especially because you
Speaker 1: say the robot, but what you really mean is the model.
Speaker 1: So every robot, right, presumably or every robot that's built
Speaker 1: more or less like that one, right, Like that's one
Speaker 1: of the key things. It's like you're not teaching one robot,
Speaker 1: you're teaching every robot ever, because it's it's software fundamentally,
Speaker 1: it's an am model. It's not hardware.
Speaker 2: Yeah, yes, with the caveat that, if you want to
Speaker 2: be this data efficient, it works best if it's like
Speaker 2: in the same like the same color of the table,
Speaker 2: the same kind of rough initial conditions of where the
Speaker 2: objects are starting, right, and the same shirt for example.
Speaker 2: So this is just with like a single shirt and
Speaker 2: not like any shirt.
Speaker 1: So there's there's like concentric circles of generalizability, right, like
Speaker 1: exact same shirt, exact same spot, exact same table versus
Speaker 1: like fold a shirt versus fold clothes, right and versus.
Speaker 1: And so is that just infinitely harder, Like how does
Speaker 1: that work? That's your big that's your big challenge at
Speaker 1: some level, right, Yeah.
Speaker 2: So generalization is one of the big one of the
Speaker 2: big challenges, not the only one, but it's one of
Speaker 2: the big challenges. And in some ways, I mean the
Speaker 2: first unlock there is just to make sure that you're
Speaker 2: collecting data not just for one shirt, but collecting it
Speaker 2: for lots of shirts, or collecting it for lots of
Speaker 2: clothing items, and ideally also collecting data with lots of
Speaker 2: tables with different textures, and also like not just visual
Speaker 2: like appearances, but also like if you're folding on a
Speaker 2: surface that has very low friction, like it's very smooth,
Speaker 2: versus a surface that like maybe on top of carpet
Speaker 2: or something that's going to behave differently when you're trying
Speaker 2: to move the shirt across the table. So having variability
Speaker 2: in the scenarios in which the robot is experiencing in
Speaker 2: the data set is important, and we've seen evidence that
Speaker 2: you set things up correctly and collect data under lots
Speaker 2: of scenarios, you can actually generalize to completely new scenarios.
Speaker 2: And in like Pile five release, for example, we found
Speaker 2: that if we collected data in roughly like one hundred
Speaker 2: different rooms, then the robot is able to do some
Speaker 2: tasks in rooms that it's never been in before.
Speaker 1: So you mentioned Pile five, So PI zero point five
Speaker 1: that's your latest model that you've released, right, tell me
Speaker 1: about that, Like, what what does that model allow robots
Speaker 1: to do? Like what robots and what settings and what tasks.
Speaker 2: Yeah, yeah, definitely. So we were focusing on generalization. So
Speaker 2: the previous model, we were focusing on capability, and we
Speaker 2: did a really complicated task of laundry folding. From there,
Speaker 2: we wanted to answer, like, Okay, that model worked in
Speaker 2: one environment. It's fairly brittle. If you put it in
Speaker 2: a new environment, it wouldn't work. And we wanted to
Speaker 2: see if we put robots in new environments with new objects,
Speaker 2: new lighting conditions, new furniture, can the robot be successful.
Speaker 2: And to do that, we collected data on these manipulators,
Speaker 2: which feels like a terrible name, but robots with two
Speaker 2: arms and wheels that can drive around kind of like
Speaker 2: a humanoid, but we're using wheels instead of legs, a
Speaker 2: bit more practical in that regard, and we train the
Speaker 2: robot to do things like tidying a bed, or wiping
Speaker 2: spills off of a surface, or putting dishes into a sink,
Speaker 2: or putting away items into drawers, taking items of clothing,
Speaker 2: dirty clothing off the floor and putting them into a
Speaker 2: laundry basket, things like that, And then we tested whether
Speaker 2: or not after collecting data like that and lots of
Speaker 2: environments aggregated with other data, including data on the internet.
Speaker 2: Can the robot then do those things in a home
Speaker 2: that has never been in before. And in some ways
Speaker 2: that sounds kind of basic, like people have no problem
Speaker 2: with if you can do it something in like one home,
Speaker 2: probably could do the same thing in another home. It's
Speaker 2: not really doesn't seem like a complicated thing for humans,
Speaker 2: but for robots that are trained on data, if they're
Speaker 2: only trained on in one place there are whole universe,
Speaker 2: is that one place they haven't ever seen any other place?
Speaker 2: This is actually kind of a big challenge for existing methods.
Speaker 2: And yeah, it was a step four. We were able
Speaker 2: to see that it definitely isn't perfect by any means,
Speaker 2: and that kind of comes to another challenge, which is reliability.
Speaker 2: But we're able to see the robot do things in
Speaker 2: homes it's never been in before, where we set it up,
Speaker 2: ask it to do things, and it does some things
Speaker 2: that are useful.
Speaker 1: So like in the classical setting where a robot is
Speaker 1: changed in one room, like it doesn't even know that
Speaker 1: room is a room. That's just like the whole world
Speaker 1: to the robot, is that world right? And if you
Speaker 1: put it in another room, it's in a completely unfamiliar
Speaker 1: world exactly.
Speaker 2: And so for example, what we were talking about, like
Speaker 2: hanging up a shirt, its whole world was like that one,
Speaker 2: like like a black tabletop that smooth, that one blue shirt,
Speaker 2: that one coat hanger. And it doesn't know about this
Speaker 2: entire universe of other shirts and other.
Speaker 1: It doesn't know that there is a category called shirt.
Speaker 1: It only knows.
Speaker 2: Yeah, it doesn't even know what shirts are.
Speaker 1: Yeah, it doesn't even know what shirts are. For pie
Speaker 1: zero point five, Like, what did you ask the robot
Speaker 1: to do? And how well did it work?
Speaker 2: Yeah, So we trained the model. We took actually a
Speaker 2: pre trading language model with also like a vision component,
Speaker 2: and we fine tuned it on a lot of data,
Speaker 2: including data from different homes across San Francisco, but actually
Speaker 2: a lot of other data too. So actually only two
Speaker 2: percent of the data was on these like mobile robots
Speaker 2: with arms. So we can store how the motors were
Speaker 2: all moving in all of our previous data and then
Speaker 2: train the model to mimic that data that we've stored.
Speaker 1: It's like it's like predicting the next word, but instead
Speaker 1: of predicting the next word, it's like predicting the next movement.
Speaker 1: Or something like yes, exactly.
Speaker 2: We've kind of trained it to predict next actions or
Speaker 2: next motor commands instead of next words. We do an
Speaker 2: additional training process to have it focus on and be
Speaker 2: good at the mobile robot data and homes. Then we
Speaker 2: set up the robot in a new home and we
Speaker 2: give it language commands, so we can give it low
Speaker 2: level language commands, or we can actually all so give
Speaker 2: it higher level commands. So the highest level of command
Speaker 2: might be cleaned the bedroom. And one of the things
Speaker 2: that we've also been thinking about more recently is can
Speaker 2: you give it a more detailed description of how you
Speaker 2: want it to clean the bedroom? But we're not quite
Speaker 2: there yet, So we could say clean the bedroom. We'd
Speaker 2: also tell it put the dirty clothes in the laundry basket,
Speaker 2: so that would be kind of a subtask. Or we
Speaker 2: can tell it like commands like pick up the shirt,
Speaker 2: put the shirt in the laundry basket. Then after we
Speaker 2: tell it that command, then it will go off and
Speaker 2: follow that command and actually in most cases realize that
Speaker 2: command successfully in the real world.
Speaker 1: How did it do.
Speaker 2: So it depends on the task. The average success rate
Speaker 2: was around eighty percent, so definitely room for improvement, and
Speaker 2: in many snares it was able to be quite successful.
Speaker 2: We also saw some some failure modes where for example,
Speaker 2: if you're trying to put dishes into a sink, sometimes
Speaker 2: one of the dishes was a cutting board, and picking
Speaker 2: up a cutting board is actually pretty tricky for the
Speaker 2: robot because you either need to slide it to the
Speaker 2: edge of the counter and then grasp it or somehow
Speaker 2: get the kind of get the finger underneath the cutting board.
Speaker 2: And so sometimes it was able to do that successfully.
Speaker 2: Sometimes it struggled and got stuck. The exciting thing though,
Speaker 2: was that it was able to We were able to
Speaker 2: kind of drop it in place as it had never
Speaker 2: been before. And I was doing things that are quite reasonable.
Speaker 1: So what are you doing now, Like, what's the next
Speaker 1: thing you're trying to get to? Yeah?
Speaker 2: Absolutely, So the next thing we're focusing on is reliability
Speaker 2: and speed. So I mentioned like around eighty percent for
Speaker 2: these tasks. How do we get that to ninety nine percent?
Speaker 2: And I think that if we can get the reliability up,
Speaker 2: that's kind of, in my mind, the main missing ingredient
Speaker 2: before we can like really have these being like useful
Speaker 2: in real world scenarios.
Speaker 1: So getting to ninety nine percent is interesting. I mean,
Speaker 1: I think of self driving cars right where it seemed
Speaker 1: sometime go I don't know, ten years ago, fifteen years ago,
Speaker 1: like they were almost there, and I know they're more
Speaker 1: almost there now. I know in San Francisco there really
Speaker 1: are self driving cars, but they're still very much at
Speaker 1: the margin of cars in the world, right, And it
Speaker 1: does seem like almost there means different things in different settings,
Speaker 1: But I don't know. Is it super hard to get
Speaker 1: from eighty percent to ninety nine percent? Does the self
Speaker 1: driving car example teach us anything for your work?
Speaker 2: The self driving car analogy is pretty good. I do
Speaker 2: think that fortunately, we may not need There are scenarios
Speaker 2: where we may not need it to be quite as
Speaker 2: reliable as cars. Cars there is a much much higher
Speaker 2: safety risk. It's much easier to hurt people, and in
Speaker 2: robots there are safety risks because you are in the
Speaker 2: physical world. But it's easier to put in software precautions
Speaker 2: in place and even hardware precautions in place to prevent
Speaker 2: that as well, So that makes it a little bit easier.
Speaker 1: I mean, nine percent probably isn't good enough for cars, right,
Speaker 1: They probably need more nines than that, whereas it may
Speaker 1: well be good enough for a house.
Speaker 2: Cleaning robots, yeah, in certain circumstances. And yeah, like we're
Speaker 2: also thinking about scenarios where maybe even less than that
Speaker 2: is fine. And if we view humans and robots working together,
Speaker 2: it's more about kind of helping the person complete the
Speaker 2: task faster or complete the task more effectively. So I
Speaker 2: think there might be scenarios like that, but still we
Speaker 2: need the performance and reliability to be higher for the
Speaker 2: robots to be faster in order to accomplish that.
Speaker 1: We'll be back in just a minute. What do you
Speaker 1: imagine as the initial real world use cases?
Speaker 2: I don't know. There's a lot of examples of robotics
Speaker 2: companies that have a tempted to kind of start with
Speaker 2: an application and hone in on that, and I think
Speaker 2: the lesson from watching those companies is that you end
Speaker 2: up then spending a lot of time on the problems
Speaker 2: of that specific application and less on developing the sort
Speaker 2: of generalist systems that we think in the long run
Speaker 2: will be more effective. And so we're very focused on
Speaker 2: understanding what are the core bottlenecks and the core missing
Speaker 2: pieces for developing these generalist models, and we think that
Speaker 2: if we had picked an application now, we would kind
Speaker 2: of lose sight of that bigger problem because we need
Speaker 2: to solve things that are specific to that application. So
Speaker 2: we're very focused on what we think are like the
Speaker 2: core technological challenges. We have certain tasks that we're working on.
Speaker 2: Some of them have been home cleaning tasks. We've also
Speaker 2: have some more kind of industrial light tasks as well,
Speaker 2: just to instantiate and actually be iterating on robots and
Speaker 2: applications could range from things and homes to things in
Speaker 2: workplaces to industrial settings. There's lots and lots of use
Speaker 2: cases for intelligent robots and intelligent kind of physical machines.
Speaker 1: What are some of the industrial tasks you've been working on.
Speaker 2: One example that I mentioned before is inserting cables. There's
Speaker 2: lots of use cases in data centers, for example, where
Speaker 2: that's a challenging task. Another example is constructing cardboard boxes
Speaker 2: and filling them with items. We've also done some packaging
Speaker 2: tasks highly relevant to lots of different kind of shipping operations.
Speaker 2: And then even folding clothes. It seems like a very
Speaker 2: home task, but it turns out that there are companies
Speaker 2: that need to fold like very large lots of clothing,
Speaker 2: and so that's also something that in the long term
Speaker 2: could be used in larger scale settings.
Speaker 1: So I've read that you have open sourced your model
Speaker 1: weights and given designs of robots to hardware companies, and
Speaker 1: I'm interested in that and that set of decisions, right,
Speaker 1: that set of sort of strategic decisions. Tell me about
Speaker 1: that sort of giving away IP basically.
Speaker 2: Right, yeah, yeah, definitely. So this is a really hard problem,
Speaker 2: especially this longer term problem of developing a general system.
Speaker 2: We think that the field is very young, and there's
Speaker 2: like a couple of reasons. One is that we think
Speaker 2: that the field needs to mature, and we think that
Speaker 2: having more people being kind of competent with using robots
Speaker 2: and using this kind of technology will be beneficial in
Speaker 2: the long term for the company, and by open sourcing things,
Speaker 2: we make it easier for people to do that. And
Speaker 2: then the second thing is, like the models that we
Speaker 2: develop right now, they're very early, and the models that
Speaker 2: we'll be developing one to three years from now are
Speaker 2: going to be far far more capable than the ones
Speaker 2: that we have now. And so it's kind of like
Speaker 2: like equivalent to like open eye open sourcing GPT to
Speaker 2: GPT three. They actually didn't open source GPT three, but like,
Speaker 2: I think that they would still be in an excellent
Speaker 2: spot today if they had.
Speaker 1: Like what could go wrong that would either prevent you
Speaker 1: as a company from succeeding or even hold back the
Speaker 1: field In general, I don't think we.
Speaker 2: Entirely know the scale of data that we need for
Speaker 2: getting really capable models. And there's a little bit of
Speaker 2: a chicken and egg problem where it's a lot easier
Speaker 2: to collect data once you have a really good model.
Speaker 2: It took like large amounts of data.
Speaker 1: Right, Or if there were thousands of robots out of
Speaker 1: the world running your model, they would just make an
Speaker 1: incredible amount of data coming into you every day, right.
Speaker 2: Yeah, yeah, exactly. So that's that's one thing I actually
Speaker 2: less maybe less a little bit less concerned about that myself.
Speaker 2: And then I think the other thing is just that
Speaker 2: there are technological challenges to getting these things to work
Speaker 2: really well. I think that I think we've had incredible
Speaker 2: progress over the last year and two months, the last
Speaker 2: like fourteen months. I think since we've started, probably more
Speaker 2: progress than I was expecting, honestly compared to when we
Speaker 2: started the company. I think it's like wild that we
Speaker 2: were able to get a robot to like unload and
Speaker 2: fold laundry like a ten minute long task.
Speaker 1: And folding laundry is like a famously hard robot problem, right,
Speaker 1: Like it's the one that people in robotics talk about
Speaker 1: when they talk about things people think are easy are
Speaker 1: actually hard for robots, right.
Speaker 2: Yeah, absolutely absolutely. I mean you have to deal with
Speaker 2: all sorts of variability and how clothes can be crumpled
Speaker 2: on each other. And also it's like there's even like
Speaker 2: really small, minor things you need to do in order
Speaker 2: to like actually get it to be flat on the
Speaker 2: table and folded nicely and even stacked. And as the
Speaker 2: task gets longer as well, there are more opportunities to
Speaker 2: make mistakes, more opportunities to get stuck. And so if
Speaker 2: you're doing a task it takes ten minutes, in those
Speaker 2: ten minutes, there's many many times where the robot can
Speaker 2: make a mistake that it can't recover from or just
Speaker 2: get stuck or something like that. And so being able
Speaker 2: to do such a task starts to kind of point
Speaker 2: at the resilience that these models can have by recovering
Speaker 2: from those mystics. Uh huh, so when we were first
Speaker 2: trying to fold laundry, like, one of the common failure
Speaker 2: modes is that it would fold the laundry like very
Speaker 2: well by my standards at the time, I would be
Speaker 2: very very happy with the robot, and then it would
Speaker 2: push the entire stack of laundry onto the ground.
Speaker 1: Sort of like teaching a toddler to fold clothes.
Speaker 2: Yeah, yeah, exactly.
Speaker 1: Was there a particular moment when you saw a robot
Speaker 1: using your model full close for ten minutes and it worked.
Speaker 2: Yeah. First off, we started with just folding a shirt
Speaker 2: starting flat on the table. We got that to work
Speaker 2: pretty quickly that it turns out to be pretty easy,
Speaker 2: and I wasn't too surprised by that. And then we
Speaker 2: moved from that to starting it in like just a
Speaker 2: random ball, like some sort of crumpled position on the table,
Speaker 2: and then you have to flatten and then fold it,
Speaker 2: and that makes a problem dramatically harder because of all
Speaker 2: the variability having to figure out how to flatten it.
Speaker 2: We were kind of stuck on that problem for at
Speaker 2: least a couple of months, where everything we're trying, the
Speaker 2: success rate of the robot was zero percent. It wasn't
Speaker 2: able to really make progress on it, and we started
Speaker 2: to see signs of life I think in August or
Speaker 2: September of last year, where we tried a new recipe
Speaker 2: where we were continue to train the model on a
Speaker 2: curated part of the data that was following a consistent strategy,
Speaker 2: and that sort of high quality post training is what
Speaker 2: really seemed to make the model work better. And then
Speaker 2: the moment that I was most excited about was the
Speaker 2: first time that I saw the model flatten and fold
Speaker 2: and stack five items in a row.
Speaker 1: Yeah.
Speaker 2: I just remember going home that night and being like
Speaker 2: so excited. It seemed like we had just like figured
Speaker 2: out this this big missing puzzle piece.
Speaker 1: So I was asking you why might it not work
Speaker 1: or what might slow the field down? And then we
Speaker 1: talked about the happy short story. But if in five
Speaker 1: years things didn't progress as quickly as you thought, what
Speaker 1: might have happened.
Speaker 2: I mentioned that I think that incorporating practice, like allowing
Speaker 2: the we're about to practice the task, should be really
Speaker 2: helpful for allowing robots to get better. We don't know
Speaker 2: what exactly that recipe will look like, and so it's
Speaker 2: like a research problem, and with any sort of research problem,
Speaker 2: you don't know exactly how hard the solution is going
Speaker 2: to be, and I think that there are some other
Speaker 2: more nuanced unknowns as well that are somewhat similar to that.
Speaker 2: And we have a large number of very talented researchers
Speaker 2: on our team because we think that there are some
Speaker 2: of these unsolved breakthroughs that are going to be needed
Speaker 2: to really truly solve this problem.
Speaker 1: So, if it does work well and things progress in
Speaker 1: that universe, what would you be worried about?
Speaker 2: Good question? I mean, if things work well, I shouldn't
Speaker 2: be too worried. In general. I do think that it's
Speaker 2: very easy in general to underestimate the challenges around actually
Speaker 2: deploying and disseminating technology that takes time, and when the
Speaker 2: technology doesn't exist yet, that means that like the world
Speaker 2: is not in a place that is like ready for
Speaker 2: that technology. I think that there's a lot of unknowns there.
Speaker 1: I mean, one of the striking things to me about, say,
Speaker 1: language models, is the people who know the most about
Speaker 1: them seem to be the most worried about them, which
Speaker 1: is generally not the case. I think historically with technology,
Speaker 1: right the possible exception of the atomic bomb, and so
Speaker 1: I'm curious. I mean those kinds of worries, like do
Speaker 1: you share them? Are there worries you have about developing
Speaker 1: a foundation model for robots about bad actors using it?
Speaker 2: Even I do think that, like, yeah, there's plenty of
Speaker 2: technology that has dual uses, and I think there are
Speaker 2: applications of technologies that are harmful. I think that a
Speaker 2: lot of the concerns in the language model community stem
Speaker 2: from imviewing these systems with greater autonomy. And I think
Speaker 2: that I work like hands on with the robots quite
Speaker 2: a bit, and I don't see a world in which
Speaker 2: they will be taking over in any way. It's very
Speaker 2: easy to just like, well, with our current iteration of robots,
Speaker 2: to just like if we threw some water on it,
Speaker 2: the robot wouldn't be in trouble.
Speaker 1: So that might be a problem for you, but I'm
Speaker 1: sure you could solve that way we're working.
Speaker 2: We're working on so we actually do have a new
Speaker 2: iteration that that is actually a lot more waterproof. But
Speaker 2: it's just not a concern that I show.
Speaker 1: Okay, interesting basically just because you think we can whatever
Speaker 1: turn it off if we need to.
Speaker 2: Yeah, and yeah, and I think, yeah, there's always going
Speaker 2: to be dual use concerns, but I think that the
Speaker 2: pros of the technology outweigh outway some of the Jobson's.
Speaker 1: Well, give me the happy story, then, like in what
Speaker 1: what number of years should we choose for a happy story?
Speaker 1: Ten is ten too soon?
Speaker 2: I don't want to put a number to it. I
Speaker 2: think that they with research, you don't know exactly how
Speaker 2: thongs things will take. And I an envision a world
Speaker 2: where the when you're developing hardware, it's it's not too
Speaker 2: hard to actually teach it to do something, and teach
Speaker 2: it to do something useful, rather than just having machines
Speaker 2: that are not particularly intelligent, like dishwashers and laundry machines
Speaker 2: and so forth.
Speaker 1: Go bigger if you would like what like what what
Speaker 1: what would be pill be teached robots to do in
Speaker 1: that world, I.
Speaker 2: Guess if we were to go bigger, I think that
Speaker 2: there's a lot of challenges around helping helping people as
Speaker 2: the age allowing them to be more independent. That that's
Speaker 2: like a huge one. I think that I don't know, manufacturing,
Speaker 2: there's all sorts of places where like there's abuse of
Speaker 2: labor practices and we can maybe like be able to
Speaker 2: eliminate those if it's a robot instead of a human. Yeah, many, many,
Speaker 2: many examples. And I think that there's also even things
Speaker 2: that are even hard to imagine because the technology doesn't exist.
Speaker 2: So a lot of the things that I'm thinking about
Speaker 2: are robots helping humans in different circumstances to allow them
Speaker 2: to be more productive. But once something exists, like you often,
Speaker 2: like people are creative and come up with new ways
Speaker 2: of how that's used.
Speaker 1: We'll be back in a minute with the lightning round. Great,
Speaker 1: let's finish with the lightning round. What's one thing that
Speaker 1: working with robots has caused you to appreciate about the
Speaker 1: human body?
Speaker 2: Our skin is pretty amazing.
Speaker 1: Huh. Well, so we didn't talk about I mean a
Speaker 1: sense of touch, or of of heat or of cold, right,
Speaker 1: I mean presumably the models you're building, the robots you're
Speaker 1: using don't have that, but they could, right, they could
Speaker 1: have a sense of touch. Is anyone working on that?
Speaker 1: Is that of interest to you?
Speaker 2: Lots of people working on it. I think it's pretty interesting.
Speaker 2: I think that the hardware technology is not super mature
Speaker 2: compared to where I'd like for it to be in
Speaker 2: terms of how robust it is. And the cheapness and
Speaker 2: the resolution that said, Like, we actually put cameras on
Speaker 2: the risks of our robot to help it get some
Speaker 2: sort of tactile and for example, if you can, if
Speaker 2: you like visually look at your finger as you make
Speaker 2: contact with an object, you can see it to form
Speaker 2: around that object, and you can actually just by looking
Speaker 2: at your finger get some notion of tactile feedback similar
Speaker 2: to what our skin gets. Yeah, and cameras are cheap,
Speaker 2: really easy, robust, way more robust and cheap than existing
Speaker 2: technology for tactile something.
Speaker 1: I've heard you say that humanoid robots are overrated, and
Speaker 1: I'm curious, why do you think that.
Speaker 2: I think that simplicity is really helpful and important when
Speaker 2: trying to develop technology. When you introduce more complexity than's needed,
Speaker 2: it slows you down a lot. And I think that
Speaker 2: the complexity that humanoids introduce. Yeah, I think that if
Speaker 2: all of the robots we were working with were humanoids,
Speaker 2: I think that we wouldn't have made anywhere near the
Speaker 2: progress that we've made because we'd be dealing with additional challenges.
Speaker 2: I also think that optimizing for ease of data collection
Speaker 2: is really important in a world where we need data,
Speaker 2: and it's a lot harder to collect and operate all
Speaker 2: of the different joints and motors of a humanoid than
Speaker 2: it is to control a simpler robot.
Speaker 1: Do you anthropomorphize robots?
Speaker 2: I hate it when people are anthrough morphize robots. I
Speaker 2: think that it is misleading because the failure modes that
Speaker 2: robots have are very different from the failure modes that
Speaker 2: people have, and it misleads people into thinking that it's
Speaker 2: going to behave in the way that people behave.
Speaker 1: Like like in what way?
Speaker 2: Oh like, if you see a robot doing something like
Speaker 2: doing a backflip, like or even folding laundry, you kind
Speaker 2: of assume that anything like like if you saw a
Speaker 2: person do that, then they probably could do a lot
Speaker 2: of other things too. And if you have to promorphize
Speaker 2: the robot, then you assume that it, like the capabilities
Speaker 2: that you see are representative as if it were like
Speaker 2: a human ah, and that it could do a backflip anywhere,
Speaker 2: or that it could fold laundry anywhere with any item
Speaker 2: of clothing.
Speaker 1: Or surely you would think a robot that could do
Speaker 1: a backflip could fold a shirt, but no.
Speaker 2: Exactly exactly, so sometimes it's fun to like assign emotions
Speaker 2: to some of the things, or say the robots having
Speaker 2: a bad day, because certainly it feels like that sometime.
Speaker 2: But when it kind of moves beyond fun and jokes,
Speaker 2: it might have consequences that I don't think makes sense.
Speaker 1: I read that there was a researcher who said they
Speaker 1: would retire if a robot tied to shoela Yes, and
Speaker 1: then one of your robots tied to shoelace, and I
Speaker 1: guess they didn't retire. But I'm curious. What would you
Speaker 1: need to see a robot do to retire.
Speaker 2: Hmm, I don't know. I guess one example that I've
Speaker 2: given before that I would love to see a robot do.
Speaker 2: I don't think this is quite retirement level, but being
Speaker 2: able to go into a kitchen that has never been
Speaker 2: in before and make a bowl of cereal pretty basic,
Speaker 2: especially compared to doing a backflip. I cannot do a
Speaker 2: backflip myself, but I could make a bowl of cereal.
Speaker 2: But it requires being able to find objects in the environment,
Speaker 2: being able to interact with delicate objects like a cereal box,
Speaker 2: maybe even use tools in order to open the cereal box.
Speaker 2: Pouring liquids. Yeah, so that's a task that I love,
Speaker 2: and I could actually even see us being able to
Speaker 2: show a demo of that without too much difficulty actually
Speaker 2: if we put our mind to it and in collected
Speaker 2: data for it. So it actually is, I think, or
Speaker 2: within reach than maybe I imagined a few years ago.
Speaker 1: Just as you're thinking about it, it's getting closer. You're like, oh, wait,
Speaker 1: we could do that.
Speaker 2: Yeah. I mean we've actually collected data of pouring cereal,
Speaker 2: like opening a cereal box and pouring it into a bowl.
Speaker 2: We haven't yet done liquid handling and pouring, but I
Speaker 2: think we're actually going to do it this week. On
Speaker 2: the Robot, I asked the hardware team to make a
Speaker 2: waterproof robot. So we're not too far. A lot of
Speaker 2: the pieces are coming together. I also, I love working
Speaker 2: with robots and so, and I'm also fairly young, I
Speaker 2: think not too old, and so I don't imagine myself
Speaker 2: retiring anytime soon.
Speaker 1: Chelsea Finn is a Stanford professor and the co founder
Speaker 1: of Physical Intelligence. You can email us at problem at
Speaker 1: pushkin dot fm, and please do email us. I read
Speaker 1: all the emails. Today's show was produced by Gabriel Hunter Chang,
Speaker 1: edited by Alexander Garreton and engineered by Sarah Bruguerrett. I'm
Speaker 1: Jacob Goldstein and we'll be back next week with another
Speaker 1: episode of What's Your Pop