Speaker 1: Pushkin.
Speaker 2: Over the past year, we've heard a lot about artificial
Speaker 2: intelligence models that are really good at manipulating language. We've
Speaker 2: heard somewhat less about AI that deals with images. It's
Speaker 2: called computer vision, and it's a huge deal, which you
Speaker 2: know obviously. Like language, vision is this core part of
Speaker 2: the experience of being human, And on a more practical level,
Speaker 2: computer vision is key for self driving cars, and for
Speaker 2: drones and for all kinds of industrial robots. As it
Speaker 2: turns out, there was this one key moment in the
Speaker 2: development of modern AI, for both vision and language, and
Speaker 2: if you understand this moment, you understand a lot about
Speaker 2: how AI works today. I'm Jacob Goldstein. This is What's
Speaker 2: Your Problem, the show where I talk to people who
Speaker 2: are trying to make technological progress. My guest today played
Speaker 2: a central role in that key moment in AI history.
Speaker 2: Her name is faith A.
Speaker 1: Lee.
Speaker 2: She's a Stanford computer scientist, the author of a memoir
Speaker 2: called The Worlds I See, the former chief scientist of
Speaker 2: AI and machine learning at Google, and just generally one
Speaker 2: of the most important innovators in the history of computer vision.
Speaker 2: I started our conversation with really a pretty general question.
Speaker 2: I asked Fay fe just to explain what computer vision
Speaker 2: is and why it's so important.
Speaker 3: So computer vision is about enabling computers and machines to
Speaker 3: have visual intelligence. What is visual intelligence? Well, the best
Speaker 3: example comes from humans who are extremely visually intelligent animals,
Speaker 3: So that we can make an omelet by knowing what
Speaker 3: is in our fridge. How do we go and take
Speaker 3: the egg out, how do we take the tomato out?
Speaker 3: How do we plan the cooking of the omelet? How
Speaker 3: do we interact with every ingredients, and how do we
Speaker 3: understand all the changes of the objects and all this
Speaker 3: is part of visual intelligence.
Speaker 2: Yeah, I mean you write in your book that vision
Speaker 2: isn't just an application of our intelligence, it is synonymous
Speaker 2: with our intelligence, which is something I want to talk
Speaker 2: more about. But before we get into human vision and
Speaker 2: how that led you into computer vision, just give me
Speaker 2: a sense of some of the applications, both the current
Speaker 2: applications of computer vision and potential future applications of computer vision.
Speaker 3: In fact, we're already using computer vision to do a
Speaker 3: lot of things. The most obvious example is all kinds
Speaker 3: of driver's assistant programs. Right, we're not having even about
Speaker 3: self driving cars. We're talking about lane detection or talking
Speaker 3: about avoiding curb sized pedestrian alert. You know, we are
Speaker 3: using computer vision in our healthcare system, in radiology, in pathology,
Speaker 3: or you know, in protecting of species. A lot of
Speaker 3: the camera traps in the in the deep forests are
Speaker 3: using computer vision to track to track different animals. So
Speaker 3: we're using computer vision already on a daily basis.
Speaker 2: And then when you dream of some applications that are
Speaker 2: not here yet but that might be here in whatever
Speaker 2: five or ten years, what do you think of what's
Speaker 2: at the top of the list.
Speaker 3: So when I dream of computer vision, I dream of
Speaker 3: all kinds of robotic application, so from self driving car
Speaker 3: to personal robots using computer vision. I dream of our
Speaker 3: biodiversity being mapped using computer vision. I dream of exploration
Speaker 3: using computer vision.
Speaker 2: Wonderful. So I want to talk about your work in
Speaker 2: computer vision, which goes back well decades now, and I
Speaker 2: want to start with work not on computers actually, but
Speaker 2: on human beings right, on understanding of how humans process
Speaker 2: visual information, right, how we make sense of what we're seeing.
Speaker 2: And in the book, you write in particular about this
Speaker 2: nineteen ninety six paper with a boring name that was
Speaker 2: a huge deal. It was called speed of processing in
Speaker 2: the human visual system. Tell me about that paper and
Speaker 2: what it meant.
Speaker 3: It's a paper of using EG, which is recording electrical
Speaker 3: brain waves to make a link between how fast can
Speaker 3: humans make a very complex visual decision when they sees something,
Speaker 3: And the particular decision humans were to make is to
Speaker 3: separate images from images containing animals and images not containing animals.
Speaker 3: And if you think about the pool of possibilities is
Speaker 3: extremely complex. It's actually mathematically just an infinite possibility because
Speaker 3: there are so many different types of animals, so many
Speaker 3: different different types of non animals. That's infinite for practical purposes.
Speaker 3: And then you put them in photos, you can get
Speaker 3: infinite possibilities of photos. Yet you show them one by
Speaker 3: one two humans they make decisions really quickly, and they
Speaker 3: make correct decisions really quickly.
Speaker 2: That really quickly, but like mind bogglingly quickly at the time, right,
Speaker 2: it was shocking just how fast it was, right milliseconds.
Speaker 3: Yeah. So the thing is we kind of sort of
Speaker 3: know we're good at see, right as a species. We
Speaker 3: know we open our eyes we see the world, but
Speaker 3: we don't really know how good and how fast.
Speaker 2: And this is we underestimate. It's it's a rare case
Speaker 2: where human beings underestimate ourselves exactly.
Speaker 3: This is a rigorous scientific study put a time, actual
Speaker 3: time to that speed of visual intelligence, and it's using
Speaker 3: modern technique. It's very smart and very very exciting.
Speaker 2: What did it mean to you when you saw that result?
Speaker 2: When you read that paper?
Speaker 3: When I read that paper, it means north star. Let
Speaker 3: me explain what does north star mean? As north star?
Speaker 2: Okay?
Speaker 3: Yeah, As a scientist, I'm driven by finding answers to
Speaker 3: the most audacious question. But as high Einstein has said,
Speaker 3: in science scientific inquiry, the hardest job is not finding
Speaker 3: solutions asking the right question because you you know, when
Speaker 3: you like we talk about visual intelligence, it's such a
Speaker 3: vast topic. What is the topic to pursue? What is
Speaker 3: the question to ask that is fundamental to visual intelligence?
Speaker 3: And how do we unlock it? When when we read
Speaker 3: that Simon Thorp paper, it convinced me that complex object categorization,
Speaker 3: the ability to classify you know, animal versus no animal,
Speaker 3: chair versus you know, table, hot dog, hot dog versus hamburger.
Speaker 3: You know, this is fundamental told to humans. It's a
Speaker 3: building block of visual intelligence. Is it has a neural
Speaker 3: correlate in human brain that shows how evolutionally evolutionarily optimized
Speaker 3: it is. So with all that evidence, it comments me
Speaker 3: object categorization is a north star to pursue.
Speaker 2: And you were a grad student at the time, right,
Speaker 2: this is sort of the thing and any ambitious grad
Speaker 2: student that's going to be doing it's like, I know,
Speaker 2: I'm interested in this field, but I need my question,
Speaker 2: I need my thing, right, And so now you've got
Speaker 2: your thing, yes, and it's categorization in particular. And you
Speaker 2: describe how earlier theories of how humans process visual input
Speaker 2: was not so categorization focused, right. It was kind of
Speaker 2: like if you just sort of thought from first principles,
Speaker 2: you would think, well, we see color and we see
Speaker 2: shapes and then we kind of make sense of it.
Speaker 2: But with this paper and related work, it show is
Speaker 2: like that's actually not it, right, and in fact our brains.
Speaker 2: You write about how there are specific regions of the
Speaker 2: brain like this region is just face, the face categorization region,
Speaker 2: and this region is the like place as we go
Speaker 2: all the time region And so it's a really different
Speaker 2: and interesting way of thinking about seeing, and it's fundamentally
Speaker 2: about just incredibly quickly putting things into categories. And so
Speaker 2: you decide to take this idea of vision and categorization
Speaker 2: and try and figure out how to how to get
Speaker 2: computers to do this, right, how to get computers to
Speaker 2: be able to categorize objects from the world. And you
Speaker 2: start building these data sets essentially of labeled images, right,
Speaker 2: And you build what seems in retrospect like a relatively
Speaker 2: small one at Caltech, and then you decide to build
Speaker 2: a really big one. Right. It comes to be called
Speaker 2: the image net. It's a thing you're famous for, NERD
Speaker 2: famous for, And I want to talk about building image net, right,
Speaker 2: So tell me about deciding to build what becomes image net.
Speaker 3: So the image net is the north Star. To me,
Speaker 3: I was in the field long enough because I finished
Speaker 3: my PhD, I started my own lab. I had this
Speaker 3: unwavering faith and believe that, you know, unlocking object recognition
Speaker 3: is part of the north is a north star, is
Speaker 3: a critical north star. And I became impatient because I
Speaker 3: realized we were not making enough progress. I realized that,
Speaker 3: especially algorithmically, we were like running in circles a little
Speaker 3: bit of optimizing very small algorithms that are not really
Speaker 3: getting to the essence of the problem, and part of
Speaker 3: the essence, which a lot of people overlooked, is actually
Speaker 3: the scale of the problem. What was really bothering me
Speaker 3: is that we were not seeing the problem. We're not
Speaker 3: seeing the mathematical problem with the scale thinking, because it's
Speaker 3: not just about being big. It's about the mathematical reason
Speaker 3: of why we should go big, and it's it's a
Speaker 3: very deep reason in general, it's a reason for what
Speaker 3: we call generalization. You have to learn enough to be
Speaker 3: able to see everything and that minds You've got.
Speaker 2: To see a lot of pictures of things that are
Speaker 2: cats and not cats to understand what.
Speaker 3: Is right that that mindset was just not you know,
Speaker 3: that's a big data mindset. It was not in the
Speaker 3: world at all at that time.
Speaker 2: So how did you get there? How did you Because
Speaker 2: what you end up doing is building this just gargantuan
Speaker 2: uh thing full of labeled images, bigger than anybody'd ever
Speaker 2: built before. Like, how did you arrive at that?
Speaker 3: That's a great question. I think that's actually the most
Speaker 3: fun but difficult part of the book to write, is
Speaker 3: you know, like dig in to my own brain. In hindsight,
Speaker 3: it's just little by little the insight and the realization epiphany,
Speaker 3: But honestly, I don't know how to analyze my own brain.
Speaker 3: I had the mathematical intuition that scale makes a difference,
Speaker 3: bigger difference than most people give credit to. I also
Speaker 3: had the neurocognitive science inspiration that early human development was
Speaker 3: exposure to the world in continuous ways. We don't like
Speaker 3: lock the baby in a dark room and show them,
Speaker 3: you know, one hundred cats. They just go out and experience.
Speaker 3: You know that experience is actually driven by big data.
Speaker 3: Maybe I was also inspired by this Internet age coming
Speaker 3: our way right like that part, I do think it's
Speaker 3: a little bit moment of just being alone, and somehow
Speaker 3: all the stars aligned in my head. I decided I'm
Speaker 3: going to try the craziest thing, and I did have
Speaker 3: a faith and believe that it was the right thing
Speaker 3: to do.
Speaker 2: And specifically, like, what was this thing that you were
Speaker 2: going to build?
Speaker 3: I'm gonna get the entire Internet of images, consisted of
Speaker 3: all the objects I can get my hands on that
Speaker 3: humans have ever taken pictures of and catalog them in
Speaker 3: a gigantic, big database. And I will use that to
Speaker 3: do two things. To train machines to recognize the entire
Speaker 3: world of objects, and also to benchmark everybody's progress. You
Speaker 3: know everybody, I mean the international community of computer vision scientists.
Speaker 2: So you will have this database and then everyone can
Speaker 2: train their computer vision models on your database and see
Speaker 2: how they do on new images.
Speaker 3: Yes, so.
Speaker 2: You have to decide. There's this interesting part of the
Speaker 2: book where you're like, Okay, I want to build a
Speaker 2: database with everything in it. How many categories of everything
Speaker 2: are there? Right, somebody's actually done that research. If you
Speaker 2: take all the things, how many kinds of things are there?
Speaker 2: What's the number?
Speaker 3: The number is the Beaderman number, and the Beierman number
Speaker 3: is a I'm proud of really giving professor or Piederman
Speaker 3: that credit. Yeah, nobody noticed that number. He wrote. He's
Speaker 3: a cognitive scientist who wrote it very very good, But
Speaker 3: I don't think it's a famous paper in the nineteen
Speaker 3: eighties guestimating or estimating with the back of the envelope
Speaker 3: computation of how many visual concepts humans see? And that
Speaker 3: is a very hard number. How do you interrogate a
Speaker 3: person and say, list me all the visual concepts. It's impossible.
Speaker 3: But he had a way of using dictionary and using
Speaker 3: visual structure to estimate, and he put a number of
Speaker 3: thirty thousand visual concepts.
Speaker 2: They're thirty thousand different sort of kinds of things, right,
Speaker 2: people can identify differentiating. Yeah, it's a lot.
Speaker 3: That's a lot.
Speaker 2: Yeah, And every concept you're setting out, if you're setting out, sorry,
Speaker 2: is that your number? Is your number?
Speaker 3: That was my number. I was obsessed with that number,
Speaker 3: and I was obsessed in a way that I feel
Speaker 3: I was kind of crazy because nobody was obsessed with
Speaker 3: that number. Nobody even knew. I think my book is
Speaker 3: the book that gave the number A which is be
Speaker 3: the most number, and I'm very proud of that.
Speaker 2: Do you can you just rattle off some of the categories.
Speaker 3: Star nosed mole, star No's mole category to itself, that's
Speaker 3: my favorite, one of my favorite categories. And Guardenian windsor
Speaker 3: flower has windsor chair. There were hundreds of dogs. I
Speaker 3: remember there were uh different kind of cars like like
Speaker 3: sports sedan and uh monocycles. It's it's a lot.
Speaker 2: So Faithley has her number, she has her big idea.
Speaker 2: She knows what she needs to build a gigantic image database.
Speaker 2: But how do you actually do that?
Speaker 1: Well? I have the answer in just a minute.
Speaker 2: Okay, so you've got your giant north Star task ahead
Speaker 2: of you. Not only do you have, you know, thirty
Speaker 2: thousand ish categories to deal with, presumably for each category
Speaker 2: you need many many thousands. So it's thousands of images
Speaker 2: per category, tens of thousands of categories. What is the
Speaker 2: order of magnitude?
Speaker 3: We're talking about tons of millions?
Speaker 2: A million this is and this is not a time
Speaker 2: where you can do this in an automated or semi
Speaker 2: automated way like you could now.
Speaker 3: No, I mean the point is, the machines cannot do it.
Speaker 3: We have to. This is a north start to push
Speaker 3: machines towards that, So you have to do it by
Speaker 3: human hen and the good.
Speaker 2: News downloading and labeling. Yeah, millions or tens of millions
Speaker 2: of images.
Speaker 3: Downloading, cleaning, labeling, and yes, that's that was the task.
Speaker 2: So now you're like Henry Ford or something right now
Speaker 2: you need an assembly line, you need a factory for
Speaker 2: creating this database.
Speaker 3: Yeah, you can put it that way. And we needed
Speaker 3: a global workforce, and eventually we found them on Amazon
Speaker 3: Mechanical Turk. It's an online global market.
Speaker 2: It's a market for project based work, right, people doing
Speaker 2: project based work. And so, so how long does it
Speaker 2: take you to build this thing? And how big is
Speaker 2: it when it's done?
Speaker 3: It took us three years. When it was done, it's
Speaker 3: fifteen million hand cleaned, sorted, curated, labeled images across twenty
Speaker 3: two thousand categories.
Speaker 2: So now you have this thing. It's called image net
Speaker 2: and basically the function of it is it itself is
Speaker 2: not useful, right, It is there to train. Well, it's
Speaker 2: useful as a means to an end. It's there for
Speaker 2: people who have models that aim to teach computer's vision
Speaker 2: to see and understand to train their models. Now there's
Speaker 2: this giant database. I mean people talk about this as
Speaker 2: kind of one of the beginnings of big data.
Speaker 3: Yes, yeah, I think it should be properly recognized as
Speaker 3: the beginning of big data in AI, because before this,
Speaker 3: there isn't this concept of big data in AI. It
Speaker 3: was just a paradigm shift from that point of view.
Speaker 2: And so you create this contest where people can come
Speaker 2: and train their models on image net, on this giant
Speaker 2: database that you've built, and then and then in the
Speaker 2: contest their models will be shown new images images not
Speaker 2: in the database, and you'll see how how good they are.
Speaker 2: And for a while it's like going okay, right, but
Speaker 2: kind of slow, like you're in the book you write
Speaker 2: about like you get a little worried. You've built this
Speaker 2: giant thing with people all around the world and it's
Speaker 2: not for a while leading to the breakthroughs that you
Speaker 2: had imagined.
Speaker 3: Yeah, it was first of all, we open source this.
Speaker 3: We didn't even though we spend a lot of sweat
Speaker 3: and tears, you know, building this, but we know the
Speaker 3: real value is to open source. So we gave it
Speaker 3: for free to the whole community. And then I wanted
Speaker 3: everybody to use it. I wanted to see this driving
Speaker 3: all of us towards the north Star. I want the
Speaker 3: field to work out. But it wasn't like an overnight success.
Speaker 3: It wasn't like everybody's running around and say, oh my god,
Speaker 3: there's image Net to use. And of course we were not.
Speaker 3: You know, we were disappointed, but we were not sitting
Speaker 3: there crying. We were just disappointed.
Speaker 2: And so there is this big moment right after a
Speaker 2: few years at one of the contests, there's a new
Speaker 2: model in three years, So tell me about that moment.
Speaker 3: So the result of two thousand and twelve came in
Speaker 3: and we saw this result coming out of Professor Jeff
Speaker 3: Hinton's lab using your network, and the difference that the
Speaker 3: error reduction compared to previous years was just much bigger,
Speaker 3: you know, and we started to realize this is a
Speaker 3: very very significant moment because there's a serious, serious breakthrough
Speaker 3: in terms of the results of image that which is
Speaker 3: the north Star problem. Right. So it was so important
Speaker 3: for me that I, you know, bought last minute plane
Speaker 3: ticket to fly to Italy to announce the Imagnet Challenge
Speaker 3: winner that year, and.
Speaker 2: You weren't going to go otherwise.
Speaker 3: I wasn't planning to go because I was still a
Speaker 3: nursing mom, so I was taking you know, I was
Speaker 3: mostly working from home at that home in that month.
Speaker 3: But I was like, this is so important that I
Speaker 3: needed to go.
Speaker 2: And so I mean, so this was someone working with
Speaker 2: Jeff Hinton and using a neural network like today Jeff Hinton.
Speaker 2: You know, if you know two names in AI, Jeff
Speaker 2: Hinton is probably one of them. People call him the
Speaker 2: godfather of kind of modern AI, right, and neural networks
Speaker 2: are essentially the thing that has worked right both for
Speaker 2: vision and for language. You know, jat GPTs neural network,
Speaker 2: and so this was a moment was like, oh, this
Speaker 2: technique that a lot of people thought wasn't going to
Speaker 2: work had kind of given up on its back.
Speaker 3: Yeah, yeah, exactly. I think it's actually a parallel story
Speaker 3: of two groups of people that had that determination seeing
Speaker 3: something that you know, maybe the mainstream wasn't seen, and
Speaker 3: then had the resilience and just perseverance to keep marching on.
Speaker 3: I was doing my north Star pursue. I was doing
Speaker 3: the big data approach. They're doing the new network algorithm,
Speaker 3: and then we converged.
Speaker 2: Uh huh. That's really elegant, right, because it's like your
Speaker 2: big data is just sitting there and you don't maybe
Speaker 2: entirely know it, but you kind of need a neural
Speaker 2: network to come in and train on it. Right, And
Speaker 2: they're over there building their neural network and they may
Speaker 2: or may not know it, but they need the big
Speaker 2: data that you're over here building, and then when it
Speaker 2: comes together, it's like, hey.
Speaker 3: It works. Yeah. So I think that's how science progresses.
Speaker 3: It's kind of spiraling up, and sometimes it takes a
Speaker 3: couple of more threats. It's not a single spral I
Speaker 3: remember very vividly that some of one of the critiques,
Speaker 3: one of the main critiques of image net by my
Speaker 3: colleagues is this is too big. We cannot even fit
Speaker 3: this into memory. What are you doing? What are you
Speaker 3: making this giant data set for when we cannot even
Speaker 3: you know, put it on a chip. And as that
Speaker 3: was happening, GPU was happening.
Speaker 2: So GPUs are the Nvidia chips that are now made
Speaker 2: in video, one of the biggest companies in the world.
Speaker 2: But they were figuring out that GPUs are particularly good
Speaker 2: for the neural.
Speaker 3: Network exactly exactly.
Speaker 2: So that moment, this moment when you guys come together
Speaker 2: and kind of create this you know, new era of
Speaker 2: computing really that we're still living in of AI is
Speaker 2: about ten years ago now, right, yeah, so just bring
Speaker 2: me to the present. Like that happened, then where are
Speaker 2: we now? I mean, it's it's kind of the same universe, right,
Speaker 2: It has advanced a lot, but the basic premise of
Speaker 2: you have neural networks training on vast, vast databases of images, like,
Speaker 2: it's basically the same.
Speaker 3: Right, So from my conceptional point of view, you're right,
Speaker 3: at that time, I was downloading the Internet of images
Speaker 3: to be honest. Now, the Internet of Images is just
Speaker 3: so vast, I don't know who can download it all.
Speaker 3: And then the GPUs is mind bogglingly advanced. Right, but
Speaker 3: you're right, the ingredients are still the same.
Speaker 2: We'll be back in a minute with the lightning round.
Speaker 2: Let's do a lightning round. Okay, what's one thing you
Speaker 2: learned running a drag cleaning shop.
Speaker 3: That's how very night to have to think. I think
Speaker 3: I learned resilience because my goal is to be a scientist.
Speaker 3: But if it takes running a dry cleaner shop to
Speaker 3: get there in the most detoured way, I'll have to
Speaker 3: do that.
Speaker 2: So you're writing the book about a high school teacher
Speaker 2: who was a very big and important influence on you,
Speaker 2: and how your advisors in grad school were an important influence.
Speaker 2: And now you have been a mentor to many people.
Speaker 2: So I'm curious, what's one tip for finding a mentor?
Speaker 3: For finding a mentor? That's a great question. I trusted
Speaker 3: them different stage. This trust meant different things. I trust
Speaker 3: their genuine intention, I trusted their wisdom, I trusted their
Speaker 3: vas and I trusted there you know, believe in me.
Speaker 3: So that was how I was lucky to find my mentors.
Speaker 2: What's one tip for being a mentor?
Speaker 3: Being a mentor is really about respecting the the person,
Speaker 3: the soul and help them to find their north star,
Speaker 3: to find their passion.
Speaker 2: If everything goes well, what problem will you be trying
Speaker 2: to solve in five years?
Speaker 3: I'm trying to usher in machines being so helpful and
Speaker 3: collaborative for humans, whether it's productivity or our wellbeing. If
Speaker 3: this includes sensors, spar sensors, virtual agents, or real robots,
Speaker 3: I think it all. You know, I'm very excited by that.
Speaker 2: Feif A Lee is a professor of computer science at
Speaker 2: Stanford and the author of the book The Worlds I See.
Speaker 2: Today's show was produced by Edith Russolo and Gabriel Hunter Chang.
Speaker 2: It was edited by Karen Chakerji and engineered by Sarah Bouger.
Speaker 2: You can email us at problem at Pushkin dot FM.
Speaker 2: I'm Jacob Goldstein, and we'll be back next week with
Speaker 2: another
Speaker 1: Episode of What You Have