Speaker 1: Pushkin. I'm Jacob Goldstein and this is What's Your Problem.
Speaker 1: Two quick things before we start the show today. One,
Speaker 1: we're going to take a little break of a few
Speaker 1: weeks after this week's episode, but we'll be back soon.
Speaker 1: And two, I want to say thanks for listening. I
Speaker 1: really appreciate it. I want to keep making a show
Speaker 1: that you like, so please feel free to email us
Speaker 1: at problem at pushkin dot fm. You can also find
Speaker 1: me on LinkedIn or on x formerly known as Twitter
Speaker 1: at Jacob Goldstein. Okay, that's the housekeeping. Now here's today's show.
Speaker 1: My guest today is Brian McClendon. He's the chief technology
Speaker 1: officer at a company called Niantic Spatial, and his problem
Speaker 1: is this, how do you make maps better? Specifically, how
Speaker 1: do you take the two dimensional maps that we have
Speaker 1: today and turn them into three dimensional maps that change
Speaker 1: over time to keep up with changes in the real world.
Speaker 1: Maps like this would be very useful for robots. For
Speaker 1: one thing, Brian's company is in fact doing a pilot
Speaker 1: with a robot delivery company, those little robots that drive
Speaker 1: down the sidewalk to bring you a pizza or whatever.
Speaker 1: But the maps could also be useful for people. For example,
Speaker 1: the company is working on a project with the US
Speaker 1: Coast Guard to build better flight simulators for helicopter pilots,
Speaker 1: and the kind of maps that the company is building
Speaker 1: could also be useful for augmented reality, and in fact,
Speaker 1: Niantic Spatial spun out of the company that made Pokemon Go,
Speaker 1: the augmented reality game. Brian's been working on maps for
Speaker 1: a long time. In the early two thousands, he worked
Speaker 1: at a mapping startup that got acquired by Google in
Speaker 1: two thousand and four, and Google turned the project he
Speaker 1: was working on into Google Earth, and after that, Brian
Speaker 1: kept working on mapping at Google until twenty fifteen. In
Speaker 1: our conversation, Brian and I discussed the problems he's working
Speaker 1: on now trying to build better maps, including why trees
Speaker 1: are his nemesis. Also, we talked about what it will
Speaker 1: take to teach Ai to understand the physical world, but
Speaker 1: to start. Brian told me the biggest lesson that he
Speaker 1: learned about mapping during his time at Google.
Speaker 2: The most interesting thing is that map data in two
Speaker 2: thousand and four, you know, was already digital. We had
Speaker 2: suppliers that you could buy map data from, and you know,
Speaker 2: map quest was out there showing you a little little
Speaker 2: map on the screen on your website, on their website.
Speaker 2: But what we discovered when we started build Google Maps,
Speaker 2: we used that same data and we showed it to
Speaker 2: our users and launched in two thousand and five. We
Speaker 2: came up with this idea of street view, where we'd
Speaker 2: go take pictures of a lot of places around around
Speaker 2: the city and publish them.
Speaker 1: Can I say, by the way, because I remember when
Speaker 1: that happened and when I heard of that, and I
Speaker 1: when I first heard of it, I was like, surely
Speaker 1: they're not going to drive cars with cameras on every
Speaker 1: street and take pictures, Like, surely that's too much. But no,
Speaker 1: that's what it was like so wildly ambitious, just on
Speaker 1: a logistical level.
Speaker 3: Absolutely, that was It was a crazy project.
Speaker 2: You know, Larry Page gets credit for coming up with
Speaker 2: the idea and then following through and supporting us as
Speaker 2: we built it.
Speaker 1: What did you say when someone was it Larry Page
Speaker 1: who said to you, hey, let's do this, Who told
Speaker 1: you about it?
Speaker 2: You know, Larry, Larry had actually taken a video camera
Speaker 2: and driven around the Stanford campus, you know, you know,
Speaker 2: sort of systematically and said, you know, why can't we
Speaker 2: do this for the whole world?
Speaker 1: What did you say when he said, why can't we
Speaker 1: do this for the whole world? What did you say?
Speaker 3: I said, he's Larry Page and it's his money. Man.
Speaker 3: If he wants to do it, we'll give it a shot.
Speaker 2: And and honestly that's you know, the support that Google
Speaker 2: had for organizing the world's information was incredibly strong.
Speaker 3: I mean, Larry was serious about this.
Speaker 2: And we initially built a very expensive data collection van
Speaker 2: that drove around a few cities. Then we started to
Speaker 2: drive other cars that were cheaper but still okay, and
Speaker 2: then that's when we launched the service. And then we
Speaker 2: eventually said we need to do this for the entire
Speaker 2: United States. And the reason we did this was that
Speaker 2: when we launched the first cities, the first thing that
Speaker 2: people said was they looked at the pictures on the screen,
Speaker 2: and then they looked at their map data, and they
Speaker 2: started complaining about the map data. How can you get
Speaker 2: this wrong? I have a picture right here, this shows
Speaker 2: a street sign and a street number, and you have
Speaker 2: it completely in a different location. So a picture is
Speaker 2: worth a thousand words and probably megabytes of actual useful data.
Speaker 2: And there hadn't been any systematic sort of extraction of
Speaker 2: the value from those pictures, because that's not how maps
Speaker 2: had been made in the past. And so the big
Speaker 2: insight we ever had was that map data to that
Speaker 2: moment was not good.
Speaker 1: And by taking pictures for the first time, people could
Speaker 1: in a scaled way look at a photo right next
Speaker 1: to right on top of the map data and see like, oh,
Speaker 1: the map is wrong, and this photo proves that. Like
Speaker 1: it was the moment when people understood how wrong maps
Speaker 1: in general were exactly.
Speaker 2: And people always kind of knew this because they always
Speaker 2: got frustrated every now and then when they'd use the
Speaker 2: maps and they would fail. But now we had this
Speaker 2: much larger complaint body, and we knew that the data
Speaker 2: that we'd been buying from third parties wasn't good, and
Speaker 2: so we decided to make our own maps. And the
Speaker 2: biggest basis for that was the street view pictures themselves.
Speaker 2: And we drove all of the US, Canada, Mexico in
Speaker 2: two thousand and eight, and by two thousand and thirteen
Speaker 2: we had driven fifty countries.
Speaker 1: So the point of street view was not just for
Speaker 1: the like, hey, you can look at pictures, it's to
Speaker 1: get better data as an input to the sort of
Speaker 1: more traditional maps. Absolutely, I never knew that. Is that
Speaker 1: why Apple Maps sucked when it first came out, because
Speaker 1: they didn't have them.
Speaker 3: I'm so glad you said. It keeps me from saying that.
Speaker 1: I remember when it happened. I remember they fired the guy.
Speaker 1: Remember they fired the guy. But is that why? Because
Speaker 1: they just were buying the normal map data and everybody
Speaker 1: was comparing it to Google and you guys had literally
Speaker 1: driven cars on every street in America to figure it out.
Speaker 2: That's exactly right, and they eventually got their Apple Apple
Speaker 2: has driven you know, many millions of miles themselves, But
Speaker 2: if you look at it worldwide, I would say that
Speaker 2: Google is still probably better.
Speaker 1: I don't have a dog in that fight. It is
Speaker 1: interesting how both Google and Apple had these We're just
Speaker 1: throwing off crazy amounts of money, right each for their
Speaker 1: own special reasons, but we're reinvesting it in an interesting
Speaker 1: way that like x Ante, I would not have guessed. Right,
Speaker 1: Like this sort of search giant and the fancy phone
Speaker 1: maker becoming kind of going to the vanguard of mapping
Speaker 1: is not obvious.
Speaker 2: So I think when Steve Jobs added Google Maps to
Speaker 2: the iPhone very very quickly. He realized it was strategic
Speaker 2: and you know, he knew that because there was a
Speaker 2: GPS on the phone and because he now had a
Speaker 2: you know, pan and zoom capable touch screen, that he
Speaker 2: had a you know, an amazing new method for helping
Speaker 2: people understand the world.
Speaker 3: And so getting that map data on there was very important.
Speaker 2: And Android and iOS both had it very quickly. But
Speaker 2: the moment you do that, you have the same problem
Speaker 2: with street view showed was that people walk around, they
Speaker 2: look at their map and they look up at the
Speaker 2: street around them, and it's wrong. And so they knew that,
Speaker 2: you know, the biggest complaint they would be getting was
Speaker 2: this factual failure from their users about their maps.
Speaker 1: Yeah, well a map that's wrong is worse than no map, arguably.
Speaker 3: Yes.
Speaker 1: I mean, were there surprises to you about, you know,
Speaker 1: sort of what what developed, what didn't developed, how people
Speaker 1: used Google Maps, things you thought might happen that didn't happen.
Speaker 2: I think one of the things that you know, we
Speaker 2: probably speculated about. But if you realized that Google Earth
Speaker 2: and Maps launched in February and June of two thousand
Speaker 2: and five.
Speaker 3: In August, Katrina happened, and.
Speaker 1: Katrina was Hurricane Katrina in New Orleans.
Speaker 2: Yeah, yeah, it was an incredible hurricane, did huge amounts
Speaker 2: of damage and the city of New Orleans was flooded,
Speaker 2: you know, completely, and this turned into a challenge for
Speaker 2: you know, everybody involved, you know, both for just escaping
Speaker 2: the city and going to the right places, but then
Speaker 2: it became about rescuing people on rooftops. And the problem
Speaker 2: now was the streets were gone and they couldn't figure
Speaker 2: out where things were. And so it turned out that
Speaker 2: the helicopter pilots at the time used Google Earth to
Speaker 2: find out where the address was on the picture and
Speaker 2: then use the picture to navigate their helicopter to find
Speaker 2: the people waiting on the roof. And there were literally
Speaker 2: hundreds of rescues done using Google Earth's bipilots, you know,
Speaker 2: a few months after we'd released it, and that was
Speaker 2: that was a big surprise for us.
Speaker 1: It is extremely interesting to me that the company right
Speaker 1: now spun out of the company that made Pokemon Go,
Speaker 1: and that there was this kind of kernel of an
Speaker 1: idea which was like, oh, hey, people you have playing
Speaker 1: Pokemon Go have taken a billion photos in the world,
Speaker 1: like truly a billion photos. This is an interesting data set.
Speaker 1: Maybe we can do something with it. I mean, if
Speaker 1: I understand right, that was before you got there, But like,
Speaker 1: how did that playoff? Did they know when they were
Speaker 1: launching Pokemon Go that like, oh, we're going to get
Speaker 1: this data? Did somebody realize it?
Speaker 2: So interestingly, I think the uh, you know, John John
Speaker 2: Hankey definitely had a very long term vision. He was
Speaker 2: thinking about not just the game itself, but about games
Speaker 2: in the real world. Now, the photos that were taken
Speaker 2: initially by Pokemon Go players were you know, taking a
Speaker 2: picture of a Pokemon into location. You know that those
Speaker 2: photos stayed private. What happened was that John, seeing the future, said,
Speaker 2: can we create a game mode where players consciously go
Speaker 2: out and record videos for us of poke stops and
Speaker 2: help us map these locations by recording videos. So it
Speaker 2: was a conscious gameplay activity where you would get in
Speaker 2: game rewards if you would upload a thirty second video
Speaker 2: to the to the game.
Speaker 3: And that was where that data came from.
Speaker 2: So it was consciously contributed by the by the players
Speaker 2: rather than sort of background collected.
Speaker 1: And and what was what was the point of that
Speaker 1: From the point of view of the company, so that
Speaker 1: we wanted to create more ar gaming experiences.
Speaker 2: And the Pokemon Go already used augmented reality to create
Speaker 2: those photos that you were talking about. Put a Pokemon
Speaker 2: sitting on a chair and take a picture of it,
Speaker 2: or so it was already using augmented reality. What we
Speaker 2: wanted to do is to be able to augment the
Speaker 2: poke stops in the outdoors, and so to do that accurately,
Speaker 2: you actually need to have a very accurate map of
Speaker 2: those pokey stops, not just where they are with a
Speaker 2: blue dot, but actually the three D model of the
Speaker 2: benches around it and the statue itself, and so creating
Speaker 2: that three D model and then being able to point
Speaker 2: your phone at it and say, I know exactly where
Speaker 2: you're standing relative to that statue. Now the Pokemon that
Speaker 2: was placed by the last person is exactly the same
Speaker 2: place that it is for you.
Speaker 1: So you need a really sophisticated model of the physical
Speaker 1: world for that to work, otherwise it'll be like floating
Speaker 1: in space a foot from where it's supposed to be.
Speaker 2: That it is exactly the problem we spent several years solving,
Speaker 2: and we only launched the product that used it. I
Speaker 2: think in twenty twenty three, and that was part of
Speaker 2: Pokemon Go and it's called Pokemon Go playgrounds, and it's
Speaker 2: the ability for players to leave Pokemon at locations in
Speaker 2: interesting formations and interesting combinations and then have other players
Speaker 2: discover them or add to them.
Speaker 1: And at what point did somebody think, oh, we can
Speaker 1: do something wholly other than gaming here, we can build
Speaker 1: a new kind of map that is, you know, more
Speaker 1: legible to a robot.
Speaker 2: I think that in some people's minds, you know, that
Speaker 2: was the thought from day one. I think John would
Speaker 2: argue in his mind he was he was really thinking
Speaker 2: in that direction. But I think, you know, from a
Speaker 2: product perspective, in a business perspective, being able to allow
Speaker 2: developers of applications to go scan their own places, not
Speaker 2: Poke stops, but any location around the world and then
Speaker 2: create these experiences. That was our first step. I think
Speaker 2: the robotics ass inspect came into play, you know, more recently,
Speaker 2: probably in the last eighteen to twenty four months, as
Speaker 2: we realize that, you know, robots struggle with finding themselves
Speaker 2: in the real world, and so they need a form
Speaker 2: of maps, don't we all.
Speaker 1: And we'll be back in just a minute. So let's
Speaker 1: talk about where you are now, like what's in the
Speaker 1: world now that you're making.
Speaker 2: I think the uh, you know, the experience of working
Speaker 2: with the data from the phones of the players taught
Speaker 2: us a lot about how to reconstruct and create these
Speaker 2: precise maps, not with super fancy survey tools but with
Speaker 2: just consumer devices.
Speaker 3: And what we got good at was using.
Speaker 2: Low quality data, which I mean, it's still very good,
Speaker 2: but it's you know, it's not professional great anything, and
Speaker 2: being able to turn that into professional grade data. And
Speaker 2: so I think that was the skill that we're bringing
Speaker 2: into the enterprise applications that you hear now.
Speaker 3: And we can do it not just at the scale
Speaker 3: of a.
Speaker 2: Statue or a single location, but we can do it
Speaker 2: at city scale. We can collect an entire city and
Speaker 2: help you find yourself.
Speaker 1: Have you collected an entire city?
Speaker 3: Yes?
Speaker 1: What city?
Speaker 2: We have all of San Francisco and pieces of other cities,
Speaker 2: and they're usually collected from drone data. Basically commissioning and
Speaker 2: saying we wanted a drone based camera, you know, follow
Speaker 2: a path, take a picture every several meters and take
Speaker 2: it from multiple angles, and then use photogrammetry to reconstruct
Speaker 2: that city at a level of detail, you know, beyond
Speaker 2: what Google can do it.
Speaker 1: And when you say drone, you mean like a flying drone,
Speaker 1: like a little quad copter what exactly.
Speaker 2: A very lightweight, you know, very limited under two hundred
Speaker 2: and fifty grams smallest ones.
Speaker 1: And so what are you doing now with your three
Speaker 1: D robot legible map of San Francisco.
Speaker 2: We are combining all of that interesting data collected on
Speaker 2: the ground with phones and with three sixty cameras with
Speaker 2: this drone data and also with satellite data and proving
Speaker 2: out that we can help robots and humans find themselves,
Speaker 2: localize themselves, you know, anywhere in the city without having
Speaker 2: to pre map it, you know, specifically at that location.
Speaker 1: So you mean like you a robot, It's just like
Speaker 1: you blindfold the robot for lack of a better word,
Speaker 1: and then take the blindfold off, and it knows right
Speaker 1: away where it is and which way it's looking.
Speaker 3: That's absolutely the goal.
Speaker 1: Yes, what do you have to do to get there?
Speaker 2: I think the challenge is that you know, the world
Speaker 2: is a complicated and varied place, and you know, trees.
Speaker 2: Trees are my nemesis in my career, and you know
Speaker 2: they are they are absence.
Speaker 1: I'm proachry. For the record, I'm approached.
Speaker 2: I understand they are visually amazing, but they are in
Speaker 2: fact fractally based. Right they have very high detail, and
Speaker 2: worse than that, they're leaves. They flutter in the wind.
Speaker 2: It's deeply inconvenient, and so when you take multiple pictures
Speaker 2: of a tree, no two pictures have the tree in
Speaker 2: the same spot.
Speaker 1: When they change rapidly with tom.
Speaker 2: Absolutely so another problem. The old way of doing this
Speaker 2: was that, you know, satellites and airplanes would take pictures
Speaker 2: of cities in the winter because the leaves were off,
Speaker 2: and that gave them more information. But if you're trying
Speaker 2: to build a model the city and localize, you actually
Speaker 2: need to see them in all their states, not just
Speaker 2: the leaf off state.
Speaker 1: So is the short answer to how you solve that problem? AI?
Speaker 2: How do you solve the tree problem? You try to
Speaker 2: look beyond the trees. You know, trees other than the
Speaker 2: tree trunk. There's not a lot of stability there, but
Speaker 2: there are many things behind the tree or next to
Speaker 2: the tree that are useful and being able to effectively
Speaker 2: in an AI. To ignore the tree is usually your
Speaker 2: best bet because there's enough remaining features in view that
Speaker 2: you can find enough to connect yourself to that location.
Speaker 1: I know you just made a deal with a delivery
Speaker 1: robot company. Like, is that the first use case? Like,
Speaker 1: what do you think the first use cases are going
Speaker 1: to be?
Speaker 2: I mean, definitely is the first use case they struggle with,
Speaker 2: effectively the blindfold problem that sometimes they don't don't have
Speaker 2: enough context to know where they are, and they need
Speaker 2: to rediscover their location, and they need to need do
Speaker 2: so accurately enough that either the autonomous controller or some
Speaker 2: human who drops in to look at where it is
Speaker 2: can know where they are in the city because city
Speaker 2: blocks tend to look the same.
Speaker 3: Unless you have that information.
Speaker 1: How does the robot get lost in the first place? Like,
Speaker 1: how does the robot find itself in that situation?
Speaker 3: There's two ways.
Speaker 2: One is I mean they do it they hit a reboot,
Speaker 2: and they literally lose connection if they are cover if
Speaker 2: for any reason enough of the world is blocked from
Speaker 2: them for long enough, they lose lose control. GPS by
Speaker 2: itself is rarely good enough because, especially in cities, the
Speaker 2: GPS has you know, reflections off of buildings, and you
Speaker 2: can be a city block away and the GPS will
Speaker 2: happily tell you bigger there so.
Speaker 1: Hu So it's insufficiently precise, particularly in cities.
Speaker 2: Yes, and big cities with big, tall buildings are absolutely
Speaker 2: the hardest challenge other than tunnels and parking critches.
Speaker 1: But that's good for you. Like that that weakness is
Speaker 1: your opportunity exactly. What tell me about the broader industry
Speaker 1: is the right word, but the broader context, you know,
Speaker 1: who else is working on things like you're working on.
Speaker 2: I mean, I think we've had, you know, companies that
Speaker 2: have worked on building digital twins of the world. They're
Speaker 2: in the micro or the macro for you know, since
Speaker 2: before Google, and Google certainly is the biggest wholesale attempt
Speaker 2: at this, but there's a need for more resolution and
Speaker 2: also to have it from suppliers other than Google, because
Speaker 2: one of the areas that I think is hard is
Speaker 2: that Google is all about recreating the public world, but
Speaker 2: they don't really have any way for a company to
Speaker 2: do their own location and add it to the real world.
Speaker 1: Oh interesting, the hospital did I read hospital as an example.
Speaker 1: We were talking about that when we're doing the prep
Speaker 1: of like how hard it is to find your way
Speaker 1: around a hospital?
Speaker 3: It is, and I just experienced that yesterday at Stanford.
Speaker 3: I think the.
Speaker 2: Challenges building a map and with the hospital, there's actually
Speaker 2: a lot of change that happens, you know, in the
Speaker 2: corridors of a hospital at any given time. The big
Speaker 2: quarriors are stable, but most of the actual patient areas
Speaker 2: are you know, constantly changeing their visual appearance, and so
Speaker 2: getting robots that are able to navigate that space accurately
Speaker 2: and reliably is certainly one of our potential use cases.
Speaker 1: So there's an app called scan a Verse that's you right,
Speaker 1: the app scan a Verse. So I downloaded that this
Speaker 1: week and astonishingly, they're estonishingly to me. Within one hundred
Speaker 1: yards of my house, there are two things, two objects
Speaker 1: that have been scanned in three D and uploaded to
Speaker 1: this app, which is interesting cool. I had no idea
Speaker 1: what's going on there? What is it? And why are
Speaker 1: people on my block taking pictures of things.
Speaker 2: Scan Versus is a great app, and we actually acquired
Speaker 2: the one person.
Speaker 3: Who wrote it, a brilliant programmer.
Speaker 2: And turned it into this product that allows capture of
Speaker 2: three D objects. And what you're seeing there is that
Speaker 2: a lot of users love to be able to go
Speaker 2: out and scan their area their location or a statue
Speaker 2: or whatever, and recreated in three D visually almost completely
Speaker 2: photo realistically, and then share that with the rest of
Speaker 2: the world. And so you're experiencing sort of the map
Speaker 2: that scanners puts up at the front of the front
Speaker 2: of them.
Speaker 1: So it's in the same way that people like to
Speaker 1: do whatever geokashig and geo guessing and whatever. It's some
Speaker 1: version of kind of sharing the real world in a
Speaker 1: digital way.
Speaker 3: Exactly is it big?
Speaker 1: I infer from the fact that they're too on my
Speaker 1: block or one on my block and one around the
Speaker 1: corner that lots of people are using it? Is that true?
Speaker 1: Do I live in a weirdly dense area?
Speaker 2: There are a lot of people using We have hundreds
Speaker 2: of thousands of users and they are passionate about this ability,
Speaker 2: and we have used we've recently, very recently added the
Speaker 2: enterprise capabilities to that app. People want to create a
Speaker 2: digital twin of a tiny little place, they can do
Speaker 2: it on their phone and upload it like you saw.
Speaker 2: If they want to recreate their factor or their business,
Speaker 2: you know, hundreds of square meters, they can do that
Speaker 2: with our enterprise version and then allow for modeling and
Speaker 2: allow for robotic training within their site using that tool.
Speaker 1: So you can use your phone and walk around your factory,
Speaker 1: say you're a little, you know, small factory or your
Speaker 1: construction site or whatever, and the three D rendering it
Speaker 1: generates is like robot legible. Then you can train your
Speaker 1: robot on the thing you made with your iPhone.
Speaker 3: That's correct.
Speaker 1: So there's this phrase you hear an AI world model, right,
Speaker 1: And it's sort of kind of a complement to a
Speaker 1: language model.
Speaker 3: Right.
Speaker 1: Instead of AI that's trained on language, it's AI in
Speaker 1: some way that's trained on the physical world. Like, tell
Speaker 1: me about that and how your work fits in with that.
Speaker 2: So the world models have been mostly focused to date
Speaker 2: on solving a sort of very specific problem, which is
Speaker 2: how to translate video, which they're mostly programmed with, into
Speaker 2: understanding of the world that's good enough that it can
Speaker 2: predict the future.
Speaker 1: Yeah, and when you say predict the future, you don't
Speaker 1: mean any like weird oracle thing, right, you mean like
Speaker 1: predict the future as well as a human can that. Like,
Speaker 1: if a fork starts to fall off the table, it's
Speaker 1: going to fall all the way to.
Speaker 3: The grounds, that's exactly right.
Speaker 2: So if something is initiated, how would it play out
Speaker 2: with the objects. And if you've seen enough forks fall
Speaker 2: and videos, if you've seen enough objects fall, then you
Speaker 2: can start to predict it.
Speaker 1: And we were so used to now AI being so
Speaker 1: good at language, it's easy to forget that it has
Speaker 1: no understanding of the physical world. I mean except in language.
Speaker 3: That's right, so weird. Yeah, it can, it can think
Speaker 3: in its brain.
Speaker 2: I know what the rules of Newton's law are, but
Speaker 2: you know, turning that into pixels on the screen is
Speaker 2: actually pretty hard. But if you train enough video models
Speaker 2: on it, then then it can figure it out. And
Speaker 2: so the I think the connecting the language model to
Speaker 2: the physical world is in some ways. There's two ways
Speaker 2: to do it, and one is to have the model
Speaker 2: understand the physics video. The other is to turn the
Speaker 2: world itself into words or into meaning. And this is
Speaker 2: you know, typically called semantics. And there's all sorts of
Speaker 2: you know, technologies that try to label information about a picture.
Speaker 2: And if you train enough of models on the labeling
Speaker 2: of the pictures, you can then label any but any
Speaker 2: picture of the real world.
Speaker 1: And this is kind of the first big breakthrough of
Speaker 1: neural networks, right, I mean, it sounds like.
Speaker 3: That cats, cats and dogs, cats and dogs.
Speaker 1: Yeah, and image net right.
Speaker 2: So if you give the information about the embeddings, it's
Speaker 2: it's called and you do that not just in cat
Speaker 2: and dog dimension, but you do it in hundreds of
Speaker 2: dimensions of objects. You can give these language models a
Speaker 2: the equivalent of a language understanding of the real world,
Speaker 2: and then they can do amazing things with it.
Speaker 1: Huh. And so how does all that relate to the
Speaker 1: work you're doing.
Speaker 2: We believe that to do that successfully you need to
Speaker 2: start with three D data because the ability to label
Speaker 2: something in three dimensions is much more accurate and with
Speaker 2: and argue with much more information than if all you
Speaker 2: have is a two D picture of it. Because in
Speaker 2: two D pictures there's always, uh, you know, obscuration of data,
Speaker 2: and things are behind things and you can't see the
Speaker 2: whole picture. But in three D you eventually see everything
Speaker 2: that is seeable.
Speaker 1: I mean is that the fun not to be reductive,
Speaker 1: but kind of to be reductive, is that the fundamental
Speaker 1: thing you were doing that is novel is making a
Speaker 1: three dimensional map and world where maps have always been
Speaker 1: two dimensions.
Speaker 2: Before I would say yes, I'm there, there are there
Speaker 2: are three D visual maps. There are you know, rare
Speaker 2: one off three maps, but trying to do a fully
Speaker 2: language embedded three D semantic map of the world so
Speaker 2: that a language, models and robots and others can understand
Speaker 2: it is what we're working on.
Speaker 1: So for zoom out like, well, two things, I guess
Speaker 1: there's two things we'll do, the sad one and the
Speaker 1: happy one. Like what might go wrong for you? Like
Speaker 1: what has to go Yeah, what might go wrong for
Speaker 1: you over the next years. It seems like you're at
Speaker 1: this moment when you have this interesting technology, you're sort
Speaker 1: of proving it out. It's not widely used yet. What
Speaker 1: are the sort of pitfalls?
Speaker 2: I mean, it was hard hard mapping the world in
Speaker 2: two D for humans. Mapping the world in three D
Speaker 2: four robots requires a level of detail and precision that
Speaker 2: nobody's ever done before. And the only way to get
Speaker 2: it done is to apply AI to the problem, to
Speaker 2: figure out how to make it more accurate, more quickly
Speaker 2: and more cheaply than has been done in the past, because,
Speaker 2: you know, carefully reconstructing the world, every square meter of it.
Speaker 2: The old way is you know, billions, if not trillion dollars,
Speaker 2: and this can't be that nobody's going to pay that.
Speaker 1: Yeah, billions maybe, but trillions definitely not. And so if
Speaker 1: you if you get it right, like give me the
Speaker 1: like the big exciting outcome, you know what I mean,
Speaker 1: like if it goes, well, what's the world look like?
Speaker 1: How is the world better?
Speaker 2: I think that if you know, as we're seeing, the
Speaker 2: ais have been able to dig into human knowledge and
Speaker 2: you know ingest all of the text of all of
Speaker 2: our scientific papers and all of our blog posts and
Speaker 2: all of the Instagram and they've synthesized this in many
Speaker 2: cases into some very interesting information that answers at least
Speaker 2: you know, personally relevant questions today. And if you talk
Speaker 2: to some mathematicians, you know they're they're they're doing pretty
Speaker 2: well in math as well. But the questions about the
Speaker 2: real world are are are simply not there yet. And
Speaker 2: but the the advantage of an AI is it doesn't
Speaker 2: stop and it doesn't stumble on too much data, and
Speaker 2: if you can give it enough information about the world,
Speaker 2: it can start to answer like really interesting questions about
Speaker 2: the city, about how the city works, about which intersections
Speaker 2: are you know too busy, and about you know, you know,
Speaker 2: flow of people and you know I think things that
Speaker 2: things that could make the city more efficient because you
Speaker 2: have a more complete vision of it, you know, especially
Speaker 2: in you know, three dimensions. You know, the two D
Speaker 2: maps for a street are fine, but the moment you
Speaker 2: go down to the subways in New York, some of
Speaker 2: those are you know, three four levels deep, and some
Speaker 2: of them are extremely popular in others are ghost towns.
Speaker 2: And and you know, I think there's opportunities to improve
Speaker 2: the infrastructure that we already have built without having to
Speaker 2: like rebuild it from scrap.
Speaker 1: We'll be back in a minute with the lightning round.
Speaker 1: We're gonna finish with the lightning round. Tell me about
Speaker 1: the Meadowbrook apartments in Lawrence, Kansas.
Speaker 2: I grew up in Meadowbrook. It was a fine apartment complex.
Speaker 2: I moved there when I was four and moved out
Speaker 2: when I was eighteen. And so the center of Google
Speaker 2: Earth on the windows PC in particular, will zoom you
Speaker 2: into the bedroom of the apartment of my apartment building.
Speaker 1: Still still yeah, that's a robust easter egg.
Speaker 2: And there's another one in that category, which is that
Speaker 2: I a friend of mine who I went to the
Speaker 2: University of Kansas with, I hired him in two thousand
Speaker 2: and five to port Google Earth to the Macintosh, and
Speaker 2: he secretly changed the center of Google Earth on the
Speaker 2: mac to be Shanook, Kansas, where he's from. But he
Speaker 2: did it to the mains main intersection there in Chanut
Speaker 2: and the city is she who loved it so much.
Speaker 2: They actually built a mural on the street at that location.
Speaker 2: So if you ever run a mac give it a
Speaker 2: try and you'll see what it says.
Speaker 1: What's one tip for becoming a pac Man champion?
Speaker 3: The old old machines have patterns.
Speaker 2: And when I was, you know, the very first pac
Speaker 2: Man machine, if you knew the five key pattern and above,
Speaker 2: you know I was able to break a million. They're
Speaker 2: really professional people. I think the high score of the
Speaker 2: Infinite Perfect Score is three point three million, but I
Speaker 2: was able to get a million. But these days, you know,
Speaker 2: I still play Miss pac Man at home. I have
Speaker 2: a home machine, and you know, it's really about reaction,
Speaker 2: and you know it's if you play it on fast mode,
Speaker 2: it's all about speed.
Speaker 1: Do I recall that Miss pac Man was more deterministic
Speaker 1: than pac Man? Is that right?
Speaker 3: It's it's the other way around.
Speaker 2: When they launched Miss pac Man, they surprised people because
Speaker 2: they had extra logic and they actually randomized it, so
Speaker 2: literally no pattern really works even from the.
Speaker 1: Oh okay, so it's a it's a kind of more
Speaker 1: elegant game maybe at some level, or you could play
Speaker 1: it for longer. Anyways, how are your skills?
Speaker 2: My my I score is turned seventy nine thousand, My
Speaker 2: friend's high score is turned in seventy five thousand and
Speaker 2: and and that's where we're at.
Speaker 1: Are you currently training to win any other video game championships? No?
Speaker 3: I I my my only my only games.
Speaker 2: I've played a lot of video games in the early
Speaker 2: eighties and some Missile Command, miss pac Man, pac Man,
Speaker 2: and Galaga.
Speaker 3: I think we're one of my main games.
Speaker 1: I liked the Big Ball on Missile Command, the Arcade
Speaker 1: Missile Command, remember had that giant ball.
Speaker 3: Almost a bowling ball, and it had had momentum.
Speaker 1: But the the only games I.
Speaker 2: Really play now are are the New York Times word games,
Speaker 2: and I still play pokem let Go.
Speaker 1: Do you have any like favorite maps from history or
Speaker 1: whatever their maps you love, or like genius cartographers.
Speaker 3: There's there's one map.
Speaker 2: There's more of a sort of a data science presentation.
Speaker 2: There was a tough tie was a guy who wrote
Speaker 2: some really great books about data visualization, and he highlighted
Speaker 2: this map which has always stuck with me.
Speaker 3: And it was a single.
Speaker 2: Sheet map that showed Napoleon's you know, march to Russia,
Speaker 2: including the size of the army as it progressed, and
Speaker 2: then it decimated itself in Russia and then shrunk and
Speaker 2: shrunk and shrunk as it returned because they were, you know,
Speaker 2: going through winter with no food and just the visual
Speaker 2: The amount of information in that map, beyond the map
Speaker 2: itself is incredible.
Speaker 1: Anything you think shouldn't be mapped.
Speaker 3: The inside of people's homes, you know, you have to
Speaker 3: have a private space.
Speaker 1: What was the last time you got lost?
Speaker 3: It's a good question.
Speaker 2: Never for very long, you know, I don't remember that.
Speaker 2: Blue dot's pretty powerful. Tell me about catalog in your life.
Speaker 2: So I started doing it in nineteen ninety three. I
Speaker 2: started recording books, I read, movies I watched, and you know,
Speaker 2: various other events in my life. And I actually backpropagated
Speaker 2: and re remembered movies I'd seen before nineteen ninety three.
Speaker 2: So I have a list of every book, every movie,
Speaker 2: as well as many other life events all in a spreadsheet,
Speaker 2: and it's you know, as I grow older, it is
Speaker 2: you know, it becomes my memory because it's not super detailed,
Speaker 2: but you know, in many cases just the event itself
Speaker 2: existing reminds me of what I was doing that year,
Speaker 2: that month, that day.
Speaker 1: What like do you look back on it, like in
Speaker 1: the way people might look at old photos or something like,
Speaker 1: how do you how do you kind of use it
Speaker 1: or engage with it?
Speaker 3: I do.
Speaker 2: I looked back at like what movies was I watching
Speaker 2: in the last semester before I graduated high school? You know,
Speaker 2: how did I spend my time that year? And was
Speaker 2: what was available? And what's going on?
Speaker 1: And you know you have qualitative notes? Do you have
Speaker 1: ranking like are there interesting columns in those?
Speaker 2: I've gone back through and I have my top forty
Speaker 2: for let's see, what do I have my top forty
Speaker 2: for books? And I have my top two hundred for movies,
Speaker 2: and I have them somewhat ranked within.
Speaker 1: That top movie, top book.
Speaker 2: Let's see. I'll still say Contact for reasons. On the movie,
Speaker 2: I just find it very compelling, And on the book,
Speaker 2: I love Scholzie and Hugh Howie, So I almost want
Speaker 2: to say Silo but I'll still say three body problem.
Speaker 1: So so both both science fiction.
Speaker 2: I am unapassionately almost entirely science fiction.
Speaker 1: It's really I guess it shouldn't be surprising, but it's
Speaker 1: particularly it's persistently notable to me how influential science fiction
Speaker 1: is in the world. Like you know, the world we
Speaker 1: live in, people built it in some significant degree because
Speaker 1: they were inspired by science fiction in really specific ways.
Speaker 2: Absolutely, Carl Sagan wrote Contact, and you know it was
Speaker 2: hugely influential that way. But Neil Stephenson wrote snow Crash
Speaker 2: and described a version of Google Earth before we built it,
Speaker 2: and both John and I had read that book and
Speaker 2: feel at least partially inspired by that. There were other
Speaker 2: things that inspired us too, but that very specific description
Speaker 2: of a visualizable, holographic three D model of the planet
Speaker 2: that you could zoom into is very powerful.
Speaker 1: Thank you so much for your time, was very interesting
Speaker 1: to talk with you.
Speaker 3: Thank you.
Speaker 1: Brian McLendon is the chief technology officer at Niantic Spatial.
Speaker 1: Our show Today was produced by Gabriel Hunter Chang. It
Speaker 1: was edited by Lydia Jean Kott and engineered by Hansdale.
Speaker 1: She I'm Jacob Goldstein and We'll be back in a
Speaker 1: couple weeks with more episodes of What's Your Problem. Thanks
Speaker 1: for listening.