WEBVTT - Google's Journey to the Edge of Search 0:00:15.316 --> 0:00:23.156 Pushkin. It feels like searching the web is a problem 0:00:23.236 --> 0:00:26.756 that's been solved. You know, it's ridiculously easy for me 0:00:26.796 --> 0:00:30.036 to say, find out when Alexander Hamilton was shot eighteen 0:00:30.076 --> 0:00:33.796 oh four, or whether they are making Sing three. Not yet, 0:00:34.156 --> 0:00:38.876 but Matthew McConaughey has expressed interest and yet. And maybe 0:00:38.876 --> 0:00:42.556 this is not surprising. The people who spend their lives 0:00:42.716 --> 0:00:46.876 working on search do not think search is solved. This 0:00:46.956 --> 0:00:50.116 is partly because the people at the frontier of search 0:00:50.596 --> 0:00:53.796 don't just want to search the web. They want to 0:00:53.796 --> 0:00:57.876 answer every question that might cross your mind, even questions 0:00:57.996 --> 0:01:06.796 you can't put into words. I'm Jacob Goldstein, and this 0:01:06.916 --> 0:01:09.636 is What's your problem? The show where on entrepreneurs and 0:01:09.676 --> 0:01:11.916 engineers talk about how they're going to change the world 0:01:12.236 --> 0:01:15.156 once they solve a few problems. My guest today is 0:01:15.316 --> 0:01:19.396 Kathy Edwards, vice president and GM of Search at Google. 0:01:20.516 --> 0:01:23.956 Cathy's problem is this, how do you teach computers to 0:01:23.956 --> 0:01:26.676 tell people what they want to know, even if they 0:01:26.716 --> 0:01:30.676 don't know how to ask. Later in the conversation we 0:01:30.756 --> 0:01:33.356 get to the frontier of what Kathy and Google are 0:01:33.396 --> 0:01:36.356 working on now, but we started with the problem they 0:01:36.396 --> 0:01:39.196 have largely solved in the six years Kathy has been 0:01:39.236 --> 0:01:42.916 at Google, the jump from search results based on keywords 0:01:43.316 --> 0:01:46.676 to search results based on natural language, the way people 0:01:46.716 --> 0:01:51.316 talk in everyday life. So one of the problems that 0:01:51.356 --> 0:01:54.556 we were working on around six years ago is this 0:01:54.716 --> 0:01:58.276 problem of natural language queries. So, if you're old enough 0:01:58.316 --> 0:02:01.756 to remember the early days of search on the Internet, 0:02:02.116 --> 0:02:05.836 there was this idea of keywordies, right, that you had 0:02:05.876 --> 0:02:09.236 to sort of take this idea you had in your 0:02:09.236 --> 0:02:11.756 mind of what you what you needed to know and 0:02:11.916 --> 0:02:15.436 figure out what were the exact right keywords to enter 0:02:15.476 --> 0:02:17.756 into the search engine to get your results back right. 0:02:17.796 --> 0:02:20.716 I mean an example as an example, very early in 0:02:20.716 --> 0:02:23.436 the you know, I remember being taught how to query 0:02:23.876 --> 0:02:28.716 back in you know, nineteen ninety nine and being told 0:02:28.796 --> 0:02:32.356 never used the word and never used the word because 0:02:32.716 --> 0:02:35.676 the word and or the word there is in almost 0:02:35.796 --> 0:02:38.516 every document on the Internet. And the way it worked 0:02:38.556 --> 0:02:41.796 back then is you did this word matching, right, and 0:02:41.836 --> 0:02:44.436 so if you had a word that was in your 0:02:44.516 --> 0:02:47.076 query and there was that same word in the document, 0:02:47.476 --> 0:02:50.876 then that document would be returned and potentially scored right. 0:02:51.276 --> 0:02:54.436 And that was very helpful if it was a word 0:02:54.476 --> 0:02:58.716 like genetics, right, which was highly specific and wasn't in 0:02:58.796 --> 0:03:00.996 a heap of documents on the internet. But the word 0:03:00.996 --> 0:03:04.196 and not very specific. And you know, in the very 0:03:04.236 --> 0:03:08.276 early days the Internet, these words weren't even weighted particularly right, 0:03:08.316 --> 0:03:11.956 The word and count for as much as the word genetics, 0:03:11.956 --> 0:03:14.716 and so a document might have a ton of the 0:03:14.796 --> 0:03:17.276 uses of the word and and one use of the 0:03:17.276 --> 0:03:20.116 word genetics, and it would score really highly, even though 0:03:20.116 --> 0:03:22.436 it wasn't particularly genetics. Folk. Now, by the time you 0:03:22.436 --> 0:03:25.556 get to Google, that part is solved, right Google buy 0:03:25.716 --> 0:03:29.156 that part is years ago? Is waiting genetics more more 0:03:29.196 --> 0:03:33.556 heavily than it's waiting the But but what what part 0:03:33.636 --> 0:03:36.516 six years ago was not solved? That's solved now or 0:03:36.556 --> 0:03:40.436 solved ish now. But we were still seeing people do 0:03:40.596 --> 0:03:44.596 these very keyword oriented queries. So they weren't saying things 0:03:44.676 --> 0:03:48.516 like what wine pairs best with chicken? Or if they were, 0:03:48.876 --> 0:03:53.036 they were doing those queries and getting not the best results, 0:03:53.076 --> 0:03:56.116 because not only is there a question of word matching 0:03:56.116 --> 0:03:59.156 and how much each word counts for, there's also the 0:03:59.276 --> 0:04:03.556 question of like does the word what appear at all? Right? 0:04:03.676 --> 0:04:07.276 Like are the answers to that question actually just documents 0:04:07.356 --> 0:04:09.716 that talk about the best wine to pair with chicken 0:04:10.076 --> 0:04:16.716 is you know, chardonnay, right, and not so much talking. 0:04:16.916 --> 0:04:18.596 You know, they didn't include the question, and so we 0:04:18.676 --> 0:04:23.756 sort of saw these like SEO documents that would spring 0:04:23.876 --> 0:04:26.236 up that would have the questions kind of baked in 0:04:26.276 --> 0:04:29.116 and an attempt to match. But those documents weren't necessarily 0:04:29.156 --> 0:04:33.436 the best answers. And so this is when we started 0:04:33.436 --> 0:04:36.356 to go just that next level deeper in our language 0:04:36.476 --> 0:04:41.956 understanding with these AI models, their language models that really 0:04:42.036 --> 0:04:46.196 can start to map out in a concept space, things 0:04:46.316 --> 0:04:49.356 like this sort of translation between how you might ask 0:04:49.396 --> 0:04:51.356 a query and then what that might look like in 0:04:51.396 --> 0:04:53.916 the document. So, to take the example you gave of 0:04:54.196 --> 0:04:57.916 what wine pairs best with chicken, even as late as 0:04:57.956 --> 0:04:59.956 six years ago when you got to Google, you're saying 0:05:00.636 --> 0:05:05.316 Google wasn't great at delivering the best results to a 0:05:05.396 --> 0:05:08.956 query like that because it was written as speech, not 0:05:09.156 --> 0:05:11.396 written as a series of keywords. So six years ago, 0:05:11.516 --> 0:05:16.596 I would have been better off typing chicken wine pairing. 0:05:17.356 --> 0:05:19.116 I would have got better results if I did that, 0:05:19.156 --> 0:05:21.636 you're saying, because that's kind of the way. That's the 0:05:21.636 --> 0:05:23.476 way Google had mapped the web. It was like a 0:05:23.516 --> 0:05:26.076 series of important words and what sites are reliable and 0:05:26.116 --> 0:05:29.876 they it just the technology wasn't there to actually try 0:05:29.876 --> 0:05:35.316 and understand the way people ask questions in real life. Absolutely, 0:05:35.716 --> 0:05:40.636 And it was this idea of bringing AI into search 0:05:41.036 --> 0:05:44.596 and having these like large scale language models. That first 0:05:44.676 --> 0:05:48.036 one was called Bert. We now use one called mom, 0:05:48.076 --> 0:05:50.796 which is get to mom. But let's talk about Let's 0:05:50.796 --> 0:05:53.836 try talking about Bert for a second. So how do 0:05:53.916 --> 0:05:58.996 you get from search results that are fundamentally keyword based 0:05:59.476 --> 0:06:03.636 to search results that are fundamentally you know, answering questions 0:06:03.676 --> 0:06:06.396 that are posed in a more natural way, like how 0:06:06.396 --> 0:06:11.076 do you make that leap? So the fundamental insight is 0:06:11.476 --> 0:06:14.236 you go from looking at these words as tokens that 0:06:14.716 --> 0:06:18.556 get matched against each other to suddenly you look at 0:06:18.596 --> 0:06:21.756 all the words in all the documents on the Internet 0:06:22.076 --> 0:06:25.156 and you create what's code an embedding space, which is 0:06:25.276 --> 0:06:27.436 essentially you can think of it as a map of 0:06:27.476 --> 0:06:32.676 the concepts that these documents know about. And suddenly, by 0:06:32.716 --> 0:06:34.636 being able to say, okay, you can take a query, 0:06:34.796 --> 0:06:39.396 map that into this concept embedding space. You'd take these documents, 0:06:39.396 --> 0:06:42.476 map that into the content embedding space. You can start 0:06:42.516 --> 0:06:47.116 to actually match together not these words, but what people 0:06:47.236 --> 0:06:50.716 actually mean what they actually mean when they ask these questions, 0:06:50.716 --> 0:06:53.676 and what they actually mean when they write these web 0:06:53.716 --> 0:06:56.836 pages on the Internet. That seems I mean, A, it 0:06:56.836 --> 0:07:00.196 seems super hard, right, and B As I'm parsing that, 0:07:00.356 --> 0:07:03.916 I'm tempted to use a lot of anthropomorphic language, right, 0:07:03.956 --> 0:07:06.676 I'm tempted to say, like, you have to go from 0:07:06.716 --> 0:07:09.076 the computer just sort of having a list of words 0:07:09.316 --> 0:07:11.556 and kind of weights around those words to a computer 0:07:12.076 --> 0:07:16.316 understanding what people mean. Like, am I right to say that? 0:07:16.436 --> 0:07:19.196 Or is that just my like layperson intuition getting in 0:07:19.196 --> 0:07:22.716 the way of what's going on? I Mean, the first 0:07:22.716 --> 0:07:25.836 thing I'll say is I think we're very far away 0:07:25.876 --> 0:07:29.996 from the computer having any sort of sentience and truly understanding. 0:07:30.356 --> 0:07:32.396 But I think it is true. It is fair to 0:07:32.476 --> 0:07:37.236 say that there is a level of deeper understanding that 0:07:37.276 --> 0:07:40.956 you're not just looking at these words as as you know, 0:07:41.036 --> 0:07:44.076 bits in a computer, but you're actually starting to model 0:07:44.236 --> 0:07:46.676 in a way that a human might, a brain might 0:07:46.756 --> 0:07:50.956 model what the concepts are. And I do think that's 0:07:50.956 --> 0:07:53.636 a first step of getting closer to this sort of 0:07:53.796 --> 0:07:59.356 natural human understanding. So is there a way to talk 0:07:59.396 --> 0:08:04.316 about how that works? It's it's pattern matching effectively right, 0:08:04.356 --> 0:08:07.916 And it just so happens that if you magnify pattern 0:08:07.956 --> 0:08:10.316 matching on a very large scale, that can be a 0:08:10.476 --> 0:08:15.396 pretty compelling understanding. And so that's the sort of big idea, 0:08:15.636 --> 0:08:19.036 the theory of how it works. I'm sure in actually 0:08:19.236 --> 0:08:23.116 building the thing in building Burt, which was this big 0:08:23.636 --> 0:08:28.716 model that did work, it wasn't that easy, right, I mean, 0:08:28.836 --> 0:08:33.076 is there a is there a story version of how 0:08:33.116 --> 0:08:36.276 you built it? So I think there were two hard 0:08:36.316 --> 0:08:39.796 points along the journey. The first hard point was just 0:08:40.996 --> 0:08:43.956 these models were being built at a scale that was 0:08:44.076 --> 0:08:48.516 unprecedented the amount of information. You know, traditional neural networks 0:08:48.556 --> 0:08:53.756 would run on thousands, maybe millions of training examples. Suddenly 0:08:53.796 --> 0:08:58.076 you're trying to model all the words on the Internet 0:08:58.116 --> 0:09:01.956 and this scale. Firstly, this scale is what gets you 0:09:02.036 --> 0:09:04.756 the amount of training to actually get the concepts model 0:09:04.836 --> 0:09:09.436 to be compelling. But frankly, the computers just couldn't process. 0:09:09.476 --> 0:09:12.236 So you're you're building this model and saying, okay, now 0:09:12.236 --> 0:09:15.596 to learn what you need to learn, read literally every 0:09:15.596 --> 0:09:18.476 word on the Internet, is that right, yes, and not 0:09:18.516 --> 0:09:22.076 read at once, because every layer of the neuronet needs 0:09:22.116 --> 0:09:25.236 to read it and reprocess it. Right. So you're reading 0:09:25.396 --> 0:09:29.196 every word, you know, a massive number of times. And 0:09:29.236 --> 0:09:31.956 at the time we didn't really have the compute power. 0:09:32.236 --> 0:09:36.836 You just needed more more computers, essentially more and more chips, 0:09:37.036 --> 0:09:40.756 more more engines to just process and process and process. 0:09:41.996 --> 0:09:47.236 So our research team had developed these these chips that 0:09:47.836 --> 0:09:51.516 these processes that were really optimized for doing a sort 0:09:51.556 --> 0:09:55.036 of deep learning work. And it was that these chips 0:09:55.076 --> 0:09:56.876 and the way that we could sort of put all 0:09:56.916 --> 0:09:59.236 the chips together at a work in concert to solve 0:09:59.316 --> 0:10:03.356 this problem that really unlocked the amount of processing power 0:10:03.356 --> 0:10:05.756 and needed to even build these models in the festival. 0:10:05.836 --> 0:10:09.196 So the binding constraint wasn't like the theory or the 0:10:09.236 --> 0:10:10.916 ideas of it, like you knew how to do it, 0:10:10.956 --> 0:10:14.956 you just didn't have enough enough horsepower to actually make 0:10:14.996 --> 0:10:18.396 it happen. Well, we knew that we could do it, 0:10:18.436 --> 0:10:21.356 we didn't know offer to be any good, right, it 0:10:20.916 --> 0:10:24.676 wasn't it, and you couldn't even try, right, yeah? Right? 0:10:25.236 --> 0:10:28.636 And so then we tried it and we found out, actually, 0:10:28.676 --> 0:10:32.476 this thing is pretty compelling. It can understand things that 0:10:32.516 --> 0:10:36.676 our models previously have never understood. You know. But I 0:10:36.716 --> 0:10:39.716 will say the second and this gets to the second 0:10:39.796 --> 0:10:44.956 hard part. We once we had these large scale language models, 0:10:44.956 --> 0:10:49.436 we didn't quite know how to put them into search ranking. 0:10:49.676 --> 0:10:52.876 This was not something that had been done before. So 0:10:52.956 --> 0:10:57.796 we have in search this incredibly rigorous methodology for testing 0:10:57.956 --> 0:11:01.676 any given change to our algorithm, and it's it's based 0:11:01.676 --> 0:11:05.556 in statistics, and it's statistically samples queries, and we look 0:11:05.556 --> 0:11:08.556 at the before and they after, and there's a scoring 0:11:08.596 --> 0:11:11.716 system to say is it better or not? And I 0:11:11.796 --> 0:11:17.756 remember looking at the early experiments from this burst integration 0:11:17.876 --> 0:11:22.236 into our search engine, and the queries that it was 0:11:22.596 --> 0:11:27.916 impacting were just queries that, honestly before we would have said, 0:11:28.396 --> 0:11:30.796 we don't know how we can solve this query. And 0:11:30.956 --> 0:11:35.796 suddenly the model was just able to figure out these 0:11:35.836 --> 0:11:41.596 sort of unspoken concepts that just our previous technology just 0:11:41.716 --> 0:11:43.956 would not have even been able to come close to. 0:11:44.516 --> 0:11:46.356 Like give me an example, like what kind of thing? 0:11:47.836 --> 0:11:50.956 So here's a really great example. This is directly from 0:11:51.556 --> 0:11:56.956 the one of the very first bit evaluations that we 0:11:56.996 --> 0:12:01.796 did internally, and the query is can you get medicine 0:12:01.876 --> 0:12:05.836 for someone? Pharmacy? Right? And so what's interesting about this 0:12:05.916 --> 0:12:09.876 question is the users looking for something very specific. They're 0:12:09.916 --> 0:12:13.356 looking like maybe my partner is sick. Can I go 0:12:13.356 --> 0:12:16.156 and pick up their prescription at the pharmacy for them? 0:12:16.316 --> 0:12:18.516 Or do they have to go and get it? Right? 0:12:18.676 --> 0:12:22.636 It's also a goodly jankie where it's half in natural 0:12:22.716 --> 0:12:25.676 language can you get medicine for someone? And half in 0:12:25.756 --> 0:12:28.676 like keyword ease, they're just typing pharmacy at the end, right, 0:12:28.716 --> 0:12:35.156 it's a weird exactly yeah. And so previously we didn't 0:12:35.196 --> 0:12:38.676 know how to pause out this intent right, this idea. 0:12:38.796 --> 0:12:42.156 You know, we could tell that it was about getting 0:12:42.156 --> 0:12:45.916 a prescription from a pharmacy, but this notion of force 0:12:46.116 --> 0:12:53.236 someone was a concept that was just slightly too complex. Oh, 0:12:53.556 --> 0:12:55.636 I didn't even understand it until now. What they mean 0:12:55.756 --> 0:12:58.596 is can I pick up someone else's prescription? That's what 0:12:58.636 --> 0:13:02.076 they're actually asking, But it's very it's poorly worded, frankly, 0:13:02.116 --> 0:13:06.876 and they'refore hard to figure out exactly right. And so previously, 0:13:06.956 --> 0:13:11.876 before Bert, we would turn these wonderful web pages saying 0:13:11.956 --> 0:13:14.996 this is how you get a prescription filled, which you 0:13:15.036 --> 0:13:19.116 can imagine if you're this user doing this query, you're like, yeah, 0:13:19.156 --> 0:13:21.316 I already know how to get a prescription that filled. 0:13:21.436 --> 0:13:24.956 Thanks for me. What I need is filled for somebody 0:13:24.956 --> 0:13:29.396 else exactly, and with Bert we were able to understand 0:13:29.476 --> 0:13:32.556 pick up this idea of the force someone and put 0:13:32.596 --> 0:13:35.396 the appropriate weight on it, that that was the sort 0:13:35.396 --> 0:13:38.956 of you know, discriminating thing in the query, that that 0:13:39.036 --> 0:13:41.356 was the key thing that the query turned on. And 0:13:42.196 --> 0:13:46.636 then we were able to show this this web page 0:13:46.676 --> 0:13:48.556 that talked about can I have a friend or family 0:13:48.596 --> 0:13:51.436 member pick up a prescription for me? And that was 0:13:51.476 --> 0:13:54.196 the sort of like aha moment where we could all 0:13:54.236 --> 0:13:56.316 just sit around and be like, Wow, this is a 0:13:56.436 --> 0:13:59.876 new level of understanding that we haven't got to previously. 0:14:02.516 --> 0:14:05.156 So with birth, Google got to the point where it 0:14:05.236 --> 0:14:08.996 was very very good at dealing with words in a deep, 0:14:09.236 --> 0:14:12.316 complex way. But words make up less and less of 0:14:12.356 --> 0:14:16.156 the Internet. Pictures and videos are a whole other story 0:14:16.716 --> 0:14:25.916 that's coming up in a minute. Now back to the show. 0:14:26.396 --> 0:14:29.156 So you have got to this place now where you 0:14:28.396 --> 0:14:32.876 have you Google have made the leap from keyword based 0:14:32.876 --> 0:14:35.876 searches to intention based search is what do people mean? Right? 0:14:35.876 --> 0:14:39.436 Which is this big interesting leap? And so I'm interested 0:14:39.476 --> 0:14:42.436 in kind of the next leap, like what's the next 0:14:42.636 --> 0:14:47.116 big hard problem you're trying to solve? What's really interesting 0:14:47.196 --> 0:14:50.876 to me is this idea of how many questions you 0:14:50.916 --> 0:14:54.556 don't ask because you don't even know the words, right, 0:14:54.996 --> 0:14:58.316 Like this is a bit of a sad story. But 0:14:58.436 --> 0:15:01.676 I have at my house this oak tree, and the 0:15:01.716 --> 0:15:04.756 oak tree I think is dead, and it's very sad 0:15:04.756 --> 0:15:08.196 for me because a very beautiful oak tree. And what's 0:15:08.236 --> 0:15:11.556 interesting is, you know, I looked at the oak tree, 0:15:11.596 --> 0:15:14.716 I'm like, wow, those leaves are kind of brown, Like, 0:15:14.796 --> 0:15:17.276 that's not it doesn't seem right to me. I wonder 0:15:17.316 --> 0:15:20.676 if there's something's wrong with the oak tree, right, But 0:15:20.756 --> 0:15:24.676 I can't necessarily right now really articulate that to a 0:15:24.716 --> 0:15:27.596 computer this fundamental question of is this oak tree dead? 0:15:27.636 --> 0:15:30.156 And if not, what can I do to save it? Right? 0:15:30.676 --> 0:15:32.436 So what I do is I go and type in 0:15:32.476 --> 0:15:36.356 some queries, I say, you know, oak tree dead? How 0:15:36.356 --> 0:15:38.956 do I know if my oak tree is dead? You know? 0:15:39.036 --> 0:15:42.556 And I get back results. But those results aren't necessarily 0:15:42.636 --> 0:15:45.876 taking what they're not taking in any context of this 0:15:46.036 --> 0:15:49.196 particular tree and what do the leaves look like? And 0:15:49.236 --> 0:15:53.756 so this idea of how can you start to ask 0:15:53.836 --> 0:15:58.796 these questions using all of the information around you, using 0:15:59.036 --> 0:16:03.396 your camera to actually capture this particular oak tree, using 0:16:03.396 --> 0:16:07.716 your location to know, you know, what are the native 0:16:07.716 --> 0:16:11.436 oaks in this area? And what's the current incidence of 0:16:12.196 --> 0:16:14.476 sudden oak death syndrome, which is a thing that I 0:16:14.516 --> 0:16:17.116 have recently learned exists. Okay, so I get why this 0:16:17.196 --> 0:16:22.116 is a hard thing to search in a text box, right, 0:16:22.236 --> 0:16:24.556 And so the thing that's interesting to me is how 0:16:24.596 --> 0:16:29.436 can we facilitate asking those types of questions where it's 0:16:29.476 --> 0:16:33.876 a mix of here's something that you're looking at, Here's 0:16:33.916 --> 0:16:37.156 something that you're saying with your words that adds to 0:16:37.236 --> 0:16:40.876 the picture. You know, here's a lemon tree that's got 0:16:40.956 --> 0:16:43.156 some black spots on it? What's wrong with it? Like? 0:16:43.196 --> 0:16:46.156 How can you help me understand what I should do 0:16:46.236 --> 0:16:49.996 about this? You know, these sorts of questions I think 0:16:50.156 --> 0:16:54.156 are right now. We have to do a tremendous amount 0:16:54.156 --> 0:16:57.676 of work to try and translate these questions into text 0:16:57.876 --> 0:17:01.276 that we would issue to a search engine. And yeah, 0:17:01.516 --> 0:17:05.316 we use that. Yeah, yeah, normal people. Yes, we're all 0:17:05.356 --> 0:17:08.196 doing it. And when you think it's you know, sometimes 0:17:08.196 --> 0:17:12.876 it's very easy, right, but sometimes you're like really having 0:17:12.916 --> 0:17:14.836 to work hard to come up with a query that 0:17:14.836 --> 0:17:16.636 will actually get you the answers that you need. And 0:17:16.676 --> 0:17:19.276 I think that's really the next frontier for us is 0:17:19.716 --> 0:17:23.196 how do we on the query side help users just 0:17:23.436 --> 0:17:30.156 naturally intuitively express whatever information need they have. And then 0:17:30.276 --> 0:17:34.276 how do we understand the whole universe of information, not 0:17:34.396 --> 0:17:38.356 just the web pages, that all the images and video 0:17:38.516 --> 0:17:43.636 and audio out there, and take that next level of 0:17:43.676 --> 0:17:47.756 like concept understanding to match those together so that we 0:17:47.796 --> 0:17:53.716 can get users even more precise answers that really help them. Great. 0:17:53.836 --> 0:18:00.356 So that's the like vast dream slash big problem. Can 0:18:00.396 --> 0:18:02.356 we reduce it a little bit so we can talk 0:18:02.396 --> 0:18:05.036 in sort of practical terms about what you're working on. 0:18:05.076 --> 0:18:08.556 I mean, I know there's this new AI model that 0:18:09.156 --> 0:18:12.516 integrates images, like you can, you know, whatever, take a 0:18:12.556 --> 0:18:14.876 picture with the camera on your phone and put in text. 0:18:14.996 --> 0:18:17.156 So like, well, you have this new model, and it, 0:18:17.356 --> 0:18:20.676 like the old one, has this worm fuzzy acronym. Right, 0:18:20.716 --> 0:18:23.516 it's called MUM, which stands for hold on, I gotta 0:18:23.516 --> 0:18:27.756 look at my notes, the multitask unified model. So like, 0:18:28.076 --> 0:18:33.036 tell me about MUM. So MOM is our next level 0:18:33.076 --> 0:18:36.676 model that you know Bert was about language. MOM is 0:18:36.716 --> 0:18:41.316 about all these different modalities of information coming together, particular 0:18:41.516 --> 0:18:44.636 images and language. I mean, is that if really images 0:18:44.676 --> 0:18:49.596 and language and we've got some limited applications of it 0:18:49.636 --> 0:18:53.876 in search today. So for example, you can take the 0:18:54.276 --> 0:18:58.756 take the photo of somebody's handbag and say you want 0:18:58.756 --> 0:19:01.396 to shop it, and that will work today. And that 0:19:01.556 --> 0:19:04.036 is like we were not able to do this previously, 0:19:04.116 --> 0:19:06.076 and that in and of itself is a big breakthrough. 0:19:06.516 --> 0:19:09.516 But there's still so much headroom, right like, this, still 0:19:09.556 --> 0:19:13.516 so much ability to say, you know, you can add 0:19:13.556 --> 0:19:18.476 sort of I would I would classify our current ability 0:19:18.516 --> 0:19:22.236 to process words in this multimodal context as you know, 0:19:22.396 --> 0:19:24.396 kind of like back in the early days days of 0:19:24.436 --> 0:19:26.636 the internet, you can say near me to find where 0:19:26.636 --> 0:19:29.196 you can buy it. Near me, you can say buy, 0:19:29.396 --> 0:19:34.396 but you can't necessarily like ask an incredibly complicated question 0:19:34.956 --> 0:19:38.116 about a picture, right like, so we're kind of back 0:19:38.156 --> 0:19:42.276 to keywords in this new pictures plus words universe. Let 0:19:42.316 --> 0:19:45.356 me ask a dumb question, why why can't you just 0:19:45.436 --> 0:19:51.036 take all of your brilliant intent AI and copy and 0:19:51.116 --> 0:19:56.516 paste it to fit with the image AI. So a 0:19:56.556 --> 0:20:01.716 couple of things. The first is that anytime we develop 0:20:02.236 --> 0:20:04.876 sort of this new technology, we also need to see 0:20:04.916 --> 0:20:07.916 how users start using it, right And so I think 0:20:07.956 --> 0:20:12.156 it's also fed say that we don't have. You know, 0:20:12.196 --> 0:20:14.196 we have a ton of people using this, but we 0:20:15.036 --> 0:20:18.156 haven't yet. There hasn't been time for that new technology 0:20:18.196 --> 0:20:21.156 to really be accepted by the world. And then we 0:20:21.196 --> 0:20:25.396 have this vast set of queries that we're doing poorly on, right. 0:20:25.436 --> 0:20:27.276 So that's the other thing you should know about Google. 0:20:27.516 --> 0:20:29.556 We spend a lot of time looking at the queries 0:20:29.596 --> 0:20:31.356 where we're failing. That's one of the other reasons we 0:20:31.396 --> 0:20:33.876 have a deep appreciation of how search is an unsolved 0:20:33.916 --> 0:20:37.236 problem because we're just constantly looking at queries whether the 0:20:37.316 --> 0:20:40.716 users clearly not getting what they're looking for. And I'll 0:20:41.116 --> 0:20:45.156 use that as a as a siege to figure out 0:20:45.196 --> 0:20:47.836 how to make things better. So do I understand you 0:20:47.916 --> 0:20:50.196 that the fundamental thing you need now is just lots 0:20:50.236 --> 0:20:52.716 of people to use this thing so you can see 0:20:53.436 --> 0:20:56.156 the weird ways people search and the things they sort 0:20:56.196 --> 0:21:00.236 of do that are hard to understand. That's certainly one 0:21:00.276 --> 0:21:03.756 of the things we need. I mean, it is fundamentally 0:21:04.316 --> 0:21:07.636 search works in service of our users, right, and so 0:21:07.876 --> 0:21:13.196 understanding the the failures is critical to how we get better. 0:21:13.396 --> 0:21:15.476 I think there are also just things that we know 0:21:15.556 --> 0:21:17.956 that we need to do on the AI and the 0:21:17.996 --> 0:21:22.396 model side that we'll continue working through, right, the ability 0:21:22.476 --> 0:21:28.156 to really bring together more of the two step process 0:21:28.196 --> 0:21:31.636 of how do you conceptually understand the words, conceptually understand 0:21:31.636 --> 0:21:35.076 the image, and then bring those two things together and 0:21:35.156 --> 0:21:37.556 have that be a bit deeper on both sides rather 0:21:37.636 --> 0:21:40.716 than just the combination together and those sorts of things. 0:21:41.676 --> 0:21:44.236 But yeah, I mean people coming in and using it 0:21:44.276 --> 0:21:46.636 and then having a bad time, we'll then make it better. 0:21:48.276 --> 0:21:53.316 It seems like there have been two main threads of 0:21:53.396 --> 0:21:57.476 AI research. One is basically language and the other is 0:21:57.876 --> 0:22:02.796 basically vision and images. I mean, it is it right 0:22:02.836 --> 0:22:05.196 to think of what you're trying to do as the 0:22:05.236 --> 0:22:10.756 synthesis of those two sort of main AI traditions. Yeah, 0:22:10.796 --> 0:22:15.956 I think so. I think it is clearly the case 0:22:16.076 --> 0:22:20.196 that just like uh, you know, with Bert, we took 0:22:20.196 --> 0:22:23.236 all these words and we got down to concepts. Right, 0:22:23.476 --> 0:22:26.236 It is clearly the case that human beings understand the 0:22:26.276 --> 0:22:28.756 world through concepts, and they do that visually, and they 0:22:28.796 --> 0:22:32.876 do that with language, and ultimately the concepts are the same. Right, 0:22:32.956 --> 0:22:36.516 So being able being able to say, okay, here's here's 0:22:36.516 --> 0:22:39.516 a concept, and we can attach to that what that 0:22:39.596 --> 0:22:43.076 concept looks like or that visual representation of that concept 0:22:43.116 --> 0:22:45.876 as much as it has one and the words surrounding 0:22:45.916 --> 0:22:50.196 that concept. That's when we can really unlock this true 0:22:51.596 --> 0:22:56.076 natural way of understanding the world that we think is 0:22:56.436 --> 0:22:58.836 going to enable people to ask all those questions that 0:22:58.876 --> 0:23:02.356 they have that they're not asking right now. Are there 0:23:03.276 --> 0:23:08.756 applications that go beyond search that come to mind if 0:23:08.756 --> 0:23:13.316 you figure this out? Yeah, I mean I think that 0:23:16.316 --> 0:23:20.596 search has this connotation of kind of find what's out there. 0:23:22.316 --> 0:23:26.476 I think there's something, you know, we're thinking about what 0:23:26.516 --> 0:23:31.156 this looks like in the generative space. So for example, 0:23:31.196 --> 0:23:35.836 if you're looking for you know, I bake birthday cakes, 0:23:36.356 --> 0:23:40.836 and sometimes I for my kids, and sometimes what my 0:23:40.916 --> 0:23:43.636 kids want, and a birthday cake just actually doesn't exist 0:23:43.796 --> 0:23:46.956 on the internet right or there's like only one or two, 0:23:47.156 --> 0:23:48.956 So then I have to come up with it myself. 0:23:48.996 --> 0:23:54.716 And like, what if AI could help us generate as 0:23:54.756 --> 0:23:59.156 sample image just based on these concepts that I could 0:23:59.196 --> 0:24:02.716 then use for inspiration. I think that's a pretty interesting concept. 0:24:03.476 --> 0:24:05.356 There's obviously a lot of things that we need to 0:24:05.356 --> 0:24:07.996 be very thoughtful about in this space as we do it, 0:24:09.396 --> 0:24:12.996 But I think this idea of extending search past the 0:24:13.076 --> 0:24:15.916 notion of connecting you with the information that's out there. 0:24:15.956 --> 0:24:21.676 To actually synthesizing new information for you is pretty interesting 0:24:21.876 --> 0:24:25.156 and something we're talking about a lot. You know. One 0:24:25.196 --> 0:24:27.316 of the things that has become clear to me talking 0:24:27.356 --> 0:24:32.716 with you is clearly, I think too narrowly about search. Right. 0:24:32.756 --> 0:24:35.756 I have this very kind of twenty years ago idea 0:24:35.836 --> 0:24:41.636 of like searching text on the web, and the web 0:24:41.716 --> 0:24:45.516 has become much less text based in that time. Right, 0:24:46.876 --> 0:24:50.196 the web includes Instagram, the web includes TikTok, and those 0:24:50.196 --> 0:24:54.196 are places where, weirdly to me, lots of people go 0:24:54.316 --> 0:24:56.956 to search like people go on TikTok to find whatever, 0:24:56.996 --> 0:24:58.716 where to go out to eat, which would never occur 0:24:58.796 --> 0:25:01.796 to me. So I mean it's that part of the 0:25:01.916 --> 0:25:06.596