WEBVTT - The AI Pioneer Developing New Kinds of Medicine 0:00:15.356 --> 0:00:15.796 Pushkin. 0:00:20.156 --> 0:00:22.156 If I were going to pick one paper from the 0:00:22.196 --> 0:00:25.316 past decade that had the biggest impact on the world, 0:00:25.876 --> 0:00:28.676 I would choose one called Attention Is All You Need, 0:00:28.916 --> 0:00:34.836 published in twenty seventeen. That paper basically invented transformer models. 0:00:35.356 --> 0:00:38.836 You've almost certainly used a transformer model if you have 0:00:39.036 --> 0:00:42.676 used chat GPT or Gemini or Claude or deep Seek. 0:00:42.996 --> 0:00:47.236 In fact, the tea in chat GPT stands for a transformer, 0:00:48.076 --> 0:00:51.196 and transformer models have turned out to be wildly useful, 0:00:51.276 --> 0:00:54.476 not just at generating language, but also at everything from 0:00:54.516 --> 0:00:57.596 generating images to predicting what proteins will look like. 0:00:58.516 --> 0:01:00.716 In fact, transformers. 0:01:00.076 --> 0:01:03.156 Are so ubiquitous and so powerful that it's easy to 0:01:03.196 --> 0:01:05.556 forget that some guy just thought them up. 0:01:06.316 --> 0:01:08.036 But in fact, some guy did. 0:01:08.116 --> 0:01:10.876 Just think up transform and I'm talking to him today 0:01:11.276 --> 0:01:19.396 on the show. I'm Jacob Goldstein and this is What's 0:01:19.436 --> 0:01:21.396 Your Problem, the show where I talk to people who 0:01:21.396 --> 0:01:24.956 are trying to make technological progress. My guest today is 0:01:25.036 --> 0:01:28.436 Yakub Uskolai. And just to be clear, Yakub was one 0:01:28.476 --> 0:01:31.796 of several co authors on that transformer paper, and on 0:01:31.836 --> 0:01:34.596 top of that, lots of other researchers were working on 0:01:34.716 --> 0:01:36.756 related things at the same time, so a lot of 0:01:36.796 --> 0:01:39.596 people were working on this, but the key idea did 0:01:39.636 --> 0:01:44.076 seem to come from Yakub. Today, Yakub is the CEO 0:01:44.236 --> 0:01:47.476 of Inceptive. That's a company that he co founded to 0:01:47.636 --> 0:01:51.076 use AI to develop new kinds of medicine, and the 0:01:51.116 --> 0:01:54.876 company is particularly focused on RNA. We talked about his 0:01:54.916 --> 0:01:57.476 work at Inceptive in the second part of our conversation. 0:01:58.236 --> 0:02:00.196 In the first part, we talked about his work on 0:02:00.316 --> 0:02:03.956 transformer models. At the time he started working on the 0:02:04.036 --> 0:02:07.236 idea for transformers, this is around a decade ago now, 0:02:07.556 --> 0:02:10.916 there were a couple of big problems with existing language models. 0:02:11.436 --> 0:02:13.956 For one thing, they were slow. They were in fact 0:02:13.996 --> 0:02:16.276 so slow that they could not even keep up with 0:02:16.356 --> 0:02:19.996 all the new training data that was becoming available. A 0:02:20.036 --> 0:02:24.716 second problem, they struggled with what are called long range dependencies. 0:02:25.196 --> 0:02:29.156 Basically in language, that's relationships between words that are far 0:02:29.236 --> 0:02:32.196 apart from each other in a sentence. So to start, 0:02:32.356 --> 0:02:34.756 I asked Yakab for an example we could use to 0:02:34.796 --> 0:02:37.756 discuss these problems and also how he came up with 0:02:37.836 --> 0:02:41.076 his big idea for how to solve them. So, pick 0:02:41.116 --> 0:02:43.916 a sentence that's going to be a good object lesson 0:02:43.956 --> 0:02:44.276 for us. 0:02:44.356 --> 0:02:47.516 Okay, so we could have the frog didn't cross the 0:02:47.556 --> 0:02:50.356 road because it was too tired. Okay, so we got 0:02:50.356 --> 0:02:51.556 our sentence. Yep. 0:02:52.236 --> 0:02:55.236 How would the sort of big, powerful but slow to 0:02:55.316 --> 0:02:57.836 train algorithm in twenty fifteen. 0:02:59.036 --> 0:03:02.596 Have processed that sentence? So basically it would have walked 0:03:02.636 --> 0:03:06.476 through that sentence word by word, and so it would 0:03:06.556 --> 0:03:11.876 walk through the sentence left to right. The frog did 0:03:12.076 --> 0:03:15.036 not cross the road because it was too tired. 0:03:15.076 --> 0:03:17.796 Which is logical, which is how I would think a 0:03:17.836 --> 0:03:18.636 system would work. 0:03:18.676 --> 0:03:21.676 It's more or less how we read, right, it's how 0:03:21.716 --> 0:03:25.076 we read, but it's not necessarily how we understand. Uh huh. 0:03:25.116 --> 0:03:28.476 That is actually one of the integral I would say 0:03:29.036 --> 0:03:31.876 for what we then how we then went about trying 0:03:31.916 --> 0:03:32.716 to speak us all up? 0:03:32.876 --> 0:03:34.796 Well, I love that. I want you to say more 0:03:34.836 --> 0:03:37.236 about it. When you say it's not how we understand, 0:03:37.276 --> 0:03:37.876 what do you mean? 0:03:38.516 --> 0:03:43.156 So? On one hand, right linearity of time forces us 0:03:43.236 --> 0:03:48.636 to almost always feel that we're communicating language in order 0:03:48.836 --> 0:03:53.876 and just linearly. It actually turns out that that's not 0:03:53.916 --> 0:03:56.956 really how we read, not even in terms of our secades, 0:03:56.956 --> 0:03:59.236 in terms of our em movements. We actually do jump 0:03:59.276 --> 0:04:01.916 back and forth quite a bit while reading, and if 0:04:01.956 --> 0:04:06.756 you look at conversations, you also have highly nonlinear elements 0:04:06.796 --> 0:04:11.076 where there's repetition, there's reference, there's basically different flavors of interruption. 0:04:11.556 --> 0:04:14.036 But sure, by and large right, we would say we 0:04:14.156 --> 0:04:17.356 certainly right them left to right right. So if you 0:04:17.516 --> 0:04:20.556 write a proper text, you don't write it as you 0:04:20.556 --> 0:04:22.276 would read it, and you also don't write it as 0:04:22.316 --> 0:04:24.196 you would talk about it. You do write it in 0:04:24.796 --> 0:04:28.196 one linear order. Now, as we read this and as 0:04:28.196 --> 0:04:34.196 we understand this, we actually form groups of words that 0:04:34.316 --> 0:04:38.156 then form meaning. Right. So an example of that is 0:04:38.596 --> 0:04:42.556 you know adjective noun, right, it's or say, in this 0:04:42.596 --> 0:04:45.756 case an article noun, it's not a frog, it's the frog. Right. 0:04:45.916 --> 0:04:48.716 We could have also said it's the green frog or 0:04:49.156 --> 0:04:49.956 the lazy frog. 0:04:50.356 --> 0:04:54.076 Right. Language has a structure, right, and there things can 0:04:54.116 --> 0:04:58.076 modify other things, and things can modify the modifiers exactly exactly. 0:04:58.156 --> 0:05:02.156 But the interesting thing now is that structure in as 0:05:01.956 --> 0:05:05.636 a as a tree structured clean hierarchy, only tells you 0:05:05.676 --> 0:05:10.876 half the story. There's so many exceptions where statistical dependencies, 0:05:10.956 --> 0:05:13.836 where modification actually happens at a distance. 0:05:14.116 --> 0:05:16.116 So okay, So just to bring this back to your 0:05:16.156 --> 0:05:19.076 sample sentence, The frog didn't cross the road because it 0:05:19.156 --> 0:05:22.956 was too tired. That word it is actually quite far 0:05:22.996 --> 0:05:25.996 from the word frog. And if you're an AI going 0:05:25.996 --> 0:05:28.956 from left to right, you may well get confused there, right, 0:05:28.996 --> 0:05:33.196 You may think it refers to road instead of to frog. 0:05:34.276 --> 0:05:37.076 So this is one of the problems you were trying 0:05:37.116 --> 0:05:39.236 to solve. And then the other one you were mentioning before, 0:05:39.556 --> 0:05:43.836 which is these models were just slow because after each word, 0:05:43.876 --> 0:05:46.956 the model just recalculates what everything means, and that just 0:05:47.156 --> 0:05:48.156 takes a long time. 0:05:48.476 --> 0:05:51.396 They can't go fast enough exactly. It takes a long time, 0:05:51.476 --> 0:05:55.116 and it doesn't play to the strengths of the computers, 0:05:55.356 --> 0:05:57.356 of the accelerators that we're using there. 0:05:57.756 --> 0:05:59.916 And when you say accelerators, I know Google has their 0:05:59.956 --> 0:06:02.956 own chips, but basically we mean GPUs. 0:06:02.356 --> 0:06:04.756 Now right, we mean GPUs, We mean. 0:06:04.636 --> 0:06:08.276 The chips that Nvidia sells. What is the nature of. 0:06:08.196 --> 0:06:10.396 Those particular ships. Yeah, So the nature of those particular 0:06:10.516 --> 0:06:16.596 chips is that instead of doing a broad variety of 0:06:16.756 --> 0:06:22.636 complex computations in sequence, they are incredibly good. They excel 0:06:23.116 --> 0:06:27.516 at performing many, many, many simple computations in parallel. And 0:06:27.556 --> 0:06:32.396 so what this hierarchical or semi hierrachical nature of language 0:06:32.956 --> 0:06:38.356 enables you to do is instead of having, so to speak, 0:06:38.476 --> 0:06:41.956 one place where you read the current word, you could 0:06:42.036 --> 0:06:46.876 now imagine you actually read every You look at everything 0:06:46.876 --> 0:06:52.196 at the same time, and you apply many simple operations 0:06:52.556 --> 0:06:55.516 at the same time to each position in your sentence. 0:06:55.796 --> 0:06:58.276 Huh So this is the big idea, I just want 0:06:58.276 --> 0:07:01.876 to because this is it, right, this is the breakthrough happening. Yes, 0:07:02.756 --> 0:07:05.836 it's basically, what if instead of reading the sentence one 0:07:05.836 --> 0:07:08.276 word at a time from left to right, we read 0:07:08.276 --> 0:07:10.036 the whole thing all at once. 0:07:10.396 --> 0:07:13.676 All at once. And now the problem is clearly something's 0:07:13.676 --> 0:07:17.076 got to give, right, so there's no fore lunch in 0:07:17.116 --> 0:07:20.116 that sense. You have to now simplify what you can 0:07:20.156 --> 0:07:22.956 do at every position when you do this all in parallel, 0:07:24.436 --> 0:07:26.716 but you can now afford to do this a bunch 0:07:26.716 --> 0:07:29.916 of times after another and revise it over time or 0:07:29.956 --> 0:07:32.796 over these steps. And so instead of walking through the 0:07:32.836 --> 0:07:35.836 sentence from beginning to end, whether an average sentence has 0:07:35.876 --> 0:07:39.076 like twenty words or so average sentence in pros, instead 0:07:39.116 --> 0:07:41.716 of walking those twenty positions, what you're doing is you're 0:07:41.756 --> 0:07:46.116 looking at every word at the same time, but in 0:07:46.156 --> 0:07:49.036 a simpler way. But now you can do that maybe 0:07:49.156 --> 0:07:53.556 five or six times, revising your understanding, and that turns 0:07:53.556 --> 0:07:59.596 out is faster, way faster on GPUs and because of 0:07:59.636 --> 0:08:01.916 this hierarchical nature of language, it's also better. 0:08:02.996 --> 0:08:06.076 So you have this idea, and as I read the 0:08:06.076 --> 0:08:08.076 little note on the paper, it was in fact your idea. 0:08:08.076 --> 0:08:09.516 I know you were working with a t but the 0:08:09.556 --> 0:08:12.796 paper credits you with the idea. So let's let's take 0:08:12.836 --> 0:08:15.476 this idea, this basic idea of look at the whole 0:08:15.476 --> 0:08:18.796 input sentence all at once, yep, a few times, and 0:08:19.876 --> 0:08:22.436 apply it to our frog sentence. Give me, give me 0:08:22.436 --> 0:08:23.436 that frog sentence again. 0:08:23.996 --> 0:08:26.396 The frog did not cross the road because it was 0:08:26.436 --> 0:08:27.916 too tired. Good. 0:08:28.276 --> 0:08:30.156 Tired is good because that's unambiguous. 0:08:30.196 --> 0:08:31.716 Hot could be either one. It could be the road 0:08:31.836 --> 0:08:33.756 or the frog, right, Hot could be hot could be 0:08:33.836 --> 0:08:36.596 the one exactly is in fact hot could either could 0:08:36.596 --> 0:08:40.476 actually either one and non referential and non referential because 0:08:40.516 --> 0:08:42.396 it was too hot outside. 0:08:41.996 --> 0:08:44.076 Outside it could be any of three things, the weather, 0:08:44.396 --> 0:08:47.396 or the frog or the road exactly. I love that 0:08:47.636 --> 0:08:52.956 tired solves the problem. So your model, this new way 0:08:52.996 --> 0:08:57.276 of doing things, how does it parse that sentence, what 0:08:57.316 --> 0:08:57.716 does it do? 0:08:58.356 --> 0:09:02.636 So basically, let's look at the word it and look 0:09:02.636 --> 0:09:04.956 at it in every single step of these you know, 0:09:05.076 --> 0:09:10.356 say a handful of times repeated operation. Imagine you're looking 0:09:10.396 --> 0:09:12.516 at this word it, that's the one that you are 0:09:12.556 --> 0:09:16.076 now trying to understand better, and you now compare it 0:09:16.276 --> 0:09:18.796 to every other word in the sense. Okay, so you 0:09:18.836 --> 0:09:22.196 compare it to the to frog that did not cross 0:09:22.316 --> 0:09:28.356 the road because two and tired, there was two and 0:09:28.516 --> 0:09:36.076 tire and initially in the first past. Already a very 0:09:36.596 --> 0:09:40.276 simple insight the model can fairly easily learn is that 0:09:40.356 --> 0:09:49.196 it could be strongly informed by frog, by road, by nothing, 0:09:50.196 --> 0:09:54.276 but not so by two or by the or maybe 0:09:54.276 --> 0:09:56.916 only to a certain extent by us. But if you 0:09:56.916 --> 0:10:00.676 want to know more about what it denotes, then it 0:10:00.716 --> 0:10:03.876 could be, you know, it could be informed by by 0:10:03.876 --> 0:10:04.476 all of these. 0:10:04.876 --> 0:10:08.036 And just to be clear, that sort of understanding arises 0:10:08.076 --> 0:10:09.476 because it has trained in. 0:10:09.396 --> 0:10:11.116 This way on lots of data. 0:10:11.156 --> 0:10:14.956 It's encountering a new sentence after reading lots of other 0:10:15.036 --> 0:10:19.236 sentences with lots of pronouns with different possible antecedents. 0:10:19.316 --> 0:10:23.156 Yeah, exactly, exactly. So Now the interesting thing is that 0:10:23.516 --> 0:10:29.836 which of the two it actually refers to, doesn't depend 0:10:30.116 --> 0:10:33.156 on only on what those other two words are. And 0:10:33.196 --> 0:10:36.396 this is why you need these subsequent steps because so 0:10:36.996 --> 0:10:39.476 let's talk with the first step. So what now happens 0:10:39.556 --> 0:10:44.116 is that, say the model identifies frog and road could 0:10:44.116 --> 0:10:46.916 have a lot to do with the word it. So 0:10:47.156 --> 0:10:51.396 now you basically copy some information from both frog and 0:10:51.636 --> 0:10:55.916 road over to it, and you don't just copy it, 0:10:55.956 --> 0:10:59.076 you kind of transform it also on the way, but 0:10:59.196 --> 0:11:02.956 you refine your understanding of it. And this is all learned, 0:11:02.996 --> 0:11:06.076 does not given by rules or you know, in any 0:11:06.116 --> 0:11:07.916 way pre specifying. 0:11:07.436 --> 0:11:11.276 Right, just by training on loge, just by training this emergency, 0:11:11.276 --> 0:11:13.636 and so that sort of the meaning of it after 0:11:13.676 --> 0:11:17.676 this first step is kind of influenced by both frog 0:11:17.756 --> 0:11:18.196 and road. 0:11:18.316 --> 0:11:23.356 Yes, both frog and road. Okay, so now we repeat 0:11:23.396 --> 0:11:27.596 this operation again and we now know that it is 0:11:27.796 --> 0:11:30.876 unsure or the model basically now has this kind of superposition. Right, 0:11:30.916 --> 0:11:34.076 it could be road, it could be frog. But now 0:11:34.196 --> 0:11:36.836 in the next step it also looks at tired, and 0:11:36.876 --> 0:11:41.116 somehow the model has learned that when it means something inanimate, 0:11:41.276 --> 0:11:46.556 that tired is not the thing. And so maybe in 0:11:46.676 --> 0:11:50.116 context of tired, it is more likely to refer to frog, 0:11:50.836 --> 0:11:54.716 and now you know, well, it is more likely and 0:11:54.796 --> 0:11:57.036 now maybe the model has figured out already, maybe needs 0:11:57.076 --> 0:12:00.516 a bit more, a few more iterations that it is 0:12:00.596 --> 0:12:03.636 most likely to refer to frog because of the presence 0:12:04.196 --> 0:12:07.036 of tired. So it has solved the problem. But it 0:12:07.036 --> 0:12:08.036 has solved the problem. 0:12:08.556 --> 0:12:12.436 So you do, you have this idea, you try it out. 0:12:12.876 --> 0:12:14.996 There's a detail that you mentioned that's kind of fun, 0:12:15.036 --> 0:12:17.596 and we kind of skipped it, but you mentioned that 0:12:17.996 --> 0:12:20.116 another one of the co authors, who has also gone 0:12:20.156 --> 0:12:22.476 on to do very big things, was about to leave 0:12:22.556 --> 0:12:25.276 Google when you sort of want to test this idea, 0:12:25.356 --> 0:12:27.476 and and that fact that he was about to leave 0:12:27.476 --> 0:12:29.956 Google was actually important to the history of this idea. 0:12:29.996 --> 0:12:33.316 Tell me about that it was important. So this Ilia Plususian, 0:12:34.636 --> 0:12:39.476 he was at the time that this started to gain 0:12:39.716 --> 0:12:44.036 any kind of speed, Elia was managing a good chunk 0:12:44.116 --> 0:12:47.676 of my organization. And the moment he really made the 0:12:47.676 --> 0:12:51.396 decision to leave the company, he had to wait ultimately 0:12:51.556 --> 0:12:55.436 for his co for his co founder, and for them 0:12:55.476 --> 0:12:58.156 to then actually get going together in earnest and so 0:12:58.236 --> 0:13:00.716 he had a few months where he knew and I 0:13:00.796 --> 0:13:04.716 also knew that he was about to leave and where 0:13:04.836 --> 0:13:06.716 you know, the right thing would of course be to 0:13:06.756 --> 0:13:11.236 transition his team to another manager, which we did immediately, 0:13:11.716 --> 0:13:14.436 but where you then suddenly was in a position of 0:13:14.476 --> 0:13:18.356 having nothing to lose and yet quite some time left 0:13:18.436 --> 0:13:21.556 to play with Google's resources and do cool stuff with interesting, 0:13:21.796 --> 0:13:25.756 interesting people. And and so that's one of those moments 0:13:25.756 --> 0:13:31.716 where suddenly your appetite for risk as a researcher just spikes, right, huh, 0:13:32.636 --> 0:13:34.556 because you have, for for a few more months, you 0:13:34.556 --> 0:13:38.796 have these resources at your disposal, you've transitioned your responsibilities. 0:13:38.876 --> 0:13:41.836 At that stage, you're just like, Okay, let's try this 0:13:41.956 --> 0:13:46.236 crazy shit and and and it's and that's literally in 0:13:46.356 --> 0:13:49.676 so many ways, was was one of the integral catalysts 0:13:50.796 --> 0:13:55.036 because that also enabled, right, this kind of mindset of 0:13:55.276 --> 0:13:58.796 we're going for this now, whatever the reason. It still 0:13:58.956 --> 0:14:02.956 you know affects other people. And so there were others 0:14:02.956 --> 0:14:06.796 who joined that collaboration really really early on, who I 0:14:06.916 --> 0:14:10.236 feel were much more excited a result, much more likely 0:14:10.276 --> 0:14:12.396 to really work on this and to really give it 0:14:12.396 --> 0:14:17.196 there all because of his you know, nothing left to lose, 0:14:17.636 --> 0:14:19.796 I'm going to go for this attitude at this. 0:14:19.716 --> 0:14:23.556 Point, Right, was there a moment when you realized it worked. 0:14:23.796 --> 0:14:27.396 There were actually a few moments. And it's interesting because 0:14:29.396 --> 0:14:32.596 on one hand, right, it's a very gradual thing, right, 0:14:32.636 --> 0:14:35.636 And initially, actually it took us many months to get 0:14:35.676 --> 0:14:39.156 to the point where we saw significant first signs of 0:14:39.196 --> 0:14:41.836 life of this not just being a curiosity but really 0:14:41.876 --> 0:14:45.196 being something that would end up being competitive. So there 0:14:45.276 --> 0:14:48.036 certainly was a moment when that started. There was another 0:14:48.116 --> 0:14:51.796 moment when we for the for the first time had 0:14:51.916 --> 0:14:56.956 one machine translation challenge, one language pair of the W 0:14:57.116 --> 0:15:00.796 and T task as it's called, where our score, our 0:15:00.836 --> 0:15:05.116 model performed better than any other single model. The point 0:15:05.116 --> 0:15:07.796 in time when I think all of us realized this 0:15:07.876 --> 0:15:12.436 is special was when we not only had the best 0:15:12.436 --> 0:15:16.036 one in one of these tasks, but in multiple and 0:15:17.276 --> 0:15:19.556 we didn't just have the best number. We also at 0:15:19.556 --> 0:15:22.316 that point were able to establish that we've gotten there 0:15:22.356 --> 0:15:27.236 with about ten times less energy or training compute spend. 0:15:27.716 --> 0:15:30.116 Wow, So you do one tenth the work and you 0:15:30.156 --> 0:15:31.076 get a better result. 0:15:31.316 --> 0:15:33.276 One tenth the work and you get a better result 0:15:33.316 --> 0:15:37.116 not just across one specific challenge, but across multiple including 0:15:37.156 --> 0:15:39.876 the hardest or of one of the harder ones. Right. 0:15:40.316 --> 0:15:45.476 And then at that stage we were still improving rapidly, 0:15:46.716 --> 0:15:50.596 and then you realize, okay, this is for real. There's 0:15:51.036 --> 0:15:53.396 because there right, It wasn't like we it wasn't that 0:15:53.436 --> 0:15:56.116 we had to squeeze those last little bits and pieces 0:15:56.156 --> 0:15:59.756 of gain out of it. It was still improving fairly rapidly, 0:16:00.796 --> 0:16:03.476 to the point where actually, by the time we actually 0:16:03.476 --> 0:16:08.196 published the paper, we again reduced the computer requirements, not 0:16:08.276 --> 0:16:11.356 quite by an entire order of magnitude, but almost right, 0:16:11.476 --> 0:16:14.516 so it still was getting faster and better at a 0:16:14.556 --> 0:16:17.076 pretty rapid rate. Wow, so we had in the paper 0:16:17.116 --> 0:16:19.836 we had some results that were those roughly ten x 0:16:19.836 --> 0:16:23.396 faster on eighthpus and what we demonstrated in terms of 0:16:23.516 --> 0:16:26.596 quality on those eight GPUs by the time we actually 0:16:26.676 --> 0:16:29.396 published the paper properly we were able to do with one. 0:16:29.276 --> 0:16:32.636 GPU, one GPU meaning one chip of the kind that 0:16:32.676 --> 0:16:35.556 people by one hundred thousand of now to build a 0:16:35.636 --> 0:16:39.116 data center exactly. So the paper actually at the end 0:16:39.876 --> 0:16:45.596 mentions other possible uses beyond language for this technology. It 0:16:45.716 --> 0:16:50.996 mentions images, audio, and video, I think explicitly. How much 0:16:51.036 --> 0:16:52.836 were you thinking about that at the time. Was that 0:16:52.956 --> 0:16:55.036 just like an afterthought or were you like, hey, wait 0:16:55.076 --> 0:16:57.356 a minute, it's not just language. 0:16:57.596 --> 0:16:59.956 By the time it was actually published at a conference, 0:16:59.996 --> 0:17:04.076 not just the preprint. By December, we had initial models 0:17:04.236 --> 0:17:07.436 on other modalities on generating images. We had the first 0:17:07.716 --> 0:17:09.836 the first at the stay. At that time they were 0:17:10.116 --> 0:17:12.636 not performing that well yet, but you know, they were 0:17:12.716 --> 0:17:15.916 rapidly getting better. We had the first prototypes actually of 0:17:16.036 --> 0:17:20.676 models working on genomic data, working on protein structure. That's 0:17:20.676 --> 0:17:24.116 good for shadow good for shadowing exactly. But then we 0:17:24.236 --> 0:17:27.236 ended up for a variety of reasons, we ended up 0:17:27.756 --> 0:17:30.716 at first focusing on applications in computer vision. 0:17:31.116 --> 0:17:33.716 The paper comes out, you know, you're working on these 0:17:33.756 --> 0:17:37.756 other applications, you're presenting the paper, it's published in various forms. 0:17:38.356 --> 0:17:42.636 What's the response like. It was interesting because the response 0:17:43.076 --> 0:17:49.836 built in deep learning AI circles basically between the pre 0:17:49.916 --> 0:17:52.236 print that I think came out and I want to 0:17:52.276 --> 0:17:56.436 say June twenty seventeen, and then the actually actual publication, 0:17:56.876 --> 0:17:59.916 to the extent that by the time the poster session 0:17:59.956 --> 0:18:03.676 happened at the conference, there was quite a crowd at 0:18:03.676 --> 0:18:07.036 the poster so we had to be shoved out of 0:18:07.116 --> 0:18:10.316 the out of the hall in which the poster session happened. 0:18:10.316 --> 0:18:14.276 About security and had very hors voices by the end 0:18:14.316 --> 0:18:18.676 of the evening, you guys were like the Beatles of 0:18:18.756 --> 0:18:23.556 the AI conference. I wouldn't say that because we weren't 0:18:23.556 --> 0:18:26.396 the Beatles, because it was really it was still very specific. 0:18:26.436 --> 0:18:27.836 You were more that you were more of the cool 0:18:27.916 --> 0:18:30.036 hipster band. You were the hipster. 0:18:29.756 --> 0:18:32.156 Band, certainly more the cool hipster band. But it was 0:18:32.196 --> 0:18:35.396 an interesting experience because there were some folks and including 0:18:35.396 --> 0:18:38.916 some greats in the field, who came by and said, Wow, 0:18:39.036 --> 0:18:40.396 this is this is cool. 0:18:40.716 --> 0:18:44.156 What has happened since has been wild. 0:18:44.596 --> 0:18:48.036 It seems wild to say the least. Yes, Is it 0:18:48.116 --> 0:18:52.196 surprising to you? Of course, many aspects are surprising. For sure. 0:18:53.796 --> 0:18:57.956 We definitely saw pretty early on already back in twenty eighteen, 0:18:57.996 --> 0:19:04.076 twenty nineteen, that something really exciting was happening here. Now 0:19:05.316 --> 0:19:08.836 I'm still surprised by with the advent of chat GPT, 0:19:10.276 --> 0:19:15.076 something that didn't go way beyond those language models that 0:19:15.116 --> 0:19:19.036 we had already seen a few years before, was suddenly 0:19:20.436 --> 0:19:23.876 the world's fastest growing consumer product. 0:19:23.836 --> 0:19:25.436 Ever, right, I think ever? 0:19:25.676 --> 0:19:26.716 Ever? Yes? 0:19:27.076 --> 0:19:31.956 And by the way, GBT stands for generative pre transformer, right, 0:19:31.996 --> 0:19:35.636 transformer is your word, that's right? So there's an interesting 0:19:36.956 --> 0:19:39.996 I don't know, business side to this right, which is, 0:19:40.356 --> 0:19:42.196 you were working for Google when you came up with this. 0:19:42.356 --> 0:19:48.316 Google presumably owned the idea, had intellectual property around. 0:19:47.996 --> 0:19:49.956 The idea has filed many a patent. 0:19:50.116 --> 0:19:52.436 Was it just a choice Google made to let everybody 0:19:52.556 --> 0:19:56.196 use it? Like when you see the fastest growing consumer 0:19:56.356 --> 0:19:58.716 product in this year of the world not only built 0:19:58.716 --> 0:20:02.076 on this idea, but using the name like and it's 0:20:02.116 --> 0:20:04.356 a different company that was five years later. 0:20:04.236 --> 0:20:04.836 Five years later. 0:20:04.876 --> 0:20:07.436 But a patent's good for more than five years? Is 0:20:07.476 --> 0:20:08.276 that a choice? 0:20:08.356 --> 0:20:10.516 Is that a stret dig choice? What's going on there? 0:20:11.036 --> 0:20:14.196 So the choice to do it in the first place, 0:20:15.036 --> 0:20:19.396 to publish it in the first place, is really based 0:20:19.476 --> 0:20:23.916 on and and rooted in a deep conviction of Google 0:20:23.956 --> 0:20:26.636 at the time, And I'm actually pretty sure it still 0:20:26.676 --> 0:20:31.596 is the case that it is. Actually these developments are 0:20:31.676 --> 0:20:34.356 the tide that floats all votes, that lifts. 0:20:33.876 --> 0:20:38.796 All votes, like a belief in progress, a belief in progress, 0:20:38.796 --> 0:20:40.596 a good old fashioned Now. 0:20:40.996 --> 0:20:45.436 It's also the case that at the time, organizationally, that 0:20:45.556 --> 0:20:51.596 specific research arm was unusually separated from the product organizations. 0:20:51.756 --> 0:20:56.476 And the reason why Brain or in general, the deep 0:20:56.556 --> 0:21:02.436 learning groups were more separated was in part historical, namely 0:21:02.676 --> 0:21:05.716 that when they started out there were no applications and 0:21:05.876 --> 0:21:08.996 the technology was not ready for being applied, and so 0:21:09.356 --> 0:21:13.996 it's completely understandable and just you know a consequence of 0:21:14.156 --> 0:21:20.196 organic developments that when this technology suddenly is on the 0:21:20.236 --> 0:21:24.836 cusp of being incredibly impactful, you're probably still under utilizing 0:21:24.876 --> 0:21:29.796 it internally and potentially also not yet treating it in 0:21:29.876 --> 0:21:32.756 the same way as you would have maybe otherwise treated 0:21:32.836 --> 0:21:35.036 previous trade secrets. 0:21:34.716 --> 0:21:39.516 For example, as it feels like this out their research project, 0:21:39.716 --> 0:21:42.276 not like what's going to be this consumer. 0:21:42.476 --> 0:21:47.316 Product exactly exactly, And to be fair, it took Open 0:21:47.356 --> 0:21:49.836 a Eye in this case a fair amount of time 0:21:50.116 --> 0:21:54.396 and to then turn this into this product, and most 0:21:54.396 --> 0:21:57.876 of that time it also from their vantage point, wasn't 0:21:57.876 --> 0:22:01.036 a product. Right. So up until all the way through 0:22:01.676 --> 0:22:06.836 chat REPT, Open Eye have published all of their GPT developments, 0:22:07.356 --> 0:22:10.036 maybe not all, but you know, their large fraction of 0:22:10.676 --> 0:22:11.316 their work on this. 0:22:11.516 --> 0:22:12.516 Yeah, they're early models. 0:22:12.516 --> 0:22:15.676 The whole models were open exactly. They were more true 0:22:15.676 --> 0:22:19.716 to their name really also believing in the same thing. 0:22:19.716 --> 0:22:22.396 And it was only really after chat GPT and after 0:22:22.476 --> 0:22:27.156 this to them also surprise to a certain extent success, 0:22:27.676 --> 0:22:31.556 that they started to become more closed as well when 0:22:31.596 --> 0:22:37.676 it comes to scientific developments in this past. You'll be 0:22:37.716 --> 0:22:54.836 back in just a minute. Let's talk about your company. 0:22:55.236 --> 0:22:58.156 When'd you decide to start Inceptive? The decision took a 0:22:58.156 --> 0:23:03.196 while and was influenced by events that happened over the 0:23:03.236 --> 0:23:07.116 course of about three months two to three months in 0:23:07.196 --> 0:23:12.196 late twenty twenty, starting with the birth of my first child. 0:23:13.116 --> 0:23:16.916 So when am I was born, two things happened. Number one, 0:23:17.516 --> 0:23:21.316 witnessing a pregnancy and a birth during a pandemic where 0:23:21.516 --> 0:23:24.916 there's a pathogen that's rapidly spreading, and so all of 0:23:24.956 --> 0:23:29.196 that was a pretty daunting experience, and everything went great, 0:23:30.276 --> 0:23:35.436 But having this new human in my arms also really 0:23:35.676 --> 0:23:41.636 made me question if I couldn't more directly affect people's 0:23:41.676 --> 0:23:45.956 lives positively with my work. And so I was at 0:23:46.036 --> 0:23:49.836 the time quite confident that indirectly it would have effect 0:23:49.916 --> 0:23:53.556 also on things like medicine, biology, etc. But I was wondering, 0:23:53.916 --> 0:23:58.196 couldn't this happen more directly if I focused more on it. 0:23:58.436 --> 0:24:00.916