Speaker 1: Pushkin.
Speaker 2: If I were going to pick one paper from the
Speaker 2: past decade that had the biggest impact on the world,
Speaker 2: I would choose one called Attention Is All You Need,
Speaker 2: published in twenty seventeen. That paper basically invented transformer models.
Speaker 2: You've almost certainly used a transformer model if you have
Speaker 2: used chat GPT or Gemini or Claude or deep Seek.
Speaker 2: In fact, the tea in chat GPT stands for a transformer,
Speaker 2: and transformer models have turned out to be wildly useful,
Speaker 2: not just at generating language, but also at everything from
Speaker 2: generating images to predicting what proteins will look like.
Speaker 1: In fact, transformers.
Speaker 2: Are so ubiquitous and so powerful that it's easy to
Speaker 2: forget that some guy just thought them up.
Speaker 1: But in fact, some guy did.
Speaker 2: Just think up transform and I'm talking to him today
Speaker 2: on the show. I'm Jacob Goldstein and this is What's
Speaker 2: Your Problem, the show where I talk to people who
Speaker 2: are trying to make technological progress. My guest today is
Speaker 2: Yakub Uskolai. And just to be clear, Yakub was one
Speaker 2: of several co authors on that transformer paper, and on
Speaker 2: top of that, lots of other researchers were working on
Speaker 2: related things at the same time, so a lot of
Speaker 2: people were working on this, but the key idea did
Speaker 2: seem to come from Yakub. Today, Yakub is the CEO
Speaker 2: of Inceptive. That's a company that he co founded to
Speaker 2: use AI to develop new kinds of medicine, and the
Speaker 2: company is particularly focused on RNA. We talked about his
Speaker 2: work at Inceptive in the second part of our conversation.
Speaker 2: In the first part, we talked about his work on
Speaker 2: transformer models. At the time he started working on the
Speaker 2: idea for transformers, this is around a decade ago now,
Speaker 2: there were a couple of big problems with existing language models.
Speaker 2: For one thing, they were slow. They were in fact
Speaker 2: so slow that they could not even keep up with
Speaker 2: all the new training data that was becoming available. A
Speaker 2: second problem, they struggled with what are called long range dependencies.
Speaker 2: Basically in language, that's relationships between words that are far
Speaker 2: apart from each other in a sentence. So to start,
Speaker 2: I asked Yakab for an example we could use to
Speaker 2: discuss these problems and also how he came up with
Speaker 2: his big idea for how to solve them. So, pick
Speaker 2: a sentence that's going to be a good object lesson
Speaker 2: for us.
Speaker 1: Okay, so we could have the frog didn't cross the
Speaker 1: road because it was too tired. Okay, so we got
Speaker 1: our sentence. Yep.
Speaker 2: How would the sort of big, powerful but slow to
Speaker 2: train algorithm in twenty fifteen.
Speaker 1: Have processed that sentence? So basically it would have walked
Speaker 1: through that sentence word by word, and so it would
Speaker 1: walk through the sentence left to right. The frog did
Speaker 1: not cross the road because it was too tired.
Speaker 2: Which is logical, which is how I would think a
Speaker 2: system would work.
Speaker 1: It's more or less how we read, right, it's how
Speaker 1: we read, but it's not necessarily how we understand. Uh huh.
Speaker 1: That is actually one of the integral I would say
Speaker 1: for what we then how we then went about trying
Speaker 1: to speak us all up?
Speaker 2: Well, I love that. I want you to say more
Speaker 2: about it. When you say it's not how we understand,
Speaker 2: what do you mean?
Speaker 1: So? On one hand, right linearity of time forces us
Speaker 1: to almost always feel that we're communicating language in order
Speaker 1: and just linearly. It actually turns out that that's not
Speaker 1: really how we read, not even in terms of our secades,
Speaker 1: in terms of our em movements. We actually do jump
Speaker 1: back and forth quite a bit while reading, and if
Speaker 1: you look at conversations, you also have highly nonlinear elements
Speaker 1: where there's repetition, there's reference, there's basically different flavors of interruption.
Speaker 1: But sure, by and large right, we would say we
Speaker 1: certainly right them left to right right. So if you
Speaker 1: write a proper text, you don't write it as you
Speaker 1: would read it, and you also don't write it as
Speaker 1: you would talk about it. You do write it in
Speaker 1: one linear order. Now, as we read this and as
Speaker 1: we understand this, we actually form groups of words that
Speaker 1: then form meaning. Right. So an example of that is
Speaker 1: you know adjective noun, right, it's or say, in this
Speaker 1: case an article noun, it's not a frog, it's the frog. Right.
Speaker 1: We could have also said it's the green frog or
Speaker 1: the lazy frog.
Speaker 2: Right. Language has a structure, right, and there things can
Speaker 2: modify other things, and things can modify the modifiers exactly exactly.
Speaker 1: But the interesting thing now is that structure in as
Speaker 1: a as a tree structured clean hierarchy, only tells you
Speaker 1: half the story. There's so many exceptions where statistical dependencies,
Speaker 1: where modification actually happens at a distance.
Speaker 2: So okay, So just to bring this back to your
Speaker 2: sample sentence, The frog didn't cross the road because it
Speaker 2: was too tired. That word it is actually quite far
Speaker 2: from the word frog. And if you're an AI going
Speaker 2: from left to right, you may well get confused there, right,
Speaker 2: You may think it refers to road instead of to frog.
Speaker 2: So this is one of the problems you were trying
Speaker 2: to solve. And then the other one you were mentioning before,
Speaker 2: which is these models were just slow because after each word,
Speaker 2: the model just recalculates what everything means, and that just
Speaker 2: takes a long time.
Speaker 1: They can't go fast enough exactly. It takes a long time,
Speaker 1: and it doesn't play to the strengths of the computers,
Speaker 1: of the accelerators that we're using there.
Speaker 2: And when you say accelerators, I know Google has their
Speaker 2: own chips, but basically we mean GPUs.
Speaker 1: Now right, we mean GPUs, We mean.
Speaker 2: The chips that Nvidia sells. What is the nature of.
Speaker 1: Those particular ships. Yeah, So the nature of those particular
Speaker 1: chips is that instead of doing a broad variety of
Speaker 1: complex computations in sequence, they are incredibly good. They excel
Speaker 1: at performing many, many, many simple computations in parallel. And
Speaker 1: so what this hierarchical or semi hierrachical nature of language
Speaker 1: enables you to do is instead of having, so to speak,
Speaker 1: one place where you read the current word, you could
Speaker 1: now imagine you actually read every You look at everything
Speaker 1: at the same time, and you apply many simple operations
Speaker 1: at the same time to each position in your sentence.
Speaker 2: Huh So this is the big idea, I just want
Speaker 2: to because this is it, right, this is the breakthrough happening. Yes,
Speaker 2: it's basically, what if instead of reading the sentence one
Speaker 2: word at a time from left to right, we read
Speaker 2: the whole thing all at once.
Speaker 1: All at once. And now the problem is clearly something's
Speaker 1: got to give, right, so there's no fore lunch in
Speaker 1: that sense. You have to now simplify what you can
Speaker 1: do at every position when you do this all in parallel,
Speaker 1: but you can now afford to do this a bunch
Speaker 1: of times after another and revise it over time or
Speaker 1: over these steps. And so instead of walking through the
Speaker 1: sentence from beginning to end, whether an average sentence has
Speaker 1: like twenty words or so average sentence in pros, instead
Speaker 1: of walking those twenty positions, what you're doing is you're
Speaker 1: looking at every word at the same time, but in
Speaker 1: a simpler way. But now you can do that maybe
Speaker 1: five or six times, revising your understanding, and that turns
Speaker 1: out is faster, way faster on GPUs and because of
Speaker 1: this hierarchical nature of language, it's also better.
Speaker 2: So you have this idea, and as I read the
Speaker 2: little note on the paper, it was in fact your idea.
Speaker 2: I know you were working with a t but the
Speaker 2: paper credits you with the idea. So let's let's take
Speaker 2: this idea, this basic idea of look at the whole
Speaker 2: input sentence all at once, yep, a few times, and
Speaker 2: apply it to our frog sentence. Give me, give me
Speaker 2: that frog sentence again.
Speaker 1: The frog did not cross the road because it was
Speaker 1: too tired. Good.
Speaker 2: Tired is good because that's unambiguous.
Speaker 1: Hot could be either one. It could be the road
Speaker 1: or the frog, right, Hot could be hot could be
Speaker 1: the one exactly is in fact hot could either could
Speaker 1: actually either one and non referential and non referential because
Speaker 1: it was too hot outside.
Speaker 2: Outside it could be any of three things, the weather,
Speaker 2: or the frog or the road exactly. I love that
Speaker 2: tired solves the problem. So your model, this new way
Speaker 2: of doing things, how does it parse that sentence, what
Speaker 2: does it do?
Speaker 1: So basically, let's look at the word it and look
Speaker 1: at it in every single step of these you know,
Speaker 1: say a handful of times repeated operation. Imagine you're looking
Speaker 1: at this word it, that's the one that you are
Speaker 1: now trying to understand better, and you now compare it
Speaker 1: to every other word in the sense. Okay, so you
Speaker 1: compare it to the to frog that did not cross
Speaker 1: the road because two and tired, there was two and
Speaker 1: tire and initially in the first past. Already a very
Speaker 1: simple insight the model can fairly easily learn is that
Speaker 1: it could be strongly informed by frog, by road, by nothing,
Speaker 1: but not so by two or by the or maybe
Speaker 1: only to a certain extent by us. But if you
Speaker 1: want to know more about what it denotes, then it
Speaker 1: could be, you know, it could be informed by by
Speaker 1: all of these.
Speaker 2: And just to be clear, that sort of understanding arises
Speaker 2: because it has trained in.
Speaker 1: This way on lots of data.
Speaker 2: It's encountering a new sentence after reading lots of other
Speaker 2: sentences with lots of pronouns with different possible antecedents.
Speaker 1: Yeah, exactly, exactly. So Now the interesting thing is that
Speaker 1: which of the two it actually refers to, doesn't depend
Speaker 1: on only on what those other two words are. And
Speaker 1: this is why you need these subsequent steps because so
Speaker 1: let's talk with the first step. So what now happens
Speaker 1: is that, say the model identifies frog and road could
Speaker 1: have a lot to do with the word it. So
Speaker 1: now you basically copy some information from both frog and
Speaker 1: road over to it, and you don't just copy it,
Speaker 1: you kind of transform it also on the way, but
Speaker 1: you refine your understanding of it. And this is all learned,
Speaker 1: does not given by rules or you know, in any
Speaker 1: way pre specifying.
Speaker 2: Right, just by training on loge, just by training this emergency,
Speaker 2: and so that sort of the meaning of it after
Speaker 2: this first step is kind of influenced by both frog
Speaker 2: and road.
Speaker 1: Yes, both frog and road. Okay, so now we repeat
Speaker 1: this operation again and we now know that it is
Speaker 1: unsure or the model basically now has this kind of superposition. Right,
Speaker 1: it could be road, it could be frog. But now
Speaker 1: in the next step it also looks at tired, and
Speaker 1: somehow the model has learned that when it means something inanimate,
Speaker 1: that tired is not the thing. And so maybe in
Speaker 1: context of tired, it is more likely to refer to frog,
Speaker 1: and now you know, well, it is more likely and
Speaker 1: now maybe the model has figured out already, maybe needs
Speaker 1: a bit more, a few more iterations that it is
Speaker 1: most likely to refer to frog because of the presence
Speaker 1: of tired. So it has solved the problem. But it
Speaker 1: has solved the problem.
Speaker 2: So you do, you have this idea, you try it out.
Speaker 2: There's a detail that you mentioned that's kind of fun,
Speaker 2: and we kind of skipped it, but you mentioned that
Speaker 2: another one of the co authors, who has also gone
Speaker 2: on to do very big things, was about to leave
Speaker 2: Google when you sort of want to test this idea,
Speaker 2: and and that fact that he was about to leave
Speaker 2: Google was actually important to the history of this idea.
Speaker 1: Tell me about that it was important. So this Ilia Plususian,
Speaker 1: he was at the time that this started to gain
Speaker 1: any kind of speed, Elia was managing a good chunk
Speaker 1: of my organization. And the moment he really made the
Speaker 1: decision to leave the company, he had to wait ultimately
Speaker 1: for his co for his co founder, and for them
Speaker 1: to then actually get going together in earnest and so
Speaker 1: he had a few months where he knew and I
Speaker 1: also knew that he was about to leave and where
Speaker 1: you know, the right thing would of course be to
Speaker 1: transition his team to another manager, which we did immediately,
Speaker 1: but where you then suddenly was in a position of
Speaker 1: having nothing to lose and yet quite some time left
Speaker 1: to play with Google's resources and do cool stuff with interesting,
Speaker 1: interesting people. And and so that's one of those moments
Speaker 1: where suddenly your appetite for risk as a researcher just spikes, right, huh,
Speaker 1: because you have, for for a few more months, you
Speaker 1: have these resources at your disposal, you've transitioned your responsibilities.
Speaker 1: At that stage, you're just like, Okay, let's try this
Speaker 1: crazy shit and and and it's and that's literally in
Speaker 1: so many ways, was was one of the integral catalysts
Speaker 1: because that also enabled, right, this kind of mindset of
Speaker 1: we're going for this now, whatever the reason. It still
Speaker 1: you know affects other people. And so there were others
Speaker 1: who joined that collaboration really really early on, who I
Speaker 1: feel were much more excited a result, much more likely
Speaker 1: to really work on this and to really give it
Speaker 1: there all because of his you know, nothing left to lose,
Speaker 1: I'm going to go for this attitude at this.
Speaker 2: Point, Right, was there a moment when you realized it worked.
Speaker 1: There were actually a few moments. And it's interesting because
Speaker 1: on one hand, right, it's a very gradual thing, right,
Speaker 1: And initially, actually it took us many months to get
Speaker 1: to the point where we saw significant first signs of
Speaker 1: life of this not just being a curiosity but really
Speaker 1: being something that would end up being competitive. So there
Speaker 1: certainly was a moment when that started. There was another
Speaker 1: moment when we for the for the first time had
Speaker 1: one machine translation challenge, one language pair of the W
Speaker 1: and T task as it's called, where our score, our
Speaker 1: model performed better than any other single model. The point
Speaker 1: in time when I think all of us realized this
Speaker 1: is special was when we not only had the best
Speaker 1: one in one of these tasks, but in multiple and
Speaker 1: we didn't just have the best number. We also at
Speaker 1: that point were able to establish that we've gotten there
Speaker 1: with about ten times less energy or training compute spend.
Speaker 2: Wow, So you do one tenth the work and you
Speaker 2: get a better result.
Speaker 1: One tenth the work and you get a better result
Speaker 1: not just across one specific challenge, but across multiple including
Speaker 1: the hardest or of one of the harder ones. Right.
Speaker 1: And then at that stage we were still improving rapidly,
Speaker 1: and then you realize, okay, this is for real. There's
Speaker 1: because there right, It wasn't like we it wasn't that
Speaker 1: we had to squeeze those last little bits and pieces
Speaker 1: of gain out of it. It was still improving fairly rapidly,
Speaker 1: to the point where actually, by the time we actually
Speaker 1: published the paper, we again reduced the computer requirements, not
Speaker 1: quite by an entire order of magnitude, but almost right,
Speaker 1: so it still was getting faster and better at a
Speaker 1: pretty rapid rate. Wow, so we had in the paper
Speaker 1: we had some results that were those roughly ten x
Speaker 1: faster on eighthpus and what we demonstrated in terms of
Speaker 1: quality on those eight GPUs by the time we actually
Speaker 1: published the paper properly we were able to do with one.
Speaker 2: GPU, one GPU meaning one chip of the kind that
Speaker 2: people by one hundred thousand of now to build a
Speaker 2: data center exactly. So the paper actually at the end
Speaker 2: mentions other possible uses beyond language for this technology. It
Speaker 2: mentions images, audio, and video, I think explicitly. How much
Speaker 2: were you thinking about that at the time. Was that
Speaker 2: just like an afterthought or were you like, hey, wait
Speaker 2: a minute, it's not just language.
Speaker 1: By the time it was actually published at a conference,
Speaker 1: not just the preprint. By December, we had initial models
Speaker 1: on other modalities on generating images. We had the first
Speaker 1: the first at the stay. At that time they were
Speaker 1: not performing that well yet, but you know, they were
Speaker 1: rapidly getting better. We had the first prototypes actually of
Speaker 1: models working on genomic data, working on protein structure. That's
Speaker 1: good for shadow good for shadowing exactly. But then we
Speaker 1: ended up for a variety of reasons, we ended up
Speaker 1: at first focusing on applications in computer vision.
Speaker 2: The paper comes out, you know, you're working on these
Speaker 2: other applications, you're presenting the paper, it's published in various forms.
Speaker 1: What's the response like. It was interesting because the response
Speaker 1: built in deep learning AI circles basically between the pre
Speaker 1: print that I think came out and I want to
Speaker 1: say June twenty seventeen, and then the actually actual publication,
Speaker 1: to the extent that by the time the poster session
Speaker 1: happened at the conference, there was quite a crowd at
Speaker 1: the poster so we had to be shoved out of
Speaker 1: the out of the hall in which the poster session happened.
Speaker 1: About security and had very hors voices by the end
Speaker 1: of the evening, you guys were like the Beatles of
Speaker 1: the AI conference. I wouldn't say that because we weren't
Speaker 1: the Beatles, because it was really it was still very specific.
Speaker 2: You were more that you were more of the cool
Speaker 2: hipster band. You were the hipster.
Speaker 1: Band, certainly more the cool hipster band. But it was
Speaker 1: an interesting experience because there were some folks and including
Speaker 1: some greats in the field, who came by and said, Wow,
Speaker 1: this is this is cool.
Speaker 2: What has happened since has been wild.
Speaker 1: It seems wild to say the least. Yes, Is it
Speaker 1: surprising to you? Of course, many aspects are surprising. For sure.
Speaker 1: We definitely saw pretty early on already back in twenty eighteen,
Speaker 1: twenty nineteen, that something really exciting was happening here. Now
Speaker 1: I'm still surprised by with the advent of chat GPT,
Speaker 1: something that didn't go way beyond those language models that
Speaker 1: we had already seen a few years before, was suddenly
Speaker 1: the world's fastest growing consumer product.
Speaker 2: Ever, right, I think ever?
Speaker 1: Ever? Yes?
Speaker 2: And by the way, GBT stands for generative pre transformer, right,
Speaker 2: transformer is your word, that's right? So there's an interesting
Speaker 2: I don't know, business side to this right, which is,
Speaker 2: you were working for Google when you came up with this.
Speaker 2: Google presumably owned the idea, had intellectual property around.
Speaker 1: The idea has filed many a patent.
Speaker 2: Was it just a choice Google made to let everybody
Speaker 2: use it? Like when you see the fastest growing consumer
Speaker 2: product in this year of the world not only built
Speaker 2: on this idea, but using the name like and it's
Speaker 2: a different company that was five years later.
Speaker 1: Five years later.
Speaker 2: But a patent's good for more than five years? Is
Speaker 2: that a choice?
Speaker 1: Is that a stret dig choice? What's going on there?
Speaker 1: So the choice to do it in the first place,
Speaker 1: to publish it in the first place, is really based
Speaker 1: on and and rooted in a deep conviction of Google
Speaker 1: at the time, And I'm actually pretty sure it still
Speaker 1: is the case that it is. Actually these developments are
Speaker 1: the tide that floats all votes, that lifts.
Speaker 2: All votes, like a belief in progress, a belief in progress,
Speaker 2: a good old fashioned Now.
Speaker 1: It's also the case that at the time, organizationally, that
Speaker 1: specific research arm was unusually separated from the product organizations.
Speaker 1: And the reason why Brain or in general, the deep
Speaker 1: learning groups were more separated was in part historical, namely
Speaker 1: that when they started out there were no applications and
Speaker 1: the technology was not ready for being applied, and so
Speaker 1: it's completely understandable and just you know a consequence of
Speaker 1: organic developments that when this technology suddenly is on the
Speaker 1: cusp of being incredibly impactful, you're probably still under utilizing
Speaker 1: it internally and potentially also not yet treating it in
Speaker 1: the same way as you would have maybe otherwise treated
Speaker 1: previous trade secrets.
Speaker 2: For example, as it feels like this out their research project,
Speaker 2: not like what's going to be this consumer.
Speaker 1: Product exactly exactly, And to be fair, it took Open
Speaker 1: a Eye in this case a fair amount of time
Speaker 1: and to then turn this into this product, and most
Speaker 1: of that time it also from their vantage point, wasn't
Speaker 1: a product. Right. So up until all the way through
Speaker 1: chat REPT, Open Eye have published all of their GPT developments,
Speaker 1: maybe not all, but you know, their large fraction of
Speaker 1: their work on this.
Speaker 2: Yeah, they're early models.
Speaker 1: The whole models were open exactly. They were more true
Speaker 1: to their name really also believing in the same thing.
Speaker 1: And it was only really after chat GPT and after
Speaker 1: this to them also surprise to a certain extent success,
Speaker 1: that they started to become more closed as well when
Speaker 1: it comes to scientific developments in this past. You'll be
Speaker 1: back in just a minute. Let's talk about your company.
Speaker 1: When'd you decide to start Inceptive? The decision took a
Speaker 1: while and was influenced by events that happened over the
Speaker 1: course of about three months two to three months in
Speaker 1: late twenty twenty, starting with the birth of my first child.
Speaker 1: So when am I was born, two things happened. Number one,
Speaker 1: witnessing a pregnancy and a birth during a pandemic where
Speaker 1: there's a pathogen that's rapidly spreading, and so all of
Speaker 1: that was a pretty daunting experience, and everything went great,
Speaker 1: But having this new human in my arms also really
Speaker 1: made me question if I couldn't more directly affect people's
Speaker 1: lives positively with my work. And so I was at
Speaker 1: the time quite confident that indirectly it would have effect
Speaker 1: also on things like medicine, biology, etc. But I was wondering,
Speaker 1: couldn't this happen more directly if I focused more on it.
Speaker 1: The next thing that happened was that alpha fold two
Speaker 1: results at CAST fourteen were published. CAST fourteen is this
Speaker 1: biannual challenge for protein structure prediction and some other related problems.
Speaker 1: This is the protein folding problem, and this is the
Speaker 1: protein folding problem exactly.
Speaker 2: The machine learning solving the protein folding problem, which had
Speaker 2: been a problem for decades given us chain of amino
Speaker 2: acids predict the three D structure of approach precisely, and
Speaker 2: humans failed and machine learning succeeded.
Speaker 1: Just amazing. Yes, it's a great example. Humans failed despite
Speaker 1: the fact that we actually understand the physics fundamentally, but
Speaker 1: we still couldn't create models that were good enough using
Speaker 1: our conceptual understanding of the processes involve.
Speaker 2: You would think an algorithm would work on that one, right,
Speaker 2: You would just think an old school set of rules,
Speaker 2: like we know what the molecules look like, we know
Speaker 2: the laws of physics. It's amazing that we couldn't predict
Speaker 2: it that way. Right. All you want to know is
Speaker 2: what shape is the protein going to be? You know
Speaker 2: all of the constituent parts, you know every atom in it,
Speaker 2: and you still couldn't predict it with a set of rules,
Speaker 2: but AI machine learning could.
Speaker 1: Amazing, Yes, and it is amazing. Actually, when you put
Speaker 1: it like this, it's important to point out that and
Speaker 1: when we say we understand it, we make massive oversimplifying
Speaker 1: assumptions because we ignore all the other players that are
Speaker 1: present when a protein folds. We ignore a lot of
Speaker 1: the kinetics of it because we say we know the structure,
Speaker 1: but the truth is, we don't know all the wiggling
Speaker 1: and all the shenanigans that happen on the way there, right,
Speaker 1: and we don't know about uh, you know, chaperone proteins
Speaker 1: that are there to influence the folding. We don't know
Speaker 1: around all sorts of other I'm doing the physics one.
Speaker 2: I'm doing the assume a frictionless plane version of protein precisely.
Speaker 1: Precisely, precisely. And the beauty is that deep learning doesn't
Speaker 1: need to make this assumption. AI doesn't need to make
Speaker 1: this assumption. AI it just looks at data, and it
Speaker 1: can look at more data than any human or even
Speaker 1: humanity eventually could look at together. It's such a good
Speaker 1: example problem to demonstrate that these models are ready for
Speaker 1: prime time in this field and ready for lots of applications,
Speaker 1: not just one or two, but men sold, and so
Speaker 1: that happens, so sold exactly. And then the third thing
Speaker 1: was that the COVID mRNA vaccines came out with astonishing
Speaker 1: ninety plus percent out of.
Speaker 2: The gate that they were still so underraty. Under the
Speaker 2: beginning of the pandemic, people were like, it'll be two
Speaker 2: or three years, and if there's sixty percent effective, that'll be.
Speaker 1: Great, exactly exactly, And so everybody forgets. Everybody forgets it.
Speaker 1: And when you look at it, this is a molecule
Speaker 1: family that was for you know, most of the time
Speaker 1: that we've known about it since the sixties, I suppose
Speaker 1: we've treated it like a neglected stepchild of molecular biology,
Speaker 1: because you're talking about marine in general. In general.
Speaker 2: Everybody loves DNA, right, DNA.
Speaker 3: Everybody loves DNA movie star, Yeah, exactly, exactly, even though
Speaker 3: now looking back, DNA is merely you know, the place
Speaker 3: where life takes its notes, maybe the hard drive and
Speaker 3: the memory.
Speaker 1: It's the book, right, it's the book. So but but
Speaker 1: at the end of the day, it was this molecule
Speaker 1: family that was about to save, you know, depending on them,
Speaker 1: tens of millions of lives and in rapid time. So
Speaker 1: all these things hold, but we have no training data
Speaker 1: to apply anything like alpha fold to this specific molecule family,
Speaker 1: no training data to speak of. We had two hundred
Speaker 1: thousand known protein structures at the time, I believe, maybe optimistically,
Speaker 1: we had maybe twelve hundred known RNA structures. And on
Speaker 1: top of that, it was also fairly clear that for
Speaker 1: RNA going directly to function would be much much more important,
Speaker 1: because it's in a certain sense a less strongly structured molecule,
Speaker 1: and other aspects of the molecule might play a bigger role.
Speaker 1: And then on top of that, the attention that generative
Speaker 1: AI was receiving overall, also now in the field of
Speaker 1: pharma or of medicine, was building, And so I ended
Speaker 1: up finding myself in a conversation where very I would
Speaker 1: say wise longtime mentor of mine pointed out that, you know,
Speaker 1: maybe ten years from now or so, somebody could tell
Speaker 1: my daughter that there was this perfect storm where this
Speaker 1: MACLE molecule with no training data was about to save
Speaker 1: the world and could do so much more in the
Speaker 1: direction of positively impacting people's lives. We didn't have training data,
Speaker 1: would be very expensive to create it, but using the
Speaker 1: technology that I've been or technologies that I'd been working
Speaker 1: on for the last I don't know, ten plus years,
Speaker 1: and the ability because of the attention that people were
Speaker 1: now giving to AI in this field the ability to
Speaker 1: raise quite a bit of money. I, in that position,
Speaker 1: chose to stay back at my cushy dream job in
Speaker 1: big tech and not actually take this opportunity to really
Speaker 1: positively impact people's lives, And that idea was not one
Speaker 1: I was willing to entertain.
Speaker 2: You couldn't just coast it out at Google and let
Speaker 2: somebody else go figure out RNA.
Speaker 1: Yeah, and it's not just RNA. I think RNA is
Speaker 1: a great starting point at the end of the day,
Speaker 1: but building models that learn from first of all, all
Speaker 1: the publicly available data that we can possibly get our
Speaker 1: hands on, but also from data that we can reasonably
Speaker 1: effectively create in our own lab. How to design molecules
Speaker 1: for specific functions is something that now is within reach
Speaker 1: and that will in the next years, in the years
Speaker 1: to come, have completely transformational impact on how we even
Speaker 1: think about what medicines are. That any opportunity to speed
Speaker 1: this up, to make this happen, even just a day
Speaker 1: sooner than it could have otherwise happened, is incredibly valuable
Speaker 1: in my opinion.
Speaker 2: As you're talking about this idea that the absence of
Speaker 2: training data is kind of seems to be at the
Speaker 2: center of it, right, It seems to be the core
Speaker 2: yeah problem, which makes sense, right, Like the reason language
Speaker 2: works so well is basically because of the Internet. I know,
Speaker 2: now we're going beyond it, but like it just happened
Speaker 2: to be that there was this incredibly giant set of
Speaker 2: natural life language that became available. We don't have anything
Speaker 2: like that for RNA, so are you. I mean, it's
Speaker 2: kind of step one at inceptive creating the data. Is
Speaker 2: that kind of what's happening?
Speaker 1: So step one that inceptive is learning to use all
Speaker 1: the data or was I think we've made a lot
Speaker 1: of focus in that direction, learning to use all the
Speaker 1: data that is available already and identify what other data
Speaker 1: we're missing, and then see how far we can get
Speaker 1: with just the publicly available data and at the same
Speaker 1: time scale up generating our own data. And it turns
Speaker 1: out that actually, because of the nature of evolution, because
Speaker 1: of how evolution isn't actually incentivized to really explore the
Speaker 1: entire space of possibilities. It is almost always given that
Speaker 1: if you are trying to design exceptional molecules, especially ones
Speaker 1: that are not say, you know, natural formats, you are
Speaker 1: basically gearing need to need novel training in it.
Speaker 2: Yeah, basically you're saying you build RNAs that don't exist
Speaker 2: in the world that have therapeutic uses, and there's no
Speaker 2: kind of definitionally no training.
Speaker 1: Yes, that exist. The funny thing is we have a
Speaker 1: few of them, and so we have existence proofs of
Speaker 1: OURNA molecules, for example, RNA viruses that actually exhibit incredibly
Speaker 1: complex different functions in ourselves, that do all sorts of
Speaker 1: things that we don't usually like. But if we could
Speaker 1: use those, you know, for good, If we could use those,
Speaker 1: you know, in ways that would actually be aimed at
Speaker 1: fighting disease rather than creating them, those kinds of functions,
Speaker 1: even just a small subset of them, would really transform
Speaker 1: medicine already. And so we know it's possible. What are
Speaker 1: you dreaming of when you say that, what are you
Speaker 1: thinking of? Specific? Okay, So, for example, right, one estimate
Speaker 1: is that in order for COVID to infect you, you
Speaker 1: would need potentially as few as five COVID genomes inside
Speaker 1: your organism that's already in five five viral particles. Five
Speaker 1: viral particles. Yeah, you inhale those, you wouldn't have to
Speaker 1: inject it you wouldn't even have to swallow it, you
Speaker 1: inhale them.
Speaker 2: If we could have a medicine that worked as well
Speaker 2: as a disease is a version of your.
Speaker 1: Truth, exactly exactly so at the end of the day, right,
Speaker 1: this medicine is able to spread in your body only
Speaker 1: into certain types of organs and tissues and cells. It
Speaker 1: does certain things there that are really quite complex, right,
Speaker 1: changing the cells behavior again not usually in this case
Speaker 1: in favorable ways, but still in ways that wouldn't have
Speaker 1: to be modified that much in order to potentially be
Speaker 1: exactly what you would need for complex multifactorial medicine. And
Speaker 1: if you could make all of that happen by just
Speaker 1: inhaling five of those molecules, then again, that would completely
Speaker 1: change how you think about medicine. Right, you have viruses
Speaker 1: that aren't immediately active, but that are inactive for long
Speaker 1: periods of time in your organism, and only under certain conditions,
Speaker 1: say under certain immune conditions, really start being reactivated. Why
Speaker 1: can't we have medicines that work in a similar way
Speaker 1: where you actually not only in a vaccination sense, but
Speaker 1: where you take a medicine for a genetic predisposition for
Speaker 1: a certain disease that you are able to take a
Speaker 1: metic design of medicine that you can take and that
Speaker 1: waits until the disease actually starts to develop, and only
Speaker 1: then and only where that disease then starts developed, becomes
Speaker 1: active and actually affects it and potentially also then alarms
Speaker 1: the doctor through a blunt test.
Speaker 2: Like for cancer cells or something. So you have some
Speaker 2: kind of prophylactic medicine in your body and it is
Speaker 2: encoded in such a way that it just hangs out there,
Speaker 2: like herpes, to take a pathological example for example, and
Speaker 2: only in certain settings does it do anything. And those
Speaker 2: settings are if you see a cancer cell, destroy it,
Speaker 2: otherwise just it there precisely.
Speaker 1: And if you can design those also in ways where
Speaker 1: you can just make them all go away. When you know,
Speaker 1: you take a say a completely harmless small molecule, and
Speaker 1: that's again entirely feasible.
Speaker 2: Sure, So, I mean you're dreaming big. These are wonderful
Speaker 2: big you know, science fiction andy dreams that I hope
Speaker 2: you figure them out. On a practical level. What's happening
Speaker 2: at the company right now? How many people work there,
Speaker 2: what are they doing, and what are they figured out
Speaker 2: so far?
Speaker 1: We're round forty. What we're doing is really exactly what
Speaker 1: we just talked about. We're basically scaling data generation experiments
Speaker 1: in our lab that allow us to assess a variety
Speaker 1: of different functions of different mostly RNA molecules actually mostly
Speaker 1: m RNA molecules at the moment, that are relevant to
Speaker 1: a pretty broad variety of different diseases. And so this
Speaker 1: ranges from things like infectious disease vaccines to sell therapies
Speaker 1: that can be applied in oncology or an auto or
Speaker 1: against autoimmune disease. We have mRNAs that we hope will
Speaker 1: eventually be effective in enzyme replacement as enzyme replacement therapies
Speaker 1: for families of a large family of rare diseases, and
Speaker 1: the list goes on. And so we're creating this or
Speaker 1: growing this training data set that eventually, on top of
Speaker 1: foundation and models that we pre trained on all publicly
Speaker 1: available data, allow us to tune those foundation models towards
Speaker 1: designing exceptional molecules for exactly those applications and many more
Speaker 1: sharing similar properties.
Speaker 2: So you basically build new mr and a model molecules
Speaker 2: and test them, and then you give that data to
Speaker 2: your model and presumably it tells you what to build next,
Speaker 2: or it helps you figure out what to build next.
Speaker 2: It's sort of a loop in that way.
Speaker 1: The models are definitely one interesting source for proposals if
Speaker 1: you wish for what to synthesize and test next, they're
Speaker 1: not the only such source, so we basically also explore
Speaker 1: kind of and maybe less guided or heuristically guided ways,
Speaker 1: but exactly so in some of the cases, it's really
Speaker 1: quite iterative. For some of those functions and for some
Speaker 1: of those modalities and diseases or disease targets, we're actually
Speaker 1: already at a point where our models can spit out
Speaker 1: entirely novel molecules that really are unlike anything they've ever
Speaker 1: seen or we've ever seen in nature, that very consistently
Speaker 1: perform quite favorably compared to pretty strong baselines by incumbents
Speaker 1: in the field.
Speaker 2: When you say perform quite favorably compared to baselines by
Speaker 2: incumbents in the field, and does that on some level
Speaker 2: mean better than what experts would think.
Speaker 1: Up, better than what experts can think up, and also
Speaker 1: better than more traditional machine learning tools can easily produce.
Speaker 2: It's like that famous moment in the Go match when
Speaker 2: alpha go made some move that like no human being
Speaker 2: ever would have thought of.
Speaker 1: Yes, so I would say we've long passed the move
Speaker 1: thirty seven in the sense that our understanding of the
Speaker 1: underlying biological phenomena is so incomplete that for most of
Speaker 1: the things that we're able to design for, we don't
Speaker 1: really understand why they happen.
Speaker 2: Huh, when you say weed, you mean at inceptive or
Speaker 2: do you mean just medicine in general?
Speaker 1: I would say just medicine in general.
Speaker 2: Okay, So Inceptive is doing this very kind of high
Speaker 2: level work, right, I mean building what will hopefully be
Speaker 2: the foundation. What's the right amount of time in the
Speaker 2: future to ask about when will we know if it works?
Speaker 2: You think five years?
Speaker 1: So the general idea of using genitive AI and similar
Speaker 1: techniques to generate therapeutics, there are some things in clinical
Speaker 1: trials that were largely designed with AI. As far as
Speaker 1: I know, we're still maybe now we have the first
Speaker 1: trials just now starting for molecules that were truly entirely
Speaker 1: designed by A.
Speaker 2: As opposed to sort of selected from a library.
Speaker 1: Or selected, influenced, exactly selected, adjusted to you, and tweaked,
Speaker 1: et cetera. Right, So that's really still only happening just now,
Speaker 1: but we will see I believe, the first success or
Speaker 1: a first success of such molecules, certainly within the next
Speaker 1: five years.
Speaker 2: What about more narrowly, the project at inceptive.
Speaker 1: It's a similar timeframe. We should be able to get
Speaker 1: molecules into the clinic in the next few years, certainly
Speaker 1: in the next handful of years. Now. These will not
Speaker 1: be molecules with where the objective that we used in
Speaker 1: their design is you know, even remotely as complex or
Speaker 1: you know, kind of the different functions that we're designing
Speaker 1: for are are not going to be even remotely as
Speaker 1: diverse as say what you would find because we used
Speaker 1: this example earlier in ourna virus. These will really be
Speaker 1: more simpler. Those will be molecules that don't do things
Speaker 1: that we couldn't possibly have done before, but that do
Speaker 1: them much better in ways that are more accessible, in
Speaker 1: ways that come with less side effects.
Speaker 2: What biotech largely is is they make protein drugs. And
Speaker 2: so if you could make an mRNA drug where you
Speaker 2: put the m RNA into the body and the body
Speaker 2: makes the protein, it wouldn't be some crazy sleeper cell
Speaker 2: that sits in your body for twenty years or whatever,
Speaker 2: but it might be a more practical alternative to today's biotech drugs.
Speaker 1: Absolutely.
Speaker 2: So you've had a kind of crash course in biology
Speaker 2: in the last few years, yes, And I'm curious, like,
Speaker 2: what is what is something that has been particularly compelling
Speaker 2: or surprising or interesting to you that you have learned
Speaker 2: about biology.
Speaker 1: They're countless things. The biggest one, or the red thread
Speaker 1: across many of them is really just how effective life
Speaker 1: is at finding solutions to problems that on one hand
Speaker 1: are incredibly robust, surprisingly robust, and on the other hand,
Speaker 1: are so different from how we would design solutions to
Speaker 1: similar problems.
Speaker 2: Uh huh.
Speaker 1: That really this comes back to this idea that we
Speaker 1: might just not be particularly well equipped in terms of
Speaker 1: cognitive capabilities to understand biology that basically, you know we
Speaker 1: are we would never think to do it this way,
Speaker 1: and how we think to do it is oftentimes much
Speaker 1: more brittle.
Speaker 2: Uh huh. Brittle is an interesting world, less, less resilient,
Speaker 2: less able to persist under different.
Speaker 1: Conditions, exactly exactly. I mean, you know, we still haven't
Speaker 1: built machines that can fix themselves, for one.
Speaker 2: Which is fundamentally the miracle of being a human being.
Speaker 1: Just fundamentally exactly, exactly exactly and so and of course
Speaker 1: this is true across the scales, right from from you know,
Speaker 1: single cells all the way to complex organisms like ourselves
Speaker 1: and and really just how many also very different kinds
Speaker 1: of solutions life has found and or and or constantly
Speaker 1: is finding. Uh. And you see this all over the place,
Speaker 1: and it's both daunting, humbling, but also incredibly inspiring when
Speaker 1: it comes to applying AI in this area, because again
Speaker 1: I think that at least so far, it's the best
Speaker 1: tool and maybe actually the only tool we have so
Speaker 1: far in face of this kind of complexity. Really design
Speaker 1: interventions that medicines that go way beyond what we were
Speaker 1: able to do or are able to do, just based
Speaker 1: on our own conceptual understanding.
Speaker 2: We'll be back in a minute with the lightning round.
Speaker 2: M hm, let's finish for the lightning round. As an
Speaker 2: inventor of the Transformer model, are there particular possible uses
Speaker 2: of it that worry you flash make you sad?
Speaker 1: I am quite concerned about the p doom doomerism, whatever
Speaker 1: you want to call it, existential fear instilling rhetoric that
Speaker 1: is in some cases actually also promoted by people by
Speaker 1: entities in the space.
Speaker 2: So just to be clear, you're you're not worried about
Speaker 2: the existential risk. You're worried about people talking.
Speaker 1: I'm worried about the about the existential risk being inflated
Speaker 1: or the perception being inflated to the extent that we
Speaker 1: actually don't look enough at some of the much more
Speaker 1: concrete and much more immediate risks. Right. I'm not going
Speaker 1: to say that the existential risk is zero. That would
Speaker 1: be silly.
Speaker 2: What is a concrete an immediate risk that is you
Speaker 2: think under.
Speaker 1: Discuss these large scale models are such defective tools in
Speaker 1: manipulating people in large numbers already today, and it's happening
Speaker 1: everywhere for many, many different purposes by in some cases
Speaker 1: benevolent and in many cases malevolent actors that I really
Speaker 1: firmly believe we need to look much more at things
Speaker 1: like enabling cryptographic certification of human generated content, because doing
Speaker 1: that with the machine generated content is not going to work.
Speaker 1: But we definitely can cryptographically certify human generated content as.
Speaker 2: Such basically watermarking or something some way to say this
Speaker 2: a human made this.
Speaker 1: Exactly what would you be.
Speaker 2: Working on if you were not working in biology on
Speaker 2: drug development?
Speaker 1: Education using using artificial intelligence to democratize access to education.
Speaker 2: What have you seen that has been impressive or compelling
Speaker 2: to you in that regard?
Speaker 1: There are lots of little examples so far and really countless.
Speaker 1: It's what's happening at the con Academy. There are many
Speaker 1: examples of AI applied to education problems in places like China,
Speaker 1: for example. You have a bunch of very compelling examples
Speaker 1: in fiction. A book I really like, like a named
Speaker 1: Neil Stephenson, The Diamond Age or a Young Ladies Illustrated
Speaker 1: primer that I recommend if you.
Speaker 2: Just everybody in AI talks about that, Well now they do.
Speaker 1: Yeah, it's yeah, well.
Speaker 2: Now they do.
Speaker 1: You liked it before?
Speaker 2: It was cool?
Speaker 1: I'm sure at one point I thought it was really
Speaker 1: really important and sure that Neil students know is that
Speaker 1: we are about to be able to build the primary
Speaker 1: and so I ended up having coffee with him to
Speaker 1: tell him, oh, that's great. So at the end of
Speaker 1: the day, maybe the biggest inspiration there is my daughter.
Speaker 1: She's four and a half now, and I think she
Speaker 1: could today read. She can read read okay, but she
Speaker 1: could read, you know, grade school level if she had
Speaker 1: access to you know, an AI tutor teaching her how
Speaker 1: to read?
Speaker 2: Does your daughter use AI use you know, AI chat
Speaker 2: butts not directly without me, But we've.
Speaker 1: Actually used chat GPT to implement an AI reading tutor
Speaker 1: that works reasonably well. I mean we basically, you know,
Speaker 1: kind of as I call it now, vibe coding, vibe coded.
Speaker 1: And I wasn't there for all of it. Took some time,
Speaker 1: but she was there for some of it. Oh, you
Speaker 1: vibe coded it with her? Yeah, well, I mean she was,
Speaker 1: she was there. You know, she witnessed a good chunk
Speaker 1: of it, Yes, although she was more interested in the
Speaker 1: image generation parts. But yeah, we have a sketch of
Speaker 1: one that she quite enjoys. So that's kind of like
Speaker 1: the extent of her at the sage using I directly.
Speaker 1: Yakabust is the CEO and.
Speaker 2: Co founder of Inceptive and the co author of the
Speaker 2: paper Attention Is All You Need. Just a quick note,
Speaker 2: This is our last episode before a break of a
Speaker 2: couple of weeks, and then we'll be back with more episodes.
Speaker 2: Please email us at problem at Pushkin dot fm. We
Speaker 2: are always looking for new guests for the show. Today's
Speaker 2: show was produced by Trinamanino and Gabriel Hunter Chang. It
Speaker 2: was edited by Alexander Garretton and engineered by Sarah muguerrett