Speaker 1: Pushkin. Humans are made of proteins. Proteins are key components
Speaker 1: of our cells and of our muscles. Proteins regulate gene
Speaker 1: expression and the immune system. And yet forever we had
Speaker 1: no idea what most proteins look like, and this was
Speaker 1: a problem. Every protein has a different shape, and to
Speaker 1: understand how any particular protein works, how it interacts with
Speaker 1: other molecules, how it keeps us healthy or causes disease,
Speaker 1: it is very helpful to understand what that particular shape is,
Speaker 1: but determining that complicated three dimensional shape was really hard.
Speaker 1: Scientists sometimes spent years trying to determine the shape of
Speaker 1: a single protein. So people started to dream this. They thought,
Speaker 1: what if we could come up with some kind of
Speaker 1: a system, some way to use a protein sequence of
Speaker 1: amino acids to reliably predict that protein's unique three dimensional shape.
Speaker 1: It would be a huge leap forward that could lead
Speaker 1: to a much deeper understanding of biology and a new
Speaker 1: wave of treatments for disease. This idea was called the
Speaker 1: protein folding problem. But after decades of work, the best
Speaker 1: protein folding models were nowhere near good enough to be
Speaker 1: scientifically useful. And then in twenty twenty a group of
Speaker 1: researchers built an AI model to try to tackle the
Speaker 1: protein folding problem, and their model was so much better
Speaker 1: than what had come before that some people thought they
Speaker 1: were cheating. In fact, they were not cheating. They had
Speaker 1: solved the protein folding problem. I'm Jacob Goldstein, and this
Speaker 1: is What's Your Problem, the show where I talk to
Speaker 1: people who are trying to make technological progress. My guest
Speaker 1: today is push Meat Coli. He's vice president of research
Speaker 1: at deep Mind, an AI research group that's part of Google.
Speaker 1: Push Me was part of the deep Mind team that
Speaker 1: solved the protein folding problem. They built an AI model
Speaker 1: called alpha fold, and alpha fold is one of the
Speaker 1: most impressive real world AI success stories that we have
Speaker 1: seen so far, and as you'll hear in our conversation,
Speaker 1: alpha fold holds lessons for AI that go beyond protein folding.
Speaker 1: One other thing you may hear in our conversation, by
Speaker 1: the way, is the occasional background beep from push meat
Speaker 1: smoke detector. Have you failed to change the battery in
Speaker 1: your smoke detector? Is that perhaps what that little chirp was?
Speaker 1: You really should do that for yourself as.
Speaker 2: Well as I know, I mean, I was trying to
Speaker 2: fix it before the recording, and like it's just so finicky,
Speaker 2: it doesn't come.
Speaker 1: Out okay, fair, fair, well with it. It makes you real.
Speaker 1: And so it is the case with protein folding that
Speaker 1: it's not just like people publishing papers. Right, There was
Speaker 1: actually this contest that was held was it every couple
Speaker 1: of years of exactlys trying to solve the protein folding problem?
Speaker 1: And this had been going on for a long time,
Speaker 1: And so is there a first moment when you compete
Speaker 1: in this contest?
Speaker 2: Yeah? So this is like an amazing thing about protein
Speaker 2: structure prediction and how visionary the community was. Like they
Speaker 2: had set up this amazing sort of fool proof way
Speaker 2: to evaluate progress, because it's very easy to sort of
Speaker 2: for sometimes scientists to fool themselves saying oh that's progress,
Speaker 2: there is no progress, right, So they had set up
Speaker 2: this contest in a very remarkable way where they had said, well,
Speaker 2: for a specific duration of time, any sort of scientists
Speaker 2: all over the world who are discovering new structures would
Speaker 2: not share them with the world. Instead, instead, they will
Speaker 2: basically send it to a secret want script vault.
Speaker 1: I love a secret fault.
Speaker 2: Yeah, yeah, I mean not like secret.
Speaker 1: I wanted to be a big vault with the wheel
Speaker 1: you turn, but yeah, I understand.
Speaker 2: Yeah, And so the idea there was that nobody except
Speaker 2: the scientists who discovered the structure knew what is the
Speaker 2: structure of this protein.
Speaker 1: So it's a perfect way to test these models doing
Speaker 1: the prediction because somebody knows the answer. But the people
Speaker 1: building the models, people like you, people like deep mind,
Speaker 1: don't actually know the answer. So you can't cheat, you
Speaker 1: can't backfit it or anything like.
Speaker 2: That exactly, right, So you don't know how good you
Speaker 2: are because like you've been training on known examples and
Speaker 2: you've been evaluating them on known examples. But when you
Speaker 2: are tested, you are tested on these amazingly new things
Speaker 2: that nobody has seen before.
Speaker 1: Yeah. Yeah, so okay. And they actually have like a
Speaker 1: numeric score that they assigned to everybody's model, right, yeah,
Speaker 1: Like it's very quantitative and it's not just like good
Speaker 1: or pretty good. It's a number, right, And what is it?
Speaker 1: Zero two one hundred?
Speaker 2: Is that the scale? Yeah, it's zero two hundred, Like
Speaker 2: that's the sort of scale. And if you look at
Speaker 2: progress in the last sort of twenty years before alpha
Speaker 2: fold one was launched. I mean it was somewhere sort
Speaker 2: of between the twenty five to forty sort of GDT
Speaker 2: sort of.
Speaker 1: Five to forty. Was it getting better slowly or was
Speaker 1: it just kind of stuck in the thirties more or less?
Speaker 2: Yeah, it was stagnating. It was like sometimes it would
Speaker 2: go to thirty eight, sometimes thirty five, and it was
Speaker 2: like in that so it was going up and down,
Speaker 2: up and down. There was no sort of remarkable sort
Speaker 2: of breakthrough.
Speaker 1: And was it all AI was like the only way
Speaker 1: people were trying to solve it? AI? Were there whole
Speaker 1: other sort of things people were thinking about trying to do.
Speaker 2: Yeah, so this is mostly not AI based solutions, right,
Speaker 2: These were sort of very well designed, hand designed systems
Speaker 2: that were carefully tuned to the problem over many, many
Speaker 2: decades with large teams working together and so on. But
Speaker 2: it was a little machine learning.
Speaker 1: So they'd been scoring in the thirties more or less.
Speaker 1: And then what year is it that deep mind? You
Speaker 1: and deep Mind show up with alpha fourth one?
Speaker 2: So twenty eighteen. So the contest actually it runs in
Speaker 2: the summer and.
Speaker 1: Then at the end of the summer they sort of
Speaker 1: give you the results or what so.
Speaker 2: At the end of the summer. I mean like by
Speaker 2: by I think July or August, they have sent the
Speaker 2: last files and then you have sent them the results
Speaker 2: and then you wait, right and you don't know what
Speaker 2: has happened. Then they invite you to a conference which
Speaker 2: happens in December, so you are like eagerly waiting what's
Speaker 2: what has happened? And they're like, oh, maybe we came last,
Speaker 2: maybe maybe we're in the middle. And then they actually
Speaker 2: revealed the leaderboard or the scores in the conference.
Speaker 1: Where were you at the time.
Speaker 2: I was in London, I was in the office. I
Speaker 2: was like really waiting, like trying to figure out like
Speaker 2: what where where were we right in terms of the career?
Speaker 2: How did we form? And we get an email from
Speaker 2: the organizers one day' hold the results are about two
Speaker 2: thounds and they say, well you are first and by
Speaker 2: a big margin, so like from thirty to forty we
Speaker 2: had gone to more than sixty.
Speaker 1: You did way better at predicting the structure of protein
Speaker 1: than anyone had ever done, including that.
Speaker 2: Yes you won by a lot, Yes we won by
Speaker 2: a lot.
Speaker 1: So that's way better than anyone has ever done. But
Speaker 1: does it mean that you're basically half right, is that
Speaker 1: what that number means?
Speaker 2: Yeah, so I think the way we I mean we
Speaker 2: are you're the best in the world, but still your
Speaker 2: predictions are pretty much sort of not very useful for
Speaker 2: any like if you're trying to figure out what a
Speaker 2: drug buying to this particular protein or like, the error
Speaker 2: is so much right that you wouldn't get a complete
Speaker 2: picture of the protein.
Speaker 1: So this sixty number, it's like good in that it's
Speaker 1: better than anyone has ever done. It's bad in that
Speaker 1: it's not scientifically useful. What number do you have to
Speaker 1: get to to be scientifically useful?
Speaker 2: Like between eighty five to ninety that's what people told
Speaker 2: us that if you get beyond eighty five to ninety,
Speaker 2: then the problem is solved.
Speaker 1: So what do you decide when you when you get
Speaker 1: this result.
Speaker 2: So we get this result and we're like, yeah, this
Speaker 2: is amazing, right, that we are the best in the world,
Speaker 2: right by a big margin. Right, So like the thesis
Speaker 2: that machine learning sort of will advanced science. Oh that's great,
Speaker 2: but the problem is not solved. And let's go back
Speaker 2: to the drawing board. And now with the information that
Speaker 2: we have in the amount of time, we have spent
Speaker 2: on this previous architecture, do we still think that this
Speaker 2: will lead us to where we want to go? And
Speaker 2: the teams thought, no, we need no completely, Yeah, we
Speaker 2: need to completely start from scratch.
Speaker 1: So your your reaction to winning this contest and doing
Speaker 1: better than anyone has ever done it predicting protein folding
Speaker 1: is let's blow up this thing that just won the contest.
Speaker 2: Yeah, throw it away. Yeah, we were like, the basic
Speaker 2: premise was proved that machine learning has a role to play, right,
Speaker 2: So that gave us a lot of confidence. But at
Speaker 2: the same time we saw it, well, this is not
Speaker 2: an elegant This is not an elegant solution, right. This
Speaker 2: is this is like two modules, Like there's machine learning,
Speaker 2: there's a machine learning module. It is making these sort
Speaker 2: of predictions which this other module is sort of trying
Speaker 2: to use. If you believe in the power of machine learning,
Speaker 2: let's do end to end, right, Let's do end to
Speaker 2: end and basically do everything so that the model takes
Speaker 2: care of it, right, rather.
Speaker 1: Than clear what was happening with that initial model that
Speaker 1: you were deciding to abandon, that.
Speaker 2: Was basically using the machine learning model in together with
Speaker 2: sort of a known framework. Right that there is a
Speaker 2: second step that was a conventional sort of step.
Speaker 1: Oh I see. So it's like you weren't all in
Speaker 1: on machine learning. You were like, well, we gonna use
Speaker 1: machine learning, but we're gonna still do this kind of
Speaker 1: the way other people have been doing it. And your
Speaker 1: response to that first result was screw it, Let's not
Speaker 1: do anything like everybody's not before. Let's just go all
Speaker 1: in on machine learning beginning to end exactly interesting. And
Speaker 1: so you do that, and you spend what two years?
Speaker 1: Is it two years between? Do I have that right?
Speaker 2: Yeah?
Speaker 1: Yeah, and then you come back in twenty twenty. You
Speaker 1: come back in twenty twenty, there's another one of these contests. Yeah,
Speaker 1: you got your new end end machine learning model.
Speaker 2: Yeah. So it was the pandemic. So this is twenty twenty, right.
Speaker 1: So nobody's gone anywhere.
Speaker 2: Yeah, nobody's going anywhere. Right. We knew like twenty nineteen
Speaker 2: was basically where we started working with this new model,
Speaker 2: and it was really tough going because we were like
Speaker 2: starting from we're starting from twenty, right, so we went
Speaker 2: at sixty. Now we are starting from twenty and twenty
Speaker 2: thirty forty. And sometimes you would stagnate at forty five
Speaker 2: fifty you were like, really should I should I had
Speaker 2: that figure model?
Speaker 1: Yeah?
Speaker 2: Yeah, So twenty by the end of twenty nineteen thought
Speaker 2: we started getting some really really cool results and we thought, okay,
Speaker 2: now we have surpassed we have definitely surpassed the previous model.
Speaker 2: We're in good territory. And we were very excited. Like
Speaker 2: at the start of twenty twenty, we were like, yeah,
Speaker 2: making progress, and then the pandemic hit.
Speaker 1: In a minute, the model gets an unanticipated test in
Speaker 1: the real world.
Speaker 2: There was this new virus that was reported sarskov two
Speaker 2: and one of the first things so somebody sort of
Speaker 2: figured out the structure of the spike protein. It was
Speaker 2: all over the newspapers, like, here's a spike protein of
Speaker 2: this new virus, but all the other proteins of the virus,
Speaker 2: the accessory proteins, nobody knew the structure. So the first
Speaker 2: we did we thought, I think we think we have
Speaker 2: the best model in the world. We should be making
Speaker 2: these predictions and sharing it with the world, but is
Speaker 2: this the right thing to do. So we spent a
Speaker 2: lot of time reaching out to biologists who looked at
Speaker 2: the prediction and said well, you need to share this,
Speaker 2: you need to share this with the world. So the
Speaker 2: start of twenty twenty was us sort of sharing the
Speaker 2: predictions from this untested model with the world because we
Speaker 2: thought they were quite good. And then throughout twenty twenty
Speaker 2: we took part in the assessment right, which ran in
Speaker 2: the summer of twenty the contest.
Speaker 1: In the contest.
Speaker 2: Exactly normally, right, the organizers don't come back to you.
Speaker 2: They just released the results at the end in December,
Speaker 2: And at the end of the summer we get this
Speaker 2: funny email saying we want to talk to you, and
Speaker 2: so we were like, yeah, like, did we do anything bad?
Speaker 2: What happened? And a few of them really had sort
Speaker 2: of suspicions. They were like, you must have cheated, right,
Speaker 2: like that your predictions are your level of performance is
Speaker 2: nowhere close to anything that we have seen ever. Right,
Speaker 2: But a few scientists in that contest had submitted a
Speaker 2: sequence a protein whose structure was not known. They were
Speaker 2: expecting that the structure would be known by the time
Speaker 2: the contest en so we'll be able to evaluate the predictions.
Speaker 2: But that structure was not known, and in fact, the
Speaker 2: structure they couldn't find. The structure.
Speaker 1: So you're saying it would be impossible to cheat because
Speaker 1: literally no human knew the structure. No way to cheat,
Speaker 1: nobody knows the answer.
Speaker 2: Yeah, yeah, So they used the prediction of alpha fold
Speaker 2: and then tried to explain their experimental data and it matched.
Speaker 2: And they are like, this model has been able to
Speaker 2: discover something that nobody knew, not even no scientists knew.
Speaker 2: Sense the model had already made new biological discoveries even
Speaker 2: before we knew it.
Speaker 1: Yeah, yeah, okay, so that's good. You're not in trouble anymore.
Speaker 1: It's clear you didn't cheat. Do they say the number?
Speaker 1: What's the number? I'm waiting for the number. How'd you do?
Speaker 2: Yeah, so we were beyond eighty five and ninety right,
Speaker 2: and then they basically said, okay, we have to announce
Speaker 2: it to the world. And so come December that was
Speaker 2: the announcement that was made by the organizers that this
Speaker 2: alpha fold too had solved the protein structure prediction problem.
Speaker 1: So is that contest done? Now? Did you just end
Speaker 1: that contest? Is nobody doing that anymore?
Speaker 2: No, the contest sort of is alive, right, it has changed,
Speaker 2: its focus has changed. So what what alpha fold two
Speaker 2: did was find the structure of these single proteins, But
Speaker 2: there are many other problems that remain, right, how do
Speaker 2: multiple proteins interact? Instance, Right, So there are other structure predictions,
Speaker 2: problems that now the contest is sort of has evolved to, right,
Speaker 2: it is sort of focusing on other types of problems
Speaker 2: that Alpha fold two did not address.
Speaker 1: If we zoom out and think about what you have done,
Speaker 1: what the team has done in you know, using machine
Speaker 1: learning to solve this scientific problem that people had been
Speaker 1: working on for a long time, Like what are the
Speaker 1: broader lessons? Like if we think about other domains, what
Speaker 1: can we infer what can we take from this?
Speaker 2: Yeah, so I think the thing that we can take
Speaker 2: from this is basically science is sort of generating a
Speaker 2: lot of data across any domain that you see, right,
Speaker 2: whether it's genomics, hydergy, physics, whether the amount of data
Speaker 2: that we are gathering about the world is much more
Speaker 2: than any single human mind can comprehend.
Speaker 1: Right.
Speaker 2: You can have the best scientists and they will not
Speaker 2: be able to sort of go through on the data
Speaker 2: that we are collecting about our world. So machine learning
Speaker 2: is this remarkable sort of tool which gives us the
Speaker 2: ability to make sense and leverage this data, right and
Speaker 2: really sort of on the path of really accelerating our
Speaker 2: understanding of the problems that we're dealing with.
Speaker 1: In the case of alpha fold, was the sort of
Speaker 1: input data the known protein structures and amino acid sequence
Speaker 1: and was that the basic training data exactly right?
Speaker 2: So it was the PDB, which was the protein database,
Speaker 2: and that had been collected by the community for many,
Speaker 2: many years, right over many decades. They have meticulously carefully
Speaker 2: deposited all the protein sequences and the corresponding structures that
Speaker 2: were discovered, right, And it had one hundred and fifty
Speaker 2: thousand examples at that time, right, sequences as well as structures,
Speaker 2: and everyone had access to the same data. Right, All
Speaker 2: the teams were training on that data.
Speaker 1: Is it right that alpha fold itself is open sourced
Speaker 1: and that there's this open source database of protein structures
Speaker 1: that have been discovered with alpha fauld? Is that right?
Speaker 2: Yeah? So when the sort of developed alpha fold, we
Speaker 2: made it available to the world. But we then said, well,
Speaker 2: it's so accurate, but it's also so fast that we
Speaker 2: will use it to find the structure for every sort
Speaker 2: of known protein. And then we made all those structures
Speaker 2: available to the world.
Speaker 1: Alpha fold has now made the structures of roughly two
Speaker 1: hundred and fifty million different proteins publicly available. We'll be
Speaker 1: back in a minute with the lightning ground. Last thing
Speaker 1: is a lightning round. Just some fast questions, okay, and
Speaker 1: then we'll be done. What's your favorite protein?
Speaker 2: Himoglobin?
Speaker 1: Why?
Speaker 2: It is sort of very pleasant to look at it. It
Speaker 2: is very symmetric, it has there and you can see
Speaker 2: it's purpose right that and the oxygen binds into that
Speaker 2: thing right from very clean protein.
Speaker 1: It's so easy to understand. It's the little thing that
Speaker 1: carries oxygen around your body. If everything goes well, what
Speaker 1: problem will you be trying to solve in say, five years.
Speaker 2: Really sort of thinking about the two big challenges sort
Speaker 2: of that humanity is facing. One is the pandemic, the
Speaker 2: other is climate change. And I think material science and
Speaker 2: quantum chemistry can impact both, but especially climate change. And
Speaker 2: I think this is something that requires a lot of work.
Speaker 1: Is there some particular problem in that domain that is
Speaker 1: analogous to protein folding? Is there some hard thing that
Speaker 1: you want to figure.
Speaker 2: Out rational material design? We are very far from there.
Speaker 2: We are still basically doing experimental stuff when we think
Speaker 2: about discovering new materials.
Speaker 1: What do you understand about AI or machine learning that
Speaker 1: most people don't understand.
Speaker 2: I think sort of AI is not magic, right, it's
Speaker 2: sort of essentially it's a series of techniques which is
Speaker 2: able to extract intelligence. But you extract intelligence from the
Speaker 2: raw material, right, So so garbage and garbage out. So
Speaker 2: what is really important is that experience need needs to
Speaker 2: be rich enough. Right, We can't just we don't become
Speaker 2: intelligent by sitting in the room, right. We become intelligent
Speaker 2: because we have amazing experiences. So it's not big data, right,
Speaker 2: it's not the bigness of the experience, but it's like
Speaker 2: the goodness of the experience, like the wide variety of
Speaker 2: sort of things that you train on and the things
Speaker 2: that you see. So I think that's very really that's
Speaker 2: really important.
Speaker 1: That thought leads you to like the optimal training data.
Speaker 1: So it's the worry that like people are making a
Speaker 1: mistake by just doing a lot of the same kind
Speaker 1: of training data.
Speaker 2: Yeah, exactly, exactly right. So if you just take one example,
Speaker 2: you repeat it multiple times, right, that's not that's not great. Again,
Speaker 2: you don't become Yeah, you don't become wise doing the
Speaker 2: same thing again and again and again.
Speaker 1: Right, what are you actually working on right now? Like
Speaker 1: what are you going to go work on today or
Speaker 1: next week.
Speaker 2: So there is a system that my team developed called Synthide,
Speaker 2: which is a system for watermarking AA generated content. So
Speaker 2: we want to be able to detect it. When you
Speaker 2: have a generated content, users should be able to detect
Speaker 2: that this is educated.
Speaker 1: Generated content, whether it's images or words or whatever, text, video.
Speaker 2: Exactly exactly. You embed this imperceptible thing within the thing
Speaker 2: that is generated that a human might not see.
Speaker 1: So the builder of the AI model Open AI could
Speaker 1: choose to embed a watermark in GPT, so that anybody
Speaker 1: who made a thing with GPT, that document would have
Speaker 1: some hidden sign that it was AI generated. It's sort
Speaker 1: of the choice of the model of builders. Yeah, thank
Speaker 1: you very much for your time. It was great to
Speaker 1: talk with you.
Speaker 2: Yeah, thanks you good. It was a pleasure.
Speaker 1: Pushmikkoli is vice president of research at Google deep Mine.
Speaker 1: Today's show was produced by Edith Russello and edited by
Speaker 1: Karen Chakerje. You can email us at problem at Pushkin
Speaker 1: dot FM. I'm Jacob Goldstein.