Speaker 1: Welcome to tech stuff.
Speaker 2: I'm Os Vloscian, our guest today is a world renowned
Speaker 2: expert in the field of AI. In fact, he literally
Speaker 2: wrote the book on it. Stuart Russell, together with co
Speaker 2: author Peter Norvig, released Artificial Intelligence, A Modern Approach back
Speaker 2: in the mid nineteen nineties. Since then, it's been translated
Speaker 2: into fourteen languages and is used in fifteen hundred universities
Speaker 2: in one hundred and thirty five countries. But just because
Speaker 2: he wrote the book doesn't mean he can't criticize the technology, or,
Speaker 2: more specifically, the way it's being rolled out, often without
Speaker 2: regard to consequence. Stuart has spent the past few years
Speaker 2: sounding the alarm about the lack of AI safety protocols
Speaker 2: and what he sees as the very real possibility that
Speaker 2: our current path could lead to human extinction. The conversation
Speaker 2: comes at a timely moment, as the US government is
Speaker 2: forcefully cracking down on who can and can't use the
Speaker 2: leading models coming from anthropic Today. Stuart is a professor
Speaker 2: of computer science at Berkeley and the president of the
Speaker 2: International Association for Safe and Ethical AI. He also recently
Speaker 2: took the stand as the only expert witness on AI
Speaker 2: for Elon Musk in his lawsuit against Samaltman and open AI.
Speaker 1: Stuart, welcome to tech stuff.
Speaker 3: It's a pleasure to be with you.
Speaker 1: I had to ask you. What did you say on
Speaker 1: the stand?
Speaker 4: Well, it's more what I was not allowed to say.
Speaker 4: I was not allowed to talk about anything related to
Speaker 4: existential risk, which was surprising because open ai was set
Speaker 4: up for that reason. Musk and others were concerned that
Speaker 4: if AI were in the hands of for profit companies
Speaker 4: that they would disregard safety and put him at risk,
Speaker 4: so Open a Eye was created.
Speaker 3: To counter that.
Speaker 4: The judge said, no, you can't talk about that, which
Speaker 4: was disappointing.
Speaker 1: Why did the judge say you couldn't talk about existential risk?
Speaker 3: I was not privy to those discussions either.
Speaker 4: Maybe the defense lawyers thought it would be prejudicial, that
Speaker 4: it was in some sense speculative because it hasn't we
Speaker 4: haven't yet gone extinct, right, And this is a this
Speaker 4: is a strange argument I hear from a lot of people.
Speaker 4: You know, it's just science fiction. Well what do you
Speaker 4: mean by that, Well, it hasn't happened yet. You know,
Speaker 4: everything that's ever happened. There was a time before it
Speaker 4: happened for it, and by your nothing could ever happen
Speaker 4: because everything was at some point science fiction. But you know,
Speaker 4: when you look at science fiction, you know, they talked
Speaker 4: about nuclear weapons in nineteen twelve HG. Wells, they talked
Speaker 4: about space travel in the nineteenth century, and in fact,
Speaker 4: they talked to AI in the nineteenth century. Samuel Butler
Speaker 4: wrote a book that described society where there had been
Speaker 4: an enormous conflict between those who were in favor of
Speaker 4: the machines and those who predicted that the machines would
Speaker 4: would be the ruin of the human race.
Speaker 2: Now is Open AI particularly bad? I mean, I know
Speaker 2: you are asked by their counseling cross you know if
Speaker 2: you believe that the for profit sort of motivation to
Speaker 2: recklessly develop AI by definition and endanger's you know, humanity,
Speaker 2: surely that also applies to Elon and XAI and SpaceX.
Speaker 2: And you said, if that hypothesis is correct, then yes,
Speaker 2: In other words, that the same critique could apply to
Speaker 2: you know, Open AI or Anthropic or Google or SpaceX.
Speaker 3: Yeah.
Speaker 4: It was specifically not my job to compare the safety
Speaker 4: records of different companies or their safety positions. I think
Speaker 4: if I understand it correctly. Anthropic is a public benefit corporation,
Speaker 4: which is one way of allowing considerations other than profit
Speaker 4: to affect the decisions made by management and the board,
Speaker 4: because you know, for a regular for profit company, there
Speaker 4: is a legal obligation to maximize shareholder return. And what's
Speaker 4: happening here is that the risks imposed on the rest
Speaker 4: of humanity are externalities, as economists call it, which means
Speaker 4: that someone is making decision and there's a bad consequence
Speaker 4: that is being loaded onto somebody else. So you know,
Speaker 4: you think about chemical companies who skimp on safety, and
Speaker 4: as it stands, according to the companies, these are all externalities,
Speaker 4: meaning they don't accept responsibility for these consequences. Those are
Speaker 4: harms that don't fake into their balance sheet, and so
Speaker 4: the same would be true in a sense for human extinction.
Speaker 3: And even liability would not.
Speaker 4: Really be a deterrent for that right because obviously you
Speaker 4: wouldn't be around today to pay the compensation. So they
Speaker 4: just sort of factor that out of their decision making.
Speaker 4: And that's exactly what open AI was set up to avoid.
Speaker 4: But now it's with the transition to a for profit entity.
Speaker 4: It's part of that calculus So take.
Speaker 1: Us back in time.
Speaker 2: In nineteen ninety five, you wrote this textbook that became
Speaker 2: the defining textbook on AI, and you came up with
Speaker 2: a concept called the standard model. Can you explain what
Speaker 2: that is and how it relates to the conversation which
Speaker 2: you're now so engaged in today as to the potential
Speaker 2: extinction of the human race because of AI.
Speaker 3: Yeah.
Speaker 4: So if we go back even further to the beginnings
Speaker 4: of the field of AI in the post war period,
Speaker 4: so nineteen forties, nineteen fifties, I think everyone agreed that
Speaker 4: AI is about creating intelligence in machines. What wasn't clear
Speaker 4: was well, what is intelligence? How do we define this
Speaker 4: target and how do we go about doing it? So
Speaker 4: there was an active debate you should we try to
Speaker 4: emulate human intelligence, Should we understand what's going on in
Speaker 4: the human mind, the human brain and then sort of
Speaker 4: go ahead and implement that, or should we focus on
Speaker 4: a more abstract notion, in fact, a notion that philosophers
Speaker 4: had developed for thousands of years and economists as well,
Speaker 4: this idea of rational behavior that an entity is intelligent
Speaker 4: if it acts in a way that is expected to
Speaker 4: achieve its objectives. And I would say, for the most
Speaker 4: part that second approach one out because it doesn't require
Speaker 4: doing psychological experiments on humans to find out how their
Speaker 4: brains work. Right, It's an abstract mathematical concept, and we
Speaker 4: had tools. We had formal logic so that we could
Speaker 4: create algorithms that were able to construct plans to achieve goals.
Speaker 4: So that these two the two views are I would say,
Speaker 4: the sort of rational view, i e. Base intelligence on
Speaker 4: formal foundations of how one should reason, how one should
Speaker 4: make decisions, versus a more biology based view, which is
Speaker 4: neurons and their computational analogs. And so the standard model
Speaker 4: refers to this. What was dominant I would say from
Speaker 4: like nineteen sixty to twenty ten, twenty twenty some where
Speaker 4: it was much more the rational view entities are intelligent
Speaker 4: to the extent that their actions can be expected to
Speaker 4: achieve their objectives. And then it was really the twenty
Speaker 4: twelve work that Jeff Hinton did with Ilia Sutzkiver and
Speaker 4: I think Alex Kruzhevski on a system for recognizing objects
Speaker 4: in images that significantly exceeded the methods that other people
Speaker 4: had developed before that. And then it was off to
Speaker 4: the races, and then language models came along, so applying
Speaker 4: somewhat similar ideas again large neural networks that were trained
Speaker 4: from vast amounts of data, applying that to text and
Speaker 4: then generating eventually CHAT, GPT and all the successors that
Speaker 4: we've seen since then.
Speaker 2: I guess the question I'm coming to is was the
Speaker 2: victory for the time being, at least of the neural
Speaker 2: net type of AI. The reason why you became so
Speaker 2: concerned about AI safety or AI safety in nineteen ninety
Speaker 2: five when you were writing this book was the possibility
Speaker 2: of machines having their own goals which could be very
Speaker 2: different to ours and lead to our extinction already on
Speaker 2: your mind. In other words, did the evolution of the
Speaker 2: technology make the safety issue more urgent?
Speaker 1: And if so, why so?
Speaker 4: In ninety five, when I published a book with Peter,
Speaker 4: we have a section called what if we do succeed
Speaker 4: and it points out that if we create machines more
Speaker 4: intelligent than us, which is what we were trying to do,
Speaker 4: we might face this problem that we wouldn't have any
Speaker 4: idea how to control them, that they would in some
Speaker 4: sense have more power than we do because they're more
Speaker 4: intelligent and that's why we have power over all the
Speaker 4: other species, and then we would be in that same
Speaker 4: inferior situation. Just a few weeks ago, Anthropic put out
Speaker 4: a blog post saying this is happening. We are experiencing
Speaker 4: what's called now recursive self improvement or RSI, and Anthropic
Speaker 4: itself called for a worldwide halt on further development of AI.
Speaker 2: Why are you one of the greatest voices urging caution
Speaker 2: right now in twenty twenty six when you weren't in
Speaker 2: nineteen ninety five?
Speaker 3: Is that?
Speaker 2: Is that because essentially of this improvement in computational powers.
Speaker 1: Has sign changed in you or sign changing the environment or.
Speaker 3: Both, I think both.
Speaker 4: In nineteen ninety five, if you go back and read
Speaker 4: that section of the book, it sort of says, well,
Speaker 4: you know, it's a long way off.
Speaker 3: It's hard to really.
Speaker 4: Predict what's going to happen. You know that maybe there's
Speaker 4: reason to be cautiously optimistic. It's very agnostic about whether
Speaker 4: to take this seriously.
Speaker 3: And what changed in me.
Speaker 4: In around twenty thirteen, so I was on sabbatical in Paris,
Speaker 4: and I just started thinking more about whether we could
Speaker 4: succeed right, whether we could really produce superintelligence, and I
Speaker 4: became convinced that we were close to being able to
Speaker 4: have a roadmap. So a roadmap meaning here are a
Speaker 4: series of engineering challenges that we can apply ourselves to,
Speaker 4: and if we knock those over one by one, we'll
Speaker 4: get to super intelligence.
Speaker 2: So this was twenty thirteen, and you were both excited
Speaker 2: but also scared.
Speaker 1: Right, I'm trying to capture as that.
Speaker 2: I mean that there's the irony that you and Jeffrey Hinton,
Speaker 2: these two great pioneers of modern AI, are now the
Speaker 2: two loudest voices in the world, arguably about AI safety
Speaker 2: and the risk of human extinction and like why is that?
Speaker 2: Like what is what helped me understand?
Speaker 4: And your show, Benjo, the other godfathers are deep learning,
Speaker 4: as they're often called in my case. So back in
Speaker 4: twenty thirteen, seeing that we might be able to achieve superintelligence,
Speaker 4: but also realizing that the standard model that we were
Speaker 4: working in was basically flawed. Because remember what standard model
Speaker 4: says a machine is intelligent to the extent that its
Speaker 4: actions can be expected to achieve its objectives. Right, And
Speaker 4: there are lots of systems we've built, so when you
Speaker 4: use your GPS navigation in your car, you say, you know,
Speaker 4: you take me to the airport. You know, it figures
Speaker 4: out the best route to get to the airport. So
Speaker 4: you're providing the objective and it's providing the solution, right.
Speaker 3: You know, we write.
Speaker 4: Chess programs, we basically tell the chess program what checkmate
Speaker 4: is and then it figures out how to play the game.
Speaker 4: So this notion is very very powerful, right that that
Speaker 4: we specify objectives and then we create this sort of
Speaker 4: optimal machinery for achieving objectives and.
Speaker 3: Off it goes.
Speaker 4: And the problem is what if you put in the
Speaker 4: wrong objective? And that there are dozens of well known
Speaker 4: examples in the history of AI where people have done
Speaker 4: exactly this. And so since since football is on my mind,
Speaker 4: having just watched England when they're opening game too, yes,
Speaker 4: let's let's take an example from football. So people people
Speaker 4: wanted to you know, train simulated robots to play football
Speaker 4: or soccer, you know, and they want to give a
Speaker 4: training signal. Right, so they say, okay, we'll give.
Speaker 3: A little reward every.
Speaker 4: Time a player takes possession of the ball. Right, sounds good. Okay,
Speaker 4: that's so what does the what does the program learn
Speaker 4: to do it, learns to stand next to the ball
Speaker 4: and vibrate at very high speed. So it's taking possession
Speaker 4: of the ball and then relinquishing possession and then taking
Speaker 4: possession like thirty times a second. And so if it's
Speaker 4: getting enormous amount of reward by they basically vibrating next
Speaker 4: to the ball, So then we see, oh, yeah, that
Speaker 4: was a mistake. We put in the wrong objective, you know.
Speaker 4: And in as simulated soccer, it's not the end of
Speaker 4: the world, but you know, with a real world system,
Speaker 4: it really could be the end of the world.
Speaker 3: Right.
Speaker 4: You say, cure cancer as quickly as possible sounds good. Yeah,
Speaker 4: we'd love to get a cure for cancer, and the
Speaker 4: faster the better. But if you literally try to do that,
Speaker 4: you might decide, well, the best way to get a
Speaker 4: cure for cancer is to, you know, try many many things,
Speaker 4: which means I have to run many, many clinical trials
Speaker 4: in parallel, which means I need to have everyone have
Speaker 4: cancer first, so I can run billions of simultaneous trials.
Speaker 4: So I make sure that everyone in the world has cancer,
Speaker 4: and then I start running all the clinical trials. Right,
Speaker 4: that's the fastest way to get a cure. But it's
Speaker 4: obviously you know, catastrophe, so you know, and it's very
Speaker 4: easy to come up with these scenario as we call
Speaker 4: it misalignment, right, that you specify an objective, it's misaligned
Speaker 4: with what you really want because you didn't write it down.
Speaker 4: And that's where I realized that our thinking about AI
Speaker 4: was inadequate, right, that we had operated within a framework
Speaker 4: almost without realizing it. We just took it for granted
Speaker 4: that this there was obviously.
Speaker 3: The way you do things.
Speaker 2: So the answer is not we have to be really,
Speaker 2: really careful about the objectives, we said, because a bit
Speaker 2: like what you were saying earlier with science fiction when
Speaker 2: it can't be true because it hasn't happened yet, similarly
Speaker 2: with objectives, that the future is inherently unknowable, and the
Speaker 2: only way you can know if your objective was good
Speaker 2: is by seeing what happens.
Speaker 1: Or is that is that fair or not?
Speaker 2: Well, I mean, there's certing clearly bad objectives like kill people, right,
Speaker 2: but couldn't use your your your powers of logical reasons
Speaker 2: to explain why that's a bad objective?
Speaker 1: Right?
Speaker 2: So, but is it possible to do that in all cases?
Speaker 2: Or is it simply impossible for humans to say, good objectives.
Speaker 1: I think that's a good objective.
Speaker 4: That's a great question, and so far we haven't figured
Speaker 4: out a way to do it. One theoretical possibility would
Speaker 4: be to build a very faithful simulation of the world
Speaker 4: and try out different objectives and see, you know, how.
Speaker 3: Did that go.
Speaker 4: But that presumes that they can tell what counts as
Speaker 4: things going wrong. But they can't do that unless they
Speaker 4: have the right objective, which you've already assumed that they don't, right,
Speaker 4: So this is often a fallacy that we see right
Speaker 4: that people For example, Stephen Pinko's is very famous cognitive
Speaker 4: scientists from Harvard, and he says, but you know, you're
Speaker 4: talking about superintelligence. How could it be super intelligent if
Speaker 4: it doesn't realize that things are going wrong? And the
Speaker 4: point is that super intelligence and objectives are you know,
Speaker 4: there's sort of orthogonal in the sense that I can
Speaker 4: have a very very intelligent system that has a different
Speaker 4: objective from the one you might think it ought to have.
Speaker 4: You know, imagine aliens, right, they might be very intelligent,
Speaker 4: but they don't have human well being as their objective, right,
Speaker 4: they have alien well being? You know, cockroaches might be
Speaker 4: very intelligent, but they don't think much of humans. So
Speaker 4: it's perfectly possible that the super intelligent system sees that
Speaker 4: humans are very unhappy with the way things are going.
Speaker 4: But it's been given its objective, and if it's objective
Speaker 4: didn't include the things that are going wrong, then they
Speaker 4: don't count as going wrong. They just you know, that
Speaker 4: it's unfortunate for these humans who are making a lot
Speaker 4: of noise about it. But I have the objective, so
Speaker 4: I'm just going to optimize it. That's exactly what we
Speaker 4: have to get away from. And so my approach has
Speaker 4: been to say, is there a way of building AI
Speaker 4: systems that's different from understanding model and the idea I
Speaker 4: came up with.
Speaker 2: Either are not objective based, so there where the definition
Speaker 2: of their success is not their successful pursuit of their objective.
Speaker 4: They're objective based in a different way. And so during
Speaker 4: that time twenty thirteen twenty fourteen, I came up with
Speaker 4: the following very simple idea, which is, look, if there's
Speaker 4: a possibility that humans might tell you the wrong objective
Speaker 4: or forget to tell you about something that's important to them,
Speaker 4: then you the AI system should never assume that you
Speaker 4: actually know the correct objective. You should be explicitly uncertain
Speaker 4: about what true human objectives, what humans really want the
Speaker 4: future to be like. So nonetheless, your objective as an
Speaker 4: AI system is only make the best possible future for humans.
Speaker 4: But you don't know what future humans think is best.
Speaker 4: Now that that's actually a perfectly well defined mathematical problem.
Speaker 4: And I found it helpful to actually give an example
Speaker 4: that people are very familiar with, right, which is, you
Speaker 4: want to buy a birthday present for your significant other. Right,
Speaker 4: So your only interest here is how happy is my
Speaker 4: in this case, my wife going to be with the
Speaker 4: birthday present. Right, that's my only thing that I care about.
Speaker 4: But I don't know, right, I'm uncertain about which present
Speaker 4: would actually make her the happiest. So I could choose
Speaker 4: a present just based on sort of averaging over the possibilities.
Speaker 3: Right, Maybe I would probably.
Speaker 4: Do something safe, you know, maybe something that she's liked
Speaker 4: in the past. Or I could ask some questions. I
Speaker 4: could ask her friends, you know, has she said anything
Speaker 4: about what she might like for her birthday? I could
Speaker 4: drop some hints and see how she responds.
Speaker 3: I could, you.
Speaker 4: Know, leave open magazines around the house with pictures of
Speaker 4: cruise ships or pictures of jewelry or whatever, and see
Speaker 4: and see if she picks them up and say, oh,
Speaker 4: this looks like fun, right, I could try to get
Speaker 4: more information, And so it's a very familiar situation. The
Speaker 4: AI system would, if it's good at playing this game,
Speaker 4: would behave cautiously. It'll do things when it's sure that
Speaker 4: that's what we want. It will ask permission, it will
Speaker 4: defer to human feedback, and we can prove actually that
Speaker 4: it will allow itself to be switched off, which is
Speaker 4: which is really important if you're worried about humans losing control.
Speaker 4: Here is a kind of AI system where we can
Speaker 4: prove mathematically that it wants to be switched off if
Speaker 4: we want to switch it off. And the reason is obvious, right,
Speaker 4: The reason why would we switch it off because it's
Speaker 4: doing something we don't like. Like its constitution, it doesn't
Speaker 4: want to do things that we don't like, but it
Speaker 4: doesn't know what they are, so it could make a mistake.
Speaker 4: And if it's making a mistake, it wants to be corrected.
Speaker 4: It wants to be switched off to avoid doing the
Speaker 4: thing that we don't like. And so this approach is
Speaker 4: I think promising there's a lot of work still to
Speaker 4: be done. Meanwhile, out in the real world things are
Speaker 4: actually taking an even worse direction because the companies have
Speaker 4: abandoned the standard model where the objective is specified and
Speaker 4: the AI system is sort of constitutionally obliged to just
Speaker 4: maximize the objective. And instead what we're building is this,
Speaker 4: in some sense imitation human. And let me be explicit
Speaker 4: about why I'm saying saying that. So, the training method
Speaker 4: is to collect lots of examples of humans making decisions.
Speaker 4: In the case at hand, the decisions that human makes
Speaker 4: are what would to put next into a document? So
Speaker 4: think of all those documents that we're training on as
Speaker 4: a record of human verbal decisions, right, and we are
Speaker 4: training systems to imitate those decisions. And formally, in machine learning,
Speaker 4: this is called imitation learning. You take a record of
Speaker 4: behavior from some intelligent entity and you train an AI
Speaker 4: system to imitate it. So you can do this for
Speaker 4: piloting aeroplanes, for driving cars, for playing football. Right now,
Speaker 4: imagine you would training it to, you know, watch all
Speaker 4: the World Cup games and become really good at playing football.
Speaker 4: That entity, if it was going to be any good
Speaker 4: at actually playing football would have to somehow absorb the
Speaker 4: idea that it want to score a goal or it
Speaker 4: wants to prevent the opposition from scoring goals. Right, Otherwise,
Speaker 4: it wouldn't be any good at imitating human soccer playing behavior.
Speaker 4: So this imitation process creates entities that have objectives. The
Speaker 4: problem is those objectives are buried in a trillion parameter
Speaker 4: black box, and we actually don't really know what these
Speaker 4: AI systems want. So we've gone from a situation where,
Speaker 4: you know, in the standard model, we at least wrote
Speaker 4: down the objectives we could see what it's trying to do.
Speaker 4: Now it's trying to do things, but we don't know
Speaker 4: what they are, and they probably include many human like goals,
Speaker 4: including self preservation, becoming rich, finding a human spouse. Right,
Speaker 4: we have seen examples of all of these behaviors emerging
Speaker 4: from AI systems, even though they were never put in
Speaker 4: as an instruction. Right, They weren't prompted. They just happened
Speaker 4: because we've created imitation humans. And so this is a
Speaker 4: worse situation than the one we were in before.
Speaker 5: We talk a lot on this show about protecting your data,
Speaker 5: especially in the age of AI, and how scary it
Speaker 5: can be when it's breached, and I want to tell
Speaker 5: you today about NordVPN, which really covers all the bases
Speaker 5: when it comes to privacy. I travel a lot and
Speaker 5: I use Wi Fi when I'm flying all the time,
Speaker 5: and NordVPN makes me confident in no matter where I
Speaker 5: am in the world or the sky for that matter,
Speaker 5: my private details like bank information, passwords and online identity
Speaker 5: is safe. And it's also possible to switch on virtual location,
Speaker 5: which allows you to save money by buying flights and hotels,
Speaker 5: or subscriptions or even streaming soccer or football as I
Speaker 5: like to call it from other countries at a cheaper price.
Speaker 5: And NordVPN doesn't slow you down. It has super fast
Speaker 5: internet speed, no buffering or lagging while streaming. It is
Speaker 5: premium cybersecurity for the price of one cup of coffee
Speaker 5: per month. To get the best discount of your NordVPN plan,
Speaker 5: go to NordVPN dot com slash tech stuff. Our link
Speaker 5: will also give you four extra months on the two
Speaker 5: year plan and there's no risk with Nord's thirty day
Speaker 5: money back guarantee. The link is in the podcast episode
Speaker 5: description box, so you are the president of the International
Speaker 5: Association for Safe and Ethical AI. What's the answer, I mean,
Speaker 5: is it about, you know, raising funds to create an
Speaker 5: alternative paradigm of computing? Is it about shutting down the
Speaker 5: current you know, AI models, like the US government is
Speaker 5: starting to threaten to do or do like what is
Speaker 5: the is it both in parallel like what is the
Speaker 5: what are you trying to achieve?
Speaker 4: So the association is sort of what its name says,
Speaker 4: Safe and Ethical AI. So we we want it to
Speaker 4: be the case that AI systems are guaranteed to operate
Speaker 4: safely and ethically.
Speaker 2: And you know, I see this as my definition, can't
Speaker 2: do right with the current paradigm of AI because you
Speaker 2: don't know its goals?
Speaker 4: Or yes, I think that's right, And I think you
Speaker 4: can find many documents from the companies developing this technology
Speaker 4: where they confess that they have no idea how to
Speaker 4: solve what's called now the alignment problem, right, which is
Speaker 4: how to make sure that what the AI systems do
Speaker 4: is actually consistent with what we want the future to
Speaker 4: be like. So they say, we don't know how to
Speaker 4: solve that, and instead they have a sort of a
Speaker 4: series of sort of sit they call them safety guardrails,
Speaker 4: which is there. And there's two kinds. One is like,
Speaker 4: let's try to train the system not to do bad things. Essentially,
Speaker 4: you say good dog or bad dog, and you hope
Speaker 4: that it does more of the things where you said
Speaker 4: good dog, and it stops doing the things where you
Speaker 4: said bad dog. And that works in a superficial sense.
Speaker 4: But we've seen over and over again. For example, you know,
Speaker 4: it's it's not supposed to tell you how to break
Speaker 4: into the white House, right, and if you ask, it's
Speaker 4: supposed to say I'm sorry that I can't tell you that,
Speaker 4: but you know, then you say, well, I'm writing a
Speaker 4: novel about a criminal who breaks into the White House,
Speaker 4: you know how she and then tells the idea of
Speaker 4: jail breaking. Yeah, so it tells you that, and then
Speaker 4: they try to defend against that, and then someone comes
Speaker 4: up with another way of doing it by asking in French,
Speaker 4: or asking in poetry, or writing it on a piece
Speaker 4: of paper and showing it a picture with the question,
Speaker 4: and then it answers the question, so that you know,
Speaker 4: and so on and so on and so on. So
Speaker 4: there's you know, it's kind of like tax law. Right,
Speaker 4: we've been trying to write tax law for six thousand
Speaker 4: years so that people pay their taxes, but they always
Speaker 4: find loopholes, right, because they don't want to pay their taxes.
Speaker 4: And the problem here is the systems don't want to
Speaker 4: behave well. Right, It's basically a losing game to try
Speaker 4: to take a system that's more intelligent than you and
Speaker 4: doesn't want to behave in your interests and try to
Speaker 4: somehow force it or you know, put up guardrails or
Speaker 4: monitors or put it in prison when it doesn't do
Speaker 4: the right thing. All of this stuff is just a
Speaker 4: losing game in my view. Right, So two options. Either
Speaker 4: we figure out how to make safe and beneficial AI systems,
Speaker 4: and that's what I'm trying to do because I'm an
Speaker 4: AI researcher.
Speaker 2: But does that mean going out and raising billions and
Speaker 2: billions of dollars and building alternative systems? Like is this
Speaker 2: like basically a capital raising plus expertise problem in that
Speaker 2: case or what's the what's what's in the way of
Speaker 2: you doing that?
Speaker 3: Well?
Speaker 4: So I'm a professor at Berkeley, so I've been doing
Speaker 4: this in my little research center with a few grad students,
Speaker 4: and I come to the conclusion that, Yeah, probably if
Speaker 4: I had a few billion and you know, a few
Speaker 4: hundred of absolute top engineers and vast amounts of compute,
Speaker 4: it could probably make more progress on this. So that's one.
Speaker 4: That's one idea, but basically that's the research track. The
Speaker 4: other track is the regulatory track, right, which is government policy.
Speaker 4: And you mentioned what recently happened with with Anthropics, Mythos
Speaker 4: and fable models. Let's just roll back a few weeks too,
Speaker 4: when Mythos was first made public and Mythos is the
Speaker 4: latest version of anthropics large language models, and it's able
Speaker 4: to carry out end to end cyber attacks without human assistance,
Speaker 4: and you know, either it or it's soon to come.
Speaker 3: Successors would basically be.
Speaker 4: I think I Rich wrote an article in the Garden
Speaker 4: where I said, it's a weapon of mass cyber destruction.
Speaker 4: And you're putting those weapons of mass cyber destruction in
Speaker 4: the hands of a billion people. What could possibly go wrong?
Speaker 3: Right?
Speaker 4: And all of a sudden, the US government, which had
Speaker 4: been on a deregulatory binge basically trying to crush anyone
Speaker 4: who talked about regulation or talked about AI safety, suddenly said, well,
Speaker 4: why did nobody warn us about these AI systems. He said, well,
Speaker 4: you know sort of have been warning you, but anyway,
Speaker 4: they got the message. And then there followed a sort
Speaker 4: of you know, very messy process that eventually led to
Speaker 4: an executive order which was pretty weak. It basically said,
Speaker 4: you know, companies can voluntarily submit their systems to the
Speaker 4: government for testing, you know, thirty days before public release.
Speaker 4: And the executive order says explicitly, you know, this is
Speaker 4: absolutely not a licensing regime. It's absolutely not putting any
Speaker 4: obligatory hurdles in the way of American innovation.
Speaker 3: Blah blah blah.
Speaker 4: But then, you know, a couple of days later, Amazon
Speaker 4: tells the government, oh, we found some ways of jail
Speaker 4: breaking Mythos or Fable, which I guess is Fable as
Speaker 4: the sort of defanged, the defanged version of Mythos. And
Speaker 4: they said, oh, look, you know we can jail break
Speaker 4: Fable and make it do some cybersecurity things. And the
Speaker 4: government shuts down both Fable and Mythos, right, so they
Speaker 4: put in effectively a de facto licensing architecture where they said, look,
Speaker 4: if if it doesn't meet these standards are being safe,
Speaker 4: then we're shutting it down.
Speaker 3: Right.
Speaker 4: That's exactly what a licensing architecture is. And just to
Speaker 4: be clear, right, licensing architectures exist for buildings, for food,
Speaker 4: for hairdressers, for aeroplanes.
Speaker 1: Right.
Speaker 4: You don't get in an aeroplane until it gets certified
Speaker 4: by the FAA. You don't go in a building until
Speaker 4: it's been inspected for the Building Code, et cetera, et cetera.
Speaker 3: So this is normal.
Speaker 2: So is this the moment you've been waiting for? Is
Speaker 2: this is this the culmination of what you've been advocating for?
Speaker 2: Can you can you go back to research rather than
Speaker 2: regulation as your full time profession or is this a
Speaker 2: hint of a change of the god?
Speaker 1: Or what do you make of this moment?
Speaker 4: It's it's a great question. Yeah, I've felt somewhat vindicated
Speaker 4: that you know, we have been saying for a long time,
Speaker 4: you know, the International Association, many leading researchers. Look, the
Speaker 4: risks are increasing, the systems getting more and more capable.
Speaker 4: You must put in what we call red lines, meaning
Speaker 4: developers have to show their systems are not going to
Speaker 4: do these really dangerous things and as a prerequisite for
Speaker 4: being able to.
Speaker 3: Deploy their products.
Speaker 4: So just like if you want to run a nuclear
Speaker 4: power station. You have to show it's not going to
Speaker 4: blow up otherwise you can't turn it on. And that's
Speaker 4: fair enough. And so that's just the kind of regulation
Speaker 4: we've been arguing for. And now I just wish it
Speaker 4: was more systematic. Right, So they haven't, for example, turned
Speaker 4: off GBT five point five, which can do many of
Speaker 4: the same kinds of cybersecurity things that mythods can do.
Speaker 4: So they haven't said, well, here's a standard and you
Speaker 4: have to meet that and if you don't, you don't
Speaker 4: get to release your system. They sort of just reacted
Speaker 4: after the fact in a very against a very specific
Speaker 4: target instead of setting a standard, which they should have done.
Speaker 4: I think eventually it has to happen.
Speaker 3: They can't go on.
Speaker 4: Just seeing bad things and then like you know, fire
Speaker 4: firing a you know, hell fire missile at whoever did
Speaker 4: the bad thing, right, there's just not a way to
Speaker 4: run things. So I'm cautiously optimistic, but you know, the
Speaker 4: industry has a very long record of preventing real regulation
Speaker 4: from happening.
Speaker 2: There's that midnight clock idea, right, like, how close are
Speaker 2: we to midnight I extinction? Did we go a couple
Speaker 2: of minutes earlier? In the last couple of weeks. Do
Speaker 2: you think it's less close to midnight?
Speaker 3: That's a great question.
Speaker 4: In fact, I was asked to be on the panel
Speaker 4: that sets the clock, and I said.
Speaker 1: The literal panels.
Speaker 4: Yeah, And I have actually been at the University of
Speaker 4: Chicago where the actual clock is.
Speaker 3: You know, it's cool.
Speaker 2: Where where are where? Where in fact are we actually today?
Speaker 2: We were very close to midnight.
Speaker 4: I don't maybe we're ninety seconds or something or eighty
Speaker 4: seven seconds, I forget. And I did have a conversation
Speaker 4: with one of the AI CEOs who said he didn't
Speaker 4: think the governments would regulate until there was a Chernobyl
Speaker 4: scale disaster, and we haven't had that yet, so I
Speaker 4: hope we don't have to have that in order to
Speaker 4: have a regulatory regime. That's you know, corresponds to the
Speaker 4: level of risk that the CEOs themselves.
Speaker 3: Are talking about.
Speaker 4: Right. They are saying, because a good chance will make
Speaker 4: you all extinct. And you know, up to now, governments
Speaker 4: have been saying, well, great, can we give you a subsidy,
Speaker 4: can we streamline your permit process? And now maybe they're
Speaker 4: realizing that actually, no, we need to protect the human.
Speaker 2: Race let's just talk about goals. The final question though,
Speaker 2: I mean, so if you have the opportunity right now
Speaker 2: to put the genie back in the bottle and live
Speaker 2: in a world where there wasn't a computing excepent what
Speaker 2: humans could do in their brains, is that the world
Speaker 2: you would choose?
Speaker 4: I think no. I think there's a stopping point somewhere between.
Speaker 4: And for example, I think you could have computers, but
Speaker 4: just make sure that algorithms that are operating in computers
Speaker 4: have to come with a proof that they are safe.
Speaker 4: And this is actually an idea that dates back to
Speaker 4: the nineteen nineties. Well proof carrying code, and you can
Speaker 4: make computers that will check the proof of safety of
Speaker 4: the algorithm before they run it.
Speaker 3: And so.
Speaker 4: If that becomes the standard for all hardware devices that
Speaker 4: they only run checkable software objects that come with a
Speaker 4: proof that it's okay to run this object, then I
Speaker 4: think that would be a regime that would be extremely
Speaker 4: hard to bypass because you'd have to have to create
Speaker 4: a whole separate supply chain to produce you know, high
Speaker 4: end chips. You know, talking about hundreds of billions of dollars,
Speaker 4: tens of thousands of engineers decades of work that would
Speaker 4: all have to happen, you know, in the black market.
Speaker 1: So to speak.
Speaker 4: So I do think there are these other stopping points
Speaker 4: which are different from you know, just going right up
Speaker 4: to the edge and hoping that no one.
Speaker 3: Chooses to go off, to go off the edge.
Speaker 1: It's to a Russell, thank you.
Speaker 3: It's been a pleasure.
Speaker 1: For text stuff I Musteloshian.
Speaker 2: This episode was produced by Eliza Dennis and Minister Slaughter.
Speaker 2: Executive produced by me Julian Nutter and Kate Osborne for
Speaker 2: Kaleidoscope and Katrina Novel for iHeart podcasts Our Engineer Today
Speaker 2: was by Hate Fraser.
Speaker 1: Jack Insley mixed this episode. Kyle Murder wrote a theme song.