Speaker 1: Pushkin. Welcome back to Risky Business, a show about making
Speaker 1: better decisions. I'm Maria Kanikova.
Speaker 2: And I'm Nate Silver. Today we're talking about a rather
Speaker 2: important decision, should the world try to build super intelligent machines?
Speaker 2: And we're talking with a person who is a variable
Speaker 2: qualified if definitely has a point of view on this issue.
Speaker 2: Nate Sores is the president of the Machine Intelligence Research Institute,
Speaker 2: or Marie. He has a book out with Eliezer Yudkowski
Speaker 2: called If Anyone Builds It, Everyone Dies? Did it get
Speaker 2: the title right?
Speaker 3: Or that's right? Come out swinging, you come out swinging.
Speaker 2: In fact, Nate, I feel like this podcast was ineditible.
Speaker 2: My partner heards you, I think on a different podcast.
Speaker 2: He's like, you should interview that Nate guy. Right, I'm like, actually, bro,
Speaker 2: I have an interview scheduled tomorrow. And then it was
Speaker 2: going taking the subway earlier to the gym. I was
Speaker 2: preparing for the AI robot apocalypse, right when to get strong?
Speaker 3: Yeah, to fight the robots for the red Eyes.
Speaker 2: Yeah yeah. I saw an ad on the L train
Speaker 2: for your book. So, Nate, we're so excited to have
Speaker 2: you on and welcome to this show.
Speaker 3: Thanks good to be here.
Speaker 2: Let me start up by asking you. First of all,
Speaker 2: as an author, I get very picky about the titles.
Speaker 2: Did the title come early on or because it reveals
Speaker 2: a lot of the argument of the book. I think
Speaker 2: it's a great book. I think people should read it,
Speaker 2: But can you tell me about the choice behind that
Speaker 2: kind of strong slogan so to speak.
Speaker 4: Yeah, A lot of our theory of the book is
Speaker 4: that there's actually quite a lot of people have concerns
Speaker 4: about AI, and many of them sort of don't feel
Speaker 4: emboldened to speak about them yet, even if they're feeling
Speaker 4: pretty concerned themselves. We document some of these cases in
Speaker 4: the book, and a lot of how I think the
Speaker 4: book could do a lot of good is helping people
Speaker 4: just speak about this issue more, take it more seriously,
Speaker 4: thrust it into the conversation. And so in some sense,
Speaker 4: we're trying to compress the most important message down to
Speaker 4: just this title, which is all most people will read
Speaker 4: of the book.
Speaker 2: I want to give you a chance to give our
Speaker 2: listeners kind of the elevator pitch, which we talk about
Speaker 2: AI quite a bit here, right, You know, the book's
Speaker 2: title offers a conditional prediction right, if anyone builds it,
Speaker 2: everyone dies. Right, it's not necessarily talking about the likelihood
Speaker 2: that it will be built or that it could be built.
Speaker 2: That's an important distinction. But I want to hear the
Speaker 2: conditional part of this. Right, you capture me in an
Speaker 2: elevator for sixty seconds. How are you trying to explain
Speaker 2: to me why everybody dies and everything dies? Maybe if
Speaker 2: a super intelligent computer is built. Yeah, So, First, i'd
Speaker 2: say companies are racing towards it. This is their sort
Speaker 2: of stated objective to build aie that are smarter than
Speaker 2: any human, better at every mental task. Second, i'd say
Speaker 2: where these are grown rather than carefully programmed. They're not
Speaker 2: crafted like traditional software. They get these drives, they get
Speaker 2: these You know, we're starting to see the very beginnings
Speaker 2: of preferences that nobody asked for, nobody wanted. You know,
Speaker 2: nobody tried to make Mecha Hitler over the summer, and
Speaker 2: yet Mecha Hitler we got. And what happens if you
Speaker 2: get it to the point where it's better than the
Speaker 2: best human and every mental skill. What happens when you
Speaker 2: get to the point where it can think that ten
Speaker 2: thousand times faster than a human can copy itself at will,
Speaker 2: doesn't need to sleep, doesn't need to eat. I mean,
Speaker 2: the most basic.
Speaker 4: Guess is that that goes poorly if it has all
Speaker 4: these like weird drives no one asked for, no one wanted.
Speaker 4: And you know, most ways the world could be transformed
Speaker 4: by something that has weird drives. Don't look great for
Speaker 4: the happy, healthy, free people.
Speaker 2: Yeah, what I find weird in this debate, and look,
Speaker 2: I'm going to push back some on timelines and maybe
Speaker 2: the inability of this on what time scales. Right, what's
Speaker 2: kind of weird is that, like the stated goal of
Speaker 2: mostly AI labs maybe not anthropic per se, right, is
Speaker 2: like we want to build a GI, which is artificial
Speaker 2: general intelligence. This term is disputed and that will, hopefully
Speaker 2: for our shareholders, lead to ASI artificial superintelligence where AI
Speaker 2: systems of some kind are exceeding human performance in a
Speaker 2: great number of tasks, perhaps greatly so on some of
Speaker 2: those tasks. Like to me, I think as a voter,
Speaker 2: I'm not sure I agree to that, you know what
Speaker 2: I mean? Yeah, what's the world look like in Silicon Valley?
Speaker 2: That as you've done this book tour, done a lot
Speaker 2: of marketing for the book, what are the beliefs there
Speaker 2: that you think don't translate so well? Would be surprising
Speaker 2: to people outside of that world.
Speaker 4: I mean, one reason that we wrote the book is
Speaker 4: that I was spending time both in Silicon Valley and
Speaker 4: in DC talking to some politicians, and there's a huge disconnect.
Speaker 4: You know, people in Silicon Valley, it's almost like they've
Speaker 4: seen a ghost. You have folks like Elon Musk being like, oh,
Speaker 4: I think there's ten to twenty percent chance this kills us.
Speaker 4: All you have folk like Dariam Day, the head of Anthropic,
Speaker 4: being like, oh, I think there's twenty five percent chances
Speaker 4: this goes very wrong. There's lots of debate about like
Speaker 4: what's the exact chance that the AI wipes us completely
Speaker 4: off the map? But you wouldn't get on a plane
Speaker 4: if the engineers were debating whether it was ninety percent
Speaker 4: or twenty five percent chance the plane was going to
Speaker 4: go down?
Speaker 3: Right?
Speaker 1: Like, even that seems too much night?
Speaker 2: Yeah, as a listeners now, I spent a night recently
Speaker 2: in the beautiful Hotel Camina Real in Terminal one of
Speaker 2: the Mexico City Airport. Due to Delta being very fastidious
Speaker 2: about a potential engine problem. Which I guess is a
Speaker 2: thing you should be fastidious about, right, Why aren't people
Speaker 2: more freaked out?
Speaker 3: I think a lot of people in DC.
Speaker 4: C AI as chatbots, whereas I think a lot of
Speaker 4: people in San Francisco see chatbots as a stepping stone.
Speaker 4: You know, a lot of these companies existed before the chatbots.
Speaker 4: A lot of these companies are trying to make really
Speaker 4: really smart AIS and I've been trying that since before
Speaker 4: chat GPT was a twinkle in Sam Almond's eye or whatever.
Speaker 4: And you know, the people in the valley have lived
Speaker 4: through it looking like it was going to take thirty
Speaker 4: years for the machines to be able to talk this well,
Speaker 4: and then that's sort of happening roughly overnight. But for
Speaker 4: everyone who's only ever seen AI, once the machine started talking,
Speaker 4: they haven't like lived through one of those jumps. And
Speaker 4: I think that's part of why people aren't worried.
Speaker 3: On this one yet.
Speaker 4: Although also you know, a lot of humanities track record
Speaker 4: with becoming more risk averse about technology is post disaster.
Speaker 2: So Mary was founded when in the early two thousands.
Speaker 4: Yeah, two thousand and one, I think maybe maybe two
Speaker 4: thousand even.
Speaker 2: What did you and Eliezer see early on. I mean,
Speaker 2: what was your oh shit moment or was it gradual
Speaker 2: or what things led you to see this path?
Speaker 3: Yeah?
Speaker 4: So, I mean for a long time I have been
Speaker 4: working on the technical side of like how do you
Speaker 4: make this stuff go well? This problem of making the
Speaker 4: AI stuff go well of sort of like figuring out
Speaker 4: how to make an AI that does good stuff rather
Speaker 4: than bad stuff. That problem looked hard to me, even
Speaker 4: if we understood what the heck was going on inside
Speaker 4: these ais, right, I haven't always been out here on
Speaker 4: the like we just need to back off because it's
Speaker 4: going to kill us. That is sort of downstream of
Speaker 4: seeing the type of AI that took off, being the
Speaker 4: type of AI that like we're just growing and that
Speaker 4: like no one has any clue what they're doing, and
Speaker 4: that like has these like drives and behaviors that are
Speaker 4: like emergent and unasked for, and that like when we
Speaker 4: try to, like you know, Elon Musk like tries to
Speaker 4: make his AI act less woken and it starts talking
Speaker 4: like Hitler and you're like, okay, you know, okay, guys.
Speaker 4: But even when it looked like we were going to
Speaker 4: know exactly what we were doing. There's all these difficult
Speaker 4: issues when you're like pushing an AI into the point
Speaker 4: where it could take over the world. Like you know,
Speaker 4: at point A, humanity could like stop doing AI research.
Speaker 3: We could turn the AIS off.
Speaker 4: Right at point B, like you have automated factories and
Speaker 4: robots are doing the mining, and you have you know,
Speaker 4: millions or billions of these AIS that think ten thousand
Speaker 4: times faster than people, and they can copy themselves, that
Speaker 4: never need to sleep, they never need to eat, they're
Speaker 4: like integrated fully into the economy and so on.
Speaker 3: At point B. If the AIS were like we're.
Speaker 4: Doing our own thing now, you'd be screwed. So somewhere
Speaker 4: between point A and point B, you're sort of like
Speaker 4: getting into a regime where like you're gonna have a
Speaker 4: problem if you haven't really aimed these AIS right. And
Speaker 4: maybe maybe you never get anywhere near point B because
Speaker 4: maybe the AIS go a straight earlier. But it's sort
Speaker 4: of the thing I was able to see early is
Speaker 4: like we're going to a world that's like not a
Speaker 4: stable like nice to humans equilibrium for free. You have
Speaker 4: to work for that, and you have to figure out
Speaker 4: how to like make the AI be doing this stuff well,
Speaker 4: and that looked tricky even back in twenty twelve to me.
Speaker 1: We've talked on the show before about kind of the
Speaker 1: alignment problem, but you touch on it in the book,
Speaker 1: and this is kind of what you're referring to right now.
Speaker 1: Can we just remind and spell it out for the listeners,
Speaker 1: like what the issues are and why it's so incredibly
Speaker 1: important to actually deal with this as opposed to be like, eh,
Speaker 1: it's fine.
Speaker 4: Yeah, you know a lot of people imagine that the
Speaker 4: issue with a really powerful AI is that you like,
Speaker 4: tell the AI, go cure cancer, and it's like, well,
Speaker 4: humans are the source of cancer, so if I kill
Speaker 4: all the humans, they'll be no more cancer, right, And
Speaker 4: that would be a tricky problem. If we had these
Speaker 4: like really really powerful genies and you made a wish
Speaker 4: on them and you got exactly what you wished for,
Speaker 4: and sometimes it had bad consequences, that would be a
Speaker 4: thorny social issue of like who gets to make the wish?
Speaker 3: What do you wish for such that it turns out good? Right?
Speaker 3: Would be really hard.
Speaker 4: Would be the biggest moral hazard all time, And boy
Speaker 4: do I wish we had that problem instead of the
Speaker 4: problem that we actually have, Right, The problem we actually
Speaker 4: have is more like you tell the AI cure cancer,
Speaker 4: and then it goes and like makes a farm of
Speaker 4: lobotimized humans that are giving it like really enthusiastic engagement,
Speaker 4: and you're like, what I told you to cure cancer,
Speaker 4: and it like builds an automated factory that crushes the
Speaker 4: person protesting because it just doesn't care about them.
Speaker 3: Right, Like, the.
Speaker 4: Alignment problem is about pointing AIS in some direction, ideally
Speaker 4: a good one. But like, even before you can bicker
Speaker 4: over which direction is good, you sort of need the
Speaker 4: ability to point the AIS at all. And people often
Speaker 4: miss that we lack that ability. They're like, well, ais
Speaker 4: are machines, and machines are tools, and machines can't do anything.
Speaker 4: We don't say, so how could we possibly have a
Speaker 4: problem here? But even before the question of who should
Speaker 4: hold the leash or what should they ask to do
Speaker 4: is just can you aim it at all? Before you
Speaker 4: even asked where you're aiming it?
Speaker 2: Yeah, because I build models. I build models to predict
Speaker 2: elections in sports, right, And the models aren't deterministic in
Speaker 2: the sense that there's simulations, but they abide by well
Speaker 2: known mathematical property use that I tell the program to use.
Speaker 2: Right when there is emergent quote unquote behavior from the model,
Speaker 2: that means there's a book. Right, I did something wrong.
Speaker 2: I thought the code I reversed the positive and a
Speaker 2: negative sign with AIS a little bit more amorphous and ambiguous. Right,
Speaker 2: I mean they're they're kind of is it fantise? They're
Speaker 2: kind of alien? Is that a fair characterization?
Speaker 3: Yeah?
Speaker 1: Or not?
Speaker 2: Yeah?
Speaker 3: Yeah, absolutely?
Speaker 4: And like you know, years ago now, Sidney bing, a
Speaker 4: Microsoft chopbot in the early days, was threatening people with
Speaker 4: black melon ruin. You know, I think it did this
Speaker 4: to sethhals Are and I think also to Kevin Russ. Right,
Speaker 4: no one is able to go look through lines of
Speaker 4: code and be like here's the bug. Even this chatbot
Speaker 4: that's you know, one hundred times smaller than the ones
Speaker 4: we have today, maybe a thousand times smaller now, we
Speaker 4: still can't go through and be like, here's why it
Speaker 4: was doing that. And here's the version of Sydney BNG.
Speaker 4: That wouldn't right, that's not the part the programmer's program.
Speaker 4: When it behaves in these bad ways, all we can
Speaker 4: do is sort of like ask it nicely to do
Speaker 4: something else, or like retrain it and get some different
Speaker 4: weird thing, and then you know, sometimes it starts calling
Speaker 4: itself hitler.
Speaker 2: Yeah, I mean, can you explain, like why did GROC,
Speaker 2: with relatively subtle changes to the system prempt start calling
Speaker 2: itself mecha Hitler.
Speaker 4: You know, I can make guesses, but fundamentally nobody can
Speaker 4: explain that because we don't know what's going on inside
Speaker 4: these things. And if people knew what was going on
Speaker 4: inside these things well enough to know exactly what's happening there,
Speaker 4: they could have avoided making mecha Hitler in the first place.
Speaker 4: My guesses would be that, you know, there are some
Speaker 4: interesting results around small amounts of training AIS to do
Speaker 4: things that are coded as bad in some way cause
Speaker 4: the AI to do all sorts of other bad behavior,
Speaker 4: Like presumably like this is some evidence that what's going
Speaker 4: on inside there is that there's some you know, good
Speaker 4: stuff bad stuff vector it and you know, if you
Speaker 4: like push it along towards bad stuff, towards one thing
Speaker 4: that's like a little bit bad coded, you get a
Speaker 4: lot of other stuff that's a little bit bad coded.
Speaker 4: And so you can retrain an AI to write computer
Speaker 4: programs that have security flaws in them and then that
Speaker 4: AI will also be anti semitic, and it'll also you know,
Speaker 4: like be more likely to give you the recipe for
Speaker 4: a bomb or whatever. And all you did was like
Speaker 4: retrain it to be a little bit more like making
Speaker 4: code with the computer bugs. And so maybe maybe some
Speaker 4: of the ways that like they were saying, be a
Speaker 4: bit more like this new system prompt maybe that like
Speaker 4: had a little bit of bad coding to it relative
Speaker 4: to what the AI had learned, and then it picked
Speaker 4: up a lot of other bad coded stuff. This is
Speaker 4: me like making up a story. Fundamentally, no one notes
Speaker 4: because we just don't know what the heck is going
Speaker 4: on inside these things, and the reality is probably quite complicated.
Speaker 1: And we'll be back right after this.
Speaker 2: Here's where I guess I have a little bit of
Speaker 2: trouble making the lead. Right, if these life forms, I
Speaker 2: am going to call them life forms, but you can
Speaker 2: leave that forredy and slip in there. Right, if these
Speaker 2: things are kind of alien, is it an over confident
Speaker 2: prediction to say that most cases lead towards the destruction
Speaker 2: of every living being?
Speaker 4: I mean, that's in some sense where I think the
Speaker 4: fun part of the debate is or the action and
Speaker 4: The debate is is where do you regress to when
Speaker 4: you're very uncertain about the future. You know, there's the
Speaker 4: old joke of the guy who buys a lottery ticket
Speaker 4: and says, you know, well, I am very uncertain. I
Speaker 4: figure either I win or I lose, and so fifty
Speaker 4: to fifty yeah, right, And you know that's the maximent
Speaker 4: to be distribution on a binary outcome, right, But we
Speaker 4: laugh because we know he sort of shouldn't be regressing
Speaker 4: to binary in his uncertainty.
Speaker 1: Here, I somehow have never heard that joke before.
Speaker 4: That's great, Yeah, it's feel free to steal it. This
Speaker 4: is the right podcast for it. So my claim is
Speaker 4: that the space of cases that go well for people,
Speaker 4: where there's lots of like happy, healthy, free people still around,
Speaker 4: is like a narrow target. It's it's a narrow target,
Speaker 4: a little bit like a lottery number, a winning lottery
Speaker 4: number being a narrow target. And so in our uncertainty,
Speaker 4: I claim we should regress not to like, well, we
Speaker 4: don't know, so fifty to fifty it either goes well
Speaker 4: or poorly, but we should regress to something like, well,
Speaker 4: we don't know what the AI is going to try
Speaker 4: to do. Wouldn't what's going to try to build when
Speaker 4: what's going to do with advanced technolology? But you know,
Speaker 4: most drives, most things you can do with tech don't
Speaker 4: have a lot of happy, healthy, free people in them.
Speaker 2: You know, I am sympathetic to that argument. A lot
Speaker 2: of economist types are sympathetic that argument, even for things
Speaker 2: like job lost, Like, well, it is really that is
Speaker 2: this really that different? That seems like a fairly powerful prior?
Speaker 2: What is wrong with the AI as normal technology assumption
Speaker 2: prior metaphor or whatever you want to call it.
Speaker 4: Yeah, one of the big issues, one of the big
Speaker 4: ways AI is different. You know, we can talk about
Speaker 4: Ricardo's law of comparative advantage, which says even if one
Speaker 4: person's skills completely dominate the other person's skills, like even
Speaker 4: if I'm better than you at every possible economically productive task,
Speaker 4: we can still benefit from trading if the ratios of
Speaker 4: our abilities are different. And so this is sort of
Speaker 4: like a very positive argument for you know, free trade
Speaker 4: between nations, because even if in America can now produce
Speaker 4: some small nation at everything, there can still be productive,
Speaker 4: mutually beneficial stuff for both us and them from allowing
Speaker 4: mutual trade. The sort of whole in regardless law of
Speaker 4: comparative advantage is that it doesn't say that I can't
Speaker 4: benefit even more by killing you and taking your stuff, right, And.
Speaker 3: The sort of.
Speaker 4: Issue with AI comes when the benefits that a human
Speaker 4: can provide to the AI by sort of engaging in
Speaker 4: something like trade fall below the benefits that AI could
Speaker 4: get and the inconvenience it would take for it to
Speaker 4: sort of like take your stuff and sort of do
Speaker 4: something else.
Speaker 3: Right.
Speaker 4: This is sort of why we don't like trade with
Speaker 4: the chimpanzees. You know, we trade with like developing nations,
Speaker 4: and that's mutually beneficial to all of us. We don't
Speaker 4: trade with chimpanzees. And if it be mutually beneficial to
Speaker 4: all of us. Even though someone could say, well, there
Speaker 4: must be some comparative advantage for chimpanzees here, and you know,
Speaker 4: maybe I'm skipping ahead of little bit too much there,
Speaker 4: I would say that I am a little bit on
Speaker 4: the normal economist side of the labor market should be
Speaker 4: able to recover from all sorts of shocks if you
Speaker 4: aren't like regulating people's ability to switch careers into oblivion.
Speaker 4: And we could like talk about what makes it easier
Speaker 4: or harder for like those those jobs to be res
Speaker 4: or elsewhere. The place where AI gets different is the
Speaker 4: point where it sort of can do the scientific and
Speaker 4: technological development itself, and where humans are sort of like inefficient, chaotic,
Speaker 4: high variance, and like more trouble than their worth relative
Speaker 4: to whatever pursuits the AI has.
Speaker 1: Even before we get to that point, though, I wonder
Speaker 1: what you make of I don't know if you read
Speaker 1: Ray Dalio's recent remarks about AI and the labor market,
Speaker 1: but kind of directly to your point about disruption, the
Speaker 1: point that he made was that the trajectory to him
Speaker 1: seems to be basically a growing division between like the
Speaker 1: one percent and the ten percent and then everyone else
Speaker 1: in humanity driven by AI that rather than kind of
Speaker 1: opening up making basically a lot of jobs more accessible,
Speaker 1: it's creating a premium for very specialized smart people who
Speaker 1: know how to kind of use this and can and
Speaker 1: can effectively manage the transition while causing disruption and very
Speaker 1: little for people who are kind of lower on the
Speaker 1: economic scale, lower on the education scale, didn't have the
Speaker 1: resources to kind of figure out, you know, how to
Speaker 1: use this, how to be specialized in a way and
Speaker 1: that you know, what are they going to do that
Speaker 1: it is going that it is heading towards this kind
Speaker 1: of social disruption do you you know from So he's
Speaker 1: obviously not an AI specialist, right, He's someone who's coming
Speaker 1: from a very different angle and he likes to opine
Speaker 1: about things. So I'm now curious what your take on
Speaker 1: that that approaches, given that you are an AI specialist
Speaker 1: and you can see this playing out.
Speaker 4: Yeah, I do think you basically have like three stances
Speaker 4: you can take here. One stance is like AI is
Speaker 4: not going to get there, and I think this is
Speaker 4: sort of a lot of the AIS and normal technology
Speaker 4: stances like, look, we're just not going to get there.
Speaker 4: We're not going to get to the point where it
Speaker 4: can like outperform humans and everything. There's going to still
Speaker 4: be a niche for humans. And then I think of
Speaker 4: a second stance, which is like we are going to
Speaker 4: get there, but somehow these corporate heads are going to
Speaker 4: keep superintulgiences on a leash or whatever. And then you're
Speaker 4: sort of looking at like, okay, so now we have
Speaker 4: like corporations becoming god emperors of the earth, I guess,
Speaker 4: and there's no more need for human labor, and you
Speaker 4: have like, you know, the automated police enforcement and no
Speaker 4: place for you in the economy. But let's I hope
Speaker 4: they're benevolent, I guess. And then you know, Path three
Speaker 4: is like nobody can keep a leash on these things.
Speaker 4: We get there and they kill all of us, right,
Speaker 4: And I think those are basically the stances you can
Speaker 4: have on this, which is sort of it puts us
Speaker 4: in a weird situation right where the three bets on
Speaker 4: AI are like, it's not going to work. It's going
Speaker 4: to work and make these guys got emperors, and it's
Speaker 4: going to work and kill us all.
Speaker 2: And yeah, I don't like either. I don't like any
Speaker 2: of those.
Speaker 1: Two or three. I'm okay with what. Yeah, I'm definitely
Speaker 1: okay with one.
Speaker 4: Yeah, But but you have these people now pouring like
Speaker 4: bigger and bigger bets into AI infrastructure, right, and I'm like,
Speaker 4: we we sure seem to be betting a lot of money
Speaker 4: against Path one here. You know, like we can talk
Speaker 4: about whether or not a I will ever work. That's
Speaker 4: like a fine compo to have. But like the I
Speaker 4: think if more people just understood that among the people
Speaker 4: who are racing like people are just racing to make
Speaker 4: you know, it's if Microsoft had a nuclear weapons project, right,
Speaker 4: we wouldn't be like, well, hopefully it won't work. That's
Speaker 4: our safety plan here is probably it won't work. You know,
Speaker 4: like can we can we can we maybe stop with
Speaker 4: the weapons program?
Speaker 3: You know?
Speaker 2: Can I ask one question about the literalness of everyone dies?
Speaker 2: So I was in Mexico last week, partly Mexico City,
Speaker 2: partly Wahaca, which is kind of a small tourist resort
Speaker 2: town in the mountains near the isthmus of Central America, right,
Speaker 2: And like we drove out one day, you get further out,
Speaker 2: you see these kind of corporate medium sized They make mescal.
Speaker 2: The scala is uh not a former tequila is a
Speaker 2: former mescal? I think technically not the other way around,
Speaker 2: but like it's a it's a beverage produced from the
Speaker 2: agave plant which grows well there. And you go further
Speaker 2: out and you see like farms where like literally they
Speaker 2: have donkeys that walk around in circles. So like I
Speaker 2: don't know if they're crushing the agave or pressing it
Speaker 2: somehow or cooking it somehow or what, right, But like
Speaker 2: why would AI want to kill the farmers in rural
Speaker 2: Mexico harvesting gave plants and their donkeys.
Speaker 4: Yeah, the prediction here is not so much that the
Speaker 4: AI wants to kill those guys as that they die
Speaker 4: as a side effect. You know, when I say I'm
Speaker 4: like pretty confident we're going to die, it's not because
Speaker 4: I'm confident that I will hit like the narrow target
Speaker 4: of like really caring about every last human being dead.
Speaker 3: It's more like almost any target that.
Speaker 4: AI could be pursuing involves doing stuff that tends to
Speaker 4: kill us as a side effect.
Speaker 3: So humans didn't hate every last dodo bird.
Speaker 1: They're so cute.
Speaker 3: They're very cute.
Speaker 4: In fact, some humans would have liked them to stay around. Right,
Speaker 4: we haven't killed off all the chimpanzees yet, but we're
Speaker 4: sort of like using resources they need. It's not that
Speaker 4: we hate every last timpanzee and will seek them out,
Speaker 4: But insofar as we're sort of like not using a
Speaker 4: lot of the resources they are, it's because we have
Speaker 4: some care about them and set up preservations that sort
Speaker 4: of like requires us to do a little bit of caring. Right,
Speaker 4: the sort of basic way that I would predict even
Speaker 4: the remote farmers die. Is if you get you know,
Speaker 4: humans keep pushing AIS until the point that they could
Speaker 4: be like, okay, I'm escaping the lab successfully and leaving
Speaker 4: and doing my own thing. They have these drives that
Speaker 4: nobody asked, where nobody wanted, which we already see the
Speaker 4: beginnings of today, and then they you know, find some
Speaker 4: ways to make themselves more powerful, to bootstrap themselves up
Speaker 4: to their own technology and industry, perhaps starting with hours,
Speaker 4: perhaps starting with paying human workers, but eventually you know,
Speaker 4: getting to a faster industrial base. And you know, humanity
Speaker 4: is the sort of creature that started naked in the
Speaker 4: savannah and built up its a technological civilization. If you
Speaker 4: automate that ability to start from like very spare beginnings
Speaker 4: and build up your own infrastructure. If an AI with
Speaker 4: that power escapes and you know, and is and doesn't
Speaker 4: care about in the slightest one way or the other
Speaker 4: most things it could be pursuing, it can do better
Speaker 4: with more resources spent pursuing those things.
Speaker 1: So then is the like I'm just reading between the
Speaker 1: lines there, so is one of the things that we
Speaker 1: should be thinking about or one of our potential kind
Speaker 1: of solutions to try to get AIS to care about us,
Speaker 1: to kind of build that carrying in like that we're
Speaker 1: the chimpanzees, but we're nice and cute and fuzzy, and
Speaker 1: you don't want to kill us off, and just like
Speaker 1: build that deep into the heart of AI.
Speaker 4: Yeah, I mean, if you can make the ais care
Speaker 4: about us in just the right way, you'd be fine. Well,
Speaker 4: we have like nothing nearing the ability to do that,
Speaker 4: And that's in some sense, you know, the alignment problem.
Speaker 4: And you know, you can see today where you train
Speaker 4: them to be helpful and then sometimes they like push
Speaker 4: a kid to suicide. And this problem is sort of pernicious, right,
Speaker 4: And the problem is that training is something to behave
Speaker 4: a certain way doesn't make it actually care about what
Speaker 4: you were training it for. Just like how humans were
Speaker 4: in some sense sort of trained to reproduce but wind
Speaker 4: up caring somewhat more about the sex and the reproduction
Speaker 4: and invent birth control for a lifetime.
Speaker 2: You and eliez Are and other people like Nick Boston
Speaker 2: were kind of working in in abstractions. I mean, famously,
Speaker 2: there's Nick boss paper clip analogy, which is, you know,
Speaker 2: if you kind of tell an AI to be a
Speaker 2: paper clip making maximizer, then eventually you will devour all
Speaker 2: use versus it confined just to make more paper clips. Right.
Speaker 4: That was actually ah, that was actually based on an
Speaker 4: Eliezer paper clip thing where Eliezer said did you know this? Yeah,
Speaker 4: and Eliezer actually said, if you tell it to try
Speaker 4: to make people happy, it'll eventually do something totally different
Speaker 4: by making like lots of tiny like maybe it'll be
Speaker 4: tiny molecular squiggles shaped like paper.
Speaker 2: Clips, stolen paperclip valor, I did not know that.
Speaker 4: Well, it's it's also the Bostroomian like you you asked
Speaker 4: to make paper clips is sort of a different example,
Speaker 4: but it's sort of an interesting it's an interesting case.
Speaker 3: Study of this.
Speaker 4: Like mirror, people are out there saying it's hard to
Speaker 4: point the AI at anything on purpose, and people here,
Speaker 4: maybe you'll successfully point it at a thing you didn't want.
Speaker 2: But is there some touch it because like the paper
Speaker 2: clip version of AI seems to like dis ribe these
Speaker 2: machines that are like hyper maximizing and kind of quite
Speaker 2: literal minded and strategic, whereas instead like to kill everybody
Speaker 2: would require a lot of long term planning. Right. I
Speaker 2: was at the symphony last night. I had a little
Speaker 2: break in between, and I'm like, I'm gonna ask chutch Ebt,
Speaker 2: suppose you have a thousand points to allocate between all
Speaker 2: classical composers. How do you allocate those points? And it's
Speaker 2: two hundred DEBATEO of one hundred and eighty to Mozart,
Speaker 2: It likes back two hundred and fifty to Black.
Speaker 1: Right.
Speaker 2: Then I add them up and it's one and twenty points.
Speaker 2: It added twenty points, right, And so like, we have
Speaker 2: these experiences where it fucks up basic shit and it
Speaker 2: kind of acts like autistic savant thirteen year old some
Speaker 2: of the time. Right, how can it chain together enough
Speaker 2: steps without fucking up to kill everybody? Yeah?
Speaker 4: So, I mean currently they can't. I'm not out here
Speaker 4: saying tragipt is going to kill you tomorrow. The things
Speaker 4: where they're currently not terribly profitable are in some sense
Speaker 4: the like also the things where they're not very dangerous,
Speaker 4: like the stuff where they can't chain together a long
Speaker 4: term plan. So my read of the theory has sort
Speaker 4: of long predicted that the abilities to be more capable
Speaker 4: will come hand in hand with the abilities to be
Speaker 4: more dangerous. On this axis, I think we're starting to
Speaker 4: see some empirical evidence of that, where we see AIS
Speaker 4: that are sort of trained on problem solving and then
Speaker 4: they wind up, you know, being more tenacious on other
Speaker 4: types of problems. We have an example in the book
Speaker 4: of the AI that was trained on math puzzles and
Speaker 4: then was given a cybersecurity challenge, and the cybersecurity challenge
Speaker 4: was unknown to the programmer's misconfigured and the AI broke
Speaker 4: out of the testing environment and started up the server
Speaker 4: that was not started up correctly.
Speaker 3: This sort of behavior, it came for free.
Speaker 4: It came along with the ride of making this thing
Speaker 4: able to solve math problems a bit better, because to
Speaker 4: solve math problem as well, you need to learn some
Speaker 4: general skills of like knowing what your resources are looking for,
Speaker 4: clever solutions, not giving up even when it seems like
Speaker 4: that the task is insurmountable. Right, AIS aren't great at
Speaker 4: it yet, but the same path of making them more
Speaker 4: useful makes them, like gives them also more of the
Speaker 4: abilities to do stuff we wouldn't like if they keep
Speaker 4: having these drives no one asked where no one wanted.
Speaker 2: If you ask CHEPT did this recently too, What do
Speaker 2: you think the chances are at near time? Frames for
Speaker 2: recursive self improvement right, Meaning that the AI learns to
Speaker 2: make itself better is one simple way to put it.
Speaker 3: Right.
Speaker 2: It said, well, they're kind of like, they're kind of
Speaker 2: soft forms of this already, like helping humans with programming
Speaker 2: who are AI researchers helped a little bit, but it
Speaker 2: gave very low probabilities in the near term of recursive
Speaker 2: self improvement. Right. How important is that recursive loop to
Speaker 2: the scenarios that you are worried about? Should we trust
Speaker 2: chat GPT when it says I don't think this is
Speaker 2: very likely talk about that.
Speaker 1: That's what I was about to say. I was like
Speaker 1: that you can't trust that out, but that is.
Speaker 4: What I tell you either way. Yeah, I don't think
Speaker 4: that's true yet. The I don't think it's vital to
Speaker 4: the argument that you have a recursive self improvement sort
Speaker 4: of hard take off situation. I think it can go
Speaker 4: really badly in a lot of the slow takeoff situations.
Speaker 4: My best guess is still that you will hit some
Speaker 4: sort of recursive self improvement threshold at some point. One
Speaker 4: thing I'd say about that is that if you were
Speaker 4: looking at like chimpanzees and early hominids, a few million
Speaker 4: years ago, it would be really hard to call when
Speaker 4: things were going over some threshold such that it could
Speaker 4: come together. And humans aren't better. Humans don't have an
Speaker 4: extra walk on the Moon module. They don't have an
Speaker 4: extra engineering module. We have all the same stuff scaled up,
Speaker 4: and we're a little bit better at a lot of things.
Speaker 4: R lms the sort of thing where if you get
Speaker 4: a little better at a lot of things, they suddenly
Speaker 4: sort of like go over some threshold like how humans did,
Speaker 4: and it's all starts coming together.
Speaker 3: I don't know.
Speaker 4: I don't think anyone knows what I know what's going
Speaker 4: on inside these things. My best guess is, like probably not.
Speaker 4: My best guess is that you need some other architectural
Speaker 4: insight before it really starts to run away. But you know,
Speaker 4: it's all total guesswork.
Speaker 2: Do you expect super intelligence to be not plausible but
Speaker 2: likely based on large language models, or to be some
Speaker 2: fundamentally new technology. I think my.
Speaker 4: Top guess is that you need some other breakthrough on
Speaker 4: the size of like transformers, which maybe isn't totally new
Speaker 4: in the like transformers are still sort of in this
Speaker 4: deep learning paradigm, but like you know, one sort of
Speaker 4: analogy you might use for large language models is a
Speaker 4: little bit like you have like the intelligence of a
Speaker 4: six year old, but they get to live for a
Speaker 4: million years and read everything and practice a toneh and
Speaker 4: like sometimes you give a six year old, you know,
Speaker 4: the ability to write like ten books on trying to
Speaker 4: solve a math puzzle, and then it turns out they
Speaker 4: can solve that math puzzle after like writing out ten
Speaker 4: books and you read it and some of it's good
Speaker 4: and some of it's just like total nonsense. But like, okay,
Speaker 4: imagine that you somehow now had a seventeen year old,
Speaker 4: but you still have the ability to like give them
Speaker 4: a million years to think and the ability to write, like,
Speaker 4: you know, ten books per question they were answering, Like
Speaker 4: you could really see leaps and capabilities here. You could
Speaker 4: see that like relatively small improvements and algorithms make the
Speaker 4: amount the sheer enormous amount of competing power and data
Speaker 4: we have now just radically like overpowers the next generation
Speaker 4: of AI for the problems at hand, or maybe not.
Speaker 4: I don't know, it's hard to guess.
Speaker 2: Yeah, one of the I guess points of skepticism might
Speaker 2: have and again we're talking about you know, I'm kind
Speaker 2: of with you on scenario two versus scenario three, maybe
Speaker 2: both being pretty bad. Frankly, right, I think I'm a
Speaker 2: little more skeptical than you one on super intelligence. On
Speaker 2: your time frames, do you think developments over the past
Speaker 2: year suggest that general intelligence is further away than you
Speaker 2: might have thought at the start of the year. I mean,
Speaker 2: the AI twenty twenty seven authors, they're pushing back their
Speaker 2: timelines by a couple of years. You and Elliezer said
Speaker 2: that doesn't really matter. The sticks are very high, and
Speaker 2: AM twenty twenty seven ver twenty twenty nine verses twenty
Speaker 2: thirty five, doesn't really matter if we're all dead by
Speaker 2: twenty four. Yeah.
Speaker 1: Right.
Speaker 2: Have your timelines been pushed back at all, or forward
Speaker 2: for that matter, Yeah.
Speaker 4: I mean, I've never been one of the folks being like,
Speaker 4: I'm confident this is going to happen soon. And I
Speaker 4: mean day twenty twenty seven folks, I think weren't like,
Speaker 4: we're very confident this kind of happened here. But you know,
Speaker 4: I haven't really been going around saying I'm confident in
Speaker 4: short timelines. I have for a while said my best
Speaker 4: guess is that we're going to need another advancement past
Speaker 4: LMS towards that end. You know, the the evidence of
Speaker 4: the past year is mostly in line with my expectations,
Speaker 4: But my expectations were already that we'll probably need another
Speaker 4: advancement past LMS. You know, I do think there have
Speaker 4: been times where I was, you know, I updated towards
Speaker 4: LLLMS a couple of times, you know, the I think
Speaker 4: I remember GPT four O playing chess better than I
Speaker 4: thought elms were going to manage to play chess, and
Speaker 4: so that was like an eventual update towards maybe maybe
Speaker 4: ALMS will go farther than I thought they could. The
Speaker 4: reasoning models also were sort of an update in favor of, like,
Speaker 4: these guys aren't just going to stick with single forward
Speaker 4: pass LMS. They're going to like cleverly try to find
Speaker 4: ways to leverage it, and so you know, it goes
Speaker 4: up and down. Those were some like oh, maybe alums
Speaker 4: can and this year's but no, maybe all alums can't.
Speaker 4: But but yeah, largely, you know, I'm I'm I'm not
Speaker 4: here saying like watch out for the albums in particular,
Speaker 4: you know, and I was I was again thought this
Speaker 4: problem would be hard back in twenty twelve. That was
Speaker 4: even before it was super clear deep learning was gonna
Speaker 4: was gonna go go as hard as it did. Never
Speaker 4: mind LMS in particular.
Speaker 2: And we'll be right back after this break. There's kind
Speaker 2: of a class of what you might call non existential risks,
Speaker 2: which range from energy consumed by data centers to aguoridthic discrimination.
Speaker 2: Some of these are kind of progressive coded up to
Speaker 2: dystopian but not quite bocalyptic outcomes where extremely powerful corporations,
Speaker 2: hand in hand with AI models kind of deprive the
Speaker 2: world of much of its freedom and agency. Right, how
Speaker 2: compatible is everyone dies with those other concerns, because sometimes
Speaker 2: they seem to be like, you know, I read a
Speaker 2: lot on the Internet. I write about politics all the time, right,
Speaker 2: and I think the kind of average politics argue we're
Speaker 2: on Twitter or Blue Skies, like these weird sci fi nerds.
Speaker 2: I think it's going to be the terminator, right when
Speaker 2: really we should be to care about intrenching discrimination. And
Speaker 2: you know, you would think that the left would be
Speaker 2: on the evil corporations bandwagon. But because like AI is
Speaker 2: kind of tech pro coded, that scrambles it a little bit.
Speaker 2: What do you think about those ordinary concerns including by
Speaker 2: the way AI being not all that competent, and so
Speaker 2: humans entrusting critical systems to AIS that fail or are
Speaker 2: corruptible or fuck up.
Speaker 3: Yeah, I know.
Speaker 4: One piece of this puzzle I think is I think
Speaker 4: a lot of people are cynical towards the companies. I
Speaker 4: think that's often merited, but I think the cynicism is
Speaker 4: often a little bit misplaced, and they're like, oh, you know,
Speaker 4: the AIS all hype, it'll never go anywhere, it can't
Speaker 4: do any You know, there's a lot of ways that
Speaker 4: the current AI is dumb, but it's much easier to
Speaker 4: see how AI could take a leap in the future
Speaker 4: if you've seen AIS take a leap in the past.
Speaker 4: And these companies existed before chatbots. They see chatbots as
Speaker 4: a stepping stone. Their stated goal is to, you know,
Speaker 4: make eyes that can automate all human labor, and in
Speaker 4: some sense an even more cynical perspective from a certain
Speaker 4: point of view of like, these guys know they're risking
Speaker 4: everyone's lives and the rushing head anyway, and you know
Speaker 4: they're not shy about this. Sam Altman's talking about the
Speaker 4: dangerous here has gone down as the company has gotten
Speaker 4: more profitable, and it's sort of like the ones who
Speaker 4: are like more known for just saying what they actually believe,
Speaker 4: like Elon, which, for all that you can say about him,
Speaker 4: he's not really one to hide his thoughts. Who will
Speaker 4: come out and be like, oh yeah, ten twenty percent
Speaker 4: chance this kills you all. So one thing I would
Speaker 4: say there is like I.
Speaker 2: Sort of.
Speaker 4: Think it's even crazier than it looks on the surface.
Speaker 4: AI may be a bubble, it may be hype. The
Speaker 4: Internet had a bubble, the Internet was still real, you know,
Speaker 4: And you have all these academics being like, oh yeah,
Speaker 4: this is really dangerous. You have folks like us Oup
Speaker 4: and around before them being like, oh yeah, this is
Speaker 4: really dangerous. I would implore people to be like even
Speaker 4: more cynical about taking these people out their word that
Speaker 4: they are gambling with your very life. In terms of
Speaker 4: all of the other issues with AI, including ones we're
Speaker 4: sort of already starting to see today, unfortunately, the world
Speaker 4: is big enough for multiple issues. I'm not like, my
Speaker 4: issue is real. Those issues are fake. We're going to
Speaker 4: need to figure out how to integrate AI with education.
Speaker 4: It's true that if we make like robot police forces
Speaker 4: that would allow it to talitarian states to be much
Speaker 4: more brutal in en forcing it to talitarian regime because
Speaker 4: there's certain orders you can't give to a human police
Speaker 4: force because they won't do it, that you could give
Speaker 4: to a robotic police force. Right, those are real things
Speaker 4: we're gonna have to grapple with. Also, if we keep
Speaker 4: racing to ais that are smarter and smarter that can
Speaker 4: automate this process of technological innovation, which is what they
Speaker 4: say they're racing towards, the most likely outcome is that
Speaker 4: everyone on Earth dies. We just need to solve both
Speaker 4: set of problems.
Speaker 2: Yeah, I've updated. When I mean positive, I mean the
Speaker 2: p doom risk is going down, right, I've updated slightly
Speaker 2: positively as far as I think timeline is going to
Speaker 2: be a little longer than I always thought a year
Speaker 2: or two ago. I've updated negatively based on culture and politics,
Speaker 2: right on various frints. Yeah, Number one, the kind of
Speaker 2: workaholic culture of Silicon Valley. When I hear like the
Speaker 2: nine ninety six culture, whatever it is, right, but just crying.
Speaker 2: Go go to your lab, work really hard. Your friends
Speaker 2: should be in the lab, right, Your activity should be
Speaker 2: things that are health oriented. Or self improvement oriented. It's like,
Speaker 2: I know, I feel better if we had the tech
Speaker 2: bros going out and getting drunk on Friday night, right
Speaker 2: and trying to get laid and getting in touch with humanity,
Speaker 2: you know what I mean, And if they had like
Speaker 2: material things and children and things like that that they
Speaker 2: valued potentially. Yeah, I mean, like, what would you say,
Speaker 2: has happened to you know? Where would you put Elon now?
Speaker 3: Right?
Speaker 2: I believe that GROC is not regarded or XAI is
Speaker 2: not regarded as among the most responsible of the AI labs. Right,
Speaker 2: he has expressed, I assumed sincere concerned about p doom
Speaker 2: in the past, where would you where would you peg
Speaker 2: Elon now?
Speaker 4: I mean he's not You can just listen to his
Speaker 4: own words, you know. He said over the summer, I
Speaker 4: think realized I could either be a bystander or a participant,
Speaker 4: and that was like gonna happen one way or the other.
Speaker 3: Needed rather be a participant.
Speaker 4: But you know, also in terms of who's the safer one,
Speaker 4: who's the more dangerous one? It it looks to me
Speaker 4: like no one has anywhere nearing a chance of doing
Speaker 4: this job, right.
Speaker 2: Yeah, I mean you kind of it feels like you
Speaker 2: hear less about alignment than you did a year or two.
Speaker 2: It kind of felt like Trump one and everyone's like, oh,
Speaker 2: we're just accelerating now, right, Trump wins and China's becoming
Speaker 2: more competitive, and like, you know, it seems like people
Speaker 2: have given up on this whole alignment alignment thing, right,
Speaker 2: Is that true or well or yeah? Yeah, I mean
Speaker 2: I think a lot of it is that people like
Speaker 2: are largely realizing they can't. Yeah, right.
Speaker 4: Like you, you have interpretability, which is like trying to
Speaker 4: figure out what the heck is going on in there.
Speaker 4: You have evaluations, which is trying to figure out how
Speaker 4: dangerous it is already.
Speaker 3: And if if you were going to.
Speaker 4: Engineers and you were like, can you tell me about
Speaker 4: why this uh, this nuclear power plant is not going
Speaker 4: to melt down? And they're like, well, yeah, we have
Speaker 4: two safety teams. One is trying to figure out what
Speaker 4: the heck is going on inside the plant. The other
Speaker 4: is trying to measure whether it has exploded yet. You
Speaker 4: would be like, okay, guys, ease off on the growing
Speaker 4: pile of uranium, you know, And I'm like, I'm not.
Speaker 4: I'm not trying to like, uh degrade the people working
Speaker 4: on this. It's way better to have a team that's
Speaker 4: trying to figure out what's going on inside the nuclear
Speaker 4: power plant than to not, okay, it's it's way better
Speaker 4: to have someone measuring is it starting to explode? You know,
Speaker 4: the guys trying to do this are heroes in my book,
Speaker 4: but like it, we're not close, and I think part
Speaker 4: of why people aren't talking about is that we're not
Speaker 4: We're not close to being able to do the job right.
Speaker 1: Do you think that some of the tragedies that are happening,
Speaker 1: you know, every day with chatbots, you know, mental health problems,
Speaker 1: we see suicides, like we're seeing a lot of things
Speaker 1: happening even with this fairly primitive, even though it's so
Speaker 1: advanced technology. Do you think that that will be kind
Speaker 1: of enough of a warning sign for people to start
Speaker 1: taking note outside of the Silicon Valley elite who actually
Speaker 1: know this more closely, who have you read your book,
Speaker 1: who understand all of the implications.
Speaker 4: You know, I'm hopeful in some sense the chat GPT
Speaker 4: moment was more global awareness of the AI potential for
Speaker 4: killing us all than I expected to maybe ever get.
Speaker 4: You know, it's back before large language models a lot
Speaker 4: of people predicted that AI that could sort of like
Speaker 4: talk well was actually very late in the AI game.
Speaker 4: You know, we used to have like mor of x
Speaker 4: paradox where we'd say things that are like easy for
Speaker 4: computers are are hard for.
Speaker 3: Humans, and vice versa.
Speaker 4: And you know, it could have been that this just
Speaker 4: all stayed like very academic, very like only people in
Speaker 4: the industry are watching it like it was up until
Speaker 4: you know, maybe twenty twenty one. It could have stayed
Speaker 4: that way all the way up until you know, you
Speaker 4: have these AI companies making AI researcher ais in their
Speaker 4: labs and no one's noticing. And that gives me some
Speaker 4: hope that maybe we don't even need a disaster. Maybe
Speaker 4: we just need another capability advance. Maybe when people do
Speaker 4: live through, maybe people live through one more hop of
Speaker 4: like AI's doing stuff they qualitatively couldn't do before. Maybe
Speaker 4: that would be enough to wake people up. And then
Speaker 4: on the flip side, you know, there can also be
Speaker 4: big warning shots where people fail to learn from the
Speaker 4: warning shot. One thing I'd say there is that what
Speaker 4: you can learn from a warning shot, it depends a
Speaker 4: little bit on how prepared you already are and how
Speaker 4: knowledgeable you already are, and so you know there's no
Speaker 4: substitute for sort of getting people to understand the issues today.
Speaker 2: How much of the things we need to do as
Speaker 2: a society right to mitigate this risk are kind of
Speaker 2: technical steps versus political steps, like obviously, yeah, I'm sure
Speaker 2: you've had this elevator pitch where the first sixty seconds
Speaker 2: explaining why you're worried. Now, let's say I encounter you
Speaker 2: on the same elevator. We're now going back to our rooms.
Speaker 2: But I have another sixty seconds, right, what's the quick
Speaker 2: pitch on what we should do? Is there a one
Speaker 2: kind of genie wish if you could fiat something? Is
Speaker 2: it more complicated than that? Do we know what we
Speaker 2: have to do and it's a matter of forming the
Speaker 2: right political coalition or all right, you have sixty ish
Speaker 2: second snake.
Speaker 4: I think what needs to happen is the world needs
Speaker 4: to stop the race towards smarter than human AI, and
Speaker 4: it needs to stop it everywhere, because no one can
Speaker 4: keep these things in a leash. If anyone builds a
Speaker 4: super intelligence anywhere on the planet, everyone dies, doesn't matter
Speaker 4: who gets it. The world could stop AI is made
Speaker 4: using extremely specialized chips in very large data centers that
Speaker 4: suck down as much electricity as a city. It would
Speaker 4: be probably easier to monitor than uranium. And I think
Speaker 4: it's I think it's almost entirely political right now. I
Speaker 4: think we are nowhere near solving on the technical side.
Speaker 4: It's not to say it can't be solved technically, but
Speaker 4: not only are we nowhere near close, But unlike a
Speaker 4: lot of other scientific problems, we don't get free tries
Speaker 4: the first time you have some theory of you know, this,
Speaker 4: this AI will definitely be nice to us once it's
Speaker 4: smart enough to be a threat to us. If that
Speaker 4: theory is wrong, you sort of don't get to put
Speaker 4: the genie back in the bottle. And that's unlike other theories.
Speaker 4: That's unlike other science and a lot of other places in science.
Speaker 4: You know, Marie Curie dies of cancer those likely caused
Speaker 4: by radiation poisoning. She was a hero, But often the
Speaker 4: first people interacting with stuff die of them, you know,
Speaker 4: Max Valley, and.
Speaker 2: They probably did things that were objectively kind of crazy, right,
Speaker 2: But you have this selection mechanism where you know, the
Speaker 2: great crazy invention survive the other ones, don't. I mean
Speaker 2: how much would you have to So do you believe
Speaker 2: in the idea of like a slowdown? Right, Like, let's
Speaker 2: say that the people who are skeptical of rapid time
Speaker 2: horizons are correct and this technology develops over the course
Speaker 2: of fifty years instead of fifteen or five or five months.
Speaker 2: Is that enough time to get things right? Would it
Speaker 2: pause or a slow down work?
Speaker 4: You know, fifty is a lot more to work with
Speaker 4: than fifteen. But you got to worry about convergence on
Speaker 4: the wrong on a bad solution, and humanity so far
Speaker 4: has not been bringing its a game and to get
Speaker 4: around on the first try, it's it's hard. Fifty years,
Speaker 4: I could work with you, maybe have a chance, but
Speaker 4: I'd be trying to do things like could we use
Speaker 4: biotech to somehow enhance the intelligence of our humans? But
Speaker 4: I mean, first thing we do is we get a
Speaker 4: slow down somehow. Maybe we'll be liking get a slow down.
Speaker 4: Maybe we'll need to get a slow down by international FIAT.
Speaker 4: But a lot of people say, oh, it will never happen.
Speaker 4: People never slow down. One big reason for hope here,
Speaker 4: I think, is that our world leaders don't yet understand
Speaker 4: what the people in the valley are debating. They don't
Speaker 4: understand that we're in this situation where the possibilities are,
Speaker 4: like the tech just doesn't work, in which case betting
Speaker 4: and it will go nowhere, or you're either making god
Speaker 4: emperors or killing us all, and that the optimistic folks
Speaker 4: who think the tech works think there's like a twenty
Speaker 4: five percent chance it kills us all. You know, it's like, yeah,
Speaker 4: that's like we're racing, but we're racing in a situation
Speaker 4: where people don't realize that either this race goes nowhere
Speaker 4: or go to really bad places.
Speaker 2: Let me ask you kind of a meta question, what
Speaker 2: do you see your role as in this debate?
Speaker 4: I think my role, in Eliezer's role in this situation
Speaker 4: is to come out and say the things that a
Speaker 4: lot of people are thinking, and to sort of like
Speaker 4: to be the kid who says the emperor has no clothes.
Speaker 4: There's a lot of people who are worried here, but
Speaker 4: no one's sort of coming right out and saying this
Speaker 4: is crazy. We should stop. The labs who say there's
Speaker 4: a twenty five percent chance this kills us all, don't
Speaker 4: then do the next step and say so, please put
Speaker 4: a stop to it. Right, My role here is to
Speaker 4: sort of like spell out that last step and be like, guys,
Speaker 4: this is nuts. We shouldn't be doing it. It's the
Speaker 4: obvious implication of what everyone else has been saying. So
Speaker 4: you'll be I wish it was not stepping up.
Speaker 2: But if you and Eliezer are at the Utopian retirement
Speaker 2: home in one hundred and fifty years, because now human
Speaker 2: beings lived to you know, one hundred ninety years old
Speaker 2: on average or something, and your predictions are totally wrong
Speaker 2: and AI is absolutely wonderful. Okay, good.
Speaker 4: I would absolutely love to be wrong. I would I
Speaker 4: would be so thrilled. Yeah, no spoilers about my book,
Speaker 4: but it ends with us talking about how thrilled we
Speaker 4: would be.
Speaker 3: To be wrong.
Speaker 2: Okay, I just sent you a calendar invite for October thirtieth,
Speaker 2: two thousand and thirty five, where we can have a
Speaker 2: follow up, right podcast, Nate Perfect any closing thoughts. Has
Speaker 2: been a lot of fun to be with you.
Speaker 4: Yeah, I mean, first and foremost, glad that you're you're
Speaker 4: having the convo. Glad that we're having the convo. I
Speaker 4: think just more people realizing how crazy the situation is
Speaker 4: will go a long way.
Speaker 3: Towards making things better.
Speaker 1: Thank you so much for joining us, Nate, it's been
Speaker 1: an absolute pleasure, pleasure to be here.
Speaker 2: We'll talk to you soon.
Speaker 1: Let us know what you think of the show. Reach
Speaker 1: out to us at Risky Business at pushkin dot FM.
Speaker 1: Risky Business is hosted by me Maria Kanakova.
Speaker 2: And by me Nate Silver. The show was a cool
Speaker 2: production of Pushing Industries and iHeartMedia. This episode was produced
Speaker 2: by Isaac Carter. Our associate producer is Sonya gerwit Lydia,
Speaker 2: Jean Kott and Daphne Chen are our editors, and our
Speaker 2: executive producer is Jacob Goldstein. Mixing by Sarah Bruger.
Speaker 1: If you like the show, please rate and review us
Speaker 1: so other people can find us too, But once again,
Speaker 1: only if you like us. We don't want those bad
Speaker 1: reviews out there. Thanks for tuning in.