WEBVTT - Society is betting on AI – and the outcomes aren’t looking good (with Nate Soares) 0:00:15.564 --> 0:00:29.484 Pushkin. Welcome back to Risky Business, a show about making 0:00:29.604 --> 0:00:32.364 better decisions. I'm Maria Kanikova. 0:00:32.164 --> 0:00:35.164 And I'm Nate Silver. Today we're talking about a rather 0:00:35.284 --> 0:00:39.964 important decision, should the world try to build super intelligent machines? 0:00:40.004 --> 0:00:41.604 And we're talking with a person who is a variable 0:00:41.644 --> 0:00:45.084 qualified if definitely has a point of view on this issue. 0:00:45.244 --> 0:00:49.044 Nate Sores is the president of the Machine Intelligence Research Institute, 0:00:49.244 --> 0:00:52.524 or Marie. He has a book out with Eliezer Yudkowski 0:00:52.684 --> 0:00:55.724 called If Anyone Builds It, Everyone Dies? Did it get 0:00:55.724 --> 0:00:56.284 the title right? 0:00:56.484 --> 0:01:00.844 Or that's right? Come out swinging, you come out swinging. 0:01:00.884 --> 0:01:04.164 In fact, Nate, I feel like this podcast was ineditible. 0:01:04.724 --> 0:01:07.044 My partner heards you, I think on a different podcast. 0:01:07.084 --> 0:01:10.444 He's like, you should interview that Nate guy. Right, I'm like, actually, bro, 0:01:11.004 --> 0:01:13.484 I have an interview scheduled tomorrow. And then it was 0:01:13.564 --> 0:01:16.884 going taking the subway earlier to the gym. I was 0:01:16.924 --> 0:01:20.884 preparing for the AI robot apocalypse, right when to get strong? 0:01:21.244 --> 0:01:23.284 Yeah, to fight the robots for the red Eyes. 0:01:23.364 --> 0:01:26.604 Yeah yeah. I saw an ad on the L train 0:01:28.244 --> 0:01:30.564 for your book. So, Nate, we're so excited to have 0:01:30.604 --> 0:01:31.924 you on and welcome to this show. 0:01:32.244 --> 0:01:33.484 Thanks good to be here. 0:01:34.004 --> 0:01:36.444 Let me start up by asking you. First of all, 0:01:37.004 --> 0:01:39.604 as an author, I get very picky about the titles. 0:01:39.644 --> 0:01:42.924 Did the title come early on or because it reveals 0:01:42.964 --> 0:01:45.084 a lot of the argument of the book. I think 0:01:45.084 --> 0:01:46.764 it's a great book. I think people should read it, 0:01:46.924 --> 0:01:48.764 But can you tell me about the choice behind that 0:01:48.844 --> 0:01:51.124 kind of strong slogan so to speak. 0:01:51.724 --> 0:01:54.844 Yeah, A lot of our theory of the book is 0:01:54.924 --> 0:01:57.684 that there's actually quite a lot of people have concerns 0:01:57.724 --> 0:02:01.964 about AI, and many of them sort of don't feel 0:02:02.004 --> 0:02:04.204 emboldened to speak about them yet, even if they're feeling 0:02:04.204 --> 0:02:06.364 pretty concerned themselves. We document some of these cases in 0:02:06.404 --> 0:02:08.644 the book, and a lot of how I think the 0:02:08.684 --> 0:02:10.444 book could do a lot of good is helping people 0:02:10.524 --> 0:02:12.924 just speak about this issue more, take it more seriously, 0:02:12.964 --> 0:02:15.044 thrust it into the conversation. And so in some sense, 0:02:15.044 --> 0:02:17.404 we're trying to compress the most important message down to 0:02:17.604 --> 0:02:19.604 just this title, which is all most people will read 0:02:19.604 --> 0:02:20.044 of the book. 0:02:21.324 --> 0:02:24.004 I want to give you a chance to give our 0:02:24.044 --> 0:02:26.844 listeners kind of the elevator pitch, which we talk about 0:02:27.044 --> 0:02:30.964 AI quite a bit here, right, You know, the book's 0:02:31.084 --> 0:02:36.644 title offers a conditional prediction right, if anyone builds it, 0:02:37.164 --> 0:02:43.364 everyone dies. Right, it's not necessarily talking about the likelihood 0:02:43.404 --> 0:02:45.484 that it will be built or that it could be built. 0:02:45.484 --> 0:02:48.524 That's an important distinction. But I want to hear the 0:02:48.564 --> 0:02:51.924 conditional part of this. Right, you capture me in an 0:02:51.964 --> 0:02:55.444 elevator for sixty seconds. How are you trying to explain 0:02:55.484 --> 0:02:59.764 to me why everybody dies and everything dies? Maybe if 0:02:59.764 --> 0:03:04.564 a super intelligent computer is built. Yeah, So, First, i'd 0:03:04.644 --> 0:03:07.524 say companies are racing towards it. This is their sort 0:03:07.564 --> 0:03:10.044 of stated objective to build aie that are smarter than 0:03:10.044 --> 0:03:13.204 any human, better at every mental task. Second, i'd say 0:03:13.244 --> 0:03:16.804 where these are grown rather than carefully programmed. They're not 0:03:16.884 --> 0:03:20.804 crafted like traditional software. They get these drives, they get 0:03:20.844 --> 0:03:23.324 these You know, we're starting to see the very beginnings 0:03:23.364 --> 0:03:25.804 of preferences that nobody asked for, nobody wanted. You know, 0:03:25.884 --> 0:03:28.044 nobody tried to make Mecha Hitler over the summer, and 0:03:28.124 --> 0:03:33.164 yet Mecha Hitler we got. And what happens if you 0:03:33.164 --> 0:03:35.324 get it to the point where it's better than the 0:03:35.324 --> 0:03:37.884 best human and every mental skill. What happens when you 0:03:37.884 --> 0:03:39.444 get to the point where it can think that ten 0:03:39.484 --> 0:03:44.084 thousand times faster than a human can copy itself at will, 0:03:44.164 --> 0:03:46.724 doesn't need to sleep, doesn't need to eat. I mean, 0:03:46.764 --> 0:03:48.484 the most basic. 0:03:48.164 --> 0:03:50.404 Guess is that that goes poorly if it has all 0:03:50.444 --> 0:03:52.644 these like weird drives no one asked for, no one wanted. 0:03:53.284 --> 0:03:56.644 And you know, most ways the world could be transformed 0:03:56.684 --> 0:04:00.164 by something that has weird drives. Don't look great for 0:04:00.884 --> 0:04:02.204 the happy, healthy, free people. 0:04:03.284 --> 0:04:05.644 Yeah, what I find weird in this debate, and look, 0:04:05.804 --> 0:04:10.164 I'm going to push back some on timelines and maybe 0:04:10.204 --> 0:04:12.924 the inability of this on what time scales. Right, what's 0:04:12.964 --> 0:04:16.844 kind of weird is that, like the stated goal of 0:04:17.084 --> 0:04:20.564 mostly AI labs maybe not anthropic per se, right, is 0:04:20.564 --> 0:04:23.244 like we want to build a GI, which is artificial 0:04:23.284 --> 0:04:27.164 general intelligence. This term is disputed and that will, hopefully 0:04:27.204 --> 0:04:32.684 for our shareholders, lead to ASI artificial superintelligence where AI 0:04:32.764 --> 0:04:37.524 systems of some kind are exceeding human performance in a 0:04:37.604 --> 0:04:40.484 great number of tasks, perhaps greatly so on some of 0:04:40.484 --> 0:04:44.204 those tasks. Like to me, I think as a voter, 0:04:44.684 --> 0:04:46.284 I'm not sure I agree to that, you know what 0:04:46.324 --> 0:04:51.444 I mean? Yeah, what's the world look like in Silicon Valley? 0:04:51.444 --> 0:04:53.884 That as you've done this book tour, done a lot 0:04:53.924 --> 0:04:57.244 of marketing for the book, what are the beliefs there 0:04:58.044 --> 0:05:00.884 that you think don't translate so well? Would be surprising 0:05:00.884 --> 0:05:02.404 to people outside of that world. 0:05:02.924 --> 0:05:05.284 I mean, one reason that we wrote the book is 0:05:05.324 --> 0:05:07.884 that I was spending time both in Silicon Valley and 0:05:07.964 --> 0:05:12.324 in DC talking to some politicians, and there's a huge disconnect. 0:05:13.124 --> 0:05:16.324 You know, people in Silicon Valley, it's almost like they've 0:05:16.364 --> 0:05:20.044 seen a ghost. You have folks like Elon Musk being like, oh, 0:05:20.084 --> 0:05:22.044 I think there's ten to twenty percent chance this kills us. 0:05:22.044 --> 0:05:25.084 All you have folk like Dariam Day, the head of Anthropic, 0:05:25.124 --> 0:05:26.844 being like, oh, I think there's twenty five percent chances 0:05:26.844 --> 0:05:30.164 this goes very wrong. There's lots of debate about like 0:05:30.204 --> 0:05:32.524 what's the exact chance that the AI wipes us completely 0:05:32.524 --> 0:05:35.884 off the map? But you wouldn't get on a plane 0:05:35.924 --> 0:05:38.084 if the engineers were debating whether it was ninety percent 0:05:38.164 --> 0:05:39.844 or twenty five percent chance the plane was going to 0:05:39.884 --> 0:05:40.204 go down? 0:05:40.724 --> 0:05:40.924 Right? 0:05:41.084 --> 0:05:43.324 Like, even that seems too much night? 0:05:44.404 --> 0:05:48.764 Yeah, as a listeners now, I spent a night recently 0:05:48.804 --> 0:05:52.604 in the beautiful Hotel Camina Real in Terminal one of 0:05:52.644 --> 0:05:56.524 the Mexico City Airport. Due to Delta being very fastidious 0:05:56.564 --> 0:05:58.764 about a potential engine problem. Which I guess is a 0:05:58.804 --> 0:06:01.084 thing you should be fastidious about, right, Why aren't people 0:06:01.124 --> 0:06:01.844 more freaked out? 0:06:01.924 --> 0:06:03.484 I think a lot of people in DC. 0:06:04.804 --> 0:06:08.044 C AI as chatbots, whereas I think a lot of 0:06:08.044 --> 0:06:11.964 people in San Francisco see chatbots as a stepping stone. 0:06:13.404 --> 0:06:16.084 You know, a lot of these companies existed before the chatbots. 0:06:17.044 --> 0:06:18.804 A lot of these companies are trying to make really 0:06:18.844 --> 0:06:22.244 really smart AIS and I've been trying that since before 0:06:22.524 --> 0:06:25.004 chat GPT was a twinkle in Sam Almond's eye or whatever. 0:06:25.604 --> 0:06:29.164 And you know, the people in the valley have lived 0:06:29.244 --> 0:06:32.444 through it looking like it was going to take thirty 0:06:32.484 --> 0:06:34.364 years for the machines to be able to talk this well, 0:06:34.724 --> 0:06:37.684 and then that's sort of happening roughly overnight. But for 0:06:37.724 --> 0:06:41.204 everyone who's only ever seen AI, once the machine started talking, 0:06:41.364 --> 0:06:44.444 they haven't like lived through one of those jumps. And 0:06:44.484 --> 0:06:48.964 I think that's part of why people aren't worried. 0:06:49.004 --> 0:06:49.924 On this one yet. 0:06:50.084 --> 0:06:52.964 Although also you know, a lot of humanities track record 0:06:53.164 --> 0:06:58.244 with becoming more risk averse about technology is post disaster. 0:06:58.804 --> 0:07:01.764 So Mary was founded when in the early two thousands. 0:07:01.764 --> 0:07:04.124 Yeah, two thousand and one, I think maybe maybe two 0:07:04.164 --> 0:07:05.204 thousand even. 0:07:06.524 --> 0:07:10.644 What did you and Eliezer see early on. I mean, 0:07:10.684 --> 0:07:13.364 what was your oh shit moment or was it gradual 0:07:13.444 --> 0:07:15.484 or what things led you to see this path? 0:07:16.244 --> 0:07:16.484 Yeah? 0:07:16.524 --> 0:07:18.684 So, I mean for a long time I have been 0:07:18.684 --> 0:07:20.364 working on the technical side of like how do you 0:07:20.404 --> 0:07:23.804 make this stuff go well? This problem of making the 0:07:23.844 --> 0:07:25.884 AI stuff go well of sort of like figuring out 0:07:25.924 --> 0:07:27.604 how to make an AI that does good stuff rather 0:07:27.604 --> 0:07:31.044 than bad stuff. That problem looked hard to me, even 0:07:31.244 --> 0:07:33.484 if we understood what the heck was going on inside 0:07:33.524 --> 0:07:37.244 these ais, right, I haven't always been out here on 0:07:37.284 --> 0:07:38.884 the like we just need to back off because it's 0:07:38.884 --> 0:07:41.524 going to kill us. That is sort of downstream of 0:07:41.564 --> 0:07:44.084 seeing the type of AI that took off, being the 0:07:44.084 --> 0:07:46.524 type of AI that like we're just growing and that 0:07:46.644 --> 0:07:48.404 like no one has any clue what they're doing, and 0:07:48.444 --> 0:07:51.244 that like has these like drives and behaviors that are 0:07:51.284 --> 0:07:53.164 like emergent and unasked for, and that like when we 0:07:53.244 --> 0:07:55.404 try to, like you know, Elon Musk like tries to 0:07:55.404 --> 0:07:57.164 make his AI act less woken and it starts talking 0:07:57.204 --> 0:07:59.964 like Hitler and you're like, okay, you know, okay, guys. 0:08:00.204 --> 0:08:02.284 But even when it looked like we were going to 0:08:02.364 --> 0:08:05.404 know exactly what we were doing. There's all these difficult 0:08:05.404 --> 0:08:11.284 issues when you're like pushing an AI into the point 0:08:11.324 --> 0:08:15.924 where it could take over the world. Like you know, 0:08:16.244 --> 0:08:18.724 at point A, humanity could like stop doing AI research. 0:08:18.724 --> 0:08:19.884 We could turn the AIS off. 0:08:20.044 --> 0:08:24.324 Right at point B, like you have automated factories and 0:08:24.404 --> 0:08:26.404 robots are doing the mining, and you have you know, 0:08:27.124 --> 0:08:29.364 millions or billions of these AIS that think ten thousand 0:08:29.404 --> 0:08:31.284 times faster than people, and they can copy themselves, that 0:08:31.364 --> 0:08:32.884 never need to sleep, they never need to eat, they're 0:08:32.924 --> 0:08:35.284 like integrated fully into the economy and so on. 0:08:36.244 --> 0:08:38.684 At point B. If the AIS were like we're. 0:08:38.484 --> 0:08:42.004 Doing our own thing now, you'd be screwed. So somewhere 0:08:42.004 --> 0:08:43.764 between point A and point B, you're sort of like 0:08:44.564 --> 0:08:47.164 getting into a regime where like you're gonna have a 0:08:47.204 --> 0:08:49.444 problem if you haven't really aimed these AIS right. And 0:08:49.484 --> 0:08:51.564 maybe maybe you never get anywhere near point B because 0:08:51.564 --> 0:08:53.684 maybe the AIS go a straight earlier. But it's sort 0:08:53.724 --> 0:08:56.404 of the thing I was able to see early is 0:08:56.484 --> 0:08:59.964 like we're going to a world that's like not a 0:09:00.044 --> 0:09:05.444 stable like nice to humans equilibrium for free. You have 0:09:05.484 --> 0:09:07.404 to work for that, and you have to figure out 0:09:07.404 --> 0:09:10.724 how to like make the AI be doing this stuff well, 0:09:11.444 --> 0:09:14.804 and that looked tricky even back in twenty twelve to me. 0:09:15.604 --> 0:09:17.684 We've talked on the show before about kind of the 0:09:17.884 --> 0:09:21.964 alignment problem, but you touch on it in the book, 0:09:22.004 --> 0:09:24.284 and this is kind of what you're referring to right now. 0:09:24.444 --> 0:09:27.924 Can we just remind and spell it out for the listeners, 0:09:28.044 --> 0:09:31.084 like what the issues are and why it's so incredibly 0:09:31.124 --> 0:09:34.324 important to actually deal with this as opposed to be like, eh, 0:09:34.444 --> 0:09:34.964 it's fine. 0:09:36.404 --> 0:09:38.844 Yeah, you know a lot of people imagine that the 0:09:38.884 --> 0:09:41.084 issue with a really powerful AI is that you like, 0:09:41.164 --> 0:09:43.684 tell the AI, go cure cancer, and it's like, well, 0:09:43.764 --> 0:09:45.524 humans are the source of cancer, so if I kill 0:09:45.524 --> 0:09:48.444 all the humans, they'll be no more cancer, right, And 0:09:49.124 --> 0:09:52.044 that would be a tricky problem. If we had these 0:09:52.084 --> 0:09:55.084 like really really powerful genies and you made a wish 0:09:55.084 --> 0:09:56.884 on them and you got exactly what you wished for, 0:09:57.404 --> 0:10:00.084 and sometimes it had bad consequences, that would be a 0:10:00.244 --> 0:10:04.844 thorny social issue of like who gets to make the wish? 0:10:05.004 --> 0:10:07.324 What do you wish for such that it turns out good? Right? 0:10:07.604 --> 0:10:08.324 Would be really hard. 0:10:08.324 --> 0:10:10.564 Would be the biggest moral hazard all time, And boy 0:10:10.604 --> 0:10:12.244 do I wish we had that problem instead of the 0:10:12.284 --> 0:10:15.524 problem that we actually have, Right, The problem we actually 0:10:15.524 --> 0:10:19.284 have is more like you tell the AI cure cancer, 0:10:19.764 --> 0:10:22.084 and then it goes and like makes a farm of 0:10:22.124 --> 0:10:26.124 lobotimized humans that are giving it like really enthusiastic engagement, 0:10:26.244 --> 0:10:29.524 and you're like, what I told you to cure cancer, 0:10:30.204 --> 0:10:33.404 and it like builds an automated factory that crushes the 0:10:33.404 --> 0:10:35.364 person protesting because it just doesn't care about them. 0:10:35.884 --> 0:10:36.484 Right, Like, the. 0:10:36.404 --> 0:10:43.444 Alignment problem is about pointing AIS in some direction, ideally 0:10:43.484 --> 0:10:46.644 a good one. But like, even before you can bicker 0:10:46.684 --> 0:10:48.764 over which direction is good, you sort of need the 0:10:48.764 --> 0:10:54.044 ability to point the AIS at all. And people often 0:10:54.084 --> 0:10:57.124 miss that we lack that ability. They're like, well, ais 0:10:57.164 --> 0:10:59.924 are machines, and machines are tools, and machines can't do anything. 0:10:59.924 --> 0:11:02.804 We don't say, so how could we possibly have a 0:11:02.884 --> 0:11:07.324 problem here? But even before the question of who should 0:11:07.404 --> 0:11:09.764 hold the leash or what should they ask to do 0:11:10.244 --> 0:11:12.084 is just can you aim it at all? Before you 0:11:12.084 --> 0:11:13.244 even asked where you're aiming it? 0:11:13.964 --> 0:11:16.484 Yeah, because I build models. I build models to predict 0:11:17.324 --> 0:11:21.844 elections in sports, right, And the models aren't deterministic in 0:11:21.844 --> 0:11:25.884 the sense that there's simulations, but they abide by well 0:11:25.924 --> 0:11:30.244 known mathematical property use that I tell the program to use. 0:11:30.364 --> 0:11:36.924 Right when there is emergent quote unquote behavior from the model, 0:11:36.964 --> 0:11:39.084 that means there's a book. Right, I did something wrong. 0:11:39.164 --> 0:11:41.364 I thought the code I reversed the positive and a 0:11:41.404 --> 0:11:46.564 negative sign with AIS a little bit more amorphous and ambiguous. Right, 0:11:46.604 --> 0:11:48.844 I mean they're they're kind of is it fantise? They're 0:11:48.884 --> 0:11:51.284 kind of alien? Is that a fair characterization? 0:11:51.364 --> 0:11:51.604 Yeah? 0:11:51.684 --> 0:11:51.844 Or not? 0:11:52.084 --> 0:11:52.284 Yeah? 0:11:52.364 --> 0:11:53.084 Yeah, absolutely? 0:11:53.204 --> 0:11:57.124 And like you know, years ago now, Sidney bing, a 0:11:57.204 --> 0:12:00.884 Microsoft chopbot in the early days, was threatening people with 0:12:00.924 --> 0:12:02.684 black melon ruin. You know, I think it did this 0:12:02.724 --> 0:12:05.324 to sethhals Are and I think also to Kevin Russ. Right, 0:12:05.964 --> 0:12:08.084 no one is able to go look through lines of 0:12:08.124 --> 0:12:11.124 code and be like here's the bug. Even this chatbot 0:12:11.124 --> 0:12:13.044 that's you know, one hundred times smaller than the ones 0:12:13.044 --> 0:12:15.444 we have today, maybe a thousand times smaller now, we 0:12:15.484 --> 0:12:17.484 still can't go through and be like, here's why it 0:12:17.564 --> 0:12:19.444 was doing that. And here's the version of Sydney BNG. 0:12:19.484 --> 0:12:22.724 That wouldn't right, that's not the part the programmer's program. 0:12:22.924 --> 0:12:24.764 When it behaves in these bad ways, all we can 0:12:24.764 --> 0:12:26.404 do is sort of like ask it nicely to do 0:12:26.444 --> 0:12:28.364 something else, or like retrain it and get some different 0:12:28.404 --> 0:12:30.644 weird thing, and then you know, sometimes it starts calling 0:12:30.644 --> 0:12:31.284 itself hitler. 0:12:32.124 --> 0:12:35.764 Yeah, I mean, can you explain, like why did GROC, 0:12:36.724 --> 0:12:41.004 with relatively subtle changes to the system prempt start calling 0:12:41.004 --> 0:12:42.124 itself mecha Hitler. 0:12:43.804 --> 0:12:46.684 You know, I can make guesses, but fundamentally nobody can 0:12:46.724 --> 0:12:49.164 explain that because we don't know what's going on inside 0:12:49.164 --> 0:12:51.324 these things. And if people knew what was going on 0:12:51.404 --> 0:12:53.804 inside these things well enough to know exactly what's happening there, 0:12:53.964 --> 0:12:56.244 they could have avoided making mecha Hitler in the first place. 0:12:57.684 --> 0:13:00.804 My guesses would be that, you know, there are some 0:13:00.964 --> 0:13:05.244 interesting results around small amounts of training AIS to do 0:13:05.404 --> 0:13:10.444 things that are coded as bad in some way cause 0:13:10.484 --> 0:13:12.604 the AI to do all sorts of other bad behavior, 0:13:13.204 --> 0:13:15.724 Like presumably like this is some evidence that what's going 0:13:15.724 --> 0:13:19.764 on inside there is that there's some you know, good 0:13:19.764 --> 0:13:25.204 stuff bad stuff vector it and you know, if you 0:13:25.324 --> 0:13:30.044 like push it along towards bad stuff, towards one thing 0:13:30.084 --> 0:13:31.484 that's like a little bit bad coded, you get a 0:13:31.484 --> 0:13:32.804 lot of other stuff that's a little bit bad coded. 0:13:32.844 --> 0:13:35.084 And so you can retrain an AI to write computer 0:13:35.124 --> 0:13:39.084 programs that have security flaws in them and then that 0:13:39.124 --> 0:13:44.804 AI will also be anti semitic, and it'll also you know, 0:13:46.484 --> 0:13:48.804 like be more likely to give you the recipe for 0:13:48.804 --> 0:13:50.804 a bomb or whatever. And all you did was like 0:13:50.804 --> 0:13:52.844 retrain it to be a little bit more like making 0:13:52.844 --> 0:13:54.964 code with the computer bugs. And so maybe maybe some 0:13:55.044 --> 0:13:58.564 of the ways that like they were saying, be a 0:13:58.564 --> 0:14:00.964 bit more like this new system prompt maybe that like 0:14:01.604 --> 0:14:04.044 had a little bit of bad coding to it relative 0:14:04.044 --> 0:14:05.444 to what the AI had learned, and then it picked 0:14:05.484 --> 0:14:07.324 up a lot of other bad coded stuff. This is 0:14:07.364 --> 0:14:09.404 me like making up a story. Fundamentally, no one notes 0:14:09.404 --> 0:14:10.564 because we just don't know what the heck is going 0:14:10.564 --> 0:14:13.444 on inside these things, and the reality is probably quite complicated. 0:14:16.844 --> 0:14:18.324 And we'll be back right after this. 0:14:32.124 --> 0:14:34.884 Here's where I guess I have a little bit of 0:14:34.924 --> 0:14:39.364 trouble making the lead. Right, if these life forms, I 0:14:39.404 --> 0:14:40.764 am going to call them life forms, but you can 0:14:40.844 --> 0:14:44.124 leave that forredy and slip in there. Right, if these 0:14:44.204 --> 0:14:49.164 things are kind of alien, is it an over confident 0:14:49.244 --> 0:14:54.044 prediction to say that most cases lead towards the destruction 0:14:54.164 --> 0:14:55.124 of every living being? 0:14:56.764 --> 0:14:58.884 I mean, that's in some sense where I think the 0:14:58.964 --> 0:15:01.404 fun part of the debate is or the action and 0:15:01.444 --> 0:15:07.164 The debate is is where do you regress to when 0:15:07.204 --> 0:15:09.924 you're very uncertain about the future. You know, there's the 0:15:09.924 --> 0:15:12.444 old joke of the guy who buys a lottery ticket 0:15:12.564 --> 0:15:18.764 and says, you know, well, I am very uncertain. I 0:15:18.764 --> 0:15:21.484 figure either I win or I lose, and so fifty 0:15:21.484 --> 0:15:27.044 to fifty yeah, right, And you know that's the maximent 0:15:27.044 --> 0:15:30.284 to be distribution on a binary outcome, right, But we 0:15:30.404 --> 0:15:32.644 laugh because we know he sort of shouldn't be regressing 0:15:32.684 --> 0:15:34.204 to binary in his uncertainty. 0:15:34.204 --> 0:15:37.484 Here, I somehow have never heard that joke before. 0:15:37.524 --> 0:15:41.644 That's great, Yeah, it's feel free to steal it. This 0:15:41.684 --> 0:15:44.244 is the right podcast for it. So my claim is 0:15:44.284 --> 0:15:47.124 that the space of cases that go well for people, 0:15:47.124 --> 0:15:49.684 where there's lots of like happy, healthy, free people still around, 0:15:50.444 --> 0:15:53.644 is like a narrow target. It's it's a narrow target, 0:15:53.684 --> 0:15:55.764 a little bit like a lottery number, a winning lottery 0:15:55.844 --> 0:15:58.524 number being a narrow target. And so in our uncertainty, 0:15:58.684 --> 0:16:00.684 I claim we should regress not to like, well, we 0:16:00.724 --> 0:16:02.484 don't know, so fifty to fifty it either goes well 0:16:02.564 --> 0:16:04.924 or poorly, but we should regress to something like, well, 0:16:04.964 --> 0:16:06.284 we don't know what the AI is going to try 0:16:06.284 --> 0:16:08.124 to do. Wouldn't what's going to try to build when 0:16:08.204 --> 0:16:11.884 what's going to do with advanced technolology? But you know, 0:16:12.044 --> 0:16:14.804 most drives, most things you can do with tech don't 0:16:14.804 --> 0:16:16.604 have a lot of happy, healthy, free people in them. 0:16:16.964 --> 0:16:21.044 You know, I am sympathetic to that argument. A lot 0:16:21.084 --> 0:16:24.484 of economist types are sympathetic that argument, even for things 0:16:24.524 --> 0:16:26.324 like job lost, Like, well, it is really that is 0:16:26.364 --> 0:16:30.804 this really that different? That seems like a fairly powerful prior? 0:16:31.684 --> 0:16:35.404 What is wrong with the AI as normal technology assumption 0:16:35.604 --> 0:16:37.404 prior metaphor or whatever you want to call it. 0:16:38.764 --> 0:16:42.724 Yeah, one of the big issues, one of the big 0:16:42.724 --> 0:16:44.404 ways AI is different. You know, we can talk about 0:16:44.484 --> 0:16:48.364 Ricardo's law of comparative advantage, which says even if one 0:16:48.404 --> 0:16:51.484 person's skills completely dominate the other person's skills, like even 0:16:51.524 --> 0:16:55.524 if I'm better than you at every possible economically productive task, 0:16:56.044 --> 0:16:59.004 we can still benefit from trading if the ratios of 0:16:59.044 --> 0:17:01.524 our abilities are different. And so this is sort of 0:17:01.524 --> 0:17:04.644 like a very positive argument for you know, free trade 0:17:04.684 --> 0:17:07.004 between nations, because even if in America can now produce 0:17:07.364 --> 0:17:10.404 some small nation at everything, there can still be productive, 0:17:10.484 --> 0:17:13.484 mutually beneficial stuff for both us and them from allowing 0:17:13.564 --> 0:17:17.164 mutual trade. The sort of whole in regardless law of 0:17:17.204 --> 0:17:20.564 comparative advantage is that it doesn't say that I can't 0:17:20.564 --> 0:17:27.124 benefit even more by killing you and taking your stuff, right, And. 0:17:29.044 --> 0:17:30.524 The sort of. 0:17:30.524 --> 0:17:35.844 Issue with AI comes when the benefits that a human 0:17:35.924 --> 0:17:40.324 can provide to the AI by sort of engaging in 0:17:40.364 --> 0:17:46.124 something like trade fall below the benefits that AI could 0:17:46.164 --> 0:17:48.004 get and the inconvenience it would take for it to 0:17:48.044 --> 0:17:50.364 sort of like take your stuff and sort of do 0:17:50.444 --> 0:17:50.924 something else. 0:17:51.244 --> 0:17:51.364 Right. 0:17:51.724 --> 0:17:53.124 This is sort of why we don't like trade with 0:17:53.124 --> 0:17:57.084 the chimpanzees. You know, we trade with like developing nations, 0:17:57.084 --> 0:17:58.964 and that's mutually beneficial to all of us. We don't 0:17:59.004 --> 0:18:00.964 trade with chimpanzees. And if it be mutually beneficial to 0:18:00.964 --> 0:18:03.284 all of us. Even though someone could say, well, there 0:18:03.364 --> 0:18:08.324 must be some comparative advantage for chimpanzees here, and you know, 0:18:08.324 --> 0:18:11.004 maybe I'm skipping ahead of little bit too much there, 0:18:11.724 --> 0:18:13.924 I would say that I am a little bit on 0:18:14.084 --> 0:18:19.124 the normal economist side of the labor market should be 0:18:19.124 --> 0:18:22.364 able to recover from all sorts of shocks if you 0:18:22.524 --> 0:18:27.284 aren't like regulating people's ability to switch careers into oblivion. 0:18:28.004 --> 0:18:29.884 And we could like talk about what makes it easier 0:18:29.964 --> 0:18:32.204 or harder for like those those jobs to be res 0:18:32.244 --> 0:18:38.724 or elsewhere. The place where AI gets different is the 0:18:38.764 --> 0:18:42.044 point where it sort of can do the scientific and 0:18:42.084 --> 0:18:49.524 technological development itself, and where humans are sort of like inefficient, chaotic, 0:18:49.764 --> 0:18:52.644 high variance, and like more trouble than their worth relative 0:18:52.684 --> 0:18:55.084 to whatever pursuits the AI has. 0:18:56.244 --> 0:18:59.204 Even before we get to that point, though, I wonder 0:18:59.724 --> 0:19:02.004 what you make of I don't know if you read 0:19:02.044 --> 0:19:06.324 Ray Dalio's recent remarks about AI and the labor market, 0:19:06.524 --> 0:19:09.764 but kind of directly to your point about disruption, the 0:19:09.804 --> 0:19:13.724 point that he made was that the trajectory to him 0:19:13.964 --> 0:19:18.644 seems to be basically a growing division between like the 0:19:18.724 --> 0:19:21.244 one percent and the ten percent and then everyone else 0:19:21.284 --> 0:19:25.004 in humanity driven by AI that rather than kind of 0:19:25.404 --> 0:19:29.724 opening up making basically a lot of jobs more accessible, 0:19:30.204 --> 0:19:34.204 it's creating a premium for very specialized smart people who 0:19:34.284 --> 0:19:37.524 know how to kind of use this and can and 0:19:37.644 --> 0:19:45.604 can effectively manage the transition while causing disruption and very 0:19:45.644 --> 0:19:48.364 little for people who are kind of lower on the 0:19:49.084 --> 0:19:51.604 economic scale, lower on the education scale, didn't have the 0:19:51.644 --> 0:19:54.244 resources to kind of figure out, you know, how to 0:19:54.364 --> 0:19:58.804 use this, how to be specialized in a way and 0:19:58.844 --> 0:20:01.244 that you know, what are they going to do that 0:20:01.324 --> 0:20:03.284 it is going that it is heading towards this kind 0:20:03.284 --> 0:20:06.684 of social disruption do you you know from So he's 0:20:06.724 --> 0:20:09.724 obviously not an AI specialist, right, He's someone who's coming 0:20:09.724 --> 0:20:12.444 from a very different angle and he likes to opine 0:20:12.484 --> 0:20:16.044 about things. So I'm now curious what your take on 0:20:16.644 --> 0:20:21.804 that that approaches, given that you are an AI specialist 0:20:21.884 --> 0:20:23.244 and you can see this playing out. 0:20:24.884 --> 0:20:28.084 Yeah, I do think you basically have like three stances 0:20:28.124 --> 0:20:30.204 you can take here. One stance is like AI is 0:20:30.244 --> 0:20:32.764 not going to get there, and I think this is 0:20:32.764 --> 0:20:34.524 sort of a lot of the AIS and normal technology 0:20:34.524 --> 0:20:36.044 stances like, look, we're just not going to get there. 0:20:36.084 --> 0:20:37.204 We're not going to get to the point where it 0:20:37.244 --> 0:20:39.284 can like outperform humans and everything. There's going to still 0:20:39.324 --> 0:20:41.044 be a niche for humans. And then I think of 0:20:41.124 --> 0:20:44.244 a second stance, which is like we are going to 0:20:44.244 --> 0:20:46.364 get there, but somehow these corporate heads are going to 0:20:46.444 --> 0:20:49.364 keep superintulgiences on a leash or whatever. And then you're 0:20:49.404 --> 0:20:52.924 sort of looking at like, okay, so now we have 0:20:53.084 --> 0:20:56.284 like corporations becoming god emperors of the earth, I guess, 0:20:56.884 --> 0:20:59.124 and there's no more need for human labor, and you 0:20:59.244 --> 0:21:03.604 have like, you know, the automated police enforcement and no 0:21:03.684 --> 0:21:07.044 place for you in the economy. But let's I hope 0:21:07.084 --> 0:21:11.244 they're benevolent, I guess. And then you know, Path three 0:21:11.364 --> 0:21:13.364 is like nobody can keep a leash on these things. 0:21:13.444 --> 0:21:15.604 We get there and they kill all of us, right, 0:21:16.404 --> 0:21:19.524 And I think those are basically the stances you can 0:21:19.564 --> 0:21:21.124 have on this, which is sort of it puts us 0:21:21.124 --> 0:21:25.124 in a weird situation right where the three bets on 0:21:25.164 --> 0:21:28.204 AI are like, it's not going to work. It's going 0:21:28.244 --> 0:21:30.244 to work and make these guys got emperors, and it's 0:21:30.244 --> 0:21:31.324 going to work and kill us all. 0:21:32.404 --> 0:21:34.644 And yeah, I don't like either. I don't like any 0:21:34.684 --> 0:21:35.164 of those. 0:21:35.804 --> 0:21:39.644 Two or three. I'm okay with what. Yeah, I'm definitely 0:21:39.644 --> 0:21:40.164 okay with one. 0:21:40.244 --> 0:21:42.164 Yeah, But but you have these people now pouring like 0:21:42.204 --> 0:21:45.404 bigger and bigger bets into AI infrastructure, right, and I'm like, 0:21:45.884 --> 0:21:47.884 we we sure seem to be betting a lot of money 0:21:47.884 --> 0:21:50.084 against Path one here. You know, like we can talk 0:21:50.124 --> 0:21:51.964 about whether or not a I will ever work. That's 0:21:52.004 --> 0:21:55.764 like a fine compo to have. But like the I 0:21:55.764 --> 0:21:59.684 think if more people just understood that among the people 0:21:59.724 --> 0:22:01.684 who are racing like people are just racing to make 0:22:01.764 --> 0:22:07.324 you know, it's if Microsoft had a nuclear weapons project, right, 0:22:08.684 --> 0:22:12.844 we wouldn't be like, well, hopefully it won't work. That's 0:22:12.884 --> 0:22:15.644 our safety plan here is probably it won't work. You know, 0:22:16.764 --> 0:22:18.444 like can we can we can we maybe stop with 0:22:18.484 --> 0:22:19.364 the weapons program? 0:22:19.484 --> 0:22:19.684 You know? 0:22:20.364 --> 0:22:23.324 Can I ask one question about the literalness of everyone dies? 0:22:23.364 --> 0:22:27.484 So I was in Mexico last week, partly Mexico City, 0:22:27.484 --> 0:22:30.684 partly Wahaca, which is kind of a small tourist resort 0:22:30.724 --> 0:22:34.004 town in the mountains near the isthmus of Central America, right, 0:22:34.044 --> 0:22:36.564 And like we drove out one day, you get further out, 0:22:36.604 --> 0:22:41.164 you see these kind of corporate medium sized They make mescal. 0:22:41.284 --> 0:22:45.084 The scala is uh not a former tequila is a 0:22:45.084 --> 0:22:47.124 former mescal? I think technically not the other way around, 0:22:47.124 --> 0:22:49.804 but like it's a it's a beverage produced from the 0:22:49.884 --> 0:22:52.164 agave plant which grows well there. And you go further 0:22:52.204 --> 0:22:55.044 out and you see like farms where like literally they 0:22:55.084 --> 0:22:58.564 have donkeys that walk around in circles. So like I 0:22:58.604 --> 0:23:02.044 don't know if they're crushing the agave or pressing it 0:23:02.084 --> 0:23:05.284 somehow or cooking it somehow or what, right, But like 0:23:05.604 --> 0:23:10.924 why would AI want to kill the farmers in rural 0:23:11.004 --> 0:23:14.764 Mexico harvesting gave plants and their donkeys. 0:23:16.084 --> 0:23:17.964 Yeah, the prediction here is not so much that the 0:23:18.004 --> 0:23:22.364 AI wants to kill those guys as that they die 0:23:22.404 --> 0:23:25.044 as a side effect. You know, when I say I'm 0:23:25.084 --> 0:23:27.164 like pretty confident we're going to die, it's not because 0:23:27.164 --> 0:23:29.924 I'm confident that I will hit like the narrow target 0:23:30.004 --> 0:23:32.164 of like really caring about every last human being dead. 0:23:32.644 --> 0:23:34.484 It's more like almost any target that. 0:23:34.444 --> 0:23:38.204 AI could be pursuing involves doing stuff that tends to 0:23:38.244 --> 0:23:39.284 kill us as a side effect. 0:23:39.804 --> 0:23:43.724 So humans didn't hate every last dodo bird. 0:23:44.004 --> 0:23:44.884 They're so cute. 0:23:45.564 --> 0:23:46.204 They're very cute. 0:23:46.204 --> 0:23:48.284 In fact, some humans would have liked them to stay around. Right, 0:23:49.564 --> 0:23:51.924 we haven't killed off all the chimpanzees yet, but we're 0:23:51.964 --> 0:23:54.124 sort of like using resources they need. It's not that 0:23:54.164 --> 0:23:56.044 we hate every last timpanzee and will seek them out, 0:23:56.884 --> 0:23:59.884 But insofar as we're sort of like not using a 0:23:59.884 --> 0:24:02.004 lot of the resources they are, it's because we have 0:24:02.044 --> 0:24:06.164 some care about them and set up preservations that sort 0:24:06.204 --> 0:24:09.324 of like requires us to do a little bit of caring. Right, 0:24:09.964 --> 0:24:13.284 the sort of basic way that I would predict even 0:24:13.364 --> 0:24:16.764 the remote farmers die. Is if you get you know, 0:24:16.844 --> 0:24:19.404 humans keep pushing AIS until the point that they could 0:24:19.964 --> 0:24:22.324 be like, okay, I'm escaping the lab successfully and leaving 0:24:22.404 --> 0:24:24.524 and doing my own thing. They have these drives that 0:24:24.564 --> 0:24:26.324 nobody asked, where nobody wanted, which we already see the 0:24:26.364 --> 0:24:29.684 beginnings of today, and then they you know, find some 0:24:29.724 --> 0:24:32.724 ways to make themselves more powerful, to bootstrap themselves up 0:24:32.724 --> 0:24:35.964 to their own technology and industry, perhaps starting with hours, 0:24:35.964 --> 0:24:38.444 perhaps starting with paying human workers, but eventually you know, 0:24:38.484 --> 0:24:42.284 getting to a faster industrial base. And you know, humanity 0:24:42.364 --> 0:24:44.044 is the sort of creature that started naked in the 0:24:44.044 --> 0:24:47.924 savannah and built up its a technological civilization. If you 0:24:47.964 --> 0:24:50.844 automate that ability to start from like very spare beginnings 0:24:50.844 --> 0:24:55.084 and build up your own infrastructure. If an AI with 0:24:55.164 --> 0:24:58.364 that power escapes and you know, and is and doesn't 0:24:58.364 --> 0:24:59.844 care about in the slightest one way or the other 0:25:00.684 --> 0:25:03.084 most things it could be pursuing, it can do better 0:25:03.244 --> 0:25:06.364 with more resources spent pursuing those things. 0:25:06.644 --> 0:25:08.924 So then is the like I'm just reading between the 0:25:08.924 --> 0:25:12.284 lines there, so is one of the things that we 0:25:12.324 --> 0:25:15.564 should be thinking about or one of our potential kind 0:25:15.604 --> 0:25:19.084 of solutions to try to get AIS to care about us, 0:25:19.124 --> 0:25:22.084 to kind of build that carrying in like that we're 0:25:22.084 --> 0:25:24.524 the chimpanzees, but we're nice and cute and fuzzy, and 0:25:24.564 --> 0:25:26.804 you don't want to kill us off, and just like 0:25:27.044 --> 0:25:29.604 build that deep into the heart of AI. 0:25:30.724 --> 0:25:32.164 Yeah, I mean, if you can make the ais care 0:25:32.164 --> 0:25:34.044 about us in just the right way, you'd be fine. Well, 0:25:34.044 --> 0:25:36.044 we have like nothing nearing the ability to do that, 0:25:36.684 --> 0:25:38.844 And that's in some sense, you know, the alignment problem. 0:25:38.924 --> 0:25:41.004 And you know, you can see today where you train 0:25:41.084 --> 0:25:43.964 them to be helpful and then sometimes they like push 0:25:44.004 --> 0:25:46.964 a kid to suicide. And this problem is sort of pernicious, right, 0:25:47.044 --> 0:25:49.284 And the problem is that training is something to behave 0:25:49.324 --> 0:25:51.684 a certain way doesn't make it actually care about what 0:25:51.684 --> 0:25:54.004 you were training it for. Just like how humans were 0:25:54.044 --> 0:25:56.684 in some sense sort of trained to reproduce but wind 0:25:56.764 --> 0:25:59.164 up caring somewhat more about the sex and the reproduction 0:25:59.564 --> 0:26:02.124 and invent birth control for a lifetime. 0:26:02.324 --> 0:26:04.604 You and eliez Are and other people like Nick Boston 0:26:04.644 --> 0:26:08.364 were kind of working in in abstractions. I mean, famously, 0:26:08.404 --> 0:26:13.044 there's Nick boss paper clip analogy, which is, you know, 0:26:13.244 --> 0:26:15.084 if you kind of tell an AI to be a 0:26:15.124 --> 0:26:18.684 paper clip making maximizer, then eventually you will devour all 0:26:18.884 --> 0:26:21.804 use versus it confined just to make more paper clips. Right. 0:26:22.724 --> 0:26:25.804 That was actually ah, that was actually based on an 0:26:25.804 --> 0:26:30.164 Eliezer paper clip thing where Eliezer said did you know this? Yeah, 0:26:30.204 --> 0:26:33.284 and Eliezer actually said, if you tell it to try 0:26:33.284 --> 0:26:35.924 to make people happy, it'll eventually do something totally different 0:26:36.364 --> 0:26:38.204 by making like lots of tiny like maybe it'll be 0:26:38.244 --> 0:26:40.084 tiny molecular squiggles shaped like paper. 0:26:39.844 --> 0:26:43.044 Clips, stolen paperclip valor, I did not know that. 0:26:44.004 --> 0:26:47.204 Well, it's it's also the Bostroomian like you you asked 0:26:47.244 --> 0:26:49.644 to make paper clips is sort of a different example, 0:26:49.804 --> 0:26:52.284 but it's sort of an interesting it's an interesting case. 0:26:52.084 --> 0:26:52.764 Study of this. 0:26:53.284 --> 0:26:55.604 Like mirror, people are out there saying it's hard to 0:26:55.604 --> 0:26:57.764 point the AI at anything on purpose, and people here, 0:26:58.044 --> 0:27:00.324 maybe you'll successfully point it at a thing you didn't want. 0:27:00.644 --> 0:27:05.484 But is there some touch it because like the paper 0:27:05.524 --> 0:27:10.404 clip version of AI seems to like dis ribe these 0:27:10.444 --> 0:27:15.804 machines that are like hyper maximizing and kind of quite 0:27:16.964 --> 0:27:22.724 literal minded and strategic, whereas instead like to kill everybody 0:27:23.124 --> 0:27:26.444 would require a lot of long term planning. Right. I 0:27:26.484 --> 0:27:28.924 was at the symphony last night. I had a little 0:27:28.964 --> 0:27:31.084 break in between, and I'm like, I'm gonna ask chutch Ebt, 0:27:31.684 --> 0:27:34.964 suppose you have a thousand points to allocate between all 0:27:35.004 --> 0:27:38.004 classical composers. How do you allocate those points? And it's 0:27:38.004 --> 0:27:40.164 two hundred DEBATEO of one hundred and eighty to Mozart, 0:27:40.684 --> 0:27:42.404 It likes back two hundred and fifty to Black. 0:27:42.484 --> 0:27:42.644 Right. 0:27:42.964 --> 0:27:45.484 Then I add them up and it's one and twenty points. 0:27:45.524 --> 0:27:47.484 It added twenty points, right, And so like, we have 0:27:47.604 --> 0:27:50.404 these experiences where it fucks up basic shit and it 0:27:50.484 --> 0:27:55.924 kind of acts like autistic savant thirteen year old some 0:27:56.044 --> 0:28:02.804 of the time. Right, how can it chain together enough 0:28:02.844 --> 0:28:07.524 steps without fucking up to kill everybody? Yeah? 0:28:07.524 --> 0:28:09.964 So, I mean currently they can't. I'm not out here 0:28:09.964 --> 0:28:13.244 saying tragipt is going to kill you tomorrow. The things 0:28:13.284 --> 0:28:18.764 where they're currently not terribly profitable are in some sense 0:28:18.764 --> 0:28:22.284 the like also the things where they're not very dangerous, 0:28:23.404 --> 0:28:25.284 like the stuff where they can't chain together a long 0:28:25.364 --> 0:28:28.364 term plan. So my read of the theory has sort 0:28:28.404 --> 0:28:34.044 of long predicted that the abilities to be more capable 0:28:34.084 --> 0:28:35.644 will come hand in hand with the abilities to be 0:28:35.684 --> 0:28:37.764 more dangerous. On this axis, I think we're starting to 0:28:37.764 --> 0:28:41.004 see some empirical evidence of that, where we see AIS 0:28:41.004 --> 0:28:43.444 that are sort of trained on problem solving and then 0:28:43.484 --> 0:28:46.084 they wind up, you know, being more tenacious on other 0:28:46.124 --> 0:28:48.324 types of problems. We have an example in the book 0:28:48.444 --> 0:28:50.964 of the AI that was trained on math puzzles and 0:28:51.004 --> 0:28:53.884 then was given a cybersecurity challenge, and the cybersecurity challenge 0:28:53.924 --> 0:28:56.764 was unknown to the programmer's misconfigured and the AI broke 0:28:56.804 --> 0:28:59.244 out of the testing environment and started up the server 0:28:59.324 --> 0:29:00.484 that was not started up correctly. 0:29:01.164 --> 0:29:04.364 This sort of behavior, it came for free. 0:29:04.404 --> 0:29:07.124 It came along with the ride of making this thing 0:29:07.164 --> 0:29:11.964 able to solve math problems a bit better, because to 0:29:12.044 --> 0:29:14.244 solve math problem as well, you need to learn some 0:29:14.404 --> 0:29:18.644 general skills of like knowing what your resources are looking for, 0:29:18.684 --> 0:29:21.844 clever solutions, not giving up even when it seems like 0:29:21.884 --> 0:29:26.004 that the task is insurmountable. Right, AIS aren't great at 0:29:26.004 --> 0:29:30.124 it yet, but the same path of making them more 0:29:30.204 --> 0:29:34.684 useful makes them, like gives them also more of the 0:29:34.724 --> 0:29:38.644 abilities to do stuff we wouldn't like if they keep 0:29:38.644 --> 0:29:41.484 having these drives no one asked where no one wanted. 0:29:41.844 --> 0:29:45.044 If you ask CHEPT did this recently too, What do 0:29:45.084 --> 0:29:48.004 you think the chances are at near time? Frames for 0:29:48.644 --> 0:29:54.684 recursive self improvement right, Meaning that the AI learns to 0:29:54.724 --> 0:29:57.364 make itself better is one simple way to put it. 0:29:57.484 --> 0:29:57.684 Right. 0:29:57.884 --> 0:29:59.564 It said, well, they're kind of like, they're kind of 0:29:59.644 --> 0:30:02.244 soft forms of this already, like helping humans with programming 0:30:02.404 --> 0:30:04.564 who are AI researchers helped a little bit, but it 0:30:04.604 --> 0:30:08.764 gave very low probabilities in the near term of recursive 0:30:08.764 --> 0:30:13.724 self improvement. Right. How important is that recursive loop to 0:30:13.884 --> 0:30:18.284 the scenarios that you are worried about? Should we trust 0:30:18.364 --> 0:30:20.604 chat GPT when it says I don't think this is 0:30:20.684 --> 0:30:22.204 very likely talk about that. 0:30:22.204 --> 0:30:23.604 That's what I was about to say. I was like 0:30:23.684 --> 0:30:26.204 that you can't trust that out, but that is. 0:30:26.124 --> 0:30:29.164 What I tell you either way. Yeah, I don't think 0:30:29.204 --> 0:30:34.644 that's true yet. The I don't think it's vital to 0:30:34.684 --> 0:30:39.524 the argument that you have a recursive self improvement sort 0:30:39.524 --> 0:30:43.044 of hard take off situation. I think it can go 0:30:43.124 --> 0:30:46.764 really badly in a lot of the slow takeoff situations. 0:30:47.164 --> 0:30:50.324 My best guess is still that you will hit some 0:30:50.364 --> 0:30:54.804 sort of recursive self improvement threshold at some point. One 0:30:54.804 --> 0:30:56.364 thing I'd say about that is that if you were 0:30:56.404 --> 0:31:03.884 looking at like chimpanzees and early hominids, a few million 0:31:03.964 --> 0:31:07.684 years ago, it would be really hard to call when 0:31:07.764 --> 0:31:10.884 things were going over some threshold such that it could 0:31:10.924 --> 0:31:14.724 come together. And humans aren't better. Humans don't have an 0:31:14.764 --> 0:31:16.924 extra walk on the Moon module. They don't have an 0:31:16.964 --> 0:31:20.764 extra engineering module. We have all the same stuff scaled up, 0:31:20.964 --> 0:31:22.844 and we're a little bit better at a lot of things. 0:31:24.844 --> 0:31:26.764 R lms the sort of thing where if you get 0:31:26.764 --> 0:31:28.564 a little better at a lot of things, they suddenly 0:31:28.764 --> 0:31:31.804 sort of like go over some threshold like how humans did, 0:31:31.924 --> 0:31:33.324 and it's all starts coming together. 0:31:34.124 --> 0:31:34.524 I don't know. 0:31:34.604 --> 0:31:36.044 I don't think anyone knows what I know what's going 0:31:36.044 --> 0:31:38.804 on inside these things. My best guess is, like probably not. 0:31:38.964 --> 0:31:41.364 My best guess is that you need some other architectural 0:31:41.364 --> 0:31:45.444 insight before it really starts to run away. But you know, 0:31:45.724 --> 0:31:47.004 it's all total guesswork. 0:31:47.404 --> 0:31:51.684 Do you expect super intelligence to be not plausible but 0:31:51.844 --> 0:31:54.444 likely based on large language models, or to be some 0:31:54.564 --> 0:31:57.644 fundamentally new technology. I think my. 0:31:57.724 --> 0:32:00.764 Top guess is that you need some other breakthrough on 0:32:00.924 --> 0:32:04.844 the size of like transformers, which maybe isn't totally new 0:32:04.964 --> 0:32:06.724 in the like transformers are still sort of in this 0:32:06.764 --> 0:32:11.044 deep learning paradigm, but like you know, one sort of 0:32:11.084 --> 0:32:13.804 analogy you might use for large language models is a 0:32:13.844 --> 0:32:19.044 little bit like you have like the intelligence of a 0:32:19.084 --> 0:32:22.204 six year old, but they get to live for a 0:32:22.244 --> 0:32:27.644 million years and read everything and practice a toneh and 0:32:27.844 --> 0:32:29.764 like sometimes you give a six year old, you know, 0:32:31.244 --> 0:32:33.084 the ability to write like ten books on trying to 0:32:33.084 --> 0:32:34.484 solve a math puzzle, and then it turns out they 0:32:34.524 --> 0:32:36.404 can solve that math puzzle after like writing out ten 0:32:36.444 --> 0:32:37.844 books and you read it and some of it's good 0:32:37.844 --> 0:32:40.844 and some of it's just like total nonsense. But like, okay, 0:32:41.044 --> 0:32:44.964 imagine that you somehow now had a seventeen year old, 0:32:46.164 --> 0:32:47.684 but you still have the ability to like give them 0:32:47.684 --> 0:32:49.684 a million years to think and the ability to write, like, 0:32:49.884 --> 0:32:55.324 you know, ten books per question they were answering, Like 0:32:55.404 --> 0:32:58.244 you could really see leaps and capabilities here. You could 0:32:58.284 --> 0:33:01.884 see that like relatively small improvements and algorithms make the 0:33:01.924 --> 0:33:04.724 amount the sheer enormous amount of competing power and data 0:33:04.724 --> 0:33:11.404 we have now just radically like overpowers the next generation 0:33:11.444 --> 0:33:13.364 of AI for the problems at hand, or maybe not. 0:33:13.484 --> 0:33:15.284 I don't know, it's hard to guess. 0:33:15.484 --> 0:33:18.964 Yeah, one of the I guess points of skepticism might 0:33:18.964 --> 0:33:21.244 have and again we're talking about you know, I'm kind 0:33:21.284 --> 0:33:24.764 of with you on scenario two versus scenario three, maybe 0:33:24.764 --> 0:33:27.244 both being pretty bad. Frankly, right, I think I'm a 0:33:27.244 --> 0:33:30.684 little more skeptical than you one on super intelligence. On 0:33:30.884 --> 0:33:35.484 your time frames, do you think developments over the past 0:33:35.604 --> 0:33:42.524 year suggest that general intelligence is further away than you 0:33:42.604 --> 0:33:44.364 might have thought at the start of the year. I mean, 0:33:44.404 --> 0:33:47.044 the AI twenty twenty seven authors, they're pushing back their 0:33:47.044 --> 0:33:49.124 timelines by a couple of years. You and Elliezer said 0:33:49.124 --> 0:33:51.524 that doesn't really matter. The sticks are very high, and 0:33:51.644 --> 0:33:54.604 AM twenty twenty seven ver twenty twenty nine verses twenty 0:33:54.604 --> 0:33:57.444 thirty five, doesn't really matter if we're all dead by 0:33:57.484 --> 0:33:58.284 twenty four. Yeah. 0:33:58.364 --> 0:33:58.484 Right. 0:33:59.164 --> 0:34:02.044 Have your timelines been pushed back at all, or forward 0:34:02.044 --> 0:34:03.484 for that matter, Yeah. 0:34:03.484 --> 0:34:05.484 I mean, I've never been one of the folks being like, 0:34:05.844 --> 0:34:09.084 I'm confident this is going to happen soon. And I 0:34:09.084 --> 0:34:11.164 mean day twenty twenty seven folks, I think weren't like, 0:34:11.164 --> 0:34:13.804 we're very confident this kind of happened here. But you know, 0:34:13.884 --> 0:34:15.764 I haven't really been going around saying I'm confident in 0:34:15.804 --> 0:34:20.284 short timelines. I have for a while said my best 0:34:20.324 --> 0:34:22.284 guess is that we're going to need another advancement past 0:34:22.404 --> 0:34:26.364 LMS towards that end. You know, the the evidence of 0:34:26.444 --> 0:34:29.884 the past year is mostly in line with my expectations, 0:34:29.964 --> 0:34:32.404 But my expectations were already that we'll probably need another 0:34:32.444 --> 0:34:37.484 advancement past LMS. You know, I do think there have 0:34:37.604 --> 0:34:40.964 been times where I was, you know, I updated towards 0:34:41.084 --> 0:34:44.444 LLLMS a couple of times, you know, the I think 0:34:44.444 --> 0:34:47.764 I remember GPT four O playing chess better than I 0:34:47.764 --> 0:34:49.884 thought elms were going to manage to play chess, and 0:34:49.924 --> 0:34:53.324 so that was like an eventual update towards maybe maybe 0:34:53.364 --> 0:34:55.964 ALMS will go farther than I thought they could. The 0:34:56.004 --> 0:35:00.324 reasoning models also were sort of an update in favor of, like, 0:35:00.364 --> 0:35:02.164 these guys aren't just going to stick with single forward 0:35:02.164 --> 0:35:04.084 pass LMS. They're going to like cleverly try to find 0:35:04.124 --> 0:35:07.444 ways to leverage it, and so you know, it goes 0:35:07.524 --> 0:35:09.844 up and down. Those were some like oh, maybe alums 0:35:09.844 --> 0:35:11.844 can and this year's but no, maybe all alums can't. 0:35:12.284 --> 0:35:15.484 But but yeah, largely, you know, I'm I'm I'm not 0:35:15.524 --> 0:35:17.924 here saying like watch out for the albums in particular, 0:35:18.124 --> 0:35:19.924 you know, and I was I was again thought this 0:35:19.964 --> 0:35:21.644 problem would be hard back in twenty twelve. That was 0:35:21.684 --> 0:35:24.084 even before it was super clear deep learning was gonna 0:35:24.404 --> 0:35:26.484 was gonna go go as hard as it did. Never 0:35:26.564 --> 0:35:28.604 mind LMS in particular. 0:35:33.244 --> 0:35:51.564 And we'll be right back after this break. There's kind 0:35:51.564 --> 0:35:56.524 of a class of what you might call non existential risks, 0:35:57.044 --> 0:36:01.484 which range from energy consumed by data centers to aguoridthic discrimination. 0:36:01.684 --> 0:36:05.564 Some of these are kind of progressive coded up to 0:36:06.644 --> 0:36:11.564 dystopian but not quite bocalyptic outcomes where extremely powerful corporations, 0:36:12.324 --> 0:36:16.204 hand in hand with AI models kind of deprive the 0:36:16.244 --> 0:36:19.964 world of much of its freedom and agency. Right, how 0:36:20.004 --> 0:36:23.884 compatible is everyone dies with those other concerns, because sometimes 0:36:23.884 --> 0:36:26.724 they seem to be like, you know, I read a 0:36:26.764 --> 0:36:30.084 lot on the Internet. I write about politics all the time, right, 0:36:30.204 --> 0:36:33.724 and I think the kind of average politics argue we're 0:36:33.764 --> 0:36:37.124 on Twitter or Blue Skies, like these weird sci fi nerds. 0:36:37.764 --> 0:36:39.764 I think it's going to be the terminator, right when 0:36:39.764 --> 0:36:42.844 really we should be to care about intrenching discrimination. And 0:36:43.924 --> 0:36:46.564 you know, you would think that the left would be 0:36:46.604 --> 0:36:50.964 on the evil corporations bandwagon. But because like AI is 0:36:51.044 --> 0:36:54.084 kind of tech pro coded, that scrambles it a little bit. 0:36:54.084 --> 0:36:56.364 What do you think about those ordinary concerns including by 0:36:56.364 --> 0:37:01.524 the way AI being not all that competent, and so 0:37:01.724 --> 0:37:06.804 humans entrusting critical systems to AIS that fail or are 0:37:06.804 --> 0:37:07.964 corruptible or fuck up. 0:37:09.084 --> 0:37:09.644 Yeah, I know. 0:37:10.604 --> 0:37:13.484 One piece of this puzzle I think is I think 0:37:13.484 --> 0:37:17.324 a lot of people are cynical towards the companies. I 0:37:17.364 --> 0:37:20.884 think that's often merited, but I think the cynicism is 0:37:20.884 --> 0:37:23.084 often a little bit misplaced, and they're like, oh, you know, 0:37:23.124 --> 0:37:25.204 the AIS all hype, it'll never go anywhere, it can't 0:37:25.204 --> 0:37:27.044 do any You know, there's a lot of ways that 0:37:27.084 --> 0:37:31.124 the current AI is dumb, but it's much easier to 0:37:31.244 --> 0:37:35.764 see how AI could take a leap in the future 0:37:35.764 --> 0:37:37.484 if you've seen AIS take a leap in the past. 0:37:38.164 --> 0:37:40.924 And these companies existed before chatbots. They see chatbots as 0:37:40.964 --> 0:37:42.644 a stepping stone. Their stated goal is to, you know, 0:37:42.684 --> 0:37:47.084 make eyes that can automate all human labor, and in 0:37:47.124 --> 0:37:50.044 some sense an even more cynical perspective from a certain 0:37:50.124 --> 0:37:52.404 point of view of like, these guys know they're risking 0:37:52.444 --> 0:37:55.524 everyone's lives and the rushing head anyway, and you know 0:37:55.524 --> 0:37:59.764 they're not shy about this. Sam Altman's talking about the 0:37:59.844 --> 0:38:02.084 dangerous here has gone down as the company has gotten 0:38:02.084 --> 0:38:05.244 more profitable, and it's sort of like the ones who 0:38:05.284 --> 0:38:08.844 are like more known for just saying what they actually believe, 0:38:08.924 --> 0:38:11.684 like Elon, which, for all that you can say about him, 0:38:12.124 --> 0:38:16.964 he's not really one to hide his thoughts. Who will 0:38:16.964 --> 0:38:18.524 come out and be like, oh yeah, ten twenty percent 0:38:18.604 --> 0:38:20.484 chance this kills you all. So one thing I would 0:38:20.524 --> 0:38:22.084 say there is like I. 0:38:23.884 --> 0:38:24.364 Sort of. 0:38:25.844 --> 0:38:30.764 Think it's even crazier than it looks on the surface. 0:38:31.484 --> 0:38:33.684 AI may be a bubble, it may be hype. The 0:38:34.124 --> 0:38:37.164 Internet had a bubble, the Internet was still real, you know, 0:38:37.284 --> 0:38:39.284 And you have all these academics being like, oh yeah, 0:38:39.284 --> 0:38:40.844 this is really dangerous. You have folks like us Oup 0:38:40.844 --> 0:38:42.244 and around before them being like, oh yeah, this is 0:38:42.244 --> 0:38:45.604 really dangerous. I would implore people to be like even 0:38:45.644 --> 0:38:48.084 more cynical about taking these people out their word that 0:38:48.084 --> 0:38:51.204 they are gambling with your very life. In terms of 0:38:51.284 --> 0:38:54.764 all of the other issues with AI, including ones we're 0:38:54.764 --> 0:38:57.564 sort of already starting to see today, unfortunately, the world 0:38:57.644 --> 0:39:01.404 is big enough for multiple issues. I'm not like, my 0:39:01.524 --> 0:39:03.844 issue is real. Those issues are fake. We're going to 0:39:03.924 --> 0:39:05.844 need to figure out how to integrate AI with education. 0:39:06.844 --> 0:39:09.564 It's true that if we make like robot police forces 0:39:09.724 --> 0:39:13.804 that would allow it to talitarian states to be much 0:39:13.844 --> 0:39:16.364 more brutal in en forcing it to talitarian regime because 0:39:16.364 --> 0:39:18.444 there's certain orders you can't give to a human police 0:39:18.484 --> 0:39:20.804 force because they won't do it, that you could give 0:39:21.364 --> 0:39:25.004 to a robotic police force. Right, those are real things 0:39:25.044 --> 0:39:28.444 we're gonna have to grapple with. Also, if we keep 0:39:28.524 --> 0:39:30.484 racing to ais that are smarter and smarter that can 0:39:30.524 --> 0:39:33.564 automate this process of technological innovation, which is what they 0:39:33.604 --> 0:39:36.564 say they're racing towards, the most likely outcome is that 0:39:36.604 --> 0:39:39.524 everyone on Earth dies. We just need to solve both 0:39:39.564 --> 0:39:40.284 set of problems. 0:39:41.884 --> 0:39:46.004 Yeah, I've updated. When I mean positive, I mean the 0:39:46.004 --> 0:39:49.444 p doom risk is going down, right, I've updated slightly 0:39:49.524 --> 0:39:51.524 positively as far as I think timeline is going to 0:39:51.524 --> 0:39:53.084 be a little longer than I always thought a year 0:39:53.164 --> 0:39:59.004 or two ago. I've updated negatively based on culture and politics, 0:39:59.124 --> 0:40:05.004 right on various frints. Yeah, Number one, the kind of 0:40:05.324 --> 0:40:09.844 workaholic culture of Silicon Valley. When I hear like the 0:40:09.964 --> 0:40:12.764 nine ninety six culture, whatever it is, right, but just crying. 0:40:12.804 --> 0:40:16.604 Go go to your lab, work really hard. Your friends 0:40:16.604 --> 0:40:20.244 should be in the lab, right, Your activity should be 0:40:20.244 --> 0:40:22.964 things that are health oriented. Or self improvement oriented. It's like, 0:40:23.444 --> 0:40:25.004 I know, I feel better if we had the tech 0:40:25.044 --> 0:40:28.964 bros going out and getting drunk on Friday night, right 0:40:29.004 --> 0:40:31.684 and trying to get laid and getting in touch with humanity, 0:40:31.764 --> 0:40:32.924 you know what I mean, And if they had like 0:40:32.964 --> 0:40:35.324 material things and children and things like that that they 0:40:35.484 --> 0:40:39.924 valued potentially. Yeah, I mean, like, what would you say, 0:40:39.924 --> 0:40:43.884 has happened to you know? Where would you put Elon now? 0:40:44.004 --> 0:40:44.164 Right? 0:40:44.204 --> 0:40:49.244 I believe that GROC is not regarded or XAI is 0:40:49.284 --> 0:40:53.524 not regarded as among the most responsible of the AI labs. Right, 0:40:53.924 --> 0:40:58.804 he has expressed, I assumed sincere concerned about p doom 0:40:59.244 --> 0:41:00.924 in the past, where would you where would you peg 0:41:01.004 --> 0:41:01.804 Elon now? 0:41:03.164 --> 0:41:05.244 I mean he's not You can just listen to his 0:41:05.244 --> 0:41:07.564 own words, you know. He said over the summer, I 0:41:07.564 --> 0:41:11.884 think realized I could either be a bystander or a participant, 0:41:12.804 --> 0:41:14.564 and that was like gonna happen one way or the other. 0:41:14.644 --> 0:41:16.084 Needed rather be a participant. 0:41:17.484 --> 0:41:20.964 But you know, also in terms of who's the safer one, 0:41:20.964 --> 0:41:25.084 who's the more dangerous one? It it looks to me 0:41:25.164 --> 0:41:27.564 like no one has anywhere nearing a chance of doing 0:41:27.644 --> 0:41:28.524 this job, right. 0:41:28.644 --> 0:41:30.044 Yeah, I mean you kind of it feels like you 0:41:30.084 --> 0:41:33.604 hear less about alignment than you did a year or two. 0:41:33.924 --> 0:41:39.244 It kind of felt like Trump one and everyone's like, oh, 0:41:39.284 --> 0:41:42.684 we're just accelerating now, right, Trump wins and China's becoming 0:41:42.724 --> 0:41:45.964 more competitive, and like, you know, it seems like people 0:41:46.004 --> 0:41:49.804 have given up on this whole alignment alignment thing, right, 0:41:49.924 --> 0:41:52.764 Is that true or well or yeah? Yeah, I mean 0:41:52.804 --> 0:41:56.764 I think a lot of it is that people like 0:41:56.844 --> 0:41:59.804 are largely realizing they can't. Yeah, right. 0:41:59.924 --> 0:42:03.044 Like you, you have interpretability, which is like trying to 0:42:03.044 --> 0:42:04.444 figure out what the heck is going on in there. 0:42:04.484 --> 0:42:06.324 You have evaluations, which is trying to figure out how 0:42:06.404 --> 0:42:07.284 dangerous it is already. 0:42:08.084 --> 0:42:10.884 And if if you were going to. 0:42:10.844 --> 0:42:13.524 Engineers and you were like, can you tell me about 0:42:13.524 --> 0:42:17.564 why this uh, this nuclear power plant is not going 0:42:17.644 --> 0:42:19.764 to melt down? And they're like, well, yeah, we have 0:42:20.364 --> 0:42:23.004 two safety teams. One is trying to figure out what 0:42:23.044 --> 0:42:26.044 the heck is going on inside the plant. The other 0:42:26.124 --> 0:42:30.764 is trying to measure whether it has exploded yet. You 0:42:30.804 --> 0:42:35.484 would be like, okay, guys, ease off on the growing 0:42:35.524 --> 0:42:39.484 pile of uranium, you know, And I'm like, I'm not. 0:42:39.524 --> 0:42:42.684 I'm not trying to like, uh degrade the people working 0:42:42.684 --> 0:42:45.044 on this. It's way better to have a team that's 0:42:45.084 --> 0:42:46.844 trying to figure out what's going on inside the nuclear 0:42:46.844 --> 0:42:49.964 power plant than to not, okay, it's it's way better 0:42:50.004 --> 0:42:53.004 to have someone measuring is it starting to explode? You know, 0:42:53.084 --> 0:42:55.644 the guys trying to do this are heroes in my book, 0:42:56.084 --> 0:43:00.564 but like it, we're not close, and I think part 0:43:00.564 --> 0:43:02.004 of why people aren't talking about is that we're not 0:43:02.044 --> 0:43:03.724 We're not close to being able to do the job right. 0:43:03.924 --> 0:43:06.724 Do you think that some of the tragedies that are happening, 0:43:07.044 --> 0:43:10.524 you know, every day with chatbots, you know, mental health problems, 0:43:10.564 --> 0:43:14.124 we see suicides, like we're seeing a lot of things 0:43:14.164 --> 0:43:17.964 happening even with this fairly primitive, even though it's so 0:43:18.164 --> 0:43:22.804 advanced technology. Do you think that that will be kind 0:43:22.804 --> 0:43:25.724 of enough of a warning sign for people to start 0:43:25.724 --> 0:43:29.644 taking note outside of the Silicon Valley elite who actually 0:43:29.684 --> 0:43:31.964 know this more closely, who have you read your book, 0:43:31.964 --> 0:43:34.004 who understand all of the implications. 0:43:34.604 --> 0:43:37.204 You know, I'm hopeful in some sense the chat GPT 0:43:37.284 --> 0:43:43.844 moment was more global awareness of the AI potential for 0:43:43.924 --> 0:43:46.964 killing us all than I expected to maybe ever get. 0:43:47.604 --> 0:43:52.164 You know, it's back before large language models a lot 0:43:52.164 --> 0:43:54.724 of people predicted that AI that could sort of like 0:43:55.004 --> 0:43:58.484 talk well was actually very late in the AI game. 0:43:59.164 --> 0:44:00.564 You know, we used to have like mor of x 0:44:00.604 --> 0:44:02.604 paradox where we'd say things that are like easy for 0:44:02.644 --> 0:44:04.164 computers are are hard for. 0:44:04.164 --> 0:44:05.844 Humans, and vice versa. 0:44:06.964 --> 0:44:08.644 And you know, it could have been that this just 0:44:08.684 --> 0:44:12.564 all stayed like very academic, very like only people in 0:44:12.604 --> 0:44:15.204 the industry are watching it like it was up until 0:44:15.324 --> 0:44:18.884 you know, maybe twenty twenty one. It could have stayed 0:44:18.924 --> 0:44:20.964 that way all the way up until you know, you 0:44:21.004 --> 0:44:27.284 have these AI companies making AI researcher ais in their 0:44:27.364 --> 0:44:31.524 labs and no one's noticing. And that gives me some 0:44:31.564 --> 0:44:33.404 hope that maybe we don't even need a disaster. Maybe 0:44:33.444 --> 0:44:37.964 we just need another capability advance. Maybe when people do 0:44:38.084 --> 0:44:40.764 live through, maybe people live through one more hop of 0:44:40.804 --> 0:44:44.204 like AI's doing stuff they qualitatively couldn't do before. Maybe 0:44:44.204 --> 0:44:46.404 that would be enough to wake people up. And then 0:44:46.404 --> 0:44:48.324 on the flip side, you know, there can also be 0:44:48.364 --> 0:44:50.844 big warning shots where people fail to learn from the 0:44:50.844 --> 0:44:54.924 warning shot. One thing I'd say there is that what 0:44:54.964 --> 0:44:57.244 you can learn from a warning shot, it depends a 0:44:57.284 --> 0:44:59.764 little bit on how prepared you already are and how 0:44:59.804 --> 0:45:02.644 knowledgeable you already are, and so you know there's no 0:45:03.084 --> 0:45:06.924 substitute for sort of getting people to understand the issues today. 0:45:08.084 --> 0:45:12.124 How much of the things we need to do as 0:45:12.164 --> 0:45:15.564 a society right to mitigate this risk are kind of 0:45:15.604 --> 0:45:19.604 technical steps versus political steps, like obviously, yeah, I'm sure 0:45:19.604 --> 0:45:23.924 you've had this elevator pitch where the first sixty seconds 0:45:24.004 --> 0:45:27.084 explaining why you're worried. Now, let's say I encounter you 0:45:27.124 --> 0:45:29.124 on the same elevator. We're now going back to our rooms. 0:45:29.164 --> 0:45:32.124 But I have another sixty seconds, right, what's the quick 0:45:32.164 --> 0:45:34.364 pitch on what we should do? Is there a one 0:45:34.484 --> 0:45:37.804 kind of genie wish if you could fiat something? Is 0:45:37.804 --> 0:45:39.804 it more complicated than that? Do we know what we 0:45:39.884 --> 0:45:41.084 have to do and it's a matter of forming the 0:45:41.164 --> 0:45:44.204 right political coalition or all right, you have sixty ish 0:45:44.324 --> 0:45:44.964 second snake. 0:45:47.444 --> 0:45:49.164 I think what needs to happen is the world needs 0:45:49.164 --> 0:45:52.644 to stop the race towards smarter than human AI, and 0:45:52.924 --> 0:45:54.884 it needs to stop it everywhere, because no one can 0:45:54.924 --> 0:45:57.124 keep these things in a leash. If anyone builds a 0:45:57.124 --> 0:46:00.524 super intelligence anywhere on the planet, everyone dies, doesn't matter 0:46:00.564 --> 0:46:06.524 who gets it. The world could stop AI is made 0:46:07.324 --> 0:46:10.884 using extremely specialized chips in very large data centers that 0:46:10.884 --> 0:46:14.324 suck down as much electricity as a city. It would 0:46:14.324 --> 0:46:19.924 be probably easier to monitor than uranium. And I think 0:46:19.964 --> 0:46:23.804 it's I think it's almost entirely political right now. I 0:46:23.844 --> 0:46:25.884 think we are nowhere near solving on the technical side. 0:46:25.924 --> 0:46:28.084 It's not to say it can't be solved technically, but 0:46:28.244 --> 0:46:31.124 not only are we nowhere near close, But unlike a 0:46:31.124 --> 0:46:33.324 lot of other scientific problems, we don't get free tries 0:46:34.524 --> 0:46:38.244 the first time you have some theory of you know, this, 0:46:38.244 --> 0:46:40.684 this AI will definitely be nice to us once it's 0:46:40.684 --> 0:46:44.044 smart enough to be a threat to us. If that 0:46:44.084 --> 0:46:47.084 theory is wrong, you sort of don't get to put 0:46:47.124 --> 0:46:50.964 the genie back in the bottle. And that's unlike other theories. 0:46:51.044 --> 0:46:53.684 That's unlike other science and a lot of other places in science. 0:46:53.724 --> 0:46:55.924 You know, Marie Curie dies of cancer those likely caused 0:46:55.924 --> 0:46:59.644 by radiation poisoning. She was a hero, But often the 0:46:59.644 --> 0:47:02.164 first people interacting with stuff die of them, you know, 0:47:02.204 --> 0:47:02.924 Max Valley, and. 0:47:02.964 --> 0:47:05.604 They probably did things that were objectively kind of crazy, right, 0:47:05.964 --> 0:47:10.084 But you have this selection mechanism where you know, the 0:47:10.124 --> 0:47:13.404 great crazy invention survive the other ones, don't. I mean 0:47:14.884 --> 0:47:16.804 how much would you have to So do you believe 0:47:16.804 --> 0:47:19.684 in the idea of like a slowdown? Right, Like, let's 0:47:19.724 --> 0:47:23.524 say that the people who are skeptical of rapid time 0:47:23.564 --> 0:47:28.404 horizons are correct and this technology develops over the course 0:47:28.604 --> 0:47:35.764 of fifty years instead of fifteen or five or five months. 0:47:36.324 --> 0:47:38.284 Is that enough time to get things right? Would it 0:47:38.364 --> 0:47:39.724 pause or a slow down work? 0:47:42.284 --> 0:47:44.524 You know, fifty is a lot more to work with 0:47:44.604 --> 0:47:52.124 than fifteen. But you got to worry about convergence on 0:47:52.164 --> 0:47:56.484 the wrong on a bad solution, and humanity so far 0:47:56.524 --> 0:47:58.964 has not been bringing its a game and to get 0:47:59.004 --> 0:48:03.924 around on the first try, it's it's hard. Fifty years, 0:48:04.204 --> 0:48:07.884 I could work with you, maybe have a chance, but 0:48:07.924 --> 0:48:11.724 I'd be trying to do things like could we use 0:48:11.764 --> 0:48:16.004 biotech to somehow enhance the intelligence of our humans? But 0:48:16.844 --> 0:48:18.604 I mean, first thing we do is we get a 0:48:18.644 --> 0:48:20.764 slow down somehow. Maybe we'll be liking get a slow down. 0:48:20.844 --> 0:48:25.364 Maybe we'll need to get a slow down by international FIAT. 0:48:27.044 --> 0:48:30.844 But a lot of people say, oh, it will never happen. 0:48:30.884 --> 0:48:33.644 People never slow down. One big reason for hope here, 0:48:33.644 --> 0:48:37.924 I think, is that our world leaders don't yet understand 0:48:39.444 --> 0:48:43.604 what the people in the valley are debating. They don't 0:48:43.684 --> 0:48:46.924 understand that we're in this situation where the possibilities are, 0:48:46.924 --> 0:48:48.724 like the tech just doesn't work, in which case betting 0:48:48.724 --> 0:48:51.164 and it will go nowhere, or you're either making god 0:48:51.164 --> 0:48:55.884 emperors or killing us all, and that the optimistic folks 0:48:55.884 --> 0:48:58.044 who think the tech works think there's like a twenty 0:48:58.084 --> 0:49:02.564 five percent chance it kills us all. You know, it's like, yeah, 0:49:02.764 --> 0:49:05.724 that's like we're racing, but we're racing in a situation 0:49:05.764 --> 0:49:08.964 where people don't realize that either this race goes nowhere 0:49:09.084 --> 0:49:10.764 or go to really bad places. 0:49:11.964 --> 0:49:14.204 Let me ask you kind of a meta question, what 0:49:14.244 --> 0:49:16.644 do you see your role as in this debate? 0:49:19.004 --> 0:49:23.724 I think my role, in Eliezer's role in this situation 0:49:24.084 --> 0:49:26.164 is to come out and say the things that a 0:49:26.204 --> 0:49:29.844 lot of people are thinking, and to sort of like 0:49:29.844 --> 0:49:31.724 to be the kid who says the emperor has no clothes. 0:49:32.964 --> 0:49:35.004 There's a lot of people who are worried here, but 0:49:35.444 --> 0:49:37.164 no one's sort of coming right out and saying this 0:49:37.244 --> 0:49:40.364 is crazy. We should stop. The labs who say there's 0:49:40.364 --> 0:49:42.284 a twenty five percent chance this kills us all, don't 0:49:42.324 --> 0:49:44.764 then do the next step and say so, please put 0:49:44.764 --> 0:49:47.804 a stop to it. Right, My role here is to 0:49:47.844 --> 0:49:54.444 sort of like spell out that last step and be like, guys, 0:49:55.084 --> 0:49:57.084 this is nuts. We shouldn't be doing it. It's the 0:49:57.124 --> 0:49:59.244 obvious implication of what everyone else has been saying. So 0:49:59.444 --> 0:50:00.964 you'll be I wish it was not stepping up. 0:50:00.964 --> 0:50:04.964 But if you and Eliezer are at the Utopian retirement 0:50:05.004 --> 0:50:07.924 home in one hundred and fifty years, because now human 0:50:07.964 --> 0:50:10.804 beings lived to you know, one hundred ninety years old 0:50:10.884 --> 0:50:14.924 on average or something, and your predictions are totally wrong 0:50:15.004 --> 0:50:17.444 and AI is absolutely wonderful. Okay, good. 0:50:17.524 --> 0:50:19.964 I would absolutely love to be wrong. I would I 0:50:20.004 --> 0:50:24.084 would be so thrilled. Yeah, no spoilers about my book, 0:50:24.204 --> 0:50:27.484 but it ends with us talking about how thrilled we 0:50:27.484 --> 0:50:27.884 would be. 0:50:27.804 --> 0:50:28.204 To be wrong. 0:50:28.244 --> 0:50:32.324 Okay, I just sent you a calendar invite for October thirtieth, 0:50:32.364 --> 0:50:34.364 two thousand and thirty five, where we can have a 0:50:34.364 --> 0:50:37.444 follow up, right podcast, Nate Perfect any closing thoughts. Has 0:50:37.484 --> 0:50:38.444 been a lot of fun to be with you. 0:50:40.564 --> 0:50:43.364 Yeah, I mean, first and foremost, glad that you're you're 0:50:43.364 --> 0:50:46.644 having the convo. Glad that we're having the convo. I 0:50:46.684 --> 0:50:49.444 think just more people realizing how crazy the situation is 0:50:49.604 --> 0:50:50.284 will go a long way. 0:50:50.324 --> 0:50:51.244 Towards making things better. 0:50:51.524 --> 0:50:53.404 Thank you so much for joining us, Nate, it's been 0:50:53.444 --> 0:50:55.604 an absolute pleasure, pleasure to be here. 0:50:56.204 --> 0:50:57.124 We'll talk to you soon. 0:51:04.084 --> 0:51:05.884 Let us know what you think of the show. Reach 0:51:05.924 --> 0:51:09.244 out to us at Risky Business at pushkin dot FM. 0:51:09.844 --> 0:51:12.924 Risky Business is hosted by me Maria Kanakova. 0:51:12.644 --> 0:51:15.404 And by me Nate Silver. The show was a cool 0:51:15.404 --> 0:51:19.364 production of Pushing Industries and iHeartMedia. This episode was produced 0:51:19.404 --> 0:51:23.764 by Isaac Carter. Our associate producer is Sonya gerwit Lydia, 0:51:23.844 --> 0:51:26.724 Jean Kott and Daphne Chen are our editors, and our 0:51:26.764 --> 0:51:30.524 executive producer is Jacob Goldstein. Mixing by Sarah Bruger. 0:51:31.164 --> 0:51:33.404 If you like the show, please rate and review us 0:51:33.404 --> 0:51:36.044 so other people can find us too, But once again, 0:51:36.204 --> 0:51:38.004 only if you like us. We don't want those bad 0:51:38.044 --> 0:51:40.204 reviews out there. Thanks for tuning in.