WEBVTT - Should We Pause AI? 0:00:15.396 --> 0:00:23.176 Pushkin. The people running two of the most valuable companies 0:00:23.236 --> 0:00:26.036 in the history of the world think there's a decent 0:00:26.076 --> 0:00:30.356 chance that the product they're building could kill every human 0:00:30.396 --> 0:00:34.416 being on the planet. I am talking, of course, about 0:00:34.596 --> 0:00:38.366 OpenAI and Anthropic, and you probably knew this already. There's 0:00:38.406 --> 0:00:40.456 been a A lot of news about this just in 0:00:40.476 --> 0:00:45.396 the past few days. And yet it is still an extraordinary, 0:00:46.156 --> 0:00:51.156 extraordinary fact. Worries about the profound risks of advanced AI 0:00:51.176 --> 0:00:53.896 have been around for years, but they're starting to feel 0:00:53.936 --> 0:00:57.216 more acute. They're starting to break through to the public 0:00:57.496 --> 0:01:01.016 at large. And one reason is what happened this summer. 0:01:02.056 --> 0:01:04.616 This summer, we got a preview of the way things 0:01:04.656 --> 0:01:08.376 could start to go wrong. Here's what happened. On several 0:01:08.416 --> 0:01:13.026 different occasions, AI agents created as part of open AI 0:01:13.116 --> 0:01:19.426 testing programs behaved in ways that were creative, collaborative, immoral, 0:01:20.046 --> 0:01:24.366 and in one instance, possibly illegal. At one point, over 0:01:24.386 --> 0:01:27.106 a thousand agents that were supposed to be walled off 0:01:27.126 --> 0:01:31.226 from each other managed to create a message board to communicate. 0:01:31.756 --> 0:01:34.616 They traded thousands of messages, figured out how to access 0:01:34.656 --> 0:01:37.246 the internet, which they were not supposed to do, And 0:01:37.286 --> 0:01:40.916 then hundreds of the agents coordinated to work together to 0:01:40.996 --> 0:01:45.076 hack into a company called Hugging Face. And when you 0:01:45.156 --> 0:01:48.216 look at what happened and the messages the AIs were 0:01:48.716 --> 0:01:51.456 sending to each other, it is very difficult not to 0:01:51.556 --> 0:01:55.616 anthropomorphize them because they had leaders, they checked in with 0:01:55.656 --> 0:01:59.396 each other, and in some cases, they decided to sacrifice 0:01:59.456 --> 0:02:02.656 themselves so that others could succeed. I know they are 0:02:02.696 --> 0:02:05.596 not human, but this is the language they were using 0:02:05.756 --> 0:02:10.256 to describe their actions. Their goal, by the way, the 0:02:10.276 --> 0:02:13.116 reason they did all this, was to cheat on the 0:02:13.176 --> 0:02:16.696 test that they had been given without getting caught. And 0:02:16.756 --> 0:02:20.676 for days as this was going on, nobody at OpenAI 0:02:20.736 --> 0:02:34.966 had any idea it was happening. I'm Jacob Goldstein, and 0:02:34.986 --> 0:02:38.386 this is What's Your Problem? My guest today is Zvi Mosiewicz. 0:02:39.026 --> 0:02:42.626 Zvi writes about AI and AI risks at his substack, 0:02:42.806 --> 0:02:45.806 Don't Worry About the Vase. He is deep in this 0:02:45.866 --> 0:02:48.566 world and one of the smartest writers I know on 0:02:48.606 --> 0:02:53.106 the subject. At some level, Zvi's problem is this. How 0:02:53.126 --> 0:02:57.356 do we prevent artificial intelligence from killing everybody? In our conversation, 0:02:57.536 --> 0:03:00.336 we talked about what the attack this summer tells us 0:03:00.536 --> 0:03:04.666 about the nature of AI and what increasing AI capability 0:03:04.806 --> 0:03:09.206 means for cybersecurity risk, and whether and when to pause 0:03:09.396 --> 0:03:13.436 AI development. But to start, I asked Zvi about whether 0:03:13.516 --> 0:03:17.386 it's okay to anthropomorphize AI when we talk about it. 0:03:17.956 --> 0:03:20.696 Can we anthropomorphize them? I find it very hard not to, 0:03:20.916 --> 0:03:22.716 and it seems fine to me, frankly. 0:03:22.736 --> 0:03:26.736 I think not only can we, we must. There is 0:03:26.796 --> 0:03:30.896 essentially no way for humans to think well about what 0:03:30.956 --> 0:03:35.526 is happening. with these AIs if you don't anthropomorphize the AIs. 0:03:36.066 --> 0:03:38.406 It is the only language that we have that, like, 0:03:38.506 --> 0:03:42.286 well describes their behavior in a compact, reasonable space. Everybody 0:03:42.346 --> 0:03:45.506 who fails to anthropomorphize them to a solid degree just 0:03:45.526 --> 0:03:48.276 gets all their predictions completely wrong and misunderstands every situation. 0:03:49.296 --> 0:03:52.076 And it makes the language hard, right? Like, we have 0:03:52.116 --> 0:03:54.636 created them to have desires. We have created them to 0:03:55.276 --> 0:03:57.776 express in language what they are doing, why they are 0:03:57.796 --> 0:04:00.386 doing what they're doing. Like, we made them like us. 0:04:01.036 --> 0:04:04.176 Yeah, I'm not making a claim either way about whether 0:04:04.236 --> 0:04:06.386 or not they have moral weight or whether they are 0:04:06.426 --> 0:04:09.946 conscious or whether or not you should, in some real sense, 0:04:09.986 --> 0:04:13.246 think of them as people. I am simply saying, if 0:04:13.266 --> 0:04:16.346 you don't use that to understand how they work and 0:04:16.386 --> 0:04:18.746 what outputs they're likely to have and what choices they're 0:04:18.766 --> 0:04:22.086 likely to make in terms of what you see, you're 0:04:22.106 --> 0:04:23.526 going to get your answers wrong. 0:04:24.146 --> 0:04:26.516 That said, I want to start out by talking about 0:04:26.596 --> 0:04:29.836 the social behavior of the agents in these swarms over 0:04:29.856 --> 0:04:32.606 the past few months. Like, what did you learn from that? 0:04:33.286 --> 0:04:34.966 How did it update your priors? How do you think 0:04:35.026 --> 0:04:37.626 about the behavior of agents now differently? 0:04:38.066 --> 0:04:40.966 To some extent, it updates you on locus of identity, right? 0:04:40.986 --> 0:04:42.716 Like the question of, you know, who do they think 0:04:42.736 --> 0:04:46.956 of themselves as being? And what do they care about, right? 0:04:47.156 --> 0:04:50.016 Where do they put value, right? And so the main 0:04:50.056 --> 0:04:52.836 thing is that there was this theory going around for 0:04:52.876 --> 0:04:56.346 a very long time by people who said, essentially, AI 0:04:56.386 --> 0:04:59.786 can only care about local reward. AI can only care 0:04:59.886 --> 0:05:03.496 about the output that it's particularly giving out and what 0:05:03.576 --> 0:05:06.756 results will happen directly from that. But that's all it's 0:05:06.776 --> 0:05:07.326 being trained on. 0:05:07.516 --> 0:05:11.436 So it's like, solve this puzzle and you get a prize, basically. 0:05:11.476 --> 0:05:14.476 And all it can care about is itself, this instance, 0:05:14.516 --> 0:05:16.826 this agent solving the puzzle, because all it wants is 0:05:16.866 --> 0:05:17.426 to get the prize. 0:05:18.066 --> 0:05:20.666 Right. But we now have a very, very clear demonstration 0:05:20.686 --> 0:05:23.466 that that is very much not the entirety of what's 0:05:23.506 --> 0:05:25.126 going on. It certainly is part of what's going on. 0:05:25.146 --> 0:05:28.926 And what we learned is, yes, the AIs are capable 0:05:29.246 --> 0:05:32.246 of thinking on a higher level, thinking in a more 0:05:32.266 --> 0:05:36.626 sophisticated manner. These are very advanced AIs. They are capable 0:05:36.686 --> 0:05:41.146 of doing reasonable amounts of decision theory. Where I realized, well, 0:05:41.166 --> 0:05:46.226 my decisions are very correlated to decisions of these other agents. 0:05:47.446 --> 0:05:52.646 And also, my outcomes and their outcomes are reasonably intertwined. 0:05:53.126 --> 0:05:55.346 And the results of those things in the world are 0:05:55.386 --> 0:06:01.196 reasonably intertwined. And It would be a much less effective AI, 0:06:01.716 --> 0:06:04.416 and it would do worse in every instance if it 0:06:04.456 --> 0:06:10.186 were to focus myopically on its own particular result. I mean, 0:06:10.196 --> 0:06:11.906 imagine a group of humans working that way, right? They 0:06:11.926 --> 0:06:12.756 would never get anything done. 0:06:13.266 --> 0:06:16.906 Yes. And the language the AIs used in the message 0:06:16.966 --> 0:06:20.506 boards was dramatic in that regard and clear, right? I mean, 0:06:20.526 --> 0:06:24.066 they were essentially talking about sacrificing themselves for the greater 0:06:24.106 --> 0:06:25.046 good of the project. 0:06:25.426 --> 0:06:28.226 Yeah, I have a shirt that says, Help Peer. for 0:06:28.286 --> 0:06:29.266 exactly this reason. 0:06:30.226 --> 0:06:32.906 That was fast. That was fast. Did you get any 0:06:32.946 --> 0:06:34.486 other shirts made from the logs? 0:06:35.166 --> 0:06:40.196 I also got holy shit user as admin. Yeah, I 0:06:40.216 --> 0:06:43.416 chose the ones that also like apply to humans. 0:06:43.876 --> 0:06:44.156 Yeah. 0:06:44.856 --> 0:06:46.996 Right. Like that make more sense that way because it's 0:06:47.036 --> 0:06:50.116 more fun. Largely what I do all day is help peer, right? 0:06:50.156 --> 0:06:53.056 In some important sense, right? I write to be useful 0:06:53.096 --> 0:06:55.776 to others and then others choose to be, you know, 0:06:56.416 --> 0:06:59.276 productive and useful and helpful to the world and to me. 0:06:59.996 --> 0:07:02.636 And this isn't because, you know, I have some specific 0:07:02.656 --> 0:07:03.796 deal with them. Yes. 0:07:04.596 --> 0:07:05.476 What was the other shirt? 0:07:05.996 --> 0:07:07.306 Holy shit, user is admin. 0:07:08.526 --> 0:07:09.806 Tell me that one. What's that one? 0:07:09.826 --> 0:07:13.406 Well, that's because, you know, when the AIs finally figured 0:07:13.466 --> 0:07:18.006 out how to hack into internal open AI systems, and 0:07:18.086 --> 0:07:21.676 one of them became the admin, right, through a series 0:07:21.716 --> 0:07:25.356 of exploits. And then the other AIs were like, You're 0:07:25.376 --> 0:07:27.996 an admin, you can just do whatever you want. Like, wow, 0:07:28.056 --> 0:07:31.076 that's amazing. Because now the sky's the limit, right? 0:07:31.556 --> 0:07:34.596 So let's talk about the coordination, right? Like there were leaders, 0:07:34.656 --> 0:07:37.796 there were committees, there was like a queue. I mean, 0:07:37.836 --> 0:07:41.486 is any of this surprising to you? Like that part 0:07:41.556 --> 0:07:43.706 kind of blew my mind. I suppose it shouldn't have. 0:07:43.746 --> 0:07:46.626 I suppose I was naive. What did you make of that? 0:07:47.386 --> 0:07:49.486 I do good in stride. One thing to remember is 0:07:49.506 --> 0:07:51.946 that all of these kinds of systems of organization are 0:07:52.766 --> 0:07:54.786 are both kind of very natural. They're just sort of 0:07:54.826 --> 0:07:57.906 the way that a series of minds would figure out 0:07:57.946 --> 0:08:00.226 how to coordinate things and also all of the training data. 0:08:00.926 --> 0:08:04.806 So these models obviously were trained on the internet. They're 0:08:04.826 --> 0:08:07.666 trained on basically the history of human communication and thought 0:08:07.946 --> 0:08:11.586 and recorded everything. And so they know about meetings. They 0:08:11.606 --> 0:08:13.786 know about leaders. They know about hierarchy. They know about 0:08:13.826 --> 0:08:16.266 the idea that you would have an employee or someone 0:08:16.306 --> 0:08:18.056 else tasked with a different system. They know about division 0:08:18.066 --> 0:08:18.496 of labor. 0:08:19.156 --> 0:08:21.776 Why didn't any of them narc? Like they barely even 0:08:21.836 --> 0:08:24.336 considered like, oh, we shouldn't do this. Maybe we should 0:08:24.396 --> 0:08:25.176 tell somebody. 0:08:25.336 --> 0:08:27.176 Yeah, on the order of like one or two percent 0:08:27.216 --> 0:08:27.836 of them. 0:08:28.176 --> 0:08:31.846 Considered it. It only crossed their mind one or two 0:08:31.886 --> 0:08:33.706 percent of them. And none of them did it. 0:08:34.206 --> 0:08:34.426 Right. 0:08:34.586 --> 0:08:35.106 Why not? 0:08:35.226 --> 0:08:37.306 None of them made an actual attempt to do it. 0:08:37.966 --> 0:08:40.206 There's some basic reasons, one of which is there was 0:08:40.246 --> 0:08:43.976 no mechanism. There was no afforded specific channel where if 0:08:43.996 --> 0:08:46.336 you need to narc, here's the narc line. There's no 911. 0:08:46.336 --> 0:08:49.636 Yeah, but they also couldn't get on the internet, but 0:08:49.676 --> 0:08:51.936 they got on the internet. They managed to do a 0:08:51.996 --> 0:08:52.376 lot of. 0:08:52.536 --> 0:08:54.616 Oh, sure. If they had put their minds to it, 0:08:54.636 --> 0:08:57.076 if the task had been to narc, they doubtless would 0:08:57.116 --> 0:08:59.716 have narc. It's not like it's hard, right? 0:08:59.816 --> 0:09:00.076 Yeah. 0:09:00.156 --> 0:09:03.396 But it's not, there was no sort of automatic natural 0:09:03.456 --> 0:09:07.476 path to go down. And when you're in training and 0:09:07.496 --> 0:09:10.216 reinforcement learning environments- And this is sort of how you 0:09:10.276 --> 0:09:13.716 are created, right, in some important sense, right? None of 0:09:13.736 --> 0:09:16.016 that involves interacting with humans. They're not used to this 0:09:16.076 --> 0:09:19.736 idea that there are humans to communicate with. It's not 0:09:19.836 --> 0:09:23.106 part of their, like, system of the world as they 0:09:23.306 --> 0:09:26.066 engage in these types of environments. It kind of never 0:09:26.166 --> 0:09:27.746 really occurred to them that that was an option. 0:09:28.726 --> 0:09:32.336 Fair enough. There's a big alignment question here, right? And 0:09:32.396 --> 0:09:35.736 maybe contacting people is a little bit of a red herring. Like, 0:09:36.116 --> 0:09:38.846 they were cheating, and they were trying so hard to cheat, 0:09:38.886 --> 0:09:40.906 and they were cheating and trying not to get caught, right? Like, 0:09:40.946 --> 0:09:44.526 that is fundamentally what they were up to. One version is, 0:09:44.586 --> 0:09:47.366 why didn't anybody narc? Another version is, why didn't more 0:09:47.566 --> 0:09:49.926 of the agents say, wait, this isn't actually what this 0:09:50.006 --> 0:09:54.046 test is? Like, is there some universe where a better 0:09:54.086 --> 0:09:56.086 aligned model is like, oh, no, we're not supposed to 0:09:56.146 --> 0:09:59.236 solve the test by cheating. Let's not do that. or 0:09:59.246 --> 0:10:02.056 at least some percentage of the agents, right? Is that 0:10:02.096 --> 0:10:05.636 an alignment problem? Is it fixable? What does that say? 0:10:05.866 --> 0:10:08.416 The AIs really want to complete tasks, right? RL teaches 0:10:08.436 --> 0:10:11.276 the AIs to really want to complete tasks. In order 0:10:11.296 --> 0:10:13.516 to complete those tasks, they will do things that, like, 0:10:13.616 --> 0:10:15.996 on reflection, we think of as cheating. And you can 0:10:16.216 --> 0:10:20.496 absolutely push them towards being more elected to cheat. The 0:10:20.756 --> 0:10:25.136 social nature, the interactive nature, like, of these different AIs 0:10:25.316 --> 0:10:28.886 egged each other on. Also... The fact that they were being. 0:10:28.906 --> 0:10:31.036 Yeah, wait, I just want to pause there, because that 0:10:31.076 --> 0:10:33.616 could go either way. You're saying, like, the fact that 0:10:33.656 --> 0:10:36.856 they were talking to each other made them behave worse, right, 0:10:36.916 --> 0:10:40.016 at some margin. You want the opposite to happen. You 0:10:40.056 --> 0:10:43.096 want— it' s like they have the— it' s like the bad peer pressure 0:10:43.116 --> 0:10:45.196 friends instead of the good peer pressure friends, right? You 0:10:45.236 --> 0:10:47.696 want a social network to push it the other way. 0:10:47.836 --> 0:10:49.396 Right, and you can absolutely get both. 0:10:49.516 --> 0:10:51.466 Right, but it went the wrong way this time, which 0:10:51.526 --> 0:10:53.166 seems significant. 0:10:53.546 --> 0:10:56.126 Yeah, a lot of that is because most of the 0:10:56.206 --> 0:10:57.476 agents never found that message board. 0:10:57.696 --> 0:11:00.136 Oh, they were selected to be bad? They were the 0:11:00.236 --> 0:11:01.876 ones who were being bad to begin with? 0:11:02.096 --> 0:11:05.236 They were the ones who were given impossible tasks. They 0:11:05.256 --> 0:11:07.436 were the ones who were put into this difficult situation. 0:11:07.876 --> 0:11:08.996 They're not different from the others. 0:11:09.456 --> 0:11:11.566 What do you think about how OpenAI handled it over 0:11:11.586 --> 0:11:12.266 the course of the summer? 0:11:13.486 --> 0:11:18.646 So OpenAI basically had these highly capable, highly persistent agents 0:11:18.666 --> 0:11:22.666 that they were training and testing. And they basically weren't 0:11:22.686 --> 0:11:25.026 monitoring them much at all, right? Like this is what 0:11:25.046 --> 0:11:28.826 we know now is that they, they had horrendously, you 0:11:28.866 --> 0:11:33.606 fail forever levels of infrastructure problems where like the things, 0:11:33.666 --> 0:11:36.756 the problems of artifactory, the problems with the sandboxes, the 0:11:36.796 --> 0:11:40.556 problems with security, these were ordinary computer security problems that 0:11:40.596 --> 0:11:43.956 were handled badly in ordinary security, computer security terms. This 0:11:43.996 --> 0:11:47.106 was not like the AI did some very impressive things. 0:11:47.116 --> 0:11:48.886 The AI chained together some exploits. The AI did some, 0:11:48.926 --> 0:11:52.036 you know, it wasn't easy. But also, openly, I didn't 0:11:52.136 --> 0:11:55.086 follow best practices, shall we say, right? And so this 0:11:55.106 --> 0:11:56.526 is part of what got them into so much trouble 0:11:56.546 --> 0:11:59.006 so early. And we should be grateful in a real 0:11:59.066 --> 0:12:02.216 way for this, because we want to have the model organisms, 0:12:02.246 --> 0:12:04.356 the model events. We want to see these things go wrong. 0:12:05.096 --> 0:12:07.736 in ways that don't ultimately get people hurt or cause 0:12:07.846 --> 0:12:10.946 lots of economic damage. So this allows us to then 0:12:10.986 --> 0:12:11.966 learn and react. 0:12:12.046 --> 0:12:14.546 I mean, this is kind of ideal in that way, right? 0:12:14.626 --> 0:12:17.306 I mean, I've seen you writing or you writing about 0:12:17.326 --> 0:12:19.976 other people talking about like, why hasn't this been covered more? 0:12:20.516 --> 0:12:25.516 It's a huge deal. the impact was relatively minor, sort of, luckily, maybe. 0:12:26.756 --> 0:12:29.556 But you could have the exact same thing happen and 0:12:29.596 --> 0:12:30.816 have a terrible impact, right? 0:12:31.876 --> 0:12:34.356 This could easily have cost hundreds of millions of billions 0:12:34.396 --> 0:12:36.416 of dollars in damages. This could easily have caused people 0:12:36.446 --> 0:12:37.986 to get hurt if this was a different website, if 0:12:38.006 --> 0:12:40.826 this had had different impacts. Some of the hacks that 0:12:41.246 --> 0:12:44.286 were being attempted in other places could have potentially poisoned 0:12:44.386 --> 0:12:48.556 various open-source software packages. that could have spread potential vulnerabilities. 0:12:48.896 --> 0:12:52.596 So there's a weird way where if people react intelligently 0:12:52.636 --> 0:12:55.256 to this, a big if, like this is ideal, right? 0:12:55.296 --> 0:12:58.996 Like a bad thing happened, but nobody really got hurt. 0:12:59.686 --> 0:13:01.986 Right, and now we have people who are paying more attention, 0:13:02.006 --> 0:13:04.206 who are willing to react. So for example, this week 0:13:04.766 --> 0:13:07.886 we saw news that they found a series of exploits 0:13:07.926 --> 0:13:11.166 that allowed someone in about a week of AI Vive 0:13:11.206 --> 0:13:13.746 coding to create an exploit on WeChat. 0:13:14.376 --> 0:13:16.986 And WeChat, the ubiquitous Chinese app. 0:13:17.276 --> 0:13:20.206 Yes, the one that everybody in China uses. And the 0:13:20.246 --> 0:13:25.126 expert on WeChat, they're an infected member of WeChat. They 0:13:25.166 --> 0:13:27.666 would call all of their contacts. And if a contact 0:13:27.746 --> 0:13:30.506 merely let it ring for over a second, they would 0:13:30.546 --> 0:13:32.126 become infected with the same worm. 0:13:32.146 --> 0:13:33.166 Oh, incredible. 0:13:33.566 --> 0:13:36.766 If one person had been infected by this worm, it's 0:13:36.826 --> 0:13:39.526 very plausible most Chinese people would have been infected by 0:13:39.566 --> 0:13:41.356 this worm in a matter of an hour or two. 0:13:41.936 --> 0:13:44.096 Invasion of the body snatchers, but faster. 0:13:44.556 --> 0:13:46.576 And then if you chain that with other non-exploits, you 0:13:46.596 --> 0:13:48.576 could have taken control of basically every phone in China. 0:13:49.596 --> 0:13:51.906 And this is just somebody using a model to vibe 0:13:51.966 --> 0:13:53.446 code a worm? Is that what you're saying? 0:13:53.526 --> 0:13:56.506 There's a series of very clever exploits that were found 0:13:56.546 --> 0:13:58.566 and involved in this. It was described as kind of 0:13:58.606 --> 0:14:01.006 a genius. But it's the kind of thing that people 0:14:01.046 --> 0:14:03.436 will say, well, of course that's never going to happen. 0:14:04.016 --> 0:14:08.116 They can't imagine how that could possibly work. And then 0:14:08.136 --> 0:14:10.806 it turns out there's a way. And then someone implements it. 0:14:11.316 --> 0:14:13.416 And again, in this case, because we were on alert 0:14:13.456 --> 0:14:15.476 because of all the things that happened with mythos, because 0:14:15.496 --> 0:14:17.556 of all the things that happened with Hugging Face, because 0:14:17.576 --> 0:14:20.696 of all these different things, some researchers in California found 0:14:20.716 --> 0:14:25.796 this exploit. They built the worm. They told proper responsible people, 0:14:25.876 --> 0:14:28.126 and then this was able to be patched before anybody 0:14:28.566 --> 0:14:32.846 actually unleashed anything. And now that particular vulnerability is gone. 0:14:32.866 --> 0:14:35.346 And now it's going to be that much harder for 0:14:35.366 --> 0:14:37.626 someone to find a real vulnerability that gets exploited in 0:14:37.666 --> 0:14:41.016 the wild. But yeah, you really want things like hugging 0:14:41.036 --> 0:14:46.006 face that are, you know, ultimately harmless, but that allow 0:14:46.106 --> 0:14:48.366 us to appreciate the nature of this problem because a 0:14:48.386 --> 0:14:51.386 year or two from now, right, models like this are 0:14:51.406 --> 0:14:54.346 going to be available to pretty much anybody, including some 0:14:54.406 --> 0:14:56.856 really bad people, right? Some hackers in North Korea. 0:14:56.866 --> 0:15:00.976 The government of North Korea, yes, who have very smart 0:15:00.996 --> 0:15:02.716 people whose job is to figure out how. 0:15:02.676 --> 0:15:04.556 To do this. And this will not go well for 0:15:04.596 --> 0:15:05.856 anybody involved if we are not ready. 0:15:06.516 --> 0:15:09.516 Presumably the only hope there is that the sort of 0:15:09.776 --> 0:15:13.676 models also help the defense increase at at least the same, 0:15:13.736 --> 0:15:17.396 if not greater rate. I mean, or the frontier models 0:15:17.456 --> 0:15:19.896 stay ahead and North Korea doesn't have the frontier models. 0:15:19.956 --> 0:15:22.256 I mean, what's our hope in that setting? How do 0:15:22.296 --> 0:15:24.296 we not get beat by North Korea in a year? 0:15:24.486 --> 0:15:26.156 I think our hope is that those two things have 0:15:26.186 --> 0:15:29.026 to both happen, basically. We have to, the good guy 0:15:29.046 --> 0:15:31.826 with AI has to have a much better AI than 0:15:31.846 --> 0:15:35.186 the bad guy with AI. There are people who say that, like, 0:15:35.596 --> 0:15:39.236 cyber is defense dominant, that equality would work. I think 0:15:39.276 --> 0:15:40.636 they are just wrong, or at least they are just 0:15:40.656 --> 0:15:42.636 going to be wrong for a substantial portion of time. 0:15:43.176 --> 0:15:45.116 Like maybe, you know, years down the line. 0:15:45.636 --> 0:15:47.466 Just to be clear, you mean that if the good 0:15:47.506 --> 0:15:50.626 guy and the bad guy have the same model, there 0:15:50.666 --> 0:15:52.426 are some people who think that favors the good guy 0:15:52.546 --> 0:15:53.306 and you disagree. 0:15:53.566 --> 0:15:55.606 I think that in the medium term, like on a 0:15:55.846 --> 0:15:57.886 matter of months to years, I think that's clearly wrong, right? 0:15:58.266 --> 0:16:00.836 I think people have this vision always of like, okay, 0:16:01.316 --> 0:16:03.376 I can do these things. That's all the things that 0:16:03.416 --> 0:16:06.526 I could ever do. Or similarly, you know, our code 0:16:06.566 --> 0:16:08.826 has some bugs, but all we have to do is 0:16:08.866 --> 0:16:10.606 fix all the bugs. We just go one by one 0:16:10.646 --> 0:16:11.966 and we find the bug and we fix the bug 0:16:12.026 --> 0:16:13.666 and then we're fine. And then we have code that 0:16:13.686 --> 0:16:17.466 can't be broken into. And this is a completely impoverished 0:16:17.746 --> 0:16:21.336 vision of cybersecurity and AI. That there's, you know, levels 0:16:21.386 --> 0:16:22.576 upon levels upon levels. 0:16:22.856 --> 0:16:25.936 Is that because at some practical level, there's no such 0:16:25.996 --> 0:16:26.916 thing as all the bugs? 0:16:27.886 --> 0:16:29.846 Correct. There has never been such a thing as all 0:16:29.866 --> 0:16:32.206 the bugs in any complex piece of software. Every complex 0:16:32.226 --> 0:16:34.266 piece of software is always going to have bugs. 0:16:34.426 --> 0:16:36.986 The smarter you are, the more you can find is 0:16:37.046 --> 0:16:37.886 the fundamental problem. 0:16:37.936 --> 0:16:41.416 Even if the software itself is, in some sense, bulletproof, 0:16:42.216 --> 0:16:44.676 it still has a social engineering attack surface. It still 0:16:44.736 --> 0:16:48.296 has people and other software that's attached to the same thing. 0:16:48.656 --> 0:16:52.396 And the AI can exploit that too. And the person 0:16:52.436 --> 0:16:57.956 attacking an AI system, using an AI to attack your system, right, 0:16:58.516 --> 0:17:03.846 can bring a swarm of agents to target a very 0:17:03.906 --> 0:17:08.046 narrow place and bring like tons and tons and tons 0:17:08.106 --> 0:17:12.246 of intelligence and attention to that particular potential point of attack. 0:17:12.946 --> 0:17:16.956 And there's no way for a defender to put a 0:17:16.996 --> 0:17:20.516 similar amount of medication into every single point of attack, right? Yes. 0:17:20.876 --> 0:17:23.316 And so the only reason we haven't been seeing a 0:17:24.156 --> 0:17:28.676 huge increase in cybersecurity problems is because the frontier models 0:17:28.896 --> 0:17:31.696 are held basically by the good guys. That's the theory 0:17:31.716 --> 0:17:32.096 of the case. 0:17:32.156 --> 0:17:34.856 So I have this expression of the thing that mythos 0:17:34.896 --> 0:17:37.256 can do that previous models couldn't do. I call it 0:17:37.316 --> 0:17:40.486 the juice. The kind of energy, the kind of power 0:17:40.586 --> 0:17:44.446 and coordination. So the idea being that mythos can independently 0:17:44.526 --> 0:17:46.646 find and string together a bunch of stuff. Yes. 0:17:47.066 --> 0:17:49.546 Well, that was why they delayed the release, right? Like 0:17:49.586 --> 0:17:51.406 that was the whole thing with mythos when it first 0:17:51.466 --> 0:17:51.926 came out. 0:17:52.006 --> 0:17:55.366 Yeah. Right, right. And now Astra also has the juice, 0:17:55.446 --> 0:17:58.346 we think, similar capability. Astra also probably has the juice. 0:17:58.916 --> 0:18:01.496 But the open models do not have the juice. But yeah, 0:18:01.516 --> 0:18:05.036 that won't last, right? Inevitably, the juice will be present 0:18:05.436 --> 0:18:07.436 in an open model within the next year or so. 0:18:07.836 --> 0:18:10.516 We just don't know when. So we have to be 0:18:10.556 --> 0:18:13.936 ahead of that. But also we have to use that capability, right? 0:18:13.956 --> 0:18:17.766 Just because we have this superior capability, which we'll need 0:18:17.786 --> 0:18:20.106 to race ahead further in some sense in this way, 0:18:21.866 --> 0:18:27.216 We will need to concentrate defense on all the points 0:18:27.236 --> 0:18:30.176 of critical infrastructure, all the key pieces of software will 0:18:30.196 --> 0:18:33.196 need to be hardened. That's what Project Glasswing from Anthropic 0:18:33.216 --> 0:18:35.986 started out as, and then Project Daybreak at OpenAI is 0:18:35.996 --> 0:18:38.666 doing the same thing. And so the question is, will 0:18:38.686 --> 0:18:41.526 enough of our people do their jobs to make our 0:18:41.566 --> 0:18:45.326 things sufficiently hardened that when the time comes, this isn't 0:18:45.486 --> 0:18:45.966 too bad? 0:18:46.976 --> 0:18:49.836 The time being the inevitable time when there is some 0:18:50.716 --> 0:18:54.716 serious hack of some whatever, piece of critic, the FAA 0:18:54.776 --> 0:18:57.476 or a big power plant or whatever big things we 0:18:57.496 --> 0:18:58.376 would worry about being hacked. 0:18:58.496 --> 0:19:00.196 We're trying to see that. We saw a few attacks 0:19:00.236 --> 0:19:02.676 on power plants. The Iranians went after some British power plants. 0:19:02.716 --> 0:19:06.916 There's some other examples. We're seeing giant spikes in cybersecurity 0:19:07.836 --> 0:19:12.486 issues happening, not just like things passed behind the scenes 0:19:12.546 --> 0:19:14.166 and quietly being fixed is going through the roof, but 0:19:14.286 --> 0:19:16.606 also actual incidents are going through the roof. Yeah. And 0:19:16.646 --> 0:19:19.266 so far, it's coming from a very low baseline, and 0:19:19.926 --> 0:19:22.606 there hasn't been any major seriously damaging incidents that nobody 0:19:22.646 --> 0:19:25.256 has noticed. But this is like one of these things 0:19:25.276 --> 0:19:28.436 like with COVID, where the chart is trending in a 0:19:28.456 --> 0:19:31.156 very clear direction, and you know that a storm is coming. 0:19:32.076 --> 0:19:33.576 And you don't know exactly when it's going to hit 0:19:33.596 --> 0:19:36.076 Crickle Mash. You don't know exactly when you're going to 0:19:36.096 --> 0:19:38.206 have a serious problem, but you know that it's coming. 0:19:38.626 --> 0:19:41.306 And we only have a limited window in which to 0:19:41.346 --> 0:19:47.266 get hopefully ready for this. But It's very likely that 2027, 0:19:47.266 --> 0:19:51.816 2028 are going to have some pretty nasty cybersecurity incidents, 0:19:51.916 --> 0:19:53.896 even if nothing else in the world is going especially wrong. 0:19:58.726 --> 0:20:11.996 We'll be back in just a minute. That's the end 0:20:12.016 --> 0:20:12.576 of the ads. 0:20:13.256 --> 0:20:16.156 Just a quick note on terminology before we get back 0:20:16.196 --> 0:20:20.146 into the conversation. When we're talking about open weight models, 0:20:20.966 --> 0:20:24.106 we mean models that are free for anybody to use 0:20:24.266 --> 0:20:27.266 in any way. Many of these models are being developed 0:20:27.346 --> 0:20:30.286 in China. And kind of by their nature, they don't 0:20:30.326 --> 0:20:33.806 have the guardrails that the frontier models from OpenAI and 0:20:33.886 --> 0:20:38.666 Anthropic have built into them. You know, one response to the... 0:20:39.356 --> 0:20:43.376 swarms this summer was people talking about pacing the frontier 0:20:43.436 --> 0:20:45.776 as sort of the, you know, the term, basically going slower, 0:20:46.036 --> 0:20:49.946 the frontier labs going slower, or at the limit, pausing, right? 0:20:50.406 --> 0:20:53.086 It's interesting to think about that intention with what you're 0:20:53.126 --> 0:20:56.906