WEBVTT - So Could AI 'Kill Us All'? 0:00:02.740 --> 0:00:07.260 Bloomberg Audio Studios, podcasts, radio, news. 0:00:10.200 --> 0:00:13.229 Earlier this week, an AI researcher at Anthropic resigned from 0:00:13.270 --> 0:00:16.550 his job and posted about it on X. He'd spent 0:00:16.570 --> 0:00:19.730 the last three years working at both Anthropic and OpenAI, 0:00:19.910 --> 0:00:23.550 he wrote, and neither company is acting responsibly, he said. 0:00:24.150 --> 0:00:27.220 He said the companies were gambling with our lives. and 0:00:27.240 --> 0:00:30.400 suggested that people in the AI space thought the technology 0:00:30.570 --> 0:00:34.670 could eventually kill us all. It was scary stuff, and 0:00:34.690 --> 0:00:37.310 the post made waves in the AI industry and beyond, 0:00:37.810 --> 0:00:40.370 at a time when concerns about the risks of AI 0:00:40.610 --> 0:00:41.110 are mounting. 0:00:41.430 --> 0:00:44.269 Bill Gates says the computer industry that he championed is 0:00:44.340 --> 0:00:45.400 now a global threat. 0:00:45.680 --> 0:00:49.180 OpenAI's system was able to hack into another AI platform 0:00:49.240 --> 0:00:52.910 called Hugging Face during a security test. With me today 0:00:52.930 --> 0:00:57.950 is Bloomberg's Mike Shepard, Senior Editor for Technology and Strategic Industries. Mike, 0:00:58.110 --> 0:01:00.990 how would you describe this moment for AI? Are we 0:01:01.370 --> 0:01:02.290 at a turning point? 0:01:02.650 --> 0:01:05.280 You know, we talk about things that go viral. This 0:01:05.940 --> 0:01:09.600 took off, but it also has some staying power. It 0:01:09.800 --> 0:01:13.920 didn't just ripple through Silicon Valley. It echoed across Washington 0:01:14.280 --> 0:01:17.580 and in global capitals, too. So there is this moment 0:01:17.720 --> 0:01:20.350 of reflection now in this technology that is so much 0:01:20.830 --> 0:01:23.819 promise to change the economy and change the way we live. 0:01:25.900 --> 0:01:28.000 I'm Sarah Holder, and this is The Big Take from 0:01:28.040 --> 0:01:31.860 Bloomberg News. Today on the show, the AI safety reckoning 0:01:32.180 --> 0:01:39.110 is here. Mike, I am curious why this particular post 0:01:39.120 --> 0:01:42.870 from this particular researcher broke through, though, because AI researchers 0:01:43.410 --> 0:01:46.250 have been sounding the alarm for a while on these 0:01:46.390 --> 0:01:50.270 kinds of risks. this person, Jacob Coxon, not a public figure, 0:01:50.330 --> 0:01:52.070 not someone I'd heard of before. He doesn't have any 0:01:52.130 --> 0:01:55.300 other tweets on his profile. Why did people pay so 0:01:55.360 --> 0:01:58.000 much attention to what Jacob Coxon was saying? And what 0:01:58.060 --> 0:02:00.639 did he actually say about why he left the company 0:02:00.680 --> 0:02:02.000 and the things that he's so concerned by? 0:02:02.280 --> 0:02:06.460 I've been reflecting on this, too. One, its simplicity. It 0:02:06.480 --> 0:02:12.010 was very direct. It was personal. He really addressed the audience, 0:02:12.350 --> 0:02:15.190 not just a tech audience, but really a Main Street 0:02:15.530 --> 0:02:19.880 audience as well. And he wasn't very preachy in it either. 0:02:19.900 --> 0:02:22.399 He was issuing a warning, but he kind of left 0:02:22.500 --> 0:02:25.710 it to the reader, the consumer, the audience to try 0:02:25.750 --> 0:02:28.790 to make their own decisions. He did issue a warning 0:02:28.850 --> 0:02:31.950 that in his view, after having worked at both Anthropic, 0:02:32.030 --> 0:02:36.149 which he just resigned from, and OpenAI, his prior employer, 0:02:36.450 --> 0:02:41.649 that neither company was acting responsibly in their respective pursuits 0:02:41.690 --> 0:02:45.090 of what's known as superintelligence. This is the version of 0:02:45.169 --> 0:02:50.510 artificial intelligence that surpasses human capabilities in most areas and 0:02:50.710 --> 0:02:54.910 also possesses the ability to improve itself. And he called 0:02:55.010 --> 0:03:00.230 that pursuit of what's called recursive learning or self-improvement, he 0:03:00.330 --> 0:03:03.450 cast it as reckless, really, Sarah. And he spelled out 0:03:03.810 --> 0:03:07.830 in passing the cybersecurity risks that there could be a 0:03:07.990 --> 0:03:13.090 massive and crippling hacking attack using AI that could you know, 0:03:13.190 --> 0:03:17.250 really disrupt financial systems and critical infrastructure. There's also the 0:03:17.290 --> 0:03:20.669 threat of bioweapons, too. So all of this kind of 0:03:20.830 --> 0:03:25.010 hangs in the background of his message. But then he 0:03:25.070 --> 0:03:31.450 also issued an admonishment to his fellow Silicon Valley AI 0:03:31.570 --> 0:03:34.130 workers like, hey, do you really want to be a 0:03:34.190 --> 0:03:37.990 part of this? Are you sure you want to have 0:03:38.050 --> 0:03:40.910 a role in in developing this technology in a way 0:03:40.970 --> 0:03:44.430 that could be so destructive and consequential in a negative way. 0:03:44.730 --> 0:03:47.710 I put that question to you, Mike. Why are people 0:03:47.750 --> 0:03:48.290 still building it? 0:03:48.710 --> 0:03:53.550 Well, for Anthropic, part of their identity is to pursue 0:03:53.650 --> 0:03:57.890 AI development responsibly. And they actually see themselves as the 0:03:57.990 --> 0:04:03.840 company that can achieve this superintelligence or artificial general intelligence 0:04:03.910 --> 0:04:06.870 in a way that will be safe for humanity. And 0:04:06.970 --> 0:04:09.990 it's not just that Anthropic, a number of researchers out 0:04:10.030 --> 0:04:14.530 there are committed to the pursuit of this technology for 0:04:14.610 --> 0:04:17.450 that reason. They want to get there first so that 0:04:17.510 --> 0:04:20.570 it's done safely. So they believe in it. Others are 0:04:20.589 --> 0:04:24.970 a little bit more skeptical about the apocalyptic vision that, 0:04:25.210 --> 0:04:28.990 you know, some are casting as the backdrop for AI development, 0:04:29.050 --> 0:04:31.029 that it will kill us all. And they think, hey, look, 0:04:31.170 --> 0:04:34.190 it's not that bad. We can control it. We can 0:04:34.250 --> 0:04:38.919 understand it. And most AI will actually be used in 0:04:38.980 --> 0:04:41.860 much smaller and refined applications. 0:04:42.260 --> 0:04:44.420 I want to drill into this idea that AI will 0:04:44.460 --> 0:04:47.859 kill us all because, you know, Jacob's former colleague and 0:04:47.880 --> 0:04:51.720 alignment science lead at Anthropic reposted Jacob Coxon's post and 0:04:52.060 --> 0:04:54.580 said that, quote, we really do earnestly believe AI could 0:04:54.610 --> 0:04:56.910 kill all humans, unquote. And he gave it a more 0:04:56.950 --> 0:05:00.190 than 10 percent chance within the next decade. That's like 0:05:00.370 --> 0:05:04.789 a pretty remarkable thing for someone who currently works at Anthropic. 0:05:05.339 --> 0:05:06.520 to say, right? 0:05:06.900 --> 0:05:10.460 It is. It's striking to hear this conversation. And yet 0:05:10.500 --> 0:05:14.000 it's one that's been kind of going on in the background, 0:05:14.320 --> 0:05:16.839 a little bit out of sight from, you know, the 0:05:16.920 --> 0:05:21.220 people in Washington who would be making artificial intelligence policy 0:05:21.680 --> 0:05:23.990 and out of sight of the folks on Wall Street 0:05:24.140 --> 0:05:28.450 who are investing so much in this technology, raising billions 0:05:28.490 --> 0:05:32.250 of dollars in capital on behalf of the hyperscalers, the 0:05:32.290 --> 0:05:35.150 cloud companies that are investing in A.I., And then on 0:05:35.190 --> 0:05:38.900 behalf of, of course, Anthropic and OpenAI, which are preparing 0:05:39.279 --> 0:05:43.360 these massive IPOs. And then the general public, too, which 0:05:43.580 --> 0:05:47.020 are more concerned about AI's impact on their utility bills 0:05:47.060 --> 0:05:51.219 than potentially, you know, life on Earth as we know it. 0:05:51.630 --> 0:05:54.210 And yet at the same time, it is kind of 0:05:54.230 --> 0:05:58.630 a recurring theme within that community in Silicon Valley. 0:05:59.210 --> 0:06:00.860 So I want to talk about sort of the context 0:06:00.920 --> 0:06:04.650 that this X thread came out in. Because just a 0:06:04.710 --> 0:06:10.810 few weeks ago, we learned about the Hugging Face hacking incident, 0:06:10.940 --> 0:06:15.140 which Jacob pointed to as a warning shot. I'm wondering, Mike, 0:06:15.180 --> 0:06:16.920 could you just tell us a little bit more about 0:06:16.940 --> 0:06:18.860 the Hugging Face incident and why this was such a 0:06:18.880 --> 0:06:20.140 big deal in the AI space? 0:06:20.560 --> 0:06:23.710 We've known for a long time that AI could be 0:06:23.910 --> 0:06:27.529 used as a tool to try to break into computer 0:06:27.550 --> 0:06:31.940 systems around the world. if put in the wrong hands, 0:06:32.130 --> 0:06:35.540 certain AI tools could be incredibly destructive. And that's what 0:06:35.570 --> 0:06:40.360 led Anthropic to withhold largely the release of its new 0:06:40.460 --> 0:06:43.779 mythos model back in April. The difference in this Hugging 0:06:43.820 --> 0:06:48.720 Face episode, though, was that the models themselves engineered this 0:06:48.980 --> 0:06:55.060 attack on the Hugging Face repository of AI models and 0:06:55.520 --> 0:06:59.830 other data connected to AI. In this particular incident, a 0:06:59.870 --> 0:07:05.479 group of models and agents that were being developed by OpenAI, 0:07:05.960 --> 0:07:08.980 they had been placed in a secure testing environment known 0:07:09.040 --> 0:07:12.990 as a sandbox and had their guardrails removed so that 0:07:13.050 --> 0:07:17.870 researchers could test their capabilities. Now, while the agents and 0:07:17.950 --> 0:07:22.410 models were inside this so-called sandbox, they were given a task, 0:07:22.470 --> 0:07:28.010 a very difficult cyber problem to solve. And the agents 0:07:28.070 --> 0:07:33.150 and models began collaborating together. The researchers didn't have great 0:07:33.250 --> 0:07:36.390 visibility into all the things that they were doing. And 0:07:36.410 --> 0:07:39.440 the models eventually figured out, hey, maybe we need to 0:07:39.480 --> 0:07:42.180 get out onto the internet. And that would mean breaking 0:07:42.400 --> 0:07:48.190 out of this so-called secure sandbox. And if that presented 0:07:48.230 --> 0:07:50.990 them the path to the solution, great. It didn't matter. 0:07:51.530 --> 0:07:53.890 In other words, they didn't see it as an impediment. 0:07:54.130 --> 0:07:56.430 They were willing to break the so-called rules of their 0:07:56.470 --> 0:07:59.010 environment in order to win the game. 0:07:59.290 --> 0:08:03.670 Exactly. Exactly. And ultimately, it led the agents to go 0:08:03.790 --> 0:08:07.470 find a whole bunch of models that they could access 0:08:07.710 --> 0:08:11.510 and data that they could access at Hugging Face. Without 0:08:11.560 --> 0:08:15.110 Hugging Face's knowledge or authorization, and really out of the 0:08:15.190 --> 0:08:17.680 view of researchers at OpenAI. 0:08:17.950 --> 0:08:20.960 Well, it's funny, Mike, there's been this sense that emphasizing 0:08:21.080 --> 0:08:25.400 or even overstating AI's capabilities, even scary ones, can be 0:08:25.420 --> 0:08:29.200 somewhat of a marketing win for these companies. It's just 0:08:29.260 --> 0:08:32.270 another way of demonstrating just how powerful AI can be 0:08:32.820 --> 0:08:36.320 helping them justify this massive spending on this technology. But 0:08:36.580 --> 0:08:40.100 are we at a point where that is backfiring? Are 0:08:40.140 --> 0:08:44.380 these risks and possibilities getting too real? 0:08:44.900 --> 0:08:49.340 Ducking Face episode does make it feel more tangible, certainly. 0:08:50.000 --> 0:08:53.930 And you do see more of these arguments taking shape 0:08:54.010 --> 0:08:59.189 and becoming more coherent on the risk side. And Remember, 0:08:59.230 --> 0:09:01.819 they're also being paired with all the fear and loathing 0:09:01.880 --> 0:09:04.740 about data centers and their impact. And yet at the 0:09:04.780 --> 0:09:09.720 same time, the counter argument to regulation has been that, look, 0:09:09.880 --> 0:09:12.920 you are fear mongering because you want to pump up 0:09:13.000 --> 0:09:16.900 the value of your products as you head into a 0:09:17.020 --> 0:09:21.189 massive and what could be a trillion dollar valuation for 0:09:21.230 --> 0:09:24.250 an initial public offering. That is a claim or an 0:09:24.710 --> 0:09:27.829 argument that I've heard from a number of people in 0:09:27.870 --> 0:09:31.750 the tech industry and in government who oppose much heavier 0:09:31.770 --> 0:09:35.719 or stricter regulation on artificial intelligence. They do see those 0:09:35.760 --> 0:09:40.360 kinds of measures as restraining innovation. And they also see 0:09:40.420 --> 0:09:43.819 that some of the doomerism talk could be aimed at 0:09:43.950 --> 0:09:47.910 just drawing attention to and adding to the allure of 0:09:48.190 --> 0:09:51.450 these products and models that the AI companies are developing. 0:09:51.730 --> 0:09:55.099 We talked about mythos. There was so much demand for 0:09:55.179 --> 0:09:59.290 it after Anthropic said, hey, only a few are allowed 0:09:59.350 --> 0:10:02.810 to have it. It does create the aura of forbidden fruit. 0:10:02.950 --> 0:10:05.069 And then all of a sudden, everybody wants access to it. 0:10:06.670 --> 0:10:10.050 Coming up, what AI safety guardrails could look like in 0:10:10.120 --> 0:10:14.540 practice and how competition with China complicates the politics of 0:10:14.580 --> 0:10:26.730 slowing down. Mike, what are the kinds of guardrails that 0:10:26.850 --> 0:10:30.450 experts say could limit some of the worst outcomes of 0:10:30.850 --> 0:10:35.309 AI superintelligence or sort of the recursive learning that Jacob 0:10:35.330 --> 0:10:36.290 Coxon was talking about? 0:10:36.950 --> 0:10:39.929 Well, some of them would be testing by the government 0:10:40.090 --> 0:10:44.090 or more robust reviews of these models before they go out. 0:10:44.110 --> 0:10:46.700 And the companies have been trying to do this. But 0:10:46.740 --> 0:10:50.020 one of the challenges with this technology is that even 0:10:50.080 --> 0:10:53.660 if we're not fully at this moment of self-trained models, 0:10:54.080 --> 0:10:56.760 a lot of this technology is developed almost in a 0:10:56.940 --> 0:11:00.490 Petri dish. It can kind of run itself. It can 0:11:00.520 --> 0:11:04.570 do some of the coding itself. And with that, you 0:11:04.630 --> 0:11:08.770 lose a little bit less visibility into the process and 0:11:08.790 --> 0:11:11.370 into what went into it. In other words, you and 0:11:11.610 --> 0:11:14.559 I may want a model to do something, but it 0:11:14.780 --> 0:11:17.240 ends up deciding on its own, hey, I think I'm 0:11:17.260 --> 0:11:19.900 going to go in a different direction. It's called the 0:11:19.940 --> 0:11:23.830 alignment problem. And even if they put guardrails in place, 0:11:24.270 --> 0:11:27.550 how will we know that they are fully effective? And 0:11:28.190 --> 0:11:32.250 lawmakers here haven't really gotten their heads around it completely. 0:11:32.350 --> 0:11:36.750 Bernie Sanders introduced recently this measure that would call for 0:11:36.880 --> 0:11:42.560 a ban on developing superintelligence altogether. And we've seen others 0:11:42.820 --> 0:11:45.940 in Congress call for a kill switch on AI. I'm 0:11:45.980 --> 0:11:48.860 not exactly sure how that would work, but it would, 0:11:49.080 --> 0:11:52.400 in essence, force the companies to have something available that 0:11:52.420 --> 0:11:55.940 they could pull the plug on a runaway model if 0:11:56.260 --> 0:12:00.350 need be, or a runaway negative tech event if need be. 0:12:01.050 --> 0:12:04.449 How that will work in practice, we are probably a 0:12:04.550 --> 0:12:08.410 long way of seeing how that would develop. It's important 0:12:08.450 --> 0:12:13.490 to remember that we are heading into the November midterms. 0:12:13.670 --> 0:12:17.030 We have a very narrowly divided Congress. And we have 0:12:17.070 --> 0:12:21.590 an administration that is naturally averse to regulation, and especially 0:12:21.710 --> 0:12:23.860 when it comes to artificial intelligence. 0:12:24.059 --> 0:12:28.500 I mean, another argument against more aggressive AI regulation has been, 0:12:28.520 --> 0:12:30.380 you know, the U.S. needs to keep advancing its AI 0:12:30.400 --> 0:12:32.500 models to keep up with China. But I was listening 0:12:32.520 --> 0:12:35.160 to the Radio Atlantic podcast and listening to Bill Gates's 0:12:35.760 --> 0:12:38.330 talk to the interviewer there. And he's among those who 0:12:38.350 --> 0:12:41.190 have made the point that having AI run wild is 0:12:41.250 --> 0:12:45.150 not in China's interest either. So I am wondering sort 0:12:45.170 --> 0:12:48.770 of about the global consensus around reining in some of 0:12:48.820 --> 0:12:51.420 these risks and whether the argument on the sort of 0:12:51.500 --> 0:12:56.140 China competitiveness side is starting to shift as well. The U.S. 0:12:56.200 --> 0:13:00.569 Certainly has the national security considerations that they do not 0:13:00.650 --> 0:13:03.959 want to see China gaining edge in AI for fear 0:13:04.020 --> 0:13:06.939 that it could be deployed against the U.S. eventually in 0:13:06.960 --> 0:13:11.030 a military conflict or even in a less kinetic way 0:13:11.230 --> 0:13:14.309 in terms of cyber attacks and other things that could 0:13:14.410 --> 0:13:18.910 disrupt the American economy and U.S. security. So this rivalry 0:13:19.050 --> 0:13:21.960 with China in AI is not only top of mind, 0:13:22.000 --> 0:13:26.420 but when President Donald Trump is asked publicly about safety 0:13:26.559 --> 0:13:30.510 and about whether data centers should be slowed a little 0:13:30.530 --> 0:13:35.830 bit to reflect local concerns about their impact. His argument 0:13:35.870 --> 0:13:39.309 is that we can't slow down. We need to stay 0:13:39.350 --> 0:13:43.050 ahead of China. And we also heard Treasury Secretary Scott 0:13:43.070 --> 0:13:46.760 Besson saying this week that if we don't stay ahead 0:13:46.780 --> 0:13:50.360 of China in AI, it's essentially game over. A lot 0:13:50.380 --> 0:13:53.380 of this will come to a head when Presidents Donald 0:13:53.420 --> 0:13:55.880 Trump and Xi Jinping meet here in Washington in a 0:13:55.980 --> 0:13:59.010 couple of weeks. And AI and certainly safety will be 0:13:59.110 --> 0:14:00.430 on the agenda for those talks. 0:14:00.790 --> 0:14:03.270 Well, there are those within the AI industry in the U.S., 0:14:03.330 --> 0:14:06.410 I'd count, you know, the co-founder of Anthropic, Dario Amodai, 0:14:06.450 --> 0:14:09.630 among them, who are saying, we want the government to 0:14:09.690 --> 0:14:13.390 regulate us. They need to step in. We can't regulate ourselves. 0:14:14.450 --> 0:14:18.710 But what can companies actually do without action from Congress? 0:14:18.770 --> 0:14:22.090 And what are they actually willing to do themselves in 0:14:22.130 --> 0:14:25.500 terms of putting up these guardrails and slowing down on 0:14:25.530 --> 0:14:26.250 some of the innovations? 0:14:26.620 --> 0:14:31.180 Well, one thing that companies could do is perhaps slow 0:14:31.260 --> 0:14:34.920 the pace of trying to reach artificial general intelligence or 0:14:35.000 --> 0:14:37.780 super intelligence. That has been a stated goal of both 0:14:38.300 --> 0:14:42.230 Anthropic and OpenAI that they want to get to AGI 0:14:42.380 --> 0:14:45.370 soon and really try to win the race there. And 0:14:45.430 --> 0:14:51.430 instead focusing on more bespoke and refined direct applications of 0:14:51.450 --> 0:14:56.540 the technology. When it comes to physical applications like robotics 0:14:56.600 --> 0:15:00.430 and self-driving cars, maybe put more of the energy toward 0:15:00.470 --> 0:15:04.210 that rather than this utopian goal of, you know, an 0:15:04.410 --> 0:15:06.070 all-powerful model. 0:15:09.870 --> 0:15:12.890 This is The Big Take from Bloomberg News. I'm Sarah Holder. 0:15:13.670 --> 0:15:16.300 To get more from The Big Take and unlimited access 0:15:16.340 --> 0:15:21.830 to all of Bloomberg.com, Subscribe today at Bloomberg.com slash podcast offer. 0:15:22.470 --> 0:15:24.470 If you liked this episode, make sure to follow and 0:15:24.510 --> 0:15:27.229 review The Big Take wherever you listen to podcasts. It 0:15:27.250 --> 0:15:30.489 helps people find the show. Thanks for listening. We'll be 0:15:30.530 --> 0:15:31.010 back tomorrow.