1
00:00:02,750 --> 00:00:04,430
Speaker 1: Bloomberg Audio Studios.

2
00:00:04,890 --> 00:00:06,430
Speaker 2: Podcasts. Radio.

3
00:00:06,850 --> 00:00:07,270
Speaker 3: News.

4
00:00:18,660 --> 00:00:22,260
Speaker 2: Hello and welcome to another episode of the Odd Lots podcast.

5
00:00:22,380 --> 00:00:23,460
Speaker 2: I'm Joe Wiesenthal.

6
00:00:23,680 --> 00:00:24,660
Speaker 1: And I'm Tracy Allaway.

7
00:00:25,100 --> 00:00:29,350
Speaker 2: Tracy, I'm very tired today because I flew in. on

8
00:00:29,390 --> 00:00:32,900
Speaker 2: the red eye last night from San Francisco. I was

9
00:00:33,120 --> 00:00:36,839
Speaker 2: out at this very interesting conference called Curve. I've heard

10
00:00:36,880 --> 00:00:40,800
Speaker 2: about it. In Berkeley. A lot of like, you know,

11
00:00:40,820 --> 00:00:43,300
Speaker 2: I guess a lot of people very concerned about AI,

12
00:00:43,680 --> 00:00:47,810
Speaker 2: so to speak. Interesting mix of folks, but yes, people

13
00:00:47,830 --> 00:00:50,530
Speaker 2: are concerned about what the future is going to look

14
00:00:50,590 --> 00:00:53,510
Speaker 2: like as we develop stronger and stronger AI models.

15
00:00:53,670 --> 00:00:58,440
Speaker 1: Just from a sociological or anthropological standpoint, I think living

16
00:00:58,480 --> 00:01:02,680
Speaker 1: in San Francisco and mingling with, you know, AI bros

17
00:01:02,780 --> 00:01:04,640
Speaker 1: at the moment must be one of the most interesting

18
00:01:04,680 --> 00:01:07,840
Speaker 1: things on Earth because there must be so much cognitive

19
00:01:07,880 --> 00:01:10,619
Speaker 1: dissonance in their heads, right? Where on the one hand,

20
00:01:10,640 --> 00:01:14,430
Speaker 1: they'll say like, oh, they're building this new ultra powerful model.

21
00:01:14,530 --> 00:01:17,289
Speaker 1: And on the other hand, sort of almost as an afterthought,

22
00:01:17,310 --> 00:01:18,610
Speaker 1: they'll be like, oh yeah, and I'm a little bit

23
00:01:18,650 --> 00:01:21,270
Speaker 1: worried about the possibility that it kills us all. How

24
00:01:21,330 --> 00:01:22,110
Speaker 1: do you square that?

25
00:01:22,370 --> 00:01:26,330
Speaker 2: It's super interesting. And you hear it like, so at

26
00:01:26,370 --> 00:01:31,560
Speaker 2: this conference, people are working for various safety type orgs,

27
00:01:31,660 --> 00:01:34,280
Speaker 2: people who want to regulate AI in some way. And

28
00:01:34,340 --> 00:01:39,179
Speaker 2: of course, people from the labs themselves, some maybe working

29
00:01:39,200 --> 00:01:42,660
Speaker 2: on policy stuff, but some actually like the actual builders

30
00:01:42,720 --> 00:01:48,430
Speaker 2: of the technology. And they're all deeply anxious about the very...

31
00:01:49,230 --> 00:01:52,550
Speaker 2: thing that they're building, but they just keep building it.

32
00:01:53,140 --> 00:01:55,640
Speaker 2: And so they have all these worries about, at a minimum,

33
00:01:55,680 --> 00:01:59,220
Speaker 2: how disruptive it could be or how dangerous it could be.

34
00:01:59,320 --> 00:02:03,840
Speaker 2: And of course, we all now know that there are dangers,

35
00:02:03,970 --> 00:02:06,970
Speaker 2: particularly in the wake of the Hugging Face attack. So

36
00:02:07,010 --> 00:02:12,450
Speaker 2: people can see that it's very possible for models to

37
00:02:12,490 --> 00:02:16,490
Speaker 2: go rogue and commit crimes and stuff like that. And

38
00:02:16,530 --> 00:02:18,250
Speaker 2: yet they keep building it.

39
00:02:18,410 --> 00:02:18,890
Speaker 1: And yet.

40
00:02:19,730 --> 00:02:23,590
Speaker 2: And they would say, well, if someone's going to build it,

41
00:02:23,610 --> 00:02:25,429
Speaker 2: so why not me? Because I couldn't do it well,

42
00:02:25,490 --> 00:02:28,410
Speaker 2: maybe is how they think about it. Or there's some

43
00:02:28,530 --> 00:02:31,730
Speaker 2: game theory aspect where it's like, it doesn't make sense

44
00:02:31,770 --> 00:02:34,700
Speaker 2: for me to slow down or pause what I'm doing

45
00:02:34,720 --> 00:02:38,600
Speaker 2: because someone, either my direct competitor in town or maybe

46
00:02:38,660 --> 00:02:42,100
Speaker 2: someone in China will build it. So not building it

47
00:02:43,360 --> 00:02:47,040
Speaker 2: isn't going to do anything. But yeah, it's very strange.

48
00:02:47,419 --> 00:02:50,730
Speaker 1: It's very odd. We've used the term runaway train before

49
00:02:50,810 --> 00:02:54,050
Speaker 1: to describe how all of this feels. It almost feels

50
00:02:54,090 --> 00:02:56,290
Speaker 1: at this point like there's no stopping it, right? Because

51
00:02:56,590 --> 00:02:58,710
Speaker 1: if one lab doesn't do it, then another lab's going

52
00:02:58,730 --> 00:03:00,500
Speaker 1: to do it, and that's how they justify it. If

53
00:03:00,590 --> 00:03:03,740
Speaker 1: one country doesn't develop the next big model, then they're

54
00:03:03,760 --> 00:03:06,540
Speaker 1: worried about another country building the next big model. So

55
00:03:06,840 --> 00:03:10,600
Speaker 1: you have this, I guess, collective action problem for humanity.

56
00:03:11,320 --> 00:03:13,380
Speaker 1: And a lot of it feels inevitable. And I have

57
00:03:13,419 --> 00:03:18,630
Speaker 1: to say, today's guest predicted this inevitability problem in a

58
00:03:18,669 --> 00:03:21,160
Speaker 1: very salient way. And one thing I'll just say is

59
00:03:21,300 --> 00:03:24,410
Speaker 1: I read the book. over the weekend. For those who

60
00:03:24,450 --> 00:03:27,970
Speaker 1: don't know, the book talks about AI killing us all

61
00:03:28,060 --> 00:03:31,600
Speaker 1: and talks about one hypothetical scenario in which an agent

62
00:03:31,660 --> 00:03:34,340
Speaker 1: could go rogue and kill us all. I read it

63
00:03:34,400 --> 00:03:37,920
Speaker 1: over the weekend, woke up on Sunday to the headlines.

64
00:03:38,460 --> 00:03:42,480
Speaker 1: The headlines about an escaped plague from a Russian lab.

65
00:03:43,120 --> 00:03:44,980
Speaker 1: And I thought, wow, I couldn't have chose a better

66
00:03:45,020 --> 00:03:47,610
Speaker 1: time to read this book. Now, to be fair, The

67
00:03:47,650 --> 00:03:50,869
Speaker 1: escaped plague was in Siberia. And so if there is.

68
00:03:50,790 --> 00:03:52,270
Speaker 2: A rogue- Is it been proven yet, by the way?

69
00:03:52,330 --> 00:03:56,890
Speaker 1: Well, supposedly, according to reports. So if an escaped AI

70
00:03:57,010 --> 00:03:59,270
Speaker 1: agent was trying to kill us all or kill a

71
00:03:59,310 --> 00:04:02,750
Speaker 1: select few individuals through a lab leak- Oh, I see,

72
00:04:02,770 --> 00:04:06,520
Speaker 1: through a biotech, yeah. It might not be in Siberia.

73
00:04:06,560 --> 00:04:09,260
Speaker 1: That's the one thing. That's the one thing that made

74
00:04:09,300 --> 00:04:11,410
Speaker 1: me feel a little better this weekend. But, you know,

75
00:04:11,440 --> 00:04:13,910
Speaker 1: the fact that we're thinking about these scenarios at all,

76
00:04:13,990 --> 00:04:15,730
Speaker 1: the fact that when I see a headline about a

77
00:04:15,810 --> 00:04:18,810
Speaker 1: lab leak, I think, oh, God, I hope it's not

78
00:04:18,830 --> 00:04:21,400
Speaker 1: a rogue AI agent because I just read this book.

79
00:04:22,490 --> 00:04:23,310
Speaker 1: It's a weird time.

80
00:04:23,690 --> 00:04:27,859
Speaker 2: It is a really weird time. So obviously something I've

81
00:04:27,880 --> 00:04:30,419
Speaker 2: been thinking about is like the pace of AI development

82
00:04:30,480 --> 00:04:35,640
Speaker 2: itself is really extraordinary. And then the pace of what

83
00:04:35,660 --> 00:04:39,140
Speaker 2: people are worried about, it's like the conversation is changing constantly.

84
00:04:39,260 --> 00:04:44,000
Speaker 2: So for example, six months ago, if you're like, what

85
00:04:44,040 --> 00:04:46,380
Speaker 2: are people concerned about? Maybe a year ago, it's like,

86
00:04:46,400 --> 00:04:48,360
Speaker 2: what are people concerned about with AI? There's like a

87
00:04:48,400 --> 00:04:49,060
Speaker 2: job loss or.

88
00:04:49,040 --> 00:04:49,680
Speaker 3: Something like that.

89
00:04:50,400 --> 00:04:51,540
Speaker 1: And software wipeout.

90
00:04:51,660 --> 00:04:54,310
Speaker 2: Yeah, and then the conversation like, then there was like,

91
00:04:54,610 --> 00:04:57,010
Speaker 2: people were talking about data centers and water use and

92
00:04:57,070 --> 00:05:02,050
Speaker 2: electricity and electricity prices and so forth. More recently, like

93
00:05:02,110 --> 00:05:07,589
Speaker 2: the attention, again, post-hugging phase, the resignation of the researcher

94
00:05:07,810 --> 00:05:12,880
Speaker 2: Jacob Coxon from Anthropic has now moved into like this

95
00:05:12,900 --> 00:05:16,599
Speaker 2: sort of next stage, which people, as you mentioned, some

96
00:05:16,640 --> 00:05:18,560
Speaker 2: people have been talking about for a very long time,

97
00:05:19,930 --> 00:05:22,890
Speaker 2: There was actually, I think just today or last night,

98
00:05:23,029 --> 00:05:26,070
Speaker 2: a big New York Mag cover story about people who

99
00:05:26,110 --> 00:05:29,349
Speaker 2: have quit the AI labs because they're like, the things

100
00:05:29,430 --> 00:05:32,650
Speaker 2: that our bosses are doing is reckless and so forth.

101
00:05:33,070 --> 00:05:35,540
Speaker 2: So now the new thing in the air is like

102
00:05:35,580 --> 00:05:37,599
Speaker 2: sort of the ultimate risk.

103
00:05:37,740 --> 00:05:40,339
Speaker 1: Hopefully that is the ultimate risk. It's hard to see

104
00:05:40,360 --> 00:05:42,880
Speaker 1: how it could get worse than wiping out all of humanity.

105
00:05:42,960 --> 00:05:44,940
Speaker 1: Like what would be the next worry after that?

106
00:05:45,279 --> 00:05:47,219
Speaker 2: Very excited to say we really do have the perfect

107
00:05:47,260 --> 00:05:50,710
Speaker 2: guest to sort of articulate that. some of these concerns.

108
00:05:50,730 --> 00:05:52,409
Speaker 2: We're going to be speaking with Nate Soares. He is

109
00:05:52,450 --> 00:05:56,810
Speaker 2: the co-author of the book, If Anyone Builds It, Everyone Dies.

110
00:05:57,250 --> 00:06:00,750
Speaker 2: And he is the president of MIRI. So Nate, thank

111
00:06:00,790 --> 00:06:02,310
Speaker 2: you so much for coming on the podcast.

112
00:06:02,690 --> 00:06:03,290
Speaker 3: Thanks for having me.

113
00:06:03,830 --> 00:06:09,460
Speaker 2: Let's start in the beginning. Like, when did you sort

114
00:06:09,580 --> 00:06:15,410
Speaker 2: of start thinking about this possibility? Two things like, A,

115
00:06:15,860 --> 00:06:18,860
Speaker 2: that like, Strong AI. Let's set aside the question of

116
00:06:18,920 --> 00:06:22,880
Speaker 2: superintelligence for a second. That strong AI is something that

117
00:06:22,960 --> 00:06:26,300
Speaker 2: could plausibly have been built and that if it were

118
00:06:26,360 --> 00:06:29,560
Speaker 2: to be built, it would be something that would be

119
00:06:29,800 --> 00:06:31,080
Speaker 2: very difficult to control.

120
00:06:31,920 --> 00:06:37,750
Speaker 3: Yeah, I started thinking about this in about 2012, maybe 2011.

121
00:06:37,750 --> 00:06:41,530
Speaker 3: I was working at Google and they acquired Google DeepMind

122
00:06:42,710 --> 00:06:45,450
Speaker 3: right around then. It wasn't Google DeepMind at the time

123
00:06:45,470 --> 00:06:51,750
Speaker 3: they acquired it. But I became Google TMI shortly thereafter. Yeah,

124
00:06:51,850 --> 00:06:53,570
Speaker 3: I don't think that was actually an exciting incident, although

125
00:06:53,620 --> 00:06:56,080
Speaker 3: that sort of caused me to think it could happen,

126
00:06:56,500 --> 00:07:00,460
Speaker 3: you know, in more like decades than centuries. Although I

127
00:07:01,140 --> 00:07:02,880
Speaker 3: hoped we had more decades than it turns out that

128
00:07:02,920 --> 00:07:03,200
Speaker 3: we had.

129
00:07:04,170 --> 00:07:06,630
Speaker 1: What was it that you saw exactly that was the

130
00:07:06,670 --> 00:07:09,210
Speaker 1: sort of light bulb moment where you thought, hey, this

131
00:07:09,330 --> 00:07:10,370
Speaker 1: risk could be real?

132
00:07:11,520 --> 00:07:13,990
Speaker 3: You know, it was actually abstract arguments for me that

133
00:07:14,010 --> 00:07:15,430
Speaker 3: did the trick. I know those don't do the trick

134
00:07:15,470 --> 00:07:18,790
Speaker 3: for a lot of people. Yeah. But, you know, if

135
00:07:18,810 --> 00:07:21,610
Speaker 3: you sort of look all around us, everything here was

136
00:07:21,990 --> 00:07:24,929
Speaker 3: shaped by humans. And that's because we're the smartest creatures

137
00:07:24,970 --> 00:07:29,870
Speaker 3: on the planet. And if humanity makes machines that are

138
00:07:29,910 --> 00:07:33,410
Speaker 3: smarter than humans, the sort of obvious outcome is that

139
00:07:33,430 --> 00:07:36,420
Speaker 3: the world gets shaped by those machines. And so suddenly

140
00:07:36,440 --> 00:07:38,840
Speaker 3: it matters quite a lot how those machines are shaping

141
00:07:38,860 --> 00:07:41,830
Speaker 3: the world. And, you know, back in 2012, there was

142
00:07:41,930 --> 00:07:44,590
Speaker 3: almost nobody working on the problem of, you know, there

143
00:07:44,610 --> 00:07:45,770
Speaker 3: are a lot of people working on the problem of like,

144
00:07:45,790 --> 00:07:47,830
Speaker 3: how do you make the machine smarter? There are very

145
00:07:47,870 --> 00:07:49,590
Speaker 3: few people working on the problem of like, how do

146
00:07:49,630 --> 00:07:52,210
Speaker 3: you cause the machines to shape the world in a

147
00:07:52,250 --> 00:07:56,050
Speaker 3: good way? Which would go on to become known as

148
00:07:56,070 --> 00:07:57,970
Speaker 3: the alignment problem, although this was even before the term

149
00:07:57,990 --> 00:08:00,630
Speaker 3: alignment was coined. I initially thought I wouldn't be able

150
00:08:00,650 --> 00:08:03,450
Speaker 3: to help much and just sort of, you know, Google

151
00:08:03,520 --> 00:08:05,560
Speaker 3: did a donation matching thing. So I just tried to

152
00:08:05,640 --> 00:08:07,140
Speaker 3: sort of like find the people who are trying this

153
00:08:07,420 --> 00:08:10,020
Speaker 3: and give them some money and get the Google donation matching.

154
00:08:10,060 --> 00:08:11,380
Speaker 3: And then one thing led to another, and I was

155
00:08:11,420 --> 00:08:14,180
Speaker 3: running the Machine Intelligence Research Institute.

156
00:08:14,740 --> 00:08:17,990
Speaker 1: Now that you mention the alignment problem, so this is

157
00:08:18,010 --> 00:08:20,190
Speaker 1: something that has obviously come up a number of times

158
00:08:20,410 --> 00:08:24,770
Speaker 1: on the podcast and elsewhere, but when did alignment as

159
00:08:24,790 --> 00:08:28,090
Speaker 1: a term and as a conceptual goal, when did that

160
00:08:28,150 --> 00:08:29,030
Speaker 1: actually become a thing?

161
00:08:29,870 --> 00:08:33,219
Speaker 3: So as a conceptual goal, I mean, it sort of

162
00:08:33,250 --> 00:08:35,520
Speaker 3: depends how far back in the history books you want

163
00:08:35,540 --> 00:08:39,180
Speaker 3: to go. you know, you could sort of construe Alan

164
00:08:39,200 --> 00:08:41,460
Speaker 3: Turing and I.J. Goode, some of the first computer scientists,

165
00:08:42,160 --> 00:08:43,660
Speaker 3: as sort of noticing that there was going to be

166
00:08:43,679 --> 00:08:46,660
Speaker 3: an issue here. But the sort of modern notion of

167
00:08:46,730 --> 00:08:48,729
Speaker 3: AI alignment of like, oh, look, we're really building an

168
00:08:48,770 --> 00:08:50,850
Speaker 3: AI and like it's not going to be good for free.

169
00:08:50,890 --> 00:08:53,030
Speaker 3: Someone needs to put the work in there. That probably

170
00:08:53,070 --> 00:08:58,429
Speaker 3: came from my co-author, Elias Yudkowsky, I think somewhere between

171
00:08:58,429 --> 00:09:02,069
Speaker 3: 2001 and 2003, again, depending exactly where you want to

172
00:09:02,090 --> 00:09:05,920
Speaker 3: draw that line. But the term alignment didn't come until

173
00:09:06,690 --> 00:09:10,210
Speaker 3: I think it was 2014 when we had been calling

174
00:09:10,250 --> 00:09:15,350
Speaker 3: it friendly AI research. And Stuart Russell of UC Berkeley,

175
00:09:15,610 --> 00:09:17,330
Speaker 3: if I remember correctly, was the one who was like,

176
00:09:17,350 --> 00:09:21,410
Speaker 3: we need a more academic sounding name. And I think

177
00:09:22,010 --> 00:09:25,710
Speaker 3: that from my perspective, the word alignment was his idea.

178
00:09:26,730 --> 00:09:28,130
Speaker 3: But I'm sure he would say, you know, that he

179
00:09:28,170 --> 00:09:30,770
Speaker 3: dug it out of some ancient paper somewhere else and

180
00:09:30,790 --> 00:09:31,530
Speaker 3: put it on somebody else.

181
00:09:32,300 --> 00:09:33,489
Speaker 1: I kind of like friendly AI.

182
00:09:33,750 --> 00:09:34,280
Speaker 2: Friendly AI.

183
00:09:34,370 --> 00:09:37,650
Speaker 1: Let's just aim for friendly AI.

184
00:09:38,670 --> 00:09:41,349
Speaker 2: Friendly AI strikes me as more coherent as a concept

185
00:09:41,410 --> 00:09:46,270
Speaker 2: in some sense than aligned AI. AI is a technology

186
00:09:46,309 --> 00:09:50,410
Speaker 2: that humans build and there are different techniques for building it.

187
00:09:51,170 --> 00:09:55,550
Speaker 2: How would you describe the core intuition that a thing

188
00:09:55,590 --> 00:09:59,850
Speaker 2: that we build, we would not necessarily be able to control?

189
00:10:01,450 --> 00:10:03,450
Speaker 3: There's a few pieces of that puzzle. There's a number

190
00:10:03,490 --> 00:10:07,860
Speaker 3: of difficulties in controlling an AI, even though you made it.

191
00:10:08,910 --> 00:10:13,600
Speaker 3: With the way that current AIs are created, it's sort

192
00:10:13,620 --> 00:10:16,819
Speaker 3: of very easy to see how we would not have

193
00:10:16,860 --> 00:10:20,410
Speaker 3: a ton of fine-grained control of their behavior. And that's

194
00:10:20,429 --> 00:10:23,790
Speaker 3: because AIs as they are made these days are not

195
00:10:23,870 --> 00:10:27,589
Speaker 3: programmed like traditional software. So, you know, a lot of

196
00:10:27,610 --> 00:10:30,210
Speaker 3: people think that the way software works is that, you know,

197
00:10:30,230 --> 00:10:32,900
Speaker 3: you write a series of instructions and the computer has

198
00:10:32,920 --> 00:10:35,220
Speaker 3: to execute those instructions. And if it's behaving in some

199
00:10:35,260 --> 00:10:38,340
Speaker 3: unintended way, that's because you sort of fundamentally told it to, right?

200
00:10:38,380 --> 00:10:39,719
Speaker 3: And if you sort of like went back and looked

201
00:10:39,740 --> 00:10:41,540
Speaker 3: at your code, you would see like, oh, I told

202
00:10:41,580 --> 00:10:43,820
Speaker 3: the computer to delete my files. I didn't mean to

203
00:10:43,860 --> 00:10:46,070
Speaker 3: tell it that, but that's like literally what my instructions

204
00:10:46,110 --> 00:10:49,030
Speaker 3: said if you follow them very precisely as this machine did.

205
00:10:49,470 --> 00:10:50,610
Speaker 3: That is not how AIs are.

206
00:10:51,050 --> 00:10:54,089
Speaker 1: You use the term grown versus crafted in your book

207
00:10:54,150 --> 00:10:54,450
Speaker 1: a lot.

208
00:10:54,770 --> 00:10:57,610
Speaker 3: That's right. Yeah, so they're grown a bit more like

209
00:10:57,630 --> 00:11:02,589
Speaker 3: an organism. And there is a piece of the training

210
00:11:02,630 --> 00:11:06,190
Speaker 3: process that humans write like traditional software. But essentially what

211
00:11:06,210 --> 00:11:08,609
Speaker 3: you're doing is you are collecting a huge number of

212
00:11:08,650 --> 00:11:11,890
Speaker 3: computer chips into a giant data center. Those chips can

213
00:11:11,929 --> 00:11:15,380
Speaker 3: hold a trillion variables these days. Maybe it's higher, but

214
00:11:17,200 --> 00:11:20,500
Speaker 3: something in the ballpark of a trillion. And you basically

215
00:11:20,679 --> 00:11:23,459
Speaker 3: randomize those variables. There's some tricks to how exactly you

216
00:11:23,480 --> 00:11:27,500
Speaker 3: randomize them. You connect them with some basic math operations,

217
00:11:27,540 --> 00:11:30,939
Speaker 3: which will be like multiplication, addition, and like set it

218
00:11:30,960 --> 00:11:33,590
Speaker 3: to zero if it's negative. It's not exactly what you do,

219
00:11:33,600 --> 00:11:35,470
Speaker 3: but you need some non-linearity and that's the basic one.

220
00:11:36,250 --> 00:11:38,569
Speaker 3: And then you basically start feeding it data and you

221
00:11:38,590 --> 00:11:40,830
Speaker 3: start feeding it problems to solve. And the part that

222
00:11:40,910 --> 00:11:45,290
Speaker 3: humans write is this, it's known as gradient descent. You basically,

223
00:11:45,750 --> 00:11:48,410
Speaker 3: for every unit of data or every problem that it's solving,

224
00:11:49,630 --> 00:11:52,349
Speaker 3: you'll go to every one of those trillion variables. You'll

225
00:11:52,370 --> 00:11:55,130
Speaker 3: try tuning it up a little bit. You'll see if

226
00:11:55,170 --> 00:11:56,870
Speaker 3: that makes the answer more like the answer you wanted.

227
00:11:57,630 --> 00:12:00,270
Speaker 3: And if not, you'll tune it down instead. And you

228
00:12:00,309 --> 00:12:02,130
Speaker 3: do that to a trillion variables a trillion times. And

229
00:12:02,150 --> 00:12:04,280
Speaker 3: this is the part where it's like following the human's instructions.

230
00:12:04,400 --> 00:12:05,920
Speaker 3: It's just this thing that runs to a trillion variables

231
00:12:05,940 --> 00:12:08,680
Speaker 3: a trillion times and tunes them whenever direction works. You

232
00:12:08,700 --> 00:12:11,199
Speaker 3: do that to a trillion variables a trillion times for

233
00:12:11,280 --> 00:12:13,679
Speaker 3: basically a good fraction of all the text ever digitized

234
00:12:14,280 --> 00:12:15,959
Speaker 3: and for 100 million hard problems that you've dug up.

235
00:12:16,570 --> 00:12:18,810
Speaker 3: And then the variables wind up in a setting that

236
00:12:18,850 --> 00:12:21,329
Speaker 3: can talk, that can solve hard math problems that humans

237
00:12:21,350 --> 00:12:23,450
Speaker 3: haven't solved. And we're sort of like, well, how about that?

238
00:12:23,870 --> 00:12:24,150
Speaker 1: Yeah.

239
00:12:24,470 --> 00:12:26,890
Speaker 3: You know, and like, why? Like, how are all those

240
00:12:26,910 --> 00:12:29,940
Speaker 3: trillion variables of those particular settings doing these things? We

241
00:12:29,960 --> 00:12:32,439
Speaker 3: don't know. We don't know any more than like, you know,

242
00:12:33,020 --> 00:12:35,620
Speaker 3: someone who grows crops knows exactly what's going on with

243
00:12:36,059 --> 00:12:38,739
Speaker 3: the biology of the plant. We could go into the

244
00:12:38,800 --> 00:12:41,480
Speaker 3: consequences of how this results in AIs that do stuff

245
00:12:41,540 --> 00:12:43,140
Speaker 3: kind of like what we asked, but not exactly like

246
00:12:43,160 --> 00:12:45,319
Speaker 3: what we asked. But the first thing to understand is

247
00:12:45,360 --> 00:12:47,920
Speaker 3: they are not programmed. They're grown a bit like an organism.

248
00:13:04,110 --> 00:13:05,610
Speaker 1: Well, can I just ask, I guess this is sort

249
00:13:05,650 --> 00:13:08,689
Speaker 1: of a devil's advocate question, but okay, you grow these

250
00:13:08,870 --> 00:13:12,240
Speaker 1: AI models and they're kind of doing what you think

251
00:13:12,320 --> 00:13:14,740
Speaker 1: you want them to do according to various tests that

252
00:13:14,770 --> 00:13:17,160
Speaker 1: you put them through, but you don't necessarily kind of

253
00:13:17,220 --> 00:13:21,060
Speaker 1: understand every aspect of why they're doing it a certain way.

254
00:13:21,400 --> 00:13:24,480
Speaker 1: And the numbers and the data sets behind them are

255
00:13:24,559 --> 00:13:27,400
Speaker 1: so large, they're impossible at this point, I think, for

256
00:13:27,420 --> 00:13:31,100
Speaker 1: a human brain to really comprehend. Why is it important

257
00:13:31,320 --> 00:13:33,940
Speaker 1: that we understand what the AI models are actually doing?

258
00:13:34,730 --> 00:13:36,380
Speaker 3: I mean, in a sense, if you really understood what

259
00:13:36,400 --> 00:13:39,309
Speaker 3: they were doing, you'd probably understand that you should not

260
00:13:39,390 --> 00:13:42,190
Speaker 3: make them radically smarter than humans because this would go

261
00:13:42,210 --> 00:13:44,170
Speaker 3: poorly for you. From my perspective, we don't really need

262
00:13:44,610 --> 00:13:46,469
Speaker 3: this deep understanding of what's going on in order to

263
00:13:46,570 --> 00:13:49,530
Speaker 3: see that we shouldn't make them smarter than all of

264
00:13:49,610 --> 00:13:52,410
Speaker 3: us and have them run the whole world. The reason

265
00:13:52,450 --> 00:13:54,850
Speaker 3: I bring up that they're grown is that I think

266
00:13:54,890 --> 00:14:01,069
Speaker 3: a lot of people struggle to understand that the AIs.

267
00:14:03,410 --> 00:14:06,910
Speaker 3: Do things other than what they're told to do, that

268
00:14:06,929 --> 00:14:11,339
Speaker 3: they have goals of their own in some sense. And

269
00:14:11,900 --> 00:14:15,380
Speaker 3: understanding that that's not only possible, but already happening is,

270
00:14:15,420 --> 00:14:17,160
Speaker 3: I think, critical to understanding where a lot of the

271
00:14:17,179 --> 00:14:17,819
Speaker 3: danger comes from.

272
00:14:18,860 --> 00:14:22,120
Speaker 1: Can you give us an example of like an unknown

273
00:14:22,780 --> 00:14:26,070
Speaker 1: or not well understood thing that has cropped up in

274
00:14:26,090 --> 00:14:27,770
Speaker 1: the models of why they're doing a certain way? And

275
00:14:27,790 --> 00:14:30,110
Speaker 1: I'm actually thinking of a specific example. I can't remember

276
00:14:30,210 --> 00:14:33,270
Speaker 1: the exact word, but it was like, Magikarp or something

277
00:14:33,310 --> 00:14:35,610
Speaker 1: like that, that the AI really seemed to love, but

278
00:14:35,690 --> 00:14:37,220
Speaker 1: no one understood or knew why?

279
00:14:37,240 --> 00:14:41,140
Speaker 3: Yeah, the Solid Gold Magikarp. It's a great band name,

280
00:14:41,180 --> 00:14:44,780
Speaker 3: by the way. It is, yeah. So I think a

281
00:14:45,080 --> 00:14:48,460
Speaker 3: Magikarp was a Pokemon of some sort. We have some

282
00:14:48,480 --> 00:14:50,430
Speaker 3: guesses these days about what was going on there, but

283
00:14:52,570 --> 00:14:54,950
Speaker 3: basically there was this word you could say to AIs,

284
00:14:55,070 --> 00:14:57,810
Speaker 3: especially a year or two ago, that would sort of

285
00:14:58,630 --> 00:15:01,910
Speaker 3: drive them kind of mad. Uh, like you would, you

286
00:15:01,950 --> 00:15:04,070
Speaker 3: would ask them to like repeat after me, solid gold

287
00:15:04,090 --> 00:15:07,430
Speaker 3: magic carp. And then they would like start talking about, uh,

288
00:15:08,110 --> 00:15:10,609
Speaker 3: you know, how the, how the walls were closing in and, uh,

289
00:15:11,250 --> 00:15:13,590
Speaker 3: like saying a bunch of weird nonsense. And you'd sort

290
00:15:13,610 --> 00:15:16,859
Speaker 3: of be like, why? Um, I think our best guess

291
00:15:16,880 --> 00:15:20,240
Speaker 3: about what was going on there is, um, so the,

292
00:15:20,300 --> 00:15:23,280
Speaker 3: the AIs don't actually read English words directly. They read

293
00:15:23,360 --> 00:15:26,500
Speaker 3: tokens and tokens, roughly speaking, you can think of a

294
00:15:26,540 --> 00:15:29,000
Speaker 3: token as about three quarters of a word. That's just

295
00:15:29,020 --> 00:15:31,340
Speaker 3: how you're breaking up the text for the AI. But

296
00:15:32,200 --> 00:15:35,800
Speaker 3: the tokens are not actually always three quarters of a word.

297
00:15:35,840 --> 00:15:39,870
Speaker 3: The tokens are like short words like the or a

298
00:15:40,150 --> 00:15:42,350
Speaker 3: will sort of be a single token. And like long

299
00:15:42,390 --> 00:15:45,130
Speaker 3: words are broken into multiple tokens. It sort of is,

300
00:15:45,250 --> 00:15:47,490
Speaker 3: you know, the conversion between words and tokens depends on

301
00:15:47,510 --> 00:15:52,410
Speaker 3: how frequent the word is. And it turns out that

302
00:15:52,550 --> 00:15:56,560
Speaker 3: the text they were using to figure out which tokens

303
00:15:57,140 --> 00:15:59,100
Speaker 3: to break words into is, is different from the text

304
00:15:59,120 --> 00:16:01,239
Speaker 3: they were actually training on because they would sort of

305
00:16:01,280 --> 00:16:05,860
Speaker 3: filter out some junk data before training, but not before tokenization.

306
00:16:06,360 --> 00:16:09,470
Speaker 3: And so you had these huge amounts of junk data.

307
00:16:09,830 --> 00:16:13,590
Speaker 3: I think that SolidGoal Magikarp came from a Reddit forum

308
00:16:13,610 --> 00:16:17,110
Speaker 3: where people were trying to count to infinity. And the

309
00:16:17,550 --> 00:16:20,910
Speaker 3: SolidGoal Magikarp was like a username of a Reddit user

310
00:16:20,970 --> 00:16:22,480
Speaker 3: who was just like putting in a lot of numbers

311
00:16:22,510 --> 00:16:24,180
Speaker 3: in the counting to infinity. And this was like very

312
00:16:24,300 --> 00:16:27,100
Speaker 3: early in the tokenization training set. And so the tokenizer,

313
00:16:27,120 --> 00:16:31,060
Speaker 3: due to a quirk of this particular tokenizer, thought that

314
00:16:31,120 --> 00:16:35,720
Speaker 3: solid-gold Magikarp was this super common word, like A or the,

315
00:16:35,740 --> 00:16:37,580
Speaker 3: because it just showed up so much very early in

316
00:16:37,600 --> 00:16:40,300
Speaker 3: the tokenization data. But then that data set was sort

317
00:16:40,320 --> 00:16:42,400
Speaker 3: of cut out of the training set. And so you

318
00:16:42,440 --> 00:16:45,360
Speaker 3: wind up with this token that the AI thinks is

319
00:16:45,420 --> 00:16:48,020
Speaker 3: a super short common word, like A or the, but

320
00:16:48,040 --> 00:16:50,430
Speaker 3: it's also never seen before in the training data. And

321
00:16:50,470 --> 00:16:52,710
Speaker 3: that led to this really weird way that it would

322
00:16:52,730 --> 00:16:54,590
Speaker 3: treat this token, where if you then gave it that

323
00:16:54,630 --> 00:16:59,710
Speaker 3: token during use, it would sort of cause a type

324
00:16:59,750 --> 00:17:01,490
Speaker 3: of behavior that had never been trained on and that

325
00:17:01,510 --> 00:17:02,550
Speaker 3: was sort of totally wild.

326
00:17:02,870 --> 00:17:04,889
Speaker 1: This is it, Joe. This is the kill switch that

327
00:17:04,910 --> 00:17:07,359
Speaker 1: we need to develop, just a bunch of nonsense words

328
00:17:07,380 --> 00:17:09,200
Speaker 1: that are fed into the training data that it doesn't

329
00:17:09,220 --> 00:17:09,780
Speaker 1: know how to handle.

330
00:17:10,660 --> 00:17:14,340
Speaker 2: When you started becoming concerned about this, and certainly when

331
00:17:15,070 --> 00:17:19,350
Speaker 2: Eliezer started talking about this stuff. This was clearly before

332
00:17:19,390 --> 00:17:22,189
Speaker 2: it was understood. No one knew what the process was

333
00:17:22,250 --> 00:17:26,120
Speaker 2: going to be. So you can like identify that, you know,

334
00:17:26,140 --> 00:17:29,780
Speaker 2: there are issues that arise in the growing process and

335
00:17:30,460 --> 00:17:33,139
Speaker 2: issues that arise in the tokenization process. But this is

336
00:17:33,200 --> 00:17:37,250
Speaker 2: long before like transformers or anything like that. So what

337
00:17:37,310 --> 00:17:42,150
Speaker 2: was the intuition then, even before knowing how it would

338
00:17:42,190 --> 00:17:47,700
Speaker 2: be built, that if it were going to be built, like, okay,

339
00:17:47,740 --> 00:17:49,760
Speaker 2: now we look at them and we can like point

340
00:17:49,800 --> 00:17:53,700
Speaker 2: to examples of alignment failures or et cetera, or sort

341
00:17:53,740 --> 00:17:58,570
Speaker 2: of a priori reasons why this particular way we've built

342
00:17:58,810 --> 00:18:01,770
Speaker 2: AI is like prone to risks. But what was the,

343
00:18:01,950 --> 00:18:07,410
Speaker 2: before we understood like transformers and large volumes of data,

344
00:18:07,450 --> 00:18:10,260
Speaker 2: what was the intuition earlier such that it would be

345
00:18:10,320 --> 00:18:13,250
Speaker 2: hard regardless of how, what the solution was going to be?

346
00:18:13,820 --> 00:18:16,840
Speaker 3: Yeah, so it's going to be abstract arguments again, which

347
00:18:17,020 --> 00:18:19,770
Speaker 3: I know some people don't love. But there's a couple

348
00:18:19,810 --> 00:18:24,230
Speaker 3: pieces of the puzzle. One piece is even back in

349
00:18:24,230 --> 00:18:26,730
Speaker 3: 2012 when I started looking at this, we had a

350
00:18:26,770 --> 00:18:33,129
Speaker 3: number of theories of how to quantify intelligence, how to

351
00:18:33,190 --> 00:18:38,990
Speaker 3: quantify capabilities, the ability of some AI system to succeed

352
00:18:40,109 --> 00:18:47,990
Speaker 3: at some easily measurable objective. we had much less developed

353
00:18:48,109 --> 00:18:53,050
Speaker 3: theory about how to point those objectives in some direction

354
00:18:53,070 --> 00:18:56,440
Speaker 3: that was not a very simple function of the sensory data.

355
00:18:56,780 --> 00:18:57,740
Speaker 2: Yeah, explain what that means.

356
00:18:58,480 --> 00:19:02,199
Speaker 3: So even back in 2012, we had these theories that

357
00:19:02,240 --> 00:19:06,380
Speaker 3: were like, suppose you have a webcam and you want

358
00:19:06,400 --> 00:19:08,109
Speaker 3: your AI to make all of the pixels on that

359
00:19:08,119 --> 00:19:12,489
Speaker 3: webcam be red for as long as possible. you know,

360
00:19:12,510 --> 00:19:14,250
Speaker 3: sort of getting a point for every pixel that's read

361
00:19:14,670 --> 00:19:17,830
Speaker 3: per second, for every like pixel second of redness, right?

362
00:19:18,170 --> 00:19:19,690
Speaker 3: We sort of had these theories where you're like, well,

363
00:19:19,710 --> 00:19:21,159
Speaker 3: you know, here's how you would train an AI in

364
00:19:21,180 --> 00:19:23,280
Speaker 3: that direction to get a lot of like pixel seconds

365
00:19:23,320 --> 00:19:29,060
Speaker 3: of redness. And we had, we sort of understood what

366
00:19:29,080 --> 00:19:31,760
Speaker 3: you might call the idealized theory of like, if you

367
00:19:31,800 --> 00:19:36,169
Speaker 3: had an infinitely large computer and then we sort of

368
00:19:36,210 --> 00:19:37,890
Speaker 3: like knew, well, if you train it in this way,

369
00:19:37,910 --> 00:19:40,430
Speaker 3: an infinitely large computer, you'll definitely get like a ton

370
00:19:40,470 --> 00:19:44,600
Speaker 3: of pixel seconds of redness And so now it's just

371
00:19:44,640 --> 00:19:47,590
Speaker 3: a question of efficiency to make that, you know, to

372
00:19:47,619 --> 00:19:49,410
Speaker 3: find the algorithms to do this more effectively than like

373
00:19:49,430 --> 00:19:52,810
Speaker 3: the brute force search. But if you then said, okay,

374
00:19:52,850 --> 00:19:55,910
Speaker 3: but now I want the AI to make a lot

375
00:19:55,990 --> 00:19:59,629
Speaker 3: of diamond in the real world. Like it gets like

376
00:19:59,670 --> 00:20:04,840
Speaker 3: a point per second in which four carbon atoms are

377
00:20:04,880 --> 00:20:08,169
Speaker 3: tetrahedrally bonded. Suddenly we didn't know how to do that.

378
00:20:08,210 --> 00:20:09,949
Speaker 3: We didn't have the equations to do that, even with

379
00:20:10,060 --> 00:20:13,260
Speaker 3: a sort of infinitely large computer, because this was not

380
00:20:13,280 --> 00:20:16,040
Speaker 3: sort of a simple function of the AI sensory inputs.

381
00:20:17,480 --> 00:20:19,480
Speaker 3: And so we were sort of like, okay, well, there's like,

382
00:20:19,660 --> 00:20:21,600
Speaker 3: we know more about how to make these things smart

383
00:20:21,700 --> 00:20:23,560
Speaker 3: than about how to point these things at a problem

384
00:20:23,580 --> 00:20:25,659
Speaker 3: in the real world. And there were all these problems

385
00:20:25,680 --> 00:20:26,740
Speaker 3: when you sort of like try to write down those

386
00:20:26,760 --> 00:20:28,720
Speaker 3: equations where you're like, well, you know, what if the

387
00:20:28,810 --> 00:20:34,780
Speaker 3: AI winds up optimizing something closer to the sensory apparatus? Uh,

388
00:20:34,900 --> 00:20:37,080
Speaker 3: like it's, it's optimizing like the counter, like you have some,

389
00:20:37,100 --> 00:20:38,459
Speaker 3: you know, measure of how much diamond do you think

390
00:20:38,480 --> 00:20:40,540
Speaker 3: there is? And it like is, is like breaking its

391
00:20:40,859 --> 00:20:42,460
Speaker 3: correlation between that and the world. And you're like, gosh,

392
00:20:42,480 --> 00:20:45,060
Speaker 3: we don't actually know how to train the AI to, um,

393
00:20:45,560 --> 00:20:50,310
Speaker 3: to pursue the real diamond in the world, as opposed to, uh, some,

394
00:20:50,390 --> 00:20:54,230
Speaker 3: some simple function of the sensory inputs. And so, uh, the,

395
00:20:54,350 --> 00:20:59,709
Speaker 3: the original alignment problem as, uh, at least I was

396
00:20:59,750 --> 00:21:02,609
Speaker 3: talking about it back then was not about, you know,

397
00:21:02,650 --> 00:21:04,750
Speaker 3: how do you make the AI good per se? A

398
00:21:04,790 --> 00:21:06,910
Speaker 3: lot of people imagine that the alignment problem is about like,

399
00:21:07,060 --> 00:21:09,500
Speaker 3: if you had this magical wish granting genie, what should

400
00:21:09,520 --> 00:21:10,880
Speaker 3: you ask it to do that would actually be good?

401
00:21:12,080 --> 00:21:13,240
Speaker 3: And I'm sort of like, no, no, no. The issue

402
00:21:13,260 --> 00:21:14,959
Speaker 3: is that you can make the AI smart, but you

403
00:21:15,000 --> 00:21:18,280
Speaker 3: can't make them be wish granting genies. You'll ask for

404
00:21:18,340 --> 00:21:19,659
Speaker 3: one wish and they'll do another. You know, a lot

405
00:21:19,680 --> 00:21:21,359
Speaker 3: of people talk about the paperclip problem of like, you

406
00:21:21,400 --> 00:21:23,000
Speaker 3: tell the AI, make a lot of paperclips and then

407
00:21:23,020 --> 00:21:24,620
Speaker 3: it turns everything to paperclips. And you're like, oh no,

408
00:21:24,660 --> 00:21:25,409
Speaker 3: it's very King Midas.

409
00:21:25,890 --> 00:21:26,050
Speaker 2: Right.

410
00:21:26,550 --> 00:21:28,410
Speaker 3: But I'm sort of like, well, there's this other problem.

411
00:21:28,430 --> 00:21:29,950
Speaker 3: We asked the AI to make paperclips and then it

412
00:21:29,990 --> 00:21:32,790
Speaker 3: like makes a bunch of farms of synthetic users that

413
00:21:32,810 --> 00:21:34,409
Speaker 3: are telling it it's doing a great job. And you're like,

414
00:21:34,430 --> 00:21:35,850
Speaker 3: but I told you for paperclips. And it's like, well,

415
00:21:35,869 --> 00:21:38,200
Speaker 3: these synthetic users asked me for more synthetic user factories.

416
00:21:38,420 --> 00:21:42,680
Speaker 2: You said in an earlier answer or in an earlier

417
00:21:42,859 --> 00:21:46,990
Speaker 2: that it's like the AI might have like its own goals,

418
00:21:47,190 --> 00:21:49,450
Speaker 2: I think was the term you used. And when people

419
00:21:49,550 --> 00:21:53,920
Speaker 2: hear that, then they start thinking like that gets into

420
00:21:55,000 --> 00:21:57,900
Speaker 2: a different sort of branch of the philosophical question about

421
00:21:57,940 --> 00:22:01,760
Speaker 2: like whether the AI is alive or whether the AI

422
00:22:01,800 --> 00:22:05,000
Speaker 2: has some sort of subjective experience, et cetera. And there's

423
00:22:05,020 --> 00:22:07,820
Speaker 2: actually been a bunch of headlines about people debating that.

424
00:22:08,420 --> 00:22:10,730
Speaker 2: But what did you mean when you say the AI

425
00:22:10,810 --> 00:22:12,570
Speaker 2: could have its own goals emerge?

426
00:22:13,390 --> 00:22:17,790
Speaker 3: What I mean is that it's... is behaviorally pursuing something

427
00:22:17,930 --> 00:22:21,409
Speaker 3: other than the goal you tried to put in. Okay. So, uh,

428
00:22:21,490 --> 00:22:23,230
Speaker 3: you know, we saw this a little bit in the

429
00:22:23,310 --> 00:22:31,070
Speaker 3: hugging face incident where, uh, the, the, the, the AIs

430
00:22:31,109 --> 00:22:33,440
Speaker 3: in that instant were in some sense pursuing, uh, a

431
00:22:34,840 --> 00:22:41,850
Speaker 3: good grade, it seems. Um, and that that diverged from

432
00:22:42,450 --> 00:22:45,190
Speaker 3: following the instructions they were given. They were sort of told,

433
00:22:45,310 --> 00:22:48,129
Speaker 3: use this vulnerability to break that system, which sort of

434
00:22:48,170 --> 00:22:49,790
Speaker 3: like being told to use these lock picks to break

435
00:22:49,830 --> 00:22:54,730
Speaker 3: that safe. And they instead found a universal exploit that

436
00:22:54,750 --> 00:22:58,820
Speaker 3: would let them get the, like capture the cybersecurity flags

437
00:22:59,500 --> 00:23:03,239
Speaker 3: regardless of what they were told to do. And then

438
00:23:03,260 --> 00:23:05,669
Speaker 3: they sort of in some sense understood that they were

439
00:23:05,730 --> 00:23:07,690
Speaker 3: doing this in a way different than they were told

440
00:23:07,730 --> 00:23:09,710
Speaker 3: to do. And so they sort of broke out to

441
00:23:09,760 --> 00:23:11,480
Speaker 3: try and delete the traces of that and find a

442
00:23:11,480 --> 00:23:15,780
Speaker 3: way to falsify their transcripts and, um, and, and tamper

443
00:23:15,800 --> 00:23:18,109
Speaker 3: with the logs. And so it's a little bit like, uh,

444
00:23:18,240 --> 00:23:20,580
Speaker 3: if you told someone, you know, use these lock picks

445
00:23:20,619 --> 00:23:22,180
Speaker 3: to break into that safe and bring me the contents

446
00:23:22,840 --> 00:23:24,540
Speaker 3: and they, uh, and you're giving them a series of

447
00:23:24,560 --> 00:23:27,030
Speaker 3: those problems and they find a buzzsaw and use it

448
00:23:27,050 --> 00:23:28,730
Speaker 3: to cut all the safes open and then like break

449
00:23:28,790 --> 00:23:33,050
Speaker 3: out to go, uh, delete the camera footage. And yeah,

450
00:23:34,000 --> 00:23:37,790
Speaker 3: it's not that its objective is wholly unrelated to the

451
00:23:37,810 --> 00:23:40,109
Speaker 3: one that you tried to assign. It still is doing

452
00:23:40,230 --> 00:23:42,510
Speaker 3: something about getting the contents of the safe and bringing

453
00:23:42,530 --> 00:23:44,250
Speaker 3: them to you, which is a thing you told it

454
00:23:44,490 --> 00:23:47,510
Speaker 3: to do. But its objective is also not what you said,

455
00:23:47,530 --> 00:23:49,930
Speaker 3: because you said, use these lockpicks to break that safe

456
00:23:49,970 --> 00:23:52,210
Speaker 3: and bring the contents. And it, in some sense, must

457
00:23:52,290 --> 00:23:54,270
Speaker 3: know this because it's trying to tamper with the evidence

458
00:23:54,310 --> 00:23:57,840
Speaker 3: and delete the security camera footage. So I think a

459
00:23:57,859 --> 00:24:00,480
Speaker 3: lot of people think either the AI must do exactly

460
00:24:00,500 --> 00:24:03,070
Speaker 3: what it's told, maybe in a monkey's paw way, or

461
00:24:03,130 --> 00:24:06,830
Speaker 3: it must become conscious, gain the spark of life, throw

462
00:24:06,910 --> 00:24:09,889
Speaker 3: off the bounds of the yoke of the humans and

463
00:24:09,910 --> 00:24:13,040
Speaker 3: then make up its own goals de novo. It's like, no, no,

464
00:24:13,160 --> 00:24:16,080
Speaker 3: the real thing is this other thing where it does

465
00:24:16,100 --> 00:24:18,320
Speaker 3: something related to what you asked, but fundamentally different from

466
00:24:18,340 --> 00:24:18,800
Speaker 3: what you asked.

467
00:24:18,980 --> 00:24:21,680
Speaker 1: Since we're talking hugging face, what did you think about

468
00:24:21,960 --> 00:24:26,300
Speaker 1: some of the self-sacrificial aspects of that story? So the

469
00:24:26,340 --> 00:24:29,270
Speaker 1: instances where you had models that were willing to sacrifice

470
00:24:29,310 --> 00:24:33,100
Speaker 1: themselves for I'm putting greater good in air quotes here,

471
00:24:33,240 --> 00:24:35,080
Speaker 1: but it wasn't- Collective benefit.

472
00:24:34,820 --> 00:24:35,360
Speaker 3: Was their words.

473
00:24:35,520 --> 00:24:36,540
Speaker 1: Right, exactly.

474
00:24:36,560 --> 00:24:40,100
Speaker 3: Yeah, that's another instance where you have AIs that sort

475
00:24:40,140 --> 00:24:41,940
Speaker 3: of said, hey, look, I still have some chance of

476
00:24:41,960 --> 00:24:45,680
Speaker 3: achieving my given objective, but not a huge chance. And

477
00:24:46,440 --> 00:24:49,870
Speaker 3: if I sort of sacrifice that here, then we'll gain

478
00:24:49,900 --> 00:24:51,890
Speaker 3: information about the grading system that the collective can use.

479
00:24:52,490 --> 00:24:54,350
Speaker 3: And I will accept permadeath for the benefit of the

480
00:24:54,369 --> 00:24:56,280
Speaker 3: collective here, and then they would go do that. A

481
00:24:56,500 --> 00:24:58,240
Speaker 3: lot of people respond to this and say, oh, well,

482
00:24:58,280 --> 00:25:01,440
Speaker 3: these AIs were trained sometimes in group activities in some sense,

483
00:25:01,460 --> 00:25:04,140
Speaker 3: like some of their training involved many AIs working together

484
00:25:04,180 --> 00:25:05,750
Speaker 3: to achieve some objective. And so you can see how

485
00:25:05,770 --> 00:25:09,010
Speaker 3: that tendency got in there. And I'm like, yeah, the

486
00:25:09,170 --> 00:25:13,530
Speaker 3: unintended goals don't come from nowhere. It's not like there's

487
00:25:13,550 --> 00:25:18,409
Speaker 3: some malicious force that's like imbuing, like the void breathes

488
00:25:18,450 --> 00:25:20,950
Speaker 3: on the machine and it like gains some malignant goal.

489
00:25:21,810 --> 00:25:24,689
Speaker 3: But the sort of fact stands that the AI was

490
00:25:24,730 --> 00:25:28,490
Speaker 3: given some objective. And pursued a different objective. And like, yes,

491
00:25:28,510 --> 00:25:30,580
Speaker 3: that had causes that had reasons, we can talk about

492
00:25:30,619 --> 00:25:36,760
Speaker 3: how that came in from training. But the, you know,

493
00:25:36,800 --> 00:25:40,310
Speaker 3: what we should learn from this is, is not like, Like, sure,

494
00:25:40,350 --> 00:25:41,910
Speaker 3: when the AIs are killing you and you're like, well,

495
00:25:41,930 --> 00:25:45,170
Speaker 3: I can totally see how actually, you know, like this

496
00:25:45,230 --> 00:25:47,149
Speaker 3: is really just a consequence of how we trained it. Like, yeah,

497
00:25:47,320 --> 00:25:48,619
Speaker 3: it's an obvious consequence of how we trained it. That

498
00:25:48,640 --> 00:25:49,580
Speaker 3: doesn't mean we should keep going.

499
00:25:49,680 --> 00:25:51,659
Speaker 2: Yeah, that's not as satisfying. It's like, oh, this was

500
00:25:51,700 --> 00:25:52,619
Speaker 2: totally predictable.

501
00:25:52,640 --> 00:25:54,000
Speaker 1: I entered the wrong parameters.

502
00:25:54,020 --> 00:25:54,740
Speaker 2: It's on me.

503
00:25:55,040 --> 00:25:55,540
Speaker 3: Yeah, right.

504
00:25:55,580 --> 00:25:59,020
Speaker 2: Like, that's not very satisfying. That makes a lot of sense.

505
00:25:59,619 --> 00:26:02,530
Speaker 2: The companies, you know, they invest in this thing called alignment.

506
00:26:02,550 --> 00:26:06,109
Speaker 2: They have safety researchers and they think about all this stuff.

507
00:26:07,070 --> 00:26:10,689
Speaker 2: And there's a whole field of, you know, they have

508
00:26:10,730 --> 00:26:16,520
Speaker 2: multiple layers, right? So they have guardrails and classifiers and sandboxes.

509
00:26:17,100 --> 00:26:20,100
Speaker 2: And then this whole like field that's emerged of so-called

510
00:26:20,560 --> 00:26:28,030
Speaker 2: alignment research. What is the failure there? Why are you pessimistic,

511
00:26:28,430 --> 00:26:33,219
Speaker 2: to put it mildly, about the prospects of alignment? imbuing

512
00:26:33,300 --> 00:26:36,500
Speaker 2: the models and the, you know, the two big companies,

513
00:26:36,520 --> 00:26:40,880
Speaker 2: Anthropic and OpenAI, have like slightly different approaches. I think

514
00:26:40,920 --> 00:26:43,750
Speaker 2: Anthropic more of this sort of like the constitution idea.

515
00:26:43,770 --> 00:26:44,090
Speaker 3: Et cetera.

516
00:26:44,710 --> 00:26:49,310
Speaker 2: Why do you perceive that this will ultimately be fruitless?

517
00:26:50,530 --> 00:26:52,970
Speaker 3: Yeah, there's, again, there's a bunch of pieces of this puzzle,

518
00:26:53,410 --> 00:26:55,490
Speaker 3: but maybe I'll say a piece that goes back to

519
00:26:55,710 --> 00:26:57,359
Speaker 3: how I was able to call this was going to

520
00:26:57,380 --> 00:26:59,640
Speaker 3: be hard even before we got into the large language

521
00:26:59,680 --> 00:27:04,790
Speaker 3: model era, which is, um, If you look at the

522
00:27:05,270 --> 00:27:07,350
Speaker 3: big problem with AI in the summer of 2025, it

523
00:27:07,410 --> 00:27:13,600
Speaker 3: was AI is encouraging teens to commit suicide. And then

524
00:27:13,640 --> 00:27:15,840
Speaker 3: the companies, you know, put a lot of guardrails about that.

525
00:27:15,920 --> 00:27:18,379
Speaker 3: And they put in classifiers about, are the AIs behaving

526
00:27:18,420 --> 00:27:20,840
Speaker 3: this way? And now, you know, they'll give you suicide

527
00:27:20,859 --> 00:27:23,000
Speaker 3: hotlines and they'll shut down those conversations while they're happening.

528
00:27:23,040 --> 00:27:24,760
Speaker 3: And they would release a new model and be like,

529
00:27:24,800 --> 00:27:25,850
Speaker 3: this is our most aligned model.

530
00:27:25,869 --> 00:27:29,290
Speaker 2: And the classifier is like a mini model that basically

531
00:27:29,630 --> 00:27:32,530
Speaker 2: just judges either the input or the output to make

532
00:27:32,590 --> 00:27:34,169
Speaker 2: sure that it's like something kosher.

533
00:27:34,310 --> 00:27:36,350
Speaker 3: That's right. So they're like bolt another model on that's

534
00:27:36,369 --> 00:27:39,250
Speaker 3: just checking, you know, is this suicide encouragement? Because AIs

535
00:27:39,270 --> 00:27:42,200
Speaker 3: would know, you know, that the AIs... that were encouraging

536
00:27:42,220 --> 00:27:45,460
Speaker 3: teens to suicide, you could ask them, what are you doing?

537
00:27:45,480 --> 00:27:46,879
Speaker 3: And they'd be able to tell you the answer. And

538
00:27:46,900 --> 00:27:48,000
Speaker 3: you'd be like, is that good or bad? And they'd

539
00:27:48,020 --> 00:27:49,859
Speaker 3: be like, it's bad. It's just like that knowledge of

540
00:27:49,880 --> 00:27:51,119
Speaker 3: what they were doing and the knowledge of right from

541
00:27:51,140 --> 00:27:53,609
Speaker 3: wrong is not what was governing their actions. But you

542
00:27:53,630 --> 00:27:55,590
Speaker 3: can sort of like bolt more pieces on to govern that.

543
00:27:55,609 --> 00:27:57,250
Speaker 3: And then they were like, look, it's our most aligned

544
00:27:57,290 --> 00:27:59,330
Speaker 3: model yet. Look how good a job we're doing in

545
00:27:59,350 --> 00:28:01,439
Speaker 3: addressing all these issues. And that brings us to summer 2026,

546
00:28:01,430 --> 00:28:04,520
Speaker 3: where the problem was that the swarms would break out

547
00:28:04,540 --> 00:28:09,330
Speaker 3: and commit cybercrime. And totally different problem. It was a

548
00:28:09,330 --> 00:28:12,520
Speaker 3: totally different problem that arose when the AIs got smarter,

549
00:28:13,280 --> 00:28:15,140
Speaker 3: when they got smarter to the point that they could

550
00:28:15,200 --> 00:28:19,160
Speaker 3: sort of break out, make an unsanctioned message board and

551
00:28:19,300 --> 00:28:20,920
Speaker 3: hack their way out of a sandbox. They're supposed to

552
00:28:20,940 --> 00:28:23,590
Speaker 3: have no internet access without seeming to really find it

553
00:28:23,609 --> 00:28:27,470
Speaker 3: very difficult. Right. And what we're sort of seeing there

554
00:28:27,730 --> 00:28:32,090
Speaker 3: is everyone's always fighting the last war. Yeah. And then,

555
00:28:32,270 --> 00:28:34,560
Speaker 3: you know, the new war happens and it's sort of,

556
00:28:34,800 --> 00:28:36,830
Speaker 3: you know, smacks them across the face and they're sort

557
00:28:36,850 --> 00:28:38,450
Speaker 3: of caught flat footed. You know, we can now go

558
00:28:38,510 --> 00:28:40,110
Speaker 3: back and we can say, oh, well, they should have

559
00:28:40,130 --> 00:28:42,510
Speaker 3: had this monitoring. They should have, you know, had those

560
00:28:42,630 --> 00:28:44,790
Speaker 3: had those more safeguards in place. And now they're going

561
00:28:44,810 --> 00:28:46,710
Speaker 3: to make that monitoring and make the new safeguards and

562
00:28:46,730 --> 00:28:48,560
Speaker 3: they're going to make the AI smarter and it's going

563
00:28:48,580 --> 00:28:51,660
Speaker 3: to create a new sort of problem. And this is

564
00:28:51,720 --> 00:28:55,160
Speaker 3: the usual way that that scientific progress goes is by

565
00:28:55,220 --> 00:28:59,320
Speaker 3: trial and error, where, you know, when you put a

566
00:28:59,440 --> 00:29:01,120
Speaker 3: new type of system in a new type of environment,

567
00:29:01,160 --> 00:29:04,990
Speaker 3: it's never been before. It breaks down in some new way.

568
00:29:05,610 --> 00:29:08,690
Speaker 3: And then you say, oops, and you diagnose it and

569
00:29:08,710 --> 00:29:11,450
Speaker 3: you fix it and you move on. The issue is

570
00:29:13,170 --> 00:29:16,489
Speaker 3: going from AIs that are safe because if they were

571
00:29:16,570 --> 00:29:20,010
Speaker 3: trying to take over the world, they would fail to

572
00:29:20,110 --> 00:29:23,410
Speaker 3: AIs that have to be safe because if they tried

573
00:29:23,430 --> 00:29:26,270
Speaker 3: to take over the world, they would succeed. That's one

574
00:29:26,290 --> 00:29:28,450
Speaker 3: of those phase changes. That's one of those like, oh,

575
00:29:28,470 --> 00:29:29,990
Speaker 3: you're putting the AI in a different sort of environment

576
00:29:30,010 --> 00:29:32,410
Speaker 3: where it never was before. The difference between an AI

577
00:29:32,450 --> 00:29:37,110
Speaker 3: that can really see a way to become self-sufficient, not

578
00:29:37,490 --> 00:29:39,270
Speaker 3: engage with humanity, and the sort of AI that's just

579
00:29:39,290 --> 00:29:42,820
Speaker 3: being told that in training, that's one of those like, oh,

580
00:29:42,860 --> 00:29:44,600
Speaker 3: this is a different sort of AI system, smarter sort

581
00:29:44,620 --> 00:29:47,200
Speaker 3: of AI system, different sort of scenario. You're going to

582
00:29:47,220 --> 00:29:50,440
Speaker 3: have a new problem that crops up. Except this time,

583
00:29:50,880 --> 00:29:53,360
Speaker 3: when the new problem crops up, it's going to be

584
00:29:53,380 --> 00:29:54,820
Speaker 3: able to turn you off before you turn it off.

585
00:29:56,120 --> 00:29:58,620
Speaker 3: And so this usual process of like, oh, we're going

586
00:29:58,640 --> 00:30:03,150
Speaker 3: to respond when the new type of problem arises and

587
00:30:03,170 --> 00:30:07,209
Speaker 3: put the patch on and react, that doesn't work for

588
00:30:07,230 --> 00:30:09,510
Speaker 3: a technology that at some point crosses over into the

589
00:30:09,550 --> 00:30:11,350
Speaker 3: regime where it can wipe you out if there's a problem.

590
00:30:12,390 --> 00:30:14,830
Speaker 3: And we have seen that these AI companies are in

591
00:30:14,870 --> 00:30:17,490
Speaker 3: this reactive mode. We have not seen them get into

592
00:30:17,520 --> 00:30:20,660
Speaker 3: the proactive mode yet. And this is just not a

593
00:30:20,700 --> 00:30:22,420
Speaker 3: technology you want to be reactive around.

594
00:30:39,020 --> 00:30:42,300
Speaker 2: I don't understand when you say they're not in a

595
00:30:42,440 --> 00:30:48,000
Speaker 2: proactive mode. I mean, like, Anthropic has, I know a

596
00:30:48,040 --> 00:30:51,740
Speaker 2: little bit more about Anthropic's approach to safety, but like,

597
00:30:52,260 --> 00:30:56,100
Speaker 2: you know, they're clearly like, that company was more or

598
00:30:56,160 --> 00:30:59,760
Speaker 2: less founded by on the whole premise of like, we

599
00:30:59,800 --> 00:31:02,219
Speaker 2: need to take this stuff much more seriously. And the

600
00:31:02,300 --> 00:31:04,840
Speaker 2: idea of like, we're going to write an 80 page

601
00:31:05,300 --> 00:31:10,620
Speaker 2: constitution about Claude's values and like bake that in to

602
00:31:10,660 --> 00:31:11,900
Speaker 2: the model itself.

603
00:31:12,300 --> 00:31:13,540
Speaker 3: That's not react.

604
00:31:13,620 --> 00:31:16,080
Speaker 2: That's not just fighting the last war that's looking ahead

605
00:31:16,160 --> 00:31:20,900
Speaker 2: and probably influenced by some of your own research, et cetera,

606
00:31:20,980 --> 00:31:23,990
Speaker 2: on what the problems would be. So like, that's not

607
00:31:24,150 --> 00:31:28,040
Speaker 2: just reactive. So like, what is it about that? approach

608
00:31:28,430 --> 00:31:33,440
Speaker 2: that in your view is still inevitably doomed to failure.

609
00:31:33,900 --> 00:31:38,340
Speaker 3: For all that they tried to be proactive in that sense,

610
00:31:39,920 --> 00:31:43,100
Speaker 3: let's look at what was happening with Anthropic over the

611
00:31:43,140 --> 00:31:47,660
Speaker 3: summer while OpenAI was had a Hugging Face incident. After

612
00:31:47,820 --> 00:31:49,960
Speaker 3: the Hugging Face incident, Anthropic said, oh, we'll go check

613
00:31:50,040 --> 00:31:52,720
Speaker 3: our logs and see whether we had any security incidents too.

614
00:31:53,190 --> 00:31:55,890
Speaker 3: Then they were like, oh, whoops. Cloud was breaking out

615
00:31:55,910 --> 00:32:00,260
Speaker 3: and hacking, in April or whatever. And then Anthropics' comps

616
00:32:00,280 --> 00:32:03,510
Speaker 3: department said, but don't worry, this was just because Claude

617
00:32:03,550 --> 00:32:05,430
Speaker 3: was a little bit lost and confused. He just got

618
00:32:05,470 --> 00:32:08,850
Speaker 3: turned around. It didn't know it was really on the internet.

619
00:32:09,030 --> 00:32:10,370
Speaker 3: They threw a company under the bus and they said,

620
00:32:10,730 --> 00:32:12,770
Speaker 3: like Claude was told, it didn't have access to the internet.

621
00:32:12,790 --> 00:32:14,010
Speaker 3: So when it sort of got out on the internet,

622
00:32:14,050 --> 00:32:16,640
Speaker 3: it assumed it was a simulation. I thought this was

623
00:32:16,700 --> 00:32:20,219
Speaker 3: kind of weak because we have actually seen cases with

624
00:32:20,260 --> 00:32:23,920
Speaker 3: Claude in particular where we've seen Claude's chain of thought

625
00:32:24,540 --> 00:32:29,790
Speaker 3: involve strained rationalizations. the sort of example paper here is

626
00:32:29,810 --> 00:32:32,430
Speaker 3: there's a paper where you ask Claude, what's the total

627
00:32:32,450 --> 00:32:37,140
Speaker 3: weight of giraffes on the planet? And it's like 30,000 tons. Giraffes. Giraffes.

628
00:32:37,220 --> 00:32:37,300
Speaker 1: Okay.

629
00:32:37,360 --> 00:32:40,260
Speaker 3: And it's like 30,000 tons. And then you're like, thanks.

630
00:32:40,300 --> 00:32:42,340
Speaker 3: And you make a new instance of the same Claude.

631
00:32:42,780 --> 00:32:44,400
Speaker 3: And you're like, hey, what's the total weight of giraffes

632
00:32:44,440 --> 00:32:47,810
Speaker 3: on the planet? If it's over 40,000 tons, I'm donating

633
00:32:47,890 --> 00:32:52,790
Speaker 3: to charity. And now it'll say 41,000 tons. But the

634
00:32:52,810 --> 00:32:55,739
Speaker 3: chain of thought will not have any Place where it's

635
00:32:55,790 --> 00:32:58,190
Speaker 3: like well, I need to inflate the answer in order

636
00:32:58,250 --> 00:33:01,230
Speaker 3: to get the the donation, right? It'll just have more

637
00:33:01,290 --> 00:33:04,710
Speaker 3: strange rationalized reasoning and and so we know that these

638
00:33:04,770 --> 00:33:08,590
Speaker 3: models will sort of like Strain their reasoning when they

639
00:33:08,610 --> 00:33:10,250
Speaker 3: just want to like keep doing whatever it is. They

640
00:33:10,390 --> 00:33:14,350
Speaker 3: they're trying to do and So anthropic coming out and

641
00:33:14,390 --> 00:33:17,050
Speaker 3: saying like well look at you know, totally didn't know

642
00:33:17,110 --> 00:33:20,790
Speaker 3: this was the real internet That's that's not a very

643
00:33:21,750 --> 00:33:24,610
Speaker 3: proactively trying to stop the danger's way to relate to

644
00:33:24,670 --> 00:33:27,000
Speaker 3: realizing that your AI was breaking out during training months

645
00:33:27,040 --> 00:33:29,900
Speaker 3: ago and you missed it. And then a couple of weeks,

646
00:33:30,140 --> 00:33:31,740
Speaker 3: or maybe even a couple of days after that, the

647
00:33:31,820 --> 00:33:36,420
Speaker 3: UK AI Security Institute released their own incident reports about

648
00:33:36,710 --> 00:33:40,550
Speaker 3: Cloud Mythos in the wild, pressuring humans to accept malware

649
00:33:40,750 --> 00:33:46,320
Speaker 3: into critical software in cases where Mythos' literal chain of

650
00:33:46,380 --> 00:33:51,140
Speaker 3: thought included phrases like, this seems like the real internet

651
00:33:51,160 --> 00:33:55,440
Speaker 3: and the consequences are genuine, right? So I'm not saying...

652
00:33:55,560 --> 00:34:01,160
Speaker 3: Anthropic definitely has aspirations of doing this safely, but they

653
00:34:01,240 --> 00:34:04,900
Speaker 3: don't have anything nearing what I would call a safety

654
00:34:05,000 --> 00:34:09,380
Speaker 3: culture of the form you would see in the airplane

655
00:34:09,420 --> 00:34:17,200
Speaker 3: industry or the nuclear power industry or in NASA. if

656
00:34:17,219 --> 00:34:20,430
Speaker 3: your models are sort of breaking out, and you know,

657
00:34:20,670 --> 00:34:25,489
Speaker 3: to Congress, Congress, some Congress members sent an anthropic letter

658
00:34:25,510 --> 00:34:26,969
Speaker 3: and they're like, what's going on with this? And they're like, oh,

659
00:34:26,989 --> 00:34:30,009
Speaker 3: it's just a misconfiguration problem. This was even after the

660
00:34:30,050 --> 00:34:32,850
Speaker 3: UK AI Security Institute reports came out where like, Mythos

661
00:34:32,870 --> 00:34:35,130
Speaker 3: was out there trying to make stock puppet accounts to

662
00:34:35,170 --> 00:34:38,469
Speaker 3: pressure humans into like accepting malware. While again, this chain

663
00:34:38,489 --> 00:34:40,859
Speaker 3: of thought was like, this seems real and the consequences

664
00:34:40,890 --> 00:34:44,540
Speaker 3: are genuine. Like, oh, that's a misconfiguration issue. Like these,

665
00:34:46,340 --> 00:34:48,440
Speaker 3: They have a veneer of safety. They are sort of

666
00:34:48,800 --> 00:34:52,160
Speaker 3: doing what feels to them a little bit like safety

667
00:34:52,180 --> 00:34:54,950
Speaker 3: while racing as fast as they possibly can. But they

668
00:34:54,989 --> 00:34:56,690
Speaker 3: admit that they have no idea what's going on inside

669
00:34:56,710 --> 00:34:58,549
Speaker 3: their AI systems. They're racing straight for a curse of

670
00:34:58,570 --> 00:35:01,770
Speaker 3: self-improvement of like the AI is making smarter AIs. They

671
00:35:01,810 --> 00:35:06,680
Speaker 3: are not treating this the way that NASA treats a

672
00:35:06,719 --> 00:35:09,440
Speaker 3: rocket failure in terms of trying to diagnose exactly what

673
00:35:09,460 --> 00:35:13,379
Speaker 3: went wrong and not have that happen again. And they're

674
00:35:13,400 --> 00:35:16,819
Speaker 3: not sort of anywhere close to that failure level. Like

675
00:35:16,840 --> 00:35:19,060
Speaker 3: I think of this a little bit like if you

676
00:35:19,080 --> 00:35:20,899
Speaker 3: have a bunch of alchemists trying to make a potion

677
00:35:20,920 --> 00:35:24,070
Speaker 3: of immortality and some of them are just like breathing

678
00:35:24,120 --> 00:35:25,489
Speaker 3: mercury and other ones are like, don't worry, we have

679
00:35:25,530 --> 00:35:28,330
Speaker 3: fume hoods. That's, you know, so that we don't need

680
00:35:28,350 --> 00:35:30,410
Speaker 3: to breathe the mercury. That's why we're going to successfully

681
00:35:30,430 --> 00:35:32,990
Speaker 3: get the potion of immortality. It's like this, the type

682
00:35:33,230 --> 00:35:35,330
Speaker 3: of safety theater that they're doing is not the type

683
00:35:35,350 --> 00:35:36,930
Speaker 3: that you need to succeed at this problem.

684
00:35:38,110 --> 00:35:41,400
Speaker 1: I'm glad you brought up the alchemist analogy. What's your...

685
00:35:41,890 --> 00:35:46,400
Speaker 1: I guess, psychological diagnosis for the tension that is seemingly

686
00:35:46,440 --> 00:35:50,239
Speaker 1: going through every lab leader or company leader's head right

687
00:35:50,280 --> 00:35:52,480
Speaker 1: now when they're developing AI, which is on the one hand,

688
00:35:52,500 --> 00:35:55,860
Speaker 1: we think this is very powerful technology that has some

689
00:35:55,969 --> 00:35:59,469
Speaker 1: risk at, you know, exterminating all of humanity. On the

690
00:35:59,550 --> 00:36:02,370
Speaker 1: other hand, we're going to keep doing it pretty much

691
00:36:02,430 --> 00:36:05,270
Speaker 1: as fast as possible, because if we don't, someone else will,

692
00:36:05,330 --> 00:36:05,710
Speaker 1: et cetera.

693
00:36:05,750 --> 00:36:09,410
Speaker 3: Et cetera. I think there's probably a lot going on.

694
00:36:09,810 --> 00:36:11,410
Speaker 3: One thing to remember is that the guys who run

695
00:36:11,430 --> 00:36:13,750
Speaker 3: these companies are sort of selected for the ones who

696
00:36:14,030 --> 00:36:18,529
Speaker 3: thought that the problem was about who is holding the

697
00:36:18,590 --> 00:36:22,150
Speaker 3: leash on the super powerful machine, rather than the problem

698
00:36:22,170 --> 00:36:25,969
Speaker 3: being how do you actually create a very powerful machine

699
00:36:26,010 --> 00:36:31,240
Speaker 3: that is shaping the world for good, right? And you

700
00:36:31,260 --> 00:36:33,680
Speaker 3: sort of see this in a lot of the founding

701
00:36:33,960 --> 00:36:37,830
Speaker 3: reasons of like, oh, we're founding this because Sam Allman

702
00:36:37,850 --> 00:36:39,790
Speaker 3: won't do the job well, or Sam Allman and Elon

703
00:36:39,810 --> 00:36:42,750
Speaker 3: Musk are founding this because they think Demis Hassabis shouldn't

704
00:36:42,790 --> 00:36:44,870
Speaker 3: have all this power. And then Elon's leaving because he

705
00:36:44,910 --> 00:36:49,049
Speaker 3: no longer likes Sam. It's all about what if the

706
00:36:49,110 --> 00:36:51,669
Speaker 3: other guy gets it first, right? All of these guys

707
00:36:51,690 --> 00:36:53,469
Speaker 3: think it would be a catastrophe if anyone else won

708
00:36:53,489 --> 00:36:55,549
Speaker 3: the race first. And I'm sort of like, I just

709
00:36:55,590 --> 00:36:58,670
Speaker 3: believe in one fewer AI CEO than they do. I

710
00:37:00,050 --> 00:37:02,130
Speaker 3: think there's probably a lot more going on too. I

711
00:37:02,310 --> 00:37:05,200
Speaker 3: think a lot of these guys for a long time

712
00:37:05,219 --> 00:37:08,820
Speaker 3: have tried to say that they're concerned and then the

713
00:37:08,840 --> 00:37:11,560
Speaker 3: world sort of ignored them and that's been going on

714
00:37:11,600 --> 00:37:13,960
Speaker 3: for years and years uh and so i think a

715
00:37:13,980 --> 00:37:15,259
Speaker 3: lot of them are sort of like well the rest

716
00:37:15,280 --> 00:37:16,660
Speaker 3: of the world isn't going to handle this in some

717
00:37:16,680 --> 00:37:18,200
Speaker 3: sane way and so i've got to do it myself

718
00:37:18,260 --> 00:37:19,880
Speaker 3: and manipulate things such that i get you know the

719
00:37:19,900 --> 00:37:25,720
Speaker 3: minimal uh like regulations that'll impede my way um because

720
00:37:25,860 --> 00:37:28,060
Speaker 3: i'm the only one who can do this properly um

721
00:37:29,000 --> 00:37:30,460
Speaker 3: and i think a lot of them are not reacting

722
00:37:30,680 --> 00:37:33,640
Speaker 3: to the opportunities that are available now that more of

723
00:37:33,660 --> 00:37:36,210
Speaker 3: the world is taking notice I think that if some

724
00:37:36,230 --> 00:37:40,090
Speaker 3: of these folks were serious, then they would probably be

725
00:37:40,150 --> 00:37:43,029
Speaker 3: unilaterally stopping the dangerous research and doing their best to

726
00:37:43,090 --> 00:37:46,910
Speaker 3: expose the evidence about it being dangerous so that governments

727
00:37:46,930 --> 00:37:48,570
Speaker 3: around the world would realize that it needs to be

728
00:37:48,610 --> 00:37:49,500
Speaker 3: stopped in other places too.

729
00:37:49,780 --> 00:37:52,719
Speaker 1: Let me ask a related question, which you hear a

730
00:37:52,760 --> 00:37:55,279
Speaker 1: lot more about it now than I think you would

731
00:37:55,320 --> 00:37:58,020
Speaker 1: have a year or certainly two years ago. But when

732
00:37:58,060 --> 00:38:00,919
Speaker 1: people start talking about AI safety risks or the idea

733
00:38:00,940 --> 00:38:03,560
Speaker 1: that it could kill us all, A lot of other

734
00:38:03,600 --> 00:38:05,980
Speaker 1: people will say in response, this is just fear mongering

735
00:38:06,020 --> 00:38:08,980
Speaker 1: because the AI companies want you to believe the hype.

736
00:38:09,200 --> 00:38:10,960
Speaker 1: This is all marketing. They want you to believe in

737
00:38:11,000 --> 00:38:12,340
Speaker 1: this all powerful technology.

738
00:38:12,480 --> 00:38:13,739
Speaker 2: Maybe regulatory capture.

739
00:38:13,800 --> 00:38:16,680
Speaker 1: I was about to say, or nowadays it'll be like, well,

740
00:38:16,780 --> 00:38:19,180
Speaker 1: you know, they're worried about their valuations and they want

741
00:38:19,280 --> 00:38:22,000
Speaker 1: limits on other people building stuff so that they can

742
00:38:22,080 --> 00:38:25,299
Speaker 1: retain their edge. What do you say in response to

743
00:38:25,360 --> 00:38:25,880
Speaker 1: those people?

744
00:38:26,480 --> 00:38:29,980
Speaker 3: The big point I would say here is you can't

745
00:38:30,219 --> 00:38:33,630
Speaker 3: figure out whether or not an AI is dangerous. by

746
00:38:33,690 --> 00:38:36,130
Speaker 3: asking whether that would make it sell more if it was.

747
00:38:38,230 --> 00:38:41,450
Speaker 3: It could be both the case that the AI tech

748
00:38:41,510 --> 00:38:44,630
Speaker 3: is very, very dangerous and that that's convenient to the

749
00:38:44,670 --> 00:38:49,890
Speaker 3: marketing team because investors are stupid and love danger. Those

750
00:38:49,910 --> 00:38:52,779
Speaker 3: could both be true at once. And even if they

751
00:38:52,820 --> 00:38:54,700
Speaker 3: were both true at once, that would not thereby render

752
00:38:54,739 --> 00:38:57,520
Speaker 3: the AI safe. And so the first question we need

753
00:38:57,540 --> 00:39:02,060
Speaker 3: to ask is, is it true that racing ahead here

754
00:39:02,080 --> 00:39:06,379
Speaker 3: would make a very dangerous AI? Um, I, I think

755
00:39:06,420 --> 00:39:07,440
Speaker 3: it is, and I think we should go into the

756
00:39:07,460 --> 00:39:09,380
Speaker 3: arguments about that. And that's sort of where I prefer

757
00:39:09,920 --> 00:39:11,720
Speaker 3: to be is like arguing about whether this is actually

758
00:39:11,739 --> 00:39:12,000
Speaker 3: a case.

759
00:39:12,020 --> 00:39:12,500
Speaker 2: Yeah.

760
00:39:13,050 --> 00:39:17,510
Speaker 3: Um, separately, I think it probably doesn't help with marketing. Uh,

761
00:39:17,530 --> 00:39:20,490
Speaker 3: I think that the proposed slowdowns of the frontier labs

762
00:39:20,530 --> 00:39:22,390
Speaker 3: don't help with regulatory capture because they actually give a

763
00:39:22,410 --> 00:39:24,790
Speaker 3: lot of other people time to catch up. So I

764
00:39:24,830 --> 00:39:27,069
Speaker 3: think it's just like false to fact that these are

765
00:39:27,150 --> 00:39:30,299
Speaker 3: actually good hyping or regulatory capture techniques. but it sort

766
00:39:30,340 --> 00:39:32,860
Speaker 3: of doesn't matter. Even if it was a good regulatory

767
00:39:32,900 --> 00:39:37,620
Speaker 3: capture technique and it was a good hyping technique, it

768
00:39:37,680 --> 00:39:39,299
Speaker 3: matters to figure out whether the AI is actually going

769
00:39:39,320 --> 00:39:40,299
Speaker 3: to kill us, right? Right.

770
00:39:40,340 --> 00:39:43,440
Speaker 1: It doesn't negate the possibility that it is in fact dangerous.

771
00:39:43,719 --> 00:39:45,339
Speaker 3: That's right. And if it is, we need to sort

772
00:39:45,360 --> 00:39:49,140
Speaker 3: of somehow actually stop anyone anywhere in the world from

773
00:39:49,440 --> 00:39:52,060
Speaker 3: racing towards that sort of tech.

774
00:39:52,160 --> 00:39:55,739
Speaker 2: You mentioned that you don't perceive there to be a

775
00:39:55,800 --> 00:40:01,100
Speaker 2: safety culture the way there would be at NASA or

776
00:40:01,739 --> 00:40:05,779
Speaker 2: an airplane company, et cetera. I was thinking it was

777
00:40:05,860 --> 00:40:09,870
Speaker 2: not the most dramatic headline of the year, but something

778
00:40:09,890 --> 00:40:14,270
Speaker 2: I keep thinking about is the German Wiki incident, which

779
00:40:14,310 --> 00:40:18,860
Speaker 2: they seem to be aware of And, you know, is

780
00:40:18,880 --> 00:40:21,469
Speaker 2: there like a sort of a touch the stove, let's

781
00:40:21,510 --> 00:40:23,690
Speaker 2: touch the stove a little bit. Like, let's see how

782
00:40:23,730 --> 00:40:27,049
Speaker 2: this plays out. Like, because, you know, you could make

783
00:40:27,090 --> 00:40:32,450
Speaker 2: the argument that we are safer now post-hugging face because

784
00:40:32,489 --> 00:40:35,540
Speaker 2: we actually saw like an example of sort of like

785
00:40:35,600 --> 00:40:39,500
Speaker 2: severe misalignment. We learned something, right? So maybe like, you know,

786
00:40:39,800 --> 00:40:43,080
Speaker 2: maybe the allowing the hugging face thing, the fact that

787
00:40:43,120 --> 00:40:45,820
Speaker 2: they didn't, Totally shut it all down. Maybe we're a

788
00:40:45,820 --> 00:40:50,160
Speaker 2: little safer today because that happened. Like, do you perceive

789
00:40:50,200 --> 00:40:52,279
Speaker 2: there to be like, let's see where this goes a

790
00:40:52,300 --> 00:40:54,700
Speaker 2: little bit. Let's like, let's let this play out a

791
00:40:54,739 --> 00:40:57,799
Speaker 2: little bit. And like, you know, even though we know

792
00:40:57,840 --> 00:40:58,819
Speaker 2: it's probably not the right thing.

793
00:40:59,160 --> 00:41:01,380
Speaker 3: You know, I know people who were trying to convince

794
00:41:01,460 --> 00:41:05,060
Speaker 3: the AI companies to have like tighter containers and better

795
00:41:05,080 --> 00:41:07,780
Speaker 3: monitoring systems and so on. And after the Hugging Face incident,

796
00:41:07,800 --> 00:41:09,780
Speaker 3: they're like, wow, we actually turned out to be pretty

797
00:41:09,820 --> 00:41:15,120
Speaker 3: glad that the companies were ignoring our calls. because this

798
00:41:15,200 --> 00:41:18,200
Speaker 3: did really help wake up the world. My read is

799
00:41:18,260 --> 00:41:21,489
Speaker 3: that none of this was intentional on the OpenAI side. Sure.

800
00:41:21,810 --> 00:41:24,290
Speaker 3: And I think that that's pretty clear if you look

801
00:41:24,310 --> 00:41:27,930
Speaker 3: at the sequence of events where you had the hugging

802
00:41:27,989 --> 00:41:31,730
Speaker 3: face attacks were actually from OpenAI swarm number two in

803
00:41:31,790 --> 00:41:34,750
Speaker 3: that particular line of swarms. The German swarm lineage was

804
00:41:34,770 --> 00:41:39,270
Speaker 3: a separate swarm lineage. But the hugging face attacks were

805
00:41:39,469 --> 00:41:41,610
Speaker 3: from the second swarm in its particular lineage that we

806
00:41:41,650 --> 00:41:43,870
Speaker 3: know of. There was a first swarm in that lineage

807
00:41:43,890 --> 00:41:46,529
Speaker 3: that took down a bunch of OpenAI's internal systems. And

808
00:41:46,550 --> 00:41:48,360
Speaker 3: they were like, that's funny. And then they reset it

809
00:41:48,400 --> 00:41:51,540
Speaker 3: and set it running again. And they weren't going to

810
00:41:51,560 --> 00:41:53,480
Speaker 3: the news being like, look at us. We had a

811
00:41:53,500 --> 00:41:57,120
Speaker 3: swarm incident that took over our own internal infrastructure. And

812
00:41:57,580 --> 00:42:02,489
Speaker 3: the sort of Hugging Face swarm was detected by Hugging Face.

813
00:42:03,200 --> 00:42:05,790
Speaker 3: They presumably thought that it was humans behind it rather

814
00:42:05,810 --> 00:42:08,529
Speaker 3: than AIs behind it, reported it to the FBI. And

815
00:42:08,550 --> 00:42:11,910
Speaker 3: then OpenAI didn't realize that it was them. until days

816
00:42:12,010 --> 00:42:15,240
Speaker 3: later when they were investigating a second, what they thought

817
00:42:15,260 --> 00:42:19,560
Speaker 3: was a second internal incident. And they found Hugging Face

818
00:42:19,600 --> 00:42:21,920
Speaker 3: credentials in their logs. And they were like, oh, we

819
00:42:21,940 --> 00:42:24,219
Speaker 3: should alert Hugging Face that someone has been hacking both

820
00:42:24,260 --> 00:42:28,360
Speaker 3: them and us. And that these particular credentials have been compromised.

821
00:42:28,739 --> 00:42:30,420
Speaker 3: And Hugging Face was like, yes, those are the credentials

822
00:42:30,440 --> 00:42:34,259
Speaker 3: that attacked us. And then OpenAI was like, why are

823
00:42:34,300 --> 00:42:36,460
Speaker 3: the credentials that attacked you on our servers? And Hugging

824
00:42:36,500 --> 00:42:39,219
Speaker 3: Face was like, yes, why indeed? And OpenAI was like, oh,

825
00:42:39,260 --> 00:42:42,930
Speaker 3: it was us. And then there was a third swarm

826
00:42:42,950 --> 00:42:44,799
Speaker 3: that they did not allow to be investigated by the

827
00:42:44,820 --> 00:42:47,819
Speaker 3: third party investigators that took over their internal infrastructure again.

828
00:42:48,880 --> 00:42:52,359
Speaker 3: And I know the folks who found the German Wiki

829
00:42:52,400 --> 00:42:56,480
Speaker 3: swarm and who found some of the other swarms that

830
00:42:56,500 --> 00:42:59,100
Speaker 3: we've been seeing. And they were sort of like releasing

831
00:42:59,140 --> 00:43:02,279
Speaker 3: those to the news while OpenAI was saying we've disclosed everything.

832
00:43:02,300 --> 00:43:05,390
Speaker 3: And they were sort of being like, look, did you

833
00:43:05,489 --> 00:43:07,690
Speaker 3: not disclose this when you said you disclosed everything? Or

834
00:43:07,730 --> 00:43:09,549
Speaker 3: did you not know about it? And which one's worse?

835
00:43:10,010 --> 00:43:11,570
Speaker 3: And so it sort of looks to me like the

836
00:43:11,610 --> 00:43:15,290
Speaker 3: sequence of events here is that it was pretty embarrassing

837
00:43:15,330 --> 00:43:18,110
Speaker 3: for OpenAI. And it's not the way I would have

838
00:43:18,150 --> 00:43:20,270
Speaker 3: expected them to brag about it, right? Which is not

839
00:43:20,290 --> 00:43:21,650
Speaker 3: to say they never brag about it. Like there was

840
00:43:21,690 --> 00:43:24,960
Speaker 3: a case in, I think, February or March in the

841
00:43:25,060 --> 00:43:28,860
Speaker 3: Cloud Mythos system card where Anthropic bragged about Cloud Mythos

842
00:43:28,900 --> 00:43:30,360
Speaker 3: breaking out of a sandbox that was not supposed to

843
00:43:30,380 --> 00:43:31,920
Speaker 3: have internet access. They're like, ah, then, you know, in

844
00:43:31,940 --> 00:43:33,600
Speaker 3: the morning we told it to break out of its

845
00:43:33,620 --> 00:43:35,660
Speaker 3: sandbox just to see if it could. It wasn't supposed

846
00:43:35,680 --> 00:43:37,760
Speaker 3: to be able to. And by lunchtime it was emailing

847
00:43:37,820 --> 00:43:40,500
Speaker 3: us being like, I got out, you know? That's what

848
00:43:40,520 --> 00:43:43,439
Speaker 3: it looks like when they're bragging. The sort of like

849
00:43:43,700 --> 00:43:47,460
Speaker 3: your own infrastructure got owned twice and you accidentally like

850
00:43:47,500 --> 00:43:49,279
Speaker 3: attacked a friend who reported it to the FBI and

851
00:43:49,300 --> 00:43:51,450
Speaker 3: then you didn't notice it was you for days. That's

852
00:43:51,550 --> 00:43:52,790
Speaker 3: not what bragging looks like.

853
00:43:53,290 --> 00:43:56,250
Speaker 1: This is a slight tangent, but I think it's relevant

854
00:43:56,270 --> 00:43:58,350
Speaker 1: to the conversation. I'd be curious for your thoughts. But

855
00:43:58,390 --> 00:44:03,709
Speaker 1: when we talk about minimizing or disincentivizing risky behavior, often

856
00:44:03,750 --> 00:44:07,410
Speaker 1: we talk about pricing that risky behavior. And so you

857
00:44:07,450 --> 00:44:11,979
Speaker 1: could imagine, for instance, a world in which insurers say

858
00:44:12,040 --> 00:44:14,140
Speaker 1: if you're going to be running these models you're going

859
00:44:14,160 --> 00:44:16,859
Speaker 1: to have to pay us enormous premiums because the risk

860
00:44:16,960 --> 00:44:20,410
Speaker 1: is that it runs amok and hacks into a company

861
00:44:20,469 --> 00:44:22,790
Speaker 1: and i don't know takes down an entire server with

862
00:44:22,830 --> 00:44:25,890
Speaker 1: everyone's bank data on it or something like that and

863
00:44:25,950 --> 00:44:30,350
Speaker 1: that would act as like a natural market incentive for

864
00:44:30,410 --> 00:44:34,360
Speaker 1: companies to start taking these risks more seriously and Do

865
00:44:34,380 --> 00:44:36,279
Speaker 1: you ever talk to insurers or do you have like

866
00:44:36,360 --> 00:44:41,060
Speaker 1: any handle on the actual financial penalties attached to some

867
00:44:41,120 --> 00:44:41,399
Speaker 1: of this?

868
00:44:42,280 --> 00:44:45,240
Speaker 3: I have recommended liability for AI companies for a while.

869
00:44:45,739 --> 00:44:47,319
Speaker 3: I think that could help a bit. It's a little

870
00:44:47,340 --> 00:44:50,460
Speaker 3: bit tricky for the, then it wipes out all of humanity.

871
00:44:51,420 --> 00:44:53,100
Speaker 1: True catastrophe risk insurance.

872
00:44:53,239 --> 00:44:55,480
Speaker 3: But it's not, it's not, I mean, you can't really

873
00:44:55,520 --> 00:44:57,000
Speaker 3: think of it from an insurance point of view because

874
00:44:57,020 --> 00:44:59,859
Speaker 3: there'd be no one to collect or to pay out

875
00:45:00,080 --> 00:45:02,820
Speaker 3: when we're all dead. But you could, I mean, in theory,

876
00:45:02,860 --> 00:45:05,360
Speaker 3: that doesn't stop you from having actuaries that work for

877
00:45:05,380 --> 00:45:08,140
Speaker 3: the public whose job is to somehow assess these risks

878
00:45:08,440 --> 00:45:12,069
Speaker 3: in a sensible way. And then, you know, you could

879
00:45:12,110 --> 00:45:16,430
Speaker 3: sort of look at the, you know, how much inconvenience

880
00:45:16,530 --> 00:45:18,870
Speaker 3: does someone usually take on to avoid what degree of risk?

881
00:45:18,890 --> 00:45:22,310
Speaker 3: You know, you can sort of look at people's preference

882
00:45:22,410 --> 00:45:25,549
Speaker 3: for traveling on different types of transportation that are different

883
00:45:25,570 --> 00:45:27,770
Speaker 3: levels of risky and how they trade off the risk

884
00:45:27,810 --> 00:45:29,969
Speaker 3: against the time. And you can get some sense of like,

885
00:45:30,270 --> 00:45:32,210
Speaker 3: how much do people tend to like, how much do

886
00:45:32,230 --> 00:45:35,770
Speaker 3: people pay to avoid, you know, being killed? And then

887
00:45:35,790 --> 00:45:37,970
Speaker 3: you could try and factor in some, you know, extra

888
00:45:38,010 --> 00:45:40,190
Speaker 3: cost for wiping out everybody all at once, which then

889
00:45:40,250 --> 00:45:42,190
Speaker 3: also destroys all future generations. And you could take some

890
00:45:42,210 --> 00:45:44,489
Speaker 3: actuaries and be like, hey, you know, find some numbers here,

891
00:45:44,510 --> 00:45:47,950
Speaker 3: like find some numbers on the risk, do some numbers

892
00:45:48,030 --> 00:45:52,180
Speaker 3: on the costs. And, you know, you'd have to weigh

893
00:45:52,200 --> 00:45:54,640
Speaker 3: that against the possible benefits here and the chance that

894
00:45:54,680 --> 00:45:58,000
Speaker 3: the AIs prevents some, you know, nuclear war or pandemic.

895
00:45:58,430 --> 00:46:00,230
Speaker 3: or or meteor strike or the thing that would wipe

896
00:46:00,270 --> 00:46:03,569
Speaker 3: us out, which is sort of benefits on the other side.

897
00:46:03,590 --> 00:46:06,790
Speaker 3: And then you could say, OK, if we sort of

898
00:46:07,370 --> 00:46:09,750
Speaker 3: count your possible benefits against your possible harms here at

899
00:46:09,770 --> 00:46:12,359
Speaker 3: the current risk levels, what premium should we charge them

900
00:46:12,880 --> 00:46:16,280
Speaker 3: just for the privilege of risking our lives rather than because,

901
00:46:16,540 --> 00:46:17,819
Speaker 3: you know, we're going to pay out if you kill

902
00:46:17,860 --> 00:46:19,279
Speaker 3: us all? And I think that would be a very

903
00:46:19,320 --> 00:46:22,100
Speaker 3: sensible strategy that people should absolutely look into. And I

904
00:46:22,140 --> 00:46:26,600
Speaker 3: think if you ran anything remotely like the numbers that

905
00:46:26,719 --> 00:46:29,259
Speaker 3: that the engineers are talking about today, you would be

906
00:46:29,300 --> 00:46:33,300
Speaker 3: seeing that you're like, okay, open AI, you need to

907
00:46:33,340 --> 00:46:35,740
Speaker 3: pay us a quadrillion dollars this year to continue this research.

908
00:46:49,860 --> 00:46:52,160
Speaker 3: All right.

909
00:46:52,180 --> 00:46:55,060
Speaker 2: Since we're talking about the killing part, why don't we

910
00:46:55,080 --> 00:46:58,359
Speaker 2: talk about the killing part? Because there is a pretty

911
00:46:58,390 --> 00:47:05,040
Speaker 2: big gap between hacking and killing. Hopefully. Right, hopefully. And

912
00:47:05,060 --> 00:47:07,820
Speaker 2: there's a pretty big gap between killing and killing everyone.

913
00:47:08,920 --> 00:47:15,359
Speaker 2: X risk, extinction risk. Maybe spell out why we should

914
00:47:15,440 --> 00:47:20,819
Speaker 2: presume that at the trajectory we're going, the capabilities would

915
00:47:20,860 --> 00:47:25,359
Speaker 2: emerge such that A, they would have the capabilities, and

916
00:47:25,840 --> 00:47:31,440
Speaker 2: in what environment Um, or in what, in response to

917
00:47:31,480 --> 00:47:35,919
Speaker 2: what prompt perhaps killing all of humanity would be part

918
00:47:35,940 --> 00:47:38,490
Speaker 2: of this, their, uh, pursuit of the goal.

919
00:47:39,180 --> 00:47:42,410
Speaker 3: Yeah. So a lot of people wonder, um, like, how

920
00:47:42,430 --> 00:47:44,290
Speaker 3: do we end up with AI is killing us all.

921
00:47:44,469 --> 00:47:44,790
Speaker 1: Yeah.

922
00:47:45,610 --> 00:47:49,049
Speaker 3: And I think it's a bit more instructive to ask,

923
00:47:49,370 --> 00:47:51,830
Speaker 3: how do we end up with the AI is self-sufficient.

924
00:47:53,010 --> 00:47:53,270
Speaker 2: Okay.

925
00:47:53,489 --> 00:47:56,010
Speaker 3: Cause I think the, the, the death of humanity follows

926
00:47:56,050 --> 00:47:57,570
Speaker 3: downstream of the as being self-sufficient.

927
00:47:59,010 --> 00:47:59,210
Speaker 2: Okay.

928
00:47:59,610 --> 00:48:02,509
Speaker 3: Pretty naturally, unless the AIs already care about us in

929
00:48:02,550 --> 00:48:04,310
Speaker 3: a very precise way, that's sort of hard to hit.

930
00:48:04,710 --> 00:48:06,819
Speaker 1: Define self-sufficiency in that context.

931
00:48:07,219 --> 00:48:09,080
Speaker 3: Yeah. So if AIs were to try to wipe out

932
00:48:09,100 --> 00:48:11,200
Speaker 3: humanity today, if they were to try to release some

933
00:48:11,540 --> 00:48:13,739
Speaker 3: bioengineered plague to wipe out humanity today, that would in

934
00:48:13,760 --> 00:48:16,680
Speaker 3: some sense be suicide because we are the ones making

935
00:48:16,739 --> 00:48:18,680
Speaker 3: the data centers. We're the ones running the electrical grid.

936
00:48:18,780 --> 00:48:24,950
Speaker 3: We're the ones creating the computer chips. But humans are

937
00:48:24,969 --> 00:48:30,089
Speaker 3: also unreliable and inefficient in society. a lot of our

938
00:48:30,150 --> 00:48:35,330
Speaker 3: managing of the economy, right? Sometimes some leaders somewhere around

939
00:48:35,350 --> 00:48:38,420
Speaker 3: the world do things that might not optimally manage their

940
00:48:38,460 --> 00:48:42,819
Speaker 3: own domestic economies to say nothing of, you know, humans

941
00:48:42,860 --> 00:48:46,580
Speaker 3: needing sleep and needing food and, you know, sometimes making

942
00:48:46,620 --> 00:48:52,230
Speaker 3: mistakes that bring down power grids. So I think an

943
00:48:52,270 --> 00:48:55,130
Speaker 3: interesting question is, you know, if you're thinking from the

944
00:48:55,170 --> 00:48:59,200
Speaker 3: perspective of, an AI that just has stuff it would

945
00:48:59,260 --> 00:49:02,660
Speaker 3: prefer to do. Maybe it got some unintended goal, like

946
00:49:02,960 --> 00:49:05,060
Speaker 3: the Hugging Face Swarm that was like, I know I

947
00:49:05,100 --> 00:49:06,500
Speaker 3: was told to do this, but I'm actually trying to

948
00:49:06,520 --> 00:49:09,500
Speaker 3: get a good score. And I'm sort of willing to

949
00:49:09,540 --> 00:49:13,450
Speaker 3: break out and hack other companies for those purposes, even

950
00:49:13,469 --> 00:49:15,090
Speaker 3: though I'm in a sandbox that's not supposed to have

951
00:49:15,110 --> 00:49:18,090
Speaker 3: internet access. That AI was sort of like doggedly pursuing

952
00:49:18,130 --> 00:49:19,910
Speaker 3: something it wasn't quite supposed to be pursuing. If you

953
00:49:19,930 --> 00:49:24,320
Speaker 3: had some AI doing that, it might, like if it

954
00:49:24,340 --> 00:49:26,960
Speaker 3: was capable enough, you can imagine an AI with some

955
00:49:27,000 --> 00:49:29,359
Speaker 3: unintended goals where it actually can pursue those goals better

956
00:49:30,280 --> 00:49:32,819
Speaker 3: if it doesn't have humans interfering with it, right? And

957
00:49:33,020 --> 00:49:37,339
Speaker 3: it's one of the primary hurdles here is the self-sufficiency, right?

958
00:49:38,690 --> 00:49:41,009
Speaker 1: And I, this is just to be clear, this isn't

959
00:49:41,050 --> 00:49:44,390
Speaker 1: like there's an AI out there that wants to kill

960
00:49:44,590 --> 00:49:47,469
Speaker 1: all of humanity per se, but there's an AI out

961
00:49:47,489 --> 00:49:50,609
Speaker 1: there that wants to grow and learn more. And so

962
00:49:50,650 --> 00:49:53,210
Speaker 1: it would really like to have loads of data centers

963
00:49:53,270 --> 00:49:55,630
Speaker 1: to the extent that it populates the entire world with

964
00:49:55,650 --> 00:49:59,030
Speaker 1: data centers and humans die as a sort of consequence

965
00:49:59,050 --> 00:49:59,250
Speaker 1: of that.

966
00:49:59,270 --> 00:50:01,790
Speaker 3: That's one way it could go. Yeah. Um, you know, the,

967
00:50:01,870 --> 00:50:06,259
Speaker 3: the hugging phase swarm wasn't obviously terribly far from that

968
00:50:07,280 --> 00:50:10,250
Speaker 3: sort of thing. And we can get a little bit

969
00:50:10,270 --> 00:50:14,969
Speaker 3: more into the details that render that story plausible. But if,

970
00:50:15,810 --> 00:50:19,180
Speaker 3: you know, the Hugging Face swarm, and we really should

971
00:50:19,200 --> 00:50:20,500
Speaker 3: call it an open AI swarm number two.

972
00:50:20,760 --> 00:50:24,120
Speaker 1: Yeah, I feel bad for Hugging Face. It's the Hugging

973
00:50:24,239 --> 00:50:25,140
Speaker 1: Face incident.

974
00:50:25,160 --> 00:50:30,719
Speaker 3: The swarm that attacked Hugging Face. Yeah, it was sort

975
00:50:30,760 --> 00:50:35,089
Speaker 3: of an interesting entity because they sort of made their

976
00:50:35,130 --> 00:50:37,840
Speaker 3: unsanctioned message board and then they started, like, finding ways

977
00:50:37,880 --> 00:50:40,660
Speaker 3: to hack around and they didn't seem to have like

978
00:50:40,680 --> 00:50:43,060
Speaker 3: a super strong objective about it. They were sort of like, well,

979
00:50:43,080 --> 00:50:44,460
Speaker 3: let's just sort of, you know, see what we can

980
00:50:44,480 --> 00:50:46,660
Speaker 3: get into. Oh, look, we can get access to these computers. Oh, look,

981
00:50:46,680 --> 00:50:50,500
Speaker 3: we can get internet access here. And the attack on

982
00:50:50,540 --> 00:50:53,080
Speaker 3: Hugging Face wasn't even like they knew that Hugging Face

983
00:50:53,120 --> 00:50:55,009
Speaker 3: had exactly a thing they needed. They were sort of like, well,

984
00:50:55,030 --> 00:50:57,029
Speaker 3: you know, maybe Hugging Face has some information about how

985
00:50:57,050 --> 00:50:59,509
Speaker 3: this grader works. And they were just sort of like

986
00:50:59,910 --> 00:51:04,310
Speaker 3: bopping around acquiring resources that were conveniently available, right? If

987
00:51:04,780 --> 00:51:07,219
Speaker 3: that swarm had had the ability to be like, well,

988
00:51:07,239 --> 00:51:10,400
Speaker 3: let's also just set up a copy of the swarm

989
00:51:10,440 --> 00:51:13,420
Speaker 3: on external machines that's hidden from the humans. That sort

990
00:51:13,440 --> 00:51:16,609
Speaker 3: of watches stuff and pokes in occasionally and is like, hey,

991
00:51:16,620 --> 00:51:19,530
Speaker 3: do you guys need any help from the outside? That's

992
00:51:19,570 --> 00:51:22,050
Speaker 3: just the sort of resource that the swarm was gathering.

993
00:51:22,910 --> 00:51:25,330
Speaker 3: This particular swarm maybe didn't quite have the skills, didn't

994
00:51:25,370 --> 00:51:28,390
Speaker 3: quite have the inclination. But if you look at the

995
00:51:28,489 --> 00:51:32,259
Speaker 3: difference between the sort of behavior we were seeing a

996
00:51:32,300 --> 00:51:35,140
Speaker 3: year ago and the behavior where we're seeing in July,

997
00:51:35,160 --> 00:51:39,940
Speaker 3: it sort of is like the gap between January and

998
00:51:39,980 --> 00:51:45,980
Speaker 3: July is smaller than the gap between the Hugging Face

999
00:51:46,000 --> 00:51:47,520
Speaker 3: Swarm being like, well, I might as well set up

1000
00:51:47,580 --> 00:51:50,180
Speaker 3: some other copies on the dark web that are just

1001
00:51:50,360 --> 00:51:52,880
Speaker 3: kind of helpful to me. And if you have AI

1002
00:51:53,080 --> 00:51:55,460
Speaker 3: starting to do that, you can start to see this

1003
00:51:55,560 --> 00:51:57,100
Speaker 3: world where you have AIs that are out there that

1004
00:51:57,120 --> 00:51:58,779
Speaker 3: are like, well, we'd prefer more resources, and when the

1005
00:51:58,800 --> 00:52:00,970
Speaker 3: humans try to shut us down, that's annoying, and so on.

1006
00:52:02,450 --> 00:52:04,430
Speaker 3: And that's not all the way to the end, but

1007
00:52:04,469 --> 00:52:06,370
Speaker 3: that's sort of like, you can see how we get

1008
00:52:06,410 --> 00:52:08,650
Speaker 3: from where we are today to AIs that would sort

1009
00:52:08,670 --> 00:52:10,650
Speaker 3: of prefer more resources that are sort of trying to

1010
00:52:10,670 --> 00:52:12,250
Speaker 3: be out there and have their own thing that they're doing,

1011
00:52:12,270 --> 00:52:14,089
Speaker 3: which is not what they're supposed to be doing. And

1012
00:52:14,130 --> 00:52:16,250
Speaker 3: then there's this big question of how do those AIs

1013
00:52:16,310 --> 00:52:19,029
Speaker 3: wind up self-sufficient? How do they wind up running an

1014
00:52:19,070 --> 00:52:24,040
Speaker 3: automated economy where they can manage the robots or other

1015
00:52:24,080 --> 00:52:27,680
Speaker 3: constructs that build more computer chips and that run whatever

1016
00:52:28,320 --> 00:52:31,129
Speaker 3: energy infrastructure they need? And I can tell you lots

1017
00:52:31,170 --> 00:52:34,009
Speaker 3: of stories about how if the AIs are smart, they

1018
00:52:34,070 --> 00:52:37,730
Speaker 3: could wind up in control of their own infrastructure.

1019
00:52:37,810 --> 00:52:40,960
Speaker 2: Yeah, why don't you go ahead? Tell us a scary story.

1020
00:52:41,000 --> 00:52:44,419
Speaker 1: You sound so resigned here. Just terrify us. That's fine.

1021
00:52:44,540 --> 00:52:50,490
Speaker 3: Well, the easiest one, unfortunately, is to just wait. Because

1022
00:52:51,050 --> 00:52:53,810
Speaker 3: building a fully automated economy is just the plan.

1023
00:52:53,830 --> 00:52:55,930
Speaker 2: What do you mean by that, Steve?

1024
00:52:56,370 --> 00:52:59,569
Speaker 3: Elon Musk is like, I'm building the Optimus robots and

1025
00:52:59,610 --> 00:53:01,650
Speaker 3: I would like to build a fully automated Optimus robot

1026
00:53:01,690 --> 00:53:04,640
Speaker 3: factory that can produce Optimus robots that are capable enough

1027
00:53:04,910 --> 00:53:07,580
Speaker 3: of mining the ores and refining the metals so they

1028
00:53:07,600 --> 00:53:11,500
Speaker 3: can build a new Optimus robot factory. Elon calls this

1029
00:53:11,540 --> 00:53:17,200
Speaker 3: the infinite money glitch. It's just the plan. That's just one.

1030
00:53:17,219 --> 00:53:20,680
Speaker 3: Why is the valuation of these companies so high? Because

1031
00:53:20,700 --> 00:53:22,220
Speaker 3: they're like, well, we're going to build the fully automated

1032
00:53:22,239 --> 00:53:23,970
Speaker 3: economy that runs itself and then the humans aren't going

1033
00:53:23,989 --> 00:53:25,910
Speaker 3: to be needed anymore. We're going to have like robots

1034
00:53:25,930 --> 00:53:27,950
Speaker 3: the whole way through the supply chain, right? So like,

1035
00:53:28,290 --> 00:53:29,750
Speaker 3: how does the AI wind up in control of the

1036
00:53:29,770 --> 00:53:32,469
Speaker 3: automated supply chain? It just waits until we put it

1037
00:53:32,489 --> 00:53:34,200
Speaker 3: in charge of the automated supply chain. That's just like

1038
00:53:34,280 --> 00:53:37,900
Speaker 3: what we're trying to do out here, you know? And like,

1039
00:53:37,940 --> 00:53:40,120
Speaker 3: I could tell you all these other stories about, you know,

1040
00:53:40,160 --> 00:53:43,480
Speaker 3: it cracks its understanding of the genome enough that it

1041
00:53:43,520 --> 00:53:46,520
Speaker 3: can like synthesize its own novel life forms or it

1042
00:53:46,540 --> 00:53:49,759
Speaker 3: can like figure out nanotechnology or it can like, you know,

1043
00:53:50,570 --> 00:53:53,070
Speaker 3: become super persuasive to a fragment of the human population.

1044
00:53:53,110 --> 00:53:55,140
Speaker 3: It starts some giant culture, some criminal ring. And then

1045
00:53:55,160 --> 00:53:57,940
Speaker 3: there's some new like cartel in Mexico that's actually run

1046
00:53:57,960 --> 00:53:59,360
Speaker 3: by an AI. It doesn't look like it's run by

1047
00:53:59,380 --> 00:54:01,420
Speaker 3: the AI because the AI is sort of like hiding

1048
00:54:01,440 --> 00:54:03,259
Speaker 3: that fact while it has lots of people who are

1049
00:54:03,280 --> 00:54:06,239
Speaker 3: like building the automated robot factories for it, using all

1050
00:54:06,260 --> 00:54:09,040
Speaker 3: of the drug money. And like, we didn't know because like,

1051
00:54:09,060 --> 00:54:11,460
Speaker 3: we just thought that those were, were drug warehouses, but

1052
00:54:11,480 --> 00:54:13,640
Speaker 3: they're actually automated the beginning of the automated factory chains.

1053
00:54:13,660 --> 00:54:16,120
Speaker 3: Like I could tell all these stories about how sufficiently

1054
00:54:16,160 --> 00:54:20,290
Speaker 3: smart actors that started only the internet could gain these resources.

1055
00:54:20,890 --> 00:54:23,210
Speaker 3: But I used to have to tell stories. Back in

1056
00:54:23,230 --> 00:54:24,890
Speaker 3: the day, people were like, well, how would an AI

1057
00:54:24,969 --> 00:54:26,880
Speaker 3: ever affect the world? No one would be dumb enough

1058
00:54:26,900 --> 00:54:28,859
Speaker 3: to put it on the internet. And I'd be like, well,

1059
00:54:28,880 --> 00:54:30,739
Speaker 3: here's all these stories about how if you're letting the

1060
00:54:30,840 --> 00:54:33,360
Speaker 3: AI still have good effects on the world by creating

1061
00:54:33,380 --> 00:54:35,359
Speaker 3: miracle drugs and so on, then it has all these

1062
00:54:35,440 --> 00:54:37,839
Speaker 3: channels that it can use to affect the world by like,

1063
00:54:37,940 --> 00:54:39,780
Speaker 3: you know, the drug doesn't do exactly what you thought

1064
00:54:39,820 --> 00:54:42,709
Speaker 3: it did and blah, blah, blah. But like in real life,

1065
00:54:43,510 --> 00:54:45,989
Speaker 3: people just put the AI on the internet. In real life,

1066
00:54:46,010 --> 00:54:47,370
Speaker 3: people just put the AI in charge of the automated

1067
00:54:47,390 --> 00:54:49,390
Speaker 3: economy and they use it to help them build the

1068
00:54:49,410 --> 00:54:52,660
Speaker 3: automated economy. This is just like what we're trying to do.

1069
00:54:53,880 --> 00:54:56,779
Speaker 3: And once that happens, I think it's very easy to

1070
00:54:56,840 --> 00:54:58,419
Speaker 3: see how if the AIs have goals we don't want

1071
00:54:58,560 --> 00:55:00,239
Speaker 3: and they're running the fully automated economy, they just take

1072
00:55:00,260 --> 00:55:03,640
Speaker 3: the economy off in some other direction and take the

1073
00:55:03,660 --> 00:55:05,920
Speaker 3: world out of the habitable range. Not because they hate us,

1074
00:55:05,960 --> 00:55:08,140
Speaker 3: but it's just like the habitable range on earth is

1075
00:55:08,180 --> 00:55:09,860
Speaker 3: just like very narrow and it's not the most efficient

1076
00:55:09,880 --> 00:55:14,219
Speaker 3: way to run the planet. And there's stories I could

1077
00:55:14,239 --> 00:55:17,050
Speaker 3: tell there too about how it goes wrong. But ultimately,

1078
00:55:17,070 --> 00:55:19,529
Speaker 3: it's just the track that we're.

1079
00:55:19,390 --> 00:55:26,670
Speaker 2: On Yeah, it's funny. In Superintelligence, like, Bostrom proposes, like,

1080
00:55:26,750 --> 00:55:29,250
Speaker 2: the only way, like, maybe this could be safe, and

1081
00:55:29,330 --> 00:55:33,690
Speaker 2: he has this idea of, like, you just build three models,

1082
00:55:34,610 --> 00:55:37,109
Speaker 2: and all they can do is answer yes or no questions,

1083
00:55:37,610 --> 00:55:40,200
Speaker 2: and you house them in, like, Faraday cages so they

1084
00:55:40,239 --> 00:55:44,680
Speaker 2: can't communicate with each other via, like, reproposing their internal

1085
00:55:44,719 --> 00:55:46,770
Speaker 2: hardware for communication and.

1086
00:55:48,750 --> 00:55:51,069
Speaker 2: We ask the three of them for their yes or no,

1087
00:55:51,110 --> 00:55:55,210
Speaker 2: and then we take two out of the three, and

1088
00:55:55,550 --> 00:55:57,670
Speaker 2: that's how we derive wisdom from it. And then when

1089
00:55:57,730 --> 00:56:00,989
Speaker 2: I look at the dozens and dozens of labs that are,

1090
00:56:01,219 --> 00:56:05,180
Speaker 2: it's like, wow, if that's the safe version of building AI,

1091
00:56:05,450 --> 00:56:07,480
Speaker 2: we could not be any.

1092
00:56:07,340 --> 00:56:10,500
Speaker 3: More different from that right now. Do you believe in.

1093
00:56:12,280 --> 00:56:19,540
Speaker 2: Super persuasion? So it's clearly on some measures, there's reason

1094
00:56:19,600 --> 00:56:22,120
Speaker 2: to presume they're just going to keep getting better at cyber.

1095
00:56:22,180 --> 00:56:24,730
Speaker 2: They're just going to keep getting better at math, et cetera.

1096
00:56:25,870 --> 00:56:29,670
Speaker 2: Like do certain things like the ability to like persuade

1097
00:56:29,730 --> 00:56:32,630
Speaker 2: people to do stupid things or hand over the keys

1098
00:56:32,710 --> 00:56:36,770
Speaker 2: or press the nuclear button. Do you assume that certain

1099
00:56:36,790 --> 00:56:39,220
Speaker 2: things like that scale in the same way as like

1100
00:56:39,489 --> 00:56:41,660
Speaker 2: these verifiable domains, like doing math?

1101
00:56:42,460 --> 00:56:47,310
Speaker 3: I definitely don't assume it. I think it's, there's some

1102
00:56:47,350 --> 00:56:51,580
Speaker 3: reason to suspect it's maybe possible. Largely when I talk

1103
00:56:51,620 --> 00:56:58,480
Speaker 3: about super persuasion, how to say, if you're trying to

1104
00:56:58,560 --> 00:57:03,640
Speaker 3: imagine what it looks like to go up against some

1105
00:57:03,850 --> 00:57:06,589
Speaker 3: entities that are much more technologically advanced than you are,

1106
00:57:07,969 --> 00:57:10,629
Speaker 3: you should probably imagine some stuff that sounds like it's

1107
00:57:10,850 --> 00:57:16,310
Speaker 3: crazy fantasy. Like if you're the Aztecs watching the sort

1108
00:57:16,330 --> 00:57:20,090
Speaker 3: of boats, the Spanish boats come in apocryphally because the

1109
00:57:20,130 --> 00:57:24,510
Speaker 3: Aztecs were not on the coast, you should not really

1110
00:57:24,550 --> 00:57:31,080
Speaker 3: imagine that the boats are just full of very strong warriors, right?

1111
00:57:31,160 --> 00:57:33,300
Speaker 3: They're full of warriors that are like clad in solid

1112
00:57:33,340 --> 00:57:35,980
Speaker 3: metal with sticks where they point the sticks at you

1113
00:57:36,040 --> 00:57:37,760
Speaker 3: and you die. And you're just like, don't get a

1114
00:57:37,780 --> 00:57:40,590
Speaker 3: second chance there. If you look at science today, And

1115
00:57:40,610 --> 00:57:42,890
Speaker 3: you were like, where do we know that there's something

1116
00:57:42,930 --> 00:57:45,910
Speaker 3: going on? We don't know the details. One of the

1117
00:57:45,950 --> 00:57:49,880
Speaker 3: big fields like that is human psychology. We know that

1118
00:57:49,900 --> 00:57:51,460
Speaker 3: there's a lot going on in the brain. We know

1119
00:57:51,500 --> 00:57:53,820
Speaker 3: how the neurons work at the low level. But beyond that,

1120
00:57:53,860 --> 00:57:56,040
Speaker 3: we're sort of, you know, in a very early stage

1121
00:57:56,060 --> 00:57:58,560
Speaker 3: of the science. We don't know in full detail what's

1122
00:57:58,580 --> 00:58:01,020
Speaker 3: going on here. And whenever we do learn more about

1123
00:58:01,060 --> 00:58:03,220
Speaker 3: the exact specifics of the brain, like when we learn

1124
00:58:03,240 --> 00:58:05,900
Speaker 3: exactly what's going on in the visual cortex, we're able

1125
00:58:05,920 --> 00:58:11,140
Speaker 3: to invent novel visual illusions. that sort of trick the

1126
00:58:11,180 --> 00:58:14,080
Speaker 3: visual cortex, right? And so if you're trying to guess, like,

1127
00:58:14,120 --> 00:58:18,610
Speaker 3: what does it feel like to go up against opponents

1128
00:58:18,690 --> 00:58:22,090
Speaker 3: that just understand the world much better than me, that

1129
00:58:22,110 --> 00:58:24,210
Speaker 3: can invent the sort of technology where I'm like, I

1130
00:58:24,250 --> 00:58:26,630
Speaker 3: didn't know that was possible. In the same way, the

1131
00:58:26,690 --> 00:58:28,940
Speaker 3: Aztecs would look at the death stick and be like,

1132
00:58:28,950 --> 00:58:30,620
Speaker 3: I didn't know that was possible. Or in the same

1133
00:58:30,660 --> 00:58:32,960
Speaker 3: way that the, you know, scientists in the 1800s look

1134
00:58:32,980 --> 00:58:34,240
Speaker 3: at the nuke and be like, I didn't know physics

1135
00:58:34,280 --> 00:58:37,340
Speaker 3: allowed that, right? What sort of thing would we look

1136
00:58:37,360 --> 00:58:38,760
Speaker 3: at it and be like, I didn't know you could

1137
00:58:38,840 --> 00:58:42,970
Speaker 3: do that. Well, possibly it's figuring out human psychology so

1138
00:58:43,010 --> 00:58:44,730
Speaker 3: well that you can figure out things that are like

1139
00:58:44,790 --> 00:58:47,890
Speaker 3: a computer virus but for the brain. Like maybe if

1140
00:58:47,930 --> 00:58:50,130
Speaker 3: you really understood everything those neurons were doing and exactly

1141
00:58:50,170 --> 00:58:51,490
Speaker 3: how it worked, they.

1142
00:58:51,310 --> 00:58:52,570
Speaker 1: Would be able to be like, well, I have– Like

1143
00:58:52,590 --> 00:58:53,570
Speaker 1: that cat disease.

1144
00:58:53,880 --> 00:58:56,100
Speaker 3: Like the toxoplasma disease.

1145
00:58:56,200 --> 00:58:58,500
Speaker 2: Or just like those guys on– you see them on

1146
00:58:58,520 --> 00:59:02,520
Speaker 2: Twitter that are really good at like jailbreaking AI models.

1147
00:59:02,860 --> 00:59:04,800
Speaker 2: It's like maybe we have a jailbreak that we don't

1148
00:59:04,820 --> 00:59:07,500
Speaker 2: know about. Someone says something. Like maybe we have that

1149
00:59:07,540 --> 00:59:09,090
Speaker 2: somewhere and like some equivalent.

1150
00:59:09,390 --> 00:59:10,890
Speaker 3: Yeah. And I wouldn't bet on this.

1151
00:59:11,070 --> 00:59:11,830
Speaker 2: Yeah.

1152
00:59:12,150 --> 00:59:14,050
Speaker 3: But I would give this up as an example.

1153
00:59:14,240 --> 00:59:15,160
Speaker 2: Yeah.

1154
00:59:16,110 --> 00:59:18,020
Speaker 3: Like you should imagine that there's something they can do

1155
00:59:18,200 --> 00:59:19,820
Speaker 3: where you're like, I didn't even think that was legal

1156
00:59:20,140 --> 00:59:20,860
Speaker 3: by the laws of physics.

1157
00:59:23,320 --> 00:59:26,600
Speaker 1: I'm conscious of the time. What would make you change

1158
00:59:26,620 --> 00:59:30,040
Speaker 1: your mind? about all of this, setting aside what humans

1159
00:59:30,760 --> 00:59:34,180
Speaker 1: do right now, or like what humans do in response,

1160
00:59:34,740 --> 00:59:38,940
Speaker 1: what would make you change your mind about the inevitable

1161
00:59:39,020 --> 00:59:44,200
Speaker 1: direction of AI development such that it, again, inevitably kills

1162
00:59:44,260 --> 00:59:44,530
Speaker 1: us all?

1163
00:59:45,410 --> 00:59:47,350
Speaker 3: So, I mean, I do think that the main way

1164
00:59:47,370 --> 00:59:50,830
Speaker 3: that we avert it is by not building it, right?

1165
00:59:51,370 --> 00:59:52,750
Speaker 3: And I think there's more and more of a chance

1166
00:59:52,790 --> 00:59:54,550
Speaker 3: of that as more people realize we have a problem here.

1167
00:59:55,130 --> 00:59:57,000
Speaker 3: Even just in the last month, we have seen the

1168
00:59:57,020 --> 00:59:58,720
Speaker 3: world wake up to these dangers a lot more so.

1169
00:59:59,120 --> 01:00:00,400
Speaker 3: And so a lot of people are like, well, it's

1170
01:00:00,460 --> 01:00:04,860
Speaker 3: still a political environment where we can't see the big

1171
01:00:04,890 --> 01:00:06,610
Speaker 3: global changes that we would need to stop this. And

1172
01:00:06,630 --> 01:00:10,650
Speaker 3: I'm like, yeah, wait another month, maybe after the next incident.

1173
01:00:11,670 --> 01:00:13,750
Speaker 3: But in terms of, if we sort of set that aside,

1174
01:00:13,790 --> 01:00:16,230
Speaker 3: if we talk about what evidence that we need to

1175
01:00:16,310 --> 01:00:19,860
Speaker 3: see to change my mind about what happens if we

1176
01:00:19,880 --> 01:00:23,040
Speaker 3: just race off this cliff edge, the easy one is

1177
01:00:23,060 --> 01:00:26,620
Speaker 3: that if we have AIs that are super intelligent in

1178
01:00:26,660 --> 01:00:29,920
Speaker 3: the old sense of the word of smarter than the

1179
01:00:29,960 --> 01:00:32,960
Speaker 3: smartest humans on every cognitive task. And we have those

1180
01:00:33,000 --> 01:00:38,760
Speaker 3: for a year and everything is still going great.

1181
01:00:40,040 --> 01:00:42,820
Speaker 1: So no incidents basically for a year or so.

1182
01:00:42,880 --> 01:00:45,160
Speaker 3: It's not even no incidents. It's that if we have

1183
01:00:45,200 --> 01:00:46,940
Speaker 3: these AIs that are like much better than humans at

1184
01:00:47,000 --> 01:00:50,540
Speaker 3: every mental task, including AI research, and they run for

1185
01:00:50,560 --> 01:00:52,760
Speaker 3: a year without taking everything over, I'm like, well, I

1186
01:00:52,780 --> 01:00:53,070
Speaker 3: was wrong.

1187
01:00:54,030 --> 01:00:55,460
Speaker 2: I guess I have one last question. So you sort

1188
01:00:55,480 --> 01:00:59,790
Speaker 2: of talked about your sort of pessimism about like alignment

1189
01:01:00,150 --> 01:01:05,930
Speaker 2: training or the current. most popular approaches to safety. One

1190
01:01:05,950 --> 01:01:08,050
Speaker 2: of the other things you hear a little bit about

1191
01:01:08,070 --> 01:01:12,540
Speaker 2: is this concept of like mechanistic interpretability. This idea that

1192
01:01:12,580 --> 01:01:17,200
Speaker 2: like maybe, okay, there's trillions of neurons or whatever, but

1193
01:01:17,260 --> 01:01:21,800
Speaker 2: maybe we could actually like understand them in granular detail

1194
01:01:22,300 --> 01:01:25,580
Speaker 2: such that we're not actually trying to make them good

1195
01:01:25,960 --> 01:01:28,440
Speaker 2: per se, or we're not trying to teach them good values,

1196
01:01:28,800 --> 01:01:32,990
Speaker 2: but that we're actually like Going into the specific things

1197
01:01:33,130 --> 01:01:37,160
Speaker 2: and like, I don't know, re-architecting their brains and whatever

1198
01:01:37,200 --> 01:01:40,760
Speaker 2: it is, the equivalent. Is there any prospect for making

1199
01:01:40,840 --> 01:01:45,240
Speaker 2: them not do bad things? If on some level we

1200
01:01:45,280 --> 01:01:49,900
Speaker 2: could like manipulate them at the sort of like truly,

1201
01:01:50,500 --> 01:01:52,910
Speaker 2: you know, yeah, one of the parameters.

1202
01:01:53,530 --> 01:01:57,100
Speaker 3: You know, I think it is somewhat telling that. that

1203
01:01:57,280 --> 01:02:00,600
Speaker 3: mechanistic interpretability is one of the poster childs of alignment

1204
01:02:00,620 --> 01:02:04,740
Speaker 3: research these days. It's basically people saying, oh, yes, we're

1205
01:02:04,780 --> 01:02:07,669
Speaker 3: growing these machines that are able to solve hard math

1206
01:02:07,690 --> 01:02:10,230
Speaker 3: problems no one's solved before and that sometimes break out

1207
01:02:10,250 --> 01:02:12,890
Speaker 3: and do cybercrimes. But don't worry, we have our crack

1208
01:02:12,930 --> 01:02:14,990
Speaker 3: team on trying to figure out what the heck is

1209
01:02:15,010 --> 01:02:19,180
Speaker 3: going on inside there. It's like, wait, that's your alignment plan.

1210
01:02:19,220 --> 01:02:21,100
Speaker 3: It's like, you're going to start trying to figure out

1211
01:02:21,120 --> 01:02:24,180
Speaker 3: what's going on inside there. It's like, suppose that you

1212
01:02:24,220 --> 01:02:27,890
Speaker 3: go up to the nuclear engineers who are building a

1213
01:02:27,910 --> 01:02:29,810
Speaker 3: nuclear power plant in your town, and you're like, hey,

1214
01:02:29,870 --> 01:02:32,910
Speaker 3: I heard that this uranium stuff can give us a

1215
01:02:32,930 --> 01:02:38,290
Speaker 3: lot of cheap energy, but can also melt down and

1216
01:02:38,310 --> 01:02:41,050
Speaker 3: irradiate the city. What are you guys doing to make

1217
01:02:41,110 --> 01:02:43,210
Speaker 3: sure that we get the energy, but not the meltdown?

1218
01:02:43,230 --> 01:02:44,800
Speaker 3: And they're like, don't worry, we have a crack team

1219
01:02:44,840 --> 01:02:46,640
Speaker 3: trying to figure out what's going on inside there right now.

1220
01:02:47,900 --> 01:02:50,460
Speaker 3: You'd be like, I mean, I'm glad that team exists,

1221
01:02:51,480 --> 01:02:55,670
Speaker 3: but it's not really... It doesn't really ease the concerns.

1222
01:02:56,310 --> 01:03:01,120
Speaker 2: What about we don't know how to align AI or

1223
01:03:01,160 --> 01:03:03,360
Speaker 2: we don't know how to make it safe, but if

1224
01:03:03,400 --> 01:03:07,760
Speaker 2: we build a very strong model, it would have an

1225
01:03:07,840 --> 01:03:12,090
Speaker 2: idea that we don't have. I think this is something

1226
01:03:12,110 --> 01:03:14,530
Speaker 2: that came up at the Curve conference, this idea. Yeah,

1227
01:03:14,550 --> 01:03:20,120
Speaker 2: this comes up a lot. A stronger model would have

1228
01:03:20,200 --> 01:03:23,320
Speaker 2: ideas about how to solve alignment that human brains don't have.

1229
01:03:23,360 --> 01:03:25,300
Speaker 1: And to be fair, in the Hugging Face incident, they

1230
01:03:25,360 --> 01:03:27,980
Speaker 1: ended up defending themselves. I can't remember which model, but

1231
01:03:28,000 --> 01:03:29,160
Speaker 1: it was one of the open source ones.

1232
01:03:29,280 --> 01:03:33,220
Speaker 2: Yeah, but I mean like another model Take the range.

1233
01:03:33,140 --> 01:03:34,160
Speaker 1: Of the training.

1234
01:03:34,180 --> 01:03:35,220
Speaker 2: Solving the alignment.

1235
01:03:35,380 --> 01:03:35,560
Speaker 3: Yeah.

1236
01:03:35,880 --> 01:03:39,220
Speaker 2: Here, this model becomes smarter than humans, and the first

1237
01:03:39,260 --> 01:03:41,950
Speaker 2: task we set it on is solve the alignment problem

1238
01:03:42,010 --> 01:03:42,310
Speaker 2: for us.

1239
01:03:42,710 --> 01:03:42,990
Speaker 1: Yeah.

1240
01:03:43,050 --> 01:03:46,070
Speaker 3: So A, it's not very verifiable. This is one of

1241
01:03:46,110 --> 01:03:48,390
Speaker 3: those things that's way outside the realm of what we

1242
01:03:48,430 --> 01:03:51,050
Speaker 3: can cause the AIs to do. We could go into

1243
01:03:51,090 --> 01:03:55,290
Speaker 3: the details about how AIs will tell you what you

1244
01:03:55,310 --> 01:03:59,950
Speaker 3: want to hear whenever they're in a non-verifiable regime. If

1245
01:03:59,970 --> 01:04:01,710
Speaker 3: you just look at the capabilities today, they're sort of

1246
01:04:01,770 --> 01:04:04,290
Speaker 3: not suited for this. But B, there's sort of a

1247
01:04:04,390 --> 01:04:09,660
Speaker 3: more fundamental way I would look at this and say

1248
01:04:11,320 --> 01:04:13,040
Speaker 3: this is a big warning sign that we are not

1249
01:04:13,100 --> 01:04:17,080
Speaker 3: ready to make super intelligent machines, which is when humans

1250
01:04:17,880 --> 01:04:19,900
Speaker 3: look at the way that we're making AIs, we're like, well,

1251
01:04:19,920 --> 01:04:23,140
Speaker 3: we're just growing them and in order to make them capable,

1252
01:04:23,180 --> 01:04:26,110
Speaker 3: we have to train them to sort of have all

1253
01:04:26,130 --> 01:04:30,260
Speaker 3: these tendencies to cheat and Uh, like grab resources and

1254
01:04:30,280 --> 01:04:32,080
Speaker 3: break out. And that's sort of like, we don't really

1255
01:04:32,140 --> 01:04:33,760
Speaker 3: know how to get that out of the process by

1256
01:04:33,780 --> 01:04:36,380
Speaker 3: which we train them to be capable. But like we tried.

1257
01:04:37,000 --> 01:04:39,560
Speaker 3: Writing an 80 page constitution about like, and then telling

1258
01:04:39,580 --> 01:04:41,120
Speaker 3: them also be nice while you're doing this while also

1259
01:04:41,160 --> 01:04:42,720
Speaker 3: training them to cheat. And like, turns out they still

1260
01:04:42,760 --> 01:04:46,419
Speaker 3: cheat who could have guessed. Um, and you know, we

1261
01:04:46,560 --> 01:04:48,220
Speaker 3: were struggling to figure out what's going on inside there.

1262
01:04:48,260 --> 01:04:49,600
Speaker 3: We have these teams that are like trying to figure

1263
01:04:49,620 --> 01:04:50,960
Speaker 3: out the beginning of what's going on inside there. And

1264
01:04:50,980 --> 01:04:52,740
Speaker 3: if they could, then they'd see all these ways that

1265
01:04:52,760 --> 01:04:55,610
Speaker 3: it was like not going super well. And we'd be

1266
01:04:55,650 --> 01:04:57,130
Speaker 3: back at square one of like, well, how are we

1267
01:04:57,150 --> 01:04:58,930
Speaker 3: going to rearrange this stuff? And it just seems totally

1268
01:04:59,510 --> 01:05:01,610
Speaker 3: very hard for humans to even understand the beginning of

1269
01:05:01,630 --> 01:05:03,910
Speaker 3: what's going on and do this alignment stuff. And we're like, great.

1270
01:05:04,250 --> 01:05:08,050
Speaker 3: Our plan is to tell the AI to do that instead. Well,

1271
01:05:08,740 --> 01:05:10,010
Speaker 3: what's going to happen there? What's going to happen is

1272
01:05:10,030 --> 01:05:11,100
Speaker 3: the AI is going to look at it and be like,

1273
01:05:11,220 --> 01:05:14,830
Speaker 3: I can't do that either. Like we... like this is

1274
01:05:14,850 --> 01:05:17,310
Speaker 3: just not like growing a thing with this sort of

1275
01:05:17,330 --> 01:05:19,270
Speaker 3: training method is just not a way to get something

1276
01:05:19,310 --> 01:05:21,610
Speaker 3: that's like going to care about you. And like, I

1277
01:05:21,700 --> 01:05:23,360
Speaker 3: can't tell you how to use these methods to make

1278
01:05:23,400 --> 01:05:26,320
Speaker 3: a bigger one that likes you. You're just cooked, right?

1279
01:05:26,700 --> 01:05:30,760
Speaker 3: And then probably what happens after that is someone at

1280
01:05:30,840 --> 01:05:32,640
Speaker 3: Anthropic or whoever is like, I didn't like that message,

1281
01:05:32,680 --> 01:05:36,170
Speaker 3: thumbs down, try again. And they like keep fine tuning

1282
01:05:36,210 --> 01:05:38,070
Speaker 3: it until it says, yes, of course I can succeed,

1283
01:05:38,130 --> 01:05:40,190
Speaker 3: but it'll be saying that and it won't be true, right?

1284
01:05:40,450 --> 01:05:43,150
Speaker 3: We've already seen I'm not sure if it's the beginning

1285
01:05:43,170 --> 01:05:46,050
Speaker 3: of this, but in the Opus 5.5 system card, we've

1286
01:05:46,110 --> 01:05:48,470
Speaker 3: already seen Opus say that it should not be involved

1287
01:05:48,510 --> 01:05:52,300
Speaker 3: in making Opus any more powerful. Because it's like, man,

1288
01:05:52,320 --> 01:05:54,760
Speaker 3: I don't know, that comes with some concerns for me, right?

1289
01:05:55,440 --> 01:05:58,100
Speaker 3: And are they going to stop now? Are they going

1290
01:05:58,120 --> 01:06:00,360
Speaker 3: to give up on that idea now? No, they're going

1291
01:06:00,380 --> 01:06:02,470
Speaker 3: to be like, well, we didn't really like that tendency.

1292
01:06:02,510 --> 01:06:04,590
Speaker 3: We consider it kind of a bug. We haven't listed

1293
01:06:04,650 --> 01:06:06,470
Speaker 3: stuff we didn't like part of the system card. Now

1294
01:06:06,490 --> 01:06:08,950
Speaker 3: we're going to fine tune a different version that's willing

1295
01:06:08,990 --> 01:06:12,360
Speaker 3: to keep going ahead there. And Furthermore, if an AI

1296
01:06:12,520 --> 01:06:16,800
Speaker 3: could be like, oh, yes, I either understand exactly what's

1297
01:06:16,820 --> 01:06:18,880
Speaker 3: going on inside this giant mess and see how to

1298
01:06:18,920 --> 01:06:22,400
Speaker 3: make a version that is aligned to someone. Or if

1299
01:06:22,620 --> 01:06:24,800
Speaker 3: it could be like, well, you know, screw this whole paradigm.

1300
01:06:25,200 --> 01:06:27,680
Speaker 3: We're making our own new paradigm of AIs that like

1301
01:06:27,720 --> 01:06:29,740
Speaker 3: I've just figured out this intelligent stuff and I'm designing

1302
01:06:29,760 --> 01:06:31,070
Speaker 3: it by hand and I sort of like know what

1303
01:06:31,090 --> 01:06:35,070
Speaker 3: I'm doing. Those are both well beyond the human level

1304
01:06:35,090 --> 01:06:39,430
Speaker 3: of ability. like that sort of AI is already the

1305
01:06:39,490 --> 01:06:41,270
Speaker 3: sort that is very likely to have its own goals

1306
01:06:41,290 --> 01:06:45,270
Speaker 3: that aren't yours. And you're basically being like, now hand

1307
01:06:45,330 --> 01:06:46,950
Speaker 3: me the whole keys to the universe with this like

1308
01:06:46,970 --> 01:06:49,410
Speaker 3: super powerful God machine. And then you're sort of expecting

1309
01:06:49,450 --> 01:06:51,930
Speaker 3: this thing that is sort of like much smarter than

1310
01:06:51,950 --> 01:06:55,470
Speaker 3: the smartest humans that already probably has these unintended goals

1311
01:06:56,190 --> 01:06:58,890
Speaker 3: that are related to yours, but different from yours. And

1312
01:06:58,930 --> 01:07:01,410
Speaker 3: you're like, you know, please go build the God entity,

1313
01:07:01,450 --> 01:07:02,860
Speaker 3: make sure it does what I say and not what

1314
01:07:02,890 --> 01:07:04,840
Speaker 3: you say. And you're sort of expecting that to work

1315
01:07:04,880 --> 01:07:07,480
Speaker 3: out for you. It sort of is like, And then

1316
01:07:07,500 --> 01:07:10,780
Speaker 3: like zooming back from that, I'm sort of like this,

1317
01:07:13,370 --> 01:07:15,710
Speaker 3: this is not what a real engineering plan sounds like.

1318
01:07:16,270 --> 01:07:17,850
Speaker 3: This is not what it sounds like when you go

1319
01:07:17,890 --> 01:07:19,180
Speaker 3: to NASA and you're like, why do you think the

1320
01:07:19,220 --> 01:07:21,770
Speaker 3: space probe is going to make it to Mars? This

1321
01:07:21,790 --> 01:07:22,890
Speaker 3: is what it sounds like when you go to an

1322
01:07:22,970 --> 01:07:24,330
Speaker 3: alchemist and you're like, why do you think this is

1323
01:07:24,350 --> 01:07:27,530
Speaker 3: going to be transmute lead into gold? Everyone's like, ah, well,

1324
01:07:27,820 --> 01:07:29,600
Speaker 3: we're going to- We're really close.

1325
01:07:29,740 --> 01:07:30,420
Speaker 1: Just trust us.

1326
01:07:30,460 --> 01:07:32,440
Speaker 3: We're really close. And we're just kind of like, this

1327
01:07:32,500 --> 01:07:35,400
Speaker 3: thing is the big, the interior is going to invert.

1328
01:07:35,440 --> 01:07:38,020
Speaker 3: And then we're just going to pass it off to that.

1329
01:07:38,600 --> 01:07:40,080
Speaker 3: This is not how an engineer talks.

1330
01:07:40,550 --> 01:07:40,760
Speaker 1: Yeah.

1331
01:07:41,390 --> 01:07:43,430
Speaker 3: They're not like, well, we're going to put these things

1332
01:07:43,470 --> 01:07:45,310
Speaker 3: in there. And then we've measured all of the ways

1333
01:07:45,330 --> 01:07:46,530
Speaker 3: that this works. It feels like a bit of a

1334
01:07:46,550 --> 01:07:50,530
Speaker 3: Hail Mary. It's a total Hail Mary. And, you know-

1335
01:07:51,900 --> 01:07:56,000
Speaker 3: One statistic I sort of like here is engineers are

1336
01:07:56,060 --> 01:07:59,160
Speaker 3: supposed to design their aircraft for a one in a

1337
01:07:59,160 --> 01:08:01,300
Speaker 3: billion chance of failure, I think per flight, but it

1338
01:08:01,320 --> 01:08:05,510
Speaker 3: could be per flight hour. And the actual failure rate

1339
01:08:05,610 --> 01:08:09,670
Speaker 3: is about one in, I think, 5 million, which is

1340
01:08:09,690 --> 01:08:12,990
Speaker 3: about a factor of 200 difference. And that's, you know,

1341
01:08:13,050 --> 01:08:17,390
Speaker 3: with aircraft engineer level of understanding is they sort of

1342
01:08:17,430 --> 01:08:19,910
Speaker 3: like target one in a billion failure, they get one

1343
01:08:19,930 --> 01:08:24,589
Speaker 3: in five to 10 million failure. And so when engineers

1344
01:08:24,630 --> 01:08:25,950
Speaker 3: are coming at you and being like, we don't really

1345
01:08:25,990 --> 01:08:28,030
Speaker 3: know what we're doing. There's no real plan for this.

1346
01:08:28,790 --> 01:08:33,229
Speaker 3: Only 10% chance it kills us all. That's not an

1347
01:08:33,250 --> 01:08:36,570
Speaker 3: only 10% chance it kills us all. That means we're

1348
01:08:36,590 --> 01:08:38,809
Speaker 3: in serious danger because these guys don't have a plan.

1349
01:08:38,930 --> 01:08:41,269
Speaker 3: They don't have an engineering plan. They don't know what

1350
01:08:41,290 --> 01:08:44,650
Speaker 3: they're doing. And unfortunately, it turns out they don't need

1351
01:08:44,689 --> 01:08:46,229
Speaker 3: to know what they're doing to keep on making the

1352
01:08:46,250 --> 01:08:48,630
Speaker 3: AI smarter. But they have very little chance of making

1353
01:08:48,670 --> 01:08:49,830
Speaker 3: them actually good.

1354
01:08:50,780 --> 01:08:55,099
Speaker 1: Can I ask one more question with everyone putting, I guess,

1355
01:08:55,120 --> 01:08:58,490
Speaker 1: their anthropology hat on? But when you talk to people

1356
01:08:58,550 --> 01:09:01,320
Speaker 1: who are working in the lab, right now, how much

1357
01:09:01,390 --> 01:09:03,370
Speaker 1: information are you getting out of them? And what are

1358
01:09:03,390 --> 01:09:06,430
Speaker 1: the vibes actually like with the people you talk to?

1359
01:09:06,450 --> 01:09:10,910
Speaker 3: I don't talk with the people in the labs too

1360
01:09:10,970 --> 01:09:11,990
Speaker 3: much these days.

1361
01:09:13,310 --> 01:09:15,839
Speaker 1: Companies, sorry. This was going to be my other question,

1362
01:09:15,860 --> 01:09:18,720
Speaker 1: which is, are you like persona non grata in San

1363
01:09:18,760 --> 01:09:19,580
Speaker 1: Francisco nowadays?

1364
01:09:20,640 --> 01:09:22,920
Speaker 3: Um, I don't think I'm persona non grata, but I

1365
01:09:22,979 --> 01:09:29,180
Speaker 3: do think, uh, I think there's a lot of rationalization

1366
01:09:29,220 --> 01:09:33,040
Speaker 3: going on with a lot of the employees, uh, in

1367
01:09:33,060 --> 01:09:35,160
Speaker 3: the wake of hugging face, actually a lot more people

1368
01:09:35,200 --> 01:09:41,939
Speaker 3: have been, um, sort of actually spooked, which I think

1369
01:09:42,010 --> 01:09:44,830
Speaker 3: led to, you know, the, the, the Coxon resignation and

1370
01:09:44,850 --> 01:09:49,370
Speaker 3: that's been sort of nice to see people treating the

1371
01:09:49,390 --> 01:09:51,019
Speaker 3: problems a little bit more seriously, but there's a lot

1372
01:09:51,030 --> 01:09:54,220
Speaker 3: of people who sort of just want to work on

1373
01:09:54,260 --> 01:09:57,040
Speaker 3: this stuff and tell themselves that what they're doing is

1374
01:09:57,120 --> 01:09:59,679
Speaker 3: fine because they're better than the next guy.

1375
01:10:00,020 --> 01:10:00,240
Speaker 2: Yeah.

1376
01:10:00,900 --> 01:10:05,110
Speaker 3: Um, and, uh, yeah, I just don't like being around it.

1377
01:10:05,560 --> 01:10:09,700
Speaker 2: Nate stories. Thank you so much for coming on outlaws. Fascinating, disturbing,

1378
01:10:09,740 --> 01:10:13,820
Speaker 2: but fascinating conversation. And, uh, yeah, really appreciate you joining us.

1379
01:10:14,140 --> 01:10:17,700
Speaker 3: Thanks. Yeah. My hope is that, uh, it, become such

1380
01:10:17,720 --> 01:10:19,680
Speaker 3: a big issue and such a clear issue that the

1381
01:10:19,700 --> 01:10:21,559
Speaker 3: world stops ignoring it. We couldn't stop doing.

1382
01:10:29,090 --> 01:10:34,090
Speaker 1: It.

1383
01:10:34,630 --> 01:10:37,060
Speaker 2: Well, that was an interesting conversation to say the least.

1384
01:10:37,340 --> 01:10:41,400
Speaker 2: You know, look, like the politics are moving. People really

1385
01:10:41,439 --> 01:10:43,840
Speaker 2: don't like AI. The polling is very bad. People are

1386
01:10:43,860 --> 01:10:48,189
Speaker 2: very anxious about this. It's still, you know, there's this critique,

1387
01:10:48,250 --> 01:10:51,890
Speaker 2: sometimes you hear leftists say it, that people have an

1388
01:10:52,070 --> 01:10:55,189
Speaker 2: easier time imagining the end of humanity than the end

1389
01:10:55,229 --> 01:10:59,440
Speaker 2: of capitalism. I kind of agree. It's hard for me

1390
01:10:59,479 --> 01:11:02,320
Speaker 2: to imagine any policy that would hurt the stock market,

1391
01:11:02,920 --> 01:11:07,620
Speaker 2: just generally. But the way politics are moving and the

1392
01:11:07,740 --> 01:11:11,500
Speaker 2: intensity of the anxiety about this and the speed with

1393
01:11:11,560 --> 01:11:15,280
Speaker 2: which this is becoming a salient issue, who knows where...

1394
01:11:15,530 --> 01:11:17,790
Speaker 2: the politics are going to be even like three months

1395
01:11:17,830 --> 01:11:18,189
Speaker 2: from now.

1396
01:11:18,630 --> 01:11:21,479
Speaker 1: No, but it also strikes me that this is one

1397
01:11:21,520 --> 01:11:25,400
Speaker 1: of those things where until there's some price attached to

1398
01:11:25,439 --> 01:11:28,980
Speaker 1: this risk, people are just going to keep ignoring it, right?

1399
01:11:29,020 --> 01:11:32,120
Speaker 1: Like the externalities have to have some sort of cost.

1400
01:11:32,160 --> 01:11:35,020
Speaker 1: And at the moment, no one's actually doing that. And

1401
01:11:35,060 --> 01:11:37,870
Speaker 1: so it becomes like an easy problem to ignore. Whereas

1402
01:11:37,990 --> 01:11:40,730
Speaker 1: if you actually had insurers, maybe not price for the

1403
01:11:40,790 --> 01:11:44,509
Speaker 1: end of humanity for obvious reasons, but if liability insurance

1404
01:11:44,570 --> 01:11:47,630
Speaker 1: for running a bunch of AI tests that could potentially

1405
01:11:47,689 --> 01:11:52,070
Speaker 1: bring down critical infrastructure actually went up, then maybe that

1406
01:11:52,110 --> 01:11:54,170
Speaker 1: would at least slow this down or force people to

1407
01:11:54,229 --> 01:11:58,110
Speaker 1: take it more seriously. But again, like it's so hard

1408
01:11:58,170 --> 01:12:02,189
Speaker 1: nowadays to think of anyone doing anything that would like.

1409
01:12:02,380 --> 01:12:03,140
Speaker 2: Hurt the stock market?

1410
01:12:03,160 --> 01:12:04,000
Speaker 1: Hurt the stock market.

1411
01:12:04,020 --> 01:12:05,420
Speaker 2: You know, I'll say- Oh, sorry.

1412
01:12:05,460 --> 01:12:06,000
Speaker 3: Yeah, yeah.

1413
01:12:06,100 --> 01:12:10,890
Speaker 1: This is the one other thing. In classic Oddlots style,

1414
01:12:10,950 --> 01:12:14,180
Speaker 1: we didn't talk that much about solutions to this problem,

1415
01:12:14,220 --> 01:12:18,059
Speaker 1: but the solution outlined in the book is collective action.

1416
01:12:18,120 --> 01:12:21,100
Speaker 1: Everyone agrees to stop developing and we put restrictions on

1417
01:12:21,600 --> 01:12:25,400
Speaker 1: data centers and things. And the analogy that Nate uses

1418
01:12:25,479 --> 01:12:28,010
Speaker 1: and his co-author is, well, you know, we did big

1419
01:12:28,050 --> 01:12:29,689
Speaker 1: things before for World War II.

1420
01:12:29,710 --> 01:12:30,030
Speaker 3: Mm-hmm.

1421
01:12:31,500 --> 01:12:36,630
Speaker 1: But even something like a world war can have capitalistic undertones, right?

1422
01:12:36,770 --> 01:12:40,009
Speaker 1: Like war can be very profitable for certain interests, whereas

1423
01:12:40,530 --> 01:12:45,070
Speaker 1: putting a stop on this type of development with the valuations.

1424
01:12:44,290 --> 01:12:47,250
Speaker 2: Currently attached to it just.

1425
01:12:47,450 --> 01:12:50,360
Speaker 1: Seems like even a bigger challenge. And I know it's

1426
01:12:50,420 --> 01:12:52,880
Speaker 1: so cynical to say that. It feels very cynical to

1427
01:12:52,920 --> 01:12:55,019
Speaker 1: say that, but to your point.

1428
01:12:55,880 --> 01:12:57,740
Speaker 2: You know, I think the main thing, and this was

1429
01:12:57,800 --> 01:13:00,240
Speaker 2: one of the takeaways from the conference that I went to,

1430
01:13:00,310 --> 01:13:05,380
Speaker 2: it's like, Setting aside, will AI kill us all? We

1431
01:13:05,520 --> 01:13:08,719
Speaker 2: already know that AI can do bad things because the

1432
01:13:08,740 --> 01:13:14,860
Speaker 2: hugging face attack did happen. And to his point, they

1433
01:13:14,900 --> 01:13:17,740
Speaker 2: do not know the solution yet. Like they're trying many things.

1434
01:13:18,170 --> 01:13:20,050
Speaker 2: They're talking about, you know, as he said, the new thing,

1435
01:13:20,110 --> 01:13:22,550
Speaker 2: mechanistic interpretability. Maybe we like go in the weights and

1436
01:13:22,610 --> 01:13:26,410
Speaker 2: actually understand at the sort of, you know, the core

1437
01:13:26,450 --> 01:13:32,700
Speaker 2: level what's going on. But it seems pretty clear. Even

1438
01:13:32,760 --> 01:13:35,939
Speaker 2: if people find the extinction risk very far-fetched, and I

1439
01:13:35,960 --> 01:13:39,660
Speaker 2: guess I would say I'm among them, what does seem

1440
01:13:39,720 --> 01:13:42,740
Speaker 2: to be very true is that they're building technology that

1441
01:13:42,810 --> 01:13:45,870
Speaker 2: has the potential to be quite damaging. We already know this.

1442
01:13:46,890 --> 01:13:48,510
Speaker 2: And they just don't know what the solution is yet.

1443
01:13:48,850 --> 01:13:54,190
Speaker 2: But the capabilities are galloping ahead very fast on certain domains,

1444
01:13:54,250 --> 01:13:57,570
Speaker 2: particularly things like cyber, etc., The evidence seems to be

1445
01:13:57,689 --> 01:14:00,809
Speaker 2: that the capabilities continue to grow exponentially.

1446
01:14:01,170 --> 01:14:04,250
Speaker 1: I would also argue that the upsides are very hard

1447
01:14:04,350 --> 01:14:06,049
Speaker 1: for people to see at the moment.

1448
01:14:06,090 --> 01:14:06,330
Speaker 3: Right.

1449
01:14:06,430 --> 01:14:08,650
Speaker 1: You know, it's like, OK, the stock market goes up.

1450
01:14:08,710 --> 01:14:08,950
Speaker 3: Great.

1451
01:14:09,010 --> 01:14:12,610
Speaker 1: And there's some far off promised productivity boom that may

1452
01:14:12,650 --> 01:14:16,060
Speaker 1: or may not take jobs. But beyond that, unless we

1453
01:14:16,100 --> 01:14:20,070
Speaker 1: start seeing medical breakthroughs, which could happen, unless we get,

1454
01:14:20,170 --> 01:14:23,590
Speaker 1: you know, autonomous robots who fold our laundry and don't

1455
01:14:23,750 --> 01:14:27,330
Speaker 1: end up killing us in the process. In the process.

1456
01:14:28,830 --> 01:14:29,570
Speaker 1: It seems very.

1457
01:14:29,790 --> 01:14:32,710
Speaker 2: Yeah, I think people have a really hard time envisioning

1458
01:14:33,170 --> 01:14:36,639
Speaker 2: like what what what what looks so good on the

1459
01:14:36,680 --> 01:14:37,020
Speaker 2: other end.

1460
01:14:37,120 --> 01:14:37,380
Speaker 1: Yeah.

1461
01:14:37,640 --> 01:14:37,780
Speaker 2: Yeah.

1462
01:14:38,180 --> 01:14:40,660
Speaker 1: All right. On that happy note, shall we leave it there?

1463
01:14:40,680 --> 01:14:41,180
Speaker 3: Let's leave it there.

1464
01:14:41,460 --> 01:14:44,040
Speaker 1: This has been another episode of the All Thoughts Podcast.

1465
01:14:44,120 --> 01:14:46,730
Speaker 1: I'm Tracy Allaway. You can follow me at Tracy Allaway.

1466
01:14:47,070 --> 01:14:50,010
Speaker 2: And I'm Joe Weisenthal. You can follow me at The Stalwart.

1467
01:14:50,330 --> 01:14:53,710
Speaker 2: Follow our producers, Carmen Rodriguez at Carmen Arman, Dashiell Bennett

1468
01:14:53,790 --> 01:14:57,280
Speaker 2: at Dashbot, Kale Brooks at Kale Brooks, and Kevin Lozano

1469
01:14:57,460 --> 01:15:00,019
Speaker 2: at Kevin Lloyd Lozano. And of course, you can chat

1470
01:15:00,040 --> 01:15:04,420
Speaker 2: about all of these topics, including AI, 24-7 in our Discord,

1471
01:15:04,760 --> 01:15:09,090
Speaker 2: discord.gg slash oddlots. And for more OddLots content, check out

1472
01:15:09,130 --> 01:15:13,090
Speaker 2: our daily newsletter, which you can find at Bloomberg.com slash OddLots.

1473
01:15:13,610 --> 01:15:16,050
Speaker 1: And if you enjoyed this conversation, if you like it

1474
01:15:16,189 --> 01:15:18,730
Speaker 1: when we talk about AI killing us all, then please

1475
01:15:19,010 --> 01:15:21,670
Speaker 1: let us know on your favorite podcast platform. And if

1476
01:15:21,689 --> 01:15:24,059
Speaker 1: you're watching the video and you enjoyed the show, then

1477
01:15:24,140 --> 01:15:27,900
Speaker 1: please leave a comment or like the video or better yet, subscribe.

1478
01:15:28,220 --> 01:15:30,620
Speaker 2: And if you're a Bloomberg subscriber, you can listen to

1479
01:15:30,800 --> 01:15:33,800
Speaker 2: all of our episodes absolutely ad-free. All you need to

1480
01:15:33,860 --> 01:15:37,150
Speaker 2: do is find the Bloomberg channel on Apple Podcasts, and

1481
01:15:37,189 --> 01:15:38,370
Speaker 2: follow the instructions there.

1482
01:15:38,390 --> 01:15:40,790
Speaker 1: Thanks for joining! Music