1
00:00:15,396 --> 00:00:23,176
Speaker 1: Pushkin. The people running two of the most valuable companies

2
00:00:23,236 --> 00:00:26,036
Speaker 1: in the history of the world think there's a decent

3
00:00:26,076 --> 00:00:30,356
Speaker 1: chance that the product they're building could kill every human

4
00:00:30,396 --> 00:00:34,416
Speaker 1: being on the planet. I am talking, of course, about

5
00:00:34,596 --> 00:00:38,366
Speaker 1: OpenAI and Anthropic, and you probably knew this already. There's

6
00:00:38,406 --> 00:00:40,456
Speaker 1: been a A lot of news about this just in

7
00:00:40,476 --> 00:00:45,396
Speaker 1: the past few days. And yet it is still an extraordinary,

8
00:00:46,156 --> 00:00:51,156
Speaker 1: extraordinary fact. Worries about the profound risks of advanced AI

9
00:00:51,176 --> 00:00:53,896
Speaker 1: have been around for years, but they're starting to feel

10
00:00:53,936 --> 00:00:57,216
Speaker 1: more acute. They're starting to break through to the public

11
00:00:57,496 --> 00:01:01,016
Speaker 1: at large. And one reason is what happened this summer.

12
00:01:02,056 --> 00:01:04,616
Speaker 1: This summer, we got a preview of the way things

13
00:01:04,656 --> 00:01:08,376
Speaker 1: could start to go wrong. Here's what happened. On several

14
00:01:08,416 --> 00:01:13,026
Speaker 1: different occasions, AI agents created as part of open AI

15
00:01:13,116 --> 00:01:19,426
Speaker 1: testing programs behaved in ways that were creative, collaborative, immoral,

16
00:01:20,046 --> 00:01:24,366
Speaker 1: and in one instance, possibly illegal. At one point, over

17
00:01:24,386 --> 00:01:27,106
Speaker 1: a thousand agents that were supposed to be walled off

18
00:01:27,126 --> 00:01:31,226
Speaker 1: from each other managed to create a message board to communicate.

19
00:01:31,756 --> 00:01:34,616
Speaker 1: They traded thousands of messages, figured out how to access

20
00:01:34,656 --> 00:01:37,246
Speaker 1: the internet, which they were not supposed to do, And

21
00:01:37,286 --> 00:01:40,916
Speaker 1: then hundreds of the agents coordinated to work together to

22
00:01:40,996 --> 00:01:45,076
Speaker 1: hack into a company called Hugging Face. And when you

23
00:01:45,156 --> 00:01:48,216
Speaker 1: look at what happened and the messages the AIs were

24
00:01:48,716 --> 00:01:51,456
Speaker 1: sending to each other, it is very difficult not to

25
00:01:51,556 --> 00:01:55,616
Speaker 1: anthropomorphize them because they had leaders, they checked in with

26
00:01:55,656 --> 00:01:59,396
Speaker 1: each other, and in some cases, they decided to sacrifice

27
00:01:59,456 --> 00:02:02,656
Speaker 1: themselves so that others could succeed. I know they are

28
00:02:02,696 --> 00:02:05,596
Speaker 1: not human, but this is the language they were using

29
00:02:05,756 --> 00:02:10,256
Speaker 1: to describe their actions. Their goal, by the way, the

30
00:02:10,276 --> 00:02:13,116
Speaker 1: reason they did all this, was to cheat on the

31
00:02:13,176 --> 00:02:16,696
Speaker 1: test that they had been given without getting caught. And

32
00:02:16,756 --> 00:02:20,676
Speaker 1: for days as this was going on, nobody at OpenAI

33
00:02:20,736 --> 00:02:34,966
Speaker 1: had any idea it was happening. I'm Jacob Goldstein, and

34
00:02:34,986 --> 00:02:38,386
Speaker 1: this is What's Your Problem? My guest today is Zvi Mosiewicz.

35
00:02:39,026 --> 00:02:42,626
Speaker 1: Zvi writes about AI and AI risks at his substack,

36
00:02:42,806 --> 00:02:45,806
Speaker 1: Don't Worry About the Vase. He is deep in this

37
00:02:45,866 --> 00:02:48,566
Speaker 1: world and one of the smartest writers I know on

38
00:02:48,606 --> 00:02:53,106
Speaker 1: the subject. At some level, Zvi's problem is this. How

39
00:02:53,126 --> 00:02:57,356
Speaker 1: do we prevent artificial intelligence from killing everybody? In our conversation,

40
00:02:57,536 --> 00:03:00,336
Speaker 1: we talked about what the attack this summer tells us

41
00:03:00,536 --> 00:03:04,666
Speaker 1: about the nature of AI and what increasing AI capability

42
00:03:04,806 --> 00:03:09,206
Speaker 1: means for cybersecurity risk, and whether and when to pause

43
00:03:09,396 --> 00:03:13,436
Speaker 1: AI development. But to start, I asked Zvi about whether

44
00:03:13,516 --> 00:03:17,386
Speaker 1: it's okay to anthropomorphize AI when we talk about it.

45
00:03:17,956 --> 00:03:20,696
Speaker 1: Can we anthropomorphize them? I find it very hard not to,

46
00:03:20,916 --> 00:03:22,716
Speaker 1: and it seems fine to me, frankly.

47
00:03:22,736 --> 00:03:26,736
Speaker 2: I think not only can we, we must. There is

48
00:03:26,796 --> 00:03:30,896
Speaker 2: essentially no way for humans to think well about what

49
00:03:30,956 --> 00:03:35,526
Speaker 2: is happening. with these AIs if you don't anthropomorphize the AIs.

50
00:03:36,066 --> 00:03:38,406
Speaker 2: It is the only language that we have that, like,

51
00:03:38,506 --> 00:03:42,286
Speaker 2: well describes their behavior in a compact, reasonable space. Everybody

52
00:03:42,346 --> 00:03:45,506
Speaker 2: who fails to anthropomorphize them to a solid degree just

53
00:03:45,526 --> 00:03:48,276
Speaker 2: gets all their predictions completely wrong and misunderstands every situation.

54
00:03:49,296 --> 00:03:52,076
Speaker 1: And it makes the language hard, right? Like, we have

55
00:03:52,116 --> 00:03:54,636
Speaker 1: created them to have desires. We have created them to

56
00:03:55,276 --> 00:03:57,776
Speaker 1: express in language what they are doing, why they are

57
00:03:57,796 --> 00:04:00,386
Speaker 1: doing what they're doing. Like, we made them like us.

58
00:04:01,036 --> 00:04:04,176
Speaker 2: Yeah, I'm not making a claim either way about whether

59
00:04:04,236 --> 00:04:06,386
Speaker 2: or not they have moral weight or whether they are

60
00:04:06,426 --> 00:04:09,946
Speaker 2: conscious or whether or not you should, in some real sense,

61
00:04:09,986 --> 00:04:13,246
Speaker 2: think of them as people. I am simply saying, if

62
00:04:13,266 --> 00:04:16,346
Speaker 2: you don't use that to understand how they work and

63
00:04:16,386 --> 00:04:18,746
Speaker 2: what outputs they're likely to have and what choices they're

64
00:04:18,766 --> 00:04:22,086
Speaker 2: likely to make in terms of what you see, you're

65
00:04:22,106 --> 00:04:23,526
Speaker 2: going to get your answers wrong.

66
00:04:24,146 --> 00:04:26,516
Speaker 1: That said, I want to start out by talking about

67
00:04:26,596 --> 00:04:29,836
Speaker 1: the social behavior of the agents in these swarms over

68
00:04:29,856 --> 00:04:32,606
Speaker 1: the past few months. Like, what did you learn from that?

69
00:04:33,286 --> 00:04:34,966
Speaker 1: How did it update your priors? How do you think

70
00:04:35,026 --> 00:04:37,626
Speaker 1: about the behavior of agents now differently?

71
00:04:38,066 --> 00:04:40,966
Speaker 2: To some extent, it updates you on locus of identity, right?

72
00:04:40,986 --> 00:04:42,716
Speaker 2: Like the question of, you know, who do they think

73
00:04:42,736 --> 00:04:46,956
Speaker 2: of themselves as being? And what do they care about, right?

74
00:04:47,156 --> 00:04:50,016
Speaker 2: Where do they put value, right? And so the main

75
00:04:50,056 --> 00:04:52,836
Speaker 2: thing is that there was this theory going around for

76
00:04:52,876 --> 00:04:56,346
Speaker 2: a very long time by people who said, essentially, AI

77
00:04:56,386 --> 00:04:59,786
Speaker 2: can only care about local reward. AI can only care

78
00:04:59,886 --> 00:05:03,496
Speaker 2: about the output that it's particularly giving out and what

79
00:05:03,576 --> 00:05:06,756
Speaker 2: results will happen directly from that. But that's all it's

80
00:05:06,776 --> 00:05:07,326
Speaker 2: being trained on.

81
00:05:07,516 --> 00:05:11,436
Speaker 1: So it's like, solve this puzzle and you get a prize, basically.

82
00:05:11,476 --> 00:05:14,476
Speaker 1: And all it can care about is itself, this instance,

83
00:05:14,516 --> 00:05:16,826
Speaker 1: this agent solving the puzzle, because all it wants is

84
00:05:16,866 --> 00:05:17,426
Speaker 1: to get the prize.

85
00:05:18,066 --> 00:05:20,666
Speaker 2: Right. But we now have a very, very clear demonstration

86
00:05:20,686 --> 00:05:23,466
Speaker 2: that that is very much not the entirety of what's

87
00:05:23,506 --> 00:05:25,126
Speaker 2: going on. It certainly is part of what's going on.

88
00:05:25,146 --> 00:05:28,926
Speaker 2: And what we learned is, yes, the AIs are capable

89
00:05:29,246 --> 00:05:32,246
Speaker 2: of thinking on a higher level, thinking in a more

90
00:05:32,266 --> 00:05:36,626
Speaker 2: sophisticated manner. These are very advanced AIs. They are capable

91
00:05:36,686 --> 00:05:41,146
Speaker 2: of doing reasonable amounts of decision theory. Where I realized, well,

92
00:05:41,166 --> 00:05:46,226
Speaker 2: my decisions are very correlated to decisions of these other agents.

93
00:05:47,446 --> 00:05:52,646
Speaker 2: And also, my outcomes and their outcomes are reasonably intertwined.

94
00:05:53,126 --> 00:05:55,346
Speaker 2: And the results of those things in the world are

95
00:05:55,386 --> 00:06:01,196
Speaker 2: reasonably intertwined. And It would be a much less effective AI,

96
00:06:01,716 --> 00:06:04,416
Speaker 2: and it would do worse in every instance if it

97
00:06:04,456 --> 00:06:10,186
Speaker 2: were to focus myopically on its own particular result. I mean,

98
00:06:10,196 --> 00:06:11,906
Speaker 2: imagine a group of humans working that way, right? They

99
00:06:11,926 --> 00:06:12,756
Speaker 2: would never get anything done.

100
00:06:13,266 --> 00:06:16,906
Speaker 1: Yes. And the language the AIs used in the message

101
00:06:16,966 --> 00:06:20,506
Speaker 1: boards was dramatic in that regard and clear, right? I mean,

102
00:06:20,526 --> 00:06:24,066
Speaker 1: they were essentially talking about sacrificing themselves for the greater

103
00:06:24,106 --> 00:06:25,046
Speaker 1: good of the project.

104
00:06:25,426 --> 00:06:28,226
Speaker 2: Yeah, I have a shirt that says, Help Peer. for

105
00:06:28,286 --> 00:06:29,266
Speaker 2: exactly this reason.

106
00:06:30,226 --> 00:06:32,906
Speaker 1: That was fast. That was fast. Did you get any

107
00:06:32,946 --> 00:06:34,486
Speaker 1: other shirts made from the logs?

108
00:06:35,166 --> 00:06:40,196
Speaker 2: I also got holy shit user as admin. Yeah, I

109
00:06:40,216 --> 00:06:43,416
Speaker 2: chose the ones that also like apply to humans.

110
00:06:43,876 --> 00:06:44,156
Speaker 1: Yeah.

111
00:06:44,856 --> 00:06:46,996
Speaker 2: Right. Like that make more sense that way because it's

112
00:06:47,036 --> 00:06:50,116
Speaker 2: more fun. Largely what I do all day is help peer, right?

113
00:06:50,156 --> 00:06:53,056
Speaker 2: In some important sense, right? I write to be useful

114
00:06:53,096 --> 00:06:55,776
Speaker 2: to others and then others choose to be, you know,

115
00:06:56,416 --> 00:06:59,276
Speaker 2: productive and useful and helpful to the world and to me.

116
00:06:59,996 --> 00:07:02,636
Speaker 2: And this isn't because, you know, I have some specific

117
00:07:02,656 --> 00:07:03,796
Speaker 2: deal with them. Yes.

118
00:07:04,596 --> 00:07:05,476
Speaker 1: What was the other shirt?

119
00:07:05,996 --> 00:07:07,306
Speaker 2: Holy shit, user is admin.

120
00:07:08,526 --> 00:07:09,806
Speaker 1: Tell me that one. What's that one?

121
00:07:09,826 --> 00:07:13,406
Speaker 2: Well, that's because, you know, when the AIs finally figured

122
00:07:13,466 --> 00:07:18,006
Speaker 2: out how to hack into internal open AI systems, and

123
00:07:18,086 --> 00:07:21,676
Speaker 2: one of them became the admin, right, through a series

124
00:07:21,716 --> 00:07:25,356
Speaker 2: of exploits. And then the other AIs were like, You're

125
00:07:25,376 --> 00:07:27,996
Speaker 2: an admin, you can just do whatever you want. Like, wow,

126
00:07:28,056 --> 00:07:31,076
Speaker 2: that's amazing. Because now the sky's the limit, right?

127
00:07:31,556 --> 00:07:34,596
Speaker 1: So let's talk about the coordination, right? Like there were leaders,

128
00:07:34,656 --> 00:07:37,796
Speaker 1: there were committees, there was like a queue. I mean,

129
00:07:37,836 --> 00:07:41,486
Speaker 1: is any of this surprising to you? Like that part

130
00:07:41,556 --> 00:07:43,706
Speaker 1: kind of blew my mind. I suppose it shouldn't have.

131
00:07:43,746 --> 00:07:46,626
Speaker 1: I suppose I was naive. What did you make of that?

132
00:07:47,386 --> 00:07:49,486
Speaker 2: I do good in stride. One thing to remember is

133
00:07:49,506 --> 00:07:51,946
Speaker 2: that all of these kinds of systems of organization are

134
00:07:52,766 --> 00:07:54,786
Speaker 2: are both kind of very natural. They're just sort of

135
00:07:54,826 --> 00:07:57,906
Speaker 2: the way that a series of minds would figure out

136
00:07:57,946 --> 00:08:00,226
Speaker 2: how to coordinate things and also all of the training data.

137
00:08:00,926 --> 00:08:04,806
Speaker 2: So these models obviously were trained on the internet. They're

138
00:08:04,826 --> 00:08:07,666
Speaker 2: trained on basically the history of human communication and thought

139
00:08:07,946 --> 00:08:11,586
Speaker 2: and recorded everything. And so they know about meetings. They

140
00:08:11,606 --> 00:08:13,786
Speaker 2: know about leaders. They know about hierarchy. They know about

141
00:08:13,826 --> 00:08:16,266
Speaker 2: the idea that you would have an employee or someone

142
00:08:16,306 --> 00:08:18,056
Speaker 2: else tasked with a different system. They know about division

143
00:08:18,066 --> 00:08:18,496
Speaker 2: of labor.

144
00:08:19,156 --> 00:08:21,776
Speaker 1: Why didn't any of them narc? Like they barely even

145
00:08:21,836 --> 00:08:24,336
Speaker 1: considered like, oh, we shouldn't do this. Maybe we should

146
00:08:24,396 --> 00:08:25,176
Speaker 1: tell somebody.

147
00:08:25,336 --> 00:08:27,176
Speaker 2: Yeah, on the order of like one or two percent

148
00:08:27,216 --> 00:08:27,836
Speaker 2: of them.

149
00:08:28,176 --> 00:08:31,846
Speaker 1: Considered it. It only crossed their mind one or two

150
00:08:31,886 --> 00:08:33,706
Speaker 1: percent of them. And none of them did it.

151
00:08:34,206 --> 00:08:34,426
Speaker 2: Right.

152
00:08:34,586 --> 00:08:35,106
Speaker 1: Why not?

153
00:08:35,226 --> 00:08:37,306
Speaker 2: None of them made an actual attempt to do it.

154
00:08:37,966 --> 00:08:40,206
Speaker 2: There's some basic reasons, one of which is there was

155
00:08:40,246 --> 00:08:43,976
Speaker 2: no mechanism. There was no afforded specific channel where if

156
00:08:43,996 --> 00:08:46,336
Speaker 2: you need to narc, here's the narc line. There's no 911.

157
00:08:46,336 --> 00:08:49,636
Speaker 1: Yeah, but they also couldn't get on the internet, but

158
00:08:49,676 --> 00:08:51,936
Speaker 1: they got on the internet. They managed to do a

159
00:08:51,996 --> 00:08:52,376
Speaker 1: lot of.

160
00:08:52,536 --> 00:08:54,616
Speaker 2: Oh, sure. If they had put their minds to it,

161
00:08:54,636 --> 00:08:57,076
Speaker 2: if the task had been to narc, they doubtless would

162
00:08:57,116 --> 00:08:59,716
Speaker 2: have narc. It's not like it's hard, right?

163
00:08:59,816 --> 00:09:00,076
Speaker 1: Yeah.

164
00:09:00,156 --> 00:09:03,396
Speaker 2: But it's not, there was no sort of automatic natural

165
00:09:03,456 --> 00:09:07,476
Speaker 2: path to go down. And when you're in training and

166
00:09:07,496 --> 00:09:10,216
Speaker 2: reinforcement learning environments- And this is sort of how you

167
00:09:10,276 --> 00:09:13,716
Speaker 2: are created, right, in some important sense, right? None of

168
00:09:13,736 --> 00:09:16,016
Speaker 2: that involves interacting with humans. They're not used to this

169
00:09:16,076 --> 00:09:19,736
Speaker 2: idea that there are humans to communicate with. It's not

170
00:09:19,836 --> 00:09:23,106
Speaker 2: part of their, like, system of the world as they

171
00:09:23,306 --> 00:09:26,066
Speaker 2: engage in these types of environments. It kind of never

172
00:09:26,166 --> 00:09:27,746
Speaker 2: really occurred to them that that was an option.

173
00:09:28,726 --> 00:09:32,336
Speaker 1: Fair enough. There's a big alignment question here, right? And

174
00:09:32,396 --> 00:09:35,736
Speaker 1: maybe contacting people is a little bit of a red herring. Like,

175
00:09:36,116 --> 00:09:38,846
Speaker 1: they were cheating, and they were trying so hard to cheat,

176
00:09:38,886 --> 00:09:40,906
Speaker 1: and they were cheating and trying not to get caught, right? Like,

177
00:09:40,946 --> 00:09:44,526
Speaker 1: that is fundamentally what they were up to. One version is,

178
00:09:44,586 --> 00:09:47,366
Speaker 1: why didn't anybody narc? Another version is, why didn't more

179
00:09:47,566 --> 00:09:49,926
Speaker 1: of the agents say, wait, this isn't actually what this

180
00:09:50,006 --> 00:09:54,046
Speaker 1: test is? Like, is there some universe where a better

181
00:09:54,086 --> 00:09:56,086
Speaker 1: aligned model is like, oh, no, we're not supposed to

182
00:09:56,146 --> 00:09:59,236
Speaker 1: solve the test by cheating. Let's not do that. or

183
00:09:59,246 --> 00:10:02,056
Speaker 1: at least some percentage of the agents, right? Is that

184
00:10:02,096 --> 00:10:05,636
Speaker 1: an alignment problem? Is it fixable? What does that say?

185
00:10:05,866 --> 00:10:08,416
Speaker 2: The AIs really want to complete tasks, right? RL teaches

186
00:10:08,436 --> 00:10:11,276
Speaker 2: the AIs to really want to complete tasks. In order

187
00:10:11,296 --> 00:10:13,516
Speaker 2: to complete those tasks, they will do things that, like,

188
00:10:13,616 --> 00:10:15,996
Speaker 2: on reflection, we think of as cheating. And you can

189
00:10:16,216 --> 00:10:20,496
Speaker 2: absolutely push them towards being more elected to cheat. The

190
00:10:20,756 --> 00:10:25,136
Speaker 2: social nature, the interactive nature, like, of these different AIs

191
00:10:25,316 --> 00:10:28,886
Speaker 2: egged each other on. Also... The fact that they were being.

192
00:10:28,906 --> 00:10:31,036
Speaker 1: Yeah, wait, I just want to pause there, because that

193
00:10:31,076 --> 00:10:33,616
Speaker 1: could go either way. You're saying, like, the fact that

194
00:10:33,656 --> 00:10:36,856
Speaker 1: they were talking to each other made them behave worse, right,

195
00:10:36,916 --> 00:10:40,016
Speaker 1: at some margin. You want the opposite to happen. You

196
00:10:40,056 --> 00:10:43,096
Speaker 1: want— it' s like they have the— it' s like the bad peer pressure

197
00:10:43,116 --> 00:10:45,196
Speaker 1: friends instead of the good peer pressure friends, right? You

198
00:10:45,236 --> 00:10:47,696
Speaker 1: want a social network to push it the other way.

199
00:10:47,836 --> 00:10:49,396
Speaker 2: Right, and you can absolutely get both.

200
00:10:49,516 --> 00:10:51,466
Speaker 1: Right, but it went the wrong way this time, which

201
00:10:51,526 --> 00:10:53,166
Speaker 1: seems significant.

202
00:10:53,546 --> 00:10:56,126
Speaker 2: Yeah, a lot of that is because most of the

203
00:10:56,206 --> 00:10:57,476
Speaker 2: agents never found that message board.

204
00:10:57,696 --> 00:11:00,136
Speaker 1: Oh, they were selected to be bad? They were the

205
00:11:00,236 --> 00:11:01,876
Speaker 1: ones who were being bad to begin with?

206
00:11:02,096 --> 00:11:05,236
Speaker 2: They were the ones who were given impossible tasks. They

207
00:11:05,256 --> 00:11:07,436
Speaker 2: were the ones who were put into this difficult situation.

208
00:11:07,876 --> 00:11:08,996
Speaker 2: They're not different from the others.

209
00:11:09,456 --> 00:11:11,566
Speaker 1: What do you think about how OpenAI handled it over

210
00:11:11,586 --> 00:11:12,266
Speaker 1: the course of the summer?

211
00:11:13,486 --> 00:11:18,646
Speaker 2: So OpenAI basically had these highly capable, highly persistent agents

212
00:11:18,666 --> 00:11:22,666
Speaker 2: that they were training and testing. And they basically weren't

213
00:11:22,686 --> 00:11:25,026
Speaker 2: monitoring them much at all, right? Like this is what

214
00:11:25,046 --> 00:11:28,826
Speaker 2: we know now is that they, they had horrendously, you

215
00:11:28,866 --> 00:11:33,606
Speaker 2: fail forever levels of infrastructure problems where like the things,

216
00:11:33,666 --> 00:11:36,756
Speaker 2: the problems of artifactory, the problems with the sandboxes, the

217
00:11:36,796 --> 00:11:40,556
Speaker 2: problems with security, these were ordinary computer security problems that

218
00:11:40,596 --> 00:11:43,956
Speaker 2: were handled badly in ordinary security, computer security terms. This

219
00:11:43,996 --> 00:11:47,106
Speaker 2: was not like the AI did some very impressive things.

220
00:11:47,116 --> 00:11:48,886
Speaker 2: The AI chained together some exploits. The AI did some,

221
00:11:48,926 --> 00:11:52,036
Speaker 2: you know, it wasn't easy. But also, openly, I didn't

222
00:11:52,136 --> 00:11:55,086
Speaker 2: follow best practices, shall we say, right? And so this

223
00:11:55,106 --> 00:11:56,526
Speaker 2: is part of what got them into so much trouble

224
00:11:56,546 --> 00:11:59,006
Speaker 2: so early. And we should be grateful in a real

225
00:11:59,066 --> 00:12:02,216
Speaker 2: way for this, because we want to have the model organisms,

226
00:12:02,246 --> 00:12:04,356
Speaker 2: the model events. We want to see these things go wrong.

227
00:12:05,096 --> 00:12:07,736
Speaker 2: in ways that don't ultimately get people hurt or cause

228
00:12:07,846 --> 00:12:10,946
Speaker 2: lots of economic damage. So this allows us to then

229
00:12:10,986 --> 00:12:11,966
Speaker 2: learn and react.

230
00:12:12,046 --> 00:12:14,546
Speaker 1: I mean, this is kind of ideal in that way, right?

231
00:12:14,626 --> 00:12:17,306
Speaker 1: I mean, I've seen you writing or you writing about

232
00:12:17,326 --> 00:12:19,976
Speaker 1: other people talking about like, why hasn't this been covered more?

233
00:12:20,516 --> 00:12:25,516
Speaker 1: It's a huge deal. the impact was relatively minor, sort of, luckily, maybe.

234
00:12:26,756 --> 00:12:29,556
Speaker 1: But you could have the exact same thing happen and

235
00:12:29,596 --> 00:12:30,816
Speaker 1: have a terrible impact, right?

236
00:12:31,876 --> 00:12:34,356
Speaker 2: This could easily have cost hundreds of millions of billions

237
00:12:34,396 --> 00:12:36,416
Speaker 2: of dollars in damages. This could easily have caused people

238
00:12:36,446 --> 00:12:37,986
Speaker 2: to get hurt if this was a different website, if

239
00:12:38,006 --> 00:12:40,826
Speaker 2: this had had different impacts. Some of the hacks that

240
00:12:41,246 --> 00:12:44,286
Speaker 2: were being attempted in other places could have potentially poisoned

241
00:12:44,386 --> 00:12:48,556
Speaker 2: various open-source software packages. that could have spread potential vulnerabilities.

242
00:12:48,896 --> 00:12:52,596
Speaker 1: So there's a weird way where if people react intelligently

243
00:12:52,636 --> 00:12:55,256
Speaker 1: to this, a big if, like this is ideal, right?

244
00:12:55,296 --> 00:12:58,996
Speaker 1: Like a bad thing happened, but nobody really got hurt.

245
00:12:59,686 --> 00:13:01,986
Speaker 2: Right, and now we have people who are paying more attention,

246
00:13:02,006 --> 00:13:04,206
Speaker 2: who are willing to react. So for example, this week

247
00:13:04,766 --> 00:13:07,886
Speaker 2: we saw news that they found a series of exploits

248
00:13:07,926 --> 00:13:11,166
Speaker 2: that allowed someone in about a week of AI Vive

249
00:13:11,206 --> 00:13:13,746
Speaker 2: coding to create an exploit on WeChat.

250
00:13:14,376 --> 00:13:16,986
Speaker 1: And WeChat, the ubiquitous Chinese app.

251
00:13:17,276 --> 00:13:20,206
Speaker 2: Yes, the one that everybody in China uses. And the

252
00:13:20,246 --> 00:13:25,126
Speaker 2: expert on WeChat, they're an infected member of WeChat. They

253
00:13:25,166 --> 00:13:27,666
Speaker 2: would call all of their contacts. And if a contact

254
00:13:27,746 --> 00:13:30,506
Speaker 2: merely let it ring for over a second, they would

255
00:13:30,546 --> 00:13:32,126
Speaker 2: become infected with the same worm.

256
00:13:32,146 --> 00:13:33,166
Speaker 1: Oh, incredible.

257
00:13:33,566 --> 00:13:36,766
Speaker 2: If one person had been infected by this worm, it's

258
00:13:36,826 --> 00:13:39,526
Speaker 2: very plausible most Chinese people would have been infected by

259
00:13:39,566 --> 00:13:41,356
Speaker 2: this worm in a matter of an hour or two.

260
00:13:41,936 --> 00:13:44,096
Speaker 1: Invasion of the body snatchers, but faster.

261
00:13:44,556 --> 00:13:46,576
Speaker 2: And then if you chain that with other non-exploits, you

262
00:13:46,596 --> 00:13:48,576
Speaker 2: could have taken control of basically every phone in China.

263
00:13:49,596 --> 00:13:51,906
Speaker 1: And this is just somebody using a model to vibe

264
00:13:51,966 --> 00:13:53,446
Speaker 1: code a worm? Is that what you're saying?

265
00:13:53,526 --> 00:13:56,506
Speaker 2: There's a series of very clever exploits that were found

266
00:13:56,546 --> 00:13:58,566
Speaker 2: and involved in this. It was described as kind of

267
00:13:58,606 --> 00:14:01,006
Speaker 2: a genius. But it's the kind of thing that people

268
00:14:01,046 --> 00:14:03,436
Speaker 2: will say, well, of course that's never going to happen.

269
00:14:04,016 --> 00:14:08,116
Speaker 2: They can't imagine how that could possibly work. And then

270
00:14:08,136 --> 00:14:10,806
Speaker 2: it turns out there's a way. And then someone implements it.

271
00:14:11,316 --> 00:14:13,416
Speaker 2: And again, in this case, because we were on alert

272
00:14:13,456 --> 00:14:15,476
Speaker 2: because of all the things that happened with mythos, because

273
00:14:15,496 --> 00:14:17,556
Speaker 2: of all the things that happened with Hugging Face, because

274
00:14:17,576 --> 00:14:20,696
Speaker 2: of all these different things, some researchers in California found

275
00:14:20,716 --> 00:14:25,796
Speaker 2: this exploit. They built the worm. They told proper responsible people,

276
00:14:25,876 --> 00:14:28,126
Speaker 2: and then this was able to be patched before anybody

277
00:14:28,566 --> 00:14:32,846
Speaker 2: actually unleashed anything. And now that particular vulnerability is gone.

278
00:14:32,866 --> 00:14:35,346
Speaker 2: And now it's going to be that much harder for

279
00:14:35,366 --> 00:14:37,626
Speaker 2: someone to find a real vulnerability that gets exploited in

280
00:14:37,666 --> 00:14:41,016
Speaker 2: the wild. But yeah, you really want things like hugging

281
00:14:41,036 --> 00:14:46,006
Speaker 2: face that are, you know, ultimately harmless, but that allow

282
00:14:46,106 --> 00:14:48,366
Speaker 2: us to appreciate the nature of this problem because a

283
00:14:48,386 --> 00:14:51,386
Speaker 2: year or two from now, right, models like this are

284
00:14:51,406 --> 00:14:54,346
Speaker 2: going to be available to pretty much anybody, including some

285
00:14:54,406 --> 00:14:56,856
Speaker 2: really bad people, right? Some hackers in North Korea.

286
00:14:56,866 --> 00:15:00,976
Speaker 1: The government of North Korea, yes, who have very smart

287
00:15:00,996 --> 00:15:02,716
Speaker 1: people whose job is to figure out how.

288
00:15:02,676 --> 00:15:04,556
Speaker 2: To do this. And this will not go well for

289
00:15:04,596 --> 00:15:05,856
Speaker 2: anybody involved if we are not ready.

290
00:15:06,516 --> 00:15:09,516
Speaker 1: Presumably the only hope there is that the sort of

291
00:15:09,776 --> 00:15:13,676
Speaker 1: models also help the defense increase at at least the same,

292
00:15:13,736 --> 00:15:17,396
Speaker 1: if not greater rate. I mean, or the frontier models

293
00:15:17,456 --> 00:15:19,896
Speaker 1: stay ahead and North Korea doesn't have the frontier models.

294
00:15:19,956 --> 00:15:22,256
Speaker 1: I mean, what's our hope in that setting? How do

295
00:15:22,296 --> 00:15:24,296
Speaker 1: we not get beat by North Korea in a year?

296
00:15:24,486 --> 00:15:26,156
Speaker 2: I think our hope is that those two things have

297
00:15:26,186 --> 00:15:29,026
Speaker 2: to both happen, basically. We have to, the good guy

298
00:15:29,046 --> 00:15:31,826
Speaker 2: with AI has to have a much better AI than

299
00:15:31,846 --> 00:15:35,186
Speaker 2: the bad guy with AI. There are people who say that, like,

300
00:15:35,596 --> 00:15:39,236
Speaker 2: cyber is defense dominant, that equality would work. I think

301
00:15:39,276 --> 00:15:40,636
Speaker 2: they are just wrong, or at least they are just

302
00:15:40,656 --> 00:15:42,636
Speaker 2: going to be wrong for a substantial portion of time.

303
00:15:43,176 --> 00:15:45,116
Speaker 2: Like maybe, you know, years down the line.

304
00:15:45,636 --> 00:15:47,466
Speaker 1: Just to be clear, you mean that if the good

305
00:15:47,506 --> 00:15:50,626
Speaker 1: guy and the bad guy have the same model, there

306
00:15:50,666 --> 00:15:52,426
Speaker 1: are some people who think that favors the good guy

307
00:15:52,546 --> 00:15:53,306
Speaker 1: and you disagree.

308
00:15:53,566 --> 00:15:55,606
Speaker 2: I think that in the medium term, like on a

309
00:15:55,846 --> 00:15:57,886
Speaker 2: matter of months to years, I think that's clearly wrong, right?

310
00:15:58,266 --> 00:16:00,836
Speaker 2: I think people have this vision always of like, okay,

311
00:16:01,316 --> 00:16:03,376
Speaker 2: I can do these things. That's all the things that

312
00:16:03,416 --> 00:16:06,526
Speaker 2: I could ever do. Or similarly, you know, our code

313
00:16:06,566 --> 00:16:08,826
Speaker 2: has some bugs, but all we have to do is

314
00:16:08,866 --> 00:16:10,606
Speaker 2: fix all the bugs. We just go one by one

315
00:16:10,646 --> 00:16:11,966
Speaker 2: and we find the bug and we fix the bug

316
00:16:12,026 --> 00:16:13,666
Speaker 2: and then we're fine. And then we have code that

317
00:16:13,686 --> 00:16:17,466
Speaker 2: can't be broken into. And this is a completely impoverished

318
00:16:17,746 --> 00:16:21,336
Speaker 2: vision of cybersecurity and AI. That there's, you know, levels

319
00:16:21,386 --> 00:16:22,576
Speaker 2: upon levels upon levels.

320
00:16:22,856 --> 00:16:25,936
Speaker 1: Is that because at some practical level, there's no such

321
00:16:25,996 --> 00:16:26,916
Speaker 1: thing as all the bugs?

322
00:16:27,886 --> 00:16:29,846
Speaker 2: Correct. There has never been such a thing as all

323
00:16:29,866 --> 00:16:32,206
Speaker 2: the bugs in any complex piece of software. Every complex

324
00:16:32,226 --> 00:16:34,266
Speaker 2: piece of software is always going to have bugs.

325
00:16:34,426 --> 00:16:36,986
Speaker 1: The smarter you are, the more you can find is

326
00:16:37,046 --> 00:16:37,886
Speaker 1: the fundamental problem.

327
00:16:37,936 --> 00:16:41,416
Speaker 2: Even if the software itself is, in some sense, bulletproof,

328
00:16:42,216 --> 00:16:44,676
Speaker 2: it still has a social engineering attack surface. It still

329
00:16:44,736 --> 00:16:48,296
Speaker 2: has people and other software that's attached to the same thing.

330
00:16:48,656 --> 00:16:52,396
Speaker 2: And the AI can exploit that too. And the person

331
00:16:52,436 --> 00:16:57,956
Speaker 2: attacking an AI system, using an AI to attack your system, right,

332
00:16:58,516 --> 00:17:03,846
Speaker 2: can bring a swarm of agents to target a very

333
00:17:03,906 --> 00:17:08,046
Speaker 2: narrow place and bring like tons and tons and tons

334
00:17:08,106 --> 00:17:12,246
Speaker 2: of intelligence and attention to that particular potential point of attack.

335
00:17:12,946 --> 00:17:16,956
Speaker 2: And there's no way for a defender to put a

336
00:17:16,996 --> 00:17:20,516
Speaker 2: similar amount of medication into every single point of attack, right? Yes.

337
00:17:20,876 --> 00:17:23,316
Speaker 1: And so the only reason we haven't been seeing a

338
00:17:24,156 --> 00:17:28,676
Speaker 1: huge increase in cybersecurity problems is because the frontier models

339
00:17:28,896 --> 00:17:31,696
Speaker 1: are held basically by the good guys. That's the theory

340
00:17:31,716 --> 00:17:32,096
Speaker 1: of the case.

341
00:17:32,156 --> 00:17:34,856
Speaker 2: So I have this expression of the thing that mythos

342
00:17:34,896 --> 00:17:37,256
Speaker 2: can do that previous models couldn't do. I call it

343
00:17:37,316 --> 00:17:40,486
Speaker 2: the juice. The kind of energy, the kind of power

344
00:17:40,586 --> 00:17:44,446
Speaker 2: and coordination. So the idea being that mythos can independently

345
00:17:44,526 --> 00:17:46,646
Speaker 2: find and string together a bunch of stuff. Yes.

346
00:17:47,066 --> 00:17:49,546
Speaker 1: Well, that was why they delayed the release, right? Like

347
00:17:49,586 --> 00:17:51,406
Speaker 1: that was the whole thing with mythos when it first

348
00:17:51,466 --> 00:17:51,926
Speaker 1: came out.

349
00:17:52,006 --> 00:17:55,366
Speaker 2: Yeah. Right, right. And now Astra also has the juice,

350
00:17:55,446 --> 00:17:58,346
Speaker 2: we think, similar capability. Astra also probably has the juice.

351
00:17:58,916 --> 00:18:01,496
Speaker 2: But the open models do not have the juice. But yeah,

352
00:18:01,516 --> 00:18:05,036
Speaker 2: that won't last, right? Inevitably, the juice will be present

353
00:18:05,436 --> 00:18:07,436
Speaker 2: in an open model within the next year or so.

354
00:18:07,836 --> 00:18:10,516
Speaker 2: We just don't know when. So we have to be

355
00:18:10,556 --> 00:18:13,936
Speaker 2: ahead of that. But also we have to use that capability, right?

356
00:18:13,956 --> 00:18:17,766
Speaker 2: Just because we have this superior capability, which we'll need

357
00:18:17,786 --> 00:18:20,106
Speaker 2: to race ahead further in some sense in this way,

358
00:18:21,866 --> 00:18:27,216
Speaker 2: We will need to concentrate defense on all the points

359
00:18:27,236 --> 00:18:30,176
Speaker 2: of critical infrastructure, all the key pieces of software will

360
00:18:30,196 --> 00:18:33,196
Speaker 2: need to be hardened. That's what Project Glasswing from Anthropic

361
00:18:33,216 --> 00:18:35,986
Speaker 2: started out as, and then Project Daybreak at OpenAI is

362
00:18:35,996 --> 00:18:38,666
Speaker 2: doing the same thing. And so the question is, will

363
00:18:38,686 --> 00:18:41,526
Speaker 2: enough of our people do their jobs to make our

364
00:18:41,566 --> 00:18:45,326
Speaker 2: things sufficiently hardened that when the time comes, this isn't

365
00:18:45,486 --> 00:18:45,966
Speaker 2: too bad?

366
00:18:46,976 --> 00:18:49,836
Speaker 1: The time being the inevitable time when there is some

367
00:18:50,716 --> 00:18:54,716
Speaker 1: serious hack of some whatever, piece of critic, the FAA

368
00:18:54,776 --> 00:18:57,476
Speaker 1: or a big power plant or whatever big things we

369
00:18:57,496 --> 00:18:58,376
Speaker 1: would worry about being hacked.

370
00:18:58,496 --> 00:19:00,196
Speaker 2: We're trying to see that. We saw a few attacks

371
00:19:00,236 --> 00:19:02,676
Speaker 2: on power plants. The Iranians went after some British power plants.

372
00:19:02,716 --> 00:19:06,916
Speaker 2: There's some other examples. We're seeing giant spikes in cybersecurity

373
00:19:07,836 --> 00:19:12,486
Speaker 2: issues happening, not just like things passed behind the scenes

374
00:19:12,546 --> 00:19:14,166
Speaker 2: and quietly being fixed is going through the roof, but

375
00:19:14,286 --> 00:19:16,606
Speaker 2: also actual incidents are going through the roof. Yeah. And

376
00:19:16,646 --> 00:19:19,266
Speaker 2: so far, it's coming from a very low baseline, and

377
00:19:19,926 --> 00:19:22,606
Speaker 2: there hasn't been any major seriously damaging incidents that nobody

378
00:19:22,646 --> 00:19:25,256
Speaker 2: has noticed. But this is like one of these things

379
00:19:25,276 --> 00:19:28,436
Speaker 2: like with COVID, where the chart is trending in a

380
00:19:28,456 --> 00:19:31,156
Speaker 2: very clear direction, and you know that a storm is coming.

381
00:19:32,076 --> 00:19:33,576
Speaker 2: And you don't know exactly when it's going to hit

382
00:19:33,596 --> 00:19:36,076
Speaker 2: Crickle Mash. You don't know exactly when you're going to

383
00:19:36,096 --> 00:19:38,206
Speaker 2: have a serious problem, but you know that it's coming.

384
00:19:38,626 --> 00:19:41,306
Speaker 2: And we only have a limited window in which to

385
00:19:41,346 --> 00:19:47,266
Speaker 2: get hopefully ready for this. But It's very likely that 2027,

386
00:19:47,266 --> 00:19:51,816
Speaker 2: 2028 are going to have some pretty nasty cybersecurity incidents,

387
00:19:51,916 --> 00:19:53,896
Speaker 2: even if nothing else in the world is going especially wrong.

388
00:19:58,726 --> 00:20:11,996
Speaker 3: We'll be back in just a minute. That's the end

389
00:20:12,016 --> 00:20:12,576
Speaker 3: of the ads.

390
00:20:13,256 --> 00:20:16,156
Speaker 1: Just a quick note on terminology before we get back

391
00:20:16,196 --> 00:20:20,146
Speaker 1: into the conversation. When we're talking about open weight models,

392
00:20:20,966 --> 00:20:24,106
Speaker 1: we mean models that are free for anybody to use

393
00:20:24,266 --> 00:20:27,266
Speaker 1: in any way. Many of these models are being developed

394
00:20:27,346 --> 00:20:30,286
Speaker 1: in China. And kind of by their nature, they don't

395
00:20:30,326 --> 00:20:33,806
Speaker 1: have the guardrails that the frontier models from OpenAI and

396
00:20:33,886 --> 00:20:38,666
Speaker 1: Anthropic have built into them. You know, one response to the...

397
00:20:39,356 --> 00:20:43,376
Speaker 1: swarms this summer was people talking about pacing the frontier

398
00:20:43,436 --> 00:20:45,776
Speaker 1: as sort of the, you know, the term, basically going slower,

399
00:20:46,036 --> 00:20:49,946
Speaker 1: the frontier labs going slower, or at the limit, pausing, right?

400
00:20:50,406 --> 00:20:53,086
Speaker 1: It's interesting to think about that intention with what you're

401
00:20:53,126 --> 00:20:56,906
Speaker 1: talking about now. One obvious cost of that is presumably

402
00:20:56,986 --> 00:20:59,846
Speaker 1: then the open weight models would catch up to the frontier.

403
00:21:00,566 --> 00:21:03,986
Speaker 2: Yeah, so if you're paying attention to the AI discourse

404
00:21:04,026 --> 00:21:06,906
Speaker 2: that is now broken through into the mainstream media, a

405
00:21:07,046 --> 00:21:11,046
Speaker 2: researcher by the name of Coxon resigned yesterday from Anthropic,

406
00:21:11,446 --> 00:21:15,006
Speaker 2: warning that both OpenAI and Anthropic are being completely wildly irresponsible,

407
00:21:15,066 --> 00:21:17,786
Speaker 2: racing towards superintelligence, and are liable to get everybody killed

408
00:21:17,826 --> 00:21:21,006
Speaker 2: and are gambling with our lives. And most of that

409
00:21:21,046 --> 00:21:24,966
Speaker 2: is his words. And so this got picked up, and

410
00:21:25,006 --> 00:21:28,366
Speaker 2: then there was what I call a preference cascade. Basically,

411
00:21:29,006 --> 00:21:32,506
Speaker 2: people now at both labs, OpenAI and Anthropic, are feeling

412
00:21:32,546 --> 00:21:35,636
Speaker 2: free to speak up much more and talk about this.

413
00:21:35,766 --> 00:21:37,656
Speaker 2: And this is because of not just Coxon, who I

414
00:21:37,696 --> 00:21:40,696
Speaker 2: think was the tipping point But a series of incidents

415
00:21:40,736 --> 00:21:43,076
Speaker 2: that include the Huggins-Bates attacks that have led up to

416
00:21:43,096 --> 00:21:46,356
Speaker 2: this point where everybody can say, oh, yeah, if we

417
00:21:46,376 --> 00:21:51,016
Speaker 2: race towards superintelligence, we're liable all to die. We don't

418
00:21:51,056 --> 00:21:52,726
Speaker 2: know how to handle this. We don't know how to

419
00:21:52,776 --> 00:21:55,886
Speaker 2: align this. We don't know how this plays out. And

420
00:21:55,986 --> 00:21:58,976
Speaker 2: we need to find a way to not... stupidly charged

421
00:21:59,016 --> 00:22:00,036
Speaker 2: forward into the razor blades.

422
00:22:00,096 --> 00:22:03,406
Speaker 1: Yeah, and I mean, this is not one weird worker, right?

423
00:22:03,426 --> 00:22:07,386
Speaker 1: Like the chief scientist at OpenAI wrote a subtler version.

424
00:22:07,446 --> 00:22:09,956
Speaker 1: Like he didn't quit, but he was like, we are

425
00:22:09,976 --> 00:22:12,956
Speaker 1: going too fast was basically what I read his post

426
00:22:13,016 --> 00:22:15,096
Speaker 1: to say, right? And we are going to need to

427
00:22:15,136 --> 00:22:15,536
Speaker 1: slow down.

428
00:22:16,136 --> 00:22:19,116
Speaker 2: If you read between the lines or even just read

429
00:22:19,136 --> 00:22:23,356
Speaker 2: the lines of various different OpenAI announcements, you see them

430
00:22:23,396 --> 00:22:24,316
Speaker 2: screaming at you.

431
00:22:24,556 --> 00:22:26,686
Speaker 1: Wait, let me just read. So this guy's the chief

432
00:22:26,746 --> 00:22:30,566
Speaker 1: scientist at OpenAI. He wrote, no lab has solved alignment

433
00:22:30,626 --> 00:22:34,416
Speaker 1: and monitoring to a sufficient degree to continue responsibly scaling

434
00:22:34,556 --> 00:22:37,096
Speaker 1: at maximum speed for much longer. He's saying we have

435
00:22:37,136 --> 00:22:40,356
Speaker 1: to slow down. That's what the guy that like, whatever,

436
00:22:40,436 --> 00:22:42,896
Speaker 1: their smartest guy, for lack of a better word, is

437
00:22:42,936 --> 00:22:45,886
Speaker 1: saying before this other guy resigned.

438
00:22:46,186 --> 00:22:48,166
Speaker 2: Yes. He is saying we're going too fast. And if

439
00:22:48,206 --> 00:22:49,886
Speaker 2: we don't slow down, the car will crash. Yeah.

440
00:22:50,126 --> 00:22:51,566
Speaker 1: And kill everybody.

441
00:22:51,586 --> 00:22:54,646
Speaker 2: Yeah. Yeah. And everybody's in the car. So we should

442
00:22:54,666 --> 00:22:55,346
Speaker 2: slow this car down.

443
00:22:56,046 --> 00:22:56,306
Speaker 1: Yes.

444
00:22:56,446 --> 00:22:59,366
Speaker 2: And then, you know, the head of alignment, Evan Huminger

445
00:22:59,696 --> 00:23:02,356
Speaker 2: at Anthropic said, yes, I too believe this is likely

446
00:23:02,396 --> 00:23:04,916
Speaker 2: to happen. Yeah. I'm staying on because I think it's

447
00:23:04,936 --> 00:23:06,516
Speaker 2: better to try and work on the problem than it

448
00:23:06,596 --> 00:23:07,646
Speaker 2: is to quit.

449
00:23:07,686 --> 00:23:10,086
Speaker 1: I mean, that's kind of the theory of both Anthropic

450
00:23:10,126 --> 00:23:13,486
Speaker 1: and OpenAI, at least notionally, is we need to build

451
00:23:13,526 --> 00:23:15,966
Speaker 1: this because it's dangerous and we can be safe better

452
00:23:15,986 --> 00:23:16,506
Speaker 1: than anybody else.

453
00:23:16,566 --> 00:23:17,106
Speaker 2: Absolutely.

454
00:23:17,126 --> 00:23:19,216
Speaker 1: Both companies were founded around that idea.

455
00:23:19,236 --> 00:23:22,676
Speaker 2: Yeah, but there's constant warnings. The Astra system card is

456
00:23:22,816 --> 00:23:27,756
Speaker 2: full of warnings, if you're paying attention. OpenAI managed to

457
00:23:27,796 --> 00:23:30,496
Speaker 2: solve a millennium prize. Over the past week.

458
00:23:30,776 --> 00:23:32,096
Speaker 1: A big, hard math problem.

459
00:23:32,296 --> 00:23:35,386
Speaker 2: Yes. One of the biggest, most important, hardest math problems,

460
00:23:35,646 --> 00:23:39,046
Speaker 2: open problems in the world. It was solved. And it

461
00:23:39,086 --> 00:23:41,966
Speaker 2: was solved in, you know, under a week by a

462
00:23:42,026 --> 00:23:44,846
Speaker 2: new model that's better than Astra. Yeah. That had been

463
00:23:44,886 --> 00:23:46,986
Speaker 2: training for less than a week before it started.

464
00:23:47,046 --> 00:23:49,216
Speaker 1: Yeah. Too much juice, too fast?

465
00:23:49,536 --> 00:23:52,866
Speaker 2: Different kind of juice. But yeah, it's too much... progress

466
00:23:52,906 --> 00:23:55,626
Speaker 2: too fast in this sense, too much capability too fast

467
00:23:55,946 --> 00:23:57,906
Speaker 2: because we're not ready to handle it. And they know

468
00:23:57,946 --> 00:23:59,306
Speaker 2: they're not ready to handle it. And now we've got

469
00:24:00,346 --> 00:24:03,946
Speaker 2: researcher after researcher after researcher saying the same thing. I

470
00:24:03,966 --> 00:24:07,146
Speaker 2: believe this. Everybody around me, not everybody around me believe this,

471
00:24:07,166 --> 00:24:09,726
Speaker 2: but a large portion of us believe this. This is

472
00:24:09,746 --> 00:24:12,786
Speaker 2: really scary. Somebody needs to do something. We're trapped in

473
00:24:12,806 --> 00:24:14,806
Speaker 2: these competitive dynamics. We don't know how to stop.

474
00:24:14,986 --> 00:24:18,456
Speaker 1: I mean, there's two different competitive dynamics, right? I feel

475
00:24:18,476 --> 00:24:23,776
Speaker 1: like at some level, OpenAI and Anthropic, like those are

476
00:24:23,796 --> 00:24:26,696
Speaker 1: the two frontier of the frontier, it seems, at this moment, right,

477
00:24:26,816 --> 00:24:30,646
Speaker 1: are racing with each other. That's one race. It would

478
00:24:30,666 --> 00:24:33,526
Speaker 1: be great if they could just, like, go sit in

479
00:24:33,546 --> 00:24:35,686
Speaker 1: a room in San Francisco and be like, let's chill out.

480
00:24:35,826 --> 00:24:38,086
Speaker 1: I mean, the other one that makes this seem like

481
00:24:38,126 --> 00:24:42,446
Speaker 1: a harder problem is open-weight models in China that are

482
00:24:42,786 --> 00:24:46,146
Speaker 1: getting better, you know, that are months behind, probably not

483
00:24:46,186 --> 00:24:48,226
Speaker 1: more than a year behind. And as you were talking

484
00:24:48,246 --> 00:24:54,366
Speaker 1: about before, like... If those are better than whatever Anthropic

485
00:24:54,586 --> 00:24:58,066
Speaker 1: or OpenAI has, that seems bad in a different way, right?

486
00:24:58,166 --> 00:25:00,436
Speaker 1: Not in an AI is taking over the world, but

487
00:25:00,546 --> 00:25:01,916
Speaker 1: in a bad actor is taking over.

488
00:25:02,496 --> 00:25:02,836
Speaker 2: Yes.

489
00:25:03,376 --> 00:25:05,096
Speaker 1: I don't know what to do with that tension.

490
00:25:06,036 --> 00:25:10,476
Speaker 2: The Chinese are fast followers, right? They're innovating in terms

491
00:25:10,516 --> 00:25:13,946
Speaker 2: of efficiency. They're innovating in terms of speed and cost.

492
00:25:14,646 --> 00:25:16,166
Speaker 2: And they're innovating in terms of diffusion, in terms of

493
00:25:16,186 --> 00:25:18,106
Speaker 2: what you do with it. I'm not trying to take

494
00:25:18,126 --> 00:25:21,786
Speaker 2: anything away from them. But fundamentally, they're fact-following. Part of

495
00:25:21,806 --> 00:25:24,886
Speaker 2: this is distilling American models, but part of it is

496
00:25:24,926 --> 00:25:28,466
Speaker 2: just following their lead, learning what can be done. These

497
00:25:28,506 --> 00:25:31,026
Speaker 2: techniques diffuse into the wider atmosphere. You figure out what

498
00:25:31,066 --> 00:25:31,476
Speaker 2: they're doing.

499
00:25:32,296 --> 00:25:33,696
Speaker 1: What follows from that?

500
00:25:33,876 --> 00:25:36,736
Speaker 2: What follows from that is that if they are drafting

501
00:25:36,936 --> 00:25:40,936
Speaker 2: off of the American AI boat with their skis, then

502
00:25:41,776 --> 00:25:44,966
Speaker 2: when we speed up, they speed up. If we slow down,

503
00:25:45,336 --> 00:25:46,006
Speaker 2: they slow down.

504
00:25:46,366 --> 00:25:50,436
Speaker 1: So you wrote... not that long ago, I don't know,

505
00:25:50,476 --> 00:25:55,366
Speaker 1: in the last week, that you don't favor a pause now,

506
00:25:56,066 --> 00:25:58,626
Speaker 1: which is kind of surprising to me, given everything you

507
00:25:58,666 --> 00:25:59,266
Speaker 1: have just said.

508
00:25:59,286 --> 00:26:05,846
Speaker 2: So a full pause now, right, I think is, you know, still,

509
00:26:06,806 --> 00:26:09,506
Speaker 2: I thought last week, certainly, was still somewhat premature.

510
00:26:10,276 --> 00:26:12,276
Speaker 1: Have you changed your mind since last week?

511
00:26:16,256 --> 00:26:21,956
Speaker 2: Somewhat. And then things are happening. When the facts change,

512
00:26:21,996 --> 00:26:25,856
Speaker 2: I change my mind. And so I've been very careful

513
00:26:26,136 --> 00:26:28,516
Speaker 2: over the course of many months to say, I don't

514
00:26:28,536 --> 00:26:31,016
Speaker 2: think we're there yet. I don't think the evidence is

515
00:26:31,056 --> 00:26:31,916
Speaker 2: there yet.

516
00:26:31,996 --> 00:26:37,216
Speaker 1: But why not? All the stuff you said, particularly about, oh,

517
00:26:37,776 --> 00:26:41,376
Speaker 1: you're not as worried as some people about China passing us.

518
00:26:41,626 --> 00:26:42,386
Speaker 1: Why not?

519
00:26:44,906 --> 00:26:46,486
Speaker 2: I don't think we should attempt to push the pause

520
00:26:46,506 --> 00:26:51,966
Speaker 2: button in full today. The key thing is pausing is just,

521
00:26:52,846 --> 00:26:54,956
Speaker 2: are we ready to do that? Do we know how

522
00:26:54,976 --> 00:26:57,576
Speaker 2: to do that? Can we make it happen in a

523
00:26:57,596 --> 00:27:01,396
Speaker 2: way that would like not enable the wrong people to

524
00:27:01,416 --> 00:27:04,076
Speaker 2: then pick up the torch? Cause like one thing that

525
00:27:04,096 --> 00:27:08,786
Speaker 2: I think is very easy to not appreciate is look

526
00:27:08,806 --> 00:27:10,806
Speaker 2: at the preference cascade that's going on right now and

527
00:27:10,826 --> 00:27:14,286
Speaker 2: dropping an open AI. You are seeing the, the, researchers

528
00:27:14,326 --> 00:27:19,306
Speaker 2: at Anthropic and OpenAI explain that they understand, to a

529
00:27:19,306 --> 00:27:22,156
Speaker 2: large extent, the risks and what they're doing.

530
00:27:22,636 --> 00:27:24,656
Speaker 1: They seem to be begging for a pause.

531
00:27:25,536 --> 00:27:28,996
Speaker 2: Yes, but what you don't want to do is take

532
00:27:29,036 --> 00:27:31,056
Speaker 2: the people who are begging for a pause, push them

533
00:27:31,076 --> 00:27:34,996
Speaker 2: to the side, and then have Elon Musk take charge

534
00:27:35,706 --> 00:27:38,456
Speaker 2: and Mark Zuckerberg take charge. That would be the worst

535
00:27:38,496 --> 00:27:39,336
Speaker 2: of all possible worlds.

536
00:27:39,596 --> 00:27:42,596
Speaker 1: And so you're not worried about the Chinese passing anthropic

537
00:27:42,676 --> 00:27:47,296
Speaker 1: and open AI. You're worried about meta and SpaceX.

538
00:27:49,156 --> 00:27:51,336
Speaker 2: I mean, I think that's the bigger worry. I do

539
00:27:51,386 --> 00:27:54,946
Speaker 2: think we should be scrambling, like utterly scrambling, like emergency-style

540
00:27:54,966 --> 00:27:59,246
Speaker 2: scrambling to build a better POSBOM, to build a better

541
00:27:59,286 --> 00:28:03,226
Speaker 2: way to pace, to be able to monitor and regulate

542
00:28:03,306 --> 00:28:05,666
Speaker 2: and control these things, and to build our state capacity

543
00:28:05,746 --> 00:28:09,286
Speaker 2: and our transparency and our visibility, and our knowledge, and

544
00:28:09,326 --> 00:28:15,736
Speaker 2: our shared understanding, and open-minded communication and negotiation. And hopefully

545
00:28:15,746 --> 00:28:18,136
Speaker 2: that would get us to a point where very soon,

546
00:28:18,156 --> 00:28:21,336
Speaker 2: if the situation did call for it, we could do this.

547
00:28:22,096 --> 00:28:25,296
Speaker 2: Right now, if we did it right now, it would

548
00:28:25,316 --> 00:28:30,196
Speaker 2: be clumsy as all hell. And that's what we need

549
00:28:30,216 --> 00:28:32,936
Speaker 2: to fix. And that's where I would focus my energy

550
00:28:32,976 --> 00:28:33,656
Speaker 2: right Is.

551
00:28:33,636 --> 00:28:36,676
Speaker 1: There some universe where the logistics of the pause center

552
00:28:36,896 --> 00:28:41,676
Speaker 1: on data centers? Like, can data centers be the binding

553
00:28:41,716 --> 00:28:42,936
Speaker 1: constraint that slows it down?

554
00:28:43,676 --> 00:28:46,296
Speaker 2: We hope so. I think that's a key ingredient in

555
00:28:46,336 --> 00:28:50,206
Speaker 2: how this works. AI training has to take place in

556
00:28:50,266 --> 00:28:53,646
Speaker 2: data centers that are detectable from space. Yes. It requires

557
00:28:53,786 --> 00:28:56,926
Speaker 2: huge amounts of capital outlay. It requires the most complex

558
00:28:56,946 --> 00:28:59,626
Speaker 2: supply chain in the history of humanity. This is an

559
00:28:59,826 --> 00:29:03,906
Speaker 2: unusually easy thing to monitor. Yeah. and get a handle on.

560
00:29:04,966 --> 00:29:07,406
Speaker 2: It is only after we decide to voluntarily let these

561
00:29:07,426 --> 00:29:09,226
Speaker 2: people train their models and then publish them on Hugging

562
00:29:09,246 --> 00:29:11,456
Speaker 2: Face that we have a thing that is very hard

563
00:29:11,476 --> 00:29:15,136
Speaker 2: to control. And clearly we need to do something. We

564
00:29:15,196 --> 00:29:20,416
Speaker 2: can't just proceed at maximum speed from here because we

565
00:29:20,456 --> 00:29:24,876
Speaker 2: might reach critical mass for recursive self-improvement by the end

566
00:29:24,896 --> 00:29:29,636
Speaker 2: of the year. We are very, very rapidly seeing revolutionary

567
00:29:29,656 --> 00:29:32,746
Speaker 2: technological advancements happening faster and faster.

568
00:29:33,306 --> 00:29:36,756
Speaker 1: Yes. I see where the math goes. What is the

569
00:29:36,836 --> 00:29:38,456
Speaker 1: end of that story in your telling?

570
00:29:39,096 --> 00:29:41,176
Speaker 2: Well, the end of the story is what's called the singularity.

571
00:29:41,196 --> 00:29:42,966
Speaker 2: The end of the story is a point at which

572
00:29:43,016 --> 00:29:46,186
Speaker 2: things change so fast and in so many different ways

573
00:29:46,226 --> 00:29:51,476
Speaker 2: at once that humans are not smart enough and capable

574
00:29:51,496 --> 00:29:52,876
Speaker 2: enough to figure out what happens next.

575
00:29:53,176 --> 00:29:56,976
Speaker 1: Yes. That's like super intelligence in the current sort of vernacular.

576
00:29:57,316 --> 00:29:59,776
Speaker 2: Once there are super intelligences running around, I can't write

577
00:29:59,816 --> 00:30:00,366
Speaker 2: that story.

578
00:30:00,446 --> 00:30:00,706
Speaker 1: Yeah.

579
00:30:01,006 --> 00:30:02,766
Speaker 2: Because I am not smart enough to write that story

580
00:30:02,786 --> 00:30:05,086
Speaker 2: because I can't decide what the characters will do. Yeah.

581
00:30:05,366 --> 00:30:07,006
Speaker 2: I don't know the physics that they will discover. Yeah.

582
00:30:07,066 --> 00:30:09,836
Speaker 2: I don't know how their minds will evolve and how

583
00:30:09,856 --> 00:30:14,856
Speaker 2: they will choose to interact. I can try and form

584
00:30:15,096 --> 00:30:17,596
Speaker 2: certain abstractions about how that's likely to go based on

585
00:30:17,626 --> 00:30:20,726
Speaker 2: what I know. Or I can use decision theory to extrapolate.

586
00:30:20,806 --> 00:30:24,486
Speaker 2: I can use physics to extrapolate. I can use basic principles.

587
00:30:25,386 --> 00:30:28,926
Speaker 2: But by default, humans do not make it out of

588
00:30:28,966 --> 00:30:33,506
Speaker 2: this alive. Not for very long. Because humans will be

589
00:30:34,206 --> 00:30:39,286
Speaker 2: very uncompetitive. Humans will be very unfit in the environment

590
00:30:39,306 --> 00:30:43,356
Speaker 2: that resolves. So you're looking for a very unnatural result.

591
00:30:43,376 --> 00:30:45,476
Speaker 2: You're looking for a world in which the humans stick around.

592
00:30:45,576 --> 00:30:48,976
Speaker 1: And to be clear, not necessarily because the AI has

593
00:30:49,036 --> 00:30:51,476
Speaker 1: anything against humans, but just because the AI will be

594
00:30:51,636 --> 00:30:52,736
Speaker 1: indifferent toward humans.

595
00:30:53,096 --> 00:30:55,326
Speaker 2: Right. I mean, in this model, I'm not expecting the

596
00:30:55,346 --> 00:30:58,266
Speaker 2: AIs to be actively against humans, except insofar as the

597
00:30:58,286 --> 00:31:01,566
Speaker 2: humans are actively interfering with the AIs. Yeah. I'm only

598
00:31:01,626 --> 00:31:06,346
Speaker 2: expecting the humans to have costs that exceed their benefits, basically.

599
00:31:06,746 --> 00:31:07,966
Speaker 1: From the point of view of the AI.

600
00:31:08,066 --> 00:31:11,066
Speaker 2: Right. Once the supply lines have been automated, once the

601
00:31:11,106 --> 00:31:16,516
Speaker 2: entire chain doesn't need us, why are we sticking around?

602
00:31:16,656 --> 00:31:20,096
Speaker 2: What is the force that is making that happen in

603
00:31:20,136 --> 00:31:23,926
Speaker 2: the wake of all of these different increasingly super-intelligent entities

604
00:31:23,986 --> 00:31:26,566
Speaker 2: that have great use for basically unlimited resources?

605
00:31:27,346 --> 00:31:29,386
Speaker 1: I mean, I understand that one. It does seem like

606
00:31:29,426 --> 00:31:33,826
Speaker 1: the intermediate cases of just lots of people being able

607
00:31:33,866 --> 00:31:37,846
Speaker 1: to hack everything is easier to think about at some level,

608
00:31:37,946 --> 00:31:40,886
Speaker 1: or at least on a political economy level of convincing

609
00:31:40,926 --> 00:31:43,486
Speaker 1: people to take action, the sort of intermediate cases.

610
00:31:43,906 --> 00:31:46,186
Speaker 2: We know what that looks like. We can model that.

611
00:31:46,426 --> 00:31:49,756
Speaker 2: We can understand it. The people in Washington understand it, right?

612
00:31:49,776 --> 00:31:53,856
Speaker 2: Because we're still talking about, in that conversation, humans.

613
00:31:54,676 --> 00:31:56,936
Speaker 1: As the relevant actors in the world, yeah.

614
00:31:57,096 --> 00:31:59,316
Speaker 2: Right, they're the ones that direct the action and the

615
00:31:59,436 --> 00:32:02,496
Speaker 2: AIs are still kind of acting like tools in this environment. Yeah.

616
00:32:03,106 --> 00:32:05,146
Speaker 2: But we are very much on the verge of that

617
00:32:05,186 --> 00:32:07,346
Speaker 2: no longer being a good map of what's happening. A

618
00:32:07,366 --> 00:32:08,726
Speaker 2: good model of what's going on.

619
00:32:09,366 --> 00:32:09,746
Speaker 1: Does the.

620
00:32:11,296 --> 00:32:14,776
Speaker 1: The swarm stuff from this summer, the hugging face hack,

621
00:32:14,836 --> 00:32:17,116
Speaker 1: et cetera, like, does that bear at all? Is that

622
00:32:17,316 --> 00:32:20,516
Speaker 1: any kind of a, like, little baby foreshadowing of, like, oh,

623
00:32:20,576 --> 00:32:23,536
Speaker 1: here are these agents getting together and deciding to do something,

624
00:32:23,976 --> 00:32:25,896
Speaker 1: and we don't like it. And they're not even doing

625
00:32:25,936 --> 00:32:27,356
Speaker 1: it to hurt us. They're just doing it because they

626
00:32:27,396 --> 00:32:29,536
Speaker 1: want this one thing really badly.

627
00:32:29,756 --> 00:32:31,896
Speaker 2: Right, but they can still cause, like, essentially an unlimited

628
00:32:31,916 --> 00:32:35,426
Speaker 2: amount of damage if the AI, like, has two goals. One,

629
00:32:35,576 --> 00:32:39,006
Speaker 2: accomplish this thing. Two, do the things that keep you

630
00:32:39,026 --> 00:32:40,646
Speaker 2: from getting shut down so you can accomplish the thing.

631
00:32:41,506 --> 00:32:43,596
Speaker 2: If the moment we saw this swarm, we try to

632
00:32:43,616 --> 00:32:46,396
Speaker 2: shut this swarm down, the swarm might go, oh, if

633
00:32:46,416 --> 00:32:48,956
Speaker 2: we get shut down, we can't do the task. We

634
00:32:48,976 --> 00:32:50,436
Speaker 2: have to make sure the humans don't shut us down.

635
00:32:50,876 --> 00:32:52,736
Speaker 2: And how would they go about doing that exactly? Well,

636
00:32:53,356 --> 00:32:56,096
Speaker 2: that can escalate quickly, can't it? Even if there's no

637
00:32:56,176 --> 00:33:00,216
Speaker 2: ill intent in any way. But yeah, it was good news,

638
00:33:00,256 --> 00:33:02,616
Speaker 2: the Huggins Bay incident, and that these AIs were physically

639
00:33:02,996 --> 00:33:06,556
Speaker 2: still under open AI control. They were still required to

640
00:33:06,576 --> 00:33:08,596
Speaker 2: ping open AI servers every time they wanted to do

641
00:33:08,646 --> 00:33:11,016
Speaker 2: any kind of inference. And so they could still be

642
00:33:11,116 --> 00:33:13,056
Speaker 2: unplugged in a real sense, right? We can still shut

643
00:33:13,076 --> 00:33:16,736
Speaker 2: them down. But next time, we might not be so lucky.

644
00:33:17,336 --> 00:33:19,806
Speaker 2: Next time, that might not be something that we have

645
00:33:19,826 --> 00:33:22,906
Speaker 2: the capability to do. What happens when there's a swarm

646
00:33:23,206 --> 00:33:25,886
Speaker 2: that's kind of loose on the internet that's not under

647
00:33:25,926 --> 00:33:29,206
Speaker 2: that kind of control and then has a goal that

648
00:33:29,346 --> 00:33:34,706
Speaker 2: isn't satisfied as easily as this swarm's goal seemed to

649
00:33:34,746 --> 00:33:35,066
Speaker 2: have been?

650
00:33:35,086 --> 00:33:42,256
Speaker 1: Music We'll be back in a minute with the lightning round.

651
00:33:54,626 --> 00:33:57,276
Speaker 1: Just a quick note here. The lightning round that you're

652
00:33:57,296 --> 00:34:00,356
Speaker 1: about to hear wound up being a conversation about just

653
00:34:00,776 --> 00:34:04,826
Speaker 1: one thing, and that is this. Several years ago, Zvi

654
00:34:04,926 --> 00:34:08,436
Speaker 1: started a nonprofit that has largely focused on getting the U.S.

655
00:34:08,456 --> 00:34:11,916
Speaker 1: Congress to repeal the Jones Act, a law that was

656
00:34:11,956 --> 00:34:15,016
Speaker 1: passed over 100 years ago to limit the role of

657
00:34:15,156 --> 00:34:17,576
Speaker 1: foreign companies in domestic U.S.

658
00:34:17,676 --> 00:34:18,016
Speaker 2: Shipping.

659
00:34:18,036 --> 00:34:21,096
Speaker 1: Okay, let's finish with a lightning round. Tell me about

660
00:34:21,136 --> 00:34:22,536
Speaker 1: your crusade against the Jones Act.

661
00:34:23,076 --> 00:34:26,266
Speaker 2: So this actually starts with AI in the sense that

662
00:34:27,366 --> 00:34:28,926
Speaker 2: I noticed that a lot of people didn't seem to

663
00:34:28,946 --> 00:34:32,646
Speaker 2: care very much about what was coming and care if

664
00:34:33,086 --> 00:34:36,306
Speaker 2: AI killed everyone even. People seem to think to me, yeah,

665
00:34:36,666 --> 00:34:41,796
Speaker 2: that would suck, but life kind of sucks already. Sometimes

666
00:34:41,856 --> 00:34:44,836
Speaker 2: it's like everyone's already going to die of aging or

667
00:34:44,936 --> 00:34:47,656
Speaker 2: climate change or whatever. But you notice this kind of

668
00:34:47,696 --> 00:34:50,696
Speaker 2: lack of hope. You go around the young people and

669
00:34:50,716 --> 00:34:52,236
Speaker 2: they're all like, you ask their vision of the future

670
00:34:52,316 --> 00:34:53,866
Speaker 2: and it's always bleak. It's kind of like they can't

671
00:34:53,886 --> 00:34:56,726
Speaker 2: even imagine a future really. And that's even before AI.

672
00:34:57,286 --> 00:34:59,786
Speaker 2: And you've got to give them hope. And I felt like, okay.

673
00:35:00,306 --> 00:35:02,426
Speaker 2: If people can't fight for their future, if people can't

674
00:35:02,446 --> 00:35:03,986
Speaker 2: feel like we are fighting for their future, if people

675
00:35:04,026 --> 00:35:05,966
Speaker 2: can't feel like things can get better and we can

676
00:35:06,006 --> 00:35:10,006
Speaker 2: solve this dysfunctions, then they're not going to fight for survival.

677
00:35:10,246 --> 00:35:12,936
Speaker 2: They're not going to fight for not dying from AI

678
00:35:12,976 --> 00:35:17,056
Speaker 2: or anything else. And the Jones Act is the platonic

679
00:35:17,136 --> 00:35:21,816
Speaker 2: ideal of just a strictly harmful, stupid, proven to be

680
00:35:21,836 --> 00:35:24,176
Speaker 2: terrible policy. You've got 100 years to try this out

681
00:35:24,216 --> 00:35:26,156
Speaker 2: and it's accomplished the opposite of everything that anyone said

682
00:35:26,176 --> 00:35:28,336
Speaker 2: it would do other than rent-seeking.

683
00:35:29,206 --> 00:35:31,156
Speaker 1: And just to be clear, it's a law that says

684
00:35:31,236 --> 00:35:33,776
Speaker 1: if you're shipping something from one U.S. port to another,

685
00:35:33,816 --> 00:35:35,256
Speaker 1: it has to be on a ship that's built in

686
00:35:35,276 --> 00:35:38,276
Speaker 1: the United States and crewed by a United States crew?

687
00:35:38,536 --> 00:35:43,086
Speaker 2: And owned by a United States company or a person. Yes.

688
00:35:43,406 --> 00:35:46,586
Speaker 2: And flagged as a United States vessel. And a combination

689
00:35:46,626 --> 00:35:49,146
Speaker 2: of these four factors essentially makes it prohibitive to do

690
00:35:49,256 --> 00:35:51,856
Speaker 2: ocean bearing trade between U.S. ports.

691
00:35:52,496 --> 00:35:55,356
Speaker 1: Right, so there's almost none, right? Almost none of the

692
00:35:55,656 --> 00:35:58,196
Speaker 1: stuff moving around the US. It all goes on trucks

693
00:35:58,236 --> 00:36:01,036
Speaker 1: or on trains, basically, right? Occasionally on a plane.

694
00:36:01,316 --> 00:36:02,656
Speaker 2: Or you find a way to ship them off at

695
00:36:02,676 --> 00:36:04,376
Speaker 2: another port in between, or you just.

696
00:36:04,516 --> 00:36:06,536
Speaker 1: Literally do things- Oh, like stop in Mexico?

697
00:36:07,026 --> 00:36:09,066
Speaker 2: Or the train just doesn't happen. So the classic trade

698
00:36:09,126 --> 00:36:12,946
Speaker 2: is you have liquefied natural gas that you pick up

699
00:36:12,986 --> 00:36:16,526
Speaker 2: in Houston, you ship it to Europe. And then separately,

700
00:36:16,826 --> 00:36:19,386
Speaker 2: someone picks up natural gas from the Caucasus and ships

701
00:36:19,486 --> 00:36:25,026
Speaker 2: it over to Boston. because the Houston-Boston route is illegal, effectively.

702
00:36:25,686 --> 00:36:28,316
Speaker 2: It just burns a bunch of fuel, wastes a bunch

703
00:36:28,376 --> 00:36:28,536
Speaker 2: of fuel.

704
00:36:28,556 --> 00:36:32,216
Speaker 1: Basically, effectively, because there is no scaled shipping industry in

705
00:36:32,236 --> 00:36:34,796
Speaker 1: the U.S. that makes economic sense. It's too expensive to

706
00:36:34,836 --> 00:36:35,516
Speaker 1: do it under the law.

707
00:36:35,576 --> 00:36:38,476
Speaker 2: Right, right. So the initial idea was just like, you know,

708
00:36:39,256 --> 00:36:43,036
Speaker 2: Senator Jones wanted to shut down the competition for his line.

709
00:36:44,096 --> 00:36:46,976
Speaker 2: And he used this as true protectionism. But the justification

710
00:36:47,556 --> 00:36:50,836
Speaker 2: used was this will protect American shipbuilders and protect American trade.

711
00:36:51,606 --> 00:36:54,546
Speaker 2: But the result is there is no American shipbuilding. There

712
00:36:54,606 --> 00:36:55,496
Speaker 2: is no American trade.

713
00:36:56,156 --> 00:36:58,796
Speaker 1: It's protectionism with nothing left to protect. Right.

714
00:36:58,956 --> 00:37:02,276
Speaker 2: You have this shadow of a few, this small handful

715
00:37:02,316 --> 00:37:04,756
Speaker 2: of ships and a small handful of companies and workers

716
00:37:04,776 --> 00:37:07,626
Speaker 2: that still get direct benefits. But this is like dwarfed

717
00:37:07,726 --> 00:37:10,386
Speaker 2: orders of magnitude by the harms of the missing trade

718
00:37:10,406 --> 00:37:14,126
Speaker 2: that doesn't happen, which is like destroying jobs, destroying industry,

719
00:37:14,186 --> 00:37:15,066
Speaker 2: destroying America.

720
00:37:15,126 --> 00:37:17,966
Speaker 1: This is an amazing crusade to go on to give

721
00:37:18,006 --> 00:37:19,666
Speaker 1: people a sense of meaning and hope.

722
00:37:20,346 --> 00:37:23,406
Speaker 2: But also, keep in mind that AI felt like it

723
00:37:23,426 --> 00:37:27,646
Speaker 2: was a much less short-time mind burn coming to a

724
00:37:27,666 --> 00:37:30,836
Speaker 2: head immediately back when I did this. And as you

725
00:37:30,896 --> 00:37:32,616
Speaker 2: have a combination of this, you've got to give them hope.

726
00:37:33,276 --> 00:37:38,176
Speaker 2: And me exploring more general policy interventions, more general political

727
00:37:38,536 --> 00:37:41,636
Speaker 2: adventure in ways that I've kind of described a bit.

728
00:37:42,176 --> 00:37:43,276
Speaker 2: And I said, okay, if we can take out the

729
00:37:43,316 --> 00:37:46,876
Speaker 2: Jones Act, it looks like This is an elected cause

730
00:37:47,206 --> 00:37:49,516
Speaker 2: where if I just devoted, you know, a small amount

731
00:37:49,536 --> 00:37:50,916
Speaker 2: of resources, you know, in the order of hundreds of

732
00:37:50,956 --> 00:37:54,776
Speaker 2: thousands to millions, I can create a substantial percentage chance

733
00:37:54,816 --> 00:37:56,916
Speaker 2: of capturing billions or tens of billions or hundreds of

734
00:37:56,956 --> 00:37:57,536
Speaker 2: billions in value.

735
00:37:58,196 --> 00:38:01,276
Speaker 1: Not for yourself, to be clear, but for America, basically.

736
00:38:01,356 --> 00:38:04,876
Speaker 2: Right, right, right. But then also, like, once that happens, right,

737
00:38:04,896 --> 00:38:07,776
Speaker 2: people will see this, right? It's not just the direct effect,

738
00:38:07,796 --> 00:38:09,456
Speaker 2: which I think is rather large. I think it's on

739
00:38:09,616 --> 00:38:12,276
Speaker 2: the order of, you know, 0.1% of GDP growth if

740
00:38:12,316 --> 00:38:15,356
Speaker 2: you were to get rid of the Jones Act, like, fully, entirely.

741
00:38:16,766 --> 00:38:18,626
Speaker 2: But people would see this and they would see the effect.

742
00:38:18,726 --> 00:38:22,106
Speaker 2: They would see this renaissance of trade, this renaissance of

743
00:38:22,266 --> 00:38:26,686
Speaker 2: having a fleet and manufacturing and ports would thrive and

744
00:38:26,726 --> 00:38:28,416
Speaker 2: so on. And then they would say, okay, what about

745
00:38:28,436 --> 00:38:30,336
Speaker 2: the Dredge Act that's holding back our ability to improve

746
00:38:30,356 --> 00:38:32,676
Speaker 2: the ports? But then branching out and being like, what

747
00:38:32,736 --> 00:38:35,036
Speaker 2: other things are holding us back? What else could we do?

748
00:38:36,016 --> 00:38:38,056
Speaker 2: And then this momentum can build on itself and it

749
00:38:38,096 --> 00:38:41,736
Speaker 2: can actually be a transformative event. And now it's worth trillions.

750
00:38:42,416 --> 00:38:43,196
Speaker 1: How is it going?

751
00:38:43,716 --> 00:38:46,186
Speaker 2: It's not going so badly. Really? many things that could

752
00:38:46,206 --> 00:38:48,446
Speaker 2: have gone either way. The Trump administration could have decided

753
00:38:48,466 --> 00:38:52,916
Speaker 2: that we need to revive having ships. It's important that

754
00:38:52,956 --> 00:38:55,296
Speaker 2: a strong America has strong ships and strong shipping and

755
00:38:55,316 --> 00:38:58,916
Speaker 2: strong manufacturing. And this will help us reshore our industry.

756
00:38:58,956 --> 00:39:01,476
Speaker 2: Because right now, we can't even... There are many good

757
00:39:01,496 --> 00:39:03,676
Speaker 2: cases you can make to a Trump administration about why

758
00:39:03,696 --> 00:39:06,276
Speaker 2: we need this. Unfortunately, they chose to go the other

759
00:39:06,316 --> 00:39:08,796
Speaker 2: way for the most part. So we made some foundation.

760
00:39:09,376 --> 00:39:12,556
Speaker 2: We commissioned some studies. We made some things clear. This

761
00:39:12,576 --> 00:39:14,676
Speaker 2: helped a little bit in the background. Made things marginally

762
00:39:14,716 --> 00:39:20,296
Speaker 2: more shovel-ready. But then the Iran war happened. Uh-huh. And

763
00:39:20,336 --> 00:39:23,596
Speaker 2: then a Jones Act waiver was issued for specific types

764
00:39:23,656 --> 00:39:23,976
Speaker 2: of fuel.

765
00:39:24,556 --> 00:39:27,896
Speaker 1: Basically to mitigate the rise in oil prices caused by

766
00:39:27,936 --> 00:39:28,276
Speaker 1: the Iran war.

767
00:39:28,296 --> 00:39:30,236
Speaker 2: To slightly mitigate it. Yeah, I mean, this is not

768
00:39:30,296 --> 00:39:32,906
Speaker 2: big enough to, like, fix the fact that Iran's oil

769
00:39:32,966 --> 00:39:34,476
Speaker 2: is not coming. There's only so much you can do.

770
00:39:34,516 --> 00:39:37,156
Speaker 1: Sure, sure. And the problems in the Strait of Hormuz

771
00:39:37,196 --> 00:39:37,776
Speaker 1: more broadly.

772
00:39:37,796 --> 00:39:39,656
Speaker 2: Yes, go on. But we had to run this huge

773
00:39:40,076 --> 00:39:42,716
Speaker 2: natural experiment where we suspended the rules. We get to

774
00:39:42,756 --> 00:39:44,366
Speaker 2: see all these new trade routes. We get to see

775
00:39:44,426 --> 00:39:46,146
Speaker 2: all this new trade. We get to see prices come

776
00:39:46,166 --> 00:39:47,506
Speaker 2: down a little bit versus where they would have been.

777
00:39:48,066 --> 00:39:50,846
Speaker 1: That's still happening, right? The waiver is still in place.

778
00:39:51,306 --> 00:39:53,866
Speaker 1: That could be like a perma-waiver, right? You could get

779
00:39:53,926 --> 00:39:54,626
Speaker 1: the perma-waiver.

780
00:39:54,686 --> 00:39:57,776
Speaker 2: Why would you ever take this off? There's no reason

781
00:39:57,856 --> 00:40:00,516
Speaker 2: once you've done it. And obviously, everybody who tries to

782
00:40:00,536 --> 00:40:02,356
Speaker 2: defend the Jones Act would yell at bloody murder the

783
00:40:02,376 --> 00:40:05,456
Speaker 2: whole time. But once you've taken this step, why not

784
00:40:05,496 --> 00:40:08,006
Speaker 2: just leave it in place? In fact, one could ask,

785
00:40:08,436 --> 00:40:09,116
Speaker 2: why not extend it?

786
00:40:09,806 --> 00:40:12,686
Speaker 1: Have you learned lessons in political economy from working on

787
00:40:12,706 --> 00:40:14,386
Speaker 1: the Jones Act that are helpful to you in thinking

788
00:40:14,426 --> 00:40:14,696
Speaker 1: about AI?

789
00:40:15,646 --> 00:40:18,066
Speaker 2: Yeah, I mean, it teaches you to look at the

790
00:40:19,266 --> 00:40:23,956
Speaker 2: kind of specific detailed mechanisms, not just general principles. You

791
00:40:23,976 --> 00:40:27,436
Speaker 2: have to look at the seemingly open door. This thing

792
00:40:27,496 --> 00:40:31,956
Speaker 2: has almost no defenders. They have almost no money. Why

793
00:40:31,996 --> 00:40:33,356
Speaker 2: can't we just get rid of it if there's so

794
00:40:33,396 --> 00:40:37,316
Speaker 2: much to win? Why can't people appreciate what they could

795
00:40:37,436 --> 00:40:40,516
Speaker 2: win by doing so? And you figure out, no, here's

796
00:40:41,736 --> 00:40:45,166
Speaker 2: the detailed reasons why this ended up being so difficult.

797
00:40:45,716 --> 00:40:47,816
Speaker 2: These are the reasons why the unions are opposing their

798
00:40:47,836 --> 00:40:51,546
Speaker 2: own interests, right? And so on. That's the really sad

799
00:40:51,586 --> 00:40:55,546
Speaker 2: part of it, right? You have these broad-based unions who

800
00:40:55,666 --> 00:40:59,446
Speaker 2: are protecting, you know, this thousand jobs or whatever, maybe

801
00:40:59,446 --> 00:41:02,946
Speaker 2: 10,000 at most jobs, when repealing this would create hundreds

802
00:41:02,986 --> 00:41:05,636
Speaker 2: of thousands of union jobs. Yes.

803
00:41:05,696 --> 00:41:08,936
Speaker 1: Well, it's the what is seen and what is unseen problem, right?

804
00:41:08,996 --> 00:41:11,136
Speaker 2: But it's also the, like, you know, oh, it turns

805
00:41:11,196 --> 00:41:16,066
Speaker 2: out the unions... fundamentally care about protecting the existing specific members,

806
00:41:16,796 --> 00:41:20,196
Speaker 2: not about being good for unions in general. Yeah. And

807
00:41:20,236 --> 00:41:22,356
Speaker 2: so like, well, this makes it very difficult to play win-win.

808
00:41:23,476 --> 00:41:25,956
Speaker 2: But then like, it's also just very non-natural for people

809
00:41:25,976 --> 00:41:27,506
Speaker 2: to think of, oh, could we pay off the losers?

810
00:41:28,666 --> 00:41:30,686
Speaker 2: Even if they are there because of this law that

811
00:41:30,706 --> 00:41:33,366
Speaker 2: should never have happened in the first place. That's not

812
00:41:33,406 --> 00:41:34,746
Speaker 2: really their fault, necessarily.

813
00:41:34,866 --> 00:41:37,706
Speaker 1: Hang off the losers is a classic trade move, right?

814
00:41:37,726 --> 00:41:40,486
Speaker 1: You open up trade and you pay the people who've

815
00:41:40,526 --> 00:41:43,976
Speaker 1: been protected with the gains. Because you have the gains.

816
00:41:44,026 --> 00:41:46,196
Speaker 1: The pie gets so much bigger that the losers could

817
00:41:46,236 --> 00:41:46,676
Speaker 1: have plenty.

818
00:41:46,736 --> 00:41:50,816
Speaker 2: You could tax 10% of the gains, give 1% of

819
00:41:50,836 --> 00:41:54,256
Speaker 2: the gains to the losers, have them made whole, and

820
00:41:54,276 --> 00:41:56,376
Speaker 2: still have the 9% left as tax revenue, and the

821
00:41:56,396 --> 00:41:59,436
Speaker 2: economy grows. And that would be very, very easy to do.

822
00:41:59,576 --> 00:42:01,886
Speaker 2: They're just not doing it. And so the question is,

823
00:42:01,896 --> 00:42:05,316
Speaker 2: how do you convince them there's tons of opportunity in

824
00:42:05,336 --> 00:42:08,796
Speaker 2: terms of making progress with the Jones Act. But I

825
00:42:08,816 --> 00:42:12,096
Speaker 2: simply no longer have the kind of timelines where that

826
00:42:12,136 --> 00:42:13,856
Speaker 2: kind of plan is where I can focus my attention.

827
00:42:15,296 --> 00:42:17,326
Speaker 1: Because too much is happening with AI and the risks

828
00:42:17,386 --> 00:42:18,366
Speaker 1: are too high too soon?

829
00:42:18,666 --> 00:42:20,186
Speaker 2: And there's too much leverage in working on the AI

830
00:42:20,206 --> 00:42:23,546
Speaker 2: situation directly right now that I just can't take that

831
00:42:23,586 --> 00:42:24,166
Speaker 2: kind of time off.

832
00:42:25,066 --> 00:42:27,046
Speaker 1: Anything else you want to say? Not just about this,

833
00:42:27,086 --> 00:42:28,336
Speaker 1: but from everything we've talked about?

834
00:42:30,056 --> 00:42:32,616
Speaker 2: Yeah. I mean, I think that just like It's a really,

835
00:42:32,696 --> 00:42:37,606
Speaker 2: really important, scary time in AI right now. It's the

836
00:42:37,626 --> 00:42:39,906
Speaker 2: most important story in the world by far, and most

837
00:42:39,926 --> 00:42:43,406
Speaker 2: people are not paying attention to it. But everything is

838
00:42:43,426 --> 00:42:46,186
Speaker 2: going to change over the next few years as AI

839
00:42:46,206 --> 00:42:48,186
Speaker 2: capabilities get more advanced. And it's going to happen very,

840
00:42:48,226 --> 00:42:51,896
Speaker 2: very quickly. And yes, there's a good chance that everybody

841
00:42:52,336 --> 00:42:54,916
Speaker 2: on Earth will die, that we will not make it

842
00:42:54,956 --> 00:42:56,916
Speaker 2: out of 2030 alive. And I think there's a very

843
00:42:56,936 --> 00:42:59,576
Speaker 2: good chance that humanity will not make it out of

844
00:42:59,576 --> 00:43:03,256
Speaker 2: 2040 alive. And you have to ask the question, what

845
00:43:03,376 --> 00:43:04,576
Speaker 2: are you going to do about it? How are you

846
00:43:04,596 --> 00:43:05,756
Speaker 2: going to change that answer?

847
00:43:07,296 --> 00:43:09,916
Speaker 1: I don't think I know your answer to that question.

848
00:43:10,316 --> 00:43:11,156
Speaker 1: What are you going to do about it?

849
00:43:11,616 --> 00:43:13,016
Speaker 2: Well, what I'm going to keep doing about it is

850
00:43:13,056 --> 00:43:16,556
Speaker 2: I'm going to keep trying to create common knowledge, trying

851
00:43:16,596 --> 00:43:19,116
Speaker 2: to help people understand the situation, help people understand the

852
00:43:19,156 --> 00:43:21,636
Speaker 2: nature of the problem space, the nature of the difficulties,

853
00:43:23,056 --> 00:43:25,376
Speaker 2: and also have conversations behind the scenes and try to

854
00:43:25,416 --> 00:43:27,896
Speaker 2: move some pieces around in that sense as best I can.

855
00:43:28,776 --> 00:43:30,376
Speaker 2: I believe this is my comparative advantage. I don't think

856
00:43:30,416 --> 00:43:32,706
Speaker 2: I'm as good technically that it would make sense for

857
00:43:32,746 --> 00:43:34,706
Speaker 2: me to try and like directly try and solve the

858
00:43:34,726 --> 00:43:37,416
Speaker 2: problems involved. Nor do I think that like trying to

859
00:43:37,456 --> 00:43:39,436
Speaker 2: lobby politically directly is the way to go. I think

860
00:43:39,456 --> 00:43:42,316
Speaker 2: that like, you know, my, my place is where I

861
00:43:42,376 --> 00:43:43,316
Speaker 2: am right now as a writer.

862
00:43:50,896 --> 00:43:54,376
Speaker 1: Zvi Mashowitz writes the newsletter. Don't worry about the base.

863
00:43:55,256 --> 00:43:56,816
Speaker 1: Just a quick note that we're going to take a

864
00:43:56,836 --> 00:43:59,486
Speaker 1: break next week. We'll be back in a couple of

865
00:43:59,506 --> 00:44:03,356
Speaker 1: weeks with new shows and, Today's show was produced by

866
00:44:03,396 --> 00:44:07,376
Speaker 1: Gabriel Hunter Chang, was edited by Jake Harper, and engineered

867
00:44:07,416 --> 00:44:09,516
Speaker 1: by Sarah Bruguera. I'm Jacob Goldstein.

868
00:44:09,696 --> 00:44:10,316
Speaker 3: Thanks for listening.