1
00:00:00,240 --> 00:00:07,200
Speaker 1: Bloomberg Audio Studios, podcasts, radio news.

2
00:00:08,720 --> 00:00:11,920
Speaker 2: The Iran War showed little signs of de escalating this week,

3
00:00:12,119 --> 00:00:15,760
Speaker 2: with Israeli forces targeting nuclear development sites in Iran, the

4
00:00:15,880 --> 00:00:19,120
Speaker 2: US attacking Iranian mind laying vessels in the Strait of Hormuz,

5
00:00:19,480 --> 00:00:24,040
Speaker 2: and Iranian drones striking Dubai International Airport. President Trump and

6
00:00:24,079 --> 00:00:27,440
Speaker 2: Iran's new Supreme leader each vowed Thursday to keep up

7
00:00:27,440 --> 00:00:30,840
Speaker 2: their respective military campaigns amid mounting oil prices.

8
00:00:31,000 --> 00:00:33,920
Speaker 1: Every day we lose about twenty percent of global oil supplies.

9
00:00:34,040 --> 00:00:36,080
Speaker 3: We are just shy of one hundred dollars a barrel

10
00:00:36,200 --> 00:00:36,919
Speaker 3: on Brent now.

11
00:00:37,280 --> 00:00:39,760
Speaker 2: In a post on truth Social Trump wrote that preventing

12
00:00:39,760 --> 00:00:42,599
Speaker 2: Iran from threatening the Middle East and having nuclear weapons

13
00:00:42,800 --> 00:00:45,160
Speaker 2: was far more important to him than the cost of oil.

14
00:00:45,680 --> 00:00:50,040
Speaker 2: Iran's new Supreme Leader, Iotolamus Stabahamane, said Iran would continue

15
00:00:50,040 --> 00:00:52,920
Speaker 2: its blockade of the Strait of Hormuz, adding that he'd

16
00:00:52,960 --> 00:00:56,680
Speaker 2: consider opening other fronts if the war continues. But the

17
00:00:56,720 --> 00:00:59,840
Speaker 2: war has already expanded it to an important new front.

18
00:01:00,240 --> 00:01:03,640
Speaker 2: How it's being waged with AI enabled weaponry. In just

19
00:01:03,680 --> 00:01:06,080
Speaker 2: the first twenty four hours of the Iran War, the

20
00:01:06,200 --> 00:01:09,319
Speaker 2: US hit more than one thousand targets. According to the

21
00:01:09,319 --> 00:01:12,399
Speaker 2: commander of US Forces, that is double the scale of

22
00:01:12,400 --> 00:01:14,880
Speaker 2: the assault that opened the Iraq War in two thousand

23
00:01:14,880 --> 00:01:15,160
Speaker 2: and three.

24
00:01:15,520 --> 00:01:19,440
Speaker 3: The decision to wage a campaign like this is intimately

25
00:01:19,520 --> 00:01:24,080
Speaker 3: linked to how remotely the US can pursue this war.

26
00:01:24,360 --> 00:01:27,520
Speaker 2: That's Bloomberg National security and tech reporter Katrina Manson.

27
00:01:27,680 --> 00:01:31,720
Speaker 3: It's clearly really important for Donald Trump not to send

28
00:01:31,840 --> 00:01:34,720
Speaker 3: US troops in on the ground, so conducting an air

29
00:01:34,760 --> 00:01:39,959
Speaker 3: war with the help of AI suddenly becomes critical to

30
00:01:40,120 --> 00:01:42,160
Speaker 3: his own pursuit of a war.

31
00:01:43,680 --> 00:01:46,880
Speaker 2: The US's first strikes on Iran came hours after the

32
00:01:46,920 --> 00:01:50,680
Speaker 2: Pentagon severed its contract with Anthropic inn AI company that

33
00:01:50,800 --> 00:01:53,880
Speaker 2: had been working directly with the US military. The Pentagon

34
00:01:53,920 --> 00:01:57,920
Speaker 2: has formally notified Anthropic that it's deemed the artificial intelligence

35
00:01:57,960 --> 00:02:01,040
Speaker 2: company and its products a supply chain risk to the

36
00:02:01,120 --> 00:02:03,840
Speaker 2: United States. Pentagoon has said it needs the ability to

37
00:02:03,960 --> 00:02:09,280
Speaker 2: use Claude for quote all lawful purposes. Anthropic, which is

38
00:02:09,360 --> 00:02:12,560
Speaker 2: best known for its clawed AI tool, wanted assurances from

39
00:02:12,600 --> 00:02:15,920
Speaker 2: the US government that its technology would not make lethal

40
00:02:15,960 --> 00:02:18,920
Speaker 2: decisions on its own or help to conduct mass surveillance

41
00:02:18,960 --> 00:02:22,400
Speaker 2: on Americans. Anthropics ouster and a move by its rival

42
00:02:22,480 --> 00:02:25,400
Speaker 2: Open Ai to start working with the Department of Defense

43
00:02:25,680 --> 00:02:29,440
Speaker 2: have struck a nerve. Some Americans are a plauding Anthropics

44
00:02:29,440 --> 00:02:33,360
Speaker 2: decision Claude went down last week because of unprecedented demand.

45
00:02:34,000 --> 00:02:36,760
Speaker 2: It's the latest turn in a long simmering debate over

46
00:02:36,800 --> 00:02:39,840
Speaker 2: how the US military works with Silicon Valley companies and

47
00:02:39,919 --> 00:02:43,040
Speaker 2: how those tech titans contribute to the development of AI

48
00:02:43,120 --> 00:02:46,520
Speaker 2: weapons systems. Katrina took a deep dive into that story,

49
00:02:46,680 --> 00:02:49,000
Speaker 2: and it's a big part of her forthcoming book on

50
00:02:49,040 --> 00:02:50,680
Speaker 2: the US military and AI.

51
00:02:51,000 --> 00:02:53,800
Speaker 3: AI may be good for scale, it may be good

52
00:02:53,840 --> 00:02:58,000
Speaker 3: for speed, but is it as good for accuracy and

53
00:02:58,200 --> 00:03:01,280
Speaker 3: saving lives as these claims have been made in the past.

54
00:03:01,760 --> 00:03:02,880
Speaker 3: We don't know yet.

55
00:03:06,480 --> 00:03:08,280
Speaker 2: I'm David Gera and this is the big take from

56
00:03:08,320 --> 00:03:11,400
Speaker 2: Bloomberg News today on the show, as Anthropic and the

57
00:03:11,400 --> 00:03:15,000
Speaker 2: Pentagon feud. How is the US integrating AI into its

58
00:03:15,000 --> 00:03:18,679
Speaker 2: war fighting machine, How is AI enabled weaponry being regulated?

59
00:03:19,040 --> 00:03:21,040
Speaker 2: And is the tech as good as it needs to

60
00:03:21,080 --> 00:03:29,560
Speaker 2: be with so many lives at risk? About five years ago,

61
00:03:29,840 --> 00:03:33,000
Speaker 2: Dario Amide and his sister left their jobs at OpenAI

62
00:03:33,120 --> 00:03:36,400
Speaker 2: to found Anthropic, an AI company meant to enforce strong

63
00:03:36,440 --> 00:03:41,840
Speaker 2: safety guardrails. Since then, Anthropic has achieved mainstream success with Claude.

64
00:03:42,200 --> 00:03:45,120
Speaker 2: It's even managed to partner with the US government, but

65
00:03:45,240 --> 00:03:47,840
Speaker 2: that all changed at the start of the year. Bloomberg

66
00:03:47,920 --> 00:03:51,000
Speaker 2: Senior editor Mike Shepherd has been following the saga closely.

67
00:03:51,360 --> 00:03:56,480
Speaker 1: The Pentagon began reviewing how it wanted to roll out

68
00:03:56,960 --> 00:04:02,120
Speaker 1: artificial intelligence across the Armed Force, and in January they

69
00:04:02,200 --> 00:04:06,320
Speaker 1: put out an order from Defense Secretary Pete Hegseth saying

70
00:04:06,360 --> 00:04:10,600
Speaker 1: that they needed to accelerate adoption across the Defense Department,

71
00:04:10,680 --> 00:04:13,720
Speaker 1: which he refers to as the Department of War. He

72
00:04:13,800 --> 00:04:17,080
Speaker 1: even put up posters around the building saying the government

73
00:04:17,120 --> 00:04:20,480
Speaker 1: wants you to use AI as part of it. They

74
00:04:20,480 --> 00:04:24,680
Speaker 1: included language that said they did not want to be

75
00:04:24,760 --> 00:04:29,240
Speaker 1: bound by any usage restrictions that might come from an

76
00:04:29,279 --> 00:04:32,520
Speaker 1: AI provider of any stripe. They didn't single out Anthropic,

77
00:04:32,880 --> 00:04:36,200
Speaker 1: but Anthropic is not only one of the leading AI

78
00:04:36,480 --> 00:04:39,440
Speaker 1: developers and providers in the world, they are one that

79
00:04:39,480 --> 00:04:42,640
Speaker 1: has stood out for what many see as its safety

80
00:04:42,680 --> 00:04:46,440
Speaker 1: first stance and its adherence to principles of trying to

81
00:04:46,880 --> 00:04:50,359
Speaker 1: develop AI with a mind toward avoiding some of the

82
00:04:50,360 --> 00:04:53,880
Speaker 1: worst case scenarios. Now. For its part, Aanthropic has expressed

83
00:04:53,920 --> 00:04:58,239
Speaker 1: continued interest in working with the military so long as

84
00:04:58,520 --> 00:05:03,360
Speaker 1: the Pentagon A military officials abide by those usage restrictions.

85
00:05:03,360 --> 00:05:06,039
Speaker 1: And the two that have really been a redline for

86
00:05:06,080 --> 00:05:09,960
Speaker 1: the company concern the use of its AI technology for

87
00:05:10,400 --> 00:05:16,080
Speaker 1: mass surveillance of Americans domestically and then also fully autonomous

88
00:05:16,120 --> 00:05:19,640
Speaker 1: weapons the climate they as the developers of the technology,

89
00:05:19,680 --> 00:05:23,200
Speaker 1: insist that they don't have full confidence that it's ready

90
00:05:23,240 --> 00:05:24,599
Speaker 1: for those uses safely.

91
00:05:25,839 --> 00:05:28,120
Speaker 2: Katrina, you have a book coming out later this month

92
00:05:28,160 --> 00:05:31,480
Speaker 2: on the Pentagon's efforts to integrate AI into warfare. I'm

93
00:05:31,520 --> 00:05:34,159
Speaker 2: curious when that initiative started. When did the Pentagon begin

94
00:05:34,839 --> 00:05:36,160
Speaker 2: looking into this an interest.

95
00:05:36,560 --> 00:05:39,520
Speaker 3: The pentagonice to say it's been developing AI for sixty years.

96
00:05:39,560 --> 00:05:42,080
Speaker 3: But the real project that I think we can set

97
00:05:42,120 --> 00:05:46,200
Speaker 3: the timerby is Project Maven, which started in twenty seventeen,

98
00:05:46,560 --> 00:05:49,320
Speaker 3: and it was an effort that really spoke to America's

99
00:05:49,320 --> 00:05:53,159
Speaker 3: concern about falling behind China. The US began to realize

100
00:05:53,200 --> 00:05:55,640
Speaker 3: it was using old tech and that this new age

101
00:05:55,640 --> 00:06:00,839
Speaker 3: of warfare was coming that would require robots, autonomy, and AI,

102
00:06:01,360 --> 00:06:05,239
Speaker 3: and Project Maven was this effort to experiment with AI.

103
00:06:05,600 --> 00:06:08,480
Speaker 3: At the time, the cutting edge was computer vision, things

104
00:06:08,520 --> 00:06:12,360
Speaker 3: that could identify objects on a video feed taken by

105
00:06:12,440 --> 00:06:16,880
Speaker 3: drones and process that quicker. But the people I spoke

106
00:06:16,920 --> 00:06:19,960
Speaker 3: to also explained to me that all along they imagined

107
00:06:20,080 --> 00:06:21,560
Speaker 3: that this would help with targeting.

108
00:06:22,080 --> 00:06:24,720
Speaker 2: You write about the software that comes out of Project

109
00:06:24,720 --> 00:06:28,120
Speaker 2: man Maven smart System, which is made by Palenteer and

110
00:06:28,160 --> 00:06:32,240
Speaker 2: incorporated technology from Amazon and Microsoft and others. Is this

111
00:06:32,240 --> 00:06:35,040
Speaker 2: something the Defense Department thought they could do in house

112
00:06:35,200 --> 00:06:37,160
Speaker 2: or was it always something that they needed the cooperation

113
00:06:37,200 --> 00:06:38,279
Speaker 2: of private industry.

114
00:06:38,760 --> 00:06:41,520
Speaker 3: It's a real core point that obviously the Pentagon is

115
00:06:41,560 --> 00:06:45,960
Speaker 3: still struggling with. For this very advanced tech, This recognition

116
00:06:46,040 --> 00:06:48,159
Speaker 3: that has been happening over the past ten years that

117
00:06:48,600 --> 00:06:51,640
Speaker 3: warfare is going to be what some call software defined,

118
00:06:52,200 --> 00:06:56,600
Speaker 3: they need commercial companies, and so when Project Maven happened

119
00:06:57,080 --> 00:06:59,080
Speaker 3: for the first time, really they were going outside the

120
00:06:59,120 --> 00:07:02,880
Speaker 3: traditional primes as they're known, the traditional defense contractors and

121
00:07:02,920 --> 00:07:07,080
Speaker 3: looking at companies like Google, Microsoft, and Amazon, but other

122
00:07:07,160 --> 00:07:10,240
Speaker 3: tiny startups as well who were just beginning to experiment

123
00:07:10,320 --> 00:07:15,200
Speaker 3: with fascinating algorithms that could detect images on wedding cakes

124
00:07:16,240 --> 00:07:19,520
Speaker 3: and then reapplying that for the tools of war, and

125
00:07:19,920 --> 00:07:25,360
Speaker 3: that transition was incredibly uncomfortable, clunky, difficult, and they're backfired

126
00:07:25,360 --> 00:07:29,040
Speaker 3: at one point rather spectacularly when in twenty eighteen, Google

127
00:07:29,080 --> 00:07:32,480
Speaker 3: workers discovered that their company was working on Project Maven,

128
00:07:32,520 --> 00:07:36,440
Speaker 3: and they feared that their tech and their traditions for

129
00:07:36,600 --> 00:07:39,320
Speaker 3: them could be subverted into what they ended up calling

130
00:07:39,400 --> 00:07:42,920
Speaker 3: in a protest letter the business of war, and the

131
00:07:42,960 --> 00:07:45,920
Speaker 3: parallel with Anthropic today isn't quite the same. The CEO

132
00:07:46,040 --> 00:07:49,080
Speaker 3: of Anthropic all Along has said he believes in national

133
00:07:49,080 --> 00:07:52,800
Speaker 3: security work more than any other AI lab. Anthropic has

134
00:07:52,880 --> 00:07:55,920
Speaker 3: lent into classified work, and the classified cloud is where

135
00:07:56,280 --> 00:08:00,680
Speaker 3: Penticon does its fighting, so they've been involved with lethal

136
00:08:00,720 --> 00:08:05,720
Speaker 3: operations in some way for more than a year. What

137
00:08:05,840 --> 00:08:08,680
Speaker 3: I think is different is he had these red lines,

138
00:08:08,760 --> 00:08:12,160
Speaker 3: and as he is trying to uphold them, tech workers

139
00:08:12,160 --> 00:08:14,360
Speaker 3: in other companies are really paying note now.

140
00:08:14,640 --> 00:08:19,240
Speaker 2: Katrina ethics are so central to this program, and I'm

141
00:08:19,320 --> 00:08:23,200
Speaker 2: curious how defense officials addressed those ethics of seeding life

142
00:08:23,280 --> 00:08:27,240
Speaker 2: or death decision making to machines to artificial intelligence.

143
00:08:27,840 --> 00:08:30,840
Speaker 3: They would argue, I think very current now to the

144
00:08:30,880 --> 00:08:35,960
Speaker 3: operations that are using AI tools against Iran, that humans

145
00:08:36,000 --> 00:08:40,120
Speaker 3: are still making the decisions. But in a very interesting

146
00:08:40,480 --> 00:08:44,520
Speaker 3: presentation by the Commander of US Forces just this week,

147
00:08:45,000 --> 00:08:48,960
Speaker 3: they've made clear that they've hit five thousand, five hundred targets.

148
00:08:49,440 --> 00:08:54,280
Speaker 3: That speed is exactly why they want AI and that scope,

149
00:08:55,000 --> 00:08:57,600
Speaker 3: and the Pentagon has had to work very hard thinking

150
00:08:57,600 --> 00:09:01,319
Speaker 3: through precision. The claim, of course is that US weapons

151
00:09:01,679 --> 00:09:04,120
Speaker 3: are the most precise in the world. But where you

152
00:09:04,280 --> 00:09:07,719
Speaker 3: decide to put them, that is where America has had

153
00:09:07,760 --> 00:09:11,880
Speaker 3: problems and the scale at which it's shooting at areas

154
00:09:11,920 --> 00:09:15,360
Speaker 3: where we know civilians are present, because Scentcom is warning

155
00:09:15,400 --> 00:09:18,240
Speaker 3: civilians to stay home, telling them to stay away from ports.

156
00:09:19,160 --> 00:09:22,080
Speaker 3: There is a margin of error. There's a decision to

157
00:09:22,240 --> 00:09:24,960
Speaker 3: accept or not accept a certain amount of collateral damage.

158
00:09:24,960 --> 00:09:27,880
Speaker 3: They haven't made public what margin they set on that

159
00:09:27,920 --> 00:09:31,679
Speaker 3: collateral damage, but these are key ethical decisions in The

160
00:09:32,240 --> 00:09:35,600
Speaker 3: targeting process is a multi stage effort where people feed

161
00:09:35,640 --> 00:09:39,280
Speaker 3: in and eventually a commander makes a decision to sign off.

162
00:09:39,720 --> 00:09:42,840
Speaker 3: But as you involve AI, that process speeds up. The

163
00:09:42,920 --> 00:09:49,360
Speaker 3: decision making time is reduced, and fears for things like

164
00:09:49,760 --> 00:09:54,360
Speaker 3: automation bias or the algorithms themselves going wrong. Hallucinating or

165
00:09:54,440 --> 00:09:58,640
Speaker 3: drifting algorithms tend to naturally get worse over time. All

166
00:09:58,679 --> 00:10:01,480
Speaker 3: of that has really yet to be worked out. Now

167
00:10:01,600 --> 00:10:04,000
Speaker 3: the US has got a policy for that. It's a directive.

168
00:10:04,400 --> 00:10:06,480
Speaker 3: Some people to me frame it more as a process

169
00:10:06,920 --> 00:10:09,640
Speaker 3: than a rule, but it does say that there needs

170
00:10:09,679 --> 00:10:13,000
Speaker 3: to be human judgment over the appropriate use of force. Well,

171
00:10:13,000 --> 00:10:16,720
Speaker 3: that's supervision, that's not necessarily making a decision. And so

172
00:10:16,840 --> 00:10:20,040
Speaker 3: I think from all the people I've spoken to, that role,

173
00:10:20,120 --> 00:10:24,240
Speaker 3: that human role is reducing, and Sentcom has been particularly

174
00:10:24,320 --> 00:10:27,280
Speaker 3: proud this week the commander to say that AI is

175
00:10:27,320 --> 00:10:31,600
Speaker 3: helping them reduce operations decisions from what used to be

176
00:10:32,200 --> 00:10:36,160
Speaker 3: days and hours two seconds.

177
00:10:37,000 --> 00:10:39,960
Speaker 2: After the break. How good is this technology really? We

178
00:10:40,080 --> 00:10:42,719
Speaker 2: hear about one test where things went awry, and we

179
00:10:42,840 --> 00:10:56,920
Speaker 2: dig into how lawmakers are responding. Proponents of AI enabled

180
00:10:56,920 --> 00:10:59,760
Speaker 2: weapons say the technology can improve decision making on the

181
00:10:59,760 --> 00:11:03,600
Speaker 2: back and put fewer troops at risk, which could save lives.

182
00:11:04,080 --> 00:11:07,400
Speaker 2: Its promise of speed makes it imperative to national security

183
00:11:07,440 --> 00:11:10,600
Speaker 2: experts who see it as essential in the widening AI

184
00:11:10,760 --> 00:11:14,280
Speaker 2: arms race with China, but Bloomberg's Katrina Manson points out

185
00:11:14,520 --> 00:11:16,160
Speaker 2: it also comes with risks.

186
00:11:16,520 --> 00:11:21,800
Speaker 3: AI is this fundamentally unpredictable black box technology. So it's

187
00:11:21,960 --> 00:11:24,560
Speaker 3: brilliant at bringing a lot of data to bear. It

188
00:11:24,679 --> 00:11:26,720
Speaker 3: just might be the wrong data, and it might be

189
00:11:26,840 --> 00:11:29,320
Speaker 3: organized in the wrong way. And if you're not giving

190
00:11:29,440 --> 00:11:32,400
Speaker 3: enough time to checking, or if you don't understand where

191
00:11:32,440 --> 00:11:34,880
Speaker 3: it can make mistakes and you put it into critical

192
00:11:34,920 --> 00:11:40,160
Speaker 3: a place in your own system, you've made yourself incredibly fallible. Now,

193
00:11:40,240 --> 00:11:43,360
Speaker 3: from the US perspective, that brings a couple of problems.

194
00:11:43,440 --> 00:11:46,360
Speaker 3: One is you may end up killing civilians, which is

195
00:11:46,400 --> 00:11:51,440
Speaker 3: against US policy. Certainly deliberate targeting inadvertent targeting under the

196
00:11:51,559 --> 00:11:54,840
Speaker 3: rule of law has to be proportionate. But the second

197
00:11:54,880 --> 00:11:56,560
Speaker 3: thing is you can also hurt your own troops and

198
00:11:56,640 --> 00:12:00,160
Speaker 3: allied troops. And the US has a history of grappling

199
00:12:00,600 --> 00:12:05,319
Speaker 3: with what are called friendly fire incidents, and AI was

200
00:12:05,400 --> 00:12:08,360
Speaker 3: meant to help clear up this kind of claus witsy

201
00:12:08,440 --> 00:12:10,680
Speaker 3: and idea of the fog of war. One of the

202
00:12:10,679 --> 00:12:13,920
Speaker 3: ways they've integrated AI into the battlefield picture is through

203
00:12:13,920 --> 00:12:17,640
Speaker 3: the system that emerges from Project Man called Maven Smart System.

204
00:12:18,240 --> 00:12:20,360
Speaker 3: For most of US, it's a really basic idea. It's

205
00:12:20,440 --> 00:12:23,800
Speaker 3: essentially just a digital map. It's Google Maps for war.

206
00:12:24,240 --> 00:12:27,040
Speaker 3: It's really hard for the military to create that because

207
00:12:27,040 --> 00:12:28,760
Speaker 3: they have data in all sorts of places, they haven't

208
00:12:28,840 --> 00:12:31,480
Speaker 3: labeled their data. They have spent the last ten years

209
00:12:31,559 --> 00:12:35,200
Speaker 3: really getting their equipment, their tech, their connectivity up to

210
00:12:35,280 --> 00:12:39,040
Speaker 3: scale where something like this could work, but then integrating

211
00:12:39,080 --> 00:12:42,199
Speaker 3: AI into it is really down to what checks they

212
00:12:42,240 --> 00:12:45,920
Speaker 3: make for it the algorithms themselves. I found cases where

213
00:12:46,320 --> 00:12:49,120
Speaker 3: the quality of the algorithm, the ability of the algorithm

214
00:12:49,160 --> 00:12:51,679
Speaker 3: to detect what it's meant to be identifying, can change

215
00:12:51,679 --> 00:12:54,520
Speaker 3: if the weather changes, if you move an algorithm from

216
00:12:54,559 --> 00:12:57,640
Speaker 3: a hot area to a green area, picking up something

217
00:12:57,640 --> 00:13:01,000
Speaker 3: as simple as a truck or a tank, andinguishing between

218
00:13:01,360 --> 00:13:03,440
Speaker 3: a man or a woman or a child, which is

219
00:13:03,600 --> 00:13:08,440
Speaker 3: really critical to those life and death decisions and the

220
00:13:08,520 --> 00:13:11,920
Speaker 3: rules under which the US military operates, really isn't at

221
00:13:11,920 --> 00:13:13,880
Speaker 3: the stage that you can give that over to algorithms.

222
00:13:14,000 --> 00:13:18,160
Speaker 3: So the risk is if they are not sufficiently tire

223
00:13:18,280 --> 00:13:21,120
Speaker 3: kicking their own algorithms when it comes to computer vision,

224
00:13:21,600 --> 00:13:24,360
Speaker 3: or when it comes to using lms and chatbots to

225
00:13:24,400 --> 00:13:27,760
Speaker 3: speed up their processes and analyze data, you could be

226
00:13:27,800 --> 00:13:30,720
Speaker 3: going in the wrong direction for quite some time before

227
00:13:30,760 --> 00:13:31,319
Speaker 3: you notice it.

228
00:13:32,800 --> 00:13:34,080
Speaker 2: Katran I wonder if you coul tell us a story

229
00:13:34,080 --> 00:13:36,040
Speaker 2: of what happens when this technology doesn't work as it's

230
00:13:36,040 --> 00:13:38,400
Speaker 2: intended to. And you write about a test that took

231
00:13:38,440 --> 00:13:40,280
Speaker 2: place back in June of twenty twenty five, maybe you

232
00:13:40,280 --> 00:13:41,920
Speaker 2: could just walk us through what happened then and what

233
00:13:42,040 --> 00:13:45,240
Speaker 2: lessons we can draw from how that unintended result could

234
00:13:45,240 --> 00:13:46,560
Speaker 2: pretend difficulties down the road.

235
00:13:46,880 --> 00:13:50,959
Speaker 3: The teams behind Maven had made computer vision and then

236
00:13:50,960 --> 00:13:52,920
Speaker 3: this was the next iteration. This is where a program

237
00:13:53,000 --> 00:13:56,840
Speaker 3: called replicator, so it would be to identify a target

238
00:13:57,200 --> 00:13:59,520
Speaker 3: and then the drone could go after the target and

239
00:13:59,559 --> 00:14:06,199
Speaker 3: then identify it and explode. And developing automatic target recognition

240
00:14:06,559 --> 00:14:09,280
Speaker 3: is one thing that relies on creating algorithms that can

241
00:14:09,320 --> 00:14:13,520
Speaker 3: find things. The other is this swarming technology coordinating between drones.

242
00:14:13,760 --> 00:14:16,559
Speaker 3: Of course they were testing this. It's expected to fail,

243
00:14:16,640 --> 00:14:19,000
Speaker 3: that's why they do it in test conditions. And the

244
00:14:19,040 --> 00:14:24,680
Speaker 3: aim was to deliver multiple thousands of drones by a

245
00:14:24,720 --> 00:14:27,360
Speaker 3: certain date. And two months before that date, there was

246
00:14:27,400 --> 00:14:31,560
Speaker 3: a test in California of some drone boats and in

247
00:14:31,560 --> 00:14:34,760
Speaker 3: this one experiment, the drone was towed out to see

248
00:14:34,800 --> 00:14:37,360
Speaker 3: before it was to be switched into autonomy mode. That

249
00:14:37,400 --> 00:14:39,720
Speaker 3: in itself was meant to be a safety measure What

250
00:14:39,840 --> 00:14:43,920
Speaker 3: happened was, inadvertently a command was sent from the dock

251
00:14:44,560 --> 00:14:47,720
Speaker 3: to the drone boat, and when the drone boat was

252
00:14:47,760 --> 00:14:51,600
Speaker 3: activated into autonomy mode without anyone realizing it, it started

253
00:14:51,640 --> 00:14:55,600
Speaker 3: trying to get away from the captain who was towing it.

254
00:14:56,040 --> 00:15:00,560
Speaker 3: The boat started accelerating, decelerating multiple times between zero and

255
00:15:00,600 --> 00:15:04,120
Speaker 3: six knots at pace semi circling action. The rope goes

256
00:15:04,400 --> 00:15:07,720
Speaker 3: taught and the captain is capsized. At that point, he's

257
00:15:07,720 --> 00:15:10,400
Speaker 3: in the water, and then the drone boat turns and

258
00:15:10,440 --> 00:15:13,880
Speaker 3: comes toward him. That's a very dangerous moment because the

259
00:15:14,080 --> 00:15:17,680
Speaker 3: rope could strangle him underwater, he could be submerged. There's

260
00:15:17,680 --> 00:15:20,400
Speaker 3: a runaway boat coming for him. A third boat was

261
00:15:20,440 --> 00:15:24,800
Speaker 3: able to intervene and save him, and he was okay.

262
00:15:24,960 --> 00:15:28,600
Speaker 3: Then what actually happened in the investigation that ensued was

263
00:15:28,600 --> 00:15:32,280
Speaker 3: that someone had mistakenly sent was called a zero command

264
00:15:32,800 --> 00:15:36,640
Speaker 3: just by pressing enter on a command line, deployed the

265
00:15:36,640 --> 00:15:39,360
Speaker 3: boat into autonomy mode, and then all of this ensued,

266
00:15:39,520 --> 00:15:41,960
Speaker 3: so they fixed it. This is a very early stage

267
00:15:41,960 --> 00:15:44,680
Speaker 3: of testing. They're meant to be putting explosives on these boats.

268
00:15:44,720 --> 00:15:47,160
Speaker 3: They're meant for these boats to be all cooperating together

269
00:15:47,520 --> 00:15:50,320
Speaker 3: and able to defend an island such as Taiwan in

270
00:15:50,320 --> 00:15:52,640
Speaker 3: the event of an invasion, And I think it showed

271
00:15:52,640 --> 00:15:55,520
Speaker 3: that the tech is really hard to deliver and just

272
00:15:55,600 --> 00:15:56,600
Speaker 3: simply not ready.

273
00:15:57,000 --> 00:15:59,840
Speaker 2: In a statement for Katrina's book, a Navy spokesperson said

274
00:15:59,880 --> 00:16:03,080
Speaker 2: that safety is always their top priority, that they have

275
00:16:03,160 --> 00:16:06,480
Speaker 2: backup systems to prevent danger. The lessons they learned during

276
00:16:06,480 --> 00:16:10,720
Speaker 2: these events drive improvements in their systems. Mike, these conversations

277
00:16:10,760 --> 00:16:13,880
Speaker 2: about ethics are happening among the Defense Department and these

278
00:16:13,880 --> 00:16:17,880
Speaker 2: private companies. Where are lawmakers? Is Congress showcasing any interest

279
00:16:17,920 --> 00:16:20,000
Speaker 2: in engaging with the subject.

280
00:16:20,200 --> 00:16:23,400
Speaker 1: Showcasing is a good word, because they do like to

281
00:16:23,480 --> 00:16:27,920
Speaker 1: showcase their interest, but actually advancing a proposal that would

282
00:16:28,400 --> 00:16:35,160
Speaker 1: codify some regulation or law even on how AI is

283
00:16:35,200 --> 00:16:37,880
Speaker 1: deployed in warfare, we are a long way from that.

284
00:16:38,040 --> 00:16:41,600
Speaker 1: There's discussion of attaching some sort of amendment to the

285
00:16:41,680 --> 00:16:47,200
Speaker 1: Annual Defense Authorization Measure, which every year usually gets caught

286
00:16:47,280 --> 00:16:50,520
Speaker 1: up in the fight of the moment, and that could

287
00:16:50,600 --> 00:16:54,400
Speaker 1: be where this ultimately ends up, But it's unclear whether

288
00:16:54,960 --> 00:16:59,120
Speaker 1: there would be enough consensus between both parties to really

289
00:16:59,160 --> 00:17:02,160
Speaker 1: come up with language. They all could agree on and

290
00:17:02,200 --> 00:17:06,960
Speaker 1: that the Trump administration would not try to torpedo somehow itself.

291
00:17:07,000 --> 00:17:10,119
Speaker 1: Remember the tech industry, a lot of the big tech

292
00:17:10,119 --> 00:17:15,600
Speaker 1: companies have really moved to align themselves with President Donald

293
00:17:15,640 --> 00:17:19,280
Speaker 1: Trump in a lot of different ways, and it's difficult

294
00:17:19,280 --> 00:17:21,480
Speaker 1: for them and a lot of other business fronts to

295
00:17:21,680 --> 00:17:24,080
Speaker 1: challenge the president, including on this.

296
00:17:24,800 --> 00:17:27,160
Speaker 2: Katrina, I would ask you lastly, just about how much

297
00:17:27,240 --> 00:17:28,879
Speaker 2: the horse is out of the barn here. I think

298
00:17:28,880 --> 00:17:31,520
Speaker 2: there'll be a lot of people listening to this who

299
00:17:32,000 --> 00:17:34,879
Speaker 2: will be impressed by how far this field has advanced,

300
00:17:34,920 --> 00:17:38,320
Speaker 2: that is a integration into warfare, and may wonder if

301
00:17:38,320 --> 00:17:41,280
Speaker 2: there's any recourse or anything that they can do they

302
00:17:41,320 --> 00:17:43,800
Speaker 2: as citizens to slow down this process.

303
00:17:44,000 --> 00:17:47,080
Speaker 3: People talk about a costless war, We've already seen that

304
00:17:47,119 --> 00:17:49,880
Speaker 3: from the US side alone. It's not a costless war.

305
00:17:50,240 --> 00:17:53,760
Speaker 3: AI is meant to give you a riskless war. And

306
00:17:54,560 --> 00:17:58,840
Speaker 3: probably where civilians and citizens come involved, are trying to

307
00:17:58,920 --> 00:18:01,800
Speaker 3: understand the contours of is there such a thing as

308
00:18:01,840 --> 00:18:04,639
Speaker 3: a riskless war? And who is harmed by that? And

309
00:18:04,720 --> 00:18:08,320
Speaker 3: if in any way AI isn't saving civilians or even

310
00:18:08,560 --> 00:18:11,919
Speaker 3: there are misfires that involve AI, then really you have

311
00:18:12,000 --> 00:18:15,440
Speaker 3: to re examine if it makes war more likely that

312
00:18:15,520 --> 00:18:19,080
Speaker 3: in itself is a change for the way that the

313
00:18:19,240 --> 00:18:23,359
Speaker 3: US had maybe been thinking about conducting wars under President

314
00:18:23,359 --> 00:18:26,840
Speaker 3: Trump himself, who of course is the main political leader

315
00:18:26,880 --> 00:18:28,959
Speaker 3: who has said I don't want any more wars.

316
00:18:29,680 --> 00:18:33,120
Speaker 2: Katrina Manson's book Project Mavin, A Marine Colonel, His Team

317
00:18:33,200 --> 00:18:36,200
Speaker 2: and the Dawn of AI Warfare comes out later this month.

318
00:18:39,720 --> 00:18:42,240
Speaker 2: This is the Big Take from Bloomberg News. I'm David Gura.

319
00:18:42,520 --> 00:18:44,800
Speaker 2: To get more from The Big Take and unlimited access

320
00:18:44,840 --> 00:18:47,800
Speaker 2: to all of Bloomberg dot com, subscribe today at Bloomberg

321
00:18:47,840 --> 00:18:51,399
Speaker 2: dot com slash podcast offer. If you like this episode,

322
00:18:51,440 --> 00:18:53,480
Speaker 2: make sure to follow and review The Big Take wherever

323
00:18:53,480 --> 00:18:55,880
Speaker 2: you listen to podcasts. It helps people find the show.

324
00:18:56,320 --> 00:18:58,160
Speaker 2: Thanks for listening. We'll be back tomorrow.