1
00:00:15,250 --> 00:00:28,890
Speaker 1: Pushkin. You're listening to Brave New Planet, a podcast about

2
00:00:28,930 --> 00:00:33,090
Speaker 1: amazing new technologies that could dramatically improve our world. Or

3
00:00:33,490 --> 00:00:36,050
Speaker 1: if we don't make wise choices, could leave us a

4
00:00:36,050 --> 00:00:40,730
Speaker 1: lot worse off. Utopia or dystopia. It's up to us.

5
00:00:47,690 --> 00:00:52,610
Speaker 1: On September twenty sixth, nineteen eighty three, the world almost

6
00:00:52,650 --> 00:00:56,570
Speaker 1: came to an end. Just three weeks earlier, the Soviet

7
00:00:56,690 --> 00:01:00,970
Speaker 1: Union had shot down Korean Airlines Flight Double O seven,

8
00:01:01,330 --> 00:01:05,050
Speaker 1: a passenger plane with two hundred and sixty nine people aboard.

9
00:01:05,290 --> 00:01:08,650
Speaker 1: I'm coming before you tonight about the Korean Airline massacre.

10
00:01:09,210 --> 00:01:12,690
Speaker 1: President Ronald Reagan addressed the nation the attack by the

11
00:01:12,730 --> 00:01:16,530
Speaker 1: Soviet Union against two hundred and sixty nine innocent men, women,

12
00:01:16,530 --> 00:01:21,210
Speaker 1: and children aboard an unarmed Korean passenger plane. This crime

13
00:01:21,250 --> 00:01:24,970
Speaker 1: against humanity must never be forgotten here or throughout the world.

14
00:01:25,610 --> 00:01:29,330
Speaker 1: Cold War tensions escalated, with the two nuclear powers on

15
00:01:29,530 --> 00:01:35,890
Speaker 1: high alert. World War three felt frighteningly possible. Then, on

16
00:01:35,930 --> 00:01:39,810
Speaker 1: September twenty sixth, in a command center outside of Moscow,

17
00:01:40,370 --> 00:01:44,890
Speaker 1: an alarm sounded. The Soviet Union's early warning system reported

18
00:01:44,930 --> 00:01:49,210
Speaker 1: the launch of multiple intercontinental ballistic missiles from bases in

19
00:01:49,250 --> 00:01:54,730
Speaker 1: the United states. Statislav Petrov, a forty four year old

20
00:01:54,770 --> 00:01:58,330
Speaker 1: member of the Soviet Air Defense Forces, was the duty

21
00:01:58,370 --> 00:02:02,930
Speaker 1: officer that night. His role was to alert Moscow that

22
00:02:03,010 --> 00:02:08,170
Speaker 1: an attack was under way, likely triggering Soviet nuclear retaliation

23
00:02:08,650 --> 00:02:14,250
Speaker 1: and all out war. Petrov spoke with BBC News in

24
00:02:14,330 --> 00:02:18,610
Speaker 1: twenty thirteen. The sirens sounded very loudly, and I just

25
00:02:18,650 --> 00:02:21,490
Speaker 1: sat there for a few seconds, staring at the screen

26
00:02:21,530 --> 00:02:26,130
Speaker 1: with the word launch displayed in bold red letters. A

27
00:02:26,210 --> 00:02:30,010
Speaker 1: minute later, the siren went off again. The second missile

28
00:02:30,090 --> 00:02:33,570
Speaker 1: was launched, then the third, and the fourth, and the fifth.

29
00:02:34,890 --> 00:02:38,570
Speaker 1: The computers changed their alerts from launch to missile strike.

30
00:02:39,810 --> 00:02:44,090
Speaker 1: Petrov's instructions were clear, report the attack on the motherland,

31
00:02:44,330 --> 00:02:48,450
Speaker 1: but something didn't make sense. If the US were attacking,

32
00:02:48,810 --> 00:02:53,570
Speaker 1: why only five missiles rather than an entire fleet? And

33
00:02:53,610 --> 00:02:57,130
Speaker 1: then I made my decision. I would not trust the computer.

34
00:02:57,810 --> 00:03:01,770
Speaker 1: I picked up the telephone handset, spoke to my superiors

35
00:03:01,810 --> 00:03:06,370
Speaker 1: and reported that the alarm was false. But I myself

36
00:03:07,010 --> 00:03:11,090
Speaker 1: was not sure. Until the very last moment. I knew

37
00:03:11,210 --> 00:03:14,090
Speaker 1: perfectly well that nobody would be able to correct my

38
00:03:14,210 --> 00:03:20,850
Speaker 1: mistake if I had made one. Petrov, of course, was right.

39
00:03:21,290 --> 00:03:23,730
Speaker 1: The false alarm was later found to be the result

40
00:03:23,770 --> 00:03:28,050
Speaker 1: of a rare and unanticipated coincidence sunlight glinting off high

41
00:03:28,050 --> 00:03:32,130
Speaker 1: altitude clouds over North Dakota at just the right angle

42
00:03:32,130 --> 00:03:36,730
Speaker 1: to fool the Soviet satellites. Statislav Petrov's story comes up

43
00:03:36,770 --> 00:03:40,690
Speaker 1: again and again in discussions of how far we should

44
00:03:40,730 --> 00:03:46,130
Speaker 1: go and turning over important decisions, especially life and death decisions,

45
00:03:46,450 --> 00:03:50,610
Speaker 1: to artificial intelligence. It's not an easy call. Think about

46
00:03:50,610 --> 00:03:55,130
Speaker 1: the split second decisions and avoiding a highway collision. Who

47
00:03:55,170 --> 00:03:58,970
Speaker 1: will ultimately do better a tire driver or a self

48
00:03:59,050 --> 00:04:04,090
Speaker 1: driving car? Nowhere is the question more fraught than on

49
00:04:04,090 --> 00:04:08,930
Speaker 1: the battlefield. As technology evolves, should weapons systems be given

50
00:04:08,970 --> 00:04:12,370
Speaker 1: the power to make life and death decisions? Or do

51
00:04:12,410 --> 00:04:16,010
Speaker 1: we need to ensure there's always a human a Stanislav

52
00:04:16,090 --> 00:04:20,850
Speaker 1: Petrov in the loop. Some people, including winners of the

53
00:04:20,850 --> 00:04:24,690
Speaker 1: Nobel Peace Prize, say that weapons should never be allowed

54
00:04:24,730 --> 00:04:28,170
Speaker 1: to make their own decisions about who or what to attack.

55
00:04:29,090 --> 00:04:33,250
Speaker 1: They're calling for a ban on what they call killer robots.

56
00:04:34,290 --> 00:04:40,770
Speaker 1: Others think that idea is well meaning but naive. Today's

57
00:04:40,810 --> 00:04:46,890
Speaker 1: big question lethal autonomous weapons. Should they ever be allowed?

58
00:04:47,810 --> 00:05:00,370
Speaker 1: If so, when, if not, can we stop them? My

59
00:05:00,450 --> 00:05:02,770
Speaker 1: name is Eric Lander. I'm a scientist who works on

60
00:05:02,810 --> 00:05:05,690
Speaker 1: ways to improve human health. I helped lead the Human

61
00:05:05,770 --> 00:05:08,850
Speaker 1: Genome Project, and today I lead the Broad Institute of

62
00:05:09,450 --> 00:05:14,010
Speaker 1: and Harvard. In the twenty first century, powerful technologies have

63
00:05:14,050 --> 00:05:17,690
Speaker 1: been appearing at a breathtaking pace, related to the Internet,

64
00:05:17,810 --> 00:05:23,570
Speaker 1: artificial intelligence, genetic engineering, and more. They have amazing potential upsides,

65
00:05:24,210 --> 00:05:26,930
Speaker 1: but we can't ignore the risks that come with them.

66
00:05:27,010 --> 00:05:31,770
Speaker 1: The decisions aren't just up to scientists or politicians, whether

67
00:05:31,810 --> 00:05:34,650
Speaker 1: we like it or not, we all of us are

68
00:05:34,690 --> 00:05:38,570
Speaker 1: the stewards of a brave New Planet. This generation's choices

69
00:05:38,610 --> 00:05:48,170
Speaker 1: will shape the future as never before. Coming up on

70
00:05:48,210 --> 00:05:53,130
Speaker 1: today's episode of Brave New Planet fully autonomous lethal weapons

71
00:05:53,690 --> 00:05:59,650
Speaker 1: or killer robots, we hear from a fighter pilot about

72
00:05:59,690 --> 00:06:02,290
Speaker 1: why it might make sense to have machines in charge

73
00:06:02,530 --> 00:06:06,810
Speaker 1: of some major battlefield decisions. I know people who have

74
00:06:07,410 --> 00:06:13,210
Speaker 1: killed civilians, and in all cases where people made mistakes,

75
00:06:13,610 --> 00:06:16,570
Speaker 1: it was just too much information. Things were happening too fast.

76
00:06:16,890 --> 00:06:19,570
Speaker 1: I speak with one of the world's leading robo ethesis.

77
00:06:19,810 --> 00:06:23,210
Speaker 1: Robots will make mistakes too, but hopefully, if done correctly,

78
00:06:23,290 --> 00:06:25,970
Speaker 1: they will make far far less mistakes than human beings.

79
00:06:26,730 --> 00:06:30,890
Speaker 1: We'll hear about some of the possible consequences of autonomous weapons.

80
00:06:31,370 --> 00:06:36,370
Speaker 1: Algorithms interacting at machine speed faster than humans couldn't respond

81
00:06:36,930 --> 00:06:40,490
Speaker 1: might result in accidents, and that's something like a flash war.

82
00:06:41,170 --> 00:06:43,970
Speaker 1: I'll speak with a leader from Human Rights Watch. The

83
00:06:44,130 --> 00:06:47,770
Speaker 1: campaign to stop Killer Robots is seeking new international lure

84
00:06:47,930 --> 00:06:51,490
Speaker 1: in the form of a new treaty. And we'll talk

85
00:06:51,530 --> 00:06:55,170
Speaker 1: with former Secretary of Defense Ash Carter. Because I'm the

86
00:06:55,170 --> 00:06:57,050
Speaker 1: guy who asked to go out the next morning after

87
00:06:57,090 --> 00:07:00,450
Speaker 1: some women and children have been accidentally killed. As suppose

88
00:07:00,490 --> 00:07:03,330
Speaker 1: I go out there, Eric and I say, oh, I

89
00:07:03,370 --> 00:07:05,490
Speaker 1: don't know how it happened. The machine did it. I

90
00:07:05,530 --> 00:07:09,970
Speaker 1: would be crucified. I should be crucified. So stay with us.

91
00:07:13,770 --> 00:07:21,690
Speaker 1: Chapter one, Stanley the self driving Car. Not long after

92
00:07:21,730 --> 00:07:25,530
Speaker 1: the first general purpose computers were invented in the nineteen forties,

93
00:07:26,010 --> 00:07:31,090
Speaker 1: some people began to dream about fully autonomous robots, machines

94
00:07:31,130 --> 00:07:35,410
Speaker 1: that used their electronic brains to navigate the world, make decisions,

95
00:07:35,450 --> 00:07:39,690
Speaker 1: and take actions. Not surprisingly, some of those dreamers were

96
00:07:39,690 --> 00:07:44,370
Speaker 1: in the US Department of Defense, specifically the Defense Advanced

97
00:07:44,410 --> 00:07:48,850
Speaker 1: Research Projects Agency or DARPA, the visionary unit behind the

98
00:07:48,890 --> 00:07:52,570
Speaker 1: creation of the Internet. They saw a lot of potential

99
00:07:52,650 --> 00:07:56,250
Speaker 1: for automating battlefields, but they knew it might take decades.

100
00:07:57,050 --> 00:08:01,050
Speaker 1: In the nineteen sixties, DARPA funded the Stanford Research inst

101
00:08:01,570 --> 00:08:05,850
Speaker 1: to build Shaky the Robots, a machine that used cameras

102
00:08:06,210 --> 00:08:09,410
Speaker 1: to move about a laboratory. In the nineteen eight ease,

103
00:08:09,530 --> 00:08:13,690
Speaker 1: it supported universities to create vehicles that could follow lanes

104
00:08:13,770 --> 00:08:18,250
Speaker 1: on a road. By the early two thousands, DARPA decided

105
00:08:18,290 --> 00:08:22,650
Speaker 1: that computers had reached the point that fully autonomous vehicles

106
00:08:23,090 --> 00:08:27,050
Speaker 1: able to navigate the real world might finally be feasible.

107
00:08:27,690 --> 00:08:32,090
Speaker 1: To find out, DARPA decided to launch a race. I

108
00:08:32,210 --> 00:08:35,650
Speaker 1: talked to someone who knew a lot about it. My

109
00:08:35,730 --> 00:08:38,010
Speaker 1: name is Sebastian Thrun. I mean the smartest person on

110
00:08:38,010 --> 00:08:41,850
Speaker 1: the planet and the best looking. That's kidding. Sebastian Thrun

111
00:08:42,010 --> 00:08:46,570
Speaker 1: gained recognition when his autonomous car, a modified Volkswagon with

112
00:08:46,610 --> 00:08:49,530
Speaker 1: a computer in the trunk and sensors on the roof,

113
00:08:50,170 --> 00:08:53,170
Speaker 1: was the first to win the DARPA Grand Challenge. A

114
00:08:53,290 --> 00:08:58,090
Speaker 1: dun Challenge was his momentous government sponsors robot raises epic RaSE,

115
00:08:58,610 --> 00:09:00,890
Speaker 1: can you bid a robot that can navigate one hundred

116
00:09:00,930 --> 00:09:05,210
Speaker 1: and thirty punishing miles through the Mohabi Desert and the

117
00:09:05,290 --> 00:09:08,570
Speaker 1: best robot like seven miles and then literally end up

118
00:09:08,810 --> 00:09:12,250
Speaker 1: in many many mini researchers had concluded can't be done.

119
00:09:12,250 --> 00:09:14,930
Speaker 1: In fact, many of my colleagues told me I'm going

120
00:09:14,970 --> 00:09:17,770
Speaker 1: to waste my time and my name if I engaged

121
00:09:17,770 --> 00:09:19,810
Speaker 1: in this kind of super hard race. And that made

122
00:09:19,850 --> 00:09:22,010
Speaker 1: you more interested in doing it, of course, and so

123
00:09:22,130 --> 00:09:25,730
Speaker 1: you built Stanley. Yeah, So my students built Stanley and

124
00:09:26,050 --> 00:09:29,730
Speaker 1: started as a class And Stanford students are great. If

125
00:09:29,730 --> 00:09:31,450
Speaker 1: you tell them go to the moon in two months,

126
00:09:31,490 --> 00:09:33,890
Speaker 1: they're going to go to the moon. So then two

127
00:09:33,930 --> 00:09:38,770
Speaker 1: thousand or five, the actual government sponsored race, how did

128
00:09:38,850 --> 00:09:43,410
Speaker 1: Stanley do. We came in first, so we are focused

129
00:09:43,730 --> 00:09:46,930
Speaker 1: insanely strongly on software and specifically on machine learning, and

130
00:09:46,970 --> 00:09:49,610
Speaker 1: that differentiated as from pretty much every other team that

131
00:09:49,730 --> 00:09:51,570
Speaker 1: focused on hardware. But the way I look at this

132
00:09:51,610 --> 00:09:54,370
Speaker 1: is there were five teams that finished this ruling race

133
00:09:54,450 --> 00:09:57,250
Speaker 1: within one year, and it's the community of the people

134
00:09:57,410 --> 00:10:01,170
Speaker 1: that build all these machines that really won. So nobody

135
00:10:01,170 --> 00:10:04,810
Speaker 1: made it a mile in the first race, and five

136
00:10:04,930 --> 00:10:07,450
Speaker 1: different teams made it more than one hundred and thirty

137
00:10:07,450 --> 00:10:10,010
Speaker 1: miles through the desert, just a year later, Yeah, that's

138
00:10:10,090 --> 00:10:13,650
Speaker 1: kind of amazing to me. That just showed how fast

139
00:10:13,730 --> 00:10:18,010
Speaker 1: this technology can possibly evolve. And what's happened since then?

140
00:10:18,490 --> 00:10:22,250
Speaker 1: I worked at Google for a vile and eventually this guy,

141
00:10:22,730 --> 00:10:25,130
Speaker 1: Larry Page came to me and says, hey, Sebastian, I

142
00:10:25,170 --> 00:10:27,450
Speaker 1: thought about this long and heart. We should build a

143
00:10:27,450 --> 00:10:29,210
Speaker 1: self driving car they can drive on all streets in

144
00:10:29,210 --> 00:10:32,370
Speaker 1: the world. And with my entire authority, I said that

145
00:10:32,570 --> 00:10:35,930
Speaker 1: cannot be done. We just had driven a desert raised

146
00:10:36,250 --> 00:10:39,050
Speaker 1: there was never pedestrians and bicycles and all the other

147
00:10:39,090 --> 00:10:41,170
Speaker 1: people that we could kill in the environment. And for me,

148
00:10:41,450 --> 00:10:44,290
Speaker 1: just the sheer imagination we would drive a self driving

149
00:10:44,290 --> 00:10:47,530
Speaker 1: car to San Francisco sounded always like a crime. So

150
00:10:47,570 --> 00:10:50,890
Speaker 1: you told Larry Page, one of the two co founders

151
00:10:51,010 --> 00:10:54,050
Speaker 1: of Google, that the idea of building a self driving

152
00:10:54,130 --> 00:10:58,090
Speaker 1: car that could navigate anywhere was just not Yeah, feelous.

153
00:10:58,170 --> 00:11:00,850
Speaker 1: Later came back and said, he Sebastian, look, I trust you,

154
00:11:00,850 --> 00:11:03,170
Speaker 1: you're the expert, but I want to explain Eric Schmidt,

155
00:11:03,210 --> 00:11:05,330
Speaker 1: then the Google CEO, and it's my co founder, surgery brain,

156
00:11:05,970 --> 00:11:08,250
Speaker 1: why it can be done. Can you give me the

157
00:11:09,090 --> 00:11:13,610
Speaker 1: technical reason? So I went home in agony, thinking about

158
00:11:14,690 --> 00:11:17,050
Speaker 1: what is the technical reason why it can be done?

159
00:11:17,730 --> 00:11:19,970
Speaker 1: And I got back the next day and I said, so,

160
00:11:20,050 --> 00:11:23,290
Speaker 1: what is it? And I said, I can't think of

161
00:11:23,370 --> 00:11:27,330
Speaker 1: any and Lomi Hoold. Eighteen months later, roughly ten engineers

162
00:11:27,810 --> 00:11:32,170
Speaker 1: we drove pretty much every street in California. Today, autonomous

163
00:11:32,250 --> 00:11:36,290
Speaker 1: technology is changing the transportation industry. About ten percent of

164
00:11:36,330 --> 00:11:39,410
Speaker 1: cars sold in the US are already capable of at

165
00:11:39,490 --> 00:11:44,010
Speaker 1: least partly guiding themselves down the highway. In twenty eighteen,

166
00:11:44,450 --> 00:11:48,890
Speaker 1: Google's self driving car company Weymo launched a self driving

167
00:11:48,970 --> 00:11:53,690
Speaker 1: taxi service in Phoenix, Arizona, initially with human backup drivers

168
00:11:53,730 --> 00:11:58,330
Speaker 1: behind every wheel, but now sometimes even without. I asked

169
00:11:58,330 --> 00:12:03,210
Speaker 1: Sebastian why he thinks this matters. We lose more than

170
00:12:03,250 --> 00:12:07,610
Speaker 1: a million people in traffic accidents every year, almost exclusively

171
00:12:08,210 --> 00:12:11,130
Speaker 1: to us not pay attention. When it was eighteen, my

172
00:12:11,530 --> 00:12:15,410
Speaker 1: best friend died in a traffic accident and it was

173
00:12:15,450 --> 00:12:19,090
Speaker 1: a split second poor decision from his friend who was

174
00:12:19,170 --> 00:12:22,450
Speaker 1: driving in who also died. To me, this is just unacceptable.

175
00:12:23,530 --> 00:12:28,850
Speaker 1: Beyond safety, Sebastian sees many other advantages for autonomy. During

176
00:12:28,850 --> 00:12:32,610
Speaker 1: a commute, you can do something else that means you're

177
00:12:32,730 --> 00:12:35,610
Speaker 1: probably willing to commute further distances. You could sleep, or

178
00:12:35,650 --> 00:12:38,730
Speaker 1: watch the movie, or do email. And then eventually people

179
00:12:38,770 --> 00:12:42,250
Speaker 1: can use cars that today can't operate them blind people,

180
00:12:42,810 --> 00:12:46,890
Speaker 1: old people, children, babies. I mean, there's an entire spectrum

181
00:12:46,930 --> 00:12:49,050
Speaker 1: of people that are kindly excluded. They would now be

182
00:12:49,090 --> 00:12:57,090
Speaker 1: able to be mobile. Chapter two the Tomahawk so darpest

183
00:12:57,090 --> 00:13:00,130
Speaker 1: efforts over the decades helped give rise to the modern

184
00:13:00,250 --> 00:13:04,970
Speaker 1: self driving car industry, which promises to make transportations safer,

185
00:13:05,330 --> 00:13:10,330
Speaker 1: more efficient, and more accessible. But the agencies primary motivation

186
00:13:10,850 --> 00:13:14,370
Speaker 1: was to bring autonomy to a different challenge, the battlefield.

187
00:13:14,770 --> 00:13:17,530
Speaker 1: I traveled to Chapel Hill, North Carolina, to meet with

188
00:13:17,570 --> 00:13:19,970
Speaker 1: someone who spends a lot of time thinking about the

189
00:13:20,010 --> 00:13:24,490
Speaker 1: consequences of autonomous technology. We both serve on a civilian

190
00:13:24,530 --> 00:13:28,810
Speaker 1: advisory board for the Defense Department. My name is Missy Cummings.

191
00:13:29,050 --> 00:13:33,210
Speaker 1: I'm a professor of electrical and computer engineering at Duke University,

192
00:13:33,770 --> 00:13:36,250
Speaker 1: and I think one of the things that people find

193
00:13:36,410 --> 00:13:39,090
Speaker 1: most interesting about me is that I was one of

194
00:13:39,090 --> 00:13:43,010
Speaker 1: the US military's first female fighter pilots in the Navy.

195
00:13:43,130 --> 00:13:46,250
Speaker 1: Did you always want to be a fighter pilot? So

196
00:13:46,290 --> 00:13:48,890
Speaker 1: when I was growing up, I did not know that

197
00:13:48,970 --> 00:13:52,570
Speaker 1: women could be pilots, and indeed, when I was growing up,

198
00:13:52,570 --> 00:13:55,970
Speaker 1: women could it be pilots. And it wasn't until the

199
00:13:56,850 --> 00:14:01,530
Speaker 1: late seventies that women actually became pilots in the military.

200
00:14:02,450 --> 00:14:06,890
Speaker 1: So I went to college. In nineteen eighty four, I

201
00:14:06,970 --> 00:14:09,290
Speaker 1: was at the Naval Academy, and then of course, in

202
00:14:09,370 --> 00:14:11,490
Speaker 1: nineteen eighty six Top Gun came out, and then I

203
00:14:12,170 --> 00:14:14,010
Speaker 1: know who doesn't want to be a pilot After you

204
00:14:14,050 --> 00:14:17,730
Speaker 1: see the movie Top Gun. Missy is tremendously proud of

205
00:14:17,770 --> 00:14:20,290
Speaker 1: the eleven years she spent in the Navy, but she

206
00:14:20,370 --> 00:14:23,330
Speaker 1: also acknowledges the challenges of being part of that first

207
00:14:23,370 --> 00:14:27,370
Speaker 1: generation of woman fighter pilots. It's no secret that the

208
00:14:27,410 --> 00:14:31,610
Speaker 1: reason I left the military was because of the hostile

209
00:14:31,650 --> 00:14:35,050
Speaker 1: attitude towards women. None of the women in that first

210
00:14:35,050 --> 00:14:37,130
Speaker 1: group stayed in to make it a career. The guys

211
00:14:37,130 --> 00:14:41,130
Speaker 1: were very angry that we were there, and I decided

212
00:14:41,130 --> 00:14:44,370
Speaker 1: to leave when they started sabotaging my flight gear. I

213
00:14:44,490 --> 00:14:47,530
Speaker 1: just thought, this is too much. If something really bad happened,

214
00:14:47,610 --> 00:14:51,370
Speaker 1: you know, I would die. When Missy Cummings left the Navy,

215
00:14:51,610 --> 00:14:55,770
Speaker 1: she decided to pursue a PhD in Human Machine interaction.

216
00:14:56,010 --> 00:14:59,970
Speaker 1: In my last three years flying IF eighteens, there were

217
00:15:00,010 --> 00:15:02,890
Speaker 1: about thirty six people I knew that died, about one

218
00:15:02,930 --> 00:15:06,090
Speaker 1: person a month. They were all training accidents. It just

219
00:15:06,130 --> 00:15:09,850
Speaker 1: really struck me how many people were dying because the

220
00:15:10,130 --> 00:15:13,250
Speaker 1: design of the airplane just did not go with the

221
00:15:13,370 --> 00:15:16,850
Speaker 1: human tendencies. And so I decided to go back to

222
00:15:16,890 --> 00:15:19,570
Speaker 1: school to find out what can be done about that.

223
00:15:20,170 --> 00:15:23,330
Speaker 1: So I went to finish my PhD at the University

224
00:15:23,370 --> 00:15:26,650
Speaker 1: of Virginia, and then I spent the next ten years

225
00:15:26,650 --> 00:15:30,490
Speaker 1: at MT learning my craft. The person I am today

226
00:15:30,730 --> 00:15:34,370
Speaker 1: is half because of the Navy and half because of MIGHT. Today,

227
00:15:34,570 --> 00:15:38,050
Speaker 1: Missy is a Duke University where she runs the Humans

228
00:15:38,050 --> 00:15:42,250
Speaker 1: an Autonomy Lab, or for short HOW. It's a nod

229
00:15:42,290 --> 00:15:46,290
Speaker 1: to the sentient computer that goes rogue in Stanley Kubrick's

230
00:15:46,330 --> 00:15:50,410
Speaker 1: film two thousand and one, A Space Odyssey. This mission

231
00:15:50,530 --> 00:15:53,330
Speaker 1: is too important for me to allow you to jeopardize it.

232
00:15:55,130 --> 00:15:58,450
Speaker 1: I don't know what you're talking about. How. I know

233
00:15:58,530 --> 00:16:01,650
Speaker 1: that you and Frank were planning to disconnect me, and

234
00:16:01,730 --> 00:16:05,570
Speaker 1: I'm afraid that's something I cannot allow to happen. And

235
00:16:05,650 --> 00:16:10,450
Speaker 1: so I intentionally named my lab how so that we

236
00:16:10,450 --> 00:16:13,330
Speaker 1: were there to stop that from happening. Right, I had

237
00:16:13,370 --> 00:16:17,290
Speaker 1: seen many friends die, not because the robot became sentient,

238
00:16:17,450 --> 00:16:21,530
Speaker 1: in fact, because the designers of the automation really had

239
00:16:21,610 --> 00:16:24,610
Speaker 1: no clue how people would or would not use this technology.

240
00:16:25,410 --> 00:16:30,570
Speaker 1: It is my life's mission statement to develop human collaborative

241
00:16:30,610 --> 00:16:34,090
Speaker 1: computer systems that work with each other to achieve something

242
00:16:34,130 --> 00:16:37,370
Speaker 1: greater than either would alone. The Humans and Autonomy Lab

243
00:16:37,450 --> 00:16:42,450
Speaker 1: works on the interactions between humans and machines across many fields,

244
00:16:43,050 --> 00:16:46,930
Speaker 1: but given her background, Missy's thought a lot about how

245
00:16:46,970 --> 00:16:51,850
Speaker 1: technology has changed the relationship between humans and their weapons.

246
00:16:52,250 --> 00:16:56,490
Speaker 1: There's a long history of us distancing ourselves from our actions.

247
00:16:57,330 --> 00:17:00,090
Speaker 1: We want to shoot somebody, we wanted to shoot them

248
00:17:00,090 --> 00:17:03,090
Speaker 1: with bows and arrows. We wanted to drop bombs from

249
00:17:03,130 --> 00:17:05,970
Speaker 1: five miles over a target. We want cruise muscles that

250
00:17:06,010 --> 00:17:09,050
Speaker 1: can kill you from another country. Right, it is human

251
00:17:09,290 --> 00:17:14,250
Speaker 1: nature to back that distance up, Missy season inherent tension.

252
00:17:15,050 --> 00:17:19,810
Speaker 1: On one hand, technology distances ourselves from killing. On the

253
00:17:19,810 --> 00:17:23,370
Speaker 1: other hand, technology is letting us design weapons that are

254
00:17:23,410 --> 00:17:28,410
Speaker 1: more accurate and less indiscriminate in their killing. Missy rotor

255
00:17:28,530 --> 00:17:32,530
Speaker 1: PhD thesis about the Tomahawk missile, an early precursor of

256
00:17:32,610 --> 00:17:37,130
Speaker 1: the autonomous weapons systems being developed today. The Tomahawk missile

257
00:17:37,210 --> 00:17:41,250
Speaker 1: has these stored maps in its brain, and as its

258
00:17:41,250 --> 00:17:44,250
Speaker 1: skimming along the nap of the Earth, it compares the

259
00:17:44,290 --> 00:17:46,930
Speaker 1: pictures that it's taking with its pictures and its database

260
00:17:47,330 --> 00:17:50,530
Speaker 1: to decide how to get to its target. This Tomahawk

261
00:17:50,610 --> 00:17:52,450
Speaker 1: was kind of a set it and forget it kind

262
00:17:52,450 --> 00:17:55,290
Speaker 1: of thing. Once you launched it, it would follow its

263
00:17:55,290 --> 00:17:57,850
Speaker 1: map to the right place and there was nobody looking

264
00:17:57,890 --> 00:18:02,370
Speaker 1: over its shoulders. Well, so the Tomahawk missile that we

265
00:18:02,410 --> 00:18:04,610
Speaker 1: saw in the Gulf War, that was a fire and

266
00:18:04,650 --> 00:18:08,730
Speaker 1: forget missile that a target would be programmed into the

267
00:18:09,290 --> 00:18:11,210
Speaker 1: sole and then it would be fired and that's where

268
00:18:11,210 --> 00:18:15,810
Speaker 1: it would go. Later, around two thousand, two thousand and three,

269
00:18:16,690 --> 00:18:20,010
Speaker 1: then GPS technology was coming online, and that's when we

270
00:18:20,050 --> 00:18:22,610
Speaker 1: got the tactical Tomahawk, which had the ability to be

271
00:18:22,690 --> 00:18:27,090
Speaker 1: redirected in flight. That success with GPS and the Tomahawk

272
00:18:27,170 --> 00:18:31,970
Speaker 1: opened the military's eyes to the ability to use them

273
00:18:32,010 --> 00:18:36,330
Speaker 1: in drones. Today's precision guided weapons are far more accurate

274
00:18:36,370 --> 00:18:39,970
Speaker 1: than the widespread aerial bombing that occurred on all sides

275
00:18:39,970 --> 00:18:44,450
Speaker 1: in World War Two, where some cities were almost entirely leveled,

276
00:18:44,810 --> 00:18:49,210
Speaker 1: resulting in huge numbers of civilian casualties. In the Gulf War,

277
00:18:49,650 --> 00:18:56,170
Speaker 1: Tomahawk missile attacks came to be called surgical strikes. I

278
00:18:56,250 --> 00:19:02,330
Speaker 1: know people who have killed civilians, and I know people

279
00:19:02,370 --> 00:19:05,330
Speaker 1: who have killed friendlies. They have dropped bombs on our

280
00:19:05,330 --> 00:19:10,530
Speaker 1: own forces and killed our own people. And in all

281
00:19:10,610 --> 00:19:15,170
Speaker 1: cases where people made mistakes, it was just too much information.

282
00:19:15,410 --> 00:19:18,730
Speaker 1: Things were happening too fast. You've seen some pictures that

283
00:19:18,770 --> 00:19:21,130
Speaker 1: you've got in a brief hours ago, and you're supposed

284
00:19:21,170 --> 00:19:24,330
Speaker 1: to know that what you're seeing now through this grainy

285
00:19:24,410 --> 00:19:27,210
Speaker 1: image thirty five thousand feet over a target is the

286
00:19:27,250 --> 00:19:30,490
Speaker 1: same image that you're being asked to bob. The Tomahawk

287
00:19:30,970 --> 00:19:35,610
Speaker 1: never missed its target. It never made a mistake unless

288
00:19:35,610 --> 00:19:39,250
Speaker 1: it was programmed as a mistake. And that's old autonomy,

289
00:19:39,370 --> 00:19:46,530
Speaker 1: and it's only gotten better over time. Chapter three, Kicking

290
00:19:46,610 --> 00:19:52,730
Speaker 1: down Doors. The Tomahawk was just a baby step toward automation.

291
00:19:53,250 --> 00:19:56,130
Speaker 1: With the ability to read maps, it could correct its course,

292
00:19:56,650 --> 00:20:00,570
Speaker 1: but it couldn't make sophisticated decisions. But what happens when

293
00:20:00,610 --> 00:20:04,410
Speaker 1: you start adding modern artificial intelligence? So where do you

294
00:20:04,410 --> 00:20:08,010
Speaker 1: see autonomous weapons going? If you could kind of map

295
00:20:08,050 --> 00:20:09,930
Speaker 1: out where are we today and where do you think

296
00:20:10,010 --> 00:20:13,570
Speaker 1: we'll be ten twenty years from now. So, in terms

297
00:20:13,610 --> 00:20:18,130
Speaker 1: of autonomy and weapons, by today's standards, the Tomahawk missile

298
00:20:18,250 --> 00:20:20,090
Speaker 1: is still one of the best ones that we have,

299
00:20:20,330 --> 00:20:23,490
Speaker 1: and it's also still one of the most advanced. Certainly,

300
00:20:23,530 --> 00:20:26,170
Speaker 1: there are research arms of the military who are trying

301
00:20:26,290 --> 00:20:31,370
Speaker 1: very hard to come up with new forms of autonomy.

302
00:20:31,930 --> 00:20:35,010
Speaker 1: There was the Predicts that came out of Lincoln Lab,

303
00:20:35,050 --> 00:20:39,690
Speaker 1: and this was basically a swarm of really tiny UAVs

304
00:20:39,690 --> 00:20:45,450
Speaker 1: that could coordinate together. A ua V, an unmanned aerial

305
00:20:45,570 --> 00:20:50,490
Speaker 1: vehicle is military speak for a drone. The Predicts the

306
00:20:50,730 --> 00:20:54,090
Speaker 1: drones that Missy was referring to. They were commissioned by

307
00:20:54,090 --> 00:20:57,210
Speaker 1: the Strategic Capabilities Office of the US Department of Defense.

308
00:20:57,810 --> 00:21:00,810
Speaker 1: These tiny flying robots are able to communicate with each

309
00:21:00,850 --> 00:21:04,250
Speaker 1: other and make split second decisions about how to move

310
00:21:04,250 --> 00:21:07,130
Speaker 1: as a group. Many researchers, have you been using bio

311
00:21:07,250 --> 00:21:12,210
Speaker 1: inspired methods? Be right? So bees have local and global intelligence.

312
00:21:12,530 --> 00:21:15,370
Speaker 1: Like a group of bees, these drones are called a

313
00:21:15,450 --> 00:21:20,650
Speaker 1: swarm collective intelligence on a shared mission. A human can

314
00:21:20,690 --> 00:21:24,250
Speaker 1: make the big picture decision and the swarm of microdrones

315
00:21:24,290 --> 00:21:27,450
Speaker 1: can then collectively decide on the most efficient way to

316
00:21:27,530 --> 00:21:30,810
Speaker 1: carry out the order in the moment. I wanted to

317
00:21:30,850 --> 00:21:35,370
Speaker 1: know why exactly this technology is necessary, so I went

318
00:21:35,410 --> 00:21:38,450
Speaker 1: to speak to someone who I was pretty sure would know.

319
00:21:38,890 --> 00:21:43,010
Speaker 1: I'm Ash Carter. Most people will probably have heard my

320
00:21:43,090 --> 00:21:48,250
Speaker 1: name as the Secretary of Defense who proceeded Gimatus. You

321
00:21:48,290 --> 00:21:50,610
Speaker 1: will know me in part from the fact that we

322
00:21:50,730 --> 00:21:53,210
Speaker 1: knew one another way back in Oxford when we were

323
00:21:53,250 --> 00:21:55,890
Speaker 1: both young scientists, and I guess agents start there. I'm

324
00:21:55,930 --> 00:21:58,530
Speaker 1: a physicist. When you were doing your PhD in physics,

325
00:21:58,570 --> 00:22:01,450
Speaker 1: I was doing my PhD in mathematics at Oxford. What

326
00:22:01,610 --> 00:22:06,170
Speaker 1: was your thesis on? It was on quantum chrominynamics. That

327
00:22:06,330 --> 00:22:09,810
Speaker 1: was the theory of quarks and gluons. And how in

328
00:22:09,850 --> 00:22:14,330
Speaker 1: the world is somebody who's an expert in quantum chromodynamics

329
00:22:14,450 --> 00:22:18,730
Speaker 1: become the Secretary of Defense. It's an interesting story. The

330
00:22:18,970 --> 00:22:24,610
Speaker 1: people who were the seniors in my field of physics,

331
00:22:25,010 --> 00:22:28,250
Speaker 1: the mentors, so to speak, were all members of the

332
00:22:28,330 --> 00:22:33,890
Speaker 1: Manhattan Project generation. They had built the bomb during World

333
00:22:33,930 --> 00:22:38,330
Speaker 1: War Two, and they were proud of what they'd done

334
00:22:38,890 --> 00:22:42,530
Speaker 1: because they believed that it had ended the war with

335
00:22:42,650 --> 00:22:45,970
Speaker 1: fewer casualties than otherwise there would have been in a

336
00:22:46,010 --> 00:22:49,130
Speaker 1: full scale of invasion of Japan, and also that it

337
00:22:49,170 --> 00:22:51,170
Speaker 1: had kept the peace through the Cold War, so they

338
00:22:51,170 --> 00:22:53,610
Speaker 1: were proud of it. However, they knew there was a

339
00:22:53,690 --> 00:22:57,570
Speaker 1: dark side, and they conveyed to me that it was

340
00:22:58,170 --> 00:23:03,810
Speaker 1: my responsibility as a scientist to be involved in these matters.

341
00:23:04,450 --> 00:23:08,690
Speaker 1: And the technology doesn't determine what the balance of good

342
00:23:08,690 --> 00:23:12,330
Speaker 1: and bad is. We human beings do. That was the lesson,

343
00:23:12,730 --> 00:23:16,130
Speaker 1: and so that's what got me started, and then my

344
00:23:16,250 --> 00:23:19,250
Speaker 1: very first Pentagon job, which was in nineteen eighty one,

345
00:23:19,570 --> 00:23:22,330
Speaker 1: right through until the last time I walked out of

346
00:23:22,370 --> 00:23:27,010
Speaker 1: the Pentagon of Sectary Defense, which was January of twenty seventeen. Now,

347
00:23:27,330 --> 00:23:32,330
Speaker 1: when you were secretary, there was a Strategic Capabilities Office

348
00:23:32,650 --> 00:23:38,330
Speaker 1: that it's been publicly reported, was experimenting with using drones

349
00:23:38,890 --> 00:23:43,410
Speaker 1: to make swarms of drones that could do things, communicate

350
00:23:43,450 --> 00:23:46,890
Speaker 1: with each other, make formations. Why would you want such things?

351
00:23:46,890 --> 00:23:49,210
Speaker 1: So it's a good question. Here's what you do with

352
00:23:49,250 --> 00:23:52,130
Speaker 1: the drone like that, You put a jammer on it,

353
00:23:52,210 --> 00:23:56,210
Speaker 1: a little radio beacon, and you fly it right into

354
00:23:56,250 --> 00:24:01,450
Speaker 1: the eye of a enemy radar. So all that radar

355
00:24:01,570 --> 00:24:06,130
Speaker 1: c's is the energy emitted by that little drone, and

356
00:24:06,170 --> 00:24:10,730
Speaker 1: it's essentially dazzled or blinded. If there's one big drone,

357
00:24:11,330 --> 00:24:14,770
Speaker 1: that radar is precious enough that the defenders going to

358
00:24:14,850 --> 00:24:18,250
Speaker 1: shoot that drone down. But if you have so many

359
00:24:18,290 --> 00:24:21,690
Speaker 1: out there, the enemy can't afford to shoot them all down.

360
00:24:22,450 --> 00:24:26,530
Speaker 1: And since they are flying right up to the radar,

361
00:24:26,850 --> 00:24:29,770
Speaker 1: they don't have to be very powerful. So there's an

362
00:24:29,770 --> 00:24:34,810
Speaker 1: application where lots of little drones can have the effect

363
00:24:34,970 --> 00:24:39,250
Speaker 1: of nullifying enemy radar. That's a pretty big deal for

364
00:24:39,290 --> 00:24:42,810
Speaker 1: a few little, little microdrones. To learn more, I went

365
00:24:42,890 --> 00:24:45,610
Speaker 1: to speak with Paul Shari. Paul's the director of the

366
00:24:45,610 --> 00:24:49,250
Speaker 1: Technology and National Security Program at the Center for a

367
00:24:49,290 --> 00:24:52,850
Speaker 1: New American Security. Before that, he worked for Ash Carter

368
00:24:52,930 --> 00:24:57,290
Speaker 1: at the Pentagon studying autonomous weapons and a recently authored

369
00:24:57,290 --> 00:25:01,570
Speaker 1: a book called Army of None, Autonomous Weapons and the

370
00:25:01,650 --> 00:25:05,810
Speaker 1: Future of War. Paul's interest in autonomous weapons began when

371
00:25:05,810 --> 00:25:09,610
Speaker 1: he served in the Army. I enlisted in the Army

372
00:25:09,850 --> 00:25:12,850
Speaker 1: to become an Army ranger. That was June of two

373
00:25:12,850 --> 00:25:15,850
Speaker 1: thousand and one, did a number of tours overseas and

374
00:25:15,970 --> 00:25:19,410
Speaker 1: the wars and a rock Afghanistan. So I'll say one

375
00:25:19,450 --> 00:25:22,450
Speaker 1: moment that stuck out for me where I really sort

376
00:25:22,450 --> 00:25:24,490
Speaker 1: of the light bulb went on about the power of

377
00:25:24,650 --> 00:25:28,370
Speaker 1: robotics in warfare. I was in a rock in two

378
00:25:28,410 --> 00:25:31,690
Speaker 1: thousand and seven. We were on a patrol, driving along

379
00:25:31,730 --> 00:25:35,410
Speaker 1: in a Striker armored vehicle. Came across an ID improvised

380
00:25:35,450 --> 00:25:38,730
Speaker 1: explosive device makeshift road type bomb, and so we called

381
00:25:38,810 --> 00:25:41,810
Speaker 1: up bomb disposal folks. So they show up and I'm

382
00:25:41,850 --> 00:25:44,730
Speaker 1: expecting the bomb tech to come out in that big

383
00:25:44,770 --> 00:25:46,450
Speaker 1: bomb suit that you might have seen in the movie

384
00:25:46,450 --> 00:25:50,130
Speaker 1: The hurt Locker, for example, and instead out rolls out

385
00:25:50,290 --> 00:25:53,730
Speaker 1: a little robot and I kind of went, oh, that

386
00:25:53,770 --> 00:25:56,930
Speaker 1: makes a lot of sense. Have the robot diffused the bomb. Well,

387
00:25:56,930 --> 00:25:58,370
Speaker 1: it turns out there's a lot of things in war

388
00:25:58,410 --> 00:26:01,210
Speaker 1: that are super dangerous where it makes sense to have

389
00:26:01,330 --> 00:26:04,570
Speaker 1: robots out on the front lines, getting people better stand

390
00:26:04,610 --> 00:26:08,010
Speaker 1: off a little bit more separation from potential threats. The

391
00:26:08,090 --> 00:26:11,850
Speaker 1: bomb diffusing robe are still remote controlled by a technician,

392
00:26:12,330 --> 00:26:15,010
Speaker 1: but ashe Carter wants to take the idea of robots

393
00:26:15,010 --> 00:26:18,530
Speaker 1: doing the most dangerous work a step further somewhere in

394
00:26:18,570 --> 00:26:21,690
Speaker 1: the future. But I'm certain will occur. Is I think

395
00:26:21,730 --> 00:26:25,250
Speaker 1: there will be robots who will be part of infantry

396
00:26:25,290 --> 00:26:28,410
Speaker 1: squads and that will do some of the most dangerous

397
00:26:28,530 --> 00:26:32,770
Speaker 1: jobs in an infantry squad, like kicking down the door

398
00:26:32,850 --> 00:26:35,090
Speaker 1: of a building and being the first one to run

399
00:26:35,170 --> 00:26:39,610
Speaker 1: in and clear the building of terrorists or whatever. That's

400
00:26:39,610 --> 00:26:42,890
Speaker 1: a job that doesn't sound like something I would like

401
00:26:43,050 --> 00:26:46,050
Speaker 1: to have a young American man or woman doing if

402
00:26:46,090 --> 00:26:53,130
Speaker 1: I could replace them with a robot. Chapter four Harpies,

403
00:26:55,330 --> 00:26:58,690
Speaker 1: Paul Shari gave me an overview of the sophisticated unmanned

404
00:26:58,730 --> 00:27:01,810
Speaker 1: systems currently used by militaries. So I think it's worth

405
00:27:01,850 --> 00:27:07,690
Speaker 1: separate running out the value of robotics versus autonomy removing

406
00:27:07,690 --> 00:27:11,970
Speaker 1: a person from decision making. So what's so special about autonomy.

407
00:27:12,210 --> 00:27:16,650
Speaker 1: The advantages there are really about speed. Machines can make

408
00:27:16,690 --> 00:27:20,970
Speaker 1: decisions faster than humans. That's why automatic breaking and automobiles

409
00:27:21,050 --> 00:27:23,810
Speaker 1: is valuable. But you could have much faster reflexes than

410
00:27:23,850 --> 00:27:28,650
Speaker 1: a person might had. Pul separates the technology into three baskets. First,

411
00:27:29,010 --> 00:27:33,690
Speaker 1: semi autonomous weapons. Semi autonomous weapons that are widely used

412
00:27:33,690 --> 00:27:36,770
Speaker 1: around the globe today, where automation is used to maybe

413
00:27:36,770 --> 00:27:40,930
Speaker 1: help identify targets, but humans are in the final decision

414
00:27:41,010 --> 00:27:46,250
Speaker 1: about which targets to attack. Second, there are supervised autonomous weapons.

415
00:27:46,610 --> 00:27:50,570
Speaker 1: There are automatic modes that can be activated on air

416
00:27:50,610 --> 00:27:54,610
Speaker 1: and missile defense systems that allow these computers to defend

417
00:27:54,810 --> 00:27:58,410
Speaker 1: the ship or ground vehicle or land base all on

418
00:27:58,450 --> 00:28:02,210
Speaker 1: its own against these incoming threats. But humans supervise these

419
00:28:02,250 --> 00:28:07,010
Speaker 1: systems in real time. They could, at least in theory, intervene. Finally,

420
00:28:07,250 --> 00:28:11,930
Speaker 1: there are fully autonomous weapon There are a few isolated

421
00:28:11,970 --> 00:28:15,770
Speaker 1: examples of what you might consider fully autonomous weapons where

422
00:28:15,810 --> 00:28:18,970
Speaker 1: there's no human oversight and they're using an offensive capacity.

423
00:28:19,450 --> 00:28:22,130
Speaker 1: The clearest today that's an operation is the Israeli Harpy

424
00:28:22,210 --> 00:28:26,530
Speaker 1: drone that can load over a wide area after about

425
00:28:26,530 --> 00:28:28,490
Speaker 1: two and a half hours at a time to search

426
00:28:28,490 --> 00:28:31,570
Speaker 1: for enemy radars, and then when it finds one, it

427
00:28:31,610 --> 00:28:35,090
Speaker 1: can attack it all on its own without any further

428
00:28:35,170 --> 00:28:38,970
Speaker 1: human approval. Once it's launched, that decision about which particular

429
00:28:38,970 --> 00:28:42,170
Speaker 1: target to attack that's delegated to the machine. It's been

430
00:28:42,210 --> 00:28:47,370
Speaker 1: sold to a handful of countries Turkey, India, China, South Korea.

431
00:28:47,530 --> 00:28:50,530
Speaker 1: I asked Missy if she saw advantages to having autonomy

432
00:28:50,570 --> 00:28:54,730
Speaker 1: built into lethal weapons. While she had reservations, she pointed

433
00:28:54,770 --> 00:28:58,810
Speaker 1: out that in some circumstances it could prevent tragedies. A

434
00:28:58,970 --> 00:29:02,610
Speaker 1: human has something called the neuromuscular lag in them. It's

435
00:29:02,610 --> 00:29:05,210
Speaker 1: about a half second delay. So you see something, you

436
00:29:05,250 --> 00:29:10,570
Speaker 1: can execute an action a half second later. So let's

437
00:29:10,610 --> 00:29:14,650
Speaker 1: say that that guided weapon fired by a human is

438
00:29:14,690 --> 00:29:17,650
Speaker 1: going into a building, and then right before it gets

439
00:29:17,650 --> 00:29:21,050
Speaker 1: to the building, at a half second, the door opens

440
00:29:21,090 --> 00:29:24,370
Speaker 1: and a child walks out. It's too late. That child

441
00:29:24,490 --> 00:29:29,010
Speaker 1: is dead. But a lethal autonomous weapon who had a

442
00:29:29,050 --> 00:29:34,850
Speaker 1: good enough perception system could immediately detect that and immediately

443
00:29:34,890 --> 00:29:38,730
Speaker 1: guide itself to a safe place to explode. That is

444
00:29:38,770 --> 00:29:48,850
Speaker 1: a possibility in the future. Chapter five bounded morality. Some

445
00:29:48,890 --> 00:29:53,130
Speaker 1: people think, and this point is controversial, that robots might

446
00:29:53,130 --> 00:29:57,050
Speaker 1: turn out to be more humane than humans. The history

447
00:29:57,050 --> 00:30:01,290
Speaker 1: of warfare has enough examples of atrocities committed by soldiers

448
00:30:01,330 --> 00:30:04,330
Speaker 1: on all sides. For example, in the middle of the

449
00:30:04,410 --> 00:30:08,050
Speaker 1: Vietnam War in March nineteen sixty eight, a company of

450
00:30:08,050 --> 00:30:11,970
Speaker 1: American soldiers attack the village in South Vietnam, killing and

451
00:30:12,130 --> 00:30:17,610
Speaker 1: estimated five hundred and four unarmed Vietnamese men, women, and children,

452
00:30:18,170 --> 00:30:22,490
Speaker 1: all noncombatants. The horrific event became known as the Melai

453
00:30:22,650 --> 00:30:27,850
Speaker 1: massacre in nineteen sixty nine. Journalist Mike Wallace of Sixty

454
00:30:27,890 --> 00:30:31,730
Speaker 1: Minutes sat down with Private Paul Medloe, one of the

455
00:30:31,810 --> 00:30:34,650
Speaker 1: soldiers involved in the massacre. Well, I'm might a kill

456
00:30:34,690 --> 00:30:38,850
Speaker 1: about ten or fifteen of them men, women, and children

457
00:30:38,810 --> 00:30:47,730
Speaker 1: and babies and babies. You're married, right, children too? How

458
00:30:47,730 --> 00:30:52,530
Speaker 1: can a father of two young children shoot babies? I

459
00:30:52,530 --> 00:30:55,370
Speaker 1: don't know when to sworn in things. Of course, the

460
00:30:55,450 --> 00:30:58,890
Speaker 1: vast majority of soldiers do not behave this way. But

461
00:30:59,050 --> 00:31:02,570
Speaker 1: humans can be thoughtlessly violent. They can act out of anger,

462
00:31:02,690 --> 00:31:07,250
Speaker 1: out of fear, they can seek revenge, they can murder senselessly.

463
00:31:07,970 --> 00:31:11,810
Speaker 1: Can robots do better? After all, robots don't get angry,

464
00:31:11,850 --> 00:31:15,650
Speaker 1: They're not impulsive. I spoke with someone who thinks that

465
00:31:16,010 --> 00:31:20,090
Speaker 1: lethal autonomous weapons could ultimately be more humane. My name

466
00:31:20,170 --> 00:31:23,450
Speaker 1: is Ronald Arkin. I'm a regents professor at the Georgia

467
00:31:23,490 --> 00:31:27,530
Speaker 1: Institute of Technology in Atlanta, Georgia. I am a roboticist

468
00:31:27,610 --> 00:31:31,290
Speaker 1: for close to thirty five years. I've been in robot

469
00:31:31,330 --> 00:31:34,490
Speaker 1: ethics for maybe the past fifteen. Ron wanted to make

470
00:31:34,530 --> 00:31:37,090
Speaker 1: it clear that he doesn't think these robots are perfect,

471
00:31:37,810 --> 00:31:40,890
Speaker 1: but they could be better than our current option. I

472
00:31:40,930 --> 00:31:45,050
Speaker 1: am absolutely not pro lethal autonomous weapons systems because I'm

473
00:31:45,050 --> 00:31:48,690
Speaker 1: not pro lethal weapons of any sort. I am against

474
00:31:48,810 --> 00:31:51,970
Speaker 1: killing in all of its manifold forms. But the problem

475
00:31:52,130 --> 00:31:56,130
Speaker 1: is that humanity persist in entering into warfare. As such,

476
00:31:56,210 --> 00:31:59,450
Speaker 1: we must better protect the innocent in the battlespace, far

477
00:31:59,570 --> 00:32:03,010
Speaker 1: better than we currently do. So Ron thinks that lethal

478
00:32:03,050 --> 00:32:07,010
Speaker 1: autonomous weapons could prevent some of the unnecessary violence that

479
00:32:07,090 --> 00:32:11,530
Speaker 1: occurs in war. Human being don't do well in warfare

480
00:32:11,570 --> 00:32:15,050
Speaker 1: in general, and that's why there's so much room for improvement.

481
00:32:15,450 --> 00:32:19,970
Speaker 1: There's unnamed fire, there's mistakes, there's carelessness, and in the

482
00:32:20,010 --> 00:32:24,250
Speaker 1: worst case, there's the commission of atrocities, and unfortunately, all

483
00:32:24,290 --> 00:32:28,450
Speaker 1: those things lead to the depths of noncombatants. And robots

484
00:32:28,490 --> 00:32:31,010
Speaker 1: will make mistakes too. They probably will make different kinds

485
00:32:31,010 --> 00:32:34,250
Speaker 1: of mistakes, but hopefully, if done correctly, they will make

486
00:32:34,290 --> 00:32:37,570
Speaker 1: far far less mistakes than human beings do in certain

487
00:32:37,650 --> 00:32:40,930
Speaker 1: narrow circumstances where human beings are prone to those errors.

488
00:32:41,130 --> 00:32:45,810
Speaker 1: So how old the robots follow these international humanitarian standards?

489
00:32:46,450 --> 00:32:50,130
Speaker 1: The way in which we explored initially is looking at

490
00:32:50,170 --> 00:32:53,370
Speaker 1: something referred to as bounded morality, which means we look

491
00:32:53,370 --> 00:32:56,770
Speaker 1: at very narrow situations. You are not allowed to drop

492
00:32:56,850 --> 00:33:02,050
Speaker 1: bombs on schools, on hospitals, mosques, or churches. So the

493
00:33:02,130 --> 00:33:05,490
Speaker 1: point is, if you know the geographic location of those,

494
00:33:05,930 --> 00:33:10,090
Speaker 1: you can demarcate those on a map, use GPS, and

495
00:33:10,210 --> 00:33:13,410
Speaker 1: you can prevent someone from pulling a trigger. But keep

496
00:33:13,450 --> 00:33:16,050
Speaker 1: in mind these systems are not only going to decide

497
00:33:16,050 --> 00:33:18,850
Speaker 1: when to engage it, but also when not to engage

498
00:33:18,930 --> 00:33:22,210
Speaker 1: a target. They can be more conservative. I believe the

499
00:33:22,250 --> 00:33:26,410
Speaker 1: potential exists to reduce noncombatant casualties and collateral damage in

500
00:33:26,450 --> 00:33:28,610
Speaker 1: almost all of its forms over what we currently have,

501
00:33:32,930 --> 00:33:36,770
Speaker 1: so autonomous weapons might operate more efficiently, reduce risk to

502
00:33:36,770 --> 00:33:41,810
Speaker 1: one's own troops, operate faster than the enemy, decreased civilian casualties,

503
00:33:42,090 --> 00:33:51,570
Speaker 1: and perhaps avoid atrocities. What could possibly go wrong? Chapter six?

504
00:33:51,930 --> 00:33:57,130
Speaker 1: What could possibly go wrong? Autonomous systems can do some

505
00:33:57,170 --> 00:34:01,530
Speaker 1: pretty remarkable things these days, but of course, robots just

506
00:34:01,650 --> 00:34:04,690
Speaker 1: do what their computer code tells them to do. The

507
00:34:04,770 --> 00:34:08,330
Speaker 1: computer code is written by humans, or, in the case

508
00:34:08,330 --> 00:34:14,010
Speaker 1: of modern artificial intelligence, automatically inferred from training data. What

509
00:34:14,170 --> 00:34:17,250
Speaker 1: happens if a robot encounters a situation that the human

510
00:34:17,570 --> 00:34:22,730
Speaker 1: where the training data didn't anticipate, well, things could go

511
00:34:22,850 --> 00:34:27,290
Speaker 1: wrong in a hurry. One of the concerns with autonomous

512
00:34:27,330 --> 00:34:30,810
Speaker 1: weapons is that they might malfunction in a way that

513
00:34:31,010 --> 00:34:36,130
Speaker 1: leads them to begin erroneously engaging targets. Robots run amock,

514
00:34:36,490 --> 00:34:41,090
Speaker 1: and this is particularly a risk for weapons that could

515
00:34:41,130 --> 00:34:45,450
Speaker 1: target on their own. Now, this builds on a flaw

516
00:34:45,610 --> 00:34:49,650
Speaker 1: and known malfunction of machine guns today called a runaway gun.

517
00:34:50,250 --> 00:34:53,490
Speaker 1: A machine gun begins firing for one reason another and

518
00:34:53,570 --> 00:34:56,610
Speaker 1: because of the nature of a machine gun where one

519
00:34:56,610 --> 00:35:00,290
Speaker 1: bullets firing cycles the automation and brings in an ex bullet.

520
00:35:00,730 --> 00:35:03,370
Speaker 1: Once it starts firing, human doesn't have to do it,

521
00:35:03,490 --> 00:35:06,010
Speaker 1: and it will continue firing bullets. The same sort of

522
00:35:06,050 --> 00:35:11,090
Speaker 1: runaway behavior can result from small in computer code, and

523
00:35:11,210 --> 00:35:16,450
Speaker 1: the problems only multiply when autonomous systems interact at high speed.

524
00:35:17,530 --> 00:35:21,210
Speaker 1: Paul Shara points to Wall Street as a harbinger of

525
00:35:21,250 --> 00:35:23,610
Speaker 1: what can go wrong, and we end up some places

526
00:35:23,650 --> 00:35:26,930
Speaker 1: like where we are in stock trading today, where many

527
00:35:26,970 --> 00:35:30,530
Speaker 1: of the decisions are highly automated, and we get things

528
00:35:30,530 --> 00:35:35,050
Speaker 1: like flash crashes. What the pack is going on down here?

529
00:35:35,650 --> 00:35:38,330
Speaker 1: I don't know. There is fear. This is capitulation. Really.

530
00:35:39,650 --> 00:35:43,850
Speaker 1: In May twenty ten, computer algorithms drove the Dow Jones

531
00:35:44,290 --> 00:35:48,250
Speaker 1: down by nearly one thousand points in thirteen minutes, the

532
00:35:48,290 --> 00:35:51,650
Speaker 1: steepest drop it had ever seen in a day. The

533
00:35:51,730 --> 00:35:57,210
Speaker 1: concern is that a world where militaries have these algorithms

534
00:35:57,250 --> 00:36:02,210
Speaker 1: interacting at machine speed, faster than humans can respond, might

535
00:36:02,330 --> 00:36:05,610
Speaker 1: result in accidents. And that's something like a flash war.

536
00:36:06,170 --> 00:36:08,370
Speaker 1: By a flash war, you mean this thing just cycling

537
00:36:08,410 --> 00:36:12,050
Speaker 1: out of control somehow, right, But the algorithms are merely

538
00:36:12,090 --> 00:36:15,970
Speaker 1: following their programming, and they escalate a conflict into a

539
00:36:16,130 --> 00:36:19,330
Speaker 1: new area of warfare, a new level of violence, in

540
00:36:19,370 --> 00:36:22,410
Speaker 1: a way that might make it harder for humans to

541
00:36:22,490 --> 00:36:25,770
Speaker 1: then dial things back and bring things back under control.

542
00:36:26,010 --> 00:36:28,890
Speaker 1: The system only knows what it's been programmed or been

543
00:36:28,930 --> 00:36:32,330
Speaker 1: trained to know. The human can bring together all of

544
00:36:32,370 --> 00:36:36,130
Speaker 1: these other pieces of information about context, and human can

545
00:36:36,250 --> 00:36:39,650
Speaker 1: understand what's at stake. So there's no Stanislav Petrov on

546
00:36:39,690 --> 00:36:43,210
Speaker 1: the loop. That's the fear, right, is that if there's

547
00:36:43,290 --> 00:36:47,610
Speaker 1: no Petrov there to say no, what might the machines

548
00:36:47,650 --> 00:36:57,090
Speaker 1: do on their own? Chapter seven, slaughter Bots. The history

549
00:36:57,090 --> 00:37:01,090
Speaker 1: of weapons technology includes well intentioned efforts to reduce violence

550
00:37:01,130 --> 00:37:05,090
Speaker 1: and suffering that end up backfiring. I tell in the

551
00:37:05,170 --> 00:37:08,210
Speaker 1: book the story of the Gatling Gun, which was invented

552
00:37:08,210 --> 00:37:11,170
Speaker 1: by Richard Gatling during the American Civil War, and he

553
00:37:11,210 --> 00:37:14,570
Speaker 1: was motivated to invent this weapon, which was a forerunner

554
00:37:14,570 --> 00:37:18,410
Speaker 1: of the machine gun, as an effort to reduce soldiers

555
00:37:18,490 --> 00:37:20,490
Speaker 1: deaths and more. He saw all of these soldiers coming

556
00:37:20,530 --> 00:37:23,090
Speaker 1: back maimed and injured from the Civil War, and he said,

557
00:37:23,090 --> 00:37:25,970
Speaker 1: would it be great if we needed fewer people to fight?

558
00:37:26,610 --> 00:37:30,210
Speaker 1: So we invented a machine that could allow four people

559
00:37:30,370 --> 00:37:32,730
Speaker 1: to deliver the same lethal effects in the battlefield as

560
00:37:32,730 --> 00:37:35,570
Speaker 1: a hundred. Now, the effect of this wasn't actually to

561
00:37:35,610 --> 00:37:38,050
Speaker 1: reduce the number of people fighting, and we got to

562
00:37:38,090 --> 00:37:42,130
Speaker 1: World War One, we saw massive devastation and a whole

563
00:37:42,210 --> 00:37:45,610
Speaker 1: generation of young men in Europe killed because of this technology.

564
00:37:46,090 --> 00:37:48,930
Speaker 1: And so I think that's a good cautionary tale as well,

565
00:37:49,450 --> 00:37:52,490
Speaker 1: that sometimes the way the technology evolves and how it's

566
00:37:52,610 --> 00:37:55,690
Speaker 1: used may not always be how we'd like it to

567
00:37:55,690 --> 00:37:59,330
Speaker 1: be used. And even if regular armies can keep autonomous

568
00:37:59,330 --> 00:38:03,690
Speaker 1: weapons within the confines of international humanitarian law, what about

569
00:38:03,810 --> 00:38:08,930
Speaker 1: rogue actors? Remember those autonomous swarms we discussed with Ash Carter,

570
00:38:09,370 --> 00:38:12,730
Speaker 1: those tiny drones that work together to block enemy radar.

571
00:38:13,250 --> 00:38:17,050
Speaker 1: What happens if the technology spreads beyond armies? What if

572
00:38:17,090 --> 00:38:20,450
Speaker 1: a terrorist adds a gun or an explosive and maybe

573
00:38:20,530 --> 00:38:26,210
Speaker 1: facial recognition technology to those little flying bots. In twenty seventeen,

574
00:38:26,450 --> 00:38:29,970
Speaker 1: Berkeley professors Stuart Russell and the Future of Life Institute

575
00:38:30,290 --> 00:38:33,650
Speaker 1: made a mock documentary called slaughter Bots, is part of

576
00:38:33,690 --> 00:38:38,410
Speaker 1: their campaign against fully autonomous lethal drones. The nation is

577
00:38:38,450 --> 00:38:42,210
Speaker 1: still recovering from yesterday's incident, which officials are describing as

578
00:38:42,250 --> 00:38:45,850
Speaker 1: some kind of automated attack which killed eleven US senators

579
00:38:45,890 --> 00:38:48,610
Speaker 1: at the Capitol Building. They flew in from every rare,

580
00:38:48,650 --> 00:38:51,290
Speaker 1: but attack just one side of the aisle. It was

581
00:38:51,770 --> 00:38:56,770
Speaker 1: people were spreading. Unlike nuclear weapons, which are difficult to build,

582
00:38:57,130 --> 00:38:59,410
Speaker 1: you know, it's not easy to obtain or work with

583
00:38:59,490 --> 00:39:03,850
Speaker 1: weapons grade uranium, the technology to create and modify autonomous

584
00:39:03,890 --> 00:39:06,650
Speaker 1: drones is getting more and more accessible. All of the

585
00:39:06,690 --> 00:39:11,290
Speaker 1: technology you need from the automation standpoint either exists in

586
00:39:11,330 --> 00:39:15,490
Speaker 1: the vehicle already or you can download from GitHub. I

587
00:39:15,610 --> 00:39:18,890
Speaker 1: asked former Secretary of Defense As Carter, if the US

588
00:39:19,010 --> 00:39:22,490
Speaker 1: government is concerned about this sort of attack, you're right

589
00:39:22,570 --> 00:39:25,570
Speaker 1: to worry about drones and Chris. It only takes a

590
00:39:25,690 --> 00:39:29,770
Speaker 1: depraved person who can go to a store and buy

591
00:39:30,290 --> 00:39:34,010
Speaker 1: a drone to at least scare people and quite possibly

592
00:39:34,050 --> 00:39:38,010
Speaker 1: threaten people hanging a gun off of it or putting

593
00:39:38,050 --> 00:39:41,090
Speaker 1: a bomb of some kind on it, and then suddenly

594
00:39:41,130 --> 00:39:43,610
Speaker 1: people don't feel safe going to the super Bowl or

595
00:39:43,770 --> 00:39:48,610
Speaker 1: landing at the municipal airport. And we can't have that.

596
00:39:48,890 --> 00:39:51,410
Speaker 1: I mean it, certainly. As your former secretary of Defense,

597
00:39:51,610 --> 00:39:53,410
Speaker 1: my job was to make sure that we didn't put

598
00:39:53,490 --> 00:39:55,930
Speaker 1: up with that kind of stuff. I'm supposed to protect

599
00:39:56,370 --> 00:40:00,010
Speaker 1: our people, and so how do I protect people against drones?

600
00:40:00,450 --> 00:40:03,890
Speaker 1: In general? They can be shot down, but they can

601
00:40:03,930 --> 00:40:07,530
Speaker 1: put more drones up than it I can conceivably shoot at.

602
00:40:08,410 --> 00:40:11,090
Speaker 1: Not to mention, shooting at things in a Super Bowl

603
00:40:11,210 --> 00:40:15,370
Speaker 1: stadium is an inherently dangerous solution to this problem. And

604
00:40:15,570 --> 00:40:18,290
Speaker 1: so there's a more subtle way of dealing with drones.

605
00:40:19,050 --> 00:40:24,250
Speaker 1: I will either jam or take over the radio link,

606
00:40:24,650 --> 00:40:27,050
Speaker 1: and then you just tell it to fly away and

607
00:40:27,450 --> 00:40:31,410
Speaker 1: go off into the countryside somewhere and crash into a field.

608
00:40:31,690 --> 00:40:35,210
Speaker 1: All right, So help me out if I have enough autonomy,

609
00:40:35,770 --> 00:40:39,170
Speaker 1: couldn't I have drones without radio links that just get

610
00:40:39,170 --> 00:40:42,490
Speaker 1: their assignment and go off and do things. Yes, and

611
00:40:42,530 --> 00:40:46,010
Speaker 1: then your mind as a defender goes to something else.

612
00:40:46,450 --> 00:40:49,210
Speaker 1: Now that they've got their idea of what they're looking

613
00:40:49,210 --> 00:40:52,890
Speaker 1: for a set in their electronic mind. Let me change

614
00:40:52,890 --> 00:40:55,370
Speaker 1: what I look like, Let me change what the stadium

615
00:40:55,410 --> 00:40:58,170
Speaker 1: looks like to it, let me change what the target

616
00:40:58,250 --> 00:41:00,290
Speaker 1: looks like. And for the Super Bowl, what do I

617
00:41:00,290 --> 00:41:04,010
Speaker 1: do about that? Well, once I know I'm being looked at,

618
00:41:04,370 --> 00:41:09,730
Speaker 1: I have the opponent in a box. A few people

619
00:41:09,810 --> 00:41:14,210
Speaker 1: know how easy facial recognition is to fool. Because I

620
00:41:14,210 --> 00:41:18,730
Speaker 1: can wear the right kind of goggles. Are stick ping

621
00:41:18,770 --> 00:41:25,050
Speaker 1: pong balls in my cheeks? There's always a stratagem memo toself.

622
00:41:25,690 --> 00:41:28,250
Speaker 1: Next time I go to Gillette Stadium for a Patriots game,

623
00:41:28,690 --> 00:41:38,530
Speaker 1: bring ping pong balls? Really? Chapter eight, The Moral Buffer.

624
00:41:39,650 --> 00:41:42,130
Speaker 1: So we have to worry about whether lethal autonomous weapons

625
00:41:42,210 --> 00:41:44,730
Speaker 1: might run them up or fall into the wrong hands.

626
00:41:45,650 --> 00:41:49,570
Speaker 1: But there may be an even deeper question. Could fully

627
00:41:49,610 --> 00:41:53,690
Speaker 1: autonomous lethal weapons change the way we think about war?

628
00:41:54,450 --> 00:41:57,130
Speaker 1: I brought this up with Army of non author Paul Shari.

629
00:41:57,410 --> 00:42:00,130
Speaker 1: So one of the concerns about autonomous weapons is that

630
00:42:00,170 --> 00:42:04,050
Speaker 1: it might lead to a breakdown in human more responsibility

631
00:42:04,050 --> 00:42:08,250
Speaker 1: for killing and war. If the weapons themselves are choosing targets,

632
00:42:08,610 --> 00:42:10,890
Speaker 1: the people no longer feel like they're the ones doing

633
00:42:10,890 --> 00:42:14,610
Speaker 1: the killing. Now, on the plus side of things, that

634
00:42:14,690 --> 00:42:17,810
Speaker 1: might mean to less post traumatic stress in war. These

635
00:42:17,810 --> 00:42:22,370
Speaker 1: things have real burdens that weigh on people, but some

636
00:42:22,570 --> 00:42:26,090
Speaker 1: argue that the burden of killing should be a requirement

637
00:42:26,210 --> 00:42:30,650
Speaker 1: of war. It's worth also asking if nobody slept uneasy

638
00:42:30,690 --> 00:42:33,770
Speaker 1: at night, what does that look like? Would there be

639
00:42:33,890 --> 00:42:36,730
Speaker 1: less restraint in war and more killing as a result.

640
00:42:37,410 --> 00:42:40,930
Speaker 1: Missy Cummings, the former fighter pilot and current Duke professor,

641
00:42:41,250 --> 00:42:43,970
Speaker 1: wrote an influential paper in two thousand and four about

642
00:42:44,010 --> 00:42:48,050
Speaker 1: how increasing the gap between a person and their actions

643
00:42:48,250 --> 00:42:55,410
Speaker 1: creates what she called a moral buffer. People ease the

644
00:42:55,450 --> 00:43:03,650
Speaker 1: psychological and emotional pain of warfare by basically superficially layering

645
00:43:03,650 --> 00:43:06,770
Speaker 1: in these other technologies to kind of make them lose

646
00:43:06,810 --> 00:43:08,930
Speaker 1: track of what they're doing. And this is actually something

647
00:43:08,930 --> 00:43:12,410
Speaker 1: that I do think it's a problem for lethal autonomous weapons.

648
00:43:12,450 --> 00:43:15,170
Speaker 1: If we send a weapon and we'd tell it to

649
00:43:15,250 --> 00:43:19,690
Speaker 1: kill one person and it kills the wrong person, then

650
00:43:19,770 --> 00:43:22,730
Speaker 1: it's very likely that people will push off their sense

651
00:43:22,770 --> 00:43:27,890
Speaker 1: of responsibility and accountability onto the autonomous agent because they say, well,

652
00:43:27,930 --> 00:43:30,530
Speaker 1: it's not my fault, it was the autonomous agent's fault.

653
00:43:30,970 --> 00:43:33,970
Speaker 1: On the other hand, Paul Scharide tells a story about

654
00:43:34,010 --> 00:43:38,130
Speaker 1: how when there's no buffer, humans rely on an implicit

655
00:43:38,250 --> 00:43:41,530
Speaker 1: sense of morality that might be hard to explain to

656
00:43:41,610 --> 00:43:44,770
Speaker 1: a robot. There was an incident early in the war

657
00:43:44,850 --> 00:43:46,970
Speaker 1: where I was part of an army ranger sniper team

658
00:43:47,450 --> 00:43:51,210
Speaker 1: up on the Afghanistan Pakistan border and we were watching

659
00:43:51,250 --> 00:43:56,050
Speaker 1: for Taliban fighters infiltrating across the border, and when dawn came,

660
00:43:56,290 --> 00:43:58,410
Speaker 1: we weren't nearly as concealed as we had hoped to be,

661
00:43:59,130 --> 00:44:02,450
Speaker 1: and very quickly a farmer came out to relieve himself

662
00:44:02,450 --> 00:44:04,690
Speaker 1: in the fields and saw us, and we knew that

663
00:44:04,730 --> 00:44:08,290
Speaker 1: we were compromised. What I did not expect was what

664
00:44:08,330 --> 00:44:10,530
Speaker 1: they did next, which was I sent a little girl

665
00:44:10,570 --> 00:44:13,450
Speaker 1: to scout at our position. She was maybe five or six,

666
00:44:14,250 --> 00:44:18,130
Speaker 1: She was not particularly sneaky. She stared directly at us

667
00:44:18,650 --> 00:44:20,930
Speaker 1: and we heard the chirping of what we later realized

668
00:44:21,010 --> 00:44:22,930
Speaker 1: was probably a radio that you had on her, and

669
00:44:23,010 --> 00:44:26,170
Speaker 1: she was reporting back information about us, and then she

670
00:44:26,330 --> 00:44:29,130
Speaker 1: left it. Not long after, some fighters did come and

671
00:44:29,170 --> 00:44:31,890
Speaker 1: then The gun fight that ensued brought out the whole valley,

672
00:44:31,930 --> 00:44:34,050
Speaker 1: so we had to leave. But later that day we

673
00:44:33,930 --> 00:44:37,210
Speaker 1: were talking about how it would treat a situation like that.

674
00:44:37,650 --> 00:44:40,210
Speaker 1: Something that just didn't come up in conversation was the

675
00:44:40,210 --> 00:44:43,170
Speaker 1: idea of shooting this little girl. Now, what's interesting is

676
00:44:43,170 --> 00:44:46,170
Speaker 1: that under the laws of war, that would have been legal.

677
00:44:47,210 --> 00:44:49,810
Speaker 1: The laws of war don't set an age for combatants.

678
00:44:50,410 --> 00:44:52,810
Speaker 1: Your status as a combatant just based on your actions,

679
00:44:53,410 --> 00:44:57,170
Speaker 1: and by scouting for the enemy, she was directly participating

680
00:44:57,170 --> 00:45:00,890
Speaker 1: on hostilities. If you had a robot that was programmed

681
00:45:00,890 --> 00:45:03,570
Speaker 1: to perfectly comply with the laws of war, it would

682
00:45:03,610 --> 00:45:07,450
Speaker 1: have shot this little girl. There are sometimes very difficult

683
00:45:07,450 --> 00:45:09,810
Speaker 1: decisions that are forced on people in but I don't

684
00:45:09,810 --> 00:45:12,090
Speaker 1: think this was one of them. But I think it's

685
00:45:12,090 --> 00:45:14,010
Speaker 1: worth asking how would a robot know the difference between

686
00:45:14,010 --> 00:45:17,330
Speaker 1: what's legal and what's right, and how would you even

687
00:45:17,330 --> 00:45:23,970
Speaker 1: begin to prehend that into a machine. Chapter nine, The

688
00:45:24,090 --> 00:45:30,650
Speaker 1: Campaign to stop killer robots. The most fundamental moral objection

689
00:45:30,730 --> 00:45:35,450
Speaker 1: to fully autonomous lethal weapons comes down to this, As

690
00:45:35,450 --> 00:45:39,250
Speaker 1: a matter of human dignity, only a human should be

691
00:45:39,290 --> 00:45:42,450
Speaker 1: able to make the decision to kill another human. Some

692
00:45:42,490 --> 00:45:47,050
Speaker 1: things are just morally wrong, regardless of the outcome, regardless

693
00:45:47,090 --> 00:45:49,970
Speaker 1: of whether or not you know, torturing one person saves

694
00:45:50,370 --> 00:45:55,490
Speaker 1: a thousand, its torture is wrong. Slavery is wrong. And

695
00:45:55,850 --> 00:45:58,410
Speaker 1: from this point of view, one might say, well, look,

696
00:45:58,490 --> 00:46:01,290
Speaker 1: it's wrong to let a machine decide whom to kill.

697
00:46:01,770 --> 00:46:04,290
Speaker 1: Humans have to make that decision. Some people have been

698
00:46:04,290 --> 00:46:09,170
Speaker 1: working hard to turn this moral view into binding international law.

699
00:46:09,450 --> 00:46:12,690
Speaker 1: So my name is Mary Warem. I'm the advocacy director

700
00:46:12,730 --> 00:46:15,970
Speaker 1: of the Arms division of Human Rights Watch. I also

701
00:46:16,090 --> 00:46:20,450
Speaker 1: coordinate this coalition of groups called the Campaign to Stop

702
00:46:20,530 --> 00:46:23,770
Speaker 1: Killer Robots, and that's a coalition of one hundred and

703
00:46:23,850 --> 00:46:28,770
Speaker 1: twelve non governmental organizations in about fifty six countries that

704
00:46:28,930 --> 00:46:32,130
Speaker 1: is working towards a single goal, which is to create

705
00:46:32,370 --> 00:46:37,570
Speaker 1: a prohibition on fully autonomous weapons. The campaign's argument is

706
00:46:37,650 --> 00:46:41,290
Speaker 1: rooted in the Geneva Conventions, a set of treaties that

707
00:46:41,450 --> 00:46:46,690
Speaker 1: establish humanitarian standards for the conduct of war. There's the

708
00:46:46,690 --> 00:46:51,090
Speaker 1: principle of distinction, which says that armed forces must recognize

709
00:46:51,130 --> 00:46:54,770
Speaker 1: civilians and may not target them. And there's the principle

710
00:46:54,890 --> 00:46:59,690
Speaker 1: of proportionality, which says that incidental civilian deaths can't be

711
00:46:59,850 --> 00:47:05,730
Speaker 1: disproportionate to an attack direct military advantage. The campaign says

712
00:47:05,890 --> 00:47:10,410
Speaker 1: killer robots fail these tests. First, they can't distinguish between

713
00:47:10,450 --> 00:47:15,450
Speaker 1: combatants and noncombatants or tell when an enemy is surrendering. Second,

714
00:47:15,530 --> 00:47:21,450
Speaker 1: they say, deciding whether civilian deaths are disproportionate inherently requires

715
00:47:21,530 --> 00:47:25,570
Speaker 1: human judgment. For these reasons and others, the campaign says,

716
00:47:26,050 --> 00:47:31,770
Speaker 1: fully autonomous lethal weapons should be banned. Getting an international

717
00:47:31,890 --> 00:47:36,010
Speaker 1: treaty to ban fully autonomous lethal weapons might seem like

718
00:47:36,050 --> 00:47:40,490
Speaker 1: a total pipe dream, except for one thing. Mary warm

719
00:47:40,530 --> 00:47:44,370
Speaker 1: In her colleagues already pulled it off for another class

720
00:47:44,410 --> 00:47:48,770
Speaker 1: of weapons, land mines. The signing of this historic treaty

721
00:47:48,930 --> 00:47:52,250
Speaker 1: at the very end of the century is this generation's

722
00:47:52,330 --> 00:47:56,810
Speaker 1: pledge to the future. The International Campaign to Ban Landmines

723
00:47:57,090 --> 00:48:00,530
Speaker 1: and its founder, Jody Williams, received the Nobel Peace Prize

724
00:48:00,570 --> 00:48:03,370
Speaker 1: in nineteen ninety seven for their work leading to the

725
00:48:03,410 --> 00:48:08,090
Speaker 1: Ottawa Convention, which banned the use, production, sale, and stockpiling

726
00:48:08,330 --> 00:48:11,810
Speaker 1: of an anti personnel mines. While one hundred and sixty

727
00:48:11,890 --> 00:48:15,370
Speaker 1: four nations joined the treaty, some of the world's major

728
00:48:15,490 --> 00:48:20,010
Speaker 1: military powers never signed it, including the United States, China,

729
00:48:20,130 --> 00:48:25,210
Speaker 1: and Russia. Still, the treaty has worked and even influence

730
00:48:25,290 --> 00:48:28,330
Speaker 1: the holdouts. So the United States did not join, but

731
00:48:28,370 --> 00:48:32,010
Speaker 1: it went on to I think prioritize clearance of anti

732
00:48:32,010 --> 00:48:35,890
Speaker 1: personnel land mines and remains the biggest donor to clearing

733
00:48:36,290 --> 00:48:40,010
Speaker 1: landlines an unexploded ordinance around the world. And then under

734
00:48:40,010 --> 00:48:44,330
Speaker 1: the Obama administration, the US committed not to use anti

735
00:48:44,330 --> 00:48:47,650
Speaker 1: personnel land mines anywhere in the world other than to

736
00:48:47,770 --> 00:48:51,890
Speaker 1: keep the option open for the Korean peninsula. So slowly,

737
00:48:52,010 --> 00:48:56,090
Speaker 1: over time countries do I think come in line. One

738
00:48:56,170 --> 00:49:00,290
Speaker 1: major difference between banning land mines and banning fully autonomous

739
00:49:00,370 --> 00:49:04,250
Speaker 1: lethal weapons is, well, it's pretty clear what a land

740
00:49:04,250 --> 00:49:08,570
Speaker 1: mine is, but a fully autonomous lethal weapon that's not

741
00:49:08,650 --> 00:49:12,650
Speaker 1: quite as obvious. Six years of discussion at the United

742
00:49:12,730 --> 00:49:16,650
Speaker 1: Nations have yet to produce a crisp definition. Trying to

743
00:49:16,690 --> 00:49:20,570
Speaker 1: define autonomy is also a very challenging task, and this

744
00:49:20,650 --> 00:49:24,370
Speaker 1: is why we focus on the need for meaningful human control.

745
00:49:25,130 --> 00:49:29,530
Speaker 1: So what exactly is meaningful human control? The ability for

746
00:49:29,570 --> 00:49:32,970
Speaker 1: the human operator and the weapon system to communicate the

747
00:49:32,970 --> 00:49:36,410
Speaker 1: ability for the human to intervene in the detection, selection

748
00:49:36,450 --> 00:49:39,850
Speaker 1: and engagement of targets if necessary to cancel the operation.

749
00:49:40,530 --> 00:49:45,330
Speaker 1: Not surprisingly, international talks about the proposed ban are complicated.

750
00:49:45,610 --> 00:49:48,170
Speaker 1: I will say that a majority of the countries who

751
00:49:48,250 --> 00:49:52,010
Speaker 1: have been talking about killer robots have called for illegally

752
00:49:52,050 --> 00:49:55,690
Speaker 1: binding instruments and international treaty. You've got the countries who

753
00:49:55,690 --> 00:49:59,690
Speaker 1: want to be helpful, like France who was proposing working groups,

754
00:50:00,170 --> 00:50:05,450
Speaker 1: Germany who's proposed political declarations on the importance of human control.

755
00:50:05,970 --> 00:50:08,970
Speaker 1: There's a lot of proposals, I think from Australia about

756
00:50:09,170 --> 00:50:13,650
Speaker 1: legal reviews of weapons. Those efforts are being rebuffed by

757
00:50:13,690 --> 00:50:17,410
Speaker 1: a smaller handful of what we call militarily powerful countries

758
00:50:17,730 --> 00:50:21,010
Speaker 1: who don't want to see new international law. The United

759
00:50:21,050 --> 00:50:24,010
Speaker 1: States and Russia have probably been amongst the most problematic

760
00:50:24,330 --> 00:50:28,530
Speaker 1: on dismissing the calls for any form of regulation. As

761
00:50:28,570 --> 00:50:31,530
Speaker 1: with the landlines, Mary Wareham sees a path forward even

762
00:50:31,570 --> 00:50:34,650
Speaker 1: if the major military powers don't join at first. We

763
00:50:34,730 --> 00:50:38,290
Speaker 1: cannot stop every potential use. What we want to do, though,

764
00:50:38,410 --> 00:50:42,210
Speaker 1: is stigmatized, so that everybody understands that even if you

765
00:50:42,210 --> 00:50:46,130
Speaker 1: could do it, it's not right and you shouldn't. Part

766
00:50:46,130 --> 00:50:49,010
Speaker 1: of the campaign strategy is to get other groups on board,

767
00:50:49,570 --> 00:50:52,970
Speaker 1: and they're making some progress. I think a big move

768
00:50:53,170 --> 00:50:56,330
Speaker 1: in our favor came in November when the United Nations

769
00:50:56,370 --> 00:51:00,170
Speaker 1: Secretary General Antonio Guterres, he made a speech in which

770
00:51:00,170 --> 00:51:03,010
Speaker 1: he called for them to be banned under international law.

771
00:51:03,330 --> 00:51:11,130
Speaker 1: Machine the power and the dispression to take human lives

772
00:51:11,170 --> 00:51:16,770
Speaker 1: are politically and acceptable, are morally impartment and should be

773
00:51:16,810 --> 00:51:25,450
Speaker 1: banned by international law. Artificial intelligence researchers have also been

774
00:51:25,490 --> 00:51:30,890
Speaker 1: expressing concern. Since twenty fifteen, more than forty five hundred

775
00:51:30,970 --> 00:51:35,170
Speaker 1: AI and robotics researchers have signed an open letter calling

776
00:51:35,210 --> 00:51:41,010
Speaker 1: for a ban on offensive autonomous weapons beyond meaningful human control.

777
00:51:41,530 --> 00:51:46,450
Speaker 1: The signers included Elon Musk, Stephen Hawking, and Demis Assabas,

778
00:51:46,530 --> 00:51:50,090
Speaker 1: the CEO of Google's Deep Mind. An excerpt from the

779
00:51:50,210 --> 00:51:55,290
Speaker 1: letter quote, if any major military power pushes ahead with

780
00:51:55,410 --> 00:52:00,730
Speaker 1: AI weapon development, a global arms race is virtually inevitable,

781
00:52:01,330 --> 00:52:06,130
Speaker 1: and the endpoint of this technological trajectory is obvious. Autonomous

782
00:52:06,170 --> 00:52:14,690
Speaker 1: weapons will become the Khalishnikov's Tomorrow Chapter ten. To ban

783
00:52:14,930 --> 00:52:19,570
Speaker 1: or not to ban? Not Everyone, however, favors the idea

784
00:52:19,610 --> 00:52:24,410
Speaker 1: of an international treaty banning all lethal autonomous weapons. In fact,

785
00:52:24,890 --> 00:52:28,090
Speaker 1: everyone else I spoke to for this episode, Ron Arkin,

786
00:52:28,290 --> 00:52:33,450
Speaker 1: Missy Cummings, Paul Shari, and Ash Carter oppose it, interestingly,

787
00:52:33,490 --> 00:52:37,650
Speaker 1: though each had a different reason and a different alternative solution.

788
00:52:38,530 --> 00:52:42,130
Speaker 1: Robo ethicist Ron Arkin thinks we'd be missing a chance

789
00:52:42,170 --> 00:52:46,370
Speaker 1: to make wars safer. Technology can, must, and should be

790
00:52:46,490 --> 00:52:50,450
Speaker 1: used to reduce noncombatant casualties. And if it's not going

791
00:52:50,490 --> 00:52:53,490
Speaker 1: to be this, you tell me what you are going

792
00:52:53,530 --> 00:52:57,010
Speaker 1: to do to address that horrible problem that exists in

793
00:52:57,010 --> 00:52:59,330
Speaker 1: the world right now, with all these innocence being slaughtered

794
00:52:59,330 --> 00:53:02,810
Speaker 1: in the battlespace. Something needs to be done, and to me,

795
00:53:03,130 --> 00:53:07,330
Speaker 1: this is one possible way. Paul Shari thinks a comprehensive

796
00:53:07,410 --> 00:53:11,090
Speaker 1: ban is just not practical. Instead, he thinks we should

797
00:53:11,090 --> 00:53:15,770
Speaker 1: focus on banning lethal autonomous weapons that specifically target people.

798
00:53:16,330 --> 00:53:20,770
Speaker 1: That is, anti personnel weapons. In fact, the Landmine Treaty

799
00:53:20,810 --> 00:53:25,170
Speaker 1: bans anti personnel land mines, but not say, anti tank

800
00:53:25,730 --> 00:53:29,050
Speaker 1: land mines. One of the challenging things about anti personnel

801
00:53:29,090 --> 00:53:32,770
Speaker 1: weapons is that you can't stop being a person if

802
00:53:32,770 --> 00:53:35,130
Speaker 1: you wan't avoid being targeted. So if you have a

803
00:53:35,170 --> 00:53:37,530
Speaker 1: weapon that's targeting tanks, you can come out of a

804
00:53:37,570 --> 00:53:40,130
Speaker 1: tank and run away. I mean, that's a good way

805
00:53:40,210 --> 00:53:44,650
Speaker 1: to effectively surrender and render yourself life of combat. If

806
00:53:44,650 --> 00:53:48,210
Speaker 1: it's even targeting, say handheld weapons. You could set down

807
00:53:48,250 --> 00:53:49,930
Speaker 1: your weapon and run away from it. So do you

808
00:53:49,970 --> 00:53:53,730
Speaker 1: think that'd be practical to actually get either a treaty

809
00:53:53,890 --> 00:53:58,930
Speaker 1: or at least an understanding that countries should forswear anti

810
00:53:58,930 --> 00:54:03,210
Speaker 1: personnel lethal autonomous weapons. I think it's easier for me

811
00:54:03,250 --> 00:54:06,690
Speaker 1: to envision how you might get to actual restraint. You

812
00:54:06,690 --> 00:54:08,730
Speaker 1: need to make sure that the weapon that countries are

813
00:54:08,730 --> 00:54:11,650
Speaker 1: giving up it's not so valuable that they can't still

814
00:54:11,690 --> 00:54:15,050
Speaker 1: defeat those you might be willing to cheat. And I

815
00:54:15,090 --> 00:54:18,970
Speaker 1: think it's really an open question how valuable autonomous weapons are.

816
00:54:19,570 --> 00:54:22,170
Speaker 1: But my suspicion is that they are not as valuable

817
00:54:22,250 --> 00:54:26,930
Speaker 1: or necessary in an anti personnel context. Former fighter pilot

818
00:54:26,930 --> 00:54:30,610
Speaker 1: and Duke professor Missy Cummings thinks it's just not feasible

819
00:54:30,650 --> 00:54:36,490
Speaker 1: to ban lethal autonomous weapons. Look, you can't ban people

820
00:54:36,730 --> 00:54:42,250
Speaker 1: developing computer code. It's not a productive conversation to start

821
00:54:42,330 --> 00:54:46,370
Speaker 1: asking for bands on technology that are almost as common

822
00:54:46,410 --> 00:54:49,530
Speaker 1: as the air we breathe. Right, So we are not

823
00:54:49,610 --> 00:54:52,970
Speaker 1: in the world of banning nuclear technologies. And because it's

824
00:54:52,970 --> 00:54:55,370
Speaker 1: a different world, we need to come up with new ideas.

825
00:54:56,290 --> 00:54:59,370
Speaker 1: What we really need is that we make sure that

826
00:54:59,450 --> 00:55:03,730
Speaker 1: we certify these technologies in advance. How do you actually

827
00:55:03,810 --> 00:55:06,890
Speaker 1: do the test certify that the weapon does at least

828
00:55:06,890 --> 00:55:09,290
Speaker 1: as well as a human. That's actually a big problem

829
00:55:09,330 --> 00:55:12,930
Speaker 1: because no one on the planet, not the Department of Defense,

830
00:55:13,290 --> 00:55:17,930
Speaker 1: not Google, not Uber, not any driverless car company understands

831
00:55:17,970 --> 00:55:22,690
Speaker 1: how to certify autonomous technologies. So four driverless cars can

832
00:55:22,690 --> 00:55:26,970
Speaker 1: come to an intersection and they will never prosecute that

833
00:55:27,050 --> 00:55:30,730
Speaker 1: intersection the same way a sun angle can change the

834
00:55:30,770 --> 00:55:33,730
Speaker 1: way that these things think. We need to come up

835
00:55:33,810 --> 00:55:36,370
Speaker 1: with some out of the box thinking about how to

836
00:55:36,370 --> 00:55:39,050
Speaker 1: test these systems to make sure that they're seeing the world.

837
00:55:39,090 --> 00:55:42,210
Speaker 1: And I'm doing that in air quotes in a way

838
00:55:42,250 --> 00:55:45,290
Speaker 1: that we are expecting them to see the world. And

839
00:55:45,330 --> 00:55:48,170
Speaker 1: this is why we need a national agenda to understand

840
00:55:48,730 --> 00:55:52,330
Speaker 1: how to do testing to get to a place that

841
00:55:52,410 --> 00:55:55,730
Speaker 1: we feel comfortable with the results those you are successful

842
00:55:55,770 --> 00:55:59,290
Speaker 1: and you get the Pentagon and the driverless car folks

843
00:55:59,370 --> 00:56:03,370
Speaker 1: to actually do real world testing, what about rest to world?

844
00:56:03,770 --> 00:56:07,450
Speaker 1: What's going to happen? So one of the problems that

845
00:56:07,490 --> 00:56:12,690
Speaker 1: we see in all technology development is that the rest

846
00:56:12,730 --> 00:56:17,890
Speaker 1: of the world doesn't agree with our standards. It is

847
00:56:17,970 --> 00:56:22,210
Speaker 1: going to be a problem going forward, So we certainly

848
00:56:22,250 --> 00:56:28,810
Speaker 1: should not circumvent testing because other countries are circumventing testing. Finally,

849
00:56:29,010 --> 00:56:32,890
Speaker 1: there's former Secretary of Defense Ash Carter Back in twenty twelve,

850
00:56:33,250 --> 00:56:35,490
Speaker 1: ashe was one of the few people who were thinking

851
00:56:35,570 --> 00:56:39,210
Speaker 1: about the consequences of autonomous technology. At the time, he

852
00:56:39,290 --> 00:56:42,010
Speaker 1: was the third ranking official in the Pentagon in charge

853
00:56:42,010 --> 00:56:45,730
Speaker 1: of weapons and technology. He decided to draft a policy,

854
00:56:46,090 --> 00:56:50,010
Speaker 1: which the Department of Defense adopted. It was issued as

855
00:56:50,050 --> 00:56:54,890
Speaker 1: Directive three thousand point zero nine Autonomy in Weapons Systems.

856
00:56:55,370 --> 00:56:59,610
Speaker 1: So I wrote this directive that said, in essence, there

857
00:56:59,610 --> 00:57:04,210
Speaker 1: will always be a human being involved in the decision

858
00:57:04,290 --> 00:57:08,530
Speaker 1: making when it comes to lethal force in the military

859
00:57:08,530 --> 00:57:10,770
Speaker 1: of the United States of America. I'm not going to

860
00:57:10,850 --> 00:57:15,130
Speaker 1: accept autonomous weapons in a literal sense because I'm the

861
00:57:15,130 --> 00:57:17,010
Speaker 1: guy who has to go out the next morning after

862
00:57:17,050 --> 00:57:20,610
Speaker 1: some women and children have been accidentally killed and explain

863
00:57:20,690 --> 00:57:23,890
Speaker 1: it to a press conference or a foreign government or

864
00:57:23,930 --> 00:57:27,490
Speaker 1: a widow. And as suppose I go out there, Eric

865
00:57:27,570 --> 00:57:30,730
Speaker 1: and I say, oh, I don't know how it happened,

866
00:57:30,730 --> 00:57:34,010
Speaker 1: the machine, did it? Are you going to allow your

867
00:57:34,250 --> 00:57:38,570
Speaker 1: Secretary of Defense to walk out and give that kind

868
00:57:38,610 --> 00:57:42,850
Speaker 1: of excuse. No way I would be crucified. I should

869
00:57:42,890 --> 00:57:45,690
Speaker 1: be crucified for giving a press conference like that, And

870
00:57:45,850 --> 00:57:49,610
Speaker 1: I didn't think any future Secretary Defense should ever be

871
00:57:49,730 --> 00:57:52,410
Speaker 1: in that position, or allow him or herself to be

872
00:57:52,450 --> 00:57:55,170
Speaker 1: in that position. That's why I wrote the directive, Because,

873
00:57:55,250 --> 00:57:58,730
Speaker 1: ashe wrote the directive that currently prevents US forces from

874
00:57:58,770 --> 00:58:02,290
Speaker 1: deploying fully autonomous lethal weapons, I was curious to know

875
00:58:02,370 --> 00:58:06,130
Speaker 1: what he thought about an international ban. I think it's

876
00:58:06,530 --> 00:58:08,930
Speaker 1: reasonable to think about a national ban, and we have

877
00:58:09,330 --> 00:58:12,090
Speaker 1: and we have one. Do I think it's reasonable that

878
00:58:12,170 --> 00:58:14,650
Speaker 1: I get everybody else to sign up to that. I

879
00:58:14,810 --> 00:58:18,170
Speaker 1: don't because I think that people will say they'll sign

880
00:58:18,250 --> 00:58:21,210
Speaker 1: up and then not do it. In general, I don't

881
00:58:21,250 --> 00:58:27,810
Speaker 1: like fakeery in serious matters, and that's too easy to fake.

882
00:58:28,650 --> 00:58:31,650
Speaker 1: That is the fake meaning to fake that they have

883
00:58:31,810 --> 00:58:36,410
Speaker 1: forsworn those weapons, and then we find out that they haven't,

884
00:58:37,130 --> 00:58:40,130
Speaker 1: and so it turns out they're doing it, and they're

885
00:58:40,210 --> 00:58:42,610
Speaker 1: lying about doing it or hiding that they're doing it.

886
00:58:42,730 --> 00:58:45,770
Speaker 1: We've run into that all the time. I remember the

887
00:58:45,770 --> 00:58:49,690
Speaker 1: Soviet Union said it signed the Biological Weapons Convention. They

888
00:58:49,810 --> 00:58:53,010
Speaker 1: ran a very large biological warfare bird. They just said

889
00:58:53,050 --> 00:58:56,650
Speaker 1: they didn't all right, but take the situation. Now, what

890
00:58:56,730 --> 00:59:00,170
Speaker 1: would be the harm of the US signing up to

891
00:59:00,210 --> 00:59:04,850
Speaker 1: such a thing, at least building the moral approbrium around

892
00:59:05,290 --> 00:59:08,210
Speaker 1: lethal autonomous weapons, because you're building something else at the

893
00:59:08,250 --> 00:59:12,250
Speaker 1: same time, which it's an illusion of safety for other people.

894
00:59:12,490 --> 00:59:16,010
Speaker 1: You're conspiring in a circumstance in which they are lied

895
00:59:16,130 --> 00:59:21,450
Speaker 1: to about their own safety, and I feel very uncomfortable

896
00:59:21,570 --> 00:59:25,210
Speaker 1: doing that. Paul Shari sums up the challenge as well.

897
00:59:25,610 --> 00:59:29,410
Speaker 1: Countries are widely divergent interviews on things like a treaty,

898
00:59:29,450 --> 00:59:32,290
Speaker 1: but there's also been some early agreement that at some

899
00:59:32,410 --> 00:59:36,370
Speaker 1: level we need humans involved in these kinds of decisions.

900
00:59:37,170 --> 00:59:39,490
Speaker 1: What's not clear is at what level is that the

901
00:59:39,610 --> 00:59:44,370
Speaker 1: level of prisiople choosing every single target, people deciding at

902
00:59:44,370 --> 00:59:46,970
Speaker 1: a higher level what kinds of targets are to be attacked.

903
00:59:47,410 --> 00:59:50,570
Speaker 1: How far are we comfortable removing humans from these decisions.

904
00:59:51,690 --> 00:59:54,650
Speaker 1: If we had all the technology in the world, what

905
00:59:54,690 --> 00:59:56,690
Speaker 1: decisions would we want humans to make it more? And

906
00:59:56,810 --> 01:00:00,810
Speaker 1: why what decisions in the world require uniquely human judgment

907
01:00:01,410 --> 01:00:04,330
Speaker 1: and why is that? And I think if we can

908
01:00:04,370 --> 01:00:07,010
Speaker 1: answer that question, will be in a much better place

909
01:00:07,050 --> 01:00:17,770
Speaker 1: to crapple with the challenge of a hotness weapons going forward. Conclusion,

910
01:00:18,410 --> 01:00:24,330
Speaker 1: choose your planet, So there you haven't fully autonomous lethal weapons.

911
01:00:25,210 --> 01:00:30,410
Speaker 1: They might keep our soldiers safer, minimize casualties, and protect civilians,

912
01:00:31,410 --> 01:00:35,490
Speaker 1: but delegating more decision making to machines might have big

913
01:00:35,610 --> 01:00:41,010
Speaker 1: risks in unanticipated situations. They might make bad decisions that

914
01:00:41,090 --> 01:00:45,130
Speaker 1: could spiral out of control with no Stanislav Petrov in

915
01:00:45,130 --> 01:00:49,250
Speaker 1: the loop. They might even lead to flash wars. The

916
01:00:49,330 --> 01:00:53,610
Speaker 1: technology might also fall into the hands of dictators and terrorists,

917
01:00:54,370 --> 01:00:57,930
Speaker 1: and it might change us as well by increasing the

918
01:00:57,970 --> 01:01:01,970
Speaker 1: moral buffer between us and our actions. But as war

919
01:01:02,090 --> 01:01:06,090
Speaker 1: gets faster and more complex, will it really be practical

920
01:01:06,170 --> 01:01:10,090
Speaker 1: to keep humans involved in decisions? Is it time to

921
01:01:10,170 --> 01:01:13,610
Speaker 1: draw a line? Should we press for an international treaty

922
01:01:13,890 --> 01:01:18,370
Speaker 1: to completely ban what some call killer robots? What about

923
01:01:18,370 --> 01:01:22,690
Speaker 1: a limited ban or just a national ban in the US?

924
01:01:23,010 --> 01:01:26,930
Speaker 1: Or would all this be naive? Would nations ever believe

925
01:01:27,010 --> 01:01:30,970
Speaker 1: each other's promises. It's hard to know, but the right

926
01:01:31,010 --> 01:01:35,010
Speaker 1: time to decide about fully autonomous lethal weapons is probably now,

927
01:01:35,450 --> 01:01:39,530
Speaker 1: before we've gone too far down the path. The question

928
01:01:39,690 --> 01:01:43,530
Speaker 1: is what can you do a lot? It turns out

929
01:01:44,370 --> 01:01:46,770
Speaker 1: you don't have to be an expert, and you don't

930
01:01:46,810 --> 01:01:49,930
Speaker 1: have to do it alone. When enough people get engaged,

931
01:01:50,090 --> 01:01:54,770
Speaker 1: we make wise choices. Invite friends over virtually for now

932
01:01:55,330 --> 01:01:58,570
Speaker 1: in person what it's safe for dinner and debate about

933
01:01:58,570 --> 01:02:02,050
Speaker 1: what we should do. Or organize a conversation for a

934
01:02:02,090 --> 01:02:04,770
Speaker 1: book club or a faith group or a campus event.

935
01:02:05,890 --> 01:02:09,370
Speaker 1: Talk to people with firsthand experience, those who have served

936
01:02:09,410 --> 01:02:13,210
Speaker 1: in the military or been refugees from war. And don't

937
01:02:13,290 --> 01:02:16,930
Speaker 1: forget to email your elected representatives to ask what they think.

938
01:02:17,650 --> 01:02:21,330
Speaker 1: That's how questions get on the national radar. You can

939
01:02:21,370 --> 01:02:24,930
Speaker 1: find lots of resources and ideas at our website Brave

940
01:02:24,970 --> 01:02:29,050
Speaker 1: New Planet dot org. It's time to choose our planet.

941
01:02:29,810 --> 01:02:40,650
Speaker 1: The future is up to us. ED don't want a

942
01:02:40,730 --> 01:02:44,090
Speaker 1: truly autonomous car. I don't want to come to garage

943
01:02:44,090 --> 01:02:46,930
Speaker 1: and the concess. I've fallen in love with the motorcycle

944
01:02:47,170 --> 01:02:55,370
Speaker 1: and I won't drive you today because I'm autonomous. Brave

945
01:02:55,450 --> 01:02:57,810
Speaker 1: New Planet is a co production of the Broad Institute

946
01:02:57,810 --> 01:03:00,930
Speaker 1: of MT and Harvard Pushkin Industries in the Boston Globe,

947
01:03:01,450 --> 01:03:04,890
Speaker 1: with support from the Alfred P. Sloan Foundation. Our show

948
01:03:04,970 --> 01:03:08,650
Speaker 1: is produced by Rebecca Lee Douglas with Mary Doo theme

949
01:03:08,730 --> 01:03:12,730
Speaker 1: song composed by Ned Porter Mastering and sound designed by

950
01:03:12,810 --> 01:03:16,770
Speaker 1: James Garver, fact checking by Joseph Fridman and a stitt

951
01:03:16,770 --> 01:03:21,330
Speaker 1: An enchant. Special thanks to Christine Heenan and Rachel Roberts

952
01:03:21,370 --> 01:03:25,930
Speaker 1: at Clarendon Communications, to Lee McGuire, Kristen Zarelli and Justine

953
01:03:25,970 --> 01:03:29,650
Speaker 1: Levin Allerhans at the Broad, to Milobelle and Heather Faine

954
01:03:29,650 --> 01:03:33,690
Speaker 1: at Pushkin, and to Eliah Edie Brode who made the

955
01:03:33,730 --> 01:03:38,970
Speaker 1: Broad Institute possible. This is brave new planet. I'm Eric Lander.