Speaker 1: Pushkin. You're listening to Brave New Planet, a podcast about
Speaker 1: amazing new technologies that could dramatically improve our world. Or
Speaker 1: if we don't make wise choices, could leave us a
Speaker 1: lot worse off. Utopia or dystopia. It's up to us.
Speaker 1: On September twenty sixth, nineteen eighty three, the world almost
Speaker 1: came to an end. Just three weeks earlier, the Soviet
Speaker 1: Union had shot down Korean Airlines Flight Double O seven,
Speaker 1: a passenger plane with two hundred and sixty nine people aboard.
Speaker 1: I'm coming before you tonight about the Korean Airline massacre.
Speaker 1: President Ronald Reagan addressed the nation the attack by the
Speaker 1: Soviet Union against two hundred and sixty nine innocent men, women,
Speaker 1: and children aboard an unarmed Korean passenger plane. This crime
Speaker 1: against humanity must never be forgotten here or throughout the world.
Speaker 1: Cold War tensions escalated, with the two nuclear powers on
Speaker 1: high alert. World War three felt frighteningly possible. Then, on
Speaker 1: September twenty sixth, in a command center outside of Moscow,
Speaker 1: an alarm sounded. The Soviet Union's early warning system reported
Speaker 1: the launch of multiple intercontinental ballistic missiles from bases in
Speaker 1: the United states. Statislav Petrov, a forty four year old
Speaker 1: member of the Soviet Air Defense Forces, was the duty
Speaker 1: officer that night. His role was to alert Moscow that
Speaker 1: an attack was under way, likely triggering Soviet nuclear retaliation
Speaker 1: and all out war. Petrov spoke with BBC News in
Speaker 1: twenty thirteen. The sirens sounded very loudly, and I just
Speaker 1: sat there for a few seconds, staring at the screen
Speaker 1: with the word launch displayed in bold red letters. A
Speaker 1: minute later, the siren went off again. The second missile
Speaker 1: was launched, then the third, and the fourth, and the fifth.
Speaker 1: The computers changed their alerts from launch to missile strike.
Speaker 1: Petrov's instructions were clear, report the attack on the motherland,
Speaker 1: but something didn't make sense. If the US were attacking,
Speaker 1: why only five missiles rather than an entire fleet? And
Speaker 1: then I made my decision. I would not trust the computer.
Speaker 1: I picked up the telephone handset, spoke to my superiors
Speaker 1: and reported that the alarm was false. But I myself
Speaker 1: was not sure. Until the very last moment. I knew
Speaker 1: perfectly well that nobody would be able to correct my
Speaker 1: mistake if I had made one. Petrov, of course, was right.
Speaker 1: The false alarm was later found to be the result
Speaker 1: of a rare and unanticipated coincidence sunlight glinting off high
Speaker 1: altitude clouds over North Dakota at just the right angle
Speaker 1: to fool the Soviet satellites. Statislav Petrov's story comes up
Speaker 1: again and again in discussions of how far we should
Speaker 1: go and turning over important decisions, especially life and death decisions,
Speaker 1: to artificial intelligence. It's not an easy call. Think about
Speaker 1: the split second decisions and avoiding a highway collision. Who
Speaker 1: will ultimately do better a tire driver or a self
Speaker 1: driving car? Nowhere is the question more fraught than on
Speaker 1: the battlefield. As technology evolves, should weapons systems be given
Speaker 1: the power to make life and death decisions? Or do
Speaker 1: we need to ensure there's always a human a Stanislav
Speaker 1: Petrov in the loop. Some people, including winners of the
Speaker 1: Nobel Peace Prize, say that weapons should never be allowed
Speaker 1: to make their own decisions about who or what to attack.
Speaker 1: They're calling for a ban on what they call killer robots.
Speaker 1: Others think that idea is well meaning but naive. Today's
Speaker 1: big question lethal autonomous weapons. Should they ever be allowed?
Speaker 1: If so, when, if not, can we stop them? My
Speaker 1: name is Eric Lander. I'm a scientist who works on
Speaker 1: ways to improve human health. I helped lead the Human
Speaker 1: Genome Project, and today I lead the Broad Institute of
Speaker 1: and Harvard. In the twenty first century, powerful technologies have
Speaker 1: been appearing at a breathtaking pace, related to the Internet,
Speaker 1: artificial intelligence, genetic engineering, and more. They have amazing potential upsides,
Speaker 1: but we can't ignore the risks that come with them.
Speaker 1: The decisions aren't just up to scientists or politicians, whether
Speaker 1: we like it or not, we all of us are
Speaker 1: the stewards of a brave New Planet. This generation's choices
Speaker 1: will shape the future as never before. Coming up on
Speaker 1: today's episode of Brave New Planet fully autonomous lethal weapons
Speaker 1: or killer robots, we hear from a fighter pilot about
Speaker 1: why it might make sense to have machines in charge
Speaker 1: of some major battlefield decisions. I know people who have
Speaker 1: killed civilians, and in all cases where people made mistakes,
Speaker 1: it was just too much information. Things were happening too fast.
Speaker 1: I speak with one of the world's leading robo ethesis.
Speaker 1: Robots will make mistakes too, but hopefully, if done correctly,
Speaker 1: they will make far far less mistakes than human beings.
Speaker 1: We'll hear about some of the possible consequences of autonomous weapons.
Speaker 1: Algorithms interacting at machine speed faster than humans couldn't respond
Speaker 1: might result in accidents, and that's something like a flash war.
Speaker 1: I'll speak with a leader from Human Rights Watch. The
Speaker 1: campaign to stop Killer Robots is seeking new international lure
Speaker 1: in the form of a new treaty. And we'll talk
Speaker 1: with former Secretary of Defense Ash Carter. Because I'm the
Speaker 1: guy who asked to go out the next morning after
Speaker 1: some women and children have been accidentally killed. As suppose
Speaker 1: I go out there, Eric and I say, oh, I
Speaker 1: don't know how it happened. The machine did it. I
Speaker 1: would be crucified. I should be crucified. So stay with us.
Speaker 1: Chapter one, Stanley the self driving Car. Not long after
Speaker 1: the first general purpose computers were invented in the nineteen forties,
Speaker 1: some people began to dream about fully autonomous robots, machines
Speaker 1: that used their electronic brains to navigate the world, make decisions,
Speaker 1: and take actions. Not surprisingly, some of those dreamers were
Speaker 1: in the US Department of Defense, specifically the Defense Advanced
Speaker 1: Research Projects Agency or DARPA, the visionary unit behind the
Speaker 1: creation of the Internet. They saw a lot of potential
Speaker 1: for automating battlefields, but they knew it might take decades.
Speaker 1: In the nineteen sixties, DARPA funded the Stanford Research inst
Speaker 1: to build Shaky the Robots, a machine that used cameras
Speaker 1: to move about a laboratory. In the nineteen eight ease,
Speaker 1: it supported universities to create vehicles that could follow lanes
Speaker 1: on a road. By the early two thousands, DARPA decided
Speaker 1: that computers had reached the point that fully autonomous vehicles
Speaker 1: able to navigate the real world might finally be feasible.
Speaker 1: To find out, DARPA decided to launch a race. I
Speaker 1: talked to someone who knew a lot about it. My
Speaker 1: name is Sebastian Thrun. I mean the smartest person on
Speaker 1: the planet and the best looking. That's kidding. Sebastian Thrun
Speaker 1: gained recognition when his autonomous car, a modified Volkswagon with
Speaker 1: a computer in the trunk and sensors on the roof,
Speaker 1: was the first to win the DARPA Grand Challenge. A
Speaker 1: dun Challenge was his momentous government sponsors robot raises epic RaSE,
Speaker 1: can you bid a robot that can navigate one hundred
Speaker 1: and thirty punishing miles through the Mohabi Desert and the
Speaker 1: best robot like seven miles and then literally end up
Speaker 1: in many many mini researchers had concluded can't be done.
Speaker 1: In fact, many of my colleagues told me I'm going
Speaker 1: to waste my time and my name if I engaged
Speaker 1: in this kind of super hard race. And that made
Speaker 1: you more interested in doing it, of course, and so
Speaker 1: you built Stanley. Yeah, So my students built Stanley and
Speaker 1: started as a class And Stanford students are great. If
Speaker 1: you tell them go to the moon in two months,
Speaker 1: they're going to go to the moon. So then two
Speaker 1: thousand or five, the actual government sponsored race, how did
Speaker 1: Stanley do. We came in first, so we are focused
Speaker 1: insanely strongly on software and specifically on machine learning, and
Speaker 1: that differentiated as from pretty much every other team that
Speaker 1: focused on hardware. But the way I look at this
Speaker 1: is there were five teams that finished this ruling race
Speaker 1: within one year, and it's the community of the people
Speaker 1: that build all these machines that really won. So nobody
Speaker 1: made it a mile in the first race, and five
Speaker 1: different teams made it more than one hundred and thirty
Speaker 1: miles through the desert, just a year later, Yeah, that's
Speaker 1: kind of amazing to me. That just showed how fast
Speaker 1: this technology can possibly evolve. And what's happened since then?
Speaker 1: I worked at Google for a vile and eventually this guy,
Speaker 1: Larry Page came to me and says, hey, Sebastian, I
Speaker 1: thought about this long and heart. We should build a
Speaker 1: self driving car they can drive on all streets in
Speaker 1: the world. And with my entire authority, I said that
Speaker 1: cannot be done. We just had driven a desert raised
Speaker 1: there was never pedestrians and bicycles and all the other
Speaker 1: people that we could kill in the environment. And for me,
Speaker 1: just the sheer imagination we would drive a self driving
Speaker 1: car to San Francisco sounded always like a crime. So
Speaker 1: you told Larry Page, one of the two co founders
Speaker 1: of Google, that the idea of building a self driving
Speaker 1: car that could navigate anywhere was just not Yeah, feelous.
Speaker 1: Later came back and said, he Sebastian, look, I trust you,
Speaker 1: you're the expert, but I want to explain Eric Schmidt,
Speaker 1: then the Google CEO, and it's my co founder, surgery brain,
Speaker 1: why it can be done. Can you give me the
Speaker 1: technical reason? So I went home in agony, thinking about
Speaker 1: what is the technical reason why it can be done?
Speaker 1: And I got back the next day and I said, so,
Speaker 1: what is it? And I said, I can't think of
Speaker 1: any and Lomi Hoold. Eighteen months later, roughly ten engineers
Speaker 1: we drove pretty much every street in California. Today, autonomous
Speaker 1: technology is changing the transportation industry. About ten percent of
Speaker 1: cars sold in the US are already capable of at
Speaker 1: least partly guiding themselves down the highway. In twenty eighteen,
Speaker 1: Google's self driving car company Weymo launched a self driving
Speaker 1: taxi service in Phoenix, Arizona, initially with human backup drivers
Speaker 1: behind every wheel, but now sometimes even without. I asked
Speaker 1: Sebastian why he thinks this matters. We lose more than
Speaker 1: a million people in traffic accidents every year, almost exclusively
Speaker 1: to us not pay attention. When it was eighteen, my
Speaker 1: best friend died in a traffic accident and it was
Speaker 1: a split second poor decision from his friend who was
Speaker 1: driving in who also died. To me, this is just unacceptable.
Speaker 1: Beyond safety, Sebastian sees many other advantages for autonomy. During
Speaker 1: a commute, you can do something else that means you're
Speaker 1: probably willing to commute further distances. You could sleep, or
Speaker 1: watch the movie, or do email. And then eventually people
Speaker 1: can use cars that today can't operate them blind people,
Speaker 1: old people, children, babies. I mean, there's an entire spectrum
Speaker 1: of people that are kindly excluded. They would now be
Speaker 1: able to be mobile. Chapter two the Tomahawk so darpest
Speaker 1: efforts over the decades helped give rise to the modern
Speaker 1: self driving car industry, which promises to make transportations safer,
Speaker 1: more efficient, and more accessible. But the agencies primary motivation
Speaker 1: was to bring autonomy to a different challenge, the battlefield.
Speaker 1: I traveled to Chapel Hill, North Carolina, to meet with
Speaker 1: someone who spends a lot of time thinking about the
Speaker 1: consequences of autonomous technology. We both serve on a civilian
Speaker 1: advisory board for the Defense Department. My name is Missy Cummings.
Speaker 1: I'm a professor of electrical and computer engineering at Duke University,
Speaker 1: and I think one of the things that people find
Speaker 1: most interesting about me is that I was one of
Speaker 1: the US military's first female fighter pilots in the Navy.
Speaker 1: Did you always want to be a fighter pilot? So
Speaker 1: when I was growing up, I did not know that
Speaker 1: women could be pilots, and indeed, when I was growing up,
Speaker 1: women could it be pilots. And it wasn't until the
Speaker 1: late seventies that women actually became pilots in the military.
Speaker 1: So I went to college. In nineteen eighty four, I
Speaker 1: was at the Naval Academy, and then of course, in
Speaker 1: nineteen eighty six Top Gun came out, and then I
Speaker 1: know who doesn't want to be a pilot After you
Speaker 1: see the movie Top Gun. Missy is tremendously proud of
Speaker 1: the eleven years she spent in the Navy, but she
Speaker 1: also acknowledges the challenges of being part of that first
Speaker 1: generation of woman fighter pilots. It's no secret that the
Speaker 1: reason I left the military was because of the hostile
Speaker 1: attitude towards women. None of the women in that first
Speaker 1: group stayed in to make it a career. The guys
Speaker 1: were very angry that we were there, and I decided
Speaker 1: to leave when they started sabotaging my flight gear. I
Speaker 1: just thought, this is too much. If something really bad happened,
Speaker 1: you know, I would die. When Missy Cummings left the Navy,
Speaker 1: she decided to pursue a PhD in Human Machine interaction.
Speaker 1: In my last three years flying IF eighteens, there were
Speaker 1: about thirty six people I knew that died, about one
Speaker 1: person a month. They were all training accidents. It just
Speaker 1: really struck me how many people were dying because the
Speaker 1: design of the airplane just did not go with the
Speaker 1: human tendencies. And so I decided to go back to
Speaker 1: school to find out what can be done about that.
Speaker 1: So I went to finish my PhD at the University
Speaker 1: of Virginia, and then I spent the next ten years
Speaker 1: at MT learning my craft. The person I am today
Speaker 1: is half because of the Navy and half because of MIGHT. Today,
Speaker 1: Missy is a Duke University where she runs the Humans
Speaker 1: an Autonomy Lab, or for short HOW. It's a nod
Speaker 1: to the sentient computer that goes rogue in Stanley Kubrick's
Speaker 1: film two thousand and one, A Space Odyssey. This mission
Speaker 1: is too important for me to allow you to jeopardize it.
Speaker 1: I don't know what you're talking about. How. I know
Speaker 1: that you and Frank were planning to disconnect me, and
Speaker 1: I'm afraid that's something I cannot allow to happen. And
Speaker 1: so I intentionally named my lab how so that we
Speaker 1: were there to stop that from happening. Right, I had
Speaker 1: seen many friends die, not because the robot became sentient,
Speaker 1: in fact, because the designers of the automation really had
Speaker 1: no clue how people would or would not use this technology.
Speaker 1: It is my life's mission statement to develop human collaborative
Speaker 1: computer systems that work with each other to achieve something
Speaker 1: greater than either would alone. The Humans and Autonomy Lab
Speaker 1: works on the interactions between humans and machines across many fields,
Speaker 1: but given her background, Missy's thought a lot about how
Speaker 1: technology has changed the relationship between humans and their weapons.
Speaker 1: There's a long history of us distancing ourselves from our actions.
Speaker 1: We want to shoot somebody, we wanted to shoot them
Speaker 1: with bows and arrows. We wanted to drop bombs from
Speaker 1: five miles over a target. We want cruise muscles that
Speaker 1: can kill you from another country. Right, it is human
Speaker 1: nature to back that distance up, Missy season inherent tension.
Speaker 1: On one hand, technology distances ourselves from killing. On the
Speaker 1: other hand, technology is letting us design weapons that are
Speaker 1: more accurate and less indiscriminate in their killing. Missy rotor
Speaker 1: PhD thesis about the Tomahawk missile, an early precursor of
Speaker 1: the autonomous weapons systems being developed today. The Tomahawk missile
Speaker 1: has these stored maps in its brain, and as its
Speaker 1: skimming along the nap of the Earth, it compares the
Speaker 1: pictures that it's taking with its pictures and its database
Speaker 1: to decide how to get to its target. This Tomahawk
Speaker 1: was kind of a set it and forget it kind
Speaker 1: of thing. Once you launched it, it would follow its
Speaker 1: map to the right place and there was nobody looking
Speaker 1: over its shoulders. Well, so the Tomahawk missile that we
Speaker 1: saw in the Gulf War, that was a fire and
Speaker 1: forget missile that a target would be programmed into the
Speaker 1: sole and then it would be fired and that's where
Speaker 1: it would go. Later, around two thousand, two thousand and three,
Speaker 1: then GPS technology was coming online, and that's when we
Speaker 1: got the tactical Tomahawk, which had the ability to be
Speaker 1: redirected in flight. That success with GPS and the Tomahawk
Speaker 1: opened the military's eyes to the ability to use them
Speaker 1: in drones. Today's precision guided weapons are far more accurate
Speaker 1: than the widespread aerial bombing that occurred on all sides
Speaker 1: in World War Two, where some cities were almost entirely leveled,
Speaker 1: resulting in huge numbers of civilian casualties. In the Gulf War,
Speaker 1: Tomahawk missile attacks came to be called surgical strikes. I
Speaker 1: know people who have killed civilians, and I know people
Speaker 1: who have killed friendlies. They have dropped bombs on our
Speaker 1: own forces and killed our own people. And in all
Speaker 1: cases where people made mistakes, it was just too much information.
Speaker 1: Things were happening too fast. You've seen some pictures that
Speaker 1: you've got in a brief hours ago, and you're supposed
Speaker 1: to know that what you're seeing now through this grainy
Speaker 1: image thirty five thousand feet over a target is the
Speaker 1: same image that you're being asked to bob. The Tomahawk
Speaker 1: never missed its target. It never made a mistake unless
Speaker 1: it was programmed as a mistake. And that's old autonomy,
Speaker 1: and it's only gotten better over time. Chapter three, Kicking
Speaker 1: down Doors. The Tomahawk was just a baby step toward automation.
Speaker 1: With the ability to read maps, it could correct its course,
Speaker 1: but it couldn't make sophisticated decisions. But what happens when
Speaker 1: you start adding modern artificial intelligence? So where do you
Speaker 1: see autonomous weapons going? If you could kind of map
Speaker 1: out where are we today and where do you think
Speaker 1: we'll be ten twenty years from now. So, in terms
Speaker 1: of autonomy and weapons, by today's standards, the Tomahawk missile
Speaker 1: is still one of the best ones that we have,
Speaker 1: and it's also still one of the most advanced. Certainly,
Speaker 1: there are research arms of the military who are trying
Speaker 1: very hard to come up with new forms of autonomy.
Speaker 1: There was the Predicts that came out of Lincoln Lab,
Speaker 1: and this was basically a swarm of really tiny UAVs
Speaker 1: that could coordinate together. A ua V, an unmanned aerial
Speaker 1: vehicle is military speak for a drone. The Predicts the
Speaker 1: drones that Missy was referring to. They were commissioned by
Speaker 1: the Strategic Capabilities Office of the US Department of Defense.
Speaker 1: These tiny flying robots are able to communicate with each
Speaker 1: other and make split second decisions about how to move
Speaker 1: as a group. Many researchers, have you been using bio
Speaker 1: inspired methods? Be right? So bees have local and global intelligence.
Speaker 1: Like a group of bees, these drones are called a
Speaker 1: swarm collective intelligence on a shared mission. A human can
Speaker 1: make the big picture decision and the swarm of microdrones
Speaker 1: can then collectively decide on the most efficient way to
Speaker 1: carry out the order in the moment. I wanted to
Speaker 1: know why exactly this technology is necessary, so I went
Speaker 1: to speak to someone who I was pretty sure would know.
Speaker 1: I'm Ash Carter. Most people will probably have heard my
Speaker 1: name as the Secretary of Defense who proceeded Gimatus. You
Speaker 1: will know me in part from the fact that we
Speaker 1: knew one another way back in Oxford when we were
Speaker 1: both young scientists, and I guess agents start there. I'm
Speaker 1: a physicist. When you were doing your PhD in physics,
Speaker 1: I was doing my PhD in mathematics at Oxford. What
Speaker 1: was your thesis on? It was on quantum chrominynamics. That
Speaker 1: was the theory of quarks and gluons. And how in
Speaker 1: the world is somebody who's an expert in quantum chromodynamics
Speaker 1: become the Secretary of Defense. It's an interesting story. The
Speaker 1: people who were the seniors in my field of physics,
Speaker 1: the mentors, so to speak, were all members of the
Speaker 1: Manhattan Project generation. They had built the bomb during World
Speaker 1: War Two, and they were proud of what they'd done
Speaker 1: because they believed that it had ended the war with
Speaker 1: fewer casualties than otherwise there would have been in a
Speaker 1: full scale of invasion of Japan, and also that it
Speaker 1: had kept the peace through the Cold War, so they
Speaker 1: were proud of it. However, they knew there was a
Speaker 1: dark side, and they conveyed to me that it was
Speaker 1: my responsibility as a scientist to be involved in these matters.
Speaker 1: And the technology doesn't determine what the balance of good
Speaker 1: and bad is. We human beings do. That was the lesson,
Speaker 1: and so that's what got me started, and then my
Speaker 1: very first Pentagon job, which was in nineteen eighty one,
Speaker 1: right through until the last time I walked out of
Speaker 1: the Pentagon of Sectary Defense, which was January of twenty seventeen. Now,
Speaker 1: when you were secretary, there was a Strategic Capabilities Office
Speaker 1: that it's been publicly reported, was experimenting with using drones
Speaker 1: to make swarms of drones that could do things, communicate
Speaker 1: with each other, make formations. Why would you want such things?
Speaker 1: So it's a good question. Here's what you do with
Speaker 1: the drone like that, You put a jammer on it,
Speaker 1: a little radio beacon, and you fly it right into
Speaker 1: the eye of a enemy radar. So all that radar
Speaker 1: c's is the energy emitted by that little drone, and
Speaker 1: it's essentially dazzled or blinded. If there's one big drone,
Speaker 1: that radar is precious enough that the defenders going to
Speaker 1: shoot that drone down. But if you have so many
Speaker 1: out there, the enemy can't afford to shoot them all down.
Speaker 1: And since they are flying right up to the radar,
Speaker 1: they don't have to be very powerful. So there's an
Speaker 1: application where lots of little drones can have the effect
Speaker 1: of nullifying enemy radar. That's a pretty big deal for
Speaker 1: a few little, little microdrones. To learn more, I went
Speaker 1: to speak with Paul Shari. Paul's the director of the
Speaker 1: Technology and National Security Program at the Center for a
Speaker 1: New American Security. Before that, he worked for Ash Carter
Speaker 1: at the Pentagon studying autonomous weapons and a recently authored
Speaker 1: a book called Army of None, Autonomous Weapons and the
Speaker 1: Future of War. Paul's interest in autonomous weapons began when
Speaker 1: he served in the Army. I enlisted in the Army
Speaker 1: to become an Army ranger. That was June of two
Speaker 1: thousand and one, did a number of tours overseas and
Speaker 1: the wars and a rock Afghanistan. So I'll say one
Speaker 1: moment that stuck out for me where I really sort
Speaker 1: of the light bulb went on about the power of
Speaker 1: robotics in warfare. I was in a rock in two
Speaker 1: thousand and seven. We were on a patrol, driving along
Speaker 1: in a Striker armored vehicle. Came across an ID improvised
Speaker 1: explosive device makeshift road type bomb, and so we called
Speaker 1: up bomb disposal folks. So they show up and I'm
Speaker 1: expecting the bomb tech to come out in that big
Speaker 1: bomb suit that you might have seen in the movie
Speaker 1: The hurt Locker, for example, and instead out rolls out
Speaker 1: a little robot and I kind of went, oh, that
Speaker 1: makes a lot of sense. Have the robot diffused the bomb. Well,
Speaker 1: it turns out there's a lot of things in war
Speaker 1: that are super dangerous where it makes sense to have
Speaker 1: robots out on the front lines, getting people better stand
Speaker 1: off a little bit more separation from potential threats. The
Speaker 1: bomb diffusing robe are still remote controlled by a technician,
Speaker 1: but ashe Carter wants to take the idea of robots
Speaker 1: doing the most dangerous work a step further somewhere in
Speaker 1: the future. But I'm certain will occur. Is I think
Speaker 1: there will be robots who will be part of infantry
Speaker 1: squads and that will do some of the most dangerous
Speaker 1: jobs in an infantry squad, like kicking down the door
Speaker 1: of a building and being the first one to run
Speaker 1: in and clear the building of terrorists or whatever. That's
Speaker 1: a job that doesn't sound like something I would like
Speaker 1: to have a young American man or woman doing if
Speaker 1: I could replace them with a robot. Chapter four Harpies,
Speaker 1: Paul Shari gave me an overview of the sophisticated unmanned
Speaker 1: systems currently used by militaries. So I think it's worth
Speaker 1: separate running out the value of robotics versus autonomy removing
Speaker 1: a person from decision making. So what's so special about autonomy.
Speaker 1: The advantages there are really about speed. Machines can make
Speaker 1: decisions faster than humans. That's why automatic breaking and automobiles
Speaker 1: is valuable. But you could have much faster reflexes than
Speaker 1: a person might had. Pul separates the technology into three baskets. First,
Speaker 1: semi autonomous weapons. Semi autonomous weapons that are widely used
Speaker 1: around the globe today, where automation is used to maybe
Speaker 1: help identify targets, but humans are in the final decision
Speaker 1: about which targets to attack. Second, there are supervised autonomous weapons.
Speaker 1: There are automatic modes that can be activated on air
Speaker 1: and missile defense systems that allow these computers to defend
Speaker 1: the ship or ground vehicle or land base all on
Speaker 1: its own against these incoming threats. But humans supervise these
Speaker 1: systems in real time. They could, at least in theory, intervene. Finally,
Speaker 1: there are fully autonomous weapon There are a few isolated
Speaker 1: examples of what you might consider fully autonomous weapons where
Speaker 1: there's no human oversight and they're using an offensive capacity.
Speaker 1: The clearest today that's an operation is the Israeli Harpy
Speaker 1: drone that can load over a wide area after about
Speaker 1: two and a half hours at a time to search
Speaker 1: for enemy radars, and then when it finds one, it
Speaker 1: can attack it all on its own without any further
Speaker 1: human approval. Once it's launched, that decision about which particular
Speaker 1: target to attack that's delegated to the machine. It's been
Speaker 1: sold to a handful of countries Turkey, India, China, South Korea.
Speaker 1: I asked Missy if she saw advantages to having autonomy
Speaker 1: built into lethal weapons. While she had reservations, she pointed
Speaker 1: out that in some circumstances it could prevent tragedies. A
Speaker 1: human has something called the neuromuscular lag in them. It's
Speaker 1: about a half second delay. So you see something, you
Speaker 1: can execute an action a half second later. So let's
Speaker 1: say that that guided weapon fired by a human is
Speaker 1: going into a building, and then right before it gets
Speaker 1: to the building, at a half second, the door opens
Speaker 1: and a child walks out. It's too late. That child
Speaker 1: is dead. But a lethal autonomous weapon who had a
Speaker 1: good enough perception system could immediately detect that and immediately
Speaker 1: guide itself to a safe place to explode. That is
Speaker 1: a possibility in the future. Chapter five bounded morality. Some
Speaker 1: people think, and this point is controversial, that robots might
Speaker 1: turn out to be more humane than humans. The history
Speaker 1: of warfare has enough examples of atrocities committed by soldiers
Speaker 1: on all sides. For example, in the middle of the
Speaker 1: Vietnam War in March nineteen sixty eight, a company of
Speaker 1: American soldiers attack the village in South Vietnam, killing and
Speaker 1: estimated five hundred and four unarmed Vietnamese men, women, and children,
Speaker 1: all noncombatants. The horrific event became known as the Melai
Speaker 1: massacre in nineteen sixty nine. Journalist Mike Wallace of Sixty
Speaker 1: Minutes sat down with Private Paul Medloe, one of the
Speaker 1: soldiers involved in the massacre. Well, I'm might a kill
Speaker 1: about ten or fifteen of them men, women, and children
Speaker 1: and babies and babies. You're married, right, children too? How
Speaker 1: can a father of two young children shoot babies? I
Speaker 1: don't know when to sworn in things. Of course, the
Speaker 1: vast majority of soldiers do not behave this way. But
Speaker 1: humans can be thoughtlessly violent. They can act out of anger,
Speaker 1: out of fear, they can seek revenge, they can murder senselessly.
Speaker 1: Can robots do better? After all, robots don't get angry,
Speaker 1: They're not impulsive. I spoke with someone who thinks that
Speaker 1: lethal autonomous weapons could ultimately be more humane. My name
Speaker 1: is Ronald Arkin. I'm a regents professor at the Georgia
Speaker 1: Institute of Technology in Atlanta, Georgia. I am a roboticist
Speaker 1: for close to thirty five years. I've been in robot
Speaker 1: ethics for maybe the past fifteen. Ron wanted to make
Speaker 1: it clear that he doesn't think these robots are perfect,
Speaker 1: but they could be better than our current option. I
Speaker 1: am absolutely not pro lethal autonomous weapons systems because I'm
Speaker 1: not pro lethal weapons of any sort. I am against
Speaker 1: killing in all of its manifold forms. But the problem
Speaker 1: is that humanity persist in entering into warfare. As such,
Speaker 1: we must better protect the innocent in the battlespace, far
Speaker 1: better than we currently do. So Ron thinks that lethal
Speaker 1: autonomous weapons could prevent some of the unnecessary violence that
Speaker 1: occurs in war. Human being don't do well in warfare
Speaker 1: in general, and that's why there's so much room for improvement.
Speaker 1: There's unnamed fire, there's mistakes, there's carelessness, and in the
Speaker 1: worst case, there's the commission of atrocities, and unfortunately, all
Speaker 1: those things lead to the depths of noncombatants. And robots
Speaker 1: will make mistakes too. They probably will make different kinds
Speaker 1: of mistakes, but hopefully, if done correctly, they will make
Speaker 1: far far less mistakes than human beings do in certain
Speaker 1: narrow circumstances where human beings are prone to those errors.
Speaker 1: So how old the robots follow these international humanitarian standards?
Speaker 1: The way in which we explored initially is looking at
Speaker 1: something referred to as bounded morality, which means we look
Speaker 1: at very narrow situations. You are not allowed to drop
Speaker 1: bombs on schools, on hospitals, mosques, or churches. So the
Speaker 1: point is, if you know the geographic location of those,
Speaker 1: you can demarcate those on a map, use GPS, and
Speaker 1: you can prevent someone from pulling a trigger. But keep
Speaker 1: in mind these systems are not only going to decide
Speaker 1: when to engage it, but also when not to engage
Speaker 1: a target. They can be more conservative. I believe the
Speaker 1: potential exists to reduce noncombatant casualties and collateral damage in
Speaker 1: almost all of its forms over what we currently have,
Speaker 1: so autonomous weapons might operate more efficiently, reduce risk to
Speaker 1: one's own troops, operate faster than the enemy, decreased civilian casualties,
Speaker 1: and perhaps avoid atrocities. What could possibly go wrong? Chapter six?
Speaker 1: What could possibly go wrong? Autonomous systems can do some
Speaker 1: pretty remarkable things these days, but of course, robots just
Speaker 1: do what their computer code tells them to do. The
Speaker 1: computer code is written by humans, or, in the case
Speaker 1: of modern artificial intelligence, automatically inferred from training data. What
Speaker 1: happens if a robot encounters a situation that the human
Speaker 1: where the training data didn't anticipate, well, things could go
Speaker 1: wrong in a hurry. One of the concerns with autonomous
Speaker 1: weapons is that they might malfunction in a way that
Speaker 1: leads them to begin erroneously engaging targets. Robots run amock,
Speaker 1: and this is particularly a risk for weapons that could
Speaker 1: target on their own. Now, this builds on a flaw
Speaker 1: and known malfunction of machine guns today called a runaway gun.
Speaker 1: A machine gun begins firing for one reason another and
Speaker 1: because of the nature of a machine gun where one
Speaker 1: bullets firing cycles the automation and brings in an ex bullet.
Speaker 1: Once it starts firing, human doesn't have to do it,
Speaker 1: and it will continue firing bullets. The same sort of
Speaker 1: runaway behavior can result from small in computer code, and
Speaker 1: the problems only multiply when autonomous systems interact at high speed.
Speaker 1: Paul Shara points to Wall Street as a harbinger of
Speaker 1: what can go wrong, and we end up some places
Speaker 1: like where we are in stock trading today, where many
Speaker 1: of the decisions are highly automated, and we get things
Speaker 1: like flash crashes. What the pack is going on down here?
Speaker 1: I don't know. There is fear. This is capitulation. Really.
Speaker 1: In May twenty ten, computer algorithms drove the Dow Jones
Speaker 1: down by nearly one thousand points in thirteen minutes, the
Speaker 1: steepest drop it had ever seen in a day. The
Speaker 1: concern is that a world where militaries have these algorithms
Speaker 1: interacting at machine speed, faster than humans can respond, might
Speaker 1: result in accidents. And that's something like a flash war.
Speaker 1: By a flash war, you mean this thing just cycling
Speaker 1: out of control somehow, right, But the algorithms are merely
Speaker 1: following their programming, and they escalate a conflict into a
Speaker 1: new area of warfare, a new level of violence, in
Speaker 1: a way that might make it harder for humans to
Speaker 1: then dial things back and bring things back under control.
Speaker 1: The system only knows what it's been programmed or been
Speaker 1: trained to know. The human can bring together all of
Speaker 1: these other pieces of information about context, and human can
Speaker 1: understand what's at stake. So there's no Stanislav Petrov on
Speaker 1: the loop. That's the fear, right, is that if there's
Speaker 1: no Petrov there to say no, what might the machines
Speaker 1: do on their own? Chapter seven, slaughter Bots. The history
Speaker 1: of weapons technology includes well intentioned efforts to reduce violence
Speaker 1: and suffering that end up backfiring. I tell in the
Speaker 1: book the story of the Gatling Gun, which was invented
Speaker 1: by Richard Gatling during the American Civil War, and he
Speaker 1: was motivated to invent this weapon, which was a forerunner
Speaker 1: of the machine gun, as an effort to reduce soldiers
Speaker 1: deaths and more. He saw all of these soldiers coming
Speaker 1: back maimed and injured from the Civil War, and he said,
Speaker 1: would it be great if we needed fewer people to fight?
Speaker 1: So we invented a machine that could allow four people
Speaker 1: to deliver the same lethal effects in the battlefield as
Speaker 1: a hundred. Now, the effect of this wasn't actually to
Speaker 1: reduce the number of people fighting, and we got to
Speaker 1: World War One, we saw massive devastation and a whole
Speaker 1: generation of young men in Europe killed because of this technology.
Speaker 1: And so I think that's a good cautionary tale as well,
Speaker 1: that sometimes the way the technology evolves and how it's
Speaker 1: used may not always be how we'd like it to
Speaker 1: be used. And even if regular armies can keep autonomous
Speaker 1: weapons within the confines of international humanitarian law, what about
Speaker 1: rogue actors? Remember those autonomous swarms we discussed with Ash Carter,
Speaker 1: those tiny drones that work together to block enemy radar.
Speaker 1: What happens if the technology spreads beyond armies? What if
Speaker 1: a terrorist adds a gun or an explosive and maybe
Speaker 1: facial recognition technology to those little flying bots. In twenty seventeen,
Speaker 1: Berkeley professors Stuart Russell and the Future of Life Institute
Speaker 1: made a mock documentary called slaughter Bots, is part of
Speaker 1: their campaign against fully autonomous lethal drones. The nation is
Speaker 1: still recovering from yesterday's incident, which officials are describing as
Speaker 1: some kind of automated attack which killed eleven US senators
Speaker 1: at the Capitol Building. They flew in from every rare,
Speaker 1: but attack just one side of the aisle. It was
Speaker 1: people were spreading. Unlike nuclear weapons, which are difficult to build,
Speaker 1: you know, it's not easy to obtain or work with
Speaker 1: weapons grade uranium, the technology to create and modify autonomous
Speaker 1: drones is getting more and more accessible. All of the
Speaker 1: technology you need from the automation standpoint either exists in
Speaker 1: the vehicle already or you can download from GitHub. I
Speaker 1: asked former Secretary of Defense As Carter, if the US
Speaker 1: government is concerned about this sort of attack, you're right
Speaker 1: to worry about drones and Chris. It only takes a
Speaker 1: depraved person who can go to a store and buy
Speaker 1: a drone to at least scare people and quite possibly
Speaker 1: threaten people hanging a gun off of it or putting
Speaker 1: a bomb of some kind on it, and then suddenly
Speaker 1: people don't feel safe going to the super Bowl or
Speaker 1: landing at the municipal airport. And we can't have that.
Speaker 1: I mean it, certainly. As your former secretary of Defense,
Speaker 1: my job was to make sure that we didn't put
Speaker 1: up with that kind of stuff. I'm supposed to protect
Speaker 1: our people, and so how do I protect people against drones?
Speaker 1: In general? They can be shot down, but they can
Speaker 1: put more drones up than it I can conceivably shoot at.
Speaker 1: Not to mention, shooting at things in a Super Bowl
Speaker 1: stadium is an inherently dangerous solution to this problem. And
Speaker 1: so there's a more subtle way of dealing with drones.
Speaker 1: I will either jam or take over the radio link,
Speaker 1: and then you just tell it to fly away and
Speaker 1: go off into the countryside somewhere and crash into a field.
Speaker 1: All right, So help me out if I have enough autonomy,
Speaker 1: couldn't I have drones without radio links that just get
Speaker 1: their assignment and go off and do things. Yes, and
Speaker 1: then your mind as a defender goes to something else.
Speaker 1: Now that they've got their idea of what they're looking
Speaker 1: for a set in their electronic mind. Let me change
Speaker 1: what I look like, Let me change what the stadium
Speaker 1: looks like to it, let me change what the target
Speaker 1: looks like. And for the Super Bowl, what do I
Speaker 1: do about that? Well, once I know I'm being looked at,
Speaker 1: I have the opponent in a box. A few people
Speaker 1: know how easy facial recognition is to fool. Because I
Speaker 1: can wear the right kind of goggles. Are stick ping
Speaker 1: pong balls in my cheeks? There's always a stratagem memo toself.
Speaker 1: Next time I go to Gillette Stadium for a Patriots game,
Speaker 1: bring ping pong balls? Really? Chapter eight, The Moral Buffer.
Speaker 1: So we have to worry about whether lethal autonomous weapons
Speaker 1: might run them up or fall into the wrong hands.
Speaker 1: But there may be an even deeper question. Could fully
Speaker 1: autonomous lethal weapons change the way we think about war?
Speaker 1: I brought this up with Army of non author Paul Shari.
Speaker 1: So one of the concerns about autonomous weapons is that
Speaker 1: it might lead to a breakdown in human more responsibility
Speaker 1: for killing and war. If the weapons themselves are choosing targets,
Speaker 1: the people no longer feel like they're the ones doing
Speaker 1: the killing. Now, on the plus side of things, that
Speaker 1: might mean to less post traumatic stress in war. These
Speaker 1: things have real burdens that weigh on people, but some
Speaker 1: argue that the burden of killing should be a requirement
Speaker 1: of war. It's worth also asking if nobody slept uneasy
Speaker 1: at night, what does that look like? Would there be
Speaker 1: less restraint in war and more killing as a result.
Speaker 1: Missy Cummings, the former fighter pilot and current Duke professor,
Speaker 1: wrote an influential paper in two thousand and four about
Speaker 1: how increasing the gap between a person and their actions
Speaker 1: creates what she called a moral buffer. People ease the
Speaker 1: psychological and emotional pain of warfare by basically superficially layering
Speaker 1: in these other technologies to kind of make them lose
Speaker 1: track of what they're doing. And this is actually something
Speaker 1: that I do think it's a problem for lethal autonomous weapons.
Speaker 1: If we send a weapon and we'd tell it to
Speaker 1: kill one person and it kills the wrong person, then
Speaker 1: it's very likely that people will push off their sense
Speaker 1: of responsibility and accountability onto the autonomous agent because they say, well,
Speaker 1: it's not my fault, it was the autonomous agent's fault.
Speaker 1: On the other hand, Paul Scharide tells a story about
Speaker 1: how when there's no buffer, humans rely on an implicit
Speaker 1: sense of morality that might be hard to explain to
Speaker 1: a robot. There was an incident early in the war
Speaker 1: where I was part of an army ranger sniper team
Speaker 1: up on the Afghanistan Pakistan border and we were watching
Speaker 1: for Taliban fighters infiltrating across the border, and when dawn came,
Speaker 1: we weren't nearly as concealed as we had hoped to be,
Speaker 1: and very quickly a farmer came out to relieve himself
Speaker 1: in the fields and saw us, and we knew that
Speaker 1: we were compromised. What I did not expect was what
Speaker 1: they did next, which was I sent a little girl
Speaker 1: to scout at our position. She was maybe five or six,
Speaker 1: She was not particularly sneaky. She stared directly at us
Speaker 1: and we heard the chirping of what we later realized
Speaker 1: was probably a radio that you had on her, and
Speaker 1: she was reporting back information about us, and then she
Speaker 1: left it. Not long after, some fighters did come and
Speaker 1: then The gun fight that ensued brought out the whole valley,
Speaker 1: so we had to leave. But later that day we
Speaker 1: were talking about how it would treat a situation like that.
Speaker 1: Something that just didn't come up in conversation was the
Speaker 1: idea of shooting this little girl. Now, what's interesting is
Speaker 1: that under the laws of war, that would have been legal.
Speaker 1: The laws of war don't set an age for combatants.
Speaker 1: Your status as a combatant just based on your actions,
Speaker 1: and by scouting for the enemy, she was directly participating
Speaker 1: on hostilities. If you had a robot that was programmed
Speaker 1: to perfectly comply with the laws of war, it would
Speaker 1: have shot this little girl. There are sometimes very difficult
Speaker 1: decisions that are forced on people in but I don't
Speaker 1: think this was one of them. But I think it's
Speaker 1: worth asking how would a robot know the difference between
Speaker 1: what's legal and what's right, and how would you even
Speaker 1: begin to prehend that into a machine. Chapter nine, The
Speaker 1: Campaign to stop killer robots. The most fundamental moral objection
Speaker 1: to fully autonomous lethal weapons comes down to this, As
Speaker 1: a matter of human dignity, only a human should be
Speaker 1: able to make the decision to kill another human. Some
Speaker 1: things are just morally wrong, regardless of the outcome, regardless
Speaker 1: of whether or not you know, torturing one person saves
Speaker 1: a thousand, its torture is wrong. Slavery is wrong. And
Speaker 1: from this point of view, one might say, well, look,
Speaker 1: it's wrong to let a machine decide whom to kill.
Speaker 1: Humans have to make that decision. Some people have been
Speaker 1: working hard to turn this moral view into binding international law.
Speaker 1: So my name is Mary Warem. I'm the advocacy director
Speaker 1: of the Arms division of Human Rights Watch. I also
Speaker 1: coordinate this coalition of groups called the Campaign to Stop
Speaker 1: Killer Robots, and that's a coalition of one hundred and
Speaker 1: twelve non governmental organizations in about fifty six countries that
Speaker 1: is working towards a single goal, which is to create
Speaker 1: a prohibition on fully autonomous weapons. The campaign's argument is
Speaker 1: rooted in the Geneva Conventions, a set of treaties that
Speaker 1: establish humanitarian standards for the conduct of war. There's the
Speaker 1: principle of distinction, which says that armed forces must recognize
Speaker 1: civilians and may not target them. And there's the principle
Speaker 1: of proportionality, which says that incidental civilian deaths can't be
Speaker 1: disproportionate to an attack direct military advantage. The campaign says
Speaker 1: killer robots fail these tests. First, they can't distinguish between
Speaker 1: combatants and noncombatants or tell when an enemy is surrendering. Second,
Speaker 1: they say, deciding whether civilian deaths are disproportionate inherently requires
Speaker 1: human judgment. For these reasons and others, the campaign says,
Speaker 1: fully autonomous lethal weapons should be banned. Getting an international
Speaker 1: treaty to ban fully autonomous lethal weapons might seem like
Speaker 1: a total pipe dream, except for one thing. Mary warm
Speaker 1: In her colleagues already pulled it off for another class
Speaker 1: of weapons, land mines. The signing of this historic treaty
Speaker 1: at the very end of the century is this generation's
Speaker 1: pledge to the future. The International Campaign to Ban Landmines
Speaker 1: and its founder, Jody Williams, received the Nobel Peace Prize
Speaker 1: in nineteen ninety seven for their work leading to the
Speaker 1: Ottawa Convention, which banned the use, production, sale, and stockpiling
Speaker 1: of an anti personnel mines. While one hundred and sixty
Speaker 1: four nations joined the treaty, some of the world's major
Speaker 1: military powers never signed it, including the United States, China,
Speaker 1: and Russia. Still, the treaty has worked and even influence
Speaker 1: the holdouts. So the United States did not join, but
Speaker 1: it went on to I think prioritize clearance of anti
Speaker 1: personnel land mines and remains the biggest donor to clearing
Speaker 1: landlines an unexploded ordinance around the world. And then under
Speaker 1: the Obama administration, the US committed not to use anti
Speaker 1: personnel land mines anywhere in the world other than to
Speaker 1: keep the option open for the Korean peninsula. So slowly,
Speaker 1: over time countries do I think come in line. One
Speaker 1: major difference between banning land mines and banning fully autonomous
Speaker 1: lethal weapons is, well, it's pretty clear what a land
Speaker 1: mine is, but a fully autonomous lethal weapon that's not
Speaker 1: quite as obvious. Six years of discussion at the United
Speaker 1: Nations have yet to produce a crisp definition. Trying to
Speaker 1: define autonomy is also a very challenging task, and this
Speaker 1: is why we focus on the need for meaningful human control.
Speaker 1: So what exactly is meaningful human control? The ability for
Speaker 1: the human operator and the weapon system to communicate the
Speaker 1: ability for the human to intervene in the detection, selection
Speaker 1: and engagement of targets if necessary to cancel the operation.
Speaker 1: Not surprisingly, international talks about the proposed ban are complicated.
Speaker 1: I will say that a majority of the countries who
Speaker 1: have been talking about killer robots have called for illegally
Speaker 1: binding instruments and international treaty. You've got the countries who
Speaker 1: want to be helpful, like France who was proposing working groups,
Speaker 1: Germany who's proposed political declarations on the importance of human control.
Speaker 1: There's a lot of proposals, I think from Australia about
Speaker 1: legal reviews of weapons. Those efforts are being rebuffed by
Speaker 1: a smaller handful of what we call militarily powerful countries
Speaker 1: who don't want to see new international law. The United
Speaker 1: States and Russia have probably been amongst the most problematic
Speaker 1: on dismissing the calls for any form of regulation. As
Speaker 1: with the landlines, Mary Wareham sees a path forward even
Speaker 1: if the major military powers don't join at first. We
Speaker 1: cannot stop every potential use. What we want to do, though,
Speaker 1: is stigmatized, so that everybody understands that even if you
Speaker 1: could do it, it's not right and you shouldn't. Part
Speaker 1: of the campaign strategy is to get other groups on board,
Speaker 1: and they're making some progress. I think a big move
Speaker 1: in our favor came in November when the United Nations
Speaker 1: Secretary General Antonio Guterres, he made a speech in which
Speaker 1: he called for them to be banned under international law.
Speaker 1: Machine the power and the dispression to take human lives
Speaker 1: are politically and acceptable, are morally impartment and should be
Speaker 1: banned by international law. Artificial intelligence researchers have also been
Speaker 1: expressing concern. Since twenty fifteen, more than forty five hundred
Speaker 1: AI and robotics researchers have signed an open letter calling
Speaker 1: for a ban on offensive autonomous weapons beyond meaningful human control.
Speaker 1: The signers included Elon Musk, Stephen Hawking, and Demis Assabas,
Speaker 1: the CEO of Google's Deep Mind. An excerpt from the
Speaker 1: letter quote, if any major military power pushes ahead with
Speaker 1: AI weapon development, a global arms race is virtually inevitable,
Speaker 1: and the endpoint of this technological trajectory is obvious. Autonomous
Speaker 1: weapons will become the Khalishnikov's Tomorrow Chapter ten. To ban
Speaker 1: or not to ban? Not Everyone, however, favors the idea
Speaker 1: of an international treaty banning all lethal autonomous weapons. In fact,
Speaker 1: everyone else I spoke to for this episode, Ron Arkin,
Speaker 1: Missy Cummings, Paul Shari, and Ash Carter oppose it, interestingly,
Speaker 1: though each had a different reason and a different alternative solution.
Speaker 1: Robo ethicist Ron Arkin thinks we'd be missing a chance
Speaker 1: to make wars safer. Technology can, must, and should be
Speaker 1: used to reduce noncombatant casualties. And if it's not going
Speaker 1: to be this, you tell me what you are going
Speaker 1: to do to address that horrible problem that exists in
Speaker 1: the world right now, with all these innocence being slaughtered
Speaker 1: in the battlespace. Something needs to be done, and to me,
Speaker 1: this is one possible way. Paul Shari thinks a comprehensive
Speaker 1: ban is just not practical. Instead, he thinks we should
Speaker 1: focus on banning lethal autonomous weapons that specifically target people.
Speaker 1: That is, anti personnel weapons. In fact, the Landmine Treaty
Speaker 1: bans anti personnel land mines, but not say, anti tank
Speaker 1: land mines. One of the challenging things about anti personnel
Speaker 1: weapons is that you can't stop being a person if
Speaker 1: you wan't avoid being targeted. So if you have a
Speaker 1: weapon that's targeting tanks, you can come out of a
Speaker 1: tank and run away. I mean, that's a good way
Speaker 1: to effectively surrender and render yourself life of combat. If
Speaker 1: it's even targeting, say handheld weapons. You could set down
Speaker 1: your weapon and run away from it. So do you
Speaker 1: think that'd be practical to actually get either a treaty
Speaker 1: or at least an understanding that countries should forswear anti
Speaker 1: personnel lethal autonomous weapons. I think it's easier for me
Speaker 1: to envision how you might get to actual restraint. You
Speaker 1: need to make sure that the weapon that countries are
Speaker 1: giving up it's not so valuable that they can't still
Speaker 1: defeat those you might be willing to cheat. And I
Speaker 1: think it's really an open question how valuable autonomous weapons are.
Speaker 1: But my suspicion is that they are not as valuable
Speaker 1: or necessary in an anti personnel context. Former fighter pilot
Speaker 1: and Duke professor Missy Cummings thinks it's just not feasible
Speaker 1: to ban lethal autonomous weapons. Look, you can't ban people
Speaker 1: developing computer code. It's not a productive conversation to start
Speaker 1: asking for bands on technology that are almost as common
Speaker 1: as the air we breathe. Right, So we are not
Speaker 1: in the world of banning nuclear technologies. And because it's
Speaker 1: a different world, we need to come up with new ideas.
Speaker 1: What we really need is that we make sure that
Speaker 1: we certify these technologies in advance. How do you actually
Speaker 1: do the test certify that the weapon does at least
Speaker 1: as well as a human. That's actually a big problem
Speaker 1: because no one on the planet, not the Department of Defense,
Speaker 1: not Google, not Uber, not any driverless car company understands
Speaker 1: how to certify autonomous technologies. So four driverless cars can
Speaker 1: come to an intersection and they will never prosecute that
Speaker 1: intersection the same way a sun angle can change the
Speaker 1: way that these things think. We need to come up
Speaker 1: with some out of the box thinking about how to
Speaker 1: test these systems to make sure that they're seeing the world.
Speaker 1: And I'm doing that in air quotes in a way
Speaker 1: that we are expecting them to see the world. And
Speaker 1: this is why we need a national agenda to understand
Speaker 1: how to do testing to get to a place that
Speaker 1: we feel comfortable with the results those you are successful
Speaker 1: and you get the Pentagon and the driverless car folks
Speaker 1: to actually do real world testing, what about rest to world?
Speaker 1: What's going to happen? So one of the problems that
Speaker 1: we see in all technology development is that the rest
Speaker 1: of the world doesn't agree with our standards. It is
Speaker 1: going to be a problem going forward, So we certainly
Speaker 1: should not circumvent testing because other countries are circumventing testing. Finally,
Speaker 1: there's former Secretary of Defense Ash Carter Back in twenty twelve,
Speaker 1: ashe was one of the few people who were thinking
Speaker 1: about the consequences of autonomous technology. At the time, he
Speaker 1: was the third ranking official in the Pentagon in charge
Speaker 1: of weapons and technology. He decided to draft a policy,
Speaker 1: which the Department of Defense adopted. It was issued as
Speaker 1: Directive three thousand point zero nine Autonomy in Weapons Systems.
Speaker 1: So I wrote this directive that said, in essence, there
Speaker 1: will always be a human being involved in the decision
Speaker 1: making when it comes to lethal force in the military
Speaker 1: of the United States of America. I'm not going to
Speaker 1: accept autonomous weapons in a literal sense because I'm the
Speaker 1: guy who has to go out the next morning after
Speaker 1: some women and children have been accidentally killed and explain
Speaker 1: it to a press conference or a foreign government or
Speaker 1: a widow. And as suppose I go out there, Eric
Speaker 1: and I say, oh, I don't know how it happened,
Speaker 1: the machine, did it? Are you going to allow your
Speaker 1: Secretary of Defense to walk out and give that kind
Speaker 1: of excuse. No way I would be crucified. I should
Speaker 1: be crucified for giving a press conference like that, And
Speaker 1: I didn't think any future Secretary Defense should ever be
Speaker 1: in that position, or allow him or herself to be
Speaker 1: in that position. That's why I wrote the directive, Because,
Speaker 1: ashe wrote the directive that currently prevents US forces from
Speaker 1: deploying fully autonomous lethal weapons, I was curious to know
Speaker 1: what he thought about an international ban. I think it's
Speaker 1: reasonable to think about a national ban, and we have
Speaker 1: and we have one. Do I think it's reasonable that
Speaker 1: I get everybody else to sign up to that. I
Speaker 1: don't because I think that people will say they'll sign
Speaker 1: up and then not do it. In general, I don't
Speaker 1: like fakeery in serious matters, and that's too easy to fake.
Speaker 1: That is the fake meaning to fake that they have
Speaker 1: forsworn those weapons, and then we find out that they haven't,
Speaker 1: and so it turns out they're doing it, and they're
Speaker 1: lying about doing it or hiding that they're doing it.
Speaker 1: We've run into that all the time. I remember the
Speaker 1: Soviet Union said it signed the Biological Weapons Convention. They
Speaker 1: ran a very large biological warfare bird. They just said
Speaker 1: they didn't all right, but take the situation. Now, what
Speaker 1: would be the harm of the US signing up to
Speaker 1: such a thing, at least building the moral approbrium around
Speaker 1: lethal autonomous weapons, because you're building something else at the
Speaker 1: same time, which it's an illusion of safety for other people.
Speaker 1: You're conspiring in a circumstance in which they are lied
Speaker 1: to about their own safety, and I feel very uncomfortable
Speaker 1: doing that. Paul Shari sums up the challenge as well.
Speaker 1: Countries are widely divergent interviews on things like a treaty,
Speaker 1: but there's also been some early agreement that at some
Speaker 1: level we need humans involved in these kinds of decisions.
Speaker 1: What's not clear is at what level is that the
Speaker 1: level of prisiople choosing every single target, people deciding at
Speaker 1: a higher level what kinds of targets are to be attacked.
Speaker 1: How far are we comfortable removing humans from these decisions.
Speaker 1: If we had all the technology in the world, what
Speaker 1: decisions would we want humans to make it more? And
Speaker 1: why what decisions in the world require uniquely human judgment
Speaker 1: and why is that? And I think if we can
Speaker 1: answer that question, will be in a much better place
Speaker 1: to crapple with the challenge of a hotness weapons going forward. Conclusion,
Speaker 1: choose your planet, So there you haven't fully autonomous lethal weapons.
Speaker 1: They might keep our soldiers safer, minimize casualties, and protect civilians,
Speaker 1: but delegating more decision making to machines might have big
Speaker 1: risks in unanticipated situations. They might make bad decisions that
Speaker 1: could spiral out of control with no Stanislav Petrov in
Speaker 1: the loop. They might even lead to flash wars. The
Speaker 1: technology might also fall into the hands of dictators and terrorists,
Speaker 1: and it might change us as well by increasing the
Speaker 1: moral buffer between us and our actions. But as war
Speaker 1: gets faster and more complex, will it really be practical
Speaker 1: to keep humans involved in decisions? Is it time to
Speaker 1: draw a line? Should we press for an international treaty
Speaker 1: to completely ban what some call killer robots? What about
Speaker 1: a limited ban or just a national ban in the US?
Speaker 1: Or would all this be naive? Would nations ever believe
Speaker 1: each other's promises. It's hard to know, but the right
Speaker 1: time to decide about fully autonomous lethal weapons is probably now,
Speaker 1: before we've gone too far down the path. The question
Speaker 1: is what can you do a lot? It turns out
Speaker 1: you don't have to be an expert, and you don't
Speaker 1: have to do it alone. When enough people get engaged,
Speaker 1: we make wise choices. Invite friends over virtually for now
Speaker 1: in person what it's safe for dinner and debate about
Speaker 1: what we should do. Or organize a conversation for a
Speaker 1: book club or a faith group or a campus event.
Speaker 1: Talk to people with firsthand experience, those who have served
Speaker 1: in the military or been refugees from war. And don't
Speaker 1: forget to email your elected representatives to ask what they think.
Speaker 1: That's how questions get on the national radar. You can
Speaker 1: find lots of resources and ideas at our website Brave
Speaker 1: New Planet dot org. It's time to choose our planet.
Speaker 1: The future is up to us. ED don't want a
Speaker 1: truly autonomous car. I don't want to come to garage
Speaker 1: and the concess. I've fallen in love with the motorcycle
Speaker 1: and I won't drive you today because I'm autonomous. Brave
Speaker 1: New Planet is a co production of the Broad Institute
Speaker 1: of MT and Harvard Pushkin Industries in the Boston Globe,
Speaker 1: with support from the Alfred P. Sloan Foundation. Our show
Speaker 1: is produced by Rebecca Lee Douglas with Mary Doo theme
Speaker 1: song composed by Ned Porter Mastering and sound designed by
Speaker 1: James Garver, fact checking by Joseph Fridman and a stitt
Speaker 1: An enchant. Special thanks to Christine Heenan and Rachel Roberts
Speaker 1: at Clarendon Communications, to Lee McGuire, Kristen Zarelli and Justine
Speaker 1: Levin Allerhans at the Broad, to Milobelle and Heather Faine
Speaker 1: at Pushkin, and to Eliah Edie Brode who made the
Speaker 1: Broad Institute possible. This is brave new planet. I'm Eric Lander.