Speaker 1: Sleep Workers is a production of I Heart Radio and
Speaker 1: Unusual Productions. I was on a patrol up in the
Speaker 1: mountains of Afghanistan, and I was doing a reconnaissance patrol
Speaker 1: with a very small ranger team, and I saw someone
Speaker 1: approaching us, young man, maybe early twenties. That's Polar, a
Speaker 1: former Army ranger, an elite special operative in the U. S. Military.
Speaker 1: There were sort of a couple of possibilities that came
Speaker 1: into my mind. One was that he was a goat hurter,
Speaker 1: just out, you know, taking his goats out to eat grass.
Speaker 1: Another was that he might be like a woodcutter that's
Speaker 1: protecting his property. And another one might be that he
Speaker 1: was someone who was had seen us. There wasn't a
Speaker 1: lot of vegetation in the area, so we were pretty exposed,
Speaker 1: and he was coming to kill us, And those all
Speaker 1: seemed like very real possibilities. PULL knew of similar situations
Speaker 1: where enemy fighters pretended to be civilians, concealing their weapons
Speaker 1: until the last minute ambush. I fairly id degree of
Speaker 1: confidence that, you know, if we got into a gunfight,
Speaker 1: we could outmatch him or three of us, but if
Speaker 1: you surprised us, he might have usially killed one of
Speaker 1: us in the process first, so I wanted to maneuver
Speaker 1: to a position where I could see him, and I
Speaker 1: found him. He was sitting on the edge of a
Speaker 1: cliff and looking out over this by valley, and he
Speaker 1: had his back to me, and I settled into a
Speaker 1: position where I could watch him very closely through my
Speaker 1: sniper rifle. It was close enough that the wind carried
Speaker 1: his voice towards me and I could hear him talking.
Speaker 1: That alarmed me because I couldn't see who was talking to.
Speaker 1: Paul didn't speak the local Afghan languages Dory or Pashtow,
Speaker 1: so he couldn't understand what the man was saying. I
Speaker 1: thought he might have had a radio and he could
Speaker 1: be reporting back to maybe a group of fighters a nearby.
Speaker 1: I was wearing that possibility versus maybe he's just you know,
Speaker 1: talking to himself or talking to his goats or something
Speaker 1: in And then eventually I heard him start singing, and
Speaker 1: it struck me that if he was singing, you know,
Speaker 1: singing out over the radio to someone in our position,
Speaker 1: it seemed like a very strange thing for a fighter
Speaker 1: to be doing. And the most like the explanation was
Speaker 1: he was just an innocent go to hurd Um had
Speaker 1: no idea that we were there, and he was singing
Speaker 1: to himself just to pass the time, and so Paul
Speaker 1: lowered his sniper rifle and he left, no provocation, no
Speaker 1: harm done to either side. But that incident really stuck
Speaker 1: with me because there was a period of time where
Speaker 1: I didn't know whether he was, you know, an innocent
Speaker 1: person or a fighter who might have been trying to
Speaker 1: kill us Um, And the actions I would take were
Speaker 1: very different in those two worlds. And now I look
Speaker 1: back when I think about autonomous weapons and I think
Speaker 1: what would a machine do? How could a machine understand
Speaker 1: the context that I did, and sort of realize it
Speaker 1: doesn't make sense that a person might be singing in
Speaker 1: a tactical way that's some strange is probably just an
Speaker 1: in person be able to do that. It's a profound question.
Speaker 1: Paul poses what happens when something like a Predator drone
Speaker 1: has as much autonomy as a self driving car and
Speaker 1: can an AI system ever understand context, which can sometimes
Speaker 1: mean the difference between life and death. In this episode
Speaker 1: will examine AI on the battlefield and the future of
Speaker 1: technology driven warfare. I'm as Velosen and this is Sleepwalkers.
Speaker 1: So care last episode we talked about the future of
Speaker 1: work and we focus on one big question, which was
Speaker 1: what can AI not do and what Paul is talking about?
Speaker 1: Identifying whether someone's a shepherd or an insurgent. Identifying targets
Speaker 1: on the battlefield seems to be one of those things, sure,
Speaker 1: But on the other hand, it's important for us to
Speaker 1: ask what is AI good at. You know, good at
Speaker 1: making predictions based on data about what might happen next,
Speaker 1: It's good at seeing patterns. So it makes me think,
Speaker 1: you know that with enough training data about battlefield interactions,
Speaker 1: it could get just as good or better than humans
Speaker 1: at this task. Yeah, and of course we don't have
Speaker 1: the counter effectual to pull story. Sadly, there's probably many
Speaker 1: stories where the army ranger doesn't wait to hear the
Speaker 1: shepherd seeing before deciding what to do and just pulls
Speaker 1: the trigger. And equally, there's probably many sad cases where
Speaker 1: it does turn out to be an insurgent and an
Speaker 1: ambush happens to the soldiers. So just from that story,
Speaker 1: it's not necessarily clear to me that humans are always
Speaker 1: or will always be better than algorithms that identifying targets.
Speaker 1: You know, I do think it raises an important question.
Speaker 1: We are increasingly comfortable outsourcing many parts of our lives
Speaker 1: to technology. You know, who are we going to fall
Speaker 1: in love with? How are we going to get to
Speaker 1: our cousin's house? You know, that's decision making. Though, in
Speaker 1: the world of war, we can also build tools to
Speaker 1: do things that people don't want to do. A lot
Speaker 1: of people have heard about Boston Dynamics. They have this
Speaker 1: extremely terrifying video of this ford legged robodog aptly named Spot,
Speaker 1: you know, that can run across bumpy ground. It can
Speaker 1: go into places that aren't safe for human beings, you know,
Speaker 1: And they've just announced this model with a claw on
Speaker 1: the back of it that can open doors. There are
Speaker 1: so many potential uses for this, from assisting military rates
Speaker 1: to sending Spot into buildings that are unsafe for humans. Yeah, like,
Speaker 1: a whole pack of Spots can actually pull a truck.
Speaker 1: There's a video of it online. Look it up. It's frightening.
Speaker 1: Teamwork is the dream work exactly? But no, like, what's
Speaker 1: going to happen when that comes to the world of
Speaker 1: war and we have packs of robodogs? Well, who knows?
Speaker 1: Right the future is unclear, and that's really the question
Speaker 1: of this episode. How will AI and robotics change the battlefield?
Speaker 1: And how good of an insurance policy is it to
Speaker 1: keep a human in the loop, to have somebody a
Speaker 1: person controlling the pack of spots. We have from Paul
Speaker 1: char at the beginning. He's now the director of Technology,
Speaker 1: g and National Security at a bipartisan think tank called
Speaker 1: the Center for a New American Security. He's recently written
Speaker 1: a book called Army of None, Autonomous Weapons and the
Speaker 1: Future of War. We're moving to a world where machines
Speaker 1: may be making some of the most important decisions on
Speaker 1: the battlefield about who lives and dies. And as Paul's
Speaker 1: experienced in Afghanistan taught him, even elite military training isn't
Speaker 1: always enough to tell who is and isn't likely to
Speaker 1: be a threat. So understandably there's a great worry in
Speaker 1: handing over target identification to a computer, especially as the
Speaker 1: stakes are life and death. And as Paul tells us,
Speaker 1: autonomous weapons are already being produced and sold around the world.
Speaker 1: The best example today of fully autonomous weapon is a
Speaker 1: drone built by Israel called the Harpy Drown and he's
Speaker 1: been sold to a number of countries Turkey, India, China,
Speaker 1: South Korea. It's an anti radar webbon that is lying
Speaker 1: a switch pattern in the sky looking for enemy radars,
Speaker 1: and when it finds one, it can then attack the
Speaker 1: radar within any further human permission. And that processes the
Speaker 1: line to an autonomous weapon that's able to go find
Speaker 1: targets and then attack them all by itself. These are
Speaker 1: not the autonomous killer robots of science fiction, setting their
Speaker 1: own goals and killing it will. We still send them
Speaker 1: into battle and tell them what to look for, but
Speaker 1: we no longer control exactly what they do when they
Speaker 1: get there. So by analogy, you might think about a
Speaker 1: self driving car. Um, there are really degrees of how
Speaker 1: much a car could be self driving. You could have
Speaker 1: some like the Tesla autopilot today, where there's a steering
Speaker 1: wheel and the human could intervene and grab control of
Speaker 1: the vehicle. UM, you might have some like Google self
Speaker 1: driving car that is no steering wheel and a person
Speaker 1: just merely a passenger. But even in a totally self
Speaker 1: driving car, the human is still choosing the destination. You're
Speaker 1: not getting in your car just in car, you know,
Speaker 1: take me where you want to go. So it's some
Speaker 1: level humans are always involved, and it's going to be
Speaker 1: the case in warfare. Um. The question is, you know,
Speaker 1: when do we cross the threshold where the humans have
Speaker 1: transfer control of some important and meaningful decisions two machines,
Speaker 1: and then what are the legal or ethical complications of that.
Speaker 1: It is a hugely important question. In the aftermath of
Speaker 1: the Second World War, a hundred and ninety six countries
Speaker 1: signed up to the Geneva Convention setting standards for behavior
Speaker 1: in battle, But how do we enforce those standards if
Speaker 1: the combatants are machines not people. Beyond having a human
Speaker 1: in the loop, one important piece of the puzzle is
Speaker 1: a healthy testing and review process, a clear understanding of
Speaker 1: how a weapon works and the decisions it will make.
Speaker 1: But according to Richard Danzig, a former secretary of the Navy,
Speaker 1: that's easier said than done. One of the things that
Speaker 1: concerns me is that the technologies are frequently highly classified.
Speaker 1: So we're for self driving cars, we typically require that
Speaker 1: there be millions of miles driven, and we insist that
Speaker 1: external regulators review them for safety. In the military context,
Speaker 1: we don't have millions of miles of experience before combat
Speaker 1: and we don't typically have any kind of third party
Speaker 1: review that says, wait a minute, here, the risks associated
Speaker 1: with this system spinning out of control. We ought to
Speaker 1: be using teams to say, hey, what could go wrong
Speaker 1: if an adversary wants to attack these and subvert them,
Speaker 1: or when they interact with other systems. This kind of
Speaker 1: war gaming is valuable, but the best laid plans and
Speaker 1: standards can crumble in the face of existential threats, real
Speaker 1: or perceived. You only need to think about Hiroshima and
Speaker 1: Nagasaki to understand how quickly restraint can give way to
Speaker 1: the desire for victory his pool Again, as countries feel
Speaker 1: that their national survival might be at stake, they're gonna
Speaker 1: be willing to take more risk put up more experimental weapons.
Speaker 1: The use of poison gas in World War One is
Speaker 1: I think a terrible example of this happening and practice.
Speaker 1: Germany was in a panic to find some kind of
Speaker 1: wonder weapon that might break that stalemate. So a desire
Speaker 1: to get an upper hand will clearly Historically we've seen
Speaker 1: lead militarios to take risks and deploy more experimental technology.
Speaker 1: This is one of the scariest things about war to
Speaker 1: me Kara, particularly war involving new technology, the potential for
Speaker 1: misunderstanding and unintended consequences. Paul mentions the almost accidental use
Speaker 1: of poison gas in World War One, and then there
Speaker 1: was a Cuban missile crisis where we almost stumbled into
Speaker 1: a nuclear war. And all this potential for misunderstanding is
Speaker 1: compounded exponentially in the world of AI because it's not
Speaker 1: just humans who are trying to read each other and
Speaker 1: make decisions. It's algorithms sort of loose in the wild,
Speaker 1: interacting with one another. The Guardian actually had a great
Speaker 1: story about this called Frank and Algorithms, The Deadly Consequences
Speaker 1: of Unpredictable Code, and the article makes the point that
Speaker 1: the stock market flash crash of was actually caused by
Speaker 1: algorithms interacting with one another. You know, it's hard not
Speaker 1: to think that this could happen in the wild, so
Speaker 1: to speak. It's a scary thought, and it's made even
Speaker 1: scarier because there's just so much potential all around for misunderstanding.
Speaker 1: And that's something Richard Danzig is seriously concerned about. When
Speaker 1: we talked about sending a ship on a mission, policy
Speaker 1: makers by and large understand what that means and how
Speaker 1: others will perceive it if the ship comes into their waters.
Speaker 1: Whereas when we start talking about artificial intelligence, policy makers,
Speaker 1: if there are five people in a room, may readily
Speaker 1: envisioned five different things. We need the people making decisions
Speaker 1: to understand both the situations they're dealing with and also
Speaker 1: how the tools they're using actually work. And that's not
Speaker 1: easy when it comes to AI. Part of the issue
Speaker 1: is the so called black box problem. Currently, we understand
Speaker 1: the principles of how a neural network uses probabilities to
Speaker 1: reach a conclusion, but we can't interrogate the millions of
Speaker 1: micro decisions it makes along the way. This is a
Speaker 1: huge barrier to understanding weapons systems powered by AI. I
Speaker 1: wanted to know how the research is developing new military
Speaker 1: technology think about the black box problem. So I spoke
Speaker 1: with Artie Prabaka, the former head of DARPA, the Defense
Speaker 1: Advanced Research Projects Agency, and Arthie shared a story about
Speaker 1: how the black box problem plays out. Many years ago, now,
Speaker 1: there was a wonderful paper from Stanford that showed a
Speaker 1: machine system that could label images. This was a girl
Speaker 1: blowing out the candles on her birthday cake, or a
Speaker 1: construction worker doing something so fairly complex analysis of what
Speaker 1: was going on in this picture, and it would get
Speaker 1: one right, and it would get ten right, and we
Speaker 1: get a hunder right. And then there was a picture
Speaker 1: that every human being would say, that's a baby holding
Speaker 1: an electric truth brush, but the machine said it's a
Speaker 1: small boy holding a baseball bat. And you know, you
Speaker 1: just look at it and you think, what what what?
Speaker 1: What were what were you thinking? And this I think
Speaker 1: it's a great illustration of the black box nature of
Speaker 1: these learning systems because they've been trained on all this
Speaker 1: volume of data, but when you look inside to say, well,
Speaker 1: what went wrong there, you know, you just see a
Speaker 1: bunch of nodes with weights from being trained, and so
Speaker 1: they're really opaque. Offi's example is kind of cute, but
Speaker 1: think about it for a moment. The difference between a
Speaker 1: baby holding an electric toothbrush and a small boy holding
Speaker 1: a baseball bat could also be in Afghanistan, the difference
Speaker 1: between Paul Shepherd and militant, in other words, the difference
Speaker 1: between life and death. Yeah, and it's a bit daunting
Speaker 1: that we are becoming more reliant on something that we
Speaker 1: continue not to understand fully, don't you think absolutely and
Speaker 1: Henry kissing you actually wrote a piece on this for
Speaker 1: the Atlantic called how the Enlightenment Ends Big Stuff for Sure,
Speaker 1: Hetty and what's kissing You? He argued that because we're
Speaker 1: unable to interrogate the output of algorithms, as we rely
Speaker 1: on them more immorti classify the world around us, we
Speaker 1: may actually start to lose the ability to reason for ourselves.
Speaker 1: It's not inevitable though, that AI will always be opaque.
Speaker 1: The EU are actually working on this policy that decisions
Speaker 1: made by AI need to be explainable to people they affect.
Speaker 1: That may be a policy that's easier said than done.
Speaker 1: Although the so called next wave of AI is all
Speaker 1: about explainable AI, and it's actually a major initiative right
Speaker 1: now at DAPPA. Here's off the again. Explainable AI has
Speaker 1: been part of starting an entire new field of inquiry
Speaker 1: and artificial intelligence to couple that kind of statistical power
Speaker 1: that machine learning systems have with systems that explain how
Speaker 1: they got the results that they got. In order for
Speaker 1: us human users to be able to know when to
Speaker 1: trust those machine learning systems and when not to trust them.
Speaker 1: What authe is describing would be a huge breakthrough in AI.
Speaker 1: Understanding how neural networks make their decisions would allow us
Speaker 1: to honess the power the technology much more safely, and
Speaker 1: not just on the battlefield. When we come back, we
Speaker 1: look at how much of the technology we take for
Speaker 1: granted in our everyday lives actually originated in the military.
Speaker 1: So DARPA, the defense agency with an annual budget of
Speaker 1: three point five billion dollars, its motto is to cast
Speaker 1: a javelin into the infinite spaces of the future. What
Speaker 1: you may not know is how much of the technology
Speaker 1: we all use every day came right out of Darper.
Speaker 1: I think about this every time I use my smartphone
Speaker 1: because that's a beautiful, seamless integration of a whole host
Speaker 1: of technologies that were sparked many many years ago by Darper.
Speaker 1: So the chip in your cell phone that talks to
Speaker 1: the cell tower is based on materials and electronics technology
Speaker 1: that was developed for communications, and radar systems that up
Speaker 1: that knows when you've rotated your phone is MEM's technology
Speaker 1: that had huge early support from DARPA. But also Series
Speaker 1: or other intelligent agents are based on the artificial intelligence
Speaker 1: research that was done. But what fueled this wave of
Speaker 1: incredible innovation at DAPPA. Well, actually, war DARPA is a
Speaker 1: very American concept. In nineteen fifty seven, the Soviet Union
Speaker 1: put the first artificial satellite on orbit. That was Sputnik.
Speaker 1: There was a lot of excitement because human beings had
Speaker 1: never done that before, but of course also quite a
Speaker 1: shock for the United States at the height of the
Speaker 1: CULD WARP. Sputnik was a reminder that in addition to
Speaker 1: working on the problems that you knew you had, you
Speaker 1: also needed to have people who came to work every
Speaker 1: day to think about those kinds of technological surprises. And
Speaker 1: so DARPA was started as a reaction to that technological
Speaker 1: surprise of Sputnik. It's mission for sixty years has been
Speaker 1: to create those kinds of technological surprises, and its history
Speaker 1: is one in which it's accomplished that mission. DARPA is
Speaker 1: known as the place that made the early stages of
Speaker 1: each revolution and artificial intelligence, and of course must memorably
Speaker 1: for starting the ARPA net and writing the protocols that
Speaker 1: became the Internet that we have today. Think about that
Speaker 1: the technology that forms the architecture of our daily lives.
Speaker 1: In the twenty one century, the Internet was created by
Speaker 1: a defense agency whose mission was to outthink the Soviet Union,
Speaker 1: and in some sense all of the technologies we've looked
Speaker 1: at so far in Sleepwalkers are the outgrowth of Dapper's work.
Speaker 1: What really enabled this was DARPA's decision to allow their
Speaker 1: technology into the US private sector and to let entrepreneurs
Speaker 1: build on top of it. Absolutely as vital was that
Speaker 1: were the companies and the entrepreneurs and the investors who
Speaker 1: saw that you could turn those those raw research results
Speaker 1: into this seamless, beautiful product that we've now we all
Speaker 1: live with all the time. Characters. Amazing to think just
Speaker 1: how much a Silicon Valley really stands on the shoulders
Speaker 1: of DARPA. But even outside of what we think of
Speaker 1: as the tech world, there's plenty of examples of military
Speaker 1: technology living with us. For example, microwaves, which were an
Speaker 1: accidental byproduct of radar technology, and then as good old Rumba,
Speaker 1: which was originally developed a mind sweeping technology and is
Speaker 1: still a mind sweeping technology in my house. Um. Yeah,
Speaker 1: you know. There's this concept of dual use technology, which
Speaker 1: we've talked about a few times. It's the idea that
Speaker 1: technology developed for the military can have civilian applications and
Speaker 1: vice versa. We talked about this with blink identity and
Speaker 1: facial recognition. Do you remember last year the project may
Speaker 1: even walk out? Yeah, Google, right, So that was a
Speaker 1: project for the Pentagon. Over three thousand employees protested that
Speaker 1: they didn't want to develop that technology. Google pulled the
Speaker 1: plug on the project. The problem is though that you know,
Speaker 1: you can say you're developing you know, AI and target
Speaker 1: recognition for the military. All you can say you're developing
Speaker 1: AI to recognize what's happening in images. But it's the
Speaker 1: same technology and once it gets into the wild, anyone
Speaker 1: who wants can use it. And that's actually something that
Speaker 1: Arthur you spoke about about innovation traveling from DARPA to
Speaker 1: the private sector. Yeah, and now DARPA actually has this
Speaker 1: younger sibling called the Defense Innovation Unit d i u X,
Speaker 1: whose job is actually to invest and incubate technologies from
Speaker 1: the private sector that could be helpful for defense. So
Speaker 1: this is basically the bridge from Silicon Valley to d C,
Speaker 1: which is taking things from the consumer space and applying
Speaker 1: them for military use. And one of these technologies is
Speaker 1: called Halo. They're basically headphones that electrify your brain in
Speaker 1: order to stimulate growth. That's right. So I went to
Speaker 1: Connecticut to test them out with former Navy Seal John Wilson,
Speaker 1: OH a seal for twelve years. I've served in Iraq
Speaker 1: multiple times, Afghanistan, I went to Mogadishue, and then South America.
Speaker 1: As you can imagine, drug warfare is still a really
Speaker 1: big issue, so we have military units down there to
Speaker 1: combat the cartels. We were gone three hundred days out
Speaker 1: of the year training and then we were deployed from
Speaker 1: once on end. So that's what we lived, breathed day
Speaker 1: in and day out. We weren't going home at night.
Speaker 1: These days, John is back with his family and he's
Speaker 1: especially interested in how new technology can help seals past
Speaker 1: and present. One such technology is the Halo headset, which
Speaker 1: d I u X invested in. The headset uses an
Speaker 1: electric current to prime the brain for so called neuroplasticity,
Speaker 1: in other words, the ability to learn and learn quickly.
Speaker 1: For us, I've recognized and when we do a pistol draw,
Speaker 1: that movement is a repetitive movement that we've done thousands
Speaker 1: and thousands and thousands of times over and over again.
Speaker 1: And what this does is it primes the brain to
Speaker 1: learn those repetitive movements faster? And have you genuinely knows
Speaker 1: the difference. When I first came across this, I've got
Speaker 1: a bunch of seals together and went out to the range.
Speaker 1: We neuro primed and we started shooting. So we got
Speaker 1: the baseline. Did this for a month, looked at our
Speaker 1: scores and our scores were light years better. And light
Speaker 1: years I mean milliseconds, right, But milliseconds on the battlefield
Speaker 1: equates the life or death. I may be as far
Speaker 1: away from being a military person as anyone could be,
Speaker 1: but John has kindly agreed to lead me through a
Speaker 1: Navy seal workout. So what we have here it looks
Speaker 1: like a beats headset right with some strange noduals. Yeah,
Speaker 1: so what we have on the top has mentioned were
Speaker 1: these little nods. What those do? Or those are going
Speaker 1: to send a current into the cortex or frontal cortex
Speaker 1: into your brain? Is that safe? That's a great question.
Speaker 1: I'm gonna bypass that question. It's safe. Turns out it
Speaker 1: is safe. It's been tested by Dapper and others. So
Speaker 1: I had to put it on and John agreed to
Speaker 1: help me use the headset to neuro prime before putting
Speaker 1: me through my pace is all right, let's let's crank
Speaker 1: it up so you can. So it's probably a good
Speaker 1: I don't know if you can feel a difference there,
Speaker 1: but we've got twenty minutes of neuro priming. It feels
Speaker 1: like something pinching in my head a bit. Yeah, it's not.
Speaker 1: It's not a comfortable feeling, right, yeah, and you imagine, no,
Speaker 1: you never do. But it's worth it, right, What is
Speaker 1: that uncomfortable sensation? What's happening that? That's electricity? Yea, so
Speaker 1: it's it's going into your brain right now, and it's
Speaker 1: getting your brain into a state of neuroplasicity. Hyper Learning,
Speaker 1: essentially is what that state allows you to do, and
Speaker 1: it just allows you to learn faster and learn more information.
Speaker 1: After a warm up, which was in fact a lot
Speaker 1: more intense than my normal workout, two three, come on up,
Speaker 1: drive up and hold right still the warm up, Yes,
Speaker 1: it's still the warm up, it was time to take
Speaker 1: the headset off and start the real thing. Or so
Speaker 1: I thought, all right, it's already to keep going. You
Speaker 1: can do twenty reps. So oh my god, oh god, ah,
Speaker 1: those people's not do what hell weeks like I thinks
Speaker 1: be worse than this. Yeah, why do you ask? John? Thankfully,
Speaker 1: the workout came to an end and without any injury. Though,
Speaker 1: to be honest, I couldn't tell if the neuropriming had
Speaker 1: worked for me. Because halo enhances the brain's ability to learn,
Speaker 1: Studies show best results when it's used over time. In
Speaker 1: other words, if I wore the headset before every workout,
Speaker 1: I might start to notice the difference in how quickly
Speaker 1: I performed. But somehow I trusted John talking about the
Speaker 1: draw time for his weapons. So the battle feels Afghanistan
Speaker 1: rugs Syria. It's my happy place. Yeah, who can understand that?
Speaker 1: It's my happy place because around people that I know
Speaker 1: would do anything for me that I love me and
Speaker 1: I love them for John. Being on the battlefield wasn't
Speaker 1: the hardest part leaving. It was transition and is not
Speaker 1: an easy thing. It's the hardest thing I've ever done.
Speaker 1: When we transition, we do it by ourselves. We're trying
Speaker 1: to solve a complex problem, which we love to do.
Speaker 1: But normally when that takes place, you have your team
Speaker 1: around you and you're going to figure it out because
Speaker 1: you know you'll never let the person the left and
Speaker 1: right of you down. When you're trying to do this
Speaker 1: by yourself, you have nobody to talk to. When it
Speaker 1: starts getting heart and you go dark is what we
Speaker 1: call us. So we go into our shell. We we
Speaker 1: kind of ostracize ourselves from society and and that's when
Speaker 1: bad stuff starts happening. John recools a recent narrow escape
Speaker 1: for one of his Sealed comrades. He was a Seal
Speaker 1: Team six guy and um ended up going through divorce.
Speaker 1: He had a newborn that he had to stay home
Speaker 1: and take care of, so he went to a really
Speaker 1: dark place. He just called me in unbeknownst to me,
Speaker 1: it just put his son to bed and he was
Speaker 1: sitting in the car with a pistol. I didn't know that.
Speaker 1: I'm just picked up the phone and asked him how
Speaker 1: he was doing, and he said he's doing okay, but
Speaker 1: he needs some help. I said, we got your brother,
Speaker 1: That's all I said. And that was enough for him
Speaker 1: to put that gun away, go back inside and take
Speaker 1: care of that little guy. Just me saying we got you.
Speaker 1: That camaraderie saved John's friends life, but returning veterans need
Speaker 1: something more than community. They need a purpose, a mission,
Speaker 1: and that can be hard to find in civilian life.
Speaker 1: You don't know you're fit, you don't know your rule
Speaker 1: in the family, in your tribe, you don't know your
Speaker 1: rule in society, and you're just trying to get by
Speaker 1: to put food on the table. And there's there's a
Speaker 1: void there now. And according to John, that's where Halo
Speaker 1: comes in. Just because we're steals doesn't mean we all
Speaker 1: want to end up doing security work. There's a lot
Speaker 1: of people that wanted to maybe go in to finance
Speaker 1: or be a lawyer. Halo allows us to succeed and
Speaker 1: accelerate at that process. So if it's people want to
Speaker 1: go back to school putting a Halo headset on before
Speaker 1: you study, people maybe wanting to get a job that
Speaker 1: requires multiple languages, you can throw on the halo and
Speaker 1: pick up those language at an accelerated pace. Do you
Speaker 1: think it's more powerful as something to believe in, like
Speaker 1: if I put this headset on, like achieve my goals?
Speaker 1: Is more powerful? Is a technological solution or is it
Speaker 1: somewhere in between those two things? I think it's probably
Speaker 1: both right. So our Strength and Conditioning coach and the
Speaker 1: steel teams. He had a bottle p E is what
Speaker 1: he wrote on it, which stood for placebo effect. It
Speaker 1: was just water, but guys would come over and saying
Speaker 1: their herd and he would just spare a little bit
Speaker 1: of this water on them and they would every single
Speaker 1: time would walk away like, oh I feel better, thanks, coach.
Speaker 1: My point is is the research their supports that this
Speaker 1: actually works. But who gives a ship even if it didn't,
Speaker 1: because people are going to believe in themselves and that's part.
Speaker 1: That's the bottle of my opinion. If it takes a
Speaker 1: headset to get there, then great, but we know that
Speaker 1: this headset works is going to help accelerate you atue
Speaker 1: in that goal. We've talked about dual use in terms
Speaker 1: of military technology that enters the civilian world and vice versa,
Speaker 1: and Halo is just that a consumer product that is
Speaker 1: also used by the armed forces, but it has a
Speaker 1: much more profound dual use. It can save soldiers lives twice,
Speaker 1: the first time on the battlefield where shaving milliseconds of
Speaker 1: reaction time coming the difference between life and death, and
Speaker 1: the second time when they return home. Haylo can help
Speaker 1: them develop new skills and perhaps even more importantly, give
Speaker 1: them the hope they need to keep going. When we
Speaker 1: come back, we return to Darper and how to ensure
Speaker 1: that we design new military technologies with worst case scenarios
Speaker 1: top of mind. One of the central contentions of Sleepwalkers
Speaker 1: is that our creations reflect us, and knowing this, we
Speaker 1: need to be deliberate about how we tell them to
Speaker 1: behave We're talking episode three of this series, The Watchman
Speaker 1: about automation bias, the very human habit of treating the
Speaker 1: output of computers as infallible even while ignoring the inputs
Speaker 1: that we've given them. And recognizing this, Arthur made it
Speaker 1: a central tenet of her tenure at Dapper to argue
Speaker 1: that technology is not inevitable. There's a tendency to give
Speaker 1: the active role to the technology. It's what the AI
Speaker 1: will do to us. I want to keep bringing us
Speaker 1: back to the fact that these technologies are our creations.
Speaker 1: They're built by human beings. We have this enormous privilege
Speaker 1: that we get to work on the powerful technologies that
Speaker 1: can shape the progress of our societies. That privilege comes
Speaker 1: with a responsibility to ask what could possibly go wrong?
Speaker 1: What could possibly go wrong? It's a legitimate question, but
Speaker 1: there's also a reason it's become a meme. It's notoriously
Speaker 1: hard to answer. This is especially true in times of war,
Speaker 1: when new technologies are often rushed into action without being
Speaker 1: fully understood. Here's poor Shari again. Old War one is
Speaker 1: a a wonderful, a terrible, um example of what can
Speaker 1: happen when we see new technologies change warfare in ways
Speaker 1: that policymakers were not prepared for. You know, the Industrial
Speaker 1: Revolution brought not just you know, locomotives, but also um,
Speaker 1: you know, cars, tanks, airplanes, machine guns that then were
Speaker 1: used to industrialized warfare in a totally new way that
Speaker 1: dramatically change this scale and the speed of killing that
Speaker 1: was possible. The Gatling gun. People still had to crank
Speaker 1: the gun, but then it automated the process of loading
Speaker 1: and firing bullets. We began this episode talking about the
Speaker 1: new dangers posed by automated weapons. Well, the Gatling gun
Speaker 1: was actually one of the world's first, and as Paul
Speaker 1: told us, its invention had a ripple effect that its
Speaker 1: inventor could not have foreseen. The inventor, Richard Gatling, did
Speaker 1: this to save lives. He was looking at um people
Speaker 1: were coming back wounded and killed from the American Civil War.
Speaker 1: He wanted to build a machine that could reduce the
Speaker 1: number of soldiers that were needed on the battlefield as
Speaker 1: a way to save lives. And that sounds like, you know,
Speaker 1: a very well meaning idea and in practice, um as
Speaker 1: the Gatlin gun involved into the machine gun. In World
Speaker 1: War One, we saw a scale of killing that was
Speaker 1: just unprecedented and in a whole a whole generation of
Speaker 1: European man wiped out a battlefield. And so I think
Speaker 1: it's it's an important cautionary tale for our ability to
Speaker 1: predict how this technology will be used. The name of
Speaker 1: this podcast, Sleepwalkers, is borrowed from a book called The Sleepwalkers,
Speaker 1: How Europe Went to War in nineteen fourteen, written by
Speaker 1: the historian Christopher Clark. And one of the big questions
Speaker 1: I've been asking is, are we had a moment like
Speaker 1: we were on the eve of World War One when
Speaker 1: we haven't fully understood the implications of our new technology.
Speaker 1: I asked Richard Danzig, the former Navy secretary, about the anels.
Speaker 1: There is an analogy from World War One. European military
Speaker 1: leaders developed mobilization plans to increase their own capabilities in
Speaker 1: the event of an attack, and they underestimated the degree
Speaker 1: to which that created rigidities and interactions, so that in
Speaker 1: the end, the railroad timetables generated a war that perhaps
Speaker 1: no one intended to be engaged in. People think that
Speaker 1: they're driving the card, and in reality, the horses of
Speaker 1: technology are frequently pulling us in directions that we don't
Speaker 1: anticipate and can't control. So are we better place now
Speaker 1: to understand the implications of AI and new technology for
Speaker 1: global conflict than Europe was in ur I don't believe
Speaker 1: our understanding of AI is greater than their understanding At
Speaker 1: that point, we will make these mistakes too. I cannot
Speaker 1: ask to make their significance or their frequency, but I'm
Speaker 1: rather confident we will lose control, that we will make
Speaker 1: mistakes of that kind and cause unintended consequences. So to me,
Speaker 1: the interesting question is not can I predict their frequency?
Speaker 1: The interesting question is what can I do in advance
Speaker 1: if I recognize that it's one of them. Is well
Speaker 1: represented by the darkast Safe Chaine project that government agency
Speaker 1: is saying if people edit genes, but it turns out
Speaker 1: they escape into the environment and proliferate. How do we
Speaker 1: program them to begin with so that we can shut
Speaker 1: them off? When it's so difficult to predict how new
Speaker 1: technologies will be used and misused, it's hugely important that
Speaker 1: we build precautions while they're still being researched and developed.
Speaker 1: Is difficult, but we have to do our best to
Speaker 1: anticipate the future dangers of a technology long before potential
Speaker 1: deployment on the battlefield. Thankfully, that philosophy governed Arthur Prabaka's
Speaker 1: work at DAPPER. What we developed was a way of
Speaker 1: grappling with the ethical implications of these technologies. It started
Speaker 1: by being open with ourselves, not just about our hopes
Speaker 1: for the technology, but also our fears, and looking at
Speaker 1: each other in the eye and saying, here's what we
Speaker 1: think really is possible, but also here's what could really
Speaker 1: go wrong. Were there any specific programs that you were
Speaker 1: tempted by as a technologist, but in the end you
Speaker 1: had to kill because they didn't meet your ethical standards.
Speaker 1: I don't really have anything to say about that. The
Speaker 1: answer is yes, but I can't give you an example
Speaker 1: because it was classified. Karen, we've talked about the design
Speaker 1: phase and thinking from r and d onwards about making
Speaker 1: new weapons system safe. But it doesn't always work out
Speaker 1: that way. Yeah, the genie does have a habit of
Speaker 1: getting out of the bottle. Um. We've talked about dual
Speaker 1: use before. Even seemingly benign technologies can be hugely destructive.
Speaker 1: The one that blows my mind is the sort of
Speaker 1: Arthur Galston, who was a plant biologist who discovered while
Speaker 1: he was a graduate student, this compound that helps soybeans
Speaker 1: flower faster. He also learned that if this compound were
Speaker 1: applied in excess, that it would cause the plant to
Speaker 1: shed its leaves. And when Galston discovered this defoliant effect,
Speaker 1: that's what was abused by biological warfare scientists who would
Speaker 1: then go on to develop Agent orange. I just got
Speaker 1: back from a trip to Vietnam actually, where the effects
Speaker 1: of Asian orange is still being felt. It's actually a
Speaker 1: gene toxin which causes deformities through the generations. So that
Speaker 1: is a truly horrific one, Kara, And it makes us
Speaker 1: think it wasn't those chemists who were releasing agent orange
Speaker 1: over Vietnam, it was the US military. So the idea
Speaker 1: of the person who creates the technology gets to control
Speaker 1: what happens to it is simply not the case, and
Speaker 1: so we need to move forward with the assumption that
Speaker 1: AI weapons will leave the laboratory and exists in the world.
Speaker 1: And the central question is how helpful it to have
Speaker 1: a human in the loop. Well, according to Richard Danzigg,
Speaker 1: humans are of increasingly limited utility. I think there are
Speaker 1: circumstances where human control is useful, but I don't think
Speaker 1: that's the most useful approach. And the reason for that
Speaker 1: is because the power is in the machine. So many
Speaker 1: decisions that we care about are made at extraordinarily high speed,
Speaker 1: and it just isn't time for the humans to assimilate
Speaker 1: them and make correct judgments. Even the president declaring or
Speaker 1: not nuclear war and responding might have fifteen minutes to
Speaker 1: make a judgment. In other words, the whole idea of
Speaker 1: a human in the loop making the final cool is
Speaker 1: something of an illusion. At the very least, it relies
Speaker 1: on us making wise decisions at lightning speed and under pressure.
Speaker 1: And there's another problem. All of the information people like
Speaker 1: the president used to make decisions has already been filtered
Speaker 1: through several computer systems. So when the president reviews information
Speaker 1: to make a tactical choice. He or she is already
Speaker 1: relying on automation. He's extraordinarily dependent on what the machines
Speaker 1: are telling him, what the sensors are interpreted to him,
Speaker 1: and what the algorithms say the trajectories of missiles that
Speaker 1: have been launched. So realistically, he's on the cart being
Speaker 1: pulled by the horses of these technologies. UM. If that's
Speaker 1: true for the president, think what it's like for the
Speaker 1: person who's a sergeant in the field manning Patriot Missile Battalion. UM.
Speaker 1: And it shows incoming missiles he has seconds or at
Speaker 1: best minutes to respond and has to make decisions. We
Speaker 1: know how fallible we are as decision makers, and we
Speaker 1: know how dependent we already are on computer systems to
Speaker 1: guide our decisions. So what can we do to prevent
Speaker 1: ourselves from stumbling into a conflict that no one wants?
Speaker 1: But I think we need to recognize that science is
Speaker 1: now diffused around the globe, and we need a common
Speaker 1: kind of understanding about how to reduce these risks, And
Speaker 1: then we need some joint planning for the contingency that
Speaker 1: these do escape. What do we do if a newly
Speaker 1: engineered genetic system gets out there into the wild. Well,
Speaker 1: that's not just a problem for the Chinese. If it
Speaker 1: happens to happen in China, technology spreads, it gets modified, copied,
Speaker 1: and hacked, and once something is out of the lab
Speaker 1: is anyone's guess as to what happens. And countries are
Speaker 1: slowly trying to establish standards for AI. But as Paul
Speaker 1: Shari argues, creating a global framework governing AI in warfare
Speaker 1: is a tool order. It's a very hard area to
Speaker 1: think about. How do we mitigate that risk because countries
Speaker 1: are not going to talk about the things that they're
Speaker 1: doing in cyberspace and the fine angel sector. They've installed
Speaker 1: things like circuit breakers that would take stocks offline if
Speaker 1: the price moves too quickly. Well, there's no referee to
Speaker 1: call time out in warfare. So if we're going to
Speaker 1: manage those risks in the military space, does have to
Speaker 1: be circuit breakers, uh if you will that people build
Speaker 1: into our own weapons systems limits on them, ways to
Speaker 1: interject human control to maintain control if things begin to
Speaker 1: move in unexpected ways. And it's worth acknowledging really upfront
Speaker 1: that there's a trade off there that every time that
Speaker 1: a military puts guard rails on a weapon system or
Speaker 1: inserts a human in the loop as a check. That's
Speaker 1: potentially slowing down the effectiveness of their weapon. And there's
Speaker 1: a risk that they are going to be afraid that
Speaker 1: an adversary might not do that and might get an
Speaker 1: edge on them. And that dynamic is really the crux
Speaker 1: of the problem here. It's hard to get to a
Speaker 1: place where countries um trust each other enough to engage
Speaker 1: in mute all frustrain. But we may not have a choice.
Speaker 1: Up until now, our defense policy has been based on
Speaker 1: the assumption of technical superiority, and as our argues, we
Speaker 1: can no longer rely on that. You have a model
Speaker 1: that is based on owning all the technology and knowing
Speaker 1: that no one else can have access to it for
Speaker 1: two or three decades, and today those assumptions simply don't hold.
Speaker 1: We are not the only people who can innovate right now.
Speaker 1: The history of new technology and warfare is frankly disturbing.
Speaker 1: When we create new weapons, we tend to use them.
Speaker 1: We've talked about the atomic bomb in Japan, and about
Speaker 1: poisonous gas in World War One, and even about the
Speaker 1: Gatling gun, one of the world's first automated weapons designed
Speaker 1: to reduce the number of combatants required to wage war.
Speaker 1: It decimated an entire generation in Europe. We haven't yet
Speaker 1: seen what happens when AI weapons systems begin to interact
Speaker 1: with one another, but chances are we will in our lifetime.
Speaker 1: So the pation in all of this can be to
Speaker 1: desperately try to hit pools on new technology, but off,
Speaker 1: y'all ues, that would be the wrong approach. Historically, we
Speaker 1: are drawn forward by the enormous potential that these technologies
Speaker 1: can enhance our lives, and at the same time we're
Speaker 1: we're repelled by the consequences that we understand could be
Speaker 1: fundamentally wrong. So I think that's the tension. But in
Speaker 1: aggregate and over time, I do think that technology has
Speaker 1: lifted us up, has advanced us. You know, when you
Speaker 1: play the parlor game of asking your friends what period
Speaker 1: in history they would rather live in. I might want
Speaker 1: to visit, but there's no other time in history I
Speaker 1: want to live in. And I think the future is
Speaker 1: going to be fraught with problems, and it's still going
Speaker 1: to be a better place than the one that we're in.
Speaker 1: It's true that technology has made so many parts of
Speaker 1: our lives easier, healthier, and safer, and it's also true
Speaker 1: that the technology we create has the potential to be
Speaker 1: ever more destructive. We've talked about dual use in this episode,
Speaker 1: and as a matter of fact, many DARPA programs have
Speaker 1: found their way into revolutionizing medicine. In the next episode,
Speaker 1: we look at some of the incredible applications of AI
Speaker 1: in the world of healthcare, from accurately predicting time of
Speaker 1: death to decoding the human genome. I'm Ozveloshin, See you
Speaker 1: next time. Sleepwalkers is a production of our Heart Radio
Speaker 1: and unusual productions. For the latest AI news, live interviews,
Speaker 1: and behind the scenes footage. Find us on Instagram, at
Speaker 1: Sleepwalker's podcast or at sleepwalkers podcast dot com. Sleepwalkers is
Speaker 1: hosted by me oz Veloshin and car hosted by me
Speaker 1: Kara Price. Were produced by Julian Weller with help from
Speaker 1: Jacopo Penzo and Taylor Chacogne. Mixing by Tristan McNeil and
Speaker 1: Julian Weller. Our story editor is Matthew Riddle. Recording assistance
Speaker 1: this episode from Chris Hambroke and Jackson Bierfeld. Sleepwalkers is
Speaker 1: executive produced by me Osvaloschen and Mangesh Hattikiller. For more
Speaker 1: podcasts from my Heart Radio, visit the I heart Radio app,
Speaker 1: Apple Podcasts, or wherever you listen to your favorite shows.