Speaker 1: Bloomberg Audio Studios, podcasts, radio news.
Speaker 2: For a long time, I had a job that wasn't acknowledged,
Speaker 2: and my wife didn't know what I did for a
Speaker 2: living other than there was a paycheck coming in.
Speaker 3: This is Will Roper. Back in the mid twenty tens,
Speaker 3: he was in charge of a then secret office within
Speaker 3: the Defense Department called the Strategic Capabilities.
Speaker 2: Office that meant developing new systems and new strategies building
Speaker 2: the future of warfare.
Speaker 3: Ruper was fascinated by how big tech was using artificial
Speaker 3: intelligence to scan photos and identify objects and what it
Speaker 3: could do for the US military.
Speaker 2: A lot of what industry was trying to do. Being
Speaker 2: able to identify buildings and roads and objects, it seemed
Speaker 2: very traceable to battlefield analogs.
Speaker 3: Roper would go on to propose what would become a
Speaker 3: powerful piece of AI technology for the military, project Maven.
Speaker 3: But the journey to build me and bring it to
Speaker 3: the battlefield hasn't been easy. There are still big questions
Speaker 3: about whether the US military should even be using algorithms
Speaker 3: in warfare.
Speaker 1: They use this language that'll always be a human in
Speaker 1: the loop. But at the same time, the whole point
Speaker 1: of using AI is to speed up, and people talk
Speaker 1: about decisions speeding up so fast that they're faster than
Speaker 1: a human can catch up with.
Speaker 3: That was Bloomberg reporter Katrina Manson.
Speaker 1: Today.
Speaker 3: On the show, she takes us inside the promise in
Speaker 3: the Peril of AI in the US military. We go
Speaker 3: inside Project Maven, the Pentagon's flagship artificial intelligence effort. You'll
Speaker 3: also hear from the US intelligence official who oversees much
Speaker 3: of Maven, and the CTO of Scentcom, the US Central
Speaker 3: Command whose troops are actually using it in battle. From
Speaker 3: Bloomberg's Washington Bureau, This is the Big Take DC podcast.
Speaker 3: I'm Sleiah Mosen. My colleague Katrina Manson covers cyber and
Speaker 3: emerging tech with a focus on national security. In the
Speaker 3: past few years, she's been hearing a lot about how
Speaker 3: AI could be the biggest thing to transform warfare since
Speaker 3: the invention of the radio or the machine gun. But
Speaker 3: she says the US military's big effort to bring AI
Speaker 3: to the battlefield really kicked off in twenty seventeen with
Speaker 3: Project Maveam.
Speaker 1: The Defense Department now likes to say they've been using
Speaker 1: AI for sixty years, but it was really that announcement
Speaker 1: that said, we want to look at AI machine learning
Speaker 1: algorithms and we want to find battlefield applications for it.
Speaker 1: As Will Ropea, one of the people who came up
Speaker 1: with the idea, said he always saw it as a
Speaker 1: way to find a better way of doing automatic target recognition.
Speaker 3: Automatic target recognition, in this case, scanning satellite imagery and
Speaker 3: other data inputs to identify potential threats and enemy locations
Speaker 3: on a battlefield and making sure the US doesn't misidentify
Speaker 3: non threats as targets and end up deploying a weapon
Speaker 3: against a hospital instead of a weapons factory. Until this point,
Speaker 3: the military relied on people to comb through hours of
Speaker 3: video footage and satellite imagery to identify these targets. But
Speaker 3: Roper thought a computer could do this faster, maybe even better.
Speaker 2: In a real battle, there's so much to see and
Speaker 2: observe to try to get down to the handful of
Speaker 2: targets that you need to prosecute with weapons, And it
Speaker 2: was clear that processing had gotten to be so good
Speaker 2: and there was so much data available that you could
Speaker 2: build these very deep neural nets.
Speaker 3: But in twenty seventeen, the US military wasn't familiar with
Speaker 3: deep neural nets, and it definitely couldn't build them.
Speaker 2: The Internet of Things did not happen to the US
Speaker 2: military when it happened to everyone else. The US military
Speaker 2: has amazing hardware, but the connectedness between it is not.
Speaker 2: So Military members leave their lives where they're connected to
Speaker 2: almost everything. They inner military lives where they're connected almost nothing.
Speaker 2: What most people, I think expect is that aircraft and
Speaker 2: ships connect like our smart devices do, and they don't
Speaker 2: not even close.
Speaker 3: So Roper proposed the military make a major push focused
Speaker 3: on automatic target recognition. But when he got into a
Speaker 3: room with senior defense officials to pitch his idea for
Speaker 3: what would become Project Maven, he didn't exactly get the
Speaker 3: response he wanted.
Speaker 2: Well, skeptics are pretty much what filled the halls of
Speaker 2: the Pentagon. It's not a place of great creativity and innovation.
Speaker 2: And I remember still having to put together a tutorial
Speaker 2: presentation about why this would feasibly work, why this was
Speaker 2: a big trend in commercial industry, and why we needed
Speaker 2: to be on the right side of the trend.
Speaker 3: Like any government bureaucracy, the Pentagon has a lot of
Speaker 3: red tape. Its approaches, nothing like the move fast and
Speaker 3: break things mentality of Silicon Valley. That's in part for
Speaker 3: good reason. The stakes are incredibly high when it comes
Speaker 3: to anything the US government decides to back, but especially
Speaker 3: when it comes to warfare. But some of those Pentagon
Speaker 3: officials did take a chance on Roper's idea, and in
Speaker 3: April twenty seventeen, the Defense Department announced Project Mavin, a
Speaker 3: way to use AI and data for actionable intelligence. What
Speaker 3: that meant was developing machine learning algorithms and then constantly
Speaker 3: retraining those algorithms with up to date, well labeled data.
Speaker 3: But all that outdated military tech that Roper was dealing
Speaker 3: with made that hard. My colleague Katrina spoke to one
Speaker 3: of the experts who worked on testing and evaluation in
Speaker 3: the early days of MAVEN.
Speaker 1: And she said it was really not very good at all.
Speaker 1: They didn't have the data labeled. There was no sense
Speaker 1: of everything that we'd been doing. When we click online
Speaker 1: and say, yes this is a traffic light, Yes this
Speaker 1: is a cat. There wasn't that same store of images
Speaker 1: for tanks or weapons factories.
Speaker 3: So Roper and his team had people go through images
Speaker 3: and make annually label them.
Speaker 2: There were thousands of people participating and labeling data so
Speaker 2: that we could train algorithms.
Speaker 3: They also needed some expertise to develop the program. The
Speaker 3: Pentagon partnered with some commercial tech companies, ones that re
Speaker 3: leaps and bounds ahead of the military when it came
Speaker 3: to AI, but in twenty eighteen that became a problem.
Speaker 1: Google was involved in the early days of Project Mabon,
Speaker 1: and in twenty eighteen thousands of Google workers protested. The
Speaker 1: key question was is just applying that ability to recognize
Speaker 1: a cat to a tank? How much moral culpability did
Speaker 1: those workers then feel? Because what might happen next is
Speaker 1: the tank might be exploded, and if it made a mistake,
Speaker 1: a human might be killed, or the targets may eventually
Speaker 1: involve places where humans are or civilians are, And in
Speaker 1: a letter to their seniors said we do not want
Speaker 1: to work on warfare technology.
Speaker 3: It brought Project me even into the spotlight, splashed across
Speaker 3: the news quote.
Speaker 4: We believe Google should not be in the business of war. Therefore,
Speaker 4: we asked that Project mayven be canceled. That Google enforced
Speaker 4: a clear policy stating that neither Google nor its contractors
Speaker 4: will ever build warfare technology.
Speaker 3: Did the protests give people inside the military any pause?
Speaker 1: I think they did. I think the military also realized
Speaker 1: it really had to grapple with, well, what are we
Speaker 1: actually asking big tech to do? These are not people
Speaker 1: who signed up to serve the flag, and the way
Speaker 1: that they have the missions are completely different, the cultures
Speaker 1: are different. At the time, you had senior defense officials
Speaker 1: flying over to the West Coast to try and understand
Speaker 1: what this completely different culture was about. And since then
Speaker 1: the people in charge of Project Maven had to pivot
Speaker 1: and find other partners that already were partners besides Google.
Speaker 1: But I think the Pentagon went through a renewed kind
Speaker 1: of outreach to Silicon Valley.
Speaker 3: And with the help of those other tech partners and
Speaker 3: thousands of workers labeling the data, the algorithms behind project
Speaker 3: may even got more and more accurate. An Army colonel
Speaker 3: named Joseph Callahan, who runs weapons fires, started taking an
Speaker 3: interest in MAVEN. He was asked to start experimenting with
Speaker 3: it in his unit, the eighteenth Airborne Court in North Carolina.
Speaker 3: He ran live ammunition exercises using the computer algorithm to
Speaker 3: identify a target like a tank, and then having a
Speaker 3: human deploy a weapon to explode that target. Testing the
Speaker 3: tech is a crucial part of the process. The whole
Speaker 3: point of machine learning is feedback, trying out an algorithm
Speaker 3: and then telling it when it's right or wrong so
Speaker 3: it can learn and do better next time. But just
Speaker 3: because the boss wants his workers to start using a
Speaker 3: shiny new tool doesn't mean everyone will fall into line.
Speaker 3: Katrina told me about one officer named Joey Temple.
Speaker 1: The very battle hardened targeting officer served five rounds in
Speaker 1: Iraq and he was offered a demonstration of what's called
Speaker 1: Maven Smart System.
Speaker 3: The platform that houses Maven in other data streams.
Speaker 1: He said, no, I don't want a demo. I don't
Speaker 1: want to demo. And he told me that he had
Speaker 1: a sense that stuff like this doesn't work. He didn't
Speaker 1: need another tool. He was fuddled by hearing so much
Speaker 1: about AI when he'd really never worked with AI before,
Speaker 1: and he was an expert. He is an expert targeter.
Speaker 1: He's done this his entire career.
Speaker 3: Initially, Temple wasn't interested, but the next month, in August
Speaker 3: of twenty twenty one, something changed his mind. Coming up
Speaker 3: how Project Maven won over battle hardened soldiers like Joey Temple,
Speaker 3: and how it's being used on actual battlefields today. At first,
Speaker 3: senior targeting officer Joey Temple wasn't interested in using Project Maven,
Speaker 3: but that changed in August twenty twenty one.
Speaker 2: Chaotic scenes as US troops at Couble Airport strove to
Speaker 2: evacuate Americans as well as Afghan civilians.
Speaker 3: It was the evacuation of US troops from Afghanistan the
Speaker 3: military had to air. I looked one hundred and twenty
Speaker 3: thousand people out of Kabble under very dangerous conditions, and
Speaker 3: Temple was put in front of a Maven scream that
Speaker 3: mapped out threats on the ground in real time.
Speaker 1: He could see people walking around the roots of planes.
Speaker 1: For him, seeing Maven's small system operate like that was
Speaker 1: a bit of a light bulb moment.
Speaker 3: The folks in charge of Maven are relying on those
Speaker 3: sorts of light bulb moments. The military needs officers willing
Speaker 3: to use Maven to train its algorithms to get better,
Speaker 3: and to link it up to data and censors and
Speaker 3: actually make it operational in the way the military runs
Speaker 3: its battles. Otherwise it's pretty much useless.
Speaker 1: You do tend to get experiences like that of Joey
Speaker 1: Temple who really are reticent at the start. And he
Speaker 1: even told me he has people under him who say, no,
Speaker 1: I don't want to use this. I mean, he calls
Speaker 1: himself a skeptic. He is constantly going to play Devil's
Speaker 1: advocate because what's so important for him is to know
Speaker 1: is this system really reliable. That's what I found so interesting.
Speaker 1: How you start integrating what military folks like to call
Speaker 1: human machine teaming. What is that actually like from the
Speaker 1: perspective of the person on the ground who's been told here,
Speaker 1: there's a newfangled toy, use this. Those people have very
Speaker 1: very deep relationships with their fellow combatants, so that the
Speaker 1: people they go to war with they trust. That's a
Speaker 1: really fundamental idea of US warfare, of any warfare. What
Speaker 1: if you've got to trust an algorithm.
Speaker 3: The military is now asking soldiers of the eighteenth Airborne
Speaker 3: Corps to trust that algorithm. It least in experiments on
Speaker 3: their base. It's now the largest operational test bed for MAVEN.
Speaker 3: Last year, Katrina flew to the base in North Carolina
Speaker 3: to see it in action herself.
Speaker 1: I was finally given an unclassified demonstration of it, and
Speaker 1: it was incredibly revealing. To see it because it wasn't
Speaker 1: quite what I expected, and they did a mock up
Speaker 1: for me using unclassified satellite imagery, but it represented a
Speaker 1: real operation that the US had conducted several months before.
Speaker 1: And it's very much like looking at a map on
Speaker 1: your phone, only this had yellow boxes blue boxes, and
Speaker 1: the blue boxes meant don't hit this, this is not
Speaker 1: a target. We've identified this as a school or a
Speaker 1: hospital or friendly forces, and yellow boxes meant what they
Speaker 1: call points of interest. So in this case, if you
Speaker 1: weren't asking the algorithm to find ships, it would box
Speaker 1: out all the ships, and then you might be able
Speaker 1: to ask it to find a specific kind of ship,
Speaker 1: maybe a warship, and they can layer that in using
Speaker 1: lots and lots of different data FEOD sources.
Speaker 3: Not just the satellite photos and videos that MAVIN was
Speaker 3: originally trained on. Now the program incorporates and analyzes so
Speaker 3: much more data. Katrina got to hear about that from
Speaker 3: Mark Manzill. He's the director of Data and Digital Innovation
Speaker 3: at the National Geospatial Intelligence Agency, which since last year
Speaker 3: has run most of MAVEN. The agency has been working
Speaker 3: closely with troops to test field and improve MAVIN on
Speaker 3: the ground.
Speaker 5: We also use radio waves bouncing off of objects, and
Speaker 5: we also use infrared to see heat light, and we
Speaker 5: also use multi spectral remote sets in so not just
Speaker 5: the visible light spectrum, but also the invisible like spectrum.
Speaker 3: Mancell told her the way he sees it, there are
Speaker 3: two big advantages of using MAVEN and military targeting.
Speaker 5: So speed is very important, right Obviously, the faster that
Speaker 5: we can take an image, the faster that we can
Speaker 5: run an algorithm and get the results is very important.
Speaker 5: But I will say the biggest advantage would be scale.
Speaker 5: The fact that we can do it on many images,
Speaker 5: you know, as fast as we can is really where
Speaker 5: we're providing the advantage.
Speaker 3: An advantage that was just recently proven, not just in
Speaker 3: a test, but on the actual battlefield.
Speaker 6: October seven, everything changed.
Speaker 3: That's Skylar Moore, the Chief Technology Officer of US Central Command,
Speaker 3: often referred to as Scentcom. Katrina managed to get her
Speaker 3: on the phone earlier this month.
Speaker 6: We immediately shifted into high gear and much moig operational
Speaker 6: tempo than we had previously. And the reason that I
Speaker 6: think we were able to and had a pretty seamless
Speaker 6: shift of using NAVAN was because we'd done about twelve
Speaker 6: months worth of digital exercises leading up to that.
Speaker 3: More talk Toatrina that the US also used ME even
Speaker 3: to identify and narrow down dozens of targets that US
Speaker 3: air strikes then fired against Iraq and Syria.
Speaker 1: That for me is the first time that I've really
Speaker 1: heard someone talk about using AI to find something and
Speaker 1: then deliver a weapon to it that actually belonged to
Speaker 1: the enemy and cause damage. It destroyed some, there were casualties,
Speaker 1: and so that cycle is really being operationalized.
Speaker 6: We've certainly had more opportunities to target in the last
Speaker 6: sixty to ninety days. We have a lot of examples
Speaker 6: of the housies presenting those types of threats.
Speaker 1: They're finding rocket launchers in Yemen, they're finding vessels in
Speaker 1: the Red Sea, and Maven has become an intrinsic part
Speaker 1: of that decision cycle to try and find out where
Speaker 1: is there a threat, what do we want to do
Speaker 1: about it, and how can we be sure.
Speaker 3: The tech is being used? But it hasn't been perfected yet.
Speaker 3: While humans at the eighteenth Airborne Corps can correctly identify
Speaker 3: a tank about eighty four percent of the time, Maven
Speaker 3: gets it closer to sixty percent, and experts have told
Speaker 3: Katrina that that number can dip even lower, as low
Speaker 3: as thirty percent when there's snow or cloud cover in
Speaker 3: an image, or if it's searching for a new type
Speaker 3: of object.
Speaker 1: A lot of the US military imagery comes from the
Speaker 1: Middle East. When you move to somewhere like Ukraine, there's snow,
Speaker 1: there's cloud cover, there's rain, so the conditions change a
Speaker 1: lot and the algorithms stop performing far less well.
Speaker 3: These challenges are some of the things that critics point
Speaker 3: to when they express concern about the US military using
Speaker 3: this technology. Another big thing that comes up is backlash
Speaker 3: against the idea of using autonomous machines which can use
Speaker 3: AI on the battlefield at all. There's a coalition of
Speaker 3: human rights and expert groups called Stop Killer Robots that's
Speaker 3: dedicated to keeping lethal autonomous machines that can use AI
Speaker 3: out of warfare. Even UN Secretary General Antonio Guterres has
Speaker 3: called for a ban on autonomous weapons systems. Here's Katrina again.
Speaker 1: An algorithm itself can be trained using data that was
Speaker 1: incorrect or incomplete. It can also be targeted by adversary attacks.
Speaker 1: The data could be changed without knowing the algorithm could
Speaker 1: be spoofed, it can be poisoned, it can lose accuracy
Speaker 1: over time. That's a very natural thing for an algorithm
Speaker 1: to do. It stops being as effective. So you have
Speaker 1: to work out at what point do you want to
Speaker 1: change retrain work with that algorithm. And then, of course,
Speaker 1: when you've got a life and death scenarios where you're
Speaker 1: deciding how much trust should be put in this algorithm,
Speaker 1: what should we do with it. It might be feasible
Speaker 1: to use an algorithm that makes mistakes if it's just
Speaker 1: going to blow up water, you know, it's trying to
Speaker 1: get a shit, but instead it gets water, a big deal.
Speaker 1: But if you're trying to get something on land and
Speaker 1: if you miss your hitting civilians or any of course,
Speaker 1: the risks are extremely high.
Speaker 3: The US military says that this is why it relies
Speaker 3: on human machine teaming. They say that the algorithm itself
Speaker 3: will never make the decision to execute an airstrike. It
Speaker 3: just identifies a target for a human commander. But Maven
Speaker 3: is making those decisions and those links to weapons faster.
Speaker 5: We want to shorten that kill chain, you know, as
Speaker 5: fast as we can.
Speaker 3: Mark Mansell again, he's the agency official who oversees the
Speaker 3: office that runs MAVEN.
Speaker 5: So instead of a human having to open up an image,
Speaker 5: scan the image, find everything on the image, and then
Speaker 5: report that out to a coimbat and commander, we want
Speaker 5: to shorten that by using the algorithms.
Speaker 1: To do that, so the combatant commander gets the choice
Speaker 1: of what to execute fies on.
Speaker 5: And that much faster.
Speaker 3: But as Katrina points out, speeding up the pace at
Speaker 3: which targets can be identified and fired on, that can
Speaker 3: be complicated.
Speaker 1: And of course there's been a huge amount of work
Speaker 1: on confirmation bias and an algorithm, and I think you
Speaker 1: have to wonder if an algorithm says this is something
Speaker 1: that we should be looking at as a target, does
Speaker 1: that make the person who's there to say yes or
Speaker 1: no to the algorithm more likely to say yes or
Speaker 1: less likely to say yes.
Speaker 3: The US military expects this kind of technology to improve,
Speaker 3: but in the meantime they're continuing to use it, and
Speaker 3: other countries around the world like Israel and Ukraine are
Speaker 3: using similar technology. That raises the question is the world
Speaker 3: ready for what that might mean?
Speaker 2: We're not ready for that. In the US military that
Speaker 2: a war of algorithms, a war of software.
Speaker 3: Keeping up is the reason Will Roper wanted to bring
Speaker 3: AI to the US military in the first place.
Speaker 2: We've always had this human advantage in the US military
Speaker 2: because we're always operating that our people are experienced and
Speaker 2: trained and they're ready to go, and we've always been
Speaker 2: able to say that as a way if we're feeling
Speaker 2: a little nervous about scenario, is that we've got the
Speaker 2: people who are ready well. With AI, it may be
Speaker 2: the first technology where you can undercut human advantage.
Speaker 3: That's a good cernedo folks of the Pentagon and in
Speaker 3: government who fear a war with say China, could put
Speaker 3: the US on the back foot. But war doesn't wait
Speaker 3: on software updates. Katrina learned that in twenty twenty two,
Speaker 3: the US helped Ukraine by using MAVEN to share so
Speaker 3: called points of interest where Russian equipment was located. Here's
Speaker 3: this stat that really stuck with me. During that process,
Speaker 3: the MAVEN smart system platform underwent more than fifty rounds
Speaker 3: of improvement in just the first ten months. Roper thinks
Speaker 3: that the people within the military are going to need
Speaker 3: to change the way they think about algorithms, because like
Speaker 3: it or not, at this point, there's no going.
Speaker 2: Back, the military is going to have to think about
Speaker 2: AI not as much like a piece of computer software,
Speaker 2: but almost like a member of the military that you
Speaker 2: train and that you trust with a certain amount of
Speaker 2: authority based on its training and pedigree. Hopefully those lessons
Speaker 2: will be learned before they're needed.
Speaker 3: Thanks for listening to The Big Take DC podcast from
Speaker 3: Bloomberg News. I'm Seleiah Mosen. This episode was produced by
Speaker 3: Julia Press and Naomi Shaven. It was fact checked by
Speaker 3: Alex Sugia. Special thanks to Katrina Manson, Margaret Sutherland, Vicky Vergalina,
Speaker 3: and Rebecca Shassen. Ben O'Brien is our mix engineer. Our
Speaker 3: story editors are Caitlin Kenney, Wendy Benjaminson, and Michael Shephard.
Speaker 3: Nicole Beemsterborer is our executive producer. Sage Bauman is our
Speaker 3: head of podcasts. If you like what you heard, please subscribe, rate,
Speaker 3: and review the show. It helps other listeners find us.
Speaker 3: Thanks for tuning in. I'll be back next week.