Speaker 1: Bloomberg Audio Studios, Podcasts, radio News.
Speaker 2: Hey, it's Sarah today on The Big Take. We're bringing
Speaker 2: you an episode of The Big Take DC hosted by
Speaker 2: my colleague, Senior Washington correspondent Salia Mosen. She's got a
Speaker 2: great story for you about the promise and peril of
Speaker 2: applying AI technology to the battlefield. I'll be back tomorrow.
Speaker 2: Here's Seleia.
Speaker 3: For a long time, I had a job that wasn't acknowledged,
Speaker 3: and my wife didn't know what I did. We're living
Speaker 3: other than there was a paycheck coming in.
Speaker 4: This is Will Roper. Back in the mid twenty tens.
Speaker 4: He was in charge of a then secret office within
Speaker 4: the Defense Department called the Strategic Capabilities.
Speaker 3: Office that meant developing new systems and new strategies building
Speaker 3: the future of warfare.
Speaker 4: Ruper was fascinated by how big tech was using artificial
Speaker 4: intelligence to scan photos and identify objects and what it
Speaker 4: could do for the US military.
Speaker 3: A lot of what industry was trying to do. Being
Speaker 3: able to identify buildings and roads and objects, it seemed
Speaker 3: very traceable to battlefield analogs.
Speaker 4: Roper would go on to propose what would become a
Speaker 4: powerful piece of AI technology for the military Project Maven.
Speaker 4: But the journey to build Maven and bring it to
Speaker 4: the battlefield hasn't been easy. There are still big questions
Speaker 4: about whether the US military should even be using algorithms
Speaker 4: in warfare.
Speaker 1: They use this language that will always be a human
Speaker 1: in the loop. But at the same time, the whole
Speaker 1: point of using AI is to speed up, and people
Speaker 1: talk about decisions speeding up so false that that false
Speaker 1: than a human can catch up with.
Speaker 4: That was Bloomberg reporter Katrina Manson today. On the show,
Speaker 4: she takes us inside the promise in the peril of
Speaker 4: AI in the US military. We go inside Project Maven,
Speaker 4: the Pentagon's flagship artificial intelligence effort. You'll also hear from
Speaker 4: the US intelligence official who receives much of Maven, and
Speaker 4: the CTO of Scentcom, the US Central Command whose troops
Speaker 4: are actually using it in battle. From Bloomberg's Washington Bureau.
Speaker 4: This is the Big Take DC podcast. I'm Salaiah Mosen.
Speaker 4: My colleague Katrina Manson covers cyber and emerging tech with
Speaker 4: a focus on national security. In the past few years,
Speaker 4: she's been hearing a lot about how AI could be
Speaker 4: the biggest thing to transform warfare since the invention of
Speaker 4: the radio or the machine gun. But she says the
Speaker 4: US military's big effort to bring AI to the battlefield
Speaker 4: really kicked off in twenty seventeen with Project Maven.
Speaker 1: The Defense Department now likes to say they've been using
Speaker 1: AI for sixty years, but it was really that announcement
Speaker 1: that said we want to look at AI machine learning algorithms,
Speaker 1: and we want to find battlefield applications for it. As
Speaker 1: Will rope In, one of the people who came up
Speaker 1: with the idea, said he always saw it as a
Speaker 1: way to find a better way of doing automatic target recognition.
Speaker 4: Automatic target recognition, in this case, scanning satellite imagery and
Speaker 4: other data inputs to identify potential threats and enemy locations
Speaker 4: on a battlefield and making sure the US doesn't misidentify
Speaker 4: non threats as targets and end up deploying a weapon
Speaker 4: against a hospital instead of a weapons factory. Until this point,
Speaker 4: the military relied on people to comb through hours of
Speaker 4: video footage and satellite imagery to identify these targets. But
Speaker 4: Roper thought a computer could do this faster, maybe even better.
Speaker 3: In a real battle, there's so much to see and
Speaker 3: observe to try to get down to the handful of
Speaker 3: targets that you need to prosecute with weapons, and it
Speaker 3: was clear that processing had gotten to be so good
Speaker 3: and there was so much data available that you could
Speaker 3: build these very deep neural nets.
Speaker 4: But in twenty seventeen, the US military wasn't familiar with
Speaker 4: deep neural nets and it definitely couldn't build them.
Speaker 3: The Internet of Things did not happen to the US
Speaker 3: military when it happened to everyone else. The US military
Speaker 3: has amazing hardware, but the connectedness between it is not.
Speaker 3: So military members leave their lives where they're connected to
Speaker 3: almost everything, and they inner military lives where they're connected
Speaker 3: to almost nothing. What most people, I think expect is
Speaker 3: that aircraft and ships connect like our smart devices do,
Speaker 3: and they don't not even close.
Speaker 4: So Rouper proposed the military make a major push focused
Speaker 4: on automatic target recognition, But when he got into a
Speaker 4: room with senior defense officials to pitch his idea for
Speaker 4: what would become Project Maven, he didn't exactly get the
Speaker 4: response he wanted.
Speaker 3: Well, skeptics are pretty much what filled the halls of
Speaker 3: the Pentagon. It's not a place of great creativity and innovation,
Speaker 3: and I remember still having to put together a tutorial
Speaker 3: presentation about why this would feasibly work, why this was
Speaker 3: a big trend in commercial industry, and why we needed
Speaker 3: to be on the right side of the trend.
Speaker 4: Like any government bureaucracy, the Pentagon has a lot of
Speaker 4: red tape. Its approaches, nothing like the move fast and
Speaker 4: break things mentality of Silicon Valley. That's in part for
Speaker 4: good reason. The stakes are incredibly high when it comes
Speaker 4: to anything the US government decides to back, but especially
Speaker 4: when it comes to warfare. But some of those Pentagon
Speaker 4: officials did take a chance on Roper's idea, and in
Speaker 4: April twenty seventeen, the Defense Department announced Project Mavin, a
Speaker 4: way to use AI and data for actionable intelligence. What
Speaker 4: that meant was developing machine learning algorithms and then constantly
Speaker 4: retraining those algorithms with up to date, well labeled data.
Speaker 4: But all that outdated military tech that Roper was dealing
Speaker 4: with made that hard. My colleague Katrina spoke to one
Speaker 4: of the experts who worked on testing and evaluation in
Speaker 4: the early days of MAVEN.
Speaker 1: And she said it was really not very good at all.
Speaker 1: They didn't have the data labeled. There was no sense
Speaker 1: of everything that we'd been doing. When we click online
Speaker 1: and say, yes, this is a traffic light, Yes this
Speaker 1: is a cat. There wasn't that same store of images
Speaker 1: for tanks or weapons factories.
Speaker 4: So Roper and his team had people go through images
Speaker 4: and manually label them.
Speaker 3: There were thousands of people participating and labeling data so
Speaker 3: that we could train algorithms.
Speaker 4: They also needed some expertise to develop the program. The
Speaker 4: Pentagon partnered with some commercial tech companies, ones that were
Speaker 4: leaps and bounds ahead of the military when it came
Speaker 4: to AI, but in twenty eighteen that became a problem.
Speaker 1: Google was involved in the early days of Project MABN,
Speaker 1: and in twenty eighteen thousands of Google workers protested. The
Speaker 1: key question was is just applying that ability to recognize
Speaker 1: a cat to a tank? How much moral culpability did
Speaker 1: those workers then feel? Because what might happen next is
Speaker 1: the tank might be exploded, and if it made a mistake,
Speaker 1: a human might be killed, or the targets may eventually
Speaker 1: involve places where humans are or civilians are, And in
Speaker 1: a letter to their seniors said we do not want
Speaker 1: to work on warfare technology.
Speaker 4: It brought Project me even into the spotlight splashed across
Speaker 4: the news quote, we.
Speaker 5: Believe Google should not be in the business of war. Therefore,
Speaker 5: we asked that Project Mayven be canceled, that Google enforced
Speaker 5: a clear policy stating that neither Google nor its contractors
Speaker 5: will ever build warfare technology.
Speaker 4: Did the protests give people inside the military any pause?
Speaker 1: I think they did. I think the military also realized
Speaker 1: it really had to grapple with, well, what are we
Speaker 1: actually asking big tech to do? These are not people
Speaker 1: who signed up to serve the flag and the way
Speaker 1: that they have the missions are completely different, the cultures
Speaker 1: are different. At the time, you had senior defense officials
Speaker 1: flying over to the West Coast to try and understand
Speaker 1: what this completely different culture was about. And since then
Speaker 1: the people in charge of Project Mayven had to pivot
Speaker 1: and find other partners that already were partners besides Google.
Speaker 1: But I think the Pentagon went through a renewed kind
Speaker 1: of outreach to Silicon Valley.
Speaker 4: And with the help of those other tech partners and
Speaker 4: thousands of workers labeling the data, the algorithms behind Project
Speaker 4: Maven got more and more accurate. By twenty twenty, an
Speaker 4: Army colonel named Joseph Callahan, who runs weapons fires, started
Speaker 4: taking an interest in Maven. He was asked to start
Speaker 4: experimenting with it in his unit, the eighteenth Airborne Core
Speaker 4: in North Carolina. He ran live ammunition exercises, using the
Speaker 4: computer algorithm to identify a target like a tank, and
Speaker 4: then having a human deploy a weapon to explode that target.
Speaker 4: Testing the tech is a crucial part of the process.
Speaker 4: The whole point of machine learning is feedback, trying out
Speaker 4: an algorithm and then telling it when it's right or
Speaker 4: wrong so it can learn and do better next time.
Speaker 4: But just because the boss wants his workers to start
Speaker 4: using a shiny new tool doesn't mean everyone will fall
Speaker 4: into line. Katrina told me about one officer named Joey Temple.
Speaker 1: The very battle hardened targeting officer served five rounds in Iraq,
Speaker 1: and he was offered a demonstration of what's called MAVEN
Speaker 1: Smart System.
Speaker 4: The platform that houses me even in other data streams, and.
Speaker 1: He said, no, I don't want a demo. I don't
Speaker 1: want a demo. And he told me that he had
Speaker 1: a sense that stuff like this doesn't work. He didn't
Speaker 1: need another tool. He was fuddled by hearing so much
Speaker 1: about AI when he'd really never worked with AI before
Speaker 1: and he was an expert. He is an expert targeter.
Speaker 1: He's done this his entire career.
Speaker 4: Initially, Temple wasn't interested, but the next month, in August
Speaker 4: of twenty twenty one, something changed his mind, coming up
Speaker 4: how Project Maven won over battle hardened soldiers like Joey Temple,
Speaker 4: and how it's being used on actual battlefields today. At first,
Speaker 4: Senior targeting Officer Joey Temple wasn't interested in using Project Mavin,
Speaker 4: but that changed in August twenty twenty.
Speaker 6: One chaotic scenes as US troops at Kabul Airport strove
Speaker 6: to evacuate Americans as well as Afghan civilians.
Speaker 4: It was the evacuation of US troops from Afghanistan. The
Speaker 4: military had to airlift one hundred and twenty thousand people
Speaker 4: out of Kabble under very dangerous conditions, and Temple was
Speaker 4: put in front of a Maven scream that mapped out
Speaker 4: threats on the ground in real time.
Speaker 1: He could see people walking around the roots of planes.
Speaker 1: For him, seeing Maven's small system operate like that was
Speaker 1: a bit of a light bulb moment.
Speaker 4: The folks in charge of Maven are relying on those
Speaker 4: sorts of light bulb moments. The military needs officers willing
Speaker 4: to use MAIVEN to train its algorithms to get better,
Speaker 4: and to link it up to data and censors and
Speaker 4: actually make it operational in the way the military runs
Speaker 4: its battles. Otherwise it's pretty much useless.
Speaker 1: You do tend to get experiences like that of Joey
Speaker 1: Temple who are reticent at the start. And he even
Speaker 1: told me he has people under him who say, no,
Speaker 1: I don't want to use this. I mean, he calls
Speaker 1: himself a skeptic. He is constantly going to play devil's
Speaker 1: advocate because what's so important for him is to know
Speaker 1: is this system really reliable? That's what I found so interesting.
Speaker 1: How you start integrating what military folks like to call
Speaker 1: human machine teaming. What is that actually like From the
Speaker 1: perspective of the person on the ground who's been told here,
Speaker 1: there's a newfangled toy, use this. Those people have very
Speaker 1: very deep relationships with their fellow combatants, who the people
Speaker 1: they go to war with they trust. That's a really
Speaker 1: fundamental idea of US warfare, of any warfare. What if
Speaker 1: you've got to trust an algorithm?
Speaker 4: The military is now asking soldiers of the eighteenth Airborne
Speaker 4: Corps to trust that algorithm, at least in experiments on
Speaker 4: their base. It's now the largest operational test bed for MAVEN.
Speaker 4: Last year, Katrina flew to the base in North Carolina
Speaker 4: to see it in action herself.
Speaker 1: I was finally given a classied demonstration of it, and
Speaker 1: it was incredibly revealing to see it because it wasn't
Speaker 1: quite what I expected, and they did a mock up
Speaker 1: for me using unclassified satellite imagery, but it represented a
Speaker 1: real operation that the US had conducted several months before.
Speaker 1: And it's very much like looking at a map on
Speaker 1: your phone, only this had yellow boxes and blue boxes,
Speaker 1: and the blue boxes meant don't hit this, this is
Speaker 1: not a target. We've identified this as a school or
Speaker 1: a hospital or friendly forces, and yellow boxes meant what
Speaker 1: they call points of interest. So in this case, if
Speaker 1: you weren't asking the algorithm to find ships, it would
Speaker 1: box out all the ships, and then you might be
Speaker 1: able to ask it to find a specific kind of ship,
Speaker 1: maybe a warship, and they can layer that in using
Speaker 1: lots and lots of different data feed sources.
Speaker 4: Not just the satellite photos and videos that MAVEN was
Speaker 4: originally trained on. Now the program incorporates and analyzes so
Speaker 4: much more data. Katrina got to hear about that from
Speaker 4: Mark Monell. He's the director of Data and Digital Innovation
Speaker 4: at the National Geospatial Intelligence Agency, which since last year
Speaker 4: has run most of MAVEN. The agency has been working
Speaker 4: closely with troops to test field and improve MAVIN on
Speaker 4: the ground.
Speaker 7: We also use radio waves bouncing off of objects, and
Speaker 7: we also use infrared to see heat light, and we
Speaker 7: also use multi spectral remote sensing, so not just the
Speaker 7: visible light spectrum, but also the invisible light spectrum.
Speaker 4: Muncell told her the way he sees it, there are
Speaker 4: two big advantages of using MAVEN and military targeting.
Speaker 7: So speed is very important, right Obviously, the faster that
Speaker 7: we can take an image, the faster that we can
Speaker 7: run an algorithm and get the results is very important.
Speaker 7: But I will say the biggest advantage would be scale.
Speaker 7: The fact that we can do it on many images,
Speaker 7: you know, as fast as we can is really where
Speaker 7: we're providing the advantage.
Speaker 4: An advantage that was just recently proven, not just in
Speaker 4: a test, but on the actual battlefield.
Speaker 6: October seventh, everything changed.
Speaker 4: That's Skyler Moore, the Chief Technology Officer of US Central Command,
Speaker 4: often referred to as Sentcom. Katrina managed to get her
Speaker 4: on the phone earlier this month.
Speaker 6: We immediately shifted into high gear in a much higher
Speaker 6: operational tempo than we had previously. And the reason that
Speaker 6: I think we were able to and had a pretty
Speaker 6: seamless shift of using NAVAN was because we'd done about
Speaker 6: twelve months worth of digital exercises leading up to that.
Speaker 1: More tool.
Speaker 4: Katrina that the US also used ME even to identify
Speaker 4: and narrow down dozens of targets that US air strikes
Speaker 4: then fired against Iraq and Syria.
Speaker 1: That for me is the first time that I've really
Speaker 1: heard someone talk about using AI to find something and
Speaker 1: then deliver a weapon to it that actually belonged to
Speaker 1: the enemy and damage it destroyed some there were casualties,
Speaker 1: and so that cycle is really being operationalized.
Speaker 6: We've certainly had more opportunities to target in a law
Speaker 6: sixty to ninety days. We have a lot of examples
Speaker 6: of the who he's presenting those types of threats.
Speaker 1: They're finding rocket launches in Yemen, they're finding vessels in
Speaker 1: the Red Sea, and Maven has become an intrinsic part
Speaker 1: of that decision cycle to try and find out where
Speaker 1: is there a threat, what do we want to do
Speaker 1: about it, and how can we be sure?
Speaker 4: The tech is being used but it hasn't been perfected
Speaker 4: yet well. Humans at the eighteenth Airborne cor can correctly
Speaker 4: identify a tank about eighty four percent of the time.
Speaker 4: Mayven gets it closer to sixty percent, and experts have
Speaker 4: told Katrina that that number can dip even lower, as
Speaker 4: low as thirty percent when there's snow or cloud cover
Speaker 4: in an image, or if it's searching for a new
Speaker 4: type of object.
Speaker 1: A lot of the US military imagery comes from the
Speaker 1: Middle East. When you move to somewhere like Ukraine, there's snow,
Speaker 1: there's cloud cover, there's rain, so the conditions change a
Speaker 1: lot and the algorithms start performing far less well.
Speaker 4: These challenges are some of the things that critics point
Speaker 4: to when they express concern about the US military using
Speaker 4: this technology. Another big thing that comes up is backlash
Speaker 4: against the idea of using autonomous machines, which can use
Speaker 4: AI on the battlefield at all. There's a coalition of
Speaker 4: human rights and expert groups called Stop Killer Robots that's
Speaker 4: dedicated to keeping lethal autonomous machines that can use AI
Speaker 4: out of warfare. Even UN Secretary General Antonio Guterrez has
Speaker 4: called for a ban on autonomous weapons systems. Here's Katrina again.
Speaker 1: An algorithm itself can be trained using data that was
Speaker 1: incorrect or incomplete. It can also be targeted by adversary attacks.
Speaker 1: The data could be changed without knowing. The algorithm could
Speaker 1: be spoofed, it can be poisoned. It can lose accuracy
Speaker 1: over time. That's a very natural thing for an algorithm
Speaker 1: to do. It stops being as effective. So you have
Speaker 1: to work out at what point do you want to
Speaker 1: change retrain work with that algorithm. And then, of course,
Speaker 1: when you've got a life and death scenarios where you're
Speaker 1: deciding how much trust should be put in this algorithm,
Speaker 1: what should we do with it. It might be feasible
Speaker 1: to use an algorithm that makes mistakes if it's just
Speaker 1: going to blow up water, you know it's trying to
Speaker 1: get a shit, but instead it gets water, a big deal.
Speaker 1: But if you're trying to get something on land and
Speaker 1: if you miss your hitting civilians or any humans, of
Speaker 1: course the risks are extremely high.
Speaker 4: The US military says that this is why it relies
Speaker 4: on human machine teaming. They say that the algorithm itself
Speaker 4: will never make the decision to execute an airstrike. It
Speaker 4: just identifies a target for a human commander. But MAVEN
Speaker 4: is making those decisions and those links to weapons faster.
Speaker 7: We want to shorten that kill chain, you know, as
Speaker 7: fast as we can.
Speaker 4: Mark Mansell again, he's the agency official who oversees the
Speaker 4: office that runs MAVEN.
Speaker 7: So instead of a human having to open up an image,
Speaker 7: scan the image, find everything on the image, and then
Speaker 7: report that out to a combat and commander, we want
Speaker 7: to shorten that by using the algorithms.
Speaker 1: To do that, so the combatant commander gets the choice
Speaker 1: of what to execute fison and that much faster.
Speaker 4: But as Katrina points out, speeding up the pace at
Speaker 4: which targets can be identified and fired on that can
Speaker 4: be complicated.
Speaker 1: And of course there's been a huge amount of work
Speaker 1: on confirmation bias and an algorithm, and I think you
Speaker 1: have to wonder if an algorithm says this is something
Speaker 1: that we should be looking at as a target, does
Speaker 1: that make the person who's there to say yes or
Speaker 1: no to the algorithm more likely to say yes or
Speaker 1: less likely to say yes, the.
Speaker 4: US military expects this kind of technology to improve, but
Speaker 4: in the meantime they're continuing to use it, and other
Speaker 4: countries around the world like Israel and Ukraine are using
Speaker 4: similar technology. That raises the question is the world ready
Speaker 4: for what that might mean?
Speaker 3: We're not ready for that. In the US military, that
Speaker 3: a war of algorithms, a war of software.
Speaker 4: Keeping up is the reason Will Roper wanted to bring
Speaker 4: AI to the US military in the first place.
Speaker 3: We've always had this human advantage in the US military
Speaker 3: because we're always operating that our people are are experienced
Speaker 3: and trained and they're ready to go, and we've always
Speaker 3: been able to say that as a way if we're
Speaker 3: feeling a little nervous about scenario, is that we've got
Speaker 3: the people who are ready well. With AI, it may
Speaker 3: be the first technology where you can undercut human advantage.
Speaker 4: That's a concern to folks of the Pentagon and in
Speaker 4: government who fear a war with say China, could put
Speaker 4: the US on the back foot. But war doesn't wait
Speaker 4: on software updates. Katrina learned that in twenty twenty two,
Speaker 4: the US helped Ukraine by using Maven to share so
Speaker 4: called points of interest where Russian equipment was located. Here's
Speaker 4: this stat that really stuck with me. During that process,
Speaker 4: the Maven smart system platform underwent more than fifty rounds
Speaker 4: of improvement in just the first ten months. Roper thinks
Speaker 4: that the people within the military are going to need
Speaker 4: to change the way they think about algorithms, because like
Speaker 4: it or not, at this point, there's no going back.
Speaker 3: The military is going to have to think about AI
Speaker 3: not as much like a piece of computer software, but
Speaker 3: almost like a member of the military that you train
Speaker 3: and that you trust with a certain amount of authority
Speaker 3: based on its training and pedigree. Hopefully those lessons will
Speaker 3: be learned before they're needed.
Speaker 4: Thanks for listening to The Big Take DC podcast from
Speaker 4: Bloomberg News. I'm Salaiah Mosen. This episode was produced by
Speaker 4: Julia Press and Naomi Shaven. It was fact checked by
Speaker 4: Alex Sugia. Special thanks to Katrina Manson, Margaret Sutherland, Vicki Vergalina,
Speaker 4: and Rebecca Shassen. Ben O'Brien is our mix engineer. Our
Speaker 4: story editors are Caitlin Kenney, Wendy Benjaminson, and Michael Shephard.
Speaker 4: Nicole Beemsterborer is our executive producer. Sage Bauman is our
Speaker 4: head of podcasts. If you like what you heard, please subscribe, rate,
Speaker 4: and review the show. It helps other listeners find us.
Speaker 4: Thanks for tuning in. I'll be back next week.