Speaker 1: Bloomberg Audio Studios, podcasts, radio news.
Speaker 2: The Iran War showed little signs of de escalating this week,
Speaker 2: with Israeli forces targeting nuclear development sites in Iran, the
Speaker 2: US attacking Iranian mind laying vessels in the Strait of Hormuz,
Speaker 2: and Iranian drones striking Dubai International Airport. President Trump and
Speaker 2: Iran's new Supreme leader each vowed Thursday to keep up
Speaker 2: their respective military campaigns amid mounting oil prices.
Speaker 1: Every day we lose about twenty percent of global oil supplies.
Speaker 3: We are just shy of one hundred dollars a barrel
Speaker 3: on Brent now.
Speaker 2: In a post on truth Social Trump wrote that preventing
Speaker 2: Iran from threatening the Middle East and having nuclear weapons
Speaker 2: was far more important to him than the cost of oil.
Speaker 2: Iran's new Supreme Leader, Iotolamus Stabahamane, said Iran would continue
Speaker 2: its blockade of the Strait of Hormuz, adding that he'd
Speaker 2: consider opening other fronts if the war continues. But the
Speaker 2: war has already expanded it to an important new front.
Speaker 2: How it's being waged with AI enabled weaponry. In just
Speaker 2: the first twenty four hours of the Iran War, the
Speaker 2: US hit more than one thousand targets. According to the
Speaker 2: commander of US Forces, that is double the scale of
Speaker 2: the assault that opened the Iraq War in two thousand
Speaker 2: and three.
Speaker 3: The decision to wage a campaign like this is intimately
Speaker 3: linked to how remotely the US can pursue this war.
Speaker 2: That's Bloomberg National security and tech reporter Katrina Manson.
Speaker 3: It's clearly really important for Donald Trump not to send
Speaker 3: US troops in on the ground, so conducting an air
Speaker 3: war with the help of AI suddenly becomes critical to
Speaker 3: his own pursuit of a war.
Speaker 2: The US's first strikes on Iran came hours after the
Speaker 2: Pentagon severed its contract with Anthropic inn AI company that
Speaker 2: had been working directly with the US military. The Pentagon
Speaker 2: has formally notified Anthropic that it's deemed the artificial intelligence
Speaker 2: company and its products a supply chain risk to the
Speaker 2: United States. Pentagoon has said it needs the ability to
Speaker 2: use Claude for quote all lawful purposes. Anthropic, which is
Speaker 2: best known for its clawed AI tool, wanted assurances from
Speaker 2: the US government that its technology would not make lethal
Speaker 2: decisions on its own or help to conduct mass surveillance
Speaker 2: on Americans. Anthropics ouster and a move by its rival
Speaker 2: Open Ai to start working with the Department of Defense
Speaker 2: have struck a nerve. Some Americans are a plauding Anthropics
Speaker 2: decision Claude went down last week because of unprecedented demand.
Speaker 2: It's the latest turn in a long simmering debate over
Speaker 2: how the US military works with Silicon Valley companies and
Speaker 2: how those tech titans contribute to the development of AI
Speaker 2: weapons systems. Katrina took a deep dive into that story,
Speaker 2: and it's a big part of her forthcoming book on
Speaker 2: the US military and AI.
Speaker 3: AI may be good for scale, it may be good
Speaker 3: for speed, but is it as good for accuracy and
Speaker 3: saving lives as these claims have been made in the past.
Speaker 3: We don't know yet.
Speaker 2: I'm David Gera and this is the big take from
Speaker 2: Bloomberg News today on the show, as Anthropic and the
Speaker 2: Pentagon feud. How is the US integrating AI into its
Speaker 2: war fighting machine, How is AI enabled weaponry being regulated?
Speaker 2: And is the tech as good as it needs to
Speaker 2: be with so many lives at risk? About five years ago,
Speaker 2: Dario Amide and his sister left their jobs at OpenAI
Speaker 2: to found Anthropic, an AI company meant to enforce strong
Speaker 2: safety guardrails. Since then, Anthropic has achieved mainstream success with Claude.
Speaker 2: It's even managed to partner with the US government, but
Speaker 2: that all changed at the start of the year. Bloomberg
Speaker 2: Senior editor Mike Shepherd has been following the saga closely.
Speaker 1: The Pentagon began reviewing how it wanted to roll out
Speaker 1: artificial intelligence across the Armed Force, and in January they
Speaker 1: put out an order from Defense Secretary Pete Hegseth saying
Speaker 1: that they needed to accelerate adoption across the Defense Department,
Speaker 1: which he refers to as the Department of War. He
Speaker 1: even put up posters around the building saying the government
Speaker 1: wants you to use AI as part of it. They
Speaker 1: included language that said they did not want to be
Speaker 1: bound by any usage restrictions that might come from an
Speaker 1: AI provider of any stripe. They didn't single out Anthropic,
Speaker 1: but Anthropic is not only one of the leading AI
Speaker 1: developers and providers in the world, they are one that
Speaker 1: has stood out for what many see as its safety
Speaker 1: first stance and its adherence to principles of trying to
Speaker 1: develop AI with a mind toward avoiding some of the
Speaker 1: worst case scenarios. Now. For its part, Aanthropic has expressed
Speaker 1: continued interest in working with the military so long as
Speaker 1: the Pentagon A military officials abide by those usage restrictions.
Speaker 1: And the two that have really been a redline for
Speaker 1: the company concern the use of its AI technology for
Speaker 1: mass surveillance of Americans domestically and then also fully autonomous
Speaker 1: weapons the climate they as the developers of the technology,
Speaker 1: insist that they don't have full confidence that it's ready
Speaker 1: for those uses safely.
Speaker 2: Katrina, you have a book coming out later this month
Speaker 2: on the Pentagon's efforts to integrate AI into warfare. I'm
Speaker 2: curious when that initiative started. When did the Pentagon begin
Speaker 2: looking into this an interest.
Speaker 3: The pentagonice to say it's been developing AI for sixty years.
Speaker 3: But the real project that I think we can set
Speaker 3: the timerby is Project Maven, which started in twenty seventeen,
Speaker 3: and it was an effort that really spoke to America's
Speaker 3: concern about falling behind China. The US began to realize
Speaker 3: it was using old tech and that this new age
Speaker 3: of warfare was coming that would require robots, autonomy, and AI,
Speaker 3: and Project Maven was this effort to experiment with AI.
Speaker 3: At the time, the cutting edge was computer vision, things
Speaker 3: that could identify objects on a video feed taken by
Speaker 3: drones and process that quicker. But the people I spoke
Speaker 3: to also explained to me that all along they imagined
Speaker 3: that this would help with targeting.
Speaker 2: You write about the software that comes out of Project
Speaker 2: man Maven smart System, which is made by Palenteer and
Speaker 2: incorporated technology from Amazon and Microsoft and others. Is this
Speaker 2: something the Defense Department thought they could do in house
Speaker 2: or was it always something that they needed the cooperation
Speaker 2: of private industry.
Speaker 3: It's a real core point that obviously the Pentagon is
Speaker 3: still struggling with. For this very advanced tech, This recognition
Speaker 3: that has been happening over the past ten years that
Speaker 3: warfare is going to be what some call software defined,
Speaker 3: they need commercial companies, and so when Project Maven happened
Speaker 3: for the first time, really they were going outside the
Speaker 3: traditional primes as they're known, the traditional defense contractors and
Speaker 3: looking at companies like Google, Microsoft, and Amazon, but other
Speaker 3: tiny startups as well who were just beginning to experiment
Speaker 3: with fascinating algorithms that could detect images on wedding cakes
Speaker 3: and then reapplying that for the tools of war, and
Speaker 3: that transition was incredibly uncomfortable, clunky, difficult, and they're backfired
Speaker 3: at one point rather spectacularly when in twenty eighteen, Google
Speaker 3: workers discovered that their company was working on Project Maven,
Speaker 3: and they feared that their tech and their traditions for
Speaker 3: them could be subverted into what they ended up calling
Speaker 3: in a protest letter the business of war, and the
Speaker 3: parallel with Anthropic today isn't quite the same. The CEO
Speaker 3: of Anthropic all Along has said he believes in national
Speaker 3: security work more than any other AI lab. Anthropic has
Speaker 3: lent into classified work, and the classified cloud is where
Speaker 3: Penticon does its fighting, so they've been involved with lethal
Speaker 3: operations in some way for more than a year. What
Speaker 3: I think is different is he had these red lines,
Speaker 3: and as he is trying to uphold them, tech workers
Speaker 3: in other companies are really paying note now.
Speaker 2: Katrina ethics are so central to this program, and I'm
Speaker 2: curious how defense officials addressed those ethics of seeding life
Speaker 2: or death decision making to machines to artificial intelligence.
Speaker 3: They would argue, I think very current now to the
Speaker 3: operations that are using AI tools against Iran, that humans
Speaker 3: are still making the decisions. But in a very interesting
Speaker 3: presentation by the Commander of US Forces just this week,
Speaker 3: they've made clear that they've hit five thousand, five hundred targets.
Speaker 3: That speed is exactly why they want AI and that scope,
Speaker 3: and the Pentagon has had to work very hard thinking
Speaker 3: through precision. The claim, of course is that US weapons
Speaker 3: are the most precise in the world. But where you
Speaker 3: decide to put them, that is where America has had
Speaker 3: problems and the scale at which it's shooting at areas
Speaker 3: where we know civilians are present, because Scentcom is warning
Speaker 3: civilians to stay home, telling them to stay away from ports.
Speaker 3: There is a margin of error. There's a decision to
Speaker 3: accept or not accept a certain amount of collateral damage.
Speaker 3: They haven't made public what margin they set on that
Speaker 3: collateral damage, but these are key ethical decisions in The
Speaker 3: targeting process is a multi stage effort where people feed
Speaker 3: in and eventually a commander makes a decision to sign off.
Speaker 3: But as you involve AI, that process speeds up. The
Speaker 3: decision making time is reduced, and fears for things like
Speaker 3: automation bias or the algorithms themselves going wrong. Hallucinating or
Speaker 3: drifting algorithms tend to naturally get worse over time. All
Speaker 3: of that has really yet to be worked out. Now
Speaker 3: the US has got a policy for that. It's a directive.
Speaker 3: Some people to me frame it more as a process
Speaker 3: than a rule, but it does say that there needs
Speaker 3: to be human judgment over the appropriate use of force. Well,
Speaker 3: that's supervision, that's not necessarily making a decision. And so
Speaker 3: I think from all the people I've spoken to, that role,
Speaker 3: that human role is reducing, and Sentcom has been particularly
Speaker 3: proud this week the commander to say that AI is
Speaker 3: helping them reduce operations decisions from what used to be
Speaker 3: days and hours two seconds.
Speaker 2: After the break. How good is this technology really? We
Speaker 2: hear about one test where things went awry, and we
Speaker 2: dig into how lawmakers are responding. Proponents of AI enabled
Speaker 2: weapons say the technology can improve decision making on the
Speaker 2: back and put fewer troops at risk, which could save lives.
Speaker 2: Its promise of speed makes it imperative to national security
Speaker 2: experts who see it as essential in the widening AI
Speaker 2: arms race with China, but Bloomberg's Katrina Manson points out
Speaker 2: it also comes with risks.
Speaker 3: AI is this fundamentally unpredictable black box technology. So it's
Speaker 3: brilliant at bringing a lot of data to bear. It
Speaker 3: just might be the wrong data, and it might be
Speaker 3: organized in the wrong way. And if you're not giving
Speaker 3: enough time to checking, or if you don't understand where
Speaker 3: it can make mistakes and you put it into critical
Speaker 3: a place in your own system, you've made yourself incredibly fallible. Now,
Speaker 3: from the US perspective, that brings a couple of problems.
Speaker 3: One is you may end up killing civilians, which is
Speaker 3: against US policy. Certainly deliberate targeting inadvertent targeting under the
Speaker 3: rule of law has to be proportionate. But the second
Speaker 3: thing is you can also hurt your own troops and
Speaker 3: allied troops. And the US has a history of grappling
Speaker 3: with what are called friendly fire incidents, and AI was
Speaker 3: meant to help clear up this kind of claus witsy
Speaker 3: and idea of the fog of war. One of the
Speaker 3: ways they've integrated AI into the battlefield picture is through
Speaker 3: the system that emerges from Project Man called Maven Smart System.
Speaker 3: For most of US, it's a really basic idea. It's
Speaker 3: essentially just a digital map. It's Google Maps for war.
Speaker 3: It's really hard for the military to create that because
Speaker 3: they have data in all sorts of places, they haven't
Speaker 3: labeled their data. They have spent the last ten years
Speaker 3: really getting their equipment, their tech, their connectivity up to
Speaker 3: scale where something like this could work, but then integrating
Speaker 3: AI into it is really down to what checks they
Speaker 3: make for it the algorithms themselves. I found cases where
Speaker 3: the quality of the algorithm, the ability of the algorithm
Speaker 3: to detect what it's meant to be identifying, can change
Speaker 3: if the weather changes, if you move an algorithm from
Speaker 3: a hot area to a green area, picking up something
Speaker 3: as simple as a truck or a tank, andinguishing between
Speaker 3: a man or a woman or a child, which is
Speaker 3: really critical to those life and death decisions and the
Speaker 3: rules under which the US military operates, really isn't at
Speaker 3: the stage that you can give that over to algorithms.
Speaker 3: So the risk is if they are not sufficiently tire
Speaker 3: kicking their own algorithms when it comes to computer vision,
Speaker 3: or when it comes to using lms and chatbots to
Speaker 3: speed up their processes and analyze data, you could be
Speaker 3: going in the wrong direction for quite some time before
Speaker 3: you notice it.
Speaker 2: Katran I wonder if you coul tell us a story
Speaker 2: of what happens when this technology doesn't work as it's
Speaker 2: intended to. And you write about a test that took
Speaker 2: place back in June of twenty twenty five, maybe you
Speaker 2: could just walk us through what happened then and what
Speaker 2: lessons we can draw from how that unintended result could
Speaker 2: pretend difficulties down the road.
Speaker 3: The teams behind Maven had made computer vision and then
Speaker 3: this was the next iteration. This is where a program
Speaker 3: called replicator, so it would be to identify a target
Speaker 3: and then the drone could go after the target and
Speaker 3: then identify it and explode. And developing automatic target recognition
Speaker 3: is one thing that relies on creating algorithms that can
Speaker 3: find things. The other is this swarming technology coordinating between drones.
Speaker 3: Of course they were testing this. It's expected to fail,
Speaker 3: that's why they do it in test conditions. And the
Speaker 3: aim was to deliver multiple thousands of drones by a
Speaker 3: certain date. And two months before that date, there was
Speaker 3: a test in California of some drone boats and in
Speaker 3: this one experiment, the drone was towed out to see
Speaker 3: before it was to be switched into autonomy mode. That
Speaker 3: in itself was meant to be a safety measure What
Speaker 3: happened was, inadvertently a command was sent from the dock
Speaker 3: to the drone boat, and when the drone boat was
Speaker 3: activated into autonomy mode without anyone realizing it, it started
Speaker 3: trying to get away from the captain who was towing it.
Speaker 3: The boat started accelerating, decelerating multiple times between zero and
Speaker 3: six knots at pace semi circling action. The rope goes
Speaker 3: taught and the captain is capsized. At that point, he's
Speaker 3: in the water, and then the drone boat turns and
Speaker 3: comes toward him. That's a very dangerous moment because the
Speaker 3: rope could strangle him underwater, he could be submerged. There's
Speaker 3: a runaway boat coming for him. A third boat was
Speaker 3: able to intervene and save him, and he was okay.
Speaker 3: Then what actually happened in the investigation that ensued was
Speaker 3: that someone had mistakenly sent was called a zero command
Speaker 3: just by pressing enter on a command line, deployed the
Speaker 3: boat into autonomy mode, and then all of this ensued,
Speaker 3: so they fixed it. This is a very early stage
Speaker 3: of testing. They're meant to be putting explosives on these boats.
Speaker 3: They're meant for these boats to be all cooperating together
Speaker 3: and able to defend an island such as Taiwan in
Speaker 3: the event of an invasion, And I think it showed
Speaker 3: that the tech is really hard to deliver and just
Speaker 3: simply not ready.
Speaker 2: In a statement for Katrina's book, a Navy spokesperson said
Speaker 2: that safety is always their top priority, that they have
Speaker 2: backup systems to prevent danger. The lessons they learned during
Speaker 2: these events drive improvements in their systems. Mike, these conversations
Speaker 2: about ethics are happening among the Defense Department and these
Speaker 2: private companies. Where are lawmakers? Is Congress showcasing any interest
Speaker 2: in engaging with the subject.
Speaker 1: Showcasing is a good word, because they do like to
Speaker 1: showcase their interest, but actually advancing a proposal that would
Speaker 1: codify some regulation or law even on how AI is
Speaker 1: deployed in warfare, we are a long way from that.
Speaker 1: There's discussion of attaching some sort of amendment to the
Speaker 1: Annual Defense Authorization Measure, which every year usually gets caught
Speaker 1: up in the fight of the moment, and that could
Speaker 1: be where this ultimately ends up, But it's unclear whether
Speaker 1: there would be enough consensus between both parties to really
Speaker 1: come up with language. They all could agree on and
Speaker 1: that the Trump administration would not try to torpedo somehow itself.
Speaker 1: Remember the tech industry, a lot of the big tech
Speaker 1: companies have really moved to align themselves with President Donald
Speaker 1: Trump in a lot of different ways, and it's difficult
Speaker 1: for them and a lot of other business fronts to
Speaker 1: challenge the president, including on this.
Speaker 2: Katrina, I would ask you lastly, just about how much
Speaker 2: the horse is out of the barn here. I think
Speaker 2: there'll be a lot of people listening to this who
Speaker 2: will be impressed by how far this field has advanced,
Speaker 2: that is a integration into warfare, and may wonder if
Speaker 2: there's any recourse or anything that they can do they
Speaker 2: as citizens to slow down this process.
Speaker 3: People talk about a costless war, We've already seen that
Speaker 3: from the US side alone. It's not a costless war.
Speaker 3: AI is meant to give you a riskless war. And
Speaker 3: probably where civilians and citizens come involved, are trying to
Speaker 3: understand the contours of is there such a thing as
Speaker 3: a riskless war? And who is harmed by that? And
Speaker 3: if in any way AI isn't saving civilians or even
Speaker 3: there are misfires that involve AI, then really you have
Speaker 3: to re examine if it makes war more likely that
Speaker 3: in itself is a change for the way that the
Speaker 3: US had maybe been thinking about conducting wars under President
Speaker 3: Trump himself, who of course is the main political leader
Speaker 3: who has said I don't want any more wars.
Speaker 2: Katrina Manson's book Project Mavin, A Marine Colonel, His Team
Speaker 2: and the Dawn of AI Warfare comes out later this month.
Speaker 2: This is the Big Take from Bloomberg News. I'm David Gura.
Speaker 2: To get more from The Big Take and unlimited access
Speaker 2: to all of Bloomberg dot com, subscribe today at Bloomberg
Speaker 2: dot com slash podcast offer. If you like this episode,
Speaker 2: make sure to follow and review The Big Take wherever
Speaker 2: you listen to podcasts. It helps people find the show.
Speaker 2: Thanks for listening. We'll be back tomorrow.