Speaker 1: Bloomberg Audio Studios, podcasts, radio news. The fact is China
Speaker 1: has cutting edge AI labs. There are only six months
Speaker 1: behind the frontier in the US. And if they get
Speaker 1: a really powerful model that can do cyber hacking, for example,
Speaker 1: and they put it in their systems and their systems
Speaker 1: are open source, it's game over. This is just going
Speaker 1: to proliferate over all around the world. So that is
Speaker 1: a very bad outcome. So let's try and be creative
Speaker 1: about at least trying with China to talk to them.
Speaker 1: You're going to need both sides to slow down. But
Speaker 1: my point about the discussion in China, which I've came
Speaker 1: into touch with when I was there, is that there
Speaker 1: is at least a discussion and openness to think about safety.
Speaker 1: The door is opened, just a crack, and so it's
Speaker 1: up to American negotiators to go there and try and
Speaker 1: give it a nudge and open it more.
Speaker 2: I'm Stephanie Lander's head of Government and Economics at Bloomberg,
Speaker 2: and this is Trumpnomics, the podcast that looks at the
Speaker 2: economic world of Donald Trump, how he's already shaking up
Speaker 2: the global economy. Modern Earth is going to happen next
Speaker 2: last week we looked ahead to this week's summit between
Speaker 2: the US and Chinese presidents and recording this on Tuesday,
Speaker 2: May twelfth, and by the time you listen to this,
Speaker 2: President Trump should already have touched down in Beijing ready
Speaker 2: for what feels like a high stakes encounter on Thursday
Speaker 2: with President Chi Jinping. Even though the expectations for concrete
Speaker 2: policy outcomes, as we heard last week, are firmly under
Speaker 2: control now. Along with trade, one of the big issues
Speaker 2: hanging over that conversation will be US China competition in
Speaker 2: advanced technologies and the race to dominate AI. Tesla's Elon
Speaker 2: Musk and Apples. Tim Cook are in the group of
Speaker 2: top executives Trump decided to invite with him to China,
Speaker 2: and there is some mild speculation that President Trump might
Speaker 2: further loosen the export controls on US cutting edge semiconductors
Speaker 2: for Chinese companies. But the safety of this race, protecting
Speaker 2: the world from the misuse of these increasingly powerful tools
Speaker 2: both countries are now capable of producing, well, that doesn't
Speaker 2: seem to be high on the agenda. This week's guest
Speaker 2: on Trumpnomics thinks that is a big and potentially dangerous
Speaker 2: missed opportunity. Sebastian Mallaby is a senior fellow at the
Speaker 2: Council and Farm Relations, author of multiple award winning books,
Speaker 2: and most recently of The Infinity Machine. Demisseivis Deep Mind
Speaker 2: and the Quest for Superintelligence. I sat down with him
Speaker 2: in our New York studio a few weeks ago now
Speaker 2: to talk about his new book, The Future of AI
Speaker 2: and the Role of China, which he has recently visited.
Speaker 2: I think you'll find what he had to say fascinating,
Speaker 2: whether hopeful or pretty alarming. Well, i'lli that to you. Sebastian. Welcome,
Speaker 2: Thank you very much for being here in the flesh
Speaker 2: in the New York studio, which is always fun. There's
Speaker 2: so many themes coming out of your book, The Infinity Machine.
Speaker 2: I wanted to start with something that we talk about
Speaker 2: a fair bit on Trumpanomics, which is the impact of
Speaker 2: AI on the economy, society, I guess especially jobs. But
Speaker 2: I think given the themes of the book, and I
Speaker 2: know things that you've been thinking about since it came out,
Speaker 2: will move on fairly quickly to the question of safety.
Speaker 2: How worried would we be about the pace of the
Speaker 2: improvement in AI's capacity, but also how likely is it
Speaker 2: that governments or anyone else will be able to bend
Speaker 2: it to their will? Humanities will So first thing first,
Speaker 2: we had a conversation with darren Os and Moglu about
Speaker 2: it recently. I mean economists. I've noticed they tend to
Speaker 2: divide into those who say, AI is going to have
Speaker 2: a very radical impact on business models, on the economy,
Speaker 2: on society, but that's going to take a long time
Speaker 2: and we will have time to prepare. There's another group
Speaker 2: who say, no, no, this is all going to happen
Speaker 2: much sooner, but it will probably be less seismic. You
Speaker 2: had all these years researching this book really up close,
Speaker 2: looking at the evolution of this technology. What's your best
Speaker 2: guess on that?
Speaker 1: Yeah. I think the second view, which is that there'll
Speaker 1: be an explosion in the power of the models and
Speaker 1: it will immediately affect the economy, is put about by
Speaker 1: the technologists who are just focused on the algorithms and
Speaker 1: on the models and what the base model can do,
Speaker 1: and it completely underestimates the rollout challenges. If you think
Speaker 1: about this idea of recursive self improvement, which preoccupies a
Speaker 1: lot of people in Silicon Valley, once you've got a
Speaker 1: machine that is good enough to write the code for
Speaker 1: the next iteration of the machine, and then it recursively
Speaker 1: self improves. It'll just explode upwards. And that is true
Speaker 1: so far as the computer science goes. It's not remotely
Speaker 1: true as far as actually applying it to the real
Speaker 1: economy goes, because you need to build the computing capacity
Speaker 1: to serve those models. So you need all the chips,
Speaker 1: and there's a shortage of chips. You then need all
Speaker 1: the power to fuel all those chips. The people who
Speaker 1: are in the center of this a talking about having
Speaker 1: to launch data centers in space. That just gives you
Speaker 1: a sense of how difficult this is going to be.
Speaker 1: And that's not to say anything about the institutional frictions
Speaker 1: of integrating AI into say a law firm or a
Speaker 1: bank or whatever. I mean, that's a non trivial challenge.
Speaker 1: You've got to get all your clients to sign off
Speaker 1: on the confidentiality of their information and so forth. So
Speaker 1: I do think that rolling it out such that it
Speaker 1: makes a difference in terms of job displacement or productivity
Speaker 1: is more kind of a decade plus than it is
Speaker 1: a kind of three to five year thing.
Speaker 2: And on your point about the exponential improvement, I guess
Speaker 2: the other thing and the colleagues who are much more
Speaker 2: focused on this than the economics team have also said
Speaker 2: this to me that when you think about sort of
Speaker 2: pie chart of all the kinds of time asks that
Speaker 2: humans do that are part of the workplace, that exponential
Speaker 2: improvement is really only in a certain subset of those tasks,
Speaker 2: and there are other areas where they're really quite far
Speaker 2: behind that are quite important for jobs. Yeah.
Speaker 1: Another way of saying the same thing is that there
Speaker 1: are sectors which are probably going to be quite difficult
Speaker 1: to disrupt, and the public sector is pretty slow at
Speaker 1: adopting tech, and that's a pretty big chunk of some economies,
Speaker 1: right construction, healthcare. There's a whole slew of economic sectors
Speaker 1: which don't seem to be very right for disruption at all.
Speaker 1: And then the ones that are right for disruption the
Speaker 1: archetypical people working on a screen doing knowledge work and analysis,
Speaker 1: which is very similar to what AI does. As I say,
Speaker 1: you've got to get the customers okay with it, and
Speaker 1: that's quite a challenge.
Speaker 2: Railways is the example that's often used, and it takes
Speaker 2: a long time for business models to adapt to that.
Speaker 2: If I think of something like the smartphone, which is
Speaker 2: extraordinarily new by historical standards, less than a decade old,
Speaker 2: day to day, all of us have been transformed by that.
Speaker 2: So is it comparable to that that it's not necessarily
Speaker 2: having a big change on the economy or on productivity,
Speaker 2: but day to day we're all being affected really quite quickly.
Speaker 1: Yeah. I think there's a lot of truth of that.
Speaker 1: That people will be using it. They already are using it.
Speaker 1: It'll occupy quite a lot of their headspace, but it's
Speaker 1: not necessarily making them more productive. One of my favorite
Speaker 1: stories about the anthropic agent agentic system, which was out
Speaker 1: in January of this year. This is before Mythos, but
Speaker 1: the earlier one. People got very excited about, Oh, this
Speaker 1: agent can organize my email, it can create a bespoke
Speaker 1: new summary for me in the morning. And so a
Speaker 1: friend of mine was implementing all these productivity enhancing hacks,
Speaker 1: and finally, after doing about four of these, she said
Speaker 1: to the agent, would you please now sync up my
Speaker 1: calendar such that you check that the train timetables on
Speaker 1: average are not too late, or some sort of extra
Speaker 1: productivity thing to which Anthropic answered or Claude answered, Are
Speaker 1: you sure you're not just giving me this taskause you're
Speaker 1: trying to avoid your own work, you know? So, I mean,
Speaker 1: I do think there's a difference between the excitement around
Speaker 1: it and the adoption of it to actually make you
Speaker 1: more productive. And then even in science, Alpha fold, which
Speaker 1: was the deep mind system published in twenty twenty end
Speaker 1: of twenty twenty for which de mister Sabis got the
Speaker 1: Nobel Prize, which unraveled protein folding. This was said at
Speaker 1: the time to be the breakthrough for faster drug discovery,
Speaker 1: and it probably will be over the longer term. But
Speaker 1: now we're six years later, there isn't a single drug
Speaker 1: that's been discovered thanks to alpha fold. Not to say
Speaker 1: that there hasn't been progressed, because now there's a much
Speaker 1: more sophisticated build out of alpha fold, with other aspects
Speaker 1: of the drug discovery process being automated by AI. So
Speaker 1: I think on a kind of ten to fifteen year horizon,
Speaker 1: yes it will massively accelerate drug discovery, but on a
Speaker 1: six year horizon hasn't.
Speaker 2: That's a really interesting example. There is a wonderful one
Speaker 2: of the good documentaries about that whole period. It's almost
Speaker 2: a sort of companion to your book or a part
Speaker 2: of your book. But there's a wonderful bit of footage
Speaker 2: where they have decided to just identify all of the
Speaker 2: proteins and then and just put them onto the Internet
Speaker 2: and have them allow researchers anywhere to access them. And
Speaker 2: they're watching in real time the uploads from all over
Speaker 2: the world on a map of the world, and it's
Speaker 2: very moving because you're sort of seeing in India and
Speaker 2: Africa or all these researchers a lot more than they
Speaker 2: thought in the space of a few hours uploading all
Speaker 2: this information. But it's fascinating to hear that they have
Speaker 2: not actually produced any new drugs on the back of it.
Speaker 2: I mean that goes to something and we move into
Speaker 2: the broader sort of policy and safety territory a little bit.
Speaker 2: But Darrenus and Moglu was on the show recently and
Speaker 2: I was checking in with him because we talked to
Speaker 2: him over time on this very issue, and his focus
Speaker 2: and his advice to governments had been to think about
Speaker 2: out you can't alter the nature of the technology, but
Speaker 2: you can change the way it's adopted, and that it
Speaker 2: was adoption was going to be the main thing that
Speaker 2: determined how jobs were affected, whether you're enhancing productivity of
Speaker 2: humans or merely just allowing businesses to replace humans. I
Speaker 2: later have a conversation with Jason Furman about that, the
Speaker 2: former head of the Council of Economic Advisors, and he
Speaker 2: thought that was just completely wishful thinking that you can't
Speaker 2: tilt the technology even to that extent that you can
Speaker 2: encourage it to be more complementary to labor as opposed
Speaker 2: to just substituting for labor. So what do you think
Speaker 2: about that?
Speaker 1: You know, I think in a race dynamic where you
Speaker 1: have multiple labs in multiple countries producing this stuff, the
Speaker 1: idea that you're going to control all of them in
Speaker 1: some direction that makes the technology complementary to labor, not replacing,
Speaker 1: does strike me. I'm kind of on Jason's side on that.
Speaker 1: The other thing to say is that the labs are
Speaker 1: going to create what the consumers want, and if the
Speaker 1: consumer's meaning in some cases the business consumers, the enterprises
Speaker 1: have an opportunity to replace labor, they want to take that,
Speaker 1: and if they don't, their competitors and some other country will,
Speaker 1: And so that the laws of competition and capitalism, I
Speaker 1: think are very much on Jason's side.
Speaker 2: I think part of that is this dynamic which is
Speaker 2: also very relevant to the book of the race with
Speaker 2: other countries, particularly the race with China. But if you're
Speaker 2: a country, if you're the UK for example, or could
Speaker 2: be any European country that's not necessarily going to be
Speaker 2: at the forefront of the technology, but clearly does want
Speaker 2: to be at the forefront or close to the front
Speaker 2: in adoption in a way that also helps the economy,
Speaker 2: helps their workers. Is there any advice from your book
Speaker 2: or from the research that you've done that you would
Speaker 2: have for government who are trying to do that?
Speaker 1: Yeah, very much. I mean, I think there is a
Speaker 1: lot of commentators, especially in Britain, draw exactly the wrong
Speaker 1: lesson about the story of deep Mind. So the classic
Speaker 1: complaint is deep Mind is a British company founded in London,
Speaker 1: and main founder is British. She's very patriotic about it.
Speaker 1: He refuses to move to Silicon Valley, and it turns
Speaker 1: out to be the most consequential sort of pioneer AI
Speaker 1: lab in the world. That's amazing, But these idiots sell
Speaker 1: it to Google in twenty fourteen and it becomes a
Speaker 1: subsidiary of an American company, and that's terrible, and I
Speaker 1: think that's wrongheaded in the sense that really the sale
Speaker 1: to Google by deep Mind in twenty fourteen was a trick,
Speaker 1: one could say, to get the American parent to pour
Speaker 1: nearly a billion dollars a year of research and development
Speaker 1: money into London, with enormous spill of effects for the
Speaker 1: tech ecosystem in London. So I think that's a win
Speaker 1: for Britain. The company stayed in London, the founder stayed
Speaker 1: in London. That's great for Britain. What was terrible for Britain,
Speaker 1: and about which Britain should beat itself up mightily, is
Speaker 1: that My book also tells the story of deep Mind
Speaker 1: trying to help the National Health Service, and both Demis's
Speaker 1: mother trained as as a nurse, his co founder, me
Speaker 1: Stuffer Sulliman, his mother trained as a nurse. For this reason,
Speaker 1: they really wanted to help the NHS. They were willing
Speaker 1: to do it pro bono for the first five years,
Speaker 1: and they started helping and rolling out AI algorithms that
Speaker 1: recognized early signs of blindness, early signs of cancer. This
Speaker 1: was just super positive, helpful stuff that was going to
Speaker 1: save the NHS time and help h based patients, and
Speaker 1: it got blown up by a stupid backlash over privacy
Speaker 1: or privacy depending on which side of the Atlantic youuron,
Speaker 1: which I mean, I really kicked the tires on this.
Speaker 1: The complaint about leaks of patient data were fictitious and
Speaker 1: so Britain because of this political climate of suspicion of tech,
Speaker 1: particularly suspicion of the subsidiary of the American tech company,
Speaker 1: created such a backlash against this collaboration between the NHS
Speaker 1: and deep Mind that both the doctors and the computer
Speaker 1: scientists at deep Mind got cold feet and the collaboration collapsed,
Speaker 1: and Britain could have been really the cutting edge country
Speaker 1: in the world on medical AI and they blew it.
Speaker 2: Putting together the two pieces of what you said, they
Speaker 2: are related because one of the reasons that there was
Speaker 2: a lack of trust was the foreign ownership, the US ownership,
Speaker 2: and particularly the association with major tech companies in Silicon
Speaker 2: Valley who have not necessarily earned our trust. Many people
Speaker 2: would say, over the last twenty years, they've made promises
Speaker 2: about privacy, about sharing of information that have been broken.
Speaker 2: Every single tech company has broken them, including Google. They
Speaker 2: have changed their terms on which even they were developing AI.
Speaker 2: In different cases there were promises even that had been
Speaker 2: built originally into the deep Mind Charter around safety and
Speaker 2: other things, which then got overridden as they needed just
Speaker 2: more and more money from the parents. So it seems
Speaker 2: to me, yes, in principle it would have been great
Speaker 2: to have the UK on the cutting edge, but also
Speaker 2: completely understandable that having given tech companies a benefit of
Speaker 2: the doubt of various other times and rather regressed that
Speaker 2: they wouldn't want to do it in the case of
Speaker 2: this very sensitive subject.
Speaker 1: Yeah, that's all true. At the same time, it would
Speaker 1: have been nice if the critics looked at the specifics
Speaker 1: or what was going on in this collaboration between deep
Speaker 1: Mind and the National Health Service, and if they'd look
Speaker 1: more carefully the specifics, they would have seen that this
Speaker 1: was positive for Britain. And if you just have a
Speaker 1: blanket view that anything to do with American tech is
Speaker 1: ipso facto a matter of suspicion because there's been bad
Speaker 1: behavior in other instances, I'm not in favor of that.
Speaker 1: I think that Britain did miss an opportunity there, but
Speaker 1: I think the issue that arises out of this is
Speaker 1: that it's in the interest of the tech companies, especially
Speaker 1: now with this super powerful technology AI, to get governance right,
Speaker 1: to be proactive in inviting governments with democratic legitimacy to
Speaker 1: hold sway over the technology. Because if you just take
Speaker 1: it unto yourself as a youngish startup and you say
Speaker 1: this is how healthcare is going to evolve, I'm doing it,
Speaker 1: it's not legitimate and people won't accept it.
Speaker 2: I mean, some of this gets to that sort of
Speaker 2: broader question of our degree of control and also the
Speaker 2: degree to which any of the people involved in this
Speaker 2: can shape the outcome in line with what they themselves
Speaker 2: want when they're in the competitive context. You have a
Speaker 2: very striking phrase in your book towards the end, as
Speaker 2: inventors dream of shaping the technology that they create, often
Speaker 2: the technology shapes them. The technology plus the business, political,
Speaker 2: and geopolitical currents that it unleashes. There's a bit in
Speaker 2: your book where it starts to feel I felt like
Speaker 2: it's some of the historical analysis that you see of
Speaker 2: the start of World War One. You have everyone individually
Speaker 2: in your story had potentially the right incentives. Was very
Speaker 2: focused on being careful, identifying risks ahead of time, managing
Speaker 2: them ahead of time, not inflicting things on society before
Speaker 2: we were ready for them. The end result has not
Speaker 2: been that at all.
Speaker 1: That's right. The central character in my book, demister Sabez,
Speaker 1: is a good person, has good values, but whether he
Speaker 1: can actually do good even if he is good, is
Speaker 1: the central conundrum. And he really wanted to make things safer.
Speaker 1: That's why he fought a secret three year war with
Speaker 1: Google trying to force them to put more safety guardrails
Speaker 1: around AI. That's why he suggested to Rishie Sunak the
Speaker 1: idea of a Bletchley Park AI safety conference, which happened
Speaker 1: in late twenty twenty three. So he does care about safety,
Speaker 1: but when other labs are willing to go fast, there
Speaker 1: isn't much a single AI leader can do. And what,
Speaker 1: of course that points you to is that the collective
Speaker 1: action problem needs to be solved by government.
Speaker 2: But what we've seen in a lot of these cases
Speaker 2: is the researchers involve the companies involved say, yes, we
Speaker 2: think it's very important to have a collective agreement. But
Speaker 2: if a government comes along with concrete rules, whether it's
Speaker 2: the I'm sure you probably feel strong about but the
Speaker 2: EU Internet Safety Rules or AI Safety rules. President Biden
Speaker 2: had a rather different set of rules. There tends to
Speaker 2: be the response, how could governments possibly understand this technology?
Speaker 2: We want some agreement, but not this one. So how
Speaker 2: do you solve that?
Speaker 1: Yeah, look, you're right that you know at the moment
Speaker 1: there is lots of state level initiatives to regulate AI
Speaker 1: in the US, and for example, open AI is actually
Speaker 1: funding the opposition to those regulatory efforts. So there is
Speaker 1: hypocrisy sometimes.
Speaker 2: And after the fact they often say yes, actually it's
Speaker 2: quite good and we've adopted it, but they never want
Speaker 2: it in advance.
Speaker 1: Yeah. On the other hand, there is a counterexample, which
Speaker 1: is that in the Biden administration when they set up
Speaker 1: the AI Safety Institute, and it included a mandatory requirement
Speaker 1: for all the frontier labs to submit their models to
Speaker 1: this institute for review. The institute didn't have a veto,
Speaker 1: which it should have done in my view, but still
Speaker 1: it was mandatory to submit. And I've spoken to the
Speaker 1: Biden people who created that executive order and they say
Speaker 1: that there was zero resistance from the labs. All the
Speaker 1: labs work more in the mode of saying, look, you
Speaker 1: need to be aware something's coming down the pike. It's
Speaker 1: very powerful. You need to get ahead of it. And
Speaker 1: when they did get ahead of it, or they started
Speaker 1: to at least move on it, there wasn't resistance. That's
Speaker 1: what the Biden people say, So I think it depends.
Speaker 1: It's a bit like with Google and deep Mind and
Speaker 1: the National Health Service in Britain. There can be good
Speaker 1: instances of technology company behavior. And again, if we paint
Speaker 1: this is true of anything, like if you paint the
Speaker 1: Chinese as uniformly bad, you can never collaborate with them
Speaker 1: on anything, and that's actually a bad outcome in itself.
Speaker 1: Same with the tech companies.
Speaker 2: Mentioned China in a US context and the group of
Speaker 2: companies that you're writing about in your book. I could
Speaker 2: totally see how you have a kind of circle of
Speaker 2: trust among those companies and shared incentives. But the thing
Speaker 2: that I think God in the way then and gets
Speaker 2: in the way often in practice, is this fear of
Speaker 2: China the outsider, which is not necessarily part of this system.
Speaker 2: And if you want to not be applying the breaks,
Speaker 2: if you want to not have these kind of control,
Speaker 2: all you need to say is what China is going
Speaker 2: to get ahead of us.
Speaker 1: Yeah. So what I think about that is that the
Speaker 1: West really needs to remember what happened in the Cold War.
Speaker 1: What happened in the Cold War in the nineteen fifties
Speaker 1: was you had simultaneously very moments of extreme tension, the
Speaker 1: Hungarian uprising, Soviet tanks going to Hungary. You have the
Speaker 1: Sewis Canal crisis at the same time, which prompts the
Speaker 1: Soviets to actually threaten the West with nuclear attack because
Speaker 1: of the Sewis intervention was viewed as imperiodists by the Russians.
Speaker 1: And simultaneously with those events you had the creation of
Speaker 1: the IAEA, the Institute for Atomic Energy and forgetting them
Speaker 1: but basically counting all the nuclear material and trying to
Speaker 1: prevent the nuclear material from going into loose nukes. Then
Speaker 1: in the sixties you get the Cuban missile crisis, enormously
Speaker 1: tense moment, and six years later you get the Nuclear
Speaker 1: Non Proliferation Treaty. So you can do what Reagan said,
Speaker 1: which is sort of trustpor verified, do arms control, even
Speaker 1: if there is competition with China. And when I went
Speaker 1: to China in March, because my book came out there
Speaker 1: first and spent eight days talking to AI leaders, both
Speaker 1: in academia and in the private sector. It was stunning
Speaker 1: to me how often they raised the subject of safety spontaneously.
Speaker 1: They would say, these agents are becoming powerful. We have
Speaker 1: a window of opportunity before they are too powerful to
Speaker 1: make them aligned with human incentives. The government should be
Speaker 1: warning people about dangers with these models. And in fact,
Speaker 1: while I was there, open Claw, this open source agent,
Speaker 1: which is not a good thing to put in your
Speaker 1: computer because it can do whatever it wants with all
Speaker 1: of the data that was being downloaded by Chinese mums
Speaker 1: and pups onto their laptops, and the government warned them
Speaker 1: not to. So the idea that China doesn't care about
Speaker 1: safety is wrong. There is a debate there, and they
Speaker 1: love regulating the Internet. That's what they're very good at
Speaker 1: in China. So why wouldn't we talk to them and say,
Speaker 1: look to probably compete on a sort of grand geostrategic
Speaker 1: military level, but we don't want this technology leaking out
Speaker 1: all over the world and being used by terrorists, criminals,
Speaker 1: nonstate actors to crash the web and do cyber hacking
Speaker 1: and build a bio weapon. Nobody wants that.
Speaker 2: You talked about the importance of trust earlier. I guess
Speaker 2: the question is have we got to a point with
Speaker 2: China where there is enough trust? I mean, the comparison
Speaker 2: with Russia is an interesting one, and that you could
Speaker 2: say in a sense that was a semi stable equilibrium.
Speaker 2: The trust to verify was one of the things that
Speaker 2: made it stable, but it was an established competition of superpowers.
Speaker 2: China is a much more in flux. We've had administrations
Speaker 2: have quite different approaches to controlling or competing or working
Speaker 2: with China. A lot of people will even listen to
Speaker 2: what you've said about your trip and say yes, they
Speaker 2: discourage their own people from using these things, just as
Speaker 2: they have much tougher controls on social media with their
Speaker 2: own population, and they don't tend to be so worried
Speaker 2: when they export all these products to the rest of
Speaker 2: the world. So do you think there's trust.
Speaker 1: That I think at a high level, what we should
Speaker 1: first get on the table is that the alternative to
Speaker 1: not even trying with China is that we just resign
Speaker 1: ourselves to having AI go viral all over the world
Speaker 1: with zero controls. The fact is China has cutting edge
Speaker 1: AI labs. There are only six months behind the frontier
Speaker 1: in the US. And if they get a really powerful
Speaker 1: model that can do cyber hacking, for example, and they
Speaker 1: put it in their systems, and their systems are open source,
Speaker 1: it's game over. This is just going to proliferate over
Speaker 1: all around the world. So that is a very bad outcome.
Speaker 1: So let's try and be creative about at least trying
Speaker 1: with China to talk to them. And I think the
Speaker 1: notion that it was easier in the Cold War not really.
Speaker 1: I mean, look, Krischev was this sort of mercurial, unpredictable,
Speaker 1: semi stable l leader, and the Russians would go to
Speaker 1: the UN and take their shoes off and bang them
Speaker 1: on the table. It was not an easy relationship in
Speaker 1: the sixties, right, and yet there was the nuclear non
Speaker 1: Proliferation Treatise. I really don't think we should give up.
Speaker 2: Okay. So the latest example we've had, and we've discussed
Speaker 2: it while back on Trumponomics, is mythos And there's obviously
Speaker 2: been a care in the way that it's been introduced,
Speaker 2: controls on who can have it. I know you feel
Speaker 2: strongly about the difference between the sort of safety differences
Speaker 2: between open source and closed source technologies, talk us through
Speaker 2: how some of the stuff we've just been talking about,
Speaker 2: how that's been illuminated or not by the example of Mythos. Yeah.
Speaker 1: So I think what's happened here is that whilst there
Speaker 1: was progress towards some AI safety regulation by governments in
Speaker 1: twenty twenty three and twenty twenty four, with national AI
Speaker 1: safety institutes being set up in multiple countries, a meeting
Speaker 1: of all those institutes in San Francisco in the fall
Speaker 1: of twenty twenty four, Once President Trump came into office,
Speaker 1: the whole system shifted towards acceleration, competition, and the momentum
Speaker 1: for regulation diet even though the technology itself was speeding ahead.
Speaker 1: So now we find ourselves in twenty twenty six and
Speaker 1: what we all knew was going to happen has happened.
Speaker 1: There is a model that, if in the wrong hands,
Speaker 1: could be used to basically steal everybody's money from their
Speaker 1: online bank accounts, crash internet systems, control dams, and so forth.
Speaker 1: Is a pretty scary prospect, and there is no government
Speaker 1: mechanism to stop that from leaking out, and there's no
Speaker 1: international mechanism because you need to do it internationally because
Speaker 1: this will be copied by the Chinese at some point,
Speaker 1: so it's a big problem. So Anthropics response to this
Speaker 1: is to say, there's no governmental governance, so we're going
Speaker 1: to do private governance. We Anthropic, a five year old
Speaker 1: startup in San Francisco, are going to arrogate unto ourselves,
Speaker 1: the power to say these are the forty entities that
Speaker 1: will get access to this technology, and the rest of you,
Speaker 1: cyber peons, you're outside the castle, outside our protection, and
Speaker 1: you can just stuff it. I don't think that's a
Speaker 1: sustainable position for a private company to be in, to
Speaker 1: be the arbiter of who is safe on the Internet
Speaker 1: and who isn't. But it's fort demurur. It's like there
Speaker 1: is no governance from the government, We're left with this
Speaker 1: private solution, which is not sustainable. So I think what
Speaker 1: this is a wake up call in terms of our
Speaker 1: earlier conversation is we don't have an alternative but to
Speaker 1: try and do something with China because they have these
Speaker 1: open source labs which are putting this AI into everybody's
Speaker 1: hands in a way that is totally not controlled. So
Speaker 1: it's not theoretical now that we might have an AI
Speaker 1: that's dangerous. We have one that's dangerous. So I think
Speaker 1: mythos is a wake up call, and looking back at
Speaker 1: the history of AI in my book, there are other
Speaker 1: moments when private labs thought that they could do governance themselves.
Speaker 1: Open AI thought that it had a kind of for profit,
Speaker 1: nonprofit capitalist hybrid that would make it all safe. It
Speaker 1: didn't work. Deep Mind tried the same thing with Google.
Speaker 1: It didn't work. Basically, individual labs just can't deliver governance.
Speaker 1: It needs to be government.
Speaker 2: If you think that the open source putting and putting
Speaker 2: open source technology into people's hands is probably the single
Speaker 2: most dangerous thing around at this level of technology, and
Speaker 2: China is still doing that despite, as you say, being
Speaker 2: so concerned about safety. Doesn't that suggest that there's the
Speaker 2: same dynamic operating in China that despite much talk and
Speaker 2: concern about safety, they are still hardwired to be moving ahead. Yeah.
Speaker 1: I think there's a couple of things there. I mean,
Speaker 1: one is that any lab that is not right at
Speaker 1: the frontier can't compete by telling customers we've got the
Speaker 1: best technology. So they compete by saying my technology is
Speaker 1: more available to you. You can download it into your
Speaker 1: server and do whatever you want with it. And that's
Speaker 1: what Meta does in the US their open source, that's
Speaker 1: what Mistra does in France they are open source. And
Speaker 1: that's what all the Chinese labs do because all of
Speaker 1: these players have in common the fact that they are
Speaker 1: not one of the big three anthropic open AI Google
Speaker 1: Deep Mind, and so this is just what followers do,
Speaker 1: and to stop them from doing that again you need
Speaker 1: some government action. And yes, the Chinese do have a
Speaker 1: sort of race mentality. They do want to build AI
Speaker 1: as fast as possible because they're frightened the falling behind
Speaker 1: the West, and that's just the normal, natural, obvious thing
Speaker 1: for them to do. In the face of an American
Speaker 1: leadership which is also racing, you're going to need both
Speaker 1: sides to slow down. But my point about the discussion
Speaker 1: in China, which I've i came in to touch with
Speaker 1: when I was there, is that there is at least
Speaker 1: a discussion and openness to think about safe. The door
Speaker 1: is opened, just a crack, and so it's up to
Speaker 1: American negotiators to go there and try and give it
Speaker 1: a nurge and open it more.
Speaker 2: It's very striking to me the conversations about AI safety,
Speaker 2: the Bletchley summit that Chally Soon I had, there was
Speaker 2: a lot of focus on killer robots, existential threats, the
Speaker 2: sort of much darker potential scenarios, the elimination of humanity,
Speaker 2: all of those things, And there was a general feeling
Speaker 2: in the industry and even among some policymakers that the
Speaker 2: focus on that was to some extent counterproductive because it
Speaker 2: was distracting from the sort of short term things that
Speaker 2: actually might cause us more harm and the impact of
Speaker 2: chatbots on children, all of those kind of things. It
Speaker 2: feels like we've now swung completely in favor of thinking
Speaker 2: about those things, those kind of social harms. Even today
Speaker 2: we had a devastating story about AI generated child porn
Speaker 2: and how that's making extremely difficult for investigators to to
Speaker 2: identify the true abuse of children and the footage and
Speaker 2: just well, you we get very depressed reading it. But
Speaker 2: sort of ironic to me that just as we've been
Speaker 2: focused on those social harms, actually those slightly more existential
Speaker 2: harms have become all the risks have become much greater.
Speaker 1: Yeah, I mean, I think that the existential risks are
Speaker 1: still there. And when I talked to Demissabis, as I
Speaker 1: did again this week, he still worries about that stuff
Speaker 1: both and by existential, what I really mean is two things.
Speaker 1: First is that bad guys get the technology and do
Speaker 1: something really catastrophic with it, build a bioweapon, do an
Speaker 1: enormous cyber attack which bloods a city by hacking a
Speaker 1: dam or something. Second, the machine itself goes rogue and
Speaker 1: starts attacking humans. And I always thought while I was
Speaker 1: writing this book that that second thing was science fiction nonsense.
Speaker 1: The machines will be more intelligent than humans, but they
Speaker 1: don't have a motive to attack humans because not evolved
Speaker 1: to survive. Unlike humans, we're always watching out for the
Speaker 1: lion on the savannah because we want to pass on
Speaker 1: our DNA. Machines don't have DNA. But then I had
Speaker 1: this rather disturbing conversation with the computer scientist Jeffrey Hinton.
Speaker 1: I went up to Toronto and sat in his kitchen
Speaker 1: for two hours, and he said to me, look, supposing
Speaker 1: you have an AI which is very powerful, but you're
Speaker 1: worried that the enemy AI will attack it. What are
Speaker 1: you going to do? You're going to tell your AI
Speaker 1: to defend itself if you see the attack coming, counterattack.
Speaker 1: Make sure you don't get killed. You've just given it
Speaker 1: a survival instinct. So now are you so comforted, Sebastian,
Speaker 1: do you think that AI won't have a survival instinct.
Speaker 1: I think this existential stuff is real. It can be
Speaker 1: addressed by engineering what they call alignment into the models,
Speaker 1: where you're basically giving instructions to the system not to
Speaker 1: turn against humans. And that's an engineering challenge which I
Speaker 1: think properly is solvable, but you need time to solve it.
Speaker 1: And the thing about alignment is that what we really
Speaker 1: care about is aligning not necessarily today's model, but the
Speaker 1: the tougher, more dangerous, more powerful one which we might
Speaker 1: have in six months or a year. And you can't
Speaker 1: invent the alignment of the future model until you've got
Speaker 1: it in the lab and now you're trying to align it.
Speaker 1: So the faster that progress goes, the less likely we
Speaker 1: are to do alignment. And that's another reason why we
Speaker 1: need government controls.
Speaker 2: The government we have currently, as you pointed out, the
Speaker 2: US administration, and obviously it's the US government that's going
Speaker 2: to have the biggest influence by definition on this at
Speaker 2: the moment, is one that is not minded towards safety,
Speaker 2: is very focused on the arms race aspect with China.
Speaker 2: Given what you've been saying about the pace of change,
Speaker 2: the fact that we already have potentially enormously damaging instruments
Speaker 2: that could be tomorrow in the wrong people's hands, is
Speaker 2: it going to really matter a lot to all of
Speaker 2: us that this administration, with those instincts was in power
Speaker 2: for at least the next two Yeah.
Speaker 1: I think there's a real danger that will turn out
Speaker 1: to be decisive on the upside. It's interesting that with Mythos,
Speaker 1: the Treasury Secretory, Scott Vesant, immediately worried that this was
Speaker 1: going to be catastrophic for the stability of the banking
Speaker 1: system because bank accounts would be hacked, and he called
Speaker 1: in the heads of the big banks and told them
Speaker 1: to take it seriously. And he apparently inside the administration
Speaker 1: favored a nationalization of Mythos because he thought it was
Speaker 1: so scary. So I think it's possible that as these
Speaker 1: models become more threatening, the Trump administration, which is flexible,
Speaker 1: one could say, might do the kind of u turn.
Speaker 1: It's done on the notion that you shouldn't intervene in
Speaker 1: other countries and start wars. It clearly forgot that principle,
Speaker 1: and so perhaps it will change on AI as well.
Speaker 2: Even the instincts that we're seeing on that seem to
Speaker 2: be much more about protecting American banks and maybe UK banks.
Speaker 2: But to your point about Anthropic really controlling who has this,
Speaker 2: is it really realistic that you can be just picking
Speaker 2: and choosing countries that get this defense.
Speaker 1: I didn't think it's realistic for Anthropic to continue to
Speaker 1: do this. Therefore, I think we have to hope that
Speaker 1: the sheer evidence of the disruptive power of the models
Speaker 1: eventually gets the US government to change his position, and
Speaker 1: then we could see a sort of norm cascade where
Speaker 1: other countries also change.
Speaker 2: Well, you say in your book that you start off
Speaker 2: as an optimist on these things, and I hope that
Speaker 2: you've still managed to sustain a little bit of optimism.
Speaker 2: But Sebastian, thank you so much.
Speaker 1: So nice to be with you.
Speaker 2: Stephanie, thanks for listening to Trumponomics from Bloomberg. It was
Speaker 2: hosted by me Stephanie Flanders. I was joined by Sebastian Mallaby,
Speaker 2: Senior Fellow at the Council and Foreign Relations and author
Speaker 2: of The Infinity Machine. DeNisi Sibi's Deep Mind and the
Speaker 2: Quest for super Intelligence. Trumponomics was produced by Samasai and
Speaker 2: Moses and with help from Amy Keane, and sound design
Speaker 2: was by Blake Maples and Kelly Garrett. To help others
Speaker 2: find this show and enjoy it, please rate and review
Speaker 2: it highly wherever you listen.