Speaker 1: Bloomberg Audio Studios, podcasts, radio news.
Speaker 2: At some point over the next twenty or thirty years,
Speaker 2: machines are going to be more capable than humans are
Speaker 2: doing most work, and so we have to decide as
Speaker 2: a society what our purpose is.
Speaker 1: What's the first solemn? The AI founder who is now
Speaker 1: at the forefront of a global race.
Speaker 2: Getting the balance right is very important. We have to
Speaker 2: deliver in the next few years. We're taking a huge
Speaker 2: bet that we're going to be able to convert this
Speaker 2: into true intelligence, and if we do, then I think
Speaker 2: the world is going to look very, very different.
Speaker 3: We will have abundant intelligence. On tap.
Speaker 1: From Bloomberg Weekend. This is the Michelle Hussein Show. I'm
Speaker 1: Michelle Hussein. Back in twenty sixteen, a computer beat a
Speaker 1: world champion in the ancient game Go. You might remember
Speaker 1: it because it certainly made the news of the time.
Speaker 1: It was a huge milestone artificial intelligence, which few of
Speaker 1: us had ever really heard about, was uncovering strategies that
Speaker 1: hadn't been seen over centuries of human players. One of
Speaker 1: those who made it happen at the startup Deep Mind,
Speaker 1: is now leading AI for Microsoft, but Mustapha Suleiman is
Speaker 1: an unusual figure in big tech. He's British for starters,
Speaker 1: He's interested in technology and public services. He used to
Speaker 1: work in local government in the UK. He's seen climate
Speaker 1: talks up close from the inside. Today though his work
Speaker 1: is at the heart of an AI arms race, the
Speaker 1: big names spending huge sums and making all sorts of
Speaker 1: promises on what it can do and when, which is
Speaker 1: why I'd say he's really worth a listen, not only
Speaker 1: to understand what's happening right now in this crucial field,
Speaker 1: but also to understand one of its most influential players.
Speaker 1: So here's my conversation with mustafas Leman, starting with his
Speaker 1: own everyday use of AI.
Speaker 2: Yesterday night, I stayed up far too late watching a film,
Speaker 2: and afterwards I added to a table that I've made
Speaker 2: in co pilot, which basically records all the films that
Speaker 2: I love, films that I want to watch, lists them
Speaker 2: by date, describes what they were about. I add my
Speaker 2: personal notes to it, gives me a link to the
Speaker 2: film poster, and I can keep just saying, okay, to
Speaker 2: add this film that I want to watch, what would
Speaker 2: be a similar one that I want to watch?
Speaker 3: So, just like you might ask.
Speaker 2: An assistant to organize your life, and I think that
Speaker 2: people will be surprised. The more obscure, the more creative,
Speaker 2: the more challenging you think the task is that you're
Speaker 2: going to ask your AI or co pilot, the better.
Speaker 1: But have you used it to do autonomous tasks? So,
Speaker 1: for example, has it booked tickets for you or bought
Speaker 1: a gift for you? Because I know this is one
Speaker 1: of the promises of co pilot, It's just not available
Speaker 1: in my region, so I haven't been able to try
Speaker 1: it out for myself yet.
Speaker 3: We're still experimenting with it.
Speaker 2: It can do it, it doesn't always get it right,
Speaker 2: so it's in like a sort of dev mode, so
Speaker 2: it's not like generally available just yet. But when it
Speaker 2: does work, it is the most magical thing you've ever seen.
Speaker 1: Okay, what are the mistakes it's made that have created
Speaker 1: problems for you? Bord a present for the wrong person, Well.
Speaker 2: It can buy the wrong thing, but obviously you can
Speaker 2: intervene and it will always ask you permission before it
Speaker 2: takes the next action and stuff, so it's it's quite safe.
Speaker 2: It's one of those things with technology, you know it
Speaker 2: promises so much and it's always sort of just around
Speaker 2: the corner.
Speaker 3: I remember, like the allure of.
Speaker 2: Getting a discman back in the nineties, and I had
Speaker 2: this dream that I was going to be roll blading
Speaker 2: down the street, and then I got one, and obviously
Speaker 2: it didn't have any kind of vibration proof shop proof
Speaker 2: technology yet, and so it would skip every time, like
Speaker 2: I turned it on its side or tried to put
Speaker 2: it in my back pocket.
Speaker 3: And so it's a funny thing. Technology.
Speaker 2: It's still magical and amazing, but it's always just got
Speaker 2: a little bit further to go. And I think in
Speaker 2: this case, it's a while yet before it's every day.
Speaker 1: And I guess that given everything you've seen, you're one
Speaker 1: of the founders of deep Mind, and you've worked at
Speaker 1: Google and had your own AI company inflection before you
Speaker 1: came to Microsoft. Does that mean that when you see
Speaker 1: these hiccups or these stages of development, you have faith.
Speaker 1: You don't get frustrated because you remember the moment when
Speaker 1: back in twenty sixteen when your AI model beat the
Speaker 1: world champion at go, which was a huge moment for AI.
Speaker 2: I think I'm very stoic about these things, like I
Speaker 2: know that it's going to work in the next six
Speaker 2: months or twelve months, or maybe the worst case, eighteen months.
Speaker 2: But you know, I think we've got a pretty good
Speaker 2: rhythm now of picking things which are not too far
Speaker 2: away but are genuinely transformative. And also, you know, it's
Speaker 2: important to remember how magical it is today. It can
Speaker 2: generate images, it can generate videos. It is already you know,
Speaker 2: sort of superhuman magical, And I think I'm just one
Speaker 2: of those people that's still inspired by what it can
Speaker 2: do at any given moment.
Speaker 1: By this time next year, could I be buying my
Speaker 1: Christmas presents using an autonomous AI agent.
Speaker 2: I'm pretty certain that you will be. Yeah, I think
Speaker 2: it's highly highly likely.
Speaker 1: So, speaking of magical things, the recent announcement that you
Speaker 1: made at Microsoft about superintelligence. That's the term which has
Speaker 1: really crept into the public debate thanks to you and
Speaker 1: others in the last few months. When you use the
Speaker 1: term superintelligence, what does it mean to you?
Speaker 2: Super intelligent in the industry today means an AI system
Speaker 2: that can learn any new task and perform better than
Speaker 2: all human combined at all tasks.
Speaker 3: So it is a very.
Speaker 2: High bar and at the moment it comes with a
Speaker 2: great deal of risk. It is not clear. In fact,
Speaker 2: it's very uncertain how we would limit its capabilities and
Speaker 2: align to get a system like that to care a
Speaker 2: lot about humans and want us to succeed and flourish.
Speaker 2: It's very unclear how we would contain and aligned a
Speaker 2: system that is so much more powerful than us. And
Speaker 2: so the framing that I prefer is one of a
Speaker 2: humanist superintelligence, one that is always in our corner, on
Speaker 2: our team, aligned to human interests, and until we can
Speaker 2: prove that it will remain safe, we won't continue to
Speaker 2: develop a system that has the potential to run away
Speaker 2: from us and to get out the box.
Speaker 3: And I think that that's an obvious thing to say.
Speaker 2: Everybody should agree to that, and yet I think it's
Speaker 2: kind of a novel position in the industry.
Speaker 3: At the moment.
Speaker 1: So is that how you're trying to set Microsoft apart?
Speaker 1: A lot of people are working on superintelligence. Are you
Speaker 1: trying to make my SELT distinctive by saying we will
Speaker 1: always use it through a humanist lens?
Speaker 2: Yes, that is our position. You know, Microsoft is a
Speaker 2: company that's been around for fifty years. It is very careful,
Speaker 2: it's highly trusted. Ninety percent of the s and P
Speaker 2: five hundred use us to provide email and operating systems
Speaker 2: and everyday productivity tools which are now among the best
Speaker 2: in the world with co Pilot, And you know, I
Speaker 2: think that we've got that reputation because the company has
Speaker 2: been careful and we're going to continue to be careful,
Speaker 2: and setting out a vision of humanist superintelligences is part
Speaker 2: of that program.
Speaker 1: So what does that mean for your rivals in that field,
Speaker 1: some of whom you work closely with, like open AI,
Speaker 1: which I'm sure we'll talk more about. But does it
Speaker 1: mean that they're the wild West and you're the moral ones.
Speaker 2: I think that everybody has to decide what they stand
Speaker 2: for and how they operate, and I don't I don't
Speaker 2: want to judge how they're operating right now. I don't
Speaker 2: see any evidence of large scale mass harm. I don't
Speaker 2: see any indication that these things are improving themselves or
Speaker 2: operating autonomously. But we all predict a time in the
Speaker 2: next five years, maybe in the next ten years, where
Speaker 2: these capabilities do start to emerge and systems like this
Speaker 2: could set their own goals, or they could improve their
Speaker 2: own code, or they could act autonomously. Those things are
Speaker 2: capabilities that I've clearly outlined as increasing the level of risk,
Speaker 2: and therefore we have to approach them with caution, with
Speaker 2: more transparency and audit, with more government engagement, and make
Speaker 2: proactive declarations about how close we are to those three capabilities.
Speaker 1: And does that mean that you won't be releasing a
Speaker 1: superintelligence tool until you're confident that it can be controlled.
Speaker 2: Yeah, I think that's right. I mean, I think for
Speaker 2: sure the containment and alignment unnecessary prerequisites their red lines,
Speaker 2: and I think everybody in the industry has to sign
Speaker 2: up to that idea. I mean, nobody wants to cause
Speaker 2: mass harmon fundamentally, even though we all disagree. Clearly, everybody
Speaker 2: is committed to the survival of our species and I
Speaker 2: would hope the kind of flourishing and well being of everybody.
Speaker 2: So that's the discussion that we're trying to push now
Speaker 2: and require everybody to ask themselves in the industry, are
Speaker 2: they building a humanist superintelligence? Does it really stand for
Speaker 2: people and for our species?
Speaker 3: First?
Speaker 1: Yeah, I'm just wondering how much confidence we can really
Speaker 1: have in that given that things do go wrong and
Speaker 1: I don't just mean AI, I mean humans do wrong. Things.
Speaker 1: All things just go wrong for other reasons. And the
Speaker 1: recent example that Microsoft had to deal with was having
Speaker 1: to cease and disable some services that were being used
Speaker 1: by the Israeli Ministry of Defense after reporting by The
Speaker 1: Guardian that suggested they could be being used for mass surveillance.
Speaker 3: That was very good reporting by the Guardian.
Speaker 2: We were very grateful for it, and as soon as
Speaker 2: we became aware of it, we actually made all the
Speaker 2: necessary changes, removed the IDF from those servers and I think,
Speaker 2: you know, they were clearly not in compliance with our
Speaker 2: terms of service, and there's an ongoing investigation internally.
Speaker 1: But I guess the broader point is that it's hard
Speaker 1: to have confidence of controls and checks and balances and uses.
Speaker 3: It is hard.
Speaker 2: I mean, these are huge and complicated systems that they
Speaker 2: carry a lot of risk, and the most that we
Speaker 2: can do is make sure that we are auditing them
Speaker 2: and we're removing actors that violate our terms of service
Speaker 2: as quickly as possible.
Speaker 1: So are the first uses of superintelligence going to be
Speaker 1: in the medical field?
Speaker 3: I think so.
Speaker 2: I think that this is probably the most exciting application
Speaker 2: of superintelligence. I think we now have systems that can
Speaker 2: diagnose any rare condition that is found in the literature
Speaker 2: significantly better than human performance, and do so with fewer
Speaker 2: tests and interventions. And that's the key thing. It's like,
Speaker 2: often it's actually quite expensive to run medical intervention tests
Speaker 2: before you get to a diagnosis, and often they can
Speaker 2: be unnecessary to and they take time, and then in
Speaker 2: that process, you know the condition came off wor in
Speaker 2: our case, our system does this significantly more cheaply, with
Speaker 2: fewer tests and with higher accuracy.
Speaker 3: We're just putting it.
Speaker 2: Through independent peer review at the moment, and soon there'll
Speaker 2: be clinical trials of it too.
Speaker 3: So I think this is very, very, very exciting.
Speaker 1: And did you push for that focus yourself, because you've
Speaker 1: been running Microsoft AI for a while, about eighteen months now.
Speaker 1: Super Intelligence was a big recent announcement, and I'm conscious
Speaker 1: as I hear you talk about it that your mother
Speaker 1: was a nurse and you, unusually for senior people in tech,
Speaker 1: you've worked in the public sector as well.
Speaker 3: Yeah, I mean this is an area that's very important
Speaker 3: to me.
Speaker 2: Obviously, my mum was in an HS nurse and you know,
Speaker 2: I'm just a big believer that technology is here to
Speaker 2: serve us. It should make our lives better, should make
Speaker 2: us more comfortable one day. I think it is going
Speaker 2: to help us to live longer, give us the option
Speaker 2: to work less if we choose to. I do think
Speaker 2: it's going to produce abundance, and I think that we
Speaker 2: have to make conscious decisions to use it for those
Speaker 2: applications first, and that that's why, you know, even back
Speaker 2: at Deep Mind, I ran our health unit, and you know,
Speaker 2: I'm very passionate about this.
Speaker 1: Abundance is such a big promise. Tell me what you
Speaker 1: mean by it, because most of the time we hear
Speaker 1: about AI destroying jobs. But are you saying that the
Speaker 1: work will be done by AI increasingly and therefore we
Speaker 1: humans won't have to work as much.
Speaker 2: I mean, I think that it is inevitable that at
Speaker 2: some point over the next twenty or thirty years, machines
Speaker 2: are going to be more capable than humans are doing
Speaker 2: most work, and that might come much sooner. And so
Speaker 2: we have to decide as a society what our purpose is,
Speaker 2: and we have to be very thoughtful about the rate
Speaker 2: of introduction of new machines because we have to make
Speaker 2: sure that that rate of displacement is counterbalanced with a
Speaker 2: mechanism to fund people and to support people through a
Speaker 2: massive transition, which for which there is probably not going
Speaker 2: to be other jobs on the other side that are
Speaker 2: more competitive than what a machine can do. And so
Speaker 2: this is a real big step change in what it
Speaker 2: is to organize society.
Speaker 1: But do you believe in the idea of a universal
Speaker 1: basic income that that's what AI being very productive making
Speaker 1: the economy more productive could unlock.
Speaker 2: Yeah, I mean, I think I've long been on record
Speaker 2: saying that I think that is both inevitable and very desirable.
Speaker 2: We already live in a world of abundance, it's just
Speaker 2: poorly distributed. We already have more calories to consume, we
Speaker 2: have an incredible amount of food wastage.
Speaker 3: It's just unevenly distributed.
Speaker 2: And now that value isn't just manifested in atoms like
Speaker 2: food and cars and physical things. It's manifested in digital
Speaker 2: goods ideas, knowledge intelligence. And that's actually great news because
Speaker 2: that can proliferate, it can spread extremely quickly around the
Speaker 2: entire world. I mean, these llms and chatbots have been
Speaker 2: the fastest spreading technology in history, basically two billion and
Speaker 2: a users in the space of three years, and that's
Speaker 2: actually amazing because it's delivering knowledge to every single person
Speaker 2: who wants to use it on their phone in the
Speaker 2: current sort of format and device they already use. So
Speaker 2: I think that we have no problem with getting technology
Speaker 2: out and available to everybody because there's going to be
Speaker 2: massive competitive forces to reduce the cost of experiencing an AI.
Speaker 2: The challenge we're going to have to figure out is
Speaker 2: how we tax and redistribute so that the transition is
Speaker 2: a healthy one.
Speaker 1: Tell me more about where these ideas and your beliefs
Speaker 1: come from. Your mother was a nurse. But the family
Speaker 1: you grew up in in London in the nineteen eighties
Speaker 1: and nineteen nineties, how would you describe it?
Speaker 3: Preally working class there. My dad was a mini cab driver,
Speaker 3: my mom works in.
Speaker 2: HS and yeah, I mean we were fairly regular, like
Speaker 2: kind of unremarkable. My parents didn't supervalue education stuff. They
Speaker 2: always tho actually go get trade and my mom would
Speaker 2: often say to me, you know, you should be a
Speaker 2: carpenter electrician, you know, leave scot sixteen and stuff. So
Speaker 2: I don't know, it comes from a place of just
Speaker 2: experiencing the kind of rougher end of things a little
Speaker 2: bit and then having just a desire to try to
Speaker 2: do the best that we can do with the short
Speaker 2: life that we have.
Speaker 1: Your dad had come to the UK from Syria. Did
Speaker 1: your parents meet in England?
Speaker 3: Yes, yeah, that's right.
Speaker 1: And what happened when you were sixteen must have had
Speaker 1: because I read that when your parents split up, you
Speaker 1: and your brother, your younger brother, were pretty much left
Speaker 1: to fend for themselves.
Speaker 3: Ah, that's true.
Speaker 2: Yeah, I don't know where you've read that, but yeah,
Speaker 2: me and my younger brother did live on our own
Speaker 2: for a few years.
Speaker 3: Yeah.
Speaker 1: Yeah, I think you mentioned this in a Guardian interview
Speaker 1: a couple of years ago when your book was coming out,
Speaker 1: But I'm really curious to know more about what it
Speaker 1: taught you and how you dealt with it at the time.
Speaker 2: I think that when you're that age, you're just sort
Speaker 2: of precocious and overconfident and fearless. We had everything we
Speaker 2: needed and it worked out pretty well.
Speaker 1: And this isn't a crucial time the last couple of
Speaker 1: years of school, and you did do well in those
Speaker 1: last couple of years of school because you ended up
Speaker 1: getting to Oxford.
Speaker 3: Yeah.
Speaker 2: I was very lucky because I had great teachers and
Speaker 2: good mentors, and I went to a very good school
Speaker 2: which was far from my home, but you could do
Speaker 2: exams to get in, so I studied really hard for
Speaker 2: when I was ten years old for the entrance exams,
Speaker 2: and it was essentially like going to a private school.
Speaker 2: Basically everyone was like very studious and got great grades
Speaker 2: and stuff. And yeah, I was lucky enough to get
Speaker 2: into Oxford, and you know, that was an amazing experience.
Speaker 1: Have you thought about what your life would have been
Speaker 1: like in that period in your mid teens had it
Speaker 1: not been for that school.
Speaker 3: Yeah, I haven't thought much about that. It would have
Speaker 3: been pretty different.
Speaker 1: And then you got to Oxford and you did actually
Speaker 1: subject pretty close to what my son is doing there
Speaker 1: right now. He's doing theology. You did philosophy and theology,
Speaker 1: and then tragedy cle to my ears as a parent,
Speaker 1: you dropped out.
Speaker 2: Yeah, well, I was very frustrated and eager to change
Speaker 2: the world and get stuff done, and so I actually
Speaker 2: dropped out. And I helped to start a telephone counseling
Speaker 2: service for young British Muslims called Muslim Youth Helpline, which
Speaker 2: I worked on for two or three years as a volunteer.
Speaker 2: And it was a secular peer to peer listing service
Speaker 2: for young British Muslims that post nine to eleven were
Speaker 2: dealing with a lot of I guess identity crisis would
Speaker 2: be the way that we would describe it in hindsight,
Speaker 2: but at the time it was just like, you know,
Speaker 2: lack of connection to community, family, parents, a lot of
Speaker 2: bullying and so yeah, I worked on that pretty much
Speaker 2: full time for two or three years.
Speaker 1: And had you experienced anti Muslim sentiment and hatred yourself
Speaker 1: at that time given the period we're talking about after
Speaker 1: nine to eleven.
Speaker 2: A little bit, Yeah, I think it was more just
Speaker 2: not so much for me, but I think a lot
Speaker 2: of people felt that, you know, we weren't sort of
Speaker 2: British enough, and people were figuring out how to both
Speaker 2: live their cultural religious identities in the context of families
Speaker 2: that often were first generation, didn't really speak the language,
Speaker 2: didn't know how to navigate the system, and then this
Speaker 2: increased kind of skepticism that we were sort of mostly
Speaker 2: terrorists or whatever.
Speaker 3: And I think that was just this general.
Speaker 2: Kind of fear and exclusion, and actually most of the
Speaker 2: time that was dealt with by just having kind and
Speaker 2: supportive conversation and having somebody be available on the other
Speaker 2: end of a telephone.
Speaker 1: Yeah, someone to talk to. I know the work of
Speaker 1: the helpline actually, so I know exactly the world that
Speaker 1: you're talking about. Did you think hard about dropping out
Speaker 1: of university and have you regretted it at any point?
Speaker 2: Since I haven't regretted it, and I probably could have
Speaker 2: thought harder about it. My tutor did encourage me to
Speaker 2: like switch subjects or change college and actually had me
Speaker 2: go meet one of his friends in a different college
Speaker 2: to go and talk about doing history or PP. But
Speaker 2: that point I was just so full of energy and
Speaker 2: like eager to start things. I started a business at
Speaker 2: the same time, sort of PDAs into restaurants to try
Speaker 2: and do restaurant booking and stuff and doing their networking.
Speaker 3: And so on.
Speaker 2: It was our very early days of the internet, so
Speaker 2: it was a bit early, but I did that for
Speaker 2: about a year or two as well. I had too
Speaker 2: much energy, I think for sitting around thinking in the
Speaker 2: dreamy spires.
Speaker 1: Now Here you are in this circle of power, right,
Speaker 1: one of a small group of people who are making
Speaker 1: decisions on a technology that is already changing all our
Speaker 1: lives and will continue to do. So, how conscious are
Speaker 1: you of that power?
Speaker 3: Very conscious?
Speaker 2: I take it very seriously and it's a great responsibility.
Speaker 2: And you know, I think that this is a moment
Speaker 2: when decisions that we make may have very lasting consequences.
Speaker 2: And I read history and I look back at times,
Speaker 2: for example, in the social media revolution, where I think
Speaker 2: it was very clear that there were potential harms that
Speaker 2: maybe fell on deaf fears for too long. Or I
Speaker 2: look back at you know, the electric car revolution that
Speaker 2: could have taken off way back in the twenties, or
Speaker 2: you know, you look at kind of smoking or oil.
Speaker 2: So I'm very sensitive to those things because it's very
Speaker 2: clear that these things will cause harm, and I think
Speaker 2: that we have to be very very careful about how
Speaker 2: we deploy them and how they're introduced into the world.
Speaker 1: Do you talk about this with your peers, with people
Speaker 1: like Sam Altman, who I know you're close to.
Speaker 2: Yes, I mean, I think everybody in the industry does.
Speaker 2: And there's definitely a group of the CEOs. Obviously, Sam
Speaker 2: and Dario from Anthropic were co founders, Me and Demis
Speaker 2: were co founders, so we all know each other very
Speaker 2: very well. And I think on the whole everybody is
Speaker 2: genuinely committed to trying to find the right path through.
Speaker 2: It's also very competitive, so there's obviously a lot of
Speaker 2: that dynamic too.
Speaker 1: Yeah. Do you recognize the term brolligarchy.
Speaker 2: I haven't heard that before, but I can figure out
Speaker 2: what it means. Yeah, I guess that's true. And it
Speaker 2: is very male centric, although it should be said Mira Marati,
Speaker 2: who's the XCTO of Opening Eyes, one of the best
Speaker 2: people in the field.
Speaker 1: Or tech bros. I'm sure you've heard that one. I
Speaker 1: don't know if you've seen Jesse Armstrong's Mountainhead, have you?
Speaker 3: No? I haven't. No.
Speaker 1: It is it is all about a group of people,
Speaker 1: one of whom is clearly Elon Musk, another one who
Speaker 1: looks like Peter Teal or sounds like Peter Teal. And
Speaker 1: it's you know, it's quite a sobering portrait. His impression
Speaker 1: of life in this quite rarefied, very powerful circle.
Speaker 2: Yeah, and I think part of the challenge is that
Speaker 2: we all spend a lot of time in Silicon Valley,
Speaker 2: where the skies are blue, and you know, life is
Speaker 2: very peaceful. I try to travel a lot. I just
Speaker 2: came back from China, a couple of weeks ago, whereas
Speaker 2: in five or six difference season. It's just staggering to
Speaker 2: get out of the bubble and see how this technology
Speaker 2: is being developed on the other side of the world.
Speaker 2: And you know, both the pace of innovation, but also
Speaker 2: you know, I think the thoughtfulness of some of the
Speaker 2: regulatory stuff. I mean, I think they are definitely thinking
Speaker 2: pretty carefully about it as well, So it's impressive. I
Speaker 2: also come back to London a lot and make sure
Speaker 2: that I spend time outside of Silicon Valley.
Speaker 1: So if you have a different impression to someone like
Speaker 1: Jesse Armstrong or others about your world, can I ask
Speaker 1: you just to because you know these key players, can
Speaker 1: you describe some of them in one word? Like if
Speaker 1: I say to Sam Ortman, what word first comes into
Speaker 1: your mind?
Speaker 4: Oh my god, Yeah, I guess courageous, because well, you know,
Speaker 4: he's obviously sort of growing his data center fleet very
Speaker 4: very aggressively.
Speaker 2: He may well turn out to be one of the
Speaker 2: great entrepreneurs of our generation. He's certainly achieved a lot,
Speaker 2: and I think now he's building data centers at a
Speaker 2: faster rate than I think anyone in the industry.
Speaker 3: And if he can pull it off. It will be
Speaker 3: pretty dramatic.
Speaker 1: It's interesting that there is an IF on that, and
Speaker 1: I understand why there are huge amounts of money being
Speaker 1: spent by open Ai. Is it a gamble? Is it
Speaker 1: not a given that it's going to pay off for them?
Speaker 2: Well, I think that chat GPT is one of the
Speaker 2: greatest products that we've seen in a generation, and so
Speaker 2: that speaks for itself. At the same time, I guess
Speaker 2: they've signed over a trillion and a half dollars of
Speaker 2: commitments for building data centers over the next five or
Speaker 2: ten years. Their revenues are quite a long way from there,
Speaker 2: so they've got a long way to go. But they're
Speaker 2: very talented teams, so I've got every confidence they can
Speaker 2: do it.
Speaker 1: Okay, So other descriptions, what word would use to describe
Speaker 1: demis Hesavis?
Speaker 3: Yeah, probably a great scientist.
Speaker 2: I mean, I think he's a great thinker and he's
Speaker 2: a good polymath. He's made massive contributions in the field
Speaker 2: multiple times, and so I think he's truly exceptional.
Speaker 1: Together and now your competitors.
Speaker 2: Yeah, but we started off as very very close friends.
Speaker 2: We work together every day for ten years and I
Speaker 2: learned a lot from him, and have huge respect for him.
Speaker 1: Do you still talk?
Speaker 3: Yeah, we texted last night.
Speaker 2: Actually I congratulated him on the Nano banana, Gemini three
Speaker 2: and five years of Alpha fold all in one week.
Speaker 1: Have you tried out Gemini three because it is creating
Speaker 1: quite a lot of waves. Do you think it is
Speaker 1: better than chat GPT?
Speaker 3: Ah, you know, they're kind of different.
Speaker 2: It's definitely got more kind of niche skills that chat
Speaker 2: GPT doesn't have, and it's very fast. CHURCHIBT is very strong,
Speaker 2: So I wouldn't I wouldn't go that far.
Speaker 1: Okay, is it better than Copilot?
Speaker 2: It can do things that Copilot can't do. I mean
Speaker 2: Copilot has features that it doesn't have, Like Copilot's actually
Speaker 2: amazing for vision. One of my favorite features is like
Speaker 2: pull out your phone and talk about the world with
Speaker 2: co Pilot in your hand, and it can see exactly
Speaker 2: what you're looking at and you can talk about it
Speaker 2: and get feedback opinions.
Speaker 3: You might be shopping for furniture or clothes.
Speaker 2: Yes, we're putting the capabilities, but we're really trying to
Speaker 2: imagine what is the day to day experience of having
Speaker 2: this really intelligent assistant at your side that can talk
Speaker 2: to you about the world and help you sort of
Speaker 2: help unblock you whenever you get stuck with something.
Speaker 1: Okay, I'm only going to do one more of these,
Speaker 1: but it's Elon Musk. How would you describe him?
Speaker 3: I guess as a bulldozer.
Speaker 2: He's kind of got superhuman capabilities to bend reality to
Speaker 2: his will and has, you know, pretty incredible track record
Speaker 2: and somehow he sort of mostly manages to pull off
Speaker 2: what appears to be impossible. But yeah, probably different kind
Speaker 2: of set of values.
Speaker 1: I have interviewed him. He ended up calling me an NPC.
Speaker 1: I guess that was probably a form of bulldozing.
Speaker 3: Yeah, I mean that's probably a good example.
Speaker 2: That sounds like exactly Elon. I kind of like that
Speaker 2: he speaks his mind. I mean, he's very unfiltered.
Speaker 1: I think you've said that your politics are different from him,
Speaker 1: and Peter Thiel's that the libertarian you know, not much
Speaker 1: value to the nation state, etc. Are it's not your world.
Speaker 1: Would you call yourself as being on the left of
Speaker 1: the political spectrum or is there a different description you'd use.
Speaker 2: I would say I'm sort of a centrist these days.
Speaker 2: I definitely started as a lefty, but yeah, I'm proud
Speaker 2: to say that I'm on the kind of center left
Speaker 2: of the spectrum. I believe that government plays an important
Speaker 2: role in society. It's a controversial thing to say in
Speaker 2: Silicon Valley, but I think regulation is necessary and it
Speaker 2: has made most technologies better. And people forget this, like
Speaker 2: cars only work because we have driver training and emissions
Speaker 2: regulations and street lights and motorway speed limits and so
Speaker 2: on and so forth, And so that's what regulation is
Speaker 2: when it works well, and we just need more of that.
Speaker 1: Do you feel isolated saying that, because clearly the Trump
Speaker 1: administration is not into regulation and the industry overall is
Speaker 1: quite happy with that.
Speaker 3: Yeah.
Speaker 2: I think that right now we are not in a
Speaker 2: mode where there.
Speaker 3: Is huge catastrophic harm.
Speaker 2: The industry has done a pretty good job of introducing
Speaker 2: very powerful chatbots in a way where the personality is
Speaker 2: sculpted to be very even handed, to be very evidence based.
Speaker 2: I think that it didn't have to go like that.
Speaker 2: It went like that, I think because the leaders have
Speaker 2: done a pretty decent job. Now, that doesn't mean that
Speaker 2: there aren't going to be issues coming up, and I'm
Speaker 2: obviously very wide eyed about them, but I don't think
Speaker 2: we're in a mode where we need emergency regulation. For example,
Speaker 2: some of the regulations that have been proposed in Europe
Speaker 2: are currently in talks to be wound back with the
Speaker 2: EUAI Act, And people shouldn't criticize that. That's like the
Speaker 2: process working. That's the regulator taking feedback, see how things
Speaker 2: work in practice. That should be celebrated. But at the
Speaker 2: same time, they have to be able to have the
Speaker 2: freedom to experiment and try new regulations. And I think
Speaker 2: that's a good thing.
Speaker 1: Do you think Mustapha? You see some of these issues
Speaker 1: differently now that you are right at the heart of
Speaker 1: one of the biggest companies of all because in your
Speaker 1: book The Coming Wave, and in a piece you wrote
Speaker 1: about the same time for Foreign Affairs, you were quite
Speaker 1: clear on how much governance was needed. In fact, you
Speaker 1: said you wanted three different kinds of governance, regimes modeled
Speaker 1: on climate change and financial stability, and arms controlled that
Speaker 1: AI and tech could learn from all three of those.
Speaker 1: And yet you sound much more sanguine now. Is it
Speaker 1: just because of where you are.
Speaker 3: I'm still calling for those things.
Speaker 2: We should have a financial stability board for AI, we
Speaker 2: should have a climate process and I can which basically
Speaker 2: goes and audits the progress that we're making. I advocated
Speaker 2: for the White House AI Principles under the Biden administration,
Speaker 2: where we made voluntary commitments as companies. At the time,
Speaker 2: it was my startup inflection, but we pushed all the
Speaker 2: other companies to do it as well, where you disclose
Speaker 2: the size of your model. We've signed up to those
Speaker 2: disclosures in Europe and with the UKAI Safety Institute. So yeah, yeah,
Speaker 2: I don't mean to say that we don't need it.
Speaker 2: I'm saying that these are such long term effects, and
Speaker 2: these governance processes take so long to build that we've
Speaker 2: basically got to start now. But what we don't need
Speaker 2: is knee jerk reaction. We don't need some panicked overreaction
Speaker 2: because that would cause a different set of.
Speaker 1: Problems the superintelligence work you're doing. And indeed, you know
Speaker 1: everything that you're doing in AI in the moment, You're
Speaker 1: in a new phase, aren't you, because now that you've
Speaker 1: got a revised agreement with open AI, you suddenly have
Speaker 1: the freedom to pursue AI independently. Help me understand how
Speaker 1: that relationship with open AI is going to work in future,
Speaker 1: because you've been collaborators Microsoft and open Air for a
Speaker 1: long time. Now you're also going to be competitors. How
Speaker 1: does that work.
Speaker 2: It's a sort of complicated setup for people to follow
Speaker 2: along to. But the bottom line is that up until
Speaker 2: a few weeks ago, Microsoft was not allowed by contract
Speaker 2: to pursue artificial general intelligence or superintelligence independently. And the
Speaker 2: deal with open Ai was that it would then go
Speaker 2: and build AGI when they signed the agreements back in
Speaker 2: twenty nineteen, and in return, Microsoft would build the AI infrastructure,
Speaker 2: the chips and the data centers, and in return from
Speaker 2: providing the chips, Microsoft would get a license to the
Speaker 2: models that have been built, and we still have that
Speaker 2: license to everything that open Ai builds up until twenty
Speaker 2: thirty two. But open Ai decided that they wanted to
Speaker 2: take on more compute and buy compute from other providers,
Speaker 2: and so they now have deals with soft Bank and
Speaker 2: many others, Oracle to build more data centers than Microsoft
Speaker 2: wanted to build for them. And so in return, we
Speaker 2: then have the right to go and develop our own
Speaker 2: AI And obviously that was a big part of me
Speaker 2: joining the company eighteen months ago. That we're now hiring
Speaker 2: out a super intelligence team and pursuing our own AI development.
Speaker 1: So was that deal a relief to you because without it,
Speaker 1: how much could you have done? As Microsoft's AI chief?
Speaker 3: Well, I mean a lot.
Speaker 2: I mean we have two hundred and eighty billion dollars
Speaker 2: of revenue all major institutions in the world, and they
Speaker 2: need to be able to use RAI features and ORAI
Speaker 2: tools every single day. So you know, we're still been
Speaker 2: a general purpose AI development shop over the last eighteen months,
Speaker 2: but now we can work on some techniques and methodologies
Speaker 2: that have the potential to exceed human performance at all tasks.
Speaker 3: And so it is a shift for us.
Speaker 1: Does your relationship with OpenAI include not poaching each other's
Speaker 1: staff because we've just had another bit of evidence of
Speaker 1: the talent war that's underway. You've just lost one of
Speaker 1: your key people to Apple.
Speaker 2: Yeah, he's going to go and run the Apple Lab.
Speaker 2: I mean, look, I think that there's huge rotation. I
Speaker 2: mean the talent in the industry. We just got a
Speaker 2: whole bunch of people from Google Deep Mind, We've got
Speaker 2: people from open Ai.
Speaker 3: I mean is there's a lot of rotation. This is
Speaker 3: part of the industry.
Speaker 2: So there's certainly no no poach agreements that would not
Speaker 2: be legal. People can go work for whoever they want
Speaker 2: to go work for, and I think that at this
Speaker 2: time in the industry, it's just very very competitive.
Speaker 1: And does that mean that you are prepared to Oh
Speaker 1: and are you already matching the kinds of sums that
Speaker 1: Meta's been spending the one hundred to two hundred million
Speaker 1: dollar packages.
Speaker 3: I don't think anyone's matching those things.
Speaker 2: Zuck's taken a particular approach that involves sort of hiring
Speaker 2: a lot of individuals rather than maybe creating a team,
Speaker 2: and I don't really think that's the right approach. I
Speaker 2: think what we did well at deep Mind is that
Speaker 2: we were very selective. What I've been doing a Microsoft
Speaker 2: over the last year and a half is adding people
Speaker 2: incrementally who fit the culture and who suit the skill
Speaker 2: set and work well with the rest of the group,
Speaker 2: weeding out those who don't, and being very very attentive
Speaker 2: to the details. So we're very much creating a team
Speaker 2: rather than a set of individuals.
Speaker 1: So you're not prepared to match salary offers on that scale,
Speaker 1: Because that was what was recently reported.
Speaker 3: Yeah, I mean that was the rumor.
Speaker 2: I think that was you know, obviously very very unprecedented
Speaker 2: and may have been the case for a few individuals,
Speaker 2: but that's certainly not the norm.
Speaker 1: Would you rather he Mark Zuckerbergelsuk as you call him,
Speaker 1: Would you rather he rethought that? Does it? Does it
Speaker 1: make life much more difficult for you?
Speaker 3: No, not at all. I mean I think that he's
Speaker 3: all in right.
Speaker 2: He's building a two gigawat data center, which will probably
Speaker 2: cost him a couple.
Speaker 3: Hundred billion dollars over the next two or three years.
Speaker 2: And I think he said publicly to Trump that he's
Speaker 2: going to spend up of six hundred billion dollars in
Speaker 2: the next three years on data centers. And you know,
Speaker 2: luckily for Microsoft, like we have thirty three gigawatts of
Speaker 2: compute capacity in our fleet. Our primary business is to
Speaker 2: build data centers and provide them to third parties, and
Speaker 2: so we can either use that for our own training
Speaker 2: or we can use it for inference where we sell
Speaker 2: it to third party. So we've got a very very
Speaker 2: good setup in that sense. We're kind of hedged against
Speaker 2: that risk.
Speaker 1: Can you see why people worry about the not just
Speaker 1: the amounts of money being spent by the industry, but
Speaker 1: the circular range of a lot of those deals. You know,
Speaker 1: you're an investor in open Ai, they're also buying services
Speaker 1: from you. Everyone seems to be in a relationship with Nvidia.
Speaker 1: There's a huge amount, a huge proportion of the of
Speaker 1: the US economy that is dependent on things continuing to
Speaker 1: go well for firms like yours.
Speaker 2: Yeah, I mean, I think that's fair. Obviously, customers and
Speaker 2: providers have often invested in one another, and you know,
Speaker 2: provides good stimulus to get us going. But it's something
Speaker 2: to watch. I'm definitely watching it carefully, and I think
Speaker 2: others are too, so you know, I think that getting
Speaker 2: the balance right is very important. We have to deliver
Speaker 2: in the next few years. I mean, every team now
Speaker 2: is building incredibly large, very powerful computers, and we're taking
Speaker 2: a huge bet that we're going to be able to
Speaker 2: convert this into true intelligence. And if we do, then
Speaker 2: I think the world is going to look very, very different.
Speaker 2: I mean, we will have abundant intelligence on tap, and
Speaker 2: I think people still underrate how phenomenal that is.
Speaker 1: You say, the world, except that this is really about
Speaker 1: two countries, the US and China. That's where almost all
Speaker 1: the labs are.
Speaker 2: That's true, and everything has concentrated even more around Silicon
Speaker 2: Valley in the last eighteen months, and that's sort of natural.
Speaker 2: I mean, I think we see that in a lot
Speaker 2: of other historical technology trends. At the same time, open
Speaker 2: source is doing very well, and the cost of production
Speaker 2: is also going through the floor. So it costs ninety
Speaker 2: percent less to ask a question of one of the
Speaker 2: best AI models in the world than it did two
Speaker 2: years ago. You don't get curves that fall that quickly
Speaker 2: in any other discipline, and obviously, when the cost goes down,
Speaker 2: everybody gets access, So I don't think there's going to
Speaker 2: be an access issue.
Speaker 1: Is there anything you look back on now that you
Speaker 1: would have done differently because you've you know, you're still
Speaker 1: only in your early forties, so you've got a lot
Speaker 1: of remarkable achievements. But is there anything left undone or
Speaker 1: that you reflect on.
Speaker 2: Yeah, I really want to nail medical superintelligence. And you know,
Speaker 2: I started deep mind Health back in twenty fifteen, and
Speaker 2: you know, we published a lot of great papers and
Speaker 2: we had a lot of clinical trials, but we didn't
Speaker 2: quite get there. And this time round, I think that's
Speaker 2: going to be the most magical application. I want to
Speaker 2: do more in energy efficiency and battery storage, developing new
Speaker 2: compounds for renewables and so on. I think that AI
Speaker 2: will really transform the energy industry. So those are my
Speaker 2: two sort of big focuses I think.
Speaker 1: And is that what your day's like That when the
Speaker 1: engineers your team are say to you, we can do
Speaker 1: this and this, you're saying, this is how I want
Speaker 1: to use it.
Speaker 2: What we're always focused on is can we build general
Speaker 2: purpose models that are as smart as humans at all capabilities.
Speaker 2: I mean, that is the number one goal. And then
Speaker 2: the question is how do we apply them, Where do
Speaker 2: we use them? What can they do for people? You know,
Speaker 2: I'm actually very proud of a lot of the use
Speaker 2: cases in co Pilot. I mean, many people are using
Speaker 2: it for companionship, emotional support, therapy, making difficult life decisions.
Speaker 2: Has given me high quality access to informations, giving me
Speaker 2: emotional support, and is helping keep me organized.
Speaker 1: And so what do you mean helping you get emotional support?
Speaker 2: So basically I have a practice that at the end
Speaker 2: of the day, when I'm in the car driving home,
Speaker 2: I have like a five or ten minute conversation with
Speaker 2: co Pilot about something that was tricky in my day
Speaker 2: or something that I felt frustrated about. Maybe emotional support's
Speaker 2: a little strong, but it's just like having a chat
Speaker 2: with a friend and kind of downloading.
Speaker 3: Really on what went well and what didn't.
Speaker 2: And once you build that memory up over time, it
Speaker 2: gets really helpful because it personalizes. So Copilot now remembers
Speaker 2: most of what you say, and it will personalize its
Speaker 2: answers to you and refer to something that you said
Speaker 2: last week, for example, or a trend or a pattern,
Speaker 2: and that is just like super helpful. I feel kind
Speaker 2: of refreshed after a conversation with I feel like I'd
Speaker 2: let go of something.
Speaker 3: It's like a burden that I've released.
Speaker 1: So it is a friend, a therapist, a family member. Almost.
Speaker 3: Yeah, it's a little bit of everything.
Speaker 2: That's always the challenge when we're trying to, you know,
Speaker 2: sort of understand a new technology and come up with
Speaker 2: words to describe something which is a little bit like
Speaker 2: many other things but is also fundamentally different. This is
Speaker 2: what an AI is now.
Speaker 1: I just wonder if there's any aspect of that that
Speaker 1: makes you pause. People go home and maybe they don't
Speaker 1: have to, they don't have to bother to all talking
Speaker 1: to others in their life in the same way because
Speaker 1: they've said it all and they've got what they need back.
Speaker 3: Yeah.
Speaker 2: Or the flip side is they don't have to take
Speaker 2: it out on their partner or their best friends. I
Speaker 2: still call my best friends every weekend and have a
Speaker 2: good old chat like so, I think if anything is
Speaker 2: actually deep in some of my relationships with my friends,
Speaker 2: I come to those conversations just feeling a little lighter
Speaker 2: and more ready to talk about other things.
Speaker 1: And your family, Where are your parents now and how
Speaker 1: do they feel about the choices you made?
Speaker 3: Yeah?
Speaker 2: No, I think my mom's very you know, she's a
Speaker 2: proud mum. She's very excited, and she uses co pilot.
Speaker 2: She used pie before this. If I went to see
Speaker 2: her three or four weeks ago and she was still
Speaker 2: using Pie and I was like, that's amazing.
Speaker 1: Which is what you founded at inflection, right.
Speaker 2: Yeah, I was quite impressed to see that she still
Speaker 2: has voice calls with it, and just throughout the whole day,
Speaker 2: We'll be walking around the garden and she'll pull out
Speaker 2: her phone and you know, she'll scan a plant and
Speaker 2: talk about, you know, whether this is a chestnut tree
Speaker 2: and which variety it is. And could I grow it
Speaker 2: at home and when we're in the park and stuff,
Speaker 2: And so it's amazing to see her just naturally use
Speaker 2: it in her everyday life, and I think that is
Speaker 2: quite gratifying.
Speaker 1: Do you see your future in the US? What's the
Speaker 1: fud because I'm sure the UK would love to have
Speaker 1: you back.
Speaker 2: I would have loved to have started Inflection in the UK.
Speaker 2: I'm very British in mentality. But one of the things
Speaker 2: that I don't like is the tall Poppy syndrome. I
Speaker 2: think that there's not enough of a culture of risk taking.
Speaker 3: There's a little bit of.
Speaker 2: A taboo around commercialization and making money and starting companies
Speaker 2: and being entrepreneurial, and there's not enough of a kind
Speaker 2: of celebration of experimentation and failure, whereas in the Valley
Speaker 2: everyone's nuts and everyone loves failure. So they're just like
Speaker 2: constantly talking about how things are going wrong and how
Speaker 2: this was a disaster, and it's very liberating. I mean,
Speaker 2: it's also very cheesy and is you know, you can
Speaker 2: great when you're when you're kind of a bit of
Speaker 2: a cynical English person, but when you get into the
Speaker 2: rhythm of it, it's great everyone's up for.
Speaker 1: It, but I'm wondering whether there is actually something. It's
Speaker 1: not just chat. It is actually like if you if
Speaker 1: you're talking about failure in that way, you are braver
Speaker 1: about it or just you know, it's it's normal, it's
Speaker 1: something you have to go through. It's a different mindset, yeah, exactly.
Speaker 3: And there's also a kind of hustle culture.
Speaker 2: It's quite humbling to see everybody just on the make
Speaker 2: trying to understand some new technology, random non technical people
Speaker 2: reading science papers, you know, in a Starbucks or whatever.
Speaker 2: There'll be someone on their laptop watching a machine learning video.
Speaker 2: You'll be reading a book in a coffee shop and
Speaker 2: you'll overhear two people who don't know each other just
Speaker 2: try to spark a conversation that might turn into something.
Speaker 3: It's just like there's just an energy.
Speaker 2: And I think I would love for that to be
Speaker 2: the case in London too, and I think it could be.
Speaker 2: But we've got to have political leaders who celebrate risk
Speaker 2: taking and business.
Speaker 1: Finally, I need you to prepare me for what AI
Speaker 1: will do to my life. All right now, do you
Speaker 1: think AI could have had this conversation with you or
Speaker 1: could it have done a better job than me. You
Speaker 1: can break it to me if you think AI could
Speaker 1: have done a better job than me interviewing.
Speaker 2: You, probably not today, but maybe if you had prepared
Speaker 2: with an AI then you know, it could have given
Speaker 2: you questions to ask and stuff. Although you did do
Speaker 2: a lot of detailed research, so your team's obviously very good.
Speaker 2: I do think that there are going to be AI reporters,
Speaker 2: you know, so I run MSN, the news service at Microsoft.
Speaker 2: One of the things I'm very excited about is how
Speaker 2: AI news reporters can reinvigorate local news. So imagine that
Speaker 2: there are hundreds of thousands of AI reporters that can
Speaker 2: make phone calls into people who are at the scene,
Speaker 2: who can verify I witness footage, conduct interviews, stitch those
Speaker 2: together into little montages, and not just do it for
Speaker 2: big national stories where the investment is justified, but do
Speaker 2: it at a very very local level to provide accurate
Speaker 2: and factually reliable information. I think that people need to
Speaker 2: have the courage to be optimistic about how these things
Speaker 2: can actually improve our system.
Speaker 1: Are you developing AI reporters for Emerson?
Speaker 3: We're exploring everything, you know.
Speaker 2: I'm also on the board of the economists and have
Speaker 2: been talking to them a lot about it. And you know,
Speaker 2: I think that we're exploring all kinds of things at
Speaker 2: the moment.
Speaker 1: So AI reporters will happen AI interviewers. I think I suspect, yes,
Speaker 1: from what you're saying.
Speaker 2: You've got you've got a little bit longer, maybe six months.
Speaker 2: I'm kidding, are you?
Speaker 1: I'm not sure you are?
Speaker 3: I am kidding. Is we had to be plenty for
Speaker 3: we are a very very long time.
Speaker 2: It's really long time to get it perfect, you know,
Speaker 2: I mean, this was this was exceptional the first.
Speaker 1: Thank you very much for talking to us.
Speaker 3: Thank you very much for Chelle. This was a lot
Speaker 3: of fun.
Speaker 1: And that's the Michelle Hussein Show for this week. If
Speaker 1: you follow us, you'll know as soon as there's a
Speaker 1: new episode, And if you'd like to email us, please
Speaker 1: use Michelle's show at Bloomberg dot net. By the way,
Speaker 1: if you've left us a rating or a comment, thank you.
Speaker 1: These conversations are also on YouTube and on Bloomberg TV,
Speaker 1: and there's a text version with my own thoughts in
Speaker 1: note form. You'll find those at Bloomberg dot com slash Weekend.
Speaker 1: Our producers are Jessica Beck and Chris Martlieu. The executive
Speaker 1: producer is Louisa Lewis. Guest booking by Dave Warren, sound
Speaker 1: engineer Richard Ward, video editing Evander Thompson and Megan Olsen.
Speaker 1: Social media is by Alex Morgan, and the music is
Speaker 1: by Bart Walshaw at Bloomberg Weekend and Francis Newnham is
Speaker 1: Editorial director of Audio and Special Projects, and our executive
Speaker 1: editor is Catherine Bell. We'd also like to thank Summersadi
Speaker 1: and the Bloomberg Podcast team and thank you for listening.
Speaker 1: Until next time, goodbye,