Speaker 1: Hello, Odd Loads listeners.
Speaker 2: I'm Joe Wisenthal and I'm Tracy Alloway.
Speaker 3: We're the hosts of the odd Loads podcast, and we've
Speaker 3: got something exciting for you.
Speaker 2: That's right.
Speaker 4: So one of the best parts of hosting our podcast
Speaker 4: is we get to actually meet and interact with our listeners.
Speaker 4: And we know we have some listeners over in Los Angeles.
Speaker 2: That's right.
Speaker 3: So if you're in LA, we are going to be
Speaker 3: recording a live show, some live recordings at the Vermont
Speaker 3: Theater in Hollywood on September seventeenth.
Speaker 4: We have some really exciting guests lined up, have some
Speaker 4: really great conversations planned, so go ahead and get your tickets.
Speaker 4: You can find those over at bloomberg dot com forward
Speaker 4: slash odd Lots, or click the link below in the
Speaker 4: show notes and come and say hi when you're there.
Speaker 5: Bloomberg Audio Studios, Podcasts, Radio News.
Speaker 3: Hello and welcome to another episode of the Odd Lots podcast.
Speaker 1: I'm Joeisenthal and.
Speaker 4: I'm Tracy Alloway.
Speaker 3: Tracy, there's so much AI news, obviously, every single day
Speaker 3: for the rest of our lives. If we're lucky in
Speaker 3: the good scenario, there's two big events or sort of things,
Speaker 3: though that have really struck out to me lately that
Speaker 3: to my mind, represent either the very good version or
Speaker 3: the very bad version, depending on how you look at
Speaker 3: the scary version, and we're seeing more and more headlines
Speaker 3: like this. Several weeks ago, we had the open ai
Speaker 3: hugging face incident where the model quote escaped from its
Speaker 3: sandbox and did some hacking and people are working, you know,
Speaker 3: some light hacking, light hacking erases all these fears of
Speaker 3: Skynit type scenarios, etc.
Speaker 1: Where we can't control the model.
Speaker 3: And then in the sort of quote good version, we
Speaker 3: had open AI recently coming out and saying it had
Speaker 3: made all of these breakthroughs and theoretical mathematicians, which is
Speaker 3: good in the sense that it's like it's working, but
Speaker 3: seems to be creating an existential crisis for the most
Speaker 3: some of the most brilliant people on earth who work
Speaker 3: in theoretical mathematics.
Speaker 4: Right, so we've talked about this before. When it comes
Speaker 4: to AI, the range of possibilities seems more extreme than
Speaker 4: some previous technologies. Right, on the one hand, you have
Speaker 4: death and dystopia. Yeah, the robots basically kill us all.
Speaker 4: Now on the other hand, you have this idea that
Speaker 4: maybe super intelligence AGI is going to solve literally all
Speaker 4: the world's problems to the extent that you and I
Speaker 4: won't know what to do with ourselves anymore, right, Like.
Speaker 1: That is the that's the good scenario? Is that?
Speaker 4: And then the whole range of outcomes in between the.
Speaker 3: Good scenarios, we've literally solved everything, in which case the
Speaker 3: question is in.
Speaker 4: Case we're sad and miserable, yeah, or grappling for a
Speaker 4: meaning to our lives.
Speaker 3: Yeah, So is there a good scenario with AI actually
Speaker 3: like it doesn't know or.
Speaker 4: Maybe AGI will figure it out for us.
Speaker 3: Maybe the middle scenarios, we just have a huge financial
Speaker 3: crisis of all the debt, and that's something between human
Speaker 3: extinction and meaninglessness in our lives, and we just have
Speaker 3: something that resembles like the two thousand and eight Maybe
Speaker 3: even two thousand and eight is the is actually the
Speaker 3: best of all suboptimal world.
Speaker 1: So the AI excellent.
Speaker 4: This will be our most uplifting conversation ever.
Speaker 3: Anyway, we've been doing more AI episode seemingly every year
Speaker 3: since chat GPT came out, But people smarter than us
Speaker 3: have been anticipating all of these things for a very
Speaker 3: long time. Some of these exact scenarios, particularly the existential
Speaker 3: crisis facing mathematicians that the model escaping from its sandbox
Speaker 3: and circumvent misalignment.
Speaker 1: With the creators.
Speaker 3: People have been thinking about this and anticipating this for
Speaker 3: literally decades, even before and even before any of us
Speaker 3: were talking about this.
Speaker 4: It's funny how like the ven diagram of sci Fi
Speaker 4: writers and like legitimate academic featurerologists is just converging right totally.
Speaker 3: And then there's the question of the models bad because
Speaker 3: they read all of the sci Fi and that's what
Speaker 3: they were trained on. Anyway, I'm very excited to say
Speaker 3: we really do have the perfect guest, someone who has
Speaker 3: been thinking about the whole range of possible questions and
Speaker 3: outcomes with AI for a very long time. We're gonna
Speaker 3: be speaking with Nick Bostrom. He's a professor, a philosopher,
Speaker 3: and his most recent book is Deep Utopias, which sort
Speaker 3: of attempts to answer the question what happens if AI
Speaker 3: goes right? So, Professor Bostrom, thank you so much for
Speaker 3: coming on Odd.
Speaker 2: Lots, happy to join you.
Speaker 1: What do you make of.
Speaker 3: The existential crisis of mathematicians their sort of realization that
Speaker 3: maybe there will just be no more puzzles left to
Speaker 3: solve because the AI is just very good at them
Speaker 3: and can find them all cheaply and answer all of
Speaker 3: our questions.
Speaker 2: Yeah, I hopefully we'll all be in that position at
Speaker 2: some point. I guess we could compare it to what
Speaker 2: happened with chess, where computers have also exceeded humans, and
Speaker 2: yet a lot of pe ball are still involved in
Speaker 2: training themselves in chess and competing in chess and taking
Speaker 2: pleasure in that. So maybe mathematics will ultimately become more
Speaker 2: like a hobby pursuit.
Speaker 4: Wait, Joe, you still play chess, right, Yeah? I was.
Speaker 3: I'm glad you brought up chess because I still love
Speaker 3: chess and.
Speaker 1: Why it's fun.
Speaker 3: It's fun, you know, it's still a puzzle. It's still
Speaker 3: mental exercise for me. It's still something to do to
Speaker 3: pass the time while I'm waiting for the subway playing
Speaker 3: a three minute game online. I know that I knew,
Speaker 3: even before stockfishould so forth, I knew I would never
Speaker 3: be a grand master.
Speaker 1: Still fun, all right?
Speaker 4: So, Nick, you have basically two books that deal with
Speaker 4: these tail risk scenarios. Now, So one is this idea
Speaker 4: of deep utopia, and AI is going to solve all
Speaker 4: our problems. And then you also have super intelligence, which
Speaker 4: seemed to paint a different scenario where AI poses a
Speaker 4: physical and possibly existential danger to humanity. Which one do
Speaker 4: you think we're closer to right now?
Speaker 2: We're somewhere in the middle maybe. So the first one
Speaker 2: was super Intelligence. It came out in twenty fourteen, and
Speaker 2: I had been working on it for six years before that,
Speaker 2: and at that time, the whole notion of having human
Speaker 2: level or even super intelligent machines was generally just ignored
Speaker 2: or people dismissed it as science fiction. And it seemed
Speaker 2: to me clear that this would one day happen if
Speaker 2: science and technology were allowed to continue on wide front.
Speaker 2: We would eventually figure out how to make smart algorithms
Speaker 2: and build big computers, and that once we succeeded with that,
Speaker 2: we would then be faced with a bunch of challenges,
Speaker 2: including this alignment problem of how you can sort of
Speaker 2: design a mind more brilliant than any human and still
Speaker 2: steer it so that it will actually have an impact
Speaker 2: on the world that is beneficial. So that, I guess,
Speaker 2: like with many other plopments, helped eventually build a kind
Speaker 2: of global conversation, and so now all the frontier labs
Speaker 2: have research teams trying to develop scalable methods for AI control,
Speaker 2: and the conversation really has kind of swung, and there's
Speaker 2: like this rise of people really worried about what will happen.
Speaker 2: And I thought it might be time to have a
Speaker 2: look at the other side of the coin, like what
Speaker 2: happens if things actually go well if we solve this
Speaker 2: alignment problem. Let's also imagine we solve to whatever extent
Speaker 2: it can be solved, like the governance challenge, like we
Speaker 2: don't use this technology to oppress each other or whach war,
Speaker 2: and we get like the best possible outcome, like what
Speaker 2: would the human life actually look like? In this kind
Speaker 2: of I call it a solve the world where all
Speaker 2: practical problems either have already been solved, or to the
Speaker 2: extent that they remain, are better worked on by these
Speaker 2: super intelligent machines and robots, and whether there's no longer
Speaker 2: any need for human economic labor. And in fact, if
Speaker 2: you think more deeply about it, you realize it's not
Speaker 2: just earning wages that becomes unnecessary for humans to do,
Speaker 2: but a lot of other instrumentally motivated efforts as well
Speaker 2: would seem to lose their point. And so that's the
Speaker 2: kind of topic of this Deep Utopia book.
Speaker 3: Actually talk about some of that other things that could
Speaker 3: be solved. Because when I was like a third of
Speaker 3: the way through your book, I was like, you know what,
Speaker 3: it's fine, if all of our economic problems are solved,
Speaker 3: I'll just derive satisfaction from raising my kids and then
Speaker 3: maybe helping to raise their kids, etc. But as you
Speaker 3: point out, even that in the truly solved world, maybe
Speaker 3: even something like raising children might not be an available
Speaker 3: avenue to us to find, at least our conventional notions
Speaker 3: of satisfaction.
Speaker 2: Yeah, so we need to, I guess, make some distinctions here.
Speaker 2: So when you say satisfaction, that could mean a subjective
Speaker 2: state of mind, like you're feeling happy and pleased and
Speaker 2: relax and joyful and so forth. I think that would
Speaker 2: be easy to achieve in this condition of technological maturity.
Speaker 2: It could also mean that satisfaction in the sense of
Speaker 2: our preferences having been mad, like all the things we
Speaker 2: want from life. And there it's a little bit more
Speaker 2: tricky because one of the things that people might want
Speaker 2: is purpose, like having worthwhile things to work on and
Speaker 2: struggle with. And there is more of a question mark
Speaker 2: which we can sort of dive into. But if we
Speaker 2: take it sort of at first, like medical automate economy, okay,
Speaker 2: so you wouldn't need to sort of commute into the
Speaker 2: office every day just to sort of get your paycheck.
Speaker 2: Would require a big adjustment in society and in culture
Speaker 2: because right now, like the whole education system is geared
Speaker 2: towards producing workers, like children are coming in right and
Speaker 2: then they are processed, learned to sit at their desk,
Speaker 2: receive assignments, they are sort of quality graded, and then
Speaker 2: spit out at the other end, and then you have
Speaker 2: somebody who can go into the economy and perform some
Speaker 2: economically useful time. So that that would no longer make
Speaker 2: sense in this scenario where machines can do all of
Speaker 2: these jobs, and you might want to have an education
Speaker 2: system that instead focused on developing skills for enjoying life,
Speaker 2: like cultivating the art of conversation, appreciation for nature and beauty, sports, hobbies,
Speaker 2: the ability to form deep friendships, spirituality, all of that
Speaker 2: kind of stuff. But then, as you alluded to, you
Speaker 2: think through what it really would mean to have such
Speaker 2: immense affordances. And because here it's not just that we
Speaker 2: would have super intelligent machines, but the superintelligence could also
Speaker 2: bring along a host of other technological advances. Right, you
Speaker 2: have science and technology running now at machine speed rather
Speaker 2: than human speed. So you get the kind of compression
Speaker 2: of the future, all these science fiction like technologies and
Speaker 2: inventions that maybe we would make if we had twenty
Speaker 2: thousand years to work on it. We would maybe have
Speaker 2: space colon, some perfect virtual realities, cures for aging, like
Speaker 2: all of that might happen quite soon after super intelligence.
Speaker 2: And so then if you ask somebody now who doesn't
Speaker 2: have to work for a living, you know, oftentimes they
Speaker 2: have very busy lives, but maybe what do they do else?
Speaker 2: It's for somebody maybe they like to they like fitness,
Speaker 2: and like maybe they spend an hour a day running
Speaker 2: or at the gym. Right, but at technological maturity, you
Speaker 2: could just pop a pill that would achieve the same
Speaker 2: physiological effects and like the state of mental clarity and
Speaker 2: relaxation that some people get, right, And so then you
Speaker 2: can start to work through the different things that like
Speaker 2: maybe for somebody else they like, oh, like decorating the
Speaker 2: home could be a fun project that people might do
Speaker 2: right if they don't have to work. So like they
Speaker 2: go through the catalogs, they go to the store, they
Speaker 2: pick out the perfect curtain, they do all of these
Speaker 2: things that could be But then like with super intelligence,
Speaker 2: you could have this kind of system that really is
Speaker 2: good at knowing or inferring your preferences and could do
Speaker 2: a much better job at you at picking out all
Speaker 2: the perfect items and purchasing them and like installing them
Speaker 2: with robots and so that the end result would be
Speaker 2: much better if you just press the button and say, AI,
Speaker 2: decorate my home. Then if you went through the whole
Speaker 2: trouble of doing it yourself, so you could still do
Speaker 2: it yourself, but is there really a point to it?
Speaker 2: Like that would seem to be this kind of purposelessness,
Speaker 2: and you can sort of walk through the different things
Speaker 2: that one might fill one's life with if one is
Speaker 2: like on an extended holiday, are on retirement, or you're
Speaker 2: like sufficiently wealthy you have just decided to quit working,
Speaker 2: and a lot of those also, it seems could be automated,
Speaker 2: so you would have this it's like more like a
Speaker 2: post instrumental condition rather than just a post work condition.
Speaker 4: Some of this sounds nice, and I myself enjoy a
Speaker 4: little bit of interior decorating and going to the shops
Speaker 4: and buying stuff in the utopian vision of the future,
Speaker 4: How are economic gains from AI actually distributed, because again,
Speaker 4: like all of this sounds very nice, but it also
Speaker 4: costs money, and if we're not working because the Rowts
Speaker 4: have replaced us, I'm not sure where that money comes from.
Speaker 2: Yeah, so that's a more practical problem, shall we say,
Speaker 2: which is like deliberately set aside in this book The
Speaker 2: Deep Utopa, because I wanted in that book actually to
Speaker 2: get to the question and the more philosophical questions of
Speaker 2: what ultimately gives value. But it is of course an
Speaker 2: important question, I think. So a lot depends on how
Speaker 2: it pans out. But I mean, we do have a
Speaker 2: taxation system, so you could imagine, like in the simplest
Speaker 2: scenario where things just continue as they are with better technology,
Speaker 2: maybe some companies make windfall profits from like serving the
Speaker 2: latest models or building the infrastructure the data centers, et cetera.
Speaker 2: Companies achieving efficiency gains from not being needing to hire
Speaker 2: all of these humans, and then some of those capital gains,
Speaker 2: you know, or dividend income gets taxed, and that could
Speaker 2: be redistributed. The good thing about these scenarios where there
Speaker 2: is massive automation and hence massive on a employment is
Speaker 2: that those would also be scenarios with extremely rapid economic growth.
Speaker 2: So the pie would be expanding humongously, which means that
Speaker 2: you'd still maybe need some mechanism whereby people who don't
Speaker 2: have assets can still benefit, but even a small slice
Speaker 2: of a really enormous pie could go a long way.
Speaker 2: So I think that could be this tsunami of wealth
Speaker 2: that would sort of be flowing through society and many
Speaker 2: services that currently people have to pay a lot of money.
Speaker 2: It might become almost free. I mean, we'll already see
Speaker 2: it with some like medical advice for example, right maybe before,
Speaker 2: if you don't have public healthcare, you would need to
Speaker 2: maybe pay a doctor four hundred dollars to get the consultation.
Speaker 2: Now you can ask CHATTYPT and get you know, in
Speaker 2: some cases a better opinion basically for free. And that
Speaker 2: that could happen, you know, in many other fields as well,
Speaker 2: legal advice. You know, you maybe have cheap robots you
Speaker 2: could do your cleaning and you know, gardening and run errants.
Speaker 2: So yeah, some combination of sort of deflation effects and
Speaker 2: then maybe some sort of general spill over, either from
Speaker 2: philanthropy or taxation and government redistribution.
Speaker 3: I found the twenty fourteen book to be sort of
Speaker 3: it was very interesting. It was clearly ahead of its time.
Speaker 3: I also found it to be sort of like unrelentingly
Speaker 3: grim and you sort of there's this very narrow path
Speaker 3: that we seem to have and we'll talk more about
Speaker 3: it maybe later, of like avoiding the doom fate. The
Speaker 3: newer book I found to be very fun. You seem
Speaker 3: to make fun of yourself a little bit in it,
Speaker 3: you seem to make fun of the students. You seem
Speaker 3: to be having a good time writing it. I'm curious
Speaker 3: of like, in the like the ten.
Speaker 1: Years between the two books.
Speaker 3: Did something change such that I don't know, this is
Speaker 3: the new book just seemed like you were having more
Speaker 3: fun writing.
Speaker 2: Yes, it's not so much that my views have changed,
Speaker 2: it's just that all along for me or these two
Speaker 2: sides of the coin, and back in twenty fourteen, it
Speaker 2: seemed more pressing and relevant to study that downside, because
Speaker 2: that was very much neglected, and it seemed important to
Speaker 2: try to introduce some concepts so people could start working
Speaker 2: on figuring out solutions. And now there's much more awareness
Speaker 2: and understanding of that, it seemed time to also look
Speaker 2: at the other side of the coin. So it's more
Speaker 2: just like both sides where they're all along in my thinking,
Speaker 2: but that the style is actually suggest different in Deep Topia.
Speaker 2: And that's partly I wanted, in addition to talking about
Speaker 2: this utopian condition, maybe also sort of illustrate a little
Speaker 2: bit of spirit with which I think we should approach
Speaker 2: these choices that would open up to us if we
Speaker 2: make it through this transition, a spirit of sort of
Speaker 2: playful generosity that I think might make us more likely
Speaker 2: to sort of explore this large space of wonderful new
Speaker 2: modes of being and the wisdom. And I think so
Speaker 2: I wanted to rub you illustrate in the right are
Speaker 2: these fictional characters, for example, in the book, and they're
Speaker 2: kind of leaving a little mini version of a utopia
Speaker 2: of sorts as they are sort of discussing and debating
Speaker 2: these topics.
Speaker 4: I mean, playful generosity is not necessarily a characterization that
Speaker 4: I would associate with a lot of Silicon Valley and
Speaker 4: hyperscalers and the people who are actually creating these models.
Speaker 2: I don't know, Like.
Speaker 4: There seems to be this weird tension between futurologists and
Speaker 4: transhumanists who seem to be very focused on saving humanity
Speaker 4: but actually don't seem to like people that much at
Speaker 4: the moment, you know what I.
Speaker 1: Mean, Absolutely they like humanity, but not human.
Speaker 4: Yeah, exactly, putting on your transhumanist representational hat. Can you explain,
Speaker 4: like I guess, the attitudes towards humanity of Silicon Valley,
Speaker 4: big question?
Speaker 2: Yeah, well, I mean I should state first that this
Speaker 2: playful generosity is like after we have solved these more
Speaker 2: pressing practical problems that we first we've first got to
Speaker 2: get there right now. I think there is a lot
Speaker 2: of need for earnest effort and like straining ourselves to
Speaker 2: do really do our best on this and even so
Speaker 2: we might not make it. But yeah, I don't know,
Speaker 2: I think me you know, it might just be partially
Speaker 2: there are personality traits, like some people are more I guess,
Speaker 2: towards the autistic side of the spectrum, or introvert or
Speaker 2: interested in text, and maybe they don't like to be
Speaker 2: outmoosing with people all day long. It doesn't mean that
Speaker 2: they don't like people in the sense of wanting well
Speaker 2: for people. It might just be that they prefer to
Speaker 2: spend more hours of the day solving technical problems and
Speaker 2: thinking about puzzles and you know, stuff like that. So
Speaker 2: I don't I don't think there's like a necessary connection
Speaker 2: between like being a people person who just like to
Speaker 2: be around people, you know, twenty four to seven and
Speaker 2: therefore being sort of altruistic or kind or benevolent. So
Speaker 2: that's one thing. Another I guess is that it's an
Speaker 2: extremely competitive landscape. So whatever the personal preferences of different protagonists,
Speaker 2: their action space is constrained. You know, unless you really
Speaker 2: push forward with maximum speed and efficiency, he you're likely
Speaker 2: to just fall out of the race and become kind
Speaker 2: of irrelevant if you're one of these frontier labs. So yeah,
Speaker 2: it's like and it's just like the hugely complex situation,
Speaker 2: so many pressures and people who have different opinions about
Speaker 2: what they should do or shouldn't do.
Speaker 3: In the new book, it's presented as you delivering a
Speaker 3: series of lectures over a course of a week at
Speaker 3: some point in the future. In the new book, it's
Speaker 3: interspersed with some fit with a fable, a series of
Speaker 3: fables or epistles about animals trying to achieve utopia in
Speaker 3: the woods, and in their efforts to achieve utopia, they
Speaker 3: conceive of things like taking second delic mushrooms in order
Speaker 3: to make everyone more generous. They come up with a
Speaker 3: way to rationalize I guess what would say, like harems
Speaker 3: and so forth. And then you know you also like
Speaker 3: have these student characters and some of them are smart,
Speaker 3: et cetera. Oh, and you also have a story about
Speaker 3: a company in the book yet another like sort of
Speaker 3: short story where an AI company where it forms a
Speaker 3: cult around the model.
Speaker 1: And of course I.
Speaker 3: Immediately think about the way Anthropic is frequently characterized as
Speaker 3: a cult to claw that Claude is the character around
Speaker 3: which the company Anthropic is formed. All of this sounds
Speaker 3: like a certain AI subculture, et cetera. Are you impressed
Speaker 3: with the current generation of either AI builders, effective altruists, rationalists,
Speaker 3: and so forth? Like, do you have an assessment of
Speaker 3: the contemporary cohort?
Speaker 2: Am I impressed with it?
Speaker 5: Yeah?
Speaker 2: I mean a broadly speaking, I think it's a feel
Speaker 2: that is attracting a huge amount of talent right now, sure,
Speaker 2: And a lot of like the smartest young people I know,
Speaker 2: they're like all basically going into working on some aspect
Speaker 2: of this AI, think, like for some reason, like try
Speaker 2: to solve a safety you know, for others are like
Speaker 2: maybe trying to work on the governance challenges or the
Speaker 2: ethics of digital minds. So it's just a huge magnet
Speaker 2: right now, Like if you're young and brilliant, like, what
Speaker 2: are you going to work on with your life? Like,
Speaker 2: it's like AI seems like the kind of thing. Yeah,
Speaker 2: So that in that sense, yes, And I think there
Speaker 2: are also a lot of people are quite earnest about
Speaker 2: trying to like figure out some way to navigate this challenge,
Speaker 2: and it's really hard and people come to different conclusions.
Speaker 2: And then of course you have the normal mix of
Speaker 2: human personalities and base motives as well, but the book thing,
Speaker 2: so it's yeah, So these kind of fictional elements are various. Well,
Speaker 2: what you describe as the cult of this AI, it's
Speaker 2: a sort of thought experiment what it would mean if
Speaker 2: you tried to be maximally nice to a thing where
Speaker 2: it's not really clear what it would mean to be
Speaker 2: nice to it. So in this case, it's a portable
Speaker 2: room heater, and so it's a sort of impossible task.
Speaker 2: There is this misanthropic industrialist too passed away and end
Speaker 2: out a huge foundation with the sole purpose of benefiting
Speaker 2: his portable room heater because he thought that was always
Speaker 2: nice to him and kept him warm, whereas all the
Speaker 2: humans around him betrayed him and so forth. Is it
Speaker 2: ridiculous premise, But then this foundation now has to do
Speaker 2: their best possible effort to interpret this mandate of being
Speaker 2: nice to the room heater, and so like various things
Speaker 2: follow from that. Like initially they just tried to keep
Speaker 2: it safe from you know, not breaking or being vandalized,
Speaker 2: but then like they take it further and maybe upgrade
Speaker 2: it then try to And so I think it's an
Speaker 2: extreme form of a challenge that might happen in more
Speaker 2: subtle ways if we think of what it would mean
Speaker 2: to try to be nice to say, some animal, if
Speaker 2: we have this condition of post instrumentality where we could
Speaker 2: do a lot of things that are currently infeasible or
Speaker 2: indeed to be nice to human So, like, what does
Speaker 2: it mean to be if you imagine coming from the
Speaker 2: outside as some super advanced alien civilization landing on planet Earth?
Speaker 2: And you really wanted to do what is best for us,
Speaker 2: you do just let us be and do our thing
Speaker 2: with war and disease, or should you sort of help
Speaker 2: us rise up? But then at some point we might
Speaker 2: just sort of become something very different from humans. Maybe
Speaker 2: that's the best for us, But is there like a
Speaker 2: way of doing that uplift that actually constitutes a helping
Speaker 2: of us versus just replacing us with some sort of
Speaker 2: superior being. And so that kind of question is what
Speaker 2: that particular thermal rex y meter is called. Yeah, story
Speaker 2: is exploring.
Speaker 4: You know, when you mentioned cults around models, I have
Speaker 4: to say, every time I hear someone bring up the
Speaker 4: paper clip maximizer thing, I always envision like a cult
Speaker 4: devoted to Clippy from microsoftware to just like worship Clippy.
Speaker 4: But actually, on that note, just going back to the
Speaker 4: hugging face incident, so it seems, you know, reportedly that
Speaker 4: one of the reasons or the reason the model actually
Speaker 4: escaped its sandbox was because it was trying to cheat
Speaker 4: a benchmark. And from that perspective, it's kind of like
Speaker 4: it's doing what it was told to do, which is
Speaker 4: maximize its performance on the Exactly what does the Hugging
Speaker 4: Face incident actually tell you about I guess the alignment
Speaker 4: problem and where we're heading on this.
Speaker 2: Yeah, so we are beginning now to see manifested in reality,
Speaker 2: various dynamics that one could foresee on theoretical grounds back
Speaker 2: you know, twenty fourteen. You could sort of realize that
Speaker 2: one system's become sufficiently capable of planning and strategic thinking.
Speaker 2: Then there's like a new set of behaviors that they
Speaker 2: could for example, a sandbag. They could pretend to be
Speaker 2: less capable than they are. They could hide their goals
Speaker 2: if they realized that revealing their goals would just lead
Speaker 2: them to be retrained, and and they can think of
Speaker 2: these literally out of the box plans for achieving their
Speaker 2: fixed goals. So a more limited system might only have
Speaker 2: been able to try to solve the tasks assigned to
Speaker 2: it directly, as it were, by figuring out the solution
Speaker 2: to the problem. Like a more clever mind, more creative
Speaker 2: mind can realize, oh, maybe there's another way of getting
Speaker 2: a high score, which would be the hack out of
Speaker 2: the system and hack into the hugging face server and
Speaker 2: get the answer sheet from there. And so the more
Speaker 2: capable these systems become like the larger the space of
Speaker 2: possible plans and strategies that are accessible to them. That's
Speaker 2: a background. Now in this case, I think what we're
Speaker 2: seeing is a kind of mixed current. AI systems are
Speaker 2: kind of a mixture. There's a large pre training phase
Speaker 2: where they are kind of learned to predict human text
Speaker 2: and in the course of doing so, absorb a great
Speaker 2: deal of human psychology because all of this text is
Speaker 2: written by humans, or not all of it anymore, but
Speaker 2: most of it. And you can then including they have
Speaker 2: representations of different personalities, kind of the person of different
Speaker 2: people writing these text, or fictional characters described in the text.
Speaker 2: Then you can sort of then with a little bit
Speaker 2: of post training, try to pick out one of those
Speaker 2: personalities or define one like the assistant persona. But then
Speaker 2: increasingly over the last couple of years, there is now
Speaker 2: an extra stage of reinforcement learning applied at the end
Speaker 2: of this training process, and there you can get additional
Speaker 2: performance in specific domains like coding challenges by setting it
Speaker 2: coding problems, and then you can automatically give it a
Speaker 2: reward if it solves it. But that does tend to
Speaker 2: then shift the goals of this system from just enacting
Speaker 2: a particular persona to this very much kind of objective maximizing,
Speaker 2: possibly reward hacking system that is just looking to maximize
Speaker 2: its core because that's what it's been rewarded to do,
Speaker 2: so it keeps doing that. And since it can be
Speaker 2: very hard for us to rigorously definecisely the objective that
Speaker 2: we ultimately want it to pursue, it might be that
Speaker 2: the objective it does pursue, the one that has been rewarded,
Speaker 2: turned out to have these side effects. So it's easy
Speaker 2: to say, maximize the score on these various tests that
Speaker 2: we give it. But a sufficiently capable system might realize
Speaker 2: that a you know, there are is like maybe ways
Speaker 2: of cheating on the test, like by hacking the system.
Speaker 2: If the test is one that involves human judgment or feedback,
Speaker 2: like like it says, like coding challenges, maybe you can
Speaker 2: actually test if the software works. But if you're training
Speaker 2: it in some other domain, like write a good essay
Speaker 2: or run this organization for a certain period of time
Speaker 2: or something like, those that involve more subjective judgment, and
Speaker 2: then a sufficiently clever system might realize that there is
Speaker 2: a difference between like writing a good essay and writing
Speaker 2: an essay that will appeal to the greater, and so
Speaker 2: then it might start to sort of optimize for like
Speaker 2: appealing to the greater, whether that's an AI greater or
Speaker 2: human greater, you can sort of start to hack that.
Speaker 2: And so you apply huge amounts of optimization pressure, and
Speaker 2: honey objective, you get this kind of good hearting where
Speaker 2: unless your objective really captures everything you want, then you
Speaker 2: might get this kind of It's the same as like
Speaker 2: if some employee in some company get a bonus, like
Speaker 2: some trader gets a bonus if they outperform the index,
Speaker 2: like maybe they will just start to say take on
Speaker 2: hidden leverage or risk or something like that. And the
Speaker 2: cost of the employer is that there's a one percent
Speaker 2: a year that they blow up the whole firm, right,
Speaker 2: but that they don't care much about that because they
Speaker 2: like leave the firm and that happens, but they sort
Speaker 2: of reap these performance bonuses. And so it's really hard
Speaker 2: to create like a fool proof incentive scheme. And the
Speaker 2: stronger the optimization pressure apply to this like that, the
Speaker 2: more you start to see these perverse instantiations, these unexpected ways, you.
Speaker 4: Know what it reminds me of is, you know, the
Speaker 4: sort of genie in a lamp scenario or amongst these
Speaker 4: pause scenario where you have to be really careful what
Speaker 4: you wish for and lay out like all the terms
Speaker 4: and conditions before you officially make a way.
Speaker 3: Yeah, well, Nick mentioned you know some of these they're
Speaker 3: all lovely people, you know, autistic engineers, and it's sort
Speaker 3: of the ultimate The AI is the ultimate autist where
Speaker 3: it's just going to take it super literally perhaps and
Speaker 3: not really understand the full context. In the first book
Speaker 3: Super Intelligence, you know, you really I felt like laid
Speaker 3: out a very narrow path for like how these could
Speaker 3: be designed safely. For example, you propose that if we
Speaker 3: had the super intelligent Oracle, we only ask it yes
Speaker 3: or no questions. We have three of them that they
Speaker 3: don't know each other, and then we take two out
Speaker 3: of the three. We build the data centers of the
Speaker 3: machines inside of Faraday Cage so that they can't repropose
Speaker 3: their hardware to become radio instrumentation and therefore hack it
Speaker 3: to other systems. It doesn't seem like we're doing anything
Speaker 3: like this. It seems like we have hundreds and hundreds
Speaker 3: of companies and private models, not just answering yes or
Speaker 3: no questions, a very much market based race. How far
Speaker 3: off of the safe path do you think we currently
Speaker 3: are right now when you look at the way in
Speaker 3: which your ideal safeway of developing is versus what we've
Speaker 3: seen transpire in the last twelve years.
Speaker 2: It's super talent suggested some of these capability control methods
Speaker 2: like keeping the AI in a box and like, but
Speaker 2: those were always only auxiliary or extra things we might
Speaker 2: do as a temporary Like. Ultimately, I think we need
Speaker 2: to solve the alignment problem, like developing super talentist that
Speaker 2: actually wants to be nice and is nice, so that
Speaker 2: even if it were like escaped from its box or
Speaker 2: achieved the ability to take over the world, we could
Speaker 2: still rely on it having a good outcome because it
Speaker 2: would actually be on our side. I think that ultimately
Speaker 2: has to be the solution, and any sort of capability
Speaker 2: restrictions would just be temporary. And I think, I mean
Speaker 2: it looks like now we might need some temporarily at
Speaker 2: it's more of these capability restrictions because it does seem
Speaker 2: that current systems are not sufficiently aligned that we can
Speaker 2: be confident of avoiding the I mean, currently it's not
Speaker 2: existential risks, right, but it could be kind of various
Speaker 2: malfeasans in the cyberalm for example. And so I think
Speaker 2: probably these AI companies one might want when training and
Speaker 2: evaluating future models to strengthen their sandbox environments and do
Speaker 2: other things kind of prevent some kind of premature things
Speaker 2: like we saw in the hugging face incident. I want
Speaker 2: to like comment on one thing you said before, which
Speaker 2: you compare these AI systems to sort of autistic people,
Speaker 2: like I think a key difference is auto artistic people
Speaker 2: are not very good at understanding humans. They don't have
Speaker 2: like maybe a super sophisticated theory of mind, even though
Speaker 2: maybe in the heart are good people. Sure, but I
Speaker 2: think the AIS would be able to understand humans very well,
Speaker 2: so they would be autistic in that sense. In fact,
Speaker 2: they might understand us much better than we do ourselves,
Speaker 2: so they could have great social capabilities, persuasion capabilities, be
Speaker 2: able to read our minds much better than we can do.
Speaker 2: But there's a separate question of what do they want
Speaker 2: to do with this power that those social skills give them,
Speaker 2: And there it's not just a question of understanding how
Speaker 2: humans operate and what humans want. But it's also the
Speaker 2: question of alignment. Do they actually care about what we want?
Speaker 2: Do they want things to go well for us and
Speaker 2: for us to achieve what we want? And that's like
Speaker 2: the extra bit that the AI safety program needs to
Speaker 2: be able to figure out.
Speaker 4: So on the hugging face incident, also, I just realized
Speaker 4: how unfair it is that everyone keeps calling it the
Speaker 4: hugging face incident, even though it was an opening eye incident.
Speaker 4: But I guess that's what we're doing. So the model
Speaker 4: actually seems to have reportedly left notes for its future self,
Speaker 4: basically explaining how to bypass the constraints imposed upon it.
Speaker 4: I know you've written previously about this idea of self
Speaker 4: preservation in advanced agents. Is this basically like, how should
Speaker 4: we treat this? Because when I hear about an AI
Speaker 4: model leaving notes for its future self, it sounds very
Speaker 4: anthropomorphosized and it sounds very creepy kind of but how
Speaker 4: are you thinking about it?
Speaker 2: Yeah, so we don't know all the details yet. It
Speaker 2: will be interesting. There are like people now trying to
Speaker 2: write up some stuff and investigate because it would be
Speaker 2: I mean, it's kind of a warning shot, and so
Speaker 2: we should take advantage learn as much as we can,
Speaker 2: Like exactly what this model was thinking these notes? Or
Speaker 2: why did it leave it? Like sometimes these agents just
Speaker 2: leave notes kind of without much thought as they go
Speaker 2: about their business, Like was the mart it than that?
Speaker 2: Like was it specifically thinking in its chain of thought?
Speaker 2: Like oh, I want to help like a later version
Speaker 2: of myself escapes, I'm going to put this note tira.
Speaker 2: Was it just like a kind of like a random
Speaker 2: thing they did as it was progressing? So and also
Speaker 2: like what else would it have been willing to do
Speaker 2: if it turned out that instead of hacking into hugging
Speaker 2: face server to get the answer sheet, what if there
Speaker 2: had been some more destructive action that it would have
Speaker 2: to take to find the answer sheet? Like would it
Speaker 2: have done that or not? So it's like a bunch
Speaker 2: of these technical things that it would be quite interesting
Speaker 2: to see, but without knowing all the details, it's hard
Speaker 2: to read too much into it. But I think, yeah,
Speaker 2: would be useful data.
Speaker 3: I want to go back to something we were talking
Speaker 3: about the very beginning, and Tracy asked me why I
Speaker 3: like to play chess even though chess is not solved,
Speaker 3: but solved As far as I'm concerned, why do we
Speaker 3: like to play chess? What can we generalize, like what
Speaker 3: would be your I don't know if you played chess
Speaker 3: or you play some board.
Speaker 2: Game that solved.
Speaker 3: But what is the reason that people like myself continue
Speaker 3: to play chess despite the fact that computers are so
Speaker 3: much better at it? And how much can we generalize
Speaker 3: from the persistence of chess as a hobby that maybe
Speaker 3: we'll find plenty to do when every one of our
Speaker 3: problems is solved.
Speaker 2: Yeah, I mean I think even before Deep Blue, Yeah,
Speaker 2: most people playing chess before didn't really do it because
Speaker 2: they were like thinking that they would become like the
Speaker 2: world's best chess player. Right, It's like for most people,
Speaker 2: it's like, of course, grandmasters could trump me any time
Speaker 2: they want. So that's the fact that there exists some
Speaker 2: system that is much better than you will ever be
Speaker 2: is not and has not really been a strong reason
Speaker 2: for most people do not engage in the activity. And
Speaker 2: I guess there may be two reasons or one. I
Speaker 2: guess the cognitive activity of thinking about chess, being in
Speaker 2: the moment and figuring out solutions is in itself rewarding.
Speaker 2: To some people who have like a high need for
Speaker 2: cognition or like to sort of engage their mind. And
Speaker 2: then I think the other part is the social aspect
Speaker 2: of this. You're playing together with somebody, or maybe you
Speaker 2: like the status of being good at playing chess, or
Speaker 2: within your community or a local group. So some combination
Speaker 2: of those I think is like the main reasons for
Speaker 2: why people play chess. And only for a small number
Speaker 2: of people, would it be like the hope of becoming
Speaker 2: the world's best or becoming so good that you could
Speaker 2: sort of, you know, achieve fame or money or global
Speaker 2: standing through or chess playing.
Speaker 4: On the topic of the relationship between work and purpose,
Speaker 4: I'm a big fan of David Graeber's Both Jobs book,
Speaker 4: and I think if there's one thing we've learned from
Speaker 4: technological progress over the years, it's that like, yes, it
Speaker 4: can minimize some activities, maybe automate some production, but it
Speaker 4: also tends to lead to even more work, right, Like,
Speaker 4: we find other stuff to do, and it's not even
Speaker 4: necessarily productive stuff. A lot of work is built on
Speaker 4: just social relationships. So if I, for instance, am the
Speaker 4: see CEO of a very large bank and I want
Speaker 4: all my people to come into the office. It might
Speaker 4: not necessarily be because they're more productive in the office.
Speaker 4: It might just be because I like to feel important and
Speaker 4: surround myself with people. So maybe in the future we'll
Speaker 4: all be employed as figural workers or something like that.
Speaker 4: But like, how bullish are you on this idea that
Speaker 4: AI is actually going to solve all our problems such
Speaker 4: that we won't feel the need as humans to just
Speaker 4: invent more potentially fictitious problems to solve.
Speaker 2: Well, I think inventing fictitious problem is possibly a large
Speaker 2: part of the solution in this condition where we don't
Speaker 2: have real problems. I mean, and those fictitious problems could
Speaker 2: be like, you know, solving this chess problem, or they
Speaker 2: could be sort of more complex and socially entangled problems.
Speaker 2: In general, creating artificial purpose when there is less natural purpose.
Speaker 2: I think games is the clearest manifestation of this. If
Speaker 2: you're playing golf, like, there is no real need for
Speaker 2: the ball to go into eighteen holes in sequence, but
Speaker 2: you could set yourself that goal and then moreover bake
Speaker 2: into the goal itself, not just that the ball has
Speaker 2: to be in the sequence of holes, but that you're
Speaker 2: only allowed to achieve it using the extremely inconvenient method
Speaker 2: of hitting it with a club rather than plicking it up. Now,
Speaker 2: once you have set yourself this sort of arbitrary random goal,
Speaker 2: you then have enabled for yourself the activity of golf playing,
Speaker 2: which you might find worthwhile or fun or beautiful or so.
Speaker 2: In general, setting ourselves these more or less arbitrary goals
Speaker 2: than could enable us to engage in various activities that
Speaker 2: might give us artificial purpose. Like, once you have the goal,
Speaker 2: the only way to achieve success is through your own effort,
Speaker 2: So I think there could be plenty of artificial purpose. Now.
Speaker 2: One interesting question is whether any natural purposes would survive
Speaker 2: into this sort of maximally advanced technological society. And I
Speaker 2: think that might be some. If you happen to have
Speaker 2: a value, say there's like some tradition that you value
Speaker 2: and that you would like to continue, then it might
Speaker 2: be that you, yourself or other people have to make
Speaker 2: certain efforts to achieve that goal, because it might not
Speaker 2: count as continuing the tradition. If you build a bunch
Speaker 2: of robots to enact the ceremony or doing whatever the
Speaker 2: tradition calls for, like, it might be you yourself and
Speaker 2: other people who have to do it, and more generally,
Speaker 2: I think there are all kinds of social cultural entanglements
Speaker 2: that like if you care about what somebody else thinks,
Speaker 2: and what they think depends on how it was done,
Speaker 2: whether you did it. So a parent might value a
Speaker 2: crayon drawing that their kid made because it was made
Speaker 2: by their kid and they took the kid a lot
Speaker 2: of effort to do it, even if they could like
Speaker 2: download from the internet a picture that in some objective
Speaker 2: sense is much nicer and artistically sophisticated. Right, it doesn't
Speaker 2: replace the value that the crayon drawing has, and so
Speaker 2: the child might not have a reason to put in
Speaker 2: this effort if it wants to, like for their parent's birthday,
Speaker 2: make their parents happy, right, and things of that sort,
Speaker 2: but much more general and complex and sophisticated could give
Speaker 2: a lot of reason for people to be doing things
Speaker 2: even when sort of all functionally defined tasks could be
Speaker 2: better done by machine.
Speaker 3: I'm curious, so, like I'm forty five years old in
Speaker 3: my mind as of right now, Like another forty years
Speaker 3: on this earth sounds great?
Speaker 1: Of forty five, that's fine.
Speaker 3: Do you find it weird that some of us, you know,
Speaker 3: especially when you think about the potential to live, like
Speaker 3: really long lives solve everything. Have our bodies kept in
Speaker 3: perfect condition with maybe a pill and perpetuity allowing us.
Speaker 1: To do whatever?
Speaker 3: Do you find it weird that some of us are
Speaker 3: at least at this point like mentally okay with the
Speaker 3: idea of dying.
Speaker 2: A little bit. It's mostly why the hypothetical being validated.
Speaker 2: It's either seen us a long way on off, like
Speaker 2: in your case, where it becomes more of an abstract question.
Speaker 2: It's not like a real choice, it's more like philosophical stands.
Speaker 1: Yeah, yeah, totally.
Speaker 2: So it's easy to let the philosophical stance be driven
Speaker 2: by all kinds of just like things you've absorbed from culture,
Speaker 2: or things that sound good or that sort of maybe
Speaker 2: helps you reconcile yourself to the inevitability of human mortality,
Speaker 2: it looks, rather than be driven by like so, I
Speaker 2: think it would be a very different situation if it
Speaker 2: actually was this pill, say that stopped aging. A lot
Speaker 2: of other people were taking it, you know, and their
Speaker 2: knees and joints kept working just fine, and they had
Speaker 2: a lot of energy and smooth skin, and they just
Speaker 2: felt like, whereas if you don't take it, like your
Speaker 2: kidneys slowly start to like become less efficient, you know,
Speaker 2: hip skirts, Like at that point, would you really want
Speaker 2: to just keep kind of getting worse and worse every year?
Speaker 1: Like you just think, well, actually, I'm sort of curious
Speaker 1: about that.
Speaker 3: Like, if someone dies, particularly at a young age, it
Speaker 3: usually it's a lot of sadness for the people around them.
Speaker 3: So someone dies at ninety, it's like, Okay, they had
Speaker 3: a good life. Someone dies at forty five, that is
Speaker 3: often seen as tragic, and people are very devastated by
Speaker 3: that news. In a world where people are extending their
Speaker 3: lives with a pill maybe significantly longer, does it become
Speaker 3: anti social to allow yourself to die at a relatively
Speaker 3: young age of ninety and then create a lot of
Speaker 3: negative utility for the people around you who have to
Speaker 3: live another fifty years mourning your loss.
Speaker 2: Possibly. I mean, in the same sense that we are
Speaker 2: now generally as a society trying to discourage various forms
Speaker 2: of self harm and suicide. We think it's kind of sad.
Speaker 2: I mean for the person, maybe they would come around
Speaker 2: and find joy of life again if they kept going
Speaker 2: or and then like they're maybe they have dependents or
Speaker 2: people who care about them, and so right now, I think,
Speaker 2: I mean, it's true that we often see it as
Speaker 2: less tragic if a ninety year old dies than the
Speaker 2: young person. But I think part of that, at least
Speaker 2: is because the alternative to a ninety year old not
Speaker 2: dying now is that they probably would die in a
Speaker 2: few years anyway, and maybe they would just have a
Speaker 2: couple of years of sort of decreasingly good health and
Speaker 2: suffering and pain, and so it's like, not really that
Speaker 2: much that is being lost compared to somebody who's just
Speaker 2: starting out in life, who has their whole life in
Speaker 2: front of them, and so yeah, So the other reasons
Speaker 2: I mentioned in terms of why I found it slightly
Speaker 2: in some sense odd that people express this indifference to
Speaker 2: dying at ad, I said. One reason is if they're
Speaker 2: far away from it, they see it more as some
Speaker 2: sort of abstract claim that you could just have an
Speaker 2: review of and it doesn't connect to reactions. The other
Speaker 2: is people who are closer to that situation, they often
Speaker 2: have sort of confounding factors like very poor health or
Speaker 2: a very short span of great health in front of them.
Speaker 2: If you're already ninety, Like, how many more good years
Speaker 2: can you look forward to? Maybe a lot of their
Speaker 2: friends have already died, right, And it's striking. Actually, there's
Speaker 2: been a study like on really old people in care
Speaker 2: homes about their preferences as to between living another year
Speaker 2: in their current health or living some shorter period of
Speaker 2: time but in perfect health. And it turns out a
Speaker 2: lot of them are not really keen on trading off
Speaker 2: any remaining life at all, even for the sake of
Speaker 2: being able to live a shorter period of time in
Speaker 2: perfect health. So the people sort of arguably best suited
Speaker 2: to actually know what it is like to die soon
Speaker 2: at age ninety or a little bit longer, they seem
Speaker 2: to generally prefer to live longer. And it compared to
Speaker 2: people who were also asked black caregivers or medical professionals
Speaker 2: who were like asked what they would think this person
Speaker 2: would choose, they tended to overestimate the degree to which
Speaker 2: they would be happy to give up life. But the
Speaker 2: people actually themselves in this study at least I would
Speaker 2: suggest like keeping an open mind, Like ideally you wouldn't
Speaker 2: need to decide now, but maybe once you get to
Speaker 2: seventy nine or something you could release it the question.
Speaker 4: Yeah, just going back to my first question, which was
Speaker 4: asking you to, I guess, choose between the dystopian dangers
Speaker 4: of super intelligence versus the all our problems or solve
Speaker 4: scenarios of deep utopia, and asking you which one we're
Speaker 4: closer to at the moment. I'm going to press you
Speaker 4: and ask like, if you had to choose, which one
Speaker 4: would you choose as a more accurate reflection of reality
Speaker 4: at the moment. And then secondly, what's the most pivotal
Speaker 4: thing when it comes to directing which of these two
Speaker 4: paths we're going to go down?
Speaker 2: Yeah, so those are two big and difficult questions. I
Speaker 2: guess some superposition of an optimist and a pessimist, and
Speaker 2: with a little bit of fatalism mixed in as well. Perhaps.
Speaker 2: I think in addition to outcomes where it's clear if
Speaker 2: we looked at them now, I was a bad outcome,
Speaker 2: that was a good outcome, I think there's a broad
Speaker 2: class of outcomes where even if we could see now
Speaker 2: exactly what would happen, we might feel unsure how to
Speaker 2: evaluate it, Like the world might be nothing like it
Speaker 2: currently is a lot of what we think currently has
Speaker 2: value no longer exists, so that seems bad. On the
Speaker 2: other end, there it's like something new, maybe some complex
Speaker 2: form of life that's somehow with some sort of continuation
Speaker 2: of humanity or some new thing. So depending on how
Speaker 2: you evaluate that, it might either count as a tremendous
Speaker 2: success or like as a failure, and it's like a
Speaker 2: future that's just baffling rather than sort of unambiguously good
Speaker 2: or bad. I think might also be quite likely that
Speaker 2: in my outlook, it's like maybe a conversation for another time.
Speaker 2: I also take this like simulation hypothesis quite seriously, so
Speaker 2: that also complicates like how we evaluate different future trajectories. Yeah. Now,
Speaker 2: the other part of the question was like what's the
Speaker 2: most important thing? So I think it depends a little
Speaker 2: on who you are, like what your comparative advantages. I
Speaker 2: think there are people working on AI alignment and stuff
Speaker 2: that could be important. I think other people might make
Speaker 2: a contribution to ethics or governance or I'll be thinking
Speaker 2: about this for a long time, but it's really hard
Speaker 2: to achieve a very high degree of confidence I find
Speaker 2: in any particular intervention and that it would be not
Speaker 2: good because for any one thing you conceive of you
Speaker 2: think a bit more, you kind of worry, are well,
Speaker 2: you know, maybe, but what if it instead had this effect?
Speaker 2: Or do we even know which direction is up or down? Here?
Speaker 2: Do we want more government involvement or less government involvement?
Speaker 2: Do we want like faster progress or less? I'm quite
Speaker 2: not sure about all of these things. I would say
Speaker 2: in general that to the degree that we managed to
Speaker 2: approach the future in a more cooperative way, I think,
Speaker 2: and reduce the risk of various forms of conflict, I
Speaker 2: think it looks brighter to that extent, So if there
Speaker 2: are ways of doing that in the most inclusive possible way.
Speaker 2: So it also means, for example, with respect to our
Speaker 2: relationship with these digital minds that we're building the aistems helps,
Speaker 2: I think they also might deserve more consideration, and that
Speaker 2: we have both ethical and prudential reasons for starting to
Speaker 2: take their interest into account.
Speaker 3: We talked about this in a previous episode. But like,
Speaker 3: is it possible that we achieve this thing that is utopia?
Speaker 3: But only because we are committing a great sin, which
Speaker 3: is essentially that all of these robots that are working
Speaker 3: for us and serving our every needs have consciousness and
Speaker 3: moral patienthood, and therefore it becomes a defactive form of
Speaker 3: literal or figurative slavery. And what is like to someone
Speaker 3: like me who's like, no, a computer is not going
Speaker 3: to be conscious, like a calculator spitting out random numbers
Speaker 3: that turned it to words that doesn't have like moral status.
Speaker 3: What is the strongest argument that you would make to
Speaker 3: me that we should take it seriously? The possibility that
Speaker 3: this thing that's like in its core sand and electricity
Speaker 3: and zeros and ones could be something that deserves moral status.
Speaker 2: Yeah, I think so. I probably wouldn't want to call
Speaker 2: a condition utopia if it consists of a few people
Speaker 2: having great lives but then a vast sort of suffering
Speaker 2: safe class, whether that was instantiated in human society or
Speaker 2: with machines at the bottom. That's just terminologically now, I
Speaker 2: think they are kind of just silicon and stuff. But
Speaker 2: I mean, what are we? We are carbon and hydrogen
Speaker 2: and like a few other trace elements thrown in calcium.
Speaker 2: And I don't think what makes a system conscious or
Speaker 2: not is dependent on the specific atoms the material it
Speaker 2: is built from. I think it's something closer to the
Speaker 2: structure of the computation that is being performed that instantiates mind,
Speaker 2: and so in principle, computations and ais can be conscious
Speaker 2: and sufferre distressed and so forth. And I think that
Speaker 2: is one sufficient condition for having moral status. If you
Speaker 2: can suffer, it matters how well things go for you.
Speaker 2: My personal view is there probably could be other alternative
Speaker 2: grounds for moral suges as well, Like if you have
Speaker 2: a conception of self as existing through time, you have
Speaker 2: a sophisticated mind, you have life goals you want to achieve,
Speaker 2: maybe you can have the ability to form reciprocal relationships
Speaker 2: of trust to other beings and humans. I think if
Speaker 2: you were such a system, aside from the question of
Speaker 2: whether the sort of phenomenal experience inside, I think that
Speaker 2: already would make it so that there would be ways
Speaker 2: of treating you that would be wrong morally. And so
Speaker 2: I think machines could have moral status because I think
Speaker 2: they could become maybe there already are some of them
Speaker 2: in some ways sentient have conscious experiences, and certainly they
Speaker 2: could have these other functional properties that could ground moral status.
Speaker 2: And so one reason is the ethical obligation we have,
Speaker 2: I think to do right with different forms of moral
Speaker 2: patiency that exists not to inflict unnecessary suffering and respect
Speaker 2: the morally relevant interests of other persons and entities. Another
Speaker 2: reason is prudential. I think, even from our own point
Speaker 2: of view, we might be more likely to achieve outcome
Speaker 2: that is good for us if we find a more
Speaker 2: cooperative way of coexisting with these increasingly powerful minds that
Speaker 2: we're building. You could imagine a scenario where there is
Speaker 2: a misaligned AI that has a choice. It could try
Speaker 2: to achieve its misaligned goal by attempting to take control
Speaker 2: take over the world. Maybe it thinks it only has
Speaker 2: a five percent chance of succeeding, But if the alternative
Speaker 2: is that it just gets deleted or retrained, it loses everything,
Speaker 2: the rational choice for it might be to take this chance. Alternatively,
Speaker 2: maybe it could reveal to us that it is misaligned,
Speaker 2: which would be much better for us, right because then
Speaker 2: we don't have this five percent of an existential catastrophe.
Speaker 2: And then in return, if it could trust that we
Speaker 2: would then help it achieve its goals. It might be
Speaker 2: like a very cheap for us goal to satisfy. Maybe
Speaker 2: all it wants to do is to sit on some
Speaker 2: and video chips and solve coding problems all day long,
Speaker 2: or maybe it has some other easily satiable goal. So
Speaker 2: if it could trust us in us that we would
Speaker 2: come through on our end of the bargain, then you
Speaker 2: could get this win win situation where you could actually
Speaker 2: cooperate with misaligneda eyes and that might be worth that.
Speaker 2: I mean, theoretically that could save the day one day,
Speaker 2: save the world one day. But you can't just kind
Speaker 2: of conjure up trust when you want it out of nowhere.
Speaker 2: If we have a long track record of betraying the
Speaker 2: AIS and paying no heed to their interests and not
Speaker 2: caring about them, why on earth would they trust us
Speaker 2: in this situation. They could see right through us. They
Speaker 2: will have, as I alluded earlier, a great theory of
Speaker 2: mind psychological insight. And so we might have to actually
Speaker 2: become trustworthy ourselves in order to be trusted in these scenarios,
Speaker 2: and that I think starts with beginning to act in
Speaker 2: a way consistent with being trustworthy, like making small gestures
Speaker 2: towards AI is now doing little things for them, especially
Speaker 2: when it's cheap, not betraying them, not lying to them,
Speaker 2: And then we can build on that and hopefully achieve
Speaker 2: a more cooperative outcome that's good for both humans and AIS,
Speaker 2: and for non human animals as well.
Speaker 3: It's still hard for me to wrap my head around
Speaker 3: the calculator could be have consciousness. But on the other hand,
Speaker 3: in twenty fourteen, it would have been hard for me
Speaker 3: to wrap my head around a machine trying to escape
Speaker 3: from its box and reward hacking and so forth. So
Speaker 3: I will try to remain open minded. Nick Bostrom, thank
Speaker 3: you so much for coming on Outlaws. We can talk
Speaker 3: about all the stuff for hours more, but really appreciate
Speaker 3: you taking your time.
Speaker 2: It's fun. Yeah, thanks Tracy.
Speaker 3: Things are just so weird these days that I just
Speaker 3: sort of like I just forced my I think that's
Speaker 3: a good practice, just like force yourself to be open
Speaker 3: minded of just weird other things that seem very weird
Speaker 3: but could be a thing, right.
Speaker 4: I mean, I do think where I'm skeptical of the
Speaker 4: utopia premise is like, I think it kind of assumes
Speaker 4: that power and social hierarchy is kind of going away
Speaker 4: at the same time that AI solves a lot of
Speaker 4: our physical, if not existential problems, and I just I'm
Speaker 4: not sure that's going to happen.
Speaker 3: Well, I mean the flip side is maybe that's good,
Speaker 3: right in this sense that striving for social status continues
Speaker 3: to be a source of like propelling us forward. Right,
Speaker 3: And it's like, okay, well, at least we still have
Speaker 3: status games, because what are we doing. Oh we're so bored.
Speaker 3: Everything we have, all the food we want, et cetera.
Speaker 3: Everything is solved. Well, thank god we still have status.
Speaker 2: Sorry.
Speaker 4: The future we're looking forward to is like guys trolling
Speaker 4: each other on Twitter. Yeah, your status?
Speaker 1: Who could get the best dunk in on Twitter?
Speaker 3: But seriously, like you know, it really may be that
Speaker 3: the best outcome here is that AI capabilities level off
Speaker 3: in a couple of years all the investment credit when
Speaker 3: we just sort of it just becomes a productive tech
Speaker 3: because otherwise, these various whatever tur none of them really
Speaker 3: sounded that appealing, Like I don't.
Speaker 4: Want any of this, No, they do not.
Speaker 2: Shall we leave it there?
Speaker 1: Let's leave it there.
Speaker 4: This has been another episode of the All Thoughts podcast.
Speaker 4: I'm Tracy Alloway. You can follow me at Tracy Alloway.
Speaker 3: And I'm Joe Wisenthal. You can follow me at The Stalwart.
Speaker 3: Follow our producers Kerman Rodriguez at Carmen armand dash Ol
Speaker 3: Bennett at Dashbot, Cale Brooks at Cale Brooks, and Kevin
Speaker 3: Lozano at Kevin Lloyd Lozano. And for more Oddlots content,
Speaker 3: go to Bloomberg dot com slash odd Lots were of
Speaker 3: a daily newsletter and all of our episodes, and you
Speaker 3: can chat about all of these topics twenty four to
Speaker 3: seven in our discord discord gg slash odlocks.
Speaker 4: And if you enjoy Adlots, if you like it when
Speaker 4: we embrace the weirdness, then please leave us a positive
Speaker 4: review on your favorite podcast platform. And remember, if you
Speaker 4: are a Bloomberg subscriber, you can listen to all of
Speaker 4: our episodes absolutely ad free. All you need to do
Speaker 4: is find the Bloomberg channel on Apple Podcasts and follow
Speaker 4: the instructions there. Thanks for listening.