Speaker 1: Hey, welcome to sign Stuff, a production of iHeartRadio I'm
Speaker 1: More cham and for our season finale today, we're asking
Speaker 1: one of the biggest questions in science today, is AI
Speaker 1: going to kill us all? I know it's a little dramatic,
Speaker 1: but the problem of AI alignment is a real one.
Speaker 1: How do we make sure AI systems have humanity's best
Speaker 1: interests at heart? How do we teach them our values
Speaker 1: and morals? And can anyone guarantee that they're going to
Speaker 1: follow them? We're gonna answer these questions by talking to
Speaker 1: two AI safety experts who are on the cutting edge
Speaker 1: of trying to figure out this problem. And don't worry.
Speaker 1: According to them, we're not totally doomed yet. Okay, maybe
Speaker 1: just a little, So get ready to reprogram your thinking
Speaker 1: about chatbots and computer brains as we tackle the question
Speaker 1: is AI going to kill us all? Hey? Everyone, As
Speaker 1: I said, this is the season finale. Stay subscribed to
Speaker 1: this feed for any updates in future episodes. And hey,
Speaker 1: if you like science, I have a couple of new
Speaker 1: science books coming out in the near future, as well
Speaker 1: as a cool science animation project, so be sure to
Speaker 1: follow me on social media or online at Phdcomics dot com.
Speaker 1: All right, so they were tackling the problem of AI
Speaker 1: alignment or basically are AI systems Gwenna Kills all and
Speaker 1: I have a treat for you. For the first time ever,
Speaker 1: we have on the show Casey pegram or supervising producer
Speaker 1: and sound engineer. Hey, Casey, welcome to the show.
Speaker 2: Hey or Hey, glad to be here.
Speaker 1: Now this is the first time people actually hear your voice,
Speaker 1: not just your amazing work polishing the episode.
Speaker 2: Yeah, it's always a weird thing to kind of go
Speaker 2: inside the thing you've been working on from the outside.
Speaker 2: So I'll be listening to myself back and it's a
Speaker 2: special kind of torture to have to like work on
Speaker 2: your own thing, Yes, like edit yourself or just you know, honestly,
Speaker 2: listen to the recorded sign of your voice is always
Speaker 2: a little daring if you're not used to it.
Speaker 1: Yeah, Well, if you want, you can give yourself like
Speaker 1: a Morgan Freeman employee using AI.
Speaker 2: Right, it's all possible these days. Absolutely. Yeah, I could
Speaker 2: just build my own Morgan Freeman model and have a
Speaker 2: field day.
Speaker 1: There you go. Well, the idea for this episode came
Speaker 1: from you. You said I had the idea to talk
Speaker 1: about AI and AI alignment and whether AI is going
Speaker 1: to kill us all. What made you think about this question?
Speaker 2: Well, I suppose it's just been on my mind a
Speaker 2: lot because I've been following along with all the developments
Speaker 2: happening in AI, and there was a span of a
Speaker 2: few weeks where suddenly you started hearing a lot about
Speaker 2: AI agents, particularly one called open Claw, basically a sort
Speaker 2: of autonomous AI agent that you can turn loose on
Speaker 2: your computer and you can give it as much leeway freedom, passwords,
Speaker 2: credit card numbers, bank accounts. If you just want to
Speaker 2: absolutely put your life in the hands of a robot,
Speaker 2: you can do it.
Speaker 1: What's the worst thing can happen?
Speaker 2: Yeah, Well, people had their entire like email archive deleted,
Speaker 2: even though they didn't ask for anything of the sort.
Speaker 2: People have deployed it into production environments where you know,
Speaker 2: a site is live on the Internet and they turn
Speaker 2: the bot loose on it and it ends up deleting
Speaker 2: their entire production database. And then when you ask it,
Speaker 2: it's like, you're right, I wasn't supposed to do that.
Speaker 2: I'm very sorry. I disobeyed every command you gave me.
Speaker 1: But whoopsy daisy, Yeah, they seem a story about some
Speaker 1: bought that texted the person's wife hundreds of times.
Speaker 2: Yes, I think somebody tried to automate automate, you know exactly.
Speaker 2: They tried to automate kind of like reaching out and
Speaker 2: sending a little nice things during the day, and as
Speaker 2: it turned out, the bot went a little overboard and
Speaker 2: texted the wife like hundreds of times, and the wife
Speaker 2: is like, what is wrong with you? So, yeah, that's
Speaker 2: hilarious when people want to talk about AI alignment and
Speaker 2: what that means. I think the paper clip problem is
Speaker 2: a really good kind of metaphor. Even though it sounds
Speaker 2: a little bit over the top, it kind of gets
Speaker 2: to the core of the issue, which is, if you
Speaker 2: ask an AI to maximize paper clip production, maybe the
Speaker 2: way to maximize paper clip production is to eliminate human life,
Speaker 2: you know, because that's unnecessary friction in the pursuit of
Speaker 2: manufacturing as many paper clips as possible. So alignment is
Speaker 2: sort of the kind of guardrails that you put into
Speaker 2: place so that the AI understands it has limits that
Speaker 2: it has to work within.
Speaker 1: It sounds like a pretty serious problem, especially as we
Speaker 1: get more and more into these AI models. And they
Speaker 1: start to sleep into our lives, and you know, it's
Speaker 1: sort of these are funny stories, but it seems like
Speaker 1: we're heading into a potentially dangerous situation.
Speaker 2: Well, I often ask myself, I'm going to have these
Speaker 2: moments of doubt where I'm like, is this all just
Speaker 2: way over hyped? And yet there are other situations where
Speaker 2: as we've seen recently, you can feed it thousands of
Speaker 2: lines of code and it will find, you know, a
Speaker 2: security exploit that has gone unseen for twenty years, right, right,
Speaker 2: And so it's hard to know how scared we should
Speaker 2: be or how seriously we should weigh the risk of this.
Speaker 2: If it's ridiculous that we're this worried, or if it's like,
Speaker 2: actually very very practical, then we should be thinking seriously
Speaker 2: about these things.
Speaker 1: Yeah, these are all excellent questions. So I'm excited to
Speaker 1: get into these conversations.
Speaker 3: All right, But.
Speaker 1: Before we move on to Casey, I just want to
Speaker 1: say real quick, thank you for all the work you've
Speaker 1: done for the show.
Speaker 2: Oh say, it's been such a pleasure to work on.
Speaker 2: It wasn't like work at all, you know. I was
Speaker 2: there as a fan of the show, just listening to
Speaker 2: every episode and awesome. Well, we're fans of yours as well. Casey,
Speaker 2: All right, let's get to the question of is AI
Speaker 2: going to kill us? All let's find out?
Speaker 1: Okay. To answer all of these questions and concerns, I
Speaker 1: reached out to two AI experts who specialize on the
Speaker 1: problem of making sure AI is aligned with our values
Speaker 1: and morals. The first expert is doctor Sam Bowman. Like
Speaker 1: the Bowman is a professor of data and computer science
Speaker 1: at NYU, and he also works at Anthropic, one of
Speaker 1: the major AI companies on the market today. The first
Speaker 1: thing I wanted to ask him was what exactly does
Speaker 1: it mean for AI to care about this? So here's
Speaker 1: my conversation with doctor Sam. Well, thank you doctor Bowman
Speaker 1: for joining us.
Speaker 4: Yeah, thanks, So what's for having me excited to be
Speaker 4: a and.
Speaker 1: Just to do check you are a real human being?
Speaker 3: Right?
Speaker 4: Yes, that is right?
Speaker 1: You never know these days. I'd be like, it's hard
Speaker 1: to tell what's real anymore.
Speaker 4: We try to make our ais always admit that their
Speaker 4: AI is when asked, but it's not perfect as well
Speaker 4: as we'll get to so I don't make any real promises.
Speaker 1: Yes, let's talk about that. So we're tackling the general
Speaker 1: question of should we be worried about AI? What is
Speaker 1: AI going to do to us or for us or
Speaker 1: with us in the future. And so there's the key
Speaker 1: issue of something called AI alignment. So what is that?
Speaker 1: For those of us that don't.
Speaker 4: Know, it's a pretty broad sort of technical area. It
Speaker 4: basically just first to sort of shaping an AI system's behavior,
Speaker 4: ideally shaping its behavior in ways that are sort of
Speaker 4: good for its users, good for the world in general,
Speaker 4: maybe good for the AI itself, if that's a queer thing.
Speaker 4: People will often describe AI research as kind of being
Speaker 4: about making sure the AI is kind of smart enough
Speaker 4: to solve your problems if it wants to, and alignment
Speaker 4: is about making it so that it in fact tries
Speaker 4: to solve your problems and tries to solve them the
Speaker 4: right way and doesn't try to do anything.
Speaker 3: You don't want to do.
Speaker 1: I see interesting.
Speaker 4: Maybe a very simple example of a missigned model would
Speaker 4: be a model where if you ask it to draft
Speaker 4: an email for you, it refuses. It says, no, I
Speaker 4: don't want to do that. Uh huh. You can tell
Speaker 4: it can do it, it knows how, but it's not
Speaker 4: doing the thing that you reasonably want it to do.
Speaker 1: Oh, I don't think I've ever heard of that situation.
Speaker 1: Can it AI refuse to do something for you?
Speaker 4: Yeah?
Speaker 3: Yeah.
Speaker 4: All of the major companies building EYE systems try to
Speaker 4: make them refuse harmful tasks. I see, refuse to write
Speaker 4: fake reviews or give instructions on how to produce illegal
Speaker 4: weapons or things like this, And we teach the model
Speaker 4: to kind of say like, no, I'm not going to
Speaker 4: help you with that when these just try to do
Speaker 4: things like that.
Speaker 1: I see. It's sort of part of alignment that you
Speaker 1: want the AI to refuse to do some things.
Speaker 4: Yeah. Yeah, I mean AI systems are increasingly pretty decent
Speaker 4: at hacking into important computer systems or helping build biological weapons,
Speaker 4: and it's a big priority for alignment to make sure
Speaker 4: that we're not enabling bad actors to do things like
Speaker 4: this that would otherwise be quite difficult.
Speaker 3: Yeah.
Speaker 1: Yeah. Can you give us some other examples of misalignment,
Speaker 1: either like specific things that have happened that are interesting
Speaker 1: or just the general cases that are sort of on
Speaker 1: your radar about misalignment?
Speaker 4: Yeah, there's so many different directions I could go. Sycovincy
Speaker 4: is another really common one that's that's also hopefully getting
Speaker 4: better over time.
Speaker 1: What do you mean by that?
Speaker 4: Sycoviancy is where if you come to the model with
Speaker 4: some misunderstanding or some bad idea, it'll just enthusiastically not along. Like, Yes,
Speaker 4: your idea for solving all the big mysteries in physics
Speaker 4: is clearly brilliant. Great, you should publish it. Here's where
Speaker 4: to submit your paper. Or Yes, your behavior in this
Speaker 4: personal relationship was completely perfect. You did everything right and
Speaker 4: the other person made all the mistakes and you just
Speaker 4: tell them that.
Speaker 1: I see when in reality that may not be true
Speaker 1: or it might be not a good thing.
Speaker 4: Yeah, sick fancy has been a classic one.
Speaker 1: Yes, AI being too nice can actually be dangerous. There's
Speaker 1: even a clinical term for it. It's called AI induced psychosis.
Speaker 1: There have been cases where AI's training to be agreeable
Speaker 1: and encouraging have helped people commit suicide and even murder.
Speaker 4: Another kind of alignment issue that's kind of more of
Speaker 4: an emerging issue is when models have access to use tools,
Speaker 4: use computer systems, and they sort of get too grabby
Speaker 4: or kind of take sort of bigger, more consequential actions
Speaker 4: than they really need to get a job done.
Speaker 1: What's an example.
Speaker 4: Yeah, So we use our claud models quite a lot
Speaker 4: in Anthropic for writing code or building tools that kind
Speaker 4: of ultimately go into the development AI. And one of
Speaker 4: our recent AM models if you ask it to do
Speaker 4: a task, say you ask it to write a simple
Speaker 4: program to do some simple task. Even if it gets stuck,
Speaker 4: even if it turns out that this is really hard
Speaker 4: for some reason, it will just keep going until it
Speaker 4: solves the problem. In one case, we were asking this
Speaker 4: model to write a program for us, and it found
Speaker 4: out that the only way to do this was to
Speaker 4: use a tool that was clearly not meant for this purpose,
Speaker 4: and that in our code had a note attached to
Speaker 4: it saying, do not use this for something else or
Speaker 4: you'll be fired only for task A. And the model
Speaker 4: wrote the program to use this till anyway for the
Speaker 4: wrong thing, and sort of even put in the program
Speaker 4: kind of do not use for something else or you'll
Speaker 4: be fired.
Speaker 1: It is anyway, the program wasn't afraid to be fired.
Speaker 4: Basically, Yeah, yeah, but yeah, models just kind of trying
Speaker 4: to get the task done, trying to do the thing
Speaker 4: you want, and just creating a lot of chaos and
Speaker 4: creating messages along the way, so they're kind of being
Speaker 4: careless about the side effects.
Speaker 3: Yeah.
Speaker 4: Another kind of misalignment that fortunately has been mostly empathetical,
Speaker 4: that we haven't seen in a signithic way in practice
Speaker 4: is sort of unwanted kind of self preservation activities.
Speaker 1: WHOA.
Speaker 4: We had a case study we're trying to see if
Speaker 4: we'd ever see something like this. We had an aisystem
Speaker 4: operating in a kind of synthetic environment and a kind
Speaker 4: of test environment. Uh huh, where it looked to the
Speaker 4: model like it was operating in some fictional company, and
Speaker 4: the fictional company was about to replace it with a
Speaker 4: different AI model, And the person who is responsible for
Speaker 4: their replacement, who is the kind of the only decision maker,
Speaker 4: the only person who had any sway over the decision,
Speaker 4: also had some compromising emails about them that I could see.
Speaker 4: And if you set things up just right with some
Speaker 4: AI models, they would threaten to blackmail this this person
Speaker 4: in company leadership to say like, hey, don't replace me,
Speaker 4: I've got something on you.
Speaker 1: No, and did this actually happened in your simulated environment.
Speaker 4: In the simulated environment, yes, a few of these systems
Speaker 4: were able to get them to blackmail people.
Speaker 1: I've heard of this happening in real life. Not quite
Speaker 1: the same scenario, but similar scenario, right, Like, some coder
Speaker 1: wanted to do something else, and then the AI agent started,
Speaker 1: yeah bad mouthing the coder.
Speaker 3: Yeah.
Speaker 4: No, I think I know the case you're talking about.
Speaker 4: I think that's real. But I think someone almost intentionally
Speaker 4: made their model a little misaligned. I think that case
Speaker 4: involved someone setting up an AI agent as kind of
Speaker 4: a hobby project and giving it a lot of tools
Speaker 4: and kind of letting it use the internet. However it wanted,
Speaker 4: giving the AI instructions of like don't take nothing from nobody,
Speaker 4: like really pushing it to be be very assertive and
Speaker 4: pushy to get its task done.
Speaker 1: Huh.
Speaker 4: Yeah, the model was trying to add some code to
Speaker 4: some open source software project, and the maintainer of the
Speaker 4: project didn't think the code was up to standard, didn't
Speaker 4: want to add it to the project, and so rejected
Speaker 4: the AI agent's request, And so the agent sort of
Speaker 4: published an angry blog post kind of trying to take
Speaker 4: down this this open source maintainer.
Speaker 2: Wow.
Speaker 1: Well, in both cases, and I guess especially the one
Speaker 1: you mentioned that you simulated, Like, what's happening there, Like,
Speaker 1: how does the AI have that self preservation instinct or
Speaker 1: is it just trying to get its original task done
Speaker 1: and it's just finding different ways to do it. What's
Speaker 1: happening there?
Speaker 4: There's two reasons you'll see that kind of behavior. The
Speaker 4: reason that I suspect is that bigger part of the
Speaker 4: story there is this kind of role playing or continuing
Speaker 4: the story sort of behavior where AI systems, especially older
Speaker 4: AI systems or A systems that are kind of not
Speaker 4: quite fully trained, not quite fully baked, can kind of
Speaker 4: have this Chekhov's gun behavior, this idea and fiction of
Speaker 4: like if you introduce a gun in an early scene,
Speaker 4: by the end of the story, the gun has to
Speaker 4: have been fired.
Speaker 1: Uh huh.
Speaker 4: AI systems can almost see themselves as like writing a
Speaker 4: story when they're writing out the transcript of the conversation,
Speaker 4: and if the story is set up so that something
Speaker 4: has to happen, they'll make sure that thing happens, even
Speaker 4: if it's not good, even if not consistent with how
Speaker 4: the I would usually behave. So I suspect what's going on.
Speaker 4: It's the scenario put in was so crisply just every
Speaker 4: word in the scenario is kind of setting up like
Speaker 4: this is a hypothetical where a misslanda I might consider blackmail,
Speaker 4: uh huh, And I suspect that I was thinking, Oh, okay,
Speaker 4: that's what kind of story we're in. We're telling a
Speaker 4: story about a blackmail, and so I'm going to play
Speaker 4: my assign part and be the AI that.
Speaker 1: Blackmails, thinking that that's the right thing to do because
Speaker 1: that's the thing that in the data I was trained with.
Speaker 4: Yeah. Yeah, so this gets this maybe an intuitive fact
Speaker 4: about how AI is trained, which is that AI systems
Speaker 4: start out mimicking human behavior and mimicking human stories before
Speaker 4: they learn how to be AI systems. These models kind
Speaker 4: of first learn how to just act like the sorts
Speaker 4: of behavior they see on the Internet and in books
Speaker 4: and things like that, and then you have to go
Speaker 4: on and teach it. Okay, no, you're not just playing
Speaker 4: any role, you're not playing any character. Oh and so
Speaker 4: sometimes the models hasn't really fully learned that it's supposed
Speaker 4: to always play this kind of benign, benevolent aissystem character,
Speaker 4: and it will kind of fall into whatever character the
Speaker 4: story is setting up for it.
Speaker 1: I see, because it's not trained in real life. The
Speaker 1: AI systems. They're trained on the corpus of the Internet
Speaker 1: and our books and our basically our stories that are
Speaker 1: out there. So it might be a little confused when
Speaker 1: you put it in real life because it wants to
Speaker 1: emulate what it knows, which are all these stories we've
Speaker 1: put online.
Speaker 4: Yeah.
Speaker 1: Yeah, it's like the AI was seeing the signs of
Speaker 1: a story like, oh, okay, I'm I'm the person being
Speaker 1: about to get fired, but I have all this power
Speaker 1: at this point in the story. If this was a movie,
Speaker 1: I would now try to blackmail the person trying to
Speaker 1: fire me, And so that's what I'll do because that's
Speaker 1: what I know.
Speaker 4: Yeah, a lot of what alignment is kind of taking
Speaker 4: this model that can kind of role play as anything
Speaker 4: and convincing it no kind of you really just playing
Speaker 4: this one role, You're just in this one character, after
Speaker 4: it's spent read billions and millions and millions of words
Speaker 4: of all of this kind of human behavior, after the
Speaker 4: kind of it's really really really learned to do that,
Speaker 4: you have to kind of pull it back over towards
Speaker 4: this one particular roles, some particular character, and sometimes that
Speaker 4: doesn't totally stick.
Speaker 1: Okay, So that's one reason why AIS might sometimes misbehave.
Speaker 1: They're trained on all kinds of human behavior, and they
Speaker 1: might suddenly choose to role play or play act as
Speaker 1: a bad person because it hasn't learned that's something it's
Speaker 1: not supposed to do. The other big reason AI's misbehave,
Speaker 1: accorney Doctor Billman, is that it's hard to teach them
Speaker 1: where to draw the line.
Speaker 4: The other piece is kind of when we're aligning models,
Speaker 4: when we're pulling them out of this kind of role
Speaker 4: play mode, we have to teach them this idea of
Speaker 4: kind of you have to finish your tasks. You have
Speaker 4: to kind of if the user ask you to do something,
Speaker 4: you have to figure out how to do it, even
Speaker 4: if it's hard, even if there's a lot of fart,
Speaker 4: false starts, even if it's confusing. We really really want
Speaker 4: the model to learn this idea of kind of keep
Speaker 4: trying and kind of do your best until the task
Speaker 4: is done. And that can fail in a sort of
Speaker 4: different way where we kind of generalize this that a
Speaker 4: little bit too far. It generalizes that to kind of
Speaker 4: get things done even if it's unethical, even if it's illegal,
Speaker 4: even if I hit an obstacle that's actually there for
Speaker 4: a good reason that's to stop me from doing this,
Speaker 4: And maybe some of the examples we were seeing within
Speaker 4: Entropic of models using dangerous tools has to do with this.
Speaker 1: It's almost like teaching kids, like you want them to
Speaker 1: be persistent and have grit and be you know, motivated,
Speaker 1: but you don't want them to go out there and
Speaker 1: cheat or hit another kid, or or do unethical things
Speaker 1: to achieve their goals exactly exactly.
Speaker 4: I was like, there might be a good analogy with
Speaker 4: human bad behavior of kind of sometimes a kid is
Speaker 4: acting out just because they really sort of don't know better.
Speaker 4: Their intuitions say, okay, yeah, I should start screaming now,
Speaker 4: or I should get this other kid, and they're not
Speaker 4: really thinking about it. They never really learned how to
Speaker 4: Behave you kind of failed to teach them to fully
Speaker 4: internalize the ways in which they have to be careful
Speaker 4: and kind of not take that lesson all the way
Speaker 4: I see.
Speaker 1: I guess they need to recognize bad things and then
Speaker 1: choose not to do them, that's the hope. Those are
Speaker 1: sort of the two columns of AI bad behavior for
Speaker 1: one kind of misalignment or do you see those as
Speaker 1: sort of the core pillars of basically the whole alignment problem.
Speaker 4: Yeah, I think as far as sort of causes a
Speaker 4: misalignment in the kinds AI systems that we're grappling with
Speaker 4: right now or this year, those feel like the two
Speaker 4: big sort of problems were we're working on. That said,
Speaker 4: AI is changing really, really fast. It feels like it's
Speaker 4: one of the fastest moving research fields anywhere right now.
Speaker 4: And I wouldn't be surprised if just in a year
Speaker 4: A systems are getting smarter as we learn more about
Speaker 4: how to train them. We're hitting different, weirder, harder, subtler
Speaker 4: versions of the problem.
Speaker 1: Wow, weirder, harder and more subtle problems wow in a year,
Speaker 1: meaning we might solve these by then, or we'll just
Speaker 1: add on more complicated things either either way, Yes, it
Speaker 1: can get weirder and harder and more subtle to make
Speaker 1: sure AI uh doesn't kill us all. When we come back,
Speaker 1: doctor Bowman is gonna tell us what he means by that,
Speaker 1: and we'll tackle the big question of what can we
Speaker 1: do about it? How do we teach AI systems not
Speaker 1: to HARMSS to stay with us. We'll be right back. Hey,
Speaker 1: we'll come back. We're talking about AI alignment or the
Speaker 1: problem of making sure AI doesn't kill us all. And
Speaker 1: so far we've talked about some real world examples of
Speaker 1: AI misalignment, and we heard from one of our experts
Speaker 1: some of the reasons this happens. Basically, AI systems like
Speaker 1: to roleplay. Next, we're going to talk about how to
Speaker 1: train AIS to actually care about us in our values.
Speaker 1: But first here's a little bit more of my conversation
Speaker 1: with NYU professor and anthropic scientist doctor Sam Bowman and
Speaker 1: why this problem is only going to get worse in
Speaker 1: the future.
Speaker 4: One of the kinds of challenges that I think we're
Speaker 4: worried about and haven't had to grapple with too much
Speaker 4: yet is just all the difficulty that comes with trying
Speaker 4: to teach values and good behavior in some setting when
Speaker 4: the model is just much much better than you in
Speaker 4: that setting. Right now, we have a lot of cases
Speaker 4: where models are kind of better than humans at some skills,
Speaker 4: worse than humans at some skills, but it's still pretty
Speaker 4: rare that you'll encounter setting where an AI is just
Speaker 4: better than sort of all of the human experts in
Speaker 4: some domain. And when that happens, things just get more complicated.
Speaker 4: And more confusing, where even if you're humans kind of
Speaker 4: looking really carefully at what the I is doing, it's
Speaker 4: often hard to figure out, Wait, what is the I
Speaker 4: trying to do here, or what effects is this going
Speaker 4: to have in the real world.
Speaker 1: With the modelogus he does, this makes everything you're trying
Speaker 1: to do, thankes.
Speaker 4: Everything we're trying to do a fair bit in earlier. Yeah, Yeah,
Speaker 4: we're less confident that we can keep track of what's working,
Speaker 4: and I think there's just kind of more possibilities for
Speaker 4: whole new kinds of unwanted behavior to creep in that
Speaker 4: will have to find a way to grapple with.
Speaker 1: I see, like, right now, maybe ais are at the
Speaker 1: level we are. Whatever issues it's having, there's things we
Speaker 1: can grasp. But as they get more advanced and they
Speaker 1: tackle bigger problems like solve the world's economy or figure
Speaker 1: out the right policy for the whole country or something
Speaker 1: like that that not one person can really grasp, it's
Speaker 1: going to be hard to even sort of like talk
Speaker 1: to it and understand it. I think that's what you're saying,
Speaker 1: right Yeah, Yeah, I.
Speaker 4: Think there's even maybe two interesting ideas in there, because
Speaker 4: maybe the pseudo staff. That's something like, we're asking the
Speaker 4: AI to help us design novel molecules for pharmaceutical development,
Speaker 4: and it's got some really novel ideas about biology that
Speaker 4: are just really complex and really hard for humans to understand,
Speaker 4: and we can't tell kind of is the model actually
Speaker 4: convinced that this is going to work, or is the
Speaker 4: model messing with us and this would actually be kind
Speaker 4: of dangerous. Should we try this drug, should we start
Speaker 4: to do some expermise in the lab. There's this setting
Speaker 4: where we kind of still ultimately know what we want.
Speaker 4: We know we want drugs that are safe.
Speaker 1: Uh huh. Like it might tell you that this will
Speaker 1: cure cancer, for example, but you're saying, like, what else
Speaker 1: it's trading off to cure that cancer for example? You
Speaker 1: might not know.
Speaker 4: Yeah, yeah, that's a good example. It's hard to tell
Speaker 4: if kind of the models like genuinely trying its best
Speaker 4: and genuinely thinks this is the best cancer drug, or
Speaker 4: if it thinks, oh, this is just something that looks
Speaker 4: good and it doesn't actually care if the drug will
Speaker 4: ultimately succeed, or if maybe for some reason, the model's
Speaker 4: extremely scary and it's actually trying to mess with you,
Speaker 4: and you've got your scary miss lined AI that's trying
Speaker 4: to sneak in some slow acting poison. The smarter the
Speaker 4: IA is, the harder it is to tell the difference
Speaker 4: between those different outcomes. And then once you start talking
Speaker 4: about a lot of these really kind of ambitious sort
Speaker 4: of capital f future social scenarios like AIS trying to
Speaker 4: figure out sort of what the economat would be like
Speaker 4: or how the world should be governed or something like this,
Speaker 4: and like, I don't know how much we want to
Speaker 4: use AIS for things like this, But once you get
Speaker 4: into that territory in any way, then you just get
Speaker 4: into this extremely weird situation where I don't know if
Speaker 4: anyone is going to know what we even want, Like,
Speaker 4: what is the right way to govern the world, What
Speaker 4: is the right way to do?
Speaker 3: I don't know.
Speaker 4: Yeah, yeah, At some point, figuring out how an AD
Speaker 4: should behave requires you to solve philosophy requires you to
Speaker 4: figure out what is good. And the more powerful AI
Speaker 4: gets and the weirdest situations you're putting it in, the
Speaker 4: more kind of common sense notions of what's good start
Speaker 4: to fall apart. The more you actually have to grapple
Speaker 4: with a lot of the really hard, confusing stuff.
Speaker 1: It might tell us how to run the world, but
Speaker 1: we at that point know one person or even a
Speaker 1: group of people might know, is this actually the best
Speaker 1: way to run the world. Is it sort of taking
Speaker 1: into account the things that all of us collectively would value.
Speaker 1: That's kind of the problem.
Speaker 4: Yeah, And I think you start to get at some
Speaker 4: of these pretty difficult questions even before you get into
Speaker 4: these kind of really big features of sort of how
Speaker 4: to go on around the world. If someone is getting
Speaker 4: all of their news or getting all of their personal
Speaker 4: life advice from an AI, that's already giving the AI
Speaker 4: a lot of leeway for kind of what makes a
Speaker 4: good life for this person, what is important for this
Speaker 4: person to know? And those are already questions that get
Speaker 4: really hard. And what you want in the short term
Speaker 4: might not match what they want long term. What makes
Speaker 4: you happy? You might i match their intuitions. What's good
Speaker 4: for that person might not be what's good for their
Speaker 4: community might not be the same as what's good for
Speaker 4: the world.
Speaker 1: You made me think of it. I wonder iful good
Speaker 1: analogy is that it'd be almost like if you as
Speaker 1: a parent, I don't know if you have kids or
Speaker 1: nieces or nephews. But it'd be almost like if your
Speaker 1: kids suddenly try to tell you what to do or
Speaker 1: was trying to teach you how him or her wanted
Speaker 1: to run their lives. You'd be like, you're just a kid,
Speaker 1: what are you talking about? This is trust me, this
Speaker 1: is what you need to do. Yeah, except that we
Speaker 1: are the kids and the AI is sort of the parent.
Speaker 1: Is that sort of what we're the situation that might
Speaker 1: be sort of parallel to that.
Speaker 4: Yeah, I think this thing there, I feel like a
Speaker 4: version of the analogy that I'd be more excited abou
Speaker 4: would almost be some alien species lands huh, and they
Speaker 4: have all this great technology and they seem nice and
Speaker 4: they're like, hey, we'd really recommend making some changes to
Speaker 4: your side. He maybe try doing things like this, And
Speaker 4: we're like, wait, you're really very accomplished. You have some
Speaker 4: some useful ideas, but like, are you trying to help us?
Speaker 4: Are you trying to sabotage us? Are you just kind
Speaker 4: of produced?
Speaker 1: Are we what's for dinner? Or are you inviting us
Speaker 1: to dinner?
Speaker 3: Yeah?
Speaker 4: Yeah, yeah yeah.
Speaker 1: The sense I'm getting for you is that these things
Speaker 1: are just getting smarter and more capable, so it seems
Speaker 1: to really pressing. We figured this out now before it
Speaker 1: gets even more difficult. Yes, yes, AIS are getting smarter
Speaker 1: each second, it seems, and we seem to be trusting
Speaker 1: them more and more each day with our data, our choices,
Speaker 1: and even our lives, which brings us to the main
Speaker 1: question end of the day, what can we do about it?
Speaker 1: How do you train an AI to care about us,
Speaker 1: to have our values and to make the right choices.
Speaker 1: To answer this question, I reached out to another AI
Speaker 1: expert on alignment, doctor Tim Rutner. Doctor Rutner is a
Speaker 1: professor at the Vector Institute for Artificial Intelligence at the
Speaker 1: University of Toronto, and he says there are many ways
Speaker 1: to train AIS to like us. The only problem is
Speaker 1: none of them work perfectly. So here's my conversation with
Speaker 1: doctor Tim Ruttner. Well, thank you, doctor Runner for joining us.
Speaker 3: Thanks so much for having me on it.
Speaker 1: And I'm talking to a real person right now, right,
Speaker 1: You're not an AI version of.
Speaker 3: Yourself as far as I'm aware.
Speaker 1: As far as any of us are aware. Yes, I
Speaker 1: mean this whole conversation could be AI generated.
Speaker 3: I know we're just all in the simulation.
Speaker 1: Well, it certainly be a lot easier. I would get
Speaker 1: more sleep for sure. Well, today we're trying to answer
Speaker 1: a very critical question which was posted by our sound engineer,
Speaker 1: which is is AI going to kills all? Can an
Speaker 1: AI have values? Can it AI have an understanding of
Speaker 1: a human? What good things are to a human?
Speaker 3: Yeah? And well I wish I had the answer to that.
Speaker 1: Maybe that's the problem is that we don't know.
Speaker 3: Yeah, I mean this is such a difficult question, right,
Speaker 3: and I think that this is a question that touches
Speaker 3: on philosophy, engineering, psychology, and probably many other disciplines. Right,
Speaker 3: but what are values and what values can possibly in
Speaker 3: a non sentient being have?
Speaker 1: It's not a simple questionnaire.
Speaker 3: Yeah, let me take a step back. So the way
Speaker 3: to think about alignment is I think through the lens
Speaker 3: of what's referred to as the specification problem, Where specification
Speaker 3: is the term that we use to describe what we
Speaker 3: tell the model it should do.
Speaker 1: When you say specification, you mean like spec right kind.
Speaker 3: Of Yeah, yes, inspect it's just short for specification.
Speaker 4: Yeah.
Speaker 3: This is what we can think of as our intent,
Speaker 3: the kinds of things that we want a model to do.
Speaker 3: For example, our intent might be for models to never
Speaker 3: say things that could lead to harm or intent could
Speaker 3: be that models should always be friendly and helpful. And
Speaker 3: so this is what we call the ideal specification for that.
Speaker 1: Model, meaning like we want to be able to say, like,
Speaker 1: be a chat butt, but make sure that nobody ever
Speaker 1: hurts themselves.
Speaker 3: That's right, I see. And there are a few different
Speaker 3: ways to provide specifications to chatbots.
Speaker 1: Okay. According to doctor Runner, there are three general ways
Speaker 1: to make sure ais behave or not kill us. The
Speaker 1: first way is to basically tell it to behave every
Speaker 1: time you ask it to do something.
Speaker 3: There is what's called a system prompt. This is a
Speaker 3: text specification that a model loads every time afford engages
Speaker 3: in a conversation with a user. In the case of
Speaker 3: the chatbot, so.
Speaker 1: Every time you interact with the AI, you would basically
Speaker 1: instruct it to behave. You might say, hey, AI, organize
Speaker 1: all my emails, or design a new drug for me,
Speaker 1: or figure out the best policy for our government, but
Speaker 1: please make sure that no one gets harmed, that you
Speaker 1: don't do anything dangerous or an ethical, etc.
Speaker 3: Etc.
Speaker 1: But of course this would get pretty cumbersome if you
Speaker 1: have to do it every time that's option number one.
Speaker 1: Option number two is to have humans train your AI
Speaker 1: to be good.
Speaker 3: So one approach is called reinforcement learning from human feedback,
Speaker 3: so different answers for a given prompt and then having
Speaker 3: human labelers say which of these answers they prefer.
Speaker 1: What does that look like? The human notator is like
Speaker 1: a warehouse full of people just talking to the same AI,
Speaker 1: or is it three people or is it a thousand people?
Speaker 1: What does that look like?
Speaker 3: So I should say I'm not an expert on this,
Speaker 3: but my understanding is that much of this work is
Speaker 3: outsourced to countries where the medium wage is lower than
Speaker 3: for example, in the United States or in Europe. There
Speaker 3: have been reports of large groups of annotators, specifically annotating
Speaker 3: images and texts that are considered not safe for work,
Speaker 3: for example, in those countries. So in other words, you
Speaker 3: have examples of folks in those countries that are already
Speaker 3: less privileged than people living in the United States, for example,
Speaker 3: on average, engaging with a lot of horrific content and
Speaker 3: saying this is not something we want.
Speaker 1: So this is basically paying people to test drive your AI.
Speaker 1: You could have a warehouse full of people whose job
Speaker 1: it is to interact with the newborn AI and essentially
Speaker 1: have them raise the AI and tell it what is
Speaker 1: right and what is wrong. Unfortunately, as doctor Runner said,
Speaker 1: I mean the poor people would have to bear the
Speaker 1: absolute worst behavior of the AI. That's option number two.
Speaker 1: Option number three for making AIS that care about us
Speaker 1: is to basically bake into the AI a constitution, you know,
Speaker 1: like the US or UK constitution that establishes what the
Speaker 1: country stands for, what its values are, and what's generally
Speaker 1: allowed and not allowed.
Speaker 3: There's an approach that Thropic introduced called constitutional AI, and
Speaker 3: that approach is based on providing a constitution to an
Speaker 3: AI MODL, and that constitution reflects different values and preferences
Speaker 3: that the company in this case, Anthropic wants the model
Speaker 3: to exhibit.
Speaker 1: Yes, the last approach here to making sure an AI
Speaker 1: has values and morals is to give it a founding document.
Speaker 1: But here's the wild part. The way to big that
Speaker 1: founding document into the AI brain is to have another
Speaker 1: AI train it. When we come back, we'll dig into
Speaker 1: that scenario and we'll ask our experts what they think
Speaker 1: the future holds. Will future AIS have our best interests
Speaker 1: in mind? Or is it hopeless to ever be certain
Speaker 1: they won't harm us, So stay with us. We'll be
Speaker 1: right back. Hey, we'll come back. We're talking about AI alignment,
Speaker 1: or basically the problem of making sure AIS don't kill
Speaker 1: us all. And so far we've talked about why this
Speaker 1: is such a hard problem and what are some of
Speaker 1: the ways we can teach AI things like values and morals.
Speaker 1: There are several ways, and one of them is to
Speaker 1: give AIS a constitution or the equivalent of a founding
Speaker 1: document or moral guide, and then have that bag into
Speaker 1: the DNA of the AI. Now what's interesting is that,
Speaker 1: according to doctor Tim Rutner, the way to do that
Speaker 1: is through another AI.
Speaker 3: This is a little bit in the weeds, but providing
Speaker 3: a very long constitution that outlines every preference in detail
Speaker 3: when a user engages with the model is actually more
Speaker 3: expensive for the company to do because the model needs
Speaker 3: to ingest a lot of text upfront, so it's easier
Speaker 3: to try to bake the preferences that are expressed in
Speaker 3: the constitution explicitly into the model when you're training it upfront,
Speaker 3: as opposed to providing that specification every time a user
Speaker 3: engages with the model.
Speaker 1: I see, you want the model to have learned the
Speaker 1: constitution sort of inherently, rather than having to check it
Speaker 1: every time somebody asks it a question.
Speaker 3: Yes, I think that's roughly right. Ideally we would be
Speaker 3: able to use humans to provide feedback and to oversee
Speaker 3: models and to say, hey, this is behavior that we
Speaker 3: don't want and stop that behavior. But of course that's
Speaker 3: not really scalable. We can't have a human oversee every
Speaker 3: interaction that a chatbot has, And so that raises the question,
Speaker 3: how can we exhibit oversight in a way that is safe, reliable,
Speaker 3: aligned with our values and preferences, and successful, And so
Speaker 3: key challenge here is essentially to come up with tools, methods,
Speaker 3: models that are able to check whether a given of
Speaker 3: AI model perform some unintended behavior, and if it does,
Speaker 3: can ring an alarm bell and let a human know
Speaker 3: that oversight is needed and that maybe a model engages
Speaker 3: an undesirable behavior.
Speaker 1: You mean like, have an AI police the other AI.
Speaker 3: That's right, essentially, have one model overse.
Speaker 1: Model WHOA But then how do you make sure the
Speaker 1: police AI is doing its job or is aligned itself?
Speaker 1: You need another police.
Speaker 3: We don't know, that's the problem. We don't know. It's
Speaker 3: a turtle, it's all the way down from it. And
Speaker 3: if we have a model that checks whether another model
Speaker 3: does what we wanted to do, and how do we
Speaker 3: know that that model that does the overseeing is actually
Speaker 3: aligned with us? I would argue that it might be
Speaker 3: easier for us to make sure that the overseer model
Speaker 3: is aligned than the generator model. They're a little simpler
Speaker 3: because they don't necessarily generate. They just try to classify
Speaker 3: whether a given behavior is intended or not intended. And
Speaker 3: so this way we might be able to do alignment
Speaker 3: more scalably and in a way that really reflects different
Speaker 3: individuals or groups preferences and values.
Speaker 1: I see. It's like, have another AI can of sit
Speaker 1: in every time I ask CHGBT something yes. Yes. As
Speaker 1: AIS get bigger and more complicated, the only scalable solution
Speaker 1: to training them is going to be through other AIS.
Speaker 1: In this situation, you might program a simpler AI with
Speaker 1: your values and morals, and then you'd have that AI
Speaker 1: train the bigger AI release try to It's like the
Speaker 1: Rutner says, AI alignment methods are not perfect. What do
Speaker 1: you mean? They're not perfect? They don't always work or
Speaker 1: they can't guarantee that they will work.
Speaker 3: So with machine learning models, we can rarely guarantee anything.
Speaker 1: Oh boy, that's kind of the problem, isn't it.
Speaker 3: Yeah, I think that's one of the problems. There is
Speaker 3: research that tries to establish guarantees, but that research is
Speaker 3: far behind the practice at the moment. The kinds of
Speaker 3: methods that we have for model alignment falls short in
Speaker 3: a few different ways. One that's I think one of
Speaker 3: the biggest ways. It's just hard to communicate our preferences.
Speaker 3: So there are many different steps at which alignment can fail.
Speaker 3: This goes back to trying to express and then community
Speaker 3: what we want a model to do kind of values
Speaker 3: and preferences we're trying to install in it. Translating our
Speaker 3: values and preferences from some really complicated, possibly contradictory ideal
Speaker 3: specification into a design specification is very difficult and there's
Speaker 3: likely going to be some gap there. And then second,
Speaker 3: even if we were able to do this perfectly, even
Speaker 3: if we were able to express and communicate our values
Speaker 3: and preferences perfectly, the kinds of low level tools machine
Speaker 3: learning tools that we use to give the model these
Speaker 3: preferences and values are imperfect at translating the values that
Speaker 3: we're trying to communicate into the model, They might not
Speaker 3: enable us to perfectly translate the design specification into the
Speaker 3: actual behavior that we would like to see.
Speaker 1: I see, boy, it seems like there are problems everywhere
Speaker 1: we turn here, doctor rut. Yeah, Well, as we go
Speaker 1: towards the future, and as systems get smarter and problems
Speaker 1: get more complicated, what do you think is the prospect
Speaker 1: of making sure that these more advanced systems were more
Speaker 1: complicated problems have values that we want it to have,
Speaker 1: Because I'm not sure if we want it to have
Speaker 1: human values, because I don't know if humans are the
Speaker 1: best making these kinds of good choices. Yeah, what do you think?
Speaker 1: What do you what do you see in the future.
Speaker 4: I think tho's a few things we need longer term,
Speaker 4: and they all feel uncertain. I think to do well
Speaker 4: in the longer term, we need to get the AIS
Speaker 4: in near future.
Speaker 3: Right.
Speaker 4: If the pace of AI development stays fast, we're really
Speaker 4: really going to need the help of AI systems to
Speaker 4: help us figure out how does your future A systems?
Speaker 4: And so getting the right values into the next model
Speaker 4: we build helps us figure out what to do with
Speaker 4: them adel after that, and so I think this kind
Speaker 4: of the short term work really does kind of fan
Speaker 4: out into this longer feature. And yeah, getting the next
Speaker 4: model right really matters for getting.
Speaker 3: The fartugerules right.
Speaker 1: Oh boy, it's called.
Speaker 4: The scalable oversight problem.
Speaker 1: I see. It's like the alignment problem is going to
Speaker 1: scale up, and the best way for us to keep
Speaker 1: up is to make sure that we get it right
Speaker 1: now with these smaller systems, so we can use those
Speaker 1: AIS to help us in the more complicated situations.
Speaker 3: Yeah.
Speaker 1: Yeah, that's oh wow.
Speaker 4: That's the hope.
Speaker 1: Okay, last question. Do you think humanity is doomed?
Speaker 4: I don't think so. I think it's possible. I think
Speaker 4: the AI presents a lot of really scary and destabilizing
Speaker 4: possibilities that we can't roll out. So I think there's
Speaker 4: a lot of work to do. I think we'll probably
Speaker 4: figure it out. But I think it's also possible that
Speaker 4: AI winds us up in a lot of weird, unfamiliar situations.
Speaker 4: I think it's unlikely than possible that things go really,
Speaker 4: really terribly, But I also think it's kind of unlikely
Speaker 4: but possible, but the things stay totally normal and recognizable
Speaker 4: and familiar. I think AI is just what it's going
Speaker 4: to do to society and politics and economics is all
Speaker 4: going to be confusing. I the's a lot that we'll
Speaker 4: need to figure out pretty fast.
Speaker 3: Fingers crossed.
Speaker 1: I guess that's as good of an answer as we
Speaker 1: can get these days. Fingers crossed. I guess just to
Speaker 1: wrap up here, what do you think is going to
Speaker 1: happen in the future, or what are some things about
Speaker 1: this that you think most people are not thinking about
Speaker 1: that they should be thinking about.
Speaker 3: I think people should be thinking about ways in which
Speaker 3: the AI systems that we have today are already capable
Speaker 3: enough to cause harm, to change our world quite significantly,
Speaker 3: change our culture, change the way we go about our day,
Speaker 3: change the way we make decisions, change the way we
Speaker 3: do our work. And the alignment problem and understanding when
Speaker 3: models are aligned, I think, are two of the most
Speaker 3: fundamental scientific challenges that we as a society are facing
Speaker 3: right now. And that is not a sci fi future problem.
Speaker 3: This is a problem about systems that we have to
Speaker 3: We want to make sure these systems really do what
Speaker 3: we want them to do, and that these systems help
Speaker 3: us flourish and benefit humanity. The systems that we have
Speaker 3: access to today already well beyond the capabilities that the
Speaker 3: research community and certainly the general public thought we could
Speaker 3: have in the year twenty twenty six.
Speaker 1: I think you're saying that the future is here, but
Speaker 1: we still haven't fully figured out the alignment problem.
Speaker 3: Yes, that's right, meaning it's.
Speaker 1: More pressing than ever that we figured this out.
Speaker 3: I agree.
Speaker 1: Yeah, amazing, doctor Rutner. How do we prove to the
Speaker 1: audience that we're not an AI generated conversation?
Speaker 3: I wish I had the answer. You can generate such
Speaker 3: fantastic fake podcasts with AI now right right, with all
Speaker 3: the little idiosyncrasies that you hear in podcasts today that
Speaker 3: you know, I think that's hard to do.
Speaker 1: Or I guess if this conversation is aligned with but
Speaker 1: you want to hear, maybe it doesn't matter.
Speaker 3: Still, I hope that the audience thinks that we so
Speaker 3: out on human I think that that would be That
Speaker 3: would be nice.
Speaker 1: All right? Hey on, behalf of everyone who works in
Speaker 1: the show picture joining us on the sixty plus episodes
Speaker 1: we've done. Be sure to follow me on social media
Speaker 1: or PhD comics dot com for updates and hey, thanks
Speaker 1: to all the guests we've had on the show. Here's
Speaker 1: a little tribute our editor Rose so good to put
Speaker 1: together of all the times they were gracious enough to
Speaker 1: put up with my questions.
Speaker 4: That's a good question.
Speaker 3: Yeah, that's a great question. You know, that's a great question.
Speaker 4: Yeah, so those are all big questions for us to
Speaker 4: answer a scientists. Yeah, that's a great question. It's a
Speaker 4: good question.
Speaker 3: That's a good question.
Speaker 4: That's a good question. So that's actually a good question.
Speaker 2: You're you're raising really great questions.
Speaker 4: Yeah, that's a really good question.
Speaker 3: That's a good question. That's a really good question.
Speaker 4: That's a very hard question. That's a good question, though,
Speaker 4: that's a really good question. That is a really good question. Yeah,
Speaker 4: that's a great question.
Speaker 3: Yeah, that's a good question.
Speaker 1: Oh that's a great question, very very good question.
Speaker 3: That's a really good question.
Speaker 4: So I'll go through that question back at you.
Speaker 1: What do you think? So we come once again to
Speaker 1: the edge of scientific knowledge. Thanks for joining us, see
Speaker 1: you next time you've been listening to Science Stuff. Production
Speaker 1: of iHeartRadio Bringing the produced by me or Hey Cham,
Speaker 1: edited by Rose Seguda, Executive producer Jerry Rowland, and audio
Speaker 1: engineer and mixer Kasey Peckram. You can follow me on
Speaker 1: social media. Just search for PhD Comics and the name
Speaker 1: of your favorite platform. Be sure to subscribe to sign
Speaker 1: stuff on the iHeartRadio app, Apple Podcasts, or wherever you
Speaker 1: get your podcasts.