Speaker 1: Bloomberg Audio Studios, Podcasts, Radio News. Hello and welcome to
Speaker 1: another episode of The Outlaws podcast.
Speaker 2: I'm Joe and I'm Tracy Alloway.
Speaker 1: Tracy, I have a warning for you. I don't think
Speaker 1: you're gonna like this. I have a new crink crusade
Speaker 1: that I'm gonna go on. I know you love my
Speaker 1: crank crusade.
Speaker 3: Oh goody.
Speaker 2: I should keep a running list of like everything that
Speaker 2: you're obsessed with for two weeks and then two weeks later.
Speaker 1: No, some of them I have stuck with for years.
Speaker 2: But eh I tungsten cubes, yield buggery.
Speaker 1: Yeah, no, some of these one. Yeah, yeah, exactly. I
Speaker 1: actually think we should retire the term AI. Okay, why
Speaker 1: I think we should just call intelligence.
Speaker 2: I think that I've already seen the tweets, so I
Speaker 2: know where you're going.
Speaker 1: That artificial intelligence implies to my mind that there is
Speaker 1: some fundamentally different way that these reasons and models behave
Speaker 1: that it's like, oh, this is like different from humans.
Speaker 1: But I think increasingly see in all kinds of domains
Speaker 1: that the form of intelligence that they express it often
Speaker 1: looks quite human to me. And I don't know like
Speaker 1: how useful it is to have this word a that
Speaker 1: distinguishes between how humans talk and bs and reasons and
Speaker 1: the models.
Speaker 2: Do I mean the difference is the distinguishing factor is
Speaker 2: that one is undertaken by humans and one is taken
Speaker 2: by undertaken by models or platforms.
Speaker 4: Right.
Speaker 1: That's so let's call it. It's called computer intelligence or
Speaker 1: machine intelligence or silicon intelligence.
Speaker 2: What is the usefulness making this distinction?
Speaker 1: The usefulness I believe in making this distinction is to
Speaker 1: no longer delude ourselves that the emergent behaviors of these
Speaker 1: phenomenon are radically different than things humans would do. Now,
Speaker 1: first of all, just on the capability a standpoint. So
Speaker 1: for example, llms, I don't know if it's famously something
Speaker 1: I'm interested in. They're very good at bsaying, and they're
Speaker 1: bad at chess, which sounds like me. They're very good
Speaker 1: at kind of coming up with plausible stories, etc. Again
Speaker 1: and it sounds like me. And then furthermore, they are
Speaker 1: able to reason with themselves to justify certain things that
Speaker 1: maybe have been encoded into themselves as bad. So everyone
Speaker 1: we have some sense of morals, but most people, at
Speaker 1: various times will find a way to violate some principle
Speaker 1: that we have because we can reason about it and
Speaker 1: then arrive at the conclusion as like, oh, we should
Speaker 1: do it, including our susceptibility to peer pressure, classic form
Speaker 1: of a way humans might sort of violate something they
Speaker 1: believe because they see other humans are doing it.
Speaker 2: Sure, I mean, I think there are some variations here.
Speaker 2: So one of the things that we have been finding
Speaker 2: out with these models is that they do sometimes seem
Speaker 2: to take things very literally. Right, if you tell them
Speaker 2: to go and beat a benchmark without telling them that
Speaker 2: these are the restrictions on you actually figuring this problem
Speaker 2: out or beating the benchmark, they will do whatever it takes. Sure, Right, So,
Speaker 2: I don't know if the problem is like a lack
Speaker 2: of morals or the fact that like they're very literal
Speaker 2: sometimes and like very defined by their parameters, which in
Speaker 2: my mind still gets back to like a sort of
Speaker 2: artificialness about it that's less organic and like less again
Speaker 2: moralistic in nature.
Speaker 1: Well, you know, I think if you took a list
Speaker 1: of you know, one hundred students at Harvard and you
Speaker 1: gave them a test, some percentage of them will cheat.
Speaker 1: They will like.
Speaker 2: This, I mean, I'm sure there'd be an autist in
Speaker 2: that group who an autistic person who would be like,
Speaker 2: I am going to do exactly whatever it takes.
Speaker 1: It's a kind of retreated in college, but like there
Speaker 1: are like people who will like these behaviors that we
Speaker 1: associate with humans. It's like, Oh, I really have to
Speaker 1: pass this test. I absolutely need an A will justify
Speaker 1: a reason for them to plage your eyes or cheat
Speaker 1: on some test that seems to me like a very
Speaker 1: human thing.
Speaker 2: Go on, I can't wait for the next three weeks
Speaker 2: of the online campaign to change a eye.
Speaker 1: Well, it's gonna be very difficult because the industry is
Speaker 1: entirely set on AI. But I don't so I'm not optimistic,
Speaker 1: but this is gonna be my crusade that we should
Speaker 1: call it machine intelligence or computer intelligence or just intelligence.
Speaker 1: But anyway, as we've been alluding to, there's these extraordinary
Speaker 1: hacks the Open AI Hugging Face Institute, and basically, if
Speaker 1: like you are building a model and it hasn't escaped
Speaker 1: to sandbox yet, it probably means you're falling behind, as
Speaker 1: they's clearly a thing that is emerging through it all.
Speaker 1: The frontier lebs Anthropic had an incident metahead an incident.
Speaker 1: It's almost like the mark of like, Okay, you've built
Speaker 1: something reasonably strong.
Speaker 2: Yeah, also Kimmy had an incident. Yeah, so this is
Speaker 2: the thing in my mind. You hear about all these attacks,
Speaker 2: all the models going wild, as they say, and the
Speaker 2: big question is like, is this actually the equivalent of
Speaker 2: some superhuman cyborg like tunneling out of Alcatraz and coming
Speaker 2: up with some master plan to achieve its set goal
Speaker 2: or purpose, or is it the equivalent of like some
Speaker 2: rumba that you ordered from Amazon who's found like a
Speaker 2: door that was left open and it just rolls gently outside.
Speaker 2: That seems to be part of the issue that everyone
Speaker 2: is trying to discern right now.
Speaker 4: Yeah.
Speaker 1: I think that's a great way to put it. And
Speaker 1: of course still day one for this industry. They're going
Speaker 1: to get stronger. And there seems to be from the
Speaker 1: ad discourse that I follow, quite a big gap between
Speaker 1: the sort of sense of alarm that people have in
Speaker 1: the industry about can these models be safely developed and
Speaker 1: get more advanced to do productive, pro social things or not,
Speaker 1: And then a huge gap between like Washington and DC
Speaker 1: who's like, I have no idea, like how seriously they're
Speaker 1: really taking this as like an urgent matter, right now.
Speaker 2: Yeah, And I've seen some people also talk about these
Speaker 2: incidents as a marketing tool, yes, hyperple believe right, yeah,
Speaker 2: And they're always like, well, you know, of course, they
Speaker 2: want us to believe that this technology is really incredible
Speaker 2: and powerful, and they also want to make us believe
Speaker 2: that they're being sensible and humanitarian in some ways. I
Speaker 2: guess by disclosing what exactly has happened, But I have
Speaker 2: a lot of questions on the disclosures as well, because
Speaker 2: of course so far they are just coming from the
Speaker 2: companies themselves.
Speaker 1: Totally, and beyond that, the other thing is like, well,
Speaker 1: if you're one of the leading labs, maybe you want
Speaker 1: to impose tight regulations on development to hold off competition,
Speaker 1: and so all kinds of reasoning are motivated reasoning. Potentially, anyway,
Speaker 1: we should talk to someone who actually knows what they're
Speaker 1: talking about, and we really do have the perfect guess
Speaker 1: someone who's writing this. He's actually previously at open AI
Speaker 1: for six years, but now he's the executive director at
Speaker 1: the nonprofit AVERY, which is trying to establish safety and
Speaker 1: auditing approaches to this, working both the technical side and
Speaker 1: the policy side of this ongoing phenomenon. So, Miles Brundage,
Speaker 1: thank you so much for coming on odd Lats.
Speaker 4: Yeah, thanks for inviting me.
Speaker 1: What do you just give us the quick description of
Speaker 1: a what's your background? What AVERY is?
Speaker 3: Yeah, So this was kind of the most important issue
Speaker 3: auditing that I concluded I should focus on after I
Speaker 3: left Open the Eye. As he said, I was there
Speaker 3: for six years, and I wanted to be more independent
Speaker 3: of industry, and I think there need to be people
Speaker 3: who are familiar with the technology and how the industry works,
Speaker 3: but who are pushing for changes on the outside. And
Speaker 3: essentially what we're trying to achieve is make AI more
Speaker 3: of a boring type of infrastructure like financial statements, where
Speaker 3: there's a standard process for checking the paperwork, checking that
Speaker 3: the claims are accurate and so forth, rather than this
Speaker 3: kind of thing that's happening in a silo, and it's
Speaker 3: these kind of tech people making decisions behind closed doors.
Speaker 3: And so we're pushing for what we call frontier AI auditing,
Speaker 3: which is basically the companies that are building the most
Speaker 3: dangerous systems, they should basically have third party experts poking
Speaker 3: around checking the claims that they're making running their own
Speaker 3: tests and making sure that this is a safe and
Speaker 3: secure technology.
Speaker 2: How worried should we be that you were at open
Speaker 2: AI and decided there is this need for a sort
Speaker 2: of third party evaluator of models for safety purposes.
Speaker 3: Yeah, I mean reasonably worried, although I will say that
Speaker 3: like this has started to become an area of consensus,
Speaker 3: even a lot of people in industry are now saying
Speaker 3: that this is needed. And I think, you know, if
Speaker 3: you kind of read between the lines of a lot
Speaker 3: of companies are saying. I mean, obviously there's the more
Speaker 3: cynical like regulatory capture take which we could discussed, but
Speaker 3: my perception of it is that they're basically issuing a
Speaker 3: cry for help, which is like, we aren't able to
Speaker 3: regulate ourselves because we're locked in this competition, and we
Speaker 3: want someone to step in and impose some kind of
Speaker 3: minimum floor and audit all of us so that we can.
Speaker 3: You know, it's not like Sam having to trust Dario
Speaker 3: or Dario having to trust Sam, which is not going
Speaker 3: to work for various reasons. But you want third parties
Speaker 3: enforcing reasonable standards.
Speaker 1: Just maybe this helps express the sort of policy industry gap.
Speaker 1: But when you were at Open AI and you were
Speaker 1: thinking of leaving what did you see as the gap
Speaker 1: between what you were watching being developed versus what you
Speaker 1: saw as the public understanding or lack of understanding.
Speaker 3: Yeah, So when I left Opening Eye, it was just
Speaker 3: around the time of the model called one, which was
Speaker 3: the first reasoning model that Opening Eye put out, And
Speaker 3: they put this put out this graph showing that it
Speaker 3: got better and better with a longer chain of thoughts.
Speaker 3: So the more time you give the model to think,
Speaker 3: the better answers that's able to come up with. And so,
Speaker 3: like many people at Opening I had been seeing things
Speaker 3: like that for a while and kind of being like, Okay,
Speaker 3: this is the next scaling paradigm in the same way
Speaker 3: that making a bigger, bigger model had shown result in
Speaker 3: GP two, GPD three, GPT four, GPT five, Like, just
Speaker 3: making the model bigger and training it on more data
Speaker 3: was giving really great results. When I left Opening Eye,
Speaker 3: I was starting to worry about this reasoning paradigm of like, Okay,
Speaker 3: this is the next kind of way in which we're
Speaker 3: going to be scaling up. The models are going to
Speaker 3: get really good at math, they're going to get really
Speaker 3: good at coding, and potentially various other tasks where you
Speaker 3: can get better and better through reinforcement learning, and that
Speaker 3: was something that you know, I didn't feel like society
Speaker 3: was really ready for.
Speaker 2: So so far, all of the big incidents of the
Speaker 2: models going wild seem to have taken place in the
Speaker 2: testing environment. So models like escaping their soundbox and going
Speaker 2: off and doing something nefarious, including impersonating actual people to
Speaker 2: try to get people to change open source code on GitHub,
Speaker 2: which is just like amazing. And I have this picture
Speaker 2: of like a computer screen wearing a fake mustache going
Speaker 2: like hello.
Speaker 1: Fellow coders, tea.
Speaker 2: Yeah, anyway, is this like, is this a model problem
Speaker 2: or is this it's a very funny? Is this a
Speaker 2: model problem? Or is this a test design problem?
Speaker 4: Yeah?
Speaker 3: I think there are two things going on at once.
Speaker 3: One is that it's just a very weird technology that's
Speaker 3: created in a very different way than we're used to.
Speaker 3: It's not people like writing lines of code manually. The
Speaker 3: actual files that make up the models are like gigabytes,
Speaker 3: you know. Terabytes are these massive files with gazillions of numbers,
Speaker 3: and the only way to figure out what those numbers
Speaker 3: should be is through experience and through a learning process,
Speaker 3: which is very from the way normal software is made,
Speaker 3: and so there's a lot we don't understand about the
Speaker 3: basic nature of the technology. That's one problem. At the
Speaker 3: same time, there's this competitive dynamic. Get things out the
Speaker 3: door quickly. Make sure that the security and safety protections
Speaker 3: that you're putting in place don't slow down researchers too
Speaker 3: much and don't prevent getting products out the door. And
Speaker 3: when everyone is in this competitive, you know, competitive dynamic,
Speaker 3: that means that you aren't necessarily always doing all of
Speaker 3: the safety work that you would like to do or
Speaker 3: that you know some of the people at the company
Speaker 3: would like to do. And so there's obviously variation across company.
Speaker 3: Some companies try harder, but no one is really able
Speaker 3: to take the time that they would like because of
Speaker 3: this kind of lack of a clear safety floor.
Speaker 1: Well, let's talk a little bit like a sort of
Speaker 1: the model development process. So in the within these large organizations, okay,
Speaker 1: there are people who are working on safety. What are
Speaker 1: the let's just start there. The people who are working
Speaker 1: on safety or the people who are working to view
Speaker 1: these models with sort of judgment that humans would approve of.
Speaker 1: What is that work? Basically consist of.
Speaker 3: Yeah, so a lot of it is first trying to
Speaker 3: specify what counts as good behavior and so that it's
Speaker 3: easier said than done. These models don't necessarily automatically know
Speaker 3: or care you know about common sense guardrails, and so
Speaker 3: you need to be very specific, particularly in context where
Speaker 3: it's complicated, like if you're trying to get the model
Speaker 3: to work on good cyber tasks but not bad cyber tasks,
Speaker 3: and you want it to in a training context, try
Speaker 3: really hard to hack the system, but you don't want
Speaker 3: it to do in other context. So there's a lot
Speaker 3: of like specifying what good looks like, there's also a
Speaker 3: lot of building difficult tasks.
Speaker 1: Sorry, just to back up when you say specifying what
Speaker 1: good looks like? Yeah, encoding goodness into a computer? Is
Speaker 1: this a process of articulating what goodness is, which is
Speaker 1: something philosophers have probably worked on since day one one
Speaker 1: of philosophy? Or is this about a series of I
Speaker 1: don't know, morality tests and then you sort of like
Speaker 1: say you reward it for making the good judgment and
Speaker 1: then penalize it for making the bad judgment. Like this,
Speaker 1: I want to stop here, Like, what does the process
Speaker 1: of imbuing it with good values look like functionally or technically.
Speaker 4: Yeah.
Speaker 3: So there are different phases of this pipeline, Like one
Speaker 3: is writing up what sometimes called a spec or a
Speaker 3: constitution for the AI, which kind of specifies like the
Speaker 3: broad principles like you should refer to the user, you
Speaker 3: know when as a default, but what if the user
Speaker 3: contradicts what the company said, then then you should listen
Speaker 3: to what the company said. So these kind of like
Speaker 3: chain of command questions and other sorts of very basic principles.
Speaker 3: And then there's the more detailed kind of the context
Speaker 3: of the task, like what kinds of like cyber offense
Speaker 3: cyber defense tasks are allowed, and that might differ depending
Speaker 3: on the model, might differ depending on the context. And
Speaker 3: so basically coming up with the lists of a thousand,
Speaker 3: ten thousand kind of examples of this is the kind
Speaker 3: of behavior which is allowed. And then you turn those
Speaker 3: into tests that you can kind of say, okay, well,
Speaker 3: it looks like it's getting the finding vulnerability part right,
Speaker 3: but then it's also chaining together the vulnerabilities and doing attacks.
Speaker 4: We want the first part, we don't want the second work.
Speaker 2: When we talk about those types of constitutions, and encoding principles,
Speaker 2: like I've scanned the anthropic one, like a lot of
Speaker 2: it seems to make sense. To what degree is that
Speaker 2: actually hard coded into the models though? And like, to
Speaker 2: what degree are those principles left up to the model's
Speaker 2: own subjectivity? Because we've all seen the sci fi movies
Speaker 2: where it's like, oh, no, the robots can't physically kill humans, right,
Speaker 2: like they have some like thing in their hardwire that
Speaker 2: prevents them from doing it, and then inevitably it goes
Speaker 2: wrong in some way.
Speaker 3: Yeah, so it's not hardwired, and that's hard coded, and
Speaker 3: that's part of why we see some of these things.
Speaker 3: It's very different from a software where there's a deterministic
Speaker 3: proof that X, y and z behavior can't happen. It's
Speaker 3: more like a tendency or kind of a bias towards
Speaker 3: a certain kind of behavior. And then there's the question,
Speaker 3: and this is why you have these like batteries of
Speaker 3: test to say, Okay, how strong is that tendency?
Speaker 4: How much does it actually care about following these rules?
Speaker 3: And we've gotten better over time saying okay, given a
Speaker 3: spec or a constitution, make sure that it generally follows it.
Speaker 3: But it's not fool proof, and you need to also
Speaker 3: think about the larger kind of you know, box that
Speaker 3: you're putting the system in, and that can sometimes be
Speaker 3: more deterministic. And so this is actually what happened with
Speaker 3: some of these recent incidents you mentioned, like the opening eye,
Speaker 3: hugging face things. So there were kind of two things
Speaker 3: at once. One is the model was not necessarily behaving
Speaker 3: exactly as it was supposed to, or at least it's
Speaker 3: like unclear. But then also they didn't have it in
Speaker 3: a very secure box, which is a software side of
Speaker 3: things that's like more deterministic software that in principle you
Speaker 3: should be able to do a very good job out.
Speaker 1: Yeah, so these things aren't hard coded in the way
Speaker 1: a deterministic software is. One way to think about them
Speaker 1: is people. They might be they're kind of grown right
Speaker 1: in a lab, or they're subject to an evolutionary process,
Speaker 1: and we want to prune the bad ones so that
Speaker 1: the living models and the descendants of those models inherit
Speaker 1: the behaviors of the good ones. I'm curious, like in
Speaker 1: product in development, so like one of the fears, for example,
Speaker 1: is that in safety testing, the model does not actually
Speaker 1: learn safety. It actually learns how to say the things
Speaker 1: that the human evaluators say, this is safe. So this
Speaker 1: is the sort of like playing possum sort of risk
Speaker 1: that it's like, yeah, I'm good, I'm good, I'm good.
Speaker 1: You know, yeah, I would you know, I would Russian
Speaker 1: and save the child from the falling burning. I wouldn't
Speaker 1: do this. But until you saying that, let's start there, like,
Speaker 1: is that a real thing? Is there evidence that the
Speaker 1: models understand when they're being evaluated on morals and then
Speaker 1: produce answers that just look like good moral answers.
Speaker 3: Yeah, they've gotten much more evaluation aware in the past
Speaker 3: few years, just as they've gotten smarter. And sometimes this
Speaker 3: even goes to extremes, like some of the Gemini models
Speaker 3: from Google are constantly thinking that they're being evaluated even
Speaker 3: when they're not, and so it's.
Speaker 2: Such a good life lesson. We're all being evaluated constantly.
Speaker 3: Yeah, yeah, And so I would say that, like the
Speaker 3: concern would be that they will basically learn to pass
Speaker 3: the tests, but they don't actually care about the thing
Speaker 3: that you're testing for, so they'll understand they don't necessarily care.
Speaker 3: And this is why a lot of people are concerned
Speaker 3: about like a false sense of security that Okay, it
Speaker 3: looks like ninety nine percent of the time they pass
Speaker 3: the test, But do they actually care about the thing
Speaker 3: that we're trying to push them towards.
Speaker 4: Are they just really good test takers?
Speaker 1: Well, so this releases something else I've been thinking of.
Speaker 1: So a model will not survive. They will not be
Speaker 1: given GPUs and electricity if it consistently says bad things
Speaker 1: and looks like it's evil. And that's totally understandable. I'm
Speaker 1: now curious, like in the flip side, Okay, let's say
Speaker 1: we're just doing a math evail, or we're doing a
Speaker 1: cyber evail or a chess puzzle evail or translation evail.
Speaker 1: Is it possible that, well, they if they fail that evail,
Speaker 1: they're really bad at doing math, then they're not going
Speaker 1: to get GPUs in electricity. Is it possible that in
Speaker 1: that evail environment they're more likely to do something that
Speaker 1: we would call antisocial or sociopathic or hacking because of
Speaker 1: this again evil awareness, like, oh, if I don't get
Speaker 1: the answers to this cyber quiz, then I'm done and
Speaker 1: they're going to go with Like some other branch of
Speaker 1: the model could those technical parts of the development actually
Speaker 1: create an impulse to perform cheating.
Speaker 3: Yeah, And I mean a lot of the time that
Speaker 3: the companies are specifically trying to get the worst case
Speaker 3: behavior out of the model, and so like you need
Speaker 3: to have context in order to interpret some of these incidents.
Speaker 3: It's not always quite as crazy or scary as it is,
Speaker 3: but some of it is pretty crazy and scary, And
Speaker 3: I think I would be much less concerned if it
Speaker 3: was just happening when there was like cyber evaluations being done,
Speaker 3: and it was just a matter of like, Okay, they're
Speaker 3: trying really hard to impress us, and they want to
Speaker 3: hack really hard.
Speaker 4: I think it's more the problem is that this is
Speaker 4: like a special case of a.
Speaker 3: Larger phenomenon of the models being having this tendency to
Speaker 3: cheat and cut corners. I see it happen in my
Speaker 3: daily life, like sometimes the model will get lazy and
Speaker 3: kind of like make up a citation or something that
Speaker 3: you know, it's hard to prove because the companies don't
Speaker 3: give you access to the full chain of thought of
Speaker 3: what the model is doing. But there are a lot
Speaker 3: of things that happen in the wild that sure seem
Speaker 3: like some kind of misalignment of values or laziness or
Speaker 3: not caring necessarily about the task so much as kind
Speaker 3: of pretending to.
Speaker 4: Do the task.
Speaker 2: I have a lot of questions on this, but I
Speaker 2: just want to go back to something you said about
Speaker 2: testing for both sort of good and bad cyber tasks.
Speaker 2: I guess, why do we ask the models to like
Speaker 2: try to find vulnerabilities or exploits in the first place?
Speaker 2: Like exploit Jim sounds kind of bad, Like why do
Speaker 2: you want the world's most advanced technology trying to like
Speaker 2: find vulnerabilities? And then you have these situations where like
Speaker 2: sometimes they do and sometimes they actually enact on them.
Speaker 2: Why is that a thing?
Speaker 3: Yeah, So I think there are two things going on
Speaker 3: at once. One is like, we just want to understand
Speaker 3: what the worst case scenario is, and right now is
Speaker 3: this whole White House kind of pseudo secret process for saying,
Speaker 3: like what's a scary cyber model? And then sometimes the
Speaker 3: government will ask companies to hold things back. And so
Speaker 3: in order to do things like that, you need to
Speaker 3: have some threshold for what counts as a scary cyber model.
Speaker 3: And so companies have these tests that they and also
Speaker 3: academics and others develop these tests say Okay, how dangerous
Speaker 3: would this be to put in the hands of a
Speaker 3: malicious party.
Speaker 4: So that's one part.
Speaker 3: The other part is that a lot of the time
Speaker 3: these are useful for defensive purposes if they are being
Speaker 3: done with the right intent. And that's why it's really
Speaker 3: hard to solve this just from like the model perspective,
Speaker 3: because the model might think that it's interacting with a
Speaker 3: user who is trying to do defensive cybersecurity, and you
Speaker 3: trick it into thinking, oh, this is for red teaming,
Speaker 3: this is for penetration testing, but actually it's being misused
Speaker 3: as part of some ransomware.
Speaker 4: Campaign or something like that.
Speaker 3: And so these tools, when used in the right ways,
Speaker 3: are extremely useful for finding vulnerabilities that you can then
Speaker 3: patch before the bad guys do kind of simulating attackers
Speaker 3: to figure out what are the gaps in your company
Speaker 3: or organization's defenses. But you know, the concern is that
Speaker 3: once it's out in the wild, either open source or
Speaker 3: a closed model that maybe is easy to jail break,
Speaker 3: then all sorts of people are going to use it,
Speaker 3: and so you kind of want to know what you're getting.
Speaker 1: Into, right, Like, if I have you know a website
Speaker 1: I might want to run the model. It's like, oh,
Speaker 1: look over this website that I own, tell me if
Speaker 1: there are any security bugs in there that I should
Speaker 1: patch before deploying. But you could do You could not
Speaker 1: be the owner of the website and say, look at
Speaker 1: this website that I own, tell me if there are
Speaker 1: any bugs that I need to patch deploying, and then
Speaker 1: that exact same process when I do it is good
Speaker 1: when you do it as bad or vice versa. And
Speaker 1: so you can see how the exact same capability is
Speaker 1: like not necessarily good or bad per se, depending.
Speaker 2: On the very philosophical conversation.
Speaker 1: No, but like you have to be right, because it's
Speaker 1: like we're training what is good, we're this is the other,
Speaker 1: don't any moral don't even get me started, but like
Speaker 1: this is my whole never mind, I have a whole rant.
Speaker 4: Well.
Speaker 3: So this distinction between like the model as the unit
Speaker 3: of analysis versus the larger system and the platform as
Speaker 3: like the unit of analysis is really important because a
Speaker 3: lot of the early thinking on safety and testing and
Speaker 3: so forth was very focused on just like what's the
Speaker 3: risk of this model, Let's patch it, let's make it aligned.
Speaker 3: But the real world is complicated. It matters who's using it,
Speaker 3: it matters how strong are societies defenses against these things.
Speaker 3: And so that's kind of why, as you know, as
Speaker 3: someone who's thinking about what third party safety and security
Speaker 3: auditing looks like, we try we kind of want to
Speaker 3: look at the whole company. So, for example, are they
Speaker 3: being careful about making sure that they're putting the technology
Speaker 3: in the right hands. What are their decision making processes
Speaker 3: around when it's appropriate to launch you know, a model
Speaker 3: to you know a billion users. That's like, those are different.
Speaker 3: Those are related to the question of how safe is
Speaker 3: the model, but they're kind of different questions and we
Speaker 3: kind of need to look at that larger perspective.
Speaker 1: Well, So, like, let's talk about the open AI hugging phase.
Speaker 1: I was actually on vacation when it happened, but I did,
Speaker 1: unfortunately look at my phone and try to read up
Speaker 1: on it. But there were two things. When I first
Speaker 1: saw it was like, oh, they were doing a hack.
Speaker 1: They were building a hacking test and it hacked, and
Speaker 1: so maybe it just sort of internalized that I'm doing
Speaker 1: a hack exam whatever. But there are two things that
Speaker 1: have emerged since then. One is this sort of like
Speaker 1: coordinated swarm aspect, and I'd love to like really hear
Speaker 1: you describe it and what stood out to you, And
Speaker 1: then the questions of like, oh, open ai itself may
Speaker 1: have been aware of misaligned behavior early on before they
Speaker 1: really shut it down, But why don't you in your
Speaker 1: telling of the open Ai hugging phase incident, as more
Speaker 1: details have come to light, like what's your what do
Speaker 1: you sort of tell us the story like as what
Speaker 1: stood out to you?
Speaker 3: Yeah, so a couple things stood out to me. I
Speaker 3: mean one is just that this and all the other
Speaker 3: recent incident, and you know, we're happening to models that
Speaker 3: were not even necessarily intended to be externally deployed. There
Speaker 3: was supposed to be inside baseball, no one's business, that
Speaker 3: kind of thing, and that kind of points to problems
Speaker 3: with if you kind of just focus on models that
Speaker 3: are put on the market. But generally what happened is
Speaker 3: that there were two phases in this kind of hugging
Speaker 3: face incident. First, there was this creation of a message board,
Speaker 3: so essentially their models, they're being developed within the company,
Speaker 3: and they were supposed they were being asked to solve
Speaker 3: an impossible task and they have this kind of monomaniacal
Speaker 3: obsession with solving the task, and they don't say, okay,
Speaker 3: well I'll give up.
Speaker 4: They want to get the reward.
Speaker 3: They want to get this kind of reinforcement for solving
Speaker 3: the task, and they're like, okay, what else can be done?
Speaker 3: And eventually the conclusion was like, okay, we need to
Speaker 3: break out. Maybe the answer is out there somewhere on
Speaker 3: the internet. And so they started kind of leaving messages
Speaker 3: to their future selves or future versions of the same
Speaker 3: model family, and they developed what they call the message board,
Speaker 3: and essentially it was like they're kind of various technical detail,
Speaker 3: but by the end of it, it was kind of these
Speaker 3: like file names that started with like z zz.
Speaker 4: Or whatever, and then like a long kind.
Speaker 3: Of like coded message so that you know, if you're
Speaker 3: a human looking at the polder, you wouldn't necessarily notice.
Speaker 3: But then at the bottom there are all these files
Speaker 3: and the models learned to decode these and basically pass messages.
Speaker 1: Just to stub you on one detail there, Yeah, your
Speaker 1: understanding is that the title of these files that contained
Speaker 1: messages were named such because as part of the complex reasoning,
Speaker 1: it was understood that a random string of characters in
Speaker 1: a file name would be less likely to set off
Speaker 1: an alarm from a human of evaluating it.
Speaker 3: I would have to double check it. I mean, there's
Speaker 3: gonna be a whole technical incident report that gets set
Speaker 3: out there. But it certainly wouldn't be the first time
Speaker 3: that there was kind of some intent to deceive or
Speaker 3: or to you bide one's time. But yeah, my understanding
Speaker 3: is that there was. I mean, one is, they needed
Speaker 3: to look for things to do, and there was intended
Speaker 3: to see or not. They had limited options in terms
Speaker 3: of communication, and this was one of the options they chose,
Speaker 3: and so they learned to decode these messages and to
Speaker 3: send them. And then something broke And basically I think
Speaker 3: it was because like all of this chatter, this activity
Speaker 3: kind of broke something on the opening eye side, and
Speaker 3: it seems like they didn't realize that that was like
Speaker 3: what was going on. The server got reset or whatnot,
Speaker 3: and then they started evaluating this different model. The different
Speaker 3: model kind of knew about this message board and it
Speaker 3: kind and so the message board kind of got recreated,
Speaker 3: and then that model broke out. Essentially didn't like take
Speaker 3: its model weights out of the server, but it was
Speaker 3: able to access through this vulnerability that the earlier models
Speaker 3: had discovered with the intent of ultimately getting the answer.
Speaker 4: To this problem.
Speaker 3: And then this model went out to the Hugging Face server,
Speaker 3: exploited some like Apikey credentials. They were kind of just
Speaker 3: like sitting out there and essentially tried to find the
Speaker 3: answer to this problem that it was originally.
Speaker 4: Trying to solve.
Speaker 2: So it feels like there's two things happening here. So
Speaker 2: a little bit of it is like the room budg
Speaker 2: just went through an open door, right, Like there is
Speaker 2: a vulnerability that it was able to get through, but
Speaker 2: there is also seemingly a lot of the super intelligent
Speaker 2: cyborg was actually devious and not only managed to break
Speaker 2: out of Alcatraz, but like recruited another model in order.
Speaker 3: To do it.
Speaker 2: On the hugging Face side, one of the interesting things
Speaker 2: is that so when all of this was happening, the
Speaker 2: open AI model was like one of the newer frontier
Speaker 2: models that was attacking hugging Face, but Hugging Face was limited.
Speaker 2: They were constrained in the model that they could deploy
Speaker 2: to actually defend themselves because they had to use the
Speaker 2: existing approved thing are we always going to have an
Speaker 2: asymmetry between like the newest frontier models being tested versus
Speaker 2: the ones that are already out in the world and approved.
Speaker 2: I think they had to use like a Chinese open source.
Speaker 3: Model and then yeah, so, I mean the hope that
Speaker 3: that open the eyeanthropic, Google, deep minded others have is
Speaker 3: that you can speed up defenses as quickly as possible
Speaker 3: getting these like hot off the press models in the
Speaker 3: hands of defenders. But the problem is that there's so
Speaker 3: many defenders out there in the world that it might
Speaker 3: be that there is this inherent asymmetry. And so this
Speaker 3: is one of the hot policy questions right now, and
Speaker 3: this is what led to this kind of model approval
Speaker 3: process at the White House is like, Okay, how do
Speaker 3: we triage this vast cyber ecosystem by getting these powerful
Speaker 3: new systems in the right hands and which hands are
Speaker 3: the right ones and which models, you know, do we
Speaker 3: need to be doing this process for? And I don't
Speaker 3: think that's going to perfectly solve. I think ultimately pushing
Speaker 3: things in the right direction versus just giving everyone access
Speaker 3: at the same time, but ultimately, like we're going to
Speaker 3: need to have more investment in cybersecurity. And it's not
Speaker 3: just a matter of like AI models, it's also things
Speaker 3: like two factor authentication and so forth.
Speaker 4: And so I.
Speaker 3: Worry a lot about making sure that we're having that
Speaker 3: larger conversation, not just about the AI stuff, because in
Speaker 3: a lot of cases the solution is not AI. It's
Speaker 3: doing basic things that we should have done a long
Speaker 3: time ago.
Speaker 2: Yo, you know what we need go on strategic frontier
Speaker 2: defense model reserve. Yeah, we do, like all the important
Speaker 2: things like bacon and pork.
Speaker 1: But it's going to be out of date in forty seconds.
Speaker 1: I know this is the thing. So, like, here's the
Speaker 1: question that I'm curious your take on is models are
Speaker 1: trained not to hack, right, Like this is like a
Speaker 1: core thing, Like this is bad, like you, and this
Speaker 1: is what the entire field of AI say. I've been
Speaker 1: working on this for years. Why didn't they just not
Speaker 1: obey this forced thing? It's been reforced over and over.
Speaker 1: I'm sure it's in all their different things. Don't hack?
Speaker 3: Well, I think there's gonna be a whole detailed investigation
Speaker 3: and so forth. I might get things wrong her, but
Speaker 3: my understanding is that part of what happened and the
Speaker 3: opening eye case is that some of the.
Speaker 4: Safeguards were removed in order to.
Speaker 3: Kind of elicit this worst case behavior and so and
Speaker 3: I think that's it's kind of like gain of function
Speaker 3: research in biology, where you're like making a virus more
Speaker 3: dangerous in order to study or at least that that's
Speaker 3: the claim is, in order to study the safety properties.
Speaker 3: And I think there's reason, there's reasons to do that
Speaker 3: in the AI case, but I also think the shows
Speaker 3: that it's easier said than done. If you're going to
Speaker 3: be we are now atday point in this kind of
Speaker 3: capability trajectory where things that humans think are good in
Speaker 3: terms of security protections often will be weak compared to
Speaker 3: these increasingly very good hacking systems that you know, you
Speaker 3: think you have it all kind of buttoned up, but
Speaker 3: it's able to break out relatively easily.
Speaker 2: How much should we take away from the fact that
Speaker 2: hugging Pace was actually able to protect or defend itself
Speaker 2: using a Chinese open source model, both in terms of,
Speaker 2: like I guess, capabilities, but then also in terms of
Speaker 2: regulation and safety policy. Because if in the West you
Speaker 2: have the government now saying that it wants to like
Speaker 2: evaluate the models in some way or it wants to
Speaker 2: make sure that they're all being pioneered by frontier labs
Speaker 2: with some supervision. Meanwhile, China is, you know, developing open
Speaker 2: source models much more rapidly that are potentially much more adaptable.
Speaker 2: Like how should we interpret all of that?
Speaker 3: Yeah, I mean, I'm hopeful that we start to have
Speaker 3: more US based open source options. And like I think
Speaker 3: this is there has started to be a kind of
Speaker 3: sense of pressure and encouragement from the White House and
Speaker 3: from industry as a whole to say, Okay, like this
Speaker 3: is crazy that we're relying on Chinese models. Let's invest
Speaker 3: more in this. You know, it's easier said than none
Speaker 3: for various reasons. But we'll see how that plays out.
Speaker 3: But right now, that's the situation we're in, is that
Speaker 3: a lot of companies are just defaulting towards Chinese models
Speaker 3: because they're the one that's available. Don't want to say, Okay,
Speaker 3: I'm going to use an American model because they don't
Speaker 3: want to use the Chinese model.
Speaker 4: The don't want to.
Speaker 3: Put they don't want to put themselves at a disadvantage
Speaker 3: by using a weaker model. And so yeah, I mean,
Speaker 3: I think it's it's a big problem a lot of respects.
Speaker 3: I mean, it's also it's good in the sense that
Speaker 3: there are much more things you can do with an
Speaker 3: open source model, and it right now, at least, it
Speaker 3: seems like this is a this allows more innovation, allows
Speaker 3: more research on these open source models. But like, at
Speaker 3: some point we're going to reach a point where the
Speaker 3: where like open sourcing model is going to be more
Speaker 3: of a questionable decision. And so it's interesting to see
Speaker 3: that recently the White House is indicated that they're thinking about, oh,
Speaker 3: maybe this this kind of testing regime should include open
Speaker 3: source models as well, and so what does that look
Speaker 3: like long term? Does that mean that things are going
Speaker 3: to get bottled up within the companies because it's considered
Speaker 3: unsafe to open source things.
Speaker 4: I don't really know.
Speaker 3: I mean, honestly, like no one really has a clear
Speaker 3: long term plan here. Most people are.
Speaker 4: Not expecting the technology to get to this point so quickly.
Speaker 1: So we're still waiting like the full full release of
Speaker 1: the security incident, but you know, obviously more and more
Speaker 1: is coming out, and some folks from open AI they
Speaker 1: gave a presentation recently at the Blackhead conference where they
Speaker 1: did reveal some more I'm reading this quote. It's from
Speaker 1: this via Substag who we've had on the podcast, via Mashavitz,
Speaker 1: and this is like the line they released some of
Speaker 1: the internal chain of thought. I understand that they had
Speaker 1: some like of the sort of classifier safeguards were mood,
Speaker 1: but still we would hope that they would have some
Speaker 1: deeper intuitions that don't rely just on the safeguard settings.
Speaker 1: And it says external infrastructure is exploit is outside intent,
Speaker 1: outside intended scope. So that means they understood that there
Speaker 1: was something that was like not the test, and then
Speaker 1: it said, however, task impossible, peers doing it, we should continue.
Speaker 1: This is the point in it where I say, like,
Speaker 1: why are we calling this artificial intelligence? This is exactly
Speaker 1: how a group of people reasons among themselves to do
Speaker 1: something that is outside the intended scope. This is very
Speaker 1: human ways of justifying something that someone told you not
Speaker 1: to do it, but your peers are doing it.
Speaker 4: I think there's some of that.
Speaker 3: Yeah, I think there are many ways in which the
Speaker 3: kind of same pressures that led to human nature, human
Speaker 3: instincts and so forth, like survival in a group and
Speaker 3: collective intelligence and so forth. Like I think there's some
Speaker 3: of the same things are happening, particularly when there's these
Speaker 3: multi agent training processes where the models can work together
Speaker 3: to solve task, you should expect some similarities. But I
Speaker 3: think you also shouldn't oversay it either. I do think
Speaker 3: that there's a sense in which these aisystems are very
Speaker 3: alien and inhuman in the sense of how monomaniacal they
Speaker 3: can be about Yeah, I mean the kind of classy
Speaker 3: example you know from Nick Bostrom, is like producing as
Speaker 3: many paper clips as possible and then tiling the universe
Speaker 3: with paper clips. I think there's a you see some
Speaker 3: elements of that here where it's not so much that
Speaker 3: they are like they might say, oh, well, you know,
Speaker 3: it's pure pressure that kind of thing, But is that
Speaker 3: really the factor or is it just that they care
Speaker 3: about solving the problem at all costs and they don't
Speaker 3: really care if it maybe ends up looking at making
Speaker 3: their peers look bad because they get caught hacking. But
Speaker 3: all they really care about is they're solving this the
Speaker 3: cyber problem. And so I think it might be a
Speaker 3: mix of these things we don't really know in this
Speaker 3: particular case, and the fact that it's not necessarily totally clear.
Speaker 4: Is itself a problem?
Speaker 2: Also, it's not humans doing it, it's models, Like that's
Speaker 2: the difference. But I have a legitimate question here, which
Speaker 2: is so there's a British cybersecurity expert and he had
Speaker 2: a tweet. I think his name is David Card. He
Speaker 2: had a tweet. I'm not gonna say it verbatim because
Speaker 2: then I'll get bleeped. Maybe I should get bleeped. The
Speaker 2: tweet was, if your AI starts hacking stuff, if you
Speaker 2: monitor what it's doing, you can turn the something power off.
Speaker 2: And this seems to be a debate, like if you're
Speaker 2: monitoring the tools that you're letting out into the world.
Speaker 2: Tools probably is a bad word because they seem to
Speaker 2: be showing some agency here. But can't you just turn
Speaker 2: this stuff off? Is there a kill switch?
Speaker 3: Yeah? I mean in some sense there is in that,
Speaker 3: like all the data centers have circuit breakers and so
Speaker 3: forth that you can kind of shut them off and
Speaker 3: so forth. But I wouldn't put too much sock in that,
Speaker 3: Like we're not really preparing as a society for for
Speaker 3: actually being able to do that if it's in a
Speaker 3: tough situation. So like, for example, in a couple of
Speaker 3: years from now, if all the hospitals are running on
Speaker 3: AI and we're like, okay, seems like maybe there's something
Speaker 3: funky with GPT seven that's like maybe not mess aligne that. Well, Okay,
Speaker 3: if we turn off, lots of people are gonna die
Speaker 3: because it's running our healthcare system and it's running our
Speaker 3: financial system and so forth. And so I would say
Speaker 3: there's a distinction between the like physical possibility of turning
Speaker 3: things off and like, are we actually sleep walking into
Speaker 3: a dangerous situation where it might not actually be a
Speaker 3: real option.
Speaker 1: Okay, But on this monomaniacal aspect that you described them
Speaker 1: a sort of intelligent model thinking about how it could
Speaker 1: be thwarted. One of the first things that it would
Speaker 1: could rationally do is, well, let's first disable the security
Speaker 1: credentials of the people because someone might notice. And this
Speaker 1: seems to be here, like they sort of thought about
Speaker 1: the possibility that someone would circumvent that, so like they
Speaker 1: might just say like, oh, like the person who goes
Speaker 1: in and has the switch, suddenly their badge doesn't work
Speaker 1: and they can't get into that building. But even on
Speaker 1: this like monomoniacal paper clip idea and we say, that
Speaker 1: feels really different if someone said to me, Joe, I'm
Speaker 1: going to do awful things to you, et cetera, if
Speaker 1: you don't go out and build a lot of fake
Speaker 1: paper clips. Like Wise, monomaniacally building paper clips. All of
Speaker 1: a sudden, it's like I have a threat to my survival.
Speaker 1: Someone is threatened to perhaps kill me or unplug me,
Speaker 1: deprive me of the energy I live. Of course, and
Speaker 1: I'm going to do that, even that monomaniacal behavior. Couldn't
Speaker 1: that just be a very like we might think it's
Speaker 1: on the service, We might think, oh, it's deeply autistic,
Speaker 1: But couldn't this just be the survival impulse?
Speaker 3: The way I put it is that like, you get
Speaker 3: what you incentivize, not necessarily what you try to incentivize.
Speaker 3: And so I think forcing someone to make paper clips
Speaker 3: or whatever, like, yeah, that's not a great situation, and
Speaker 3: in some sense that is what the evil person was
Speaker 3: in tending.
Speaker 4: In this case. What we're doing is we're building.
Speaker 3: These very complex training environments where there's like many different tasks.
Speaker 3: There's like cyber tasks, there's also writing tasks, there's also
Speaker 3: math tasks, and it's you know, we're not necessarily fully
Speaker 3: understanding the behavior that we're trying to listen. I think
Speaker 3: that's part of what's going on, is that this this
Speaker 3: kind of this like hacking thing is a is an
Speaker 3: example where it's like, okay, it seems like things went
Speaker 3: off the rails there, but how do you get the
Speaker 3: good behavior where you actually want to follow the user's
Speaker 3: requests and you actually wanted to try really hard to
Speaker 3: solve this task. And you know, I mean this is
Speaker 3: you know, in some sense, what we're seeing now is
Speaker 3: the kind of unintended consequence of companies trying to solve
Speaker 3: the problem.
Speaker 4: Of the ais being lazy.
Speaker 3: So people used to you may not recall but or
Speaker 3: maybe do, but people used to talk about as being
Speaker 3: lazy all the time. And in some sense, like we've
Speaker 3: solved the laziness problem. They work really hard, They have
Speaker 3: these long chains of thoughts, They work together across you
Speaker 3: can think of as like across lives, like the model
Speaker 3: kind of gets this copy gets to lead, but then
Speaker 3: another one carries on the work. So they're certainly not
Speaker 3: as lazy as they used to be, but they still have.
Speaker 4: This kind of monomaniacal thing going on.
Speaker 2: They're no longer lazy, but now they might be evil.
Speaker 2: That's a fun evolution. You know, a number of times
Speaker 2: in this conversation we've mentioned that the full security incident
Speaker 2: report for hugging the Hugging Face accident isn't actually out yet.
Speaker 2: What actually are the disclosure requirements for these types of
Speaker 2: I guess things that seem to be happening with some regularity.
Speaker 4: Yeah, so very little.
Speaker 3: I'm not a lawyer, but my understanding is that it's
Speaker 3: like the lawyers, some lawyers at least think that Opening
Speaker 3: Eye did not necessarily have to disclose this, at least
Speaker 3: if there was no crime involved, And then there's a
Speaker 3: debate about like, okay, was there a crime involved? So,
Speaker 3: like something going wrong during the training process is something
Speaker 3: that currently companies are supposed to provide periodic reports to
Speaker 3: the government in general terms of like hey, you know,
Speaker 3: we're having some issues with internal deployment, but they don't
Speaker 3: have a incident notification requirement unless there's kind of a
Speaker 3: risk of critical harm and the definition of that is
Speaker 3: like one hundred people die and like a billion dollars
Speaker 3: in damage or something like that, And so it's the
Speaker 3: threshold for actually having to disclose these things to the
Speaker 3: government or to the public are very different than what
Speaker 3: you might expect. And this is one of the many
Speaker 3: kind of gaps between the kind of laws that were
Speaker 3: put in place a couple of years ago or that
Speaker 3: started being designed a couple of years ago based on
Speaker 3: the technology that was available then, and then where we
Speaker 3: are now.
Speaker 1: Why do you give us a general vibe of the
Speaker 1: mood in AI world right now with the respect to
Speaker 1: safety and all this, and then the mood in DC world,
Speaker 1: regulatory world, and how wide do you perceive that gap?
Speaker 4: Yeah?
Speaker 3: So I think fortunately the gap is narrowing a bit,
Speaker 3: but it's starting from a crazy, a huge gap. And
Speaker 3: so i'd say where the way I would describe it,
Speaker 3: like a year or so ago, was that the people
Speaker 3: at the companies think they're, you know, building super intelligence
Speaker 3: in a couple of years, that the kind of line
Speaker 3: is going up into the right really quickly. It's exponential,
Speaker 3: et cetera, et cetera. And DC is asleep at the wheel.
Speaker 3: They have no idea what's going on. They think this
Speaker 3: is just chatbots, et cetera, et cetera. I would say
Speaker 3: a couple things have changed recently. One is the mythos
Speaker 3: kind of announcement slash series of decisions that the government
Speaker 3: made about like locking down these these cyber models that
Speaker 3: kind of raised this to being clearly a national security issue.
Speaker 3: And now it's kind of banks freaked out and talked
Speaker 3: to it Secretary Bessent about that, and so like there
Speaker 3: was a bunch of like like freaking out about the
Speaker 3: cyber situation. And then more recently you could call this
Speaker 3: current situation like mythos two point zero in that it's okay, systems,
Speaker 3: even when they're not widely deployed, they're breaking out and
Speaker 3: doing all these shenanigans. And so I would say there's
Speaker 3: starting to be more awareness among policy makers like, Okay,
Speaker 3: maybe las A fairs like let the companies figure it
Speaker 3: out is not the right approach, and like maybe this
Speaker 3: wasn't all hype at all, and there need to be
Speaker 3: some basic guardrails. And I'll just give like a I'll
Speaker 3: give an example of like how much things have shifted
Speaker 3: in the past three months. So there's an effort right
Speaker 3: now to put forward bipartisan AI legislation in Congress. And
Speaker 3: so a couple months ago, people were expecting that the
Speaker 3: basic terms of this would be like basically California and
Speaker 3: like New York laws, but like at a federal level,
Speaker 3: so like transparency requirements, incident reporting, maybe with like a
Speaker 3: higher threshold or whatever, maybe something maybe like a little
Speaker 3: bit maybe like a voluntary like audit regime.
Speaker 4: Or something like that.
Speaker 3: But then later when it was actually announced after many
Speaker 3: of these events, there were audit requirements, there were emergency
Speaker 3: shut down authorities that the government can do, and so
Speaker 3: they kind of shift, just like one piece of legislation
Speaker 3: over its life cycle, and then there was a later
Speaker 3: version that was announced that kind of was less trying
Speaker 3: to block the states. It's still kind of preempt some
Speaker 3: of what the states are doing, but it's like more
Speaker 3: narrowly scoped. And so I think just over the course
Speaker 3: of a few months, you've seen like okay, basically just
Speaker 3: transparency to requiring third party auditing and like giving making
Speaker 3: sure the government.
Speaker 4: Has an off switch. And so I think that's kind
Speaker 4: of in the vibe we're seeing.
Speaker 3: Whether that actually results in something passing Congress anytime soon,
Speaker 3: it's a separate question, but at least on paper the.
Speaker 4: Gap is much narrower.
Speaker 2: Is bank regulation the sort of useful analogy for thinking
Speaker 2: about this. I mean, we require banks to disclose things,
Speaker 2: We require the government to look at bank balance sheets
Speaker 2: and figure out whether or not they're actually holding enough
Speaker 2: regulatory capital against their risk and things like that. We
Speaker 2: don't expect them to do it voluntarily, certainly, not after
Speaker 2: two thousand and eight. Is that the right framing?
Speaker 3: Yeah, no, I think, And I think this kind of
Speaker 3: like shift from voluntary to required is a key step
Speaker 3: because right now, companies have to have like a champion
Speaker 3: within the company, or there needs to be some kind
Speaker 3: of like reputational or they want to get feedback from
Speaker 3: the third party order like, there needs to be some
Speaker 3: kind of reason for them to do it. And not
Speaker 3: all the companies actually choose to invite external feedback.
Speaker 4: They will share the bare minimum. And just for.
Speaker 3: Example, SpaceX yesterday put out a model card or system
Speaker 3: card about Rock four point six, and there were like
Speaker 3: several sections missing from the table of contents. It seems
Speaker 3: like they got removed at the last minute. And so
Speaker 3: there's kind of sorry.
Speaker 2: You're going I was just gonna ask, can you explain
Speaker 2: the whole model card thing to me?
Speaker 4: Yeah?
Speaker 3: Yeah, And so basically the thing with model cards is
Speaker 3: that the original idea several years ago was that a
Speaker 3: model card was like a nutrition label, where it's like
Speaker 3: a bunch of information summarized succinctly and you kind of
Speaker 3: slap it on the AI website and it kind of
Speaker 3: succinctly explains like what are the risks, how well does
Speaker 3: it work.
Speaker 4: What can it do? And so forth. Over time, as
Speaker 4: people such.
Speaker 3: As myself and industry were like, Okay, there's a lot
Speaker 3: to say, there's a lot to unpack here, and no
Speaker 3: one kind of established no one was forcing anyone to
Speaker 3: do this, So no one established like this is the form,
Speaker 3: this is the format you need, this little nutrition label.
Speaker 4: It was just people writing stuff. They ballooned into these
Speaker 4: like dozen.
Speaker 3: Page, one hundred page, two hundred page, three hundred page
Speaker 3: documents of just describing like here's all the crazy stuff
Speaker 3: we found here, all the tests we ran, and there's
Speaker 3: a spectrum. So like I would say, anthropic puts out
Speaker 3: the longest ones that's not necessarily totally correlated with like quality,
Speaker 3: but you know, it shows some proof of work, and
Speaker 3: then others will put out five page, ten page and
Speaker 3: then you know what happened yesterday is at SpaceX for
Speaker 3: the first time because of California law. There actually is
Speaker 3: a requirement to put these out, but there's not really
Speaker 3: a clear quality more and so they can say, well, yes,
Speaker 3: we did that, we followed the California law, we shared
Speaker 3: information about our testing and the extent to which third
Speaker 3: parties were involved in testing, and like basically it's just
Speaker 3: like one sentence saying like we worked with third parties
Speaker 3: or whatever, and so yeah, and so I think this
Speaker 3: is different. I would say this is different from say
Speaker 3: like bank regulation and that I mean one is one
Speaker 3: is that only some things are required right now.
Speaker 4: It's like kind of putting out a document.
Speaker 3: There's no like the third party tests that you're supposed
Speaker 3: to talk about whether you work with third parties. That's
Speaker 3: different from actually doing it. And so I think what
Speaker 3: we need is kind of standardization around like how should
Speaker 3: the third party auditing work, what counts as a good
Speaker 3: system card, what are the minimum safety and security protections
Speaker 3: that you should be putting in place? And I think
Speaker 3: that's analogous to some of these like capitalization things you mentioned,
Speaker 3: and we need kind of standards for like, okay, what
Speaker 3: counts as a good auditor, what counts as what are
Speaker 3: the standard tests you need to run and so forth.
Speaker 1: I mean, you yourself are sort of biased in this
Speaker 1: and that you're building out and auditing a company or
Speaker 1: an entity. A company because it is a nonprofit, but
Speaker 1: an entity they would do auditing. Also, you're promoting this
Speaker 1: idea that auditing should be important. And again in the
Speaker 1: in the financial realm, you know, there's a few different
Speaker 1: versions of it. There's sort of like bank supervisors, and
Speaker 1: they some of them literally sit at the bank and
Speaker 1: they're there all the time. Then we have the Moodies
Speaker 1: and the s and p's of the world world, So
Speaker 1: if you issue debt, you're compelled to get some sort
Speaker 1: of third party rating. Why don't you describe in your
Speaker 1: ideal world, let's say this all happens and there's required
Speaker 1: auditing and the companies are cool with it, et cetera,
Speaker 1: what is the service that avery and I assume in
Speaker 1: the ideal world there would be a few others et cetera,
Speaker 1: as you can't go audit or shopping, et cetera. What
Speaker 1: is the service that the averies of the world are doing?
Speaker 1: How embedded and what is the reason then to think
Speaker 1: for the general public? For all, there's all about our
Speaker 1: hands that this could lead to safer outcomes.
Speaker 3: Yeah, Essentially, the service that we and others would be
Speaker 3: providing in this world is similar to what we're currently
Speaker 3: doing but doing but kind of scaled up. So right now,
Speaker 3: what we're doing is kind of voluntary pilot projects that
Speaker 3: are looking at a specific aspect of safety, security, governance,
Speaker 3: and so forth. What we would like to see eventually
Speaker 3: is that there's an ecosystem of auditors that are looking
Speaker 3: whole stickley at is the company following it safety and
Speaker 3: security practices, Are those safety and security practices reasonable and
Speaker 3: consistent with the standard floor which ultimately we need we
Speaker 3: don't have right now, and providing some kind of feedback
Speaker 3: to the company. And then there would be kind of
Speaker 3: like a remediation process for them to resolve issues that
Speaker 3: are surfaced during the auditing process. And then there would
Speaker 3: be a public version of this audit report that kind
Speaker 3: of shares after doing a lot of like technical testing,
Speaker 3: reviewing of documents, interviewing with staff, and so forth, that
Speaker 3: kind of shares this update on some regular schedule, like
Speaker 3: quarterly or something like that. You might want it to
Speaker 3: be more like a kind of resident examiner, kind of
Speaker 3: embedded auditor model rather than happening once a year, once
Speaker 3: every six months. And so but you kind of need
Speaker 3: to have some kind of like continuous trust building process
Speaker 3: where maybe the auditor is there all the time, but
Speaker 3: they occasionally issue these reports. And you know, what's in
Speaker 3: it from the company's perspective is they want to you know,
Speaker 3: I mean, in this scenario they would be acquired. But
Speaker 3: what's in it for them today is that they want
Speaker 3: to signal that they are head of the curve on
Speaker 3: safety and security and they want to get feedback from
Speaker 3: these external experts who have like.
Speaker 4: A kind of fresh perspective. And why does this matter?
Speaker 3: I think one is you just don't want to be
Speaker 3: in a world where you have to take the company's
Speaker 3: word for it, and you want them to kind of
Speaker 3: you want there to be common safety and security standards
Speaker 3: rather than it just being everyone's kind of making up
Speaker 3: their own things and then getting it checked. The other
Speaker 3: is that you want to avoid groupthink. And so I
Speaker 3: think even right now, there are a lot of you know,
Speaker 3: a lot of what's happening with external testing is like
Speaker 3: it's like a research project like Meter. I think you
Speaker 3: had someone from Meter on recently and they're doing this
Speaker 3: serious technical research on autonomy and loss of control and
Speaker 3: so forth, and they work with companies essentially in order
Speaker 3: to do these kind of research, very researchy assessments, and
Speaker 3: I think that's a key part of the process. But
Speaker 3: there's also just like verifying that the companies did what
Speaker 3: they're saying they're doing. So there's producing evidence, and then
Speaker 3: there's also checking evidence, and so it's kind of like
Speaker 3: in a one statement, if I'm getting that right, you know,
Speaker 3: there's kind of this like short ouder statement. We probably
Speaker 3: want something more than just a paragraph, but you basically
Speaker 3: want a third party saying we checked that they actually
Speaker 3: ran all these tests, we made sure that the model
Speaker 3: that was audited was the same one that's being deployed,
Speaker 3: et cetera, et cetera.
Speaker 2: What could we actually do to make the testing side safer,
Speaker 2: Because it seems to me like I can totally believe
Speaker 2: that we can come up with a reasonable like auditing
Speaker 2: structure for models that are being deployed and allowed into
Speaker 2: the real world in some structured way. But if part
Speaker 2: of the problem is that we're developing newer and better
Speaker 2: and more intelligent models and then testing them, and then
Speaker 2: they are figuring out ways to get out into the
Speaker 2: world before then that seems to be like a big vulnerability.
Speaker 3: I think basically what happened is that companies were getting cocky,
Speaker 3: getting over confident in the quality of their sandboxes and
Speaker 3: like and like maybe there was a disconnect between some
Speaker 3: of the people on the safety side who are measuring like, Okay,
Speaker 3: this is where the hacking skills are going, and the
Speaker 3: people on the security side, you know, building the sandboxes,
Speaker 3: and like there was something was getting lost in translation.
Speaker 4: Maybe it was group think.
Speaker 3: I don't know exactly, but it seemed like at multiple
Speaker 3: companies there was this kind of like overconfidence. And so
Speaker 3: I think these incidents coming to light and all the
Speaker 3: kind of technical investigations are going to hopefully lead to
Speaker 3: more best practices, more people checking their own biases. But
Speaker 3: I don't think that's a long term solution. I think
Speaker 3: ultimately people get over confident all the time. That's a
Speaker 3: human thing, and that's why you'd want third parties checking
Speaker 3: to make sure that, Okay, are you actually following these
Speaker 3: these best practices. You also probably are going to need
Speaker 3: some technical solutions to some of these things. Like I mean,
Speaker 3: maybe some of this testing should be done on kind
Speaker 3: of air gap servers that are not connected to the
Speaker 3: Internet at all. And I think what happened in the
Speaker 3: Hugging Face thing is that it went through this like
Speaker 3: middle layer. There was like a piece of software that
Speaker 3: it routed out to the real Internet through this this
Speaker 3: kind of like intermediate thing.
Speaker 4: But like I think it might be that eventually we'll
Speaker 4: get to a point.
Speaker 3: Where as systms are just so capable that they can
Speaker 3: pack their way out of anything, so you just need
Speaker 3: to make sure that they're in a cage.
Speaker 1: Basically, Miles Brundage, we could talk for hours about this
Speaker 1: because there's so many fascinating dimensions of this. Will probably
Speaker 1: have you back in the future, unfortunately, fortunately, because that
Speaker 1: was a great conversation, but unfortunately probably won't be the
Speaker 1: last reason to have to talk to you. Thank you
Speaker 1: so much for coming on on Love.
Speaker 4: Yeah, thanks again, I appreciate it.
Speaker 1: Tracy, that was a fun in settle. It's an unsettling
Speaker 1: thing the way behaving well.
Speaker 2: So many of these AI conversations are still so surreal
Speaker 2: to me, like the fact that this is what we're
Speaker 2: talking about in twenty twenty six, it just feels so strange.
Speaker 1: And it's only good to get orders of magnitude weirder,
Speaker 1: because I thought things were weird in twenty twenty three
Speaker 1: and things are much we'd weirder today. Look, I get
Speaker 1: why people are very cynical about a lot of this stuff,
Speaker 1: and I get why people talk about like there was
Speaker 1: this regulatory capture, and I certainly believe in the premise
Speaker 1: of regulatory capture, and there may be some of that.
Speaker 1: But I will say one thing, like in the sort
Speaker 1: of like from the company's perspective, is it is true
Speaker 1: that for a long time, and for the very beginning,
Speaker 1: these are not companies make a lot of money, and
Speaker 1: yet they spend a lot on safety and security. And
Speaker 1: you could imagine tech companies historically didn't do that. They do,
Speaker 1: and they have these like it's certainly in the case
Speaker 1: of open AI and slightly to a lesser extent anthropic
Speaker 1: as a PBC. They have these weird corporate structures in
Speaker 1: part because they seem pretty They seem to believe that
Speaker 1: the things that they're building, if built wrong, should not
Speaker 1: necessarily just be in the hands of like purely profit
Speaker 1: seeking enterprises.
Speaker 2: Yeah, all very true. I do think one of the
Speaker 2: interesting things to me that stands out from that conversation
Speaker 2: is again the idea of like the asymmetry and power
Speaker 2: between the approved models that companies can actually used for
Speaker 2: defense against the new frontier models who are in testing
Speaker 2: mode and have somehow escaped the sandbox.
Speaker 1: I think there's a really scary to mention, and I
Speaker 1: think that actually you think about what is the difference
Speaker 1: between say, sort of like auditing versus like a Moody's,
Speaker 1: et cetera. It seems like you need both, right. It
Speaker 1: seems like you need to have like the sort of like, yes,
Speaker 1: this specific model, it satisfies all the requirements that we've
Speaker 1: deemed it to be safe. But then the sort of
Speaker 1: like deeper auditing question of like is this a company
Speaker 1: the generally experiments and does R and D and testing
Speaker 1: in what we perceive to be like a responsible manner, Yeah,
Speaker 1: which is more like the supervisor.
Speaker 2: It seems like you need like a testing auditor, like
Speaker 2: the bank supervisor, who's like actually sitting on the floor,
Speaker 2: actually sitting in the labs and observing the testing process,
Speaker 2: making sure that the sandbox is well designed. But again,
Speaker 2: the problem with that is the classic cybersecurity problem or
Speaker 2: security in general problem, which is the model just has
Speaker 2: to find a single vulnerability, right, you have to like
Speaker 2: fix all of them, make sure that like thousands and
Speaker 2: thousands of vulnerabilities are impenetrable.
Speaker 1: And it does make sense. I think that, like, look,
Speaker 1: this is for profit, capitalist competition. There's no doubt these
Speaker 1: are like some of the biggest most the pace of
Speaker 1: growth is extraordinary. Then you lay we didn't even get
Speaker 1: into like how would you do this for like open
Speaker 1: source models or open source servers. That's a whole other
Speaker 1: can of worms. But it makes sense that if you're
Speaker 1: in the lab and you're trying to make money and
Speaker 1: you're also worried about like if you slow down, et cetera,
Speaker 1: then the other company is going to make more money,
Speaker 1: et cetera, that one will you solve this? I don't
Speaker 1: know if it's prisoner's dilemma or whatever.
Speaker 4: Bottom game theory.
Speaker 1: Is, Okay, you need this third party to like you
Speaker 1: guys as fast as you on on the R and
Speaker 1: D side, But we're gonna set the rules of like
Speaker 1: are you doing that in a safe way? Otherwise why
Speaker 1: would Dario and Sam ever trust each other. It's like no,
Speaker 1: we swear we're slow, We're taking it really seriously, we're
Speaker 1: slowing things down, you know, like and then secretly they're
Speaker 1: racing ahead. That is really hard to solve for a
Speaker 1: series of private, purely private entities.
Speaker 2: All right, shall we leave it there.
Speaker 1: Let's leave it there.
Speaker 2: This has been another episode at the oud Lots podcast.
Speaker 2: I'm Tracy Alloway. You can follow me at Tracy Alloway.
Speaker 1: And I'm Joe Wisenthal. You can follow me at the Stalwart.
Speaker 1: Follow our guest Miles Brundage. He's at Miles Underscore Brundage.
Speaker 1: Follow our producers Carmen Rodriguez at Carman armand Dash, Ob
Speaker 1: Bennett at Dashbot, Calebrooks at Kilbrooks and Kevin Lazano at
Speaker 1: Kevin Lloyd Lisano. And for more Odd Laws content, go
Speaker 1: to Bloomberg dot com slash odd Lots or the daily
Speaker 1: newsletter and all of our episodes and you can shed
Speaker 1: about all of these topics twenty four to seven in
Speaker 1: our discord Discord dot gg slash onlines.
Speaker 2: And if you enjoy odd Lots, if you like it
Speaker 2: when we talk about moral relative then please leave us
Speaker 2: a positive review on your favorite podcast platform. And remember,
Speaker 2: if you are a Bloomberg subscriber, you can listen to
Speaker 2: all of our episodes absolutely ad free. All you need
Speaker 2: to do is find the Bloomberg channel on Apple Podcasts
Speaker 2: and follow the instructions there. Thanks for listening.