S1: Daniel.
S2: Hey. How are you?
S3: Good. How are you doing?
S2: Not too bad.
S3: Great. Thanks for doing this, by the way.
S2: Totally.
S3: Yeah. Yeah. It was. It was. You were the one
S3: who said we should do it on camera, so we're
S3: doing it on camera. Uh, I think there's a lot
S3: of people excited about it, so, uh. I'm happy. I
S3: think we have, uh, so the chat is apparently a
S3: little bit messed up in terms of like time delay
S3: and stuff. Uh, but we'll just kind of ignore the
S3: chat for a bit and just assume there's an audience
S3: who cares. And yeah, just do this between us. Uh, sound. Okay.
S2: Sounds perfect.
S3: Great. Uh, nice to meet you. By the way, you
S3: and I have never actually talked on a call before. Like,
S3: we've only interacted on the internet, so. Good to meet you.
S2: Yeah, good to meet you.
S3: Cool. Okay, so what I thought we'd do is just.
S3: I kind of want to start off by just giving like,
S3: a little bit of a summary for people of what,
S3: what the interaction was. Uh, and then I'll just let
S3: you kind of give your take on what your opinion was,
S3: and then we'll just go and have a discussion from there. Cool.
S2: Um, or maybe you can give the background of like
S2: what your opinion was that I responded to.
S3: Yeah, yeah. So I'm going to start from where I, yeah.
S3: So okay. So for anyone who is like just new
S3: to this or okay, so what has happened is I
S3: made a video, uh, of 18 minutes of me just
S3: trashing Dario Amodei. Okay. Who was the CEO of anthropic? Uh,
S3: just saying. Basically I think he's a bad person. Okay. Or, um,
S3: bad person is maybe not the word I use, but
S3: it was, it was along those lines. And I basically said,
S3: the problem is, is that what? And I walked through
S3: in this video, uh, Dario's, uh, position on AI safety.
S3: And for anyone who doesn't, um, understand some of the context,
S3: this can only give a little more information. I am
S3: a little bit fortunate in that I happen to be
S3: when I was doing my master's, I was at Oxford,
S3: which is where Nick Bostrom was. And Nick Bostrom is
S3: a person who was like very early in or like
S3: a big name in the AI safety debates. So I
S3: went and I heard him speak at some event. I
S3: forget what it was exactly. Uh, and so, and then
S3: that made me read up on all of the AI
S3: safety discussions. And this was like 2016. So I'm like
S3: a little bit like more well versed in AI safety as, um,
S3: than your average person would be just because I just
S3: happen to be at the place where they were talking
S3: about it, like where it was kind of was a
S3: big idea at the time. They have like a whole
S3: thing called like the future of humanity. Um, and so
S3: I make this video where I basically walk through what
S3: Dario believes and then walk through what Dario wants. And
S3: then I say basically, uh, no, uh, fuck that. We're
S3: not doing that. Uh, everything Dario wants is bad. And
S3: I think that we should all reject it, and we
S3: should all just completely reject Dario completely because he's trying
S3: to lead society down a path that is bad. And
S3: specifically my argument comes down to this, which is that
S3: Dario believes that AI has a 25% chance of going
S3: extremely poorly for society. That's his position on it. Okay.
S3: And in that world, we want to he wants to
S3: make sure it goes well. And his basically, his proposal
S3: is like, help me regulate all of my competitors. And, um,
S3: let me continue to run AI so that he gets
S3: put in a position where he runs the safety of
S3: AI and he can keep you all safe. Uh, make
S3: sure that you're all safe by not giving you access
S3: to anything dangerous. Um, only he gets access to that.
S3: And so I basically called that out and made fun
S3: of it for 17 straight minutes. Uh, and then Daniel,
S3: you wrote a post and then a number. We went
S3: back and forth a bit. Do you want to explain
S3: what you said, what your position is on this?
S2: Yeah, I don't really remember what I said, but I
S2: but what, what you said just now reminds me of
S2: like what my position is. So, um, so I guess
S2: at a high level, my position on this, um, is
S2: that what you're saying? Sounds perfectly logical. And I, I
S2: actually think we probably agree 80 to 95% on like
S2: pretty much everything here. Um, and I kind of just
S2: have a way of thinking about this, which is a
S2: little bit different. Um, so I essentially think that if
S2: we take certain things into account about what we want
S2: to happen in the world and what we don't want
S2: to happen in the world, the question then becomes, okay,
S2: what do we do about it? Okay. So, so let's
S2: just step through these because I think we probably actually agree. Um,
S2: do we agree that we don't want like a government, um,
S2: authoritarian government who has like AGI, ASI or whatever that
S2: just like subjugates the population.
S3: Yeah. Subjugation of the population is bad. So I think.
S2: Yeah, that's bad. Yeah. Um, so we don't want that
S2: from the government, but we also don't want in your
S2: mind like a Dario type or like a, you know,
S2: super rich, ultra rich, like oligarchy, who's on the corporate
S2: side doing the same thing. Right. Both, both like government
S2: or corporate. It doesn't matter. Like either way, if it
S2: ends up with us subjugated, that's bad. So 1,000% agree
S2: with you there. So I agree with what you're kind
S2: of like attacking. I think you're attacking the wrong person
S2: in the wrong entity. Um all right. So let's keep going.
S2: Do you agree that it would be bad if the
S2: public suddenly received like, um. Let's say mythos or further level, um,
S2: ability to, um, create bioweapons, um, planned terrorist attacks, uh,
S2: create end to end like vulnerability research all the way
S2: to exploitation.
S3: So I think you have to break that. So if
S3: we're talking the cyber capabilities. No, I don't think that'd
S3: be bad. I think that'd be good. I think like
S3: from a cyber perspective, if the whole public got access
S3: to like mythos plus plus, let's call it like mythos seven,
S3: and it's like the greatest hacker of all time. Like
S3: I still think that's net positive for the world because
S3: like cybersecurity is not magic. So like, there are still
S3: ways to protect, like, it's not like, you know, a
S3: mythos seven can not just hack anyone, even if like,
S3: there has to be some issue to hack. Right? Um,
S3: so I'm not really, I don't.
S2: Okay, well, maybe this is our, maybe this is our
S2: first level of disagreement. Um, uh, let's see. Okay, I
S2: guess we can go tactically down into here. So it's
S2: not my main point, but but maybe this is part
S2: of the main point. So you're saying that it would
S2: not be bad for Society. Because keep in mind all
S2: attackers who have been attacking us this whole time. They
S2: would also have this capability, right?
S3: Sure.
S2: Because the whole public does. Everyone has it. Um, you're
S2: saying this would not be a problem?
S3: No, no, I mean, it would not. A problem is
S3: a bit of an understatement. Like, obviously it would be
S3: inconvenient and it would be like a little painful. Um,
S3: but like, is it a thing that I would go
S3: to bed worrying about? Like, no, not really and I don't. Okay.
S3: I think that's what's true of a lot of people
S3: in cybersecurity, as we're pointing out that like, I kind
S3: of asked that question of like, how bad would it
S3: be if mythos level capabilities were available to everyone? And
S3: the answer was, like most people in cybersecurity that I,
S3: of course, biased by the people who follow me, but
S3: like it's a pretty big pool. Uh, we're like, yeah,
S3: it'd be a little painful in the short run. I
S3: think that there's no reason to take actions. And I
S3: think this is sort of comes back to the original
S3: position I have, which is there's no reason to take
S3: actions based on those types of fears. If the actions
S3: themselves require us to do things that are like wholly illiberal. Right?
S3: In a generic sense. Liberal, right.
S2: Okay. Okay. So a couple of things there. Um, so I,
S2: I think we're very far apart on how bad this
S2: would be. Um, so let me ask you this. Do
S2: you not believe that most of our software and, and
S2: mythos is essentially like showing us this? And it's not
S2: just mythos, like a good AI stack on a previous
S2: level model can actually do the same thing. I've been
S2: doing this forever. Um, that it can reveal the fact
S2: that our entire infrastructure, basically everything that our society, by
S2: the way, your video's frozen for me. I don't know
S2: if it's frozen for other people, it's probably probably just
S2: my computer. Um, but just in case it's not, um, basic,
S2: basically my argument is, um, and, and just like you
S2: were saying, people agree with you about this. There's lots
S2: of people agree with me about this. Um, our infrastructure
S2: is basically a giant soup sandwich. It is a complete mess. Um,
S2: Linux is massively still vulnerable. Windows is massively still vulnerable.
S2: All the critical infrastructure runs on software that is full
S2: of holes, which we simply haven't taken the time to
S2: look for. And if we did take the time to
S2: look for it, we would find serious issues. So if
S2: the whole world, including all the bad guys, suddenly get
S2: the ability to uncover those, that is not an inconvenience.
S2: That is seriously bad. Especially if you do it all
S2: at once and you just release the model if this
S2: suddenly becomes good.
S3: But your your take on it was that this is
S3: not just mythos thing. It's also true of things like,
S3: you know, if you're running a lot of stacks with like, okay,
S3: other models that you can use to do, but those,
S3: those do exist. And it's not like nothing bad has
S3: happened yet.
S2: No, no. But they require extraordinary. Sorry. They require extraordinary
S2: skill to create and manage. And only a few people
S2: have them, right. Me and my buddies, we have them lots.
S2: Lots of other hacker buddies. Like we have these stacks.
S2: And so it is a you know, the top of
S2: the top has these stacks and has been able to
S2: do this type of stuff. And the models like improve them.
S2: But the more you go mythos mythos plus plus, it's
S2: like a single prompt. I want you to achieve this.
S3: Believe me. Yeah, yeah. Well, I don't think we, I
S3: don't think we disagree on the premise here. I just
S3: think that basically what happens as a result of it
S3: is if you were to do something like, let's say,
S3: hold these models off for release to try to like
S3: solve some problem in advance, like try to gate access, whatever,
S3: like the solution to that might be, it's just going
S3: to be so much worse than just saying like, let's
S3: just rip the band aid off. Some problems will happen.
S3: We'll fix the problems like no, nothing is going to
S3: like the world will go on in very normal ways.
S3: Like I think that if you release those capabilities into
S3: the world, the world will go on and it will
S3: just be like, yeah, some bad stuff happened along the
S3: way and we fixed it and we're good. Okay. And
S3: so like, I don't think that there's any major reason
S3: to be like in a panic about that besides, like,
S3: you know, compared to the types of especially the types
S3: of proposals we're talking about, which would be like, okay, well,
S3: now we need to limit what open rate models can
S3: be put into the public, right? We need like, these
S3: are extremely draconian measures to prevent something I think is
S3: a shrug, you know?
S2: Okay, well, a shrug, okay, a shrug. Okay. All of
S2: our financial systems are based on this software. All of
S2: our economic systems are based on those financial systems. Um,
S2: our critical infrastructure for running, uh, water, energy, everything, everything
S2: is based on running that same exact shitty software, which
S2: it sounds like we agree. We know it's shitty and
S2: we are now releasing into open source models or or
S2: openly releasing mythos from closed source models, whatever. We're just
S2: giving it to the entire world. And you believe that
S2: is a shrug.
S3: I believe it is like, yeah, effectively, I think that
S3: basically we will have.
S2: Well, I think we found I think we found the disagreement. Uh,
S2: I think that is just wrong.
S3: Okay. But like, but that I think gets actually to
S3: like the deeper point, which is like, okay, so imagine
S3: that you're sitting there going like, okay, I believe that
S3: like all of these terrible, that mythos is this amazing
S3: model and that if we release it to the public,
S3: like lots of terrible, terrible things happen, right? The, the
S3: follow on steps and actions from that are like, okay,
S3: so what do you do? Like what is, what's your
S3: proposed plan to that?
S2: Totally. 100% agree. So let me just say this. I
S2: want to I want to make this very sort of
S2: agreement based. I agree with you that if this were
S2: a minor situation, okay, if this were a minor situation
S2: and it wasn't that big of a deal, and then
S2: you have someone like anthropic or someone like Sam or
S2: Dario or whoever, and they're like, mm, I want to
S2: control everything. And like, we can't let this get released.
S2: And they're clearly doing this for many reasons. Then I
S2: see that the same way it seems like you're seeing it,
S2: which is these guys are just trying to control things
S2: like this is nasty. And I guess what I would
S2: make a 17 minute video bashing them just like you did.
S2: The difference is I don't believe that is what's happening.
S2: I believe what's happening is they believe, just like I do,
S2: and tons of people in the military do, and tons
S2: of people who, in my opinion, who are close to
S2: this are like, no, that would be bad. That would
S2: be seriously bad.
S3: Well, to clarify, I believe Dario believes that too. So
S3: of course, one of the things that makes me diverge
S3: from some of the people who just believe it's regulatory
S3: capture play is that I fully believe Dario believes like,
S3: so I don't I don't for any moment suggest that
S3: Dario is doing it all just to exploit something for
S3: his own personal gain. I think Dario is is deeply
S3: versed in the world of like, you know, less wrong
S3: ideology of, you know, less wrong being a blog for
S3: anyone who doesn't know. It's not a.
S2: Rationalist.
S3: Movement. Yeah. You know, e world. Like I think he's
S3: so deep into that he believes it fully. So I
S3: don't accuse him of not believing, uh, what I think
S3: is that basically, um, the belief that he has about
S3: the risks, right? And then the measures he wants to
S3: take in order to address those risks are like completely
S3: out of weight from each other. And for one of
S3: the reasons we found we disagree is that, of course,
S3: I don't think that releasing mythos publicly would have been
S3: the end of the world. But at the second time.
S3: On the other hand, I also think the measures he's
S3: proposing can lead to some extraordinarily terrible things, right? You know,
S3: the measures he's proposing is basically to say, I want
S3: to make sure that China does not have capability to
S3: develop models. Okay. Good models at least, I guess. Do you?
S3: I guess the question is, do you agree with his
S3: proposal there? Like, should we be blocking distillation and should
S3: we be blocking open weight models? Like what do you
S3: what do you think there?
S2: Right. So this is completely consistent with the belief that
S2: I have, which I believe is the same one that
S2: he has that mass release at extremely high pace of
S2: all this stuff is very bad. So we're trying to
S2: like control the flow and the speed to like get
S2: society ready. Um, so I don't believe we should give
S2: this to the CCP immediately. I don't believe we should
S2: give all these capabilities to the public immediately. I think
S2: it needs to be done slowly, um, and carefully and thoughtfully. Um.
S2: I don't believe that he is. People will characterize him
S2: as being like, oh, he hates open source. He hates
S2: all this stuff. I think what he's saying, and I
S2: think is clear from all his interviews, what he's saying
S2: is like, we got to be careful. Okay. So so
S2: let me ask you this. What would a person who
S2: agreed with what I just said, what would they do
S2: relative to a person who is saying, I want to
S2: control everything and I don't want to give to anyone?
S3: No, they would do, of course, exactly what Dario's doing
S3: because I don't believe that it's about I don't believe
S3: that Dario's doing it because he wants to control it
S3: for his own good. I believe Dario is doing it,
S3: and I know a lot of people in cybersecurity disagree
S3: with me, but I believe that Dario is doing it
S3: because he wants to control it. For your good and
S3: my good and all of our goods. Right. Um, Dario
S3: believes this is for the betterment of everyone for the
S3: reasons you're stating, like. Right. The problem I have with
S3: it is that the what he's basically proposing is that
S3: we trust him and that we take a hit on
S3: our side. So, I mean, if you think about it
S3: this way, if we had done what anthropic had asked
S3: for us and has been asking for for years and
S3: years and years and all of their policy proposals and stuff,
S3: especially for the last year, uh, we would have to
S3: pay opus level pricing for an opus level model. Okay.
S3: Whereas as of a couple of days ago, now I
S3: can pay one fourth of the opus pricing to get
S3: an opus. Basically an opus clone like GLM 5.2 is
S3: an opus clone, okay. And it costs one fifth of
S3: opus 4.8. And that is like a huge net benefit
S3: to society and to every American, like basically to every
S3: American company, to every, you know, Western company, to every company. Uh,
S3: that's huge. That's a hugely beneficial thing that we would
S3: have not allowed had we followed anthropic policy proposal.
S2: Uh, is he saying we should limit things like GLM 5.2?
S3: He is very clear that we need to limit distillation.
S3: He wants the US government to make an attempt to
S3: limit distillation more. Okay, so he's like, that's one of
S3: their policy proposals from like six months ago.
S2: Well, isn't distillation just isn't distillation? Isn't he trying to
S2: limit like the CCP's ability to distill from our top models?
S3: Yes. And that's what GLM did, obviously. GLM is like
S3: clearly an opus clone. Like, I think it would be
S3: very hard to believe that they trained GLM without hitting,
S3: without basically training it on Opus. Like they, they behave identically.
S2: Yeah. I feel like this is a little bit of
S2: a diversion, but I'm happy to go down. So basically
S2: I think it's, it's blatantly okay. Do you agree that
S2: the CCP's China's overall strategy of just blatantly, blatantly just
S2: stealing from everyone on the planet to better their country
S2: is like not a good thing?
S3: Well, well, if they're releasing it to the public, like
S3: as an open weight model the way that they have been.
S3: As long as that is there.
S2: For the for the purpose of damaging the United States,
S2: that is, the strategic goal is to.
S3: Yeah. But how how did they damage the United States
S3: for releasing GLM 5.2? Like that's just it just doesn't
S3: I don't.
S2: See no, no, the overall Chinese AI strategy. Um, and
S2: to just to get my bias here. So I'm like ex-military, uh,
S2: I wasn't like super elite or anything like that. But
S2: my bias is that like, I see, I see things
S2: in terms of like, what, what happens if the adversaries win?
S2: What happens if we win? What does it mean to
S2: want somebody to win? Why does it matter? And ultimately, like,
S2: I'm a Star Trek liberal. So like, I think we
S2: should all just get along anyway. But in the meantime,
S2: the CCP is stealing everything, trying to win because they
S2: just want to win everything. So the strategy for China
S2: is to completely underpin US dominance via AI, by harvesting
S2: everything through distillation and making it open source and getting
S2: it into open source as fast as possible.
S3: But that's but you think about it from my perspective
S3: as just like an American citizen, which by the way,
S3: I know a lot of people on the internet think
S3: I'm not. But like American citizen here, I.
S2: Was trying to figure that out. I'm like, you do
S2: an amazing American accent. Yeah. For an Oxford guy.
S3: Yeah, I'm American. Just happened to move around too much. Um,
S3: you know, uh, the thing is, as like, an American
S3: running like a company and, like, living my life and
S3: all of that, it feels a very much like China's
S3: helping me, okay? They're giving me great models at low prices. Right? Um,
S3: open weight stopping these. Okay. And then the question and
S3: then the question becomes, if you want me to believe
S3: that it's all like, this is a plan from China
S3: to hurt Americans and all of that. It's like you
S3: have to be willing to give something quite, quite strong there.
S3: Otherwise you're asking like people like me to give up
S3: all of the benefits that we get from this in
S3: exchange for like, some global foreign policy that really doesn't
S3: seem to affect us much at all. Like, I don't
S3: notice these things in our normal life. I notice like
S3: the fact that, okay, I now can't even use this
S3: model or I can't do that. Like for my daily
S3: life and for my work. Chinese models giving me like
S3: open weight models is like just positive for most Americans.
S2: Yeah. And they're subsidizing it. So it's cheaper too. It's
S2: like very nice of them. So, so basically it's true.
S3: Wait, but that's not true because from the perspective of inference.
S3: So if you meant inference there, that's not true because
S3: like Google Cloud's pricing is, is like $1 per input token,
S3: which would be compared to like $4 per input token
S3: for opus. And of course, Google Cloud's like, that's just
S3: Google cloud. It's not China subsidizing that. Of course, you
S3: could be saying like that they capitalizing the cost into the.
S2: No, I'm saying like.
S3: They really are.
S2: I'm talking about like EC2, like a whole bunch of
S2: Chinese models that are available that people can use as
S2: alternatives to US models. They are super cheap, and you're
S2: literally taking all of the stuff in your harness and
S2: all your cool ideas, and it's going off to China.
S2: I'm sure they're doing nothing bad with it, right?
S3: But that's not true for like, but that's not true.
S3: If you're running it on like Google Cloud. So if
S3: you run any of these.
S2: Oh, sure sure, sure.
S3: Hosted. Right. And they're still cheaper. So the point is
S3: it's not about stealing our data because they're not getting
S3: the data. It's not about that. They're forcing us to
S3: use their AI APIs because they're not. And that's actually
S3: one of the things is that that's very hard for
S3: me to accept, I think very hard for a lot
S3: of people to accept, which is like Anthropic's plan is
S3: to say, uh, we and maybe a few other labs
S3: in America will have the best models. You will have
S3: to go through us for your own safety. Um, we
S3: will block the companies that were willing to give it
S3: to you cheaper and better because for your own safety. And, um,
S3: and no, we don't have any intention or plan of
S3: giving you cheaper models, right. We don't have any intention
S3: or plan of giving you better models that will be
S3: able that will serve your actual purposes. So you see
S3: why that's such a hard sell. Like politically.
S2: I hear you, I hear you. That sounds it sounds awesome.
S2: Sounds amazing. Um, it sounds amazing. It really does. Like
S2: I'm not being facetious there. Um, the, the issue is that, okay,
S2: let me continue to propose this, this training strategy, um, the,
S2: the concept of, um, the US companies having a stranglehold,
S2: like super strong control, the best ice that just keeps
S2: getting better and better. And China is kind of behind
S2: that is not okay for China. So they decided, I
S2: don't know when like a year and a half ago,
S2: three years, who knows to go the open source model,
S2: just really everything public, public, public. Um, now, so I
S2: happen to study China quite a bit. I've read like
S2: dozens of books about it. They, they just clearly state
S2: what their strategy is, which is to win, and for
S2: the U.S. to become a minor power with them being
S2: a major power like this is not conjecture. Like, they
S2: just write about it. They talk about it openly because
S2: they think no one is is actually reading anything. They're
S2: writing because it's in Chinese. So it's very clear what
S2: they're actually trying to do. Right. Um, and the fact
S2: that they're giving out, oh, we're just free and open.
S2: It's not free and open in China. It's not free
S2: and open in Hong Kong, which they took over. Um,
S2: it's not like they're going to give us all these
S2: models and then we're going to be like, yeah, so
S2: what are alternatives to Chinese government? And they're like, yeah,
S2: that's a perfectly, you know, natural question you're allowed to ask.
S2: It is allowed. We are allowed to ask it in
S2: the US. We're not allowed to ask it in China.
S2: So like, I don't think I mean, you're you're you're
S2: kind of jumping ahead to a world in which I
S2: agree with you. Okay. Both of you and I agree
S2: in the Star Trek world. What are you talking about?
S2: All this should be open source. I 1,000% agree with you.
S2: This should all be open source. We shouldn't have these
S2: crazy governments like me. And you are probably Star Trek liberals.
S2: I'm talking about how we get from here to there.
S2: And I don't want an extra step where CCP runs
S2: all of this. Um, now, now, let me ask you this.
S2: You're saying that the actions anthropic is taking is like really,
S2: really bad. Wasn't his action to go and get all
S2: of his competitors and get them all on board and
S2: do this shared community release of like, we're going to
S2: give this to hundreds of companies, they're going to come back,
S2: we're going to roll it out slowly for the purpose
S2: of doing the thing that you said, which is you
S2: kind of like, do the controlled burns in the, uh,
S2: in the forest. You burn things down, you clean things up,
S2: then you start releasing it. And now I would agree
S2: with you, once you've done that, maybe it's okay now
S2: we can release more, but you don't just drop it. Um,
S2: because here's the thing, dude, they don't they don't hate money.
S2: They like money. They would have made so much more
S2: money if they just released mythos.
S3: Of course. And, and, and believe me, it's because I
S3: believe that they're believers. Like, I think it comes down
S3: back to that now on the China point, like it's
S3: important to understand. Like, so I actually, I studied at
S3: the University of Hong Kong law school for a year, uh,
S3: the year prior to Hong Kong, basically getting completely, you know, done, uh,
S3: and that the protests that kicked off actually kicked off
S3: from the University of Hong Kong law school. So I
S3: was like at the epicenter of this point. Um, and
S3: I have tons of friends who are, uh, from mainland China,
S3: including one who by all accounts, I will, you know,
S3: can pretty openly say he is like clearly a spy, uh,
S3: for China. He works for in the US for a
S3: tech company. Um, and I happen to know his background. Right.
S3: Well enough to know that he would be a spy
S3: for China. Um, and so, but the fact of the
S3: matter is, and like, no, of course I don't think
S3: that's great. Um, you know, at the end of the day,
S3: I'm sitting here going, yeah, but they are giving us
S3: models that. No. Yes. If I asked about Tiananmen Square,
S3: it would deny that. Right. Like it will deny that request. Okay.
S3: But I don't typically have to ask my AI models
S3: about Tiananmen Square. Right? Like it's not it's not a
S3: thing that comes up when I'm doing like any type
S3: of cybersecurity work, right? Those types of like, can they
S3: influence it with pro-Chinese bias is such like an abstract
S3: concept of how you might influence a population in comparison
S3: to like, what if only Dario controlled it? And now
S3: when you mentioned it like, okay, but that it was
S3: that he went and got all of his competitors together
S3: to this is just what we call like collusion, like
S3: it's illegal for a reason and normal circumstances that would
S3: be a crime. Uh, we're willing to accept it because
S3: we pass it off through the safety exception. But in
S3: normal circumstances, putting together, like all of the leaders of
S3: all of the major companies of a Get competitors and saying,
S3: how about we cooperate on this would just would just
S3: be illegal because that would be a cartel, right? So
S3: we don't want cooperation. I do not want cooperation between
S3: these labs. I want them to compete.
S2: Okay. But they're competing. You're familiar with like the Malek,
S2: like you're chasing this thing that you know is going
S2: to kill you and you're you're aggressively chasing a thing.
S2: Everyone knows if we get there, it's going to kill us,
S2: but we can't let the other person win. So we
S2: accelerate like that.
S3: That's that's what they live in. That's what they believe. Right?
S2: Well, no, but that's what they would do if they
S2: weren't collaborating in some sort of way towards a positive outcome. Right.
S2: So so they literally said, and Sam has also said
S2: this earlier, um, I've a whole, whole nother talk show
S2: on Sam, but, um, they're not just saying, let's collaborate.
S2: They're saying, let's collaborate with this thing, check this out.
S2: This concept of government is the idea that the people
S2: are involved. So it's them, all of them getting together
S2: with the government to say, hey, what is the best
S2: thing that we should do? That is literally the consortium
S2: that that he is arguing for and, and saying he's saying, look,
S2: we need to be regulated. People need to be involved.
S2: We want all of our competitors to be involved. We're
S2: trying to control the speed that this comes out. And
S2: that's literally what he's proposed and.
S3: Done, because I don't want them to cooperate on the
S3: speed with which it comes out, because that means that
S3: they can both say, hey, because what I thought when
S3: they didn't, by the way, in the chat, there is
S3: a guy named Daniel that I worked with as a
S3: developer for years ago. That's why I laughed because like, yo,
S3: what's up Daniel? Uh, just like it's been like eight
S3: years since I talked to that guy. Uh, but anyways, um,
S3: back on the point if we go back to that though,
S3: is that if you think that like, uh, when mythos
S3: came out and they said, we're not going to release
S3: mythos to anyone, I was like, watch it. Because what
S3: happens is, of course, if Sam Altman releases his version
S3: of mythos and like, then of course it kind of
S3: forces Dario to release mythos, Right. So the game like
S3: the game here of competition forces them into the world
S3: that like I want to live in, which is that like,
S3: we just get the best models and we get the
S3: best prices and we get it the fastest. That's the
S3: world that I want. And so the, the collusion they're
S3: trying to do is to avoid that world, but it's
S3: coming at a cost and the cost is not on them.
S3: The cost is on us. They're already trillionaires. Right. It's
S3: like the permanent underclass that loses from this. It's all
S3: the people who are not trillionaires and do not control
S3: the AI, you know, or are not friends with anthropic,
S3: because of course.
S2: We still have. We still have. I'm sorry.
S3: Yeah, but but they but but opus being much more
S3: expensive than what we got from the labs that he
S3: doesn't want to have it now, but also importantly, um,
S3: you know, you mentioned what I was about to say there, uh,
S3: I forget whatever that was it.
S2: Okay. So I, I think, I think I'm hearing something
S2: strange in the argument. The reason I reacted so viscerally
S2: to your video is because you were saying Daria was
S2: a bad person. Now you're telling me I agree with you.
S2: He's a good person. But the output, the output that
S2: he is asking for would be bad for us. No, no, no.
S3: I don't think he's a good. I don't think he's
S3: a good person just because he genuinely believes. I mean,
S3: like I said, I think in the video there are
S3: people who have believed in like ethnic cleansing and they, like,
S3: genuinely believe in it. Uh, but it's like still evil. Right.
S3: And I think it's the same as that to he
S3: genuinely believes he's keeping us safe and that it's better
S3: for everyone to keep us safe and that he's doing
S3: that for safety. Um, the problem is, of course, is
S3: that he is willing by doing that, he is willing
S3: to basically create a society in which very few people
S3: are benefiting because at least they didn't die. Okay. I think, uh, what.
S2: Are you talking. He still wants to release all these models.
S2: He still wants to release them.
S3: No, no, no, he wants to release the current level
S3: of models. He's very clear on that. The next. and
S3: the next generations of models we will never receive. Okay,
S3: he is very clear on the fact that he believes.
S2: What are you talking about? They're in development. They're going
S2: to come out. What? What are you talking about?
S3: It's his quotes. He literally says, in the future, you know,
S3: we might have to treat this like nuclear weapons, not
S3: as a safety risk, but rather as a risk to
S3: society and to humanity, and that we would need tighter controls.
S3: You know, basically the point being then it is not
S3: a public good. It becomes, you know, not not a
S3: public good is the wrong term there. It is not
S3: a good that he would be willing to release to
S3: the public ever. I think Dario would not deny that.
S3: I think.
S2: But if you look at the context.
S3: Of that, Dario does not want to release all future
S3: models if they become sufficiently smart.
S2: He's talking about in the case where it becomes certain
S2: amount of dangerous. And like you shouldn't just drop it
S2: on the world, which is my point. Okay. Which I
S2: think we disagree about. Yeah. But we disagree about the
S2: potential impact of that. He believes, as I do, that
S2: the potential impact of dropping. Let me give you a
S2: better example. Okay, let's call this mythos six or mythos
S2: seven or whatever. Okay. Um, give me I don't even
S2: know if I should say this prompted. Um, yeah. Okay.
S2: Imagine the creation of a wet lab combined with doing
S2: the research for creating, uh, uh, different, uh, pathogens that
S2: combine the worst of the worst. And the problem with
S2: this is that the knowledge has been there. Like if
S2: you could jailbreak or whatever potentially been there, but as
S2: you start to move into AGI, ASI maybe mythos 6728
S2: who knows when it's going to happen? It starts to
S2: be like, oh, actually, if you make a cut here
S2: in this particular virus and you add this, um, okay,
S2: this will kill everyone. This will effectively do what your goal,
S2: you did a slash goal. And guess what? I made
S2: it for you. The problem has always been, okay, that
S2: knowledge is fine. Kind of like with nuclear. That's fine.
S2: The knowledge is there. But you don't have a lab. Well,
S2: guess what? I'm going to give you all the stuff
S2: to go and make this in a lab. And I'm
S2: going to help you automate the process. And now we've
S2: actually built it. This little canister here now has the
S2: thing you believe that is safe to provide that level
S2: of knowledge to the world, the entire world. Give it
S2: to CCP, give it to Joe next door.
S3: Importantly, here, this this kind of moves into the world
S3: of what we consider like, I think Nick Bostrom refers
S3: to it as like this idea that you can make
S3: nukes like in your microwave, right? If there is some
S3: piece of technology that like we discover that would, whether
S3: that's through AI or something else that would allow like
S3: every random person to go make like civilization ending technology
S3: at home easily, right? That would be catastrophically bad for
S3: the existence of humanity. That would be like probably civilization
S3: ending if it got out. Like that's what we're talking
S3: about there. The problem is, of course, I don't believe this. Okay.
S3: I think that we've not seen sufficient evidence that AI
S3: will get to a level that will allow that to
S3: be possible. And with that being the case, what happens is,
S3: is that we're starting to be asked to accept, again,
S3: like fairly liberal positions of like, we will allow for
S3: the AI labs to control larger and larger. I hear
S3: you get the best models. All in favor of this
S3: theoretical possibility. And now I'm sitting here going, well, let's
S3: say that Dario says the counter to this is, well, Zach,
S3: you don't have the access to the information like I.
S3: Dario do. Dario. I'm Dario and I can see the
S3: model does it. The problem is of course I don't
S3: want Dario to be able to see this either. Because
S3: if it is at that level. Of course I think
S3: it should be like completely expropriated, you know, expropriated out
S3: of his hands and like the government should just roll in.
S3: But he doesn't want that either, which is kind of
S3: a good signal that this is not that. Right. This
S3: is not telling people how to.
S2: What do you mean? He is literally saying we should
S2: be regulated. This stuff is super dangerous.
S3: Regulation is not what you would do if you believed
S3: that you had civilization ending technology in your hands. And
S3: I think that.
S2: He believes it's it's about to come. He believes it's
S2: starting like it's very clear from his statements.
S3: Do you think that Dario. It's a very interesting question,
S3: is if I believed. So this is one area where
S3: I do think it becomes very clear where Dario and
S3: I disagree. The strongest, besides just about what we believe
S3: comes in the future. If I believed that that was
S3: coming in the future, there would be one and only
S3: one appropriate answer, which is to hand it over to
S3: the government and say, look, guys, this is now a
S3: matter of your this is on, you.
S2: Know, now we're now we're getting to it.
S3: No, no, no, but not as a matter of regulation,
S3: but as just full on nationalization, because most people would
S3: accept that you should not allow civilization ending technology to
S3: be developed in a private lab, uh, and making trillions
S3: of dollars. But Dario referred to that as communism. Okay.
S3: So Dario very openly said, this is communist. Okay.
S2: So say that again. Say that again. What did he
S2: say was communist?
S3: Uh, nationalization. So when he said that some people argue
S3: now that because of the risks that it should be nationalized,
S3: he said, guys, you're yo yoing between, uh, full, um,
S3: basically full libertarianism and full communism. And so is of course,
S3: the implication being that nationalization was the communist position. He
S3: did not he if if, if Dario came out tomorrow
S3: and said, I think that the US government should nationalize
S3: all AI labs, I'd say I'm sorry to Dario because
S3: at least he has. I would say he's wrong because
S3: of course, I don't believe in the pre. The premise
S3: that this is going to become make a wet lab
S3: at home, kill everyone technology. But I would at least
S3: say he's drawn the correct conclusions from the series of
S3: things he believes the problem. The biggest problem I have
S3: with Dario is that he wants to believe that, uh,
S3: this is civilization ending technology and that he should be
S3: able to make $1 trillion off of it. And that's
S3: like super weird. Like, why would anyone think that?
S2: What do you mean? So, so, so I think our
S2: positions are extremely close here, Extremely close. I think this
S2: is a framing issue. Okay. So first of all, the
S2: government has no ability to do anything. They they have
S2: no AI expertise. They can't make anything. We can't nationalize crap. Um,
S2: guess what an actual nationalization looks like. An actual nationalization.
S2: Looks like the labs get together and they're like, hey,
S2: we're inviting the government in to be a huge part
S2: of this because this is super important, which happens to
S2: be the exact thing that he's asking for.
S3: No one. So if I went and asked my parents
S3: who don't know about AI or my brother who's a lawyer,
S3: or any one of my friends who aren't in technology
S3: to explain that premise to me, which is that this
S3: technology is being developed and it might kill everyone in
S3: the world, right? It might cause basically mass, mass, mass extinction,
S3: which is the belief that he has. Right. And then
S3: he said, okay, but they don't believe that this should
S3: be nationalized and they just believe it should be regulated.
S3: All of the people, all of those normal non-tech people
S3: would go, what? That's absurd. Obviously, we don't want to
S3: see people, uh, having any sort of. You know, you're
S3: basically it's a negative externality that you're basically saying, what
S3: if we killed everyone and made $1 trillion in the meantime?
S3: I mean, like, what are you talking about? Of course
S3: you want to support that.
S2: What does nationalization mean? What I'm arguing to you is
S2: that there is a smooth spectrum between the government having
S2: control and oversight and transparency into all U.S. companies who
S2: are pushing towards the same goal. And there's like some
S2: sort of internal consortium there with the government actually having
S2: veto power and control. And and actually the labs are
S2: the ones doing the work because the government doesn't there
S2: is no nationalization there. Like, there's not a government thing
S2: that is just like anthropic and just like Google. And
S2: it's just as good.
S3: It would be anthropic. You would expropriate it, it would
S3: be yours. Now you own 100% of the shares. Game over. Right?
S3: It would be anthropic.
S2: Yeah. Okay. So imagine you're Dario. Do you think that's
S2: a good idea right now, given our administration?
S3: Of course. Dario does not believe this.
S2: Okay. Do you believe it's a good idea? Okay. Let
S2: me ask you this. Do you think it's a great
S2: idea to hand all the power of of anthropic and Sam?
S2: Sam and Dario should go over and meet with Hegseth
S2: and Trump and say, hey, this is all yours. Um,
S2: we just got a piece out of this game because
S2: you guys are super you guys are super talented in this,
S2: and it's all yours. Take it.
S3: Wait, I, I absolutely don't believe that because of course,
S3: I don't believe the things that Dario believes about the world.
S3: But if I was sitting in Dario's shoes believing the
S3: things Dario believes, yes, I would, because I think that
S3: even though we might say like, oh, well, we don't
S3: think that the Trump administration would do a good job
S3: or whatever you might believe. Uh, at the end of
S3: the day, a public process is like, that is the
S3: best system we have, what they're saying. And the reason
S3: that I have such objections to what Dario's position is,
S3: is to say, you can't trust this government which was elected.
S3: You can trust Dario, who wasn't elected. As long as
S3: we get a light level of regulation from the government. No,
S3: I think that absolutely, if you were to move to
S3: this fake extreme world, which I do not believe in,
S3: you would believe in in just full nationalization. No one
S3: should be profiting off of taking a 25% risk of
S3: everyone dying. Okay. So like, that's a very basic premise,
S3: which is like, if you believe there's a 25% chance
S3: of extinction risk, which admittedly is not what Dario said.
S3: It said 25% chance of very, very, very poorly. He
S3: didn't call it extinction risk. But if you believe 25%
S3: chance of going very, very, very poorly, I don't believe
S3: that we should be driving that as a for profit
S3: profit enterprise. That would be basically just saying if if
S3: it goes well, we become your new trillionaire overlords. If
S3: it goes poorly, you all die and who cares, right?
S3: That's that's just not an acceptable state of the world.
S2: I hear you. So so I think you're talking about
S2: worlds that don't exist. Zach. That's what I think you're
S2: talking about.
S3: 100% agree because I'm talking about the world's Dario has
S3: put forward.
S2: No no no. I think you're talking about worlds that
S2: don't exist. Okay. I believe that this progress is is happening. Um,
S2: I believe Dario when he says they were surprised by mythos.
S2: Do you agree with that?
S3: Of course. Yes.
S2: Okay. So he was surprised. He's like, Holy crap, this
S2: is way better than I thought it was going to be.
S2: This is happening faster than I thought it was going
S2: to happen. Okay. Do you agree that most AI experts
S2: are worried about the future and quickly being able to
S2: have this kind of capability?
S3: Yeah. Most AI experts, yes. But that's that's a narrow
S3: pool of technology.
S2: No, no. But what what I'm what I'm saying to you, Zach,
S2: is that there is a consensus. Everyone knows. Every book
S2: written knows. Holy crap. We are going into something very strange. Uh,
S2: we're at the point where things like auto research can
S2: just make AI better, right? And it's slowly happening. Like,
S2: this is such an inevitable thing.
S3: Yes. Okay. Okay. But yeah. Go on.
S2: So so it's it's inevitable. We agree on that if
S2: it's inevitable and.
S3: Well I.
S2: Don't. And hand in handing it.
S3: Yes. Yeah.
S2: Handing it to. Well I don't I don't think anthropic.
S2: If they shut down tomorrow they literally turned everything off
S2: and went home. That would not. It would be a blip.
S2: It would be a blip.
S3: Of.
S2: Course. And overall.
S3: Progress. Anthropic should shut down. I'm you know.
S2: Right, right. So this is why I'm saying that that
S2: we're not arguing in a realistic way. If we're saying
S2: this is inevitable and anthropic shutting down tomorrow would would
S2: only be a blip in the progress. Then the reality
S2: we're dealing with is this is happening and we need
S2: to try to do the best we possibly can. Yes. Okay.
S2: And one, one thing that is not the best we
S2: can possibly do. One option is handing it over to
S2: the current US administration, and walking away and going home
S2: with your billions and starting a farm. That would be
S2: super cowardly.
S3: But but okay, but let's get back to the point.
S3: What I believe the best thing to do is that
S3: they just keep on building these models in a competitive way.
S3: Where OpenAI builds their models and anthropic builds their models,
S3: and maybe Google decides to build models again, and China
S3: builds their models. And we do not do anything as
S3: the US government to regulate China so that the American
S3: people can get the best models at the best prices. Right.
S3: That's what I believe. Okay. Uh, fundamentally, because if we
S3: accept the worlds of like runaway AI, that might kill
S3: everyone and all of these very much extreme positions that like,
S3: if you believe all of this is true, uh, it
S3: leads you to conclusions that are still different from what
S3: Dario is doing. Okay. Which would be like, you should
S3: believe in nationalization at that point. Like if you are
S3: publicly minded, right? Uh, well.
S2: But right now, he wouldn't want to give it to them.
S2: He wouldn't want to give it to them.
S3: Who gave? But who gave Dario the choice to make
S3: that call? If it is what he says it is,
S3: Dario has no right. Okay. It doesn't matter that I
S3: don't I don't want you know, it's like it doesn't matter.
S3: It's like at the end of the day.
S2: He has no right.
S3: To say yes.
S2: He has no right to do what I don't understand
S2: what he shouldn't be allowed to do.
S3: To decide to say, if you, like, believe all of
S3: these things and you're actually sitting in the shoes of
S3: saying it's going to kill everyone, or it might kill everyone,
S3: whatever it may be, unless somebody really takes leadership and
S3: Dario decides that Dario should be the person to take
S3: that leadership instead of, for example, the public, like the
S3: US government, right?
S2: Like Dario, Dario is taking a pro-u.s. position, which is
S2: the US should get there first.
S3: Yeah, but but Dario is not doing that through any
S3: sort of public apparatus. Dario is doing that through a private.
S2: Yes he is. Yes he is.
S3: He's a private company.
S2: He's attacking Jensen uh Wong publicly against everyone else, saying
S2: we should not be shipping our best GPUs to China.
S3: I know, but the point is, is that he's taking
S3: a pro-American position that also benefits.
S2: It's not because he benefits. No, it's not because he benefits.
S2: It's because he believes.
S3: It's because he believes that it will make him remain
S3: in power so that he can lead the world to
S3: the best path of the future. Like I think he
S3: does believe that he would be the best person to
S3: lead us to not dying. Okay. I think he does
S3: believe that. The problem is, is that.
S2: He believes he's one of the best people. One.
S3: Exactly. The point is, I find the whole premise like
S3: kind of repulsive, which is that like, if I were
S3: sitting like, I believe just to say it from this angle,
S3: like I believe very much in myself, right? I believe
S3: like I am a very good guy and all of that.
S3: But if I was sitting in the shoes of saying like, okay,
S3: I have this great technology, blah, blah, blah, and if
S3: I were saying like, and I trust myself more than
S3: the public system, I would still hand it over. Okay.
S3: And I would hand it over because I believe that
S3: there's like a process issue here, which is that like,
S3: no one picked me to decide that, okay, even if
S3: I pick me. Dario was saying pick me and this can.
S3: If you think about this future state of the world,
S3: imagine that. Like you believe in this recursive take off, like,
S3: you know, self-improvement, you know, at each point. Dario could
S3: keep saying, well, I'm actually the best person to be
S3: the dictator and I'm actually the best person to be
S3: in charge. And I'm actually the best person to be. Right.
S3: We don't want that. I want to make sure that
S3: that doesn't happen. I want to make sure that if
S3: this fast take off thing happens and that self-improvement happens,
S3: it is not controlled by somebody who thinks that they
S3: should be in charge of it.
S2: Yeah. So this is fascinating that we're talking about this.
S2: I'm reading the culture series right now. Um, I just
S2: got done with the games one and, uh, whole bunch
S2: of conversation about this. It's, it's crazy. So it got
S2: me to reading, uh, the Machiavelli stuff because, um, that
S2: just came out on Dwarkesh podcast and that got me
S2: to reading, um, Hobbes Leviathan, which is like, which goes
S2: back to Plato, which is philosopher king. And, um, what
S2: if someone believes that they're the right person or whatever,
S2: and they just think they're the best and you want
S2: to give them all the power? So I see the
S2: thing that you're getting at, which is like you're saying,
S2: it's kind of gross when someone is like, I am
S2: the good guy, I'm going to do my best. Right.
S3: Yeah. Do my best individually without, you know, obviously everyone should.
S2: No no no no no. But see, here's, here's here's
S2: my bigger point. Okay. I think there is a really
S2: gross narrative which says that if somebody believes something in
S2: a positive future, they believe that they can help in
S2: some way. They believe they can contribute and they they
S2: go into the arena and they fight for what they
S2: believe in, that people like us should sit on the
S2: side and we should be cynical. We should assume the
S2: worst of them. We should attack them. And I think
S2: that is super wrong. I think it's super fucked up.
S3: But I don't assume the worst of Dario. I assume
S3: the best. Dario. Dario is terrible for it.
S2: No no no. You are assuming the worst of Dario
S2: that your video is assuming the worst of Dario.
S3: Which thing do you think that I believe about Dario.
S3: That's not true because I, I thought in my view
S3: that I expressed every opinion Dario has the way Dario
S3: has them, right. Like I, I tried to um you know,
S3: I'm not making any assumptions about his intention that is different.
S3: I tried to say this is his intention is what
S3: I really believe is true. If, if I don't think
S3: there's anything in that video from that perspective that Dario
S3: would say was wrong, at least besides the part where
S3: I call him shitty. Right?
S2: Um, no, no, but but what? You're what you're saying.
S2: I'm sorry. Go ahead.
S3: The next piece of that, though, is to say just
S3: because somebody is like you're saying, okay, they're out there,
S3: they're trying to make the world better. They're trying to,
S3: you know, like I said, there are people who've committed like,
S3: like genocide under those terms, right? They genuinely believe that like,
S3: you know, you know something, some group is after them
S3: or whatever. And then and they're out in the arena
S3: trying to tell people not to, you know, trust this
S3: ethnic minority group. And then people are critical of them.
S3: And we say that they, you know, you see, it
S3: doesn't it doesn't change anything. It's like it doesn't change
S3: the in conclusion, which is that like, in the end,
S3: what Dario wants is bad for all of us. And
S3: he wants it kind of knowing that he kind of
S3: knows it's bad for all of us because he doesn't
S3: believe it's as bad as what could happen, which would
S3: be like extinction or like genocide or things like that. Right.
S2: Okay. So it sounds like you're arguing against E.
S3: Oh, yeah. Oh, 100%. Yeah.
S2: Well, so, so what I haven't studied is how close
S2: he is to e my read on him is like
S2: he doesn't care at all about that shit. He literally
S2: is going in a different direction. Um, I hear, okay,
S2: let me just I'm going to steal your position. Um,
S2: he thinks he understands the world. He thinks he's trying
S2: to do the right thing and he's wrong in some
S2: sort of way, but he is mostly good and he
S2: just is like Jesus like syndrome, where it's like, I'm,
S2: I'm going to save the world or whatever, and he's wrong.
S2: And if too many people give him the power, um,
S2: he'll end up doing bad things for us because he
S2: doesn't want us to have all the toys or whatever.
S3: Not, not quite what I would say. I would say
S3: it's not just because that would not make him bad.
S3: That wouldn't make him a bad person. Okay, so being
S3: wrong doesn't make someone bad, right? Um, what it is,
S3: is that he is wrong in a specific way, which
S3: leads him to a conclusion which is, uh, on its
S3: own grounds, immoral, which is basically to say that he
S3: should be trusted with an amount of power that no
S3: one entrusted him with, and that he can seize that
S3: power for himself unilaterally. Okay. Because he happens to control
S3: AI because he happens to be the CEO of anthropic.
S2: But he's not doing that.
S3: But he is, because he's trying to maintain private control
S3: over AI while believing in this self recursive self improvement world.
S3: Like that's just not how I would play it if
S3: I were in his shoes. Now, of course, I don't
S3: believe any of this, so I would not I'm not
S3: asking Dario to hand over to the government. I'm asking
S3: Dario to just compete. Just play the game so we
S3: can keep this ball moving. You know, keep the game rolling.
S2: But he is competing and he's inviting collaboration from his
S2: competitors and inviting the government.
S3: So what I'm asking is like of people listening to this,
S3: what I want is for you to basically not vote
S3: for politicians that would believe in supporting the types of
S3: policies that Dario wants. Okay. I think we should not
S3: support Dario's policies are bad for most people. Uh, unless
S3: you believe in extinction risk. If you believe in extinction risk, okay, fine.
S3: We can play a different game. But if you don't
S3: believe in extinction risk, how could not?
S2: How could you not believe in extinction risk if most
S2: AI experts believe in it?
S3: Well, okay, so that's like there have been plenty of
S3: historical examples of like people believing, like groups of people
S3: believing things that aren't true. Right?
S2: No, no, but but, but I, I gave you an
S2: example of the wet lab and the the advancements of AI.
S2: So so let me just pitch this to you. Um,
S2: it will not be long before open source models and
S2: closed source models will be able to do that end
S2: to end and lab, uh, wet lab thing and the
S2: end to end cyber, um, zero days that that will
S2: not be long. Do you agree with that?
S3: No, no, no, no, I, I don't I don't necessarily
S3: have a strong opinion of course end in zero days. Yeah. Sure.
S3: That's not that's not that's already kind of there. Um okay.
S3: Do I believe that the extinction risk I mean zero
S3: day doesn't cause extinction risk by and large. Okay.
S2: Can I, can I have a quick technical sort of
S2: breakout here? Um, the, the reason that mythos advanced, in
S2: my opinion, based on being, uh, somewhat of an AI
S2: expert compared to a true like AI researcher, I'm basically
S2: a four year old, but, um, My understanding of this, okay,
S2: is that mythos and all the good models, the way
S2: the good models are getting better is because they're better
S2: at pursuing goals long term. There is no difference between
S2: cyber and building a wet lab when you generalize. This
S2: is all the same shit. And the the speed in
S2: which these are improving. Go ahead.
S3: But there's like other parts of this, like, of course,
S3: I am not a biology expert, but this is one
S3: of the problems is that like, neither are any of
S3: these AI experts either. And so what happens is, is
S3: that like, and I can give a really great example
S3: of this. When I was at the University of Hong Kong,
S3: one of the professors there happens to be one of
S3: the leading experts in nuclear non-proliferation. Uh, he was cited
S3: in the I c, J blah, blah, blah. Okay. And
S3: I remember I tried to get him involved in AI
S3: safety stuff because of course, it has a lot of
S3: parallels to nuclear non-proliferation. And he said that when he
S3: tried to discuss it with his AI, you know, people
S3: from the AI part of HKU, basically they said, you
S3: don't know what you're talking about on nuclear non-proliferation. So
S3: he said, okay, well, I'm now. Right. And so what
S3: happens is, is that like the AI safety people can
S3: genuinely convince themselves, they have logic out all of the
S3: way in which these wet labs might be created and stuff.
S3: But like AI experts are not an expert in wet
S3: lab creation and like biological weapons and things of that nature.
S3: We haven't had some sort of discourse on this. And
S3: then we also don't know what happens as a result.
S3: Like what if AI also produces some sort of like
S3: biology protection mechanism, right? Like we don't know. And so
S3: it's like, these are risks that I think are just like,
S3: very like, uh, they, they're kind of like, we don't
S3: really know that's true. And yet we're being asked to
S3: accept things now. And I think so we have to
S3: kind of ramp up. But like, I just want to
S3: say it from this angle, if we want to believe that,
S3: like if Dario really believes this, there's also this piece
S3: of advice I have, which is to say, currently AI
S3: safety ism is losing the political, um, war, right? Like
S3: I think that the position that I'm expressing in one
S3: reason that some people like yourself might get frustrated is
S3: because like, my argumentation is actually quite popular and it's
S3: quite easy for like people to accept and latch on to,
S3: whereas to accept the safety as an argument, you have
S3: to like kind of go a lot deeper and believe
S3: a bunch of things. If you believe the safety ism position,
S3: you need to switch the political framing of it to
S3: give people something positive other than just, we'll protect you
S3: from the risk that, you know, you don't believe in
S3: in the first place.
S2: I hear you.
S3: Right now, they're not doing that right now. They're basically
S3: saying you give up everything for your own safety so
S3: that we can control you. No one's gonna accept that.
S3: It's just not going to work. So, like, their plan
S3: doesn't work.
S2: Yeah. So so the thing is, Zach. Yes, your argument
S2: is very popular. Your argument is basically you see these
S2: guys like they're, oh, man, they're just making all this
S2: stuff and they're trying to control us. And, um, they've
S2: got all the best stuff and we should have all
S2: the best stuff too. And Dario was like, guys, this
S2: is fucking bad. This is really fucking bad. We have
S2: to be very careful here. Let's get the government involved.
S2: Let's get everyone involved. We need to regulate this. This
S2: is like I'm still going to do it. Like I
S2: still have a company and I still think I have
S2: a good path going forward. But we need to be
S2: super careful about this. They were the first people to
S2: release their own dirty laundry. The guy has consistently been
S2: releasing all, all the sort of flaws in their shit.
S2: And describing this consistently put his company on the line
S2: multiple times, um, financially to, to stand up for, for
S2: ethical red lines and ultimately, like the stuff is going
S2: to happen and he is not trying to say is
S2: so this, I disagree with you. He's trying to produce
S2: some negative thing in the future. What he's trying to
S2: do is saying this is bad, it's happening. Everyone's going there.
S2: It's going to happen no matter what. Let's be careful
S2: and release slowly and make sure that the US wins
S2: before CCP. That's the argument.
S3: But also, when we say the US wins, like let's
S3: keep in mind this is not the, you know, unlike
S3: the CCP, which is like, okay, obviously can control any
S3: Chinese company. Like there were people were already suggesting like,
S3: what if anthropic just moved to Europe? Right? It's not like,
S3: you know, also at another example, it's like when the
S3: Department of War said, we want to use your models
S3: in this way. Dario kind of refused. So it's not
S3: like it's controlled by the US government to the same degree.
S3: So saying America wins is a bit unfair because like,
S3: it's not an American model. It's just a model made
S3: by an American company, which of course is like a
S3: fairly floating concept. I mean, you could, you know, but.
S2: Let's make an American consortium to try to do this.
S2: So we win before China.
S3: Yes. But he's not giving. So this is the issue.
S3: He's not giving anything. He's asking us to believe in
S3: all those risks, which I don't believe in. And he's
S3: asking us to believe in that and do things that
S3: will hurt us, like all of the people who are
S3: not him. Right. And his group. Uh, he's not giving
S3: us anything. And so you sit there going, ah, I'd
S3: rather the Chinese model than believing that this risk, you know,
S3: this is how most. I mean, isn't this like, I've
S3: not watched this Star Wars thing, but isn't this the
S3: whole like the Chancellor took over to protect the people
S3: from some made up risk or some shit like this
S3: happens all the time. Like Napoleon did it. Exactly right.
S3: That's a common story, Zach.
S2: The the emperor was an evil person. Like just straight
S2: up pure evil. Like wanted to do evil things for
S2: evil people.
S3: You met Sam Altman, like, I mean, come on. Like,
S3: people talk about this all the time.
S2: That's a different talk show. Sam Altman is not Dario.
S2: They are the opposite.
S3: I to some degree. Yeah, sure. I concede that. Of
S3: course I believe that there's a difference there.
S2: Um, so so here's, here's my thing, Zach. I think
S2: the situation is I believe Dario is a moral actor.
S2: I don't think it's perfect. Like I'm super pissed about
S2: my my prices. I wish I had more stuff in
S2: my subscription. Like I've been pissed off at anthropic plenty
S2: of times. The issue is, you're not explaining to me
S2: how a moral actor which you agree that that Dario is,
S2: he's behaving as a moral actor. You think he's ultimately
S2: going to lead somehow to an immoral thing? I'm saying
S2: I can't see any difference between how I would behave
S2: and how Dario would behave, given my model of the world.
S3: Yeah, but I would hope that if you were in
S3: Dario's position and actually sitting there, you would turn around
S3: and go, oh, God, and believe the things Dario believes.
S3: You would go, oh, God. Actually, probably it shouldn't be
S3: Daniel deciding it should be the government. Okay. But importantly,
S3: which he referred to as communism. Uh, but importantly, I
S3: don't believe any of these things. And so therefore, I
S3: don't believe in a world where Americans are made worse
S3: off or not just Americans like the whole, you know,
S3: I live in Norway. I don't want my Norwegian family
S3: made worse off. I don't like I don't want anyone
S3: made worse off to protect from a risk that I
S3: have not seen playing out or seen proven in any
S3: meaningful sense. Well, I do not think we should be
S3: willing to accept that trade off. And if that is
S3: the trade off that we're being asked to accept, how
S3: about if Dario believes this is the end of the world?
S3: He makes some concessions on his side to give us
S3: things back, as opposed to just saying we'll block the
S3: model that would do, you know, give you cheaper prices
S3: and things of that nature and not just prices, but
S3: also better capabilities. We will block that one and you
S3: get nothing. I mean, he's got to come if he
S3: wants to save the world. That's a trade he's got
S3: to be willing to make, surely. But he's not doing it.
S2: Dude, it's only been a few months. What are you
S2: talking about? We're talking. We're talking about. Okay, potentially. Okay.
S2: This is we just disagree on this. I just think
S2: you're wrong about the actual threats to the world. But
S2: I'm going to ask you to empathize and just accept
S2: it for the sake of argument, right? For the sake
S2: of argument, if we are correct, me and Dario and
S2: these other AI experts or whatever. If we are right
S2: about that, then he is behaving as a as a
S2: good actor.
S3: No, you should if you. That's my point is that
S3: if he if I was sitting in his shoes and
S3: I believed that thing.
S2: What.
S3: Would you do? He. I would basically say this needs
S3: to be nationalized because this is not something that should
S3: be privately owned.
S2: Right. But nationalization doesn't. Nationalization doesn't solve anything. It doesn't
S2: solve anything.
S3: That's. But that's the point. That's not for us to decide.
S3: That's that's that's my point is that that's the thing.
S2: It is up to us to decide.
S3: If not us in the sense of Dario. I mean,
S3: it's up for us in the sense of like American
S3: people who can vote and stuff like that. But it's
S3: not us up to Dario to be making that call.
S3: And so.
S2: You have.
S3: To.
S2: You have to look at the actual thing that would
S2: happen if he did that, if he walked off the.
S3: Field. That's not up to him. That's not that's my
S3: point is that at the end of the day, there's
S3: a public system that needs to be played. He is making.
S3: That's my point is he is he is acting like
S3: somebody picked him to make that call and no one did. Okay.
S3: If you found out tomorrow that a company in Alabama
S3: was making nuclear weapons in their garage, right? You would
S3: immediately say, go see, like, send the troops in. Cease
S3: that shit, Zach.
S2: All of us are are supposed to be making that call.
S2: Every single one of us starting with just happens to
S2: have a company.
S3: No, but as a it's in a.
S2: Different than me. I would make the same decision. Anyone
S2: in that position should make the same decision.
S3: No, you should make the decision to hand over to
S3: the public system. That's like my position.
S2: And just leave and not participate anymore.
S3: Participate? Sure. But like it's not up to you to
S3: decide it then. So like to basically say this becomes
S3: a thing that is controlled entirely by the government. Okay.
S3: You can still go to work every day, help make
S3: things safer, whatever you want to. Do.
S2: You think if he thought things were this bad, he
S2: should completely give it to the government and just hand
S2: over full control?
S3: The same we would accept. But this is the same
S3: thing we would accept if someone was making nuclear weapons,
S3: which is what he compares it to. If you found
S3: out that a company in Alabama was making nuclear weapons
S3: in their garage, you would say, this is unacceptable and
S3: we should go take it. Of course, like you would
S3: believe that. I don't think anyone would think.
S2: No, but but but tactically but tactically the way it
S2: would work.
S3: That's the problem. The tactics are not up for us. Right.
S3: That's that's that's something that should be handed to a public.
S2: But that nationalization in that model, why wouldn't it be? Okay. Okay.
S2: So let's say I'm in charge of the government now.
S2: I'm in charge of the nationalization program. My first step.
S2: Get all these AI companies in here. All the US
S2: American AI companies. Get them all in here. This is
S2: now nationalized. We're using these resources of these companies to
S2: move forward.
S3: Yeah. That's of course what you would. That's how nationalization
S3: would work of course. But it's.
S2: Very. How different is that from what he is proposing?
S2: I'm arguing it's private.
S3: Ownership that he's going to IPO. Like who cares about ownership?
S2: Who cares? Who cares about shares and ownership debate?
S3: That's the whole debate. It's like, should you be able
S3: to build nuclear weapons and have private ownership?
S2: That's not the debate. The debate is whether or not
S2: the government is involved in controlling.
S3: I don't want the government, and I don't want the
S3: FDA for nuclear weapons. Right. Like, it's like it's a silly.
S2: You're asking for complete nationalization.
S3: No I'm not. I'm asking for competition. It's just if
S3: you believe the crazy things Dario believes, I would be
S3: asking for nationalization. I believe that people should be that
S3: all of these models need to compete so that we
S3: don't run into a world where all of. I think
S3: the biggest risk for AI is that it gets controlled
S3: by a few small companies, and that's why I want
S3: everyone to be competing openly. I want us to make
S3: it so that distillation is like a right. You get
S3: to distill models, okay? Because I want there to be
S3: a ton of labs all competing in parallel. And that
S3: is because I don't believe in like the existential risk here. Okay.
S3: I believe the biggest risk is Dario's company becoming king, right?
S3: That's horrible for society. It would be horrible to have
S3: $150 trillion company and everyone else be unemployed. That's what
S3: I don't want.
S2: If that were he were trying to do, he would
S2: not have to.
S3: It's just that could happen.
S2: No, but I'm saying he's explicitly like doing the opposite.
S3: No, I mean, his company's gone from whatever to one
S3: point whatever trillion. I mean.
S2: No, but he's the one that started glasswing. He's the
S2: one that did all of this.
S3: But yeah, but glass, this was the other point I
S3: forgot to make, which is like Glasswing didn't even have
S3: any app set companies in it. Why didn't they give
S3: it to them is because they're going to try to
S3: compete with apps. So obviously that was like there were
S3: aspects of Glasswing that were not purely altruistic, right? Like,
S3: why would you have not given it to an application
S3: security company that has more access to code than they do?
S3: I mean, just because they don't want to compete.
S2: They gave it to the people with the actual infrastructure
S2: that they were worried about.
S3: There are plenty, like there are plenty of application security
S3: companies that would have the ability to use mythos to
S3: make a large lot of code safer. Right. You could
S3: have done that. They chose not to.
S2: Yeah. I mean, that's an implementation thing, right? But I mean.
S3: But it's an implement. It's kind of the point here
S3: in the comments that Glasswing is like becomes when it's
S3: an implementation that doesn't reflect the full details of safety ism,
S3: you start to go like, it's not really that wasn't
S3: Glasswing was not really a safety thing. Okay. Glasswing large.
S2: How could it be anything? Okay. Okay. If someone who
S2: believes they're behaving morally comes up and says, hey, I
S2: don't want to release this immediately. I need to give
S2: it to people who have infrastructure so they can defend. Yes. Like,
S2: how is that not a moral action?
S3: That part's normal. That's normal. Okay. And then what happens?
S3: They go, I need to give it to these people. Um,
S3: can you come up with a list? And then the
S3: marketing department comes up with a list and they say, oh,
S3: we should probably not include the appsec people, because.
S2: Now you're back to assuming the worst of Sam or
S2: of Dario.
S3: No, no, no, Dario is not even in the room
S3: for that discussion. Right. It's like that's what happens because
S3: it's consistent with safety ism. He's not released it to
S3: the world. So Dario's not going to have a problem
S3: with this, right? But like, so that's the one thing
S3: to understand is that I argue that Dario genuinely believes
S3: in safety, blah, blah, blah, but it doesn't mean that
S3: every action Dario takes is only about safety. It just
S3: means it's consistent with safety. So Glasswing is consistent with
S3: safety because he didn't release it to the world. Right.
S3: But it is not only about safety, okay? There was
S3: choices made in there that were not about safety, the
S3: choice to use fire.
S2: Well, he's also running. He's also running a company. He's
S2: also trying to make money. Sure, sure, sure. Who else
S2: would be in the position to do this other than
S2: someone who has the ability to start a company and
S2: run one?
S3: Say it to damn him. I just said it to say,
S3: you know, when we look at it and say yes,
S3: but he did the right thing with Glasswing like I
S3: think he did like a thing with Glasswing, right? He
S3: did like the 5050 of it. It's like, no, I'm
S3: not going to give him a huge amount of credit
S3: for Glasswing. Right. I'm not going to sit there and
S3: go like, round of applause for giving it to CrowdStrike,
S3: but not to, you.
S2: Know, I'm just saying it's.
S3: Extreme. Give it to you.
S2: Know, I'm just saying it's extremely consistent with the view
S2: that I've presented here.
S3: Of course. And I, I agree, and this is one
S3: of the things is that you can view almost everything
S3: anthropic does is consistent with safety ism, right? Uh, and
S3: I think that, that, that's one reason why I have
S3: come to believe between that and Dario's, the fact that
S3: Dario does much like the Chinese do, writes everything he
S3: thinks and plans down in long winded essays that he
S3: thinks no one will read. And then you read them.
S3: It's like, you know, there are. When I prepared for
S3: the video, I mean, I probably read like 20,000 words
S3: written by the guy, right?
S2: Uh, he's read all that stuff.
S3: He's prolific at writing these things. Okay. Um, yeah. You know,
S3: and this is why I believe that Dario believes all this,
S3: but what Dario, there is a part of AI safety ism,
S3: which is that like, you know, if you go back
S3: to the origins of like, if you think like Nick Bostrom, like, um, superintelligence,
S3: which would be like 2015 book, right. Uh, which would
S3: have been basically around the time when yod would have
S3: been writing all this stuff. You know, basically one of
S3: the positions that's argued is like that AI is aligned,
S3: fully aligned with our goals. But, uh, given that it's aligned,
S3: whoever controls it has like immense power, right? And this
S3: is kind of your concern of like, what if the
S3: CCP had it right? That'd be like really bad, right?
S3: If like, if like the Chinese Communist Party has the ability,
S3: has this recursive self-improving thing, and then you get there first.
S3: And Dario believes in this world where like whoever, if,
S3: if we can control it, which I think he thinks
S3: is a little bit unclear, but as he assumes is like,
S3: let's just assume it's true because there's some probability where
S3: it's true. Um, you know, there was this whole wing
S3: of safety ism, which is like, but then what if
S3: the company who's in charge of it decides to do
S3: something evil with it? Right? Okay. And, and like, that's
S3: a part of this, which is like, seriously, like, if
S3: you believe in this recursive self improvement plan where like
S3: suddenly whoever gets the best models will be permanently ahead.
S3: You have to be concerned about what the company who
S3: controls it would do with it, right?
S2: So here's my argument to you. So if you've read
S2: all his essays and stuff like that, what's clear to
S2: me from his essays and you can't always know the
S2: mind of someone, right? My, in my view, what Dario
S2: would want the absolute most, okay, is we get to
S2: the other side of this risk situation. Either it burns down,
S2: we build it back up, or we mitigate it. Somehow.
S2: We get to the other side. It. We have a
S2: we have ACI. We're now building up humanity like all
S2: the good things are happening or whatever. There's still companies.
S2: He's still one of the companies. He's making tons of money.
S2: There's tons of open source models and they're doing great
S2: to everyone has tons of AI and he's still making
S2: money and the company is great and thriving. That is
S2: from his essays, the thing he's trying to build.
S3: I have never once heard him say that he wants
S3: there to be open source models. If there's a quote
S3: on that, like send it to me because I haven't
S3: heard him say it.
S2: Yeah. I mean, there's no indication to me that he
S2: doesn't want open source models other than he doesn't want
S2: the most dangerous stuff to go into open source models immediately,
S2: because that would be bad for the reasons we talked about.
S3: No, because he's talked to pretty extensively about how he
S3: believes one of the problems with open weight models is
S3: that you can remove any guardrails that might exist. So
S3: I guess like so from that perspective, it'd be hard
S3: for him to argue for open weight models because then
S3: you could just remove the guardrails.
S2: Right? Well, I mean, that's saying the same thing I'm saying,
S2: assuming that that we're something we could potentially solve. He's not.
S2: In other words, he's not trying to limit the intelligence
S2: that people have access to other than for the reason
S2: of this risk thing. So he's not trying to get
S2: he's not trying to go to a place. He's not
S2: trying to go to a place where him and his
S2: rich buddies control intelligence and people have to go through him.
S2: That's not what he's trying to do.
S3: No, I think he is. I think he's doing it
S3: for safety reasons, though, like he very much would love
S3: if I think if you told Dario tomorrow that like
S3: all AI goes through anthropic, right? Um, he would say
S3: that's great because it allows him to ensure that everyone
S3: is safe.
S2: No, but what I'm saying.
S3: Is surely you agree he would say that that that
S3: would be good because it would allow him to keep
S3: everyone safe.
S2: Yeah, I don't know. I think he would probably like
S2: other people trying the same thing at the same time. Yeah,
S2: I agree.
S3: They would probably. Yes.
S2: Yeah. Yeah. Agreed. Agreed. So what what I would challenge
S2: you on is I think um, and maybe we could
S2: find quotes. Maybe we could use AI to do this. Um,
S2: I think you're wrong about the, the final thing he's
S2: trying to get to, I think he wants there to
S2: be open models, lots of competition. He's not the only company.
S2: Everyone's safe, everyone's fine. And everyone has access to tons
S2: of intelligence, one of which is him.
S3: Yeah, but I'm not gonna. I'm not going to let
S3: us play out this world where, like, we have to
S3: first hand over all this power to these guys, and
S3: then one day they will cede it back and make
S3: sure that everyone lives in harmony. Like, I think it's
S3: just the best course of action. And I think, sorry,
S3: I probably have to wrap up, but like, and I'll
S3: let you say something after this. But the best course
S3: of action is fundamentally, everyone's competing for building the best
S3: AI models so that that way, like we don't end
S3: up in a world where it's controlled by one or
S3: 2 or 3 people that maybe we shouldn't have in control, right?
S3: Let them all compete, let everyone have access. And then like,
S3: if it turns out that there's some sort of like,
S3: if when we get closer to some of these really
S3: tangible risks, like, okay, like let's have a discussion maybe,
S3: but like right now, I'm not, I'm not willing to
S3: cave on any of this until I, until there's like
S3: some sort of reason to, why would we hand over
S3: to the, you know, Napoleon, just so that about some fear,
S3: we don't know if it's true.
S2: We are extremely close. It is happening. Look, Dario just
S2: got surprised by this. Anthropic just got surprised by this.
S2: This is happening a lot faster than I think. You
S2: think it's happening. Which would explain his behavior. So, um. Yeah,
S2: I would just say, uh, great conversation. Uh, I'm glad
S2: we did it. And, uh, I think this really does
S2: illuminate the differences, which, like I said before, we're not
S2: disagreeing on what we're trying to achieve. I think it's
S2: just this, uh, this middle path that we're disagreeing on.
S3: Yeah. No, I think this was good. Um, I think obviously,
S3: I think anyone listening probably now can form. I think
S3: there's very little left to ambiguity. Like people can now
S3: decide which, which way they think.
S2: Well, that's a success.
S3: Yeah, exactly. So thanks for doing this. Um, yeah, totally.
S3: I will then just stop the stream. I guess I
S3: will leave, you'll be on for one second. So okay,
S3: for everyone listening, see you guys. Uh, I'll probably be
S3: on Twitter, so I'll probably post some stuff. So, uh.
S3: See y'all.