Speaker 1: Pushkin.
Speaker 2: Hey everybody, Nate here, jumping in before the short so
Speaker 2: that we've been having a lot of fun answering your
Speaker 2: listener questions. So far, we've covered things like Pascal's Wager,
Speaker 2: the hot hand fallacy, or the fallacy of the hot
Speaker 2: hand fallacy actually, and the expected value of learning new languages.
Speaker 2: We want to keep doing this kind of thing, so
Speaker 2: send us all your questions about risk decision making, game theory, poker,
Speaker 2: you name it. Reach out to us on social media
Speaker 2: or email at Risky Business at pushkin dot fm. Even
Speaker 2: if you're not a premium subscriber, this is a great
Speaker 2: way to support the show, so you can keep sharing
Speaker 2: it free of charge. We look forward to hearing from you.
Speaker 1: Welcome back to Risky Business, a show about making better decisions.
Speaker 1: I'm Maria Kanakova.
Speaker 2: And I'm Nate Silver.
Speaker 1: Today on the show is going to be a little
Speaker 1: bit doomtastic.
Speaker 2: Yeah, I mean, I don't know if it's like worse
Speaker 2: than thinking about like the global economy going into a
Speaker 2: recession because of dumbfuck tariff policies. This is all about
Speaker 2: how we're all going to die in seven years instead. No,
Speaker 2: I'm just kidding. This is a very very intelligent and
Speaker 2: well written and thoughtful report called AI twenty twenty seven
Speaker 2: that we're going to spend the whole show on because
Speaker 2: think it's such an interesting to talk about, but that
Speaker 2: you know, includes some dystopian possibilities.
Speaker 1: I would say it does indeed, so let's get into
Speaker 1: it and hope that you guys are all still here
Speaker 1: to listen to us in seven years.
Speaker 2: The contrast is interesting between like all the chaos we're
Speaker 2: seeing with tariff policy in terms of starting a trade
Speaker 2: war with China and then other types of chaos. It's
Speaker 2: interesting to kind of look at this. I mean, I
Speaker 2: wouldn't call it a more optimistic future exactly, but like
Speaker 2: but like on a different trajectory of like a future
Speaker 2: that's going to change very fast, according to these authors,
Speaker 2: with profound implications for you know, everything, the human species.
Speaker 2: These researchers and authors are saying that everything is going
Speaker 2: to change profoundly.
Speaker 1: And.
Speaker 2: Even though there is some hedging hair, this is kind
Speaker 2: of their base case scenario, and like you know, base
Speaker 2: case number one and base case number two differ. There's
Speaker 2: like kind of a choose your own adventure at some
Speaker 2: point in this report. But they're both very different than
Speaker 2: the status quo, right, and the notionately hear from that,
Speaker 2: like everything becomes different. If ais become substantially more intelligent
Speaker 2: than human beings, people can debate and we will debate
Speaker 2: on this program what that means. But yeah, do you
Speaker 2: want to do you want to contextualize this more? Do
Speaker 2: you want to tell people to actors are of this report?
Speaker 1: Absolutely? Absolutely? So the report is authored officially by five people,
Speaker 1: and I think unofficially there's also a sixth We've got
Speaker 1: Eli Leifelund, who's a super forecaster and he was ranked
Speaker 1: first on RAN's forecasting initiative, so he is someone who
Speaker 1: is very good at kind of looking at the future
Speaker 1: trying to predict what's going to happen. You have Jonas
Speaker 1: Palmer who's a VC at Macroscopic Ventures. Thomas Larson was
Speaker 1: a former executive director of the Center for AI Policy,
Speaker 1: so that is a center that advises both sides of
Speaker 1: the aisle on you know, how how AI is going
Speaker 1: to go? And Romeo Dean, who is part of Harvard's
Speaker 1: AI Safety Student Team, so someone who is still a student,
Speaker 1: still learning, but kind of the next generation of people
Speaker 1: looking at AI. And finally, we have Daniel coch Taylor,
Speaker 1: who basically had written a report back in twenty twenty one.
Speaker 1: He was a AI researcher and he looked at predictions
Speaker 1: for AI for twenty twenty six, and it turns out
Speaker 1: that his predictions were pretty spot on, and so open
Speaker 1: AI actually hired Daniel as a result of this report.
Speaker 2: Obviously he now left, yes.
Speaker 1: And then and then he left exactly.
Speaker 2: And importantly, so there's also Scott Alexander exactly.
Speaker 1: So I was saying, he's the he's the person who
Speaker 1: kind of is in the background, and he's you guys
Speaker 1: might know him as the author behind astral Codux.
Speaker 2: And I know Scott. He's one of the kind of
Speaker 2: fathers of what you might call rationalism. I think Scott,
Speaker 2: when I interview him from a book, was happy enough
Speaker 2: with that term and accused me or co opted me
Speaker 2: into also being a rationalist. These people are somewhat adjacent
Speaker 2: to the effective altruist, but not quite right. They're just
Speaker 2: trying to apply a sort of thoughtful, rigorous, quantitative lens
Speaker 2: to big picture problem, including existential risk, of which most
Speaker 2: people in this community believe that AI is both an
Speaker 2: existential risk and also kind of an existential opportunity, right,
Speaker 2: that it could transform things. You talk to Sam Alt
Speaker 2: and he'll say We're going to cure cancer and illuminate
Speaker 2: poverty and whatever else. Right, And Scott's also an excellent writer.
Speaker 2: And so let me disclose something which is slightly important here.
Speaker 2: So I actually was approached by some of the authors
Speaker 2: of this report a couple of months ago, I guess
Speaker 2: it was in February ish, just to give feedback and
Speaker 2: chat with them. So I'm working off the draft version, right,
Speaker 2: which I do not believe they changed very much. So
Speaker 2: my notes pertained to an earlier draft. I did not
Speaker 2: have time this morning to go back and re.
Speaker 1: Read it, so I was not on the inside loops.
Speaker 1: So I did not get an earlier draft. And I've
Speaker 1: read this draft and basically just to kind of big
Speaker 1: picture sum it up, it outlines two scenarios, right, two
Speaker 1: major scenarios for how AI might change the world as
Speaker 1: soon as twenty thirty. Now important note like that that
Speaker 1: date is kind of hedged. It might be sooner, it
Speaker 1: might be later. There's kind of a there's a confidence
Speaker 1: interval there. But the two different scenarios. One basically we
Speaker 1: do twenty thirty, humanity disappears and is taken over by AI.
Speaker 1: The positive report is in twenty thirty. Basically we get
Speaker 1: AIS that are aligned to our interests, and we get
Speaker 1: kind of this AI utopia where AI is actually help
Speaker 1: make life much better for everyone and make the standard
Speaker 1: of living much higher. But the crucial turning point is
Speaker 1: before twenty thirty, and the crucial kind of question at
Speaker 1: the center of this is will we be able to
Speaker 1: design AIS that are truly aligned to human interests rather
Speaker 1: than just appear to be aligned and kind of lying
Speaker 1: to us while actually following their own agenda. And how
Speaker 1: we handle that is kind of the lynchpin. And it's
Speaker 1: actually interesting, Nate, that you started out with China because
Speaker 1: a lot of the policy choices and a lot of
Speaker 1: what they see as kind of the decision points that
Speaker 1: will affect the future of humanity actually hinge on the
Speaker 1: US China dynamic, how they compete with each other, and
Speaker 1: how that sometimes might basically clash against safety concerns because
Speaker 1: no one wants to be left behind. Can we manage
Speaker 1: that effectively and can kind of that transition work in
Speaker 1: our favor as opposed to against us. I think that
Speaker 1: this is kind of one of the big questions here,
Speaker 1: and so it's funny that we're seeing all of this
Speaker 1: trade war right now as the sideport is coming out.
Speaker 2: Yeah, look, I think this exercise is partly just a
Speaker 2: forecasting exercise, right, I mean, obviously it's just kind of
Speaker 2: like fork at the bottom where we learn to have
Speaker 2: an AI slow down or we kind of are pressing
Speaker 2: fully on the accelerator, right, Like, in some ways, scenarios
Speaker 2: are like not that different, right, Either one assumes remarkable
Speaker 2: rates of technological growth that I think even AI I'm
Speaker 2: never quite sure who to call an optimist or a pessimist, right,
Speaker 2: even AI believers you know, might think is a little
Speaker 2: bit aggressive. Right. But what they want to do is
Speaker 2: they want to have like a specific, fleshed out scenario
Speaker 2: for how the world would look. Like it's kind of
Speaker 2: like a modal scenario. And like I think they'd say that, like,
Speaker 2: we're not totally sure about either of these necessarily, right,
Speaker 2: And I don't think they'd be as like pedantic as
Speaker 2: to say, if you do X, Y, and Z, then
Speaker 2: we'll save the world and have utopia, and if you don't,
Speaker 2: then we'll all die. Right. I think they probably say
Speaker 2: it's unclear and there's kind of like risk either way.
Speaker 2: We wanted to go through the scenario like fleshing out
Speaker 2: like what the world might look like. Right. I do
Speaker 2: think one thing that's important is that whatever decisions are
Speaker 2: made now could get locked in, right that you pass
Speaker 2: certain points in overturn and it becomes very hard to accelerate,
Speaker 2: like an arms race. This is you know, what we
Speaker 2: found during the Cold War for example. I mean, one
Speaker 2: of the big things I look at is like, do
Speaker 2: we force the AI to be transparent in its thinking
Speaker 2: with humans? Right? Like, now there's been a movement toward
Speaker 2: the AI will actually explicate it's thinking more. I'll ask
Speaker 2: you a query open AI. The Chinese models to this too, right,
Speaker 2: and I'll say, I am thinking about X, Y and Z,
Speaker 2: and I'm looking up PD and Q and now I'm
Speaker 2: reconsidering this. It actually has this chain of thought process, right,
Speaker 2: which is explicated in English. You know. One concern is
Speaker 2: that what if the AI just kind of communicates to
Speaker 2: another in these implicit vectors that's inferring from all the
Speaker 2: texts it has. It's kind of unintelligible to human beings, right,
Speaker 2: and maybe kind of quote unquote thinking in that way
Speaker 2: in the first place, and then does us the favor
Speaker 2: of like translating back to it goes from to kind
Speaker 2: of this big bag of numbers is one day I
Speaker 2: researcher called it, right, and then it translates back into
Speaker 2: English or whatever language you want. Really, in the end,
Speaker 2: what if it just skip cuts out that last step right,
Speaker 2: then we can't kind of like check what AI is doing.
Speaker 2: Then it can behave deceptively more easily. So you know,
Speaker 2: so that part seems to be important. I want to
Speaker 2: hear your your first impressions before I kind of poison
Speaker 2: the well too much.
Speaker 1: Well, my first impressions is that the alignment problem is
Speaker 1: a very real one and an incredibly important one to solve.
Speaker 1: And what I got from this is that actually, the
Speaker 1: problem that I've had with like these initial AI lms
Speaker 1: is the kernel of what they're seeing there. Right. So
Speaker 1: you and I have talked about this on the show
Speaker 1: in the past, and I've said, well, my problem is
Speaker 1: that when I'm a domain expert, right, I start seeing
Speaker 1: some inaccuracies, and I start seeing like places where like
Speaker 1: it either just didn't do well or made shit up
Speaker 1: or or whatever it is. Now I think it's very
Speaker 1: clear that those problems are going to go away, right,
Speaker 1: that that is going to get much much better. However,
Speaker 1: the kernel of it's showing me something, but that might
Speaker 1: just be you know, I have no way of verifying
Speaker 1: if that's what's going on, what it's reading, like, how
Speaker 1: it's I don't want to say thinking about it, even
Speaker 1: though in the report they do use thinking, but.
Speaker 3: It's normalizing too much, I think is correct.
Speaker 2: I think it's.
Speaker 1: Okay, we'll stick to that language. Yeah, okay, So, so
Speaker 1: how it's thinking about it that those little problems and
Speaker 1: like the little the glitches and the things that it
Speaker 1: might be doing where it starts actually glitching on purpose,
Speaker 1: are not going to be visible to the human eye.
Speaker 1: And so one of the main things that they say
Speaker 1: here is that as AI internal R and D gets
Speaker 1: rapidly faster, so that means basically AI's researching AI, right,
Speaker 1: and so internally they start developing new models, and as
Speaker 1: they kind of surpass human ability to monitor it, it
Speaker 1: becomes progressively more difficult to figure out. Okay, is the
Speaker 1: AI actually doing what I want it to do? Is
Speaker 1: the output that it's giving me its actual thought process?
Speaker 1: And is it accurate or is it like trying to
Speaker 1: deceive me? But it's actually kind of inserting certain things
Speaker 1: on purpose because it has different goals, right, because it
Speaker 1: is actually secretly misaligned, but it's very good at persuading
Speaker 1: me that it's aligned. Because one of the things that
Speaker 1: actually came out of this report, and I was like, huh,
Speaker 1: you know this is interesting is if we get this
Speaker 1: remarkable improvement in AI, it will also remarkably improve at
Speaker 1: persuading us right as part of it. Making Yeah, so
Speaker 1: this is but I've never even thought about that. I
Speaker 1: was like, okay, fine, But one of the things that
Speaker 1: I do buy is that it's going to be very
Speaker 1: difficult for us to monitor it and to figure out
Speaker 1: like is it truly aligned with human wants, with human desires,
Speaker 1: with human goals, And the experts who are capable of
Speaker 1: doing that, I think are actually going to dwindle right
Speaker 1: as AI starts proliferating in society, And so to me,
Speaker 1: that is something that is actually quite worrisome and that
Speaker 1: is something that we really need to be paying attention to.
Speaker 2: Now.
Speaker 1: Just to fast forward a little bit, in their doomsday
Speaker 1: scenario in twenty thirty one, AI takes over it basically
Speaker 1: like suddenly releases some chemical agents, right, and humanity ties
Speaker 1: and the rest of the stragglers are taken care of
Speaker 1: by drones, et cetera.
Speaker 2: I don't even like it. It doesn't even quick and
Speaker 2: painless death. I will say, let's hope we.
Speaker 1: Don't know what the chemical agents are. Might not be
Speaker 1: quick and painless. Some chemical agents are actually a very
Speaker 1: painful death thing. So let's hope. Let's hope it's quick
Speaker 1: and painless. Quickly, quick, yes, stuff quick, Okay, hopefully some
Speaker 1: chemical agents are not quick. I hope it's quick and painless.
Speaker 1: But if they're actually capable of deception at that high level,
Speaker 1: then you technically don't even need them to do it.
Speaker 1: If we're trusting medicine and all sorts of things to
Speaker 1: the AIS, it's pretty easy for it to actually manipulate
Speaker 1: something and actually insert something into codes et cetera that
Speaker 1: will fuck up humanity in a way that we can't
Speaker 1: that we can actually figure out at the moment. Right,
Speaker 1: Like the way I think of it, and this is
Speaker 1: not from the this is not from the paper, but
Speaker 1: this is just the way that my mind processed. It
Speaker 1: is like think about DNA, right, like you have these
Speaker 1: remarkably complex, huge strands of data, and as we've found out,
Speaker 1: but it's taken forever, one tiny mutation can actually be fatal, right,
Speaker 1: but you can't spot that mutation. Sometimes that mutation isn't
Speaker 1: fatal immediately, but we'll only manifest at a certain point
Speaker 1: in time. That's the way that my mind tried to
Speaker 1: kind of try to conceptualize what this actually means. And
Speaker 1: so I think that you know, that would be easy
Speaker 1: for a deceptive AI to do, and to me like that.
Speaker 1: That's kind of the big takeaway from this report is
Speaker 1: that we need to make sure that we are building
Speaker 1: AIS that will not deceive right that they're capabilities, they
Speaker 1: explain them in an honest way, and that honesty and
Speaker 1: trust is actually prioritized over other things, even though it
Speaker 1: might slow down research, it might slow down other things,
Speaker 1: but that that kind of alignment step is absolutely crucial
Speaker 1: at the beginning because otherwise humans are human, right, they're
Speaker 1: easily manipulated. And we often trust that computers are quote
Speaker 1: unquote rational because they're computers, but they're not. They have
Speaker 1: their own inputs, they have their own weights, they have
Speaker 1: their own values, and that could just lead us down
Speaker 1: a dark path.
Speaker 2: Yeah, so let me follow up with this pushback. I
Speaker 2: guess right, Like, first of all, I don't know that
Speaker 2: human so easily persuaded. This is my big critique with
Speaker 2: like all the misinformation people who say, well, misinformation is
Speaker 2: the biggest problem on society basis. It's like people are
Speaker 2: actually pretty stubborn and they're kind of sound pretentious. They're
Speaker 2: kind of basian and how they formulate their beliefs, right,
Speaker 2: they have some notion of reality. They're looking at the
Speaker 2: credibility of the person who is telling them these remarks.
Speaker 2: If it's an unpersuasive source, it might make them less
Speaker 2: likely to believe they're balancing with other information with their
Speaker 2: lived experience so called. Right. You know, part of the
Speaker 2: reason that like I am skeptical of AIS being super
Speaker 2: persuasive is like you know that it's an AI. You know,
Speaker 2: it's trying to persuade you, you know what I mean. So, like,
Speaker 2: if you go and play poker against like a really
Speaker 2: chatty player I Phil hemm With or Scott Seeber or
Speaker 2: someone like that, right, you know, on some level of
Speaker 2: the best play is just to totally ignore it. Right,
Speaker 2: You know that they are trying to sweet talk you
Speaker 2: into doing exactly what they want you to do. And
Speaker 2: so the best play is to disengage, or literally you
Speaker 2: can randomize your movis VI have some notion of what
Speaker 2: the game, theoretical optimal play might be, right, or salesmen
Speaker 2: or politicians have reputations for being Oh he's a little
Speaker 2: too smooth, Gavin Newsome a little too fucking smooth, right,
Speaker 2: I don't find Gavin news some persuasive at all, Right,
Speaker 2: use a little too from the hair gel to the
Speaker 2: constantly shifting vibes. I mean, I don't really find Gavin
Speaker 2: new Some persuasive at all, even though like an aim,
Speaker 2: I'd say, boy, Gavin new some good looking guys look
Speaker 2: gravelly throated, But you know whatever, I mean. Look the
Speaker 2: big critique I have of this project, and by the way,
Speaker 2: I think this is an amazing project in addition to
Speaker 2: like wonderful writing if you viewed on the web not
Speaker 2: your phone, so these very cool like little infographic that
Speaker 2: updates everything from like the market value. If they don't
Speaker 2: call it open AI, they call it open brain, I
Speaker 2: guess is what they settle on for a substitui.
Speaker 1: Yeah, they call everything something else just to make sure
Speaker 1: that they're not stepping on any toes. So they have
Speaker 1: open brain and and they have deep scent from China.
Speaker 2: I wonder which one that could be. I wonder. But
Speaker 2: it's beautifully presented and written, and like I appreciate they're
Speaker 2: going out on a limb here, you know, I mean,
Speaker 2: I think they have. It's been fairly well received. They've
Speaker 2: gotten some push back both from inside and I think
Speaker 2: outside the AI safety community. Right, but they're putting their
Speaker 2: necks in the line, hear. They will look if things
Speaker 2: look pretty normal, if or it looks pretty normal in
Speaker 2: twenty thirty two or whatever, right, then they will look
Speaker 2: dumb for having published this.
Speaker 1: Well, and they actually have that right as some scenario
Speaker 1: that you end up looking stupid if everything goes well.
Speaker 2: But that's okay.
Speaker 1: Now, can I push back on the persuasion thing a
Speaker 1: little bit, just on two things. So, first of all,
Speaker 1: the poker example is not actually a particularly applicable one here,
Speaker 1: because you know that you're playing poker and you know
Speaker 1: that someone is trying to get information and deceive you.
Speaker 1: The tricky thing. So this is kind of when I
Speaker 1: spend time with con artists. The best con artists aren't
Speaker 1: Gavin Newso like, they're not car salesmen. You have no
Speaker 1: idea they're trying to persuade you to do something. They
Speaker 1: are just like nice, affable people who are incredibly charismatic,
Speaker 1: and even in the poker community, by the way, like
Speaker 1: some of the biggest grifters who like it comes out
Speaker 1: later on, we're just like stealing money and doing all
Speaker 1: of these things. Are charming, right, They're not sleazy looking
Speaker 1: like they have no signs of, oh, I'm a salesman.
Speaker 1: I'm trying to sell you something. The people who are
Speaker 1: actually good at persuasion, you do not realize you are
Speaker 1: being persuaded. And I think people are incredibly easy to
Speaker 1: kind of to subtly lead in a certain direction if
Speaker 1: you know how to do it, and I think ais
Speaker 1: could could do that, and they might persuade you when
Speaker 1: you don't even think they're trying to persuade you. You might
Speaker 1: just ask like, can you please summarize this research report
Speaker 1: and the way that it frames it right, The way
Speaker 1: that it summarizes it just subtly changes the way that
Speaker 1: you think of this issue. We see that in psych
Speaker 1: studies all the time, by the way, where you have
Speaker 1: different articles presented in slightly different orders, slightly different ways,
Speaker 1: and people from the same political beliefs, you know, same
Speaker 1: starting point, come away with different impressions of what kind
Speaker 1: of the right course of action is or what this
Speaker 1: is actually trying to tell you. Because the way the
Speaker 1: information is presented actually influences how you think about it.
Speaker 1: It's very very easy to do subtle manipulations like that.
Speaker 1: And if we're relying on AI on a large scale
Speaker 1: for a lot of our lives, I think that if
Speaker 1: it has like a quote unquote master plan, you know,
Speaker 1: the way that they present in this report, then persuasion
Speaker 1: in that sense is actually going to be.
Speaker 2: You'll know you're being manipulated, right, that's the issue.
Speaker 1: No, you don't know. That's the thing.
Speaker 2: People will be manipulated because it's AI and that I
Speaker 2: don't but I don't know.
Speaker 1: I honestly, like Nate, I applaud your belief in human's
Speaker 1: ability to adjust to this, but I don't know that
Speaker 1: they will because I've just seen enough people who are
Speaker 1: incredibly intelligent fall for cons and then be very unpersuadable
Speaker 1: that they have been conned, right, instead doubling down and
Speaker 1: saying no, I have not so humans are stubborn, but
Speaker 1: they're also stubborn and saying I have not been deceived,
Speaker 1: I have not been manipulated, when in fact they have
Speaker 1: to protect their ego and to protect their view of
Speaker 1: themselves as people who are not capable of being manipulated
Speaker 1: or deceived. And I think that that is incredibly powerful.
Speaker 1: And I think that that's going to push against your optimism.
Speaker 1: I hope you're right, but from what I know, I
Speaker 1: don't think you are.
Speaker 2: I'm not quite sure to call it optimism some I
Speaker 2: guess maybe we do likely different views of human nature.
Speaker 2: But like there's not yet a substantial market for like
Speaker 2: AI driven art we're writing, and I'm sure there will
Speaker 2: be one eventually, right, But like people understand that context matters, right,
Speaker 2: that you could heavy I create a rip off the
Speaker 2: Mona Lisa, but you can also buy rip off the
Speaker 2: Mona Lisa and Canal Street for five bucks, right, and
Speaker 2: like it? You know, So it's it's the intentionality of
Speaker 2: the act and the context of the speaker. Now, sound
Speaker 2: like super woke, I guess right where you're coming from.
Speaker 2: I think that actually is how humans communicate. Like art
Speaker 2: that might be pointless dribble coming from somebody can be
Speaker 2: something different coming from a Jackson Bollock or whatever. You know.
Speaker 1: Absolutely, I think that that's a really important point. By
Speaker 1: the way, I think it's a different point, but I
Speaker 1: think that that is a very important point. I think
Speaker 1: context does matter.
Speaker 2: We'll be right back after this message. I was buttering
Speaker 2: up this report before. My big critique of it is like,
Speaker 2: where are the human beings in this? Or put another way,
Speaker 2: kind of like where is the politics? Right? They're trying
Speaker 2: not to use any remotely controversial real names, right, so
Speaker 2: you open brain for example, so where's where is President Trump?
Speaker 2: Let me steal a qick search to make sure their
Speaker 2: named Trump does not appear to know the name.
Speaker 1: So do they do? Actually, I don't know if this existed.
Speaker 1: Maybe they took your criticism in this, but they do
Speaker 1: have like the vice president and the president like they
Speaker 1: do put politicians in this version of the report. They
Speaker 1: don't have names, but they say the vice president in
Speaker 1: one of these scenarios, you know, handily wins the election
Speaker 1: of twenty twenty eight. We have one vice president.
Speaker 2: They have general secretary. I think I'm not sure if they.
Speaker 3: I mean general secretary, general secretary resembles she yep, and
Speaker 3: the vice president kind of resembles jd Vance, Right, I
Speaker 3: don't think the president resembles Trump at all, Right, is
Speaker 3: kind of the same.
Speaker 1: No, they did character like, yeah, they tried to side
Speaker 1: stuff that if we think it's.
Speaker 2: All happening in the next four years, then you know,
Speaker 2: presidential politics matter quite a bit. I mean, I know,
Speaker 2: I is such a fucking you know, I was jogging
Speaker 2: earlier on the East Side and I was listening to
Speaker 2: the S Reclin interview with Thomas Friedman. It's such a
Speaker 2: fucking yuppie fucking thing. Right, It's okay, It's okay.
Speaker 1: You're allowed to be a young.
Speaker 2: Not a huge fan of necessarily, but like you know,
Speaker 2: as well versed some like geopolitics and like China issues,
Speaker 2: like yeah, China, you've just been back from Chinese, Like yeah,
Speaker 2: China's kind of winning, you know what I mean, And
Speaker 2: like I'm not sure how how Trump's hawkishness on China,
Speaker 2: but like kind of imbecillically executed hawkishness on China, Like
Speaker 2: I'm not sure how that figure's in to this. Right,
Speaker 2: If we're reducing US China trade, that probably does produce
Speaker 2: an ai slow down maybe more for US if we're
Speaker 2: like if they're not exporting their like raw earth materials
Speaker 2: and and so forth, but we're making it hard for
Speaker 2: them to get in video chips, so they probably have
Speaker 2: like lots of workarounds and things like that. Maybe maybe
Speaker 2: Trump sheriffs are good. I would like to ask the
Speaker 2: author's this report, because it means that we're going to
Speaker 2: like have slower AI progress. And I'm not joking, right,
Speaker 2: that increases the hostility between the US and China in
Speaker 2: the long run, right, I mean that's if we were
Speaker 2: send it all the terariffs tomorrow. I think we still
Speaker 2: permanently or let's not say permanent, let's say at least
Speaker 2: for a decade or so, have injured us standing in
Speaker 2: the world. And so I don't know how that figures in.
Speaker 2: And I'm like, I'm also not sure like kind of
Speaker 2: quote what the rational response might be. But one thing
Speaker 2: they tried, let me make sure that they kept us
Speaker 2: into their report, right, so they actually have their implied
Speaker 2: approval rating for how people feel about open brain, which
Speaker 2: is there not very sense AI. I think this actually
Speaker 2: is some feedback who that they took into account. Right,
Speaker 2: they originally had it slightly less negative, but they have
Speaker 2: this being persistently negative and then getting more negative overtime.
Speaker 2: It was a little softer in the in the previous
Speaker 2: version that I saw, so they did change that one
Speaker 2: thing at some stage. But like the fact that like
Speaker 2: AI scares people, it scares people for both good and
Speaker 2: bad reasons, but I think mostly for valid reasons, right,
Speaker 2: that the fear is fairly bipartisan. That the biggest AI
Speaker 2: accelerators are now these kind of Republican techno optimists who
Speaker 2: are not looking particularly wise given how it's going with
Speaker 2: the first ninety days whatever we are, the Trump administration
Speaker 2: and the likelihood of like a substantial political backlash right,
Speaker 2: which could lead to dumb types of regulations. But like
Speaker 2: in a part of it too is like okay, AI
Speaker 2: they're saying can do not just computer desk jobs, but
Speaker 2: like all types of things, right, And like humans kind
Speaker 2: of play this role initially as supervisors, and then literally
Speaker 2: within a couple of years people start to say, you
Speaker 2: know what, am I really adding much much value here? Right?
Speaker 2: You kind of have like these legacy jobs and there's
Speaker 2: a lot of money. I think mostly scenarios imagine very
Speaker 2: fast economic growth, although maybe very lumpy, right for some
Speaker 2: parts of the world and not others. But we're kind
Speaker 2: of just sitting around with a lot of idle time.
Speaker 2: It might be good for live poker Maria. Right, all
Speaker 2: of a sudden, all these smart people, their open earth,
Speaker 2: their open brain, excuse me, stock is now worth billions
Speaker 2: of dollars, right, and like nothing to do because the
Speaker 2: AI is doing all their work. Right, they have a
Speaker 2: lot of fucking time to play some fucking texas hold
Speaker 2: them right, that.
Speaker 1: Is, that is one way of thinking about it. Let's
Speaker 1: let's go back to your your earlier point, which I
Speaker 1: actually think is an important one because obviously they were
Speaker 1: trying to do as all you know, super forecasting tries
Speaker 1: to do is you try to create a rape that
Speaker 1: will work in multiple scenarios. Right. You can't tie it
Speaker 1: too much to like the present moment, otherwise your forecasts
Speaker 1: are going to be quite biased. However, I do think
Speaker 1: that what you raise kind of our current situation with China,
Speaker 1: et cetera, has very real implications. Given that this is
Speaker 1: kind of the central dynamic of this report that their
Speaker 1: predictions are based on, I think that it's incredibly valid
Speaker 1: to actually speculate, you know, how will if at all,
Speaker 1: this effect the timeline of the predictions, the possible the
Speaker 1: likelihood of the two scenarios. And I will also say
Speaker 1: that one of the things in the report is that
Speaker 1: all of these negotiations on like will we slow down?
Speaker 2: Will we not?
Speaker 1: How aligned is it? This all takes place in secret, right,
Speaker 1: Like we don't know the humans don't know that it's
Speaker 1: going on. We don't know what's happening behind the scenes,
Speaker 1: and we don't know what the decision makers are kind
Speaker 1: of thinking. And so for all we know, you know,
Speaker 1: President Trump is meeting with Sam Altman and trying to
Speaker 1: trying to could do some of these things. And it's
Speaker 1: funny because we were kind of pushing for transparency in
Speaker 1: one way, but there's a lot of things here that
Speaker 1: are very much not transparent.
Speaker 2: Yeah, it's kind of the deep state, right, but also
Speaker 2: also a lot of the negotiations are now AI versus AI, right.
Speaker 2: And look, I'm not sure that AIS will have that
Speaker 2: trust both with the external actor and internally. I'm skeptical
Speaker 2: of that. Right. If that does happen, they kind of
Speaker 2: think this might be good because the AIS will probably
Speaker 2: behave in like literally game theory optimal way right and
Speaker 2: understand these things and make I guess, like fewer mistakes
Speaker 2: than humans might if.
Speaker 1: They're properly aligned. Like that's a crucial thing because in
Speaker 1: the doomsday scenario, AI negotiates with AI, but they conspire
Speaker 1: to destroy humanity. Right, So there are two scenarios. One
Speaker 1: it's actually properly aligned, so AI negotiates with AI, game
Speaker 1: theory works out and we end up you know, democracy
Speaker 1: and wonderful things. But in the other one, where they're misaligned,
Speaker 1: AI negotiates with AI to create a new AI basically
Speaker 1: and destroy humanity. So it can go one way or
Speaker 1: the other depending on that alignment step. First of all,
Speaker 1: I mean the Utilian.
Speaker 2: You didn't see that utopian to me, right, I'm not
Speaker 2: sure that it did it.
Speaker 1: It actually seemed quite distoping to me, Like it seemed
Speaker 1: incredibly distised.
Speaker 2: It's kind of like, you know, look, but at least
Speaker 2: we're still alive, that we'll have chures, we'll probably live longer.
Speaker 2: And again, lots of lots of lots of poker. The
Speaker 2: ail you writing Silver Bulletin and and hosting our podcast. Right,
Speaker 2: let me let me back up a little bit, because
Speaker 2: I think we maybe take for granted that some of
Speaker 2: these premises are are kind of controversial, right, so they
Speaker 2: have a breakpoint I think in twenty twenty six, well
Speaker 2: says our why aren't CREA increased potentially be on twenty
Speaker 2: twenty six? Right, So that's kind of the breakpoint is
Speaker 2: like twenty seven is this inflection point? I think I'm
Speaker 2: using that term correctly in this context.
Speaker 3: You know.
Speaker 2: So I'm reading this report and up to twenty twenty
Speaker 2: six and like, thumbs up, YIAA. This seems like very
Speaker 2: smart and detailed about, like, you know, how the economy
Speaker 2: is reacting and how politics reacting in the race dynamic
Speaker 2: with China. Maybe there needs to be little bit more
Speaker 2: Trump in there. I understand why politically they didn't want
Speaker 2: to get into that mess, right, But like, so there's
Speaker 2: kind of three different things here, right. One is a
Speaker 2: notion of what's sometimes called AGI, or artificial general intelligence.
Speaker 2: And if you ask one hundred different researcher you get
Speaker 2: one hundred different definitions of what AGI is. But you know,
Speaker 2: I think it is based to like being able to
Speaker 2: do a large majority of things that a human being
Speaker 2: could do competently, siming we're limiting it to kind of
Speaker 2: like desk job type tasks. Right, anything that can be
Speaker 2: done remotely is sometimes definition that is used or through
Speaker 2: remote work, right, because clearly AIS are inferior to humans
Speaker 2: and like sorting and folding laundry or things like that, right,
Speaker 2: that requires certain type of intelligence. Right. If you use
Speaker 2: the kind of desk job definition, then like AI is
Speaker 2: already pretty close to AGI. Right. I use large language
Speaker 2: models all the freaking time, and they're not perfect for everything.
Speaker 2: I felt like, you know, in terms of like being
Speaker 2: able to do the large majority of desk work at
Speaker 2: levels ranging from competent intern to super genius, Like on average,
Speaker 2: it's probably pretty close to being generally intelligent by that definition, right.
Speaker 1: If you're the one using it. I just want to
Speaker 1: like once again point that out because one of the
Speaker 1: things that they say in the report is that as
Speaker 1: it gets more and more involved what we're asking AI
Speaker 1: to do, it's like the human process to evaluate whether
Speaker 1: it's accurate and whether it's making mistakes will get longer
Speaker 1: and longer. And I think they say like for every
Speaker 1: like basically one day of work, it'll take several it's
Speaker 1: like a two to one ratio at the beginning for
Speaker 1: how long it will take humans to verify the output. Right,
Speaker 1: So you think, like you think you save time by
Speaker 1: having aids do this, but if you want it to
Speaker 1: actually develop correctly, then you need a team and it
Speaker 1: takes them twice as long to verify that what the
Speaker 1: AI did is actually true and actually valid and actually aligned,
Speaker 1: et cetera, et cetera. Now you're not asking it to
Speaker 1: do things that require that amount of time, but there
Speaker 1: do need to be little caveats to how we how
Speaker 1: we think about their usefulness and how you are able
Speaker 1: to evaluate the output versus Southern scenarios.
Speaker 2: When I use AI, the things that it's best with
Speaker 2: are things that like save any time. Right where I
Speaker 2: feeded a bunch of different names for different college basketball teams.
Speaker 2: We work in our NCAA model. I'm like, take these
Speaker 2: seven different naming conventions that are all different and create
Speaker 2: a cross reference table of these, which is kind of
Speaker 2: like a hard task. You need to have a little
Speaker 2: context about basketball. And it did that very well. Right,
Speaker 2: That's something I could have done. It might have taken
Speaker 2: an hour or two, but you know, instead, so it
Speaker 2: could do it. It's in a few minutes, and it
Speaker 2: gets faster. It's like, oh, I've learned from this from
Speaker 2: you before, Nate, so now I can be faster and
Speaker 2: doing this type of task in the future. I was
Speaker 2: at the poker tournament down in Florida last week and like, uh,
Speaker 2: you know, I ask open research. Excuse me, god, is that?
Speaker 1: See exactly right? It all because deep research, deep research,
Speaker 1: deep research.
Speaker 2: I asked deep reach.
Speaker 1: Reading too many of these twenty twenty seven reports.
Speaker 2: Super brand gets to pull a bunch of stock market
Speaker 2: data for me. And then I'm playing at poker hand
Speaker 2: and like, make a really thin, sexy value bet with
Speaker 2: like fourth pair. No one knows that means, right, I
Speaker 2: bet a very weekend. I thought the other guy would
Speaker 2: call it with an even weaker hand, and I was right.
Speaker 2: And I feel like I'm such a fucking stud here
Speaker 2: valuating fourth pair, And well AI does the work for me,
Speaker 2: and then of course I like bust out of the
Speaker 2: tournament an hour later. And meanwhile, you know, deep research
Speaker 2: bungles this particular task. But in general, AI has been
Speaker 2: very reliable. But the point is that, like there's like
Speaker 2: a a inflection point where like I'm asking you to
Speaker 2: do things that like are just a faster version of
Speaker 2: what I could do myself, I would at the moment
Speaker 2: I ask AI, be like, I want you design a
Speaker 2: new NCA model for me with these parameters, because like
Speaker 2: I wouldn't know how to test it. But anyway, I'm
Speaker 2: mean long wanted to hear so AGI, we're gonna get AGI,
Speaker 2: or at least we're going to get someone calling something
Speaker 2: AGI soon. Right, Artificial super intelligence where it's doing things
Speaker 2: much better than human beings. I think this report takes
Speaker 2: for granted or not it takes for granted. It has
Speaker 2: lots of documentation about its assumptions, but it's saying, Okay,
Speaker 2: this trajectory has been very robust so far, and people
Speaker 2: make all types of bullshit predictions of the fact that
Speaker 2: these guys in particular have made accurate predictions of the
Speaker 2: past is certainly worth something, I think, right, But they're like, okay,
Speaker 2: you kind of follow the scaling law, and before too
Speaker 2: much longer, you know, AI starts to be more intelligent
Speaker 2: than human beings. You can debate what intelligent means if
Speaker 2: you want, but do superhuman of things or and or
Speaker 2: do them very fast? I think might be different, right,
Speaker 2: And NA that can do things very fast is and
Speaker 2: maybe it's certainly a component of intelligence. Right, But like,
Speaker 2: but I don't take for granted that like quote unquote,
Speaker 2: AI can reliably extrapolate beyond the data set. I just
Speaker 2: think that, like it's not an absurdly pavlogic. It may
Speaker 2: even be like the base case are close to the
Speaker 2: base case, but like that's not assumable from first principles.
Speaker 2: I don't think we've all seen lots of trend charts
Speaker 2: of thing. You know, if you look at a chart
Speaker 2: of Japan's GDP in the nineteen eighties, he might have said, Okay, well,
Speaker 2: Japan's gonna take over the world and be and people
Speaker 2: bought this and now it's kind of hasn't grown for
Speaker 2: like forty years basically, right, And so like we've all
Speaker 2: seen lots of curves that go up and then it's
Speaker 2: actually an estra where the fuck you call it? Where
Speaker 2: like it begins to bend the other way at some
Speaker 2: point and and we can't tell until later. Right. The
Speaker 2: other thing is like the a billy of AI to
Speaker 2: plan and manipulate the physical world. I mean some of
Speaker 2: these things where they're talking about, like you know, brain
Speaker 2: uploading and dice and swarms and nanobots, like you know, there,
Speaker 2: I would literally wager money against this happening on the
Speaker 2: time scales that they're talking about, right, they double the timescale. Okay,
Speaker 2: then I might start to give some more probability. And look,
Speaker 2: I'm willing to be wrong about that. I guess we'll
Speaker 2: all be dead anyway, fifty percent likely in this scenario.
Speaker 1: But like, like this scenario fifty percent you do like AA.
Speaker 2: You know, the physical world requires sensory input and lots
Speaker 2: of parts of our brain that AI is not as
Speaker 2: effective at manipulating. It also requires like being given or
Speaker 2: commandeering resources somehow. By the way, this is like a
Speaker 2: little bit of a problem for the United States. I
Speaker 2: mean we are behind China by like quite a bit
Speaker 2: in robotics and related things. Right, So, like, I don't
Speaker 2: know what happens if like we have the brainier, smarter AIS,
Speaker 2: but like they're very good at manufacturing and machinery. So like,
Speaker 2: what if we have the brains and they have the
Speaker 2: brawn so to speak, right, and they have maybe a
Speaker 2: more authoritarian but functional infrastructure. So I don't know what
Speaker 2: happens then, right, Like, but the abilitiy of AIS to
Speaker 2: comment to your resources to control the physical world to me,
Speaker 2: seems far fetched on these timelines in part because of politics, right,
Speaker 2: I mean the fact that it takes so long to
Speaker 2: build a new building in the US, or a new
Speaker 2: subway station or a new highway, and the fact that
Speaker 2: our politics is kind of sclerotic. Right. And look, I
Speaker 2: mean I don't want to sound like too pessimistic, but
Speaker 2: if you read the book I recommended last week, fight,
Speaker 2: I mean, we basically did not have a fully competent
Speaker 2: president for the last two years, and I would argue
Speaker 2: that we don't have one for the next four years. Right, So,
Speaker 2: like all these kind of things that we have to
Speaker 2: plan for, like who's doing that fucking planning? Our government's
Speaker 2: kind of dysfunctional. And you know, maybe that means we
Speaker 2: just lose to China. Right, Maybe that means we lose
Speaker 2: to China, at least we'll have like nice cars. I guess.
Speaker 1: We'll be back right after this. I think that your
Speaker 1: point about the growth trajectories not necessarily being reliable is
Speaker 1: a very valid one. The Japanic example is great. You know,
Speaker 1: malefusion population growth is another is another big one. Right,
Speaker 1: We thought that population would explode and instead we're actually
Speaker 1: seeing population decline, so you know, the world does change.
Speaker 1: The thing that I think they rely on is that
Speaker 1: that AIS are capable of designing just this incredible technology
Speaker 1: much more quickly so that our building process and all
Speaker 1: of that gets sped up one hundredfold from what it
Speaker 1: is right now. But it's still, at least at this point,
Speaker 1: needs humans to implement it right, and needs all of
Speaker 1: these different workers. And so yeah, I think there are
Speaker 1: some assumptions built into here that I hope, like I
Speaker 1: hope that that timeline isn't feasible, and I do think
Speaker 1: that there are things that are holding us back. All
Speaker 1: the same, I think it's really I think it's interesting.
Speaker 1: One of the reasons I like this report is that
Speaker 1: it forces you to think about these things right and
Speaker 1: try to game out some of these worst case scenarios
Speaker 1: to try to prevent them, which I think is always
Speaker 1: an important thought exercise. I do want to go back
Speaker 1: to kind of their good scenario, which just so bad
Speaker 1: scenario is, you know, we're all wiped out by a
Speaker 1: chemical warfare that the AI is release on us. Good
Speaker 1: scenario is that you know, everyone gets a universal basic
Speaker 1: income and AI does everything and no one has to
Speaker 1: do anything and we can just you know, at play poker. Yeah,
Speaker 1: and that just as as you suggested, that seems actually
Speaker 1: like a very dystopian scenario where people can become much
Speaker 1: more easy to brainwash, control, et cetera, et cetera. It's
Speaker 1: like a dumbing down, right where where we're not challenged
Speaker 1: to produce good art, to advance in any sort of way.
Speaker 1: Just to me, it does not seem like a very meaningful.
Speaker 2: Well, the question is there's not way another if you
Speaker 2: read AI twenty twenty seven, which I highly recommend that
Speaker 2: you read it. There's also another post by a pseudonymous
Speaker 2: poster called el Rudolph L who wrote something called A
Speaker 2: History of the Future twenty twenty five to twenty forty,
Speaker 2: which is very detailed but goes through kind of like
Speaker 2: what this looks like at like more of a human level,
Speaker 2: how society evolves, how the economy evolves, how work evolves, right,
Speaker 2: and like very detail, just like AI twenty twenty seven
Speaker 2: is they're kind of focused on the parts of A
Speaker 2: twenty seven, then I think it kind of deliberately ignores
Speaker 2: maybe can call them mild blind spots or whatever, right,
Speaker 2: But like, but that's interesting because that kind of thinks
Speaker 2: about what types of jobs are there in the future.
Speaker 2: There are probably lots of lawyers actually, right, because you know,
Speaker 2: the law is very sluggish to change, especially in a
Speaker 2: constitutional system where there are lots of vitail points, right,
Speaker 2: probably high end service sector. You know, you go to
Speaker 2: a restaurant because everyone's for a lot of people are
Speaker 2: rich now, right, and you're flattered by the attractive young
Speaker 2: server and things like that is kind of highly kind
Speaker 2: of like catered and curated experiences. I guess I have
Speaker 2: some faith in humanities beilieved to fight back quote unquote
Speaker 2: against like two scenarios that it might not really like
Speaker 2: either one, you know what I mean. And like the
Speaker 2: scenario where like AI is producing like ten percent GDP
Speaker 2: growth or or whatever. Right, man, it's great a few
Speaker 2: own stocks that are exposed to AI and tech companies probably, right,
Speaker 2: but it's also making that money on the backs of
Speaker 2: mass job displacement. And like, you know, it kind of
Speaker 2: a sert confident in the long run that human beings
Speaker 2: fine productive things to do, and and you know, massive
Speaker 2: employment has been predicted many times and never really occurred, right,
Speaker 2: But like, but it's not occurring this fast where they
Speaker 2: think the world ends in six years of whether they're
Speaker 2: predicting or we have utopia in six years, and like
Speaker 2: just the ability of like human society to like deal
Speaker 2: with that change at these time scales leads to like
Speaker 2: more chaos than I think they're more predicting. I think also,
Speaker 2: and I told them this too, right, I think also
Speaker 2: leads to more constraints. Right that you hit bottlenecks if
Speaker 2: you have five things you have to do, right, and
Speaker 2: you have the world's fastest computer, et cetera, et cetera,
Speaker 2: But there's like a power outage in your neighborhood, right,
Speaker 2: and that's a that's a bottle deck. Right, Maybe there
Speaker 2: are ways around it if you're like go to home
Speaker 2: depot and buy a generator or you know what I mean.
Speaker 2: But like, but the point is that, like you're often
Speaker 2: defined like the slowest link, and politics are sometimes the
Speaker 2: slowest link. But also but like also, you know, I
Speaker 2: think the report maybe under states, and I think kind
Speaker 2: of in general, the a safty community like maybe understates
Speaker 2: the ability of like human beings to cause harm to
Speaker 2: other human beings with AI, right, that concern kind of
Speaker 2: gets brushed off as like to pedestrian or like like to.
Speaker 1: Say, Pedestrian was saying, yeah, the exact word I was
Speaker 1: thinking of. I think that's a good I mean, I
Speaker 1: think that's a great place to end it, because yes,
Speaker 1: we do need to be concerned about all of these
Speaker 1: things about AI. But like that that phrase I think
Speaker 1: is very crucial, like do not underestimate the ability of
Speaker 1: humans to cause harm to other humans. And I think
Speaker 1: that that's you know, it's not a very opt it's
Speaker 1: not a very pleasant place to end, but I think
Speaker 1: it's a really important place to end. And I think
Speaker 1: that that's a very valid kind of way of reflecting
Speaker 1: on this name.
Speaker 2: Or to trust AIS too much, right, I gener I
Speaker 2: think that concern is like somewhat misplays but like if
Speaker 2: we're handing over critical systems to AI, right, it can
Speaker 2: cause problems if it's very smart and deceives us and
Speaker 2: doesn't like us very much. It can also cause problems
Speaker 2: if it has hallucinations, bugs in critical areas where it
Speaker 2: isn't as robust and hasn't really been tested yet that
Speaker 2: are outside of its domain yep, or there could be espionage. Anyway,
Speaker 2: we will have plenty of time, although maybe only seven
Speaker 2: more years actually to like explore, explore these scenarios.
Speaker 1: Yes, and in seven years we'll be like, welcome back
Speaker 1: to the final episode of Risky Business, because the prediction
Speaker 1: is we're all going to be done tomorrow. Oh but yeah,
Speaker 1: this was an interesting exercise, and I think my my
Speaker 1: p doom has slightly gone up as a result of
Speaker 1: reading this. But I also remain optimistic that humans can
Speaker 1: and can do good as well as harm.
Speaker 2: Yeah, my interest in learning Chinese has increased as a
Speaker 2: result of recent developments. I know about my pe doo.
Speaker 1: All right, yeah, let's do some language immersion. I'm with you.
Speaker 1: That's it for today. If you're a premium subscriber, we
Speaker 1: will be answering a question about whether MFAs can ever
Speaker 1: be plus ev right after the credits, And if you're
Speaker 1: not a subscriber, it's not too late. For six ninety
Speaker 1: nine a month the price of a midber, you get
Speaker 1: access to all these conversations and all premium content across
Speaker 1: the Pushkin Network. Risky Business is hosted by me Maria Kannakova.
Speaker 2: And by me Nate Silver. The show is a co
Speaker 2: production of Pushkin Industries and iHeartMedia. This episode was produced
Speaker 2: by Isabel Carter. Our associate producer is Sonya Gerwitz. Sally
Speaker 2: Helm is our editor, and our executive producer is Jacob Goldstein.
Speaker 2: Mixing by Sarah Bruger. If you like the show, please
Speaker 2: rate and review us. You know we'd like we take
Speaker 2: a four or five. We take the five, rate and
Speaker 2: review us to other people. Thank you for listening.