Speaker 1: Pushkin.
Speaker 2: Hey, it's Jacob.
Speaker 1: There's another Pushkin show called Risky Business. It's a show
Speaker 1: about decision making and strategic thinking. It's hosted by Nate
Speaker 1: Silver and Riya Kanakova, and they just had a really
Speaker 1: interesting episode about the future of AI. So we're going
Speaker 1: to play that episode for you right now. We'll be
Speaker 1: back later this week with the regular episode of What's
Speaker 1: Your Problem.
Speaker 3: Welcome back to Risky Business, a show about making better decisions.
Speaker 3: I'm Maria Kanakova.
Speaker 2: And I'm Nate Silver.
Speaker 3: Hey. Today on the show is going to be a
Speaker 3: little bit doomtastic.
Speaker 2: Yeah, I mean, I don't know if it's like worse
Speaker 2: and thinking about like the global economy going into a
Speaker 2: recession because of dumb fuck tariff policies about how we're
Speaker 2: all going to die in seven years and staid no,
Speaker 2: I'm just kidding. This is a very very intelligent and
Speaker 2: well written and thoughtful report called AI twenty twenty seven
Speaker 2: that we're to spend the whole show on I because
Speaker 2: think it's such an interesting to talk about, but that
Speaker 2: you know, includes some dystopian possibilities. I would say it does.
Speaker 3: Indeed, So let's get into it and hope that you
Speaker 3: guys are all still here to listen to us in
Speaker 3: seven years.
Speaker 2: The contrast is interesting between like all the chaos we're
Speaker 2: seeing with tariff policy in terms of starting a trade
Speaker 2: war with China and then other types of chaos. It's
Speaker 2: interesting to kind of look at this. I mean, I
Speaker 2: wouldn't call it a more optimistic future exactly, like but
Speaker 2: like on a different trajectory of like a future that's
Speaker 2: going to change very fast, according to these authors, with
Speaker 2: profound implications for you know, everything, the humans species. These
Speaker 2: researchers and authors are saying that everything is going to
Speaker 2: change profoundly.
Speaker 3: And.
Speaker 2: Even though there is some hedging hair, this is kind
Speaker 2: of their base case scenario. And like you know, base
Speaker 2: case number one and base case number two differ. There's
Speaker 2: like kind of a choose your own adventure at some
Speaker 2: point in this report, but they're both very different than
Speaker 2: the status quo, right, and the notionately hear from that,
Speaker 2: like everything becomes different If ais become substantially more intelligent
Speaker 2: than human beings. People can debate and we will debate
Speaker 2: on this program what that means. But yeah, do you
Speaker 2: want to do you want to contextualize this for me?
Speaker 2: Do you want to tell people too. Yeah, there's are
Speaker 2: of this report.
Speaker 3: Absolutely absolutely. So the report is authored officially by five people,
Speaker 3: but I think unofficially there's also a sixth. We've got
Speaker 3: Eli Leiflund, who's a super forecaster and he was ranked
Speaker 3: first on Rand's forecasting initiative, so he is someone who
Speaker 3: is very good at kind of looking at the future
Speaker 3: trying to predict what's going to happen. You have Jonas Walmer,
Speaker 3: who's a VC at Macroscopic Ventures. Thomas Larson was the
Speaker 3: former executive director of the Center for AI Policy, so
Speaker 3: that is a center that advises both sides of the
Speaker 3: aisle on how AI is going to go. And Romeo Dean,
Speaker 3: who is part of Harvard's AI Safety Student Team, so
Speaker 3: someone who is still a student, still learning, but kind
Speaker 3: of the next generation of people looking at AI. And
Speaker 3: finally we have Daniel Cochtaylo, who basically had written a
Speaker 3: report back in twenty twenty one. He was a AI
Speaker 3: researcher and he looked at predictions for AI for twenty
Speaker 3: twenty six and it turns out that his predictions were
Speaker 3: pretty spot on, and so Open AI actually hired Daniel
Speaker 3: as a result of this report.
Speaker 2: Obviously he left, yes, and.
Speaker 3: Then and then he left exactly.
Speaker 2: And importantly, So there's also Scott Alexander exactly.
Speaker 3: So I was saying, he's the he's the person who
Speaker 3: kind of is in the background, and he's you guys
Speaker 3: might know him as the author behind astral Codex.
Speaker 2: And I know Scott. He's one of the kind of
Speaker 2: fathers of what you might call rationalism. I think Scott,
Speaker 2: when I interviewed him from a book, was happy enough
Speaker 2: with that term and accused me or co opted me
Speaker 2: into also being a rationalist. These people are somewhat adjacent
Speaker 2: to the effective altruist, but not quite right. They're just
Speaker 2: trying to apply a sort of thoughtful, rigorous, quantitative lens
Speaker 2: to big picture problems, including existential risk, of which most
Speaker 2: people in this community believe that AI is both an
Speaker 2: existential risk and also kind of an existential opportunity, right,
Speaker 2: that it could transform things. You talk to Sam Alp
Speaker 2: and he'll say, we're going to cure cancer and elimina
Speaker 2: poverty and whatever else. Right. And Scott's also an excellent writer.
Speaker 2: And so let me disclose something which is slightly important here.
Speaker 2: So I actually was approached by some of the authors
Speaker 2: of this report a couple of months ago, I guess
Speaker 2: it was in February ish, just to give feedback and
Speaker 2: chat with them. So I'm working off the draft version right,
Speaker 2: which I do not leave. They changed very much, so
Speaker 2: my notes pertained to an earlier draft. I did not
Speaker 2: have time this morning to go back and re.
Speaker 3: Read it, so I was not on the inside loops.
Speaker 3: So I did not get an earlier draft. And I've
Speaker 3: read this draft and basically just to kind of big
Speaker 3: picture sum it up, it outlines two scenarios, right, two
Speaker 3: major scenarios for how AI might change the world as
Speaker 3: soon as twenty thirty. Now important note like that that
Speaker 3: date is could have hedged. It might be sooner, it
Speaker 3: might be later. There's kind of a there's a confidence
Speaker 3: interval there. But the two different scenarios. One basically we
Speaker 3: do twenty thirty, humanity disappears and is taken over by AI.
Speaker 3: The positive report is in twenty thirty basically we get
Speaker 3: AIS that are aligned to our interests, and we get
Speaker 3: kind of this AI utopia where AIS actually help make
Speaker 3: life much better for everyone and make the standard of
Speaker 3: living much higher. But the crucial turning point is before
Speaker 3: twenty thirty. And the crucial kind of question at the
Speaker 3: center of this is will we be able to design
Speaker 3: AIS that are truly aligned to human interests rather than
Speaker 3: just appear to be aligned and kind of lying to
Speaker 3: us while actually following their own agenda. And how we
Speaker 3: handle that is kind of the lynchpin. And it's actually interesting,
Speaker 3: Nate that you started out with China, because a lot
Speaker 3: of the policy choices and a lot of what they
Speaker 3: see as kind of the decision points will affect the
Speaker 3: future of humanity actually hinge on the US China dynamic,
Speaker 3: how they compete with each other, and how that sometimes
Speaker 3: might basically clash against safety concerns because no one wants
Speaker 3: to be left behind. Can we manage that effectively and
Speaker 3: can kind of that transition work in our favor as
Speaker 3: opposed to against us. I think that this is kind
Speaker 3: of one of the big questions here, And so it's
Speaker 3: funny that we're seeing all of this trade war right
Speaker 3: now as the sideport is coming out.
Speaker 2: Yeah. Look, I think this exercise is partly just a
Speaker 2: forecasting exercise, right, I mean, obviously it's just kind of
Speaker 2: like fork at the bottom where we learn to have
Speaker 2: an AI slow down or we kind of are pressing
Speaker 2: fully on the accelerator, right, Like, in some ways, scenarios
Speaker 2: are like not that different, right, either one assumes remarkable
Speaker 2: rates of technological growth that I think even AI I'm
Speaker 2: never quite sure who to call an optimist or a pessimist, right,
Speaker 2: even AI believers you know, might think is a little
Speaker 2: bit aggressive. Right, But what they want to do is
Speaker 2: they want to have like a specific, fleshed out scenario
Speaker 2: for how the world would look like. It's kind of
Speaker 2: like a modal scenario. And like, I think they'd say that, like,
Speaker 2: we're not totally sure about either of these necessarily, right,
Speaker 2: And I don't think they'd be as like pedantic as
Speaker 2: to say, if you do X, Y and Z, then
Speaker 2: we'll save the world and have utopia, and if you don't,
Speaker 2: then we'll all die, Right. I think they'd probably say
Speaker 2: it's unclear and there's kind of like risk either way.
Speaker 2: We wanted to go through the scenario of like fleshing
Speaker 2: out like what the world might look like, right. I
Speaker 2: do think one thing that's important is that whatever decisions
Speaker 2: are made now could get locked in, right that you
Speaker 2: pass certain points in overturn and it becomes very hard
Speaker 2: to decelerate like an arms race. This is you know
Speaker 2: what we found during the Cold War for example. I
Speaker 2: mean one of the big things I look at is like,
Speaker 2: do we force the AI to be transparent in its
Speaker 2: thinking with humans? Right? Like, now there's been a movement
Speaker 2: toward the ai'll actually explicate it's thinking more. I'll ask
Speaker 2: it a query open AI. The Chinese models to this too, right,
Speaker 2: and I'll say, I am thinking about X, Y and Z,
Speaker 2: and I'm looking up PD and Q and now I'm
Speaker 2: reconsidering this. It's actually has this chain of thought process, right,
Speaker 2: which is explicated in English. You know. One concern is
Speaker 2: that what if the AI just kind of communicates to
Speaker 2: another in these implicit vectors that's inferring from all the
Speaker 2: texts it has. It's kind of unintelligible to human beings, right,
Speaker 2: and maybe kind of quote unquote thinking in that way
Speaker 2: in the first place, and then does us the favor
Speaker 2: of like translating back so it it goes from English
Speaker 2: to kind of this big bag of numbers is one
Speaker 2: a researcher called it, right, and then it translates it
Speaker 2: back into English or whatever land which you want. Really,
Speaker 2: in the end, what if it just skip cuts out
Speaker 2: that last step, right, then we can't kind of like
Speaker 2: check what AI is doing. Then it can behave deceptively
Speaker 2: more easily. So you know, so that part seems to
Speaker 2: be important. I want to hear your your first impressions
Speaker 2: before I kind of poison the well too much.
Speaker 3: Well, my first impressions is that the alignment problem is
Speaker 3: a very real one and an incredibly important one to solve.
Speaker 3: And what I got from this is that actually, the
Speaker 3: problem that I've had with like these initial AI lms
Speaker 3: is the kernel of what they're seeing there. Right, So
Speaker 3: you and I have talked about this on the show
Speaker 3: in the past, and I've said, well, my problem is
Speaker 3: that when I'm a domain expert, right, I start seeing
Speaker 3: some inaccuracies, and I start seeing like places where like
Speaker 3: it either just didn't do well or made shit up
Speaker 3: or whatever it is. Now, I think it's very clear
Speaker 3: that those problems aren't going to go away, right, that
Speaker 3: that is going to get much much better. However, the
Speaker 3: kernel of it's showing me some thing, but that might
Speaker 3: just be you know, I have no way of verifying
Speaker 3: if that's what's going on, what it's reading, like, how
Speaker 3: it's I don't want to say thinking about it, even
Speaker 3: though in the report they do use thinking, but it's normal.
Speaker 2: Vizing too much, I think is correct. I think it's.
Speaker 3: Okay, we'll stick to that language. Yeah, okay, So, so
Speaker 3: how it's thinking about it that those little problems and
Speaker 3: like the little the glitches and the things that it
Speaker 3: might be doing where it starts actually glitching on purpose
Speaker 3: are not going to be visible to the human eye.
Speaker 3: And so one of the main things that they say
Speaker 3: here is that as AI internal R and D gets
Speaker 3: rapidly faster, so that means basically AI's researching AIS, right,
Speaker 3: and so internally they start developing new models, and as
Speaker 3: they kind of surpass human ability to monitor it, it
Speaker 3: becomes progressively more difficult to figure out, Okay, is the
Speaker 3: AI actually doing what I want to do? Is the
Speaker 3: output that it's giving me its actual thought process, and
Speaker 3: is it accurate or is it like trying to deceive me?
Speaker 3: But it's actually kind of inserting certain things on purpose
Speaker 3: because it has different goals, right, because it is actually
Speaker 3: secretly misaligned, but it's very good at persuading me that
Speaker 3: it's aligned. Because one of the things that actually came
Speaker 3: out of this report and I was like, huh, you
Speaker 3: know this is interesting is if we get this remarkable
Speaker 3: improvement in AI, it will also remarkably improve at persuading us, right,
Speaker 3: so as part of making yeah, so this is this
Speaker 3: is but I've never even thought about that. I was like, okay, fine,
Speaker 3: But one of the things that I do buy is
Speaker 3: that it's going to be very difficult for us to
Speaker 3: monitor it and to figure out like is it truly
Speaker 3: aligned with human wants, with human desires, with human goals,
Speaker 3: And the experts who are capable of doing that, I
Speaker 3: think are actually going to dwindle right as AI starts
Speaker 3: proliferating in society. And so to me, that is something
Speaker 3: that is actually quite worrisome, and that is something that
Speaker 3: we really need to be paying attention to now. Just
Speaker 3: to fast forward a little bit, in their doomsday scenario
Speaker 3: in twenty thirty one, AI takes over it basically like
Speaker 3: suddenly releases some chemical agents, right, and humanity dies and
Speaker 3: the rest of the stragglers are taken care of by drones,
Speaker 3: et cetera. I don't even like it's it doesn't even.
Speaker 2: Quick and painless death.
Speaker 3: I will say, let's hope we don't know what the
Speaker 3: chemical agents are might not be quick and painless. Some
Speaker 3: chemical agents are actually a very painful death thing. So
Speaker 3: let's hope let's hope it's quick and painless. Quick, quick, yes,
Speaker 3: stuff quick? Okay, hopefully some chemical agents are not quick.
Speaker 3: Let's hope it's it's quick and painless. But if they're
Speaker 3: actually capable of deception at that high level, then you
Speaker 3: technically don't even need them to do it. If we're
Speaker 3: trusting medicine and all sorts of things to the AIS,
Speaker 3: it's pretty easy for it to actually manipulate something and
Speaker 3: actually insert something into codes, et cetera that will fuck
Speaker 3: up humanity in a way that we can't that we
Speaker 3: can't actually figure out at the moment, right, Like the
Speaker 3: way I think of it, And this is not from
Speaker 3: the this is not from the paper, but this is
Speaker 3: just the way that like my mind processed. It is
Speaker 3: like think about DNA, right, Like you have these remarkably complex,
Speaker 3: huge strands of data, and as we've found out, but
Speaker 3: it's taken forever, one tiny mutation can actually be fatal, right,
Speaker 3: but you can't spot that mutation. Sometimes that mutation isn't
Speaker 3: fatal immediately but will only manifest at a certain point
Speaker 3: in time. That's the way that my mind tried to
Speaker 3: kind of try to conceptualize what this actually means. And
Speaker 3: so I think that, you know, that would be easy
Speaker 3: for a deceptive AI to do and to me like that,
Speaker 3: that's kind of the big takeaway from this report is
Speaker 3: that we need to make sure that we are building
Speaker 3: AIS that will not deceive right that they're capabilities, they
Speaker 3: explain them in an honest way, and that honesty and
Speaker 3: trust is actually prioritized over other things, even though it
Speaker 3: might slow down research, it might slow down other things,
Speaker 3: but that that kind of alignment step is absolutely crucial
Speaker 3: at the beginning, because otherwise humans are human, right, they're
Speaker 3: easily manipulated, And we often trust that computers are quote
Speaker 3: unquote rational because they're computers, but they're not. They have
Speaker 3: their own inputs, they have their own weights, they have
Speaker 3: their own values, and that could just lead us down
Speaker 3: a dark path.
Speaker 2: Yeah, so let me follow up with this pushback. I
Speaker 2: guess right, Like, first of all, I don't know that
Speaker 2: humans are so easily persuaded. This is my big critique
Speaker 2: with like all the misinformation people who say, well, misinformation
Speaker 2: is the biggest problem that society basis. It's like people
Speaker 2: are actually pretty stubborn and they're kind of sound pretentious.
Speaker 2: They're kind of basian and how they formulate their beliefs, right,
Speaker 2: they have some notion of reality. They're looking at the
Speaker 2: credibility of the person who is telling them these remarks.
Speaker 2: If it's an unpersuasive source, it might make them less
Speaker 2: likely to believe they're balancing what other information with their
Speaker 2: lived experience so called Right. You know, part of the
Speaker 2: reason that like I am skeptical of AIS being super
Speaker 2: persuasive is like you know that it's an AI. You
Speaker 2: know it's trying to persuade you. You know what I mean.
Speaker 2: Like if you go and play poker against like a
Speaker 2: really chatty player, if Phil Hemmi with or Scott Sieber
Speaker 2: or someone like that, Right, you know, on some level
Speaker 2: of the best play is just to totally ignore it. Right,
Speaker 2: You know that they are trying to sweet talk you
Speaker 2: into doing exactly what they want you to do, and
Speaker 2: so the best play is to disengage or literally you
Speaker 2: can randomize your movis videos no notion of what the
Speaker 2: game theoretical optimal play might be, right, or salesmen or
Speaker 2: politicians have reputations for being Oh he's a little too smooth.
Speaker 2: Gavin newsome a little too fucking smooth. Right, I don't
Speaker 2: find Gavin news some persuasive at all. Right, use a
Speaker 2: little too from the hair gel to the constantly shifting vibes.
Speaker 2: I mean, I don't really find Gavin news some persuasive
Speaker 2: at all, even though like an AI, I'd say, boy,
Speaker 2: Gavin new some good looking guys look gravelly throated. But
Speaker 2: you know whatever, I mean. Look, the big critique I
Speaker 2: have of this project, and by the way, I think
Speaker 2: this is an amazing project in addition to like wonderful
Speaker 2: writing if you viewed on the web not your phone.
Speaker 2: So these very cool, like little infographic that updates everything
Speaker 2: from like the market value. If they don't call it
Speaker 2: open aye, they call it open brain. I guess is
Speaker 2: what they settle on for a substitute.
Speaker 3: Yeah, they call everything something else, just to make sure
Speaker 3: that they're not stepping on any toes. So they have
Speaker 3: open Brain, and then they have Deep Scent from China.
Speaker 2: I wonder which one that could be. I wonder. But
Speaker 2: it's beautifully presented and written, and like I appreciate they're
Speaker 2: going out on a limb here, you know, I mean,
Speaker 2: I think they have. It's been fairly well received. They've
Speaker 2: gotten some pushback both from inside and I think outside
Speaker 2: the AI safety community. Right, but they're putting their necks
Speaker 2: in the line, hear. They will look if things look
Speaker 2: pretty normal, if looks pretty normal in twenty thirty two
Speaker 2: or whatever, right, then they will look dumb for having
Speaker 2: published this.
Speaker 3: Well, and they actually have that right as a scenario
Speaker 3: that you end up looking stupid if everything goes well.
Speaker 3: But that's okay. Now, can I push back on the
Speaker 3: persuasion thing a little bit, just on two things. So
Speaker 3: first of all, the poker example is not actually a
Speaker 3: particularly applicable one here, because you know that you're playing
Speaker 3: poker and you know that someone is trying to get
Speaker 3: information and deceive you the tricky thing. So this is
Speaker 3: kind of when I spend time with con artists. The
Speaker 3: best con artists aren't Gavin Newsome, Like, they're not car salesmen.
Speaker 3: You have no idea they're trying to persuade you to
Speaker 3: do something. They are just like nice, affable people who
Speaker 3: are incredibly charismatic, and even in the poker community, by
Speaker 3: the way, like some of the biggest grifters who like
Speaker 3: it comes out later on, we're just like stealing money
Speaker 3: and doing all of these things. Are charming, right, They're
Speaker 3: not sleazy looking like they have no signs of, oh,
Speaker 3: I'm a salesman. I'm trying to sell you something. The
Speaker 3: people who are actually good at persuasion you do not
Speaker 3: realize you are being persuaded. And I think people are
Speaker 3: incredibly easy to kind of to subtly lead in a
Speaker 3: certain direction if you know how to do it, and
Speaker 3: I think ais could do that, and they might persuade
Speaker 3: you when you don't even think they're trying to persuade you.
Speaker 3: You might just ask like, can you please summarize this
Speaker 3: research report and the way that it frames it right,
Speaker 3: the way that it summarizes it just subtly changes the
Speaker 3: way that you think of this issue. We see that
Speaker 3: in psych studies all the time, by the way, where
Speaker 3: you have different articles presented in slightly different orders, slightly
Speaker 3: different ways, and people from the same political beliefs, you know,
Speaker 3: same starting point, come away with different impressions of what
Speaker 3: kind of the right course of action is or what
Speaker 3: this is actually trying to tell you. Because the way
Speaker 3: the information is presented actually influences how you think about it.
Speaker 3: It's very very easy to do subtle manipulations like that.
Speaker 3: And if we're relying on AI on a large scale
Speaker 3: for a lot of our lives, I think that if
Speaker 3: it has like a quote unquote master plan, you know,
Speaker 3: the way that they present in this report, then persuasion
Speaker 3: in that sense is actually going to be.
Speaker 2: You'll know you're being manipulated, right, that's the issue.
Speaker 3: No, you don't know. That's the thing.
Speaker 2: People will be manipulated because it's AI and that I
Speaker 2: don't but I don't know.
Speaker 3: I honestly, like Nate, I applaud your belief in human's
Speaker 3: ability to adjust to this, but I don't know that
Speaker 3: they will because I've just seen enough people who are
Speaker 3: incredibly intelligent fall for cons and then be very unpersuadable
Speaker 3: that they have been conned, right, instead doubling down and
Speaker 3: saying no, I have not. So humans are stubborn, but
Speaker 3: they're also stubborn and saying I have not been deceived,
Speaker 3: I have not been manipulated, when in fact, they have
Speaker 3: to protect their ego and to protect their view of
Speaker 3: themselves as people who are not capable of being manipulated
Speaker 3: or deceived, and I think that that is incredibly powerful,
Speaker 3: and I think that that's going to push against your optimism.
Speaker 3: I hope you're right, but from what I know, I
Speaker 3: don't think you are.
Speaker 2: I'm not quite sure you call it optimism some I
Speaker 2: guess maybe we do like different views of human nature.
Speaker 2: But like there's not yet a substantial market for like
Speaker 2: AI driven art we're writing, and I'm sure there will
Speaker 2: be one eventually, right, But like people understand that context matters, right,
Speaker 2: that you could heavy I create a rip off the
Speaker 2: Mona Lisa, but you can also buy rip off the
Speaker 2: Mona Lisa and Canal Street for five bucks, right, and
Speaker 2: like it. You know, So it's it's the intentionality of
Speaker 2: the act and the context of the speaker. Now, sound
Speaker 2: like super woke, I guess right where you're coming from,
Speaker 2: A I think that actually is how humans communicate. Like
Speaker 2: art that might be pointless drivel coming from somebody can
Speaker 2: be something to coming from a Jackson Ballock or whatever.
Speaker 1: You know.
Speaker 3: Absolutely, I think that that's a really important point. By
Speaker 3: the way, I think it's a different point, but I
Speaker 3: think that that is a very important point. I think
Speaker 3: context does matter.
Speaker 2: We'll be right back after this message. I was buttering
Speaker 2: up this report before. My big critique of it is like,
Speaker 2: where are the human beings in this? Or put another way,
Speaker 2: kind of like where is the politics? Right? They're trying
Speaker 2: not to use any remotely controversial real names, right, so
Speaker 2: you have open brain for example, So where's where is
Speaker 2: President Trump? Let me steal a quick search to make
Speaker 2: sure their named Trump does not appear to know the name.
Speaker 3: So they do. Actually, I don't know if this existed.
Speaker 3: Maybe they took your criticism in this, but they do
Speaker 3: have like the vice president and the president like they
Speaker 3: do put politicians in this version of the report. They
Speaker 3: don't have names, but they say the vice president in
Speaker 3: one of these scenarios, you know, handily wins the election
Speaker 3: of twenty twenty eight. We have one vice president.
Speaker 2: They have general secretary. I think I'm not sure if
Speaker 2: they I mean general secretary. General secretary resembles she yep,
Speaker 2: and the vice president kind of resembles jd Vance, Right,
Speaker 2: I don't think the president resembles Trump at all.
Speaker 3: Right, it's kind of the no, they didn't character like, yeah,
Speaker 3: they tried to side stuff that if.
Speaker 2: We think it's all happening in the next four years then,
Speaker 2: you know, presidential politics matter quite a bit. I mean,
Speaker 2: I know, I h this is such a fucking you know,
Speaker 2: I was jogging earlier on the East Side and I
Speaker 2: was listening to the Sraclient interview with Thomas Friedman. It's
Speaker 2: such a fucking yuppie fucking thing, right, It's.
Speaker 3: Okay, It's okay, You're allowed to be a.
Speaker 2: YOUNGESTI even not a huge fan of necessarily, but like,
Speaker 2: you know, as well versed some like geopolitics and like
Speaker 2: China issues, is like, yeah, China, You've just been back
Speaker 2: from Chinese. Like, yeah, China's kind of winning, you know
Speaker 2: what I mean. And like, I'm not sure how how
Speaker 2: Trump's hawkishness on China, but like kind of imbecilically executed
Speaker 2: hawkishness on China, Like I'm not sure how that figure's
Speaker 2: in to this. Right, If we're reducing US China trade,
Speaker 2: that probably does produce an AI slow down, maybe more
Speaker 2: for US if we're like if they're not exporting their
Speaker 2: like raw earth materials and and so forth, but we're
Speaker 2: making it hard for them to get in video chips,
Speaker 2: so they probably have like lots of workarounds and things
Speaker 2: like that. Maybe maybe Trump tariffs are good. I would
Speaker 2: like to ask the authors this report, because it means
Speaker 2: that we're gonna like have slower AI progress. And I'm
Speaker 2: not joking, right, that increases the hostility between the US
Speaker 2: and China in the long run, right, I mean that's
Speaker 2: if we were send it all the terrafs. I think
Speaker 2: we still permanently or let's not say permanent, let's say
Speaker 2: at least for a decade or so, have injured us
Speaker 2: standing in the world. And so I don't know how
Speaker 2: that figures in. And I'm like, I'm also not sure
Speaker 2: like kind of quote what the rational response might be.
Speaker 2: But one thing they tried to let me make sure
Speaker 2: that they kept us into their report, right, So they
Speaker 2: actually have their implied approval rating for how people feel
Speaker 2: about open brain, which is there not very self open AI.
Speaker 2: I think this actually is some feedback who that they
Speaker 2: took into account. Right. They originally had it slightly less negative,
Speaker 2: but they have this being persistently negative and then getting
Speaker 2: more negative over time. It was a little softer in
Speaker 2: the in the previous version that that I saw, so
Speaker 2: they did change that one thing on at some stage.
Speaker 2: But like the fact that like AI scares people, it
Speaker 2: scares people for both good and bad reasons, but I
Speaker 2: think mostly for valid reasons, right, that the fear is
Speaker 2: fairly bipartisan. That the biggest AI accelerators are now these
Speaker 2: kind of Republican techno optimists who are not looking particularly
Speaker 2: wise given how it's going with the first ninety days
Speaker 2: whatever we are, the Trump administration, and the likelihood of
Speaker 2: like a substantial political backlash, right, which could lead to
Speaker 2: dumb types of regulations. But like, you know, part of
Speaker 2: it too is like okay, AI they're saying can do
Speaker 2: not just computer disk jobs, but like all types of things, right,
Speaker 2: And like humans kind of play this role initially as supervisors,
Speaker 2: and then literally within a couple of years people start
Speaker 2: to say, you know what, am I really adding much
Speaker 2: value here? Right? You kind of have like these legacy jobs,
Speaker 2: and there is a lot of money. I think most
Speaker 2: these scenarios imagine very fast economic grow although maybe very lumpy,
Speaker 2: right for some parts of the world and not others.
Speaker 2: We're kind of just sitting around with a lot of
Speaker 2: idle time. It might be good for live poker, Maria. Right,
Speaker 2: all of a sudden, all these smart people, their open earth,
Speaker 2: their open brain, excuse me, stock is now worth billions
Speaker 2: of dollars, right, and like nothing to do because the
Speaker 2: AI is doing all their work, right, they have a
Speaker 2: lot of fucking time to play some fucking texas hold
Speaker 2: them right.
Speaker 3: That is, that is the one way of thinking about it.
Speaker 3: Let's go back to your earlier point, which I actually
Speaker 3: think is an important one, because obviously they were trying
Speaker 3: to do as all you know, super forecasting tries to
Speaker 3: do is you try to create a report that will
Speaker 3: work in multiple scenarios.
Speaker 2: Right.
Speaker 3: You can't tie it too much to like the present moment,
Speaker 3: otherwise your forecasts are going to be quite biased. However,
Speaker 3: I do think that what you raise kind of our
Speaker 3: current situation with et cetera, has very real implications. Given
Speaker 3: that this is kind of the central dynamic of this
Speaker 3: report that their predictions are based on, I think that
Speaker 3: it's incredibly valid to actually speculate, you know, how will
Speaker 3: if at all, this effect the timeline of the predictions,
Speaker 3: the possible the likelihood of the two scenarios. And I
Speaker 3: will also say that one of the things in the
Speaker 3: report is that all of these negotiations on like will
Speaker 3: we slow down, will we not? How aligned is it?
Speaker 3: This all takes place in secret, right, Like we don't
Speaker 3: know the humans don't know that it's going on. We
Speaker 3: don't know what's happening behind the scenes, and we don't
Speaker 3: know what the decision makers are kind of thinking. And
Speaker 3: so for all we know, you know, President Trump is
Speaker 3: meeting with Sam Altman and trying to trying to kind
Speaker 3: of do some of these things. And it's funny because
Speaker 3: we were kind of pushing for transparency in one way,
Speaker 3: but there's a lot of things here that are very
Speaker 3: much not transparent.
Speaker 2: Yeah, it's kind of the deep state, right, but also
Speaker 2: also a lot than negotiations are now AI versus AI, right,
Speaker 2: And look, I'm not sure that AIS will have that
Speaker 2: trust both with the external actor and internally. I'm skeptical
Speaker 2: of that. Right. If that does happen, they kind of
Speaker 2: think this might be good because the AIS will probably
Speaker 2: behave in like it literally game theory optimal way, right
Speaker 2: and understand these things and make I guess, like fewer
Speaker 2: mistakes than humans might say, if they're properly aligned.
Speaker 3: Like that's a crucial thing because in the doomsday scenario,
Speaker 3: AI negotiates with AI, but they conspire to destroy humanity. Right,
Speaker 3: So there are two scenarios. One, it's actually properly aligned,
Speaker 3: so AI negotiates with AI. Game theory works out, and
Speaker 3: we end up with you know, democracy and wonderful things.
Speaker 3: But in the other one, where they're misaligned, AI negotiates
Speaker 3: with AI to create a new AI basically and destroy humanity.
Speaker 3: So it can go one way or the other depending
Speaker 3: on that alignment step.
Speaker 2: First of all, I mean, the utopian didn't seem that
Speaker 2: utopian to me, right, I'm not sure that no, it
Speaker 2: did it.
Speaker 3: It actually seemed quite distolling to me, Like it seems
Speaker 3: incredibly disturbed. It's kind of like, you know, I look,
Speaker 3: but at least we're still alive, we'll have chures.
Speaker 2: Will probably live longer. And again, lots of lots of
Speaker 2: lots of poker the ail you're writing Silver Bulletin and
Speaker 2: and hosting our podcast. Right, let me let me back
Speaker 2: up a little bit, because I think we maybe take
Speaker 2: for granted that some of these premises are are kind
Speaker 2: of controversial, right, so they have a breakpoint. I think
Speaker 2: in twenty twenty six, well says Art, why aren't certainly
Speaker 2: increased potentially be on twenty twenty six, right, So that's
Speaker 2: kind of the breakpoint. Is like twenty seven is this
Speaker 2: inflection point? I think I'm using that term correctly in
Speaker 2: this context. You know. So I'm reading this report and
Speaker 2: up to twenty twenty six and like, thumbs up, yia,
Speaker 2: this seems like very smart and detailed about, like, you know,
Speaker 2: how the economy is reacting and how politics are reacting
Speaker 2: in the in the race dynamic with China. Maybe there
Speaker 2: needs to be a little bit more Trump in there.
Speaker 2: I understand why politically they didn't want to get into
Speaker 2: that mess, right, But like, so there's kind of three
Speaker 2: different things here, right. One is a notion of what's
Speaker 2: sometimes called AGI, or artificial general intelligence. And if you
Speaker 2: ask a hundred different researcher you get a hundred different
Speaker 2: definitions of what AGI is. But you know, I think
Speaker 2: it is based like being able to do a large
Speaker 2: majority of things that a human being could do competently,
Speaker 2: something we're limiting it to kind of like desk job
Speaker 2: type tasks. Right, anything that can be done remotely is
Speaker 2: sometimes definition that is used or through remote work, right,
Speaker 2: because clearly AIS are inferior to humans and like sorting
Speaker 2: and folding laundryth or things like that, right that require
Speaker 2: certain type of intelligence. Right, if you use the kind
Speaker 2: of desk job definition, then like AI is already pretty
Speaker 2: close to AGI. Right. I use large language models all
Speaker 2: the freaking time, and they're not perfect for everything. I
Speaker 2: felt like, you know, in terms of like being able
Speaker 2: to do the large majority of desk work at levels
Speaker 2: ranging from competent intern to super genius, Like on average,
Speaker 2: it's probably pretty close to being generally intelligent by that definition, right,
Speaker 2: if you're the.
Speaker 3: One using it. I just want to like once again
Speaker 3: point that out because one of the things that they
Speaker 3: say in the report is that as it gets more
Speaker 3: and more involved what we're asking AI to do, it's
Speaker 3: like the human process to evaluate whether it's accurate and
Speaker 3: whether it's making mistakes will get longer and longer. And
Speaker 3: I think they say, like for every like basically one
Speaker 3: day of work, it'll take several it's like a two
Speaker 3: to one ratio at the beginning for how long it
Speaker 3: will take humans to verify the output. Right, So you think, like,
Speaker 3: you think you save time by having AI do this,
Speaker 3: but if you want it to actually develop correctly, then
Speaker 3: you need a team and it takes them twice as
Speaker 3: long to verify that. What the AI did is actually
Speaker 3: true and actually valid and actually aligned, et cetera, et cetera.
Speaker 3: Now you're not asking it to do things that require
Speaker 3: that amount of time, but there do need to be
Speaker 3: little caveats to how we how we think about their
Speaker 3: usefulness and how you are able to evaluate the output
Speaker 3: versus Southern scenarios.
Speaker 2: When I use AI the things it's best with our
Speaker 2: things that like save any time. Right where I feeded
Speaker 2: a bunch of different names for different college basketball teams
Speaker 2: were working in our NCAA model, I'm like, take these
Speaker 2: seven different naming conventions that are all different and create
Speaker 2: a cross reference table of these, which is kind of
Speaker 2: like a hard task. You need to have a little
Speaker 2: context about basketball. And it did that very well. Right,
Speaker 2: That's something I could have done. It might have taken
Speaker 2: an hour or two, but you know, instead, instead it
Speaker 2: could do it. It's in a few minutes, and it gets faster.
Speaker 2: It's like, oh, I've learned from this from you before, Nate,
Speaker 2: so now I can be faster doing this type of
Speaker 2: task in the future. I was at the poker tournament
Speaker 2: down in Florida last week and like, uh, you know,
Speaker 2: I ask open research. Excuse me, God, is that like.
Speaker 3: See exactly right. It all because deep research, Deep research,
Speaker 3: Deep research, I asked, deep reach, eating too many of
Speaker 3: these twenty twenty seven reports, brand gets.
Speaker 2: To pull a bunch of stock market data for me.
Speaker 2: And then I'm playing at poker hand and like and
Speaker 2: make a really thin sexy value bet with like fourth pair.
Speaker 2: No one knows that means right, I bet a very weekend.
Speaker 2: I thought the other guy would call it with an
Speaker 2: even weaker hand, and I was right. And I feel
Speaker 2: like I'm such a fucking stud here valuating fourth pair,
Speaker 2: And well AI does the work for me, and then
Speaker 2: of course I like bust out of the tournament an
Speaker 2: hour later, and meanwhile, you know, deep research bungles this
Speaker 2: particular task. But in general AI has been very reliable.
Speaker 2: But the point is that like there's like a a
Speaker 2: inflection point where like I'm asking you to do things
Speaker 2: that like are just a faster version of what I
Speaker 2: could do myself. I would at the moment I ask, aid, be, like,
Speaker 2: I want you design a new NCA model for me
Speaker 2: with these parameters, because like I wouldn't know how to
Speaker 2: test it. But anyway, I mean long wanted to hear,
Speaker 2: so AGI, we're going to get AGI, or at least
Speaker 2: we're going to get someone calling something AGI soon. Right,
Speaker 2: artificial super intelligence where it's doing things much better than
Speaker 2: human beings. I think this report takes for granted or
Speaker 2: not takes for granted. It has lots of documentation about
Speaker 2: its assumptions, but it's saying, Okay, this trajectory has been
Speaker 2: very robust so far, and people make all types of
Speaker 2: bullshit predictions. Of the fact that these guys in particular
Speaker 2: have made accurate predictions in the past is certainly worth something,
Speaker 2: I think, Right, But they're like, Okay, you kind of
Speaker 2: follow the scaling law, and before too much longer, you know,
Speaker 2: AI starts to be more intelligent than human beings. You
Speaker 2: can debate what intelligent means if you want, but do
Speaker 2: superhuman types of things or and or do them very fast,
Speaker 2: you thing might be different, right, And and I can
Speaker 2: do things very fast. Is maybe it's certainly a component
Speaker 2: of intelligence, right, But like, but I don't take for
Speaker 2: granted that like quote unquote AI I can reliably extrapolate
Speaker 2: beyond the data set. I just think that, like, it's
Speaker 2: not an absurdly pavologic It may even be like the
Speaker 2: base case are close to the base case, but like
Speaker 2: that's not assumable from first principles. I don't think we've
Speaker 2: all seen lots of trend charts of the you know,
Speaker 2: if you look at a chart of Japan's GDP in
Speaker 2: the nineteen eighties, you might have said, Okay, well, Japan's
Speaker 2: gonna take over the world, and being people bought this
Speaker 2: and now it's kind of hasn't grown for like forty
Speaker 2: years basically, right, And so like we've all seen lots
Speaker 2: of curves that go up and then it's actually an
Speaker 2: estra where the fuck you call it? Where like it
Speaker 2: begins to bend the other way at some point and
Speaker 2: and we can't tell until later. Right. The other thing
Speaker 2: is like the abillity of AI to plan and manipulate
Speaker 2: the physical world. I mean some of these things where
Speaker 2: they're talking about, like you know, brain uploading and dice
Speaker 2: and swarms and nanobots like you know there, I would
Speaker 2: literally wager money against this happening on the time scales
Speaker 2: that they're talking about. Right, they double the timescale, Okay,
Speaker 2: then I might start to give some more probability. And look,
Speaker 2: I'm willing to be wrong about that. I guess we'll
Speaker 2: I'll be dead anyway, fifty percent likely in this scenario,
Speaker 2: but like.
Speaker 3: This scenario fifty percent to do like AA.
Speaker 2: You know, the physical world requires sensory input and lots
Speaker 2: of parts of our brain that AI is not as
Speaker 2: effective at manipulating. It also requires like being given or
Speaker 2: commandeering resources somehow. By the way, this is like a
Speaker 2: little bit of a problem for the United States. I mean,
Speaker 2: we are behind China by like quite a bit in
Speaker 2: robotics and related things, right, So, like, I don't know
Speaker 2: what happens if like we have the brainier, smarter AIS,
Speaker 2: but like they're very good at manufacturing machinery. So like,
Speaker 2: what if we have the brains and they have the
Speaker 2: brawn so to speak, right, and they have maybe a
Speaker 2: more authoritarian but functional infrastructure. So I don't know what
Speaker 2: happens then, right, Like, but the ability of AIS to
Speaker 2: comment to your resources to control the physical world, to
Speaker 2: me seems far fetched on these timelines, in part because
Speaker 2: of politics, right, I mean, the fact that it takes
Speaker 2: so long to build a new building in the US
Speaker 2: or a new subway station or a new highway, and
Speaker 2: the fact that our politics is kind of sclerotic. Right,
Speaker 2: And look, I mean I don't want to sound like
Speaker 2: too pessimistic, but if you read the book I recommended
Speaker 2: last week, fight, I mean, we basically did not have
Speaker 2: a fully competent president for the last two years, and
Speaker 2: I would argue that we don't have one for the
Speaker 2: next four years. Right, So like all these kind of
Speaker 2: things that we have to plan for, like who's doing
Speaker 2: that fucking planning? Our government's kind of dysfunctional, and you know,
Speaker 2: maybe that means we just lose to China, right, Maybe
Speaker 2: that means we lose to China, at least we'll have
Speaker 2: like nice cars. I guess.
Speaker 3: We'll be back right after this. I think that your
Speaker 3: point about the growth trajectories not necessarily being reliable is
Speaker 3: a very valid one. The Japanic example is great. You know,
Speaker 3: malfusion population growth is another is another big one. Right,
Speaker 3: we thought that population would explode and instead we're actually
Speaker 3: seeing population decline, So you know, the world does change.
Speaker 3: The thing that I think they rely on is that
Speaker 3: the AIS are capable of designing just this incredible technology
Speaker 3: much more quickly, so that our building process and all
Speaker 3: of that gets sped up one hundredfold from what it
Speaker 3: is right now. But it's still at least at this
Speaker 3: point needs humans to implement it right, and needs all
Speaker 3: of these different workers. And so yeah, I think there's
Speaker 3: some assumptions built into here that I hope, like I
Speaker 3: hope that that timeline isn't feasible, and I do think
Speaker 3: that there are things that are holding us back all
Speaker 3: the same. I think it's really I think it's interesting.
Speaker 3: One of the reasons I like this report is that
Speaker 3: it forces you to think about these things right and
Speaker 3: try to game out some of these worst case scenarios
Speaker 3: to try to prevent them, which I think is always
Speaker 3: an important thought exercise. I do want to go back
Speaker 3: to kind of their good scenario, which just so bad
Speaker 3: scenario is, you know, we're all wiped out by a
Speaker 3: chemical warfare that the AI is release on us. Good
Speaker 3: scenario is that you know, everyone gets a universal basic
Speaker 3: income and AI does everything and no one has to
Speaker 3: do anything and we can just live, you know, happily
Speaker 3: with Maria at play poker. Yeah, and that just as
Speaker 3: you suggest it, that seems actually like a very dystopian
Speaker 3: scenario where people can become much more easy to brainwash, control,
Speaker 3: et cetera, et cetera. It's like a dumbing town, right
Speaker 3: where where we're not challenged to produce good art to
Speaker 3: advance in any sort of way. Just to me, it
Speaker 3: does not seem like a very meaningful The.
Speaker 2: Question is there's not way another if you read AI
Speaker 2: twenty twenty seven, which I highly recommend that you read it.
Speaker 2: There's also another post by a pseudonymous poster called L.
Speaker 2: Rudolph L who wrote something called A History of the
Speaker 2: Future twenty twenty five to twenty forty, which is very
Speaker 2: detailed but goes through kind of like what this looks
Speaker 2: like at like more of a human level, how society evolves,
Speaker 2: how the economy evolves, how work evolves, right, and like
Speaker 2: very detailed, just like AI twenty twenty seven. Is that
Speaker 2: kind of focused on the parts of A twenty seven,
Speaker 2: then I think it kind of deliberately ignores maybe can
Speaker 2: call them mild blind spots or whatever, right, But like,
Speaker 2: but that's interesting because that kind of thinks about, like
Speaker 2: what types of jobs are there in the future. There
Speaker 2: a probably lots of lawyers actually, right, because you know,
Speaker 2: the law is very sluggish to change, especially in a
Speaker 2: constitutional system where there are lots of vitail points. Right,
Speaker 2: probably high end service sector. You know, you go to
Speaker 2: a restaurant because everyone's for a lot of people are
Speaker 2: rich now, right, and you're flattered by the attractive young
Speaker 2: server and things like that is kind of highly kind
Speaker 2: of like catered and curated experiences. I guess I have
Speaker 2: some faith in humanities abilieved to fight back quote unquote
Speaker 2: against like two scenarios that it might not really like
Speaker 2: either one, you know what I mean. And like the
Speaker 2: scenario where like AI is producing like ten percent GDP
Speaker 2: growth or or whatever. Right, man, it's great a few
Speaker 2: own stocks that are exposed to AI and tech companies probably, right,
Speaker 2: but it's also making that money on the backs of
Speaker 2: mass job displacement, and like, you know, it kind of
Speaker 2: a cert of confident in the long run that human
Speaker 2: beings fine productive things to do. And you know, massive
Speaker 2: employment has been predicted many times and never really occurred, right,
Speaker 2: But like, but it's not occurring this fast where they
Speaker 2: think the world ends in six years or whether they're
Speaker 2: predicting or we have utopia in six years, and like
Speaker 2: just the ability of like human society to like deal
Speaker 2: with that change at these times scales leads to like
Speaker 2: more chaos than I think. They're more predicting. But I
Speaker 2: think also and I told them this too, right, I
Speaker 2: think also leads to more constraints. Right that you hit bottlenecks.
Speaker 2: If you have five things you have to do, right
Speaker 2: and you have the world's fastest computer, et cetera, et cetera.
Speaker 2: But there's like a power outage in your neighborhood, right,
Speaker 2: and that's a that's a bottle deck. Right, Maybe there
Speaker 2: are ways around it. Youre like go to home depot
Speaker 2: and buy a generator or you know what I mean.
Speaker 2: But like, but the point is that like you're often
Speaker 2: defined by like the slowest link, and politics are sometimes
Speaker 2: the slowest link, but also by like Also, you know,
Speaker 2: I think the report maybe under states, and I think
Speaker 2: kind of in general the a saify community like maybe
Speaker 2: understates the ability of like human beings to cause harm
Speaker 2: to other human beings with AI, Right, that concern kind
Speaker 2: of gets brushed off, is like to pedestrian or like
Speaker 2: say to pedestri And I was saying, yeah.
Speaker 3: The exact word I was thinking of. I think that's
Speaker 3: a good I mean, I think that's a great place
Speaker 3: to end it, because yes, we do need to be
Speaker 3: concerned about all of these things about AI. But like
Speaker 3: that phrase I think is very crucial, like do not
Speaker 3: underestimate the ability of humans to cause harm to other humans.
Speaker 3: And I think that that's you know, it's not a
Speaker 3: very optim it's not a very pleasant place to end,
Speaker 3: but I think it's a really important place to end.
Speaker 3: And I think that that's a very valid kind of
Speaker 3: way of reflecting on this name.
Speaker 2: Or to trust AIS too much, right, I generally think
Speaker 2: that concern is like somewhat misplays but like if we're
Speaker 2: handing over critical systems to AI, right, it can cause
Speaker 2: problems if it's very smart and deceives us and doesn't
Speaker 2: like us very much. It also cause problems if it
Speaker 2: has hallucinations, bugs in critical areas where it isn't as
Speaker 2: robust and hasn't really been tested yet that are outside
Speaker 2: of its domain yep, or there could be espionage. Anyway,
Speaker 2: we will have plenty of time, although maybe only seven
Speaker 2: more years actually to like explore expore these scenarios.
Speaker 3: Yes, and in seven years we'll be like, welcome back
Speaker 3: to the final episode of Risky Business, because the prediction
Speaker 3: is we're all going to be done tomorrow. But yeah,
Speaker 3: this was an interesting exercise, and I think my my
Speaker 3: p doom has slightly gone up as a result of
Speaker 3: reading this, but I also remain optimistic that humans can
Speaker 3: can do good as well as harm.
Speaker 2: Yeah, my interest in learning Chinese has increased as a
Speaker 2: result of recent developments I know about my pe doome.
Speaker 3: All right, Yeah, let's let's do some language emmergis and
Speaker 3: I'm with you. That's it for today. If you're a
Speaker 3: premium subscriber, we will be answering a question about whether
Speaker 3: mface can ever be plus ev right after the credits.
Speaker 3: And if you're not a subscriber, it's not too late.
Speaker 3: For six ninety nine a month the price of a midber,
Speaker 3: you get access to all these conversations and all premium
Speaker 3: content across the Pushkin Network. Risky Business is hosted by
Speaker 3: me Maria Kannakova.
Speaker 2: And by me Nate Silver. The show is a co
Speaker 2: production of Pushkin Industries and iHeartMedia. This episode was produced
Speaker 2: by Isabel Carter. Our associate producer is Sonia Gerwit. Sally
Speaker 2: helm is our editor, and our executive producer is Jacob Goldstein.
Speaker 2: Mixing by Sarah Bruger. If you like the show, please
Speaker 2: rate and review us. You know we like we take
Speaker 2: a four or five. We take the five. Rate and
Speaker 2: review us. To other people, Thank you for listening.