1
00:00:15,604 --> 00:00:28,764
Speaker 1: Pushkin. Welcome back to Risky Business, a show about making

2
00:00:28,844 --> 00:00:31,484
Speaker 1: better decisions. I'm Maria Kanakova.

3
00:00:31,204 --> 00:00:32,284
Speaker 2: And I'm Nate Silver.

4
00:00:32,844 --> 00:00:35,324
Speaker 3: Today on the show, we're gonna be talking about how

5
00:00:35,524 --> 00:00:38,164
Speaker 3: chat GPT plays poker, i e.

6
00:00:38,364 --> 00:00:38,764
Speaker 2: Poorly.

7
00:00:38,924 --> 00:00:40,964
Speaker 3: If chat gipt is getting ready for the World Series

8
00:00:40,964 --> 00:00:43,124
Speaker 3: of Poker like us, then it's got a lot of

9
00:00:43,124 --> 00:00:43,764
Speaker 3: homework to do.

10
00:00:44,084 --> 00:00:48,924
Speaker 1: Yes. And after we talk about that, we'll talk about

11
00:00:48,924 --> 00:00:53,924
Speaker 1: Harvard two point zero aka the fact that foreign students

12
00:00:54,004 --> 00:00:59,804
Speaker 1: are now being potentially barred from attending Harvard University. Harvard

13
00:00:59,884 --> 00:01:02,764
Speaker 1: has sued the administration. Anyway, we'll be talking about what's

14
00:01:02,804 --> 00:01:06,284
Speaker 1: going on with that and what the implications are for

15
00:01:06,364 --> 00:01:10,764
Speaker 1: the future of the United States and kind of research development,

16
00:01:11,124 --> 00:01:21,684
Speaker 1: brain power and the US competitive edge. So for people

17
00:01:21,764 --> 00:01:25,044
Speaker 1: who don't subscribe to Silver Bulletin, which you absolutely should,

18
00:01:25,204 --> 00:01:27,364
Speaker 1: Nate and I both had poker posts this last week.

19
00:01:27,524 --> 00:01:32,004
Speaker 1: Mine was about cheating, was about chat GPT, but his

20
00:01:32,124 --> 00:01:34,924
Speaker 1: post this week was one of the funniest things I've

21
00:01:34,964 --> 00:01:39,804
Speaker 1: read in a while about his attempts to get chat

22
00:01:39,844 --> 00:01:45,244
Speaker 1: GPT to simulate a hand of cash game poker. And

23
00:01:46,244 --> 00:01:48,284
Speaker 1: one of the reasons I mean, I found this amusing

24
00:01:48,324 --> 00:01:51,244
Speaker 1: because I mean, first of all, it fused cards together

25
00:01:51,284 --> 00:01:53,804
Speaker 1: for an image. From the beginning to the end, it

26
00:01:53,924 --> 00:01:56,924
Speaker 1: was pretty spot on in terms of being spot on wrong.

27
00:01:57,124 --> 00:02:00,084
Speaker 1: But Nay, You're also someone who is constantly writing about

28
00:02:00,084 --> 00:02:03,004
Speaker 1: how good AI is at so many things, and so

29
00:02:03,524 --> 00:02:08,084
Speaker 1: it was funny to me to see it fall so

30
00:02:08,644 --> 00:02:14,084
Speaker 1: short of something that actually tests intelligence as opposed to

31
00:02:14,764 --> 00:02:17,604
Speaker 1: being able to kind of pull together a lot of

32
00:02:17,644 --> 00:02:19,204
Speaker 1: things and spit out an answer.

33
00:02:19,724 --> 00:02:21,284
Speaker 3: There are some reasons why I'm just sid in testing

34
00:02:21,404 --> 00:02:25,124
Speaker 3: chatchipt on poker, but they kind of fall into two

35
00:02:25,204 --> 00:02:29,244
Speaker 3: big buckets. Right, What is it like poker played in

36
00:02:29,324 --> 00:02:31,844
Speaker 3: the context of a real hand or in this case,

37
00:02:31,884 --> 00:02:35,604
Speaker 3: a real text based simulation I guess of a hand.

38
00:02:36,004 --> 00:02:40,884
Speaker 3: It really does require quite a few skills, right. There

39
00:02:40,964 --> 00:02:44,484
Speaker 3: is the pure math part of it. What is the equilibrium,

40
00:02:44,524 --> 00:02:47,964
Speaker 3: the Nash equilibrium, the gto strategy that you're solving for.

41
00:02:48,484 --> 00:02:52,244
Speaker 3: There's also making adjustments for how other players play. There's

42
00:02:52,284 --> 00:02:55,884
Speaker 3: the conversation you're having, the physical reads that you're getting

43
00:02:56,164 --> 00:02:58,884
Speaker 3: their stuff, like knowing what the rules are right, which

44
00:02:59,524 --> 00:03:03,284
Speaker 3: might seem trivial, but we've yeah, chatchpt also.

45
00:03:03,124 --> 00:03:03,804
Speaker 1: Does not know.

46
00:03:03,924 --> 00:03:06,164
Speaker 2: It seems to be no, it is fun to look.

47
00:03:06,204 --> 00:03:07,804
Speaker 3: I mean, I'm sure we've all had hands where we

48
00:03:07,924 --> 00:03:10,644
Speaker 3: like misread a board or I you know, I had

49
00:03:10,684 --> 00:03:13,324
Speaker 3: a big hand in the main event at the when

50
00:03:13,404 --> 00:03:15,604
Speaker 3: I play back in Florida back in April or whatever,

51
00:03:15,644 --> 00:03:18,484
Speaker 3: right where like a guy had like a twenty five

52
00:03:18,484 --> 00:03:20,524
Speaker 3: thousand dollars chip that was like almost the same color

53
00:03:20,524 --> 00:03:23,524
Speaker 3: as the felt, which is not good chipkit maintenance, by

54
00:03:23,564 --> 00:03:25,964
Speaker 3: the way, Seminal hard Rock. I like the WPT, but

55
00:03:26,524 --> 00:03:29,044
Speaker 3: that chip kit it's got to be improved, right, But yeah,

56
00:03:29,044 --> 00:03:32,564
Speaker 3: you're tired, you make mistakes like that. But and you

57
00:03:32,604 --> 00:03:36,884
Speaker 3: also have to have like a lot of short term memory,

58
00:03:37,644 --> 00:03:40,004
Speaker 3: which sounds trivial, but you you know, you want to

59
00:03:40,004 --> 00:03:44,404
Speaker 3: remember how you arrived at this current spot in the hand, right,

60
00:03:44,804 --> 00:03:47,004
Speaker 3: you have to keep crack of all these stack sizes,

61
00:03:47,084 --> 00:03:50,524
Speaker 3: which again seems trivial, but like it's a good test

62
00:03:50,564 --> 00:03:55,404
Speaker 3: of general intelligence of a certain type, especially a live

63
00:03:55,524 --> 00:03:57,004
Speaker 3: poker hand, like I'm asking.

64
00:03:56,804 --> 00:03:58,444
Speaker 2: Chat ept to simulate.

65
00:03:58,644 --> 00:04:00,844
Speaker 3: Right. The other reason is that like I don't think

66
00:04:00,924 --> 00:04:06,404
Speaker 3: engineers in Silicon Valley are trying to optimize it for poker.

67
00:04:07,404 --> 00:04:09,844
Speaker 3: And the reason that's important is because like, look, there

68
00:04:09,884 --> 00:04:14,364
Speaker 3: are benchmarks like math Olympiad problems that these LM's large

69
00:04:14,404 --> 00:04:17,884
Speaker 3: language models like CHATCHPT, Claude, et cetera compete on, right,

70
00:04:17,924 --> 00:04:19,884
Speaker 3: And they're like, well, we be bragging that we can now,

71
00:04:20,124 --> 00:04:23,804
Speaker 3: you know, beat all but the Nobel Prize mathematicians on

72
00:04:24,044 --> 00:04:26,244
Speaker 3: x percentage of math problems and things like that. And

73
00:04:26,284 --> 00:04:29,004
Speaker 3: there are a couple of issues with that, right. Like

74
00:04:29,044 --> 00:04:33,924
Speaker 3: one is that, like, clearly, if you train a transformer model,

75
00:04:33,964 --> 00:04:37,284
Speaker 3: a machine learning model on a particular type of problem,

76
00:04:37,804 --> 00:04:42,084
Speaker 3: like it can do fairly well or very well often, right,

77
00:04:42,164 --> 00:04:45,884
Speaker 3: Like poker is to a first approximation, and it's an

78
00:04:45,884 --> 00:04:49,884
Speaker 3: important approximation. Right, But if you have poker dedicated tools,

79
00:04:50,244 --> 00:04:53,844
Speaker 3: I wouldn't consider a solver and AI that's a technical

80
00:04:53,844 --> 00:04:56,484
Speaker 3: distinction I think might not be that important for our listeners.

81
00:04:56,484 --> 00:04:58,484
Speaker 3: But like, you know, if you want to use computers

82
00:04:58,484 --> 00:05:01,444
Speaker 3: to play very good poker, then they can play very

83
00:05:01,524 --> 00:05:04,644
Speaker 3: very good poker. Right. The question is like, can you

84
00:05:04,764 --> 00:05:09,204
Speaker 3: take a text based model and have this organic property

85
00:05:09,644 --> 00:05:12,924
Speaker 3: where intelligence emerges and converts from the data set towards

86
00:05:12,964 --> 00:05:16,604
Speaker 3: superintelligence without training it on poker specifically. And the answer,

87
00:05:16,604 --> 00:05:18,804
Speaker 3: at least as of last week when I did this,

88
00:05:18,924 --> 00:05:20,564
Speaker 3: is no, it fails miserably.

89
00:05:22,004 --> 00:05:25,164
Speaker 1: Yeah, and I think it's actually important. One of the

90
00:05:25,204 --> 00:05:27,884
Speaker 1: things that came to my mind when I was reading

91
00:05:27,924 --> 00:05:32,164
Speaker 1: your piece was that poker has been a benchmark for

92
00:05:32,724 --> 00:05:37,324
Speaker 1: AI development well before LMS, Right, It has been kind

93
00:05:37,324 --> 00:05:40,524
Speaker 1: of the gold standard that teams all over the world

94
00:05:41,244 --> 00:05:46,004
Speaker 1: have been working on for decades because it is a

95
00:05:46,124 --> 00:05:51,004
Speaker 1: much more complex game in many senses than chess, then

96
00:05:51,084 --> 00:05:55,724
Speaker 1: even go, and it's a game where, you know, if

97
00:05:55,764 --> 00:05:59,244
Speaker 1: computers can actually manage to outthink human players on a

98
00:05:59,284 --> 00:06:02,724
Speaker 1: broader scale, that would mean something in terms of kind

99
00:06:02,724 --> 00:06:08,564
Speaker 1: of broader intelligence capabilities. And before LMS were developed, that

100
00:06:08,604 --> 00:06:12,524
Speaker 1: had not happened. Right, There had been computer programs, kind

101
00:06:12,524 --> 00:06:16,284
Speaker 1: of AI algorithms that have been able to beat heads

102
00:06:16,404 --> 00:06:20,004
Speaker 1: up one on one opponents in poker. But when it

103
00:06:20,044 --> 00:06:22,404
Speaker 1: came to full ring games, So for people who don't

104
00:06:22,444 --> 00:06:24,444
Speaker 1: know poker, that means basically, when you put it in

105
00:06:24,484 --> 00:06:27,724
Speaker 1: a rich environment with you know, six players, eight players

106
00:06:28,364 --> 00:06:31,644
Speaker 1: and you have a computer there, it was still not

107
00:06:31,924 --> 00:06:34,884
Speaker 1: quite there. And one of the you know, one of

108
00:06:34,924 --> 00:06:37,924
Speaker 1: the reasons why poker so interesting is it's not just math, right,

109
00:06:38,004 --> 00:06:41,324
Speaker 1: especially when you have multiple players. There's so many dynamics

110
00:06:41,364 --> 00:06:45,924
Speaker 1: and there's actually a very thorny problems for AI, which

111
00:06:46,004 --> 00:06:51,084
Speaker 1: is that if you program something to follow an algorithm

112
00:06:51,164 --> 00:06:55,644
Speaker 1: right to be GTO game theory optimal, then it can

113
00:06:55,764 --> 00:06:58,364
Speaker 1: be exploited by a human if the human figures out

114
00:06:58,404 --> 00:07:01,484
Speaker 1: the GTO strategy. Now, if it were another human, then

115
00:07:01,564 --> 00:07:04,964
Speaker 1: you'd adjust immediately right the moment the human adjusts, the

116
00:07:05,084 --> 00:07:07,524
Speaker 1: other human would adjust and kind of figure it out.

117
00:07:07,964 --> 00:07:11,004
Speaker 1: But if you're an a if you're a computer, and

118
00:07:11,044 --> 00:07:15,124
Speaker 1: you build in that adjustment parameter, what ends up happening

119
00:07:15,164 --> 00:07:17,724
Speaker 1: is that the model adjusts too much and too quickly,

120
00:07:18,124 --> 00:07:22,524
Speaker 1: and so that lacks kind of some of the nuance

121
00:07:22,764 --> 00:07:26,484
Speaker 1: and flexibility that marks that's kind of the hallmark of

122
00:07:26,484 --> 00:07:29,604
Speaker 1: the great poker players, right, kind of that ability to

123
00:07:30,244 --> 00:07:34,444
Speaker 1: into it when it's time to slightly deviate. And I mean,

124
00:07:34,484 --> 00:07:39,444
Speaker 1: it would be terrifying but amazing if a model could spontaneously,

125
00:07:39,844 --> 00:07:43,124
Speaker 1: like an LLM, could actually spontaneously figure that out and

126
00:07:43,324 --> 00:07:46,484
Speaker 1: so far, I mean, instead instead of doing that, Nate,

127
00:07:46,804 --> 00:07:49,484
Speaker 1: you know, your model decided that the person who won

128
00:07:49,524 --> 00:07:52,404
Speaker 1: the hand actually lost the hand, right, It just got

129
00:07:52,524 --> 00:07:57,244
Speaker 1: basic things completely wrong, and you just you know, obviously

130
00:07:57,284 --> 00:08:00,044
Speaker 1: we've talked about pe doom a lot on the show.

131
00:08:00,404 --> 00:08:02,444
Speaker 1: I think you and I are both worried about that.

132
00:08:02,684 --> 00:08:05,244
Speaker 1: But when I see this poker stuff and like, oh man.

133
00:08:05,844 --> 00:08:08,844
Speaker 3: Yeah, I don't know how much to update. But look

134
00:08:08,884 --> 00:08:13,524
Speaker 3: my say in the post, I mean, like my appreciation

135
00:08:13,684 --> 00:08:18,124
Speaker 3: and expectations for large language models have increased a lot

136
00:08:18,164 --> 00:08:22,484
Speaker 3: over the past year. I had a pretty good experience

137
00:08:22,524 --> 00:08:24,484
Speaker 3: working with them when I was building the NCAA tournament

138
00:08:24,484 --> 00:08:27,044
Speaker 3: model that we host silver bulletin and just helpful kind

139
00:08:27,084 --> 00:08:31,164
Speaker 3: of Swiss army knives or like annoying data related tasks.

140
00:08:32,444 --> 00:08:35,764
Speaker 3: It's late at night, you forget the Stata command for something,

141
00:08:35,764 --> 00:08:37,684
Speaker 3: You're like, right, give me the right command and write

142
00:08:37,684 --> 00:08:39,724
Speaker 3: five lines of code, and like usually it works, and

143
00:08:39,764 --> 00:08:41,964
Speaker 3: it kind of now shows you like the chain of

144
00:08:42,004 --> 00:08:46,604
Speaker 3: reasoning and like what it's thinking. Right, And but you know,

145
00:08:46,644 --> 00:08:49,164
Speaker 3: people are saying that we're going to have like AI

146
00:08:50,044 --> 00:08:55,044
Speaker 3: revolutionizing all programming or you know, well, first of all,

147
00:08:55,044 --> 00:08:57,244
Speaker 3: the big claim is going to have AGI, meaning that

148
00:08:57,364 --> 00:09:01,164
Speaker 3: a machine. It's a little ambiguous when people say, does

149
00:09:01,204 --> 00:09:05,964
Speaker 3: this mean like chat GPT itself is AGI? Right, Well,

150
00:09:06,044 --> 00:09:10,564
Speaker 3: clearly not. It's very far from human being in some things, right,

151
00:09:11,044 --> 00:09:14,844
Speaker 3: Does that mean that a combination of transformer or machine

152
00:09:14,884 --> 00:09:17,644
Speaker 3: learning based technologies is or zooming out the definition of AI.

153
00:09:17,724 --> 00:09:21,124
Speaker 3: It's really AI still, right, so that you have chechiptiza

154
00:09:21,164 --> 00:09:25,164
Speaker 3: seventy percent of things well, and then you specialized applications

155
00:09:25,204 --> 00:09:28,964
Speaker 3: for twenty percent, and then there are ten percent where

156
00:09:28,964 --> 00:09:31,684
Speaker 3: you're not doing it very well. Like that would still

157
00:09:31,724 --> 00:09:34,204
Speaker 3: be a big achievement, very disruptive achievement. I'm not sure

158
00:09:34,204 --> 00:09:37,044
Speaker 3: it would kind of quite count as AGI. Like one

159
00:09:37,044 --> 00:09:38,844
Speaker 3: thing you could do is, like some of these new

160
00:09:39,684 --> 00:09:44,524
Speaker 3: models will say, okay, I can detect that you ask

161
00:09:44,644 --> 00:09:49,124
Speaker 3: for something that requires mathematical precisions. So let's take a

162
00:09:49,124 --> 00:09:53,324
Speaker 3: simpler example. Right, Let's say I want to take the

163
00:09:53,564 --> 00:10:00,004
Speaker 3: distance between pick two cities, Boise, Idaho and Tampa, Florida. Right,

164
00:10:00,684 --> 00:10:04,164
Speaker 3: that's pretty easy to calculate mathematically if you look up

165
00:10:04,684 --> 00:10:07,844
Speaker 3: the latlong coordinates for Boise and Tampa, and then there

166
00:10:07,884 --> 00:10:09,604
Speaker 3: are various formulas that.

167
00:10:09,604 --> 00:10:10,204
Speaker 2: You can use.

168
00:10:10,324 --> 00:10:13,964
Speaker 3: Right, So a pure data set where it's just crunching

169
00:10:14,044 --> 00:10:17,164
Speaker 3: texts might not find that answer because there's not necessarily

170
00:10:17,164 --> 00:10:20,404
Speaker 3: a lot of examples of like what's the distance between

171
00:10:20,404 --> 00:10:22,964
Speaker 3: Boise and Tampa? Right, Whereas you would have that for

172
00:10:23,124 --> 00:10:26,644
Speaker 3: like La and new York or something. However, you can

173
00:10:26,684 --> 00:10:31,164
Speaker 3: set up kind of agentic meaning agent ais where they're like, okay,

174
00:10:31,444 --> 00:10:34,084
Speaker 3: well this person asked a question that triggered a routine

175
00:10:34,084 --> 00:10:36,404
Speaker 3: of mine. Where now I'm going to go and I

176
00:10:36,484 --> 00:10:39,284
Speaker 3: know to search the internet for the coordinates or maybe

177
00:10:39,324 --> 00:10:41,964
Speaker 3: it's stored somewhere right of Boise and Tampa. And then

178
00:10:42,004 --> 00:10:44,044
Speaker 3: the formula. I know the formula and I can apply it, right,

179
00:10:44,124 --> 00:10:46,604
Speaker 3: so like you have the scaffolding. It's sometimes called a

180
00:10:46,644 --> 00:10:49,924
Speaker 3: different processes on top of one another. And I guess

181
00:10:49,964 --> 00:10:51,324
Speaker 3: that kind of I guess that counts as I mean,

182
00:10:51,324 --> 00:10:54,244
Speaker 3: it's kind of how humans solve problems, right, They're like, oh,

183
00:10:54,244 --> 00:10:56,484
Speaker 3: and now I have to go look something up, right,

184
00:10:56,924 --> 00:11:00,924
Speaker 3: It's still I think subtly different. So one implication I

185
00:11:00,924 --> 00:11:09,164
Speaker 3: think this has is artificial general intelligence versus super intelligence ASI,

186
00:11:09,484 --> 00:11:14,084
Speaker 3: sometimes called where there's like an explosion of intelligence just

187
00:11:14,124 --> 00:11:16,684
Speaker 3: from kind of mining text or other data sources. I

188
00:11:17,724 --> 00:11:20,484
Speaker 3: have actually become a little bit more skeptical of that,

189
00:11:20,764 --> 00:11:24,564
Speaker 3: or I think it's presumptive to assume it can find that.

190
00:11:24,644 --> 00:11:26,524
Speaker 3: I mean, like, let me give you another example, right,

191
00:11:26,604 --> 00:11:30,204
Speaker 3: like one I had an idea for either a silver

192
00:11:30,244 --> 00:11:33,324
Speaker 3: bulletin post or maybe even a risky business segment, right

193
00:11:33,324 --> 00:11:38,204
Speaker 3: where like I asked, chatchipt deep research, design a meal

194
00:11:39,324 --> 00:11:46,964
Speaker 3: with foods that have no clear precedent in other countries, right, huh?

195
00:11:46,444 --> 00:11:48,964
Speaker 3: And I thought, and the idea was that, like, you know,

196
00:11:49,004 --> 00:11:53,724
Speaker 3: at some point somebody invented pizza or spaghetti or whatever else, right,

197
00:11:53,884 --> 00:11:54,684
Speaker 3: or or I.

198
00:11:54,604 --> 00:11:57,684
Speaker 1: Mean at some point somebody invented bread and figured out

199
00:11:57,724 --> 00:12:00,404
Speaker 1: that you can take flour and like bake shit with it.

200
00:12:00,404 --> 00:12:00,884
Speaker 1: It's amazing.

201
00:12:01,004 --> 00:12:03,484
Speaker 3: Yes, you have to be able to buy these ingredientstead

202
00:12:03,484 --> 00:12:06,844
Speaker 3: of Trader Joe's or a Whole Foods. Right. I was

203
00:12:06,844 --> 00:12:08,924
Speaker 3: just going to like hire professional chef and have a

204
00:12:08,924 --> 00:12:11,764
Speaker 3: dinner part and ask how these creations went, right, And

205
00:12:11,804 --> 00:12:14,844
Speaker 3: it's like and all they recipes were like here's salmon

206
00:12:14,964 --> 00:12:18,524
Speaker 3: with miso paste, right, really creative and things like that, right,

207
00:12:18,564 --> 00:12:21,484
Speaker 3: and like it didn't have any ability to like extrapolate

208
00:12:21,524 --> 00:12:25,404
Speaker 3: beyond the data set. I'm sure these preparations are fine.

209
00:12:25,444 --> 00:12:28,644
Speaker 3: It was like their little chefy, meaning like things that

210
00:12:28,804 --> 00:12:31,524
Speaker 3: like have one too many ingredients, right, and like and

211
00:12:31,564 --> 00:12:34,884
Speaker 3: I don't, yeah, look so bad chefe.

212
00:12:35,164 --> 00:12:36,004
Speaker 2: Yeah.

213
00:12:36,084 --> 00:12:38,604
Speaker 3: And meanwhile, I've been like I've been interviewing candidates for

214
00:12:38,684 --> 00:12:43,484
Speaker 3: like a sports assistant position at Silver Bulletin, and one

215
00:12:43,524 --> 00:12:46,484
Speaker 3: of the questions is, how are you using AI? These

216
00:12:46,484 --> 00:12:48,644
Speaker 3: are very bright I want to say kids. A lot

217
00:12:48,684 --> 00:12:53,284
Speaker 3: of them are are recent college grads, you know, your

218
00:12:53,284 --> 00:12:56,804
Speaker 3: early twenties to mid thirties, basically right, and very technically

219
00:12:56,884 --> 00:13:00,204
Speaker 3: proficient people. And asked them how they're using a GI

220
00:13:00,284 --> 00:13:03,724
Speaker 3: and their experiences like kind of similar to mine. They're like, Yeah,

221
00:13:03,924 --> 00:13:07,804
Speaker 3: for a certain task, it's very helpful. For certain task,

222
00:13:07,844 --> 00:13:10,284
Speaker 3: it's somewhat helpful for certain tasks, not helpful at all.

223
00:13:10,284 --> 00:13:14,644
Speaker 3: I can get you in trouble, right, which again contrast

224
00:13:14,684 --> 00:13:16,404
Speaker 3: with the experience of people at the AI l apps

225
00:13:16,404 --> 00:13:19,644
Speaker 3: to say, yeah, it's going to like displace all programming

226
00:13:20,204 --> 00:13:21,164
Speaker 3: within a year or two.

227
00:13:21,524 --> 00:13:23,364
Speaker 2: I mean, I know you should weigh in here.

228
00:13:23,644 --> 00:13:27,484
Speaker 1: So I have an interesting kind of anecdote to add

229
00:13:27,524 --> 00:13:30,764
Speaker 1: to this that's not poker related, but that talks about

230
00:13:30,844 --> 00:13:35,204
Speaker 1: kind of some of this nuance and complexity and the

231
00:13:35,284 --> 00:13:39,124
Speaker 1: fact that you know that all of these lms still

232
00:13:39,364 --> 00:13:43,524
Speaker 1: are wanting in certain major respects. So I had the

233
00:13:43,604 --> 00:13:46,764
Speaker 1: chance last week to speak to the CEO of one

234
00:13:46,804 --> 00:13:50,684
Speaker 1: of the major AI companies. I won't name which company

235
00:13:50,724 --> 00:13:56,164
Speaker 1: it was, but one of the big ones. And someone

236
00:13:56,204 --> 00:14:02,284
Speaker 1: had asked him earlier, what was something that was surprising

237
00:14:02,484 --> 00:14:05,444
Speaker 1: about the way that this AI functioned, and he said,

238
00:14:05,484 --> 00:14:10,164
Speaker 1: you know, one of the mistakes that people is kind

239
00:14:10,204 --> 00:14:12,924
Speaker 1: of they trust it, And what we know is that

240
00:14:13,044 --> 00:14:16,844
Speaker 1: it's remarkably accurate ninety percent of the time, right, and

241
00:14:16,884 --> 00:14:19,444
Speaker 1: then ten percent of the time it isn't. But the

242
00:14:19,484 --> 00:14:23,124
Speaker 1: ten percent is getting harder and harder for humans to

243
00:14:23,204 --> 00:14:26,324
Speaker 1: spot because it doesn't make errors in a way that's

244
00:14:26,364 --> 00:14:30,324
Speaker 1: intuitive for people, right. It makes errors in different ways.

245
00:14:30,604 --> 00:14:33,964
Speaker 1: It's like computer errors AI errs, that's not the way

246
00:14:33,964 --> 00:14:36,524
Speaker 1: that the human mind makes errors. He's like, but like,

247
00:14:36,564 --> 00:14:38,884
Speaker 1: it's fine because we have really smart people who can

248
00:14:38,924 --> 00:14:41,844
Speaker 1: figure out where it's making errors and can work with

249
00:14:41,924 --> 00:14:44,324
Speaker 1: it and can kind of calibrate it and help. And

250
00:14:44,404 --> 00:14:47,844
Speaker 1: so my question, my follow up question, was, Okay, you

251
00:14:47,884 --> 00:14:52,204
Speaker 1: know what happens when the new generation that's being brought

252
00:14:52,284 --> 00:14:56,244
Speaker 1: up with this AI comes up and they've been educated

253
00:14:56,244 --> 00:14:58,244
Speaker 1: with it and they've been using it the whole time,

254
00:14:58,604 --> 00:15:02,724
Speaker 1: and they didn't necessarily get the same education that we got, right,

255
00:15:02,804 --> 00:15:06,324
Speaker 1: because they aren't the incentives are different, and they're not

256
00:15:06,364 --> 00:15:09,764
Speaker 1: getting the deep knowledge. They're not actually able to figure

257
00:15:09,764 --> 00:15:12,124
Speaker 1: out what are the programming errors because this person was

258
00:15:12,164 --> 00:15:14,404
Speaker 1: talking that, you know, some of the biggest potential of

259
00:15:14,404 --> 00:15:18,044
Speaker 1: AIS and like programming and biology and kind of those

260
00:15:18,244 --> 00:15:20,364
Speaker 1: those types of things, and I said, well, what if

261
00:15:20,364 --> 00:15:23,404
Speaker 1: that person doesn't have the neuroscience and the biology background

262
00:15:23,444 --> 00:15:26,964
Speaker 1: or the programming background. And what the CEO said was,

263
00:15:27,164 --> 00:15:29,884
Speaker 1: this is a major problem, and what we're hoping is

264
00:15:29,884 --> 00:15:32,804
Speaker 1: that we can develop even more advanced AI to help

265
00:15:33,484 --> 00:15:36,604
Speaker 1: and to help fix the problems that AI is causing.

266
00:15:36,724 --> 00:15:38,484
Speaker 1: Because he was like, yeah, this is a big problem

267
00:15:38,524 --> 00:15:41,324
Speaker 1: and we're not quite sure, like we're hoping for the best,

268
00:15:41,364 --> 00:15:44,564
Speaker 1: but we don't know. And the fact that this was

269
00:15:44,604 --> 00:15:49,284
Speaker 1: our conversation didn't exactly inspire me to heights of thinking

270
00:15:49,644 --> 00:15:54,484
Speaker 1: this is going to replace intelligence or become AGI or

271
00:15:54,564 --> 00:16:00,084
Speaker 1: what did you call, Nate the other ASI artificial super intelligent? Yeah,

272
00:16:00,164 --> 00:16:04,084
Speaker 1: artificial superintelligence. So it didn't inspire me that that was

273
00:16:04,124 --> 00:16:06,844
Speaker 1: going to happen, and instead it was like an uh

274
00:16:06,844 --> 00:16:11,084
Speaker 1: oh moment where like, what happened when the humans who

275
00:16:11,084 --> 00:16:14,084
Speaker 1: are kind of helping push it along and make it better?

276
00:16:14,604 --> 00:16:18,524
Speaker 1: When that generation, like when they age out and they're

277
00:16:18,884 --> 00:16:22,604
Speaker 1: new people out there who who lack that sort of

278
00:16:22,644 --> 00:16:26,284
Speaker 1: skill and expertise and who grew up with AI that

279
00:16:26,724 --> 00:16:29,724
Speaker 1: is ninety percent accurate or even ninety five percent accurate,

280
00:16:29,964 --> 00:16:32,844
Speaker 1: but they can't. It's harder and harder to spot that

281
00:16:32,964 --> 00:16:37,004
Speaker 1: five and ten percent. And that to me was actually, like,

282
00:16:37,284 --> 00:16:39,604
Speaker 1: it's a worrisome thing, and it's clearly worrisome to the

283
00:16:39,604 --> 00:16:43,964
Speaker 1: people who are developing these technologies as well. And so

284
00:16:44,844 --> 00:16:47,844
Speaker 1: that I think is something that dovetails with what you

285
00:16:47,884 --> 00:16:50,644
Speaker 1: were talking about here. What happens Nate if all of

286
00:16:50,644 --> 00:16:54,284
Speaker 1: a sudden, all poker players are training with chatchipt right,

287
00:16:54,564 --> 00:16:58,524
Speaker 1: and that's that's actually great for us, but not great

288
00:16:58,564 --> 00:16:59,484
Speaker 1: for the future of poker.

289
00:17:02,804 --> 00:17:14,924
Speaker 3: And we'll be right back after this break. I use

290
00:17:15,564 --> 00:17:19,924
Speaker 3: large language models for tasks that I have strong domain

291
00:17:20,204 --> 00:17:23,444
Speaker 3: knowledge over and could do myself. But they're a labor

292
00:17:23,484 --> 00:17:28,724
Speaker 3: saving device. Often they say substantial labor, right, Like I

293
00:17:28,804 --> 00:17:31,164
Speaker 3: use them to copy edit articles I posted the newsletter,

294
00:17:31,204 --> 00:17:33,884
Speaker 3: and I you know, I'm a fairly proficient user of

295
00:17:33,884 --> 00:17:36,844
Speaker 3: the English language and like for programming, right Like you know,

296
00:17:37,084 --> 00:17:40,924
Speaker 3: if you're like using AI to like design a website

297
00:17:41,764 --> 00:17:44,964
Speaker 3: and some of the functionality breaks, right, that might be okay.

298
00:17:45,124 --> 00:17:47,284
Speaker 2: You can patch it and fix it. Right, if I'm

299
00:17:47,324 --> 00:17:48,124
Speaker 2: doing things.

300
00:17:47,884 --> 00:17:52,764
Speaker 3: Involving modeling, I'm trying to project the value of basketball

301
00:17:52,804 --> 00:17:55,884
Speaker 3: teams or football teams, or basketball players or football players. Right,

302
00:17:56,364 --> 00:17:58,164
Speaker 3: and there's a bug where all of a sudden, like

303
00:17:58,284 --> 00:18:01,164
Speaker 3: one of the worst players in the league is rated

304
00:18:01,164 --> 00:18:03,084
Speaker 3: as being very good. Right. I mean, and you see

305
00:18:03,084 --> 00:18:06,684
Speaker 3: this in poker, where like you know, AI is relying.

306
00:18:06,764 --> 00:18:11,684
Speaker 3: You get punished in poker for sloppy application of imprecise

307
00:18:11,724 --> 00:18:15,484
Speaker 3: application of heuristics. Like one of the things like the

308
00:18:15,524 --> 00:18:18,324
Speaker 3: AI did so at first, I had it simulate just

309
00:18:18,444 --> 00:18:21,164
Speaker 3: one hand, and it had like a backstore for each player.

310
00:18:21,164 --> 00:18:22,684
Speaker 3: It fucked that up, right, then like Oka, I'm gonna

311
00:18:22,724 --> 00:18:25,164
Speaker 3: use deep research now and have it simulate like an

312
00:18:25,284 --> 00:18:26,244
Speaker 3: orbit of eight hands.

313
00:18:26,284 --> 00:18:27,124
Speaker 2: So that was a little better.

314
00:18:27,164 --> 00:18:30,004
Speaker 3: It took more compute time, right, Some of the hands

315
00:18:30,004 --> 00:18:32,524
Speaker 3: were decent, but it you know, it didn't know how

316
00:18:32,564 --> 00:18:35,644
Speaker 3: to like read aboard correctly and keep track of stacks.

317
00:18:35,684 --> 00:18:39,244
Speaker 3: But like, but when it was over able to overcome

318
00:18:39,284 --> 00:18:43,124
Speaker 3: those errors, it also had poor strategy and it gave

319
00:18:43,204 --> 00:18:45,724
Speaker 3: various excuses. One thing it said is like, well, the

320
00:18:45,884 --> 00:18:48,764
Speaker 3: quality of text based content on poker on the internet

321
00:18:49,444 --> 00:18:53,404
Speaker 3: really sucks, Right, it's aware of that is it says

322
00:18:53,444 --> 00:18:56,124
Speaker 3: it's worse than go or chess, although when I talk

323
00:18:56,124 --> 00:18:59,324
Speaker 3: to people who know how AI's played chess, it's also

324
00:18:59,404 --> 00:19:02,644
Speaker 3: a disaster. Even something like wordle I heard from a user.

325
00:19:02,724 --> 00:19:06,644
Speaker 3: You know you think wordle is a word game and

326
00:19:06,684 --> 00:19:08,884
Speaker 3: it does very well, but like it can't quite figure

327
00:19:08,884 --> 00:19:10,964
Speaker 3: out the strike sure of the problem, right, it kind

328
00:19:10,964 --> 00:19:13,244
Speaker 3: of was learning a little bit more by by a

329
00:19:13,364 --> 00:19:14,924
Speaker 3: rote And to be clear, like this is a very

330
00:19:14,964 --> 00:19:18,204
Speaker 3: good strategy for like many types of problems. And also

331
00:19:18,324 --> 00:19:20,524
Speaker 3: if you were to say, okay, here's a bunch of

332
00:19:20,604 --> 00:19:23,404
Speaker 3: we bought data from PIO solver or gto wizard. These

333
00:19:23,404 --> 00:19:27,084
Speaker 3: are solvers if you don't know listeners or a database

334
00:19:27,124 --> 00:19:28,924
Speaker 3: of high stakes hands. Like if you trained it on

335
00:19:28,964 --> 00:19:33,084
Speaker 3: that and then had GPTs say okay, poker question, I'm

336
00:19:33,124 --> 00:19:35,684
Speaker 3: going to call them especially trained database, then it would

337
00:19:35,724 --> 00:19:37,444
Speaker 3: do well. And maybe if people are criticizing its for

338
00:19:38,004 --> 00:19:41,524
Speaker 3: on poker, then the labs will start to do that. Right,

339
00:19:41,644 --> 00:19:44,764
Speaker 3: but it's still kind of like slightly cheating from the

340
00:19:44,804 --> 00:19:48,724
Speaker 3: standpoint of super intelligence. Also, it says like when you

341
00:19:48,724 --> 00:19:50,484
Speaker 3: asked me to do a whole bunch of things at once, right,

342
00:19:50,684 --> 00:19:52,844
Speaker 3: So I told it, give me characters, give me eight

343
00:19:52,924 --> 00:19:55,324
Speaker 3: characters who have like they were all kind of like

344
00:19:55,404 --> 00:19:57,324
Speaker 3: ethnic stereotypes, right, it was.

345
00:19:57,804 --> 00:19:59,604
Speaker 1: They were very funny.

346
00:20:01,364 --> 00:20:04,084
Speaker 3: It was kind of yeah and great. It was kind

347
00:20:04,124 --> 00:20:06,684
Speaker 3: of at the one hand, so half the players were women.

348
00:20:06,764 --> 00:20:08,804
Speaker 3: That was very feminist of it, right, But then they're

349
00:20:08,844 --> 00:20:12,364
Speaker 3: all ethnic stereotypes of different kinds, right, and so it

350
00:20:12,404 --> 00:20:16,844
Speaker 3: was kind of both woken and unwoke, right the opposite. Yeah,

351
00:20:17,364 --> 00:20:19,324
Speaker 3: it was kind of good about like matching players playing

352
00:20:19,364 --> 00:20:24,004
Speaker 3: styles with with their actions, but like, but it couldn't

353
00:20:24,084 --> 00:20:27,084
Speaker 3: keep track of stack size. It's like, yeah, keep track

354
00:20:27,084 --> 00:20:28,684
Speaker 3: of stacks. I have to store a bunch of stuff

355
00:20:28,684 --> 00:20:30,204
Speaker 3: in memory, and then when you ask me to do

356
00:20:30,244 --> 00:20:33,324
Speaker 3: a whole bunch of things at once, right, and like

357
00:20:33,684 --> 00:20:38,004
Speaker 3: you know, and even including like so in the hand

358
00:20:38,044 --> 00:20:40,924
Speaker 3: I show a sylver bulletin it like it misreads the board,

359
00:20:41,004 --> 00:20:43,844
Speaker 3: doesn't recognize that a pair of nines has a higher

360
00:20:43,844 --> 00:20:46,884
Speaker 3: two pair. If you ask it that out right, then

361
00:20:46,924 --> 00:20:49,804
Speaker 3: it thinks about it more and gets the question right, right,

362
00:20:49,844 --> 00:20:52,324
Speaker 3: But like it loses track of things in the context

363
00:20:52,844 --> 00:20:55,924
Speaker 3: of these scaffolding situations where it has to keep track

364
00:20:55,924 --> 00:20:57,524
Speaker 3: of a bunch of things at once and it doesn't

365
00:20:57,524 --> 00:21:00,964
Speaker 3: know it doesn't know what to prioritize right. Like you know,

366
00:21:01,044 --> 00:21:02,524
Speaker 3: you can make a lot of mistakes in poker if

367
00:21:02,524 --> 00:21:04,964
Speaker 3: you don't know which hand be twitch hand. That's a

368
00:21:05,004 --> 00:21:07,404
Speaker 3: more elementary mistake than anything else.

369
00:21:08,564 --> 00:21:10,764
Speaker 1: Absolutely, So there are a few things that stand out

370
00:21:10,764 --> 00:21:11,204
Speaker 1: to me here.

371
00:21:11,524 --> 00:21:11,804
Speaker 3: One.

372
00:21:12,164 --> 00:21:16,724
Speaker 1: I mean, it's a computer, right, like we humans have

373
00:21:16,844 --> 00:21:20,164
Speaker 1: working memory capacity problems. It should be able to keep

374
00:21:20,204 --> 00:21:22,284
Speaker 1: all these things like this is what it's good at.

375
00:21:22,604 --> 00:21:26,044
Speaker 1: But the problem that you're kind of that you're illustrating

376
00:21:26,124 --> 00:21:29,084
Speaker 1: is it doesn't know how to prioritize right, and that

377
00:21:29,084 --> 00:21:32,644
Speaker 1: that is actually that is a major problem. And also

378
00:21:32,884 --> 00:21:36,564
Speaker 1: when we keep saying thinking. But one of the first things,

379
00:21:36,724 --> 00:21:39,244
Speaker 1: so when I was just learning poker, one of the

380
00:21:39,244 --> 00:21:41,364
Speaker 1: first lessons I had with Phil Galfon, who is one

381
00:21:41,404 --> 00:21:43,004
Speaker 1: of the people who kind of I worked with and

382
00:21:43,044 --> 00:21:47,404
Speaker 1: who taught me a lot. I remember very early on

383
00:21:47,524 --> 00:21:49,444
Speaker 1: he said, you know, I can give you a bunch

384
00:21:49,484 --> 00:21:51,604
Speaker 1: of charts and a bunch of outputs, and you can

385
00:21:51,644 --> 00:21:55,204
Speaker 1: memorize them and you'll be a very decent player very quickly.

386
00:21:55,284 --> 00:21:56,804
Speaker 1: Like I know, I can give you this shit, you

387
00:21:56,844 --> 00:21:58,684
Speaker 1: can memorize it, you can spit it back at me.

388
00:21:58,804 --> 00:22:01,164
Speaker 1: You'll be fine, He's like, but I don't want you

389
00:22:01,204 --> 00:22:03,884
Speaker 1: to do that, because that's going to make you a

390
00:22:03,924 --> 00:22:06,364
Speaker 1: fine poker player, and you'll do fine in the short run,

391
00:22:06,444 --> 00:22:09,124
Speaker 1: but you'll never be a great poker player because you

392
00:22:09,204 --> 00:22:12,004
Speaker 1: have no idea why you're doing it. You're not actually thinking,

393
00:22:12,204 --> 00:22:15,964
Speaker 1: you don't understand. You've just done a rote memorization, which

394
00:22:16,084 --> 00:22:18,484
Speaker 1: is might make you more money in the short term,

395
00:22:18,564 --> 00:22:20,244
Speaker 1: but in the long term is actually going to be

396
00:22:20,284 --> 00:22:24,084
Speaker 1: detrimental to you because you're going to do stuff unthinkingly

397
00:22:24,204 --> 00:22:25,364
Speaker 1: because you've memorized it.

398
00:22:25,644 --> 00:22:25,964
Speaker 3: He said.

399
00:22:25,964 --> 00:22:28,804
Speaker 1: What I want you to do is instead think through

400
00:22:28,804 --> 00:22:33,004
Speaker 1: things and figure out, Okay, why right? For every single play,

401
00:22:33,524 --> 00:22:36,044
Speaker 1: why am I doing this? Why would I play this

402
00:22:36,204 --> 00:22:38,604
Speaker 1: hand and not this hand? Why would I raise this

403
00:22:38,684 --> 00:22:41,364
Speaker 1: type of hand and not this type of hand? Why?

404
00:22:41,484 --> 00:22:42,844
Speaker 3: Why? Why? Why? Why?

405
00:22:43,164 --> 00:22:46,244
Speaker 1: And that's something that to this day has stayed with

406
00:22:46,284 --> 00:22:49,204
Speaker 1: me because it's such an important thing to remember when

407
00:22:49,204 --> 00:22:51,924
Speaker 1: you're making a decision. Right, And this doesn't even have

408
00:22:51,964 --> 00:22:53,844
Speaker 1: to be poker, It can be any sort of decision.

409
00:22:54,324 --> 00:22:57,364
Speaker 1: Why am I choosing this action? Why is it better

410
00:22:57,404 --> 00:23:00,084
Speaker 1: than all of the other actions? Right? And if you

411
00:23:00,364 --> 00:23:05,364
Speaker 1: even if you feed you know, piosalvereign gto Wizard outputs

412
00:23:05,444 --> 00:23:11,044
Speaker 1: two lllms, they'll be doing some rote of memorization. You know,

413
00:23:11,444 --> 00:23:14,324
Speaker 1: at this point they're not understanding the why, and so

414
00:23:15,444 --> 00:23:18,924
Speaker 1: that will lead them still to make big mistakes, which

415
00:23:18,964 --> 00:23:22,524
Speaker 1: happens when you just learn solver outputs, no matter how

416
00:23:22,564 --> 00:23:26,004
Speaker 1: sophisticated those outputs might be and how correct they might

417
00:23:26,044 --> 00:23:29,684
Speaker 1: be in one specific hand, in one specific spot, if

418
00:23:29,764 --> 00:23:33,244
Speaker 1: you don't understand the why, you're either going to overgeneralize

419
00:23:33,284 --> 00:23:37,204
Speaker 1: it right or misapply it like you're going to screw

420
00:23:37,204 --> 00:23:40,484
Speaker 1: it up because you don't understand the underlying reasoning. And

421
00:23:40,844 --> 00:23:46,124
Speaker 1: there's the difference between looking like you're thinking and actually thinking.

422
00:23:46,604 --> 00:23:49,484
Speaker 1: And yeah, you know, as a human you have other pitfalls,

423
00:23:49,524 --> 00:23:52,164
Speaker 1: and even if you're you know, thinking through things, you

424
00:23:52,164 --> 00:23:54,164
Speaker 1: can mess up, So you mess up in different ways.

425
00:23:54,364 --> 00:23:57,524
Speaker 1: But I think that's a really important distinction and probably

426
00:23:57,564 --> 00:24:02,964
Speaker 1: one of the reasons why AGI is not as close

427
00:24:03,084 --> 00:24:08,324
Speaker 1: like poker. Actually illustrates in a very practical way, why

428
00:24:08,564 --> 00:24:10,524
Speaker 1: the notion that you can just have a lot of

429
00:24:10,604 --> 00:24:12,964
Speaker 1: data and all of a sudden have this you know,

430
00:24:13,404 --> 00:24:17,524
Speaker 1: flurry of insight might not be as I guess, as

431
00:24:17,564 --> 00:24:19,164
Speaker 1: intuitive as one might think.

432
00:24:19,124 --> 00:24:21,244
Speaker 3: When one thing, Chatchip you told me when I asked

433
00:24:21,284 --> 00:24:24,204
Speaker 3: to like audit itself, is like, well, poker's very difficult

434
00:24:24,204 --> 00:24:30,484
Speaker 3: because it's adversarial. You know, understand that there's some game

435
00:24:30,484 --> 00:24:33,644
Speaker 3: theory there, but it's adversarial, and like there's no one

436
00:24:33,724 --> 00:24:36,124
Speaker 3: I mean, there is like I guess some superstructure of

437
00:24:36,164 --> 00:24:39,164
Speaker 3: like a solution for all poker hands if you get

438
00:24:39,284 --> 00:24:42,364
Speaker 3: very zoomed out, right, but like but like subtle things

439
00:24:42,404 --> 00:24:47,764
Speaker 3: when you have to calculate like an adversarial equilibrium on

440
00:24:47,844 --> 00:24:52,404
Speaker 3: the fly basically, and it's that's very difficult.

441
00:24:52,524 --> 00:24:55,404
Speaker 1: Yeah, I mean think about not even poker, but think

442
00:24:55,404 --> 00:24:58,684
Speaker 1: about trying to do a game theory solution for multiple players.

443
00:24:59,004 --> 00:24:59,204
Speaker 3: Right.

444
00:24:59,244 --> 00:25:01,644
Speaker 1: It's hard enough trying to do a payoff matrix for

445
00:25:01,684 --> 00:25:04,724
Speaker 1: two players when you're really trying to think through it

446
00:25:04,884 --> 00:25:06,844
Speaker 1: and you're trying to think of all of the different

447
00:25:06,884 --> 00:25:09,364
Speaker 1: payoffs and figuring out, you know, how to exactly do

448
00:25:09,444 --> 00:25:12,284
Speaker 1: you wait them, how exactly do you structure that. Now

449
00:25:12,324 --> 00:25:15,684
Speaker 1: when it's three players, four players, I mean, it's incredibly

450
00:25:15,724 --> 00:25:19,604
Speaker 1: difficult to do that accurately. And so even if we

451
00:25:19,724 --> 00:25:22,044
Speaker 1: just zoom out from poker, like this is just an

452
00:25:22,124 --> 00:25:26,284
Speaker 1: incredible a very tough problem, even if you understand game theory.

453
00:25:26,524 --> 00:25:29,964
Speaker 1: Plus I mean, I think that anyone who has used solvers,

454
00:25:30,084 --> 00:25:33,004
Speaker 1: and anyone who's talked about this understands anyone who doesn't

455
00:25:33,004 --> 00:25:36,044
Speaker 1: even play poker but has worked with algorithms, the common

456
00:25:36,124 --> 00:25:39,404
Speaker 1: saying garbage in, garbage out is absolutely accurate.

457
00:25:39,484 --> 00:25:39,684
Speaker 3: Right.

458
00:25:40,124 --> 00:25:44,124
Speaker 1: The solver works based on what ranges of hands you

459
00:25:45,244 --> 00:25:48,604
Speaker 1: as a human put in there, right, and what responses

460
00:25:48,684 --> 00:25:51,924
Speaker 1: you allow or don't allow, right, And yes, it will

461
00:25:51,924 --> 00:25:55,924
Speaker 1: come up with an equilibrium strategy. But if you were wrong, right,

462
00:25:56,364 --> 00:25:59,084
Speaker 1: if there's a player who's playing a totally different range,

463
00:25:59,484 --> 00:26:01,924
Speaker 1: if there's a player who's playing a totally different strategy,

464
00:26:02,084 --> 00:26:04,924
Speaker 1: all of a sudden, your solution means nothing. And as

465
00:26:04,924 --> 00:26:07,604
Speaker 1: a human you can figure that out, and you can

466
00:26:07,684 --> 00:26:11,564
Speaker 1: kind of make adjustments from baseline, right, if you understand

467
00:26:11,564 --> 00:26:15,244
Speaker 1: what the baseline strategy as you can adjust as an LM,

468
00:26:15,564 --> 00:26:18,884
Speaker 1: as an AI who is learning that but doesn't kind

469
00:26:18,884 --> 00:26:21,844
Speaker 1: of have that nuanced experience at least at this point.

470
00:26:22,044 --> 00:26:24,604
Speaker 1: I don't pretend to know what AI is going to

471
00:26:24,604 --> 00:26:26,604
Speaker 1: be capable of in you know, in five years and

472
00:26:26,684 --> 00:26:30,804
Speaker 1: ten years, but at least at this point they're not

473
00:26:31,044 --> 00:26:39,444
Speaker 1: capable of making those sorts of nuanced extrapolations and figuring out, Okay,

474
00:26:39,524 --> 00:26:42,764
Speaker 1: what were my inputs accurate, right, or were my inputs

475
00:26:42,804 --> 00:26:43,524
Speaker 1: not quite accurate.

476
00:26:43,604 --> 00:26:45,844
Speaker 3: Look, one thing human beings are good at is that

477
00:26:45,884 --> 00:26:50,604
Speaker 3: human beings are relatively good estimators, and chatchipt can be

478
00:26:50,724 --> 00:26:53,004
Speaker 3: sometimes if it's like, Okay, take all this text on

479
00:26:53,044 --> 00:26:55,484
Speaker 3: the web and kind of give me the average of that,

480
00:26:55,604 --> 00:26:57,884
Speaker 3: Like there are some applications where it's pretty good. But like,

481
00:26:57,924 --> 00:27:01,164
Speaker 3: you know, the other day, I was getting getting a

482
00:27:01,244 --> 00:27:05,164
Speaker 3: drink with my partner and a guy who looked like

483
00:27:05,164 --> 00:27:07,724
Speaker 3: he was homeless comes in and like hands like the

484
00:27:08,044 --> 00:27:12,964
Speaker 3: hosts slash barked like a note saying call nine to

485
00:27:12,964 --> 00:27:15,804
Speaker 3: one one or something like that, right, and you kind

486
00:27:15,844 --> 00:27:18,804
Speaker 3: of have to make this judgment call about like is

487
00:27:18,844 --> 00:27:24,204
Speaker 3: this like a paranoid schizophrenic or is someone actually in danger?

488
00:27:24,404 --> 00:27:26,804
Speaker 3: And like I think was pretty clear. I mean they're

489
00:27:26,804 --> 00:27:30,804
Speaker 3: probably you know, two bartenders and eight customers clear to everybody.

490
00:27:30,804 --> 00:27:32,564
Speaker 3: Like this guy I think is not like acutely in

491
00:27:32,644 --> 00:27:35,764
Speaker 3: danger or anything. But I'd never quite experienced situation like

492
00:27:35,804 --> 00:27:38,004
Speaker 3: that before. And like the fact that we can use

493
00:27:38,044 --> 00:27:42,844
Speaker 3: like our general intelligence about like human behavior Yeah, Like

494
00:27:43,124 --> 00:27:44,964
Speaker 3: I think everyone made the right decision that we're not

495
00:27:45,004 --> 00:27:47,364
Speaker 3: going to call the police either to rat on him

496
00:27:47,444 --> 00:27:49,684
Speaker 3: or to say he was in danger. And like you know,

497
00:27:49,724 --> 00:27:52,364
Speaker 3: and that kind of thing is is is hard for

498
00:27:53,524 --> 00:27:55,684
Speaker 3: language yells to do, you know.

499
00:27:57,124 --> 00:27:59,044
Speaker 1: No, I mean I think in general, like this just

500
00:27:59,804 --> 00:28:03,004
Speaker 1: illustrates a really good point, which is that humans, even

501
00:28:03,204 --> 00:28:06,284
Speaker 1: not the smartest humans, are much smarter and much more

502
00:28:06,324 --> 00:28:09,044
Speaker 1: capable in very basic ways that we take for granted

503
00:28:10,004 --> 00:28:14,004
Speaker 1: than a lot of kind of super intelligences. Right, Like

504
00:28:14,724 --> 00:28:18,004
Speaker 1: it took forever for a robot to be able to

505
00:28:18,004 --> 00:28:20,564
Speaker 1: pick up an egg, right, It's something that we don't

506
00:28:20,604 --> 00:28:23,804
Speaker 1: even think about, but this was like a robotics problem

507
00:28:23,844 --> 00:28:27,324
Speaker 1: that was absolutely unsolvable, Like how do you get a

508
00:28:27,364 --> 00:28:30,364
Speaker 1: robot to pick up an egg without without breaking it?

509
00:28:30,724 --> 00:28:33,684
Speaker 1: And we do it without just without thinking about it.

510
00:28:33,964 --> 00:28:38,244
Speaker 1: And we make judgments all the time, just silly judgments

511
00:28:38,564 --> 00:28:41,204
Speaker 1: that we don't think twice about that are so easy

512
00:28:41,244 --> 00:28:44,204
Speaker 1: for the human mind, but that are that we don't

513
00:28:44,204 --> 00:28:49,084
Speaker 1: even realize we're making right things about safety, things about

514
00:28:49,164 --> 00:28:52,644
Speaker 1: just perceptions of the world. So I think that you know,

515
00:28:53,604 --> 00:28:57,124
Speaker 1: humans are much smarter than than we think in like

516
00:28:57,324 --> 00:29:00,804
Speaker 1: in dumb ways, if that makes sense, Like in waste

517
00:29:00,884 --> 00:29:03,244
Speaker 1: that seemed like they're not hard problems, but those are

518
00:29:03,284 --> 00:29:05,164
Speaker 1: actually some of the hardest problems to solve.

519
00:29:05,404 --> 00:29:05,924
Speaker 2: I just think of.

520
00:29:05,844 --> 00:29:08,884
Speaker 3: There being four levels, right, Like one more AI performance

521
00:29:08,924 --> 00:29:12,084
Speaker 3: better than any human, two where AI performs better than

522
00:29:12,204 --> 00:29:16,884
Speaker 3: like all but expert level humans, Three where AI gives

523
00:29:16,964 --> 00:29:20,164
Speaker 3: kind of a passable substitute performance but like you know,

524
00:29:20,844 --> 00:29:24,484
Speaker 3: not quite professional high grade level, and four where it's

525
00:29:24,524 --> 00:29:28,284
Speaker 3: just inept right to the point of being comically inept.

526
00:29:28,324 --> 00:29:31,364
Speaker 3: And like, you know, my heuristic is that like within

527
00:29:31,404 --> 00:29:34,604
Speaker 3: a few years we might have roughly an even divide

528
00:29:34,604 --> 00:29:37,364
Speaker 3: between those four and I'm counting, by the way, tasks

529
00:29:37,404 --> 00:29:39,684
Speaker 3: that involve manipulting the physical environment, which I think AI

530
00:29:39,724 --> 00:29:42,844
Speaker 3: will mostly be pretty bad at. Right, I don't know

531
00:29:42,884 --> 00:29:46,204
Speaker 3: how much I should extrapolate from the poker example. It

532
00:29:46,324 --> 00:29:52,364
Speaker 3: was just so incongruent with these predictions of like imminent AGI, right,

533
00:29:54,124 --> 00:29:57,364
Speaker 3: where a year ago would have said, okay, yeah, of

534
00:29:57,404 --> 00:29:59,324
Speaker 3: course suck at poker. Right, it's not really trained on

535
00:29:59,364 --> 00:30:01,524
Speaker 3: this and it's a hard problem, and ha ha ha, Right,

536
00:30:01,564 --> 00:30:05,124
Speaker 3: But like if you want to have these complicated, like structured,

537
00:30:05,364 --> 00:30:09,124
Speaker 3: nested tasks, like I'm much less worried now in periods

538
00:30:09,164 --> 00:30:15,124
Speaker 3: of oh, let's say five years of AI like being

539
00:30:15,164 --> 00:30:18,724
Speaker 3: able to build like atiscal model to like forecast elections

540
00:30:18,844 --> 00:30:21,444
Speaker 3: or the NFL or whatever else, because that like just

541
00:30:21,484 --> 00:30:25,484
Speaker 3: requires like a superstructure of lots of little tasks, all

542
00:30:25,484 --> 00:30:29,604
Speaker 3: of which are fuck upable, And if you get the

543
00:30:29,604 --> 00:30:33,044
Speaker 3: superstructure wrong too, then you're kind of just drawing dead.

544
00:30:33,084 --> 00:30:35,484
Speaker 3: To use a poker term, right if any if you

545
00:30:35,524 --> 00:30:38,404
Speaker 3: have to chain together twenty steps and any step, there's

546
00:30:38,444 --> 00:30:40,764
Speaker 3: a ninety percent chance you get it right. Well, point

547
00:30:40,884 --> 00:30:43,044
Speaker 3: nine to the twentieth power means you're almost.

548
00:30:42,804 --> 00:30:44,284
Speaker 2: Sure to fuck something up.

549
00:30:45,284 --> 00:30:48,644
Speaker 1: Yeah, fuck upable is a really great word to describe this.

550
00:30:48,924 --> 00:30:51,564
Speaker 1: Nate to kind of to sum this up, you did

551
00:30:51,604 --> 00:30:54,044
Speaker 1: ask chat GBT why it was bad at poker? What

552
00:30:54,564 --> 00:30:55,204
Speaker 1: did it tell you?

553
00:30:55,964 --> 00:30:58,364
Speaker 3: Yeah? No, I mean it's said various things. It said, Look,

554
00:30:58,364 --> 00:31:01,044
Speaker 3: you stressed me out by having me having me have

555
00:31:01,124 --> 00:31:02,684
Speaker 3: to make up these players and things like that.

556
00:31:03,044 --> 00:31:05,284
Speaker 1: Trust it out, You trusted.

557
00:31:06,804 --> 00:31:08,844
Speaker 3: It said the data that it trained on on the

558
00:31:08,844 --> 00:31:11,444
Speaker 3: internet is which seems realistic.

559
00:31:11,564 --> 00:31:14,124
Speaker 1: Yeah, it's a data problem. It's a yeah, it's not me.

560
00:31:14,164 --> 00:31:18,324
Speaker 2: It's you. It's it's complicated.

561
00:31:18,484 --> 00:31:22,604
Speaker 3: Inform about like how like it doesn't realize how important

562
00:31:22,724 --> 00:31:25,884
Speaker 3: stack sizes are because it's just a bunch of tokens.

563
00:31:25,924 --> 00:31:29,444
Speaker 3: Like all the input you give it is generated into tokens, right,

564
00:31:29,524 --> 00:31:32,084
Speaker 3: and like the word the is not very important. The

565
00:31:32,124 --> 00:31:34,524
Speaker 3: fact that Maria has fifty two thousand dollars in chips

566
00:31:35,244 --> 00:31:38,244
Speaker 3: is very important, right, And it doesn't know how to

567
00:31:38,284 --> 00:31:41,444
Speaker 3: distinguish those from its transformer architecture. So like, look, it

568
00:31:41,484 --> 00:31:43,804
Speaker 3: is good at like it is good at verbal reasoning.

569
00:31:43,844 --> 00:31:46,364
Speaker 3: Like again, I I, you know we are talking about

570
00:31:46,404 --> 00:31:48,684
Speaker 3: we don't other segments where we're like amazed by AI progress.

571
00:31:48,804 --> 00:31:53,444
Speaker 3: Like it's very good at verbal reasoning, or at least

572
00:31:53,444 --> 00:31:55,684
Speaker 3: faking that. But in ways I if it's faking, it's

573
00:31:55,684 --> 00:31:57,964
Speaker 3: doing a pretty good job. Right, Like it's very good

574
00:31:58,004 --> 00:32:02,244
Speaker 3: at verbal stuff for the most part. Right, And AI

575
00:32:02,324 --> 00:32:06,764
Speaker 3: is traded on poker, are very good at mimicking Selver solutions,

576
00:32:06,844 --> 00:32:09,884
Speaker 3: right and like and so like and machine learning can

577
00:32:09,884 --> 00:32:12,964
Speaker 3: get you a lot of ways when you have good data,

578
00:32:13,084 --> 00:32:14,564
Speaker 3: but like when you don't have the data in the

579
00:32:14,564 --> 00:32:17,124
Speaker 3: training set and it's not seeming to extrapolate very well,

580
00:32:17,204 --> 00:32:20,284
Speaker 3: and then these complex tasks that require more and more

581
00:32:20,324 --> 00:32:23,324
Speaker 3: compute like, I have not been impressed by deep research,

582
00:32:23,324 --> 00:32:25,044
Speaker 3: which is where it goes away and says I'm going

583
00:32:25,124 --> 00:32:29,244
Speaker 3: to perform several tasks for you that require deeper thought, right,

584
00:32:29,244 --> 00:32:30,724
Speaker 3: and then it takes fifteen minutes you come back and like,

585
00:32:30,724 --> 00:32:32,844
Speaker 3: are you fuck this up? Right? And like, anyway, so

586
00:32:32,964 --> 00:32:35,044
Speaker 3: I think it has affected my priors a little bit.

587
00:32:36,364 --> 00:32:39,084
Speaker 1: Yeah, that's that's really interesting. And I love, by the way, Nate,

588
00:32:39,164 --> 00:32:42,364
Speaker 1: that you know, even though that this is an AI

589
00:32:42,724 --> 00:32:45,644
Speaker 1: which is supposed to kind of be better than humans

590
00:32:45,684 --> 00:32:48,444
Speaker 1: in so many ways, that use such human excuses for

591
00:32:48,524 --> 00:32:51,044
Speaker 1: why it fucked up. You know, you stressed me out,

592
00:32:51,124 --> 00:32:53,564
Speaker 1: you gave me that data. It's not my fault.

593
00:32:53,644 --> 00:32:54,564
Speaker 2: This is hard.

594
00:32:55,324 --> 00:32:59,004
Speaker 1: And on that note, let's let's take a break and

595
00:32:59,044 --> 00:33:02,124
Speaker 1: talk a little bit about Harvard and the other type

596
00:33:02,164 --> 00:33:16,444
Speaker 1: of intelligence, human intelligence. Nay, We've talked on the show

597
00:33:16,524 --> 00:33:20,924
Speaker 1: a few times now about kind of the crusade that

598
00:33:21,084 --> 00:33:24,284
Speaker 1: Donald Trump seems to have against higher education, specifically the

599
00:33:24,324 --> 00:33:28,604
Speaker 1: Ivy League and even more specifically Harvard University. Last time

600
00:33:28,644 --> 00:33:34,364
Speaker 1: we spoke was when basically the administration had threatened to

601
00:33:34,524 --> 00:33:37,844
Speaker 1: pull funding if Harvard didn't comply with a certain set

602
00:33:37,884 --> 00:33:43,644
Speaker 1: of demands. Harvard said fuck you and sued and yeah,

603
00:33:43,724 --> 00:33:48,044
Speaker 1: and tried to go that route, and the funding was frozen.

604
00:33:48,604 --> 00:33:52,164
Speaker 1: And then the administration last week, so we're taping this

605
00:33:52,804 --> 00:33:58,084
Speaker 1: on Wednesday, the twenty eighth of May. So last week,

606
00:33:58,124 --> 00:34:02,324
Speaker 1: the administration decided that it wanted to punish Harvard even

607
00:34:02,364 --> 00:34:04,364
Speaker 1: more because you know, they didn't like the fact that

608
00:34:04,404 --> 00:34:07,524
Speaker 1: Harvard wasn't complying, that it was being defiant, that it

609
00:34:07,564 --> 00:34:10,564
Speaker 1: was taking them to court, and so said, hey, Harvard,

610
00:34:10,604 --> 00:34:15,844
Speaker 1: you can no longer have foreign students. Anyone who has

611
00:34:16,284 --> 00:34:19,604
Speaker 1: visa's effective immediately. Those are going to be revoked. So

612
00:34:19,764 --> 00:34:23,804
Speaker 1: Harvard's class of twenty twenty five that's graduating next week

613
00:34:23,964 --> 00:34:27,684
Speaker 1: is going to be your last basically foreign student class,

614
00:34:28,244 --> 00:34:31,244
Speaker 1: and we're not going to be granting visas anymore to

615
00:34:31,564 --> 00:34:35,404
Speaker 1: those students. Harvard has sued again, so now we have

616
00:34:35,444 --> 00:34:39,004
Speaker 1: a second set of lawsuits. The initial court cases have

617
00:34:39,164 --> 00:34:43,044
Speaker 1: frozen that. But then after that happened in the last

618
00:34:43,084 --> 00:34:47,324
Speaker 1: few days, the administration said Marco Rubio actually said that

619
00:34:47,404 --> 00:34:52,244
Speaker 1: we are going to freeze interviews for all foreign student visas,

620
00:34:52,524 --> 00:34:56,564
Speaker 1: not just Harvard. So they not only you know, went

621
00:34:56,604 --> 00:34:59,444
Speaker 1: after Harvard, but doubled down and said, you know what,

622
00:34:59,484 --> 00:35:01,884
Speaker 1: you can sue us and you can freeze this. But

623
00:35:02,284 --> 00:35:05,484
Speaker 1: if we don't grant the interviews, last laugh is with us.

624
00:35:06,884 --> 00:35:07,124
Speaker 3: Yeah.

625
00:35:07,164 --> 00:35:09,324
Speaker 2: Were they likely to win the lawsuit or are the

626
00:35:09,644 --> 00:35:09,964
Speaker 2: likely to?

627
00:35:11,044 --> 00:35:11,244
Speaker 3: Yeah?

628
00:35:11,284 --> 00:35:13,004
Speaker 1: I think so. I think that Harvard is likely to

629
00:35:13,044 --> 00:35:16,884
Speaker 1: win the lawsuit. Yes, However, from the lee I'm not

630
00:35:16,884 --> 00:35:19,804
Speaker 1: a lawyer, from the legal analysis that I have seen,

631
00:35:20,204 --> 00:35:23,444
Speaker 1: they're likely to win the lawsuit. But Nate, let's remember

632
00:35:23,764 --> 00:35:26,924
Speaker 1: that that requires first of all, a lot of times, right,

633
00:35:27,044 --> 00:35:31,084
Speaker 1: because how many times can we appeal this? And ultimately, well,

634
00:35:31,284 --> 00:35:34,044
Speaker 1: if it goes to the Supreme Court, it ends up

635
00:35:34,084 --> 00:35:37,004
Speaker 1: being these days less of a legal issue and more

636
00:35:37,004 --> 00:35:38,084
Speaker 1: of a political issue.

637
00:35:39,004 --> 00:35:41,124
Speaker 3: Look, I don't think the Supreme Court is likely to

638
00:35:41,164 --> 00:35:44,124
Speaker 3: be very sympathetic to Trump on this type of issue,

639
00:35:44,164 --> 00:35:48,244
Speaker 3: in part because, like you know, clearly they are somewhat

640
00:35:48,244 --> 00:35:51,004
Speaker 3: transparent about like they are not doing it for like

641
00:35:51,004 --> 00:35:53,324
Speaker 3: some security concerning and they're doing it because they don't

642
00:35:53,404 --> 00:35:57,884
Speaker 3: like some of the speech that Harvard is making. But

643
00:35:57,924 --> 00:35:59,964
Speaker 3: it's almost certainly First Amendment protect.

644
00:35:59,724 --> 00:36:02,804
Speaker 1: Yeah, exactly, and I actually think that the So I

645
00:36:02,804 --> 00:36:06,404
Speaker 1: don't know if you read Stephen Pinker's op ed in

646
00:36:06,444 --> 00:36:08,804
Speaker 1: the New York Times, I don't remember what the op

647
00:36:08,924 --> 00:36:12,444
Speaker 1: ed was called, but he termed this Harvard derangement syndrome, right,

648
00:36:12,564 --> 00:36:15,724
Speaker 1: kind of playing off of Trump arrangement syndrome and full disclosure.

649
00:36:16,084 --> 00:36:19,724
Speaker 1: Steve Pinker was my undergrad advisor, so someone I know, well,

650
00:36:19,764 --> 00:36:23,204
Speaker 1: someone i'm you know, I'm still close with, so I'm

651
00:36:23,284 --> 00:36:26,164
Speaker 1: very sympathetic. But I thought it was an excellent way

652
00:36:26,284 --> 00:36:29,644
Speaker 1: of framing what's going on. And he is someone who

653
00:36:29,644 --> 00:36:32,444
Speaker 1: has attacked Harvard. He's a tenured professor at Harvard, but

654
00:36:32,524 --> 00:36:36,844
Speaker 1: he's attacked Harvard many, many times for not standing up

655
00:36:36,844 --> 00:36:41,444
Speaker 1: for free speech, for not protecting conservative viewpoints, for being

656
00:36:41,524 --> 00:36:43,964
Speaker 1: a little bit too woke. Right. He has been one

657
00:36:43,964 --> 00:36:46,284
Speaker 1: of the main people who said, hey, like, we have

658
00:36:46,364 --> 00:36:49,444
Speaker 1: an issue here. But this his op ed was, this

659
00:36:49,524 --> 00:36:51,724
Speaker 1: ain't the way, right, this is not the way to

660
00:36:52,404 --> 00:36:55,564
Speaker 1: solve this problem. And you're actually attacking the students who

661
00:36:55,604 --> 00:36:58,884
Speaker 1: are some of my most critical, thinking, least woke people

662
00:36:59,644 --> 00:37:01,004
Speaker 1: who are on campus.

663
00:37:01,644 --> 00:37:02,964
Speaker 3: But yeah, look, I mean I kind of want to

664
00:37:03,004 --> 00:37:06,084
Speaker 3: zoom out and say, like, what is the equilibrium here,

665
00:37:06,124 --> 00:37:09,244
Speaker 3: Like what is a Trump administration trying to achieve? What

666
00:37:09,444 --> 00:37:12,244
Speaker 3: is Harvard trying to achieve? Like I support maybe some

667
00:37:12,324 --> 00:37:14,524
Speaker 3: listeners will get pissed off, right, you know, I think

668
00:37:15,164 --> 00:37:20,164
Speaker 3: clearly some of these colleges are not complying with like

669
00:37:20,844 --> 00:37:24,164
Speaker 3: the spirit and maybe the letter of the Supreme Court's

670
00:37:24,284 --> 00:37:28,244
Speaker 3: affirmative Action decisions, right, and so like, I wouldn't mind

671
00:37:28,324 --> 00:37:32,924
Speaker 3: if like the Trump administration's like aggressively enforcing the law

672
00:37:33,044 --> 00:37:36,884
Speaker 3: on those or things like diversity statements, which are enforced

673
00:37:37,284 --> 00:37:39,964
Speaker 3: political speech. Although Harvard's a private organization, so it's a

674
00:37:39,964 --> 00:37:41,964
Speaker 3: little bit more complicated, right, But like I you know,

675
00:37:42,004 --> 00:37:45,884
Speaker 3: I don't it's such a blunt tool to like target

676
00:37:46,244 --> 00:37:49,404
Speaker 3: foreign students. And even if you hate Harvard, you know,

677
00:37:49,444 --> 00:37:52,164
Speaker 3: as a patriotic American, I want the best and brightest

678
00:37:52,164 --> 00:37:54,764
Speaker 3: from all around the world to come here and contribute

679
00:37:54,804 --> 00:37:58,004
Speaker 3: to our economy and pay our taxes and everything else, right,

680
00:37:58,084 --> 00:38:01,484
Speaker 3: And like it's just like purely destroying the value of

681
00:38:01,484 --> 00:38:04,204
Speaker 3: these very bright students from helping our economy.

682
00:38:04,444 --> 00:38:07,204
Speaker 1: Yes, it's destroying our intellectual capital. So, by the way,

683
00:38:07,284 --> 00:38:09,364
Speaker 1: when we're taping this right now, I'm actually in Hongo,

684
00:38:10,084 --> 00:38:12,884
Speaker 1: and I just found out today that several universities in

685
00:38:12,964 --> 00:38:16,964
Speaker 1: Hong Kong have offered blanket admissions to all students who

686
00:38:17,164 --> 00:38:20,244
Speaker 1: can show that they were admitted to Harvard and can

687
00:38:20,284 --> 00:38:22,244
Speaker 1: no longer go because they are a foreign student. They said,

688
00:38:22,284 --> 00:38:24,324
Speaker 1: you don't even have to apply, you can just come here,

689
00:38:24,604 --> 00:38:26,844
Speaker 1: and these are some of the best universities in Hong Kong.

690
00:38:26,884 --> 00:38:30,004
Speaker 1: And that's super smart, right, Like, universities all over the

691
00:38:30,004 --> 00:38:32,124
Speaker 1: world should be doing stuff like that right now, because

692
00:38:32,484 --> 00:38:35,684
Speaker 1: what the administration is doing is kind of bankrupting at

693
00:38:35,724 --> 00:38:39,484
Speaker 1: the future of kind of intellectual development in the country

694
00:38:39,484 --> 00:38:42,404
Speaker 1: by saying, oh, foreign students, you know, we've we've just

695
00:38:43,004 --> 00:38:46,924
Speaker 1: frozen all these applications. So to me, like that's super smart, right,

696
00:38:47,004 --> 00:38:50,284
Speaker 1: Like that is the we've we talked about this several

697
00:38:50,324 --> 00:38:53,324
Speaker 1: months ago, Nate, Right, what can China do to capitalize

698
00:38:53,364 --> 00:38:55,724
Speaker 1: on the moment it's doing it. It's being like, come here,

699
00:38:55,884 --> 00:38:57,564
Speaker 1: like you don't even have to apply, you don't have

700
00:38:57,604 --> 00:38:59,684
Speaker 1: to do anything to show us the letter that showed

701
00:38:59,684 --> 00:39:02,484
Speaker 1: you were accepted and welcome. Come.

702
00:39:02,684 --> 00:39:04,884
Speaker 3: Yeah. I went to London School of Economics for my

703
00:39:04,964 --> 00:39:06,964
Speaker 3: junior year and if I were them, i'd be doing

704
00:39:07,484 --> 00:39:10,444
Speaker 3: cart wheels. And they always got a lot of foreign students.

705
00:39:10,124 --> 00:39:12,644
Speaker 1: Right absolutely, I mean I think everyone should be doing

706
00:39:12,644 --> 00:39:14,844
Speaker 1: this right now, and I just do not see I

707
00:39:14,844 --> 00:39:16,684
Speaker 1: mean a lot of these things. At this point, it

708
00:39:16,804 --> 00:39:19,884
Speaker 1: just derangement syndrome seems right, because there does not seem

709
00:39:19,884 --> 00:39:23,604
Speaker 1: to be any strategic value to this. There's not any

710
00:39:24,804 --> 00:39:27,404
Speaker 1: any value in terms of trying to get at the

711
00:39:27,444 --> 00:39:30,524
Speaker 1: types of problems that the Trump administration says it wants

712
00:39:30,564 --> 00:39:33,124
Speaker 1: to get at, because it's not even addressing that right,

713
00:39:33,204 --> 00:39:36,484
Speaker 1: Like at the originally anti Semitism was the pretext for this,

714
00:39:36,964 --> 00:39:39,484
Speaker 1: and this is just like so far divorced from it.

715
00:39:39,844 --> 00:39:43,284
Speaker 1: Everyone is having their visas revoked or not granted, or

716
00:39:43,324 --> 00:39:46,164
Speaker 1: their interview is not even granted. So it's just, you know,

717
00:39:46,244 --> 00:39:48,764
Speaker 1: the pretense is now crumbling. We always knew it was

718
00:39:48,844 --> 00:39:51,244
Speaker 1: just a pretense, but at this point, it's just not

719
00:39:51,284 --> 00:39:56,084
Speaker 1: accomplishing anything other than to further could of create this

720
00:39:56,324 --> 00:40:02,124
Speaker 1: absolute chasm in higher education and future brain power, future

721
00:40:02,204 --> 00:40:07,724
Speaker 1: research ability, future creativity of the United States basically mortgaging

722
00:40:07,764 --> 00:40:11,084
Speaker 1: its future right saying that we were going to give

723
00:40:11,164 --> 00:40:13,604
Speaker 1: up one of the biggest, if not the biggest edge

724
00:40:13,644 --> 00:40:17,284
Speaker 1: that the United States has had historically, which is kind

725
00:40:17,324 --> 00:40:23,844
Speaker 1: of this brain power creativity place where people can can

726
00:40:23,924 --> 00:40:30,084
Speaker 1: develop and find support for for for their ideas, and

727
00:40:30,204 --> 00:40:35,644
Speaker 1: now that is not possible and if this actually stands, right,

728
00:40:35,724 --> 00:40:39,004
Speaker 1: if foreign students are not allowed into US universities, because

729
00:40:39,084 --> 00:40:42,724
Speaker 1: right now, like I said, this interview process being paused

730
00:40:42,764 --> 00:40:45,564
Speaker 1: for everyone, not just Harvard. I mean, that is going

731
00:40:45,644 --> 00:40:51,484
Speaker 1: to be just detrimental in so many respects to what

732
00:40:51,684 --> 00:40:52,364
Speaker 1: happens here.

733
00:40:53,644 --> 00:40:57,644
Speaker 3: Yeah, the next So Harvard is not a very sympathetic

734
00:40:57,764 --> 00:41:00,044
Speaker 3: case for reasons that are partly they're full right, but

735
00:41:00,204 --> 00:41:02,404
Speaker 3: like you know, so far, the Trump admin has been

736
00:41:02,404 --> 00:41:05,484
Speaker 3: pretty smart about kind of which schools they they pick on.

737
00:41:05,564 --> 00:41:07,004
Speaker 3: I mean, I don't know what their in game is

738
00:41:07,004 --> 00:41:09,164
Speaker 3: here or what Harvard's is really you know, JD. Vans

739
00:41:09,164 --> 00:41:11,164
Speaker 3: had it tweet. I don't know where he was like,

740
00:41:11,204 --> 00:41:16,124
Speaker 3: he's oddly transparent sometimes about his thinking, and he's like,

741
00:41:16,204 --> 00:41:19,404
Speaker 3: well look just like, look, these colleges aren't serious about

742
00:41:19,444 --> 00:41:23,324
Speaker 3: complying with the law citing like the students for fair admissions.

743
00:41:23,324 --> 00:41:25,964
Speaker 3: I think it's called which was the affirmative action decision,

744
00:41:26,004 --> 00:41:28,044
Speaker 3: which I do think they've kind of not been serious

745
00:41:28,044 --> 00:41:30,444
Speaker 3: about complying with in some cases case by case basis,

746
00:41:30,444 --> 00:41:32,884
Speaker 3: and so like, unless we kind of show real muscle,

747
00:41:33,524 --> 00:41:35,444
Speaker 3: then what can we do. We have to fight them

748
00:41:35,484 --> 00:41:38,804
Speaker 3: because they're in the wrong, and so we're overreaching. It

749
00:41:38,844 --> 00:41:43,484
Speaker 3: is a subtext of that, right, which is like maybe

750
00:41:43,524 --> 00:41:46,124
Speaker 3: not crazy and they have a real leader because Harvard

751
00:41:46,124 --> 00:41:50,364
Speaker 3: isn't that sympathetic perceptions of higher education that's gone way down,

752
00:41:50,524 --> 00:41:52,804
Speaker 3: so it's like harder to marshall public opinion on the

753
00:41:53,004 --> 00:41:56,724
Speaker 3: side of this, right, and like and like, you know,

754
00:41:56,804 --> 00:41:59,724
Speaker 3: I saw some Harvard professors on Twitter. Ever we can

755
00:41:59,764 --> 00:42:01,684
Speaker 3: be like this is the greatest threat to American I'm like,

756
00:42:02,044 --> 00:42:05,124
Speaker 3: please don't say that, because it's it's it's maybe more

757
00:42:05,124 --> 00:42:07,004
Speaker 3: of a threat to American competitoris and not sure it's

758
00:42:07,004 --> 00:42:09,044
Speaker 3: a threat to democracy per se.

759
00:42:09,444 --> 00:42:12,244
Speaker 1: Right, No, I think there are bigger threats to democracy.

760
00:42:13,644 --> 00:42:16,564
Speaker 3: But like, but what's Harvard's end game? Like I think, ok,

761
00:42:16,604 --> 00:42:21,444
Speaker 3: if they could go back and tone down some of

762
00:42:21,484 --> 00:42:23,844
Speaker 3: the things say down for the past ten years, and

763
00:42:23,884 --> 00:42:27,604
Speaker 3: I think they maybe would, but like now, I mean,

764
00:42:27,604 --> 00:42:29,884
Speaker 3: if they back down, then they look weak and that

765
00:42:29,924 --> 00:42:32,724
Speaker 3: also might hurt their recruitment own nothing, they'd have real problems.

766
00:42:32,764 --> 00:42:34,484
Speaker 3: But like, what if you're the president of Heart What's

767
00:42:34,524 --> 00:42:37,404
Speaker 3: name Alan Garber? Is that right? Yep? If you're Alan Garber, Maria,

768
00:42:37,484 --> 00:42:39,684
Speaker 3: what do you do right now? And you know Harvard?

769
00:42:40,964 --> 00:42:44,404
Speaker 1: Yeah, yeah, I mean, you're in a horrible position. I

770
00:42:44,444 --> 00:42:47,364
Speaker 1: think you're doing what you can, which is trying to

771
00:42:47,764 --> 00:42:50,724
Speaker 1: stand up to the Trump administration. So Alan Garber is

772
00:42:50,764 --> 00:42:53,484
Speaker 1: trying to actually walk a very fine line. He's trying

773
00:42:53,484 --> 00:42:56,484
Speaker 1: to stand up to Trump, and he's you know, spearheading

774
00:42:56,524 --> 00:43:03,084
Speaker 1: these lawsuits, and he is leading committees into the claims

775
00:43:03,084 --> 00:43:07,604
Speaker 1: of anti Semitism and kind of and discrimination on campus,

776
00:43:07,604 --> 00:43:11,164
Speaker 1: the anti wokeness. So he is actually trying to address

777
00:43:11,284 --> 00:43:15,644
Speaker 1: all of these things in different ways. And I have

778
00:43:15,764 --> 00:43:19,644
Speaker 1: no idea, you know, how successful that will end up being,

779
00:43:20,404 --> 00:43:22,204
Speaker 1: but it is. It's a it's a tough you know,

780
00:43:22,244 --> 00:43:24,884
Speaker 1: he's been put in a really tough spot because obviously

781
00:43:24,924 --> 00:43:29,404
Speaker 1: he hasn't been president for long. The last president departed

782
00:43:29,484 --> 00:43:34,644
Speaker 1: on lessons tellar circumstances, as did a number of presidents

783
00:43:34,644 --> 00:43:40,684
Speaker 1: of Ivy League institutions, and you know, you're picking up

784
00:43:40,724 --> 00:43:45,564
Speaker 1: the presidency at a very difficult point in time where, yeah,

785
00:43:45,684 --> 00:43:48,524
Speaker 1: you have to acknowledge that Harvard has made mistakes, right,

786
00:43:48,604 --> 00:43:52,524
Speaker 1: not just Harvard, a lot of universities I think were

787
00:43:52,844 --> 00:43:56,164
Speaker 1: trying to be I don't even want to use the

788
00:43:56,204 --> 00:43:59,084
Speaker 1: word woke, because I don't think that's accurate. Here but

789
00:43:59,284 --> 00:44:03,764
Speaker 1: we're becoming, you know, places where people were afraid of

790
00:44:03,884 --> 00:44:07,484
Speaker 1: voicing their minds. And I wrote, I think almost a

791
00:44:07,604 --> 00:44:10,284
Speaker 1: decade ago, I wrote a piece for The New Yorker

792
00:44:11,164 --> 00:44:15,164
Speaker 1: about basically, you know, the problems in academia that come

793
00:44:15,204 --> 00:44:18,484
Speaker 1: from not having enough conservatives right that it actually has

794
00:44:18,524 --> 00:44:21,204
Speaker 1: trickle down effects in research and the types of things

795
00:44:21,204 --> 00:44:23,284
Speaker 1: that are being researched in the findings that are allowed

796
00:44:23,324 --> 00:44:26,204
Speaker 1: to be voiced, and that it can be incredibly problematic.

797
00:44:27,124 --> 00:44:30,244
Speaker 1: And so I think you need to acknowledge that, and

798
00:44:30,364 --> 00:44:33,204
Speaker 1: those are things that need to be rectified. At the

799
00:44:33,324 --> 00:44:37,604
Speaker 1: same time, you are currently fighting this battle against you know,

800
00:44:37,684 --> 00:44:40,844
Speaker 1: Donald Trump, who's trying to destroy higher education in the

801
00:44:40,964 --> 00:44:44,964
Speaker 1: United States, and so that is a you know, overriding priority,

802
00:44:45,004 --> 00:44:47,484
Speaker 1: which doesn't dismiss all of the things that have gone

803
00:44:47,484 --> 00:44:50,044
Speaker 1: wrong and all of the things that are wrong with academia,

804
00:44:50,124 --> 00:44:52,444
Speaker 1: but it's just a it's a kind of this fine

805
00:44:52,484 --> 00:44:55,044
Speaker 1: balance where you're like, yeah, I know I'm not sympathetic

806
00:44:55,124 --> 00:44:57,684
Speaker 1: because I've fucked up and I've done all of these things.

807
00:44:58,004 --> 00:45:00,004
Speaker 1: And yet right now I'm one of the people best

808
00:45:00,004 --> 00:45:03,684
Speaker 1: positioned to fight this, and so let's just try to

809
00:45:03,724 --> 00:45:06,884
Speaker 1: figure out how to prioritize this and how to not

810
00:45:07,044 --> 00:45:11,084
Speaker 1: spread ourselves tooth in so that we can actually protect

811
00:45:11,164 --> 00:45:14,524
Speaker 1: higher education and then start to remedy the problems that

812
00:45:14,564 --> 00:45:17,324
Speaker 1: we have internally, because I do think those are problems

813
00:45:17,324 --> 00:45:19,644
Speaker 1: that need to be remedied, right, Like, you can't just

814
00:45:19,684 --> 00:45:22,364
Speaker 1: be like, well, now we're all united against Trump, so

815
00:45:22,444 --> 00:45:24,804
Speaker 1: let's forget we ever did anything wrong. I think that

816
00:45:24,964 --> 00:45:28,444
Speaker 1: all of these things you remember, Nate, like less than

817
00:45:28,604 --> 00:45:31,324
Speaker 1: you know. Like six months ago, I was railing against

818
00:45:31,404 --> 00:45:34,124
Speaker 1: Harvard because I was really pissed at the way they

819
00:45:34,124 --> 00:45:36,844
Speaker 1: were reacting to a lot of things. Now I think

820
00:45:37,044 --> 00:45:39,884
Speaker 1: now I'm supporting it because they're being attacked. It's the

821
00:45:39,964 --> 00:45:43,484
Speaker 1: classic psychology thing, right where you can actually unite a

822
00:45:43,524 --> 00:45:47,604
Speaker 1: lot of people who who criticize you when you attack, right,

823
00:45:47,644 --> 00:45:50,204
Speaker 1: all of a sudden, these various factions, when they feel

824
00:45:50,244 --> 00:45:54,324
Speaker 1: these life threatening attacks from the outside, all of a sudden,

825
00:45:54,324 --> 00:45:56,644
Speaker 1: these factions start uniting. And I think that's what's happening

826
00:45:56,684 --> 00:45:59,924
Speaker 1: at Harvard right now, as it should be. But we

827
00:46:00,084 --> 00:46:03,044
Speaker 1: still have to realize that, you know, there are all

828
00:46:03,084 --> 00:46:05,444
Speaker 1: of these different issues and it's a really I mean,

829
00:46:05,684 --> 00:46:08,604
Speaker 1: it's a shit show. That's a technical term.

830
00:46:08,684 --> 00:46:11,924
Speaker 2: Nate yeah, Look, I'm not afraid to use a term woke.

831
00:46:12,004 --> 00:46:14,724
Speaker 3: I mean, you know, critical race theory and intersectionality, all

832
00:46:14,764 --> 00:46:18,524
Speaker 3: these things kind of come from an extremely academic context, right,

833
00:46:20,484 --> 00:46:25,964
Speaker 3: And Look, the thing I'm probably most concerned about is

834
00:46:26,004 --> 00:46:28,124
Speaker 3: the fact that I think the quality of academic research

835
00:46:28,204 --> 00:46:32,004
Speaker 3: is often has lots of problems, but like political bias

836
00:46:32,364 --> 00:46:36,044
Speaker 3: in predictable directions is among those problems. I'm trying to

837
00:46:36,044 --> 00:46:38,604
Speaker 3: think of a politically experienced solution that the Trump administration

838
00:46:39,444 --> 00:46:41,644
Speaker 3: might like and that people would find tolerable. On the

839
00:46:41,644 --> 00:46:44,644
Speaker 3: Harvard side, Right, we'll collect data on the political affiliation

840
00:46:45,244 --> 00:46:50,044
Speaker 3: of our students and faculty. We will make concerted efforts

841
00:46:50,124 --> 00:46:52,964
Speaker 3: to I'm not going to try to frame this right

842
00:46:53,124 --> 00:46:56,644
Speaker 3: conservati efforts like higher conservative scholars. Right, maybe you have

843
00:46:56,844 --> 00:46:59,324
Speaker 3: like you know, if you have an ethnic studies department,

844
00:46:59,404 --> 00:47:00,284
Speaker 3: you have a conservative stay.

845
00:47:00,364 --> 00:47:03,004
Speaker 1: No, I mean that isn't that you know exactly what

846
00:47:04,484 --> 00:47:07,164
Speaker 1: people have been trying to reverse by kind of with

847
00:47:07,204 --> 00:47:09,924
Speaker 1: the affirmative action rulings, right, that you have quotos for

848
00:47:10,404 --> 00:47:13,084
Speaker 1: certain types of people. So I think that the better

849
00:47:13,124 --> 00:47:16,804
Speaker 1: way to do it is basically having blind hiring, just

850
00:47:16,844 --> 00:47:20,244
Speaker 1: like you have blind admissions. You are willing. You just

851
00:47:20,324 --> 00:47:22,644
Speaker 1: don't you don't ask any of that, and you don't

852
00:47:22,684 --> 00:47:26,564
Speaker 1: disqualify people because they have conservative leanings, right, and that

853
00:47:26,604 --> 00:47:30,564
Speaker 1: it's not even it's not a thing because you don't

854
00:47:30,604 --> 00:47:33,204
Speaker 1: really care, Like I don't care if my uh, you know,

855
00:47:33,404 --> 00:47:37,044
Speaker 1: neuroscience professor, Well maybe not neuroscience as insofar as I

856
00:47:37,084 --> 00:47:42,564
Speaker 1: study psychology, but if my astrophysics professor is, you know,

857
00:47:43,724 --> 00:47:47,324
Speaker 1: what their political views are, as long as they correctly,

858
00:47:47,484 --> 00:47:48,804
Speaker 1: you know, as long as they're one of the best

859
00:47:48,884 --> 00:47:52,044
Speaker 1: astrophysicists in the world. So so I don't think that

860
00:47:52,044 --> 00:47:56,164
Speaker 1: that's you know, that what you're proposing is affirmative action

861
00:47:56,284 --> 00:47:57,084
Speaker 1: for politics.

862
00:47:57,444 --> 00:48:00,004
Speaker 3: I'm just saying, like it's slightly stupid, but like maybe

863
00:48:00,044 --> 00:48:00,524
Speaker 3: maybe jd.

864
00:48:00,604 --> 00:48:02,924
Speaker 2: Vance would like it.

865
00:48:03,564 --> 00:48:06,644
Speaker 1: But but you're saying, like do something that he would like.

866
00:48:06,924 --> 00:48:09,404
Speaker 1: Columbia tried to do everything and had all of their

867
00:48:09,444 --> 00:48:12,844
Speaker 1: funding frozen anyway, right, So like this is the problem

868
00:48:12,884 --> 00:48:16,604
Speaker 1: that like actually doing trying to appease is not the

869
00:48:16,604 --> 00:48:20,004
Speaker 1: correct play here. I think that's actually the completely wrong

870
00:48:20,084 --> 00:48:24,284
Speaker 1: game theoretical way to respond here. That's because it's not working.

871
00:48:24,844 --> 00:48:27,444
Speaker 3: And that's a meta thing across everything that the White

872
00:48:27,444 --> 00:48:29,764
Speaker 3: House does. You see it playing out Ontari's too, where

873
00:48:29,804 --> 00:48:34,084
Speaker 3: on the one hand, they reverse course and undercut themselves

874
00:48:35,204 --> 00:48:38,964
Speaker 3: all the time, right, you know, and they're sloppy enough

875
00:48:39,004 --> 00:48:42,004
Speaker 3: where they probably maybe if they're doing this right, they

876
00:48:42,044 --> 00:48:44,484
Speaker 3: could have had more likelihood of success in the courts

877
00:48:44,524 --> 00:48:46,684
Speaker 3: than they have. Right, on either hand, they don't necessarily

878
00:48:46,724 --> 00:48:49,084
Speaker 3: hold up their quid pro quote. This is kind of

879
00:48:49,204 --> 00:48:53,044
Speaker 3: very transactional and explicit, right, So they're not even like

880
00:48:54,084 --> 00:48:59,324
Speaker 3: presenting clear enough information to make themselves easy to bargain with,

881
00:48:59,524 --> 00:49:05,564
Speaker 3: even if it were some theoretical optimal solution or win win.

882
00:49:06,564 --> 00:49:08,964
Speaker 1: Yeah. So I mean, so I guess, you know, we're

883
00:49:09,484 --> 00:49:12,284
Speaker 1: just in a very sticky spot and we'll have to

884
00:49:12,324 --> 00:49:14,244
Speaker 1: see how it plays out. I think Harvard is doing

885
00:49:14,324 --> 00:49:16,684
Speaker 1: its best, and you know, I wish that I could

886
00:49:16,684 --> 00:49:18,804
Speaker 1: give it, you know, a playbook, Harvard, this is what

887
00:49:18,844 --> 00:49:22,924
Speaker 1: you should be doing, but but I can't. So let's

888
00:49:23,084 --> 00:49:26,484
Speaker 1: just let's hold out hope. I think you are.

889
00:49:26,924 --> 00:49:30,764
Speaker 3: I mean, there are real changes that like, yeah, Harvard should, and.

890
00:49:30,724 --> 00:49:33,444
Speaker 1: I think Stephen and I think Stephen Pinker has advocated

891
00:49:33,484 --> 00:49:35,924
Speaker 1: for a lot of them. Like, as I said, I'm biased,

892
00:49:36,004 --> 00:49:40,044
Speaker 1: but I think that Harvard and Harvard should listen to

893
00:49:40,204 --> 00:49:44,724
Speaker 1: Steve Pinker. That's my advice. I think he's one of

894
00:49:44,764 --> 00:49:47,924
Speaker 1: the smartest people I know who actually knows a lot

895
00:49:47,924 --> 00:49:51,004
Speaker 1: about this and has very rational views and can advise

896
00:49:51,044 --> 00:49:54,804
Speaker 1: on this. It is almost eleven pm for me in

897
00:49:54,844 --> 00:49:56,884
Speaker 1: Hong Kong, so I think we're gonna rap it for

898
00:49:56,924 --> 00:49:59,684
Speaker 1: this week. The World to Years of Poker has already started.

899
00:50:00,004 --> 00:50:01,444
Speaker 1: You and I are not going to be out there

900
00:50:01,484 --> 00:50:05,044
Speaker 1: for another It's another basically two weeks, I think for

901
00:50:05,124 --> 00:50:08,004
Speaker 1: us week and a half. So let's wish all of

902
00:50:08,044 --> 00:50:10,684
Speaker 1: our Risky Business listeners who are already in Vegas for

903
00:50:10,684 --> 00:50:13,804
Speaker 1: the World Series of Poker good luck. We hope that

904
00:50:13,884 --> 00:50:16,364
Speaker 1: you all crush it at the tables and we're looking

905
00:50:16,364 --> 00:50:20,684
Speaker 1: forward to joining you soon. Let us know what you

906
00:50:20,724 --> 00:50:23,124
Speaker 1: think of the show. Reach out to us at Risky

907
00:50:23,164 --> 00:50:26,964
Speaker 1: Business at Pushkin dot fm. And by the way, if

908
00:50:27,004 --> 00:50:29,564
Speaker 1: you're a Pushkin Plus subscriber, we have some bonus content

909
00:50:29,604 --> 00:50:32,164
Speaker 1: for you that's coming up right after the credits.

910
00:50:32,484 --> 00:50:35,724
Speaker 3: And if you're not subscribing yet, consider signing up for

911
00:50:35,924 --> 00:50:38,084
Speaker 3: just six ninety nine a month. What a nice price

912
00:50:38,444 --> 00:50:41,204
Speaker 3: you get access to all that premium content and ad

913
00:50:41,244 --> 00:50:43,884
Speaker 3: for listening across Pushkin's entire network of shows.

914
00:50:44,404 --> 00:50:47,444
Speaker 1: Risky Business is hosted by me Maria Kannakova.

915
00:50:47,044 --> 00:50:49,404
Speaker 3: And by me Nate Silver. The show is a co

916
00:50:49,484 --> 00:50:53,564
Speaker 3: production of Pushkin Industries and iHeartMedia. This episode was produced

917
00:50:53,604 --> 00:50:57,484
Speaker 3: by Isabelle Carter. Our associate producer is Sonia Gerwin. Sally

918
00:50:57,564 --> 00:51:00,564
Speaker 3: Helm is our editor, and our executive producer is Jacob Bolstein.

919
00:51:01,084 --> 00:51:02,284
Speaker 2: Mixing by Sarah Bruger.

920
00:51:02,804 --> 00:51:04,004
Speaker 1: Thanks so much for tuning in.