WEBVTT - Why ChatGPT Sucks at Poker 0:00:15.604 --> 0:00:28.764 Pushkin. Welcome back to Risky Business, a show about making 0:00:28.844 --> 0:00:31.484 better decisions. I'm Maria Kanakova. 0:00:31.204 --> 0:00:32.284 And I'm Nate Silver. 0:00:32.844 --> 0:00:35.324 Today on the show, we're gonna be talking about how 0:00:35.524 --> 0:00:38.164 chat GPT plays poker, i e. 0:00:38.364 --> 0:00:38.764 Poorly. 0:00:38.924 --> 0:00:40.964 If chat gipt is getting ready for the World Series 0:00:40.964 --> 0:00:43.124 of Poker like us, then it's got a lot of 0:00:43.124 --> 0:00:43.764 homework to do. 0:00:44.084 --> 0:00:48.924 Yes. And after we talk about that, we'll talk about 0:00:48.924 --> 0:00:53.924 Harvard two point zero aka the fact that foreign students 0:00:54.004 --> 0:00:59.804 are now being potentially barred from attending Harvard University. Harvard 0:00:59.884 --> 0:01:02.764 has sued the administration. Anyway, we'll be talking about what's 0:01:02.804 --> 0:01:06.284 going on with that and what the implications are for 0:01:06.364 --> 0:01:10.764 the future of the United States and kind of research development, 0:01:11.124 --> 0:01:21.684 brain power and the US competitive edge. So for people 0:01:21.764 --> 0:01:25.044 who don't subscribe to Silver Bulletin, which you absolutely should, 0:01:25.204 --> 0:01:27.364 Nate and I both had poker posts this last week. 0:01:27.524 --> 0:01:32.004 Mine was about cheating, was about chat GPT, but his 0:01:32.124 --> 0:01:34.924 post this week was one of the funniest things I've 0:01:34.964 --> 0:01:39.804 read in a while about his attempts to get chat 0:01:39.844 --> 0:01:45.244 GPT to simulate a hand of cash game poker. And 0:01:46.244 --> 0:01:48.284 one of the reasons I mean, I found this amusing 0:01:48.324 --> 0:01:51.244 because I mean, first of all, it fused cards together 0:01:51.284 --> 0:01:53.804 for an image. From the beginning to the end, it 0:01:53.924 --> 0:01:56.924 was pretty spot on in terms of being spot on wrong. 0:01:57.124 --> 0:02:00.084 But Nay, You're also someone who is constantly writing about 0:02:00.084 --> 0:02:03.004 how good AI is at so many things, and so 0:02:03.524 --> 0:02:08.084 it was funny to me to see it fall so 0:02:08.644 --> 0:02:14.084 short of something that actually tests intelligence as opposed to 0:02:14.764 --> 0:02:17.604 being able to kind of pull together a lot of 0:02:17.644 --> 0:02:19.204 things and spit out an answer. 0:02:19.724 --> 0:02:21.284 There are some reasons why I'm just sid in testing 0:02:21.404 --> 0:02:25.124 chatchipt on poker, but they kind of fall into two 0:02:25.204 --> 0:02:29.244 big buckets. Right, What is it like poker played in 0:02:29.324 --> 0:02:31.844 the context of a real hand or in this case, 0:02:31.884 --> 0:02:35.604 a real text based simulation I guess of a hand. 0:02:36.004 --> 0:02:40.884 It really does require quite a few skills, right. There 0:02:40.964 --> 0:02:44.484 is the pure math part of it. What is the equilibrium, 0:02:44.524 --> 0:02:47.964 the Nash equilibrium, the gto strategy that you're solving for. 0:02:48.484 --> 0:02:52.244 There's also making adjustments for how other players play. There's 0:02:52.284 --> 0:02:55.884 the conversation you're having, the physical reads that you're getting 0:02:56.164 --> 0:02:58.884 their stuff, like knowing what the rules are right, which 0:02:59.524 --> 0:03:03.284 might seem trivial, but we've yeah, chatchpt also. 0:03:03.124 --> 0:03:03.804 Does not know. 0:03:03.924 --> 0:03:06.164 It seems to be no, it is fun to look. 0:03:06.204 --> 0:03:07.804 I mean, I'm sure we've all had hands where we 0:03:07.924 --> 0:03:10.644 like misread a board or I you know, I had 0:03:10.684 --> 0:03:13.324 a big hand in the main event at the when 0:03:13.404 --> 0:03:15.604 I play back in Florida back in April or whatever, 0:03:15.644 --> 0:03:18.484 right where like a guy had like a twenty five 0:03:18.484 --> 0:03:20.524 thousand dollars chip that was like almost the same color 0:03:20.524 --> 0:03:23.524 as the felt, which is not good chipkit maintenance, by 0:03:23.564 --> 0:03:25.964 the way, Seminal hard Rock. I like the WPT, but 0:03:26.524 --> 0:03:29.044 that chip kit it's got to be improved, right, But yeah, 0:03:29.044 --> 0:03:32.564 you're tired, you make mistakes like that. But and you 0:03:32.604 --> 0:03:36.884 also have to have like a lot of short term memory, 0:03:37.644 --> 0:03:40.004 which sounds trivial, but you you know, you want to 0:03:40.004 --> 0:03:44.404 remember how you arrived at this current spot in the hand, right, 0:03:44.804 --> 0:03:47.004 you have to keep crack of all these stack sizes, 0:03:47.084 --> 0:03:50.524 which again seems trivial, but like it's a good test 0:03:50.564 --> 0:03:55.404 of general intelligence of a certain type, especially a live 0:03:55.524 --> 0:03:57.004 poker hand, like I'm asking. 0:03:56.804 --> 0:03:58.444 Chat ept to simulate. 0:03:58.644 --> 0:04:00.844 Right. The other reason is that like I don't think 0:04:00.924 --> 0:04:06.404 engineers in Silicon Valley are trying to optimize it for poker. 0:04:07.404 --> 0:04:09.844 And the reason that's important is because like, look, there 0:04:09.884 --> 0:04:14.364 are benchmarks like math Olympiad problems that these LM's large 0:04:14.404 --> 0:04:17.884 language models like CHATCHPT, Claude, et cetera compete on, right, 0:04:17.924 --> 0:04:19.884 And they're like, well, we be bragging that we can now, 0:04:20.124 --> 0:04:23.804 you know, beat all but the Nobel Prize mathematicians on 0:04:24.044 --> 0:04:26.244 x percentage of math problems and things like that. And 0:04:26.284 --> 0:04:29.004 there are a couple of issues with that, right. Like 0:04:29.044 --> 0:04:33.924 one is that, like, clearly, if you train a transformer model, 0:04:33.964 --> 0:04:37.284 a machine learning model on a particular type of problem, 0:04:37.804 --> 0:04:42.084 like it can do fairly well or very well often, right, 0:04:42.164 --> 0:04:45.884 Like poker is to a first approximation, and it's an 0:04:45.884 --> 0:04:49.884 important approximation. Right, But if you have poker dedicated tools, 0:04:50.244 --> 0:04:53.844 I wouldn't consider a solver and AI that's a technical 0:04:53.844 --> 0:04:56.484 distinction I think might not be that important for our listeners. 0:04:56.484 --> 0:04:58.484 But like, you know, if you want to use computers 0:04:58.484 --> 0:05:01.444 to play very good poker, then they can play very 0:05:01.524 --> 0:05:04.644 very good poker. Right. The question is like, can you 0:05:04.764 --> 0:05:09.204 take a text based model and have this organic property 0:05:09.644 --> 0:05:12.924 where intelligence emerges and converts from the data set towards 0:05:12.964 --> 0:05:16.604 superintelligence without training it on poker specifically. And the answer, 0:05:16.604 --> 0:05:18.804 at least as of last week when I did this, 0:05:18.924 --> 0:05:20.564 is no, it fails miserably. 0:05:22.004 --> 0:05:25.164 Yeah, and I think it's actually important. One of the 0:05:25.204 --> 0:05:27.884 things that came to my mind when I was reading 0:05:27.924 --> 0:05:32.164 your piece was that poker has been a benchmark for 0:05:32.724 --> 0:05:37.324 AI development well before LMS, Right, It has been kind 0:05:37.324 --> 0:05:40.524 of the gold standard that teams all over the world 0:05:41.244 --> 0:05:46.004 have been working on for decades because it is a 0:05:46.124 --> 0:05:51.004 much more complex game in many senses than chess, then 0:05:51.084 --> 0:05:55.724 even go, and it's a game where, you know, if 0:05:55.764 --> 0:05:59.244 computers can actually manage to outthink human players on a 0:05:59.284 --> 0:06:02.724 broader scale, that would mean something in terms of kind 0:06:02.724 --> 0:06:08.564 of broader intelligence capabilities. And before LMS were developed, that 0:06:08.604 --> 0:06:12.524 had not happened. Right, There had been computer programs, kind 0:06:12.524 --> 0:06:16.284 of AI algorithms that have been able to beat heads 0:06:16.404 --> 0:06:20.004 up one on one opponents in poker. But when it 0:06:20.044 --> 0:06:22.404 came to full ring games, So for people who don't 0:06:22.444 --> 0:06:24.444 know poker, that means basically, when you put it in 0:06:24.484 --> 0:06:27.724 a rich environment with you know, six players, eight players 0:06:28.364 --> 0:06:31.644 and you have a computer there, it was still not 0:06:31.924 --> 0:06:34.884 quite there. And one of the you know, one of 0:06:34.924 --> 0:06:37.924 the reasons why poker so interesting is it's not just math, right, 0:06:38.004 --> 0:06:41.324 especially when you have multiple players. There's so many dynamics 0:06:41.364 --> 0:06:45.924 and there's actually a very thorny problems for AI, which 0:06:46.004 --> 0:06:51.084 is that if you program something to follow an algorithm 0:06:51.164 --> 0:06:55.644 right to be GTO game theory optimal, then it can 0:06:55.764 --> 0:06:58.364 be exploited by a human if the human figures out 0:06:58.404 --> 0:07:01.484 the GTO strategy. Now, if it were another human, then 0:07:01.564 --> 0:07:04.964 you'd adjust immediately right the moment the human adjusts, the 0:07:05.084 --> 0:07:07.524 other human would adjust and kind of figure it out. 0:07:07.964 --> 0:07:11.004 But if you're an a if you're a computer, and 0:07:11.044 --> 0:07:15.124 you build in that adjustment parameter, what ends up happening 0:07:15.164 --> 0:07:17.724 is that the model adjusts too much and too quickly, 0:07:18.124 --> 0:07:22.524 and so that lacks kind of some of the nuance 0:07:22.764 --> 0:07:26.484 and flexibility that marks that's kind of the hallmark of 0:07:26.484 --> 0:07:29.604 the great poker players, right, kind of that ability to 0:07:30.244 --> 0:07:34.444 into it when it's time to slightly deviate. And I mean, 0:07:34.484 --> 0:07:39.444 it would be terrifying but amazing if a model could spontaneously, 0:07:39.844 --> 0:07:43.124 like an LLM, could actually spontaneously figure that out and 0:07:43.324 --> 0:07:46.484 so far, I mean, instead instead of doing that, Nate, 0:07:46.804 --> 0:07:49.484 you know, your model decided that the person who won 0:07:49.524 --> 0:07:52.404 the hand actually lost the hand, right, It just got 0:07:52.524 --> 0:07:57.244 basic things completely wrong, and you just you know, obviously 0:07:57.284 --> 0:08:00.044 we've talked about pe doom a lot on the show. 0:08:00.404 --> 0:08:02.444 I think you and I are both worried about that. 0:08:02.684 --> 0:08:05.244 But when I see this poker stuff and like, oh man. 0:08:05.844 --> 0:08:08.844 Yeah, I don't know how much to update. But look 0:08:08.884 --> 0:08:13.524 my say in the post, I mean, like my appreciation 0:08:13.684 --> 0:08:18.124 and expectations for large language models have increased a lot 0:08:18.164 --> 0:08:22.484 over the past year. I had a pretty good experience 0:08:22.524 --> 0:08:24.484 working with them when I was building the NCAA tournament 0:08:24.484 --> 0:08:27.044 model that we host silver bulletin and just helpful kind 0:08:27.084 --> 0:08:31.164 of Swiss army knives or like annoying data related tasks. 0:08:32.444 --> 0:08:35.764 It's late at night, you forget the Stata command for something, 0:08:35.764 --> 0:08:37.684 You're like, right, give me the right command and write 0:08:37.684 --> 0:08:39.724 five lines of code, and like usually it works, and 0:08:39.764 --> 0:08:41.964 it kind of now shows you like the chain of 0:08:42.004 --> 0:08:46.604 reasoning and like what it's thinking. Right, And but you know, 0:08:46.644 --> 0:08:49.164 people are saying that we're going to have like AI 0:08:50.044 --> 0:08:55.044 revolutionizing all programming or you know, well, first of all, 0:08:55.044 --> 0:08:57.244 the big claim is going to have AGI, meaning that 0:08:57.364 --> 0:09:01.164 a machine. It's a little ambiguous when people say, does 0:09:01.204 --> 0:09:05.964 this mean like chat GPT itself is AGI? Right, Well, 0:09:06.044 --> 0:09:10.564 clearly not. It's very far from human being in some things, right, 0:09:11.044 --> 0:09:14.844 Does that mean that a combination of transformer or machine 0:09:14.884 --> 0:09:17.644 learning based technologies is or zooming out the definition of AI. 0:09:17.724 --> 0:09:21.124 It's really AI still, right, so that you have chechiptiza 0:09:21.164 --> 0:09:25.164 seventy percent of things well, and then you specialized applications 0:09:25.204 --> 0:09:28.964 for twenty percent, and then there are ten percent where 0:09:28.964 --> 0:09:31.684 you're not doing it very well. Like that would still 0:09:31.724 --> 0:09:34.204 be a big achievement, very disruptive achievement. I'm not sure 0:09:34.204 --> 0:09:37.044 it would kind of quite count as AGI. Like one 0:09:37.044 --> 0:09:38.844 thing you could do is, like some of these new 0:09:39.684 --> 0:09:44.524 models will say, okay, I can detect that you ask 0:09:44.644 --> 0:09:49.124 for something that requires mathematical precisions. So let's take a 0:09:49.124 --> 0:09:53.324 simpler example. Right, Let's say I want to take the 0:09:53.564 --> 0:10:00.004 distance between pick two cities, Boise, Idaho and Tampa, Florida. Right, 0:10:00.684 --> 0:10:04.164 that's pretty easy to calculate mathematically if you look up 0:10:04.684 --> 0:10:07.844 the latlong coordinates for Boise and Tampa, and then there 0:10:07.884 --> 0:10:09.604 are various formulas that. 0:10:09.604 --> 0:10:10.204 You can use. 0:10:10.324 --> 0:10:13.964 Right, So a pure data set where it's just crunching 0:10:14.044 --> 0:10:17.164 texts might not find that answer because there's not necessarily 0:10:17.164 --> 0:10:20.404 a lot of examples of like what's the distance between 0:10:20.404 --> 0:10:22.964 Boise and Tampa? Right, Whereas you would have that for 0:10:23.124 --> 0:10:26.644 like La and new York or something. However, you can 0:10:26.684 --> 0:10:31.164 set up kind of agentic meaning agent ais where they're like, okay, 0:10:31.444 --> 0:10:34.084 well this person asked a question that triggered a routine 0:10:34.084 --> 0:10:36.404 of mine. Where now I'm going to go and I 0:10:36.484 --> 0:10:39.284 know to search the internet for the coordinates or maybe 0:10:39.324 --> 0:10:41.964 it's stored somewhere right of Boise and Tampa. And then 0:10:42.004 --> 0:10:44.044 the formula. I know the formula and I can apply it, right, 0:10:44.124 --> 0:10:46.604 so like you have the scaffolding. It's sometimes called a 0:10:46.644 --> 0:10:49.924 different processes on top of one another. And I guess 0:10:49.964 --> 0:10:51.324 that kind of I guess that counts as I mean, 0:10:51.324 --> 0:10:54.244 it's kind of how humans solve problems, right, They're like, oh, 0:10:54.244 --> 0:10:56.484 and now I have to go look something up, right, 0:10:56.924 --> 0:11:00.924 It's still I think subtly different. So one implication I 0:11:00.924 --> 0:11:09.164 think this has is artificial general intelligence versus super intelligence ASI, 0:11:09.484 --> 0:11:14.084 sometimes called where there's like an explosion of intelligence just 0:11:14.124 --> 0:11:16.684 from kind of mining text or other data sources. I 0:11:17.724 --> 0:11:20.484 have actually become a little bit more skeptical of that, 0:11:20.764 --> 0:11:24.564 or I think it's presumptive to assume it can find that. 0:11:24.644 --> 0:11:26.524 I mean, like, let me give you another example, right, 0:11:26.604 --> 0:11:30.204 like one I had an idea for either a silver 0:11:30.244 --> 0:11:33.324 bulletin post or maybe even a risky business segment, right 0:11:33.324 --> 0:11:38.204 where like I asked, chatchipt deep research, design a meal 0:11:39.324 --> 0:11:46.964 with foods that have no clear precedent in other countries, right, huh? 0:11:46.444 --> 0:11:48.964 And I thought, and the idea was that, like, you know, 0:11:49.004 --> 0:11:53.724 at some point somebody invented pizza or spaghetti or whatever else, right, 0:11:53.884 --> 0:11:54.684 or or I. 0:11:54.604 --> 0:11:57.684 Mean at some point somebody invented bread and figured out 0:11:57.724 --> 0:12:00.404 that you can take flour and like bake shit with it. 0:12:00.404 --> 0:12:00.884 It's amazing. 0:12:01.004 --> 0:12:03.484 Yes, you have to be able to buy these ingredientstead 0:12:03.484 --> 0:12:06.844 of Trader Joe's or a Whole Foods. Right. I was 0:12:06.844 --> 0:12:08.924 just going to like hire professional chef and have a 0:12:08.924 --> 0:12:11.764 dinner part and ask how these creations went, right, And 0:12:11.804 --> 0:12:14.844 it's like and all they recipes were like here's salmon 0:12:14.964 --> 0:12:18.524 with miso paste, right, really creative and things like that, right, 0:12:18.564 --> 0:12:21.484 and like it didn't have any ability to like extrapolate 0:12:21.524 --> 0:12:25.404 beyond the data set. I'm sure these preparations are fine. 0:12:25.444 --> 0:12:28.644 It was like their little chefy, meaning like things that 0:12:28.804 --> 0:12:31.524 like have one too many ingredients, right, and like and 0:12:31.564 --> 0:12:34.884 I don't, yeah, look so bad chefe. 0:12:35.164 --> 0:12:36.004 Yeah. 0:12:36.084 --> 0:12:38.604 And meanwhile, I've been like I've been interviewing candidates for 0:12:38.684 --> 0:12:43.484 like a sports assistant position at Silver Bulletin, and one 0:12:43.524 --> 0:12:46.484 of the questions is, how are you using AI? These 0:12:46.484 --> 0:12:48.644 are very bright I want to say kids. A lot 0:12:48.684 --> 0:12:53.284 of them are are recent college grads, you know, your 0:12:53.284 --> 0:12:56.804 early twenties to mid thirties, basically right, and very technically 0:12:56.884 --> 0:13:00.204 proficient people. And asked them how they're using a GI 0:13:00.284 --> 0:13:03.724 and their experiences like kind of similar to mine. They're like, Yeah, 0:13:03.924 --> 0:13:07.804 for a certain task, it's very helpful. For certain task, 0:13:07.844 --> 0:13:10.284 it's somewhat helpful for certain tasks, not helpful at all. 0:13:10.284 --> 0:13:14.644 I can get you in trouble, right, which again contrast 0:13:14.684 --> 0:13:16.404 with the experience of people at the AI l apps 0:13:16.404 --> 0:13:19.644 to say, yeah, it's going to like displace all programming 0:13:20.204 --> 0:13:21.164 within a year or two. 0:13:21.524 --> 0:13:23.364 I mean, I know you should weigh in here. 0:13:23.644 --> 0:13:27.484 So I have an interesting kind of anecdote to add 0:13:27.524 --> 0:13:30.764 to this that's not poker related, but that talks about 0:13:30.844 --> 0:13:35.204 kind of some of this nuance and complexity and the 0:13:35.284 --> 0:13:39.124 fact that you know that all of these lms still 0:13:39.364 --> 0:13:43.524 are wanting in certain major respects. So I had the 0:13:43.604 --> 0:13:46.764 chance last week to speak to the CEO of one 0:13:46.804 --> 0:13:50.684 of the major AI companies. I won't name which company 0:13:50.724 --> 0:13:56.164 it was, but one of the big ones. And someone 0:13:56.204 --> 0:14:02.284 had asked him earlier, what was something that was surprising 0:14:02.484 --> 0:14:05.444 about the way that this AI functioned, and he said, 0:14:05.484 --> 0:14:10.164 you know, one of the mistakes that people is kind 0:14:10.204 --> 0:14:12.924 of they trust it, And what we know is that 0:14:13.044 --> 0:14:16.844 it's remarkably accurate ninety percent of the time, right, and 0:14:16.884 --> 0:14:19.444 then ten percent of the time it isn't. But the 0:14:19.484 --> 0:14:23.124 ten percent is getting harder and harder for humans to 0:14:23.204 --> 0:14:26.324 spot because it doesn't make errors in a way that's 0:14:26.364 --> 0:14:30.324 intuitive for people, right. It makes errors in different ways. 0:14:30.604 --> 0:14:33.964 It's like computer errors AI errs, that's not the way 0:14:33.964 --> 0:14:36.524 that the human mind makes errors. He's like, but like, 0:14:36.564 --> 0:14:38.884 it's fine because we have really smart people who can 0:14:38.924 --> 0:14:41.844 figure out where it's making errors and can work with 0:14:41.924 --> 0:14:44.324 it and can kind of calibrate it and help. And 0:14:44.404 --> 0:14:47.844 so my question, my follow up question, was, Okay, you 0:14:47.884 --> 0:14:52.204 know what happens when the new generation that's being brought 0:14:52.284 --> 0:14:56.244 up with this AI comes up and they've been educated 0:14:56.244 --> 0:14:58.244 with it and they've been using it the whole time, 0:14:58.604 --> 0:15:02.724 and they didn't necessarily get the same education that we got, right, 0:15:02.804 --> 0:15:06.324 because they aren't the incentives are different, and they're not 0:15:06.364 --> 0:15:09.764 getting the deep knowledge. They're not actually able to figure 0:15:09.764 --> 0:15:12.124 out what are the programming errors because this person was 0:15:12.164 --> 0:15:14.404 talking that, you know, some of the biggest potential of 0:15:14.404 --> 0:15:18.044 AIS and like programming and biology and kind of those 0:15:18.244 --> 0:15:20.364 those types of things, and I said, well, what if 0:15:20.364 --> 0:15:23.404 that person doesn't have the neuroscience and the biology background 0:15:23.444 --> 0:15:26.964 or the programming background. And what the CEO said was, 0:15:27.164 --> 0:15:29.884 this is a major problem, and what we're hoping is 0:15:29.884 --> 0:15:32.804 that we can develop even more advanced AI to help 0:15:33.484 --> 0:15:36.604 and to help fix the problems that AI is causing. 0:15:36.724 --> 0:15:38.484 Because he was like, yeah, this is a big problem 0:15:38.524 --> 0:15:41.324 and we're not quite sure, like we're hoping for the best, 0:15:41.364 --> 0:15:44.564 but we don't know. And the fact that this was 0:15:44.604 --> 0:15:49.284 our conversation didn't exactly inspire me to heights of thinking 0:15:49.644 --> 0:15:54.484 this is going to replace intelligence or become AGI or 0:15:54.564 --> 0:16:00.084 what did you call, Nate the other ASI artificial super intelligent? Yeah, 0:16:00.164 --> 0:16:04.084 artificial superintelligence. So it didn't inspire me that that was 0:16:04.124 --> 0:16:06.844 going to happen, and instead it was like an uh 0:16:06.844 --> 0:16:11.084 oh moment where like, what happened when the humans who 0:16:11.084 --> 0:16:14.084 are kind of helping push it along and make it better? 0:16:14.604 --> 0:16:18.524 When that generation, like when they age out and they're 0:16:18.884 --> 0:16:22.604 new people out there who who lack that sort of 0:16:22.644 --> 0:16:26.284 skill and expertise and who grew up with AI that 0:16:26.724 --> 0:16:29.724 is ninety percent accurate or even ninety five percent accurate, 0:16:29.964 --> 0:16:32.844 but they can't. It's harder and harder to spot that 0:16:32.964 --> 0:16:37.004 five and ten percent. And that to me was actually, like, 0:16:37.284 --> 0:16:39.604 it's a worrisome thing, and it's clearly worrisome to the 0:16:39.604 --> 0:16:43.964 people who are developing these technologies as well. And so 0:16:44.844 --> 0:16:47.844 that I think is something that dovetails with what you 0:16:47.884 --> 0:16:50.644 were talking about here. What happens Nate if all of 0:16:50.644 --> 0:16:54.284 a sudden, all poker players are training with chatchipt right, 0:16:54.564 --> 0:16:58.524 and that's that's actually great for us, but not great 0:16:58.564 --> 0:16:59.484 for the future of poker. 0:17:02.804 --> 0:17:14.924 And we'll be right back after this break. I use 0:17:15.564 --> 0:17:19.924 large language models for tasks that I have strong domain 0:17:20.204 --> 0:17:23.444 knowledge over and could do myself. But they're a labor 0:17:23.484 --> 0:17:28.724 saving device. Often they say substantial labor, right, Like I 0:17:28.804 --> 0:17:31.164 use them to copy edit articles I posted the newsletter, 0:17:31.204 --> 0:17:33.884 and I you know, I'm a fairly proficient user of 0:17:33.884 --> 0:17:36.844 the English language and like for programming, right Like you know, 0:17:37.084 --> 0:17:40.924 if you're like using AI to like design a website 0:17:41.764 --> 0:17:44.964 and some of the functionality breaks, right, that might be okay. 0:17:45.124 --> 0:17:47.284 You can patch it and fix it. Right, if I'm 0:17:47.324 --> 0:17:48.124 doing things. 0:17:47.884 --> 0:17:52.764 Involving modeling, I'm trying to project the value of basketball 0:17:52.804 --> 0:17:55.884 teams or football teams, or basketball players or football players. Right, 0:17:56.364 --> 0:17:58.164 and there's a bug where all of a sudden, like 0:17:58.284 --> 0:18:01.164 one of the worst players in the league is rated 0:18:01.164 --> 0:18:03.084 as being very good. Right. I mean, and you see 0:18:03.084 --> 0:18:06.684 this in poker, where like you know, AI is relying. 0:18:06.764 --> 0:18:11.684 You get punished in poker for sloppy application of imprecise 0:18:11.724 --> 0:18:15.484 application of heuristics. Like one of the things like the 0:18:15.524 --> 0:18:18.324 AI did so at first, I had it simulate just 0:18:18.444 --> 0:18:21.164 one hand, and it had like a backstore for each player. 0:18:21.164 --> 0:18:22.684 It fucked that up, right, then like Oka, I'm gonna 0:18:22.724 --> 0:18:25.164 use deep research now and have it simulate like an 0:18:25.284 --> 0:18:26.244 orbit of eight hands. 0:18:26.284 --> 0:18:27.124 So that was a little better. 0:18:27.164 --> 0:18:30.004 It took more compute time, right, Some of the hands 0:18:30.004 --> 0:18:32.524 were decent, but it you know, it didn't know how 0:18:32.564 --> 0:18:35.644 to like read aboard correctly and keep track of stacks. 0:18:35.684 --> 0:18:39.244 But like, but when it was over able to overcome 0:18:39.284 --> 0:18:43.124 those errors, it also had poor strategy and it gave 0:18:43.204 --> 0:18:45.724 various excuses. One thing it said is like, well, the 0:18:45.884 --> 0:18:48.764 quality of text based content on poker on the internet 0:18:49.444 --> 0:18:53.404 really sucks, Right, it's aware of that is it says 0:18:53.444 --> 0:18:56.124 it's worse than go or chess, although when I talk 0:18:56.124 --> 0:18:59.324 to people who know how AI's played chess, it's also 0:18:59.404 --> 0:19:02.644 a disaster. Even something like wordle I heard from a user. 0:19:02.724 --> 0:19:06.644 You know you think wordle is a word game and 0:19:06.684 --> 0:19:08.884 it does very well, but like it can't quite figure 0:19:08.884 --> 0:19:10.964 out the strike sure of the problem, right, it kind 0:19:10.964 --> 0:19:13.244 of was learning a little bit more by by a 0:19:13.364 --> 0:19:14.924 rote And to be clear, like this is a very 0:19:14.964 --> 0:19:18.204 good strategy for like many types of problems. And also 0:19:18.324 --> 0:19:20.524 if you were to say, okay, here's a bunch of 0:19:20.604 --> 0:19:23.404 we bought data from PIO solver or gto wizard. These 0:19:23.404 --> 0:19:27.084 are solvers if you don't know listeners or a database 0:19:27.124 --> 0:19:28.924 of high stakes hands. Like if you trained it on 0:19:28.964 --> 0:19:33.084 that and then had GPTs say okay, poker question, I'm 0:19:33.124 --> 0:19:35.684 going to call them especially trained database, then it would 0:19:35.724 --> 0:19:37.444 do well. And maybe if people are criticizing its for 0:19:38.004 --> 0:19:41.524 on poker, then the labs will start to do that. Right, 0:19:41.644 --> 0:19:44.764 but it's still kind of like slightly cheating from the 0:19:44.804 --> 0:19:48.724 standpoint of super intelligence. Also, it says like when you 0:19:48.724 --> 0:19:50.484 asked me to do a whole bunch of things at once, right, 0:19:50.684 --> 0:19:52.844 So I told it, give me characters, give me eight 0:19:52.924 --> 0:19:55.324 characters who have like they were all kind of like 0:19:55.404 --> 0:19:57.324 ethnic stereotypes, right, it was. 0:19:57.804 --> 0:19:59.604 They were very funny. 0:20:01.364 --> 0:20:04.084 It was kind of yeah and great. It was kind 0:20:04.124 --> 0:20:06.684 of at the one hand, so half the players were women. 0:20:06.764 --> 0:20:08.804 That was very feminist of it, right, But then they're 0:20:08.844 --> 0:20:12.364 all ethnic stereotypes of different kinds, right, and so it 0:20:12.404 --> 0:20:16.844 was kind of both woken and unwoke, right the opposite. Yeah, 0:20:17.364 --> 0:20:19.324 it was kind of good about like matching players playing 0:20:19.364 --> 0:20:24.004 styles with with their actions, but like, but it couldn't 0:20:24.084 --> 0:20:27.084 keep track of stack size. It's like, yeah, keep track 0:20:27.084 --> 0:20:28.684 of stacks. I have to store a bunch of stuff 0:20:28.684 --> 0:20:30.204 in memory, and then when you ask me to do 0:20:30.244 --> 0:20:33.324 a whole bunch of things at once, right, and like 0:20:33.684 --> 0:20:38.004 you know, and even including like so in the hand 0:20:38.044 --> 0:20:40.924 I show a sylver bulletin it like it misreads the board, 0:20:41.004 --> 0:20:43.844 doesn't recognize that a pair of nines has a higher 0:20:43.844 --> 0:20:46.884 two pair. If you ask it that out right, then 0:20:46.924 --> 0:20:49.804 it thinks about it more and gets the question right, right, 0:20:49.844 --> 0:20:52.324 But like it loses track of things in the context 0:20:52.844 --> 0:20:55.924 of these scaffolding situations where it has to keep track 0:20:55.924 --> 0:20:57.524 of a bunch of things at once and it doesn't 0:20:57.524 --> 0:21:00.964 know it doesn't know what to prioritize right. Like you know, 0:21:01.044 --> 0:21:02.524 you can make a lot of mistakes in poker if 0:21:02.524 --> 0:21:04.964 you don't know which hand be twitch hand. That's a 0:21:05.004 --> 0:21:07.404 more elementary mistake than anything else. 0:21:08.564 --> 0:21:10.764 Absolutely, So there are a few things that stand out 0:21:10.764 --> 0:21:11.204 to me here. 0:21:11.524 --> 0:21:11.804 One. 0:21:12.164 --> 0:21:16.724 I mean, it's a computer, right, like we humans have 0:21:16.844 --> 0:21:20.164 working memory capacity problems. It should be able to keep 0:21:20.204 --> 0:21:22.284 all these things like this is what it's good at. 0:21:22.604 --> 0:21:26.044 But the problem that you're kind of that you're illustrating 0:21:26.124 --> 0:21:29.084 is it doesn't know how to prioritize right, and that 0:21:29.084 --> 0:21:32.644 that is actually that is a major problem. And also 0:21:32.884 --> 0:21:36.564 when we keep saying thinking. But one of the first things, 0:21:36.724 --> 0:21:39.244 so when I was just learning poker, one of the 0:21:39.244 --> 0:21:41.364 first lessons I had with Phil Galfon, who is one 0:21:41.404 --> 0:21:43.004 of the people who kind of I worked with and 0:21:43.044 --> 0:21:47.404 who taught me a lot. I remember very early on 0:21:47.524 --> 0:21:49.444 he said, you know, I can give you a bunch 0:21:49.484 --> 0:21:51.604 of charts and a bunch of outputs, and you can 0:21:51.644 --> 0:21:55.204 memorize them and you'll be a very decent player very quickly. 0:21:55.284 --> 0:21:56.804 Like I know, I can give you this shit, you 0:21:56.844 --> 0:21:58.684 can memorize it, you can spit it back at me. 0:21:58.804 --> 0:22:01.164 You'll be fine, He's like, but I don't want you 0:22:01.204 --> 0:22:03.884 to do that, because that's going to make you a 0:22:03.924 --> 0:22:06.364 fine poker player, and you'll do fine in the short run, 0:22:06.444 --> 0:22:09.124 but you'll never be a great poker player because you 0:22:09.204 --> 0:22:12.004 have no idea why you're doing it. You're not actually thinking, 0:22:12.204 --> 0:22:15.964 you don't understand. You've just done a rote memorization, which 0:22:16.084 --> 0:22:18.484 is might make you more money in the short term, 0:22:18.564 --> 0:22:20.244 but in the long term is actually going to be 0:22:20.284 --> 0:22:24.084 detrimental to you because you're going to do stuff unthinkingly 0:22:24.204 --> 0:22:25.364 because you've memorized it. 0:22:25.644 --> 0:22:25.964 He said. 0:22:25.964 --> 0:22:28.804 What I want you to do is instead think through 0:22:28.804 --> 0:22:33.004 things and figure out, Okay, why right? For every single play, 0:22:33.524 --> 0:22:36.044