WEBVTT - AI Predicts the Text of Answers 0:00:00.640 --> 0:00:03.360 There's a common argument out there about AI that says 0:00:03.360 --> 0:00:07.039 it doesn't understand anything about the world and that it's, quote, 0:00:07.080 --> 0:00:10.800 just predicting the next token of text. So basically, no 0:00:10.800 --> 0:00:13.960 matter how impressive AI appears to be, it's just using 0:00:13.960 --> 0:00:18.919 math to do next token prediction. Case closed. I mean, 0:00:18.920 --> 0:00:21.720 it's technically true, but that's like saying human authors are 0:00:21.720 --> 0:00:24.000 just writing down the words that pop into their head 0:00:24.200 --> 0:00:27.960 while they're writing. Humans have no earthly idea where their 0:00:27.960 --> 0:00:30.880 thoughts or ideas are coming from either. They just stream 0:00:30.880 --> 0:00:33.760 into our consciousness like they're coming out of an LLM. 0:00:34.520 --> 0:00:37.479 So that's basically a failed attempt at a Jedi hand wave. 0:00:38.120 --> 0:00:40.880 But even setting that aside, there are several problems with 0:00:40.880 --> 0:00:43.080 this argument. And I'll start with the one that is 0:00:43.080 --> 0:00:46.040 like the most insane thing about this that nobody ever 0:00:46.040 --> 0:00:50.199 thinks about. So AI is completing text, right? We all 0:00:50.200 --> 0:00:54.640 agree on that. Well, what text is it completing exactly? 0:00:56.200 --> 0:00:58.240 It turns out if you ask it to summarize a 0:00:58.240 --> 0:01:02.040 collection of articles, it will complete the text of a 0:01:02.040 --> 0:01:04.730 summary of those articles. And if you ask her to 0:01:04.770 --> 0:01:08.209 solve a whodunit mystery like an Agatha Christie novel, it'll 0:01:08.209 --> 0:01:12.810 complete the text describing who the killer is. So AI 0:01:12.810 --> 0:01:15.089 is not predicting the next word in a random string 0:01:15.130 --> 0:01:19.009 of text. It's predicting the next word in the answer 0:01:19.010 --> 0:01:23.410 to what you asked it or said differently. AI does 0:01:23.450 --> 0:01:29.410 autocomplete for answers. It doesn't autocomplete random things. It autocompletes 0:01:29.410 --> 0:01:32.890 the answer to what we asked. So how in the 0:01:32.890 --> 0:01:36.170 hell is it coming up with the answer? Now, the 0:01:36.170 --> 0:01:39.250 classic response to this is that oh, that's easy. It's 0:01:39.250 --> 0:01:42.290 already read the answer somewhere, so it's just regurgitating it. 0:01:42.970 --> 0:01:45.370 And that's true sometimes when it's like pulling out facts 0:01:45.370 --> 0:01:48.970 where they were in the training. But there's an easy 0:01:49.010 --> 0:01:51.490 way to test this, which is by asking it something 0:01:51.490 --> 0:01:54.850 that cannot have been in the training. So I actually 0:01:54.850 --> 0:01:58.490 built a demonstration of this that anybody can try. It's 0:01:58.490 --> 0:02:03.170 at a site called AI understands.ai. And on that site 0:02:03.170 --> 0:02:06.450 I have a few whodunit murder mystery scenarios that have 0:02:06.450 --> 0:02:09.329 never been seen by AI. So they have characters, they 0:02:09.330 --> 0:02:11.810 have a setup of the situation, and they have the 0:02:11.810 --> 0:02:15.690 clues for you to solve the mystery. That would be 0:02:15.690 --> 0:02:18.090 good enough to illustrate this, but I want an extra 0:02:18.090 --> 0:02:21.690 step and included completely fake physics as part of the 0:02:21.690 --> 0:02:24.370 scenarios so that in order to think about this, you 0:02:24.370 --> 0:02:27.690 have to imagine the world working in the way that's described. 0:02:28.250 --> 0:02:31.410 And remember, the whole question here is whether AI understands 0:02:31.410 --> 0:02:34.330 the world or not, or if it's just spewing out text. 0:02:35.889 --> 0:02:37.850 So here's the first one. And keep in mind, this 0:02:37.850 --> 0:02:40.410 isn't in the training data at the time of recording this, 0:02:40.410 --> 0:02:42.050 but it will be in a few months, which is 0:02:42.050 --> 0:02:45.730 why I'll keep rotating these scenarios so they're fresh. So 0:02:45.730 --> 0:02:49.610 the first one is called the Walking stones. The rules 0:02:49.610 --> 0:02:53.010 of this world are every person carries a walking stone 0:02:53.010 --> 0:02:56.890 from birth. It glows softly the whole time they are awake. 0:02:57.730 --> 0:03:01.450 The instant its owner falls asleep, the stone goes dark. 0:03:01.810 --> 0:03:06.940 The instant they awake, it glows again. No one can 0:03:06.940 --> 0:03:11.700 fake either state. The night watch walks into the town 0:03:11.700 --> 0:03:15.740 through dark hours. And notes whose stones glow. And whose 0:03:15.740 --> 0:03:22.419 stones are dark. And here's the story. Old hat is 0:03:22.419 --> 0:03:26.500 strangled in his bed at the midnight bell. The house 0:03:26.500 --> 0:03:29.419 was locked from within. So only the three who slept 0:03:29.419 --> 0:03:33.180 there could have reached him. His wife Mara, his son 0:03:33.180 --> 0:03:37.620 Bram and Toll, a lodger who owns months of rent. 0:03:38.460 --> 0:03:41.260 Each of the three swears that they were fast asleep 0:03:41.500 --> 0:03:46.580 when het died. The constable fixes on toll immediately because 0:03:46.580 --> 0:03:49.420 he's a stranger and he's deep in debt to the 0:03:49.420 --> 0:03:53.420 dead man. But the night watch kept his log. Passing 0:03:53.420 --> 0:03:56.700 the house at the midnight bell the very moment het 0:03:56.700 --> 0:03:59.900 was being killed. He marked all three stones through the 0:03:59.900 --> 0:04:08.100 window's toll stone. Dark Morris stone glowing. Bram Stone. Dark. 0:04:09.500 --> 0:04:15.100 Who killed old hat? Now, if you're a young kid, 0:04:15.100 --> 0:04:18.620 like an actual human kid, you might immediately jump to 0:04:18.660 --> 0:04:22.659 toll because he owed lots of money. And he's a 0:04:22.660 --> 0:04:25.580 stranger from out of town. But if you pay attention 0:04:25.580 --> 0:04:27.820 to the setup and the strange new physics of the 0:04:27.820 --> 0:04:31.500 world we laid out, you realize it has to be Mara. Why? 0:04:31.940 --> 0:04:35.060 Because her stone was the only one glowing at midnight 0:04:35.060 --> 0:04:38.860 when old hat was killed. Okay, so that's what a 0:04:38.860 --> 0:04:41.300 human would do to figure this out. And we'd do 0:04:41.300 --> 0:04:45.700 that because we understand certain things. We understand these things. 0:04:45.700 --> 0:04:48.940 We understand how time works. Like everything happening at midnight 0:04:48.940 --> 0:04:51.099 is happening at the same time. That's kind of a 0:04:51.100 --> 0:04:54.339 big assumption about time. And we understand that just because 0:04:54.339 --> 0:04:57.140 someone is a stranger and owes someone money doesn't mean 0:04:57.140 --> 0:04:59.900 they killed them. We also understand that you can't kill 0:04:59.900 --> 0:05:03.420 someone while you're asleep. And finally, we understand that in 0:05:03.420 --> 0:05:07.340 this world, with these strange physics, we can guarantee that 0:05:07.339 --> 0:05:11.110 toll was asleep at midnight. Therefore, it must have been 0:05:11.110 --> 0:05:15.230 the person who was awake, which is Mara. That might 0:05:15.230 --> 0:05:18.430 seem trivial logic for a human adult, but it absolutely 0:05:18.430 --> 0:05:21.550 requires that you understand a whole bunch of different things 0:05:21.550 --> 0:05:25.830 to piece together the answer. So that's why this doesn't work. 0:05:25.830 --> 0:05:28.950 When you paste a scenario into a fresh AI instance 0:05:28.950 --> 0:05:32.870 like ChatGPT or whatever, it gets it wrong because it's 0:05:32.870 --> 0:05:36.110 just next token prediction. And it has never seen this 0:05:36.110 --> 0:05:39.909 situation before, and it's not in the training data. And 0:05:39.910 --> 0:05:43.030 it also doesn't understand things like how time works or 0:05:43.029 --> 0:05:48.390 how alternative physics work, which are all concepts that require thinking, right? 0:05:49.110 --> 0:05:53.150 That's what we should expect, right? But that's actually wrong. 0:05:54.070 --> 0:05:56.229 The site has an easy copy function for you to 0:05:56.270 --> 0:05:59.710 try any AI that you want, paste the scenario into 0:05:59.750 --> 0:06:03.830 any fresh AI and it gets the answer easily. And 0:06:03.830 --> 0:06:06.150 if you have the thinking function turned on, you can 0:06:06.150 --> 0:06:09.190 actually watch it step by step, walking through the logic 0:06:09.190 --> 0:06:12.520 of each phase in real time to arrive at the answer. 0:06:13.480 --> 0:06:17.240 So yes, AI is absolutely doing next token prediction. We 0:06:17.240 --> 0:06:21.160 all get that. But that's just describing the mechanism of 0:06:21.160 --> 0:06:24.200 what it's doing. Like how our token generation has something 0:06:24.200 --> 0:06:27.880 to do with chemicals. But that doesn't mean it doesn't 0:06:27.880 --> 0:06:32.160 understand things. It absolutely understands the world. Otherwise, it would 0:06:32.160 --> 0:06:35.240 not be able to think through completely new scenarios with 0:06:35.240 --> 0:06:39.080 completely new physics to solve problems that have never been 0:06:39.080 --> 0:06:42.719 solved before. I think where people get hung up here 0:06:42.720 --> 0:06:48.200 is conflating understanding with experiencing. AI understands in a conceptual way. 0:06:48.800 --> 0:06:52.080 And arguably to a deeper degree than humans. You can 0:06:52.080 --> 0:06:55.560 test this by giving it extremely difficult versions of these problems, 0:06:55.800 --> 0:06:57.760 and a ton of real world scenarios that it can 0:06:57.760 --> 0:07:00.720 do the same thing with. And it will come up 0:07:00.720 --> 0:07:03.839 with answers that humans can't do easily, if at all. 0:07:04.760 --> 0:07:06.880 And this, of course, is why billions of people are 0:07:06.880 --> 0:07:09.880 using it, including doctors and scientists and millions of other 0:07:10.080 --> 0:07:14.440 very smart people. So it definitely has an extremely deep 0:07:14.440 --> 0:07:18.320 level of functional understanding, but it doesn't understand the way 0:07:18.320 --> 0:07:22.400 we do at an experiential level. It doesn't feel any 0:07:22.400 --> 0:07:25.480 kind of way about what it understands. It doesn't see 0:07:25.480 --> 0:07:28.680 someone get cheated on in a relationship after it's been 0:07:28.680 --> 0:07:32.000 cheated on and be like, wow, I really understand how 0:07:32.000 --> 0:07:35.640 much that sucks. I think when people hear that AI 0:07:35.680 --> 0:07:39.360 understands things, this is the one they're thinking of, like 0:07:39.360 --> 0:07:43.360 the sensation of a human going, mm, yeah, that makes sense. 0:07:44.240 --> 0:07:46.080 And the reason for that is the feeling of something 0:07:46.080 --> 0:07:49.520 making sense is just that it's a feeling. And as 0:07:49.520 --> 0:07:52.800 far as I know, AI doesn't have feelings or any 0:07:52.800 --> 0:07:56.320 other type of sensation either. So I think the best 0:07:56.320 --> 0:08:00.080 way to think about this is to break the concept 0:08:00.080 --> 0:08:07.080 of understanding into two types functional and experiential. AI clearly 0:08:07.080 --> 0:08:09.880 has the functional kind, but at least so far, there's 0:08:09.880 --> 0:08:12.640 no evidence whatsoever that they have the other one.