WEBVTT - Humans Need Entropy 0:00:00.200 --> 0:00:04.000 I've had several thoughts on the Carpathian Dwarkesh conversation that 0:00:04.000 --> 0:00:08.200 took place in late October of 25, but the one 0:00:08.200 --> 0:00:10.760 that keeps haunting me is something Karpathy just kind of 0:00:10.800 --> 0:00:14.520 casually mentioned before moving on to another topic. I think 0:00:14.520 --> 0:00:17.160 it might be the biggest idea in the whole conversation. 0:00:18.040 --> 0:00:22.640 He was talking about human model similarities and he says 0:00:23.400 --> 0:00:28.160 humans collapse during the course of their lives. Children have 0:00:28.160 --> 0:00:31.560 an overfit, yet they will say stuff that will shock 0:00:31.560 --> 0:00:37.000 you because they're not yet collapsed. But we adults, we 0:00:37.000 --> 0:00:39.920 end up revisiting the same thoughts. We end up saying 0:00:39.920 --> 0:00:43.600 more and more the same stuff. The learning rate goes down, 0:00:43.960 --> 0:00:48.440 the collapse continues to get worse, and then everything deteriorates. 0:00:50.400 --> 0:00:54.880 End quote. Since my 20s, I've been terrified of this 0:00:54.880 --> 0:00:58.280 happening to me. It pierces my soul whenever my partner 0:00:58.280 --> 0:01:03.840 says things like, I knew you were going to say that. Ouch. 0:01:04.440 --> 0:01:10.339 Predictable humor or wit isn't another example. How many older 0:01:10.340 --> 0:01:13.020 people do you know who tell the same stories and 0:01:13.020 --> 0:01:16.420 jokes over and over? They watch the same shows. They 0:01:16.420 --> 0:01:19.540 listen to the same five bands, and then eventually, like 0:01:19.540 --> 0:01:25.940 2 or 1, their aperture slowly shrinks until they die. Luckily, 0:01:26.380 --> 0:01:31.140 Karpathy gives a solution right after we have to find 0:01:31.340 --> 0:01:36.179 sources of entropy. When we were kids, everything was entropy 0:01:36.180 --> 0:01:41.740 because everything was new. So we were constantly changing our preferences, 0:01:41.740 --> 0:01:46.500 our behaviors, our language and everything. It made us fresh, unpredictable, 0:01:47.140 --> 0:01:49.980 which is highly related to the concept I'm obsessed with 0:01:49.980 --> 0:01:54.100 from Shannon's theory of information, which in his model defines 0:01:54.100 --> 0:01:59.460 information as the part of transmission that isn't repeated or noise. 0:02:00.740 --> 0:02:03.220 I think about this constantly when I'm giving talks or 0:02:03.220 --> 0:02:07.340 participating in panels or whatever, or when I'm watching someone 0:02:07.340 --> 0:02:11.040 else do so. The main thing I'm asking myself, especially 0:02:11.040 --> 0:02:14.520 for my own content, is how much of this is new? 0:02:14.760 --> 0:02:17.359 How often will I'm presenting this? Will the viewer be 0:02:17.360 --> 0:02:20.960 pleasantly surprised? If the answer is not very often I 0:02:21.000 --> 0:02:24.720 redo it or I start over. I'm actively doing a 0:02:24.720 --> 0:02:29.280 bunch of stuff in addition to pathological reading to maximize 0:02:29.280 --> 0:02:33.600 entropy in my life. I'm reading a lot of old 0:02:33.600 --> 0:02:37.480 books on writing, like rhetorical figures and stuff like that, 0:02:37.760 --> 0:02:40.760 to try to get fresh phrases into my mind. I 0:02:40.800 --> 0:02:44.960 regularly reread and listen to Christopher Hitchens books and debates. 0:02:45.560 --> 0:02:49.400 Just having exposure to that level of non cliché language. 0:02:50.040 --> 0:02:53.880 And I'm currently building in cloud code a skill called 0:02:53.880 --> 0:02:57.480 Increase Entropy that incorporates all of this old and fresh 0:02:57.480 --> 0:03:01.280 language like a particle accelerator. So I can point it 0:03:01.280 --> 0:03:05.200 at a thought or a piece of content and basically 0:03:05.200 --> 0:03:08.280 come up with novel ways of saying the same thing. 0:03:08.680 --> 0:03:10.239 So I give it the way that I would say 0:03:10.240 --> 0:03:13.360 it in a kind of like just breaks me out 0:03:13.400 --> 0:03:16.940 of my mold. I even went so far in 2024 0:03:16.940 --> 0:03:21.340 to create an AI prompt in fabric that would rate talks, blogs, panels, 0:03:21.340 --> 0:03:25.340 or whatever for wows per minute, meaning how often a 0:03:25.340 --> 0:03:29.620 given piece of content surprised the audience. I mean, this 0:03:29.620 --> 0:03:33.660 was a problem before AI, and now many are delegating 0:03:33.660 --> 0:03:36.780 even more and more of their thinking to a system 0:03:36.780 --> 0:03:41.140 that learns by crunching mediocrity from the internet. I can 0:03:41.140 --> 0:03:46.500 see things getting way worse in this respect. I guess 0:03:46.500 --> 0:03:50.660 it's somewhat comforting that this happens to both AI models 0:03:50.660 --> 0:03:54.380 and to people. It makes the whole thing more human somehow. 0:03:55.060 --> 0:03:58.580 And hearing Karpathy say it so plainly was jarring to 0:03:58.620 --> 0:04:02.100 me in a pleasant way. At least for us humans. 0:04:02.100 --> 0:04:05.620 The solution seems something like recognize that this is a 0:04:05.620 --> 0:04:10.340 problem that starts for everyone in there, probably like mid 0:04:10.380 --> 0:04:14.620 to late 20s, and constantly seek and consume sources of 0:04:14.620 --> 0:04:18.740 novelty and freshness to maintain young mind.