1 00:00:00,200 --> 00:00:04,000 S1: I've had several thoughts on the Carpathian Dwarkesh conversation that 2 00:00:04,000 --> 00:00:08,200 S1: took place in late October of 25, but the one 3 00:00:08,200 --> 00:00:10,760 S1: that keeps haunting me is something Karpathy just kind of 4 00:00:10,800 --> 00:00:14,520 S1: casually mentioned before moving on to another topic. I think 5 00:00:14,520 --> 00:00:17,160 S1: it might be the biggest idea in the whole conversation. 6 00:00:18,040 --> 00:00:22,640 S1: He was talking about human model similarities and he says 7 00:00:23,400 --> 00:00:28,160 S1: humans collapse during the course of their lives. Children have 8 00:00:28,160 --> 00:00:31,560 S1: an overfit, yet they will say stuff that will shock 9 00:00:31,560 --> 00:00:37,000 S1: you because they're not yet collapsed. But we adults, we 10 00:00:37,000 --> 00:00:39,920 S1: end up revisiting the same thoughts. We end up saying 11 00:00:39,920 --> 00:00:43,600 S1: more and more the same stuff. The learning rate goes down, 12 00:00:43,960 --> 00:00:48,440 S1: the collapse continues to get worse, and then everything deteriorates. 13 00:00:50,400 --> 00:00:54,880 S1: End quote. Since my 20s, I've been terrified of this 14 00:00:54,880 --> 00:00:58,280 S1: happening to me. It pierces my soul whenever my partner 15 00:00:58,280 --> 00:01:03,840 S1: says things like, I knew you were going to say that. Ouch. 16 00:01:04,440 --> 00:01:10,339 S1: Predictable humor or wit isn't another example. How many older 17 00:01:10,340 --> 00:01:13,020 S1: people do you know who tell the same stories and 18 00:01:13,020 --> 00:01:16,420 S1: jokes over and over? They watch the same shows. They 19 00:01:16,420 --> 00:01:19,540 S1: listen to the same five bands, and then eventually, like 20 00:01:19,540 --> 00:01:25,940 S1: 2 or 1, their aperture slowly shrinks until they die. Luckily, 21 00:01:26,380 --> 00:01:31,140 S1: Karpathy gives a solution right after we have to find 22 00:01:31,340 --> 00:01:36,179 S1: sources of entropy. When we were kids, everything was entropy 23 00:01:36,180 --> 00:01:41,740 S1: because everything was new. So we were constantly changing our preferences, 24 00:01:41,740 --> 00:01:46,500 S1: our behaviors, our language and everything. It made us fresh, unpredictable, 25 00:01:47,140 --> 00:01:49,980 S1: which is highly related to the concept I'm obsessed with 26 00:01:49,980 --> 00:01:54,100 S1: from Shannon's theory of information, which in his model defines 27 00:01:54,100 --> 00:01:59,460 S1: information as the part of transmission that isn't repeated or noise. 28 00:02:00,740 --> 00:02:03,220 S1: I think about this constantly when I'm giving talks or 29 00:02:03,220 --> 00:02:07,340 S1: participating in panels or whatever, or when I'm watching someone 30 00:02:07,340 --> 00:02:11,040 S1: else do so. The main thing I'm asking myself, especially 31 00:02:11,040 --> 00:02:14,520 S1: for my own content, is how much of this is new? 32 00:02:14,760 --> 00:02:17,359 S1: How often will I'm presenting this? Will the viewer be 33 00:02:17,360 --> 00:02:20,960 S1: pleasantly surprised? If the answer is not very often I 34 00:02:21,000 --> 00:02:24,720 S1: redo it or I start over. I'm actively doing a 35 00:02:24,720 --> 00:02:29,280 S1: bunch of stuff in addition to pathological reading to maximize 36 00:02:29,280 --> 00:02:33,600 S1: entropy in my life. I'm reading a lot of old 37 00:02:33,600 --> 00:02:37,480 S1: books on writing, like rhetorical figures and stuff like that, 38 00:02:37,760 --> 00:02:40,760 S1: to try to get fresh phrases into my mind. I 39 00:02:40,800 --> 00:02:44,960 S1: regularly reread and listen to Christopher Hitchens books and debates. 40 00:02:45,560 --> 00:02:49,400 S1: Just having exposure to that level of non cliché language. 41 00:02:50,040 --> 00:02:53,880 S1: And I'm currently building in cloud code a skill called 42 00:02:53,880 --> 00:02:57,480 S1: Increase Entropy that incorporates all of this old and fresh 43 00:02:57,480 --> 00:03:01,280 S1: language like a particle accelerator. So I can point it 44 00:03:01,280 --> 00:03:05,200 S1: at a thought or a piece of content and basically 45 00:03:05,200 --> 00:03:08,280 S1: come up with novel ways of saying the same thing. 46 00:03:08,680 --> 00:03:10,239 S1: So I give it the way that I would say 47 00:03:10,240 --> 00:03:13,360 S1: it in a kind of like just breaks me out 48 00:03:13,400 --> 00:03:16,940 S1: of my mold. I even went so far in 2024 49 00:03:16,940 --> 00:03:21,340 S1: to create an AI prompt in fabric that would rate talks, blogs, panels, 50 00:03:21,340 --> 00:03:25,340 S1: or whatever for wows per minute, meaning how often a 51 00:03:25,340 --> 00:03:29,620 S1: given piece of content surprised the audience. I mean, this 52 00:03:29,620 --> 00:03:33,660 S1: was a problem before AI, and now many are delegating 53 00:03:33,660 --> 00:03:36,780 S1: even more and more of their thinking to a system 54 00:03:36,780 --> 00:03:41,140 S1: that learns by crunching mediocrity from the internet. I can 55 00:03:41,140 --> 00:03:46,500 S1: see things getting way worse in this respect. I guess 56 00:03:46,500 --> 00:03:50,660 S1: it's somewhat comforting that this happens to both AI models 57 00:03:50,660 --> 00:03:54,380 S1: and to people. It makes the whole thing more human somehow. 58 00:03:55,060 --> 00:03:58,580 S1: And hearing Karpathy say it so plainly was jarring to 59 00:03:58,620 --> 00:04:02,100 S1: me in a pleasant way. At least for us humans. 60 00:04:02,100 --> 00:04:05,620 S1: The solution seems something like recognize that this is a 61 00:04:05,620 --> 00:04:10,340 S1: problem that starts for everyone in there, probably like mid 62 00:04:10,380 --> 00:04:14,620 S1: to late 20s, and constantly seek and consume sources of 63 00:04:14,620 --> 00:04:18,740 S1: novelty and freshness to maintain young mind.