S1: I've had several thoughts on the Carpathian Dwarkesh conversation that
S1: took place in late October of 25, but the one
S1: that keeps haunting me is something Karpathy just kind of
S1: casually mentioned before moving on to another topic. I think
S1: it might be the biggest idea in the whole conversation.
S1: He was talking about human model similarities and he says
S1: humans collapse during the course of their lives. Children have
S1: an overfit, yet they will say stuff that will shock
S1: you because they're not yet collapsed. But we adults, we
S1: end up revisiting the same thoughts. We end up saying
S1: more and more the same stuff. The learning rate goes down,
S1: the collapse continues to get worse, and then everything deteriorates.
S1: End quote. Since my 20s, I've been terrified of this
S1: happening to me. It pierces my soul whenever my partner
S1: says things like, I knew you were going to say that. Ouch.
S1: Predictable humor or wit isn't another example. How many older
S1: people do you know who tell the same stories and
S1: jokes over and over? They watch the same shows. They
S1: listen to the same five bands, and then eventually, like
S1: 2 or 1, their aperture slowly shrinks until they die. Luckily,
S1: Karpathy gives a solution right after we have to find
S1: sources of entropy. When we were kids, everything was entropy
S1: because everything was new. So we were constantly changing our preferences,
S1: our behaviors, our language and everything. It made us fresh, unpredictable,
S1: which is highly related to the concept I'm obsessed with
S1: from Shannon's theory of information, which in his model defines
S1: information as the part of transmission that isn't repeated or noise.
S1: I think about this constantly when I'm giving talks or
S1: participating in panels or whatever, or when I'm watching someone
S1: else do so. The main thing I'm asking myself, especially
S1: for my own content, is how much of this is new?
S1: How often will I'm presenting this? Will the viewer be
S1: pleasantly surprised? If the answer is not very often I
S1: redo it or I start over. I'm actively doing a
S1: bunch of stuff in addition to pathological reading to maximize
S1: entropy in my life. I'm reading a lot of old
S1: books on writing, like rhetorical figures and stuff like that,
S1: to try to get fresh phrases into my mind. I
S1: regularly reread and listen to Christopher Hitchens books and debates.
S1: Just having exposure to that level of non cliché language.
S1: And I'm currently building in cloud code a skill called
S1: Increase Entropy that incorporates all of this old and fresh
S1: language like a particle accelerator. So I can point it
S1: at a thought or a piece of content and basically
S1: come up with novel ways of saying the same thing.
S1: So I give it the way that I would say
S1: it in a kind of like just breaks me out
S1: of my mold. I even went so far in 2024
S1: to create an AI prompt in fabric that would rate talks, blogs, panels,
S1: or whatever for wows per minute, meaning how often a
S1: given piece of content surprised the audience. I mean, this
S1: was a problem before AI, and now many are delegating
S1: even more and more of their thinking to a system
S1: that learns by crunching mediocrity from the internet. I can
S1: see things getting way worse in this respect. I guess
S1: it's somewhat comforting that this happens to both AI models
S1: and to people. It makes the whole thing more human somehow.
S1: And hearing Karpathy say it so plainly was jarring to
S1: me in a pleasant way. At least for us humans.
S1: The solution seems something like recognize that this is a
S1: problem that starts for everyone in there, probably like mid
S1: to late 20s, and constantly seek and consume sources of
S1: novelty and freshness to maintain young mind.