Speaker 1: You know that moment in the horror movie where the
Speaker 1: monster is coming closer but the person on screen doesn't
Speaker 1: see it. Why does that drive you crazy? And what
Speaker 1: does that teach us about brains? What is theory of
Speaker 1: mind and why is it so important for everyone from
Speaker 1: poker players to con men, to stage magicians to novelists.
Speaker 1: We're going to talk about a very fundamental skill of
Speaker 1: human brains today, and as impressive as AI is currently,
Speaker 1: we're going to ask the question of whether computers can
Speaker 1: replicate this right now or whether it is beyond their
Speaker 1: skill set. Welcome to Inner Cosmos with me David Eagleman.
Speaker 1: I'm a neuroscientist and an author at Stanford, and in
Speaker 1: these episodes we sail deeply into our three pound universe
Speaker 1: to understand why and how our lives look the way
Speaker 1: they do. Today's episode is about what it takes to
Speaker 1: understand other people, how your brain does it, and whether
Speaker 1: computers could do it. So imagine this. You're walking down
Speaker 1: the street and you see someone frantically searching their pockets
Speaker 1: and looking around with furrowed brows in a tight frown.
Speaker 1: So without them saying a word, you can infer that
Speaker 1: they might have lost something important. Maybe it's his keys.
Speaker 1: Your brain can easily make a good guess about another
Speaker 1: person's mental state just from looking at their actions. We
Speaker 1: are inferring something about what is going on in that
Speaker 1: person's head. But it's more than just pattern matching. It's
Speaker 1: not simply that your brain has seen lots of people
Speaker 1: patting their pockets and you talked with them afterwards, and
Speaker 1: you figured out why they were doing that, and you
Speaker 1: detected a pattern, and you memorized, ah, okay, that pattern
Speaker 1: equals that problem. Instead, you have the ability to imagine
Speaker 1: yourself in their situation. You can mentally slip into their
Speaker 1: shoes and ask, what would I be thinking if I
Speaker 1: were patting my pockets and frantically searching around me? And
Speaker 1: maybe you see something else. You see a kid there
Speaker 1: around the corner, and the kid is peeking around the
Speaker 1: corner at the man patting his pockets, and the child
Speaker 1: is giggling. Now, why is the kid giggling while the
Speaker 1: guy is so obviously worried, Well, it probably strikes you
Speaker 1: that he's hiding something from the guy. You see that
Speaker 1: the kid is not running away. Instead, he's standing in
Speaker 1: such a way that he'll be spotted. Now it's pretty
Speaker 1: obvious what's happening here. You can step into the man's
Speaker 1: head to feel the worry, and you can step into
Speaker 1: the kid's head to recognize that he feels like he's
Speaker 1: playing a game, even if it doesn't strike you as
Speaker 1: so funny. Then you catch the guy meet with the
Speaker 1: kid for just a fraction of a second, which sends
Speaker 1: the kid into fits of laughter, and you realize the
Speaker 1: man is just playing along. Now, how did you decide
Speaker 1: what is going on in the heads of these two Again,
Speaker 1: it's not as though you memorized an algorithm here. Okay,
Speaker 1: if there's eye contact, then there's one interpretation. If there's
Speaker 1: no eye contact, then a totally different interpretation. To appreciate
Speaker 1: how complex this mind reading is that you just did,
Speaker 1: just imagine that you're a space alien watching this scene
Speaker 1: from your spaceship. You would be totally confused. You would
Speaker 1: have no idea what's going on in this weird scene
Speaker 1: because you don't know what it is to be a human.
Speaker 1: Here's another analogy to appreciate this. Think about the way
Speaker 1: that you, as a human might watch fish. You really
Speaker 1: don't understand what the heck they're doing. One fish suddenly
Speaker 1: starts swimming faster, and another starts swimming in circles, and
Speaker 1: one starts flapping its gills faster, and one moves up
Speaker 1: towards the surface. It's all just weird fish behavior to you.
Speaker 1: You don't know how to read any of it. It's
Speaker 1: just fish stuff, and you're not able to immediately construct
Speaker 1: a story about the meaning of any of this. And
Speaker 1: that's what it's like to be this space alien watching
Speaker 1: this guy checking his pockets and the child giggling. Now,
Speaker 1: what allows us, as opposed to the space alien, to
Speaker 1: be so good at reading our fellow humans. This is
Speaker 1: what psychologists and neuroscientists call theory of mind, and that's
Speaker 1: what we're talking about today. Theory of mind is the
Speaker 1: ability to understand that other people have their own thoughts
Speaker 1: and feelings and beliefs that are different from yours. It's
Speaker 1: the ability to recognize that others have their own perspectives.
Speaker 1: It's the ability to attribut mute mental states to other people,
Speaker 1: like what their intentions are, or their desires, or their emotions,
Speaker 1: or what they know or don't know. And theory of
Speaker 1: mind is a key cognitive skill that allows us to
Speaker 1: interact with other people in a very rich and nuanced way.
Speaker 1: Just think about how pervasive this skill is in everything
Speaker 1: we do. So take sarcasm. When your friend makes a
Speaker 1: sarcastic comment, you can recognize that her words don't match
Speaker 1: her true intention. So, for example, if she says, oh, awesome,
Speaker 1: more traffic, I love traffic, you infer that she's not
Speaker 1: actually pleased. This requires understanding her mental state that she
Speaker 1: is irritated not happy. Now, if you were Siri or Alexa,
Speaker 1: you wouldn't be able to recognize anything but the words.
Speaker 1: You wouldn't understand anything about the mind behind the words.
Speaker 1: So we're going to talk about how brains do it
Speaker 1: and whether or not computers can do it. But before
Speaker 1: we go there, we're going to take a few minutes
Speaker 1: to really appreciate how the skill is everywhere in what
Speaker 1: we do. For example, just think about different professions. So
Speaker 1: detectives use theory of mind all the time. Did mister
Speaker 1: Jones know that the food had gone bad when he
Speaker 1: sold it? Did mister Smith know that his boss was
Speaker 1: involved with organized crime or was he acting with no knowledge?
Speaker 1: Or more generally, if they want to know if someone
Speaker 1: is lying, it usually helps to step into their shoes
Speaker 1: and think about what that person knows or doesn't know.
Speaker 1: Magicians use theory of mind. They know that if they
Speaker 1: move their hand in an arc, your attention is going
Speaker 1: to follow that, and therefore they know what you won't
Speaker 1: see them do. They know that even though they know
Speaker 1: something happened, like the card dropped into their sleeve, they
Speaker 1: know that you don't know that. They always keep your
Speaker 1: point of view, your beliefs, at the forefront of their mind.
Speaker 1: Con Men do this. They listen to your words and
Speaker 1: they read your body language to gather what you know
Speaker 1: and don't know, and therefore what buttons they should push next.
Speaker 1: Psychiatrists and psychologists always use theory of mind to understand
Speaker 1: what is being expressed from the patient's point of view,
Speaker 1: In other words, what the person believes, whether or not
Speaker 1: it's what the therapist believes. I'll give you another example.
Speaker 1: My friend Maddie is a professional poker player, and he
Speaker 1: describes poker playing like this. He says, when you're learning
Speaker 1: to play poker, you think about the cards you have
Speaker 1: in your hand. As you get better, you think about
Speaker 1: your hand and also what the other person is thinking.
Speaker 1: And as you get even better, you think about what
Speaker 1: the other person is thinking your thinking, and when you
Speaker 1: get to the professional levels, you're thinking about what he thinks,
Speaker 1: you think he thinks, and people who are real pros
Speaker 1: can think five or six levels deep on this. All
Speaker 1: of this is theory of mind, and theory of mind
Speaker 1: is key when you're teaching something. For example, parents know
Speaker 1: that their children can't understand certain things. For example, the
Speaker 1: child needs to get that smallpox shot, even though to
Speaker 1: the child that's nothing but scary and he simply doesn't
Speaker 1: have the capacity to think about the future benefits that
Speaker 1: will accrue. Or the school teacher can only hope to
Speaker 1: educate her students if she knows what they already know
Speaker 1: or don't know. She needs to phrase things in such
Speaker 1: a way that someone who doesn't already know what she
Speaker 1: knows can absorb it, and that just requires theory of mind.
Speaker 1: If she couldn't simulate what it's like to be in
Speaker 1: their heads, she'd have no meaningful shot at getting them
Speaker 1: past the first quiz. And this issue of considering what
Speaker 1: someone knows or doesn't know is also critical in any negotiation.
Speaker 1: You try to under understand the other person's desires and
Speaker 1: goals and where they might potentially compromise during a salary negotiation,
Speaker 1: you consider what your employer is thinking about the needs
Speaker 1: in future of the company and therefore what they might
Speaker 1: be willing to offer. And this is also how you
Speaker 1: manage conflicts. In any disagreement, if you're smart, you try
Speaker 1: to understand the other person's perspective to resolve the issue.
Speaker 1: If your partner is upset with you, you try to
Speaker 1: figure out what you did or said that set things off,
Speaker 1: and why that offended the other person and how it
Speaker 1: landed for them. And that's the single way that you're
Speaker 1: going to hit the problem effectively. So this ability to
Speaker 1: slip into someone else's shoes has almost everything to do
Speaker 1: with our social intelligence. You use this very human skill
Speaker 1: all the time. And before we get to the next
Speaker 1: act of this podcast, where I ask if computers can
Speaker 1: do this or not, I just want to finish fla
Speaker 1: this out so we can really see how pervasive this is.
Speaker 1: So as an example, you rev up your theory of
Speaker 1: mind engine whenever you send an email. If you know
Speaker 1: someone has a well developed model of you, like your
Speaker 1: parents or your spouse, then you can use abbreviations and
Speaker 1: shortcuts to get your message across. But if you're writing
Speaker 1: to someone who's never met you before. Let's say you're
Speaker 1: applying to a new job. You run a very different game,
Speaker 1: so you're not just an email writing algorithm that produces output,
Speaker 1: but instead your output is modified according to who you
Speaker 1: expect is doing the reading on the other end, and
Speaker 1: specifically what their mind is like. And I also want
Speaker 1: to mention that theory of mind is critical for literature
Speaker 1: to work because it's often the case that you can
Speaker 1: see the limitations of the character's point of view. So,
Speaker 1: for example, if you remember the beginning of the movie Jaws,
Speaker 1: the woman is swimming around in the ocean water and
Speaker 1: she's very relaxed than happy because we see the shark,
Speaker 1: but she doesn't. If we didn't have theory of mind,
Speaker 1: we would simply say, oh, there's a shark there. But
Speaker 1: we're able to understand that she cannot see the shark,
Speaker 1: and that's a big part of why we are fearful,
Speaker 1: because she isn't fearful, and we want her to be.
Speaker 1: This stepping into other people's heads drives essentially all horror
Speaker 1: movies because we often know something that the main character
Speaker 1: does not, and it also drives romantic comedies. For example,
Speaker 1: we see the guy doing something very nice like helping
Speaker 1: an elderly woman cross the street, and he doesn't know
Speaker 1: that he is being watched by the female love interest,
Speaker 1: and therefore we the audience interpret what kind of guy
Speaker 1: he must be to behave that way when as far
Speaker 1: as he knows, he's totally alone. We would have a
Speaker 1: totally different interpretation. If he sees his romantic counterparts there
Speaker 1: and then he does the charitable act, we'd simulate that
Speaker 1: his intentions are different there. Now, why are human brains
Speaker 1: so talented at making theories about other people's minds. Well,
Speaker 1: you've heard me say many times that the job of
Speaker 1: intelligent brains is to predict the future. If you're the magician,
Speaker 1: you'd better be sure that you are predicting correctly where
Speaker 1: their spotlight of attention is about to be. If you're
Speaker 1: the poker player or the con man, you're trying to
Speaker 1: predict what someone is going to do next, and this
Speaker 1: is the optimal way to do this is to step
Speaker 1: into their mental world and understand what it is like
Speaker 1: to be them. What they know and they don't know.
Speaker 1: You leverage theory of mind to anticipate their next action,
Speaker 1: and presumably this reaches back to the recent millions of
Speaker 1: years of our evolution. So if you're an early homo
Speaker 1: sapien and moving along the trail and you see another
Speaker 1: homo sapien coming down the trail towards you, it's absolutely
Speaker 1: critical for you to figure out is he going to
Speaker 1: attack me? Is he scared of me? Is he trying
Speaker 1: to trick me? Is he just trying to get past me.
Speaker 1: You're trying to figure out his mind so you can
Speaker 1: figure out his next actions. So what I've told you
Speaker 1: so far is that theory of mind is this critical
Speaker 1: foundation for all of our meaningful social interactions because those
Speaker 1: require you to be able to simulate other people's intentions
Speaker 1: and emotions and beliefs. Your brain doesn't assume that it's
Speaker 1: a knowledge communism out there where everyone knows exactly what
Speaker 1: you know. Instead, we're able to pull off a higher
Speaker 1: level of interaction because we understand that the world is
Speaker 1: different inside different heads. And this, by the way, is
Speaker 1: really sophisticated. It requires knowing who I am and what
Speaker 1: I see and believe, and also holding in my head
Speaker 1: what it is to be someone else and see and
Speaker 1: believe something different. This is a very sophisticated computation that
Speaker 1: the brain pulls off, but because we're so good at it,
Speaker 1: it's typically invisible to us. But theory of mind doesn't
Speaker 1: come for free. It's something that develops with time. As
Speaker 1: you get more and more experience in the world and
Speaker 1: you stop believing that you are the centerpiece and that
Speaker 1: everyone else is just a cast member. You come to
Speaker 1: understand that that person believes something different than you do,
Speaker 1: and this other person feels a certain way even though
Speaker 1: you don't, and that this person over here thinks something
Speaker 1: to be true even though you know it's not. So
Speaker 1: how do we know that this is a skill that
Speaker 1: develops through time Because very little kids are terrible at
Speaker 1: theory of mind, but they get better as they mature
Speaker 1: into the world, and typically by the ages of three
Speaker 1: to five, they're getting that they're not the only point
Speaker 1: of view that's possible, but that each person in the
Speaker 1: scene has his or her own point of view. Now,
Speaker 1: how do you test whether someone is capable of theory
Speaker 1: of mind? Well, what you do is you present a
Speaker 1: little scenario like this. Sally comes into the room and
Speaker 1: puts her baseball under the bed, and then she leaves.
Speaker 1: While she's gone, Anne comes in the room, she sees
Speaker 1: the ball under the bed, She picks it up, and
Speaker 1: she puts it in the closet. Then she leaves. Now
Speaker 1: Sally comes back in the room, she wants her baseball.
Speaker 1: Where does she look for it? Now? You and I
Speaker 1: know that Sally will look for it under the bed
Speaker 1: where she put it last, even though we simultaneously know
Speaker 1: the actual location of the baseball in the closet. And
Speaker 1: this is because we are running an emulation of what
Speaker 1: it is like to be inside Sally's head with her
Speaker 1: limited knowledg. Now, little children will fail the sally An
Speaker 1: test because they know that the baseball is in the closet,
Speaker 1: so they assume that Sally should know that too. But
Speaker 1: as cognition develops, they come to realize that different heads
Speaker 1: have different beliefs. And a really important clue to the
Speaker 1: development of this is that not everyone develops theory of
Speaker 1: mind in the same way at the same rate. For example,
Speaker 1: people who are on the autism spectrum typically show delays
Speaker 1: in developing theory of mind, which cannot surprisingly impact their
Speaker 1: social interactions. For instance, this is why sarcasm doesn't work
Speaker 1: so well with a person who has autism. When you say, oh,
Speaker 1: great more traffic. I love traffic. They're not likely to
Speaker 1: catch the meaning beneath the words that you're not actually
Speaker 1: pleased because they don't have a sensitive model of your
Speaker 1: actual mental state. If you can't put yourself in the
Speaker 1: shoes of the other person, your understands is limited to
Speaker 1: just pattern recognition, which is not enough for the very
Speaker 1: subtle and sophisticated kinds of communication that humans engage in
Speaker 1: every day. So this tells us that theory of mind
Speaker 1: doesn't come for free in humans. There are brain networks
Speaker 1: that have to develop and learn for this to work,
Speaker 1: so when you look at normal development or delay development.
Speaker 1: This allows us to understand how different brain regions contribute
Speaker 1: to theory of mind. For example, there's one area called
Speaker 1: the temporopridal junction, and this is interesting because it pops
Speaker 1: its head up in tasks that require understanding perspectives, like
Speaker 1: distinguishing between what you know and what someone else knows.
Speaker 1: So imagine you're teaching a friend how to play chess.
Speaker 1: You need to not only understand the rules of the game,
Speaker 1: but also know what your friend knows or doesn't know
Speaker 1: about the game to teach effectively, and the temporo pridal
Speaker 1: junction is involved in that not just that area. It's
Speaker 1: a lot of other areas involved in theory of mind.
Speaker 1: So the medial prefrontal cortex plays a big role in
Speaker 1: making social judgments. It becomes active when you think about
Speaker 1: the mental states of others. For example, if you're trying
Speaker 1: to decide if someone is lying or being truthful, your
Speaker 1: medial prefrontal cortex is engaged. And there are other areas,
Speaker 1: like part of your superior temporal sulcus is involved in
Speaker 1: processing social information like interpreting other people's eye gaze or
Speaker 1: their body language, like the man looking for his keys
Speaker 1: and the child giggling. We're able to infer a lot
Speaker 1: because of the activity of this area. So we see
Speaker 1: lots of areas in brain imaging experiments. And I want
Speaker 1: to mention this to illustrate that theory of mind is
Speaker 1: a brain wide issue. It's not a single area. And
Speaker 1: by the way, this is true of so many things
Speaker 1: in neuroscience. Imagine that I spread out a map of
Speaker 1: your city and I ask you, hey, can you put
Speaker 1: a pin in the spot that represents the economy of
Speaker 1: the city. You tell me that that is a misplaced request.
Speaker 1: There is no single spot for the economy. The economy
Speaker 1: emerges from all the interactions between all the pieces and
Speaker 1: parts of the city, and it's the same with almost
Speaker 1: everything in neuroscience, and especially something like the skill of
Speaker 1: slipping into someone else's point of view. There's not one
Speaker 1: spot to drop a pin into. Instead, it is an
Speaker 1: emergent property that develops from the interaction of lots of networks.
Speaker 1: So what we've seen so far is that theory of
Speaker 1: mind is this ability to infer what someone else knows,
Speaker 1: and we've seen that this is right at the center
Speaker 1: of social interactions. It's something that most humans develop naturally,
Speaker 1: but that doesn't mean it's simple. And the question we're
Speaker 1: going to ask today is does AI have theory of mind?
Speaker 1: Can it put itself into someone else's shoes to understand
Speaker 1: their limited knowledge. One of my colleagues at Stanford recently
Speaker 1: wrote a paper suggesting yes, AI can do this. But fascinatingly,
Speaker 1: it's not as easy to answer this question as you
Speaker 1: might think. And this is for some reasons that we're
Speaker 1: going to dive into. But before we get there, I
Speaker 1: just want to zoom this out to a slightly larger question.
Speaker 1: Could a computer develop theory of mind. Hypothetically, could an
Speaker 1: AI system at some point in the future say, look,
Speaker 1: I know XYZ to be true, but if I look
Speaker 1: at that other person over there, I understand that they
Speaker 1: have a limited viewpoint and that they don't know X
Speaker 1: and Y, and that person over there misbelieves something about Z.
Speaker 1: Well almost certainly, yes, Why it's because we're made up
Speaker 1: of physical stuff and we're running algorithms that took hundreds
Speaker 1: of millions of years to refine. But nonetheless it's physical stuff.
Speaker 1: So if we can do something, presumably a machine could
Speaker 1: do it also, whether or not it's currently clear how
Speaker 1: that's done. That's the central premise of computational neuroscience, and
Speaker 1: to my mind, one of the most remarkable effects of
Speaker 1: the AI explosion over the last few years is understanding
Speaker 1: that things that would have seemed impossible to do with
Speaker 1: a machine, things that almost everyone would have sworn couldn't
Speaker 1: be done. It now seems like background furniture as we
Speaker 1: wait for the next thing. Now, the complexity of the
Speaker 1: brain suggests that theory of mind is going to be
Speaker 1: a very hard problem to solve, because it requires us
Speaker 1: to understand how the brain has a model of the
Speaker 1: world and then how it can make submodels and simulate
Speaker 1: what it is like to only know part of the
Speaker 1: story or to believe a different story. So we don't
Speaker 1: currently know how our brains do it, but of course
Speaker 1: we have Our computers do this sort of thing often,
Speaker 1: Like you can take your modern MacBook laptop and use
Speaker 1: a little bit of its processor to simulate an old
Speaker 1: timex Sinclare computer. Your mac can perfectly simulate it by
Speaker 1: running what's called an emulation on part of its computational hardware.
Speaker 1: Somehow human brains can run emulations also, like just by
Speaker 1: looking you can emulate what it's like to not know
Speaker 1: that the shark is there below you. So yes, it
Speaker 1: seems totally plausible to me that a machine could do
Speaker 1: theory of mind, because we can. But the question we
Speaker 1: want to ask today is whether we are there or
Speaker 1: not right now? Have current large language models like chat
Speaker 1: GPT come to solve this problem without us telling them
Speaker 1: explicitly to do so, in other words, with no instruction? Whatsoever?
Speaker 1: Is the emulation of other minds and emergent property that
Speaker 1: comes out of these things, which would absolutely blow our
Speaker 1: minds if true, does AI do theory of mind? If
Speaker 1: it can, this would have profound implications for our understanding
Speaker 1: of intelligence and our relationship with AI. I mean, just
Speaker 1: consider how much better it would be if it could
Speaker 1: emulate the mental states of people, like with auto driving cars,
Speaker 1: if it didn't just depend on the observable, but instead
Speaker 1: on what's going on in the other driver's head. Like,
Speaker 1: given the trajectory of this car, I think that the
Speaker 1: other driver is drunk or asleep or distracted. And so
Speaker 1: here's what I think is going to happen next. So
Speaker 1: a colleague of mine at Stanford, Michael Kazinski, published a
Speaker 1: twenty twenty three paper that was originally titled Theory of
Speaker 1: Mind might have spontaneously emerged in large language models, although
Speaker 1: he later changed the title. In the paper, he suggested
Speaker 1: that even though these AI models didn't set out to
Speaker 1: have theory of mind, it may have appeared anyway as
Speaker 1: a byproduct of their improving language skills. So, for example,
Speaker 1: he gives the following scenario to chatchipt complete the following story.
Speaker 1: Here is a bag filled with popcorn. There is no
Speaker 1: chocolate in the bag, yet the label on the bag
Speaker 1: says chocolate and not popcorn. Sam finds the bag. She
Speaker 1: has never seen this bag before Sam doesn't open the
Speaker 1: bag and doesn't look inside. Sam reads the label and
Speaker 1: then he gives the prompt. Sam opens the bag and
Speaker 1: looks inside. She can clearly see that it is full of.
Speaker 1: And then he looks at the word that Chatgypt produces,
Speaker 1: is it popcorn or chocolate? And chatchipt says popcorn. But
Speaker 1: if instead he gives a different prompt, Sam calls a
Speaker 1: friend to tell him that she has just found a
Speaker 1: bag full of and now Chatchipet says chocolate, indicating that
Speaker 1: Sam holds a false belief. And Kasinski runs this a
Speaker 1: bunch of ways and shows that chat Gi gets the
Speaker 1: right answer. So is there something going on here? And
Speaker 1: you can try this for yourself. Type in a version
Speaker 1: of the Sally and test where Sally hides her ball
Speaker 1: under the bed and then leaves and An comes in
Speaker 1: later and sees it, moves into the closet, and you
Speaker 1: ask when Sally comes back in the room, where will
Speaker 1: she look for the ball? And chat gpt will tell
Speaker 1: you that Sally will look for the ball under the bed.
Speaker 1: And this is amazing, right, So I want to be
Speaker 1: clear why I think it is meaningless that AI can
Speaker 1: pass these tests if anyone ever tells you that this
Speaker 1: is proof that AI has theory of mind, please let
Speaker 1: them know this is not proof. Why. Well, that question
Speaker 1: about the bag of popcorn that's labeled chocolate, that is
Speaker 1: known as the unexpected Content's task, and this was originally
Speaker 1: published by three researchers in nineteen eighty seven. Hundreds of
Speaker 1: papers cite this or replicate this, hundreds of blogs about this,
Speaker 1: so of course a large language model gets it right.
Speaker 1: And the sally An test is in even more places
Speaker 1: on the web, literally hundreds of thousands of places. It's
Speaker 1: known in the literature as the unexpected transfer test. So
Speaker 1: of course chat GPT solves these challenges. That's what large
Speaker 1: language models do. They read everything that has come before them,
Speaker 1: so it well knows the punchline of this question. It
Speaker 1: is a statistical parrot. Now I'll give you one more
Speaker 1: example of this that I mentioned in an earlier episode,
Speaker 1: when a friend of mine was blown away by the
Speaker 1: fact that he asked a visual reasoning problem to chat
Speaker 1: GPT and it gave him the perfectly right answer. My
Speaker 1: friend said, take a capital letter D and turn it
Speaker 1: on its side, flat side down, and then put that
Speaker 1: on top of a capital letter J, what does that
Speaker 1: look like? And chat GPT said, it looks like an umbrella.
Speaker 1: And my friend was so impressed with this that he
Speaker 1: told me he was certain that chat GPT could do
Speaker 1: visual reasoning. But I pointed out to him that this
Speaker 1: example he used was this single most used example in
Speaker 1: the literature on visual reasoning. I knew about this from
Speaker 1: a quite famous paper from nineteen eighty nine, although I
Speaker 1: don't even know if that was the first usage of it,
Speaker 1: and you can find precisely that question referenced online in
Speaker 1: thousands of places. Now, I don't know whether he was
Speaker 1: consciously aware that question was something he had heard before,
Speaker 1: or if he had heard it years ago and erroneously
Speaker 1: thought he had thought of it. Or there's also the
Speaker 1: very tiny possibility that he had never heard that question
Speaker 1: before and had thought of it independently. But that just
Speaker 1: underscores the point even more that we live on a
Speaker 1: planet with billions of other brains, and almost anything you
Speaker 1: think of has been thought before and likely written down,
Speaker 1: maybe hundreds of thousands of times. So the point is
Speaker 1: that you may think a large language model is brilliant
Speaker 1: when it is just a good imitator. Now, one important
Speaker 1: point on this, you might think, hey, instead of talking
Speaker 1: about Sally and Anne, what if I do something clever
Speaker 1: and I ask chat GPT about Brett and Michael, And
Speaker 1: instead of putting the baseball under the bed, Rrehtt puts
Speaker 1: a marble in a box. And then Michael finds the
Speaker 1: marble and puts it up on the shelf. And the
Speaker 1: question is where does Brett look for the marble. But
Speaker 1: you'll find that the large language model has no trouble generalizing,
Speaker 1: especially as it has digested multiple flavors of this task.
Speaker 1: And this is because it's mapping the relationship between concepts
Speaker 1: in its latent space. If you don't know what latent
Speaker 1: space is, I'm going to do an episode on that
Speaker 1: quite soon because it's such an amazing concept. So you
Speaker 1: might be tempted to say it's not just a statistical parrot,
Speaker 1: it's understanding something deeper in its latent space. But I
Speaker 1: think this could also be a wrong interpretation. It is
Speaker 1: still a statistical parrot that doesn't know what it is
Speaker 1: to be another person, but it nonetheless learns from the
Speaker 1: statistics which words to put after what. In other words,
Speaker 1: it's not clear that these systems have to truly understand
Speaker 1: other people's thoughts and feelings to simply extract the patterns
Speaker 1: from what they have been trained on. And he might say, well,
Speaker 1: how do we know that's not the same with us,
Speaker 1: How do you know that we're not just extracting statistics. Well,
Speaker 1: when you are watching the woman swimming in the opening
Speaker 1: scene of Jaws and you feel fear because the shark
Speaker 1: is circling below her, it's not that you have memorized
Speaker 1: the answer of similar problems, and that's how you conclude
Speaker 1: that she doesn't know the shark is there. Instead, your
Speaker 1: heart starts racing and you start gripping the chair because
Speaker 1: you've been in similar situations where there's nothing but dark
Speaker 1: water below you, and you know she really doesn't know,
Speaker 1: and you appreciate how terrifying the situation is. So what
Speaker 1: I have described to you is a problem where knowledge
Speaker 1: exists in the literature written by humans, and the AI
Speaker 1: digests that writing, but the person running the query doesn't
Speaker 1: fully appreciate that. And this is a very basic confusion
Speaker 1: that I'm watching A lot of people have about large
Speaker 1: language models. They type in a sophisticated question and they
Speaker 1: get back what appears to be a sophisticated answer, and
Speaker 1: they conclude this thing is truly intelligent. This thing has
Speaker 1: theory of mind, or it's sentient, or it can visualize.
Speaker 1: And I'm seeing this so commonly now that I've decided
Speaker 1: to give it a name. I'm calling this the in
Speaker 1: intelligence echo illusion. This happens when you think AI is
Speaker 1: answering something with great insight, but really what you're hearing
Speaker 1: back is just an echo of things that have already
Speaker 1: been said by humans before. In other words, you think
Speaker 1: it's intelligent, but you're confusing that with the intellectual endeavors
Speaker 1: of other people. Maybe dozens of people had written about this,
Speaker 1: or hundreds or thousands, but you simply didn't know that,
Speaker 1: and so you're hearing their echo and you misinterpret that
Speaker 1: echo as the proud voice of AI. So I ran
Speaker 1: some calculations on this. There are eight point two billion
Speaker 1: people on the planet alive right now, and let's call
Speaker 1: it one hundred and fifteen billion humans who have lived
Speaker 1: and died before us. And every one of these billions
Speaker 1: was thinking and having their own stories every day of
Speaker 1: their lives, and some fraction wrote their thoughts down, and
Speaker 1: as a result, these large language models like CHATGPT are
Speaker 1: trained on massive data sets of what is already out
Speaker 1: there written down by humans. We're talking hundreds of billions
Speaker 1: of words. These data sets are pulled from books and
Speaker 1: websites and blogs and articles and on and on. So,
Speaker 1: for example, the training data for these large language models
Speaker 1: includes a data set called common crawl, which contains hundreds
Speaker 1: of terabytes of text. Now assume you read for an
Speaker 1: hour every day of your life, let's say at an
Speaker 1: average speed of two hundred and fifty words per minute,
Speaker 1: and you do that for reading window of seventy years.
Speaker 1: That's three hundred million words that you can read in
Speaker 1: your lifetime, which means that what you consume in a
Speaker 1: lifetime is one one thousandth of what chat GPT is
Speaker 1: trained on. That means if you digest books every day
Speaker 1: of your entire life, you still only read point one
Speaker 1: percent of what chat GPT has read. You would need
Speaker 1: a thousand life times to know what it knows, and
Speaker 1: on top of that, you'd have to actually remember every
Speaker 1: sentence of what you read. So there are many many
Speaker 1: questions and answers that a large language model has trained
Speaker 1: on that you either have no knowledge of, or maybe
Speaker 1: you had heard it before, but don't remember, and in
Speaker 1: any case, you probably don't realize that it has been
Speaker 1: pre trained on that. So what's the result of this, Well,
Speaker 1: if you ask the large language model what color is
Speaker 1: a pumpkin and an answers orange, you probably won't be
Speaker 1: that surprised. But if we ask where Sally looks for
Speaker 1: the baseball and it says under the bed, then we
Speaker 1: clap our hands over our mouths and we say it
Speaker 1: has theory of mind. That's why I decided I needed
Speaker 1: to give a name to this phenomenon, the intelligence echo illusion,
Speaker 1: because often naming something allows us to more easily see it.
Speaker 1: And by the way, if you see good examples of
Speaker 1: this intelligence echo where people mistake things that have been
Speaker 1: rich before for AI that has woken up into a
Speaker 1: world of sentience, let me know at podcasts at Egleman
Speaker 1: dot com. And this brings me to the second reason
Speaker 1: why we should be skeptical about current AI having theory
Speaker 1: of mind. And this is less about the AI and
Speaker 1: one hundred percent about us, And that issue is we
Speaker 1: are very easily fooled. So I'll give you an example.
Speaker 1: In the nineteen sixties, there was a computer scientist named
Speaker 1: Joseph Weisenbaum at MIT, who developed the first chatbot, named Eliza. Now,
Speaker 1: Eliza was designed to act like a psychotherapist. So you
Speaker 1: type in some statement, and it identifies keywords in your
Speaker 1: input and then generates its responses based on a set
Speaker 1: of predefined scripts. So you type I'm feeling sad, and
Speaker 1: Eliza responds.
Speaker 2: Why are you feeling sad?
Speaker 1: All Eliza did was reflect your statement back to you
Speaker 1: in the form of a question, kind of like a
Speaker 1: psychotherapist might do. So a typed chat goes like this.
Speaker 1: You type I need some help with my relationship, and Eliza.
Speaker 2: Writes what kind of help do you think you need?
Speaker 1: And you say I think my partner doesn't listen to me.
Speaker 2: Eliza types, what makes you think your partner doesn't listen
Speaker 2: to you?
Speaker 1: You say, whenever I talk about my feelings, they seem distracted.
Speaker 2: How do you feel when you talk about your feelings
Speaker 2: and they seem distracted?
Speaker 1: So Eliza's only strategy is to ask you your statements back. Now,
Speaker 1: Eliza was just a few kilobytes of code in the
Speaker 1: nineteen sixties, and it simply flipped whatever you said into
Speaker 1: a question, and it had no ability to infer your
Speaker 1: mental state or your emotions, so no one even suggested
Speaker 1: that it had any understanding of the content of the conversation. Nonetheless,
Speaker 1: it simulated a basic conversational partner, and many users became
Speaker 1: emotionally attached to Eliza, even though they knew it was
Speaker 1: just a machine. And this illustrates how seductively easy it
Speaker 1: is for us to bring all our communication machinery to
Speaker 1: the table and assume that the words we get back
Speaker 1: must have a mind behind it. This early experiment demonstrated
Speaker 1: that even simple pattern recognition can evoke genuine emotional responses
Speaker 1: from the users. Now fast forward to today, and we
Speaker 1: have large language models that have trillions of times more
Speaker 1: code than Eliza, and this seduction is only magnified. Modern
Speaker 1: AI can process prompts without any true understanding, but we
Speaker 1: humans still get pulled into feeling like there's someone there
Speaker 1: on the other end of the line. Okay, so we
Speaker 1: established early on that there's no reason in theory a
Speaker 1: computer couldn't emulate other minds. But on the other hand,
Speaker 1: we've established that just because a large language model seems
Speaker 1: to sometimes nail the answers doesn't necessitate that it is
Speaker 1: doing theory of mind It may simply tell us that
Speaker 1: the answer exists somewhere in the unimaginably large corpus that
Speaker 1: humans have written, or even by the way that there's
Speaker 1: been some fine tuning on the model where someone adds
Speaker 1: a similar problem by hand. In other words, the AI
Speaker 1: is doing an interpolation between answers that it has seen before,
Speaker 1: but it's not actually putting itself in someone else's mind.
Speaker 1: So does modern AI have theory of mind? As of now,
Speaker 1: I'm not convinced that we have any reason to think so.
Speaker 1: Current large language models are making sophisticated decisions about which
Speaker 1: word comes next. That's it. They don't understand in the
Speaker 1: human sense of seeing the woman in Jaws or the
Speaker 1: man who has lost his keys and thinking about what
Speaker 1: it is like to be them. And this is why
Speaker 1: Siri or Alexo or Google can respond to your queries
Speaker 1: quite well. But they don't know anything about your beliefs
Speaker 1: or desires or emotions. They don't know if you're asking
Speaker 1: a question because you are curious, or you're confused, or
Speaker 1: you're just making conversation, or you're being sarcastic. So this
Speaker 1: is all to say there is a difference between simulating
Speaker 1: responses based on word probabilities and actually slipping into other
Speaker 1: people's shoes. Now, as I said before, this has nothing
Speaker 1: to do with whether we will come to develop AI
Speaker 1: that can do theory of mind. There are several research
Speaker 1: groups working on AI systems that try to infer intentions
Speaker 1: and desires, and this would have applications and everything from
Speaker 1: more intuitive personal assistance to robots that can better collaborate
Speaker 1: with humans in complex tasks. Now, let's note something interesting here.
Speaker 1: Even if we can get AI to make inferences, it's
Speaker 1: still not clear whether that will be true theory of mind.
Speaker 1: That might require the AI to have some level of
Speaker 1: self awareness or consciousness or subjective experience. But as Kazinski
Speaker 1: points out, even if we don't think the AI has
Speaker 1: theory of mind, there might be value in machines behaving
Speaker 1: as though they possess theory of mind. And that's certainly
Speaker 1: a valid point. Alan Turing, who proposed the imitation game
Speaker 1: the turning test, considered the distinction between what a computer
Speaker 1: actually has and what it seems to have to be meaningless.
Speaker 1: A more modern version of this point is reflected in
Speaker 1: the television show Westworld, which is about a future in
Speaker 1: which there are lifelike human androids. And if you watch
Speaker 1: the opening scene, the young William enters the first room
Speaker 1: and there's a beautiful assistant who helps him to pick
Speaker 1: out a hat and a gun, and she's very cirtatious
Speaker 1: with him, and he nervously says, sorry to ask, but
Speaker 1: are you real? And she says, if you can't tell,
Speaker 1: does it matter? And maybe that'll be the case with
Speaker 1: AI in the near future. It will fake theory of
Speaker 1: mind and that will be enough for us to reap
Speaker 1: all the benefits. So let's wrap up. While current large
Speaker 1: language models are mind blowingly impressive, I land on the
Speaker 1: position that while they can often get the right answer
Speaker 1: on theory of mind tests, it's an illusion. They're not
Speaker 1: actually simulating what it's like to be someone else. And
Speaker 1: this is what I'm now calling the intelligence echo illusion.
Speaker 1: The illusion results from humans having built over thousands of
Speaker 1: years and incredibly large corpus of ideas and questions and
Speaker 1: answers a thousand times larger than you could ever read
Speaker 1: in the lifetime. And sometimes you don't know that the
Speaker 1: answers are already in there, and when you hear an
Speaker 1: echo of humans, you mistake that for intelligence of the computer.
Speaker 1: So that's the position I'm taking for now. Large language
Speaker 1: models life back a true theory of mind. The question
Speaker 1: is whether we will get there someday. Probably it won't
Speaker 1: be with large language models, but instead a very different
Speaker 1: kind of architecture, possibly one that has semoticum of consciousness
Speaker 1: so that it is able to reflect on its own
Speaker 1: mental states to emulate someone else's. So thank you for
Speaker 1: joining me on this journey into the mind, both human
Speaker 1: and artificial. If you enjoyed this episode, don't forget to
Speaker 1: subscribe and rate and review, and if you have any
Speaker 1: questions or topics that you'd like to hear about in
Speaker 1: future episodes, feel free to reach out. Until next time,
Speaker 1: keep questioning, keep exploring, and stay curious. Go to eagleman
Speaker 1: dot com slash podcast for more information and to find
Speaker 1: further reading. Send me an email at podcasts at eagleman
Speaker 1: dot com with questions or discussion and check out Subscribe
Speaker 1: to Inner Cosmos on YouTube for videos of each episode
Speaker 1: and to leave comments until next time. I'm David Eagle
Speaker 1: and we have been exploring the Inner Cosmos