Speaker 1: Can we read minds?
Speaker 2: For those of you who follow the media around neuroscience,
Speaker 2: it certainly seems like it. You've seen articles and heard
Speaker 2: things about mind reading.
Speaker 1: So what is it? Could we run mind reading in.
Speaker 2: Airports so that we know who is carrying a bomb
Speaker 2: or hatching a plot?
Speaker 1: Where are we right now with this technology?
Speaker 2: How close or far are we from being able to
Speaker 2: know if you were thinking about some particular thing you
Speaker 2: did or intend to do. Welcome to Inner Cosmos. I'm
Speaker 2: David Eagleman, a neuroscientist at Stanford and this podcast is
Speaker 2: all about the intersection of brain science with our lives,
Speaker 2: both now and in the future. Today's episode is about
Speaker 2: mind reading.
Speaker 1: What does it mean? Where are we with it now?
Speaker 2: Will we be able to read your mind by measuring
Speaker 2: something from your brain? So strap in for a wild
Speaker 2: ride today. So imagine this twenty years from now. You
Speaker 2: walk into the airport to take a flight across the nation.
Speaker 2: But there are no metal detectors here anymore, or those
Speaker 2: scanners that rotate around you looking for things hidden on
Speaker 2: your body. Instead of simply knowing what you have on
Speaker 2: your body. You find the airport now has portable brain
Speaker 2: scanning such that the authorities can tell who is thinking
Speaker 2: about bringing a bomb onto the airplane. And don't picture
Speaker 2: a big, clunky fMRI machine like we have now, because
Speaker 2: in two decades, brain scanning could certainly be done at
Speaker 2: a distance, getting some signature of brain activity of every
Speaker 2: person walking through Now could this work? I want to
Speaker 2: put aside any legal and ethical conundrums for just a moment,
Speaker 2: because I want to ask would this be technically possible?
Speaker 2: So let's start with what do we mean by mind reading?
Speaker 2: This is a term that's always been talked about by
Speaker 2: magicians and FBI agents and forensic psychologists, and what it
Speaker 2: means historically is reading somebody's body language or the words
Speaker 2: they choose, or their body posture to understand what might
Speaker 2: be going on inside their heads under the hood where
Speaker 2: you can't really see what's happening. And so the next
Speaker 2: step for mind reading is to use technology to allow
Speaker 2: us to do this a little bit better than an
Speaker 2: observant person would. And one example of this is the
Speaker 2: lie detector test. So I'm going to do a separate
Speaker 2: episode on lie detection. So I'll just say briefly here
Speaker 2: that light detection is used in courts all over the
Speaker 2: world it's not actually detecting a lie, it is simply
Speaker 2: detecting a stress response that is usually associated with lying.
Speaker 2: So this technology started in the early nineteen hundreds when
Speaker 2: people realized that often when somebody is lying, there's a
Speaker 2: stress response. You can detect when people are stressed about
Speaker 2: the lie. Now, this has been researched with breathing, blood pressure,
Speaker 2: heart rate, pupil dilation, and the most common measure is
Speaker 2: what's called the galvanic skin response, and all these are
Speaker 2: external signals that the person is stressed. Now, these aren't
Speaker 2: terrible measures, but they're not perfect either, and that's why
Speaker 2: they're accepted by some court systems and not by others.
Speaker 2: And they're not perfect because, among other things, there are
Speaker 2: people that are very good at lying and it doesn't
Speaker 2: stress them out in any way to do so, and
Speaker 2: so a galvanic skin response or something else is not
Speaker 2: going to reveal anything about stress in their situation. So
Speaker 2: the point I just want to make here is there's
Speaker 2: actually quite a gap between the notion of a lie
Speaker 2: and something that you can read off the skin. And
Speaker 2: you can see this isn't exactly mind reading here, but
Speaker 2: it's something removed by a few degrees, So what about
Speaker 2: reading something directly from the brain. Just before I get
Speaker 2: into that, I'll just mention that some years ago I
Speaker 2: talked with a startup that was using electron cephlography or EEG,
Speaker 2: with little electrodes that you stick on the outside of
Speaker 2: the head. And this company's market was long distance truckers,
Speaker 2: And so what they did is they would sew these
Speaker 2: EEG electrodes into a baseball cap so that the trucker
Speaker 2: could wear the cap and the system could tell if
Speaker 2: he was getting sleepy, and if so, then the system
Speaker 2: would do things like alert the driver to pull over.
Speaker 2: So it was a pretty straightforward use case, but it
Speaker 2: turned out the company was having a very hard time
Speaker 2: getting traction for one main reason. The truckers were concerned
Speaker 2: that their bosses would be able to read their minds.
Speaker 2: Now that struck me as interesting, because we're never going
Speaker 2: to get very far in knowing what is happening inside
Speaker 2: a brain with a crude technology like EEG. People are
Speaker 2: using EEG to let's say, control a wheelchair, because you
Speaker 2: can find a few signals that can be reliably distinguished,
Speaker 2: and then you can use one signal to mean go
Speaker 2: and another one to mean stop and left and right.
Speaker 2: But in today's discussion about reading minds, I want to
Speaker 2: ask if we can use technology to really level up
Speaker 2: simply is somebody having a stress response? Or can we
Speaker 2: distinguish four signals to direct a wheelchair? But instead, what
Speaker 2: precisely is this person thinking about? Who do we do
Speaker 2: mind reading via more direct measurements of the brain and
Speaker 2: how close are we to that right now? Now, what
Speaker 2: I'm going to tell you about the field that the
Speaker 2: media typically calls mind reading started getting the foundations laid
Speaker 2: in the nineteen fifties when people first started measuring signals
Speaker 2: from the brain. And what they found is that these
Speaker 2: brain signals were well correlated with things in the outside world. So,
Speaker 2: for example, when someone sees a horizontal line, that activates
Speaker 2: particular neurons that makes them pop off in the visual cortex.
Speaker 2: So in theory, if you had an electrode dunked into
Speaker 2: the visual cortex and you didn't know what the person
Speaker 2: was seeing, but then you saw these cells pop off,
Speaker 2: you could know that they were seeing a horizon the line.
Speaker 2: This is a very crude form of mind reading. Now,
Speaker 2: with almost everyone on the planet there are not electrodes
Speaker 2: in their brains, so how could you read something meaningful
Speaker 2: about their brain activity. So some decades ago, functional magnetic
Speaker 2: resonance imaging or fMRI burst onto the scene, and this
Speaker 2: technique allows you to non invasively measure the activity across
Speaker 2: the whole brain. The problem is that it's really low resolution,
Speaker 2: so instead of knowing what individual neurons might be doing,
Speaker 2: you now can only see great, big blobs that represent
Speaker 2: activity that covers hundreds of thousands or millions of neurons.
Speaker 2: So people didn't think that fMRI was going to be
Speaker 2: able to tell you much about the details of what
Speaker 2: someone was seeing, like exactly what kind of line or
Speaker 2: picture or video someone is watching. But some years ago
Speaker 2: go researchers in England and in America started doing some
Speaker 2: very cool analyzes. And what they did is they looked
Speaker 2: at lots of visual inputs and they measured the corresponding
Speaker 2: outputs from the fMRI. So you show something and you
Speaker 2: measure the brain activity, and then you do the next
Speaker 2: one and the next one. And what they found is
Speaker 2: that even though the individual blobs can't tell you much,
Speaker 2: if you look at the patterns of blobs with lots
Speaker 2: and lots of example inputs, you can then use machine
Speaker 2: learning here to help you, and you can start pulling out.
Speaker 1: Some pretty good correlations.
Speaker 2: You can start knowing what was presented visually just by
Speaker 2: looking at the details of this multi blob output. And
Speaker 2: this really hit a new watermark in twenty eleven when
Speaker 2: Jack Gallant and his team at Berkeley did an amazing experiment.
Speaker 2: He had participants watch videos for hours and hours and
Speaker 2: they measured each frame of the video and exactly how
Speaker 2: the brain responded, tons and tons of input and output.
Speaker 2: And then they could show the participant a video and
Speaker 2: just by looking at the fMRI brain activity these patterns
Speaker 2: of blobs, the researchers could crudely reconstruct what the video
Speaker 2: must have been that.
Speaker 1: Was being looked at.
Speaker 2: So they look at your brain activity and the computer
Speaker 2: reconstructs this must have been something like a bird flying
Speaker 2: from right to left across the screen, and the real
Speaker 2: video was an eagle flying from right to left across
Speaker 2: the screen. Or here's the reconstruction of a person talking
Speaker 2: or a whale breaching or whatever. And the original videos
Speaker 2: that were shown to the participants were really really similar.
Speaker 2: It was extremely impressive, and I'd encourage you to watch
Speaker 2: their videos, which I've linked at Eagleman dot com slash
Speaker 2: podcast now, as you can imagine the media blowing up
Speaker 2: at this point, and everyone called this mind reading, and
Speaker 2: it's only gotten more impressive and fine tuned from there.
Speaker 2: Now the computer also spits out the words the concepts
Speaker 2: that are associated with a video while you're watching, and
Speaker 2: the Lawns team has reversed the machine learning to show
Speaker 2: maps of where concepts sit in the brain. So if
Speaker 2: it's the word marriage or job or dog or whatever,
Speaker 2: you can see which areas of the brain care about
Speaker 2: these concepts.
Speaker 1: Now. I want to note that the maps.
Speaker 2: Of concepts are slightly different for each participant, and I'm
Speaker 2: going to come back to this in a bit, but
Speaker 2: for now, I just want to emphasize how mind blowing
Speaker 2: this all is. And when I first read this paper
Speaker 2: about reconstructing videos in twenty eleven, I was so inspired
Speaker 2: and immediately had an idea in my lab to see
Speaker 2: if we could measure dreams. Because dreams are simply activity
Speaker 2: in the primary visual cortex. Teams are experienced as visual
Speaker 2: because there's activity in the visual part of your brain,
Speaker 2: and your brain interprets that as seeing, so even though
Speaker 2: your eyes are closed, you are having full, rich visual experience.
Speaker 2: So I figured we should be able to just measure
Speaker 2: that and then interview people about what they roughly saw
Speaker 2: in their dreams. Or perhaps we give them a couple
Speaker 2: of choices about scenes that might have appeared in their
Speaker 2: dream and see which one they pick. Anyway, I started
Speaker 2: this project, but before I could finish it, my colleagues
Speaker 2: in Japan, led by Yukiyaso Komatani, published a beautiful paper
Speaker 2: on this same thing. His was the same approach as
Speaker 2: what Galant had done at Berkeley. You gather a bunch
Speaker 2: of data about people watching videos in the fMRI, and
Speaker 2: then you use a machine learning model to do the
Speaker 2: neural decoding. Now, with dreaming, it's a little more difficult
Speaker 2: to know whether you're actually getting this right, because the
Speaker 2: people have to wake up and tell you what they remember,
Speaker 2: and then you compare that to what you think you
Speaker 2: might have decoded. And of course you can't know the
Speaker 2: exact timing of when they saw what. But nonetheless this
Speaker 2: gave the rough start to a dream decoder that could
Speaker 2: tell you what you were dreaming about, and the same
Speaker 2: idea with figuring out what someone might have seen, you
Speaker 2: can do the same thing with what someone heard. This
Speaker 2: started in twenty twelve in the lab of Robert Knight,
Speaker 2: also at Berkeley, who showed that you could reconstruct what
Speaker 2: words a person was hearing by using surface electrodes right
Speaker 2: on the surface of the brain. This is called intracranial
Speaker 2: EEG or IEG. So by presenting lots of spoken words
Speaker 2: to them and measuring the neural activity here, they could
Speaker 2: reconstruct which word a person was hearing. And a paper
Speaker 2: just came out in twenty twenty three showing now that
Speaker 2: you can use even more sophisticated machine learning models to
Speaker 2: reconstruct the music that someone is listening to. Just from
Speaker 2: eavesdropping on the neural activity, you can figure out what
Speaker 2: the music must have been. And specifically, this new paper
Speaker 2: reconstructs Pink Floyd's song Another Brick in the Wall. So
Speaker 2: they play the music to the person and they look
Speaker 2: at all this up and down electrical activity on the
Speaker 2: surface of the cortex, and they can figure out the
Speaker 2: words of the song, and they can figure out all
Speaker 2: kinds of other information about the prosity of the music,
Speaker 2: the rhythm and intonation and stress and so on, and
Speaker 2: they can put this all together into a reconstruction of
Speaker 2: what the whole song must have been just from.
Speaker 1: The neural activity.
Speaker 2: So I want you to notice one thing. This is
Speaker 2: a totally different measure than what the the other labs did.
Speaker 2: This lab is using intracranial EEG, which gives you really
Speaker 2: fast small signals, whereas fMRI gives really slow large signals.
Speaker 2: But it doesn't matter. For a machine learning model. All
Speaker 2: you have to do is give lots of inputs and
Speaker 2: measure lots of outputs, and as long as the output
Speaker 2: is reliably correlated to that input, it doesn't really matter
Speaker 2: what the measuring device is. You can still look at
Speaker 2: the output and say, okay, look, this must have been
Speaker 2: the input. This must have been the video shown because
Speaker 2: it led to this pattern of activity in the fMRI,
Speaker 2: or this must have been the music played because it
Speaker 2: led to this pattern of activity in the intracranial EEG.
Speaker 2: And you can take this same decoding approach with other
Speaker 2: parts of the brain too, not just to decode the
Speaker 2: input that someone must have seen or heard, but also
Speaker 2: to understand what the intention is for an output, in
Speaker 2: other words, how the brain wants to move the body.
Speaker 2: In other words, you can eavesdrop on the visual cortex
Speaker 2: to understand what's in front of someone's eyes, or on
Speaker 2: the auditory cortex to understand what's piping into their ears.
Speaker 1: And in the same way, you can decode.
Speaker 2: The signals from their motor system to understand how they're
Speaker 2: trying to move. And this is exactly what neuroscience is
Speaker 2: doing with people who are paralyzed. Typically, a person who
Speaker 2: is paralyzed has damage to their spinal cord such that
Speaker 2: the signals in their brain aren't able to plunge down
Speaker 2: the back and control the muscles. But their brain is
Speaker 2: still generating the signals. It's just that the signals aren't
Speaker 2: getting to their final destination because of a broken roadway
Speaker 2: or a degenerated roadway. But the key is that their
Speaker 2: brain is still generating the signals just fine. So if
Speaker 2: you could listen to those signals in the motor cortex,
Speaker 2: could you exploit those to make the action happened in
Speaker 2: the outside world. Yes, And this kind of incredible work
Speaker 2: has been happening for two decades now. It started when
Speaker 2: researchers at Emery University implanted a brain computer interface into
Speaker 2: a locked in patient named Johnny Ray, and this allowed
Speaker 2: him to control a computer cursor on the screen simply
Speaker 2: by imagining the movement. His motor cortex couldn't get the
Speaker 2: signals passed a damaged spinal cord, but the implant could
Speaker 2: listen to those signals and pass along the message to
Speaker 2: a computer. And by two thousand and six there was
Speaker 2: a former football player named Matt Nagel who was paralyzed,
Speaker 2: and he was able to get a brain computer interface,
Speaker 2: a little grid of almost one hundred electrodes implanted directly
Speaker 2: into his motor cortex, and that allowed him to control
Speaker 2: lights and open an email, and play the video game
Speaker 2: Pong and draw circles on the screen. And that's mind reading.
Speaker 2: His brain imagined moving his muscles, which caused activity in
Speaker 2: his motor cortex, and the researchers measured that neural activity
Speaker 2: with the electrodes and crudely decoded that to figure out
Speaker 2: the intention. And by twenty eleven, Andrew Schwartz and his
Speaker 2: colleagues at the University of Pittsburgh built a beautiful robotic arm,
Speaker 2: and a woman who had become paralyzed could imagine making
Speaker 2: a movement with her arm, and the robotic arm moves. Now,
Speaker 2: as I said, when you want to move your arm.
Speaker 2: The signals travel from your motor cortex, down your spinal cord,
Speaker 2: to your peripheral nerves, and to your muscle fibers. So
Speaker 2: with this woman, the signals recorded from the brain just
Speaker 2: took a different route, racing along wires connected to motors
Speaker 2: instead of neurons connected to muscles. In an interview, this
Speaker 2: woman said, I'd so much rather have my brain than
Speaker 2: my legs. Why did she say this, Because if you
Speaker 2: have the brain, you can read signals from it to
Speaker 2: make a robotic body do things. You can decode the
Speaker 2: activity in the brain. Now, in episode two, I talked
Speaker 2: about many of these examples and a lot more, and
Speaker 2: I pointed out that we shouldn't be surprised that we
Speaker 2: can figure out how to move robotic arms with our thoughts.
Speaker 2: It's the same process by which your brain learns to
Speaker 2: control your natural, fleshy limbs. As a baby, you flailed
Speaker 2: your appendages around, and you bit your toes, and you
Speaker 2: grabbed your crib bars, and you poked yourself in the eye,
Speaker 2: and you turn yourself over. For years you did that,
Speaker 2: and that's how your muscles learned to mind read the
Speaker 2: command center of the brain. Eventually you could walk and
Speaker 2: run and do cartwheels and ice skate and so on,
Speaker 2: because your brain generates the signals and the muscles do
Speaker 2: as commanded, and so the brain naturally reads from these systems,
Speaker 2: and we can do so now artificially. So nowadays you
Speaker 2: have people who are paralyzed successfully running brain controlled electric wheelchairs.
Speaker 1: It's the same idea.
Speaker 2: You measure brain activity in some way, either with surface
Speaker 2: electrodes on the outside, or with higher resolution like electrodes
Speaker 2: on the surface of the brain, or even with a
Speaker 2: grid of electrodes poked a little bit into the brain
Speaker 2: tissue and you read what the motor intention is, like
Speaker 2: move forward or turn right, and you control the chair accordingly.
Speaker 2: And this sort of work is exploding. A few years ago,
Speaker 2: Stanford researchers implanted two little arrays of microelectrodes in a
Speaker 2: small part of the brain, the premotor cortex, in a
Speaker 2: person who was paralyzed, and then the person was able
Speaker 2: to do mind writing, which was turning the signals from
Speaker 2: the motor cortex into handwritten letters on the computer screen.
Speaker 2: So the person thinks about making the hand motions to
Speaker 2: write big letters, and then you see the lets get
Speaker 2: drawn on the computer screen and the person could do
Speaker 2: about ninety characters a minute.
Speaker 1: And then just in August.
Speaker 2: Twenty twenty three, two amazing papers came out side by
Speaker 2: side in the journal Nature, and by the way, I'm
Speaker 2: linking all these papers at eagleman dot com slash podcast.
Speaker 2: The first paper was from Stanford. They were working with
Speaker 2: a person with als. So this person had a growing
Speaker 2: paralysis and at this point couldn't speak understandably anymore. So
Speaker 2: the team put in two small electrode arrays. These are little,
Speaker 2: tiny square grids of electrodes. They're small, They're like three
Speaker 2: point two millimeters on his side, way smaller than a penny,
Speaker 2: and they dunk these into an area just in front
Speaker 2: of the motor cortex, a pre motor area called ventral
Speaker 2: area six. Okay, I'm going to skip all the details,
Speaker 2: because the important part is that the researcher said, look,
Speaker 2: given these signals worth seeing, what is the probability of
Speaker 2: the sound the person is trying to say, and also
Speaker 2: what are the statistics of the English language, And they
Speaker 2: combine those probabilities to figure out the most likely sequence
Speaker 2: of words the person is trying to say, just by
Speaker 2: looking at signals in the brain, and it worked really well.
Speaker 2: If the person tried to say any one of fifty words,
Speaker 2: the computer got it right nine out of ten times,
Speaker 2: And when the person trained up on a huge vocabulary
Speaker 2: of over one hundred thousand words, the computer got it
Speaker 2: right three out.
Speaker 1: Of four times.
Speaker 2: This is incredible because the person is not using his
Speaker 2: mouth but only brain signals, and the researchers could decode
Speaker 2: those and hear the person talking with them. And as
Speaker 2: I said, at the exact same time that paper came out,
Speaker 2: a second paper in nature came out from UCE San
Speaker 2: Francisco and UC Berkeley. It was the same basic idea,
Speaker 2: but here they were using surface electrodes on the brain.
Speaker 2: This intracordical eeg. So imagine two post it notes up
Speaker 2: against the surface of your brain. It sort of looks
Speaker 2: like that anyway, Slightly different signals spread over sensory and
Speaker 2: motor cortex. And they worked with a woman who had
Speaker 2: had a brainstem stroke eighteen years ago, and she can't
Speaker 2: speak and she can't type. So just imagine being able
Speaker 2: to think clearly but have no capacity for output. It's
Speaker 2: a nightmare, right, So they were not only able to
Speaker 2: decode her words what she was trying to say, but
Speaker 2: they were also able to decode what signals her brain
Speaker 2: was trying to send to her facial muscles, and they
Speaker 2: made an avatar on the screen that would speak her words.
Speaker 1: And show her facial expressions.
Speaker 2: And this is so amazing because it's taking the signals
Speaker 2: that are hidden on the inside of the skull and
Speaker 2: surfacing these, exposing these, turning these into something we can
Speaker 2: understand on the outside and watch on the computer screen.
Speaker 2: So this certainly seems like the kind of thing we
Speaker 2: would want when we talk about mind reading. And I
Speaker 2: also want to say that the researchers and surgeons who
Speaker 2: are doing this are doing incredible pioneering work. But after
Speaker 2: we've lived with these mind bending results for a while,
Speaker 2: it seems easy and obvious that this should be possible.
Speaker 2: We're finding these signals that are often complex and covered
Speaker 2: giant swaths of neurons, but you know that they're correlated
Speaker 2: with movements or pictures or sounds, and you simply throw
Speaker 2: machine learning at it to decode the relationship. And once
Speaker 2: you've read the brain signals and correlated them to their output,
Speaker 2: you can now figure out the intention. So all this
Speaker 2: is amazing news about where we are right now. And
Speaker 2: if you were to drop off from this podcast now,
Speaker 2: you'd probably say that we have achieved mind reading. But
Speaker 2: here comes the plot twist into Act two. It seems
Speaker 2: like mind reading is here, and if you judged just
Speaker 2: from magazine and news articles, you would be certain that
Speaker 2: we're reading minds and that the next step is to
Speaker 2: read everyone's mind and their innermost thoughts. But I want
Speaker 2: to explain why we are actually quite distant from that.
Speaker 2: What I've been telling you about is brain reading, and
Speaker 2: it's amazing, but we should reserve the term mind reading
Speaker 2: for reading the contents of your mind, your thoughts, And
Speaker 2: as stunning as all that current research is, it's not
Speaker 2: reading thoughts.
Speaker 1: Why because unlike a.
Speaker 2: Picture or a piece of music, or a decision about
Speaker 2: which way to go next, a real thought that crosses
Speaker 2: your mind can be a pretty different creature. Now, with
Speaker 2: everything I've told you so far, you might say, well, fine,
Speaker 2: it seems like reading someone's mind is just the next
Speaker 2: layer of resolution, like going from black and white TV
Speaker 2: to color TV. But I'm going to suggest that analogy
Speaker 2: is a false one and we're going to unpack that now.
Speaker 1: So really take stock of the thoughts.
Speaker 2: That you have in a day, their complexity and their weirdness,
Speaker 2: that'll allow you to understand whether and how we could
Speaker 2: really do mind reading. So the other day I was
Speaker 2: driving on the highway and I glanced at a billboard,
Speaker 2: and I really took stock of the three second stream
Speaker 2: of thoughts in my head. It went something like this, Oh,
Speaker 2: there's the billboard again for that software company. Tommy first
Speaker 2: showed me that app six years ago, or maybe it
Speaker 2: was seven years ago. Tommy's beard was so bushy, which
Speaker 2: looked unusual on such a young face. Whatever happened to
Speaker 2: him and that girl that he was dating, what was
Speaker 2: her name, Jackie Jane?
Speaker 1: I wonder how.
Speaker 2: Tommy's doing now. Ah, that reminds me. I need to
Speaker 2: send him an email, because I think he sent me
Speaker 2: one a few weeks ago about a paper he wanted
Speaker 2: to co author. Now, if you could read out that thought,
Speaker 2: that's what I would call mind reading.
Speaker 1: Not what's the picture you're.
Speaker 2: Seeing or the sound you're hearing, but that kind of thought,
Speaker 2: in all of its weirdness and richness. Now you may
Speaker 2: be thinking, but wait, didn't I just hear about some
Speaker 2: other nature paper in twenty twenty three that seems to
Speaker 2: do just that reads people's thoughts sort of. My colleague
Speaker 2: Alex huss At Ut Austin and his grad student Jerry
Speaker 2: Tang did a wonderful piece of work. He used fMRI
Speaker 2: and married it to a large language model GPT one.
Speaker 2: They put people in the scanner and they had them
Speaker 2: listen over many days to sixteen hours of storytelling podcasts
Speaker 2: like The Moth. So the researchers know the words that
Speaker 2: were said, and they measure what's happening in the brain
Speaker 2: and they throw the AI model at it to get
Speaker 2: a clearer and clearer picture of what words cause activation
Speaker 2: in what areas. Then you have the person think about
Speaker 2: a little story. They say the words in their head,
Speaker 2: like quote, he was coming down a hill at me
Speaker 2: on a skateboard and he was going really fast and
Speaker 2: he stopped just in time. And you measure the brain
Speaker 2: activity and you stick the AI model on it to
Speaker 2: tell you what words were being thought of. Now, the
Speaker 2: way they use GPT one is to narrow down the
Speaker 2: word sequences by asking what words are most likely to
Speaker 2: come next. So you take all this noisy brain activity
Speaker 2: and you force it down a straight and narrow path
Speaker 2: that makes it sound sentency. So the decoder doesn't get
Speaker 2: all the words, but the idea, the hope, is that
Speaker 2: it gets the gist, right, So does it well?
Speaker 1: Sort of?
Speaker 2: It depends how loosely you're willing to interpret things. So,
Speaker 2: for the sentence about the skateboard, that internally thought sentence
Speaker 2: again is he was coming down a hill at me
Speaker 2: on a skateboard and he was going really fast and
Speaker 2: he stopped just in time. And the decoder comes up
Speaker 2: with he couldn't get to me fast enough. He drove
Speaker 2: straight up into my lane and tried to ram me
Speaker 2: which is not really like the original sentence at all,
Speaker 2: but if you squint really hard, you could say the
Speaker 2: gist is similar. Or here's another example. The participant thought quote,
Speaker 2: look for a message from my wife saying that she
Speaker 2: had changed her mind and that she was coming back,
Speaker 2: And the dakoder translated this as to see her. For
Speaker 2: some reason, I thought she would come to me and
Speaker 2: say she misses me.
Speaker 1: So exciting stuff.
Speaker 2: But it's far from perfect for even a simple sentence.
Speaker 2: So let's return to this issue of whether you could
Speaker 2: tell if someone is trying to carry a bomb onto
Speaker 2: a flight. You might argue that even if the system
Speaker 2: is imperfect, you might be able to see something cooking
Speaker 2: under the surface about a bomb, and maybe that could
Speaker 2: get us close to mind reading. The problem, of course,
Speaker 2: is that lots of people in the airport are thinking
Speaker 2: about a bomb. For almost everybody, that's I sure hope
Speaker 2: there's not a bomb on this plane. And certainly if
Speaker 2: you know that your government in twenty years has perfected
Speaker 2: remote neuroimaging and is seeing if you're thinking about a bomb.
Speaker 1: Then you can be guaranteed that everyone is going to
Speaker 1: be thinking of a bomb.
Speaker 2: It's like asking someone to not think of a yellow bird.
Speaker 2: As soon as I ask that, you can't help but
Speaker 2: to think about a yellow bird.
Speaker 1: The more you try not to.
Speaker 2: The more your brain is screaming off with yellow feathers.
Speaker 2: But what I want to make really clear is that
Speaker 2: the problem with mind reading is even far tougher than that.
Speaker 2: We need to recognize that our thoughts are much richer
Speaker 2: than sequences of words. When you really take into account
Speaker 2: all the color and emotion of our thoughtscape most of
Speaker 2: it is beyond just the words. So really consider thoughts
Speaker 2: that you have. I'll give you another one of mine
Speaker 2: from the other day. I walked into a restaurant and
Speaker 2: I smelled a syrup being poorn on a pancake, and
Speaker 2: that reminded me of the time that I was in
Speaker 2: the tenth grade and sat with my friend at the
Speaker 2: International House of Pancakes.
Speaker 1: I was wearing.
Speaker 2: A striped sweater and my friend was in his def
Speaker 2: Leopard concert t shirt, and we had the thrill of
Speaker 2: being away from our parents and out on our own,
Speaker 2: just entering our teen years, and we were flirting with
Speaker 2: the waitress who had a German accent, who was at
Speaker 2: seven years older than us, and she was kind enough
Speaker 2: to give us the illusion of flirting back, or so
Speaker 2: we thought at the time, but now a better interpretation
Speaker 2: would perhaps be that she just wanted a better tip.
Speaker 2: And I asked her to help me with a German
Speaker 2: vocabulary word that I was working on, and she got
Speaker 2: uncomfortable and wouldn't do it. And suddenly I was seized
Speaker 2: with the suspicion that she wasn't actually German after all,
Speaker 2: but for some unimaginable reason, had been pretending to be German.
Speaker 2: Was she in fact American or from somewhere else in Europe?
Speaker 2: What was her reason for telling us she was German
Speaker 2: and putting on that accent when she clearly didn't know
Speaker 2: how to speak the German language. Now that memory, that
Speaker 2: moment in time involves territories all over my brain. The
Speaker 2: taste of the pancakes, the smell of the coffee, the
Speaker 2: sights and colors of the pancake house, the sound of
Speaker 2: my friend's fits of laughter, but also things that are
Speaker 2: more subtle, more difficult to simply re out, like my
Speaker 2: interpretation then and now of the reasons behind her behavior,
Speaker 2: my feelings of doubt and skepticism, and for that matter,
Speaker 2: of a confused and unrequited pubescent crush and mystery. This
Speaker 2: is the type of thing that's not stored in anyone
Speaker 2: blob in the brain. It's instead stored in a distributed
Speaker 2: fashion that covers all territories of the brain. And more importantly,
Speaker 2: it's not just words that I'm thinking of. Most of
Speaker 2: it is subtle events and emotions, and I have to
Speaker 2: actively work to translate them into words to even communicate
Speaker 2: this memory on a podcast. But words is not how
Speaker 2: I experience it as it flips through my mind, and
Speaker 2: hence the challenge of reading out something like that. So,
Speaker 2: despite the tremendous progress that's going on, I want us
Speaker 2: to retain this distinction between brain reading guessing what video
Speaker 2: you're seeing or song you're hearing, or word you want
Speaker 2: to say, and mind reading for reading out a rich
Speaker 2: internal experience of some thought or reminiscence or plan for
Speaker 2: the future. Now you might say, Okay, look, that's a
Speaker 2: tough problem to get the real stream of consciousness kind
Speaker 2: of thoughts. But this should become possible as we get
Speaker 2: better resolution in measuring the brain. So let's consider whether
Speaker 2: that challenge seems realistic. So one thing we might need
Speaker 2: would be a technology that could measure that tends to
Speaker 2: hundreds of little electrical spikes every second in every single
Speaker 2: one of the eighty six billion neurons in the cortex
Speaker 2: in real time and record those results. Now, that might
Speaker 2: sound like a straightforward technology problem, and in this century
Speaker 2: we're used to going out and solving big problems. But
Speaker 2: I want to be clear that this would make terraflops
Speaker 2: of data every minute, and that's a ton of data.
Speaker 2: But let's imagine that we flex all our technical muscles
Speaker 2: and figure out how to solve that. Amazing, But that's
Speaker 2: not actually the problem. The technical feat is just the
Speaker 2: warm up problem. The fundamental problem is the interpretation. Let's
Speaker 2: say I handed you this file of the eighty six
Speaker 2: billion neurons popping off for two seconds, and I told
Speaker 2: you this is a particular memory. So you're looking in
Speaker 2: great detail at a pattern, a swirl of billions of neurons.
Speaker 2: But imagine that I don't tell you if this was
Speaker 2: measured from Fred's brain or from Susie's brain, this exact
Speaker 2: pattern among the neurons would mean totally different things in
Speaker 2: different heads. Fred grew up in China and is thinking
Speaker 2: about the time that he climbed the tree and saw
Speaker 2: the panoramic view, and his mother, who suffered from anxiety,
Speaker 2: yelled at him because she was scared because she had
Speaker 2: lost her younger brother in an accident. While in Susie's
Speaker 2: brain this pattern would need something entirely different. She grew
Speaker 2: up in New Mexico, and for her it would mean
Speaker 2: she's thinking about the time that she went on a
Speaker 2: hike and saw wild horses and got in trouble because
Speaker 2: her friend had parked his truck behind her mother's car,
Speaker 2: and her mother couldn't get out and they had to
Speaker 2: run all the way down the mountain to move his truck. Now,
Speaker 2: how could such different memories have the same pattern of neurons.
Speaker 2: It's because who you are is determined by the exact
Speaker 2: wiring of your brain, which is determined by everything that
Speaker 2: has come in, all your memories and experiences of your life.
Speaker 2: Everything about your identity is stored in the particular configuration
Speaker 2: of your tens of billions of neurons and hundreds of
Speaker 2: trillions of connection. That's what makes you you. When you
Speaker 2: learned that a cloud is the same thing as fog,
Speaker 2: just at a different elevation, or you learned that Isaac
Speaker 2: Newton was into alchemy, or you learned that the Bissie
Speaker 2: breed of dog is from Africa and has a curled tail,
Speaker 2: this is all stored by changes in your network. That's
Speaker 2: what it means to learn something and to recall it later.
Speaker 2: It means your network has changed to store the information.
Speaker 2: And so as you go through life and you have
Speaker 2: your unique experiences, your brain gets wired up in a
Speaker 2: totally different way than someone else's, with the end result
Speaker 2: being that your enormous neural forest, your inner cosmos, is
Speaker 2: totally different than someone else's. And that means that if
Speaker 2: I measured neuron number sixty nine hundred and thirty five
Speaker 2: in your head, it would tell me little to nothing
Speaker 2: about what that neuron firing means in someone else's head.
Speaker 2: And this is of course what the researchers at Berkeley
Speaker 2: and Austin and Stanford find. To guess at the words
Speaker 2: that someone is trying to communicate, they have to train
Speaker 2: the model on each person individually.
Speaker 1: The AI model.
Speaker 2: Trained on one person's brain has no chance of reading
Speaker 2: another person's brain. So to do any sort of meaningful
Speaker 2: mind reading, we would need to do what's called system identification.
Speaker 2: What is system identification, It's a process in engineering of
Speaker 2: figuring out the internal structure of a system by sending
Speaker 2: in inputs and looking at the outputs. Like if I
Speaker 2: hit this button, it beeps, and I hit that button
Speaker 2: and it boops, and so on through all the buttons
Speaker 2: and all the combinations, and I can get a pretty
Speaker 2: good idea of what the system must look like on
Speaker 2: the inside. You give lots of inputs, you look at
Speaker 2: all the outputs, and that's how you figure out the
Speaker 2: parameters of a control system, or the dynamics of a
Speaker 2: physical system, or in this case, the structure of a
Speaker 2: neural network a person's brain. The key is that if
Speaker 2: you want to do meaningful mind reading, you'd really have
Speaker 2: to know essentially everything about an individual person.
Speaker 1: So I want to know two things here.
Speaker 2: First, In a sense, this system identification is what long
Speaker 2: term relationships are about. Let's say with a spouse, you
Speaker 2: see a person in lots of different situations, and in
Speaker 2: each you see how they react and how they handle things,
Speaker 2: And this is what it means to get to know
Speaker 2: someone well. But second, we can never do it with
Speaker 2: very high precision. For anyone who's been in long term relationships,
Speaker 2: you know that even with years of experience, our models
Speaker 2: of another person are always impoverished. One of the things
Speaker 2: that I enjoyed the most about being in a relationship
Speaker 2: with my spouse is to watch when she has a
Speaker 2: funny facial expression or stares at something on you usual
Speaker 2: that I wouldn't have expected her to stare at, and
Speaker 2: trying to figure out what in the world is going
Speaker 2: on inside her head. And like any person in a
Speaker 2: long term relationship, I'll tell you that often I have
Speaker 2: no idea, or I'll make a guess or have an assumption,
Speaker 2: but I am almost certainly doomed to be incorrect. And
Speaker 2: this is an example of a more general issue that
Speaker 2: I've mentioned in other episodes, which is that all we
Speaker 2: have is our own internal model of the world, and
Speaker 2: we can never really know what's happening inside someone else's head,
Speaker 2: and so we make our guesses, and we say things
Speaker 2: to people like, oh, I know exactly how you feel,
Speaker 2: and we mean well when we say that, but we're
Speaker 2: always off in our assumptions. It is difficult or impossible
Speaker 2: to understand exactly how someone.
Speaker 1: Else is feeling. Why.
Speaker 2: It's because we've all had a history of years and
Speaker 2: years of deep experience in the world, things that were
Speaker 2: scary to us or tiddle or amazing or dumbfounding or
Speaker 2: attention grabbing, and all of these form pathways in the
Speaker 2: unspeakably large neural forest of our tens of billions of
Speaker 2: neurons and hundreds of trillions of connections. So when you
Speaker 2: zoom passed the billboard on the highway, the one that
Speaker 2: made me think of my student and his beard and
Speaker 2: his girlfriend whose name I couldn't remember, and the email
Speaker 2: that I owed him, you would have a totally different
Speaker 2: cascade of thoughts, same visual input, totally different internal waterfall.
Speaker 2: And unless I knew really everything in your life, it
Speaker 2: would be impossible for me to decode your particular idiosyncratic
Speaker 2: stream of consciousness. Why because I'd be tasked with trying
Speaker 2: to get some sense of a neural system that is
Speaker 2: built from mind boggling complexity that has been developing over
Speaker 2: years and decades of experience, every single experience left, and
Speaker 2: its fingerprint in the deep forests of your brain. So
Speaker 2: while some general structures tend to be the same, like
Speaker 2: where your visual system is, and how your temperature regulation works,
Speaker 2: and the hormones that control your digestive system and so on,
Speaker 2: all the rest is shaped uniquely, which is what makes
Speaker 2: one person different from the next. As you have followed
Speaker 2: your thin trajectory of space and time from your parents,
Speaker 2: to your hometown, to your relationships, your school, friends, your sports,
Speaker 2: your tragedies, your successes, your heart breaks, your life, that's
Speaker 2: what has led your brain to be unique. Now you
Speaker 2: can get a graduate student in the lab to lie
Speaker 2: down in the brain scanner and you can show scene
Speaker 2: after seeing to figure out the rough shape of their
Speaker 2: semantic network. In other words, which areas tend to be
Speaker 2: activated by which words or concepts, And this is what
Speaker 2: the labs at Berkeley and U T Austin and others do.
Speaker 2: But it's not at all clear how you would get
Speaker 2: something meaning from a stranger in the airport whose brain
Speaker 2: you didn't already know in detail, Like, how could you
Speaker 2: know if that guy is thinking, I disagree with this
Speaker 2: political position, and so my goal is to murderously sneak
Speaker 2: a bomb onto the airplane. Now, I just want to
Speaker 2: be really clear about the assertion I'm making. This challenge
Speaker 2: we face is not a scientific challenge of programming better
Speaker 2: AI decoders.
Speaker 1: It's not a technical issue.
Speaker 2: Instead, it's about knowing you as an individual. We are
Speaker 2: never going to get to real rich mind reading unless
Speaker 2: we have something in the future that accompanies you and
Speaker 2: tracks every single experience you have from the time you're
Speaker 2: an infant, every scary moment, every class you sat through,
Speaker 2: every conversation you had, every book you read, every movie
Speaker 2: you saw, every sexual encounter. This device would have to
Speaker 2: record everything you see and hear in experience, and then
Speaker 2: maybe we could combine that device's data with a sophisticated
Speaker 2: model of your genetics, or maybe combine that with a
Speaker 2: bunch of input output system identification, and then maybe we'd
Speaker 2: be close. So you might think, fine, if we could
Speaker 2: actually record everything in someone's life, then we'd be able
Speaker 2: to do it that way. After all, you might point
Speaker 2: out people in the near future might wear portable cameras
Speaker 2: every moment of their life. But really the problem of
Speaker 2: system identification is even harder than that, and that's because
Speaker 2: your internal life isn't just predicated on your inputs and outputs.
Speaker 2: So the brain reading tasks that I told you about
Speaker 2: at the beginning, the researchers know exactly what stimulus was
Speaker 2: presented and know exactly the brain signals that resulted, and
Speaker 2: that's what the AI is trained up on. But lots
Speaker 2: of things in our inside life don't come with a
Speaker 2: clear external input. Example, let's say our recording device detects
Speaker 2: that you meet with a friend at a cafe, and
Speaker 2: it measures all the conversation and your corresponding brain signals,
Speaker 2: but it wouldn't know that you thought about that encounter
Speaker 2: later you saw an analogy between something that was said
Speaker 2: during that conversation to something else in your life. Two
Speaker 2: weeks later, you have a tiny revelation about the behavior
Speaker 2: of your date back when you were a kid and
Speaker 2: went to the high school prom. So for many of
Speaker 2: the things screaming around in your internal cosmos. There's no
Speaker 2: external input, there's nothing to film, and this is true
Speaker 2: of most of your internal thoughtscape You might see somebody
Speaker 2: sitting in the booth next to you, or you might
Speaker 2: walk through the shopping mall and you see different people
Speaker 2: and you think whatever you think about them. They're attractive,
Speaker 2: they're unattractive, they look like my friend from back home,
Speaker 2: they look sad, they look pleased.
Speaker 1: Whatever.
Speaker 2: The challenge for training and input output system like the
Speaker 2: in the research papers, is that the overwhelming majority of
Speaker 2: your thoughts are private. You don't say your thoughts out loud,
Speaker 2: and no one from the outside ever knows what's going
Speaker 2: on in there, and therefore a researcher trying to train
Speaker 2: up an AI model doesn't have the data to train
Speaker 2: the model with. You might walk down the street and
Speaker 2: think that person looks lost, or that person has weird
Speaker 2: shoes and so on, but these are thoughts that you
Speaker 2: never express, so we can't suss out the input output mapping.
Speaker 2: It's not just a matter of having sophisticated enough algorithms
Speaker 2: to measure and store and analyze the patterns. It's that
Speaker 2: the patterns are inherently unpredictable because we don't know what
Speaker 2: causes most of them and how those patterns lead to
Speaker 2: the next. And I just want to mention one more
Speaker 2: reason why mind reading won't be done in airports, not
Speaker 2: now and not in two centuries from now. You need
Speaker 2: the part participants cooperation to train up the decoder and
Speaker 2: to apply the decoder. If people didn't want to train
Speaker 2: up on this, they could think whatever random thing they
Speaker 2: wanted and mess up the whole process. So yet another
Speaker 2: reason why system identification will be difficult unless you want
Speaker 2: to be system identified. So meaningful mind reading reading the
Speaker 2: stream of consciousness isn't going to happen in our lifetimes.
Speaker 2: Maybe some new kind of recording technology and brain reading
Speaker 2: loop will get developed in a thousand years from now,
Speaker 2: but this is not for us or our near term descendants. Instead,
Speaker 2: we will continue to live in a world of mysterious
Speaker 2: strangers stuck inside their own skulls, each person the sole
Speaker 2: inhabitant of his or her internal world.
Speaker 1: So let's wrap up.
Speaker 2: There's been excitement and genuine an incredible progress, but this
Speaker 2: should be carefully distinguished as brain reading. When we read
Speaker 2: the brain patterns to know what a person is seeing
Speaker 2: or what music they're hearing, or read the premotor cortex
Speaker 2: to understand what words they want to say. These are
Speaker 2: technological feats that would have blown the hairback of the
Speaker 2: best neuroscientists fifty years ago.
Speaker 1: But it's not mind reading.
Speaker 2: Mind reading would be an understanding of the universe of
Speaker 2: swirling thoughts, which is where we spend most of our lives.
Speaker 2: Just look at a person sitting at the bus stop,
Speaker 2: or walking silently down the street, or sitting at the library.
Speaker 2: Their minds are swirling with an internal world, considering what
Speaker 2: they're going to do and what they're going to say,
Speaker 2: and maybe reminiscing about the past or feeling regret over
Speaker 2: a possible present that could have been. It's not about
Speaker 2: the final output of what they're trying to say, or
Speaker 2: the input of what's on their eyes or ears.
Speaker 1: It's much deeper than that.
Speaker 2: It's the entire rest of the brain, beyond the little
Speaker 2: territories of input and output. To read the rest of
Speaker 2: that area, to read the private thoughts of a person
Speaker 2: would require an individualized understanding, a system identification that we
Speaker 2: need to track essentially every event in their life and
Speaker 2: probe every single neural reaction and have some way to
Speaker 2: at least guess or impute what all the internal washing
Speaker 2: machine of thought is doing. We are centuries away from that,
Speaker 2: and in a sense that's good news. It means that
Speaker 2: we can enjoy helping people who are mute express themselves,
Speaker 2: or a person who's paralyzed move a wheelchair or a robot,
Speaker 2: or otherwise make things happen in the world. But happily,
Speaker 2: it also means that we retain, for the foreseeable pretty
Speaker 2: distant future, sure the sanctity and privacy of the swirling
Speaker 2: galaxies in our inner cosmos. Go to eagleman dot com
Speaker 2: slash podcast for more information and to find further reading
Speaker 2: and to see videos. Send me an email at podcasts
Speaker 2: at eagleman dot com with questions or discussion, and I'll
Speaker 2: be making mailbag podcasts to address those until next time.
Speaker 2: I'm David Eagleman, and this is Inner Cosmos.