Speaker 1: What is a brain computer interface? How far along is
Speaker 1: this field? Can we evesdrop on the brain so that
Speaker 1: a person who has lost the ability to move can
Speaker 1: use their brain to control a computer cursor or a
Speaker 1: robotic arm. Can someone who has lost the ability to
Speaker 1: speak send brain signals to a decoder and hear their
Speaker 1: voice again? Can we restore autonomy and dignity and eventually
Speaker 1: do so so seamlessly that the technology disappears and the
Speaker 1: person reappears In the future, where will the ethical boundaries
Speaker 1: be between restoring function and spying on private thought? And
Speaker 1: who owns the stream of neural data that represents you?
Speaker 1: Welcome to Inner Cosmos with me David Eagleman. I'm a
Speaker 1: neuroscientist and author at Stanford and in these episodes we
Speaker 1: sail deeply into our three pound universe to understand why
Speaker 1: and how our lives look the way they do. This week,
Speaker 1: we're talking about technology for reading the brain. Now. One
Speaker 1: thing that I find fascinating is that ancient cultures didn't
Speaker 1: care at all about the brain. They generally would just
Speaker 1: throw it out at autopsy, and it's understandable why it
Speaker 1: just looks and feels like a huge, squishy walnut. If
Speaker 1: you could sit and stare at a brain in action,
Speaker 1: you wouldn't see anything happening. So it's taken centuries and
Speaker 1: a lot of technology to realize that, in fact, the
Speaker 1: brain is alive with lots of tiny cells, microscopically tiny,
Speaker 1: and these cells are transmitting electrical signals tens or one
Speaker 1: hundred times every second for each cell. And you have
Speaker 1: eighty six billion of these cells. So this big, squishy
Speaker 1: walnut is one of the busiest things on the planet.
Speaker 1: But because it is so fragile, Mother Nature surrounds the
Speaker 1: brain with an armored bunker plating the skull, and that
Speaker 1: provides a huge challenge if you want to go in
Speaker 1: there and eavesdrop on what the cells are doing. Now,
Speaker 1: why would you want to spy on these cells? Well,
Speaker 1: imagine if your thoughts could exit the skull as easily
Speaker 1: as words leave your mouth. Now, there's a sense in
Speaker 1: which we always do this. We use keyboards, touch screens,
Speaker 1: and voice assistants, but all of those are detours. They
Speaker 1: force the brain to root its intentions through muscle, and
Speaker 1: that's fine if your muscles work. The problem is that
Speaker 1: lots of people, millions of our neighbors and friends don't
Speaker 1: have a way to get the information out of their
Speaker 1: brain because something about the brain or the brain's pathways
Speaker 1: or the muscles are not working, and therefore their brain
Speaker 1: knows what they want to do or say, but there's
Speaker 1: no way to get that information out. And this is
Speaker 1: where the idea of a brain computer interface comes in.
Speaker 1: What you'll hear referred to as a BCEI brain computer interface.
Speaker 1: The idea of a BCI is to listen directly to
Speaker 1: the neural patterns that mean move or speak or select,
Speaker 1: and then you use some device to translate those patterns
Speaker 1: directly into activation in the outside world. Now, as I said,
Speaker 1: this is a huge deal for all the people for
Speaker 1: whom the path from intention to movement has been interrupted
Speaker 1: by disease or injury. The intent is still alive and
Speaker 1: well in the cortex, and BCIs are the bridge back.
Speaker 1: They turn silent plans into text or voice or cursor
Speaker 1: control or reaching and grasping. But the story will, at
Speaker 1: least in theory, reach beyond the medical because once you
Speaker 1: can read out the programs for say this word or
Speaker 1: press that key, now you've built a communication channel between
Speaker 1: biological tissue and silicon, and that opens new forms of
Speaker 1: interaction that our species has barely begun to imagine. Now,
Speaker 1: let me not get ahead of myself yet, because as
Speaker 1: we're going to see today, we are still at the
Speaker 1: earliest stages of this technology. But this is what we're
Speaker 1: going to talk about at the end. Now, you can
Speaker 1: build bceiyes in lots of flavors. Some rest on the scalp,
Speaker 1: Others sit on the surface of the brain. Others poke
Speaker 1: tiny wires called electrodes into the surface of the brain
Speaker 1: or even down deep into the brain for some purposes.
Speaker 1: Some of these BCIs only read the electrical activity. Others
Speaker 1: will also write with electrical patterns that the brain experiences
Speaker 1: as touch or sound or sight. In every case, the
Speaker 1: principle is the same. Brains issue commands, and they're very
Speaker 1: fast and complex internal language of electrical spikes. This is
Speaker 1: a language that we haven't nearly decoded yet, but machines
Speaker 1: can learn to translate that language through a lot of
Speaker 1: trial and error. Huge populations of neurons are playing some
Speaker 1: symphony piece, and these decoders learn how to hear the
Speaker 1: music and root the commands to a cursor or a
Speaker 1: speaker or a robotic arm or whatever. Now. The issue
Speaker 1: is that when we talk about it, it all seems
Speaker 1: very straightforward and easy, but actually getting in there and
Speaker 1: getting technology that can record from these microscopic little cells,
Speaker 1: having these little changes in their electrical potential of tens
Speaker 1: of millivolts, and making a system that lasts, and then
Speaker 1: putting all the data together to understand what this very
Speaker 1: tiny sampling of neurons, maybe a few hundred out of
Speaker 1: hundreds of billions of neurons. It turns out this is
Speaker 1: a massive engineering challenge and there are a million practical questions.
Speaker 1: How reliable are these systems outside the lab? Can they
Speaker 1: survive infection and signal drift? What about battery life? What's
Speaker 1: the surgical risk? When does insurance cover these? So there's
Speaker 1: a huge gap between a beautiful proof of principle and
Speaker 1: a device that changes lives every day, and crossing that
Speaker 1: gap is the real work of the field right now.
Speaker 1: Now there's also a second issue. As soon as we
Speaker 1: start talking about reading the brain, the questions start to surface,
Speaker 1: what exactly are we reading? Is it intended movements? That's
Speaker 1: one thing is that inner speech? Is it where you
Speaker 1: place your attention? You can imagine situations in which there
Speaker 1: are things that you don't want everyone knowing. We're used
Speaker 1: to the skull having some sort of sanctity. So where
Speaker 1: will the ethical boundaries be between restoring function and evesdropping
Speaker 1: on private thought? Who's going to own the stream of
Speaker 1: data that is literally you? How do we guarantee consent
Speaker 1: and security and dignity when the interface is not on
Speaker 1: your desk but inside your skull. So, even in the
Speaker 1: face of all the tough questions coming down the pike,
Speaker 1: it's hard not to feel awe at what's already possible.
Speaker 1: Who have been locked inside their bodies are communicating again.
Speaker 1: They're talking with their loved ones for the first time
Speaker 1: in years. And the technology keeps improving every month, smarter algorithms,
Speaker 1: better sensors, cleaner signals, and crucially designs that move from
Speaker 1: the hospital to the home. So today I want to
Speaker 1: explore what that looks like and where we are in
Speaker 1: the process and where things are going. So I sat
Speaker 1: down with my colleague Sergei Stavisky. Sergei is at the
Speaker 1: UC Davis Neuroprosthetics Lab, which he co directs with neurosurgeon
Speaker 1: David Brandman. With their collaborators, they work on BCIs that
Speaker 1: restore communication and they're pushing towards systems that are fast
Speaker 1: and expressive and practical for everyday life. So here's my
Speaker 1: interview with Sergei Staviski.
Speaker 2: A brain computer interface is a device that interacts between
Speaker 2: technology and a brains. You have the brain, you have
Speaker 2: some way of getting information in or out, and you
Speaker 2: have some computation that's happening. And that computation it could
Speaker 2: be happening inside the body, so it could be a
Speaker 2: chip that does everything in the brain, or it could
Speaker 2: be sending that information to a laptop next to the person,
Speaker 2: or even to the cloud for more computation.
Speaker 1: Now, one of your interests is that you know, over
Speaker 1: a century ago people figured out you could dunk an
Speaker 1: electrode into the brain the thin wire and because cells
Speaker 1: are communicating with little electrical signals, you're you can eavesdrop
Speaker 1: on that and you can also stimulate the cell to
Speaker 1: do whatever. So tell us about the history of this,
Speaker 1: how people have thought about, let's eavesdrop on the brain
Speaker 1: and turn that into something.
Speaker 2: So starting in the sixties and seventies and eighties, especially
Speaker 2: working in animal models, people realized, yeah, you can put
Speaker 2: electrodes into the brain, and you can get up close
Speaker 2: next to an individual brain cell a neuron, and when
Speaker 2: that neuron's firing, it's genera a big electric field, a
Speaker 2: tiny electric field, but big relative to the electrode right
Speaker 2: next to it, And so.
Speaker 3: We know that that neuron is firing.
Speaker 2: And then there was a whole decades of systems neuroscience
Speaker 2: which was relating those patterns of activity to what typically
Speaker 2: the animal was doing. So a classic example from the
Speaker 2: eighties would be a monkey is moving his arm up
Speaker 2: or down, or left or right, and you can see
Speaker 2: that maybe a neuron fires more when the arm is
Speaker 2: moving to the left, and say, okay, that neuron has
Speaker 2: a left or preferred direction. We're starting to build some
Speaker 2: mental map of how that brain activity relates to movements.
Speaker 2: Of course, it's much more complicated, and the whole field
Speaker 2: of neuroscience is trying to understand how individual neurons and
Speaker 2: hundreds of neurons and whole large assemblies of neurons generate behavior.
Speaker 2: Starting around the two thousands, the field had felt that
Speaker 2: we had enough of a rudimentary understanding of how movement
Speaker 2: is encoded in the brain that this could be used
Speaker 2: for a medical application.
Speaker 3: And kind of in my world.
Speaker 2: That's been focused on restoring movement to people with paralysis.
Speaker 3: So in two.
Speaker 2: Thousand and four it was a big landmark event that
Speaker 2: was when the original brain Gate trial. So this was
Speaker 2: led by John Donahue in Lee Hagberg at Brown University
Speaker 2: in Masteronal Hospital. They put what was called a multi
Speaker 2: electro array, so instead of a single wire like you
Speaker 2: mentioned in the beginning, now imagine a hundred of those
Speaker 2: little wires kind of all stacked together, recording from thus
Speaker 2: about one hundred neurons. And they showed that these arrays
Speaker 2: could be put in a person with paralysis, and even
Speaker 2: though that person hadn't moved in a decade. I think
Speaker 2: the first guy was a young man in his twenties
Speaker 2: who had been paralyzed from the neck down due to
Speaker 2: a knife wound from like a bar fight. So he
Speaker 2: hadn't moved in many, many years. But they put that
Speaker 2: electro array in the motor cortex, the part of the
Speaker 2: brain that normally sends commands to the arm, and when
Speaker 2: he tried to move his arm, lo and behold, those
Speaker 2: neurons fired away. And so kind of the main risk
Speaker 2: had been solved, which is would the brain even still
Speaker 2: try to generate movements because you might think, well, use
Speaker 2: it or lose it. Right, the person's paralyzed, why would
Speaker 2: their brain still generate movement commands. Fortunately it still does,
Speaker 2: and people were able to decode those signals.
Speaker 1: And just as a quick reminder to everybody, the brain
Speaker 1: is saying, okay, I want you to make these movements,
Speaker 1: and then those shoot down down the spinal cord and
Speaker 1: out to the peripheral nervous system and move the muscles.
Speaker 1: And so in this case you're hearing the original command,
Speaker 1: but there's some break in the roadway plunging down the
Speaker 1: spinal cord and out such that the body never gets
Speaker 1: the signals correctly exactly.
Speaker 2: We're bypassing the injury. We're going to the source. So
Speaker 2: where's the command coming from?
Speaker 1: So this was back in two thousand and four, what
Speaker 1: was his name, Matt Nagel. Is that researchers are able
Speaker 1: to listen to what the neurons are intending, and then
Speaker 1: the field has really taken off since then in the
Speaker 1: past two decades. For example, with motor movement, originally it
Speaker 1: was just on a computer screen you could move a
Speaker 1: cursor around. Nowadays people are thinking about Hey, could you
Speaker 1: actually use an exoskeleton to move the arm physically?
Speaker 3: Yeah, or even stimulate those paralyzed muscles.
Speaker 2: So there's these functional electrical stimulation systems or epidural spinal stimulation,
Speaker 2: both for walking and for the arm. So you can
Speaker 2: really close the loop. You can decode what movement the
Speaker 2: person's trying to make.
Speaker 3: It.
Speaker 2: Oh, they're trying to move their arm forward to grab something,
Speaker 2: and then you can have that move a robotic arm.
Speaker 2: You could have that move an exoskeleton, or if they
Speaker 2: also have a stimulator that's implanted under the skin with
Speaker 2: wires going to the muscles or going outside of the spine,
Speaker 2: you can stimulate the body and actually have the person's
Speaker 2: own formally paralyzed muscles make that movement. It's not at
Speaker 2: the level that you or I let a healthy person
Speaker 2: is moving their arm, but it does work. There's been
Speaker 2: some really amazing studies in the last decade doing that.
Speaker 1: Yeah, exactly right, Okay, great, So that's how people have
Speaker 1: been using brain computer interfaces to move a paralyzed body. Now,
Speaker 1: something that several groups have gotten interested in in recent
Speaker 1: years is what if somebody can't speak anymore? So, what
Speaker 1: are the reasons. First of all, that somebody can't speak.
Speaker 2: So one common one is neurodegenerative diseases like ALS. So
Speaker 2: ALS is a terrible disease, hemiotrophic lateral sclerosis, right and
Speaker 2: right now there's no cure. We can't stop it with
Speaker 2: a drug or other therapy.
Speaker 1: Also known as Luke Gerrig's disease.
Speaker 2: That's right, yeah, and almost everyone who has ALS will
Speaker 2: gradually lose the ability to move their body. But also
Speaker 2: that means what we call the speech articulators, so their lips,
Speaker 2: their jaw, their tongue, their diaphragm, and so their speech
Speaker 2: becomes harder and harder to understand, and eventually you wind
Speaker 2: up what's called locked in, so really not able to
Speaker 2: move at all. And of course this is a terrible situation.
Speaker 2: And if there were a way to restore the ability
Speaker 2: to communicate, so like before decoding not now not they
Speaker 2: are movements that're trying to make, or the leg movements,
Speaker 2: but what are the words that're trying to make, or
Speaker 2: what are the movements of those articulars that they're trying
Speaker 2: to make. What's are they trying to produce? Then we
Speaker 2: can have this person communicate again and talk again through
Speaker 2: a computer.
Speaker 1: If you want to figure out what somebody is trying
Speaker 1: to say, where do you put the electrodes?
Speaker 3: Yeah, and that is the big question. So there are
Speaker 3: a lot of ideas.
Speaker 2: One idea would be the broker's area, which was thought
Speaker 2: to plan speech. Another idea would be the motor cortex,
Speaker 2: which would be kind of the last planning to command generation.
Speaker 2: So the part of the brain that's really sending signals
Speaker 2: to the muscles. And then there's a wide part of
Speaker 2: the brain that are called the language network.
Speaker 3: So this is the temporal lobe.
Speaker 2: It's canonically thought of for perceiving language, but also heavily
Speaker 2: involved in producing language. So there are a lot of
Speaker 2: possible choices. One of the challenges for developing a speech
Speaker 2: ne or prosthesis is there's no animal model. So when
Speaker 2: the field was trying to have people walk again or
Speaker 2: people move their arms again, we had a huge head
Speaker 2: start because you could say, okay, where can you code
Speaker 2: the walking or the arm moved of a rat or
Speaker 2: a monkey or another animal. Well, animals don't talk, they
Speaker 2: don't have language, so we don't have that kind of
Speaker 2: guidance for us, and what we do have are less
Speaker 2: precise measurements from other humans. A lot of the really
Speaker 2: important work from the last decade or twenty years was
Speaker 2: done with electrocorticography. So people with epilepsy often will have
Speaker 2: electrodes put under their skull, typically on top of their
Speaker 2: brain or even in their brain to for the neurologists
Speaker 2: to identify.
Speaker 3: Where the teacher is coming from.
Speaker 2: But these people are then in the hospital for a
Speaker 2: couple of weeks, and this is a gold mine for
Speaker 2: human neuroscience. A lot of what we know about direct
Speaker 2: brain recordings and how they relate to human specific behaviors,
Speaker 2: whether that's speaking or language, or imagination or memory.
Speaker 3: Or mood, all of these things.
Speaker 2: A lot of that comes from this sort of opportunistic
Speaker 2: recording people who are they're in the hospital anyway, they're
Speaker 2: kind of bored, they're waiting for the neurologists to have
Speaker 2: enough data, and so it's very easy to ask them, hey, do.
Speaker 3: You want to read a sentence off a screen.
Speaker 2: So from that we already knew that this sensory motor cortex.
Speaker 2: So the motor and the sensory cortex was a prime area,
Speaker 2: and in our brain Gate clinical trial, that's where we
Speaker 2: ended up putting electrodes, so in the motor part, basically
Speaker 2: the part of the brain that would typically send commands
Speaker 2: to the muscles.
Speaker 1: Great, so it's essentially like the last train station before
Speaker 1: it plunges down towards the muscles. Okay, so you're eavesdropping
Speaker 1: there and you're sticking these little electrode or raise these
Speaker 1: little square jobs where they have sixty four electrodes on
Speaker 1: the one and four of those.
Speaker 2: We used four of them, so yeah, four all along
Speaker 2: this precentral gyrus.
Speaker 1: So you're listening to these neurons and you're trying to
Speaker 1: decode what the person is intending to say from that.
Speaker 1: And one question, were you worried at the beginning that
Speaker 1: that wouldn't be enough data or did you feel like, look,
Speaker 1: with two hundred fifty six neurons, we can figure out
Speaker 1: what's going on in terms of what was trying to
Speaker 1: be articulated.
Speaker 2: When I started the project, I was pretty worried. So
Speaker 2: kind of the prior work is we had shown that
Speaker 2: with about one hundred electrodes in a different part of
Speaker 2: the brain, the hand part of motor cortex, we could
Speaker 2: decode speech, but very poorly. There I was classifying between
Speaker 2: the thirty nine phonemes in American English, if I recall
Speaker 2: about thirty three percent accuracy, So that's way better than chance.
Speaker 2: It showed there's information, but that is not good enough
Speaker 2: to understand.
Speaker 3: What someone's saying.
Speaker 1: Tell us what a phoneme is.
Speaker 3: A phoneme is a building block of speech.
Speaker 2: So I think most people are familiar with the syllables,
Speaker 2: think of a phoneme as a little bit smaller than that.
Speaker 2: So good, ooh E. Right, there's consonants, there's vowels. Different
Speaker 2: languages have different phonemes, but in English, depending on the
Speaker 2: dialect or accent, between thirty nine forty one. These are
Speaker 2: the typical ways we break down English.
Speaker 1: Got So you're recording from these neurons, and you were saying,
Speaker 1: can I figure out what phoneme person is trying to
Speaker 1: say right now and right now just from looking at
Speaker 1: this array of neural activity?
Speaker 3: That's exactly right.
Speaker 2: And a little bit before that, my colleagues at Stanford,
Speaker 2: and that was also the lab that I did my
Speaker 2: post doctoral training, and so I started that project then
Speaker 2: moved on. They had implanted one hundred and twenty eight
Speaker 2: electrodes in the motor cortex of a woman with als,
Speaker 2: and with that they were able to decode what words
Speaker 2: she was saying with about seventy five percent accuracy with
Speaker 2: a large vocabulary of one hundred and twenty five thousand words.
Speaker 2: So that was a really really exciting moment for the
Speaker 2: field because that was really banging at the door of
Speaker 2: making this useful for general communication. Now, three out of
Speaker 2: four words correct is amazing. It was way better than
Speaker 2: anything that ever been done before. But you can't have
Speaker 2: a conversation that way. It's just too frustrating. There's too
Speaker 2: many mistakes.
Speaker 1: And so when we will give us a sense of
Speaker 1: the type of mistake, So the person is intending to
Speaker 1: say the word brain, but the neural activity is decoded
Speaker 1: by the computer, and the computer says, oh, he's trying
Speaker 1: to say panda bear or whatever.
Speaker 3: Well it could be panda bear, it's more likely.
Speaker 1: So the the.
Speaker 2: Way that these systems work is well, one way they work.
Speaker 2: The way our systems work is we're decoding from neural
Speaker 2: activity to phonemes and then those phonemes get assembled into
Speaker 2: words using a dictionary.
Speaker 3: And a language model.
Speaker 2: And in fact, if you look at a dictionary, there's
Speaker 2: that phonetic spelling which most people don't use but if
Speaker 2: you want to figure out how to actually pronounce a word.
Speaker 3: You can look at that.
Speaker 2: So the types of mistakes it would more likely make
Speaker 2: would be similar sounding words.
Speaker 3: So if someone's trying to say brain, maybe they'd get barn.
Speaker 1: Yeah.
Speaker 2: And in some contexts you can understand, oh, I hurt
Speaker 2: my barn, I think you maybe you know you got
Speaker 2: an accident, you hurt your brain. But if there's enough
Speaker 2: of those, it just kind of breaks down. And the
Speaker 2: analogy I'd give is when you're typing on your smartphone.
Speaker 2: Most of us are a little bit clumsy. We make
Speaker 2: a lot of typos. The autocorrect can help up to
Speaker 2: a point, but there's this sort of steep cliff where
Speaker 2: if we're making too many typos, the autocrack so the
Speaker 2: language model cannot keep up, and all of a sudden
Speaker 2: you just get gibberish coming out.
Speaker 3: So that's kind of where things were.
Speaker 2: You could it wasn't gibberish, right, that's overstating it, but
Speaker 2: it was not there for communication day to day.
Speaker 1: So you worked with a man who is forty five
Speaker 1: years old, if I'm rememory correctly, and he had als
Speaker 1: and hadn't articulated in about five years. Is that right?
Speaker 2: Yet he was severely disarthuric, meaning most people couldn't understand him,
Speaker 2: and he volunteered for this brain gate to clinical trial
Speaker 2: that we are one of four sights of which meant
Speaker 2: that after a bunch of tests and imaging scans and
Speaker 2: other things, once we determined that it was a good
Speaker 2: fit and it was safe to move forward. He'd had
Speaker 2: this surgery where doctor Brandman, my collaudrator, put these four
Speaker 2: multi electro to rays into his speech motor cortex.
Speaker 3: We waited a couple of weeks.
Speaker 2: For everything to heal up, and then we went to
Speaker 2: his house where all of our equipment was already pre staged.
Speaker 2: We literally plugged him in. So there's this system is wired,
Speaker 2: so it's not wireless yet. And the way we started
Speaker 2: it was we needed what's called training data in the
Speaker 2: machine learning sense, so we needed the algorithms to see
Speaker 2: a bunch of examples of him trying to say words,
Speaker 2: and then what the neural activity looked like, and what
Speaker 2: this actually looked like in the room was picture a
Speaker 2: person in a wheelchair looking at a computer screen. We
Speaker 2: put up what seemed like random sentences. The text would appear,
Speaker 2: it would turn green, he would try to speak, and
Speaker 2: then he would stop. And we just did this for
Speaker 2: about thirty minutes. And one of the big questions at
Speaker 2: the time was how much data do you need to
Speaker 2: make this work? And the conventional wisdom would it was
Speaker 2: that it would take a lot of data. Previous studies
Speaker 2: had waited many, many weeks before they tried to decode
Speaker 2: what's someone was trying to say. The AI fields that
Speaker 2: we were borrowing tools from, for example, automated dictation when
Speaker 2: you talk to your smartphone, those models are trained with
Speaker 2: millions of hours so huge scrapes data sets to get
Speaker 2: them to be able to understand speech. But it turned
Speaker 2: out that because we had these electrodes in the part
Speaker 2: of part of the brain that's controlling speech movements, it
Speaker 2: has what's called a very high signal to noise ratio.
Speaker 2: There's a really clear signal about what movements the body's
Speaker 2: trying to make and thus what sounds is trying to produce.
Speaker 2: And so after just thirty minutes of him reading these sentences,
Speaker 2: we were looking at our little dashboard on the side
Speaker 2: on our computers and it was showing us what we
Speaker 2: call the word error rate. Or the phoneme error rate,
Speaker 2: so how many words or phonemes were being incorrectly decoded.
Speaker 2: And we saw that that was at the point where
Speaker 2: we thought, okay, this thing can actually work, and so
Speaker 2: we said, okay, now we're gonna do something very special.
Speaker 2: We're gonna kind of flipless, which so to speak, and
Speaker 2: now as you try to speak, you're going to see
Speaker 2: words hopefully appearing at the bottom of the screen. And
Speaker 2: we have a cool video of this, and so everyone's
Speaker 2: kind of holding their breath and very excited, and the
Speaker 2: prompt appeared, and he tries to speak, and the first
Speaker 2: two words appeared correctly, and actually, at that point everyone
Speaker 2: broke out in tears and laughter and clapping.
Speaker 3: We actually paused.
Speaker 2: For a few minutes and hugs, and his family was
Speaker 2: there to watch it, in a really amazing moment, and
Speaker 2: then we said, all right, let's get back to work,
Speaker 2: and we kept going. And on that day we had
Speaker 2: set a relatively modest goal. So we were using what's
Speaker 2: called a fifty word vocabulary, meaning the sentences he could
Speaker 2: say with it were restricted to fifty words, and you
Speaker 2: can still say a few things, and that's obviously not
Speaker 2: pragmatically useful, but that was to just to get going.
Speaker 2: We had less than a one percent error rate using
Speaker 2: this fifty word vocabulary, so almost every word was correct.
Speaker 3: That was huge.
Speaker 2: So we'd already established that, like some previous clinical throw participants,
Speaker 2: his brain was still active when he was trying to speak.
Speaker 2: So good, all right, that was the big one of
Speaker 2: the bigger risks. Were we getting good in neural signals
Speaker 2: from these electroder arrays? Yes, we were getting beautiful neural signals,
Speaker 2: in fact, some of the best I've seen in my career.
Speaker 2: And then did we need a ton of data? And
Speaker 2: the answer was no, we were getting enough that we
Speaker 2: could train these machine learning algorithms to map the neural
Speaker 2: activity patterns to the words okay.
Speaker 1: And for the listeners, I'm going to link the video
Speaker 1: which shows when the family started to cry and so
Speaker 1: I found that very moving. And so how long will
Speaker 1: these electrodes last? And you'd be able to get good
Speaker 1: signal out of this?
Speaker 2: For Casey that is a key question, and the answers
Speaker 2: we just don't know. So at this point he has
Speaker 2: had this for about two years. We just had a
Speaker 2: preprint a few months ago showing that out past six
Speaker 2: hundred and fifty days the system is still going strong.
Speaker 2: So this is huge because there was always some concern
Speaker 2: that maybe these electrodes would stop recording neurons after a
Speaker 2: few months or.
Speaker 1: And why it's because of scar tissue building up around
Speaker 1: the electrode.
Speaker 2: There are a lot of potential factors. So yeah, whenever
Speaker 2: you have a foreign body in the brain, the body
Speaker 2: in the brain does not want that thing, So scar
Speaker 2: tissue can form, can be at the microscale, just around
Speaker 2: the electrode tip, which makes it harder to record individual neurons.
Speaker 2: That sort of think of it like you're moving further
Speaker 2: away from someone you're listening to, or there's padding between
Speaker 2: you and them. It kind of it muffles the signal.
Speaker 2: It could be at a more of a macro scale
Speaker 2: where it can actually pull the electrodes out of the brain,
Speaker 2: and that's happened in some other studies.
Speaker 1: The way that your skin pushes a splinter out.
Speaker 2: Yeah, I think that's a good analogy. So that's on
Speaker 2: the biological response. Also, these are electrodes, so the materials
Speaker 2: can fail, The insulation can fail over time, the metal
Speaker 2: can get kind of chipped away or even away at
Speaker 2: the wires, could disconnect, and there's a lot of failure modes,
Speaker 2: but in this case, the records offar is really really encouraging.
Speaker 2: So two years out, it's working great. The accuracy has
Speaker 2: actually gotten better, and our preprint is now ninety nine
Speaker 2: percent accurate, both because we have more data and we've
Speaker 2: had more time to just improve the algorithms and keep
Speaker 2: trying new things. And he is now using this as
Speaker 2: his primary means of communication.
Speaker 1: And so a couple of things. One is, when you
Speaker 1: decode the neural activity, you could just print that as
Speaker 1: words on the screen, but you guys went a step further.
Speaker 2: Yeah, So in our first few months, what we did
Speaker 2: is called text to speech, So the words would appear
Speaker 2: as text on the screen initially, and then when a
Speaker 2: whole utter and so a sentence or it could be
Speaker 2: a whole paragraph, he would use his eyes to look
Speaker 2: at a button on the screen and basically there's a
Speaker 2: done button, and after he hits the done button, the
Speaker 2: computer will read out loud what he said, and we
Speaker 2: basically made a deep fake of his voice, so it
Speaker 2: sounds a lot like he did before he got als.
Speaker 2: It's not perfect, but it really does sound quite a
Speaker 2: lot like him. Technology has progressed a lot, even in
Speaker 2: the last couple of years. Most of the time people
Speaker 2: worry about all the ill uses of faking someone's voice,
Speaker 2: but this is maybe one of the few cases where
Speaker 2: it's actually a really wonderful thing.
Speaker 1: So you got his voice from videos when he was younger,
Speaker 1: before the als had set in.
Speaker 2: Yeah, we asked him and his family and they provided
Speaker 2: us a bunch of things. And actually he had done
Speaker 2: a podcast before, so we had really good material.
Speaker 1: So when he thinks of a sentence, the neural activities decoded,
Speaker 1: the sentence gets reconstructed, and then you turn it into
Speaker 1: his voice. Yes, now that's what you showed in twenty
Speaker 1: twenty four, and you just recently had a paper five
Speaker 1: months ago or so. Tell us about that.
Speaker 2: Yeah, So everything before, even though it could be said
Speaker 2: out loud, ultimately the informations in the form of text.
Speaker 2: And I think we can all appreciate that a lot
Speaker 2: gets lost just through texts.
Speaker 3: There's no intonation.
Speaker 2: You can't indicate that maybe you're being sarcastic. It's less expressive. Right,
Speaker 2: There's a lot of rich nuance that we all convey
Speaker 2: in our voice and through text that's lost, and the
Speaker 2: other problem is the latency or the immediacy. So if
Speaker 2: I was talking to you and I could only write,
Speaker 2: it would be very easy for you to accidentally interrupt me,
Speaker 2: or to just not for me not to be able
Speaker 2: to get a word in, because by the time I've
Speaker 2: finished a sentence and selected a bund to speak it
Speaker 2: out loud, maybe you've already moved on to the next topic.
Speaker 2: Maybe if there's other people in the room, they're talking right. So,
Speaker 2: for all of these reasons, we really wanted to do
Speaker 2: not what we call brain to text, but what we
Speaker 2: call brain to voice, and that means go immediately from
Speaker 2: neuroactivity to sound. This is a hard problem for a
Speaker 2: lot of reasons, one of which is it has to
Speaker 2: be in super fast. You want sound to happen within
Speaker 2: about thirty millisecond. That's kind of matching the natural latency
Speaker 2: of brain to moving the muscles to vibrating air that
Speaker 2: someone can hear. And so because of that, first of all,
Speaker 2: we had to decode these neuro signals very quickly. It
Speaker 2: limits the kind of algorithms we can use. We have
Speaker 2: less data to work with. Right, you can't look into
Speaker 2: the future, there's no autocorrect. You can't look at the
Speaker 2: entire sentence to figure out based on context, like, Oh,
Speaker 2: I reached down to pet the cot. No, you probably
Speaker 2: meant kat because you don't usually pet a cot. You
Speaker 2: can't do that if you're doing brain to voice. As
Speaker 2: soon as you try to say I, you need to
Speaker 2: have the sound eye reached. Right. It just has to
Speaker 2: flow constantly. But we were able to, through a bunch
Speaker 2: of complicated engineering work, get really far in there. And
Speaker 2: where the state of the art in that paper that
Speaker 2: you're referring to is is it is very immediate, So
Speaker 2: the latency is under thirty milliseconds, and it's mostly intelligible,
Speaker 2: but not consistently intelligible. So about fifty six percent of
Speaker 2: words could be understood by someone. It's a big step forward,
Speaker 2: but it's not good enough for daily use. Right. I
Speaker 2: already said earlier that we out of four words is
Speaker 2: not good enough, So you know, one out of two
Speaker 2: words is definitely not good enough.
Speaker 1: So when there's a mistake, what kind of mistake is it?
Speaker 1: Is it barn for brain and therefore sort of intelligible,
Speaker 1: or is it is it worse than that?
Speaker 2: Yeah, it tends to sound like slurry speech, or maybe
Speaker 2: like if someone's mumbling, so sometimes you can get the
Speaker 2: gist of it. The length tends to be the same
Speaker 2: because it's still capturing we call the envelope of speech.
Speaker 2: So if you're saying a short word or a long word,
Speaker 2: that comes through it very clearly, but maybe some of
Speaker 2: the phonemes are a little garbled, and so you can't
Speaker 2: tell exactly what's being said.
Speaker 1: Got it, Because each phoneme that the brain is encoding for,
Speaker 1: you're translating that right away. Thirty milli seconds later that's
Speaker 1: coming out of the speaker.
Speaker 2: Yeah, we just don't have enough signal to noise ratio.
Speaker 2: We don't have enough precisions. So it's like if you
Speaker 2: have a really bad digital camera, really grainy camera, and
Speaker 2: you're trying to parse the scene. You know, sometimes you
Speaker 2: can see what's going on, and other times you just
Speaker 2: can't quite make out. I know that is that a
Speaker 2: person or a ball?
Speaker 3: Is that?
Speaker 2: You know? What does that word say? If it's really grainy,
Speaker 2: you just can't see so well. And although we have
Speaker 2: two hundred and fifty six electros, which sounds like a lot,
Speaker 2: the brain has almost one hundred billion neurons. There's probably
Speaker 2: multiple billions that are involved in just speech and language.
Speaker 2: So in some ways as a miracle that works at all,
Speaker 2: that we're sampling from such a small number of neurons
Speaker 2: and able to reconstruct the sounds that the person's trying
Speaker 2: to make.
Speaker 1: And if I'm remembering in that paper, you also showed
Speaker 1: sort of short singing.
Speaker 2: Yeah, So we wanted to demonstrate that this approach could
Speaker 2: do more than just transmit the words, because we kind
Speaker 2: of already had that with brain to text. Now it
Speaker 2: could do it immediately, so that solves that interruption or
Speaker 2: being heard right away problem. But we wanted to provide
Speaker 2: a proof of concept that this could also be expressive,
Speaker 2: so we had a couple experiments that did that. In
Speaker 2: one of them, he was asked to say sentences as
Speaker 2: either a question or a statement. And in English, when
Speaker 2: we ask a question, can we increase the pitch at
Speaker 2: the end, So he was able to do that. We
Speaker 2: had him emphasize specific words, and you know, you use
Speaker 2: that to change the meaning of what you're saying. So
Speaker 2: this is classic from a different study, sentence that you
Speaker 2: can say in seven different ways, which is I never
Speaker 2: said she stole my money. Now I can say I
Speaker 2: never said she stole my money. I never said she
Speaker 2: stole my money. Right, I'm slightly changing the connotation depending
Speaker 2: on which word I'm stressing. And so we had a
Speaker 2: task where he said that sentence emphasizing all the different
Speaker 2: words and lo and behold.
Speaker 1: Yes.
Speaker 2: From the neuroactivity, we could identify which word he was stressing.
Speaker 2: And so then we had another task where we would
Speaker 2: give him a sentence and we would capitalize a word
Speaker 2: and he was supposed to emphasize that. And then the
Speaker 2: last one is what you were referring to is we
Speaker 2: call a simple singing task. So it was only three notes,
Speaker 2: but basically he could say whatever he wanted to say,
Speaker 2: but at three different pitch levels, so you could say,
Speaker 2: you know, like bah bah bah or like you know,
Speaker 2: la law da. So that task he was able to
Speaker 2: do quite well. He's not going to be singing in
Speaker 2: the opera yet, but it shows the path forward and
Speaker 2: where our lab and many others are working now is
Speaker 2: how do we build on this? So does that mean
Speaker 2: better algorithms? There's always new innovations in the artificial intelligence
Speaker 2: world and just neuroscience making sense of these signals.
Speaker 3: Does that mean putting more electrodes?
Speaker 1: In.
Speaker 2: Certainly that's of interest, and there's a lot of really
Speaker 2: exciting work happening in there. Does that mean maybe putting
Speaker 2: electrodes in additional parts of the brain, so kind of
Speaker 2: at a simplistic level, people think of left versus right
Speaker 2: brain as having some differences with maybe more of these
Speaker 2: what are called parlinguistic elements of voice encoded more on
Speaker 2: the right side of the brain. That's something we'd like
Speaker 2: to find out and we hope to in the future,
Speaker 2: or do we need to put it in other parts
Speaker 2: of the speech network.
Speaker 1: By the way, just to flesh that out for listeners.
Speaker 1: You know, on the left side of the brain, you've
Speaker 1: got a lot involved with language. When people get damage there,
Speaker 1: they let's say, lose the ability to articulate, to produce sentences,
Speaker 1: to understand census. But when people get damage in equivalent
Speaker 1: areas mirror images on the right side, they can get
Speaker 1: what's called a musia, which is the inability to understand
Speaker 1: music anymore. Because as you say, that's where intonation, the
Speaker 1: prosity of language seems to be encoded. So good, this
Speaker 1: is a good segue into the future, then, which is
Speaker 1: first of all, I'm curious what you think is the
Speaker 1: answer you just posed. Is it getting better electrodes, more electrodes,
Speaker 1: is it getting better algorithms? Is there a limitation in
Speaker 1: the signals and noise ratio? Where's the lowest hanging fruit
Speaker 1: for getting improvements? Here?
Speaker 3: Can I go with d all of the above? I
Speaker 3: think we do need all of these things.
Speaker 2: So already we are seeing with our data and this
Speaker 2: current participant that with the same electrodes, we are able
Speaker 2: to squeeze more information out with better algorithms and just
Speaker 2: better understanding what the brain is doing. And there's a
Speaker 2: lot going on there. It's not just the movements. We're
Speaker 2: seeing things like neural error signals. We're seeing prosody and
Speaker 2: intonation encoded. Right. All of these things are kind of
Speaker 2: mixed together in these brain signals we're measuring, and there's
Speaker 2: a lot of science that goes into disentangling them and
Speaker 2: figure out what they mean. What are you trying to
Speaker 2: pay attention to for given application. So that's all moving forward,
Speaker 2: and so we're just learning a ton about how the
Speaker 2: human brain produces speech because we didn't have this opportunity
Speaker 2: at this precision before. There's now only a handful of
Speaker 2: humans in the whole world that have had electrodes that
Speaker 2: measure individual neurons as they try to speak. So we're
Speaker 2: learning a lot, but certainly more electrodes is better, So
Speaker 2: in our trial as we move forward, we intend to
Speaker 2: put more electrodes in. There are now multiple companies that
Speaker 2: are building fully implanted intracortical electrodes, so similar type of
Speaker 2: electrodes that go right up to the neurons, but they
Speaker 2: all have a thousand or more electrodes or recording sites.
Speaker 2: So we're talking about at least a four x if
Speaker 2: not more improved in the density or the count of electrodes.
Speaker 2: And I think that's going to make everything work just
Speaker 2: so much better.
Speaker 1: And of course companies were working on making this wireless
Speaker 1: as well, Neurallink being I guess the first one to
Speaker 1: do it, but other companies moving that way as well,
Speaker 1: so that you could have something that's fully packaged and
Speaker 1: a person can just speak with no wires hanging out.
Speaker 3: Yeah, that is very important.
Speaker 2: So the wired systems we have now, they are what
Speaker 2: is available. They're good for research there in some ways simpler.
Speaker 2: They've been shown to be safe for quite a long time,
Speaker 2: but they're limiting right fully implanted is the way to go,
Speaker 2: and we can look at other medical devices. So there's
Speaker 2: these wild photos of pacemakers in the fifties and it
Speaker 2: was basically like a car battery on a cart with
Speaker 2: you some amplifiers and kind of primitive. They're not computers,
Speaker 2: they're electronics, and then there's a wire going to someone's chest.
Speaker 3: It kept them alive and it showed that this worked.
Speaker 2: But of course today millions and millions of people are
Speaker 2: walking around very healthy with pacemakers that are small and
Speaker 2: their packaged and titanium or other very inert safe materials.
Speaker 3: They have battery.
Speaker 2: Some of them now can be wirelessly recharged. So I
Speaker 2: think this is a well trodden path and we're going
Speaker 2: to absolutely see this with brain computer interfaces. They're going
Speaker 2: to be fully implanted, they're going to be wireless. Data
Speaker 2: is going to come out through radio or lasers or
Speaker 2: other means to get data out of the brain, and
Speaker 2: power is going to go in and it's going to
Speaker 2: be great. Great.
Speaker 1: Now, Okay, let me ask you this. A lot of
Speaker 1: people are very familiar with neuralink. They've heard about it.
Speaker 1: Even though as I mentioned, this idea of recording from
Speaker 1: brains has been happening for a very long time.
Speaker 2: Now.
Speaker 1: What neuralink is doing is implanting very tiny electrodes robotically,
Speaker 1: and it's fully implantable, and so that's part of why
Speaker 1: it's famous. But also part of why it's famous this
Speaker 1: is because it's Elon and there's this mystique about it,
Speaker 1: the sort of idea that everyone will someday get a neuralink.
Speaker 1: Now I have my doubts because it's an open head
Speaker 1: surgery still, even though it's with the robot. But let's
Speaker 1: look towards the future in terms of what use would
Speaker 1: it be to have a brain computer interface for somebody
Speaker 1: without a problem speaking or moving.
Speaker 2: Yeah, I don't think that application, the killer app so
Speaker 2: to speak, has been discovered yet.
Speaker 3: You know, there's times where I'm lying.
Speaker 2: In bed and I kind of wish i could send
Speaker 2: a text message without having to reach for my phone.
Speaker 2: But I'm not going to get a brain surgery to
Speaker 2: do that. I'm going to just reach for my phone.
Speaker 2: So what I think we're going to see is a
Speaker 2: widening of the medical applications. So I think there's gonna
Speaker 2: be many, many more medical needs that can be addressed
Speaker 2: with brain technology, whether stroke, things like sustaining memory in
Speaker 2: the longer term, or dealing with age related decline or
Speaker 2: even Alzheimer's. So there's going to be different types of
Speaker 2: BCIs for different problems. But in terms of fully implanted,
Speaker 2: kind of invasivec eyes for really healthy people, no one
Speaker 2: has yet shown a benefit that I think is worthwhile. Now,
Speaker 2: could I imagine it? Certainly one could imagine it. So,
Speaker 2: you know, if you could have a device in your brain,
Speaker 2: let's say it would allow you to feel more alert
Speaker 2: or to sleep less, right, so kind of modulating some
Speaker 2: circadian rhythms or energy level or attention. One could imagine
Speaker 2: that that kind of like a performance enhancing drug that
Speaker 2: could be done with a neurotechnology or neural interface. But
Speaker 2: no one's done that yet in a way that's compelling.
Speaker 2: People have talked about could it be kind of like
Speaker 2: a coprocessor for your brain, like you know, somehow you
Speaker 2: just know things. It's like having a smart AI assistant,
Speaker 2: but it's inside your mind and it's much more seamless.
Speaker 3: But that is a really long way away.
Speaker 2: I mean, we have we're struggling to get you know,
Speaker 2: crude vision in so people can can read a page. Now,
Speaker 2: I mean, that's amazing, that's like very state of the art.
Speaker 2: Or someone can slowly walk who has a spinal cord injury,
Speaker 2: or someone can talk but not as eloquently as before
Speaker 2: their als or before their stroke. So, given where we
Speaker 2: are now, I think we're quite a ways away from
Speaker 2: like beaming information in Oh.
Speaker 1: I totally agree with you on that. I do wonder
Speaker 1: twenty five years from now, let's say, right if you
Speaker 1: just took a short cut of said, okay, look, I
Speaker 1: want to listen to your covert speech things are not
Speaker 1: saying out loud, and then I want to plug the
Speaker 1: answer right back into your auditory cort text as though
Speaker 1: you're hearing it, and then you know, beam wirelessly to
Speaker 1: open AI or whatever exists in twenty five years from now. Yeah,
Speaker 1: the question is could you ask a question and hear
Speaker 1: the answer that way?
Speaker 2: My prediction is yes, I think that could be done.
Speaker 2: I mean also, I think that could be done the
Speaker 2: next five years. It just would still require a surgery
Speaker 2: to be done accurately, And so would anyone want it?
Speaker 2: Would we as a society choose to allow? It?
Speaker 3: Gets into debates of people's agency over their health.
Speaker 1: Are there moral or ethical questions about that.
Speaker 2: I think these are just general kind of medical and
Speaker 2: societal questions of do we allow people to take medical
Speaker 2: risks to get certain abilities that they otherwise wouldn't have.
Speaker 1: One of the issues is about brain privacy, right, the
Speaker 1: question of let's say I'm doing something that's recording my
Speaker 1: covert thoughts, by which I mean, you know something that
Speaker 1: I'm thinking, but I haven't actually pushed it out to
Speaker 1: my motor cortex to say it yet. Who's the company
Speaker 1: who has access to that? Do I want anybody accessing that?
Speaker 2: I think that's yeah, that's a real concern. We're not
Speaker 2: there yet, so to be clear, there's no BCI that
Speaker 2: can decode covert thought yet exactly.
Speaker 1: I'm talking twenty five years from Yeah. Yeah, I mean,
Speaker 1: this is one of the conundrums about where this is heading.
Speaker 2: Well, we're already dealing with inklings of that. So, for example,
Speaker 2: in our system, because our participant is using this for
Speaker 2: his day to day life. For example, one thing that
Speaker 2: we implement was a privacy mode where if he toggles
Speaker 2: a button, it no longer saves that data. This is
Speaker 2: a academic clinical trial. In general, we're really loath to
Speaker 2: give up any data I mean, it's so precious and
Speaker 2: then these people are making these commitments to science, but
Speaker 2: we also want to be respectful that he might need
Speaker 2: to have a really private conversation and we don't want
Speaker 2: to even have any ability to access that. So that's
Speaker 2: already something we're dealing with in the context of a
Speaker 2: medical trial from an academic medical center. I think this
Speaker 2: is a very high trust scenario. Of course, when you
Speaker 2: have companies that are building these, we're going to want
Speaker 2: to think about we have what rights do in that
Speaker 2: case patients or customers have to the data? Can the
Speaker 2: data be used to improve the algorithms? Who owns the
Speaker 2: benefit of that? What happens if a government subpoena?
Speaker 3: Is it? Right? Now, we have.
Speaker 2: This speech PCI for people with vocal tracked paralysis, meaning
Speaker 2: that they know exactly what they're trying to say. The
Speaker 2: words are clearly formed in their mind. They are trying
Speaker 2: to speak it. Those commands are not reaching the muscles. Okay,
Speaker 2: So we've shown that there is a very compelling therapy there.
Speaker 2: Industry is going to come in and kind of productize it.
Speaker 2: I think this is going to turn into medical device
Speaker 2: in the next five years. There is a much larger
Speaker 2: patient population though with aphasia due to stroke, So there
Speaker 2: the problem is one step further upstream, meaning.
Speaker 1: I mean they can't speak language by the way face.
Speaker 3: Yes, well, there's different types.
Speaker 2: So sometimes within aphasia that means they can't understand language,
Speaker 2: but with expressive aphasia that means in many patients cases
Speaker 2: they want to communicate, they really know what they're trying
Speaker 2: to say in sort of in a meaning sense, but
Speaker 2: they can't find the right words for it. It's almost like,
Speaker 2: you know, sometimes I can't remember a word, but that's
Speaker 2: rare and I can usually remember it or explain in
Speaker 2: other words. But if I couldn't remember most of the words,
Speaker 2: that would be really frustrating and debilitating.
Speaker 3: And there's millions of.
Speaker 2: People that have strokes and partially recover but never fully recover.
Speaker 2: They have a language disorder. Many of them have perfectly
Speaker 2: normal intelligence and their personalities preserved and kind of everything
Speaker 2: else is there, but they just can't form words.
Speaker 3: Can we help them?
Speaker 2: And this is something that our lab and many others
Speaker 2: are starting to think about. The idea is, can we
Speaker 2: basically do this thing that we've done with a speech BCI,
Speaker 2: but now make a language BCI can we put electrodes
Speaker 2: somewhere in the language network and that is a lot
Speaker 2: of the brain that's both a good and a bad thing.
Speaker 3: Could we decode the meaning and this.
Speaker 2: Is kind of getting close to this idea of a thought,
Speaker 2: which is not a very well defined term, but could
Speaker 2: we decode the semantic meaning of what they're trying to
Speaker 2: communicate and have let's say, a tablet in front of
Speaker 2: them print out a sentence or speak a sentence where
Speaker 2: they're saying, I'm happy to see you, or could you
Speaker 2: hand me some water? Or my nose itches or I'm
Speaker 2: not feeling well well right, that thought, that communication intent
Speaker 2: is still in there for many of these patients. We're
Speaker 2: trying to develop a medical technology to help them, but
Speaker 2: that starts getting pretty close to sounding like mind reading.
Speaker 2: And so yeah, I think as an ethical question this
Speaker 2: will potentially become relevant in the coming years if this
Speaker 2: medical project succeeds.
Speaker 1: It's interesting because we mean different things by mind reading.
Speaker 1: There are all these different levels of it, so even
Speaker 1: what somebody is trying to say often masks what they're thinking.
Speaker 1: I'm trying to remember this quotation from the poet Oliver Goldsmith,
Speaker 1: who said something like I think the real purpose of
Speaker 1: language is not to communicate intent but to hide it.
Speaker 1: So anyway, so if somebody says, hey, you know, I'm
Speaker 1: happy to see you, or I you know, whatever the
Speaker 1: thing is they're saying, it may or may not be
Speaker 1: what their thoughts actually are. Is that's what their language is.
Speaker 2: Yeah, so we're still talking. I'm still talking about decoding
Speaker 2: communication and tent and that's sort of I think we
Speaker 2: find it a little bit reassuring because it's an active process.
Speaker 2: It's not like right now that we're nowhere close no
Speaker 2: one even has an inkling of how to make a
Speaker 2: device that can like read everything you know. You know,
Speaker 2: you're not actively thinking about it, but it just knows
Speaker 2: your whole childhood and all your deepest secrets and you
Speaker 2: know what you think about everyone around you. That I
Speaker 2: would not even know how to start to do that,
Speaker 2: But for thinking what you're thinking actively or what you're
Speaker 2: trying to communicate, that seems plausible. And there's some studies
Speaker 2: using imaging that kind of you know, can do above
Speaker 2: chance dey coding which someone's trying to communicate. We have
Speaker 2: some preliminary data others do as well, So I think
Speaker 2: that might happen.
Speaker 1: So let me ask you a few things. When will
Speaker 1: paralysis be solved?
Speaker 2: I think there will be approved BCIs for paralysis in
Speaker 2: about five years. That doesn't mean they'll be available everywhere.
Speaker 2: They might be only available in certain markets. Maybe only
Speaker 2: a few hospitals will initially be providing them, but that
Speaker 2: will grow rapidly.
Speaker 3: Will it mean.
Speaker 2: Paralysis is cured? I think that's too strong a term.
Speaker 2: Maybe that means you can walk slowly, you can move
Speaker 2: your arm, but you maybe can't tie your shoelace.
Speaker 3: Initially.
Speaker 2: You can move a computer cursor really well, but that's
Speaker 2: not the same thing as playing the piano.
Speaker 3: So I think the capabilities will keep getting better.
Speaker 1: And with als and dysarthria where someone can't articulate, well,
Speaker 1: what are we looking at?
Speaker 3: Your prediction, it's actually the same.
Speaker 2: I think that the speech bring computer interfaces are going
Speaker 2: to move very fast. I think that and cursor will
Speaker 2: probably be one of the first approved systems, even though
Speaker 2: people have been trying to move robot arms or paralyzed limbs.
Speaker 3: For much longer.
Speaker 2: So if you're trying to decode what someone's trying to say,
Speaker 2: or decode them trying to move a computer cursor or
Speaker 2: right of the keyboard the thing that they're trying to
Speaker 2: control as a computer, and those are ubiquitous, they're everywhere, they're.
Speaker 3: Cheap, they work really well.
Speaker 2: If you're trying to decode what someone's trying to move
Speaker 2: with their arm, you either need to move a robot arm.
Speaker 2: Robot arms are hard, they break often, they're not as
Speaker 2: precise as people are.
Speaker 3: You know, where does it go? Does it go on
Speaker 3: your wheelchair?
Speaker 2: Is it there with you in the shower, if it's
Speaker 2: mounted on like if you have an amputation, is.
Speaker 3: It mounted on your stump or on your shoulder? That
Speaker 3: is hard. There's a lot of challenges there.
Speaker 2: So kind of the readout part for speech is very
Speaker 2: hard because it's very fast. There's a lot of information
Speaker 2: per second. But once you have that solved, making use
Speaker 2: of it is actually really easy. You just send texts
Speaker 2: to their computer or their phone, or you have their
Speaker 2: tablet talk mix sound and that's something you can carry
Speaker 2: with you all the time and it's really reliable. So
Speaker 2: because for all those reasons, I think we're going to
Speaker 2: have speech and also computer use BCIs hopefully starting to
Speaker 2: hit the market in the next five years.
Speaker 1: Great and when you think about fifty years from now,
Speaker 1: when you think about as you're retiring and you look
Speaker 1: around the field, what do you say.
Speaker 2: I think BCIs will be well, the term may not
Speaker 2: even mean anything because it's going to be so wide.
Speaker 2: I think many of the diseases that we struggle with
Speaker 2: today are going to be treated with some sort of
Speaker 2: technology inside the head or interacting with the head.
Speaker 3: Maybe it's somehow not.
Speaker 2: Invasive, whether that's paralysis, which is going to be I
Speaker 2: think much faster than that. Or will we have systems
Speaker 2: that help us regulate our mood, Will they treat psychiatric issues,
Speaker 2: Will they perhaps reconnect parts of the brain that have
Speaker 2: been disconnected due to aging or damage, or injury or stroke.
Speaker 2: If we're talking about fifty years, a lot can happen
Speaker 2: in fifty years, right, I mean technology is moving very quickly.
Speaker 2: The interfaces will get better. So instead of talking about
Speaker 2: instead of me being right now excited about recording from
Speaker 2: a thousand neurons, in fifty years, could we be interfacing
Speaker 2: with one hundred thousand or a million neurons.
Speaker 3: I think that's really plausible.
Speaker 2: Through tiny nano wires or biohybrids or focused beams that
Speaker 2: are non invasive.
Speaker 3: A lot can happen.
Speaker 2: In fifty years, our neuroscience, I think, will be a
Speaker 2: lot more advanced.
Speaker 3: We will not be limited to right now.
Speaker 2: We mostly understand the peripheres, We understand movement, We understand
Speaker 2: the senses really well because it's really easy to experimentally
Speaker 2: manipulate those.
Speaker 3: We as soon as you get.
Speaker 2: Into the kind of the inside the center cognition intelligence,
Speaker 2: how do we problem solve creativity? We don't understand that
Speaker 2: really well, but I think at fifty years we will.
Speaker 2: And part of that is because as we make these
Speaker 2: medical systems, we will have access to human brains. So
Speaker 2: think of this as a flywheel. So let's say someone
Speaker 2: has a few thousand electrodes because they have a stroke
Speaker 2: and they want to communicate. Maybe these are spread across
Speaker 2: several different brain areas because you get different pieces of it.
Speaker 2: Or maybe you get the prosody in one area primarily
Speaker 2: and you get what they're trying to say in the
Speaker 2: motor cortex. But you get some planning benefit and language
Speaker 2: benefit from the temporal lobe. Okay, so let's say you
Speaker 2: have four or five six areas that you're recording from. Well,
Speaker 2: now you have a wealth of information that you can
Speaker 2: use for other things. So some of these patients are
Speaker 2: going to develop dementia over time, or they might be depressed,
Speaker 2: or they might have OCD, And instead of having to
Speaker 2: do a new brain implant with all the new risks
Speaker 2: of that, you can just look at the data you're
Speaker 2: already collecting and try to relate that to their mood
Speaker 2: or what are they looking at? What are they trying
Speaker 2: to remember? Oh, they're trying to remember where they put
Speaker 2: their keys. Hey, Actually, because we have electrodes in the
Speaker 2: temporal lobe, it's close to the hippocampus, it's cortex, it's
Speaker 2: part of the memory system as well, everything's kind of
Speaker 2: spread out. Well, maybe now we're seeing some neural correlative
Speaker 2: that memory process. Maybe we can even ask if they're
Speaker 2: willing to do another clinical trail where we stimulate and
Speaker 2: try to boost that memory, try to kind of help
Speaker 2: nudget be remembered correctly. I think when we're talking about
Speaker 2: fifty years that's going to happen. And so through this
Speaker 2: process we're going to learn a lot more about how
Speaker 2: the human mind works and thus how to fix it.
Speaker 1: That was my interview with Sergei Stavisky, a neuroscientist that
Speaker 1: you see Davis and co director of the Neuroprosthetics Lab.
Speaker 1: We talked about what BCIs can do, what they might
Speaker 1: do soon, and how will navigate the human questions that
Speaker 1: they raise. What we talked about today was how a
Speaker 1: person's intention can find its way back into the world
Speaker 1: when bodies have lost function. Brain computer interfaces are opening
Speaker 1: a new lane right now. These technologies are crude in
Speaker 1: some ways, but they're getting better fast. Each year they
Speaker 1: get a little faster and more expressive. So this is
Speaker 1: how BCIs can restore autonomy and intimacy and dignity. And
Speaker 1: when it's done right, you don't see the technology at all,
Speaker 1: You just see the person again. So here's how I
Speaker 1: see it. In the next five years, BCIs are going
Speaker 1: to start looking less like research product and more like appliances.
Speaker 1: We're going to have fully implantable systems for communication. In
Speaker 1: other words, at some point in the future, we'll be
Speaker 1: looking at a small surgery, a wireless puck that goes in,
Speaker 1: and a setup that takes minutes instead of hours. You'll
Speaker 1: turn on your speech BCI or your BCI that controls
Speaker 1: a computer cursor, and the key thing will be reliability,
Speaker 1: these decoders will hold steady through years, and also identity.
Speaker 1: The voice is going to sound just like you, your cadence,
Speaker 1: your prosity, your humor at the end of a sentence.
Speaker 1: Maybe rehab teams will have a neural therapist who tunes
Speaker 1: your decoder the way that an audiologist tunes a cochlear implant.
Speaker 1: And if I had a guess, this will all become
Speaker 1: normal rather than newsworthy. Now around ten years out, we'll
Speaker 1: get good feedback of signals moving in both directions. So
Speaker 1: a person who is suffering from paralysis will can control
Speaker 1: her hand through say electrodes in her motor cortex, and
Speaker 1: you have another interface, say electrodes in her somatosentury cortex,
Speaker 1: that's inputting information so that she feels a push back
Speaker 1: with electrically evoked touch, and that loop makes the movements
Speaker 1: smooth and automatic. This is all going to continue getting
Speaker 1: smaller and better. Soon will have thin film options to
Speaker 1: reduce the surgical footprints. The decoders will auto calibrate, they'll
Speaker 1: borrow tricks from language models, and they'll figure out how
Speaker 1: to adjust to your neural dynamics when you're tired or
Speaker 1: stressed or boosted on caffeine. Eventually your BCI will speak
Speaker 1: the same API language as your phone and home devices,
Speaker 1: so that you can text or adjust the lights or
Speaker 1: turn on appliances without moving a limb or making a sound.
Speaker 1: And crucially, the privacy architecture is to evolve like inner
Speaker 1: speech stays off limits by default, and your neural stream
Speaker 1: lives behind consent gates. We'll need to have a kind
Speaker 1: of airplane mode for the mind. Okay, And if I
Speaker 1: were going to speculate on a quarter century from now,
Speaker 1: I'm thinking that what we're looking at is very high
Speaker 1: bandwidth arrays. These might be micro needles or flexible meshes,
Speaker 1: or electrode stents living on the inside of the blood vessels.
Speaker 1: Whatever the technology, it's going to give us coverage that
Speaker 1: approaches the dexterousness of natural hand control. Imagine playing a
Speaker 1: piano with one of these. Imagine prosthetics and exoskeletons that
Speaker 1: feel less like machines and more like natural limbs because
Speaker 1: the brain sees and feels them just as part of
Speaker 1: the body. And for communication, we'll get the full richness
Speaker 1: of natural speech. Just imagine talking with a person with
Speaker 1: a BCI and you hear the emphasis of ups and
Speaker 1: downs of speech, and their laughter and their little half
Speaker 1: swallowed syllables when people are negotiating, turn taking and singing.
Speaker 1: And soon enough, I think, in our lifetimes for sure,
Speaker 1: the science fiction edge of this all is going to
Speaker 1: start to glow. So imagine a scene like this when
Speaker 1: you step onto a train maybe thirty five years from now.
Speaker 1: People are sitting there. It's crowded, and they're all speaking
Speaker 1: private messages to their friends who are somewhere else. There's
Speaker 1: no sound, the train is quiet. Each person's decoder is
Speaker 1: locked onto their attempted speech, not their idle thoughts, and
Speaker 1: every message is signed with a cryptographic water mark that
Speaker 1: proves it came from that person's neural key. So you're
Speaker 1: looking at a silent train car, but it's filled with conversations.
Speaker 1: Or just imagine something simpler. Here's a carpenter who lost
Speaker 1: his hand, but he's back at work with a prosthetic
Speaker 1: hand that streams touch information into the brain pressure and temperature.
Speaker 1: But also he can feel the details of the grain.
Speaker 1: He can tell the difference between pine and oak just
Speaker 1: by running his sensory packed robotic fingers over it. And
Speaker 1: the key is that He doesn't think about the device
Speaker 1: at all. He just builds, just like you use the
Speaker 1: high bandwidth sensory devices on your own hand, and you
Speaker 1: rarely stop to think about it. Eventually, there'll be a
Speaker 1: lot of legislation in place, because there are going to
Speaker 1: be hard lines we choose as a society not to cross.
Speaker 1: Not all thoughts should be digitized. We're going to need
Speaker 1: neuro rights with teeth, will need on device processing that
Speaker 1: keeps data local where maybe you have your own descendant
Speaker 1: of modern day LLMS living with you in your brain.
Speaker 1: Whatever the case, will presumably keep asking philosophical questions about
Speaker 1: our brains and ourselves, but we'll get to do it
Speaker 1: with better and better tools than we have now. And
Speaker 1: I think what this means is that we have more
Speaker 1: in common with our ancestors of a thousand years ago
Speaker 1: than we do with our descendants a century from now.
Speaker 1: Go to Eagleman dot com slash podcast for more information
Speaker 1: and to find further reading. Send me an email at
Speaker 1: podcasts at eagleman dot com with questions or discussion and
Speaker 1: check out Subscribe to Inner Cosmos on YouTube for videos
Speaker 1: of each episode and to leave comments. Until next time.
Speaker 1: I'm David eagleman, and this is inner cosmos.