WEBVTT - EP127 "What happens when we marry brains to machines?" with Sergey Stavisky 0:00:05.200 --> 0:00:10.000 What is a brain computer interface? How far along is 0:00:10.160 --> 0:00:14.120 this field? Can we evesdrop on the brain so that 0:00:14.160 --> 0:00:18.040 a person who has lost the ability to move can 0:00:18.200 --> 0:00:22.120 use their brain to control a computer cursor or a 0:00:22.239 --> 0:00:25.880 robotic arm. Can someone who has lost the ability to 0:00:26.040 --> 0:00:30.760 speak send brain signals to a decoder and hear their 0:00:30.880 --> 0:00:36.360 voice again? Can we restore autonomy and dignity and eventually 0:00:36.800 --> 0:00:41.720 do so so seamlessly that the technology disappears and the 0:00:41.760 --> 0:00:46.920 person reappears In the future, where will the ethical boundaries 0:00:46.960 --> 0:00:52.320 be between restoring function and spying on private thought? And 0:00:52.400 --> 0:00:57.400 who owns the stream of neural data that represents you? 0:01:00.640 --> 0:01:03.880 Welcome to Inner Cosmos with me David Eagleman. I'm a 0:01:03.920 --> 0:01:07.720 neuroscientist and author at Stanford and in these episodes we 0:01:07.840 --> 0:01:12.560 sail deeply into our three pound universe to understand why 0:01:12.680 --> 0:01:31.839 and how our lives look the way they do. This week, 0:01:31.840 --> 0:01:36.760 we're talking about technology for reading the brain. Now. One 0:01:36.760 --> 0:01:40.480 thing that I find fascinating is that ancient cultures didn't 0:01:40.520 --> 0:01:44.160 care at all about the brain. They generally would just 0:01:44.680 --> 0:01:48.720 throw it out at autopsy, and it's understandable why it 0:01:48.880 --> 0:01:53.360 just looks and feels like a huge, squishy walnut. If 0:01:53.360 --> 0:01:57.200 you could sit and stare at a brain in action, 0:01:57.880 --> 0:02:03.200 you wouldn't see anything happening. So it's taken centuries and 0:02:03.240 --> 0:02:06.160 a lot of technology to realize that, in fact, the 0:02:06.200 --> 0:02:11.680 brain is alive with lots of tiny cells, microscopically tiny, 0:02:12.040 --> 0:02:15.960 and these cells are transmitting electrical signals tens or one 0:02:16.040 --> 0:02:18.920 hundred times every second for each cell. And you have 0:02:19.080 --> 0:02:23.880 eighty six billion of these cells. So this big, squishy 0:02:23.919 --> 0:02:27.799 walnut is one of the busiest things on the planet. 0:02:28.680 --> 0:02:32.560 But because it is so fragile, Mother Nature surrounds the 0:02:32.600 --> 0:02:36.839 brain with an armored bunker plating the skull, and that 0:02:36.919 --> 0:02:40.080 provides a huge challenge if you want to go in 0:02:40.120 --> 0:02:44.600 there and eavesdrop on what the cells are doing. Now, 0:02:44.639 --> 0:02:47.400 why would you want to spy on these cells? Well, 0:02:47.840 --> 0:02:52.800 imagine if your thoughts could exit the skull as easily 0:02:53.160 --> 0:02:57.440 as words leave your mouth. Now, there's a sense in 0:02:57.480 --> 0:03:00.840 which we always do this. We use keyboards, touch screens, 0:03:00.919 --> 0:03:05.079 and voice assistants, but all of those are detours. They 0:03:05.160 --> 0:03:09.160 force the brain to root its intentions through muscle, and 0:03:09.240 --> 0:03:13.520 that's fine if your muscles work. The problem is that 0:03:13.720 --> 0:03:17.680 lots of people, millions of our neighbors and friends don't 0:03:17.680 --> 0:03:21.000 have a way to get the information out of their 0:03:21.040 --> 0:03:24.959 brain because something about the brain or the brain's pathways 0:03:25.080 --> 0:03:28.639 or the muscles are not working, and therefore their brain 0:03:28.800 --> 0:03:31.640 knows what they want to do or say, but there's 0:03:31.639 --> 0:03:35.280 no way to get that information out. And this is 0:03:35.320 --> 0:03:39.800 where the idea of a brain computer interface comes in. 0:03:40.160 --> 0:03:44.520 What you'll hear referred to as a BCEI brain computer interface. 0:03:45.000 --> 0:03:48.360 The idea of a BCI is to listen directly to 0:03:48.440 --> 0:03:52.320 the neural patterns that mean move or speak or select, 0:03:52.680 --> 0:03:56.800 and then you use some device to translate those patterns 0:03:56.880 --> 0:04:01.840 directly into activation in the outside world. Now, as I said, 0:04:01.840 --> 0:04:04.160 this is a huge deal for all the people for 0:04:04.200 --> 0:04:09.480 whom the path from intention to movement has been interrupted 0:04:09.520 --> 0:04:12.920 by disease or injury. The intent is still alive and 0:04:12.960 --> 0:04:16.800 well in the cortex, and BCIs are the bridge back. 0:04:17.279 --> 0:04:22.239 They turn silent plans into text or voice or cursor 0:04:22.320 --> 0:04:27.120 control or reaching and grasping. But the story will, at 0:04:27.200 --> 0:04:30.839 least in theory, reach beyond the medical because once you 0:04:30.920 --> 0:04:34.240 can read out the programs for say this word or 0:04:34.320 --> 0:04:38.400 press that key, now you've built a communication channel between 0:04:38.520 --> 0:04:43.919 biological tissue and silicon, and that opens new forms of 0:04:44.080 --> 0:04:49.760 interaction that our species has barely begun to imagine. Now, 0:04:49.839 --> 0:04:51.960 let me not get ahead of myself yet, because as 0:04:52.000 --> 0:04:54.599 we're going to see today, we are still at the 0:04:54.760 --> 0:04:58.240 earliest stages of this technology. But this is what we're 0:04:58.279 --> 0:05:01.279 going to talk about at the end. Now, you can 0:05:01.360 --> 0:05:05.160 build bceiyes in lots of flavors. Some rest on the scalp, 0:05:05.480 --> 0:05:08.840 Others sit on the surface of the brain. Others poke 0:05:09.160 --> 0:05:12.440 tiny wires called electrodes into the surface of the brain 0:05:12.880 --> 0:05:15.480 or even down deep into the brain for some purposes. 0:05:16.240 --> 0:05:20.520 Some of these BCIs only read the electrical activity. Others 0:05:20.560 --> 0:05:25.120 will also write with electrical patterns that the brain experiences 0:05:25.200 --> 0:05:28.200 as touch or sound or sight. In every case, the 0:05:28.240 --> 0:05:32.360 principle is the same. Brains issue commands, and they're very 0:05:32.520 --> 0:05:36.760 fast and complex internal language of electrical spikes. This is 0:05:36.800 --> 0:05:41.000 a language that we haven't nearly decoded yet, but machines 0:05:41.120 --> 0:05:44.159 can learn to translate that language through a lot of 0:05:44.360 --> 0:05:48.839 trial and error. Huge populations of neurons are playing some 0:05:49.240 --> 0:05:54.000 symphony piece, and these decoders learn how to hear the 0:05:54.080 --> 0:05:58.240 music and root the commands to a cursor or a 0:05:58.279 --> 0:06:02.040 speaker or a robotic arm or whatever. Now. The issue 0:06:02.080 --> 0:06:04.120 is that when we talk about it, it all seems 0:06:04.200 --> 0:06:08.240 very straightforward and easy, but actually getting in there and 0:06:08.320 --> 0:06:12.320 getting technology that can record from these microscopic little cells, 0:06:12.560 --> 0:06:16.520 having these little changes in their electrical potential of tens 0:06:16.520 --> 0:06:20.640 of millivolts, and making a system that lasts, and then 0:06:20.720 --> 0:06:23.960 putting all the data together to understand what this very 0:06:24.120 --> 0:06:28.320 tiny sampling of neurons, maybe a few hundred out of 0:06:28.640 --> 0:06:32.200 hundreds of billions of neurons. It turns out this is 0:06:32.240 --> 0:06:37.640 a massive engineering challenge and there are a million practical questions. 0:06:38.000 --> 0:06:41.640 How reliable are these systems outside the lab? Can they 0:06:41.680 --> 0:06:46.480 survive infection and signal drift? What about battery life? What's 0:06:46.560 --> 0:06:50.559 the surgical risk? When does insurance cover these? So there's 0:06:50.800 --> 0:06:55.520 a huge gap between a beautiful proof of principle and 0:06:55.800 --> 0:07:00.440 a device that changes lives every day, and crossing that 0:07:00.560 --> 0:07:03.440 gap is the real work of the field right now. 0:07:04.160 --> 0:07:06.320 Now there's also a second issue. As soon as we 0:07:06.360 --> 0:07:10.840 start talking about reading the brain, the questions start to surface, 0:07:11.000 --> 0:07:14.880 what exactly are we reading? Is it intended movements? That's 0:07:14.920 --> 0:07:18.360 one thing is that inner speech? Is it where you 0:07:18.520 --> 0:07:22.120 place your attention? You can imagine situations in which there 0:07:22.160 --> 0:07:25.760 are things that you don't want everyone knowing. We're used 0:07:25.760 --> 0:07:29.640 to the skull having some sort of sanctity. So where 0:07:29.680 --> 0:07:36.080 will the ethical boundaries be between restoring function and evesdropping 0:07:36.120 --> 0:07:40.040 on private thought? Who's going to own the stream of 0:07:40.120 --> 0:07:44.320 data that is literally you? How do we guarantee consent 0:07:44.440 --> 0:07:48.680 and security and dignity when the interface is not on 0:07:48.720 --> 0:07:52.280 your desk but inside your skull. So, even in the 0:07:52.280 --> 0:07:54.800 face of all the tough questions coming down the pike, 0:07:55.320 --> 0:07:59.840 it's hard not to feel awe at what's already possible. 0:07:59.840 --> 0:08:04.040 Who have been locked inside their bodies are communicating again. 0:08:04.360 --> 0:08:07.080 They're talking with their loved ones for the first time 0:08:07.160 --> 0:08:12.360 in years. And the technology keeps improving every month, smarter algorithms, 0:08:12.440 --> 0:08:17.960 better sensors, cleaner signals, and crucially designs that move from 0:08:17.960 --> 0:08:20.960 the hospital to the home. So today I want to 0:08:21.000 --> 0:08:23.600 explore what that looks like and where we are in 0:08:23.600 --> 0:08:26.480 the process and where things are going. So I sat 0:08:26.520 --> 0:08:29.800 down with my colleague Sergei Stavisky. Sergei is at the 0:08:29.840 --> 0:08:34.280 UC Davis Neuroprosthetics Lab, which he co directs with neurosurgeon 0:08:34.480 --> 0:08:38.720 David Brandman. With their collaborators, they work on BCIs that 0:08:38.840 --> 0:08:43.400 restore communication and they're pushing towards systems that are fast 0:08:43.520 --> 0:08:47.600 and expressive and practical for everyday life. So here's my 0:08:47.679 --> 0:08:50.280 interview with Sergei Staviski. 0:08:53.920 --> 0:08:58.120 A brain computer interface is a device that interacts between 0:08:58.200 --> 0:09:01.000 technology and a brains. You have the brain, you have 0:09:01.240 --> 0:09:04.200 some way of getting information in or out, and you 0:09:04.280 --> 0:09:07.560 have some computation that's happening. And that computation it could 0:09:07.559 --> 0:09:09.240 be happening inside the body, so it could be a 0:09:09.320 --> 0:09:12.240 chip that does everything in the brain, or it could 0:09:12.240 --> 0:09:15.800 be sending that information to a laptop next to the person, 0:09:15.880 --> 0:09:18.079 or even to the cloud for more computation. 0:09:18.480 --> 0:09:21.080 Now, one of your interests is that you know, over 0:09:21.120 --> 0:09:23.440 a century ago people figured out you could dunk an 0:09:23.480 --> 0:09:27.480 electrode into the brain the thin wire and because cells 0:09:27.480 --> 0:09:33.320 are communicating with little electrical signals, you're you can eavesdrop 0:09:33.440 --> 0:09:36.440 on that and you can also stimulate the cell to 0:09:36.480 --> 0:09:39.800 do whatever. So tell us about the history of this, 0:09:41.080 --> 0:09:43.880 how people have thought about, let's eavesdrop on the brain 0:09:43.960 --> 0:09:45.240 and turn that into something. 0:09:45.480 --> 0:09:49.440 So starting in the sixties and seventies and eighties, especially 0:09:49.480 --> 0:09:52.800 working in animal models, people realized, yeah, you can put 0:09:52.800 --> 0:09:55.720 electrodes into the brain, and you can get up close 0:09:55.760 --> 0:09:58.079 next to an individual brain cell a neuron, and when 0:09:58.080 --> 0:10:01.199 that neuron's firing, it's genera a big electric field, a 0:10:01.240 --> 0:10:03.520 tiny electric field, but big relative to the electrode right 0:10:03.559 --> 0:10:05.160 next to it, And so. 0:10:05.080 --> 0:10:06.520 We know that that neuron is firing. 0:10:06.559 --> 0:10:09.679 And then there was a whole decades of systems neuroscience 0:10:09.679 --> 0:10:13.240 which was relating those patterns of activity to what typically 0:10:13.280 --> 0:10:16.560 the animal was doing. So a classic example from the 0:10:16.559 --> 0:10:20.240 eighties would be a monkey is moving his arm up 0:10:20.320 --> 0:10:22.920 or down, or left or right, and you can see 0:10:22.920 --> 0:10:26.240 that maybe a neuron fires more when the arm is 0:10:26.280 --> 0:10:28.360 moving to the left, and say, okay, that neuron has 0:10:28.360 --> 0:10:30.960 a left or preferred direction. We're starting to build some 0:10:31.400 --> 0:10:34.800 mental map of how that brain activity relates to movements. 0:10:34.800 --> 0:10:37.240 Of course, it's much more complicated, and the whole field 0:10:37.240 --> 0:10:40.679 of neuroscience is trying to understand how individual neurons and 0:10:40.760 --> 0:10:44.920 hundreds of neurons and whole large assemblies of neurons generate behavior. 0:10:45.320 --> 0:10:50.160 Starting around the two thousands, the field had felt that 0:10:50.240 --> 0:10:53.280 we had enough of a rudimentary understanding of how movement 0:10:53.520 --> 0:10:57.200 is encoded in the brain that this could be used 0:10:57.360 --> 0:10:58.719 for a medical application. 0:10:59.520 --> 0:11:01.240 And kind of in my world. 0:11:01.040 --> 0:11:04.440 That's been focused on restoring movement to people with paralysis. 0:11:04.480 --> 0:11:05.400 So in two. 0:11:05.280 --> 0:11:07.600 Thousand and four it was a big landmark event that 0:11:07.760 --> 0:11:10.319 was when the original brain Gate trial. So this was 0:11:10.400 --> 0:11:13.720 led by John Donahue in Lee Hagberg at Brown University 0:11:13.720 --> 0:11:16.240 in Masteronal Hospital. They put what was called a multi 0:11:16.240 --> 0:11:18.880 electro array, so instead of a single wire like you 0:11:19.040 --> 0:11:21.600 mentioned in the beginning, now imagine a hundred of those 0:11:21.600 --> 0:11:24.959 little wires kind of all stacked together, recording from thus 0:11:25.040 --> 0:11:29.240 about one hundred neurons. And they showed that these arrays 0:11:29.280 --> 0:11:31.480 could be put in a person with paralysis, and even 0:11:31.520 --> 0:11:34.400 though that person hadn't moved in a decade. I think 0:11:34.600 --> 0:11:36.559 the first guy was a young man in his twenties 0:11:36.600 --> 0:11:39.559 who had been paralyzed from the neck down due to 0:11:39.600 --> 0:11:42.560 a knife wound from like a bar fight. So he 0:11:42.600 --> 0:11:46.000 hadn't moved in many, many years. But they put that 0:11:46.040 --> 0:11:48.600 electro array in the motor cortex, the part of the 0:11:48.600 --> 0:11:52.199 brain that normally sends commands to the arm, and when 0:11:52.240 --> 0:11:54.680 he tried to move his arm, lo and behold, those 0:11:54.720 --> 0:11:57.960 neurons fired away. And so kind of the main risk 0:11:58.080 --> 0:12:02.080 had been solved, which is would the brain even still 0:12:02.120 --> 0:12:05.040 try to generate movements because you might think, well, use 0:12:05.080 --> 0:12:07.800 it or lose it. Right, the person's paralyzed, why would 0:12:07.800 --> 0:12:10.880 their brain still generate movement commands. Fortunately it still does, 0:12:11.679 --> 0:12:14.640 and people were able to decode those signals. 0:12:14.320 --> 0:12:16.680 And just as a quick reminder to everybody, the brain 0:12:16.800 --> 0:12:18.920 is saying, okay, I want you to make these movements, 0:12:19.000 --> 0:12:21.880 and then those shoot down down the spinal cord and 0:12:21.880 --> 0:12:24.440 out to the peripheral nervous system and move the muscles. 0:12:24.840 --> 0:12:28.200 And so in this case you're hearing the original command, 0:12:28.720 --> 0:12:33.120 but there's some break in the roadway plunging down the 0:12:33.160 --> 0:12:36.120 spinal cord and out such that the body never gets 0:12:36.160 --> 0:12:37.720 the signals correctly exactly. 0:12:37.760 --> 0:12:39.880 We're bypassing the injury. We're going to the source. So 0:12:39.920 --> 0:12:41.320 where's the command coming from? 0:12:41.360 --> 0:12:43.320 So this was back in two thousand and four, what 0:12:43.320 --> 0:12:46.360 was his name, Matt Nagel. Is that researchers are able 0:12:46.400 --> 0:12:49.400 to listen to what the neurons are intending, and then 0:12:49.760 --> 0:12:51.760 the field has really taken off since then in the 0:12:51.800 --> 0:12:56.120 past two decades. For example, with motor movement, originally it 0:12:56.200 --> 0:12:58.680 was just on a computer screen you could move a 0:12:58.679 --> 0:13:03.079 cursor around. Nowadays people are thinking about Hey, could you 0:13:03.160 --> 0:13:06.719 actually use an exoskeleton to move the arm physically? 0:13:07.120 --> 0:13:09.840 Yeah, or even stimulate those paralyzed muscles. 0:13:09.880 --> 0:13:14.880 So there's these functional electrical stimulation systems or epidural spinal stimulation, 0:13:15.000 --> 0:13:17.959 both for walking and for the arm. So you can 0:13:18.320 --> 0:13:20.800 really close the loop. You can decode what movement the 0:13:20.840 --> 0:13:21.559 person's trying to make. 0:13:21.520 --> 0:13:21.559 It. 0:13:21.600 --> 0:13:23.960 Oh, they're trying to move their arm forward to grab something, 0:13:24.559 --> 0:13:26.960 and then you can have that move a robotic arm. 0:13:27.240 --> 0:13:29.880 You could have that move an exoskeleton, or if they 0:13:29.920 --> 0:13:33.480 also have a stimulator that's implanted under the skin with 0:13:33.559 --> 0:13:36.480 wires going to the muscles or going outside of the spine, 0:13:36.679 --> 0:13:39.880 you can stimulate the body and actually have the person's 0:13:39.880 --> 0:13:44.200 own formally paralyzed muscles make that movement. It's not at 0:13:44.240 --> 0:13:46.280 the level that you or I let a healthy person 0:13:46.320 --> 0:13:48.560 is moving their arm, but it does work. There's been 0:13:48.559 --> 0:13:51.280 some really amazing studies in the last decade doing that. 0:13:51.480 --> 0:13:54.080 Yeah, exactly right, Okay, great, So that's how people have 0:13:54.160 --> 0:13:58.679 been using brain computer interfaces to move a paralyzed body. Now, 0:13:58.760 --> 0:14:01.800 something that several groups have gotten interested in in recent 0:14:01.880 --> 0:14:05.480 years is what if somebody can't speak anymore? So, what 0:14:05.520 --> 0:14:08.040 are the reasons. First of all, that somebody can't speak. 0:14:08.360 --> 0:14:11.960 So one common one is neurodegenerative diseases like ALS. So 0:14:12.040 --> 0:14:16.000 ALS is a terrible disease, hemiotrophic lateral sclerosis, right and 0:14:16.080 --> 0:14:18.839 right now there's no cure. We can't stop it with 0:14:19.240 --> 0:14:21.240 a drug or other therapy. 0:14:21.120 --> 0:14:22.560 Also known as Luke Gerrig's disease. 0:14:22.600 --> 0:14:26.200 That's right, yeah, and almost everyone who has ALS will 0:14:26.240 --> 0:14:28.960 gradually lose the ability to move their body. But also 0:14:29.080 --> 0:14:32.640 that means what we call the speech articulators, so their lips, 0:14:32.680 --> 0:14:35.760 their jaw, their tongue, their diaphragm, and so their speech 0:14:35.800 --> 0:14:39.120 becomes harder and harder to understand, and eventually you wind 0:14:39.200 --> 0:14:41.480 up what's called locked in, so really not able to 0:14:41.520 --> 0:14:44.840 move at all. And of course this is a terrible situation. 0:14:45.680 --> 0:14:48.800 And if there were a way to restore the ability 0:14:48.840 --> 0:14:53.480 to communicate, so like before decoding not now not they 0:14:53.560 --> 0:14:55.480 are movements that're trying to make, or the leg movements, 0:14:55.520 --> 0:14:57.280 but what are the words that're trying to make, or 0:14:57.280 --> 0:14:59.160 what are the movements of those articulars that they're trying 0:14:59.160 --> 0:15:02.600 to make. What's are they trying to produce? Then we 0:15:02.640 --> 0:15:05.680 can have this person communicate again and talk again through 0:15:05.720 --> 0:15:06.160 a computer. 0:15:06.440 --> 0:15:08.520 If you want to figure out what somebody is trying 0:15:08.560 --> 0:15:11.120 to say, where do you put the electrodes? 0:15:11.360 --> 0:15:13.400 Yeah, and that is the big question. So there are 0:15:13.400 --> 0:15:14.200 a lot of ideas. 0:15:14.240 --> 0:15:16.720 One idea would be the broker's area, which was thought 0:15:16.760 --> 0:15:21.200 to plan speech. Another idea would be the motor cortex, 0:15:21.240 --> 0:15:26.440 which would be kind of the last planning to command generation. 0:15:26.520 --> 0:15:28.440 So the part of the brain that's really sending signals 0:15:28.480 --> 0:15:32.640 to the muscles. And then there's a wide part of 0:15:32.720 --> 0:15:34.880 the brain that are called the language network. 0:15:34.920 --> 0:15:36.200 So this is the temporal lobe. 0:15:36.800 --> 0:15:39.760 It's canonically thought of for perceiving language, but also heavily 0:15:39.760 --> 0:15:41.840 involved in producing language. So there are a lot of 0:15:41.920 --> 0:15:46.400 possible choices. One of the challenges for developing a speech 0:15:46.400 --> 0:15:49.840 ne or prosthesis is there's no animal model. So when 0:15:50.240 --> 0:15:52.760 the field was trying to have people walk again or 0:15:52.760 --> 0:15:55.360 people move their arms again, we had a huge head 0:15:55.360 --> 0:15:58.160 start because you could say, okay, where can you code 0:15:58.440 --> 0:16:01.040 the walking or the arm moved of a rat or 0:16:01.080 --> 0:16:04.720 a monkey or another animal. Well, animals don't talk, they 0:16:04.720 --> 0:16:09.360 don't have language, so we don't have that kind of 0:16:09.400 --> 0:16:12.960 guidance for us, and what we do have are less 0:16:13.120 --> 0:16:16.520 precise measurements from other humans. A lot of the really 0:16:16.600 --> 0:16:19.080 important work from the last decade or twenty years was 0:16:19.440 --> 0:16:23.480 done with electrocorticography. So people with epilepsy often will have 0:16:23.840 --> 0:16:26.760 electrodes put under their skull, typically on top of their 0:16:26.800 --> 0:16:30.400 brain or even in their brain to for the neurologists 0:16:30.400 --> 0:16:31.280 to identify. 0:16:30.880 --> 0:16:32.160 Where the teacher is coming from. 0:16:32.440 --> 0:16:34.040 But these people are then in the hospital for a 0:16:34.040 --> 0:16:36.560 couple of weeks, and this is a gold mine for 0:16:36.720 --> 0:16:39.520 human neuroscience. A lot of what we know about direct 0:16:39.520 --> 0:16:42.760 brain recordings and how they relate to human specific behaviors, 0:16:42.800 --> 0:16:46.480 whether that's speaking or language, or imagination or memory. 0:16:46.760 --> 0:16:48.280 Or mood, all of these things. 0:16:48.440 --> 0:16:51.080 A lot of that comes from this sort of opportunistic 0:16:51.160 --> 0:16:53.240 recording people who are they're in the hospital anyway, they're 0:16:53.320 --> 0:16:55.960 kind of bored, they're waiting for the neurologists to have 0:16:56.120 --> 0:16:58.560 enough data, and so it's very easy to ask them, hey, do. 0:16:58.560 --> 0:17:00.680 You want to read a sentence off a screen. 0:17:00.760 --> 0:17:03.960 So from that we already knew that this sensory motor cortex. 0:17:04.080 --> 0:17:08.879 So the motor and the sensory cortex was a prime area, 0:17:08.960 --> 0:17:12.000 and in our brain Gate clinical trial, that's where we 0:17:12.080 --> 0:17:15.359 ended up putting electrodes, so in the motor part, basically 0:17:15.680 --> 0:17:17.879 the part of the brain that would typically send commands 0:17:17.920 --> 0:17:18.679 to the muscles. 0:17:18.920 --> 0:17:23.359 Great, so it's essentially like the last train station before 0:17:23.400 --> 0:17:27.440 it plunges down towards the muscles. Okay, so you're eavesdropping 0:17:27.480 --> 0:17:31.679 there and you're sticking these little electrode or raise these 0:17:31.680 --> 0:17:34.280 little square jobs where they have sixty four electrodes on 0:17:34.280 --> 0:17:35.960 the one and four of those. 0:17:35.920 --> 0:17:38.560 We used four of them, so yeah, four all along 0:17:38.600 --> 0:17:40.680 this precentral gyrus. 0:17:40.760 --> 0:17:44.640 So you're listening to these neurons and you're trying to 0:17:44.840 --> 0:17:49.760 decode what the person is intending to say from that. 0:17:50.280 --> 0:17:53.600 And one question, were you worried at the beginning that 0:17:53.600 --> 0:17:56.720 that wouldn't be enough data or did you feel like, look, 0:17:56.760 --> 0:17:59.640 with two hundred fifty six neurons, we can figure out 0:17:59.680 --> 0:18:02.240 what's going on in terms of what was trying to 0:18:02.320 --> 0:18:03.080 be articulated. 0:18:03.480 --> 0:18:06.359 When I started the project, I was pretty worried. So 0:18:07.200 --> 0:18:09.360 kind of the prior work is we had shown that 0:18:09.400 --> 0:18:11.679 with about one hundred electrodes in a different part of 0:18:11.720 --> 0:18:14.800 the brain, the hand part of motor cortex, we could 0:18:14.800 --> 0:18:18.479 decode speech, but very poorly. There I was classifying between 0:18:18.480 --> 0:18:22.040 the thirty nine phonemes in American English, if I recall 0:18:22.119 --> 0:18:25.760 about thirty three percent accuracy, So that's way better than chance. 0:18:25.800 --> 0:18:27.960 It showed there's information, but that is not good enough 0:18:27.960 --> 0:18:29.280 to understand. 0:18:28.880 --> 0:18:29.440 What someone's saying. 0:18:29.480 --> 0:18:30.679 Tell us what a phoneme is. 0:18:31.240 --> 0:18:33.720 A phoneme is a building block of speech. 0:18:33.800 --> 0:18:36.240 So I think most people are familiar with the syllables, 0:18:36.560 --> 0:18:38.320 think of a phoneme as a little bit smaller than that. 0:18:38.440 --> 0:18:43.200 So good, ooh E. Right, there's consonants, there's vowels. Different 0:18:43.280 --> 0:18:47.159 languages have different phonemes, but in English, depending on the 0:18:47.160 --> 0:18:50.880 dialect or accent, between thirty nine forty one. These are 0:18:50.960 --> 0:18:53.959 the typical ways we break down English. 0:18:54.000 --> 0:18:57.760 Got So you're recording from these neurons, and you were saying, 0:18:57.760 --> 0:19:00.720 can I figure out what phoneme person is trying to 0:19:00.760 --> 0:19:02.919 say right now and right now just from looking at 0:19:02.960 --> 0:19:04.520 this array of neural activity? 0:19:04.720 --> 0:19:05.600 That's exactly right. 0:19:05.680 --> 0:19:09.040 And a little bit before that, my colleagues at Stanford, 0:19:09.080 --> 0:19:10.720 and that was also the lab that I did my 0:19:10.760 --> 0:19:13.800 post doctoral training, and so I started that project then 0:19:13.840 --> 0:19:17.600 moved on. They had implanted one hundred and twenty eight 0:19:17.720 --> 0:19:22.320 electrodes in the motor cortex of a woman with als, 0:19:22.840 --> 0:19:26.000 and with that they were able to decode what words 0:19:26.000 --> 0:19:29.639 she was saying with about seventy five percent accuracy with 0:19:29.680 --> 0:19:31.920 a large vocabulary of one hundred and twenty five thousand words. 0:19:32.080 --> 0:19:35.520 So that was a really really exciting moment for the 0:19:35.520 --> 0:19:38.000 field because that was really banging at the door of 0:19:38.040 --> 0:19:42.639 making this useful for general communication. Now, three out of 0:19:42.640 --> 0:19:45.719 four words correct is amazing. It was way better than 0:19:45.720 --> 0:19:48.320 anything that ever been done before. But you can't have 0:19:48.359 --> 0:19:50.919 a conversation that way. It's just too frustrating. There's too 0:19:50.920 --> 0:19:51.640 many mistakes. 0:19:52.520 --> 0:19:54.399 And so when we will give us a sense of 0:19:54.400 --> 0:19:57.199 the type of mistake, So the person is intending to 0:19:57.240 --> 0:20:01.119 say the word brain, but the neural activity is decoded 0:20:01.160 --> 0:20:03.440 by the computer, and the computer says, oh, he's trying 0:20:03.440 --> 0:20:05.159 to say panda bear or whatever. 0:20:05.359 --> 0:20:07.800 Well it could be panda bear, it's more likely. 0:20:07.880 --> 0:20:10.480 So the the. 0:20:11.320 --> 0:20:14.600 Way that these systems work is well, one way they work. 0:20:14.680 --> 0:20:17.280 The way our systems work is we're decoding from neural 0:20:17.280 --> 0:20:20.600 activity to phonemes and then those phonemes get assembled into 0:20:20.640 --> 0:20:22.840 words using a dictionary. 0:20:22.440 --> 0:20:23.439 And a language model. 0:20:23.760 --> 0:20:25.720 And in fact, if you look at a dictionary, there's 0:20:25.760 --> 0:20:28.160 that phonetic spelling which most people don't use but if 0:20:28.160 --> 0:20:30.520 you want to figure out how to actually pronounce a word. 0:20:30.520 --> 0:20:31.199 You can look at that. 0:20:31.280 --> 0:20:34.120 So the types of mistakes it would more likely make 0:20:34.240 --> 0:20:36.600 would be similar sounding words. 0:20:36.600 --> 0:20:39.800 So if someone's trying to say brain, maybe they'd get barn. 0:20:40.480 --> 0:20:40.920 Yeah. 0:20:40.960 --> 0:20:44.280 And in some contexts you can understand, oh, I hurt 0:20:44.320 --> 0:20:46.720 my barn, I think you maybe you know you got 0:20:46.760 --> 0:20:49.240 an accident, you hurt your brain. But if there's enough 0:20:49.280 --> 0:20:51.560 of those, it just kind of breaks down. And the 0:20:51.560 --> 0:20:54.320 analogy I'd give is when you're typing on your smartphone. 0:20:54.320 --> 0:20:56.560 Most of us are a little bit clumsy. We make 0:20:56.560 --> 0:20:59.760 a lot of typos. The autocorrect can help up to 0:20:59.800 --> 0:21:02.879 a point, but there's this sort of steep cliff where 0:21:03.160 --> 0:21:06.200 if we're making too many typos, the autocrack so the 0:21:06.280 --> 0:21:08.440 language model cannot keep up, and all of a sudden 0:21:08.720 --> 0:21:10.200 you just get gibberish coming out. 0:21:10.680 --> 0:21:12.920 So that's kind of where things were. 0:21:13.080 --> 0:21:15.280 You could it wasn't gibberish, right, that's overstating it, but 0:21:15.680 --> 0:21:33.400 it was not there for communication day to day. 0:21:33.520 --> 0:21:36.719 So you worked with a man who is forty five 0:21:36.800 --> 0:21:40.000 years old, if I'm rememory correctly, and he had als 0:21:40.240 --> 0:21:43.760 and hadn't articulated in about five years. Is that right? 0:21:43.960 --> 0:21:47.480 Yet he was severely disarthuric, meaning most people couldn't understand him, 0:21:47.840 --> 0:21:51.080 and he volunteered for this brain gate to clinical trial 0:21:51.200 --> 0:21:55.200 that we are one of four sights of which meant 0:21:55.359 --> 0:21:59.600 that after a bunch of tests and imaging scans and 0:21:59.640 --> 0:22:02.600 other things, once we determined that it was a good 0:22:02.640 --> 0:22:04.800 fit and it was safe to move forward. He'd had 0:22:04.800 --> 0:22:08.560 this surgery where doctor Brandman, my collaudrator, put these four 0:22:08.960 --> 0:22:11.600 multi electro to rays into his speech motor cortex. 0:22:12.400 --> 0:22:14.240 We waited a couple of weeks. 0:22:13.920 --> 0:22:16.720 For everything to heal up, and then we went to 0:22:16.760 --> 0:22:19.280 his house where all of our equipment was already pre staged. 0:22:19.840 --> 0:22:23.320 We literally plugged him in. So there's this system is wired, 0:22:23.400 --> 0:22:26.480 so it's not wireless yet. And the way we started 0:22:26.520 --> 0:22:29.320 it was we needed what's called training data in the 0:22:29.359 --> 0:22:32.640 machine learning sense, so we needed the algorithms to see 0:22:33.040 --> 0:22:35.479 a bunch of examples of him trying to say words, 0:22:35.480 --> 0:22:37.600 and then what the neural activity looked like, and what 0:22:37.680 --> 0:22:40.240