Speaker 1: Can a person who is blind come to see through
Speaker 1: her tongue? Can a baby be born without ears? What
Speaker 1: does it like to have the smell of a dog?
Speaker 1: And what does any of this have to do with
Speaker 1: airplane pilots or Westworld or potato head. Welcome to Inner
Speaker 1: Cosmos with me, David Eagleman. I'm a neuroscientist and an author,
Speaker 1: and my fascination for a very long time has been
Speaker 1: how brains perceive reality, because the strange part is that
Speaker 1: we're not seeing most of the action that's going on
Speaker 1: out there. So today we're going to dive into that
Speaker 1: and we're going to see how we might expand our perception.
Speaker 1: We're built out of really small stuff like DNA, and
Speaker 1: we're embedded in a very large cosmos, and we're not
Speaker 1: particularly good at receiving reality at either of those scales.
Speaker 1: And that's because we've evolved to deal with reality. It's
Speaker 1: it's very thin slice in between at the level of
Speaker 1: rivers and apples and rabbits and stuff like that. But
Speaker 1: even here, at our level of perception, we're not seeing
Speaker 1: most of the action that's going on. So take, for example,
Speaker 1: the colors of our world, So picture the reds and
Speaker 1: blues and greens and purples. These are light waves that
Speaker 1: bounce off objects and hit these specialized receptors at the
Speaker 1: back of our eyes, and then we perceive these colors,
Speaker 1: but we're not seeing all the light waves that are
Speaker 1: out there. In fact, what we see is less than
Speaker 1: a ten trillionth of the light waves out there. So
Speaker 1: if you look at what's called the electromagnetic spectrum, you
Speaker 1: have radio waves and microwaves and X rays and gamma rays.
Speaker 1: All these are light. They're just different frequencies. These are
Speaker 1: passing through your body right now, and you're completely unaware
Speaker 1: of them because your biology doesn't come with the right
Speaker 1: receptors to pick those up. They are light, but they're
Speaker 1: not visible light. There are thousands of cell phone conversations
Speaker 1: passing through you right now, and you're completely blind to them. Now,
Speaker 1: it's not that these other wavelengths of light are inherently unseeable.
Speaker 1: Snakes include some infrared light in their reality, and honeybees
Speaker 1: include ultraviolet light in their view of the world. And
Speaker 1: of course we build machines in the dashboards of our
Speaker 1: cars to pick up on signals in the radio frequency range,
Speaker 1: and we build machines and hospitals to pick up on
Speaker 1: the X ray range and so on. But you can't
Speaker 1: sense any of these things by yourself, at least not yet,
Speaker 1: because you don't come equipped with the proper sensors. Now,
Speaker 1: what this means is that our experience of reality is
Speaker 1: it's trained by our biology, and that goes against this
Speaker 1: common sense notion that our eyes and our ears and
Speaker 1: our fingertips are just picking up on the objective reality
Speaker 1: out there. Instead, what this means is that our brains
Speaker 1: are sampling just a little bit of the world. Now,
Speaker 1: across the animal kingdom, different animals pick up on different
Speaker 1: parts of reality. So take the tick. It's blind and
Speaker 1: death and in its little world, the important signals are
Speaker 1: temperature and body odor butteric acid, and that's all it
Speaker 1: picks up on, and that's how it constructs its reality.
Speaker 1: For a fish called the black ghost knife fish, its
Speaker 1: sensory world is all about electrical fields and the perturbations
Speaker 1: of those fields when it's passing a rock or another creature,
Speaker 1: and that's always picking up on. For the echolocating bat,
Speaker 1: its reality is constructed out of air compression waves that
Speaker 1: bounce off objects and come back to them. So for
Speaker 1: these different animals. That's the slice of their ecosystem that
Speaker 1: they can pick up on, and that's all they're seeing.
Speaker 1: And we have a word for this in science. This
Speaker 1: is called the umveldt, which is the German word for
Speaker 1: the surrounding world. Now, every animal is very limited in
Speaker 1: the umveldt that it can pick up on, but presumably
Speaker 1: every animal assumes that it's umveldt is the entire objective
Speaker 1: reality that's out there, because why would you ever stop
Speaker 1: to imagine that there's something beyond what you can sense. Instead,
Speaker 1: we all accept reality as it is presented to us.
Speaker 1: So let's do a consciousness raiser on this. Imagine that
Speaker 1: you are your family dog, and your whole world is
Speaker 1: about smelling. So you've got this long snout that has
Speaker 1: two hundred million scent receptors in it, and you have
Speaker 1: wet nostrils that attract and trap scent molecules. And your
Speaker 1: nostrils even have slits so you can take these big
Speaker 1: nose fulls of air. You have floppy ears to kick
Speaker 1: up more scent. Everything is about smell for you. So
Speaker 1: one day you stop in your tracks with a revelation
Speaker 1: and you look at your human owners and you think,
Speaker 1: what is it like to have the pitiful little nose
Speaker 1: of a human? What is it like when they take
Speaker 1: a little feeble nose full of air? How can a
Speaker 1: human not know that there's a cat one hundred yards
Speaker 1: away or that their best friend was on this very
Speaker 1: spot six hours ago. But because we're humans that we've
Speaker 1: never experienced that world of smell, we don't miss it
Speaker 1: and we don't even think about it. Because we are
Speaker 1: firmly settled into our umvelt we don't feel like there's
Speaker 1: a black hole of smell that we're missing there. We
Speaker 1: think we've got the whole world. But the question is
Speaker 1: do we have to be stuck in the umveldt into
Speaker 1: which we were born? So as a neuroscientist, I've always
Speaker 1: been interested in the way that our technology might allow
Speaker 1: us to expand our umveldt and how that's going to
Speaker 1: change the experience of being human. So we're already quite
Speaker 1: good at marrying our technology to our biology. You may
Speaker 1: know this, but there are hundreds of thousands of people
Speaker 1: walking around with artificial hearing and artificial vision. The way
Speaker 1: this works, for example, with artificial hearing is you have
Speaker 1: a microphone and you digitize the signal and you put
Speaker 1: an electrode strip directly into the inner ear. Or with
Speaker 1: artificial vision, you have what's called a retinal implant, where
Speaker 1: you take a camera and you digitize this signal and
Speaker 1: you plug an electrode grid directly into the back of
Speaker 1: the eye and the optic nerve. Now, as recently as
Speaker 1: twenty five years ago, there were a lot of scientists
Speaker 1: who these technologies were never going to work. Why. It's
Speaker 1: because these technologies speak the language of silicon valley and
Speaker 1: zeros and ones, and it's not exactly the same dialect
Speaker 1: as our natural biological sense organs. But the fact is
Speaker 1: that these technologies work. The brain figures out how to
Speaker 1: use the signals just fine. Now, how do we understand that?
Speaker 1: The key to understanding this requires diving one level deeper.
Speaker 1: Your three pounds of brain tissue are not hearing or
Speaker 1: seeing the world around you directly. It's not that your
Speaker 1: eyes are piping in light or your ears are piping
Speaker 1: sound in. Instead, your brain is locked in a crypt
Speaker 1: of silence and darkness inside your skull. All inever experiences
Speaker 1: are electrochemical signals that stream in along different data cables.
Speaker 1: That's all it has to work with are these little
Speaker 1: electrical spikes and chemical releases. It's just a world of
Speaker 1: spikes running around in darkness inside there, and in ways
Speaker 1: that we're still working to understand. The brain is shockingly
Speaker 1: good at taking these signals running around and extracting patterns
Speaker 1: into those patterns that as signs meaning, and with that meaning,
Speaker 1: you have subjective experience. So the brain is an organ
Speaker 1: that converts sparks in the dark into a picture show
Speaker 1: of your world. All the hues and aromas and emotions
Speaker 1: and sensations of your life. These are encoded in trillions
Speaker 1: of signals zipping around in the blackness. So you know
Speaker 1: when you watch a beautiful screensaver on your computer screen,
Speaker 1: that's just built out of zeros and ones and transistors,
Speaker 1: and it's somehow the same thing that's happening with your
Speaker 1: experience of the world. Let's understand this just a little
Speaker 1: bit more. Imagine that you traveled over to an island
Speaker 1: of people who are all born blind, so they all
Speaker 1: read by braille. They feel tiny patterns of inputs on
Speaker 1: their fingertips. So you watch them read a book and
Speaker 1: they're brushing over the small bumps with their fingers and
Speaker 1: you watch them laugh and cry at the book they're reading,
Speaker 1: and you might wonder how can they fit all that
Speaker 1: emotion into the tip of their finger. So you explain
Speaker 1: to them that when you read a novel, you aim
Speaker 1: these spheres on your face towards visual patterns of lines
Speaker 1: and curves on a page, and each of your eyes
Speaker 1: has a lawn of cells that catch photons, and in
Speaker 1: this way you can register the shapes of the symbols.
Speaker 1: And you tell them that you have memorized a set
Speaker 1: of rules by which different shapes on the page represent
Speaker 1: different sounds. So for each squiggle that you detect with
Speaker 1: your eyes, you recite a small sound in your head,
Speaker 1: imagining what you would hear if someone were speaking that
Speaker 1: out loud. And so the resulting pattern of neurochemical signaling
Speaker 1: makes you laugh or cry. You couldn't blame the islanders
Speaker 1: for finding your story difficult to understand. How do you
Speaker 1: fit all that emotion into two spheres on your head? Okay,
Speaker 1: So you or they would finally have to allow something,
Speaker 1: which is that the fingertip or the eyeball is just
Speaker 1: the peripheral device that converts information from the outside world
Speaker 1: into spikes in the brain. And then the brain does
Speaker 1: all the hard work of the interpretation. You and the
Speaker 1: Islanders would break bread over the fact that in the end,
Speaker 1: it's all about the trillions of spikes racing around in
Speaker 1: the brain, and that the method of entry simply isn't
Speaker 1: the part that matters, because your brain doesn't know and
Speaker 1: it doesn't care where it gets the data from. Whatever
Speaker 1: information comes in from the outside, it just figures out
Speaker 1: what to do with it. And this is a very
Speaker 1: efficient kind of machine. It is essentially a general purpose
Speaker 1: computing device. It just takes in everything and it figures
Speaker 1: out what it's going to do with it. And in
Speaker 1: my work, I've proposed that this freeze up mother nature
Speaker 1: to tinker around with different sorts of input channels. So
Speaker 1: I've argued in my talks and books and papers that
Speaker 1: we can send information into the brain via unusual pathways.
Speaker 1: And I call this the pH model of evolution. And
Speaker 1: I don't want to get too technical here, but pH
Speaker 1: stands for potato head, and I use this name to
Speaker 1: emphasize that all these sensors that we know in love,
Speaker 1: like our eyes and our ears and our fingertips, these
Speaker 1: are merely peripheral, plug and play devices, you stick them
Speaker 1: in and you're good to go, just like with a
Speaker 1: potato head. Where you attach these devices, the brain figures
Speaker 1: out what to do with the data that comes in.
Speaker 1: And by the way, when you look across the animal kingdom,
Speaker 1: you find lots of interesting peripheral devices. So snakes have
Speaker 1: heat pits with which they detect the infrared light. And
Speaker 1: the black ghost knifefish has electro receptors up and down
Speaker 1: its body. That's how it detects the changes in the
Speaker 1: electrical field. And there's an animal called the star nosed mole,
Speaker 1: which essentially has this nose with twenty two fingers on it,
Speaker 1: and it moves around through its three dimensional tunnel system
Speaker 1: and feels around and constructs a model of its world
Speaker 1: that way. And many birds and cows and insects have
Speaker 1: specializations so that they can feel the magnetic field of
Speaker 1: the planet. This is called magneto reception, and they navigate
Speaker 1: that way. The idea with the potato head model is
Speaker 1: that Mother Nature doesn't have to continually redesign the brain
Speaker 1: every time she introduces some new peripheral device. Instead, with
Speaker 1: the principles of brain operation already established, all she has
Speaker 1: to do is worry about designing new peripheral devices to
Speaker 1: pick up on new information from the world. So in
Speaker 1: the same way you can plug an arbitrary nose or
Speaker 1: eyes or mouth into potato head. Likewise, nature plugs all
Speaker 1: kinds of instrumentation into the brain for the purpose of
Speaker 1: detecting these energy sources in the outside world. Now, the
Speaker 1: idea of looking at our peripheral sensors like individuals standalone
Speaker 1: devices might seem bizarre, because, after all, aren't there thousands
Speaker 1: of genes involved with building these devices, and don't these
Speaker 1: genes overlap with other pieces and parts of the body.
Speaker 1: Can we really look at the nose or the eye,
Speaker 1: or the ear or the tongue as a device that
Speaker 1: stands alone. So I started studying this question because I thought,
Speaker 1: if the potato head model is correct, wouldn't that suggest
Speaker 1: we might find switches in the genetics that lead to
Speaker 1: the presence or absence of a peripheral device. And as
Speaker 1: it turns out, that's precisely what can happen. So, for example,
Speaker 1: some babies are born completely missing a nose, and they
Speaker 1: also lack the nasal cavity and the whole system for smelling.
Speaker 1: This is called a rhinia. Now, these kind of mutations,
Speaker 1: they seem startling and difficult to fathom, But in our
Speaker 1: plug and play framework, a rhinia is predictable. With a
Speaker 1: slight tweak of the genes, the peripheral device simply doesn't
Speaker 1: get built. Or consider other babies who are born normal,
Speaker 1: but they have no eyes. This is called anophthalmia, and
Speaker 1: others are born without tongues. Some babies are born without ears,
Speaker 1: that's called anotia. Some children are born without any pain receptors,
Speaker 1: and more generally, others are born without any touch receptors.
Speaker 1: This is called nafia. And so when we look at
Speaker 1: these situations, it becomes clear that our peripheral detectors unpack
Speaker 1: because of specific genetic programs. And if you have a
Speaker 1: minor mauthfunction in the genes, that can halt the program,
Speaker 1: and then the brain just doesn't get that particular data
Speaker 1: stream of information from the world, whether that's smell, molecules,
Speaker 1: or photons or air compression waves or touch or whatever.
Speaker 1: For me, the lesson that comes together here is that
Speaker 1: nature designs ways of extracting information from the world world,
Speaker 1: and these unpack with their own little genetic instructions. Now,
Speaker 1: what this implies is that there's nothing really fundamental about
Speaker 1: the devices that you and I come to the table
Speaker 1: with our eyes and our ears and our nose and
Speaker 1: our fingertips. It's just what we've inherited from a complex
Speaker 1: road of evolution. But that particular collection of sensors might
Speaker 1: not have to be what we stick with, because the
Speaker 1: brain's ability to decode different kinds of incoming information implies
Speaker 1: the crazy prediction that you might be able to get
Speaker 1: some sensory cable going into the brain to carry a
Speaker 1: different kind of sensory information. For example, what if you
Speaker 1: took a data stream from a video camera and converted
Speaker 1: that into touch on your skin. Would the brain eventually
Speaker 1: be able to interpret the visual world simply by feeling it?
Speaker 1: And this is the stranger than fiction world of sensory substitution.
Speaker 1: Sensory substitution refers to the idea of feeding information into
Speaker 1: the brain via unusual sensory channels and the brain just
Speaker 1: figures out what to do with the information. Now, that
Speaker 1: might sound speculative, but the first paper demonstrating this was
Speaker 1: published in the journal Nature in nineteen sixty nine. There
Speaker 1: was a scientist named Paul Baki Rita and he put
Speaker 1: blind people in a modified dental chair and he set
Speaker 1: up a video feed and he would put something in
Speaker 1: front of the camera and then the person would feel
Speaker 1: that poked into their back with a grid of solenoids.
Speaker 1: So if he put a coffee cup in front of
Speaker 1: the camera, they would feel the shape of a coffee
Speaker 1: cup in their back. Or he puts a telephone in
Speaker 1: front of the camera, and they feel a telephone in
Speaker 1: their back. And amazingly, people who were blind got pretty
Speaker 1: good at being able to determine what was in front
Speaker 1: of them the camera, just by feeling it in the
Speaker 1: small of their back. So Baki Rita summarized his findings
Speaker 1: by saying, quote, the brain is able to use incoming
Speaker 1: information from the skin as if it were coming from
Speaker 1: the eyes end quote. The subjective experience for the blind
Speaker 1: people who are feeling this in their back was that
Speaker 1: visual objects were located out there instead of on the
Speaker 1: skin of their back. In other words, it was something
Speaker 1: like vision. And think about it this way. When you're
Speaker 1: at the coffee shop and you see your friend waving
Speaker 1: at you across the way, the photons from your friend
Speaker 1: are impinging on your photoreceptors in your eye. But you
Speaker 1: don't perceive that the signal is at your eyes or
Speaker 1: in your brain. You perceive that your friend is out
Speaker 1: there waving at you from a distance. And so it
Speaker 1: goes with the users of baki Rita's modified dental chair.
Speaker 1: They were perceiving the object out there now. Amazingly. While
Speaker 1: baki Rita's device was the first to hit public consciousness,
Speaker 1: it was not actually the first attempt at sensory substitution.
Speaker 1: On the other side of the world, at the end
Speaker 1: of the eighteen nineties, a Polish ophthalmologist developed a crude
Speaker 1: device for people who were blind. He put a single
Speaker 1: photo cell on the forehead of a blind person, and
Speaker 1: the more light that hit it, the louder a sound
Speaker 1: would be in the person's ear, so based on the
Speaker 1: sound's intensity, the blind person could tell where there were
Speaker 1: lights or where there were dark areas. Unfortunately, the whole
Speaker 1: device was very large and heavy, and of course it
Speaker 1: was only one pixel of resolution, so it never got
Speaker 1: any traction. But in nineteen sixty another group in Poland
Speaker 1: picked up the ball and ran with it. They recognized
Speaker 1: that hearing is critical for the blind, so they turned
Speaker 1: to passing in the light information via touch. They built
Speaker 1: a helmet that had all these vibratory motors in it,
Speaker 1: and they essentially drew the images on the head, and
Speaker 1: blind participants were able to move around in these specially
Speaker 1: prepared rooms that were painted to enhance the contrast of
Speaker 1: the door frames and the furniture edges. It worked. Unfortunately,
Speaker 1: it was also heavy and would get very hot, and
Speaker 1: so the world had to wait. But the proof of
Speaker 1: principle was starting to emerge. Now, why did these strange
Speaker 1: approaches work. It's because input to the brain, whether that's
Speaker 1: from photons of the eyes, or air compression waves of
Speaker 1: the ears, or pressure on the skin, they're all converted
Speaker 1: into the common currency of electrical signals. So as long
Speaker 1: as the incoming spikes carry information that represents something important
Speaker 1: about the outside world, the brain will learn how to
Speaker 1: interpret it. The vast forests of your brain cells in
Speaker 1: the dark, they don't care about how the spikes get there.
Speaker 1: They just do their work on it. Now, there have
Speaker 1: been all kinds of incarnations of sensory substitution for the blind.
Speaker 1: One also from the nineteen sixties, is called the sonic glasses.
Speaker 1: It takes a video feed right in front of you
Speaker 1: and turns that into a sound landscape. So as things
Speaker 1: move around and get closer and farther, it sounds like
Speaker 1: it sounds like a cacophony. But after some time, blind
Speaker 1: people start getting really good at understanding what is in
Speaker 1: front of them just based on what they're hearing through
Speaker 1: their ears. And the best example of this is a
Speaker 1: program that you can download on your cell phone called
Speaker 1: the Voice. Note that the three middle letters are oh,
Speaker 1: I see anyway. This is developed by an engineer named
Speaker 1: Meyer in the Netherlands, and it started as a bulky project,
Speaker 1: but it can now be downloaded on your phone. You
Speaker 1: point your phone camera at things and the program converts
Speaker 1: what the phone sees into sounds. The app is amazing
Speaker 1: and you can download this onto your phone and start
Speaker 1: walking around in the world with it and really understand
Speaker 1: what's going on when you convert site into sound. And
Speaker 1: my colleagues all over the world, like Jamie Warren and
Speaker 1: emirah Medi, have been running science experiments on these sorts
Speaker 1: of approaches. And by the way, the century substitution doesn't
Speaker 1: have to be through the ears. Another version is called
Speaker 1: the brain port, and this is a little grid. It's
Speaker 1: called an electro techtile grid. It sits on your tongue
Speaker 1: and gives little shocks. So you have a camera and
Speaker 1: that video feed gets turned into these little shocks on
Speaker 1: your tongue. It feels like pop rocks in your mouth.
Speaker 1: And blind people can get so good at using this
Speaker 1: that they can throw a ball into a basket where
Speaker 1: they can navigate a complex obstacle. Course they can come
Speaker 1: to see through their tongue. Now that sounds completely insane, right,
Speaker 1: but remember all vision, ever is, are these electrical signals
Speaker 1: coursing around in your brain. Your brain doesn't know where
Speaker 1: the signals come from, it just figures out what to
Speaker 1: do with them. So my laboratory set out some years
Speaker 1: ago to solve sensory substitution for people who are deaf,
Speaker 1: and we wanted to make it so that the sound
Speaker 1: from the world gets converted in some ways so that
Speaker 1: a deaf person can understand what is being said. So
Speaker 1: with my graduate student, Scott Novic, we built a vest.
Speaker 1: Now this is not a normal vest. This is a
Speaker 1: vest that zips up tight around the torso and it
Speaker 1: has thirty two little motors on it. And these are
Speaker 1: vibratory motors like the buzzer on your cell phone, but
Speaker 1: thirty two of them, and they're distributed pretty evenly around
Speaker 1: your waist in your back, and each motor represents a
Speaker 1: different frequency of sound from low to high. And by
Speaker 1: breaking up sound in this way, this is the same
Speaker 1: thing that your inner ear does a part of your
Speaker 1: interear called the cochlea, So we have essentially transferred the
Speaker 1: cochlea to the torso, so it captures sound and turns
Speaker 1: that into these patterns vibration. So some years ago we
Speaker 1: started to test this in conjunction with the deaf community.
Speaker 1: Our first participant was a guy named Jonathan. He was
Speaker 1: thirty seven years old. He had a master's degree, and
Speaker 1: he had been born profoundly deaf, which means there was
Speaker 1: a part of his umvelt that was unavailable to him.
Speaker 1: So we had Jonathan wear the vest and train with
Speaker 1: it for four days, two hours a day, and by
Speaker 1: the fifth day he was pretty good at identifying the
Speaker 1: words that were being said to him. So you say
Speaker 1: the word dog, and Jonathan feels a pattern of vibrations
Speaker 1: all over the vest, and his job is simply to
Speaker 1: write on the dry erase board what he thinks the
Speaker 1: word might have been, and by day five, he could
Speaker 1: get this mostly right. Now. We had trained him on
Speaker 1: a limited number of words, what's called a closed set,
Speaker 1: but when we switched to a new set of words,
Speaker 1: once he had never heard before, he was able to
Speaker 1: perform well above chance. And he learned more and more
Speaker 1: quickly with every new set. And this suggested he wasn't
Speaker 1: just memorizing some answers. He was actually learning how to
Speaker 1: hear with the vest. He was translating the complicated pattern
Speaker 1: of vibrations into an understanding of what was being said. Now,
Speaker 1: he wasn't doing this consciously, because the patterns are too
Speaker 1: complicated for that, but his brain was unlocking the meaning
Speaker 1: of this. And by the way, this is just like you.
Speaker 1: So listening to this podcast, you're not thinking, oh, Eagelman
Speaker 1: is saying some high frequencies and now some low and
Speaker 1: some medium, so that must be a s sound. Instead,
Speaker 1: You've just practiced hearing your whole life, and eventually you
Speaker 1: become pretty good at using your ears and your brain.
Speaker 1: But when you were born, you didn't know how to
Speaker 1: use your ears, but your brain looked for correlations, things
Speaker 1: that went together. So you would watch your mother's mouth
Speaker 1: moving and you get spikes coming down your auditory nerve,
Speaker 1: and you figure out that those go together. Or as
Speaker 1: a baby, you clap your hands and you get a
Speaker 1: different pattern of spikes coming down your auditory nerve. Or
Speaker 1: you bang on the bars of your cage, or you
Speaker 1: babble with your mouth, and these all correlate with particular
Speaker 1: patterns coming in along this nerve, and eventually these patterns
Speaker 1: become what philosophers call a qualia, which is a private,
Speaker 1: subjective experience of hearing. You don't have to think about
Speaker 1: what all the spikes mean. They just get translated into
Speaker 1: a direct perceptual experience. Okay, so back to the vest.
Speaker 1: So we tested the vest with lots of participants in
Speaker 1: the deaf community, and in fact, we even built a
Speaker 1: miniature vest because it turned out that one of the
Speaker 1: people we were working with had a daughter who was
Speaker 1: born deaf and blind. So we made this little miniature
Speaker 1: vest for her and it picked up on the sounds
Speaker 1: of the world and translated this into patterns of vibration
Speaker 1: on her skin. And so her grandmother took her around
Speaker 1: the lab and touched her feet on things and said,
Speaker 1: this is hard, this is soft, this is going up,
Speaker 1: this is going down, and so on, and this allowed
Speaker 1: the little girl to tap into a larger part of
Speaker 1: her umveldt. Then we made a smaller version of the vest,
Speaker 1: just a chest strap, and we began testing that with
Speaker 1: some other children. But eventually we were able to shrink
Speaker 1: the whole system down to a wristband, and that opens
Speaker 1: up the technology for a much larger population, and we
Speaker 1: spun this off of the lab as a company called
Speaker 1: Neo Sensory, and one of our first users was a
Speaker 1: wonderful guy here in the San Francisco Bay area named
Speaker 1: phil and we videoed him talking in sign language about
Speaker 1: what the wristband meant to him. So I'm going to
Speaker 1: quote him here as a translation from the sign language used.
Speaker 1: He signed quote, it makes me feel a natural connection
Speaker 1: with everyone around me. Sometimes I perceive wow. I can
Speaker 1: tell what a sound feels like if someone calls my name,
Speaker 1: or if there's some kind of noise nearby, or my
Speaker 1: dog's barking, or even my wife calling me from far away. Philip,
Speaker 1: I feel her call my name and I go to her.
Speaker 1: So we tested lots of people who were deaf in
Speaker 1: the Bay area, and people reported things to us like
Speaker 1: I'm picking up on running water or birds or the
Speaker 1: oven timer. And when wearing it at work, I had
Speaker 1: a really good experience, like when people were talking in
Speaker 1: the room, I could feel what they were saying and
Speaker 1: it helped me lip read better. And as a quick
Speaker 1: side note, we went to interview lots of people who
Speaker 1: are deaf, and I came to understand that lots of
Speaker 1: deaf people live in nice apartments in one particular location,
Speaker 1: which is right next to the railroad track, because the
Speaker 1: sound of the howling trains passing by doesn't register with
Speaker 1: them and bother them, so they can live comfortably in
Speaker 1: a steeply discounted apartment that's perfectly nice. But people who
Speaker 1: are hearing don't want that apartment. Anyway, back to the story,
Speaker 1: users started telling us that they were picking up on
Speaker 1: things that they didn't even know existed, like that microwaves beeped,
Speaker 1: or that their car blinker made a clicking sound, or
Speaker 1: that if they accidentally left the air blower on at work,
Speaker 1: that it was making a noise, or for that matter,
Speaker 1: the loudness of toilets flushing, or the they had left
Speaker 1: the sink running. And they started feeling things like the
Speaker 1: laughter of their children on their skin. And they were
Speaker 1: able to distinguish which child was talking and which of
Speaker 1: their dogs was barking. And with time, people just get
Speaker 1: better and better at picking up the sounds of the
Speaker 1: world as patterns of vibration on their skin. And with
Speaker 1: one of our users, I asked him, what was it
Speaker 1: like when he hears the dog bark? Does he register? Oh,
Speaker 1: there were just vibrations on my wrist, and so now
Speaker 1: I have to translate that that must have been a
Speaker 1: dog barking. And he said, no, I just hear the
Speaker 1: dog barking out there, which sounds crazy, right, but remember
Speaker 1: that's all that's going on with your ears. You hear
Speaker 1: the sound out there even though it's actually happening in
Speaker 1: here in your head. Now, after we were years into
Speaker 1: this project, I began to discover that the idea of
Speaker 1: converting touch to sound is not even new. I found
Speaker 1: a paper from nineteen twenty three. There was a psychologist
Speaker 1: at Northwestern University called Robert Galt, and he heard about
Speaker 1: a deaf and blind ten year old girl who claimed
Speaker 1: to be able to feel sound through her fingertips. So
Speaker 1: he was skeptical, and so he ran an experiment. He
Speaker 1: stopped up her ears and wrapped her head in a
Speaker 1: woolen blanket, and he put her finger against the diaphragm
Speaker 1: of a device which converted his voice signal into vibrations.
Speaker 1: So Galt sat in a closet and spoke to her
Speaker 1: through the device, and so her only chance to understand
Speaker 1: what he was saying was from the vibrations on her fingertip.
Speaker 1: And what he reported is that it worked. She was
Speaker 1: able to tell what he was saying through her fingertips.
Speaker 1: And in the early nineteen thirties, and educator at a
Speaker 1: school in Massachusetts developed a technique for two deaf and
Speaker 1: blind students. Being deaf, they needed a way to read
Speaker 1: the lips of speakers, but they were blind as well,
Speaker 1: so that couldn't work. So the technique consists of placing
Speaker 1: a hand over the face and neck of the person
Speaker 1: who is speaking, so the thumb rests lightly on the
Speaker 1: lips and the fingers fan out to cover the neck
Speaker 1: and cheek, and in this way they can feel the
Speaker 1: lips moving and the vocal courts vibrating, and even the
Speaker 1: air coming out of the nostrils. And by the way,
Speaker 1: because these two original students were named Tad and Oma,
Speaker 1: this technique is known as the Tadoma technique, and thousands
Speaker 1: of deaf and blind children have been taught this method
Speaker 1: and they can obtain proficiency understanding language almost to the
Speaker 1: point of those with hearing. So the key thing to
Speaker 1: note for our purposes is that all the information is
Speaker 1: coming in through their sense of touch. And in the
Speaker 1: nineteen seventies the death inventor Dmitri Konewski came up with
Speaker 1: a two channel vibrotactile device, one of which captures the
Speaker 1: low frequencies and the other the high frequencies, and these
Speaker 1: two vibratory motors sit on the wrists. And in the
Speaker 1: nineteen eighties some other people came up with things like
Speaker 1: this too, which all demonstrated the power of sensory substitution.
Speaker 1: The problem was that all these devices were too large,
Speaker 1: and they typically just had one motor or two motors,
Speaker 1: and they got too hot, and it was not practical
Speaker 1: for people to wear these. It's only now that we're
Speaker 1: able to capitalize on a whole constellation of tech advances
Speaker 1: to run this in a wristband in real time. And
Speaker 1: so I'm really happy to say that the neosensory wrist
Speaker 1: band is now on risks all over the world. And
Speaker 1: what's cool is that this technology is a game changer
Speaker 1: because the only other solution for deafness is a cochlear implant,
Speaker 1: and that's something that requires about one hundred thousand dollars
Speaker 1: and an invasive surgery. But the riskband we can build
Speaker 1: for one hundred times cheaper, and that opens up the
Speaker 1: technology globally, even for the poorest countries in the world.
Speaker 1: And that's one of the reasons we've been able to
Speaker 1: get this into underfunded schools for the deaf all over
Speaker 1: the globe, and we've had many wonderful philanthropists help us
Speaker 1: do that because this is such a different scale of
Speaker 1: solution that's simple and inexpensive and takes advantage of a
Speaker 1: very strange principle of the brain sensory substitution. And we've
Speaker 1: just released something else that's having real impact. It's a
Speaker 1: version of the same idea, but it's not for people
Speaker 1: who are deaf, but instead people who are having normal
Speaker 1: age related hearing loss, which almost always happens in the
Speaker 1: high freaquent and c which is why people who are
Speaker 1: getting older and losing hearing start having a harder time
Speaker 1: understanding women and children because their voices tend to be
Speaker 1: at a higher frequency. So we develop cutting edge machine
Speaker 1: learning that sits on the wristband and listens in real
Speaker 1: time just for the high frequency parts of speech. So
Speaker 1: for example, it just listens for an S or a
Speaker 1: Z or a B or a K, and the wristband
Speaker 1: signals in different ways each time it hears one of
Speaker 1: those speech sounds. And so the key is when you're
Speaker 1: losing your high frequency hearing, your ears are still doing
Speaker 1: fine at the medium and low frequencies. Those are getting
Speaker 1: to the brain. The risk band is just clarifying what's
Speaker 1: happening at the high frequencies, and your brain learns to
Speaker 1: fuse these signals from your ear and from your skin,
Speaker 1: so it puts together what it heard from the ear
Speaker 1: with what it's getting through the wristband, and after a
Speaker 1: few weeks people develop much clear were hearing. And as
Speaker 1: an interesting side note, people don't always notice that they're
Speaker 1: getting better, but everyone around them does, and if they
Speaker 1: forget to put on the wristband, they get yelled at.
Speaker 1: So that's an example of pushing some information into the
Speaker 1: brain via an unusual channel while most of the information
Speaker 1: is coming in the normal way. And I'll also tell
Speaker 1: you something else amazing that we found, which is that
Speaker 1: the wristband works incredibly well for reducing tonitis, which is
Speaker 1: ringing in the ears. So a couple of research labs
Speaker 1: had previously shown that tonitis can be reduced from something
Speaker 1: called bi modal stimulation, which just means that you have
Speaker 1: sounds and you have touch that are synchronized. So that's
Speaker 1: two modes or by modal. Now, the previous research had
Speaker 1: done this by combining tones beppep with shocks on the tongues,
Speaker 1: and that worked to drive down the ringing in the ears.
Speaker 1: So we did the same thing with the wristband and
Speaker 1: it works the same. We've published our data on this
Speaker 1: that people with tonightis get clinically significant improvement. Now, why
Speaker 1: does something like that work. There are some sophisticated arguments
Speaker 1: and debates about why this works, but I think this
Speaker 1: simple explanation is that we're just teaching the brain what
Speaker 1: is a real external sound, because those get confirmation on
Speaker 1: the wristband when you hear boo boo boo boop you're feeling,
Speaker 1: But the tonitis, the internal gets no verification on the wrist,
Speaker 1: and so the brain figures out that's fake. News and
Speaker 1: it drives it down. Now, we're doing all kinds of
Speaker 1: other experiments using the wristband for sensory substitution. So, for example,
Speaker 1: we've begun to study this as a device for balance.
Speaker 1: So there are many people who have problems with balance
Speaker 1: because of their inner ear. They don't realize when their
Speaker 1: body is tilting. So in our experiments, they wear the
Speaker 1: risk band and they also wear a small collar clip,
Speaker 1: and the collar clip has a motion detector and a
Speaker 1: gyroscope in it, and it can detect your orientation whether
Speaker 1: you're standing straight or you're tilting one way or another,
Speaker 1: and it just sends that information to the wristband, so
Speaker 1: you become aware if you're tilting and you know in
Speaker 1: which direction. So it goes BIZ when you tilt and
Speaker 1: BZ when you turn the other way. And this is
Speaker 1: simply taking what your inner ear would normally do, and
Speaker 1: if there's something wrong with it, it's just sending it
Speaker 1: in through a different channel. And beyond deafness and balance,
Speaker 1: we're doing other things like working with prosthetics. So when
Speaker 1: somebody gets an amputation, they get an artificial leg prosthetic.
Speaker 1: And what we did is we put sensors on the
Speaker 1: leg so that you can feel the information on the wristband.
Speaker 1: So we're taking an artificial limb and by putting angle
Speaker 1: and pressure sensors on it, we are restoring the sensory
Speaker 1: input that you would have from it through wrist and
Speaker 1: that allows patients to learn much more quickly how to
Speaker 1: walk with their new prosthetic limb. Now, beyond sensory substitution,
Speaker 1: how can we use a technology like this to add
Speaker 1: a completely new kind of sense to actually expand the
Speaker 1: human umvelt? For example, could we feed real time data
Speaker 1: from the internet directly into somebody and could they develop
Speaker 1: a direct perceptual experience. So some years ago we did
Speaker 1: an experiment in the lab where a participant feels a
Speaker 1: real time streaming feed from the net of data for
Speaker 1: five seconds, and then he's holding a tablet and two
Speaker 1: buttons appear and he has to make a choice. He
Speaker 1: doesn't know what's going on, but he makes his choice
Speaker 1: and then gets feedback after a second and a half. Now,
Speaker 1: here's the thing. The subject has no idea what all
Speaker 1: these patterns mean, but we're seeing if he can get
Speaker 1: better at figure out which button to press. And he
Speaker 1: doesn't know that what we're feeding is real time data
Speaker 1: from the stock market, and he's making buy and sell decisions,
Speaker 1: and the feedback is telling him whether he did the
Speaker 1: right thing or not. And what we're seeing is can
Speaker 1: we expand the human umvelt so that he comes to
Speaker 1: have a direct perceptual experience of the economic movements of
Speaker 1: the planet. Here's another experiment which I showed it teds
Speaker 1: some years ago in a talk. We scrape the web
Speaker 1: for any hashtag and we do an automated sentiment analysis,
Speaker 1: which means are people using positive words or negative words
Speaker 1: or neutral and we feed that into the vest or
Speaker 1: the wristband. And this allows a person to feel what's
Speaker 1: going on in the community of millions of people and
Speaker 1: to be plugged into the aggregate emotion of giant crowds
Speaker 1: all at the same time. And that's a new kind
Speaker 1: of human experience because you can you can't know normally
Speaker 1: how a population is feeling. It's a bigger experience than
Speaker 1: a human can normally have. And we're working on feeling
Speaker 1: signals that exist out there but are normally invisible to you.
Speaker 1: So imagine that instead of a police officer having to
Speaker 1: have a drug dog, they could instead feel the odors
Speaker 1: around them that they normally couldn't. So imagine building an
Speaker 1: array of molecular detectors and instead of needing the dog
Speaker 1: with its huge snout, they can just directly experience that
Speaker 1: level of smell themselves through vibrations on the skin. And
Speaker 1: we're doing things with robotic surgery. So normally, when a
Speaker 1: surgeon is doing a robotic surgery, they have to keep
Speaker 1: looking up at the monitors to understand what's going on
Speaker 1: with the patient. But imagine being able to simply feel
Speaker 1: the data from the patient, the heart rate and the
Speaker 1: breathing and so on, simply feeling it as you're going
Speaker 1: and not needing to keep looking at the monitors. Another
Speaker 1: thing we've been working on for a while is expanding
Speaker 1: the umvelt of drone pilots. So in this case, we
Speaker 1: have the vest streaming nine different measures from a quad copter,
Speaker 1: so the pitch and yaw and roll and orientation and heading,
Speaker 1: and that improves the pilot's ability to fly it because
Speaker 1: it's essentially like the drone pilot is extending his skin
Speaker 1: up there onto the drone far away. He's becoming one
Speaker 1: with the drone. He can learn how to fly it better.
Speaker 1: In the fog or in the darkness, because essentially he
Speaker 1: is becoming one with the drone. Or something that's related
Speaker 1: to this is imagine taking a modern airplane cockpit which
Speaker 1: is full of gages and instead of trying to read
Speaker 1: the whole thing, you just feel it. Because we live
Speaker 1: in a world of information now and there's a difference
Speaker 1: between accessing big data and experiencing it. And we're also
Speaker 1: exploring to expand your body to a different location. So
Speaker 1: imagine that you feel everything that a robot feels. So
Speaker 1: you send an avatar robot on a rescue mission into
Speaker 1: a place that's very dangerous, like after an earthquake, with
Speaker 1: collapsed buildings and dangerous chemicals, and you feel what the
Speaker 1: avatar robot is feeling. So you can close the feedback
Speaker 1: loop between action and perception. And we're interested in using
Speaker 1: this for the military to reduce friendly fire, which is
Speaker 1: when a person gets killed just because one of their
Speaker 1: colleagues makes a mistake and shoots them. So with our
Speaker 1: chest strap and some encrypted position information, you can tell
Speaker 1: where your friendlies are in any moment because you're feeling them.
Speaker 1: You know their location right on your body, like Fred
Speaker 1: is off to my left because I can feel a
Speaker 1: slight vibration, but now he's getting closer to me, so
Speaker 1: the vibration gets more intense. And now I know that
Speaker 1: Steve is behind the wall over there because I can
Speaker 1: feel him moving around even though I can't see him,
Speaker 1: and Tom is behind me back there. You don't have
Speaker 1: to rely on vision because you're feeling where everyone is.
Speaker 1: So with one of our engineers, Mike Perata, we built
Speaker 1: a version of this and we demonstrated it by turning
Speaker 1: to fiction. We had our vest make a cameo on
Speaker 1: the show Westworld. So if you saw season two, episode seven,
Speaker 1: the storyline is that private military contractors drop into Westworld
Speaker 1: to take care of these out of control robots called
Speaker 1: the Hosts. And we set this up so that the
Speaker 1: military contractors in the show are wearing our vests that
Speaker 1: let them feel the location of the Hosts on their bodies,
Speaker 1: and that's how they know exactly how to target them.
Speaker 1: So as they're moving around, they can feel, oh, there's
Speaker 1: a robot over there, and there's a robot on the
Speaker 1: other side of that thing, and there's a robot the
Speaker 1: dark over there, and they can aim at them appropriately. Now,
Speaker 1: as it turns out, all the military contractors eventually get killed,
Speaker 1: so the vest is not necessarily going to save your
Speaker 1: life if things really hit the fan with robot consciousness,
Speaker 1: but that's a different episode, and we've used this same
Speaker 1: concept for people who are blind. We set this up
Speaker 1: in collaboration with some colleagues at Google who have light
Speaker 1: ar in their offices. Light ar is like sonar, but
Speaker 1: with light and so with light ar you can know
Speaker 1: the location of everything and everybody moving around in the offices,
Speaker 1: and we tapped into that data stream and we brought
Speaker 1: in blind participants and they could feel where everyone was.
Speaker 1: So if there's someone on your right, you feel a
Speaker 1: vibration on your right, and as they get closer, it
Speaker 1: gets more intense than as they go away it gets
Speaker 1: less intense, and you can feel them moving around you
Speaker 1: and you can even feel when they're walking around behind you,
Speaker 1: which is better than sighted vision. And on top of that,
Speaker 1: we also added navigation. So our participants had never been
Speaker 1: to these offices before, but we type into the system
Speaker 1: a particular conference room to go to, and the person
Speaker 1: then feels on their vest a buzzing on the front,
Speaker 1: so they walk straight and then they feel a buzz
Speaker 1: on their left, and they turn left and then they
Speaker 1: feel a diagonal buzz and they know that the conference
Speaker 1: room is diagonally over there, and they were able to
Speaker 1: navigate this way on top of feeling who is around them,
Speaker 1: and so in this way, they're not getting real vision,
Speaker 1: but they're getting a lot of incredibly important information in
Speaker 1: a very simple way. And there's really no end to
Speaker 1: the possibilities on the horizon with sensory substitution and sensory expansion.
Speaker 1: One experiment we did involves using these smart watches that
Speaker 1: can measure things like your heart rate and hurried variability
Speaker 1: in galvanic skin response, and so we tapped into the
Speaker 1: API for that and we put the data on the internet,
Speaker 1: and then you feel that on the wristband, so you
Speaker 1: can feel these normally invisible states of your body. But
Speaker 1: the interesting part is when you take the watch off
Speaker 1: and give it to someone else, let's say your spouse,
Speaker 1: so that now you are feeling the physiologic responses of
Speaker 1: another person. You're tapped into their internal signals. Now, I
Speaker 1: have no idea if this is good or bad for marriages,
Speaker 1: but this is an experiment we're trying because humans are
Speaker 1: at a point now where we can open up new
Speaker 1: folds in the possibility space. There are things we can
Speaker 1: experiment with to have new kinds of senses and bodies,
Speaker 1: and we can feel things like not only other people's physiology,
Speaker 1: but things like entire factories or traffic patterns. In general,
Speaker 1: what this gives us is a new approach to data.
Speaker 1: Our visual systems are fundamentally really good at blobs and
Speaker 1: edges and motion, but they're limited in what they can
Speaker 1: attend to. They can only do one thing at a time,
Speaker 1: and that's not very good for high dimensional data. But
Speaker 1: your body is very good at multidimensional data, which is
Speaker 1: why you can balance on one leg and you're getting
Speaker 1: feedback from all these different muscle groups. You're taking in
Speaker 1: high dimensional data and dealing with it all at once
Speaker 1: and with the right sorts of data compression. I think
Speaker 1: there's no limits to the kind of data that we
Speaker 1: can take in. We have about seventy different experiments running
Speaker 1: on this, and if you're interested, go to neosensory dot
Speaker 1: com slash developers and you can see all the various
Speaker 1: cool projects that we in the community in general has done.
Speaker 1: So the possibilities are endless here. Just imagine an astronaut
Speaker 1: being able to feel the overall health of the International
Speaker 1: Space Station, or for that matter, having you feel the
Speaker 1: invisible states of your own health, like your blood sugar
Speaker 1: and the state of your microbiome, or having three one
Speaker 1: hundred and sixty degree vision or seeing an infrared or ultraviolet.
Speaker 1: So the key is this, as we move into the future,
Speaker 1: we're going to be increasingly able to choose our own
Speaker 1: peripheral devices. We don't have to wait for Mother Nature's
Speaker 1: sensory gifts on her time scales, because instead, like any
Speaker 1: good parent, what she's given us are the tools that
Speaker 1: we need to go out and define our own trajectory.
Speaker 1: So the question now is how do you want to
Speaker 1: experience your universe? That's all for this week. To find
Speaker 1: out more and to share your thoughts, head over to
Speaker 1: eagleman dot com slash podcasts. Any questions or discussions that
Speaker 1: you have please email podcasts at eagleman dot com and
Speaker 1: I will be addressing those on future episodes. Until next time,
Speaker 1: I'm David nigelm In signing off to you from the
Speaker 1: Inner Cosmos.