Speaker 1: Welcome to tech Stuff, a production of I Heart Radios,
Speaker 1: How Stuff Works. Hey there, and welcome to tech Stuff.
Speaker 1: I'm your host, Jonathan Strickland. I'm an executive producer with
Speaker 1: How Stuff Works and I Heart Radio and I love
Speaker 1: all things tech and listener Robert Casey pinned me on
Speaker 1: Twitter with a request that I do an episode about
Speaker 1: touch screens and stylus is or STYLI Now. My original
Speaker 1: co host, Chris Palette, and I covered this topic on
Speaker 1: an ancient episode of tech Stuff that published all the
Speaker 1: way back on October nineteen, two thousand nine, Holy Cow,
Speaker 1: ten years ago. But I think it's past time to
Speaker 1: revisit this topic and give it the full modern day
Speaker 1: tech stuff treatment. Now, there are a few interesting things
Speaker 1: about touch screens in general that I'd like to get
Speaker 1: all the way. One is that it's pretty ubiquitous today.
Speaker 1: It's it's it's a user interface that you find everywhere
Speaker 1: for everything from mobile or mobile devices to lots of
Speaker 1: different laptop and desktop displays. Now, granted, I've never owned
Speaker 1: a laptop or desktop display that had touch screen technology
Speaker 1: incorporated in it, largely because I didn't see a lot
Speaker 1: of use in that Based on how I tend to
Speaker 1: interact with computers, Not to say that there's no use
Speaker 1: for it, just that the way I use computers, it
Speaker 1: wouldn't make sense for me. Usually I don't have the
Speaker 1: display so close that reaching out and touching it would
Speaker 1: be terribly easy or comfortable, And most of what I
Speaker 1: use computers for requires lots of typing, which isn't great
Speaker 1: on most touch screen implementations. I guess you could pare
Speaker 1: it with voice recognition and get more use out of it,
Speaker 1: But I am curious if any of you guys use
Speaker 1: computers with touch displays and what do you use them for?
Speaker 1: As I'm sure there are plenty of use cases where
Speaker 1: it is incredibly handy so to speak. Oh and there
Speaker 1: will probably be a lot of unintentional puns in this episode,
Speaker 1: and maybe a couple of intended ones will be but
Speaker 1: a touch away, so to speak. Anyway, touch screens are everywhere,
Speaker 1: but their history is fairly recent. Another thing I find
Speaker 1: really interesting about them is that there are a lot
Speaker 1: of different ways to go about it, and the end
Speaker 1: result aims to be the same, but there are several
Speaker 1: approaches to implementing touch screens, and each implementation has its
Speaker 1: advantages and disadvantages, so we'll cover those in this episode.
Speaker 1: And yet, one more thing I think is interesting is
Speaker 1: really just how innovative touch screen devices have been. If
Speaker 1: you look back at the science fiction films and TV
Speaker 1: series from the nineteen fifties and even into the nineteen sixties,
Speaker 1: when touch screen technology was first being described, you'll rarely
Speaker 1: see examples of that idea. Touch Screens were such a
Speaker 1: leap forward that spec lative fiction writers weren't really imagining
Speaker 1: it as a user interface. That's why in series like
Speaker 1: Star Trek, the original series, you'll see characters interacting with
Speaker 1: physical dials and knobs and levers that are the controls
Speaker 1: of a twenty third century spaceship. You know, you look
Speaker 1: at those controls today and you think, oh, that looks antiquated.
Speaker 1: Unlike a lot of technologies we've seen over the last
Speaker 1: several decades, touch screens weren't heavily predicted in fiction. And
Speaker 1: I think I'll have to do an episode dedicated to
Speaker 1: tech that writers described years before it became a reality.
Speaker 1: That's an interesting subject of its own, Like the types
Speaker 1: of technology that science fiction writers predicted before it happened.
Speaker 1: You know, things like you know, geosynchronous satellites, but that's
Speaker 1: for another episode. Okay, so we're ready for our history lesson,
Speaker 1: which is you longtime listeners know is sort of a
Speaker 1: requirement in every tech stuff episode. Before I dive in,
Speaker 1: I want to give a shout out to Florence Ion's
Speaker 1: article from Touch to Plays to the Surface, A brief
Speaker 1: history of touch screen technology in Ours Technica. It's a
Speaker 1: fantastic summary of the evolution of the technology, and if
Speaker 1: you want to learn more about the history of touch screens,
Speaker 1: I urge you to seek it out. It was not
Speaker 1: the only source I used when getting all the history stuff,
Speaker 1: but it was a great resource. Also. I normally break
Speaker 1: up the history and the description of how technology works
Speaker 1: into different parts of episodes typically, but in this case,
Speaker 1: I think it works better to describe how each version
Speaker 1: of the technology works as we get to them, and
Speaker 1: as a peak behind the curtain. I came to that
Speaker 1: decision after I was about a third of the way
Speaker 1: done typing out all of my notes, so I actually
Speaker 1: went back and revised my notes quite a bit and
Speaker 1: rearranged things because I did not like the way the
Speaker 1: episode flowed in its original form. So all that being said,
Speaker 1: where did the idea for the touch screen interface come
Speaker 1: from well before there were touch screens that could interpret
Speaker 1: the touch of a finger or stylus, there was the
Speaker 1: light pen. The first light pen was part of a
Speaker 1: system that IBM design called Whirlwind, which the company built
Speaker 1: for Norad. But the way touch screens work is different
Speaker 1: from the way light pens work. With touch screens, the
Speaker 1: technology for detecting a point of contact is generally built
Speaker 1: into or behind or sometimes in front of the screen itself.
Speaker 1: With a light pen, the detector is actually in the
Speaker 1: pen side of the interface, not on the screen side,
Speaker 1: so the screen is a nert. It's the pen that's active.
Speaker 1: Light pens have a photoelectric cell built into them, in
Speaker 1: other words, a sensor that detects light, and typically light
Speaker 1: pen is tethered to the computer system it's connected to.
Speaker 1: It's actually physically connected with a cable. Holding a light
Speaker 1: pen up to a screen would allow the light pen
Speaker 1: to register when the monitors electron beams scanned across that point,
Speaker 1: because monitors in those days were based off the old
Speaker 1: cathode ray tube technology, which uses an electron gun that
Speaker 1: shoots a beam of electrons in row after row after row,
Speaker 1: so it goes it scans across the screen and then
Speaker 1: down the screen. So it goes one line across, then
Speaker 1: moves down the line, goes across, moves down the line, etcetera, etcetera.
Speaker 1: And these electrons then hit against phost four points on
Speaker 1: the back side of the screen and it generates light. Now,
Speaker 1: because the light pins were tethered to the computer system,
Speaker 1: the computer would pick up precisely where on the screen
Speaker 1: the light pin was sitting, and it did this by
Speaker 1: cross referencing the time of contact with the position of
Speaker 1: the electron beam at that moment. So the light pin
Speaker 1: would detect this electron beam, and that message would be
Speaker 1: sent to the computer, and the computer knew where the
Speaker 1: electron beam was at that precise instant, and that way
Speaker 1: it knew where the point of contact was. Now, this
Speaker 1: is largely an outlier of the touchscreen topic, but it's
Speaker 1: kind of a predecessor, so I thought it would include it. Now,
Speaker 1: unlike a lot of other technologies, which tend to get
Speaker 1: pretty muddy when you start asking questions like where did
Speaker 1: this idea come from? We can be reasonably certain that
Speaker 1: Eric Arthur Johnson or E. A. Johnson proposed the first
Speaker 1: technological solution to creating a touch screen computer interface. Johnson
Speaker 1: was an engineer at what was at that point the
Speaker 1: Royal Radar Establishment, which was a research facility in Malvern, England,
Speaker 1: and as the name suggests, this facility was chiefly focused
Speaker 1: on developing new radar technologies, and it wasn't It was
Speaker 1: operating as a research organization that worked closely with British
Speaker 1: Armed Forces. It would later become the Defense Evaluation and
Speaker 1: Research Agency after merging with a few other organizations, and
Speaker 1: later still it would become part of a defense contractor
Speaker 1: in England called Kinetic spelled with a que. Johnson was
Speaker 1: working on improving the user interface for air traffic control staff.
Speaker 1: He wrote an article and had the title touch Display
Speaker 1: a novel input output device for computers. It was published
Speaker 1: in the journal Electronics Letters on October uh in nineteen.
Speaker 1: I said on October, I should say in October nine,
Speaker 1: because I don't have the precise date of when in
Speaker 1: October it came out. But in nineteen sixty seven he
Speaker 1: published a follow up piece titled touch Displays a Programmed
Speaker 1: Man Machine Interface that further developed this concept and fleshed
Speaker 1: it out, and he was describing what would become the
Speaker 1: capacitive touch screen interface. And I also find this interesting
Speaker 1: because for many years, it seemed the majority of consumer
Speaker 1: products that had touch screens used an alternative to the
Speaker 1: capacity of approach, known as resistive touch screens, and those
Speaker 1: two technologies make up the majority of the touch screens
Speaker 1: we tend to encounter. I'll explain the differences between them
Speaker 1: when we get to each, but first before I get
Speaker 1: into the differences, what our capacity of touch screens and
Speaker 1: how do they register touch? Well, they only register a
Speaker 1: touch if the substance touching the screen can hold an
Speaker 1: electrical charge, So stuff like our skin. Our skin can
Speaker 1: hold an electrical charge. It's electrically conductive. So if you've
Speaker 1: ever used a touch screen while wearing gloves and nothing happened,
Speaker 1: it's probably because the capacity of touch screen was not
Speaker 1: able to detect any sort of electrical connection. The gloves
Speaker 1: were acting like insulators. They inhibit electrical charge. That's why
Speaker 1: there are companies out there that sell gloves that have
Speaker 1: conductive wire or conductive pads at the fingertips. And it's
Speaker 1: also why you can't use something like an inert plastic
Speaker 1: stylus on a capacity of screen. You can use a
Speaker 1: stylus that has a conductive surface at the hip, that
Speaker 1: would work. But if it's just a plastic stick, for example,
Speaker 1: it wouldn't activate the screen. But you could use something
Speaker 1: else like um, you know, like a hot dog, which
Speaker 1: is I understand it used to be a thing in
Speaker 1: South Korea. People would use hot dogs to activate their
Speaker 1: capacitive touch screens when it was too cold for them
Speaker 1: to not wear gloves, which tells me you could probably
Speaker 1: make a killing in soul by selling screen cleaners dedicated
Speaker 1: to eradicating weener grease from your screens. But how do
Speaker 1: the capacity of screens actually detect touch. There's a couple
Speaker 1: of different approaches, but the general idea is to create
Speaker 1: a surface that holds an electrical charge, and in many
Speaker 1: implementations this is done with a grid of very fine
Speaker 1: conductive wires running in rows and columns on an x
Speaker 1: y grid. In other words, so like a net, sensors
Speaker 1: pick up changes in this electrical charge when they happen.
Speaker 1: So if an object act that conducts electricity makes contact
Speaker 1: with the screen, so for example, your finger, there's a
Speaker 1: change in that electrical charge. Technically the change is a
Speaker 1: drop in voltage. So by detecting where that change happens
Speaker 1: along those x y coordinates, A microprocessor can interpret the
Speaker 1: touch and associate it with whatever command you wanted to execute. So,
Speaker 1: in the example of activating an icon on a screen,
Speaker 1: the icon represents the execution of a particular app or program.
Speaker 1: When the microprocessor or touch sensor detects a contact at
Speaker 1: the location of such an icon, it interprets that touch
Speaker 1: to mean execute the program associated with the image at
Speaker 1: these coordinates on the display, and then it will launch
Speaker 1: the app. A similar thing happens with gesture controls like
Speaker 1: swiping or pinching. Engineers and programmers have to build in
Speaker 1: this capability so that the system can interpret the meaning
Speaker 1: behind the gestures and thus produced the appropriate result, But
Speaker 1: the actual detection of a touch all comes down to
Speaker 1: that change and voltage. Early capacity of screens could really
Speaker 1: only detect one point of contact, so if you tried
Speaker 1: touching the screen with more than one finger, typically the
Speaker 1: screen would only register that first touch. This is because
Speaker 1: the sensing technology was limited. Early sensors would estimate the
Speaker 1: point of contact. It wasn't incredibly precise. It was precise
Speaker 1: enough for general use, but you couldn't really get fine
Speaker 1: tuning with it. Later implementations would incorporate better sensors, and
Speaker 1: eventually you'd find capacity of screens in which each row
Speaker 1: or column of wires had its own associated sensors, which
Speaker 1: increased their accuracy, and it opened up the possibility of
Speaker 1: a capacity of screen with multi touch capability. Johnson would
Speaker 1: receive a U S patent for his invention in nineteen
Speaker 1: sixty nine. So if you'd like an engineer's explanation of
Speaker 1: the basic technology behind these capacity of touch screens, you
Speaker 1: two can search for u S Patent three million, four
Speaker 1: hundred eighty two thousand, two hundred forty one. The patent
Speaker 1: includes circuit diagrams and a flow chart that are helpful
Speaker 1: to understand how it works as well. The capacity of
Speaker 1: screen was a great innovation, but it saw little adoption
Speaker 1: over the following few years. An alternative approach would get
Speaker 1: a bit more traction. In the short term. You could
Speaker 1: say the tech world couldn't resist it. I'll explain more
Speaker 1: after this break in nineteen seventy, Dr George Samuel Hurst
Speaker 1: invented the first resistive touch screen. In the late nineteen
Speaker 1: forties through the nineteen fifties, he had worked at the
Speaker 1: oak Ridge National Laboratory and R and D facility that
Speaker 1: is funded by the U. S Department of Energy. Herst
Speaker 1: has been working in the field of atomic physics, developing
Speaker 1: stuff like radiation detectors at nuclear testing sites. In nineteen
Speaker 1: sixty six, he accepted a job as professor of physics
Speaker 1: at the University of Kentucky. While in that role, he
Speaker 1: continued doing research into atomic physics, but his team was
Speaker 1: running into some obstacles, particularly in the use of a
Speaker 1: vander graphic accelerator, which is an electro static generator that
Speaker 1: can be used as a particle accelerator. To quote the Minerals,
Speaker 1: Metals and Materials Society, which has a PDF about these devices, quote,
Speaker 1: a high potential difference is built up and maintained on
Speaker 1: a smooth conducting surface by the continuous transfer of positive
Speaker 1: static charges from a moving belt to the surface. When
Speaker 1: used as a particle accelerator, and ion source is located
Speaker 1: inside the high voltage terminal. Ions are accelerated from the
Speaker 1: source to the target by the electric voltage between the
Speaker 1: high voltage supply and ground. Now that sounds complicated, but
Speaker 1: it's essentially a vandergraph generator and you've probably seen one
Speaker 1: of these, maybe non in person, but probably at least
Speaker 1: in a picture or video. They typically look like silver
Speaker 1: orbs that can give off a spectacular spark when they operate.
Speaker 1: There's typically a large silver orb on a pedestal, and
Speaker 1: then there's a smaller silver orb that's located a certain
Speaker 1: distance away from the large one, and when you turn
Speaker 1: it on, inside that pedestal, there is a belt that's
Speaker 1: running in a loop, and the belt is essentially building
Speaker 1: up a positive charge inside that large silver orb, and
Speaker 1: when the the voltage difference is enough between the large
Speaker 1: silver orb and the small silver orb, it will create
Speaker 1: a spark between the two. It can be really really spectacular.
Speaker 1: Now I'll have to do a full episode on those
Speaker 1: in the future. Let's get back to touch screens. So
Speaker 1: Hearst was working with others on his team to come
Speaker 1: up with what they called an electrically conductive paper in
Speaker 1: order to work with these vander Graph accelerators and to
Speaker 1: make their notes more um efficient. The paper would be
Speaker 1: able to pinpoint contact, and when mapped to an x
Speaker 1: Y coordinate system, could be used to specify a particular
Speaker 1: location of contact. So Herst thought, wait a minute, this
Speaker 1: could be used as an interface for computers, not just
Speaker 1: for registering a specific point in space for the purposes
Speaker 1: of research. So Heirst returned to work for the oak
Speaker 1: Ridge National Laboratory in ninety and he refined his idea.
Speaker 1: He worked with nine other Eggheads to create the first
Speaker 1: resistive touch screen. Okay, so, a resistive touch screen has
Speaker 1: a couple of layers, one of which is conductive and
Speaker 1: the other of which is resistive, meaning it resists the
Speaker 1: flow of electrons through the material, and separating those two
Speaker 1: layers are small spacers. Spacers are essentially little blocks of
Speaker 1: non conductive material. They act as as support structure. They
Speaker 1: keep the two layers from being in contact with one another,
Speaker 1: and there's also usually a scratch resistant layer on top
Speaker 1: of the surface that faces the user. Because using a
Speaker 1: resistive screen touch screen means that you're actually having to
Speaker 1: apply pressure on the touch screen. You're not just touching it,
Speaker 1: you're actually pressing it. So when you press a resistive screen,
Speaker 1: you apply that little bit of pressure. The conductive and
Speaker 1: resistive layers move closer together, they're flexible, and eventually they
Speaker 1: touch each other. Now both layers have an electrical current
Speaker 1: running through them, and when they make contact, the electrical
Speaker 1: field changes, and sensors and a microprocessor detect and analyze
Speaker 1: that point of contact and register it so that the
Speaker 1: device does whatever it is you wanted to do, from
Speaker 1: allowing you to make a digital signature to executing a command. Now,
Speaker 1: unlike a capac poet of screen, resistive screens don't require
Speaker 1: the point of contact to come from an electrically conductive material.
Speaker 1: A resistive screen doesn't care if the thing touching the
Speaker 1: screen is your finger, or it's a hot dog wiener,
Speaker 1: or a plastic stylus or a rock or whatever it
Speaker 1: is that's applying the pressure. All the electrical activity is
Speaker 1: contained within those layers that make up the outer part
Speaker 1: of the screen. So you can operate a device with
Speaker 1: a resistive touch screen even if you're wearing non conductive gloves.
Speaker 1: And if the screen were to get a little wet,
Speaker 1: you never want your electronics to really get wet. But
Speaker 1: let's say a little water gets on it, it wouldn't
Speaker 1: affect the performance. That's different from a capacitive screen. If
Speaker 1: you've ever had a smartphone get a little bit of
Speaker 1: water on it. Let's say it's a very light rain
Speaker 1: and you're trying to use your smartphone, you might have
Speaker 1: encountered some problems with it. Well. Capacitive touch screens don't
Speaker 1: work so well when they get a little water on them.
Speaker 1: It messes with this ability to detect the actual point
Speaker 1: of contact with a conductive surface. So resistive screens don't
Speaker 1: have that issue, although you still shouldn't really operate them
Speaker 1: in the rain. Now that being said, resistive touch screens
Speaker 1: tend to be harder to read and to see what's
Speaker 1: on the screen. They require more layers than a capacity
Speaker 1: of screen does, and they tend to reflect a lot
Speaker 1: more ambient light than capacitive screens. Plus, while they are
Speaker 1: pretty hardy, they do rely on detecting pressure, and depending
Speaker 1: on how hard people are pushing while they're trying to
Speaker 1: use these things, it can cause some wear and tear
Speaker 1: on the device. If the spacers that separate those two
Speaker 1: screens get damaged, then the screen can end up having
Speaker 1: points of contact before you've even touched it, which sends
Speaker 1: erroneous signals to the microprocess or it doesn't actually know
Speaker 1: where you're trying to touch it because it's getting conflicting information. Also,
Speaker 1: they were limited to detecting a single point of contact
Speaker 1: which eliminated the possibility for a multi touch Still, because
Speaker 1: they could stand up to a lot of punishment and
Speaker 1: they could work in different environments, they found a lot
Speaker 1: of applications in different technologies, particularly in stuff like military tech. Now,
Speaker 1: just one year after Hurst's resistive touch screen approach made
Speaker 1: the news, the University of Illinois introduced the PLATO for system.
Speaker 1: PLATO stands for Programmed Logic for Automatic Teaching Operations. One
Speaker 1: of the components of this system was an orange plasma
Speaker 1: display with touch screen capability, But unlike the previous inventions,
Speaker 1: this approach relied upon an infrared touch panel. All right, well, then,
Speaker 1: how does an infrared touch panel work. Well, imagine that
Speaker 1: you have an array of l e ED s that
Speaker 1: emit light in the infrared spectrum, so they're like tiny
Speaker 1: little infrared flashlights. Infrared is outside the visible spectrum of light.
Speaker 1: So to us it would seem as if an LED
Speaker 1: light was off because we can't see that light. But
Speaker 1: in fact, it would be beaming this infrared light across
Speaker 1: the surface of a screen, and on the other side
Speaker 1: of the beam would be a photo cell. So, in
Speaker 1: other words, a light sensor, and it's a sensor at
Speaker 1: tuned specifically to detecting that frequency of infrared light that
Speaker 1: it was paired with. So if you could see these lights,
Speaker 1: it would look like a laser grid, kind of like
Speaker 1: admission impossible or Tom Cruise is coming down from the ceiling.
Speaker 1: It's pretty awesome scene. But that's what infrared touch screen
Speaker 1: would look like if you could see those infrared beams. Now,
Speaker 1: if the beam remains unbroken, then it's clear there's no
Speaker 1: contact with the screen. If the sensors keep on picking
Speaker 1: up the light, they just say, all right, nothing's touching.
Speaker 1: But if something that blocks the light that's between the
Speaker 1: l e ED that's emitting the light and the photo cell,
Speaker 1: that interruption would indicate that something has touched the screen.
Speaker 1: And by arranging the l e d s and the
Speaker 1: photo cells and having them paired up in columns and
Speaker 1: in rows as a grid system, then you have a
Speaker 1: whole net of those invisible beams. Touching a point on
Speaker 1: the screen would interrupt both horizontal and vertical beams along
Speaker 1: the surface, So a microprocessor could detect which photo cells
Speaker 1: had registered the interruption and then plot the point where
Speaker 1: that happened. So it's very similar to plotting a point
Speaker 1: on a grid in math class. Like the resist of screens,
Speaker 1: this approach had the benefit of working with any light
Speaker 1: blocking material. It did not have to be electrically conductive,
Speaker 1: so a plastic stylus works just as well as a
Speaker 1: finger if the technology is implemented properly. Also, there's no
Speaker 1: need to work in thin metallic wires in the screen itself,
Speaker 1: because the screen is not what's detecting the touch. It's
Speaker 1: this laser grid essentially not really lasers, but this led
Speaker 1: grid with the photo cells, so the screen wouldn't have
Speaker 1: any wires in it. It would be brighter and provide
Speaker 1: more clarity than early capacity and resist of versions could.
Speaker 1: That was a big advantage. The infrared approach would see
Speaker 1: use in the nineteen eighties in the HP one fifty.
Speaker 1: That was a computer system from well then Hewlett Packard
Speaker 1: before they became just HP and it costs the princely
Speaker 1: sum of two thousand, seven hundred ninety five dollars in
Speaker 1: ninety three, but if we have adjust that for inflation,
Speaker 1: that means in today's money it would cost about seven thousand,
Speaker 1: two hundred dollars. Yikes. The HP one fifty version, or
Speaker 1: portly had some issues that that made it less practical.
Speaker 1: So I imagine that, uh, the fact that it wasn't
Speaker 1: working perfectly and that the price tag was so high
Speaker 1: meant it didn't get a whole lot of traction in
Speaker 1: the market. Later on, devices like the Sony E reader
Speaker 1: would actually adopt this technology. Now around the same time
Speaker 1: that the infrared system debuted in the mid nineteen seventies,
Speaker 1: g why Zapp Finel and Chris Herold of the Architecture
Speaker 1: Machine Group at M I T. And I know I
Speaker 1: butchered their names. I apologize anyway, they created a touch
Speaker 1: screen interface that could detect not just touch but also pressure.
Speaker 1: I mean it wasn't like a resistive screen in that sense.
Speaker 1: It could actually detect how much pressure you were applying
Speaker 1: to the screen. In fact, the system could detect up
Speaker 1: to eight different signals from a single touch point, including torque,
Speaker 1: which meant you could push your finger on the screen
Speaker 1: and you add some pressure to it and then twist
Speaker 1: your finger, and the interface could detect that you were
Speaker 1: making this twisting motion, and you could have that imagined
Speaker 1: as some sort of effect in a program. What kind
Speaker 1: of effect would depend entirely on the programming, so there's
Speaker 1: no specific application, but it could be used for all
Speaker 1: sorts of different stuff. So they published their work in
Speaker 1: nine with the title one point Touch Input of Vector
Speaker 1: Information from Computer Displays, and it was published in the
Speaker 1: Computer Graphics Periodical, So if you want to read up
Speaker 1: on that, you can. There's also illustrations of how it worked.
Speaker 1: There's also a YouTube video of a demonstration as a
Speaker 1: very early demonstration of this nineteen seventies era technology. But
Speaker 1: it's pretty fascinating to see at work. And we're not
Speaker 1: done with the different ways to achieve touch screen functionality.
Speaker 1: There's still some more to chat about. Engineer Nimish Meta
Speaker 1: developed a solution for the first multi touch system in
Speaker 1: nineteen two decades before the iPhone. Now it's important to
Speaker 1: note that this was not so much a touch screen
Speaker 1: as it was a control interface like a touch pad,
Speaker 1: not a touch screen, so think like a keyboard and
Speaker 1: mouse or something along those lines. And it consisted of
Speaker 1: a pane of glass with a translucent layer of plastic
Speaker 1: on that glass, giving it sort of a frosted appearance.
Speaker 1: There was a camera mounted below or behind the glass pane,
Speaker 1: and that would detect points of contact. Essentially, it was
Speaker 1: looking for dark spots to appear on that surface. That
Speaker 1: would indicate that a finger was there blocking the light.
Speaker 1: This isn't that different from what Microsoft would use on
Speaker 1: the surface tables a couple of decades later. I'll chat
Speaker 1: about that in a second. So in this case, the
Speaker 1: camera would detect these dark spots and through software, the
Speaker 1: system would be able to interpret where those points of
Speaker 1: contact were in relation to what was being displayed on
Speaker 1: a screen. One benefit of this approach was that you
Speaker 1: weren't actually making contact with the same surface you were
Speaker 1: looking at, so you weren't smearing your grubby hands all
Speaker 1: over the same surface you were trying to read. But
Speaker 1: then again, you could argue the whole purpose of creating
Speaker 1: a touch screen interface is to remove the barrier between
Speaker 1: humans and the machines they're working on and make the
Speaker 1: experience more intuitive. We have to learn how to use
Speaker 1: things like a computer mouse or a keyboard. It's not
Speaker 1: hard to learn, but it does mean manipulating something along
Speaker 1: one surface while looking at another. So, for example, with
Speaker 1: a traditional computer, you would use a keyboard and a
Speaker 1: mouse on a plane that's at a ninety degree angle
Speaker 1: with the display you're looking at right. So if you
Speaker 1: think of horizontal and vertical. Your hands are manipulating objects
Speaker 1: on a horizontal plane while you're looking at the reactions
Speaker 1: on a vertical plane. So manipulating the mouse to move
Speaker 1: a cursor requires your brain to translate the motion along
Speaker 1: one axis of movement to a display that's on a
Speaker 1: different axis. Now, once we learn how to do this,
Speaker 1: and admittedly it does not take very long, it becomes
Speaker 1: second nature, so it's not a big deal. But a
Speaker 1: touch screen removes that necessity entirely because the thing you're
Speaker 1: looking at and the thing you're manipulating is the same surface.
Speaker 1: Ine Myron Krueger introduced another input method that wasn't strictly
Speaker 1: a touch screen, but is similar enough to merit in collusion.
Speaker 1: In this episode, Krueger's system could track a user's hands.
Speaker 1: It was a vision based system, meaning it employed cameras
Speaker 1: to detect and track hand motion and poses, so I
Speaker 1: could detect multiple hands so impaired with the proper software,
Speaker 1: it could translate the actions of those multiple hands into
Speaker 1: commands for a program. But in this case, the system
Speaker 1: would respond to what is called dwell time. Hand gestures
Speaker 1: or poses would correspond to specific commands, and a user
Speaker 1: would have to hold his or her hands within view
Speaker 1: of this system long enough for it to register that
Speaker 1: it was in fact a signal to do something. Well,
Speaker 1: this wasn't directly related to touch screen technology, it is
Speaker 1: important in the history of gestural interaction, which does intertwined
Speaker 1: with touch screen technology quite a bit. A lot of
Speaker 1: our interactions with touch screens depend upon not just points
Speaker 1: of contact, but specific gestures swiping, pinching, that kind of thing,
Speaker 1: and I figured should at least touch on Krueger's work.
Speaker 1: Krueger wrote several books on technology that are pretty fascinating.
Speaker 1: I feel I should probably dedicate a full episode to
Speaker 1: him at some point, and we're not finished yet, so
Speaker 1: when we come back, I'll talk more about multi touch
Speaker 1: systems that actually did rely on making contact with a screen.
Speaker 1: But first let's take another quick break. So the first
Speaker 1: screen to feature multi touch isn't the surface. It's not
Speaker 1: the iPhone, though you wouldn't necessarily know that based upon
Speaker 1: the marketing around those devices. A Bell Labs researcher named
Speaker 1: Bob Boy created the first multi touch capacitive screen in Actually,
Speaker 1: to be fair, it really was an array of capacitive
Speaker 1: touch sensors that were mounted on a transparent film that
Speaker 1: could be added as an overlay on top of a
Speaker 1: CRT monitor, So monitor itself wouldn't have the touch screen
Speaker 1: built into it. It would actually be a peripheral you
Speaker 1: could add to the monitor. This prototype never emerged out
Speaker 1: of the lab for broader application, but it was able
Speaker 1: to register the touch of multiple points of contact, and
Speaker 1: thus you could create applications that would allow for that
Speaker 1: and to you know, create new commands for how this
Speaker 1: could work. And there are a couple of other approaches
Speaker 1: to multi touch. One demonstrated by Jeff Hahn in two
Speaker 1: thousand and six, used a rear projection system, a sheet
Speaker 1: of acrylic and an LED that created frustrated total internal
Speaker 1: reflection or f t i R, which sounds to me
Speaker 1: like meditating a self discovery only to find out you're
Speaker 1: actually a total jerk. But that's not what it actually means.
Speaker 1: So to get into the nitty gritty of the technology
Speaker 1: is more than a little bit complicated, but I'll give
Speaker 1: it a shot from a very high level. So imagine
Speaker 1: you have a sheet of really clear material like acrylic. Okay,
Speaker 1: so you've got a sheet of acrylic. Now imagine that
Speaker 1: we're looking at this sheet of acrylic from a side
Speaker 1: edge right, so the top surface is uh is and
Speaker 1: the bottom surface are facing you know, up and down.
Speaker 1: From our perspective, we're just looking at it from the side,
Speaker 1: and you've got a bunch of infrared LEDs mounted on
Speaker 1: either end of this acrylic sheet. Below the acrylic sheet
Speaker 1: and facing upwards is an infrared camera. So the control
Speaker 1: surface in this case would be above the sheet from
Speaker 1: your perspective. The camera is below the sheet from your perspective.
Speaker 1: So total internal reflection gives you a hint at what's
Speaker 1: actually at play here. Those L E D s are
Speaker 1: beaming light, infrared light into the edge of the acrylic
Speaker 1: at a specific angle it's called the critical angle, which
Speaker 1: results in the beam reflecting perfectly within the acrylics. So
Speaker 1: imagine that on one side you have the beam UH
Speaker 1: position in such a way that it's angled downward from
Speaker 1: your perspective. The beam goes down, hits the inner bottom
Speaker 1: edge of the acrylic, bounces up with no refraction. It's
Speaker 1: it's a perfect reflection, and then encounters the upper edge
Speaker 1: from your perspective, bounces off that again perfectly reflected, and
Speaker 1: does so all through the entire length of this acrylic sheet. Now,
Speaker 1: if you could see the beams of infrared light, you
Speaker 1: would see how they were criss crossing around inside this
Speaker 1: acrylic bouncing off those inner surfaces of either face of
Speaker 1: the sheet. And it happens because physics. I mean, it
Speaker 1: gets more complicated than that. But if I were to
Speaker 1: jump into it, I would have to talk about Snell's
Speaker 1: law and the refractive index, and honestly, it would get
Speaker 1: super complex and it would be hard to describe without
Speaker 1: the use of visual aids. So I'm just gonna take
Speaker 1: a shortcut in this case and just say it works
Speaker 1: because of physics. Anyway. The result is you have these
Speaker 1: perfectly reflective beams of infrared light bouncing around inside this
Speaker 1: acrylic sheet. But if you were to touch the surface
Speaker 1: of the sheet on the active side, the top side,
Speaker 1: you frustrate this total internal reflection. Some of that light
Speaker 1: that was being reflected inside the acrylic sheet at the
Speaker 1: point of contact can pass from the surface to your finger,
Speaker 1: so reflection is no longer total. And the infrared count
Speaker 1: camera that's mounted beneath the sheets, pointed up at it
Speaker 1: will detect that change in the reflectivity at the point
Speaker 1: of contact, registering it as a touch, and this system
Speaker 1: can detect multiple points of contact on the same surface,
Speaker 1: so it is a multi touch approach. In two thousand seven,
Speaker 1: Microsoft showed off a table sized computer system that it
Speaker 1: called the Surface. Since then, Microsoft has used the name
Speaker 1: Surface for some of its other product us, largely the
Speaker 1: tablet style computers. But the early version of the Surface
Speaker 1: was much much larger, and it was a collaborative workspace
Speaker 1: where multiple people could stand around this interactive table, the
Speaker 1: top of which was a computer display, and it could
Speaker 1: detect multiple points of contact on that computer display. You
Speaker 1: could manipulate virtual objects, you could play games, you could
Speaker 1: do a lot of different stuff. The Surface worked using
Speaker 1: some of the methods I've already mentioned in this episode.
Speaker 1: Inside the table was a projector, and the projector was
Speaker 1: projecting the images that you would see on the actual surface,
Speaker 1: So what you were looking at was really a projection
Speaker 1: being shot against the backside of the screen you were
Speaker 1: looking at. So, in other words, the Surfaces screen was
Speaker 1: what we would call a rear projection screen, very much
Speaker 1: like rear projection televisions. Also inside the Surface were cameras
Speaker 1: that could detect the points of contact on the opp
Speaker 1: outside of the screen on your side. In other words,
Speaker 1: Microsoft also designed a program that could recognize patterns that
Speaker 1: were printed on special stickers. Then they could put those
Speaker 1: stickers onto solid objects. So if you place one of
Speaker 1: those small objects on the surface, the camera underneath would
Speaker 1: be able to recognize the pattern on that sticker and
Speaker 1: then execute an associated command, which could be anything. But
Speaker 1: one version of this one version I saw was that
Speaker 1: you could have sort of a synthesizer application, one that
Speaker 1: could play pre rendered styles of music, and each sticker
Speaker 1: would represent maybe a specific tone or a sound pattern
Speaker 1: or a sound effect, and by arranging a series of
Speaker 1: these objects on the surface, you could build a sound
Speaker 1: So you could create a series of sounds, like in
Speaker 1: a particular rhythm. By manipulating these objects and changing the
Speaker 1: location on the surface might do things like change the
Speaker 1: pitch or the volume of each sound. So you would
Speaker 1: have this interactive kind of music surface to work with.
Speaker 1: And that's just one example of what you could do
Speaker 1: with this type of technology. There were lots of potential applications.
Speaker 1: Microsoft would actually bring that version of the Surface to
Speaker 1: c E. S. Two thousand eight, but the company was
Speaker 1: also quick to say that the technology wasn't actually consumer tech. Rather,
Speaker 1: this was technology that businesses would be able to purchase
Speaker 1: for their own purposes. So you might have it in
Speaker 1: a retail establishment, you might have it in an entertainment establishment.
Speaker 1: One of the versions I heard about was being used
Speaker 1: in a Las Vegas bar where you could play games
Speaker 1: on the table, or you could use your table to
Speaker 1: send messages to other people around the bar on their tables,
Speaker 1: which kind of skeeths me out a little bit. But
Speaker 1: then again, I'm not a bar person, so maybe I'm
Speaker 1: just the wrong kind of guy to look into that
Speaker 1: sort of thing. It just seems like another way to
Speaker 1: kind of harass people without them, you know, wanting it.
Speaker 1: Who am I to say? The same year that Microsoft
Speaker 1: first demonstrated the surface, Apple introduced the iPhone, and again,
Speaker 1: while Apple didn't invent capacitive touch or even capacity of
Speaker 1: multi touch, heck, even the gestures associated with gestural interaction
Speaker 1: on the iPhone were already described by other people in
Speaker 1: other systems, but the packaging of all of those features
Speaker 1: in a sleek smartphone form factor wowed the crowds. The
Speaker 1: iPhone brought touchscreen technology into the spotlight for lots of people,
Speaker 1: when earlier it had been a type of user interface
Speaker 1: that really only applied to electronics in niche applications and implementations.
Speaker 1: The iPhone was not the first consumer gadget to rely
Speaker 1: on touch screen interactions, but I think it's safe to
Speaker 1: say that Apple got it so right that it changed
Speaker 1: the game for everyone, and it became the go to
Speaker 1: interface for mobile handheld electronics. Other than the methods I've
Speaker 1: already covered, there are a couple of more rare forms
Speaker 1: of touch screens out there. One is the surface acoustic
Speaker 1: wave touch screen. Now, as that name implies, this version
Speaker 1: of a touch screen relies on sound, specifically sounds that
Speaker 1: are in the ultrasonic frequencies. Those are at such a
Speaker 1: high frequency range that they are imperceptible to us, we
Speaker 1: cannot hear them. Ultrasonic speakers would be along the edge
Speaker 1: of the screen that would emit these high pitched sound waves,
Speaker 1: and those sound waves reflect back and forth across the surface,
Speaker 1: kind of like waves go across the water, and when
Speaker 1: something would come into contact with the screen, it would
Speaker 1: disrupt the path of those waves. And again it would
Speaker 1: be a lot like if something large were to get
Speaker 1: into a a wavy pool of water. Now, with water,
Speaker 1: the waves are are really big, particularly compared to ultrasonic frequencies,
Speaker 1: and that kind of makes it a little hard to
Speaker 1: see what the effects are in this interrupted path system.
Speaker 1: But it's much easier to see the change with ultrasonic
Speaker 1: waves because they are so tiny, and sensors detect the
Speaker 1: point of interruption to determine where you touched the screen. So,
Speaker 1: in other words, they detect where are the waves no
Speaker 1: longer able to travel unimpeded, and that is clearly the
Speaker 1: point of contact. They're also touch screens called near field
Speaker 1: imaging touch screens. These screens have technology that monitors the
Speaker 1: electromagnetic field on the glass screen, and when something comes
Speaker 1: close to the screens surface, it interferes with that electromagnetic field,
Speaker 1: and the system detects that and interprets it as a touch.
Speaker 1: These sorts of screens can also be pretty rugged, and
Speaker 1: so they are frequently used for stuff like industrial or
Speaker 1: military applications. And there we are. That's the history and
Speaker 1: operation of touch screens. Uh. It's pretty complicated because, like
Speaker 1: many other technologies, there were a lot of people taking
Speaker 1: many different approaches, all in an effort to achieve similar goals,
Speaker 1: and some of what I've described has also been used
Speaker 1: in other types of interfaces that don't involve a touch
Speaker 1: screen at all, such as the gesture controls used in
Speaker 1: systems like the Microsoft Connect peripheral for Xbox systems. Now,
Speaker 1: I think it's safe to say that the Connect was
Speaker 1: largely a failed experiment. But I don't think it was
Speaker 1: because it didn't work, because for the most part it
Speaker 1: did work. Now, there were some rather egregious exceptions to
Speaker 1: that rule, but for the most part it worked. Rather,
Speaker 1: I think it failed because the system never evolved beyond
Speaker 1: a gimmick or oddity in the eyes of most owners. Uh.
Speaker 1: You could argue a large reason for that was just
Speaker 1: a lack of very compelling content in the library of
Speaker 1: games and applications that supported Connect interactivity. Still, the Connect
Speaker 1: relied on a lot of work that was being done
Speaker 1: in the field of touch screen user interfaces and just
Speaker 1: your controls, So while it's not a direct application of
Speaker 1: the technology, it is definitely related to it. Likewise, there
Speaker 1: have been several systems for everything from virtual environments to
Speaker 1: art installations that have used similar technologies to some touchscreen implementations.
Speaker 1: Most of these have been visually or optically based, so
Speaker 1: in other words, they use cameras to track the gestures, motions,
Speaker 1: and poses of people within a physical space in order
Speaker 1: to create some sort of effect. You may have been
Speaker 1: one of these installations or or applications where your movements
Speaker 1: through the space are reflected in some way. Maybe it's
Speaker 1: a video effect, maybe it's sound. But a lot of
Speaker 1: that also has related technologies to the ones that went
Speaker 1: into developing touch screens. Now, considering the ubiquity of mobile devices,
Speaker 1: I expect will continue to see advancements in touch screen
Speaker 1: technology over the years. It may involve new approaches to
Speaker 1: achieving the results, or it may involve refined implementation of
Speaker 1: existing approaches to improve the overall experience. And I'm not
Speaker 1: sure if it will translate to all our electronics. I
Speaker 1: think there are some implementations where touch screens make a
Speaker 1: lot of sense, and in others maybe not so much.
Speaker 1: Like I'm still curious if people with desktop or laptop
Speaker 1: displays that include touch sensitivity really use that feature all
Speaker 1: that often. I mean, maybe they do. I'm only basing
Speaker 1: this off my own anecdotal evidence, which obviously is limited
Speaker 1: and therefore largely worthless. But it's hard for me to
Speaker 1: imagine using a touch screen laptop or our desktop display regularly.
Speaker 1: In fact, when i use my Microsoft Surface tablet as
Speaker 1: a laptop, because I've got all the connected keyboard I
Speaker 1: can use with it. When I'm using it, like in
Speaker 1: that form factor, I totally forget that the screen actually
Speaker 1: has touch capability, that I could reach out and touch
Speaker 1: things on the screen instead of using the mouse pad. Uh.
Speaker 1: But also I have to admit, as Tori will tell
Speaker 1: you without a moment's hesitation, I'm old, and so it's
Speaker 1: entirely possible that I'm the odd man out here. I
Speaker 1: do think it's true that even when an interface works,
Speaker 1: and it works well, it's not necessarily the best interface
Speaker 1: for everything. So if I'm not the odd one out
Speaker 1: and most people find touch screens unnecessary for laptops or desktops,
Speaker 1: maybe we won't seem as many of those types of
Speaker 1: devices with that feature included in the future. Kind of
Speaker 1: like how televisions for a while all had three D capability,
Speaker 1: and then people said, I don't want three D. I
Speaker 1: don't care for it. It's too irritating. And now if
Speaker 1: it is a feature, it's rarely one of the main
Speaker 1: ones mentioned on the box. For those televisions we might
Speaker 1: see the same thing with the touch screen text for
Speaker 1: certain types of electronics, but for things like mobile devices,
Speaker 1: it totally makes sense, and I expect we will continue
Speaker 1: to see uh it used there and improve in that implementation.
Speaker 1: And that wraps up this discussion about touch screens. Obviously
Speaker 1: I could have gone into a lot more detail about
Speaker 1: each of those, but that would have required a whole
Speaker 1: mini series on it, and honestly, I'm not sure that
Speaker 1: that I could do all of that without losing my mind.
Speaker 1: So we're gonna wrap up this episode. If you guys
Speaker 1: have suggestions for future episodes, you can write me the
Speaker 1: email addresses tech Stuff at how stuff works dot com.
Speaker 1: You can drop by the website that's tech stuff podcast
Speaker 1: dot com. There you're gonna find an archive of all
Speaker 1: of our shows, including the two thousand nine episode where
Speaker 1: I first talked about touch screens with my co host
Speaker 1: Chris Palette, as well every other episode of tech Stuff.
Speaker 1: You'll also find links to our social media presence and
Speaker 1: a link to our online store, where every purchase you
Speaker 1: make goes to help the show, and we greatly appreciate it,
Speaker 1: and I will talk to you again really soon. Text
Speaker 1: Stuff is a action of i heart Radio's How Stuff Works.
Speaker 1: For more podcasts from I heart Radio, visit the i
Speaker 1: heart Radio app, Apple Podcasts, or wherever you listen to
Speaker 1: your favorite shows.