Speaker 1: Get in touch with technology with tech Stuff from how
Speaker 1: stuff works dot com. Hey there, and welcome to tech Stuff.
Speaker 1: I'm your host, Jonathan Strickland. I'm an executive producer at
Speaker 1: how stuff Works and I love all things tech. And
Speaker 1: in the last episode, I covered the history and technology
Speaker 1: behind speech recognition. So today we're going to look at
Speaker 1: a related concept called natural language processing or natural language understanding.
Speaker 1: The two are are related. This technology and speech recognition
Speaker 1: are both part of what make voice assistants like Sirie,
Speaker 1: Alexa and Google Assistant work, though there are other technologies
Speaker 1: that also go into that. Now, this is a huge
Speaker 1: topic and as a long and fascinating history, so this
Speaker 1: episode is just going to be the start of it.
Speaker 1: In the next episode, I will conclude a discussion on
Speaker 1: natural language processing and go into the history of these
Speaker 1: actual voice assistants. So, on a high level, what is
Speaker 1: natural language processing? Well, simply put, it's programming a machine
Speaker 1: to interpret language the way we use it we human beings.
Speaker 1: So in an ideal implementation, which would also require advanced
Speaker 1: artificial intelligence, you could speak to a machine or type
Speaker 1: whatever you like into a terminal and it would be
Speaker 1: able to understand what you meant. What your commands were,
Speaker 1: no matter how you worded the phrase. In turn, the
Speaker 1: machine would be able to generate responses that made linguistic
Speaker 1: sense to us, and we could in effect hold entire
Speaker 1: conversations with those machines. This, as it turns out, is
Speaker 1: a very difficult challenge. Even creating a machine that can
Speaker 1: respond to basic commands delivered in a natural language is
Speaker 1: really really hard to do, and we haven't yet cracked
Speaker 1: the nut on making a machine that can actually hold
Speaker 1: a real conversation with us. Yet we can sometimes forget
Speaker 1: that machines do not natively understand human language. Machines process
Speaker 1: information in machine code, which is difficult for humans to understand.
Speaker 1: I almost said impossible for humans to understand, but really
Speaker 1: it's just impractical. It's incredibly difficult. So, for example, computers
Speaker 1: that run on binary systems process all information in zeros
Speaker 1: and ones. Ultimately, when you get down to it, so
Speaker 1: if you were to look at a sheet of zeros
Speaker 1: and ones, it would probably seem completely incomprehensible to you,
Speaker 1: although to a computer it could seem perfectly logical. Our
Speaker 1: language is equally incomprehensible to machines. Programming languages make it
Speaker 1: easier for humans to make machines do what we want
Speaker 1: them to do. Programming languages create a level of abstraction
Speaker 1: between human language and machine language. It's kind of a
Speaker 1: meeting ground in the middle. Programming languages tend to be
Speaker 1: highly structured with specific strict sets of rules. Programming within
Speaker 1: those rules will get you the results you want, assuming
Speaker 1: your code is good, but if you stray outside those rules,
Speaker 1: you start to get errors. Human language is much more
Speaker 1: variable and complicated and ambiguous, and that's something that machines
Speaker 1: are not very good at handling. Now, if you've ever
Speaker 1: played a text based adventure from way back in the day,
Speaker 1: like Zork, you know that those adventure games have a
Speaker 1: very limited vocabulary. The game can accept certain commands, but
Speaker 1: only because the programmer built in the option in the game.
Speaker 1: They incorporated that in the game's design. So you might
Speaker 1: be able to type something like go north or just north,
Speaker 1: and the game understands you want your character to move
Speaker 1: to a new location that's to the north of your
Speaker 1: current location. But maybe you type something else, maybe you
Speaker 1: type jog north or saunter north, and the programmer didn't
Speaker 1: think of that. They didn't come up with all the
Speaker 1: different ways you have describe the way you want to
Speaker 1: move north, so you might get a result that says
Speaker 1: something like I didn't understand that, or you can't do
Speaker 1: that here. Computers only have the illusion of understanding us.
Speaker 1: They don't actually know what we mean when we say something,
Speaker 1: at least not natively. Now, that meant that for most
Speaker 1: of our history with computers, humans have had to learn
Speaker 1: how to work with machines, not the other way around.
Speaker 1: We have had to learn commands and syntax that machines accept,
Speaker 1: and if we try to word those commands in a
Speaker 1: different way, we tend to get an error. Natural language
Speaker 1: processing attempts to flip the tables on this relationship and
Speaker 1: teach machines how to work with humans so that we
Speaker 1: don't have to go through any sort of learning curve.
Speaker 1: We don't need to formulate our our commands in a
Speaker 1: specific way to be understood. The technology works on our terms,
Speaker 1: or as close to those as we can manage. That
Speaker 1: means that programmers have to build systems that can parse
Speaker 1: language for meaning, and it also means having to build
Speaker 1: tools and machines that can handle stuff that you typically
Speaker 1: encounter in higher level language courses. So here's a quick
Speaker 1: rundown on some of the stuff a natural language processing
Speaker 1: approach has to take into account. First, you have grammar. Now,
Speaker 1: grammar can refer to the study of language, but generally speaking,
Speaker 1: when we say grammar, or at least when I'm using
Speaker 1: the term in the context of natural language processing, I
Speaker 1: mean a set of rules for the organization of components
Speaker 1: of a language into meaningful statements or sentences. This is
Speaker 1: a broad concept. It is a big, big idea. It
Speaker 1: actually encompasses a couple of other also big ideas that
Speaker 1: are important in natural language processing. One of those is
Speaker 1: the concept of morphology. Morphology has to do with word forms.
Speaker 1: Words consist of more themes, and a word can actually
Speaker 1: have multiple moreph themes. So, for example, let's take a
Speaker 1: word like sky divers. Sky divers technically has four more themes,
Speaker 1: and they are sky dive er and s sky divers.
Speaker 1: The more themes only make sense if we put them
Speaker 1: in that particular order. For the word skydivers, dive skiers
Speaker 1: does not mean the same thing. Actually, it doesn't mean
Speaker 1: anything at all. So a good system will have to
Speaker 1: understand morphology and know how words can and cannot be formed.
Speaker 1: So again, with skydivers and knows all right, well, I
Speaker 1: know the word sky, I know what that means. I
Speaker 1: know what the word dive means. Er means that this
Speaker 1: is not an action. This is actually an entity that
Speaker 1: engages in that action. Right. A sky diver is someone
Speaker 1: who's skydives, and the s SO says it's plural, so
Speaker 1: that there's more than one skydiver. That's what morphology is
Speaker 1: all about. This is this sort of internal logic of
Speaker 1: word formation. Syntax is another big concept within grammar. Syntax, however,
Speaker 1: does not refer to word formation. It refers to sentence structure.
Speaker 1: How do we arrange words to make meaningful sentences. For example,
Speaker 1: the sentence you must have patience, my young Padawan. That
Speaker 1: follows good syntax, but patients you must have my young
Speaker 1: Padawan is a bit hanky because Yoda is all over
Speaker 1: the place with his syntax. In addition to grammar, you
Speaker 1: also have to take into account semantics. Now, that is
Speaker 1: the study of the meaning within language. This is a
Speaker 1: tricky one because there's a lot to unwrap here. For example,
Speaker 1: words and phrases can actually stand for different meanings. They
Speaker 1: can denote different ideas. We might use many different phrases
Speaker 1: or words to describe the same concept. Right, So we
Speaker 1: might use a usen or more different ways to say
Speaker 1: the same thing, or we might use two similar words
Speaker 1: or phrases to describe very different concepts. We might even
Speaker 1: use the same phrase to describe wildly different things or
Speaker 1: with very different meanings. Semantics gets down to what we
Speaker 1: actually mean when we say something. If you've ever had
Speaker 1: a discussion with someone and that person says, you know
Speaker 1: what I meant, that's essentially a statement that indicates semantically
Speaker 1: the meaning was clear, even if the phrasing did not
Speaker 1: indicate it on the face of things. Then there is
Speaker 1: pragmatics that's all about context. Contextual information is incredibly important
Speaker 1: in communication, and it relates a little bit to semantics.
Speaker 1: Semantics is about structure, and pragmatics is about context. So
Speaker 1: if I say the weather sure is nice today, on
Speaker 1: the face of it, that sounds like I'm in favor
Speaker 1: of the way the weather is. Right, it sounds like, oh,
Speaker 1: I like how the weather is. But if I say
Speaker 1: that same phrase while I'm standing in a downpour and
Speaker 1: I'm clearly not happy, I'm obviously being sarcastic. I mean
Speaker 1: the opposite of what I actually said. The context of
Speaker 1: the situation changes the meaning of what I am saying,
Speaker 1: even though the actual phrasing would seem to indicate the
Speaker 1: opposite of what my meaning was. As we develop more
Speaker 1: technology that can communicate with us, we have to take
Speaker 1: pragmatics into consideration, or else machines are going to be
Speaker 1: misinterpreting what we actually mean when we say stuff. So
Speaker 1: machines are going to have to learn how to deal
Speaker 1: with stuff like sarcasm. Yeah. Right. Then we have phonology,
Speaker 1: that is the sound of a language. I talked a
Speaker 1: little bit about this in the Speech Recognition podcast about
Speaker 1: how different languages have different phonemes. So I'm not going
Speaker 1: to dwell on that again. You can listen to the
Speaker 1: Speech Recognition podcast to learn more about it. But it
Speaker 1: is an important element in languages, especially when you get
Speaker 1: into uh natural language processing that is taking verbal input
Speaker 1: and not just textual input. Then you have lexicons that's
Speaker 1: the total vocabulary for a system. Ideally, alexicon has not
Speaker 1: just the words, but some sort of metadata attached that
Speaker 1: indicate the meaning of words or the relationship of words
Speaker 1: with one another. Though you can fudge this a little
Speaker 1: bit depending upon the implementation of the system. I'll talk
Speaker 1: a lot more about that throughout these podcasts. Now, these
Speaker 1: can be tricky concepts for human beings, let alone for machines.
Speaker 1: Machines are very good at following strict sets of instructions,
Speaker 1: but language can sometimes defy logic. Think of rules that
Speaker 1: apply to your native language, then just think of the
Speaker 1: exceptions that exist to those rules. Every language has exceptions
Speaker 1: for rules that are established, and depending upon the rule
Speaker 1: and the exception, there may seem to be no rhyme
Speaker 1: or reason for the deviation and from the rule. Moreover,
Speaker 1: if we want machines that are capable of understanding us
Speaker 1: and responding to our language in a meaningful way, those
Speaker 1: machines need to be able to handle the idiosyncrasies of
Speaker 1: individual speakers. To some extent. There may be regional turns
Speaker 1: of phrase or vocabulary that don't extend to the general
Speaker 1: population of speakers of the respected language. So you might
Speaker 1: encounter a person who speaks in local idioms quite a bit,
Speaker 1: and if those are not frequently used in the broader
Speaker 1: general population of that language, then you're gonna have a
Speaker 1: lot of communication errors between that person and a machine
Speaker 1: that is trying to process that language. Ideally, machines would
Speaker 1: be able to understand whatever we say and interpret the
Speaker 1: meaning correctly, although we haven't even gotten to a world
Speaker 1: where human beings can do that reliably, So I don't
Speaker 1: know why I'm holding machines up to such a high standard.
Speaker 1: We definitely would want them to reach a certain love
Speaker 1: of confidence and and capability, however that machines just are
Speaker 1: not quite there yet. I'm going to talk a lot
Speaker 1: more about the history of natural language processing in just
Speaker 1: a moment, but first let's take a quick break to
Speaker 1: thank our sponsor. The history of natural language processing is
Speaker 1: pretty darn complicated because it involves multiple lines of research
Speaker 1: and lots of different disciplines. So we have all sorts
Speaker 1: of things that play into this, like hidden Markov models
Speaker 1: I talked about those in the Speech Recognition podcast, neural networks,
Speaker 1: referencing language using mathematical vectors, and a lot more contributing
Speaker 1: to the evolution of natural language processing, and a lot
Speaker 1: of disciplines like not just computer science, but linguistics and psychology.
Speaker 1: So there's not like a single line I can follow
Speaker 1: where it's a lad to be led to see. So
Speaker 1: we're gonna be jumping around a little bit. However, one
Speaker 1: of the sources I want to call out that I
Speaker 1: used while I was researching this episode was a paper
Speaker 1: written by Karen Spark Jones called Natural Language Processing a
Speaker 1: Historical Review. It's pretty dense, it's pretty technical, but it's
Speaker 1: also available to read online if you want a more
Speaker 1: thorough treatment of the history of the technology up to
Speaker 1: two thousand. I'm gonna be skimming over quite a bit
Speaker 1: of it because, as I say, it gets really deep
Speaker 1: and really technical, and it uses a lot of shorthand
Speaker 1: to reference things, which meant that I had to do
Speaker 1: a lot of jumping down research rabbit holes to learn more.
Speaker 1: But it was a very useful starting point for this research.
Speaker 1: And also it was published in two thousand one. Obviously
Speaker 1: a lot has happened since then. We're almost two decades
Speaker 1: out from that. But I'm gonna start at the beginning
Speaker 1: and then work my way up to what's going on today.
Speaker 1: So early work in natural language processing it actually surprised me.
Speaker 1: I was surprised at how old it was. It actually
Speaker 1: dates all the way back to the nineteen forties. Physicist
Speaker 1: and computer scientist Andrew Donald Booth proposed using computers to
Speaker 1: translate passages from one language into another, which is the
Speaker 1: type of natural language processing. You have to be able
Speaker 1: to recognize the words of one language and then map
Speaker 1: them to a similar meaning in a different language. Now,
Speaker 1: Booth's approach involved creating a word for word model. If
Speaker 1: the model couldn't find a match between two words, it
Speaker 1: would automatically discard the last letter on the input word
Speaker 1: and try again. It would do this until it found
Speaker 1: a match, or if it didn't find a match, you've
Speaker 1: got an error. But it would find a match, it
Speaker 1: would search its memory to see if the ending of
Speaker 1: the input word could give information about what the ending
Speaker 1: does to the meaning of the word. So, for example,
Speaker 1: if you were using this to tr inslate from English
Speaker 1: into Russian and you use the word writer, maybe writer
Speaker 1: does not show up in the Russian lexicon, but right
Speaker 1: does w R I T E. So the translating program
Speaker 1: tries to translate writer from English into Russian, cannot find
Speaker 1: a Russian equivalent to writer, drops the r looks for
Speaker 1: the Russian word for right, and it finds it. Then says,
Speaker 1: all right, well, in English, what does writer remain? What
Speaker 1: does that are due to the word right and it
Speaker 1: looks at its memory and finds out that the letter
Speaker 1: R makes a a noun out of the verb, but
Speaker 1: it creates an entity that does the action, which is
Speaker 1: to right. Then it looks in the Russian lexicon and says,
Speaker 1: all right, well, is there a word in that lexicon
Speaker 1: that matches this meaning. It's kind of a slow, laborious
Speaker 1: way of doing things, but was also very very early.
Speaker 1: I mean it was the following year, in nine, Warren
Speaker 1: Weaver produced a memorandum about machine translation, and Weaver admitted
Speaker 1: in the memorandum that such an application would likely be
Speaker 1: much more challenging than what he understood it to be,
Speaker 1: but that he was quote willing to expose my ignorance,
Speaker 1: hoping that will be slightly shielded by my intentions in
Speaker 1: the quote. And I think that's rather charming. In that memo,
Speaker 1: Weaver cites a letter he wrote to Professor Norbert Wiener
Speaker 1: of M I T. And that included the following paragraph.
Speaker 1: So here's a full paragraph. Actually it's two paragraphs from
Speaker 1: the memorandum recognizing fully, even though necessarily vaguely, the semantic
Speaker 1: difficulties because of multiple meanings, etcetera. I have wondered if
Speaker 1: it were unthinkable to design a computer which would translate,
Speaker 1: even if it would only translate only scientific material, where
Speaker 1: the semantic difficulties are very notably less, and even if
Speaker 1: it did produce an inelegant but intelligible result, it would
Speaker 1: seem to me worthwhile also knowing nothing official about, but
Speaker 1: having guests and inferred considerable about powerful new mechanized methods
Speaker 1: and cryptography methods which I believe succeed even when one
Speaker 1: does not know what language has been coded. One naturally
Speaker 1: wonders if the problem of translation could conceivably be treated
Speaker 1: as a problem in cryptography. When I look at an
Speaker 1: article in Russian, I say, this is really written in English,
Speaker 1: but it has been coded in some strange symbols I
Speaker 1: will now proceed to decode. So he got this idea
Speaker 1: because of activities that were going on in World War Two,
Speaker 1: where teams were trying to decode messages. And they might
Speaker 1: decode the message, they might figure out what letters correspond
Speaker 1: to the code, but it may even be in a
Speaker 1: totally different language than when they speak. So while they
Speaker 1: are able to decode the message into a native language,
Speaker 1: they are not able to speak that language. He says, well,
Speaker 1: what if we just take that same step, and now
Speaker 1: we treat the other language as a code in of
Speaker 1: itself and try to translate that into English or or
Speaker 1: decrypt it into English. Weaver are acknowledged that the word
Speaker 1: into word approach that Booth and his contemporaries were relying
Speaker 1: upon had limited utility. He wrote, quote, it is in
Speaker 1: fact amply clear that a translation procedure that does little
Speaker 1: more than handle a one to one correspondence of words
Speaker 1: cannot hope to be useful for problems of literary translation
Speaker 1: in which style is important, and in which the problems
Speaker 1: of idiom, multiple meanings, etcetera. Are frequent. End quote. So
Speaker 1: there he's saying, you can't just take a foreign word,
Speaker 1: translate it into whatever the closest equivalent in English is,
Speaker 1: and hope to get the same meaning, especially in literary works,
Speaker 1: because they are all these different turns of phrase and
Speaker 1: cultural meanings that will get lost. In that translation. You
Speaker 1: would have something that might technically be considered more or
Speaker 1: less correct, but would not be actually correct. You wouldn't
Speaker 1: be getting across the meaning of the author in that translation.
Speaker 1: You would just have words in a syntactical order that
Speaker 1: would make sense from a syntax perspective. In other words,
Speaker 1: you would have sentences that held up grammatically, but they
Speaker 1: wouldn't necessarily have the meaning of the original writing. Weaver's
Speaker 1: proposal was to perhaps expand the word into word model
Speaker 1: and create a system that would analyze not just the
Speaker 1: target word, but the words adjacent to the target in
Speaker 1: order to determine the context of the word the meaning
Speaker 1: of the word. As we'll see when we get a
Speaker 1: little bit further down in the timeline, this is one
Speaker 1: of the methods that folks working in in natural language
Speaker 1: processing incorporated into their approach. So this was incredibly forward
Speaker 1: thinking of Weaver. On January seven, ninety four, researchers from
Speaker 1: IBM and Georgetown University demonstrated a system that was able
Speaker 1: to translate around sixty sentences from Russian into English automatically. Now,
Speaker 1: the process wasn't exactly painless. It required an operator to
Speaker 1: take a sentence written in Russian but transcribed for the
Speaker 1: English alphabet. It wasn't in the cyrillic alphabet. The person
Speaker 1: would then encode that sentence on punch cards. They would
Speaker 1: feed the punch cards into a seven oh one computer.
Speaker 1: I mentioned the seven oh one that was an IBM system,
Speaker 1: but I mentioned that in the previous episode and speech recognition.
Speaker 1: Then they would wait for the translation program's response, which
Speaker 1: would take a few seconds. The program would attempt to
Speaker 1: translate the words from Russian to English. The demonstration was impressive,
Speaker 1: but it was limited in scope. The program had alexicon
Speaker 1: of only two fifty words or so, and it required
Speaker 1: extensive programming to cope with syntax because word order in
Speaker 1: Russian is different then word order in English, and you
Speaker 1: can think of the programming as including metadata. The researchers
Speaker 1: would tag Russian words with little signs that related to
Speaker 1: specific rules. So, for example, one of the terms the
Speaker 1: system could translate was a Russian two word phrase. It
Speaker 1: was g dial major, which is I'm butchering the Russian pronunciation,
Speaker 1: but in English it means major general. But the word
Speaker 1: order is reversed in Russian. If you did a strict
Speaker 1: word to word translation, you would get general major with
Speaker 1: the translation, because that's the order that the Russian phrase
Speaker 1: would put it in. So the programmers would tag each
Speaker 1: word with a rule to kind of give the idea
Speaker 1: of of what what you would what you should follow
Speaker 1: when you're making these translations, and by you I mean
Speaker 1: the computer system. So the word for general got the
Speaker 1: assignment of rule twenty one and the rule for major
Speaker 1: got the sign on. So when the system encountered a word,
Speaker 1: it would look up any related rules to that word.
Speaker 1: So if it comes across a word that has the
Speaker 1: associated rule one, it would say, all right, this rule
Speaker 1: tells me I have to go back over the message
Speaker 1: and look to see if there was a rule twenty
Speaker 1: one word in that same phrase, And if it finds
Speaker 1: a rule twenty one word, it would then know I
Speaker 1: need to reverse the order of these two words. This
Speaker 1: this uh word order that appears in Russian needs to
Speaker 1: be flipped for English. Now that's a pretty laborious process
Speaker 1: and it doesn't work great for larger lexicons. The larger
Speaker 1: the vocabulary, the more complex the sentences can become, the
Speaker 1: more exceptions and rules you're going to encounter. It would
Speaker 1: be really hard to implement this on a big scale,
Speaker 1: but it was an impressive display of machine translation. The
Speaker 1: system was essentially a vocabulary list and a long series
Speaker 1: of if then rules. If the word is this, then
Speaker 1: look for this. If that is there, then switch the
Speaker 1: word order. Essentially according to articles, it could translate sentences
Speaker 1: designed for the system in about six seconds. But again
Speaker 1: it was designed for the system, very limited vocabulary, so
Speaker 1: limited implementation there. And it's good to point out that
Speaker 1: a lot of work and machine translation around this time
Speaker 1: focused on English and Russian, which is no big surprise.
Speaker 1: Keep in mind the time scale we're talking about the
Speaker 1: nineteen fifties. Here, the USA and the then USS are
Speaker 1: we're not on great terms. Both countries were using pretty
Speaker 1: much every means at their disposal to analyze one another,
Speaker 1: to spy on one another, to maneuver to make certain
Speaker 1: the other nation didn't get a superior position. And we
Speaker 1: saw a lot of technological development during this period, including
Speaker 1: the space race that was all wrapped up in this
Speaker 1: Cold War issue as well, and perhaps as no big surprise,
Speaker 1: the US government was pretty keen to fund research and
Speaker 1: development in machine translation up to a point. That is,
Speaker 1: in nineteen sixty six, Joseph Wisenbaum published a computer program
Speaker 1: called Eliza. I've talked about Eliza in previous episodes of
Speaker 1: Tech Stuff. This was a primitive chat bought text based
Speaker 1: chat bot. It mimicked a Rogerian psychotherapist. That's a discipline
Speaker 1: that was pioneered by the psychologist Carl Rogers. It's sometimes
Speaker 1: also called persons centered therapy. Eliza was strictly this text
Speaker 1: based terminal operation. You would see a line of text
Speaker 1: pop up. It would ask you how what how you're doing?
Speaker 1: You can type stuff in and then it would respond
Speaker 1: to you, so you would get the responses that appeared
Speaker 1: to be semi intelligent. Typically it would be a question
Speaker 1: to ask for more information, or sometimes it would be
Speaker 1: a phrase to change the subject. So you might say
Speaker 1: something along the lines of I'm so angry right now,
Speaker 1: and Eliza might respond with what has made you angry?
Speaker 1: So Eliza has flipped this around in order to sustain
Speaker 1: the conversation. Then you could type in something else. Maybe
Speaker 1: you type in everything is going wrong today, and Eliza
Speaker 1: might respond with can you give me an example? And
Speaker 1: then so on. Eliza would give the appearance of understanding
Speaker 1: the subject, but in reality it was simply taking the input,
Speaker 1: analyzing the parts of speech, then sending back a very
Speaker 1: similar message or a related message in an effort to
Speaker 1: keep the conversation going. Like it might just be a placeholder.
Speaker 1: The program did not understand language or context beyond being
Speaker 1: able to parse the basic parts of a sentence and
Speaker 1: then rearrange them or go with several stock responses when
Speaker 1: it didn't have a way of figuring out what it
Speaker 1: should do. NIX also saw something else that would end
Speaker 1: up creating a bit of a big setback for natural
Speaker 1: language processor researchers. But I'll explain more about that when
Speaker 1: we come back after a quick break to thank our sponsors. Okay,
Speaker 1: So nineteen sixty six, what happened that set back research
Speaker 1: in this field. Well, that's when a report was published
Speaker 1: that had a dramatic impact on funding for R and
Speaker 1: D and machine translation. It was called the ALPAC Report.
Speaker 1: ALPAC a l p a C stood for Automatic Language
Speaker 1: Processing Advisory Committee. This was a group consisting of various
Speaker 1: experts and fields ranging from computer science to linguistics to psychology,
Speaker 1: and the U. S. Government had established the committee back
Speaker 1: in nineteen sixty four, and they had a very simple assignment,
Speaker 1: or at least simple on the surface, which was evaluate
Speaker 1: the progress that was being made an automatic machine translation
Speaker 1: across the board, look at what everyone's working on. Give
Speaker 1: us an idea of where we are and where we're headed.
Speaker 1: The nineteen sixty six report essentially concluded that the field
Speaker 1: was still in its infancy, and that before any real
Speaker 1: advancements could happen, a lot more basic research in the
Speaker 1: field of computational linguistics would be required. So essentially, the
Speaker 1: report was saying, we're trying to move at a full gallop,
Speaker 1: but we still aren't really sure how to get on
Speaker 1: the horse. I'm paraphrasing, of course. One result of this
Speaker 1: was that the US government began to scale back grants
Speaker 1: for research in the field of machine translation. This was,
Speaker 1: unfortunately exactly the opposite thing that needed to happen. The
Speaker 1: US government wanted more immediate results and decided, well, if
Speaker 1: you're not going to get results right away, we're gonna
Speaker 1: take that money away and put it to use somewhere else.
Speaker 1: And that made funding scarce, and it likely prolonged the
Speaker 1: amount of time it took to advance the discipline. Although
Speaker 1: I should stress work was still being performed in the
Speaker 1: United States as well as elsewhere. It's not like this
Speaker 1: brought everything to a standstill. It just slowed down quite
Speaker 1: a bit. By teen sixty seven, NLP research was straining
Speaker 1: against technological limitations. They were starting to feel the the
Speaker 1: very limit of what computers were able to do. Even
Speaker 1: advanced systems could take upwards of seven minutes to analyze
Speaker 1: a long sentence. Programming was still largely in a similar language,
Speaker 1: so it wasn't easy to do. And you would still
Speaker 1: have to interact with machines using punch cards, so that
Speaker 1: was also laborious, and heaven help you if you dropped
Speaker 1: all your punch cards and you forgot to number them,
Speaker 1: because then you've ruined your program. Work was progressing on
Speaker 1: the linguistic side, but the technological side was kind of
Speaker 1: lagging behind at this point. One of the big decisions
Speaker 1: researchers had to make around this time was what were
Speaker 1: they going to focus on first while building out computational linguistics.
Speaker 1: Because it's such a huge problem you couldn't really tackle
Speaker 1: it wholesale. You needed to kind of focus on specifics.
Speaker 1: So should research focus on syntax that all about sentence
Speaker 1: form and structure, as I mentioned earlier, or should it
Speaker 1: focus on semantics, which is more about the underlying meaning
Speaker 1: of what was said and less about the structure of
Speaker 1: how it was said. And ultimately, most researchers, not all
Speaker 1: of them, but most of them decided to focus on syntax.
Speaker 1: For one thing, it seemed like a more analytical thing
Speaker 1: to concentrate on. Right like, you could define rules more
Speaker 1: easily for syntax than you could for semantics, and semantic
Speaker 1: ambiguity could be fudged a bit. You can rely heavily
Speaker 1: on output words that had a broad meaning. So using
Speaker 1: a word with a broad meaning might not produce a specific,
Speaker 1: precise result, but at least could be quote not wrong
Speaker 1: end quote. So if a word might have several translations
Speaker 1: ranging from hut to villa to bungalow to mansion, the
Speaker 1: output word might be building because the translating program might
Speaker 1: not know which variation of that translation it should go with,
Speaker 1: but knows that all of those different examples fall into
Speaker 1: a larger category called building. So that's not precise, but
Speaker 1: it gets the job done. You you would understand what
Speaker 1: the the actual noun was. In general, you would know
Speaker 1: it was a building. You might not know that it
Speaker 1: was a home, and you might not know what kind
Speaker 1: of home it was, but you would at least know
Speaker 1: that it was a structure. So much of the work
Speaker 1: in the late nineteen sixties focused on solving syntax problems
Speaker 1: for computers, with the researchers saying will worry about semantics later.
Speaker 1: Some notable groups went against the flow and decided to
Speaker 1: tackle semantics and semantically driven processing, partly because they recognized
Speaker 1: it as being a really tough problem and some engineers
Speaker 1: just love solving really hard problems. That's kind of what
Speaker 1: thrills them, and so they chose to go that route.
Speaker 1: They began building out semantic categories and worked on semantic
Speaker 1: pattern matching using semantic networks as a means of knowledge representation.
Speaker 1: Karen Spark Jones, who wrote that that history I mentioned earlier,
Speaker 1: suggests that it was in the late nineteen sixties that
Speaker 1: the research moved out of its initial phase and into
Speaker 1: a second phase, and that second phase was largely marked
Speaker 1: by the incorporation of artificial intelligence, including incorporating world knowledge
Speaker 1: in processing natural language. In nineteen sixty eight, Terry Winograd,
Speaker 1: who today is a Professor Emeritus of Computer Science at
Speaker 1: Stanford University, was working in M I. T. S AI
Speaker 1: Lab as part of his postgraduate studies, and he began
Speaker 1: to work on a virtual world he would call s
Speaker 1: h R D l U sued blue. Um, that's what
Speaker 1: I'm going to call it is sued blue. It consisted
Speaker 1: of virtual objects on a virtual table, so it's all imaginary, right.
Speaker 1: He then programmed a grammar and lexicon specifically for this
Speaker 1: very very limited imaginary world. So in anything that did
Speaker 1: not involve the things that were in this imaginary world,
Speaker 1: namely the table and these virtual objects, that didn't need
Speaker 1: to be dealt with it all because it was immaterial,
Speaker 1: It didn't exist in this universe. So he only had
Speaker 1: to focus on the elements he had created, and that
Speaker 1: limited the scope of his work and made it more manageable.
Speaker 1: His design even included the concept of persistence and memory.
Speaker 1: So imagine a table with a collection of five objects
Speaker 1: on it. So you've got imaginary table, You've got five
Speaker 1: imaginary objects on it. Two of the five imaginary objects
Speaker 1: are spheres. One of them is a green sphere, and
Speaker 1: one of them is a red sphere. You then type
Speaker 1: in a command into a terminal that is that's giving
Speaker 1: you information about this virtual world, and you say, I
Speaker 1: want to move the red sphere over to the far
Speaker 1: end of the table. And then you send another command,
Speaker 1: only this time you don't specify red sphere. You just
Speaker 1: say move the sphere back. Whino grad system could actually
Speaker 1: remember ber that you had previously moved the red sphere,
Speaker 1: and it would apply your command to the red sphere
Speaker 1: again under the assumption that's what you meant. When you
Speaker 1: didn't specify, you must have meant the same sphere that
Speaker 1: you had just moved. This is a concept that we're
Speaker 1: seeing rolled out into voice assistance today, like Google Assistant.
Speaker 1: It's the ability to reference something you've already accessed without
Speaker 1: having to specify what you're talking about. So if I
Speaker 1: asked a voice assistant what the weather will be like today,
Speaker 1: and then I follow that up after I get the information,
Speaker 1: I say what about tomorrow, the system that has this
Speaker 1: kind of capability could infer that what I meant was
Speaker 1: what will the weather be like tomorrow, even though I
Speaker 1: didn't say it specifically. Like that. That's pretty advanced for
Speaker 1: nineteen sixty eight, even though it was for this very
Speaker 1: restricted virtual world with a limited number of variables. However,
Speaker 1: win no Grad discovered that the secret to his success
Speaker 1: was largely in this restriction. As you expec ended the
Speaker 1: virtual world to incorporate more elements, it made the problem
Speaker 1: exponentially harder. His work, by the way, was an early
Speaker 1: example of what we call anapho resolution, and an anaphour
Speaker 1: is what I was talking about second ago. It's a
Speaker 1: word or phrase that refers to an earlier word or
Speaker 1: phrase within a discourse. So if I said move the
Speaker 1: red sphere to the left, then I said, now move
Speaker 1: it back the It obviously refers to the red sphere.
Speaker 1: You would understand that, but a machine wouldn't necessarily understand it.
Speaker 1: You would have to say move the red sphere to
Speaker 1: the left, move the red sphere back. And even with back,
Speaker 1: that has an element of memory to it, because the
Speaker 1: system has to remember where the red sphere used to be. Why.
Speaker 1: No Grad's approach was one of the early attempts to
Speaker 1: incorporate anapho resolution into NLP models. Other models concentrated on
Speaker 1: translating word by word or sentence by sentence. They were
Speaker 1: incapable of maintaining relationships between between words beyond that. That
Speaker 1: shift marked a change in attitude among NLP researchers of
Speaker 1: the time. A growing number of researchers felt that world
Speaker 1: knowledge and artificial intelligence was necessary if we wanted machines
Speaker 1: to be able to analyze and act upon longer forms
Speaker 1: of discourse. The early approaches to NLP were best suited
Speaker 1: to short, self contained passages in ninety one, AREPA launched
Speaker 1: the Speech Understanding Research Program. I also mentioned that in
Speaker 1: the Speech recognition episode it was very important for the
Speaker 1: development of speech recognition. The goal of that program was
Speaker 1: to advance not only speech recognition but also n LP
Speaker 1: research so that a computer could not just detect and
Speaker 1: transcribe speech, but also respond to it in some meaningful way,
Speaker 1: for example being able to UH index all that information
Speaker 1: so that it is searchable. The program lasted five years. However,
Speaker 1: at the conclusion, the agency was not satisfied with the results,
Speaker 1: which technically delivered upon what was asked, but a pretty
Speaker 1: limited implementation, so are BUT decided to cut funding. They
Speaker 1: stopped the project. This was another big blow to research
Speaker 1: in the United States, which had viewed the project as
Speaker 1: a positive development ever since the ALPAC report had pulled
Speaker 1: the RUG out from under the funding earlier. Now, I've
Speaker 1: got a lot more to say about the development of
Speaker 1: natural language processing and where we are now, as well
Speaker 1: as the history of the various voice assistants that we're
Speaker 1: familiar with today. But it's time to conclude this episode.
Speaker 1: In our next episode, we'll pick up where I left
Speaker 1: off today and we'll continue down and talk about all
Speaker 1: of our beloved friends like Syrie and Alexa. Now, if
Speaker 1: you have suggestions or future episodes of tech Stuff, right me.
Speaker 1: Let me know what you want to hear. There might
Speaker 1: be a specific technology or a company, a person in tech.
Speaker 1: Maybe there's someone you want me to interview or have
Speaker 1: on as a special guest host. You can send me
Speaker 1: an email. The address for the show is tech Stuff
Speaker 1: at how stuff works dot com, or you can drop
Speaker 1: me a line on Facebook or Twitter. The handle of
Speaker 1: both of those is tech Stuff H s W. Don't forget.
Speaker 1: You can follow us on Instagram. I want to see
Speaker 1: you guys over there, and I'll talk to you again
Speaker 1: really soon for more on this and thousands of other topics,
Speaker 1: because it how stuff works dot com