Speaker 1: Welcome to tech Stuff, a production from I Heart Radio.
Speaker 1: Hey there, and welcome to tech Stuff. I'm your host,
Speaker 1: Jonathan Strickland. I'm an executive producer with I Heart Radio
Speaker 1: and love all things tech, and this is time for
Speaker 1: a classic tech Stuff episode. This episode originally published on
Speaker 1: June twenty three, two thousand fourteen. It is titled passing
Speaker 1: the Turing Test, something that we frequently associate with artificial
Speaker 1: intelligence and actual you know, like machines thinking or at
Speaker 1: least appearing to think. So let's listen in to this
Speaker 1: classic episode. The Touring Test is named after Alan Turing,
Speaker 1: and we've done a full episode on Alan Turing way
Speaker 1: back when, back in November. Yeah, phenomenal person, amazing thinker. Yeah,
Speaker 1: one of the like the grandfather of computer science. Also
Speaker 1: tragic life story which we went into detail back in
Speaker 1: that story. And so you might wonder why, right, So
Speaker 1: what does that have to do was the the guy
Speaker 1: who was the essentially the father of computer science or
Speaker 1: grandfather of computer science? What does he have to do
Speaker 1: with the story about a computer program in June two
Speaker 1: thousand fourteen, a computer program with an interesting name, Eugene
Speaker 1: Goostman passing the Turing test. What does that all have
Speaker 1: to do with each other? Well? To answer that, we
Speaker 1: have to ask what is a Turing test? Well, as
Speaker 1: it turns out, Away back in the nineteen fifties, he
Speaker 1: started envisioning uh I thought experiment. Yeah, he published a
Speaker 1: paper in a journal called Mind in nineteen fifty called Computing,
Speaker 1: Machinery and Intelligence, and he sort of laid out his
Speaker 1: thought experiment there. It's interesting because what he did was
Speaker 1: he took this idea of a party game and then
Speaker 1: adapted it for computers. Right. The party game that it's
Speaker 1: based on would have three participants, an interrogator, a man,
Speaker 1: and a woman um, all situated so that they can't
Speaker 1: see one another, and the interrogator is supposed to ask
Speaker 1: questions um to try to figure out which of the
Speaker 1: participants is the dude and which is the lady exactly.
Speaker 1: And it's the dude's job to try and mislead the
Speaker 1: interrogator to believe that that he, in fact is the
Speaker 1: lady and the other one is the man. It's the
Speaker 1: lady's job to say, hey, I want you to get
Speaker 1: this right. I'm the lady. That other guy that's the dude.
Speaker 1: And so the interrogator has to ask questions. Now, obviously
Speaker 1: they can't see each other. Because they could see each other,
Speaker 1: then that would probably give things away. Probably, they really
Speaker 1: shouldn't be able to hear each other because that could
Speaker 1: also give things away. Sure, and if you use handwritten notes,
Speaker 1: then you the interrogator might be able to make judgments
Speaker 1: based on the handwriting style. So it should really be typewritten, right,
Speaker 1: So you want to you want to remove as many
Speaker 1: easily identifiable traits from this game as possible to make
Speaker 1: it all about the questions and the answers. Now, Touring said,
Speaker 1: what if we were to take the same basic premise,
Speaker 1: but instead of having two human interviewees, replace one of
Speaker 1: those humans with a machine. Now, if that machine can
Speaker 1: convince the interrogator that the machine itself is a human being,
Speaker 1: it would be a pretty phenomenal achievement. Or you know,
Speaker 1: just generally, if the interrogator wasn't sure which of the
Speaker 1: two interrogate, right, whether which one is the human, which
Speaker 1: one is the machine, or even if there could be
Speaker 1: a case where you don't know. It may be that
Speaker 1: you have two humans that you're interrogating, and it may
Speaker 1: be one of those things where you have to you know,
Speaker 1: if you don't know for a fact that one of
Speaker 1: them the machine, that makes it even harder, right, at
Speaker 1: least assuming that the computer program is sophisticated enough. Now,
Speaker 1: Touring was saying that we don't have any machines right
Speaker 1: now that can do this, but I envision a time
Speaker 1: when computers will be able to do such a thing
Speaker 1: where if you were to interrogate a computer, you would
Speaker 1: get back responses that would be uh, convincing enough for
Speaker 1: it to make it difficult to determine if it were
Speaker 1: man or machine. And so he said that he predicted
Speaker 1: in fifty years time, which would be the year two thousand,
Speaker 1: that computers and software be sophisticate enough that interrogators would
Speaker 1: only be able to guess correctly seventy percent of the time,
Speaker 1: meaning that they would be fooled by the computer pent
Speaker 1: of the time. Right. So, uh, this was just kind
Speaker 1: of a thing he was coming up with, like an idea,
Speaker 1: a prediction, not necessarily a test. Although very much like
Speaker 1: Moore's observation became Moore's law, this became what is known
Speaker 1: as the Turing test. People talk about a Touring test
Speaker 1: as a machine capable of fooling people into thinking it's
Speaker 1: another person at least thirty percent of the time after
Speaker 1: five minutes of conversation. Very good point. Yes, it needs
Speaker 1: to be five minutes of conversation. If you are just
Speaker 1: getting maybe two or three responses, that might not be
Speaker 1: enough for you to be able to draw a conclusion
Speaker 1: that you feel good about. If after five minutes you
Speaker 1: still are not entirely, entirely certain, then that might say
Speaker 1: that this machine, in fact has passed the Turing test.
Speaker 1: And this has been extrapolated to mean something about machine
Speaker 1: intelligence because Turing himself tied the idea of of how
Speaker 1: we perceive a machines intelligence directly to artificial intelligence and
Speaker 1: and and even how we perceive human intelligence. Because here's
Speaker 1: Touring's idea is a little cheeky, and I love the
Speaker 1: fact that it's so cheeky. So Turing kind of said,
Speaker 1: would you say such a machine as intelligent if it
Speaker 1: appears to be intelligent, is it fair to say that
Speaker 1: is intelligent? Touring? So why not? Because I'm only able
Speaker 1: to tell that I am intelligent, right because of my experience.
Speaker 1: I'm only able to have my own personal experience. I
Speaker 1: can't experience what someone else's life is like, all right,
Speaker 1: sitting across from each other, Jonathan and I can only
Speaker 1: assume that the other one is intelligent, right, And it's
Speaker 1: because the other person is displaying traits that we associate
Speaker 1: with intelligence. They seem to be able to take an information,
Speaker 1: respond to it, make decisions. And based on the fact
Speaker 1: that we ourselves also do that thing, we go ahead
Speaker 1: and say, all right, well, they clearly have the same
Speaker 1: features that I have, which includes intelligence. Uh. Now, he says,
Speaker 1: why would we not extend that same courtesy to a
Speaker 1: machine if it also appeared to display those same uh features?
Speaker 1: He says, doesn't matter if the computer is quote unquote thinking.
Speaker 1: If it's it can fool you. Yeah, if it's able
Speaker 1: to to simulate it well enough, you might as well
Speaker 1: say it's intelligent, because simulate it is probably a kinder
Speaker 1: phrase for that than fool. Yes. Yeah, well I know
Speaker 1: it's it is fooling essentially, I mean, because ultimately you're
Speaker 1: talking about a computer programmer who's making this happen. So
Speaker 1: nowadays we think of this as the Turing test. Can
Speaker 1: a machine of the time are more fool someone after
Speaker 1: five minutes of conversation into thinking it's a human? Now,
Speaker 1: this is really hard to do. This is is non trivial.
Speaker 1: I mean it sounds almost simple. Yeah, the concept is simple,
Speaker 1: the execution incredibly difficult. Because here's the thing, human language varied.
Speaker 1: We have unstructured, unpredictable ways to say things like if
Speaker 1: if I were to tell you that it's a hundred
Speaker 1: degrees outside and it is humid like the humidities, and
Speaker 1: you go out there, then I'm sure all of our
Speaker 1: listeners would have slightly different ways to express their thoughts
Speaker 1: on the conditions outside. Some of them would probably contain
Speaker 1: colorful metaphors. Mine very likely when it contain colorful metaphors,
Speaker 1: particularly if I had to be outside for any length
Speaker 1: of time. But that's the point. We would all have
Speaker 1: different ways of saying this. So how do you make
Speaker 1: a computer program able to interpret all the myriad of
Speaker 1: ways we can all express the same thought, let alone
Speaker 1: any thought? All right? This is what's referred to as
Speaker 1: natural language recognition, and it's a really huge problem in
Speaker 1: artificial intelligence and and and a lot of other speech
Speaker 1: related computer programming. Exactly. Yeah, this is where a computer
Speaker 1: program has to be able to parse the language so
Speaker 1: it recognizes things like this word is a noun, this
Speaker 1: word is a verb, this this word alters this other word. Uh.
Speaker 1: And not only does it need to be able to
Speaker 1: recognize it, it it needs to be able to respond in kind. Right,
Speaker 1: So if you have appropriately in some way or another share. Yeah,
Speaker 1: you could have a computer program that's literally making up
Speaker 1: sentences or you know, the approximation of a sentence randomly,
Speaker 1: where it's just pulling strings of words and placing them
Speaker 1: in a sequence and then presenting them. But that wouldn't
Speaker 1: be convincing at all. If I were to say, hello,
Speaker 1: how are you today, and Lauren was to say blue
Speaker 1: panther pickup jump down street, I'd be like what? And
Speaker 1: even that was closer to being a sentence than some
Speaker 1: of the random stuff that you would see if it
Speaker 1: was just truly completely So it also has to be
Speaker 1: able to uh endure a five minute long conversation, like
Speaker 1: we said, in order to pass the Turing test. So
Speaker 1: you can't have too much repetition or that gives it away. Absolutely.
Speaker 1: If you've ever been playing a video game and all
Speaker 1: of the NPCs say the same thing over and over
Speaker 1: and over again, yeah, if if all the chat bot
Speaker 1: says is hey, listen, then you're clearly playing Zelda and
Speaker 1: you're not actually having a decent conversation. That being said,
Speaker 1: there is someone I know who plays a ferry at
Speaker 1: the Georgia Renaissance Festival, and hey, listen is heavily represented
Speaker 1: in a repertoire. It's pretty amazing. No, it's pretty awesome.
Speaker 1: But at any rate, Yeah, so so these are these
Speaker 1: are big problems. You have to build a database of words,
Speaker 1: you have to be able to figure out what kind
Speaker 1: of syntax are you going for. It's a wide open,
Speaker 1: huge problem, and solving this problem can be really beneficial
Speaker 1: in lots of ways. We'll talk a little bit about
Speaker 1: that later. It's it's beyond just making a program that
Speaker 1: seems human, right, That's that's one way of looking at it.
Speaker 1: But there are a lot of other benefits that come
Speaker 1: along with it, which will chat about towards the end
Speaker 1: of the show. But for right now, let's go into
Speaker 1: this story behind Eugene Goostman and whether or not it
Speaker 1: was actually the first chat bought to pass the Turing test. Yeah. So,
Speaker 1: first of all, we've got three programmers in this story
Speaker 1: of Vladimir Vslov, Eugene Demchinko and Sarage you listen. As
Speaker 1: you may guess from their names, they all hail from
Speaker 1: Russia and the Ukraine as well, and although not all,
Speaker 1: not all of them still live there. But starting in
Speaker 1: two thousand one they got to work on this, right, Yeah,
Speaker 1: they were trying to design a computer program that would
Speaker 1: pose specifically as a third teen year old boy from
Speaker 1: Odessa in the Ukraine. Yeah, and that meant that they
Speaker 1: had specific parameters that they could work within. It automatically
Speaker 1: helped reduce some of that unpredictability and that lack of
Speaker 1: restriction that you would have if you were to just say,
Speaker 1: this is a fluent adult speaker of a given language. Yeah, Yeah,
Speaker 1: giving him these you know, setting up the expectation from
Speaker 1: the judges that you know, this is a non native
Speaker 1: English speaker. It's it's a kid essentially. Um, you know
Speaker 1: that they'll expect him to have limited knowledge of the
Speaker 1: world and and different subject areas, and a limited understanding
Speaker 1: of English vocabulary and grammar and all that kind of stuff.
Speaker 1: So you're already managing expectations. That's going to come into
Speaker 1: play when we talk about some of the criticisms about this.
Speaker 1: Although although I do think it's a very clever way
Speaker 1: and and a lot of previous chatbots have have had
Speaker 1: similar yes kind of approaches. Yeah, because like we said,
Speaker 1: if you were to take a quote unquote pure approach
Speaker 1: to this, it's really really challenging. So yeah, by by
Speaker 1: limiting this, the judges have an idea of well, this
Speaker 1: this could be a thirteen year old boy or it
Speaker 1: could be a computer program. It means that the computer
Speaker 1: program doesn't have to be as sophisticated as one that
Speaker 1: would be completely fluent and have you know, an adult's experiences,
Speaker 1: uh and ability to communicate. So that was step one
Speaker 1: and uh, Eugene Goosman took part in the competition. UH
Speaker 1: that had five total chatbots. It was it was one
Speaker 1: of five that took place on the sixtieth anniversary of
Speaker 1: Turing's death, and the program managed to full thirty three
Speaker 1: percent of the judges into thinking it was actually a person. Uh,
Speaker 1: and it had there were thirty judges from what I understand,
Speaker 1: So that was where you got all the headlines of
Speaker 1: chat bought. There are computer beats touring tests, which already
Speaker 1: is not accurate. There were some slight um hiccups in
Speaker 1: a little bit of the news reporting that we'll get
Speaker 1: into that part of the story later. But you know,
Speaker 1: the key takeaways I think here are that this was
Speaker 1: a this was a competition that was celebrating touring awesome um.
Speaker 1: And that there were thirty judges, some of whom were celebrities, yes,
Speaker 1: including an actor who had appeared on Red Dwarf right. Yes.
Speaker 1: And then a couple of years before that, because because
Speaker 1: you said they started working on this in two two
Speaker 1: thousand one, this was not the first time that Gustman
Speaker 1: had entered competition. UH. Two years previously, that same software
Speaker 1: had convinced twenty of judges at a similar competition that
Speaker 1: was held at Bletchley Park. Now, but Bletchley Park that's
Speaker 1: where Touring helped crack the Enigma machine, the encoding device
Speaker 1: that the German military was using during World War Two.
Speaker 1: So this was a big celebration. It was at the
Speaker 1: centennial celebrating his birth, and it ended up falling just
Speaker 1: short of passing the Turing test. Uh. The organizer of
Speaker 1: the the more recent event, the one in which Gustman
Speaker 1: ran away with quote unquote beating the Turing test. UH.
Speaker 1: That organizer was Kevin Warwick. That name may sound familiar
Speaker 1: to some of our listeners. If you've ever heard us
Speaker 1: talk about cyborgs. He's the guy who had an r
Speaker 1: f I D chip surgically implanted into him. His wife
Speaker 1: also did at one point. Yes, they could communicate with
Speaker 1: each other through them. Some unflattering news media sometimes refers
Speaker 1: to him as Captain Cyborg. Yeah. There there are some
Speaker 1: critics who say that he, uh, he courts publicity in
Speaker 1: a manner that is unbecoming of of of a scientist. Yes,
Speaker 1: really of anybody. Yes, yes, uh, those are the critics
Speaker 1: who say that. By the way, I just want to
Speaker 1: make that clear. At any rate, He said that this
Speaker 1: was the first time a chatbot had passed the Turing
Speaker 1: test at an event, or the conversation was open ended,
Speaker 1: meaning that they had not previously uh decided upon a
Speaker 1: specific topic or line of questioning that the judges were
Speaker 1: allowed to say whatever they wanted to the chat bot
Speaker 1: and response, which obviously makes it harder because you have
Speaker 1: to have a much wider breadth of potential responses. Yeah. Yeah,
Speaker 1: because again, if you were to say, all right, this
Speaker 1: chat bot is just going to talk about, um, I
Speaker 1: don't know, sporting events from last year, well, then you
Speaker 1: can prepare pretty well for that. Yeah, exactly. So it's
Speaker 1: one of those again, it's one of those things where
Speaker 1: the unrestricted nature adds in a degree of difficulty. It's
Speaker 1: time for us to take a quick break, but will
Speaker 1: be right back. So why would you need to make
Speaker 1: the qualification that this is an open ended approach and
Speaker 1: this is the first chat bot to manage it. That
Speaker 1: would be because despite what you may have heard, Eugene
Speaker 1: Goosband was not the first program to beat the Turing test,
Speaker 1: not by a long shot. So work on these sort
Speaker 1: of chatbots, these kind of artificial conversationalists. That's real recent, right,
Speaker 1: I mean they just started doing that like what like
Speaker 1: like maybe three or four years ago or two thousand,
Speaker 1: one of the earliest, I mean that's when they started
Speaker 1: with Goose. Yeah. No, in the nineteen sixties and seventies.
Speaker 1: Say what back in the mid nineteen sixties there was Eliza,
Speaker 1: which was written by Joseph Weisenbaum. If you're pronouncing it
Speaker 1: in the correct German, that's correct. Excellent, I'm finally learning.
Speaker 1: I'm sure. I'm sure he pronounces at Wisenbaum. But yeah,
Speaker 1: but no, it would be Weisenbaum at any rate. Um,
Speaker 1: this this was a program um that would respond to
Speaker 1: human conversation in what ideally would be a relevant way. Yes,
Speaker 1: it was obviously an early attempt. It was not meant
Speaker 1: to be a program that takes on the Turing test.
Speaker 1: It was really a thought again, kind of like a
Speaker 1: thought experiment, the idea of what does it take to
Speaker 1: create a piece of software that can react to questions
Speaker 1: and make it make sense. At that point, it was
Speaker 1: more a can we do this? Then let's do this
Speaker 1: for real? Yeah, and these are the sort of foundations
Speaker 1: that you have to lay in order for other things
Speaker 1: like the Eugene Goostman program to be uh successful. So
Speaker 1: he created a language analyzer. Now this specifically would look
Speaker 1: at words that users would put in and then compare
Speaker 1: them against a database of words that were stored in
Speaker 1: the computer's memory. And then also created scripts. Now, in
Speaker 1: this case, the scripts were sets of rules. They're kind
Speaker 1: of like, you know, like a protocol or an algorithm
Speaker 1: in a way. These rules dictated how Eliza would respond
Speaker 1: to messages in order to cut down on that huge,
Speaker 1: massive number of variables we were talking about, that the
Speaker 1: whole unrestricted, unpredictable thing, And so they would have different
Speaker 1: uh kind of like like overlays. Think of him as
Speaker 1: an overlay that would kind of guide Aliza's responses, And
Speaker 1: the most well known one was called Doctor, which put
Speaker 1: Eliza in the role of a Rogerian psychiatrist. Uh. This
Speaker 1: is the person who responds to everything with a question.
Speaker 1: Al right. It's that it's that passive interview style where
Speaker 1: where you know, you repeat back. You know, if if
Speaker 1: if I go, oh, man, like I'm I'm I'm really
Speaker 1: sad about my cat, Oh, tell what is it about
Speaker 1: your cat that makes you sad? And then you can
Speaker 1: say things like you know, And that's one of those
Speaker 1: things that where as a conversation starts to wind down,
Speaker 1: you then have another line of questions, So tell me
Speaker 1: more about your mother, like that the whole tell me
Speaker 1: about your mother thing. That's really coming back to this
Speaker 1: kind of model of psychiatrists. But yeah, if you've ever
Speaker 1: heard the joke about them responding to anything with another question,
Speaker 1: so I just just taking the last word and turning
Speaker 1: it into a question what am I paying you for?
Speaker 1: Why do you think you're paying for? That kind of thing? Uh?
Speaker 1: So that's what that's how Aliza spotted and in fact,
Speaker 1: you can find examples of the Eliza proscript transcripts, and
Speaker 1: even even the actual there's there. There are ports you
Speaker 1: could say, or people have essentially created their own version
Speaker 1: of Eliza just using the originalize as a guide. You
Speaker 1: can find tons of them on the web, and you
Speaker 1: can attempt to have a conversation with them. It's not
Speaker 1: terribly compelling, but it's it's kind of fun. Usually within
Speaker 1: maybe four or five exchanges, you've already run into something
Speaker 1: where you're like, well, this can't be a person, or
Speaker 1: if it's a person, it's the weirdest person I've ever
Speaker 1: conversed with. But but again, it wasn't really an earnest
Speaker 1: attempt to to create something that would pass the quote
Speaker 1: unquote tearing tests, right, which I don't think was being
Speaker 1: referred to as such at that point yet. Yeah, it
Speaker 1: was kind of people knew about tourings prediction, but it
Speaker 1: wasn't so much called a Turing test. Another chat bought
Speaker 1: that premiered in the early nineteen seventies went even further
Speaker 1: and actually was an attempt to try and pass the
Speaker 1: Turing test in a very specific approach, kind of like
Speaker 1: you know, Eugene Goostman is a very specific approach to
Speaker 1: narrow down that those parameters. In this case, this one
Speaker 1: was called Perry p A R R y A. Right,
Speaker 1: and Kenneth Colby created it too to emulate a patient
Speaker 1: who has paranoid schizophrenia. Yeah, someone who has you know,
Speaker 1: sort of a persecution complex. Uh. They imagined that that
Speaker 1: there are people or other entities that are out to
Speaker 1: get them, and in this uh specific case, he kinda
Speaker 1: he kind of really embraced this approach. It reminds me
Speaker 1: of people who create a like a Dungeons and Dragon's character,
Speaker 1: but then give their character an entire backstory. Yeah. Yeah, yeah,
Speaker 1: this this Perry persona was an entire persona. It was
Speaker 1: a twenty eight year old man with a job as
Speaker 1: a post office clerk who was single, had no brothers
Speaker 1: and sisters, and rarely saw his parents. He had specific hobbies.
Speaker 1: He liked to go to the movies and horse racing. Yeah,
Speaker 1: he liked to bet on the horses. Uh. And he
Speaker 1: placed about with bookies in the past, right, and that
Speaker 1: he realized later bookies have an association with the criminal
Speaker 1: underworld and that therefore the mafia knew about him and
Speaker 1: we're out to get him. Now, now all of that
Speaker 1: might sound very ridiculous to you if you never had
Speaker 1: any kind of interaction with someone who suffers from paranoid schizophrenia,
Speaker 1: that can seem like, well, that seems cartoonish. But no, this,
Speaker 1: this kind of thinking is not uncommon, you know, whether
Speaker 1: it's whether it's a criminal organization or the government or
Speaker 1: some other even unnamed entity. Oh absolutely, uh, it's it's
Speaker 1: very much realistic in terms of that kind of diagnosis.
Speaker 1: And and if I was speaking about it with humor
Speaker 1: in my voice a moment ago, it's it's only because
Speaker 1: I am absolutely tickled that the programmers of this of
Speaker 1: this program built it, built it in. Yeah, I mean
Speaker 1: it's it's actually pretty entertaining that they went so far
Speaker 1: as to make this whole, uh, this whole backstory to explain,
Speaker 1: because that's what gives it the believability, and then they
Speaker 1: ended up testing it in conversation with a human. Perry
Speaker 1: would gradually start to introduce his thoughts quote unquote thoughts
Speaker 1: about being persecuted, and would respond sensitively to anything said
Speaker 1: about his appearance, family, or religious beliefs. I've actually seen
Speaker 1: lots of transcripts of conversations with Perry, and sure enough,
Speaker 1: it's one of those things where you know, you might
Speaker 1: have a few exchanges and then Perry ends up saying
Speaker 1: something that seems really odd, but not so odd as
Speaker 1: to seem artificial. It just seems like it's a non sequitur,
Speaker 1: you know, something like, well, that's what they want you
Speaker 1: to think. And if you were to say, who are
Speaker 1: they the mafia? Who did you think I was talking about?
Speaker 1: Like the mafia after you? Of course they're after me.
Speaker 1: They know who I am. That kind of stuff, and
Speaker 1: it's disturbing, like it's you know, when you know what
Speaker 1: it is, it's kind of amusing, but like if you're
Speaker 1: in the middle of a conversation, join, oh this poor person. Yeah.
Speaker 1: And so in order to test it, Kolby did a
Speaker 1: couple of different things. Did one test where eight psychiatrists
Speaker 1: interviewed both Perry and a human patient via teletypewriter. So
Speaker 1: in both cases the the the psychiatrists could not see
Speaker 1: who they were interviewing. This is going back to the
Speaker 1: kind of the original Touring test idea, or at least
Speaker 1: Touring's proposed experiment, and in this case, only two of
Speaker 1: the eight were able to identify that one of the
Speaker 1: interviewees was human. And the other was a machine. In
Speaker 1: a second test, Colby presented a group of a hundred psychiatrists,
Speaker 1: almost a psychics, transcripts of interviews between Perry changed the Yeah,
Speaker 1: he had he had pairs of exactly I foresee no.
Speaker 1: But they gave these psychiatrists transcripts of interviews between an
Speaker 1: interviewer and Perry, and an interviewer and a human patient,
Speaker 1: and forty out of the responded. I don't know if
Speaker 1: the other sixty just never got it or if they didn't.
Speaker 1: Response rates are variable, right, so I'll the forty who responded,
Speaker 1: nineteen of them guessed incorrectly. So that's almost a fifty percent,
Speaker 1: you know, getting you know, right up there with a
Speaker 1: pretty impressive amount. Now, again, we have to look at
Speaker 1: the fact that Perry is operating under a very restricted,
Speaker 1: uh set of rules. We're talking about paranoid schizophrenic, someone
Speaker 1: who we would assume would occasionally an non normative answers
Speaker 1: exactly two conversational pieces. And and again it's it's a
Speaker 1: limited time that you're having with this person or this
Speaker 1: entity in this case, this this program, and uh. Because
Speaker 1: the psychiatrists had a specific expectation of the type of
Speaker 1: interactions they were going to see that could have affected
Speaker 1: their their uh answer right. So ideally, in the perfect situation,
Speaker 1: you would have this interview happening where you have no
Speaker 1: expectation as to what the answers should be. In other words,
Speaker 1: you don't know ahead of time that the interviewee is
Speaker 1: having any kind of other, you know, any kind of
Speaker 1: restrictions upon that person, so that you would be interviewing
Speaker 1: anyone like any average person. But that's obviously not what
Speaker 1: we're talking about here, nor was it the one for
Speaker 1: Eugene Goost Bond. So I've also seen, by the way,
Speaker 1: transcripts of people who set up Eliza and Perry to
Speaker 1: talk to each other. So you have Eliza acting as
Speaker 1: the Rogerian psychiatrist and Perry the paranoid schizophrenic, having bizarre conversations,
Speaker 1: and they usually don't last very long because Perry gets upset,
Speaker 1: And obviously by gets upset, I just mean that Perry
Speaker 1: ends up essentially shutting down the conversation because Eliza just
Speaker 1: wants to ask questions and Perry gets suspicious of people
Speaker 1: who are asking questions, and by again get suspicious, I'm
Speaker 1: saying following specific rules that make it feel like this.
Speaker 1: Computer programs getting suspicious, but they are entertaining. If you
Speaker 1: ever do a search on line, just look for Eliza
Speaker 1: Perry transcripts. There. There are a few of them, and
Speaker 1: they're all pretty entertaining. But so since since that time,
Speaker 1: I mean obviously, lots and lots of chatbots have been
Speaker 1: created for multiple reasons. Oh yeah, well, you know, some
Speaker 1: of them are trying to test the Turing test, and
Speaker 1: others are trying to fool you into giving out your
Speaker 1: credit card information or plaining on a link that has
Speaker 1: malicious uh links to malicious software. Yeah, anyone who's been
Speaker 1: on any kind of chat program, specifically like AIM has
Speaker 1: probably encountered this at least once or twice, where they're
Speaker 1: they're getting an unsolicited message from someone or something or something. Yeah,
Speaker 1: if you type a couple of times, you realize, oh,
Speaker 1: this is not actually this is an attempt for to
Speaker 1: either get information from me or have me click on
Speaker 1: a link. Yeah, that's a that's a thing. But there
Speaker 1: are some examples of uh, I guess saying legitimate is weird,
Speaker 1: but there are some examples of they're more and more
Speaker 1: scholarly attempts. Yeah, like a PC therapist that one was
Speaker 1: by Joseph Weintraub, and it fooled of its judges into
Speaker 1: thinking that it was human. Of ten judges, of course,
Speaker 1: so like, if I'm doing my math correctly, that's five
Speaker 1: that I believe you are. Unless we're talking about quantum judges. Um,
Speaker 1: these judges were both right and wrong at the same time.
Speaker 1: And um, and it was it was a whimsical program.
Speaker 1: I guess you could say, yeah, it had come some
Speaker 1: I think it's fair to say its answers could be
Speaker 1: pretty smart ass. Also, I read some of these transcripts
Speaker 1: and it actually surprised me that enough judges thought that
Speaker 1: it was a person. Maybe they thought it was a
Speaker 1: person who was purposefully attempting to fool them into thinking
Speaker 1: it was a computer, huh, you know, because it would
Speaker 1: say things like I compute, therefore I am that kind
Speaker 1: of stuff where it was specifically yeah. So you're looking
Speaker 1: at this stuff and you're thinking, all right, well, maybe
Speaker 1: this is kind of going back to that original touring
Speaker 1: test party game, where the the effort is for a
Speaker 1: person like the person who's being interviewed is trying to
Speaker 1: throw everybody off, and that's perfectly within the rules, unless
Speaker 1: you stay upfront. No, just be honest in your in
Speaker 1: your answers. There's nothing in here, by the way that
Speaker 1: says that the interviewee has to tell the truth necessarily,
Speaker 1: unless you just state that as a parameter at the beginning.
Speaker 1: So in other words, you could you could be like,
Speaker 1: I'm totally the computer and you're the human being interviewed. Uh.
Speaker 1: I don't know if that's a fair way of saying
Speaker 1: that the device won or lost, but it is a possibility. Uh.
Speaker 1: Then we have a two thousand eleven. Now, this one
Speaker 1: is a really pretty impressive one. And this was clever Butt,
Speaker 1: which was made by a fellow named Rollo Carpenter, and
Speaker 1: it fooled fifty nine point three percent of a live
Speaker 1: audience and an event in India with more than a
Speaker 1: thousand people. Yeah, the way this worked was that the
Speaker 1: audience watched as interviewers interacted via text with either clever
Speaker 1: Bot or a human in the course of a four
Speaker 1: minute interview. So it's a little shorter than what Turing
Speaker 1: had said, but not by not by a whole lot.
Speaker 1: In four mints is still a good amount of time.
Speaker 1: Sure that that is twenty less, that's true, that's true.
Speaker 1: So keep that in mind. But but at any rate,
Speaker 1: it was you know, pretty interesting experience. And also from
Speaker 1: why I read they misidentified, they thought that the human
Speaker 1: was a computer sixty of the time. Um, because they
Speaker 1: didn't necessarily just say that it was a computer or
Speaker 1: it was. So now we see that there are a
Speaker 1: few examples of chat bots quote unquote passing the Turing test.
Speaker 1: So what does that mean. Does it mean that the
Speaker 1: machines are actually thinking? Um? No, I mean, and it's
Speaker 1: not to say that that computers don't have a certain
Speaker 1: amount of machine intelligence, but there's absolutely a distinction between
Speaker 1: that and what we consider to be human intelligence. That's true.
Speaker 1: The programmers themselves have said that this doesn't mean a
Speaker 1: machine is able to think. Uh, they're just able to
Speaker 1: interpret commands then to follow a set of rules to
Speaker 1: make a response, which is still pretty cool. And and
Speaker 1: it doesn't It certainly doesn't mean that the Turing test
Speaker 1: is worthless as an exercise. No, it is. In fact,
Speaker 1: it's improving our ability to create programs that that can
Speaker 1: understand or at least respond to natural language. Natural language
Speaker 1: recognition is one of those big things where if you're
Speaker 1: really able to crack it, then you can have some
Speaker 1: amazing opportunities open up, and we've seen this recently with
Speaker 1: stuff like Siri. Oh absolutely being able to speak to
Speaker 1: your computer rather than having to I mean, even if
Speaker 1: you could speak to your computer through a keyboard and
Speaker 1: have it understand what you're what you're saying, I mean,
Speaker 1: like it's it's the reason why Google spends so much
Speaker 1: time and money, and it's search algorithms of of trying
Speaker 1: to figure out what you really mean when you search
Speaker 1: for a certain phrase, because traditionally, you know, before we
Speaker 1: really got into the natural language recognition era, it meant
Speaker 1: that in order to work with a computer, you had
Speaker 1: to work with a computer on the computer's terms. You
Speaker 1: had to learn the commands, you had to learn the
Speaker 1: way to navigate a computer system in order for it
Speaker 1: to do what you wanted it to do. Once you
Speaker 1: get to a point where natural language recognition software is
Speaker 1: robust enough the computer is working on your terms, you
Speaker 1: you can put in however you're thinking, like whatever, whatever
Speaker 1: mental exercise you've gone through to ask this computer to
Speaker 1: do something, whatever you do to kind of express a thought,
Speaker 1: uh as a command to this computer, the computer can
Speaker 1: then interpret then respond and it's not just for serving
Speaker 1: you back whatever information you happen to be looking for.
Speaker 1: It's I mean, I mean, we're talking about being able
Speaker 1: to just look at a computer and say, you know,
Speaker 1: I really want a graph that looks blue and has
Speaker 1: these percentages in it and is about this thing, and
Speaker 1: it just doesn't. Yeah, like I want to see what
Speaker 1: the population distribution of Atlanta is in a bar chart
Speaker 1: or something, and then it could bring that shows out,
Speaker 1: finds that information, put it into a bar chart, and yeah,
Speaker 1: that's pretty phenomenal stuff. We have a little bit more
Speaker 1: to say about passing the Turing test, but before we
Speaker 1: get to that, let's take another quick break. We see
Speaker 1: other examples of machine intelligence everywhere, things like pattern recognition,
Speaker 1: probabilistic predictions, for example, Pandora. You know the Music Genome project.
Speaker 1: It's yeah, that's that's pattern recognition. Yeah, it's looking for
Speaker 1: elements of songs that you say you like and then
Speaker 1: looking for other stuff that's not in the specific category
Speaker 1: you mentioned or the specific examples you mentioned, and there's
Speaker 1: something else you probably will like because you like these
Speaker 1: other things that also have this stuff in it. Uh,
Speaker 1: you know, sometimes that's less functional than other times. It
Speaker 1: makes me think of Patton Oswald has a great routine
Speaker 1: about TiVo that does the same sort of thing where
Speaker 1: he says, you know, TiVo is great. I mean, I like,
Speaker 1: I love Westerns, and I'll have it set to TiVo
Speaker 1: a Western for me, and I come back, and then
Speaker 1: they'll be all these other Westerns that will be suggested.
Speaker 1: I didn't even know about. Thank you, TiVo. But then
Speaker 1: sometimes TiVo gets it wrong and I come back and
Speaker 1: everything has got horses in it, because Westerns have horses
Speaker 1: in it. So I've got my little pony and cartoons
Speaker 1: with horses and unicorns and things, and I have to say, no, TiVo,
Speaker 1: that's a bad TiVo. But TiVo says, but you said
Speaker 1: you liked horses. Same sort of thing. Like when you
Speaker 1: get more sophisticated than the the computer program starts to
Speaker 1: anticipate things and makes these probabilistic models. These these models
Speaker 1: where there there are certain percentages associated with various responses,
Speaker 1: and it goes with whichever one seems to be the
Speaker 1: most prevalent, assoning it meets a threshold. If this sounds
Speaker 1: familiar to you, it's because that's how IBM S. Watson worked, right,
Speaker 1: which is a really good example of natural language recognition, absolutely,
Speaker 1: because not only was it able to recognize natural language,
Speaker 1: it had to interpret things like wordplay that Jeopardy do.
Speaker 1: This is the machine that went up on Jeopardy and
Speaker 1: beat the returning champions or former champions. Uh And you know,
Speaker 1: if you've ever played Jeopardy or watched Jeopardy, you know
Speaker 1: that there are categories that depend on things like puns
Speaker 1: or hominem's or other forms of word play. So it
Speaker 1: has to parse all of that, and that's even more
Speaker 1: complicated than just taking a simple sentence and figuring out,
Speaker 1: all right, what are the potential responses to whatever this
Speaker 1: this this phrase is so great, great example. You know,
Speaker 1: they would end up coming up with a potential answer.
Speaker 1: It would assign a percentage of how quote unquote sure
Speaker 1: it was that that was the right answer, and if
Speaker 1: the percentage was higher than its threshold, which I think
Speaker 1: was something like that, it would buzz in and give
Speaker 1: that as a guess. Sometimes it was wrong, but it
Speaker 1: was right a lot of the time, so that's kind
Speaker 1: of cool. Uh So, getting back to Eugene Gene, the
Speaker 1: main machine as I called him in in my notes
Speaker 1: at one point. Uh And of course, I'm anthropomorphizing when
Speaker 1: I say him, it's a it's a it's an it. Well,
Speaker 1: it has a dude name. Yeah, it has a dude
Speaker 1: name and a dude persona. But it's ultimately, isn't it.
Speaker 1: Would you say that perhaps some of the reporting around
Speaker 1: this was was maybe a little misleading or at least
Speaker 1: hype is well, you know, Okay, the entire Eugene Gooseman
Speaker 1: chat bought sounds really cool. I haven't met it personally, No,
Speaker 1: I haven't either, although you can. There is an Internet version,
Speaker 1: and I'm not sure that it's the same version that's
Speaker 1: being used in competition, because I've seen some transcripts from
Speaker 1: the Internet version and they don't seem good at all.
Speaker 1: They seem bad, right, I guess you'd have to talk
Speaker 1: to some actual thirteen year old boys from actually, yeah,
Speaker 1: that this is part of it. You know. There's certainly
Speaker 1: been some some questions among uh natural language AI enthusiasts
Speaker 1: online about whether we're really just lowering our expectations for
Speaker 1: human communication. Which, yeah, that's that's a totally different way
Speaker 1: of looking at and a depressing one to say that, Oh, well,
Speaker 1: if you come from this place and if you are
Speaker 1: of this age, then I expect you to only be
Speaker 1: able to communicate at this level, right, which is depressing.
Speaker 1: It certainly is well, but you know, but it's a
Speaker 1: valid point, I think. I think it's a good thing
Speaker 1: to be thinking about in this kind of situation. UM.
Speaker 1: You know. Beyond that though, and I certainly don't want
Speaker 1: to downplay the apparent achievements of its programmers, because I
Speaker 1: haven't programmed any capable chatbots today, I it's been ever
Speaker 1: since I did. But there are a few things that
Speaker 1: are just a little bit shady about the news UM.
Speaker 1: First off, the original press release, which came out of
Speaker 1: the University of Reading, I believe so UM stated that
Speaker 1: a quote supercomputer had achieved this feat, and perhaps charitably,
Speaker 1: it was a mistake or misunderstanding on the part of
Speaker 1: the writer of the press release, but some skeptics have
Speaker 1: suggested that it was in fact a purposeful publicity play
Speaker 1: that in fact worked, because a whole lot of news
Speaker 1: headlines around the interwebs repeated the error very excitedly. Yes,
Speaker 1: because it was not a supercomputer. It was a computer
Speaker 1: running a piece of software. It's a program, Yes, the
Speaker 1: program that did the work. I mean, the computer just
Speaker 1: provided the horsepower. Right, it's the software that did all
Speaker 1: the action work, and it was not It wasn't on
Speaker 1: a supercomputer. Little little known fact. Supercomputers have better things
Speaker 1: to do than run chat bought software generally. Yeah. Yeah,
Speaker 1: we're talking about things like figuring out global weather changing
Speaker 1: change patterns and things like that, you know, or or
Speaker 1: the way that money works. Chat bots low on their
Speaker 1: priority list. It's like number seven at least. Um And
Speaker 1: and also that this press release in question was largely
Speaker 1: a quotation from Dr Kevin Warwick. Kevin Warwick, of course
Speaker 1: being the fellow who organized this entire competition. Um who
Speaker 1: who's an engineer and a futurist, um and also the
Speaker 1: instigator and or enjoyer of a certain amount of hype
Speaker 1: and debate about future technologies. Yeah, he is, obviously you
Speaker 1: can tell. This is the guy who elected to have
Speaker 1: surgery performed on him so he could have that r
Speaker 1: F I D chip and call himself aside Wars. This
Speaker 1: is someone who not only embraces this these ideas of futurism,
Speaker 1: but is actively trying to promote them and get to
Speaker 1: them that we're not even saying that that's a bad thing.
Speaker 1: What we are saying, is that that may give him
Speaker 1: somewhat of a bias when it comes to proclaiming a
Speaker 1: computer software piece of computer software being an amazing achievement
Speaker 1: that beat the Turing test, right sure. I mean he
Speaker 1: admits basically to to being a provocateur. He he says
Speaker 1: that that's really his job, you know, it is to
Speaker 1: get people excited about tech and engineering and in the future.
Speaker 1: And we get that like that that we agree that's
Speaker 1: our job too. We think it's red. We think that
Speaker 1: perhaps I don't want to put words into your mount Lauren.
Speaker 1: I think perhaps that there's a different way of going
Speaker 1: about it where you can still be excited, but you
Speaker 1: can be a little more grounded in the way you
Speaker 1: present things, because I I also think the achievement of
Speaker 1: creating a chatbot that could be uh convincing is a
Speaker 1: fantastic achievement. I mean, it's something that, even under any
Speaker 1: number of qualifications, incredibly challenging to do, no matter how
Speaker 1: you frame it. Um. I do think, however, that if
Speaker 1: you seem to over inflate the achievement, you run the
Speaker 1: danger of making people feel jaded about it later, which
Speaker 1: I think Computer Wolf exactly. Yeah, exactly. So it's one
Speaker 1: of those things where you know, you have to take
Speaker 1: the context into account, right and don't don't downplay the achievement,
Speaker 1: but don't sit there and say, like, ah ha, now
Speaker 1: we have intelligent computers everywhere. That's not that's not true either.
Speaker 1: I saw there's a great Wired article that specifically went
Speaker 1: into uh, kind of debunking the whole beating the Turing
Speaker 1: test thing and again kind of saying the same thing
Speaker 1: we're saying, like, take the context into account, and and
Speaker 1: part of that article. They ended up asking a cognitive
Speaker 1: scientist named Gary Marcus of n y U about this,
Speaker 1: and Marcus proposed a new version of the Turing test
Speaker 1: because he says the old version is not really a
Speaker 1: measurement of machine intelligence. Uh, it does kind of illustrate
Speaker 1: ways of creating natural language recognition and clever ways to
Speaker 1: fool the human side, huh. And that it was very
Speaker 1: valid historically at the time because you know, text textual
Speaker 1: communication was new and exciting and it was you know,
Speaker 1: it pushed the field forward, it really did. But now
Speaker 1: we've gotten to a point where fooling the person on
Speaker 1: the other side of a keyboard is not necessarily the
Speaker 1: goal that we should be looking at. He proposes, the
Speaker 1: next version of the Turing test should be that a
Speaker 1: computer software, uh, like any kind of program that wants
Speaker 1: to beat it, what it has to do is first
Speaker 1: quote unquote watch a movie, television show, YouTube video something,
Speaker 1: some kind of video media, and then be able to
Speaker 1: respond to questions about it. So sort of like here,
Speaker 1: let me show you this ten minute video on car safety,
Speaker 1: and then asking questions about specifically about the video, what
Speaker 1: happened after they fastened the seat belt, that kind of thing,
Speaker 1: and if the computer program is able to answer it,
Speaker 1: then that would be a much more convincing touring test
Speaker 1: than just kind of spewing out a script, which is,
Speaker 1: you know, it's adding another layer of difficulty on top
Speaker 1: of an already difficult task. But that's that's the whole
Speaker 1: only way you can go forward. Otherwise we're just going
Speaker 1: to see increasingly sophisticated chat box. Yeah, and and that
Speaker 1: is actually a very difficult and interesting problem. What wasn't
Speaker 1: it just recently that there was a computer that we
Speaker 1: we taught I mean not us personally, but that humanity,
Speaker 1: some researchers and we're taught taught to identify cats pictures
Speaker 1: of cats. That was the AI program that essentially went
Speaker 1: through thousands and thousands of uh I think it was
Speaker 1: images and videos and then became able to identify cats.
Speaker 1: It essentially defined what a cat was because no one
Speaker 1: taught it right. It learned what cats are based upon
Speaker 1: their appearance, on their appearance and can look at pictures
Speaker 1: of cats and say that is totally a cat, essentially
Speaker 1: saying that thing that is in that video is the
Speaker 1: same as this other thing that's in this picture. That's
Speaker 1: the same as this other thing that's in this totally
Speaker 1: different video, which sounds trivial and hilarious, and it kind
Speaker 1: of is hilarious. Also easy because I mean, come on,
Speaker 1: everything on the internet has cats in it. Yeah, that
Speaker 1: is That is kind of a gimme, isn't it? But
Speaker 1: but still no, it is. It is cool. I mean
Speaker 1: because just just that level of image recognition, I mean
Speaker 1: being able to take an object and look at it
Speaker 1: from a different angle than you were taught or that's
Speaker 1: a different color, different different size. Yeah, all these things.
Speaker 1: All of these things are easy for us, hard for computers,
Speaker 1: so seeing something make that breakthrough is really exciting. Anyway,
Speaker 1: we thought we would take that story and kind of
Speaker 1: break it down for you guys, explain how it's still
Speaker 1: cool but maybe not as cool as the way some
Speaker 1: of the headlines are saying. Right, and also say hey
Speaker 1: internet journalists, UM, step up your game. Yeah, I understand
Speaker 1: that you want people to read your stuff. Oh yeah,
Speaker 1: And and you're under deadline pressure and that's terrible. But
Speaker 1: let's look hard. Let's represent reality, shall we don't just
Speaker 1: don't just spit out press releases the way that you
Speaker 1: found them. Yeah, and uh, and I would be criminal
Speaker 1: to to neglect to mention Noel. Our producer reminded me
Speaker 1: that obviously this is a very important field of study
Speaker 1: because we want to be able to tell the difference
Speaker 1: between computers and humans when the future of Blade Runner
Speaker 1: becomes our reality and you're chasing down a replicant and
Speaker 1: you have to determine if it's actually a replicate or
Speaker 1: human being. And that wraps up this classic episode about
Speaker 1: passing the Turing Test. Honestly, the whole Turing Test thing,
Speaker 1: that general concept has morphed over the decades, and it's
Speaker 1: it's more like shorthand right. Passing the Touring test isn't
Speaker 1: so much about passing a specific hypothetical test. It's more
Speaker 1: about this idea of creating a machine that of here's
Speaker 1: to be indistinguishable from a human as far as you know,
Speaker 1: processing and communicating information so that you feel like the
Speaker 1: machine is truly intelligent. I think the turning test is
Speaker 1: one of those concepts that changes over time, and and
Speaker 1: it's more like you start to find out what it's
Speaker 1: not rather than what it is. It's an interesting things,
Speaker 1: very much like our artificial intelligence itself, or even machine consciousness.
Speaker 1: These are all concepts that are are kind of wibbly wobbly.
Speaker 1: As dr who might say, well, if you have suggestions
Speaker 1: for topics I should cover in future episodes of tech Stuff,
Speaker 1: reach out to me. The best way to do that
Speaker 1: is over on Twitter. The handle for the show is
Speaker 1: text Stuff HSW and I'll talk to you again really soon.
Speaker 1: Text Stuff is an I Heart Radio production. For more
Speaker 1: podcasts from my Heart Radio, visit the i Heart Radio app,
Speaker 1: Apple Podcasts, or wherever you listen to your favorite shows.