Speaker 1: Get in test with technology with text stuff from dot com.
Speaker 1: Say everyone, and welcome to text Uff. I'm Jonathan Strickland,
Speaker 1: and uh, you know, something hit the news recently. You
Speaker 1: may have seen it, in fact, you may have seen
Speaker 1: it in multiple places about a program, a computer program
Speaker 1: passing the Turing test, possibly the first such program to
Speaker 1: ever pass the Turing test ever. And we had planned
Speaker 1: on doing an episode on this, and then on top
Speaker 1: of that, one of our listeners, Nick on Twitter said,
Speaker 1: just saw this headline on Google News. A computer just
Speaker 1: passed the Turing test in landmark trial. So we knew
Speaker 1: that the timing was perfect. Thank you Nick for writing in.
Speaker 1: We are really ready to talk about this, but in
Speaker 1: order to do that, it's important that we, you know,
Speaker 1: kind of lay some groundwork. So first of all, the
Speaker 1: Touring test is named after Alan Turing, and we've done
Speaker 1: a full episode on Alan Turing way back when, back
Speaker 1: in November of Yeah. Phenomenal person, amazing thinker. Yeah, one
Speaker 1: of the like the grandfather of computer science. Also tragic
Speaker 1: life story which we went into detail back in that story.
Speaker 1: And so you might wonder why, right, So what does
Speaker 1: that have to do? Was the guy who was the
Speaker 1: essentially the father of computer science or grandfather of computer science.
Speaker 1: What does he have to do with the story about
Speaker 1: a computer program in June two thousand fourteen, a computer
Speaker 1: program with an interesting name, Eugene Goostman passing the Turing test.
Speaker 1: What does that all have to do with each other? Well?
Speaker 1: To answer that, we have to ask what is a
Speaker 1: Turing test? Well, as it turns out Away, back in
Speaker 1: the nineteen fifties, he started envisioning uh I thought experiment. Yeah,
Speaker 1: he published a paper in a journal called Mind in
Speaker 1: ninety fifty called Computing, Machinery and Intelligence, and he sort
Speaker 1: of laid out his thought experiment there. And it's interesting
Speaker 1: because what he did was he took this idea of
Speaker 1: a party game and then adapted it for computers. Right.
Speaker 1: The party game that it's based on would have three participants,
Speaker 1: an interrogator, a man, and a woman um all situated
Speaker 1: so that they can't see one another, and the interrogator
Speaker 1: is supposed to ask questions to try to figure out
Speaker 1: which of the participants is the dude and which is
Speaker 1: the lady exactly, And it's the dude's job to try
Speaker 1: and mislead the interrogator to believe that that he, in fact,
Speaker 1: is the lady and the other one is the man.
Speaker 1: It's the lady's job to say, hey, I want you
Speaker 1: to get this right. I'm the lady. That other guy
Speaker 1: that's the dude. And so the interrogator has to ask questions. Now,
Speaker 1: obviously they can't see each other. Because they could see
Speaker 1: each other, then that would probably give things away. Probably,
Speaker 1: they really shouldn't be able to hear each other because
Speaker 1: that could also give things away. Sure, and if you
Speaker 1: use handwritten notes, then you the interrogator might be able
Speaker 1: to make judgments based on the handwriting style. So it
Speaker 1: should really be typewritten, right, So you want to you
Speaker 1: want to remove as many easily identifiable traits from this
Speaker 1: game as possible to make it all about the questions
Speaker 1: and the answers. Now, Touring said, what if we were
Speaker 1: to take the same basic premise, but instead of having
Speaker 1: two human interviewees, replace one of those humans with a machine. Now,
Speaker 1: if that machine can convince the interrogator that the machine
Speaker 1: itself is a human being, it would be a pretty
Speaker 1: phenomenal achievement. Or you know, just generally, if the interrogator
Speaker 1: wasn't sure which of the two interrogate ease, right, which
Speaker 1: one is the human, which one is the machine? Or
Speaker 1: even if there could be a case where you don't know.
Speaker 1: It may be that you have two humans that you're interrogating,
Speaker 1: and it may be one of those things where you
Speaker 1: have to you know, if you don't know for a
Speaker 1: fact that one of them the machine, that makes it
Speaker 1: even hard, right, at least assuming that the computer program
Speaker 1: is sophisticated enough. Now, Touring was saying that we don't
Speaker 1: have any machines right now that can do this, but
Speaker 1: I envision a time when computers will be able to
Speaker 1: do such a thing where if you were to interrogate
Speaker 1: a computer, you would get back responses that would be uh,
Speaker 1: convincing enough for it to make it difficult to determine
Speaker 1: if it were man or machine. And so he said
Speaker 1: that he predicted in fifty years time, which would be
Speaker 1: the year two thousand, that computers and software be sophisticate
Speaker 1: enough that interrogators would only be able to guess correctly
Speaker 1: seventy of the time, meaning they would be fooled by
Speaker 1: the computer of the time. Right, So, uh, this was
Speaker 1: just kind of a thing he was coming up with,
Speaker 1: like an idea, a prediction not necessarily a test, although
Speaker 1: very much like Moore's observation became Moore's law, this became
Speaker 1: what is known as the touring test. People talk about
Speaker 1: a touring test as a machine capable of fooling people
Speaker 1: into thinking it's another person at least thirty of the
Speaker 1: time after five minutes of conversation. Very good point. Yes,
Speaker 1: it needs to be five minutes of conversation. If you
Speaker 1: are just getting maybe two or three responses, that might
Speaker 1: not be enough for you to be able to draw
Speaker 1: a conclusion that you feel good about. If after five
Speaker 1: minutes you still are not entirely, entirely certain, then that
Speaker 1: might say that this machine, in fact has passed the
Speaker 1: touring test. And this has been extrapolated to mean something
Speaker 1: about machine intelligence because Turing himself tied the idea of
Speaker 1: of how we perceive a machines intelligence directly to artificial
Speaker 1: intelligence and and and even how we perceive human intelligence.
Speaker 1: Because here's Touring's idea is a little cheeky, and I
Speaker 1: love the fact that's so cheeky. So Turing kind of said,
Speaker 1: would you say such a machine as intelligent if it
Speaker 1: appears to be intelligent, is it fair to say that
Speaker 1: is intelligent touring. So why not because I'm only able
Speaker 1: to tell that I am intelligent, right, because of my experience.
Speaker 1: I'm only able to have my own personal experience. I
Speaker 1: can't experience what someone else's life is like. All right,
Speaker 1: Sitting across from each other, Jonathan and I can only
Speaker 1: assume that the other one is intelligent, right, And it's
Speaker 1: because the other person is displaying traits that we associate
Speaker 1: with intelligence. They they seem to be able to take
Speaker 1: an information, respond to it, make decisions. And based on
Speaker 1: the fact that we ourselves also do that thing, we
Speaker 1: go ahead and say, all right, well, they clearly have
Speaker 1: the same features that I have, which includes intelligence. Uh. Now,
Speaker 1: he says, why would we not extend that same courtesy
Speaker 1: to a machine if it also appeared to display those
Speaker 1: same uh features? He says, doesn't matter if the computer
Speaker 1: is quote unquote thinking. If it's it can fool you. Yeah,
Speaker 1: if it's able to to simulate it well enough, you
Speaker 1: might as well say it's intelligent, because stimulate is probably
Speaker 1: a kinder phrase for that than fool. Yeah. Yeah, well
Speaker 1: I know it's it is fooling essentially, I mean, because
Speaker 1: ultimately you're talking about a computer programmer, who's making this happen?
Speaker 1: So nowadays we think of this as the Turing test.
Speaker 1: Can a machine of the time are more fool someone
Speaker 1: after five minutes of conversation into thinking it's a human? Now,
Speaker 1: this is really hard to do. This is yeah, this
Speaker 1: is non trivial. I mean it sounds almost simple. Yeah,
Speaker 1: the concept is simple, the execution incredibly difficult. Because here's
Speaker 1: the thing, human language varied. We have unstructured, unpredictable ways
Speaker 1: to say things like if if I were to tell
Speaker 1: you that it's a hundred degrees outside and it is
Speaker 1: humid like the humidities at and you go out there,
Speaker 1: then I'm sure all of our listeners would have slightly
Speaker 1: different ways to express their thoughts on the conditions out side.
Speaker 1: Some of them would probably contain colorful metaphors, probably mine
Speaker 1: very likely when it contain colorful metaphors, particularly if I
Speaker 1: had to be outside for any length of time. But
Speaker 1: that's the point. We would all have different ways of
Speaker 1: saying this. So how do you make a computer program
Speaker 1: able to interpret all the myriad of ways we can
Speaker 1: all express the same thought, let alone any thought. Right,
Speaker 1: this is what's referred to as natural language. Recognition, and
Speaker 1: it's a really huge problem in artificial intelligence and and
Speaker 1: and a lot of other speech related computer programming. Exactly. Yeah,
Speaker 1: this is where a computer program has to be able
Speaker 1: to parse the language so it recognizes things like this
Speaker 1: word is a noun, this word is a verb, this
Speaker 1: this word alters this other word. Uh. And not only
Speaker 1: does it need to be able to recognize it, it it
Speaker 1: needs to be able to respond in kind. Right, So
Speaker 1: if you appropriately in some way or another share, Yeah,
Speaker 1: you could have a computer program that's literally making up
Speaker 1: sentences or you know, the approximation of a sentence randomly,
Speaker 1: where it's just pulling strings of words and placing them
Speaker 1: in a sequence and then presenting them. But that wouldn't
Speaker 1: be convincing at all. If I were to say, Hello,
Speaker 1: how are you today, and Lauren was to say blue
Speaker 1: panther pickup jump down street, I'd be like what? And
Speaker 1: even that was closer to being a sentence than some
Speaker 1: of the random stuff that you would see if it
Speaker 1: was just truly completely So it also has to be
Speaker 1: able to uh endure a five minute long conversation, like
Speaker 1: we said, in order to pass the Turing test. So
Speaker 1: you can't have too much repetition or that gives it
Speaker 1: away absolutely. If you've ever been playing a video game
Speaker 1: and all of the NPCs say the same thing over
Speaker 1: and over and over again. Yeah, if if all the
Speaker 1: chat bot says is hey listen, then you're clearly playing
Speaker 1: Zelda and you're not actually having a decent conversation. That
Speaker 1: being said, there is someone I know who plays a
Speaker 1: ferry at the Georgia Renaissance Festival, and hey listen is
Speaker 1: heavily represented in a repertoire. Pretty amazing, No, it's pretty awesome,
Speaker 1: but at any rate, Yeah, So, so these are these
Speaker 1: are big problems. You have to build a database of words,
Speaker 1: you have to be able to figure out what kind
Speaker 1: of syntax are you going for. It's a wide open,
Speaker 1: huge problem, and solving this problem can be really beneficial
Speaker 1: in lots of ways. We'll talk a little bit about
Speaker 1: that later. It's it's beyond just making a program that
Speaker 1: seems human, right, That's that's one way of looking at it.
Speaker 1: But there are a lot of other benefits that come
Speaker 1: along with it, which will chat about towards the end
Speaker 1: of the show. But for right now, let's go into
Speaker 1: this story behind Eugene Goostman and whether or not it
Speaker 1: was actually the first chat bought to pass the Turing test. Yeah. So,
Speaker 1: first of all, we've got three programmers in this story
Speaker 1: of Vladimir Vassilov, Eugene Demchinko and Sarage you listen. As
Speaker 1: you may guess from their names, they all hail from
Speaker 1: Russia and the Ukraine as well, and although not all
Speaker 1: the not all of them still live there. But starting
Speaker 1: in two thousand one, they got to work on this, right, Yeah,
Speaker 1: they were trying to design a computer program that would
Speaker 1: pose specifically as a thirteen year old boy from Odessa
Speaker 1: in the Ukraine. Yeah, and that meant that they had
Speaker 1: specific parameters that they could work within. It automatically helped
Speaker 1: reduce some of that unpredictability and that lack of restriction
Speaker 1: that you would have if you were to just say,
Speaker 1: this is a fluent adult speaker of a given language. Yeah, yeah,
Speaker 1: giving him these you know, setting up the expectation from
Speaker 1: the judges that you know, this is a non native
Speaker 1: English speaker. It's it's a kid essentially, Um, you know
Speaker 1: that they'll expect him to have limited knowledge of the
Speaker 1: world and and different subject areas and a limited understanding
Speaker 1: of English. Vocabulary and grammar and all that kind of stuff.
Speaker 1: So you're already managing expectations. That's going to come into
Speaker 1: play when we talk about some of the criticisms about this,
Speaker 1: although although I do think it's a very clever way
Speaker 1: and in a lot of previous chat bots have have
Speaker 1: had similar, yes, similar kind of approaches. Yeah, because like
Speaker 1: we said, if you're to take a quote unquote pure
Speaker 1: approach to this, it's really really challenging. So yeah, by
Speaker 1: by limiting this, the judges have an idea of well,
Speaker 1: this this could be a thirteen year old boy or
Speaker 1: it could be a computer program. It means that the
Speaker 1: computer program doesn't have to be as sophisticated as one
Speaker 1: that would be completely fluent and have you know, an
Speaker 1: adult's experiences, uh and ability to communicate. So that was
Speaker 1: step one. And uh. Eugene Goosman took part in the
Speaker 1: competition UH that had five total chatbots. It was it
Speaker 1: was one of five that took place on the sixtieth
Speaker 1: anniversary of Touring's death, and the program managed to full
Speaker 1: thirty three percent of the judges into thinking it was
Speaker 1: actually a person. UH. And it had there were thirty judges.
Speaker 1: From what I understand. So that was where you got
Speaker 1: all the headlines of chat bought. There are computer beats
Speaker 1: touring tests, which already is not accurate. There were some
Speaker 1: slight um hiccups in a little bit of the news
Speaker 1: reporting that We'll get into that part of the story later,
Speaker 1: but you know, the key takeaways I think here are
Speaker 1: that this was a this was a competition that was
Speaker 1: celebrating touring awesome um and that there were thirty judges,
Speaker 1: some of whom were celebrities, yes, including an actor who
Speaker 1: had appeared on Red Dwarf right. Yes. And then a
Speaker 1: couple of years before that, because because you said they
Speaker 1: started working on this on two th two thousand one,
Speaker 1: this was not the first time that Gustman had entered competition.
Speaker 1: Two years previously, that same software had convinced twenty of
Speaker 1: judges at a similar competition that was held at Bletchley Park. Now,
Speaker 1: but Bletchley Park's where touring helped crack the Enigma machine,
Speaker 1: the the encoding device that the German military was using
Speaker 1: during World War Two. So this was a big celebration.
Speaker 1: It was at the centennial celebrating his birth, and it
Speaker 1: ended up falling just short of passing the Turing Test. Uh.
Speaker 1: The organizer of the the more recent event, the one
Speaker 1: in which Goostman ran away with quote unquote beating the
Speaker 1: Turing test. Uh. That organizer was Kevin Warwick. That name
Speaker 1: may sound familiar to some of our listeners if you've
Speaker 1: ever heard us talk about cyborgs. He's the guy who
Speaker 1: had an r f I D chip surgically implanted into him.
Speaker 1: His wife also did at one point. Yes, they could
Speaker 1: communicate with each other through them. Some unflattering news media
Speaker 1: sometimes refers to him as Captain Cyborg. Yeah. There there
Speaker 1: are some critics who say that he uh, he courts
Speaker 1: publicity in a manner that is um unbeco umming of
Speaker 1: of of a scientist, yes, really of anybody. Yes, yes,
Speaker 1: uh those are the critics who say that. By the way,
Speaker 1: I just want to make that clear. At any rate,
Speaker 1: He said that this was the first time a chat
Speaker 1: bot had passed the Turing test at an event where
Speaker 1: the conversation was open ended, meaning that they had not
Speaker 1: previously uh decided upon a specific topic or line of questioning.
Speaker 1: That the judges were allowed to say whatever they wanted
Speaker 1: to the chat bot and response, which obviously makes it
Speaker 1: harder because you have to have a much wider breadth
Speaker 1: of potential responses. Yeah. Yeah, because again, if you were
Speaker 1: to say, all right, this chat bot is just going
Speaker 1: to talk about, um, I don't know, sporting events from
Speaker 1: last year, Well, then you can prepare pretty well for that. Yeah, exactly.
Speaker 1: So it's one of those again, it's one of those
Speaker 1: things where the unrestricted nature adds in a degree of difficulty.
Speaker 1: So why would you need to make the qualification that
Speaker 1: this is an open ended approach and this is the
Speaker 1: first chat to manage it. That would be because despite
Speaker 1: what you may have heard, Eugene Goosband was not the
Speaker 1: first program to beat the Turing test, not by a
Speaker 1: long shot. So work on these sort of chatbots, these
Speaker 1: kind of artificial conversationalists. That's real recent, right, I mean
Speaker 1: they just started doing that like what like like maybe
Speaker 1: three or four years ago or two thousand, one of
Speaker 1: the earliest, I mean that's when they started with Goosman. Yeah. No,
Speaker 1: in the nineteen sixties and seventies, say, back in the
Speaker 1: mid nineteen sixties there was Eliza, which was written by
Speaker 1: Joseph Weisenbaum. If you're pronouncing it in the correct German,
Speaker 1: that's correct. Excellent, I'm finally learning. I'm sure, I'm sure
Speaker 1: he pronounces it Wisenbaum, but yeah, but no, it would
Speaker 1: be Weisenbaum at any rate. Um. This this was a
Speaker 1: program UM that would respond to human conversation and what
Speaker 1: ideally would be a relevant way. Yes, it was obviously
Speaker 1: an early attempt. It was not meant to be a
Speaker 1: program that takes on the Turing test. It was really
Speaker 1: a thought again, kind of like a thought experiment, the
Speaker 1: idea of what does it take to create a piece
Speaker 1: of software that can react to questions and make it
Speaker 1: make sense. At that point, it was more a can
Speaker 1: we do this? Then let's do this for real? Yeah,
Speaker 1: and these are the sort of foundations that you have
Speaker 1: to lay in order for other things like the Eugene
Speaker 1: Goostman program to be uh successful. So he created a
Speaker 1: language analyzer. Now this specifically would look at words that
Speaker 1: users would put in and then compare them against a
Speaker 1: database of words that were stored in the computer's memory.
Speaker 1: And then also created scripts. Now, in this case, the
Speaker 1: scripts were sets of rules. They're kind of like, you know,
Speaker 1: like a protocol or an algorithm in a way. These
Speaker 1: rules dictated how Eliza would respond to messages in order
Speaker 1: to cut down on that huge, massive number of variables
Speaker 1: we were talking about, the whole unrestricted, unpredictable thing, and
Speaker 1: so they would have different uh kind of like like overlays.
Speaker 1: Think of him as an overlay that would kind of
Speaker 1: guide Aliza's responses. And the most well known one was
Speaker 1: called doctor, which put Eliza in the role of a
Speaker 1: Rogerian psychiatrist. Uh. This is the person who responds to
Speaker 1: everything with a question. Al Right. It's that it's that
Speaker 1: passive interview style where where you know, you repeat back.
Speaker 1: You know, if if if I go, oh, man, like
Speaker 1: I'm I'm I'm really sad about my cat, Oh, tell
Speaker 1: what is it about your cat that makes you sad?
Speaker 1: And then you can say things like you know, And
Speaker 1: that's one of those things where as a conversation starts
Speaker 1: to wind down, you then have another line of questions,
Speaker 1: So tell me more about your mother, like that the
Speaker 1: whole tell me about your mother thing. That's really coming
Speaker 1: back to this kind of model of psychiatrists. But yeah,
Speaker 1: if you've ever heard the joke about them responding to
Speaker 1: anything with another question, just just taking the last word
Speaker 1: and turning it into a question, what am I paying
Speaker 1: you for Why do you think your page for that
Speaker 1: kind of thing? Uh So that's what That's how Eliza responded,
Speaker 1: And in fact, you can find examples of the Eliza
Speaker 1: program script transcripts and even even the actual there's there.
Speaker 1: There are ports, you could say, or people have essentially
Speaker 1: created their own version of Eliza just using the original
Speaker 1: Liza as a guide. You can find tons of them
Speaker 1: on the web, and you can attempt to have a
Speaker 1: conversation with them. It's not terribly compelling, but it's it's
Speaker 1: kind of fun. Usually within maybe four or five exchanges,
Speaker 1: you've already run into something where you're like, well, this
Speaker 1: can't be a person, or if it's a person, it's
Speaker 1: the weirdest person I've ever conversed with. But but again,
Speaker 1: it wasn't really an earnest attempt to to create something
Speaker 1: that would pass the quote unquote tearing tests, which I
Speaker 1: don't think was being referred to as such at that
Speaker 1: point yet. Yeah, it was kind of people knew about
Speaker 1: tourings prediction, but it wasn't so much called a turing test.
Speaker 1: Another chat bought that premiered in the early nineties seventies
Speaker 1: went even further and actually was an attempt to try
Speaker 1: and pass the Turing test in a very specific approach,
Speaker 1: kind of like you know, Eugene Goostman, is a very
Speaker 1: specific approach to narrow down that those parameters. In this case,
Speaker 1: this one was called Perry p A R R Y
Speaker 1: A Right, and Kenneth Colby created it too to emulate
Speaker 1: a patient who has paranoid schizophrenia. Yeah, someone who has
Speaker 1: you know, sort of a persecution complex. Uh. They imagined
Speaker 1: that's that there are people or other entities that are
Speaker 1: out to get them. And in this uh specific case,
Speaker 1: he he kinda he kind of really embraced this approach.
Speaker 1: It reminds me of people who create a like a
Speaker 1: Dungeons and Dragon's character, but then give their character an
Speaker 1: entire backstory. Yeah. Yeah, yeah, this this Perry persona was
Speaker 1: an entire persona. It was a twenty eight year old
Speaker 1: man with a job as a post office clerk who
Speaker 1: was single, had no brothers and sisters, and rarely saw
Speaker 1: his parents. He had specific hobbies. He liked to of
Speaker 1: the movies and horse racing. Yeah, he liked to bet
Speaker 1: on the horses and placed a bet with bookies in
Speaker 1: the past. Right, and that he realized later bookies have
Speaker 1: an association with the criminal underworld, and that therefore the
Speaker 1: mafia knew about him and we're out to get him. Now. Now,
Speaker 1: all of that might sound very ridiculous to you if
Speaker 1: you never had any kind of interaction with someone who
Speaker 1: suffers from paranoid schizophrenia, that can seem like, well, that
Speaker 1: seems cartoonish. But no, this, this kind of thinking is
Speaker 1: not uncommon, you know, whether it's whether it's a criminal
Speaker 1: organization or the government or some other even unnamed entity. Oh. Absolutely, Uh,
Speaker 1: it's it's very much realistic in terms of that kind
Speaker 1: of diagnosis. And and if I was speaking about it
Speaker 1: with humor in my voice a moment ago, it's it's
Speaker 1: only because I am absolutely tickled that the programmers of
Speaker 1: this of this program built it, built it in. Yeah,
Speaker 1: it's it's actually pretty entertaining that they went so far
Speaker 1: as to make this whole uh, this whole backstory to explain,
Speaker 1: because that's what gives it the believability, right, and then
Speaker 1: they ended up testing it in conversation with a human.
Speaker 1: Perry would gradually start to introduce his thoughts quote unquote
Speaker 1: thoughts about being persecuted, and would respond sensitively to anything
Speaker 1: said about his appearance, family, or religious beliefs. I've actually
Speaker 1: seen lots of transcripts of conversations with Perry, and sure enough,
Speaker 1: it's one of those things where you know, you might
Speaker 1: have a few exchanges and then Perry ends up saying
Speaker 1: something that seems really odd, but not so odd as
Speaker 1: to seem artificial. It just seems like it's a non sequitur,
Speaker 1: you know, something like, well, that's what they want you
Speaker 1: to think. And if you were to say, who are
Speaker 1: they the mafia? Who did you think I was talking about?
Speaker 1: Like the mafia after you? Of course they're after me.
Speaker 1: They know who I am. That kind of stuff, and
Speaker 1: it's disturbing, like it's you know, when you know what
Speaker 1: it is, it's kind of amusing, but like if you're
Speaker 1: in the middle of a conversation, oh, this poor person. Yeah.
Speaker 1: And so in order to test it, uh Kolby did
Speaker 1: a couple of different things. He did one test where
Speaker 1: eight psychiatrists interviewed both Perry and a human patient via teletypewriter.
Speaker 1: So in both cases, the the the psychiatrists could not
Speaker 1: see who they were interviewing. This is going back to
Speaker 1: the kind of the original touring test. Idea, or at
Speaker 1: least Touring's proposed experiment. And in this case, only two
Speaker 1: of the eight were able to identify that one of
Speaker 1: the interviewees was human and the other was a machine.
Speaker 1: In a second test, Colby presented a group of a
Speaker 1: hundreds psychiatrists, most a psychics, transcripts of interviews between Perry. Yeah,
Speaker 1: he had, he had pairs of exactly food, I foresee. No.
Speaker 1: But they gave these psychiatrists transcripts of interviews between an
Speaker 1: interviewer and Perry, and an interviewer and a human patient,
Speaker 1: and forty out of the one responded. I don't know
Speaker 1: if the other sixty just never got it or if
Speaker 1: they didn't take The response rates are variable, right, so
Speaker 1: I'll the forty who responded, nineteen of them guessed incorrectly.
Speaker 1: So that's almost a fifty percent, you know, getting you know,
Speaker 1: right up there with a pretty impressive amount. Now, again,
Speaker 1: we have to look at the fact that Perry is
Speaker 1: operating under a very restricted set of rules. We're talking
Speaker 1: about paranoid schizophrenic, someone who we would assume would occasionally
Speaker 1: have a non normative answers exactly to conversational pieces. And
Speaker 1: and again it's it's a limited time that you're having
Speaker 1: with this person or this entity in this case, this
Speaker 1: this program, and uh, because the psychiatrists had a specific
Speaker 1: expectation of the type of interactions they were going to
Speaker 1: see that could have affected their their answer right. So ideally,
Speaker 1: in the perfect situation, you would have this in review
Speaker 1: happening where you have no expectation as to what the
Speaker 1: answers should be. In other words, you don't know ahead
Speaker 1: of time that the interviewee is having any kind of
Speaker 1: other any kind of restrictions upon that person, so that
Speaker 1: you would be interviewing anyone like any average person. But
Speaker 1: that's obviously not what we're talking about here, nor was
Speaker 1: it the one for Eugene Goosband. So I've also seen
Speaker 1: by the way transcripts of people who set up Eliza
Speaker 1: and Perry to talk to each other. So you have
Speaker 1: Eliza acting as the Rogerian psychiatrist and Perry the paranoid schizophrenic,
Speaker 1: having bizarre conversations, and they usually don't last very long
Speaker 1: because Perry gets upset, And obviously by gets upset, I
Speaker 1: just mean that Perry ends up essentially shutting down the
Speaker 1: conversation because Eliza just wants to ask questions, and Perry
Speaker 1: gets suspicious of people who are asking questions, and by
Speaker 1: again get suspicious, I'm saying following specific rules that make
Speaker 1: it feel like this computer programs getting suspicious, but they
Speaker 1: are entertaining. If you ever do a search online, just
Speaker 1: look for Eliza Perry transcripts. There. There are a few
Speaker 1: of them, and they're all pretty entertaining. But so since
Speaker 1: since that time, I mean obviously, lots and lots of
Speaker 1: chatbots have been created for multiple reasons. Oh yeah, well,
Speaker 1: you know, some of them are trying to test the
Speaker 1: Turing test, and others are trying to fool you into
Speaker 1: giving out your credit card information or planing on a
Speaker 1: link that has malicious uh links to malicious software. Yeah.
Speaker 1: Anyone who's been on any kind of chat program, specifically
Speaker 1: like AIM has probably encountered this at least once or twice,
Speaker 1: where they're they're getting an unsolicited message from someone or
Speaker 1: something or something. Yeah, and if you type a couple
Speaker 1: of times, you realize, oh, this is not actually this
Speaker 1: is an attempt for to either get information from me
Speaker 1: or have me click on a link. Yeah, that's a
Speaker 1: that's the thing. But there are some example of I
Speaker 1: guess saying legitimate is weird. But there are some examples
Speaker 1: of other more and more scholarly attempts. Yeah, like a
Speaker 1: PC therapist. Uh. That one was by Joseph Weintraub, and
Speaker 1: it fooled of its judges into thinking that it was human.
Speaker 1: Of ten judges, of course, so if I'm doing my
Speaker 1: math correctly, that's five that I believe you are. Unless
Speaker 1: we're talking about quantum judges. Um, these judges were both
Speaker 1: right and wrong at the same time. And um, and
Speaker 1: it was it was a whimsical program. I guess you
Speaker 1: could say, yeah, it had come some I think it's
Speaker 1: fair to say its answers could be pretty smart ass. Also,
Speaker 1: I read some of these transcripts and it actually surprised
Speaker 1: me that enough judges thought that it was a person.
Speaker 1: Maybe they thought it was a person who was purposefully
Speaker 1: attempting to fool them into thinking it was a computer. Uh,
Speaker 1: you know, because it would say things like I compute,
Speaker 1: therefore I am that kind of stuff where it was
Speaker 1: specifically Yeah, so you're looking at this stuff and you're thinking,
Speaker 1: all right, well, maybe this is kind of going back
Speaker 1: to that original touring test party game where the the
Speaker 1: effort is for a person like the person who's being
Speaker 1: interviewed is trying to throw everybody off, And that's perfectly
Speaker 1: within the rules unless you stay upfront. No, just be
Speaker 1: honest in your in your answers. There's nothing in here,
Speaker 1: by the way that says that the interviewee has to
Speaker 1: tell the truth necessarily unless you just state that as
Speaker 1: a parameter at the beginning. So in other words, you
Speaker 1: could you could be like, I'm totally the computer and
Speaker 1: you're the human being interviewed. Uh. I don't know if
Speaker 1: that's a fair way of saying that the device won
Speaker 1: or lost, but it is a possibility. Uh. Then we
Speaker 1: have a two thousand eleven. Now, this one is a
Speaker 1: really pretty impressive one. And this was clever Butt, which
Speaker 1: was made by a fellow named Rollo Carpenter, and it
Speaker 1: fooled fifty nine point three percent of a live audience
Speaker 1: and an event in India with more than a thousand people. Yeah,
Speaker 1: the way this worked was that the audience watched as
Speaker 1: interviewers interacted via text with either clever Bot or a
Speaker 1: human in the course of a four minute interview. So
Speaker 1: it's a little shorter than what Turing had said, but
Speaker 1: not by not by a whole lot. In fourmants is
Speaker 1: still a good amount of time. Sure that that is
Speaker 1: twent less, that's true. That's true, so keep that in mind.
Speaker 1: But but at any rate, it was you know, pretty
Speaker 1: interesting experience. And also from why I read they misidentified,
Speaker 1: they thought that the human was a computer sixty of
Speaker 1: the time. Um, because they didn't necessarily just say that
Speaker 1: it was a computer or it was clever. But so
Speaker 1: now we see that there are a few examples of
Speaker 1: chat bots quote unquote passing the Turing test. So what
Speaker 1: does that mean? Does it mean that the machines are
Speaker 1: actually thinking? Um? No, I mean, and it's not to
Speaker 1: say that that computers don't have a certain amount of
Speaker 1: machine intelligence, but there's absolutely a distinction between that and
Speaker 1: what we consider to be human intelligence. That's true. The
Speaker 1: programmers themselves have said that this doesn't mean a machine
Speaker 1: is able to think. Uh, they're just able to interpret
Speaker 1: commands then to follow a set of rules to make
Speaker 1: a response, which is still pretty cool. And and it
Speaker 1: doesn't It certainly doesn't mean that the Turing test is
Speaker 1: worthless as an exercise. No, it is. In fact, it's
Speaker 1: improving our ability to create programs that that can understand
Speaker 1: or at least respond to natural language. Natural language recognition
Speaker 1: is one of those big things where if you're really
Speaker 1: able to crack it, then you can have some amazing
Speaker 1: opportunities open up. And we've seen this recently with stuff
Speaker 1: like Siri. Oh absolutely being able to speak to your
Speaker 1: computer rather than having to I mean, even if you
Speaker 1: could speak to your computer through a keyboard and have
Speaker 1: it understand what you're what you're saying, I mean, like
Speaker 1: it's it's the reason why Google spend so much time
Speaker 1: and money, and it's search algorithms of of trying to
Speaker 1: figure out what you really mean when you search for
Speaker 1: a certain phrase, because traditionally, you know, before we really
Speaker 1: got into the natural language recognition era, it meant that
Speaker 1: in order to work with a computer, you had to
Speaker 1: work with a computer on the computer's terms. You had
Speaker 1: to learn the commands, you had to learn the way
Speaker 1: to navigate a computer system in order for it to
Speaker 1: do what you wanted it to do. Once you get
Speaker 1: to a point where natural language recognition software is robust
Speaker 1: enough the computer is working on your terms, you you
Speaker 1: can put in however you're thinking, like whatever, whatever mental
Speaker 1: exercise you've gone through to ask this computer to do something.
Speaker 1: Whatever you do to kind of express a thought, uh
Speaker 1: as a command to this computer. The computer can then
Speaker 1: interpret then respond And it's not just for serving you
Speaker 1: back whatever information you happen to be looking for. It's
Speaker 1: I mean, I mean, we're talking about being able to
Speaker 1: just look at a computer and say, you know, I
Speaker 1: really want a graph that looks blue and has these
Speaker 1: percentages in it and it is about this thing, and
Speaker 1: it just does it. Yeah, like I want to see
Speaker 1: what the population distribution of Atlanta is in a bar
Speaker 1: chart or something, and then it could bring that, goes out,
Speaker 1: finds that information, put it into a bar chart, and yeah,
Speaker 1: that's pretty phenomenal stuff. We see other examples of machine
Speaker 1: intelligence everywhere, things like pattern recognition, probabilistic predictions, for example, Pandora.
Speaker 1: You know the Music Genome project. It's yeah, that's that's
Speaker 1: pattern recognition. It's looking for elements of songs that you
Speaker 1: say you like, and then looking for other stuff that's
Speaker 1: not in the specific category you mentioned or the specific
Speaker 1: examples you mentioned, and there's something else you probably will
Speaker 1: like because you like these other things that also have
Speaker 1: this stuff in it. Sure, Uh, you know sometimes that's
Speaker 1: less functional than other times. It makes me think of
Speaker 1: Patton Oswald has a great routine about Evo that does
Speaker 1: the same sort of thing where he says, you know,
Speaker 1: TiVo is great. I mean, I like, I love Westerns,
Speaker 1: and I'll have it set to TiVo a Western for me,
Speaker 1: and I come back, and then there'll be all these
Speaker 1: other Westerns that will be suggested. I didn't even know about.
Speaker 1: Thank you, TiVo. But then sometimes TiVo gets it wrong
Speaker 1: and I come back and everything has got horses in it,
Speaker 1: because Westerns have horses in it. So I've got my
Speaker 1: little pony and cartoons with horses and unicorns and things,
Speaker 1: and I have to say, no, TiVo, that's a bad TiVo.
Speaker 1: But TiVo says, but you said you liked horses. Same
Speaker 1: sort of thing. Like when you get more sophisticated than
Speaker 1: the the computer program starts to anticipate things and makes
Speaker 1: these probabilistic models. These these models where there there are
Speaker 1: certain percentages associated with various responses, and it goes with
Speaker 1: whichever one seems to be the most prevalent, assuming it
Speaker 1: meets a threshold. If this sounds familiar to you, it's
Speaker 1: because that's how ibms Watson worked, which is a really
Speaker 1: good example of natural language recognition absolutely because not only
Speaker 1: was it able to recognize natural language, it had to
Speaker 1: interpret things like word play Jeopardy. This is the machine
Speaker 1: that went up on Jeopardy and beat the returning champions
Speaker 1: or former champions. Uh And you know, if you've ever
Speaker 1: played Jeopardy or watch Jeopardy, you know that there are
Speaker 1: categories that depend on things like puns or hominem's or
Speaker 1: other forms of word play. So it has to parse
Speaker 1: all of that, and that's even more complicated than just
Speaker 1: taking a simple sentence and figuring out, all, right, what
Speaker 1: are the potential responses to whatever this this this phrase
Speaker 1: is so great, great example. You know, it would end
Speaker 1: up coming up with a potential answer. It would assign
Speaker 1: a percentage of how quote unquote sure it was that
Speaker 1: that was the right answer, and if the percentage was
Speaker 1: higher than its threshold, which I think was something like that,
Speaker 1: it would buzz in and give that as a guess.
Speaker 1: Sometimes it was wrong, but it was right a lot
Speaker 1: of the time, so that's kind of cool. Uh So,
Speaker 1: getting back to Eugene Gene, the main machine as I
Speaker 1: called him in in my notes, at one point uh,
Speaker 1: and of course I'm anthropomorphizing when I say him, it's
Speaker 1: a it's a it's an it. Well, it has a
Speaker 1: dude name. Yeah, it has a dude name and a
Speaker 1: dude persona. But it's ultimately, isn't it. Would you say
Speaker 1: that perhaps some of the reporting around this was was
Speaker 1: maybe a little misleading or at least hype is well,
Speaker 1: you know, okay, the entire Eugene Gooseman chat bought sounds
Speaker 1: really cool. I haven't met it personally, No, I haven't either,
Speaker 1: although you can. There is an Internet version, and I'm
Speaker 1: not sure that it's the same version that's being used
Speaker 1: in competition, because I've seen some transcripts from the Internet
Speaker 1: version and they don't seem good at all. They seem bad, right,
Speaker 1: I guess you'd have to talk to some actual thirteen
Speaker 1: year old boys from actually yeah, that well, this is
Speaker 1: part of it. You know. There's certainly been some some
Speaker 1: questions among uh natural language AI enthusiasts online about whether
Speaker 1: we're really just lowering our expectations for human communication. Which, yeah,
Speaker 1: that's that's a totally different way of looking at and
Speaker 1: a depressing one to say that, Oh, well, if you
Speaker 1: come from this place and if you are of this age,
Speaker 1: then I expect you to only be able to communicate
Speaker 1: at this level, right, which is depressing. It certainly is well,
Speaker 1: but you know, but it's a valid point. I think.
Speaker 1: I think it's a good thing to be thinking about
Speaker 1: in this kind of situation. UM. You know. Beyond that though,
Speaker 1: and I certainly don't want to downplay the apparent achievements
Speaker 1: of its programmers, because I haven't programmed any capable chatbots
Speaker 1: today ever since I did. But there are a few
Speaker 1: things that are just a little bit shady about the
Speaker 1: news UM. First off, the original press release, which came
Speaker 1: out of the University of Reading, I believe so UM
Speaker 1: stated that a quote supercomputer had achieved this feat, and
Speaker 1: perhaps charitably, it was a mistake or misunderstanding on the
Speaker 1: part of the writer of the press release, But some
Speaker 1: skeptics have suggested that it was in fact a purposeful
Speaker 1: publicity play that in fact worked, because a whole lot
Speaker 1: of news headlines around the interwebs repeated the error very excitedly. Yes,
Speaker 1: because it was not a supercomputer. It was a computer
Speaker 1: running a piece of software program. Yes, the program that
Speaker 1: did the work. I mean, the computer just provided the horsepower. Right,
Speaker 1: it's the software that did all the action work, and
Speaker 1: it was not it wasn't on a supercomputer. Little little
Speaker 1: known fact. Supercomputers have better things to do than run
Speaker 1: chatbot software generally. Yeah. Yeah, we're talking about things like
Speaker 1: figuring out global weather changing change pad and some things
Speaker 1: like that, you know, or or the way that money works.
Speaker 1: Chat Bots low on their priority list. It's like number
Speaker 1: seven at least. Um And and also that this press
Speaker 1: release in question was largely a quotation from Dr Kevin Warwick.
Speaker 1: Kevin Warwick, of course being the fellow who organized this
Speaker 1: entire competition, who who's an engineer and a futurist um
Speaker 1: and also the instigator and or enjoyer of a certain
Speaker 1: amount of hype and debate about future technologies. Yeah, he is,
Speaker 1: obviously you can tell. This is the guy who elected
Speaker 1: to have surgery performed on him so he could have
Speaker 1: h that r F I D chip and call himself
Speaker 1: a cybors. This is someone who not only embraces this
Speaker 1: these ideas of futurism, but is actively trying to promote
Speaker 1: them and get to them that we're not even saying
Speaker 1: that that's a bad thing. What we are saying is
Speaker 1: that may give him somewhat of a bias when it
Speaker 1: comes to proclaiming a computer software piece of computer software
Speaker 1: being an amazing achievement that beat the Turing test, right sure.
Speaker 1: I mean he admits basically to to being a provocateur.
Speaker 1: He he says that that's really his job, you know,
Speaker 1: is to get people excited about tech and engineering and
Speaker 1: and the future. And we get that like that that
Speaker 1: we agree that a job too. We think it's red
Speaker 1: We think that perhaps I don't want to put words
Speaker 1: into your mouth, Lauren, I think perhaps that there's a
Speaker 1: different way of going about it where you can still
Speaker 1: be excited, but you can be a little more grounded
Speaker 1: in the way you present things, because I I also
Speaker 1: think the achievement of creating a chatbot that could be
Speaker 1: uh convincing is a fantastic achievement. I mean, it's something that,
Speaker 1: even under any number of qualifications, incredibly challenging to do,
Speaker 1: no matter how you frame it. Um. I do think, however,
Speaker 1: that if you seem to over inflate the achieve it,
Speaker 1: you run the danger of making people feel jaded about
Speaker 1: it later, which I think Computer Wolf exactly, yeah, exactly.
Speaker 1: So it's one of those things where you know, you
Speaker 1: have to take the context into account, right and don't
Speaker 1: don't downplay the achievement. But don't sit there and say like,
Speaker 1: ah ha, now we have intelligent computers everywhere. That's not
Speaker 1: that's not true either. I saw there's a great Wired
Speaker 1: article that specifically went into uh, kind of debunking the
Speaker 1: whole beating the Turing test thing and again kind of
Speaker 1: saying the same thing we're saying, like take the context
Speaker 1: into account, and and part of that article they ended
Speaker 1: up asking a cognitive scientist named Gary Marcus of n
Speaker 1: y U about this, and Marcus proposed a new version
Speaker 1: of the Turing test because he says, the old version
Speaker 1: is not really a measurement of machine intelligence. Uh, it
Speaker 1: does kind of illustrate ways of creating natural language recognition
Speaker 1: and clever way is to fool the human side, uh huh.
Speaker 1: And that it was very valid historically at the time
Speaker 1: because you know, text textual communication was new and exciting
Speaker 1: and it was you know, it pushed the field forward,
Speaker 1: it really did. But now we've gotten to a point
Speaker 1: where fooling the person on the other side of a
Speaker 1: keyboard is not necessarily the goal that we should be
Speaker 1: looking at he proposes. The next version of the Turing
Speaker 1: test should be that a computer software, uh, like any
Speaker 1: kind of program that wants to beat it, what it
Speaker 1: has to do is first quote unquote watch a movie,
Speaker 1: television show, YouTube video, something, some kind of video media,
Speaker 1: and then be able to respond to questions about it.
Speaker 1: So sort of like here, let me show you this
Speaker 1: ten minute video on car safety, and then asking questions
Speaker 1: about specifically about the video, what happened after they fastened
Speaker 1: the seat belt, that kind of thing, and if the
Speaker 1: computer program is able to answer it, then that would
Speaker 1: be a much more convincing touring test than just kind
Speaker 1: of spewing out a script, which is, you know, it's
Speaker 1: adding another layer of difficulty on top of an already
Speaker 1: difficult task. But that's that's the whole only way you
Speaker 1: can go forward. Otherwise we're just going to see increasingly
Speaker 1: sophisticated chat box. Yeah, and and that is actually a
Speaker 1: very difficult and interesting problem. What wasn't it just recently
Speaker 1: that there was a computer that we we taught I
Speaker 1: mean not us personally, but that humanity, some researchers, and
Speaker 1: we're taught taught to identify cats pictures of cats that
Speaker 1: was the AI program that essentially went through thousands and
Speaker 1: thousands of uh I think it was images and videos
Speaker 1: and then became able to identify cats, essentially defined what
Speaker 1: a cat was because no one taught it right. It
Speaker 1: learned what cats are based upon their appearance, on their
Speaker 1: appearance and can look at pictures of cats and say
Speaker 1: that is totally a cat, essentially saying that thing that
Speaker 1: is in that video is the same as this other
Speaker 1: thing that's in this picture. That's the same as this
Speaker 1: other thing that's in this totally different video, which sounds
Speaker 1: trivial and hilarious, and it kind of is hilarious. Also
Speaker 1: easy because I mean, come on, everything on the internet
Speaker 1: has cats in it. Yeah, that is That is kind
Speaker 1: of a gimme, isn't it? But still no, it is.
Speaker 1: It is cool. I mean because just just that level
Speaker 1: of image recognition, I mean being able to take an
Speaker 1: object and look at it from a different angle than
Speaker 1: you were taught or that's a different color, different different size. Yeah,
Speaker 1: all these things. All of these things are easy for us,
Speaker 1: hard for computers. So seeing something make that breakthrough is
Speaker 1: really exciting. Anyway, we thought we would take that story
Speaker 1: and kind of break it down for you guys, explain
Speaker 1: how it's still cool but maybe not as cool as
Speaker 1: the way some of the headlines are saying. Right, and
Speaker 1: also say, hey internet journalists, um, step up your game. Yeah,
Speaker 1: I understand that you want people to read your stuff.
Speaker 1: Oh yeah, and you're under deadline pressure and that's terrible.
Speaker 1: But that's that's hard. Let's reality. We don't just don't
Speaker 1: just spit up us releases the way that you found them. Yeah,
Speaker 1: and uh, and I would be criminal to to neglect
Speaker 1: to mention, knowl Our producer reminded me that obviously this
Speaker 1: is a very important field of study because we want
Speaker 1: to be able to tell the difference between computers and
Speaker 1: humans when the future of Blade Runner becomes our reality
Speaker 1: and you're chasing down a replicant and you have to
Speaker 1: determine if it's actually a replicate or human. Being very
Speaker 1: good point, computer programmers, just make sure you don't explain
Speaker 1: sort of you know, what you do if you see
Speaker 1: in turtle laying on back and you've decided not to
Speaker 1: turn it over, because that's like that's like our gimme, Yeah, yeah,
Speaker 1: we got yeah, we need that. So everything else fair game.
Speaker 1: All right, So that wraps up this discussion. Guys. First
Speaker 1: of all, thank you so much Nick for for asking
Speaker 1: us about that. And if anyone else wants to ask
Speaker 1: us to talk about anything in particular right now, I
Speaker 1: would highly suggest you use Twitter, Facebook, or Tumbler. We
Speaker 1: have to handle tech stuff hs W. We will soon
Speaker 1: have an email address, but coming any day now. Yeah,
Speaker 1: we need we're making this transition from one kind of
Speaker 1: email server to a different one. We're getting new email addresses,
Speaker 1: and as of right now, as we're recording this podcast,
Speaker 1: I think tech Stuff does not yet have one. Our
Speaker 1: future address will be tech stuff at how stuff works
Speaker 1: dot com. So if you're listening to this in try it. Yeah,
Speaker 1: it should be fine. But if you're listening to this
Speaker 1: the day it comes out and you wanted to send
Speaker 1: us a message and it's bouncing back, try Twitter, Facebook
Speaker 1: or Tumbler and we will get your message there. And yes,
Speaker 1: we are working on this. We'll have it up as
Speaker 1: soon as we possibly can, and we'll talk to you
Speaker 1: again really soon for more on this and bathens of
Speaker 1: other topics. Because it how stuff works dot com