Speaker 1: Welcome to Text, a production from my Heart Radio. Hey there,
Speaker 1: and welcome to tech stuff. I'm your host, Jonathan Strickland.
Speaker 1: I'm an executive producer with I Heart Radio and I
Speaker 1: love all things tech, and today I want to tackle
Speaker 1: a really interesting, complicated, and potentially scary topic, and that
Speaker 1: is predictive text generation. And I know that sounds weird
Speaker 1: to say potentially scary, but you know, stick with me.
Speaker 1: I'm sure many of you have seen social media posts
Speaker 1: that say things like type I am the on your
Speaker 1: phone and then generate a result using the middle option
Speaker 1: of predictive text. So you know, just for example, I
Speaker 1: did that. If I did that on my phone, then
Speaker 1: I get I am the only one who can help
Speaker 1: me with this. Oh two, real predictive text. I mean,
Speaker 1: I'm the only one who researches and writes these episodes.
Speaker 1: That's it's way too real. But the whole meme of
Speaker 1: using predictive text to generate seemingly meaningful or you know,
Speaker 1: sometimes wildly absurd phrases is just part of what I
Speaker 1: want to talk about today. Now. The reason this topic
Speaker 1: jumped at me is because of a recent news article
Speaker 1: that I read over on the Verge The article that
Speaker 1: was written by Kim Lyons has the title a college
Speaker 1: student used GPT three to write fake blog posts and
Speaker 1: ended up at the top of Hacker News. Now, as
Speaker 1: the headline indicates, a computer science student used a predictive
Speaker 1: text engine called GPT three, a beta build of it
Speaker 1: in fact, that stands for Generative pre Trained Transformer, and
Speaker 1: then generated a blog that was featured on a site
Speaker 1: called hacker News as if it were a piece written
Speaker 1: by a flesh and blood human being. What's more, a
Speaker 1: threat on Reddit showed that only a few people were
Speaker 1: picking up on the feeling that something hinky was going on,
Speaker 1: and that perhaps the blog post had not been written
Speaker 1: but generated. And Lions goes on to point out that
Speaker 1: the fact that there's a lot of, you know, not
Speaker 1: very good writing on the Internet makes it a little
Speaker 1: harder to sus out a decent generated post as opposed
Speaker 1: to a written one. It's not so much that the
Speaker 1: AI has become super awesome at writing, but rather that
Speaker 1: we've kind of lowered the bar more than a little.
Speaker 1: This kind of plays into the whole concept of a
Speaker 1: touring test. So, just to go off on a tangent here,
Speaker 1: this isn't in my notes, I'm just going to speak
Speaker 1: off the cuff. The Touring test is named after Alan Touring,
Speaker 1: famous computer scientist, and the idea. Nowadays, it's kind of
Speaker 1: evolved into this idea of you have a series of
Speaker 1: interviews that a person does over a computer, and some
Speaker 1: of the interviewees are people and some of them are
Speaker 1: chat bots essentially, and the goal of this whole exercise
Speaker 1: is to see if the person who's doing the interview
Speaker 1: can consistently tell if the other entity on the other
Speaker 1: side of the interview is a person or if it's
Speaker 1: a chat bought. And if you pass with a certain percentage,
Speaker 1: you would say that the chat bought has passed the
Speaker 1: Touring test, that people are unable to tell the difference
Speaker 1: between the chat bought and a real human being, and
Speaker 1: that this is kind of one of the markers for
Speaker 1: artificial intelligence. We're gonna be dipping into that sort of
Speaker 1: thing with this discussion as well. So today I'm really
Speaker 1: wanted to dive into the whole concept of predictive text
Speaker 1: and how it's done and how it could absolutely destroy
Speaker 1: platforms like Facebook in the future. That's all I'm going
Speaker 1: to end this episode, So stick around, but we have
Speaker 1: to build on this gradually, So let's start at the
Speaker 1: very beginning, which, according to this woman who's singing outside
Speaker 1: my window, is a very good place to start. And
Speaker 1: we are going to start with a particularly tricky concept
Speaker 1: for a former English Lit Major to try and explain,
Speaker 1: and this is called a Markov model. It's named after
Speaker 1: a mathematician named Andre Andreyevitch Markov, and he was born
Speaker 1: in Russia in eighteen fifty six, and he did a
Speaker 1: lot of work on an area of mathematics called stochastic processes.
Speaker 1: But that just raises another question, right, what does stochastic mean? Well,
Speaker 1: a stochastic variable is one that is randomly determined. A
Speaker 1: stocastic system has a random probability pattern that you can study,
Speaker 1: but you can't dickt it precisely. There's always uncertainty. So
Speaker 1: you can assign probabilities as to how the pattern will form,
Speaker 1: but those are just indications of how likely a particular
Speaker 1: pattern will form, not a guarantee. So let's take a
Speaker 1: very simple example, and let's pick something really random. Let's
Speaker 1: talk about my two year old niece. So let's say
Speaker 1: my niece is standing in the middle of a room
Speaker 1: and I walk in. Now, based on my past interactions
Speaker 1: with this random creature, I know my niece is likely
Speaker 1: to do one of three things. She is going to
Speaker 1: run at me and grab my hand, and then boss
Speaker 1: me around and put me someplace and tell me I
Speaker 1: have to stay there. She's going to run away from
Speaker 1: me and then hide and then demand very loudly that
Speaker 1: I come find her. She is not, i should add,
Speaker 1: quite grasped the concept of hiding. Or she is going
Speaker 1: to ignore me and say and or dance. Those are
Speaker 1: the things that she typically does. There are other things
Speaker 1: she might do as well, but they happen much less frequently.
Speaker 1: So let's say I want to sketch out this scenario
Speaker 1: on paper. I might start with the scenario is my
Speaker 1: nieces in a room and I come into the room.
Speaker 1: Then I would draw a little bubbles on my paper
Speaker 1: to represent the potential actions or states as we would
Speaker 1: call them, in a Markov chain that could follow this
Speaker 1: input of me walking into the room. Now, based on
Speaker 1: the number of times I've seen her respond before, I
Speaker 1: could wait each of those states with a certain probability. If,
Speaker 1: for example, she runs at me and grabs my hand
Speaker 1: then bosses me around more than half the time I
Speaker 1: can wait that outcome, as you know, And does that
Speaker 1: mean the next time I walk into a room that
Speaker 1: she's going to do that? No, each incident is random.
Speaker 1: I'm just illustrating how likely a particular outcome is going
Speaker 1: to be. I would then assign probabilities for the other
Speaker 1: two outcomes I outlined, and and maybe just ignore all
Speaker 1: the outliers and say that one of them is you know, likely,
Speaker 1: which means the third one is only five percent likely
Speaker 1: to happen because it has to add up to now.
Speaker 1: The example I just gave is ridiculously simple, despite the
Speaker 1: fact that my niece is already incredibly complicated, And it
Speaker 1: just gives us the odds of one starting state that
Speaker 1: I'm me walking into a room that then transitions into
Speaker 1: one of three outcome states. Markov models can have lots
Speaker 1: of variables, with some variables dependent upon the value of
Speaker 1: other variables. So you might see a chain as something
Speaker 1: like if outcome A happens and there's a sixty chance
Speaker 1: that it will, then there's a thirty percent chance that
Speaker 1: a subsequent outcome A three will happen, And it can
Speaker 1: become a really complex branching path of possibilities, but we
Speaker 1: can stick with simple. Let's take the coin flip, the
Speaker 1: classic example of a random variable. We know that the
Speaker 1: odds of a fair coin landing heads up are and
Speaker 1: landing tails up. Our fifty percent. Flipping a coin many
Speaker 1: thousands of times should show that collectively you're gravitating towards
Speaker 1: those probabilities, that about half of your coin flips will
Speaker 1: be heads and the other half will be tails. But
Speaker 1: that does not mean you won't get on streaks where
Speaker 1: you flip heads over and over. Allah, Rosencrantz and Guildenstern
Speaker 1: are dead. And if you don't know that reference, I
Speaker 1: highly recommend that you read that play or you watch
Speaker 1: the excellent film version that has Tim Roth and Gary
Speaker 1: Oldman in it, because it is fantastic and it kind
Speaker 1: of dives into a fun discussion of probabilities and what
Speaker 1: does that actually mean Anyway, The odds of flipping a
Speaker 1: coin heads are for a single coin flip, but what
Speaker 1: about a second coin flip. Well, if we look at
Speaker 1: just that flip in isolation, that second coin flip, it's
Speaker 1: still a fifty pc chance that's going to land on heads.
Speaker 1: But if we frame it a different way, if we
Speaker 1: ask the question, what what are the odds of flipping
Speaker 1: heads twice in a row? This is a different question
Speaker 1: because you're not thinking about individual flips. You're saying, what
Speaker 1: are the odds of this happening twice sequentially? Well, now
Speaker 1: we have to take the odds of it happening once,
Speaker 1: which is, and then we have to multiply it against itself.
Speaker 1: It's a fifty chance again that it would happen twice.
Speaker 1: So oft is let me do the math. It is
Speaker 1: or one four. So if you were to do a
Speaker 1: pair of coin flips, and you were to repeat this
Speaker 1: experiment over and over and over again over the long run,
Speaker 1: you would find that of those sequences would end up
Speaker 1: with heads followed by heads. But what if we wanted
Speaker 1: to say, how what are the odds of flipping three
Speaker 1: heads in a row? Well, then we have to have
Speaker 1: it again. So instead of one out of every four trials,
Speaker 1: we would see one out of every eight, or twelve
Speaker 1: point five percent. And we can keep extending this out.
Speaker 1: We can figure out the odds of some ridiculously long
Speaker 1: stretch of flipping heads in a row. Now in Rosen, Cranston,
Speaker 1: Gillenstern are dead. We are told that it happens and
Speaker 1: astonishing ninety two times in a row, that streak has
Speaker 1: a probability of one in five octillion. That would be
Speaker 1: a five followed by twenty seven zeros. This does not
Speaker 1: mean that it would be impossible, but it is unfathomably unlikely.
Speaker 1: Clemson University has a useful lecture available online in the
Speaker 1: form of a presentation, and it's titled Introduction to Markov Models,
Speaker 1: and it uses weather forecasting as an example. And their
Speaker 1: example takes three initial states, sunny, rainy, and cloudy. Consequently,
Speaker 1: those are also the three potential output states, so each
Speaker 1: state can transition into three states, including transitioning into itself,
Speaker 1: so you could go sunny to cloudy, sunny too rainy,
Speaker 1: or sunny to sunny. That's a valid result as well.
Speaker 1: And in their example, the ideas that we have based
Speaker 1: on past observations figured out the probability for specific forecasts
Speaker 1: based on whatever the current weather happens to be. So,
Speaker 1: for example, we've figured out that rain tomorrow is likely
Speaker 1: if it's raining today, but it's only likely if it's
Speaker 1: just cloudy or sunny today. So if it's cloudy, if
Speaker 1: it's sunny, if it's raining today, that we'll see rain tomorrow.
Speaker 1: But our model would need to have probabilities assigned to
Speaker 1: each pair of starting and ending states. So I'm gonna
Speaker 1: follow through with that just for the purposes of this conversation.
Speaker 1: And we've covered the probabilities of tomorrow being rainy based
Speaker 1: on whatever today's weather is. But the example from Clemson
Speaker 1: also gives the other two outcomes states. So if we're
Speaker 1: looking at the probability of tomorrow being cloudy, we see
Speaker 1: that based on our past observations, that if today is sunny,
Speaker 1: it's a chance of cloudy tomorrow. If today is rainy,
Speaker 1: it's a thirty percent chance, and if today is cloudy,
Speaker 1: there's a fifty percent chance. And finally, if we want
Speaker 1: to know if it's going to be sunny tomorrow, again
Speaker 1: this is all just based on the example. We see
Speaker 1: that if today is sunny, there's an eight percent chance
Speaker 1: that tomorrow will be too. If today is rainy, it's
Speaker 1: just a five percent chance. If today is cloudy there's
Speaker 1: a fifteen percent chance. Now, the reason we need to
Speaker 1: know all of these probabilities will become clear in a second.
Speaker 1: And again these are just examples, they don't reflect real data.
Speaker 1: Markov got very clever and began to use math to
Speaker 1: describe probabilities for predictions that are further out than one state. So,
Speaker 1: for example, you might say, what is the probability that,
Speaker 1: if today is cloudy, that tomorrow will be sunny and
Speaker 1: that the following day will be rainy. This is kind
Speaker 1: of similar to us asking the question of what are
Speaker 1: the odds of flipping heads two or three times in
Speaker 1: a row, except we're looking at the probabilities of weather
Speaker 1: that are based on what our current conditions happen to be.
Speaker 1: So using the example probabilities that were used in that lecture,
Speaker 1: we would find that sunny days follow cloudy days just
Speaker 1: fifteen percent of the time, So there's a fifteen percent
Speaker 1: chance that tomorrow will be cloudy if today is sunny,
Speaker 1: and rainy days follow sunny days twenty per scent of
Speaker 1: the time. So if tomorrow is sunny, there's a twenty
Speaker 1: chance the day after tomorrow will be rainy. So then
Speaker 1: that means that if today's cloudy, we've got that fift
Speaker 1: chance tomorrow will be sunny, and if it is sunny,
Speaker 1: there's a chance that the day after tomorrow will be rainy.
Speaker 1: So we have to multiply those probabilities together. We have
Speaker 1: to multiply that by twenty or point one five times
Speaker 1: point two. That gives us point zero three, which we
Speaker 1: convert to a percentage. That means there's just a three
Speaker 1: percent chance that if today is cloudy, tomorrow will be sunny,
Speaker 1: and the day after tomorrow will be rainy. That's just
Speaker 1: a three percent chance of that happening. And the further
Speaker 1: out we try to predict a particular sequence of whether,
Speaker 1: the lower the probability will be, meaning you know it
Speaker 1: could happen. It's not like it's impossible, but it gets
Speaker 1: less likely the further out we go from our initial state.
Speaker 1: So a Markov model is a stochastic model that describes
Speaker 1: putten chill sequences. It is temporal in nature. That means
Speaker 1: we are really concerned with the state of things and
Speaker 1: how those states will change over time, and it gives
Speaker 1: us a way to explain how current states will depend
Speaker 1: upon previous states. It's not just about predicting the future,
Speaker 1: but also understanding the present. Why are things the way
Speaker 1: they are right now? And it gives us the chance
Speaker 1: to weigh the predictions of the future based upon past
Speaker 1: observational data. This is why we see weather forecasts that
Speaker 1: give us percentages for rainy days, Like a chance for
Speaker 1: rain tells us that it's probably a good idea to
Speaker 1: bring an umbrella if we're going outside, because based on
Speaker 1: past observations, there's a decent chance it's going to rain today. Now,
Speaker 1: let's get more complicated. What if we don't actually know
Speaker 1: the current state of the weather. Let's say that you
Speaker 1: are stuck inside and you can't see out a window,
Speaker 1: you have no windows in the room you're in, and
Speaker 1: someone else comes into your room and says, what's the
Speaker 1: weather like outside? Well, the only hint that we have
Speaker 1: in this experience is if the person that comes in
Speaker 1: is carrying an umbrella or not. We don't actually know
Speaker 1: the current state. We can only make an educated guess
Speaker 1: based on the presence or absence of an umbrella. The
Speaker 1: reality of the current state is hidden from us. This
Speaker 1: leads us to a type of sequential analysis that's used
Speaker 1: in computer science, the hidden Markov model. So with these models,
Speaker 1: we're trying to learn more about the initial states by
Speaker 1: analyzing the outcomes that we can observe. And another way
Speaker 1: of putting it is we're trying to answer the question
Speaker 1: Why are things how they are right now? Why did
Speaker 1: this happen? Let's look back and figure out the probability
Speaker 1: that a particular initial state led to what is going
Speaker 1: on right now now. The whole reason I spent time
Speaker 1: talking about Markov models and probability is that it ties
Speaker 1: heavily into predictive text. It's also used in tons of
Speaker 1: other computational processes and analysis, from natural language analysis to
Speaker 1: genome sequencing. It's really powerful stuff. If we think about language,
Speaker 1: we know that there are certain rules to things. You
Speaker 1: can't just string random letters in a sequence and expect
Speaker 1: that to make a word that other people can understand.
Speaker 1: We have developed languages that have their own vocabularies and
Speaker 1: syntax and grammars. We know that in English, for example,
Speaker 1: the letter Q is nearly always followed by the letter you.
Speaker 1: We know that it would be very odd to see
Speaker 1: the letter H follow right behind the letter J in English.
Speaker 1: And so we can start building out a dictionary and
Speaker 1: a matrix, and the dictionary would include lots of common words,
Speaker 1: and the matrix would include basic rules to help us
Speaker 1: identify when someone is making a typo or misspelling something.
Speaker 1: And with these tools we could build out a method
Speaker 1: for predicting a letter based on the letters that were
Speaker 1: already typed. So if I typed T and then H,
Speaker 1: my predictive text might helpfully offer out the letter E
Speaker 1: because I frequently type the word the If I ignore
Speaker 1: that and I hit the letter A, I might get
Speaker 1: the prompt of using van or thank or maybe even
Speaker 1: thanks or maybe something else. And we're starting down that
Speaker 1: journey toward generative text. When we come back, I'll explain
Speaker 1: more about this and some really cool experiments with using
Speaker 1: machine learning and what that all means. But first let's
Speaker 1: take a quick break. Okay, So we're building out a
Speaker 1: tool that quote unquote understands basic probabilities of words appearing
Speaker 1: in a given language in a given order, and it
Speaker 1: understands that, for example, a Q will be followed by
Speaker 1: you nearly of the time in English. We build into
Speaker 1: this model all sorts of probabilities, so that words that
Speaker 1: are more common are going to pop up as autocomplete
Speaker 1: options more frequently than uncommon words. But we can do
Speaker 1: better than this. We can pair this with a learning model.
Speaker 1: Learning models evolve over time, They adjust based on the
Speaker 1: input fed to them, and we're talking about lots and
Speaker 1: lots of input, they refine themselves, so, in other words,
Speaker 1: they learn. So with learning models are predictive text begins
Speaker 1: to adjust to the specific individual who uses the predictive
Speaker 1: text over time. Like a phone. So let's say you
Speaker 1: and I each have the same particular model of smartphone,
Speaker 1: and we're both running the same operating system version and everything,
Speaker 1: like our phones are are essentially identical, at least at
Speaker 1: casual glance. And we've both been using these phones for
Speaker 1: a few weeks. And in that time, you and I
Speaker 1: have each used our phones to send various messages to
Speaker 1: our friends, our family, our colleagues, you know, your arch nemesis,
Speaker 1: Ben Bolan, you know the usual. As we do that,
Speaker 1: our predictive text keyboards start to pick up on how
Speaker 1: we use words, and it can build up a frequency matrix,
Speaker 1: which isn't just looking at words that are common in general,
Speaker 1: but words that are common to us as individuals, and
Speaker 1: the way that we use words, and sometimes the way
Speaker 1: we generate words. Maybe you happen to use the word
Speaker 1: balder dash a lot, and so you start typing the
Speaker 1: word and the autocomplete for balder dash will jump up
Speaker 1: much faster than it would if I were typing it
Speaker 1: on my phone, because my phone has never heard me
Speaker 1: use that, so it doesn't automatically assume that's what I'm typing.
Speaker 1: Maybe I use the word folder roll a lot, and
Speaker 1: the same happens with my phone compared to yours. The
Speaker 1: models learned the words we use, not and not just
Speaker 1: the words that the words we create as well. So
Speaker 1: let's say that I was, for some reason a big
Speaker 1: fan of How I Met Your Mother, which I'm not.
Speaker 1: But let's say that I am a big fan of
Speaker 1: Neil Patrick Harris, which is true, and his character often
Speaker 1: says that is wait for it, legendary. Uh, And it
Speaker 1: might extend the word legendary. So to do that, I
Speaker 1: might throw in a whole bunch of extra ease at
Speaker 1: the beginning of legendary. Well, my phone might pick up
Speaker 1: that I tend to do this, and so it includes
Speaker 1: that as a legitimate word, even though any sort of
Speaker 1: spelling check would say this ain't a word, stop it,
Speaker 1: But my phone's predictive text is going to include it
Speaker 1: as saying this is something that is meaningful and thus
Speaker 1: a valid option. Also, the phones can learn to adapt
Speaker 1: to our own sense of syntax and grammar. Perhaps for
Speaker 1: purposes of a particular effect. One of us tends to
Speaker 1: tweak the syntax of the language that we're communicating in
Speaker 1: for some reason. Maybe it's for comedic effect and it's
Speaker 1: not following the established rules of grammar for English. But
Speaker 1: our phone starts to understand that's how we communicate, based
Speaker 1: on how we order our words and how we generate
Speaker 1: our phrases, you know, how we communicate that. While our
Speaker 1: choices aren't necessarily in alignment with an established formal system,
Speaker 1: they represent a particular approach to communicating. Predictive text can
Speaker 1: start to get a handle on that if it's built properly,
Speaker 1: and even someone who communicates in an idiosyncratic way might
Speaker 1: find that their phone is offering up particularly relevant suggestions.
Speaker 1: So how does all this work? How do machines actually
Speaker 1: learn stuff? Well, there's not one single method, but there
Speaker 1: are a collection of related processes that computer scientists develop
Speaker 1: to train machines. And you can look at two major
Speaker 1: types of categories of machine learning, and there are a
Speaker 1: lot of subtypes under each of these, and those would
Speaker 1: be supervised learning and unsupervised learning. Supervised learning involves training
Speaker 1: a computer model using known input and output information, so
Speaker 1: Let's take an example that I like to use a lot,
Speaker 1: and it's about image recognition. So let's say you're teaching
Speaker 1: a computer to recognize images of coffee mugs, and you
Speaker 1: have an enormous supply of images, millions of them. Some
Speaker 1: of them contain coffee mugs and various shapes and sizes
Speaker 1: and colors and orientations, and the lighting can be different.
Speaker 1: You might have the handle pointing to the left, and
Speaker 1: some or pointing to the right or the other. Some
Speaker 1: cases it might be on its side. But you've got
Speaker 1: tons of these, and you also have millions of images
Speaker 1: of other stuff. Some of it might not even resemble
Speaker 1: a mug remotely. Maybe it's an airplane or Christopher walkin.
Speaker 1: Others might look kind of like a mug, you know,
Speaker 1: it might be a glass or a bowl or something similar. Now,
Speaker 1: as a human being, you can tell straight away if
Speaker 1: the image you've got in front of you represents a
Speaker 1: coffee mug or not, But machines don't inherently possess this ability.
Speaker 1: You could feed one photo of a generic off white
Speaker 1: coffee mug, the handle happens to be pointed to the left,
Speaker 1: and you tag that photo as a coffee mug, you
Speaker 1: give meta data to the computer to classify that as
Speaker 1: a coffee mug. And if you create a database of images,
Speaker 1: maybe you do a search for coffee mug, that one
Speaker 1: would come up as a result because of all the
Speaker 1: work you've done with tagging this thing and effectively telling
Speaker 1: the computer this is what I mean by coffee mug. However,
Speaker 1: if you fed a new image and this one is
Speaker 1: of a red coffee mug that's of a different size,
Speaker 1: maybe the photo has different lighting conditions, maybe the mug
Speaker 1: is a little closer to the camera, the handles point
Speaker 1: to the right and on the left, would the computer
Speaker 1: automatically know that that's a coffee mug. No, it hasn't
Speaker 1: learned that. So you would have to build a predictive
Speaker 1: model for a computer to follow based on the known
Speaker 1: input and outputs. Your output is you want the computer
Speaker 1: to classify photos as either having a coffee mug in
Speaker 1: them or not, And you might use an artificial neural network.
Speaker 1: In this case, you're creating nodes that accept input, then
Speaker 1: they apply some sort of decision making process to that
Speaker 1: input and then pass it along further along the network.
Speaker 1: You can almost think of nodes as essentially making a
Speaker 1: yes or no judgment on a piece of data. Does
Speaker 1: the input qualify or does it not? Does it have
Speaker 1: this particular aspect of whatever it is you're looking at,
Speaker 1: in our case, coffee mugs or does it lack that?
Speaker 1: With our mug example, it could be a simple question
Speaker 1: like is this mug shaped? But the nodes are asking
Speaker 1: lots of questions and making lots of judgments and passing
Speaker 1: them throughout the neural network until you get to the
Speaker 1: final output, the final judgment of is this a coffee
Speaker 1: mug or is it not? And computer scientists influence how
Speaker 1: the computer processes information. They adjust the waiting of answers
Speaker 1: waiting as in like weight, as in heavy W E
Speaker 1: I G H T waiting. So you create your model,
Speaker 1: you use nodes that are making a series of judgments
Speaker 1: on images. You wait those decisions so that you're hopefully
Speaker 1: going toward a more accurate result, and you feed your
Speaker 1: photos through and you look at the output. Now you
Speaker 1: know whether the photos have a coffee mug in them
Speaker 1: or not. You're looking to see if the computer can
Speaker 1: recognize that. So you're looking to see if your model
Speaker 1: succeeded or failed. And then you go back and you
Speaker 1: make adjustments to your neural network. You adjust the waitings
Speaker 1: of those decisions so that the nodes process information in
Speaker 1: a slightly different way, and you always have the goal
Speaker 1: of improving the accuracy of the overall system. You feed
Speaker 1: the images through it again, and you do this over
Speaker 1: and over. You train the computer model so that it
Speaker 1: gets more accurate as you make these adjustments, and ultimately
Speaker 1: you get to a system that can accept brand new images,
Speaker 1: ones that haven't been deliberately chosen, and then sort those
Speaker 1: into images that either are of a coffee mug or
Speaker 1: are not. And this is in an area called classification.
Speaker 1: So in our simple example, images just fall into two
Speaker 1: broad classifications, photos with mugs or photos without, though we're
Speaker 1: gonna get a little more complicated a little bit later,
Speaker 1: so you can have all sorts of classifications. Medical imaging
Speaker 1: systems make use of this sort of machine learning process
Speaker 1: to indicate whether or not an image of a of
Speaker 1: a tumor is benign or not. Handwriting recognition program ms
Speaker 1: do this to speech recognition can do this as well,
Speaker 1: so supervised learning systems can also use a different approach
Speaker 1: called regression as a means of training a system regression
Speaker 1: is all about predicting a continuous response, like how much
Speaker 1: electricity a community is going to need over time. It's
Speaker 1: about predicting things to which you can assign real numbers. So,
Speaker 1: for example, predicting a change in temperature, temperature happens to
Speaker 1: have a value that is a real number, so that
Speaker 1: falls into this category that's supervised learning, where we have
Speaker 1: the known inputs and known outputs. We know definitively if
Speaker 1: the information the computer generates is accurate or not because
Speaker 1: we can actually check its work. It's kind of like
Speaker 1: a teacher grading student tests and then working with a
Speaker 1: student who has a low score to get a better
Speaker 1: understanding of subject matter, and then on the next test
Speaker 1: hopefully they score better, and you keep working with that
Speaker 1: student over and over until they have reached a high
Speaker 1: of level of consistency of being correct. Unsupervised learning is
Speaker 1: more about finding patterns or meaning in data where no
Speaker 1: such patterns or meaning is initially obvious. When we talk
Speaker 1: about sifting through big data to find patterns, this is
Speaker 1: the kind of thing we're talking about. Those patterns might
Speaker 1: be subtle, or they might only be obvious when you're
Speaker 1: dealing with truly enormous amounts of information. We humans are
Speaker 1: really good at spotting patterns up to a point. It's
Speaker 1: part of our survival mechanism. Recognizing patterns helped ancient humans
Speaker 1: recognize prey or predators, so it's a key element to
Speaker 1: the survival of our species. But when you get to really,
Speaker 1: really big quantities of data, it's hard for us to
Speaker 1: see patterns. It would be kind of like if you
Speaker 1: jumped off a boat in the middle of the ocean
Speaker 1: and then you were told to look for patterns that
Speaker 1: are the size of New Zealand you'd be lost right away.
Speaker 1: The scale is something we can't deal with. But computer
Speaker 1: systems can handle data far more efficiently than we can,
Speaker 1: and that means they can potentially spot patterns where we
Speaker 1: would not. Unsupervised learning techniques are best for this, and
Speaker 1: they have a few different approaches. One is clustering, which
Speaker 1: is pretty much what sounds like. The system looks for
Speaker 1: groupings and data indications of clusters, pattern clusters. And now
Speaker 1: I need to get back to my image recognition coffee
Speaker 1: mug analogy. If we were just feeding images that are
Speaker 1: either a coffee mug on a neutral background or something else,
Speaker 1: then we could go supervised learning all the way. But
Speaker 1: if we wanted to create a system that could recognize
Speaker 1: if a coffee mug were in a larger scene, like
Speaker 1: a crowded kitchen table, lots of other stuff is on it,
Speaker 1: we could probably rely a bit on unsupervised learning, in
Speaker 1: which we would use clustering to teach the system to
Speaker 1: look for data that collectively appears to represent a coffee mug.
Speaker 1: We're trying to create a system that can pick out
Speaker 1: the shape of a coffee mug in an image that
Speaker 1: has a lot of other shapes in it. The system
Speaker 1: needs to understand which shapes, which lines and curves represent
Speaker 1: the borders of objects. So what is a coffee mug
Speaker 1: as opposed to say, a tablecloth or a shadow or
Speaker 1: a bowl with a spoon next to it. Unsupervised pattern
Speaker 1: recognition can lead to that outcome. Again, it requires a
Speaker 1: lot of training. You feed millions of images to a
Speaker 1: system numerous times to refine this approach. The method often
Speaker 1: relies upon hidden Markov models. Oh and this also ties
Speaker 1: into something else that's you know, tangentially related. But I
Speaker 1: thought I would bring it up in case you guys
Speaker 1: have been experiencing it as much as I have. If
Speaker 1: you've noticed a lot more instances of websites demanding that
Speaker 1: you prove you're not a robot with a capture. By
Speaker 1: the way, this is a good reminder that if you
Speaker 1: go to the tech stuff store at t public dot
Speaker 1: com slash stores slash tech Stuff, you can get a
Speaker 1: shirt or you know, dare I say, a coffee mug
Speaker 1: with this capture robot idea on it. A lot of
Speaker 1: those captures involve a series of photos, and it's your
Speaker 1: job to click all the photos that have something specific
Speaker 1: in them, you know, like bicycles or crosswalks, or traffic
Speaker 1: lights or fire hydrants. If you've wondered why that is, well,
Speaker 1: it all comes down to good traffic versus bad traffic.
Speaker 1: There's a lot of traffic out there that is uh
Speaker 1: powered by butts for various reasons, and that can clog
Speaker 1: things up, and so systems and companies like Google want
Speaker 1: to prioritize traffic that's good traffic. It represents actual people
Speaker 1: trying to do stuff, and give them preferential access to
Speaker 1: other methods that might be malevolent or just might end
Speaker 1: up making things run slower if they get unfettered access.
Speaker 1: And the reason these captions are getting so difficult is
Speaker 1: because machine learning and image recognition software has gotten really good,
Speaker 1: and so to protect against bad traffic, companies like Google
Speaker 1: are using difficult capture systems that present fuzzy, dimly lit,
Speaker 1: or otherwise you know, bad photographs to you, and your
Speaker 1: job is to stare at them, possibly on a tiny
Speaker 1: smartphone screen, and figure out which ones are legit. The
Speaker 1: whole goal is to present photos that are so lousy
Speaker 1: that machines can't really deal with them. The problem is,
Speaker 1: over the long run, machines get better than doing this
Speaker 1: sort of stuff, whereas we kind of, you know, we
Speaker 1: have a cap on our performance. There will come a
Speaker 1: point where an image will be get you know, too
Speaker 1: fuzzy or too dim for us to make out if
Speaker 1: there's a fire hydrant in there or not. The machines
Speaker 1: will always get better at stuff at this than than
Speaker 1: we are over the long run. Heck, older capture systems
Speaker 1: are completely obsolete now because computer systems can complete them
Speaker 1: at a success rate that's actually higher than humans. We've
Speaker 1: got a lot of science fiction stories about machines becoming
Speaker 1: sentient and ruining humanity, but the truth of the matter
Speaker 1: is they don't need sentients to be disruptive. If they
Speaker 1: are directed by someone for a specific malevolent purpose, that's
Speaker 1: bad enough, even if the machines aren't really you know,
Speaker 1: thinking for themselves. Okay, but let's get back to predictive text.
Speaker 1: After all of this. You could create a machine learning
Speaker 1: model that has a huge database of words, you know,
Speaker 1: a dictionary, and you could program the system to classify
Speaker 1: the words. You can sus out which words are nouns
Speaker 1: and verbs and adjectives, and then apply rules to how
Speaker 1: those words can go together to make sentences. Or you
Speaker 1: could just you know, analyze a ton of literature and
Speaker 1: have the computer kind of figure that out for itself,
Speaker 1: just through statistical analysis, understand how words fit together based
Speaker 1: upon the history of the written word, at least in
Speaker 1: modern English. For example, if you went further back to
Speaker 1: like old English, first of all, your vocabulary would be
Speaker 1: totally different, but your grammar would be too, and suddenly
Speaker 1: things would not make much sense. It would everything would
Speaker 1: sound like yoda. So the system could go through millions
Speaker 1: of pages of materials building a statistical model that shows
Speaker 1: how frequently certain words pair together and in which order. Effectively,
Speaker 1: you're analyzing how humans put letters together to make words,
Speaker 1: and words together to make sentences. You could move up
Speaker 1: from there. You could try and analyze how sentences come
Speaker 1: together to make up paragraphs, but it starts to get tricky. However,
Speaker 1: you can work on a system that can present a
Speaker 1: series of sentences that are related enough to be a
Speaker 1: coherent presentation of ideas, at least in the short run.
Speaker 1: It might not be super compelling or as effective as
Speaker 1: what a human could do, but it could be a
Speaker 1: lot more impressive than just, you know, a string of
Speaker 1: totally unrelated words. When we come back, I'll talk a
Speaker 1: bit more about how computer systems can put words together
Speaker 1: for us and what that could mean in the future.
Speaker 1: But first let's take another quick break. Okay, So, AI systems,
Speaker 1: if sophisticated enough, can use stuff like hidden Markov models
Speaker 1: and machine learning to put together strings of words that,
Speaker 1: from a probability standpoint, a statistical standpoint, at least are
Speaker 1: likely to make some sense. There's no guarantee it will
Speaker 1: actually make sense, but if things are going well, the
Speaker 1: phrases will be grammatically correct, and if they're going really well,
Speaker 1: the word choice will be reasonable enough to pass muster.
Speaker 1: But this is still pretty hard. Computer systems typically lack
Speaker 1: the ability to build on context and meaning because they're
Speaker 1: effectively looking for what is most likely to come next,
Speaker 1: rather than looking back at what has already come before.
Speaker 1: Does that make sense, Well, let me put it in
Speaker 1: another way. In our weather example, I talked about how
Speaker 1: the predictions for future weather depended on current weather. So
Speaker 1: what is it doing today? If it is sunny today,
Speaker 1: there's an eight percent chance it will be sunny tomorrow
Speaker 1: according to our example. But the predictions don't depend upon
Speaker 1: the weather that came earlier, like what happened yesterday. The
Speaker 1: system doesn't care about yesterday's weather. We might care because
Speaker 1: we're using long trends of weather to act as our
Speaker 1: data source to train the computer model, you know, to
Speaker 1: create those probabilities. But yesterday's weather, as far as the
Speaker 1: computer system is concerned, has no impact on tomorrow's weather.
Speaker 1: So if yesterday we're rainy in today is sunny, the
Speaker 1: computer doesn't really care. It just cares that today is sunny.
Speaker 1: The same thing can hold true with systems that are
Speaker 1: creating predictive text. The goal with standard predictive text is
Speaker 1: to save users time and effort by suggesting likely words
Speaker 1: as you, you know, start typing, So if you start
Speaker 1: typing the word technology, at some point, the system recognizes
Speaker 1: the letter pattern and offers that up as an option,
Speaker 1: And for words that are frequently used in pairs, you'll
Speaker 1: get those suggestions right away after you type the first word.
Speaker 1: Since this is typically presented as an option, you know,
Speaker 1: something you can choose to use or not. It's pretty
Speaker 1: simple to avoid going wrong unless you, as a user,
Speaker 1: fumble things and accidentally picked the wrong word, which can
Speaker 1: get kind of embarrassing, or if it autocompletes after the fact,
Speaker 1: thinking that you made a spelling error and then you
Speaker 1: have accidentally spelled Tim mentions name as Tim Munchkin and
Speaker 1: I am deeply sorry for that. Auto replies with email
Speaker 1: get a little more complicated as the system is analyzing
Speaker 1: the message that is coming into you before formulating a
Speaker 1: possible response. So I have email systems that do this
Speaker 1: for me. And one common example for me is that
Speaker 1: our sales team here at our company will send me
Speaker 1: an email asking if I'm okay running a particular sponsors
Speaker 1: ads on my show. Now, normally I like to do
Speaker 1: research on my sponsors, so I'll take time to look
Speaker 1: into things and then respond myself. But sometimes the request
Speaker 1: is for a sponsor I'm familiar with and I definitely
Speaker 1: want or you know, occasionally definitely do not want on
Speaker 1: my show, and I'll see on my phone that I
Speaker 1: have the option to pick a quick reply of something
Speaker 1: like sure or yes, that's fine, or something similar. In
Speaker 1: this case, the email program is using natural language systems
Speaker 1: and predictive text to suss out that there is a
Speaker 1: request and that the common responses I might make to
Speaker 1: that request should be options. Now, it's not that the
Speaker 1: computer system actually understands the nature of this request, but
Speaker 1: more like the structure of a request. In other words,
Speaker 1: it's saying, this looks like it's a yes or no question.
Speaker 1: Let's present him with responses that are in a yes
Speaker 1: or no format. The fact that the system doesn't really
Speaker 1: have a deeper understanding can become evident in other use cases.
Speaker 1: So for example, Janelle Shane, who is a research scientist
Speaker 1: and who has a delightful blog called AI Weirdness, took
Speaker 1: time to try and train a machine learning system to
Speaker 1: tell jokes. It became clear that the system could construct
Speaker 1: something resembling a classic question slash punchline style of joke.
Speaker 1: But it was also clear that the punchline rarely had
Speaker 1: any connection to the question. It actually reminded me a
Speaker 1: lot of how little kids like my two year old
Speaker 1: niece tell jokes. These jokes are some of my favorite
Speaker 1: in the world, not because the jokes are inherently funny,
Speaker 1: but because they are absurd and they show how little
Speaker 1: children can recognize the structure, but not how to build
Speaker 1: an actual joke. My favorite of the AI generated jokes
Speaker 1: almost got it right, and it went like this, what
Speaker 1: do you get when you cross a dinosaur? They get
Speaker 1: a lawyer's I mean, that's that's almost a real joke.
Speaker 1: I actually love that one. Shane pointed out the bit
Speaker 1: that I mentioned earlier that these systems have next to
Speaker 1: no short term memory, and so building any lengthy response
Speaker 1: is pretty much impossible because the computer system is so
Speaker 1: focused on choosing the word that comes next without an
Speaker 1: understanding of the connection or context of what came earlier.
Speaker 1: And you may have come across stuff like a social
Speaker 1: media post that says something along the lines of I
Speaker 1: fed a computer ten thousand movie scripts and asked it
Speaker 1: to write the next you know, Highlander movie or whatever,
Speaker 1: and then you get a little screenplay, and inevitably they
Speaker 1: end up being silly and absurd, with crazy stage directions
Speaker 1: and dialogue and descriptions. They also tend to be written
Speaker 1: entirely by human beings. Most AI systems are incapable of
Speaker 1: keeping things consistent, like character names. A computer system might
Speaker 1: create a character name and give that character align, but
Speaker 1: that name is not likely to return later on in
Speaker 1: the screenplay. It's not necessarily going to show up in
Speaker 1: any stage directions or descriptions. It ends up being more
Speaker 1: dreamlike and free form. It's still absurd, but it's not
Speaker 1: as internally consistent. So if you come across a long
Speaker 1: piece of absurd ast humor that was quote unquote written
Speaker 1: by a computer, chances are it wasn't. It was written
Speaker 1: by a person who was emulating the dreamlike absurdism of
Speaker 1: computer generated text. They're still really funny, they're just not
Speaker 1: necessarily actually generated by a computer. So about that blog
Speaker 1: post that ran on Hacker News. How did that get
Speaker 1: past so many people? It started with Liam Poor, a
Speaker 1: college student, a computer scientist, who made contact with a
Speaker 1: PhD student who in turn had access to a private
Speaker 1: beta build of the GPT three autocomplete tool. Poor created
Speaker 1: a blog post title and an introduction to serve as
Speaker 1: the launch point for the system to build upon. And
Speaker 1: together they ran a few trials with this machine learning
Speaker 1: system and auto generated text system and uh with those prompts,
Speaker 1: and then Poor picked one of the results to submit
Speaker 1: as a legit blog post. Now, I'm going to read
Speaker 1: a little section of it. Now, the blog post title
Speaker 1: was feeling unproductive, maybe you should stop overthinking. And here's
Speaker 1: a segment that comes from the middle of the blog post. Quote.
Speaker 1: When you engage in creative thinking, your brain starts working
Speaker 1: more efficiently. It becomes more active and more open to
Speaker 1: new ideas. It also helps you think outside the box
Speaker 1: and look at things from a different perspective. So how
Speaker 1: does this all tie into productivity. Well, if you're a creator,
Speaker 1: then you should be engaging in creative thinking on a
Speaker 1: regular basis. The more you do it, the better your
Speaker 1: brain becomes at thinking up ideas. This makes it easier
Speaker 1: for you to work on your projects because you won't
Speaker 1: get stuck as often. End quote. Now the phrasing makes sense.
Speaker 1: It's in a very casual style, and other parts of
Speaker 1: the blog post get, you know, even more casual, sometimes
Speaker 1: straying into grammatical error territory. It's not terribly precise, nor
Speaker 1: is it saying anything really. The example I gave to
Speaker 1: a friend of mine is that this blog post is
Speaker 1: just like if I said, you know, if I'm caught
Speaker 1: outside when it starts pouring down rain, I get wet.
Speaker 1: I mean, yeah, that statement is true, but it's also,
Speaker 1: you know, not saying anything, or at least not anything
Speaker 1: that isn't already evident. All that being said, the blog
Speaker 1: post impresses the heck out of me. And that's because
Speaker 1: the paragraphs follow in a logical pattern. It's not well written,
Speaker 1: but there's so much bad writing out there that it
Speaker 1: also doesn't stand out. If I had read this without
Speaker 1: knowing a computer generated it, I'm not certain I would
Speaker 1: pick up on it again. Not because it's great writing,
Speaker 1: but because I've read a lot of really bad writing
Speaker 1: out there. Heck, I've probably written some of it. Think
Speaker 1: of some of the content farms out there that post
Speaker 1: thousands of blog posts a day. There's not as many
Speaker 1: as there were maybe you know, five years ago, but
Speaker 1: there's still quite a few. Well, a lot of that
Speaker 1: content is written in a very quick, slap dash style,
Speaker 1: and and no, no shade being thrown at the writers.
Speaker 1: They're trying to make a living, but it's not exactly
Speaker 1: well crafted work. This piece could have passed for one
Speaker 1: of those, and the piece does actually seem to build
Speaker 1: on itself. New paragraphs reference a point made in an
Speaker 1: earlier paragraph, something that you didn't see so much of
Speaker 1: in other systems. New paragraphs build on those earlier ones,
Speaker 1: not in substantial ways, but there is a coherent link
Speaker 1: from one paragraph to the next. It's not as free
Speaker 1: form and absurd as other generative texts that I've seen.
Speaker 1: As for the autocorrect on our phones, those get more
Speaker 1: individualized as we use them. Like I said, if I
Speaker 1: type a proper name like my dog tim Bolt, my
Speaker 1: phone starts to pick up on this that it's a
Speaker 1: word that has a particular meaning to me, that it's
Speaker 1: also a proper noun because I always capitalize it, and
Speaker 1: that it's not a typo, it's not a misspelling. So
Speaker 1: while the name wasn't in my phone's dictionary when I
Speaker 1: first got it, it has been added to that now
Speaker 1: that I've been using it so much, and it can
Speaker 1: even auto complete the name as I start to type.
Speaker 1: Now we have some really impressive examples of generated text
Speaker 1: or generated language applications in AI. A couple of years ago,
Speaker 1: Google demonstrated how the Google Assistant could make a phone
Speaker 1: call to a real human being operated business and make
Speaker 1: an appointment for you. In a demonstration, the assistant called
Speaker 1: a hair salon and had a brief conversation with the
Speaker 1: salon employee to okay, haircut appointment, and it all sounded,
Speaker 1: you know, fairly natural. This approach to natural language recognition
Speaker 1: and generative language is really powerful stuff. In this case,
Speaker 1: the assistant was relying upon certain parameters. Right The assistant
Speaker 1: knew which salon the user wanted to call. They knew
Speaker 1: the time frame that the user had outlined as being appropriate. Uh.
Speaker 1: In this particular demonstration, it was an appointment slot anytime
Speaker 1: between ten am and twelve pm, and knew what day
Speaker 1: the user wanted an appointment and had all the basics,
Speaker 1: and then the assistant could respond to questions and statements
Speaker 1: from the salon employee on the phone and book the appointment,
Speaker 1: all without obviously revealing that it was an AI program.
Speaker 1: The appearance is that the assistant is able to have
Speaker 1: persistent knowledge, but that's more of an illusion than anything else,
Speaker 1: it does show that computer scientists are making a lot
Speaker 1: of progress towards building systems that can generate language at
Speaker 1: if it's not deeply meaningful, can at least be useful.
Speaker 1: I'll close out was something that I covered at the
Speaker 1: IBM Think Conference back in twenty nineteen. To demonstrate the
Speaker 1: power of the Watson platform, which is a foundation for
Speaker 1: various applications that all tap into deep AI processes, IBM
Speaker 1: organized a debate between a debate champion and a system
Speaker 1: called Project debater or, and the debate was on the
Speaker 1: topic of subsidizing preschools. IBM had drawn the pro side
Speaker 1: of the argument, and I got to watch this debate
Speaker 1: live in person, and it was impressive. Not that I
Speaker 1: felt that Watson was able to outmaneuver the skilled, logical,
Speaker 1: eloquent human champion, but it was able to construct a
Speaker 1: pretty sound and consistent argument. It wasn't as strong and rhetoric,
Speaker 1: but it appeared to parse the flow of the debate
Speaker 1: properly for the most part, constructing arguments and supporting them
Speaker 1: with information wherever possible. It didn't come across as quite human,
Speaker 1: but it was still really impressive. I think it will
Speaker 1: be quite some time before machines can generate text or
Speaker 1: speech at a level that compares with skilled humans, you know,
Speaker 1: humans who incorporate so many things from creativity to insight
Speaker 1: to intelligence in order to build communication. But progress is
Speaker 1: being made all the time, and thanks to a surplus
Speaker 1: of you know, not so great communication out there, we're
Speaker 1: more likely to not notice the computer generated stuff as
Speaker 1: it improves. This opens up a lot of thorny problems.
Speaker 1: We've already got a problem with fake news. In a
Speaker 1: world where computer systems could generate endless blog posts and
Speaker 1: articles supporting narratives that don't reflect the truth, we're really
Speaker 1: going to be in trouble. And I think that's why
Speaker 1: this news about the blog post passing for a real
Speaker 1: article should scare platforms like Facebook. If we reach a
Speaker 1: point where computers can lad Facebook with fake news and
Speaker 1: other computers are running bots that interact with that fake news,
Speaker 1: fewer people are going to stick around on that platform.
Speaker 1: They're going to it's just gonna get a turned to
Speaker 1: a cess pit of of total nonsense. You know, some
Speaker 1: people stick around, but a lot of people are just
Speaker 1: gonna bail. People have been bailing already. We're gonna see
Speaker 1: a lot more leave, and once the advertisers get win
Speaker 1: that the majority of activity on Facebook isn't even human
Speaker 1: and therefore doesn't represent actual potential customers, advertising money will
Speaker 1: start to dry up, and then even a behemoth like
Speaker 1: Facebook could crumble. Now I'm not saying this is going
Speaker 1: to happen quickly, but I think it definitely could and
Speaker 1: probably will happen at least in some respect over the
Speaker 1: course of the next few years. So hey, Facebook, maybe
Speaker 1: think about your oncoming existential crisis and you know, get
Speaker 1: ahead of it. It would be good for everybody, including
Speaker 1: your shareholders, and I know you really care about those alright.
Speaker 1: That wraps up this episode of tech Stuff and how
Speaker 1: artificial intelligence and machine learning and predictive text are all
Speaker 1: evolving rapidly in ways that are both cool and you know, concerning,
Speaker 1: if we're being totally honest, But I want to know
Speaker 1: what you guys think. I also want to know if
Speaker 1: you have any suggestions for future episodes of tech Stuff.
Speaker 1: Reach out to me on Twitter. The handle is text
Speaker 1: stuff h s W and I'll talk to you again
Speaker 1: really soon. Text Stuff is an I Heart Radio production.
Speaker 1: For more podcasts from My Heart Radio, visit the I
Speaker 1: heart radio, app, Apple podcasts, or wherever you listen to
Speaker 1: your favorite shows.