Speaker 1: Brought to you by Toyota. Let's go places. Welcome to
Speaker 1: Forward Thinking, Pater and welcome to Forward Thinking, the podcast
Speaker 1: that looks at the future and says makes you think
Speaker 1: all the world's a sunny day. I'm Jonathan Strickland Obama,
Speaker 1: and I'm Joe mcformick. So today we're going to talk
Speaker 1: about something that I think is a pretty interesting topic,
Speaker 1: and that's the automated description of images. This is yet
Speaker 1: another topic I came across in Alexis Magicals Five Intriguing
Speaker 1: Things Email, which if you're not signed up for you
Speaker 1: should get on that it. It is one of my
Speaker 1: favorite sources of daily delights on the Internet. So what
Speaker 1: is this idea, Well, it's what it sounds like. But
Speaker 1: I'll start with an analogy. Okay, imagine you're going to
Speaker 1: something like Google image search. Um, now, what happens when
Speaker 1: you do a Google image search? You type in some
Speaker 1: words and it comes back with images. Well, that's kind
Speaker 1: of strange because how does it translate the difference between
Speaker 1: a collection of pixels on the one hand and the
Speaker 1: words you've typed in. Well, one of the things, obviously,
Speaker 1: is that there's data associated with images medigata, right, Um,
Speaker 1: it's either captions that people have manually typed in, or
Speaker 1: perhaps keywords that they've attached to those images, or something
Speaker 1: else on the on the website's page that's going to
Speaker 1: clue you into what that image is about, like a
Speaker 1: file name or whatever. You could also use an approach
Speaker 1: that is sort of refined by humans. So you could
Speaker 1: have humans sitting there working on your algorithm where they
Speaker 1: go through image after image from selected keywords and say
Speaker 1: this is a good match for that keyboard and this
Speaker 1: is a bad match for that keyword, and that sort
Speaker 1: of helps you, uh, connect words to images. Or let's say,
Speaker 1: what if there was no text associated with an image,
Speaker 1: could you still do it? Well, in some cases you
Speaker 1: probably could, right, because we've gotten to a certain level
Speaker 1: with image recognition. Uh, there are automated programs that can
Speaker 1: look at this and say this is a human face
Speaker 1: or this is a cat, as we have discussed before
Speaker 1: on the show. And well, I'm sure right, and we
Speaker 1: we I think we don't know the full extent to
Speaker 1: which artificial intelligence like that already figures into something like
Speaker 1: Google Image Search. I wouldn't be surprised if that was
Speaker 1: a small part of it. But obviously we're relying heavily
Speaker 1: on text associated with images. Okay, but now let's take
Speaker 1: that same last example, just identifying an image with no
Speaker 1: associated text, and say, could we do that with a
Speaker 1: complex scenario. So it's not just a picture of a
Speaker 1: human face, or say a bowling ball, which would be
Speaker 1: pretty easy to recognize, as you know, it's round, it's
Speaker 1: got three holes, but something like there is a pizza
Speaker 1: sitting in a bathtub, or a man throwing a sandwich
Speaker 1: off a cliff. I've got a lot of food related
Speaker 1: imagery in your in your head. Well, I said those
Speaker 1: because I actually searched for them earlier before i'd had lunch. Yeah, alright,
Speaker 1: and I found no images of someone throwing a sandwich
Speaker 1: off a cliff. Well, why would anyone want to throw
Speaker 1: a sandwich off a cliff? I don't know, But you
Speaker 1: know what, I would be really surprised if there wasn't
Speaker 1: at least one picture of that out there somewhere. If
Speaker 1: there's not, there's definitely a stock photography opportunity lying in Wait, yeah,
Speaker 1: you know what we're doing after the podcast. Um, there's
Speaker 1: a reason that this would be hard, right to describe
Speaker 1: a complex image not just one thing, but a complex
Speaker 1: sort of scene that requires a sentence to describe it, right,
Speaker 1: with propositional phrases and and situational um relativity. Yeah, exactly,
Speaker 1: you have to be able to describe the relationship between
Speaker 1: all the different elements that are inside that picture. Right.
Speaker 1: And so that last example is what we're going to
Speaker 1: talk about today, how computers can look at an image
Speaker 1: with a complex scene taking place and turn that into
Speaker 1: a correct and accurate description made out of words. And
Speaker 1: this this goes into all, you know, a key element
Speaker 1: of artificial intelligence. It's not just the ability to describe something,
Speaker 1: but the ability to recognize it. It's something that can
Speaker 1: go beyond just describing a picture. And we talk a
Speaker 1: lot about how there are things that that we humans
Speaker 1: are really good at. It's it comes naturally to us,
Speaker 1: it's the way we work. But they are things that
Speaker 1: do not necessarily come naturally to the machines we make um.
Speaker 1: And the example I gave was if you had a
Speaker 1: group of people, you know, you've got a bunch of
Speaker 1: people together, and you told all of them, I want
Speaker 1: you to draw this picture, and you describe the picture
Speaker 1: to them. And in my case, I said, well, just
Speaker 1: imagine that there's a young lady saying at a table
Speaker 1: reading a book, and just make the picture as detailed
Speaker 1: as you can. But all that's all I give is
Speaker 1: just the elements that have to be there. Is there's
Speaker 1: a seated lady. Uh that she's at a table and
Speaker 1: she's reading a book. And so you could have all
Speaker 1: these different types of interpretations of that request. Sure, I
Speaker 1: might draw a picture of a lady sitting in a
Speaker 1: cafe reading a book. Or it could be a desk
Speaker 1: at a library. There's so many different opportunities there. Uh,
Speaker 1: could just be a kitchen table whatever. But if I
Speaker 1: took all the pictures that everyone drew, and then I
Speaker 1: went to a separate group of people and I showed
Speaker 1: them those all those different pictures, and I all right,
Speaker 1: so what do all these pictures have in common? Pretty
Speaker 1: sure that a lot of people would end up saying, Okay,
Speaker 1: these are all pictures of a young lady reading a
Speaker 1: book at a table. It would be kind of weird
Speaker 1: if in every picture she was reading tech war. That
Speaker 1: would be a very weird. For lots of reasons that
Speaker 1: would be weird. But at any rate, yeah, I mean,
Speaker 1: we would have essentially people answering the same you know,
Speaker 1: giving the same basic description. Now, they're a lot of
Speaker 1: things going on in that scenario I just described. It's
Speaker 1: not just the fact that you're able to recognize things,
Speaker 1: it's that you're able to draw the conclusion that all
Speaker 1: these different pictures, even though the details are different, are
Speaker 1: showing you essentially the same thing, which is something that
Speaker 1: would be necessary if we were using an image search
Speaker 1: that relied on this automated image description. Right, well, I mean,
Speaker 1: and furthermore, we as humans are able to recognize a
Speaker 1: lady sitting at a table reading a book from any
Speaker 1: angle that it's drawn from right right, pretty easily. Yeah,
Speaker 1: we can tell. We can tell like that this is
Speaker 1: the book, this is the lady, this is the table.
Speaker 1: If you show just pixels to a machine that hasn't
Speaker 1: had any way of telling the difference between, you know,
Speaker 1: what these pixels actually mean, they might not That machine
Speaker 1: might not be able to tell that there are distinct
Speaker 1: elements in that picture, right, it may all just look
Speaker 1: like one thing. So there are a lot of complications here, uh,
Speaker 1: same sort of We brought this example up a few times,
Speaker 1: same sort of thing, like I know what a cup
Speaker 1: is because I've seen a cup, but I was told
Speaker 1: this is a cup, and you're able to extrapolate many
Speaker 1: different kinds of cups. Exactly. You have an ideal in
Speaker 1: your head. It's sort of the platonic ideal of a cup,
Speaker 1: and there are many ways that a actual cup can
Speaker 1: vary that theme, but somehow you always recognize the theme.
Speaker 1: I recognize every single cup in existence as an imperfect
Speaker 1: realization of the ideal cup that's in my mind. Uh so,
Speaker 1: which has grimace on it, by the way, But at
Speaker 1: any rate, Yeah, and so the again, a computer, you
Speaker 1: could if you gave it an image and you programmed
Speaker 1: in some software and said, this particular image that you're
Speaker 1: seeing here is a cup. But then you took a
Speaker 1: totally different kind of cup, different shape, different size, different color.
Speaker 1: The computer is not necessarily going to know what that is,
Speaker 1: right right, it's going to say, well, this one's blue. Yeah,
Speaker 1: you need a huge shorter and it's from a different angle.
Speaker 1: It's it's one of those things where you you know
Speaker 1: there's a difficult problem and there's not necessarily a simple
Speaker 1: solution to fix it. The computer understands what's going on.
Speaker 1: So the point I'm trying to make is that this
Speaker 1: is a non trivial computer problem that a lot of
Speaker 1: people have worked on for a long time, and it's
Speaker 1: absolutely amazing to see how much progress we've made. Absolutely
Speaker 1: and furthermore, this is only half of the issue that
Speaker 1: we're dealing with here overall. Because once, I mean, once
Speaker 1: you can teach a computer to identify, for example, a cup,
Speaker 1: how do you get it to to explain what that
Speaker 1: cup is doing in relation to the other things? How
Speaker 1: does it describe it in a way that actually makes
Speaker 1: sense to That's a natural language problem, exactly right, because
Speaker 1: the computer doesn't think in English or whatever language you
Speaker 1: wanted to spit out right. Yeah, that's and we've talked
Speaker 1: about natural language issues as well, the idea that machine
Speaker 1: language and natural language are extremely different, and in fact,
Speaker 1: programming languages are a bridge between pure machine language and
Speaker 1: human language. You might not think it if you're not
Speaker 1: a programmer and you look at raw code, you might think, well,
Speaker 1: this isn't language that any human would understand, But in
Speaker 1: fact that's precisely what it is. It's meant to be
Speaker 1: that that bridging material. Uh So, getting computers so that
Speaker 1: they can interpret the natural language innately is a very
Speaker 1: challenging issue. We've said it before that you can word
Speaker 1: the exact same thought numerous ways. Human language is highly,
Speaker 1: highly redundant. Yes, they're all different kinds, and the differences
Speaker 1: in in word choice might express subtle differences in tone
Speaker 1: and things like that, but you can basically describe the
Speaker 1: same thing a jillion different ways different spellings. So, for example,
Speaker 1: if you're playing an old text adventure game like Zorc, yes,
Speaker 1: you could type walk down the hall or go down
Speaker 1: the hall, and that text parsing system might be smart
Speaker 1: enough to know either one and get you down the hall.
Speaker 1: So like, okay, I know what the person just said,
Speaker 1: but if you type mosey on down the corridor, it
Speaker 1: may very well say I don't know what you're talking about. Right,
Speaker 1: What did Zorc say when you said something I'm sorry,
Speaker 1: I don't understand what you mean, or something along those lines,
Speaker 1: or you were eaten by a group if it's too
Speaker 1: frustrated at you, if you're too confusing too often grew Yeah, no,
Speaker 1: but that's a great example the idea that you know, obviously,
Speaker 1: those those programs were only capable of accepting commands that
Speaker 1: had been pre programmed into them, and anything that went
Speaker 1: outside those parameters was an error. It was missing. The
Speaker 1: message we would get is I didn't understand what you
Speaker 1: had to say, but in reality it could have just
Speaker 1: as well been found not found. Right. Okay, So now
Speaker 1: we're combining these two different artificial intelligence problems. On one
Speaker 1: the complex problem of looking at a scene and recognizing
Speaker 1: what's going on there, connecting that to other images and context,
Speaker 1: and the other one making sense of it in a
Speaker 1: written language. And uh, why does this matter? Yeah? Why
Speaker 1: why do we even want to do this if it's
Speaker 1: so difficult? A lot a lot of reasons. One of
Speaker 1: the Well, first of all, we talked about image search
Speaker 1: and just being able to to automate this would make
Speaker 1: image search way more efficient for the things that we're
Speaker 1: looking for. We could be much more specific. Oh man,
Speaker 1: if only I could tell it. So. Part of my
Speaker 1: job is searching for stock images that I end up
Speaker 1: publishing on our website and and human labeling of stock images.
Speaker 1: If you guys have never had to search for stock
Speaker 1: images for your job before, let me tell you it's
Speaker 1: one of the most joyous and terrifying things on the planet.
Speaker 1: Because no matter what you type in, what you're going
Speaker 1: to get back is some weird clip art, some sexy
Speaker 1: ladies doing some stuff that may or may not have
Speaker 1: anything to do with what you just said and may not,
Speaker 1: and and maybe what you are actually looking you might
Speaker 1: also find some like truly weird images, like a an overweight,
Speaker 1: shirtless gentleman sitting at a table with a paper bag
Speaker 1: over his head, knife and fork in his hands, eating
Speaker 1: a cartoon hamster. I mean, it's some weird stuff on there. Seriously,
Speaker 1: I am almost positive that there is a stock photograph
Speaker 1: out there somewhere of somebody throwing a sandwich off a
Speaker 1: cliff for like they've made it for diet purposes or
Speaker 1: something like that. Didn't get endless examples of that by that, right?
Speaker 1: If I searched for man throwing a sandwich off a cliff,
Speaker 1: I should have tried this before we started, but I'm
Speaker 1: almost positive I would not get that. Instead, I get
Speaker 1: a lady in a bikini sitting on a pinball machine. Yeah,
Speaker 1: that's that's pretty accurate. So just increasing the accuracy of
Speaker 1: image search would be one reason, right, And and there
Speaker 1: are lots of different reasons why our our image searches
Speaker 1: on these things are imperfect. A large part of it
Speaker 1: is that you've got people gaming the system. They're essentially
Speaker 1: putting in any tag word they can possibly think of
Speaker 1: because they want their images to be the ones that
Speaker 1: are purchased. But but if you want to play fair,
Speaker 1: then having an automated description would be best because you
Speaker 1: can't curate everything by human eye. It would just take
Speaker 1: too long. We're generating too much content for that to
Speaker 1: be a realistic possibility. But another is that it could
Speaker 1: be a huge help for people who have visual impairments.
Speaker 1: So someone who is reading a news story, you know,
Speaker 1: someone who has who has some sort of visual impairment,
Speaker 1: maybe they're blind, and there could be pictures that give
Speaker 1: more context to whatever the story is, but they miss
Speaker 1: out on that if there's not an actual description of
Speaker 1: what that picture is. Particularly, I mean, there's some content
Speaker 1: out there where the caption might be playful but doesn't
Speaker 1: actually tell you what the picture is. Absolutely, so this
Speaker 1: would be a big help for people who are in
Speaker 1: that situation. It also could speed up web access for
Speaker 1: people who have limited connectivity to the Internet. Perhaps it's
Speaker 1: over through a cellular network, and it may be that
Speaker 1: there's some important information they need to get hold of.
Speaker 1: But you know, if they're trying to load pictures, it's
Speaker 1: just taking too long for it to load any kind
Speaker 1: of thing. You could have a quick summary of those pictures.
Speaker 1: That would really speed things up. Because actually, I also
Speaker 1: think it's just an important contribution to general artificial intelligence. Absolutely,
Speaker 1: if you're trying to create a system that can mimic
Speaker 1: all of the functions of the human mind, well, one
Speaker 1: of the main things humans do is look at something
Speaker 1: and describe it, right, and you know, the description is
Speaker 1: just part of it. There's also the interaction, right by
Speaker 1: by recognizing things in our environment, we know how to proceed.
Speaker 1: We can make decisions on how to proceed. So if,
Speaker 1: for example, we walk into a room and we noticed
Speaker 1: that there are a lot of pedestals around us, and
Speaker 1: they're delicate vases on each pedestal, we know not to
Speaker 1: go swing in our arms everywhere willy nilly. But you know,
Speaker 1: a robot would not necessarily be able to tell that
Speaker 1: a a vause sitting on a pestel was not in
Speaker 1: fact a single piece. It might it might interpret that
Speaker 1: as a column that's a good point, you know, or
Speaker 1: that there's even even if it could recognize that was
Speaker 1: an object sitting on another object the vase was delicate
Speaker 1: or that it was worth not smashing. Right, yeah, robots
Speaker 1: Teaching robots value is a very tricky thing. Also, they
Speaker 1: hate vauses they do. It's I think, I think you
Speaker 1: program them, I think programming. All well, it's because we
Speaker 1: gave them the basic personality of Gallagher is the problem. Okay, okay,
Speaker 1: so we've so automated image description is a very difficult problem,
Speaker 1: but it's also very worth solving. But in the long
Speaker 1: term for artificial intelligence, and in some specific cases, in
Speaker 1: the short term. Who's actually working on this? Where did
Speaker 1: this come from? Well, I mean there are lots of
Speaker 1: different people in computer science working on this problem. But
Speaker 1: the thing that kind of spur spurred on this particular
Speaker 1: podcast you discovered, right, Yeah, it was well, as I said,
Speaker 1: it was through Alexis Madrigals five Intriguing Things newsletter, and
Speaker 1: it was a link to the Google Research blog, which
Speaker 1: is a cool little blog. Some of it's definitely over
Speaker 1: the average reader's head, but it's also just very interesting.
Speaker 1: And yeah, yeah, um, this specific blog entry is from
Speaker 1: Google UK uh and it was posted by a bunch
Speaker 1: of research scientists. It's it's one of those things where
Speaker 1: if you go into the Google research blog, they do
Speaker 1: get um more technical than your average blog does. They're
Speaker 1: not so technical as to be completely incomprehensible, but I
Speaker 1: will say they're really good about linking out two terms
Speaker 1: that you might be unfamiliar, which is important because I
Speaker 1: had to click on every single one of those links.
Speaker 1: I did so much reading for this one blog post,
Speaker 1: so I could really get a handle on what they
Speaker 1: were saying. But again, it illustrates the complexity of the problem.
Speaker 1: So again we don't mean to suggest that these Google
Speaker 1: researchers are the only people working on this problem, or
Speaker 1: that their approach is the only way to do it.
Speaker 1: We're constraint on it because it was really well documented
Speaker 1: and it was just published on November sevente We are
Speaker 1: recording this on November two. Any first, so it was
Speaker 1: of immediate interest to us. Yes, okay, so how are
Speaker 1: we doing this? Well? First, you have to identify what
Speaker 1: needs to be done before you can figure out how
Speaker 1: to do it right. You have to figure out the
Speaker 1: the things that have to happen in order for this
Speaker 1: to be a possibility. And they identified several things, including
Speaker 1: computer vision, which is how machines acquire and analyze images.
Speaker 1: So how do they get the images in the first place.
Speaker 1: Is it purely through code? Is it actual visual you know?
Speaker 1: Is like like a camera system. I mean, if you're
Speaker 1: talking about robotics, and it's probably a camera system because
Speaker 1: they're looking around in their environment. It could just be
Speaker 1: sampling from the Internet or something like right right, Well
Speaker 1: with with something like an automated search, it could all
Speaker 1: be code. Like it could be that there's no quote
Speaker 1: unquote looking at the image, right, but so there's that.
Speaker 1: There's also object detection, which sounds really easy, but it's
Speaker 1: incredibly hard. So this is what I was talking about,
Speaker 1: being able to recognize in visual objects within an image.
Speaker 1: So what separates an object from its background? If I've
Speaker 1: got a shot, a top down shot of a table,
Speaker 1: and there's a book sitting on the middle of that table,
Speaker 1: then when I look down, I can see that there's
Speaker 1: a book and there's a table, and I recognize those
Speaker 1: is two different things. But like I was saying before,
Speaker 1: if it's a machine and it doesn't have this way
Speaker 1: of of telling the difference there, it may just think
Speaker 1: of that as a pattern that's on a table, you know,
Speaker 1: or even a raised part of that table. If it
Speaker 1: can detect depth, it's not an it's not a cut
Speaker 1: and dry thing. So getting a point where you can
Speaker 1: have a computer that can tell that there are multiple
Speaker 1: objects within a scene, that's already a challenge, although we've
Speaker 1: gone a far away to actually do that. But imagine
Speaker 1: that you are looking at if you want to think
Speaker 1: about how hard this is, imagine you're thinking at a
Speaker 1: at a overgrown field and there's someone in a gilly
Speaker 1: suit out there. Gilly suit are those camos suits that
Speaker 1: have all the plant type material hanging off of them.
Speaker 1: Some of its artificial, some of it may actually be
Speaker 1: gathered from wherever you're going. Those camouflage suits are really convincing.
Speaker 1: It's really hard to pick someone who's someone who's good
Speaker 1: at at being uh covert. You may not even know
Speaker 1: that there's a person there, and so that's as hard
Speaker 1: as that is for us. That about the equivalent of
Speaker 1: looking at a book on a table for a computer exactly. Yeah.
Speaker 1: You know, until you're able to teach a machine how
Speaker 1: to how to see in a way, then it's going
Speaker 1: to have it's going to be just as as difficult
Speaker 1: to detect that as it would be for us to
Speaker 1: see that guy in the gilly suit in the middle
Speaker 1: of the overgrown field. Um. But once you do get
Speaker 1: to that, you still have other things to keep in mind,
Speaker 1: like classification and labeling. So this is a measurement of
Speaker 1: how accurately a program can assign correct labels to an image. Uh.
Speaker 1: I love the example, like if you looked at certain
Speaker 1: examples within the Google blog post and eventually took you
Speaker 1: to a picture of a dog wearing a sombrero, so
Speaker 1: you had dog hat. There's a hat on a dog,
Speaker 1: a wide brimmed hat. Yeah, these are all important elements.
Speaker 1: That's part of that classification and labeling. Uh. You know,
Speaker 1: instead of just saying that's one ugly dog because not
Speaker 1: being a weird growth, yeah, that would be that would
Speaker 1: be more than that. It's it's uh dog with the
Speaker 1: hat and not just dog with something on it. Yeah,
Speaker 1: so again not dog with stack of pancakes. Yeah, And
Speaker 1: identifying the fact that there is that relationship that there
Speaker 1: is a hat on a dog, not just that there's
Speaker 1: a hat and a dog in the picture, but how
Speaker 1: did those two objects relate to one another within the
Speaker 1: context of that picture. So that's also pretty cool. So
Speaker 1: as We've said, lots of folks have been working on
Speaker 1: the problem of how to do this thing, and a
Speaker 1: common approach has been working on linking computer systems that
Speaker 1: understand what's going on in pictures with computer systems that
Speaker 1: understand what's going on in sentences, and letting the two
Speaker 1: match up images and phrases. But these scientists at Google
Speaker 1: were approaching it from a different way. They're trying to
Speaker 1: create a system where the two halves work together directly
Speaker 1: with the same data, rather than comparing and contrasting two
Speaker 1: separate sets of data. Okay, um, so they were. They
Speaker 1: were inspired to do this by recent language translation research
Speaker 1: in which one half of the system would create a
Speaker 1: diagram of a sentence in one language, say English, and
Speaker 1: the second half would look at that diagram and generate
Speaker 1: a sentence from it in another language, say French. Yeah.
Speaker 1: And again this was a part of classification. It wasn't
Speaker 1: It wasn't just a word to word, you know, what
Speaker 1: is the what is the analogous word in this other
Speaker 1: language for this the one that we're detecting here, but
Speaker 1: rather what is the meaning of this sentence and what
Speaker 1: is the what is the phrase in this other language
Speaker 1: that has that same meaning exactly, because because doing word
Speaker 1: for word translations, if you've ever worked in another language,
Speaker 1: often doesn't at all. Right, and you realize as you
Speaker 1: say it to someone who's a native speaker of that language,
Speaker 1: is said, that's a very weird way of putting what
Speaker 1: you just said. Um, so yeah, it's it's a really
Speaker 1: interesting idea, and the way they do it is particularly technical.
Speaker 1: So I'm going to take this from a very kind
Speaker 1: of high level because well, first and what I gotta
Speaker 1: be totally honest, it goes so technical. I definitely don't
Speaker 1: understand all of it. So we're talking about neural networks. Yeah,
Speaker 1: so I'm trying to take it from a high enough
Speaker 1: level where I feel like I still have a general
Speaker 1: grip on what's happening. But I'm if I dove down
Speaker 1: any deeper than I would most likely be giving at
Speaker 1: least equal amounts information and misinformation. Yeah. Yeah, but so so,
Speaker 1: like like just said, we're talking about artificial neural networks,
Speaker 1: which are the systems that these language researchers were using,
Speaker 1: and you decided to go with the same thing. Yeah,
Speaker 1: they're systems that are attempting to replicate something that happens
Speaker 1: inside an organic brain. Yeah, so anything that is an
Speaker 1: artificial neural network is trying to to kind of mimic
Speaker 1: nature essentially at some level. Some of them are more
Speaker 1: like the neural networks you would see in our brains.
Speaker 1: Some of them react like neurons, but they are arranged
Speaker 1: in a way that's very different from the way our
Speaker 1: brains are arranged. So, uh, you know, it's when we
Speaker 1: say these neural networks, just keep in mind we're not
Speaker 1: necessarily talking about an artificial brain. It's not the same thing.
Speaker 1: It's more like the tiny constituents, you know, millions of
Speaker 1: which make up a brain. Right. Think of each neuron
Speaker 1: as capable of of doing a various processes, whatever those
Speaker 1: processes might be, on information that comes to it, and
Speaker 1: then send it on to other neurons, which will then
Speaker 1: add their own element of whatever it may be. So
Speaker 1: one of them might be all right, whenever I get
Speaker 1: input from uh, this other neuron, I know to perform
Speaker 1: this specific mathematic process and then I know to pass
Speaker 1: it on to this other neuron. And uh, even that's
Speaker 1: an oversimplification, but it gives you kind of an idea
Speaker 1: of what's going on. So one of the two types
Speaker 1: of artificial neural networks used by Google, and keep in
Speaker 1: mind there are lots of different variations of artificial neural networks.
Speaker 1: It's called a convolutional neural network. And it's funny because
Speaker 1: when I think convoluted, I don't think of that as
Speaker 1: being a positive thing. But convolutional neural network in this
Speaker 1: case is a feed forward neural network, which means there's
Speaker 1: an actual pathway to follow that has a beginning and
Speaker 1: an end. So think of it like, uh, you know,
Speaker 1: it's it's gonna start and and a destination. Uh and
Speaker 1: and things always begin at the start and they always
Speaker 1: end up at the destination. And the pathway has lots
Speaker 1: of neurons along it that can do work upon whatever
Speaker 1: the input is. So you have this is the start
Speaker 1: um and it's essentially there to classify objects within an image,
Speaker 1: and that data ends up getting encoded according to however
Speaker 1: you've programmed that network. Al Right, you wind up with
Speaker 1: this with this really huge amount of data coming off
Speaker 1: of a single image, right, and all of that gets
Speaker 1: fed into the second artificial neural network, which is called
Speaker 1: a recurrent neural network, which is not a direct pathway.
Speaker 1: This is an interconnected model that creates a what is
Speaker 1: called a directed cycle. So there are several subsets of
Speaker 1: this kind of network. Uh. The fully recurrent network is
Speaker 1: the probably the easiest to imagine. That's one where every
Speaker 1: single neuron has a direct connection with our directed connection
Speaker 1: with every single other neuron within the network. Uh. Now,
Speaker 1: these obviously get way more complicated the more neurons you add.
Speaker 1: This is one of the reasons why having an artificial
Speaker 1: brain model is so hard, because you're talking on the
Speaker 1: order of eighty billion neurons to make a simple artificial brain.
Speaker 1: Eighty billion units that have interconnections with not necessarily every
Speaker 1: other node like every other neuron, but enough to make
Speaker 1: this complicated and slow on the classical computing scale, and
Speaker 1: then away getting back to their's. Uh, this one is
Speaker 1: what will end up uh processing all that information to
Speaker 1: describe the images that were classified from the first one.
Speaker 1: So the first one classifies all the stuff. This one
Speaker 1: is what creates the language used to describe those images
Speaker 1: to make them meaningful to a human audience, so that
Speaker 1: when we get the description, it actually it reflects whatever
Speaker 1: the picture shows and doesn't like It's not like a
Speaker 1: lady sitting down at a table reading a book and
Speaker 1: it says frog dancing on skyscraper. Well hopefully, I mean,
Speaker 1: I mean the system is certainly not perfect. It's it's
Speaker 1: a really good system, and it's neat because this this
Speaker 1: these neural networks. The other thing that these do that
Speaker 1: I didn't mention before is they can learn. They can
Speaker 1: you know, once once a process goes through, if you
Speaker 1: start making that process, you know, replicating that process, either
Speaker 1: by feeding the same thing through over and over feeding
Speaker 1: similar things through, it starts to pick up on that.
Speaker 1: So this is very similar to that idea we had
Speaker 1: about feeding in the the thousands and thousands of images
Speaker 1: and videos of cats and how the machine was able
Speaker 1: to learn what a cat was without anyone telling it
Speaker 1: what a cat was the same sort of thing here.
Speaker 1: It's it's that same process. It ends up kind of
Speaker 1: like a memory. How our memories are pathways of neurons
Speaker 1: that fire in a specific sequence more or less, and
Speaker 1: every time we remember it, we're replicating that as close
Speaker 1: as we can. Anyway, Uh, similar to what's going on here.
Speaker 1: Pretty cool And like I said, to get more more
Speaker 1: detailed is beyond me, yea, beyond beyond any of us
Speaker 1: sitting at this table. Um, I would check out that
Speaker 1: blog post if you get a chance, because especially because
Speaker 1: it's got a great image where it sort of shows
Speaker 1: the difference between what a good description of an image
Speaker 1: looks like and what I failed description looks like. Yeah,
Speaker 1: radiance in between. Sy sure it had this rating system
Speaker 1: or has it probably hasn't been stroid since we've made
Speaker 1: this podcast. I hope that. Otherwise we're futile and telling
Speaker 1: you check it out. We're backward thinkers. Um. But but
Speaker 1: but yeah, so so humans have ranked the photos by like, well,
Speaker 1: this totally has this is an accurate description all the
Speaker 1: way to this is not what that is. So, for example,
Speaker 1: like a person riding a motorcycle on a dirt road
Speaker 1: is described with the sentence a person riding a motorcycle
Speaker 1: on a dirt road sort of a dirt road. Actually,
Speaker 1: I would say it looks kind of like a motocross track.
Speaker 1: But that's enough. Yeah, okay, And then it would have
Speaker 1: sort of like describes with minor errors. So there was
Speaker 1: one that says close up of a cat laying on
Speaker 1: a couch, it's a cat sitting on a bed. But
Speaker 1: cat in the photo, it's not really a close up,
Speaker 1: but yeah, you know, it's sort of sort of what's
Speaker 1: going on. Um. My favorite was the one that's labeled
Speaker 1: a refrigerator filled with lots of food and drinks and
Speaker 1: in fact it's some kind of parking sign with stickers
Speaker 1: all for it. Yeah, in a in a busy city tunnel.
Speaker 1: It looks like and if you look at the images
Speaker 1: the way that they they show how the computer quote
Speaker 1: unquote sees the image, which really it's just it's just
Speaker 1: a visualization for our benefit, but it shows with different
Speaker 1: color boxes around each individual element to show how it
Speaker 1: how it's picking the mountain labeling it. It gives you
Speaker 1: an idea like when you when you think of it
Speaker 1: that way, you think, wow, this is a lot more
Speaker 1: complicated than I imagine. I mean, we we often think like,
Speaker 1: I don't know what you guys think. I shouldn't say
Speaker 1: we I said. I often think of automated image description.
Speaker 1: I often will imagine the simplest of images in my head,
Speaker 1: just thinking, like, you know, it would look like an
Speaker 1: Apple commercial, you know, white background with one solitary image
Speaker 1: in the middle, and the description of what that is.
Speaker 1: I don't necessarily think, oh wait, no, this refers to
Speaker 1: every kind of image, including you know, pictures of me
Speaker 1: and my friends at at a restaurant with lots of
Speaker 1: other stuff going on. Like when you think about that
Speaker 1: and all the different elements that can appear in a
Speaker 1: single picture, then you realize, wow, this is this is
Speaker 1: really amazing that they've been able to create anything remotely
Speaker 1: approaching automated image, because you know, that's that's the thing
Speaker 1: that our eyes do. They naturally pick out items of
Speaker 1: interest from visual scenes. So the same sort of thing
Speaker 1: again applies with robotics. I mean, you know, I know
Speaker 1: this is mostly about image description, but the same kind
Speaker 1: of of processing is really important for machines, especially machines
Speaker 1: that are going to be interacting with humans on a
Speaker 1: more frequent basis, to be able to recognize an environment,
Speaker 1: not just to pick out the potential obstacles that a
Speaker 1: robot might encounter so I can maneuver around them, but
Speaker 1: also just to understand, h, these are the elements within
Speaker 1: this scene that I need to be careful around because
Speaker 1: they are either people and thus I don't want to
Speaker 1: injure them, or they are things that are delicate and
Speaker 1: I don't want to break them, or these are the
Speaker 1: objects that I need to interact with, and this is
Speaker 1: how I might go about interacting with them exactly. And
Speaker 1: of course, beyond that, to communicate information about the environment.
Speaker 1: And I mean that's something that would be amazingly helpful
Speaker 1: if a robot could describe to you what happened three
Speaker 1: minutes ago, yeah, you know well, And and for robots,
Speaker 1: we often talk about robots used for first responders, a
Speaker 1: robot that could be able to not just look for
Speaker 1: signs of life, but actively describe the environment back to operators,
Speaker 1: so that you know, say like, oh, hey, that pillar
Speaker 1: is down over there, and the floor is caved in
Speaker 1: over here, and there's a pile of debris and yeahah yeah,
Speaker 1: because there are a lot of sensors on a robot
Speaker 1: that can detect things that don't necessarily translate into direct
Speaker 1: visual data for us, you know, not just the cameras
Speaker 1: but other things, and for red sensors, stuff that you know,
Speaker 1: would need some processing on our end to make it meaningful.
Speaker 1: But if it's able to communicate directly that information, that
Speaker 1: would be incredibly helpful. So uh, very important part of
Speaker 1: artificial intelligence. And again I think it also illustrates that
Speaker 1: artificial intelligence is a much bigger idea than just a
Speaker 1: machine that quote unquote thinks like we do. That's the
Speaker 1: way a lot of us will describe like that. That's
Speaker 1: kind of my go to thought whenever I hear the
Speaker 1: words artificial intelligence. Mainly I guess because of Hollywood, But
Speaker 1: the reality is that it encompasses way more than that. So, uh,
Speaker 1: excellent work on uncovering that little item and and suggesting it, Joe.
Speaker 1: I think it made a great topic. As I've said,
Speaker 1: it weren't me, Well, you saw it and then you
Speaker 1: brought it to our attention. If it hadn't been for you,
Speaker 1: we probably been talking about something that would not have
Speaker 1: required me to fire my entire brain at it for
Speaker 1: a day and a half of solid research. Well, I
Speaker 1: spend most of my time trying to figure out weird
Speaker 1: ways to make you exercise your neurons because I need
Speaker 1: those healthy for something I'm planning on doing in a
Speaker 1: couple of months. That's that's all good, good, that's good.
Speaker 1: I personally thank you, Joe, because otherwise I'd just be
Speaker 1: saying here drooling. So I you know, sure your plan
Speaker 1: is likely nefarious, but I'm gonna go along with it
Speaker 1: because it's benefiting me in the short term. What's your
Speaker 1: blood type? So forward thinking? If you have suggestions for
Speaker 1: future topics, you should get in touch with us and
Speaker 1: let us know what those are. Our email addresses FW
Speaker 1: thinking at how stuff Works dot com, or drop us
Speaker 1: a line on Google Plus, on Facebook or on Twitter.
Speaker 1: At Twitter and Google Plus, we have the handle fw thinking.
Speaker 1: Just search for that at Facebook will pop right up.
Speaker 1: Let us know what you want to hear about, or
Speaker 1: maybe you want to chime in, or maybe you found
Speaker 1: that that stock image of a of a man throwing
Speaker 1: a sandwich off a cliff. Oh I want to see that.
Speaker 1: If you send it in, I will replace whatever image
Speaker 1: I originally put with this podcast and use that and
Speaker 1: uh and we'll even we'll even throw credit in. We'll
Speaker 1: say that you know you're the one who found it
Speaker 1: for us. So well, assuming now their photo permissions, yes, yes,
Speaker 1: we have to be able to we have to be able,
Speaker 1: we have to be able to use it with permission.
Speaker 1: So if it's if it's a stock photo that we
Speaker 1: you know, from a stock company, that we actually use,
Speaker 1: then we can we can totally do that. And uh
Speaker 1: so as long as that all liigns up I know
Speaker 1: that's a lot of ifs. You'll get a credit for it,
Speaker 1: and just think you two will have have contributed to
Speaker 1: our forward thinking quest of people throwing food off of
Speaker 1: high elevations. I don't know at any rate. Get in
Speaker 1: touch with us and we will talk to you again
Speaker 1: really soon. For more on this topic in the future
Speaker 1: of technology, I'll visit forward thinking dot Com, brought to
Speaker 1: you by Toyota. Let's Go Places,