Speaker 1: Get in touch with technology with tech Stuff from how
Speaker 1: stuff works dot com. Hey there, and welcome to tech Stuff.
Speaker 1: I'm your host, Jonathan Strickland. I'm an executive producer and
Speaker 1: I love all things tech. And here's a fun fact
Speaker 1: about getting older. As you age, stuff that you once
Speaker 1: thought was impossible will become not only possible, but will
Speaker 1: become the norm, and future generations won't even think about
Speaker 1: what it must have been like before the impossible was commonplace.
Speaker 1: Now this is true for every generation. It's not like
Speaker 1: this is, you know, a brand new, groundbreaking observation. Plenty
Speaker 1: of people have made it before me, but I want
Speaker 1: to talk about a specific implementation. For example, with photography,
Speaker 1: it used to be pretty difficult to manipulate pictures convincingly.
Speaker 1: There have been photo and tools for decades, but generally
Speaker 1: it took a great deal of skill and training, plus
Speaker 1: access to specialized equipment to pull it off, especially in
Speaker 1: the old film days. Now, gradually, tools like Photoshop made
Speaker 1: it easier to manipulate digital images. Now it still requires
Speaker 1: a certain level of skill to pull off a really
Speaker 1: convincing job, and it's easy to make really badly manipulated images.
Speaker 1: But as these tools became widely available, people began to
Speaker 1: learn how to use them. We had to come to
Speaker 1: the realization that we cannot necessarily believe our own eyes
Speaker 1: when we're looking at a digital image. Now, the same
Speaker 1: thing is happening with video footage. It's quite possible to
Speaker 1: fake video footage, though again, if you want to do
Speaker 1: it really well, it requires some skill, some specialized tools,
Speaker 1: and to really get some expertise in it in order
Speaker 1: to do it in a way that's really convincing. But
Speaker 1: it's a pretty recent techn logical capability in fiction, however,
Speaker 1: it's been around for a long time. I remember seeing
Speaker 1: the movie Rising Sun back in the early nineteen nineties. Now,
Speaker 1: in that film, Wesley Snipes plays a detective and Sean
Speaker 1: Connery in his most convincing roles, since he played an
Speaker 1: Egyptian immortal posing as a Spaniard with a Scottish accent
Speaker 1: in Highlander, would play a Japanese customs and culture expert
Speaker 1: Sean Connery. Anyway, the film is a mystery thriller, and
Speaker 1: while the two are investigating a murder, they come into
Speaker 1: possession of some video footage, and they find out that
Speaker 1: video footage has actually been manipulated it was planted for
Speaker 1: them to find so that would put them on the
Speaker 1: wrong trail. One person's face was replaced with someone else's,
Speaker 1: and in a somewhat comedic scene, the video editor who
Speaker 1: is explaining this casually swaps the heads of Snipes and
Speaker 1: Connor in real time in a video feed to show
Speaker 1: off this capability, which might have been a tad unrealistic,
Speaker 1: but today we are in a world in which video
Speaker 1: manipulation of increasingly convincing quality is achievable in real time.
Speaker 1: In fact, these days, it's possible to use sophisticated computer
Speaker 1: algorithms to allow for manipulation of captured video, almost as
Speaker 1: if the video was a computer generated cartoon reacting to
Speaker 1: real time inputs like a video game controller, only instead
Speaker 1: of it being a video game, it's a real person
Speaker 1: on video. There are, of course, lots of ways this
Speaker 1: technology could be used unethically, and one of the best
Speaker 1: known has been the focus of this whole conversation around
Speaker 1: video manipulation, and it comes from a former Reddit user
Speaker 1: who went by the handle deep fakes. Now that handle
Speaker 1: has become the shorthand for the general practice, which frequently,
Speaker 1: but not exclusively, would involve replace the face of an
Speaker 1: actor in a pornographic scene with someone else's face like
Speaker 1: that of a celebrity, which is pretty darn unethical and creepy.
Speaker 1: The name itself was a reference to the technology used
Speaker 1: in the approach, so it relies on a process called
Speaker 1: deep learning. Deep learning is a type of machine learning.
Speaker 1: It's a sub type of machine learning that utilizes artificial
Speaker 1: neural networks. And I've talked an awful lot about those
Speaker 1: kind of networks recently, so I'm not going to go
Speaker 1: over the whole thing again. Will give just a really
Speaker 1: quick rundown to say, you have nodes, artificial neurons in
Speaker 1: these networks that receive input from potentially multiple other nodes,
Speaker 1: and then on that input, your artificial neuron that you're
Speaker 1: looking at will perform some sort of weighted operation and
Speaker 1: produce a single output. That single output can move on
Speaker 1: to become one of many inputs for a different ARTIFICI
Speaker 1: shoal neuron in that network, and so on and so forth.
Speaker 1: Deep learning networks are very very large artificial neural networks,
Speaker 1: and they can accept a large amount of training data.
Speaker 1: This is a scalable approach. This means the larger the
Speaker 1: network and the more data you can feed it, the
Speaker 1: better it performs. This is different from any other machine
Speaker 1: learning models. Those tend to hit a performance plateau once
Speaker 1: you hit a certain size, which means that if you
Speaker 1: were to add more nodes to the network, you wouldn't
Speaker 1: necessarily see a comparative increase in performance. You would you
Speaker 1: would kind of flatten out over time. In fourteen, a
Speaker 1: deep learning expert named Andrew ng gave a talk at
Speaker 1: Stanford about the best use cases for deep learning, and
Speaker 1: he mentioned that it was particularly good at supervised learning tasks. Now,
Speaker 1: these are the types of computer problems in which we
Speaker 1: humans already know the answer, such as is there a
Speaker 1: cat in this photograph? Humans can pick up on that
Speaker 1: right away, assuming someone has not carefully hidden a cat
Speaker 1: in a very busy image. But for a computer, this
Speaker 1: is a much more difficult problem. Even if the picture
Speaker 1: has a cat center stage, it can be tough for
Speaker 1: a computer to figure that out using a supervised learning approach.
Speaker 1: With a deep learning network, you can train a system
Speaker 1: to recognize cats and images with a high degree of
Speaker 1: success if you have a large amount of training data
Speaker 1: to train the network to recognize cats. Now, in this
Speaker 1: other case, that I was talking about. The Reddit user
Speaker 1: called deep fakes started posting on Reddit in late twenties seventeen.
Speaker 1: The user made an open source code version of a
Speaker 1: deep learning algorithm and made it available for the purposes
Speaker 1: of video manipulation and anyone could take advantage of it. Now. Specifically,
Speaker 1: this algorithm was designed for face swapping. The algorithm would
Speaker 1: allow you to put the face of one person onto
Speaker 1: the body of another in video form, and it wasn't
Speaker 1: always convincing. In fact, it could often be easily detectable
Speaker 1: as fake if someone had not trained the model properly
Speaker 1: before creating the video. But it did open up a
Speaker 1: can of worms once the practice started getting media coverage. However,
Speaker 1: the actual technology to pull this off was already a
Speaker 1: couple of years old when deep fakes shared it. Back
Speaker 1: in there was a group of researchers from Stanford then
Speaker 1: and also the University of Erlanger Nuremberg and the Max
Speaker 1: Planck Institute for Informatics who collectively published a paper that
Speaker 1: was titled Face to Face Real Time Face Capture and
Speaker 1: Reenactment of r GB Videos and that's a face, the
Speaker 1: number two and face. The paper details the methodology the
Speaker 1: group used to create a pretty incredible effect. The algorithm
Speaker 1: could take the facial expressis from one person and transfer
Speaker 1: them in real time to a video target. It was
Speaker 1: like turning the video into a digital puppet. So you
Speaker 1: might have a video loop of a celebrity running, and
Speaker 1: preferably it's a loop that's easily repeatable without the repeat
Speaker 1: being terribly noticeable, and that's your target video. So if
Speaker 1: you just let it run, you just would see video
Speaker 1: of someone sitting down, maybe looking around a little bit,
Speaker 1: but that's it, nothing special. Then you would have a
Speaker 1: source subject sitting in view of a consumer quality webcam,
Speaker 1: no special equipment here, and that person could make different expressions,
Speaker 1: including opening and closing their mouths, and the video target
Speaker 1: would match them move for move, like a digital puppet. Moreover,
Speaker 1: the source subject didn't have to wear any special gear.
Speaker 1: They didn't have to have any special markers, none of
Speaker 1: those dots that you would see with motion capture. None
Speaker 1: of that was necessary. All the algorithm needed was a
Speaker 1: video feed from a monocular camera, so you didn't even
Speaker 1: need depth perception for this. There's a video of their
Speaker 1: work on YouTube that shows off this process and includes
Speaker 1: a loop of George W. Bush sitting for an interview.
Speaker 1: The source subject can manipulate the face that Bush makes
Speaker 1: just by making faces of his own, and the algorithm
Speaker 1: would map those movements to the target video. And it's
Speaker 1: pretty wild to see an image of moving image of
Speaker 1: George W. Bush responding in real time to all of
Speaker 1: these different facial expressions this guy is making. So how
Speaker 1: did the team do this? It's one thing to say
Speaker 1: a deep learning algorithm gave them this capability, but that's
Speaker 1: not really an explanation. The paper definitively spells this out
Speaker 1: in real technical detail. It starts off the explanation by saying, quote,
Speaker 1: in our method, we first reconstruct the shape identity of
Speaker 1: the target actor using a new global non rigid model
Speaker 1: BA fast bundling approach based on a prerecorded training sequence.
Speaker 1: As this pre process is performed globally on a set
Speaker 1: of training frames, we can resolve geometric ambiguities common to
Speaker 1: binocular reconstruction. At runtime. We tracked both the expressions of
Speaker 1: the source and target actors video by a dense analysis
Speaker 1: by synthesis approach based on a statistical facial prior end quote,
Speaker 1: and it goes on in that vein throughout the paper
Speaker 1: which means it gets pretty dense. But I think we
Speaker 1: can suss out what's going on from a high level
Speaker 1: if we just take a moment. But first, I'm going
Speaker 1: to take a moment of my own to thank my sponsor.
Speaker 1: So how did the Face to Face team build this tool? Well,
Speaker 1: for each target video, they would collect a large sample
Speaker 1: of footage and images and feed it to this deep
Speaker 1: learning algorithm. This would be necess sarry to identify all
Speaker 1: the points on the face that would move with various expressions,
Speaker 1: as well as to capture images of the inside of
Speaker 1: the target's mouth when he or she spoke. This is
Speaker 1: because the video loop they used to create the manipulated
Speaker 1: video would feature the target subject, typically with his or
Speaker 1: her mouth closed, so it might be a section in
Speaker 1: which the subject was sitting down for an interview and
Speaker 1: listening to an interviewer's questions but not responding yet they
Speaker 1: were just listening. The additional video would provide information about
Speaker 1: the inside of the target subject's mouth, which could be
Speaker 1: rendered in time on the target video performance when it
Speaker 1: came time to do that. Their approach improved the scanning
Speaker 1: technique to build face templates for both the source subject
Speaker 1: who provides all the expressions and the target subject, who
Speaker 1: mimics all the expressions. As the source subject makes different
Speaker 1: facial expressions, the computer face template detects how the subjects
Speaker 1: face changes or deforms over time. The computer model then
Speaker 1: takes that information, saying, all right, well the lips moved
Speaker 1: in this way, there was a grimace here or a
Speaker 1: smile there, and transfer those motions to the targets face template,
Speaker 1: which is matched to the target's actual face. This process
Speaker 1: transfers the expressions over to the target, and so when
Speaker 1: the source subject grimaces, the target grimaces, if the source
Speaker 1: subject just it's still the target. Video will continue to
Speaker 1: loop and the targets face won't change. The more video
Speaker 1: footage you can get of your target and your source,
Speaker 1: the better the computer algorithms are that create those face templates,
Speaker 1: and the more natural the manipulation will appear on the
Speaker 1: finished video. You also want a really good amount of
Speaker 1: footage just to get all that extra information you need,
Speaker 1: like the inside of the mouth, so that that can
Speaker 1: all be extrapolated properly. You have to design a tool
Speaker 1: that can encode an image called the training image, and
Speaker 1: then decodes this data to reconstruct the image. So imagine
Speaker 1: you've got a picture. The encoder essentially creates data based
Speaker 1: on that image. It's like a description of that image.
Speaker 1: The decoder takes the description and tries to rebuild the
Speaker 1: image based on the description. I think of this like
Speaker 1: that scene in Willy Wonka where Mike TV gets broken
Speaker 1: up in a million little pieces and then gets reconstructed
Speaker 1: on the television screen. So the second image is not
Speaker 1: a copy. It's not like you made a copy of
Speaker 1: the first one. It's like you built a new image
Speaker 1: based on the first one. By the way, when you
Speaker 1: start off with these uh these algorithms, those reconstructions tend
Speaker 1: to look pretty bad. You have to continually train and
Speaker 1: train and train and train the model so that it
Speaker 1: gets better and better at producing a close representation of
Speaker 1: the original image. When it does, it's reconstru struction. And
Speaker 1: you would have both essentially decoders for both your source
Speaker 1: subject and your target subject. Use the same encoder for both,
Speaker 1: but two different decoders, one dedicated to your source, one
Speaker 1: dedicated to your target. Then you would feed the reconstructed
Speaker 1: images through the system again and again. This is called
Speaker 1: back propagation. You do this over millions of times, typically
Speaker 1: to improve this process, and then you're ready to really
Speaker 1: switch switch things up. So let's say we've got two people.
Speaker 1: We've got person one and we've got person two, and
Speaker 1: you've been feeding images of both of these people through
Speaker 1: the same encoder, but of course you have dedicated decoders
Speaker 1: to produce the reconstruction. So person one has decode er
Speaker 1: one and person two has decoder two. Now let's say
Speaker 1: you're ready to put person two's face on person one's body. Well,
Speaker 1: you would feed an image of person one into the encoder,
Speaker 1: but you use the decoder for a person to to
Speaker 1: reconstruct the image, and what you get is persons who's
Speaker 1: face but mimicking the expression from person one. You, or
Speaker 1: rather the computer algorithm, does this frame by frame on video,
Speaker 1: and you end up with a video appearing to feature
Speaker 1: one person when in fact it's just their face on
Speaker 1: top of someone else, and it's their face making the
Speaker 1: exact same expressions as whoever was originally in that video.
Speaker 1: Now back over to deep fakes. Before long after the
Speaker 1: Reddit user initially posted this code, folks over at Reddit,
Speaker 1: we're taking this open source code and making more advanced
Speaker 1: software based off of it. Soon there were desktop apps
Speaker 1: that would take over all the hard parts of this process,
Speaker 1: all the codey bits, if you will, of training a model.
Speaker 1: Some of them would guide users into creating the data
Speaker 1: that would be used to train the model and go
Speaker 1: all the way through the process of creating the final
Speaker 1: fake videos. Even with some of the more sophisticated versions,
Speaker 1: there were tell tales signs of tampering. Typically some blurring
Speaker 1: around images, particularly near chins and mouths. Those would be signs.
Speaker 1: If there was any flicker, that was a sign if
Speaker 1: you didn't take enough time to train the model. Typically
Speaker 1: you would want to do several days of training at least.
Speaker 1: If you didn't take that time, you might see some
Speaker 1: really nasty blurring and flickering, and it would be a
Speaker 1: dead giveaway that this was tampered. Video in writer, director,
Speaker 1: and comedian Jordan's Peel demonstrated the power of this technology.
Speaker 1: He showed how, with his impersonation of Barack Obama and
Speaker 1: some manipulation software, he could create a fake public service
Speaker 1: address when which the president would appear to say things
Speaker 1: that he normally would never say. The technology behind this
Speaker 1: made use of what is called a long short term
Speaker 1: memory network or l s TM, to go into the
Speaker 1: mechanics of that would require another podcast, but using an
Speaker 1: approach similar to what I've already described, a team was
Speaker 1: able to make a video of Obama apparently lip syncing
Speaker 1: Peel's satirical message. The goal of this p s A
Speaker 1: was beyond alert because fakes are getting harder to spot.
Speaker 1: The University of Washington showed off this and They're Synthesizing
Speaker 1: Obama project in which they took the audio from one
Speaker 1: of President Obama's speeches and then used it to animate
Speaker 1: his face in video from a different address that he
Speaker 1: gave during his presidency. So in this example, the person
Speaker 1: in the target video is the same person as the
Speaker 1: source for the audio. But the point was pretty clear
Speaker 1: that tech would soon make it possible to fake someone
Speaker 1: saying or doing something. It just takes the right algorithms,
Speaker 1: the right amount of training data, and the right amount
Speaker 1: of time to get the model trained up enough to
Speaker 1: do it smoothly. Now, this technology could be used to
Speaker 1: do stuff that isn't related to malicious deception or for
Speaker 1: pornography or anything along those lines. It could be used
Speaker 1: in television and film for lots of stuff, including potentially
Speaker 1: adding in actors who have passed away into a film.
Speaker 1: Paired with similar work that's going on in voice synthesis,
Speaker 1: you could end up with a convincing replacement, which means
Speaker 1: we could make movies with dead actors taking on new
Speaker 1: parts because we can synthesize their speech, we can synthesize
Speaker 1: their appearance. You would still have someone else acting out
Speaker 1: the part physically, but you would replace their image with
Speaker 1: this actor's image. Or maybe you would want to use
Speaker 1: this kind of technology just to make everyone think you
Speaker 1: can cut a rug. This brings me to the University
Speaker 1: of California, Berkeley and is the subject of a paper
Speaker 1: titled Everybody Dance Now. The goal is a simple concept
Speaker 1: that's actually really hard to pull off. What if you
Speaker 1: were to take the movements of a professional dancer and
Speaker 1: then map those movements onto the body of someone who
Speaker 1: wasn't a dancer. What if you could create a video
Speaker 1: in which literally anyone would appear to move like a skilled,
Speaker 1: trained dancer. And how the heck would that be possible. Well,
Speaker 1: at the heart of the team's efforts was something I
Speaker 1: talked about in a recent episode of tech Stuff about
Speaker 1: an AI generated portrait, and that would be generative adversarial
Speaker 1: networks or g A n s. These use a pair
Speaker 1: of artificial neural networks in competition against each other. So
Speaker 1: since I covered this recently, i'll just give again a
Speaker 1: super quick high level summary. You've got one network that
Speaker 1: has a specific job, such as trying to create an
Speaker 1: original image of a cat. We'll go back to the
Speaker 1: cat pictures. That's one of my favorite ones because it
Speaker 1: was one of the early use cases of neural networks
Speaker 1: that I remember encountering when I was doing research. Now,
Speaker 1: let's say you've got your second network. Your second network
Speaker 1: has the specific job of evaluating pictures of cats to
Speaker 1: determine if they are valid, meaning is this a real
Speaker 1: picture of a cat that's part of the training material
Speaker 1: that I'm accepting, or is this, in fact a fake
Speaker 1: that was created by a computer program the other neural network.
Speaker 1: So you've got one network trying to fool the other network.
Speaker 1: And these networks get better at what they do over time,
Speaker 1: they improve, So your counterfeit network is getting better and
Speaker 1: better at making fake pictures of cats, and your detector
Speaker 1: network is getting better and better at detecting fake images
Speaker 1: of cats. Now, typically this requires humans to give feedback
Speaker 1: or tweaking weight values along the networks, but they do
Speaker 1: get better over time. So if the network trying to
Speaker 1: create a picture of a cat gets the feedback of sorry, buddy,
Speaker 1: but they're onto you, then it can try again and
Speaker 1: adjust it's approach slightly in an effort to fool the
Speaker 1: second network. If the second network gets the feedback you'll
Speaker 1: let this one slip by and it's fake, then it
Speaker 1: will adjust or it will be adjusted to look out
Speaker 1: for any tailtale signs that it had missed in that
Speaker 1: earlier evaluation. Over time, the two networks working against each
Speaker 1: other will create the ultimate result of better and better
Speaker 1: computer generated content, whether it's an image of a cat
Speaker 1: or a sonnet, or a song or a video. Now
Speaker 1: that doesn't mean that these computer generated things are at
Speaker 1: the same level as human generated stuff, especially when it
Speaker 1: comes to text. I've seen a lot of song lyrics
Speaker 1: that were inscrutable even by my old man standards. So
Speaker 1: I think that we're a long way away from getting
Speaker 1: to a point where they can fool us in every case.
Speaker 1: But with video they're getting pretty darn good. Now, this
Speaker 1: team had two groups of subjects, and so you had
Speaker 1: your source subjects and your target subjects. The source in
Speaker 1: this case, were the people who could dance, so like
Speaker 1: ballet dancers, hip hop dancers and that sort of stuff.
Speaker 1: They legit know how to move. They would demonstrate various
Speaker 1: dances on video. The second group of subjects were your
Speaker 1: target subjects. These were not trained dancers. They were to
Speaker 1: go through a series of moves and poses, essentially aping
Speaker 1: as best they could the movements of trained dancers, and
Speaker 1: the goal of this pair of networks was to smooth
Speaker 1: the movements out and adjust the timing so that these
Speaker 1: untrained dancers would appear to move more like their groovy
Speaker 1: source subject counterparts. I'll explain more in just a moment,
Speaker 1: but first let's take another quick break to thank our sponsor.
Speaker 1: According to the Everybody Dance Now paper, the team would
Speaker 1: transfer motion between the sources to the target through an
Speaker 1: end to end pixel based pipeline. So here's how that's done.
Speaker 1: Because if you're like me, that phrase meant next to
Speaker 1: nothing to you. So specifically, the group used three stages
Speaker 1: to take the movements of one person and transpose them
Speaker 1: to a target person. Those three were pose detection, global
Speaker 1: pose normalization, and mapping from normalized pose stick figures to
Speaker 1: the target subject. Post detection involves teaching machines, in other words,
Speaker 1: computers how to interpret images to determine where key body
Speaker 1: points are, like elbows, knees, hips, shoulders, the head, that
Speaker 1: kind of stuff. That first requires that you teach the
Speaker 1: machine to recognize those points in the first place. So
Speaker 1: first you have to train a machine to recogniz eyes
Speaker 1: those points and identify them with a target level of accuracy.
Speaker 1: It's pretty typical to represent these joints as as points
Speaker 1: in a stick figure, so each point represents another joint
Speaker 1: or point of articulation. The lines represent the trunk of
Speaker 1: the body, the limbs, the head. You end up with
Speaker 1: a stick figure. If your machine learning mechanism was a
Speaker 1: good one, the machine should be able to overlay a
Speaker 1: stick figure on top of any image of a person posing,
Speaker 1: and the stick figure should more or less conform to
Speaker 1: that image, including where the actual joints are. So if
Speaker 1: you have someone standing there in the classic Peter Pan
Speaker 1: pose of their their fists on their hips uh and
Speaker 1: their their arms out of kimbo, then it should draw
Speaker 1: a stick figure that's essentially aping the same thing and
Speaker 1: be able to overlay it on top of the original image.
Speaker 1: Now these days this can be done in real time. So,
Speaker 1: for example, there's a team at Google Creative Lab that
Speaker 1: used a machine learning model of pose net and created
Speaker 1: a JavaScript version with TensorFlow, which is an open source
Speaker 1: software library often used for machine learning. And with this
Speaker 1: tool you can do real time pose estimation through a
Speaker 1: browser and a webcam. The application doesn't have any technology
Speaker 1: related to identifying the person in the image. It's just
Speaker 1: quote unquote interested in what the person is doing, not
Speaker 1: who the person is. So you can actually run this
Speaker 1: on your own machine in a browser, and you can
Speaker 1: pose in front of a webcam and you'll see the
Speaker 1: little stick figure uh painted on top of your image
Speaker 1: on the computer. Essentially, so every time you move, every
Speaker 1: time you bend a joint, you will see the stick
Speaker 1: figure doing the same thing, um mapped on top of you.
Speaker 1: The Berkeley team made use of a pre trained pose detector,
Speaker 1: meaning they didn't build a new one, which helps save
Speaker 1: a lot of time and expense on their project. Now
Speaker 1: people come in all shapes and sizes. In the video
Speaker 1: the team released, they showed off subjects who included a
Speaker 1: woman who appeared to be of around average height and
Speaker 1: a man who appeared to be pretty darn Tallman transfer
Speaker 1: method that would only work between a subject and a
Speaker 1: target who are of similar shape and size would be
Speaker 1: pretty limited. So the purpose of the global pose normalization
Speaker 1: stage is to account for all the differences between the
Speaker 1: source and the target subjects and the locations within the
Speaker 1: frame of the camera. Without this step, the motion transfer
Speaker 1: might appear ghoulish. We don't have all the same proportions, right,
Speaker 1: so a mismatch might mean a target's limbs would appear
Speaker 1: to bend in places that were clearly not natural joints.
Speaker 1: All you need to do is see an arm bend
Speaker 1: where an arm isn't supposed to bend, and that's going
Speaker 1: to ski the out quite a bit. Makes an effective
Speaker 1: horror movie experience, but not one that would produce convincing
Speaker 1: motion transfer. Now, there are a lot of ways that
Speaker 1: the team could have gone about normalizing the poses, but
Speaker 1: their choice seems particularly clever to me. They measured the
Speaker 1: heights and ankle positions of the various subjects and used
Speaker 1: linear mapping between the closest and farthest ankle positions in
Speaker 1: both videos to normalize the stick figure for the target subjects.
Speaker 1: The program would calculate the scale of the figure as
Speaker 1: well as the scale of motion from frame to frame.
Speaker 1: And I think that's pretty darn cool because it wasn't
Speaker 1: just accounting for the size of the subjects to get
Speaker 1: all the joints right, but also to make sure the
Speaker 1: scale of the movements with respect to the body size
Speaker 1: and proportions would remain the same. So a tall person
Speaker 1: with really long limbs moving their arms in really big, big,
Speaker 1: bold gestures, if you tried to transfer that motion to
Speaker 1: someone who was of smaller stature, it could really look disturbing.
Speaker 1: But by using this scaling approach, the movements on the
Speaker 1: smaller person would be proportionate in size to the movements
Speaker 1: of the larger person. The team would use two of
Speaker 1: the Generative Adversarial Network setups to work on making a
Speaker 1: convincing final video. The first was dedicated to image to
Speaker 1: image translation, attempting to manipulate the image of the target
Speaker 1: subjects that would follow the motions made from the pose
Speaker 1: detection process, and like all g a N setups, this
Speaker 1: included the generator, which would attempt to create a convincing
Speaker 1: sequence of images, and the discriminators, which tried to weed
Speaker 1: out the quote unquote fake sequences from the generator from
Speaker 1: the ground truth data that was being fed to it.
Speaker 1: The second g N set set up was specifically dedicated
Speaker 1: to add detail and realism to the faces of the
Speaker 1: target subjects. In some frames this appears to have worked
Speaker 1: pretty well, and others there's a bit of an uncanny
Speaker 1: valley thing or maybe even horror movie type element going on,
Speaker 1: similar to how some of the AI generated portraits that
Speaker 1: I talked about in the previous episode introduced a bit
Speaker 1: of unrealistic qualities to the various images. When shooting video
Speaker 1: of the target subjects, the team captured images at one
Speaker 1: hundred twenty frames per second to get enough data for
Speaker 1: each subject. The sessions lasted for about twenty minutes. They
Speaker 1: used smartphone cameras to do it, since many smartphones allow
Speaker 1: you to shoot video at this kind of frame rate
Speaker 1: these days. They had their target subjects where close fitting
Speaker 1: clothing that wasn't prone to wrinkling because the post recognition
Speaker 1: tool they were using wasn't designed to encode information about clothing.
Speaker 1: As for the source videos, the ones that would actually
Speaker 1: create the motions that would be transferred to the targets,
Speaker 1: the team didn't have to worry about capturing images at
Speaker 1: such a high frame rate. They could use videos of
Speaker 1: just reasonable quality, meaning decent resolution and frame rate, and
Speaker 1: their post detection tool would do its work and create
Speaker 1: the stick figure that would serve as the guide for
Speaker 1: the target motions later on. Because of that, the team
Speaker 1: can really use any online video of sufficient quality to
Speaker 1: act as the source information for motion transfer. It doesn't
Speaker 1: have to be a video shot specifically for that purpose.
Speaker 1: In fact, one of the example videos the team used
Speaker 1: in their demonstration was from a Bruno Mars music video
Speaker 1: for That's what I Like. Before applying the motion transfer,
Speaker 1: the team smoothed pose key points to reduce jitter in
Speaker 1: the final output, and then the team applied the motion transfer.
Speaker 1: The stick figure motions were then transferred to the target
Speaker 1: subjects and the result is pretty interesting. It is not seamless.
Speaker 1: You can definitely tell something odd is going on, but
Speaker 1: it is an indication of where things are going and
Speaker 1: using adversarial networks could lead to more convincing motion transfers
Speaker 1: in the future. Now, this could lead to all sorts
Speaker 1: of stuff nefarious and otherwise. You could imagine using it
Speaker 1: to transform an average actor into a martial arts master,
Speaker 1: or it might allow directors more freedom of casting, knowing
Speaker 1: that if the actors they choose don't possess certain physical skills,
Speaker 1: they can use this kind of technology to fake it,
Speaker 1: but would also be used to fake footage to make
Speaker 1: it looks like people like specific people are doing stuff
Speaker 1: that they are not doing. It could be used to
Speaker 1: spread misinformation and it likely will be, which means we'll
Speaker 1: need to be on the lookout for signs of fakes,
Speaker 1: which are going to get harder and harder to detect
Speaker 1: as time goes on. And hey, you guys remember DARPA,
Speaker 1: right because I just did a whole series of episodes
Speaker 1: about them. Well, that agency has funded programs dedicated to
Speaker 1: automating various forensic tools, including tools that could be used
Speaker 1: to detect AI created forgeries in video and audio. Often
Speaker 1: the secret is in the eyes. Most of these neural
Speaker 1: networks are trained on still images, so you send thousands
Speaker 1: or tens of thousands of images if you have them,
Speaker 1: of your various subjects, your target, and your source. But
Speaker 1: most published still images don't show people with their eyes closed.
Speaker 1: So I've moved my movements and blinking tends to be
Speaker 1: a little wonky in these fake videos. You might watch
Speaker 1: one for a while and think, huh, that's weird. This
Speaker 1: guy hasn't blinked for like ten minutes, or when they
Speaker 1: blink it looks really strange. Well, that's an indication that
Speaker 1: it's a fake video. There are other ones as well,
Speaker 1: but DARPA is understandably keeping those quiet because not you know,
Speaker 1: if if they publish how they figure out AI created
Speaker 1: videos are in fact faked, then that gives the fakers
Speaker 1: enough information to go back and improve their models. So
Speaker 1: we're likely to see something akin to what happened with capture.
Speaker 1: Specialists will develop new tools to detect a I generated media.
Speaker 1: AI developers will then create more sophisticated models, and so
Speaker 1: it becomes kind of an arms race a seesaw, and
Speaker 1: one benefit is that AI as a whole will improve,
Speaker 1: but we may not be able to believe it when
Speaker 1: we see it. Well, that wraps up this episode of fascinating,
Speaker 1: somewhat disturbing topic, and uh, I'm sure we're gonna hear
Speaker 1: a lot more about this in the years to come.
Speaker 1: We've seen a lot of of sites banning deep fakes
Speaker 1: outright because of the misinformation that they can spread. So
Speaker 1: we're already seeing a reaction to this in various online communities,
Speaker 1: so that's very interesting to me. But we're definitely gonna
Speaker 1: keep seeing this continue. It's a it's a valid area
Speaker 1: of AI research, so we will have to wait and
Speaker 1: see how it all plays out. If you guys have
Speaker 1: any suggestions for future episodes of tech Stuff, why not
Speaker 1: send me a message. You can go over to our
Speaker 1: website that is Text Stuff podcast dot com. You'll find
Speaker 1: all the different ways to contact me. I look forward
Speaker 1: to hearing from you. Make sure you check out our
Speaker 1: store over at t public dot com slash tech Stuff
Speaker 1: by some merchandise. You can make sure that you get
Speaker 1: all the really cool T shirts like prove to Me
Speaker 1: You're not a Robot. That one's pretty appropriate for this
Speaker 1: particular episode. And remember every single purchase you make goes
Speaker 1: to help the show, so we greatly appreciate it. Also,
Speaker 1: if you haven't heard, we have been nominated for an
Speaker 1: I Heart Radio Podcast Award. It's the first year I
Speaker 1: Heart Radio is giving out podcast awards. We are nominated
Speaker 1: in the Science and Technology category. You can go online
Speaker 1: and visit the I Heart Radio Podcast Awards page and
Speaker 1: vote up to five times a day for your favorite podcasts.
Speaker 1: If you wanted to. You could dedicate all five of
Speaker 1: those votes every single day to us. I would not
Speaker 1: complain if you did that. It would be really cool
Speaker 1: to win that award, but make sure you check it out.
Speaker 1: There may be lots of shows there that you truly
Speaker 1: love and you want to throw your support behind them.
Speaker 1: That would be really cool with you, And I'll talk
Speaker 1: to you again really soon for more on this and
Speaker 1: bousands of other topics because it has to have four.
Speaker 1: Stock com