Speaker 1: December of twenty twenty, during the pandemic cold winter morning,
Speaker 1: the security guard on a train platform and the outskirts
Speaker 1: of Saint Louis was working his job and suddenly saw
Speaker 1: a man in a security booth who wasn't supposed to
Speaker 1: be there. So he approached this man and told him
Speaker 1: to leave, and then the verbal altercation ensued and another
Speaker 1: man walked up behind him, so he was surrounded by
Speaker 1: these two guys who were getting more and more agitated
Speaker 1: with him. The next thing he knew, he was assaulted
Speaker 1: by these two guys, struck in the head, and when
Speaker 1: he was on the ground, they continued to beat him
Speaker 1: and hit him in the head and hit ananimous chest
Speaker 1: and his ribs. The guys ran off and Michael and
Speaker 1: the victim suffered concussion that day and he couldn't remember anything.
Speaker 1: So what the police were really left with to solve
Speaker 1: this case is the same thing that they're left with
Speaker 1: in a lot of cases where there's no witnesses.
Speaker 2: A random crime, a victim with no memory, and nobody
Speaker 2: seems to have seen anything.
Speaker 1: A video camera was the only witness to this crime.
Speaker 1: This kind of fuzzy, blurry surveillance video captured a still
Speaker 1: of these perpetrators, and that was basically the only thing
Speaker 1: that they had to go on.
Speaker 2: Douglas McMillan is a reporter for the Washington Post, So Doug,
Speaker 2: first off, for real, thanks for being down to talk
Speaker 2: with me about all this. Of course, Doug doesn't usually
Speaker 2: cover cops or the justice system. His usual beat at
Speaker 2: the Post is corporate accountability and technology. But there was
Speaker 2: something about this case in Saint Louis that caught us attention.
Speaker 1: There's a lot of crimes, hundreds or thousands of solved
Speaker 1: cases out there where the only evidence was a camera,
Speaker 1: The only evidence was a photo of this in perpetrator.
Speaker 1: And the potential of this technology is we have this
Speaker 1: seemingly magical tool to help solve those crimes.
Speaker 2: The technology he's talking about here is AI facial recognition.
Speaker 2: Police around the country are increasingly using this on cases
Speaker 2: that they can't find a witness for and this was
Speaker 2: one of those cases. The investigation into who had assaulted
Speaker 2: Michael Feldman had pretty much gone cold.
Speaker 1: Until about eight months after the incident. We believe the
Speaker 1: police officers they decided, oh hey, we have these grainy
Speaker 1: surveillance images, let's try to run them through the facial
Speaker 1: recognition system and see if that will give us any matches.
Speaker 1: And so they did that and they got some matches back.
Speaker 1: We believe it was between five and ten maybe of
Speaker 1: different possible matches. And the one they chose, the one
Speaker 1: that they thought looked the most like the perpetrator of
Speaker 1: the crime, or one of the two perpetrators of the crime,
Speaker 1: was this gentleman named Christopher Gatlin. And then they kind
Speaker 1: of were off and running. In this case, they had
Speaker 1: a suspect. Then this was the only thing that police
Speaker 1: had to arrest this man, take away his freedom, you know,
Speaker 1: came down to this AI program.
Speaker 2: I mean, because the thing here that of course we're
Speaker 2: sort of circling around, is that that's not the guy,
Speaker 2: Christopher Gatlin's innocence.
Speaker 3: I'm afraid.
Speaker 2: From Kaleidoscope in iHeart Podcasts. This is kill Switch. I'm
Speaker 2: Dexter Thomas.
Speaker 3: I'm sorry, I'm sorry. Good bye.
Speaker 1: So they went out investigating Christopher Gatlin and the first
Speaker 1: thing they did was print out a picture of his mugshot,
Speaker 1: and they took the picture of Chris Gatlin to the
Speaker 1: apartment building where Michael Feldman lives and they conducted a
Speaker 1: photo lineup with him, and they basically sat Michael Feldman
Speaker 1: down and gave him the photos, put them on the
Speaker 1: table in front of him, and instructed him to kind
Speaker 1: of spend time looking at each one and going through
Speaker 1: these photos, take.
Speaker 4: A look through all of them and see if anything.
Speaker 5: Rings a bell.
Speaker 2: Eight months after the assault, the cops visited the victim,
Speaker 2: Michael Feldman's house, and they brought some pictures with them.
Speaker 2: One of those pictures was of Chris Gatlin. And we
Speaker 2: know exactly what happened next because it's all on record.
Speaker 1: So we got the body camera a video of this
Speaker 1: photo lineup, and the officers who conducted this photo lineup
Speaker 1: did not follow the proper procedures for basically getting an
Speaker 1: affair and impartial identification of this witness. And you can
Speaker 1: see throughout the process when Michael Feldman's going through these photos,
Speaker 1: these police officers, who are really supposed to be just
Speaker 1: standing off to the side and not saying anything, are
Speaker 1: kind of coaching him and giving him kind of little
Speaker 1: clues throughout the process on which guy they want him
Speaker 1: to pick.
Speaker 4: Okay, let's think about before the incident, he went up
Speaker 4: and he talked to him a little bit right, Let's
Speaker 4: start to focus on things before you actually got any
Speaker 4: altercation of it, maybe even before you actually initially went.
Speaker 3: To talk to him. Just take a minute kind.
Speaker 4: Of ponder, think.
Speaker 1: And at one point he actually picks up the picture
Speaker 1: of Chris Gatlan and puts it aside, and he keeps
Speaker 1: on looking through these photos, and at one point he says,
Speaker 1: I want to say it's him pointing to a different guy.
Speaker 3: I want to say, jan I remember my memory is
Speaker 3: I want to say to him.
Speaker 1: When he does that, one of the detectives steps forward
Speaker 1: and says, okay. Instead of, you know, accepting that as
Speaker 1: an identification, the detective encourages him to continue going and
Speaker 1: continue thinking about the interaction. Think about, you know, was
Speaker 1: a guy wearing anything, Was he wearing a hat? Did
Speaker 1: you recognize anything about his face, his eyes, their clothing
Speaker 1: he had.
Speaker 3: I don't remember really.
Speaker 1: I thought he had like a hat on or something, or.
Speaker 4: Stocking something on. So let's picture the picture of these
Speaker 4: two guys wearing that, you know, a stocking cap or something.
Speaker 4: If you need to use your hands, if you've got
Speaker 4: to put hands on the papers.
Speaker 3: That's okay.
Speaker 1: And so you know, he kind of like, you know,
Speaker 1: prods Michael into thinking that he didn't pick the right
Speaker 1: person and then going back and kind of changing his mind,
Speaker 1: And eventually he does go back and he pulls the
Speaker 1: picture of Chris Gatland out and he points to that
Speaker 1: and he says it was him, kid.
Speaker 3: All right.
Speaker 4: I just remember he getting really angry quick.
Speaker 2: Just the mannerism I'm just trying to and the eyes probably.
Speaker 4: How pissed off he got.
Speaker 3: As not to be in that.
Speaker 1: You couldn't stand there.
Speaker 3: I think.
Speaker 5: Okay.
Speaker 2: So police take that identification and they're able to get
Speaker 2: a warrant to arrest Chris Gatlin. They have no other
Speaker 2: evidence other than that an algorithm picked him out. He
Speaker 2: spent seventeen months in jail. So, Doug, I've read about
Speaker 2: Chris Gatland's case, but you've met him in person. What's
Speaker 2: he like.
Speaker 1: He's a very sweet, charming guy, father of four kids.
Speaker 1: He did not have a criminal record, so the reason
Speaker 1: that he was in this system. So the fish recognition
Speaker 1: database in Saint Louis has over two hundred and fifty
Speaker 1: thousand mug shots of people, including people who were pulled
Speaker 1: over for traffic stops and arrested for speeding or arrested
Speaker 1: for very minor things. Saint Louis has a long history
Speaker 1: of over policing, especially black minority, low income populations, handing
Speaker 1: out tickets to people just so that police can meet
Speaker 1: their quotas. So Chris, he had had a few different
Speaker 1: traffic infractions, and he had one burglary charge from a
Speaker 1: few years earlier. It was ultimately dropped right prosecutors. But
Speaker 1: that's the reason that he came out, and that's the
Speaker 1: reason he was in this system.
Speaker 2: It's incredible that you could be in a system that
Speaker 2: would lead you to get arrested based off of a
Speaker 2: traffic ticket. I have a completely clean record, with the
Speaker 2: one exception that a literal decade ago, I was speeding
Speaker 2: to work on the freeway. I was late for work.
Speaker 2: I got pulled over. Now, I didn't get arrested, but
Speaker 2: had somebody decide to go a little harder on me? Sure,
Speaker 2: I suppose they could have. And I'm just imagining myself
Speaker 2: being at home one day and police coming to my
Speaker 2: door and saying, Hey, we're arresting you because we think
Speaker 2: you did something that I've never heard of in a
Speaker 2: place that I don't.
Speaker 1: Go, because you were in this database.
Speaker 2: Yeah, because I was five miles over the speed limit
Speaker 2: one day going for work. Yeah, so maybe this is
Speaker 2: a good time to say that all facial recognition technology
Speaker 2: isn't the same, or maybe more importantly, that it all
Speaker 2: doesn't come from the same place. The program that the
Speaker 2: Saint Louis Cops use searches the internal databases of the
Speaker 2: police department. But there's another very popular tool. It's able
Speaker 2: to pull data straight from the Internet. It's called clearview AI.
Speaker 1: Clearview scrapes together images from Facebook, from LinkedIn, from Venmo accounts,
Speaker 1: from news articles, from all of the public web sources
Speaker 1: that it can find images with people's face.
Speaker 2: Clearview is wild, and they've got billions of photos in there.
Speaker 1: Yeah, they say they have billions of images. And there's
Speaker 1: a question about whether it's legal for them to scrape
Speaker 1: these images and put them in a database, because they
Speaker 1: do not have they never really got permission from anybody
Speaker 1: to get these images.
Speaker 2: Clearview AI's website boats that they've scraped over fifty billion images.
Speaker 2: And again this is just from the general Internet. So
Speaker 2: when the police run a search, you could be in there,
Speaker 2: even if you've never had a run in with law enforcement.
Speaker 2: Do we even know how many police departments are using this.
Speaker 1: No, it's very hard to tell because they're not required
Speaker 1: to disclose this anyway. In most places, they're not required
Speaker 1: to disclose that they're using these tools. So the best
Speaker 1: proxy for that number is what the companies who make
Speaker 1: this software have announced. Clearview AI has said that it
Speaker 1: has thousands. I think they have said over three thousand
Speaker 1: police customers. I did a pretty exhaustive public record search
Speaker 1: for police departments that are using this tools. We only
Speaker 1: came up with a list of about seventy that we
Speaker 1: could verify that are using it in some way, and
Speaker 1: of those, we sought public records on individual cases, and
Speaker 1: we could only get individual case on something like thirty
Speaker 1: or forty of them. Getting to the bottom of you know,
Speaker 1: which police departments are using this, how they're using it
Speaker 1: is very difficult because for the most part, there's no
Speaker 1: requirement for them to make public how and when they're
Speaker 1: using the tools, and that is allowing police to mostly
Speaker 1: use these tools in secrets.
Speaker 2: But sometimes it's not the machines making the mistakes, it's
Speaker 2: the cops. More on that after the break, it's not
Speaker 2: too hard to understand why a police department would be
Speaker 2: interested in facial recognition tools. It seems pretty logical. I mean,
Speaker 2: you're having a hard time crack in a case and
Speaker 2: you either don't have any eyewitnesses, or you can't rely
Speaker 2: on the witnesses you have, or maybe, like with Michael Feldman,
Speaker 2: the victims suffered an injury that affected their memory of
Speaker 2: the event, or maybe too much time has passed for
Speaker 2: somebody to really remember the details. Eyewitness accounts are full
Speaker 2: of problems for law enforcement because human memory is unreliable
Speaker 2: and it can be influenced by people's biases. So there's
Speaker 2: this potential magical solution. You in put a face into
Speaker 2: the system, It searches a database and spits out a match.
Speaker 2: All you gotta do is go find that person and boom.
Speaker 2: Case closed.
Speaker 1: So we know that there are at least eight Americans
Speaker 1: who have been wrongly arrested due to police misuse of
Speaker 1: facial recognition.
Speaker 2: And just to be clear here, when he says wrongly arrested,
Speaker 2: that's not just his opinion. What he means is that
Speaker 2: the police admitted afterwards that they got the wrong person.
Speaker 1: Interestingly, one of those cases this man named Jason Vernaw
Speaker 1: who is based in Miami. In that case, the computer
Speaker 1: got the right person, but the police had fed the
Speaker 1: computer an image of the wrong person. So they fed
Speaker 1: it an image who they thought this was the perpetrator
Speaker 1: of somebody in a bank committing fraud. But it turns
Speaker 1: out they were mistaken and the surveillance image that they
Speaker 1: put in the system was just the wrong person to
Speaker 1: begin with.
Speaker 2: So that was the one time they got it right.
Speaker 1: Well, yeah, so the computer actually picked the right person,
Speaker 1: but the police were relying on this image they got
Speaker 1: from the bank and they just pulled an image from
Speaker 1: the wrong time.
Speaker 2: So the bank pulled an image from their security cameras
Speaker 2: and sent it to the police, and this image was
Speaker 2: Jason Vernow, but it was from a completely different time
Speaker 2: of the day. The facial recognition software had correctly identified Jason,
Speaker 2: but the police failed to make some obvious checks.
Speaker 1: What this case shows was that the police put the
Speaker 1: image in their system and it popped up Jason Vernow
Speaker 1: and they went out and arrested the guy, rather than
Speaker 1: taking a second and saying, okay, is we have a suspect.
Speaker 1: Let's see if, like, can we prove that Jason ver
Speaker 1: Now was tied to this crime. And in the case
Speaker 1: of a financial there is a lot of potential evidence there.
Speaker 1: There's a check that he supposedly cashed, a faulty check. Well,
Speaker 1: this check of his signature matches. There are witnesses, there's
Speaker 1: a bank teller who took let's see if the bank
Speaker 1: teller can confirm that was him. They didn't do any
Speaker 1: of that. They apparently didn't do any real investigative steps.
Speaker 1: They just took the word of the machine and went
Speaker 1: not arrested him. It speaks to kind of this phenomenon
Speaker 1: that seems to be happening with police as they are
Speaker 1: trusting this kind of they're kind of imbuing this software
Speaker 1: with this magical ability to lead them to the right suspect.
Speaker 1: Some researchers call this automation bias, that when a computer
Speaker 1: is telling you an answer, you're more likely to believe
Speaker 1: that that is the correct answer, even at the expense
Speaker 1: of your kind of typical due diligence, your typical you know,
Speaker 1: just rational brain and just normal steps that you should
Speaker 1: think through and follow before you go out and arrest
Speaker 1: somebody take away somebody's freedom.
Speaker 2: Automation bias is just one kind of bias here, though,
Speaker 2: I mean, there was something unique about this particular person though, right,
Speaker 2: compared to the other cases. Yeah, he was white, and
Speaker 2: that's the one that they got the computer got right. Yeah.
Speaker 3: Yeah.
Speaker 2: The facial recognition has been shown to often get certain
Speaker 2: people wrong. Right.
Speaker 1: Yeah. There's two things going on here. One, facial recognition
Speaker 1: has struggled with darker complexions, and part of this is
Speaker 1: due to how these programs are trained. So when the
Speaker 1: first face rerecation algorithms were trained, they were trained on
Speaker 1: lighter skin tones. There's a federal research lab it's called
Speaker 1: NIST NIST that does testing into how well facial recognition
Speaker 1: performs in laboratory settings, and they found in twenty nineteen
Speaker 1: that certain demographic groups, including African Americans, and certain facial
Speaker 1: rection algorithms could be up to one hundred times more
Speaker 1: likely to be mismatched than lighter skin tones. That was
Speaker 1: about six years ago, and the industry says, and clear
Speaker 1: View in other companies says that their algorithms have improved
Speaker 1: dramatically over that time, and there's some evidence that they
Speaker 1: have gotten better. However, we ultimately don't know how accurate
Speaker 1: and how reliable this software is in the way that
Speaker 1: law enforcement use it because it's never been tested in
Speaker 1: the way that law enforcement use it, the way that
Speaker 1: that federal lab tests things. It's looking at basically perfect lighting,
Speaker 1: a perfectly framed profile photo in perfect conditions. How does
Speaker 1: that algorithm do in those settings? Unlike how police use
Speaker 1: these tools, which is, we're going to use this grainy
Speaker 1: surveillance image shot at a weird angle from above, usually
Speaker 1: usually if very poor lighting, and usually the face is
Speaker 1: partly obscure or it could be. In the case in
Speaker 1: Saint Louis, the guy was actually wearing a COVID mask
Speaker 1: that was covering part of his and he was wearing
Speaker 1: a hood that covered another part of his face. So
Speaker 1: this is the typical setting where police are using these tools.
Speaker 2: There's a case where a cartoon came up in a
Speaker 2: match at one point, right.
Speaker 1: Yeah, So because clear VII just has these I guess
Speaker 1: web crawlers that are just scraping the Internet, and so
Speaker 1: the things that come up and the Clearview results sometimes
Speaker 1: are bizarre. So in this one case in Ohio, two
Speaker 1: of the search results that I think we're in the
Speaker 1: top ten of the search results were Michael Jordan just
Speaker 1: a picture of Michael Jordan and I by the way,
Speaker 1: I reached out to Michael Jordan's rep when I published
Speaker 1: this in a story, and she didn't have any comment
Speaker 1: on whether he might have been implicated into crime in Ohio.
Speaker 1: And then the other one was yeah, just like it
Speaker 1: illustrated a cartoon image of a black man. So it
Speaker 1: just makes you like, kind of makes it you scratch
Speaker 1: your head as to you know, what is this tool
Speaker 1: that police are relying on if it's feeding them, you know,
Speaker 1: garbage as results.
Speaker 2: And giving them Michael Jordan committed this crime in the
Speaker 2: top ten.
Speaker 1: Right, which right, Yeah.
Speaker 2: Again a big allegedly here, but I feel pretty sure
Speaker 2: they'd have a tough time connecting him to the scene
Speaker 2: of whatever that that particular crime was.
Speaker 1: Yeah, And if this is such an advanced algorithm, why
Speaker 1: can't their computer figure out that's Michael Jordan and just
Speaker 1: take him out? And why can't a computer figure out
Speaker 1: that that's a cartoon black man and not a real person.
Speaker 1: I just ca't like a lot of the people who
Speaker 1: are who put forward facial recognition as this science, they
Speaker 1: have sort of cast it in the kind of the
Speaker 1: cloak of this is a scientific tool. The police are
Speaker 1: now using a lot of those arguments kind of break
Speaker 1: down when you see stuff like this, when you see, oh, well,
Speaker 1: your scientific, highly advanced tool, you know, brought back a
Speaker 1: picture of you know, cartoon black Man. How do the
Speaker 1: science arrive at that?
Speaker 2: Facial recognition as a technology is probably a whole other episod.
Speaker 2: But if you've ever used face ID to unlock your iPhone,
Speaker 2: you've already used the version of facial recognition on yourself.
Speaker 2: In some ways, the technology isn't all that different from
Speaker 2: how it started in the nineteen sixties, when researchers were
Speaker 2: trying to figure out how to identify facial features like
Speaker 2: the bridge of the nose, the edge of the nose,
Speaker 2: or the eyes and measure the distance between them. They'd
Speaker 2: store all those measurements as data for a single face.
Speaker 2: Back in the sixties, this stuff was pretty rudimentary, but
Speaker 2: AI's made it more accurate because you can train the
Speaker 2: systems on more faces, which means theoretically it's more likely
Speaker 2: to tell your face from someone else's. But you've probably
Speaker 2: had your face ID fail on you before, and that's
Speaker 2: under good conditions. It's just trying to match your face
Speaker 2: with well your face. If you add a few million
Speaker 2: other faces into the mix, the potential for mistakes multiplies
Speaker 2: by a lot. But even with those issues, police are
Speaker 2: increasingly leaning on this technology. It seems like police are
Speaker 2: presenting the fact that facial recognition software has returned a match,
Speaker 2: whether it's true or false, as evidence. But that seems
Speaker 2: it seems backwards, right.
Speaker 1: Yeah, Well, not only that, but it's on the one hand,
Speaker 1: they actually acknowledge that facial recognition is not enough to
Speaker 1: make a case. They say this in their police rules
Speaker 1: and their public statements over and over and over. If
Speaker 1: you hear police talk about facial recognition, they say to
Speaker 1: the public, this is only an investigative tool. We are
Speaker 1: only going to use this to find investigative leads that
Speaker 1: we then go out and corroborate. And in some places
Speaker 1: that's actually the law. There's six states where actually police
Speaker 1: are required to corroborate any leads that they given facial recgnition.
Speaker 1: What we found in looking at cases all around the
Speaker 1: country was that the police were not doing that. Sometimes
Speaker 1: they do, but often they will just take it at
Speaker 1: face value that this software hit is enough, and then
Speaker 1: they will use that to go out and arrest the
Speaker 1: per I encountered this really surprising thing, which is I
Speaker 1: talked to a number of prosecutors and police who told
Speaker 1: me that, well, yeah, we did corroboration. As soon as
Speaker 1: we got the name from the software, we went and
Speaker 1: we looked at that person's other photos and it visually
Speaker 1: looked like the same person. And they said that that
Speaker 1: is corroboration, visual corroboration of a match.
Speaker 2: That sounds like you're corroborating evidence on your own, like
Speaker 2: you're becoming a witness at that point.
Speaker 1: As a police officer, this thing comes up over and
Speaker 1: over is that many people look alike, and humans are
Speaker 1: very bad at distinguishing between two people who are similar.
Speaker 2: You would think that this is where technology could help,
Speaker 2: but AI isn't much better at telling people apart. The
Speaker 2: first known case of a wrongful arrest made by AI
Speaker 2: was Robert Williams in Detroit in twenty twenty. Doug was
Speaker 2: able to get the police interrogation video. William's case is
Speaker 2: really interesting because we have the interrogation video of this
Speaker 2: and I was just watching this and it's incredible. So
Speaker 2: this is you I know you've seen this video. So
Speaker 2: this is where they're you know, these two detectives are
Speaker 2: sitting in this interrogation room. Rob Williams is sitting across
Speaker 2: the table from them, and he's clearly confused. And they
Speaker 2: bring out these papers and I'll play it from here.
Speaker 5: December twenty sixth, around one pm? Where were you.
Speaker 3: Home? Home?
Speaker 5: Can you do me a favorite? Is that you?
Speaker 3: You know? No?
Speaker 5: Not even closed it? No, I'm pissed.
Speaker 3: Keep going.
Speaker 5: Is that you? No, that's not you at all? Not
Speaker 5: there either. You can't y'all can't tail that. I'm one hundred,
Speaker 5: so I'm one of the pictures. We actually got facial recognition.
Speaker 5: I heard that it was probably favorite recognition.
Speaker 3: That is not me.
Speaker 2: That's me on my d right, you're you're smiling and
Speaker 2: I'm smiling too. As bleak as this is. But there's
Speaker 2: something almost funny in this exchange.
Speaker 1: Well, they because they present the pictures as if it's
Speaker 1: evidence in their favor, and then as soon as he
Speaker 1: sees them and reacts to them, suddenly it becomes evidence
Speaker 1: in his favor because he's like, are you looking at
Speaker 1: the same picture I'm looking at because it's clearly not me?
Speaker 1: And it's funny. You can kind of hear in the
Speaker 1: officer's voice, well, well, it's we have facial recognition, as
Speaker 1: if you know, as if that's somehow, you know, bolsters
Speaker 1: his case that this is this is accurate, as if
Speaker 1: that's going to change his mind that the picture that
Speaker 1: he's looking at is actually him when he knows it's
Speaker 1: not him, As if the phrase facial recognition is itself
Speaker 1: kind of evidence of the police being correct.
Speaker 2: You know, he holds a picture up to his face, like, look,
Speaker 2: of course, this isn't me. What are you talking about, Kate?
Speaker 2: Forget the software, look at me. The thing that hits
Speaker 2: you about this interaction is that when the cop finally
Speaker 2: looks at the picture, you could hear the room go
Speaker 2: quiet for a second, and nobody's arguing with Robert Williams,
Speaker 2: and it kind of sounds like he's agreeing, like it's
Speaker 2: the first time he's actually looking at it, like he
Speaker 2: had so much confidence in the machine that he didn't
Speaker 2: bother to look with his own eyes.
Speaker 1: They by and large are not telling even the people
Speaker 1: that they identify and arrest using his tools that they
Speaker 1: use them, and that's leading to people either finding out
Speaker 1: kind of in offhand ways in interrogation that oh, well,
Speaker 1: you know the computer picked you. Well, what do you
Speaker 1: mean by the computer. A lot of times, they're probably
Speaker 1: not finding out at all, which is concerning because one
Speaker 1: of the pillars of our courts and our justice system
Speaker 1: is that you need to be able to face your accuser.
Speaker 1: And so if your accuser is this algorithm, this computer program,
Speaker 1: but you're not even being told that it was used,
Speaker 1: let alone given any of the details about how it works.
Speaker 1: So all of these things are being kind of kept
Speaker 1: vague or sometimes completely kept hidden from the people that
Speaker 1: these tools are being used to investigate and ultimately arrest.
Speaker 2: So what does this mean for the future of policing
Speaker 2: and how should we be using these technologies if at all?
Speaker 2: That's after the break. People are being investigated and arrested
Speaker 2: and not being told that police are using these facial
Speaker 2: recognition tools to find them. You don't need a law
Speaker 2: agree to feel that something about that is just kind
Speaker 2: of off, and you wouldn't be wrong. Here in the US,
Speaker 2: there's a specific set of rules that might apply here,
Speaker 2: the Brady rules. I want to talk about that. So
Speaker 2: can we talk about how Brady rules can can potentially
Speaker 2: figure into an arrest that is made with AI facial
Speaker 2: recognition as a part of that or is the sole
Speaker 2: part of it?
Speaker 1: Yeah, So basically, when you prosecute somebody, the prosecution is
Speaker 1: required by the courts and required by the Constitution to
Speaker 1: share any evidence that speaks to the guilt or the
Speaker 1: innocence of the person that they're prosecuting, so whired to
Speaker 1: share that with the defense. There's a question now the
Speaker 1: courts are grappling with, which is does somebody's identification by
Speaker 1: an AI software? Should that be brought into the courtroom?
Speaker 1: And should the should the defendant be given a chance
Speaker 1: to know everything about that software? And right now this
Speaker 1: is playing out in courtrooms across the country.
Speaker 2: Basically putting the AI in a manner of speaking, putting
Speaker 2: the AI on the witness stand and saying how reliable
Speaker 2: really are you? Yeah.
Speaker 1: A very good indicator of how that's going to go
Speaker 1: is the companies themselves have disclaimers saying this does not
Speaker 1: hold up in courts. Clearview AI is the one that
Speaker 1: I'm very familiar with. They have language and all of
Speaker 1: their contracts with police departments saying this is not admissible
Speaker 1: evidence in court. And also basically, please don't ask us
Speaker 1: to come into court, because nobody from Clearview AI is
Speaker 1: going to come and sit in court and defend this software.
Speaker 1: The very companies who make it and advertise it and
Speaker 1: market it as this great, amazing technology tool, they would
Speaker 1: not even stand behind it.
Speaker 2: In your research, in your reporting, you've seen what looks
Speaker 2: like and what defendants are saying wrongful arrest based on AI.
Speaker 2: Have the police who made these arrests or who made
Speaker 2: those decisions. Has there been any repercussions, any discipline, any
Speaker 2: guidance for them that you've seen.
Speaker 1: I mean, the answer is no, as far as I
Speaker 1: can tell, the individual officers have not publicly faced any punishment,
Speaker 1: although you know, you never know what happens behind closed doors.
Speaker 1: Typically police unions forbid police from publicly talking about punishment
Speaker 1: of individuals. Now, we've seen settlements paid out of three
Speaker 1: hundred thousand dollars to a few of these people who
Speaker 1: are wrong fully arrested. Is this going to change the
Speaker 1: behavior of these individuals. There's only six states that have
Speaker 1: laws mandating specific disclosures and mandate specific things about how
Speaker 1: these tools cannot be used. So you know, the vast
Speaker 1: majority of the country police are kind of still just
Speaker 1: kind of figuring out. It's really early days.
Speaker 2: So just to back up here, we've been talking about
Speaker 2: the problems with this technology, what about the upsides? And
Speaker 2: it's interesting because if you ask people how facial recognition
Speaker 2: could be used to catch criminals, there's not really any
Speaker 2: high profile cases we could point to. And before you
Speaker 2: say January sixth, well that's complicated.
Speaker 1: A lot of people attribute facial recognition with helping to
Speaker 1: I identify these people and to bring them to justice.
Speaker 1: And yes, facial recond did play a role. However, on
Speaker 1: that day, there was a lot of evidence being collected
Speaker 1: through social media posts that these people were making on
Speaker 1: their own videos that these people were taking and posting.
Speaker 1: I think the federal investigators even use of high tech
Speaker 1: methods to grab the cellular location data of everybody who
Speaker 1: is in the capital that day, and I think they
Speaker 1: were able to identify a lot of people that way.
Speaker 1: So facial actation in that case was one of many
Speaker 1: tools that were used, which is sort of how it's
Speaker 1: designed to be used, is you know, not as kind
Speaker 1: of the sole investigative lead, but as one of several. So,
Speaker 1: you know, other than that, there aren't that many well
Speaker 1: known cases out there that we can point to of
Speaker 1: you know, this was the thing that brought down, you know,
Speaker 1: a murder case that we've been trying to solve for
Speaker 1: many years. But I bet we will start to hear
Speaker 1: about some of that, and I would love to kind
Speaker 1: of have more of those case studies we can pick apart.
Speaker 1: I think the public deserves to know about both. The
Speaker 1: public should also be aware of the good and the
Speaker 1: benefit because as we kind of make laws or think
Speaker 1: about regulating how to ring this in, it's going to
Speaker 1: be important to understand that balance and to get it right.
Speaker 1: I think that there's a danger and rushing to a
Speaker 1: blanket ban on this technology.
Speaker 2: I want to play some for you really quick. This
Speaker 2: is a really I think powerful moment from your podcast,
Speaker 2: and I was hoping I could have have you speak
Speaker 2: to this a little bit here.
Speaker 5: They lean on the technology because I think we are
Speaker 5: taught that computers don't make mistakes.
Speaker 4: Humans do.
Speaker 2: Now I'm sitting there like, how can you do this?
Speaker 3: You can't do this?
Speaker 1: Like, no way, how can you just put these charges
Speaker 1: on them?
Speaker 3: And I'm telling you that that's not me.
Speaker 2: I know I was in this, So how do I
Speaker 2: beat a machine?
Speaker 3: So?
Speaker 2: I mean, I guess I guess my question is here
Speaker 2: these I'm just gonna call them AI arrests. How do
Speaker 2: these arrests affect the people who are being accused.
Speaker 1: That was a clip of our episode of Post Reports,
Speaker 1: and we heard these patterns of experience which were very notable.
Speaker 1: So a few things. Wrongful arrests are not a new thing.
Speaker 1: The technology did not bring about wrongful arrests, but the
Speaker 1: idea of you know, usually there's some other incidental relationship
Speaker 1: some somebody has to a crime before they get into
Speaker 1: a wrongful orest. In these cases, the AI is plucking
Speaker 1: you out of thin air in some cases. And in
Speaker 1: one case, this man, Karan Reid, he had never actually
Speaker 1: even been to the state of a Louisiana and he
Speaker 1: was resident in Atlanta, so he's literally been plucked from
Speaker 1: the other side of the country and brought into this
Speaker 1: crime that he had literally no proximity to whatsoever. So
Speaker 1: this what does that do to the human brain when
Speaker 1: you are just suddenly poof inside of a criminal investigation
Speaker 1: that you have no connection to whatsoever. It's like baseline
Speaker 1: is a word that came up over and over. A
Speaker 1: lot of people just kind of got stuck on that idea,
Speaker 1: even when they sat in jail for one or two
Speaker 1: or three weeks or months in the case of Chris Gatlin,
Speaker 1: who was in jail for seventeen months. Some of them
Speaker 1: had this feeling of and I think you just heard
Speaker 1: it in the clip. If the computer is telling you
Speaker 1: that I did it, then how am I going to
Speaker 1: convince you? Otherwise? How am I going to beat a computer?
Speaker 1: Because you know, it goes back to this thing that
Speaker 1: we were talking about before, this automation bias of you know,
Speaker 1: he never heard of that term, but he instinctively realized
Speaker 1: that he was going against a power that was much
Speaker 1: greater than him and in some ways probably much greater
Speaker 1: than if a witness had picked you for a crime.
Speaker 1: It's like it's a computer picked you for a crime
Speaker 1: and put your name on the line and put your
Speaker 1: name into this investigation.
Speaker 2: So it's our belief in the computer that's what you're
Speaker 2: up against.
Speaker 1: You know, many of the people that we talked to
Speaker 1: and the cases that are public, many of them feel
Speaker 1: like they kind of were the lucky ones. And maybe
Speaker 1: they got out because one of them had a mole
Speaker 1: on his face and his lawyer discovered that the guy
Speaker 1: and the surveillance image didn't have a mole. Or there
Speaker 1: was one woman, Portia Woodriff who was eight months pregnant
Speaker 1: at the time she has arrested, and you know, there
Speaker 1: was nothing in the interviews with the victims that said
Speaker 1: there's a pregnant woman involved. So that's ultimately one of
Speaker 1: the reasons that she got out. But are there many
Speaker 1: other people who didn't get lucky and might might be
Speaker 1: behind bars still because of facial recognition, you know, wrongfully
Speaker 1: put them there.
Speaker 2: I mean, when you put it like that, you start
Speaker 2: to make me think that maybe we're not hearing horror stories.
Speaker 2: Those are the success stories.
Speaker 1: This was pretty horrible for these people, not just them
Speaker 1: in some of the cases. You know, their their children,
Speaker 1: their young children watch them get arrested in their front lawns,
Speaker 1: and that's trauma.
Speaker 3: Yeah.
Speaker 1: One of the one of the men that we've talked
Speaker 1: about before, Robert Williams, who's arrested in Detroit. You know,
Speaker 1: his kids watched them get arrested in the front lawn
Speaker 1: and he says, to this day, they still talk about
Speaker 1: that incident. One one of his daughters. This is kind
Speaker 1: of heartbreaking. One of his daughters every once in a
Speaker 1: while comes up to him and says, Daddy, I think
Speaker 1: I've solved it. I found the real man who went
Speaker 1: and stole those watches, and she pointed to like a
Speaker 1: cartoon character as like that was the one that was
Speaker 1: the real guy who stole the watches and not daddy.
Speaker 2: Oh my god.
Speaker 1: So these, I mean, these experiences are now baked into
Speaker 1: their lives and their childhoods. And then that's you know,
Speaker 1: I'm a parent of small kids, and that's a very
Speaker 1: sad thing. And yes, they were lucky, probably luckier than
Speaker 1: other people whose stories we don't know about, but also
Speaker 1: these are horrible experiences.
Speaker 2: I'm not sure what y'all think about this. I'm still
Speaker 2: trying to figure that out myself, and I know a
Speaker 2: lot of what we've heard sounds pretty terrifying. So for
Speaker 2: one of my last questions, I asked Doug what he
Speaker 2: thought the next few years were going to look like
Speaker 2: with this technology, and he answered in a way that
Speaker 2: I didn't really expect.
Speaker 1: We're still at a point of fascination with this technology,
Speaker 1: and in places around the country, maybe not in the
Speaker 1: big cities, where I mean you do see kind of
Speaker 1: a skepticism of tech tools and concerned about privacy and surveillance,
Speaker 1: but in many of the places in this country, you know,
Speaker 1: in smaller towns around the country, police are adopting these
Speaker 1: things and are excited and there's a fascination, and they're
Speaker 1: being sold to the public as this kind of magical tool,
Speaker 1: like in Evansville, Indiana, in Florence, Kentucky, in Saint John's County, Florida,
Speaker 1: these smaller town places where you know, the local population
Speaker 1: may not know that the police have started to like
Speaker 1: rely on these tools quite a bit. So a simple
Speaker 1: goal for my reporting has just been to just kind
Speaker 1: of encourage people to ask questions to your go to
Speaker 1: your city council meeting. You'll actually learn a lot. You
Speaker 1: don't have to sit on the sidelines of this discussion
Speaker 1: because you're part of it. You are a person that
Speaker 1: is in front of these surveillance cameras too. So now
Speaker 1: there's a risk of you, everybody in this country of
Speaker 1: getting pulled into an investigation due to, you know, a
Speaker 1: misuse of this technology. So you should everybody should think
Speaker 1: about themselves as being part of this discussion about whether
Speaker 1: and how we use this technology going forward.
Speaker 2: Thank you so much for listening to our first episode
Speaker 2: of The New kill Switch, and a very special thanks
Speaker 2: to our friends over the Washington Post. Podcast Post reports
Speaker 2: who helped us with this episode. You can check out
Speaker 2: more Doug's reporting over there and stay with us. We
Speaker 2: have so much more ahead. This show is all about
Speaker 2: what's happening right now between humans, between machines, how we're
Speaker 2: being shaped by our own creations, and a whole lot
Speaker 2: beyond that. So let us know what you think, and
Speaker 2: even if there's something you want us to cover, you
Speaker 2: can hit us up at kill Switch at Kaleidoscope dot NYC,
Speaker 2: or you could hit me at dex Digi on the
Speaker 2: Gram or Blue Sky if that's more your thing. Kill
Speaker 2: Switch is hosted by me Dexter Thomas. It's produced by
Speaker 2: Shena Ozaki, Daryl Potts, and Kate Osbourne. Our theme song
Speaker 2: is by Kyle Murdoch, who also mixed the show. From Kaleidoscope.
Speaker 2: Our executive producers are Ozwa Washin, Mangesh Hutti Goodur, and
Speaker 2: Kate Osborne from iHeart, our executive producers are Katrina Norvil
Speaker 2: and Nick Etur. That's it for me, catch on the
Speaker 2: next one. H