Speaker 1: Sleepwalkers is a production of our heart Radio and unusual productions.
Speaker 1: Every single thing that you do on a social network
Speaker 1: or on a website is essentially recorded. How many pages
Speaker 1: you visited, what did you click on, how did you
Speaker 1: get to that website? What page did you leave on?
Speaker 1: How many photos have you ever uploaded? Where were those
Speaker 1: photos uploaded? How many places have you checked into? Who
Speaker 1: have you tagged? What photographs have you been tagged in with? Whom?
Speaker 1: Where were those photographs taken? Who's in your friendship circle?
Speaker 1: Who did you go to school with? There are algorithms
Speaker 1: and machine learning technologies that connect all of that data
Speaker 1: together and start to find patterns. That's Lisa italian Aretti speaking.
Speaker 1: She's a digital sociologist and tech ethics advocate with a
Speaker 1: big focus on data. I think some of the data
Speaker 1: that's quite concerning is facial recognition technology, where machines are
Speaker 1: being fed huge amounts of images pictures of people's faces,
Speaker 1: and that can come from dating websites, from photographs of
Speaker 1: you on Facebook or Instagram or anywhere on the Internet.
Speaker 1: You don't even have to upload it yourself. It could
Speaker 1: be somebody else that's uploaded it for you. With all
Speaker 1: that data out in the wild. All it takes is
Speaker 1: for somebody to suck it up and they can start
Speaker 1: connecting the dots. So there were researchers last year from
Speaker 1: Stanford University who without the user's permission, scraped thirty thousand
Speaker 1: photographs from a dating website that was public rights. So
Speaker 1: they took it that they could just use those photographs
Speaker 1: for research. Because it was a dating site, people will
Speaker 1: also asked the data points, um, what is your sexual orientation?
Speaker 1: So gay, straight, by right, and so they had all
Speaker 1: of that data connected, so they could then connect faces
Speaker 1: to sexual orientation and essentially they built an algorithm that
Speaker 1: they said could detect just by somebody's face if they
Speaker 1: were gay or straight or by the algorithm worked, it
Speaker 1: could predict sexual orientation from photographs with accuracy for men
Speaker 1: and accuracy for women based on just five photographs. And
Speaker 1: Stanford isn't the only place using facial recognition to categorize people.
Speaker 1: Governments are too all around the world. This is Sleepwalkers Welcome.
Speaker 1: I'm Azveloshin. In the last episode, we looked at what
Speaker 1: happens when artificial intelligence digests huge data sets to find
Speaker 1: patterns and make predictions, like what's the most typical move synopsis?
Speaker 1: Or is that moral cancerous? But advances in machine learning
Speaker 1: are also making us more and more legible. In today's episode,
Speaker 1: we ask what happens when we become the data set
Speaker 1: and the power to predict is turned on us. Here's
Speaker 1: Lisa again. The extra terrifying thing is why creates this
Speaker 1: kind of technology? Like? Who does it serve? Why do
Speaker 1: we need this? If you take this type of technology,
Speaker 1: feed it to a citywide CCTV surveillance system and say
Speaker 1: that you go to a place like Saudi Arabia where
Speaker 1: being gay is considered a crime, and suddenly you're just
Speaker 1: what pulling people off the street and arresting them because
Speaker 1: you're gay because the computer said so, so like now
Speaker 1: you're going to prison. This may sound like a terrifying
Speaker 1: Minority Report style future, but actually it's here today. I
Speaker 1: Cara Highs. So Lisa was talking about what happens when
Speaker 1: governments start to connect the dots of all this data,
Speaker 1: but it's already happening with private enterprise. For example, insurance
Speaker 1: companies can now see someone who joins a Facebook group
Speaker 1: about a genetic mutation and use that data to guess
Speaker 1: that that person may have the genetic mutation and the
Speaker 1: condition that's associated, and then the computer says, let's raise
Speaker 1: their premium or let's deny them care. And so we
Speaker 1: can use this peroxy data to do something The New
Speaker 1: York Times has recently called proxy discrimination. And I think
Speaker 1: something that's even more widely applicable is this definition of
Speaker 1: surveillance capitalism, which is essentially that data is not just
Speaker 1: data anymore, it's money. Companies can use data to make
Speaker 1: predictions about future behavior and that can make them profit. Right,
Speaker 1: that surveillance capitalism. I using information about joining a Facebook
Speaker 1: group to make an insurance decision, and it's I don't know,
Speaker 1: it's a little bit scary. In the US, it's all
Speaker 1: about capitalism, But in other countries surveillance is used for
Speaker 1: other purposes. For example, in China, it's about social control.
Speaker 1: But in China, this same massive ingestion of data and
Speaker 1: statistical modeling is used for governance. So let's take a
Speaker 1: closer look at China and how they're using technology to
Speaker 1: amplify the power of the state. They have this notion
Speaker 1: of a social credit score, which is how good of
Speaker 1: a citizen you are according to the government. They have
Speaker 1: literally hundreds of millions of cameras around and they can
Speaker 1: basically do things like you've been out at the bar,
Speaker 1: you know, until to the last couple of nights. That's
Speaker 1: not really what a good upstanding citizen would do. So
Speaker 1: your social credit score can get ding because you've been
Speaker 1: spending too much time at night at a bar. That's
Speaker 1: Dr Alex Kilpatrick. He and Mary Haskett co founded Blink Identity,
Speaker 1: a facial recognition company that can recognize the users face
Speaker 1: in yes, the blink of an eye about no point
Speaker 1: four seconds. Here's Blink co founder Mary on Chinese surveillance.
Speaker 1: They have camera, so you're recorded jaywalking, and so your
Speaker 1: score goes down, you know, and you automatically get a ticket,
Speaker 1: which again doesn't sound like that they give a deal
Speaker 1: if you really believe in law and order, except if
Speaker 1: your score isn't high enough, you can't buy a plane ticket,
Speaker 1: you have to travel by bus, you know, you can't
Speaker 1: live in certain areas. And you can obviously see how
Speaker 1: this could be abused. I mean, it doesn't take much
Speaker 1: of an imagination. Other factors that can bring down your
Speaker 1: social credit score include what you buy at the store,
Speaker 1: your online browsing, and even having a friend with a
Speaker 1: low score. This use of generalized surveillance can keep a
Speaker 1: whole population in check, which is more or less the
Speaker 1: explicit goal of the Communist Party of China. This can
Speaker 1: be stifling for the average hand Chinese citizen. For minorities,
Speaker 1: it can be much much worse, right, you know, more
Speaker 1: specifically and more dauntingly, it can be used for internment,
Speaker 1: you know, in the case of the weaker minority in China,
Speaker 1: using data that is being created by this minority to
Speaker 1: just communicate, you know, one person communicating with another person,
Speaker 1: those who are connected that both wagas right, five weeks
Speaker 1: are gathering the same place. Now that we know that,
Speaker 1: you know, what are we going to do with that data? Oh,
Speaker 1: We're going to send these people to re education camps.
Speaker 1: And as technology improves, so does the state's ability to
Speaker 1: project power. China today is cleaning the floor with the
Speaker 1: Americans on voice and facial recognition technology. That's the Embrema,
Speaker 1: an expert on global political risk and the founder of
Speaker 1: the Eurasia Group. The Chinese have much more data. You
Speaker 1: also have a government that is consolidating the data and
Speaker 1: allocating it for different types of purposes. And you have
Speaker 1: no presumption of privacy whatsoever. With no presumption of privacy,
Speaker 1: the amount of data you can collect from your citizens
Speaker 1: grows exponentially, and that actually gives you a huge technological advantage.
Speaker 1: This is how Kai Fu Lee explains it. What makes
Speaker 1: sense AI algorithm work better is how much data you
Speaker 1: used to train it. And that's the beauty of deep learning.
Speaker 1: You just keep throwing data at it and it just
Speaker 1: performs better. And Kai fully understands this world better than most.
Speaker 1: His fund, Signivation Ventures, has invested in meg v, a
Speaker 1: facial recognition company valued at four billion dollars. Kaifu also
Speaker 1: around Google China, so he knows the landscape. China simply
Speaker 1: has more data than the US due to not only
Speaker 1: the large number of users, but also the depth in
Speaker 1: which Chinese users use the Internet for ordering food, for
Speaker 1: shared bicycles, for mobile payments. So the AI will actually
Speaker 1: just perform better because it's trained out more data. Some
Speaker 1: of that data is taken from citizens using surveillance, but
Speaker 1: according to Kaifu, much is freely given. I think the
Speaker 1: Chinese culture and the Chinese people are more pragmatic, so
Speaker 1: that if the software delivers pragmatic value, they ask fewer questions.
Speaker 1: For example, you know, we've funded an app that loans
Speaker 1: money to ask you a couple of questions and takes
Speaker 1: data from your phone at the same level as the
Speaker 1: Facebook would take data from your phone, and it's SAPs
Speaker 1: the money to you instantly if it decides to lend
Speaker 1: money to you. I think in the US people my
Speaker 1: question do I really want to give my data to
Speaker 1: a lending application? And is it appropriate to consider the
Speaker 1: makeup my phone as a part of that give me
Speaker 1: giving me money or my zip codes? Because that might
Speaker 1: reflect certain things about me. The breadth and depth of
Speaker 1: data in China, both from a larger population and a
Speaker 1: much deeper integration with technology, gives China a serious competitive advantage.
Speaker 1: Where the Americans have much better scientists, the Chinese were
Speaker 1: able to buy a lot of science. Now, when you
Speaker 1: talk to specialists in this field, they will tell you
Speaker 1: that in many parts of Ai, great data and okay
Speaker 1: scientists will frequently beat great scientists and okay data. Character
Speaker 1: scary thing is that China is using that technological superiority
Speaker 1: to build a very different kind of state, you know,
Speaker 1: one in which the price of descent is intolerably high.
Speaker 1: You know. So according to Ian public demonstrations have fallen marketly, right,
Speaker 1: because if you know you're being watched, you're probably less
Speaker 1: likely to commit public displays of civil disobedience. Right. We
Speaker 1: talked about the vigas earlier in Cashgar, which is a
Speaker 1: Muslim city in China. Vigas have to register to go
Speaker 1: into the mosque, and once they're inside, they face a
Speaker 1: bank of cameras like many many surveillance cameras, and go
Speaker 1: figure Muslims stopped going to mosque voluntarily well, because going
Speaker 1: to mask is an act of civil disobedience where they are.
Speaker 1: Even if it's not explicitly stated, it's it's heavily implied, right.
Speaker 1: I mean, I think about it for myself, Like if
Speaker 1: I were living in an area in the United States
Speaker 1: where going to temple was going to land me in
Speaker 1: an internment camp, I would not be going voluntarily if
Speaker 1: I knew there were security cameras all over my temple. Absolutely,
Speaker 1: And the crazy thing is you wouldn't even have to
Speaker 1: know if those security cameras work. It's like on the
Speaker 1: o TO in England, we have a lot of speed
Speaker 1: cameras and no one knows that they're actually doing it.
Speaker 1: It's staying apart from once every five years you've got
Speaker 1: a ticket. But it still slows people down the panopticon
Speaker 1: even if they can't do that. But people think they
Speaker 1: can do that, right. I mean, you don't need a
Speaker 1: hundred percent certainty. You just need a government that is
Speaker 1: starting to get that capacity and make it known and
Speaker 1: have a few people that are sort of strung up
Speaker 1: as examples, and suddenly everyone's scared. And this isn't only
Speaker 1: happening in China. According to Ian, it was a key
Speaker 1: part of Bashar al Assad's strategy in the Syrian Civil War.
Speaker 1: Assade got some help from the Russians, who gave them
Speaker 1: a couple of hundred computer scientists to go in work
Speaker 1: with the Syrian military and identify on social media and
Speaker 1: on text messaging who wore those Syrian citizens that were
Speaker 1: nodes of dissent and within six months no more moderate
Speaker 1: opposition in Syria. They specifically, we're looking into individual Syrian
Speaker 1: citizens that we're saying things about the regime that we're untoward,
Speaker 1: that we're connected to influencers that were helping to organize protests,
Speaker 1: and suddenly, you know, a bunch of those people were
Speaker 1: rounded up and some were never heard from again. And
Speaker 1: as I mentioned in terms of China, you don't have
Speaker 1: to do that with many people before people start ratting
Speaker 1: out their friends, being scared of talking to anyone not
Speaker 1: going out. The system worked. We may feel comfortably far
Speaker 1: from the battlefields of Syria here in the US, and
Speaker 1: from the overwhelming number of surveillance cameras on practically every
Speaker 1: street corner in China. But the more effective these technologies are,
Speaker 1: the more likely they are to be adopted by others. Now,
Speaker 1: in other countries, you're going to have a confluence of
Speaker 1: both them liking the model and the Chinese directly exporting it.
Speaker 1: Who were those countries, Well, look at One Belt, One Road,
Speaker 1: the you know, trillion plus dollar investments that the Chinese
Speaker 1: are making all over the world in Pakistan, Southeast Asia,
Speaker 1: you know, Cambodia, laughs, a whole bunch of countries. And
Speaker 1: when you look at those countries and you see that
Speaker 1: the Chinese are providing the money in this conditionality in return,
Speaker 1: some that conditionalities use Chinese standards for technology that's in
Speaker 1: many of these contracts. And with the spread of Chinese
Speaker 1: standards of technology comes the spread of Chinese style surveillance,
Speaker 1: which could ultimately make the whole world trend more authoritarian.
Speaker 1: So as that happens, these governments are going to say,
Speaker 1: ah ha, we get the money from China, we use
Speaker 1: their technology. We're stuck with their system, but we can
Speaker 1: use it to ensure that our people stay in power. Again,
Speaker 1: it's easy to let all of this feel comfortably far away,
Speaker 1: but remember the Internet doesn't have borders, so you don't
Speaker 1: have to be in China for the Chinese state to
Speaker 1: access your data. Yeah. I don't know how many people
Speaker 1: know this, but Grinder is actually owned by a Chinese company,
Speaker 1: Grinder the dating MP. Yes, and actually there have been
Speaker 1: articles about the fact that the US government is trying
Speaker 1: to force China's hands so that we can buy it
Speaker 1: back because there's so much user data that this company
Speaker 1: now owns. Well, that Grinder use a data is basically
Speaker 1: being seen by the US government as a strategic asset.
Speaker 1: It is a strategic asset. I mean, if you think
Speaker 1: about it, if somebody is on a military base or
Speaker 1: in a barrack and trying to connect with someone on Grinder,
Speaker 1: they're turning their location services data on because they want
Speaker 1: to see people in their area and if they're turning
Speaker 1: that location services data on, they're basically making themselves vulnerable
Speaker 1: to the company that owns the data, because it's basically
Speaker 1: saying here I am, here, I am and not only that,
Speaker 1: here I am, here, I am, and I'm gay, and
Speaker 1: that can lead to some possibilities of blackmail. Even today, exactly,
Speaker 1: there was an article in the interface that was written
Speaker 1: by this guy, Casey Newton, chat history, photos, videos, real
Speaker 1: time location. All of that is connected to a user's
Speaker 1: email address, and that means that the user's identity can
Speaker 1: be very easily learned. That's pretty scary. And even if
Speaker 1: don't use Grinder, you might use Flash of Clans or Fortnite,
Speaker 1: which are very popular gaming apps also owned by the Chinese.
Speaker 1: Now we keep saying the Chinese. To be clear, these
Speaker 1: apps aren't owned by the Chinese government, that owned by
Speaker 1: Chinese companies. What the actions of the US government imply
Speaker 1: by trying to force this company to sell Grinder back
Speaker 1: is that they don't believe in the distinction. And do
Speaker 1: you think the US government would really be working that
Speaker 1: hard to get back a gay dating app if they
Speaker 1: didn't think that there was not a murky separation between
Speaker 1: the government and companies in China. Right, So every time
Speaker 1: we give data away, I mean, we're aware that it
Speaker 1: opens us up to targeted ads on Facebook. We talked
Speaker 1: about those with Gillian. We're not aware that that data
Speaker 1: may end up in the hands of potentially hostile foreign government.
Speaker 1: So once again, Sleepwalking. We've been talking about how foreign
Speaker 1: governments are using AI, but when we come back, we'll
Speaker 1: look at how the police and courts are using it
Speaker 1: at home in America. Kara, it's easy to look at
Speaker 1: China and to see the big bad wolf. They're using
Speaker 1: surveillance technology for the wholesale suppression of an ethnic minority.
Speaker 1: They have a social credit score that con emit access
Speaker 1: to opportunities and even travel, But algorithms also determined outcomes
Speaker 1: here in the US exactly if you think about it,
Speaker 1: we do use social ratings. If you use Uber and
Speaker 1: have a rating lower than four out of five stars,
Speaker 1: you can't get a car right and you can't get
Speaker 1: alone if you have a low FICO credit score. And
Speaker 1: our criminal justice system also uses algorithmic ratings to decide
Speaker 1: people's fate. When I got arrested at sixteen, I was
Speaker 1: in high school and John F. Kennedy High School. That's
Speaker 1: Glenn Rodriguez. When he was a baby, Glenn's mother was murdered,
Speaker 1: and when he was three, his father committed suicide. From
Speaker 1: then on, Glenn was raised by his grandmother and he
Speaker 1: searched for belonging. The kid who wants to feel accepted,
Speaker 1: wants to feel a part of something right. Whatever the
Speaker 1: group was up for, I was down. And if they
Speaker 1: were going one step forward, I would take two. We
Speaker 1: pretty much planned a robbery at a car dealership in
Speaker 1: Queen's and we entered the premises. We took three cars.
Speaker 1: There was a twenty five year old men in there,
Speaker 1: and he initially pulled a gun, and so I had
Speaker 1: a gun and I shot him. Glenn was arrested and
Speaker 1: convicted of second degree murder. He was sentenced to twenty
Speaker 1: five years in jail, and he was still a high schooler.
Speaker 1: You feel powerless, you feel hopeless, especially at that age.
Speaker 1: So the way I saw it was, this is my life.
Speaker 1: You know, I'm probably gonna die in jail, and so
Speaker 1: whatever it is that I have to do, I need
Speaker 1: to survive. One of the things that I learned very
Speaker 1: quickly is that in prison, one of the only things
Speaker 1: that is respected is violence, and so in order for
Speaker 1: you to survive in there, you have to be violent,
Speaker 1: because otherwise you'd be come pray. In time, Glenn established
Speaker 1: his reputation and started to feel safer. With that security
Speaker 1: and getting older, his thinking began to change. And it
Speaker 1: wasn't until later, to like my mid twhenies, when I
Speaker 1: started saying, you know what, I need to reverse this
Speaker 1: trend if I want to have any chance at parole.
Speaker 1: Glenn had to reverse thirteen years of behavior to survive
Speaker 1: in prison. He had learned to behave one way, but
Speaker 1: to get out he had to behave another. I reveiled
Speaker 1: myself of the Puppies behind Bars program, so I was
Speaker 1: training service dogs for wounded war veterans for five years.
Speaker 1: That was an amazing experience, right because throughout incarceration it's
Speaker 1: almost like you build a wall around yourself with the dogs.
Speaker 1: You can't fake it with the dogs. If you're trying
Speaker 1: to like teach them a command, sometimes you may have
Speaker 1: to be silly, which guess what, in prison, being silly
Speaker 1: is not acceptable. That's perceived as a weakness. But with
Speaker 1: that program, often times you had the resort of being
Speaker 1: silly and throwing yourself on the floor and giggling loud
Speaker 1: and making all kinds of crazy sounds to try to
Speaker 1: get the dogs attend gin Right. To a very large extent,
Speaker 1: I believe that that program kind of helped me regain
Speaker 1: my humanity as well as helping Glenn. Personally. Taking part
Speaker 1: in prison programs for the public good is looked upon
Speaker 1: favorably by parole boards. So everything I did, I wanted
Speaker 1: to document the kind of showcases what I've done, this
Speaker 1: is who I am today. As part of the process,
Speaker 1: there's also the Campus Risk Assessment COMPUS stands for Correctional
Speaker 1: Offender Management Profiling for Alternative Sanctions. It's an algorithm that
Speaker 1: claims to be able to predict how likely a defendant
Speaker 1: is to commit another crime based on a list of
Speaker 1: a hundred and thirty seven questions. Since being developed in
Speaker 1: its estimated COMPASS has been used to assess more than
Speaker 1: one million defendants, including Glenn. You meet with this person
Speaker 1: a few months before your schedule parole board day, and
Speaker 1: they asked you a series of questions. And so when
Speaker 1: he got to the disciplinary history section of the Campus
Speaker 1: Risk Assessment, there was a list of offenses right for
Speaker 1: him to check off. Yes or no for the past
Speaker 1: twenty four months, and it was all known. And anyone
Speaker 1: who has any experience with prison would tell you that
Speaker 1: that is almost impossible to do, right, because misbehavior reports
Speaker 1: can be for something as simple as having too many pillows,
Speaker 1: something as simple as your parents hanging off your bot,
Speaker 1: your sneaker is untied. It takes a lot of energy
Speaker 1: to dodge a misbehavior report during the course of a year,
Speaker 1: let alone ten, and in my case, it had been eleven.
Speaker 1: And then I heard him read the question and he says,
Speaker 1: does this person appear to have notable disciplinary issues? And
Speaker 1: he says yes, And I was like, hold up, wait
Speaker 1: a second, did I just hear you right? Because I
Speaker 1: just heard you say that I have notable disciplinary issues?
Speaker 1: Do you do realize that I haven't had a misbehavior
Speaker 1: report in over a decade? Right? And his answer was well,
Speaker 1: I was told that if there's any instance of misbehavior
Speaker 1: at any point, I have to check yes for this answer.
Speaker 1: So I was like, okay, So at that point there
Speaker 1: was nothing I could do. I'm appearing before the parole
Speaker 1: board panel. I presented to them a portfolio that was
Speaker 1: approximately one hundred pages, had letters to support Now the
Speaker 1: Compass is saying that I'm a disciplinary issue and so
Speaker 1: I shouldn't be released. But I was denied because of
Speaker 1: the fact that I scored high on Compass. They played
Speaker 1: a safe and kept me in. It may have been
Speaker 1: less than five minutes the hearing. I waited twenty six
Speaker 1: years to sit in front of a panel of three
Speaker 1: people for less than five minutes. No one wants to
Speaker 1: be the one to go against Compass, and next you know,
Speaker 1: something goes wrong and now your job is on the
Speaker 1: line because you departed from Compass, which is taken as
Speaker 1: factual and scientific. In time, Glenn went before another parole
Speaker 1: board and this time they freed him against the recommendation
Speaker 1: of Compass. And now Glenn has built a life for
Speaker 1: himself working with teenagers at risk, giving conceration at the
Speaker 1: Center for Community Alternatives. But he's still being affected by
Speaker 1: the algorithm. Compass does not end upon your release because
Speaker 1: the same Compass risk assessment that's considered for your release.
Speaker 1: The term is how you're going to be supervised upon release.
Speaker 1: There's a number of restrictions that I have. I have
Speaker 1: a curve you. I'm still haunted by a Compass. Despite
Speaker 1: turning his life around Compass is still limiting Glen's freedom,
Speaker 1: and that should haunt all of us. According to propublic A,
Speaker 1: Compass inaccurately labels black defendants as likely to reoffend twice
Speaker 1: as often as white defendants. Algorithmic discrimination isn't government policy
Speaker 1: here in the US like it is against the weak
Speaker 1: as in China, but it still exists. There's this issue
Speaker 1: where you can have computer scientists building a more accurate algorithm,
Speaker 1: but on account of dubious input factors like gender or
Speaker 1: race or religion, you've created something that's unconstitutional. That's Jason
Speaker 1: tchet It. He introduced us to Glenn, and he's the
Speaker 1: founder of Justice Codes and a legal affairs writer for
Speaker 1: the American Bar Association Journal. There's this predisposition to believe
Speaker 1: that math doesn't carry all of the biases that humans do.
Speaker 1: It's an objective science. I think we need to dispel
Speaker 1: that idea. Jason is describing the very human habit of
Speaker 1: taking computer output as gospel truth. It's called automation bias,
Speaker 1: and it's why parole boards often don't feel comfortable over
Speaker 1: writing algorithms like Compass, and why some people follow their
Speaker 1: GPS even when it has them driving into the ocean.
Speaker 1: This idea that somehow, because math is an underlying force
Speaker 1: to these tools, makes them more objective or beyond certain
Speaker 1: types of scrutiny is wrong. Computer algorithms are being used
Speaker 1: to determine human fate today, whether it's Compassed in the
Speaker 1: US or the social credit score in China, So we
Speaker 1: have to scrutinize them and understand that their output is
Speaker 1: not necessarily neutral. The foundational principle of AI is using
Speaker 1: historical data to predict what will happen next, and that
Speaker 1: in itself is a challenge to our culture because the
Speaker 1: American dream is built on the idea that we have
Speaker 1: a capacity to change, that we can move from rags
Speaker 1: to riches, from the penitentiary to the boardroom. And it's
Speaker 1: not just an American narrative. It's Scrooges change of heart
Speaker 1: delivering the turkey to the Cratchets on Christmas Day. It's
Speaker 1: Saint Paul's conversion on the Road to Damascus. It's at
Speaker 1: the very heart of Western culture. But algorithms like Compass
Speaker 1: aren't built to see the potential in people. They're designed
Speaker 1: to calculate risk based on past actions, and Compass isn't
Speaker 1: the only example of algorithms being used in our criminal
Speaker 1: justice system. When we come back, we go right inside
Speaker 1: the NYPD to understand how new technology is powering policing.
Speaker 1: It's a freezing cold day in New York City when
Speaker 1: we arrive at the NYPD headquarters, and before we even
Speaker 1: get into the main building, Julie and I have to
Speaker 1: pass through airport style security and naturally give up some data,
Speaker 1: including submitting a selfie in a kiosk. Yes, right now,
Speaker 1: our society is holding big conversations about body cameras, police accountability,
Speaker 1: and government monitoring, so we had to ask how does
Speaker 1: one of the most recognizable police forces in the world
Speaker 1: handle our data. My name is Benjamin Singleton, director of
Speaker 1: Analytics at the NYPD, so I probably spend half my
Speaker 1: day writing code and half my day and meetings. The
Speaker 1: police Department collects records as a regular course of our business.
Speaker 1: We respond to nine when one calls, We take crime reports,
Speaker 1: we make arrests, We issue moving summons. Is when you
Speaker 1: you know, speed in the city. These are examples of
Speaker 1: the kind of data that we collect. You know, I
Speaker 1: think there's probably some sentiment that there are back doors
Speaker 1: into various systems UM, but the NYPD is governed by
Speaker 1: the same legal processes as any other UM law enforcement agency.
Speaker 1: If we want data from an outside company or vendor,
Speaker 1: we get a search warrant from a judge or through
Speaker 1: a d a's office issue with subpoena, and that's how
Speaker 1: we collect our data. Their cameras throughout the subways, white
Speaker 1: corset turnstyles, and easy pause readers on the roads and
Speaker 1: more in New York. So what might the NYPD know
Speaker 1: about me? If you haven't sort of stood in front
Speaker 1: of a police officer who hasn't taken a report by hand,
Speaker 1: we probably don't have records on you. That being said,
Speaker 1: we do collect data through sensors like license plate readers UM,
Speaker 1: and we do have data sharing agreements with some other
Speaker 1: criminal justice agencies like corrections, like the courts, and so
Speaker 1: there's obviously opportunities for that kind of data to enter
Speaker 1: our realm. But one thing that's built into every single
Speaker 1: NYPD application is an auditing track. So anytime you look
Speaker 1: at any piece of information, no matter what system you're in,
Speaker 1: that's being audited. And so we have a very large
Speaker 1: internal affairs bureau, and people have gotten in trouble before
Speaker 1: for misuse of computer systems, and so I think that
Speaker 1: that's an important check that's reassuring to hear. But why
Speaker 1: collects so much data in the first place. I think
Speaker 1: that the next frontier of machine learning in policing is
Speaker 1: bringing decisions and information into the hands of cops who
Speaker 1: need to make decisions quickly. We recently rolled out tens
Speaker 1: of thousands of mobile phones to all of our cops,
Speaker 1: and putting a computer in their hands has really changed
Speaker 1: the way that the police. When you have more information,
Speaker 1: you can make better decisions. So we could be responding
Speaker 1: to a job at a specific location in a building,
Speaker 1: and we know what's happened at that building before we
Speaker 1: responded to nine. When one calls their last week in apartment,
Speaker 1: you know for see and in that interaction it led
Speaker 1: to some sort of altercation or we found out that
Speaker 1: that person that we interacted have had some sort of issue. Well,
Speaker 1: the cop who's working today might not be the same
Speaker 1: cop who is working a week ago. And so how
Speaker 1: do I convey that information? Maybe in a phone as
Speaker 1: a as a pop up, as a notification that tells
Speaker 1: you take extra time, take caution. Um, this sort of
Speaker 1: incident happened. Using data to give office as context is
Speaker 1: hard to argue against if it can lead to safer
Speaker 1: interactions for everyone. But of course, what many people find
Speaker 1: more concerning is ambient surveillance. Surveillance that happens all the time,
Speaker 1: and despite much pressure, the NYPD has yet to release
Speaker 1: an explicit facial recognition policy. And where do the effort
Speaker 1: stand on facial recognition technology. Our Facial Identification Section, which
Speaker 1: sits under the Detective Bureau, is a group of trained
Speaker 1: detectives investigators. They use a tool and algorithm that compares
Speaker 1: faces that we might get from a surveillance photo, and
Speaker 1: they run that algorithm, get potential matches and then conduct
Speaker 1: an investigation. But it's not as simple as you know,
Speaker 1: a facial recognition hit occurs and that's suddenly licensed to
Speaker 1: make an arrest. It doesn't generate probable cause for us.
Speaker 1: We still require much more evidence in order to make
Speaker 1: a determination that that hit is truly viable and something
Speaker 1: we can act on. But there are cases where that
Speaker 1: technology has been used as part of an arrest or prosecution.
Speaker 1: In the absence of an explicit policy, Ben wasn't able
Speaker 1: to answer the question live in the room, but we
Speaker 1: did get a statement from the m I p D.
Speaker 1: The NYPD has moved deliberately and responsibly in the use
Speaker 1: of facial recognition software. There is no NYPD case where
Speaker 1: an arrest or prosecution was brought on the basis of
Speaker 1: facial recognition. The NYPD uses it on a case by
Speaker 1: case basis, and the case must always be supported by
Speaker 1: further investigation before any arrest is made. The NYPD has
Speaker 1: absolutely no interest in wholesale surveillance, which would be an
Speaker 1: enormous and entirely pointless task. We have little choice but
Speaker 1: to trust. But that said, Bend is beat convincingly about
Speaker 1: how the NYPD actually uses technology to police themselves. There's
Speaker 1: also statistical tools around fairness that can actually measure whether
Speaker 1: an algorithm is fair, whether it's causing bias, etcetera. And
Speaker 1: so we're very interested in utilizing these metrics and we
Speaker 1: fully embrace them. We we want to get better, and
Speaker 1: we're taking a conservative approach because we know how high
Speaker 1: stakes this is. The stakes are high and the path
Speaker 1: is murky. I didn't know what to expect at the NYPD.
Speaker 1: Would they optimize purely for reducing crime or they take
Speaker 1: a broader view of justice. Personally, I found ben reassuring,
Speaker 1: but the potential for abuse remains. So how do we
Speaker 1: here in America god against that abuse. Well, let's return
Speaker 1: to Mary Haskett who found a blink identity with Alex Kilpatrick.
Speaker 1: Anytime you're using face wreck without consent, it's going to
Speaker 1: get abused because why wouldn't it. And and here's the problem.
Speaker 1: I don't think it's appropriate to ask a police department
Speaker 1: to just voluntarily not use a tool that's awesome for them.
Speaker 1: I mean you need to have a different level. You
Speaker 1: need to have your governor, federal states, some government governing
Speaker 1: body needs to be saying sorry, this is not appropriate.
Speaker 1: This is violating people's rights. The difference between what is
Speaker 1: happening in China and in the US is not technological,
Speaker 1: it's cultural and political. Edward Snowden had a phrase for this,
Speaker 1: turnkey tyranny, meaning that the technical infrastructure of mass surveillance
Speaker 1: already exists, and that we're only protected by our values
Speaker 1: and our laws. And thinking of China, I think there
Speaker 1: are some profoundly creepy things that we are right on
Speaker 1: the edge of starting to see. There's cameras everywhere. If
Speaker 1: you add face recognition. It's not just oh, they saw
Speaker 1: my face, they saw that I went to Starbucks. It's
Speaker 1: where you were every day, every time, all of her
Speaker 1: history and all that gets saved. It's my pattern of
Speaker 1: where I go when I'm outdoors forever. Five years ago,
Speaker 1: I would have said that could never happen here. Part
Speaker 1: of the reason Mary has so much about privacy is
Speaker 1: that she knows how quickly facial recognition is spreading. In fact,
Speaker 1: in twenty eight Blink Identity raise money from Live Nation
Speaker 1: ticket Master's parent company to allow future concert goers to
Speaker 1: use their faces instead of their tickets. We wanted to
Speaker 1: be a case study of how to do this in
Speaker 1: a way that preserves individual privacy and respects the individual,
Speaker 1: and maybe that will help set a precedence and maybe
Speaker 1: some of these other objectionable use cases just won't be
Speaker 1: able to take off. Facial recognition and other AI technologies
Speaker 1: are being developed all over the world, and we can't
Speaker 1: trust everyone to be as conscientious as Mary and Alex
Speaker 1: In America, the liberty we take for granted is hard
Speaker 1: one and fragile, and cases like Glens show what can
Speaker 1: happen when algorithms are blind trusted to determine outcomes. So
Speaker 1: much hangs in the balance right now about our technological future,
Speaker 1: and the decisions we take will affect our lives profoundly
Speaker 1: and echo through the lives of our children too. I
Speaker 1: mentioned Charles Dickens Christmas Carol earlier to me. One of
Speaker 1: the most powerful scenes in the book is Scrooge seeing
Speaker 1: for the first time the chains he has made for
Speaker 1: himself through his own decisions. Nowadays, we would call those
Speaker 1: decisions and those chains longitudinal data, and they'd be very
Speaker 1: hard to get rid of. They're the record that Lenn
Speaker 1: couldn't shake, that might deny a Chinese citizen a plane ticket,
Speaker 1: or deny you health insurance because of your social media activity.
Speaker 1: But data can also set us free. In the next episode,
Speaker 1: we investigate what's possible when our data is used to
Speaker 1: help us. From a dying man brought back to his youth,
Speaker 1: to movies and music that read our bodies while they play,
Speaker 1: and what happens when Alexa becomes part of the family.
Speaker 1: How long that Giuliana dived with m hmmm, I don't
Speaker 1: know that one. I am still learning more about dinosaurs
Speaker 1: us some dinosaur trivially. Sleepwalkers is a production of I
Speaker 1: Heart Radio and Unusual productions. For the latest AI news,
Speaker 1: live interviews, and behind the scenes footage, find us on Instagram,
Speaker 1: at Sleepwalker's podcast or at Sleepwalker's podcast dot com. Special
Speaker 1: thanks this episode to Laurie Arlam and Lucy Brady. Sleepwalkers
Speaker 1: is hosted by me Osbaloshen and co hosted by me
Speaker 1: Kara Price, with produced by Julian Weller with help from
Speaker 1: Jacopo Penzo and Taylor Koin. Mixing by Tristan McNeil and
Speaker 1: Julian Weller. Our story editor is Matthew Riddle. Recording assistance
Speaker 1: this episode from Dina Bridgett, Rachel London, and Phil Bodger.
Speaker 1: Sleepwalkers is Exactly You, produced by me Osloschen and Mangesh Hattikiler.
Speaker 1: For more podcasts from my Heart Radio, visit the I
Speaker 1: Heart Radio app, Apple Podcasts, or wherever you listen to
Speaker 1: your favorite shows.