WEBVTT - The Watchmen 0:00:04.600 --> 0:00:08.360 Sleepwalkers is a production of our heart Radio and unusual productions. 0:00:13.800 --> 0:00:16.920 Every single thing that you do on a social network 0:00:17.640 --> 0:00:22.720 or on a website is essentially recorded. How many pages 0:00:22.840 --> 0:00:25.119 you visited, what did you click on, how did you 0:00:25.160 --> 0:00:27.920 get to that website? What page did you leave on? 0:00:28.080 --> 0:00:30.760 How many photos have you ever uploaded? Where were those 0:00:30.760 --> 0:00:33.880 photos uploaded? How many places have you checked into? Who 0:00:33.920 --> 0:00:37.480 have you tagged? What photographs have you been tagged in with? Whom? 0:00:37.880 --> 0:00:41.200 Where were those photographs taken? Who's in your friendship circle? 0:00:41.360 --> 0:00:44.600 Who did you go to school with? There are algorithms 0:00:44.600 --> 0:00:48.960 and machine learning technologies that connect all of that data 0:00:49.000 --> 0:00:54.200 together and start to find patterns. That's Lisa italian Aretti speaking. 0:00:54.720 --> 0:00:58.520 She's a digital sociologist and tech ethics advocate with a 0:00:58.560 --> 0:01:01.880 big focus on data. I think some of the data 0:01:02.000 --> 0:01:07.600 that's quite concerning is facial recognition technology, where machines are 0:01:07.640 --> 0:01:13.080 being fed huge amounts of images pictures of people's faces, 0:01:13.120 --> 0:01:17.320 and that can come from dating websites, from photographs of 0:01:17.400 --> 0:01:21.640 you on Facebook or Instagram or anywhere on the Internet. 0:01:21.680 --> 0:01:23.640 You don't even have to upload it yourself. It could 0:01:23.640 --> 0:01:26.440 be somebody else that's uploaded it for you. With all 0:01:26.480 --> 0:01:28.959 that data out in the wild. All it takes is 0:01:29.000 --> 0:01:31.080 for somebody to suck it up and they can start 0:01:31.160 --> 0:01:35.360 connecting the dots. So there were researchers last year from 0:01:35.440 --> 0:01:41.440 Stanford University who without the user's permission, scraped thirty thousand 0:01:41.640 --> 0:01:45.920 photographs from a dating website that was public rights. So 0:01:46.360 --> 0:01:48.680 they took it that they could just use those photographs 0:01:48.720 --> 0:01:52.200 for research. Because it was a dating site, people will 0:01:52.240 --> 0:01:55.800 also asked the data points, um, what is your sexual orientation? 0:01:56.400 --> 0:02:00.240 So gay, straight, by right, and so they had all 0:02:00.280 --> 0:02:03.920 of that data connected, so they could then connect faces 0:02:03.960 --> 0:02:08.040 to sexual orientation and essentially they built an algorithm that 0:02:08.120 --> 0:02:11.560 they said could detect just by somebody's face if they 0:02:11.600 --> 0:02:15.920 were gay or straight or by the algorithm worked, it 0:02:15.960 --> 0:02:20.839 could predict sexual orientation from photographs with accuracy for men 0:02:21.000 --> 0:02:26.320 and accuracy for women based on just five photographs. And 0:02:26.400 --> 0:02:30.360 Stanford isn't the only place using facial recognition to categorize people. 0:02:30.880 --> 0:02:48.720 Governments are too all around the world. This is Sleepwalkers Welcome. 0:02:48.840 --> 0:02:51.840 I'm Azveloshin. In the last episode, we looked at what 0:02:51.919 --> 0:02:56.040 happens when artificial intelligence digests huge data sets to find 0:02:56.080 --> 0:03:00.920 patterns and make predictions, like what's the most typical move synopsis? 0:03:01.040 --> 0:03:05.440 Or is that moral cancerous? But advances in machine learning 0:03:05.480 --> 0:03:10.040 are also making us more and more legible. In today's episode, 0:03:10.040 --> 0:03:13.080 we ask what happens when we become the data set 0:03:13.440 --> 0:03:16.640 and the power to predict is turned on us. Here's 0:03:16.680 --> 0:03:21.560 Lisa again. The extra terrifying thing is why creates this 0:03:21.840 --> 0:03:26.000 kind of technology? Like? Who does it serve? Why do 0:03:26.080 --> 0:03:29.720 we need this? If you take this type of technology, 0:03:30.120 --> 0:03:35.840 feed it to a citywide CCTV surveillance system and say 0:03:35.880 --> 0:03:38.760 that you go to a place like Saudi Arabia where 0:03:38.840 --> 0:03:42.360 being gay is considered a crime, and suddenly you're just 0:03:42.400 --> 0:03:45.160 what pulling people off the street and arresting them because 0:03:45.800 --> 0:03:49.680 you're gay because the computer said so, so like now 0:03:49.720 --> 0:03:53.920 you're going to prison. This may sound like a terrifying 0:03:54.000 --> 0:03:59.880 Minority Report style future, but actually it's here today. I 0:04:00.080 --> 0:04:03.840 Cara Highs. So Lisa was talking about what happens when 0:04:04.000 --> 0:04:07.000 governments start to connect the dots of all this data, 0:04:07.240 --> 0:04:10.880 but it's already happening with private enterprise. For example, insurance 0:04:10.880 --> 0:04:14.119 companies can now see someone who joins a Facebook group 0:04:14.200 --> 0:04:17.880 about a genetic mutation and use that data to guess 0:04:18.120 --> 0:04:20.680 that that person may have the genetic mutation and the 0:04:20.720 --> 0:04:24.520 condition that's associated, and then the computer says, let's raise 0:04:24.560 --> 0:04:27.200 their premium or let's deny them care. And so we 0:04:27.240 --> 0:04:30.599 can use this peroxy data to do something The New 0:04:30.680 --> 0:04:34.480 York Times has recently called proxy discrimination. And I think 0:04:34.520 --> 0:04:39.240 something that's even more widely applicable is this definition of 0:04:39.480 --> 0:04:43.640 surveillance capitalism, which is essentially that data is not just 0:04:43.760 --> 0:04:48.400 data anymore, it's money. Companies can use data to make 0:04:48.640 --> 0:04:53.160 predictions about future behavior and that can make them profit. Right, 0:04:53.240 --> 0:04:57.719 that surveillance capitalism. I using information about joining a Facebook 0:04:57.760 --> 0:05:00.839 group to make an insurance decision, and it's I don't know, 0:05:00.880 --> 0:05:02.840 it's a little bit scary. In the US, it's all 0:05:02.839 --> 0:05:06.279 about capitalism, But in other countries surveillance is used for 0:05:06.440 --> 0:05:10.960 other purposes. For example, in China, it's about social control. 0:05:11.320 --> 0:05:14.360 But in China, this same massive ingestion of data and 0:05:14.440 --> 0:05:18.440 statistical modeling is used for governance. So let's take a 0:05:18.440 --> 0:05:21.520 closer look at China and how they're using technology to 0:05:21.600 --> 0:05:26.400 amplify the power of the state. They have this notion 0:05:26.400 --> 0:05:28.040 of a social credit score, which is how good of 0:05:28.040 --> 0:05:30.279 a citizen you are according to the government. They have 0:05:30.520 --> 0:05:32.760 literally hundreds of millions of cameras around and they can 0:05:32.800 --> 0:05:35.120 basically do things like you've been out at the bar, 0:05:35.320 --> 0:05:37.200 you know, until to the last couple of nights. That's 0:05:37.200 --> 0:05:39.359 not really what a good upstanding citizen would do. So 0:05:39.400 --> 0:05:41.920 your social credit score can get ding because you've been 0:05:41.920 --> 0:05:44.400 spending too much time at night at a bar. That's 0:05:44.480 --> 0:05:48.440 Dr Alex Kilpatrick. He and Mary Haskett co founded Blink Identity, 0:05:48.600 --> 0:05:52.080 a facial recognition company that can recognize the users face 0:05:52.240 --> 0:05:54.800 in yes, the blink of an eye about no point 0:05:54.880 --> 0:05:59.000 four seconds. Here's Blink co founder Mary on Chinese surveillance. 0:05:59.360 --> 0:06:02.200 They have camera, so you're recorded jaywalking, and so your 0:06:02.240 --> 0:06:04.839 score goes down, you know, and you automatically get a ticket, 0:06:04.880 --> 0:06:07.000 which again doesn't sound like that they give a deal 0:06:07.080 --> 0:06:08.960 if you really believe in law and order, except if 0:06:09.000 --> 0:06:11.440 your score isn't high enough, you can't buy a plane ticket, 0:06:11.480 --> 0:06:13.280 you have to travel by bus, you know, you can't 0:06:13.320 --> 0:06:15.960 live in certain areas. And you can obviously see how 0:06:16.000 --> 0:06:18.120 this could be abused. I mean, it doesn't take much 0:06:18.120 --> 0:06:21.320 of an imagination. Other factors that can bring down your 0:06:21.360 --> 0:06:24.320 social credit score include what you buy at the store, 0:06:24.680 --> 0:06:27.840 your online browsing, and even having a friend with a 0:06:27.839 --> 0:06:30.920 low score. This use of generalized surveillance can keep a 0:06:30.960 --> 0:06:33.320 whole population in check, which is more or less the 0:06:33.360 --> 0:06:36.440 explicit goal of the Communist Party of China. This can 0:06:36.480 --> 0:06:40.280 be stifling for the average hand Chinese citizen. For minorities, 0:06:40.440 --> 0:06:43.080 it can be much much worse, right, you know, more 0:06:43.120 --> 0:06:47.120 specifically and more dauntingly, it can be used for internment, 0:06:47.279 --> 0:06:49.680 you know, in the case of the weaker minority in China, 0:06:50.240 --> 0:06:53.880 using data that is being created by this minority to 0:06:54.040 --> 0:06:57.560 just communicate, you know, one person communicating with another person, 0:06:57.720 --> 0:07:00.640 those who are connected that both wagas right, five weeks 0:07:00.640 --> 0:07:02.719 are gathering the same place. Now that we know that, 0:07:03.440 --> 0:07:05.240 you know, what are we going to do with that data? Oh, 0:07:05.240 --> 0:07:08.040 We're going to send these people to re education camps. 0:07:08.520 --> 0:07:12.120 And as technology improves, so does the state's ability to 0:07:12.200 --> 0:07:16.480 project power. China today is cleaning the floor with the 0:07:16.480 --> 0:07:22.200 Americans on voice and facial recognition technology. That's the Embrema, 0:07:22.280 --> 0:07:25.160 an expert on global political risk and the founder of 0:07:25.160 --> 0:07:29.440 the Eurasia Group. The Chinese have much more data. You 0:07:29.560 --> 0:07:32.720 also have a government that is consolidating the data and 0:07:32.880 --> 0:07:35.560 allocating it for different types of purposes. And you have 0:07:35.600 --> 0:07:39.280 no presumption of privacy whatsoever. With no presumption of privacy, 0:07:39.440 --> 0:07:41.800 the amount of data you can collect from your citizens 0:07:41.880 --> 0:07:46.760 grows exponentially, and that actually gives you a huge technological advantage. 0:07:47.040 --> 0:07:49.880 This is how Kai Fu Lee explains it. What makes 0:07:49.880 --> 0:07:53.240 sense AI algorithm work better is how much data you 0:07:53.360 --> 0:07:56.440 used to train it. And that's the beauty of deep learning. 0:07:56.640 --> 0:07:58.720 You just keep throwing data at it and it just 0:07:58.840 --> 0:08:02.880 performs better. And Kai fully understands this world better than most. 0:08:03.200 --> 0:08:06.360 His fund, Signivation Ventures, has invested in meg v, a 0:08:06.400 --> 0:08:10.960 facial recognition company valued at four billion dollars. Kaifu also 0:08:11.000 --> 0:08:15.840 around Google China, so he knows the landscape. China simply 0:08:15.960 --> 0:08:19.080 has more data than the US due to not only 0:08:19.080 --> 0:08:22.080 the large number of users, but also the depth in 0:08:22.120 --> 0:08:25.960 which Chinese users use the Internet for ordering food, for 0:08:26.000 --> 0:08:30.960 shared bicycles, for mobile payments. So the AI will actually 0:08:30.960 --> 0:08:33.480 just perform better because it's trained out more data. Some 0:08:33.600 --> 0:08:37.559 of that data is taken from citizens using surveillance, but 0:08:37.640 --> 0:08:42.120 according to Kaifu, much is freely given. I think the 0:08:42.240 --> 0:08:46.800 Chinese culture and the Chinese people are more pragmatic, so 0:08:46.920 --> 0:08:53.120 that if the software delivers pragmatic value, they ask fewer questions. 0:08:53.559 --> 0:08:57.080 For example, you know, we've funded an app that loans 0:08:57.120 --> 0:09:00.280 money to ask you a couple of questions and takes 0:09:00.360 --> 0:09:02.280 data from your phone at the same level as the 0:09:02.320 --> 0:09:05.600 Facebook would take data from your phone, and it's SAPs 0:09:05.640 --> 0:09:08.080 the money to you instantly if it decides to lend 0:09:08.120 --> 0:09:10.400 money to you. I think in the US people my 0:09:10.559 --> 0:09:12.840 question do I really want to give my data to 0:09:12.920 --> 0:09:17.200 a lending application? And is it appropriate to consider the 0:09:17.280 --> 0:09:19.360 makeup my phone as a part of that give me 0:09:19.559 --> 0:09:22.880 giving me money or my zip codes? Because that might 0:09:22.960 --> 0:09:27.000 reflect certain things about me. The breadth and depth of 0:09:27.080 --> 0:09:30.360 data in China, both from a larger population and a 0:09:30.440 --> 0:09:35.480 much deeper integration with technology, gives China a serious competitive advantage. 0:09:35.720 --> 0:09:39.520 Where the Americans have much better scientists, the Chinese were 0:09:39.520 --> 0:09:41.600 able to buy a lot of science. Now, when you 0:09:41.679 --> 0:09:44.360 talk to specialists in this field, they will tell you 0:09:44.400 --> 0:09:48.680 that in many parts of Ai, great data and okay 0:09:48.679 --> 0:09:54.280 scientists will frequently beat great scientists and okay data. Character 0:09:54.320 --> 0:09:57.880 scary thing is that China is using that technological superiority 0:09:57.920 --> 0:10:00.280 to build a very different kind of state, you know, 0:10:00.320 --> 0:10:03.559 one in which the price of descent is intolerably high. 0:10:03.960 --> 0:10:08.200 You know. So according to Ian public demonstrations have fallen marketly, right, 0:10:08.200 --> 0:10:11.080 because if you know you're being watched, you're probably less 0:10:11.120 --> 0:10:14.480 likely to commit public displays of civil disobedience. Right. We 0:10:14.559 --> 0:10:17.360 talked about the vigas earlier in Cashgar, which is a 0:10:17.400 --> 0:10:20.640 Muslim city in China. Vigas have to register to go 0:10:20.720 --> 0:10:24.560 into the mosque, and once they're inside, they face a 0:10:24.640 --> 0:10:28.520 bank of cameras like many many surveillance cameras, and go 0:10:28.679 --> 0:10:33.360 figure Muslims stopped going to mosque voluntarily well, because going 0:10:33.360 --> 0:10:37.160 to mask is an act of civil disobedience where they are. 0:10:37.280 --> 0:10:40.920 Even if it's not explicitly stated, it's it's heavily implied, right. 0:10:41.000 --> 0:10:42.920 I mean, I think about it for myself, Like if 0:10:42.960 --> 0:10:45.320 I were living in an area in the United States 0:10:45.320 --> 0:10:47.840 where going to temple was going to land me in 0:10:47.880 --> 0:10:51.160 an internment camp, I would not be going voluntarily if 0:10:51.200 --> 0:10:54.559 I knew there were security cameras all over my temple. Absolutely, 0:10:54.640 --> 0:10:57.080 And the crazy thing is you wouldn't even have to 0:10:57.160 --> 0:10:59.880 know if those security cameras work. It's like on the 0:11:00.000 --> 0:11:01.480 o TO in England, we have a lot of speed 0:11:01.520 --> 0:11:03.920 cameras and no one knows that they're actually doing it. 0:11:03.960 --> 0:11:05.959 It's staying apart from once every five years you've got 0:11:05.960 --> 0:11:09.240 a ticket. But it still slows people down the panopticon 0:11:09.880 --> 0:11:12.720 even if they can't do that. But people think they 0:11:12.720 --> 0:11:14.599 can do that, right. I mean, you don't need a 0:11:14.679 --> 0:11:17.319 hundred percent certainty. You just need a government that is 0:11:17.360 --> 0:11:19.680 starting to get that capacity and make it known and 0:11:19.760 --> 0:11:21.400 have a few people that are sort of strung up 0:11:21.400 --> 0:11:26.480 as examples, and suddenly everyone's scared. And this isn't only 0:11:26.520 --> 0:11:29.840 happening in China. According to Ian, it was a key 0:11:29.880 --> 0:11:33.280 part of Bashar al Assad's strategy in the Syrian Civil War. 0:11:33.600 --> 0:11:36.520 Assade got some help from the Russians, who gave them 0:11:36.559 --> 0:11:39.600 a couple of hundred computer scientists to go in work 0:11:39.640 --> 0:11:43.480 with the Syrian military and identify on social media and 0:11:43.480 --> 0:11:47.720 on text messaging who wore those Syrian citizens that were 0:11:47.840 --> 0:11:52.640 nodes of dissent and within six months no more moderate 0:11:52.640 --> 0:11:58.720 opposition in Syria. They specifically, we're looking into individual Syrian 0:11:58.800 --> 0:12:02.240 citizens that we're saying things about the regime that we're untoward, 0:12:02.480 --> 0:12:06.520 that we're connected to influencers that were helping to organize protests, 0:12:06.720 --> 0:12:10.440 and suddenly, you know, a bunch of those people were 0:12:10.559 --> 0:12:13.520 rounded up and some were never heard from again. And 0:12:13.760 --> 0:12:16.640 as I mentioned in terms of China, you don't have 0:12:16.720 --> 0:12:19.800 to do that with many people before people start ratting 0:12:19.840 --> 0:12:22.560 out their friends, being scared of talking to anyone not 0:12:22.679 --> 0:12:27.959 going out. The system worked. We may feel comfortably far 0:12:28.040 --> 0:12:31.080 from the battlefields of Syria here in the US, and 0:12:31.120 --> 0:12:34.559 from the overwhelming number of surveillance cameras on practically every 0:12:34.559 --> 0:12:38.400 street corner in China. But the more effective these technologies are, 0:12:38.840 --> 0:12:41.720 the more likely they are to be adopted by others. Now, 0:12:41.760 --> 0:12:44.840 in other countries, you're going to have a confluence of 0:12:44.880 --> 0:12:48.720 both them liking the model and the Chinese directly exporting it. 0:12:49.000 --> 0:12:51.720 Who were those countries, Well, look at One Belt, One Road, 0:12:52.000 --> 0:12:55.120 the you know, trillion plus dollar investments that the Chinese 0:12:55.120 --> 0:12:58.720 are making all over the world in Pakistan, Southeast Asia, 0:12:58.760 --> 0:13:01.640 you know, Cambodia, laughs, a whole bunch of countries. And 0:13:01.840 --> 0:13:04.760 when you look at those countries and you see that 0:13:04.800 --> 0:13:08.000 the Chinese are providing the money in this conditionality in return, 0:13:08.080 --> 0:13:11.520 some that conditionalities use Chinese standards for technology that's in 0:13:11.640 --> 0:13:14.400 many of these contracts. And with the spread of Chinese 0:13:14.400 --> 0:13:18.040 standards of technology comes the spread of Chinese style surveillance, 0:13:18.080 --> 0:13:21.439 which could ultimately make the whole world trend more authoritarian. 0:13:22.080 --> 0:13:26.120 So as that happens, these governments are going to say, 0:13:26.360 --> 0:13:29.520 ah ha, we get the money from China, we use 0:13:29.600 --> 0:13:33.720 their technology. We're stuck with their system, but we can 0:13:33.920 --> 0:13:37.319 use it to ensure that our people stay in power. Again, 0:13:37.800 --> 0:13:41.160 it's easy to let all of this feel comfortably far away, 0:13:41.200 --> 0:13:44.680 but remember the Internet doesn't have borders, so you don't 0:13:44.679 --> 0:13:46.800 have to be in China for the Chinese state to 0:13:46.840 --> 0:13:49.280 access your data. Yeah. I don't know how many people 0:13:49.320 --> 0:13:52.440 know this, but Grinder is actually owned by a Chinese company, 0:13:52.640 --> 0:13:55.280 Grinder the dating MP. Yes, and actually there have been 0:13:55.400 --> 0:13:58.600 articles about the fact that the US government is trying 0:13:58.640 --> 0:14:01.320 to force China's hands so that we can buy it 0:14:01.400 --> 0:14:05.720 back because there's so much user data that this company 0:14:05.760 --> 0:14:08.760 now owns. Well, that Grinder use a data is basically 0:14:08.760 --> 0:14:11.280 being seen by the US government as a strategic asset. 0:14:11.679 --> 0:14:13.480 It is a strategic asset. I mean, if you think 0:14:13.480 --> 0:14:16.920 about it, if somebody is on a military base or 0:14:16.960 --> 0:14:21.600 in a barrack and trying to connect with someone on Grinder, 0:14:21.800 --> 0:14:24.400 they're turning their location services data on because they want 0:14:24.400 --> 0:14:26.760 to see people in their area and if they're turning 0:14:26.800 --> 0:14:31.080 that location services data on, they're basically making themselves vulnerable 0:14:31.560 --> 0:14:34.320 to the company that owns the data, because it's basically 0:14:34.320 --> 0:14:36.720 saying here I am, here, I am and not only that, 0:14:36.840 --> 0:14:39.760 here I am, here, I am, and I'm gay, and 0:14:39.920 --> 0:14:43.400 that can lead to some possibilities of blackmail. Even today, exactly, 0:14:43.720 --> 0:14:45.920 there was an article in the interface that was written 0:14:45.920 --> 0:14:49.400 by this guy, Casey Newton, chat history, photos, videos, real 0:14:49.400 --> 0:14:52.760 time location. All of that is connected to a user's 0:14:52.760 --> 0:14:56.040 email address, and that means that the user's identity can 0:14:56.040 --> 0:14:59.880 be very easily learned. That's pretty scary. And even if 0:15:00.040 --> 0:15:03.840 don't use Grinder, you might use Flash of Clans or Fortnite, 0:15:03.840 --> 0:15:07.200 which are very popular gaming apps also owned by the Chinese. 0:15:07.200 --> 0:15:09.720 Now we keep saying the Chinese. To be clear, these 0:15:09.760 --> 0:15:13.080 apps aren't owned by the Chinese government, that owned by 0:15:13.200 --> 0:15:16.320 Chinese companies. What the actions of the US government imply 0:15:16.520 --> 0:15:19.440 by trying to force this company to sell Grinder back 0:15:20.200 --> 0:15:23.000 is that they don't believe in the distinction. And do 0:15:23.040 --> 0:15:24.960 you think the US government would really be working that 0:15:25.080 --> 0:15:29.320 hard to get back a gay dating app if they 0:15:29.360 --> 0:15:32.240 didn't think that there was not a murky separation between 0:15:32.240 --> 0:15:36.720 the government and companies in China. Right, So every time 0:15:36.760 --> 0:15:39.000 we give data away, I mean, we're aware that it 0:15:39.040 --> 0:15:41.560 opens us up to targeted ads on Facebook. We talked 0:15:41.560 --> 0:15:45.400 about those with Gillian. We're not aware that that data 0:15:45.440 --> 0:15:50.040 may end up in the hands of potentially hostile foreign government. 0:15:50.160 --> 0:15:55.120 So once again, Sleepwalking. We've been talking about how foreign 0:15:55.160 --> 0:15:58.120 governments are using AI, but when we come back, we'll 0:15:58.120 --> 0:16:00.080 look at how the police and courts are using it 0:16:00.280 --> 0:16:09.320 at home in America. Kara, it's easy to look at 0:16:09.440 --> 0:16:12.200 China and to see the big bad wolf. They're using 0:16:12.240 --> 0:16:16.440 surveillance technology for the wholesale suppression of an ethnic minority. 0:16:16.280 --> 0:16:18.880 They have a social credit score that con emit access 0:16:18.880 --> 0:16:23.160 to opportunities and even travel, But algorithms also determined outcomes 0:16:23.360 --> 0:16:26.240 here in the US exactly if you think about it, 0:16:26.360 --> 0:16:29.160 we do use social ratings. If you use Uber and 0:16:29.240 --> 0:16:31.280 have a rating lower than four out of five stars, 0:16:31.360 --> 0:16:33.560 you can't get a car right and you can't get 0:16:33.560 --> 0:16:36.080 alone if you have a low FICO credit score. And 0:16:36.400 --> 0:16:40.320 our criminal justice system also uses algorithmic ratings to decide 0:16:40.360 --> 0:16:45.400 people's fate. When I got arrested at sixteen, I was 0:16:45.440 --> 0:16:48.120 in high school and John F. Kennedy High School. That's 0:16:48.160 --> 0:16:51.480 Glenn Rodriguez. When he was a baby, Glenn's mother was murdered, 0:16:51.520 --> 0:16:54.920 and when he was three, his father committed suicide. From 0:16:54.960 --> 0:16:57.920 then on, Glenn was raised by his grandmother and he 0:16:58.000 --> 0:17:02.200 searched for belonging. The kid who wants to feel accepted, 0:17:02.440 --> 0:17:05.040 wants to feel a part of something right. Whatever the 0:17:05.080 --> 0:17:08.080 group was up for, I was down. And if they 0:17:08.119 --> 0:17:11.199 were going one step forward, I would take two. We 0:17:11.240 --> 0:17:14.760 pretty much planned a robbery at a car dealership in 0:17:14.840 --> 0:17:20.240 Queen's and we entered the premises. We took three cars. 0:17:20.720 --> 0:17:24.240 There was a twenty five year old men in there, 0:17:24.920 --> 0:17:27.520 and he initially pulled a gun, and so I had 0:17:27.520 --> 0:17:32.000 a gun and I shot him. Glenn was arrested and 0:17:32.080 --> 0:17:35.679 convicted of second degree murder. He was sentenced to twenty 0:17:35.720 --> 0:17:38.560 five years in jail, and he was still a high schooler. 0:17:38.760 --> 0:17:41.720 You feel powerless, you feel hopeless, especially at that age. 0:17:41.880 --> 0:17:44.159 So the way I saw it was, this is my life. 0:17:44.400 --> 0:17:46.320 You know, I'm probably gonna die in jail, and so 0:17:46.440 --> 0:17:48.080 whatever it is that I have to do, I need 0:17:48.160 --> 0:17:50.199 to survive. One of the things that I learned very 0:17:50.280 --> 0:17:53.399 quickly is that in prison, one of the only things 0:17:53.440 --> 0:17:55.919 that is respected is violence, and so in order for 0:17:56.000 --> 0:17:58.479 you to survive in there, you have to be violent, 0:17:58.920 --> 0:18:02.200 because otherwise you'd be come pray. In time, Glenn established 0:18:02.200 --> 0:18:06.320 his reputation and started to feel safer. With that security 0:18:06.359 --> 0:18:09.879 and getting older, his thinking began to change. And it 0:18:09.960 --> 0:18:12.560 wasn't until later, to like my mid twhenies, when I 0:18:12.640 --> 0:18:15.119 started saying, you know what, I need to reverse this 0:18:15.240 --> 0:18:17.879 trend if I want to have any chance at parole. 0:18:18.240 --> 0:18:22.120 Glenn had to reverse thirteen years of behavior to survive 0:18:22.119 --> 0:18:24.720 in prison. He had learned to behave one way, but 0:18:24.800 --> 0:18:27.760 to get out he had to behave another. I reveiled 0:18:27.800 --> 0:18:30.399 myself of the Puppies behind Bars program, so I was 0:18:30.520 --> 0:18:33.399 training service dogs for wounded war veterans for five years. 0:18:33.720 --> 0:18:36.959 That was an amazing experience, right because throughout incarceration it's 0:18:36.960 --> 0:18:40.840 almost like you build a wall around yourself with the dogs. 0:18:41.280 --> 0:18:43.440 You can't fake it with the dogs. If you're trying 0:18:43.480 --> 0:18:45.800 to like teach them a command, sometimes you may have 0:18:45.880 --> 0:18:48.679 to be silly, which guess what, in prison, being silly 0:18:48.720 --> 0:18:51.240 is not acceptable. That's perceived as a weakness. But with 0:18:51.320 --> 0:18:53.960 that program, often times you had the resort of being 0:18:53.960 --> 0:18:57.160 silly and throwing yourself on the floor and giggling loud 0:18:57.160 --> 0:18:59.159 and making all kinds of crazy sounds to try to 0:18:59.160 --> 0:19:01.840 get the dogs attend gin Right. To a very large extent, 0:19:01.920 --> 0:19:04.160 I believe that that program kind of helped me regain 0:19:04.200 --> 0:19:08.840 my humanity as well as helping Glenn. Personally. Taking part 0:19:08.880 --> 0:19:11.400 in prison programs for the public good is looked upon 0:19:11.480 --> 0:19:14.520 favorably by parole boards. So everything I did, I wanted 0:19:14.560 --> 0:19:17.239 to document the kind of showcases what I've done, this 0:19:17.320 --> 0:19:20.280 is who I am today. As part of the process, 0:19:20.320 --> 0:19:25.080 there's also the Campus Risk Assessment COMPUS stands for Correctional 0:19:25.160 --> 0:19:30.720 Offender Management Profiling for Alternative Sanctions. It's an algorithm that 0:19:30.880 --> 0:19:33.680 claims to be able to predict how likely a defendant 0:19:33.760 --> 0:19:36.359 is to commit another crime based on a list of 0:19:36.359 --> 0:19:39.760 a hundred and thirty seven questions. Since being developed in 0:19:41.200 --> 0:19:43.800 its estimated COMPASS has been used to assess more than 0:19:43.880 --> 0:19:48.639 one million defendants, including Glenn. You meet with this person 0:19:48.800 --> 0:19:51.640 a few months before your schedule parole board day, and 0:19:52.320 --> 0:19:54.640 they asked you a series of questions. And so when 0:19:54.640 --> 0:19:58.680 he got to the disciplinary history section of the Campus 0:19:58.760 --> 0:20:01.960 Risk Assessment, there was a list of offenses right for 0:20:02.040 --> 0:20:03.920 him to check off. Yes or no for the past 0:20:03.920 --> 0:20:06.280 twenty four months, and it was all known. And anyone 0:20:06.359 --> 0:20:08.880 who has any experience with prison would tell you that 0:20:08.880 --> 0:20:12.320 that is almost impossible to do, right, because misbehavior reports 0:20:12.560 --> 0:20:15.800 can be for something as simple as having too many pillows, 0:20:15.840 --> 0:20:18.440 something as simple as your parents hanging off your bot, 0:20:18.840 --> 0:20:21.720 your sneaker is untied. It takes a lot of energy 0:20:21.760 --> 0:20:24.760 to dodge a misbehavior report during the course of a year, 0:20:25.119 --> 0:20:27.959 let alone ten, and in my case, it had been eleven. 0:20:29.520 --> 0:20:31.720 And then I heard him read the question and he says, 0:20:31.760 --> 0:20:36.159 does this person appear to have notable disciplinary issues? And 0:20:36.200 --> 0:20:39.359 he says yes, And I was like, hold up, wait 0:20:39.400 --> 0:20:41.639 a second, did I just hear you right? Because I 0:20:41.760 --> 0:20:44.800 just heard you say that I have notable disciplinary issues? 0:20:45.040 --> 0:20:47.199 Do you do realize that I haven't had a misbehavior 0:20:47.200 --> 0:20:50.480 report in over a decade? Right? And his answer was well, 0:20:50.520 --> 0:20:53.920 I was told that if there's any instance of misbehavior 0:20:54.000 --> 0:20:56.359 at any point, I have to check yes for this answer. 0:20:56.520 --> 0:20:58.760 So I was like, okay, So at that point there 0:20:58.800 --> 0:21:04.119 was nothing I could do. I'm appearing before the parole 0:21:04.119 --> 0:21:07.479 board panel. I presented to them a portfolio that was 0:21:07.520 --> 0:21:12.800 approximately one hundred pages, had letters to support Now the 0:21:12.840 --> 0:21:16.800 Compass is saying that I'm a disciplinary issue and so 0:21:16.840 --> 0:21:19.399 I shouldn't be released. But I was denied because of 0:21:19.400 --> 0:21:21.800 the fact that I scored high on Compass. They played 0:21:21.800 --> 0:21:24.760 a safe and kept me in. It may have been 0:21:24.840 --> 0:21:27.640 less than five minutes the hearing. I waited twenty six 0:21:27.720 --> 0:21:30.120 years to sit in front of a panel of three 0:21:30.119 --> 0:21:33.000 people for less than five minutes. No one wants to 0:21:33.000 --> 0:21:35.399 be the one to go against Compass, and next you know, 0:21:35.480 --> 0:21:37.720 something goes wrong and now your job is on the 0:21:37.760 --> 0:21:41.840 line because you departed from Compass, which is taken as 0:21:42.080 --> 0:21:48.320 factual and scientific. In time, Glenn went before another parole 0:21:48.359 --> 0:21:52.120 board and this time they freed him against the recommendation 0:21:52.160 --> 0:21:54.439 of Compass. And now Glenn has built a life for 0:21:54.520 --> 0:21:58.119 himself working with teenagers at risk, giving conceration at the 0:21:58.160 --> 0:22:02.280 Center for Community Alternatives. But he's still being affected by 0:22:02.280 --> 0:22:06.760 the algorithm. Compass does not end upon your release because 0:22:06.800 --> 0:22:10.840 the same Compass risk assessment that's considered for your release. 0:22:10.960 --> 0:22:13.760 The term is how you're going to be supervised upon release. 0:22:14.000 --> 0:22:16.280 There's a number of restrictions that I have. I have 0:22:16.359 --> 0:22:21.800 a curve you. I'm still haunted by a Compass. Despite 0:22:21.800 --> 0:22:26.480 turning his life around Compass is still limiting Glen's freedom, 0:22:26.520 --> 0:22:29.680 and that should haunt all of us. According to propublic A, 0:22:29.760 --> 0:22:34.320 Compass inaccurately labels black defendants as likely to reoffend twice 0:22:34.359 --> 0:22:39.480 as often as white defendants. Algorithmic discrimination isn't government policy 0:22:39.520 --> 0:22:41.800 here in the US like it is against the weak 0:22:41.840 --> 0:22:46.320 as in China, but it still exists. There's this issue 0:22:46.680 --> 0:22:50.639 where you can have computer scientists building a more accurate algorithm, 0:22:50.960 --> 0:22:55.159 but on account of dubious input factors like gender or 0:22:55.280 --> 0:22:59.600 race or religion, you've created something that's unconstitutional. That's Jason 0:22:59.640 --> 0:23:02.600 tchet It. He introduced us to Glenn, and he's the 0:23:02.600 --> 0:23:05.840 founder of Justice Codes and a legal affairs writer for 0:23:05.880 --> 0:23:10.359 the American Bar Association Journal. There's this predisposition to believe 0:23:10.480 --> 0:23:15.840 that math doesn't carry all of the biases that humans do. 0:23:16.000 --> 0:23:18.840 It's an objective science. I think we need to dispel 0:23:19.080 --> 0:23:23.159 that idea. Jason is describing the very human habit of 0:23:23.200 --> 0:23:28.560 taking computer output as gospel truth. It's called automation bias, 0:23:28.680 --> 0:23:31.720 and it's why parole boards often don't feel comfortable over 0:23:31.760 --> 0:23:35.320