WEBVTT - A Solution for Algorithmic Bias 0:00:15.356 --> 0:00:21.756 Pushkin from Pushkin Industries. This is Deep Background, the show 0:00:21.796 --> 0:00:24.796 where we explore the stories behind the stories in the news. 0:00:25.116 --> 0:00:28.996 I'm Noah Feldman. How did you find this podcast? Did 0:00:28.996 --> 0:00:31.196 you see an ad for it on your phone? If so, 0:00:31.676 --> 0:00:33.596 that ad might have shown up for you, because, based 0:00:33.596 --> 0:00:36.316 on information about you that's out there on the Internet, 0:00:36.716 --> 0:00:40.396 a computer algorithm decided that this show might be the 0:00:40.516 --> 0:00:43.436 kind of thing that you would like. Algorithms like that 0:00:43.476 --> 0:00:47.716 are all around us. Some are far more consequential than others. 0:00:48.076 --> 0:00:50.156 But I'm glad you're listening to the show. But hey, 0:00:50.156 --> 0:00:53.076 if you're applying for credit, an algorithm could actually evaluate 0:00:53.076 --> 0:00:55.396 your credit worthiness, and the stakes are a little higher 0:00:55.436 --> 0:00:57.676 there than whether you're listening to this podcast or to 0:00:57.716 --> 0:01:00.516 Trevor Noah's. If I'm looking for a job, an algorithm 0:01:00.516 --> 0:01:03.076 could go through all of the job applicants to try 0:01:03.116 --> 0:01:06.036 to do a first cut before the employer decides who 0:01:06.076 --> 0:01:08.796 they're going to interview. In some cities, algorithms are even 0:01:08.836 --> 0:01:11.116 being used by the police to try to predict the 0:01:11.156 --> 0:01:13.436 probability that there's going to be a crime in a 0:01:13.476 --> 0:01:16.796 particular place and to decide where they're going to focus 0:01:16.996 --> 0:01:20.716 the police efforts and actually send the cops. Nicole Ternally 0:01:21.036 --> 0:01:24.316 is a fellow at Brooking Center for Technology Innovation. She's 0:01:24.316 --> 0:01:28.876 been studying how algorithms like this work and how they fail. 0:01:29.276 --> 0:01:32.636 She recently co wrote a report for Brookings about algorithmic bias, 0:01:32.996 --> 0:01:37.596 or in other words, how computers can be racist. So 0:01:37.636 --> 0:01:40.076 if we look at an algorithm like a black box, 0:01:40.476 --> 0:01:43.356 it starts with an input and it ends with an output. 0:01:43.756 --> 0:01:46.076 When it comes to the input, you know, big data 0:01:46.116 --> 0:01:49.876 has made it very easy to actually harness volumes of 0:01:49.956 --> 0:01:54.116 data they're about us, these reference points, about individuals, and 0:01:54.196 --> 0:01:56.596 to create you know, I think some input or what 0:01:56.636 --> 0:02:00.476 we call training data that essentially trains the algorithm to 0:02:00.716 --> 0:02:03.716 adapt to what our behaviors are. In many cases, what 0:02:03.756 --> 0:02:06.476 goes into the algorithm can be accurate. There are certain 0:02:06.516 --> 0:02:11.396 things that your listeners do online that are discrete, our objective, 0:02:11.596 --> 0:02:14.516 are true in terms of your search queries, in terms 0:02:14.556 --> 0:02:18.876 of your online profile. But when you have developers that 0:02:19.476 --> 0:02:22.956 put in training data that in some respects may be 0:02:23.116 --> 0:02:27.756 biased or skewed, it creates challenges or what technologists have 0:02:27.836 --> 0:02:30.996 called garbage in So what I mean by that, if 0:02:30.996 --> 0:02:33.396 you're developing an algorithm and this is actually a case 0:02:33.436 --> 0:02:35.796 So this is not something that we're making up. Like 0:02:35.876 --> 0:02:38.956 the Compass algorithm, which was designed to help judges make 0:02:38.996 --> 0:02:42.396 better predictions on the amount of time that a defendant 0:02:42.436 --> 0:02:46.476 should be detained before sentencing. And let's say the training 0:02:46.556 --> 0:02:51.116 data used to train that algorithm is based upon criminal 0:02:51.196 --> 0:02:55.556 justice stats or criminal behavior stats. It's no secret that 0:02:55.636 --> 0:02:59.716 in this country, African American men in particular are more 0:02:59.756 --> 0:03:03.476 likely to experience arrest. And if they are more likely 0:03:03.516 --> 0:03:07.996 to experience arrest, which oftentimes leads to incarceration, they will 0:03:08.156 --> 0:03:12.036 overwhelmingly make up the majority of the training data. So 0:03:12.076 --> 0:03:16.316 that input, when it gets to the output, it then 0:03:16.956 --> 0:03:22.836 may disproportionately affect African American defendants by suggesting that they 0:03:22.916 --> 0:03:27.316 have a longer detainment before sentencing. So that's an example 0:03:27.356 --> 0:03:31.756 where we have some background bias in our society. Right, 0:03:31.836 --> 0:03:35.716 the system is already raged against African Americans. Arrests are disproportioned, 0:03:35.716 --> 0:03:39.156 African American jailing is disproportionate of African Americans. And then 0:03:39.276 --> 0:03:42.116 once the data is trained, the data that emerges will 0:03:42.196 --> 0:03:45.916 also reflect those pre existing biases. But how do you 0:03:45.956 --> 0:03:48.436 know the problem was the algorithm? How do you know 0:03:48.476 --> 0:03:52.876 the problem isn't Rather the underlying deep structures of racism 0:03:52.916 --> 0:03:56.996 in the United States that created the circumstances where arrests 0:03:57.036 --> 0:04:01.876 and imprisonment are disproportionately acts that happen to African Americans 0:04:02.356 --> 0:04:05.156 rather than to white people. In other words, that is 0:04:05.236 --> 0:04:07.436 the form of racial bias. How do we know that 0:04:07.476 --> 0:04:09.796 the algorithm is actually making it worse as opposed to 0:04:09.836 --> 0:04:13.556 just reflecting the existing realities of race, You know, I 0:04:13.556 --> 0:04:15.556 think it's both. I mean, on the one hand, I 0:04:15.596 --> 0:04:17.836 think that we do have the issue where it is 0:04:18.196 --> 0:04:22.076 representative of the existing societal concerns that we have. A 0:04:22.276 --> 0:04:26.556 mathematical model is not necessarily going to correct or remedy. 0:04:26.596 --> 0:04:29.436 I think the historical biases that many groups have suffered. 0:04:29.716 --> 0:04:33.876 This we're talking about structural and systemic racism and discrimination 0:04:33.916 --> 0:04:36.876 that just won't go away from a computer model. But 0:04:36.916 --> 0:04:39.476 I also think that part of what we're seeing, and 0:04:39.516 --> 0:04:42.316 this is in my attempt to not say that developers 0:04:42.356 --> 0:04:45.956 are racist, that it all depends on who's at the 0:04:45.996 --> 0:04:49.356 table when developing that algorithm. And so there's two things 0:04:49.396 --> 0:04:51.956 that are going on when you look at the tech space. One, 0:04:52.036 --> 0:04:55.476 you have a very limited pool of diversity that happens 0:04:55.476 --> 0:04:58.556 in these professions you know, the data science profession in 0:04:58.556 --> 0:05:04.116 and of itself is underrepresentative of historically disadvantaged groups, women, 0:05:04.276 --> 0:05:08.276 people of color, older Americans, etc. And that can be 0:05:08.316 --> 0:05:12.076 problematic as these algorithms become much more ubiquitous in such society. 0:05:12.436 --> 0:05:14.756 And then you have the other issue of implicit bias, 0:05:14.796 --> 0:05:18.356 which comes from this unconscious understanding of how the world works. 0:05:18.436 --> 0:05:21.276 Let me give a good example of that. Amazon just 0:05:21.316 --> 0:05:25.476 a few months ago released an employment algorithm that was 0:05:25.596 --> 0:05:31.636 trying to find candidates for their engineering department. The training 0:05:31.716 --> 0:05:34.196 data that was used by the developers went on the 0:05:34.236 --> 0:05:37.716 historical data of that department, which tended to be white men, 0:05:38.236 --> 0:05:41.676 and as a result, the algorithm kicked out any resume 0:05:41.796 --> 0:05:44.556 that had any hint of a person being from an 0:05:44.556 --> 0:05:49.036 all women's college or having a women's group represented. So 0:05:49.116 --> 0:05:51.756 that word, in and of itself, because it's not necessarily 0:05:51.836 --> 0:05:56.836 associated with engineering professions, struck down the opportunities of them 0:05:56.836 --> 0:06:00.516 to become a more diverse workforce in terms of that department. 0:06:00.876 --> 0:06:03.356 You know, Amazon later retracted that and took it off 0:06:03.356 --> 0:06:06.036 the market, But you see what I mean? So right, 0:06:06.116 --> 0:06:08.676 I mean, so you mentioned a group of fascinating things there. 0:06:08.676 --> 0:06:11.676 So one is the composition of the tech world, and 0:06:11.756 --> 0:06:14.756 there I think every reasonable person can agree that it 0:06:14.796 --> 0:06:18.076 can only be better on its own terms, totally independent 0:06:18.156 --> 0:06:20.996 of whether it affects the implicit bias phenomenon. But in general, 0:06:21.036 --> 0:06:22.476 we would love to see we need to see as 0:06:22.476 --> 0:06:27.076 a society much greater diverse representation of previously disadvantage groups 0:06:27.156 --> 0:06:30.316 in or currently disadvantage groups in the tech role. Then 0:06:30.356 --> 0:06:33.796 they have the implicit bias example where you're drawing on 0:06:34.076 --> 0:06:37.236 existing data. So you could sort of imagine why Amazon 0:06:37.276 --> 0:06:38.956 wants to figure out who to hire, and so they 0:06:38.956 --> 0:06:40.676 put into the data that people they have hired because 0:06:40.676 --> 0:06:43.556 they think, hey, we're pretty awesome and sure enough. Then 0:06:43.556 --> 0:06:46.116 that just suggests that they replicate the thing that they 0:06:46.156 --> 0:06:48.436 have already. Let me ask about the flip side of that, 0:06:48.476 --> 0:06:51.236 not nicle the potentially positive side. So take a couple 0:06:51.276 --> 0:06:54.316 of examples. You've mentioned determination of either bail or of 0:06:54.316 --> 0:06:57.316 criminal sentence on the one hand. Another example would be 0:06:57.876 --> 0:07:02.676 employment determinations. These are all cases where we know from 0:07:02.836 --> 0:07:06.516 years and years of collected data that human decision makers 0:07:06.556 --> 0:07:10.436 are systematically biased against people of and we try to 0:07:10.516 --> 0:07:13.596 debias people by different methods. We have an appeals process 0:07:13.596 --> 0:07:15.636 where you can appeal and say I've been discriminated against, 0:07:15.836 --> 0:07:19.036 very hard to win. We have rules that say don't 0:07:19.076 --> 0:07:22.116 be biased, and we even have lots of decision makers 0:07:22.116 --> 0:07:25.116 who in their hearts are not biased, and yet you 0:07:25.156 --> 0:07:28.276 show them statistically what they've done over the long run, 0:07:28.276 --> 0:07:32.716 and sure enough their behavior does reflect bias. Now, don't 0:07:32.756 --> 0:07:38.036 algorithms potentially offer a liberatory solution an equality solution here? 0:07:38.156 --> 0:07:40.116 Because the one thing that we can say about an 0:07:40.116 --> 0:07:43.516 algorithm is, unlike a human, if you give it a rule, 0:07:44.116 --> 0:07:47.716 it will follow the rule, and there's no O G. 0:07:47.956 --> 0:07:50.116 I thought I was following the rule, but I really wasn't. 0:07:50.596 --> 0:07:54.676 So if you, in principle, tell the algorithm not to 0:07:54.716 --> 0:07:58.876 consider race, or you tell the algorithm not to consider 0:07:58.996 --> 0:08:01.036 various factors that look like proxies for race, and you 0:08:01.036 --> 0:08:03.556 can even train the algorithms so that it is less 0:08:03.596 --> 0:08:07.516 inclined to rely on those proxies, then presumably you could 0:08:07.516 --> 0:08:11.236 have a decision maker making decisions about criminal justice, making 0:08:11.236 --> 0:08:15.916 decisions about employment that are less biased, less racist than 0:08:16.236 --> 0:08:20.436 the best intentioned human being. Because humans have an unconscious, 0:08:20.676 --> 0:08:24.516 and in our unconscious we might be biased, but algorithms 0:08:24.676 --> 0:08:27.316 don't have an unconscious mind. Yeah, you know, I have 0:08:27.396 --> 0:08:29.196 to say I got to push back on that, and 0:08:29.236 --> 0:08:32.276 I'll tell you why. I think because we see more 0:08:32.396 --> 0:08:35.636 of the digital economy rush to market, we're not dealing 0:08:35.676 --> 0:08:38.476 with an environment where we see this level of diligence 0:08:38.516 --> 0:08:41.956 when it comes to know what is the bias impact 0:08:42.196 --> 0:08:45.996 on certain groups? Have we been able to use race 0:08:46.076 --> 0:08:50.876 as a proxy to create an anti bias experimentation? Are 0:08:50.916 --> 0:08:54.796 we auditing our algorithm in ways that we can ensure 0:08:54.956 --> 0:08:59.236 from its development to its execution that we're identifying what 0:08:59.316 --> 0:09:02.116 that bias may look like? And I think we need 0:09:02.156 --> 0:09:06.076 to go forward and put together some framework not on 0:09:06.116 --> 0:09:09.076 all algorithms. I think Netflix does a pretty good job 0:09:09.276 --> 0:09:11.476 recommending the types of movies that I like to watch. 0:09:11.596 --> 0:09:13.116 I'm not I mean, that's true, but I'm not so 0:09:13.156 --> 0:09:15.396 sure that we should even assume that those aren't biased. 0:09:15.596 --> 0:09:18.676 They are. I think those are because they're also going 0:09:18.756 --> 0:09:20.676 to pick out features. They also know what your zip 0:09:20.676 --> 0:09:22.996 code is. If they take up zip code, then at 0:09:23.036 --> 0:09:25.476 least implicity, they're also recognizing race, you know, I mean, 0:09:25.836 --> 0:09:27.756 that's right. So I'll just take my case. I'm an 0:09:27.756 --> 0:09:30.916 African American woman who's middle age, who loves to watch 0:09:30.956 --> 0:09:33.036 you know, black romance Flix, and let me tell you, 0:09:33.036 --> 0:09:35.836 every time Netflix recommends one, I'm happy. I mean, the 0:09:35.836 --> 0:09:38.076 only problem I get, you know, challenged by it is 0:09:38.116 --> 0:09:40.636 when the content runs out and not investing in, you know, 0:09:40.756 --> 0:09:44.436 more more programmers or developers to develop more content for 0:09:44.476 --> 0:09:46.716 people like me. But maybe that's because they're not feeding 0:09:46.716 --> 0:09:48.636 those two people like me, right, I'm a middle aged 0:09:48.676 --> 0:09:51.316 white guy. They're not telling me to watch those, But 0:09:51.396 --> 0:09:54.476 maybe if they did, I'd watch them, I'd like them, 0:09:54.596 --> 0:09:56.516 and if that happened, there would be developers. So in 0:09:56.516 --> 0:09:58.876 that sense, you know, it may be that that there 0:09:58.876 --> 0:10:00.836 is an implicit bias there. They're just assuming that the 0:10:00.996 --> 0:10:02.996 reason that I don't watch them is that I haven't 0:10:02.996 --> 0:10:06.756 watched that many you know, African American romantic comedies in 0:10:06.556 --> 0:10:09.796 recent years and so. But you know, but itself is 0:10:09.836 --> 0:10:12.516 not a neutral fact, right, because no one's advertising them 0:10:12.556 --> 0:10:14.556 to me. Netflix isn't telling me to watch them. If 0:10:14.556 --> 0:10:16.756 they told me, that might have a different impact. That's right, 0:10:16.796 --> 0:10:18.836 And that's I mean, I always say to people. When 0:10:19.716 --> 0:10:23.156 the alright movement and the white conservative movement became a 0:10:23.196 --> 0:10:26.356 big thing and hate speech on Facebook, I was kind 0:10:26.356 --> 0:10:28.276 of surprised that I didn't know that this was happening 0:10:28.316 --> 0:10:31.156 in the same playground in which I also, you know, visit. 0:10:31.396 --> 0:10:33.876 And that's because my algorithm or is not made of 0:10:34.556 --> 0:10:37.796 white supremacists. It's you know, more liberals that sort of 0:10:37.836 --> 0:10:40.156 speak the same language and feel the same way about 0:10:40.156 --> 0:10:43.436 certain issues. What about the hardcase nicle, I mean, you know, 0:10:43.516 --> 0:10:47.476 let's say I'm applying for credit and they've got information, 0:10:47.796 --> 0:10:49.716 you know that Let's say I have allowed I didn't 0:10:49.796 --> 0:10:52.156 check the box to make it private. That says, hey, 0:10:52.236 --> 0:10:56.756 Feldman's been searching for payday loans, and the algorithm notices 0:10:56.836 --> 0:10:59.156 something that is intuitively very plausible, which is that if 0:10:59.156 --> 0:11:02.476 I'm so desperate that I'm looking into payday loans, probably 0:11:02.636 --> 0:11:05.476 that means I'm a slightly less good credit risk than 0:11:05.556 --> 0:11:08.116 someone who hasn't been yet searching for payday loans, because 0:11:08.116 --> 0:11:09.756 you know, eventually, I'm going to start looking for that 0:11:09.756 --> 0:11:12.436 the minute I really need it. So if I'm trying 0:11:12.436 --> 0:11:15.636 to lend money, and that's there's nothing inherently racially determinative 0:11:15.796 --> 0:11:18.276 about that. It may not even be about wealth in general. 0:11:18.276 --> 0:11:20.116 It's just about how much money I have right this minute, 0:11:20.196 --> 0:11:22.676 or how little I have right this minute. That might 0:11:22.716 --> 0:11:25.996 be great from the standpoint of the credit company, and 0:11:26.036 --> 0:11:28.156 they might actually be able to do a better job 0:11:28.556 --> 0:11:31.796 of setting the correct interest rate for me based on 0:11:31.876 --> 0:11:35.596 that information. Does that still disturbute? Does it still make 0:11:35.636 --> 0:11:36.956 you think that that's a problem or is that more 0:11:36.956 --> 0:11:38.956 like Netflix? It's not that big a deal they're telling 0:11:38.996 --> 0:11:42.036 you know, They're they're fitting the data to the objective. 0:11:42.476 --> 0:11:44.316 You know. I think it's interesting because I struggle with it. 0:11:44.356 --> 0:11:47.236 I think that there are cases where, you know, our 0:11:47.356 --> 0:11:50.996 online behavior will indicate certain characteristics about us. Though it 0:11:51.036 --> 0:11:53.196 is somewhat problematic because what if I was searching for 0:11:53.236 --> 0:11:56.196 the payday loan for my uncle? Right, not necessarily for myself, 0:11:57.156 --> 0:11:59.596 But in the end, I think we have to be 0:11:59.676 --> 0:12:03.476 very sensitive, or the algorithmic operator has to be sensitive 0:12:03.556 --> 0:12:05.876 to the extent to which they're denying credit to these 0:12:05.916 --> 0:12:09.756 groups versus out the groups. Right. One thing, for example, 0:12:09.756 --> 0:12:12.196 we say in the paper, which I think is just profound, is, 0:12:12.236 --> 0:12:15.956 as an African American who maybe serve more higher interest 0:12:16.036 --> 0:12:19.916 credit card rates, what if I see that ad come 0:12:19.996 --> 0:12:22.516 through and I click it just because I'm interested to 0:12:22.516 --> 0:12:25.716 see why I'm getting this ad. Automatically, I will be 0:12:25.756 --> 0:12:28.956 served similar ads, right, so it automatically places me in 0:12:28.956 --> 0:12:32.596 that high credit risk category. The challenge that we're having now, Noah, 0:12:32.636 --> 0:12:34.996 is that as an individual consumer, I have no way 0:12:35.036 --> 0:12:38.956 of recurating what my identity is. I want to ask 0:12:38.996 --> 0:12:41.596 you a kind of final big picture question, and it's 0:12:41.756 --> 0:12:45.156 when you survey this whole environment, the possibilities of regulation, 0:12:45.196 --> 0:12:48.116 you're testifying on the hill about it, you're doing reports 0:12:48.156 --> 0:12:53.676 on it. Are you in general optimistic about the future 0:12:53.796 --> 0:12:57.676 of the possibility to regulate algorithms and to also turn 0:12:57.716 --> 0:13:01.596 algorithms to good with respect to fairness and equality, or 0:13:01.636 --> 0:13:05.716 are you, on balance pessimistic and think that the terrible 0:13:05.916 --> 0:13:08.836 legacies of discrimination that we have in our country are 0:13:08.916 --> 0:13:12.876 like just to be either continued or even made worse 0:13:13.116 --> 0:13:16.396 by virtue of this technological development. You know, I'm a 0:13:16.436 --> 0:13:21.876 technologist who's optimistic about the use of technology, you know, 0:13:21.956 --> 0:13:25.516 I think of it this way. I think as technology evolves, 0:13:25.596 --> 0:13:28.436 we are faced with this challenge whether or not the 0:13:28.516 --> 0:13:31.676 technology coopts the user or the user has something to 0:13:31.716 --> 0:13:35.476 do with the technologies agency, right, And so I'm one 0:13:35.516 --> 0:13:38.756 of those people, particularly in this case of algorithms, which 0:13:38.796 --> 0:13:42.476 has just become so interesting to many of us because 0:13:42.556 --> 0:13:45.156 it's got this explainability portion, and then it has stuff 0:13:45.156 --> 0:13:47.996 that we don't even know how to dissect and unpack that. 0:13:48.116 --> 0:13:50.476 I think what we're trying to do in this particular case, 0:13:50.556 --> 0:13:52.756 Noah's just get ahead of it and to be much 0:13:52.796 --> 0:13:55.556 more proactive in talking about it. I mean, my goal 0:13:56.276 --> 0:13:59.676 is to bring to the forefront those algorithms that are 0:13:59.676 --> 0:14:03.676 allowing older Americans to age in place, those algorithms that 0:14:03.676 --> 0:14:08.636 are catching chronic disease and some of the worst abilitating diseases, 0:14:08.676 --> 0:14:12.516 and as ahead of time because of the precision of 0:14:12.556 --> 0:14:17.236 the technology, we're seeing, you know, better customization of educational 0:14:17.276 --> 0:14:21.036 curricula for students because algorithms are able to identify learning 0:14:21.076 --> 0:14:24.356 styles much faster than a teacher can. And so I 0:14:24.356 --> 0:14:27.436 don't want us to be a society which turns our 0:14:27.476 --> 0:14:31.276 back against the technology and the innovation, because that's part 0:14:31.316 --> 0:14:34.996 of this whole new revolution of our shift for manufacturing 0:14:35.036 --> 0:14:37.956 into I think this digital age where it does matter. 0:14:38.236 --> 0:14:41.116 I'm gonna tell you honestly, what really concerns me is 0:14:41.156 --> 0:14:44.316 the fact that the less information or the less diffused 0:14:44.396 --> 0:14:47.076 that these algorithms are, the more likely you'll be on 0:14:47.196 --> 0:14:50.396 the wrong side of digital opportunity, and the more likely 0:14:50.436 --> 0:14:52.716 that your community may not get some of the services 0:14:52.716 --> 0:14:55.156 that have come out of an algorithmic economy. I mean, 0:14:55.236 --> 0:14:57.756 imagine living in a community where they don't have your data. 0:14:58.156 --> 0:15:00.396 You're not your data is not being harnessed for any 0:15:00.396 --> 0:15:03.516 type of productive algorithm. You find yourself in a state 0:15:03.516 --> 0:15:07.276 where you have more chronic disease, more incarceration, less levels 0:15:07.276 --> 0:15:10.356 of educational achievement. Better to be then to be out. Yeah, 0:15:10.396 --> 0:15:12.196 and I'll just say this in final I mean, I 0:15:12.236 --> 0:15:15.196 think for those of us that are in this space, 0:15:15.276 --> 0:15:18.956 I think we take into consideration the fairness and accuracy 0:15:18.956 --> 0:15:22.356 conversations as well the ethical conversations, But our main goal 0:15:22.436 --> 0:15:25.476 is to deploy this responsibly. And if we can come 0:15:25.516 --> 0:15:28.316 up with more responsible frameworks that incorporate many of the 0:15:28.356 --> 0:15:31.156 aspects that we've talked about today, I think we're on 0:15:31.196 --> 0:15:35.156 the brink of actually unpacking what could potentially become the 0:15:35.236 --> 0:15:38.956 next big game changer for people you know that have 0:15:39.116 --> 0:15:42.796 had to rely upon wrong decisions or humans who are 0:15:42.836 --> 0:15:45.076 biased to do such. So I want to say, you know, 0:15:45.116 --> 0:15:46.996 in all honesty, I do agree with you that there's 0:15:46.996 --> 0:15:50.676 a promise of algorithms to sort of break down the barriers, 0:15:50.996 --> 0:15:53.036 but it has to be done responsibly with the right 0:15:53.076 --> 0:15:56.436 people at the table to talk about it. Niculternally thank 0:15:56.436 --> 0:15:59.356 you so much for a really fascinating and rich discussion 0:15:59.476 --> 0:16:01.836 and for sharing your knowledge and expertise with us. Thank 0:16:01.876 --> 0:16:18.996 you now, thank you for having me appreciate you. My 0:16:19.076 --> 0:16:21.556 conversation with Nicole made me want to talk to somebody 0:16:21.596 --> 0:16:23.316 who was doing the kind of work that she was 0:16:23.356 --> 0:16:26.636 just talking about, someone who was thinking about how algorithms 0:16:26.676 --> 0:16:30.756 can break down barriers rather than create them. So I 0:16:30.796 --> 0:16:34.156 called up Talia Gillis. Talia is a PhD student in 0:16:34.156 --> 0:16:37.636 a business economics at Harvard. She's also a former student 0:16:37.676 --> 0:16:40.396 of mine who holds a law degree from Harvard, and 0:16:40.476 --> 0:16:44.476 she's been researching how banks and other lenders use algorithms 0:16:44.476 --> 0:16:47.676 to determine interest rates on loans. She thinks that the 0:16:47.716 --> 0:16:51.476 way they're doing it right now isn't working, but she 0:16:51.596 --> 0:16:55.516 has an idea for a better way. Talia, thank you 0:16:55.596 --> 0:16:58.436 so much for joining me. It's great to have you, 0:16:58.676 --> 0:17:01.316 and it's great to talk to you as it were 0:17:01.796 --> 0:17:04.756 on air about something that we've talked about lots and 0:17:04.796 --> 0:17:09.596 lots of times in the office, your research, because there's 0:17:09.676 --> 0:17:12.796 very much on how we can fix the problem of 0:17:12.876 --> 0:17:17.756 algorithmic bias. Tell me what it is that is the 0:17:17.796 --> 0:17:21.476 core of your approach. What is your original idea about 0:17:21.516 --> 0:17:23.956 what we can do to make things better. So I 0:17:23.996 --> 0:17:26.516 think the core of the approach is, first of all, 0:17:26.556 --> 0:17:29.236 to recognize that it's it's very hard to know a 0:17:29.356 --> 0:17:33.756 priori what exactly the bias or what direction the bias 0:17:33.836 --> 0:17:35.676 is going to go in, and so I'm very much 0:17:35.716 --> 0:17:39.276 focused on the credit pricing context. And in the credit 0:17:39.276 --> 0:17:43.036 pricing context, it's true that a lot of the kind 0:17:43.036 --> 0:17:47.156 of input variables into a credit pricing decision suffer from 0:17:47.196 --> 0:17:49.716 some sort of bias. But what's important to keep in 0:17:49.756 --> 0:17:53.876 mind is that kind of some biases might get worse 0:17:53.916 --> 0:17:57.996 in the algorithmic context, but actually big data might in 0:17:58.236 --> 0:18:00.876 for other types of biases, make things things better in 0:18:00.916 --> 0:18:05.196 a way. I think there's two large, separate categories of bias. 0:18:05.316 --> 0:18:08.156 So the first is what I call kind of inputs 0:18:08.156 --> 0:18:10.636 that result from a bias world, and the idea there 0:18:10.756 --> 0:18:14.316 is that there's some kind of pre existing discrimination, and 0:18:14.356 --> 0:18:17.076 so there might be disparities between men and women, or 0:18:17.116 --> 0:18:20.796 between blacks and whites that kind of originate partially from 0:18:20.836 --> 0:18:24.076 that discrimination. So that's what you're calling biased world, and 0:18:24.116 --> 0:18:26.796 that is you're going to apply for a loan. If 0:18:26.796 --> 0:18:28.676 you make less money and you have more debt, you're 0:18:28.676 --> 0:18:30.396 not going to get as good terms for the loan. 0:18:30.876 --> 0:18:34.396 But that's not because the lender in particular, that's because 0:18:34.876 --> 0:18:37.116 you live in a society where there's background sexist and 0:18:37.116 --> 0:18:40.516 there's background racism. The world is already biased. And so 0:18:40.556 --> 0:18:42.476 in that sense, that's biased world. And then what's the 0:18:42.476 --> 0:18:46.436 second category? And so the second category is inputs that 0:18:46.516 --> 0:18:49.476 are bias because they result from some kind of bias measurement. 0:18:50.156 --> 0:18:52.796 You can think of that as, for example, the way 0:18:52.876 --> 0:18:55.596 we measure someone's income. You know, we might put a 0:18:55.636 --> 0:18:58.756 lot of weight on someone who has one regular job, 0:18:58.876 --> 0:19:03.796 regular paycheck, and we fully capture their income and compare 0:19:03.836 --> 0:19:06.836 that to someone who kind of has multiple jobs, maybe 0:19:07.116 --> 0:19:11.396 isn't in a formal employee employer relationship, like an uber driver, 0:19:11.916 --> 0:19:15.116 and then we kind of discount their income or don't 0:19:15.196 --> 0:19:18.356 measure it properly, or or don't have the ability to 0:19:18.396 --> 0:19:21.956 fully capture what they're earning. And so the more you 0:19:21.996 --> 0:19:25.116 might look at two people who they're underlying income is similar, 0:19:25.476 --> 0:19:27.996 but because of the way we're measuring a person's income, 0:19:28.356 --> 0:19:31.516 then we consider kind of the second person to have 0:19:31.596 --> 0:19:34.516 a lower income. So that's an example of bias in 0:19:34.516 --> 0:19:36.716 the way we're measuring where we're whether we mean to 0:19:36.916 --> 0:19:39.276 or not, we're systematically giving an advantage to someone who 0:19:39.316 --> 0:19:41.516 works in a nine to five as opposed to someone 0:19:41.516 --> 0:19:43.956 who's in the gig economy. And that's what you're calling 0:19:44.156 --> 0:19:47.476 bias in measurement. And now how would you go about 0:19:47.596 --> 0:19:50.276 measuring which kind of bias or what kind of bias 0:19:50.596 --> 0:19:54.996 is in fact found in the algorithm. So it's it's 0:19:55.076 --> 0:19:59.356 quite difficult to in reality perfectly distinguish between these two biases. 0:19:59.836 --> 0:20:03.076 Also because very often, like an example I gave with income, 0:20:03.156 --> 0:20:05.956 it might be a combination of those two. And what 0:20:05.996 --> 0:20:08.156 do you do if you're in not biased world, but 0:20:08.356 --> 0:20:11.476 biased German situation where you're worried not about the backround 0:20:11.476 --> 0:20:14.716 discrimination in the world, but more worried that you're measuring 0:20:14.716 --> 0:20:17.516 the wrong things in the algorithm and as a result, 0:20:18.236 --> 0:20:22.236 you know, having a bad effect on their community of color. 0:20:23.276 --> 0:20:28.676 So with bias measurement, what's interesting about the algorithmic context 0:20:28.876 --> 0:20:31.276 is that it actually might mitigate a lot of the 0:20:31.316 --> 0:20:35.236 harms that we're concerned about in the context of bias measurement. 0:20:35.876 --> 0:20:39.196 So if you take, for example, credit scores, there have 0:20:39.236 --> 0:20:42.276 been many claims that credit scores are biased against minorities, 0:20:42.316 --> 0:20:46.436 and that's because they measure certain qualities of credit worthiness 0:20:46.516 --> 0:20:50.356 that are more representative of let's say, white borrowers. So 0:20:50.756 --> 0:20:54.316 it puts a lot of weight on people repaying previous 0:20:54.356 --> 0:20:56.676 loans on time, but it might not give any weight 0:20:56.716 --> 0:21:00.236 to people who regularly made let's say rent payments, which 0:21:00.316 --> 0:21:02.876 might actually also be a very good measure of a 0:21:02.916 --> 0:21:06.756 person's credit worthiness. So in a world in which we 0:21:06.796 --> 0:21:09.596 put a lot a lot of weight on a credit score, 0:21:10.396 --> 0:21:12.556 if we moved to a world of kind of machine 0:21:12.596 --> 0:21:15.276 learning and big data, we might get a whole new 0:21:15.356 --> 0:21:19.076 richness of indicators of a person's credit worthiness. So let's 0:21:19.076 --> 0:21:22.036 say the algorithm, how did your full history of payments 0:21:22.196 --> 0:21:26.036 or your full kind of consumer history. Then we might 0:21:26.076 --> 0:21:29.116 be getting a lot more information out about a person's 0:21:29.156 --> 0:21:32.916 credit worthiness that was before only limited to the credit score. 0:21:33.036 --> 0:21:35.236 So this is an example where if we could identify 0:21:35.916 --> 0:21:38.756 the bias in the measurement, then we could do better 0:21:38.876 --> 0:21:42.196 with the algorithm. Yes, what about a situation where the 0:21:42.236 --> 0:21:44.796 opposite is happening where the algorithm is taking into account 0:21:44.836 --> 0:21:47.956 things that are producing measurement bias. How do we know 0:21:48.036 --> 0:21:51.236 that that's happening. So I think that the key is 0:21:51.276 --> 0:21:55.076 that we never truly know what's going on. Say more 0:21:55.076 --> 0:21:56.516 about that, because I think that scares a lot of 0:21:56.516 --> 0:21:58.876 people with respect to these algorithms. What does it mean 0:21:58.876 --> 0:22:01.756 to say we never truly know what's going on? Well, 0:22:02.116 --> 0:22:03.796 on the one hand, you could say it's scary, But 0:22:03.836 --> 0:22:05.876 on the other hand, you could say that any attempt 0:22:05.876 --> 0:22:09.956 to say it's necessarily bad or necessarily going to hurt populations. 0:22:10.556 --> 0:22:13.116 It's going to be a difficult position to defend because 0:22:13.396 --> 0:22:16.316 it's kind of more of an empirical question that requires 0:22:16.356 --> 0:22:19.396