1
00:00:15,356 --> 00:00:21,756
Speaker 1: Pushkin from Pushkin Industries. This is Deep Background, the show

2
00:00:21,796 --> 00:00:24,796
Speaker 1: where we explore the stories behind the stories in the news.

3
00:00:25,116 --> 00:00:28,996
Speaker 1: I'm Noah Feldman. How did you find this podcast? Did

4
00:00:28,996 --> 00:00:31,196
Speaker 1: you see an ad for it on your phone? If so,

5
00:00:31,676 --> 00:00:33,596
Speaker 1: that ad might have shown up for you, because, based

6
00:00:33,596 --> 00:00:36,316
Speaker 1: on information about you that's out there on the Internet,

7
00:00:36,716 --> 00:00:40,396
Speaker 1: a computer algorithm decided that this show might be the

8
00:00:40,516 --> 00:00:43,436
Speaker 1: kind of thing that you would like. Algorithms like that

9
00:00:43,476 --> 00:00:47,716
Speaker 1: are all around us. Some are far more consequential than others.

10
00:00:48,076 --> 00:00:50,156
Speaker 1: But I'm glad you're listening to the show. But hey,

11
00:00:50,156 --> 00:00:53,076
Speaker 1: if you're applying for credit, an algorithm could actually evaluate

12
00:00:53,076 --> 00:00:55,396
Speaker 1: your credit worthiness, and the stakes are a little higher

13
00:00:55,436 --> 00:00:57,676
Speaker 1: there than whether you're listening to this podcast or to

14
00:00:57,716 --> 00:01:00,516
Speaker 1: Trevor Noah's. If I'm looking for a job, an algorithm

15
00:01:00,516 --> 00:01:03,076
Speaker 1: could go through all of the job applicants to try

16
00:01:03,116 --> 00:01:06,036
Speaker 1: to do a first cut before the employer decides who

17
00:01:06,076 --> 00:01:08,796
Speaker 1: they're going to interview. In some cities, algorithms are even

18
00:01:08,836 --> 00:01:11,116
Speaker 1: being used by the police to try to predict the

19
00:01:11,156 --> 00:01:13,436
Speaker 1: probability that there's going to be a crime in a

20
00:01:13,476 --> 00:01:16,796
Speaker 1: particular place and to decide where they're going to focus

21
00:01:16,996 --> 00:01:20,716
Speaker 1: the police efforts and actually send the cops. Nicole Ternally

22
00:01:21,036 --> 00:01:24,316
Speaker 1: is a fellow at Brooking Center for Technology Innovation. She's

23
00:01:24,316 --> 00:01:28,876
Speaker 1: been studying how algorithms like this work and how they fail.

24
00:01:29,276 --> 00:01:32,636
Speaker 1: She recently co wrote a report for Brookings about algorithmic bias,

25
00:01:32,996 --> 00:01:37,596
Speaker 1: or in other words, how computers can be racist. So

26
00:01:37,636 --> 00:01:40,076
Speaker 1: if we look at an algorithm like a black box,

27
00:01:40,476 --> 00:01:43,356
Speaker 1: it starts with an input and it ends with an output.

28
00:01:43,756 --> 00:01:46,076
Speaker 1: When it comes to the input, you know, big data

29
00:01:46,116 --> 00:01:49,876
Speaker 1: has made it very easy to actually harness volumes of

30
00:01:49,956 --> 00:01:54,116
Speaker 1: data they're about us, these reference points, about individuals, and

31
00:01:54,196 --> 00:01:56,596
Speaker 1: to create you know, I think some input or what

32
00:01:56,636 --> 00:02:00,476
Speaker 1: we call training data that essentially trains the algorithm to

33
00:02:00,716 --> 00:02:03,716
Speaker 1: adapt to what our behaviors are. In many cases, what

34
00:02:03,756 --> 00:02:06,476
Speaker 1: goes into the algorithm can be accurate. There are certain

35
00:02:06,516 --> 00:02:11,396
Speaker 1: things that your listeners do online that are discrete, our objective,

36
00:02:11,596 --> 00:02:14,516
Speaker 1: are true in terms of your search queries, in terms

37
00:02:14,556 --> 00:02:18,876
Speaker 1: of your online profile. But when you have developers that

38
00:02:19,476 --> 00:02:22,956
Speaker 1: put in training data that in some respects may be

39
00:02:23,116 --> 00:02:27,756
Speaker 1: biased or skewed, it creates challenges or what technologists have

40
00:02:27,836 --> 00:02:30,996
Speaker 1: called garbage in So what I mean by that, if

41
00:02:30,996 --> 00:02:33,396
Speaker 1: you're developing an algorithm and this is actually a case

42
00:02:33,436 --> 00:02:35,796
Speaker 1: So this is not something that we're making up. Like

43
00:02:35,876 --> 00:02:38,956
Speaker 1: the Compass algorithm, which was designed to help judges make

44
00:02:38,996 --> 00:02:42,396
Speaker 1: better predictions on the amount of time that a defendant

45
00:02:42,436 --> 00:02:46,476
Speaker 1: should be detained before sentencing. And let's say the training

46
00:02:46,556 --> 00:02:51,116
Speaker 1: data used to train that algorithm is based upon criminal

47
00:02:51,196 --> 00:02:55,556
Speaker 1: justice stats or criminal behavior stats. It's no secret that

48
00:02:55,636 --> 00:02:59,716
Speaker 1: in this country, African American men in particular are more

49
00:02:59,756 --> 00:03:03,476
Speaker 1: likely to experience arrest. And if they are more likely

50
00:03:03,516 --> 00:03:07,996
Speaker 1: to experience arrest, which oftentimes leads to incarceration, they will

51
00:03:08,156 --> 00:03:12,036
Speaker 1: overwhelmingly make up the majority of the training data. So

52
00:03:12,076 --> 00:03:16,316
Speaker 1: that input, when it gets to the output, it then

53
00:03:16,956 --> 00:03:22,836
Speaker 1: may disproportionately affect African American defendants by suggesting that they

54
00:03:22,916 --> 00:03:27,316
Speaker 1: have a longer detainment before sentencing. So that's an example

55
00:03:27,356 --> 00:03:31,756
Speaker 1: where we have some background bias in our society. Right,

56
00:03:31,836 --> 00:03:35,716
Speaker 1: the system is already raged against African Americans. Arrests are disproportioned,

57
00:03:35,716 --> 00:03:39,156
Speaker 1: African American jailing is disproportionate of African Americans. And then

58
00:03:39,276 --> 00:03:42,116
Speaker 1: once the data is trained, the data that emerges will

59
00:03:42,196 --> 00:03:45,916
Speaker 1: also reflect those pre existing biases. But how do you

60
00:03:45,956 --> 00:03:48,436
Speaker 1: know the problem was the algorithm? How do you know

61
00:03:48,476 --> 00:03:52,876
Speaker 1: the problem isn't Rather the underlying deep structures of racism

62
00:03:52,916 --> 00:03:56,996
Speaker 1: in the United States that created the circumstances where arrests

63
00:03:57,036 --> 00:04:01,876
Speaker 1: and imprisonment are disproportionately acts that happen to African Americans

64
00:04:02,356 --> 00:04:05,156
Speaker 1: rather than to white people. In other words, that is

65
00:04:05,236 --> 00:04:07,436
Speaker 1: the form of racial bias. How do we know that

66
00:04:07,476 --> 00:04:09,796
Speaker 1: the algorithm is actually making it worse as opposed to

67
00:04:09,836 --> 00:04:13,556
Speaker 1: just reflecting the existing realities of race, You know, I

68
00:04:13,556 --> 00:04:15,556
Speaker 1: think it's both. I mean, on the one hand, I

69
00:04:15,596 --> 00:04:17,836
Speaker 1: think that we do have the issue where it is

70
00:04:18,196 --> 00:04:22,076
Speaker 1: representative of the existing societal concerns that we have. A

71
00:04:22,276 --> 00:04:26,556
Speaker 1: mathematical model is not necessarily going to correct or remedy.

72
00:04:26,596 --> 00:04:29,436
Speaker 1: I think the historical biases that many groups have suffered.

73
00:04:29,716 --> 00:04:33,876
Speaker 1: This we're talking about structural and systemic racism and discrimination

74
00:04:33,916 --> 00:04:36,876
Speaker 1: that just won't go away from a computer model. But

75
00:04:36,916 --> 00:04:39,476
Speaker 1: I also think that part of what we're seeing, and

76
00:04:39,516 --> 00:04:42,316
Speaker 1: this is in my attempt to not say that developers

77
00:04:42,356 --> 00:04:45,956
Speaker 1: are racist, that it all depends on who's at the

78
00:04:45,996 --> 00:04:49,356
Speaker 1: table when developing that algorithm. And so there's two things

79
00:04:49,396 --> 00:04:51,956
Speaker 1: that are going on when you look at the tech space. One,

80
00:04:52,036 --> 00:04:55,476
Speaker 1: you have a very limited pool of diversity that happens

81
00:04:55,476 --> 00:04:58,556
Speaker 1: in these professions you know, the data science profession in

82
00:04:58,556 --> 00:05:04,116
Speaker 1: and of itself is underrepresentative of historically disadvantaged groups, women,

83
00:05:04,276 --> 00:05:08,276
Speaker 1: people of color, older Americans, etc. And that can be

84
00:05:08,316 --> 00:05:12,076
Speaker 1: problematic as these algorithms become much more ubiquitous in such society.

85
00:05:12,436 --> 00:05:14,756
Speaker 1: And then you have the other issue of implicit bias,

86
00:05:14,796 --> 00:05:18,356
Speaker 1: which comes from this unconscious understanding of how the world works.

87
00:05:18,436 --> 00:05:21,276
Speaker 1: Let me give a good example of that. Amazon just

88
00:05:21,316 --> 00:05:25,476
Speaker 1: a few months ago released an employment algorithm that was

89
00:05:25,596 --> 00:05:31,636
Speaker 1: trying to find candidates for their engineering department. The training

90
00:05:31,716 --> 00:05:34,196
Speaker 1: data that was used by the developers went on the

91
00:05:34,236 --> 00:05:37,716
Speaker 1: historical data of that department, which tended to be white men,

92
00:05:38,236 --> 00:05:41,676
Speaker 1: and as a result, the algorithm kicked out any resume

93
00:05:41,796 --> 00:05:44,556
Speaker 1: that had any hint of a person being from an

94
00:05:44,556 --> 00:05:49,036
Speaker 1: all women's college or having a women's group represented. So

95
00:05:49,116 --> 00:05:51,756
Speaker 1: that word, in and of itself, because it's not necessarily

96
00:05:51,836 --> 00:05:56,836
Speaker 1: associated with engineering professions, struck down the opportunities of them

97
00:05:56,836 --> 00:06:00,516
Speaker 1: to become a more diverse workforce in terms of that department.

98
00:06:00,876 --> 00:06:03,356
Speaker 1: You know, Amazon later retracted that and took it off

99
00:06:03,356 --> 00:06:06,036
Speaker 1: the market, But you see what I mean? So right,

100
00:06:06,116 --> 00:06:08,676
Speaker 1: I mean, so you mentioned a group of fascinating things there.

101
00:06:08,676 --> 00:06:11,676
Speaker 1: So one is the composition of the tech world, and

102
00:06:11,756 --> 00:06:14,756
Speaker 1: there I think every reasonable person can agree that it

103
00:06:14,796 --> 00:06:18,076
Speaker 1: can only be better on its own terms, totally independent

104
00:06:18,156 --> 00:06:20,996
Speaker 1: of whether it affects the implicit bias phenomenon. But in general,

105
00:06:21,036 --> 00:06:22,476
Speaker 1: we would love to see we need to see as

106
00:06:22,476 --> 00:06:27,076
Speaker 1: a society much greater diverse representation of previously disadvantage groups

107
00:06:27,156 --> 00:06:30,316
Speaker 1: in or currently disadvantage groups in the tech role. Then

108
00:06:30,356 --> 00:06:33,796
Speaker 1: they have the implicit bias example where you're drawing on

109
00:06:34,076 --> 00:06:37,236
Speaker 1: existing data. So you could sort of imagine why Amazon

110
00:06:37,276 --> 00:06:38,956
Speaker 1: wants to figure out who to hire, and so they

111
00:06:38,956 --> 00:06:40,676
Speaker 1: put into the data that people they have hired because

112
00:06:40,676 --> 00:06:43,556
Speaker 1: they think, hey, we're pretty awesome and sure enough. Then

113
00:06:43,556 --> 00:06:46,116
Speaker 1: that just suggests that they replicate the thing that they

114
00:06:46,156 --> 00:06:48,436
Speaker 1: have already. Let me ask about the flip side of that,

115
00:06:48,476 --> 00:06:51,236
Speaker 1: not nicle the potentially positive side. So take a couple

116
00:06:51,276 --> 00:06:54,316
Speaker 1: of examples. You've mentioned determination of either bail or of

117
00:06:54,316 --> 00:06:57,316
Speaker 1: criminal sentence on the one hand. Another example would be

118
00:06:57,876 --> 00:07:02,676
Speaker 1: employment determinations. These are all cases where we know from

119
00:07:02,836 --> 00:07:06,516
Speaker 1: years and years of collected data that human decision makers

120
00:07:06,556 --> 00:07:10,436
Speaker 1: are systematically biased against people of and we try to

121
00:07:10,516 --> 00:07:13,596
Speaker 1: debias people by different methods. We have an appeals process

122
00:07:13,596 --> 00:07:15,636
Speaker 1: where you can appeal and say I've been discriminated against,

123
00:07:15,836 --> 00:07:19,036
Speaker 1: very hard to win. We have rules that say don't

124
00:07:19,076 --> 00:07:22,116
Speaker 1: be biased, and we even have lots of decision makers

125
00:07:22,116 --> 00:07:25,116
Speaker 1: who in their hearts are not biased, and yet you

126
00:07:25,156 --> 00:07:28,276
Speaker 1: show them statistically what they've done over the long run,

127
00:07:28,276 --> 00:07:32,716
Speaker 1: and sure enough their behavior does reflect bias. Now, don't

128
00:07:32,756 --> 00:07:38,036
Speaker 1: algorithms potentially offer a liberatory solution an equality solution here?

129
00:07:38,156 --> 00:07:40,116
Speaker 1: Because the one thing that we can say about an

130
00:07:40,116 --> 00:07:43,516
Speaker 1: algorithm is, unlike a human, if you give it a rule,

131
00:07:44,116 --> 00:07:47,716
Speaker 1: it will follow the rule, and there's no O G.

132
00:07:47,956 --> 00:07:50,116
Speaker 1: I thought I was following the rule, but I really wasn't.

133
00:07:50,596 --> 00:07:54,676
Speaker 1: So if you, in principle, tell the algorithm not to

134
00:07:54,716 --> 00:07:58,876
Speaker 1: consider race, or you tell the algorithm not to consider

135
00:07:58,996 --> 00:08:01,036
Speaker 1: various factors that look like proxies for race, and you

136
00:08:01,036 --> 00:08:03,556
Speaker 1: can even train the algorithms so that it is less

137
00:08:03,596 --> 00:08:07,516
Speaker 1: inclined to rely on those proxies, then presumably you could

138
00:08:07,516 --> 00:08:11,236
Speaker 1: have a decision maker making decisions about criminal justice, making

139
00:08:11,236 --> 00:08:15,916
Speaker 1: decisions about employment that are less biased, less racist than

140
00:08:16,236 --> 00:08:20,436
Speaker 1: the best intentioned human being. Because humans have an unconscious,

141
00:08:20,676 --> 00:08:24,516
Speaker 1: and in our unconscious we might be biased, but algorithms

142
00:08:24,676 --> 00:08:27,316
Speaker 1: don't have an unconscious mind. Yeah, you know, I have

143
00:08:27,396 --> 00:08:29,196
Speaker 1: to say I got to push back on that, and

144
00:08:29,236 --> 00:08:32,276
Speaker 1: I'll tell you why. I think because we see more

145
00:08:32,396 --> 00:08:35,636
Speaker 1: of the digital economy rush to market, we're not dealing

146
00:08:35,676 --> 00:08:38,476
Speaker 1: with an environment where we see this level of diligence

147
00:08:38,516 --> 00:08:41,956
Speaker 1: when it comes to know what is the bias impact

148
00:08:42,196 --> 00:08:45,996
Speaker 1: on certain groups? Have we been able to use race

149
00:08:46,076 --> 00:08:50,876
Speaker 1: as a proxy to create an anti bias experimentation? Are

150
00:08:50,916 --> 00:08:54,796
Speaker 1: we auditing our algorithm in ways that we can ensure

151
00:08:54,956 --> 00:08:59,236
Speaker 1: from its development to its execution that we're identifying what

152
00:08:59,316 --> 00:09:02,116
Speaker 1: that bias may look like? And I think we need

153
00:09:02,156 --> 00:09:06,076
Speaker 1: to go forward and put together some framework not on

154
00:09:06,116 --> 00:09:09,076
Speaker 1: all algorithms. I think Netflix does a pretty good job

155
00:09:09,276 --> 00:09:11,476
Speaker 1: recommending the types of movies that I like to watch.

156
00:09:11,596 --> 00:09:13,116
Speaker 1: I'm not I mean, that's true, but I'm not so

157
00:09:13,156 --> 00:09:15,396
Speaker 1: sure that we should even assume that those aren't biased.

158
00:09:15,596 --> 00:09:18,676
Speaker 1: They are. I think those are because they're also going

159
00:09:18,756 --> 00:09:20,676
Speaker 1: to pick out features. They also know what your zip

160
00:09:20,676 --> 00:09:22,996
Speaker 1: code is. If they take up zip code, then at

161
00:09:23,036 --> 00:09:25,476
Speaker 1: least implicity, they're also recognizing race, you know, I mean,

162
00:09:25,836 --> 00:09:27,756
Speaker 1: that's right. So I'll just take my case. I'm an

163
00:09:27,756 --> 00:09:30,916
Speaker 1: African American woman who's middle age, who loves to watch

164
00:09:30,956 --> 00:09:33,036
Speaker 1: you know, black romance Flix, and let me tell you,

165
00:09:33,036 --> 00:09:35,836
Speaker 1: every time Netflix recommends one, I'm happy. I mean, the

166
00:09:35,836 --> 00:09:38,076
Speaker 1: only problem I get, you know, challenged by it is

167
00:09:38,116 --> 00:09:40,636
Speaker 1: when the content runs out and not investing in, you know,

168
00:09:40,756 --> 00:09:44,436
Speaker 1: more more programmers or developers to develop more content for

169
00:09:44,476 --> 00:09:46,716
Speaker 1: people like me. But maybe that's because they're not feeding

170
00:09:46,716 --> 00:09:48,636
Speaker 1: those two people like me, right, I'm a middle aged

171
00:09:48,676 --> 00:09:51,316
Speaker 1: white guy. They're not telling me to watch those, But

172
00:09:51,396 --> 00:09:54,476
Speaker 1: maybe if they did, I'd watch them, I'd like them,

173
00:09:54,596 --> 00:09:56,516
Speaker 1: and if that happened, there would be developers. So in

174
00:09:56,516 --> 00:09:58,876
Speaker 1: that sense, you know, it may be that that there

175
00:09:58,876 --> 00:10:00,836
Speaker 1: is an implicit bias there. They're just assuming that the

176
00:10:00,996 --> 00:10:02,996
Speaker 1: reason that I don't watch them is that I haven't

177
00:10:02,996 --> 00:10:06,756
Speaker 1: watched that many you know, African American romantic comedies in

178
00:10:06,556 --> 00:10:09,796
Speaker 1: recent years and so. But you know, but itself is

179
00:10:09,836 --> 00:10:12,516
Speaker 1: not a neutral fact, right, because no one's advertising them

180
00:10:12,556 --> 00:10:14,556
Speaker 1: to me. Netflix isn't telling me to watch them. If

181
00:10:14,556 --> 00:10:16,756
Speaker 1: they told me, that might have a different impact. That's right,

182
00:10:16,796 --> 00:10:18,836
Speaker 1: And that's I mean, I always say to people. When

183
00:10:19,716 --> 00:10:23,156
Speaker 1: the alright movement and the white conservative movement became a

184
00:10:23,196 --> 00:10:26,356
Speaker 1: big thing and hate speech on Facebook, I was kind

185
00:10:26,356 --> 00:10:28,276
Speaker 1: of surprised that I didn't know that this was happening

186
00:10:28,316 --> 00:10:31,156
Speaker 1: in the same playground in which I also, you know, visit.

187
00:10:31,396 --> 00:10:33,876
Speaker 1: And that's because my algorithm or is not made of

188
00:10:34,556 --> 00:10:37,796
Speaker 1: white supremacists. It's you know, more liberals that sort of

189
00:10:37,836 --> 00:10:40,156
Speaker 1: speak the same language and feel the same way about

190
00:10:40,156 --> 00:10:43,436
Speaker 1: certain issues. What about the hardcase nicle, I mean, you know,

191
00:10:43,516 --> 00:10:47,476
Speaker 1: let's say I'm applying for credit and they've got information,

192
00:10:47,796 --> 00:10:49,716
Speaker 1: you know that Let's say I have allowed I didn't

193
00:10:49,796 --> 00:10:52,156
Speaker 1: check the box to make it private. That says, hey,

194
00:10:52,236 --> 00:10:56,756
Speaker 1: Feldman's been searching for payday loans, and the algorithm notices

195
00:10:56,836 --> 00:10:59,156
Speaker 1: something that is intuitively very plausible, which is that if

196
00:10:59,156 --> 00:11:02,476
Speaker 1: I'm so desperate that I'm looking into payday loans, probably

197
00:11:02,636 --> 00:11:05,476
Speaker 1: that means I'm a slightly less good credit risk than

198
00:11:05,556 --> 00:11:08,116
Speaker 1: someone who hasn't been yet searching for payday loans, because

199
00:11:08,116 --> 00:11:09,756
Speaker 1: you know, eventually, I'm going to start looking for that

200
00:11:09,756 --> 00:11:12,436
Speaker 1: the minute I really need it. So if I'm trying

201
00:11:12,436 --> 00:11:15,636
Speaker 1: to lend money, and that's there's nothing inherently racially determinative

202
00:11:15,796 --> 00:11:18,276
Speaker 1: about that. It may not even be about wealth in general.

203
00:11:18,276 --> 00:11:20,116
Speaker 1: It's just about how much money I have right this minute,

204
00:11:20,196 --> 00:11:22,676
Speaker 1: or how little I have right this minute. That might

205
00:11:22,716 --> 00:11:25,996
Speaker 1: be great from the standpoint of the credit company, and

206
00:11:26,036 --> 00:11:28,156
Speaker 1: they might actually be able to do a better job

207
00:11:28,556 --> 00:11:31,796
Speaker 1: of setting the correct interest rate for me based on

208
00:11:31,876 --> 00:11:35,596
Speaker 1: that information. Does that still disturbute? Does it still make

209
00:11:35,636 --> 00:11:36,956
Speaker 1: you think that that's a problem or is that more

210
00:11:36,956 --> 00:11:38,956
Speaker 1: like Netflix? It's not that big a deal they're telling

211
00:11:38,996 --> 00:11:42,036
Speaker 1: you know, They're they're fitting the data to the objective.

212
00:11:42,476 --> 00:11:44,316
Speaker 1: You know. I think it's interesting because I struggle with it.

213
00:11:44,356 --> 00:11:47,236
Speaker 1: I think that there are cases where, you know, our

214
00:11:47,356 --> 00:11:50,996
Speaker 1: online behavior will indicate certain characteristics about us. Though it

215
00:11:51,036 --> 00:11:53,196
Speaker 1: is somewhat problematic because what if I was searching for

216
00:11:53,236 --> 00:11:56,196
Speaker 1: the payday loan for my uncle? Right, not necessarily for myself,

217
00:11:57,156 --> 00:11:59,596
Speaker 1: But in the end, I think we have to be

218
00:11:59,676 --> 00:12:03,476
Speaker 1: very sensitive, or the algorithmic operator has to be sensitive

219
00:12:03,556 --> 00:12:05,876
Speaker 1: to the extent to which they're denying credit to these

220
00:12:05,916 --> 00:12:09,756
Speaker 1: groups versus out the groups. Right. One thing, for example,

221
00:12:09,756 --> 00:12:12,196
Speaker 1: we say in the paper, which I think is just profound, is,

222
00:12:12,236 --> 00:12:15,956
Speaker 1: as an African American who maybe serve more higher interest

223
00:12:16,036 --> 00:12:19,916
Speaker 1: credit card rates, what if I see that ad come

224
00:12:19,996 --> 00:12:22,516
Speaker 1: through and I click it just because I'm interested to

225
00:12:22,516 --> 00:12:25,716
Speaker 1: see why I'm getting this ad. Automatically, I will be

226
00:12:25,756 --> 00:12:28,956
Speaker 1: served similar ads, right, so it automatically places me in

227
00:12:28,956 --> 00:12:32,596
Speaker 1: that high credit risk category. The challenge that we're having now, Noah,

228
00:12:32,636 --> 00:12:34,996
Speaker 1: is that as an individual consumer, I have no way

229
00:12:35,036 --> 00:12:38,956
Speaker 1: of recurating what my identity is. I want to ask

230
00:12:38,996 --> 00:12:41,596
Speaker 1: you a kind of final big picture question, and it's

231
00:12:41,756 --> 00:12:45,156
Speaker 1: when you survey this whole environment, the possibilities of regulation,

232
00:12:45,196 --> 00:12:48,116
Speaker 1: you're testifying on the hill about it, you're doing reports

233
00:12:48,156 --> 00:12:53,676
Speaker 1: on it. Are you in general optimistic about the future

234
00:12:53,796 --> 00:12:57,676
Speaker 1: of the possibility to regulate algorithms and to also turn

235
00:12:57,716 --> 00:13:01,596
Speaker 1: algorithms to good with respect to fairness and equality, or

236
00:13:01,636 --> 00:13:05,716
Speaker 1: are you, on balance pessimistic and think that the terrible

237
00:13:05,916 --> 00:13:08,836
Speaker 1: legacies of discrimination that we have in our country are

238
00:13:08,916 --> 00:13:12,876
Speaker 1: like just to be either continued or even made worse

239
00:13:13,116 --> 00:13:16,396
Speaker 1: by virtue of this technological development. You know, I'm a

240
00:13:16,436 --> 00:13:21,876
Speaker 1: technologist who's optimistic about the use of technology, you know,

241
00:13:21,956 --> 00:13:25,516
Speaker 1: I think of it this way. I think as technology evolves,

242
00:13:25,596 --> 00:13:28,436
Speaker 1: we are faced with this challenge whether or not the

243
00:13:28,516 --> 00:13:31,676
Speaker 1: technology coopts the user or the user has something to

244
00:13:31,716 --> 00:13:35,476
Speaker 1: do with the technologies agency, right, And so I'm one

245
00:13:35,516 --> 00:13:38,756
Speaker 1: of those people, particularly in this case of algorithms, which

246
00:13:38,796 --> 00:13:42,476
Speaker 1: has just become so interesting to many of us because

247
00:13:42,556 --> 00:13:45,156
Speaker 1: it's got this explainability portion, and then it has stuff

248
00:13:45,156 --> 00:13:47,996
Speaker 1: that we don't even know how to dissect and unpack that.

249
00:13:48,116 --> 00:13:50,476
Speaker 1: I think what we're trying to do in this particular case,

250
00:13:50,556 --> 00:13:52,756
Speaker 1: Noah's just get ahead of it and to be much

251
00:13:52,796 --> 00:13:55,556
Speaker 1: more proactive in talking about it. I mean, my goal

252
00:13:56,276 --> 00:13:59,676
Speaker 1: is to bring to the forefront those algorithms that are

253
00:13:59,676 --> 00:14:03,676
Speaker 1: allowing older Americans to age in place, those algorithms that

254
00:14:03,676 --> 00:14:08,636
Speaker 1: are catching chronic disease and some of the worst abilitating diseases,

255
00:14:08,676 --> 00:14:12,516
Speaker 1: and as ahead of time because of the precision of

256
00:14:12,556 --> 00:14:17,236
Speaker 1: the technology, we're seeing, you know, better customization of educational

257
00:14:17,276 --> 00:14:21,036
Speaker 1: curricula for students because algorithms are able to identify learning

258
00:14:21,076 --> 00:14:24,356
Speaker 1: styles much faster than a teacher can. And so I

259
00:14:24,356 --> 00:14:27,436
Speaker 1: don't want us to be a society which turns our

260
00:14:27,476 --> 00:14:31,276
Speaker 1: back against the technology and the innovation, because that's part

261
00:14:31,316 --> 00:14:34,996
Speaker 1: of this whole new revolution of our shift for manufacturing

262
00:14:35,036 --> 00:14:37,956
Speaker 1: into I think this digital age where it does matter.

263
00:14:38,236 --> 00:14:41,116
Speaker 1: I'm gonna tell you honestly, what really concerns me is

264
00:14:41,156 --> 00:14:44,316
Speaker 1: the fact that the less information or the less diffused

265
00:14:44,396 --> 00:14:47,076
Speaker 1: that these algorithms are, the more likely you'll be on

266
00:14:47,196 --> 00:14:50,396
Speaker 1: the wrong side of digital opportunity, and the more likely

267
00:14:50,436 --> 00:14:52,716
Speaker 1: that your community may not get some of the services

268
00:14:52,716 --> 00:14:55,156
Speaker 1: that have come out of an algorithmic economy. I mean,

269
00:14:55,236 --> 00:14:57,756
Speaker 1: imagine living in a community where they don't have your data.

270
00:14:58,156 --> 00:15:00,396
Speaker 1: You're not your data is not being harnessed for any

271
00:15:00,396 --> 00:15:03,516
Speaker 1: type of productive algorithm. You find yourself in a state

272
00:15:03,516 --> 00:15:07,276
Speaker 1: where you have more chronic disease, more incarceration, less levels

273
00:15:07,276 --> 00:15:10,356
Speaker 1: of educational achievement. Better to be then to be out. Yeah,

274
00:15:10,396 --> 00:15:12,196
Speaker 1: and I'll just say this in final I mean, I

275
00:15:12,236 --> 00:15:15,196
Speaker 1: think for those of us that are in this space,

276
00:15:15,276 --> 00:15:18,956
Speaker 1: I think we take into consideration the fairness and accuracy

277
00:15:18,956 --> 00:15:22,356
Speaker 1: conversations as well the ethical conversations, But our main goal

278
00:15:22,436 --> 00:15:25,476
Speaker 1: is to deploy this responsibly. And if we can come

279
00:15:25,516 --> 00:15:28,316
Speaker 1: up with more responsible frameworks that incorporate many of the

280
00:15:28,356 --> 00:15:31,156
Speaker 1: aspects that we've talked about today, I think we're on

281
00:15:31,196 --> 00:15:35,156
Speaker 1: the brink of actually unpacking what could potentially become the

282
00:15:35,236 --> 00:15:38,956
Speaker 1: next big game changer for people you know that have

283
00:15:39,116 --> 00:15:42,796
Speaker 1: had to rely upon wrong decisions or humans who are

284
00:15:42,836 --> 00:15:45,076
Speaker 1: biased to do such. So I want to say, you know,

285
00:15:45,116 --> 00:15:46,996
Speaker 1: in all honesty, I do agree with you that there's

286
00:15:46,996 --> 00:15:50,676
Speaker 1: a promise of algorithms to sort of break down the barriers,

287
00:15:50,996 --> 00:15:53,036
Speaker 1: but it has to be done responsibly with the right

288
00:15:53,076 --> 00:15:56,436
Speaker 1: people at the table to talk about it. Niculternally thank

289
00:15:56,436 --> 00:15:59,356
Speaker 1: you so much for a really fascinating and rich discussion

290
00:15:59,476 --> 00:16:01,836
Speaker 1: and for sharing your knowledge and expertise with us. Thank

291
00:16:01,876 --> 00:16:18,996
Speaker 1: you now, thank you for having me appreciate you. My

292
00:16:19,076 --> 00:16:21,556
Speaker 1: conversation with Nicole made me want to talk to somebody

293
00:16:21,596 --> 00:16:23,316
Speaker 1: who was doing the kind of work that she was

294
00:16:23,356 --> 00:16:26,636
Speaker 1: just talking about, someone who was thinking about how algorithms

295
00:16:26,676 --> 00:16:30,756
Speaker 1: can break down barriers rather than create them. So I

296
00:16:30,796 --> 00:16:34,156
Speaker 1: called up Talia Gillis. Talia is a PhD student in

297
00:16:34,156 --> 00:16:37,636
Speaker 1: a business economics at Harvard. She's also a former student

298
00:16:37,676 --> 00:16:40,396
Speaker 1: of mine who holds a law degree from Harvard, and

299
00:16:40,476 --> 00:16:44,476
Speaker 1: she's been researching how banks and other lenders use algorithms

300
00:16:44,476 --> 00:16:47,676
Speaker 1: to determine interest rates on loans. She thinks that the

301
00:16:47,716 --> 00:16:51,476
Speaker 1: way they're doing it right now isn't working, but she

302
00:16:51,596 --> 00:16:55,516
Speaker 1: has an idea for a better way. Talia, thank you

303
00:16:55,596 --> 00:16:58,436
Speaker 1: so much for joining me. It's great to have you,

304
00:16:58,676 --> 00:17:01,316
Speaker 1: and it's great to talk to you as it were

305
00:17:01,796 --> 00:17:04,756
Speaker 1: on air about something that we've talked about lots and

306
00:17:04,796 --> 00:17:09,596
Speaker 1: lots of times in the office, your research, because there's

307
00:17:09,676 --> 00:17:12,796
Speaker 1: very much on how we can fix the problem of

308
00:17:12,876 --> 00:17:17,756
Speaker 1: algorithmic bias. Tell me what it is that is the

309
00:17:17,796 --> 00:17:21,476
Speaker 1: core of your approach. What is your original idea about

310
00:17:21,516 --> 00:17:23,956
Speaker 1: what we can do to make things better. So I

311
00:17:23,996 --> 00:17:26,516
Speaker 1: think the core of the approach is, first of all,

312
00:17:26,556 --> 00:17:29,236
Speaker 1: to recognize that it's it's very hard to know a

313
00:17:29,356 --> 00:17:33,756
Speaker 1: priori what exactly the bias or what direction the bias

314
00:17:33,836 --> 00:17:35,676
Speaker 1: is going to go in, and so I'm very much

315
00:17:35,716 --> 00:17:39,276
Speaker 1: focused on the credit pricing context. And in the credit

316
00:17:39,276 --> 00:17:43,036
Speaker 1: pricing context, it's true that a lot of the kind

317
00:17:43,036 --> 00:17:47,156
Speaker 1: of input variables into a credit pricing decision suffer from

318
00:17:47,196 --> 00:17:49,716
Speaker 1: some sort of bias. But what's important to keep in

319
00:17:49,756 --> 00:17:53,876
Speaker 1: mind is that kind of some biases might get worse

320
00:17:53,916 --> 00:17:57,996
Speaker 1: in the algorithmic context, but actually big data might in

321
00:17:58,236 --> 00:18:00,876
Speaker 1: for other types of biases, make things things better in

322
00:18:00,916 --> 00:18:05,196
Speaker 1: a way. I think there's two large, separate categories of bias.

323
00:18:05,316 --> 00:18:08,156
Speaker 1: So the first is what I call kind of inputs

324
00:18:08,156 --> 00:18:10,636
Speaker 1: that result from a bias world, and the idea there

325
00:18:10,756 --> 00:18:14,316
Speaker 1: is that there's some kind of pre existing discrimination, and

326
00:18:14,356 --> 00:18:17,076
Speaker 1: so there might be disparities between men and women, or

327
00:18:17,116 --> 00:18:20,796
Speaker 1: between blacks and whites that kind of originate partially from

328
00:18:20,836 --> 00:18:24,076
Speaker 1: that discrimination. So that's what you're calling biased world, and

329
00:18:24,116 --> 00:18:26,796
Speaker 1: that is you're going to apply for a loan. If

330
00:18:26,796 --> 00:18:28,676
Speaker 1: you make less money and you have more debt, you're

331
00:18:28,676 --> 00:18:30,396
Speaker 1: not going to get as good terms for the loan.

332
00:18:30,876 --> 00:18:34,396
Speaker 1: But that's not because the lender in particular, that's because

333
00:18:34,876 --> 00:18:37,116
Speaker 1: you live in a society where there's background sexist and

334
00:18:37,116 --> 00:18:40,516
Speaker 1: there's background racism. The world is already biased. And so

335
00:18:40,556 --> 00:18:42,476
Speaker 1: in that sense, that's biased world. And then what's the

336
00:18:42,476 --> 00:18:46,436
Speaker 1: second category? And so the second category is inputs that

337
00:18:46,516 --> 00:18:49,476
Speaker 1: are bias because they result from some kind of bias measurement.

338
00:18:50,156 --> 00:18:52,796
Speaker 1: You can think of that as, for example, the way

339
00:18:52,876 --> 00:18:55,596
Speaker 1: we measure someone's income. You know, we might put a

340
00:18:55,636 --> 00:18:58,756
Speaker 1: lot of weight on someone who has one regular job,

341
00:18:58,876 --> 00:19:03,796
Speaker 1: regular paycheck, and we fully capture their income and compare

342
00:19:03,836 --> 00:19:06,836
Speaker 1: that to someone who kind of has multiple jobs, maybe

343
00:19:07,116 --> 00:19:11,396
Speaker 1: isn't in a formal employee employer relationship, like an uber driver,

344
00:19:11,916 --> 00:19:15,116
Speaker 1: and then we kind of discount their income or don't

345
00:19:15,196 --> 00:19:18,356
Speaker 1: measure it properly, or or don't have the ability to

346
00:19:18,396 --> 00:19:21,956
Speaker 1: fully capture what they're earning. And so the more you

347
00:19:21,996 --> 00:19:25,116
Speaker 1: might look at two people who they're underlying income is similar,

348
00:19:25,476 --> 00:19:27,996
Speaker 1: but because of the way we're measuring a person's income,

349
00:19:28,356 --> 00:19:31,516
Speaker 1: then we consider kind of the second person to have

350
00:19:31,596 --> 00:19:34,516
Speaker 1: a lower income. So that's an example of bias in

351
00:19:34,516 --> 00:19:36,716
Speaker 1: the way we're measuring where we're whether we mean to

352
00:19:36,916 --> 00:19:39,276
Speaker 1: or not, we're systematically giving an advantage to someone who

353
00:19:39,316 --> 00:19:41,516
Speaker 1: works in a nine to five as opposed to someone

354
00:19:41,516 --> 00:19:43,956
Speaker 1: who's in the gig economy. And that's what you're calling

355
00:19:44,156 --> 00:19:47,476
Speaker 1: bias in measurement. And now how would you go about

356
00:19:47,596 --> 00:19:50,276
Speaker 1: measuring which kind of bias or what kind of bias

357
00:19:50,596 --> 00:19:54,996
Speaker 1: is in fact found in the algorithm. So it's it's

358
00:19:55,076 --> 00:19:59,356
Speaker 1: quite difficult to in reality perfectly distinguish between these two biases.

359
00:19:59,836 --> 00:20:03,076
Speaker 1: Also because very often, like an example I gave with income,

360
00:20:03,156 --> 00:20:05,956
Speaker 1: it might be a combination of those two. And what

361
00:20:05,996 --> 00:20:08,156
Speaker 1: do you do if you're in not biased world, but

362
00:20:08,356 --> 00:20:11,476
Speaker 1: biased German situation where you're worried not about the backround

363
00:20:11,476 --> 00:20:14,716
Speaker 1: discrimination in the world, but more worried that you're measuring

364
00:20:14,716 --> 00:20:17,516
Speaker 1: the wrong things in the algorithm and as a result,

365
00:20:18,236 --> 00:20:22,236
Speaker 1: you know, having a bad effect on their community of color.

366
00:20:23,276 --> 00:20:28,676
Speaker 1: So with bias measurement, what's interesting about the algorithmic context

367
00:20:28,876 --> 00:20:31,276
Speaker 1: is that it actually might mitigate a lot of the

368
00:20:31,316 --> 00:20:35,236
Speaker 1: harms that we're concerned about in the context of bias measurement.

369
00:20:35,876 --> 00:20:39,196
Speaker 1: So if you take, for example, credit scores, there have

370
00:20:39,236 --> 00:20:42,276
Speaker 1: been many claims that credit scores are biased against minorities,

371
00:20:42,316 --> 00:20:46,436
Speaker 1: and that's because they measure certain qualities of credit worthiness

372
00:20:46,516 --> 00:20:50,356
Speaker 1: that are more representative of let's say, white borrowers. So

373
00:20:50,756 --> 00:20:54,316
Speaker 1: it puts a lot of weight on people repaying previous

374
00:20:54,356 --> 00:20:56,676
Speaker 1: loans on time, but it might not give any weight

375
00:20:56,716 --> 00:21:00,236
Speaker 1: to people who regularly made let's say rent payments, which

376
00:21:00,316 --> 00:21:02,876
Speaker 1: might actually also be a very good measure of a

377
00:21:02,916 --> 00:21:06,756
Speaker 1: person's credit worthiness. So in a world in which we

378
00:21:06,796 --> 00:21:09,596
Speaker 1: put a lot a lot of weight on a credit score,

379
00:21:10,396 --> 00:21:12,556
Speaker 1: if we moved to a world of kind of machine

380
00:21:12,596 --> 00:21:15,276
Speaker 1: learning and big data, we might get a whole new

381
00:21:15,356 --> 00:21:19,076
Speaker 1: richness of indicators of a person's credit worthiness. So let's

382
00:21:19,076 --> 00:21:22,036
Speaker 1: say the algorithm, how did your full history of payments

383
00:21:22,196 --> 00:21:26,036
Speaker 1: or your full kind of consumer history. Then we might

384
00:21:26,076 --> 00:21:29,116
Speaker 1: be getting a lot more information out about a person's

385
00:21:29,156 --> 00:21:32,916
Speaker 1: credit worthiness that was before only limited to the credit score.

386
00:21:33,036 --> 00:21:35,236
Speaker 1: So this is an example where if we could identify

387
00:21:35,916 --> 00:21:38,756
Speaker 1: the bias in the measurement, then we could do better

388
00:21:38,876 --> 00:21:42,196
Speaker 1: with the algorithm. Yes, what about a situation where the

389
00:21:42,236 --> 00:21:44,796
Speaker 1: opposite is happening where the algorithm is taking into account

390
00:21:44,836 --> 00:21:47,956
Speaker 1: things that are producing measurement bias. How do we know

391
00:21:48,036 --> 00:21:51,236
Speaker 1: that that's happening. So I think that the key is

392
00:21:51,276 --> 00:21:55,076
Speaker 1: that we never truly know what's going on. Say more

393
00:21:55,076 --> 00:21:56,516
Speaker 1: about that, because I think that scares a lot of

394
00:21:56,516 --> 00:21:58,876
Speaker 1: people with respect to these algorithms. What does it mean

395
00:21:58,876 --> 00:22:01,756
Speaker 1: to say we never truly know what's going on? Well,

396
00:22:02,116 --> 00:22:03,796
Speaker 1: on the one hand, you could say it's scary, But

397
00:22:03,836 --> 00:22:05,876
Speaker 1: on the other hand, you could say that any attempt

398
00:22:05,876 --> 00:22:09,956
Speaker 1: to say it's necessarily bad or necessarily going to hurt populations.

399
00:22:10,556 --> 00:22:13,116
Speaker 1: It's going to be a difficult position to defend because

400
00:22:13,396 --> 00:22:16,316
Speaker 1: it's kind of more of an empirical question that requires

401
00:22:16,356 --> 00:22:19,396
Speaker 1: investigation rather than something that you can determine ahead of time.

402
00:22:20,036 --> 00:22:22,556
Speaker 1: Now tell you that sounds logically correct to me. You know,

403
00:22:22,596 --> 00:22:25,076
Speaker 1: you to You don't know for sure something's bad or

404
00:22:25,076 --> 00:22:26,676
Speaker 1: good until you test it out and you have to

405
00:22:26,836 --> 00:22:29,356
Speaker 1: examine it, and principle I agree with you. But what

406
00:22:29,356 --> 00:22:32,716
Speaker 1: would you say to someone who said, well, look, we

407
00:22:32,756 --> 00:22:34,636
Speaker 1: know how the world works in general, and the world

408
00:22:34,676 --> 00:22:39,876
Speaker 1: doesn't turn out so well. Often traditionally discriminated against groups,

409
00:22:39,956 --> 00:22:44,756
Speaker 1: and so our instinct is that we expect to find

410
00:22:45,156 --> 00:22:51,116
Speaker 1: discrimination rather than to find magic, whereby an algorithmic measurement

411
00:22:51,156 --> 00:22:54,116
Speaker 1: does better than a human How would you respond to

412
00:22:54,156 --> 00:22:57,636
Speaker 1: that kind of systemic skepticism that I think one very

413
00:22:57,676 --> 00:23:01,036
Speaker 1: reasonably hears from people who are concerned that existing bias

414
00:23:01,076 --> 00:23:05,396
Speaker 1: will be made worse by algorithms, rather than being optimistic

415
00:23:05,436 --> 00:23:08,996
Speaker 1: about the capacity of algorithms to block certain kinds of bias.

416
00:23:10,316 --> 00:23:12,836
Speaker 1: You'd have to, I mean, particularly in the credit context,

417
00:23:12,996 --> 00:23:15,356
Speaker 1: you'd have to be very sensitive to the fact that

418
00:23:15,556 --> 00:23:19,396
Speaker 1: credit markets are not working for large segments of the

419
00:23:19,516 --> 00:23:22,916
Speaker 1: US population. So many people in the US don't have

420
00:23:22,996 --> 00:23:26,596
Speaker 1: access to credit. Many people don't have credit histories, they

421
00:23:26,596 --> 00:23:30,476
Speaker 1: don't have credit scores. So if you were defending the

422
00:23:30,556 --> 00:23:34,116
Speaker 1: status quo and credit pricing, then you would have a

423
00:23:34,196 --> 00:23:39,076
Speaker 1: really big difficulty in terms of actually kind of blocking

424
00:23:39,076 --> 00:23:42,836
Speaker 1: the potential that this technological move has in terms of

425
00:23:42,876 --> 00:23:46,716
Speaker 1: expanding access and creating kind of access to credit markets

426
00:23:46,756 --> 00:23:50,676
Speaker 1: to populations that before have just simply been excluded from

427
00:23:50,676 --> 00:23:53,396
Speaker 1: these markets. So things are so bad right now, you're

428
00:23:53,436 --> 00:23:55,396
Speaker 1: saying that it would be crazy enough to at least

429
00:23:55,436 --> 00:23:58,756
Speaker 1: give this or try because the existing stays quo is

430
00:23:58,996 --> 00:24:02,516
Speaker 1: deeply discriminatory yeah, I think there's this kind of a

431
00:24:02,596 --> 00:24:06,956
Speaker 1: serious entrenchment of pre existing disadvantage in credit markets. And

432
00:24:06,956 --> 00:24:09,916
Speaker 1: when you think that credit markets are very important tool

433
00:24:10,036 --> 00:24:13,956
Speaker 1: not just to kind of not want to replicate disadvantage,

434
00:24:13,996 --> 00:24:17,076
Speaker 1: but also credit markets play a very important role at

435
00:24:17,156 --> 00:24:20,996
Speaker 1: producing wealth or allowing people to kind of come out

436
00:24:21,036 --> 00:24:23,996
Speaker 1: of some kind of situation in which they were blocked

437
00:24:24,036 --> 00:24:27,156
Speaker 1: from kind of expanding their possibilities, because if you can't

438
00:24:27,156 --> 00:24:31,996
Speaker 1: borrow money, you can invest in yourself exactly, exactly right, Okay,

439
00:24:31,996 --> 00:24:35,796
Speaker 1: So go back then to the question of how we

440
00:24:35,876 --> 00:24:40,516
Speaker 1: run a test to see whether we've got biased measurement.

441
00:24:40,676 --> 00:24:43,436
Speaker 1: How do we make sure that we're making things better

442
00:24:43,476 --> 00:24:45,956
Speaker 1: with the algorithm with respect to measurement bias, not making

443
00:24:45,956 --> 00:24:47,876
Speaker 1: things worse. How would you test that in real world?

444
00:24:48,636 --> 00:24:51,356
Speaker 1: So I think what's key is to have a kind

445
00:24:51,396 --> 00:24:54,116
Speaker 1: of baseline in which the key question is when I

446
00:24:54,196 --> 00:24:57,516
Speaker 1: move from that baseline to a new situation, how are

447
00:24:57,556 --> 00:25:01,436
Speaker 1: things changing. And so the key question to me is,

448
00:25:01,516 --> 00:25:04,756
Speaker 1: if we're comparing kind of a traditional pricing situation to

449
00:25:04,796 --> 00:25:08,716
Speaker 1: an algorithmic pricing situation, what's happening? And to do that

450
00:25:09,156 --> 00:25:10,916
Speaker 1: I would do is I would take kind of the

451
00:25:10,956 --> 00:25:15,076
Speaker 1: algorithmic pricing function. And again, the big advantage in a

452
00:25:15,116 --> 00:25:19,236
Speaker 1: way of the algorithmic context is that even before you

453
00:25:19,316 --> 00:25:23,236
Speaker 1: actually apply your decision rule or your prediction to a

454
00:25:23,276 --> 00:25:25,956
Speaker 1: new borrower who comes through the door, you're able to

455
00:25:25,996 --> 00:25:30,836
Speaker 1: say something about the algorithm itself. So it sounds like overall,

456
00:25:30,916 --> 00:25:35,556
Speaker 1: the key tool of social science that you think needs

457
00:25:35,556 --> 00:25:39,076
Speaker 1: to be used to help us overcome the possibilities of

458
00:25:39,076 --> 00:25:44,196
Speaker 1: different kinds of algorithmic bias is the experiment. It's experiment

459
00:25:44,276 --> 00:25:47,836
Speaker 1: by setting a baseline, then experiment and see what happens

460
00:25:47,836 --> 00:25:51,716
Speaker 1: when the algorithm is applied, and then compare them and

461
00:25:51,756 --> 00:25:53,996
Speaker 1: then make a judgment afterwards. It sounds like you're saying,

462
00:25:54,276 --> 00:25:58,396
Speaker 1: we never know from just looking at an algorithm what's

463
00:25:58,396 --> 00:25:59,876
Speaker 1: going to happen, whether it's going to make the world

464
00:25:59,916 --> 00:26:01,476
Speaker 1: a worst place, where that's going to make the world

465
00:26:01,476 --> 00:26:04,316
Speaker 1: a better place. We always have to test it. And

466
00:26:04,476 --> 00:26:06,836
Speaker 1: in a sense that seems to me very scientific, right,

467
00:26:06,956 --> 00:26:10,516
Speaker 1: very economic scientific. Run the experiment and see see what

468
00:26:10,596 --> 00:26:13,716
Speaker 1: comes out on the other side. Do other people agree

469
00:26:13,756 --> 00:26:15,676
Speaker 1: with you? I mean, how far out are you on

470
00:26:15,716 --> 00:26:19,236
Speaker 1: the edge and calling for experiment in every case? Well,

471
00:26:19,276 --> 00:26:22,756
Speaker 1: I think there's several difficulties. I think the first difficulty

472
00:26:23,156 --> 00:26:26,636
Speaker 1: is kind of maybe a legal theoretical difficulty, and that

473
00:26:26,836 --> 00:26:30,836
Speaker 1: is that traditionally, the way we've always thought about kind

474
00:26:30,836 --> 00:26:35,796
Speaker 1: of discrimination and evaluating a lender for discrimination purposes was

475
00:26:35,916 --> 00:26:38,836
Speaker 1: kind of the exact opposite. It was considering what are

476
00:26:38,876 --> 00:26:42,556
Speaker 1: the inputs into the decision to price alone, and not

477
00:26:42,596 --> 00:26:45,156
Speaker 1: what's the outcome. So this whole way of thinking about

478
00:26:45,236 --> 00:26:48,076
Speaker 1: testing is very focused on the outcome of a pricing rule.

479
00:26:48,316 --> 00:26:51,676
Speaker 1: So in legal terms, instead of asking about discriminatory intent

480
00:26:52,196 --> 00:26:55,156
Speaker 1: by the person making the decision, you're asking about whether

481
00:26:55,156 --> 00:26:58,836
Speaker 1: there's a disparate impact on people at the end and

482
00:26:58,876 --> 00:27:02,036
Speaker 1: the outcomes. That's right, Do you want us to focus

483
00:27:02,116 --> 00:27:06,036
Speaker 1: on outputs? Exactly? Exactly? So there's quite kind of this

484
00:27:06,156 --> 00:27:08,956
Speaker 1: fundamental shift that I think needs to take place in

485
00:27:09,276 --> 00:27:12,316
Speaker 1: moving from being very focused on what poes into a

486
00:27:12,356 --> 00:27:15,636
Speaker 1: decision or what poes into an algorithm and saying there's

487
00:27:15,676 --> 00:27:19,236
Speaker 1: not much progress that we can make on focusing just

488
00:27:19,356 --> 00:27:21,396
Speaker 1: on the inputs. We really need to go to the

489
00:27:21,436 --> 00:27:31,436
Speaker 1: outcomes and consider the outcomes more seriously. I think Talia's

490
00:27:31,516 --> 00:27:36,196
Speaker 1: idea for fixing algorithmic bias has some profound implications. One

491
00:27:36,276 --> 00:27:39,636
Speaker 1: of the things about algorithms is you can't really know

492
00:27:39,676 --> 00:27:41,996
Speaker 1: what all the inputs are because often, in the case

493
00:27:42,036 --> 00:27:45,076
Speaker 1: of a sophisticated algorithm which is based on machine learning,

494
00:27:45,596 --> 00:27:48,196
Speaker 1: we don't know how it's learning from the data. We

495
00:27:48,316 --> 00:27:50,676
Speaker 1: just know that it's looking at every aspect of the data,

496
00:27:50,716 --> 00:27:53,316
Speaker 1: but we don't know exactly what its true inputs are.

497
00:27:53,956 --> 00:27:57,196
Speaker 1: In other words, we know what data it's training on,

498
00:27:57,476 --> 00:27:59,796
Speaker 1: but we don't know what features of the data it

499
00:27:59,916 --> 00:28:03,196
Speaker 1: cares the most about. And that's why Talia sees her

500
00:28:03,236 --> 00:28:06,076
Speaker 1: approach of running experiments and looking at the outputs as

501
00:28:06,116 --> 00:28:11,276
Speaker 1: the only potential solution. She's if people begin gradually to

502
00:28:11,276 --> 00:28:13,716
Speaker 1: see things the way that Taya does, I wonder if

503
00:28:13,716 --> 00:28:16,836
Speaker 1: that could lead us to a new paradigm more broadly

504
00:28:16,836 --> 00:28:19,716
Speaker 1: about how we think of discrimination. It might lead us

505
00:28:19,756 --> 00:28:22,636
Speaker 1: away from the old way of asking, well, was the

506
00:28:22,676 --> 00:28:25,476
Speaker 1: person of making the decision or racist, and towards a

507
00:28:25,516 --> 00:28:28,196
Speaker 1: newer way of thinking, which says, who cares what the

508
00:28:28,236 --> 00:28:30,516
Speaker 1: person was thinking about? What we want to see is

509
00:28:30,516 --> 00:28:33,476
Speaker 1: whether the system as a whole is producing outcomes that

510
00:28:33,596 --> 00:28:37,596
Speaker 1: we think are fair and just. That's all in the future,

511
00:28:37,916 --> 00:28:40,676
Speaker 1: and right now the Trump administration is proposing regulations that

512
00:28:40,756 --> 00:28:44,116
Speaker 1: actually go in the opposite direction. The Department of Housing

513
00:28:44,116 --> 00:28:47,116
Speaker 1: and Urban Development has proposed a new rule that would

514
00:28:47,156 --> 00:28:50,636
Speaker 1: make it harder for banks, or landlords or homeowners insurance

515
00:28:50,636 --> 00:28:53,876
Speaker 1: companies to be sued for using algorithms that result in

516
00:28:53,916 --> 00:28:58,516
Speaker 1: discriminatory lending practices, and the Trump administration has gone to

517
00:28:58,596 --> 00:29:02,116
Speaker 1: the courts more broadly to suggest that they think there

518
00:29:02,156 --> 00:29:05,476
Speaker 1: needs to be a stronger showing of actual racist intent

519
00:29:05,956 --> 00:29:10,476
Speaker 1: before discrimination claims can be leveled. So the trend line

520
00:29:10,876 --> 00:29:14,756
Speaker 1: is not the line that Talia is calling for, nor

521
00:29:14,996 --> 00:29:17,636
Speaker 1: is it the line that machine learning and artificial intelligence

522
00:29:17,796 --> 00:29:20,676
Speaker 1: would suggest for us. In its most extreme form, the

523
00:29:20,716 --> 00:29:25,196
Speaker 1: Trump administration approach might actually allow racist bias to be

524
00:29:25,316 --> 00:29:29,796
Speaker 1: imported into the functioning of algorithmic systems, and exactly the

525
00:29:29,836 --> 00:29:33,476
Speaker 1: way that Nicole wants to avoid. We'll be watching very

526
00:29:33,476 --> 00:29:37,476
Speaker 1: closely going forward to see how those proposed Trump administration

527
00:29:37,516 --> 00:29:41,076
Speaker 1: regulations are treated, how the courts address the question of bias,

528
00:29:41,076 --> 00:30:00,076
Speaker 1: and most profoundly, how algorithms shape justice in the future. Now,

529
00:30:00,116 --> 00:30:02,196
Speaker 1: I want to move to a new segment of deep background,

530
00:30:02,356 --> 00:30:05,756
Speaker 1: something we're calling Sound of the Week. For me, this week,

531
00:30:06,036 --> 00:30:10,236
Speaker 1: a defining moment in sound was this everybody to know.

532
00:30:10,876 --> 00:30:15,556
Speaker 1: There are no circumstances in which I will ask Russels

533
00:30:15,876 --> 00:30:19,316
Speaker 1: to delay. We're leaving on the thirty first of October.

534
00:30:19,636 --> 00:30:24,236
Speaker 1: No ifs or butts. We will not accept any attempt

535
00:30:24,316 --> 00:30:27,876
Speaker 1: to go back on our promises or scrub that referendum.

536
00:30:28,636 --> 00:30:31,996
Speaker 1: That's Boris Johnson on Monday, making a public statement in

537
00:30:32,036 --> 00:30:34,676
Speaker 1: front of ten Downing Street, where for the moment he

538
00:30:34,796 --> 00:30:37,276
Speaker 1: still lives and works as the Prime Minister of the

539
00:30:37,356 --> 00:30:43,156
Speaker 1: United Kingdom. But things have changed a lot since then. First,

540
00:30:43,196 --> 00:30:47,676
Speaker 1: in a remarkable development, the Parliament of Great Britain, including

541
00:30:47,756 --> 00:30:52,436
Speaker 1: a group of rebels from Johnson's own Conservative Party, actually

542
00:30:52,716 --> 00:30:57,436
Speaker 1: voted that the United Kingdom cannot crash out of the

543
00:30:57,476 --> 00:31:01,836
Speaker 1: European Union with a no deal brexit. Johnson will actually

544
00:31:01,876 --> 00:31:06,396
Speaker 1: be required by this law to seek an extension from

545
00:31:06,396 --> 00:31:10,796
Speaker 1: the European Union so that Britain does not leave the

546
00:31:10,876 --> 00:31:15,956
Speaker 1: Union without some kind of a deal. Johnson's position has

547
00:31:15,996 --> 00:31:19,516
Speaker 1: been all along that this would be terrible for his

548
00:31:19,596 --> 00:31:22,556
Speaker 1: negotiating position with the European Union since he's got nothing

549
00:31:22,596 --> 00:31:27,156
Speaker 1: to threaten, but Parliament didn't care. Having lost this vote,

550
00:31:27,436 --> 00:31:30,316
Speaker 1: Johnson then turned around and did two things. One more

551
00:31:30,356 --> 00:31:34,276
Speaker 1: shocking than next. First, he kicked out of the Conservative

552
00:31:34,276 --> 00:31:39,036
Speaker 1: Party twenty one members of Parliament who had voted against him.

553
00:31:39,476 --> 00:31:43,116
Speaker 1: This was sufficiently shocking that his own brother, Joe Johnson,

554
00:31:43,516 --> 00:31:48,436
Speaker 1: actually resigned from his seat in Parliament and from Johnson's cabinet,

555
00:31:48,756 --> 00:31:52,236
Speaker 1: saying that he felt a conflict between the national interest

556
00:31:52,316 --> 00:31:56,476
Speaker 1: and his family loyalties. Then, having taken that radical step,

557
00:31:56,956 --> 00:32:00,636
Speaker 1: Johnson asked Parliament to vote for a snap election. Now

558
00:32:00,676 --> 00:32:03,036
Speaker 1: it takes two thirds of Parliament to vote for a

559
00:32:03,036 --> 00:32:05,636
Speaker 1: snap election for it to happen right away, and the

560
00:32:05,676 --> 00:32:09,076
Speaker 1: Conservatives didn't get it. So where we are now is it?

561
00:32:09,156 --> 00:32:12,316
Speaker 1: Boris Johnson can't get out of the European Union on

562
00:32:12,356 --> 00:32:14,996
Speaker 1: October thirty first, whether he wants to or not. And

563
00:32:15,556 --> 00:32:18,356
Speaker 1: so far, at least he still doesn't have a general

564
00:32:18,396 --> 00:32:20,876
Speaker 1: election in which he could try to ask the voters

565
00:32:21,036 --> 00:32:24,676
Speaker 1: to change the government in order to change this law.

566
00:32:25,716 --> 00:32:31,036
Speaker 1: Stunning developments, historically significant moments in the history of British politics.

567
00:32:31,796 --> 00:32:35,796
Speaker 1: What's their more profound meaning, I'll tell you what's been

568
00:32:35,876 --> 00:32:40,516
Speaker 1: on my mind. There's a deep contradiction between the idea

569
00:32:40,596 --> 00:32:44,036
Speaker 1: of a referendum that would allow the public as a

570
00:32:44,076 --> 00:32:47,036
Speaker 1: whole to decide on an important question like whether to

571
00:32:47,116 --> 00:32:51,276
Speaker 1: leave the European Union and parliamentary democracy, which is based

572
00:32:51,316 --> 00:32:54,676
Speaker 1: on the idea that the people choose representatives who then

573
00:32:54,676 --> 00:32:58,876
Speaker 1: exercise their practical judgment and their wisdom to implement the

574
00:32:58,996 --> 00:33:03,116
Speaker 1: policies of the country. Notice how this contradiction has driven

575
00:33:03,196 --> 00:33:05,516
Speaker 1: Britain into a kind of paralysis that I would say

576
00:33:05,516 --> 00:33:09,436
Speaker 1: even veers occasionally on madness. First, the public says leave,

577
00:33:09,836 --> 00:33:13,356
Speaker 1: but it doesn't say how to leave. Then it tells

578
00:33:13,436 --> 00:33:16,876
Speaker 1: its elected representatives figure out how to do it, and

579
00:33:16,956 --> 00:33:20,876
Speaker 1: they can't figure it out. They can't agree. Proposal after

580
00:33:20,956 --> 00:33:25,276
Speaker 1: proposal gets blocked. Good idea follows bad idea follows bad

581
00:33:25,356 --> 00:33:28,596
Speaker 1: idea follows good idea, and nothing seems to work itself out.

582
00:33:28,796 --> 00:33:31,516
Speaker 1: And the whole time the politicians are saying, well, we

583
00:33:31,596 --> 00:33:33,916
Speaker 1: can't reach an agreement, but we know we have to

584
00:33:33,956 --> 00:33:36,636
Speaker 1: give effect to the will of the people in the referendum.

585
00:33:37,196 --> 00:33:40,276
Speaker 1: This is the product of a mismatch between the idea

586
00:33:40,316 --> 00:33:42,516
Speaker 1: that you can take a snapshot of public opinion at

587
00:33:42,556 --> 00:33:45,276
Speaker 1: a given moment and call that a referendum, and the

588
00:33:45,356 --> 00:33:47,276
Speaker 1: idea that the best way to run a government is

589
00:33:47,276 --> 00:33:51,636
Speaker 1: in fact through electing representatives and have them use their judgment.

590
00:33:52,596 --> 00:33:56,636
Speaker 1: So is there a way out of this contradiction for Britain.

591
00:33:57,316 --> 00:33:59,156
Speaker 1: If I were an optimist, I would say that the

592
00:33:59,196 --> 00:34:01,556
Speaker 1: British could call a new election and that the outcome

593
00:34:01,556 --> 00:34:05,276
Speaker 1: of that election would somehow clarify whether people favored a

594
00:34:05,396 --> 00:34:08,436
Speaker 1: change or not. But I don't actually believe that a

595
00:34:08,476 --> 00:34:11,356
Speaker 1: new election is going to make things any clearer with

596
00:34:11,396 --> 00:34:15,276
Speaker 1: respect to the contradiction between the referendum and ordinary voting.

597
00:34:15,836 --> 00:34:19,036
Speaker 1: So then you might imagine, how about another referendum that

598
00:34:19,076 --> 00:34:24,196
Speaker 1: asks people, well, what did you change your mind? Do

599
00:34:24,276 --> 00:34:27,156
Speaker 1: you want us to leave without a deal? If so,

600
00:34:27,276 --> 00:34:30,156
Speaker 1: what kind of a deal? Notice that almost immediately you

601
00:34:30,236 --> 00:34:33,876
Speaker 1: get into the kind of details that a referendum cannot answer.

602
00:34:34,316 --> 00:34:36,876
Speaker 1: There's no way that a referendum can do anything other

603
00:34:36,876 --> 00:34:39,476
Speaker 1: than ask up or down do you want this or

604
00:34:39,636 --> 00:34:42,876
Speaker 1: do you want that? The only way that that's problem

605
00:34:42,876 --> 00:34:45,476
Speaker 1: could be solved would be if a specific deal were

606
00:34:45,476 --> 00:34:46,956
Speaker 1: put in front of the British people and they were

607
00:34:46,956 --> 00:34:49,116
Speaker 1: asked whether to take it or not. And even then

608
00:34:49,236 --> 00:34:52,076
Speaker 1: that would leave the question of what to do afterwards.

609
00:34:52,596 --> 00:34:55,916
Speaker 1: It emerges that the British have simply gone down a

610
00:34:56,076 --> 00:35:00,876
Speaker 1: rabbit hole of contradiction between these two modes of democracy,

611
00:35:01,156 --> 00:35:05,396
Speaker 1: direct democracy by referendum and representative democracy by parliament, and

612
00:35:05,476 --> 00:35:07,436
Speaker 1: the only way they're going to get out of it

613
00:35:07,476 --> 00:35:09,836
Speaker 1: is if they abandon one of the these two modes

614
00:35:09,836 --> 00:35:13,356
Speaker 1: of action. They're not going to abandon in Parliament, which is,

615
00:35:13,396 --> 00:35:18,556
Speaker 1: by most accounts the oldest continuously running political body in

616
00:35:18,636 --> 00:35:22,076
Speaker 1: any democratic country. At least I hope they won't. What

617
00:35:22,156 --> 00:35:24,476
Speaker 1: they might learn is that if you've got something as

618
00:35:24,476 --> 00:35:27,676
Speaker 1: good as Parliament is, maybe you should stay away from

619
00:35:27,716 --> 00:35:31,356
Speaker 1: their referendums. And if that happens, then over time the

620
00:35:31,476 --> 00:35:34,516
Speaker 1: British will be able to re establish the norm of

621
00:35:34,596 --> 00:35:39,916
Speaker 1: parliamentary supremacy and parliamentary sovereignty. It ain't perfect, but it's

622
00:35:39,916 --> 00:35:42,956
Speaker 1: worked for a long long time. And if there's one

623
00:35:43,036 --> 00:35:45,476
Speaker 1: takeaway from the briggs At fiasco is that when the

624
00:35:45,516 --> 00:35:48,276
Speaker 1: British try to deviate from it, they do not know

625
00:35:48,356 --> 00:35:54,836
Speaker 1: what they're doing. Deep Background is brought to you by

626
00:35:54,876 --> 00:35:58,556
Speaker 1: Pushkin Industries. Our producer is Lydia Genecott, with engineering by

627
00:35:58,596 --> 00:36:02,676
Speaker 1: Jason Gambrell and Jason Roskowski. Our showrunner is Sophie mckibbon.

628
00:36:02,996 --> 00:36:06,076
Speaker 1: Our theme music is composed by Luis GERA special thanks

629
00:36:06,076 --> 00:36:09,716
Speaker 1: to the Pushkin Brass, Malcolm Gladwell, Jacob Weisberg, and Miah Lobel.

630
00:36:10,156 --> 00:36:12,436
Speaker 1: I'm Noah Feldman. You can follow me on Twitter at

631
00:36:12,436 --> 00:36:15,396
Speaker 1: Noah R. Feldman. This is deep background