1
00:00:01,560 --> 00:00:04,320
Speaker 1: So why is it fairness part of our process here?

2
00:00:04,880 --> 00:00:08,760
Speaker 1: It's because, well, as data scientists and statisticians and researchers,

3
00:00:08,840 --> 00:00:12,560
Speaker 1: we had good intentions. We lack those mechanisms for action.

4
00:00:13,080 --> 00:00:15,520
Speaker 1: We lack things in our process that force us to

5
00:00:15,560 --> 00:00:18,880
Speaker 1: consider hard questions. We need to use our brains a

6
00:00:18,920 --> 00:00:21,440
Speaker 1: little bit more than we need to for other problems

7
00:00:21,440 --> 00:00:24,400
Speaker 1: that we solve every day. And so why don't we

8
00:00:24,440 --> 00:00:27,920
Speaker 1: solve these hard problems? It's because we lack incentives as

9
00:00:27,960 --> 00:00:32,360
Speaker 1: a community data scientist to do something. Um, it's a

10
00:00:32,400 --> 00:00:36,000
Speaker 1: hard problem, and we have no transparency and no accountability

11
00:00:36,320 --> 00:00:38,920
Speaker 1: for the models that we produce. Right, So that means

12
00:00:38,960 --> 00:00:42,800
Speaker 1: that we have little hard business reason to prioritize fairness

13
00:00:42,800 --> 00:00:46,320
Speaker 1: and to spend time working on addressing this hard problem. Well,

14
00:00:46,320 --> 00:00:49,160
Speaker 1: you see a black tech green money. Let's talk about

15
00:00:49,400 --> 00:00:53,320
Speaker 1: algorithmic bias. You probably like, yo will. What in the

16
00:00:53,360 --> 00:00:58,400
Speaker 1: world is algorithmic bias? The wikipedias is it describes systematic

17
00:00:58,720 --> 00:01:03,240
Speaker 1: and repeatable error in the computer system that create unfair outcomes,

18
00:01:03,280 --> 00:01:07,480
Speaker 1: such as privileging one category over another in ways different

19
00:01:07,720 --> 00:01:11,440
Speaker 1: from the intended function of the algorithm. Now we can

20
00:01:11,680 --> 00:01:14,959
Speaker 1: debate whether these things are intended or not intended. But

21
00:01:14,959 --> 00:01:17,679
Speaker 1: that's a different conversation for another day. But these canna

22
00:01:17,680 --> 00:01:20,800
Speaker 1: have a direct impact on you when it determines which

23
00:01:21,040 --> 00:01:24,720
Speaker 1: political ads you see, or how many cops are deployed

24
00:01:24,720 --> 00:01:28,480
Speaker 1: in your neighborhood, or even your insurance premiums, how much

25
00:01:28,520 --> 00:01:32,319
Speaker 1: you pay for insurance. It was a study that show

26
00:01:32,840 --> 00:01:36,200
Speaker 1: even though black Americans are four times more likely to

27
00:01:36,240 --> 00:01:40,399
Speaker 1: have kidney failure, an algorithm to determine the priority of

28
00:01:40,480 --> 00:01:44,360
Speaker 1: patients on a kidney transplant list put black patients lower

29
00:01:44,400 --> 00:01:46,800
Speaker 1: on the list than white patients, even when all other

30
00:01:46,840 --> 00:01:50,640
Speaker 1: factors remain identical. So today on Black Tech, Green Money,

31
00:01:50,680 --> 00:01:53,640
Speaker 1: we're hearing from Matthew Finney, who's a data scientist is

32
00:01:53,680 --> 00:01:57,000
Speaker 1: strategy consultant at Harvard. He was a speaking from Afro

33
00:01:57,040 --> 00:02:00,760
Speaker 1: Tech World and in his day job, he phillips AI

34
00:02:01,000 --> 00:02:04,760
Speaker 1: decision systems to help large organizations and make an impact

35
00:02:04,760 --> 00:02:09,040
Speaker 1: on their most challenging business emission problems. I can sometimes

36
00:02:09,040 --> 00:02:12,560
Speaker 1: be a reluctant technologist, don't get me wrong. In the

37
00:02:12,639 --> 00:02:16,799
Speaker 1: last decade we have made some amazing feats with artificial intelligence.

38
00:02:17,160 --> 00:02:19,400
Speaker 1: We've been able to figure out what you want to

39
00:02:19,520 --> 00:02:22,760
Speaker 1: buy before you knew you wanted it we can have

40
00:02:23,040 --> 00:02:27,080
Speaker 1: a self driving, artificially intelligent electric car, and if that

41
00:02:27,160 --> 00:02:30,440
Speaker 1: was enough, we put it in space. We've trained AI

42
00:02:30,639 --> 00:02:34,760
Speaker 1: to read mammograms with particular skill at diagnosing a set

43
00:02:34,800 --> 00:02:38,840
Speaker 1: of highly invasive cancers that radiologists had missed, but we

44
00:02:38,919 --> 00:02:42,359
Speaker 1: still hadn't figured out how to make our technology treat

45
00:02:42,400 --> 00:02:44,440
Speaker 1: others the way that we would want to be treated.

46
00:02:44,760 --> 00:02:47,000
Speaker 1: So I promise I'm not just gonna stick to that

47
00:02:47,080 --> 00:02:51,000
Speaker 1: gloom and doom topic today. So what are we gonna do. First,

48
00:02:51,000 --> 00:02:54,880
Speaker 1: we're gonna define and measure algorithm bias. Then we're gonna

49
00:02:54,960 --> 00:02:57,520
Speaker 1: figure out how we can isolate the root causes of

50
00:02:57,560 --> 00:03:00,520
Speaker 1: poor algorithm behavior, and finally, we're going to learn how

51
00:03:00,560 --> 00:03:03,160
Speaker 1: we can all take action to make algorithms more fair.

52
00:03:03,639 --> 00:03:07,280
Speaker 1: So let's get started. I want to evaluate algorithmic bias

53
00:03:07,360 --> 00:03:10,280
Speaker 1: here through the lens of a case study, and we'll

54
00:03:10,360 --> 00:03:13,760
Speaker 1: learn how to, through this case study, apply the tools

55
00:03:13,800 --> 00:03:18,320
Speaker 1: more generally. Kidneys are really important. Obviously, their main function

56
00:03:18,400 --> 00:03:21,440
Speaker 1: in our body is to help us filter out waste,

57
00:03:21,840 --> 00:03:24,280
Speaker 1: and so there's a metric of kidney function called the

58
00:03:24,280 --> 00:03:29,880
Speaker 1: glomerular filtration rate that's very important for diagnosed and kidney

59
00:03:29,919 --> 00:03:33,680
Speaker 1: disease However, this metric is really hard to measure directly.

60
00:03:34,320 --> 00:03:36,320
Speaker 1: If you were going to measure directly, you need to

61
00:03:36,360 --> 00:03:38,720
Speaker 1: collect the waste from the kidney over the period of

62
00:03:38,760 --> 00:03:41,839
Speaker 1: twenty four hours. So it's not practical, it's not fun

63
00:03:41,920 --> 00:03:44,840
Speaker 1: for anyone. That's why in the seventies they developed an

64
00:03:44,880 --> 00:03:50,520
Speaker 1: algorithmic way to estimate this metric. UH Doctors can take

65
00:03:50,560 --> 00:03:54,120
Speaker 1: a sample of your blood and measure the level of

66
00:03:54,320 --> 00:03:59,680
Speaker 1: asset called creatomy that's in your blood sample, and there's

67
00:03:59,680 --> 00:04:03,480
Speaker 1: a Russian equation that takes that crowdning metric and turns

68
00:04:03,480 --> 00:04:07,120
Speaker 1: it into a kidney function index, this creating any metric

69
00:04:07,280 --> 00:04:10,960
Speaker 1: that they use. When researchers were developing the model, they

70
00:04:11,000 --> 00:04:15,600
Speaker 1: realized that creating is highly sensitive to someone's muscle mass,

71
00:04:15,920 --> 00:04:18,800
Speaker 1: you know, given that it's actually a byproduct of muscle activity.

72
00:04:19,000 --> 00:04:21,480
Speaker 1: And so when they were trying to make the algorithm

73
00:04:21,640 --> 00:04:25,719
Speaker 1: as accurate as they could, researchers determined that because African

74
00:04:25,760 --> 00:04:30,720
Speaker 1: Americans have higher muscle mass, they have higher baseline crawdning levels,

75
00:04:30,800 --> 00:04:33,200
Speaker 1: and so they decided that they were going to adjust

76
00:04:33,279 --> 00:04:36,720
Speaker 1: the c k D EPI algorithm, this kidney function algorithm,

77
00:04:36,760 --> 00:04:41,080
Speaker 1: to increase kidney function index scorers for African Americans to

78
00:04:41,400 --> 00:04:45,360
Speaker 1: control for this muscle difference. Here, a higher kidney function

79
00:04:45,440 --> 00:04:49,440
Speaker 1: score indicates that your kidney is healthier, so African Americans

80
00:04:49,440 --> 00:04:52,520
Speaker 1: were being given kidney index scores that were showing their

81
00:04:52,560 --> 00:04:55,400
Speaker 1: kidneys were healthier than a white person with the same

82
00:04:55,600 --> 00:04:59,760
Speaker 1: observable metrics. Interestingly, the United States is the only place

83
00:04:59,800 --> 00:05:03,000
Speaker 1: in the world that we do this race correction for

84
00:05:03,160 --> 00:05:05,960
Speaker 1: kidney functions, and there are many other places in the

85
00:05:06,000 --> 00:05:08,279
Speaker 1: world where we have a large population of people with

86
00:05:08,360 --> 00:05:13,159
Speaker 1: African heritage. This is because people see that this correction

87
00:05:13,360 --> 00:05:17,320
Speaker 1: is unfair. There are two specific definitions of fairness that

88
00:05:17,360 --> 00:05:21,400
Speaker 1: we use in the algorithm community. The first is group fairness,

89
00:05:22,200 --> 00:05:24,920
Speaker 1: and the idea behind group fairness is that in your

90
00:05:25,000 --> 00:05:28,400
Speaker 1: data set, you have groups that are identifiable and they

91
00:05:28,440 --> 00:05:32,240
Speaker 1: should be treated similarly to the population as a whole. Right,

92
00:05:32,320 --> 00:05:35,080
Speaker 1: So a group could be all people with blue eyes,

93
00:05:35,680 --> 00:05:40,200
Speaker 1: people with red hair, everyone who lives in Minnesota, all men,

94
00:05:40,920 --> 00:05:45,520
Speaker 1: people of Latin heritage. All those are examples of groups.

95
00:05:45,560 --> 00:05:47,920
Speaker 1: And if you have an algorithm that is grouped fair

96
00:05:48,279 --> 00:05:51,800
Speaker 1: that means that the algorithm treats all of these groups

97
00:05:51,839 --> 00:05:55,200
Speaker 1: similarly to the rest of the population. Regardless of whether

98
00:05:55,279 --> 00:05:58,880
Speaker 1: or not the algorithm has that information about the sensitive attribute.

99
00:05:58,880 --> 00:06:01,080
Speaker 1: That means someone's in a group or not. So let's

100
00:06:01,080 --> 00:06:05,760
Speaker 1: look at the second definition, individual fairness. Individual fairness means

101
00:06:05,800 --> 00:06:09,799
Speaker 1: that similar individuals should be treated similarly. In An example

102
00:06:09,839 --> 00:06:13,080
Speaker 1: of that is, let's say you have two people who

103
00:06:13,120 --> 00:06:17,279
Speaker 1: have equal incomes and equal credit history, and they're applying

104
00:06:17,320 --> 00:06:19,480
Speaker 1: for credit at a bank, and the bank uses an

105
00:06:19,520 --> 00:06:23,760
Speaker 1: algorithmic decision system to determine whether or not to extend

106
00:06:23,800 --> 00:06:27,039
Speaker 1: credit and a certain credit limit to the customers. So,

107
00:06:27,160 --> 00:06:29,880
Speaker 1: given that they had the same income and the same

108
00:06:29,920 --> 00:06:33,520
Speaker 1: credit history, even though one is male and the other's female,

109
00:06:33,800 --> 00:06:36,760
Speaker 1: both individuals should get the same credit limit if the

110
00:06:36,760 --> 00:06:39,919
Speaker 1: algorithm is individually fair. So now let's dive into this

111
00:06:40,000 --> 00:06:44,720
Speaker 1: kidney function algorithm again and let's think is this algorithm fair.

112
00:06:45,080 --> 00:06:47,279
Speaker 1: So first we'll look at the group fairness of the

113
00:06:47,360 --> 00:06:51,120
Speaker 1: c K D E P I algorithm. UM. The chart

114
00:06:51,160 --> 00:06:53,599
Speaker 1: here on the rank is taking a look at the

115
00:06:53,640 --> 00:06:57,640
Speaker 1: media number of days that adults in the United States

116
00:06:57,880 --> 00:07:01,159
Speaker 1: who received kidney transplants spent on the waiting list for

117
00:07:01,160 --> 00:07:05,960
Speaker 1: a kidney before they receive the transplant. UM something stands

118
00:07:05,960 --> 00:07:10,040
Speaker 1: out almost immediately here, and it's that African Americans can

119
00:07:10,120 --> 00:07:15,760
Speaker 1: spend over twice as long as Caucasians on the waiting

120
00:07:15,800 --> 00:07:18,920
Speaker 1: list for a kidney in the United States. Right, So,

121
00:07:19,120 --> 00:07:22,440
Speaker 1: African Americans are spending years on the waiting list, and

122
00:07:22,520 --> 00:07:25,080
Speaker 1: part of this is because of the c K D

123
00:07:25,280 --> 00:07:29,920
Speaker 1: e PI algorithm that's giving them higher kidney functions scores

124
00:07:30,320 --> 00:07:33,000
Speaker 1: even though their kidney might not be functioning well, and

125
00:07:33,040 --> 00:07:35,280
Speaker 1: that puts them at a lower priority on the waiting

126
00:07:35,320 --> 00:07:38,960
Speaker 1: list for a kidney. So this is treating African Americans

127
00:07:39,040 --> 00:07:42,640
Speaker 1: as a group different from groups of other Americans, and

128
00:07:42,680 --> 00:07:45,720
Speaker 1: that's something we should be concerned about. This algorithm is

129
00:07:45,720 --> 00:07:49,320
Speaker 1: not group fair. So now let's consider is this algorithm

130
00:07:49,360 --> 00:07:54,200
Speaker 1: individually fair. Individual fairness means that we treat similar individuals similarly.

131
00:07:54,440 --> 00:07:57,800
Speaker 1: And in this algorithm, we can have two individuals who

132
00:07:57,840 --> 00:08:00,560
Speaker 1: have the same muscle mass and the a level of

133
00:08:00,560 --> 00:08:03,360
Speaker 1: creating me measured in their blood. But if one of

134
00:08:03,400 --> 00:08:05,480
Speaker 1: them is white and one of them is black, they're

135
00:08:05,480 --> 00:08:09,280
Speaker 1: going to get different scores for their kidney function, such

136
00:08:09,360 --> 00:08:12,080
Speaker 1: that the black person will get a score indicating a

137
00:08:12,120 --> 00:08:17,120
Speaker 1: healthier kidney than the white person. Um this is concerning, right,

138
00:08:17,240 --> 00:08:21,280
Speaker 1: This is not individually fair and the medical community starting

139
00:08:21,320 --> 00:08:24,200
Speaker 1: to come around to this. So last year in the

140
00:08:24,280 --> 00:08:27,880
Speaker 1: Journal of the American Medical Association, they published an article

141
00:08:28,320 --> 00:08:31,760
Speaker 1: asking to reconsider the use of race and the kidney

142
00:08:31,760 --> 00:08:34,600
Speaker 1: function algorithm. And there was a sentence here that I

143
00:08:34,640 --> 00:08:37,240
Speaker 1: thought was really important. With the e G. F Our

144
00:08:37,280 --> 00:08:41,640
Speaker 1: equation that's being used, it asserts that existing organ function

145
00:08:42,240 --> 00:08:46,120
Speaker 1: is different between individuals who are identical except for race.

146
00:08:46,960 --> 00:08:51,880
Speaker 1: Race is causing African Americans to get unfavorable scores of

147
00:08:51,920 --> 00:08:55,200
Speaker 1: their kidney measurement function that might lead them to get

148
00:08:55,240 --> 00:08:57,719
Speaker 1: a lower priority on the waiting list to receive an

149
00:08:57,800 --> 00:09:02,240
Speaker 1: organ that's desperately needed. This might seem obvious that these

150
00:09:02,320 --> 00:09:07,120
Speaker 1: types of scenarios are bad, right, and we shouldn't be

151
00:09:07,240 --> 00:09:10,440
Speaker 1: using race for something that could have unfair outcomes that

152
00:09:10,520 --> 00:09:15,319
Speaker 1: cause life or death situations for people. But this keeps

153
00:09:15,360 --> 00:09:18,600
Speaker 1: happening over and over again. Any week you can open

154
00:09:18,679 --> 00:09:22,080
Speaker 1: up the newspaper and see a new algorithm that was

155
00:09:22,200 --> 00:09:25,640
Speaker 1: racist or sexist. You know, name YOURYSM. There's an algorithm

156
00:09:25,720 --> 00:09:29,480
Speaker 1: that is suffering from it. So let's talk about how

157
00:09:29,520 --> 00:09:32,000
Speaker 1: and why this happens. First, I want to just talk

158
00:09:32,040 --> 00:09:36,040
Speaker 1: about how we make models. Algorithmic models are function of

159
00:09:36,160 --> 00:09:41,439
Speaker 1: three things, technology, people, and process. On the technical front,

160
00:09:41,679 --> 00:09:44,320
Speaker 1: you know, that's where we consider the data that you're

161
00:09:44,400 --> 00:09:47,840
Speaker 1: using to train your model and the specific algorithm for example,

162
00:09:48,080 --> 00:09:50,160
Speaker 1: so that could be a neural network, that could be

163
00:09:50,200 --> 00:09:53,480
Speaker 1: a linear progression, that could be anything in between. On

164
00:09:53,559 --> 00:09:56,480
Speaker 1: the people front, you know, that's where we consider the

165
00:09:56,640 --> 00:10:00,480
Speaker 1: role of people like myself, data scientists, business owners who

166
00:10:00,840 --> 00:10:03,640
Speaker 1: come up with the business requirements for these algorithms, and

167
00:10:03,679 --> 00:10:06,880
Speaker 1: the end users who actually take the algorithms and put

168
00:10:06,920 --> 00:10:10,000
Speaker 1: them into practice to make decisions. And the last component

169
00:10:10,080 --> 00:10:13,720
Speaker 1: here are the processes, the processes that we use to

170
00:10:13,800 --> 00:10:17,199
Speaker 1: tread our models, to evaluate our models, and apply them

171
00:10:17,200 --> 00:10:21,840
Speaker 1: in practice. And by breaking down the process of building

172
00:10:21,840 --> 00:10:25,319
Speaker 1: a model into these three components, we can evaluate them

173
00:10:25,360 --> 00:10:28,400
Speaker 1: individually when we want to determine the root cause of

174
00:10:28,440 --> 00:10:32,480
Speaker 1: algorithmic fairness or algorithmic bias. So how did we make

175
00:10:32,600 --> 00:10:35,720
Speaker 1: a biased kidney function model in the context of these

176
00:10:35,760 --> 00:10:40,400
Speaker 1: three components. First, let's look at technology. So when researchers

177
00:10:40,440 --> 00:10:43,479
Speaker 1: were developing the c K D E p I algorithm,

178
00:10:43,520 --> 00:10:46,840
Speaker 1: they had many different ways that they could consider that

179
00:10:46,920 --> 00:10:50,920
Speaker 1: we're technologically feasible to measure and estimate e g. F R.

180
00:10:51,080 --> 00:10:54,280
Speaker 1: There was a direct way of measuring at gloom earlier

181
00:10:54,440 --> 00:10:59,400
Speaker 1: filtration rate, which was very difficult but not impossible, and

182
00:10:59,440 --> 00:11:02,000
Speaker 1: we could have on with that as medical community. There

183
00:11:02,000 --> 00:11:05,120
Speaker 1: were other alternatives to things that we can measure in

184
00:11:05,160 --> 00:11:08,800
Speaker 1: the blood Beyond looking at the creatomy, which is sensitive

185
00:11:08,840 --> 00:11:11,920
Speaker 1: to muscle mass. We could have instead decided to look

186
00:11:11,960 --> 00:11:15,440
Speaker 1: at sistat and see, which is another indicator of kidney

187
00:11:15,440 --> 00:11:18,880
Speaker 1: function that has no sensitivity to muscle muscle mass. And

188
00:11:19,040 --> 00:11:23,000
Speaker 1: there were also better ways of measuring muscle mass that

189
00:11:23,040 --> 00:11:27,080
Speaker 1: were technologically possible beyond just looking at someone's race to

190
00:11:27,240 --> 00:11:31,080
Speaker 1: estimate muscle mass. Right, So technology wasn't the constraint here

191
00:11:31,080 --> 00:11:35,400
Speaker 1: that let us to have a unfair algorithm for measuring

192
00:11:35,480 --> 00:11:39,160
Speaker 1: kidney function. Let's evaluate the people. Now it's gonna sound

193
00:11:39,160 --> 00:11:41,320
Speaker 1: like I'm glossing over this one, but I really do

194
00:11:41,480 --> 00:11:45,240
Speaker 1: want to assume the researcher's best intentions here when they

195
00:11:45,280 --> 00:11:48,880
Speaker 1: decided to build this regression model for measuring kidney function.

196
00:11:49,360 --> 00:11:52,080
Speaker 1: And I also want to assume that the doctors have

197
00:11:52,640 --> 00:11:55,520
Speaker 1: only the best intentions and the best interests of their

198
00:11:55,559 --> 00:11:58,440
Speaker 1: patients and mind when they make decisions on ordering this

199
00:11:58,559 --> 00:12:02,360
Speaker 1: test and recommen patients for kidney transplants, So I don't

200
00:12:02,400 --> 00:12:05,080
Speaker 1: think that people are the constraint here either. That led

201
00:12:05,160 --> 00:12:07,480
Speaker 1: us to have a biased model. So now let's look

202
00:12:07,520 --> 00:12:11,280
Speaker 1: at the process. The process here for building this model

203
00:12:11,600 --> 00:12:15,160
Speaker 1: was optimized for overall accuracy of the model. So we

204
00:12:15,240 --> 00:12:19,120
Speaker 1: mentioned how when researchers decided to include race in the

205
00:12:19,200 --> 00:12:22,840
Speaker 1: model that they were training, they got a slight overall

206
00:12:22,880 --> 00:12:25,760
Speaker 1: accuracy boost in the model, and that was the driving

207
00:12:25,760 --> 00:12:28,360
Speaker 1: factor in the decision to include race as a predictor

208
00:12:28,400 --> 00:12:31,280
Speaker 1: of kidney function. That process, that's where I want to

209
00:12:31,280 --> 00:12:35,840
Speaker 1: dive deeper. That's where our failure was. We had a

210
00:12:35,920 --> 00:12:41,720
Speaker 1: process that was optimized for accuracy and not for fairness objectives,

211
00:12:42,080 --> 00:12:46,720
Speaker 1: and because of that, that's how researchers developed a kidney

212
00:12:46,720 --> 00:12:49,880
Speaker 1: function model that was biased racially and had led to

213
00:12:50,000 --> 00:13:05,040
Speaker 1: unfair outcomes. A couple of years ago, the US Department

214
00:13:05,040 --> 00:13:09,560
Speaker 1: of Education Civil Rights Data Collection released information showing that

215
00:13:09,720 --> 00:13:13,240
Speaker 1: black and Latino students lack access at the high school

216
00:13:13,320 --> 00:13:18,360
Speaker 1: level to high level science and math classes and predominantly

217
00:13:18,360 --> 00:13:22,600
Speaker 1: white schools, calculus was offered across fifty percent of them.

218
00:13:23,360 --> 00:13:29,080
Speaker 1: In predominantly minority schools, just thirty three physics sixty seven

219
00:13:29,120 --> 00:13:34,160
Speaker 1: percent for white, forty percent for minority, algebra eight fo

220
00:13:34,280 --> 00:13:38,000
Speaker 1: percent for white, seventy one percent for minority. Now this

221
00:13:38,120 --> 00:13:42,080
Speaker 1: matters because these have downstream effects. High aptitude in these

222
00:13:42,160 --> 00:13:46,520
Speaker 1: STEM fields us higher representation in STEM careers. So when

223
00:13:46,559 --> 00:13:50,040
Speaker 1: we're not represented well, the systems don't get built for

224
00:13:50,160 --> 00:13:54,520
Speaker 1: us or even with our input appropriately considered. So how

225
00:13:54,520 --> 00:13:57,640
Speaker 1: can these systems that weren't built with our input play

226
00:13:57,679 --> 00:14:01,840
Speaker 1: out negatively in our communities? As a data scientist, you know,

227
00:14:02,360 --> 00:14:06,080
Speaker 1: we are in a profession where there's a high emphasis

228
00:14:06,120 --> 00:14:10,920
Speaker 1: on overall accuracy and a number of procedural technical controls

229
00:14:10,960 --> 00:14:14,560
Speaker 1: that promote that. On the technical side, we have many

230
00:14:14,679 --> 00:14:20,920
Speaker 1: metrics like just overall vanilla accuracy, MSc, precision recall, you

231
00:14:21,040 --> 00:14:25,120
Speaker 1: name it, specialized metrics to measure the accuracy of our models.

232
00:14:25,400 --> 00:14:28,800
Speaker 1: And then we have procedures like p testing that help

233
00:14:28,880 --> 00:14:32,000
Speaker 1: us make determinations about whether or not we should deploy

234
00:14:32,040 --> 00:14:35,160
Speaker 1: a certain model into practice. But we don't have that

235
00:14:35,280 --> 00:14:39,680
Speaker 1: same infrastructure for fairness. Um. As someone who's been in

236
00:14:39,680 --> 00:14:42,360
Speaker 1: the room where it happens, you know, I can tell

237
00:14:42,400 --> 00:14:45,960
Speaker 1: you where I think specifically, this type of process breakdown

238
00:14:46,280 --> 00:14:50,040
Speaker 1: affected our our kidney function model that we've been evaluating.

239
00:14:50,520 --> 00:14:54,400
Speaker 1: So let's look at specific things that they missed. UM. First,

240
00:14:54,520 --> 00:14:57,160
Speaker 1: let's address this chart here on the right. This is

241
00:14:57,200 --> 00:15:00,840
Speaker 1: a chart that shows muscle mass by ray among a

242
00:15:00,920 --> 00:15:04,640
Speaker 1: population of the US adults. The blue line represents white

243
00:15:04,680 --> 00:15:08,680
Speaker 1: Americans and the red line represents Black Americans. So we

244
00:15:08,720 --> 00:15:12,120
Speaker 1: can see that while on average, black Americans have a

245
00:15:12,160 --> 00:15:16,720
Speaker 1: slightly higher muscle mass and white Americans UM, this shift

246
00:15:16,880 --> 00:15:20,480
Speaker 1: is so slight that the distributions of muscle mass by

247
00:15:20,600 --> 00:15:24,280
Speaker 1: race overlap almost entirely. What this tells me as a

248
00:15:24,360 --> 00:15:27,840
Speaker 1: data scientist and a statistician is that an individual's race

249
00:15:27,920 --> 00:15:32,080
Speaker 1: tells me next to nothing about that person's muscle mass.

250
00:15:32,240 --> 00:15:36,600
Speaker 1: And so, as a researcher developing a kidney function algorithm,

251
00:15:36,720 --> 00:15:39,280
Speaker 1: if I was concerned about muscle mass, I would have

252
00:15:39,360 --> 00:15:42,000
Speaker 1: seen this chart and said, Wow, race is not a

253
00:15:42,040 --> 00:15:44,560
Speaker 1: predictor for muscle mass. That's going to help us, uh

254
00:15:44,720 --> 00:15:47,440
Speaker 1: improve the accuracy of our algorithm in a way. That's fair,

255
00:15:48,120 --> 00:15:51,880
Speaker 1: because you know, if we treat individuals as just members

256
00:15:51,960 --> 00:15:54,440
Speaker 1: of a race, we're actually not going to give that

257
00:15:54,520 --> 00:15:57,880
Speaker 1: person the best healthcare. So nothing in their process forced

258
00:15:57,880 --> 00:16:01,840
Speaker 1: them to look at whether or not race is predictive um,

259
00:16:02,360 --> 00:16:05,720
Speaker 1: in in in a broad sense for their objective, which

260
00:16:05,760 --> 00:16:09,280
Speaker 1: was to control for muscle mass. Nothing also forced them

261
00:16:09,320 --> 00:16:13,160
Speaker 1: to consider what the impact of using race would be

262
00:16:13,240 --> 00:16:16,920
Speaker 1: on the fairness of their model. So they didn't consider

263
00:16:17,400 --> 00:16:21,320
Speaker 1: the societal impacts of using race and healthcare. They also

264
00:16:21,360 --> 00:16:26,160
Speaker 1: didn't consider, um, how that would impact individuals you know,

265
00:16:26,200 --> 00:16:28,280
Speaker 1: who are on the waiting list for a kidney, and

266
00:16:28,320 --> 00:16:31,960
Speaker 1: how that might lead to individuals who are equally qualified

267
00:16:31,960 --> 00:16:37,240
Speaker 1: to receive a kidney uh be uh differentially prioritized on

268
00:16:37,320 --> 00:16:41,800
Speaker 1: the list to receive that kidney based on race. So

269
00:16:41,960 --> 00:16:46,120
Speaker 1: why isn't fairness part of our process here? Um? It's

270
00:16:46,160 --> 00:16:49,960
Speaker 1: because well, as data scientists and statisticians and researchers, we

271
00:16:50,040 --> 00:16:54,200
Speaker 1: had good intentions. We lack those mechanisms for action. We

272
00:16:54,360 --> 00:16:57,120
Speaker 1: lack things in our process that forced us to consider

273
00:16:57,480 --> 00:17:02,239
Speaker 1: hard questions. UM. It would be really easy to say

274
00:17:02,280 --> 00:17:05,640
Speaker 1: that we have biased algorithms because there are biased individuals

275
00:17:05,720 --> 00:17:09,080
Speaker 1: who want to encode their bias and the algorithms. UM.

276
00:17:09,080 --> 00:17:11,679
Speaker 1: And while I can't rule that out completely, let me

277
00:17:11,720 --> 00:17:15,520
Speaker 1: tell you that of the time that is not the case. Right.

278
00:17:15,840 --> 00:17:22,760
Speaker 1: Here's my hypothesis. Fairness is context specific um, meaning that

279
00:17:23,119 --> 00:17:27,120
Speaker 1: depending on what type of algorithm we're training, there might

280
00:17:27,119 --> 00:17:30,800
Speaker 1: be a different fairness subjective, and there might be different

281
00:17:30,880 --> 00:17:34,600
Speaker 1: rules for what's fair and what's unfair. So, for example,

282
00:17:34,840 --> 00:17:37,879
Speaker 1: there could be some healthcare scenarios where race is actually

283
00:17:37,920 --> 00:17:41,280
Speaker 1: an important predictor of a person to have overall health

284
00:17:41,359 --> 00:17:46,080
Speaker 1: or or risk for a disease, and those scenarivos might

285
00:17:46,080 --> 00:17:50,480
Speaker 1: be areas where it's fair to include race in an algorithm.

286
00:17:50,520 --> 00:17:53,439
Speaker 1: But it's something like this kidney function algorithm, we can

287
00:17:53,480 --> 00:17:56,840
Speaker 1: see that including race is clearly unfair. Um. And it's

288
00:17:56,880 --> 00:17:59,760
Speaker 1: because that there are these multiple notions of fairness with

289
00:18:00,040 --> 00:18:04,840
Speaker 1: different context dependencies that fairness is actually a hard problem

290
00:18:04,920 --> 00:18:08,399
Speaker 1: to solve. And for data scientists, you know, this is

291
00:18:08,400 --> 00:18:12,879
Speaker 1: a hard problem without a unique, closed form mathematical solutions,

292
00:18:13,480 --> 00:18:15,600
Speaker 1: meaning we need to use our brains a little bit

293
00:18:15,600 --> 00:18:17,879
Speaker 1: more than we need to for other problems that we

294
00:18:17,920 --> 00:18:21,160
Speaker 1: solve every day. And so why don't we solve these

295
00:18:21,200 --> 00:18:24,800
Speaker 1: hard problems. It's because we lack incentives as a community

296
00:18:24,840 --> 00:18:29,280
Speaker 1: data scientist to do something. Um, it's a hard problem,

297
00:18:29,359 --> 00:18:32,840
Speaker 1: and we have no transparency and no accountability for the

298
00:18:32,880 --> 00:18:35,480
Speaker 1: models that we produce. Right, So that means that we

299
00:18:35,520 --> 00:18:39,480
Speaker 1: have little hard business reason to prioritize fairness and to

300
00:18:39,480 --> 00:18:42,840
Speaker 1: spend time working on addressing this hard problem if no

301
00:18:42,840 --> 00:18:44,959
Speaker 1: one's ever going to be able to see, you know,

302
00:18:45,280 --> 00:18:47,520
Speaker 1: the steps that we took to address it and the

303
00:18:47,520 --> 00:18:53,119
Speaker 1: impact of our work. So, considering this process and mechanism

304
00:18:53,160 --> 00:18:57,520
Speaker 1: failure for fairness, how will we end algorithmic bias? So

305
00:18:57,600 --> 00:19:01,520
Speaker 1: I want to return to this idea, yeah, that algorithmic

306
00:19:01,560 --> 00:19:06,679
Speaker 1: models are a function of three major components technology, people,

307
00:19:07,000 --> 00:19:10,560
Speaker 1: and process. This is actually a question I asked often,

308
00:19:11,040 --> 00:19:14,720
Speaker 1: and I've asked in conversations about algorithm algorithmic fairness with

309
00:19:14,760 --> 00:19:20,760
Speaker 1: all kinds of people technologists, computer scientists, mathematicians, lawyers, ethicist, activists,

310
00:19:21,359 --> 00:19:25,280
Speaker 1: policy makers, and sociologists and many more. Right, And so

311
00:19:25,400 --> 00:19:27,800
Speaker 1: I found through these conversations and through some of my

312
00:19:27,880 --> 00:19:31,720
Speaker 1: own research that there are many existing approaches to addressing

313
00:19:31,720 --> 00:19:35,760
Speaker 1: algorithmic bias, and they generally fall in the technology and

314
00:19:35,840 --> 00:19:39,680
Speaker 1: people the veins. And so that's what we're looking at here,

315
00:19:40,400 --> 00:19:43,480
Speaker 1: just a couple of those different approaches that are already

316
00:19:43,480 --> 00:19:47,879
Speaker 1: out there that allows to address algorithmic fairness on the

317
00:19:47,880 --> 00:19:52,040
Speaker 1: technology front. I want to highlight that we already do

318
00:19:52,240 --> 00:19:56,040
Speaker 1: have class of algorithms that are always fair or fair

319
00:19:56,080 --> 00:20:00,480
Speaker 1: within certain constraints, and we're not always using them our work.

320
00:20:00,720 --> 00:20:04,600
Speaker 1: That's the problem. But there are tools out there that

321
00:20:04,680 --> 00:20:08,600
Speaker 1: allows to implement these very directly. So IBM, for example,

322
00:20:08,720 --> 00:20:12,920
Speaker 1: recently released a toolkit called AI Fairness three sixty UM

323
00:20:12,960 --> 00:20:17,080
Speaker 1: and it has fair machine learning algorithms and machine learning

324
00:20:17,280 --> 00:20:21,919
Speaker 1: diagnostics already implemented in Python that can be adapted to

325
00:20:22,320 --> 00:20:25,480
Speaker 1: any other type of prediction problem. Now, if you're a

326
00:20:25,480 --> 00:20:29,440
Speaker 1: little bit more adventurous, there's also a community of academics

327
00:20:29,760 --> 00:20:33,800
Speaker 1: who are on the cutting edge of research of algorithmic fairness.

328
00:20:33,880 --> 00:20:36,840
Speaker 1: And I'll point out the Symposium on the Foundations of

329
00:20:36,960 --> 00:20:40,000
Speaker 1: Responsible Computing as one place where you can go and

330
00:20:40,119 --> 00:20:43,160
Speaker 1: learn about a lot of those really cutting cutting edge

331
00:20:43,240 --> 00:20:46,840
Speaker 1: research topics. All these videos from the symposium are actually

332
00:20:46,920 --> 00:20:50,000
Speaker 1: publicly available on YouTube, so that you can add your

333
00:20:50,080 --> 00:20:53,720
Speaker 1: leisure learn about these topics from the academics who developed

334
00:20:53,760 --> 00:20:57,480
Speaker 1: them themselves. On the people front, right, we have a

335
00:20:57,560 --> 00:21:02,000
Speaker 1: lot of existing organizations that attack length education and tackling

336
00:21:02,040 --> 00:21:04,600
Speaker 1: the social movement component of this as well. Just to

337
00:21:04,680 --> 00:21:08,119
Speaker 1: name a few of organizations that are doing many great things.

338
00:21:08,560 --> 00:21:11,359
Speaker 1: Are we have data for black Lives and the Algorithmic

339
00:21:11,520 --> 00:21:15,199
Speaker 1: Justice League that are tackling that social movements and social

340
00:21:15,200 --> 00:21:21,680
Speaker 1: activism approach to encouraging algorithmic fairness. And then there's also

341
00:21:21,720 --> 00:21:25,360
Speaker 1: an organization called AI for All that is UH tackling

342
00:21:25,400 --> 00:21:28,359
Speaker 1: the education. So given that we see a lot of

343
00:21:28,440 --> 00:21:31,920
Speaker 1: existing work out there on the technology and people fronts,

344
00:21:32,200 --> 00:21:34,960
Speaker 1: I want to turn our attention to process where there's

345
00:21:35,040 --> 00:21:40,359
Speaker 1: relatively less existing work, and that's where the focus of

346
00:21:40,400 --> 00:21:43,800
Speaker 1: my research is what mechanisms can help us to build

347
00:21:43,840 --> 00:21:48,320
Speaker 1: fair algorithmic models. I'll return to those challenges that we

348
00:21:48,400 --> 00:21:52,240
Speaker 1: discussed before, the fact that algorithm fairness is hard to

349
00:21:52,240 --> 00:21:55,000
Speaker 1: define and hard to measure, and because of a lack

350
00:21:55,000 --> 00:21:59,080
Speaker 1: of transparency and accountability, we have a few incentives to

351
00:21:59,119 --> 00:22:02,080
Speaker 1: actually go in an and tackle the heart problem. So

352
00:22:02,240 --> 00:22:04,840
Speaker 1: first I want to propose an approach that will allow

353
00:22:04,960 --> 00:22:07,880
Speaker 1: us to make this hard problem a little bit easier

354
00:22:07,920 --> 00:22:11,360
Speaker 1: for us to solve. And it's called a fairness statement.

355
00:22:11,720 --> 00:22:15,680
Speaker 1: So what is a fairness statement? That's an application specific

356
00:22:15,720 --> 00:22:20,159
Speaker 1: commitment to defined and measurable fairness goals. The scope of

357
00:22:20,160 --> 00:22:23,720
Speaker 1: this fairness's statement is going to include defining the relevant

358
00:22:23,760 --> 00:22:27,600
Speaker 1: fairness objective or constraint for the specific algorithm that we're

359
00:22:27,600 --> 00:22:31,840
Speaker 1: working on developing. So, for example, that could be we

360
00:22:31,880 --> 00:22:35,480
Speaker 1: want to make sure that African American people and white

361
00:22:35,480 --> 00:22:41,879
Speaker 1: people received similar kidney functions scores for similar actual kidney function. Now,

362
00:22:42,640 --> 00:22:46,040
Speaker 1: now that we've defined a fairness objective, we can document

363
00:22:46,080 --> 00:22:50,240
Speaker 1: potential sources of bias that might impact our fairness subjective

364
00:22:50,720 --> 00:22:55,240
Speaker 1: and also the downstream impact will see two individuals or groups, right,

365
00:22:55,359 --> 00:22:57,680
Speaker 1: So this might be the place where we raise well,

366
00:22:57,720 --> 00:23:01,800
Speaker 1: if our algorithms racial racially bias, we might see African

367
00:23:01,840 --> 00:23:05,719
Speaker 1: Americans play prioritize at a lower priority on the kidney

368
00:23:05,720 --> 00:23:09,040
Speaker 1: waiting list, and I might leave to adverse healthcare outcomes

369
00:23:09,080 --> 00:23:14,840
Speaker 1: for that population. Finally, once we've documented the source of biases,

370
00:23:15,080 --> 00:23:19,600
Speaker 1: we can identify appropriate procedural and technical controls that we

371
00:23:19,600 --> 00:23:23,520
Speaker 1: would would take to mitigate the unacceptable risks. Right. So

372
00:23:23,640 --> 00:23:26,640
Speaker 1: that could be, for example, implementing one of the classes

373
00:23:26,640 --> 00:23:30,240
Speaker 1: of fair algorithms that we discussed before. One of the

374
00:23:30,359 --> 00:23:33,160
Speaker 1: key benefits of the fairness statement is that it gives

375
00:23:33,280 --> 00:23:36,720
Speaker 1: data scientists a named goal they can work towards, and

376
00:23:36,840 --> 00:23:40,000
Speaker 1: that helps them informed choices and trade offs in the

377
00:23:40,080 --> 00:23:45,360
Speaker 1: development of algorithms and the deployment. So, for example, if

378
00:23:45,400 --> 00:23:48,720
Speaker 1: we had a fairness statement that was in place for

379
00:23:48,760 --> 00:23:51,880
Speaker 1: the researchers who developed the c k d EPI algorithm

380
00:23:51,960 --> 00:23:56,120
Speaker 1: for kidney function UH, that might have helped them say, hey,

381
00:23:56,240 --> 00:23:58,840
Speaker 1: we could include race and have a slight bump and

382
00:23:59,000 --> 00:24:04,960
Speaker 1: overall accuracy for our algorithm. But that presents a high

383
00:24:05,160 --> 00:24:09,480
Speaker 1: risk of unfair outcomes. Therefore, the cost of this solution

384
00:24:09,920 --> 00:24:15,440
Speaker 1: outweighs the small benefit of controlling for race and measuring

385
00:24:15,560 --> 00:24:20,080
Speaker 1: kidney function. Now, the other key thing fit here is

386
00:24:20,119 --> 00:24:26,680
Speaker 1: that this allows algorithmic developers to catch problems early, at

387
00:24:26,720 --> 00:24:29,240
Speaker 1: the stage when the algorithm is still in development and

388
00:24:29,280 --> 00:24:32,960
Speaker 1: before it's been deployed into the world. This might mean

389
00:24:33,000 --> 00:24:36,520
Speaker 1: that we catch an issue before it actually creates harm

390
00:24:36,720 --> 00:24:39,879
Speaker 1: for people in real life. So now that we've talked

391
00:24:39,880 --> 00:24:43,800
Speaker 1: about how we can make the UH fairness problem a

392
00:24:43,800 --> 00:24:46,480
Speaker 1: little bit less hard, now let's talk about how we

393
00:24:46,520 --> 00:24:50,560
Speaker 1: can incentivize people to actually tackle it. I want to

394
00:24:50,600 --> 00:24:55,159
Speaker 1: propose an approach called the algorithmic Practice audit. So what

395
00:24:55,359 --> 00:24:58,399
Speaker 1: is this? As an independent third party review of an

396
00:24:58,480 --> 00:25:03,320
Speaker 1: organization's algorithmic the season outcomes. On the process front, we

397
00:25:03,400 --> 00:25:07,359
Speaker 1: might evaluate questions like are we using a representative training

398
00:25:07,440 --> 00:25:10,679
Speaker 1: data set to trade our model. We might also question

399
00:25:10,680 --> 00:25:14,280
Speaker 1: whether or not the organization is using fair classes of

400
00:25:14,320 --> 00:25:19,920
Speaker 1: algorithms when they exist to train models. On the outcome front,

401
00:25:20,160 --> 00:25:24,080
Speaker 1: we might evaluate the actual fairness objective that was in

402
00:25:24,200 --> 00:25:28,280
Speaker 1: the fairness statement. Is the model meeting the stated fairness goals.

403
00:25:29,560 --> 00:25:31,960
Speaker 1: We might also be able to look at whether or

404
00:25:32,000 --> 00:25:35,840
Speaker 1: not biases introduced by humans in the last mile of

405
00:25:35,880 --> 00:25:39,399
Speaker 1: the algorithmic decision making process, right, so in that stage

406
00:25:39,440 --> 00:25:42,159
Speaker 1: where the algorithm has made a prediction and then it

407
00:25:42,200 --> 00:25:44,320
Speaker 1: takes a human to go and implement it and turn

408
00:25:44,359 --> 00:25:46,919
Speaker 1: it into a decision. So a key benefit of this

409
00:25:47,560 --> 00:25:50,600
Speaker 1: is that it's a forcing function that allows our data

410
00:25:50,680 --> 00:25:56,040
Speaker 1: scientists to actually invest time in algorithmic fairness because there

411
00:25:56,040 --> 00:26:00,000
Speaker 1: are penalties. There are real penalties um to not actually

412
00:26:00,320 --> 00:26:03,400
Speaker 1: having a fair algorithm. And another key benefit is that

413
00:26:03,440 --> 00:26:06,040
Speaker 1: this can be a signal for your organization to your

414
00:26:06,080 --> 00:26:10,280
Speaker 1: customers and shareholders that any algorithmic services you provide are

415
00:26:10,320 --> 00:26:14,240
Speaker 1: correct and fair. Right, So imagine that you're a customer

416
00:26:14,480 --> 00:26:16,879
Speaker 1: and you can transact with an organization that you know

417
00:26:16,960 --> 00:26:20,080
Speaker 1: has fair algorithms and that is certified as such, or

418
00:26:20,119 --> 00:26:23,360
Speaker 1: you can spend your money with another organization where it's

419
00:26:23,400 --> 00:26:26,280
Speaker 1: upanly or whether or not their algorithms are fair. You

420
00:26:26,359 --> 00:26:28,760
Speaker 1: might choose as a customer to spend your money with

421
00:26:28,800 --> 00:26:33,200
Speaker 1: an organization that has fair algorithms. Now, if you're a shareholder,

422
00:26:33,440 --> 00:26:36,960
Speaker 1: you might also be at more confident in an organization

423
00:26:37,359 --> 00:26:41,760
Speaker 1: that you know is UH spending time and energy on

424
00:26:41,840 --> 00:26:44,960
Speaker 1: algorithmic fairness, because that might be a signal to you

425
00:26:45,280 --> 00:26:47,840
Speaker 1: that the organization won't end up on the front page

426
00:26:47,880 --> 00:26:50,919
Speaker 1: of the New York Times for having an unfair racist

427
00:26:50,960 --> 00:26:56,640
Speaker 1: algorithm in the future. And I want to just highlight

428
00:26:56,840 --> 00:27:00,280
Speaker 1: that while this seems like a hard problem, these types

429
00:27:00,320 --> 00:27:03,720
Speaker 1: of mechanisms actually work and we can implement them to

430
00:27:03,800 --> 00:27:08,000
Speaker 1: make change in the way that algorithm predictions happen. So

431
00:27:08,320 --> 00:27:11,280
Speaker 1: let's look at the example of the system Risk indicator.

432
00:27:12,200 --> 00:27:16,119
Speaker 1: In the Dutch government developed the system Risk Indicator to

433
00:27:16,200 --> 00:27:20,320
Speaker 1: detect benefit fraud. Right, But while the government developed it,

434
00:27:20,320 --> 00:27:23,040
Speaker 1: it was only applied by a certain number of cities,

435
00:27:23,280 --> 00:27:27,040
Speaker 1: and the cities that applied this algorithm um only applied

436
00:27:27,080 --> 00:27:30,520
Speaker 1: it to some of the applications for benefits that they received,

437
00:27:30,680 --> 00:27:35,480
Speaker 1: and specifically it was applied in low income and immigrant neighborhoods,

438
00:27:35,560 --> 00:27:39,760
Speaker 1: So these populations of people were specifically targeted by the

439
00:27:39,800 --> 00:27:46,080
Speaker 1: algorithm to identify possible benefit risk. This is unfair and

440
00:27:46,080 --> 00:27:50,320
Speaker 1: and the Dutch court actually UH did an investigation and

441
00:27:50,440 --> 00:27:54,760
Speaker 1: found just as much. UM. They shut down this algorithmic

442
00:27:54,800 --> 00:27:59,760
Speaker 1: system because of the possibility of discrimination based on socio

443
00:27:59,840 --> 00:28:04,080
Speaker 1: economic status, ethnicity, and religion. Essentially, what they found was

444
00:28:04,160 --> 00:28:08,520
Speaker 1: that the algorithm did not meet the stated fairness objectives

445
00:28:08,560 --> 00:28:12,159
Speaker 1: of the Dutch government because it was discriminating against people

446
00:28:12,440 --> 00:28:16,920
Speaker 1: based on immutable characteristics. And because of that, they stopped

447
00:28:17,040 --> 00:28:22,360
Speaker 1: using this algorithm UH in benefit processing for Dutch citizens

448
00:28:22,359 --> 00:28:26,560
Speaker 1: and residents. So we know it works. What will you

449
00:28:26,640 --> 00:28:30,119
Speaker 1: do to create fair algorithms? I want to leave you

450
00:28:30,200 --> 00:28:33,720
Speaker 1: with a couple of my suggestions, UM, and this is

451
00:28:33,760 --> 00:28:38,120
Speaker 1: something that we can tackle as organizations and also as individuals.

452
00:28:39,080 --> 00:28:42,120
Speaker 1: In an organization, you might question whether or not you're

453
00:28:42,280 --> 00:28:47,280
Speaker 1: using existing classes of fair algorithms, such as those released

454
00:28:47,280 --> 00:28:51,480
Speaker 1: by IBM and the AI three sixty tool kit. You

455
00:28:51,560 --> 00:28:54,480
Speaker 1: might also consider whether or not you have mechanisms in

456
00:28:54,560 --> 00:28:58,240
Speaker 1: place to ensure algorithm fairness, such as the Fairness Statement

457
00:28:58,280 --> 00:29:02,080
Speaker 1: and the algorithmic Practice audit. As an individual, you might

458
00:29:02,400 --> 00:29:06,440
Speaker 1: do an inventory of all the algorithmic decisions that occur

459
00:29:06,520 --> 00:29:09,680
Speaker 1: in your life. You know, with customers that you work with,

460
00:29:09,680 --> 00:29:13,320
Speaker 1: with companies that you buy from, with your employer, with

461
00:29:13,480 --> 00:29:18,120
Speaker 1: your apartment building. These are everywhere. And then once you've

462
00:29:18,160 --> 00:29:22,160
Speaker 1: done that inventory, you might request and review algorithmic audits

463
00:29:22,200 --> 00:29:24,240
Speaker 1: from the organizations that you know are making some of

464
00:29:24,240 --> 00:29:40,480
Speaker 1: the most impactful decisions about you using algorithms. Black Tag

465
00:29:40,520 --> 00:29:42,520
Speaker 1: Green Money is a production of Black the af Road

466
00:29:42,560 --> 00:29:45,600
Speaker 1: Say from the Black Effect podcast Network and iHeart Media.

467
00:29:46,000 --> 00:29:49,239
Speaker 1: Is produced by Morgan Dabon and me Well Lucas, with

468
00:29:49,280 --> 00:29:53,800
Speaker 1: aditional productive support by Love Beach Merissa Lewis. Special thank

469
00:29:53,800 --> 00:29:56,360
Speaker 1: you to mikead Davis, your main Hall of It Necessarianto

470
00:29:57,120 --> 00:29:59,680
Speaker 1: learn by guests and other technistuff does the Innovatives an

471
00:29:59,680 --> 00:30:03,000
Speaker 1: Afro tech dot com and join your Black Tech Green Money.

472
00:30:03,520 --> 00:30:07,080
Speaker 1: Leave us a five star rady on iTunes. Go get

473
00:30:07,120 --> 00:30:09,560
Speaker 1: your money. Peace in Love,