Speaker 1: Pushkin from Pushkin Industries. This is Deep Background, the show
Speaker 1: where we explore the stories behind the stories in the news.
Speaker 1: I'm Noah Feldman. How did you find this podcast? Did
Speaker 1: you see an ad for it on your phone? If so,
Speaker 1: that ad might have shown up for you, because, based
Speaker 1: on information about you that's out there on the Internet,
Speaker 1: a computer algorithm decided that this show might be the
Speaker 1: kind of thing that you would like. Algorithms like that
Speaker 1: are all around us. Some are far more consequential than others.
Speaker 1: But I'm glad you're listening to the show. But hey,
Speaker 1: if you're applying for credit, an algorithm could actually evaluate
Speaker 1: your credit worthiness, and the stakes are a little higher
Speaker 1: there than whether you're listening to this podcast or to
Speaker 1: Trevor Noah's. If I'm looking for a job, an algorithm
Speaker 1: could go through all of the job applicants to try
Speaker 1: to do a first cut before the employer decides who
Speaker 1: they're going to interview. In some cities, algorithms are even
Speaker 1: being used by the police to try to predict the
Speaker 1: probability that there's going to be a crime in a
Speaker 1: particular place and to decide where they're going to focus
Speaker 1: the police efforts and actually send the cops. Nicole Ternally
Speaker 1: is a fellow at Brooking Center for Technology Innovation. She's
Speaker 1: been studying how algorithms like this work and how they fail.
Speaker 1: She recently co wrote a report for Brookings about algorithmic bias,
Speaker 1: or in other words, how computers can be racist. So
Speaker 1: if we look at an algorithm like a black box,
Speaker 1: it starts with an input and it ends with an output.
Speaker 1: When it comes to the input, you know, big data
Speaker 1: has made it very easy to actually harness volumes of
Speaker 1: data they're about us, these reference points, about individuals, and
Speaker 1: to create you know, I think some input or what
Speaker 1: we call training data that essentially trains the algorithm to
Speaker 1: adapt to what our behaviors are. In many cases, what
Speaker 1: goes into the algorithm can be accurate. There are certain
Speaker 1: things that your listeners do online that are discrete, our objective,
Speaker 1: are true in terms of your search queries, in terms
Speaker 1: of your online profile. But when you have developers that
Speaker 1: put in training data that in some respects may be
Speaker 1: biased or skewed, it creates challenges or what technologists have
Speaker 1: called garbage in So what I mean by that, if
Speaker 1: you're developing an algorithm and this is actually a case
Speaker 1: So this is not something that we're making up. Like
Speaker 1: the Compass algorithm, which was designed to help judges make
Speaker 1: better predictions on the amount of time that a defendant
Speaker 1: should be detained before sentencing. And let's say the training
Speaker 1: data used to train that algorithm is based upon criminal
Speaker 1: justice stats or criminal behavior stats. It's no secret that
Speaker 1: in this country, African American men in particular are more
Speaker 1: likely to experience arrest. And if they are more likely
Speaker 1: to experience arrest, which oftentimes leads to incarceration, they will
Speaker 1: overwhelmingly make up the majority of the training data. So
Speaker 1: that input, when it gets to the output, it then
Speaker 1: may disproportionately affect African American defendants by suggesting that they
Speaker 1: have a longer detainment before sentencing. So that's an example
Speaker 1: where we have some background bias in our society. Right,
Speaker 1: the system is already raged against African Americans. Arrests are disproportioned,
Speaker 1: African American jailing is disproportionate of African Americans. And then
Speaker 1: once the data is trained, the data that emerges will
Speaker 1: also reflect those pre existing biases. But how do you
Speaker 1: know the problem was the algorithm? How do you know
Speaker 1: the problem isn't Rather the underlying deep structures of racism
Speaker 1: in the United States that created the circumstances where arrests
Speaker 1: and imprisonment are disproportionately acts that happen to African Americans
Speaker 1: rather than to white people. In other words, that is
Speaker 1: the form of racial bias. How do we know that
Speaker 1: the algorithm is actually making it worse as opposed to
Speaker 1: just reflecting the existing realities of race, You know, I
Speaker 1: think it's both. I mean, on the one hand, I
Speaker 1: think that we do have the issue where it is
Speaker 1: representative of the existing societal concerns that we have. A
Speaker 1: mathematical model is not necessarily going to correct or remedy.
Speaker 1: I think the historical biases that many groups have suffered.
Speaker 1: This we're talking about structural and systemic racism and discrimination
Speaker 1: that just won't go away from a computer model. But
Speaker 1: I also think that part of what we're seeing, and
Speaker 1: this is in my attempt to not say that developers
Speaker 1: are racist, that it all depends on who's at the
Speaker 1: table when developing that algorithm. And so there's two things
Speaker 1: that are going on when you look at the tech space. One,
Speaker 1: you have a very limited pool of diversity that happens
Speaker 1: in these professions you know, the data science profession in
Speaker 1: and of itself is underrepresentative of historically disadvantaged groups, women,
Speaker 1: people of color, older Americans, etc. And that can be
Speaker 1: problematic as these algorithms become much more ubiquitous in such society.
Speaker 1: And then you have the other issue of implicit bias,
Speaker 1: which comes from this unconscious understanding of how the world works.
Speaker 1: Let me give a good example of that. Amazon just
Speaker 1: a few months ago released an employment algorithm that was
Speaker 1: trying to find candidates for their engineering department. The training
Speaker 1: data that was used by the developers went on the
Speaker 1: historical data of that department, which tended to be white men,
Speaker 1: and as a result, the algorithm kicked out any resume
Speaker 1: that had any hint of a person being from an
Speaker 1: all women's college or having a women's group represented. So
Speaker 1: that word, in and of itself, because it's not necessarily
Speaker 1: associated with engineering professions, struck down the opportunities of them
Speaker 1: to become a more diverse workforce in terms of that department.
Speaker 1: You know, Amazon later retracted that and took it off
Speaker 1: the market, But you see what I mean? So right,
Speaker 1: I mean, so you mentioned a group of fascinating things there.
Speaker 1: So one is the composition of the tech world, and
Speaker 1: there I think every reasonable person can agree that it
Speaker 1: can only be better on its own terms, totally independent
Speaker 1: of whether it affects the implicit bias phenomenon. But in general,
Speaker 1: we would love to see we need to see as
Speaker 1: a society much greater diverse representation of previously disadvantage groups
Speaker 1: in or currently disadvantage groups in the tech role. Then
Speaker 1: they have the implicit bias example where you're drawing on
Speaker 1: existing data. So you could sort of imagine why Amazon
Speaker 1: wants to figure out who to hire, and so they
Speaker 1: put into the data that people they have hired because
Speaker 1: they think, hey, we're pretty awesome and sure enough. Then
Speaker 1: that just suggests that they replicate the thing that they
Speaker 1: have already. Let me ask about the flip side of that,
Speaker 1: not nicle the potentially positive side. So take a couple
Speaker 1: of examples. You've mentioned determination of either bail or of
Speaker 1: criminal sentence on the one hand. Another example would be
Speaker 1: employment determinations. These are all cases where we know from
Speaker 1: years and years of collected data that human decision makers
Speaker 1: are systematically biased against people of and we try to
Speaker 1: debias people by different methods. We have an appeals process
Speaker 1: where you can appeal and say I've been discriminated against,
Speaker 1: very hard to win. We have rules that say don't
Speaker 1: be biased, and we even have lots of decision makers
Speaker 1: who in their hearts are not biased, and yet you
Speaker 1: show them statistically what they've done over the long run,
Speaker 1: and sure enough their behavior does reflect bias. Now, don't
Speaker 1: algorithms potentially offer a liberatory solution an equality solution here?
Speaker 1: Because the one thing that we can say about an
Speaker 1: algorithm is, unlike a human, if you give it a rule,
Speaker 1: it will follow the rule, and there's no O G.
Speaker 1: I thought I was following the rule, but I really wasn't.
Speaker 1: So if you, in principle, tell the algorithm not to
Speaker 1: consider race, or you tell the algorithm not to consider
Speaker 1: various factors that look like proxies for race, and you
Speaker 1: can even train the algorithms so that it is less
Speaker 1: inclined to rely on those proxies, then presumably you could
Speaker 1: have a decision maker making decisions about criminal justice, making
Speaker 1: decisions about employment that are less biased, less racist than
Speaker 1: the best intentioned human being. Because humans have an unconscious,
Speaker 1: and in our unconscious we might be biased, but algorithms
Speaker 1: don't have an unconscious mind. Yeah, you know, I have
Speaker 1: to say I got to push back on that, and
Speaker 1: I'll tell you why. I think because we see more
Speaker 1: of the digital economy rush to market, we're not dealing
Speaker 1: with an environment where we see this level of diligence
Speaker 1: when it comes to know what is the bias impact
Speaker 1: on certain groups? Have we been able to use race
Speaker 1: as a proxy to create an anti bias experimentation? Are
Speaker 1: we auditing our algorithm in ways that we can ensure
Speaker 1: from its development to its execution that we're identifying what
Speaker 1: that bias may look like? And I think we need
Speaker 1: to go forward and put together some framework not on
Speaker 1: all algorithms. I think Netflix does a pretty good job
Speaker 1: recommending the types of movies that I like to watch.
Speaker 1: I'm not I mean, that's true, but I'm not so
Speaker 1: sure that we should even assume that those aren't biased.
Speaker 1: They are. I think those are because they're also going
Speaker 1: to pick out features. They also know what your zip
Speaker 1: code is. If they take up zip code, then at
Speaker 1: least implicity, they're also recognizing race, you know, I mean,
Speaker 1: that's right. So I'll just take my case. I'm an
Speaker 1: African American woman who's middle age, who loves to watch
Speaker 1: you know, black romance Flix, and let me tell you,
Speaker 1: every time Netflix recommends one, I'm happy. I mean, the
Speaker 1: only problem I get, you know, challenged by it is
Speaker 1: when the content runs out and not investing in, you know,
Speaker 1: more more programmers or developers to develop more content for
Speaker 1: people like me. But maybe that's because they're not feeding
Speaker 1: those two people like me, right, I'm a middle aged
Speaker 1: white guy. They're not telling me to watch those, But
Speaker 1: maybe if they did, I'd watch them, I'd like them,
Speaker 1: and if that happened, there would be developers. So in
Speaker 1: that sense, you know, it may be that that there
Speaker 1: is an implicit bias there. They're just assuming that the
Speaker 1: reason that I don't watch them is that I haven't
Speaker 1: watched that many you know, African American romantic comedies in
Speaker 1: recent years and so. But you know, but itself is
Speaker 1: not a neutral fact, right, because no one's advertising them
Speaker 1: to me. Netflix isn't telling me to watch them. If
Speaker 1: they told me, that might have a different impact. That's right,
Speaker 1: And that's I mean, I always say to people. When
Speaker 1: the alright movement and the white conservative movement became a
Speaker 1: big thing and hate speech on Facebook, I was kind
Speaker 1: of surprised that I didn't know that this was happening
Speaker 1: in the same playground in which I also, you know, visit.
Speaker 1: And that's because my algorithm or is not made of
Speaker 1: white supremacists. It's you know, more liberals that sort of
Speaker 1: speak the same language and feel the same way about
Speaker 1: certain issues. What about the hardcase nicle, I mean, you know,
Speaker 1: let's say I'm applying for credit and they've got information,
Speaker 1: you know that Let's say I have allowed I didn't
Speaker 1: check the box to make it private. That says, hey,
Speaker 1: Feldman's been searching for payday loans, and the algorithm notices
Speaker 1: something that is intuitively very plausible, which is that if
Speaker 1: I'm so desperate that I'm looking into payday loans, probably
Speaker 1: that means I'm a slightly less good credit risk than
Speaker 1: someone who hasn't been yet searching for payday loans, because
Speaker 1: you know, eventually, I'm going to start looking for that
Speaker 1: the minute I really need it. So if I'm trying
Speaker 1: to lend money, and that's there's nothing inherently racially determinative
Speaker 1: about that. It may not even be about wealth in general.
Speaker 1: It's just about how much money I have right this minute,
Speaker 1: or how little I have right this minute. That might
Speaker 1: be great from the standpoint of the credit company, and
Speaker 1: they might actually be able to do a better job
Speaker 1: of setting the correct interest rate for me based on
Speaker 1: that information. Does that still disturbute? Does it still make
Speaker 1: you think that that's a problem or is that more
Speaker 1: like Netflix? It's not that big a deal they're telling
Speaker 1: you know, They're they're fitting the data to the objective.
Speaker 1: You know. I think it's interesting because I struggle with it.
Speaker 1: I think that there are cases where, you know, our
Speaker 1: online behavior will indicate certain characteristics about us. Though it
Speaker 1: is somewhat problematic because what if I was searching for
Speaker 1: the payday loan for my uncle? Right, not necessarily for myself,
Speaker 1: But in the end, I think we have to be
Speaker 1: very sensitive, or the algorithmic operator has to be sensitive
Speaker 1: to the extent to which they're denying credit to these
Speaker 1: groups versus out the groups. Right. One thing, for example,
Speaker 1: we say in the paper, which I think is just profound, is,
Speaker 1: as an African American who maybe serve more higher interest
Speaker 1: credit card rates, what if I see that ad come
Speaker 1: through and I click it just because I'm interested to
Speaker 1: see why I'm getting this ad. Automatically, I will be
Speaker 1: served similar ads, right, so it automatically places me in
Speaker 1: that high credit risk category. The challenge that we're having now, Noah,
Speaker 1: is that as an individual consumer, I have no way
Speaker 1: of recurating what my identity is. I want to ask
Speaker 1: you a kind of final big picture question, and it's
Speaker 1: when you survey this whole environment, the possibilities of regulation,
Speaker 1: you're testifying on the hill about it, you're doing reports
Speaker 1: on it. Are you in general optimistic about the future
Speaker 1: of the possibility to regulate algorithms and to also turn
Speaker 1: algorithms to good with respect to fairness and equality, or
Speaker 1: are you, on balance pessimistic and think that the terrible
Speaker 1: legacies of discrimination that we have in our country are
Speaker 1: like just to be either continued or even made worse
Speaker 1: by virtue of this technological development. You know, I'm a
Speaker 1: technologist who's optimistic about the use of technology, you know,
Speaker 1: I think of it this way. I think as technology evolves,
Speaker 1: we are faced with this challenge whether or not the
Speaker 1: technology coopts the user or the user has something to
Speaker 1: do with the technologies agency, right, And so I'm one
Speaker 1: of those people, particularly in this case of algorithms, which
Speaker 1: has just become so interesting to many of us because
Speaker 1: it's got this explainability portion, and then it has stuff
Speaker 1: that we don't even know how to dissect and unpack that.
Speaker 1: I think what we're trying to do in this particular case,
Speaker 1: Noah's just get ahead of it and to be much
Speaker 1: more proactive in talking about it. I mean, my goal
Speaker 1: is to bring to the forefront those algorithms that are
Speaker 1: allowing older Americans to age in place, those algorithms that
Speaker 1: are catching chronic disease and some of the worst abilitating diseases,
Speaker 1: and as ahead of time because of the precision of
Speaker 1: the technology, we're seeing, you know, better customization of educational
Speaker 1: curricula for students because algorithms are able to identify learning
Speaker 1: styles much faster than a teacher can. And so I
Speaker 1: don't want us to be a society which turns our
Speaker 1: back against the technology and the innovation, because that's part
Speaker 1: of this whole new revolution of our shift for manufacturing
Speaker 1: into I think this digital age where it does matter.
Speaker 1: I'm gonna tell you honestly, what really concerns me is
Speaker 1: the fact that the less information or the less diffused
Speaker 1: that these algorithms are, the more likely you'll be on
Speaker 1: the wrong side of digital opportunity, and the more likely
Speaker 1: that your community may not get some of the services
Speaker 1: that have come out of an algorithmic economy. I mean,
Speaker 1: imagine living in a community where they don't have your data.
Speaker 1: You're not your data is not being harnessed for any
Speaker 1: type of productive algorithm. You find yourself in a state
Speaker 1: where you have more chronic disease, more incarceration, less levels
Speaker 1: of educational achievement. Better to be then to be out. Yeah,
Speaker 1: and I'll just say this in final I mean, I
Speaker 1: think for those of us that are in this space,
Speaker 1: I think we take into consideration the fairness and accuracy
Speaker 1: conversations as well the ethical conversations, But our main goal
Speaker 1: is to deploy this responsibly. And if we can come
Speaker 1: up with more responsible frameworks that incorporate many of the
Speaker 1: aspects that we've talked about today, I think we're on
Speaker 1: the brink of actually unpacking what could potentially become the
Speaker 1: next big game changer for people you know that have
Speaker 1: had to rely upon wrong decisions or humans who are
Speaker 1: biased to do such. So I want to say, you know,
Speaker 1: in all honesty, I do agree with you that there's
Speaker 1: a promise of algorithms to sort of break down the barriers,
Speaker 1: but it has to be done responsibly with the right
Speaker 1: people at the table to talk about it. Niculternally thank
Speaker 1: you so much for a really fascinating and rich discussion
Speaker 1: and for sharing your knowledge and expertise with us. Thank
Speaker 1: you now, thank you for having me appreciate you. My
Speaker 1: conversation with Nicole made me want to talk to somebody
Speaker 1: who was doing the kind of work that she was
Speaker 1: just talking about, someone who was thinking about how algorithms
Speaker 1: can break down barriers rather than create them. So I
Speaker 1: called up Talia Gillis. Talia is a PhD student in
Speaker 1: a business economics at Harvard. She's also a former student
Speaker 1: of mine who holds a law degree from Harvard, and
Speaker 1: she's been researching how banks and other lenders use algorithms
Speaker 1: to determine interest rates on loans. She thinks that the
Speaker 1: way they're doing it right now isn't working, but she
Speaker 1: has an idea for a better way. Talia, thank you
Speaker 1: so much for joining me. It's great to have you,
Speaker 1: and it's great to talk to you as it were
Speaker 1: on air about something that we've talked about lots and
Speaker 1: lots of times in the office, your research, because there's
Speaker 1: very much on how we can fix the problem of
Speaker 1: algorithmic bias. Tell me what it is that is the
Speaker 1: core of your approach. What is your original idea about
Speaker 1: what we can do to make things better. So I
Speaker 1: think the core of the approach is, first of all,
Speaker 1: to recognize that it's it's very hard to know a
Speaker 1: priori what exactly the bias or what direction the bias
Speaker 1: is going to go in, and so I'm very much
Speaker 1: focused on the credit pricing context. And in the credit
Speaker 1: pricing context, it's true that a lot of the kind
Speaker 1: of input variables into a credit pricing decision suffer from
Speaker 1: some sort of bias. But what's important to keep in
Speaker 1: mind is that kind of some biases might get worse
Speaker 1: in the algorithmic context, but actually big data might in
Speaker 1: for other types of biases, make things things better in
Speaker 1: a way. I think there's two large, separate categories of bias.
Speaker 1: So the first is what I call kind of inputs
Speaker 1: that result from a bias world, and the idea there
Speaker 1: is that there's some kind of pre existing discrimination, and
Speaker 1: so there might be disparities between men and women, or
Speaker 1: between blacks and whites that kind of originate partially from
Speaker 1: that discrimination. So that's what you're calling biased world, and
Speaker 1: that is you're going to apply for a loan. If
Speaker 1: you make less money and you have more debt, you're
Speaker 1: not going to get as good terms for the loan.
Speaker 1: But that's not because the lender in particular, that's because
Speaker 1: you live in a society where there's background sexist and
Speaker 1: there's background racism. The world is already biased. And so
Speaker 1: in that sense, that's biased world. And then what's the
Speaker 1: second category? And so the second category is inputs that
Speaker 1: are bias because they result from some kind of bias measurement.
Speaker 1: You can think of that as, for example, the way
Speaker 1: we measure someone's income. You know, we might put a
Speaker 1: lot of weight on someone who has one regular job,
Speaker 1: regular paycheck, and we fully capture their income and compare
Speaker 1: that to someone who kind of has multiple jobs, maybe
Speaker 1: isn't in a formal employee employer relationship, like an uber driver,
Speaker 1: and then we kind of discount their income or don't
Speaker 1: measure it properly, or or don't have the ability to
Speaker 1: fully capture what they're earning. And so the more you
Speaker 1: might look at two people who they're underlying income is similar,
Speaker 1: but because of the way we're measuring a person's income,
Speaker 1: then we consider kind of the second person to have
Speaker 1: a lower income. So that's an example of bias in
Speaker 1: the way we're measuring where we're whether we mean to
Speaker 1: or not, we're systematically giving an advantage to someone who
Speaker 1: works in a nine to five as opposed to someone
Speaker 1: who's in the gig economy. And that's what you're calling
Speaker 1: bias in measurement. And now how would you go about
Speaker 1: measuring which kind of bias or what kind of bias
Speaker 1: is in fact found in the algorithm. So it's it's
Speaker 1: quite difficult to in reality perfectly distinguish between these two biases.
Speaker 1: Also because very often, like an example I gave with income,
Speaker 1: it might be a combination of those two. And what
Speaker 1: do you do if you're in not biased world, but
Speaker 1: biased German situation where you're worried not about the backround
Speaker 1: discrimination in the world, but more worried that you're measuring
Speaker 1: the wrong things in the algorithm and as a result,
Speaker 1: you know, having a bad effect on their community of color.
Speaker 1: So with bias measurement, what's interesting about the algorithmic context
Speaker 1: is that it actually might mitigate a lot of the
Speaker 1: harms that we're concerned about in the context of bias measurement.
Speaker 1: So if you take, for example, credit scores, there have
Speaker 1: been many claims that credit scores are biased against minorities,
Speaker 1: and that's because they measure certain qualities of credit worthiness
Speaker 1: that are more representative of let's say, white borrowers. So
Speaker 1: it puts a lot of weight on people repaying previous
Speaker 1: loans on time, but it might not give any weight
Speaker 1: to people who regularly made let's say rent payments, which
Speaker 1: might actually also be a very good measure of a
Speaker 1: person's credit worthiness. So in a world in which we
Speaker 1: put a lot a lot of weight on a credit score,
Speaker 1: if we moved to a world of kind of machine
Speaker 1: learning and big data, we might get a whole new
Speaker 1: richness of indicators of a person's credit worthiness. So let's
Speaker 1: say the algorithm, how did your full history of payments
Speaker 1: or your full kind of consumer history. Then we might
Speaker 1: be getting a lot more information out about a person's
Speaker 1: credit worthiness that was before only limited to the credit score.
Speaker 1: So this is an example where if we could identify
Speaker 1: the bias in the measurement, then we could do better
Speaker 1: with the algorithm. Yes, what about a situation where the
Speaker 1: opposite is happening where the algorithm is taking into account
Speaker 1: things that are producing measurement bias. How do we know
Speaker 1: that that's happening. So I think that the key is
Speaker 1: that we never truly know what's going on. Say more
Speaker 1: about that, because I think that scares a lot of
Speaker 1: people with respect to these algorithms. What does it mean
Speaker 1: to say we never truly know what's going on? Well,
Speaker 1: on the one hand, you could say it's scary, But
Speaker 1: on the other hand, you could say that any attempt
Speaker 1: to say it's necessarily bad or necessarily going to hurt populations.
Speaker 1: It's going to be a difficult position to defend because
Speaker 1: it's kind of more of an empirical question that requires
Speaker 1: investigation rather than something that you can determine ahead of time.
Speaker 1: Now tell you that sounds logically correct to me. You know,
Speaker 1: you to You don't know for sure something's bad or
Speaker 1: good until you test it out and you have to
Speaker 1: examine it, and principle I agree with you. But what
Speaker 1: would you say to someone who said, well, look, we
Speaker 1: know how the world works in general, and the world
Speaker 1: doesn't turn out so well. Often traditionally discriminated against groups,
Speaker 1: and so our instinct is that we expect to find
Speaker 1: discrimination rather than to find magic, whereby an algorithmic measurement
Speaker 1: does better than a human How would you respond to
Speaker 1: that kind of systemic skepticism that I think one very
Speaker 1: reasonably hears from people who are concerned that existing bias
Speaker 1: will be made worse by algorithms, rather than being optimistic
Speaker 1: about the capacity of algorithms to block certain kinds of bias.
Speaker 1: You'd have to, I mean, particularly in the credit context,
Speaker 1: you'd have to be very sensitive to the fact that
Speaker 1: credit markets are not working for large segments of the
Speaker 1: US population. So many people in the US don't have
Speaker 1: access to credit. Many people don't have credit histories, they
Speaker 1: don't have credit scores. So if you were defending the
Speaker 1: status quo and credit pricing, then you would have a
Speaker 1: really big difficulty in terms of actually kind of blocking
Speaker 1: the potential that this technological move has in terms of
Speaker 1: expanding access and creating kind of access to credit markets
Speaker 1: to populations that before have just simply been excluded from
Speaker 1: these markets. So things are so bad right now, you're
Speaker 1: saying that it would be crazy enough to at least
Speaker 1: give this or try because the existing stays quo is
Speaker 1: deeply discriminatory yeah, I think there's this kind of a
Speaker 1: serious entrenchment of pre existing disadvantage in credit markets. And
Speaker 1: when you think that credit markets are very important tool
Speaker 1: not just to kind of not want to replicate disadvantage,
Speaker 1: but also credit markets play a very important role at
Speaker 1: producing wealth or allowing people to kind of come out
Speaker 1: of some kind of situation in which they were blocked
Speaker 1: from kind of expanding their possibilities, because if you can't
Speaker 1: borrow money, you can invest in yourself exactly, exactly right, Okay,
Speaker 1: So go back then to the question of how we
Speaker 1: run a test to see whether we've got biased measurement.
Speaker 1: How do we make sure that we're making things better
Speaker 1: with the algorithm with respect to measurement bias, not making
Speaker 1: things worse. How would you test that in real world?
Speaker 1: So I think what's key is to have a kind
Speaker 1: of baseline in which the key question is when I
Speaker 1: move from that baseline to a new situation, how are
Speaker 1: things changing. And so the key question to me is,
Speaker 1: if we're comparing kind of a traditional pricing situation to
Speaker 1: an algorithmic pricing situation, what's happening? And to do that
Speaker 1: I would do is I would take kind of the
Speaker 1: algorithmic pricing function. And again, the big advantage in a
Speaker 1: way of the algorithmic context is that even before you
Speaker 1: actually apply your decision rule or your prediction to a
Speaker 1: new borrower who comes through the door, you're able to
Speaker 1: say something about the algorithm itself. So it sounds like overall,
Speaker 1: the key tool of social science that you think needs
Speaker 1: to be used to help us overcome the possibilities of
Speaker 1: different kinds of algorithmic bias is the experiment. It's experiment
Speaker 1: by setting a baseline, then experiment and see what happens
Speaker 1: when the algorithm is applied, and then compare them and
Speaker 1: then make a judgment afterwards. It sounds like you're saying,
Speaker 1: we never know from just looking at an algorithm what's
Speaker 1: going to happen, whether it's going to make the world
Speaker 1: a worst place, where that's going to make the world
Speaker 1: a better place. We always have to test it. And
Speaker 1: in a sense that seems to me very scientific, right,
Speaker 1: very economic scientific. Run the experiment and see see what
Speaker 1: comes out on the other side. Do other people agree
Speaker 1: with you? I mean, how far out are you on
Speaker 1: the edge and calling for experiment in every case? Well,
Speaker 1: I think there's several difficulties. I think the first difficulty
Speaker 1: is kind of maybe a legal theoretical difficulty, and that
Speaker 1: is that traditionally, the way we've always thought about kind
Speaker 1: of discrimination and evaluating a lender for discrimination purposes was
Speaker 1: kind of the exact opposite. It was considering what are
Speaker 1: the inputs into the decision to price alone, and not
Speaker 1: what's the outcome. So this whole way of thinking about
Speaker 1: testing is very focused on the outcome of a pricing rule.
Speaker 1: So in legal terms, instead of asking about discriminatory intent
Speaker 1: by the person making the decision, you're asking about whether
Speaker 1: there's a disparate impact on people at the end and
Speaker 1: the outcomes. That's right, Do you want us to focus
Speaker 1: on outputs? Exactly? Exactly? So there's quite kind of this
Speaker 1: fundamental shift that I think needs to take place in
Speaker 1: moving from being very focused on what poes into a
Speaker 1: decision or what poes into an algorithm and saying there's
Speaker 1: not much progress that we can make on focusing just
Speaker 1: on the inputs. We really need to go to the
Speaker 1: outcomes and consider the outcomes more seriously. I think Talia's
Speaker 1: idea for fixing algorithmic bias has some profound implications. One
Speaker 1: of the things about algorithms is you can't really know
Speaker 1: what all the inputs are because often, in the case
Speaker 1: of a sophisticated algorithm which is based on machine learning,
Speaker 1: we don't know how it's learning from the data. We
Speaker 1: just know that it's looking at every aspect of the data,
Speaker 1: but we don't know exactly what its true inputs are.
Speaker 1: In other words, we know what data it's training on,
Speaker 1: but we don't know what features of the data it
Speaker 1: cares the most about. And that's why Talia sees her
Speaker 1: approach of running experiments and looking at the outputs as
Speaker 1: the only potential solution. She's if people begin gradually to
Speaker 1: see things the way that Taya does, I wonder if
Speaker 1: that could lead us to a new paradigm more broadly
Speaker 1: about how we think of discrimination. It might lead us
Speaker 1: away from the old way of asking, well, was the
Speaker 1: person of making the decision or racist, and towards a
Speaker 1: newer way of thinking, which says, who cares what the
Speaker 1: person was thinking about? What we want to see is
Speaker 1: whether the system as a whole is producing outcomes that
Speaker 1: we think are fair and just. That's all in the future,
Speaker 1: and right now the Trump administration is proposing regulations that
Speaker 1: actually go in the opposite direction. The Department of Housing
Speaker 1: and Urban Development has proposed a new rule that would
Speaker 1: make it harder for banks, or landlords or homeowners insurance
Speaker 1: companies to be sued for using algorithms that result in
Speaker 1: discriminatory lending practices, and the Trump administration has gone to
Speaker 1: the courts more broadly to suggest that they think there
Speaker 1: needs to be a stronger showing of actual racist intent
Speaker 1: before discrimination claims can be leveled. So the trend line
Speaker 1: is not the line that Talia is calling for, nor
Speaker 1: is it the line that machine learning and artificial intelligence
Speaker 1: would suggest for us. In its most extreme form, the
Speaker 1: Trump administration approach might actually allow racist bias to be
Speaker 1: imported into the functioning of algorithmic systems, and exactly the
Speaker 1: way that Nicole wants to avoid. We'll be watching very
Speaker 1: closely going forward to see how those proposed Trump administration
Speaker 1: regulations are treated, how the courts address the question of bias,
Speaker 1: and most profoundly, how algorithms shape justice in the future. Now,
Speaker 1: I want to move to a new segment of deep background,
Speaker 1: something we're calling Sound of the Week. For me, this week,
Speaker 1: a defining moment in sound was this everybody to know.
Speaker 1: There are no circumstances in which I will ask Russels
Speaker 1: to delay. We're leaving on the thirty first of October.
Speaker 1: No ifs or butts. We will not accept any attempt
Speaker 1: to go back on our promises or scrub that referendum.
Speaker 1: That's Boris Johnson on Monday, making a public statement in
Speaker 1: front of ten Downing Street, where for the moment he
Speaker 1: still lives and works as the Prime Minister of the
Speaker 1: United Kingdom. But things have changed a lot since then. First,
Speaker 1: in a remarkable development, the Parliament of Great Britain, including
Speaker 1: a group of rebels from Johnson's own Conservative Party, actually
Speaker 1: voted that the United Kingdom cannot crash out of the
Speaker 1: European Union with a no deal brexit. Johnson will actually
Speaker 1: be required by this law to seek an extension from
Speaker 1: the European Union so that Britain does not leave the
Speaker 1: Union without some kind of a deal. Johnson's position has
Speaker 1: been all along that this would be terrible for his
Speaker 1: negotiating position with the European Union since he's got nothing
Speaker 1: to threaten, but Parliament didn't care. Having lost this vote,
Speaker 1: Johnson then turned around and did two things. One more
Speaker 1: shocking than next. First, he kicked out of the Conservative
Speaker 1: Party twenty one members of Parliament who had voted against him.
Speaker 1: This was sufficiently shocking that his own brother, Joe Johnson,
Speaker 1: actually resigned from his seat in Parliament and from Johnson's cabinet,
Speaker 1: saying that he felt a conflict between the national interest
Speaker 1: and his family loyalties. Then, having taken that radical step,
Speaker 1: Johnson asked Parliament to vote for a snap election. Now
Speaker 1: it takes two thirds of Parliament to vote for a
Speaker 1: snap election for it to happen right away, and the
Speaker 1: Conservatives didn't get it. So where we are now is it?
Speaker 1: Boris Johnson can't get out of the European Union on
Speaker 1: October thirty first, whether he wants to or not. And
Speaker 1: so far, at least he still doesn't have a general
Speaker 1: election in which he could try to ask the voters
Speaker 1: to change the government in order to change this law.
Speaker 1: Stunning developments, historically significant moments in the history of British politics.
Speaker 1: What's their more profound meaning, I'll tell you what's been
Speaker 1: on my mind. There's a deep contradiction between the idea
Speaker 1: of a referendum that would allow the public as a
Speaker 1: whole to decide on an important question like whether to
Speaker 1: leave the European Union and parliamentary democracy, which is based
Speaker 1: on the idea that the people choose representatives who then
Speaker 1: exercise their practical judgment and their wisdom to implement the
Speaker 1: policies of the country. Notice how this contradiction has driven
Speaker 1: Britain into a kind of paralysis that I would say
Speaker 1: even veers occasionally on madness. First, the public says leave,
Speaker 1: but it doesn't say how to leave. Then it tells
Speaker 1: its elected representatives figure out how to do it, and
Speaker 1: they can't figure it out. They can't agree. Proposal after
Speaker 1: proposal gets blocked. Good idea follows bad idea follows bad
Speaker 1: idea follows good idea, and nothing seems to work itself out.
Speaker 1: And the whole time the politicians are saying, well, we
Speaker 1: can't reach an agreement, but we know we have to
Speaker 1: give effect to the will of the people in the referendum.
Speaker 1: This is the product of a mismatch between the idea
Speaker 1: that you can take a snapshot of public opinion at
Speaker 1: a given moment and call that a referendum, and the
Speaker 1: idea that the best way to run a government is
Speaker 1: in fact through electing representatives and have them use their judgment.
Speaker 1: So is there a way out of this contradiction for Britain.
Speaker 1: If I were an optimist, I would say that the
Speaker 1: British could call a new election and that the outcome
Speaker 1: of that election would somehow clarify whether people favored a
Speaker 1: change or not. But I don't actually believe that a
Speaker 1: new election is going to make things any clearer with
Speaker 1: respect to the contradiction between the referendum and ordinary voting.
Speaker 1: So then you might imagine, how about another referendum that
Speaker 1: asks people, well, what did you change your mind? Do
Speaker 1: you want us to leave without a deal? If so,
Speaker 1: what kind of a deal? Notice that almost immediately you
Speaker 1: get into the kind of details that a referendum cannot answer.
Speaker 1: There's no way that a referendum can do anything other
Speaker 1: than ask up or down do you want this or
Speaker 1: do you want that? The only way that that's problem
Speaker 1: could be solved would be if a specific deal were
Speaker 1: put in front of the British people and they were
Speaker 1: asked whether to take it or not. And even then
Speaker 1: that would leave the question of what to do afterwards.
Speaker 1: It emerges that the British have simply gone down a
Speaker 1: rabbit hole of contradiction between these two modes of democracy,
Speaker 1: direct democracy by referendum and representative democracy by parliament, and
Speaker 1: the only way they're going to get out of it
Speaker 1: is if they abandon one of the these two modes
Speaker 1: of action. They're not going to abandon in Parliament, which is,
Speaker 1: by most accounts the oldest continuously running political body in
Speaker 1: any democratic country. At least I hope they won't. What
Speaker 1: they might learn is that if you've got something as
Speaker 1: good as Parliament is, maybe you should stay away from
Speaker 1: their referendums. And if that happens, then over time the
Speaker 1: British will be able to re establish the norm of
Speaker 1: parliamentary supremacy and parliamentary sovereignty. It ain't perfect, but it's
Speaker 1: worked for a long long time. And if there's one
Speaker 1: takeaway from the briggs At fiasco is that when the
Speaker 1: British try to deviate from it, they do not know
Speaker 1: what they're doing. Deep Background is brought to you by
Speaker 1: Pushkin Industries. Our producer is Lydia Genecott, with engineering by
Speaker 1: Jason Gambrell and Jason Roskowski. Our showrunner is Sophie mckibbon.
Speaker 1: Our theme music is composed by Luis GERA special thanks
Speaker 1: to the Pushkin Brass, Malcolm Gladwell, Jacob Weisberg, and Miah Lobel.
Speaker 1: I'm Noah Feldman. You can follow me on Twitter at
Speaker 1: Noah R. Feldman. This is deep background