Speaker 1: Pushkin. You're listening to Brave New Planet, a podcast about
Speaker 1: amazing new technologies that could dramatically improve our world. Or
Speaker 1: if we don't make wise choices, could leave us a
Speaker 1: lot worse off, Utopia or dystopia. It's up to us.
Speaker 1: On November eleventh, twenty sixteen, the Babelfish burst from fiction
Speaker 1: into reality. The Babelfish was conceived forty years ago in
Speaker 1: Douglas Adam's science fiction classic The Hitchhiker's Guide to the Galaxy.
Speaker 1: In the story, a hapless Earthling finds himself a stowaway
Speaker 1: on a Vogon spaceship. When the alien captain starts an
Speaker 1: announcement over the loudspeaker, his companion tells him to stick
Speaker 1: a small yellow fish in his ear. Listen, it's important,
Speaker 1: it's a I can't just put this in your ear.
Speaker 1: Suddenly he's able to understand the language. The Babelfish is small, yellow,
Speaker 1: leech like and probably the oddest thing in the universe.
Speaker 1: It feeds on brainwave energy, whose ambing all unconscious frequencies,
Speaker 1: the practical upshot of which is that if you stick
Speaker 1: one in your ear, you instantly understand anything said to
Speaker 1: you in any form of language. At the time, the
Speaker 1: idea of sticking an instantaneous universal translator in your ear
Speaker 1: seems charmingly absurd, But a couple of years ago, Google
Speaker 1: and other companies announced plants to start selling Babelfish well
Speaker 1: not fish actually, but earbuds that do the same thing.
Speaker 1: The key breakthrough came in November twenty sixteen, when Google
Speaker 1: replaced the technology behind its translate program. Overnight, the Internet
Speaker 1: realized that something extraordinary had happened. A Japanese computer scientist
Speaker 1: ran a quick test. He dashed off his own Japanese
Speaker 1: translation of the opening lines of Ernest Hemingway's short story
Speaker 1: The Snows of Kilmanjaro, and dared Google Translate to turn
Speaker 1: it back into English. Here's the opening passage from the
Speaker 1: Simon and Schuster audio book. Kilimanjaro is a snow covered
Speaker 1: mountain nineteen thousand, seven hundred and ten feet high and
Speaker 1: is said to be the highest mountain in Africa. Its
Speaker 1: western summit is called the Massai Nagaji Nagai, the House
Speaker 1: of God. Close to the western summit there is the
Speaker 1: dried and frozen carcass of a leopard. No one has
Speaker 1: explained what the leopard seeking at that altitude. Let's just
Speaker 1: consider that last sentence. No one has explained what the
Speaker 1: leopard was seeking at that altitude. One day earlier, Google
Speaker 1: had mangled the back translation quote. Whether the leopard had
Speaker 1: what the demand at that altitude? There is no that
Speaker 1: nobody explained. But now Google Translate returned quote. No one
Speaker 1: has ever explained what leopard wanted at that altitude. It
Speaker 1: was perfect except for a missing the the what explained
Speaker 1: the great leap? While Google had built a predictive algorithm
Speaker 1: that taught itself how to translate between English and Japanese
Speaker 1: by training on a vast library of examples and tweaking
Speaker 1: its connections to get better and better at predicting the
Speaker 1: right answer. In anyways, the algorithm was a black box.
Speaker 1: No one understood precisely how it worked, but it did
Speaker 1: amazingly well. Predictive algorithms turn out to be remarkably general.
Speaker 1: They can be applied to predict which movies a Netflix
Speaker 1: user will want to see next, or whether an eye
Speaker 1: exam or a mammogram indicates disease. But it doesn't stop there.
Speaker 1: Predictive algorithms or also being trained to make societal decisions
Speaker 1: who to hire for a job, whether to approve a
Speaker 1: mortgage application, what students to let into a college, what
Speaker 1: a rest ease to let out on bail? But what
Speaker 1: exactly are these big black boxes learning from massive data sets?
Speaker 1: Are they gaining deep new insights about people? Or might
Speaker 1: they sometimes be automating systemic biases? Today's big question when
Speaker 1: should predictive algorithms be allowed to make big decisions about people?
Speaker 1: And before they judge us, should we have the right
Speaker 1: to know what's inside the black box? My name is
Speaker 1: Eric Lander. I'm a scientist who works on ways to
Speaker 1: improve human health. I helped lead the Human Genome Project,
Speaker 1: and today I lead the Broad Institute of MIT and Harvard.
Speaker 1: In the twenty first century, powerful technologies have been appearing
Speaker 1: at a breathtaking pace, related to the Internet, artificial intelligence,
Speaker 1: genetic engineering, and more. They have amazing potential upsides, but
Speaker 1: we can't ignore the risks that come with them. The
Speaker 1: decisions aren't just up to scientists or politicians. Whether we
Speaker 1: like it or not, we all of us are the
Speaker 1: stewards of a brave New planet. This generation's choices will
Speaker 1: shape the future as never before. Coming up on today's
Speaker 1: episode of Brave New Planet, predictive algorithms. We hear from
Speaker 1: a physician at Google about how this technology might help
Speaker 1: keep millions of people with diabetes from going blind, and
Speaker 1: the idea was, well, if you could retrain the model,
Speaker 1: you could get to more patients to screen them for disease.
Speaker 1: The first iteration of the model was on par with
Speaker 1: the US board sortified ophomologists. I speak with an AI
Speaker 1: researcher about how predictive algorithms sometimes learn to be sexist
Speaker 1: and racist. If you typed in I am a white man,
Speaker 1: you would get positive sentiment. If you typed in I
Speaker 1: am a black lesbian, for example, negative sentiment. We hear
Speaker 1: how algorithms are affecting the criminal justice system. For black defendants,
Speaker 1: it was much more likely to incorrectly predict that they
Speaker 1: were going to go on to come in a future
Speaker 1: crime when they didn't, and for white defend it was
Speaker 1: much more likely to predict that they were going to
Speaker 1: go on to not commit a future crime when they did.
Speaker 1: And we hear from a policy expert about whether these
Speaker 1: systems should be regulated. A lot of the horror stories
Speaker 1: are about fully implemented tools that were in work for years.
Speaker 1: There's never a pause button. To reevaluate or look at
Speaker 1: how a system is working real time. Stay with us
Speaker 1: Chapter one, The Big Black Box. To better understand these algorithms,
Speaker 1: I decided to speak with one of the creators of
Speaker 1: the technology that transformed Google Translate. My name is Greg Kurado,
Speaker 1: and I'm a distinguished scientist at Google Research. Early in
Speaker 1: his career, Greg had trained neuroscience, but he soon shifted
Speaker 1: his focus from organic intelligence to artificial and not turned
Speaker 1: out to be really a very lucky moment, because I
Speaker 1: was becoming interested in artificial intelligence at exactly the moment
Speaker 1: that artificial intelligence was changing so much. Ever since the
Speaker 1: field of artificial intelligence started more than sixty years ago,
Speaker 1: there have been two warring approaches about how to teach
Speaker 1: machines to do human tasks. We might call them human
Speaker 1: rules versus machine learning. The way that we used to
Speaker 1: try to get computers to recognize patterns was to program
Speaker 1: into them specific rules. So we would say, oh, well,
Speaker 1: you can tell the difference between a cat and a
Speaker 1: dog by how long it's whiskers are and what kind
Speaker 1: of fur it has and does it have stripes? And
Speaker 1: trying to put these rules into computers. It kind of worked,
Speaker 1: but it made for a lot of mistakes. The other
Speaker 1: approach was machine learning, let the computer figure everything out
Speaker 1: for itself, somewhat like the biological brain. The machine learning
Speaker 1: system is actually built of tiny little decision makers or neurons.
Speaker 1: They start out connected very much in random ways, but
Speaker 1: we give the system feedback. So, for example, if it's
Speaker 1: guessing between a cat and a dog and it gets
Speaker 1: one wrong, we tell the system that it got one wrong,
Speaker 1: and we make little changes inside so that it's much
Speaker 1: more likely to recognize that cat as a cat and
Speaker 1: not mistake it for a dog. Over time, the system
Speaker 1: gets better and better and better. Machine learning had been
Speaker 1: around for decades with rather unimpressive results. The number of
Speaker 1: connections and neurons in those early systems was pretty small.
Speaker 1: We didn't realize until about two and ten that computers
Speaker 1: had gotten fast enough and the data sets were big
Speaker 1: enough that these systems could actually learn from patterns and
Speaker 1: learn from data better than we could describe rules manually.
Speaker 1: Machine learning made huge leaps. Google itself became the leading
Speaker 1: driver of machine learning. In twenty eleven, Krrado joined with
Speaker 1: two colleagues to form a unit called Google Brain. Among
Speaker 1: other things, they applied a machine learning approach to language translation.
Speaker 1: The strategy turned out to be remarkably effective. It doesn't
Speaker 1: learn French the way you would learn French in high school.
Speaker 1: It learns French the way you would learn French at home,
Speaker 1: much more like the way that a child learns the language.
Speaker 1: We give the machine the English sentence, and then we
Speaker 1: give it an example of a French translation of that
Speaker 1: whole sentence. We show a whole lot of them, probably
Speaker 1: more French and English sentences than you could read in
Speaker 1: your whole life. And by seeing so many examples of
Speaker 1: entire sentences, the system is able to learn, oh, this
Speaker 1: is how I would say this in French. That's actually,
Speaker 1: at this point about as good as a biling human
Speaker 1: would produce. Soon Google was training predictive algorithms for all
Speaker 1: sorts of purposes. We use neural network predictors to help
Speaker 1: rank search results, tell people organize their photos, to recognize speech,
Speaker 1: to find driving directions, to help complete emails. Really anything
Speaker 1: that you can think of where there's some notion of
Speaker 1: finding a pattern or making a prediction, artificial intelligence might
Speaker 1: be at play. Predictive algorithms would become ubiquitous in commerce.
Speaker 1: They let Netflix know which movies to recommend to each customer,
Speaker 1: Amazon to suggest products users might be interested in purchasing,
Speaker 1: and much more well, they're shockingly useful, they can also
Speaker 1: be inscrutable. Modern neural networks are like a black box.
Speaker 1: Understanding how they make their predictions can be surprisingly difficult.
Speaker 1: When you build an artificial neural network, you do not
Speaker 1: necessarily understand exactly the final state of how it works.
Speaker 1: Figuring out how it works becomes its own science project.
Speaker 1: One thing we do know. Predictive algorithms are especially sensitive,
Speaker 1: so the choice of examples used to train them. The
Speaker 1: systems learn to imitate the examples in the data that
Speaker 1: they see. You don't know how well they will do
Speaker 1: on things that are very different. So, for example, if
Speaker 1: you train a system to recognize cats and dogs, but
Speaker 1: you only ever show it border collies and tabbycats, it's
Speaker 1: not clear what it will do. When you show it
Speaker 1: a picture of chihuahua, all it's ever seen as border collies,
Speaker 1: it may not get the right answer. So its concept
Speaker 1: of dog is going to be limited by the dogs
Speaker 1: it's seen. That's right, and this is why diversity of
Speaker 1: data in machine learning systems is so important. You have
Speaker 1: to have a data set that represents the entire spectrum
Speaker 1: of possibilities that you expect the system to work under.
Speaker 1: Teaching algorithms turns out to be not so different than
Speaker 1: teaching people. They learn what they see. Chapter two, retina fundoscopy.
Speaker 1: It's cool that predictive algorithms can learn to translate languages
Speaker 1: and suggest movies, but what about more life changing applications.
Speaker 1: My name is Lily Ping. I am a physician by training,
Speaker 1: and I am a product manager at Google. I went
Speaker 1: to visit doctor Ping because she and her colleagues are
Speaker 1: using predictive algorithms to help millions of people avoid going blind. So.
Speaker 1: Diabetic retinopathy is a complication of diabetes that affects the
Speaker 1: back of the eye, the retina. One of the devastating
Speaker 1: complications is vision loss. All patients that have diabetes need
Speaker 1: to be screened once a year for a diabetic retnopathy.
Speaker 1: This is an asymptomatic disease, which means that you do
Speaker 1: not feel the symptoms. You don't experienced vision loss until
Speaker 1: it's too late. Now, diabetes is epidemic around the world.
Speaker 1: How many diabetics are there? Though by most estimates, there
Speaker 1: are over four hundred million patients in the world with diabetes.
Speaker 1: How do you screen a patient to see whether they
Speaker 1: have diabetic retinopathy. You need to have a special camera
Speaker 1: while a fundis camera and it takes a picture through
Speaker 1: the people of the back of the eye. We have
Speaker 1: a very small supply of retina specialists and eye doctors
Speaker 1: and they do a lot more than reading images, so
Speaker 1: they needed to scale the reading of these images. Four
Speaker 1: hundred million people with diabetes. There just aren't enough specialists
Speaker 1: for all the retinal images that need reading, especially in
Speaker 1: some countries in Asia where resources are limited and the
Speaker 1: incidence of diabetes is skyrocketing. Two hospitals in southern India
Speaker 1: recognize the problem and reached out to Google for help.
Speaker 1: That point, Google was already sort of well known for
Speaker 1: image recognition. We were classifying cats and dogs and consumer images,
Speaker 1: and the idea was, well, if you could retrain the
Speaker 1: model to recognize diabetic retinopathy, you could potentially help the
Speaker 1: hospitals in India get to more patients to screen them
Speaker 1: for disease. How did you and your colleagues set out
Speaker 1: to attack this problem? So when I first started the project,
Speaker 1: we had about one hundred thirty thousand images from eye
Speaker 1: hospitals in India as well as a screening program in
Speaker 1: the US. Also, we gathered the army of opthalmologists to
Speaker 1: grade them eight hundred eighty thousand diagnoses or rendered on
Speaker 1: one hundred thirty thousand images. So we took this training
Speaker 1: data and we put it in a machine learning model
Speaker 1: and had to do The first iteration of the model
Speaker 1: was on par with the US board sortified ophomologists. Since then,
Speaker 1: we've made some improvements to the model, and the initial
Speaker 1: training took about how long The first time we train
Speaker 1: a model, it may have taken a couple of weeks,
Speaker 1: But then the second time you train the next models
Speaker 1: and next models, it's just it's shorter and shorter, sometimes overnight,
Speaker 1: sometimes overnight. Well, yes, all right, And by contrast, how
Speaker 1: long does it take to train a board certified ophthimologist,
Speaker 1: So that usually takes at least five years, and then
Speaker 1: you also have additional fellowship ears to specialize in the retina.
Speaker 1: And at the end of that you only have one
Speaker 1: board certified ophthimologist. Yes, at the end of that you'd
Speaker 1: have one very very well trained doctor, but that's not scaled. Yes,
Speaker 1: So by contrast, a model like this scales worldwide and
Speaker 1: never fatigues. It consistently gives the same diagnosis on the
Speaker 1: same image, and it obviously takes a much shorter time
Speaker 1: to train. That being said, it does a very very
Speaker 1: narrow task that is just a very small portion of
Speaker 1: what that doctor can do. The retina screening tools already
Speaker 1: being used in India, It was recently approved in Europe
Speaker 1: and its under review in the United States. Groups around
Speaker 1: the world are now working on other challenges in medical imaging,
Speaker 1: like detecting breast cancers at earlier stages. But I was
Speaker 1: particularly struck by a surprising discovery by Lily's team that
Speaker 1: unexpected information about patients was hiding in their retinal pictures.
Speaker 1: In the fundess image, there are blood vessels, and so
Speaker 1: one of the thoughts that we had was, because you
Speaker 1: can see these vessels, I wonder if we can predict
Speaker 1: cardiovascular disease from the same image. So we did an
Speaker 1: experiment where we took fundess images and we train a
Speaker 1: model to predict whether or not that patient would have
Speaker 1: a heart attack in five years. We found that we
Speaker 1: could tell whether or not this patient may have a
Speaker 1: a vascular event much better than doctors. It speaks to
Speaker 1: what might be in this data that we've overlooked. The
Speaker 1: model could make predictions that doctors couldn't from the same
Speaker 1: type of data. It turned out the computer could also
Speaker 1: do a reasonable job of predicting a patient sex, age,
Speaker 1: and smoking status. The first time I did this with
Speaker 1: an ahomologist, I think she thought I was trolling her.
Speaker 1: I said, well, here pictures. Guess which one is a woman,
Speaker 1: Guess which one is a man. Guess which one's a smoker,
Speaker 1: Guess which one is young. Right, these are all tasks
Speaker 1: that doctors don't generally do with these images. It turns
Speaker 1: out the model was right ninety eight ninety nine percent
Speaker 1: of the time. That being said, there are much easier
Speaker 1: ways of getting the sex of a patience. So so,
Speaker 1: while scientifically interesting, this is one of the most useless
Speaker 1: clinical predictions ever. So how far can it go? If
Speaker 1: you gave preference for rock music or not? What do
Speaker 1: you think? You know? We tried predicting happiness. That didn't work,
Speaker 1: So I'm guessing rock music. Oh, probably not, but who knows. So.
Speaker 1: Predictive algorithms can learn a remarkable range of tasks, and
Speaker 1: they can even discover hidden patterns that humans miss. We
Speaker 1: just have to give them enough training data to learn from.
Speaker 1: Sounds pretty fantastic. What could possibly go wrong? Chapter three?
Speaker 1: What could possibly go wrong? If predictive algorithms can use
Speaker 1: massive data to discover unexpected connections between your eye and
Speaker 1: your heart, what might they be learning about, say, human society.
Speaker 1: To answer this question, I took a trip to speak
Speaker 1: with Kate Crawford, the co founder and co director of
Speaker 1: the AI Now Institute at New York University. When we
Speaker 1: begin and we were the world's first AI institute dedicated
Speaker 1: to studying the social implications of these tools. To me,
Speaker 1: these are the biggest challenges that we face right now,
Speaker 1: simply because we've spent decades looking at these questions from
Speaker 1: a technical lens at the expense of looking at them
Speaker 1: at a social and an ethical lens. I knew about
Speaker 1: Kate's work because we served together on a working group
Speaker 1: about artificial intelligence for the US National Institutes of Health.
Speaker 1: I also knew she had an interesting background. I grew
Speaker 1: up in Australia. I studied a really strange grab bag
Speaker 1: of disciplines. I studied law, I studied philosophy, and then
Speaker 1: I got really interested in computer science, and this was
Speaker 1: happening at the same time as I was writing electronic
Speaker 1: music on large scale modulus synthesizers, and that's still a
Speaker 1: thing that I do today. It's almost like the opposite
Speaker 1: of artificial intelligence because it's so analog, so I absolutely
Speaker 1: love it for that reason. In the year two thousand
Speaker 1: and Kate's band released an album entitled twenty twenty that
Speaker 1: included a pression song called Machines work so that people
Speaker 1: have time to think. It's funny because we use a
Speaker 1: sample from an early IBM promotional film that was made
Speaker 1: in the nineteen sixties, which says machines can do the
Speaker 1: work so that people have time to think, and we
Speaker 1: actually ended up sort of cutting it and splicing it
Speaker 1: in the track, so it ends up saying that people
Speaker 1: can do the work so that machines have time to think.
Speaker 1: And strangely, the more that I've been working in the
Speaker 1: sort of machine learning space, I think, yeah, there's a
Speaker 1: lot of ways in which actually people are doing the
Speaker 1: work so that machines can do all the thinking. Kate
Speaker 1: gave me a crash course on how predictive algorithms not
Speaker 1: only teach themselves language skills, but also in the process
Speaker 1: acquire human prejudices, even in something as seemingly benign as
Speaker 1: language translation. So in many cases, if you say, translate
Speaker 1: a sentence like she is a doctor into a language
Speaker 1: like Turkish, and then you translate it back into English,
Speaker 1: and you're saying Turkish because Turkish has pronouns that are
Speaker 1: not gendered precisely, and so you would expect that you
Speaker 1: would get the same sentence back, but you do not.
Speaker 1: It will say he is a doctor, so she is
Speaker 1: a doctor was translated into gender neutral Turkish as all
Speaker 1: beer doctor, which was then back translated into English as
Speaker 1: he is a doctor. In fact, you could see how
Speaker 1: much the predictive algorithms had learned about gender roles. Just
Speaker 1: by giving Google Translate a bunch of gender neutral sentences
Speaker 1: in Turkish. You got he is an engineer, she is
Speaker 1: a cook. He is a soldier, but she is a teacher.
Speaker 1: He is a friend, but she is a lover. He
Speaker 1: is happy and she is unhappy. I find that one
Speaker 1: particularly odd, and it's not just language translation that's problematic.
Speaker 1: The same sort of issues arise in language understanding. Predictive
Speaker 1: algorithms were trained to learn analogies by reading lots of texts,
Speaker 1: they concluded that dog is to puppy as cat is
Speaker 1: to kitten, and man is to king as woman is
Speaker 1: to queen. But they also automatically inferred that man is
Speaker 1: to computer programmer as woman is to homemaker. And with
Speaker 1: the rise of social media, Google used text on the
Speaker 1: Internet to train predictive algorithms to infer the sentiment of
Speaker 1: tweets and online reviews. Is it a positive sentiment? Is
Speaker 1: it a negative sentiment? I believe it was Google who
Speaker 1: released their sentiment engine, and you could just try it online,
Speaker 1: you know, put in a sentiment and see what you'd get.
Speaker 1: And again, similar problems emerged. If you typed in I
Speaker 1: am a white man, you would get positive sentiment. If
Speaker 1: you typed in a black lesbian, for example, negative sentiment.
Speaker 1: Just as Greg Korado explained with chihuahuas and border collies,
Speaker 1: the predictive algorithms were learning from the examples they found
Speaker 1: in the world, and those examples reflected a lot about
Speaker 1: past practices and prejudices. If we think about where you
Speaker 1: might be scraping large amounts of text from say Reddit,
Speaker 1: for example, and you're not thinking about how that sentiment
Speaker 1: might be biased against certain groups, then you're just basically
Speaker 1: importing that directly into your tool. But it's not just
Speaker 1: conversations on Reddit. There's the cautionary tale of what happens
Speaker 1: when Amazon let a computer teach itself how to sift
Speaker 1: through mountains of resumes for computer programming jobs to find
Speaker 1: the best candidates to interview. So they set up this system,
Speaker 1: they designed it, and what they found was a very
Speaker 1: quickly this system had learned to discard and really demote
Speaker 1: the applications from women. And typically if you had a
Speaker 1: women's college mentioned, and even if you had the word
Speaker 1: women's on your resume, your application would go to the
Speaker 1: bottom of the pile. All right, So how does it
Speaker 1: learn that? So, first of all, we take a look
Speaker 1: at who is generally hired by Amazon, and of course
Speaker 1: they have a very heavily skewed male workforce, and so
Speaker 1: the system is learning that these are the sorts of
Speaker 1: people who will tend to be hired and promoted. And
Speaker 1: it is not a surprise then that they actually found
Speaker 1: it impossible to really retrain this system. They ended up
Speaker 1: abandoning this tool because simply correcting for a bias is
Speaker 1: very hard to do when all of your ground truth
Speaker 1: data is so profoundly skewed in a particular direction. So
Speaker 1: Amazon dropped this particular machine learning project and Google fixed
Speaker 1: the Turkish to English problem. Today, Google Translate gives both
Speaker 1: he is a doctor and she is a doctor as
Speaker 1: translation options. But biases keep popping up in predictive algorithms
Speaker 1: in many settings, there's no systematic way to prevent them. Instead,
Speaker 1: spotting and fixing biases has become a game of whacamole
Speaker 1: Chapter four quarterbacks. Perhaps it's no surprise that algorithms trained
Speaker 1: in the wild west of the Internet or on tech
Speaker 1: industry hiring practices learned serious biases. But what about more
Speaker 1: sober settings like a hospital. I talked with someone recently
Speaker 1: discovered similar problems with potentially life threatening consequences. Hi am
Speaker 1: Christine Vogeli. I'm the director of evaluation research at Partner's
Speaker 1: Healthcare here in Boston. Partner's Healthcare, recently rebranded as mass
Speaker 1: General Brigham, is the largest healthcare provider in Massachusetts, a
Speaker 1: system that has six thousand doctors and a dozen hospitals
Speaker 1: and serves more than a million patients. As Christine explained
Speaker 1: to me, the role of healthcare providers in the US
Speaker 1: has been shifting. The responsibility for controlling costs and ensuring
Speaker 1: high quality services is now being put down on the
Speaker 1: hospitals and the doctors. And to me, this makes a
Speaker 1: lot of sense, Right, we really should be the ones
Speaker 1: responsible for ensuring that there's good quality care and that
Speaker 1: we're doing it efficiently. Healthcare providers are especially focusing their
Speaker 1: attention on what they call high risk patients. Really, what
Speaker 1: it means is that they have both multiple chronic illnesses
Speaker 1: and relatively acute chronic illnesses. So give me a set
Speaker 1: of conditions that a patient might have, right, So somebody,
Speaker 1: for example, with cardiovascular disease co occurring with diabetes, and
Speaker 1: you know, maybe they also have depression. They're just kind
Speaker 1: of suffering and trying to get used to having that
Speaker 1: complex illness and how to manage it. Partners Healthcare offers
Speaker 1: a program to help these complex patients. We have a
Speaker 1: nurse or social worker who works as a care manager
Speaker 1: who help everything from education to care coordination services. But
Speaker 1: really that care manager works essentially as a quarterback, arranges
Speaker 1: everything but also provides hands on care to the patient
Speaker 1: and the caregiver. Yeah, I think it's a wonder how
Speaker 1: we expect patients to go figure out all the things
Speaker 1: they're supposed to be doing and how to interact with
Speaker 1: the medical system without a quarterback. It's incredibly complex. These
Speaker 1: patients have multiple specialists who are interacting with the primary
Speaker 1: care physician. They need somebody to be able to tie
Speaker 1: it together and be able to create a care plan
Speaker 1: for them that they can follow, and it pulls everything
Speaker 1: together from all those specialists. Partners Healthcare found that providing
Speaker 1: complex patients with quarterbacks both saved money and improved patient's health.
Speaker 1: For example, they had fewer emergency visits each year, so
Speaker 1: Partners developed a program to identify the top three percent
Speaker 1: of patients with the greatest need for the service. Most
Speaker 1: were recommended by their physicians, but they also used a
Speaker 1: predictive algorithm provided by a major health insurance company that
Speaker 1: assigns each patient a risk score. What does the algorithm do?
Speaker 1: When you look at the web page, it really describes
Speaker 1: itself as a tool to help identify high risk patients.
Speaker 1: And that term is really interesting term to me. What
Speaker 1: makes a patient high risk? So I think from an
Speaker 1: insurance perspective, risk means these patients are going to be
Speaker 1: expensive from a healthcare organization perspective, these are patients who
Speaker 1: we think we could help, and that's the fundamental challenge
Speaker 1: on this one. When the team began to look closely
Speaker 1: at the results, they noticed that people recommended by the
Speaker 1: algorithm were strikingly different than those recommended by their doctor.
Speaker 1: We noticed that black patients overall were underrepresented patients with
Speaker 1: similar numbers of chronic illnesses. If they were black, they
Speaker 1: had a lower riskcore than if they were white, and
Speaker 1: that didn't make sense. Just black patients identified by the
Speaker 1: algorithm turned out to have twenty six percent more chronic
Speaker 1: illnesses than white patients with the same risk scores. So
Speaker 1: what was wrong with the algorithm? It was because given
Speaker 1: a certain level of illness, black and minority patients tend
Speaker 1: to use fewer healthcare services, and whites tend to use
Speaker 1: more even if they have the same level of chronic
Speaker 1: even if they have the same level of chronic conditions.
Speaker 1: That's right, So in some sense, the algorithm is correctly
Speaker 1: predicting the cost associated with the patient, but not the
Speaker 1: need exactly. It predicts costs very well, but we're interested
Speaker 1: in understanding patients who are sick and have needs It's
Speaker 1: important to say that the algorithm only used information about
Speaker 1: insurance claims and medical costs. It didn't use any information
Speaker 1: about a patient's race. But of course these factors are
Speaker 1: correlated with race due to longstanding issues in American society. Frankly,
Speaker 1: we have fewer minority physicians and we do white physicians.
Speaker 1: So the level of trust minorities with the healthcare system,
Speaker 1: we've observed it's lower. And we also know that there
Speaker 1: are just systematic barriers to care that certain groups of
Speaker 1: patients experience more so. For example, race and poverty go
Speaker 1: together and job flexibility. So all these issues with scheduling,
Speaker 1: being able to come in, being able to access services
Speaker 1: are just heightened for minority populations relative to white populations.
Speaker 1: So someone who just has less economic resources might not
Speaker 1: be able to get off work, might not be able
Speaker 1: to get off work, might not have the flexibility with
Speaker 1: childcare to be able to come in for a visit
Speaker 1: when they need to. Exactly, so it means that if
Speaker 1: one only relied on the algorithm, you wouldn't be targeting
Speaker 1: the right people. Yes, we would be targeting more advantaged
Speaker 1: patients who tend to use a lot of healthcare services
Speaker 1: when they corrected the problem, the proportion black patients in
Speaker 1: the high risk group jumped from eighteen percent to forty
Speaker 1: seven percent. Christine, together with colleagues from several other institutions,
Speaker 1: wrote up a paper describing their findings. It was published
Speaker 1: in Science, the nation's leading research journal, in twenty nineteen.
Speaker 1: It made a big splash, not least because many other
Speaker 1: hospital systems we're using the algorithm and others like it.
Speaker 1: We've since changed the algorithm that we use to one
Speaker 1: that uses exclusively information about chronic illness and not healthcare utilization,
Speaker 1: and has that worked. We're still testing. We think it's
Speaker 1: going to work, but as in all of these things,
Speaker 1: you really need to test it. You need to understand
Speaker 1: and see if there's actually any biases. In the end,
Speaker 1: you can't just adopt an algorithm. It's very important to
Speaker 1: be very conscious about what you're predicting. It's also very
Speaker 1: important to think about what are the factors you're putting
Speaker 1: into that prediction algorithm. Even if you believe the ingredients
Speaker 1: so right, you do actually have to see how it
Speaker 1: works in practice. Anything that has to do with people's lives,
Speaker 1: you know, you have to be transparent about it. Chapter
Speaker 1: five Compass Transparency. Christine Vogeli and her colleagues were able
Speaker 1: to get to the bottom of the issue with the
Speaker 1: medical risk prediction because they had ready access to the
Speaker 1: partners healthcare data and could test the algorithm. Unfortunately, that's
Speaker 1: not always the case. I traveled to New York to
Speaker 1: speak with a person who's arguably done more than anyone
Speaker 1: to focus attention on the consequences of algorithmic bias. My
Speaker 1: name is Julia Anguin. I'm a journalist. I've been writing
Speaker 1: about technology for twenty five years, mostly at the Wealthy
Speaker 1: Journal and Pro Publicat. Julia grew up in Silicon Valley
Speaker 1: as the child of a mathematician and a chemist. She
Speaker 1: studied math at the University of Chicago, but decided to
Speaker 1: pursue a career your in journalism. Her quantitative skills gave
Speaker 1: her a unique lens to report on the societal implications
Speaker 1: of technology, and she eventually became interested in investigating high
Speaker 1: stakes algorithms. When I learned that there was actually an
Speaker 1: algorithm that judges used to help decide what to sentence people,
Speaker 1: I was stunned. I thought, this is shocking. I can't
Speaker 1: believe this exists, and I'm going to investigate it. What
Speaker 1: we're talking about is a score that is assigned to
Speaker 1: criminal defendants in many jurisdictions in this country that aims
Speaker 1: to predict whether they will go on to commit a
Speaker 1: future crime. It's known as a risk assessment score, and
Speaker 1: the one that we chose to look at was called
Speaker 1: the Compass Risk Assessment Score. Based on the answers to
Speaker 1: a long list of questions, Compass gives defendants a risk
Speaker 1: score from one to ten. In some jurisdictions, judges use
Speaker 1: the Compass score to decide whether defendants should be released
Speaker 1: on bail before trial. In others, judges use it to
Speaker 1: decide the length of sentence to impose undefendants who plead
Speaker 1: guilty or who were convicted a trial. Julia had a
Speaker 1: suspicion that the algorithm might reflect bias against black defendants.
Speaker 1: Attorney General Eric Holder had actually given a big speech
Speaker 1: saying he was concerned about the use of these growers
Speaker 1: and whether they were exacerbating racial bias, and so that
Speaker 1: was one of the reasons we wanted to investigate. But
Speaker 1: investigating wasn't easy. Unlike Christine Vogeli had partner's healthcare. Julia
Speaker 1: couldn't inspect the Compass algorithm itself. Now, Compass isn't a
Speaker 1: modern neural network who was developed by a company that's
Speaker 1: now called Equivand and it's a much simpler algorithm. It's
Speaker 1: basically a linear equation that should be easy to understand.
Speaker 1: But it's a black box of a different sort. The
Speaker 1: algorithm is opaque because to date Fant has insisted on
Speaker 1: keeping it a trade secret. Julia also had no way
Speaker 1: to download defendants Compass scores from her website, so she
Speaker 1: had to gather the data herself. Her team decided to
Speaker 1: focus on Broward County, Florida. Florida has great public records laws,
Speaker 1: and so we filed a public records request and we
Speaker 1: did end up getting eighteen thousand scores. We got scores
Speaker 1: for everyone who was arrested for a two year period.
Speaker 1: Eighteen thousand scores. All right, So then what did you
Speaker 1: do to evaluate these scores? Well, first thing we did
Speaker 1: when we got the eighteen thousand scores was actually we
Speaker 1: just threw them into a bar chart black and white defendants.
Speaker 1: We immediately noticed there was really different looking distributions for
Speaker 1: black defendants. The scores were evenly distributed, meaning one through
Speaker 1: ten lowest risk to highest Chris. There's equal numbers of
Speaker 1: black defendants in every one of those buckets. For white defendants,
Speaker 1: the scores were heavily clustered in the low risk range.
Speaker 1: And so we thought, there's two options. All the white
Speaker 1: people getting scored in Broward County are legitimately really low risk.
Speaker 1: They're all Mother Teresa, or there's something weird going on.
Speaker 1: Julia sworted the defendants and to those who were rearrested
Speaker 1: over the next two years and those who weren't. She
Speaker 1: compared the compass scores that had been assigned to each group.
Speaker 1: For black defendants, it was much more likely to incorrectly
Speaker 1: predict that they were going to go on to commit
Speaker 1: a future crime when they didn't, and for white defendants,
Speaker 1: it was much more likely to predict that they were
Speaker 1: going to go on to not commit a future crime
Speaker 1: when they did. They were twice as many false positives
Speaker 1: for black defendants as white and twice as many false
Speaker 1: negatives for white defendants as black defendants. Julia described the
Speaker 1: story of two people whose arrest histories illustrate this difference.
Speaker 1: A young eighteen year old black girl named Brecia Borden,
Speaker 1: who had been arrested after picking up kid's bicycle from
Speaker 1: their front yard. Riding at a few blocks. The mom
Speaker 1: came out yelled at her, so that's my kid's bike.
Speaker 1: She gave it back, but actually by then the neighbor
Speaker 1: had called the police, and so she was arrested for that.
Speaker 1: And we compared her with a white man who had
Speaker 1: stolen about eighty dollars worth of stuff from a drug
Speaker 1: store Vernon Prater. When teenager Brecia Borden got booked into jail,
Speaker 1: she got a high compass score and eight, predicting a
Speaker 1: high risk that she'd get re arrested, And Vernon Prader,
Speaker 1: he got a low score a three. Now he had
Speaker 1: already committed two armed robberies and had served time. She
Speaker 1: was eighteen. She given back the bike, and of course
Speaker 1: these scores turned out to be completely wrong. She did
Speaker 1: not go on to commit a future crime in the
Speaker 1: next two years, and he actually went on to break
Speaker 1: into a warehouse steal thousands of dollars of electronics and
Speaker 1: he's serving a ten year ten And so that's what
Speaker 1: the difference between a false positive and a false negative
Speaker 1: looks like. It looks like Fresha Borden and Vernon Prater
Speaker 1: Chapter six Criminal Attitudes. Julia Anguin and her team spent
Speaker 1: over a year doing research. In May twenty sixteen, Pro
Speaker 1: Publica published their article headlined machine Bias. The subtitle quote
Speaker 1: their software used across the country to predict future criminals,
Speaker 1: and it's biased against blacks. Julia's team released all the
Speaker 1: data they had collected so that anyone could check or
Speaker 1: dispute their conclusions. What happened next was truly remarkable. The
Speaker 1: Pro Publica article provoked an outcry for some statisticians, who
Speaker 1: argued that the data actually proved moved Compass wasn't biased.
Speaker 1: How could they reach the opposite conclusion. It turned out
Speaker 1: the answer depended on how you define bias. Pro Publica
Speaker 1: had to analyze the Compass scores by looking backward after
Speaker 1: the outcomes were known among people who are not re arrested.
Speaker 1: They found that black people had been assigned much higher
Speaker 1: risk scores than white people. That seemed pretty unfair, but
Speaker 1: statisticians use the word bias to describe how a predictor
Speaker 1: performs when looking forward before the outcomes happened. It turns
Speaker 1: out that black people and white people who received the
Speaker 1: same risk score had roughly the same chance of being rearrested.
Speaker 1: That seems pretty fair, So whether Compass was fair or
Speaker 1: unfair depended on your definition of fairness. This sparked an
Speaker 1: explosion of academic research. Matt Maticians showed there's no way
Speaker 1: out of the problem. They proved a theorem saying it's
Speaker 1: impossible to build a risk predictor that's fair when looking
Speaker 1: both backward and forward unless the arrest rates for black
Speaker 1: people and white people are identical, which they aren't. The
Speaker 1: pro public article also focused at tension on many other
Speaker 1: ways in which COMPASS scores are biased, like the healthcare
Speaker 1: algorithm that Christine Vogeli studied. Compass scores don't explicitly ask
Speaker 1: about a person's race, but race is closely correlated with
Speaker 1: both the training data and the inputs to the algorithm. First,
Speaker 1: the training data, COMPASS isn't actually trained to predict the
Speaker 1: probability that a person will commit another crime. Instead, it's
Speaker 1: trained to predict whether a person will be arrested for
Speaker 1: committing another crime. The problem is there's abundant evidence that
Speaker 1: in situations where black people and white people commit crimes
Speaker 1: at the same rate, for example, illegal drug use, black
Speaker 1: people are much more likely to get arrested, so Compass
Speaker 1: is being trained on an unfair outcome. Second, the questionnaire
Speaker 1: used to calculate Compass scores is pretty revealing. Some sections
Speaker 1: assess peers, work, and social environment. The questions include how
Speaker 1: many of your friends and acquaintances have ever been arrested?
Speaker 1: How many have been crime victims? How often do you
Speaker 1: have trouble paying bills. Other sections are titled criminal personality
Speaker 1: and criminal attitude. They ask people to agree or disagree
Speaker 1: with such statements as the law doesn't help average people,
Speaker 1: or many people get into trouble because society has given
Speaker 1: them no education, jobs, or future. In a nutshell, the
Speaker 1: predictor penalizes defendants who are honest enough to admit they
Speaker 1: live in high crime neighborhoods or they don't fully trust
Speaker 1: the system. From the questionnaire, it's not hard to guess
Speaker 1: how a teenage black girl arrested for something so minor
Speaker 1: is writing someone else's bicycle a few blocks and returning
Speaker 1: it might have received a COMPASS score of eight. And
Speaker 1: it's not hard to imagine why racially correlated questions would
Speaker 1: do a good job of predicting racially correlated arrest rates.
Speaker 1: Pro PUBLICA didn't win a Pulitzer Prize for its article,
Speaker 1: but it was a remarkable public service Chapter seven Minority report.
Speaker 1: Putting aside the details of Compass, I wanted to find
Speaker 1: out more about the role of predictive algorithms in courts.
Speaker 1: I reached out to one of the leading legal scholars
Speaker 1: in the country. I'm Martha Minnow. I'm a law professor
Speaker 1: at Harvard, and I have recently immersed myself in issues
Speaker 1: of algorithmic fairness. Martha Minnow has a remarkable resume. From
Speaker 1: two thousand and nine to twenty seventeen, she served as
Speaker 1: dean of the Harvard Law School, following now Supreme Court
Speaker 1: Justice Elaina Kagan. Martha also served on the board of
Speaker 1: the government sponsored Legal Services Corporation, which provides legal assistance
Speaker 1: to low income Americans. She was appointed by her former
Speaker 1: law student, President Barack Obama. It became very interested in
Speaker 1: and concerned about the increasing use of algorithms in worlds
Speaker 1: that touch on my preoccupations with equal protection, do process,
Speaker 1: constitutional rights, fairness, anti discrimination. Martha recently co signed a
Speaker 1: statement with twenty six other lawyers and scientists raising quote
Speaker 1: grave concerns about the use of predictive algorithms for pre
Speaker 1: trial risk assessment. I asked her how courts had gotten
Speaker 1: involved in the business of prediction. Criminal's justice system has
Speaker 1: flirted with the use of prediction forever, including discussions from
Speaker 1: the nineteenth century on in this country about dangerousness and
Speaker 1: whether people should be detained prevactively. So far, that's not
Speaker 1: permitted in the United States. It appears in Minority Report
Speaker 1: and other interesting movies. The movie starring Tom Cruise tells
Speaker 1: the story of a future in which the PreCrime division
Speaker 1: of the police arrest people for crimes they haven't yet committed.
Speaker 1: I'm placing you under arrest for the future, murder, Sarah marks.
Speaker 1: We are arresting individuals who've broken no law, but they will.
Speaker 1: The use of prediction in the context of sentencing is
Speaker 1: part of this rather large sphere of discretion that judges
Speaker 1: have to decide what kind of sentence fits the crime
Speaker 1: you're saying. In sentencing, one is allowed to use essentially
Speaker 1: information from the pre crime division about crimes that haven't
Speaker 1: been committed yet. Well, I am horrified by that suggestion,
Speaker 1: but I think it's fair to raise it as a concern.
Speaker 1: The problem is if we actually acknowledge purposes of the
Speaker 1: criminal justice system, some of them start to get into
Speaker 1: the future. So if one purpose is simply incapacitation, prevent
Speaker 1: this person from walking the streets because they might hurt
Speaker 1: someone else, there's a prediction built in. So judges have
Speaker 1: been factoring in predictions about a defendant's future behavior for
Speaker 1: a long time. And judges certainly aren't perfect. They can
Speaker 1: be biased or sometimes just cranky. There are even studies
Speaker 1: showing the judges hand down harsher sentences before lunch breaks
Speaker 1: than after. Now, the defenders of risk Prediction score will say, well,
Speaker 1: it's always not what's the ideal but compared to what
Speaker 1: and if the alternative is we're relying entirely on the
Speaker 1: individual judges and their prejudices, their lack of education, what
Speaker 1: they had for lunch. Isn't this better that it will
Speaker 1: provide some kind of scaffold for more consistency. Journalist Julia
Speaker 1: Anguin has heard the same arguments some good friends right
Speaker 1: who really believe in the use of these criminal risks
Speaker 1: score algorithms. I've said to me, look, Julia, the fact
Speaker 1: is judges are terribly biased, and this is an improvement,
Speaker 1: and my feeling is That's probably true for some judges
Speaker 1: and maybe less true for other judges. But I don't
Speaker 1: think it is a reason to automate bias, right, Like
Speaker 1: I don't understand why you say, Okay, humans are flawed,
Speaker 1: so why don't we make a flawed algorithm and bake
Speaker 1: it into every decision, because then it's really intractable. Martha
Speaker 1: also worries that numerical risk scores are misleading. The judges
Speaker 1: think high numbers mean people are very likely to commit
Speaker 1: violent crime. In fact, the actual probability of violence is
Speaker 1: very low, about eight percent according to a public assessment,
Speaker 1: And she thinks numerical scores can lull judges into a
Speaker 1: false sense of certainty. There's an appearance of objectivity because
Speaker 1: it's math, but is it really Then for lawyers, they
Speaker 1: may have had no math, no numeracy education since high school.
Speaker 1: Many people go to a law in part because they
Speaker 1: don't want to do anything with numbers. And there is
Speaker 1: a larger problem, which is the deference to expertise, particularly
Speaker 1: scientific expertise. Finally, I wanted to ask Martha if defendants
Speaker 1: have a constitutional right to know what's inside the black
Speaker 1: box that's helping to term in their fate. I confess
Speaker 1: I thought the answer was an obvious yes until I
Speaker 1: read a twenty sixteen decision by Wisconsin's Supreme Court. The
Speaker 1: defendant in that case, Eric Loomis, pled guilty to operating
Speaker 1: a car without the owner's permission and fleeing a traffic officer.
Speaker 1: When Loomis was sentenced, the presentencing report given to the
Speaker 1: judge included a Compass score that predicted Loomis had a
Speaker 1: high risk for committing future crimes. He was sentenced to
Speaker 1: six years in prison. Loomis appealed, arguing that his inability
Speaker 1: to inspect the Compass algorithm violated his constitutional right to
Speaker 1: due process. Wisconsin's Supreme Court ultimately decided that Loomis had
Speaker 1: no right to know how Compass worked. Why. First, the
Speaker 1: Wisconsin court said the score was just one of several
Speaker 1: inputs to the judge's sentencing decision. Second, the court said
Speaker 1: even if Lomas didn't know how the score was determined,
Speaker 1: he could still dispute its accuracy. Lomas appealed to the
Speaker 1: US Supreme Court, but it declined to hear the case.
Speaker 1: I find that troubling and not persuasive. It was up
Speaker 1: to you, how would you change the law. I actually
Speaker 1: would require transparency for any use of any algorithm by
Speaker 1: a government agency or court that has the consequence of
Speaker 1: influencing not just deciding, but influencing decisions about individual's rights.
Speaker 1: And those rights could be rights to liberty, property opportunities.
Speaker 1: So transparency, transparency, and am be able to see what
Speaker 1: this algorithm does, absolutely and have the code and be
Speaker 1: able to give it to your own lawyer and your
Speaker 1: own experts. But should a state be able to buy
Speaker 1: a computer program that's proprietary. I mean it would say, well,
Speaker 1: I'd love to give it to you, but it's proprietary.
Speaker 1: I can't. Should that be okay? I think not, because
Speaker 1: if that then limits the transparency, that seems a breach.
Speaker 1: But you know, this is a major problem, the outsourcing
Speaker 1: of government activity that has the effect of bypassing restrictions.
Speaker 1: Take another example, when the US government hires private contractors
Speaker 1: to engage in war activities, they are not governed by
Speaker 1: the same rules that govern the US military. She's saying
Speaker 1: that government can get around constitutional limitations on the government
Speaker 1: by just outsourcing it to somebody who's not the government.
Speaker 1: It's currently the case, and I think that's wrong for
Speaker 1: her part, journalist Julia Angwin is baffled by the Wisconsin
Speaker 1: Court's ruling. I mean, we have this idea that you
Speaker 1: should be able to argue against whatever accusations are made.
Speaker 1: But I don't know how you make an argument against
Speaker 1: a score, like the score says you're seven, but you
Speaker 1: think you're a four. How do you make that argument
Speaker 1: If you don't know how that seven was calculated? You
Speaker 1: can't make an argument that you're a four Chapter eight
Speaker 1: robo recruiter. Even if you never find yourself in a
Speaker 1: criminal court filling out a compass questionnaire, that doesn't mean
Speaker 1: you won't be judged by a predictive algorithm. There's actually
Speaker 1: a good chance it will happen the next time you
Speaker 1: go looking for a job. I spoke to a scientist
Speaker 1: at a high tech company that screens job applicants. My
Speaker 1: name is Lindsay Zulaga, and I'm actually educated as a physicist,
Speaker 1: but now working for a company called higher View. Higher
Speaker 1: View is a video interviewing platform. Companies create an interview,
Speaker 1: candidates can take it at any time that's convenient for them,
Speaker 1: So they go through the questions and they record themselves answer.
Speaker 1: So it's really a great substitute for kind of the
Speaker 1: resume phone screening part of the process. When a candidate
Speaker 1: takes a video interview, they're creating thousands of unique points
Speaker 1: of data. A candidate's verbal and nonverbal cues give us
Speaker 1: insight into their emotional, engagement, thinking, and problem solving style.
Speaker 1: This combination of cutting edge AI and validated science is
Speaker 1: the perfect partner for making data driven talent decisions. Higher View.
Speaker 1: You know, we'll have a customer and they are hiring
Speaker 1: for something like a call center, say it's sales calls.
Speaker 1: And what we do is we look at past employees
Speaker 1: that applied, and we look at their video interviews. We
Speaker 1: look at the words they said, tone of voice, pauses,
Speaker 1: and facial expressions, things like that, and we look for
Speaker 1: patterns in how those people with good sales numbers behave
Speaker 1: as compared to people as low sales numbers. And then
Speaker 1: we have this algorithm that scores new candidates as they
Speaker 1: come in, and so we help kind of get those
Speaker 1: more promising candidates to the top of the pile so
Speaker 1: they're seen more quickly. So Higher View trains a predictive
Speaker 1: algorithm on video interviews of past applicants who turned out
Speaker 1: to be successful employees. But how does higher View know
Speaker 1: its program isn't learning sexism or racism or other similar biases.
Speaker 1: There are lots of reasons to worry. For example, studies
Speaker 1: from M I. T have shown that facial recognition algorithms
Speaker 1: can have a hard time reading emotions from black people's faces.
Speaker 1: And how would Higher Views program evaluate videos from people
Speaker 1: who might look or sound different than the average employee, say,
Speaker 1: people who don't speak English as a native language, who
Speaker 1: are disabled, who are on the autism spectrum, or even
Speaker 1: people who are just a little quirky. Well, Lindsay says
Speaker 1: Higher View tests for certain kinds of bias, So we
Speaker 1: audit the algorithm after the fact and see if it's
Speaker 1: scoring different groups differently in terms of age, race, and gender.
Speaker 1: So if we do see that happening a lot of times,
Speaker 1: that's probably coming from the training data. So maybe there
Speaker 1: is only one female software engineer in this data set,
Speaker 1: the model might mimic that bias. If we do see
Speaker 1: any of that adverse impact, we simply remove the features
Speaker 1: that are causing it, so we can say this model
Speaker 1: is being sexist. How does the model even know what
Speaker 1: gender the person is? So we look at all the features,
Speaker 1: and we find the features that are the most correlated
Speaker 1: to gender. If there are, we simply remove some of
Speaker 1: those features. I asked lindsay why people should believe higher
Speaker 1: views or any company's assurances, or whether something more was needed.
Speaker 1: You seem thoughtful about this, but there will be many
Speaker 1: people coming into the industry over time might not be
Speaker 1: as thoughtful or as sophisticated as you are. Do you
Speaker 1: think it would be a good idea to have third
Speaker 1: parties come in to certify the audits for bias? I
Speaker 1: know that's a hard question, I guess I I kind
Speaker 1: of lean towards no. So you're talking about having a
Speaker 1: third party entity that comes in an assess and certifies
Speaker 1: the audit. You know, because you've described what I think
Speaker 1: is a really impressive process. But of course, how do
Speaker 1: we know it's true? You know, you could reveal all
Speaker 1: your algorithms, but probably not the thing you want to do,
Speaker 1: And so the next best thing is a certifier says yes,
Speaker 1: this audit has been done. Probably you know your financials
Speaker 1: presumably get audited. Why not the result of the algorithm?
Speaker 1: I guess a little bit. The reason I'm not sure
Speaker 1: about the certification is just. It is mostly just because
Speaker 1: I feel like I don't know how it would work exactly,
Speaker 1: Like you're right totally that finances are audited. I haven't
Speaker 1: thought about it enough to have like a strong opinion
Speaker 1: that it should happen, because it's like, Okay, we have
Speaker 1: all these different models, it's constantly changing. How to do
Speaker 1: they audit every single model all the time. I was
Speaker 1: impressed with Lindsay's willingness as a scientist to think in
Speaker 1: real time about a hard question, and it turns out
Speaker 1: she kept thinking about it afterwards. A few months later,
Speaker 1: she wrote back to me to say that she changed
Speaker 1: her mind. We do have a lot of private information,
Speaker 1: but if we don't share it, people tend to assume
Speaker 1: the worst. So I've decided, after thinking about it quite
Speaker 1: a bit, that I definitely support the third party auditing
Speaker 1: of algorithms. Sometimes people you assume we're doing horrible, horrible things,
Speaker 1: and that can be frustrating. But I do think the
Speaker 1: more transparent we can be about what we are doing
Speaker 1: is important. Several months later, Lindsay emailed again to say
Speaker 1: that Higher View was now undergoing a third party audit.
Speaker 1: She says she's excited to learn from the results, Chapter nine,
Speaker 1: confronting the black box so higher view at first, Reluctant
Speaker 1: says it's now engaging external auditors. What about Equivant, whose
Speaker 1: Compass scores can heavily influence prison sentences, but which is
Speaker 1: steadfastly refused to let anyone even see how their simple
Speaker 1: algorithm works. Well. Just before we release this podcast, I
Speaker 1: checked back with them. A company spokesperson wrote that Equivant
Speaker 1: now agrees that the Compass scoring process quote should be
Speaker 1: made available for third party examination, but they weren't releasing
Speaker 1: it yet because they first wanted to file for copyright
Speaker 1: protection on their simple algorithm. So we're still waiting. You
Speaker 1: might ask, should it be up to the companies to decide?
Speaker 1: Aren't there laws or regulations? The answer is there's not much.
Speaker 1: Governments are just now waking up to the idea that
Speaker 1: they have a role to play. I traveled back to
Speaker 1: New York City to talk to someone who's been involved
Speaker 1: in this question. My name's Rashida Richardson, and I'm a
Speaker 1: civil rights lawyer that focuses on the social implications of
Speaker 1: artificial intelligence. Rashida served as the director of policy research
Speaker 1: at a i Now Institute at NYU, where she worked
Speaker 1: with Kate Crawford, the Australian expert and algorithmic bias that
Speaker 1: I spoke to earlier in the episode. In twenty eighteen,
Speaker 1: New York City became the first jurisdiction in the US
Speaker 1: to create a task force to come up with recommendations
Speaker 1: about government use of predictive algorithms, or, as they call them,
Speaker 1: automated decision systems. Unfortunately, the task force bogged down in
Speaker 1: details and wasn't very productive. In response, Rashida led a
Speaker 1: group of twenty seven experts that wrote a fifty six
Speaker 1: page shadow report entitled Confronting Black Boxes that offered concrete proposals.
Speaker 1: New York City it turns out, uses quite a few
Speaker 1: algorithms to make major decisions. You have the school matching algorithms.
Speaker 1: You have an algorithm used by the Child Welfare agency here.
Speaker 1: You have public benefits algorithms that are used to determine
Speaker 1: who will qualify or have their public benefits, whether that's
Speaker 1: Medicaid or temporary food assistance terminated, or whether they'll receive
Speaker 1: access to those benefits. You have a gang database which
Speaker 1: tries to identify who is likely to be in a gang,
Speaker 1: and that's both used by the DA's office and the
Speaker 1: police department. If you had to make a yes, how
Speaker 1: many predictive algorithms are used by the City of New York,
Speaker 1: I'd say upwards to thirty and I'm underestimating with that number.
Speaker 1: How many of these thirty plus algorithms are transparent about
Speaker 1: how they work, about their code. None. So what should
Speaker 1: New York do? It was up to you what should
Speaker 1: be the behavior of a responsible city with respect to
Speaker 1: the algorithms it uses. I think the first step is
Speaker 1: creating greater transparency, some annual acknowledgement of what is being used,
Speaker 1: how it's being used, whether it's been tested or had
Speaker 1: a validation study. And then you would also want general
Speaker 1: information about the inputs or factors that are used by
Speaker 1: these systems to make predictions, because in some cases you
Speaker 1: have factors that are just discriminatory or proxies for protected
Speaker 1: status is like race, gender, ability status. All right, So
Speaker 1: step one, disclose what systems you're using. Yes, And then
Speaker 1: the second step, I think is creating a system of audits,
Speaker 1: both prior to procurement and then once procured, ongoing auditing
Speaker 1: of the system to at least have a gauge on
Speaker 1: what it's doing real time. A lot of the horror
Speaker 1: stories we hear are about fully implemented tools that we're
Speaker 1: in works for years. There's never your pause button to
Speaker 1: reevaluate or look at how a system is working real time.
Speaker 1: And even when I did studies on the use of
Speaker 1: predictive policing systems, I looked at thirteen jurisdictions, only one
Speaker 1: of them actually did a retrospective review of their system.
Speaker 1: So what's your theory about how do you get the
Speaker 1: auditing done? If you are going to outsource to third parties,
Speaker 1: I think it's going to have to be some approval
Speaker 1: process to assess their level of independence, but also any
Speaker 1: conflict of interest she use that may come up, and
Speaker 1: then also doing some thinking about what types of expertise
Speaker 1: are needed, because I think if you don't necessarily have
Speaker 1: someone who understands that social context or even the history
Speaker 1: of a certain government sector, then you could have a
Speaker 1: tool that is technically accurate and meets all of the
Speaker 1: technical standards, but is still reproducing harm because it's not
Speaker 1: paying attention to that social context. Should a government be
Speaker 1: permitted to purchase an automated decision system where the code
Speaker 1: can't be disclosed by contract now, and in fact, there's
Speaker 1: movement around creating more provisions that vendors must waive trade
Speaker 1: secrecy claims once they enter a contract with the government.
Speaker 1: Rashida says, we need laws to regulate the use of
Speaker 1: predictive algorithms, both by governments and by private companies like
Speaker 1: higher View. We're beginning to see bills being explored in
Speaker 1: different states. Massachusetts, Vermont, and Washington DC are considering setting
Speaker 1: up commissions to look at the government use of predictive algorithms.
Speaker 1: Idaho recently passed a first in the nation law requiring
Speaker 1: that pre trial risk algorithms be free of bias and transparent.
Speaker 1: It blocks manufacturers of tools like Compass from claiming trade
Speaker 1: secret protection. And at the national level, a bill was
Speaker 1: recently introduced in the US Congress, the Algorithmic Accountability Act.
Speaker 1: The bill would require that private companies ensure certain types
Speaker 1: of algorithms are audited for bias. Unfortunately, it doesn't require
Speaker 1: that the results of the audit are made public, so
Speaker 1: there's still a long way to go. Rashida thinks it's
Speaker 1: important that regulations don't just focus on technical issues. They
Speaker 1: need to look at the larger context. Part of the
Speaker 1: problems that were identif fine with these systems is that
Speaker 1: they're amplifying and reproducing a lot of the historical and
Speaker 1: current discrimination that we see in society. There are large
Speaker 1: questions we've been unable to answer as a society of
Speaker 1: how do you deal with the compounded effect of fifty
Speaker 1: years of discrimination? And we don't have a simple answer,
Speaker 1: and there's not necessarily going to be a technical solution.
Speaker 1: But I think having access to more data in an
Speaker 1: understanding of how these systems are working will help us
Speaker 1: evaluate whether these tools are even being evalue added and
Speaker 1: addressing the larger social questions. Finally, Kate Crawford says laws
Speaker 1: alone likely won't be enough. There's another thing we need
Speaker 1: to focus on. In the end, it really matters who
Speaker 1: is in the room designing these systems. If you have
Speaker 1: people sitting around a conference table, they all look the same.
Speaker 1: Perhaps they all did the same type of engineering degree.
Speaker 1: Perhaps they're all men. Perhaps they're all pretty middle class
Speaker 1: or pretty well off. They're going to be designing systems
Speaker 1: that reflect their worldview. What we're learning is that the
Speaker 1: more diverse those rooms are, and the more we can
Speaker 1: question those kinds of assumptions, the better we can actually
Speaker 1: design systems for a diverse world. Conclusion, choose your planet.
Speaker 1: So there you have it, Steorides of the Brave New Planet.
Speaker 1: Predictive algorithms, a sixty year old dream of artificial intelligence
Speaker 1: machines making human like decisions has finally become a reality.
Speaker 1: If a task can be turned into a prediction problem,
Speaker 1: and if you've got a mountain of training data, algorithms
Speaker 1: can learn to do the job. Countless applications are possible,
Speaker 1: translating languages instantaneously, providing expert medical diagnoses for eye diseases
Speaker 1: and cancer to patients anywhere, improving drug development, all at
Speaker 1: levels comparable to or better than human experts. But it's
Speaker 1: also letting governments and companies make automatic decisions about you,
Speaker 1: whether you should get admitted to college, be hired for
Speaker 1: a job, get a loan, get housing assistance, be granted bail,
Speaker 1: or get medical attention. The problem is that algorithms that
Speaker 1: learn to make human like decisions based on past human
Speaker 1: outcomes can acquire a lot of human biases about gender, race, class,
Speaker 1: and more often masquerading as objective judgment. Even worse, you
Speaker 1: usually don't have a right even to know you're being
Speaker 1: judged by a machine, or what's inside the black box,
Speaker 1: or whether the algorithms are accurate or fair. Should laws
Speaker 1: require that automated decision systems used by governments or companies
Speaker 1: be transparent? Should they require public auditing for a curacy
Speaker 1: and fairness? And what exactly is fairness? Anyway? Governments are
Speaker 1: just beginning to wake up to these issues, and they're
Speaker 1: not sure what they should do. In the coming years,
Speaker 1: they'll decide what rules to set, or perhaps to do
Speaker 1: nothing at all. So what can you do a lot?
Speaker 1: It turns out you don't have to be an expert
Speaker 1: and you don't have to do it alone. Start by
Speaker 1: learning a bit more. Invite friends over virtually or in
Speaker 1: person when it's saved for dinner and debate about what
Speaker 1: we should do. Or organize a conversation at a book club,
Speaker 1: a faith group, or a campus event. And then email
Speaker 1: your city or state representatives to ask what they're doing
Speaker 1: about the issue, maybe even proposing first steps like setting
Speaker 1: up a task force. When people get engaged, action happens.
Speaker 1: You'll find lots of resources and ideas at our website,
Speaker 1: Brave New Planet dot org. It's time to choose our planet.
Speaker 1: The future is up to us, James. Brave New Planet
Speaker 1: is a co production of the Broad Institute of Mt
Speaker 1: and Harvard Pushkin Industries in the Boston Globe, with support
Speaker 1: from the Alfred P. Sloan Foundation. Our show is produced
Speaker 1: by Rebecca Lee Douglas with Mary Doo theme song composed
Speaker 1: by Ned Porter, mastering and sound designed by James Garver,
Speaker 1: fact checking by Joseph Fridman, and a Stitt and Enchant.
Speaker 1: Special Thanks to Christine Heenan and Rachel Roberts at Clarendon Communications,
Speaker 1: to Lee McGuire, Kristen Zarelli and Justine Levin Allerhand at
Speaker 1: the Broad, to Milobelle and Heather Faine at Pushkin, and
Speaker 1: to Eli and Edy Brode who made Broad Institute possible.
Speaker 1: This is Brave New Planet. I'm Ericlander.