1 00:00:09,760 --> 00:00:13,240 Speaker 1: Hello, and welcome to another edition of the Odd Thoughts Podcast. 2 00:00:13,320 --> 00:00:17,040 Speaker 1: I'm Tracy Allaway and I'm Joe Wis. So, Joe, you 3 00:00:17,079 --> 00:00:21,200 Speaker 1: know what I was thinking? No, I don't actually Okay, 4 00:00:21,360 --> 00:00:23,560 Speaker 1: what were you thinking? That was a pretty open ended 5 00:00:23,640 --> 00:00:26,720 Speaker 1: question in retrospect. That would have been pretty impressive if 6 00:00:26,760 --> 00:00:29,120 Speaker 1: I had got it right. All right, What I was 7 00:00:29,160 --> 00:00:31,920 Speaker 1: thinking is that we have a tendency on this show 8 00:00:32,000 --> 00:00:36,159 Speaker 1: to talk a lot about quantitative trading. We talk a 9 00:00:36,159 --> 00:00:41,360 Speaker 1: lot about high frequency trading, systematic trading strategies, and we 10 00:00:41,400 --> 00:00:45,760 Speaker 1: don't actually talk that much about the things that underpin 11 00:00:46,040 --> 00:00:48,479 Speaker 1: all those strategies. Do you know what I'm talking about. 12 00:00:49,000 --> 00:00:52,279 Speaker 1: I've thought about this before, So we just sort of 13 00:00:52,320 --> 00:00:55,160 Speaker 1: speak in the abstract about what quants are doing, or 14 00:00:55,160 --> 00:00:59,600 Speaker 1: we're like, oh, humans have no chance anymore at beating 15 00:00:59,600 --> 00:01:03,240 Speaker 1: the l agorhythms. But there's sort of in the general discussion. 16 00:01:03,320 --> 00:01:05,160 Speaker 1: And I think we're a little bit better than this year, 17 00:01:05,520 --> 00:01:07,640 Speaker 1: to be honest, But I think of the general discussion 18 00:01:07,680 --> 00:01:10,360 Speaker 1: about all these things we don't really like dive into, 19 00:01:10,440 --> 00:01:12,559 Speaker 1: like you know, the math and how it really works, 20 00:01:12,560 --> 00:01:15,600 Speaker 1: and you know who comes up with all this stuff, right, right? 21 00:01:15,680 --> 00:01:17,640 Speaker 1: So you hit the nail exactly on the head. So 22 00:01:17,720 --> 00:01:20,400 Speaker 1: we don't talk that much about the algorithms, and there 23 00:01:20,520 --> 00:01:24,039 Speaker 1: is this perception out there that the algorithms are the 24 00:01:24,120 --> 00:01:28,720 Speaker 1: maths doing all this stuff, are better than human beings. 25 00:01:28,920 --> 00:01:32,600 Speaker 1: And I think that's a question. You know, I don't 26 00:01:32,600 --> 00:01:34,840 Speaker 1: think we actually have the answer to that. It's just 27 00:01:34,880 --> 00:01:38,319 Speaker 1: an assumption that people have made. And in recent years 28 00:01:38,360 --> 00:01:41,119 Speaker 1: we have seen lots of people start to talk about 29 00:01:41,200 --> 00:01:46,119 Speaker 1: weaknesses in the algorithms that underlie a lot of these strategies. 30 00:01:46,160 --> 00:01:50,840 Speaker 1: And I should just say here, when we talk about algorithms, yes, 31 00:01:51,200 --> 00:01:54,920 Speaker 1: you think trading, you think finance, but algorithms kind of 32 00:01:55,000 --> 00:01:57,240 Speaker 1: rule our lives in a lot of different ways. Now. 33 00:01:57,320 --> 00:02:00,760 Speaker 1: You see a lot of the applications in business, whether 34 00:02:00,800 --> 00:02:06,800 Speaker 1: it's dynamic pricing on Amazon, or online advertising or content curation. 35 00:02:07,520 --> 00:02:10,520 Speaker 1: Think about the Netflix movies that get shown to you 36 00:02:10,639 --> 00:02:15,119 Speaker 1: all the time. Algorithms and the assumptions underpinning them are 37 00:02:15,200 --> 00:02:18,880 Speaker 1: basically everywhere at the moment, and we're not really asking 38 00:02:19,160 --> 00:02:22,000 Speaker 1: enough of the tough questions about them. I think my 39 00:02:22,320 --> 00:02:26,120 Speaker 1: most frequent thinking about algorithms these days is when I 40 00:02:26,200 --> 00:02:28,400 Speaker 1: drive out of the city and I used the Ways 41 00:02:28,520 --> 00:02:31,200 Speaker 1: app and it tells me like different paths to to 42 00:02:31,360 --> 00:02:34,280 Speaker 1: beat traffic, and sometimes it seems like it gets it 43 00:02:34,320 --> 00:02:37,760 Speaker 1: totally wrong. Usually it does an amazing job, and sometimes 44 00:02:37,760 --> 00:02:40,680 Speaker 1: I second guess it and regret it. But sometimes it's like, 45 00:02:40,840 --> 00:02:43,640 Speaker 1: how did I get in this situation? So I have 46 00:02:43,800 --> 00:02:48,320 Speaker 1: firsthand experience with Occasionally they don't work perfectly right, Okay, 47 00:02:48,360 --> 00:02:51,320 Speaker 1: everyone has an algorithm experience, like you watch that one 48 00:02:51,360 --> 00:02:54,680 Speaker 1: bad movie on Netflix and then suddenly for the rest 49 00:02:54,680 --> 00:02:58,960 Speaker 1: of your life you're getting pushed sequels, or in my case, 50 00:02:59,040 --> 00:03:02,640 Speaker 1: because I have a twenty month old daughter, when I 51 00:03:02,720 --> 00:03:04,760 Speaker 1: go on I, it just thinks that all I want 52 00:03:04,880 --> 00:03:07,560 Speaker 1: to watch is children shows. So those are some of 53 00:03:07,560 --> 00:03:10,040 Speaker 1: the downsides of algorithms, but I think we can do 54 00:03:10,120 --> 00:03:12,560 Speaker 1: better than that. We have a really good guest to 55 00:03:12,720 --> 00:03:16,880 Speaker 1: talk about all the assumptions that are actually going into 56 00:03:16,919 --> 00:03:20,760 Speaker 1: these things, and it is someone I've spoken to before, 57 00:03:21,280 --> 00:03:24,600 Speaker 1: back in my previous life over at the Financial Times. 58 00:03:24,880 --> 00:03:29,000 Speaker 1: It's Frank Pascal. He's professor of law at the University 59 00:03:29,080 --> 00:03:32,519 Speaker 1: of Maryland. He's also the author of an entire book 60 00:03:32,600 --> 00:03:45,960 Speaker 1: on this subject called The Black Box Society. Frank, thank 61 00:03:46,000 --> 00:03:49,320 Speaker 1: you so much for coming on. Thank you, Tracy and Joe, 62 00:03:49,320 --> 00:03:53,720 Speaker 1: it's great to be here. So you've clearly identified some 63 00:03:53,800 --> 00:03:56,280 Speaker 1: of the issues with algorithms. I think you give it 64 00:03:56,320 --> 00:03:58,880 Speaker 1: away a little bit in the title of your book, 65 00:03:59,040 --> 00:04:02,560 Speaker 1: The Black Box Society. These are things that are shaping 66 00:04:02,600 --> 00:04:05,800 Speaker 1: our society, but we don't actually know a lot about 67 00:04:05,960 --> 00:04:08,400 Speaker 1: what's going into them. Essentially, there are a black box. 68 00:04:08,720 --> 00:04:12,000 Speaker 1: What piqued your interest in this subject, you know, it's 69 00:04:12,000 --> 00:04:15,400 Speaker 1: a long history. I began back in two thousand and 70 00:04:15,440 --> 00:04:18,119 Speaker 1: six looking at search engines, and at the time I 71 00:04:18,160 --> 00:04:20,360 Speaker 1: was just so enthused about the way in which they 72 00:04:20,360 --> 00:04:22,880 Speaker 1: helped you find things and how much easier they made 73 00:04:22,920 --> 00:04:25,719 Speaker 1: research and finding music and movies and stuff. And my 74 00:04:25,800 --> 00:04:27,640 Speaker 1: first articles on them were all about how you could 75 00:04:27,680 --> 00:04:30,400 Speaker 1: expand fair use doctrine to get people more access to them. 76 00:04:30,880 --> 00:04:33,200 Speaker 1: But then I started looking at the dark side and seeing, 77 00:04:33,279 --> 00:04:35,680 Speaker 1: you know, there are all these disputes about what should 78 00:04:35,680 --> 00:04:38,719 Speaker 1: be highly ranked or not so highly ranked, and people 79 00:04:38,720 --> 00:04:41,279 Speaker 1: that had really embarrassing or untrue stories about them that 80 00:04:41,400 --> 00:04:43,160 Speaker 1: got you know, really high in the rankings, and they 81 00:04:43,160 --> 00:04:45,480 Speaker 1: were trying to fight that, and so then I started 82 00:04:45,560 --> 00:04:48,880 Speaker 1: writing about the search engines. And then then the financial 83 00:04:48,880 --> 00:04:52,200 Speaker 1: crisis happened and I was just fascinated by that, and 84 00:04:52,360 --> 00:04:54,560 Speaker 1: I found that there were a lot of parallels between 85 00:04:54,600 --> 00:04:57,279 Speaker 1: tech and finance and how they used algorithms. I just 86 00:04:57,279 --> 00:04:59,880 Speaker 1: want to I'm already really excited about where this episode 87 00:05:00,120 --> 00:05:03,800 Speaker 1: is going just based on that answer, before we get 88 00:05:03,839 --> 00:05:05,839 Speaker 1: to like the you know, the finance stuff. I'm just 89 00:05:05,880 --> 00:05:08,760 Speaker 1: thinking about. You know, you mentioned the early days of 90 00:05:09,200 --> 00:05:12,600 Speaker 1: search engines and how it seemed like this amazing new 91 00:05:12,680 --> 00:05:15,240 Speaker 1: thing ended up having dark side. It reminds me it 92 00:05:15,279 --> 00:05:18,520 Speaker 1: feels like that debate is really a sort of mirrored 93 00:05:18,520 --> 00:05:21,719 Speaker 1: in the discussion of social media these days, and the 94 00:05:21,720 --> 00:05:26,000 Speaker 1: way algorithms turn up news that reinforce our biases or 95 00:05:26,040 --> 00:05:29,359 Speaker 1: reinforce our bubbles, and so it feels like probably what 96 00:05:29,480 --> 00:05:31,479 Speaker 1: you were looking at back in two thousand and five 97 00:05:31,520 --> 00:05:34,520 Speaker 1: two thousand six must feel very similar when you see 98 00:05:34,520 --> 00:05:37,080 Speaker 1: people talking about the kind of things that Facebook and 99 00:05:37,080 --> 00:05:40,560 Speaker 1: Twitter surface, absolutely, and it does feel like deja vu 100 00:05:40,600 --> 00:05:43,320 Speaker 1: all over. The gain to Congress is credit. I actually 101 00:05:43,360 --> 00:05:45,480 Speaker 1: was called before the House Judiciary Committee in two thousand 102 00:05:45,520 --> 00:05:47,240 Speaker 1: and eight to talk about some of this stuff. They 103 00:05:47,240 --> 00:05:49,960 Speaker 1: didn't do much then, but actually last week I was 104 00:05:50,000 --> 00:05:52,240 Speaker 1: just before another House committee, and it looks like they 105 00:05:52,279 --> 00:05:54,640 Speaker 1: really do get it. I mean there's been a whole 106 00:05:54,800 --> 00:05:57,080 Speaker 1: phase change, I think, and I think it is because 107 00:05:57,120 --> 00:06:00,719 Speaker 1: of exactly what you're describing, Joe, this awareness that, uh, 108 00:06:00,760 --> 00:06:02,960 Speaker 1: we just don't know where a lot of the ads 109 00:06:03,160 --> 00:06:05,359 Speaker 1: or bots or other things that we see on Twitter 110 00:06:05,400 --> 00:06:08,279 Speaker 1: or Facebook, where they're even coming from. So, since we're 111 00:06:08,320 --> 00:06:10,520 Speaker 1: on this topic, and I assume we'll get to some 112 00:06:10,560 --> 00:06:14,479 Speaker 1: of the more finance oriented algorithmic stuff later, but what 113 00:06:14,520 --> 00:06:19,600 Speaker 1: do you think are the most insidious applications of algorithms 114 00:06:19,600 --> 00:06:22,640 Speaker 1: in our sort of day to day, non trading financial 115 00:06:22,680 --> 00:06:25,880 Speaker 1: lives nowadays? The thing that I am most troubled by 116 00:06:26,120 --> 00:06:30,520 Speaker 1: is the fact that you could have stealth health profiles 117 00:06:30,560 --> 00:06:33,200 Speaker 1: of yourself out there. So we now know that there's 118 00:06:33,200 --> 00:06:35,440 Speaker 1: sort of a digital doppel ganga. We all have this 119 00:06:35,560 --> 00:06:39,360 Speaker 1: digital second self out there that is the aggregation of 120 00:06:39,400 --> 00:06:42,760 Speaker 1: all the different profiles that are you know, from data brokers, 121 00:06:43,000 --> 00:06:46,880 Speaker 1: from the giant online companies like Facebook, Google, Twitter, etcetera. 122 00:06:47,080 --> 00:06:49,640 Speaker 1: And what I worry about is that you know, these 123 00:06:49,680 --> 00:06:53,560 Speaker 1: things could be used to manipulate people. So, for example, 124 00:06:53,640 --> 00:06:57,520 Speaker 1: there's concerned about what's called vulnerability based marketing, where people 125 00:06:57,520 --> 00:07:01,440 Speaker 1: are trying to market to the gullible others, and that's 126 00:07:01,480 --> 00:07:05,480 Speaker 1: that's been exposed and by various governmental entities. So those 127 00:07:05,520 --> 00:07:07,200 Speaker 1: I think are really troubling. You know, if they've got 128 00:07:07,240 --> 00:07:09,960 Speaker 1: something that says, oh, this person is really upset, and 129 00:07:10,000 --> 00:07:11,960 Speaker 1: when they're really upset, you know, that's the best time 130 00:07:12,000 --> 00:07:14,560 Speaker 1: to target them for a really expensive purchase or something 131 00:07:15,040 --> 00:07:17,120 Speaker 1: that's worrisome. There are lots of other ones I can 132 00:07:17,160 --> 00:07:19,400 Speaker 1: give that are outside the consumer side. There are more 133 00:07:19,440 --> 00:07:22,080 Speaker 1: on the work side, law enforcement side. But that's the start. 134 00:07:22,120 --> 00:07:24,320 Speaker 1: I think. Keep giving them tell tell us a little 135 00:07:24,320 --> 00:07:27,280 Speaker 1: bit some of the other ones too, sure. So, I mean, 136 00:07:27,280 --> 00:07:30,120 Speaker 1: there are scores that are about whether someone is likely 137 00:07:30,160 --> 00:07:33,080 Speaker 1: to be a fraud or not. And these scores people 138 00:07:33,120 --> 00:07:36,000 Speaker 1: don't even know that they exist. People don't know that 139 00:07:36,040 --> 00:07:38,080 Speaker 1: they exist, or they know that there are credit scores 140 00:07:38,120 --> 00:07:40,200 Speaker 1: out there and that the three major credit bureaus are 141 00:07:40,200 --> 00:07:43,119 Speaker 1: calculating those. But I did this article called the Scored 142 00:07:43,200 --> 00:07:45,679 Speaker 1: Society with Daniel Citron, and we looked at the research 143 00:07:45,800 --> 00:07:48,080 Speaker 1: on all these other scores that are out there, you know, 144 00:07:48,120 --> 00:07:52,480 Speaker 1: about whether people are reliable as employees, their medication adherent score, 145 00:07:52,520 --> 00:07:55,320 Speaker 1: whether they're likely to adhere to a medication regime. You know, 146 00:07:55,400 --> 00:07:57,080 Speaker 1: all these sorts of things that you know, people don't 147 00:07:57,120 --> 00:08:00,240 Speaker 1: know about, and that oftentimes they're not accurate, you know, 148 00:08:00,240 --> 00:08:02,840 Speaker 1: So you have a problem that like one guy complained, 149 00:08:02,840 --> 00:08:04,520 Speaker 1: and I think this was actually in a Bloomberg article, 150 00:08:04,840 --> 00:08:06,880 Speaker 1: he complained that he was on a list of diabetics 151 00:08:07,360 --> 00:08:09,640 Speaker 1: and he's not diabetic. And the irony here is that 152 00:08:09,680 --> 00:08:12,040 Speaker 1: the companies then come back and say, well, it's not 153 00:08:12,200 --> 00:08:13,880 Speaker 1: really a list of the diabetics, it's a list of 154 00:08:13,880 --> 00:08:17,080 Speaker 1: the diabetic concerned, and so you can't prove that we're 155 00:08:17,080 --> 00:08:19,720 Speaker 1: wrong because we think you're diabetic concerned and that's our opinion. 156 00:08:20,400 --> 00:08:22,320 Speaker 1: So so there are there's a lot of problems in 157 00:08:22,400 --> 00:08:24,280 Speaker 1: terms of like there's lists out there, and people are 158 00:08:24,280 --> 00:08:27,360 Speaker 1: not really responsible for the lists, and nobody's really looking 159 00:08:27,400 --> 00:08:30,040 Speaker 1: to make sure that they're accurate, and we don't know 160 00:08:30,160 --> 00:08:32,960 Speaker 1: how far or the applications of them and how they 161 00:08:33,040 --> 00:08:35,680 Speaker 1: might be used to deny opportunity or to otherwise, you know, 162 00:08:35,760 --> 00:08:39,240 Speaker 1: classify people in ways that are negative. Now, you mentioned 163 00:08:39,480 --> 00:08:43,040 Speaker 1: credit scores just then, and in many respects, these were 164 00:08:43,080 --> 00:08:47,080 Speaker 1: sort of the first algorithmic scoring models that we saw 165 00:08:47,120 --> 00:08:51,160 Speaker 1: in the consumer space, and their history kind of parallels 166 00:08:51,200 --> 00:08:54,640 Speaker 1: some of the concerns that we're seeing erupt now over 167 00:08:55,400 --> 00:08:59,880 Speaker 1: various types of new data driven algorithms. Can you maybe 168 00:09:00,000 --> 00:09:03,200 Speaker 1: of us a sort of potted history of credit scores 169 00:09:03,360 --> 00:09:06,160 Speaker 1: and how they reflect some of the conversation that we're 170 00:09:06,160 --> 00:09:08,600 Speaker 1: having now. Yeah, I think credit scores are a great 171 00:09:08,600 --> 00:09:11,680 Speaker 1: place to start, and the irony is this, and we're 172 00:09:11,720 --> 00:09:14,520 Speaker 1: seeing exactly the same rhetoric now. In the sixties and 173 00:09:14,559 --> 00:09:17,680 Speaker 1: early seventies, there was a big concern that the decisions 174 00:09:17,679 --> 00:09:23,360 Speaker 1: made by individual loan officers bank employees were discriminatory, and 175 00:09:23,440 --> 00:09:25,720 Speaker 1: so a lot of people said, well, you can't keep 176 00:09:25,760 --> 00:09:29,400 Speaker 1: doing these discriminatory decisions. You need to have an objective 177 00:09:29,480 --> 00:09:32,199 Speaker 1: metric in terms of how you decide who you give 178 00:09:32,240 --> 00:09:35,440 Speaker 1: credit to, what rate should give it to, etcetera. And 179 00:09:35,480 --> 00:09:38,080 Speaker 1: so that led to more demand for these sort of 180 00:09:38,080 --> 00:09:41,360 Speaker 1: scoring systems that were They were developed in the fifties, 181 00:09:41,400 --> 00:09:43,679 Speaker 1: but you know, they weren't in huge demand, but then 182 00:09:43,679 --> 00:09:45,960 Speaker 1: they overthrew the sixties and seventies they were in more demand. 183 00:09:46,720 --> 00:09:49,200 Speaker 1: And then what ends up happening is that they go 184 00:09:49,480 --> 00:09:53,120 Speaker 1: from being a relatively simple, you know, sort of setup 185 00:09:53,160 --> 00:09:56,440 Speaker 1: criteria to very complex and adding in more and more 186 00:09:56,520 --> 00:09:59,960 Speaker 1: data and ways of transforming the data, and there see 187 00:10:00,040 --> 00:10:02,000 Speaker 1: creative because a lot of companies they don't want to 188 00:10:02,000 --> 00:10:04,400 Speaker 1: try to get a patent on these things because there's 189 00:10:04,400 --> 00:10:06,520 Speaker 1: already prior art out there, so they want to protect 190 00:10:06,520 --> 00:10:10,320 Speaker 1: them legally as trade secrets. And the concern that a 191 00:10:10,320 --> 00:10:13,240 Speaker 1: lot of people have had over the past forty fifty 192 00:10:13,320 --> 00:10:17,719 Speaker 1: years is that the scoring systems are are they incorporating 193 00:10:17,800 --> 00:10:20,800 Speaker 1: data that's accurate? How are they incorporating it? Do they 194 00:10:20,840 --> 00:10:23,000 Speaker 1: have disparate impacts? There are some studies that show they 195 00:10:23,040 --> 00:10:26,520 Speaker 1: have disparate impacts on minority groups and others, And so 196 00:10:26,520 --> 00:10:29,280 Speaker 1: there's been a lot of controversy over these scoring systems 197 00:10:29,679 --> 00:10:32,080 Speaker 1: and that controversy, but but if we look back to 198 00:10:32,120 --> 00:10:35,720 Speaker 1: their beginnings, it's exactly the same case that's now being 199 00:10:35,760 --> 00:10:37,560 Speaker 1: made for AI. There's all these people that say, we 200 00:10:37,600 --> 00:10:41,040 Speaker 1: need like AI driven in algorithm driven police departments because 201 00:10:41,080 --> 00:10:43,760 Speaker 1: the police are is prejudiced, and we need AI driven 202 00:10:43,840 --> 00:10:47,439 Speaker 1: hiring because HR is prejudiced. But what we found in 203 00:10:47,440 --> 00:10:49,920 Speaker 1: the credit score context is that the flight to a 204 00:10:49,960 --> 00:10:54,439 Speaker 1: computerized algorithmic process that has its own biases in it. 205 00:10:54,920 --> 00:10:57,400 Speaker 1: And secondly, that the data in all of these systems 206 00:10:57,679 --> 00:11:01,319 Speaker 1: is still data collected by humans. You don't have robots 207 00:11:01,440 --> 00:11:04,360 Speaker 1: creating the credit data or the crime data or other data. 208 00:11:04,600 --> 00:11:07,560 Speaker 1: And because that data can itself have all sorts of 209 00:11:07,559 --> 00:11:10,800 Speaker 1: biases in it, a lot of times the algorithms are 210 00:11:10,840 --> 00:11:13,000 Speaker 1: not what's really the key actor. What's key is the 211 00:11:13,080 --> 00:11:15,120 Speaker 1: data and it still has the same problems that the 212 00:11:15,120 --> 00:11:18,680 Speaker 1: old system had now. Just to play Devil's advocate for 213 00:11:18,720 --> 00:11:21,080 Speaker 1: a second, as I think a lot of people would 214 00:11:21,120 --> 00:11:24,720 Speaker 1: acknowledge there's noise that could get into the data. It 215 00:11:24,760 --> 00:11:26,760 Speaker 1: has to get humans have to deal with it, so 216 00:11:26,840 --> 00:11:31,040 Speaker 1: that introduces error. The flip side would be, Okay, yes, 217 00:11:31,679 --> 00:11:34,960 Speaker 1: it's kind of messy, but because we can get reasonably 218 00:11:35,040 --> 00:11:39,360 Speaker 1: accurate profiles of would be borrowers, we can lend to 219 00:11:39,480 --> 00:11:42,760 Speaker 1: more people, and we can lend to them at a 220 00:11:42,800 --> 00:11:45,320 Speaker 1: lower rate because we can feel confident that they're not 221 00:11:45,440 --> 00:11:49,200 Speaker 1: one of the fraudsters or they're not inclined to default 222 00:11:49,240 --> 00:11:52,920 Speaker 1: on their debt. And so the devil's argument argument would be, yes, 223 00:11:52,920 --> 00:11:56,040 Speaker 1: you can pinpoint all these problems, but what's unseen is 224 00:11:56,440 --> 00:12:01,120 Speaker 1: greater credit, credit availability and cheaper credit. That's a great point, 225 00:12:01,200 --> 00:12:02,560 Speaker 1: and I mean I think that this is one of 226 00:12:02,559 --> 00:12:04,840 Speaker 1: those areas where we're gonna have to make a lot 227 00:12:04,880 --> 00:12:06,840 Speaker 1: of tough trade offs. And this is something that's happening 228 00:12:06,880 --> 00:12:11,200 Speaker 1: in the general fairness in machine learning community. There's now 229 00:12:11,240 --> 00:12:13,160 Speaker 1: a lot of researchers and computer science and law that 230 00:12:13,200 --> 00:12:15,960 Speaker 1: are working on this sort of issue. And I think 231 00:12:16,000 --> 00:12:17,720 Speaker 1: that what we're gonna have to try to do is, 232 00:12:18,280 --> 00:12:22,280 Speaker 1: you know, maintain some of the efficiencies, but also try 233 00:12:22,360 --> 00:12:24,840 Speaker 1: to get rid of and maybe lose a little bit 234 00:12:24,880 --> 00:12:26,760 Speaker 1: of efficiency at the edges, but get rid of some 235 00:12:26,840 --> 00:12:29,680 Speaker 1: of the discriminatory side effects. The other thing I would 236 00:12:29,679 --> 00:12:31,840 Speaker 1: say is about that is that you know, there's a 237 00:12:31,920 --> 00:12:35,880 Speaker 1: really interesting book about the financial crisis which said um 238 00:12:36,280 --> 00:12:40,400 Speaker 1: call to call for judgment um, and the author makes 239 00:12:40,440 --> 00:12:45,400 Speaker 1: the argument that these systems they fooled us into thinking 240 00:12:45,440 --> 00:12:49,719 Speaker 1: that we could calculate risk better than we actually could, right. 241 00:12:50,160 --> 00:12:52,720 Speaker 1: And so the concern there was that if you because 242 00:12:52,720 --> 00:12:54,280 Speaker 1: one of the other things happened with credit scoring is 243 00:12:54,320 --> 00:12:56,679 Speaker 1: they said, well, it's not just a binary you get 244 00:12:56,679 --> 00:12:59,679 Speaker 1: credit or you don't. Someone with a very low score, 245 00:12:59,720 --> 00:13:01,319 Speaker 1: maybe we give them credit, but at a very high 246 00:13:01,360 --> 00:13:03,560 Speaker 1: interest rate. Someone with a high score, we give them 247 00:13:03,640 --> 00:13:06,800 Speaker 1: very low interest rate. And so those sorts of judgments, 248 00:13:07,240 --> 00:13:11,080 Speaker 1: they can be seen one by one as being very efficient, 249 00:13:11,200 --> 00:13:14,679 Speaker 1: but then they can create these larger systemic effects that 250 00:13:14,720 --> 00:13:17,720 Speaker 1: you know, it's hard to really anticipate at the beginning, 251 00:13:18,320 --> 00:13:20,800 Speaker 1: to what extent our credit score is able to be 252 00:13:20,920 --> 00:13:25,239 Speaker 1: manipulated by people who understand the sort of basic factors 253 00:13:25,440 --> 00:13:28,360 Speaker 1: or ingredients that are going into them. Because I remember 254 00:13:28,440 --> 00:13:30,439 Speaker 1: some of the research that came out of the financial 255 00:13:30,480 --> 00:13:35,760 Speaker 1: crisis showed that you had bunchings of credit scores applying 256 00:13:35,880 --> 00:13:39,240 Speaker 1: for mortgages around certain cut off points, which kind of 257 00:13:39,280 --> 00:13:42,760 Speaker 1: suggested that someone was aware of what was going on 258 00:13:42,880 --> 00:13:46,199 Speaker 1: and was really, you know, keeping an eye on the FICO. 259 00:13:46,760 --> 00:13:50,560 Speaker 1: That is such a great question. So I mean, there 260 00:13:50,559 --> 00:13:54,439 Speaker 1: are all of these online forums that argue that you 261 00:13:54,520 --> 00:13:58,760 Speaker 1: can figure out these secret signals, you know, for example, 262 00:13:58,920 --> 00:14:00,920 Speaker 1: like have four credit hard is because if you have 263 00:14:01,360 --> 00:14:03,640 Speaker 1: less than four, you might be seen as having too few, 264 00:14:03,800 --> 00:14:06,080 Speaker 1: and more than four you're seen as having too many. 265 00:14:06,600 --> 00:14:08,920 Speaker 1: Those sorts of little things. And there are people that 266 00:14:09,120 --> 00:14:12,200 Speaker 1: are constantly on these forums saying one thing or the other. 267 00:14:12,920 --> 00:14:15,880 Speaker 1: What I have heard though from the empirical researchers and 268 00:14:16,200 --> 00:14:18,439 Speaker 1: from a guy named Aaron Reiki at a place called Upturn, 269 00:14:19,040 --> 00:14:21,800 Speaker 1: which is a think tank that focuses on algorithmic fairness, 270 00:14:22,520 --> 00:14:27,000 Speaker 1: is that ultimately this sort of extra data or very 271 00:14:27,000 --> 00:14:29,480 Speaker 1: obscure data, or data where it's hard to explain the 272 00:14:29,520 --> 00:14:32,880 Speaker 1: exact effect of it, that it's hard to use that 273 00:14:32,960 --> 00:14:36,040 Speaker 1: to manipulate your credit score, massage your credit score into 274 00:14:36,640 --> 00:14:42,320 Speaker 1: higher and that really what drives it is timeliness of payments. 275 00:14:42,320 --> 00:14:45,240 Speaker 1: So this has led some people to say incasing group 276 00:14:45,280 --> 00:14:47,720 Speaker 1: algorithm watch. Um, I think there've been people there that 277 00:14:47,760 --> 00:14:50,520 Speaker 1: have argued, look, we have to we we should just 278 00:14:50,560 --> 00:14:53,200 Speaker 1: simplify this system, don't make it so secret because eighty 279 00:14:53,960 --> 00:14:58,320 Speaker 1: of the value is in very obvious things like your 280 00:14:58,320 --> 00:15:00,920 Speaker 1: payment history. But there are people out there that still 281 00:15:00,920 --> 00:15:14,720 Speaker 1: are trying to work the margins yet. So just on 282 00:15:14,840 --> 00:15:17,440 Speaker 1: that point, you know, if we have questions about how 283 00:15:17,480 --> 00:15:20,280 Speaker 1: these algorithms are working, what sort of assumptions they're making, 284 00:15:20,320 --> 00:15:23,720 Speaker 1: what sort of data they're looking at, then surely the 285 00:15:23,800 --> 00:15:27,120 Speaker 1: solution to this is to throw them open to some 286 00:15:27,200 --> 00:15:29,680 Speaker 1: more scrutiny. And in fact, again if you look at 287 00:15:29,720 --> 00:15:33,520 Speaker 1: the credit score history, that's exactly what happened. There was 288 00:15:33,600 --> 00:15:38,000 Speaker 1: such an uproar over credit scores that eventually, I mean 289 00:15:38,040 --> 00:15:40,160 Speaker 1: it took them a while, but eventually the US government 290 00:15:40,160 --> 00:15:43,080 Speaker 1: said you have to at least make everyone's credit score 291 00:15:43,160 --> 00:15:46,360 Speaker 1: available to them, and that's why nowadays you can go 292 00:15:46,440 --> 00:15:50,000 Speaker 1: online and find your credit score and then eventually, of course, 293 00:15:50,040 --> 00:15:53,200 Speaker 1: your your credit score gets hacked, um in a major 294 00:15:53,320 --> 00:15:56,560 Speaker 1: data breach. But that's probably a slight tangent for US. 295 00:15:57,120 --> 00:16:00,600 Speaker 1: I think that was very important transparency legislation. I think 296 00:16:00,600 --> 00:16:03,200 Speaker 1: the problem though, is that what we're finding now is 297 00:16:03,280 --> 00:16:08,080 Speaker 1: that the companies are arbitraging around that because there is 298 00:16:08,120 --> 00:16:11,400 Speaker 1: a core credit score that they will give you, but 299 00:16:11,480 --> 00:16:15,160 Speaker 1: it turns out that in certain applications or certain companies 300 00:16:15,200 --> 00:16:18,280 Speaker 1: want a variation on it, or something that has additional 301 00:16:18,360 --> 00:16:23,640 Speaker 1: data or transformation or bespoke type of score that you 302 00:16:23,680 --> 00:16:26,040 Speaker 1: don't get. And so this is the worry that I 303 00:16:26,120 --> 00:16:29,240 Speaker 1: have is that, you know, if you don't have financial 304 00:16:29,240 --> 00:16:32,240 Speaker 1: regulators and even legislators that are constantly keeping up with 305 00:16:32,760 --> 00:16:36,480 Speaker 1: the newest tricks of the industry, you're going to fall behind. 306 00:16:36,720 --> 00:16:39,680 Speaker 1: And the purposes of the pick right now is almost 307 00:16:39,720 --> 00:16:43,720 Speaker 1: fifty years old. UM, it's underlying purpose will be defeated. Frank, 308 00:16:43,760 --> 00:16:46,040 Speaker 1: I wanna go back real quickly something you were talking 309 00:16:46,040 --> 00:16:48,040 Speaker 1: about earlier, and you're talking about how we all have 310 00:16:48,200 --> 00:16:51,040 Speaker 1: this sort of separate self, which is a collection of 311 00:16:51,080 --> 00:16:55,600 Speaker 1: our characteristics and certain attributes about us. When people talk 312 00:16:55,680 --> 00:17:00,080 Speaker 1: about targeting through advertising and things like that, is it 313 00:17:00,120 --> 00:17:04,840 Speaker 1: as simple as Okay, I want to target Frank and 314 00:17:05,040 --> 00:17:07,960 Speaker 1: so let me call up the information or is it 315 00:17:08,040 --> 00:17:11,880 Speaker 1: I want to target people who are vulnerable about X, 316 00:17:12,480 --> 00:17:17,119 Speaker 1: and Frank, you'll be on the list? Like how like? Really? 317 00:17:17,160 --> 00:17:20,159 Speaker 1: Like what kind of is the visibility on the individual 318 00:17:20,240 --> 00:17:23,480 Speaker 1: level or is it on the attribute level? By and large, 319 00:17:24,000 --> 00:17:27,240 Speaker 1: all that I've heard in terms of the empirical research 320 00:17:27,400 --> 00:17:30,760 Speaker 1: on the mainstream marketing is that is on the sort 321 00:17:30,800 --> 00:17:33,679 Speaker 1: of aggregate level, and so you're buying and selling, you know, 322 00:17:34,040 --> 00:17:38,560 Speaker 1: very very large sets of individuals. However, you've got to 323 00:17:38,600 --> 00:17:41,879 Speaker 1: have two caveats to that. One is that the field 324 00:17:41,880 --> 00:17:46,240 Speaker 1: of reidentification research that people like Latania Sweeney and Arvin 325 00:17:46,320 --> 00:17:49,359 Speaker 1: Nara Ryan have been doing for over a decade, that 326 00:17:49,440 --> 00:17:52,720 Speaker 1: they are constantly finding new ways to re identify people 327 00:17:53,280 --> 00:17:56,919 Speaker 1: from what seemed like anonymised data. There was recently some 328 00:17:57,160 --> 00:18:00,320 Speaker 1: finding in Germany where someone re identified or put back 329 00:18:00,359 --> 00:18:03,520 Speaker 1: together a huge number of people's search engine queries back 330 00:18:03,560 --> 00:18:06,040 Speaker 1: to their name. So this sort of thing happens all 331 00:18:06,080 --> 00:18:07,800 Speaker 1: the time. The second is that you know, if you 332 00:18:07,880 --> 00:18:10,160 Speaker 1: drill down closely enough, I mean, imagine if you were 333 00:18:10,200 --> 00:18:12,919 Speaker 1: like to say, I want to advertise to this person 334 00:18:13,040 --> 00:18:15,960 Speaker 1: that has you know, there are enough identifiable characteristics about 335 00:18:15,960 --> 00:18:18,560 Speaker 1: people if you know something about them that maybe you can, 336 00:18:18,720 --> 00:18:21,160 Speaker 1: you know, try to buy a direct profile on them. 337 00:18:21,400 --> 00:18:22,840 Speaker 1: I know that that was sort of in the air 338 00:18:22,880 --> 00:18:25,919 Speaker 1: when the UH Congress got rid of the I s 339 00:18:25,920 --> 00:18:27,879 Speaker 1: P privacy rules. All these people said, We're going to 340 00:18:27,960 --> 00:18:31,399 Speaker 1: crowdfund a purchase of congress members internet search history or 341 00:18:31,440 --> 00:18:35,320 Speaker 1: browsing history so so, and and there's articles out there 342 00:18:35,320 --> 00:18:38,920 Speaker 1: about UM in China, apparently you can buy these sort 343 00:18:38,920 --> 00:18:42,120 Speaker 1: of dossiers. An article by Theo Rostow is up there 344 00:18:42,280 --> 00:18:45,520 Speaker 1: called what Happens when an acquaintance bis your data. So 345 00:18:45,680 --> 00:18:47,960 Speaker 1: I think that we are on the cusp of some 346 00:18:48,040 --> 00:18:51,960 Speaker 1: pretty rapid unravelings of privacy, but as of now it's 347 00:18:52,040 --> 00:18:54,719 Speaker 1: much more done on the aggregate level. I want to 348 00:18:54,720 --> 00:18:57,760 Speaker 1: segue over to the world of finance because that's one 349 00:18:57,760 --> 00:19:01,720 Speaker 1: place where, of course, out rhythms have kind of been 350 00:19:01,720 --> 00:19:05,040 Speaker 1: in the headlines for many years now, often in a 351 00:19:05,119 --> 00:19:09,679 Speaker 1: negative connotation. People complain about high frequency trading, and that 352 00:19:09,800 --> 00:19:12,359 Speaker 1: it's ruining the market and making it more difficult for 353 00:19:12,480 --> 00:19:17,239 Speaker 1: human beings to actually outperform. What's your take on the 354 00:19:17,359 --> 00:19:22,280 Speaker 1: rise of algorithmic trading and how it's changed markets. So 355 00:19:22,320 --> 00:19:24,359 Speaker 1: I wrote an article a couple of years ago called 356 00:19:24,440 --> 00:19:27,439 Speaker 1: laws Acceleration of Finance, where I looked into some of 357 00:19:27,480 --> 00:19:30,639 Speaker 1: the developments in h f T and dark pools and 358 00:19:30,720 --> 00:19:34,080 Speaker 1: some of these other things. And you know, I've had 359 00:19:34,119 --> 00:19:37,439 Speaker 1: two perspectives then, one being exactly your point about you know, 360 00:19:37,520 --> 00:19:42,400 Speaker 1: some of the risks involved here, worries about systemic risk 361 00:19:42,520 --> 00:19:45,800 Speaker 1: or financial stability if something like the flash crash happened again, 362 00:19:46,400 --> 00:19:51,439 Speaker 1: And also worried about the way in which speed was 363 00:19:51,520 --> 00:19:54,960 Speaker 1: made into this very high value through what I thought 364 00:19:55,000 --> 00:19:57,840 Speaker 1: were pretty bad legal choices, that you could find other 365 00:19:57,880 --> 00:20:00,720 Speaker 1: ways of breaking the tie of say two people bids 366 00:20:00,760 --> 00:20:03,000 Speaker 1: for a given a lot of stock, you know, came 367 00:20:03,040 --> 00:20:05,040 Speaker 1: in at the same time, rather than trying to just 368 00:20:05,160 --> 00:20:08,240 Speaker 1: pull out the decimal point further and further. I will 369 00:20:08,280 --> 00:20:10,600 Speaker 1: say now that it's you know, three years later, I 370 00:20:10,640 --> 00:20:14,800 Speaker 1: think I probably was too concerned about the financial stability 371 00:20:14,800 --> 00:20:17,240 Speaker 1: consequences of h f T, like I mean, and maybe 372 00:20:17,280 --> 00:20:19,720 Speaker 1: I'll be proven wrong. But it does seem that you know, 373 00:20:19,760 --> 00:20:22,320 Speaker 1: now we're almost eight years out from the flash crash, 374 00:20:22,480 --> 00:20:25,280 Speaker 1: and not that much seems to have happened along those 375 00:20:25,320 --> 00:20:28,480 Speaker 1: lines that would be on a really high systemic level. 376 00:20:28,480 --> 00:20:30,560 Speaker 1: Although you know, maybe I've just missed some stuff there. 377 00:20:31,280 --> 00:20:34,200 Speaker 1: I would I still do keep with my earlier point though, 378 00:20:34,240 --> 00:20:38,639 Speaker 1: that I think that this is um that the emphasis 379 00:20:38,680 --> 00:20:42,680 Speaker 1: on speed will eventually, I mean, has has some negative 380 00:20:43,240 --> 00:20:45,680 Speaker 1: side effects, and that we're seeing this in terms of 381 00:20:45,760 --> 00:20:49,080 Speaker 1: you know, just the level of confusion, uh, and frustration 382 00:20:49,119 --> 00:20:52,240 Speaker 1: among some traders about the you know how how who 383 00:20:52,280 --> 00:20:54,520 Speaker 1: gets access to which data and at what price and 384 00:20:54,560 --> 00:20:56,320 Speaker 1: stuff like that. You know, it doesn't seem like it's 385 00:20:56,400 --> 00:21:00,560 Speaker 1: very economically productive use of people's efforts there. Frank the 386 00:21:00,640 --> 00:21:04,160 Speaker 1: other big way in which and I imagine a lot 387 00:21:04,200 --> 00:21:07,680 Speaker 1: of our finance listeners think about this a lot. It 388 00:21:07,840 --> 00:21:11,159 Speaker 1: just questions about like, is this gonna be so good 389 00:21:11,200 --> 00:21:14,440 Speaker 1: that you know, we have a handful of people who 390 00:21:14,520 --> 00:21:20,080 Speaker 1: sort of program the algorithms, mathematicians, uh, physicists, and that 391 00:21:20,160 --> 00:21:22,680 Speaker 1: there's essentially no jobs for anyone else. And I think, 392 00:21:22,720 --> 00:21:26,399 Speaker 1: you know, you obviously hear this anxiety in finance, but 393 00:21:26,480 --> 00:21:28,760 Speaker 1: you also hear in other areas, and you here with 394 00:21:28,880 --> 00:21:32,720 Speaker 1: people concerned about the future of truck drivers if self 395 00:21:32,840 --> 00:21:35,439 Speaker 1: driving cars and self driving trucks get too good, and 396 00:21:35,760 --> 00:21:39,320 Speaker 1: it's just an endless discussion in your work on this, 397 00:21:39,760 --> 00:21:42,880 Speaker 1: what do you feel is are we framing the question right? 398 00:21:43,320 --> 00:21:46,359 Speaker 1: Like how should we think about this idea of where 399 00:21:46,560 --> 00:21:51,640 Speaker 1: he will humans uh lose out to the algorithms and work? Oh? 400 00:21:51,680 --> 00:21:54,360 Speaker 1: I love that question, and I have been writing a 401 00:21:54,400 --> 00:21:58,119 Speaker 1: manuscript on robotics and automation for the past couple of years, 402 00:21:58,240 --> 00:22:00,159 Speaker 1: and I think it is you know the question, and 403 00:22:00,840 --> 00:22:04,280 Speaker 1: I think with respect to the in finance, one of 404 00:22:04,320 --> 00:22:07,119 Speaker 1: the reasons why you see a lot of replacement of 405 00:22:07,280 --> 00:22:10,879 Speaker 1: I think traitors or others you know, with these automated 406 00:22:10,920 --> 00:22:14,000 Speaker 1: systems is because they sort of made the system too simple. 407 00:22:14,040 --> 00:22:16,240 Speaker 1: You know, if you're just maximizing as the price or 408 00:22:16,320 --> 00:22:19,239 Speaker 1: to you know, get a certain return, it can be 409 00:22:19,320 --> 00:22:22,000 Speaker 1: something a computer can do. And I think that when 410 00:22:22,040 --> 00:22:25,120 Speaker 1: you look at other areas where there are multiple competing values, 411 00:22:25,600 --> 00:22:27,760 Speaker 1: that's harder to automate, you know. And I think of 412 00:22:27,800 --> 00:22:30,480 Speaker 1: like a teacher trying to decide whether the he or 413 00:22:30,480 --> 00:22:32,840 Speaker 1: she is gonna yell at somebody who's a disruption or 414 00:22:32,920 --> 00:22:35,560 Speaker 1: kick them out or try some other approach, you know, 415 00:22:35,600 --> 00:22:38,159 Speaker 1: in terms of something more subtle, um, I think of 416 00:22:38,240 --> 00:22:41,119 Speaker 1: doctors and the type of complexity of what they're recommending, 417 00:22:41,200 --> 00:22:43,680 Speaker 1: and even you know, personal trainers other people with high 418 00:22:43,760 --> 00:22:46,919 Speaker 1: touch professions. So I think that my sense is that 419 00:22:46,960 --> 00:22:49,040 Speaker 1: you know that there are probably going to be a 420 00:22:49,040 --> 00:22:53,399 Speaker 1: lot of jobs left out there that whenever you can't 421 00:22:53,400 --> 00:22:57,119 Speaker 1: sort of just optimize as to a mathematical equation. And 422 00:22:57,160 --> 00:22:59,080 Speaker 1: I remember this one great quote. I don't forget, I 423 00:22:59,080 --> 00:23:01,840 Speaker 1: forget who said it, but they that you know, choosing 424 00:23:01,880 --> 00:23:06,800 Speaker 1: how to optimize for a given value is a math problem, 425 00:23:06,840 --> 00:23:10,520 Speaker 1: but choosing what to optimize for is not a math problem. 426 00:23:10,520 --> 00:23:12,159 Speaker 1: And I think that will be the bottom line in 427 00:23:12,240 --> 00:23:17,120 Speaker 1: terms of the automation debate. M I have a slightly conceptual, 428 00:23:17,200 --> 00:23:20,920 Speaker 1: well a very conceptual question when it comes to algorithms 429 00:23:21,040 --> 00:23:25,400 Speaker 1: and finance, especially in a sort of risk management setting. 430 00:23:26,000 --> 00:23:31,959 Speaker 1: Do you think algorithms are ultimately forward looking or backward looking? 431 00:23:32,520 --> 00:23:36,959 Speaker 1: Because often they're focused on extrapolating the future, but they 432 00:23:37,000 --> 00:23:40,760 Speaker 1: extrapolate that future from a set of historic data and 433 00:23:41,240 --> 00:23:46,520 Speaker 1: current data. So the question of whether the past data 434 00:23:47,040 --> 00:23:49,719 Speaker 1: is going to be reflective of the future is one 435 00:23:49,760 --> 00:23:53,520 Speaker 1: of the biggest problems for UM big data driven AI 436 00:23:53,760 --> 00:23:56,720 Speaker 1: algorithmic systems, and I think that you know, we need 437 00:23:56,760 --> 00:24:00,000 Speaker 1: to really be able to answer that and to realize 438 00:24:00,080 --> 00:24:02,120 Speaker 1: is that if we have systems that are just going 439 00:24:02,160 --> 00:24:04,840 Speaker 1: to be based on past data without much opportunity to 440 00:24:04,880 --> 00:24:07,760 Speaker 1: think about how things will change or should change, that's 441 00:24:07,760 --> 00:24:09,960 Speaker 1: a big problem. The best example I can think of 442 00:24:10,000 --> 00:24:11,800 Speaker 1: that is one that you know, Kathy O'Neill gives in 443 00:24:11,800 --> 00:24:14,520 Speaker 1: her book Weapons of Math Destruction, where she talks about 444 00:24:14,560 --> 00:24:17,879 Speaker 1: how if you have an algorithm that is you tell people, 445 00:24:18,600 --> 00:24:21,840 Speaker 1: uh in hr department hire people based on who did 446 00:24:21,840 --> 00:24:23,920 Speaker 1: the best in the past at this firm, or who 447 00:24:23,920 --> 00:24:26,399 Speaker 1: made the most money or what have you. Well, if 448 00:24:26,440 --> 00:24:28,959 Speaker 1: it turns out that you know, in the past also 449 00:24:29,560 --> 00:24:32,000 Speaker 1: there was a forms of discrimination where they were always 450 00:24:32,040 --> 00:24:34,719 Speaker 1: hiring a certain type of person. You may not just 451 00:24:34,800 --> 00:24:39,199 Speaker 1: be baking in assumptions about how people's qualities relate to 452 00:24:39,280 --> 00:24:42,639 Speaker 1: how they perform, but also you're baking into the future 453 00:24:42,640 --> 00:24:46,160 Speaker 1: prediction system the discrimination that existed in the past. So 454 00:24:46,320 --> 00:24:50,359 Speaker 1: both of those have to really be considered. So frank 455 00:24:50,440 --> 00:24:55,080 Speaker 1: you obviously raise a lot of disturbing questions and it's 456 00:24:55,080 --> 00:24:58,160 Speaker 1: hard to wrap our head around how you would begin 457 00:24:58,200 --> 00:25:01,200 Speaker 1: to solve many of these problems I'm thinking about. You know, recently, 458 00:25:01,240 --> 00:25:03,840 Speaker 1: their Facebook going back to them has gotten a lot 459 00:25:03,880 --> 00:25:06,600 Speaker 1: of trouble because people have found that it's very easy 460 00:25:06,680 --> 00:25:11,160 Speaker 1: to create ads that discriminate against races, and then Facebook 461 00:25:11,240 --> 00:25:13,399 Speaker 1: rushes out to put out a fix, and then people 462 00:25:13,480 --> 00:25:16,800 Speaker 1: discover another way to do the same thing. It seems 463 00:25:16,840 --> 00:25:19,000 Speaker 1: like it's sort of like this, you know, they're trying 464 00:25:19,040 --> 00:25:21,199 Speaker 1: to hold back the tide, and there's almost nothing they 465 00:25:21,240 --> 00:25:25,120 Speaker 1: can do with the creation that they've built. But big picture, 466 00:25:25,800 --> 00:25:30,320 Speaker 1: what are some approaches to think about there could take on, um, 467 00:25:30,359 --> 00:25:33,800 Speaker 1: you know, all the issues you raise head on. I'm 468 00:25:33,800 --> 00:25:37,399 Speaker 1: really glad you brought up the Facebook example and the 469 00:25:37,480 --> 00:25:41,960 Speaker 1: problem with the discrimination, the potential for housing discrimination with 470 00:25:42,160 --> 00:25:46,880 Speaker 1: the racial affinity classifiers that advertisers are given on Facebook, 471 00:25:47,440 --> 00:25:49,760 Speaker 1: and I have a couple of thoughts on it. I mean, 472 00:25:49,880 --> 00:25:52,560 Speaker 1: it's funny. I'm I am a bit of an old 473 00:25:52,640 --> 00:25:56,080 Speaker 1: fashioned or an old timer in the sense that I 474 00:25:56,119 --> 00:26:00,359 Speaker 1: think what we're finally seeing is that you can't really 475 00:26:00,440 --> 00:26:03,840 Speaker 1: run the largest media company in the world, which is 476 00:26:03,840 --> 00:26:06,119 Speaker 1: what I think of. Facebook and YouTube were the largest 477 00:26:06,160 --> 00:26:08,399 Speaker 1: some of the largest media companies in the world. You 478 00:26:08,480 --> 00:26:11,399 Speaker 1: can't run them via robot, you can't just run them 479 00:26:11,480 --> 00:26:15,600 Speaker 1: via AI. That all the touted gains and efficiency via 480 00:26:15,680 --> 00:26:22,360 Speaker 1: automation of content, automation of advertisers, preferences, and news feeds, 481 00:26:22,960 --> 00:26:25,920 Speaker 1: that all of those come at great cost, and we're 482 00:26:25,920 --> 00:26:29,240 Speaker 1: finally discovering the costs. It's it's a lot like discovering 483 00:26:29,240 --> 00:26:32,359 Speaker 1: global warming, you know, I mean, the carbon although it 484 00:26:32,400 --> 00:26:34,639 Speaker 1: happens faster. Right, we had sort of a carbon driven 485 00:26:34,920 --> 00:26:38,960 Speaker 1: industrial revolution that was amazing for decades, and now we're discovering, 486 00:26:39,080 --> 00:26:41,200 Speaker 1: Wait a second, we've got a fundamentally retool or else 487 00:26:41,240 --> 00:26:43,840 Speaker 1: we're going to cook ourselves. I think it's very similar 488 00:26:43,840 --> 00:26:47,520 Speaker 1: with respect to this automation of content online that we 489 00:26:48,040 --> 00:26:51,040 Speaker 1: have just been amazed at how much profit these companies 490 00:26:51,080 --> 00:26:53,960 Speaker 1: could make. But now all of a sudden, we're saying, wow, 491 00:26:54,080 --> 00:26:56,160 Speaker 1: we're going to have to fundamentally retool how they work. 492 00:26:56,200 --> 00:26:58,399 Speaker 1: And fortunately they are doing that. And I've heard that 493 00:26:58,480 --> 00:27:00,600 Speaker 1: YouTube is going to hire thousands of people in the 494 00:27:00,600 --> 00:27:04,400 Speaker 1: wake of all the complaints about exploitative child directed content. 495 00:27:04,920 --> 00:27:08,119 Speaker 1: That's uh that has been discovered over the past few months. 496 00:27:08,520 --> 00:27:11,359 Speaker 1: So that's going to bidence their profit margins, no doubt, 497 00:27:11,440 --> 00:27:13,480 Speaker 1: but it's also going to make them, I think, higher 498 00:27:13,560 --> 00:27:16,679 Speaker 1: quality entities, and it's going to create a precedent for 499 00:27:16,760 --> 00:27:21,119 Speaker 1: how we should de automate lots of other fields I think, um, 500 00:27:21,160 --> 00:27:26,840 Speaker 1: including journalism. So yeah, oh okay, let's leave it there. Um. 501 00:27:26,880 --> 00:27:30,480 Speaker 1: Frank Pascal, professor of law at the University of Maryland 502 00:27:30,760 --> 00:27:34,040 Speaker 1: and author of The Black Box Society. Was really great 503 00:27:34,200 --> 00:27:36,960 Speaker 1: having you on. Thanks so much, Oh, thank you the 504 00:27:37,119 --> 00:27:51,400 Speaker 1: terrific questions. I really enjoyed it. Joe. I was kind 505 00:27:51,400 --> 00:27:54,000 Speaker 1: of joking when I cut him off at the journalism point, 506 00:27:54,080 --> 00:27:57,320 Speaker 1: but of course, I mean, we see the applications of 507 00:27:57,520 --> 00:28:02,080 Speaker 1: artificial intelligence and algorithm to our own fields. We at 508 00:28:02,080 --> 00:28:06,040 Speaker 1: Bloomberg have an automated news service that spits out automated 509 00:28:06,119 --> 00:28:09,040 Speaker 1: news and it usually does a pretty good job. Yeah. No, 510 00:28:09,480 --> 00:28:11,439 Speaker 1: it does a good job. And I like that you 511 00:28:11,480 --> 00:28:13,320 Speaker 1: cut off there because I feel like that's a that's 512 00:28:13,359 --> 00:28:16,760 Speaker 1: a whole that's a whole, separate, whole, other episode, very 513 00:28:16,840 --> 00:28:21,720 Speaker 1: raw emotions from our part, that's right, But in general, 514 00:28:21,800 --> 00:28:25,000 Speaker 1: I mean I found that conversation absolutely fascinating and hit 515 00:28:25,119 --> 00:28:28,920 Speaker 1: on so many of the big news items of our day. Actually, 516 00:28:29,040 --> 00:28:31,840 Speaker 1: not just in finance, of course, but also in the 517 00:28:31,840 --> 00:28:36,159 Speaker 1: world of media and politics. It's interesting how many of 518 00:28:36,240 --> 00:28:40,280 Speaker 1: the different things we're talking about today end up coming 519 00:28:40,320 --> 00:28:44,000 Speaker 1: back to algorithms in some way. So obviously so much 520 00:28:44,320 --> 00:28:48,920 Speaker 1: discussion about racial concerns in this country, and as Frank 521 00:28:49,000 --> 00:28:52,760 Speaker 1: pointed out, you know, algorithms may play a big role 522 00:28:52,920 --> 00:28:56,760 Speaker 1: in uh, you know, exacerbating a problem that we'd like 523 00:28:56,840 --> 00:28:58,920 Speaker 1: to fix or and so whether we see that in 524 00:28:59,280 --> 00:29:01,720 Speaker 1: law enforced man or we see that on Facebook, and 525 00:29:01,760 --> 00:29:06,640 Speaker 1: then of course other areas markets, robotics, like it all 526 00:29:06,800 --> 00:29:09,080 Speaker 1: seems to be coming back to the same topic, or 527 00:29:09,080 --> 00:29:10,760 Speaker 1: at least if we try we can help bring it 528 00:29:10,760 --> 00:29:13,240 Speaker 1: back to the same topic. Yeah, and of course it 529 00:29:13,280 --> 00:29:15,560 Speaker 1: also gets to the importance of data, which is a 530 00:29:15,600 --> 00:29:18,520 Speaker 1: conversation that I think you and I have had more 531 00:29:18,560 --> 00:29:21,320 Speaker 1: than once on this show. The people that hold the 532 00:29:21,320 --> 00:29:24,280 Speaker 1: best data nowadays are probably the people who are going 533 00:29:24,320 --> 00:29:28,080 Speaker 1: to do the best competitively. So I expect this will 534 00:29:28,160 --> 00:29:30,560 Speaker 1: just harden some of the scramble that we've seen for 535 00:29:31,080 --> 00:29:35,320 Speaker 1: special data, you know, data that's not available to everyone else, 536 00:29:35,480 --> 00:29:38,200 Speaker 1: especially in the world of finance totally. And I also 537 00:29:38,280 --> 00:29:42,800 Speaker 1: think like it's pretty clear that there's gonna need to 538 00:29:42,840 --> 00:29:45,840 Speaker 1: be some system of redress, right Like, I don't think 539 00:29:45,880 --> 00:29:49,000 Speaker 1: that society is going to accept it. If Okay, you 540 00:29:49,120 --> 00:29:53,480 Speaker 1: got put on the at risk for diabetes list, you 541 00:29:53,560 --> 00:29:56,200 Speaker 1: have no risk for diabetes whatsoever, or it's very low, 542 00:29:56,240 --> 00:29:58,560 Speaker 1: but sorry, there's nothing you can do about it, right Like, 543 00:29:58,640 --> 00:30:01,200 Speaker 1: I don't think that's gonna be a tolerable situation. The 544 00:30:01,280 --> 00:30:05,480 Speaker 1: job of cleaning data, the job of helping people sort 545 00:30:05,520 --> 00:30:08,320 Speaker 1: of have accurate data out there, getting rid of inaccurate 546 00:30:08,400 --> 00:30:12,200 Speaker 1: data seems like this mammoth task that humans are probably 547 00:30:12,240 --> 00:30:15,880 Speaker 1: going to need to be uh employed in. Maybe a 548 00:30:15,960 --> 00:30:20,200 Speaker 1: first test of this is convincing Netflix to break your 549 00:30:20,200 --> 00:30:25,120 Speaker 1: account out of its um I guess Toddler TV specific suggestions, 550 00:30:25,160 --> 00:30:27,440 Speaker 1: and for me it's you know, I watched one rom 551 00:30:27,520 --> 00:30:30,360 Speaker 1: com three years ago, and now it's suggesting an endless 552 00:30:30,440 --> 00:30:34,160 Speaker 1: stream of Julia Roberts movies. So you joke and you 553 00:30:34,200 --> 00:30:36,520 Speaker 1: think about the fact that, like you're still dealing with it, 554 00:30:36,600 --> 00:30:38,360 Speaker 1: and you're like, I'm never going to solve any of 555 00:30:38,360 --> 00:30:41,960 Speaker 1: these real problems that we have in society. Yeah, alright, Well, 556 00:30:42,000 --> 00:30:45,160 Speaker 1: on that note, Uh, this has been another edition of 557 00:30:45,200 --> 00:30:48,240 Speaker 1: the Odd Lots podcast I'm Tracy Alloway. You can follow 558 00:30:48,280 --> 00:30:51,920 Speaker 1: me on Twitter at Tracy Alloway, and I'm Joe wise Inhal. 559 00:30:52,040 --> 00:30:55,120 Speaker 1: You can follow me on Twitter at the Stalwart, and 560 00:30:55,280 --> 00:30:59,400 Speaker 1: you can follow Frank at Frank Pasquill p A s 561 00:30:59,480 --> 00:31:04,040 Speaker 1: qu A l E. And our producer Sarah Patterson at 562 00:31:04,120 --> 00:31:06,760 Speaker 1: Sarah pett with two teams. Thanks for listening.