1 00:00:07,800 --> 00:00:11,000 Speaker 1: Hello, Odd Lots listeners, It's Tracy Alloway. As you may know, 2 00:00:11,119 --> 00:00:14,720 Speaker 1: Odd Lots is hosting its first ever live event on Thursday, 3 00:00:14,720 --> 00:00:17,800 Speaker 1: September nineteen in New York City. Join me and Joe 4 00:00:17,800 --> 00:00:20,759 Speaker 1: Wisenthal as we host an evening of great conversation and 5 00:00:20,920 --> 00:00:24,080 Speaker 1: live music. The Odd Blots Variety Show will feature some 6 00:00:24,160 --> 00:00:28,560 Speaker 1: new and old Odd Lots guests, including the economist Stephanie Kelton, 7 00:00:28,960 --> 00:00:32,720 Speaker 1: Sam Antar, the former Crazy Eddie CFO, and convicted Felon 8 00:00:32,960 --> 00:00:35,879 Speaker 1: will give us some of his best stories. Meanwhile, Lee 9 00:00:35,960 --> 00:00:39,879 Speaker 1: book Eye, once dubbed the philosopher King of Sovereign Debt Lawyers, 10 00:00:40,000 --> 00:00:43,199 Speaker 1: is coming out of retirement for us. Salton, posar of 11 00:00:43,240 --> 00:00:45,839 Speaker 1: Credit Swiss and formerly of the U. S. Treasury, will 12 00:00:45,880 --> 00:00:49,040 Speaker 1: be on stage with Brad Setzer, Senior Fellow at the 13 00:00:49,080 --> 00:00:53,360 Speaker 1: Council for Foreign Relations, talking all things bonds and trade. 14 00:00:53,680 --> 00:00:57,040 Speaker 1: Will also have some markets the music courtesy of Moral Hazard, 15 00:00:57,160 --> 00:01:00,360 Speaker 1: the most important country singer in economics, and an early 16 00:01:00,400 --> 00:01:03,520 Speaker 1: Odd Thoughts guest, and even Joe has promised us a 17 00:01:03,600 --> 00:01:06,600 Speaker 1: song or two. You can stick around after the show 18 00:01:06,680 --> 00:01:08,840 Speaker 1: for some drinks and a chance to chat with us 19 00:01:08,840 --> 00:01:11,800 Speaker 1: and our guests in person, so keep listening to Odd 20 00:01:11,840 --> 00:01:13,920 Speaker 1: Lots to learn how to sign up to see us 21 00:01:14,000 --> 00:01:38,160 Speaker 1: live on Thursday, September nineteenth. We hope to see you there. Hello, 22 00:01:38,240 --> 00:01:41,280 Speaker 1: and welcome to another episode of the Odd Loots podcast. 23 00:01:41,360 --> 00:01:46,080 Speaker 1: I'm Tracy Halloway and I'm Joe wisnal So Joe. I 24 00:01:46,120 --> 00:01:49,000 Speaker 1: feel and I think we've discussed this before, but it 25 00:01:49,040 --> 00:01:52,280 Speaker 1: feels like the world is sort of at an inflection 26 00:01:52,360 --> 00:01:56,680 Speaker 1: point right now. Uh you think, I mean you think so? Yeah. 27 00:01:57,280 --> 00:02:00,280 Speaker 1: Isn't it always kind of an inflection point? Yeah, guess 28 00:02:00,280 --> 00:02:03,360 Speaker 1: that's true. But at least in markets, it feels like, 29 00:02:03,480 --> 00:02:06,560 Speaker 1: you know, warnings that were late in the cycle that 30 00:02:06,640 --> 00:02:08,720 Speaker 1: we could get a recession at the end of this 31 00:02:08,800 --> 00:02:12,320 Speaker 1: year or in those definitely seem to be heating up. 32 00:02:13,760 --> 00:02:16,800 Speaker 1: I think what I would say and where maybe we 33 00:02:16,880 --> 00:02:19,639 Speaker 1: would uh most likely agree, is not that the world 34 00:02:19,720 --> 00:02:22,680 Speaker 1: is that at an inflection point per se, but that 35 00:02:22,840 --> 00:02:26,880 Speaker 1: what comes and goes is periods when suddenly people feel 36 00:02:26,919 --> 00:02:28,720 Speaker 1: the turn is about to come. And we've had a 37 00:02:28,800 --> 00:02:32,040 Speaker 1: series of these over the last ten years, whether it 38 00:02:32,160 --> 00:02:37,320 Speaker 1: was Q four of last year early with the economy 39 00:02:37,360 --> 00:02:40,600 Speaker 1: going down, the euro crisis. From time to time, it's 40 00:02:40,639 --> 00:02:44,440 Speaker 1: like there's this global collective anxiety that rises, and whether 41 00:02:44,480 --> 00:02:47,000 Speaker 1: it's anymore real or not is up for debate, but 42 00:02:47,080 --> 00:02:49,560 Speaker 1: I would absolutely agree right now you're getting a lot 43 00:02:49,600 --> 00:02:53,880 Speaker 1: of like recession calls, bear market calls, started the easy 44 00:02:53,960 --> 00:02:57,600 Speaker 1: cycle calls, things like that, A wave of global anxiety. 45 00:02:57,680 --> 00:02:59,839 Speaker 1: That's a good way to put it. And this isn't really, 46 00:03:00,320 --> 00:03:02,480 Speaker 1: as you mentioned, the first time that we've seen this, 47 00:03:02,680 --> 00:03:06,600 Speaker 1: and I'm not just talking about markets. We've seen sort 48 00:03:06,639 --> 00:03:10,880 Speaker 1: of inflection points happen in politics recently, right, So Brexit 49 00:03:11,000 --> 00:03:15,040 Speaker 1: springs to mind, the election of Donald Trump as well. Yeah, 50 00:03:15,080 --> 00:03:17,079 Speaker 1: and I think the fact that we have another election 51 00:03:17,280 --> 00:03:20,480 Speaker 1: coming up in the US again, it's one of these 52 00:03:20,520 --> 00:03:24,440 Speaker 1: points where people wanted to call some sort of meaningful 53 00:03:24,560 --> 00:03:28,079 Speaker 1: turn in something or or there's no doubt we're we're 54 00:03:28,080 --> 00:03:31,400 Speaker 1: hearing a lot of that these days. Right. So all 55 00:03:31,440 --> 00:03:33,639 Speaker 1: of this is a roundabout way of saying that we're 56 00:03:33,680 --> 00:03:36,520 Speaker 1: getting a lot of forecasts and a lot of predictions, 57 00:03:36,640 --> 00:03:39,160 Speaker 1: a lot of people trying to call or see it 58 00:03:39,200 --> 00:03:42,200 Speaker 1: into the future. And so I thought it would probably 59 00:03:42,240 --> 00:03:45,520 Speaker 1: be a good idea to do an episode on forecasting. 60 00:03:46,840 --> 00:03:48,680 Speaker 1: I love this idea. I mean, one thing that I've 61 00:03:48,680 --> 00:03:51,280 Speaker 1: always wanted to do and never done? Is you know, 62 00:03:51,800 --> 00:03:55,360 Speaker 1: obviously on TV I talked to people all the time 63 00:03:55,520 --> 00:03:57,960 Speaker 1: and they make forecasts like, oh, we're in this stock 64 00:03:58,160 --> 00:04:01,280 Speaker 1: or interested in this sector, or we're telling clients to 65 00:04:01,280 --> 00:04:03,880 Speaker 1: do this, and I've never gone back and it would 66 00:04:03,880 --> 00:04:05,920 Speaker 1: just be too much work for me, and like actually 67 00:04:05,960 --> 00:04:09,200 Speaker 1: made a database of all their calls. But I've always 68 00:04:09,240 --> 00:04:12,040 Speaker 1: thought that's like a fascinating project or that you know 69 00:04:12,200 --> 00:04:14,760 Speaker 1: that because who knows everybody? You know, people just make 70 00:04:14,800 --> 00:04:17,839 Speaker 1: these calls and how often do they ever get revisited 71 00:04:17,880 --> 00:04:21,480 Speaker 1: to see if the person was actually right or useful 72 00:04:21,480 --> 00:04:24,440 Speaker 1: in some way? Totally, And I have a feeling that 73 00:04:24,480 --> 00:04:26,719 Speaker 1: our guests for this episode is going to have something 74 00:04:26,760 --> 00:04:30,240 Speaker 1: to say on that particular topic. So without further ado, 75 00:04:30,279 --> 00:04:32,599 Speaker 1: why don't I go ahead and bring him in. Our 76 00:04:32,640 --> 00:04:37,600 Speaker 1: guest is Phil Tetlock. He is Annenberg University Professor at 77 00:04:37,600 --> 00:04:41,800 Speaker 1: the University of Pennsylvania. He's also the author of numerous 78 00:04:41,880 --> 00:04:46,400 Speaker 1: books and papers on forecasting. So, Phil, welcome to the show. 79 00:04:47,560 --> 00:04:50,240 Speaker 1: I'm glad to be here. So Phil, I guess my 80 00:04:50,320 --> 00:04:53,440 Speaker 1: first question is, you know, one of the thrust of 81 00:04:53,480 --> 00:04:56,400 Speaker 1: a lot of your work is that experts tend to 82 00:04:56,480 --> 00:05:00,760 Speaker 1: get forecasting wrong. So is it weird? It doesn't feel 83 00:05:00,760 --> 00:05:05,360 Speaker 1: weird that you're sort of the expert on why experts fail. Um. 84 00:05:05,640 --> 00:05:08,240 Speaker 1: I guess I've gotten accustomed to it because I've been 85 00:05:08,279 --> 00:05:11,320 Speaker 1: doing it for about thirty five years now. I started 86 00:05:11,320 --> 00:05:16,640 Speaker 1: out just after I got tenure at Berkeley and been 87 00:05:16,800 --> 00:05:21,880 Speaker 1: tracking the accuracy of experts predictions pretty much continuously since then. 88 00:05:22,200 --> 00:05:25,880 Speaker 1: So I'm sometimes called an expert hologist, which is, of 89 00:05:25,880 --> 00:05:29,479 Speaker 1: course the field that does not does not exist. The 90 00:05:29,520 --> 00:05:32,640 Speaker 1: basic idea of keeping track of of how accurate people 91 00:05:32,680 --> 00:05:35,160 Speaker 1: are is it? Is it really good idea? And and 92 00:05:35,360 --> 00:05:40,000 Speaker 1: insofar as we all kept track of ourselves and the forecasting, 93 00:05:40,080 --> 00:05:43,159 Speaker 1: making how well calibrated we are, I think the research 94 00:05:43,200 --> 00:05:46,040 Speaker 1: suggests we would actually get a little bit better at it. 95 00:05:47,000 --> 00:05:50,200 Speaker 1: So you've actually done what I've said for years that 96 00:05:50,240 --> 00:05:54,080 Speaker 1: I've wanted to do, which is creates some database of 97 00:05:54,160 --> 00:05:56,640 Speaker 1: forecasts and actually go back and look at who's right 98 00:05:56,680 --> 00:06:00,360 Speaker 1: and who's wrong. How did you get the thought that 99 00:06:00,360 --> 00:06:03,440 Speaker 1: that would be worth doing initially? And what does such 100 00:06:03,480 --> 00:06:09,080 Speaker 1: a database look like? Because most forecasts are not particularly binary. 101 00:06:09,240 --> 00:06:13,680 Speaker 1: People might say I chance or seventy five chance, so 102 00:06:13,760 --> 00:06:16,960 Speaker 1: usually things aren't just this will happen or this won't happen. 103 00:06:17,480 --> 00:06:21,920 Speaker 1: And often a forecast can be wrong, but the methodology 104 00:06:22,040 --> 00:06:26,320 Speaker 1: turned out to be right, or maybe someone unpredicted chance 105 00:06:26,360 --> 00:06:29,040 Speaker 1: and that was still the right framework and the five 106 00:06:29,120 --> 00:06:32,400 Speaker 1: percent odds did hit. So how do you go even 107 00:06:32,440 --> 00:06:36,039 Speaker 1: go about constructing a database of forecasts and measuring what 108 00:06:36,120 --> 00:06:38,839 Speaker 1: turned out to be right or wrong? Well, that's what 109 00:06:38,920 --> 00:06:42,000 Speaker 1: ten ures for it takes. It takes a long time. 110 00:06:42,720 --> 00:06:46,800 Speaker 1: The key thing to note is that your right that 111 00:06:46,880 --> 00:06:49,440 Speaker 1: there are only two conditions under which you can definitively 112 00:06:49,480 --> 00:06:52,359 Speaker 1: say at particular forecast was right or wrong, and that 113 00:06:52,520 --> 00:06:54,400 Speaker 1: is that the forecast it was rash enough to say 114 00:06:54,400 --> 00:06:57,479 Speaker 1: a hundred percent chance and it didn't happen, or zero 115 00:06:57,520 --> 00:06:59,680 Speaker 1: percent chance that it did happen, in which case you 116 00:06:59,680 --> 00:07:03,080 Speaker 1: know inclusively that that specific forecast was wrong. But a 117 00:07:03,160 --> 00:07:05,680 Speaker 1: forecaster says, you know, there's a seventy percent chance, like 118 00:07:05,760 --> 00:07:08,599 Speaker 1: Nate Silver said there's a seventy percent chance of Hillary 119 00:07:08,640 --> 00:07:13,800 Speaker 1: Clinton winning the election in November? Uh was was was 120 00:07:13,920 --> 00:07:16,760 Speaker 1: Nate Silver wrong? Or or do we happen to inhabit 121 00:07:16,760 --> 00:07:21,920 Speaker 1: a world that was likely on November first? So the 122 00:07:22,160 --> 00:07:25,480 Speaker 1: solution to the problem is statistical. You need you need 123 00:07:25,520 --> 00:07:29,040 Speaker 1: to keep track of lots of forecasts over time. If 124 00:07:29,040 --> 00:07:31,440 Speaker 1: we collect hundreds of of your forecasts over the course 125 00:07:31,480 --> 00:07:34,760 Speaker 1: of a year, and we find that when you say 126 00:07:34,800 --> 00:07:37,320 Speaker 1: there's a seventy percent chance of things happening, those things 127 00:07:37,320 --> 00:07:39,440 Speaker 1: happen about seventy percent of the time. We say there's 128 00:07:39,440 --> 00:07:41,800 Speaker 1: a ninety percent chance those things happen about ninety percent 129 00:07:41,880 --> 00:07:44,240 Speaker 1: of the time, and so forth. If there's a close 130 00:07:44,280 --> 00:07:48,240 Speaker 1: correspondence between your subjective probability estimates and the object of 131 00:07:48,320 --> 00:07:51,040 Speaker 1: frequency with which events occur, you can be said to 132 00:07:51,040 --> 00:07:55,200 Speaker 1: be reasonably well calibrated. And that's a very desirable feature 133 00:07:55,200 --> 00:07:59,560 Speaker 1: of forecasters. So when you run your analysis, your statistical 134 00:07:59,560 --> 00:08:03,600 Speaker 1: analysy says, of all these expert forecasts, and I'm assuming 135 00:08:03,600 --> 00:08:06,560 Speaker 1: some non expert forecasts as well, what did you find? 136 00:08:07,200 --> 00:08:09,320 Speaker 1: You find that people are not very well calibrated. For 137 00:08:09,360 --> 00:08:13,320 Speaker 1: a start, there's a lot of overconfidence. And when uh, 138 00:08:13,920 --> 00:08:16,040 Speaker 1: many many experts would would would say things are an 139 00:08:16,040 --> 00:08:18,920 Speaker 1: eighty or ninety percent likely or even likely, they would 140 00:08:18,960 --> 00:08:22,120 Speaker 1: occur sixty or seventy percent of the time. So there 141 00:08:22,120 --> 00:08:25,800 Speaker 1: were big gaps between subject of probabilities and object of probabilities. 142 00:08:25,800 --> 00:08:29,800 Speaker 1: There's a lot of overconfidence. People were not qualifying their 143 00:08:29,840 --> 00:08:34,880 Speaker 1: forecast appropriately, and that is one of the better replicated 144 00:08:34,920 --> 00:08:39,880 Speaker 1: findings in my field, which is cognitive social psychology of judgment. 145 00:08:40,760 --> 00:08:43,760 Speaker 1: People are over confident, that tend to be over confident. 146 00:08:43,920 --> 00:08:45,800 Speaker 1: Everybody's over confident. There are even there are a few 147 00:08:45,800 --> 00:08:49,000 Speaker 1: souls who were even systematically under confident and who don't 148 00:08:49,000 --> 00:08:52,120 Speaker 1: have enough confidence in their judgment. But the modal, if 149 00:08:52,120 --> 00:08:54,160 Speaker 1: you had to bet on what kind of mistake people 150 00:08:54,200 --> 00:08:57,680 Speaker 1: are making at any given moment, over confidence would be 151 00:08:57,679 --> 00:09:01,320 Speaker 1: the better bet. You know, I think about something. One 152 00:09:01,360 --> 00:09:04,400 Speaker 1: of our colleagues, I'll give a shout out to him, 153 00:09:04,520 --> 00:09:08,040 Speaker 1: Lorc and Roch Kelly. He works with us here at Bloomberg, 154 00:09:08,080 --> 00:09:13,559 Speaker 1: and he's very fond of strategists at banks and other 155 00:09:14,200 --> 00:09:18,760 Speaker 1: experts who love to give a forecast on things like, oh, 156 00:09:18,840 --> 00:09:23,720 Speaker 1: we see a forty percent forecast of this person winning 157 00:09:23,760 --> 00:09:27,000 Speaker 1: the election, or forty percent forecast of this country leaving 158 00:09:27,040 --> 00:09:29,440 Speaker 1: the EU in the next five years, and it's like 159 00:09:29,480 --> 00:09:32,640 Speaker 1: the perfect number because it's you know, it's still on. 160 00:09:33,000 --> 00:09:36,480 Speaker 1: It feels unlikely, but it's close enough to that if 161 00:09:36,520 --> 00:09:38,720 Speaker 1: it happens, you know, you can still say, oh, I 162 00:09:38,720 --> 00:09:40,760 Speaker 1: told you it was significant. But if it doesn't happen, 163 00:09:40,880 --> 00:09:43,960 Speaker 1: you could say it was unlikely. And I'm curious in 164 00:09:44,080 --> 00:09:47,600 Speaker 1: your findings and in your research. You know, we think 165 00:09:47,600 --> 00:09:52,080 Speaker 1: of forecasts, our predictions is the point is to be accurate. 166 00:09:52,120 --> 00:09:54,640 Speaker 1: But it feels like a lot of the reason people 167 00:09:54,920 --> 00:09:57,560 Speaker 1: give forecast is just to be interesting, just to have 168 00:09:57,600 --> 00:09:59,719 Speaker 1: their voice heard, just to get their clients to pick 169 00:09:59,760 --> 00:10:02,920 Speaker 1: up the phone. And I'm curious how that plays into 170 00:10:02,960 --> 00:10:06,160 Speaker 1: your analysis of forecasts when the whole purpose is maybe 171 00:10:06,160 --> 00:10:08,960 Speaker 1: not even to get it right, it's just to provoke 172 00:10:09,000 --> 00:10:10,880 Speaker 1: a thought or to have your name out in the news. 173 00:10:11,520 --> 00:10:15,439 Speaker 1: I think that's a very perceptive observation. It's a very 174 00:10:15,480 --> 00:10:18,640 Speaker 1: delicate dance that people play. On the one hand, you 175 00:10:18,679 --> 00:10:21,320 Speaker 1: want to say things that sound interesting so people don't 176 00:10:21,400 --> 00:10:22,920 Speaker 1: roll their eyes and things that this is boring and 177 00:10:22,960 --> 00:10:25,439 Speaker 1: a useless conversation. On the other hand, you don't want 178 00:10:25,440 --> 00:10:27,280 Speaker 1: to say things that are so interesting that they could 179 00:10:27,280 --> 00:10:32,199 Speaker 1: prove to be wrong later. So you're there seems to 180 00:10:32,240 --> 00:10:34,400 Speaker 1: be sort of in the sweet spot zone. Or if 181 00:10:34,400 --> 00:10:36,120 Speaker 1: you were to use language, you say, well, I think 182 00:10:36,120 --> 00:10:38,880 Speaker 1: there's a distinct possibility that Putin's next move is going 183 00:10:38,920 --> 00:10:42,599 Speaker 1: to be on Belarus or on Estonia. Uh, it's a 184 00:10:42,640 --> 00:10:45,920 Speaker 1: distinct possibility, is wonderful, and exactly the same way is 185 00:10:45,920 --> 00:10:48,360 Speaker 1: pretty good. I mean, if it happens, I say, hey, 186 00:10:48,400 --> 00:10:51,440 Speaker 1: I told you distinct possibility, and if it doesn't happen, 187 00:10:51,440 --> 00:10:54,000 Speaker 1: I said, I merely said it was possible. So you 188 00:10:54,000 --> 00:10:56,679 Speaker 1: you're you're, you're covering yourself very nicely. It is as 189 00:10:56,720 --> 00:11:01,160 Speaker 1: though the art of punditry is the art of appearing 190 00:11:01,160 --> 00:11:03,320 Speaker 1: to go out on a limb without actually going out 191 00:11:03,320 --> 00:11:09,480 Speaker 1: on a limb. So, I mean, Joe referred to forecasts 192 00:11:09,480 --> 00:11:12,600 Speaker 1: as non binary. But I feel like instinctively a lot 193 00:11:12,600 --> 00:11:15,520 Speaker 1: of people want to know whether something will or will 194 00:11:15,600 --> 00:11:19,160 Speaker 1: not happen, and yet we have all these, uh, forecasting 195 00:11:19,240 --> 00:11:21,480 Speaker 1: calls that are sort of, you know, thirty percent chance, 196 00:11:21,559 --> 00:11:24,920 Speaker 1: forty percent chance. Our probability is a cop out in 197 00:11:24,960 --> 00:11:27,839 Speaker 1: that case, is it's something that people hide under. Well, 198 00:11:27,840 --> 00:11:31,480 Speaker 1: probable probabilities are not a cop out. If you're participating 199 00:11:31,600 --> 00:11:35,640 Speaker 1: in forecasting tournaments in which we can systematically track how 200 00:11:35,679 --> 00:11:39,920 Speaker 1: often you're things happen, and if things you say likely 201 00:11:39,960 --> 00:11:42,560 Speaker 1: happen forty of the time, you're pretty well calibrated, and 202 00:11:42,559 --> 00:11:46,000 Speaker 1: it's not simply being used as a cop out. So, okay, 203 00:11:46,040 --> 00:11:50,400 Speaker 1: So you've built this database and you've been tracking forecasters 204 00:11:50,400 --> 00:11:54,319 Speaker 1: ability for years, and you mentioned that there are forecasting 205 00:11:54,360 --> 00:11:56,520 Speaker 1: tournaments and we can really track this stuff, and that 206 00:11:56,600 --> 00:12:00,120 Speaker 1: we can track how well calibrated forecasters are, not on 207 00:12:00,200 --> 00:12:03,720 Speaker 1: any sort of individual prediction, but by whether over time 208 00:12:04,320 --> 00:12:08,320 Speaker 1: their predictions of likelihood events happened seven out of ten 209 00:12:08,360 --> 00:12:11,319 Speaker 1: times and so forth. What are some of the interesting 210 00:12:11,400 --> 00:12:16,400 Speaker 1: patterns you've discovered besides that people tend to be overconfident, 211 00:12:16,840 --> 00:12:20,680 Speaker 1: what kind of people, what kind of approaches tend to 212 00:12:20,720 --> 00:12:24,520 Speaker 1: distinguish the better forecasters from the worst ones, Because ultimately, 213 00:12:24,520 --> 00:12:26,920 Speaker 1: I think that's sort of the point of your research. 214 00:12:27,480 --> 00:12:30,199 Speaker 1: There are there are two classic biases that we have 215 00:12:30,320 --> 00:12:32,880 Speaker 1: found over the years, and one of them is that 216 00:12:32,960 --> 00:12:35,760 Speaker 1: people are too quick to make up their minds, and 217 00:12:35,800 --> 00:12:38,040 Speaker 1: the other is that people are too slow to change them. 218 00:12:38,320 --> 00:12:41,559 Speaker 1: And it's the combination of those two things that causes 219 00:12:41,679 --> 00:12:47,360 Speaker 1: chronic over confidence. So I'm curious, beyond sort of individual 220 00:12:47,520 --> 00:12:51,080 Speaker 1: characteristics that make people a good or bad forecaster, did 221 00:12:51,080 --> 00:12:55,800 Speaker 1: you notice any discrepancy in the type of forecasts being made, Like, 222 00:12:55,880 --> 00:13:00,520 Speaker 1: for instance, did political forecasting tend to be more or 223 00:13:00,600 --> 00:13:06,320 Speaker 1: less accurate than something like economic or financial forecasting. Oh, 224 00:13:06,360 --> 00:13:09,240 Speaker 1: it really depends on the timeframe and and the and 225 00:13:09,360 --> 00:13:12,480 Speaker 1: the types of questions you're asking. I think economic and 226 00:13:12,520 --> 00:13:16,040 Speaker 1: political forecasting can be pretty hazardous to your reputation. I 227 00:13:16,080 --> 00:13:19,040 Speaker 1: think what we noticed more than anything was that the 228 00:13:19,400 --> 00:13:22,320 Speaker 1: types of forecasters who tended to be better did tend 229 00:13:22,360 --> 00:13:24,720 Speaker 1: to be a little bit more boring. There they were 230 00:13:24,760 --> 00:13:26,720 Speaker 1: more likely to say on the one hand, on the 231 00:13:26,720 --> 00:13:28,520 Speaker 1: other hand, they were they were, they were, they were 232 00:13:28,520 --> 00:13:31,240 Speaker 1: engaging in more explicit balancing and say, well, there's this 233 00:13:31,320 --> 00:13:33,120 Speaker 1: causal for us, and there's that causal for us, and 234 00:13:33,120 --> 00:13:35,280 Speaker 1: you have to balance them against each other. So the 235 00:13:35,600 --> 00:13:40,040 Speaker 1: types of forecasting talk that make forecasters appealing to the 236 00:13:40,080 --> 00:13:43,040 Speaker 1: media tend to details that tend to make forecasts less 237 00:13:43,160 --> 00:13:45,839 Speaker 1: less accurate. So a forecaster is going to be more 238 00:13:45,840 --> 00:13:48,080 Speaker 1: appealing to the media, it would seem if they if 239 00:13:48,080 --> 00:13:50,000 Speaker 1: they can come up with a compelling sound bite and 240 00:13:50,040 --> 00:13:52,120 Speaker 1: they say something like, well, you know, I think the 241 00:13:52,440 --> 00:13:55,040 Speaker 1: Saudi regime is going to collapse within the next twelve 242 00:13:55,080 --> 00:13:57,880 Speaker 1: to twenty four months. That's that's a that's a very 243 00:13:58,440 --> 00:14:01,280 Speaker 1: dramatic forecast. It would have a lot of consequences for 244 00:14:01,360 --> 00:14:05,000 Speaker 1: the Middle East and from World Politics. A forecast who says, well, 245 00:14:05,040 --> 00:14:07,000 Speaker 1: you know, there are people have been predicting the major 246 00:14:07,040 --> 00:14:09,640 Speaker 1: regime change in Saudi Arabia off and on for the 247 00:14:09,679 --> 00:14:14,000 Speaker 1: last forty years. It hasn't happened yet. The base rate 248 00:14:14,040 --> 00:14:17,280 Speaker 1: prediction is it is it's not very likely. Um, there 249 00:14:17,280 --> 00:14:21,720 Speaker 1: are some reasons for some concern, but you know, you 250 00:14:21,760 --> 00:14:24,120 Speaker 1: can steal your eyes start to glaze over. Listening to 251 00:14:24,400 --> 00:14:28,680 Speaker 1: the more accurate forecasters tend to to bore people. Yeah, 252 00:14:28,840 --> 00:14:32,000 Speaker 1: So I'm just thinking so at the time that we're 253 00:14:32,040 --> 00:14:36,320 Speaker 1: recording this episode, just in the last day, and by 254 00:14:36,320 --> 00:14:38,320 Speaker 1: the time you're hearing this, this would be old news. 255 00:14:38,400 --> 00:14:41,600 Speaker 1: But in the last day, uh Deutsche Bank, for example, 256 00:14:41,680 --> 00:14:45,960 Speaker 1: announced a major restructuring of its bank. And I'm just 257 00:14:46,040 --> 00:14:48,800 Speaker 1: thinking about, how like the imperative for the news media 258 00:14:49,280 --> 00:14:53,360 Speaker 1: is immediately to find people who will come on this 259 00:14:53,440 --> 00:14:57,320 Speaker 1: morning and say something about whether this restructuring of the 260 00:14:57,360 --> 00:14:59,840 Speaker 1: bank is likely to be enough, did it go far enough, 261 00:15:00,120 --> 00:15:04,000 Speaker 1: will restore the bank to robust profitability and so forth, 262 00:15:04,640 --> 00:15:08,680 Speaker 1: And it sounds like that imperative is almost exactly the 263 00:15:08,680 --> 00:15:11,960 Speaker 1: opposite of what's likely to make a good forecaster. And 264 00:15:12,080 --> 00:15:14,800 Speaker 1: anyone who already has their mind made up but already 265 00:15:14,840 --> 00:15:18,240 Speaker 1: has a strong view on the efficacy of the plan, 266 00:15:19,120 --> 00:15:22,800 Speaker 1: at least going by your heuristic that the people that 267 00:15:22,920 --> 00:15:27,280 Speaker 1: good forecasting is not correlated with a quick judgment or 268 00:15:27,680 --> 00:15:31,000 Speaker 1: you know, quick decision making, which seems like we're kind 269 00:15:31,000 --> 00:15:34,000 Speaker 1: of like highlighting most likely highlighting some of the worst 270 00:15:34,000 --> 00:15:37,040 Speaker 1: people we could be highlighting. Well, it really you have 271 00:15:37,120 --> 00:15:39,280 Speaker 1: to make a decision about what kind of business you're in. 272 00:15:39,400 --> 00:15:42,160 Speaker 1: If you're in the accuracy business, you're going to look 273 00:15:42,200 --> 00:15:44,800 Speaker 1: for the kinds of forecasters we've been looking for in 274 00:15:44,840 --> 00:15:47,280 Speaker 1: the work we've been doing with the intelligence community and 275 00:15:47,320 --> 00:15:50,520 Speaker 1: elsewhere where. These are going to be forecasters who are 276 00:15:50,560 --> 00:15:53,120 Speaker 1: not very entertaining. If you're in the entertainment business, you're 277 00:15:53,120 --> 00:15:55,200 Speaker 1: going to be looking for people who are entertaining. There's 278 00:15:55,240 --> 00:15:59,720 Speaker 1: a separation in Bloomberg and probably mostly another sophisticated media 279 00:15:59,760 --> 00:16:03,000 Speaker 1: comes these between analytics and and and the front end. Right, 280 00:16:21,800 --> 00:16:25,680 Speaker 1: So what about experience, Like to what degree can if 281 00:16:25,760 --> 00:16:29,080 Speaker 1: you're an expert, presumably you have you know, probably decades 282 00:16:29,120 --> 00:16:32,080 Speaker 1: of experience, and you know you've been studying a particular 283 00:16:32,120 --> 00:16:35,080 Speaker 1: subject matter for a long time, you've noticed patterns, or 284 00:16:35,120 --> 00:16:38,760 Speaker 1: you can reach for historical analogies to describe a current 285 00:16:38,800 --> 00:16:44,480 Speaker 1: situation and extrapolate from that. Does experience help offset the 286 00:16:44,600 --> 00:16:49,440 Speaker 1: problems of overconfidence at all? Sometimes it depends what you 287 00:16:49,480 --> 00:16:53,760 Speaker 1: do with the experience. Um. People have different styles of 288 00:16:53,800 --> 00:16:58,160 Speaker 1: thinking and and some people with experience become extremely skilled 289 00:16:58,680 --> 00:17:02,920 Speaker 1: at creating very comp telling, very articulate justifications why they 290 00:17:03,000 --> 00:17:07,440 Speaker 1: must be right. Uh. So experience can actually solidify dogmatism 291 00:17:07,560 --> 00:17:11,000 Speaker 1: for people with that cognitive style, and either for other people, 292 00:17:11,160 --> 00:17:14,320 Speaker 1: experience melos of them and they become more tuned to 293 00:17:14,359 --> 00:17:18,200 Speaker 1: the limitations of their prior world views and uh, they 294 00:17:18,320 --> 00:17:22,520 Speaker 1: introduce more appropriate qualifications, may become better calibrated. So but 295 00:17:22,600 --> 00:17:26,840 Speaker 1: but it's not a one trajectory. Be people people mature 296 00:17:27,080 --> 00:17:30,919 Speaker 1: in different ways. Talk to us about how you train 297 00:17:31,080 --> 00:17:34,879 Speaker 1: people to get better. So obviously, like let's start with 298 00:17:34,960 --> 00:17:38,119 Speaker 1: the assumption that there are some people that aren't just 299 00:17:38,200 --> 00:17:41,639 Speaker 1: looking for media sound bites and maybe as uh to 300 00:17:41,760 --> 00:17:45,640 Speaker 1: use your example there in the intelligence community, and they 301 00:17:45,680 --> 00:17:49,400 Speaker 1: really want to make better forecasts about how things will 302 00:17:50,000 --> 00:17:52,360 Speaker 1: happen in the future. They want to be better at 303 00:17:52,359 --> 00:17:56,680 Speaker 1: predicting save Vladimir Putin's next move? What is that? How 304 00:17:56,680 --> 00:17:58,920 Speaker 1: do you start and how do you what's the general 305 00:17:58,920 --> 00:18:02,840 Speaker 1: approach to becoming better at that? Well, I would say 306 00:18:03,040 --> 00:18:05,400 Speaker 1: the starting point is again not going to be all 307 00:18:05,440 --> 00:18:09,560 Speaker 1: that exciting and it and its bears a strong resemblance 308 00:18:09,600 --> 00:18:12,600 Speaker 1: to what Danny Kahneman proposes in his best selling book 309 00:18:12,680 --> 00:18:16,080 Speaker 1: Thinking Fast and Slow. Uh, it is start with the 310 00:18:16,119 --> 00:18:18,560 Speaker 1: base rates. And if you look at the base rates, 311 00:18:18,840 --> 00:18:22,119 Speaker 1: you you'll see something quite interesting, and that is people 312 00:18:22,240 --> 00:18:25,480 Speaker 1: frequently claim, to go back to the beginning of our conversation, 313 00:18:25,880 --> 00:18:28,679 Speaker 1: people frequently claim that they're at an inflection point in history. 314 00:18:29,280 --> 00:18:31,280 Speaker 1: If you if you look at how many inflection points 315 00:18:31,280 --> 00:18:34,040 Speaker 1: there have been, it's just a very long list. The 316 00:18:34,119 --> 00:18:37,679 Speaker 1: vast majority of claims about inflection points have been false positives. 317 00:18:38,200 --> 00:18:41,439 Speaker 1: So you you would, naturally, I think, be wary of 318 00:18:41,560 --> 00:18:45,119 Speaker 1: claims about inflection points. Another claim you'd be wary of 319 00:18:45,320 --> 00:18:49,080 Speaker 1: is military coups, the revolutions. They're relatively rare events. So 320 00:18:49,200 --> 00:18:51,520 Speaker 1: someone who's making a dramatic claim but it's going to 321 00:18:51,600 --> 00:18:54,080 Speaker 1: be a regime change in a particular country within a 322 00:18:54,119 --> 00:18:58,920 Speaker 1: particular timeframe, the likelihood their being right is pretty low. Still. 323 00:18:59,240 --> 00:19:02,959 Speaker 1: Joe mentioned us in the intro, but what role do 324 00:19:03,000 --> 00:19:07,920 Speaker 1: you think accountability plays in the sort of forecasting industry 325 00:19:08,000 --> 00:19:12,879 Speaker 1: because it feels to me like, given the volume of 326 00:19:13,040 --> 00:19:15,800 Speaker 1: media that's out there right now, you know, either social 327 00:19:15,840 --> 00:19:19,679 Speaker 1: media or traditional forms of media, it feels like you 328 00:19:19,720 --> 00:19:21,719 Speaker 1: have a lot of people who will make, you know, 329 00:19:21,800 --> 00:19:25,080 Speaker 1: say a hundred predictions, and maybe one or two of 330 00:19:25,119 --> 00:19:28,240 Speaker 1: those are right, and then they get trumpeted for those 331 00:19:28,320 --> 00:19:31,560 Speaker 1: right calls, or you know, they laud themselves for those 332 00:19:31,560 --> 00:19:35,000 Speaker 1: correct calls, and people sort of forget about the other 333 00:19:35,160 --> 00:19:38,600 Speaker 1: nine calls that were wrong, and no one bothers to 334 00:19:38,640 --> 00:19:41,159 Speaker 1: go back and check on them. Because there's just so 335 00:19:41,600 --> 00:19:45,399 Speaker 1: much forecasting and so much information out there in the world. 336 00:19:45,560 --> 00:19:51,679 Speaker 1: So how can we develop accountability for forecasting by running 337 00:19:51,680 --> 00:19:56,919 Speaker 1: forecasting tournaments? Um? What prediction markets are? Forecasting tournaments are 338 00:19:56,960 --> 00:20:01,040 Speaker 1: excellent ways of allowing people to tract their accuracy on 339 00:20:01,240 --> 00:20:05,200 Speaker 1: judgment calls for which there aren't ready financial market equivalence. 340 00:20:05,640 --> 00:20:08,040 Speaker 1: How do these I've never I'm not familiar with these, 341 00:20:08,040 --> 00:20:11,120 Speaker 1: So how does they are forecasting tournament work? You ask 342 00:20:11,200 --> 00:20:16,480 Speaker 1: people to put subjective probabilities on events that are specified 343 00:20:16,480 --> 00:20:19,760 Speaker 1: by well defined questions, such as whether Prutent is going 344 00:20:19,800 --> 00:20:25,240 Speaker 1: to be the president of Russia after and those probability 345 00:20:25,280 --> 00:20:27,639 Speaker 1: either in that in that case, it would be a 346 00:20:27,640 --> 00:20:31,119 Speaker 1: probability that would be yes or no, and it should 347 00:20:31,160 --> 00:20:33,840 Speaker 1: sum up to one point zero. There are lots of 348 00:20:33,840 --> 00:20:35,840 Speaker 1: ways of doing it, but they all boiled down to 349 00:20:35,920 --> 00:20:39,960 Speaker 1: the core idea, which is which is keeping score. Sticking 350 00:20:40,080 --> 00:20:44,600 Speaker 1: with the intelligence community framework, how does the role of 351 00:20:44,800 --> 00:20:48,520 Speaker 1: group think play into this and avoiding group think? Because 352 00:20:48,520 --> 00:20:51,120 Speaker 1: if we think back to what are considered a lot 353 00:20:51,160 --> 00:20:55,880 Speaker 1: of the intelligence disasters over the last several years, the 354 00:20:55,960 --> 00:20:59,800 Speaker 1: idea of some idea takes hold and no one feels 355 00:21:00,440 --> 00:21:04,159 Speaker 1: comfortable yelling stop, and suddenly everyone can deals on the 356 00:21:04,240 --> 00:21:07,600 Speaker 1: same idea. Is that something that in your work you 357 00:21:07,680 --> 00:21:10,960 Speaker 1: focus on sort of like these cascades where someone puts 358 00:21:11,000 --> 00:21:13,800 Speaker 1: forth an idea and everyone feels compelled to fall online, 359 00:21:14,080 --> 00:21:18,520 Speaker 1: or there's extreme pressure to voice a concern or voice 360 00:21:18,520 --> 00:21:22,000 Speaker 1: skepticism on things. And are there are ways or strategies 361 00:21:22,040 --> 00:21:27,359 Speaker 1: that aspiring forecasters can use to eliminate or reduce that bias. 362 00:21:27,960 --> 00:21:31,760 Speaker 1: Group think is a big problem, and that's why in 363 00:21:31,840 --> 00:21:36,240 Speaker 1: forecasting tournaments we typically have people make judgments independently of 364 00:21:36,280 --> 00:21:39,240 Speaker 1: each other, um at least initially. It doesn't mean that 365 00:21:39,320 --> 00:21:41,800 Speaker 1: all team or group decisions are going to be bad ones, 366 00:21:42,240 --> 00:21:45,320 Speaker 1: But it does mean that forecasters need to value accuracy 367 00:21:45,359 --> 00:21:48,960 Speaker 1: above all. If your primary goal is pleasing your boss, 368 00:21:48,960 --> 00:21:50,960 Speaker 1: and you have an opinionated boss, it's going to be 369 00:21:51,080 --> 00:21:53,200 Speaker 1: very hard for you to offer that boss an opinion 370 00:21:53,720 --> 00:21:57,000 Speaker 1: with a probability judgment that points to a policy different 371 00:21:57,000 --> 00:22:00,840 Speaker 1: from what the boss prefers. So it it's another version 372 00:22:00,840 --> 00:22:03,280 Speaker 1: of the question what business are you in? Are you 373 00:22:03,320 --> 00:22:05,679 Speaker 1: in the entertainment business, are you in the pleasing your 374 00:22:05,680 --> 00:22:09,600 Speaker 1: boss business? Or you in the accuracy business. Accuracy business 375 00:22:09,720 --> 00:22:12,119 Speaker 1: is often not the first business people are in, talking, 376 00:22:12,240 --> 00:22:15,520 Speaker 1: not even the second business people are in. People are 377 00:22:15,640 --> 00:22:18,720 Speaker 1: trying to have successful careers, they're trying to avoid embarrassment. 378 00:22:18,800 --> 00:22:21,520 Speaker 1: There's a lot of other things people are doing a 379 00:22:21,600 --> 00:22:24,320 Speaker 1: scie from accuracy. But so forecasting tournaments create a really 380 00:22:24,320 --> 00:22:27,880 Speaker 1: weird social environment. They create a world that's only one 381 00:22:27,920 --> 00:22:31,719 Speaker 1: thing matters, and that is minimizing the gaps between your 382 00:22:31,720 --> 00:22:36,320 Speaker 1: probability judgments and reality over the long term. And that's it. 383 00:22:36,400 --> 00:22:39,560 Speaker 1: That's that's that's that's the sole objective I'm still trying 384 00:22:39,560 --> 00:22:43,159 Speaker 1: to understand. So, as you point out, like forecasting tournaments 385 00:22:43,840 --> 00:22:47,360 Speaker 1: are very weird because in the real world, that's not 386 00:22:47,560 --> 00:22:50,679 Speaker 1: how predictions are made, and people are aware of what 387 00:22:50,720 --> 00:22:53,320 Speaker 1: other people are thinking and talking about. So when you 388 00:22:53,440 --> 00:22:56,560 Speaker 1: consult and when you talk with people, how do you 389 00:22:56,640 --> 00:23:00,520 Speaker 1: foster a culture? And I guess this is really what 390 00:23:00,600 --> 00:23:04,800 Speaker 1: matters to the end consumers of your research, is how 391 00:23:04,840 --> 00:23:08,679 Speaker 1: do you foster a culture where more people feel that 392 00:23:08,720 --> 00:23:12,320 Speaker 1: they're in the accuracy business. It really helps if it 393 00:23:12,359 --> 00:23:15,560 Speaker 1: comes from the top. People are looking at your typically 394 00:23:15,560 --> 00:23:18,920 Speaker 1: look up for the normative cues about what's appropriate. So 395 00:23:19,040 --> 00:23:21,280 Speaker 1: if you have a boss who's open to being wrong, 396 00:23:21,520 --> 00:23:24,080 Speaker 1: that that helps a lot. So, Phil, I know the 397 00:23:24,119 --> 00:23:27,880 Speaker 1: majority of your work has to do with statistical analysis 398 00:23:28,000 --> 00:23:31,280 Speaker 1: of probabilities of forecasts, but I'm curious, could you give 399 00:23:31,400 --> 00:23:34,600 Speaker 1: us a sort of case study that you've come across 400 00:23:34,760 --> 00:23:38,240 Speaker 1: of a forecast that has gone very, very wrong and 401 00:23:38,280 --> 00:23:40,800 Speaker 1: that sort of brings together some of the themes or 402 00:23:40,920 --> 00:23:43,560 Speaker 1: lessons that you've been talking about. I can go back 403 00:23:43,560 --> 00:23:45,720 Speaker 1: to the very beginning and when I was doing this 404 00:23:45,760 --> 00:23:49,440 Speaker 1: work and everybody thought we were at a major inflection point, 405 00:23:50,480 --> 00:23:53,560 Speaker 1: virtually everybody, and they were more or less right about it. 406 00:23:53,600 --> 00:23:55,600 Speaker 1: I mean, the most inflection point calls are wrong, but 407 00:23:55,640 --> 00:23:57,720 Speaker 1: they were right that the Soviet Union At the time 408 00:23:57,720 --> 00:23:59,520 Speaker 1: I was starting off on this work was at an 409 00:23:59,520 --> 00:24:03,760 Speaker 1: inflection point. People didn't have any idea where the Soviet 410 00:24:03,840 --> 00:24:06,919 Speaker 1: Union was going to go. The Conservatives thought that the 411 00:24:06,920 --> 00:24:10,280 Speaker 1: Soviet Union was incapable of reforming itself. The liberals thought 412 00:24:10,320 --> 00:24:13,600 Speaker 1: that Reagan was driving the Soviet Union into nio Stalinist 413 00:24:13,640 --> 00:24:17,600 Speaker 1: retrenchments would become more aggressive. Yet Gorbachev came along in 414 00:24:17,680 --> 00:24:21,359 Speaker 1: March of five. He became the General Party Secretary, and 415 00:24:21,400 --> 00:24:25,359 Speaker 1: he proceeded, with Glasgow's embarrasster to liberalize the Soviet Union 416 00:24:25,400 --> 00:24:28,800 Speaker 1: in ways that we're really astonishing now. After the fact, 417 00:24:28,880 --> 00:24:31,160 Speaker 1: the Conservatives said, hey, we forced them to do it, 418 00:24:31,280 --> 00:24:33,560 Speaker 1: but they didn't really expect the Soviet Union was capable 419 00:24:33,560 --> 00:24:36,880 Speaker 1: of reforming itself beforehand, and the Liberals said, well, we knew, 420 00:24:36,880 --> 00:24:39,159 Speaker 1: we knew it all along because the Soviet economy was 421 00:24:39,160 --> 00:24:41,719 Speaker 1: crumbling and the Soviets were needed to do this, and 422 00:24:41,760 --> 00:24:45,320 Speaker 1: Reagan had no role in it at all. So the 423 00:24:45,320 --> 00:24:49,320 Speaker 1: paradox was that nobody was really very close at all 424 00:24:49,359 --> 00:24:52,800 Speaker 1: to predicting what would happen, but everybody after the fact 425 00:24:52,880 --> 00:24:56,400 Speaker 1: had a confident explanation for what would happen. Is there 426 00:24:56,440 --> 00:25:01,399 Speaker 1: anything that people could have done better prior to the 427 00:25:01,480 --> 00:25:06,439 Speaker 1: events unfolding that could have made their forecast better, or 428 00:25:06,480 --> 00:25:08,040 Speaker 1: is it the kind of thing it was so novel 429 00:25:08,880 --> 00:25:12,479 Speaker 1: and you know, it's kind of so unexpected that this 430 00:25:12,480 --> 00:25:14,879 Speaker 1: would just be a really hard thing to forecast in 431 00:25:14,880 --> 00:25:18,240 Speaker 1: any meaningful sense. It was a hard thing to forecast. 432 00:25:18,359 --> 00:25:21,920 Speaker 1: But there were clues that Gorbachev was different, and even 433 00:25:21,960 --> 00:25:24,879 Speaker 1: a conservative like Margaret Thatcher was signaling that based on 434 00:25:24,880 --> 00:25:28,600 Speaker 1: our early meetings with Gorbachev, I think that the key 435 00:25:28,680 --> 00:25:32,280 Speaker 1: factor here is how fast are you willing to change 436 00:25:32,280 --> 00:25:35,320 Speaker 1: your mind in response to the the incoming evidence. So after 437 00:25:35,359 --> 00:25:40,239 Speaker 1: Gorbachev became General Secretary in early there were there were 438 00:25:40,280 --> 00:25:43,439 Speaker 1: lots of little bits of news that suggested this was 439 00:25:43,640 --> 00:25:46,040 Speaker 1: going to be a different style of leadership, and maybe 440 00:25:46,080 --> 00:25:48,680 Speaker 1: not just a different style of leadership, a different substance 441 00:25:48,760 --> 00:25:51,960 Speaker 1: different that a different substance of policies would be pursued, 442 00:25:52,720 --> 00:25:55,040 Speaker 1: and it would be it was your willingness, I think, 443 00:25:55,040 --> 00:25:59,560 Speaker 1: to make small, rapid adjustments in response to the news. 444 00:26:00,240 --> 00:26:02,600 Speaker 1: It wasn't and there was any one big item that 445 00:26:02,880 --> 00:26:05,600 Speaker 1: absolutely turned the case, but there were lots of little bit, 446 00:26:05,720 --> 00:26:10,479 Speaker 1: little bits of news that created over time that good, 447 00:26:10,640 --> 00:26:13,480 Speaker 1: good forecasters could it could attend to. And I think 448 00:26:13,520 --> 00:26:16,400 Speaker 1: that's one of the finding features of the best forecasters 449 00:26:16,520 --> 00:26:20,520 Speaker 1: is that they're more granular. They make distinctions among more 450 00:26:20,560 --> 00:26:24,080 Speaker 1: degrees of maybe than normal people do. An old joke 451 00:26:24,240 --> 00:26:27,360 Speaker 1: in my field there's people can can really only distinguish 452 00:26:27,440 --> 00:26:31,200 Speaker 1: three degrees of uncertainty, yes, now, and maybe the best 453 00:26:31,280 --> 00:26:34,600 Speaker 1: forecasters are people who know the difference between a forty 454 00:26:34,680 --> 00:26:40,159 Speaker 1: sixty bed and a sixty forty bed or even so 455 00:26:41,000 --> 00:26:43,160 Speaker 1: it's some someone analog I said, the would be no doubt. 456 00:26:43,160 --> 00:26:45,119 Speaker 1: For example, if I said, you know, good poker players 457 00:26:45,119 --> 00:26:47,120 Speaker 1: could do that, and you say, well, sure, they must 458 00:26:47,160 --> 00:26:49,080 Speaker 1: be able to do that in a repeated play game. 459 00:26:49,160 --> 00:26:52,000 Speaker 1: We get rapid quantitative feedback. But I said, well, good 460 00:26:52,200 --> 00:26:55,840 Speaker 1: geopolitical and economic forecasters also do that, and you say, well, 461 00:26:55,840 --> 00:26:59,280 Speaker 1: it cannot really be possible, and and answers, yes it can. 462 00:27:00,160 --> 00:27:03,480 Speaker 1: So do you have a favorite forecaster? You know, it's 463 00:27:03,480 --> 00:27:06,600 Speaker 1: like asking who are my favorite children? Right, I know 464 00:27:06,680 --> 00:27:10,359 Speaker 1: I'm not I would I wouldn't take any particular person 465 00:27:10,400 --> 00:27:12,880 Speaker 1: as a favorite forecasting but I think there are lots 466 00:27:12,920 --> 00:27:16,040 Speaker 1: of very admirable people out there. Well, in the beginning, 467 00:27:16,040 --> 00:27:19,760 Speaker 1: you mentioned Nate Silver and his prediction or not prediction, 468 00:27:19,880 --> 00:27:22,119 Speaker 1: but his assessment maybe a better way to put it, 469 00:27:22,160 --> 00:27:26,000 Speaker 1: that Hillary had a sevent chance of winning the election. 470 00:27:26,440 --> 00:27:32,560 Speaker 1: He's someone who has a very sort of clear understanding 471 00:27:32,840 --> 00:27:36,439 Speaker 1: of probabilities and he puts h you know, he has 472 00:27:36,440 --> 00:27:39,639 Speaker 1: a difference between eighty five and seventy and fifty and 473 00:27:39,680 --> 00:27:42,600 Speaker 1: twenty and probably fits very well into your database. By 474 00:27:42,640 --> 00:27:47,639 Speaker 1: and large, do his seventy forecasts proved to be right 475 00:27:47,640 --> 00:27:51,000 Speaker 1: about seven out of ten times. I have not analyzed 476 00:27:51,080 --> 00:27:54,959 Speaker 1: Nate's data, but Nate does have data, and he analyzes 477 00:27:55,000 --> 00:27:58,119 Speaker 1: it himself, and he has reported how well calibrated his 478 00:27:58,200 --> 00:28:01,320 Speaker 1: boy is the forecast on the five. I tend to 479 00:28:01,359 --> 00:28:06,640 Speaker 1: be in both sports and political forecasts, and I think 480 00:28:06,640 --> 00:28:09,880 Speaker 1: they're pretty good, uh, in the sense of being well calibrated. 481 00:28:10,119 --> 00:28:13,400 Speaker 1: I don't know the data on resolution, but on calibration, 482 00:28:13,400 --> 00:28:17,080 Speaker 1: they're scoring pretty well. There are two key fassets of 483 00:28:17,080 --> 00:28:20,240 Speaker 1: being a good forecaster when you're doing subjective probability scoring. 484 00:28:21,000 --> 00:28:23,480 Speaker 1: One of them is what I mentioned earlier, calibration. So 485 00:28:23,600 --> 00:28:25,960 Speaker 1: when you say seventy percent likely to seventy percent, things 486 00:28:26,000 --> 00:28:30,200 Speaker 1: happen seventy percent of the time. The other is resolution. Now, 487 00:28:30,280 --> 00:28:32,760 Speaker 1: so there's there's a sneaky and lazy way of being 488 00:28:33,000 --> 00:28:35,960 Speaker 1: well calibrated. So if you're a weather forecaster in Seattle 489 00:28:36,119 --> 00:28:40,120 Speaker 1: and it rains sixty percent at the time, and you say, hey, 490 00:28:39,760 --> 00:28:42,520 Speaker 1: I'm just gonna say there's sixty percent likelihood of rain 491 00:28:42,560 --> 00:28:44,520 Speaker 1: every day, and you know what, I'm going to be 492 00:28:44,520 --> 00:28:47,200 Speaker 1: well calibrated because rain will happen sixty percent of the time. 493 00:28:48,040 --> 00:28:51,000 Speaker 1: You you would be you'd be well calibrated, but you'd 494 00:28:51,040 --> 00:28:54,800 Speaker 1: be very uninteresting. So there's another property you need to 495 00:28:54,840 --> 00:28:58,280 Speaker 1: ask a forecasters beyond calibration, and that is you need 496 00:28:58,360 --> 00:29:01,160 Speaker 1: to ask them, are you good at assigning much higher 497 00:29:01,160 --> 00:29:03,640 Speaker 1: probabilities to things that happen than to things that don't. 498 00:29:03,920 --> 00:29:07,280 Speaker 1: Are you good at being justifiably decisive? So you want 499 00:29:07,280 --> 00:29:10,360 Speaker 1: forecasters who are two things that you want there appropriately humble, 500 00:29:10,640 --> 00:29:14,680 Speaker 1: which means well calibrated, but they're also justifiably decisive. They 501 00:29:14,960 --> 00:29:18,480 Speaker 1: say interesting, decisive things when they have a warrant for 502 00:29:18,680 --> 00:29:22,000 Speaker 1: saying those things, so that it's a combination of those 503 00:29:22,040 --> 00:29:24,160 Speaker 1: two things that makes some most a so called super 504 00:29:24,200 --> 00:29:28,720 Speaker 1: forecaster in our work. And um, but I think that 505 00:29:29,280 --> 00:29:32,240 Speaker 1: the Nate Silver group at five three is doing doing 506 00:29:32,280 --> 00:29:34,200 Speaker 1: the right things and I think a number of other 507 00:29:34,280 --> 00:29:36,880 Speaker 1: organizations are starting to do the right things as well. 508 00:29:37,800 --> 00:29:42,080 Speaker 1: So on that note, how should forecasters deal with tail 509 00:29:42,200 --> 00:29:45,440 Speaker 1: risk events? Because of course, as you as you put it, 510 00:29:45,560 --> 00:29:48,120 Speaker 1: you know, you could just sort of do an average 511 00:29:48,160 --> 00:29:50,920 Speaker 1: of probabilities and you might look very smart and very 512 00:29:51,000 --> 00:29:54,160 Speaker 1: well calibrated. But at some point there is a chance 513 00:29:54,240 --> 00:29:57,000 Speaker 1: that a big unexpected event is going to come out 514 00:29:57,000 --> 00:30:00,640 Speaker 1: of nowhere and sort of shift the entire regime of 515 00:30:00,680 --> 00:30:04,240 Speaker 1: statistics in some way. How should forecasters deal with those 516 00:30:04,360 --> 00:30:10,480 Speaker 1: kind of unforeseen risks? I think you want to think 517 00:30:10,520 --> 00:30:14,000 Speaker 1: in terms of shades of gray rather than black and white. 518 00:30:14,480 --> 00:30:16,400 Speaker 1: It's not that things are there, there are some things 519 00:30:16,400 --> 00:30:19,120 Speaker 1: are foreseeable and other things are unforeseeable. That there are 520 00:30:19,120 --> 00:30:22,320 Speaker 1: black swans and then there are white swans. There are 521 00:30:22,360 --> 00:30:26,480 Speaker 1: swans of varying degrees of grayness. And the best forecasters, 522 00:30:26,520 --> 00:30:29,800 Speaker 1: I think, recognize that there is a continuum um and 523 00:30:29,880 --> 00:30:33,680 Speaker 1: that tail risk is a problem, and you have to 524 00:30:33,760 --> 00:30:36,840 Speaker 1: judge how important it is. You will never miss a war, 525 00:30:37,080 --> 00:30:40,040 Speaker 1: or you'll never miss a disaster if you always predict disaster, 526 00:30:41,000 --> 00:30:43,000 Speaker 1: but that will that the cost you're you're paying in 527 00:30:43,080 --> 00:30:46,880 Speaker 1: false positives is ridiculous. The question is how high a 528 00:30:46,960 --> 00:30:49,080 Speaker 1: price are you willing to pay for making lots of 529 00:30:49,080 --> 00:30:51,920 Speaker 1: false positive predictions about you know, the DATO is going 530 00:30:51,960 --> 00:30:54,280 Speaker 1: to fall below two thousand in the next six months, 531 00:30:54,600 --> 00:30:56,840 Speaker 1: that sort of thing, and and and and you know 532 00:30:56,920 --> 00:31:00,240 Speaker 1: by by by futures contracts based on that belief. Skin 533 00:31:00,280 --> 00:31:02,880 Speaker 1: in the game, as it were. Those are judgment calls, 534 00:31:02,960 --> 00:31:07,160 Speaker 1: and you never escape making probability judgments, even though the 535 00:31:07,200 --> 00:31:11,120 Speaker 1: probabilities may be extremely small. Is very difficult to say 536 00:31:11,120 --> 00:31:14,120 Speaker 1: whether someone is well calibrated and distinguishing between events that 537 00:31:14,160 --> 00:31:16,080 Speaker 1: are one and ten thousand likely and one on a 538 00:31:16,160 --> 00:31:19,200 Speaker 1: chillion likely. Right, But there's a huge difference between one 539 00:31:19,200 --> 00:31:23,200 Speaker 1: and ten thousand and one and the chillion right. So, Phil, 540 00:31:23,600 --> 00:31:26,880 Speaker 1: you mentioned you know, people predicting war or natural disasters. 541 00:31:27,000 --> 00:31:33,040 Speaker 1: It does feel sometimes like the people who forecast negative events, 542 00:31:33,120 --> 00:31:36,680 Speaker 1: you know, recession is coming, war is coming, Donald Trump 543 00:31:36,720 --> 00:31:39,520 Speaker 1: is up ending the global order, those sorts of things 544 00:31:39,760 --> 00:31:43,320 Speaker 1: that they seem to make more waves or more in 545 00:31:43,560 --> 00:31:46,920 Speaker 1: roads than people who predict either a status quo or 546 00:31:47,640 --> 00:31:51,800 Speaker 1: positive trends. Do you think people like to hear dire 547 00:31:51,920 --> 00:31:58,080 Speaker 1: forecasts more than extremely optimistic forecasts. I don't know. If 548 00:31:58,120 --> 00:32:00,800 Speaker 1: they like to hear them more, but they seem to 549 00:32:00,800 --> 00:32:04,360 Speaker 1: find them more interesting and they pay more attention to them. 550 00:32:04,480 --> 00:32:08,680 Speaker 1: Can the best forecasters, or the super forecasters as you 551 00:32:08,760 --> 00:32:12,800 Speaker 1: call them, can they always articulate their approach or do 552 00:32:12,960 --> 00:32:16,560 Speaker 1: some people who just have some sort of deeper intuitive sense. 553 00:32:16,600 --> 00:32:19,920 Speaker 1: And I'm thinking about you use the poker analogy. And 554 00:32:19,960 --> 00:32:22,120 Speaker 1: one of the things about poker is that there's different 555 00:32:22,440 --> 00:32:25,040 Speaker 1: ways to play it. So some people are extremely mathematical 556 00:32:25,040 --> 00:32:28,360 Speaker 1: in their forecast, they calculate everything. Others seem too much 557 00:32:28,440 --> 00:32:31,960 Speaker 1: more clearly operate on feel, and they just have a 558 00:32:31,960 --> 00:32:34,440 Speaker 1: good feel for whatever reason that turns out to be 559 00:32:34,480 --> 00:32:37,920 Speaker 1: a successful strategy for them too. Is there arrange in 560 00:32:37,960 --> 00:32:40,480 Speaker 1: the approaches that people use? Or some people can very 561 00:32:40,520 --> 00:32:43,120 Speaker 1: methodically lay out their approach like a date silver where 562 00:32:43,120 --> 00:32:46,840 Speaker 1: they build all these models versus a more intuitive fuel 563 00:32:46,880 --> 00:32:51,200 Speaker 1: based approach that maybe can't be written down as well. Yeah, 564 00:32:51,240 --> 00:32:53,800 Speaker 1: a long time ago, Malcolm Bladder wrote a book I 565 00:32:53,800 --> 00:32:56,280 Speaker 1: don't know if you remember it called Blink. And there 566 00:32:56,320 --> 00:32:58,200 Speaker 1: are some people in my fields who wrote a much 567 00:32:58,280 --> 00:33:04,240 Speaker 1: less well known book. So a rejoinder called think You've 568 00:33:04,240 --> 00:33:07,640 Speaker 1: Got a duel? Isn't here between people who endorse blink 569 00:33:07,720 --> 00:33:11,880 Speaker 1: and people who endorse think I lean towards the think end. 570 00:33:11,920 --> 00:33:15,320 Speaker 1: I'm not precluding the possibility that when you're dealing with 571 00:33:16,000 --> 00:33:19,440 Speaker 1: events that occur over and over again, and there's lots 572 00:33:19,440 --> 00:33:22,400 Speaker 1: of repeated play, and there's lots of opportunity to build 573 00:33:22,440 --> 00:33:28,120 Speaker 1: up deep experience and automated cognitive processing, that some people 574 00:33:28,160 --> 00:33:30,840 Speaker 1: can become intuitively very good at it. Uh And they 575 00:33:30,880 --> 00:33:33,560 Speaker 1: may even be doing rather complex calculations in their head 576 00:33:33,760 --> 00:33:36,680 Speaker 1: very rapidly. So it's not that the intuition is is 577 00:33:36,760 --> 00:33:39,360 Speaker 1: eesp here. It could it could be that some of 578 00:33:39,360 --> 00:33:44,200 Speaker 1: the best forecasters are not have simply overlearned the probability 579 00:33:44,240 --> 00:33:48,080 Speaker 1: calculation heuristics to the point where they got they unfold 580 00:33:48,160 --> 00:33:51,560 Speaker 1: virtually automatically. This isn't like a master pianist, right, It 581 00:33:51,640 --> 00:33:53,920 Speaker 1: doesn't have to think about every key and just you know, 582 00:33:54,040 --> 00:33:55,960 Speaker 1: the great tennis player doesn't have to think about where 583 00:33:55,960 --> 00:33:59,560 Speaker 1: his arm is. That sort of thing. So, Phil, I 584 00:33:59,600 --> 00:34:01,560 Speaker 1: feel like I have to ask, will you give us 585 00:34:01,600 --> 00:34:08,880 Speaker 1: a forecast? Okay, I'll give you a forecast, even though 586 00:34:08,920 --> 00:34:11,920 Speaker 1: I'm not a forecast, just just for you, just for you, 587 00:34:12,040 --> 00:34:13,880 Speaker 1: I will, I will, I will give you a forecast 588 00:34:14,320 --> 00:34:17,440 Speaker 1: and that is I don't think the forecast and forecasting 589 00:34:17,440 --> 00:34:20,920 Speaker 1: practices are going to change very fast. I think that 590 00:34:20,960 --> 00:34:25,040 Speaker 1: the majority of people are making forecasts because they don't 591 00:34:25,040 --> 00:34:27,160 Speaker 1: want to offend people in power, They don't want to 592 00:34:27,160 --> 00:34:31,680 Speaker 1: offend clients because they want to be entertaining in media 593 00:34:31,719 --> 00:34:34,960 Speaker 1: performances or elsewhere. Uh. And the accuracy will continue to 594 00:34:35,000 --> 00:34:38,919 Speaker 1: be a very, very secondary goal, but that gradually over 595 00:34:38,960 --> 00:34:42,640 Speaker 1: the next ten twenty, through years of forecasting tournaments and 596 00:34:42,640 --> 00:34:45,720 Speaker 1: prediction markets, will become an increasingly common way for people 597 00:34:45,880 --> 00:34:50,600 Speaker 1: to resolve certain categories of disagreements. Well, hopefully your your 598 00:34:50,600 --> 00:34:55,399 Speaker 1: appearance on this podcast will help marginally change the trajectory 599 00:34:55,400 --> 00:34:59,560 Speaker 1: of the forecasting industry over the next several decades. But 600 00:34:59,680 --> 00:35:03,800 Speaker 1: really appreciate you coming out a fascinating, fascinating discussion. Thanks 601 00:35:03,800 --> 00:35:20,839 Speaker 1: so much. Okay, take care. So Joe, I'm just going 602 00:35:20,880 --> 00:35:24,360 Speaker 1: to point out that Phil's forecast did not contain a 603 00:35:24,480 --> 00:35:30,840 Speaker 1: probability oh huh. Even though he did a pretty uncontroversial forecast, 604 00:35:31,680 --> 00:35:35,600 Speaker 1: then he didn't put a precise number on it. But 605 00:35:35,760 --> 00:35:38,080 Speaker 1: I really, I really like that conversation. I think it's 606 00:35:38,239 --> 00:35:41,080 Speaker 1: just a great topic because we see this all the 607 00:35:41,160 --> 00:35:43,279 Speaker 1: time and not just in the fact that people come 608 00:35:43,320 --> 00:35:47,600 Speaker 1: on give forecasts and never are really held accountable for them, 609 00:35:47,600 --> 00:35:50,000 Speaker 1: but just in what is the purpose of forecast? And 610 00:35:50,239 --> 00:35:54,359 Speaker 1: how many times we all encounter people who give forecasts 611 00:35:54,719 --> 00:35:58,000 Speaker 1: but whose job is clearly not accuracy, and I'm thinking 612 00:35:58,000 --> 00:35:59,080 Speaker 1: a lot of their I think there are a lot 613 00:35:59,120 --> 00:36:02,319 Speaker 1: of asset manager is like this who use stories as 614 00:36:02,320 --> 00:36:05,719 Speaker 1: a way of gathering clients, and maybe they tell a 615 00:36:05,719 --> 00:36:09,120 Speaker 1: bearished story or a conspiratorial story about the fed or whatever, 616 00:36:09,640 --> 00:36:12,080 Speaker 1: but that story really has nothing to do with how 617 00:36:12,120 --> 00:36:16,080 Speaker 1: they then go on and invest. Right, certainly in the 618 00:36:16,120 --> 00:36:19,319 Speaker 1: investment industry, there are plenty of people whose positions just 619 00:36:19,480 --> 00:36:22,759 Speaker 1: don't add up to the world viewpoint that they tend 620 00:36:22,800 --> 00:36:25,440 Speaker 1: to express, which you know, again, I would think of 621 00:36:25,440 --> 00:36:28,319 Speaker 1: a lot of the really bearished people out there who 622 00:36:28,440 --> 00:36:30,640 Speaker 1: for the past eight years have been saying, you know, 623 00:36:30,719 --> 00:36:33,960 Speaker 1: move into cash by gold, the end is coming, and 624 00:36:34,080 --> 00:36:36,880 Speaker 1: yet clearly they are in the business of putting cash 625 00:36:36,920 --> 00:36:39,560 Speaker 1: to work in some way or another. So you know, 626 00:36:40,040 --> 00:36:42,719 Speaker 1: it seems a little bit out of sync. You know, 627 00:36:42,840 --> 00:36:44,719 Speaker 1: we didn't you know, we only talked a little bit. 628 00:36:44,760 --> 00:36:48,000 Speaker 1: And I'm sure if we read his work, uh and 629 00:36:48,239 --> 00:36:51,719 Speaker 1: studied it, there'll be more depth, but in terms of 630 00:36:51,760 --> 00:36:55,919 Speaker 1: becoming a better forecaster, this idea of just sort of 631 00:36:56,520 --> 00:37:00,640 Speaker 1: looking at the the the default and what point is well, 632 00:37:00,719 --> 00:37:04,040 Speaker 1: coups are pretty rare, and wars are pretty rare, and 633 00:37:04,200 --> 00:37:08,239 Speaker 1: regime change is pretty rare, and sort of starting there, 634 00:37:08,239 --> 00:37:10,520 Speaker 1: I mean, another thing in stock markets is like bear 635 00:37:10,600 --> 00:37:14,319 Speaker 1: markets are really rare, and stock market crashes are really rare. 636 00:37:15,000 --> 00:37:18,400 Speaker 1: And so this sort of idea of like, as you 637 00:37:18,440 --> 00:37:20,759 Speaker 1: put in the beginning, and as we're talking about we're 638 00:37:20,800 --> 00:37:23,839 Speaker 1: at a moment in which lots of people are talking 639 00:37:23,840 --> 00:37:27,120 Speaker 1: about inflection points and it feels like things might turn. 640 00:37:28,000 --> 00:37:30,880 Speaker 1: But just starting from this assumption that we're probably not 641 00:37:31,040 --> 00:37:33,480 Speaker 1: an inflection point and that most of these turns that 642 00:37:33,560 --> 00:37:36,200 Speaker 1: we think are they're bound to happen now probably won't 643 00:37:36,200 --> 00:37:39,400 Speaker 1: happen now, it's like it's an interesting starting point. I mean, 644 00:37:39,440 --> 00:37:42,200 Speaker 1: obviously that's not enough, because sometimes the turns do happen, 645 00:37:42,760 --> 00:37:46,400 Speaker 1: but starting from that assumption, it's like that you're probably 646 00:37:46,440 --> 00:37:49,640 Speaker 1: the right move is to fade the expectations of a turn. 647 00:37:50,239 --> 00:37:53,399 Speaker 1: See it feels like an interesting toll hole to get 648 00:37:53,440 --> 00:37:57,640 Speaker 1: into two then move from there, right, But see, I 649 00:37:57,680 --> 00:38:00,960 Speaker 1: think this is where human psychology and references come in, 650 00:38:01,040 --> 00:38:04,080 Speaker 1: because no one is going to remember or reward you 651 00:38:04,280 --> 00:38:08,319 Speaker 1: for continuously calling the status quo correctly, but they will 652 00:38:08,400 --> 00:38:12,840 Speaker 1: probably remember you if you did call the big regime 653 00:38:12,920 --> 00:38:15,560 Speaker 1: shift or the big bear market. And that's why so 654 00:38:15,600 --> 00:38:19,160 Speaker 1: many people remember quite a few names that predicted the 655 00:38:19,160 --> 00:38:23,360 Speaker 1: two thousand and eight financial crisis, but no one remembers 656 00:38:23,360 --> 00:38:26,800 Speaker 1: as clearly, you know, people who called the gigantic rally 657 00:38:26,880 --> 00:38:30,680 Speaker 1: that we had after two thousand nine the only you know, 658 00:38:30,880 --> 00:38:33,200 Speaker 1: the only one, and like an exception to someone like 659 00:38:33,200 --> 00:38:36,480 Speaker 1: Warren Buffett who just buy stocks and doesn't do anything 660 00:38:36,520 --> 00:38:40,160 Speaker 1: fancy and has done so there's no seriously, like there's 661 00:38:40,160 --> 00:38:42,360 Speaker 1: a few people like that. I feel like Warren Buffett 662 00:38:42,440 --> 00:38:44,839 Speaker 1: is always the exception. Yeah, that's true. Like you can't 663 00:38:44,880 --> 00:38:49,200 Speaker 1: just point to Buffett, but that's true. All right. Well, 664 00:38:49,200 --> 00:38:52,359 Speaker 1: on that note, h this has been another episode of 665 00:38:52,400 --> 00:38:55,400 Speaker 1: the ad Thoughts podcast. I'm Tracy Allaway. You can follow 666 00:38:55,400 --> 00:38:58,440 Speaker 1: me on Twitter at Tracy Alloway and I'm Joe Why 667 00:38:58,480 --> 00:39:02,040 Speaker 1: Isn't Though? You can follow me on Twitter at the Stalwart. 668 00:39:02,440 --> 00:39:05,880 Speaker 1: You should follow our guest on Twitter, Phillip Tetlock. He's 669 00:39:06,040 --> 00:39:09,520 Speaker 1: at p Tetlock, and you should follow our producer on Twitter, 670 00:39:09,600 --> 00:39:12,920 Speaker 1: Laura Carlson. She's at Laura M. Carlson, as well as 671 00:39:12,920 --> 00:39:17,200 Speaker 1: the Bloomberg head of podcasts, Francesca Levi at Francesca Today. 672 00:39:17,400 --> 00:39:21,120 Speaker 1: And be sure to check out all of Bloomberg's podcasts 673 00:39:21,160 --> 00:39:25,800 Speaker 1: on Twitter under the handle at podcasts. Thanks for listening.