WEBVTT - How to Forecast the Future 0:00:07.800 --> 0:00:11.000 Hello, Odd Lots listeners, It's Tracy Alloway. As you may know, 0:00:11.119 --> 0:00:14.720 Odd Lots is hosting its first ever live event on Thursday, 0:00:14.720 --> 0:00:17.800 September nineteen in New York City. Join me and Joe 0:00:17.800 --> 0:00:20.759 Wisenthal as we host an evening of great conversation and 0:00:20.920 --> 0:00:24.080 live music. The Odd Blots Variety Show will feature some 0:00:24.160 --> 0:00:28.560 new and old Odd Lots guests, including the economist Stephanie Kelton, 0:00:28.960 --> 0:00:32.720 Sam Antar, the former Crazy Eddie CFO, and convicted Felon 0:00:32.960 --> 0:00:35.879 will give us some of his best stories. Meanwhile, Lee 0:00:35.960 --> 0:00:39.879 book Eye, once dubbed the philosopher King of Sovereign Debt Lawyers, 0:00:40.000 --> 0:00:43.199 is coming out of retirement for us. Salton, posar of 0:00:43.240 --> 0:00:45.839 Credit Swiss and formerly of the U. S. Treasury, will 0:00:45.880 --> 0:00:49.040 be on stage with Brad Setzer, Senior Fellow at the 0:00:49.080 --> 0:00:53.360 Council for Foreign Relations, talking all things bonds and trade. 0:00:53.680 --> 0:00:57.040 Will also have some markets the music courtesy of Moral Hazard, 0:00:57.160 --> 0:01:00.360 the most important country singer in economics, and an early 0:01:00.400 --> 0:01:03.520 Odd Thoughts guest, and even Joe has promised us a 0:01:03.600 --> 0:01:06.600 song or two. You can stick around after the show 0:01:06.680 --> 0:01:08.840 for some drinks and a chance to chat with us 0:01:08.840 --> 0:01:11.800 and our guests in person, so keep listening to Odd 0:01:11.840 --> 0:01:13.920 Lots to learn how to sign up to see us 0:01:14.000 --> 0:01:38.160 live on Thursday, September nineteenth. We hope to see you there. Hello, 0:01:38.240 --> 0:01:41.280 and welcome to another episode of the Odd Loots podcast. 0:01:41.360 --> 0:01:46.080 I'm Tracy Halloway and I'm Joe wisnal So Joe. I 0:01:46.120 --> 0:01:49.000 feel and I think we've discussed this before, but it 0:01:49.040 --> 0:01:52.280 feels like the world is sort of at an inflection 0:01:52.360 --> 0:01:56.680 point right now. Uh you think, I mean you think so? Yeah. 0:01:57.280 --> 0:02:00.280 Isn't it always kind of an inflection point? Yeah, guess 0:02:00.280 --> 0:02:03.360 that's true. But at least in markets, it feels like, 0:02:03.480 --> 0:02:06.560 you know, warnings that were late in the cycle that 0:02:06.640 --> 0:02:08.720 we could get a recession at the end of this 0:02:08.800 --> 0:02:12.320 year or in those definitely seem to be heating up. 0:02:13.760 --> 0:02:16.800 I think what I would say and where maybe we 0:02:16.880 --> 0:02:19.639 would uh most likely agree, is not that the world 0:02:19.720 --> 0:02:22.680 is that at an inflection point per se, but that 0:02:22.840 --> 0:02:26.880 what comes and goes is periods when suddenly people feel 0:02:26.919 --> 0:02:28.720 the turn is about to come. And we've had a 0:02:28.800 --> 0:02:32.040 series of these over the last ten years, whether it 0:02:32.160 --> 0:02:37.320 was Q four of last year early with the economy 0:02:37.360 --> 0:02:40.600 going down, the euro crisis. From time to time, it's 0:02:40.639 --> 0:02:44.440 like there's this global collective anxiety that rises, and whether 0:02:44.480 --> 0:02:47.000 it's anymore real or not is up for debate, but 0:02:47.080 --> 0:02:49.560 I would absolutely agree right now you're getting a lot 0:02:49.600 --> 0:02:53.880 of like recession calls, bear market calls, started the easy 0:02:53.960 --> 0:02:57.600 cycle calls, things like that, A wave of global anxiety. 0:02:57.680 --> 0:02:59.839 That's a good way to put it. And this isn't really, 0:03:00.320 --> 0:03:02.480 as you mentioned, the first time that we've seen this, 0:03:02.680 --> 0:03:06.600 and I'm not just talking about markets. We've seen sort 0:03:06.639 --> 0:03:10.880 of inflection points happen in politics recently, right, So Brexit 0:03:11.000 --> 0:03:15.040 springs to mind, the election of Donald Trump as well. Yeah, 0:03:15.080 --> 0:03:17.079 and I think the fact that we have another election 0:03:17.280 --> 0:03:20.480 coming up in the US again, it's one of these 0:03:20.520 --> 0:03:24.440 points where people wanted to call some sort of meaningful 0:03:24.560 --> 0:03:28.079 turn in something or or there's no doubt we're we're 0:03:28.080 --> 0:03:31.400 hearing a lot of that these days. Right. So all 0:03:31.440 --> 0:03:33.639 of this is a roundabout way of saying that we're 0:03:33.680 --> 0:03:36.520 getting a lot of forecasts and a lot of predictions, 0:03:36.640 --> 0:03:39.160 a lot of people trying to call or see it 0:03:39.200 --> 0:03:42.200 into the future. And so I thought it would probably 0:03:42.240 --> 0:03:45.520 be a good idea to do an episode on forecasting. 0:03:46.840 --> 0:03:48.680 I love this idea. I mean, one thing that I've 0:03:48.680 --> 0:03:51.280 always wanted to do and never done? Is you know, 0:03:51.800 --> 0:03:55.360 obviously on TV I talked to people all the time 0:03:55.520 --> 0:03:57.960 and they make forecasts like, oh, we're in this stock 0:03:58.160 --> 0:04:01.280 or interested in this sector, or we're telling clients to 0:04:01.280 --> 0:04:03.880 do this, and I've never gone back and it would 0:04:03.880 --> 0:04:05.920 just be too much work for me, and like actually 0:04:05.960 --> 0:04:09.200 made a database of all their calls. But I've always 0:04:09.240 --> 0:04:12.040 thought that's like a fascinating project or that you know 0:04:12.200 --> 0:04:14.760 that because who knows everybody? You know, people just make 0:04:14.800 --> 0:04:17.839 these calls and how often do they ever get revisited 0:04:17.880 --> 0:04:21.480 to see if the person was actually right or useful 0:04:21.480 --> 0:04:24.440 in some way? Totally, And I have a feeling that 0:04:24.480 --> 0:04:26.719 our guests for this episode is going to have something 0:04:26.760 --> 0:04:30.240 to say on that particular topic. So without further ado, 0:04:30.279 --> 0:04:32.599 why don't I go ahead and bring him in. Our 0:04:32.640 --> 0:04:37.600 guest is Phil Tetlock. He is Annenberg University Professor at 0:04:37.600 --> 0:04:41.800 the University of Pennsylvania. He's also the author of numerous 0:04:41.880 --> 0:04:46.400 books and papers on forecasting. So, Phil, welcome to the show. 0:04:47.560 --> 0:04:50.240 I'm glad to be here. So Phil, I guess my 0:04:50.320 --> 0:04:53.440 first question is, you know, one of the thrust of 0:04:53.480 --> 0:04:56.400 a lot of your work is that experts tend to 0:04:56.480 --> 0:05:00.760 get forecasting wrong. So is it weird? It doesn't feel 0:05:00.760 --> 0:05:05.360 weird that you're sort of the expert on why experts fail. Um. 0:05:05.640 --> 0:05:08.240 I guess I've gotten accustomed to it because I've been 0:05:08.279 --> 0:05:11.320 doing it for about thirty five years now. I started 0:05:11.320 --> 0:05:16.640 out just after I got tenure at Berkeley and been 0:05:16.800 --> 0:05:21.880 tracking the accuracy of experts predictions pretty much continuously since then. 0:05:22.200 --> 0:05:25.880 So I'm sometimes called an expert hologist, which is, of 0:05:25.880 --> 0:05:29.479 course the field that does not does not exist. The 0:05:29.520 --> 0:05:32.640 basic idea of keeping track of of how accurate people 0:05:32.680 --> 0:05:35.160 are is it? Is it really good idea? And and 0:05:35.360 --> 0:05:40.000 insofar as we all kept track of ourselves and the forecasting, 0:05:40.080 --> 0:05:43.159 making how well calibrated we are, I think the research 0:05:43.200 --> 0:05:46.040 suggests we would actually get a little bit better at it. 0:05:47.000 --> 0:05:50.200 So you've actually done what I've said for years that 0:05:50.240 --> 0:05:54.080 I've wanted to do, which is creates some database of 0:05:54.160 --> 0:05:56.640 forecasts and actually go back and look at who's right 0:05:56.680 --> 0:06:00.360 and who's wrong. How did you get the thought that 0:06:00.360 --> 0:06:03.440 that would be worth doing initially? And what does such 0:06:03.480 --> 0:06:09.080 a database look like? Because most forecasts are not particularly binary. 0:06:09.240 --> 0:06:13.680 People might say I chance or seventy five chance, so 0:06:13.760 --> 0:06:16.960 usually things aren't just this will happen or this won't happen. 0:06:17.480 --> 0:06:21.920 And often a forecast can be wrong, but the methodology 0:06:22.040 --> 0:06:26.320 turned out to be right, or maybe someone unpredicted chance 0:06:26.360 --> 0:06:29.040 and that was still the right framework and the five 0:06:29.120 --> 0:06:32.400 percent odds did hit. So how do you go even 0:06:32.440 --> 0:06:36.039 go about constructing a database of forecasts and measuring what 0:06:36.120 --> 0:06:38.839 turned out to be right or wrong? Well, that's what 0:06:38.920 --> 0:06:42.000 ten ures for it takes. It takes a long time. 0:06:42.720 --> 0:06:46.800 The key thing to note is that your right that 0:06:46.880 --> 0:06:49.440 there are only two conditions under which you can definitively 0:06:49.480 --> 0:06:52.359 say at particular forecast was right or wrong, and that 0:06:52.520 --> 0:06:54.400 is that the forecast it was rash enough to say 0:06:54.400 --> 0:06:57.479 a hundred percent chance and it didn't happen, or zero 0:06:57.520 --> 0:06:59.680 percent chance that it did happen, in which case you 0:06:59.680 --> 0:07:03.080 know inclusively that that specific forecast was wrong. But a 0:07:03.160 --> 0:07:05.680 forecaster says, you know, there's a seventy percent chance, like 0:07:05.760 --> 0:07:08.599 Nate Silver said there's a seventy percent chance of Hillary 0:07:08.640 --> 0:07:13.800 Clinton winning the election in November? Uh was was was 0:07:13.920 --> 0:07:16.760 Nate Silver wrong? Or or do we happen to inhabit 0:07:16.760 --> 0:07:21.920 a world that was likely on November first? So the 0:07:22.160 --> 0:07:25.480 solution to the problem is statistical. You need you need 0:07:25.520 --> 0:07:29.040 to keep track of lots of forecasts over time. If 0:07:29.040 --> 0:07:31.440 we collect hundreds of of your forecasts over the course 0:07:31.480 --> 0:07:34.760 of a year, and we find that when you say 0:07:34.800 --> 0:07:37.320 there's a seventy percent chance of things happening, those things 0:07:37.320 --> 0:07:39.440 happen about seventy percent of the time. We say there's 0:07:39.440 --> 0:07:41.800 a ninety percent chance those things happen about ninety percent 0:07:41.880 --> 0:07:44.240 of the time, and so forth. If there's a close 0:07:44.280 --> 0:07:48.240 correspondence between your subjective probability estimates and the object of 0:07:48.320 --> 0:07:51.040 frequency with which events occur, you can be said to 0:07:51.040 --> 0:07:55.200 be reasonably well calibrated. And that's a very desirable feature 0:07:55.200 --> 0:07:59.560 of forecasters. So when you run your analysis, your statistical 0:07:59.560 --> 0:08:03.600 analysy says, of all these expert forecasts, and I'm assuming 0:08:03.600 --> 0:08:06.560 some non expert forecasts as well, what did you find? 0:08:07.200 --> 0:08:09.320 You find that people are not very well calibrated. For 0:08:09.360 --> 0:08:13.320 a start, there's a lot of overconfidence. And when uh, 0:08:13.920 --> 0:08:16.040 many many experts would would would say things are an 0:08:16.040 --> 0:08:18.920 eighty or ninety percent likely or even likely, they would 0:08:18.960 --> 0:08:22.120 occur sixty or seventy percent of the time. So there 0:08:22.120 --> 0:08:25.800 were big gaps between subject of probabilities and object of probabilities. 0:08:25.800 --> 0:08:29.800 There's a lot of overconfidence. People were not qualifying their 0:08:29.840 --> 0:08:34.880 forecast appropriately, and that is one of the better replicated 0:08:34.920 --> 0:08:39.880 findings in my field, which is cognitive social psychology of judgment. 0:08:40.760 --> 0:08:43.760 People are over confident, that tend to be over confident. 0:08:43.920 --> 0:08:45.800 Everybody's over confident. There are even there are a few 0:08:45.800 --> 0:08:49.000 souls who were even systematically under confident and who don't 0:08:49.000 --> 0:08:52.120 have enough confidence in their judgment. But the modal, if 0:08:52.120 --> 0:08:54.160 you had to bet on what kind of mistake people 0:08:54.200 --> 0:08:57.680 are making at any given moment, over confidence would be 0:08:57.679 --> 0:09:01.320 the better bet. You know, I think about something. One 0:09:01.360 --> 0:09:04.400 of our colleagues, I'll give a shout out to him, 0:09:04.520 --> 0:09:08.040 Lorc and Roch Kelly. He works with us here at Bloomberg, 0:09:08.080 --> 0:09:13.559 and he's very fond of strategists at banks and other 0:09:14.200 --> 0:09:18.760 experts who love to give a forecast on things like, oh, 0:09:18.840 --> 0:09:23.720 we see a forty percent forecast of this person winning 0:09:23.760 --> 0:09:27.000 the election, or forty percent forecast of this country leaving 0:09:27.040 --> 0:09:29.440 the EU in the next five years, and it's like 0:09:29.480 --> 0:09:32.640 the perfect number because it's you know, it's still on. 0:09:33.000 --> 0:09:36.480 It feels unlikely, but it's close enough to that if 0:09:36.520 --> 0:09:38.720 it happens, you know, you can still say, oh, I 0:09:38.720 --> 0:09:40.760 told you it was significant. But if it doesn't happen, 0:09:40.880 --> 0:09:43.960 you could say it was unlikely. And I'm curious in 0:09:44.080 --> 0:09:47.600 your findings and in your research. You know, we think 0:09:47.600 --> 0:09:52.080 of forecasts, our predictions is the point is to be accurate. 0:09:52.120 --> 0:09:54.640 But it feels like a lot of the reason people 0:09:54.920 --> 0:09:57.560 give forecast is just to be interesting, just to have 0:09:57.600 --> 0:09:59.719 their voice heard, just to get their clients to pick 0:09:59.760 --> 0:10:02.920 up the phone. And I'm curious how that plays into 0:10:02.960 --> 0:10:06.160 your analysis of forecasts when the whole purpose is maybe 0:10:06.160 --> 0:10:08.960 not even to get it right, it's just to provoke 0:10:09.000 --> 0:10:10.880 a thought or to have your name out in the news. 0:10:11.520 --> 0:10:15.439 I think that's a very perceptive observation. It's a very 0:10:15.480 --> 0:10:18.640 delicate dance that people play. On the one hand, you 0:10:18.679 --> 0:10:21.320 want to say things that sound interesting so people don't 0:10:21.400 --> 0:10:22.920 roll their eyes and things that this is boring and 0:10:22.960 --> 0:10:25.439 a useless conversation. On the other hand, you don't want 0:10:25.440 --> 0:10:27.280 to say things that are so interesting that they could 0:10:27.280 --> 0:10:32.199 prove to be wrong later. So you're there seems to 0:10:32.240 --> 0:10:34.400 be sort of in the sweet spot zone. Or if 0:10:34.400 --> 0:10:36.120 you were to use language, you say, well, I think 0:10:36.120 --> 0:10:38.880 there's a distinct possibility that Putin's next move is going 0:10:38.920 --> 0:10:42.599 to be on Belarus or on Estonia. Uh, it's a 0:10:42.640 --> 0:10:45.920 distinct possibility, is wonderful, and exactly the same way is 0:10:45.920 --> 0:10:48.360 pretty good. I mean, if it happens, I say, hey, 0:10:48.400 --> 0:10:51.440 I told you distinct possibility, and if it doesn't happen, 0:10:51.440 --> 0:10:54.000 I said, I merely said it was possible. So you 0:10:54.000 --> 0:10:56.679 you're you're, you're covering yourself very nicely. It is as 0:10:56.720 --> 0:11:01.160 though the art of punditry is the art of appearing 0:11:01.160 --> 0:11:03.320 to go out on a limb without actually going out 0:11:03.320 --> 0:11:09.480 on a limb. So, I mean, Joe referred to forecasts 0:11:09.480 --> 0:11:12.600 as non binary. But I feel like instinctively a lot 0:11:12.600 --> 0:11:15.520 of people want to know whether something will or will 0:11:15.600 --> 0:11:19.160 not happen, and yet we have all these, uh, forecasting 0:11:19.240 --> 0:11:21.480 calls that are sort of, you know, thirty percent chance, 0:11:21.559 --> 0:11:24.920 forty percent chance. Our probability is a cop out in 0:11:24.960 --> 0:11:27.839 that case, is it's something that people hide under. Well, 0:11:27.840 --> 0:11:31.480 probable probabilities are not a cop out. If you're participating 0:11:31.600 --> 0:11:35.640 in forecasting tournaments in which we can systematically track how 0:11:35.679 --> 0:11:39.920 often you're things happen, and if things you say likely 0:11:39.960 --> 0:11:42.560 happen forty of the time, you're pretty well calibrated, and 0:11:42.559 --> 0:11:46.000 it's not simply being used as a cop out. So, okay, 0:11:46.040 --> 0:11:50.400 So you've built this database and you've been tracking forecasters 0:11:50.400 --> 0:11:54.319 ability for years, and you mentioned that there are forecasting 0:11:54.360 --> 0:11:56.520 tournaments and we can really track this stuff, and that 0:11:56.600 --> 0:12:00.120 we can track how well calibrated forecasters are, not on 0:12:00.200 --> 0:12:03.720 any sort of individual prediction, but by whether over time 0:12:04.320 --> 0:12:08.320 their predictions of likelihood events happened seven out of ten 0:12:08.360 --> 0:12:11.319 times and so forth. What are some of the interesting 0:12:11.400 --> 0:12:16.400 patterns you've discovered besides that people tend to be overconfident, 0:12:16.840 --> 0:12:20.680 what kind of people, what kind of approaches tend to 0:12:20.720 --> 0:12:24.520 distinguish the better forecasters from the worst ones, Because ultimately, 0:12:24.520 --> 0:12:26.920 I think that's sort of the point of your research. 0:12:27.480 --> 0:12:30.199 There are there are two classic biases that we have 0:12:30.320 --> 0:12:32.880 found over the years, and one of them is that 0:12:32.960 --> 0:12:35.760 people are too quick to make up their minds, and 0:12:35.800 --> 0:12:38.040 the other is that people are too slow to change them. 0:12:38.320 --> 0:12:41.559 And it's the combination of those two things that causes 0:12:41.679 --> 0:12:47.360 chronic over confidence. So I'm curious, beyond sort of individual 0:12:47.520 --> 0:12:51.080 characteristics that make people a good or bad forecaster, did 0:12:51.080 --> 0:12:55.800 you notice any discrepancy in the type of forecasts being made, Like, 0:12:55.880 --> 0:13:00.520 for instance, did political forecasting tend to be more or 0:13:00.600 --> 0:13:06.320 less accurate than something like economic or financial forecasting. Oh, 0:13:06.360 --> 0:13:09.240 it really depends on the timeframe and and the and 0:13:09.360 --> 0:13:12.480 the types of questions you're asking. I think economic and 0:13:12.520 --> 0:13:16.040 political forecasting can be pretty hazardous to your reputation. I 0:13:16.080 --> 0:13:19.040 think what we noticed more than anything was that the 0:13:19.400 --> 0:13:22.320 types of forecasters who tended to be better did tend 0:13:22.360 --> 0:13:24.720 to be a little bit more boring. There they were 0:13:24.760 --> 0:13:26.720 more likely to say on the one hand, on the 0:13:26.720 --> 0:13:28.520 other hand, they were they were, they were, they were 0:13:28.520 --> 0:13:31.240 engaging in more explicit balancing and say, well, there's this 0:13:31.320 --> 0:13:33.120 causal for us, and there's that causal for us, and 0:13:33.120 --> 0:13:35.280 you have to balance them against each other. So the 0:13:35.600 --> 0:13:40.040 types of forecasting talk that make forecasters appealing to the 0:13:40.080 --> 0:13:43.040 media tend to details that tend to make forecasts less 0:13:43.160 --> 0:13:45.839 less accurate. So a forecaster is going to be more 0:13:45.840 --> 0:13:48.080 appealing to the media, it would seem if they if 0:13:48.080 --> 0:13:50.000 they can come up with a compelling sound bite and 0:13:50.040 --> 0:13:52.120 they say something like, well, you know, I think the 0:13:52.440 --> 0:13:55.040 Saudi regime is going to collapse within the next twelve 0:13:55.080 --> 0:13:57.880 to twenty four months. That's that's a that's a very 0:13:58.440 --> 0:14:01.280 dramatic forecast. It would have a lot of consequences for 0:14:01.360 --> 0:14:05.000 the Middle East and from World Politics. A forecast who says, well, 0:14:05.040 --> 0:14:07.000 you know, there are people have been predicting the major 0:14:07.040 --> 0:14:09.640 regime change in Saudi Arabia off and on for the 0:14:09.679 --> 0:14:14.000 last forty years. It hasn't happened yet. The base rate 0:14:14.040 --> 0:14:17.280 prediction is it is it's not very likely. Um, there 0:14:17.280 --> 0:14:21.720 are some reasons for some concern, but you know, you 0:14:21.760 --> 0:14:24.120 can steal your eyes start to glaze over. Listening to 0:14:24.400 --> 0:14:28.680 the more accurate forecasters tend to to bore people. Yeah, 0:14:28.840 --> 0:14:32.000 So I'm just thinking so at the time that we're 0:14:32.040 --> 0:14:36.320 recording this episode, just in the last day, and by 0:14:36.320 --> 0:14:38.320 the time you're hearing this, this would be old news. 0:14:38.400 --> 0:14:41.600 But in the last day, uh Deutsche Bank, for example, 0:14:41.680 --> 0:14:45.960 announced a major restructuring of its bank. And I'm just 0:14:46.040 --> 0:14:48.800 thinking about, how like the imperative for the news media 0:14:49.280 --> 0:14:53.360 is immediately to find people who will come on this 0:14:53.440 --> 0:14:57.320 morning and say something about whether this restructuring of the 0:14:57.360 --> 0:14:59.840 bank is likely to be enough, did it go far enough, 0:15:00.120 --> 0:15:04.000 will restore the bank to robust profitability and so forth, 0:15:04.640 --> 0:15:08.680 And it sounds like that imperative is almost exactly the 0:15:08.680 --> 0:15:11.960 opposite of what's likely to make a good forecaster. And 0:15:12.080 --> 0:15:14.800 anyone who already has their mind made up but already 0:15:14.840 --> 0:15:18.240 has a strong view on the efficacy of the plan, 0:15:19.120 --> 0:15:22.800 at least going by your heuristic that the people that 0:15:22.920 --> 0:15:27.280 good forecasting is not correlated with a quick judgment or 0:15:27.680 --> 0:15:31.000 you know, quick decision making, which seems like we're kind 0:15:31.000 --> 0:15:34.000 of like highlighting most likely highlighting some of the worst 0:15:34.000 --> 0:15:37.040 people we could be highlighting. Well, it really you have 0:15:37.120 --> 0:15:39.280 to make a decision about what kind of business you're in. 0:15:39.400 --> 0:15:42.160 If you're in the accuracy business, you're going to look 0:15:42.200 --> 0:15:44.800 for the kinds of forecasters we've been looking for in 0:15:44.840 --> 0:15:47.280 the work we've been doing with the intelligence community and 0:15:47.320 --> 0:15:50.520 elsewhere where. These are going to be forecasters who are 0:15:50.560 --> 0:15:53.120 not very entertaining. If you're in the entertainment business, you're 0:15:53.120 --> 0:15:55.200 going to be looking for people who are entertaining. There's 0:15:55.240 --> 0:15:59.720 a separation in Bloomberg and probably mostly another sophisticated media 0:15:59.760 --> 0:16:03.000 comes these between analytics and and and the front end. Right, 0:16:21.800 --> 0:16:25.680 So what about experience, Like to what degree can if 0:16:25.760 --> 0:16:29.080 you're an expert, presumably you have you know, probably decades 0:16:29.120 --> 0:16:32.080 of experience, and you know you've been studying a particular 0:16:32.120 --> 0:16:35.080 subject matter for a long time, you've noticed patterns, or 0:16:35.120 --> 0:16:38.760 you can reach for historical analogies to describe a current 0:16:38.800 --> 0:16:44.480 situation and extrapolate from that. Does experience help offset the 0:16:44.600 --> 0:16:49.440 problems of overconfidence at all? Sometimes it depends what you 0:16:49.480 --> 0:16:53.760 do with the experience. Um. People have different styles of 0:16:53.800 --> 0:16:58.160 thinking and and some people with experience become extremely skilled 0:16:58.680 --> 0:17:02.920 at creating very comp telling, very articulate justifications why they 0:17:03.000 --> 0:17:07.440 must be right. Uh. So experience can actually solidify dogmatism 0:17:07.560 --> 0:17:11.000 for people with that cognitive style, and either for other people, 0:17:11.160 --> 0:17:14.320 experience melos of them and they become more tuned to 0:17:14.359 --> 0:17:18.200 the limitations of their prior world views and uh, they 0:17:18.320 --> 0:17:22.520 introduce more appropriate qualifications, may become better calibrated. So but 0:17:22.600 --> 0:17:26.840 but it's not a one trajectory. Be people people mature 0:17:27.080 --> 0:17:30.919 in different ways. Talk to us about how you train 0:17:31.080 --> 0:17:34.879 people to get better. So obviously, like let's start with 0:17:34.960 --> 0:17:38.119 the assumption that there are some people that aren't just 0:17:38.200 --> 0:17:41.639 looking for media sound bites and maybe as uh to 0:17:41.760 --> 0:17:45.640 use your example there in the intelligence community, and they 0:17:45.680 --> 0:17:49.400 really want to make better forecasts about how things will 0:17:50.000 --> 0:17:52.360 happen in the future. They want to be better at 0:17:52.359 --> 0:17:56.680 predicting save Vladimir Putin's next move? What is that? How 0:17:56.680 --> 0:17:58.920 do you start and how do you what's the general 0:17:58.920 --> 0:18:02.840 approach to becoming better at that? Well, I would say 0:18:03.040 --> 0:18:05.400 the starting point is again not going to be all 0:18:05.440 --> 0:18:09.560 that exciting and it and its bears a strong resemblance 0:18:09.600 --> 0:18:12.600 to what Danny Kahneman proposes in his best selling book 0:18:12.680 --> 0:18:16.080 Thinking Fast and Slow. Uh, it is start with the 0:18:16.119 --> 0:18:18.560 base rates. And if you look at the base rates, 0:18:18.840 --> 0:18:22.119 you you'll see something quite interesting, and that is people 0:18:22.240 --> 0:18:25.480 frequently claim, to go back to the beginning of our conversation, 0:18:25.880 --> 0:18:28.679 people frequently claim that they're at an inflection point in history. 0:18:29.280 --> 0:18:31.280 If you if you look at how many inflection points 0:18:31.280 --> 0:18:34.040 there have been, it's just a very long list. The 0:18:34.119 --> 0:18:37.679 vast majority of claims about inflection points have been false positives. 0:18:38.200 --> 0:18:41.439 So you you would, naturally, I think, be wary of 0:18:41.560 --> 0:18:45.119 claims about inflection points. Another claim you'd be wary of 0:18:45.320 --> 0:18:49.080 is military coups, the revolutions. They're relatively rare events. So 0:18:49.200 --> 0:18:51.520 someone who's making a dramatic claim but it's going to 0:18:51.600 --> 0:18:54.080 be a regime change in a particular country within a 0:18:54.119 --> 0:18:58.920 particular timeframe, the likelihood their being right is pretty low. Still. 0:18:59.240 --> 0:19:02.959 Joe mentioned us in the intro, but what role do 0:19:03.000 --> 0:19:07.920 you think accountability plays in the sort of forecasting industry 0:19:08.000 --> 0:19:12.879 because it feels to me like, given the volume of 0:19:13.040 --> 0:19:15.800 media that's out there right now, you know, either social 0:19:15.840 --> 0:19:19.679 media or traditional forms of media, it feels like you 0:19:19.720 --> 0:19:21.719 have a lot of people who will make, you know, 0:19:21.800 --> 0:19:25.080 say a hundred predictions, and maybe one or two of 0:19:25.119 --> 0:19:28.240 those are right, and then they get trumpeted for those 0:19:28.320 --> 0:19:31.560 right calls, or you know, they laud themselves for those 0:19:31.560 --> 0:19:35.000 correct calls, and people sort of forget about the other 0:19:35.160 --> 0:19:38.600 nine calls that were wrong, and no one bothers to 0:19:38.640 --> 0:19:41.159 go back and check on them. Because there's just so 0:19:41.600 --> 0:19:45.399 much forecasting and so much information out there in the world. 0:19:45.560 --> 0:19:51.679 So how can we develop accountability for forecasting by running 0:19:51.680 --> 0:19:56.919 forecasting tournaments? Um? What prediction markets are? Forecasting tournaments are 0:19:56.960 --> 0:20:01.040 excellent ways of allowing people to tract their accuracy on 0:20:01.240 --> 0:20:05.200 judgment calls for which there aren't ready financial market equivalence. 0:20:05.640 --> 0:20:08.040 How do these I've never I'm not familiar with these, 0:20:08.040 --> 0:20:11.120 So how does they are forecasting tournament work? You ask 0:20:11.200 --> 0:20:16.480 people to put subjective probabilities on events that are specified 0:20:16.480 --> 0:20:19.760 by well defined questions, such as whether Prutent is going 0:20:19.800 --> 0:20:25.240 to be the president of Russia after and those probability 0:20:25.280 --> 0:20:27.639 either in that in that case, it would be a 0:20:27.640 --> 0:20:31.119 probability that would be yes or no, and it should 0:20:31.160 --> 0:20:33.840 sum up to one point zero. There are lots of 0:20:33.840 --> 0:20:35.840 ways of doing it, but they all boiled down to 0:20:35.920 --> 0:20:39.960 the core idea, which is which is keeping score. Sticking 0:20:40.080 --> 0:20:44.600 with the intelligence community framework, how does the role of 0:20:44.800 --> 0:20:48.520 group think play into this and avoiding group think? Because 0:20:48.520 --> 0:20:51.120 if we think back to what are considered a lot 0:20:51.160 --> 0:20:55.880 of the intelligence disasters over the last several years, the 0:20:55.960 --> 0:20:59.800 idea of some idea takes hold and no one feels 0:21:00.440 --> 0:21:04.159 comfortable yelling stop, and suddenly everyone can deals on the 0:21:04.240 --> 0:21:07.600 same idea. Is that something that in your work you 0:21:07.680 --> 0:21:10.960 focus on sort of like these cascades where someone puts 0:21:11.000 --> 0:21:13.800 forth an idea and everyone feels compelled to fall online, 0:21:14.080 --> 0:21:18.520 or there's extreme pressure to voice a concern or voice 0:21:18.520 --> 0:21:22.000 skepticism on things. And are there are ways or strategies 0:21:22.040 --> 0:21:27.359 that aspiring forecasters can use to eliminate or reduce that bias. 0:21:27.960 --> 0:21:31.760 Group think is a big problem, and that's why in 0:21:31.840 --> 0:21:36.240 forecasting tournaments we typically have people make judgments independently of 0:21:36.280 --> 0:21:39.240 each other, um at least initially. It doesn't mean that 0:21:39.320 --> 0:21:41.800 all team or group decisions are going to be bad ones, 0:21:42.240 --> 0:21:45.320 But it does mean that forecasters need to value accuracy 0:21:45.359 --> 0:21:48.960 above all. If your primary goal is pleasing your boss, 0:21:48.960 --> 0:21:50.960 and you have an opinionated boss, it's going to be 0:21:51.080 --> 0:21:53.200 very hard for you to offer that boss an opinion 0:21:53.720 --> 0:21:57.000 with a probability judgment that points to a policy different 0:21:57.000 --> 0:22:00.840 from what the boss prefers. So it it's another version 0:22:00.840 --> 0:22:03.280 of the question what business are you in? Are you 0:22:03.320 --> 0:22:05.679 in the entertainment business, are you in the pleasing your 0:22:05.680 --> 0:22:09.600 boss business? Or you in the accuracy business. Accuracy business 0:22:09.720 --> 0:22:12.119 is often not the first business people are in, talking, 0:22:12.240 --> 0:22:15.520 not even the second business people are in. People are 0:22:15.640 --> 0:22:18.720 trying to have successful careers, they're trying to avoid embarrassment. 0:22:18.800 --> 0:22:21.520 There's a lot of other things people are doing a 0:22:21.600 --> 0:22:24.320 scie from accuracy. But so forecasting tournaments create a really 0:22:24.320 --> 0:22:27.880 weird social environment. They create a world that's only one 0:22:27.920 --> 0:22:31.719 thing matters, and that is minimizing the gaps between your 0:22:31.720 --> 0:22:36.320 probability judgments and reality over the long term. And that's it. 0:22:36.400 --> 0:22:39.560 That's that's that's that's the sole objective I'm still trying 0:22:39.560 --> 0:22:43.159 to understand. So, as you point out, like forecasting tournaments 0:22:43.840 --> 0:22:47.360 are very weird because in the real world, that's not 0:22:47.560 --> 0:22:50.679 how predictions are made, and people are aware of what 0:22:50.720 --> 0:22:53.320 other people are thinking and talking about. So when you 0:22:53.440 --> 0:22:56.560 consult and when you talk with people, how do you 0:22:56.640 --> 0:23:00.520 foster a culture? And I guess this is really what 0:23:00.600 --> 0:23:04.800 matters to the end consumers of your research, is how 0:23:04.840 --> 0:23:08.679 do you foster a culture where more people feel that 0:23:08.720 --> 0:23:12.320 they're in the accuracy business. It really helps if it 0:23:12.359 --> 0:23:15.560 comes from the top. People are looking at your typically 0:23:15.560 --> 0:23:18.920 look up for the normative cues about what's appropriate. So 0:23:19.040 --> 0:23:21.280 if you have a boss who's open to being wrong, 0:23:21.520 --> 0:23:24.080 that that helps a lot. So, Phil, I know the 0:23:24.119 --> 0:23:27.880 majority of your work has to do with statistical analysis 0:23:28.000 --> 0:23:31.280 of probabilities of forecasts, but I'm curious, could you give 0:23:31.400 --> 0:23:34.600 us a sort of case study that you've come across 0:23:34.760 --> 0:23:38.240 of a forecast that has gone very, very wrong and 0:23:38.280 --> 0:23:40.800