WEBVTT - Barry Ritholtz's Masters in Business: Philip Tetlock Interview 0:00:03.240 --> 0:00:07.560 This is Masters in Business with Barry Ridholts on Bloomberg Radio. 0:00:08.080 --> 0:00:11.600 This week on the podcast, I have a really fascinating guest. 0:00:11.840 --> 0:00:15.960 His name is Philip Tetlock. Here's a professor of loosely 0:00:16.040 --> 0:00:20.200 let's call its psychology and political science at uh both 0:00:20.320 --> 0:00:23.840 Wharton and the Arts and Sciences School at the University 0:00:23.880 --> 0:00:28.240 of Pennsylvania. Professor Tatlock is really a fascinating guy. I 0:00:28.400 --> 0:00:32.080 first got to know of his work through a book 0:00:32.120 --> 0:00:36.199 he wrote well over a decade UH ago called Expert 0:00:36.240 --> 0:00:40.400 Political Judgment, and it really cast an enormous shadow of 0:00:40.479 --> 0:00:45.440 doubt on all of these self proclaimed experts and and 0:00:45.560 --> 0:00:49.839 he used the field of political science as his basic 0:00:49.960 --> 0:00:55.040 investigation arena, but really it applied to everything from investing 0:00:55.160 --> 0:01:01.279 to economics to anywhere people prognosticate about the future and 0:01:01.320 --> 0:01:05.720 make um confident sounding assertions about what's going to happen 0:01:05.840 --> 0:01:09.520 one to five ten years in the future. He pretty 0:01:09.560 --> 0:01:14.680 much destroyed that entire line of thinking and showed that 0:01:14.800 --> 0:01:17.880 the average so called expert is no better than the 0:01:17.920 --> 0:01:20.880 average person walking down the street. That was well over 0:01:20.920 --> 0:01:25.080 a decade ago, and he tells an absolutely fascinating story 0:01:25.120 --> 0:01:29.679 of how we're familiar with DARPA, nette the Defense Research 0:01:29.720 --> 0:01:33.080 Projects invented the Internet and all these other really cool things. Well, 0:01:33.120 --> 0:01:36.160 the intelligence community has their own version of DARPA and 0:01:36.200 --> 0:01:40.640 it's called IARPA. And they basically approached Professor Tetlock and said, 0:01:41.120 --> 0:01:43.840 we're fascinated by your work and we'd like to find 0:01:43.840 --> 0:01:48.800 out if there are a way to to either identify 0:01:48.920 --> 0:01:53.559 or develop those outlawyers in your studies who actually turned 0:01:53.600 --> 0:01:57.160 out to be pretty good forecasters. And as it turns out, 0:01:57.720 --> 0:02:02.320 forecasting is a humble skill that can be I don't 0:02:02.320 --> 0:02:06.360 want to say learned by just about anybody, but you 0:02:06.440 --> 0:02:10.960 can undertake a number of steps to make your own 0:02:11.200 --> 0:02:16.400 forecasting better than it might have been otherwise. And it's 0:02:16.480 --> 0:02:21.800 fairly rational and fairly straightforward and really really fascinating. And 0:02:21.800 --> 0:02:24.680 so you'll hear the story of of not only how 0:02:24.720 --> 0:02:28.480 the original book came about, but how the new book 0:02:28.520 --> 0:02:32.800 came about and how he modified the previous view. It 0:02:32.840 --> 0:02:36.920 turns out, looking twelve months and further out, it's still 0:02:36.960 --> 0:02:39.240 pretty much a crapshoot. No one knows what's going to happen. 0:02:39.639 --> 0:02:42.280 But on shorter periods of time and on very very 0:02:42.320 --> 0:02:45.720 specific things, you can undertake a number of steps that 0:02:45.760 --> 0:02:51.840 will help your own ability to discern probable outcomes be 0:02:52.000 --> 0:02:55.720 much better. I think if you're at all a statistics 0:02:55.800 --> 0:02:59.680 or probability want you're gonna find this absolutely fascinating. Anyone 0:02:59.720 --> 0:03:03.840 who's an investor, a trader, who who deals with market 0:03:04.000 --> 0:03:07.080 or economic forecast might also find this to be really 0:03:07.120 --> 0:03:11.560 really interesting. So, without any further ado, my conversation with 0:03:11.639 --> 0:03:18.600 Professor Philip Tetlock. This is Masters in Business with Barry 0:03:18.680 --> 0:03:22.880 Ridholts on Bloomberg Radio. This week on Masters in Business 0:03:22.880 --> 0:03:26.520 on Bloomberg Radio, I have a special guest, Philip Tetlock, 0:03:26.680 --> 0:03:30.120 professor at the University of Pennsylvania, where he is cross 0:03:30.160 --> 0:03:32.560 appointed at both the Wharton School and the School of 0:03:32.680 --> 0:03:36.920 Arts and Sciences. Perhaps best known as the author of 0:03:37.480 --> 0:03:40.920 Expert Political Judgment, How Good Is It and How Can 0:03:40.960 --> 0:03:43.760 We Know? Which has won a number of awards. We'll 0:03:43.760 --> 0:03:46.640 talk about that in a little while. His most recent book, 0:03:47.200 --> 0:03:51.600 also winning accolades, is called super Forecasting, The Art and 0:03:51.720 --> 0:03:55.560 Science of Prediction. The Economist magazine named it to one 0:03:55.600 --> 0:03:59.960 of its best book lists of Professor Tatlock is also 0:04:00.320 --> 0:04:04.560 a co principal investigator of the Good Judgment Project, which 0:04:04.600 --> 0:04:09.240 is a multi year study looking at improving the accuracy 0:04:09.280 --> 0:04:13.880 of probability judgments of real world high stake events. Professor 0:04:13.920 --> 0:04:16.520 tat Luck, Welcome to Bloomberg. Oh, thank you, great to 0:04:16.520 --> 0:04:19.960 be here. So I'm familiar with you from your your 0:04:20.160 --> 0:04:22.920 earlier work, some of your earlier books, which I found 0:04:23.320 --> 0:04:28.120 absolutely fascinating. But let's just start this segment talking very 0:04:28.240 --> 0:04:32.640 very generally. How come there's such an enormous appetite for 0:04:32.960 --> 0:04:38.479 political and economic punditry, including all the predictions and forecasts 0:04:38.480 --> 0:04:41.280 that go with that. Well, it would be deeply dissonant 0:04:41.440 --> 0:04:45.600 for people to um to come to the conclusion that 0:04:46.160 --> 0:04:49.280 it doesn't matter who's elected president. It doesn't matter whether 0:04:49.320 --> 0:04:51.719 we raise taxes or cut them, whether we help the 0:04:51.800 --> 0:04:54.200 Ukraine or don't help the Ukraine, or intervene in Syria, 0:04:54.320 --> 0:04:56.680 or don't intervene in Syria, or assign free trade packs 0:04:56.760 --> 0:04:59.240 or don't do that. Um, we could just toss a coin, 0:04:59.320 --> 0:05:02.400 because nobody the is on either side of the political 0:05:02.440 --> 0:05:06.440 spectrum has any demonstrated ability to assign good probabilities to 0:05:06.480 --> 0:05:09.760 the consequences of those policy options. Uh So it's to 0:05:09.800 --> 0:05:14.920 say that sounds nihilistic, it's dissonant. Um people just don't 0:05:14.960 --> 0:05:16.800 want to live that way. People don't want to live 0:05:16.800 --> 0:05:19.760 that They want structure. And for the same reason that 0:05:19.880 --> 0:05:22.800 people have turned to psychics and which doctors and all 0:05:22.839 --> 0:05:25.919 sorts of things over the past. We we we we 0:05:25.920 --> 0:05:28.839 we we see guidance, predictive guidance, and we seek it. 0:05:29.800 --> 0:05:32.560 There's a demand for it even when the supply of 0:05:33.000 --> 0:05:38.040 good forecasting is extremely limited. So we're recording this in 0:05:38.440 --> 0:05:42.080 the month of March not too long ago, the cover 0:05:42.200 --> 0:05:46.320 of Barons magazine has a picture of Hillary Clinton and 0:05:47.200 --> 0:05:50.680 Donald Trump, which one is better for investors, and the 0:05:50.880 --> 0:05:56.039 usually conservative Barons grudgingly says, we think Hillary will be 0:05:56.040 --> 0:05:59.080 better for the stock market then than that Donald will. 0:05:59.560 --> 0:06:02.400 Don't we run the risk of putting way too much 0:06:02.440 --> 0:06:05.839 credit for good markets and good economies on the president 0:06:05.920 --> 0:06:08.880 and vice versa. When things are bad. Don't we blame 0:06:08.960 --> 0:06:11.280 them too much for what's gone wrong. I think there's 0:06:11.320 --> 0:06:14.480 a great deal of superstitious reasoning about the impact that 0:06:14.480 --> 0:06:17.919 that that leaders have on organizational performance and also the 0:06:17.960 --> 0:06:21.080 impact that presidents have on on the economy. The number 0:06:21.080 --> 0:06:24.400 of leaders that presidents have to influence economic outcomes is 0:06:24.600 --> 0:06:27.640 quite limited um, but there is this deep intuition we 0:06:27.680 --> 0:06:29.800 have that leaders should be accountable for what happens on 0:06:29.839 --> 0:06:32.200 their watch, like the captain of the ship. It really 0:06:32.240 --> 0:06:38.080 doesn't matter whether it's a storm suddenly emerges out of nowhere. Uh. 0:06:38.120 --> 0:06:40.760 You you, you want to hold somebody accountable and and 0:06:40.800 --> 0:06:43.000 that that is also deeply wired into us as it's 0:06:43.000 --> 0:06:45.160 how someone has to be accountable for it. So, so 0:06:45.240 --> 0:06:49.279 let's ask the obvious question, why are experts so often 0:06:49.440 --> 0:06:52.440 so wrong in their forecasts. Well, there are a couple 0:06:52.480 --> 0:06:55.680 of theories about that. One is that the problem lies 0:06:55.680 --> 0:06:58.400 in the experts and how the experts think, uh, and 0:06:58.440 --> 0:07:02.320 they could do better if they had used better analytical 0:07:02.320 --> 0:07:06.000 tools and we're more self critical and creative and thoughtful. 0:07:06.520 --> 0:07:08.480 And the other theory is that we just live in 0:07:08.520 --> 0:07:12.720 a radically unpredictable world in which originally nobody can do 0:07:12.760 --> 0:07:15.520 appreciably better than chance over extended periods of time. It's 0:07:15.560 --> 0:07:19.440 a world royaled by black Swanish dark gray Swanish events 0:07:19.760 --> 0:07:24.640 uh and uh, com buyer beware. Fair enough, So let's 0:07:24.680 --> 0:07:28.960 talk about good judgment in predicting future events, or at 0:07:29.040 --> 0:07:33.160 least good analytical processes. And along that that line, I 0:07:33.160 --> 0:07:35.760 would be remiss if I didn't ask what is the 0:07:35.800 --> 0:07:40.400 Briar score and why is it so important the Briar score. Well, 0:07:40.400 --> 0:07:44.120 the Briar score was originally developed by some statisticians UH 0:07:44.760 --> 0:07:48.160 working with meteorologists who wanted to keep score. And this 0:07:48.200 --> 0:07:51.360 goes all the way back to and it's a very 0:07:51.360 --> 0:07:54.800 simple idea. UH. You want to minimize the gaps between 0:07:54.800 --> 0:07:58.440 probability and reality. So your code reality is either zero 0:07:58.560 --> 0:08:01.160 or one, depending on whether the event occurred or didn't occur, 0:08:01.720 --> 0:08:04.600 and you have probability judgments that range from zero to one, 0:08:05.440 --> 0:08:08.280 and you you get a really good Brier score if 0:08:08.320 --> 0:08:11.560 you assign probabilities very close to zero to things that 0:08:11.600 --> 0:08:14.440 don't happen, and probability is very close to one point 0:08:14.520 --> 0:08:16.880 zero two things that do happen. So it's your ability 0:08:17.000 --> 0:08:21.680 to be justifiably decisive to make appropriately extreme judgments as 0:08:21.720 --> 0:08:25.040 the circumstances dictate, but to avoid driving off a cliff 0:08:25.160 --> 0:08:29.480 in the pursuit of that objective by making extremely overconfident 0:08:29.560 --> 0:08:32.000 judgments and saying ntent of things that don't happen and 0:08:32.960 --> 0:08:35.480 of things that do well. Once you're thinking in terms 0:08:35.520 --> 0:08:39.000 of probability, aren't you already light years ahead of the 0:08:39.080 --> 0:08:44.120 folks who are that decisive in making these bolds outlawyer declarations. 0:08:44.160 --> 0:08:47.640 But isn't that much more nuanced than thoughtful? Then here's 0:08:47.679 --> 0:08:51.280 what's gonna happen A And of course there's a big 0:08:51.320 --> 0:08:54.280 possibility that it just doesn't go that way. That's right. 0:08:54.320 --> 0:08:56.559 The probability scale and principle, if you want to look 0:08:56.600 --> 0:09:00.080 at it where a mathematician would is infinitely divisible, an 0:09:00.080 --> 0:09:02.480 infinite number of points between zero and one point zero. 0:09:02.520 --> 0:09:06.600 And obviously people can't distinguish that many levels of uncertainty. Now, 0:09:06.720 --> 0:09:10.640 when IBM S. Watson was playing in the Jeopardy competition 0:09:10.840 --> 0:09:14.280 and beat the best human players, uh, you might have 0:09:14.360 --> 0:09:16.920 noticed that under its answers, occasionally there would be this 0:09:16.960 --> 0:09:20.520 little uh Baysian probability estimate of how confident Watson was 0:09:20.520 --> 0:09:22.920 in his answer at point eight seven three six two 0:09:22.960 --> 0:09:27.680 or something. Uh So, these these types of form forms 0:09:27.720 --> 0:09:31.800 of artificial intelligence do try to uh make extremely granular 0:09:31.800 --> 0:09:34.600 distinctions among degrees of uncertainty. Human beings can't make that 0:09:34.640 --> 0:09:38.520 many uh distinctions among degrees of uncertainty in most environments. 0:09:38.559 --> 0:09:40.120 I mean, there's some environments where we can pull out 0:09:40.160 --> 0:09:42.280 a calculator and do it. If you can do it, 0:09:42.320 --> 0:09:45.720 with card tricks and so forth um or poker um. 0:09:45.880 --> 0:09:47.960 But there are there are real limits on how granular 0:09:48.040 --> 0:09:50.120 you can become when it comes to whether there's going 0:09:50.160 --> 0:09:52.480 to be a country leaving the Eurozone in the next year, 0:09:52.720 --> 0:09:54.920 or whether there's going to be a violent Sino Japanese 0:09:54.920 --> 0:09:58.280 class in the East China Sea, or things of that sort. 0:09:58.320 --> 0:10:00.319 How many degrees can you distinguish their Is it just 0:10:00.480 --> 0:10:02.559 yes or no? Or that yes maybe no? Or can 0:10:02.559 --> 0:10:05.520 you make finer distinctions than that? I'm Barry Ridhults. You're 0:10:05.559 --> 0:10:08.720 listening to Masters in Business on Bloomberg Radio. My special 0:10:08.720 --> 0:10:12.600 guest today is Professor Philip Tetlock of the University of Pennsylvania. 0:10:12.960 --> 0:10:15.360 He is the author of a number of books, but 0:10:15.440 --> 0:10:17.560 the one I want to talk about right now was 0:10:18.080 --> 0:10:21.959 the award winning Expert Political Judgment, How good is it 0:10:22.040 --> 0:10:24.600 and how can we know? Uh? This won a number 0:10:24.600 --> 0:10:29.679 of awards, the Graumeyer Award for Ideas Improving Political Order, 0:10:30.000 --> 0:10:33.600 the Woodrow Wilson Foundation Award for Political Science, and the 0:10:33.720 --> 0:10:39.240 Robert E. Lan Award for a Political Psychology. Let's let's 0:10:39.240 --> 0:10:41.680 start with a quote of yours that I want to 0:10:41.679 --> 0:10:46.720 get some some feedback on people who make predictions in 0:10:46.760 --> 0:10:50.880 their business who appear as experts on TV get quoted 0:10:50.960 --> 0:10:55.679 in newspaper articles advised, governments and businesses are no better 0:10:55.960 --> 0:10:59.719 than the rest of us at making forecasts. How is 0:10:59.760 --> 0:11:02.720 that possible? It turns out that you reached the point 0:11:02.720 --> 0:11:06.880 of diminishing marginal returns for knowledge quite quickly and a 0:11:06.880 --> 0:11:08.880 lot of the domains we care the most about. Now 0:11:08.880 --> 0:11:11.920 what does that mean? UM? When I started off doing 0:11:12.040 --> 0:11:16.040 research on political judgment, UM, one of the greatest psychologists 0:11:16.080 --> 0:11:20.800 and on the planet advised me, Daniel Kahneman and the 0:11:20.840 --> 0:11:27.480 Nobel Prize award winning psychologists most recent book, Thinking Fast Right. 0:11:27.559 --> 0:11:30.760 And it was a lunchtime conversation about thirty years ago 0:11:31.080 --> 0:11:34.080 in which he said rather casually that he thought that 0:11:34.120 --> 0:11:36.280 the experts I was interviewing from my early work on 0:11:36.320 --> 0:11:39.400 expert political judgment would have a hard time doing better 0:11:39.520 --> 0:11:43.560 than UM an attentive reader of the New York Times. UM, 0:11:43.800 --> 0:11:45.480 which is, you know, a kind of a fancy way 0:11:45.480 --> 0:11:48.960 of saying more or less what you just said. Uh. Now, 0:11:49.120 --> 0:11:51.040 he he didn't know that as a fact. He was 0:11:51.080 --> 0:11:54.600 offering that as an hypothesis. Uh. And I think the 0:11:54.679 --> 0:11:57.160 right way to look at this is, UM. It is 0:11:57.240 --> 0:11:59.920 an hypothesis we can be continually testing. It's it's not 0:12:00.200 --> 0:12:02.240 always going to be the case that experts are going 0:12:02.280 --> 0:12:05.000 to fall short, but they're going to fall short much 0:12:05.040 --> 0:12:10.240 more often than we would expect. Um well, how much 0:12:10.320 --> 0:12:13.559 of that is random? And when they're right and at 0:12:13.559 --> 0:12:18.360 a certain point don't don't the either investing or let's 0:12:18.360 --> 0:12:22.240 call it, voting public have a reasonable belief that the 0:12:22.360 --> 0:12:25.560 supposed experts know what they're talking about. When they make 0:12:25.600 --> 0:12:29.680 a forecast. They expect them to be considerably better than 0:12:30.600 --> 0:12:33.280 I think as as someone once called it a dart 0:12:33.320 --> 0:12:36.720 throwing monkey. Right, Well, there's a there's the big question 0:12:36.760 --> 0:12:39.320 that we want the answer to, and then they're all 0:12:39.440 --> 0:12:42.960 these proxy cues that we kind of latch onto in 0:12:43.000 --> 0:12:45.320 the hope that those proxy cues will get us closer 0:12:45.360 --> 0:12:47.040 to the answer. So the big question we want the 0:12:47.080 --> 0:12:49.679 answer to say is whether the U. S. Economy is 0:12:49.720 --> 0:12:53.319 going into her session next year, whether that the Dow 0:12:53.440 --> 0:12:56.080 is going to be over twenty thousand or under fifteen thousand. 0:12:56.120 --> 0:12:59.360 There there's some big questions that people in the financial 0:12:59.360 --> 0:13:03.360 community or political community want answers to, And there are 0:13:03.400 --> 0:13:06.040 various people who passed through our lives passed through your 0:13:06.120 --> 0:13:09.640 radio station, passed there in the op ed pages of newspapers, 0:13:09.640 --> 0:13:13.360 on television and so forth, who offer opinions on these things, 0:13:13.600 --> 0:13:16.240 and they come with various types of credentials. You might say, 0:13:16.320 --> 0:13:18.280 so and so is the muckety muck professor a bl 0:13:18.520 --> 0:13:20.960 bludy blum, or you might say that so and so 0:13:21.040 --> 0:13:22.960 when a Nobel prize. You might say that so and 0:13:22.960 --> 0:13:24.920 so is worth ten billion dollars, or you might do 0:13:25.000 --> 0:13:29.120 a lot of things you could say about that and um. 0:13:29.360 --> 0:13:32.640 The The interesting question is, uh, do those things give 0:13:32.679 --> 0:13:35.080 us much guidance on how accurate what the person is saying? 0:13:35.280 --> 0:13:38.319 So we're hoping that they do. We're we're hoping to say, well, 0:13:38.320 --> 0:13:40.200 this person must know what he or she is talking 0:13:40.200 --> 0:13:42.240 about by virtue of the fact that this person has 0:13:42.280 --> 0:13:46.200 done X, Y or z. But the relationship between having 0:13:46.240 --> 0:13:49.600 done X, Y or Z and accuracy is unknown. And 0:13:49.640 --> 0:13:52.400 the more honest we are about our ignorance, the more 0:13:52.440 --> 0:13:55.600 honest we are about when we're using proxy cues for 0:13:55.679 --> 0:13:58.839 judging how credible source of advice is, the better off 0:13:58.880 --> 0:14:00.440 we're going to be in the long term. And and 0:14:00.640 --> 0:14:03.360 by the way that applies to me too, that's fascinating. 0:14:03.440 --> 0:14:07.640 So so let me ask, um, when you put out 0:14:07.760 --> 0:14:11.800 expert political judgment, had anyone really done a full on 0:14:12.720 --> 0:14:17.480 quantitative analysis of how accurate experts were, at least in 0:14:17.520 --> 0:14:21.200 the political field. Had had anyone tried to figure out, Hey, 0:14:21.280 --> 0:14:26.680 let's figure out exactly how right or wrong these folks are? Before? Interestingly, 0:14:27.160 --> 0:14:29.080 very little work had been done. There was a little 0:14:29.080 --> 0:14:31.680 bit of work assessing the accurate Well, there's quite a 0:14:31.680 --> 0:14:34.240 bit of work assessing the work of weather forecasters. There 0:14:34.320 --> 0:14:38.880 was some work assessing the accuracy of expert bridge players, uh, 0:14:38.960 --> 0:14:41.720 And there was some work assessing the accuracy of economist. 0:14:41.760 --> 0:14:45.040 The Federal Reserve in Philadelphia and elsewhere had been doing 0:14:45.120 --> 0:14:49.000 some some studies along those lines. But as for assessing 0:14:49.040 --> 0:14:51.960 the accuracy of political pundits, at the time my book 0:14:52.000 --> 0:14:55.160 came out, I think there was extremely little work on 0:14:55.240 --> 0:14:58.960 that subject. So here's a quote from the book, and 0:14:59.200 --> 0:15:01.760 I want to get some um feedback from you on this. 0:15:02.240 --> 0:15:06.680 When they're wrong, they're rarely held accountable, and they rarely admitted. 0:15:07.160 --> 0:15:11.600 They insist they were just off on timing, or blindsided 0:15:11.640 --> 0:15:15.840 by an improbable events, or almost right or wrong for 0:15:15.880 --> 0:15:21.240 the right reasons. Well, if you're a pundit, You're playing 0:15:21.280 --> 0:15:25.040 a complicated game. Uh. If I'm a pundit on your 0:15:25.080 --> 0:15:27.720 show or anyone show, and I need I need to 0:15:27.760 --> 0:15:29.760 make it sound as though I know what I'm talking about. 0:15:30.120 --> 0:15:31.840 I need to make it sound as though I'm telling 0:15:32.120 --> 0:15:36.440 the listeners something they didn't know before. I also need 0:15:36.480 --> 0:15:39.480 to preserve my long term credibility, which means I have 0:15:39.600 --> 0:15:42.040 to have some escape clauses. So if the claims I 0:15:42.080 --> 0:15:44.000 make about the future turn out to be wrong, I 0:15:44.040 --> 0:15:46.560 need I need some way about of walking away from it. 0:15:47.160 --> 0:15:50.880 So when you have this problem of um, you have 0:15:50.920 --> 0:15:53.480 a career as a pundit, you need to be saying 0:15:53.520 --> 0:15:56.360 something surprising, but you also need to preserve your long 0:15:56.480 --> 0:15:59.400 term credibility. That's a real dilemma the pundit is in. 0:16:00.080 --> 0:16:03.200 So the typical way pundits cope with this is by 0:16:03.240 --> 0:16:07.840 saying something dramatic, uh, like Canada will disintegrate or the 0:16:07.880 --> 0:16:11.360 Eurozone will disintegrate, or Pudin will reinvade the Ukraine. But 0:16:11.880 --> 0:16:15.000 build in some waffle words like this could happen, or 0:16:15.000 --> 0:16:17.800 this might happen, or there's a distinct possibility this will happen. 0:16:18.280 --> 0:16:22.000 Now distinct possibility is one of those wonderful phrases, because 0:16:22.200 --> 0:16:25.240 if it happens, I can say, hey, I told you 0:16:25.280 --> 0:16:27.680 there was a distinct possibility. If it doesn't happen, I 0:16:27.680 --> 0:16:30.040 can say, hey, I just that was possible, So I'm 0:16:30.080 --> 0:16:33.880 covered either way. Uh, And that UM, it helps to 0:16:33.920 --> 0:16:37.840 explain why pundits um and and indeed why traditionally people 0:16:37.880 --> 0:16:40.400 in the US intelligence community as well UM have relied 0:16:40.440 --> 0:16:43.840 so heavily on vague verbiage forecasting because they need to 0:16:43.880 --> 0:16:47.200 be saying something that sounds informative, but they need uh 0:16:47.280 --> 0:16:50.080 strategy for preserving their long term credibility at the same time. 0:16:50.360 --> 0:16:54.400 So that explains why they don't admit error. Although I 0:16:54.440 --> 0:16:58.800 would argue nobody bats a thousand, admitting error shows that 0:16:58.840 --> 0:17:02.920 you have a little humility and recognize that it's not easy. 0:17:03.560 --> 0:17:05.600 In the last minute we have in this segment, the 0:17:05.680 --> 0:17:08.560 real question I have is why don't we hold these 0:17:08.560 --> 0:17:11.359 folks accountable. Well, here's the thing, Barry, they don't even 0:17:11.400 --> 0:17:13.639 think that they're wrong. If I if I say there 0:17:13.680 --> 0:17:16.479 was a distinct possibility that Putin is going to invade 0:17:16.600 --> 0:17:19.800 Estonia this this coming year, and he doesn't do it, 0:17:20.000 --> 0:17:25.440 I'm gonna interpret distinct possibilities having met a very low probability, 0:17:25.480 --> 0:17:27.040 and if he does do it, I'm going to interpret 0:17:27.080 --> 0:17:30.000 distinct possibilities having med a very high probability. I'm gonna 0:17:30.160 --> 0:17:32.280 we we tend to be somewhat self serving in our 0:17:32.320 --> 0:17:35.480 own mental calculus. I say, well, you know, we we interpret, 0:17:35.560 --> 0:17:37.320 we give ourselves a benefit of the doubt. On how 0:17:37.320 --> 0:17:40.399 we interpret distinct possibility adjusted, we give the benefit of 0:17:40.400 --> 0:17:43.480 the doubt. You know political pundits who favor our political party. 0:17:43.520 --> 0:17:46.159 I'm Barry rid Helts. You're listening to Masters in Business 0:17:46.160 --> 0:17:49.119 on Bloomberg Radio. My special guest today is Professor of 0:17:49.160 --> 0:17:53.360 Philip Tetlock. He is the author of Expert Political Judgment, 0:17:53.480 --> 0:17:56.439 How Good Is It? As well as Super Forecasters, a 0:17:56.440 --> 0:17:58.800 new book that just came out to great acclaim. He 0:17:58.840 --> 0:18:02.480 teaches at the Universe City of Pennsylvania, Wharton And and 0:18:02.560 --> 0:18:07.800 let's jump right in to the Hedgehogs versus Fox's discussion. 0:18:07.840 --> 0:18:11.640 You referenced this throughout really throughout um the second book 0:18:11.640 --> 0:18:14.240 a lot, and if I recall correctly, you mentioned it 0:18:14.280 --> 0:18:17.240 a few times in the first book, which I've read 0:18:17.280 --> 0:18:20.639 a while ago. Uh. For those people who may not 0:18:20.760 --> 0:18:25.800 have read Isaiah Berlin's essay explain to us what is 0:18:25.920 --> 0:18:31.560 the hedgehog and the fox? So Isaiah Berlin was British 0:18:31.840 --> 0:18:38.680 um scholar, political philosopher, political historian, philosopher who um took 0:18:38.800 --> 0:18:42.520 a quote from the Greek warrior poet or Kilicus from 0:18:43.480 --> 0:18:47.040 years ago and he built a really interesting essay around 0:18:47.040 --> 0:18:50.560 it and the quote. One of the few surviving fragments 0:18:50.560 --> 0:18:54.280 of this man's work was that the fox knows many things, 0:18:54.280 --> 0:18:57.520 but the hedgehog knows one big thing. And he intended 0:18:57.600 --> 0:19:00.760 that to capture different styles of things. King he thought 0:19:00.800 --> 0:19:03.840 that um, some some thinkers were much closer to being 0:19:04.160 --> 0:19:07.720 um uh foxes. Shakespeare, I think was one of his 0:19:07.800 --> 0:19:11.200 classic examples of a fox who could just a very 0:19:11.320 --> 0:19:15.320 multifaceted view of human nature, and other writers um he 0:19:15.400 --> 0:19:19.120 thought could be could be pigeonholed better as hedgehogs. Now 0:19:19.280 --> 0:19:24.240 we use this fox hedgehog distinction um in the work 0:19:24.280 --> 0:19:29.320 on political Judgment because it rather captures rather well uh 0:19:29.440 --> 0:19:32.920 different styles of thinking. Um. You could be, for example, 0:19:33.040 --> 0:19:36.200 a hedgehog of very many different political sorts. You could 0:19:36.240 --> 0:19:38.320 be a free market hedgehog, or you could be a 0:19:38.320 --> 0:19:44.920 Marxist hedgehog. You could be an environmentalist, uh, doomster hedgehog, 0:19:45.440 --> 0:19:51.679 or you could be a boomster um utopian kind of 0:19:51.800 --> 0:19:54.040 you techno utopian sort of hedgehog. You're going to find 0:19:54.040 --> 0:19:56.439 a cost effective substitutes for whatever we're running out of. 0:19:56.800 --> 0:20:02.800 So there there are many different forms of hedgehog left right, pessimistic, optimistic, 0:20:03.400 --> 0:20:06.960 and we identified many of them in the early work, 0:20:07.000 --> 0:20:09.679 and we tracked their their their accuracy, and we compared 0:20:09.720 --> 0:20:12.879 their accuracy to that of more fox like forecasters, and 0:20:12.920 --> 0:20:14.800 we found a couple of things. One is that the 0:20:14.840 --> 0:20:17.520 hedgehogs tend to have pretty bad batting average when you 0:20:17.520 --> 0:20:20.760 look at all their predictions, uh, their their overall batting 0:20:20.760 --> 0:20:24.000 average is pretty bad. We also found that the hedgehogs 0:20:24.000 --> 0:20:27.000 tended to be more prominent, They're more attractive to the media. 0:20:27.080 --> 0:20:30.000 The media like the kinds of sound bites that hedgehogs 0:20:30.000 --> 0:20:34.000 can deliver. And we found that the hedge hugs um 0:20:34.240 --> 0:20:37.800 also tended to have at least a few home runs. Matt, 0:20:37.880 --> 0:20:40.080 if you're making a lot of pretty extreme predictions on 0:20:40.080 --> 0:20:41.879 a wide range of subjects, at least a few of 0:20:41.880 --> 0:20:45.600 them are almost by chance going to be accurate, whereas 0:20:45.600 --> 0:20:48.760 the foxes are more making more moderate probability judgments, and 0:20:48.760 --> 0:20:52.919 they have less claim on home runs. So um, you 0:20:52.960 --> 0:20:55.920 get on somewhat ironic situation that the worst forecasters have 0:20:56.000 --> 0:21:00.560 the greatest media prominence. Isn't that inherent to the process 0:21:00.760 --> 0:21:05.080 of not only having a real specific expertise in one 0:21:05.119 --> 0:21:09.159 area as opposed to being a generalist, but also making 0:21:09.200 --> 0:21:12.960 those outlier forecasts. I use a slide in my presentation 0:21:13.560 --> 0:21:17.080 about a particular pundit who every year for the past 0:21:17.119 --> 0:21:22.600 seven years has forecast a seven like crash every year, 0:21:23.000 --> 0:21:26.560 and you would think the media would eventually say, hey, 0:21:26.600 --> 0:21:29.639 this guy is just consistently wrong. But it's such an 0:21:29.680 --> 0:21:33.520 outrageous forecast and it gets people so excited they love 0:21:33.600 --> 0:21:37.199 to bring them back on. Isn't that the nature of 0:21:37.760 --> 0:21:41.320 sensationalism that the hedgehogs are going to be more, especially 0:21:41.359 --> 0:21:45.199 today where everything is on clicks and views. Who's going 0:21:45.240 --> 0:21:48.640 to generate more clicks a rational sober well, we don't 0:21:48.680 --> 0:21:52.080 really know what's gonna happen, versus the sky is falling 0:21:52.119 --> 0:21:55.240 and everybody loves that. Well, I would just suggest that 0:21:55.240 --> 0:21:58.240 that people be better off if they were more honest 0:21:58.280 --> 0:22:01.320 about the functions that are served by consuming different types 0:22:01.359 --> 0:22:04.200 of information. So you if you said to yourself, look, 0:22:04.200 --> 0:22:06.280 I want to be entertained. I want I want to 0:22:06.320 --> 0:22:08.760 see somebody who's saying outrageous things and I'm gonna be 0:22:08.920 --> 0:22:11.280 But but I'm not gonna base my probability judgments on them. 0:22:11.280 --> 0:22:12.840 I just going to want to be entertained by these 0:22:13.000 --> 0:22:15.040 amazing stories as person is going to tell about how 0:22:15.040 --> 0:22:16.960 the Saudi regime is going to disintegrate and how we're 0:22:16.960 --> 0:22:19.639 on the verge of World War three. Uh, this is 0:22:19.680 --> 0:22:21.879 this is really entertaining stuff as opposed to listening to 0:22:21.880 --> 0:22:24.920 this much more at tentative, nuanced fox like forecaster who's 0:22:24.920 --> 0:22:27.679 saying on the wine hand, on the other hand, drones 0:22:27.760 --> 0:22:31.320 on and on. Why there's really only about probability of 0:22:31.359 --> 0:22:33.399 the Saudi regime changing in the next twenty four months. 0:22:33.800 --> 0:22:37.280 Can we really view the world through the lens of 0:22:37.040 --> 0:22:40.760 a of a single defining idea or is that, as 0:22:40.800 --> 0:22:45.480 the disclaimer says, for entertainment purposes only. Well, we need 0:22:45.560 --> 0:22:48.840 to be very clear about the functions that are being served. Uh. 0:22:49.119 --> 0:22:52.000 Some of these big ideas are very useful lenses for 0:22:52.080 --> 0:22:56.240 viewing the world um at particular moments in history and 0:22:56.400 --> 0:22:59.520 in conjunction with other ideas. So I'm not saying that 0:22:59.600 --> 0:23:03.359 the intellectual apparatus is useless. I'm saying that it's what's 0:23:03.400 --> 0:23:07.520 really dangerous is when you have a smart person who 0:23:07.600 --> 0:23:10.840 runs too far with a big idea and fails to 0:23:10.880 --> 0:23:13.000 see that the complexity of the world puts a lot 0:23:13.040 --> 0:23:15.359 of breaks on it. So one of our rules of 0:23:15.400 --> 0:23:18.359 thumb for distinguishing better forecasters and worse forecasters on the 0:23:18.400 --> 0:23:21.080 media is the ratio of the number of times they 0:23:21.119 --> 0:23:24.720 say however versus moreover. So if you have a high 0:23:24.760 --> 0:23:27.480 however over moreover ratio, that means you're a fox. That 0:23:27.480 --> 0:23:29.080 means you're boring. That means are probably going to be 0:23:29.160 --> 0:23:32.280