1
00:00:03,240 --> 00:00:07,560
Speaker 1: This is Masters in Business with Barry Ridholts on Bloomberg Radio.

2
00:00:08,080 --> 00:00:11,600
Speaker 1: This week on the podcast, I have a really fascinating guest.

3
00:00:11,840 --> 00:00:15,960
Speaker 1: His name is Philip Tetlock. Here's a professor of loosely

4
00:00:16,040 --> 00:00:20,200
Speaker 1: let's call its psychology and political science at uh both

5
00:00:20,320 --> 00:00:23,840
Speaker 1: Wharton and the Arts and Sciences School at the University

6
00:00:23,880 --> 00:00:28,240
Speaker 1: of Pennsylvania. Professor Tatlock is really a fascinating guy. I

7
00:00:28,400 --> 00:00:32,080
Speaker 1: first got to know of his work through a book

8
00:00:32,120 --> 00:00:36,199
Speaker 1: he wrote well over a decade UH ago called Expert

9
00:00:36,240 --> 00:00:40,400
Speaker 1: Political Judgment, and it really cast an enormous shadow of

10
00:00:40,479 --> 00:00:45,440
Speaker 1: doubt on all of these self proclaimed experts and and

11
00:00:45,560 --> 00:00:49,839
Speaker 1: he used the field of political science as his basic

12
00:00:49,960 --> 00:00:55,040
Speaker 1: investigation arena, but really it applied to everything from investing

13
00:00:55,160 --> 00:01:01,279
Speaker 1: to economics to anywhere people prognosticate about the future and

14
00:01:01,320 --> 00:01:05,720
Speaker 1: make um confident sounding assertions about what's going to happen

15
00:01:05,840 --> 00:01:09,520
Speaker 1: one to five ten years in the future. He pretty

16
00:01:09,560 --> 00:01:14,680
Speaker 1: much destroyed that entire line of thinking and showed that

17
00:01:14,800 --> 00:01:17,880
Speaker 1: the average so called expert is no better than the

18
00:01:17,920 --> 00:01:20,880
Speaker 1: average person walking down the street. That was well over

19
00:01:20,920 --> 00:01:25,080
Speaker 1: a decade ago, and he tells an absolutely fascinating story

20
00:01:25,120 --> 00:01:29,679
Speaker 1: of how we're familiar with DARPA, nette the Defense Research

21
00:01:29,720 --> 00:01:33,080
Speaker 1: Projects invented the Internet and all these other really cool things. Well,

22
00:01:33,120 --> 00:01:36,160
Speaker 1: the intelligence community has their own version of DARPA and

23
00:01:36,200 --> 00:01:40,640
Speaker 1: it's called IARPA. And they basically approached Professor Tetlock and said,

24
00:01:41,120 --> 00:01:43,840
Speaker 1: we're fascinated by your work and we'd like to find

25
00:01:43,840 --> 00:01:48,800
Speaker 1: out if there are a way to to either identify

26
00:01:48,920 --> 00:01:53,559
Speaker 1: or develop those outlawyers in your studies who actually turned

27
00:01:53,600 --> 00:01:57,160
Speaker 1: out to be pretty good forecasters. And as it turns out,

28
00:01:57,720 --> 00:02:02,320
Speaker 1: forecasting is a humble skill that can be I don't

29
00:02:02,320 --> 00:02:06,360
Speaker 1: want to say learned by just about anybody, but you

30
00:02:06,440 --> 00:02:10,960
Speaker 1: can undertake a number of steps to make your own

31
00:02:11,200 --> 00:02:16,400
Speaker 1: forecasting better than it might have been otherwise. And it's

32
00:02:16,480 --> 00:02:21,800
Speaker 1: fairly rational and fairly straightforward and really really fascinating. And

33
00:02:21,800 --> 00:02:24,680
Speaker 1: so you'll hear the story of of not only how

34
00:02:24,720 --> 00:02:28,480
Speaker 1: the original book came about, but how the new book

35
00:02:28,520 --> 00:02:32,800
Speaker 1: came about and how he modified the previous view. It

36
00:02:32,840 --> 00:02:36,920
Speaker 1: turns out, looking twelve months and further out, it's still

37
00:02:36,960 --> 00:02:39,240
Speaker 1: pretty much a crapshoot. No one knows what's going to happen.

38
00:02:39,639 --> 00:02:42,280
Speaker 1: But on shorter periods of time and on very very

39
00:02:42,320 --> 00:02:45,720
Speaker 1: specific things, you can undertake a number of steps that

40
00:02:45,760 --> 00:02:51,840
Speaker 1: will help your own ability to discern probable outcomes be

41
00:02:52,000 --> 00:02:55,720
Speaker 1: much better. I think if you're at all a statistics

42
00:02:55,800 --> 00:02:59,680
Speaker 1: or probability want you're gonna find this absolutely fascinating. Anyone

43
00:02:59,720 --> 00:03:03,840
Speaker 1: who's an investor, a trader, who who deals with market

44
00:03:04,000 --> 00:03:07,080
Speaker 1: or economic forecast might also find this to be really

45
00:03:07,120 --> 00:03:11,560
Speaker 1: really interesting. So, without any further ado, my conversation with

46
00:03:11,639 --> 00:03:18,600
Speaker 1: Professor Philip Tetlock. This is Masters in Business with Barry

47
00:03:18,680 --> 00:03:22,880
Speaker 1: Ridholts on Bloomberg Radio. This week on Masters in Business

48
00:03:22,880 --> 00:03:26,520
Speaker 1: on Bloomberg Radio, I have a special guest, Philip Tetlock,

49
00:03:26,680 --> 00:03:30,120
Speaker 1: professor at the University of Pennsylvania, where he is cross

50
00:03:30,160 --> 00:03:32,560
Speaker 1: appointed at both the Wharton School and the School of

51
00:03:32,680 --> 00:03:36,920
Speaker 1: Arts and Sciences. Perhaps best known as the author of

52
00:03:37,480 --> 00:03:40,920
Speaker 1: Expert Political Judgment, How Good Is It and How Can

53
00:03:40,960 --> 00:03:43,760
Speaker 1: We Know? Which has won a number of awards. We'll

54
00:03:43,760 --> 00:03:46,640
Speaker 1: talk about that in a little while. His most recent book,

55
00:03:47,200 --> 00:03:51,600
Speaker 1: also winning accolades, is called super Forecasting, The Art and

56
00:03:51,720 --> 00:03:55,560
Speaker 1: Science of Prediction. The Economist magazine named it to one

57
00:03:55,600 --> 00:03:59,960
Speaker 1: of its best book lists of Professor Tatlock is also

58
00:04:00,320 --> 00:04:04,560
Speaker 1: a co principal investigator of the Good Judgment Project, which

59
00:04:04,600 --> 00:04:09,240
Speaker 1: is a multi year study looking at improving the accuracy

60
00:04:09,280 --> 00:04:13,880
Speaker 1: of probability judgments of real world high stake events. Professor

61
00:04:13,920 --> 00:04:16,520
Speaker 1: tat Luck, Welcome to Bloomberg. Oh, thank you, great to

62
00:04:16,520 --> 00:04:19,960
Speaker 1: be here. So I'm familiar with you from your your

63
00:04:20,160 --> 00:04:22,920
Speaker 1: earlier work, some of your earlier books, which I found

64
00:04:23,320 --> 00:04:28,120
Speaker 1: absolutely fascinating. But let's just start this segment talking very

65
00:04:28,240 --> 00:04:32,640
Speaker 1: very generally. How come there's such an enormous appetite for

66
00:04:32,960 --> 00:04:38,479
Speaker 1: political and economic punditry, including all the predictions and forecasts

67
00:04:38,480 --> 00:04:41,280
Speaker 1: that go with that. Well, it would be deeply dissonant

68
00:04:41,440 --> 00:04:45,600
Speaker 1: for people to um to come to the conclusion that

69
00:04:46,160 --> 00:04:49,280
Speaker 1: it doesn't matter who's elected president. It doesn't matter whether

70
00:04:49,320 --> 00:04:51,719
Speaker 1: we raise taxes or cut them, whether we help the

71
00:04:51,800 --> 00:04:54,200
Speaker 1: Ukraine or don't help the Ukraine, or intervene in Syria,

72
00:04:54,320 --> 00:04:56,680
Speaker 1: or don't intervene in Syria, or assign free trade packs

73
00:04:56,760 --> 00:04:59,240
Speaker 1: or don't do that. Um, we could just toss a coin,

74
00:04:59,320 --> 00:05:02,400
Speaker 1: because nobody the is on either side of the political

75
00:05:02,440 --> 00:05:06,440
Speaker 1: spectrum has any demonstrated ability to assign good probabilities to

76
00:05:06,480 --> 00:05:09,760
Speaker 1: the consequences of those policy options. Uh So it's to

77
00:05:09,800 --> 00:05:14,920
Speaker 1: say that sounds nihilistic, it's dissonant. Um people just don't

78
00:05:14,960 --> 00:05:16,800
Speaker 1: want to live that way. People don't want to live

79
00:05:16,800 --> 00:05:19,760
Speaker 1: that They want structure. And for the same reason that

80
00:05:19,880 --> 00:05:22,800
Speaker 1: people have turned to psychics and which doctors and all

81
00:05:22,839 --> 00:05:25,919
Speaker 1: sorts of things over the past. We we we we

82
00:05:25,920 --> 00:05:28,839
Speaker 1: we we see guidance, predictive guidance, and we seek it.

83
00:05:29,800 --> 00:05:32,560
Speaker 1: There's a demand for it even when the supply of

84
00:05:33,000 --> 00:05:38,040
Speaker 1: good forecasting is extremely limited. So we're recording this in

85
00:05:38,440 --> 00:05:42,080
Speaker 1: the month of March not too long ago, the cover

86
00:05:42,200 --> 00:05:46,320
Speaker 1: of Barons magazine has a picture of Hillary Clinton and

87
00:05:47,200 --> 00:05:50,680
Speaker 1: Donald Trump, which one is better for investors, and the

88
00:05:50,880 --> 00:05:56,039
Speaker 1: usually conservative Barons grudgingly says, we think Hillary will be

89
00:05:56,040 --> 00:05:59,080
Speaker 1: better for the stock market then than that Donald will.

90
00:05:59,560 --> 00:06:02,400
Speaker 1: Don't we run the risk of putting way too much

91
00:06:02,440 --> 00:06:05,839
Speaker 1: credit for good markets and good economies on the president

92
00:06:05,920 --> 00:06:08,880
Speaker 1: and vice versa. When things are bad. Don't we blame

93
00:06:08,960 --> 00:06:11,280
Speaker 1: them too much for what's gone wrong. I think there's

94
00:06:11,320 --> 00:06:14,480
Speaker 1: a great deal of superstitious reasoning about the impact that

95
00:06:14,480 --> 00:06:17,919
Speaker 1: that that leaders have on organizational performance and also the

96
00:06:17,960 --> 00:06:21,080
Speaker 1: impact that presidents have on on the economy. The number

97
00:06:21,080 --> 00:06:24,400
Speaker 1: of leaders that presidents have to influence economic outcomes is

98
00:06:24,600 --> 00:06:27,640
Speaker 1: quite limited um, but there is this deep intuition we

99
00:06:27,680 --> 00:06:29,800
Speaker 1: have that leaders should be accountable for what happens on

100
00:06:29,839 --> 00:06:32,200
Speaker 1: their watch, like the captain of the ship. It really

101
00:06:32,240 --> 00:06:38,080
Speaker 1: doesn't matter whether it's a storm suddenly emerges out of nowhere. Uh.

102
00:06:38,120 --> 00:06:40,760
Speaker 1: You you, you want to hold somebody accountable and and

103
00:06:40,800 --> 00:06:43,000
Speaker 1: that that is also deeply wired into us as it's

104
00:06:43,000 --> 00:06:45,160
Speaker 1: how someone has to be accountable for it. So, so

105
00:06:45,240 --> 00:06:49,279
Speaker 1: let's ask the obvious question, why are experts so often

106
00:06:49,440 --> 00:06:52,440
Speaker 1: so wrong in their forecasts. Well, there are a couple

107
00:06:52,480 --> 00:06:55,680
Speaker 1: of theories about that. One is that the problem lies

108
00:06:55,680 --> 00:06:58,400
Speaker 1: in the experts and how the experts think, uh, and

109
00:06:58,440 --> 00:07:02,320
Speaker 1: they could do better if they had used better analytical

110
00:07:02,320 --> 00:07:06,000
Speaker 1: tools and we're more self critical and creative and thoughtful.

111
00:07:06,520 --> 00:07:08,480
Speaker 1: And the other theory is that we just live in

112
00:07:08,520 --> 00:07:12,720
Speaker 1: a radically unpredictable world in which originally nobody can do

113
00:07:12,760 --> 00:07:15,520
Speaker 1: appreciably better than chance over extended periods of time. It's

114
00:07:15,560 --> 00:07:19,440
Speaker 1: a world royaled by black Swanish dark gray Swanish events

115
00:07:19,760 --> 00:07:24,640
Speaker 1: uh and uh, com buyer beware. Fair enough, So let's

116
00:07:24,680 --> 00:07:28,960
Speaker 1: talk about good judgment in predicting future events, or at

117
00:07:29,040 --> 00:07:33,160
Speaker 1: least good analytical processes. And along that that line, I

118
00:07:33,160 --> 00:07:35,760
Speaker 1: would be remiss if I didn't ask what is the

119
00:07:35,800 --> 00:07:40,400
Speaker 1: Briar score and why is it so important the Briar score. Well,

120
00:07:40,400 --> 00:07:44,120
Speaker 1: the Briar score was originally developed by some statisticians UH

121
00:07:44,760 --> 00:07:48,160
Speaker 1: working with meteorologists who wanted to keep score. And this

122
00:07:48,200 --> 00:07:51,360
Speaker 1: goes all the way back to and it's a very

123
00:07:51,360 --> 00:07:54,800
Speaker 1: simple idea. UH. You want to minimize the gaps between

124
00:07:54,800 --> 00:07:58,440
Speaker 1: probability and reality. So your code reality is either zero

125
00:07:58,560 --> 00:08:01,160
Speaker 1: or one, depending on whether the event occurred or didn't occur,

126
00:08:01,720 --> 00:08:04,600
Speaker 1: and you have probability judgments that range from zero to one,

127
00:08:05,440 --> 00:08:08,280
Speaker 1: and you you get a really good Brier score if

128
00:08:08,320 --> 00:08:11,560
Speaker 1: you assign probabilities very close to zero to things that

129
00:08:11,600 --> 00:08:14,440
Speaker 1: don't happen, and probability is very close to one point

130
00:08:14,520 --> 00:08:16,880
Speaker 1: zero two things that do happen. So it's your ability

131
00:08:17,000 --> 00:08:21,680
Speaker 1: to be justifiably decisive to make appropriately extreme judgments as

132
00:08:21,720 --> 00:08:25,040
Speaker 1: the circumstances dictate, but to avoid driving off a cliff

133
00:08:25,160 --> 00:08:29,480
Speaker 1: in the pursuit of that objective by making extremely overconfident

134
00:08:29,560 --> 00:08:32,000
Speaker 1: judgments and saying ntent of things that don't happen and

135
00:08:32,960 --> 00:08:35,480
Speaker 1: of things that do well. Once you're thinking in terms

136
00:08:35,520 --> 00:08:39,000
Speaker 1: of probability, aren't you already light years ahead of the

137
00:08:39,080 --> 00:08:44,120
Speaker 1: folks who are that decisive in making these bolds outlawyer declarations.

138
00:08:44,160 --> 00:08:47,640
Speaker 1: But isn't that much more nuanced than thoughtful? Then here's

139
00:08:47,679 --> 00:08:51,280
Speaker 1: what's gonna happen A And of course there's a big

140
00:08:51,320 --> 00:08:54,280
Speaker 1: possibility that it just doesn't go that way. That's right.

141
00:08:54,320 --> 00:08:56,559
Speaker 1: The probability scale and principle, if you want to look

142
00:08:56,600 --> 00:09:00,080
Speaker 1: at it where a mathematician would is infinitely divisible, an

143
00:09:00,080 --> 00:09:02,480
Speaker 1: infinite number of points between zero and one point zero.

144
00:09:02,520 --> 00:09:06,600
Speaker 1: And obviously people can't distinguish that many levels of uncertainty. Now,

145
00:09:06,720 --> 00:09:10,640
Speaker 1: when IBM S. Watson was playing in the Jeopardy competition

146
00:09:10,840 --> 00:09:14,280
Speaker 1: and beat the best human players, uh, you might have

147
00:09:14,360 --> 00:09:16,920
Speaker 1: noticed that under its answers, occasionally there would be this

148
00:09:16,960 --> 00:09:20,520
Speaker 1: little uh Baysian probability estimate of how confident Watson was

149
00:09:20,520 --> 00:09:22,920
Speaker 1: in his answer at point eight seven three six two

150
00:09:22,960 --> 00:09:27,680
Speaker 1: or something. Uh So, these these types of form forms

151
00:09:27,720 --> 00:09:31,800
Speaker 1: of artificial intelligence do try to uh make extremely granular

152
00:09:31,800 --> 00:09:34,600
Speaker 1: distinctions among degrees of uncertainty. Human beings can't make that

153
00:09:34,640 --> 00:09:38,520
Speaker 1: many uh distinctions among degrees of uncertainty in most environments.

154
00:09:38,559 --> 00:09:40,120
Speaker 1: I mean, there's some environments where we can pull out

155
00:09:40,160 --> 00:09:42,280
Speaker 1: a calculator and do it. If you can do it,

156
00:09:42,320 --> 00:09:45,720
Speaker 1: with card tricks and so forth um or poker um.

157
00:09:45,880 --> 00:09:47,960
Speaker 1: But there are there are real limits on how granular

158
00:09:48,040 --> 00:09:50,120
Speaker 1: you can become when it comes to whether there's going

159
00:09:50,160 --> 00:09:52,480
Speaker 1: to be a country leaving the Eurozone in the next year,

160
00:09:52,720 --> 00:09:54,920
Speaker 1: or whether there's going to be a violent Sino Japanese

161
00:09:54,920 --> 00:09:58,280
Speaker 1: class in the East China Sea, or things of that sort.

162
00:09:58,320 --> 00:10:00,319
Speaker 1: How many degrees can you distinguish their Is it just

163
00:10:00,480 --> 00:10:02,559
Speaker 1: yes or no? Or that yes maybe no? Or can

164
00:10:02,559 --> 00:10:05,520
Speaker 1: you make finer distinctions than that? I'm Barry Ridhults. You're

165
00:10:05,559 --> 00:10:08,720
Speaker 1: listening to Masters in Business on Bloomberg Radio. My special

166
00:10:08,720 --> 00:10:12,600
Speaker 1: guest today is Professor Philip Tetlock of the University of Pennsylvania.

167
00:10:12,960 --> 00:10:15,360
Speaker 1: He is the author of a number of books, but

168
00:10:15,440 --> 00:10:17,560
Speaker 1: the one I want to talk about right now was

169
00:10:18,080 --> 00:10:21,959
Speaker 1: the award winning Expert Political Judgment, How good is it

170
00:10:22,040 --> 00:10:24,600
Speaker 1: and how can we know? Uh? This won a number

171
00:10:24,600 --> 00:10:29,679
Speaker 1: of awards, the Graumeyer Award for Ideas Improving Political Order,

172
00:10:30,000 --> 00:10:33,600
Speaker 1: the Woodrow Wilson Foundation Award for Political Science, and the

173
00:10:33,720 --> 00:10:39,240
Speaker 1: Robert E. Lan Award for a Political Psychology. Let's let's

174
00:10:39,240 --> 00:10:41,680
Speaker 1: start with a quote of yours that I want to

175
00:10:41,679 --> 00:10:46,720
Speaker 1: get some some feedback on people who make predictions in

176
00:10:46,760 --> 00:10:50,880
Speaker 1: their business who appear as experts on TV get quoted

177
00:10:50,960 --> 00:10:55,679
Speaker 1: in newspaper articles advised, governments and businesses are no better

178
00:10:55,960 --> 00:10:59,719
Speaker 1: than the rest of us at making forecasts. How is

179
00:10:59,760 --> 00:11:02,720
Speaker 1: that possible? It turns out that you reached the point

180
00:11:02,720 --> 00:11:06,880
Speaker 1: of diminishing marginal returns for knowledge quite quickly and a

181
00:11:06,880 --> 00:11:08,880
Speaker 1: lot of the domains we care the most about. Now

182
00:11:08,880 --> 00:11:11,920
Speaker 1: what does that mean? UM? When I started off doing

183
00:11:12,040 --> 00:11:16,040
Speaker 1: research on political judgment, UM, one of the greatest psychologists

184
00:11:16,080 --> 00:11:20,800
Speaker 1: and on the planet advised me, Daniel Kahneman and the

185
00:11:20,840 --> 00:11:27,480
Speaker 1: Nobel Prize award winning psychologists most recent book, Thinking Fast Right.

186
00:11:27,559 --> 00:11:30,760
Speaker 1: And it was a lunchtime conversation about thirty years ago

187
00:11:31,080 --> 00:11:34,080
Speaker 1: in which he said rather casually that he thought that

188
00:11:34,120 --> 00:11:36,280
Speaker 1: the experts I was interviewing from my early work on

189
00:11:36,320 --> 00:11:39,400
Speaker 1: expert political judgment would have a hard time doing better

190
00:11:39,520 --> 00:11:43,560
Speaker 1: than UM an attentive reader of the New York Times. UM,

191
00:11:43,800 --> 00:11:45,480
Speaker 1: which is, you know, a kind of a fancy way

192
00:11:45,480 --> 00:11:48,960
Speaker 1: of saying more or less what you just said. Uh. Now,

193
00:11:49,120 --> 00:11:51,040
Speaker 1: he he didn't know that as a fact. He was

194
00:11:51,080 --> 00:11:54,600
Speaker 1: offering that as an hypothesis. Uh. And I think the

195
00:11:54,679 --> 00:11:57,160
Speaker 1: right way to look at this is, UM. It is

196
00:11:57,240 --> 00:11:59,920
Speaker 1: an hypothesis we can be continually testing. It's it's not

197
00:12:00,200 --> 00:12:02,240
Speaker 1: always going to be the case that experts are going

198
00:12:02,280 --> 00:12:05,000
Speaker 1: to fall short, but they're going to fall short much

199
00:12:05,040 --> 00:12:10,240
Speaker 1: more often than we would expect. Um well, how much

200
00:12:10,320 --> 00:12:13,559
Speaker 1: of that is random? And when they're right and at

201
00:12:13,559 --> 00:12:18,360
Speaker 1: a certain point don't don't the either investing or let's

202
00:12:18,360 --> 00:12:22,240
Speaker 1: call it, voting public have a reasonable belief that the

203
00:12:22,360 --> 00:12:25,560
Speaker 1: supposed experts know what they're talking about. When they make

204
00:12:25,600 --> 00:12:29,680
Speaker 1: a forecast. They expect them to be considerably better than

205
00:12:30,600 --> 00:12:33,280
Speaker 1: I think as as someone once called it a dart

206
00:12:33,320 --> 00:12:36,720
Speaker 1: throwing monkey. Right, Well, there's a there's the big question

207
00:12:36,760 --> 00:12:39,320
Speaker 1: that we want the answer to, and then they're all

208
00:12:39,440 --> 00:12:42,960
Speaker 1: these proxy cues that we kind of latch onto in

209
00:12:43,000 --> 00:12:45,320
Speaker 1: the hope that those proxy cues will get us closer

210
00:12:45,360 --> 00:12:47,040
Speaker 1: to the answer. So the big question we want the

211
00:12:47,080 --> 00:12:49,679
Speaker 1: answer to say is whether the U. S. Economy is

212
00:12:49,720 --> 00:12:53,319
Speaker 1: going into her session next year, whether that the Dow

213
00:12:53,440 --> 00:12:56,080
Speaker 1: is going to be over twenty thousand or under fifteen thousand.

214
00:12:56,120 --> 00:12:59,360
Speaker 1: There there's some big questions that people in the financial

215
00:12:59,360 --> 00:13:03,360
Speaker 1: community or political community want answers to, And there are

216
00:13:03,400 --> 00:13:06,040
Speaker 1: various people who passed through our lives passed through your

217
00:13:06,120 --> 00:13:09,640
Speaker 1: radio station, passed there in the op ed pages of newspapers,

218
00:13:09,640 --> 00:13:13,360
Speaker 1: on television and so forth, who offer opinions on these things,

219
00:13:13,600 --> 00:13:16,240
Speaker 1: and they come with various types of credentials. You might say,

220
00:13:16,320 --> 00:13:18,280
Speaker 1: so and so is the muckety muck professor a bl

221
00:13:18,520 --> 00:13:20,960
Speaker 1: bludy blum, or you might say that so and so

222
00:13:21,040 --> 00:13:22,960
Speaker 1: when a Nobel prize. You might say that so and

223
00:13:22,960 --> 00:13:24,920
Speaker 1: so is worth ten billion dollars, or you might do

224
00:13:25,000 --> 00:13:29,120
Speaker 1: a lot of things you could say about that and um.

225
00:13:29,360 --> 00:13:32,640
Speaker 1: The The interesting question is, uh, do those things give

226
00:13:32,679 --> 00:13:35,080
Speaker 1: us much guidance on how accurate what the person is saying?

227
00:13:35,280 --> 00:13:38,319
Speaker 1: So we're hoping that they do. We're we're hoping to say, well,

228
00:13:38,320 --> 00:13:40,200
Speaker 1: this person must know what he or she is talking

229
00:13:40,200 --> 00:13:42,240
Speaker 1: about by virtue of the fact that this person has

230
00:13:42,280 --> 00:13:46,200
Speaker 1: done X, Y or z. But the relationship between having

231
00:13:46,240 --> 00:13:49,600
Speaker 1: done X, Y or Z and accuracy is unknown. And

232
00:13:49,640 --> 00:13:52,400
Speaker 1: the more honest we are about our ignorance, the more

233
00:13:52,440 --> 00:13:55,600
Speaker 1: honest we are about when we're using proxy cues for

234
00:13:55,679 --> 00:13:58,839
Speaker 1: judging how credible source of advice is, the better off

235
00:13:58,880 --> 00:14:00,440
Speaker 1: we're going to be in the long term. And and

236
00:14:00,640 --> 00:14:03,360
Speaker 1: by the way that applies to me too, that's fascinating.

237
00:14:03,440 --> 00:14:07,640
Speaker 1: So so let me ask, um, when you put out

238
00:14:07,760 --> 00:14:11,800
Speaker 1: expert political judgment, had anyone really done a full on

239
00:14:12,720 --> 00:14:17,480
Speaker 1: quantitative analysis of how accurate experts were, at least in

240
00:14:17,520 --> 00:14:21,200
Speaker 1: the political field. Had had anyone tried to figure out, Hey,

241
00:14:21,280 --> 00:14:26,680
Speaker 1: let's figure out exactly how right or wrong these folks are? Before? Interestingly,

242
00:14:27,160 --> 00:14:29,080
Speaker 1: very little work had been done. There was a little

243
00:14:29,080 --> 00:14:31,680
Speaker 1: bit of work assessing the accurate Well, there's quite a

244
00:14:31,680 --> 00:14:34,240
Speaker 1: bit of work assessing the work of weather forecasters. There

245
00:14:34,320 --> 00:14:38,880
Speaker 1: was some work assessing the accuracy of expert bridge players, uh,

246
00:14:38,960 --> 00:14:41,720
Speaker 1: And there was some work assessing the accuracy of economist.

247
00:14:41,760 --> 00:14:45,040
Speaker 1: The Federal Reserve in Philadelphia and elsewhere had been doing

248
00:14:45,120 --> 00:14:49,000
Speaker 1: some some studies along those lines. But as for assessing

249
00:14:49,040 --> 00:14:51,960
Speaker 1: the accuracy of political pundits, at the time my book

250
00:14:52,000 --> 00:14:55,160
Speaker 1: came out, I think there was extremely little work on

251
00:14:55,240 --> 00:14:58,960
Speaker 1: that subject. So here's a quote from the book, and

252
00:14:59,200 --> 00:15:01,760
Speaker 1: I want to get some um feedback from you on this.

253
00:15:02,240 --> 00:15:06,680
Speaker 1: When they're wrong, they're rarely held accountable, and they rarely admitted.

254
00:15:07,160 --> 00:15:11,600
Speaker 1: They insist they were just off on timing, or blindsided

255
00:15:11,640 --> 00:15:15,840
Speaker 1: by an improbable events, or almost right or wrong for

256
00:15:15,880 --> 00:15:21,240
Speaker 1: the right reasons. Well, if you're a pundit, You're playing

257
00:15:21,280 --> 00:15:25,040
Speaker 1: a complicated game. Uh. If I'm a pundit on your

258
00:15:25,080 --> 00:15:27,720
Speaker 1: show or anyone show, and I need I need to

259
00:15:27,760 --> 00:15:29,760
Speaker 1: make it sound as though I know what I'm talking about.

260
00:15:30,120 --> 00:15:31,840
Speaker 1: I need to make it sound as though I'm telling

261
00:15:32,120 --> 00:15:36,440
Speaker 1: the listeners something they didn't know before. I also need

262
00:15:36,480 --> 00:15:39,480
Speaker 1: to preserve my long term credibility, which means I have

263
00:15:39,600 --> 00:15:42,040
Speaker 1: to have some escape clauses. So if the claims I

264
00:15:42,080 --> 00:15:44,000
Speaker 1: make about the future turn out to be wrong, I

265
00:15:44,040 --> 00:15:46,560
Speaker 1: need I need some way about of walking away from it.

266
00:15:47,160 --> 00:15:50,880
Speaker 1: So when you have this problem of um, you have

267
00:15:50,920 --> 00:15:53,480
Speaker 1: a career as a pundit, you need to be saying

268
00:15:53,520 --> 00:15:56,360
Speaker 1: something surprising, but you also need to preserve your long

269
00:15:56,480 --> 00:15:59,400
Speaker 1: term credibility. That's a real dilemma the pundit is in.

270
00:16:00,080 --> 00:16:03,200
Speaker 1: So the typical way pundits cope with this is by

271
00:16:03,240 --> 00:16:07,840
Speaker 1: saying something dramatic, uh, like Canada will disintegrate or the

272
00:16:07,880 --> 00:16:11,360
Speaker 1: Eurozone will disintegrate, or Pudin will reinvade the Ukraine. But

273
00:16:11,880 --> 00:16:15,000
Speaker 1: build in some waffle words like this could happen, or

274
00:16:15,000 --> 00:16:17,800
Speaker 1: this might happen, or there's a distinct possibility this will happen.

275
00:16:18,280 --> 00:16:22,000
Speaker 1: Now distinct possibility is one of those wonderful phrases, because

276
00:16:22,200 --> 00:16:25,240
Speaker 1: if it happens, I can say, hey, I told you

277
00:16:25,280 --> 00:16:27,680
Speaker 1: there was a distinct possibility. If it doesn't happen, I

278
00:16:27,680 --> 00:16:30,040
Speaker 1: can say, hey, I just that was possible, So I'm

279
00:16:30,080 --> 00:16:33,880
Speaker 1: covered either way. Uh, And that UM, it helps to

280
00:16:33,920 --> 00:16:37,840
Speaker 1: explain why pundits um and and indeed why traditionally people

281
00:16:37,880 --> 00:16:40,400
Speaker 1: in the US intelligence community as well UM have relied

282
00:16:40,440 --> 00:16:43,840
Speaker 1: so heavily on vague verbiage forecasting because they need to

283
00:16:43,880 --> 00:16:47,200
Speaker 1: be saying something that sounds informative, but they need uh

284
00:16:47,280 --> 00:16:50,080
Speaker 1: strategy for preserving their long term credibility at the same time.

285
00:16:50,360 --> 00:16:54,400
Speaker 1: So that explains why they don't admit error. Although I

286
00:16:54,440 --> 00:16:58,800
Speaker 1: would argue nobody bats a thousand, admitting error shows that

287
00:16:58,840 --> 00:17:02,920
Speaker 1: you have a little humility and recognize that it's not easy.

288
00:17:03,560 --> 00:17:05,600
Speaker 1: In the last minute we have in this segment, the

289
00:17:05,680 --> 00:17:08,560
Speaker 1: real question I have is why don't we hold these

290
00:17:08,560 --> 00:17:11,359
Speaker 1: folks accountable. Well, here's the thing, Barry, they don't even

291
00:17:11,400 --> 00:17:13,639
Speaker 1: think that they're wrong. If I if I say there

292
00:17:13,680 --> 00:17:16,479
Speaker 1: was a distinct possibility that Putin is going to invade

293
00:17:16,600 --> 00:17:19,800
Speaker 1: Estonia this this coming year, and he doesn't do it,

294
00:17:20,000 --> 00:17:25,440
Speaker 1: I'm gonna interpret distinct possibilities having met a very low probability,

295
00:17:25,480 --> 00:17:27,040
Speaker 1: and if he does do it, I'm going to interpret

296
00:17:27,080 --> 00:17:30,000
Speaker 1: distinct possibilities having med a very high probability. I'm gonna

297
00:17:30,160 --> 00:17:32,280
Speaker 1: we we tend to be somewhat self serving in our

298
00:17:32,320 --> 00:17:35,480
Speaker 1: own mental calculus. I say, well, you know, we we interpret,

299
00:17:35,560 --> 00:17:37,320
Speaker 1: we give ourselves a benefit of the doubt. On how

300
00:17:37,320 --> 00:17:40,399
Speaker 1: we interpret distinct possibility adjusted, we give the benefit of

301
00:17:40,400 --> 00:17:43,480
Speaker 1: the doubt. You know political pundits who favor our political party.

302
00:17:43,520 --> 00:17:46,159
Speaker 1: I'm Barry rid Helts. You're listening to Masters in Business

303
00:17:46,160 --> 00:17:49,119
Speaker 1: on Bloomberg Radio. My special guest today is Professor of

304
00:17:49,160 --> 00:17:53,360
Speaker 1: Philip Tetlock. He is the author of Expert Political Judgment,

305
00:17:53,480 --> 00:17:56,439
Speaker 1: How Good Is It? As well as Super Forecasters, a

306
00:17:56,440 --> 00:17:58,800
Speaker 1: new book that just came out to great acclaim. He

307
00:17:58,840 --> 00:18:02,480
Speaker 1: teaches at the Universe City of Pennsylvania, Wharton And and

308
00:18:02,560 --> 00:18:07,800
Speaker 1: let's jump right in to the Hedgehogs versus Fox's discussion.

309
00:18:07,840 --> 00:18:11,640
Speaker 1: You referenced this throughout really throughout um the second book

310
00:18:11,640 --> 00:18:14,240
Speaker 1: a lot, and if I recall correctly, you mentioned it

311
00:18:14,280 --> 00:18:17,240
Speaker 1: a few times in the first book, which I've read

312
00:18:17,280 --> 00:18:20,639
Speaker 1: a while ago. Uh. For those people who may not

313
00:18:20,760 --> 00:18:25,800
Speaker 1: have read Isaiah Berlin's essay explain to us what is

314
00:18:25,920 --> 00:18:31,560
Speaker 1: the hedgehog and the fox? So Isaiah Berlin was British

315
00:18:31,840 --> 00:18:38,680
Speaker 1: um scholar, political philosopher, political historian, philosopher who um took

316
00:18:38,800 --> 00:18:42,520
Speaker 1: a quote from the Greek warrior poet or Kilicus from

317
00:18:43,480 --> 00:18:47,040
Speaker 1: years ago and he built a really interesting essay around

318
00:18:47,040 --> 00:18:50,560
Speaker 1: it and the quote. One of the few surviving fragments

319
00:18:50,560 --> 00:18:54,280
Speaker 1: of this man's work was that the fox knows many things,

320
00:18:54,280 --> 00:18:57,520
Speaker 1: but the hedgehog knows one big thing. And he intended

321
00:18:57,600 --> 00:19:00,760
Speaker 1: that to capture different styles of things. King he thought

322
00:19:00,800 --> 00:19:03,840
Speaker 1: that um, some some thinkers were much closer to being

323
00:19:04,160 --> 00:19:07,720
Speaker 1: um uh foxes. Shakespeare, I think was one of his

324
00:19:07,800 --> 00:19:11,200
Speaker 1: classic examples of a fox who could just a very

325
00:19:11,320 --> 00:19:15,320
Speaker 1: multifaceted view of human nature, and other writers um he

326
00:19:15,400 --> 00:19:19,120
Speaker 1: thought could be could be pigeonholed better as hedgehogs. Now

327
00:19:19,280 --> 00:19:24,240
Speaker 1: we use this fox hedgehog distinction um in the work

328
00:19:24,280 --> 00:19:29,320
Speaker 1: on political Judgment because it rather captures rather well uh

329
00:19:29,440 --> 00:19:32,920
Speaker 1: different styles of thinking. Um. You could be, for example,

330
00:19:33,040 --> 00:19:36,200
Speaker 1: a hedgehog of very many different political sorts. You could

331
00:19:36,240 --> 00:19:38,320
Speaker 1: be a free market hedgehog, or you could be a

332
00:19:38,320 --> 00:19:44,920
Speaker 1: Marxist hedgehog. You could be an environmentalist, uh, doomster hedgehog,

333
00:19:45,440 --> 00:19:51,679
Speaker 1: or you could be a boomster um utopian kind of

334
00:19:51,800 --> 00:19:54,040
Speaker 1: you techno utopian sort of hedgehog. You're going to find

335
00:19:54,040 --> 00:19:56,439
Speaker 1: a cost effective substitutes for whatever we're running out of.

336
00:19:56,800 --> 00:20:02,800
Speaker 1: So there there are many different forms of hedgehog left right, pessimistic, optimistic,

337
00:20:03,400 --> 00:20:06,960
Speaker 1: and we identified many of them in the early work,

338
00:20:07,000 --> 00:20:09,679
Speaker 1: and we tracked their their their accuracy, and we compared

339
00:20:09,720 --> 00:20:12,879
Speaker 1: their accuracy to that of more fox like forecasters, and

340
00:20:12,920 --> 00:20:14,800
Speaker 1: we found a couple of things. One is that the

341
00:20:14,840 --> 00:20:17,520
Speaker 1: hedgehogs tend to have pretty bad batting average when you

342
00:20:17,520 --> 00:20:20,760
Speaker 1: look at all their predictions, uh, their their overall batting

343
00:20:20,760 --> 00:20:24,000
Speaker 1: average is pretty bad. We also found that the hedgehogs

344
00:20:24,000 --> 00:20:27,000
Speaker 1: tended to be more prominent, They're more attractive to the media.

345
00:20:27,080 --> 00:20:30,000
Speaker 1: The media like the kinds of sound bites that hedgehogs

346
00:20:30,000 --> 00:20:34,000
Speaker 1: can deliver. And we found that the hedge hugs um

347
00:20:34,240 --> 00:20:37,800
Speaker 1: also tended to have at least a few home runs. Matt,

348
00:20:37,880 --> 00:20:40,080
Speaker 1: if you're making a lot of pretty extreme predictions on

349
00:20:40,080 --> 00:20:41,879
Speaker 1: a wide range of subjects, at least a few of

350
00:20:41,880 --> 00:20:45,600
Speaker 1: them are almost by chance going to be accurate, whereas

351
00:20:45,600 --> 00:20:48,760
Speaker 1: the foxes are more making more moderate probability judgments, and

352
00:20:48,760 --> 00:20:52,919
Speaker 1: they have less claim on home runs. So um, you

353
00:20:52,960 --> 00:20:55,920
Speaker 1: get on somewhat ironic situation that the worst forecasters have

354
00:20:56,000 --> 00:21:00,560
Speaker 1: the greatest media prominence. Isn't that inherent to the process

355
00:21:00,760 --> 00:21:05,080
Speaker 1: of not only having a real specific expertise in one

356
00:21:05,119 --> 00:21:09,159
Speaker 1: area as opposed to being a generalist, but also making

357
00:21:09,200 --> 00:21:12,960
Speaker 1: those outlier forecasts. I use a slide in my presentation

358
00:21:13,560 --> 00:21:17,080
Speaker 1: about a particular pundit who every year for the past

359
00:21:17,119 --> 00:21:22,600
Speaker 1: seven years has forecast a seven like crash every year,

360
00:21:23,000 --> 00:21:26,560
Speaker 1: and you would think the media would eventually say, hey,

361
00:21:26,600 --> 00:21:29,639
Speaker 1: this guy is just consistently wrong. But it's such an

362
00:21:29,680 --> 00:21:33,520
Speaker 1: outrageous forecast and it gets people so excited they love

363
00:21:33,600 --> 00:21:37,199
Speaker 1: to bring them back on. Isn't that the nature of

364
00:21:37,760 --> 00:21:41,320
Speaker 1: sensationalism that the hedgehogs are going to be more, especially

365
00:21:41,359 --> 00:21:45,199
Speaker 1: today where everything is on clicks and views. Who's going

366
00:21:45,240 --> 00:21:48,640
Speaker 1: to generate more clicks a rational sober well, we don't

367
00:21:48,680 --> 00:21:52,080
Speaker 1: really know what's gonna happen, versus the sky is falling

368
00:21:52,119 --> 00:21:55,240
Speaker 1: and everybody loves that. Well, I would just suggest that

369
00:21:55,240 --> 00:21:58,240
Speaker 1: that people be better off if they were more honest

370
00:21:58,280 --> 00:22:01,320
Speaker 1: about the functions that are served by consuming different types

371
00:22:01,359 --> 00:22:04,200
Speaker 1: of information. So you if you said to yourself, look,

372
00:22:04,200 --> 00:22:06,280
Speaker 1: I want to be entertained. I want I want to

373
00:22:06,320 --> 00:22:08,760
Speaker 1: see somebody who's saying outrageous things and I'm gonna be

374
00:22:08,920 --> 00:22:11,280
Speaker 1: But but I'm not gonna base my probability judgments on them.

375
00:22:11,280 --> 00:22:12,840
Speaker 1: I just going to want to be entertained by these

376
00:22:13,000 --> 00:22:15,040
Speaker 1: amazing stories as person is going to tell about how

377
00:22:15,040 --> 00:22:16,960
Speaker 1: the Saudi regime is going to disintegrate and how we're

378
00:22:16,960 --> 00:22:19,639
Speaker 1: on the verge of World War three. Uh, this is

379
00:22:19,680 --> 00:22:21,879
Speaker 1: this is really entertaining stuff as opposed to listening to

380
00:22:21,880 --> 00:22:24,920
Speaker 1: this much more at tentative, nuanced fox like forecaster who's

381
00:22:24,920 --> 00:22:27,679
Speaker 1: saying on the wine hand, on the other hand, drones

382
00:22:27,760 --> 00:22:31,320
Speaker 1: on and on. Why there's really only about probability of

383
00:22:31,359 --> 00:22:33,399
Speaker 1: the Saudi regime changing in the next twenty four months.

384
00:22:33,800 --> 00:22:37,280
Speaker 1: Can we really view the world through the lens of

385
00:22:37,040 --> 00:22:40,760
Speaker 1: a of a single defining idea or is that, as

386
00:22:40,800 --> 00:22:45,480
Speaker 1: the disclaimer says, for entertainment purposes only. Well, we need

387
00:22:45,560 --> 00:22:48,840
Speaker 1: to be very clear about the functions that are being served. Uh.

388
00:22:49,119 --> 00:22:52,000
Speaker 1: Some of these big ideas are very useful lenses for

389
00:22:52,080 --> 00:22:56,240
Speaker 1: viewing the world um at particular moments in history and

390
00:22:56,400 --> 00:22:59,520
Speaker 1: in conjunction with other ideas. So I'm not saying that

391
00:22:59,600 --> 00:23:03,359
Speaker 1: the intellectual apparatus is useless. I'm saying that it's what's

392
00:23:03,400 --> 00:23:07,520
Speaker 1: really dangerous is when you have a smart person who

393
00:23:07,600 --> 00:23:10,840
Speaker 1: runs too far with a big idea and fails to

394
00:23:10,880 --> 00:23:13,000
Speaker 1: see that the complexity of the world puts a lot

395
00:23:13,040 --> 00:23:15,359
Speaker 1: of breaks on it. So one of our rules of

396
00:23:15,400 --> 00:23:18,359
Speaker 1: thumb for distinguishing better forecasters and worse forecasters on the

397
00:23:18,400 --> 00:23:21,080
Speaker 1: media is the ratio of the number of times they

398
00:23:21,119 --> 00:23:24,720
Speaker 1: say however versus moreover. So if you have a high

399
00:23:24,760 --> 00:23:27,480
Speaker 1: however over moreover ratio, that means you're a fox. That

400
00:23:27,480 --> 00:23:29,080
Speaker 1: means you're boring. That means are probably going to be

401
00:23:29,160 --> 00:23:32,280
Speaker 1: kicked off the show. And if you have more more accurate,

402
00:23:32,359 --> 00:23:34,520
Speaker 1: more likely to be more accurate. But that's right, you're

403
00:23:34,560 --> 00:23:37,719
Speaker 1: gonna have better briar score. I'm Barry Ridholts. You're listening

404
00:23:37,760 --> 00:23:41,120
Speaker 1: to Masters in Business on Bloomberg Radio. My special guest

405
00:23:41,160 --> 00:23:45,400
Speaker 1: this week, professor of phil Tetlock of the University of Pennsylvania.

406
00:23:45,840 --> 00:23:50,080
Speaker 1: His most recent book, super Forecasting, The Art and Science

407
00:23:50,240 --> 00:23:53,800
Speaker 1: of Prediction. So let's jump right into this because I

408
00:23:53,840 --> 00:23:56,880
Speaker 1: have so many questions about this, and let me start

409
00:23:56,880 --> 00:24:00,600
Speaker 1: out by just asking how many piano tuners are there Chicago,

410
00:24:01,840 --> 00:24:04,920
Speaker 1: UH somewhere between about eight and I think so. So

411
00:24:05,320 --> 00:24:09,280
Speaker 1: it's a fascinating question because your initial reaction is to

412
00:24:09,440 --> 00:24:12,800
Speaker 1: shrug and say, I don't know. The variation I've heard

413
00:24:12,800 --> 00:24:15,360
Speaker 1: of that is how many cardiac surgeons are there in London?

414
00:24:16,119 --> 00:24:19,639
Speaker 1: UM or what's the empire state building way? That's another

415
00:24:19,680 --> 00:24:22,600
Speaker 1: good question. So so let's talk. Let's talk a little

416
00:24:22,600 --> 00:24:26,840
Speaker 1: bit about what do most people do when presented with

417
00:24:26,880 --> 00:24:29,399
Speaker 1: a question like that? Interesting? You know, some of the

418
00:24:29,480 --> 00:24:32,320
Speaker 1: high tech firms in Silicon Valley were quite fon fond

419
00:24:32,359 --> 00:24:36,480
Speaker 1: of asking off off the out of left field questions

420
00:24:36,480 --> 00:24:38,520
Speaker 1: of this sort because they thought it was a great

421
00:24:38,520 --> 00:24:41,760
Speaker 1: way of testing how well people think on their feet. UM.

422
00:24:41,800 --> 00:24:44,240
Speaker 1: In the book, we call these Ferremi questions, named after

423
00:24:44,320 --> 00:24:47,520
Speaker 1: the great Italian American physicist and Rico Ferremi, who developed

424
00:24:47,520 --> 00:24:50,840
Speaker 1: the first um UH nuclear reactors keep part of the

425
00:24:50,840 --> 00:24:55,160
Speaker 1: Manhattan Project developing the bomb, and Ferremy was fond of

426
00:24:55,440 --> 00:24:59,600
Speaker 1: UH posing these oddball questions to his students. He what

427
00:24:59,640 --> 00:25:02,159
Speaker 1: do he wanted to the students to do was to

428
00:25:02,280 --> 00:25:06,320
Speaker 1: take an seemingly intractable problem and break it down into

429
00:25:06,400 --> 00:25:09,840
Speaker 1: parts or components that were more tractable, So you might

430
00:25:09,880 --> 00:25:12,320
Speaker 1: not have any idea how many. No one has any

431
00:25:12,320 --> 00:25:14,639
Speaker 1: idea initially on how many how many piano tuners that

432
00:25:14,720 --> 00:25:17,800
Speaker 1: might be in Chicago. But you make guestiments about the

433
00:25:17,800 --> 00:25:20,800
Speaker 1: population of Chicago. Walk us through that because you you

434
00:25:20,920 --> 00:25:23,359
Speaker 1: go through about seven steps and you get pretty close

435
00:25:23,400 --> 00:25:26,000
Speaker 1: to the correct answer. Right. Well, you're you're making a

436
00:25:26,040 --> 00:25:29,480
Speaker 1: lot of guests and it's not just the breaking down

437
00:25:29,640 --> 00:25:32,440
Speaker 1: and get trying to get the answer. What you're doing

438
00:25:32,480 --> 00:25:35,760
Speaker 1: in the process is you're revealing sources of ignorance, and

439
00:25:35,960 --> 00:25:38,359
Speaker 1: your colleagues on your team, for example, if it's because

440
00:25:38,359 --> 00:25:41,640
Speaker 1: our forecasters often work together on teams, uh, your colleagues

441
00:25:41,680 --> 00:25:44,800
Speaker 1: can help you correct help correct your errors. So what's

442
00:25:44,840 --> 00:25:47,760
Speaker 1: the population of Chicago, I don't really know, between two

443
00:25:47,760 --> 00:25:51,280
Speaker 1: point five and four million, and maybe you know some

444
00:25:51,280 --> 00:25:54,679
Speaker 1: somewhere in the middle there, what proportion of the population

445
00:25:54,720 --> 00:25:57,760
Speaker 1: would have a piano and so forth? You see, if

446
00:25:57,760 --> 00:25:59,440
Speaker 1: you would, you would break it down and you you

447
00:25:59,440 --> 00:26:02,800
Speaker 1: you try to mind eventually how how how many people

448
00:26:02,840 --> 00:26:06,240
Speaker 1: could conceivably make a living working as as piano tuners

449
00:26:06,480 --> 00:26:09,280
Speaker 1: given the number of people who own pianos in Chicago

450
00:26:09,400 --> 00:26:12,199
Speaker 1: and their willingness to pay for the services of piano

451
00:26:12,240 --> 00:26:16,600
Speaker 1: tuners um and so so Faremy did this. He One

452
00:26:16,800 --> 00:26:20,200
Speaker 1: legend is that he tried to uh infer the strength

453
00:26:20,200 --> 00:26:24,919
Speaker 1: of the first atomic blast um by dropping little pieces

454
00:26:24,960 --> 00:26:26,840
Speaker 1: of paper when the when, when the when the winds

455
00:26:26,880 --> 00:26:29,600
Speaker 1: came in front, and by estimating how far the winds blew.

456
00:26:29,640 --> 00:26:32,879
Speaker 1: And I think he was off by about forty or

457
00:26:32,920 --> 00:26:35,520
Speaker 1: fifty percent. But you know that for Faremy estimates, that's

458
00:26:35,560 --> 00:26:37,520
Speaker 1: not too bad. It's it's a lot better than simply

459
00:26:37,520 --> 00:26:39,320
Speaker 1: shrugging your shoulders and saying I have no idea. I

460
00:26:39,359 --> 00:26:42,240
Speaker 1: say a ten Kelton glass as post of twenty. But

461
00:26:43,160 --> 00:26:48,720
Speaker 1: that's that's fascinating. I'm absolutely entranced by Firm's paradox, which says,

462
00:26:48,880 --> 00:26:51,800
Speaker 1: where where is everybody? You know, it's a giant universe

463
00:26:51,840 --> 00:26:55,119
Speaker 1: filled with different galaxies and hundreds of billions of stars?

464
00:26:55,160 --> 00:26:58,760
Speaker 1: And are we really the only intelligent life here? And

465
00:26:58,960 --> 00:27:02,439
Speaker 1: I found most of the various arguments both ways to

466
00:27:02,520 --> 00:27:06,200
Speaker 1: be lacking. It's really it's really a fascinating, fascinating debate.

467
00:27:06,680 --> 00:27:10,639
Speaker 1: But let's let's stick with this. So so looking at

468
00:27:10,960 --> 00:27:13,560
Speaker 1: what the empire state building ways, or or how many

469
00:27:13,600 --> 00:27:18,240
Speaker 1: piano tuners in Chicago show us how to break down

470
00:27:19,119 --> 00:27:24,040
Speaker 1: unknown questions into component parts and make reasonable assessments and

471
00:27:24,080 --> 00:27:28,960
Speaker 1: reasonable valuations on each of those segments. Um. So, so

472
00:27:29,080 --> 00:27:33,760
Speaker 1: what do you find about teams of super forecasters? How

473
00:27:33,840 --> 00:27:37,600
Speaker 1: much better are they at at these sort of predictions

474
00:27:37,920 --> 00:27:45,960
Speaker 1: than the average person, or prediction markets or just regular pundits. Well, Um,

475
00:27:46,040 --> 00:27:50,960
Speaker 1: the teams have super forecasters truly astonished us because the

476
00:27:51,280 --> 00:27:54,280
Speaker 1: statisticians were telling us the right thing to do here

477
00:27:55,000 --> 00:27:59,600
Speaker 1: was to have each individual top performer make judgments completely

478
00:27:59,640 --> 00:28:02,840
Speaker 1: independ at lay of the others, um, rather than allowing

479
00:28:02,840 --> 00:28:05,359
Speaker 1: them to contaminate each other. And you get conformity, and

480
00:28:05,359 --> 00:28:07,240
Speaker 1: you get group think, and you got to get kind

481
00:28:07,240 --> 00:28:09,800
Speaker 1: of a blur rather than a number of distinct points

482
00:28:09,800 --> 00:28:12,159
Speaker 1: of view, and then you can combine them statistically somehow.

483
00:28:12,640 --> 00:28:16,760
Speaker 1: So Um, we were the only competitor in the forecasting

484
00:28:16,760 --> 00:28:19,400
Speaker 1: tournament sponsored by the U S Intelligence community that used

485
00:28:19,400 --> 00:28:22,760
Speaker 1: teams um And but we we hedged our bets. We

486
00:28:22,760 --> 00:28:25,159
Speaker 1: weren't sure that teams would work. We ran an experiment

487
00:28:25,240 --> 00:28:27,800
Speaker 1: and we randomly assigned people to teams, and we random

488
00:28:27,840 --> 00:28:30,360
Speaker 1: and had other people work as individuals. Uh. And we

489
00:28:30,359 --> 00:28:33,399
Speaker 1: were truly surprised that the teams functioned as well as

490
00:28:33,440 --> 00:28:37,280
Speaker 1: they did UM, and it's an interesting question of why

491
00:28:37,440 --> 00:28:41,600
Speaker 1: our teams were so dynamic and open minded relative to

492
00:28:41,640 --> 00:28:45,200
Speaker 1: many teams you see in actual organizations. But before you

493
00:28:45,240 --> 00:28:47,680
Speaker 1: answer that question, let's let's put a little flesh on

494
00:28:47,720 --> 00:28:51,600
Speaker 1: the bone with some numbers. So teams of ordinary forecasters

495
00:28:51,680 --> 00:28:55,800
Speaker 1: beat the wisdom of the crowd by about ten. They

496
00:28:55,800 --> 00:29:01,120
Speaker 1: were bested by prediction markets UM and addiction markets beat

497
00:29:01,240 --> 00:29:05,800
Speaker 1: ordinary teams by about and then the super teams of

498
00:29:05,880 --> 00:29:10,880
Speaker 1: the best forecasters beat the prediction markets by anywhere from fifteen.

499
00:29:12,280 --> 00:29:16,680
Speaker 1: So these folks working in groups really are the outliers.

500
00:29:16,920 --> 00:29:19,240
Speaker 1: None of the other groups are even close to them

501
00:29:19,280 --> 00:29:24,120
Speaker 1: in terms of of accuracy. Why do you think that is, Well, yeah,

502
00:29:24,200 --> 00:29:26,080
Speaker 1: and I'll just add one other thing to that. They

503
00:29:26,080 --> 00:29:31,040
Speaker 1: weren't just outperforming prediction markets in the public sphere, they

504
00:29:31,040 --> 00:29:34,600
Speaker 1: are also outperforming intelligence analysts who were working, you know,

505
00:29:34,640 --> 00:29:38,360
Speaker 1: behind a veil of of of classified information. And it's

506
00:29:38,400 --> 00:29:40,760
Speaker 1: just a remarkable thing. And I think US intelligence community,

507
00:29:40,760 --> 00:29:43,240
Speaker 1: which is much maligned for many things, deserves some credit

508
00:29:43,440 --> 00:29:46,440
Speaker 1: for its willingness to sponsor of forecasting tournament that has

509
00:29:46,440 --> 00:29:49,240
Speaker 1: the potential to be embarrassing for government bureaucracy. How often

510
00:29:49,280 --> 00:29:52,360
Speaker 1: do you see a government bureaucracy, uh, spend money, a

511
00:29:52,360 --> 00:29:54,800
Speaker 1: lot of money on a project that has the potential

512
00:29:54,840 --> 00:29:57,479
Speaker 1: to be to yield results that you're fairly embarrassing. How

513
00:29:57,520 --> 00:30:00,680
Speaker 1: hard is it to cultivate these skills or is it

514
00:30:00,960 --> 00:30:04,960
Speaker 1: just a matter of internalizing these ten bullet points? I

515
00:30:04,960 --> 00:30:08,160
Speaker 1: wouldn't say just a matter. It's it's no, it's an

516
00:30:08,200 --> 00:30:10,800
Speaker 1: it's a non trivial thing to do. Uh it's it's

517
00:30:10,800 --> 00:30:15,760
Speaker 1: it's pretty hard. Um. And uh so let's let's take

518
00:30:15,800 --> 00:30:17,360
Speaker 1: get an example of one. And what do you think

519
00:30:17,440 --> 00:30:22,720
Speaker 1: is an important um, an important commandment of super forecasting?

520
00:30:23,680 --> 00:30:29,520
Speaker 1: You want to just pick up one at random? Sure? Okay, Um, well,

521
00:30:29,520 --> 00:30:32,400
Speaker 1: one of them has to do with granularity and uh

522
00:30:32,600 --> 00:30:35,479
Speaker 1: it's it's it's actually grounded in a story that about

523
00:30:36,360 --> 00:30:39,920
Speaker 1: President Obama and how are he reacted to his advisors

524
00:30:39,960 --> 00:30:45,520
Speaker 1: who were um offering him um somewhat conflicting probabilities on

525
00:30:45,560 --> 00:30:48,200
Speaker 1: how likely it was that Osama bin Laden was residing

526
00:30:48,200 --> 00:30:51,440
Speaker 1: in a particular compound in Abadabad, Pakistan. As we all

527
00:30:51,480 --> 00:30:55,200
Speaker 1: know now, he was indeed residing there, and the President

528
00:30:55,240 --> 00:30:58,280
Speaker 1: did authorize a n a vcal mission, and that resulted

529
00:30:58,320 --> 00:31:03,640
Speaker 1: in Osama bin Lad's death. UM. Now, when the President

530
00:31:03,720 --> 00:31:07,640
Speaker 1: was confronted by these probability estimates from really smart people

531
00:31:07,640 --> 00:31:11,080
Speaker 1: at the top of the intelligence apparatus, ranging from about

532
00:31:11,320 --> 00:31:16,200
Speaker 1: you know, thirty five or up to about um the

533
00:31:16,200 --> 00:31:19,320
Speaker 1: President's reaction was an interesting one. If if you Hee

534
00:31:19,360 --> 00:31:22,040
Speaker 1: had computed the average of the median estimate of the

535
00:31:22,080 --> 00:31:24,040
Speaker 1: advice he was getting, he would have said, looks like

536
00:31:24,040 --> 00:31:28,440
Speaker 1: about a s probability. But instead he said something interesting.

537
00:31:28,560 --> 00:31:31,240
Speaker 1: He said, well, look, look, guys, this is a coin flip.

538
00:31:31,320 --> 00:31:35,800
Speaker 1: It's a fifty fifty thing. Um. Now, the President, I

539
00:31:35,840 --> 00:31:38,400
Speaker 1: think is a very intelligent person, and I think he's

540
00:31:38,440 --> 00:31:41,800
Speaker 1: capable of being very granular in his assessments of uncertainty.

541
00:31:42,000 --> 00:31:44,640
Speaker 1: And if you doubt it, think about the following thought experiment,

542
00:31:44,640 --> 00:31:48,040
Speaker 1: which is appropriate given it approaching March Madness. He follows

543
00:31:48,080 --> 00:31:51,880
Speaker 1: March Madness he uh, and basketball. He's a basketball fan.

544
00:31:52,280 --> 00:31:55,240
Speaker 1: Imagine he'd been sitting around with friends waiting for Duke

545
00:31:55,320 --> 00:31:58,280
Speaker 1: to play some team in March Matt Martin the March

546
00:31:58,320 --> 00:32:02,040
Speaker 1: Madness tournament, and uh, they offered him exactly the same

547
00:32:02,080 --> 00:32:05,360
Speaker 1: probabilities about whether Duke would win the win the game.

548
00:32:05,920 --> 00:32:08,160
Speaker 1: You know, somewhere between thirty five and ninety five, with

549
00:32:08,240 --> 00:32:11,040
Speaker 1: the center of gravity of opinion around seventy, would you

550
00:32:11,080 --> 00:32:13,240
Speaker 1: have said it sounds like a fifty fifty thing or

551
00:32:13,280 --> 00:32:17,240
Speaker 1: would you have said, MM sounds like about UM three

552
00:32:17,240 --> 00:32:21,000
Speaker 1: to one, Duke UM, I think to ask the question

553
00:32:21,080 --> 00:32:24,320
Speaker 1: is to answer it. He would have seen opportunity for

554
00:32:24,400 --> 00:32:27,360
Speaker 1: being much more granular in making bets about sports than

555
00:32:27,400 --> 00:32:30,160
Speaker 1: he would and making estimates about the likelihood of a

556
00:32:30,160 --> 00:32:34,480
Speaker 1: particular terrorist being in a particular location. UM. Now it

557
00:32:34,840 --> 00:32:38,480
Speaker 1: turns out that UM for many categories of problems where

558
00:32:38,520 --> 00:32:41,640
Speaker 1: we think it's impossible to be more granular, it is possible.

559
00:32:43,040 --> 00:32:45,760
Speaker 1: And that's one of the things super forecasters have learned

560
00:32:45,960 --> 00:32:49,560
Speaker 1: that there's a difference between fifty and sometimes they can

561
00:32:49,560 --> 00:32:53,200
Speaker 1: even make distinctions between fifty and now we quote that

562
00:32:53,280 --> 00:32:56,840
Speaker 1: the Chief Risk Officer of UM a q r M,

563
00:32:57,560 --> 00:33:01,800
Speaker 1: the hedge fund UMAST is the head of that, and

564
00:33:02,120 --> 00:33:06,080
Speaker 1: the chief risk Officer is Aaron Brown. And when we

565
00:33:06,120 --> 00:33:09,640
Speaker 1: talked with with Aaron, he you know, he he's also

566
00:33:09,680 --> 00:33:13,320
Speaker 1: a really serious poker player, UM, and he said, well,

567
00:33:13,320 --> 00:33:15,360
Speaker 1: he can tell it different. World class poker player and

568
00:33:15,360 --> 00:33:18,400
Speaker 1: a talented amateur on the basis that the world class

569
00:33:18,440 --> 00:33:23,400
Speaker 1: player UM knows the difference, and then he paused that

570
00:33:23,640 --> 00:33:30,120
Speaker 1: maybe it's more like UM or indeed two, how granular

571
00:33:30,160 --> 00:33:32,120
Speaker 1: can you get in poker? Well, poker is a game

572
00:33:32,160 --> 00:33:35,120
Speaker 1: with repeated play, quick clear feedback. It's possible to get

573
00:33:35,160 --> 00:33:37,880
Speaker 1: more granular on poker than it is about the location

574
00:33:38,240 --> 00:33:40,240
Speaker 1: of terrorists or about whether countries are going to leave

575
00:33:40,240 --> 00:33:43,560
Speaker 1: the Eurozone. But it's an open question of how granular

576
00:33:43,720 --> 00:33:46,600
Speaker 1: you can get UM. And you need to grapple with

577
00:33:46,640 --> 00:33:51,440
Speaker 1: this distinction between precision and pseudo precision UM. And that's

578
00:33:51,440 --> 00:33:54,360
Speaker 1: one of the things. Super forecasters are just very thoughtful

579
00:33:54,400 --> 00:33:57,200
Speaker 1: people who pushed the frontiers of knowledge as far as

580
00:33:57,200 --> 00:33:59,520
Speaker 1: they can, and that means sometimes pushing them a little

581
00:33:59,520 --> 00:34:02,160
Speaker 1: too far, in which case they retreat. If people want

582
00:34:02,160 --> 00:34:04,800
Speaker 1: to find your work, just google Philip Tetlock and they'll

583
00:34:04,800 --> 00:34:09,279
Speaker 1: be able to dig up all of your various publications, books, writings, etcetera. Yeah,

584
00:34:09,320 --> 00:34:12,160
Speaker 1: Google scholars probably a little faster, but all right. If

585
00:34:12,200 --> 00:34:14,839
Speaker 1: you've enjoyed this conversation, be sure and check out all

586
00:34:14,920 --> 00:34:18,879
Speaker 1: eighty three or so of our prior conversations. Be sure

587
00:34:18,920 --> 00:34:21,279
Speaker 1: and follow me on Twitter at rid Halts and check

588
00:34:21,320 --> 00:34:24,960
Speaker 1: out my daily column on Bloomberg View dot com. I'm

589
00:34:25,000 --> 00:34:28,760
Speaker 1: Barry Ridhults. You're listening to Masters in Business on Bloomberg Radio.

590
00:34:28,960 --> 00:34:32,200
Speaker 1: Welcome to the podcast. This is Barry Ridhills, Professor Tetlock.

591
00:34:32,239 --> 00:34:34,520
Speaker 1: If I don't remember to say this later, thank you

592
00:34:34,600 --> 00:34:36,719
Speaker 1: so much for being so generous with your time. This

593
00:34:36,800 --> 00:34:40,759
Speaker 1: is really um been a fascinating conversation. I have a

594
00:34:40,760 --> 00:34:43,879
Speaker 1: lot of things to go over with you in the

595
00:34:44,000 --> 00:34:47,480
Speaker 1: last twenty minutes or so we have, but there are

596
00:34:47,480 --> 00:34:50,319
Speaker 1: a few questions that I'm just dying to ask you

597
00:34:50,360 --> 00:34:56,360
Speaker 1: because it's your lat previous book really was very influential

598
00:34:56,920 --> 00:35:00,280
Speaker 1: to me on expert political judgment. It was that book,

599
00:35:00,480 --> 00:35:03,800
Speaker 1: and it was a prior book called The Fortune Sellers

600
00:35:03,840 --> 00:35:06,960
Speaker 1: that really was more of a media criticism of this

601
00:35:07,160 --> 00:35:11,120
Speaker 1: parade of people who would come through the studios make

602
00:35:11,200 --> 00:35:17,040
Speaker 1: their outland just forecast, be completely wrong, never be held accountable,

603
00:35:17,040 --> 00:35:20,319
Speaker 1: and then they would get called back and the more outlandish, uh,

604
00:35:20,440 --> 00:35:23,200
Speaker 1: the better there were. There was an author who wrote

605
00:35:23,200 --> 00:35:26,400
Speaker 1: a book I'm trying to remember what year the book was.

606
00:35:26,440 --> 00:35:30,759
Speaker 1: It was called The Tao Jones T. A. O. Bennett Goodspeed,

607
00:35:30,760 --> 00:35:36,840
Speaker 1: and he called these folks the articulate incompetence plural, meaning

608
00:35:36,880 --> 00:35:40,120
Speaker 1: that they're very good salespeople. They can speak, but really

609
00:35:40,160 --> 00:35:43,439
Speaker 1: they have no expert knowledge. And if you've spent any

610
00:35:43,480 --> 00:35:48,000
Speaker 1: time in green rooms in various television studios, there's something

611
00:35:48,040 --> 00:35:53,319
Speaker 1: to that. So let me ask the basic question that

612
00:35:53,360 --> 00:35:58,000
Speaker 1: we kind of skirted around during the broadcast portion. Why

613
00:35:58,040 --> 00:36:03,919
Speaker 1: are we so enamored with forecasting and forecasters despite their

614
00:36:04,040 --> 00:36:09,520
Speaker 1: terrible track records. Well, I think because there's a lot

615
00:36:09,520 --> 00:36:13,880
Speaker 1: of motivated reasoning going on. As we noted earlier, there's

616
00:36:13,880 --> 00:36:15,680
Speaker 1: this tendency to use a lot of a do a

617
00:36:15,719 --> 00:36:18,400
Speaker 1: lot of a verbiage forecasting, to to paint a dramatic

618
00:36:18,440 --> 00:36:21,680
Speaker 1: scenario and then hold it together with some very weak

619
00:36:21,840 --> 00:36:24,680
Speaker 1: verbs like this might happen or could happen or the

620
00:36:24,719 --> 00:36:28,759
Speaker 1: distinct possibility of this happening. Um. So, there's this interesting

621
00:36:28,800 --> 00:36:32,560
Speaker 1: tendency that the pundits have of engaging our attention with

622
00:36:32,600 --> 00:36:37,439
Speaker 1: a vivid scenario disintegration of the Saudi regime or you know, uh,

623
00:36:37,719 --> 00:36:42,200
Speaker 1: Cino Japanese wars, something Tom Clancy issue on that kind

624
00:36:42,239 --> 00:36:44,840
Speaker 1: of scale, um and and and but to stitch it

625
00:36:44,880 --> 00:36:48,160
Speaker 1: all together with terms weasel weasel word terms that allow

626
00:36:48,239 --> 00:36:53,360
Speaker 1: them to retreat later on. And UM, we don't distinguish

627
00:36:53,440 --> 00:36:55,680
Speaker 1: very clearly in our own minds. We we don't think

628
00:36:55,680 --> 00:36:57,440
Speaker 1: we want to hold as they say, all of the

629
00:36:57,440 --> 00:37:00,960
Speaker 1: fault lies with the pundits. They couldn't do this unless

630
00:37:01,040 --> 00:37:04,239
Speaker 1: unless we were willing partners. UM. And I think that

631
00:37:04,560 --> 00:37:07,400
Speaker 1: you know here here I am talking to radio station

632
00:37:07,440 --> 00:37:11,680
Speaker 1: one of the most influential companies in the world, Bloomberg. UM.

633
00:37:11,760 --> 00:37:18,560
Speaker 1: Bloomberg is a major purchaser of expertise. UM. Bloomberg could

634
00:37:18,600 --> 00:37:22,279
Speaker 1: actually change the world to some degree if it implemented

635
00:37:22,400 --> 00:37:27,239
Speaker 1: systematic uh, if it implemented systems for tracking the accuracy

636
00:37:27,239 --> 00:37:29,560
Speaker 1: of many of the people who came through. If part

637
00:37:29,600 --> 00:37:32,120
Speaker 1: of the price for getting onto Bloomberg was that you

638
00:37:32,160 --> 00:37:34,920
Speaker 1: had to demonstrate that you were engaging in some kind

639
00:37:34,920 --> 00:37:37,799
Speaker 1: of rigorous scorekeeping, and Bloomberg could flash up some of

640
00:37:37,840 --> 00:37:41,560
Speaker 1: some batting average statistics. UM, as you as you appear,

641
00:37:42,160 --> 00:37:46,160
Speaker 1: Um you Bloomberg could increase the collective IQ of our society.

642
00:37:46,560 --> 00:37:50,120
Speaker 1: It could increase the collective IQ of the conversation. Um.

643
00:37:50,280 --> 00:37:54,160
Speaker 1: When most pundits stay away, Well, well that's the question

644
00:37:54,200 --> 00:37:56,880
Speaker 1: I mean. I I say Bloomberg because Bloomberg is so influential,

645
00:37:56,920 --> 00:37:58,440
Speaker 1: I think a lot of a lot of punets would say, well,

646
00:37:58,440 --> 00:37:59,680
Speaker 1: I'm not I'm not going to run away and high

647
00:37:59,719 --> 00:38:02,600
Speaker 1: from blue Burg. But the other media could do this

648
00:38:02,640 --> 00:38:04,440
Speaker 1: to the Wall Street Journal in New York Times. There

649
00:38:04,440 --> 00:38:07,600
Speaker 1: are lots of major media science that have the leverage

650
00:38:08,120 --> 00:38:11,120
Speaker 1: that could induce pundits to be much more intellectually honest.

651
00:38:11,440 --> 00:38:16,280
Speaker 1: They choose not to exercise that option. Um probably because

652
00:38:16,320 --> 00:38:19,560
Speaker 1: they don't perceive a great market demand for it. We

653
00:38:19,800 --> 00:38:24,319
Speaker 1: were talking during the break about my usage of a

654
00:38:24,360 --> 00:38:27,480
Speaker 1: little app called follow up then dot com. Whenever I

655
00:38:27,520 --> 00:38:31,040
Speaker 1: see an outregeous forecast, I just shoot an email to

656
00:38:31,160 --> 00:38:35,160
Speaker 1: a specific date, so Gold going to five thousand dollars,

657
00:38:35,200 --> 00:38:38,759
Speaker 1: and I send the forecast. I send the email out

658
00:38:38,880 --> 00:38:41,879
Speaker 1: to Oh, well, let's let's let's give them a year,

659
00:38:42,280 --> 00:38:45,560
Speaker 1: so we'll send a forecast out an email out March

660
00:38:45,640 --> 00:38:49,240
Speaker 1: one at follow up then dot com with the headline

661
00:38:49,280 --> 00:38:53,000
Speaker 1: and the web address of the article that made this forecast.

662
00:38:53,440 --> 00:38:57,319
Speaker 1: And then a year later, or if I write March one,

663
00:38:57,360 --> 00:39:01,800
Speaker 1: five years later, comes the email back that specifically gives

664
00:39:01,800 --> 00:39:04,239
Speaker 1: me that lank. Oh it's a reminder. Here's what this

665
00:39:04,280 --> 00:39:08,200
Speaker 1: person said a year ago. And occasionally I get to

666
00:39:08,239 --> 00:39:11,960
Speaker 1: do an article about, Hey, here's a wild forecast that

667
00:39:12,239 --> 00:39:16,280
Speaker 1: someone made and it's been completely wrong. So let's talk

668
00:39:16,800 --> 00:39:19,960
Speaker 1: about your book. You kind of call me out for

669
00:39:20,560 --> 00:39:24,400
Speaker 1: calling someone else out, and I'm curious as to your

670
00:39:24,440 --> 00:39:28,120
Speaker 1: perspective on this. So in two thousand and ten, when

671
00:39:28,120 --> 00:39:32,080
Speaker 1: the FED was in the midst of doing quantitative easing, uh,

672
00:39:32,120 --> 00:39:35,520
Speaker 1: there was a letter published and I believe online and

673
00:39:35,640 --> 00:39:37,279
Speaker 1: it ran in the Wall Street Journal in a number

674
00:39:37,320 --> 00:39:41,120
Speaker 1: of places warning that quantitative easing was going to cause

675
00:39:41,760 --> 00:39:44,879
Speaker 1: hyper inflation and collapse of the dollar and all these

676
00:39:44,960 --> 00:39:48,040
Speaker 1: terrible things. And so I figured, you've got to give

677
00:39:48,080 --> 00:39:53,279
Speaker 1: those people three years. So I set a reminder for

678
00:39:53,320 --> 00:39:56,439
Speaker 1: three years later. And three years later it popped up. Hey,

679
00:39:56,560 --> 00:39:59,480
Speaker 1: the dollars at multi year highs. There there is no

680
00:39:59,600 --> 00:40:03,560
Speaker 1: hyper inflation, there's there's deflation. These guys were wrong, and

681
00:40:03,600 --> 00:40:05,560
Speaker 1: so I called them out about it, and it went

682
00:40:06,040 --> 00:40:09,400
Speaker 1: totally viral, got picked up by a dozen different media outlets,

683
00:40:09,800 --> 00:40:13,560
Speaker 1: and a book called super Forecasting. Now at by today

684
00:40:13,760 --> 00:40:17,440
Speaker 1: we're six years forward, um, and we still have a

685
00:40:17,480 --> 00:40:23,360
Speaker 1: strong dollar and and no inflation. What is the issue

686
00:40:23,520 --> 00:40:29,480
Speaker 1: with an ambiguous forecast? With no specific I described forecasts

687
00:40:29,520 --> 00:40:33,239
Speaker 1: as an asset class, a price, and a specific date.

688
00:40:34,160 --> 00:40:36,919
Speaker 1: If you leave out the specific date, do you get

689
00:40:36,960 --> 00:40:40,879
Speaker 1: to say we're never wrong because there's just wait, you'll see.

690
00:40:41,000 --> 00:40:44,720
Speaker 1: Is is that a fair defense of that? It's definitely

691
00:40:44,760 --> 00:40:47,319
Speaker 1: not fair, but it it is the state of the

692
00:40:47,440 --> 00:40:49,839
Speaker 1: art at the moment. You know. There's an old communism

693
00:40:49,920 --> 00:40:53,040
Speaker 1: joke that that that's rather a proposed here um the

694
00:40:53,800 --> 00:40:57,400
Speaker 1: Soviet revolutionarily on Trotsky after he was thrown out of

695
00:40:57,400 --> 00:41:00,000
Speaker 1: the Soviet Union by Stalin, when it went around giving

696
00:41:00,040 --> 00:41:03,480
Speaker 1: talks to the left wing audiences around the world, and um,

697
00:41:03,560 --> 00:41:06,560
Speaker 1: one probably apocryphal story has it that he went once

698
00:41:06,600 --> 00:41:09,960
Speaker 1: when he was introduced to an audience of followers. Uh,

699
00:41:10,400 --> 00:41:14,200
Speaker 1: the speaker said it was was proclaiming Leon trotskya visionary

700
00:41:14,239 --> 00:41:17,000
Speaker 1: who could see far, far into the future. And you're saying,

701
00:41:17,000 --> 00:41:19,080
Speaker 1: and you know, comrades, the ultimate proof of the far

702
00:41:19,200 --> 00:41:22,560
Speaker 1: sightedness of of of of comrade Trotsky, not one of

703
00:41:22,600 --> 00:41:26,480
Speaker 1: his predictions has yet come true. That's how far sighted

704
00:41:26,520 --> 00:41:29,880
Speaker 1: he is. The the old joke about market forecasting is

705
00:41:30,239 --> 00:41:32,719
Speaker 1: you could give a price level or a date, but

706
00:41:32,800 --> 00:41:36,440
Speaker 1: never both at once. And that's just another way. So

707
00:41:36,440 --> 00:41:39,760
Speaker 1: so how long do we allow a forecast to persist

708
00:41:39,800 --> 00:41:42,560
Speaker 1: before we say, all right, at this point it's been

709
00:41:42,840 --> 00:41:45,560
Speaker 1: X number of years, you we're gonna have to put

710
00:41:45,560 --> 00:41:52,000
Speaker 1: you in the incorrect column. Well, there's there's no absolute rule,

711
00:41:52,640 --> 00:41:55,120
Speaker 1: because that's that's the way vague verbiage is vague. Verbiage

712
00:41:55,200 --> 00:41:59,960
Speaker 1: is vague because it's it's just it's it's always a slippery,

713
00:42:00,120 --> 00:42:04,400
Speaker 1: vague um thing that no nobody knows how to quantify

714
00:42:04,480 --> 00:42:07,400
Speaker 1: it and and keeps it, keeps the pundit safe. Um.

715
00:42:07,440 --> 00:42:09,520
Speaker 1: I mean, I have lots of forecasters from the earlier

716
00:42:09,560 --> 00:42:13,040
Speaker 1: work who predicted that Canada would disintegrate, or Nigeria would disintegrate,

717
00:42:13,120 --> 00:42:15,640
Speaker 1: or there are a lot of disintegration scenarios out there

718
00:42:15,640 --> 00:42:18,600
Speaker 1: that haven't happened yet. Um. And if you call them

719
00:42:18,640 --> 00:42:21,399
Speaker 1: on it, are they going to say, well, we're it's

720
00:42:21,520 --> 00:42:25,399
Speaker 1: it's only seen it could still happen, absolutely and and

721
00:42:25,600 --> 00:42:29,560
Speaker 1: wholeheartedly believe it. That's right. So how much of this

722
00:42:29,640 --> 00:42:33,880
Speaker 1: is just simply humans not being immune to human behavior?

723
00:42:34,440 --> 00:42:37,839
Speaker 1: You're a psychologist, Let's let's let's take that tact. Is

724
00:42:37,880 --> 00:42:43,080
Speaker 1: this just ordinary human behavior refusal to accept responsibility for error,

725
00:42:43,120 --> 00:42:46,160
Speaker 1: not wanting to admit being wrong, not wanting to do

726
00:42:46,200 --> 00:42:52,080
Speaker 1: anything that reduces their potential um status within the hierarchy.

727
00:42:52,120 --> 00:42:55,160
Speaker 1: Is that all this is well, we're moving into world

728
00:42:55,160 --> 00:42:58,320
Speaker 1: that's requiring us to make ever subtler distinctions among degrees

729
00:42:58,320 --> 00:43:00,759
Speaker 1: of uncertainty. This orts of distinction as we didn't have

730
00:43:00,840 --> 00:43:04,080
Speaker 1: to make in our evolutionary past um when we were

731
00:43:04,080 --> 00:43:06,359
Speaker 1: wandering the savannah plains of Africa. You know there were

732
00:43:06,480 --> 00:43:08,520
Speaker 1: are there is a lion or isn't a lions during

733
00:43:08,520 --> 00:43:10,920
Speaker 1: in the long grass and you're gonna you're gonna make

734
00:43:10,920 --> 00:43:12,680
Speaker 1: you have to make a judgment call pretty darn quickly.

735
00:43:12,680 --> 00:43:14,400
Speaker 1: And if you dontle very long, and maybe you're not

736
00:43:14,480 --> 00:43:17,080
Speaker 1: likely to pass your genes onto the next generation. You're

737
00:43:17,120 --> 00:43:21,359
Speaker 1: better off being wrong but jumping the gun then having

738
00:43:21,400 --> 00:43:24,479
Speaker 1: a higher, better track record. But if you're wrong once,

739
00:43:24,520 --> 00:43:28,160
Speaker 1: well then it's catastrophic in terms of progeny. And that

740
00:43:28,200 --> 00:43:31,360
Speaker 1: makes a lot of sense. That sort of thinking is

741
00:43:31,400 --> 00:43:33,520
Speaker 1: why we have a tendency to do all sorts of

742
00:43:33,560 --> 00:43:38,680
Speaker 1: things that just are inappropriate in investing but worked really

743
00:43:38,680 --> 00:43:44,600
Speaker 1: well way back when that's right and um, if you

744
00:43:44,640 --> 00:43:47,880
Speaker 1: can imagine a scenario, here's the here's the big problem

745
00:43:47,960 --> 00:43:52,240
Speaker 1: with tail risks and scenarios. Um, most of the time

746
00:43:52,440 --> 00:43:56,319
Speaker 1: people underrate them, but as soon as the scenario has

747
00:43:56,360 --> 00:43:59,800
Speaker 1: called to their attention, they overrated. Well, it's very very

748
00:44:00,120 --> 00:44:02,880
Speaker 1: caul for people to strike the right balance in dealing

749
00:44:02,880 --> 00:44:06,080
Speaker 1: with tailor risk scenario. So so that whole recency effect

750
00:44:06,160 --> 00:44:09,680
Speaker 1: thing is since nobody, very few people were forecasting the

751
00:44:09,800 --> 00:44:13,160
Speaker 1: sort of financial crisis we had in O eight oh nine,

752
00:44:13,719 --> 00:44:18,520
Speaker 1: and since then it's been nothing but catastrophic forecasts from

753
00:44:18,960 --> 00:44:21,880
Speaker 1: the recession never ended, We're going to turn into a depression.

754
00:44:22,160 --> 00:44:25,880
Speaker 1: Here comes another eight seven like stock crash, the parade

755
00:44:25,880 --> 00:44:30,239
Speaker 1: of horribles just have not now auto loans and the

756
00:44:30,280 --> 00:44:33,960
Speaker 1: new subprime it's going to be just like, uh, is

757
00:44:34,000 --> 00:44:37,000
Speaker 1: this just that that tel risk factor is so recent

758
00:44:37,000 --> 00:44:39,799
Speaker 1: in people's minds and they missed it coming, and so

759
00:44:39,880 --> 00:44:42,719
Speaker 1: now they're just like every general fights the last war.

760
00:44:43,120 --> 00:44:46,560
Speaker 1: These folks are still fighting the previous financial crisis and

761
00:44:46,600 --> 00:44:49,680
Speaker 1: the diplomats try to avoid the last war. Um, yeah,

762
00:44:49,800 --> 00:44:52,440
Speaker 1: I think that's right. I mean, you are you you

763
00:44:52,680 --> 00:44:55,279
Speaker 1: you go from Iraq to Syrian Libya or you that

764
00:44:55,400 --> 00:44:58,440
Speaker 1: you go from one error to another. And Iraq was

765
00:44:58,520 --> 00:45:01,520
Speaker 1: far enough after the Aetnam that it looked like all

766
00:45:01,560 --> 00:45:04,120
Speaker 1: it takes as a generation before those lessons are lost,

767
00:45:04,520 --> 00:45:06,560
Speaker 1: more or less yet more or less. All right, so

768
00:45:06,640 --> 00:45:10,560
Speaker 1: I only have you for another ten minutes, and um,

769
00:45:10,600 --> 00:45:14,320
Speaker 1: my last question before I jumped to my my standard questions.

770
00:45:14,520 --> 00:45:18,160
Speaker 1: Burton Malkiel said, when investors moved from stock to stock

771
00:45:18,360 --> 00:45:21,600
Speaker 1: or mutual fund to funds as if they were selecting

772
00:45:21,640 --> 00:45:25,719
Speaker 1: and discarding cards in a game of jin rummie, what

773
00:45:25,800 --> 00:45:30,680
Speaker 1: does this tell us about human's ability to participate in

774
00:45:30,920 --> 00:45:34,160
Speaker 1: uncertain equity markets? Well, this is another one of these

775
00:45:34,400 --> 00:45:38,400
Speaker 1: the ten commandments that we formulated from observing the super forecasters.

776
00:45:38,440 --> 00:45:42,880
Speaker 1: They're acutely aware of the principle of error balancing. If

777
00:45:42,920 --> 00:45:45,400
Speaker 1: you look at the research literature on human judgment, there

778
00:45:45,440 --> 00:45:47,120
Speaker 1: are two kinds of errors people can make in that

779
00:45:47,239 --> 00:45:50,239
Speaker 1: situation you're describing. One of them is the error of

780
00:45:50,440 --> 00:45:54,520
Speaker 1: excess of volatility, of um jumping every time there's a

781
00:45:54,560 --> 00:45:58,200
Speaker 1: little bit of news and exaggerating its diagnostic value visa

782
00:45:58,520 --> 00:46:01,839
Speaker 1: deep market trends. And the other big mistake you can

783
00:46:01,880 --> 00:46:04,960
Speaker 1: make is excessive rigidity and being so committed to a

784
00:46:05,000 --> 00:46:07,400
Speaker 1: particular preconception about where the future is going that you

785
00:46:07,440 --> 00:46:09,440
Speaker 1: just ignore the news altogether. And you don't you fit,

786
00:46:09,760 --> 00:46:12,480
Speaker 1: you fit, you failed to do any updating. So it's

787
00:46:12,520 --> 00:46:15,040
Speaker 1: it's it's kind of principal error balance things. But like

788
00:46:15,040 --> 00:46:17,839
Speaker 1: writing learning how to ride a bicycle. Um, I mean,

789
00:46:17,840 --> 00:46:21,080
Speaker 1: I could talk for hours about the principles of error

790
00:46:21,160 --> 00:46:23,279
Speaker 1: balancing and everybody is like, yeah, yeah, I kind of

791
00:46:23,320 --> 00:46:24,719
Speaker 1: get at Sure you can make one air, you can

792
00:46:24,760 --> 00:46:27,080
Speaker 1: make the other. But the only way it's really going

793
00:46:27,120 --> 00:46:29,239
Speaker 1: to sink into people's heads if they is if they

794
00:46:29,280 --> 00:46:32,600
Speaker 1: go to forecasting tournaments and they actually practice making judgments,

795
00:46:32,640 --> 00:46:34,680
Speaker 1: get on the bike and try to ride it. Uh.

796
00:46:34,760 --> 00:46:37,719
Speaker 1: And that's what going to forecasting tournaments all about. That

797
00:46:37,719 --> 00:46:39,960
Speaker 1: that's why we were continuing to run the gj open

798
00:46:40,040 --> 00:46:42,520
Speaker 1: dot com, which is a forecasting tournament where people can

799
00:46:42,520 --> 00:46:46,640
Speaker 1: indeed uh work to cultivate their skills. That sounds pretty

800
00:46:46,680 --> 00:46:52,600
Speaker 1: pretty fascinating. So that's that's really interesting for that. Actually,

801
00:46:52,600 --> 00:46:53,960
Speaker 1: we have a little more time, so I'm going to

802
00:46:54,080 --> 00:46:58,360
Speaker 1: keep bangalway on some of these questions. In the first

803
00:46:58,400 --> 00:47:02,040
Speaker 1: book on in the previous book on Expert Political Judgment,

804
00:47:03,440 --> 00:47:06,320
Speaker 1: you know, I look at these is really two sides

805
00:47:06,880 --> 00:47:11,160
Speaker 1: to the same coin. The first book talks about what

806
00:47:11,360 --> 00:47:16,080
Speaker 1: is essentially long term forecasts really a year and further out,

807
00:47:16,880 --> 00:47:20,120
Speaker 1: and they tend to be wrong. The book on super

808
00:47:20,160 --> 00:47:23,600
Speaker 1: forecasting is really looking at a year or less. So

809
00:47:23,680 --> 00:47:27,279
Speaker 1: are are they really saying two different things where we're

810
00:47:27,280 --> 00:47:31,680
Speaker 1: really looking at two different types of forecast, two different lengths.

811
00:47:31,920 --> 00:47:36,280
Speaker 1: I think that's a superb point. There are different time

812
00:47:36,280 --> 00:47:39,920
Speaker 1: periods in the different studies. UM. I'm much more optimistic

813
00:47:40,080 --> 00:47:43,560
Speaker 1: that we can improve forecasting using the right selection, training,

814
00:47:43,640 --> 00:47:48,439
Speaker 1: teaming tools in shorter time periods up to about a year.

815
00:47:49,239 --> 00:47:52,000
Speaker 1: I become progressively more pessimistic when you go out to

816
00:47:52,080 --> 00:47:54,600
Speaker 1: the longer reaches of three, five, ten years that were

817
00:47:54,600 --> 00:47:58,399
Speaker 1: included in Expert Political Judgment, and there I think it's

818
00:47:58,480 --> 00:48:01,919
Speaker 1: it's going to be very, very difficult to do much better, UM,

819
00:48:02,400 --> 00:48:03,960
Speaker 1: do much better than chance. How many times do you

820
00:48:04,000 --> 00:48:08,120
Speaker 1: have to shuffle a deck of cards, um, until it's

821
00:48:08,160 --> 00:48:12,160
Speaker 1: perfectly random? That's a good question. Well, I think the

822
00:48:12,480 --> 00:48:17,040
Speaker 1: UM I would guess five. The statistician and magician Percy

823
00:48:17,120 --> 00:48:21,200
Speaker 1: dot Com is Stanford. I think he estimated at seven. Well,

824
00:48:21,520 --> 00:48:24,440
Speaker 1: life is like shuffling the cards? How how many months,

825
00:48:24,440 --> 00:48:26,880
Speaker 1: how many years have to go by before so many

826
00:48:27,200 --> 00:48:31,840
Speaker 1: random contingencies accumulate that no no human being could conceivably

827
00:48:31,840 --> 00:48:36,480
Speaker 1: have anticipated anything that far out. And um O our

828
00:48:36,520 --> 00:48:40,120
Speaker 1: current best guests for the kinds of geopolitical geoeconomic questions

829
00:48:40,120 --> 00:48:41,879
Speaker 1: we're looking at, it is around a year or so.

830
00:48:42,719 --> 00:48:48,480
Speaker 1: Isn't that just the nature of society and a complex

831
00:48:48,480 --> 00:48:55,000
Speaker 1: system such as fill in the blank, stock markets, economy, geopolitics, elections,

832
00:48:55,040 --> 00:49:01,040
Speaker 1: anything along those lines. They're so sensitive to an all conditions,

833
00:49:01,360 --> 00:49:04,200
Speaker 1: they're nonlinear that you end up with a little change

834
00:49:04,239 --> 00:49:08,720
Speaker 1: here has out sized impact further down the road. Really,

835
00:49:08,760 --> 00:49:11,880
Speaker 1: what we're saying is the universe is pretty random beyond

836
00:49:11,920 --> 00:49:17,719
Speaker 1: twelve months. Uh, it's practically impossible to make any sort

837
00:49:17,760 --> 00:49:23,200
Speaker 1: of realistic forecast with any degree of specificity. Um about

838
00:49:23,560 --> 00:49:26,399
Speaker 1: certain things. I made a bet four years ago with

839
00:49:26,560 --> 00:49:28,680
Speaker 1: um I won't mention his name, but he's been a

840
00:49:28,719 --> 00:49:31,799
Speaker 1: guest on the show as to I was wondering who

841
00:49:31,960 --> 00:49:36,520
Speaker 1: was the GOP nominee likely to be? This is literally

842
00:49:36,640 --> 00:49:42,480
Speaker 1: three years ago after after the election, and he said, well,

843
00:49:42,560 --> 00:49:44,560
Speaker 1: what about the Democrats? And I said, well, it's easy,

844
00:49:44,600 --> 00:49:46,960
Speaker 1: that'll be Hillary, but who I have no idea who

845
00:49:46,960 --> 00:49:49,600
Speaker 1: the Republican is going to be. So we made a bet,

846
00:49:49,640 --> 00:49:54,120
Speaker 1: and so far looking pretty good. Um. But I don't

847
00:49:54,120 --> 00:49:56,640
Speaker 1: know if I got lucky, just got lucky and assumed

848
00:49:56,680 --> 00:49:59,799
Speaker 1: it was she was up next, she was next in line. Uh.

849
00:50:00,400 --> 00:50:03,759
Speaker 1: Under normal circumstances, when you're looking out to three or

850
00:50:03,800 --> 00:50:08,160
Speaker 1: four years, there are so many contingencies. Can anybody really

851
00:50:08,680 --> 00:50:11,640
Speaker 1: I got lucky. I'm not pretending I have any expertise

852
00:50:11,960 --> 00:50:16,840
Speaker 1: in politics or anything else, But can anybody consistently have

853
00:50:17,080 --> 00:50:22,040
Speaker 1: any sort of acumen thinking out more than twelve months?

854
00:50:23,080 --> 00:50:24,920
Speaker 1: It's it's really hard. I mean, there are there are

855
00:50:24,960 --> 00:50:29,880
Speaker 1: some categories of questions where great longer foresight is possible.

856
00:50:30,000 --> 00:50:32,399
Speaker 1: I think the Hillary thing was in the cards for

857
00:50:32,400 --> 00:50:35,719
Speaker 1: for for for quite a long time. Um. But for

858
00:50:35,840 --> 00:50:37,719
Speaker 1: most of these things, I mean, who mean to take

859
00:50:37,760 --> 00:50:41,000
Speaker 1: some extreme examples, I mean, in nineteen forty Dwight Eisenhower

860
00:50:41,040 --> 00:50:44,239
Speaker 1: was an anonymous Army colonel. In nineteen fifty two years

861
00:50:44,280 --> 00:50:48,880
Speaker 1: president United States in twelve years, right, uh in um Um.

862
00:50:49,000 --> 00:50:52,680
Speaker 1: Jimmy Carter was an anonymous peanut farmer in the nineteen

863
00:50:52,719 --> 00:50:56,000
Speaker 1: sixty four. In nineteen seventy six, he was being elected

864
00:50:56,000 --> 00:50:59,400
Speaker 1: president United States. Twelve years is a huge amount of

865
00:50:59,440 --> 00:51:03,040
Speaker 1: time in politics, So I don't think anybody really is

866
00:51:03,040 --> 00:51:07,000
Speaker 1: going to suppose there's very much possibility there UM, But

867
00:51:07,320 --> 00:51:09,120
Speaker 1: as you get closer and closer, it gets more and

868
00:51:09,160 --> 00:51:11,760
Speaker 1: more possible. That's not all that surprising. It's an analogy.

869
00:51:11,800 --> 00:51:13,319
Speaker 1: Will be like to Snell and I char when you

870
00:51:13,360 --> 00:51:16,160
Speaker 1: visit your optometrist and engage, it's easier and easier the

871
00:51:16,160 --> 00:51:19,720
Speaker 1: closer up you get for most things. UM. The trouble

872
00:51:19,800 --> 00:51:22,799
Speaker 1: is we're just not very well tuned to the parameters UM.

873
00:51:22,960 --> 00:51:24,879
Speaker 1: And if somebody can tell a really good story about

874
00:51:24,880 --> 00:51:27,360
Speaker 1: a relatively far off future about you know, the United

875
00:51:27,360 --> 00:51:30,840
Speaker 1: States is moving toward a techno utopia and which DDP

876
00:51:31,000 --> 00:51:35,600
Speaker 1: will will skyrocket as intelligent machines do amazing things for

877
00:51:35,680 --> 00:51:38,319
Speaker 1: us in the fourth and Dulsta Revolution. That that meant

878
00:51:38,360 --> 00:51:44,319
Speaker 1: may that scenario may indeed materialized by its UM, but

879
00:51:44,480 --> 00:51:49,400
Speaker 1: the likelihood of scenarios of that sort being accurate UM

880
00:51:49,560 --> 00:51:53,000
Speaker 1: is extremely low. I have a t shirt at home.

881
00:51:53,760 --> 00:51:56,400
Speaker 1: It says Where's my jet pack? I was promised the

882
00:51:56,480 --> 00:52:01,680
Speaker 1: jet pack by the year two thousand under forty four characters. Right.

883
00:52:01,800 --> 00:52:06,239
Speaker 1: That's right. When when you look back at future forecasts

884
00:52:06,280 --> 00:52:09,879
Speaker 1: from decades ago, and we now have enough for them

885
00:52:09,880 --> 00:52:14,120
Speaker 1: that we can look back years as to what people

886
00:52:14,160 --> 00:52:18,319
Speaker 1: were expecting from the future. What's fascinating is all the

887
00:52:18,400 --> 00:52:24,800
Speaker 1: amazing technology, technological developments, all the advantages of hardware, software, biotechnology,

888
00:52:24,840 --> 00:52:29,040
Speaker 1: medicine that we practically take for granted. They weren't the

889
00:52:29,080 --> 00:52:32,640
Speaker 1: things that people were forecasting. It was colonizing Mars and

890
00:52:33,160 --> 00:52:37,879
Speaker 1: other sort of hoverboards and other such things. Uh So,

891
00:52:38,040 --> 00:52:42,400
Speaker 1: even when you're thinking in terms of giant technological changes,

892
00:52:42,800 --> 00:52:44,840
Speaker 1: and of course there's a handful of people, you know,

893
00:52:45,000 --> 00:52:48,719
Speaker 1: Arthur C. Clarke is notorious for having forecast everything from

894
00:52:48,719 --> 00:52:53,879
Speaker 1: cell phones to satellites to to what have you. Um

895
00:52:53,960 --> 00:52:58,759
Speaker 1: what does this say about our ability to understand the few,

896
00:52:58,960 --> 00:53:02,440
Speaker 1: the present and extra appelate to the future. What it

897
00:53:02,480 --> 00:53:05,600
Speaker 1: suggests is we'd be better off if we were aware

898
00:53:05,600 --> 00:53:12,200
Speaker 1: of our limitations, achieved a certain baseline of appropriate humility,

899
00:53:12,520 --> 00:53:16,200
Speaker 1: and got in the habit of keeping score, and resisted

900
00:53:16,239 --> 00:53:21,680
Speaker 1: being sucked into clever scenarios and storytellers, and resisted being

901
00:53:22,080 --> 00:53:26,360
Speaker 1: seduced by credentials. If we could manage to do those things,

902
00:53:26,400 --> 00:53:30,480
Speaker 1: I think we would um proceed through life, making investments

903
00:53:30,480 --> 00:53:34,200
Speaker 1: and political decisions with better calibrated probabilities. And I think

904
00:53:34,200 --> 00:53:36,520
Speaker 1: we would be better office individuals and we'd be better

905
00:53:36,520 --> 00:53:40,360
Speaker 1: officers of society. That sounds that sounds tremendous. On a

906
00:53:40,400 --> 00:53:43,600
Speaker 1: related note to that, because because those those seven bullet

907
00:53:43,600 --> 00:53:48,399
Speaker 1: points are very significant. Let's let's talk about uncertainty, which

908
00:53:48,440 --> 00:53:53,200
Speaker 1: you is a is a uh concept that is dotted

909
00:53:53,239 --> 00:53:58,239
Speaker 1: throughout actually both books. Um, what is uncertainty and what

910
00:53:58,280 --> 00:54:02,160
Speaker 1: does it mean for individuals just trying to navigate their

911
00:54:02,200 --> 00:54:05,799
Speaker 1: way through the world. Do we understand uncertainty? Uh? Do

912
00:54:05,840 --> 00:54:08,920
Speaker 1: we misunderstand it? What? What exactly is it relative to

913
00:54:09,520 --> 00:54:16,480
Speaker 1: thinking about the future? Well, um, there are some types

914
00:54:16,520 --> 00:54:22,000
Speaker 1: of problems where the probabilities can be readily computed. We

915
00:54:22,040 --> 00:54:24,640
Speaker 1: can compute the probability of drawing an asis spades from

916
00:54:24,640 --> 00:54:29,200
Speaker 1: a randomly shuffled deck, um very accurately, uh, too many

917
00:54:29,200 --> 00:54:31,600
Speaker 1: decimal points if we want whatever one out of over

918
00:54:31,680 --> 00:54:34,440
Speaker 1: fifty two works out too. We can do that with

919
00:54:34,520 --> 00:54:37,920
Speaker 1: coin toss games and so forth. Um. So there are

920
00:54:38,000 --> 00:54:41,960
Speaker 1: some games in which the classic rules of statistics want

921
00:54:42,000 --> 00:54:46,400
Speaker 1: oh one very clearly apply and there are well defined probabilities.

922
00:54:46,520 --> 00:54:50,080
Speaker 1: In other words, we know what the range of outcomes are.

923
00:54:50,239 --> 00:54:52,279
Speaker 1: We just don't know what the specific outcome is going

924
00:54:52,320 --> 00:54:55,400
Speaker 1: to be. Got a well defined sampling universe. You've got clear, quick,

925
00:54:55,560 --> 00:55:00,200
Speaker 1: clear feedback about your about your predictions. Um, my of

926
00:55:00,239 --> 00:55:02,560
Speaker 1: the world, most of the world isn't like that. It's

927
00:55:02,600 --> 00:55:04,960
Speaker 1: not like it's definitely not like that. And and and

928
00:55:05,000 --> 00:55:08,200
Speaker 1: the question is what are the limits? How useful is

929
00:55:08,200 --> 00:55:12,560
Speaker 1: it to apply probabilistic forms of reasoning um outside their

930
00:55:12,560 --> 00:55:15,400
Speaker 1: traditional domains of application. Then, in a sense, is what

931
00:55:15,440 --> 00:55:18,160
Speaker 1: the U. S. Intelligence community really wanted to explore? I mean,

932
00:55:18,200 --> 00:55:21,200
Speaker 1: can we do better than say, distinct possibility, which, when

933
00:55:21,200 --> 00:55:23,319
Speaker 1: you look at it carefully, is such an elastic term.

934
00:55:23,320 --> 00:55:26,480
Speaker 1: It could mean anything from one percent to nine. So

935
00:55:26,680 --> 00:55:29,600
Speaker 1: let's talk a little bit about the intelligence community and

936
00:55:29,640 --> 00:55:33,879
Speaker 1: the Defense Department. Um, how did you get involved with

937
00:55:34,000 --> 00:55:40,560
Speaker 1: DARPA and the the competition, the forecasting competition? Right? Well, Um,

938
00:55:40,600 --> 00:55:43,920
Speaker 1: it was I r as DARA, the Intelligence Advanced Research

939
00:55:43,920 --> 00:55:47,000
Speaker 1: Projects Agency is post to DARPA, but it's it's it's

940
00:55:47,000 --> 00:55:49,200
Speaker 1: a cousin and and it and it models itself to

941
00:55:49,239 --> 00:55:51,719
Speaker 1: some degree. I think after dark by it it really

942
00:55:51,760 --> 00:55:56,800
Speaker 1: wants to do radical earth change, world changing forms of research.

943
00:55:57,320 --> 00:55:59,680
Speaker 1: And I think changing how we think about uncertainty would

944
00:55:59,680 --> 00:56:02,080
Speaker 1: would be would be pretty fundamental. Maybe not as fundamentals

945
00:56:02,280 --> 00:56:05,520
Speaker 1: inventing the Internet, but way up there. Um, it would

946
00:56:05,520 --> 00:56:08,600
Speaker 1: be a big deal, um to to to change how

947
00:56:08,640 --> 00:56:10,640
Speaker 1: we how we go about doing things. I think our

948
00:56:10,760 --> 00:56:13,320
Speaker 1: our democracy would would be transformed. I think the finance

949
00:56:13,320 --> 00:56:15,640
Speaker 1: industry would be transformed. It would not it would not

950
00:56:15,680 --> 00:56:19,200
Speaker 1: be not a small thing. These would not be small

951
00:56:19,200 --> 00:56:21,520
Speaker 1: things that how long have they been running this contest?

952
00:56:21,600 --> 00:56:23,759
Speaker 1: So they started. They approached my wife and May when

953
00:56:23,800 --> 00:56:26,120
Speaker 1: we were still on the faculty at University California, Berkeley

954
00:56:26,160 --> 00:56:29,200
Speaker 1: about six years ago, and we had a nice um

955
00:56:29,719 --> 00:56:32,360
Speaker 1: um a set of drinks over at the Clermont Hotel

956
00:56:32,400 --> 00:56:36,839
Speaker 1: in Berkeley, and um we um. We were just astonished

957
00:56:37,200 --> 00:56:39,680
Speaker 1: that the U. S Intelligence community was prepared to run

958
00:56:39,680 --> 00:56:42,120
Speaker 1: a series of forecasting tournaments. I mean, I predicted that

959
00:56:42,160 --> 00:56:43,600
Speaker 1: they would never want to do anything like that. So

960
00:56:43,680 --> 00:56:46,080
Speaker 1: it's kind of ironic, right that I I forecast that

961
00:56:46,120 --> 00:56:48,799
Speaker 1: forecasting tournaments would be impossible. And I know it's being

962
00:56:48,840 --> 00:56:50,640
Speaker 1: a bit of a HEDGEHOWK. What I what I said is, look,

963
00:56:50,680 --> 00:56:55,200
Speaker 1: government bureaucracies don't give a slingshot money to David right

964
00:56:55,320 --> 00:56:57,960
Speaker 1: Glass doesn't give sl slingshot money to David. Why would

965
00:56:58,040 --> 00:57:01,440
Speaker 1: be a massive influential government you're oocracy fifty billion dollars

966
00:57:01,520 --> 00:57:04,280
Speaker 1: or so, I wanted to spread millions of dollars around

967
00:57:04,280 --> 00:57:06,839
Speaker 1: to a bunch of small scale academic competition to see

968
00:57:06,840 --> 00:57:08,360
Speaker 1: whether or not they can do a better job of

969
00:57:08,400 --> 00:57:11,800
Speaker 1: assigning realistic probabilities to things of national security significance. And

970
00:57:12,600 --> 00:57:15,160
Speaker 1: this it didn't make any sense given the normal rules

971
00:57:15,160 --> 00:57:18,400
Speaker 1: of bureocratic behavior in Washington, d c UM and so

972
00:57:18,480 --> 00:57:21,880
Speaker 1: I was too Hedgehogy, I was wrong about that. Now

973
00:57:21,880 --> 00:57:23,960
Speaker 1: when they I'm delighted. I was wrong to say, at

974
00:57:24,040 --> 00:57:26,360
Speaker 1: least when when they came to you the prior book,

975
00:57:26,400 --> 00:57:29,480
Speaker 1: the Expert Political Judgment book, you had really run a

976
00:57:29,560 --> 00:57:34,200
Speaker 1: form of this. You would assembled a mast over eighty

977
00:57:34,240 --> 00:57:38,600
Speaker 1: two thousand separate forecasts from several thousand, was it or

978
00:57:38,680 --> 00:57:42,560
Speaker 1: several hundred political forecasters? It was it was a smaller number.

979
00:57:42,560 --> 00:57:45,960
Speaker 1: It was in the hundreds, but um the um. Yes.

980
00:57:46,000 --> 00:57:48,840
Speaker 1: In a sense, Expert Political Judgment was a small scale

981
00:57:49,240 --> 00:57:52,560
Speaker 1: dry run for what I RPA did on Expert Political

982
00:57:52,640 --> 00:57:56,080
Speaker 1: Judgment was run more on a shoestring budget, whereas UH

983
00:57:56,560 --> 00:57:58,800
Speaker 1: the r PA forecasting tournaments were run on on a

984
00:57:58,880 --> 00:58:00,920
Speaker 1: much more in a much more per fessional, large scale

985
00:58:00,920 --> 00:58:03,520
Speaker 1: basis with and you know, and one of the nice

986
00:58:03,520 --> 00:58:05,720
Speaker 1: things about the R project, and people worry about the

987
00:58:05,760 --> 00:58:07,840
Speaker 1: applicability of research and things like this, but this was

988
00:58:07,880 --> 00:58:10,400
Speaker 1: all independently monitored by the U S intelligence community. I mean,

989
00:58:10,400 --> 00:58:13,560
Speaker 1: these these forecasts were submitted at nine am Eastern time

990
00:58:13,640 --> 00:58:15,400
Speaker 1: every day on the day, every DA, every day. The

991
00:58:15,400 --> 00:58:19,640
Speaker 1: forecasting tournaments are running over four years UM, so there

992
00:58:19,720 --> 00:58:23,720
Speaker 1: is a very clear paper trail. So what what is

993
00:58:23,760 --> 00:58:26,640
Speaker 1: the state of the forecasting contests these days? Is it

994
00:58:26,760 --> 00:58:29,960
Speaker 1: something that they've put aside? What what is the takeaway

995
00:58:29,960 --> 00:58:34,760
Speaker 1: from all that? The takeaways are that it is possible

996
00:58:35,120 --> 00:58:38,720
Speaker 1: to make better probability estimates of events that many people

997
00:58:38,720 --> 00:58:42,000
Speaker 1: thought it would be impossible to estimate probabilistically. And it's

998
00:58:42,040 --> 00:58:45,320
Speaker 1: possible to do that by engaging in systematic talent spawning,

999
00:58:45,480 --> 00:58:47,800
Speaker 1: which you can only do if you're tracking score. And

1000
00:58:47,800 --> 00:58:51,280
Speaker 1: it's also possible to do by designing good training modules,

1001
00:58:51,320 --> 00:58:54,680
Speaker 1: by putting together teams that are open to dissent and

1002
00:58:54,920 --> 00:58:57,680
Speaker 1: uh know how to do precision questioning of each other's assumptions,

1003
00:58:58,120 --> 00:59:00,040
Speaker 1: and also by doing a little bit of algory with

1004
00:59:00,160 --> 00:59:05,320
Speaker 1: mcmagic um. So do you think the result of that

1005
00:59:05,440 --> 00:59:12,880
Speaker 1: contest has changed the way the US intelligence community recruits talent,

1006
00:59:13,120 --> 00:59:16,640
Speaker 1: trains talent, and makes forecasts about future events. Well, you'd

1007
00:59:16,640 --> 00:59:18,919
Speaker 1: have to ask the U S intelligence community about how

1008
00:59:18,960 --> 00:59:22,600
Speaker 1: exactly things have changed. My understanding is that the National

1009
00:59:22,600 --> 00:59:26,880
Speaker 1: Intelligence Council now does try to quantify it's probability estimates

1010
00:59:27,000 --> 00:59:30,760
Speaker 1: rather than using just vague verbiage forecasting UM. It has

1011
00:59:30,760 --> 00:59:33,360
Speaker 1: probability ranges. I think it tries to distinguish at least

1012
00:59:33,360 --> 00:59:35,840
Speaker 1: seven degrees of uncertainty, which is a lot more than three.

1013
00:59:36,200 --> 00:59:38,240
Speaker 1: H is more than five, which was which was the

1014
00:59:38,240 --> 00:59:41,320
Speaker 1: preceding number. I think they may be underestimating themselves. I

1015
00:59:41,360 --> 00:59:43,200
Speaker 1: think they could probably get up to ten or fifteen

1016
00:59:43,240 --> 00:59:46,439
Speaker 1: if they if they wanted to. Uh. But I think

1017
00:59:46,520 --> 00:59:48,560
Speaker 1: they're moving in the right direction. I think there's growing

1018
00:59:48,600 --> 00:59:53,040
Speaker 1: interest in crowdsourcing forecast There's growing recognition that UM, the

1019
00:59:53,160 --> 00:59:56,840
Speaker 1: average forecast derived from a group of forecasters, is often

1020
00:59:56,960 --> 00:59:59,560
Speaker 1: more accurate than most of the individuals from whom the

1021
00:59:59,560 --> 01:00:02,120
Speaker 1: average was derived. It sounds kind of magical, but it

1022
01:00:02,760 --> 01:00:06,040
Speaker 1: makes it it is true. UM not always true, but

1023
01:00:06,080 --> 01:00:09,640
Speaker 1: it's a good way to bet we We've I've been

1024
01:00:09,640 --> 01:00:13,520
Speaker 1: critical of some of the prediction markets, not because the

1025
01:00:13,520 --> 01:00:17,280
Speaker 1: theory underlying them is wrong. But very often they're narrow,

1026
01:00:17,480 --> 01:00:21,000
Speaker 1: they're not diverse, they're not incentivized. All the various things

1027
01:00:21,000 --> 01:00:24,480
Speaker 1: you need for a prediction market to work is often missing.

1028
01:00:25,040 --> 01:00:28,640
Speaker 1: Um And sometimes the better as the participants are are

1029
01:00:28,760 --> 01:00:31,920
Speaker 1: so similar to each other, it's hard to extrapolate that

1030
01:00:32,000 --> 01:00:37,080
Speaker 1: out to other other factors. Um. Uh, These these sort

1031
01:00:37,120 --> 01:00:40,920
Speaker 1: of contests and the various prediction markets. Can we describe

1032
01:00:40,960 --> 01:00:44,800
Speaker 1: these as moneyball for the intelligence community? Is it just

1033
01:00:45,480 --> 01:00:49,400
Speaker 1: quantifying data in a way that hasn't been done previously

1034
01:00:49,880 --> 01:00:52,480
Speaker 1: to intelligence forecasts. I think that's a great way to

1035
01:00:52,520 --> 01:00:56,120
Speaker 1: describe it. Just money ballfing the intelligence community. Um. I

1036
01:00:56,120 --> 01:00:58,880
Speaker 1: think it's the movie The The Old World was a

1037
01:00:58,920 --> 01:01:03,360
Speaker 1: world with baseball scouts. Uh, cl Clint Eastwood, Trusty baseball

1038
01:01:03,360 --> 01:01:07,240
Speaker 1: scouts too. You know, we're gradually being displaced by these

1039
01:01:07,320 --> 01:01:10,480
Speaker 1: number crunchers. Um. We're never going to do away with

1040
01:01:10,680 --> 01:01:13,800
Speaker 1: people who have deep qualitative insights into the subject matter.

1041
01:01:13,880 --> 01:01:17,160
Speaker 1: There are crucial source of inputs. But the question is

1042
01:01:17,360 --> 01:01:20,640
Speaker 1: what roles should we be playing as the world changes?

1043
01:01:20,960 --> 01:01:23,880
Speaker 1: And I think human judgment will always be playing a

1044
01:01:23,880 --> 01:01:28,200
Speaker 1: critical role when we're dealing with human beings. Um Um.

1045
01:01:28,240 --> 01:01:33,160
Speaker 1: But there are useful tools for combining human judgment, and

1046
01:01:33,200 --> 01:01:35,720
Speaker 1: you can get more out of it than previously supposed.

1047
01:01:36,240 --> 01:01:38,080
Speaker 1: That makes a lot of sense. But before we get

1048
01:01:38,080 --> 01:01:42,680
Speaker 1: to our our favorite standard questions, anything from super forecasting,

1049
01:01:42,800 --> 01:01:46,240
Speaker 1: I might have missed that you want to uh add

1050
01:01:46,400 --> 01:01:50,200
Speaker 1: as as worth thinking about before we uh we get

1051
01:01:50,240 --> 01:01:52,280
Speaker 1: into a little bit of your history. Well, the thing

1052
01:01:52,360 --> 01:01:55,680
Speaker 1: that I most hope if I mean I'm getting older now,

1053
01:01:55,720 --> 01:01:58,000
Speaker 1: I mean I've been doing this stuff for thirty plus years, UM.

1054
01:01:58,120 --> 01:02:01,200
Speaker 1: And the thing that I most hope lasting legacy of

1055
01:02:01,240 --> 01:02:05,280
Speaker 1: this work, and I hope it improves US foreign policy

1056
01:02:05,280 --> 01:02:09,080
Speaker 1: and intelligence analysis, but I also hope it improves our democracy.

1057
01:02:09,120 --> 01:02:11,120
Speaker 1: And I think in the in the closing chapter, we

1058
01:02:11,160 --> 01:02:14,960
Speaker 1: talk about the debate between Paul Krugman and Nil Ferguson

1059
01:02:15,600 --> 01:02:18,080
Speaker 1: on various issues, and how it more resembles a food

1060
01:02:18,120 --> 01:02:22,240
Speaker 1: fight than it does a serious debate between extremely intelligent people,

1061
01:02:22,280 --> 01:02:25,800
Speaker 1: which which both of them obviously are. UM. And the

1062
01:02:25,920 --> 01:02:29,160
Speaker 1: question is, could we use forecasting tournaments? Could we structure

1063
01:02:29,200 --> 01:02:32,680
Speaker 1: them in ways uh to facilitate more civilized debates on

1064
01:02:32,760 --> 01:02:35,280
Speaker 1: issues that matter. So that's why I wrote a piece

1065
01:02:35,280 --> 01:02:37,200
Speaker 1: in The New York Times several months ago with Peter

1066
01:02:37,240 --> 01:02:41,480
Speaker 1: Skoblick on how we could do that with Iranian nuclear deal? Um,

1067
01:02:41,760 --> 01:02:44,880
Speaker 1: and when a one way to proceed would be to say, Okay,

1068
01:02:44,920 --> 01:02:47,480
Speaker 1: you've got hawks, you've got doves. You have different opinions

1069
01:02:47,480 --> 01:02:49,960
Speaker 1: about what the long term consequences of signing this deal are.

1070
01:02:50,800 --> 01:02:53,320
Speaker 1: We don't know for sure which historical trajectory were on.

1071
01:02:53,640 --> 01:02:56,360
Speaker 1: Why don't the hawks generate five questions that they think

1072
01:02:56,360 --> 01:02:58,800
Speaker 1: they have a comparative advantage in answering? Why don't the

1073
01:02:58,840 --> 01:03:01,080
Speaker 1: doves generate five questions and they think they have a

1074
01:03:01,080 --> 01:03:04,320
Speaker 1: comparative advantage in answering? And you know what, victory will

1075
01:03:04,320 --> 01:03:06,720
Speaker 1: have a clear cut meaning here if the if the

1076
01:03:06,760 --> 01:03:09,200
Speaker 1: doves can answer the dove questions better than the hawks,

1077
01:03:09,280 --> 01:03:11,160
Speaker 1: and they can answer the hawk questions better, then the

1078
01:03:11,200 --> 01:03:14,640
Speaker 1: doves win and vice versa for the hawks. Uh now, um,

1079
01:03:14,800 --> 01:03:16,840
Speaker 1: anyone take you up on that? Well, we do have

1080
01:03:16,880 --> 01:03:19,080
Speaker 1: a number of people who are participating in the tournament,

1081
01:03:19,200 --> 01:03:21,960
Speaker 1: and um, one of the people, and g j open

1082
01:03:22,080 --> 01:03:24,800
Speaker 1: dot com has written a memo on on where where

1083
01:03:24,800 --> 01:03:26,600
Speaker 1: we are right now? The moderate seemed to be doing

1084
01:03:26,720 --> 01:03:28,760
Speaker 1: the best at the moment, but you know that game

1085
01:03:28,840 --> 01:03:30,919
Speaker 1: is far from over. I mean, this is just very

1086
01:03:30,920 --> 01:03:34,120
Speaker 1: early stages of a long term process. Yeah, where you're

1087
01:03:34,160 --> 01:03:37,160
Speaker 1: one of what a tenure treaty, it's quite a way

1088
01:03:37,240 --> 01:03:40,080
Speaker 1: is to go. So so when you when you describe that,

1089
01:03:40,120 --> 01:03:42,680
Speaker 1: and you mentioned debates, I immediately thought of the political

1090
01:03:42,680 --> 01:03:46,040
Speaker 1: debates this year, which at least on the GEOP side,

1091
01:03:46,040 --> 01:03:51,320
Speaker 1: have been not your usual policy debates. UM. And I'd

1092
01:03:51,400 --> 01:03:54,680
Speaker 1: love to see some of the tenants from super forecasting

1093
01:03:55,200 --> 01:03:58,200
Speaker 1: find its way to uh, the political parties and and

1094
01:03:58,240 --> 01:04:01,720
Speaker 1: see if we can have a little more substantive discussion

1095
01:04:01,760 --> 01:04:04,760
Speaker 1: about when this happens, here's what happens in the future,

1096
01:04:04,800 --> 01:04:07,560
Speaker 1: and then hold these folks accountable. That really doesn't seem

1097
01:04:07,600 --> 01:04:11,360
Speaker 1: to happen on whether with political experts or politicians. We

1098
01:04:11,440 --> 01:04:13,240
Speaker 1: really don't hold their feet to the fire much do

1099
01:04:13,480 --> 01:04:15,560
Speaker 1: How far into the future might it be when in

1100
01:04:15,600 --> 01:04:19,720
Speaker 1: a in a presidential election, the presidential candidates take pride

1101
01:04:19,760 --> 01:04:22,800
Speaker 1: in what their briar scores are. UM, I think it's

1102
01:04:22,800 --> 01:04:25,280
Speaker 1: a long way off, judging by what's going on this year,

1103
01:04:25,360 --> 01:04:27,960
Speaker 1: to say the least. So So let's talk about UM,

1104
01:04:28,040 --> 01:04:31,640
Speaker 1: let's talk a little bit about you personally, rather than

1105
01:04:31,840 --> 01:04:34,040
Speaker 1: than some of the books and the ideas that you've

1106
01:04:34,160 --> 01:04:39,240
Speaker 1: you've put forth which which have been absolutely fascinating. Um,

1107
01:04:39,280 --> 01:04:42,040
Speaker 1: So how did you find your way? You went, you

1108
01:04:42,360 --> 01:04:44,680
Speaker 1: became a you went to Yale, you got your pH

1109
01:04:44,800 --> 01:04:48,800
Speaker 1: d in psychology? How did you find your way into forecasting?

1110
01:04:48,840 --> 01:04:54,400
Speaker 1: This really seems far afield from the traditional UM academic

1111
01:04:55,480 --> 01:04:58,760
Speaker 1: realm of that. I was always a pretty strange psychologist.

1112
01:04:59,440 --> 01:05:02,479
Speaker 1: I I had interest that it took me pretty deep

1113
01:05:02,480 --> 01:05:05,800
Speaker 1: into social science, into political science in particular, into into

1114
01:05:05,880 --> 01:05:09,120
Speaker 1: areas of business. But they always interested in organization as

1115
01:05:09,160 --> 01:05:12,360
Speaker 1: I was interested in societies and cultures and large entities

1116
01:05:12,400 --> 01:05:15,280
Speaker 1: that were not you know that obviously psychology matters there,

1117
01:05:15,320 --> 01:05:18,760
Speaker 1: but it's it's a stretch for a psychologist. So in

1118
01:05:18,760 --> 01:05:20,960
Speaker 1: my early work I did do a fair amount of

1119
01:05:21,000 --> 01:05:24,360
Speaker 1: experimental UM work, but I was also also did a

1120
01:05:24,360 --> 01:05:26,760
Speaker 1: lot of archival and naturalistic work. So it was a

1121
01:05:26,800 --> 01:05:30,440
Speaker 1: kind of a natural progression for me. Um, who are

1122
01:05:30,440 --> 01:05:35,680
Speaker 1: your early mentors? Uh? Well? Um uh. Peter Suitfeld was

1123
01:05:35,720 --> 01:05:38,800
Speaker 1: my very first mentor in Canada. I was an undergraduate

1124
01:05:38,880 --> 01:05:42,480
Speaker 1: University of British Columbia, and and he was wonderfully supportive

1125
01:05:42,560 --> 01:05:44,800
Speaker 1: and of me, and and he believed in me, and

1126
01:05:44,840 --> 01:05:48,360
Speaker 1: he he really told me that you know I would

1127
01:05:48,440 --> 01:05:51,080
Speaker 1: probably have a pretty good time if I went to

1128
01:05:51,240 --> 01:05:53,880
Speaker 1: graduate school at Yale and I took it on faith.

1129
01:05:53,920 --> 01:05:56,360
Speaker 1: And I did that, and I met a number of

1130
01:05:56,360 --> 01:05:59,840
Speaker 1: people at Yale who helped me. Um. The guy who

1131
01:06:00,320 --> 01:06:03,600
Speaker 1: coined the term group think, Irving Janice, was one of

1132
01:06:03,600 --> 01:06:06,000
Speaker 1: the people I worked with, and he was quite an

1133
01:06:06,080 --> 01:06:10,240
Speaker 1: unusual psychologist. Also. UM. When I got to Berkeley, of course,

1134
01:06:10,440 --> 01:06:13,400
Speaker 1: UM Daniel Koneman came along in a few years, and

1135
01:06:13,800 --> 01:06:16,280
Speaker 1: he certainly had an influence on me. I already had

1136
01:06:16,280 --> 01:06:19,200
Speaker 1: a PhD, and I was I was just recently tenured faculty.

1137
01:06:19,240 --> 01:06:24,960
Speaker 1: But k Koneman is as a lot of very influential guy.

1138
01:06:25,120 --> 01:06:27,760
Speaker 1: He's he's just a lot smarter than than most of us.

1139
01:06:27,760 --> 01:06:30,280
Speaker 1: So it's a it's a good idea to listen very

1140
01:06:30,280 --> 01:06:34,440
Speaker 1: carefully when he speaks. I really enjoyed, UM, thinking fast

1141
01:06:34,520 --> 01:06:39,800
Speaker 1: and thinking slow, the metaphor for that entire two stage

1142
01:06:41,240 --> 01:06:45,880
Speaker 1: way to look at how humans make decisions, either fast

1143
01:06:45,880 --> 01:06:49,560
Speaker 1: and instinctual or longer and thoughtful, really just seems to

1144
01:06:49,600 --> 01:06:53,280
Speaker 1: make a lot of sense. UM. What other books, uh,

1145
01:06:53,600 --> 01:06:58,320
Speaker 1: have you really enjoyed? Whether books have been especially influential

1146
01:06:58,360 --> 01:07:00,720
Speaker 1: to you. Another person who influenced me is just um

1147
01:07:01,080 --> 01:07:06,000
Speaker 1: uptown here at Columbia University. Robert Jervis his book Perception, Misperception,

1148
01:07:06,040 --> 01:07:08,560
Speaker 1: International Politics. It came out when I was in graduate

1149
01:07:08,600 --> 01:07:11,960
Speaker 1: school and I could feel myself being tugged towards these topics.

1150
01:07:12,000 --> 01:07:15,640
Speaker 1: It was. It's a brilliant analysis of mistakes that have

1151
01:07:15,720 --> 01:07:22,880
Speaker 1: caused unnecessary wars, that perception and misperception in international politics.

1152
01:07:21,800 --> 01:07:27,640
Speaker 1: By anything else stands out is as interesting or unusual

1153
01:07:27,680 --> 01:07:34,200
Speaker 1: to you. Well, um, I mean life evolves in funny, quirky,

1154
01:07:34,240 --> 01:07:36,520
Speaker 1: path dependent ways. I mean, you can you can look

1155
01:07:36,560 --> 01:07:37,920
Speaker 1: back on your life and you can say, well, it

1156
01:07:37,960 --> 01:07:40,240
Speaker 1: was kind of inevitable this happened or that happened. But

1157
01:07:40,280 --> 01:07:42,800
Speaker 1: a lot of the things that that led to my

1158
01:07:42,920 --> 01:07:45,800
Speaker 1: early forecasting tournament work where I think kind of quirky,

1159
01:07:45,880 --> 01:07:48,160
Speaker 1: I mean, it was really kind of quirky. Thats. A

1160
01:07:48,200 --> 01:07:50,200
Speaker 1: scholar as young as I was was appointed to a

1161
01:07:50,280 --> 01:07:53,000
Speaker 1: National Research Council committee when I was just thirty or

1162
01:07:53,040 --> 01:07:56,040
Speaker 1: thirty one year four. Uh, when I when I was

1163
01:07:56,080 --> 01:07:59,000
Speaker 1: that young, um by far the most junior member a

1164
01:07:59,040 --> 01:08:01,360
Speaker 1: committee like that, and had a lot of senior scientists

1165
01:08:01,400 --> 01:08:03,520
Speaker 1: on it. But it gave me opportunities to meet a

1166
01:08:03,560 --> 01:08:06,440
Speaker 1: lot of people, and it connected me to resources that

1167
01:08:06,480 --> 01:08:08,800
Speaker 1: made it possible to do the early forecasting tournament work.

1168
01:08:09,120 --> 01:08:10,880
Speaker 1: It also impressed on me the need to do it

1169
01:08:11,280 --> 01:08:15,400
Speaker 1: because there there we were five liberals and conservatives, all

1170
01:08:15,440 --> 01:08:17,920
Speaker 1: had very strong opinions about the Soviet Union and where

1171
01:08:17,960 --> 01:08:20,720
Speaker 1: things were going. And the liberals thought that Reagan was

1172
01:08:20,760 --> 01:08:23,320
Speaker 1: sending us to where the nuclear apocalypse, and the conservatives

1173
01:08:23,320 --> 01:08:27,519
Speaker 1: thought that then the Soviet Union wasn't ere as an

1174
01:08:27,520 --> 01:08:31,120
Speaker 1: evil empire would never change from within, essentially, as you

1175
01:08:31,200 --> 01:08:32,960
Speaker 1: just had to keep up endless pressure and maybe it

1176
01:08:32,960 --> 01:08:35,240
Speaker 1: would eventually crack. But you know, they didn't help a

1177
01:08:35,240 --> 01:08:37,680
Speaker 1: lot any and they certainly didn't see garbage Of as

1178
01:08:37,760 --> 01:08:41,559
Speaker 1: much of a change agent. Initially, um each side, neither

1179
01:08:41,600 --> 01:08:45,360
Speaker 1: side really predicted gorbachalv and what Garbagechov did inside the

1180
01:08:45,400 --> 01:08:49,479
Speaker 1: Soviet Union in the internal transformations that occurred. Both sides

1181
01:08:49,520 --> 01:08:53,040
Speaker 1: could readily explain after the fact what happened. UM. So

1182
01:08:53,080 --> 01:08:56,240
Speaker 1: it was this mismatch between virtually zero predictab ability and

1183
01:08:56,400 --> 01:09:01,120
Speaker 1: virtually perfect expost explanatory ability that troubled me. And I thought, well,

1184
01:09:01,160 --> 01:09:04,800
Speaker 1: you know, if if debates this important, like World War three,

1185
01:09:05,320 --> 01:09:08,840
Speaker 1: are are being conducted, this shodily. You know, surely there's

1186
01:09:08,840 --> 01:09:11,200
Speaker 1: a better way to do this um And that's what

1187
01:09:11,360 --> 01:09:13,720
Speaker 1: led to the early work on expert political judgment. It

1188
01:09:13,800 --> 01:09:15,960
Speaker 1: was a way to try, what can we do to

1189
01:09:16,280 --> 01:09:19,200
Speaker 1: to keep score and and if we do keep score,

1190
01:09:19,560 --> 01:09:24,040
Speaker 1: can we identify um better ways of making judgments. You

1191
01:09:24,160 --> 01:09:27,200
Speaker 1: describe something that is an enormous pet peeve of mine.

1192
01:09:27,240 --> 01:09:30,519
Speaker 1: In the markets, nobody knows what happens day to day.

1193
01:09:30,600 --> 01:09:35,240
Speaker 1: There is zero predictive analysis, and then on any given day,

1194
01:09:35,280 --> 01:09:38,760
Speaker 1: the market's up five hundred points, it's down five hundred points,

1195
01:09:38,840 --> 01:09:44,120
Speaker 1: and ex post there is always a fantastic narrative explaining exactly,

1196
01:09:44,600 --> 01:09:48,040
Speaker 1: here's why oil shot up and why the market rallied

1197
01:09:48,360 --> 01:09:52,120
Speaker 1: three hundred points, or here's why this terrible thing happened

1198
01:09:52,120 --> 01:09:56,080
Speaker 1: and the market dropped five hundred points. But nobody is saying,

1199
01:09:56,160 --> 01:09:59,479
Speaker 1: if this happens tomorrow, then here's a result. It's always

1200
01:09:59,520 --> 01:10:02,120
Speaker 1: an after the fact narrative that that seems to be

1201
01:10:02,160 --> 01:10:07,480
Speaker 1: consistent across lots of different uh fields, not just politics,

1202
01:10:07,520 --> 01:10:12,960
Speaker 1: but markets and economics. And after the fact we're fantastic storytellers.

1203
01:10:12,960 --> 01:10:15,760
Speaker 1: Before the fact, we have no idea, And there there

1204
01:10:15,760 --> 01:10:18,439
Speaker 1: are situations where we really want to continue doing that,

1205
01:10:18,439 --> 01:10:21,360
Speaker 1: though it's not always bad. I mean, the National Transportation

1206
01:10:21,400 --> 01:10:25,680
Speaker 1: Safety Board, for example, conducts these ex post postmortems on

1207
01:10:25,880 --> 01:10:28,679
Speaker 1: plane plane crashes. One of the reasons why air travel

1208
01:10:28,720 --> 01:10:30,800
Speaker 1: has become as safe as it is is because they're

1209
01:10:30,800 --> 01:10:33,880
Speaker 1: so good at doing these postmortems. Obviously, they can't predict

1210
01:10:33,920 --> 01:10:36,240
Speaker 1: which planes are going to go down, but they become

1211
01:10:36,320 --> 01:10:41,600
Speaker 1: really pretty adept at identifying the critical factors that underlie

1212
01:10:41,880 --> 01:10:45,600
Speaker 1: plane accidents, and as a result, the rules for pilots

1213
01:10:45,640 --> 01:10:49,719
Speaker 1: and the design features of aircraft have changed in ways

1214
01:10:49,920 --> 01:10:53,080
Speaker 1: that make us all safer. Safer so um. But they're

1215
01:10:53,080 --> 01:10:55,599
Speaker 1: not just making up a story for the six o'clock news.

1216
01:10:55,680 --> 01:10:59,040
Speaker 1: They're saying, hey, you know the whole shuttle investigation, the

1217
01:10:59,120 --> 01:11:04,160
Speaker 1: O ring fair, that's right. Therefore, that's the most most

1218
01:11:05,120 --> 01:11:09,800
Speaker 1: youth worlds. Most sectors don't have a black box that say, hey,

1219
01:11:09,840 --> 01:11:13,679
Speaker 1: here's why the engine failed at two oh seven and fifteen.

1220
01:11:14,280 --> 01:11:17,000
Speaker 1: That's right. So there are different types of postmortems, and

1221
01:11:17,080 --> 01:11:20,800
Speaker 1: some of them are constrained by well defined bodies of

1222
01:11:20,840 --> 01:11:25,880
Speaker 1: scientific knowledge and investigative procedure that reduced the serious risk

1223
01:11:25,880 --> 01:11:28,439
Speaker 1: of capitalizing on chance, and others are just sort of

1224
01:11:28,479 --> 01:11:30,640
Speaker 1: make it up as you go, and I think the

1225
01:11:30,640 --> 01:11:32,479
Speaker 1: things we're talking about and make it up as you go.

1226
01:11:32,880 --> 01:11:36,320
Speaker 1: But there are approaches to doing case studies and learning

1227
01:11:36,360 --> 01:11:38,880
Speaker 1: from the past that are very disciplined and focused and

1228
01:11:38,920 --> 01:11:41,559
Speaker 1: can make a safer and wealthier and happier. The The

1229
01:11:41,720 --> 01:11:46,000
Speaker 1: recent book Um the Checklist talks about how much better

1230
01:11:46,120 --> 01:11:51,720
Speaker 1: surgical procedures and outcomes have been since surgeons started using checklists,

1231
01:11:51,760 --> 01:11:55,440
Speaker 1: including wash your hands, which very often was just assumed

1232
01:11:56,120 --> 01:11:59,200
Speaker 1: um that it was done properly with a certain disinfectant

1233
01:11:59,200 --> 01:12:02,400
Speaker 1: in a certain life of time. But we've we've apparently

1234
01:12:02,479 --> 01:12:09,080
Speaker 1: dramatically reduced operating instruments left in abdomens and keeping count

1235
01:12:09,120 --> 01:12:13,240
Speaker 1: of the number of sponges, and that's improved the subsequent outcome,

1236
01:12:13,720 --> 01:12:18,200
Speaker 1: just as the National Transportation Safety Board has improved the

1237
01:12:18,200 --> 01:12:21,400
Speaker 1: safety level of of travel. That's right. So the big

1238
01:12:21,479 --> 01:12:25,760
Speaker 1: question for us is when is learning possible? When can

1239
01:12:25,800 --> 01:12:28,600
Speaker 1: we learn to do certain categories of things better? And

1240
01:12:28,640 --> 01:12:31,240
Speaker 1: when are we just spinning our wheels and deluding ourselves?

1241
01:12:31,479 --> 01:12:34,559
Speaker 1: Are we spinning our wheels and deluding ourselves about financial markets?

1242
01:12:34,560 --> 01:12:37,400
Speaker 1: So many political issues, but we're actually making real progress

1243
01:12:37,400 --> 01:12:41,400
Speaker 1: in the domains of medicine, or airline safety or or whatnot.

1244
01:12:41,400 --> 01:12:43,880
Speaker 1: And it's a it's a mixed picture. Um. And I

1245
01:12:43,920 --> 01:12:46,799
Speaker 1: suppose what we're trying to do with this forecasting tournament

1246
01:12:46,960 --> 01:12:49,640
Speaker 1: work is to bring some of the rigor that has

1247
01:12:49,680 --> 01:12:52,360
Speaker 1: worked in these more scientific domains to bring it to

1248
01:12:52,400 --> 01:12:55,479
Speaker 1: bear in in domains that they are more or less

1249
01:12:55,479 --> 01:12:59,240
Speaker 1: like the wild West. Huh. Um. So let's you've been

1250
01:12:59,280 --> 01:13:03,160
Speaker 1: doing this now for you said, thirty years. What's changed

1251
01:13:03,200 --> 01:13:06,400
Speaker 1: in this industry more than anything? What is the significant

1252
01:13:07,400 --> 01:13:12,000
Speaker 1: progress in in the forecasting and prediction industry? Um, during

1253
01:13:12,040 --> 01:13:18,479
Speaker 1: the course of your career. That's a hard question. Um,

1254
01:13:18,880 --> 01:13:21,840
Speaker 1: They're not all there there there, that's right, well there there.

1255
01:13:21,880 --> 01:13:25,200
Speaker 1: I I think that our knowledge of the imperfections and

1256
01:13:25,240 --> 01:13:28,720
Speaker 1: human judgment, thanks to a lot of the comment inspired

1257
01:13:28,760 --> 01:13:33,160
Speaker 1: research programs, I think we've made discernible progress there. To

1258
01:13:33,439 --> 01:13:36,280
Speaker 1: the least, I think we've I think some of the

1259
01:13:36,280 --> 01:13:38,920
Speaker 1: statistical tools have improved in various ways. I think some

1260
01:13:39,000 --> 01:13:42,240
Speaker 1: of the tools for running teams have even improved. I mean,

1261
01:13:42,280 --> 01:13:44,400
Speaker 1: I think we can do There are versions of the

1262
01:13:44,439 --> 01:13:47,439
Speaker 1: Delphi procedure, for example, which was developed a long time ago,

1263
01:13:47,479 --> 01:13:51,000
Speaker 1: but has got the DELFI better, right, so, which is

1264
01:13:51,040 --> 01:13:52,519
Speaker 1: what I remember. I was saying that a lot of

1265
01:13:52,520 --> 01:13:54,280
Speaker 1: people thought it was kind of crazy to use teams.

1266
01:13:54,520 --> 01:13:57,320
Speaker 1: You're better off having a lot of independent observers. But

1267
01:13:57,400 --> 01:13:59,800
Speaker 1: there's a way to get the benefits of independence and

1268
01:13:59,840 --> 01:14:02,880
Speaker 1: the benefits of creative interaction at the same time. And

1269
01:14:02,960 --> 01:14:05,679
Speaker 1: one way to do that is by going getting everybody

1270
01:14:05,680 --> 01:14:08,639
Speaker 1: to make their judgments anonymously. So you give your probability judgment,

1271
01:14:08,640 --> 01:14:11,200
Speaker 1: your explanation, I give mine, and everybody around the table

1272
01:14:11,240 --> 01:14:13,960
Speaker 1: gives theirs, and we circulate, and we we circulate that

1273
01:14:14,160 --> 01:14:16,400
Speaker 1: and nobody knows who said what. So the high status

1274
01:14:16,479 --> 01:14:20,120
Speaker 1: guy isn't swaying everything the way off. It often happens

1275
01:14:20,120 --> 01:14:24,000
Speaker 1: in groups. UM, and everybody's expressing your judgments anonymously is don't,

1276
01:14:24,360 --> 01:14:27,439
Speaker 1: so they're insulated from the group thing pressure. And you

1277
01:14:27,479 --> 01:14:29,320
Speaker 1: can do that two or three times, and then the

1278
01:14:29,400 --> 01:14:32,679
Speaker 1: question is how much better is the resulting group judgment

1279
01:14:32,760 --> 01:14:35,760
Speaker 1: after you go through this process. Uh, Then it would

1280
01:14:35,760 --> 01:14:38,559
Speaker 1: have been if you'd simply say, taken um an unweighted

1281
01:14:38,600 --> 01:14:41,360
Speaker 1: average of each of the individual group group group judgments,

1282
01:14:41,640 --> 01:14:44,639
Speaker 1: and the answer is this better? How much better? Uh?

1283
01:14:45,080 --> 01:14:47,439
Speaker 1: I think met analysis suggests probably in the vicinity of

1284
01:14:47,479 --> 01:14:53,280
Speaker 1: ten percent better is you know? That's real? Right? And

1285
01:14:53,360 --> 01:14:56,719
Speaker 1: if if you're talking about avoiding a war or finding

1286
01:14:56,720 --> 01:14:59,800
Speaker 1: a terrorist or anything along those lines, that's a real

1287
01:15:00,320 --> 01:15:03,960
Speaker 1: worthwhile pursuit. It's nothing to sniff at, nothing to sniff at.

1288
01:15:04,120 --> 01:15:07,280
Speaker 1: So now let me ask you for your forecast. What

1289
01:15:07,400 --> 01:15:10,800
Speaker 1: are the next major changes? What are the next shifts

1290
01:15:10,840 --> 01:15:12,920
Speaker 1: that are going to come in in the world of

1291
01:15:13,000 --> 01:15:15,920
Speaker 1: forecasts and predictions? What do you see? Perhaps the better

1292
01:15:15,960 --> 01:15:18,960
Speaker 1: way to say this to ask this is what do

1293
01:15:19,040 --> 01:15:22,400
Speaker 1: you see as the influence of your work on on

1294
01:15:22,479 --> 01:15:26,800
Speaker 1: the forecast and community? I see huge potential here, um

1295
01:15:27,960 --> 01:15:30,880
Speaker 1: i R, which funded the first forecasting tournament, is going

1296
01:15:30,920 --> 01:15:34,080
Speaker 1: to be funding to follow up forecasting tournaments, um and

1297
01:15:34,120 --> 01:15:36,280
Speaker 1: I think many of your readers might be interested in these,

1298
01:15:36,280 --> 01:15:39,280
Speaker 1: and there will be calls for volunteers to participate in

1299
01:15:39,280 --> 01:15:42,240
Speaker 1: one form or another. One of them is focusing not

1300
01:15:42,360 --> 01:15:45,559
Speaker 1: so much on the accuracy of your forecast as on

1301
01:15:45,640 --> 01:15:49,479
Speaker 1: the probitive value of the explanations you generate. The probate

1302
01:15:49,520 --> 01:15:52,000
Speaker 1: of value. Are you good at for or after the fact?

1303
01:15:52,120 --> 01:15:54,960
Speaker 1: Are you good at explaining things? Um and? And again,

1304
01:15:55,160 --> 01:15:58,960
Speaker 1: the extent to which we can crowdsource aspects of problem

1305
01:15:59,040 --> 01:16:03,639
Speaker 1: solving and then eventually marrying that. I think the forecasting, UM,

1306
01:16:03,960 --> 01:16:07,880
Speaker 1: I think that's a very ambitious project. It's it's it's

1307
01:16:07,880 --> 01:16:11,120
Speaker 1: in the very early stages. It hasn't been launched yet, um,

1308
01:16:11,160 --> 01:16:15,040
Speaker 1: but I'm optimistic that it will. UM. Um. So what

1309
01:16:15,120 --> 01:16:18,599
Speaker 1: this is called the I r PA forecast things. It's

1310
01:16:18,640 --> 01:16:22,760
Speaker 1: called create create c R E A T E. Uh.

1311
01:16:22,800 --> 01:16:28,400
Speaker 1: It's it's it stands for the flex reasoning. That's right,

1312
01:16:28,600 --> 01:16:31,679
Speaker 1: it's an acronym like that. All right, that sounds interesting.

1313
01:16:31,680 --> 01:16:34,040
Speaker 1: I'll definitely I'll search for that and all link to

1314
01:16:34,080 --> 01:16:36,680
Speaker 1: that and that. The other is another competition which is

1315
01:16:36,680 --> 01:16:40,880
Speaker 1: called h FC, the Hybrid Forecasting Competition, which will be UM,

1316
01:16:41,920 --> 01:16:46,000
Speaker 1: humans and machines and human machine combinations, uh, trying to

1317
01:16:46,040 --> 01:16:51,200
Speaker 1: make predictions, which I am quite optimistic about. Um. So

1318
01:16:51,240 --> 01:16:53,600
Speaker 1: it's Watson working with somebody, well, it would be that

1319
01:16:53,600 --> 01:16:54,880
Speaker 1: would be one way of doing it. There are a

1320
01:16:54,880 --> 01:16:56,640
Speaker 1: lot of there are a lot of possible machines, are

1321
01:16:56,680 --> 01:16:58,640
Speaker 1: a lot of possible models that people could work with.

1322
01:16:58,720 --> 01:17:00,880
Speaker 1: And the question is are you better off just using

1323
01:17:00,880 --> 01:17:02,880
Speaker 1: the model or you better off just using the person,

1324
01:17:03,000 --> 01:17:04,680
Speaker 1: or you're better when are you better off using the

1325
01:17:04,680 --> 01:17:08,439
Speaker 1: combination of the model and the person, and the one

1326
01:17:08,520 --> 01:17:10,720
Speaker 1: truth is we really don't know the answers to these

1327
01:17:10,800 --> 01:17:13,800
Speaker 1: questions right now, and we're hoping to learn more. So

1328
01:17:13,840 --> 01:17:16,120
Speaker 1: I think that's these are these are really important projects.

1329
01:17:16,120 --> 01:17:18,280
Speaker 1: And I think the other big thing that that I'm

1330
01:17:18,400 --> 01:17:21,280
Speaker 1: very focused on because it has relevance to improving the

1331
01:17:21,360 --> 01:17:24,920
Speaker 1: debate in our debates in our society, is competitions to

1332
01:17:25,000 --> 01:17:29,920
Speaker 1: generate better questions. It's not to generate better questions. It's

1333
01:17:29,960 --> 01:17:32,320
Speaker 1: not just about forecasting. It's I mean, you can you

1334
01:17:32,360 --> 01:17:34,479
Speaker 1: can forecast trivial pursuits, and you can become a great

1335
01:17:34,479 --> 01:17:37,240
Speaker 1: forecaster in trivial pursuits. And really, the world is not

1336
01:17:37,280 --> 01:17:40,080
Speaker 1: a better place for it? Or what? What? What? Your

1337
01:17:40,120 --> 01:17:42,360
Speaker 1: world will be a better place when we join super

1338
01:17:42,400 --> 01:17:47,160
Speaker 1: forecasting skills. Two questions on which big policy debates pivot.

1339
01:17:47,520 --> 01:17:50,080
Speaker 1: So you say, if we knew the answer to this question,

1340
01:17:50,160 --> 01:17:52,599
Speaker 1: would we have invaded Iraq? Or if we had known

1341
01:17:52,600 --> 01:17:54,120
Speaker 1: the answer, if we had known the answer that if

1342
01:17:54,120 --> 01:17:55,679
Speaker 1: we if we knew the answer this question, what would

1343
01:17:55,680 --> 01:17:57,479
Speaker 1: would have changed how we what we do in Syria,

1344
01:17:57,600 --> 01:18:00,920
Speaker 1: the Ukraine, or with respect to tax policy, with respect

1345
01:18:00,960 --> 01:18:05,720
Speaker 1: to uh FED policy or whatnot uh so uh generating

1346
01:18:05,720 --> 01:18:12,519
Speaker 1: probitive questions, generating high quality explanations, human machine competitions. I

1347
01:18:12,560 --> 01:18:15,200
Speaker 1: think these are three really important areas for the future.

1348
01:18:15,360 --> 01:18:20,040
Speaker 1: That's fascinating. The last two questions, Um, this is always

1349
01:18:20,080 --> 01:18:21,920
Speaker 1: interesting and I'm trying to figure out the best way

1350
01:18:21,960 --> 01:18:25,599
Speaker 1: to its phraser for you. Normally, I say, what advice

1351
01:18:25,600 --> 01:18:29,000
Speaker 1: would you give to a millennial or someone just graduating

1352
01:18:29,080 --> 01:18:31,880
Speaker 1: college who are going into your field? But I don't

1353
01:18:31,920 --> 01:18:34,519
Speaker 1: know whether that's the field of psychology or the field

1354
01:18:34,560 --> 01:18:40,040
Speaker 1: of analyzing and improving forecasts and predictions. But let me ask,

1355
01:18:40,080 --> 01:18:43,800
Speaker 1: in an open ended fashion, what advice would you give

1356
01:18:43,880 --> 01:18:46,519
Speaker 1: to someone just coming out of school starting their career

1357
01:18:46,880 --> 01:18:50,160
Speaker 1: who wants to follow in your footsteps. Well, it's an

1358
01:18:50,160 --> 01:18:53,080
Speaker 1: interesting point that I really don't have a field anymore.

1359
01:18:53,120 --> 01:18:56,000
Speaker 1: I mean, the University of Pennsylvania when they hired me,

1360
01:18:56,080 --> 01:18:58,000
Speaker 1: they didn't really know where to put me. So I'm

1361
01:18:58,040 --> 01:19:01,800
Speaker 1: partly warden, partly psychology, partly a political science, and Nannenburg

1362
01:19:02,280 --> 01:19:04,960
Speaker 1: um so a lot of different But it's really the

1363
01:19:05,320 --> 01:19:09,760
Speaker 1: you're really studying this, let's let's just, for lack of

1364
01:19:09,800 --> 01:19:13,639
Speaker 1: a better phrase, you're studying the science of decision making

1365
01:19:14,880 --> 01:19:17,840
Speaker 1: that we're studying human judgment and the extent to which

1366
01:19:17,880 --> 01:19:21,200
Speaker 1: human judgment can be improved using a variety of tools,

1367
01:19:21,200 --> 01:19:23,160
Speaker 1: some of them drawn from psychology, some of them drawn

1368
01:19:23,200 --> 01:19:25,519
Speaker 1: from statistics, some of them drawn from organization theory, a

1369
01:19:25,560 --> 01:19:29,080
Speaker 1: lot of different tools. So someone who wanted to go

1370
01:19:29,160 --> 01:19:32,840
Speaker 1: into that field, what advice would you give them? I

1371
01:19:32,880 --> 01:19:38,600
Speaker 1: would say, Um, that there is no clear path to

1372
01:19:38,840 --> 01:19:42,320
Speaker 1: where I am right now. Um that it's it's it's

1373
01:19:42,360 --> 01:19:45,879
Speaker 1: not clear to me, Um, where you would go because

1374
01:19:46,040 --> 01:19:48,920
Speaker 1: the work I'm doing is so weird and interdisciplinary it

1375
01:19:48,920 --> 01:19:52,400
Speaker 1: doesn't fit into any of the existing university niches, Which

1376
01:19:52,439 --> 01:19:56,160
Speaker 1: is kind of funny because most university niches have become

1377
01:19:56,200 --> 01:19:59,520
Speaker 1: more and more specific and more and more narrowly focused,

1378
01:20:00,000 --> 01:20:02,799
Speaker 1: And you're going in the opposite direction, pulling from three

1379
01:20:02,880 --> 01:20:07,280
Speaker 1: distinct plus the whole quantitative side of it, three distinct

1380
01:20:07,360 --> 01:20:10,720
Speaker 1: areas of practice with a heavy math overlay. Yeah. I

1381
01:20:10,920 --> 01:20:13,800
Speaker 1: don't claim to be much more general than many of

1382
01:20:13,840 --> 01:20:16,760
Speaker 1: the specialists my specialist colleagues. I think I'm just more

1383
01:20:16,800 --> 01:20:20,160
Speaker 1: specialized in a weird way. I mean, I work is

1384
01:20:20,240 --> 01:20:24,480
Speaker 1: very specialized and focused. It just draws on different components

1385
01:20:24,479 --> 01:20:28,200
Speaker 1: of different disciplines in a very focused way. Um, so

1386
01:20:28,240 --> 01:20:30,960
Speaker 1: that's intriguing. I think people who do in new discipline work,

1387
01:20:31,000 --> 01:20:33,240
Speaker 1: you know, they're not Leonardo da Vinci. I mean, there

1388
01:20:33,240 --> 01:20:35,280
Speaker 1: aren't any Leonardo DaVinci as far as I can tell

1389
01:20:35,600 --> 01:20:38,559
Speaker 1: right now in the university world. Uh. Um, what we

1390
01:20:38,720 --> 01:20:40,640
Speaker 1: what we do is we we we we and we

1391
01:20:40,720 --> 01:20:43,920
Speaker 1: need to carve out a very specialized research programs that

1392
01:20:43,960 --> 01:20:47,920
Speaker 1: deliver have tangible deliverables. Uh that we were really on

1393
01:20:47,960 --> 01:20:50,840
Speaker 1: a tight accountability leash in this forecasting tournament. It was

1394
01:20:51,400 --> 01:20:53,720
Speaker 1: you know, we had we were submitting forecast nine am

1395
01:20:53,760 --> 01:20:55,960
Speaker 1: Eastern time every every day. It was. It was a

1396
01:20:56,040 --> 01:20:59,759
Speaker 1: very rigorous process and we needed to have very focused group,

1397
01:21:00,479 --> 01:21:04,160
Speaker 1: tangible deliverables and a focused process that that seems like

1398
01:21:04,240 --> 01:21:09,280
Speaker 1: that's of great value to both business and government. Final question,

1399
01:21:09,800 --> 01:21:13,200
Speaker 1: what is it that you know about forecasting today that

1400
01:21:13,320 --> 01:21:15,479
Speaker 1: you wish you knew when you started down this road

1401
01:21:15,600 --> 01:21:19,160
Speaker 1: thirty years ago? Well? What I wish I knew? What

1402
01:21:19,320 --> 01:21:22,559
Speaker 1: I what? So the early work was mostly about cursing

1403
01:21:22,560 --> 01:21:26,160
Speaker 1: the darkness. It was about cognitive bias and how we're

1404
01:21:26,160 --> 01:21:28,479
Speaker 1: prisoners of our preconceptions, how we have a heart that

1405
01:21:28,520 --> 01:21:30,240
Speaker 1: we're too quick to make up our minds were too

1406
01:21:30,240 --> 01:21:32,320
Speaker 1: slow to change them. It was a rather dark purtrait

1407
01:21:32,360 --> 01:21:36,400
Speaker 1: of human nature. UM. And there's some reason for being pessimistic,

1408
01:21:36,640 --> 01:21:39,639
Speaker 1: given the way we think about UM politics and history

1409
01:21:39,680 --> 01:21:43,040
Speaker 1: and economics for much of the time UM. The later

1410
01:21:43,080 --> 01:21:45,040
Speaker 1: work has been more about lighting candles. It has a

1411
01:21:45,080 --> 01:21:48,040
Speaker 1: more upbeat flavor that there are specific things you can

1412
01:21:48,080 --> 01:21:51,000
Speaker 1: do to become more open minded, at least about relatively

1413
01:21:51,000 --> 01:21:53,120
Speaker 1: near term futures. And if you can become more open

1414
01:21:53,120 --> 01:21:55,639
Speaker 1: minded about relatively near term futures, maybe you can become

1415
01:21:55,640 --> 01:21:58,320
Speaker 1: a little more open minded about medium and longer term futures.

1416
01:21:58,400 --> 01:22:01,040
Speaker 1: Maybe you can be better able to see how alternative

1417
01:22:01,040 --> 01:22:04,760
Speaker 1: perspectives might have some merit UM. And I think that

1418
01:22:04,840 --> 01:22:07,920
Speaker 1: when you feel you're in competition with the other side,

1419
01:22:08,000 --> 01:22:09,559
Speaker 1: and the other side might be getting to the truth

1420
01:22:09,600 --> 01:22:12,600
Speaker 1: faster than you, that has a very salutary effect. I

1421
01:22:13,000 --> 01:22:14,840
Speaker 1: think it will tend to make us a bit more

1422
01:22:14,880 --> 01:22:18,000
Speaker 1: open minded. Thank you so much for being so generous

1423
01:22:18,040 --> 01:22:21,639
Speaker 1: with your time, Professor Tetlock. Uh. We've been speaking with

1424
01:22:21,640 --> 01:22:26,120
Speaker 1: Professor Philip Tetlock of the University of Pennsylvania UM both

1425
01:22:26,120 --> 01:22:30,720
Speaker 1: Wharton and other schools, author of super Forecasting, The art

1426
01:22:30,920 --> 01:22:36,519
Speaker 1: and science of prediction, as well as expert political judgment,

1427
01:22:36,800 --> 01:22:39,280
Speaker 1: How Good is It? How Can We Know? And a

1428
01:22:39,400 --> 01:22:42,720
Speaker 1: number of other books. If you've enjoyed this conversation, be

1429
01:22:42,760 --> 01:22:44,479
Speaker 1: sure and look up an Inch or Down an Inch

1430
01:22:44,800 --> 01:22:47,160
Speaker 1: on Apple iTunes and you'll see all of our other

1431
01:22:47,760 --> 01:22:53,200
Speaker 1: eighty three or so um previous conversations. I want to

1432
01:22:53,280 --> 01:22:55,920
Speaker 1: thank Mike bat Nick for doing the deep dive and

1433
01:22:55,960 --> 01:23:00,000
Speaker 1: helping me on the research with this. I'm Barry Ridhltz.

1434
01:23:00,040 --> 01:23:08,920
Speaker 1: You've been listening to Masters and Business on Bloomberg radioh