Speaker 1: This is Masters in Business with Barry Ridholts on Bloomberg Radio.
Speaker 1: This week on the podcast, I have a really fascinating guest.
Speaker 1: His name is Philip Tetlock. Here's a professor of loosely
Speaker 1: let's call its psychology and political science at uh both
Speaker 1: Wharton and the Arts and Sciences School at the University
Speaker 1: of Pennsylvania. Professor Tatlock is really a fascinating guy. I
Speaker 1: first got to know of his work through a book
Speaker 1: he wrote well over a decade UH ago called Expert
Speaker 1: Political Judgment, and it really cast an enormous shadow of
Speaker 1: doubt on all of these self proclaimed experts and and
Speaker 1: he used the field of political science as his basic
Speaker 1: investigation arena, but really it applied to everything from investing
Speaker 1: to economics to anywhere people prognosticate about the future and
Speaker 1: make um confident sounding assertions about what's going to happen
Speaker 1: one to five ten years in the future. He pretty
Speaker 1: much destroyed that entire line of thinking and showed that
Speaker 1: the average so called expert is no better than the
Speaker 1: average person walking down the street. That was well over
Speaker 1: a decade ago, and he tells an absolutely fascinating story
Speaker 1: of how we're familiar with DARPA, nette the Defense Research
Speaker 1: Projects invented the Internet and all these other really cool things. Well,
Speaker 1: the intelligence community has their own version of DARPA and
Speaker 1: it's called IARPA. And they basically approached Professor Tetlock and said,
Speaker 1: we're fascinated by your work and we'd like to find
Speaker 1: out if there are a way to to either identify
Speaker 1: or develop those outlawyers in your studies who actually turned
Speaker 1: out to be pretty good forecasters. And as it turns out,
Speaker 1: forecasting is a humble skill that can be I don't
Speaker 1: want to say learned by just about anybody, but you
Speaker 1: can undertake a number of steps to make your own
Speaker 1: forecasting better than it might have been otherwise. And it's
Speaker 1: fairly rational and fairly straightforward and really really fascinating. And
Speaker 1: so you'll hear the story of of not only how
Speaker 1: the original book came about, but how the new book
Speaker 1: came about and how he modified the previous view. It
Speaker 1: turns out, looking twelve months and further out, it's still
Speaker 1: pretty much a crapshoot. No one knows what's going to happen.
Speaker 1: But on shorter periods of time and on very very
Speaker 1: specific things, you can undertake a number of steps that
Speaker 1: will help your own ability to discern probable outcomes be
Speaker 1: much better. I think if you're at all a statistics
Speaker 1: or probability want you're gonna find this absolutely fascinating. Anyone
Speaker 1: who's an investor, a trader, who who deals with market
Speaker 1: or economic forecast might also find this to be really
Speaker 1: really interesting. So, without any further ado, my conversation with
Speaker 1: Professor Philip Tetlock. This is Masters in Business with Barry
Speaker 1: Ridholts on Bloomberg Radio. This week on Masters in Business
Speaker 1: on Bloomberg Radio, I have a special guest, Philip Tetlock,
Speaker 1: professor at the University of Pennsylvania, where he is cross
Speaker 1: appointed at both the Wharton School and the School of
Speaker 1: Arts and Sciences. Perhaps best known as the author of
Speaker 1: Expert Political Judgment, How Good Is It and How Can
Speaker 1: We Know? Which has won a number of awards. We'll
Speaker 1: talk about that in a little while. His most recent book,
Speaker 1: also winning accolades, is called super Forecasting, The Art and
Speaker 1: Science of Prediction. The Economist magazine named it to one
Speaker 1: of its best book lists of Professor Tatlock is also
Speaker 1: a co principal investigator of the Good Judgment Project, which
Speaker 1: is a multi year study looking at improving the accuracy
Speaker 1: of probability judgments of real world high stake events. Professor
Speaker 1: tat Luck, Welcome to Bloomberg. Oh, thank you, great to
Speaker 1: be here. So I'm familiar with you from your your
Speaker 1: earlier work, some of your earlier books, which I found
Speaker 1: absolutely fascinating. But let's just start this segment talking very
Speaker 1: very generally. How come there's such an enormous appetite for
Speaker 1: political and economic punditry, including all the predictions and forecasts
Speaker 1: that go with that. Well, it would be deeply dissonant
Speaker 1: for people to um to come to the conclusion that
Speaker 1: it doesn't matter who's elected president. It doesn't matter whether
Speaker 1: we raise taxes or cut them, whether we help the
Speaker 1: Ukraine or don't help the Ukraine, or intervene in Syria,
Speaker 1: or don't intervene in Syria, or assign free trade packs
Speaker 1: or don't do that. Um, we could just toss a coin,
Speaker 1: because nobody the is on either side of the political
Speaker 1: spectrum has any demonstrated ability to assign good probabilities to
Speaker 1: the consequences of those policy options. Uh So it's to
Speaker 1: say that sounds nihilistic, it's dissonant. Um people just don't
Speaker 1: want to live that way. People don't want to live
Speaker 1: that They want structure. And for the same reason that
Speaker 1: people have turned to psychics and which doctors and all
Speaker 1: sorts of things over the past. We we we we
Speaker 1: we we see guidance, predictive guidance, and we seek it.
Speaker 1: There's a demand for it even when the supply of
Speaker 1: good forecasting is extremely limited. So we're recording this in
Speaker 1: the month of March not too long ago, the cover
Speaker 1: of Barons magazine has a picture of Hillary Clinton and
Speaker 1: Donald Trump, which one is better for investors, and the
Speaker 1: usually conservative Barons grudgingly says, we think Hillary will be
Speaker 1: better for the stock market then than that Donald will.
Speaker 1: Don't we run the risk of putting way too much
Speaker 1: credit for good markets and good economies on the president
Speaker 1: and vice versa. When things are bad. Don't we blame
Speaker 1: them too much for what's gone wrong. I think there's
Speaker 1: a great deal of superstitious reasoning about the impact that
Speaker 1: that that leaders have on organizational performance and also the
Speaker 1: impact that presidents have on on the economy. The number
Speaker 1: of leaders that presidents have to influence economic outcomes is
Speaker 1: quite limited um, but there is this deep intuition we
Speaker 1: have that leaders should be accountable for what happens on
Speaker 1: their watch, like the captain of the ship. It really
Speaker 1: doesn't matter whether it's a storm suddenly emerges out of nowhere. Uh.
Speaker 1: You you, you want to hold somebody accountable and and
Speaker 1: that that is also deeply wired into us as it's
Speaker 1: how someone has to be accountable for it. So, so
Speaker 1: let's ask the obvious question, why are experts so often
Speaker 1: so wrong in their forecasts. Well, there are a couple
Speaker 1: of theories about that. One is that the problem lies
Speaker 1: in the experts and how the experts think, uh, and
Speaker 1: they could do better if they had used better analytical
Speaker 1: tools and we're more self critical and creative and thoughtful.
Speaker 1: And the other theory is that we just live in
Speaker 1: a radically unpredictable world in which originally nobody can do
Speaker 1: appreciably better than chance over extended periods of time. It's
Speaker 1: a world royaled by black Swanish dark gray Swanish events
Speaker 1: uh and uh, com buyer beware. Fair enough, So let's
Speaker 1: talk about good judgment in predicting future events, or at
Speaker 1: least good analytical processes. And along that that line, I
Speaker 1: would be remiss if I didn't ask what is the
Speaker 1: Briar score and why is it so important the Briar score. Well,
Speaker 1: the Briar score was originally developed by some statisticians UH
Speaker 1: working with meteorologists who wanted to keep score. And this
Speaker 1: goes all the way back to and it's a very
Speaker 1: simple idea. UH. You want to minimize the gaps between
Speaker 1: probability and reality. So your code reality is either zero
Speaker 1: or one, depending on whether the event occurred or didn't occur,
Speaker 1: and you have probability judgments that range from zero to one,
Speaker 1: and you you get a really good Brier score if
Speaker 1: you assign probabilities very close to zero to things that
Speaker 1: don't happen, and probability is very close to one point
Speaker 1: zero two things that do happen. So it's your ability
Speaker 1: to be justifiably decisive to make appropriately extreme judgments as
Speaker 1: the circumstances dictate, but to avoid driving off a cliff
Speaker 1: in the pursuit of that objective by making extremely overconfident
Speaker 1: judgments and saying ntent of things that don't happen and
Speaker 1: of things that do well. Once you're thinking in terms
Speaker 1: of probability, aren't you already light years ahead of the
Speaker 1: folks who are that decisive in making these bolds outlawyer declarations.
Speaker 1: But isn't that much more nuanced than thoughtful? Then here's
Speaker 1: what's gonna happen A And of course there's a big
Speaker 1: possibility that it just doesn't go that way. That's right.
Speaker 1: The probability scale and principle, if you want to look
Speaker 1: at it where a mathematician would is infinitely divisible, an
Speaker 1: infinite number of points between zero and one point zero.
Speaker 1: And obviously people can't distinguish that many levels of uncertainty. Now,
Speaker 1: when IBM S. Watson was playing in the Jeopardy competition
Speaker 1: and beat the best human players, uh, you might have
Speaker 1: noticed that under its answers, occasionally there would be this
Speaker 1: little uh Baysian probability estimate of how confident Watson was
Speaker 1: in his answer at point eight seven three six two
Speaker 1: or something. Uh So, these these types of form forms
Speaker 1: of artificial intelligence do try to uh make extremely granular
Speaker 1: distinctions among degrees of uncertainty. Human beings can't make that
Speaker 1: many uh distinctions among degrees of uncertainty in most environments.
Speaker 1: I mean, there's some environments where we can pull out
Speaker 1: a calculator and do it. If you can do it,
Speaker 1: with card tricks and so forth um or poker um.
Speaker 1: But there are there are real limits on how granular
Speaker 1: you can become when it comes to whether there's going
Speaker 1: to be a country leaving the Eurozone in the next year,
Speaker 1: or whether there's going to be a violent Sino Japanese
Speaker 1: class in the East China Sea, or things of that sort.
Speaker 1: How many degrees can you distinguish their Is it just
Speaker 1: yes or no? Or that yes maybe no? Or can
Speaker 1: you make finer distinctions than that? I'm Barry Ridhults. You're
Speaker 1: listening to Masters in Business on Bloomberg Radio. My special
Speaker 1: guest today is Professor Philip Tetlock of the University of Pennsylvania.
Speaker 1: He is the author of a number of books, but
Speaker 1: the one I want to talk about right now was
Speaker 1: the award winning Expert Political Judgment, How good is it
Speaker 1: and how can we know? Uh? This won a number
Speaker 1: of awards, the Graumeyer Award for Ideas Improving Political Order,
Speaker 1: the Woodrow Wilson Foundation Award for Political Science, and the
Speaker 1: Robert E. Lan Award for a Political Psychology. Let's let's
Speaker 1: start with a quote of yours that I want to
Speaker 1: get some some feedback on people who make predictions in
Speaker 1: their business who appear as experts on TV get quoted
Speaker 1: in newspaper articles advised, governments and businesses are no better
Speaker 1: than the rest of us at making forecasts. How is
Speaker 1: that possible? It turns out that you reached the point
Speaker 1: of diminishing marginal returns for knowledge quite quickly and a
Speaker 1: lot of the domains we care the most about. Now
Speaker 1: what does that mean? UM? When I started off doing
Speaker 1: research on political judgment, UM, one of the greatest psychologists
Speaker 1: and on the planet advised me, Daniel Kahneman and the
Speaker 1: Nobel Prize award winning psychologists most recent book, Thinking Fast Right.
Speaker 1: And it was a lunchtime conversation about thirty years ago
Speaker 1: in which he said rather casually that he thought that
Speaker 1: the experts I was interviewing from my early work on
Speaker 1: expert political judgment would have a hard time doing better
Speaker 1: than UM an attentive reader of the New York Times. UM,
Speaker 1: which is, you know, a kind of a fancy way
Speaker 1: of saying more or less what you just said. Uh. Now,
Speaker 1: he he didn't know that as a fact. He was
Speaker 1: offering that as an hypothesis. Uh. And I think the
Speaker 1: right way to look at this is, UM. It is
Speaker 1: an hypothesis we can be continually testing. It's it's not
Speaker 1: always going to be the case that experts are going
Speaker 1: to fall short, but they're going to fall short much
Speaker 1: more often than we would expect. Um well, how much
Speaker 1: of that is random? And when they're right and at
Speaker 1: a certain point don't don't the either investing or let's
Speaker 1: call it, voting public have a reasonable belief that the
Speaker 1: supposed experts know what they're talking about. When they make
Speaker 1: a forecast. They expect them to be considerably better than
Speaker 1: I think as as someone once called it a dart
Speaker 1: throwing monkey. Right, Well, there's a there's the big question
Speaker 1: that we want the answer to, and then they're all
Speaker 1: these proxy cues that we kind of latch onto in
Speaker 1: the hope that those proxy cues will get us closer
Speaker 1: to the answer. So the big question we want the
Speaker 1: answer to say is whether the U. S. Economy is
Speaker 1: going into her session next year, whether that the Dow
Speaker 1: is going to be over twenty thousand or under fifteen thousand.
Speaker 1: There there's some big questions that people in the financial
Speaker 1: community or political community want answers to, And there are
Speaker 1: various people who passed through our lives passed through your
Speaker 1: radio station, passed there in the op ed pages of newspapers,
Speaker 1: on television and so forth, who offer opinions on these things,
Speaker 1: and they come with various types of credentials. You might say,
Speaker 1: so and so is the muckety muck professor a bl
Speaker 1: bludy blum, or you might say that so and so
Speaker 1: when a Nobel prize. You might say that so and
Speaker 1: so is worth ten billion dollars, or you might do
Speaker 1: a lot of things you could say about that and um.
Speaker 1: The The interesting question is, uh, do those things give
Speaker 1: us much guidance on how accurate what the person is saying?
Speaker 1: So we're hoping that they do. We're we're hoping to say, well,
Speaker 1: this person must know what he or she is talking
Speaker 1: about by virtue of the fact that this person has
Speaker 1: done X, Y or z. But the relationship between having
Speaker 1: done X, Y or Z and accuracy is unknown. And
Speaker 1: the more honest we are about our ignorance, the more
Speaker 1: honest we are about when we're using proxy cues for
Speaker 1: judging how credible source of advice is, the better off
Speaker 1: we're going to be in the long term. And and
Speaker 1: by the way that applies to me too, that's fascinating.
Speaker 1: So so let me ask, um, when you put out
Speaker 1: expert political judgment, had anyone really done a full on
Speaker 1: quantitative analysis of how accurate experts were, at least in
Speaker 1: the political field. Had had anyone tried to figure out, Hey,
Speaker 1: let's figure out exactly how right or wrong these folks are? Before? Interestingly,
Speaker 1: very little work had been done. There was a little
Speaker 1: bit of work assessing the accurate Well, there's quite a
Speaker 1: bit of work assessing the work of weather forecasters. There
Speaker 1: was some work assessing the accuracy of expert bridge players, uh,
Speaker 1: And there was some work assessing the accuracy of economist.
Speaker 1: The Federal Reserve in Philadelphia and elsewhere had been doing
Speaker 1: some some studies along those lines. But as for assessing
Speaker 1: the accuracy of political pundits, at the time my book
Speaker 1: came out, I think there was extremely little work on
Speaker 1: that subject. So here's a quote from the book, and
Speaker 1: I want to get some um feedback from you on this.
Speaker 1: When they're wrong, they're rarely held accountable, and they rarely admitted.
Speaker 1: They insist they were just off on timing, or blindsided
Speaker 1: by an improbable events, or almost right or wrong for
Speaker 1: the right reasons. Well, if you're a pundit, You're playing
Speaker 1: a complicated game. Uh. If I'm a pundit on your
Speaker 1: show or anyone show, and I need I need to
Speaker 1: make it sound as though I know what I'm talking about.
Speaker 1: I need to make it sound as though I'm telling
Speaker 1: the listeners something they didn't know before. I also need
Speaker 1: to preserve my long term credibility, which means I have
Speaker 1: to have some escape clauses. So if the claims I
Speaker 1: make about the future turn out to be wrong, I
Speaker 1: need I need some way about of walking away from it.
Speaker 1: So when you have this problem of um, you have
Speaker 1: a career as a pundit, you need to be saying
Speaker 1: something surprising, but you also need to preserve your long
Speaker 1: term credibility. That's a real dilemma the pundit is in.
Speaker 1: So the typical way pundits cope with this is by
Speaker 1: saying something dramatic, uh, like Canada will disintegrate or the
Speaker 1: Eurozone will disintegrate, or Pudin will reinvade the Ukraine. But
Speaker 1: build in some waffle words like this could happen, or
Speaker 1: this might happen, or there's a distinct possibility this will happen.
Speaker 1: Now distinct possibility is one of those wonderful phrases, because
Speaker 1: if it happens, I can say, hey, I told you
Speaker 1: there was a distinct possibility. If it doesn't happen, I
Speaker 1: can say, hey, I just that was possible, So I'm
Speaker 1: covered either way. Uh, And that UM, it helps to
Speaker 1: explain why pundits um and and indeed why traditionally people
Speaker 1: in the US intelligence community as well UM have relied
Speaker 1: so heavily on vague verbiage forecasting because they need to
Speaker 1: be saying something that sounds informative, but they need uh
Speaker 1: strategy for preserving their long term credibility at the same time.
Speaker 1: So that explains why they don't admit error. Although I
Speaker 1: would argue nobody bats a thousand, admitting error shows that
Speaker 1: you have a little humility and recognize that it's not easy.
Speaker 1: In the last minute we have in this segment, the
Speaker 1: real question I have is why don't we hold these
Speaker 1: folks accountable. Well, here's the thing, Barry, they don't even
Speaker 1: think that they're wrong. If I if I say there
Speaker 1: was a distinct possibility that Putin is going to invade
Speaker 1: Estonia this this coming year, and he doesn't do it,
Speaker 1: I'm gonna interpret distinct possibilities having met a very low probability,
Speaker 1: and if he does do it, I'm going to interpret
Speaker 1: distinct possibilities having med a very high probability. I'm gonna
Speaker 1: we we tend to be somewhat self serving in our
Speaker 1: own mental calculus. I say, well, you know, we we interpret,
Speaker 1: we give ourselves a benefit of the doubt. On how
Speaker 1: we interpret distinct possibility adjusted, we give the benefit of
Speaker 1: the doubt. You know political pundits who favor our political party.
Speaker 1: I'm Barry rid Helts. You're listening to Masters in Business
Speaker 1: on Bloomberg Radio. My special guest today is Professor of
Speaker 1: Philip Tetlock. He is the author of Expert Political Judgment,
Speaker 1: How Good Is It? As well as Super Forecasters, a
Speaker 1: new book that just came out to great acclaim. He
Speaker 1: teaches at the Universe City of Pennsylvania, Wharton And and
Speaker 1: let's jump right in to the Hedgehogs versus Fox's discussion.
Speaker 1: You referenced this throughout really throughout um the second book
Speaker 1: a lot, and if I recall correctly, you mentioned it
Speaker 1: a few times in the first book, which I've read
Speaker 1: a while ago. Uh. For those people who may not
Speaker 1: have read Isaiah Berlin's essay explain to us what is
Speaker 1: the hedgehog and the fox? So Isaiah Berlin was British
Speaker 1: um scholar, political philosopher, political historian, philosopher who um took
Speaker 1: a quote from the Greek warrior poet or Kilicus from
Speaker 1: years ago and he built a really interesting essay around
Speaker 1: it and the quote. One of the few surviving fragments
Speaker 1: of this man's work was that the fox knows many things,
Speaker 1: but the hedgehog knows one big thing. And he intended
Speaker 1: that to capture different styles of things. King he thought
Speaker 1: that um, some some thinkers were much closer to being
Speaker 1: um uh foxes. Shakespeare, I think was one of his
Speaker 1: classic examples of a fox who could just a very
Speaker 1: multifaceted view of human nature, and other writers um he
Speaker 1: thought could be could be pigeonholed better as hedgehogs. Now
Speaker 1: we use this fox hedgehog distinction um in the work
Speaker 1: on political Judgment because it rather captures rather well uh
Speaker 1: different styles of thinking. Um. You could be, for example,
Speaker 1: a hedgehog of very many different political sorts. You could
Speaker 1: be a free market hedgehog, or you could be a
Speaker 1: Marxist hedgehog. You could be an environmentalist, uh, doomster hedgehog,
Speaker 1: or you could be a boomster um utopian kind of
Speaker 1: you techno utopian sort of hedgehog. You're going to find
Speaker 1: a cost effective substitutes for whatever we're running out of.
Speaker 1: So there there are many different forms of hedgehog left right, pessimistic, optimistic,
Speaker 1: and we identified many of them in the early work,
Speaker 1: and we tracked their their their accuracy, and we compared
Speaker 1: their accuracy to that of more fox like forecasters, and
Speaker 1: we found a couple of things. One is that the
Speaker 1: hedgehogs tend to have pretty bad batting average when you
Speaker 1: look at all their predictions, uh, their their overall batting
Speaker 1: average is pretty bad. We also found that the hedgehogs
Speaker 1: tended to be more prominent, They're more attractive to the media.
Speaker 1: The media like the kinds of sound bites that hedgehogs
Speaker 1: can deliver. And we found that the hedge hugs um
Speaker 1: also tended to have at least a few home runs. Matt,
Speaker 1: if you're making a lot of pretty extreme predictions on
Speaker 1: a wide range of subjects, at least a few of
Speaker 1: them are almost by chance going to be accurate, whereas
Speaker 1: the foxes are more making more moderate probability judgments, and
Speaker 1: they have less claim on home runs. So um, you
Speaker 1: get on somewhat ironic situation that the worst forecasters have
Speaker 1: the greatest media prominence. Isn't that inherent to the process
Speaker 1: of not only having a real specific expertise in one
Speaker 1: area as opposed to being a generalist, but also making
Speaker 1: those outlier forecasts. I use a slide in my presentation
Speaker 1: about a particular pundit who every year for the past
Speaker 1: seven years has forecast a seven like crash every year,
Speaker 1: and you would think the media would eventually say, hey,
Speaker 1: this guy is just consistently wrong. But it's such an
Speaker 1: outrageous forecast and it gets people so excited they love
Speaker 1: to bring them back on. Isn't that the nature of
Speaker 1: sensationalism that the hedgehogs are going to be more, especially
Speaker 1: today where everything is on clicks and views. Who's going
Speaker 1: to generate more clicks a rational sober well, we don't
Speaker 1: really know what's gonna happen, versus the sky is falling
Speaker 1: and everybody loves that. Well, I would just suggest that
Speaker 1: that people be better off if they were more honest
Speaker 1: about the functions that are served by consuming different types
Speaker 1: of information. So you if you said to yourself, look,
Speaker 1: I want to be entertained. I want I want to
Speaker 1: see somebody who's saying outrageous things and I'm gonna be
Speaker 1: But but I'm not gonna base my probability judgments on them.
Speaker 1: I just going to want to be entertained by these
Speaker 1: amazing stories as person is going to tell about how
Speaker 1: the Saudi regime is going to disintegrate and how we're
Speaker 1: on the verge of World War three. Uh, this is
Speaker 1: this is really entertaining stuff as opposed to listening to
Speaker 1: this much more at tentative, nuanced fox like forecaster who's
Speaker 1: saying on the wine hand, on the other hand, drones
Speaker 1: on and on. Why there's really only about probability of
Speaker 1: the Saudi regime changing in the next twenty four months.
Speaker 1: Can we really view the world through the lens of
Speaker 1: a of a single defining idea or is that, as
Speaker 1: the disclaimer says, for entertainment purposes only. Well, we need
Speaker 1: to be very clear about the functions that are being served. Uh.
Speaker 1: Some of these big ideas are very useful lenses for
Speaker 1: viewing the world um at particular moments in history and
Speaker 1: in conjunction with other ideas. So I'm not saying that
Speaker 1: the intellectual apparatus is useless. I'm saying that it's what's
Speaker 1: really dangerous is when you have a smart person who
Speaker 1: runs too far with a big idea and fails to
Speaker 1: see that the complexity of the world puts a lot
Speaker 1: of breaks on it. So one of our rules of
Speaker 1: thumb for distinguishing better forecasters and worse forecasters on the
Speaker 1: media is the ratio of the number of times they
Speaker 1: say however versus moreover. So if you have a high
Speaker 1: however over moreover ratio, that means you're a fox. That
Speaker 1: means you're boring. That means are probably going to be
Speaker 1: kicked off the show. And if you have more more accurate,
Speaker 1: more likely to be more accurate. But that's right, you're
Speaker 1: gonna have better briar score. I'm Barry Ridholts. You're listening
Speaker 1: to Masters in Business on Bloomberg Radio. My special guest
Speaker 1: this week, professor of phil Tetlock of the University of Pennsylvania.
Speaker 1: His most recent book, super Forecasting, The Art and Science
Speaker 1: of Prediction. So let's jump right into this because I
Speaker 1: have so many questions about this, and let me start
Speaker 1: out by just asking how many piano tuners are there Chicago,
Speaker 1: UH somewhere between about eight and I think so. So
Speaker 1: it's a fascinating question because your initial reaction is to
Speaker 1: shrug and say, I don't know. The variation I've heard
Speaker 1: of that is how many cardiac surgeons are there in London?
Speaker 1: UM or what's the empire state building way? That's another
Speaker 1: good question. So so let's talk. Let's talk a little
Speaker 1: bit about what do most people do when presented with
Speaker 1: a question like that? Interesting? You know, some of the
Speaker 1: high tech firms in Silicon Valley were quite fon fond
Speaker 1: of asking off off the out of left field questions
Speaker 1: of this sort because they thought it was a great
Speaker 1: way of testing how well people think on their feet. UM.
Speaker 1: In the book, we call these Ferremi questions, named after
Speaker 1: the great Italian American physicist and Rico Ferremi, who developed
Speaker 1: the first um UH nuclear reactors keep part of the
Speaker 1: Manhattan Project developing the bomb, and Ferremy was fond of
Speaker 1: UH posing these oddball questions to his students. He what
Speaker 1: do he wanted to the students to do was to
Speaker 1: take an seemingly intractable problem and break it down into
Speaker 1: parts or components that were more tractable, So you might
Speaker 1: not have any idea how many. No one has any
Speaker 1: idea initially on how many how many piano tuners that
Speaker 1: might be in Chicago. But you make guestiments about the
Speaker 1: population of Chicago. Walk us through that because you you
Speaker 1: go through about seven steps and you get pretty close
Speaker 1: to the correct answer. Right. Well, you're you're making a
Speaker 1: lot of guests and it's not just the breaking down
Speaker 1: and get trying to get the answer. What you're doing
Speaker 1: in the process is you're revealing sources of ignorance, and
Speaker 1: your colleagues on your team, for example, if it's because
Speaker 1: our forecasters often work together on teams, uh, your colleagues
Speaker 1: can help you correct help correct your errors. So what's
Speaker 1: the population of Chicago, I don't really know, between two
Speaker 1: point five and four million, and maybe you know some
Speaker 1: somewhere in the middle there, what proportion of the population
Speaker 1: would have a piano and so forth? You see, if
Speaker 1: you would, you would break it down and you you
Speaker 1: you try to mind eventually how how how many people
Speaker 1: could conceivably make a living working as as piano tuners
Speaker 1: given the number of people who own pianos in Chicago
Speaker 1: and their willingness to pay for the services of piano
Speaker 1: tuners um and so so Faremy did this. He One
Speaker 1: legend is that he tried to uh infer the strength
Speaker 1: of the first atomic blast um by dropping little pieces
Speaker 1: of paper when the when, when the when the winds
Speaker 1: came in front, and by estimating how far the winds blew.
Speaker 1: And I think he was off by about forty or
Speaker 1: fifty percent. But you know that for Faremy estimates, that's
Speaker 1: not too bad. It's it's a lot better than simply
Speaker 1: shrugging your shoulders and saying I have no idea. I
Speaker 1: say a ten Kelton glass as post of twenty. But
Speaker 1: that's that's fascinating. I'm absolutely entranced by Firm's paradox, which says,
Speaker 1: where where is everybody? You know, it's a giant universe
Speaker 1: filled with different galaxies and hundreds of billions of stars?
Speaker 1: And are we really the only intelligent life here? And
Speaker 1: I found most of the various arguments both ways to
Speaker 1: be lacking. It's really it's really a fascinating, fascinating debate.
Speaker 1: But let's let's stick with this. So so looking at
Speaker 1: what the empire state building ways, or or how many
Speaker 1: piano tuners in Chicago show us how to break down
Speaker 1: unknown questions into component parts and make reasonable assessments and
Speaker 1: reasonable valuations on each of those segments. Um. So, so
Speaker 1: what do you find about teams of super forecasters? How
Speaker 1: much better are they at at these sort of predictions
Speaker 1: than the average person, or prediction markets or just regular pundits. Well, Um,
Speaker 1: the teams have super forecasters truly astonished us because the
Speaker 1: statisticians were telling us the right thing to do here
Speaker 1: was to have each individual top performer make judgments completely
Speaker 1: independ at lay of the others, um, rather than allowing
Speaker 1: them to contaminate each other. And you get conformity, and
Speaker 1: you get group think, and you got to get kind
Speaker 1: of a blur rather than a number of distinct points
Speaker 1: of view, and then you can combine them statistically somehow.
Speaker 1: So Um, we were the only competitor in the forecasting
Speaker 1: tournament sponsored by the U S Intelligence community that used
Speaker 1: teams um And but we we hedged our bets. We
Speaker 1: weren't sure that teams would work. We ran an experiment
Speaker 1: and we randomly assigned people to teams, and we random
Speaker 1: and had other people work as individuals. Uh. And we
Speaker 1: were truly surprised that the teams functioned as well as
Speaker 1: they did UM, and it's an interesting question of why
Speaker 1: our teams were so dynamic and open minded relative to
Speaker 1: many teams you see in actual organizations. But before you
Speaker 1: answer that question, let's let's put a little flesh on
Speaker 1: the bone with some numbers. So teams of ordinary forecasters
Speaker 1: beat the wisdom of the crowd by about ten. They
Speaker 1: were bested by prediction markets UM and addiction markets beat
Speaker 1: ordinary teams by about and then the super teams of
Speaker 1: the best forecasters beat the prediction markets by anywhere from fifteen.
Speaker 1: So these folks working in groups really are the outliers.
Speaker 1: None of the other groups are even close to them
Speaker 1: in terms of of accuracy. Why do you think that is, Well, yeah,
Speaker 1: and I'll just add one other thing to that. They
Speaker 1: weren't just outperforming prediction markets in the public sphere, they
Speaker 1: are also outperforming intelligence analysts who were working, you know,
Speaker 1: behind a veil of of of classified information. And it's
Speaker 1: just a remarkable thing. And I think US intelligence community,
Speaker 1: which is much maligned for many things, deserves some credit
Speaker 1: for its willingness to sponsor of forecasting tournament that has
Speaker 1: the potential to be embarrassing for government bureaucracy. How often
Speaker 1: do you see a government bureaucracy, uh, spend money, a
Speaker 1: lot of money on a project that has the potential
Speaker 1: to be to yield results that you're fairly embarrassing. How
Speaker 1: hard is it to cultivate these skills or is it
Speaker 1: just a matter of internalizing these ten bullet points? I
Speaker 1: wouldn't say just a matter. It's it's no, it's an
Speaker 1: it's a non trivial thing to do. Uh it's it's
Speaker 1: it's pretty hard. Um. And uh so let's let's take
Speaker 1: get an example of one. And what do you think
Speaker 1: is an important um, an important commandment of super forecasting?
Speaker 1: You want to just pick up one at random? Sure? Okay, Um, well,
Speaker 1: one of them has to do with granularity and uh
Speaker 1: it's it's it's actually grounded in a story that about
Speaker 1: President Obama and how are he reacted to his advisors
Speaker 1: who were um offering him um somewhat conflicting probabilities on
Speaker 1: how likely it was that Osama bin Laden was residing
Speaker 1: in a particular compound in Abadabad, Pakistan. As we all
Speaker 1: know now, he was indeed residing there, and the President
Speaker 1: did authorize a n a vcal mission, and that resulted
Speaker 1: in Osama bin Lad's death. UM. Now, when the President
Speaker 1: was confronted by these probability estimates from really smart people
Speaker 1: at the top of the intelligence apparatus, ranging from about
Speaker 1: you know, thirty five or up to about um the
Speaker 1: President's reaction was an interesting one. If if you Hee
Speaker 1: had computed the average of the median estimate of the
Speaker 1: advice he was getting, he would have said, looks like
Speaker 1: about a s probability. But instead he said something interesting.
Speaker 1: He said, well, look, look, guys, this is a coin flip.
Speaker 1: It's a fifty fifty thing. Um. Now, the President, I
Speaker 1: think is a very intelligent person, and I think he's
Speaker 1: capable of being very granular in his assessments of uncertainty.
Speaker 1: And if you doubt it, think about the following thought experiment,
Speaker 1: which is appropriate given it approaching March Madness. He follows
Speaker 1: March Madness he uh, and basketball. He's a basketball fan.
Speaker 1: Imagine he'd been sitting around with friends waiting for Duke
Speaker 1: to play some team in March Matt Martin the March
Speaker 1: Madness tournament, and uh, they offered him exactly the same
Speaker 1: probabilities about whether Duke would win the win the game.
Speaker 1: You know, somewhere between thirty five and ninety five, with
Speaker 1: the center of gravity of opinion around seventy, would you
Speaker 1: have said it sounds like a fifty fifty thing or
Speaker 1: would you have said, MM sounds like about UM three
Speaker 1: to one, Duke UM, I think to ask the question
Speaker 1: is to answer it. He would have seen opportunity for
Speaker 1: being much more granular in making bets about sports than
Speaker 1: he would and making estimates about the likelihood of a
Speaker 1: particular terrorist being in a particular location. UM. Now it
Speaker 1: turns out that UM for many categories of problems where
Speaker 1: we think it's impossible to be more granular, it is possible.
Speaker 1: And that's one of the things super forecasters have learned
Speaker 1: that there's a difference between fifty and sometimes they can
Speaker 1: even make distinctions between fifty and now we quote that
Speaker 1: the Chief Risk Officer of UM a q r M,
Speaker 1: the hedge fund UMAST is the head of that, and
Speaker 1: the chief risk Officer is Aaron Brown. And when we
Speaker 1: talked with with Aaron, he you know, he he's also
Speaker 1: a really serious poker player, UM, and he said, well,
Speaker 1: he can tell it different. World class poker player and
Speaker 1: a talented amateur on the basis that the world class
Speaker 1: player UM knows the difference, and then he paused that
Speaker 1: maybe it's more like UM or indeed two, how granular
Speaker 1: can you get in poker? Well, poker is a game
Speaker 1: with repeated play, quick clear feedback. It's possible to get
Speaker 1: more granular on poker than it is about the location
Speaker 1: of terrorists or about whether countries are going to leave
Speaker 1: the Eurozone. But it's an open question of how granular
Speaker 1: you can get UM. And you need to grapple with
Speaker 1: this distinction between precision and pseudo precision UM. And that's
Speaker 1: one of the things. Super forecasters are just very thoughtful
Speaker 1: people who pushed the frontiers of knowledge as far as
Speaker 1: they can, and that means sometimes pushing them a little
Speaker 1: too far, in which case they retreat. If people want
Speaker 1: to find your work, just google Philip Tetlock and they'll
Speaker 1: be able to dig up all of your various publications, books, writings, etcetera. Yeah,
Speaker 1: Google scholars probably a little faster, but all right. If
Speaker 1: you've enjoyed this conversation, be sure and check out all
Speaker 1: eighty three or so of our prior conversations. Be sure
Speaker 1: and follow me on Twitter at rid Halts and check
Speaker 1: out my daily column on Bloomberg View dot com. I'm
Speaker 1: Barry Ridhults. You're listening to Masters in Business on Bloomberg Radio.
Speaker 1: Welcome to the podcast. This is Barry Ridhills, Professor Tetlock.
Speaker 1: If I don't remember to say this later, thank you
Speaker 1: so much for being so generous with your time. This
Speaker 1: is really um been a fascinating conversation. I have a
Speaker 1: lot of things to go over with you in the
Speaker 1: last twenty minutes or so we have, but there are
Speaker 1: a few questions that I'm just dying to ask you
Speaker 1: because it's your lat previous book really was very influential
Speaker 1: to me on expert political judgment. It was that book,
Speaker 1: and it was a prior book called The Fortune Sellers
Speaker 1: that really was more of a media criticism of this
Speaker 1: parade of people who would come through the studios make
Speaker 1: their outland just forecast, be completely wrong, never be held accountable,
Speaker 1: and then they would get called back and the more outlandish, uh,
Speaker 1: the better there were. There was an author who wrote
Speaker 1: a book I'm trying to remember what year the book was.
Speaker 1: It was called The Tao Jones T. A. O. Bennett Goodspeed,
Speaker 1: and he called these folks the articulate incompetence plural, meaning
Speaker 1: that they're very good salespeople. They can speak, but really
Speaker 1: they have no expert knowledge. And if you've spent any
Speaker 1: time in green rooms in various television studios, there's something
Speaker 1: to that. So let me ask the basic question that
Speaker 1: we kind of skirted around during the broadcast portion. Why
Speaker 1: are we so enamored with forecasting and forecasters despite their
Speaker 1: terrible track records. Well, I think because there's a lot
Speaker 1: of motivated reasoning going on. As we noted earlier, there's
Speaker 1: this tendency to use a lot of a do a
Speaker 1: lot of a verbiage forecasting, to to paint a dramatic
Speaker 1: scenario and then hold it together with some very weak
Speaker 1: verbs like this might happen or could happen or the
Speaker 1: distinct possibility of this happening. Um. So, there's this interesting
Speaker 1: tendency that the pundits have of engaging our attention with
Speaker 1: a vivid scenario disintegration of the Saudi regime or you know, uh,
Speaker 1: Cino Japanese wars, something Tom Clancy issue on that kind
Speaker 1: of scale, um and and and but to stitch it
Speaker 1: all together with terms weasel weasel word terms that allow
Speaker 1: them to retreat later on. And UM, we don't distinguish
Speaker 1: very clearly in our own minds. We we don't think
Speaker 1: we want to hold as they say, all of the
Speaker 1: fault lies with the pundits. They couldn't do this unless
Speaker 1: unless we were willing partners. UM. And I think that
Speaker 1: you know here here I am talking to radio station
Speaker 1: one of the most influential companies in the world, Bloomberg. UM.
Speaker 1: Bloomberg is a major purchaser of expertise. UM. Bloomberg could
Speaker 1: actually change the world to some degree if it implemented
Speaker 1: systematic uh, if it implemented systems for tracking the accuracy
Speaker 1: of many of the people who came through. If part
Speaker 1: of the price for getting onto Bloomberg was that you
Speaker 1: had to demonstrate that you were engaging in some kind
Speaker 1: of rigorous scorekeeping, and Bloomberg could flash up some of
Speaker 1: some batting average statistics. UM, as you as you appear,
Speaker 1: Um you Bloomberg could increase the collective IQ of our society.
Speaker 1: It could increase the collective IQ of the conversation. Um.
Speaker 1: When most pundits stay away, Well, well that's the question
Speaker 1: I mean. I I say Bloomberg because Bloomberg is so influential,
Speaker 1: I think a lot of a lot of punets would say, well,
Speaker 1: I'm not I'm not going to run away and high
Speaker 1: from blue Burg. But the other media could do this
Speaker 1: to the Wall Street Journal in New York Times. There
Speaker 1: are lots of major media science that have the leverage
Speaker 1: that could induce pundits to be much more intellectually honest.
Speaker 1: They choose not to exercise that option. Um probably because
Speaker 1: they don't perceive a great market demand for it. We
Speaker 1: were talking during the break about my usage of a
Speaker 1: little app called follow up then dot com. Whenever I
Speaker 1: see an outregeous forecast, I just shoot an email to
Speaker 1: a specific date, so Gold going to five thousand dollars,
Speaker 1: and I send the forecast. I send the email out
Speaker 1: to Oh, well, let's let's let's give them a year,
Speaker 1: so we'll send a forecast out an email out March
Speaker 1: one at follow up then dot com with the headline
Speaker 1: and the web address of the article that made this forecast.
Speaker 1: And then a year later, or if I write March one,
Speaker 1: five years later, comes the email back that specifically gives
Speaker 1: me that lank. Oh it's a reminder. Here's what this
Speaker 1: person said a year ago. And occasionally I get to
Speaker 1: do an article about, Hey, here's a wild forecast that
Speaker 1: someone made and it's been completely wrong. So let's talk
Speaker 1: about your book. You kind of call me out for
Speaker 1: calling someone else out, and I'm curious as to your
Speaker 1: perspective on this. So in two thousand and ten, when
Speaker 1: the FED was in the midst of doing quantitative easing, uh,
Speaker 1: there was a letter published and I believe online and
Speaker 1: it ran in the Wall Street Journal in a number
Speaker 1: of places warning that quantitative easing was going to cause
Speaker 1: hyper inflation and collapse of the dollar and all these
Speaker 1: terrible things. And so I figured, you've got to give
Speaker 1: those people three years. So I set a reminder for
Speaker 1: three years later. And three years later it popped up. Hey,
Speaker 1: the dollars at multi year highs. There there is no
Speaker 1: hyper inflation, there's there's deflation. These guys were wrong, and
Speaker 1: so I called them out about it, and it went
Speaker 1: totally viral, got picked up by a dozen different media outlets,
Speaker 1: and a book called super Forecasting. Now at by today
Speaker 1: we're six years forward, um, and we still have a
Speaker 1: strong dollar and and no inflation. What is the issue
Speaker 1: with an ambiguous forecast? With no specific I described forecasts
Speaker 1: as an asset class, a price, and a specific date.
Speaker 1: If you leave out the specific date, do you get
Speaker 1: to say we're never wrong because there's just wait, you'll see.
Speaker 1: Is is that a fair defense of that? It's definitely
Speaker 1: not fair, but it it is the state of the
Speaker 1: art at the moment. You know. There's an old communism
Speaker 1: joke that that that's rather a proposed here um the
Speaker 1: Soviet revolutionarily on Trotsky after he was thrown out of
Speaker 1: the Soviet Union by Stalin, when it went around giving
Speaker 1: talks to the left wing audiences around the world, and um,
Speaker 1: one probably apocryphal story has it that he went once
Speaker 1: when he was introduced to an audience of followers. Uh,
Speaker 1: the speaker said it was was proclaiming Leon trotskya visionary
Speaker 1: who could see far, far into the future. And you're saying,
Speaker 1: and you know, comrades, the ultimate proof of the far
Speaker 1: sightedness of of of of comrade Trotsky, not one of
Speaker 1: his predictions has yet come true. That's how far sighted
Speaker 1: he is. The the old joke about market forecasting is
Speaker 1: you could give a price level or a date, but
Speaker 1: never both at once. And that's just another way. So
Speaker 1: so how long do we allow a forecast to persist
Speaker 1: before we say, all right, at this point it's been
Speaker 1: X number of years, you we're gonna have to put
Speaker 1: you in the incorrect column. Well, there's there's no absolute rule,
Speaker 1: because that's that's the way vague verbiage is vague. Verbiage
Speaker 1: is vague because it's it's just it's it's always a slippery,
Speaker 1: vague um thing that no nobody knows how to quantify
Speaker 1: it and and keeps it, keeps the pundit safe. Um.
Speaker 1: I mean, I have lots of forecasters from the earlier
Speaker 1: work who predicted that Canada would disintegrate, or Nigeria would disintegrate,
Speaker 1: or there are a lot of disintegration scenarios out there
Speaker 1: that haven't happened yet. Um. And if you call them
Speaker 1: on it, are they going to say, well, we're it's
Speaker 1: it's only seen it could still happen, absolutely and and
Speaker 1: wholeheartedly believe it. That's right. So how much of this
Speaker 1: is just simply humans not being immune to human behavior?
Speaker 1: You're a psychologist, Let's let's let's take that tact. Is
Speaker 1: this just ordinary human behavior refusal to accept responsibility for error,
Speaker 1: not wanting to admit being wrong, not wanting to do
Speaker 1: anything that reduces their potential um status within the hierarchy.
Speaker 1: Is that all this is well, we're moving into world
Speaker 1: that's requiring us to make ever subtler distinctions among degrees
Speaker 1: of uncertainty. This orts of distinction as we didn't have
Speaker 1: to make in our evolutionary past um when we were
Speaker 1: wandering the savannah plains of Africa. You know there were
Speaker 1: are there is a lion or isn't a lions during
Speaker 1: in the long grass and you're gonna you're gonna make
Speaker 1: you have to make a judgment call pretty darn quickly.
Speaker 1: And if you dontle very long, and maybe you're not
Speaker 1: likely to pass your genes onto the next generation. You're
Speaker 1: better off being wrong but jumping the gun then having
Speaker 1: a higher, better track record. But if you're wrong once,
Speaker 1: well then it's catastrophic in terms of progeny. And that
Speaker 1: makes a lot of sense. That sort of thinking is
Speaker 1: why we have a tendency to do all sorts of
Speaker 1: things that just are inappropriate in investing but worked really
Speaker 1: well way back when that's right and um, if you
Speaker 1: can imagine a scenario, here's the here's the big problem
Speaker 1: with tail risks and scenarios. Um, most of the time
Speaker 1: people underrate them, but as soon as the scenario has
Speaker 1: called to their attention, they overrated. Well, it's very very
Speaker 1: caul for people to strike the right balance in dealing
Speaker 1: with tailor risk scenario. So so that whole recency effect
Speaker 1: thing is since nobody, very few people were forecasting the
Speaker 1: sort of financial crisis we had in O eight oh nine,
Speaker 1: and since then it's been nothing but catastrophic forecasts from
Speaker 1: the recession never ended, We're going to turn into a depression.
Speaker 1: Here comes another eight seven like stock crash, the parade
Speaker 1: of horribles just have not now auto loans and the
Speaker 1: new subprime it's going to be just like, uh, is
Speaker 1: this just that that tel risk factor is so recent
Speaker 1: in people's minds and they missed it coming, and so
Speaker 1: now they're just like every general fights the last war.
Speaker 1: These folks are still fighting the previous financial crisis and
Speaker 1: the diplomats try to avoid the last war. Um, yeah,
Speaker 1: I think that's right. I mean, you are you you
Speaker 1: you go from Iraq to Syrian Libya or you that
Speaker 1: you go from one error to another. And Iraq was
Speaker 1: far enough after the Aetnam that it looked like all
Speaker 1: it takes as a generation before those lessons are lost,
Speaker 1: more or less yet more or less. All right, so
Speaker 1: I only have you for another ten minutes, and um,
Speaker 1: my last question before I jumped to my my standard questions.
Speaker 1: Burton Malkiel said, when investors moved from stock to stock
Speaker 1: or mutual fund to funds as if they were selecting
Speaker 1: and discarding cards in a game of jin rummie, what
Speaker 1: does this tell us about human's ability to participate in
Speaker 1: uncertain equity markets? Well, this is another one of these
Speaker 1: the ten commandments that we formulated from observing the super forecasters.
Speaker 1: They're acutely aware of the principle of error balancing. If
Speaker 1: you look at the research literature on human judgment, there
Speaker 1: are two kinds of errors people can make in that
Speaker 1: situation you're describing. One of them is the error of
Speaker 1: excess of volatility, of um jumping every time there's a
Speaker 1: little bit of news and exaggerating its diagnostic value visa
Speaker 1: deep market trends. And the other big mistake you can
Speaker 1: make is excessive rigidity and being so committed to a
Speaker 1: particular preconception about where the future is going that you
Speaker 1: just ignore the news altogether. And you don't you fit,
Speaker 1: you fit, you failed to do any updating. So it's
Speaker 1: it's it's kind of principal error balance things. But like
Speaker 1: writing learning how to ride a bicycle. Um, I mean,
Speaker 1: I could talk for hours about the principles of error
Speaker 1: balancing and everybody is like, yeah, yeah, I kind of
Speaker 1: get at Sure you can make one air, you can
Speaker 1: make the other. But the only way it's really going
Speaker 1: to sink into people's heads if they is if they
Speaker 1: go to forecasting tournaments and they actually practice making judgments,
Speaker 1: get on the bike and try to ride it. Uh.
Speaker 1: And that's what going to forecasting tournaments all about. That
Speaker 1: that's why we were continuing to run the gj open
Speaker 1: dot com, which is a forecasting tournament where people can
Speaker 1: indeed uh work to cultivate their skills. That sounds pretty
Speaker 1: pretty fascinating. So that's that's really interesting for that. Actually,
Speaker 1: we have a little more time, so I'm going to
Speaker 1: keep bangalway on some of these questions. In the first
Speaker 1: book on in the previous book on Expert Political Judgment,
Speaker 1: you know, I look at these is really two sides
Speaker 1: to the same coin. The first book talks about what
Speaker 1: is essentially long term forecasts really a year and further out,
Speaker 1: and they tend to be wrong. The book on super
Speaker 1: forecasting is really looking at a year or less. So
Speaker 1: are are they really saying two different things where we're
Speaker 1: really looking at two different types of forecast, two different lengths.
Speaker 1: I think that's a superb point. There are different time
Speaker 1: periods in the different studies. UM. I'm much more optimistic
Speaker 1: that we can improve forecasting using the right selection, training,
Speaker 1: teaming tools in shorter time periods up to about a year.
Speaker 1: I become progressively more pessimistic when you go out to
Speaker 1: the longer reaches of three, five, ten years that were
Speaker 1: included in Expert Political Judgment, and there I think it's
Speaker 1: it's going to be very, very difficult to do much better, UM,
Speaker 1: do much better than chance. How many times do you
Speaker 1: have to shuffle a deck of cards, um, until it's
Speaker 1: perfectly random? That's a good question. Well, I think the
Speaker 1: UM I would guess five. The statistician and magician Percy
Speaker 1: dot Com is Stanford. I think he estimated at seven. Well,
Speaker 1: life is like shuffling the cards? How how many months,
Speaker 1: how many years have to go by before so many
Speaker 1: random contingencies accumulate that no no human being could conceivably
Speaker 1: have anticipated anything that far out. And um O our
Speaker 1: current best guests for the kinds of geopolitical geoeconomic questions
Speaker 1: we're looking at, it is around a year or so.
Speaker 1: Isn't that just the nature of society and a complex
Speaker 1: system such as fill in the blank, stock markets, economy, geopolitics, elections,
Speaker 1: anything along those lines. They're so sensitive to an all conditions,
Speaker 1: they're nonlinear that you end up with a little change
Speaker 1: here has out sized impact further down the road. Really,
Speaker 1: what we're saying is the universe is pretty random beyond
Speaker 1: twelve months. Uh, it's practically impossible to make any sort
Speaker 1: of realistic forecast with any degree of specificity. Um about
Speaker 1: certain things. I made a bet four years ago with
Speaker 1: um I won't mention his name, but he's been a
Speaker 1: guest on the show as to I was wondering who
Speaker 1: was the GOP nominee likely to be? This is literally
Speaker 1: three years ago after after the election, and he said, well,
Speaker 1: what about the Democrats? And I said, well, it's easy,
Speaker 1: that'll be Hillary, but who I have no idea who
Speaker 1: the Republican is going to be. So we made a bet,
Speaker 1: and so far looking pretty good. Um. But I don't
Speaker 1: know if I got lucky, just got lucky and assumed
Speaker 1: it was she was up next, she was next in line. Uh.
Speaker 1: Under normal circumstances, when you're looking out to three or
Speaker 1: four years, there are so many contingencies. Can anybody really
Speaker 1: I got lucky. I'm not pretending I have any expertise
Speaker 1: in politics or anything else, But can anybody consistently have
Speaker 1: any sort of acumen thinking out more than twelve months?
Speaker 1: It's it's really hard. I mean, there are there are
Speaker 1: some categories of questions where great longer foresight is possible.
Speaker 1: I think the Hillary thing was in the cards for
Speaker 1: for for for quite a long time. Um. But for
Speaker 1: most of these things, I mean, who mean to take
Speaker 1: some extreme examples, I mean, in nineteen forty Dwight Eisenhower
Speaker 1: was an anonymous Army colonel. In nineteen fifty two years
Speaker 1: president United States in twelve years, right, uh in um Um.
Speaker 1: Jimmy Carter was an anonymous peanut farmer in the nineteen
Speaker 1: sixty four. In nineteen seventy six, he was being elected
Speaker 1: president United States. Twelve years is a huge amount of
Speaker 1: time in politics, So I don't think anybody really is
Speaker 1: going to suppose there's very much possibility there UM, But
Speaker 1: as you get closer and closer, it gets more and
Speaker 1: more possible. That's not all that surprising. It's an analogy.
Speaker 1: Will be like to Snell and I char when you
Speaker 1: visit your optometrist and engage, it's easier and easier the
Speaker 1: closer up you get for most things. UM. The trouble
Speaker 1: is we're just not very well tuned to the parameters UM.
Speaker 1: And if somebody can tell a really good story about
Speaker 1: a relatively far off future about you know, the United
Speaker 1: States is moving toward a techno utopia and which DDP
Speaker 1: will will skyrocket as intelligent machines do amazing things for
Speaker 1: us in the fourth and Dulsta Revolution. That that meant
Speaker 1: may that scenario may indeed materialized by its UM, but
Speaker 1: the likelihood of scenarios of that sort being accurate UM
Speaker 1: is extremely low. I have a t shirt at home.
Speaker 1: It says Where's my jet pack? I was promised the
Speaker 1: jet pack by the year two thousand under forty four characters. Right.
Speaker 1: That's right. When when you look back at future forecasts
Speaker 1: from decades ago, and we now have enough for them
Speaker 1: that we can look back years as to what people
Speaker 1: were expecting from the future. What's fascinating is all the
Speaker 1: amazing technology, technological developments, all the advantages of hardware, software, biotechnology,
Speaker 1: medicine that we practically take for granted. They weren't the
Speaker 1: things that people were forecasting. It was colonizing Mars and
Speaker 1: other sort of hoverboards and other such things. Uh So,
Speaker 1: even when you're thinking in terms of giant technological changes,
Speaker 1: and of course there's a handful of people, you know,
Speaker 1: Arthur C. Clarke is notorious for having forecast everything from
Speaker 1: cell phones to satellites to to what have you. Um
Speaker 1: what does this say about our ability to understand the few,
Speaker 1: the present and extra appelate to the future. What it
Speaker 1: suggests is we'd be better off if we were aware
Speaker 1: of our limitations, achieved a certain baseline of appropriate humility,
Speaker 1: and got in the habit of keeping score, and resisted
Speaker 1: being sucked into clever scenarios and storytellers, and resisted being
Speaker 1: seduced by credentials. If we could manage to do those things,
Speaker 1: I think we would um proceed through life, making investments
Speaker 1: and political decisions with better calibrated probabilities. And I think
Speaker 1: we would be better office individuals and we'd be better
Speaker 1: officers of society. That sounds that sounds tremendous. On a
Speaker 1: related note to that, because because those those seven bullet
Speaker 1: points are very significant. Let's let's talk about uncertainty, which
Speaker 1: you is a is a uh concept that is dotted
Speaker 1: throughout actually both books. Um, what is uncertainty and what
Speaker 1: does it mean for individuals just trying to navigate their
Speaker 1: way through the world. Do we understand uncertainty? Uh? Do
Speaker 1: we misunderstand it? What? What exactly is it relative to
Speaker 1: thinking about the future? Well, um, there are some types
Speaker 1: of problems where the probabilities can be readily computed. We
Speaker 1: can compute the probability of drawing an asis spades from
Speaker 1: a randomly shuffled deck, um very accurately, uh, too many
Speaker 1: decimal points if we want whatever one out of over
Speaker 1: fifty two works out too. We can do that with
Speaker 1: coin toss games and so forth. Um. So there are
Speaker 1: some games in which the classic rules of statistics want
Speaker 1: oh one very clearly apply and there are well defined probabilities.
Speaker 1: In other words, we know what the range of outcomes are.
Speaker 1: We just don't know what the specific outcome is going
Speaker 1: to be. Got a well defined sampling universe. You've got clear, quick,
Speaker 1: clear feedback about your about your predictions. Um, my of
Speaker 1: the world, most of the world isn't like that. It's
Speaker 1: not like it's definitely not like that. And and and
Speaker 1: the question is what are the limits? How useful is
Speaker 1: it to apply probabilistic forms of reasoning um outside their
Speaker 1: traditional domains of application. Then, in a sense, is what
Speaker 1: the U. S. Intelligence community really wanted to explore? I mean,
Speaker 1: can we do better than say, distinct possibility, which, when
Speaker 1: you look at it carefully, is such an elastic term.
Speaker 1: It could mean anything from one percent to nine. So
Speaker 1: let's talk a little bit about the intelligence community and
Speaker 1: the Defense Department. Um, how did you get involved with
Speaker 1: DARPA and the the competition, the forecasting competition? Right? Well, Um,
Speaker 1: it was I r as DARA, the Intelligence Advanced Research
Speaker 1: Projects Agency is post to DARPA, but it's it's it's
Speaker 1: a cousin and and it and it models itself to
Speaker 1: some degree. I think after dark by it it really
Speaker 1: wants to do radical earth change, world changing forms of research.
Speaker 1: And I think changing how we think about uncertainty would
Speaker 1: would be would be pretty fundamental. Maybe not as fundamentals
Speaker 1: inventing the Internet, but way up there. Um, it would
Speaker 1: be a big deal, um to to to change how
Speaker 1: we how we go about doing things. I think our
Speaker 1: our democracy would would be transformed. I think the finance
Speaker 1: industry would be transformed. It would not it would not
Speaker 1: be not a small thing. These would not be small
Speaker 1: things that how long have they been running this contest?
Speaker 1: So they started. They approached my wife and May when
Speaker 1: we were still on the faculty at University California, Berkeley
Speaker 1: about six years ago, and we had a nice um
Speaker 1: um a set of drinks over at the Clermont Hotel
Speaker 1: in Berkeley, and um we um. We were just astonished
Speaker 1: that the U. S Intelligence community was prepared to run
Speaker 1: a series of forecasting tournaments. I mean, I predicted that
Speaker 1: they would never want to do anything like that. So
Speaker 1: it's kind of ironic, right that I I forecast that
Speaker 1: forecasting tournaments would be impossible. And I know it's being
Speaker 1: a bit of a HEDGEHOWK. What I what I said is, look,
Speaker 1: government bureaucracies don't give a slingshot money to David right
Speaker 1: Glass doesn't give sl slingshot money to David. Why would
Speaker 1: be a massive influential government you're oocracy fifty billion dollars
Speaker 1: or so, I wanted to spread millions of dollars around
Speaker 1: to a bunch of small scale academic competition to see
Speaker 1: whether or not they can do a better job of
Speaker 1: assigning realistic probabilities to things of national security significance. And
Speaker 1: this it didn't make any sense given the normal rules
Speaker 1: of bureocratic behavior in Washington, d c UM and so
Speaker 1: I was too Hedgehogy, I was wrong about that. Now
Speaker 1: when they I'm delighted. I was wrong to say, at
Speaker 1: least when when they came to you the prior book,
Speaker 1: the Expert Political Judgment book, you had really run a
Speaker 1: form of this. You would assembled a mast over eighty
Speaker 1: two thousand separate forecasts from several thousand, was it or
Speaker 1: several hundred political forecasters? It was it was a smaller number.
Speaker 1: It was in the hundreds, but um the um. Yes.
Speaker 1: In a sense, Expert Political Judgment was a small scale
Speaker 1: dry run for what I RPA did on Expert Political
Speaker 1: Judgment was run more on a shoestring budget, whereas UH
Speaker 1: the r PA forecasting tournaments were run on on a
Speaker 1: much more in a much more per fessional, large scale
Speaker 1: basis with and you know, and one of the nice
Speaker 1: things about the R project, and people worry about the
Speaker 1: applicability of research and things like this, but this was
Speaker 1: all independently monitored by the U S intelligence community. I mean,
Speaker 1: these these forecasts were submitted at nine am Eastern time
Speaker 1: every day on the day, every DA, every day. The
Speaker 1: forecasting tournaments are running over four years UM, so there
Speaker 1: is a very clear paper trail. So what what is
Speaker 1: the state of the forecasting contests these days? Is it
Speaker 1: something that they've put aside? What what is the takeaway
Speaker 1: from all that? The takeaways are that it is possible
Speaker 1: to make better probability estimates of events that many people
Speaker 1: thought it would be impossible to estimate probabilistically. And it's
Speaker 1: possible to do that by engaging in systematic talent spawning,
Speaker 1: which you can only do if you're tracking score. And
Speaker 1: it's also possible to do by designing good training modules,
Speaker 1: by putting together teams that are open to dissent and
Speaker 1: uh know how to do precision questioning of each other's assumptions,
Speaker 1: and also by doing a little bit of algory with
Speaker 1: mcmagic um. So do you think the result of that
Speaker 1: contest has changed the way the US intelligence community recruits talent,
Speaker 1: trains talent, and makes forecasts about future events. Well, you'd
Speaker 1: have to ask the U S intelligence community about how
Speaker 1: exactly things have changed. My understanding is that the National
Speaker 1: Intelligence Council now does try to quantify it's probability estimates
Speaker 1: rather than using just vague verbiage forecasting UM. It has
Speaker 1: probability ranges. I think it tries to distinguish at least
Speaker 1: seven degrees of uncertainty, which is a lot more than three.
Speaker 1: H is more than five, which was which was the
Speaker 1: preceding number. I think they may be underestimating themselves. I
Speaker 1: think they could probably get up to ten or fifteen
Speaker 1: if they if they wanted to. Uh. But I think
Speaker 1: they're moving in the right direction. I think there's growing
Speaker 1: interest in crowdsourcing forecast There's growing recognition that UM, the
Speaker 1: average forecast derived from a group of forecasters, is often
Speaker 1: more accurate than most of the individuals from whom the
Speaker 1: average was derived. It sounds kind of magical, but it
Speaker 1: makes it it is true. UM not always true, but
Speaker 1: it's a good way to bet we We've I've been
Speaker 1: critical of some of the prediction markets, not because the
Speaker 1: theory underlying them is wrong. But very often they're narrow,
Speaker 1: they're not diverse, they're not incentivized. All the various things
Speaker 1: you need for a prediction market to work is often missing.
Speaker 1: Um And sometimes the better as the participants are are
Speaker 1: so similar to each other, it's hard to extrapolate that
Speaker 1: out to other other factors. Um. Uh, These these sort
Speaker 1: of contests and the various prediction markets. Can we describe
Speaker 1: these as moneyball for the intelligence community? Is it just
Speaker 1: quantifying data in a way that hasn't been done previously
Speaker 1: to intelligence forecasts. I think that's a great way to
Speaker 1: describe it. Just money ballfing the intelligence community. Um. I
Speaker 1: think it's the movie The The Old World was a
Speaker 1: world with baseball scouts. Uh, cl Clint Eastwood, Trusty baseball
Speaker 1: scouts too. You know, we're gradually being displaced by these
Speaker 1: number crunchers. Um. We're never going to do away with
Speaker 1: people who have deep qualitative insights into the subject matter.
Speaker 1: There are crucial source of inputs. But the question is
Speaker 1: what roles should we be playing as the world changes?
Speaker 1: And I think human judgment will always be playing a
Speaker 1: critical role when we're dealing with human beings. Um Um.
Speaker 1: But there are useful tools for combining human judgment, and
Speaker 1: you can get more out of it than previously supposed.
Speaker 1: That makes a lot of sense. But before we get
Speaker 1: to our our favorite standard questions, anything from super forecasting,
Speaker 1: I might have missed that you want to uh add
Speaker 1: as as worth thinking about before we uh we get
Speaker 1: into a little bit of your history. Well, the thing
Speaker 1: that I most hope if I mean I'm getting older now,
Speaker 1: I mean I've been doing this stuff for thirty plus years, UM.
Speaker 1: And the thing that I most hope lasting legacy of
Speaker 1: this work, and I hope it improves US foreign policy
Speaker 1: and intelligence analysis, but I also hope it improves our democracy.
Speaker 1: And I think in the in the closing chapter, we
Speaker 1: talk about the debate between Paul Krugman and Nil Ferguson
Speaker 1: on various issues, and how it more resembles a food
Speaker 1: fight than it does a serious debate between extremely intelligent people,
Speaker 1: which which both of them obviously are. UM. And the
Speaker 1: question is, could we use forecasting tournaments? Could we structure
Speaker 1: them in ways uh to facilitate more civilized debates on
Speaker 1: issues that matter. So that's why I wrote a piece
Speaker 1: in The New York Times several months ago with Peter
Speaker 1: Skoblick on how we could do that with Iranian nuclear deal? Um,
Speaker 1: and when a one way to proceed would be to say, Okay,
Speaker 1: you've got hawks, you've got doves. You have different opinions
Speaker 1: about what the long term consequences of signing this deal are.
Speaker 1: We don't know for sure which historical trajectory were on.
Speaker 1: Why don't the hawks generate five questions that they think
Speaker 1: they have a comparative advantage in answering? Why don't the
Speaker 1: doves generate five questions and they think they have a
Speaker 1: comparative advantage in answering? And you know what, victory will
Speaker 1: have a clear cut meaning here if the if the
Speaker 1: doves can answer the dove questions better than the hawks,
Speaker 1: and they can answer the hawk questions better, then the
Speaker 1: doves win and vice versa for the hawks. Uh now, um,
Speaker 1: anyone take you up on that? Well, we do have
Speaker 1: a number of people who are participating in the tournament,
Speaker 1: and um, one of the people, and g j open
Speaker 1: dot com has written a memo on on where where
Speaker 1: we are right now? The moderate seemed to be doing
Speaker 1: the best at the moment, but you know that game
Speaker 1: is far from over. I mean, this is just very
Speaker 1: early stages of a long term process. Yeah, where you're
Speaker 1: one of what a tenure treaty, it's quite a way
Speaker 1: is to go. So so when you when you describe that,
Speaker 1: and you mentioned debates, I immediately thought of the political
Speaker 1: debates this year, which at least on the GEOP side,
Speaker 1: have been not your usual policy debates. UM. And I'd
Speaker 1: love to see some of the tenants from super forecasting
Speaker 1: find its way to uh, the political parties and and
Speaker 1: see if we can have a little more substantive discussion
Speaker 1: about when this happens, here's what happens in the future,
Speaker 1: and then hold these folks accountable. That really doesn't seem
Speaker 1: to happen on whether with political experts or politicians. We
Speaker 1: really don't hold their feet to the fire much do
Speaker 1: How far into the future might it be when in
Speaker 1: a in a presidential election, the presidential candidates take pride
Speaker 1: in what their briar scores are. UM, I think it's
Speaker 1: a long way off, judging by what's going on this year,
Speaker 1: to say the least. So So let's talk about UM,
Speaker 1: let's talk a little bit about you personally, rather than
Speaker 1: than some of the books and the ideas that you've
Speaker 1: you've put forth which which have been absolutely fascinating. Um,
Speaker 1: So how did you find your way? You went, you
Speaker 1: became a you went to Yale, you got your pH
Speaker 1: d in psychology? How did you find your way into forecasting?
Speaker 1: This really seems far afield from the traditional UM academic
Speaker 1: realm of that. I was always a pretty strange psychologist.
Speaker 1: I I had interest that it took me pretty deep
Speaker 1: into social science, into political science in particular, into into
Speaker 1: areas of business. But they always interested in organization as
Speaker 1: I was interested in societies and cultures and large entities
Speaker 1: that were not you know that obviously psychology matters there,
Speaker 1: but it's it's a stretch for a psychologist. So in
Speaker 1: my early work I did do a fair amount of
Speaker 1: experimental UM work, but I was also also did a
Speaker 1: lot of archival and naturalistic work. So it was a
Speaker 1: kind of a natural progression for me. Um, who are
Speaker 1: your early mentors? Uh? Well? Um uh. Peter Suitfeld was
Speaker 1: my very first mentor in Canada. I was an undergraduate
Speaker 1: University of British Columbia, and and he was wonderfully supportive
Speaker 1: and of me, and and he believed in me, and
Speaker 1: he he really told me that you know I would
Speaker 1: probably have a pretty good time if I went to
Speaker 1: graduate school at Yale and I took it on faith.
Speaker 1: And I did that, and I met a number of
Speaker 1: people at Yale who helped me. Um. The guy who
Speaker 1: coined the term group think, Irving Janice, was one of
Speaker 1: the people I worked with, and he was quite an
Speaker 1: unusual psychologist. Also. UM. When I got to Berkeley, of course,
Speaker 1: UM Daniel Koneman came along in a few years, and
Speaker 1: he certainly had an influence on me. I already had
Speaker 1: a PhD, and I was I was just recently tenured faculty.
Speaker 1: But k Koneman is as a lot of very influential guy.
Speaker 1: He's he's just a lot smarter than than most of us.
Speaker 1: So it's a it's a good idea to listen very
Speaker 1: carefully when he speaks. I really enjoyed, UM, thinking fast
Speaker 1: and thinking slow, the metaphor for that entire two stage
Speaker 1: way to look at how humans make decisions, either fast
Speaker 1: and instinctual or longer and thoughtful, really just seems to
Speaker 1: make a lot of sense. UM. What other books, uh,
Speaker 1: have you really enjoyed? Whether books have been especially influential
Speaker 1: to you. Another person who influenced me is just um
Speaker 1: uptown here at Columbia University. Robert Jervis his book Perception, Misperception,
Speaker 1: International Politics. It came out when I was in graduate
Speaker 1: school and I could feel myself being tugged towards these topics.
Speaker 1: It was. It's a brilliant analysis of mistakes that have
Speaker 1: caused unnecessary wars, that perception and misperception in international politics.
Speaker 1: By anything else stands out is as interesting or unusual
Speaker 1: to you. Well, um, I mean life evolves in funny, quirky,
Speaker 1: path dependent ways. I mean, you can you can look
Speaker 1: back on your life and you can say, well, it
Speaker 1: was kind of inevitable this happened or that happened. But
Speaker 1: a lot of the things that that led to my
Speaker 1: early forecasting tournament work where I think kind of quirky,
Speaker 1: I mean, it was really kind of quirky. Thats. A
Speaker 1: scholar as young as I was was appointed to a
Speaker 1: National Research Council committee when I was just thirty or
Speaker 1: thirty one year four. Uh, when I when I was
Speaker 1: that young, um by far the most junior member a
Speaker 1: committee like that, and had a lot of senior scientists
Speaker 1: on it. But it gave me opportunities to meet a
Speaker 1: lot of people, and it connected me to resources that
Speaker 1: made it possible to do the early forecasting tournament work.
Speaker 1: It also impressed on me the need to do it
Speaker 1: because there there we were five liberals and conservatives, all
Speaker 1: had very strong opinions about the Soviet Union and where
Speaker 1: things were going. And the liberals thought that Reagan was
Speaker 1: sending us to where the nuclear apocalypse, and the conservatives
Speaker 1: thought that then the Soviet Union wasn't ere as an
Speaker 1: evil empire would never change from within, essentially, as you
Speaker 1: just had to keep up endless pressure and maybe it
Speaker 1: would eventually crack. But you know, they didn't help a
Speaker 1: lot any and they certainly didn't see garbage Of as
Speaker 1: much of a change agent. Initially, um each side, neither
Speaker 1: side really predicted gorbachalv and what Garbagechov did inside the
Speaker 1: Soviet Union in the internal transformations that occurred. Both sides
Speaker 1: could readily explain after the fact what happened. UM. So
Speaker 1: it was this mismatch between virtually zero predictab ability and
Speaker 1: virtually perfect expost explanatory ability that troubled me. And I thought, well,
Speaker 1: you know, if if debates this important, like World War three,
Speaker 1: are are being conducted, this shodily. You know, surely there's
Speaker 1: a better way to do this um And that's what
Speaker 1: led to the early work on expert political judgment. It
Speaker 1: was a way to try, what can we do to
Speaker 1: to keep score and and if we do keep score,
Speaker 1: can we identify um better ways of making judgments. You
Speaker 1: describe something that is an enormous pet peeve of mine.
Speaker 1: In the markets, nobody knows what happens day to day.
Speaker 1: There is zero predictive analysis, and then on any given day,
Speaker 1: the market's up five hundred points, it's down five hundred points,
Speaker 1: and ex post there is always a fantastic narrative explaining exactly,
Speaker 1: here's why oil shot up and why the market rallied
Speaker 1: three hundred points, or here's why this terrible thing happened
Speaker 1: and the market dropped five hundred points. But nobody is saying,
Speaker 1: if this happens tomorrow, then here's a result. It's always
Speaker 1: an after the fact narrative that that seems to be
Speaker 1: consistent across lots of different uh fields, not just politics,
Speaker 1: but markets and economics. And after the fact we're fantastic storytellers.
Speaker 1: Before the fact, we have no idea, And there there
Speaker 1: are situations where we really want to continue doing that,
Speaker 1: though it's not always bad. I mean, the National Transportation
Speaker 1: Safety Board, for example, conducts these ex post postmortems on
Speaker 1: plane plane crashes. One of the reasons why air travel
Speaker 1: has become as safe as it is is because they're
Speaker 1: so good at doing these postmortems. Obviously, they can't predict
Speaker 1: which planes are going to go down, but they become
Speaker 1: really pretty adept at identifying the critical factors that underlie
Speaker 1: plane accidents, and as a result, the rules for pilots
Speaker 1: and the design features of aircraft have changed in ways
Speaker 1: that make us all safer. Safer so um. But they're
Speaker 1: not just making up a story for the six o'clock news.
Speaker 1: They're saying, hey, you know the whole shuttle investigation, the
Speaker 1: O ring fair, that's right. Therefore, that's the most most
Speaker 1: youth worlds. Most sectors don't have a black box that say, hey,
Speaker 1: here's why the engine failed at two oh seven and fifteen.
Speaker 1: That's right. So there are different types of postmortems, and
Speaker 1: some of them are constrained by well defined bodies of
Speaker 1: scientific knowledge and investigative procedure that reduced the serious risk
Speaker 1: of capitalizing on chance, and others are just sort of
Speaker 1: make it up as you go, and I think the
Speaker 1: things we're talking about and make it up as you go.
Speaker 1: But there are approaches to doing case studies and learning
Speaker 1: from the past that are very disciplined and focused and
Speaker 1: can make a safer and wealthier and happier. The The
Speaker 1: recent book Um the Checklist talks about how much better
Speaker 1: surgical procedures and outcomes have been since surgeons started using checklists,
Speaker 1: including wash your hands, which very often was just assumed
Speaker 1: um that it was done properly with a certain disinfectant
Speaker 1: in a certain life of time. But we've we've apparently
Speaker 1: dramatically reduced operating instruments left in abdomens and keeping count
Speaker 1: of the number of sponges, and that's improved the subsequent outcome,
Speaker 1: just as the National Transportation Safety Board has improved the
Speaker 1: safety level of of travel. That's right. So the big
Speaker 1: question for us is when is learning possible? When can
Speaker 1: we learn to do certain categories of things better? And
Speaker 1: when are we just spinning our wheels and deluding ourselves?
Speaker 1: Are we spinning our wheels and deluding ourselves about financial markets?
Speaker 1: So many political issues, but we're actually making real progress
Speaker 1: in the domains of medicine, or airline safety or or whatnot.
Speaker 1: And it's a it's a mixed picture. Um. And I
Speaker 1: suppose what we're trying to do with this forecasting tournament
Speaker 1: work is to bring some of the rigor that has
Speaker 1: worked in these more scientific domains to bring it to
Speaker 1: bear in in domains that they are more or less
Speaker 1: like the wild West. Huh. Um. So let's you've been
Speaker 1: doing this now for you said, thirty years. What's changed
Speaker 1: in this industry more than anything? What is the significant
Speaker 1: progress in in the forecasting and prediction industry? Um, during
Speaker 1: the course of your career. That's a hard question. Um,
Speaker 1: They're not all there there there, that's right, well there there.
Speaker 1: I I think that our knowledge of the imperfections and
Speaker 1: human judgment, thanks to a lot of the comment inspired
Speaker 1: research programs, I think we've made discernible progress there. To
Speaker 1: the least, I think we've I think some of the
Speaker 1: statistical tools have improved in various ways. I think some
Speaker 1: of the tools for running teams have even improved. I mean,
Speaker 1: I think we can do There are versions of the
Speaker 1: Delphi procedure, for example, which was developed a long time ago,
Speaker 1: but has got the DELFI better, right, so, which is
Speaker 1: what I remember. I was saying that a lot of
Speaker 1: people thought it was kind of crazy to use teams.
Speaker 1: You're better off having a lot of independent observers. But
Speaker 1: there's a way to get the benefits of independence and
Speaker 1: the benefits of creative interaction at the same time. And
Speaker 1: one way to do that is by going getting everybody
Speaker 1: to make their judgments anonymously. So you give your probability judgment,
Speaker 1: your explanation, I give mine, and everybody around the table
Speaker 1: gives theirs, and we circulate, and we we circulate that
Speaker 1: and nobody knows who said what. So the high status
Speaker 1: guy isn't swaying everything the way off. It often happens
Speaker 1: in groups. UM, and everybody's expressing your judgments anonymously is don't,
Speaker 1: so they're insulated from the group thing pressure. And you
Speaker 1: can do that two or three times, and then the
Speaker 1: question is how much better is the resulting group judgment
Speaker 1: after you go through this process. Uh, Then it would
Speaker 1: have been if you'd simply say, taken um an unweighted
Speaker 1: average of each of the individual group group group judgments,
Speaker 1: and the answer is this better? How much better? Uh?
Speaker 1: I think met analysis suggests probably in the vicinity of
Speaker 1: ten percent better is you know? That's real? Right? And
Speaker 1: if if you're talking about avoiding a war or finding
Speaker 1: a terrorist or anything along those lines, that's a real
Speaker 1: worthwhile pursuit. It's nothing to sniff at, nothing to sniff at.
Speaker 1: So now let me ask you for your forecast. What
Speaker 1: are the next major changes? What are the next shifts
Speaker 1: that are going to come in in the world of
Speaker 1: forecasts and predictions? What do you see? Perhaps the better
Speaker 1: way to say this to ask this is what do
Speaker 1: you see as the influence of your work on on
Speaker 1: the forecast and community? I see huge potential here, um
Speaker 1: i R, which funded the first forecasting tournament, is going
Speaker 1: to be funding to follow up forecasting tournaments, um and
Speaker 1: I think many of your readers might be interested in these,
Speaker 1: and there will be calls for volunteers to participate in
Speaker 1: one form or another. One of them is focusing not
Speaker 1: so much on the accuracy of your forecast as on
Speaker 1: the probitive value of the explanations you generate. The probate
Speaker 1: of value. Are you good at for or after the fact?
Speaker 1: Are you good at explaining things? Um and? And again,
Speaker 1: the extent to which we can crowdsource aspects of problem
Speaker 1: solving and then eventually marrying that. I think the forecasting, UM,
Speaker 1: I think that's a very ambitious project. It's it's it's
Speaker 1: in the very early stages. It hasn't been launched yet, um,
Speaker 1: but I'm optimistic that it will. UM. Um. So what
Speaker 1: this is called the I r PA forecast things. It's
Speaker 1: called create create c R E A T E. Uh.
Speaker 1: It's it's it stands for the flex reasoning. That's right,
Speaker 1: it's an acronym like that. All right, that sounds interesting.
Speaker 1: I'll definitely I'll search for that and all link to
Speaker 1: that and that. The other is another competition which is
Speaker 1: called h FC, the Hybrid Forecasting Competition, which will be UM,
Speaker 1: humans and machines and human machine combinations, uh, trying to
Speaker 1: make predictions, which I am quite optimistic about. Um. So
Speaker 1: it's Watson working with somebody, well, it would be that
Speaker 1: would be one way of doing it. There are a
Speaker 1: lot of there are a lot of possible machines, are
Speaker 1: a lot of possible models that people could work with.
Speaker 1: And the question is are you better off just using
Speaker 1: the model or you better off just using the person,
Speaker 1: or you're better when are you better off using the
Speaker 1: combination of the model and the person, and the one
Speaker 1: truth is we really don't know the answers to these
Speaker 1: questions right now, and we're hoping to learn more. So
Speaker 1: I think that's these are these are really important projects.
Speaker 1: And I think the other big thing that that I'm
Speaker 1: very focused on because it has relevance to improving the
Speaker 1: debate in our debates in our society, is competitions to
Speaker 1: generate better questions. It's not to generate better questions. It's
Speaker 1: not just about forecasting. It's I mean, you can you
Speaker 1: can forecast trivial pursuits, and you can become a great
Speaker 1: forecaster in trivial pursuits. And really, the world is not
Speaker 1: a better place for it? Or what? What? What? Your
Speaker 1: world will be a better place when we join super
Speaker 1: forecasting skills. Two questions on which big policy debates pivot.
Speaker 1: So you say, if we knew the answer to this question,
Speaker 1: would we have invaded Iraq? Or if we had known
Speaker 1: the answer, if we had known the answer that if
Speaker 1: we if we knew the answer this question, what would
Speaker 1: would have changed how we what we do in Syria,
Speaker 1: the Ukraine, or with respect to tax policy, with respect
Speaker 1: to uh FED policy or whatnot uh so uh generating
Speaker 1: probitive questions, generating high quality explanations, human machine competitions. I
Speaker 1: think these are three really important areas for the future.
Speaker 1: That's fascinating. The last two questions, Um, this is always
Speaker 1: interesting and I'm trying to figure out the best way
Speaker 1: to its phraser for you. Normally, I say, what advice
Speaker 1: would you give to a millennial or someone just graduating
Speaker 1: college who are going into your field? But I don't
Speaker 1: know whether that's the field of psychology or the field
Speaker 1: of analyzing and improving forecasts and predictions. But let me ask,
Speaker 1: in an open ended fashion, what advice would you give
Speaker 1: to someone just coming out of school starting their career
Speaker 1: who wants to follow in your footsteps. Well, it's an
Speaker 1: interesting point that I really don't have a field anymore.
Speaker 1: I mean, the University of Pennsylvania when they hired me,
Speaker 1: they didn't really know where to put me. So I'm
Speaker 1: partly warden, partly psychology, partly a political science, and Nannenburg
Speaker 1: um so a lot of different But it's really the
Speaker 1: you're really studying this, let's let's just, for lack of
Speaker 1: a better phrase, you're studying the science of decision making
Speaker 1: that we're studying human judgment and the extent to which
Speaker 1: human judgment can be improved using a variety of tools,
Speaker 1: some of them drawn from psychology, some of them drawn
Speaker 1: from statistics, some of them drawn from organization theory, a
Speaker 1: lot of different tools. So someone who wanted to go
Speaker 1: into that field, what advice would you give them? I
Speaker 1: would say, Um, that there is no clear path to
Speaker 1: where I am right now. Um that it's it's it's
Speaker 1: not clear to me, Um, where you would go because
Speaker 1: the work I'm doing is so weird and interdisciplinary it
Speaker 1: doesn't fit into any of the existing university niches, Which
Speaker 1: is kind of funny because most university niches have become
Speaker 1: more and more specific and more and more narrowly focused,
Speaker 1: And you're going in the opposite direction, pulling from three
Speaker 1: distinct plus the whole quantitative side of it, three distinct
Speaker 1: areas of practice with a heavy math overlay. Yeah. I
Speaker 1: don't claim to be much more general than many of
Speaker 1: the specialists my specialist colleagues. I think I'm just more
Speaker 1: specialized in a weird way. I mean, I work is
Speaker 1: very specialized and focused. It just draws on different components
Speaker 1: of different disciplines in a very focused way. Um, so
Speaker 1: that's intriguing. I think people who do in new discipline work,
Speaker 1: you know, they're not Leonardo da Vinci. I mean, there
Speaker 1: aren't any Leonardo DaVinci as far as I can tell
Speaker 1: right now in the university world. Uh. Um, what we
Speaker 1: what we do is we we we we and we
Speaker 1: need to carve out a very specialized research programs that
Speaker 1: deliver have tangible deliverables. Uh that we were really on
Speaker 1: a tight accountability leash in this forecasting tournament. It was
Speaker 1: you know, we had we were submitting forecast nine am
Speaker 1: Eastern time every every day. It was. It was a
Speaker 1: very rigorous process and we needed to have very focused group,
Speaker 1: tangible deliverables and a focused process that that seems like
Speaker 1: that's of great value to both business and government. Final question,
Speaker 1: what is it that you know about forecasting today that
Speaker 1: you wish you knew when you started down this road
Speaker 1: thirty years ago? Well? What I wish I knew? What
Speaker 1: I what? So the early work was mostly about cursing
Speaker 1: the darkness. It was about cognitive bias and how we're
Speaker 1: prisoners of our preconceptions, how we have a heart that
Speaker 1: we're too quick to make up our minds were too
Speaker 1: slow to change them. It was a rather dark purtrait
Speaker 1: of human nature. UM. And there's some reason for being pessimistic,
Speaker 1: given the way we think about UM politics and history
Speaker 1: and economics for much of the time UM. The later
Speaker 1: work has been more about lighting candles. It has a
Speaker 1: more upbeat flavor that there are specific things you can
Speaker 1: do to become more open minded, at least about relatively
Speaker 1: near term futures. And if you can become more open
Speaker 1: minded about relatively near term futures, maybe you can become
Speaker 1: a little more open minded about medium and longer term futures.
Speaker 1: Maybe you can be better able to see how alternative
Speaker 1: perspectives might have some merit UM. And I think that
Speaker 1: when you feel you're in competition with the other side,
Speaker 1: and the other side might be getting to the truth
Speaker 1: faster than you, that has a very salutary effect. I
Speaker 1: think it will tend to make us a bit more
Speaker 1: open minded. Thank you so much for being so generous
Speaker 1: with your time, Professor Tetlock. Uh. We've been speaking with
Speaker 1: Professor Philip Tetlock of the University of Pennsylvania UM both
Speaker 1: Wharton and other schools, author of super Forecasting, The art
Speaker 1: and science of prediction, as well as expert political judgment,
Speaker 1: How Good is It? How Can We Know? And a
Speaker 1: number of other books. If you've enjoyed this conversation, be
Speaker 1: sure and look up an Inch or Down an Inch
Speaker 1: on Apple iTunes and you'll see all of our other
Speaker 1: eighty three or so um previous conversations. I want to
Speaker 1: thank Mike bat Nick for doing the deep dive and
Speaker 1: helping me on the research with this. I'm Barry Ridhltz.
Speaker 1: You've been listening to Masters and Business on Bloomberg radioh