Speaker 1: Hello, Odd Lots listeners, It's Tracy Alloway. As you may know,
Speaker 1: Odd Lots is hosting its first ever live event on Thursday,
Speaker 1: September nineteen in New York City. Join me and Joe
Speaker 1: Wisenthal as we host an evening of great conversation and
Speaker 1: live music. The Odd Blots Variety Show will feature some
Speaker 1: new and old Odd Lots guests, including the economist Stephanie Kelton,
Speaker 1: Sam Antar, the former Crazy Eddie CFO, and convicted Felon
Speaker 1: will give us some of his best stories. Meanwhile, Lee
Speaker 1: book Eye, once dubbed the philosopher King of Sovereign Debt Lawyers,
Speaker 1: is coming out of retirement for us. Salton, posar of
Speaker 1: Credit Swiss and formerly of the U. S. Treasury, will
Speaker 1: be on stage with Brad Setzer, Senior Fellow at the
Speaker 1: Council for Foreign Relations, talking all things bonds and trade.
Speaker 1: Will also have some markets the music courtesy of Moral Hazard,
Speaker 1: the most important country singer in economics, and an early
Speaker 1: Odd Thoughts guest, and even Joe has promised us a
Speaker 1: song or two. You can stick around after the show
Speaker 1: for some drinks and a chance to chat with us
Speaker 1: and our guests in person, so keep listening to Odd
Speaker 1: Lots to learn how to sign up to see us
Speaker 1: live on Thursday, September nineteenth. We hope to see you there. Hello,
Speaker 1: and welcome to another episode of the Odd Loots podcast.
Speaker 1: I'm Tracy Halloway and I'm Joe wisnal So Joe. I
Speaker 1: feel and I think we've discussed this before, but it
Speaker 1: feels like the world is sort of at an inflection
Speaker 1: point right now. Uh you think, I mean you think so? Yeah.
Speaker 1: Isn't it always kind of an inflection point? Yeah, guess
Speaker 1: that's true. But at least in markets, it feels like,
Speaker 1: you know, warnings that were late in the cycle that
Speaker 1: we could get a recession at the end of this
Speaker 1: year or in those definitely seem to be heating up.
Speaker 1: I think what I would say and where maybe we
Speaker 1: would uh most likely agree, is not that the world
Speaker 1: is that at an inflection point per se, but that
Speaker 1: what comes and goes is periods when suddenly people feel
Speaker 1: the turn is about to come. And we've had a
Speaker 1: series of these over the last ten years, whether it
Speaker 1: was Q four of last year early with the economy
Speaker 1: going down, the euro crisis. From time to time, it's
Speaker 1: like there's this global collective anxiety that rises, and whether
Speaker 1: it's anymore real or not is up for debate, but
Speaker 1: I would absolutely agree right now you're getting a lot
Speaker 1: of like recession calls, bear market calls, started the easy
Speaker 1: cycle calls, things like that, A wave of global anxiety.
Speaker 1: That's a good way to put it. And this isn't really,
Speaker 1: as you mentioned, the first time that we've seen this,
Speaker 1: and I'm not just talking about markets. We've seen sort
Speaker 1: of inflection points happen in politics recently, right, So Brexit
Speaker 1: springs to mind, the election of Donald Trump as well. Yeah,
Speaker 1: and I think the fact that we have another election
Speaker 1: coming up in the US again, it's one of these
Speaker 1: points where people wanted to call some sort of meaningful
Speaker 1: turn in something or or there's no doubt we're we're
Speaker 1: hearing a lot of that these days. Right. So all
Speaker 1: of this is a roundabout way of saying that we're
Speaker 1: getting a lot of forecasts and a lot of predictions,
Speaker 1: a lot of people trying to call or see it
Speaker 1: into the future. And so I thought it would probably
Speaker 1: be a good idea to do an episode on forecasting.
Speaker 1: I love this idea. I mean, one thing that I've
Speaker 1: always wanted to do and never done? Is you know,
Speaker 1: obviously on TV I talked to people all the time
Speaker 1: and they make forecasts like, oh, we're in this stock
Speaker 1: or interested in this sector, or we're telling clients to
Speaker 1: do this, and I've never gone back and it would
Speaker 1: just be too much work for me, and like actually
Speaker 1: made a database of all their calls. But I've always
Speaker 1: thought that's like a fascinating project or that you know
Speaker 1: that because who knows everybody? You know, people just make
Speaker 1: these calls and how often do they ever get revisited
Speaker 1: to see if the person was actually right or useful
Speaker 1: in some way? Totally, And I have a feeling that
Speaker 1: our guests for this episode is going to have something
Speaker 1: to say on that particular topic. So without further ado,
Speaker 1: why don't I go ahead and bring him in. Our
Speaker 1: guest is Phil Tetlock. He is Annenberg University Professor at
Speaker 1: the University of Pennsylvania. He's also the author of numerous
Speaker 1: books and papers on forecasting. So, Phil, welcome to the show.
Speaker 1: I'm glad to be here. So Phil, I guess my
Speaker 1: first question is, you know, one of the thrust of
Speaker 1: a lot of your work is that experts tend to
Speaker 1: get forecasting wrong. So is it weird? It doesn't feel
Speaker 1: weird that you're sort of the expert on why experts fail. Um.
Speaker 1: I guess I've gotten accustomed to it because I've been
Speaker 1: doing it for about thirty five years now. I started
Speaker 1: out just after I got tenure at Berkeley and been
Speaker 1: tracking the accuracy of experts predictions pretty much continuously since then.
Speaker 1: So I'm sometimes called an expert hologist, which is, of
Speaker 1: course the field that does not does not exist. The
Speaker 1: basic idea of keeping track of of how accurate people
Speaker 1: are is it? Is it really good idea? And and
Speaker 1: insofar as we all kept track of ourselves and the forecasting,
Speaker 1: making how well calibrated we are, I think the research
Speaker 1: suggests we would actually get a little bit better at it.
Speaker 1: So you've actually done what I've said for years that
Speaker 1: I've wanted to do, which is creates some database of
Speaker 1: forecasts and actually go back and look at who's right
Speaker 1: and who's wrong. How did you get the thought that
Speaker 1: that would be worth doing initially? And what does such
Speaker 1: a database look like? Because most forecasts are not particularly binary.
Speaker 1: People might say I chance or seventy five chance, so
Speaker 1: usually things aren't just this will happen or this won't happen.
Speaker 1: And often a forecast can be wrong, but the methodology
Speaker 1: turned out to be right, or maybe someone unpredicted chance
Speaker 1: and that was still the right framework and the five
Speaker 1: percent odds did hit. So how do you go even
Speaker 1: go about constructing a database of forecasts and measuring what
Speaker 1: turned out to be right or wrong? Well, that's what
Speaker 1: ten ures for it takes. It takes a long time.
Speaker 1: The key thing to note is that your right that
Speaker 1: there are only two conditions under which you can definitively
Speaker 1: say at particular forecast was right or wrong, and that
Speaker 1: is that the forecast it was rash enough to say
Speaker 1: a hundred percent chance and it didn't happen, or zero
Speaker 1: percent chance that it did happen, in which case you
Speaker 1: know inclusively that that specific forecast was wrong. But a
Speaker 1: forecaster says, you know, there's a seventy percent chance, like
Speaker 1: Nate Silver said there's a seventy percent chance of Hillary
Speaker 1: Clinton winning the election in November? Uh was was was
Speaker 1: Nate Silver wrong? Or or do we happen to inhabit
Speaker 1: a world that was likely on November first? So the
Speaker 1: solution to the problem is statistical. You need you need
Speaker 1: to keep track of lots of forecasts over time. If
Speaker 1: we collect hundreds of of your forecasts over the course
Speaker 1: of a year, and we find that when you say
Speaker 1: there's a seventy percent chance of things happening, those things
Speaker 1: happen about seventy percent of the time. We say there's
Speaker 1: a ninety percent chance those things happen about ninety percent
Speaker 1: of the time, and so forth. If there's a close
Speaker 1: correspondence between your subjective probability estimates and the object of
Speaker 1: frequency with which events occur, you can be said to
Speaker 1: be reasonably well calibrated. And that's a very desirable feature
Speaker 1: of forecasters. So when you run your analysis, your statistical
Speaker 1: analysy says, of all these expert forecasts, and I'm assuming
Speaker 1: some non expert forecasts as well, what did you find?
Speaker 1: You find that people are not very well calibrated. For
Speaker 1: a start, there's a lot of overconfidence. And when uh,
Speaker 1: many many experts would would would say things are an
Speaker 1: eighty or ninety percent likely or even likely, they would
Speaker 1: occur sixty or seventy percent of the time. So there
Speaker 1: were big gaps between subject of probabilities and object of probabilities.
Speaker 1: There's a lot of overconfidence. People were not qualifying their
Speaker 1: forecast appropriately, and that is one of the better replicated
Speaker 1: findings in my field, which is cognitive social psychology of judgment.
Speaker 1: People are over confident, that tend to be over confident.
Speaker 1: Everybody's over confident. There are even there are a few
Speaker 1: souls who were even systematically under confident and who don't
Speaker 1: have enough confidence in their judgment. But the modal, if
Speaker 1: you had to bet on what kind of mistake people
Speaker 1: are making at any given moment, over confidence would be
Speaker 1: the better bet. You know, I think about something. One
Speaker 1: of our colleagues, I'll give a shout out to him,
Speaker 1: Lorc and Roch Kelly. He works with us here at Bloomberg,
Speaker 1: and he's very fond of strategists at banks and other
Speaker 1: experts who love to give a forecast on things like, oh,
Speaker 1: we see a forty percent forecast of this person winning
Speaker 1: the election, or forty percent forecast of this country leaving
Speaker 1: the EU in the next five years, and it's like
Speaker 1: the perfect number because it's you know, it's still on.
Speaker 1: It feels unlikely, but it's close enough to that if
Speaker 1: it happens, you know, you can still say, oh, I
Speaker 1: told you it was significant. But if it doesn't happen,
Speaker 1: you could say it was unlikely. And I'm curious in
Speaker 1: your findings and in your research. You know, we think
Speaker 1: of forecasts, our predictions is the point is to be accurate.
Speaker 1: But it feels like a lot of the reason people
Speaker 1: give forecast is just to be interesting, just to have
Speaker 1: their voice heard, just to get their clients to pick
Speaker 1: up the phone. And I'm curious how that plays into
Speaker 1: your analysis of forecasts when the whole purpose is maybe
Speaker 1: not even to get it right, it's just to provoke
Speaker 1: a thought or to have your name out in the news.
Speaker 1: I think that's a very perceptive observation. It's a very
Speaker 1: delicate dance that people play. On the one hand, you
Speaker 1: want to say things that sound interesting so people don't
Speaker 1: roll their eyes and things that this is boring and
Speaker 1: a useless conversation. On the other hand, you don't want
Speaker 1: to say things that are so interesting that they could
Speaker 1: prove to be wrong later. So you're there seems to
Speaker 1: be sort of in the sweet spot zone. Or if
Speaker 1: you were to use language, you say, well, I think
Speaker 1: there's a distinct possibility that Putin's next move is going
Speaker 1: to be on Belarus or on Estonia. Uh, it's a
Speaker 1: distinct possibility, is wonderful, and exactly the same way is
Speaker 1: pretty good. I mean, if it happens, I say, hey,
Speaker 1: I told you distinct possibility, and if it doesn't happen,
Speaker 1: I said, I merely said it was possible. So you
Speaker 1: you're you're, you're covering yourself very nicely. It is as
Speaker 1: though the art of punditry is the art of appearing
Speaker 1: to go out on a limb without actually going out
Speaker 1: on a limb. So, I mean, Joe referred to forecasts
Speaker 1: as non binary. But I feel like instinctively a lot
Speaker 1: of people want to know whether something will or will
Speaker 1: not happen, and yet we have all these, uh, forecasting
Speaker 1: calls that are sort of, you know, thirty percent chance,
Speaker 1: forty percent chance. Our probability is a cop out in
Speaker 1: that case, is it's something that people hide under. Well,
Speaker 1: probable probabilities are not a cop out. If you're participating
Speaker 1: in forecasting tournaments in which we can systematically track how
Speaker 1: often you're things happen, and if things you say likely
Speaker 1: happen forty of the time, you're pretty well calibrated, and
Speaker 1: it's not simply being used as a cop out. So, okay,
Speaker 1: So you've built this database and you've been tracking forecasters
Speaker 1: ability for years, and you mentioned that there are forecasting
Speaker 1: tournaments and we can really track this stuff, and that
Speaker 1: we can track how well calibrated forecasters are, not on
Speaker 1: any sort of individual prediction, but by whether over time
Speaker 1: their predictions of likelihood events happened seven out of ten
Speaker 1: times and so forth. What are some of the interesting
Speaker 1: patterns you've discovered besides that people tend to be overconfident,
Speaker 1: what kind of people, what kind of approaches tend to
Speaker 1: distinguish the better forecasters from the worst ones, Because ultimately,
Speaker 1: I think that's sort of the point of your research.
Speaker 1: There are there are two classic biases that we have
Speaker 1: found over the years, and one of them is that
Speaker 1: people are too quick to make up their minds, and
Speaker 1: the other is that people are too slow to change them.
Speaker 1: And it's the combination of those two things that causes
Speaker 1: chronic over confidence. So I'm curious, beyond sort of individual
Speaker 1: characteristics that make people a good or bad forecaster, did
Speaker 1: you notice any discrepancy in the type of forecasts being made, Like,
Speaker 1: for instance, did political forecasting tend to be more or
Speaker 1: less accurate than something like economic or financial forecasting. Oh,
Speaker 1: it really depends on the timeframe and and the and
Speaker 1: the types of questions you're asking. I think economic and
Speaker 1: political forecasting can be pretty hazardous to your reputation. I
Speaker 1: think what we noticed more than anything was that the
Speaker 1: types of forecasters who tended to be better did tend
Speaker 1: to be a little bit more boring. There they were
Speaker 1: more likely to say on the one hand, on the
Speaker 1: other hand, they were they were, they were, they were
Speaker 1: engaging in more explicit balancing and say, well, there's this
Speaker 1: causal for us, and there's that causal for us, and
Speaker 1: you have to balance them against each other. So the
Speaker 1: types of forecasting talk that make forecasters appealing to the
Speaker 1: media tend to details that tend to make forecasts less
Speaker 1: less accurate. So a forecaster is going to be more
Speaker 1: appealing to the media, it would seem if they if
Speaker 1: they can come up with a compelling sound bite and
Speaker 1: they say something like, well, you know, I think the
Speaker 1: Saudi regime is going to collapse within the next twelve
Speaker 1: to twenty four months. That's that's a that's a very
Speaker 1: dramatic forecast. It would have a lot of consequences for
Speaker 1: the Middle East and from World Politics. A forecast who says, well,
Speaker 1: you know, there are people have been predicting the major
Speaker 1: regime change in Saudi Arabia off and on for the
Speaker 1: last forty years. It hasn't happened yet. The base rate
Speaker 1: prediction is it is it's not very likely. Um, there
Speaker 1: are some reasons for some concern, but you know, you
Speaker 1: can steal your eyes start to glaze over. Listening to
Speaker 1: the more accurate forecasters tend to to bore people. Yeah,
Speaker 1: So I'm just thinking so at the time that we're
Speaker 1: recording this episode, just in the last day, and by
Speaker 1: the time you're hearing this, this would be old news.
Speaker 1: But in the last day, uh Deutsche Bank, for example,
Speaker 1: announced a major restructuring of its bank. And I'm just
Speaker 1: thinking about, how like the imperative for the news media
Speaker 1: is immediately to find people who will come on this
Speaker 1: morning and say something about whether this restructuring of the
Speaker 1: bank is likely to be enough, did it go far enough,
Speaker 1: will restore the bank to robust profitability and so forth,
Speaker 1: And it sounds like that imperative is almost exactly the
Speaker 1: opposite of what's likely to make a good forecaster. And
Speaker 1: anyone who already has their mind made up but already
Speaker 1: has a strong view on the efficacy of the plan,
Speaker 1: at least going by your heuristic that the people that
Speaker 1: good forecasting is not correlated with a quick judgment or
Speaker 1: you know, quick decision making, which seems like we're kind
Speaker 1: of like highlighting most likely highlighting some of the worst
Speaker 1: people we could be highlighting. Well, it really you have
Speaker 1: to make a decision about what kind of business you're in.
Speaker 1: If you're in the accuracy business, you're going to look
Speaker 1: for the kinds of forecasters we've been looking for in
Speaker 1: the work we've been doing with the intelligence community and
Speaker 1: elsewhere where. These are going to be forecasters who are
Speaker 1: not very entertaining. If you're in the entertainment business, you're
Speaker 1: going to be looking for people who are entertaining. There's
Speaker 1: a separation in Bloomberg and probably mostly another sophisticated media
Speaker 1: comes these between analytics and and and the front end. Right,
Speaker 1: So what about experience, Like to what degree can if
Speaker 1: you're an expert, presumably you have you know, probably decades
Speaker 1: of experience, and you know you've been studying a particular
Speaker 1: subject matter for a long time, you've noticed patterns, or
Speaker 1: you can reach for historical analogies to describe a current
Speaker 1: situation and extrapolate from that. Does experience help offset the
Speaker 1: problems of overconfidence at all? Sometimes it depends what you
Speaker 1: do with the experience. Um. People have different styles of
Speaker 1: thinking and and some people with experience become extremely skilled
Speaker 1: at creating very comp telling, very articulate justifications why they
Speaker 1: must be right. Uh. So experience can actually solidify dogmatism
Speaker 1: for people with that cognitive style, and either for other people,
Speaker 1: experience melos of them and they become more tuned to
Speaker 1: the limitations of their prior world views and uh, they
Speaker 1: introduce more appropriate qualifications, may become better calibrated. So but
Speaker 1: but it's not a one trajectory. Be people people mature
Speaker 1: in different ways. Talk to us about how you train
Speaker 1: people to get better. So obviously, like let's start with
Speaker 1: the assumption that there are some people that aren't just
Speaker 1: looking for media sound bites and maybe as uh to
Speaker 1: use your example there in the intelligence community, and they
Speaker 1: really want to make better forecasts about how things will
Speaker 1: happen in the future. They want to be better at
Speaker 1: predicting save Vladimir Putin's next move? What is that? How
Speaker 1: do you start and how do you what's the general
Speaker 1: approach to becoming better at that? Well, I would say
Speaker 1: the starting point is again not going to be all
Speaker 1: that exciting and it and its bears a strong resemblance
Speaker 1: to what Danny Kahneman proposes in his best selling book
Speaker 1: Thinking Fast and Slow. Uh, it is start with the
Speaker 1: base rates. And if you look at the base rates,
Speaker 1: you you'll see something quite interesting, and that is people
Speaker 1: frequently claim, to go back to the beginning of our conversation,
Speaker 1: people frequently claim that they're at an inflection point in history.
Speaker 1: If you if you look at how many inflection points
Speaker 1: there have been, it's just a very long list. The
Speaker 1: vast majority of claims about inflection points have been false positives.
Speaker 1: So you you would, naturally, I think, be wary of
Speaker 1: claims about inflection points. Another claim you'd be wary of
Speaker 1: is military coups, the revolutions. They're relatively rare events. So
Speaker 1: someone who's making a dramatic claim but it's going to
Speaker 1: be a regime change in a particular country within a
Speaker 1: particular timeframe, the likelihood their being right is pretty low. Still.
Speaker 1: Joe mentioned us in the intro, but what role do
Speaker 1: you think accountability plays in the sort of forecasting industry
Speaker 1: because it feels to me like, given the volume of
Speaker 1: media that's out there right now, you know, either social
Speaker 1: media or traditional forms of media, it feels like you
Speaker 1: have a lot of people who will make, you know,
Speaker 1: say a hundred predictions, and maybe one or two of
Speaker 1: those are right, and then they get trumpeted for those
Speaker 1: right calls, or you know, they laud themselves for those
Speaker 1: correct calls, and people sort of forget about the other
Speaker 1: nine calls that were wrong, and no one bothers to
Speaker 1: go back and check on them. Because there's just so
Speaker 1: much forecasting and so much information out there in the world.
Speaker 1: So how can we develop accountability for forecasting by running
Speaker 1: forecasting tournaments? Um? What prediction markets are? Forecasting tournaments are
Speaker 1: excellent ways of allowing people to tract their accuracy on
Speaker 1: judgment calls for which there aren't ready financial market equivalence.
Speaker 1: How do these I've never I'm not familiar with these,
Speaker 1: So how does they are forecasting tournament work? You ask
Speaker 1: people to put subjective probabilities on events that are specified
Speaker 1: by well defined questions, such as whether Prutent is going
Speaker 1: to be the president of Russia after and those probability
Speaker 1: either in that in that case, it would be a
Speaker 1: probability that would be yes or no, and it should
Speaker 1: sum up to one point zero. There are lots of
Speaker 1: ways of doing it, but they all boiled down to
Speaker 1: the core idea, which is which is keeping score. Sticking
Speaker 1: with the intelligence community framework, how does the role of
Speaker 1: group think play into this and avoiding group think? Because
Speaker 1: if we think back to what are considered a lot
Speaker 1: of the intelligence disasters over the last several years, the
Speaker 1: idea of some idea takes hold and no one feels
Speaker 1: comfortable yelling stop, and suddenly everyone can deals on the
Speaker 1: same idea. Is that something that in your work you
Speaker 1: focus on sort of like these cascades where someone puts
Speaker 1: forth an idea and everyone feels compelled to fall online,
Speaker 1: or there's extreme pressure to voice a concern or voice
Speaker 1: skepticism on things. And are there are ways or strategies
Speaker 1: that aspiring forecasters can use to eliminate or reduce that bias.
Speaker 1: Group think is a big problem, and that's why in
Speaker 1: forecasting tournaments we typically have people make judgments independently of
Speaker 1: each other, um at least initially. It doesn't mean that
Speaker 1: all team or group decisions are going to be bad ones,
Speaker 1: But it does mean that forecasters need to value accuracy
Speaker 1: above all. If your primary goal is pleasing your boss,
Speaker 1: and you have an opinionated boss, it's going to be
Speaker 1: very hard for you to offer that boss an opinion
Speaker 1: with a probability judgment that points to a policy different
Speaker 1: from what the boss prefers. So it it's another version
Speaker 1: of the question what business are you in? Are you
Speaker 1: in the entertainment business, are you in the pleasing your
Speaker 1: boss business? Or you in the accuracy business. Accuracy business
Speaker 1: is often not the first business people are in, talking,
Speaker 1: not even the second business people are in. People are
Speaker 1: trying to have successful careers, they're trying to avoid embarrassment.
Speaker 1: There's a lot of other things people are doing a
Speaker 1: scie from accuracy. But so forecasting tournaments create a really
Speaker 1: weird social environment. They create a world that's only one
Speaker 1: thing matters, and that is minimizing the gaps between your
Speaker 1: probability judgments and reality over the long term. And that's it.
Speaker 1: That's that's that's that's the sole objective I'm still trying
Speaker 1: to understand. So, as you point out, like forecasting tournaments
Speaker 1: are very weird because in the real world, that's not
Speaker 1: how predictions are made, and people are aware of what
Speaker 1: other people are thinking and talking about. So when you
Speaker 1: consult and when you talk with people, how do you
Speaker 1: foster a culture? And I guess this is really what
Speaker 1: matters to the end consumers of your research, is how
Speaker 1: do you foster a culture where more people feel that
Speaker 1: they're in the accuracy business. It really helps if it
Speaker 1: comes from the top. People are looking at your typically
Speaker 1: look up for the normative cues about what's appropriate. So
Speaker 1: if you have a boss who's open to being wrong,
Speaker 1: that that helps a lot. So, Phil, I know the
Speaker 1: majority of your work has to do with statistical analysis
Speaker 1: of probabilities of forecasts, but I'm curious, could you give
Speaker 1: us a sort of case study that you've come across
Speaker 1: of a forecast that has gone very, very wrong and
Speaker 1: that sort of brings together some of the themes or
Speaker 1: lessons that you've been talking about. I can go back
Speaker 1: to the very beginning and when I was doing this
Speaker 1: work and everybody thought we were at a major inflection point,
Speaker 1: virtually everybody, and they were more or less right about it.
Speaker 1: I mean, the most inflection point calls are wrong, but
Speaker 1: they were right that the Soviet Union At the time
Speaker 1: I was starting off on this work was at an
Speaker 1: inflection point. People didn't have any idea where the Soviet
Speaker 1: Union was going to go. The Conservatives thought that the
Speaker 1: Soviet Union was incapable of reforming itself. The liberals thought
Speaker 1: that Reagan was driving the Soviet Union into nio Stalinist
Speaker 1: retrenchments would become more aggressive. Yet Gorbachev came along in
Speaker 1: March of five. He became the General Party Secretary, and
Speaker 1: he proceeded, with Glasgow's embarrasster to liberalize the Soviet Union
Speaker 1: in ways that we're really astonishing now. After the fact,
Speaker 1: the Conservatives said, hey, we forced them to do it,
Speaker 1: but they didn't really expect the Soviet Union was capable
Speaker 1: of reforming itself beforehand, and the Liberals said, well, we knew,
Speaker 1: we knew it all along because the Soviet economy was
Speaker 1: crumbling and the Soviets were needed to do this, and
Speaker 1: Reagan had no role in it at all. So the
Speaker 1: paradox was that nobody was really very close at all
Speaker 1: to predicting what would happen, but everybody after the fact
Speaker 1: had a confident explanation for what would happen. Is there
Speaker 1: anything that people could have done better prior to the
Speaker 1: events unfolding that could have made their forecast better, or
Speaker 1: is it the kind of thing it was so novel
Speaker 1: and you know, it's kind of so unexpected that this
Speaker 1: would just be a really hard thing to forecast in
Speaker 1: any meaningful sense. It was a hard thing to forecast.
Speaker 1: But there were clues that Gorbachev was different, and even
Speaker 1: a conservative like Margaret Thatcher was signaling that based on
Speaker 1: our early meetings with Gorbachev, I think that the key
Speaker 1: factor here is how fast are you willing to change
Speaker 1: your mind in response to the the incoming evidence. So after
Speaker 1: Gorbachev became General Secretary in early there were there were
Speaker 1: lots of little bits of news that suggested this was
Speaker 1: going to be a different style of leadership, and maybe
Speaker 1: not just a different style of leadership, a different substance
Speaker 1: different that a different substance of policies would be pursued,
Speaker 1: and it would be it was your willingness, I think,
Speaker 1: to make small, rapid adjustments in response to the news.
Speaker 1: It wasn't and there was any one big item that
Speaker 1: absolutely turned the case, but there were lots of little bit,
Speaker 1: little bits of news that created over time that good,
Speaker 1: good forecasters could it could attend to. And I think
Speaker 1: that's one of the finding features of the best forecasters
Speaker 1: is that they're more granular. They make distinctions among more
Speaker 1: degrees of maybe than normal people do. An old joke
Speaker 1: in my field there's people can can really only distinguish
Speaker 1: three degrees of uncertainty, yes, now, and maybe the best
Speaker 1: forecasters are people who know the difference between a forty
Speaker 1: sixty bed and a sixty forty bed or even so
Speaker 1: it's some someone analog I said, the would be no doubt.
Speaker 1: For example, if I said, you know, good poker players
Speaker 1: could do that, and you say, well, sure, they must
Speaker 1: be able to do that in a repeated play game.
Speaker 1: We get rapid quantitative feedback. But I said, well, good
Speaker 1: geopolitical and economic forecasters also do that, and you say, well,
Speaker 1: it cannot really be possible, and and answers, yes it can.
Speaker 1: So do you have a favorite forecaster? You know, it's
Speaker 1: like asking who are my favorite children? Right, I know
Speaker 1: I'm not I would I wouldn't take any particular person
Speaker 1: as a favorite forecasting but I think there are lots
Speaker 1: of very admirable people out there. Well, in the beginning,
Speaker 1: you mentioned Nate Silver and his prediction or not prediction,
Speaker 1: but his assessment maybe a better way to put it,
Speaker 1: that Hillary had a sevent chance of winning the election.
Speaker 1: He's someone who has a very sort of clear understanding
Speaker 1: of probabilities and he puts h you know, he has
Speaker 1: a difference between eighty five and seventy and fifty and
Speaker 1: twenty and probably fits very well into your database. By
Speaker 1: and large, do his seventy forecasts proved to be right
Speaker 1: about seven out of ten times. I have not analyzed
Speaker 1: Nate's data, but Nate does have data, and he analyzes
Speaker 1: it himself, and he has reported how well calibrated his
Speaker 1: boy is the forecast on the five. I tend to
Speaker 1: be in both sports and political forecasts, and I think
Speaker 1: they're pretty good, uh, in the sense of being well calibrated.
Speaker 1: I don't know the data on resolution, but on calibration,
Speaker 1: they're scoring pretty well. There are two key fassets of
Speaker 1: being a good forecaster when you're doing subjective probability scoring.
Speaker 1: One of them is what I mentioned earlier, calibration. So
Speaker 1: when you say seventy percent likely to seventy percent, things
Speaker 1: happen seventy percent of the time. The other is resolution. Now,
Speaker 1: so there's there's a sneaky and lazy way of being
Speaker 1: well calibrated. So if you're a weather forecaster in Seattle
Speaker 1: and it rains sixty percent at the time, and you say, hey,
Speaker 1: I'm just gonna say there's sixty percent likelihood of rain
Speaker 1: every day, and you know what, I'm going to be
Speaker 1: well calibrated because rain will happen sixty percent of the time.
Speaker 1: You you would be you'd be well calibrated, but you'd
Speaker 1: be very uninteresting. So there's another property you need to
Speaker 1: ask a forecasters beyond calibration, and that is you need
Speaker 1: to ask them, are you good at assigning much higher
Speaker 1: probabilities to things that happen than to things that don't.
Speaker 1: Are you good at being justifiably decisive? So you want
Speaker 1: forecasters who are two things that you want there appropriately humble,
Speaker 1: which means well calibrated, but they're also justifiably decisive. They
Speaker 1: say interesting, decisive things when they have a warrant for
Speaker 1: saying those things, so that it's a combination of those
Speaker 1: two things that makes some most a so called super
Speaker 1: forecaster in our work. And um, but I think that
Speaker 1: the Nate Silver group at five three is doing doing
Speaker 1: the right things and I think a number of other
Speaker 1: organizations are starting to do the right things as well.
Speaker 1: So on that note, how should forecasters deal with tail
Speaker 1: risk events? Because of course, as you as you put it,
Speaker 1: you know, you could just sort of do an average
Speaker 1: of probabilities and you might look very smart and very
Speaker 1: well calibrated. But at some point there is a chance
Speaker 1: that a big unexpected event is going to come out
Speaker 1: of nowhere and sort of shift the entire regime of
Speaker 1: statistics in some way. How should forecasters deal with those
Speaker 1: kind of unforeseen risks? I think you want to think
Speaker 1: in terms of shades of gray rather than black and white.
Speaker 1: It's not that things are there, there are some things
Speaker 1: are foreseeable and other things are unforeseeable. That there are
Speaker 1: black swans and then there are white swans. There are
Speaker 1: swans of varying degrees of grayness. And the best forecasters,
Speaker 1: I think, recognize that there is a continuum um and
Speaker 1: that tail risk is a problem, and you have to
Speaker 1: judge how important it is. You will never miss a war,
Speaker 1: or you'll never miss a disaster if you always predict disaster,
Speaker 1: but that will that the cost you're you're paying in
Speaker 1: false positives is ridiculous. The question is how high a
Speaker 1: price are you willing to pay for making lots of
Speaker 1: false positive predictions about you know, the DATO is going
Speaker 1: to fall below two thousand in the next six months,
Speaker 1: that sort of thing, and and and and you know
Speaker 1: by by by futures contracts based on that belief. Skin
Speaker 1: in the game, as it were. Those are judgment calls,
Speaker 1: and you never escape making probability judgments, even though the
Speaker 1: probabilities may be extremely small. Is very difficult to say
Speaker 1: whether someone is well calibrated and distinguishing between events that
Speaker 1: are one and ten thousand likely and one on a
Speaker 1: chillion likely. Right, But there's a huge difference between one
Speaker 1: and ten thousand and one and the chillion right. So, Phil,
Speaker 1: you mentioned you know, people predicting war or natural disasters.
Speaker 1: It does feel sometimes like the people who forecast negative events,
Speaker 1: you know, recession is coming, war is coming, Donald Trump
Speaker 1: is up ending the global order, those sorts of things
Speaker 1: that they seem to make more waves or more in
Speaker 1: roads than people who predict either a status quo or
Speaker 1: positive trends. Do you think people like to hear dire
Speaker 1: forecasts more than extremely optimistic forecasts. I don't know. If
Speaker 1: they like to hear them more, but they seem to
Speaker 1: find them more interesting and they pay more attention to them.
Speaker 1: Can the best forecasters, or the super forecasters as you
Speaker 1: call them, can they always articulate their approach or do
Speaker 1: some people who just have some sort of deeper intuitive sense.
Speaker 1: And I'm thinking about you use the poker analogy. And
Speaker 1: one of the things about poker is that there's different
Speaker 1: ways to play it. So some people are extremely mathematical
Speaker 1: in their forecast, they calculate everything. Others seem too much
Speaker 1: more clearly operate on feel, and they just have a
Speaker 1: good feel for whatever reason that turns out to be
Speaker 1: a successful strategy for them too. Is there arrange in
Speaker 1: the approaches that people use? Or some people can very
Speaker 1: methodically lay out their approach like a date silver where
Speaker 1: they build all these models versus a more intuitive fuel
Speaker 1: based approach that maybe can't be written down as well. Yeah,
Speaker 1: a long time ago, Malcolm Bladder wrote a book I
Speaker 1: don't know if you remember it called Blink. And there
Speaker 1: are some people in my fields who wrote a much
Speaker 1: less well known book. So a rejoinder called think You've
Speaker 1: Got a duel? Isn't here between people who endorse blink
Speaker 1: and people who endorse think I lean towards the think end.
Speaker 1: I'm not precluding the possibility that when you're dealing with
Speaker 1: events that occur over and over again, and there's lots
Speaker 1: of repeated play, and there's lots of opportunity to build
Speaker 1: up deep experience and automated cognitive processing, that some people
Speaker 1: can become intuitively very good at it. Uh And they
Speaker 1: may even be doing rather complex calculations in their head
Speaker 1: very rapidly. So it's not that the intuition is is
Speaker 1: eesp here. It could it could be that some of
Speaker 1: the best forecasters are not have simply overlearned the probability
Speaker 1: calculation heuristics to the point where they got they unfold
Speaker 1: virtually automatically. This isn't like a master pianist, right, It
Speaker 1: doesn't have to think about every key and just you know,
Speaker 1: the great tennis player doesn't have to think about where
Speaker 1: his arm is. That sort of thing. So, Phil, I
Speaker 1: feel like I have to ask, will you give us
Speaker 1: a forecast? Okay, I'll give you a forecast, even though
Speaker 1: I'm not a forecast, just just for you, just for you,
Speaker 1: I will, I will, I will give you a forecast
Speaker 1: and that is I don't think the forecast and forecasting
Speaker 1: practices are going to change very fast. I think that
Speaker 1: the majority of people are making forecasts because they don't
Speaker 1: want to offend people in power, They don't want to
Speaker 1: offend clients because they want to be entertaining in media
Speaker 1: performances or elsewhere. Uh. And the accuracy will continue to
Speaker 1: be a very, very secondary goal, but that gradually over
Speaker 1: the next ten twenty, through years of forecasting tournaments and
Speaker 1: prediction markets, will become an increasingly common way for people
Speaker 1: to resolve certain categories of disagreements. Well, hopefully your your
Speaker 1: appearance on this podcast will help marginally change the trajectory
Speaker 1: of the forecasting industry over the next several decades. But
Speaker 1: really appreciate you coming out a fascinating, fascinating discussion. Thanks
Speaker 1: so much. Okay, take care. So Joe, I'm just going
Speaker 1: to point out that Phil's forecast did not contain a
Speaker 1: probability oh huh. Even though he did a pretty uncontroversial forecast,
Speaker 1: then he didn't put a precise number on it. But
Speaker 1: I really, I really like that conversation. I think it's
Speaker 1: just a great topic because we see this all the
Speaker 1: time and not just in the fact that people come
Speaker 1: on give forecasts and never are really held accountable for them,
Speaker 1: but just in what is the purpose of forecast? And
Speaker 1: how many times we all encounter people who give forecasts
Speaker 1: but whose job is clearly not accuracy, and I'm thinking
Speaker 1: a lot of their I think there are a lot
Speaker 1: of asset manager is like this who use stories as
Speaker 1: a way of gathering clients, and maybe they tell a
Speaker 1: bearished story or a conspiratorial story about the fed or whatever,
Speaker 1: but that story really has nothing to do with how
Speaker 1: they then go on and invest. Right, certainly in the
Speaker 1: investment industry, there are plenty of people whose positions just
Speaker 1: don't add up to the world viewpoint that they tend
Speaker 1: to express, which you know, again, I would think of
Speaker 1: a lot of the really bearished people out there who
Speaker 1: for the past eight years have been saying, you know,
Speaker 1: move into cash by gold, the end is coming, and
Speaker 1: yet clearly they are in the business of putting cash
Speaker 1: to work in some way or another. So you know,
Speaker 1: it seems a little bit out of sync. You know,
Speaker 1: we didn't you know, we only talked a little bit.
Speaker 1: And I'm sure if we read his work, uh and
Speaker 1: studied it, there'll be more depth, but in terms of
Speaker 1: becoming a better forecaster, this idea of just sort of
Speaker 1: looking at the the the default and what point is well,
Speaker 1: coups are pretty rare, and wars are pretty rare, and
Speaker 1: regime change is pretty rare, and sort of starting there,
Speaker 1: I mean, another thing in stock markets is like bear
Speaker 1: markets are really rare, and stock market crashes are really rare.
Speaker 1: And so this sort of idea of like, as you
Speaker 1: put in the beginning, and as we're talking about we're
Speaker 1: at a moment in which lots of people are talking
Speaker 1: about inflection points and it feels like things might turn.
Speaker 1: But just starting from this assumption that we're probably not
Speaker 1: an inflection point and that most of these turns that
Speaker 1: we think are they're bound to happen now probably won't
Speaker 1: happen now, it's like it's an interesting starting point. I mean,
Speaker 1: obviously that's not enough, because sometimes the turns do happen,
Speaker 1: but starting from that assumption, it's like that you're probably
Speaker 1: the right move is to fade the expectations of a turn.
Speaker 1: See it feels like an interesting toll hole to get
Speaker 1: into two then move from there, right, But see, I
Speaker 1: think this is where human psychology and references come in,
Speaker 1: because no one is going to remember or reward you
Speaker 1: for continuously calling the status quo correctly, but they will
Speaker 1: probably remember you if you did call the big regime
Speaker 1: shift or the big bear market. And that's why so
Speaker 1: many people remember quite a few names that predicted the
Speaker 1: two thousand and eight financial crisis, but no one remembers
Speaker 1: as clearly, you know, people who called the gigantic rally
Speaker 1: that we had after two thousand nine the only you know,
Speaker 1: the only one, and like an exception to someone like
Speaker 1: Warren Buffett who just buy stocks and doesn't do anything
Speaker 1: fancy and has done so there's no seriously, like there's
Speaker 1: a few people like that. I feel like Warren Buffett
Speaker 1: is always the exception. Yeah, that's true. Like you can't
Speaker 1: just point to Buffett, but that's true. All right. Well,
Speaker 1: on that note, h this has been another episode of
Speaker 1: the ad Thoughts podcast. I'm Tracy Allaway. You can follow
Speaker 1: me on Twitter at Tracy Alloway and I'm Joe Why
Speaker 1: Isn't Though? You can follow me on Twitter at the Stalwart.
Speaker 1: You should follow our guest on Twitter, Phillip Tetlock. He's
Speaker 1: at p Tetlock, and you should follow our producer on Twitter,
Speaker 1: Laura Carlson. She's at Laura M. Carlson, as well as
Speaker 1: the Bloomberg head of podcasts, Francesca Levi at Francesca Today.
Speaker 1: And be sure to check out all of Bloomberg's podcasts
Speaker 1: on Twitter under the handle at podcasts. Thanks for listening.