Speaker 1: Hello, and welcome to Stephanomics, the podcast that brings the
Speaker 1: global economy to you. And this week we have a
Speaker 1: treat in the form of a conversation with Daniel Kannerman,
Speaker 1: the most influential and respected psychologist in the world. He's
Speaker 1: one of a small number of non economists to have
Speaker 1: won the Nobel Prize for Economics for his contribution to
Speaker 1: the school of behavioral economics. With his collaborator Amos Verski,
Speaker 1: he also wrote what I'm sure is the best selling
Speaker 1: psychology book of all time, Thinking Fast and Slow, along
Speaker 1: with Olivier Siboni and cass Sunstein. He has a new
Speaker 1: book just coming out in paperback, Noise The Flaw in
Speaker 1: Human Judgment. Olivier is a professor of Strategy and Business
Speaker 1: Policy at h GC Paris, and I'm pleased to say
Speaker 1: he joined us for the interview. But I started by
Speaker 1: asking Kannerman to tell us what this great flaw in
Speaker 1: human judgment was that he caused noise? Define uh, what
Speaker 1: noises in relation to its cousin more familiar cousin, which
Speaker 1: is biased? Uh. And we're talk in general about errors
Speaker 1: of judgment, and the context when we talk about errors
Speaker 1: of judgment is to compare judgment to measurement. That's where
Speaker 1: the whole notions of accuracy and accuracy of measurement and error.
Speaker 1: The regulous treatment of errors stems so com from the
Speaker 1: comparison of judgment to measurement. Now, in measurement, when you're
Speaker 1: measuring the same object multiple times with the very fine rulers,
Speaker 1: we're not going to get the same result every time.
Speaker 1: That is, if the ruler is fine enough, there's going
Speaker 1: to be variability. That variability, that's the variability that we
Speaker 1: call noise. So the errors some areas are positive of
Speaker 1: the errors are negative, and the variability of the air
Speaker 1: a noise. The average error is a bias, so that
Speaker 1: you can have positive bias or negative bias. And it
Speaker 1: turns out that in the discussion of errors of judgment
Speaker 1: we have focused on systematic errors on biases, and very
Speaker 1: little attention has been paid in the discussion of error
Speaker 1: and two noise to variability. But in the theory of measurements,
Speaker 1: bias and noise actually have equivalent weight, and there is
Speaker 1: reason to believe that more inaccuracy and judgment is often
Speaker 1: due to noise than to bias. And this is you
Speaker 1: know where the book came to be. And I guess
Speaker 1: it's worth saying. We're used to thinking of the bias
Speaker 1: that you'll be systematically leaning in one direction or another
Speaker 1: in the directions that we in the decisions that we take.
Speaker 1: But I guess the point of the noise is that
Speaker 1: it's not predictable. And I guess crucially we fail to
Speaker 1: understand not just that there is noise, but the stint
Speaker 1: of the noise. We ask experts something and we expect
Speaker 1: there to be only a relatively small variation in their decisions,
Speaker 1: and actually there's a huge variation. So I guess it's
Speaker 1: it's worth talking through one of those examples just to
Speaker 1: give a sense of what you're talking about. I can
Speaker 1: describe the example from what the study began, and this
Speaker 1: It began about eight years ago when I was doing
Speaker 1: some consulting in an insurance company and I conducted a
Speaker 1: fairly between experiment that today we would call a noise audit,
Speaker 1: where cases were constructed which were very common representative of
Speaker 1: the work of underwriters in that company, and then the
Speaker 1: same cases were presented to several dozen underwriters about fifty
Speaker 1: years I recall, and we looked at the variability they
Speaker 1: looked at. They put a dollar value on those cases. Now,
Speaker 1: notice those cases worth were fictions. They were constructed as such,
Speaker 1: but they were very typical. And the idea was that
Speaker 1: if underwriters vary in their judgments of those hypothetical cases,
Speaker 1: they would also vary in their judgments of real case. Now,
Speaker 1: I ask executives in the company a question that I
Speaker 1: think anybody was listening to this would also ask themselves.
Speaker 1: If you look at two underwriters and you should pick
Speaker 1: that random and our large the difference do you expect
Speaker 1: to find between them in percentages? That is, you take
Speaker 1: the two underwriters, you compute their average of their judgments
Speaker 1: the difference of their judgrens. To divide the difference by
Speaker 1: the average, what percentage do you expect most people or
Speaker 1: many people? There is a really common answer to that question.
Speaker 1: People expect about them pass and this was also true
Speaker 1: of the executives in that company. We don't expect judgments
Speaker 1: to be perfectly, but we expect them not to disagree wildly. Now,
Speaker 1: the real number that we observed in the experiment was
Speaker 1: about fifty five zero five times larger than expected, and
Speaker 1: that's really the origin of the book. So it looked
Speaker 1: worth studying, not only because there was a lot of noise,
Speaker 1: but because the noise came as complete news to the organization.
Speaker 1: They were unaware that they had a noise problem. So
Speaker 1: we started with slogan, which is and it turned out
Speaker 1: that there's a lot of noise everywhere, that wherever there
Speaker 1: is judgment, there is noise, and there is more of
Speaker 1: it than you think, and that is really the motivation
Speaker 1: for the book. Olivia joined me very soon and they
Speaker 1: started working in the book together. Cast joined us later. Um,
Speaker 1: that's the story in a year ago the book. It
Speaker 1: is interesting. You know, obviously you're from different fields and Olivia,
Speaker 1: I'm interested in you are drawn to this because obviously
Speaker 1: it has very clear relevance for business strategy and the
Speaker 1: way companies think about sort of what it is they're doing. Well.
Speaker 1: As Danny has just described it, noise is unwanted variability
Speaker 1: in judgments. And this only becomes a problem when you're
Speaker 1: an organization. Noise is a disease of organizations. If you
Speaker 1: are an individual, we will never know how noisy you are.
Speaker 1: You are noisy, by the way, and you are you
Speaker 1: are subject to the same sources of noise that we
Speaker 1: are all subject to it organizations. But where we expect
Speaker 1: consistency is when people in an organization are making judgments
Speaker 1: on behalf of the organization, as in the example of
Speaker 1: the underwriters that Danny was just talking about, and when
Speaker 1: we expect those judgments to be reasonably consistent. If you
Speaker 1: look at another example we've looked at, which is the
Speaker 1: judicial system, we expect that the decisions that judges renders
Speaker 1: should not be too dependent on the identity of the judge.
Speaker 1: Of course, again we expect some variability, but we expect
Speaker 1: general consistency. And the challenge for organizations of any kind
Speaker 1: is to actually achieve something approaching consistency, because first they
Speaker 1: need to realize how much inconsistency they have. They need
Speaker 1: to realize how much noise there is, and as then
Speaker 1: he pointed out, they're not aware of that. It comes
Speaker 1: as a complete surprise when they realize that. So it's
Speaker 1: a huge organizational problem for private enterprises, but also for administrations,
Speaker 1: for government, for non government organizations, for any organization of
Speaker 1: any kind that has many people making judgments and that
Speaker 1: expects consistency. There's so many different strands of this. I
Speaker 1: think there's one which is a straightforward sort of natural
Speaker 1: justice perspective. But some of the examples in the book,
Speaker 1: there's some of the ones that perhaps were most familiar
Speaker 1: with is the variation in sentencing for the same case
Speaker 1: depending on whether the judges football team one or lost
Speaker 1: at the weekend or whatever. It maybe. But it also
Speaker 1: raises a question about what is the nature of expertise
Speaker 1: If we like to think that an experts are experts
Speaker 1: in part because they understand the body of knowledge and
Speaker 1: have a shared understanding of that the way the world
Speaker 1: works in that particular expertise, And what a lot of
Speaker 1: these examples suggests is that they're all experts in their
Speaker 1: own way, and they're all coming up with completely different conclusions.
Speaker 1: Does want come away from this thinking that experts are
Speaker 1: not necessarily helpful for organizations. I think you've come away
Speaker 1: thinking that there are really two different sorts of experts,
Speaker 1: and that we should be clearer in our articulation of
Speaker 1: that distinction. There are experts whose track records can actually
Speaker 1: be evaluated, whose expertise can be quantified measured against a
Speaker 1: gold stand earth. So if you're a forecaster and you
Speaker 1: make short term economic forecasts, and each quarter we can
Speaker 1: check how oft you were and it turns out that
Speaker 1: your forecast historically been very good, we can say you're
Speaker 1: a true expert as forecasting. Now, if you're making thirty
Speaker 1: year forecasts, how much of an expert you are does
Speaker 1: not depend on how good your forecasts are. It depends
Speaker 1: on how much respect we accord you as a forecaster.
Speaker 1: And those are the experts that we call respect experts
Speaker 1: because they're experts, not because they have demonstrable experts ese,
Speaker 1: but because they have convinced others of their expertise, because
Speaker 1: we have respect for their expertise. That is not a
Speaker 1: criticism of those experts, by the way, because in many fields,
Speaker 1: all you can be is a respect expert. Even the
Speaker 1: underwriters that Danny was talking about, we'll never know if
Speaker 1: they've actually set the right premium for an insurance policy.
Speaker 1: So we have respect for them because they are convincing,
Speaker 1: because they can are securely their reasoning in a compelling manner,
Speaker 1: because they have experience, because they have gained confidence in
Speaker 1: the way they do their job. But they are not
Speaker 1: the same sort of experts as the experts whose expertise
Speaker 1: can be demonstrated. And what we argue in Noise is
Speaker 1: that it's important to know what sort of experts you're
Speaker 1: dealing with. When you're dealing with experts, Danny kind of.
Speaker 1: And you talk about good decision hygiene as being the
Speaker 1: sort of equivalent of washing your hands so that you
Speaker 1: can have the limited infection from noise. What what does
Speaker 1: that look like for a for a policymaker or an organization. Well, well,
Speaker 1: whole notion of the hygiene is in contrast to common
Speaker 1: efforts of the biases trying to reduce various biases in
Speaker 1: the thinking of the judgment of organizations and individuals. Uh,
Speaker 1: the bias thing is very much like medication or vaccination.
Speaker 1: It's specific to a particular disease. Hygiene, like washing your hands,
Speaker 1: is non specific. But as you don't know what germs
Speaker 1: you're killing, if you're lucky, you'll never know. And uh,
Speaker 1: that's that's the nature of hygiene. And when you think
Speaker 1: about noise, the only way that we could think of
Speaker 1: improving judgment of reducing noise is by taking steps which
Speaker 1: are generally steps to improve the quality of judgment, but
Speaker 1: are not oriented to particular biases or two combat particular biases.
Speaker 1: What would be a good example of good hygiene where
Speaker 1: you could otherwise have a very noisy and unfair decisions.
Speaker 1: A standard example, and actually an example that was very
Speaker 1: influential on our thinking, is how to conduct hiring interviews.
Speaker 1: And there has been a lot of research on hiring
Speaker 1: interviews and they fall into two broad families. Unstructured interviews,
Speaker 1: that's the common procedure where you talk to the candidate,
Speaker 1: you try to form a general impression, You have a
Speaker 1: conversation with the candidate, there is some human contact, and
Speaker 1: at the end of the process you you make a
Speaker 1: decision or you form an impression of that candid. A
Speaker 1: structured interview is very different. In a structured interview, you
Speaker 1: have a list of topics that you want to think about.
Speaker 1: For example, you want to assess various attributes of the candidate,
Speaker 1: how original, how reliable, many attributes that may be relevant
Speaker 1: to a particular job. And in a structured interview, you
Speaker 1: think about each of these areas in turn and you
Speaker 1: conduct an interview. That is, you ask questions that pertain
Speaker 1: to that particular area, actually write down a grade or
Speaker 1: ranking or rating for that before switching to the next topic.
Speaker 1: So that's a structured interview. Now it turns out that
Speaker 1: neither kind of interview is very good because because basically
Speaker 1: performance in on jobs is very difficult to predict and
Speaker 1: it doesn't depend only on the characteristics of the individual.
Speaker 1: But structured interview are distinctly superior to unstructured interview. And
Speaker 1: so structuring is we think a good idea, and when
Speaker 1: you're making a decision and you're considering various options, you
Speaker 1: might want to consider the options as if there were
Speaker 1: candidates and assess the various attributes of an option. And
Speaker 1: the important feature here that you delay the global intuition.
Speaker 1: You delay the formation of the global impression. Intuition. One
Speaker 1: of the problems of intuitive thinking that it comes very fast.
Speaker 1: They form first impressions, and in unstructured interviews, typically an
Speaker 1: impression is formed very quickly and most of the rest
Speaker 1: of the conversation is to justify the national impression. In
Speaker 1: a structured interview, that's not the case. You deal with
Speaker 1: topics one at the time and you try to delay
Speaker 1: the global view of the candidate until all the information
Speaker 1: is So that's an example of decision. But I mean,
Speaker 1: one of the points about intuition is it's not very controllable.
Speaker 1: So I'm just wondering. I mean, you know, you and
Speaker 1: Olivier and and and behavioral economists may be very aware
Speaker 1: of all the biases and all the noise that you've
Speaker 1: just talked about. But when you're sitting interviewing a candidate
Speaker 1: or a potential colleague at university or whatever, how do
Speaker 1: you actually stop yourself from having a first impression of someone,
Speaker 1: Because by definition of first impression comes unbidden. Oh, you
Speaker 1: will undoubtedly form impresference, there is no question. But but
Speaker 1: if you have a set of questions that you want
Speaker 1: to ask about the person's reliability or about the extent
Speaker 1: of their experience on similar on a similar, unsimilar task,
Speaker 1: those specific questions are going to fail your mind, and
Speaker 1: they're going to push the intuition aside to some extent,
Speaker 1: and you will have an opportunity that you normally do
Speaker 1: not have of disconfirming your initial impression, of finding things
Speaker 1: out that actually do not fit the initial impression. In general,
Speaker 1: an unstructured interview, impressions are self reinforces. You justify your
Speaker 1: initial impression, and that is a source of noise, And
Speaker 1: by structuring the process you reduce that source of noise.
Speaker 1: An additional thing you can do to limit the problem
Speaker 1: that you're pointing out Stephanie is to have different people
Speaker 1: or different sources of information evaluate the different dimensions that
Speaker 1: you are looking at. So, if you're evaluating candidates for
Speaker 1: you know, intelligence, technical skills, and fit with the culture
Speaker 1: of the company, let's assume these are the three dimensions
Speaker 1: of your job description. In an unstructured interview, you would
Speaker 1: form an overall picture of the person and you would
Speaker 1: then raid them on the three dimensions, but they would
Speaker 1: be strongly correlated with each other because there would be
Speaker 1: a positive or a negative halo around the person, and
Speaker 1: you would say they are great and everything, or they're
Speaker 1: bad on everything. Now suppose that we say you, Stephanie,
Speaker 1: are going to conduct the interview about the technical skills,
Speaker 1: or maybe in fact, we're going to have a technical
Speaker 1: test to evaluate the technical skills. Someone else is going
Speaker 1: to evaluate the fit with the company, and someone else
Speaker 1: is going to evaluate how smart the person is, or again,
Speaker 1: perhaps we're going to have the test of how smart
Speaker 1: the person is. Now you've got three independent data points
Speaker 1: that do not influence each other, and you have a
Speaker 1: much more structured process to make your decision. You would
Speaker 1: have anounced this before, but of course, a lot of
Speaker 1: people entering the job market now find that they're at
Speaker 1: least the first couple of rounds, depending on how popular
Speaker 1: the job is is, and they're talking to a computer
Speaker 1: or they have there is a an AI element to
Speaker 1: their application process. We may not like it, we may
Speaker 1: think it's not true to our great sense of intuition
Speaker 1: about people. But from a fairness perspective and from a
Speaker 1: decision hygiene perspective, is that a better way to go?
Speaker 1: We need to be careful here, because there is an
Speaker 1: answer in principle, and there is an answer in practice.
Speaker 1: In principle, any form of structured decision making that reduces
Speaker 1: noise would in fact enhance the quality of the decisions.
Speaker 1: So if you have an algorithm making decisions, there is
Speaker 1: going to be less noise there. But of course the
Speaker 1: question is how good is the algorithm? How good are
Speaker 1: those AI systems that people sit in front of, And
Speaker 1: I'm sure there are good ones, but from the ones
Speaker 1: that I've seen personally in my admittedly limited experience, there
Speaker 1: isn't much evidence, and there isn't very good quality evidence
Speaker 1: that what these software packages are testing for is actually
Speaker 1: what you're looking for. It's actually quite hard for most
Speaker 1: companies to define what it is that they're looking for,
Speaker 1: and there is no evidence that I've seen that there's
Speaker 1: any correlation between when those software packages look for and
Speaker 1: job success is actually highly correlated. So in practice I'm
Speaker 1: quite skeptical about what I see in the market. In theory,
Speaker 1: I have to agree that it makes some sense, but
Speaker 1: my worry is that companies are using this mostly to
Speaker 1: save time and money, not to actually improve the quality
Speaker 1: of their decision. Here, I would hope that what is
Speaker 1: true in theory can be made true in practice. And
Speaker 1: one characteristics of algorithms and is that they're improvable. They're
Speaker 1: much more improvable than people are, and and they can
Speaker 1: be corrected by by data on quality. They can be
Speaker 1: made to predict more accurately. So this this is really
Speaker 1: an issue of the quality of constructing algorithms. And there
Speaker 1: are many algorithms that are of poor quality out there
Speaker 1: on the market, and there is a widespread suspicion of
Speaker 1: algorithms which makes us prone to reject them. But by
Speaker 1: and large, I think this is the future. In the future,
Speaker 1: there will be more and more of those algorithms, and
Speaker 1: their quality will be getting better and better every year
Speaker 1: because there will be data there will be feedback, and
Speaker 1: the feedback can be incorporated into an algorithm much more
Speaker 1: efficiently than it can be in the human judgment. So uh,
Speaker 1: here I join Olivia's skepticism about most of the algorithms
Speaker 1: that exist, but I really want to register and mode
Speaker 1: of optimism about the future of that kind of operation.
Speaker 1: There's one trend which is about eliminating the human element
Speaker 1: to some extent, or at least having it in a
Speaker 1: more regular form in an algorithm, a more consistent form structured.
Speaker 1: Of course, the other big trend in business strategy and
Speaker 1: conversations about companies is the move is encouragement of diversity
Speaker 1: and to encourage businesses in a sense to have a
Speaker 1: wider variety of humans doing the judgment. And I wonder
Speaker 1: whether that even goes against some of the things that
Speaker 1: you're talking about. You know, one of the ways that
Speaker 1: companies might have previously eliminated noise, not necessarily error, but noise,
Speaker 1: would have been having lots of identical people or making
Speaker 1: the decisions all of these white men sitting in their boards.
Speaker 1: If you now have a greatly much more diversity, you
Speaker 1: might be more true to the range of human experience,
Speaker 1: but you'll be getting a lot more noise. Well, that
Speaker 1: is certainly true, but in in principle, we we want
Speaker 1: to distinguish between the process of generating a judgment and
Speaker 1: the final judgment. In the process of generating a judgment,
Speaker 1: diversity is very welcome. That is, you want multiple points
Speaker 1: of view, you want people ptise to enter into the
Speaker 1: participate in the conversation. But when a final judgment is made,
Speaker 1: we want a process that reduces noises. So diversity is
Speaker 1: very useful, and you know it's it's you can think
Speaker 1: of that in terms of, say, witnesses to a crime.
Speaker 1: So you're better off if the witnesses are in different
Speaker 1: places and see the event from different perspectives. And you're
Speaker 1: certainly better off if if the witnesses don't talk to
Speaker 1: each other and they are independent of each other. And
Speaker 1: so thinking along those lines gives you an idea that
Speaker 1: you do want diversity, but you want also the kind
Speaker 1: of independence and the kind of goal directiveness that reduces
Speaker 1: noise in the final journey. Diversity in the outcome of
Speaker 1: these decisions, in the judgment that you produce in the end.
Speaker 1: It's good for some things, but for most it's not.
Speaker 1: When you when you go to the doctor and the
Speaker 1: doctor tells you, oh, you have is disease. And then
Speaker 1: you go to another doctor and he tells you you
Speaker 1: have that disease. You don't say, oh, that's wonderful, it's diversity.
Speaker 1: You say one of these two doctors is wrong, maybe both.
Speaker 1: So whenever we think that there is a correct answer,
Speaker 1: diversity in the outcome is not good. Maybe the way
Speaker 1: to get to the correct outcome is to harness the
Speaker 1: diversity of the perspectives or through multiple witnesses, and that
Speaker 1: is one of the remedies that you can have to
Speaker 1: reduce noise. But as an organization, what you're aiming for
Speaker 1: is not every person having their own opinion. It's any
Speaker 1: person having the best possible judgment. And we've talked about business,
Speaker 1: we've talked about justice or you know, sentencing and decisions
Speaker 1: within the criminal justice system, and an area that comes
Speaker 1: up a little bit in your book, but obviously it's
Speaker 1: kind of front and center of people's minds at the
Speaker 1: moment when we think of the decisions being taken around
Speaker 1: the war in in Ukraine. Is that in a critical
Speaker 1: moments of foreign policy or military strategy decisions, you know,
Speaker 1: you can't necessarily enlist a lot of people and listen
Speaker 1: to their structured answers on a set of questions in
Speaker 1: reaching your judgment about how to respond to Russia or
Speaker 1: how to how to respond to one of the sort
Speaker 1: of very pressing situations that can arise in foreign policy.
Speaker 1: So I just wonder whether you whether you'd reflected on that,
Speaker 1: you know, if you're Anthony Blink in the sector of state,
Speaker 1: or if you're President Biden, or or that matter, at
Speaker 1: a Russian general how what does decision hygiene look like
Speaker 1: in those kind of situations where there's inevitably going to
Speaker 1: be a limited number of people that you can call
Speaker 1: on and imperfect information hygiene is something that an individual
Speaker 1: can follow. That is, there are better and less good
Speaker 1: ways of individual judgments. You want to cover all the
Speaker 1: bases you are to think, You want to the extent
Speaker 1: possible to think of all possible consequences, you know, as
Speaker 1: salient example in the Ukraine War, is it looks unlikely
Speaker 1: that that people who started that war knew that Finland
Speaker 1: and Sweden would want to join NATO, because you know,
Speaker 1: after all, this was supposed to keep NATO away. So
Speaker 1: when I have the feeling that not all the bases
Speaker 1: were covered in making those critical, so there are all
Speaker 1: we can hope for is that we have people with
Speaker 1: a lot of experience, because it turns out that there
Speaker 1: is genuine intuitive experience that can develop over time with
Speaker 1: institution within certain kinds of decisions and choices. And we
Speaker 1: also want people who, even under sometime pressure, UH, can
Speaker 1: follow the basic dictates of decision hype. I've noticed a
Speaker 1: professor kind of and that a lot of interviews with
Speaker 1: you are quite long, a long than the sort of
Speaker 1: the norm for whatever program it is, And I suspect
Speaker 1: it's because it's so it's so it's always so fascinating
Speaker 1: to listen to you, and you always want to have
Speaker 1: another question. But we're going to run out of time.
Speaker 1: So thank you very much for coming on Stephonomics, and
Speaker 1: thanks thank you to Olivier Simbony, thank you, thank you
Speaker 1: very much. That's it for this episode of Stephonomics. We'll
Speaker 1: be back next week. In the meantime, do please rate
Speaker 1: the show if you like it, and check out the
Speaker 1: Bloomberg Terminal and News website for more economic news and
Speaker 1: views on the global economy. You can also follow our
Speaker 1: economics on Twitter. This episode was produced by Magnus Henrickson
Speaker 1: and Summer Said, with special thanks to Professor Daniel Kannerman
Speaker 1: and Olivier Sibon. Mike Sasso is executive producer of Stephonomics
Speaker 1: and the head of Bloomberg Podcast is Francesca Levi. The