1
00:00:05,200 --> 00:00:07,880
Speaker 1: Hello, and welcome to Stephanomics, the podcast that brings the

2
00:00:07,880 --> 00:00:10,240
Speaker 1: global economy to you. And this week we have a

3
00:00:10,320 --> 00:00:13,520
Speaker 1: treat in the form of a conversation with Daniel Kannerman,

4
00:00:13,840 --> 00:00:17,840
Speaker 1: the most influential and respected psychologist in the world. He's

5
00:00:17,840 --> 00:00:20,119
Speaker 1: one of a small number of non economists to have

6
00:00:20,160 --> 00:00:23,200
Speaker 1: won the Nobel Prize for Economics for his contribution to

7
00:00:23,239 --> 00:00:26,960
Speaker 1: the school of behavioral economics. With his collaborator Amos Verski,

8
00:00:27,560 --> 00:00:30,360
Speaker 1: he also wrote what I'm sure is the best selling

9
00:00:30,360 --> 00:00:34,320
Speaker 1: psychology book of all time, Thinking Fast and Slow, along

10
00:00:34,320 --> 00:00:37,199
Speaker 1: with Olivier Siboni and cass Sunstein. He has a new

11
00:00:37,200 --> 00:00:40,440
Speaker 1: book just coming out in paperback, Noise The Flaw in

12
00:00:40,560 --> 00:00:44,360
Speaker 1: Human Judgment. Olivier is a professor of Strategy and Business

13
00:00:44,360 --> 00:00:46,960
Speaker 1: Policy at h GC Paris, and I'm pleased to say

14
00:00:47,000 --> 00:00:49,479
Speaker 1: he joined us for the interview. But I started by

15
00:00:49,479 --> 00:00:52,280
Speaker 1: asking Kannerman to tell us what this great flaw in

16
00:00:52,400 --> 00:01:02,680
Speaker 1: human judgment was that he caused noise? Define uh, what

17
00:01:02,840 --> 00:01:07,520
Speaker 1: noises in relation to its cousin more familiar cousin, which

18
00:01:07,600 --> 00:01:12,039
Speaker 1: is biased? Uh. And we're talk in general about errors

19
00:01:12,040 --> 00:01:15,600
Speaker 1: of judgment, and the context when we talk about errors

20
00:01:15,640 --> 00:01:20,119
Speaker 1: of judgment is to compare judgment to measurement. That's where

21
00:01:21,000 --> 00:01:25,880
Speaker 1: the whole notions of accuracy and accuracy of measurement and error.

22
00:01:26,920 --> 00:01:31,520
Speaker 1: The regulous treatment of errors stems so com from the

23
00:01:31,600 --> 00:01:35,560
Speaker 1: comparison of judgment to measurement. Now, in measurement, when you're

24
00:01:35,560 --> 00:01:41,160
Speaker 1: measuring the same object multiple times with the very fine rulers,

25
00:01:41,480 --> 00:01:43,880
Speaker 1: we're not going to get the same result every time.

26
00:01:44,319 --> 00:01:47,200
Speaker 1: That is, if the ruler is fine enough, there's going

27
00:01:47,240 --> 00:01:52,360
Speaker 1: to be variability. That variability, that's the variability that we

28
00:01:52,440 --> 00:01:56,400
Speaker 1: call noise. So the errors some areas are positive of

29
00:01:56,480 --> 00:01:59,680
Speaker 1: the errors are negative, and the variability of the air

30
00:02:00,000 --> 00:02:04,840
Speaker 1: a noise. The average error is a bias, so that

31
00:02:04,920 --> 00:02:08,520
Speaker 1: you can have positive bias or negative bias. And it

32
00:02:08,639 --> 00:02:11,679
Speaker 1: turns out that in the discussion of errors of judgment

33
00:02:12,320 --> 00:02:17,280
Speaker 1: we have focused on systematic errors on biases, and very

34
00:02:17,360 --> 00:02:20,360
Speaker 1: little attention has been paid in the discussion of error

35
00:02:20,840 --> 00:02:25,480
Speaker 1: and two noise to variability. But in the theory of measurements,

36
00:02:25,960 --> 00:02:30,320
Speaker 1: bias and noise actually have equivalent weight, and there is

37
00:02:30,440 --> 00:02:35,360
Speaker 1: reason to believe that more inaccuracy and judgment is often

38
00:02:35,480 --> 00:02:39,080
Speaker 1: due to noise than to bias. And this is you

39
00:02:39,120 --> 00:02:41,280
Speaker 1: know where the book came to be. And I guess

40
00:02:41,320 --> 00:02:43,720
Speaker 1: it's worth saying. We're used to thinking of the bias

41
00:02:43,800 --> 00:02:47,320
Speaker 1: that you'll be systematically leaning in one direction or another

42
00:02:47,400 --> 00:02:50,240
Speaker 1: in the directions that we in the decisions that we take.

43
00:02:50,560 --> 00:02:53,320
Speaker 1: But I guess the point of the noise is that

44
00:02:53,440 --> 00:02:57,080
Speaker 1: it's not predictable. And I guess crucially we fail to

45
00:02:57,200 --> 00:03:00,360
Speaker 1: understand not just that there is noise, but the stint

46
00:03:00,440 --> 00:03:03,079
Speaker 1: of the noise. We ask experts something and we expect

47
00:03:03,160 --> 00:03:07,600
Speaker 1: there to be only a relatively small variation in their decisions,

48
00:03:08,120 --> 00:03:10,520
Speaker 1: and actually there's a huge variation. So I guess it's

49
00:03:10,560 --> 00:03:13,360
Speaker 1: it's worth talking through one of those examples just to

50
00:03:13,360 --> 00:03:15,600
Speaker 1: give a sense of what you're talking about. I can

51
00:03:15,840 --> 00:03:20,120
Speaker 1: describe the example from what the study began, and this

52
00:03:20,560 --> 00:03:23,200
Speaker 1: It began about eight years ago when I was doing

53
00:03:23,240 --> 00:03:29,040
Speaker 1: some consulting in an insurance company and I conducted a

54
00:03:29,040 --> 00:03:32,720
Speaker 1: fairly between experiment that today we would call a noise audit,

55
00:03:33,320 --> 00:03:40,520
Speaker 1: where cases were constructed which were very common representative of

56
00:03:40,640 --> 00:03:44,200
Speaker 1: the work of underwriters in that company, and then the

57
00:03:44,280 --> 00:03:48,400
Speaker 1: same cases were presented to several dozen underwriters about fifty

58
00:03:48,440 --> 00:03:52,920
Speaker 1: years I recall, and we looked at the variability they

59
00:03:53,000 --> 00:03:56,520
Speaker 1: looked at. They put a dollar value on those cases. Now,

60
00:03:56,560 --> 00:04:01,720
Speaker 1: notice those cases worth were fictions. They were constructed as such,

61
00:04:02,040 --> 00:04:05,040
Speaker 1: but they were very typical. And the idea was that

62
00:04:05,080 --> 00:04:10,160
Speaker 1: if underwriters vary in their judgments of those hypothetical cases,

63
00:04:10,440 --> 00:04:14,160
Speaker 1: they would also vary in their judgments of real case. Now,

64
00:04:14,480 --> 00:04:17,800
Speaker 1: I ask executives in the company a question that I

65
00:04:17,880 --> 00:04:21,800
Speaker 1: think anybody was listening to this would also ask themselves.

66
00:04:22,080 --> 00:04:26,240
Speaker 1: If you look at two underwriters and you should pick

67
00:04:26,320 --> 00:04:30,600
Speaker 1: that random and our large the difference do you expect

68
00:04:30,600 --> 00:04:35,279
Speaker 1: to find between them in percentages? That is, you take

69
00:04:35,320 --> 00:04:38,279
Speaker 1: the two underwriters, you compute their average of their judgments

70
00:04:38,320 --> 00:04:41,360
Speaker 1: the difference of their judgrens. To divide the difference by

71
00:04:41,400 --> 00:04:46,240
Speaker 1: the average, what percentage do you expect most people or

72
00:04:46,400 --> 00:04:50,120
Speaker 1: many people? There is a really common answer to that question.

73
00:04:50,520 --> 00:04:54,640
Speaker 1: People expect about them pass and this was also true

74
00:04:55,040 --> 00:04:58,599
Speaker 1: of the executives in that company. We don't expect judgments

75
00:04:58,640 --> 00:05:04,760
Speaker 1: to be perfectly, but we expect them not to disagree wildly. Now,

76
00:05:05,640 --> 00:05:11,039
Speaker 1: the real number that we observed in the experiment was

77
00:05:11,080 --> 00:05:17,480
Speaker 1: about fifty five zero five times larger than expected, and

78
00:05:17,839 --> 00:05:21,360
Speaker 1: that's really the origin of the book. So it looked

79
00:05:21,800 --> 00:05:26,480
Speaker 1: worth studying, not only because there was a lot of noise,

80
00:05:26,920 --> 00:05:31,480
Speaker 1: but because the noise came as complete news to the organization.

81
00:05:31,920 --> 00:05:35,440
Speaker 1: They were unaware that they had a noise problem. So

82
00:05:36,279 --> 00:05:40,320
Speaker 1: we started with slogan, which is and it turned out

83
00:05:40,320 --> 00:05:43,200
Speaker 1: that there's a lot of noise everywhere, that wherever there

84
00:05:43,279 --> 00:05:46,039
Speaker 1: is judgment, there is noise, and there is more of

85
00:05:46,080 --> 00:05:50,800
Speaker 1: it than you think, and that is really the motivation

86
00:05:50,839 --> 00:05:54,960
Speaker 1: for the book. Olivia joined me very soon and they

87
00:05:55,040 --> 00:05:58,920
Speaker 1: started working in the book together. Cast joined us later. Um,

88
00:06:00,839 --> 00:06:04,120
Speaker 1: that's the story in a year ago the book. It

89
00:06:04,279 --> 00:06:07,760
Speaker 1: is interesting. You know, obviously you're from different fields and Olivia,

90
00:06:07,839 --> 00:06:10,960
Speaker 1: I'm interested in you are drawn to this because obviously

91
00:06:11,000 --> 00:06:14,200
Speaker 1: it has very clear relevance for business strategy and the

92
00:06:14,240 --> 00:06:17,320
Speaker 1: way companies think about sort of what it is they're doing. Well.

93
00:06:17,360 --> 00:06:21,320
Speaker 1: As Danny has just described it, noise is unwanted variability

94
00:06:21,320 --> 00:06:25,000
Speaker 1: in judgments. And this only becomes a problem when you're

95
00:06:25,000 --> 00:06:29,279
Speaker 1: an organization. Noise is a disease of organizations. If you

96
00:06:29,360 --> 00:06:33,120
Speaker 1: are an individual, we will never know how noisy you are.

97
00:06:33,160 --> 00:06:34,680
Speaker 1: You are noisy, by the way, and you are you

98
00:06:34,680 --> 00:06:37,719
Speaker 1: are subject to the same sources of noise that we

99
00:06:37,800 --> 00:06:40,719
Speaker 1: are all subject to it organizations. But where we expect

100
00:06:40,760 --> 00:06:45,320
Speaker 1: consistency is when people in an organization are making judgments

101
00:06:45,360 --> 00:06:48,760
Speaker 1: on behalf of the organization, as in the example of

102
00:06:48,800 --> 00:06:51,719
Speaker 1: the underwriters that Danny was just talking about, and when

103
00:06:51,760 --> 00:06:54,800
Speaker 1: we expect those judgments to be reasonably consistent. If you

104
00:06:54,839 --> 00:06:57,240
Speaker 1: look at another example we've looked at, which is the

105
00:06:57,520 --> 00:07:03,120
Speaker 1: judicial system, we expect that the decisions that judges renders

106
00:07:03,120 --> 00:07:06,839
Speaker 1: should not be too dependent on the identity of the judge.

107
00:07:06,880 --> 00:07:09,960
Speaker 1: Of course, again we expect some variability, but we expect

108
00:07:10,480 --> 00:07:16,000
Speaker 1: general consistency. And the challenge for organizations of any kind

109
00:07:16,280 --> 00:07:21,680
Speaker 1: is to actually achieve something approaching consistency, because first they

110
00:07:21,720 --> 00:07:25,280
Speaker 1: need to realize how much inconsistency they have. They need

111
00:07:25,320 --> 00:07:27,720
Speaker 1: to realize how much noise there is, and as then

112
00:07:27,720 --> 00:07:30,160
Speaker 1: he pointed out, they're not aware of that. It comes

113
00:07:30,160 --> 00:07:33,080
Speaker 1: as a complete surprise when they realize that. So it's

114
00:07:32,520 --> 00:07:38,400
Speaker 1: a huge organizational problem for private enterprises, but also for administrations,

115
00:07:38,400 --> 00:07:44,120
Speaker 1: for government, for non government organizations, for any organization of

116
00:07:44,160 --> 00:07:48,560
Speaker 1: any kind that has many people making judgments and that

117
00:07:48,640 --> 00:07:51,480
Speaker 1: expects consistency. There's so many different strands of this. I

118
00:07:51,480 --> 00:07:54,680
Speaker 1: think there's one which is a straightforward sort of natural

119
00:07:54,760 --> 00:07:57,440
Speaker 1: justice perspective. But some of the examples in the book,

120
00:07:57,600 --> 00:08:00,000
Speaker 1: there's some of the ones that perhaps were most familiar

121
00:08:00,160 --> 00:08:04,160
Speaker 1: with is the variation in sentencing for the same case

122
00:08:04,280 --> 00:08:07,920
Speaker 1: depending on whether the judges football team one or lost

123
00:08:07,960 --> 00:08:11,040
Speaker 1: at the weekend or whatever. It maybe. But it also

124
00:08:11,600 --> 00:08:15,080
Speaker 1: raises a question about what is the nature of expertise

125
00:08:15,880 --> 00:08:19,000
Speaker 1: If we like to think that an experts are experts

126
00:08:19,080 --> 00:08:23,760
Speaker 1: in part because they understand the body of knowledge and

127
00:08:24,440 --> 00:08:27,360
Speaker 1: have a shared understanding of that the way the world

128
00:08:27,400 --> 00:08:30,280
Speaker 1: works in that particular expertise, And what a lot of

129
00:08:30,280 --> 00:08:33,439
Speaker 1: these examples suggests is that they're all experts in their

130
00:08:33,440 --> 00:08:36,520
Speaker 1: own way, and they're all coming up with completely different conclusions.

131
00:08:36,520 --> 00:08:40,680
Speaker 1: Does want come away from this thinking that experts are

132
00:08:41,120 --> 00:08:45,079
Speaker 1: not necessarily helpful for organizations. I think you've come away

133
00:08:45,200 --> 00:08:48,160
Speaker 1: thinking that there are really two different sorts of experts,

134
00:08:48,200 --> 00:08:50,840
Speaker 1: and that we should be clearer in our articulation of

135
00:08:50,880 --> 00:08:55,080
Speaker 1: that distinction. There are experts whose track records can actually

136
00:08:55,080 --> 00:08:59,360
Speaker 1: be evaluated, whose expertise can be quantified measured against a

137
00:08:59,440 --> 00:09:02,360
Speaker 1: gold stand earth. So if you're a forecaster and you

138
00:09:02,400 --> 00:09:06,280
Speaker 1: make short term economic forecasts, and each quarter we can

139
00:09:06,360 --> 00:09:09,400
Speaker 1: check how oft you were and it turns out that

140
00:09:09,480 --> 00:09:12,040
Speaker 1: your forecast historically been very good, we can say you're

141
00:09:12,040 --> 00:09:15,680
Speaker 1: a true expert as forecasting. Now, if you're making thirty

142
00:09:15,720 --> 00:09:20,760
Speaker 1: year forecasts, how much of an expert you are does

143
00:09:20,800 --> 00:09:23,560
Speaker 1: not depend on how good your forecasts are. It depends

144
00:09:23,600 --> 00:09:27,840
Speaker 1: on how much respect we accord you as a forecaster.

145
00:09:28,520 --> 00:09:32,320
Speaker 1: And those are the experts that we call respect experts

146
00:09:32,360 --> 00:09:37,160
Speaker 1: because they're experts, not because they have demonstrable experts ese,

147
00:09:37,280 --> 00:09:40,480
Speaker 1: but because they have convinced others of their expertise, because

148
00:09:40,480 --> 00:09:43,440
Speaker 1: we have respect for their expertise. That is not a

149
00:09:43,520 --> 00:09:46,880
Speaker 1: criticism of those experts, by the way, because in many fields,

150
00:09:46,960 --> 00:09:49,439
Speaker 1: all you can be is a respect expert. Even the

151
00:09:49,520 --> 00:09:53,079
Speaker 1: underwriters that Danny was talking about, we'll never know if

152
00:09:53,120 --> 00:09:56,320
Speaker 1: they've actually set the right premium for an insurance policy.

153
00:09:56,440 --> 00:09:59,119
Speaker 1: So we have respect for them because they are convincing,

154
00:09:59,160 --> 00:10:02,040
Speaker 1: because they can are securely their reasoning in a compelling manner,

155
00:10:02,360 --> 00:10:06,080
Speaker 1: because they have experience, because they have gained confidence in

156
00:10:06,120 --> 00:10:08,240
Speaker 1: the way they do their job. But they are not

157
00:10:08,320 --> 00:10:11,280
Speaker 1: the same sort of experts as the experts whose expertise

158
00:10:11,320 --> 00:10:14,599
Speaker 1: can be demonstrated. And what we argue in Noise is

159
00:10:14,640 --> 00:10:16,880
Speaker 1: that it's important to know what sort of experts you're

160
00:10:16,880 --> 00:10:25,959
Speaker 1: dealing with. When you're dealing with experts, Danny kind of.

161
00:10:26,040 --> 00:10:29,360
Speaker 1: And you talk about good decision hygiene as being the

162
00:10:29,360 --> 00:10:32,080
Speaker 1: sort of equivalent of washing your hands so that you

163
00:10:32,120 --> 00:10:36,160
Speaker 1: can have the limited infection from noise. What what does

164
00:10:36,200 --> 00:10:40,360
Speaker 1: that look like for a for a policymaker or an organization. Well, well,

165
00:10:40,480 --> 00:10:46,520
Speaker 1: whole notion of the hygiene is in contrast to common

166
00:10:46,600 --> 00:10:52,720
Speaker 1: efforts of the biases trying to reduce various biases in

167
00:10:52,800 --> 00:10:57,439
Speaker 1: the thinking of the judgment of organizations and individuals. Uh,

168
00:10:58,880 --> 00:11:03,360
Speaker 1: the bias thing is very much like medication or vaccination.

169
00:11:03,920 --> 00:11:09,679
Speaker 1: It's specific to a particular disease. Hygiene, like washing your hands,

170
00:11:09,840 --> 00:11:13,040
Speaker 1: is non specific. But as you don't know what germs

171
00:11:13,040 --> 00:11:16,560
Speaker 1: you're killing, if you're lucky, you'll never know. And uh,

172
00:11:17,040 --> 00:11:22,080
Speaker 1: that's that's the nature of hygiene. And when you think

173
00:11:22,120 --> 00:11:24,800
Speaker 1: about noise, the only way that we could think of

174
00:11:25,320 --> 00:11:31,000
Speaker 1: improving judgment of reducing noise is by taking steps which

175
00:11:31,000 --> 00:11:35,320
Speaker 1: are generally steps to improve the quality of judgment, but

176
00:11:35,520 --> 00:11:41,480
Speaker 1: are not oriented to particular biases or two combat particular biases.

177
00:11:41,800 --> 00:11:44,360
Speaker 1: What would be a good example of good hygiene where

178
00:11:44,400 --> 00:11:47,280
Speaker 1: you could otherwise have a very noisy and unfair decisions.

179
00:11:47,520 --> 00:11:50,640
Speaker 1: A standard example, and actually an example that was very

180
00:11:50,640 --> 00:11:56,320
Speaker 1: influential on our thinking, is how to conduct hiring interviews.

181
00:11:56,400 --> 00:11:59,280
Speaker 1: And there has been a lot of research on hiring

182
00:11:59,320 --> 00:12:06,400
Speaker 1: interviews and they fall into two broad families. Unstructured interviews,

183
00:12:06,440 --> 00:12:11,559
Speaker 1: that's the common procedure where you talk to the candidate,

184
00:12:11,640 --> 00:12:14,040
Speaker 1: you try to form a general impression, You have a

185
00:12:14,080 --> 00:12:17,520
Speaker 1: conversation with the candidate, there is some human contact, and

186
00:12:17,640 --> 00:12:20,920
Speaker 1: at the end of the process you you make a

187
00:12:20,960 --> 00:12:23,880
Speaker 1: decision or you form an impression of that candid. A

188
00:12:24,000 --> 00:12:29,160
Speaker 1: structured interview is very different. In a structured interview, you

189
00:12:29,240 --> 00:12:32,280
Speaker 1: have a list of topics that you want to think about.

190
00:12:32,640 --> 00:12:39,000
Speaker 1: For example, you want to assess various attributes of the candidate,

191
00:12:39,520 --> 00:12:44,880
Speaker 1: how original, how reliable, many attributes that may be relevant

192
00:12:44,920 --> 00:12:49,240
Speaker 1: to a particular job. And in a structured interview, you

193
00:12:49,600 --> 00:12:53,880
Speaker 1: think about each of these areas in turn and you

194
00:12:54,800 --> 00:12:58,640
Speaker 1: conduct an interview. That is, you ask questions that pertain

195
00:12:58,760 --> 00:13:03,439
Speaker 1: to that particular area, actually write down a grade or

196
00:13:03,640 --> 00:13:08,280
Speaker 1: ranking or rating for that before switching to the next topic.

197
00:13:08,600 --> 00:13:12,400
Speaker 1: So that's a structured interview. Now it turns out that

198
00:13:12,640 --> 00:13:16,920
Speaker 1: neither kind of interview is very good because because basically

199
00:13:17,000 --> 00:13:21,000
Speaker 1: performance in on jobs is very difficult to predict and

200
00:13:21,080 --> 00:13:24,719
Speaker 1: it doesn't depend only on the characteristics of the individual.

201
00:13:25,200 --> 00:13:31,000
Speaker 1: But structured interview are distinctly superior to unstructured interview. And

202
00:13:31,080 --> 00:13:35,800
Speaker 1: so structuring is we think a good idea, and when

203
00:13:35,800 --> 00:13:39,920
Speaker 1: you're making a decision and you're considering various options, you

204
00:13:40,040 --> 00:13:43,160
Speaker 1: might want to consider the options as if there were

205
00:13:43,200 --> 00:13:48,280
Speaker 1: candidates and assess the various attributes of an option. And

206
00:13:49,200 --> 00:13:54,880
Speaker 1: the important feature here that you delay the global intuition.

207
00:13:55,360 --> 00:13:59,920
Speaker 1: You delay the formation of the global impression. Intuition. One

208
00:14:00,000 --> 00:14:03,840
Speaker 1: of the problems of intuitive thinking that it comes very fast.

209
00:14:04,280 --> 00:14:09,480
Speaker 1: They form first impressions, and in unstructured interviews, typically an

210
00:14:09,520 --> 00:14:12,720
Speaker 1: impression is formed very quickly and most of the rest

211
00:14:12,760 --> 00:14:17,840
Speaker 1: of the conversation is to justify the national impression. In

212
00:14:17,840 --> 00:14:20,880
Speaker 1: a structured interview, that's not the case. You deal with

213
00:14:21,000 --> 00:14:23,720
Speaker 1: topics one at the time and you try to delay

214
00:14:23,760 --> 00:14:27,200
Speaker 1: the global view of the candidate until all the information

215
00:14:27,320 --> 00:14:31,320
Speaker 1: is So that's an example of decision. But I mean,

216
00:14:31,320 --> 00:14:35,800
Speaker 1: one of the points about intuition is it's not very controllable.

217
00:14:35,880 --> 00:14:38,320
Speaker 1: So I'm just wondering. I mean, you know, you and

218
00:14:38,480 --> 00:14:43,280
Speaker 1: Olivier and and and behavioral economists may be very aware

219
00:14:43,360 --> 00:14:45,600
Speaker 1: of all the biases and all the noise that you've

220
00:14:45,640 --> 00:14:48,520
Speaker 1: just talked about. But when you're sitting interviewing a candidate

221
00:14:48,640 --> 00:14:52,720
Speaker 1: or a potential colleague at university or whatever, how do

222
00:14:52,760 --> 00:14:56,920
Speaker 1: you actually stop yourself from having a first impression of someone,

223
00:14:56,920 --> 00:15:00,640
Speaker 1: Because by definition of first impression comes unbidden. Oh, you

224
00:15:00,760 --> 00:15:05,200
Speaker 1: will undoubtedly form impresference, there is no question. But but

225
00:15:06,240 --> 00:15:09,120
Speaker 1: if you have a set of questions that you want

226
00:15:09,160 --> 00:15:13,040
Speaker 1: to ask about the person's reliability or about the extent

227
00:15:13,120 --> 00:15:17,520
Speaker 1: of their experience on similar on a similar, unsimilar task,

228
00:15:18,360 --> 00:15:22,800
Speaker 1: those specific questions are going to fail your mind, and

229
00:15:22,840 --> 00:15:26,240
Speaker 1: they're going to push the intuition aside to some extent,

230
00:15:26,600 --> 00:15:30,080
Speaker 1: and you will have an opportunity that you normally do

231
00:15:30,200 --> 00:15:35,280
Speaker 1: not have of disconfirming your initial impression, of finding things

232
00:15:35,280 --> 00:15:39,960
Speaker 1: out that actually do not fit the initial impression. In general,

233
00:15:40,040 --> 00:15:46,760
Speaker 1: an unstructured interview, impressions are self reinforces. You justify your

234
00:15:46,800 --> 00:15:50,720
Speaker 1: initial impression, and that is a source of noise, And

235
00:15:50,880 --> 00:15:55,720
Speaker 1: by structuring the process you reduce that source of noise.

236
00:15:56,120 --> 00:15:59,600
Speaker 1: An additional thing you can do to limit the problem

237
00:15:59,680 --> 00:16:04,400
Speaker 1: that you're pointing out Stephanie is to have different people

238
00:16:04,800 --> 00:16:08,760
Speaker 1: or different sources of information evaluate the different dimensions that

239
00:16:08,840 --> 00:16:12,280
Speaker 1: you are looking at. So, if you're evaluating candidates for

240
00:16:13,080 --> 00:16:18,200
Speaker 1: you know, intelligence, technical skills, and fit with the culture

241
00:16:18,240 --> 00:16:20,800
Speaker 1: of the company, let's assume these are the three dimensions

242
00:16:20,800 --> 00:16:24,040
Speaker 1: of your job description. In an unstructured interview, you would

243
00:16:24,440 --> 00:16:28,080
Speaker 1: form an overall picture of the person and you would

244
00:16:28,440 --> 00:16:30,640
Speaker 1: then raid them on the three dimensions, but they would

245
00:16:30,680 --> 00:16:33,640
Speaker 1: be strongly correlated with each other because there would be

246
00:16:33,680 --> 00:16:36,320
Speaker 1: a positive or a negative halo around the person, and

247
00:16:36,360 --> 00:16:38,960
Speaker 1: you would say they are great and everything, or they're

248
00:16:38,960 --> 00:16:42,920
Speaker 1: bad on everything. Now suppose that we say you, Stephanie,

249
00:16:42,960 --> 00:16:45,520
Speaker 1: are going to conduct the interview about the technical skills,

250
00:16:45,960 --> 00:16:48,600
Speaker 1: or maybe in fact, we're going to have a technical

251
00:16:48,600 --> 00:16:51,720
Speaker 1: test to evaluate the technical skills. Someone else is going

252
00:16:51,760 --> 00:16:54,720
Speaker 1: to evaluate the fit with the company, and someone else

253
00:16:54,800 --> 00:16:57,280
Speaker 1: is going to evaluate how smart the person is, or again,

254
00:16:57,320 --> 00:17:00,440
Speaker 1: perhaps we're going to have the test of how smart

255
00:17:00,480 --> 00:17:03,280
Speaker 1: the person is. Now you've got three independent data points

256
00:17:03,280 --> 00:17:06,000
Speaker 1: that do not influence each other, and you have a

257
00:17:06,080 --> 00:17:09,160
Speaker 1: much more structured process to make your decision. You would

258
00:17:09,160 --> 00:17:10,720
Speaker 1: have anounced this before, but of course, a lot of

259
00:17:10,760 --> 00:17:14,760
Speaker 1: people entering the job market now find that they're at

260
00:17:14,840 --> 00:17:17,119
Speaker 1: least the first couple of rounds, depending on how popular

261
00:17:17,160 --> 00:17:19,840
Speaker 1: the job is is, and they're talking to a computer

262
00:17:20,160 --> 00:17:24,320
Speaker 1: or they have there is a an AI element to

263
00:17:24,440 --> 00:17:28,000
Speaker 1: their application process. We may not like it, we may

264
00:17:28,040 --> 00:17:31,200
Speaker 1: think it's not true to our great sense of intuition

265
00:17:31,240 --> 00:17:35,040
Speaker 1: about people. But from a fairness perspective and from a

266
00:17:35,040 --> 00:17:38,040
Speaker 1: decision hygiene perspective, is that a better way to go?

267
00:17:38,640 --> 00:17:40,720
Speaker 1: We need to be careful here, because there is an

268
00:17:40,720 --> 00:17:43,080
Speaker 1: answer in principle, and there is an answer in practice.

269
00:17:43,960 --> 00:17:49,200
Speaker 1: In principle, any form of structured decision making that reduces

270
00:17:49,280 --> 00:17:53,160
Speaker 1: noise would in fact enhance the quality of the decisions.

271
00:17:53,160 --> 00:17:56,399
Speaker 1: So if you have an algorithm making decisions, there is

272
00:17:56,440 --> 00:17:58,560
Speaker 1: going to be less noise there. But of course the

273
00:17:58,680 --> 00:18:01,440
Speaker 1: question is how good is the algorithm? How good are

274
00:18:01,520 --> 00:18:05,680
Speaker 1: those AI systems that people sit in front of, And

275
00:18:06,480 --> 00:18:09,480
Speaker 1: I'm sure there are good ones, but from the ones

276
00:18:09,560 --> 00:18:14,320
Speaker 1: that I've seen personally in my admittedly limited experience, there

277
00:18:14,440 --> 00:18:19,280
Speaker 1: isn't much evidence, and there isn't very good quality evidence

278
00:18:19,400 --> 00:18:24,840
Speaker 1: that what these software packages are testing for is actually

279
00:18:25,040 --> 00:18:27,640
Speaker 1: what you're looking for. It's actually quite hard for most

280
00:18:27,640 --> 00:18:29,840
Speaker 1: companies to define what it is that they're looking for,

281
00:18:30,520 --> 00:18:34,200
Speaker 1: and there is no evidence that I've seen that there's

282
00:18:34,320 --> 00:18:38,440
Speaker 1: any correlation between when those software packages look for and

283
00:18:39,040 --> 00:18:45,720
Speaker 1: job success is actually highly correlated. So in practice I'm

284
00:18:45,800 --> 00:18:50,639
Speaker 1: quite skeptical about what I see in the market. In theory,

285
00:18:51,480 --> 00:18:53,280
Speaker 1: I have to agree that it makes some sense, but

286
00:18:53,400 --> 00:18:55,879
Speaker 1: my worry is that companies are using this mostly to

287
00:18:56,040 --> 00:18:59,280
Speaker 1: save time and money, not to actually improve the quality

288
00:18:59,280 --> 00:19:03,359
Speaker 1: of their decision. Here, I would hope that what is

289
00:19:03,400 --> 00:19:06,680
Speaker 1: true in theory can be made true in practice. And

290
00:19:07,160 --> 00:19:13,720
Speaker 1: one characteristics of algorithms and is that they're improvable. They're

291
00:19:13,800 --> 00:19:17,680
Speaker 1: much more improvable than people are, and and they can

292
00:19:17,720 --> 00:19:22,320
Speaker 1: be corrected by by data on quality. They can be

293
00:19:22,400 --> 00:19:26,560
Speaker 1: made to predict more accurately. So this this is really

294
00:19:26,600 --> 00:19:32,200
Speaker 1: an issue of the quality of constructing algorithms. And there

295
00:19:32,240 --> 00:19:36,320
Speaker 1: are many algorithms that are of poor quality out there

296
00:19:36,320 --> 00:19:40,639
Speaker 1: on the market, and there is a widespread suspicion of

297
00:19:40,720 --> 00:19:45,560
Speaker 1: algorithms which makes us prone to reject them. But by

298
00:19:45,600 --> 00:19:49,920
Speaker 1: and large, I think this is the future. In the future,

299
00:19:50,040 --> 00:19:53,040
Speaker 1: there will be more and more of those algorithms, and

300
00:19:53,119 --> 00:19:57,640
Speaker 1: their quality will be getting better and better every year

301
00:19:58,000 --> 00:20:01,040
Speaker 1: because there will be data there will be feedback, and

302
00:20:01,080 --> 00:20:04,920
Speaker 1: the feedback can be incorporated into an algorithm much more

303
00:20:04,960 --> 00:20:11,200
Speaker 1: efficiently than it can be in the human judgment. So uh,

304
00:20:11,480 --> 00:20:16,960
Speaker 1: here I join Olivia's skepticism about most of the algorithms

305
00:20:16,960 --> 00:20:20,000
Speaker 1: that exist, but I really want to register and mode

306
00:20:20,000 --> 00:20:23,959
Speaker 1: of optimism about the future of that kind of operation.

307
00:20:30,040 --> 00:20:33,760
Speaker 1: There's one trend which is about eliminating the human element

308
00:20:33,960 --> 00:20:37,520
Speaker 1: to some extent, or at least having it in a

309
00:20:37,520 --> 00:20:42,000
Speaker 1: more regular form in an algorithm, a more consistent form structured.

310
00:20:42,840 --> 00:20:48,160
Speaker 1: Of course, the other big trend in business strategy and

311
00:20:49,200 --> 00:20:53,800
Speaker 1: conversations about companies is the move is encouragement of diversity

312
00:20:54,000 --> 00:20:57,159
Speaker 1: and to encourage businesses in a sense to have a

313
00:20:57,200 --> 00:21:00,840
Speaker 1: wider variety of humans doing the judgment. And I wonder

314
00:21:00,840 --> 00:21:03,359
Speaker 1: whether that even goes against some of the things that

315
00:21:03,400 --> 00:21:04,960
Speaker 1: you're talking about. You know, one of the ways that

316
00:21:05,000 --> 00:21:09,159
Speaker 1: companies might have previously eliminated noise, not necessarily error, but noise,

317
00:21:09,960 --> 00:21:12,240
Speaker 1: would have been having lots of identical people or making

318
00:21:12,240 --> 00:21:14,879
Speaker 1: the decisions all of these white men sitting in their boards.

319
00:21:15,040 --> 00:21:18,159
Speaker 1: If you now have a greatly much more diversity, you

320
00:21:18,240 --> 00:21:22,439
Speaker 1: might be more true to the range of human experience,

321
00:21:22,560 --> 00:21:25,639
Speaker 1: but you'll be getting a lot more noise. Well, that

322
00:21:25,840 --> 00:21:34,920
Speaker 1: is certainly true, but in in principle, we we want

323
00:21:34,960 --> 00:21:39,359
Speaker 1: to distinguish between the process of generating a judgment and

324
00:21:39,880 --> 00:21:44,080
Speaker 1: the final judgment. In the process of generating a judgment,

325
00:21:44,400 --> 00:21:49,439
Speaker 1: diversity is very welcome. That is, you want multiple points

326
00:21:49,440 --> 00:21:56,199
Speaker 1: of view, you want people ptise to enter into the

327
00:21:56,600 --> 00:22:02,200
Speaker 1: participate in the conversation. But when a final judgment is made,

328
00:22:02,560 --> 00:22:07,439
Speaker 1: we want a process that reduces noises. So diversity is

329
00:22:07,640 --> 00:22:11,720
Speaker 1: very useful, and you know it's it's you can think

330
00:22:11,760 --> 00:22:15,399
Speaker 1: of that in terms of, say, witnesses to a crime.

331
00:22:16,240 --> 00:22:20,600
Speaker 1: So you're better off if the witnesses are in different

332
00:22:20,640 --> 00:22:24,560
Speaker 1: places and see the event from different perspectives. And you're

333
00:22:24,600 --> 00:22:28,520
Speaker 1: certainly better off if if the witnesses don't talk to

334
00:22:28,560 --> 00:22:31,560
Speaker 1: each other and they are independent of each other. And

335
00:22:31,640 --> 00:22:36,360
Speaker 1: so thinking along those lines gives you an idea that

336
00:22:36,440 --> 00:22:41,040
Speaker 1: you do want diversity, but you want also the kind

337
00:22:41,040 --> 00:22:45,800
Speaker 1: of independence and the kind of goal directiveness that reduces

338
00:22:45,880 --> 00:22:49,919
Speaker 1: noise in the final journey. Diversity in the outcome of

339
00:22:49,960 --> 00:22:52,560
Speaker 1: these decisions, in the judgment that you produce in the end.

340
00:22:53,359 --> 00:22:55,719
Speaker 1: It's good for some things, but for most it's not.

341
00:22:56,440 --> 00:22:58,560
Speaker 1: When you when you go to the doctor and the

342
00:22:58,600 --> 00:23:00,840
Speaker 1: doctor tells you, oh, you have is disease. And then

343
00:23:00,880 --> 00:23:02,600
Speaker 1: you go to another doctor and he tells you you

344
00:23:02,640 --> 00:23:05,520
Speaker 1: have that disease. You don't say, oh, that's wonderful, it's diversity.

345
00:23:05,560 --> 00:23:08,800
Speaker 1: You say one of these two doctors is wrong, maybe both.

346
00:23:09,400 --> 00:23:12,320
Speaker 1: So whenever we think that there is a correct answer,

347
00:23:12,960 --> 00:23:16,480
Speaker 1: diversity in the outcome is not good. Maybe the way

348
00:23:16,480 --> 00:23:18,680
Speaker 1: to get to the correct outcome is to harness the

349
00:23:18,720 --> 00:23:22,520
Speaker 1: diversity of the perspectives or through multiple witnesses, and that

350
00:23:22,680 --> 00:23:24,359
Speaker 1: is one of the remedies that you can have to

351
00:23:24,400 --> 00:23:28,080
Speaker 1: reduce noise. But as an organization, what you're aiming for

352
00:23:28,600 --> 00:23:33,080
Speaker 1: is not every person having their own opinion. It's any

353
00:23:33,160 --> 00:23:37,000
Speaker 1: person having the best possible judgment. And we've talked about business,

354
00:23:37,000 --> 00:23:41,960
Speaker 1: we've talked about justice or you know, sentencing and decisions

355
00:23:42,000 --> 00:23:45,920
Speaker 1: within the criminal justice system, and an area that comes

356
00:23:46,000 --> 00:23:47,880
Speaker 1: up a little bit in your book, but obviously it's

357
00:23:47,920 --> 00:23:49,800
Speaker 1: kind of front and center of people's minds at the

358
00:23:49,840 --> 00:23:52,520
Speaker 1: moment when we think of the decisions being taken around

359
00:23:52,560 --> 00:23:57,480
Speaker 1: the war in in Ukraine. Is that in a critical

360
00:23:57,560 --> 00:24:04,399
Speaker 1: moments of foreign policy or military strategy decisions, you know,

361
00:24:04,440 --> 00:24:08,840
Speaker 1: you can't necessarily enlist a lot of people and listen

362
00:24:08,920 --> 00:24:12,800
Speaker 1: to their structured answers on a set of questions in

363
00:24:12,840 --> 00:24:17,119
Speaker 1: reaching your judgment about how to respond to Russia or

364
00:24:17,119 --> 00:24:19,600
Speaker 1: how to how to respond to one of the sort

365
00:24:19,600 --> 00:24:22,879
Speaker 1: of very pressing situations that can arise in foreign policy.

366
00:24:23,280 --> 00:24:26,000
Speaker 1: So I just wonder whether you whether you'd reflected on that,

367
00:24:26,040 --> 00:24:27,920
Speaker 1: you know, if you're Anthony Blink in the sector of state,

368
00:24:28,000 --> 00:24:31,919
Speaker 1: or if you're President Biden, or or that matter, at

369
00:24:32,000 --> 00:24:36,600
Speaker 1: a Russian general how what does decision hygiene look like

370
00:24:36,800 --> 00:24:39,639
Speaker 1: in those kind of situations where there's inevitably going to

371
00:24:39,720 --> 00:24:42,200
Speaker 1: be a limited number of people that you can call

372
00:24:42,240 --> 00:24:49,760
Speaker 1: on and imperfect information hygiene is something that an individual

373
00:24:49,920 --> 00:24:54,359
Speaker 1: can follow. That is, there are better and less good

374
00:24:54,600 --> 00:24:59,040
Speaker 1: ways of individual judgments. You want to cover all the

375
00:24:59,040 --> 00:25:02,159
Speaker 1: bases you are to think, You want to the extent

376
00:25:02,240 --> 00:25:05,600
Speaker 1: possible to think of all possible consequences, you know, as

377
00:25:05,640 --> 00:25:09,280
Speaker 1: salient example in the Ukraine War, is it looks unlikely

378
00:25:09,880 --> 00:25:14,879
Speaker 1: that that people who started that war knew that Finland

379
00:25:14,880 --> 00:25:18,399
Speaker 1: and Sweden would want to join NATO, because you know,

380
00:25:18,640 --> 00:25:22,040
Speaker 1: after all, this was supposed to keep NATO away. So

381
00:25:22,240 --> 00:25:24,600
Speaker 1: when I have the feeling that not all the bases

382
00:25:24,640 --> 00:25:30,920
Speaker 1: were covered in making those critical, so there are all

383
00:25:31,000 --> 00:25:34,280
Speaker 1: we can hope for is that we have people with

384
00:25:34,359 --> 00:25:37,680
Speaker 1: a lot of experience, because it turns out that there

385
00:25:37,800 --> 00:25:42,240
Speaker 1: is genuine intuitive experience that can develop over time with

386
00:25:42,400 --> 00:25:47,480
Speaker 1: institution within certain kinds of decisions and choices. And we

387
00:25:47,560 --> 00:25:52,560
Speaker 1: also want people who, even under sometime pressure, UH, can

388
00:25:52,680 --> 00:25:56,840
Speaker 1: follow the basic dictates of decision hype. I've noticed a

389
00:25:56,920 --> 00:25:58,679
Speaker 1: professor kind of and that a lot of interviews with

390
00:25:58,720 --> 00:26:01,240
Speaker 1: you are quite long, a long than the sort of

391
00:26:01,280 --> 00:26:03,640
Speaker 1: the norm for whatever program it is, And I suspect

392
00:26:03,640 --> 00:26:07,560
Speaker 1: it's because it's so it's so it's always so fascinating

393
00:26:07,600 --> 00:26:08,960
Speaker 1: to listen to you, and you always want to have

394
00:26:08,960 --> 00:26:12,040
Speaker 1: another question. But we're going to run out of time.

395
00:26:12,320 --> 00:26:15,119
Speaker 1: So thank you very much for coming on Stephonomics, and

396
00:26:15,160 --> 00:26:18,679
Speaker 1: thanks thank you to Olivier Simbony, thank you, thank you

397
00:26:18,760 --> 00:26:26,800
Speaker 1: very much. That's it for this episode of Stephonomics. We'll

398
00:26:26,840 --> 00:26:29,320
Speaker 1: be back next week. In the meantime, do please rate

399
00:26:29,359 --> 00:26:31,199
Speaker 1: the show if you like it, and check out the

400
00:26:31,200 --> 00:26:34,760
Speaker 1: Bloomberg Terminal and News website for more economic news and

401
00:26:34,840 --> 00:26:37,520
Speaker 1: views on the global economy. You can also follow our

402
00:26:37,600 --> 00:26:41,800
Speaker 1: economics on Twitter. This episode was produced by Magnus Henrickson

403
00:26:41,920 --> 00:26:45,320
Speaker 1: and Summer Said, with special thanks to Professor Daniel Kannerman

404
00:26:45,400 --> 00:26:49,520
Speaker 1: and Olivier Sibon. Mike Sasso is executive producer of Stephonomics

405
00:26:49,640 --> 00:27:04,680
Speaker 1: and the head of Bloomberg Podcast is Francesca Levi. The