WEBVTT - Silencing the ‘Noise’ Behind Bad Corporate Decisionmaking 0:00:05.200 --> 0:00:07.880 Hello, and welcome to Stephanomics, the podcast that brings the 0:00:07.880 --> 0:00:10.240 global economy to you. And this week we have a 0:00:10.320 --> 0:00:13.520 treat in the form of a conversation with Daniel Kannerman, 0:00:13.840 --> 0:00:17.840 the most influential and respected psychologist in the world. He's 0:00:17.840 --> 0:00:20.119 one of a small number of non economists to have 0:00:20.160 --> 0:00:23.200 won the Nobel Prize for Economics for his contribution to 0:00:23.239 --> 0:00:26.960 the school of behavioral economics. With his collaborator Amos Verski, 0:00:27.560 --> 0:00:30.360 he also wrote what I'm sure is the best selling 0:00:30.360 --> 0:00:34.320 psychology book of all time, Thinking Fast and Slow, along 0:00:34.320 --> 0:00:37.199 with Olivier Siboni and cass Sunstein. He has a new 0:00:37.200 --> 0:00:40.440 book just coming out in paperback, Noise The Flaw in 0:00:40.560 --> 0:00:44.360 Human Judgment. Olivier is a professor of Strategy and Business 0:00:44.360 --> 0:00:46.960 Policy at h GC Paris, and I'm pleased to say 0:00:47.000 --> 0:00:49.479 he joined us for the interview. But I started by 0:00:49.479 --> 0:00:52.280 asking Kannerman to tell us what this great flaw in 0:00:52.400 --> 0:01:02.680 human judgment was that he caused noise? Define uh, what 0:01:02.840 --> 0:01:07.520 noises in relation to its cousin more familiar cousin, which 0:01:07.600 --> 0:01:12.039 is biased? Uh. And we're talk in general about errors 0:01:12.040 --> 0:01:15.600 of judgment, and the context when we talk about errors 0:01:15.640 --> 0:01:20.119 of judgment is to compare judgment to measurement. That's where 0:01:21.000 --> 0:01:25.880 the whole notions of accuracy and accuracy of measurement and error. 0:01:26.920 --> 0:01:31.520 The regulous treatment of errors stems so com from the 0:01:31.600 --> 0:01:35.560 comparison of judgment to measurement. Now, in measurement, when you're 0:01:35.560 --> 0:01:41.160 measuring the same object multiple times with the very fine rulers, 0:01:41.480 --> 0:01:43.880 we're not going to get the same result every time. 0:01:44.319 --> 0:01:47.200 That is, if the ruler is fine enough, there's going 0:01:47.240 --> 0:01:52.360 to be variability. That variability, that's the variability that we 0:01:52.440 --> 0:01:56.400 call noise. So the errors some areas are positive of 0:01:56.480 --> 0:01:59.680 the errors are negative, and the variability of the air 0:02:00.000 --> 0:02:04.840 a noise. The average error is a bias, so that 0:02:04.920 --> 0:02:08.520 you can have positive bias or negative bias. And it 0:02:08.639 --> 0:02:11.679 turns out that in the discussion of errors of judgment 0:02:12.320 --> 0:02:17.280 we have focused on systematic errors on biases, and very 0:02:17.360 --> 0:02:20.360 little attention has been paid in the discussion of error 0:02:20.840 --> 0:02:25.480 and two noise to variability. But in the theory of measurements, 0:02:25.960 --> 0:02:30.320 bias and noise actually have equivalent weight, and there is 0:02:30.440 --> 0:02:35.360 reason to believe that more inaccuracy and judgment is often 0:02:35.480 --> 0:02:39.080 due to noise than to bias. And this is you 0:02:39.120 --> 0:02:41.280 know where the book came to be. And I guess 0:02:41.320 --> 0:02:43.720 it's worth saying. We're used to thinking of the bias 0:02:43.800 --> 0:02:47.320 that you'll be systematically leaning in one direction or another 0:02:47.400 --> 0:02:50.240 in the directions that we in the decisions that we take. 0:02:50.560 --> 0:02:53.320 But I guess the point of the noise is that 0:02:53.440 --> 0:02:57.080 it's not predictable. And I guess crucially we fail to 0:02:57.200 --> 0:03:00.360 understand not just that there is noise, but the stint 0:03:00.440 --> 0:03:03.079 of the noise. We ask experts something and we expect 0:03:03.160 --> 0:03:07.600 there to be only a relatively small variation in their decisions, 0:03:08.120 --> 0:03:10.520 and actually there's a huge variation. So I guess it's 0:03:10.560 --> 0:03:13.360 it's worth talking through one of those examples just to 0:03:13.360 --> 0:03:15.600 give a sense of what you're talking about. I can 0:03:15.840 --> 0:03:20.120 describe the example from what the study began, and this 0:03:20.560 --> 0:03:23.200 It began about eight years ago when I was doing 0:03:23.240 --> 0:03:29.040 some consulting in an insurance company and I conducted a 0:03:29.040 --> 0:03:32.720 fairly between experiment that today we would call a noise audit, 0:03:33.320 --> 0:03:40.520 where cases were constructed which were very common representative of 0:03:40.640 --> 0:03:44.200 the work of underwriters in that company, and then the 0:03:44.280 --> 0:03:48.400 same cases were presented to several dozen underwriters about fifty 0:03:48.440 --> 0:03:52.920 years I recall, and we looked at the variability they 0:03:53.000 --> 0:03:56.520 looked at. They put a dollar value on those cases. Now, 0:03:56.560 --> 0:04:01.720 notice those cases worth were fictions. They were constructed as such, 0:04:02.040 --> 0:04:05.040 but they were very typical. And the idea was that 0:04:05.080 --> 0:04:10.160 if underwriters vary in their judgments of those hypothetical cases, 0:04:10.440 --> 0:04:14.160 they would also vary in their judgments of real case. Now, 0:04:14.480 --> 0:04:17.800 I ask executives in the company a question that I 0:04:17.880 --> 0:04:21.800 think anybody was listening to this would also ask themselves. 0:04:22.080 --> 0:04:26.240 If you look at two underwriters and you should pick 0:04:26.320 --> 0:04:30.600 that random and our large the difference do you expect 0:04:30.600 --> 0:04:35.279 to find between them in percentages? That is, you take 0:04:35.320 --> 0:04:38.279 the two underwriters, you compute their average of their judgments 0:04:38.320 --> 0:04:41.360 the difference of their judgrens. To divide the difference by 0:04:41.400 --> 0:04:46.240 the average, what percentage do you expect most people or 0:04:46.400 --> 0:04:50.120 many people? There is a really common answer to that question. 0:04:50.520 --> 0:04:54.640 People expect about them pass and this was also true 0:04:55.040 --> 0:04:58.599 of the executives in that company. We don't expect judgments 0:04:58.640 --> 0:05:04.760 to be perfectly, but we expect them not to disagree wildly. Now, 0:05:05.640 --> 0:05:11.039 the real number that we observed in the experiment was 0:05:11.080 --> 0:05:17.480 about fifty five zero five times larger than expected, and 0:05:17.839 --> 0:05:21.360 that's really the origin of the book. So it looked 0:05:21.800 --> 0:05:26.480 worth studying, not only because there was a lot of noise, 0:05:26.920 --> 0:05:31.480 but because the noise came as complete news to the organization. 0:05:31.920 --> 0:05:35.440 They were unaware that they had a noise problem. So 0:05:36.279 --> 0:05:40.320 we started with slogan, which is and it turned out 0:05:40.320 --> 0:05:43.200 that there's a lot of noise everywhere, that wherever there 0:05:43.279 --> 0:05:46.039 is judgment, there is noise, and there is more of 0:05:46.080 --> 0:05:50.800 it than you think, and that is really the motivation 0:05:50.839 --> 0:05:54.960 for the book. Olivia joined me very soon and they 0:05:55.040 --> 0:05:58.920 started working in the book together. Cast joined us later. Um, 0:06:00.839 --> 0:06:04.120 that's the story in a year ago the book. It 0:06:04.279 --> 0:06:07.760 is interesting. You know, obviously you're from different fields and Olivia, 0:06:07.839 --> 0:06:10.960 I'm interested in you are drawn to this because obviously 0:06:11.000 --> 0:06:14.200 it has very clear relevance for business strategy and the 0:06:14.240 --> 0:06:17.320 way companies think about sort of what it is they're doing. Well. 0:06:17.360 --> 0:06:21.320 As Danny has just described it, noise is unwanted variability 0:06:21.320 --> 0:06:25.000 in judgments. And this only becomes a problem when you're 0:06:25.000 --> 0:06:29.279 an organization. Noise is a disease of organizations. If you 0:06:29.360 --> 0:06:33.120 are an individual, we will never know how noisy you are. 0:06:33.160 --> 0:06:34.680 You are noisy, by the way, and you are you 0:06:34.680 --> 0:06:37.719 are subject to the same sources of noise that we 0:06:37.800 --> 0:06:40.719 are all subject to it organizations. But where we expect 0:06:40.760 --> 0:06:45.320 consistency is when people in an organization are making judgments 0:06:45.360 --> 0:06:48.760 on behalf of the organization, as in the example of 0:06:48.800 --> 0:06:51.719 the underwriters that Danny was just talking about, and when 0:06:51.760 --> 0:06:54.800 we expect those judgments to be reasonably consistent. If you 0:06:54.839 --> 0:06:57.240 look at another example we've looked at, which is the 0:06:57.520 --> 0:07:03.120 judicial system, we expect that the decisions that judges renders 0:07:03.120 --> 0:07:06.839 should not be too dependent on the identity of the judge. 0:07:06.880 --> 0:07:09.960 Of course, again we expect some variability, but we expect 0:07:10.480 --> 0:07:16.000 general consistency. And the challenge for organizations of any kind 0:07:16.280 --> 0:07:21.680 is to actually achieve something approaching consistency, because first they 0:07:21.720 --> 0:07:25.280 need to realize how much inconsistency they have. They need 0:07:25.320 --> 0:07:27.720 to realize how much noise there is, and as then 0:07:27.720 --> 0:07:30.160 he pointed out, they're not aware of that. It comes 0:07:30.160 --> 0:07:33.080 as a complete surprise when they realize that. So it's 0:07:32.520 --> 0:07:38.400 a huge organizational problem for private enterprises, but also for administrations, 0:07:38.400 --> 0:07:44.120 for government, for non government organizations, for any organization of 0:07:44.160 --> 0:07:48.560 any kind that has many people making judgments and that 0:07:48.640 --> 0:07:51.480 expects consistency. There's so many different strands of this. I 0:07:51.480 --> 0:07:54.680 think there's one which is a straightforward sort of natural 0:07:54.760 --> 0:07:57.440 justice perspective. But some of the examples in the book, 0:07:57.600 --> 0:08:00.000 there's some of the ones that perhaps were most familiar 0:08:00.160 --> 0:08:04.160 with is the variation in sentencing for the same case 0:08:04.280 --> 0:08:07.920 depending on whether the judges football team one or lost 0:08:07.960 --> 0:08:11.040 at the weekend or whatever. It maybe. But it also 0:08:11.600 --> 0:08:15.080 raises a question about what is the nature of expertise 0:08:15.880 --> 0:08:19.000 If we like to think that an experts are experts 0:08:19.080 --> 0:08:23.760 in part because they understand the body of knowledge and 0:08:24.440 --> 0:08:27.360 have a shared understanding of that the way the world 0:08:27.400 --> 0:08:30.280 works in that particular expertise, And what a lot of 0:08:30.280 --> 0:08:33.439 these examples suggests is that they're all experts in their 0:08:33.440 --> 0:08:36.520 own way, and they're all coming up with completely different conclusions. 0:08:36.520 --> 0:08:40.680 Does want come away from this thinking that experts are 0:08:41.120 --> 0:08:45.079 not necessarily helpful for organizations. I think you've come away 0:08:45.200 --> 0:08:48.160 thinking that there are really two different sorts of experts, 0:08:48.200 --> 0:08:50.840 and that we should be clearer in our articulation of 0:08:50.880 --> 0:08:55.080 that distinction. There are experts whose track records can actually 0:08:55.080 --> 0:08:59.360 be evaluated, whose expertise can be quantified measured against a 0:08:59.440 --> 0:09:02.360 gold stand earth. So if you're a forecaster and you 0:09:02.400 --> 0:09:06.280 make short term economic forecasts, and each quarter we can 0:09:06.360 --> 0:09:09.400 check how oft you were and it turns out that 0:09:09.480 --> 0:09:12.040 your forecast historically been very good, we can say you're 0:09:12.040 --> 0:09:15.680 a true expert as forecasting. Now, if you're making thirty 0:09:15.720 --> 0:09:20.760 year forecasts, how much of an expert you are does 0:09:20.800 --> 0:09:23.560 not depend on how good your forecasts are. It depends 0:09:23.600 --> 0:09:27.840 on how much respect we accord you as a forecaster. 0:09:28.520 --> 0:09:32.320 And those are the experts that we call respect experts 0:09:32.360 --> 0:09:37.160 because they're experts, not because they have demonstrable experts ese, 0:09:37.280 --> 0:09:40.480 but because they have convinced others of their expertise, because 0:09:40.480 --> 0:09:43.440 we have respect for their expertise. That is not a 0:09:43.520 --> 0:09:46.880 criticism of those experts, by the way, because in many fields, 0:09:46.960 --> 0:09:49.439 all you can be is a respect expert. Even the 0:09:49.520 --> 0:09:53.079 underwriters that Danny was talking about, we'll never know if 0:09:53.120 --> 0:09:56.320 they've actually set the right premium for an insurance policy. 0:09:56.440 --> 0:09:59.119 So we have respect for them because they are convincing, 0:09:59.160 --> 0:10:02.040 because they can are securely their reasoning in a compelling manner, 0:10:02.360 --> 0:10:06.080 because they have experience, because they have gained confidence in 0:10:06.120 --> 0:10:08.240 the way they do their job. But they are not 0:10:08.320 --> 0:10:11.280 the same sort of experts as the experts whose expertise 0:10:11.320 --> 0:10:14.599 can be demonstrated. And what we argue in Noise is 0:10:14.640 --> 0:10:16.880 that it's important to know what sort of experts you're 0:10:16.880 --> 0:10:25.959 dealing with. When you're dealing with experts, Danny kind of. 0:10:26.040 --> 0:10:29.360 And you talk about good decision hygiene as being the 0:10:29.360 --> 0:10:32.080 sort of equivalent of washing your hands so that you 0:10:32.120 --> 0:10:36.160 can have the limited infection from noise. What what does 0:10:36.200 --> 0:10:40.360 that look like for a for a policymaker or an organization. Well, well, 0:10:40.480 --> 0:10:46.520 whole notion of the hygiene is in contrast to common 0:10:46.600 --> 0:10:52.720 efforts of the biases trying to reduce various biases in 0:10:52.800 --> 0:10:57.439 the thinking of the judgment of organizations and individuals. Uh, 0:10:58.880 --> 0:11:03.360 the bias thing is very much like medication or vaccination. 0:11:03.920 --> 0:11:09.679 It's specific to a particular disease. Hygiene, like washing your hands, 0:11:09.840 --> 0:11:13.040 is non specific. But as you don't know what germs 0:11:13.040 --> 0:11:16.560 you're killing, if you're lucky, you'll never know. And uh, 0:11:17.040 --> 0:11:22.080 that's that's the nature of hygiene. And when you think 0:11:22.120 --> 0:11:24.800 about noise, the only way that we could think of 0:11:25.320 --> 0:11:31.000 improving judgment of reducing noise is by taking steps which 0:11:31.000 --> 0:11:35.320 are generally steps to improve the quality of judgment, but 0:11:35.520 --> 0:11:41.480 are not oriented to particular biases or two combat particular biases. 0:11:41.800 --> 0:11:44.360 What would be a good example of good hygiene where 0:11:44.400 --> 0:11:47.280 you could otherwise have a very noisy and unfair decisions. 0:11:47.520 --> 0:11:50.640 A standard example, and actually an example that was very 0:11:50.640 --> 0:11:56.320 influential on our thinking, is how to conduct hiring interviews. 0:11:56.400 --> 0:11:59.280 And there has been a lot of research on hiring 0:11:59.320 --> 0:12:06.400 interviews and they fall into two broad families. Unstructured interviews, 0:12:06.440 --> 0:12:11.559 that's the common procedure where you talk to the candidate, 0:12:11.640 --> 0:12:14.040 you try to form a general impression, You have a 0:12:14.080 --> 0:12:17.520 conversation with the candidate, there is some human contact, and 0:12:17.640 --> 0:12:20.920 at the end of the process you you make a 0:12:20.960 --> 0:12:23.880 decision or you form an impression of that candid. A 0:12:24.000 --> 0:12:29.160 structured interview is very different. In a structured interview, you 0:12:29.240 --> 0:12:32.280 have a list of topics that you want to think about. 0:12:32.640 --> 0:12:39.000 For example, you want to assess various attributes of the candidate, 0:12:39.520 --> 0:12:44.880 how original, how reliable, many attributes that may be relevant 0:12:44.920 --> 0:12:49.240 to a particular job. And in a structured interview, you 0:12:49.600 --> 0:12:53.880 think about each of these areas in turn and you 0:12:54.800 --> 0:12:58.640 conduct an interview. That is, you ask questions that pertain 0:12:58.760 --> 0:13:03.439 to that particular area, actually write down a grade or 0:13:03.640 --> 0:13:08.280 ranking or rating for that before switching to the next topic. 0:13:08.600 --> 0:13:12.400 So that's a structured interview. Now it turns out that 0:13:12.640 --> 0:13:16.920 neither kind of interview is very good because because basically 0:13:17.000 --> 0:13:21.000 performance in on jobs is very difficult to predict and 0:13:21.080 --> 0:13:24.719 it doesn't depend only on the characteristics of the individual. 0:13:25.200 --> 0:13:31.000 But structured interview are distinctly superior to unstructured interview. And 0:13:31.080 --> 0:13:35.800 so structuring is we think a good idea, and when 0:13:35.800 --> 0:13:39.920 you're making a decision and you're considering various options, you 0:13:40.040 --> 0:13:43.160 might want to consider the options as if there were 0:13:43.200 --> 0:13:48.280 candidates and assess the various attributes of an option. And 0:13:49.200 --> 0:13:54.880 the important feature here that you delay the global intuition. 0:13:55.360 --> 0:13:59.920 You delay the formation of the global impression. Intuition. One 0:14:00.000 --> 0:14:03.840 of the problems of intuitive thinking that it comes very fast. 0:14:04.280 --> 0:14:09.480 They form first impressions, and in unstructured interviews, typically an 0:14:09.520 --> 0:14:12.720 impression is formed very quickly and most of the rest 0:14:12.760 --> 0:14:17.840 of the conversation is to justify the national impression. In 0:14:17.840 --> 0:14:20.880 a structured interview, that's not the case. You deal with 0:14:21.000 --> 0:14:23.720 topics one at the time and you try to delay 0:14:23.760 --> 0:14:27.200 the global view of the candidate until all the information 0:14:27.320 --> 0:14:31.320 is So that's an example of decision. But I mean, 0:14:31.320 --> 0:14:35.800 one of the points about intuition is it's not very controllable. 0:14:35.880 --> 0:14:38.320 So I'm just wondering. I mean, you know, you and 0:14:38.480 --> 0:14:43.280 Olivier and and and behavioral economists may be very aware 0:14:43.360 --> 0:14:45.600 of all the biases and all the noise that you've 0:14:45.640 --> 0:14:48.520 just talked about. But when you're sitting interviewing a candidate 0:14:48.640 --> 0:14:52.720 or a potential colleague at university or whatever, how do 0:14:52.760 --> 0:14:56.920 you actually stop yourself from having a first impression of someone, 0:14:56.920 --> 0:15:00.640 Because by definition of first impression comes unbidden. Oh, you 0:15:00.760 --> 0:15:05.200 will undoubtedly form impresference, there is no question. But but 0:15:06.240 --> 0:15:09.120 if you have a set of questions that you want 0:15:09.160 --> 0:15:13.040 to ask about the person's reliability or about the extent 0:15:13.120 --> 0:15:17.520 of their experience on similar on a similar, unsimilar task, 0:15:18.360 --> 0:15:22.800 those specific questions are going to fail your mind, and 0:15:22.840 --> 0:15:26.240 they're going to push the intuition aside to some extent, 0:15:26.600 --> 0:15:30.080 and you will have an opportunity that you normally do 0:15:30.200 --> 0:15:35.280 not have of disconfirming your initial impression, of finding things 0:15:35.280 --> 0:15:39.960 out that actually do not fit the initial impression. In general, 0:15:40.040 --> 0:15:46.760 an unstructured interview, impressions are self reinforces. You justify your 0:15:46.800 --> 0:15:50.720 initial impression, and that is a source of noise, And 0:15:50.880 --> 0:15:55.720 by structuring the process you reduce that source of noise. 0:15:56.120 --> 0:15:59.600 An additional thing you can do to limit the problem 0:15:59.680 --> 0:16:04.400 that you're pointing out Stephanie is to have different people 0:16:04.800 --> 0:16:08.760 or different sources of information evaluate the different dimensions that 0:16:08.840 --> 0:16:12.280 you are looking at. So, if you're evaluating candidates for 0:16:13.080 --> 0:16:18.200 you know, intelligence, technical skills, and fit with the culture 0:16:18.240 --> 0:16:20.800 of the company, let's assume these are the three dimensions 0:16:20.800 --> 0:16:24.040 of your job description. In an unstructured interview, you would 0:16:24.440 --> 0:16:28.080 form an overall picture of the person and you would 0:16:28.440 --> 0:16:30.640 then raid them on the three dimensions, but they would 0:16:30.680 --> 0:16:33.640 be strongly correlated with each other because there would be 0:16:33.680 --> 0:16:36.320 a positive or a negative halo around the person, and 0:16:36.360 --> 0:16:38.960 you would say they are great and everything, or they're 0:16:38.960 --> 0:16:42.920 bad on everything. Now suppose that we say you, Stephanie, 0:16:42.960 --> 0:16:45.520 are going to conduct the interview about the technical skills, 0:16:45.960 --> 0:16:48.600 or maybe in fact, we're going to have a technical 0:16:48.600 --> 0:16:51.720 test to evaluate the technical skills. Someone else is going 0:16:51.760 --> 0:16:54.720 to evaluate the fit with the company, and someone else 0:16:54.800 --> 0:16:57.280 is going to evaluate how smart the person is, or again, 0:16:57.320 --> 0:17:00.440 perhaps we're going to have the test of how smart 0:17:00.480 --> 0:17:03.280 the person is. Now you've got three independent data points 0:17:03.280 --> 0:17:06.000 that do not influence each other, and you have a 0:17:06.080 --> 0:17:09.160 much more structured process to make your decision. You would 0:17:09.160 --> 0:17:10.720 have anounced this before, but of course, a lot of 0:17:10.760 --> 0:17:14.760 people entering the job market now find that they're at 0:17:14.840 --> 0:17:17.119 least the first couple of rounds, depending on how popular 0:17:17.160 --> 0:17:19.840 the job is is, and they're talking to a computer 0:17:20.160 --> 0:17:24.320 or they have there is a an AI element to 0:17:24.440 --> 0:17:28.000 their application process. We may not like it, we may 0:17:28.040 --> 0:17:31.200 think it's not true to our great sense of intuition 0:17:31.240 --> 0:17:35.040 about people. But from a fairness perspective and from a 0:17:35.040 --> 0:17:38.040 decision hygiene perspective, is that a better way to go? 0:17:38.640 --> 0:17:40.720 We need to be careful here, because there is an 0:17:40.720 --> 0:17:43.080 answer in principle, and there is an answer in practice. 0:17:43.960 --> 0:17:49.200 In principle, any form of structured decision making that reduces 0:17:49.280 --> 0:17:53.160 noise would in fact enhance the quality of the decisions. 0:17:53.160 --> 0:17:56.399 So if you have an algorithm making decisions, there is 0:17:56.440 --> 0:17:58.560 going to be less noise there. But of course the 0:17:58.680 --> 0:18:01.440 question is how good is the algorithm? How good are 0:18:01.520 --> 0:18:05.680 those AI systems that people sit in front of, And 0:18:06.480 --> 0:18:09.480 I'm sure there are good ones, but from the ones 0:18:09.560 --> 0:18:14.320 that I've seen personally in my admittedly limited experience, there 0:18:14.440 --> 0:18:19.280 isn't much evidence, and there isn't very good quality evidence 0:18:19.400 --> 0:18:24.840 that what these software packages are testing for is actually 0:18:25.040 --> 0:18:27.640 what you're looking for. It's actually quite hard for most 0:18:27.640 --> 0:18:29.840 companies to define what it is that they're looking for, 0:18:30.520 --> 0:18:34.200 and there is no evidence that I've seen that there's 0:18:34.320 --> 0:18:38.440 any correlation between when those software packages look for and 0:18:39.040 --> 0:18:45.720 job success is actually highly correlated. So in practice I'm 0:18:45.800 --> 0:18:50.639 quite skeptical about what I see in the market. In theory, 0:18:51.480 --> 0:18:53.280 I have to agree that it makes some sense, but 0:18:53.400 --> 0:18:55.879 my worry is that companies are using this mostly to 0:18:56.040 --> 0:18:59.280 save time and money, not to actually improve the quality 0:18:59.280 --> 0:19:03.359 of their decision. Here, I would hope that what is 0:19:03.400 --> 0:19:06.680 true in theory can be made true in practice. And 0:19:07.160 --> 0:19:13.720 one characteristics of algorithms and is that they're improvable. They're 0:19:13.800 --> 0:19:17.680 much more improvable than people are, and and they can 0:19:17.720 --> 0:19:22.320 be corrected by by data on quality. They can be 0:19:22.400 --> 0:19:26.560 made to predict more accurately. So this this is really 0:19:26.600 --> 0:19:32.200 an issue of the quality of constructing algorithms. And there 0:19:32.240 --> 0:19:36.320 are many algorithms that are of poor quality out there 0:19:36.320 --> 0:19:40.639 on the market, and there is a widespread suspicion of 0:19:40.720 --> 0:19:45.560 algorithms which makes us prone to reject them. But by 0:19:45.600 --> 0:19:49.920 and large, I think this is the future. In the future, 0:19:50.040 --> 0:19:53.040 there will be more and more of those algorithms, and 0:19:53.119 --> 0:19:57.640 their quality will be getting better and better every year 0:19:58.000 --> 0:20:01.040 because there will be data there will be feedback, and 0:20:01.080 --> 0:20:04.920 the feedback can be incorporated into an algorithm much more 0:20:04.960 --> 0:20:11.200 efficiently than it can be in the human judgment. So uh, 0:20:11.480 --> 0:20:16.960 here I join Olivia's skepticism about most of the algorithms 0:20:16.960 --> 0:20:20.000 that exist, but I really want to register and mode 0:20:20.000 --> 0:20:23.959 of optimism about the future of that kind of operation. 0:20:30.040 --> 0:20:33.760 There's one trend which is about eliminating the human element 0:20:33.960 --> 0:20:37.520 to some extent, or at least having it in a 0:20:37.520 --> 0:20:42.000 more regular form in an algorithm, a more consistent form structured. 0:20:42.840 --> 0:20:48.160 Of course, the other big trend in business strategy and 0:20:49.200 --> 0:20:53.800 conversations about companies is the move is encouragement of diversity 0:20:54.000 --> 0:20:57.159 and to encourage businesses in a sense to have a 0:20:57.200 --> 0:21:00.840 wider variety of humans doing the judgment. And I wonder 0:21:00.840 --> 0:21:03.359 whether that even goes against some of the things that 0:21:03.400 --> 0:21:04.960 you're talking about. You know, one of the ways that 0:21:05.000 --> 0:21:09.159 companies might have previously eliminated noise, not necessarily error, but noise, 0:21:09.960 --> 0:21:12.240 would have been having lots of identical people or making 0:21:12.240 --> 0:21:14.879 the decisions all of these white men sitting in their boards. 0:21:15.040 --> 0:21:18.159 If you now have a greatly much more diversity, you 0:21:18.240 --> 0:21:22.439 might be more true to the range of human experience, 0:21:22.560 --> 0:21:25.639 but you'll be getting a lot more noise. Well, that 0:21:25.840 --> 0:21:34.920 is certainly true, but in in principle, we we want 0:21:34.960 --> 0:21:39.359 to distinguish between the process of generating a judgment and 0:21:39.880 --> 0:21:44.080 the final judgment. In the process of generating a judgment, 0:21:44.400 --> 0:21:49.439 diversity is very welcome. That is, you want multiple points 0:21:49.440 --> 0:21:56.199 of view, you want people ptise to enter into the 0:21:56.600 --> 0:22:02.200 participate in the conversation. But when a final judgment is made, 0:22:02.560 --> 0:22:07.439 we want a process that reduces noises. So diversity is 0:22:07.640 --> 0:22:11.720 very useful, and you know it's it's you can think 0:22:11.760 --> 0:22:15.399 of that in terms of, say, witnesses to a crime. 0:22:16.240 --> 0:22:20.600 So you're better off if the witnesses are in different 0:22:20.640 --> 0:22:24.560 places and see the event from different perspectives. And you're 0:22:24.600 --> 0:22:28.520 certainly better off if if the witnesses don't talk to 0:22:28.560 --> 0:22:31.560 each other and they are independent of each other. And 0:22:31.640 --> 0:22:36.360 so thinking along those lines gives you an idea that 0:22:36.440 --> 0:22:41.040 you do want diversity, but you want also the kind 0:22:41.040 --> 0:22:45.800 of independence and the kind of goal directiveness that reduces 0:22:45.880 --> 0:22:49.919 noise in the final journey. Diversity in the outcome of 0:22:49.960 --> 0:22:52.560 these decisions, in the judgment that you produce in the end. 0:22:53.359 --> 0:22:55.719 It's good for some things, but for most it's not. 0:22:56.440 --> 0:22:58.560 When you when you go to the doctor and the 0:22:58.600 --> 0:23:00.840 doctor tells you, oh, you have is disease. And then 0:23:00.880 --> 0:23:02.600 you go to another doctor and he tells you you 0:23:02.640 --> 0:23:05.520 have that disease. You don't say, oh, that's wonderful, it's diversity. 0:23:05.560 --> 0:23:08.800 You say one of these two doctors is wrong, maybe both. 0:23:09.400 --> 0:23:12.320 So whenever we think that there is a correct answer, 0:23:12.960 --> 0:23:16.480 diversity in the outcome is not good. Maybe the way 0:23:16.480 --> 0:23:18.680 to get to the correct outcome is to harness the 0:23:18.720 --> 0:23:22.520 diversity of the perspectives or through multiple witnesses, and that 0:23:22.680 --> 0:23:24.359 is one of the remedies that you can have to 0:23:24.400 --> 0:23:28.080 reduce noise. But as an organization, what you're aiming for 0:23:28.600 --> 0:23:33.080 is not every person having their own opinion. It's any 0:23:33.160 --> 0:23:37.000 person having the best possible judgment. And we've talked about business, 0:23:37.000 --> 0:23:41.960 we've talked about justice or you know, sentencing and decisions 0:23:42.000 --> 0:23:45.920 within the criminal justice system, and an area that comes 0:23:46.000 --> 0:23:47.880 up a little bit in your book, but obviously it's 0:23:47.920 --> 0:23:49.800 kind of front and center of people's minds at the 0:23:49.840 --> 0:23:52.520 moment when we think of the decisions being taken around 0:23:52.560 --> 0:23:57.480 the war in in Ukraine. Is that in a critical 0:23:57.560 --> 0:24:04.399 moments of foreign policy or military strategy decisions, you know, 0:24:04.440 --> 0:24:08.840 you can't necessarily enlist a lot of people and listen 0:24:08.920 --> 0:24:12.800 to their structured answers on a set of questions in 0:24:12.840 --> 0:24:17.119 reaching your judgment about how to respond to Russia or 0:24:17.119 --> 0:24:19.600 how to how to respond to one of the sort 0:24:19.600 --> 0:24:22.879 of very pressing situations that can arise in foreign policy. 0:24:23.280 --> 0:24:26.000 So I just wonder whether you whether you'd reflected on that, 0:24:26.040 --> 0:24:27.920 you know, if you're Anthony Blink in the sector of state, 0:24:28.000 --> 0:24:31.919 or if you're President Biden, or or that matter, at 0:24:32.000 --> 0:24:36.600 a Russian general how what does decision hygiene look like 0:24:36.800 --> 0:24:39.639 in those kind of situations where there's inevitably going to 0:24:39.720 --> 0:24:42.200 be a limited number of people that you can call 0:24:42.240 --> 0:24:49.760 on and imperfect information hygiene is something that an individual 0:24:49.920 --> 0:24:54.359 can follow. That is, there are better and less good 0:24:54.600 --> 0:24:59.040 ways of individual judgments. You want to cover all the 0:24:59.040 --> 0:25:02.159 bases you are to think, You want to the extent 0:25:02.240 --> 0:25:05.600 possible to think of all possible consequences, you know, as 0:25:05.640 --> 0:25:09.280 salient example in the Ukraine War, is it looks unlikely 0:25:09.880 --> 0:25:14.879 that that people who started that war knew that Finland 0:25:14.880 --> 0:25:18.399 and Sweden would want to join NATO, because you know, 0:25:18.640 --> 0:25:22.040 after all, this was supposed to keep NATO away. So 0:25:22.240 --> 0:25:24.600 when I have the feeling that not all the bases 0:25:24.640 --> 0:25:30.920 were covered in making those critical, so there are all 0:25:31.000 --> 0:25:34.280 we can hope for is that we have people with 0:25:34.359 --> 0:25:37.680 a lot of experience, because it turns out that there 0:25:37.800 --> 0:25:42.240 is genuine intuitive experience that can develop over time with 0:25:42.400 --> 0:25:47.480 institution within certain kinds of decisions and choices. And we 0:25:47.560 --> 0:25:52.560 also want people who, even under sometime pressure, UH, can 0:25:52.680 --> 0:25:56.840 follow the basic dictates of decision hype. I've noticed a 0:25:56.920 --> 0:25:58.679 professor kind of and that a lot of interviews with 0:25:58.720 --> 0:26:01.240 you are quite long, a long than the sort of 0:26:01.280 --> 0:26:03.640 the norm for whatever program it is, And I suspect 0:26:03.640 --> 0:26:07.560 it's because it's so it's so it's always so fascinating 0:26:07.600 --> 0:26:08.960 to listen to you, and you always want to have 0:26:08.960 --> 0:26:12.040 another question. But we're going to run out of time. 0:26:12.320 --> 0:26:15.119 So thank you very much for coming on Stephonomics, and 0:26:15.160 --> 0:26:18.679 thanks thank you to Olivier Simbony, thank you, thank you 0:26:18.760 --> 0:26:26.800 very much. That's it for this episode of Stephonomics. We'll 0:26:26.840 --> 0:26:29.320 be back next week. In the meantime, do please rate 0:26:29.359 --> 0:26:31.199 the show if you like it, and check out the 0:26:31.200 --> 0:26:34.760 Bloomberg Terminal and News website for more economic news and 0:26:34.840 --> 0:26:37.520