WEBVTT - Why Doctors Use Evidence but Managers (Mostly) Don’t 0:00:15.356 --> 0:00:33.556 Pushkin. I'm Jacob Goldstein, and this is what's your problem. 0:00:33.796 --> 0:00:37.396 My guest today is Helen Pearson. She's a science journalist. 0:00:37.556 --> 0:00:40.196 She writes for the journal Nature, and she recently wrote 0:00:40.236 --> 0:00:44.836 a book called Beyond Belief, How Evidence Shows What Really works. 0:00:45.396 --> 0:00:48.556 Helen's problem is this, how do you get people to 0:00:48.636 --> 0:00:52.636 make important decisions based on high quality evidence rather than 0:00:52.996 --> 0:00:56.076 hunches or habits. To a large extent, the book is 0:00:56.076 --> 0:00:59.996 a history of what Helen calls the evidence revolution, and 0:01:00.036 --> 0:01:04.876 this is a remarkably recent revolution. In her telling, fifty 0:01:04.956 --> 0:01:08.596 years ago, she writes, doctors did not typically try to 0:01:08.676 --> 0:01:12.316 sort through through the high quality published studies to figure 0:01:12.316 --> 0:01:15.436 out what they should do. Instead, they often did whatever 0:01:15.436 --> 0:01:18.196 they'd been taught in medical school, or what the senior 0:01:18.236 --> 0:01:20.796 doctor in the room said to do, even if the 0:01:20.796 --> 0:01:24.996 evidence said otherwise. But a few doctors spent their careers 0:01:25.036 --> 0:01:28.076 trying to change that, sorting through all the evidence to 0:01:28.076 --> 0:01:31.276 figure out what worked best in different situations, and trying 0:01:31.276 --> 0:01:33.996 to get their fellow doctors to pay attention. In more 0:01:34.036 --> 0:01:36.956 recent decades, that spirit has started to spread to other fields, 0:01:37.156 --> 0:01:41.756 feels like management and crime prevention and education. Later in 0:01:41.796 --> 0:01:45.076 the conversation, Helen and I talk about some of those fields, 0:01:45.356 --> 0:01:48.876 places where the evidence is still often unclear, and where 0:01:48.916 --> 0:01:51.476 people often ignore the evidence even when it is clear. 0:01:52.076 --> 0:01:54.396 We also talk about how she uses evidence in her 0:01:54.396 --> 0:01:57.156 own life and about the prospects for the spread of 0:01:57.276 --> 0:02:01.236 evidence based decision making in the world. To start, we 0:02:01.316 --> 0:02:03.876 talk about one doctor who did a lot to get 0:02:03.876 --> 0:02:08.636 the evidence revolution going in medicine. His name was Ian Chalmers. 0:02:09.356 --> 0:02:13.156 Ian is really the reason I ended up writing the book. 0:02:13.436 --> 0:02:15.316 He was training to be a doctor in the kind 0:02:15.316 --> 0:02:18.596 of late nineteen sixties early nineteen seventies. He was working 0:02:18.636 --> 0:02:22.276 in obstetrics and gynecology in Wales at the time, and 0:02:22.316 --> 0:02:24.156 he noticed this sort of odd thing, which was that 0:02:24.236 --> 0:02:27.036 when doctors would hand out advice to women with the 0:02:27.036 --> 0:02:29.516 same condition, they would give different advice. So, you know, 0:02:29.556 --> 0:02:31.636 some doctors would think it was really important to have 0:02:31.876 --> 0:02:35.396 prenatal vitamins or not to drink during pregnancy, and other 0:02:35.476 --> 0:02:38.236 doctors would say no such thing. And this was kind 0:02:38.236 --> 0:02:41.196 of so extreme that Ian once told me that when 0:02:41.196 --> 0:02:43.596 he was woken in the night to treat a woman, 0:02:44.476 --> 0:02:46.476 the first thing he would say was what's wrong, and 0:02:46.516 --> 0:02:48.476 the second thing he would say was who's treating her? 0:02:48.916 --> 0:02:50.596 Because that was going to change. You know what he 0:02:50.636 --> 0:02:54.436 did next raight, So what generally happened was that everyone 0:02:55.156 --> 0:02:57.356 followed the advice of the most senior doctor in the room, 0:02:57.396 --> 0:03:00.356 which is sometimes now called eminence based medicine. 0:03:00.796 --> 0:03:05.196 Tell me about his work in Gaza and how that 0:03:05.196 --> 0:03:06.636 writer influenced him. 0:03:07.196 --> 0:03:10.276 Yeah, so that was another part of his train, which 0:03:10.316 --> 0:03:12.916 was also kind of influential in him realizing that a 0:03:12.956 --> 0:03:15.476 lot of what he'd been taught in medical school was wrong. 0:03:15.836 --> 0:03:19.836 So he worked on a refugee camp in Gaza during 0:03:19.836 --> 0:03:24.116 his medical training, and during that time he was treating 0:03:24.196 --> 0:03:28.316 children who had measles, and he had been taught in 0:03:28.356 --> 0:03:30.836 medical school that you shouldn't give antibiotics to a child 0:03:30.836 --> 0:03:34.876 with measles, even if they were becoming quite sick, because 0:03:34.956 --> 0:03:37.836 antibiotics treated bacterial infection, so why would you give it 0:03:37.876 --> 0:03:40.756 to a child with a viral infection. So he followed 0:03:40.756 --> 0:03:43.916 the advice he'd been taught, and he didn't give the antibiotics. 0:03:43.956 --> 0:03:47.116 But then later when he returned to the UK, he 0:03:47.236 --> 0:03:50.356 discovered that actually this advice he'd been given was tragically wrong, 0:03:50.756 --> 0:03:53.916 and that there were in fact about six control clinical 0:03:53.956 --> 0:03:57.516 trials that suggested it did help to give some children 0:03:57.516 --> 0:04:01.916 antibiotics prophylactically because they were developing a secondary bacterial infection 0:04:01.956 --> 0:04:03.036 along of the viral one. 0:04:03.196 --> 0:04:06.196 And in fact, as I understand it, like he concluded 0:04:06.236 --> 0:04:08.836 when he went back to the UK and studied the 0:04:08.836 --> 0:04:12.316 advice that children under his care had probably died or 0:04:12.356 --> 0:04:14.716 possibly died because he had done the wrong thing. 0:04:14.956 --> 0:04:15.876 That's exactly right. 0:04:16.076 --> 0:04:17.956 Yeah, So it's so a huge kind of you know, 0:04:18.156 --> 0:04:20.356 turning point for him of realizing that. 0:04:20.876 --> 0:04:24.916 So let me ask the native question, why had he 0:04:25.076 --> 0:04:28.476 been taught something that was not consistent with the evidence. 0:04:28.996 --> 0:04:33.276 Because that was that was standard practice in medical schools 0:04:33.316 --> 0:04:35.916 at the time. Most doctors weren't really being taught about 0:04:35.916 --> 0:04:37.996 clinical trials in the sixties and seventies. 0:04:38.196 --> 0:04:39.756 Hard though it is to believe. 0:04:39.876 --> 0:04:42.196 It is weird, right, it is hard to believe. I mean, 0:04:42.236 --> 0:04:44.476 I guess is part of it the sort of formal 0:04:44.996 --> 0:04:47.876 you know, randomized controlled trial. Well you have a placebo 0:04:48.036 --> 0:04:50.036 arm and a treatment arm, and that people running the 0:04:50.076 --> 0:04:53.436 trial don't know which is which. That was actually relatively 0:04:53.556 --> 0:04:54.436 new still. 0:04:54.276 --> 0:04:57.596 Right, exactly, Yes, So we feel like randomized trials have 0:04:57.676 --> 0:05:00.276 been around for ages, but they are relatively new. I 0:05:00.276 --> 0:05:02.236 mean around the time that you know that Ian was 0:05:02.276 --> 0:05:04.396 having the you know that this realization that he hadn't 0:05:04.436 --> 0:05:08.396 been taught about trials. They were really only then becoming commonplace. 0:05:09.116 --> 0:05:14.076 Oh so you have this sort of research changing and 0:05:14.116 --> 0:05:18.436 the practices of doctors are lagging. That is sort of 0:05:18.476 --> 0:05:21.556 the moment when Ian Chalmers walks onto the scene. And 0:05:21.756 --> 0:05:25.156 it's this moment when there is now research but doctors, 0:05:25.836 --> 0:05:28.796 weirdly to our sensibilities, are not paying attention to it. 0:05:29.516 --> 0:05:31.916 What does Chalmers do when he realizes this? 0:05:33.516 --> 0:05:39.036 So what he did was he became very passionate about 0:05:39.156 --> 0:05:42.916 randomized trials. Actually, and I discovered this with other people 0:05:43.116 --> 0:05:45.196 I spoke to, is that there's something about this method 0:05:45.276 --> 0:05:46.996 which is very captivating to people. 0:05:47.436 --> 0:05:51.596 It's so elegant, right, like intellectually, like the it's blinded, 0:05:51.636 --> 0:05:55.396 there's a placebo control. Like I recognize that they're imperfect, 0:05:55.436 --> 0:05:57.156 and people love to talk about the problems with them, 0:05:57.436 --> 0:06:02.196 but it's wildly elegant and not intuitive. Right People want 0:06:02.236 --> 0:06:04.156 to just like give something to somebody and see if 0:06:04.196 --> 0:06:06.396 they get better, and say it works if they get better, 0:06:06.476 --> 0:06:09.116 and like, for so many reasons, a randomized trial is 0:06:09.196 --> 0:06:09.996 so much better than that. 0:06:10.996 --> 0:06:13.356 And that argument goes on all the time today. I mean, 0:06:13.356 --> 0:06:16.396 that's what mostly happens in policy is we just introduce 0:06:16.476 --> 0:06:19.396 policies right without bothering to test whether they're effective and 0:06:19.436 --> 0:06:21.916 assuming they do good when some do harm. So that 0:06:21.916 --> 0:06:23.276 I don't think that's an old argument. 0:06:23.596 --> 0:06:25.956 We will get to policy. Yes. So this is this 0:06:26.036 --> 0:06:28.676 moment people are falling in love with randomized trials, and 0:06:28.676 --> 0:06:33.916 more importantly for our story, Ian Chalmers is realizing that 0:06:33.956 --> 0:06:36.836 this evidence is not making its way into medical practice. 0:06:36.876 --> 0:06:37.956 What does he do about that? 0:06:38.796 --> 0:06:43.236 So he decides to do this pioneering exercise where he 0:06:43.476 --> 0:06:47.956 is going to collect all of the controlled clinical trials. 0:06:47.996 --> 0:06:50.716 They weren't all randomized, but controlled trials which had been done, 0:06:50.756 --> 0:06:54.876 which had tested treatments in pregnancy and childbirth. Okay, and 0:06:55.196 --> 0:06:57.716 at the time, so this is in the eighties, I mean, 0:06:57.716 --> 0:07:01.036 that was an extraordinarily difficult thing to do because he 0:07:01.036 --> 0:07:04.036 couldn't just go online and you know, search up these trials. 0:07:04.876 --> 0:07:08.836 So he eventually ended up recruiting hundreds of people to 0:07:08.916 --> 0:07:11.196 sort of go to libraries and scour through the medical 0:07:11.236 --> 0:07:14.836 journals and really exhaustively try and track down these clinical trials. 0:07:15.116 --> 0:07:19.156 It's amazing that it's almost an archaeological exercise, right, like 0:07:19.516 --> 0:07:23.636 people have generated all this evidence not in ancient times, 0:07:23.676 --> 0:07:25.276 but in the last ten or twenty years, and it's 0:07:25.276 --> 0:07:28.716 already been sort of lost. Right. He is an obgyn, right, 0:07:28.756 --> 0:07:30.956 and he's having to have people will look for the 0:07:30.996 --> 0:07:33.796 research in his field exactly exactly. 0:07:33.836 --> 0:07:36.916 It's extraordinary and that is still a real difficult problem today, 0:07:36.956 --> 0:07:39.276 which I think is absolutely mind boggling, because a lot 0:07:39.316 --> 0:07:41.796 of research is published and yet people still struggle to 0:07:41.836 --> 0:07:43.036 find it on databases. 0:07:43.396 --> 0:07:47.796 So they're doing this archaeological expedition to look for recent research. 0:07:47.796 --> 0:07:48.436 And what happened. 0:07:49.756 --> 0:07:52.276 So it takes him about ten years, him and his 0:07:52.356 --> 0:07:57.076 many colleagues, and eventually he doesn't just get the evidence together. 0:07:57.116 --> 0:07:59.276 I think this is what's so interesting about it is 0:07:59.676 --> 0:08:04.396 he realizes that he needs to synthesize the evidence, and 0:08:04.636 --> 0:08:06.516 you know, not just to look at one trial at 0:08:06.556 --> 0:08:08.676 a time, because if you look at one trial, it 0:08:08.796 --> 0:08:11.236 risks being wrong, or you get a group of trials 0:08:11.236 --> 0:08:14.076 which may be conflicting, maybe some of them are poorly done, 0:08:14.716 --> 0:08:17.236 maybe it's just through chance that they point in different directions. 0:08:17.236 --> 0:08:21.156 And so he took this quite radical step of synthesizing 0:08:21.156 --> 0:08:24.916 the evidence using a method called a systematic review and 0:08:25.716 --> 0:08:28.716 ended up doing hundreds of systematic reviews to look at 0:08:28.756 --> 0:08:33.036 all of these different approaches in pregnancy and childbirth and 0:08:33.116 --> 0:08:35.556 eventually publishes all of this after ten years in these 0:08:35.556 --> 0:08:37.316 two massive books. 0:08:36.996 --> 0:08:39.396 Which I also have on my book case following me. 0:08:40.436 --> 0:08:44.436 So, they revealed that a lot of practices in pregnancy 0:08:44.436 --> 0:08:47.756 and childbirth were not based on evidence or were actually 0:08:47.756 --> 0:08:53.356 doing harm. So for example, even things like invasive episiotomies, 0:08:53.396 --> 0:08:56.436 which is a surgical incision which is supposed to give 0:08:56.476 --> 0:08:59.236 babies a bigger hole to get out, there wasn't really 0:08:59.236 --> 0:09:03.436 evidence to support these practices, and like shaving women's pubic 0:09:03.436 --> 0:09:06.596 care in labor. So countless women had been subjected to 0:09:06.676 --> 0:09:10.356 these really quite invasive and degrading practices because doctors assumed 0:09:10.396 --> 0:09:13.796 they were the right thing to do. And Ian's work 0:09:14.236 --> 0:09:17.436 by synthesizing evidence in this way helped bring an end 0:09:17.516 --> 0:09:20.436 to them. So he was sort of inspired by what 0:09:20.476 --> 0:09:23.156 he had done and thought, why can't we do this 0:09:23.236 --> 0:09:27.556 across all specialties of medicine, you know, synthesize the evidence 0:09:27.596 --> 0:09:29.796 in heart disease and diabetes, in cancer and so on, 0:09:29.996 --> 0:09:33.636 And that led to this group which he helped found. 0:09:33.676 --> 0:09:36.676 He was seminal in founding called the Cochrane Collaboration, and 0:09:36.756 --> 0:09:38.676 they set out and are still trying to do this 0:09:38.716 --> 0:09:43.236 today to synthesize evidence which shows which treatments are of 0:09:43.396 --> 0:09:46.396 help and do not harm across these different groups. And 0:09:46.716 --> 0:09:49.996 nowadays the word sort of like a Cochrane review if 0:09:49.996 --> 0:09:53.396 you're here that it's sort of like associated with doing 0:09:53.436 --> 0:09:56.436 the kind of highest standard, you know, gold standard type 0:09:56.476 --> 0:09:57.836 of review of evidence. 0:09:58.836 --> 0:10:00.836 And I would say a lot of it at least 0:10:01.036 --> 0:10:04.196 is just online, right, like you can go look if 0:10:04.236 --> 0:10:08.436 you're thinking of taking a supplement, say, you can just 0:10:08.516 --> 0:10:11.996 go look. And there are a lot of Cochrane reviews 0:10:12.156 --> 0:10:18.276 of whatever, magnesium for something something, and I find it useful. 0:10:18.276 --> 0:10:21.436 I mean quite often they say there's not enough evidence 0:10:21.436 --> 0:10:23.876 to say one way or the other. Right, that's typically 0:10:23.916 --> 0:10:26.116 with the kinds of things that I'm looking at, you know, 0:10:26.156 --> 0:10:29.316 over the counter things or whatever. There just isn't enough 0:10:29.356 --> 0:10:31.756 research to have a clear answer. And that is what 0:10:31.796 --> 0:10:34.036 they often say. It makes sense, Yes. 0:10:33.756 --> 0:10:34.956 It is often the conclusion. 0:10:34.996 --> 0:10:37.356 And I mean it's not just so Cochran reviews now 0:10:37.396 --> 0:10:39.836 are a tiny fraction of the number of systematic reviews 0:10:39.836 --> 0:10:41.836 in the world. So Cochran was really sort of seminal 0:10:41.876 --> 0:10:44.156 in sort of showing that this was a really good 0:10:44.236 --> 0:10:47.796 way to make sense of conflicting bodies of evidence. But 0:10:47.956 --> 0:10:50.276 now that you know, thousands are coming out every year, 0:10:50.396 --> 0:10:53.876 So in general, a systematic review is a better way 0:10:53.956 --> 0:10:55.676 to get an overview of what the evidence says than 0:10:55.716 --> 0:10:57.996 looking at just one study. Now, of course, it's it's 0:10:58.036 --> 0:11:01.716 not completely straightforward because like randomized trials, some systematic reviews 0:11:01.716 --> 0:11:05.156 aren't very good either, So that's a challenge too. 0:11:05.276 --> 0:11:08.636 Well, it's always something yes, well, I mean that's Cochrane 0:11:08.676 --> 0:11:11.476 is useful as a brand in that way, right, Like 0:11:11.596 --> 0:11:15.436 I trust their ability to assess the evidence, even if 0:11:15.476 --> 0:11:17.396 what they say is we don't know, Like we don't know. 0:11:17.396 --> 0:11:22.076 It's a very respectable answer as a general matter. What's 0:11:22.116 --> 0:11:23.116 a Charmer's like. 0:11:26.116 --> 0:11:31.236 Quite extraordinary actually just the most I don't think I've 0:11:31.276 --> 0:11:34.836 ever met anyone who is so extraordinarily modest about his 0:11:34.876 --> 0:11:37.396 accomplishments and downplays them so much, to the point where 0:11:37.396 --> 0:11:40.956 he denies sort of starting the cock con collaboration. I 0:11:40.996 --> 0:11:42.956 think he says something along the lines of he came 0:11:43.036 --> 0:11:49.756 up with the idea so and also very exacting and critical. 0:11:49.796 --> 0:11:52.036 I mean it was interesting when he retired or decided 0:11:52.076 --> 0:11:54.076 to step away. He didn't retire from Cockraney decided he 0:11:54.116 --> 0:11:56.156 wanted to move on. He actually set up a prize 0:11:56.556 --> 0:11:59.396 for the group that offered the most astute criticism of 0:11:59.476 --> 0:12:00.476 the cock Con collaboration. 0:12:00.796 --> 0:12:02.156 Oh that's really admirable. 0:12:02.396 --> 0:12:03.076 Yes, exactly. 0:12:03.076 --> 0:12:05.796 And they still have that attitude of kind of inviting 0:12:05.836 --> 0:12:08.636 criticism because of course that's exactly what you know, he 0:12:09.116 --> 0:12:10.876 and the group have done all these years is sort 0:12:10.916 --> 0:12:13.556 of speak truth to power. So they need to invite 0:12:13.596 --> 0:12:15.276 that criticism themselves. 0:12:15.996 --> 0:12:18.516 Who won the prize and what did they say? 0:12:19.796 --> 0:12:21.996 I don't know, it's one every years. 0:12:22.716 --> 0:12:25.516 Yeah, so I could imagine him having a pretty high 0:12:25.516 --> 0:12:30.236 tolerance for disagreeableness. Say right, I mean what he did 0:12:30.996 --> 0:12:34.116 was in a sort of cultural way in terms of 0:12:34.156 --> 0:12:39.596 the culture of physicians, the culture of medicine. I don't know, 0:12:39.916 --> 0:12:42.996 antagonistic is maybe too strong a word, but he said 0:12:42.996 --> 0:12:45.196 things that presumably lots of people didn't want to hear. 0:12:45.396 --> 0:12:47.476 He said, to doctors, you're doing it wrong, both in 0:12:47.516 --> 0:12:50.436 a specific way, you shouldn't be doing this particular intervention, 0:12:50.676 --> 0:12:53.436 and also in a general way, like he implicitly or 0:12:53.436 --> 0:12:56.756 explicitly was saying you're thinking about the practice of medicine wrong. 0:12:57.476 --> 0:13:00.516 That's right, that's exactly right. I discovered this as I 0:13:00.556 --> 0:13:03.716 sort of was reporting the book and kind of speaking 0:13:03.716 --> 0:13:07.436 to these people who were quite seminal in bringing evidence 0:13:07.476 --> 0:13:09.556 based practice into their fields. They often have this sort 0:13:09.596 --> 0:13:12.876 of rebellious streak to them and are kind of willing 0:13:12.956 --> 0:13:14.956 to speak truth to power, because yes, it does. I mean, 0:13:14.956 --> 0:13:17.716 you're basically saying to people, you know, ort to doctors, 0:13:17.756 --> 0:13:20.156 you've been doing it wrong all these years. And he 0:13:21.276 --> 0:13:23.436 himself said he thought he was able to do that 0:13:23.596 --> 0:13:25.996 because he was sort of he was a bit of 0:13:25.996 --> 0:13:29.516 an outsider actually, Like he worked in Oxford, but he 0:13:29.596 --> 0:13:31.516 wasn't sort of part of all the kind of you know, 0:13:31.956 --> 0:13:35.156 high for lutin academic circles. He was a bit on 0:13:35.196 --> 0:13:37.516 the outside and he felt that that allowed him to 0:13:37.596 --> 0:13:41.956 maybe do this speaking of truth to power, which maybe 0:13:42.036 --> 0:13:43.196 others what unable to do. 0:13:43.756 --> 0:13:47.556 So where's medicine today? To what extent is medicine today 0:13:47.556 --> 0:13:48.316 evidence based? 0:13:49.476 --> 0:13:53.956 Well more than it was then. I think I used 0:13:54.276 --> 0:13:56.116 a figure in my book of about sort of sixty 0:13:56.156 --> 0:14:02.076 percent of healthcare is probably evidence based or at least 0:14:02.156 --> 0:14:05.636 draws on evidence based clinical guidelines. So nowadays what happens 0:14:05.716 --> 0:14:10.356 is there are typically systematic reviews of clinical trials or 0:14:10.396 --> 0:14:12.716 are the evidence, and on the back of that, then 0:14:13.596 --> 0:14:16.796 professional groups prepare clinical guidelines and the clinical guidelines of 0:14:16.876 --> 0:14:18.916 what might be used in the clinic to help actually 0:14:18.916 --> 0:14:19.996 make a decision. 0:14:19.876 --> 0:14:23.636 Right, so, like the National Association of Cardiologists will have 0:14:23.796 --> 0:14:26.996 guidelines on whatever who should get statins or who should 0:14:26.996 --> 0:14:28.556 get what interventions exactly. 0:14:28.596 --> 0:14:30.756 And that, I mean, that's quite interesting because it means 0:14:30.796 --> 0:14:33.516 that the evidence is kind of baked into the processes, 0:14:33.876 --> 0:14:36.116 which which I find quite interesting. It's sort of made it. 0:14:36.236 --> 0:14:37.836 I mean, I won't say it's easy, but it is 0:14:37.916 --> 0:14:40.516 relatively easy for doctors now to use evidence because that's 0:14:40.516 --> 0:14:42.596 the expectation. Whereas I'm sure we'll get onto this, but 0:14:42.636 --> 0:14:44.596 you look at other fields and that's just not the case. 0:14:45.916 --> 0:14:47.876 But of course there are huge challenges, you know, lots 0:14:47.916 --> 0:14:51.636 of criticism still of evidence based medicine. Some people criticize, 0:14:51.676 --> 0:14:54.476 for example, you know, this sort of special place that 0:14:54.516 --> 0:14:57.276 the randomized control trial has and point out that randomized 0:14:57.316 --> 0:15:00.516 control trials can be wrong or poorly done. And then 0:15:00.876 --> 0:15:03.796 of course, you know, pushback in the world against evidence 0:15:03.836 --> 0:15:07.116 based practices. Anyway, and people wanting to go their own way. So, 0:15:07.756 --> 0:15:10.756 in no way is the kind of evidence revolution that 0:15:10.796 --> 0:15:12.756 I write about, you know, obviously done. 0:15:12.836 --> 0:15:15.276 It's evolving, it's new in some way. 0:15:16.316 --> 0:15:18.836 So if you say it's sixty percent evidence based for 0:15:18.876 --> 0:15:22.076 the other forty, I could imagine two things. Right Clearly, 0:15:22.076 --> 0:15:24.796 there are a lot of settings, Like medicine is complicated. 0:15:24.796 --> 0:15:27.356 People are complicated, the body is complicated, lots of sick 0:15:27.356 --> 0:15:29.396 people have lots of things wrong with them. So you 0:15:29.396 --> 0:15:32.076 can imagine lots of cases where there just isn't evidence, 0:15:32.116 --> 0:15:33.756 where it's just not clear what should I do for 0:15:33.796 --> 0:15:36.636 this patient right now? Well, there's lots of different conflicting 0:15:36.676 --> 0:15:38.876 things that I could do in their trade offs. The 0:15:38.956 --> 0:15:41.996 other piece is there is evidence and people just don't 0:15:42.036 --> 0:15:45.156 follow it. So like how much of each of those 0:15:45.236 --> 0:15:45.596 is there? 0:15:46.036 --> 0:15:47.756 Yeah, so I can't give you a percentage on each 0:15:47.836 --> 0:15:49.356 of those. But I mean, just to speak to the 0:15:49.356 --> 0:15:52.316 first one though, I mean, what's really interesting about evidence 0:15:52.316 --> 0:15:55.756 based medicine is that evidence alone doesn't tell you what 0:15:55.876 --> 0:15:59.116 to do. So evidence is one part of the decision 0:15:59.116 --> 0:16:02.236 making process, and I think that's a common misunderstanding actually 0:16:02.316 --> 0:16:04.036 around it is that like you look at the evidence 0:16:04.076 --> 0:16:05.836 and it will tell you, you know, whether you should take 0:16:05.876 --> 0:16:08.036 this drug. But that's not the case because what what 0:16:08.156 --> 0:16:10.276 a doctor or who whoever is you know, a doctor, 0:16:10.316 --> 0:16:12.956 hopefully impatient together are doing is saying, Okay, you know, 0:16:12.996 --> 0:16:15.756 this is the evidence on what treatments are effective. This 0:16:15.836 --> 0:16:18.876 is the clinical expertise of the doctor who absolutely you know, 0:16:18.916 --> 0:16:21.596 they've built up all this expertise over the years. We're 0:16:21.596 --> 0:16:23.676 not throwing that out the window. And then you as 0:16:23.716 --> 0:16:26.596 a patient have values and preferences which have to feed 0:16:26.636 --> 0:16:29.236 into this decision. Right, if you've decided this isn't a 0:16:29.276 --> 0:16:31.676 treatment path you want, then that has to be respected. So, 0:16:32.076 --> 0:16:33.476 but that's still evidence based. 0:16:33.316 --> 0:16:37.276 Medicine, and that's still in your sixty percent easy one. Yeah, yeah, 0:16:37.356 --> 0:16:39.836 Like tell me about the forty percent that's not evidence based. 0:16:39.836 --> 0:16:41.076 What is what's going on there? 0:16:41.676 --> 0:16:45.436 Well, in some places, evidence based medicine is less common 0:16:45.516 --> 0:16:50.116 or accepted, so people might choose traditional Chinese medicine, for example, 0:16:50.396 --> 0:16:53.716 So there may be other practices around medicine which people prefer. 0:16:54.396 --> 0:16:57.676 In some cases, You're absolutely right, the evidence isn't really 0:16:58.596 --> 0:17:01.716 there or it might be misapplied. So there's the kind 0:17:01.756 --> 0:17:04.356 of all the human factors that come into it where 0:17:04.356 --> 0:17:09.116 people may do choice or inadvertently not follow the evidence. 0:17:09.196 --> 0:17:15.316 Ice Pak. 0:17:13.956 --> 0:17:28.356 Will be back in just a minute. Okay, So let's 0:17:28.396 --> 0:17:31.516 talk about evidence outside of medicine. A big chunk of 0:17:31.556 --> 0:17:35.276 your book is that, And it does seem to some 0:17:35.356 --> 0:17:39.876 extent at least, that this evidence based revolution, as you 0:17:39.916 --> 0:17:43.356 call it, that Ian Chalmers helped us start, has directly 0:17:43.436 --> 0:17:47.796 influenced people in other fields. You know, and you write 0:17:47.836 --> 0:17:54.916 about economics and crime and management and parenting, and you know, 0:17:55.116 --> 0:17:58.116 I will say, like I covered economics for a long time, 0:17:58.236 --> 0:18:01.916 and in economics they've gotten better about evidence. It seems 0:18:01.956 --> 0:18:04.476 to me that in a lot of fields outside of medicine, 0:18:05.236 --> 0:18:08.116 it's just frustrating. It's just frustrating when you sort of 0:18:08.156 --> 0:18:10.316 try and study the evidence. It seems like there's often 0:18:10.396 --> 0:18:13.116 not good evidence. When there is, people don't follow the evidence. 0:18:13.196 --> 0:18:16.116 I mean, in a broad way, what's your assessment of 0:18:16.156 --> 0:18:19.116 how evidence as a way of looking at the world 0:18:19.236 --> 0:18:20.716 is doing outside of medicine. 0:18:21.876 --> 0:18:24.596 So I think it really depends which field you're looking 0:18:24.636 --> 0:18:28.316 in of how like I suppose I ended up thinking 0:18:28.316 --> 0:18:31.116 that each of these fields was, you know, weaving its 0:18:31.156 --> 0:18:33.356 own path through evidence, and some of them had directly 0:18:33.356 --> 0:18:34.996 looked at medicine and gone, yeah, we want to do 0:18:35.036 --> 0:18:38.116 that too. We should think our practices shouldn't be based 0:18:38.156 --> 0:18:42.316 on conventional wisdom an anecdote. And in others, like in 0:18:42.356 --> 0:18:45.316 economics example, there's kind of you know, evidences that the 0:18:45.396 --> 0:18:48.156 use of randomized trials has really swept through international development, 0:18:48.276 --> 0:18:51.676 through development economics. But yeah, I think I don't think 0:18:51.676 --> 0:18:54.676 any of them have advanced as far as medicine. And 0:18:54.716 --> 0:19:00.156 I think that that is partly because of the histories 0:19:00.196 --> 0:19:03.196 of the of these fields and the people who've been 0:19:03.236 --> 0:19:05.516 championing it, and the different barriers that it's faced. 0:19:05.876 --> 0:19:12.236 It seems harder in most other domains to do useful 0:19:12.396 --> 0:19:16.036 you know, studies that have real external validity where you 0:19:16.076 --> 0:19:18.756 can look at something in one place and say this 0:19:18.916 --> 0:19:20.916 same intervention is going to work in other places, Like 0:19:20.956 --> 0:19:24.516 the world is messy, is a fundamental problem with trying 0:19:24.556 --> 0:19:25.476 to generate evidence. 0:19:25.756 --> 0:19:28.356 Yeah, it makes things really really hard. So let's say 0:19:28.356 --> 0:19:31.596 you want to, I don't know, you know, test whether 0:19:31.956 --> 0:19:35.316 giving people incentives to have vaccines works. And I mean 0:19:35.356 --> 0:19:37.836 I talked about a trial in the book that was 0:19:37.876 --> 0:19:42.156 done where in India people were offered steel dinner plates 0:19:42.276 --> 0:19:44.996 and bags of lentils in order to incentivize parents to 0:19:45.036 --> 0:19:47.756 compliclents had their children vaccinated, and it seemed to be effective. 0:19:47.836 --> 0:19:49.996 Right now, you can't just lift and shift that and 0:19:50.036 --> 0:19:52.716 put it into you know, the UK. You know, you 0:19:52.756 --> 0:19:55.396 have to think about the different cultures and you know, 0:19:55.396 --> 0:19:57.596 the different ways incentives might work, and so on and 0:19:57.596 --> 0:19:57.996 so forth. 0:19:58.036 --> 0:20:01.836 I mean that particular example, like you do see conditional 0:20:01.836 --> 0:20:04.836 cash transfers in other countries where you know, if parents 0:20:04.836 --> 0:20:07.796 bring their children for vaccination and send them to school, 0:20:07.836 --> 0:20:10.916 then they get a monthly stipend from the government. Like 0:20:11.876 --> 0:20:14.396 those seem to work, Like those that one does seem 0:20:14.436 --> 0:20:15.996 to generalize at least. 0:20:15.876 --> 0:20:18.196 Some yes, it has. I mean that's built up. 0:20:18.356 --> 0:20:20.556 That's unusual really and having such a body of evidence 0:20:20.596 --> 0:20:21.956 to it, that's built up over time. 0:20:22.156 --> 0:20:24.916 Although it's funny like that one, I understand that we 0:20:24.916 --> 0:20:27.356 shouldn't rely on common sense is one of the themes 0:20:27.356 --> 0:20:29.556 of the book. But like, if you give people money 0:20:29.596 --> 0:20:31.876 to do something that is actually helpful to them, they 0:20:31.876 --> 0:20:34.956 will do it. Is basically what that finding says, and 0:20:35.036 --> 0:20:36.196 it's true. 0:20:36.236 --> 0:20:40.076 Right, but it took evidence to convince policymakers of that. Yeah, fair, 0:20:40.516 --> 0:20:43.396 So the counter argument, you know, You're absolutely right that 0:20:43.436 --> 0:20:46.556 this point about generalization is like a really big problem 0:20:46.596 --> 0:20:48.556 across all of these fields, right that what works in 0:20:48.596 --> 0:20:51.116 one place might not work in another. Now, some people 0:20:51.116 --> 0:20:53.796 would say, well why bother? You know, well why bother 0:20:53.916 --> 0:20:56.236 getting evidence at all? Like what's the point of testing it? 0:20:56.716 --> 0:21:01.156 But you generally, over time, you know, if something it 0:21:01.236 --> 0:21:03.676 might not work exactly the same. But if you start 0:21:03.716 --> 0:21:05.516 to build up a body of evidence that shows, yeah, 0:21:05.556 --> 0:21:08.436 actually this seems to like the conditional cash transfers have 0:21:08.636 --> 0:21:11.236