WEBVTT - There's a Problem with Your Study 0:00:03.120 --> 0:00:06.000 Welcome to Stuff to Blow your Mind from how Stuff 0:00:06.000 --> 0:00:14.000 Works dot com. Hey, welcome to Stuff to Blow your Mind. 0:00:14.040 --> 0:00:16.600 My name is Robert Lamb, and I'm Julie Douglas. You know, Julie. 0:00:16.640 --> 0:00:19.079 Science has done some good things for us. It's done 0:00:19.079 --> 0:00:21.239 a lot of good things. I mean, it has really 0:00:21.600 --> 0:00:26.439 forwarded humanity right and made us, in some ways the 0:00:26.880 --> 0:00:30.080 kind of success story of the species that we are. Yeah, 0:00:30.120 --> 0:00:32.599 it's kind of the skeleton of human culture, that the 0:00:32.600 --> 0:00:35.599 thing upon which we grow and continue to grow. And 0:00:35.720 --> 0:00:37.160 you know, and you can look at just about any 0:00:37.159 --> 0:00:41.960 area right, medical science, exploration of inner and outer space, um, 0:00:42.080 --> 0:00:45.479 increasing knowledge of the self, the brain, the connection, our 0:00:45.520 --> 0:00:47.960 connection to the from the brain to the body. I mean, 0:00:48.000 --> 0:00:50.800 pretty much everything we talk about every week is is 0:00:50.800 --> 0:00:54.040 a testament to what science is doing and has done 0:00:54.120 --> 0:00:58.200 for humans. And while the pursuit of science, what we 0:00:58.240 --> 0:01:01.600 think about as science, has been around for a very 0:01:01.640 --> 0:01:04.839 long time, this pursuit of knowledge and truth, the word 0:01:04.920 --> 0:01:09.120 scientists is only one hundred and eighty years old. Before that, 0:01:09.160 --> 0:01:12.560 a person might be called a natural philosopher. And before 0:01:12.560 --> 0:01:16.800 that you had economists, you had philosophers, and what we 0:01:16.880 --> 0:01:21.360 now call scientists. All co mingling under the same roof, 0:01:21.680 --> 0:01:25.000 and this affected how science and and how we think 0:01:25.040 --> 0:01:28.200 of it was defined and pursued. And we sort of 0:01:28.200 --> 0:01:31.720 assumed that science and the scientific method were in place 0:01:31.760 --> 0:01:34.839 from the get go, but in fact they hadn't really 0:01:34.920 --> 0:01:39.000 been defined in the rules tightened, uh, you know, until 0:01:39.000 --> 0:01:43.120 a couple of hundred years ago, because economists were pulling 0:01:43.160 --> 0:01:49.360 for deductive reasoning, right, and scientists we're saying, no, I 0:01:49.400 --> 0:01:53.440 think there's there's more of this inductive reasoning, which is 0:01:53.480 --> 0:01:56.640 this premise that you you take an idea and then 0:01:56.680 --> 0:01:58.920 you try to take it down to the studs and 0:01:58.960 --> 0:02:00.760 prove it wrong, even theo you might want it to 0:02:00.760 --> 0:02:04.160 be right. And the whole idea there is that you're 0:02:04.160 --> 0:02:06.800 trying to get at this kind of truth. And this 0:02:06.880 --> 0:02:09.720 is now something called the scientific method. But we sort 0:02:09.720 --> 0:02:12.520 of take this for granted, this this fact that this 0:02:12.600 --> 0:02:16.280 is only a fairly recent development in the long history 0:02:16.280 --> 0:02:19.480 of humans. Yeah, I mean, it's it's basically how science works. 0:02:19.800 --> 0:02:22.400 There were people who managed to make it work in 0:02:22.400 --> 0:02:24.600 the past, but it was until recently that we actually 0:02:24.680 --> 0:02:26.560 said this is what works, and this is why we 0:02:26.560 --> 0:02:31.359 should stick to. Now. A lot of the advances in 0:02:32.360 --> 0:02:35.639 century you can, you can boil down to a simple 0:02:35.680 --> 0:02:39.040 idea trust, but very verify. And this plays into our 0:02:39.280 --> 0:02:44.120 peer view system. Right, one scientist writes a paper, or 0:02:44.120 --> 0:02:46.160 a team of scientists write a paper, maybe there's a 0:02:46.200 --> 0:02:49.160 big breakthrough in it, maybe not. But then the idea 0:02:49.320 --> 0:02:51.800 is that their peers come along, look at the paper 0:02:52.040 --> 0:02:56.320 and try to replicate the results, just you know, tear 0:02:56.360 --> 0:02:59.119 it apart, see what's happening in the paper and say, yes, 0:02:59.200 --> 0:03:01.520 I agree this is working, or I have problems with 0:03:01.560 --> 0:03:05.200 this or that, or this is complete bunk. Yeah. I mean, 0:03:05.200 --> 0:03:08.880 it's this idea that science can police itself. And yet 0:03:09.440 --> 0:03:14.120 we have some statistics coming out that point to other 0:03:14.200 --> 0:03:17.960 factors going on and that perhaps we're not pursuing knowledge 0:03:17.960 --> 0:03:21.600 for knowledge itself in some cases or truth and and 0:03:21.960 --> 0:03:24.680 we'll discuss more of those factors in a bit. According 0:03:24.680 --> 0:03:27.519 to the Economists article How Science Goes Wrong, in two 0:03:27.520 --> 0:03:31.160 thousand twelve, biotech firm am Jin reported that they could 0:03:31.240 --> 0:03:35.920 reproduce just six of fifty three landmark studies in cancer research, 0:03:36.560 --> 0:03:39.960 and earlier Bear the drug company, managed to repeat just 0:03:40.040 --> 0:03:44.360 a quarter of sixty seven similarly important papers. Now we're 0:03:44.360 --> 0:03:47.400 not taking on this topic today because we think that 0:03:47.440 --> 0:03:49.800 we are experts on this topic, but by no stretch 0:03:49.880 --> 0:03:52.480 of imagination are we. But we do rely on a 0:03:52.520 --> 0:03:54.680 lot of studies, and so we wanted to point this 0:03:54.800 --> 0:03:58.240 out today to just for ourselves better understand what are 0:03:58.280 --> 0:04:02.760 the conditions that lead to a good, solid study or experiment. 0:04:02.920 --> 0:04:06.400 What are the conditions that lead to dubious data? Yeah, 0:04:06.440 --> 0:04:08.800 certainly worth keeping in mind too when you find yourself 0:04:09.000 --> 0:04:13.320 reading science journalism articles, you know, uh that asking yourself, well, 0:04:13.360 --> 0:04:16.320 what is the study? You know, what are there problems 0:04:16.320 --> 0:04:18.520 and it what could the problems be? Because of what 0:04:18.560 --> 0:04:21.880 we'll discuss, there are a number of problems that can 0:04:21.880 --> 0:04:25.839 and do occur in modern peer of viewed science. Now, 0:04:25.880 --> 0:04:28.720 one of the things that will come up sometimes when 0:04:28.880 --> 0:04:32.400 people write on this topic is careerism as one of 0:04:32.440 --> 0:04:36.479 the factors that is problematic. And that's because we've all 0:04:36.520 --> 0:04:42.120 heard the maximum published or parish right, and the spirit 0:04:42.240 --> 0:04:44.200 of it is not so bad. I mean, the spirit 0:04:44.240 --> 0:04:47.400 of it is really like less than a threat and 0:04:47.560 --> 0:04:51.680 more like, hey, this is a challenge to push science forward. 0:04:52.200 --> 0:04:56.799 Put forth your multiple lines of evidence, your hypotheses, your theories, 0:04:57.600 --> 0:04:59.800 because we all want to share information. We wanted to 0:05:00.040 --> 0:05:02.480 it apart, we want to try to validate it or 0:05:02.560 --> 0:05:06.320 invalidate it, and generally create a better understanding of the 0:05:06.400 --> 0:05:09.720 topic or the issue. So again it's an attempt at 0:05:09.800 --> 0:05:14.479 reaching some sort of truth. And yet the reality of 0:05:14.600 --> 0:05:19.000 published or perish now is more that it's this kind 0:05:19.040 --> 0:05:21.520 of pressure to produce. So it's not enough for say 0:05:21.560 --> 0:05:25.120 a faculty member at university to write a few really 0:05:25.160 --> 0:05:28.400 good papers a year. Now they have this pressure to 0:05:28.440 --> 0:05:33.160 write several And so there's this idea that questionable results 0:05:33.400 --> 0:05:37.520 could come out of this, and instead of maybe making 0:05:37.560 --> 0:05:40.280 it to a first tier journal, maybe that data goes 0:05:40.320 --> 0:05:44.640 to a third tier journal. And yet it shouldn't necessarily 0:05:44.680 --> 0:05:48.839 go any place. And the problem, as outlined in the 0:05:48.880 --> 0:05:52.799 Economist article how Science Goes Wrong, is quote. In order 0:05:52.839 --> 0:05:57.640 to safeguard their exclusivity, the leading journals impose high rejection 0:05:57.720 --> 0:06:02.880 rates in excess of submitted manuscripts. The most striking findings 0:06:02.960 --> 0:06:06.240 have the greatest chance of making it onto the page. 0:06:06.480 --> 0:06:10.000 Little wonder that one in three researchers knows of a 0:06:10.080 --> 0:06:13.440 colleague who has pepped up a paper by say, excluding 0:06:13.600 --> 0:06:18.159 inconvenient data from results based on a gut feeling. So 0:06:18.279 --> 0:06:22.200 we're talking about here is cherry picking information. And then 0:06:22.240 --> 0:06:25.840 all of this, this kind of careerism is compounded by 0:06:25.839 --> 0:06:29.599 the pressure to generate grant funding. So there's this idea 0:06:29.720 --> 0:06:33.520 that more and more scientists are having a bigger percentage 0:06:33.520 --> 0:06:38.760 of their salary covered by contingent or research funding dollars. 0:06:38.800 --> 0:06:40.680 So that means that you now have this pressure to 0:06:40.880 --> 0:06:44.960 keep the flow of funding going with positive results. So 0:06:45.000 --> 0:06:46.800 you can say, yeah, see, this is exactly what I 0:06:46.800 --> 0:06:51.640 thought was going to happen, proving out um. That shouldn't 0:06:51.680 --> 0:06:55.640 be the case. There shouldn't be those sort of strings 0:06:55.680 --> 0:06:59.000 tied to it, and in an ideal world that wouldn't 0:06:59.040 --> 0:07:01.680 be the case. But that's what we're dealing with. And 0:07:01.680 --> 0:07:05.279 then there's this failures to prove hypothesis are actually rarely 0:07:05.360 --> 0:07:08.760 offered for publication, let alone accepted. Uh, you know, and 0:07:08.920 --> 0:07:10.920 you can sort of squirrel away a lot of this 0:07:11.080 --> 0:07:14.000 to you know, what scientific journal doesn't want to be 0:07:14.080 --> 0:07:17.000 on the forefront of science, you know, full of amazing 0:07:17.040 --> 0:07:21.040 new discoveries and and and wonderful new ideas, right that's 0:07:21.160 --> 0:07:24.120 you know, that's really essential to the overall drive of science. 0:07:24.120 --> 0:07:26.240 You don't want to fill your your paper with a 0:07:26.240 --> 0:07:29.120 bunch of failures, right, But the failures are important, right. 0:07:29.120 --> 0:07:31.800 You need to know what hasn't worked so you can 0:07:31.800 --> 0:07:33.600 try and figure out what does work. You need to 0:07:33.600 --> 0:07:36.040 know what's false so you can figure out what's true. Yet, 0:07:36.040 --> 0:07:39.240 in two thousand thirteen, negative results are accounted for only 0:07:39.320 --> 0:07:41.680 fourteen percent of published papers, and that was down from 0:07:41.720 --> 0:07:46.600 thirty in nineteen ninety. And then in a similar vein 0:07:46.880 --> 0:07:50.880 we see the peer of view process UM often sees 0:07:51.000 --> 0:07:53.600 peers missing the errors in the paper. The very thing 0:07:53.600 --> 0:07:56.200 they're supposed to do is, you know, figure out what's 0:07:56.240 --> 0:07:59.840 what's potentially wrong with this work. So both of these 0:08:00.040 --> 0:08:04.240 into handicap the process to a certain extent. You know, 0:08:04.360 --> 0:08:08.160 it's interesting because my daughter's school has nine different design 0:08:08.200 --> 0:08:11.440 principles of education, and there this is something that they 0:08:11.480 --> 0:08:14.800 actually present to the student. So kindergarten reserve being taught 0:08:15.320 --> 0:08:20.160 about failure and actually celebrating failure for this very reason, 0:08:20.240 --> 0:08:23.440 because the idea, again is that you cannot have successes 0:08:23.480 --> 0:08:27.240 without failures. And uh makes me think about Edison and 0:08:27.280 --> 0:08:29.920 the light bulb and the hundred plus iterations of the 0:08:30.000 --> 0:08:33.080 light bulb, all the failures that proceeded those, And yet 0:08:33.200 --> 0:08:36.199 that's not the flashy stuff, right, that's not what necessarily 0:08:36.240 --> 0:08:39.240 a first tier journal is going after. Like, hey, tell 0:08:39.240 --> 0:08:42.920 me about your spectacular failure. Yeah, Like I keep thinking 0:08:42.960 --> 0:08:45.800 about science in terms of slime mold. We did an 0:08:45.800 --> 0:08:47.720 episode on the slime mold way back, where you would 0:08:47.720 --> 0:08:50.520 put a slime mold in a maze, and it's solving 0:08:50.559 --> 0:08:52.560 the maze to get to resources on the outside of 0:08:52.559 --> 0:08:55.360 the maze. And so these tendrils of slime mold or 0:08:55.480 --> 0:08:57.319 trailing through the mazing if they reach a dead end, 0:08:57.320 --> 0:08:59.959 and that tendril dies and fades back and it doesn't 0:09:00.000 --> 0:09:02.199 go down that way again. And science kind of works 0:09:02.400 --> 0:09:05.440 the same way that you need to know which where 0:09:05.480 --> 0:09:08.160 the dead ends are, otherwise you're just gonna keep sending 0:09:08.160 --> 0:09:11.800 your tendils down there. Well. And then it's also this 0:09:12.280 --> 0:09:16.400 such an elegant analogy because they're going after that sugar, right, 0:09:16.840 --> 0:09:20.000 that that resource, and so they're eventually going to find 0:09:20.000 --> 0:09:23.280 themselves to the success story of the resource. But then 0:09:23.360 --> 0:09:25.880 it becomes this question of is that resource that piece 0:09:25.880 --> 0:09:29.720 of sugar that the slime bowl is after. Is this 0:09:29.960 --> 0:09:32.840 truth or is this money? And we'll talk a little 0:09:32.880 --> 0:09:35.400 bit more about that later, but I thought at this 0:09:35.440 --> 0:09:37.679 point I would go ahead and drop in a little 0:09:37.679 --> 0:09:42.920 information about over generalization and extrapolation of results, because this 0:09:42.960 --> 0:09:46.400 can occur in two ways. The first is applying findings 0:09:46.440 --> 0:09:49.840 from one target group to another target group within the 0:09:49.880 --> 0:09:53.319 same population. So an example would be you have this 0:09:53.440 --> 0:09:57.720 new cholesterol drug and it's been tested on females age 0:09:58.520 --> 0:10:02.320 uh it is thirty two fifty. Well, you can't make 0:10:02.440 --> 0:10:04.880 the assumption that the drug can also do the same 0:10:04.920 --> 0:10:08.040 thing for a different population, say women over sixty five 0:10:08.200 --> 0:10:12.320 or men. The second fallacy is applying the survey results 0:10:12.320 --> 0:10:14.720 to population is not living in the area in the survey. 0:10:14.800 --> 0:10:17.080 So this is this to me was very clear cut. 0:10:18.240 --> 0:10:20.559 Let's say that you're trying to establish the mortality rate 0:10:20.559 --> 0:10:23.560 for a certain neighborhood within a zip code. All right, 0:10:23.760 --> 0:10:26.120 you do the research, you do the surveying, and then 0:10:26.160 --> 0:10:29.679 you you've got your data. Now it would be beneficial 0:10:29.720 --> 0:10:32.760 to find out what other neighborhoods mortality rate. But you 0:10:32.840 --> 0:10:35.520 make the assumption that just because the borders of this 0:10:35.600 --> 0:10:37.839 other neighborhood are butting up against the one that you've 0:10:37.880 --> 0:10:41.040 just surveyed, that they have the same mortality rate. Well, 0:10:41.080 --> 0:10:43.720 that is erroneous thinking, because as we know and we 0:10:43.760 --> 0:10:46.360 have seen over and over again you can have really 0:10:46.440 --> 0:10:50.560 poor neighborhoods betting up against very prosperous ones and that 0:10:50.640 --> 0:10:52.960 excuse the data because the very protuluh ones are going 0:10:53.000 --> 0:10:56.719 to have a far different mortality rate than a poor neighborhood. 0:10:57.480 --> 0:10:59.959 And yet these are some of the things that lead 0:11:00.080 --> 0:11:03.720 can with studies and experiments. And then of course there 0:11:03.840 --> 0:11:08.360 is conflict of interest, which is a big one. UH. 0:11:08.360 --> 0:11:10.760 And we can date a lot of this back to 0:11:10.880 --> 0:11:13.600 the Bio Dule Act of nineteen eighty and this came 0:11:13.600 --> 0:11:17.600 along to encourage technology transfer from universities to industry. The 0:11:17.600 --> 0:11:22.480 IDBA being that it would facilitate financial relationships between academic 0:11:22.520 --> 0:11:27.719 biomedical researchers and the biotechnology industry. And you know, obviously 0:11:28.200 --> 0:11:30.480 there's a lot of good that was going to come 0:11:30.480 --> 0:11:32.040 out of this and has come out of this. Uh. 0:11:32.400 --> 0:11:35.480 They leave these uh. These relationships lead to the development 0:11:35.480 --> 0:11:39.400 of improved drugs and medical devices. UH. But on the 0:11:39.400 --> 0:11:43.040 other hand, there's this huge financial aspect of the relationship. 0:11:43.040 --> 0:11:46.960 Of financial relationship emerges relationships that can cause conflicts of 0:11:47.000 --> 0:11:51.079 interest between a researchers scientific and ethical principles and that 0:11:51.160 --> 0:11:54.160 gleam of financial gain coming background to what you said 0:11:54.160 --> 0:11:56.280 about what is the what is the bait on the 0:11:56.320 --> 0:11:59.680 outside of the maze? Is it? Is it knowledge and understanding? 0:11:59.800 --> 0:12:03.719 Is it? Is it increasing our scientific understanding of a 0:12:03.760 --> 0:12:07.160 particular ailment? Or is it mere financial gain? And of course, 0:12:07.760 --> 0:12:11.680 financial gain for a biomedical corporation tends to boil down 0:12:11.720 --> 0:12:14.199 to treatment, the drugs that can be thrown at a 0:12:14.200 --> 0:12:16.800 particular ailment, the medical devices that can be thrown at 0:12:16.800 --> 0:12:19.720 a particular ailment. And in a two thousand nine study 0:12:19.760 --> 0:12:24.440 from Dr Rueschmajas, Assistant Professor of Radiation Oncology at the 0:12:24.480 --> 0:12:28.360 University of Michigan Medical School, compared a thousand, five thirty 0:12:28.360 --> 0:12:32.760 four studies involving cancer research, found that studies with with 0:12:32.920 --> 0:12:37.920 industry funding focused on treatment again drugs, medical devices sixty 0:12:38.480 --> 0:12:41.000 of the time compared to thirty six percent of the 0:12:41.040 --> 0:12:44.800 time for other studies not funded by industry, and the 0:12:44.840 --> 0:12:49.360 studies funded by industry focused on epidemiology, prevention, risk factors, 0:12:49.400 --> 0:12:53.120 screening and other diagnostic methods only twenty percent of the 0:12:53.160 --> 0:12:57.720 time versus forty seven for studies with no declared industry funding. 0:12:57.800 --> 0:13:01.520 So the take home here seems to be the more 0:13:01.640 --> 0:13:05.560 money is involved from these from the biotech industry. The 0:13:05.640 --> 0:13:08.840 more focus there is going to be on the mere 0:13:08.880 --> 0:13:12.040 treatment of an ailment versus uh um, you know, actually 0:13:12.080 --> 0:13:15.560 being able to prevent it or figure out how to 0:13:15.600 --> 0:13:19.280 screen it through looking at risk factors, which my lead 0:13:19.320 --> 0:13:23.640 to misleading statistics or interpretation about the data. And what 0:13:23.760 --> 0:13:27.960 I'm talking about is absolute versus relative percentages. This is 0:13:28.040 --> 0:13:32.120 from the article bad Science, Common Problems and research Articles. 0:13:32.120 --> 0:13:35.560 This was published on Health Readings. Quote supposed that there 0:13:35.679 --> 0:13:38.880 was a medical problem that caused two people in one 0:13:38.960 --> 0:13:41.880 million to have a stroke, and suppose there was a 0:13:41.920 --> 0:13:45.080 treatment that would reduce the problem to only one person 0:13:45.480 --> 0:13:48.360 in one million. This would be an improvement of point 0:13:48.520 --> 0:13:53.880 zero zero zero one percent in an absolute sense, or 0:13:54.240 --> 0:13:57.640 or as this author says, no big deal, right. However, 0:13:58.440 --> 0:14:03.120 if it had been hoarded using relative percentages, it could 0:14:03.160 --> 0:14:07.960 have been stated quote new medical treatment yields a rediction 0:14:08.160 --> 0:14:11.560 and reduction and risk of stroke, and this would be 0:14:11.640 --> 0:14:15.600 very misleading. But it's unfortunately a common practice that you 0:14:15.640 --> 0:14:18.400 see from time to time, and so again you see 0:14:18.440 --> 0:14:22.080 how that's it's not exactly wrong. It is a fifty 0:14:22.360 --> 0:14:27.280 percent reduction in the two and one million people, but 0:14:27.360 --> 0:14:31.680 it's not really accurate saying. It's just how how do 0:14:31.720 --> 0:14:34.040 you end up using? How the effect the overall statistics 0:14:34.160 --> 0:14:39.520 that you're dealing with. Yeah, semantics matter. Now. Another area 0:14:39.600 --> 0:14:43.360 of concern is that of unpublished clinical trials. A two 0:14:43.360 --> 0:14:46.720 thousand twelve study from Yale School of Medicine researchers found 0:14:46.760 --> 0:14:50.160 that fewer than half of a sample of trials primarily 0:14:50.240 --> 0:14:53.040 or partially funded by the National Institutes of Health were 0:14:53.040 --> 0:14:56.560 published within thirty months of completing the clinical trial. So, 0:14:56.600 --> 0:14:59.720 in other words, the research refindings here are not being 0:15:00.000 --> 0:15:03.800 emanated half the time, So the scientific process is disrupted, 0:15:04.000 --> 0:15:07.840 undermining the effort and the available material for peer of view. Now, 0:15:07.920 --> 0:15:11.160 according to study author Dr Joseph Ross, they're probably a 0:15:11.240 --> 0:15:14.000 number of reasons for lack of publication, such as not 0:15:14.000 --> 0:15:16.120 getting accepted by a journal and we already hit on 0:15:16.160 --> 0:15:20.240 the high rejection rates, or not prioritizing the dissemination of 0:15:20.280 --> 0:15:25.400 research findings in the study. Either way, this disrupts the process. 0:15:25.440 --> 0:15:29.600 This disrupts the strengths of the peer of view system. 0:15:29.640 --> 0:15:35.360 Another factor is something called selective observation. Now, you've probably 0:15:35.400 --> 0:15:41.000 experienced your own selective observation before. My example is every 0:15:41.000 --> 0:15:44.880 time I get into the shower of my phone rings, right. Uh, 0:15:44.880 --> 0:15:47.000 and it's a perception that is based on the annoyance 0:15:47.000 --> 0:15:49.680 of my phone ringing and my inability to get to it. 0:15:50.960 --> 0:15:54.600 But then I, you know, I tended to disregard all 0:15:54.640 --> 0:15:56.680 the times that my phone didn't ring well as in 0:15:56.760 --> 0:15:59.680 the shower, and so I was practicing confirmation biased and 0:16:00.080 --> 0:16:04.760 oring the other data, skewing my own statistics. So selective 0:16:04.760 --> 0:16:07.840 observation in science is essentially trying to land on a 0:16:07.880 --> 0:16:11.960 conclusion based on an existing bias or belief. For example, 0:16:12.400 --> 0:16:14.840 a researcher who studying obesity may have a bias that 0:16:14.920 --> 0:16:18.720 obese people lack will power, and as a result, they 0:16:18.760 --> 0:16:21.920 may construct an experiment that involved a plate of donuts 0:16:21.920 --> 0:16:25.040 and a conference froom work. But if that researcher only 0:16:25.080 --> 0:16:29.840 records data about ABS subjects and doesn't record non ABS subjects, well, 0:16:29.920 --> 0:16:32.600 then they have a biased experiment on their hands. In 0:16:32.600 --> 0:16:35.600 other words, Uh, if they don't go out of their 0:16:35.640 --> 0:16:39.119 way to try to prove themselves wrong, they're not exercising 0:16:39.120 --> 0:16:42.960 the principles of scientific method. All right, you know, let's 0:16:42.960 --> 0:16:45.480 take a quick break and when we come back we 0:16:45.520 --> 0:16:58.560 will discuss weird science. All right, we're back. Weird science, 0:16:58.840 --> 0:17:03.760 weird psychology, and I'm not talking about the eighties classic 0:17:03.840 --> 0:17:07.480 as it is. Um No, weird is a phenomenon that 0:17:07.520 --> 0:17:10.440 plagues a lot of psychology and other social science studies. 0:17:10.840 --> 0:17:14.120 This is when the Protestants are overwhelmingly This is where 0:17:14.160 --> 0:17:18.800 weird comes in Western E for educated, and they're from 0:17:19.320 --> 0:17:23.840 I for industrialized, ARE for rich, and D for democratic countries. 0:17:24.000 --> 0:17:29.720 So weird humans are serving as the basic test subjects 0:17:29.840 --> 0:17:32.560 in a lot of these studies. And you can also 0:17:32.720 --> 0:17:36.480 add in that weird humans are also often college students 0:17:36.720 --> 0:17:41.200 in the United States, participating in studies for class credit. So, 0:17:41.440 --> 0:17:44.600 especially in the social sciences, the risk is that so 0:17:44.680 --> 0:17:48.959 called weird populations are actually the outliers of human population 0:17:49.240 --> 0:17:52.520 as opposed to a good standard example of human behavior. 0:17:53.040 --> 0:17:55.680 And you know, you see this, You see shades of 0:17:55.760 --> 0:17:57.040 this time and time again. Right. You look at a 0:17:57.080 --> 0:17:59.280 study and it was clearly a study that was conducted 0:18:00.040 --> 0:18:04.760 on campus with students, and in your better studies you 0:18:04.800 --> 0:18:08.200 see them branching out from that and saying, uh um, well, 0:18:08.280 --> 0:18:10.040 all right, when this first study we looked at students, 0:18:10.080 --> 0:18:12.719 but then we went into an impoverished neighborhood or in 0:18:12.720 --> 0:18:16.000 some cases, then we looked at some US participants. Then 0:18:16.000 --> 0:18:18.080 when we also went and looked at some participants in 0:18:18.160 --> 0:18:21.880 Hong Kong, that sort of thing. Um. And so obviously 0:18:21.920 --> 0:18:23.680 there are a lot there's a lot to consider here 0:18:23.920 --> 0:18:26.639 with the software of psychology, right, because there's so much 0:18:26.640 --> 0:18:30.240 about human culture and uh and in your relations within 0:18:30.320 --> 0:18:33.800 your particular group. But it also bleeds into the hardware 0:18:33.880 --> 0:18:38.920 of physiology. In two thousand fourteen, Liverpool University had a 0:18:38.960 --> 0:18:43.520 study examining rapid eye movements called cicades among groups of 0:18:43.760 --> 0:18:48.639 mainland Chinese, British Chinese, and white British test subjects, and 0:18:48.640 --> 0:18:51.640 he found that Chinese ethnicity was more of a factor 0:18:51.680 --> 0:18:55.919 than culture in high cicade counts. So the mainland Chinese 0:18:55.960 --> 0:18:59.600 groups scored high cicade numbers as did the British Chinese counterparts, 0:19:00.119 --> 0:19:03.760 despite the many cultural differences between the two groups. So 0:19:04.800 --> 0:19:08.200 lead author Dr Paul Knox argued, quote, the human brain 0:19:08.359 --> 0:19:12.320 is not just amazingly complex in general, but also highly 0:19:12.440 --> 0:19:18.960 variable across the human population. Mm hmm. And that variability 0:19:19.280 --> 0:19:23.120 takes us to the next entry here, which is animals. Now, 0:19:23.160 --> 0:19:28.080 we have talked about how much rodents have um contributed 0:19:28.119 --> 0:19:31.680 to science, and they absolutely have, but we do have 0:19:31.800 --> 0:19:37.840 problems where animal studies do not reliably predict human outcomes. 0:19:38.480 --> 0:19:40.840 And this topic is really a complex one, but there's 0:19:40.840 --> 0:19:44.080 a paper on the topic by Michael B. Bracken who's 0:19:44.119 --> 0:19:47.480 from Yale University, and he writes in his paper why 0:19:47.560 --> 0:19:50.119 animal studies are often poor predictors of human reactions to 0:19:50.160 --> 0:19:54.879 exposure that one reason is probably because animal experiments do 0:19:55.000 --> 0:19:58.080 not translate into replications, and human trials are into cancer 0:19:58.160 --> 0:20:04.680 chemoprevention because as they're poorly designed, conducted, and analyzed. Now, 0:20:04.720 --> 0:20:07.919 another possible contribution to failure to replicate the results of 0:20:08.040 --> 0:20:12.359 animal research and humans is that reviews and summaries of 0:20:12.440 --> 0:20:16.840 evidence from animal research are inadequate when it comes to methodology, 0:20:16.960 --> 0:20:20.560 and one survey, only one in ten thousand Meadline records 0:20:20.600 --> 0:20:23.879 of animal studies were tagged as being meta analysis, is 0:20:24.359 --> 0:20:28.520 compared to one and one thousand human studies, and in 0:20:28.640 --> 0:20:32.399 recent reports, the poor quality of research was documented by 0:20:32.400 --> 0:20:35.200 a comprehensive search of Medline, which found only twenty five 0:20:35.480 --> 0:20:40.600 systematic reviews of animal research. Other studies similarly found only 0:20:40.760 --> 0:20:44.359 thirty and fifty seven systematic reviews of any type of 0:20:44.400 --> 0:20:48.840 animal researcher so Um. The reason that Bracken points us 0:20:48.840 --> 0:20:51.760 out is because he says these kind of deficiencies are 0:20:51.840 --> 0:20:57.280 important because animal research often provides the rationale for hypotheses 0:20:57.359 --> 0:21:02.200 studied by epidemiologists and clinical researchers. Moreover, if you look 0:21:02.240 --> 0:21:06.280 at the genetics of this, it gets even more muddled. 0:21:06.440 --> 0:21:09.560 And the reason for that is because with rodents, and 0:21:09.600 --> 0:21:11.160 one of the reasons why we use them is because 0:21:11.200 --> 0:21:15.280 we can change their genetic background within a couple of generations. 0:21:15.400 --> 0:21:19.640 We can tinker with the genes. And that's great because 0:21:19.680 --> 0:21:24.640 that can really help us to study certain conditions. However, Um, 0:21:24.800 --> 0:21:29.800 those rodents would yield really consistent results and disease expression. 0:21:30.840 --> 0:21:33.960 But humans, we are far more wild West when it 0:21:34.000 --> 0:21:37.720 comes to genetics and the genetic background, and that would 0:21:37.800 --> 0:21:40.639 factor in how the human disease is expressed, and this 0:21:40.680 --> 0:21:46.200 would yield a mismatching results between humans and animals. It's 0:21:46.200 --> 0:21:50.680 a layer cake of animal confusion. Indeed, it is um. Now, 0:21:50.720 --> 0:21:53.560 on top of everything we've discussed here, there are plenty 0:21:53.600 --> 0:21:57.879 of additional methodological pitfalls, and we're we're gonna include a 0:21:58.000 --> 0:22:00.199 link on the landing page of this episode to a 0:22:00.200 --> 0:22:02.640 fabulous page that has a list of about sixty of them, 0:22:02.960 --> 0:22:04.600 and we're not going to go into all into detail 0:22:04.600 --> 0:22:05.680 on all of them here, but just to give you 0:22:05.720 --> 0:22:08.479 an example, this includes the likes of the cebo effect, 0:22:08.520 --> 0:22:12.400 which we've discussed at length before, and in which the 0:22:12.400 --> 0:22:17.480 the individual receiving the sugar pill ends up actually getting 0:22:17.560 --> 0:22:21.159 some sort of biological benefit from from the medication or 0:22:21.160 --> 0:22:24.960 the fake medication, uh carry over effect, where the results 0:22:24.960 --> 0:22:29.159 of one study are are observed in a secondary study 0:22:29.600 --> 0:22:32.639 without realizing it. And then magnitude blindness, the tendency to 0:22:32.680 --> 0:22:38.040 become preoccupied with statistically significant results that never nevertheless have 0:22:38.119 --> 0:22:41.240 a small magnitude on effect. I feel like that comes 0:22:41.280 --> 0:22:45.320 into play a lot when I look at, um, some 0:22:45.400 --> 0:22:48.080 of this stuff that's new and that's being reported in 0:22:48.080 --> 0:22:50.280 the media. It's very exciting, right, you know, oh wow, 0:22:50.359 --> 0:22:52.440 look at this insight, and then when you get into 0:22:52.440 --> 0:22:56.879 the specifics of the study, it's just it's not that significant, right, 0:22:57.000 --> 0:23:00.639 doesn't quite match up to that snappy headline. All right, 0:23:00.720 --> 0:23:03.960 So how does science correct course? What can be done 0:23:03.960 --> 0:23:08.720 about these problems we've discussed? Well, um, just to talk 0:23:09.040 --> 0:23:13.520 briefly about the use of statistics and managing potential conflicts, 0:23:13.760 --> 0:23:17.600 those financial conflicts we we mentioned earlier, conflicts of interest. Um. 0:23:17.640 --> 0:23:20.159 The general idea that the expert put put forth is 0:23:20.200 --> 0:23:23.800 that we need to simplify, standardized and better enforced policies 0:23:23.840 --> 0:23:27.960 to manage financial conflicts of interest, and that science needs 0:23:28.000 --> 0:23:30.479 to keep a better eye on statistics, by which we mean, 0:23:30.520 --> 0:23:34.120 of course, the statistical validity and the statistical errors inherent 0:23:34.200 --> 0:23:37.840 in the system. Another thing is to encourage replication. And 0:23:37.880 --> 0:23:41.120 again this is from the Economist article quote. Some government 0:23:41.119 --> 0:23:45.879 funding agencies, including America's National Institutes of Health, which dish 0:23:45.920 --> 0:23:49.240 out thirty billion on research each year, are working out 0:23:49.240 --> 0:23:52.800 how to best encourage replication and growing numbers of scientists, 0:23:52.880 --> 0:23:58.880 especially young ones, understand statistics. Another area is allocating space 0:23:59.119 --> 0:24:03.119 and journals were uninteresting studies, which which is which is 0:24:03.119 --> 0:24:05.399 crazy because if you think about it in terms of, say, um, 0:24:05.640 --> 0:24:08.040 you know, a literary fiction publication, you would never in 0:24:08.040 --> 0:24:10.880 a million years have anyone suggests, hey, we should make 0:24:11.000 --> 0:24:14.760