WEBVTT - Research tips from SYSK 0:00:01.160 --> 0:00:04.600 Welcome to Stuff You Should Know from House Stuff Works 0:00:04.600 --> 0:00:13.000 dot com. Hey, and welcome to the podcast. I'm Josh 0:00:13.039 --> 0:00:17.960 Clark with Charles W. Chuck Bryant and Jerry This Stuff 0:00:17.960 --> 0:00:22.600 you Should Know. Um, Josh, We're gonna do something weird today. 0:00:22.720 --> 0:00:25.360 I'm gonna do a listener mail at the head of 0:00:25.360 --> 0:00:29.120 the podcast. I know, all right, what? All right, let's 0:00:29.120 --> 0:00:32.720 do it. Okay, this is from b Wait, hold on, 0:00:32.760 --> 0:00:34.960 can you have the listener mail? Music going? Oh? I 0:00:34.960 --> 0:00:37.080 don't know, Jerry, should we go the whole nine yards? 0:00:37.080 --> 0:00:41.680 So let's do it? People might freak out? All right? 0:00:41.720 --> 0:00:46.120 This is from Bianca. Uh voice is what I'm gonna say. 0:00:46.320 --> 0:00:48.280 I think that's great. Hey, guys wrote you not too 0:00:48.320 --> 0:00:50.920 long ago asking about Hey, research your own podcast. It 0:00:51.040 --> 0:00:53.280 just got back from a class where we talked about 0:00:53.760 --> 0:00:58.800 research misrepresentation and journal articles. Apparently journals don't publish everything 0:00:58.800 --> 0:01:01.360 that is submitted, and lot of researchers don't even publish 0:01:01.360 --> 0:01:04.319 their studies they don't like the results. Uh. Some laws 0:01:04.319 --> 0:01:07.479 have been put into place to prevent misrepresentation, such as 0:01:07.520 --> 0:01:10.760 researchers having to register their studies before they get results 0:01:10.800 --> 0:01:14.120 and journals only accepting preregistered studies. But apparently this is 0:01:14.120 --> 0:01:17.559 not happening at all, even though it is now technically law. 0:01:18.160 --> 0:01:21.520 This ends with the general public being misinformed about methods 0:01:21.520 --> 0:01:24.399 and drugs that work. For example, there are twenty five 0:01:24.440 --> 0:01:28.000 studies proving a drug works, twenty five that don't. It's 0:01:28.040 --> 0:01:31.360 more likely that twenty of the positive results have been 0:01:31.360 --> 0:01:34.480 published and only one or two of the negative. Uh. 0:01:34.480 --> 0:01:37.640 And that is from Bianca, and that led us to 0:01:37.840 --> 0:01:42.160 this article on our own website. Ten signs that that 0:01:42.240 --> 0:01:46.480 study is bogus, and here it is nice Chuck. Well, 0:01:46.480 --> 0:01:49.200 we get asked a lot about research from people usually 0:01:49.200 --> 0:01:52.840 in college. Are like, you guys are professional researchers. How 0:01:52.840 --> 0:01:54.880 do I know I'm doing a good job and getting 0:01:54.880 --> 0:01:58.920 good info? And it's getting harder and harder these days, 0:01:59.160 --> 0:02:01.720 it really is. You know. One sign that I've learned 0:02:01.800 --> 0:02:06.200 is if you are searching about his study, and all 0:02:06.240 --> 0:02:11.639 of the hits that come back are from different news organizations, 0:02:11.680 --> 0:02:13.800 and they're all within like a two three day period 0:02:14.160 --> 0:02:17.960 from a year ago, nothing like, nothing more recent than 0:02:18.000 --> 0:02:21.919 that than somebody released a sensational study and no one 0:02:22.000 --> 0:02:25.160 did any actual effort into investigating it, and there was 0:02:25.200 --> 0:02:28.040 no follow up. If you dig deep enough, somebody might 0:02:28.080 --> 0:02:29.600 have done follow up or something like that, but for 0:02:29.639 --> 0:02:32.880 the most part, it was just something that splashed across 0:02:32.919 --> 0:02:35.680 the headlines, which more often than not is the is 0:02:35.720 --> 0:02:38.440 the case as far as science reporting goes. So that's 0:02:38.440 --> 0:02:43.600 a bonus. That's the eleven boom. How about that? Yeah, 0:02:43.840 --> 0:02:46.239 should we just start banging these out let's do it? 0:02:46.280 --> 0:02:51.280 Al Do you have some other clever well apart and 0:02:51.360 --> 0:02:53.600 parcel with that, I don't know if it's clever. You 0:02:53.680 --> 0:02:56.359 do come across people who you know can be trusted 0:02:56.400 --> 0:03:00.400 and relied upon to do good science reporting. So like 0:03:00.560 --> 0:03:04.200 Ed Young is one, another guy named Ben gold Acre 0:03:04.280 --> 0:03:06.839 has something called bad Science I don't remember what, Yeah, 0:03:06.880 --> 0:03:09.200 outlet he's with. And then there's a guy I think 0:03:09.240 --> 0:03:13.360 scientific American named John Horgan who's awesome. Yeah, or some 0:03:13.480 --> 0:03:16.799 journalism organizations that have been around instood the test of 0:03:16.840 --> 0:03:18.639 time that you know are really doing it right, like 0:03:18.919 --> 0:03:23.680 a nature Yeah. Scientific Americans are like really science Yeah, 0:03:23.760 --> 0:03:26.600 Like I feel I feel really good about using those sources. Yeah, 0:03:26.639 --> 0:03:29.520 but even they can you know, there's there's something called 0:03:29.600 --> 0:03:32.520 scientis um where there's a lot of like faith in 0:03:32.639 --> 0:03:36.200 dogma associated with the scientific process, and you know, you 0:03:36.240 --> 0:03:40.640 have to root through that as well. Try it. I'm done. Uh. 0:03:40.800 --> 0:03:42.600 The first one that they have here on the list 0:03:42.680 --> 0:03:46.280 is that it's unrepeatable, and that's a big one. UM. 0:03:46.320 --> 0:03:49.840 The Center for Open Science did a study, UH was 0:03:49.880 --> 0:03:52.120 a project really where they took two hundred and seventy 0:03:52.760 --> 0:03:55.120 researchers and they said, you know what, take these one 0:03:55.200 --> 0:03:59.720 hundred studies that have been published already psychological studies and 0:04:00.120 --> 0:04:03.840 pour over them. And uh in just last year. They 0:04:03.880 --> 0:04:06.200 took him a while, took them several years. They said, 0:04:06.200 --> 0:04:08.280 you know what, more than half of these can't even 0:04:08.320 --> 0:04:11.880 be repeated using the same methods. They're not reproducible. Nope, 0:04:11.920 --> 0:04:15.760 not reproducible. That's a big one. And and ones that 0:04:15.800 --> 0:04:18.360 means that they they when they carried out they followed 0:04:18.360 --> 0:04:22.280 the methodology. UM Scientific method podcasts. You should listen to 0:04:22.360 --> 0:04:25.000 that one. That was a good one. That they found 0:04:25.040 --> 0:04:27.560 that their results were just not what the what the 0:04:27.600 --> 0:04:31.160 people published, not anywhere near them. UM. For example, they 0:04:31.240 --> 0:04:34.880 used one as an example where a study found that 0:04:34.920 --> 0:04:40.599 men were terrible at a determining whether a woman was 0:04:40.680 --> 0:04:44.200 giving them UM some sort of like a clues to 0:04:44.279 --> 0:04:49.640 attraction or just being friendly, sexy, sexy stuff or yeah, 0:04:49.839 --> 0:04:53.320 or good to meet you or buzz off jerky um. 0:04:53.440 --> 0:04:56.080 And they did the study again and as part of 0:04:56.120 --> 0:05:00.680 this Open Science Center for Open Science study or survey, 0:05:00.720 --> 0:05:04.279 and they found that that was not reproducible or that 0:05:04.360 --> 0:05:06.880 they came up with totally different results. And that was 0:05:06.960 --> 0:05:09.640 just one of many. Yeah, and in this case specifically, 0:05:09.680 --> 0:05:13.159 they looked into that study and they found that it was, um, 0:05:13.320 --> 0:05:15.320 one was in the United Kingdom, one was in the 0:05:15.400 --> 0:05:17.640 United States. May have something to do with it. But 0:05:17.800 --> 0:05:21.840 the point is, Chuck, is if you're talking about humanity, 0:05:22.160 --> 0:05:24.839 I don't think that the study was like the American 0:05:25.000 --> 0:05:28.320 male is terrible at it. It's men are terrible at it. Right. 0:05:28.400 --> 0:05:30.479 So that means that whether it's in the UK, which 0:05:30.480 --> 0:05:33.680 is basically the US with an accent and a penchant 0:05:33.760 --> 0:05:38.920 for t I'm just kidding UK s soon, Um, the 0:05:39.600 --> 0:05:44.680 it should be universal. Yeah, you know, unless you're saying, no, 0:05:44.880 --> 0:05:51.160 it's just this only applies to American men, the American men, right, 0:05:51.400 --> 0:05:54.400 then it's not even study. Yeah. The next one we 0:05:54.440 --> 0:06:00.680 have is, uh, it's it's plausible, not necessarily a provable. 0:06:00.800 --> 0:06:04.400 And this is a big one because and I think, um, 0:06:04.480 --> 0:06:07.839 we're talking about observational studies here more than lab experiments, 0:06:07.880 --> 0:06:10.120 because with observational studies, you know, you sit in a 0:06:10.240 --> 0:06:14.080 room and get asked three questions about something, and all 0:06:14.120 --> 0:06:15.920 these people get asked the same questions, and then they 0:06:15.960 --> 0:06:19.400 pour over the data and they draw out their own observations. 0:06:20.200 --> 0:06:23.000 And one of the very famously an observational study that 0:06:23.080 --> 0:06:26.479 led to false results found a correlation between having a 0:06:26.520 --> 0:06:31.200 type A personality and UM being prone to risk for 0:06:31.360 --> 0:06:35.360 heart attacks and UM. For a long time, you know 0:06:35.640 --> 0:06:38.000 that the news outlets were like, oh, yes, of course, 0:06:38.080 --> 0:06:40.800 that makes total sense. This study proved what we've all 0:06:40.839 --> 0:06:44.680 known all along, UM, And then it came out that no, 0:06:44.880 --> 0:06:49.760 actually what was going on was a well known anomaly 0:06:50.240 --> 0:06:54.720 where you have a five percent um risk that chance 0:06:54.800 --> 0:06:59.240 will produce something that looks like a statistically significant correlation 0:06:59.440 --> 0:07:01.680 when it's not all when really it's just total chance. 0:07:02.200 --> 0:07:06.320 And science is aware of this, especially with observational studies, 0:07:07.080 --> 0:07:10.280 because the more questions you have, the more opportunity you 0:07:10.320 --> 0:07:13.600 have for that five percent chance to create a seemingly 0:07:13.680 --> 0:07:19.480 statistically significant correlation when really it's not there. It was 0:07:19.520 --> 0:07:22.280 just random chance where if somebody else goes back and 0:07:22.520 --> 0:07:25.360 does the same same study, they're not going to come 0:07:25.440 --> 0:07:30.200 up with the same results. But the if a researcher 0:07:31.240 --> 0:07:36.680 is I would guess willfully blind to that five percent chance, um, 0:07:36.800 --> 0:07:38.960 they will go ahead and produce the study and be like, no, 0:07:39.040 --> 0:07:41.200 it's true, here's the results right here, go ahead and 0:07:41.200 --> 0:07:43.440 report on it and make my career. Yeah. Well, and 0:07:43.440 --> 0:07:45.960 they also might be looking for something inter affect Chances 0:07:45.960 --> 0:07:50.360 are they are, Um, it's not just some random studying. Look, 0:07:50.440 --> 0:07:51.720 let's just see what we get if we ask a 0:07:51.720 --> 0:07:54.040 bunch of weird questions. It's like, hey, we're looking to 0:07:54.040 --> 0:07:57.679 try and prove something most likely so that better minehoff 0:07:57.720 --> 0:07:59.920 thing might come into play where you're kind of chair 0:08:00.160 --> 0:08:03.600 picking data. Yeah, that's a big problem that kind of 0:08:03.640 --> 0:08:05.280 comes up. A lot of these are really kind of 0:08:05.320 --> 0:08:08.200 interrelated to totally. The other big thing that's in related 0:08:08.280 --> 0:08:11.680 is how the media reports on science these days. Yeah, 0:08:11.720 --> 0:08:14.840 you know, he's a big deal. Yeah, Like in John 0:08:14.880 --> 0:08:17.480 Oliver just recently went off on this and NPR did 0:08:17.480 --> 0:08:21.120 a thing on it, like they might even like the 0:08:21.160 --> 0:08:23.920 researcher might say plausible, but it doesn't get portrayed that 0:08:23.920 --> 0:08:27.240 way in the media. Sure, remember that poor kid who 0:08:27.280 --> 0:08:31.360 thought he found the ancient Mayan city. The media just 0:08:31.400 --> 0:08:33.840 took it and ran with it. You know, yeah, I 0:08:33.840 --> 0:08:36.640 think there was a lot of maybe or it's possible, 0:08:36.760 --> 0:08:38.480 we need to go check kind of thing. That mean, 0:08:38.520 --> 0:08:41.240 He's like, no, he discovered an ancient Mayan city never 0:08:41.280 --> 0:08:43.439 known before. Yeah, and let's put it in a headline. 0:08:43.520 --> 0:08:45.720 And that's I mean, that's the That's just kind of 0:08:45.760 --> 0:08:48.559 the way it is these days. Do you have to 0:08:48.600 --> 0:08:50.160 be able to sort through it? I guess that's what 0:08:50.200 --> 0:08:52.760 we're doing here, aren't we, Chuck, We're telling everybody how 0:08:52.800 --> 0:08:55.120 to sort through it, or at the very least take 0:08:55.520 --> 0:09:00.320 scientific reporting with a grain of salt, right, Not like 0:09:00.360 --> 0:09:03.880 you don't necessarily have the time to go through and 0:09:04.000 --> 0:09:06.640 double that research and then check on that research and 0:09:07.640 --> 0:09:09.400 you know, right, so take it with a grain of salt. 0:09:10.640 --> 0:09:16.840 Um unsound samples. Uh, here was this study that basically said, um, 0:09:17.200 --> 0:09:20.040 how you lost your virginity. It's going to have a 0:09:20.160 --> 0:09:24.600 very large impact and play a role on how you 0:09:24.679 --> 0:09:27.640 feel about sex and experienced sex for the rest of 0:09:27.679 --> 0:09:33.320 your life. Yeah, it's possible. Sure, it seems logical, so 0:09:33.440 --> 0:09:37.240 we'll just go with it. But when you um only 0:09:38.040 --> 0:09:44.240 interview college students and uh, you don't you only interview 0:09:44.720 --> 0:09:47.760 heterosexual people, Then you can't really say you've done a 0:09:47.840 --> 0:09:50.840 robust study, now, can't you. Plus you also take out 0:09:50.840 --> 0:09:55.480 of the sample size your sample population, anybody who reports 0:09:55.520 --> 0:09:59.720 having had a violent encounter, Throw them out, that data out. 0:10:00.080 --> 0:10:02.400 That's not gonna inform how you feel about sex, right exactly. 0:10:02.440 --> 0:10:05.320 You're just narrowing it down further and further and again, 0:10:05.480 --> 0:10:08.839 cherry picking the data by throwing people out of your 0:10:08.880 --> 0:10:12.120 population sample that don't that will throw off the data 0:10:12.160 --> 0:10:14.320 that you want. Yeah, and I've never heard of this 0:10:14.400 --> 0:10:18.520 acronym weird and um. A lot of these studies are 0:10:18.559 --> 0:10:21.960 conducted by professors in academics. So a lot of times 0:10:21.960 --> 0:10:25.160 you've got college students as your sample, and there's something 0:10:25.160 --> 0:10:31.600 called weird Western educated from industrialized rich in democratic countries. Right, 0:10:31.600 --> 0:10:34.840 those are the participants in the studies studies subject. But 0:10:34.840 --> 0:10:39.280 then they will say, men, right, well, what about the 0:10:39.400 --> 0:10:44.040 gay man in Africa? Like you didn't ask him? So 0:10:44.120 --> 0:10:47.040 that was that's actually a really really big deal. Um. 0:10:47.080 --> 0:10:50.120 In two thousand and ten, these three researchers did a 0:10:50.640 --> 0:10:55.479 a survey of a ton of social science and behavioral 0:10:55.520 --> 0:11:00.240 science studies found that eight percent of them used your 0:11:00.600 --> 0:11:03.920 study participants. So basically it was college kids for eighty 0:11:04.080 --> 0:11:07.960 percent of these papers. And they surveyed a bunch of 0:11:07.960 --> 0:11:10.720 papers and they took it a little further and they 0:11:10.760 --> 0:11:15.359 said that, um, people who fit into the weird category 0:11:15.760 --> 0:11:18.320 only make up twelve percent of the world population, but 0:11:18.360 --> 0:11:21.400 they represent eight percent of the population of these studies. 0:11:21.800 --> 0:11:27.800 And a college student, chuck in North America, Europe, Israel, 0:11:28.040 --> 0:11:32.520 or Australia is four thousand times more likely to be 0:11:32.640 --> 0:11:35.840 in a scientific study than anyone else on the planet. 0:11:36.440 --> 0:11:40.880 And they're basing psychology and behavioral sciences are basing their 0:11:40.920 --> 0:11:45.600 findings onto everybody else based on this this small tranch 0:11:45.920 --> 0:11:49.400 of humanity. And that's a that's a big problem. That's 0:11:49.400 --> 0:11:52.680 extremely misleading. Yeah, and it's also a little insulting because 0:11:52.760 --> 0:11:56.360 what they're essentially saying is like, this is who matters. 0:11:57.200 --> 0:12:00.280 Well also, yeah, but what's sad is this is who 0:12:00.440 --> 0:12:03.439 I am going to go to the trouble of recruiting 0:12:03.520 --> 0:12:07.760 for my study. It's just sheer laziness. And I'm sure 0:12:07.760 --> 0:12:08.960 a lot of them are like, well, I don't have 0:12:09.000 --> 0:12:13.040 the funding to do that. I guess I see that. 0:12:13.080 --> 0:12:16.400 But at the same time, I guarantee there's a tremendous 0:12:16.440 --> 0:12:19.560 amount of laziness involved. Yeah, or maybe if you don't 0:12:19.559 --> 0:12:24.400 have the money, maybe don't do that study. Is it 0:12:24.480 --> 0:12:26.640 that simple? I'm probably over simple plying. I don't know. 0:12:26.720 --> 0:12:28.360 I'm sure we're going to hear from some people in 0:12:28.400 --> 0:12:33.760 academia about this one. We'll stop using weird participants, or 0:12:33.800 --> 0:12:37.880 at the very least say, um, like this is sexual 0:12:38.360 --> 0:12:43.680 Dartmouth students. This applies to them, not everybody in the world. 0:12:46.000 --> 0:12:50.160 Of these studies where used those people as study participants, 0:12:50.200 --> 0:12:52.720 and they're not even they're not even emblematic of the 0:12:52.720 --> 0:12:56.160 rest of the human race. Like college students are shown 0:12:56.480 --> 0:13:00.720 to see the world differently than other people around the world. 0:13:01.280 --> 0:13:03.200 And so it's not like you can be like, well, 0:13:03.240 --> 0:13:06.000 it still works, you can still extrapolate. It's like flawed 0:13:06.080 --> 0:13:09.680 in every way, shape and form. Probably take a break. Come, yeah, 0:13:09.720 --> 0:13:12.040 let's take a break because, uh, you're get a little 0:13:12.080 --> 0:13:15.280 hot under the collar. I love it, man. Uh, we'll 0:13:15.320 --> 0:13:40.640 be right back after this. Just so much, All right, 0:13:40.640 --> 0:13:47.559 what's next, buddy? Uh? Very small sample sizes. Right, if 0:13:47.600 --> 0:13:53.600 you do a study with twenty mice, then you're not 0:13:54.160 --> 0:13:58.120 doing a good enough study. No, So They used this 0:13:58.280 --> 0:14:02.640 um in the the article. They use the idea of 0:14:02.679 --> 0:14:07.480 ten thousand smokers and ten thousand non smokers, and they said, okay, 0:14:07.520 --> 0:14:11.080 if you have a population sample that size, that's not bad. 0:14:11.120 --> 0:14:13.400 It's a pretty good start. And you find that fifty 0:14:13.800 --> 0:14:16.360 of the smokers developed lung cancer but only five percent 0:14:16.440 --> 0:14:19.760 of non smokers did, then your study has what's called 0:14:19.880 --> 0:14:24.520 a high power. UM. It's if if you had something 0:14:24.560 --> 0:14:27.400 like ten smokers and ten non smokers, and two of 0:14:27.480 --> 0:14:30.920 the smokers developed lung cancer and one developed lung cancer 0:14:30.960 --> 0:14:35.160 as well, you have very little power and you should 0:14:35.160 --> 0:14:38.960 have a very little confidence in your findings. But regardless, 0:14:39.400 --> 0:14:41.960 it's still going to get reported if it's a sexy 0:14:42.000 --> 0:14:47.200 idea yea for sure. Um. And because these are kind 0:14:47.200 --> 0:14:49.400 of overlapping in a lot of ways, it was I 0:14:49.440 --> 0:14:53.080 want to mention this guy, a scientist named Ulrich uh 0:14:53.360 --> 0:14:57.440 Dirnegle Uh. He and his colleague Malcolm McCloud have been trying, 0:14:57.600 --> 0:14:59.200 I mean, and there are a lot of scientists that 0:14:59.240 --> 0:15:01.760 are trying to clean this up because they know it's 0:15:01.760 --> 0:15:04.560 a problem. But he co wrote an article in Nature. 0:15:04.920 --> 0:15:10.160 Uh that's called robust research. Colon institutions must do their 0:15:10.200 --> 0:15:14.000 part for reproducibility. So this kind of ties back into 0:15:14.000 --> 0:15:18.120 the reproducing things, like we said earlier, and his whole ideas, 0:15:18.160 --> 0:15:21.360 you know what good funding they should tie funding too 0:15:21.360 --> 0:15:25.160 good institutional practices, like you shouldn't get the money if 0:15:25.160 --> 0:15:28.920 you can't show that you're doing it right. Um, and 0:15:28.960 --> 0:15:30.800 he said that would just weed out a lot of stuff. 0:15:31.520 --> 0:15:36.680 Here's one staggering stat for reproducibility and small, simple size. Uh. 0:15:36.880 --> 0:15:41.400 Biomedical researchers for drug companies reported that of their only 0:15:42.840 --> 0:15:45.840 of the papers that they published or even reproducible, that 0:15:46.000 --> 0:15:50.560 was like an insider stat And it doesn't matter. They 0:15:50.640 --> 0:15:54.000 still drugs are still going to market, yeah, which is 0:15:54.040 --> 0:15:56.800 that's a really good example of why this does matter 0:15:56.880 --> 0:15:59.720 to the average person. You know, like if you hear 0:15:59.800 --> 0:16:05.240 something thing like um uh, monkeys like to cuddle with 0:16:05.320 --> 0:16:10.040 one another because they are reminded of their mothers. Study shows. 0:16:10.720 --> 0:16:12.600 Do you could just be like, oh, that's great, I'm 0:16:12.600 --> 0:16:15.360 going to share that on the internet. Doesn't really affect 0:16:15.400 --> 0:16:19.200 you in any way. But when their studies being conducted 0:16:19.240 --> 0:16:23.840 that are that are creating drugs that could kill you 0:16:24.000 --> 0:16:26.680 or not treat you or that kind of thing, And 0:16:26.840 --> 0:16:30.680 is it's attracting money and funding and that kind of 0:16:30.680 --> 0:16:36.960 stuff that's like that's harmful. Yeah. Absolutely. I found another survey, 0:16:37.160 --> 0:16:40.800 did you like that terrible study idea? That it came up? 0:16:41.600 --> 0:16:47.480 The monkeys like the cuddle A hundred and forty trainees 0:16:47.520 --> 0:16:52.360 at the MD Anderson Cancer Center in Houston, Texas, Thank 0:16:52.400 --> 0:16:54.560 you Houston for being so kind to us at a 0:16:54.640 --> 0:16:58.680 recent show. They found that nearly a third of these 0:16:58.880 --> 0:17:04.600 UM trainees felt pressure to support their mentors work. It's 0:17:04.640 --> 0:17:07.440 like to get ahead or not get fired. So that's 0:17:07.440 --> 0:17:11.080 another issue. As you've got these trainees or residents, uh, 0:17:11.080 --> 0:17:13.160 and you have these mentors, and even if you disagree 0:17:13.280 --> 0:17:15.880 or don't think it's a great study, you're you're pressured 0:17:15.920 --> 0:17:18.080 into just going along with it. I could see that 0:17:18.720 --> 0:17:21.159 for sure. There's there seems to be a huge hierarchy. 0:17:21.240 --> 0:17:24.439 And UM science can a lab. You know, you've got 0:17:24.480 --> 0:17:26.600 the person who runs the lab, it's their lab, and 0:17:27.000 --> 0:17:30.720 go against them. But there are people UM like Science 0:17:30.840 --> 0:17:34.199 and Nature to great journals are updating their guidelines right now. 0:17:34.280 --> 0:17:39.960 They're introducing checklists. UM Science hired statisticians to their panel 0:17:40.760 --> 0:17:44.320 of reviewing editors, not just other you know, peer reviewed 0:17:44.600 --> 0:17:49.520 like they actually actually hired numbers people specifically because that's 0:17:49.520 --> 0:17:52.240 a bigger process. That's a huge part of studies. It's 0:17:52.280 --> 0:17:56.639 like these this mind breaking statistical analysis that can be 0:17:56.720 --> 0:17:59.800 used for good or ill. And I mean, I don't 0:17:59.800 --> 0:18:03.840 think the average scientists necessarily is a whiz at that, 0:18:04.160 --> 0:18:06.560 although I it has to be part of training, but 0:18:06.560 --> 0:18:09.360 not necessarily. And that's a different kind of beast altogether. 0:18:09.960 --> 0:18:12.880 Um Stats, we talked about it earlier. I took a 0:18:12.920 --> 0:18:16.119 stats class in college, had so much trouble that was 0:18:16.200 --> 0:18:19.640 awful at it. It really just it's a special kind 0:18:19.640 --> 0:18:25.240 of is it even matter? Yeah, I didn't get it. 0:18:25.960 --> 0:18:29.800 I passed it, though I passed it because my professor 0:18:29.880 --> 0:18:37.359 took pity on me. Um that Ulric during durnagal Ulric 0:18:37.680 --> 0:18:42.359 narnagal Um. He is a he's a big time crusader 0:18:42.440 --> 0:18:46.800 for his jam making sure that science is good science. 0:18:47.119 --> 0:18:49.520 One of the things um he crusades against is the 0:18:49.640 --> 0:18:52.840 idea of remembering that virginity study where they just threw 0:18:52.880 --> 0:18:55.560 out anybody who had a violent encounter for their first 0:18:56.160 --> 0:18:59.960 sexual experience. UM. Apparently that's a big deal with annam 0:19:00.320 --> 0:19:02.720 studies as well. If you're studying the effects of a 0:19:02.800 --> 0:19:06.120 drug or something like there was this one in the article. Um, 0:19:06.160 --> 0:19:09.600 if you're studying the effects of a stroke drug and 0:19:09.680 --> 0:19:12.680 you've got a control group of mice that are taking 0:19:12.720 --> 0:19:15.000 the drug or that aren't taking the drug, and then 0:19:15.000 --> 0:19:18.080 a test group that are getting the drug. Um, and 0:19:18.080 --> 0:19:22.360 then like three mice from the test group die even 0:19:22.359 --> 0:19:24.040 though they're on the stroke drug. They die of a 0:19:24.080 --> 0:19:27.880 massive stroke, and you just literally and figuratively throw them 0:19:27.920 --> 0:19:30.879 out of the study, um, and don't include them in 0:19:30.920 --> 0:19:34.879 the results. That changes the data. And he's been on 0:19:35.440 --> 0:19:37.600 a peer of you on a paper before. He's like, no, 0:19:37.720 --> 0:19:39.880 this doesn't pass peer of you. You can't just throw 0:19:39.960 --> 0:19:42.680 out what happened to these three rodents? You started with ten, 0:19:42.680 --> 0:19:45.720 there's only seven reported in the end. What happened to 0:19:45.760 --> 0:19:49.040 those three? And how many of them just don't report 0:19:49.119 --> 0:19:51.879 the ten? They're like, oh, we only started with seven. 0:19:51.920 --> 0:19:55.080 Were going, you know, well, I was about to say 0:19:55.080 --> 0:19:56.800 I get the urge. I don't get it because it's 0:19:56.800 --> 0:19:58.960 not right. But I think what happens is you work 0:19:59.040 --> 0:20:01.480 so hard at something yeah yeah, and you're like, how 0:20:01.520 --> 0:20:04.000 can I just walk away from two years of this 0:20:04.359 --> 0:20:08.199 because it didn't get a result? Okay, The point of 0:20:08.320 --> 0:20:11.160 real science, though you have to walk away from it. Well, 0:20:11.280 --> 0:20:14.320 you have to publish that. And that's the other thing too, 0:20:14.320 --> 0:20:18.159 And I guarantee scientists will say, hey, man, try getting 0:20:18.160 --> 0:20:21.120 a negative paper published in a good journal. These days, 0:20:21.200 --> 0:20:23.399 you don't want that kind of stuff. But part of 0:20:23.400 --> 0:20:25.960 it also is I don't think it's enough to just 0:20:26.000 --> 0:20:28.880 have to be published in like a journal. You want 0:20:28.880 --> 0:20:31.000 to make the news cycle as well. That makes it 0:20:31.040 --> 0:20:34.000 even better, right, Um So, I think there's a lot 0:20:34.040 --> 0:20:36.760 of factors involved. But ultimately, if you take all that 0:20:36.800 --> 0:20:38.960 stuff away, if you take the culture away from it, 0:20:39.520 --> 0:20:42.960 you're if you get negative results, you're supposed to publish 0:20:43.000 --> 0:20:45.199 that so that some other scientists can come along and 0:20:45.240 --> 0:20:48.480 be like, oh, somebody else already did this using these 0:20:48.520 --> 0:20:50.520 methods that I was going to use. I'm not gonna 0:20:50.560 --> 0:20:54.040 waste two years of my career because somebody else already did. 0:20:54.280 --> 0:20:57.199 Thank you, buddy for saving me this time and trouble 0:20:57.200 --> 0:21:00.120 and effort to know that this does not work. You've 0:21:00.160 --> 0:21:02.880 proven this doesn't work when you saw it to prove 0:21:02.920 --> 0:21:04.960 it does work, you actually proved it didn't work. That's 0:21:05.000 --> 0:21:09.760 part of science. Yeah, I wish there wasn't a negative 0:21:10.720 --> 0:21:14.359 connotation to a negative result, because to me, it's the 0:21:14.440 --> 0:21:17.479 value is the same as proving something does work as 0:21:17.520 --> 0:21:22.439 proving something doesn't work. Right again, it's just not Yeah, 0:21:22.480 --> 0:21:24.879 but I'm not sexy either, so maybe that's why I 0:21:24.880 --> 0:21:27.359 get it. Uh. Here's one that I didn't know was 0:21:27.400 --> 0:21:31.359 a thing, predatory publishing. You never heard of this. So 0:21:31.520 --> 0:21:35.720 here's the scenario. You're a doctor or a scientists and um, 0:21:35.840 --> 0:21:39.719 you get an email from a journal that says, hey, 0:21:39.920 --> 0:21:42.080 you got anything interesting for us. I've heard about your 0:21:42.080 --> 0:21:43.959 work and you say, well, actually, do I have this 0:21:44.760 --> 0:21:47.320 this study right here? They say, cool, we'll publish it. 0:21:47.560 --> 0:21:50.159 You go great, my career is taking off. Then you 0:21:50.200 --> 0:21:54.240 get a bill he says, where's my three grand for 0:21:54.280 --> 0:21:57.040 publishing your article? And you're like, I don't owe you 0:21:57.080 --> 0:22:00.560 three grand, all right, give us two? And you know 0:22:00.560 --> 0:22:02.360 I can't even give you two. And if you fight 0:22:02.440 --> 0:22:06.040 them long enough, maybe they'll drop it and never work 0:22:06.080 --> 0:22:09.160 with you again. Or maybe it'll just be like, well, 0:22:09.200 --> 0:22:13.920 we'll talk to your next court. Exactly. That's called predatory publishing. 0:22:13.920 --> 0:22:16.159 And it is I'm not sure how new it is. 0:22:16.560 --> 0:22:19.120 Maybe it's pretty new. Is it pretty new? But it's 0:22:19.119 --> 0:22:24.600 a thing now where uh, you can pay essentially to 0:22:24.760 --> 0:22:29.840 get something published. Yes, you can, um it kind of 0:22:29.880 --> 0:22:32.320 it's kind of like who's who in behavioral science is 0:22:32.440 --> 0:22:35.960 kind of thing, you know. Um. And apparently it's new 0:22:36.000 --> 0:22:39.720 because it's a result of open source academic journals, which 0:22:39.720 --> 0:22:43.360 a lot of people push for, including Aaron Schwartz very 0:22:43.400 --> 0:22:46.280 famously who like took a bunch of academic articles and 0:22:46.280 --> 0:22:50.320 published him online and was prosecuted heavily for it. Persecuted. 0:22:50.400 --> 0:22:53.399 You could even say, um, but the idea that science 0:22:53.440 --> 0:22:56.800 is behind this paywall, which is another great article from 0:22:56.800 --> 0:23:00.480 Priceonomics by the way, um, really just takes a lot 0:23:00.480 --> 0:23:03.600 of people off. So they started open source journals right, 0:23:04.160 --> 0:23:07.920 and as a result, predatory publishers came about and said, okay, yeah, 0:23:08.040 --> 0:23:10.679 let's make this free, but we need to make our 0:23:10.720 --> 0:23:13.919 money anyway, so we're going to charge the academic who 0:23:14.000 --> 0:23:16.760 wrote the study for publishing it. Well yeah, and and 0:23:16.920 --> 0:23:22.040 sometimes now it's just a flat out scam operation. There's 0:23:22.040 --> 0:23:26.280 this guy named Jeffrey Beale who is a research librarian. 0:23:26.400 --> 0:23:30.240 He is my new hero because he's truly like one 0:23:30.280 --> 0:23:33.879 of these dudes that has uh, he's trying to make 0:23:33.880 --> 0:23:36.840 a difference and he's not profiting from this, but he's 0:23:36.920 --> 0:23:41.760 spending a lot of time by creating a list of 0:23:41.760 --> 0:23:45.960 of predatory publishers. Yeah, a significant list too. Yeah, how 0:23:45.960 --> 0:23:51.600 many four thousand of them right now? Um. Some of 0:23:51.640 --> 0:23:54.840 these companies flat out lie, like they're literally based out 0:23:54.840 --> 0:23:57.760 of Pakistan or Nigeria and they say no, we're in 0:23:57.800 --> 0:24:01.919 New York. Yea publisher, So it's just a flat out scam. 0:24:02.040 --> 0:24:05.359 Or they lie about their review practices. Um, like they 0:24:05.440 --> 0:24:08.439 might not have any review practices and they straight up 0:24:08.520 --> 0:24:11.359