WEBVTT - SYSK Selects: Research tips from SYSK 0:00:00.040 --> 0:00:04.080 M Hey everybody, it's me Josh and for this week's 0:00:04.200 --> 0:00:07.240 s Y s K Selects, I've chosen our guide to 0:00:07.440 --> 0:00:11.800 research tips. It's a surprisingly good episode that shares the 0:00:11.840 --> 0:00:15.000 ins and outs of keeping from being duped online by 0:00:15.040 --> 0:00:18.119 bad information and how to read between the lines on 0:00:18.200 --> 0:00:22.079 sensational science reporting, all sorts of stuff like that. And 0:00:22.120 --> 0:00:25.160 you might notice in this episode Chuck sounds different than usual. 0:00:25.480 --> 0:00:27.560 That's because this is during the period that he was 0:00:27.600 --> 0:00:30.480 transitioning into a person with a full set of teeth, 0:00:31.360 --> 0:00:33.199 so that adds to the hilarity of the whole thing. 0:00:33.479 --> 0:00:35.800 I hope you enjoyed this as much as we did, 0:00:36.240 --> 0:00:42.599 making it. Welcome to Stuff You Should Know, a production 0:00:42.640 --> 0:00:50.880 of My Heart Radios Have Stuff Works. Hey, and welcome 0:00:50.920 --> 0:00:53.559 to the podcast. I'm Josh Clark with Charles W. Chuck 0:00:53.640 --> 0:01:00.600 Bryant and Jerry This Stuff you Should Know. Um, Josh, 0:01:00.640 --> 0:01:03.480 we're gonna do something weird today and we're gonna do 0:01:03.520 --> 0:01:05.840 a listener mail at the head of the podcast, right, 0:01:05.880 --> 0:01:10.759 all right, what, all right, let's do it. Okay, this 0:01:10.840 --> 0:01:14.200 is from being the listener mail music going. Oh, I 0:01:14.200 --> 0:01:18.720 don't know, should we go the whole nine yards? People 0:01:18.800 --> 0:01:23.000 might freak out all right, this is from Bianca. Uh 0:01:23.840 --> 0:01:26.720 VoiceAge is what I'm gonna say. I think that's great. Hey, 0:01:26.720 --> 0:01:28.640 guys wrote you not too long ago asking about Hey 0:01:28.720 --> 0:01:31.360 research your own podcast. It just got back from a 0:01:31.400 --> 0:01:36.240 class where we talked about research misrepresentation and journal articles. 0:01:36.440 --> 0:01:39.280 Apparently journals don't publish everything that is submitted. A lot 0:01:39.280 --> 0:01:41.800 of researchers don't even publish their studies they don't like 0:01:41.840 --> 0:01:44.400 the results. Some laws have been put into place to 0:01:44.440 --> 0:01:48.720 prevent misrepresentation, such as researchers having to register their studies 0:01:48.760 --> 0:01:52.600 before they get results and journals only accepting preregistered studies. 0:01:52.640 --> 0:01:55.480 But apparently this is not happening at all, even though 0:01:55.560 --> 0:01:58.360 it is now technically law. This ends with the general 0:01:58.400 --> 0:02:01.640 public being missing for about methods and drugs that work. 0:02:01.880 --> 0:02:04.720 For example, there are twenty five studies proving a drug 0:02:04.760 --> 0:02:09.440 works that don't. It's more likely that twenty of the 0:02:09.480 --> 0:02:11.880 positive results have been published and only one or two 0:02:11.919 --> 0:02:16.040 of the negative. Uh. And that is from Bianca, and 0:02:16.160 --> 0:02:20.120 that led us to this article on our own website. 0:02:20.560 --> 0:02:23.840 Ten signs that that study is bogus and here it 0:02:23.880 --> 0:02:26.720 is nice Chuck. Well, we get asked a lot about 0:02:26.760 --> 0:02:29.919 research from people, usually in college are like you guys 0:02:29.960 --> 0:02:33.360 are professional researchers. How do I know I'm doing a 0:02:33.360 --> 0:02:36.959 good job and getting good info? And it's getting harder 0:02:36.960 --> 0:02:39.959 and harder these days, it really is. You know. One 0:02:40.000 --> 0:02:43.080 sign that I've learned is if you are searching about 0:02:43.080 --> 0:02:47.800 his study and all of the hits that come back 0:02:48.480 --> 0:02:51.760 are from different news organizations and they're all within like 0:02:51.760 --> 0:02:55.680 a two three day period from a year ago, nothing like, 0:02:55.960 --> 0:02:59.560 nothing more recent than that than somebody released the sensational 0:02:59.600 --> 0:03:04.000 study and no one did any actual effort into investigating it, 0:03:04.040 --> 0:03:06.639 and there was no follow up. If you dig deep enough, 0:03:06.720 --> 0:03:08.640 somebody might have done follow up or something like that, 0:03:08.639 --> 0:03:11.280 but for the most part, it was just something that 0:03:11.400 --> 0:03:14.440 splashed across the headlines, which more often than not is 0:03:14.520 --> 0:03:17.000 the is the case as far as science reporting goes. 0:03:17.320 --> 0:03:22.840 So that's a bonus, that's the eleventh boom. How about that? Yeah, 0:03:23.080 --> 0:03:25.519 should we just start banging these out, let's do it? 0:03:25.600 --> 0:03:30.519 Or do you have some other clever well apart and 0:03:30.600 --> 0:03:32.840 parcel with that. I don't know. If it's clever, you 0:03:32.919 --> 0:03:35.600 do come across people who you know can be trusted 0:03:35.640 --> 0:03:39.640 and relied upon to do good science reporting. So like 0:03:39.800 --> 0:03:43.440 ed Young is one another guy named Ben gold Acre 0:03:43.520 --> 0:03:46.520 has something called bad science. I don't remember what outlet 0:03:46.560 --> 0:03:49.000 he's with. And then there's a guy I think scientific 0:03:49.000 --> 0:03:53.160 American named John Horgan who's awesome. Yeah. Or some journalism 0:03:53.280 --> 0:03:56.040 organizations that have been around and stood the test of 0:03:56.120 --> 0:03:57.880 time that you know are really doing it right, like 0:03:58.200 --> 0:04:02.920 a nature yeah, scientific amor kids are like really science. Yeah, 0:04:03.000 --> 0:04:05.840 Like I feel I feel really good about using those sources. Yeah, 0:04:05.880 --> 0:04:08.760 but even they can you know, there's there's something called 0:04:08.840 --> 0:04:11.760 scientis um where there's a lot of like faith in 0:04:11.880 --> 0:04:15.440 dogma associated with the scientific process, and you know you 0:04:15.480 --> 0:04:19.880 have to root through that as well. Try it. I'm done. Uh. 0:04:20.040 --> 0:04:21.839 The first one that they have here on the list 0:04:21.920 --> 0:04:25.479 is that it's unrepeatable, and that's a big one. UM. 0:04:25.560 --> 0:04:29.080 The Center for Open Science did a study, uh was 0:04:29.120 --> 0:04:31.760 a project really where they took two hundred and seventy 0:04:32.000 --> 0:04:34.400 researchers and they said, you know what, take these one 0:04:34.440 --> 0:04:39.000 hundred studies that have been published already psychological studies and 0:04:39.120 --> 0:04:42.839 just pour over them. And uh, in just last year. 0:04:43.000 --> 0:04:45.440 It took him a while, took them several years. They said, 0:04:45.440 --> 0:04:47.520 you know what, more than half of these can't even 0:04:47.560 --> 0:04:51.119 be repeated using the same methods. They're not reproducible. Nope, 0:04:51.160 --> 0:04:54.960 not reproducible. That's a big one. And and ones that 0:04:55.040 --> 0:04:57.680 means that they when they carried out they followed the 0:04:57.720 --> 0:05:01.520 methodology UM signed sific method podcasts. You should listen to 0:05:01.600 --> 0:05:03.880 that one. That was a good one that they they 0:05:03.920 --> 0:05:06.680 found that their results were just not what the what 0:05:06.720 --> 0:05:10.239 the people published, not anywhere near them. UM. For example, 0:05:10.279 --> 0:05:14.000 they used one as an example where a study found 0:05:14.040 --> 0:05:19.599 that men were terrible at a determining whether a woman 0:05:19.760 --> 0:05:23.320 was giving them um, some sort of like a clues 0:05:23.360 --> 0:05:27.520 to attraction or just being friendly sexy sexy stuff or 0:05:28.040 --> 0:05:30.640 friends or yeah or good to meet you or buzz 0:05:30.640 --> 0:05:34.559 off jerk yeah. Um. And they did the study again 0:05:34.600 --> 0:05:37.360 and as part of this Open Science Center for Open 0:05:37.400 --> 0:05:41.160 Science study your survey, and they found that that was 0:05:41.760 --> 0:05:45.800 not reproducible or that they came up with totally different results. 0:05:45.800 --> 0:05:47.720 And that was just one of many. Yeah. And in 0:05:47.760 --> 0:05:50.480 this case specifically, they looked into that study and they 0:05:50.520 --> 0:05:53.960 found that it was UM. One was in the United Kingdom, 0:05:54.040 --> 0:05:56.159 one was in the United States. May have something to 0:05:56.160 --> 0:05:58.320 do with it. But the point is, Chuck, is if 0:05:58.360 --> 0:06:02.480 you're talking about humanity, I don't think that the study 0:06:02.600 --> 0:06:05.880 was like the American male is terrible at it. It's 0:06:06.080 --> 0:06:08.200 men are terrible at it. Right. So that means that 0:06:08.240 --> 0:06:10.840 whether it's in the UK, which is basically the US 0:06:11.320 --> 0:06:15.040 with an accent and a penchant for t I'm just 0:06:15.120 --> 0:06:21.880 kidding UK s soon, UM, the it should be universal. Yeah, 0:06:22.040 --> 0:06:24.560 you know, I agreed, unless you're saying no, it's just 0:06:24.760 --> 0:06:28.680 this only applies to American men, right, or the American 0:06:28.720 --> 0:06:33.159 men right, then it's not even study. Yeah. Uh. The 0:06:33.200 --> 0:06:38.840 next one we have is, uh, it's it's plausible, not 0:06:38.880 --> 0:06:42.640 necessarily provable. And this is a big one because and 0:06:42.680 --> 0:06:45.680 I think, um, we're talking about observational studies here more 0:06:45.720 --> 0:06:48.960 than lab experiments, because with observational studies, you know, you 0:06:49.040 --> 0:06:52.320 sit in a room and get asked three questions about something, 0:06:53.000 --> 0:06:54.880 and all these people get asked the same questions, and 0:06:54.880 --> 0:06:57.520 then they pour over the data and they draw out 0:06:57.560 --> 0:07:02.160 their own observations. And one very famously, an observational study 0:07:02.160 --> 0:07:05.640 that led to false results found a correlation between having 0:07:05.640 --> 0:07:10.240 a type A personality and um, being prone to risk 0:07:10.320 --> 0:07:14.440 for heart attacks and um for a long time, you 0:07:14.480 --> 0:07:16.920 know that the news outlets were like, oh, yes, of 0:07:16.960 --> 0:07:19.840 course that makes total sense. This study proved what we've 0:07:19.840 --> 0:07:23.360 all known all along, um, And then it came out 0:07:23.480 --> 0:07:27.160 that no, actually what was going on was a well 0:07:27.200 --> 0:07:33.240 known anomaly where you have a five percent um risk 0:07:33.320 --> 0:07:37.120 that chance will produce something that looks like a statistically 0:07:37.200 --> 0:07:43.000 significant correlation when really it's just total chance. And science 0:07:43.080 --> 0:07:46.880 is aware of this, especially with observational studies, because the 0:07:46.960 --> 0:07:49.840 more questions you have, the more opportunity you have for 0:07:49.920 --> 0:07:54.560 that five percent chance to create a seemingly statistically significant 0:07:54.920 --> 0:07:59.679 correlation when really it's not there. It was just random chance, 0:07:59.720 --> 0:08:02.560 where if somebody else goes back and does the same 0:08:02.760 --> 0:08:04.960 same study, they're not going to come up with the 0:08:04.960 --> 0:08:11.400 same results. But the if a researcher is I would guess, 0:08:11.440 --> 0:08:16.360 willfully blind to that five percent chance, um, they will 0:08:16.360 --> 0:08:18.200 go ahead and produce the study and be like, no, 0:08:18.280 --> 0:08:20.440 it's true, here's the results right here, go ahead and 0:08:20.440 --> 0:08:22.680 report on it and make my career. Yeah. Well, and 0:08:22.680 --> 0:08:25.200 they also might be looking for something in affect chances 0:08:25.240 --> 0:08:29.360 are they are, Um, it's not just some random studying. 0:08:29.680 --> 0:08:30.960 Let's just see what we get if we ask a 0:08:30.960 --> 0:08:33.280 bunch of weird questions. It's like, hey, we're looking to 0:08:33.280 --> 0:08:36.920 try and prove something most likely, so that better minehoff 0:08:36.960 --> 0:08:39.320 thing might come into play where you're kind of cherry 0:08:39.360 --> 0:08:42.840 picking data. Yeah, that's a big problem that kind of 0:08:42.880 --> 0:08:44.520 comes up. A lot of these are really kind of 0:08:44.559 --> 0:08:47.440 interrelated to totally. The other big thing that's in related 0:08:47.520 --> 0:08:50.880 is how the media reports on science these days. Yeah, 0:08:50.960 --> 0:08:54.400 you know, he's a big deal. Yeah, Like John Oliver 0:08:54.559 --> 0:08:56.800 just recently went off on this and NPR did a 0:08:56.840 --> 0:09:00.840 thing on it, like they might even like the researcher 0:09:00.920 --> 0:09:03.719 might say plausible, but it doesn't get portrayed that way 0:09:03.880 --> 0:09:06.720 in the media. Sure. Remember that poor kid who thought 0:09:06.800 --> 0:09:10.840 he found the ancient Mayan city. The media just took 0:09:10.840 --> 0:09:13.280 it and ran with it. You know, Yeah, I think 0:09:13.280 --> 0:09:16.080 there was a lot of maybe or it's possible, we 0:09:16.120 --> 0:09:18.160 need to go check kind of thing. That mean, he's like, no, 0:09:18.480 --> 0:09:21.480 he discovered an ancient Mayan city never known before. Yeah, 0:09:21.520 --> 0:09:23.480 and let's put it in a headline. And that's I mean, 0:09:23.520 --> 0:09:25.600 that's the That's just kind of the way it is 0:09:25.679 --> 0:09:28.520 these days. Do you have to be able to sort 0:09:28.559 --> 0:09:30.680 through And I guess that's what we're doing here, aren't we, Chuck, 0:09:30.679 --> 0:09:33.120 We're telling everybody how to sort through it, or at 0:09:33.120 --> 0:09:38.920 the very least take scientific reporting with a grain of salt, yes, right, 0:09:38.920 --> 0:09:41.319 and not well, like you don't necessarily have the time 0:09:41.360 --> 0:09:44.760 to go through and double that research and then check 0:09:44.800 --> 0:09:47.960 on that research, and you know, right, so take it 0:09:48.000 --> 0:09:53.120 with a grain of salt. Um unsound samples. Uh. Here 0:09:53.200 --> 0:09:57.199 was his study that basically said, um, how you lost 0:09:57.240 --> 0:10:00.800 your virginity is going to have a very a large 0:10:00.960 --> 0:10:04.600 impact and play a role on how you feel about 0:10:04.840 --> 0:10:08.239 sex and experienced sex for the rest of your life. Yeah, 0:10:08.520 --> 0:10:13.200 it's possible. Sure, it seems logical, so we'll just go 0:10:13.320 --> 0:10:19.600 with it. But when you, um, only interview college students 0:10:20.400 --> 0:10:25.880 and uh you don't you only interview heterosexual people, then 0:10:25.920 --> 0:10:27.840 you can't really say you've done a robust study, now, 0:10:27.880 --> 0:10:31.439 can't you. Plus you also take out of the sample 0:10:31.520 --> 0:10:35.600 size your sample population, anybody who reports having had a 0:10:35.679 --> 0:10:39.439 violent encounter, Throw them out, that data out, because that's 0:10:39.440 --> 0:10:41.640 not gonna inform how you feel about sex, right exactly. 0:10:41.679 --> 0:10:44.560 You're just narrowing it down further and further and again 0:10:44.720 --> 0:10:48.079 cherry picking the data by throwing people out of your 0:10:48.120 --> 0:10:51.360 population sample that don't that will throw off the data 0:10:51.400 --> 0:10:53.560 that you want. Yeah, and I've never heard of this 0:10:53.640 --> 0:10:57.679 acronym weird and UM. A lot of these studies are 0:10:57.800 --> 0:11:00.920 conducted by professors in acade make so a lot of 0:11:00.960 --> 0:11:04.160 times you've got college students as your sample, and there's 0:11:04.160 --> 0:11:10.839 something called weird Western educated from industrialized, rich in democratic countries. Right, 0:11:10.840 --> 0:11:14.080 those are the participants in the studies studies subject. But 0:11:14.080 --> 0:11:18.559 then they will say, men, right, well, what about the 0:11:18.640 --> 0:11:22.920 gay man in Africa? Right, like you didn't ask him, 0:11:23.120 --> 0:11:26.280 so that was that's actually a really really big deal. Um. 0:11:26.320 --> 0:11:29.240 In two thousand and ten, they these three researchers did 0:11:29.280 --> 0:11:34.719 a survey of a ton of social science and behavioral 0:11:34.760 --> 0:11:39.480 science studies found that eight percent of them used weird 0:11:39.840 --> 0:11:43.200 study participants. So basically it was college kids for eighty 0:11:43.320 --> 0:11:47.200 percent of these papers. And they surveyed a bunch of 0:11:47.200 --> 0:11:49.960 papers and they took it a little further and they 0:11:50.000 --> 0:11:54.599 said that, um, people who fit into the weird category 0:11:55.000 --> 0:11:57.560 only make up twelve percent of the world population, but 0:11:57.600 --> 0:12:00.680 they represent eight percent of the population of these studies. 0:12:01.040 --> 0:12:07.040 And a college student chuck in North America, Europe, Israel, 0:12:07.280 --> 0:12:11.760 or Australia is four thousand times more likely to be 0:12:11.880 --> 0:12:15.079 in a scientific study than anyone else on the planet. 0:12:15.679 --> 0:12:20.120 And they're basing psychology and behavioral sciences are basing their 0:12:20.160 --> 0:12:24.880 findings onto everybody else based on this this small tranche 0:12:25.160 --> 0:12:28.640 of humanity, and that's a that's a big problem. That's 0:12:28.640 --> 0:12:31.959 extremely misleading. Yeah, and it's also a little insulting because 0:12:32.000 --> 0:12:35.559 what they're essentially saying is like, this is who matters. 0:12:36.440 --> 0:12:39.520 Well also, yeah, but what's sad is this is who 0:12:39.679 --> 0:12:42.679 I am going to go to the trouble of recruiting 0:12:42.760 --> 0:12:47.000 for my study. It's just sheer laziness. And I'm sure 0:12:47.000 --> 0:12:48.200 a lot of them are like, well, I don't have 0:12:48.240 --> 0:12:52.280 the funding to do that. I guess I see that, 0:12:52.320 --> 0:12:55.640 But at the same time, I guarantee there's a tremendous 0:12:55.679 --> 0:12:58.800 amount of laziness involved. Yeah, or maybe if you don't 0:12:58.800 --> 0:13:03.640 have the money, maybe don't do that study. Is it 0:13:03.720 --> 0:13:06.120 that simple? I'm probably over simplifying. I don't know. I'm 0:13:06.120 --> 0:13:08.280 sure we're going to hear from some people in academia 0:13:08.280 --> 0:13:13.160 about this one. We'll stop using weird participants, or at 0:13:13.200 --> 0:13:19.400 the very least say, um, like this is sexual dartmouth students. Yeah, 0:13:19.720 --> 0:13:25.400 this applies to them, not everybody in the world of 0:13:25.440 --> 0:13:29.400 these studies where they use those people as study participants 0:13:29.440 --> 0:13:31.960 and they're not even they're not even emblematic of the 0:13:31.960 --> 0:13:35.400 rest of the human race. Like college students are shown 0:13:35.720 --> 0:13:39.960 to see the world differently than other people around the world. 0:13:40.679 --> 0:13:42.640 So it's not like you can be like, well, it's 0:13:42.640 --> 0:13:45.400 still works, you can still extrapolate. It's like flawed in 0:13:45.440 --> 0:13:48.920 every way, shape and form. Probably take a break. Come, yeah, 0:13:48.960 --> 0:13:51.280 let's take a break because you can get a little 0:13:51.280 --> 0:13:54.520 hot under the collar. I love it. Man. Uh, we'll 0:13:54.559 --> 0:14:09.800 be right back after this. Just god so much. Sorry, 0:14:19.720 --> 0:14:25.800 all right, what's next, buddy? Uh? Very small sample sizes. Right, 0:14:26.800 --> 0:14:32.800 if you do a study with twenty mice, then you're 0:14:32.840 --> 0:14:37.400 not doing a good enough study. No, So they used 0:14:37.400 --> 0:14:41.480 this um in the in the article, they use the 0:14:41.600 --> 0:14:45.040 idea of ten thousand smokers and ten thousand non smokers, 0:14:45.920 --> 0:14:49.040 and they said, okay, if you have a population sample 0:14:49.120 --> 0:14:51.560 that size, that's not bad. It's a pretty good start. 0:14:51.920 --> 0:14:54.840 And you find that of the smokers developed lung cancer, 0:14:54.880 --> 0:14:58.360 but only five of non smokers did, then your study 0:14:58.400 --> 0:15:02.840 has what's called a high power Yeah. Um, it's if 0:15:02.880 --> 0:15:06.000 if you had something like ten smokers and ten non smokers, 0:15:06.360 --> 0:15:09.200 and to the smokers developed lung cancer and one developed 0:15:09.680 --> 0:15:13.800 lung cancer. As well, you have very little power, and 0:15:13.960 --> 0:15:18.360 you should have very little confidence in your findings. But regardless, 0:15:18.760 --> 0:15:22.880 it's still going to get reported if it's a sexy idea, Yeah, 0:15:23.160 --> 0:15:27.360 for sure. Um. And because these are kind of overlapping 0:15:27.400 --> 0:15:29.080 in a lot of ways, it was I want to 0:15:29.080 --> 0:15:34.560 mention this guy, a scientist named Ulrich uh Dirnegle Uh. 0:15:34.600 --> 0:15:37.240 He and his colleague Malcolm McCloud have been trying I mean, 0:15:37.360 --> 0:15:39.400 and there are a lot of scientists that are trying 0:15:39.440 --> 0:15:41.640 to clean this up because they know it's a problem. 0:15:42.200 --> 0:15:45.280 But he co wrote an article in Nature. Uh that's 0:15:45.320 --> 0:15:51.880 called robust research colin institutions must do their part for reproduceability. 0:15:51.960 --> 0:15:55.280 So this kind of ties back into the reproducing things, 0:15:55.280 --> 0:15:57.760 like we said earlier, and his whole ideas, you know 0:15:57.800 --> 0:16:00.960 what good funding they should tie fund ing two good 0:16:01.000 --> 0:16:04.640 institutional practices, like you shouldn't get the money if you 0:16:04.680 --> 0:16:08.440 can't show that you're doing it right. Um. And he 0:16:08.480 --> 0:16:10.200 said that would just weed out a lot of stuff. 0:16:10.920 --> 0:16:15.720 Here's one staggering stat for reproducibility and small sample size. 0:16:16.240 --> 0:16:20.800 Biomedical researchers for drug companies reported that of their only 0:16:22.240 --> 0:16:25.120 of the papers that they publish are even reproducible and 0:16:25.200 --> 0:16:29.040 that was like an insider stat and it doesn't matter. 0:16:29.840 --> 0:16:33.320 They's the drugs are still going to market. Yeah, which 0:16:33.320 --> 0:16:35.760 is that's a really good example of why this does 0:16:35.880 --> 0:16:38.880 matter to the average person. You know, like if you 0:16:38.960 --> 0:16:44.640 hear something like um uh, monkeys like to cuddle with 0:16:44.720 --> 0:16:50.000 one another because they are reminded of their mothers. Study shows, right, 0:16:50.080 --> 0:16:51.960 do you can just be like, oh, that's great, I'm 0:16:52.000 --> 0:16:54.760 going to share that on the internet. Doesn't really affect 0:16:54.760 --> 0:16:58.600 you in any way. But when their studies being conducted 0:16:58.640 --> 0:17:03.240 that are that are creating drugs that could kill you 0:17:03.360 --> 0:17:06.080 or not treat you or that kind of thing, And 0:17:06.240 --> 0:17:10.439 is it's attracting money and funding and that kind of stuff, 0:17:10.760 --> 0:17:16.320 that's like that's harmful. Yeah. Absolutely. I found another survey 0:17:16.560 --> 0:17:19.960 did you like that terrible study idea that it came 0:17:20.040 --> 0:17:26.119 up like the monkey's like the cuddle A hundred and 0:17:26.160 --> 0:17:30.560 forty trainees at the MD Anderson Cancer Center in Houston, Texas, 0:17:31.560 --> 0:17:33.879 Thank you Houston for being so kind to us at 0:17:33.920 --> 0:17:37.760 a recent show. They found that nearly a third of 0:17:37.800 --> 0:17:42.800 these um trainees felt pressure to support their mentors work, 0:17:44.040 --> 0:17:46.840 like to get ahead or not get fired. So that's 0:17:46.840 --> 0:17:50.480 another issue. As you've got these trainees or residents, uh, 0:17:50.480 --> 0:17:52.560 and you have these mentors, and even if you disagree 0:17:52.680 --> 0:17:55.280 or don't think it's a great study, you're you're pressured 0:17:55.320 --> 0:17:57.480 into just going along with it. I could see that 0:17:58.119 --> 0:18:00.560 for sure. There's there seems to be a huge hierarchy. 0:18:00.640 --> 0:18:03.880 And UM Science, Ye canna lab. You know you've got 0:18:03.880 --> 0:18:06.000 the person who runs the lab. It's their lab, and 0:18:06.400 --> 0:18:10.120 go against them. But there are people UM like Science 0:18:10.240 --> 0:18:13.640 and Nature to great journals are updating their guidelines right now. 0:18:13.680 --> 0:18:19.359 They're introducing checklists. UM Science hired statisticians to their panel 0:18:20.280 --> 0:18:24.119 reviewing editors, not just other you know, peer reviewed like 0:18:24.160 --> 0:18:28.960 they actually actually hard numbers people specifically, because that's a 0:18:29.200 --> 0:18:31.800 the process that's a huge part of studies. It's like 0:18:31.880 --> 0:18:36.359 these this mind breaking statistical analysis. It can be used 0:18:36.359 --> 0:18:39.400 for good or ill. And I mean, I don't think 0:18:39.400 --> 0:18:43.879 the average scientists necessarily is a whiz at that, although 0:18:44.119 --> 0:18:46.560 it has to be part of training, but not necessarily. 0:18:46.560 --> 0:18:49.960 And that's a different kind of beast altogether. UM stats, 0:18:50.119 --> 0:18:53.000 we talked about it earlier. I took a stats class 0:18:53.000 --> 0:18:56.560 in college. So much trouble that was awful at it. 0:18:56.560 --> 0:19:03.920 It really just it's a special kind of is even mad. Yeah, 0:19:03.920 --> 0:19:07.879 I didn't get it. I passed it, though I passed 0:19:07.880 --> 0:19:11.760 it because my professor took pity on me. Um that 0:19:12.000 --> 0:19:20.640 ulric Durna durnagal ulric arnaal Um. He is a he's 0:19:20.680 --> 0:19:24.919 a big time crusader for his jam making sure that 0:19:24.960 --> 0:19:27.560 science is good science. One of the things um he 0:19:27.600 --> 0:19:31.439 crusades against is the idea of remembering that virginity study 0:19:31.440 --> 0:19:33.720 where they just threw out anybody who had a violent 0:19:33.800 --> 0:19:38.159 encounter for their first sexual experience. UM. Apparently that's a 0:19:38.160 --> 0:19:41.440 big deal with animal studies as well. If you're studying 0:19:41.440 --> 0:19:43.679 the effects of a drug or something like there was 0:19:43.800 --> 0:19:46.200 there one in the article. Um. If you're studying the 0:19:46.200 --> 0:19:49.880 effects of a stroke drug and you've got a control 0:19:49.880 --> 0:19:53.000 group of mice that are taking the drug or that 0:19:53.040 --> 0:19:55.159 aren't taking the drug, and then a test group that 0:19:55.200 --> 0:19:59.160 are getting the drug. Um, and then like three mice 0:19:59.280 --> 0:20:02.920 from the group die even though they're on the stroke drug. 0:20:02.920 --> 0:20:05.960 They die of a massive stroke, and you just literally 0:20:06.000 --> 0:20:09.480 and figuratively throw them out of the study, um, and 0:20:09.520 --> 0:20:13.040 don't include them in the results. That changes the data. 0:20:13.160 --> 0:20:15.760 And he's been on a peer of you on a 0:20:15.800 --> 0:20:18.480 paper before He's like, no, this doesn't pass peer of you. 0:20:18.480 --> 0:20:21.399 You can't just throw out what happened to these three rodents? 0:20:21.400 --> 0:20:24.360 You started with ten, there's only seven reported in the end. 0:20:24.560 --> 0:20:27.200 What happened to those three? And how many of them 0:20:27.480 --> 0:20:30.520 just don't report the ten? They're like, oh, we only 0:20:30.520 --> 0:20:33.960 started with seven. Were going, you know, well, I was 0:20:34.000 --> 0:20:35.720 about to say I get the urge. I don't get 0:20:35.720 --> 0:20:37.920 it because it's not right. But I think what happens 0:20:37.960 --> 0:20:40.320 is you work so hard at something. Yeah, yeah, and 0:20:40.359 --> 0:20:42.640 you're like, how can I just walk away from two 0:20:42.720 --> 0:20:46.360 years of this because it didn't get a result? Okay? 0:20:46.640 --> 0:20:49.680 The point of real science, though, you have to walk 0:20:49.680 --> 0:20:52.800 away from it, Well, you have to publish that. And 0:20:52.840 --> 0:20:56.280 that's the other thing too, And I guarantee scientists will say, hey, man, 0:20:56.920 --> 0:20:59.919 try getting a negative paper published in a good journal. 0:21:00.040 --> 0:21:02.399 These days, you don't want that kind of stuff. But 0:21:02.480 --> 0:21:05.080 part of it also is I don't think it's enough 0:21:05.119 --> 0:21:07.720 to just have to be published in like a journal. 0:21:08.000 --> 0:21:09.960 You want to make the news cycle as well. That 0:21:10.080 --> 0:21:13.120 makes it even better, right, Um, So, I think there's 0:21:13.119 --> 0:21:15.880 a lot of factors involved, but ultimately, if you take 0:21:15.920 --> 0:21:18.000 all that stuff away, if you take the culture away, 0:21:18.000 --> 0:21:21.800 from it. You're if you get negative results, you're supposed 0:21:21.840 --> 0:21:24.239 to publish that so that some other scientists can come 0:21:24.240 --> 0:21:26.959 along and be like, oh, somebody else already did this 0:21:27.320 --> 0:21:29.560 using these methods that I was going to use. I'm 0:21:29.560 --> 0:21:32.680 not gonna waste two years of my career because somebody 0:21:32.680 --> 0:21:35.760 else already did. Thank you, buddy for saving me this 0:21:35.840 --> 0:21:38.360 time and trouble and effort to know that this does 0:21:38.359 --> 0:21:41.880 not work. You've proven this doesn't work. When you start 0:21:42.000 --> 0:21:44.160 to prove it does work, you actually proved it didn't work. 0:21:44.200 --> 0:21:48.560 That's part of science. Yeah, And I wish there wasn't 0:21:48.600 --> 0:21:53.320 a negative connotation to a negative result because to me, 0:21:53.359 --> 0:21:56.520 it's the value is the same as proving something does 0:21:56.560 --> 0:21:59.679 work as proving something doesn't work. Right. Again, it's just 0:21:59.800 --> 0:22:03.639 not it's sexy. Yeah, but I'm not sexy either, so 0:22:03.680 --> 0:22:06.119 maybe that's why I get it. Uh. Here's one that 0:22:06.160 --> 0:22:10.040 I didn't know was a thing. Predatory publishing. You never 0:22:10.080 --> 0:22:12.960 heard of this. So here's the scenario. You're a doctor 0:22:13.040 --> 0:22:17.440 or a scientist and um, you get an email from 0:22:17.480 --> 0:22:20.760 a journal that says, hey, you got anything interesting for us. 0:22:20.800 --> 0:22:22.520 I've heard about your work, and you say, well, I 0:22:22.520 --> 0:22:25.920 actually do I have this this study right here. They say, cool, 0:22:25.960 --> 0:22:29.120 we'll publish it. You go, great, my career is taking off. 0:22:29.320 --> 0:22:32.280 Then you get a bill he says, where's my three 0:22:32.359 --> 0:22:36.160 grand for publishing your article? And you're like, I don't 0:22:36.160 --> 0:22:39.760 owe you three grand, all right, give us two. And 0:22:39.760 --> 0:22:41.479 you're like, I can't even give you two. And if 0:22:41.520 --> 0:22:44.600 you fight them long enough, maybe they'll drop it and 0:22:44.960 --> 0:22:48.560 never work with you again, or maybe it'll just be like, well, 0:22:48.600 --> 0:22:53.280 we'll talk to your next quarter. Exactly. That's called predatory publishing, 0:22:53.320 --> 0:22:55.600 and it's a I'm not sure how new it is, 0:22:55.920 --> 0:22:58.800 maybe it's is it pretty new, but it's a thing 0:22:58.840 --> 0:23:07.040 now where uh, you can pay essentially to get something published. Yes, 0:23:07.119 --> 0:23:09.800 you can. Um it kind of it's kind of like 0:23:09.840 --> 0:23:14.160 who's who in behavioral science is kind of thing you know. Um. 0:23:14.160 --> 0:23:16.919 And apparently it's new because it's a result of open 0:23:17.000 --> 0:23:20.199 source academic journals, which a lot of people push for, 0:23:20.400 --> 0:23:24.200 including Aaron Schwartz very famously who like took a bunch 0:23:24.200 --> 0:23:28.000 of academic articles and published him online and was prosecuted 0:23:28.040 --> 0:23:31.720 heavily for it. Persecuted. You could even say, um, but 0:23:31.800 --> 0:23:34.560 the idea that science is behind this paywall, which is 0:23:34.600 --> 0:23:38.719 another great article from prisonomics by the way, Um, it 0:23:38.760 --> 0:23:40.719 really just ticks a lot of people off. So they 0:23:40.720 --> 0:23:45.000 started open source journals right and as a result, predatory 0:23:45.080 --> 0:23:48.399 publishers came about and said, Okay, yeah, let's make this free, 0:23:49.240 --> 0:23:50.919 but we need to make our money anyway, so we're 0:23:50.920 --> 0:23:54.159 going to charge the academic who wrote the study for 0:23:54.240 --> 0:23:57.719 publishing it. Well yeah, and and sometimes now it's just 0:23:57.800 --> 0:24:02.480 a flat out scam operation. There's this guy named Jeffrey 0:24:02.520 --> 0:24:07.240 Beale who is a research librarian. He is my new 0:24:07.280 --> 0:24:10.439 hero because he's truly like one of these dudes that 0:24:10.840 --> 0:24:14.200