WEBVTT - How does scientific peer review work? 0:00:07.840 --> 0:00:10.840 Science is supposed to be our best way of revealing 0:00:10.840 --> 0:00:14.360 the truth about the universe. But science is constantly being 0:00:14.440 --> 0:00:17.920 updated and corrected, and sometimes we learn after the fact 0:00:17.960 --> 0:00:20.439 that a study was flawed or even was shoddy to 0:00:20.480 --> 0:00:20.920 begin with. 0:00:21.480 --> 0:00:23.880 So how do scientists decide. 0:00:23.440 --> 0:00:26.560 Whether a new result is robust or not? And how 0:00:26.600 --> 0:00:29.000 does the general public know when the science is settled 0:00:29.160 --> 0:00:32.559 or about to be upended. No system is perfect, but 0:00:32.600 --> 0:00:34.800 it's important that the process is transparent. 0:00:35.240 --> 0:00:36.080 So today we're. 0:00:35.920 --> 0:00:37.920 Going to shine a light on the inner workings of 0:00:37.920 --> 0:00:41.280 one of the crucial parts of the scientific process, peer review. 0:00:41.960 --> 0:00:44.960 How does it work? Who are these peers? Can peer 0:00:45.000 --> 0:00:48.200 reviewed papers be wrong? And does that mean that all 0:00:48.280 --> 0:00:52.320 of science is questionable? Welcome to Daniel and Kelly's peer 0:00:52.360 --> 0:01:06.800 reviewed universe. Hi. 0:01:07.040 --> 0:01:10.639 I'm Daniel. I'm a particle physicist and I've published over 0:01:10.720 --> 0:01:14.160 a thousand papers, but haven't read most of them. 0:01:14.360 --> 0:01:14.520 Hi. 0:01:14.600 --> 0:01:18.600 I'm Kelly Wieder Smith and I'm confused. What does that mean? Daniel? 0:01:18.640 --> 0:01:21.839 How have you published over a thousand papers at the LHC. 0:01:22.040 --> 0:01:27.360 Does everybody who is reading Oh that's cheating, man, that's cheating. 0:01:27.600 --> 0:01:31.280 Well, the way the collaboration works is everybody who's contributed 0:01:31.319 --> 0:01:33.559 to the project is an author. And when you're an author, 0:01:34.040 --> 0:01:37.119 you get included on every paper that comes out during 0:01:37.160 --> 0:01:40.959 your authorship period. And ATLAS is a huge collaboration of 0:01:41.000 --> 0:01:43.039 lots of clever people. We put out about one hundred 0:01:43.040 --> 0:01:45.959 papers a year. Every single one has my name on it, 0:01:46.000 --> 0:01:48.400 and I haven't read most of them. I couldn't even 0:01:48.440 --> 0:01:50.280 explain the titles of some of them to you. 0:01:50.840 --> 0:01:54.480 So in my so fields have their own cultures. In 0:01:54.520 --> 0:02:00.720 my field, in my culture, that would ethically not be okay. 0:02:01.120 --> 0:02:03.680 You should be responsible for what is in every paper 0:02:03.960 --> 0:02:06.120 that your name is on, in my opinion. 0:02:06.080 --> 0:02:09.400 And I was in another collaboration previously where things work differently, 0:02:09.440 --> 0:02:11.720 where you weren't an author on a paper unless you 0:02:11.800 --> 0:02:14.200 asked to be, and nobody questioned it. If you had 0:02:14.200 --> 0:02:16.040 the right to be an author, if you were on 0:02:16.080 --> 0:02:17.560 the list, you could be. But it was an opt 0:02:17.600 --> 0:02:20.160 in and I thought that was much better. The lengths 0:02:20.160 --> 0:02:22.960 of all the list was shorter, and authorship meant something. 0:02:23.760 --> 0:02:26.360 But here you can ask to be removed, but you 0:02:26.360 --> 0:02:28.080 have to go through this process and it would be 0:02:28.080 --> 0:02:30.640 like I'd be doing it twice a week every week. 0:02:30.720 --> 0:02:35.120 So wow, So what does your CV include all of 0:02:35.160 --> 0:02:37.520 those papers or just the ones that you feel you've 0:02:37.560 --> 0:02:39.160 actually contributed to, So. 0:02:39.080 --> 0:02:41.880 That depends the CV I make and share. People only 0:02:41.919 --> 0:02:44.839 includes papers that I actually did anything on, or had 0:02:44.880 --> 0:02:48.840 ideas for, or really contributed to. But sometimes an institution 0:02:49.120 --> 0:02:52.200 wants a list of your official papers, and then I 0:02:52.280 --> 0:02:54.600 have to like include all I don't even know the 0:02:54.680 --> 0:02:57.079 number one thousand plus papers at this point. 0:02:57.320 --> 0:02:58.080 Oh poor dude. 0:02:58.440 --> 0:03:01.240 Now, so. 0:03:02.800 --> 0:03:04.320 How do you get them? Ball? Do you just copy 0:03:04.400 --> 0:03:05.400 them from Google Scholar? 0:03:05.880 --> 0:03:08.880 Yeah, Particle physics is pretty good at automation and stuff 0:03:08.919 --> 0:03:12.760 like this, so we have our own databases of papers 0:03:12.800 --> 0:03:14.760 and it's pretty easy to download that kind of stuff. 0:03:15.120 --> 0:03:17.480 But yeah, it's pretty tough to have so many papers. 0:03:17.840 --> 0:03:20.600 Oh, so many pages you have to print out. The 0:03:20.680 --> 0:03:23.160 file for the PDA for your resume is so big. 0:03:23.720 --> 0:03:25.600 How about you, Kelly, how many papers do you haven't? 0:03:25.680 --> 0:03:27.639 Have you finished that paper from your thesis yet? 0:03:27.919 --> 0:03:32.000 Oh I'm feeling angry now, Dan, I have not. 0:03:32.800 --> 0:03:36.200 I've been busy with other things, including my new Ducks 0:03:36.240 --> 0:03:39.800 and geese. But when goats and goats, which I won't 0:03:39.840 --> 0:03:40.200 mention of. 0:03:40.200 --> 0:03:42.400 My co authors, do your ducks and geese and goats? 0:03:42.520 --> 0:03:46.440 Go on your CV. That's my question. No, uh, students supervised? 0:03:46.480 --> 0:03:46.720 Come on? 0:03:46.960 --> 0:03:48.600 Oh yeah, no, I've got that list. 0:03:48.720 --> 0:03:53.800 But in the Great, the Great Goat, I think I've 0:03:53.840 --> 0:03:56.160 got I haven't I haven't counted. I think I've got 0:03:56.240 --> 0:03:59.800 something like thirty that. My pace has slowed since I 0:04:00.560 --> 0:04:03.160 writing POPSI, but you know I still get a couple 0:04:03.200 --> 0:04:03.880 in there Every year. 0:04:04.240 --> 0:04:07.400 My CV has something whimsical and unseerious on it. Yours 0:04:07.560 --> 0:04:08.840 is totally professional. 0:04:09.440 --> 0:04:16.839 Ah, yes, what whimsical thing do you have on yours? 0:04:17.120 --> 0:04:17.240 Oh? 0:04:17.360 --> 0:04:19.120 I'm just gonna leave that as an easter eggs. Oh. 0:04:19.279 --> 0:04:21.000 I put it in there to see if anybody actually 0:04:21.040 --> 0:04:23.560 reads my CV, because if they don't respond to that, 0:04:23.680 --> 0:04:26.120 I'm like, hmm, you didn't really read it. Okay, yeah 0:04:26.400 --> 0:04:26.840 check it out. 0:04:26.880 --> 0:04:29.120 All right, Well, I'll be looking up your CV. Do 0:04:29.120 --> 0:04:30.440 you haven't updated? Here's the question. 0:04:30.960 --> 0:04:34.320 Most scholars don't update their CV, but like once every 0:04:34.360 --> 0:04:35.000 half decade. 0:04:35.960 --> 0:04:37.960 Is your online CV updated? 0:04:38.800 --> 0:04:40.520 Yeah? I think I updated a couple months ago. 0:04:40.720 --> 0:04:42.680 Oh all right, bravo, good out. 0:04:43.560 --> 0:04:45.799 All right, Well today we're not talking about the thousands 0:04:45.839 --> 0:04:48.559 of publications that Daniel has, but we're all very proud 0:04:48.600 --> 0:04:50.359 of Daniel for all of his hard. 0:04:50.200 --> 0:04:54.279 Work, or the tens of geese and ducks that Kelly 0:04:54.400 --> 0:04:56.600 is currently raising, though we're very proud of her and 0:04:56.720 --> 0:04:58.680 her literally extended family. 0:04:58.920 --> 0:05:01.720 Well, I don't have that many, and I wouldn't claim 0:05:01.839 --> 0:05:03.840 other people's ducks that I didn't help out with as 0:05:03.920 --> 0:05:04.559 my own ducks. 0:05:04.680 --> 0:05:07.040 Oh wow, now no. 0:05:09.240 --> 0:05:13.440 But anyway, today we are answering a question from a 0:05:13.680 --> 0:05:16.599 superfan and we're going to go into a bit more detail. 0:05:16.680 --> 0:05:19.400 This question inspired us to make a whole episode about 0:05:19.440 --> 0:05:21.520 the process of peer reviews. So let's go ahead and 0:05:21.600 --> 0:05:25.159 hear what Stephen from British Columbia has to say. 0:05:25.960 --> 0:05:29.800 Hi, Daniel and Kelly, this is show super fan Stephen 0:05:29.960 --> 0:05:33.960 from British Columbia, Canada. My question today has to do 0:05:34.160 --> 0:05:37.640 with how scientific studies are peer reviewed? 0:05:38.360 --> 0:05:38.880 Is it me? 0:05:39.720 --> 0:05:43.920 Or is science denial spreading these days? And it came 0:05:44.040 --> 0:05:48.640 to my attention recently that not all peer reviewed studies 0:05:48.880 --> 0:05:53.080 are replicated, And then it occurred to me, doesn't this 0:05:53.320 --> 0:05:57.800 automatically cast doubt on a discovery if a study is 0:05:57.920 --> 0:06:01.800 not replicated but is still considered peer reviewed. 0:06:02.880 --> 0:06:04.080 I would love to. 0:06:04.200 --> 0:06:09.040 Hear an explanation about how the peer reviewing process works 0:06:09.520 --> 0:06:13.440 and how scientists determine when a fact is a fact, 0:06:14.040 --> 0:06:18.200 a finding is a finding, and how the community comes. 0:06:18.000 --> 0:06:20.599 To a consensus on new discoveries. 0:06:21.360 --> 0:06:23.680 I also think your audience. 0:06:23.320 --> 0:06:27.719 Could use some tools for how to determine what studies 0:06:27.839 --> 0:06:32.760 are legitimate and how to spot a bad study. How 0:06:33.000 --> 0:06:38.679 should one respond to someone who claims a scientific study 0:06:39.200 --> 0:06:42.960 is bogus. Thank you Daniel and Kelly for this great 0:06:43.040 --> 0:06:47.000 podcast and for explaining science topics to the masses. 0:06:47.320 --> 0:06:49.960 Thank you Stephen for asking this question. I think it's 0:06:50.040 --> 0:06:52.680 really the right moment to dig into this. You know, 0:06:52.760 --> 0:06:56.160 when we see institutions being attacked, when we see science 0:06:56.279 --> 0:06:59.440 being denied, when we see the whole process of science 0:06:59.520 --> 0:07:02.240 being q and I think it's important to shine some 0:07:02.400 --> 0:07:04.479 light on how does this all work, what does it mean, 0:07:04.720 --> 0:07:07.200 especially for those folks who are not scientists, who don't know, 0:07:07.640 --> 0:07:09.560 what does it mean for a paper to be peer reviewed? 0:07:09.920 --> 0:07:12.560 What is bad science? Thank you Stephen for asking this 0:07:12.680 --> 0:07:15.080 question and giving us the chance to dig into all 0:07:15.160 --> 0:07:15.280 of this. 0:07:15.640 --> 0:07:18.000 That's right, and so the question in particular is about 0:07:18.200 --> 0:07:22.320 peer review for scientific manuscripts. But peer review actually happens 0:07:22.360 --> 0:07:25.200 at multiple stages in the process of doing science, and 0:07:25.320 --> 0:07:27.600 so we thought we would take a step back and 0:07:27.720 --> 0:07:29.960 talk about peer review from the very beginning, which is 0:07:30.000 --> 0:07:34.640 when your idea becomes a grant. And this is one 0:07:34.680 --> 0:07:37.960 of the less fun parts about science is writing this 0:07:38.120 --> 0:07:40.960 grant because peer review for grants is harsh. 0:07:41.480 --> 0:07:43.560 This is like the science version of when a build 0:07:43.600 --> 0:07:45.400 becomes a law. We should have written a song for this. 0:07:45.640 --> 0:07:47.920 No, well we still can. You know, we could write 0:07:47.920 --> 0:07:51.320 it and then we can add it to the top. No, 0:07:51.440 --> 0:07:52.520 we won't do that to everyone. 0:07:52.840 --> 0:07:55.640 So Kelly, tell me what is your process of grant writing? 0:07:55.680 --> 0:07:57.640 First of all, who do you submit your grants to? 0:07:58.240 --> 0:07:59.760 And what's it like for you to prepare that? 0:08:00.440 --> 0:08:03.080 So I submit my grants to the National Science Foundation. 0:08:03.360 --> 0:08:06.640 And actually it's possible that in the last six months 0:08:06.720 --> 0:08:09.120 the process has changed a bit because I know the 0:08:09.200 --> 0:08:11.600 new administration is shaking things up. 0:08:11.720 --> 0:08:14.560 The agency formerly known as the National Science Foundation. 0:08:15.160 --> 0:08:16.400 No, what is it called now? 0:08:16.520 --> 0:08:17.040 I'm joking? 0:08:17.480 --> 0:08:20.240 Oh god, anything is believable, Daniel. 0:08:20.280 --> 0:08:24.520 At this point, all right, the institution still known as 0:08:24.560 --> 0:08:28.040 the NSF or the National Science Foundation. So for this 0:08:28.280 --> 0:08:31.320 you write a grant, and there's different programs that accept 0:08:31.360 --> 0:08:33.640 different kinds of grants, but often the grant is like 0:08:33.760 --> 0:08:38.319 fifteen pages where you go into extreme detail and what 0:08:38.559 --> 0:08:41.319 we know about the question you want to ask based 0:08:41.360 --> 0:08:43.959 on the literature that's already out there, extreme detail and 0:08:44.040 --> 0:08:46.840 the experimental design, and then a lot of detail about 0:08:47.040 --> 0:08:49.920 what you think the implications will be of your results. 0:08:50.160 --> 0:08:53.360 So you talk about why this question is interesting, to 0:08:53.600 --> 0:08:56.440 argue that your work will help answer it, and then 0:08:56.480 --> 0:08:58.360 you lay out in detail how are you going to 0:08:58.360 --> 0:09:00.800 spend every dollar, who's going to do what, and when 0:09:00.840 --> 0:09:02.559 you get like a whole gan chart for like when 0:09:02.600 --> 0:09:04.760 everything is going to happen yep, And then you have 0:09:04.800 --> 0:09:07.520 to lay out also like the products for the grant, right, yeah. 0:09:07.440 --> 0:09:09.400 Yeah, the budgets and so the you're supposed to There's 0:09:09.440 --> 0:09:12.080 a section called intellectual merit where you talk about how 0:09:12.120 --> 0:09:14.120 many papers you think you're going to end up producing, 0:09:14.200 --> 0:09:16.200 what big questions you're going to answer, And then there's 0:09:16.240 --> 0:09:18.720 also a section called broader impacts, where you talk about 0:09:18.760 --> 0:09:22.040 how your work is going to benefit society writ large. 0:09:22.559 --> 0:09:24.559 And you know, for my grants in the past, it's 0:09:24.600 --> 0:09:26.800 been things like, you know, my information is going to 0:09:26.840 --> 0:09:30.880 be helpful to fisheries managers who are managing fish populations, 0:09:30.920 --> 0:09:33.640 which is like a literally a multi billion dollar industry 0:09:34.320 --> 0:09:36.120 for like people who want to go out fishing and stuff. 0:09:36.120 --> 0:09:38.599 So most of my research has been on fish and 0:09:38.679 --> 0:09:42.320 their parasites. Also intellectual merit will include things like how 0:09:42.360 --> 0:09:44.839 many students you're going to train and what skills you're 0:09:44.880 --> 0:09:46.640 going to give them that they can then use when 0:09:46.679 --> 0:09:49.160 they go out in the workforce. And depending on the 0:09:49.200 --> 0:09:52.440 field you're in, sometimes your broader impacts will be things like, Hey, 0:09:52.520 --> 0:09:55.760 this could be what we learn about the brains of 0:09:55.840 --> 0:09:57.800 this fish could one day help us produce a new 0:09:57.880 --> 0:10:01.720 medication for anxiety or something like. So you talk about 0:10:01.720 --> 0:10:04.320 how your stuff would impact the broader society. 0:10:04.559 --> 0:10:07.040 And so for those folks who might think, hey, scientists 0:10:07.080 --> 0:10:10.240 are just cashing in on the government gravy train, tell 0:10:10.360 --> 0:10:12.040 us like, is this the kind of thing you throw 0:10:12.120 --> 0:10:15.439 together in an afternoon and then can confidently think it 0:10:15.559 --> 0:10:16.240 will be funded? 0:10:16.440 --> 0:10:17.360 Oh my god? Okay. 0:10:17.520 --> 0:10:19.959 So one of the reasons, to be honest, one of 0:10:19.960 --> 0:10:23.360 the reasons I sort of transitioned away from academia in 0:10:23.480 --> 0:10:27.000 the most purest sense, so I'm like academic adjacent now, 0:10:27.640 --> 0:10:31.800 but was because I spent too many Christmas Eves working 0:10:31.880 --> 0:10:35.040 on my grants because they were due like sometime in January, 0:10:35.720 --> 0:10:40.080 and they just they were like so many I would 0:10:40.080 --> 0:10:41.880 say months of work. In a lot of cases, if 0:10:41.880 --> 0:10:43.800 it's a new idea, if you're just cleaning up an 0:10:43.840 --> 0:10:47.199 old grant, then it can be once of work. And 0:10:47.320 --> 0:10:50.440 then so many great grants are submitted, and when they 0:10:50.559 --> 0:10:54.000 go to review, you have three peers of yours who 0:10:54.080 --> 0:10:57.200 are in closely related fields review it, and then there's 0:10:57.320 --> 0:11:00.480 a panel discussion where they talk about their views of 0:11:00.559 --> 0:11:02.840 your grant in front of a bunch of other experts 0:11:02.840 --> 0:11:05.480 who can also weigh in. And then all those reviews 0:11:05.559 --> 0:11:09.000 go to program directors who pick out of like the 0:11:09.120 --> 0:11:12.320 twenty best grants, you know, maybe they can only fund 0:11:12.440 --> 0:11:15.640 fifteen or something. And then they pick some on the 0:11:15.760 --> 0:11:17.679 East coast, some on the West coast, some in the 0:11:17.800 --> 0:11:20.839 center of the country, some from major research institutions, some 0:11:21.000 --> 0:11:24.000 from institutions that focus a lot on undergraduate research, and 0:11:24.200 --> 0:11:27.960 they have research programs as well, And so so many 0:11:28.320 --> 0:11:32.000 really great grants don't make it into the pile that 0:11:32.120 --> 0:11:33.040 get funded. 0:11:33.120 --> 0:11:35.880 Meaning that people have read them, have said this is excellent, 0:11:36.000 --> 0:11:39.400 the science is good, it's interesting, it's important. We would 0:11:39.440 --> 0:11:42.000 benefit as a society if we did this. It's all 0:11:42.080 --> 0:11:42.840 well thought. 0:11:42.679 --> 0:11:46.719 Through, but no, that is exactly right, yes, and it 0:11:46.960 --> 0:11:49.679 is soul crushing. I know so many people who are like, 0:11:49.800 --> 0:11:51.640 I gave up Christmas Eve on this grant, and this 0:11:51.760 --> 0:11:53.959 is the third time I've submitted it, and it just 0:11:54.080 --> 0:11:56.720 never makes it quite high enough, even though everybody thinks 0:11:56.760 --> 0:11:59.360 it's amazing. And then quickly just to mention for follow up. 0:11:59.400 --> 0:12:01.920 If you do get grant, every year you have to 0:12:01.960 --> 0:12:04.640 write a progress report saying what you've done, have you 0:12:04.679 --> 0:12:07.040 stuck to your timeline, how have you spent the money 0:12:07.080 --> 0:12:09.920 that you've been given, are you going to hit your timelines? 0:12:10.040 --> 0:12:12.760 And what have you done that impacts society? And so 0:12:12.920 --> 0:12:15.320 every year you have to report on that and then 0:12:15.360 --> 0:12:17.560 at the end you have to write a bigger report. 0:12:17.800 --> 0:12:19.800 So you need to be justifying what you're doing with 0:12:19.880 --> 0:12:21.120 the money every step of the way. 0:12:21.360 --> 0:12:23.559 And before we turn it around and ask me questions, 0:12:24.000 --> 0:12:26.200 who are the peers here, Who are folks who are 0:12:26.320 --> 0:12:28.640 reading your grants and commenting on it. 0:12:29.000 --> 0:12:31.480 There are folks in the same discipline at other universities. 0:12:31.600 --> 0:12:34.800 So you also have to fill out a detailed excel 0:12:34.880 --> 0:12:37.240 sheet where you talk about who in the field who 0:12:37.280 --> 0:12:40.520 does similar work, has been your student or your postdoc 0:12:40.840 --> 0:12:42.839 or your mentor or a co author. 0:12:42.679 --> 0:12:43.480 You might have a conflict. 0:12:43.640 --> 0:12:47.160 Yeah, and anyone who might like you for you and 0:12:47.240 --> 0:12:50.280 who might be too nice to you when they review 0:12:50.320 --> 0:12:52.360 the grant. So it has to be only people that 0:12:52.440 --> 0:12:54.520 you haven't worked with or haven't co author of paper 0:12:54.559 --> 0:12:57.559 with for something like five to ten years. I guess 0:12:57.600 --> 0:13:00.240 they assume that the liking of someone wears off about 0:13:00.240 --> 0:13:03.920 half a decade. But it can be tough because you know, 0:13:04.000 --> 0:13:05.880 these fields are kind of small, but you know, usually 0:13:05.920 --> 0:13:09.600 it's other people at research institutions who do similar work, 0:13:09.640 --> 0:13:12.000 are familiar with the literature and can tell if what 0:13:12.080 --> 0:13:14.480 you're proposing makes sense or not and is good or not. 0:13:14.760 --> 0:13:16.520 And so the thing to take away from this is 0:13:16.960 --> 0:13:20.280 getting a grand funded is hard, right. Not only is 0:13:20.320 --> 0:13:22.040 it a huge amount of work, but you have to 0:13:22.120 --> 0:13:26.920 survive an excruciating process where really only the best are selected. 0:13:27.400 --> 0:13:30.520 It's sort of like watching the Olympics and you're wondering, like, oh, 0:13:30.640 --> 0:13:32.960 here's somebody from this country and somebody from that country, 0:13:33.000 --> 0:13:35.959 and you know that each person has already won some 0:13:36.080 --> 0:13:40.199 sort of like really competitive national competition just to even 0:13:40.360 --> 0:13:44.079 be there at the Olympics representing their country. And so 0:13:44.240 --> 0:13:46.679 like every grant that's funded, even if it seems silly 0:13:46.840 --> 0:13:49.679 and it's about like the sex lives of ducks, you 0:13:49.880 --> 0:13:54.480 know that it's been like excruciatingly written, reviewed in detail, 0:13:54.679 --> 0:13:57.480 questioned by experts without conflicts of interest, and found to 0:13:57.520 --> 0:14:00.840 be excellent, beaten out lots of other grand These are 0:14:00.920 --> 0:14:04.839 not slush funds shoveled to scientists, so you know, just 0:14:04.920 --> 0:14:07.559 like do whatever they want with. These are hard one 0:14:07.760 --> 0:14:10.760 funds to do science that the agency thinks are a 0:14:10.840 --> 0:14:11.360 good idea. 0:14:11.720 --> 0:14:15.480 Yeah, it is absolutely excruciating to do these things. It 0:14:15.559 --> 0:14:17.440 feels great when you get it funded. And I said, 0:14:17.440 --> 0:14:20.120 I hate writing grants. Sometimes I enjoy the process of 0:14:20.280 --> 0:14:23.240 like figuring out just the right experimental design for a question. 0:14:23.400 --> 0:14:26.080 That's like fun for me. But in general, when the 0:14:26.120 --> 0:14:29.840 grant doesn't get funded, it sucks. But anyway, all right, 0:14:29.960 --> 0:14:31.560 so what's your experience. 0:14:31.400 --> 0:14:34.000 Yeah, my experience is very similar. You know, we come 0:14:34.080 --> 0:14:35.960 up with an idea, we spend a lot of time 0:14:36.080 --> 0:14:39.720 polishing it, often doing a lot of the work in 0:14:39.880 --> 0:14:43.600 advance that we need to demonstrate that the work described 0:14:43.680 --> 0:14:47.800 it's reasonable. Right. If it's too far forward thinking, then 0:14:48.080 --> 0:14:50.120 you won't get the money because they're like, well, that 0:14:50.280 --> 0:14:52.400 might work, but can you really prove it. It's too 0:14:52.480 --> 0:14:54.960 much of a risk. If you've already done too much 0:14:55.000 --> 0:14:57.200 of the work, then they're like, oh, this is already done, 0:14:57.200 --> 0:14:59.160 why would we fund it? So it's really sort of 0:14:59.240 --> 0:15:01.320 on the edge there of like you've done the initial 0:15:01.440 --> 0:15:04.480 work for free right or on some other grant or whatever. 0:15:04.880 --> 0:15:07.920 So you prove that the idea is valid and can fly, 0:15:08.200 --> 0:15:10.120 but not so much that they're like, why would we 0:15:10.200 --> 0:15:12.080 fund this? You've already done it, which is often a 0:15:12.160 --> 0:15:16.200 delicate balance. And my feeling with grants is submit it 0:15:16.320 --> 0:15:19.720 and forget it, because such a tiny fraction ever come 0:15:19.800 --> 0:15:21.800 back with money that you just got to like let 0:15:21.880 --> 0:15:24.680 it go, like, hey, I've submitted it, and I never 0:15:24.800 --> 0:15:28.680 expect to hear back, you know, anything positive, And so 0:15:28.800 --> 0:15:31.120 I just sort of like give up emotionally on everyone 0:15:31.200 --> 0:15:34.200 because otherwise it's too hard, you know, it's too hard 0:15:34.240 --> 0:15:36.760 to deal with, yea. So my process is very similar 0:15:36.920 --> 0:15:40.800 to yours, with a couple of exceptions. Sometimes I submit 0:15:40.920 --> 0:15:44.240 to private foundations, like I recently found a private foundation 0:15:44.440 --> 0:15:48.360 that likes to fund projects that are too blue sky 0:15:48.880 --> 0:15:52.280 that were rejected by the NSF for being like way 0:15:52.360 --> 0:15:56.560 too out there, and that submission process was like write 0:15:56.560 --> 0:16:00.360 a paragraph, send us your rejected NSF grant with the views, 0:16:00.640 --> 0:16:01.080 and that's it. 0:16:01.360 --> 0:16:01.600 Whoa. 0:16:02.040 --> 0:16:03.880 And that one actually just came back and they just 0:16:03.960 --> 0:16:07.800 gave me some money to build my cosmic ray smartphone 0:16:07.840 --> 0:16:10.880 telescope down here at Irvine. So yeah, that was actually 0:16:10.880 --> 0:16:13.040 a really positive experience graduations. 0:16:13.160 --> 0:16:14.040 Yeah, that's awesome. 0:16:14.680 --> 0:16:18.120 Yeah exactly, because you know, private foundations can do whatever 0:16:18.160 --> 0:16:20.480 they like with your money. What we talked about previously 0:16:20.600 --> 0:16:24.000 was mostly the process of applying to government institutions, the 0:16:24.160 --> 0:16:27.800 NIH the NSF. Most of my fund incomes from Department 0:16:27.840 --> 0:16:30.600 of Energy, Office of Science, which funds particle physics and 0:16:30.680 --> 0:16:34.160 lots of other stuff in the United States. But private 0:16:34.200 --> 0:16:36.480 fundations can do anything. Like the Bill Gates Foundation, you 0:16:36.560 --> 0:16:38.600 can't even apply to. They have to like reach out 0:16:38.640 --> 0:16:38.840 to you. 0:16:39.320 --> 0:16:39.520 Wow. 0:16:39.640 --> 0:16:42.120 And you know Mackenzie Scott just like gives money to 0:16:42.160 --> 0:16:44.040 people that don't even apply. They just get a phone 0:16:44.080 --> 0:16:46.680 call saying like, by the way, here comes a million 0:16:46.720 --> 0:16:47.640 dollars or more. 0:16:47.840 --> 0:16:48.040 Whoa. 0:16:48.400 --> 0:16:50.600 So yeah, private foundations are weird. 0:16:51.440 --> 0:16:55.200 Yeah, so I'm sure that they make solid choices about 0:16:55.200 --> 0:16:56.800 who they're going to give their money. To But I 0:16:56.840 --> 0:16:58.760 guess one way to sort of check how much you 0:16:58.960 --> 0:17:01.080 can trust an article as you can look in the 0:17:01.160 --> 0:17:03.440 acknowledgment section at the end and say, oh, this was 0:17:03.480 --> 0:17:05.600 funded by the National Science Foundation, And then you know 0:17:06.200 --> 0:17:08.680 this went through rigorous peer review. And just because something 0:17:08.800 --> 0:17:11.320 doesn't go through rigorous peer review at the grant stage 0:17:11.640 --> 0:17:14.280 doesn't mean it's bad science. But at least you know 0:17:14.320 --> 0:17:16.720 if it went through a government agency, it has been 0:17:16.760 --> 0:17:19.199 sort of gone over with a fine tooth coust And the. 0:17:19.200 --> 0:17:22.879 Thing that's frustrating to me is this process is so inefficient. 0:17:23.200 --> 0:17:26.760 Scientists spend so much of their time preparing grant proposals 0:17:26.920 --> 0:17:30.120 and having them rejected. And you know, each of these 0:17:30.280 --> 0:17:32.320 is a great idea, each of them. If we did 0:17:32.359 --> 0:17:35.600 it would benefit society in terms of sheer knowledge or 0:17:36.240 --> 0:17:39.800 you know, new technology or something. We have all these 0:17:39.840 --> 0:17:43.399 smart people constantly pitching great ideas to the government, the 0:17:43.480 --> 0:17:45.960 government saying, yeah, that's awesome, we'd love to do it, 0:17:46.119 --> 0:17:50.280 but we can't. Like, why don't we just double triple 0:17:51.160 --> 0:17:54.800 funding for science. It would be like better for us 0:17:54.840 --> 0:17:57.800 every dollar we spend, we know comes back two threefold 0:17:57.880 --> 0:18:00.960 in terms of like economic output. It just seems crazy 0:18:01.000 --> 0:18:04.160 to me to reject all of these excellent ideas. 0:18:04.440 --> 0:18:07.240 I mean, I gotta say, not every idea I've reviewed 0:18:07.320 --> 0:18:08.840 on a panel has been excellent. 0:18:09.000 --> 0:18:10.600 No, but there are plenty above threshold. 0:18:10.680 --> 0:18:11.960 Yes, absolutely, absolutely. 0:18:12.040 --> 0:18:14.960 There's always a grant every round where my heart breaks 0:18:15.000 --> 0:18:17.080 a little that it didn't get funded because I thought, oh, 0:18:17.200 --> 0:18:19.760 that is so cool, but there wasn't enough money to 0:18:19.960 --> 0:18:21.960 cover it. And so yes, there's a lot of good 0:18:22.000 --> 0:18:23.040 work that's not getting done. 0:18:23.160 --> 0:18:26.000 And I agree. I'm a grant reviewer often and I 0:18:26.160 --> 0:18:28.240 see grants I'm like, no, this is not well thought out, 0:18:28.400 --> 0:18:30.840 or there's a flaw here, or this isn't cutting edge. 0:18:30.920 --> 0:18:32.840 You know, somebody did this last year, and it's important 0:18:33.080 --> 0:18:35.240 that this stuff gets reviewed and gets reviewed in a 0:18:35.320 --> 0:18:38.320 fair way. But there's so many that are above threshold, 0:18:38.520 --> 0:18:40.720 and only a fraction of those get funded, and it 0:18:40.800 --> 0:18:42.440 just seems to me to be a waste. But whatever. 0:18:42.880 --> 0:18:46.159 I'm not a political person. I understand these things. But 0:18:46.240 --> 0:18:48.359 I want folks out there to realize that every grant 0:18:48.400 --> 0:18:51.880 you've seen that's been funded has been reviewed and excruciating 0:18:51.960 --> 0:18:54.960 detailed before dollar one was even sent to the institution. 0:18:55.240 --> 0:18:55.639 That's right. 0:18:55.760 --> 0:18:57.480 I don't think we need to get into lots of 0:18:57.560 --> 0:18:59.359 detail about this, but every once in a while, my 0:18:59.440 --> 0:19:01.480 husband and I will have chats about what would be 0:19:01.560 --> 0:19:04.800 a better system for funding grants, Like maybe every four 0:19:04.920 --> 0:19:07.359 years you get a chance to submit and that way 0:19:07.400 --> 0:19:09.720 you don't have to write grants the other three years 0:19:09.840 --> 0:19:12.000