Speaker 1: Science is supposed to be our best way of revealing
Speaker 1: the truth about the universe. But science is constantly being
Speaker 1: updated and corrected, and sometimes we learn after the fact
Speaker 1: that a study was flawed or even was shoddy to
Speaker 1: begin with.
Speaker 2: So how do scientists decide.
Speaker 1: Whether a new result is robust or not? And how
Speaker 1: does the general public know when the science is settled
Speaker 1: or about to be upended. No system is perfect, but
Speaker 1: it's important that the process is transparent.
Speaker 2: So today we're.
Speaker 1: Going to shine a light on the inner workings of
Speaker 1: one of the crucial parts of the scientific process, peer review.
Speaker 1: How does it work? Who are these peers? Can peer
Speaker 1: reviewed papers be wrong? And does that mean that all
Speaker 1: of science is questionable? Welcome to Daniel and Kelly's peer
Speaker 1: reviewed universe. Hi.
Speaker 3: I'm Daniel. I'm a particle physicist and I've published over
Speaker 3: a thousand papers, but haven't read most of them.
Speaker 4: Hi.
Speaker 2: I'm Kelly Wieder Smith and I'm confused. What does that mean? Daniel?
Speaker 2: How have you published over a thousand papers at the LHC.
Speaker 1: Does everybody who is reading Oh that's cheating, man, that's cheating.
Speaker 3: Well, the way the collaboration works is everybody who's contributed
Speaker 3: to the project is an author. And when you're an author,
Speaker 3: you get included on every paper that comes out during
Speaker 3: your authorship period. And ATLAS is a huge collaboration of
Speaker 3: lots of clever people. We put out about one hundred
Speaker 3: papers a year. Every single one has my name on it,
Speaker 3: and I haven't read most of them. I couldn't even
Speaker 3: explain the titles of some of them to you.
Speaker 1: So in my so fields have their own cultures. In
Speaker 1: my field, in my culture, that would ethically not be okay.
Speaker 1: You should be responsible for what is in every paper
Speaker 1: that your name is on, in my opinion.
Speaker 3: And I was in another collaboration previously where things work differently,
Speaker 3: where you weren't an author on a paper unless you
Speaker 3: asked to be, and nobody questioned it. If you had
Speaker 3: the right to be an author, if you were on
Speaker 3: the list, you could be. But it was an opt
Speaker 3: in and I thought that was much better. The lengths
Speaker 3: of all the list was shorter, and authorship meant something.
Speaker 3: But here you can ask to be removed, but you
Speaker 3: have to go through this process and it would be
Speaker 3: like I'd be doing it twice a week every week.
Speaker 1: So wow, So what does your CV include all of
Speaker 1: those papers or just the ones that you feel you've
Speaker 1: actually contributed to, So.
Speaker 3: That depends the CV I make and share. People only
Speaker 3: includes papers that I actually did anything on, or had
Speaker 3: ideas for, or really contributed to. But sometimes an institution
Speaker 3: wants a list of your official papers, and then I
Speaker 3: have to like include all I don't even know the
Speaker 3: number one thousand plus papers at this point.
Speaker 2: Oh poor dude.
Speaker 3: Now, so.
Speaker 2: How do you get them? Ball? Do you just copy
Speaker 2: them from Google Scholar?
Speaker 3: Yeah, Particle physics is pretty good at automation and stuff
Speaker 3: like this, so we have our own databases of papers
Speaker 3: and it's pretty easy to download that kind of stuff.
Speaker 3: But yeah, it's pretty tough to have so many papers.
Speaker 2: Oh, so many pages you have to print out. The
Speaker 2: file for the PDA for your resume is so big.
Speaker 3: How about you, Kelly, how many papers do you haven't?
Speaker 3: Have you finished that paper from your thesis yet?
Speaker 2: Oh I'm feeling angry now, Dan, I have not.
Speaker 1: I've been busy with other things, including my new Ducks
Speaker 1: and geese. But when goats and goats, which I won't
Speaker 1: mention of.
Speaker 3: My co authors, do your ducks and geese and goats?
Speaker 3: Go on your CV. That's my question. No, uh, students supervised?
Speaker 3: Come on?
Speaker 2: Oh yeah, no, I've got that list.
Speaker 1: But in the Great, the Great Goat, I think I've
Speaker 1: got I haven't I haven't counted. I think I've got
Speaker 1: something like thirty that. My pace has slowed since I
Speaker 1: writing POPSI, but you know I still get a couple
Speaker 1: in there Every year.
Speaker 3: My CV has something whimsical and unseerious on it. Yours
Speaker 3: is totally professional.
Speaker 2: Ah, yes, what whimsical thing do you have on yours?
Speaker 4: Oh?
Speaker 3: I'm just gonna leave that as an easter eggs. Oh.
Speaker 3: I put it in there to see if anybody actually
Speaker 3: reads my CV, because if they don't respond to that,
Speaker 3: I'm like, hmm, you didn't really read it. Okay, yeah
Speaker 3: check it out.
Speaker 2: All right, Well, I'll be looking up your CV. Do
Speaker 2: you haven't updated? Here's the question.
Speaker 1: Most scholars don't update their CV, but like once every
Speaker 1: half decade.
Speaker 2: Is your online CV updated?
Speaker 3: Yeah? I think I updated a couple months ago.
Speaker 2: Oh all right, bravo, good out.
Speaker 1: All right, Well today we're not talking about the thousands
Speaker 1: of publications that Daniel has, but we're all very proud
Speaker 1: of Daniel for all of his hard.
Speaker 3: Work, or the tens of geese and ducks that Kelly
Speaker 3: is currently raising, though we're very proud of her and
Speaker 3: her literally extended family.
Speaker 1: Well, I don't have that many, and I wouldn't claim
Speaker 1: other people's ducks that I didn't help out with as
Speaker 1: my own ducks.
Speaker 3: Oh wow, now no.
Speaker 1: But anyway, today we are answering a question from a
Speaker 1: superfan and we're going to go into a bit more detail.
Speaker 1: This question inspired us to make a whole episode about
Speaker 1: the process of peer reviews. So let's go ahead and
Speaker 1: hear what Stephen from British Columbia has to say.
Speaker 4: Hi, Daniel and Kelly, this is show super fan Stephen
Speaker 4: from British Columbia, Canada. My question today has to do
Speaker 4: with how scientific studies are peer reviewed?
Speaker 1: Is it me?
Speaker 4: Or is science denial spreading these days? And it came
Speaker 4: to my attention recently that not all peer reviewed studies
Speaker 4: are replicated, And then it occurred to me, doesn't this
Speaker 4: automatically cast doubt on a discovery if a study is
Speaker 4: not replicated but is still considered peer reviewed.
Speaker 3: I would love to.
Speaker 5: Hear an explanation about how the peer reviewing process works
Speaker 5: and how scientists determine when a fact is a fact,
Speaker 5: a finding is a finding, and how the community comes.
Speaker 4: To a consensus on new discoveries.
Speaker 3: I also think your audience.
Speaker 4: Could use some tools for how to determine what studies
Speaker 4: are legitimate and how to spot a bad study. How
Speaker 4: should one respond to someone who claims a scientific study
Speaker 4: is bogus. Thank you Daniel and Kelly for this great
Speaker 4: podcast and for explaining science topics to the masses.
Speaker 3: Thank you Stephen for asking this question. I think it's
Speaker 3: really the right moment to dig into this. You know,
Speaker 3: when we see institutions being attacked, when we see science
Speaker 3: being denied, when we see the whole process of science
Speaker 3: being q and I think it's important to shine some
Speaker 3: light on how does this all work, what does it mean,
Speaker 3: especially for those folks who are not scientists, who don't know,
Speaker 3: what does it mean for a paper to be peer reviewed?
Speaker 3: What is bad science? Thank you Stephen for asking this
Speaker 3: question and giving us the chance to dig into all
Speaker 3: of this.
Speaker 1: That's right, and so the question in particular is about
Speaker 1: peer review for scientific manuscripts. But peer review actually happens
Speaker 1: at multiple stages in the process of doing science, and
Speaker 1: so we thought we would take a step back and
Speaker 1: talk about peer review from the very beginning, which is
Speaker 1: when your idea becomes a grant. And this is one
Speaker 1: of the less fun parts about science is writing this
Speaker 1: grant because peer review for grants is harsh.
Speaker 3: This is like the science version of when a build
Speaker 3: becomes a law. We should have written a song for this.
Speaker 1: No, well we still can. You know, we could write
Speaker 1: it and then we can add it to the top. No,
Speaker 1: we won't do that to everyone.
Speaker 3: So Kelly, tell me what is your process of grant writing?
Speaker 3: First of all, who do you submit your grants to?
Speaker 3: And what's it like for you to prepare that?
Speaker 1: So I submit my grants to the National Science Foundation.
Speaker 1: And actually it's possible that in the last six months
Speaker 1: the process has changed a bit because I know the
Speaker 1: new administration is shaking things up.
Speaker 3: The agency formerly known as the National Science Foundation.
Speaker 2: No, what is it called now?
Speaker 3: I'm joking?
Speaker 2: Oh god, anything is believable, Daniel.
Speaker 1: At this point, all right, the institution still known as
Speaker 1: the NSF or the National Science Foundation. So for this
Speaker 1: you write a grant, and there's different programs that accept
Speaker 1: different kinds of grants, but often the grant is like
Speaker 1: fifteen pages where you go into extreme detail and what
Speaker 1: we know about the question you want to ask based
Speaker 1: on the literature that's already out there, extreme detail and
Speaker 1: the experimental design, and then a lot of detail about
Speaker 1: what you think the implications will be of your results.
Speaker 3: So you talk about why this question is interesting, to
Speaker 3: argue that your work will help answer it, and then
Speaker 3: you lay out in detail how are you going to
Speaker 3: spend every dollar, who's going to do what, and when
Speaker 3: you get like a whole gan chart for like when
Speaker 3: everything is going to happen yep, And then you have
Speaker 3: to lay out also like the products for the grant, right, yeah.
Speaker 1: Yeah, the budgets and so the you're supposed to There's
Speaker 1: a section called intellectual merit where you talk about how
Speaker 1: many papers you think you're going to end up producing,
Speaker 1: what big questions you're going to answer, And then there's
Speaker 1: also a section called broader impacts, where you talk about
Speaker 1: how your work is going to benefit society writ large.
Speaker 1: And you know, for my grants in the past, it's
Speaker 1: been things like, you know, my information is going to
Speaker 1: be helpful to fisheries managers who are managing fish populations,
Speaker 1: which is like a literally a multi billion dollar industry
Speaker 1: for like people who want to go out fishing and stuff.
Speaker 1: So most of my research has been on fish and
Speaker 1: their parasites. Also intellectual merit will include things like how
Speaker 1: many students you're going to train and what skills you're
Speaker 1: going to give them that they can then use when
Speaker 1: they go out in the workforce. And depending on the
Speaker 1: field you're in, sometimes your broader impacts will be things like, Hey,
Speaker 1: this could be what we learn about the brains of
Speaker 1: this fish could one day help us produce a new
Speaker 1: medication for anxiety or something like. So you talk about
Speaker 1: how your stuff would impact the broader society.
Speaker 3: And so for those folks who might think, hey, scientists
Speaker 3: are just cashing in on the government gravy train, tell
Speaker 3: us like, is this the kind of thing you throw
Speaker 3: together in an afternoon and then can confidently think it
Speaker 3: will be funded?
Speaker 2: Oh my god? Okay.
Speaker 1: So one of the reasons, to be honest, one of
Speaker 1: the reasons I sort of transitioned away from academia in
Speaker 1: the most purest sense, so I'm like academic adjacent now,
Speaker 1: but was because I spent too many Christmas Eves working
Speaker 1: on my grants because they were due like sometime in January,
Speaker 1: and they just they were like so many I would
Speaker 1: say months of work. In a lot of cases, if
Speaker 1: it's a new idea, if you're just cleaning up an
Speaker 1: old grant, then it can be once of work. And
Speaker 1: then so many great grants are submitted, and when they
Speaker 1: go to review, you have three peers of yours who
Speaker 1: are in closely related fields review it, and then there's
Speaker 1: a panel discussion where they talk about their views of
Speaker 1: your grant in front of a bunch of other experts
Speaker 1: who can also weigh in. And then all those reviews
Speaker 1: go to program directors who pick out of like the
Speaker 1: twenty best grants, you know, maybe they can only fund
Speaker 1: fifteen or something. And then they pick some on the
Speaker 1: East coast, some on the West coast, some in the
Speaker 1: center of the country, some from major research institutions, some
Speaker 1: from institutions that focus a lot on undergraduate research, and
Speaker 1: they have research programs as well, And so so many
Speaker 1: really great grants don't make it into the pile that
Speaker 1: get funded.
Speaker 3: Meaning that people have read them, have said this is excellent,
Speaker 3: the science is good, it's interesting, it's important. We would
Speaker 3: benefit as a society if we did this. It's all
Speaker 3: well thought.
Speaker 2: Through, but no, that is exactly right, yes, and it
Speaker 2: is soul crushing. I know so many people who are like,
Speaker 2: I gave up Christmas Eve on this grant, and this
Speaker 2: is the third time I've submitted it, and it just
Speaker 2: never makes it quite high enough, even though everybody thinks
Speaker 2: it's amazing. And then quickly just to mention for follow up.
Speaker 1: If you do get grant, every year you have to
Speaker 1: write a progress report saying what you've done, have you
Speaker 1: stuck to your timeline, how have you spent the money
Speaker 1: that you've been given, are you going to hit your timelines?
Speaker 1: And what have you done that impacts society? And so
Speaker 1: every year you have to report on that and then
Speaker 1: at the end you have to write a bigger report.
Speaker 1: So you need to be justifying what you're doing with
Speaker 1: the money every step of the way.
Speaker 3: And before we turn it around and ask me questions,
Speaker 3: who are the peers here, Who are folks who are
Speaker 3: reading your grants and commenting on it.
Speaker 1: There are folks in the same discipline at other universities.
Speaker 1: So you also have to fill out a detailed excel
Speaker 1: sheet where you talk about who in the field who
Speaker 1: does similar work, has been your student or your postdoc
Speaker 1: or your mentor or a co author.
Speaker 3: You might have a conflict.
Speaker 1: Yeah, and anyone who might like you for you and
Speaker 1: who might be too nice to you when they review
Speaker 1: the grant. So it has to be only people that
Speaker 1: you haven't worked with or haven't co author of paper
Speaker 1: with for something like five to ten years. I guess
Speaker 1: they assume that the liking of someone wears off about
Speaker 1: half a decade. But it can be tough because you know,
Speaker 1: these fields are kind of small, but you know, usually
Speaker 1: it's other people at research institutions who do similar work,
Speaker 1: are familiar with the literature and can tell if what
Speaker 1: you're proposing makes sense or not and is good or not.
Speaker 3: And so the thing to take away from this is
Speaker 3: getting a grand funded is hard, right. Not only is
Speaker 3: it a huge amount of work, but you have to
Speaker 3: survive an excruciating process where really only the best are selected.
Speaker 3: It's sort of like watching the Olympics and you're wondering, like, oh,
Speaker 3: here's somebody from this country and somebody from that country,
Speaker 3: and you know that each person has already won some
Speaker 3: sort of like really competitive national competition just to even
Speaker 3: be there at the Olympics representing their country. And so
Speaker 3: like every grant that's funded, even if it seems silly
Speaker 3: and it's about like the sex lives of ducks, you
Speaker 3: know that it's been like excruciatingly written, reviewed in detail,
Speaker 3: questioned by experts without conflicts of interest, and found to
Speaker 3: be excellent, beaten out lots of other grand These are
Speaker 3: not slush funds shoveled to scientists, so you know, just
Speaker 3: like do whatever they want with. These are hard one
Speaker 3: funds to do science that the agency thinks are a
Speaker 3: good idea.
Speaker 1: Yeah, it is absolutely excruciating to do these things. It
Speaker 1: feels great when you get it funded. And I said,
Speaker 1: I hate writing grants. Sometimes I enjoy the process of
Speaker 1: like figuring out just the right experimental design for a question.
Speaker 1: That's like fun for me. But in general, when the
Speaker 1: grant doesn't get funded, it sucks. But anyway, all right,
Speaker 1: so what's your experience.
Speaker 3: Yeah, my experience is very similar. You know, we come
Speaker 3: up with an idea, we spend a lot of time
Speaker 3: polishing it, often doing a lot of the work in
Speaker 3: advance that we need to demonstrate that the work described
Speaker 3: it's reasonable. Right. If it's too far forward thinking, then
Speaker 3: you won't get the money because they're like, well, that
Speaker 3: might work, but can you really prove it. It's too
Speaker 3: much of a risk. If you've already done too much
Speaker 3: of the work, then they're like, oh, this is already done,
Speaker 3: why would we fund it? So it's really sort of
Speaker 3: on the edge there of like you've done the initial
Speaker 3: work for free right or on some other grant or whatever.
Speaker 3: So you prove that the idea is valid and can fly,
Speaker 3: but not so much that they're like, why would we
Speaker 3: fund this? You've already done it, which is often a
Speaker 3: delicate balance. And my feeling with grants is submit it
Speaker 3: and forget it, because such a tiny fraction ever come
Speaker 3: back with money that you just got to like let
Speaker 3: it go, like, hey, I've submitted it, and I never
Speaker 3: expect to hear back, you know, anything positive, And so
Speaker 3: I just sort of like give up emotionally on everyone
Speaker 3: because otherwise it's too hard, you know, it's too hard
Speaker 3: to deal with, yea. So my process is very similar
Speaker 3: to yours, with a couple of exceptions. Sometimes I submit
Speaker 3: to private foundations, like I recently found a private foundation
Speaker 3: that likes to fund projects that are too blue sky
Speaker 3: that were rejected by the NSF for being like way
Speaker 3: too out there, and that submission process was like write
Speaker 3: a paragraph, send us your rejected NSF grant with the views,
Speaker 3: and that's it.
Speaker 4: Whoa.
Speaker 3: And that one actually just came back and they just
Speaker 3: gave me some money to build my cosmic ray smartphone
Speaker 3: telescope down here at Irvine. So yeah, that was actually
Speaker 3: a really positive experience graduations.
Speaker 2: Yeah, that's awesome.
Speaker 3: Yeah exactly, because you know, private foundations can do whatever
Speaker 3: they like with your money. What we talked about previously
Speaker 3: was mostly the process of applying to government institutions, the
Speaker 3: NIH the NSF. Most of my fund incomes from Department
Speaker 3: of Energy, Office of Science, which funds particle physics and
Speaker 3: lots of other stuff in the United States. But private
Speaker 3: fundations can do anything. Like the Bill Gates Foundation, you
Speaker 3: can't even apply to. They have to like reach out
Speaker 3: to you.
Speaker 2: Wow.
Speaker 3: And you know Mackenzie Scott just like gives money to
Speaker 3: people that don't even apply. They just get a phone
Speaker 3: call saying like, by the way, here comes a million
Speaker 3: dollars or more.
Speaker 4: Whoa.
Speaker 3: So yeah, private foundations are weird.
Speaker 1: Yeah, so I'm sure that they make solid choices about
Speaker 1: who they're going to give their money. To But I
Speaker 1: guess one way to sort of check how much you
Speaker 1: can trust an article as you can look in the
Speaker 1: acknowledgment section at the end and say, oh, this was
Speaker 1: funded by the National Science Foundation, And then you know
Speaker 1: this went through rigorous peer review. And just because something
Speaker 1: doesn't go through rigorous peer review at the grant stage
Speaker 1: doesn't mean it's bad science. But at least you know
Speaker 1: if it went through a government agency, it has been
Speaker 1: sort of gone over with a fine tooth coust And the.
Speaker 3: Thing that's frustrating to me is this process is so inefficient.
Speaker 3: Scientists spend so much of their time preparing grant proposals
Speaker 3: and having them rejected. And you know, each of these
Speaker 3: is a great idea, each of them. If we did
Speaker 3: it would benefit society in terms of sheer knowledge or
Speaker 3: you know, new technology or something. We have all these
Speaker 3: smart people constantly pitching great ideas to the government, the
Speaker 3: government saying, yeah, that's awesome, we'd love to do it,
Speaker 3: but we can't. Like, why don't we just double triple
Speaker 3: funding for science. It would be like better for us
Speaker 3: every dollar we spend, we know comes back two threefold
Speaker 3: in terms of like economic output. It just seems crazy
Speaker 3: to me to reject all of these excellent ideas.
Speaker 2: I mean, I gotta say, not every idea I've reviewed
Speaker 2: on a panel has been excellent.
Speaker 3: No, but there are plenty above threshold.
Speaker 2: Yes, absolutely, absolutely.
Speaker 1: There's always a grant every round where my heart breaks
Speaker 1: a little that it didn't get funded because I thought, oh,
Speaker 1: that is so cool, but there wasn't enough money to
Speaker 1: cover it. And so yes, there's a lot of good
Speaker 1: work that's not getting done.
Speaker 3: And I agree. I'm a grant reviewer often and I
Speaker 3: see grants I'm like, no, this is not well thought out,
Speaker 3: or there's a flaw here, or this isn't cutting edge.
Speaker 3: You know, somebody did this last year, and it's important
Speaker 3: that this stuff gets reviewed and gets reviewed in a
Speaker 3: fair way. But there's so many that are above threshold,
Speaker 3: and only a fraction of those get funded, and it
Speaker 3: just seems to me to be a waste. But whatever.
Speaker 3: I'm not a political person. I understand these things. But
Speaker 3: I want folks out there to realize that every grant
Speaker 3: you've seen that's been funded has been reviewed and excruciating
Speaker 3: detailed before dollar one was even sent to the institution.
Speaker 2: That's right.
Speaker 1: I don't think we need to get into lots of
Speaker 1: detail about this, but every once in a while, my
Speaker 1: husband and I will have chats about what would be
Speaker 1: a better system for funding grants, Like maybe every four
Speaker 1: years you get a chance to submit and that way
Speaker 1: you don't have to write grants the other three years
Speaker 1: and there's fewer people in the pool and you're more
Speaker 1: likely to get it. I think that's not the answer,
Speaker 1: but I wonder if there's some way we could save
Speaker 1: scientists from spending all of this time on grants that
Speaker 1: don't get funded, or maybe we should just throw twenty
Speaker 1: times as much money at the scientific community. There's our solution.
Speaker 3: Well, you know, it used to be fifty years ago
Speaker 3: that the philosophy is a little bit different that the
Speaker 3: government funded people rather than projects. And they were like, oh,
Speaker 3: scientist X at this university, you're a smart lady. You've
Speaker 3: done good work. We'll just keep giving you money and
Speaker 3: it'll just go for a while. As long as you
Speaker 3: keep doing something, we'll keep giving you money. And I
Speaker 3: think that we used to be the major model, and
Speaker 3: it's just not that way anymore. Now. It's projects, and
Speaker 3: so if you have a lab at a university, it's
Speaker 3: sort of like running a small business. You know, you
Speaker 3: have to constantly be pitching grants to get funds. You know,
Speaker 3: you're like running a store at the mall. Katrina likes
Speaker 3: to say, you're constantly looking for new ways to contribute.
Speaker 3: The one holdover that I'm aware of is actually experimental
Speaker 3: particle physics is still a little bit of the older model.
Speaker 3: They run competitions for junior faculty, and if you win
Speaker 3: one of these awards, like I want a Outstanding Junior
Speaker 3: Investigator Award when I was a very young professor, then
Speaker 3: you sort of get in the club and then they
Speaker 3: mostly fund you, and your find can go up if
Speaker 3: you do great, and go down if you're less productive.
Speaker 3: But it's much more stable than typically I think, because
Speaker 3: these particle physics projects last like twenty years. You know,
Speaker 3: we build a collider, we expect to use it for
Speaker 3: twenty five years. That's my thinking for why they still
Speaker 3: do it in the sort of older model. But it
Speaker 3: means I don't have to write as many grants because
Speaker 3: I do have one more stable source of funding.
Speaker 2: Wow. So even to this day, you still get a
Speaker 2: chunk of money every year.
Speaker 3: I have to submit a grant every three years to
Speaker 3: propose new work and to tell them what I did
Speaker 3: in the last three years, and so far every time
Speaker 3: my grant has been renewed and continued. So yeah, I've
Speaker 3: been funded from the Armament of Energy for a couple
Speaker 3: of decades now, which is very nice. Yeah, and I'm
Speaker 3: very grateful to the Department of Energy, thank you very much,
Speaker 3: and to all the taxpayers who support them.
Speaker 1: Oh you government, shill, I'm just kidding. I am so
Speaker 1: happy for you. That's awesome. I know there's a couple
Speaker 1: grants for people in the medical field that work that
Speaker 1: way also where they fund your lab and just trust
Speaker 1: that you're doing awesome stuff and you will continue to
Speaker 1: do awesome stuff with that money.
Speaker 2: And that sounds pretty sweet.
Speaker 3: Yeah, it's pretty nice. Yeah. I think you're talking about
Speaker 3: like the Howard Hughes Awards, for example.
Speaker 2: That sounds very yeah. Yeah yeah. All right, Well, my
Speaker 2: jealousy aside.
Speaker 1: Let's all take a break, and when we get back,
Speaker 1: we're going to talk about the next phase when peer
Speaker 1: review is done, and.
Speaker 2: That is when you're about to start your experiments.
Speaker 3: All right, we are back and we are talking about
Speaker 3: peer review. Thanks to a question from a listener, and
Speaker 3: we dug into the process of peer review for grant
Speaker 3: proposals and grant writing. And now let's talk about what
Speaker 3: it's like to review an experiment while it's running, before
Speaker 3: the paper is even sent to the journal. Kelly, I
Speaker 3: mostly do research on particles that don't have rights and
Speaker 3: don't have institutional review boards protecting them. What's it like
Speaker 3: to do an experiment on living creatures with emotions that
Speaker 3: can feel pain and have people looking over your shoulder.
Speaker 2: Oh, it can be pretty stressful.
Speaker 1: So say you get that National Science Foundation grant, they
Speaker 1: won't actually release the funds to you until you've shown
Speaker 1: that you've acquired certain permits and protocols so that your
Speaker 1: institution is giving you permission to do the research. So
Speaker 1: my PhD work required collecting some fish out of estuaries
Speaker 1: in California. So first I had to talk to the
Speaker 1: state government and fill out permits to get permission to
Speaker 1: go collect those fish. So I had to convince the
Speaker 1: California government that I wasn't going to take too many,
Speaker 1: that there were plenty of these fish out there, that
Speaker 1: what I was going to do to them was asking
Speaker 1: a worthwhile scientific.
Speaker 3: Question, Kelly, what are you going to do to.
Speaker 1: Them, I'm going to hear well the questions I'm going
Speaker 1: to ask of them that they will be contributing in
Speaker 1: a meaningful way to science. And so first you have
Speaker 1: to get that permission to take the animals out of
Speaker 1: the wild, and then you need to fill out a
Speaker 1: protocol through the Institutional Animal Care and Use Committee, which
Speaker 1: is iya Cook. It includes five people, and they include
Speaker 1: folks like veterinarians, an outside member, so like somebody who's
Speaker 1: part of the community who is just sort of going
Speaker 1: to weigh in on what the general public feels about
Speaker 1: the work that's being done. Sometimes you'll also have like
Speaker 1: an ethicist or a lawyer in there, and often you'll
Speaker 1: have like a faculty member on there also.
Speaker 3: And what do these people want? Do they want to
Speaker 3: say yes? Do they want to say no? Are they
Speaker 3: just totally disinterested? Like what are their motivations? Why is
Speaker 3: like a rando from the public doing this? Anyway?
Speaker 1: So, if I'm being completely honest, my field in the
Speaker 1: past did some uncomfortable things to animals, and you know,
Speaker 1: they didn't use, for example, anesthetic when they were doing
Speaker 1: some surgeries, and they should have things like that. And
Speaker 1: so this committee is trying to make sure that you
Speaker 1: are treating the animals as nice as possible and using
Speaker 1: the fewest animals that you need to get answers to
Speaker 1: your questions. So you need to convince them that you
Speaker 1: have read up on the most effective anesthetics for whatever
Speaker 1: animal it is that you're using. You also often have
Speaker 1: to take a bunch of online training courses where you
Speaker 1: show I am well trained and the best way to
Speaker 1: treat these animals and I know how these anesthetics work.
Speaker 1: You often have to prove to them that you have
Speaker 1: like teamed up with someone who can make sure you're
Speaker 1: doing the process correctly, and you need to convince them
Speaker 1: that you've thought really hard about the exact number of
Speaker 1: animals that you're going to use for these experiments. You know,
Speaker 1: their job isn't to stop science, and they're not necessarily
Speaker 1: trying to figure you're out of the question that's being
Speaker 1: asked is good or not. They assume that if you
Speaker 1: have funding from the National Science Foundation, that has already happened.
Speaker 1: But their goal is to just make sure that nothing
Speaker 1: unethical happens and that the animals who are contributing their
Speaker 1: lives to science have the best life possible.
Speaker 3: Oh that sounds nice. It is, and it's nice to
Speaker 3: see the process sort of self correcting, like, yeah, okay,
Speaker 3: we trust scientists. Actually, maybe they need some more eyeballs
Speaker 3: on them because they have conflicts of interest, and so
Speaker 3: it's good to have other people who don't have those
Speaker 3: conflicts looking over your shoulder and making sure you're doing
Speaker 3: things with the right way.
Speaker 1: That's nice, it is, And I also appreciate that there's
Speaker 1: a veterinarian on there. The veterinarian will like check in
Speaker 1: pretty regularly. I have training in parasitology, but I don't
Speaker 1: necessarily have training in care of fish or something.
Speaker 2: And I've done a bunch of reading.
Speaker 1: But it's nice to have a veterinarian check in every
Speaker 1: once in a while and offer their expertise to make
Speaker 1: sure that these animals really are being treated as well
Speaker 1: as you can in a lab setting. And I have
Speaker 1: never worked on humans, but if you are doing things
Speaker 1: with humans, including just like sending out survey to humans
Speaker 1: that might ask questions that would make people feel uncomfortable,
Speaker 1: you have to pass those procedures through an institutional review board.
Speaker 1: So there's a similar procedure for working through humans. So
Speaker 1: I guess you don't have anything like that for particle
Speaker 1: physics because we don't care what you do to the particles.
Speaker 3: No, we don't. But I did one time come home
Speaker 3: from work to find Katrina collecting samples for an experiment
Speaker 3: from our children. Oh, And I was like, hmm, shouldn't
Speaker 3: you be asking the other parent pofore experimenting on your children,
Speaker 3: like you have a conflict of interest here? And she
Speaker 3: was like, it's just a saliva sample. I'm like, the
Speaker 3: children can't consent to. So we have a lot of
Speaker 3: fun joking about that.
Speaker 2: Did you give permission? That's good? That's good.
Speaker 1: So, Daniel, there's this new thing that's getting sort of
Speaker 1: big in science that my friends have been talking about.
Speaker 1: I haven't had a chance to do this yet, but
Speaker 1: pre registering a scientific study, have you done this yet?
Speaker 3: We don't actually do this in particle physics, because this
Speaker 3: is another way that particle physics is weird, is that
Speaker 3: we publish our studies when we get a negative result.
Speaker 3: Like lots of fields of science, you might say I
Speaker 3: have an idea for how we might learn something, and
Speaker 3: you know, you do this study and then it didn't
Speaker 3: work or you learned that there's nothing new there like okay,
Speaker 3: bears eat salmon. We already knew that you know like
Speaker 3: or whatever. And often those studies don't get published if
Speaker 3: the result isn't statistically significant or you didn't learn something new,
Speaker 3: and there's a statistical issue there, which is that this
Speaker 3: can lead to a bias in our data and our understanding.
Speaker 3: The effect is called PA hacking, and it means that
Speaker 3: sometimes things that are not real can appear to be
Speaker 3: significant just due to random fluctuation. Say, for example, I'm
Speaker 3: testing a bunch of coins to see if they're fare.
Speaker 3: I flip each one one hundred times. Then like most
Speaker 3: of the time, I'm going to get fifty heads, but occasionally,
Speaker 3: just due to random fluctuations, I'm going to get one
Speaker 3: that gets like seventy heads right. And the more I
Speaker 3: do this experiment, the more likely I am to have
Speaker 3: one that's a fluctuation. And so if I only publish
Speaker 3: the ones that seem to be sick magnificant, that that
Speaker 3: crossed some statistical threshold to be weird, it's going to
Speaker 3: look like these coins are not fair, when in reality
Speaker 3: they are. That's what PA hacking is. P refers to
Speaker 3: the probability for the alternative hypothesis that the coin is
Speaker 3: fair to fluctuate randomly to look like it's not fair.
Speaker 3: And so the way to counteract this is to publish
Speaker 3: negative results, to say, look, I did all these experiments
Speaker 3: and the coin came out fair. We already knew that, yawn.
Speaker 3: But it's important that we include this context so that
Speaker 3: if one in a thousand studies says, look I saw
Speaker 3: one in a thousand times effect, you know what it means.
Speaker 3: And often we do this by pre registering studies by
Speaker 3: saying I'm going to go out and do this study
Speaker 3: and I don't know the results yet, but I'm going
Speaker 3: to publish it either way, right, And so that's a
Speaker 3: way to counteract p hacking.
Speaker 1: So in my field, what you've just described, I think
Speaker 1: we would call the file draw effect. And so the
Speaker 1: idea here is that, yeah, when you get a null
Speaker 1: result or a result that's less interesting, you might try
Speaker 1: to publish it somewhere, but you're less likely to put
Speaker 1: in the effort to try to publish it in a
Speaker 1: lower tier journal that you're going to get less credit
Speaker 1: from your institution for having published in. And that could
Speaker 1: make it look like an effect is there, because when
Speaker 1: you randomly get a positive effect, it gets published and
Speaker 1: otherwise it doesn't. For us, pe hacking is when you
Speaker 1: didn't get the result that you wanted, but you're looking
Speaker 1: at your data and you're like, oh, you know, I
Speaker 1: wasn't actually asking a question about this other thing, but
Speaker 1: if I look at my data, there's actually a statistically
Speaker 1: significant result there, and so I'm going to publish on that,
Speaker 1: and you know, maybe I'll mention that this wasn't the
Speaker 1: initial thing, but I found this, you know, significant relationship
Speaker 1: that's positive, So I'll publish on that. And the deal
Speaker 1: there is, you know, like you said, randomly, you would
Speaker 1: expect to get results that look like something is really
Speaker 1: going on. And if you just are searching through your
Speaker 1: data set, you're more likely to find something that's significant.
Speaker 1: And that wasn't the question you're originally asking. And so
Speaker 1: the reason we do pre registering a scientific study is
Speaker 1: you'll say, ahead of time, I am specifically looking at
Speaker 1: the relationship between I don't know, parasite and how often
Speaker 1: a fish darts. And so if I go on and
Speaker 1: I publish a paper about parasites and how active a
Speaker 1: fish is, you could say, hey, did you just find
Speaker 1: that result? And that you changed your paper to be
Speaker 1: about that because it looked interesting. And so anyway, this
Speaker 1: is how we make sure that you're not looking for
Speaker 1: significant results and just publishing whatever you find.
Speaker 3: And in particle physics we're lucky enough that we always
Speaker 3: publish because a negative result is still interesting. Like if
Speaker 3: you look for a particle because you think it's there
Speaker 3: and the universe would make more sense for it to exist,
Speaker 3: like the Higgs boson, then if you see it, great,
Speaker 3: you publish it. If you don't, that's still interesting. You
Speaker 3: still want to know, oh, there isn't a Higgs Boson,
Speaker 3: because if we smash particles together often enough and we
Speaker 3: don't see it, we can say something about the likelihood
Speaker 3: of it not existing, because if it existed, we would
Speaker 3: have seen it. Now. We still are susceptible to p
Speaker 3: hacking because we will sometimes see fluctuations like data will
Speaker 3: just like look like a new particle sometimes, and to
Speaker 3: that we have a very stringent criteria. It's called this
Speaker 3: five sigma threshold, which means if we only claim the
Speaker 3: particle is discovered if it's very very very unlikely to
Speaker 3: come from a fluctuation, but we still in principle could
Speaker 3: get fooled. And that's why we always have duplicate experiments,
Speaker 3: like at the largechange in cloud, we have two of
Speaker 3: these big detectors and they're totally independent, and that's why
Speaker 3: you expect to see the same physics in both. And
Speaker 3: if you don't, you know something this squarely.
Speaker 1: Yeah, And in my field, we're getting better where if
Speaker 1: you get no result or negative results, as long as
Speaker 1: you can show that you did a sound experiment and
Speaker 1: the answer is no, then you can still publish it somewhere.
Speaker 1: And so Public Library of Science or PLUS is a
Speaker 1: journal that encourages folks to submit they're null results as
Speaker 1: long as the science was good and you can defend
Speaker 1: what you did to try to make sure that you're
Speaker 1: not getting this file drawer effects stuff.
Speaker 2: So you get the no answers just as often as
Speaker 2: the yes answers, and.
Speaker 3: That's helpful, right. It saves people time because if Kelly
Speaker 3: had a great idea and it turned out the answer
Speaker 3: is no, and then she just throws it in the
Speaker 3: file drawer, then Sam across the country is going to
Speaker 3: try the same thing someday and waste their time. So
Speaker 3: it's it's good to get this stuff out there.
Speaker 1: Right, And you know, in my field, wasting time also
Speaker 1: often means that some animals may have died in the process.
Speaker 1: And so to me, I feel like there's this additional
Speaker 1: ethical requirement that you get anything that you learned from
Speaker 1: these animals out so that nobody has to go and
Speaker 1: repeat it and it you know, more animals might have
Speaker 1: to pass away for science. So anyway, good to get
Speaker 1: it out there.
Speaker 3: All right, So then let's talk more about what happens
Speaker 3: when you think you have an interesting result and you've
Speaker 3: written it up and you want to publish it.
Speaker 1: And we're going to leave everybody on a ledge for
Speaker 1: just a second, and when we get back from the break,
Speaker 1: we'll talk more about the fascinating world of peer review.
Speaker 1: All right, Daniel, you have a significant result. You've got
Speaker 1: a result that looks good, or at least even if
Speaker 1: it's no, you're sure it was no because the experiment
Speaker 1: was done.
Speaker 2: Well, what do you do next?
Speaker 3: So we write it up, and when all of our
Speaker 3: co authors are happy with it, the first thing we
Speaker 3: do is we post it online. So particle physics is
Speaker 3: a very very online science, and we invented this thing
Speaker 3: called the archive where scientists post their papers and it's
Speaker 3: called a preprint because we posted before we submitted to
Speaker 3: the journal. So for example, I finished a paper this
Speaker 3: week and yesterday it appeared on the archive, and nobody's
Speaker 3: reviewed it, nobody's confirmed it. It's just out there for
Speaker 3: other scientists to look at. And in a week or so,
Speaker 3: if we don't get like screaming fits from somebody saying
Speaker 3: you stole my idea or this is deeply wrong, then
Speaker 3: we'll submit it to a journal. So the process for
Speaker 3: particle physics is first post it online, then submitted to
Speaker 3: a journal where it's going to get reviewed, and that's
Speaker 3: going to take months and months, but it's so slow.
Speaker 3: The particle physicis just read the pre print server and
Speaker 3: read those papers, and nobody really cares when it comes
Speaker 3: out in the journal later because we already have seen
Speaker 3: the paper months earlier.
Speaker 1: But what if the peer reviewed version catches an error?
Speaker 1: Do you go back and update the archive? Versions out?
Speaker 3: Okay, you can update it. And usually papers in the
Speaker 3: archive are like version three, version four, and there's comments
Speaker 3: in between. You can compare them. You can see the
Speaker 3: mistakes getting fixed, absolutely, And some papers appear in the
Speaker 3: archive and then are never published and that's fine. I'd
Speaker 3: actually had one paper like ten years ago put on
Speaker 3: the archive. It took like more than a year to review,
Speaker 3: and then they rejected it because the paper already had
Speaker 3: like thirty citations, and so they were like, why publish this,
Speaker 3: It's already out there, people are reading it and using it.
Speaker 2: Huh, that's such a stupid reason.
Speaker 3: Yeah, it was really dumb. But then we just gave up,
Speaker 3: and now it's like one of my more cited papers
Speaker 3: and it was never published. The system is silly. It
Speaker 3: is silly, but I know that in other field is different.
Speaker 3: Like my friends in astronomy, they never put papers on
Speaker 3: the archive until after they've been reviewed, and they think
Speaker 3: that if they send it to a journal, it's already
Speaker 3: out in the archive. Those journal referees are going to
Speaker 3: find that disrespectful like that they didn't wait for them
Speaker 3: to chime in. So it's definitely a field by field
Speaker 3: culture kind of thing. How does it work for you guys?
Speaker 1: Yeah, so y'all were definitely the trailblazers. We subsequently came
Speaker 1: up with bio Archive, and I remember when I was
Speaker 1: initially wanting to submit to bio archive, I was hesitant
Speaker 1: to do it because some of our journals did say
Speaker 1: your paper should not be available anywhere else, because we
Speaker 1: want to be the only place where you can find it.
Speaker 1: And that's because journals make money off of people who
Speaker 1: download their articles, and so they didn't want someone else
Speaker 1: to be able to get another version for free somewhere else.
Speaker 1: And on bio archive or archive, all of the articles
Speaker 1: are free. But it seems like it's now become acceptable
Speaker 1: that you can put papers in these pre print servers
Speaker 1: like bio Archive, and so now a lot of people
Speaker 1: will do that, and they do it for a variety
Speaker 1: of reasons. Some of it is they want to get
Speaker 1: feedback early, they want to get the results out out
Speaker 1: there early. But sometimes it's also like if you've got
Speaker 1: a PhD student who wants to go looking for a job.
Speaker 1: On their resume, if they finished a study, they'll write
Speaker 1: in preparation next to the manuscript. But in preparation could
Speaker 1: mean I've got the data and I still need to
Speaker 1: do all the stats and I haven't even started writing.
Speaker 1: Or it could mean I'm about to submit it to
Speaker 1: a journal, and if you've put it on bio archive,
Speaker 1: you're saying, look, it's done. I just need to start
Speaker 1: the peer review process, or even it's in peer review,
Speaker 1: because that process can take a long time. So anyway,
Speaker 1: it's becoming a lot more common to put your papers
Speaker 1: on bio archive, just to make them easier to access
Speaker 1: for other people, and just to show that you really
Speaker 1: have actually finished that study finally, after a decade or
Speaker 1: however long it took, the.
Speaker 3: Clock has still taken on your thesis. There, Kelly, we'll see.
Speaker 2: I'll get into preparation.
Speaker 3: In preparation, so then you send it to a journal,
Speaker 3: and you pick a journal, like if you think it's
Speaker 3: a really exciting paper, you send it to Nature or Science.
Speaker 3: If you think it's less exciting, you send it to
Speaker 3: a journal with a smaller impact factor. So in particle
Speaker 3: physics we often publish in like Physical Review or in
Speaker 3: Journal of highind Physics or these kinds of journals, and
Speaker 3: it goes to an editor, and an editor finds people
Speaker 3: to review it. These are the peers, and they reach
Speaker 3: out to folks they know who are experts in the field,
Speaker 3: who aren't your supervisor or your brother or this kind
Speaker 3: of thing, and ask them to review these things. And
Speaker 3: Katrina is a senior editor on a journal, so I
Speaker 3: see this process all the time, and she has to
Speaker 3: read the paper, think about who might know about it,
Speaker 3: and then find people to review it. And usually you
Speaker 3: have one to three people read this paper and give
Speaker 3: comments on it. But finding reviewers can be tough.
Speaker 1: So for our grants that we talked about earlier, you
Speaker 1: have to make sure that the person reviewing the grant
Speaker 1: doesn't have a conflict of interest. They're not your brother
Speaker 1: or your supervisor or something like that. And in my field,
Speaker 1: the same holds true for when you're reviewing manuscripts. Is
Speaker 1: that true for your field as well?
Speaker 3: Yeah? Absolutely, Yeah. You have to say, oh, I can't
Speaker 3: read this one because that was my student or something
Speaker 3: like that.
Speaker 2: Yeah. How long does review usually take for your.
Speaker 3: I'd say it's you know, like four to eight weeks
Speaker 3: or something like that.
Speaker 1: Wow, how about you, guys, I'd be happy four to
Speaker 1: eight weeks. That sounds good, but it can be you know,
Speaker 1: sometimes six months, and at six months you start writing
Speaker 1: the editor being.
Speaker 2: Like, come on man, this is crazy. But yeah, it
Speaker 2: can be hard to find reviewers.
Speaker 3: Because reviewers are just other scientists busy writing their own
Speaker 3: grants and writing their own papers and doing all their
Speaker 3: work and picking up their kids from daycare and stuff
Speaker 3: like this and trying to get through that mountain of laundry.
Speaker 3: You know. I think that people forget that, like science
Speaker 3: is just people, right, and people are busy. Yeah, And
Speaker 3: if you're out there and you've written some treatise on
Speaker 3: the fundamental physics of the universe and nobody has read it,
Speaker 3: it's not because we're gatekeeping. Is just because we're busy,
Speaker 3: you know. And there's lots of those. And I often
Speaker 3: say no to editors who ask me to review stuff
Speaker 3: because I'm just too busy to do it in a
Speaker 3: timely manner.
Speaker 2: Yeah, no, me as well.
Speaker 1: And I think it's worth noting that while we're doing
Speaker 1: this review work, we're doing it for free. Yeah, Like
Speaker 1: when you're on a grant review panel, you can lose
Speaker 1: weeks of time to reviewing these things and you don't
Speaker 1: get reimbursed for that time. And the same thing goes
Speaker 1: So when I review a manuscript, I read it for
Speaker 1: one day, takes me a couple hours to read carefully,
Speaker 1: and then I sit on it for a few days,
Speaker 1: and that's the main thing that is occupying my brain
Speaker 1: in the background for those days. And that could be
Speaker 1: like I could have been thinking about the introduction to
Speaker 1: my next book or something, but instead I'm thinking about
Speaker 1: the methods to.
Speaker 2: Make sure it made sense.
Speaker 1: And then I spend another three to four hours reading
Speaker 1: it again and writing in detail my comments. And if
Speaker 1: I think there's some literature they missed, I go and
Speaker 1: I find it and like it's a multi day process.
Speaker 1: When I commit to reviewing a paper, and reviewers don't
Speaker 1: get paid for that, nor do the editors.
Speaker 2: How long does it take you?
Speaker 3: Well, I wish that all of my papers were reviewed
Speaker 3: by somebody as thorough as you. But my process is similar,
Speaker 3: like an initial read, some thoughts, and then let it sit.
Speaker 3: Sometimes I'll discuss it with people in my group. We'll
Speaker 3: read it together, and then I read it again in detail.
Speaker 3: And then, especially if the review is negative, I wait
Speaker 3: and I sit on it, and I come back later
Speaker 3: I'm like, was this fair? And also, most importantly, was
Speaker 3: I nice enough on my comments? Constructive and helpful and
Speaker 3: not like just negative or harsh in any way? I'd
Speaker 3: like try to take a sandpaper to anything that's negative
Speaker 3: and smooth it over, because I've been on the other
Speaker 3: end of harsh reviews and it's it doesn't help anybody
Speaker 3: to get zingers in there and like especially because a
Speaker 3: lot of my papers are written by young people and
Speaker 3: it's often their first paper, and then I have to
Speaker 3: show them this review that's like, this guy's a jerk.
Speaker 3: This isn't necessary. I have to explain to them that
Speaker 3: the anonymity here is protecting them. But you know, most
Speaker 3: reviewers are fair and are thoughtful and are courteous, and
Speaker 3: so often we get like helpful feedback, like what about this,
Speaker 3: or have you thought about this question? Or what would
Speaker 3: you say to this concern? Do you find your feedback
Speaker 3: to be mostly useful?
Speaker 1: Yeah, But most of the time the feedback I get
Speaker 1: improves the paper. I did have one reviewer when it
Speaker 1: was one of my first papers who told me they
Speaker 1: were very disappointed in me.
Speaker 3: It's like, you're a jerk, disappointed, thanks dead.
Speaker 2: And that was because my supplemental materials were too long.
Speaker 2: I was like, God, give me a break.
Speaker 1: But anyway, so in my field, we're moving towards double
Speaker 1: blind review, so the reviewers are not supposed to know
Speaker 1: the names of the authors, and the authors are not
Speaker 1: supposed to know who reviewed their paper. And that's a
Speaker 1: really great idea. So that you feel like you can
Speaker 1: honestly tell someone if their paper was good or not.
Speaker 1: You don't have to worry about someone getting mad at
Speaker 1: you in particular, though in practice there's usually like four
Speaker 1: people who could review your paper, so you know, like
Speaker 1: you know, it was Joe or Francceine and Franccene tends
Speaker 1: to be meaner and so.
Speaker 2: But anyway, so yeah, what about mirror field? Is it
Speaker 2: blind or double blind?
Speaker 3: It's only blind to one direction. The review is anonymous,
Speaker 3: but you see who the authors are. I know, in
Speaker 3: some other fields, like in top computer science conferences, for example,
Speaker 3: it's double blind, and that's helpful. But also it's not
Speaker 3: that hard to figure it out. If you wanted to
Speaker 3: know who these people are, you could figure it out.
Speaker 3: But you might also be wondering, like, what is the
Speaker 3: job of the reviewer exactlytly does reviewer have to make
Speaker 3: sure the experiment was right? Like if I'm checking Kelly's work,
Speaker 3: I'm the reviewer, do you have to go out and
Speaker 3: do the experiment and make sure she was right? And
Speaker 3: that's the things that peer review is not replication. The
Speaker 3: job of the reviewer is to ask do the claims
Speaker 3: of the paper are they supported by the evidence provided,
Speaker 3: is their logic? There is there mathematical tissue between the
Speaker 3: work that's been done and the conclusions that are being claimed.
Speaker 3: And also is it well written in the citations? And finally,
Speaker 3: is it interesting? Is it important? Does it play a
Speaker 3: role in the scientific conversation? Which is a little bit
Speaker 3: subjective of course, but science is by the people and
Speaker 3: for the people. But if you're working on something nobody's
Speaker 3: interested in, then nobody's going to read your paper. Then
Speaker 3: a journal might say this is solid work, but nobody cares.
Speaker 3: It's a boring question, or you didn't learn anything interesting.
Speaker 3: The reviewer's job is not to say, hey, I did
Speaker 3: this experiment also, and I know that this is a
Speaker 3: feature of the universe. That's not the task of the
Speaker 3: Reviewer's not their job to do that for you. It's
Speaker 3: also not the task of the reviewer to say, like, hey,
Speaker 3: this is a cool idea. I would have done this differently,
Speaker 3: and so you now need to go back and do
Speaker 3: all these additional studies that I think are important. And
Speaker 3: you see this a lot in peer review that reviews
Speaker 3: come back and say this is cool, but also do
Speaker 3: xyz and like that's frustrating to me, because that's not
Speaker 3: the job, but the reviewer.
Speaker 2: Yes, yep, Nope. That's super frustrating.
Speaker 1: And I think it's worth noting that sometimes bad papers
Speaker 1: get through this procedure, and I one of the ways
Speaker 1: that bad papers get through is that reviewers aren't required
Speaker 1: or and this makes sense, but they're not responsible for
Speaker 1: going through, for example, the codes for the statistical models
Speaker 1: that you ran. And so if somebody you know, check
Speaker 1: their codes a bunch of times, but they forgot a
Speaker 1: minus signed somewhere, the result could be wrong and maybe
Speaker 1: nobody knows. And I have known a couple of people
Speaker 1: who afterwards have gone to use those models again for
Speaker 1: some other question and caught their mistake. And then a
Speaker 1: good scientist will contact that journal and say, I have
Speaker 1: I have to retract my paper, or I have to
Speaker 1: submit errata and let everybody know I forgot the minus
Speaker 1: signed these it's a different result, and that's really painful.
Speaker 1: But at least they're being honest, and that's great, and
Speaker 1: I think that's really important. Or some people are dishonest
Speaker 1: and there's no way for the reviewer to know that
Speaker 1: my field had somebody where they were one of the
Speaker 1: biggest names in the field, and there's this new requirement
Speaker 1: in our field.
Speaker 2: What's a couple decades old? Now I'm old.
Speaker 1: But where you have to put the data that you use,
Speaker 1: that you collected as part of the experiment online, somewhere
Speaker 1: in a public place where people can download your data.
Speaker 1: And again, you know, in a field where animals are
Speaker 1: being used to collect data. It's great to know that
Speaker 1: those data could be used by other people to ask questions.
Speaker 1: You can get more information out of them. But it
Speaker 1: also means that if someone gets suspect results, you can
Speaker 1: pull their data and rerun the models. And when somebody
Speaker 1: was looking at the data, it was clear that the
Speaker 1: numbers just didn't make sense, like one column was always
Speaker 1: the prior column times three and it anyway, after more scrutiny,
Speaker 1: it became clear that this person was making up their data.
Speaker 1: But you know, we have this new check where your
Speaker 1: data have to be available to everyone else, and that
Speaker 1: has helped us sort of tease out people who are
Speaker 1: being dishonest. So the system is evolving and getting better
Speaker 1: over time. But still sometimes you get stuff through.
Speaker 3: You certainly do, and there are folks out there who
Speaker 3: are like combing through papers to find this stuff and
Speaker 3: there's a scientist, for example, her name is Elizabeth Bick,
Speaker 3: and this is her passion. She combs through old papers
Speaker 3: and finds evidence of like photoshopping in biology. You know,
Speaker 3: you take a picture of your gel and it's supposed
Speaker 3: to have this blob and she finds, oh, this blob
Speaker 3: is the same as that blob, or it's been reverted
Speaker 3: or whatever. And so there are definitely lots of ways
Speaker 3: that you could find this stuff now that we couldn't
Speaker 3: have done beforehand. But peer review is not a guarantee
Speaker 3: that this hasn't been done, like whereviewers should look for
Speaker 3: this stuff and call it out if you see it.
Speaker 3: But stuff definitely can get through. It's not a perfect system,
Speaker 3: you know, it's one of these. It's the terrible system.
Speaker 3: But it's also the best that we have so far.
Speaker 2: Just in aside here, I am a massive wimp.
Speaker 1: If I like purposefully fabricated data in this era where
Speaker 1: everything is like public and people can peek behind the curtain.
Speaker 2: I would never sleep again. I'd be like, someone's gonna
Speaker 2: find me out, and it would be like I would
Speaker 2: be miserable the rest of my life.
Speaker 1: I'd rather have a bunch of low impact papers than
Speaker 1: worry that I was gonna get called out for lying,
Speaker 1: but anyway, I am a whim.
Speaker 3: Yeah, And so the highest standard really is not just
Speaker 3: that something has been pure reviewed, but that something has
Speaker 3: been independently replicated. Like you might have a scientist who's
Speaker 3: doing totally solid work and see some effect in their lab,
Speaker 3: but doesn't realize that it's an artifact due to some
Speaker 3: condition the humidity or the local gravity or the trains
Speaker 3: or something are causing some influence on their experiments, And
Speaker 3: so you want people on the other side of the
Speaker 3: world who built it differently, who made different assumptions, who
Speaker 3: sensitive different stuff to reproduce it. You might remember this
Speaker 3: excitement of about high temperature superconductors. A couple of years
Speaker 3: ago LK ninety nine. Korean scientists claimed to have created
Speaker 3: this room temperature, cheap superconductor which would revolutionize the industry,
Speaker 3: and so very quickly people were out there trying to
Speaker 3: replicate it, and people were excited. But until another independent
Speaker 3: group built it and showed that it was real, nobody
Speaker 3: really accepted it and thought we were in a new era.
Speaker 3: And personally, for example, when we were discovering the Higgs boson,
Speaker 3: I saw the data accumulating around a mass of one
Speaker 3: hundred and twenty five. We were looking at it constantly.
Speaker 3: It was building up and up and up. But until
Speaker 3: I heard that my colleagues around the ring also saw
Speaker 3: an effect at the same place, I didn't personally think, Okay, yeah,
Speaker 3: we found this thing. And so this sort of independent
Speaker 3: replication is really sort of the highest standard. What do
Speaker 3: you think, Kelly, I.
Speaker 1: Think so, And to me, this is one of the
Speaker 1: current weaknesses in science, at least in my field. So
Speaker 1: replicating data is absolutely critical. But if you write to
Speaker 1: the National Science Foundation, for example, and you're like, oh,
Speaker 1: I just want to do the same experiment that Maria did,
Speaker 1: but I'm going to do it on another continent, it's
Speaker 1: going to be hard to get money for that because
Speaker 1: it's not a new idea, and it's also probably not
Speaker 1: going to get published in a top tier journal. And
Speaker 1: so if you are training PhD students who are going
Speaker 1: to want to get jobs and they're just replic quote unquote,
Speaker 1: just replicating somebody else's work, that's not going to help
Speaker 1: them get a job as much as following up on
Speaker 1: their own exciting new idea. And so we've got this
Speaker 1: incentive structure. It's actually not super great for encouraging people
Speaker 1: to replicate each other's studies, but I do think it's
Speaker 1: absolutely critical, and I would love to see us sort
Speaker 1: of work on that incentive structure to make replication just
Speaker 1: as important as the initial thing that got done.
Speaker 3: That's interesting. In particle physics, there's maybe a slightly healthier environment.
Speaker 3: We have a few signals of new physics that we've
Speaker 3: seen in experiments that we're all curious about, but nobody
Speaker 3: really accepts because it doesn't quite make sense. And then
Speaker 3: there's been another generation of experiments to follow up on those.
Speaker 3: For example, there's a very significant signal of dark matter
Speaker 3: an experiment in Italy called DOMA, and nobody really believes
Speaker 3: it because we've never seen it in another experiment. And
Speaker 3: people have set up other experiments very similar to DAMA,
Speaker 3: but in another part of the world with different conditions,
Speaker 3: in a different cave, for example, and not seeing the
Speaker 3: same effects. And so those were experiments definitely inspired by
Speaker 3: this signal to test this in other conditions. And in
Speaker 3: the last few years there was this quote unquote discovery
Speaker 3: of a fifth force in this experiment in Hungary, and
Speaker 3: folks in Berkeley are trying to replicate it, and there's
Speaker 3: an experiment in Italy trying to probe it as well,
Speaker 3: And so there's definitely like follow up work. But usually
Speaker 3: those follow up experiments are a little bit broader, and
Speaker 3: they try to not just check this one new result,
Speaker 3: but also learn something else along the way to make it,
Speaker 3: so they're also covering new ground. But I agree we
Speaker 3: should definitely have replication. But it comes back to funding, right,
Speaker 3: Like if your reviewer, what would you rather fund, Like
Speaker 3: let's check Kelly study, or let's do this brand new
Speaker 3: thing that could tell us something new about the universe.
Speaker 1: Yeah, And when I write a grant, I try to
Speaker 1: work my own replication in there, like as part of
Speaker 1: asking a new question, I'm going to repeat the experiment,
Speaker 1: but maybe put a little tweak But then I can
Speaker 1: at least make sure that I'm still getting the same
Speaker 1: results in you know, a different space or with slightly
Speaker 1: different lighting or something like that. And so, and you know,
Speaker 1: especially when to defend my field a little bit, especially
Speaker 1: when animals need to be euthanized as part of the experiment,
Speaker 1: you know, you might be a little bit hesitant to
Speaker 1: be like, well, this is just for replication, Well, if
Speaker 1: you already got an answer, do animals need to get so? Anyway,
Speaker 1: So we do also try to ask additional questions while
Speaker 1: trying to replicate, but it would be nice if we
Speaker 1: had more incentive for that.
Speaker 3: And there's folks out there just doing this, Like you
Speaker 3: might have heard of the replication crisis that comes out
Speaker 3: of people finding papers in the literature and saying like,
Speaker 3: all right, let's reproduce this, let's see if it holds up.
Speaker 3: And this is one reason that like p hacking is
Speaker 3: a thing people talk about, because we discovered that some
Speaker 3: of the results in literature are just statistical artifacts that
Speaker 3: were selected in order to get a paper out there.
Speaker 3: And so I think what you're seeing in a broader
Speaker 3: sentence is science is self correcting. Just like we saw
Speaker 3: that we needed to add external reviewers and members of
Speaker 3: the public when we're talking about the experiments you do
Speaker 3: on animals. Now we see like, okay, we need some
Speaker 3: sort of way to protect against this kind of abuse
Speaker 3: as well. And so you know, the process is a
Speaker 3: living thing. It's not like science is a crisp philosophy.
Speaker 3: What we call science has changed over the last ten years,
Speaker 3: fifty years, one hundred years and it will continue to
Speaker 3: evolve and I hope keep delivering great truths about the universe.
Speaker 1: Yeah, me too, And so Steven asked, how can you
Speaker 1: know if something is good science or not? I feel
Speaker 1: like that was the big push behind the question. And
Speaker 1: so what do you think when you read a paper
Speaker 1: for the first time?
Speaker 2: What do you look for?
Speaker 3: Well, you know, science has to stand on its own,
Speaker 3: So the thing I look for is like, does the
Speaker 3: paper make sense? Does it lay out an argument? Do
Speaker 3: the conclusions follow from the evidence presented? That's the most
Speaker 3: important thing to me. But before I'm in a person
Speaker 3: something believe that something is real part of the universe, yeah,
Speaker 3: I need to see it reproduced by another group. I've
Speaker 3: often reproduced papers myself, especially if they're like statistical, just
Speaker 3: to make sure I understand, like, how is this calculation
Speaker 3: being done? Exactly? How do you get from step three
Speaker 3: to four? Because I really want to understand what are
Speaker 3: the assumptions being made and think about whether those are
Speaker 3: broad enough. So I would say that, yes, peer review
Speaker 3: papers can be wrong, but that's part of science, and
Speaker 3: the highest standard I think is independent replication.
Speaker 1: What do you think, Kelly, Yeah, no, I agree. I mean,
Speaker 1: when I read a paper, I'll look to see, you know,
Speaker 1: what journal is it in. So lately there in this
Speaker 1: day and age, there are some new what we call
Speaker 1: predatory journals where they will encourage people to submit, but
Speaker 1: they publish just about anything they get. And peer review
Speaker 1: isn't so much about peers having a chance to turn
Speaker 1: down a study, but they'll like give a.
Speaker 2: Little bit of input. But the paper gets published one
Speaker 2: way or another.
Speaker 1: So you need to look to see what kind of
Speaker 1: journal it was published in, and then you know, if
Speaker 1: it's a field I know about, I'll look to see
Speaker 1: did you cite the paper that I would expect you
Speaker 1: to be citing. Was your literature search deep enough? And
Speaker 1: where your experiments designed? Well, what else could those results
Speaker 1: have meant? And then who funded the study and stuff
Speaker 1: like that. But you know, if you are a member
Speaker 1: of the general public and you can't do all of
Speaker 1: that and you're just reading a pop side summary, I
Speaker 1: would look for again, like where is this being published in?
Speaker 1: You know, like if it's The Atlantic by ed Young,
Speaker 1: it's probably great. And if the science journalist had other
Speaker 1: scientists way in on what might have been wrong about
Speaker 1: the study, that's a good sign, and so you look
Speaker 1: for where you're getting the information, how critical they seem
Speaker 1: to be, and stuff like that. What do you look
Speaker 1: for in a pop side article.
Speaker 3: Yeah, in a pop side article, I definitely look to
Speaker 3: see whether they have talked to people who are not authors,
Speaker 3: you know, other people in the field who know this stuff,
Speaker 3: and is it just a press release from the university,
Speaker 3: or as a journalist who's actually thought about this stuff
Speaker 3: and written something up. Those are definitely the things I
Speaker 3: look for in popside press releases, because there's the temptation
Speaker 3: and the sort of marketplace of ideas to overinflate the
Speaker 3: meaning of an incremental study.
Speaker 1: Yep, yep, all right, So in general I would say
Speaker 1: that you know, we have this process in place to
Speaker 1: try to make sure that the best ideas are the
Speaker 1: ones that move forward, and that we're all checking to
Speaker 1: make sure no one missed anything important along the way.
Speaker 1: Some stuff gets through either by mistake or because some
Speaker 1: people are unscrupulous, but hopefully that doesn't happen that often.
Speaker 1: But you know, the system is evolving at all times.
Speaker 1: And if you have a study that you're interested in,
Speaker 1: but you don't know if you can trust the press
Speaker 1: release or whatever. You can send it to us, and
Speaker 1: if it happens to me in our wheelhouse, we're happy
Speaker 1: to weigh in and tell you what sort of you know,
Speaker 1: set off our alarm bells in our heads, or what
Speaker 1: we liked about the study, and we're happy to help
Speaker 1: people figure out what was done well and what was
Speaker 1: just squeaking through.
Speaker 3: All right, Thanks very much, Steven for asking this question,
Speaker 3: if for shining a light on the inner workings of science, and.
Speaker 1: If you'd like to reach out to us, send us
Speaker 1: an email at questions at Daniel and Kelly dot org
Speaker 1: and let's see what Stephen had to say about our answer.
Speaker 6: Hi, Daniel and Kelly, it's Steve here. Thanks so much
Speaker 6: for answering my question. I think you shed a lot
Speaker 6: of light on some topics that most of the general
Speaker 6: public are not aware of, and I really appreciate that.
Speaker 6: It's actually really fascinating to learn that just because something
Speaker 6: is peer reviewed doesn't mean it's one hundred percent fact.
Speaker 6: And it's definitely a lot to take away here, so
Speaker 6: I appreciate you guys diving into the topic and looking
Speaker 6: forward to the next episode.
Speaker 1: Daniel and Kelly's Extraordinary Universe is produced by iHeartRadio.
Speaker 2: We would love to hear from you, We really would.
Speaker 3: We want to know what questions you have about this
Speaker 3: extraordinary universe.
Speaker 1: We want to know your thoughts on recent shows, suggestions
Speaker 1: for future shows.
Speaker 2: If you contact us, we will get back to you.
Speaker 3: We really mean it. Answer every message. Email us at
Speaker 3: Questions at Danielankelly.
Speaker 1: Dot org, or you can find us on social media.
Speaker 1: We have accounts on x, Instagram, Blue Sky and on
Speaker 1: all of those platforms. You can find us at D
Speaker 1: and K Universe.
Speaker 3: Don't be shy, write to us