Speaker 1: Hi, I'm Aaron Welsh and this is This Podcast will
Speaker 1: Kill You. You're tuning in to the latest episode of
Speaker 1: the tp w K Y Book Club, where I chat
Speaker 1: with authors of popular science and medicine books about their
Speaker 1: latest work. Since starting this series a few years ago,
Speaker 1: I've gotten to cover some amazing books and I appreciate
Speaker 1: so many of you reaching out with your suggestions for
Speaker 1: books to feature. Keep those recommendations coming, please, and if
Speaker 1: you'd like to take a look at the full list
Speaker 1: of books that we've covered in this series, as well
Speaker 1: as get a sneak peek at ones that are coming
Speaker 1: up in future episodes, head on over to our bookshop
Speaker 1: dot org affiliate page, which you can find on our
Speaker 1: website This Podcast will Kill You dot com. Under the
Speaker 1: extras tab on the bookshop page, you'll find several podcast
Speaker 1: related lists, including one for this book Club and the
Speaker 1: TPWKY Kids book Club, which if you're not following us
Speaker 1: on social media you absolutely should be because Aaron Epdyke
Speaker 1: has been putting together videos reviewing children's books. It is
Speaker 1: such a great resource for sciencey kids books for all
Speaker 1: ages and if you want to share your thoughts on
Speaker 1: these episodes, make topic suggestions submit a first hand account.
Speaker 1: You can get in touch with us using the contact
Speaker 1: us form on our website. Two last things before moving
Speaker 1: on to the Book of the Week, and that is
Speaker 1: to please rate, review and subscribe. It really does help
Speaker 1: us out. And second, you can now find full video
Speaker 1: versions of most of our newest episodes on YouTube. Make
Speaker 1: sure you're subscribed to the exactly Right Media YouTube channel
Speaker 1: so you never miss a new episode drop. Belief is
Speaker 1: a powerful force. It shapes every facet of our lives
Speaker 1: and transforms perception into reality. What we believe to be
Speaker 1: true is not always what is actually true, something I'm
Speaker 1: sure we can all relate to. Maybe you've debated with
Speaker 1: a friend over the answer to a trivia question, like
Speaker 1: you both know the right answer, but your answers are
Speaker 1: somehow different. Or maybe you've had a heated exchange with
Speaker 1: a relative who firmly believes that the moon landing was faked.
Speaker 1: How do we decide what we believe? How can we
Speaker 1: know that what we believe is the truth, and how
Speaker 1: can we convince others of that? These are precisely the
Speaker 1: questions that Adam Kucharski, who is professor at the London
Speaker 1: School of hygiene and tropical medicine asks. In his latest book, Proof,
Speaker 1: The Art and Science of Certainty, Kucharski, who is a
Speaker 1: mathematician that works on infectious disease outbreaks, explores how we
Speaker 1: are inundated with information and increasingly misinformation, that we have
Speaker 1: to evaluate to determine whether or not we should incorporate
Speaker 1: it into our decision making. This extends beyond personal decisions
Speaker 1: which root is best to take to work, what to
Speaker 1: make for dinner. Our world is built upon structures of
Speaker 1: proof with varying degrees of support. That car that you
Speaker 1: drive to work is manufactured under rigorous safety testing, meaning
Speaker 1: there are established guidelines for what is considered safe and
Speaker 1: how to test that same thing. With the food we eat,
Speaker 1: the medicines we take, the buildings we spend time in.
Speaker 1: We don't question so many of our beliefs. To do
Speaker 1: so would leave you frozen, uncertain of which direction to
Speaker 1: move in, what to trust. You'd have no time to
Speaker 1: actually live your life. But when we do scrutinize our certainty,
Speaker 1: we might find a gulf between our beliefs and someone else's,
Speaker 1: and those beliefs and the objective truth. Where does that
Speaker 1: incongruity originate. Why are we skeptical about some things and
Speaker 1: not others? What does it take to make up our
Speaker 1: mind and what does it take to change it? That
Speaker 1: answer might not be the same for everyone. An enlightening
Speaker 1: blend of philosophical musings, political commentary, statistical exploration, and personal reflection.
Speaker 1: Proof is a fascinating read, particularly as this unceasing flood
Speaker 1: of information, both good and bad, shows no sign of stopping.
Speaker 1: Let's take a quick break and then get into things.
Speaker 1: Professor Kochowski, thank you so much for joining me today.
Speaker 2: Thanks for having me.
Speaker 1: I am thrilled to talk with you about your latest book, Proof,
Speaker 1: The Art and Science of Certainty. And before we dig
Speaker 1: into the various forms of proof and how we determine
Speaker 1: a threshold for proof or what different types of proof
Speaker 1: exists for certain situations, I want to start at the
Speaker 1: very beginning. What is proof? Is there a standard definition?
Speaker 2: Yes?
Speaker 3: I think that's a great question, and that I think.
Speaker 3: My background's in math, so I think a lot of
Speaker 3: MICA training was around this idea that you can have
Speaker 3: this definitive knowledge that something is true, and I think
Speaker 3: it's something that people grappled with across fields I mean,
Speaker 3: more of the stories that really struck me was Abraham Lincoln,
Speaker 3: when he was training to be a lawyer, came across
Speaker 3: this word demonstrate and yeah, this kind of beyond reasonable doubt,
Speaker 3: this certainty, and he's like, I don't really understand what
Speaker 3: this is as a concept, and he actually went back
Speaker 3: to all of these ancient Greek mathematical texts to understand
Speaker 3: how can we take what the knowledge we have, build
Speaker 3: on that prove new theorems, use that to prove subsequent knowledge.
Speaker 3: But I think one of the things that was really
Speaker 3: the motivation for the book, and something that I think
Speaker 3: anyone who works with information and decision making and evidence
Speaker 3: happens across very often is it can become quite a
Speaker 3: shifting concepts. I mean, even in mathematics, things that people
Speaker 3: thought were proven turned out had some hidden assumptions or
Speaker 3: human judgments that were kind of lurking there and caused
Speaker 3: a lot of that to collapse. So I think it's
Speaker 3: it's a kind of fascinating concept because it's something that's
Speaker 3: so important in life, not just having knowledge that we
Speaker 3: gradually accrue, but from many of the things we care about,
Speaker 3: whether it's dealing with them, emergency, whether it's a legal case,
Speaker 3: whether it's even just a kind of minor business decision
Speaker 3: in our day, we have to work out where we
Speaker 3: set the bar and how we evaluate what we've got.
Speaker 3: And I think for me that was really the launching
Speaker 3: off point to explore this. You know, how do we
Speaker 3: converge on certainty and what happens when it goes wrong?
Speaker 1: Thinking about the difference between proof and certainty and truth, like,
Speaker 1: what is the relationship between those concepts?
Speaker 3: Yeah, I think that's a great question, and without going
Speaker 3: down the kind of philosophical rabbit or it could have
Speaker 3: been a book.
Speaker 2: On you what is reality?
Speaker 3: Yeah, But I think the way that I approached it
Speaker 3: is just to look at how people thought about this
Speaker 3: in different fields. And again even going back to Lincoln
Speaker 3: and much earlier, there was this this appeal of this certainty,
Speaker 3: this idea that there could be this universe truth, and
Speaker 3: it's why a lot of fields ended up borrowing for mathematics.
Speaker 3: You see it in the US Declaration of Independence. Yeah,
Speaker 3: we hold these truths to be self evident. The visional
Speaker 3: draft was we hold these truths to be sacred and undeniable.
Speaker 3: But Benjamin Franklin didn't like that, because it sounded like
Speaker 3: they were kind of appealing to some divine authority and
Speaker 3: self evidence is just borrowed directly for maths, it's just
Speaker 3: a given truth. And unfortunately it turned out a lot
Speaker 3: of these things about equality weren't self evident. But I
Speaker 3: think that that story of how you think about these things,
Speaker 3: and even when we see in the legal world, a
Speaker 3: lot of it was originally derived from concepts around matter,
Speaker 3: around probability. If you talk about, you know, some of
Speaker 3: these thresholds preponderance of evidence, you're saying it's more likely
Speaker 3: than not, and you're kind of borrowing a lot of
Speaker 3: these kind of probability based ideas, and even in the
Speaker 3: world kind of more experimental design as that kind of developed,
Speaker 3: a lot of it was about I mean, actually some
Speaker 3: of these early studies were almost trying to discount some
Speaker 3: of the influences of religion. You're wanting to understand cause
Speaker 3: of effect in the world, rather than just appealing to
Speaker 3: some other influence.
Speaker 2: And then it for a lot of people it became a.
Speaker 3: Discussion of how do you take the evidence you have
Speaker 3: and how do you link that to a conclusion that
Speaker 3: you want to make, and where do you set the
Speaker 3: bar for that do you try and get ever closer
Speaker 3: to certainty? And there's actually a lot of statistical tension
Speaker 3: about one hundred years ago, no statistical debates. Kind of
Speaker 3: sounds a bit boring, but it was actually this real. Yeah,
Speaker 3: people just almost I wouldn't talk to each other because
Speaker 3: there was this tension between do you just try and
Speaker 3: get ever closer to the truth or do you have
Speaker 3: a framework that allows you to make decisions? And I
Speaker 3: think a lot of times in life we don't get
Speaker 3: to do the academic I'm just going to sit on
Speaker 3: the fence yet I just I don't know, and I'm
Speaker 3: just not going to do anything with life or actions
Speaker 3: that often we have to decide we do something or
Speaker 3: we don't do something, or you know, we say someone's
Speaker 3: guilty or we let them go three or there's these
Speaker 3: decisions we have to make. And so that process of
Speaker 3: interacting with evidence is much more pressure and I think
Speaker 3: that that was one of the real big tensions that
Speaker 3: never fully got resolved.
Speaker 2: Actually, even how.
Speaker 3: We teach statistics at school, we kind of smushed together
Speaker 3: these two very different philosophies, one of this ever higher
Speaker 3: bar for evidence and one where we're sort of outlining
Speaker 3: a framework to make a decision based on the knowledge
Speaker 3: we have.
Speaker 1: When it comes to public health and medicine, there's a
Speaker 1: lot more pressing, you know, need to make decisions, and
Speaker 1: yet this decision is often dragged out for long periods
Speaker 1: of time, and sometimes that is at the urging of,
Speaker 1: you know, someone who has incentive to drag out a decision.
Speaker 1: So one of the examples that you talk about is
Speaker 1: Austin Bradford Hill, who is talking about this relationship between
Speaker 1: cigarettes and lung cancer and saying, oh, we have the
Speaker 1: we have some evidence, and there's still a lot of skepticism,
Speaker 1: but we have enough to make a decision. We cannot
Speaker 1: use uncertainty as an excuse for inaction. Do you feel
Speaker 1: like that, like we've ever truly learned that as a society,
Speaker 1: or has it been you know, players like the tobacco
Speaker 1: industry saying oh, no, this uncertainty, you know, we need
Speaker 1: to push for more and more and more evidence.
Speaker 3: Yeah.
Speaker 2: I think that's so, that's a really good question.
Speaker 3: I think that's a really good example of almost kind
Speaker 3: of weaponized certainty, that you can always set the bar higher,
Speaker 3: and any aspects of life, you can set the bar
Speaker 3: high and higher and higher to the point where you
Speaker 3: just won't do anything. And in action, of course, is
Speaker 3: in itself a decision. And I think Bradford Hill's work,
Speaker 3: you know, he was extremely thoughtful in how we approached
Speaker 3: this because something like smoking, you can't really design it
Speaker 3: like a try. You can't get people to randomly take
Speaker 3: up smoking and see if they get cancer. There's obviously
Speaker 3: ethical reasons about there's also just timeline reasons. You know,
Speaker 3: if you look at the timescale of the intervention versus
Speaker 3: what happened, you might have to wait decades to have
Speaker 3: that clear signal. And so he did a lot of
Speaker 3: poning work with others, linking together the various sort of
Speaker 3: non random data sets you had available. Because one of
Speaker 3: the criticisms, of course, as any days, is yes, smoke
Speaker 3: is more like to get cancer, but maybe there's a
Speaker 3: genetic reason that makes them all like to smoke and
Speaker 3: get this. And he outlined a lot of the ways
Speaker 3: we can think about cause and effect, and I think
Speaker 3: that's a very useful set of concept and some of
Speaker 3: it the obvious ones of the cause that needs to
Speaker 3: come before the effect, or that you know, if you
Speaker 3: have the strength of association more of cigarettes makes you
Speaker 3: more likely to get cancer, if you see that across
Speaker 3: multiple countries, or if you can start to think about,
Speaker 3: you know, the biological plausibility. We see carstinogens in other
Speaker 3: kind of situations as well. None of those things on
Speaker 3: their own is conclusive. You can start to build this
Speaker 3: evidence space. And he made this really good point that
Speaker 3: any knowledge we have, even if it's very confident knowledge,
Speaker 3: is always subject to further refinement. But we still have
Speaker 3: that knowledge at that point in time, and we can
Speaker 3: seek further information. There's been lots more studies of smoking
Speaker 3: since they're early ones, but also that's information that we
Speaker 3: have to do something with. And I think we often
Speaker 3: particularly in the situations with emerging threats or kind of
Speaker 3: early concerns about things, whether it's a health intervention we
Speaker 3: think might be harmful. I mean, what of the examples
Speaker 3: are given the book is the work at the FDA
Speaker 3: around flidamide, which was this for sickness in pregnancy, and
Speaker 3: there was actually a lot of concerns about safety for babies,
Speaker 3: and the FDA blocked it as a result. But on
Speaker 3: the other hand, you get things where there might be
Speaker 3: a lot of value. For example, in reducing smoking for
Speaker 3: health outcomes, and even if there's an uncertainty, And Bradford
Speaker 3: Hill made this nice point of actually the standard you
Speaker 3: should apply for taking action kind of depends a bit
Speaker 3: on the situation you're dealing with, if it's a fairly
Speaker 3: cheap action to take, if it's not too disruptive for people.
Speaker 3: But actually, in his argument he said, smoking is something
Speaker 3: people really enjoy, so we need a kind of.
Speaker 2: Higher barrier if you're get it.
Speaker 3: And I think it's a reasonable point if you're going
Speaker 3: to tell a lot of people to change how they
Speaker 3: live their lives, that the evidence you need is perhaps
Speaker 3: different for something where you can take some action and
Speaker 3: you can unwind that. So it is those kind of
Speaker 3: trade offs that you have available that obviously need to
Speaker 3: play in as well.
Speaker 1: Let's take a short break and when we get back,
Speaker 1: there's still so much to discuss. Welcome back everyone. I've
Speaker 1: been chatting with doctor Adam Kucharski about his book Proof,
Speaker 1: The Art and Science of Certainty. Let's get back into
Speaker 1: things right. The thresholds for certainty is it can be
Speaker 1: different depending on the situation, and then there's also these
Speaker 1: personal thresholds for certainty or evidence, you know, how much
Speaker 1: information do we need? And one of the things that
Speaker 1: you discuss in your book as well is sort of
Speaker 1: what happens when evidence flies in the face of our
Speaker 1: personal beliefs, and how sometimes even despite a mountain of evidence,
Speaker 1: we can just still feel like that's not possible. We
Speaker 1: can't reject it. It's not an intuitive truth, you know,
Speaker 1: what happens, Like what is show us about sort of
Speaker 1: the personal nature of proof and certainty.
Speaker 3: Yeah, I think that's one of the things that really
Speaker 3: kind of struck me in researching that. I mean, even
Speaker 3: in some of these kind of mathematical puzzles examples, it's
Speaker 3: things that you know, I'd come across as a kid
Speaker 3: and convince myself, Oh, that's just that's the answer to
Speaker 3: the puzzle.
Speaker 2: And it was only years later when I was explaining it.
Speaker 3: To someone else or someone else had asked me about it,
Speaker 3: and I sort of went through the thing that convinced me,
Speaker 3: and it just didn't convince them at all. And I
Speaker 3: think that's really interesting. I think we focus a lot on,
Speaker 3: you know, how science works, how methods work, what convinces us,
Speaker 3: and you see this in even a lot of studies
Speaker 3: around political beliefs that people will often try and convince
Speaker 3: others with arguments that convince them, and then you get
Speaker 3: this gap and it's almost like that just fails. And
Speaker 3: I think that's a really interesting step to explore, well
Speaker 3: why does that film. One of the things that I
Speaker 3: find even just kind of in some of the modern
Speaker 3: tools we have in the modern there we are quite
Speaker 3: striking is where we have this desire to explain things. So, yeah,
Speaker 3: few years ago to talk to a bunch of people
Speaker 3: working on AI, and there's a lot of concern about
Speaker 3: I think things that self driving cars that we don't
Speaker 3: understand why they make mistakes. We need that explainability. We
Speaker 3: can't have things we don't trust. And actually, in medicine,
Speaker 3: we have all sorts of things that we know work.
Speaker 3: We now often they work, we don't fully understand the
Speaker 3: physics and biology something like anesthesia, for example. We can
Speaker 3: control the effect it's going to have, but actually all
Speaker 3: the underlying biology and kind physics mechanisms is still more
Speaker 3: work to be done. Things like defibrillation. You know, if
Speaker 3: you give a heart a shock, you can kind of
Speaker 3: reset it against Some of that's understood, but there's still
Speaker 3: those kind of gaps in knowledge, but we know that
Speaker 3: these are useful tools. And even yet, if you run
Speaker 3: a clinical trial, you can assess how effective a treatment is,
Speaker 3: but that on its own will just tell you the effect.
Speaker 3: It won't tell you necessarily all the mechanisms that are
Speaker 3: going on to explain it. But there's these tools that
Speaker 3: we've got, and we've got the evidence to take action
Speaker 3: and use these things.
Speaker 2: We're very happy with.
Speaker 3: And there's other areas of life where actually that inability
Speaker 3: to explain something kind of really bothers us. You know,
Speaker 3: even if self driving cars were much safer other than humans,
Speaker 3: as humans we are start looking to a book, humans
Speaker 3: are not good at driving. You know, it's not a
Speaker 3: massively high bar, but I think it would still make
Speaker 3: people uncomfortable. Even if they would say twice as safe
Speaker 3: in cities were as well very well defined, I think
Speaker 3: it still bother people if every now and again there
Speaker 3: was just an accident that we had no real idea
Speaker 3: of what was happening. And I think that's really important
Speaker 3: to bridge because I think that particularly when you get
Speaker 3: that gap in understanding, that's room for other explanations to
Speaker 3: kind of creep in. And I think that's where we
Speaker 3: start see emergence where things like conspiracy theories, whether it's
Speaker 3: things with kind of incorrect logic, Often it is that
Speaker 3: gap between what we're seeing and the understanding of why
Speaker 3: that's happening.
Speaker 2: I think humans have this.
Speaker 3: Very in many ways, very powerful desire to explain what
Speaker 3: they're seeing. But in some cases where the explanation is
Speaker 3: very hard to untangle, it can lead us astray.
Speaker 1: That's fascinating to think of that, the gap between understanding
Speaker 1: and what is happening. We don't understand how anesthesia works,
Speaker 1: or how Thailand are a menif it truly works. But
Speaker 1: we do understand how vaccines work, for instance, and yet
Speaker 1: there's so many conspiracy theories and misinformation surrounding this thing
Speaker 1: that we do know how it works. I guess what
Speaker 1: good does evidence do if we do not take it
Speaker 1: into account and are not open to it.
Speaker 3: Yeah, And I think for me, a lot of it
Speaker 3: is just understanding at what point that breaks down. I mean,
Speaker 3: even if you if you look at some of the
Speaker 3: COVID vaccines for example, you know, or or even some
Speaker 3: of the kind of other debates around climate interventions. I
Speaker 3: think often it gets very into debating some element of
Speaker 3: the technology, and I think often it's actually just people
Speaker 3: disliked some of the control that was exerted over them
Speaker 3: through mandates or for other things. And actually, you know,
Speaker 3: if you've got an intervention you're unhappy with, you can
Speaker 3: disagree with the intervention of saying look, yeah, for example,
Speaker 3: we know that intervention works, but I disagree with how
Speaker 3: you're implementing it, or you can disagree that the intervention
Speaker 3: actually has an effect, even one step down and just
Speaker 3: say you know, actually, I think there's there's sort of
Speaker 3: deeper problems, or maybe the disease isn't a threat. And
Speaker 3: I think often those kind of levels get get tangled up,
Speaker 3: and I think in a lot of conversations I've had
Speaker 3: with people, often they're sort of deep down concern or
Speaker 3: the thing that they're approaching it with isn't necessarily that
Speaker 3: they've just out of nowhere decided that this isn't a threat,
Speaker 3: or that that technology doesn't work. It's actually in some
Speaker 3: of these instances things are a bit more marginal, and
Speaker 3: you could say, you know, you can make an argument
Speaker 3: either way even if the underlying intervention is effective or
Speaker 3: is going to have this You know, you can make
Speaker 3: there's moral and this it's not just about an sort
Speaker 3: of epilogical question, as I think, kind of understanding where
Speaker 3: those drivers are and also just in our own arguments.
Speaker 3: I think sometimes, you know, I have the conversation with
Speaker 3: people and I think I'm just arguing about the kind
Speaker 3: of the nuances of whether intervention is a good idea
Speaker 3: or not, and they're actually arguing whether it's a problem
Speaker 3: in the first place. We see it vaccines, I guess
Speaker 3: examples more polarized, but even something in climate You know,
Speaker 3: you can have a lot of people who just agree
Speaker 3: on the nature of the effect of climate change. They
Speaker 3: agree on the different levers that we probably have available
Speaker 3: a society, but they might strongly disagree about actually how
Speaker 3: we prioritize those and all of the trade offs. And
Speaker 3: I think it's just understanding what level we're on and
Speaker 3: where the evidence might stop and where it might then
Speaker 3: just be other things that are filtering in on a
Speaker 3: personal level.
Speaker 1: This idea of proof and certainty and truth. It seems
Speaker 1: very intuitive in a lot of ways today, but this
Speaker 1: maybe wasn't always the case, Like when did the concepts
Speaker 1: of truth and the need for these self evident truths
Speaker 1: or certainty or proof when did these come to be?
Speaker 1: And then you know in what fields or what areas
Speaker 1: where they initially applied.
Speaker 3: Yeah, I think that's a great It's easier just to
Speaker 3: think of like the world and sort of science and
Speaker 3: evidences always was as it is. I mean even in mathematics,
Speaker 3: it's idea that we had a universal truth wasn't the
Speaker 3: same throughout history. If you go back to the ancient Egyptians,
Speaker 3: ancient Babylonians, they were much more focused on problems solving.
Speaker 3: A lot of their texts are kind of these these
Speaker 3: kind of puzzles and very much things around kind of
Speaker 3: practical everyday problems. And even if you look at their
Speaker 3: formulas for an area of a circle, they're quite approximate,
Speaker 3: and if you're building something that needs quite a large circle,
Speaker 3: you're probably going to be okay using those, but it's
Speaker 3: not going to give you that really precise truth no
Speaker 3: matter what problem you're working on. And that's something where
Speaker 3: the ancient Greeks mathetations like Euclid came in and tried
Speaker 3: to put things on much more solid footing. So you've
Speaker 3: got these concepts like pie that if you want the
Speaker 3: area of a circle that will just be universally true
Speaker 3: and you won't have this issue of your kind of
Speaker 3: approximation breaks down. And it was then, I think that
Speaker 3: as it sort of came into the Enlightenment here, it
Speaker 3: was very appealing for people that you could have these
Speaker 3: undeniable truths about the world. And I think that's where
Speaker 3: a lot of other fields started growing them as well.
Speaker 3: But even in medicine, if you look at this study
Speaker 3: of cause and effect, a lot of that it was
Speaker 3: the sort of medieval Arabic world that a lot of
Speaker 3: that started to emerge. So a lot of the kind
Speaker 3: of superstition, this idea that disease or conditions just kind
Speaker 3: of come out of nowhere and it's bad people or
Speaker 3: you know, someone's a witch. All this kind of stuff
Speaker 3: that was going around in much of Europe at the time.
Speaker 3: There's a lot of early writings inn around sort of
Speaker 3: the eleventh century saying these aren't supernatural, there's natural causes
Speaker 3: and we can study them. Yeah, we can study them,
Speaker 3: we can work out what the cause of effects were
Speaker 3: a lot of early attempts to try and think about
Speaker 3: concepts that we would now call things like having a
Speaker 3: control group or thinking about how we kind of you know,
Speaker 3: would divide and treat some people and not treat some people,
Speaker 3: and then compend the difference. The conclusions didn't always work out.
Speaker 3: I mean, there was I think one of the earlier
Speaker 3: studies was someone who'd identified correctly the symptoms of meningitis,
Speaker 3: but then concluded that blood letting was really effective for it,
Speaker 3: which probably something in their study design had gone astray.
Speaker 3: But again, it just kind of really and it's one
Speaker 3: of the things you look back on and you think
Speaker 3: it's just it's pretty obvious that we should be doing
Speaker 3: it that way. But even coming into the twentieth century,
Speaker 3: if you look at something like analyzing a medical treatment,
Speaker 3: a lot of the early studies did an alternation method,
Speaker 3: because if you think about it, rather than randomized patients,
Speaker 3: you could just say, well, the first patient that comes in,
Speaker 3: I'm going to treat, the second I won't, the third
Speaker 3: I will, forth they won't, and on average you should
Speaker 3: get something that any other sources of variability should balance out,
Speaker 3: and the difference between those groups should be on average
Speaker 3: down to the treatment effect.
Speaker 2: But Bradford Hill actually, who did a.
Speaker 3: Lot of the pilneering work in the early clinical trial space,
Speaker 3: noticed that the groups were often imbalance because what's happening
Speaker 3: as patients were coming in and doctors were maybe subconsciously that, oh,
Speaker 3: maybe that person looks a bit ill, I'll enroll them,
Speaker 3: or maybe they don't meet the diagnosis it And actually
Speaker 3: a lot of the early randomization wasn't statistical. It was
Speaker 3: just it was to sort of keep humans from themselves
Speaker 3: because we couldn't trust subconscious judgment. So a lot of
Speaker 3: the early randomization in medicine wasn't about the statistical properties
Speaker 3: of the trial design. It was just about making sure
Speaker 3: human didn't muck things up basically with their internal biases.
Speaker 1: Well, I mean, we'll find a way, I'm sure somehow
Speaker 1: in some way. It's interesting to think about this idea
Speaker 1: of like self evident truths thinking back to you okay, yes,
Speaker 1: or superstition and this person is a witch based on
Speaker 1: these signs or whatever was that also viewed as proof.
Speaker 3: The story of those trials by old deal is a
Speaker 3: fascinating one as well, because they were used for a
Speaker 3: long time. You could have trial by ord deal like
Speaker 3: by water by fi whatever, and then you could also
Speaker 3: chose trial by duels. So basically the big criminals always
Speaker 3: pick that, and people start to notice like, oh, you know,
Speaker 3: if God is deciding which one's innocent, God tends to
Speaker 3: pick the bigger one like pretty much every time, which.
Speaker 2: Is I think there was that that came.
Speaker 3: But actually one of the reasons they stopped using them
Speaker 3: is a lot of the religious scholars became concerned that
Speaker 3: they were basically trying to by running those trials, you're
Speaker 3: essentially trying to get God to do your work for you,
Speaker 3: and that felt for them a bit awkward because you're
Speaker 3: sort of on demand saying, hey, can you come and
Speaker 3: make a decision for us, which they it sort of
Speaker 3: got quite uncomfortable with. But even those early systems, I
Speaker 3: mean early juries in England were kind of fascinating because
Speaker 3: they they weren't the structure that we had today. They
Speaker 3: kind of did their own investigation. So often someone was
Speaker 3: accused and then they went off and accused someone else
Speaker 3: and kind of did their own thing, and it was
Speaker 3: it was only over time that system kind of evolved
Speaker 3: of having that way and converging something and I think
Speaker 3: that's we talk a lot about the problem of black boxes,
Speaker 3: but to some extent, juries and talking to legal scholars
Speaker 3: was kind of interesting with this that it's not so
Speaker 3: much about getting to the truth. It's having a system
Speaker 3: where you can reach a decision and you've got kind
Speaker 3: of that finality or send me from finality to that decision,
Speaker 3: and having a system that works, rather than you know,
Speaker 3: your one hundred percent convinced of that. And I think
Speaker 3: we see that kind of across different fields, of that
Speaker 3: emergence of truth. And you said, what's kind of obvious
Speaker 3: and what self evident? I mean, one of the other
Speaker 3: things that I found kind of interesting was how many
Speaker 3: mathematicians were deeply influenced by religion. So Newton, for example,
Speaker 3: Isaac Newton, driving all these equations and theories about planets
Speaker 3: and planetary emotion, he saw that it was God keeping
Speaker 3: the universe in balance, and he was essentially just observing
Speaker 3: divine influence. So for him, although he was doing a
Speaker 3: lot of this scientific work, he saw that there was
Speaker 3: this external influence keeping it all in place along the way.
Speaker 3: So even quite far through history, you had these kind
Speaker 3: of other baseline explanations going on. I think even in
Speaker 3: the modern era. I think the way sometimes we tell
Speaker 3: the story of science I think is sometimes almost a
Speaker 3: bit disingenuous. If you read a scientific paper, it's kind
Speaker 3: of yet there's this problem, and I decided to run
Speaker 3: this experiment and I got these results. But I think
Speaker 3: there's also just that element of what was the hunch
Speaker 3: that made you think that that might be an interesting
Speaker 3: thing to investigate? What was that spark of inspiration? I think,
Speaker 3: even in this era of AI, it's a really interesting question,
Speaker 3: because AI can kind of process and mimic human decisions
Speaker 3: as we write them down. But I think there's often
Speaker 3: that kind of spark or that idea that would lead
Speaker 3: you to do something that just people wouldn't have tried before.
Speaker 3: And that's much much harder to articulate. So it's not
Speaker 3: necessarily that kind of obviousness that we might have had
Speaker 3: in another era. But I think there still are those
Speaker 3: things which are quite hard to explain in where that
Speaker 3: evidence might have initially sparked from.
Speaker 1: One of the things that you mentioned was the use
Speaker 1: of proof and evidence in the legal system, and I
Speaker 1: feel like this was a really fascinating discussion in your
Speaker 1: book as well, where this is employed as like, you know,
Speaker 1: proof beyond a reasonable doubt or innocent until proven guilty.
Speaker 1: What does this show us about like the variable level
Speaker 1: of evidence needed to make a decision, and I guess
Speaker 1: like the different forms that proof can take in this setting.
Speaker 3: It's a really interesting question about how different societies have
Speaker 3: even said that that balance me Essentially, in a legal case,
Speaker 3: there's two main errors you can make that someone can
Speaker 3: be guilty and you can let them go three or
Speaker 3: they can be in you can convict them. And William Blackstone,
Speaker 3: who is a legal scholar, in the seventeen sixties came
Speaker 3: up with wats known as Blackstone's ratio. He said it's
Speaker 3: better for ten guilty people to go free than one
Speaker 3: innocent to be convicted. And Benjamin Franklin actually and even
Speaker 3: accortiately said it's better for one hundred guilty people to
Speaker 3: go free of one isn't to be convicted.
Speaker 2: Seeing that as a kind of balance.
Speaker 3: Other cultures, particularly some communist regimes in twentieth century, set
Speaker 3: it the other way. It's like it was better for
Speaker 3: ten innocents to be in prison than warner guilty to
Speaker 3: go free, because there's this kind of trade off and
Speaker 3: where they're seeing it as the worst error. And actually
Speaker 3: some analysis looking at US legal cases, obviously they don't
Speaker 3: try and target these error rates, but you can sort
Speaker 3: of infer how people are valuing this. A lot of
Speaker 3: them seem to land between that kind of Blackstone and
Speaker 3: Franklin ratio of error. But then, of course, yes, the
Speaker 3: different evidence and how it makes it its way into
Speaker 3: the court room, particularly some of the examples historically of
Speaker 3: kind of things like early probability. And again, one of
Speaker 3: the challenges here is what What's god I talked to
Speaker 3: called the weak evidence problem, and I think a lot
Speaker 3: of how we navigate life is around probabilities that are
Speaker 3: quite likely. You know, a lot of probability theory was
Speaker 3: originally developed around like dice games and things. You know,
Speaker 3: you can study and you can quantify. But in legal cases,
Speaker 3: we often have this weak evidence problem where someone ends
Speaker 3: up in some extremely bad looking situation from a guilt
Speaker 3: point of view, and you're like, well, it's extremely unlikely
Speaker 3: this is just a coincidence. But then if you think
Speaker 3: about it, like, well, this person might just be a normal,
Speaker 3: everyday person. That what, it's extremely unlikely too that they're guilty.
Speaker 3: So you have these two extremely unlikely events, and a
Speaker 3: lot of statistics just isn't equipped to handle that. And
Speaker 3: so there's this notion it's called the prosecutor's fallacy where
Speaker 3: people say, well, this is the probability that that will
Speaker 3: all be a coincidence, and therefore that's the probability they're innocent,
Speaker 3: But of course you've got to weigh it against the
Speaker 3: fact that it's extremely unlikely they're guilty as well. And
Speaker 3: we see this even in other areas. So the work
Speaker 3: we do dealing with like emerging threats and you know,
Speaker 3: pre COVID, there were some studies and actually we did
Speaker 3: a TV show where you sort of say, oh, it's
Speaker 3: you know, a pandemic could just be round the corner.
Speaker 3: Or there's another study that the World Bank I think
Speaker 3: put it at one percent, and you're like, well, what
Speaker 3: is that?
Speaker 2: That's is that right? We was that a good prediction?
Speaker 2: Was that bad? And it's these these very unlikely events.
Speaker 3: I think in legal cases again, for that weak evidence problem,
Speaker 3: it's less about do we definitively work out with high probability,
Speaker 3: which is true, and it's more just we have to
Speaker 3: converge on the best explanation for what we've seen given
Speaker 3: those two possibilities, and in reality we may never have
Speaker 3: certainty about where we are. And I think it's something
Speaker 3: that kind of struck me both think about that, and
Speaker 3: then also thinking about a lot of people who have
Speaker 3: to plan for emergencies and very unlikely events. Thinking a
Speaker 3: lot of the way we traditionally think about probability can
Speaker 3: very quickly lead us astray because I think we're so
Speaker 3: used to having this idea. Well, I can be ninety
Speaker 3: nine percent sure that this happened, But actually it's much
Speaker 3: more about that balancing app that we have to perform.
Speaker 1: Let's take a quick break here, We'll be back before
Speaker 1: you know it. Welcome back, everyone, I'm here chatting with
Speaker 1: doctor Adam Kucharski about his book Proof. Let's get into
Speaker 1: some more questions thinking about this in the context of COVID,
Speaker 1: when you know, things were evolving very rapidly, the situation
Speaker 1: was evolving rapidly, and the general public and you know,
Speaker 1: of course government officials wanted answers and wanted decisions. You know,
Speaker 1: what is the best thing to do, wear masks, not
Speaker 1: wear masks, sanitized groceries, all these things that were just
Speaker 1: constant questions and neat and people wanting hard answers like
Speaker 1: just yes, period. And of as someone who was on
Speaker 1: the informational front lines of the COVID pandemic, what was
Speaker 1: your relationship with uncertainty, Like at that time, did you
Speaker 1: struggle with feeling like we don't have enough information yet?
Speaker 1: You know, how did that feel? I guess in your position?
Speaker 3: Yeah, I think I mean those those kinds of situations
Speaker 3: were normously chanted, both in terms of evidence generation and
Speaker 3: communication and then obviously the political decision making that comes
Speaker 3: off the.
Speaker 2: Back of it.
Speaker 3: I think in many of those situations, I found it
Speaker 3: useful to kind of convert in some cases uncertainty around
Speaker 3: the exact estimate to just just to kind of poorly
Speaker 3: what situation we're in. So for example, when I think
Speaker 3: it was the delta variant emerge and we did a
Speaker 3: lot of the work identifying the early advantage it had,
Speaker 3: and it really wasn't.
Speaker 2: Was it for thirty percent? Was it forty percent? Was
Speaker 2: it sixty percent?
Speaker 3: But essentially all of those were a big problem and
Speaker 3: it's kind of arguing like is this you know, is
Speaker 3: this a disaster or just a catastrophe or just very
Speaker 3: very bad And it's like right, right, Yeah, from a policy,
Speaker 3: you don't need to kind of necessarily commune out as that.
Speaker 3: You just say, like we're very confident that it's going
Speaker 3: to take off a couple of things I think that
Speaker 3: jumped out for me. I think one was the need
Speaker 3: to triangulate across data sources. I think sometimes people have
Speaker 3: this idea of science that you go out and you
Speaker 3: run a study, and that study gives you the answer.
Speaker 3: It doesn't give you the answer. And there were quite
Speaker 3: a few of the early skepticism were saying, well, actually,
Speaker 3: this study wasn't definitive, and this study wasn't definitive. But
Speaker 3: once you start to look at all of those, you know,
Speaker 3: you start to look at the evacuations flights, you start
Speaker 3: to look at the testing data and the contact tracing
Speaker 3: and the big testing of you know, some of the
Speaker 3: cruise ships. You start to look at the clinical data.
Speaker 3: All of those signals start to drag you in the
Speaker 3: same way. And again, each bits of those evidence on
Speaker 3: their own might have problems, but you can start to
Speaker 3: bring together and draw that into a conclusion. I think
Speaker 3: we saw that across the pandemic that if you view
Speaker 3: it very much as like I'm going to get the
Speaker 3: perfect study it's going to give me the answer, you'll struggle.
Speaker 3: But often you can actually find a lot of complementary
Speaker 3: data sources that all for variants or a lot of
Speaker 3: that early severity were all pointing in the same direction.
Speaker 3: I think it's harder, obviously when they're pointing in different directions,
Speaker 3: as we saw with some of the interventions, where it
Speaker 3: was less clear because different countries, different economies, certain things
Speaker 3: did affect behavior and other things in different ways.
Speaker 2: But I think the other challenge.
Speaker 3: That kind of jumped out, and I think a lot
Speaker 3: of the health issues we deal with in the US,
Speaker 3: UK and the modern era are non contagious, so they're
Speaker 3: very much kind of individual you know, things like cancer,
Speaker 3: things like heart disease. It's just very much individual focus.
Speaker 3: So you have someone who's ill, do you treat them,
Speaker 3: do not treat them. If you don't treat them, that's
Speaker 3: someone who's one person who's going.
Speaker 2: To get worse.
Speaker 3: But contagious health threats have this dependence where you know
Speaker 3: a problem can get worse and that problem can then
Speaker 3: accelerate in very different ways. I think that was something
Speaker 3: that was quite a challenge to communicate, cause I think
Speaker 3: a lot of people had this notion of you've got
Speaker 3: normal life and then you could do something else that's
Speaker 3: not normal life. And I would obviously just prefer it
Speaker 3: to be normal. But I think as we saw globally,
Speaker 3: you didn't get that status quote. I mean, that was
Speaker 3: that was gone, and no country had They had varying
Speaker 3: levels of normality, but no country had like just you know,
Speaker 3: pretend absolutely nothing happened. You either had in varying degrees
Speaker 3: depending on the structured society and advantages they had in
Speaker 3: terms of demography and healthcare and other stuff, big changes
Speaker 3: in behavior or borders, whatever, or you saw a.
Speaker 2: Huge amount of death.
Speaker 3: And I think that's something that can be from an
Speaker 3: evidence point of view, much more challenging, because I think
Speaker 3: just in life, we're much more used to those kind
Speaker 3: of linear problems where you know, like with cancer or something,
Speaker 3: these are event tragic events that happen sort of distributed
Speaker 3: across the population, rather than something that the worse it gets,
Speaker 3: the worse that worseness accelerates.
Speaker 1: You mentioned how we have these different data sources, these
Speaker 1: different you know, studies that are all leading us in
Speaker 1: a certain direction, and we have by this point in
Speaker 1: time developed ways to measure both the quantity and quality
Speaker 1: of evidence. I really enjoyed your discussion on un randomized
Speaker 1: controlled trials because this quote unquote gold standard of medical
Speaker 1: studies that might not always be the gold standard, And
Speaker 1: I was hoping you could tell me a little bit
Speaker 1: about the times when the true gold standard might not be,
Speaker 1: for instance, an RCT, but it might be something else entirely,
Speaker 1: or it might be unethical to do a randomized control
Speaker 1: trial in that situation.
Speaker 3: Yeah, I think we've seen quite a lot of examples
Speaker 3: where treating it as that kind of cookie cutter, this
Speaker 3: is the only method we can use coleadisant problems. I mean,
Speaker 3: smoking cancers are very well known one that we couldn't
Speaker 3: just have in action because you can't get that level
Speaker 3: of perfection.
Speaker 2: I mean actually even the first.
Speaker 3: Randomized control trial in modern medicine, which is ninety forty seven,
Speaker 3: so stretcht to mycin, a trial for TB.
Speaker 2: Austin Bradford Hill, who led.
Speaker 3: That, made the point that actually Stretto mycin had some
Speaker 3: very promising looking lab data and kind of early signals,
Speaker 3: and he suggested it would have been unethical to withhold
Speaker 3: it from patients if it was available. But actually as
Speaker 3: a ninety forty seven there were currency controls. The UK
Speaker 3: and its post war state couldn't get enough dollars to
Speaker 3: buy uptimize it. So there wasn't enough to go around.
Speaker 3: So in that situation, they said it would be ethical
Speaker 3: to randomize because there's not enough of it. So there's
Speaker 3: not enough of it, you might as well randomize and
Speaker 3: just learn something along the way. And I think we've
Speaker 3: seen that in other situations. I mean other sort of
Speaker 3: examples that you see where things are very difficult to randomize.
Speaker 3: You can think about natural experiments, a lot of the
Speaker 3: well known moms the Vietnam Draft, where people essentially randomly
Speaker 3: assigned to go to war based on their birthdays. A
Speaker 3: lot of economists have done in Nobrock pires winning work
Speaker 3: using that to understand the effects of war on subsequent
Speaker 3: life outcomes, because it's not something where you can fully
Speaker 3: design that experiment, but you can then make use of
Speaker 3: what you have available. So I think a lot of
Speaker 3: it just comes down to this issue we want to
Speaker 3: understand called effect, and the benefit of randomization is a
Speaker 3: lot of the other things that would influence whether or
Speaker 3: not you know, someone's getting a vaccine and someone's getting
Speaker 3: a disease, because you're randomizing on the vaccine, on average,
Speaker 3: those will cancel out as effects, so it gives you
Speaker 3: that quite neat benefit. But of course, you've also got
Speaker 3: the challenge that you might run a population in one
Speaker 3: group when one population that doesn't generalize to someone else.
Speaker 2: You've also got the time issue.
Speaker 3: So for diseases that evolve, you know, you might run
Speaker 3: a trial now against flu or COVID or something a
Speaker 3: year later that's going to be a different variant, and
Speaker 3: to what extent can you carry over those conclusions. I
Speaker 3: think we see a lot of examples in the literature where,
Speaker 3: for instance, someone might run a trial in one population
Speaker 3: for one disease, for flu, for example, and then see
Speaker 3: a very different result when people look at population patterns elsewhere,
Speaker 3: because it's a different immunse structure, it's a different strain,
Speaker 3: it's a different time period. And yeah, I think we
Speaker 3: can't just say, well that study from a few years
Speaker 3: ago as the gold standard, we're only going to use
Speaker 3: that one. We have to think about how these things
Speaker 3: move along. I mean that being said, though, I think
Speaker 3: in COVID there were missed opportunities I think to gather
Speaker 3: much stronger data. I think it's very hard to justify
Speaker 3: running those kinds of studies As a threat increases. I
Speaker 3: think when epidemic's going up, your time to kind of
Speaker 3: try and randomize it. I mean, I think essentially countries
Speaker 3: have to take that threat, as the evidence suggests, but
Speaker 3: I think particularly as countries lifted measures, that was often
Speaker 3: just done in quite an ad hoc way, and we
Speaker 3: could have done much more kind of staging. In the UK,
Speaker 3: there were some early studies, for example, of can we
Speaker 3: use rapid tests so people test themselves every day rather
Speaker 3: than quarantining for like a week or two, and then
Speaker 3: in practice a lot of people just didn't bother But
Speaker 3: apart from that, you know, I think there's a lot
Speaker 3: of these debates we're still having, and we probably could
Speaker 3: have got better answers for that with some higher quality studies,
Speaker 3: so not necessarily even an RCT, just just making use
Speaker 3: of what we had with more observations.
Speaker 1: One thing that I feel like during the COVID pandemic,
Speaker 1: especially the early months, was this desire from the public
Speaker 1: to have the answers. And I feel like there's a
Speaker 1: lot of variation in how willing someone is to say
Speaker 1: I don't know. And I'm wondering your feelings on this.
Speaker 1: Do you feel that scientists in particular have a difficult
Speaker 1: time so saying that they don't know the answer to
Speaker 1: something like do we need to embrace uncertainty more in
Speaker 1: as scientists or do you feel like there's that we
Speaker 1: are embracing it but just not communicating it. Well, Yeah,
Speaker 1: I think.
Speaker 3: That's a really good question. It's kind of how I
Speaker 3: guess how personality in politics and all these things go.
Speaker 3: So and I think, I mean, there's been good reviews
Speaker 3: of evidence showing that the overstated certainty just just undermines
Speaker 3: trust and confidence, whether it's kind vaccine to its other things.
Speaker 3: So saying yeah, this is one hundred percent, say there's
Speaker 3: absolutely no risk, and if there's even a tiny risk,
Speaker 3: you then kind of undermined that.
Speaker 2: Yeah. One of the challenges with kind of that over.
Speaker 3: Certainty, I think, particularly once you make that public statement,
Speaker 3: it's very hard to back that, and we saw that
Speaker 3: with some of the airborne trying some even health organizations
Speaker 3: say it's not airborne fact. It's very difficult to then
Speaker 3: walk that back. But I think it's fine line because
Speaker 3: you don't just want to say we have no idea.
Speaker 3: You want to try and communicate the way that evidence.
Speaker 3: I think some countries did that better than others, particularly
Speaker 3: them Mark Singapore s pling to mind on their reopening
Speaker 3: where they said, this is the data we're looking at
Speaker 3: to do this, that might change and this is kind
Speaker 3: of how we have to work things through.
Speaker 2: But I think one of the one of the.
Speaker 3: Difficulty I think because any emergency it goes on for
Speaker 3: that long, is you know, you have some people who
Speaker 3: have very loudly said something's one hundred less, one hundred
Speaker 3: times less severe than it is, and then they're kind
Speaker 3: of very nailed onto to having to keep promoting that.
Speaker 3: And I think it is There was one that one
Speaker 3: of the government advisory committees I sat on. You know,
Speaker 3: so a lot of the early alpha very and early
Speaker 3: delta very and a lot of this early severity came
Speaker 3: out of this group. And there was a phrase that
Speaker 3: became used quite a lot, which was tell me why
Speaker 3: I'm wrong. If you have that discussion where you want
Speaker 3: to get criticism, if you present stuff and especially people
Speaker 3: more senior and say is this is this correct, it's
Speaker 3: very hard for people to kind of come in and say, oh, yeah,
Speaker 3: actually I've sported a problem, especially if there's a power
Speaker 3: dynamics or seniority and other things. So I think there
Speaker 3: was a lot of that thing where people present work
Speaker 3: and right, tell me why I'm wrong, tell me what
Speaker 3: I'm missing. And I think that's quite a healthy attitude
Speaker 3: in that kind of environment to be much more, you know,
Speaker 3: looking for weaknesses and being able to.
Speaker 2: Kind of lay out.
Speaker 3: And I remember, actually I think it was when was
Speaker 3: it the gamma variant is sort of emerging in Latin
Speaker 3: America And I gave immediate interview, and when they write
Speaker 3: it up, it was basically, you know, Dotowski doesn't really know.
Speaker 2: It was the kind of open but in that situation,
Speaker 2: we didn't.
Speaker 3: And it is hard to do because I think, you know,
Speaker 3: especially people asking you questions around your area of expertise,
Speaker 3: I think in terms of how to balance that not
Speaker 3: just saying I don't know, but saying, well, we do
Speaker 3: know this, and we can make some judgment. And there
Speaker 3: was this wonderful study in the nineteen fifty one is
Speaker 3: by the CIA analyst, and it was about words we
Speaker 3: use when we're unsure and words about judgment, and it
Speaker 3: basically realized that people used probable and possible to meet
Speaker 3: all sorts of things, and they all know had kind
Speaker 3: of different notions. And he said, yeah, humans will go
Speaker 3: out of their way to making a judgment about something
Speaker 3: that we'll often you know, the risk is you get
Speaker 3: the uncertainty where we're like very hazy, like, oh it's
Speaker 3: you know, it's a definite possibility. And actually, in some cases,
Speaker 3: like with you know, if you've got an emerging threat
Speaker 3: and you've got experts, you do actually want them to
Speaker 3: put a number on it, you know, even if there's uncertainty,
Speaker 3: you want them to say, I am sixty percent sure
Speaker 3: that this is the case. And there's been a lot
Speaker 3: of nice work, you know, even around things like super forecasting,
Speaker 3: where people make those predictions and you can go back
Speaker 3: and then look because you know, if people are well
Speaker 3: calibrated in their uncertainty. Yeah, if you say you're fifty
Speaker 3: percent sure about a list of things, about fifty percent
Speaker 3: at the time those things should happen. So about half
Speaker 3: the things on that list should occur. So there are
Speaker 3: these situations where I think we can get better just
Speaker 3: about thinking about our own uncertainty. And one of the
Speaker 3: things that I actually tried to do, I've tried to
Speaker 3: do a lot of kind of emerging threats is even
Speaker 3: just just writing down what you think is going to happen.
Speaker 3: Because I think we're great, you know, the human mind
Speaker 3: at like kind of rationalizing. Oh yeah, maybe I did
Speaker 3: think that, And so yeah, I did quite quite a
Speaker 3: lot of like where where you could state I actually
Speaker 3: think the vaccine is going to be pretty good, or
Speaker 3: you know, I think this, and like this is where
Speaker 3: social media, when it was maybe slightly less polarized, is
Speaker 3: quite helpful because you could just put a post out.
Speaker 3: And I think I was always very careful. I didn't
Speaker 3: delete any of my tweets during COVID because I was like,
Speaker 3: I actually want that record, and there were time I
Speaker 3: got one. You know, I was in Singapore in feb
Speaker 3: twenty twenty and their policy was don't wear a mask
Speaker 3: unless you have symptoms, and I think I tweeted I
Speaker 3: was like, yeah, that seems like a sensible policy, and
Speaker 3: that seems quite at evidence space. And now we'd probably
Speaker 3: with some of the studies not look back on that
Speaker 3: as being the best post. But so yeah, I think
Speaker 3: it's almost that as well as overstated certainty. I think
Speaker 3: it's also holding ourselves to account, even IF's just you know,
Speaker 3: privately about how confident we were and what played out.
Speaker 1: I want to close out by asking you about the
Speaker 1: subtitle of your book, which is the art and science
Speaker 1: of certainty? And I want to know about the art
Speaker 1: part of this, what is the art s?
Speaker 3: So I think for me it was the more I
Speaker 3: dug into this, the more I saw these other elements
Speaker 3: beyond kind of pure logic, pure observation coming in. I mean,
Speaker 3: even if you look at what was essentially a bit
Speaker 3: of a mathematical civil war in the late nineteenth century,
Speaker 3: where a lot of these ancient Greek theorems, you know,
Speaker 3: things about the properties of triangles started to break down
Speaker 3: because people started to draw shapes on spheres and other
Speaker 3: structures and come up with functions that these supposedly proven
Speaker 3: theorems didn't hold. And I think one of the reasons
Speaker 3: that was really controversial was there was this idea that
Speaker 3: there's a universal truth out there about the world, and
Speaker 3: actually in this situation, it kind of depended on what
Speaker 3: assumptions humans were making and what we were willing to
Speaker 3: kind of define.
Speaker 2: And even in this.
Speaker 3: Supposedly pure subjects, there's still these debates around well, it
Speaker 3: kind of depends on which one you want to pick,
Speaker 3: and that will change the answer. I think even in science,
Speaker 3: it's a lot of these situations where you know, we
Speaker 3: can accumulate the evidence, but then you have disagreement about
Speaker 3: where you set the freshal. I mean that this kind
Speaker 3: of five percent cut off has become very popular, this
Speaker 3: sort of p value that the chance that you'd get
Speaker 3: a result that extreme if there was nothing going on
Speaker 3: or you're no, no hypothesis was wrong. But that was
Speaker 3: kind of arbitrary. I mean it was partly picked just
Speaker 3: for convenience, that this was you one hundred years ago.
Speaker 3: The calculations just a bit easier if they picked a
Speaker 3: value one fished a lot to work, just easier to
Speaker 3: pick a value around point zero five. And others who
Speaker 3: were more pragmatic, you know, working in business on something
Speaker 3: and thinking, well, actually the evidence is a bit weaker,
Speaker 3: but that's still useful to it. So there's this kind
Speaker 3: of human balancing act. And yeah, we saw it again,
Speaker 3: things like legal cases where how much you value different
Speaker 3: types of errors depends a lot on the individuals. I mean,
Speaker 3: one of the examples that that fan fascinating in the
Speaker 3: book was Einstein, when you moved to the US, got
Speaker 3: very angry about peer review because he sent some to
Speaker 3: send something to a journal and it came back as like, oh,
Speaker 3: we've got another opinion on it, and he was like
Speaker 3: whoaah whoa, Like, why haven't you.
Speaker 2: Just accepted my work?
Speaker 3: And actually Max Max Plank, who published some of his
Speaker 3: like amazing early papers, Plank made that point that actually,
Speaker 3: I would rather kind of publish a few things that
Speaker 3: are a bit you know, nonsense, than this is me
Speaker 3: paraphrasing the miss a really important idea. So for him,
Speaker 3: his threshold was like, I want to set the threshold low.
Speaker 3: Admittedly mainly amongst kind of physicists he knew, because I
Speaker 3: don't want to set it too high and miss a
Speaker 3: good idea, and.
Speaker 2: I think we all we all have those.
Speaker 3: This kind of that's where the art, I think creeps
Speaker 3: in that that kind of subjectivity in not just the evidence,
Speaker 3: I think one of For me, the real difference was
Speaker 3: something like proof is it's not just generating data. It's
Speaker 3: how that data interacts with the world and the decisions
Speaker 3: we make. And I think that's where things get really interesting.
Speaker 3: It's like where do we actually set the bar for
Speaker 3: evidence and then both to convince ourselves but then go
Speaker 3: out and convince others too well.
Speaker 1: Professor Kocharski, thank you so much for joining me today.
Speaker 1: This This was such an enlightening conversation, and I really did.
Speaker 1: I loved your book Proof, so I appreciate you coming
Speaker 1: onto the show.
Speaker 2: Thanks great to talk.
Speaker 1: A big thank you again to doctor Adam Kocharski for
Speaker 1: taking the time to chat with me. If you enjoyed
Speaker 1: today's episode and would like to learn more, check out
Speaker 1: our website This podcast will Kill You dot com, where
Speaker 1: I'll post a link to where you can find Proof,
Speaker 1: the Art and Science of Certainty, as well as a
Speaker 1: link to DODR. Kocharski's website where you can also find
Speaker 1: his other book, The Rules of Contagion, Why Things Spread
Speaker 1: and Why They Stop and Don't Forget. You can also
Speaker 1: check out our website for all sorts of other cool things,
Speaker 1: including but not limited to, transcripts, quarantine recipes, show notes,
Speaker 1: and references for all of our episodes, links to merch
Speaker 1: our bookshop dot Org, affiliate account, our Goodreads list, a
Speaker 1: first hand account, form, and music by Bloodmobile. Speaking of which,
Speaker 1: thank you to Bloodmobile for providing the music for this
Speaker 1: episode and all of our episodes. Thank you to Leona
Speaker 1: Squilacci and Tom Bryfocal for our audio mixing, and thanks
Speaker 1: to you listeners for listening. I hope you liked this
Speaker 1: episode and our loving still being part of the TPWKY
Speaker 1: book Club. A special thank you, as always to our
Speaker 1: fantastic patrons. We appreciate your support so very much. Well,
Speaker 1: until next time, keep washing those hands, FU