Speaker 1: Welcome to Stuff to Blow Your Mind, the production of
Speaker 1: My Heart Radio. Hey, welcome to Stuff to Blow Your Mind.
Speaker 1: My name is Robert Lamb, and I'm Joe McCormick. And
Speaker 1: in today's episode, we are going to be focusing on
Speaker 1: a topic that is already something that's very well known
Speaker 1: to people who are familiar with quantitative research and statistics,
Speaker 1: but less known to the general public. And uh and
Speaker 1: I think that's a tragedy because it's an idea that
Speaker 1: should really be part of everybody's basic critical thinking toolkit,
Speaker 1: no matter what your job is. And so in order
Speaker 1: to introduce this concept, I thought it would be best
Speaker 1: to start with a with a direct illustration from the
Speaker 1: real world of people reaching incorrect conclusions by not understanding
Speaker 1: the subject of today's episode. And so the illustration I
Speaker 1: want to start with is an interesting story told by
Speaker 1: the psycholo, just Daniel Kaneman, that's about the illusory power
Speaker 1: of screaming at pilots. Uh. So, the context of the
Speaker 1: story is that Knemon says he was giving a lecture
Speaker 1: about positive reinforcement to a group of flight instructures. I
Speaker 1: think this was in the nineteen sixties, and Kaneman was
Speaker 1: trying to inform them about what he believed at the
Speaker 1: time was the best consensus of scientific research on learning
Speaker 1: and reinforcement, which was at the time that if these
Speaker 1: flight instructors wanted their students to have the best possible outcomes,
Speaker 1: they should focus more on praising the students when they
Speaker 1: did well than on chewing them out when they did
Speaker 1: something wrong. And Kneman says that when he finished his talk,
Speaker 1: one of the flight instructors that he had been giving
Speaker 1: this lecture two got up and tried to dispute him.
Speaker 1: He said, no, you're wrong, And so the direct quote
Speaker 1: Economy gives from the instructor here is on many occasions,
Speaker 1: I have praised flight cadets for clean execution of some
Speaker 1: aero attic maneuver, and in general when they try it
Speaker 1: again they do worse. On the other hand, I've often
Speaker 1: screamed at cadets for bad execution, and in general they
Speaker 1: do better the next time. So please don't tell us
Speaker 1: that reinforcement works and punishment does not, because the opposite
Speaker 1: is the case. So you might think he has a
Speaker 1: good point here if you accept that this flight instructor
Speaker 1: has had a lot of direct experience working with students,
Speaker 1: and you trust him to remember the relative frequency of
Speaker 1: these events pretty well, you might assume that he has
Speaker 1: a meaningful rebuke to ekonom In. Here again, he says
Speaker 1: that most of the time, after a cadet does something
Speaker 1: bad and he screams at them, they do better the
Speaker 1: next time. And after a cadet does something good and
Speaker 1: he praises them, they actually do worse the next time.
Speaker 1: So if he's remembering these experiences correctly, and he's had
Speaker 1: a lot of them, it would really seem like evidence
Speaker 1: that praise has a negative effect on learning, maybe by
Speaker 1: making the student pilots soft and overconfident or something, and
Speaker 1: getting chewed out is good for skill development. I think
Speaker 1: it's quite easy to see the allure of this, this
Speaker 1: false conclusion, right right, And it's and you can also
Speaker 1: easily imagine how you kind of build upon this with
Speaker 1: certain loosely backed up you know, folk ideas about how
Speaker 1: you encourage people and how people learn, and you got
Speaker 1: to stay on them if they if you tell them
Speaker 1: they're doing a good job, they'll get lazy, right, folk wisdom,
Speaker 1: tough guy mentality. Yeah, But Knemon saw something different in
Speaker 1: this response, and he says that he immediately set up
Speaker 1: an experiment on the spot to demonstrate the flaw in
Speaker 1: the flight instructors thinking here, so I want to read
Speaker 1: from Knomen's description, he says, I immediately arranged a demonstration
Speaker 1: in which each participant tossed two coins at a target
Speaker 1: behind his back without any feedback. We measured the distances
Speaker 1: from the target and could see that those who had
Speaker 1: done best the first time had mostly deteriorated their second try,
Speaker 1: and vice versa. But I knew that this demonstration would
Speaker 1: not undo the effects of lifelong exposure to a perverse contingency.
Speaker 1: So to explain this, this experiment a little bit better, right,
Speaker 1: he has people stand with their backs to a target
Speaker 1: so they couldn't see it, and they would take two
Speaker 1: attempts to throw a coin and hit the target without
Speaker 1: any feedback of any kind. So they're not getting praised,
Speaker 1: they're not getting chewed out, nothing. Uh. And after staging
Speaker 1: a number of these, he found again what he suspected,
Speaker 1: that the people who were the closest on the first
Speaker 1: throw did worse on their second throw, and the people
Speaker 1: who were farthest away on their first throw tended to
Speaker 1: do better on the second throw. So what kandiment is
Speaker 1: actually demonstrating here is something that doesn't really have anything
Speaker 1: to do with learning or reinforcement, or really skills or
Speaker 1: even human psychology. Instead, this demonstration is showing the effects
Speaker 1: of chance, luck, and statistics. What he was showing is
Speaker 1: the subject we're talking about today, regression to the mean. Uh.
Speaker 1: You'll you'll see that phrase a lot in in scientific
Speaker 1: literature and in statistics. But if it helps to put
Speaker 1: it in more everyday terms, anytime you see regression to
Speaker 1: the mean, you can translate it in your head as
Speaker 1: trending toward the average, trending toward the average. So, to
Speaker 1: make the coin tossing illustration even clearer, imagine you throw
Speaker 1: the coin not twice, but that you throw the coin
Speaker 1: a hundred times. So you stand there throwing the coin
Speaker 1: a hundred times. And then let's say afterwards you average
Speaker 1: together the distance from the target across all a hundred throws,
Speaker 1: and you'll come up with some kind of average distance
Speaker 1: from target. Uh, just to make up a number for
Speaker 1: the sake of argument. Common doesn't give this. But let's
Speaker 1: say the average distance from the target across all your
Speaker 1: throws is nine centimeters. And remember that you're getting no
Speaker 1: feedback at all. Here, so it's unlikely that you will
Speaker 1: be getting much better as you go on. So even
Speaker 1: that the average distance from the target is nine cimeters,
Speaker 1: if you throw a coin once and it happens to
Speaker 1: be two centimeters from the targets really close, is your
Speaker 1: next throw likely to be about the same as that one,
Speaker 1: better or worse. Obviously, it is overwhelmingly likely that your
Speaker 1: next throw will be worse, just due to chance, probably
Speaker 1: closer to the average of nine cimeters away. And the
Speaker 1: same goes for throws that are really far off. If
Speaker 1: you throw something three hundred centimeters off, your next random toss,
Speaker 1: just by chance, is likely to be much better, much closer. So,
Speaker 1: simply put, most of the time, if you're sampling something
Speaker 1: in a series over time, if one sample produces an
Speaker 1: extreme value, the next one in the series is more
Speaker 1: likely to be closer to the average instead of extreme
Speaker 1: in the same way. In my experience, Uh, this is
Speaker 1: This is why it can sometimes be liberating to start
Speaker 1: off a game of bowling with just a disastrous gutter ball,
Speaker 1: because because I know that I'm good enough that that's
Speaker 1: probably not gonna happen twice in a row, but it's
Speaker 1: definitely going to happen at some point in the game
Speaker 1: because I'm not that good, you know, I like playing
Speaker 1: you know, once a year or even with less frequency
Speaker 1: these days. Oh yeah. And it's also like why I
Speaker 1: think a lot of us have intuitions that when you
Speaker 1: try something for the first time and you do really
Speaker 1: good on the first attempt, that makes you kind of
Speaker 1: nervous because you just know you're probably not gonna live
Speaker 1: up to that repeatedly. Yeah, like if you get you
Speaker 1: get a strike that first time, then that that first
Speaker 1: um what is it round? I can't even remember. This
Speaker 1: is how frequently I bowl. Um, the first role. So
Speaker 1: the first role first, the first column, you know. So
Speaker 1: the tendency of regression to the mean or or trending
Speaker 1: towards the average is pretty obvious when you're dealing with
Speaker 1: something like lots of random coin tosses with no feedback,
Speaker 1: But it becomes much more obscure when you're dealing with,
Speaker 1: say a more a more limited numbers of outcomes. In
Speaker 1: the series you're looking at and introducing possibly influential variables
Speaker 1: like pilot skill and instructor feedback. After all, we would
Speaker 1: expect that some variables having to do with instructor feedback
Speaker 1: should have an effect on pilot skill, right, That's the
Speaker 1: point of teaching is to have an effect over time.
Speaker 1: And after all, in this one scenario, the Konomen describes
Speaker 1: the the instructor believed that his verbal abuse of the
Speaker 1: students was so motivating that it made them instantly better
Speaker 1: on the stick. And you can't necessarily rule that out,
Speaker 1: but it's unlikely. I think. I'm convinced that regression to
Speaker 1: the mean could more easily explain this flight instructor's belief
Speaker 1: that screaming at pilots for screw ups made them better
Speaker 1: at planes, because, again, on average, even in the absence
Speaker 1: of any feedback at all, if a pilot in training
Speaker 1: executes a maneuver perfectly, the random fluctuation from one execut
Speaker 1: usian to the next will tend to mean that their
Speaker 1: next attempt probably won't be as good as that really
Speaker 1: good when the last time. And likewise, if they make
Speaker 1: a major error totally botcha maneuver, they're more likely to
Speaker 1: do better the next time just by chance. Both of
Speaker 1: these tendencies are regression towards the mean. But then Conomon
Speaker 1: actually draws a really interesting observation about about about our
Speaker 1: psychology and about culture from this fact, so, to quote
Speaker 1: him directly, this was a joyous moment in which I
Speaker 1: understood an important truth about the world. Because we tend
Speaker 1: to reward others when they do well and punish them
Speaker 1: when they do badly, and because there is regression to
Speaker 1: the mean, it is part of the human condition that
Speaker 1: we are statistically punished for rewarding others and rewarded for
Speaker 1: punishing them. And that was one of those things that
Speaker 1: when I read it, I was just like, oh my god,
Speaker 1: that's so true. Um, yeah, yeah, And in this specific instance,
Speaker 1: it makes me think about the special fact of reversion
Speaker 1: to the mean, fallacies on motivating belief in the effectiveness
Speaker 1: of of not just screaming at pilots in this one case,
Speaker 1: but all kinds of punishment behaviors, for example, corporal punishment.
Speaker 1: Thankfully you hear this less often these days, but I
Speaker 1: remember when I was younger, I used to hear people
Speaker 1: who would defend the parental practice of spanking children by saying,
Speaker 1: you know, I don't I don't care what the site
Speaker 1: scientists say. I don't care what the research says. I
Speaker 1: know from experience that it works to the extent that
Speaker 1: comments like this were based on any real experience and
Speaker 1: observation and not just sort of a free form, self
Speaker 1: justifying statement that had nothing to do with experience. I
Speaker 1: bet a lot of it was fallacious inference of causation
Speaker 1: actually based on regression to the mean, just like in
Speaker 1: this commument example. But anyway, I thought it would be
Speaker 1: interesting to talk a bit more about regression to the
Speaker 1: mean today because it's one of those things that, again,
Speaker 1: once you see it, it's it's pretty simple, it's actually
Speaker 1: actually pretty clear. But understanding it can help you have
Speaker 1: a better sense of how good science works and help
Speaker 1: keep you from drawing hasty inferences in everyday life. Yeah,
Speaker 1: because it is. It is interesting how kind of an
Speaker 1: insidious the results can be, the idea that that again,
Speaker 1: praise is ultimately punished because there's going to be a
Speaker 1: regression to the mean, to to to to the mean,
Speaker 1: and then likewise there can be this illusion, uh that
Speaker 1: uh that's screaming at pilots and so forth is going
Speaker 1: to be the successful way to go about things. Um So, yeah,
Speaker 1: this is I think this is an important episode to
Speaker 1: cover because it's the kind of thing that it's the
Speaker 1: kind of tool you kind of need tucked in your
Speaker 1: back pocket, even if you're just doing something like like
Speaker 1: scanning science headlines on a you know, a news server
Speaker 1: or social media message board. Yeah, because of course, understanding
Speaker 1: regression to the mean is extremely important in what scientists
Speaker 1: do when they design good experiments. If you don't take
Speaker 1: into account regression to the mean, you can incorrectly believe
Speaker 1: you have discovered some kind of tiger repellent or something. Uh.
Speaker 1: This concern plays a huge role in the history of medicine.
Speaker 1: It's part of the design of good medical research, or
Speaker 1: really any field that seeks to find remedies for problems.
Speaker 1: So consider a very basic hypothetical, uh patent medicine, say
Speaker 1: from a hundred years ago. So you know, you have
Speaker 1: you have a foot pain that you've never really had before. Uh.
Speaker 1: You know, you want it to go away. So you
Speaker 1: go to the store and you buy a bottle of
Speaker 1: Doctor Field Grades No Fail Pantasy for tumors, ulcers, cramps,
Speaker 1: and rooms, and you you pull the cork out, you
Speaker 1: chug it, and then the next day your foot feels better.
Speaker 1: Now you can conclude from this that the Doctor Field
Speaker 1: Grades cured you. But how do you know actually that
Speaker 1: the feelings in your foot didn't just regress to the mean,
Speaker 1: because the average is a low amount or no amount
Speaker 1: of foot pain. And if you don't have a medication
Speaker 1: that's tested with control groups and and randomized allocation into
Speaker 1: the groups, then how do you know that that the
Speaker 1: medicine actually did anything at all? Yeah? Yeah, So many
Speaker 1: of the examples you see for this and the applications,
Speaker 1: you're dealing with some sort of situation in the world
Speaker 1: where there is fluctuation and or change happening, often separately
Speaker 1: from whatever is being tested. So in this case, yeah,
Speaker 1: the Doctor Field Greats could have just been like just water,
Speaker 1: It just just you know, but there is the illusion
Speaker 1: that it worked because things got better. But if you
Speaker 1: don't have a control group and to you know, to
Speaker 1: drive home what that is, that would be like if
Speaker 1: you had a had like three different groups and a
Speaker 1: study of Doctor Field Greats elixir. Here, one group was
Speaker 1: taking Doctor Feel Greats elixer, another group was taking I
Speaker 1: don't know, let's say a half dose of Feel Grade
Speaker 1: or maybe a competitor's tonic. And then one group, the
Speaker 1: control group was taking nothing was or was taking you know,
Speaker 1: just water or something to that effect something completely innate. Uh.
Speaker 1: And that would be that would be a a group
Speaker 1: that you would judge the results of the other categories by, right,
Speaker 1: And you would need to randomly sort the people into
Speaker 1: those groups. So it wasn't just that, you know, the
Speaker 1: only the people with real severe foot pain. We're taking
Speaker 1: the doctor field grades, because the more extreme their pain
Speaker 1: to begin with, probably the more likely they are to
Speaker 1: have that pain be lessened or go away over time,
Speaker 1: just naturally. Right. And uh. And I'm going to have
Speaker 1: a more specific example of this a little later in
Speaker 1: the podcast. So if you if you still don't get it,
Speaker 1: just hang on. We'll we'll have another example in a bit.
Speaker 1: I was looking at an article in the British Medical
Speaker 1: Journal from nine that was just a collection of different
Speaker 1: examples of regression to the mean in real life medical research.
Speaker 1: This was by j Martin Bland and Douglas J. Altman
Speaker 1: called statistics notes some examples of regression towards the mean,
Speaker 1: and they point out a very common type of example,
Speaker 1: so the this will be similar to what we just
Speaker 1: talked about. The author's right. In clinical practice, there are
Speaker 1: many measurements such as weight, serum cholesterol concentration, or blood pressure,
Speaker 1: for which particularly high or low values are signs of
Speaker 1: underlying disease or risk factors for disease. People with extreme
Speaker 1: values of the measurements, such as high blood pressure may
Speaker 1: be treated to bring their values closer to the mean.
Speaker 1: If they are measured again, we will observe that the
Speaker 1: mean of the extreme group is now closer to the
Speaker 1: mean of the whole population. That is, it is reduced.
Speaker 1: This should not be interpreted as showing the effect of
Speaker 1: the treatment. Even if subjects are not treated, the mean
Speaker 1: blood pressure will go down owing to regression towards the means.
Speaker 1: So again, something starts with an extreme value in certain
Speaker 1: types of cases, you would just expect it to have
Speaker 1: a less extreme value the next time due to random fluctuation.
Speaker 1: Uh So again, you know this could fill you with
Speaker 1: despair because you might wonder, well, then how could you
Speaker 1: ever know if a treatment was effective or not. But again,
Speaker 1: this is where the standard practices of science based medicine
Speaker 1: come to play. Instead of just taking people with some
Speaker 1: extreme measurement and giving them a treatment, you randomize them
Speaker 1: into test groups and control groups like we were just
Speaker 1: talking about. So if you have a large enough sample,
Speaker 1: you properly randomize the groups. People with the extreme starting
Speaker 1: conditions will somewhat regress toward the mean, but they will
Speaker 1: all regress toward the mean on average the same rate
Speaker 1: whether they're receiving a real potential treatment or they're in
Speaker 1: the placebo group. But if the treatment actually does something helpful,
Speaker 1: this effect will manifest as the difference between the two groups.
Speaker 1: So good scientific research good medical research has methods for
Speaker 1: excluding the effects of reversion to the mean on their findings.
Speaker 1: We have the tools, but we can still fall into
Speaker 1: the trap of regression to the mean fallacies, especially in
Speaker 1: our day to day lives drawing inferences the way that
Speaker 1: that the pilot and in common story did, or or
Speaker 1: even science if we're not careful and deliberate about designing experiments.
Speaker 1: And in addition to just a methodology design that has
Speaker 1: you know, randomized groups and control groups, there are also
Speaker 1: ways of trying to counteract regression to the mean, just
Speaker 1: through statistical methods that are maybe less reliable, But there
Speaker 1: are statistical methods people can use to try to apply
Speaker 1: sort of modifiers to data in order to estimate regression
Speaker 1: to the mean and UH and counteract its effects. So
Speaker 1: again we have tools within scientific research to to figure
Speaker 1: this out, and it's a lot of what science does
Speaker 1: is trying to sort out the difference between regression to
Speaker 1: the mean and actual effects of interventions. But in our
Speaker 1: day to day lives, we still fall for regression to
Speaker 1: the mean fallacies all the time. Yeah, and it's important
Speaker 1: to realize too that it's not just a situation where
Speaker 1: regression towards the mean could create an illusion of something
Speaker 1: working when it doesn't. Uh, you know, sometimes it can
Speaker 1: just potentially overstate um the effects something. For an example
Speaker 1: of that that I was looking at was that regression
Speaker 1: towards the mean or the failure to account for it
Speaker 1: can also overstate the effectiveness of something like traffic light cameras.
Speaker 1: Is it making a difference and cutting down on accidents perhaps,
Speaker 1: but any actual effectiveness could potentially be overstated by failure
Speaker 1: to account for just regression towards the mean. Oh yeah,
Speaker 1: So where do you tend to install things like that?
Speaker 1: High acts like problem areas? Right, So, if there's like
Speaker 1: a stretch of road that has a lot of problems
Speaker 1: on it, people really speeding a lot there or crashing
Speaker 1: a lot there, that might be where you stage the intervention.
Speaker 1: It's possible some things like that fluctuate naturally over time
Speaker 1: in different locations. Yeah, and you put the cameras in place,
Speaker 1: and it could have an effect, but maybe not as
Speaker 1: much of an effect as it looks like it is
Speaker 1: taking place. Again, if you don't factor regression towards the
Speaker 1: mean into the study. Right now, While our TM is
Speaker 1: a very important phenomenon to understand and take into account,
Speaker 1: it certainly doesn't apply to every sequence of values you
Speaker 1: could repeatedly sample, So you also have to be careful
Speaker 1: not to apply it in situations where it isn't warranted.
Speaker 1: I was you know, there are a million examples you
Speaker 1: could cite. One that came to my mind is the
Speaker 1: orbital decay of a satellite. Let's say you've got a
Speaker 1: communication satellite in lower orbit and you get a reading
Speaker 1: on its altitude and the reading is lower than the
Speaker 1: satellites average altitude. Uh. Now, you might say, hey, I
Speaker 1: think this means we need to program a reboost to
Speaker 1: insert it back into the orbit where it's supposed to be.
Speaker 1: And somebody could erroneously apply regression to the mean here
Speaker 1: and say, no, we don't need to do that. The
Speaker 1: satellite might just return to its average altitude. It doesn't
Speaker 1: apply in this scenario, even though you are taking repeated
Speaker 1: measurements of a value over time, because we know things
Speaker 1: about the physical characteristics determining the orbit of satellites and
Speaker 1: in lower thorbit uh and that due to factors like
Speaker 1: atmospheric drag, their altitude tends to trend steadily downward over
Speaker 1: time in a consistent direction down, down, down, So eventually
Speaker 1: you will need a reboost in order to put it
Speaker 1: back up to the correct distance. So regression to the
Speaker 1: mean applies to certain kinds of data that are repeatedly
Speaker 1: sampled data where there is natural random fluctuation back and forth,
Speaker 1: not a steady trend in the data in one direction
Speaker 1: on the relevant time scale. The other thing that's important
Speaker 1: to understand is that systems where you expect to find
Speaker 1: regression to the mean are systems in which the repeated
Speaker 1: data values you're sampling are to some degree determined by
Speaker 1: luck or chance. If a series of values is influenced
Speaker 1: almost entirely by deterministic influence, like in the satellite example,
Speaker 1: by like the laws of physics, or by some extremely
Speaker 1: reliable skill with little room for variation, values don't really
Speaker 1: regress towards the mean in the same way, because there's
Speaker 1: just less random fluctuation back and forth to begin with.
Speaker 1: The more chance and random variation plays a role in
Speaker 1: the outcome, the more you will tend to observe regression
Speaker 1: towards the mean after an extreme sample in in whatever
Speaker 1: it is you're looking at. Yeah, I've read that the
Speaker 1: progression towards the mean is is not to be confused
Speaker 1: with the law of large numbers. For example, Uh, this
Speaker 1: is the the law that that states as a sample
Speaker 1: size becomes larger, the sample mean gets closer to the
Speaker 1: expected value. So a coin flipping example is key here.
Speaker 1: Flip a coin and the random results are going to
Speaker 1: ultimately average out to a point five proportion. But if
Speaker 1: you only flip the coin ten times, you might not
Speaker 1: see this breakdown. Um. And this also applies to say,
Speaker 1: even odds on the rolling of a of a D
Speaker 1: six of a six sided die. Uh So for example,
Speaker 1: too regular people, that's just to die. That is nerves
Speaker 1: like us, it's a D six. Yeah, D six is
Speaker 1: what I could get my hands on because I was like, well,
Speaker 1: I'm gonna do an example. I'm gonna try it myself.
Speaker 1: So while I was putting together notes for this, I
Speaker 1: went ahead and rolled ten times, and I got even
Speaker 1: even odd even odd even even even even odd. So
Speaker 1: that's that's seven to three in favor of even. So
Speaker 1: it might make you wonder, well, is this die broken?
Speaker 1: Does this D six need to go away? Because it
Speaker 1: can't be trusted to roll? Uh? You know a balanced
Speaker 1: array of odd and even numbers. Well, no, that's not
Speaker 1: the case. Uh. And if I were to roll this,
Speaker 1: say another hundred times, another thousand times, I would see
Speaker 1: things even out even more to where we would see this, uh,
Speaker 1: this point five proportion of odd versus even. Right. So
Speaker 1: these are not exactly the same thing, regression to the
Speaker 1: mean and the law of large numbers, but they are
Speaker 1: closely related. Both observations require you to think about statistical
Speaker 1: tendencies over time, over a time period of repeated sampling,
Speaker 1: and both are prim ust on the knowledge that repeated
Speaker 1: samples will tend towards the average. But regression to the
Speaker 1: mean has to do with the idea that if you
Speaker 1: start with an extreme observation and there is some role
Speaker 1: of chance or luck in determining the value of this observation,
Speaker 1: the next time you sample it, it's more likely to
Speaker 1: be closer to the average. The law of large numbers
Speaker 1: is that if in the real world, the more times
Speaker 1: you run something, the closer your outcomes in the real
Speaker 1: world will will be to the sort of perfect mathematical
Speaker 1: average that you would estimate just given the chances to
Speaker 1: begin with. Now, I want to come back to regression
Speaker 1: towards the mean in um in medical studies because I
Speaker 1: found a really interesting one that came out earlier this year.
Speaker 1: Uh So, a lot of a lot of the examples
Speaker 1: you find involving regression to the mean involved sports or economics,
Speaker 1: and I found this one discussed in a New York
Speaker 1: Times article again from earlier this year titled Intense strength
Speaker 1: training does not ease knee pain, study finds by Gina Colada.
Speaker 1: Uh this referring to a study published in JAMMA that
Speaker 1: entailed an eighteen month clinical trial involving three seventy seven participants. Okay,
Speaker 1: so the basic situation, the setup for this paper is
Speaker 1: that a lot of people have knee osteoarthritis and one
Speaker 1: of the go to treatment recommendations has long been strength training.
Speaker 1: So in this study they decided to look into it
Speaker 1: with three basic groups, one that received intense strength training,
Speaker 1: another that received moderate strength training, and another that received
Speaker 1: counseling on healthy living. So that third group, that's the
Speaker 1: control group, They did not have any amount of strength training,
Speaker 1: just uh, you know, some positive counseling about healthy living. Sure,
Speaker 1: so the researchers here apparently actually expected to see the
Speaker 1: intense strength training take the lead that they were looking
Speaker 1: to identify what has been just sort of accepted wisdom,
Speaker 1: um and and again that this has been the predominant
Speaker 1: treatment idea. But in instead they found that the results
Speaker 1: were the same for all three groups. Quote, everyone reported
Speaker 1: slightly less pain, including those who had received only counseling.
Speaker 1: Now why is that, Well, as Colotta points out, there's
Speaker 1: there's always room for other effects, especially say the placebo effect.
Speaker 1: Uh but regression to the mean is also a heavy
Speaker 1: consideration here and certainly could work in congress with the
Speaker 1: placebo effect. Right, So you don't necessarily have to assume
Speaker 1: that the counseling actually helped to heal people's knees, though
Speaker 1: it may have in in in some it may have
Speaker 1: had some kind of mechanistic effect in in some way
Speaker 1: a mind body kind of thing, But you would also
Speaker 1: just expect over time, people who have an extreme starting position,
Speaker 1: who are starting with a lot of knee pain, to
Speaker 1: get gradually better over time. Yeah. So a Colatta rights
Speaker 1: quote are the right, as symptoms tend to surge and subside,
Speaker 1: and people tend to seek out treatments when the pain
Speaker 1: is at its peak, when it declines, as it would
Speaker 1: have anyway, they ascribed the improvement to the treatment. Uh.
Speaker 1: So you know, this would this would roughly equate to
Speaker 1: yelling at your knee when it's in pain, and it
Speaker 1: really makes it certainly relates to many other health scenarios
Speaker 1: as well, various medications and even things like prayer and
Speaker 1: you know, supernatural um treatments and attempts to to deal
Speaker 1: with pain, etcetera. Yeah, I mean it could apply to
Speaker 1: to any intervention that is aimed at influencing something that
Speaker 1: is naturally variable on its own, right. Yeah, and you
Speaker 1: know something that's again any kind of system in which
Speaker 1: change occurs, when fluctuation occurs. Uh, you know, you can
Speaker 1: you can see this applying to not only physical pain,
Speaker 1: but also uh, emotional distress things of that nature. You know.
Speaker 1: So again, I think this is an important tool to
Speaker 1: have in our our logic tool kit. Thank thank Now.
Speaker 1: There are even cases where I'm tempted to think about
Speaker 1: the application of regression to the mean, but where it's
Speaker 1: probably a lot harder to quantify exactly what the effects are.
Speaker 1: It's cases where it can be difficult to separate out,
Speaker 1: say the effects of some kind of deterministic influence like
Speaker 1: skill versus how how strong the effect of chance or
Speaker 1: luck is. But I think about things even in the
Speaker 1: world of the arts, like I think about, you know,
Speaker 1: the sophomore album by by a band that has like
Speaker 1: a really stellar debut album. Uh, you know, often that
Speaker 1: is perceived is disappointing, and you have to wonder, like, Okay,
Speaker 1: is it is that often true? Because I don't know
Speaker 1: if people get famous and it goes to their heads
Speaker 1: and then they you know, they get full of themselves
Speaker 1: and make something dumb, or is it because when somebody
Speaker 1: has a debut album that's really well received, to some extent,
Speaker 1: it's so good partially because of luck or chance, and
Speaker 1: that's an outlier that you're as you're starting sample, yeah, yeah,
Speaker 1: And certainly this is an area that's there's a lot
Speaker 1: more subjectivity here and and so it's not the kind
Speaker 1: of thing you can that's really have a control group
Speaker 1: for anything. But but I think it is quite interesting,
Speaker 1: and I did find as I was looking around for
Speaker 1: some jazzy or examples or possible examples of aggression to
Speaker 1: the mean. Um, I found one that that actually gets
Speaker 1: into a little bit into the idea of you know,
Speaker 1: first and second album. But also uh, the idea of
Speaker 1: follow up films and Hollywood sequels has pointed out both
Speaker 1: good Yeah has pointed out by Joanna Deong in two
Speaker 1: thousand eighteen on the blogs scientifically sound movie sequels are
Speaker 1: potentially a great example of aggression to the mean. Quote,
Speaker 1: Hollywood sequels are only made if the original film is
Speaker 1: a quote unquote high quality success. But the average quality
Speaker 1: of sequels will be closer to the mean than average
Speaker 1: quality of originals of sequels because of regression to the means,
Speaker 1: So sequels tend to be of lower quality than the original. Now,
Speaker 1: I might somewhat dispute the premise here that Hollywood sequels
Speaker 1: are only made to films that are high quality to
Speaker 1: begin with. Um, right, But but I still think this
Speaker 1: is onto something because there is a movie that gets
Speaker 1: a sequel tends to have something about it something that
Speaker 1: people are responding to, whether it's a movie that I
Speaker 1: would like or not. Right, I mean, sometimes obviously the
Speaker 1: situation is the film just made a lot of money.
Speaker 1: I mean, I guess that's the key thing. It didn't
Speaker 1: make a lot of money. If so, producers are going
Speaker 1: to be more inclined to say, let's do that again,
Speaker 1: Let's have that experience again of all that money coming in.
Speaker 1: And sometimes this this certainly matches up with a quality film.
Speaker 1: You have something that really captures people's imagination and is
Speaker 1: of high quality. And uh and you know, so it's
Speaker 1: really firing on all cylinders. But you know, and yes,
Speaker 1: certainly in some cases it's just the right film at
Speaker 1: the right time. Or or maybe it has nothing to
Speaker 1: do with the film itself. Maybe it's who's in it,
Speaker 1: or I don't know what's going on in the zeitgeist
Speaker 1: during that particular era. Well, the way I would think
Speaker 1: about this is, and I think that again, this is
Speaker 1: onto something. It highlights that when we experience confusion where
Speaker 1: we say, like, wow, you know, the Exorcist is such
Speaker 1: a great horror movie and the Exorcist too is so bad?
Speaker 1: How could that be the case? You know, why is it.
Speaker 1: Why is such a bad sequel to such a great movie.
Speaker 1: It's because of the comparison of the original to the
Speaker 1: sequel that we're experiencing this confusion. Another way you could
Speaker 1: just look at it is most horror movies are direc
Speaker 1: most movies are bad, and it is only by comparing
Speaker 1: the The Exorcist Too to The Exorcist that you notice
Speaker 1: this steep drop off where Another way of looking at
Speaker 1: it is that The Exorcist Too is bad like most
Speaker 1: horror movies are, and the first one was an outlier
Speaker 1: at the beginning. It was a first film in a
Speaker 1: series that happened to be really good to cut above. Yeah, absolutely, like, yeah,
Speaker 1: I think this is the correct way to look at it,
Speaker 1: and also keeping in mind it just how amazing it
Speaker 1: is that any film gets completed, like even bad film,
Speaker 1: Like a lot of people probably work pretty hard to
Speaker 1: make that happen, even if the end results don't really
Speaker 1: please anyone at all. But but yeah, I think this
Speaker 1: is also an interesting inversion of the opening example of
Speaker 1: yelling at pilots as well, because most of the time,
Speaker 1: if a flawed movie comes out, people are not clamoring
Speaker 1: for the sequel. Um sequels are rarely guaranteed, so you're
Speaker 1: not often going to hear things like, oh, well that
Speaker 1: wasn't great. I hope the next one is an improvement.
Speaker 1: I mean some people say that, some people I've said
Speaker 1: things like that before, where it will be like, oh,
Speaker 1: really flawed film, but maybe there's like a cool idea.
Speaker 1: I kind of wish it would they would remake it,
Speaker 1: even though there's no like logical reason that there would
Speaker 1: be like a there would be money behind that idea. Well,
Speaker 1: I guess it's kind of different when you're talking about
Speaker 1: a one off creative project versus something. I mean, we
Speaker 1: live in a kind of different era now because we
Speaker 1: were at the height of this you know, cinematic universe
Speaker 1: thing with a huge number of the big budget movies
Speaker 1: that come out, the big event movies are not one
Speaker 1: off creative products, but they are a product that exists
Speaker 1: within some kind of franchise or universe or something. So
Speaker 1: you just know automatically that there's gonna be another one,
Speaker 1: whether this one is good or not. Yeah, like either
Speaker 1: it's an established film universe where like, you know, they
Speaker 1: put out another Marvel movie and it's just terrible. Well,
Speaker 1: obviously there's enough momentum, they're not going to stop. They're
Speaker 1: not gonna be like, oh, well, lesson learned, Well we'll
Speaker 1: stop them. No, no, there's gonna be another. Another example
Speaker 1: of this might be a successful franchise in another medium,
Speaker 1: say a book series, so like the Harry Potter books
Speaker 1: for example, or I don't know, Lord of the Rings,
Speaker 1: where you know that once they make the Fellowship of
Speaker 1: the Rings, there's going to be a follow up, They're
Speaker 1: gonna do another one. So in these ways, unless it's
Speaker 1: the seventies and it's uh, that Lord of the Rings
Speaker 1: movie that that ends with Helm's Deep, well, but they
Speaker 1: picked that up eventually, but yeah, okay, but but yeah,
Speaker 1: probably the Harry Hotter films are a better example. And
Speaker 1: there may be spe specific you know things about how
Speaker 1: that wasn't guaranteed either, uh, you know, the economic reality
Speaker 1: can always come into play. But for the most part,
Speaker 1: like those were when when that started, you knew they
Speaker 1: were going to keep making these at least they were
Speaker 1: going to make a follow up, so you could have
Speaker 1: comments like, well, there that was this was kind of
Speaker 1: flawed in some of the some of its execution. I
Speaker 1: hope that they fixed that in the next film for
Speaker 1: the most part. Yeah, with one offs, this is not
Speaker 1: the case. It's like, if if this film fizzles, then
Speaker 1: only you know, a few, like rare people are going
Speaker 1: to be clamoring for a sequel or dreaming about what
Speaker 1: the sequel would be. Yeah. I think this observation but
Speaker 1: regression to the mean and movie sequels is actually very
Speaker 1: on point, but more so for the films of yester year,
Speaker 1: where the more the more common thing was you'd have
Speaker 1: a an independent sort of creative product that it's its
Speaker 1: own thing, and then if it resonated with somebody, if
Speaker 1: it did well, there would be sequels. I think it's
Speaker 1: a little it applies a little bit less today when
Speaker 1: there's just you know, we're in the world of France,
Speaker 1: Chises and extended universes and there's just sort of like
Speaker 1: a guaranteed ongoing uh conveyor belt of of new stuff
Speaker 1: within the Marvel world or the Star Wars world or whatever. Yeah,
Speaker 1: but I think it it is a worthwhile way to
Speaker 1: think about creative the creative process, and you know, as
Speaker 1: opposed to some of these alternate sort of folk wisdomy
Speaker 1: ways of thinking about it. For example, on Weird House, cinema.
Speaker 1: We recently talked about Toby Hooper. Toby Hooper is one
Speaker 1: of those directors who's often you'll often you'll see descriptions.
Speaker 1: I think we've even read part of a review where
Speaker 1: they they really they talk about, oh, well, you know
Speaker 1: he put out Texas Chainsaw Mascre directed that film and
Speaker 1: this was great. It was, you know, just a real
Speaker 1: lightning bolt um to the cinematic world into horror itself
Speaker 1: as a genre. And then the idea that well, he
Speaker 1: was never able to capture that magic again, you know,
Speaker 1: that is his career was just like one long slide
Speaker 1: after that, which I don't think it is a fair assessment,
Speaker 1: especially if you employ regression to the mean you the
Speaker 1: idea being that, yeah, he did kind of get lightning
Speaker 1: in a bottle with that with that first big film,
Speaker 1: that that he was able to to really bring something
Speaker 1: together that is an outlier, um, but that that that's
Speaker 1: just going to happen. That's just the way these things work, right,
Speaker 1: So most movies aren't that good. So you know, the
Speaker 1: random chance of like how good his ideas and execution
Speaker 1: are from one year to the next is going to
Speaker 1: set in and you might have a different idea about
Speaker 1: his career. If you were to say, like randomly chronologically
Speaker 1: reorder all his movies, right, you know, like if you
Speaker 1: were to put the worst ones earlier on or something,
Speaker 1: people might feel differently about it. Yeah, well then they
Speaker 1: would talk about, well, okay, TCM was peak Toby Hooper,
Speaker 1: like this was his peak output. Because this is the
Speaker 1: kind of the kind of view of an artist's you know,
Speaker 1: creative trajectory that we tend to want to um to
Speaker 1: follow along, you know, because it's more story shaped, the
Speaker 1: idea of ascent and then eventually decent that there's gonna
Speaker 1: be h is going to be a period of high
Speaker 1: noon in their creative out output, and sometimes that does
Speaker 1: match up with the reality. But I don't know even
Speaker 1: then we I think we tend to overlook the dogs
Speaker 1: in the filmographies of people we love, you know. Oh
Speaker 1: yeah uh. But then again, I mean, this is interesting
Speaker 1: because in talking about regression to the mean applying to
Speaker 1: creative products like movies, we are acknowledging that the creative
Speaker 1: process is not purely a product of talent and skill,
Speaker 1: that there is a significant amount of chance and luck
Speaker 1: involved in something like how good a movie turns out?
Speaker 1: To be um, and it's hard to know exactly how
Speaker 1: to like how to picture that influence of chance and luck.
Speaker 1: You know, like, what what is that in the creative process.
Speaker 1: It's obviously true because there are people who can be
Speaker 1: incredibly skilled in one instance and then I don't know,
Speaker 1: things just don't go right the next time, and to
Speaker 1: make something that nobody really likes. But uh, but that's
Speaker 1: that's just not often how people like to think about
Speaker 1: creative talents and people like to think about the creative
Speaker 1: process like it is much more strictly deterministic. Yeah, yeah,
Speaker 1: Or or you look at things like the Star Wars
Speaker 1: films and you kind of like fall into this idea
Speaker 1: of thinking, this is stuff that is mind out of
Speaker 1: the mythic earth, and then you know, it just makes
Speaker 1: sense that things would accumulate and get better. So um,
Speaker 1: but really looking back on it, especially if you actually
Speaker 1: like watch documentaries, and there's some great ones about the
Speaker 1: production of those films, like it's it's amazing that Star Wars,
Speaker 1: the first one in New Hope was as good as
Speaker 1: it was, and then it's nothing short of I mean,
Speaker 1: it's it's just a pure miracle that the second one
Speaker 1: was so much better and like really nailed it. Like
Speaker 1: if if the second film had had floundered, I mean,
Speaker 1: just imagine how different the cinematical landscape would have been
Speaker 1: for decades to come. Yeah, So it's it's amazing if
Speaker 1: the first film in a series is good, and it's
Speaker 1: super amazing if the second one is good. And and
Speaker 1: this is why I think we often find too that
Speaker 1: if if part one in part two, if something are
Speaker 1: of high quality, then you've got to look out for
Speaker 1: that part three because that part three, that part three
Speaker 1: may be coming to get you. But likewise, if a
Speaker 1: part two is rubbish, um, you know, subjectively, then then
Speaker 1: part three might pick it up and uh and get
Speaker 1: things back on track. So you certainly see that that
Speaker 1: kind of fluctuation as well. I have a question I
Speaker 1: actually don't know the answer to, but this would be
Speaker 1: interesting in terms of I don't know the high performing output,
Speaker 1: whether that is in whether that is a creative endeavor
Speaker 1: like you know, writing books or creating movies, or whether
Speaker 1: that's something even like athletics, like athletic performance, do you
Speaker 1: expect to see more random fluctuation in the performance of
Speaker 1: collaborative output versus individual output? So say, um, do you
Speaker 1: expect more influence of random chance and fluctuation in the
Speaker 1: quality of uh books written by a single author versus
Speaker 1: you know, movies that have the input of hundreds of
Speaker 1: thousands of people? Uh? Or in in the realm of
Speaker 1: say sports, like, do you expect more random variation in
Speaker 1: the output of an individual athletes like you know, an
Speaker 1: individual gymnast or something, or in team sports? Yeah? I
Speaker 1: can see it going both ways, because yeah, if you
Speaker 1: think too hard to about even just like the film analogy,
Speaker 1: you can easily get into discussions of like, Okay, well
Speaker 1: was it the same cast and crew that are producing
Speaker 1: the sequel? Uh? You know, what happens when the budget
Speaker 1: is different, what happens when there are other constraints, what
Speaker 1: happens when suddenly there are a whole bunch of producers
Speaker 1: that have their ideas about what things should be. I mean,
Speaker 1: there's so many different factors to take into place. Uh.
Speaker 1: You know, with this example that you know, perhaps doesn't
Speaker 1: bear too close of scrutiny, but but but it's but
Speaker 1: it's still I think serves as a nice um illustration
Speaker 1: of the overall trend that we're talking about here. Well,
Speaker 1: it does bring up the fact that since I mentioned
Speaker 1: athletes that, you know, I don't know a lot about sports.
Speaker 1: I'm not a big sports fan. But but clearly, but
Speaker 1: regression to the mean is something that has widely been
Speaker 1: applied to the world of sports. Uh for example, in
Speaker 1: the observation that often after having a really stellar season,
Speaker 1: either an individual athlete or a sports team will be
Speaker 1: perceived to underperform the next season. And again that very
Speaker 1: well could have something to do with regression to the mean. Like,
Speaker 1: you know, the fact that they're observed having an amazing
Speaker 1: season is actually an outlier. You're starting your expectations then
Speaker 1: and saying like, Okay, now they're going to be the
Speaker 1: best forever. Just by random fluctuation over time, you would
Speaker 1: expect their next season to probably be not as good
Speaker 1: as the first. I wonder to what an extent this
Speaker 1: can be applied to, say, the world of the culinary arts,
Speaker 1: or even just like various food crops, like say the
Speaker 1: selecting a cantalope at the grocery store, that sort of thing.
Speaker 1: I mean, I guess it would apply to pretty much
Speaker 1: anything where you're sampling in a series over time. There's
Speaker 1: plenty of random fluctuation in what you're sampling and the
Speaker 1: first thing you sample is an outlier in some way
Speaker 1: really good or really bad. If those things true, then
Speaker 1: you can probably expect you're going to see some regression
Speaker 1: one way or the other. Yeah. Yeah. By the way,
Speaker 1: I was looking around for like really stellar examples of
Speaker 1: a sequel film that is widely believed to be uh rubbish,
Speaker 1: and I think The Exorcist Too is the primary example.
Speaker 1: Like you get into some of the other examples that
Speaker 1: pop up, I feel like there's room for disagreement. Um.
Speaker 1: For instance, Texas Chainsaw Masker two is one which I
Speaker 1: saw popping up on some of these lists for disappointing sequels.
Speaker 1: But I think that's entirely based on who you ask.
Speaker 1: I think if you ask us, we will agree that
Speaker 1: that that t c M two is is actually a
Speaker 1: great film. It's different from the first one perhaps if
Speaker 1: you go into if you go into part two with
Speaker 1: the expectations you had for part one, you may see
Speaker 1: it as a dip in quality. But depending on what
Speaker 1: else you're bringing to the table, you might see it
Speaker 1: as an increase in in quality or at least or
Speaker 1: something that maybe is different but on par with the original.
Speaker 1: I mean it's certainly not for everybody. I mean, it
Speaker 1: is a it is a gross, disgusting film in in
Speaker 1: a way like the first one, probably even grosser, but
Speaker 1: also a sort of satirical masterpiece. Um. But I just
Speaker 1: had another thought when you said that The Exorcist Too
Speaker 1: is regarded as like one of the best examples of
Speaker 1: a sequel. That's really rubbish. I mean, it makes me
Speaker 1: also wonder about the pretty high estimation critics generally have
Speaker 1: of The Exorcist three. Makes me wonder if the effect
Speaker 1: of the Exorcist to being so bad actually makes people
Speaker 1: sort of over you know, they're like they're ready to
Speaker 1: be impressed by the Exorcist three. Yeah. Yeah, I wonder
Speaker 1: if that's the case too with it with especially when
Speaker 1: you have a situation with a part three coming back
Speaker 1: and restoring uh some I don't know, some level of
Speaker 1: quality to a franchise. I mean there's also like the
Speaker 1: Star Trek h example, right, I mean that was long
Speaker 1: the Long held up as an example of like, okay,
Speaker 1: you have you even Star Treks and your odd Star Treks, right. Uh.
Speaker 1: And I think you've made a similar case for the
Speaker 1: Faster and Furious movies, right, I mean once you get
Speaker 1: to a certain point in the series, I think it's
Speaker 1: pretty much all uh, you know, a nitrous boosted brain.
Speaker 1: It's it gets you know, it's all like we're driving
Speaker 1: cars in space now and flying and all that. But um,
Speaker 1: but for the earlier ones, yeah, i'd say the odd
Speaker 1: ones are better, Like, uh, three is the first one
Speaker 1: where it really starts getting ludicrously weird. Four is kind
Speaker 1: of a uh, and then five starts. Five is when
Speaker 1: the rock shows up, and then but by seven year
Speaker 1: Golden All right, well, we're gonna go ahead and close
Speaker 1: this one out here. But we'd obviously love to hear
Speaker 1: from everyone about this about regression towards the mean, just
Speaker 1: in our daily lives, in various scientific studies. Perhaps you
Speaker 1: have thoughts about how this applies to something we've discussed
Speaker 1: on the show in the past, because I know we've
Speaker 1: we've mentioned regression to the mean in passing before, but
Speaker 1: certainly we've never taken the opportunity to really dive into
Speaker 1: it and explain it like we did today. Yeah, I
Speaker 1: know it's come up in passing, just in us making
Speaker 1: comments here and there about like the import of of
Speaker 1: randomized trials and control groups and all that. In the meantime,
Speaker 1: if you would like to listen to other episodes of
Speaker 1: Stuff to Blow Your Mind, you will find them wherever
Speaker 1: you get your podcast. Just look for the Stuff to
Speaker 1: Blow your Mind podcast feed. We have our core episodes
Speaker 1: on Tuesdays and Thursday's, Artifact episodes on Wednesday, listener mail
Speaker 1: on Monday's. On Fridays, we do a little bit of
Speaker 1: a weird house cinema. That's our times. We just talk
Speaker 1: about some sort of a strange film. Uh, and you know,
Speaker 1: teas apart what makes it strange? Uh, let's see what else. So, yeah,
Speaker 1: you go to Stuff to Blow your Mind dot com
Speaker 1: that will send you to the I heart listing for
Speaker 1: our show. And there's a button there for a store
Speaker 1: if you want to click on that. Wealth then you
Speaker 1: can buy some merchandise that has Stuff to Blow your
Speaker 1: Mind logos and whatnot on it, or perhaps Weird House
Speaker 1: Cinema logos and whatnot on it, and you can get mugs,
Speaker 1: t shirts, all that kind of stuff. Uh, just for fun,
Speaker 1: you know, or but if you want to be all
Speaker 1: business about it, you can make it work to get
Speaker 1: our merge. Now, now, this this is just for fun, Like,
Speaker 1: don't use these these clothing items to clothe your nakedness.
Speaker 1: These are just for fun. This is just purely extra clothing. Okay,
Speaker 1: this should not be your core clothing. Huge thanks as
Speaker 1: always to our excellent audio producer Seth Nicholas Johnson. If
Speaker 1: you would like to get in touch with us with
Speaker 1: feedback on this episode or any other, to suggest a
Speaker 1: topic for the future, just to say hello, you can
Speaker 1: email us at contact at stuff to Blow your Mind
Speaker 1: dot com. Stuff to Blow Your Mind is production of
Speaker 1: I Heart Radio. For more podcasts for my heart Radio,
Speaker 1: visit the iHeart Radio app, Apple Podcasts, or wherever you're
Speaker 1: listening to your favorite shows.