Speaker 1: Welcome to Stuff You Should Know, a production of five
Speaker 1: Heart Radios How Stuff Works. Hey, and welcome to the podcast.
Speaker 1: I'm Josh Clark, and there's Charles W. Chuck Bryant over there,
Speaker 1: and we've got the scoop. Jerry's around here somewhere, and
Speaker 1: this is Stuff you Should Know. Off to a great start.
Speaker 1: She's in hers, she is she's got this remote thing
Speaker 1: going on. Uh, it's like the COVID special, that's right. Uh.
Speaker 1: And this was this has been what I've been wanting
Speaker 1: to do since two thousand sixteen, and why it seems
Speaker 1: like the fire kind of went down on it and
Speaker 1: now it's the fires back up again. In election season,
Speaker 1: I thought no better time than to talk election polling
Speaker 1: and this weird sort of black magic which is really
Speaker 1: not black magic at all. And it was, um, the
Speaker 1: polling wasn't even really that off. No, it was great.
Speaker 1: There was a furious fear we'll talk about in a second,
Speaker 1: but there's a furious reaction by the media just left
Speaker 1: polling and polsters out to dry, saying like you you're terrible.
Speaker 1: Your whole craft is is useless. As the pollsters went
Speaker 1: back after election night on two thousand and sixteen, which,
Speaker 1: by the way, is a bit of a surprise to
Speaker 1: everybody involved. Um. Yeah, when the pollsters went back and
Speaker 1: looked at their stuff, they said, wait a minute, no,
Speaker 1: this is this is all fine. It was you guys, media,
Speaker 1: You screwed up. You don't know what polling is or
Speaker 1: what it does, or how to talk about it. Most importantly, Yeah,
Speaker 1: and then you public, you have no idea what's going on.
Speaker 1: You just see some percentages and you automatically leaked to
Speaker 1: some conclusions and this is way off. So it's impartant
Speaker 1: that the media was misrepresenting it. Some polls weren't very good.
Speaker 1: And then, um, the public in general just needs to
Speaker 1: be a bit more educated on statistics to understand what
Speaker 1: they're hearing. And that's what we're here for. Because I
Speaker 1: I took statistics three times in college, the same course
Speaker 1: at Georgia. At Georgia, I took one of those classes.
Speaker 1: I hated it, intro to statistics, right, yeah, boy, I
Speaker 1: hated the class. The professor. Finally I walked up to
Speaker 1: her on the last day of the third time, was
Speaker 1: like please, and she bumped my d up to a
Speaker 1: seat and I was I never looked back. She say
Speaker 1: you have a one in four chance, and you're like,
Speaker 1: but what does that mean? Right? What is four? But
Speaker 1: so if I can understand this after doing some research,
Speaker 1: then anybody can understand at least the gist of it
Speaker 1: enough to understand polling and not be taken in by
Speaker 1: bad representation of what poll results are. Yeah. So, if
Speaker 1: you remember, in there were polsters saying or I'm sorry
Speaker 1: and I'm gonna say that wrong over and over again.
Speaker 1: You had media saying that Hillary Clinton is going to
Speaker 1: win in a landslide. Um, she's got a chance to
Speaker 1: win some set as high is ninety five, She's gonna
Speaker 1: win the popular vote by three percentage points. Um, all
Speaker 1: the all the battleground states in the Midwest, Um, she's
Speaker 1: gonna win those narrowly. And it did not work out
Speaker 1: that way. And like you said, there was a furor
Speaker 1: over how could everyone be this wrong with the polling.
Speaker 1: And there's man named Nate Silver, who everyone probably knows
Speaker 1: at this point, who has made his name as a
Speaker 1: data specialist and runs the five thirty eight blog and said,
Speaker 1: you know what, um pulling is flawed. And that's probably
Speaker 1: the first thing that everyone should understand is all polling
Speaker 1: is a little bit flawed. Um, state polling is is
Speaker 1: definitely a little more flawed than national polling. But here's
Speaker 1: the deal, everybody, these polls from we're not only not
Speaker 1: so far off, but historically, dating back to since nineteen
Speaker 1: seventy two, they actually performed a little better than a
Speaker 1: lot of elections. Yeah, and the state polling, while worse
Speaker 1: than average, wasn't a far off from the average error rate.
Speaker 1: So what do you what do you want? So there's
Speaker 1: a lot of stuff, Like we said that there was
Speaker 1: a lot of post mortem that was done on the
Speaker 1: two thousand and sixteen polls and what what was gotten
Speaker 1: wrong and what was gotten right, And we'll talk about
Speaker 1: that later. But um, the point is is that overall
Speaker 1: it wasn't that far off. And so the the idea
Speaker 1: isn't that the polls failure, that there's something inherently flawed
Speaker 1: with polling, or that there's even something inherently wrong with
Speaker 1: the media. Like I want to go on record here,
Speaker 1: especially in this climate, the media is not our enemy.
Speaker 1: Like any healthy democracy needs a vital, robust, independent media
Speaker 1: is free from bias as an objective to reality and
Speaker 1: good injustice as possible. But there's also such a thing
Speaker 1: as a twenty four hour news cycle, and you've got
Speaker 1: to fill that. And that's given the rise of opinion
Speaker 1: news and pundits and um and basically trying to capture
Speaker 1: as much market share as possible, which is definitely the
Speaker 1: wrong track for media in general. But I just want
Speaker 1: to go on record, while we're gonna be kind of
Speaker 1: beating the media up a little bit, that does not
Speaker 1: mean that the media has inherently flawed or evil or
Speaker 1: or seeks to um to to kill you and your
Speaker 1: family and your family dog. So Silver goes back and
Speaker 1: a bunch of people go back and look at um
Speaker 1: history and kind of what went wrong here in as
Speaker 1: far as the polling goes. He says, you know what,
Speaker 1: we went back for the past twelve presidential cycles since
Speaker 1: nineteen seventy two, and he said the polling air was
Speaker 1: four point one. He said in that national polling air
Speaker 1: was three point one, so technically by a full point
Speaker 1: it was. It was a full point better. He said, Uh,
Speaker 1: we predicted that she would win the popular vote by
Speaker 1: three percentage points. She actually did win the popular vote
Speaker 1: by two percentage points. Um. The state polls were the
Speaker 1: real difference maker. They actually did underperform at a five
Speaker 1: point to error rate, and that doesn't sound like that much.
Speaker 1: I think the overall or rate for state poll since
Speaker 1: nineteen four point eight, so four point eight five point
Speaker 1: two doesn't sound like much. But if you're talking about
Speaker 1: a percentage of error and just a handful of swing states,
Speaker 1: that can make something look like a landslide even though
Speaker 1: you lose a popular vote. That's exactly what happened, right,
Speaker 1: That's exactly what happened, because you gotta remember Trump didn't
Speaker 1: win the popular vote, he won the electoral college, and
Speaker 1: it came down to those swing states. But the fact
Speaker 1: that they were off just by point four points from
Speaker 1: the average for the error rate um goes to show
Speaker 1: you just how close that race actually was, which again
Speaker 1: is the opposite of how it was being broadcast throughout
Speaker 1: the election. It was supposed to be a landslide, like
Speaker 1: Hillary Clinton might as well just be like taking measurements
Speaker 1: for curtains in the oval office right now, Like it
Speaker 1: was just that set So it was presented one way,
Speaker 1: when in reality, if you really looked at the polls
Speaker 1: and the polling results, if you looked at them with
Speaker 1: a sober face, it was a much closer race than
Speaker 1: it appeared or than it was being broadcast. I haven't
Speaker 1: had a sober face since that night. So, uh, we
Speaker 1: should talk about the margin of error. Um, in polling.
Speaker 1: Anytime you see a poll, it's you. They talk about
Speaker 1: the margin of air. It's usually plus plus or minus
Speaker 1: three or four, and that is on each side. So
Speaker 1: for each candidate's poll, uh. In other words, it could
Speaker 1: be a potential like seven to eight point swing and
Speaker 1: still be within that margin of error. So when Trump
Speaker 1: is winning States by a point two percent margin or
Speaker 1: a point five or a point seven percent margin, that's well,
Speaker 1: well well well within the margin of error, right right,
Speaker 1: So um, that margin of error, by the way, is
Speaker 1: just bill tim. We'll talk about it a little more
Speaker 1: and a little bit, but it's like there's just no
Speaker 1: way around it. If to to get around any margin
Speaker 1: of air, you would have to literally go through and
Speaker 1: interview every single voter in America and then compile the
Speaker 1: evidence or their their data perfectly without any miss keys
Speaker 1: or anything like that. And it's just impossible. So everyone
Speaker 1: accepts that any poll is going to have a margin error,
Speaker 1: but you want to keep it within plus or minus
Speaker 1: three points. Right be four. Yeah, So a little history
Speaker 1: of polling. Um, we've always been, um, pretty spell bound
Speaker 1: by poles in this country. We put a lot of
Speaker 1: stock and polls, especially the presidential race. Um. The word
Speaker 1: straw pole, if you've ever heard that, that comes from
Speaker 1: the idea that you hold up a piece of straw
Speaker 1: to see which way the wind is blowing. So a
Speaker 1: straw pole is kind of like, here's how things stand
Speaker 1: today on something like this is why the way the
Speaker 1: wind is blowing today on this matter. Yeah, and they're
Speaker 1: just kind of informal. They used to take them like
Speaker 1: on train cars, as journalists would ask people who they
Speaker 1: were on the train with, who they're going to vote for.
Speaker 1: Nothing like formal or anything, but it it was. It
Speaker 1: does kind of reveal how long standing our fascination with
Speaker 1: poles really really is. Yeah, it got pretty serious in
Speaker 1: the nineteen thirties, specifically the nineteen thirty six election, where
Speaker 1: a literary digest it was, it was a pretty big
Speaker 1: magazine at the time, pulled it's subscribers and it's just
Speaker 1: kind of funny even seeing this sence they predicted a
Speaker 1: landslide wind for Republican Republican alf landon over fd R.
Speaker 1: So if you've never heard of alf Landon, uh, you
Speaker 1: know I because alf Landon did not beat f DR. Uh.
Speaker 1: And the magazine's editor said, you know what, we didn't
Speaker 1: even think about the fact that we just pulled our
Speaker 1: subscribers and that they're wealthy people are at least wealthier
Speaker 1: on average, and they're probably going to vote Republicans. So um,
Speaker 1: alf Landon was their man. Right. So if you go
Speaker 1: out it's even today and just interview Republicans and say, hey,
Speaker 1: who you're going to vote for, and then take that
Speaker 1: results and apply it to the entire population of the nation,
Speaker 1: you've got a flawed pole. And that's what Literary digested.
Speaker 1: But in doing so, they established this kind of they
Speaker 1: pointed out a real design flaw that now is just
Speaker 1: one of the first basic things that anybody conducting a
Speaker 1: pole gets rid of. That's right. Gallup came on the scene.
Speaker 1: They galloped onto the scene. Uh so sorry, And they
Speaker 1: were one of the first big polling companies to say,
Speaker 1: all right, we gotta get this right. We gotta get
Speaker 1: a representation of all of America here. So we're gonna
Speaker 1: send our people door to door. We're gonna go to
Speaker 1: every zip code in America, and they did that from
Speaker 1: n to nineteen four UH and got basically within about
Speaker 1: three percentage points, doing a pretty good job. UM, but
Speaker 1: it was really expensive. So in the eighties, in the
Speaker 1: mid eighties, they switched to calling people on telephone, which
Speaker 1: which I mean that that's still today. That is the
Speaker 1: gold standard is for a human being to dial up
Speaker 1: another human being and ask them some questions. And we'll
Speaker 1: talk a little more about it. But what what Gallup
Speaker 1: does and what you does, and what a few others
Speaker 1: UM do is it's called randomized sampling or probability sampling,
Speaker 1: which is where you basically leave it to chance that
Speaker 1: any voter registered voter in America is going to get
Speaker 1: a phone call from you, So that what what Gallup
Speaker 1: is doing and what Pew does is called randomized sampling
Speaker 1: or probability sampling, where the any voter in America has
Speaker 1: an equal chance of receiving a phone call from Gallop
Speaker 1: or from Pew and being asked these questions. And it
Speaker 1: worked pretty well for a while when they moved from
Speaker 1: UH in person over to the phone, because they were
Speaker 1: still asking people questions and they could still UM get
Speaker 1: their answers and harass them, which is a big thing
Speaker 1: is we'll see about the this type of sampling. Um,
Speaker 1: the problem is when people started to use caller I D,
Speaker 1: they stopped picking up the phone as much, and so
Speaker 1: the response rate went down dramatic. Yeah, so they would
Speaker 1: call people using random digit dialing, which is a computer
Speaker 1: system where it fed in an area code and then
Speaker 1: the first three digits and then randomly dialed the last four.
Speaker 1: So you've got a pretty good start there on the
Speaker 1: random sampling. But even then they said, you know what,
Speaker 1: women tend to answer the phone more than men. So
Speaker 1: to truly randomize that whenever whoever picks up the phone,
Speaker 1: we have to then follow up and say we want
Speaker 1: to talk to the person in the house who's had
Speaker 1: the most recent birthday, further randomizing. Um. I got it
Speaker 1: kind of a laugh about this because I don't know
Speaker 1: that I've ever, literally ever seen my father pick up
Speaker 1: a telephone in his life, or at least growing up
Speaker 1: for the first eighteen years of my life, I don't
Speaker 1: think I ever saw him answer the phone. It's all
Speaker 1: ham radio, huh. Not not he went into that, but
Speaker 1: just literally not not one time. He would just let
Speaker 1: it ring. If no one was around if my mom
Speaker 1: wasn't around to answer it, and granted it was usually
Speaker 1: never for him, no one ever called to talk to him.
Speaker 1: But I picked up on that and my friends used
Speaker 1: to get really frustrated back before texting that I would
Speaker 1: just never answer my phone. And I always just thought
Speaker 1: it was an option, Like when the phone rings, it
Speaker 1: doesn't mean you're obligated. It just means now you have
Speaker 1: an option you can answer it or not. Well, technically
Speaker 1: that's true. I mean like it depends no, you don't
Speaker 1: have to answer the phone, but it depends on you know,
Speaker 1: who in your life could possibly be calling. I didn't
Speaker 1: think it was rude or anything. I just thought it
Speaker 1: was literally, like, you know, I'm gonna hedge my beds here,
Speaker 1: that one of my friends isn't stuck on the side
Speaker 1: of the road. They can leave a message and if
Speaker 1: they are, I'll go get them. So, UM, what you're
Speaker 1: talking about, Chuck, is what's called a non response, and
Speaker 1: that's factored into the response rate, which with phone pulling
Speaker 1: from nine the nine eight until the nineteen nineties, um
Speaker 1: it was manageable. You. I think the response rate peaked
Speaker 1: at thirty six percent in n which is good. Now
Speaker 1: it's down to like nine percent, because, like I said,
Speaker 1: people have callorad D and if some unknown numbers calling,
Speaker 1: you typically don't answer. And that actually affects things because
Speaker 1: there is a certain kind of person who answers the
Speaker 1: phone no matter what, and they are not like every
Speaker 1: single American, and that actually factors into the kind of
Speaker 1: pole you're connecting. Plus also you want like a certain
Speaker 1: amount of responses. I think out of a sample size,
Speaker 1: you want a minimum of eight hundred survey responses. And
Speaker 1: back in the day, when you got a thirty six
Speaker 1: percent response rate, meaning thirty six percent of those people
Speaker 1: you called would answer the phone and go through all
Speaker 1: of the questions and answer them fully and complete the survey.
Speaker 1: Um since it was down to nine percent, you went
Speaker 1: from having to call between two thousand people to to
Speaker 1: up to nine thousand people now just to get eight
Speaker 1: hundred surveys completed. And that made the whole thing a
Speaker 1: lot more expensive. On the one hand, because it was expensive,
Speaker 1: it meant that there were fewer and fewer companies that
Speaker 1: could conduct these polls, which meant the polls you were
Speaker 1: seeing were more and more legitimate. But on the other hand,
Speaker 1: it also UM usually decreased sample size a little bit, because,
Speaker 1: as as Gallup pointed out, like you can kind of
Speaker 1: fiddle with the numbers a little bit with a smaller
Speaker 1: response rate and smaller sample size. Yeah, and it also
Speaker 1: led to robo calls because of expense, because of people
Speaker 1: not answering their phone as much, and those systems. Uh.
Speaker 1: I mean, I love how Dave Ruce put it. He
Speaker 1: said they they range from okay to terrible, UM, and
Speaker 1: how well they work online polls in these other new techniques.
Speaker 1: But I think we should take a break and then
Speaker 1: talk about, uh what I found the very interesting UM
Speaker 1: way that they further randomized this thing from this point
Speaker 1: forward right after this. All right, so we've already talked
Speaker 1: about the fact that they randomly called someone, and then
Speaker 1: they take one further step on that that call by saying,
Speaker 1: let me speak with whoever had the most recent birthday,
Speaker 1: even if it's I guess you're your three year old, right,
Speaker 1: And and one other thing I kind of made mention
Speaker 1: to it that I have to interject, dude, like harassing people,
Speaker 1: like if you've been picked by this computer, if your
Speaker 1: phone number has been picked, they're going to keep calling
Speaker 1: you and calling you. And that is because, as a
Speaker 1: person who doesn't normally participate in phone surveys, you are
Speaker 1: a specific kind of person that you you can't be
Speaker 1: left out of the population because you represent a large
Speaker 1: number of people and they want your opinion. So part
Speaker 1: of this phone standard of calling people is to call
Speaker 1: them over and over and over again to basically harass
Speaker 1: them into participating to get their answers for this serve
Speaker 1: because it's as important, if not more important, sometimes than
Speaker 1: the people who are like, oh, yeah, I'd love to
Speaker 1: answer this phone survey. Two totally different kinds of people. Yeah. Absolutely.
Speaker 1: And I was totally kidding, by the way, to the
Speaker 1: listener when I said they will speak to a three
Speaker 1: year old. They they asked the most recent birthday of
Speaker 1: someone a voting age obviously, right. Um, So then you've
Speaker 1: got a pretty pretty decent random sampling to begin with.
Speaker 1: And then you have to start uh the process of waiting, uh,
Speaker 1: which comes in a lot of different forms. Um. If
Speaker 1: you want an example of like a really good political poll,
Speaker 1: it's gonna be paid for by a neutral source. It's
Speaker 1: not gonna be um like you know, a CNN pole
Speaker 1: or a Fox News poll or a superpack or anything
Speaker 1: like that. Um, you're gonna have a random sample of
Speaker 1: the public, which we just talked about. You're gonna be
Speaker 1: dialing cell phones and landlines these days. That's a big one. Also,
Speaker 1: they'll ask you if you have a cell phone and
Speaker 1: a landline, and if you say yes, I have both,
Speaker 1: they're going to adjust your response based on the fact
Speaker 1: that you had a higher chance of being selected because
Speaker 1: you have two numbers that the computer could have picked right.
Speaker 1: And another thing is, like you mentioned, they're gonna keep
Speaker 1: calling you the best ones, use live interviewers still. And
Speaker 1: then what they want to do, and this last one
Speaker 1: is really important, is you're they're gonna try and improve
Speaker 1: the accuracy of the results by waiting the response to match.
Speaker 1: What they want to do is just match a real
Speaker 1: world demographic, age race, your income level, your education level,
Speaker 1: and all of that stuff is factored in, and all
Speaker 1: the stuff is weighed out because, um, well we'll talk
Speaker 1: about it, but you know, there are many different kinds
Speaker 1: of Americans, and if you want a really good sampling
Speaker 1: of different kinds of Americans, you're gonna, like, like you said,
Speaker 1: I have to fidget with the numbers to make it
Speaker 1: a true representative population right. So, um, because even if
Speaker 1: you just get it exactly right demographically and waited, which,
Speaker 1: like you said, we'll talk about some more in a second,
Speaker 1: you still have that marge and of error. And again
Speaker 1: that's that um, you know, plus or minus three points,
Speaker 1: and that means that it could be cent or it
Speaker 1: could be they don't know, but somewhere between that, most
Speaker 1: of your answers are going to be the like, the
Speaker 1: correct answers somewhere in there. That's what that means with
Speaker 1: that that margin of error, And the reason that that's
Speaker 1: built in is because it is basically impossible to perfectly
Speaker 1: represent the larger population through random sampling. You're just not
Speaker 1: going to pick everybody correctly just by the fact that
Speaker 1: it's random and it's a sample. Yeah, and that's important
Speaker 1: because um, like that's why you hear so much hay
Speaker 1: being made over a double digit lead in a poll um,
Speaker 1: which Biden had sort of semi recently. I know it's
Speaker 1: it's gotten a lot tighter since then, but you know,
Speaker 1: when Biden was up by I think like ten percentage points,
Speaker 1: people were flipping out because you know, like we said,
Speaker 1: it's plus or minus four for each candidate. So that's
Speaker 1: a total of eight. And so basically the press start
Speaker 1: screaming like he's outside of the margin of error. Everybody
Speaker 1: like nothing can beat him, right right, yeah, But now
Speaker 1: things are back within that margin. I saw PBS News
Speaker 1: Our UM they interview Mark Shields and David Brooks. Um.
Speaker 1: Brooks is a New York Times columnists and I think
Speaker 1: Mark Shields is an independent columnist. And one of them
Speaker 1: actually said, and this is in July, America has clearly
Speaker 1: made up its mind on who's going to be the
Speaker 1: next president. And I was like, this is July. Did
Speaker 1: you not learn anything from two thousand and sixteen or
Speaker 1: you I couldn't believe that those words, And there's a
Speaker 1: matter of factly, yeah, it's irresponsible. And there's been studies
Speaker 1: about this too that have suggested that that words like that,
Speaker 1: that um polls that say chance of winning, that this
Speaker 1: kind of stuff like actually has a negative impact on
Speaker 1: the leader because it makes people think, well, I don't
Speaker 1: need to go out and vote. Everybody else is going
Speaker 1: to go vote, and the turnout might be lower than otherwise.
Speaker 1: There's also people that just who well, there's people who
Speaker 1: dispute that. They say, yes, it makes sense intuitively and anecdotally,
Speaker 1: but we've yet to actually see genuine data that that
Speaker 1: says clearly that this has this effect. But it's something
Speaker 1: that's still being studied right now whether it actually does
Speaker 1: or not well. And I also saw an article the
Speaker 1: other day about the the quote unquote silent majority, and
Speaker 1: that another reason those polls were so wrong back then
Speaker 1: and they're saying are probably wrong now is because there
Speaker 1: are there they They say that there's a substantial block
Speaker 1: of voters who vary privately and secretly vote for Trump. Yeah,
Speaker 1: they're the term for them among pollsters as shy Trump voters.
Speaker 1: They won't admit that they're going to vote for Trump,
Speaker 1: but they're going to vote for Trump, and that that
Speaker 1: affects polls. I saw that that's actually not been proven
Speaker 1: to actually exist. Um, but I think it was a Pew.
Speaker 1: There's a really great Pew article. If this stuff is
Speaker 1: speaking to you at all, go check out Pews Key
Speaker 1: Things to Know about Election Polling in the US and
Speaker 1: Now has a bunch of great links that you should
Speaker 1: follow in there. And there's also Sideline UM. They have
Speaker 1: Surveys and Polling, which is a guide for journalists to polling.
Speaker 1: But I I found out you don't actually have to
Speaker 1: be a journalist to read it online. So if you
Speaker 1: want to go check those out, they have some great like,
Speaker 1: um like just some breakdowns of some of the stuff
Speaker 1: we're talking about, but also about how to read polls
Speaker 1: and what to trust and look for in general. Uh.
Speaker 1: And then a little known fact, PEW was actually originally
Speaker 1: called uh pupu until six when Star Wars came out
Speaker 1: and they're like, we gotta change our name now, guys. Yeah,
Speaker 1: I can't do it, man, it is dant o rama
Speaker 1: today with you. Huh. So back to the waiting thing. Um.
Speaker 1: And by the way, we should mention that Gallup said
Speaker 1: if they wanted to um increase that sample size and
Speaker 1: actually get the margin of error down to like plus
Speaker 1: our minus two, that they could do that, but that
Speaker 1: would be like a literal increase in the cost. So like,
Speaker 1: everyone just please live with plus or minus three or
Speaker 1: four points. Yeah, and now everybody generally does. And and
Speaker 1: Dave uses this really good example. Dave Russ helped us
Speaker 1: out with this, and he said, um, this this margin
Speaker 1: of air is best understood where if you selected a
Speaker 1: hundred marbles UM five there's a jar five five hundred
Speaker 1: blue marbles, and you pick out a hundred of them. Um,
Speaker 1: you might pick out fifty of each one time. And
Speaker 1: then what he said, five hundred marbles. Oh no, I'm sorry,
Speaker 1: a thousand marbles. I've lost my marbles. Yes, there's a
Speaker 1: thousand marbles. Okay, five hundred are red and five hundred
Speaker 1: of blue. Your task, Chuck, is to pick out a hundred.
Speaker 1: So you go to the trouble picking out a hundred
Speaker 1: fifty or red, fifty or blue. And I say do
Speaker 1: it again, and this time is forty seven and fifty three,
Speaker 1: and keep saying again again, right, and I smack my
Speaker 1: riding crop on the desk. That's that you're sitting at
Speaker 1: times because you gets super turned on. Yeah, I did
Speaker 1: it a hundred times because dear leader told me to.
Speaker 1: And at the end you get a little bell curve
Speaker 1: and basically a plus or minus four. Right, So yeah,
Speaker 1: almost all of them, this is what's confidence involved. Almost
Speaker 1: all of them are going to fall in that bell curve.
Speaker 1: There's going to be some outler, it's just gonna be
Speaker 1: that one time where it was just absolutely insane. You
Speaker 1: actually picked one hundred red marbles randomly blindfolded from this
Speaker 1: jar that that's that's so insignificant statistically, it's just such
Speaker 1: an outlier. But almost all the we're gonna be in there.
Speaker 1: So when you're pulling like this large group of people,
Speaker 1: like American voters, and of them are falling within a
Speaker 1: couple of percentage points of either side of this this middle,
Speaker 1: you can pretty much feel confident about that. And that
Speaker 1: is the basis of of election polling, of political polling,
Speaker 1: of all polling, really that they have this built in
Speaker 1: margin that they know exists, but everybody can live with it.
Speaker 1: The problem is is when you're hovering around that fifty
Speaker 1: mark and you're talking about a two party system, one
Speaker 1: of them has like and the other one has but
Speaker 1: there's a plus or minus of like two points. It
Speaker 1: means we have no idea. And some people would say, well,
Speaker 1: why even do polling, because what you're showing there is
Speaker 1: not who's going to win. That's not the point of polling.
Speaker 1: But the point of polling is to take a snapshot
Speaker 1: of how America or whoever you're polling is feeling that
Speaker 1: moment about who to elect, about what laws to pass,
Speaker 1: about religion, about um. The Cleveland Indians. It doesn't matter, right,
Speaker 1: like the the the that's what a poll does. But
Speaker 1: you can pervert polls into making them talk a different
Speaker 1: language and say, hey, he look at this percentage. You
Speaker 1: take these polls and you convert them into something else.
Speaker 1: Now you have something like a chance this person is
Speaker 1: gonna win. Go shout that, Wolf Splitz heer, and Wolf
Speaker 1: Flitzer goes and shouts it as loud as he can. So, uh,
Speaker 1: we need to talk a little bit more about waiting.
Speaker 1: I mentioned earlier that there's other things they do to
Speaker 1: sort of, um tip the scale, and that sounds like
Speaker 1: a bad term, so I guess I shouldn't say it
Speaker 1: that way. But um things they do to make it
Speaker 1: equitable and a true representative of the American population. For instance, UM,
Speaker 1: African American voters make up twelve percent of voters. So
Speaker 1: if they did a poll and in the end they
Speaker 1: only got six percent of respondents so were African American,
Speaker 1: then they just double it. Basically, Um, if the respondents
Speaker 1: were overwhelmingly Caucasian, they would weight that down to their
Speaker 1: true representative number, which is about I think sixty six percent. Yeah,
Speaker 1: the electorate is white, and of U of white people
Speaker 1: respond or eight percent of the people that respond are white,
Speaker 1: then they're going to kick that down. And again, this
Speaker 1: is just adjusting the poll to the proper weight, so
Speaker 1: you have a really legitimate snapshot. And you know, if
Speaker 1: it sounds crazy that there using a thousand people's responses,
Speaker 1: uh and drawing that out to the size of the
Speaker 1: voting population of America, it is. But if you're a statistician,
Speaker 1: it isn't you know. I mean, you know, it reliably
Speaker 1: works as long as you present it with plus or
Speaker 1: minus this margin of error. Um, it's as crazy as
Speaker 1: just an average Joe on the street. It does to
Speaker 1: be like they ask a thousand people and we're supposed
Speaker 1: to know and extrapolate that, and a statistician who our
Speaker 1: number wants and data wants would say, yeah, that's exactly
Speaker 1: what that means. Shut up, that's really all. That's really
Speaker 1: all you need. But it really is a testimony to
Speaker 1: the power of those those statistics in that data and
Speaker 1: the analysis of them. Yeah, waiting is really important. It
Speaker 1: goes far beyond just like age political party. Um. I
Speaker 1: think gallup uses eight different variables. The New York Times
Speaker 1: see in a college poll uses ten, like and they
Speaker 1: include things like marital status and home ownership PU uses
Speaker 1: twelve variables. Um. They ask things like do you have
Speaker 1: home internet access? To you have volunteers or do you
Speaker 1: engage in volunteerism um, And all of these things have
Speaker 1: been owned to be associated. So like, if you're a
Speaker 1: white woman age sixty five to seventy five who volunteers
Speaker 1: twice a month and lives in the suburbs, you're a
Speaker 1: very specific person where you you there's a group of
Speaker 1: people out there who vote like a certain way, and
Speaker 1: you represent like all those people with that. So they
Speaker 1: they'll wait the results based on these additional questions that
Speaker 1: you're into. They don't just ask you do you are
Speaker 1: you going to vote for Trump or Biden? And there's
Speaker 1: also built into that question a really important point, are
Speaker 1: you going to vote? Yeah, that's a that's a huge
Speaker 1: thing we haven't talked about. It's one thing to pull
Speaker 1: registered voters. But here in America, somehow, uh, presidential elections
Speaker 1: only get about six turnout still, which is shameful, shameful
Speaker 1: and crazy. But um, that's another podcast. But uh so,
Speaker 1: most of the really really good polls drilled down and
Speaker 1: to get a real, real good representation of what might
Speaker 1: actually happen. They try to drill down to whether or
Speaker 1: not you're most likely to actually vote, because he cares
Speaker 1: what your pain is if you're not gonna vote, and
Speaker 1: they and I mean they generally take your word for
Speaker 1: it that you're telling the truth, you know. Um yeah,
Speaker 1: but they do have like nine I think pu Yeah,
Speaker 1: Pew has nine questions that they basically used to establish
Speaker 1: that you're you are planning on voting, like you're actually
Speaker 1: going to vote, You're not full of hot air, you know. Yeah.
Speaker 1: I don't know those questions are, but I imagine they
Speaker 1: have to do with do you know where you're pulling places?
Speaker 1: Do you have transportation? Stuff? Like I was thinking they
Speaker 1: were going to be like, are you really really gonna vote?
Speaker 1: Was like question three, and they just kept adding release
Speaker 1: right you So, um, So you've got these these people
Speaker 1: who have been called and they have answered these questions,
Speaker 1: and they have participated in this survey whether they wanted
Speaker 1: to or not, and they've they've finally done it. Built
Speaker 1: into that margin of error built into the poll. Is
Speaker 1: that understandable margin verror that just comes from the fact
Speaker 1: that it's a randomized sample, right, But what PE and
Speaker 1: any other legitimate UM group polling group will point out
Speaker 1: is that the margin is actually greater than that. That
Speaker 1: the margin of error for the average poll according to Pew,
Speaker 1: is that it's something more like six points, not not
Speaker 1: three or four, it's actually six. And the reason why
Speaker 1: it's built on top of that margin of error that's
Speaker 1: that's automatically part of the poll just by the virtue
Speaker 1: of it being a randomized sample. Are things like the
Speaker 1: person typing in the wrong key accidentally that those kind
Speaker 1: of things add up, or that the question isn't worded
Speaker 1: clearly enough that anybody who hears it knows the intent
Speaker 1: and knows what their answer is, that there's some sort
Speaker 1: of um miscommunication involved. There's also things that they can't control,
Speaker 1: for like people who have pseudo opinions who don't want
Speaker 1: to sound dumb, so they just answer yes or no
Speaker 1: by sense something they really don't care about either way
Speaker 1: UM and because they don't actually have an opinion, that
Speaker 1: actually that that waits things the wrong way. So when
Speaker 1: you add all these stuff these things up, UM, you
Speaker 1: have these additional um uh errors that lead to different
Speaker 1: to like a bias overall in in the the UM
Speaker 1: the poll, which can affect the outcome. But again, the
Speaker 1: companies that have the money to conduct like these genuinely
Speaker 1: big gold standard polls, are they they know enough to
Speaker 1: know how to kind of factor control for those as
Speaker 1: much as possible. But still, what Pew says is, if
Speaker 1: you're listening to a poll and somebody says plus or
Speaker 1: minds three points, you should probably go ahead and double
Speaker 1: that in your mind, double it in your mind, double
Speaker 1: your W, your pleasure wr fun W, your margin error.
Speaker 1: So let's take a break and we're gonna come back
Speaker 1: and talk about what exactly they think went wrong with
Speaker 1: those state polls right after this, Sorry, George, all right,
Speaker 1: So I think it's generally acknowledged that the um and
Speaker 1: again I want to say the polling was was off,
Speaker 1: but apparently the pulling wasn't off, but the way it
Speaker 1: was reported on was off. But what really happened in
Speaker 1: seen what was off was the state polling, and what
Speaker 1: they think they've, like you said, gone back and obsessed
Speaker 1: over these polls since then, um, you know, because they
Speaker 1: were already statistical walks, but when something like this happens,
Speaker 1: they really sort of get worked into a dander and
Speaker 1: get to the bottom of it. I mean people were
Speaker 1: calling for the end of polling. She said it was
Speaker 1: a failed profession. Basically, he was like, I'm getting rich
Speaker 1: off this man. Yeah, we can't end polling. Jimmy Pugh
Speaker 1: was like, stop stop talking like that. So what happened
Speaker 1: in is uh they think is that uh, a lot
Speaker 1: of non educated white people came out in big, big
Speaker 1: numbers for Donald Trump. And that was a sort of
Speaker 1: a new not a new factor because they had always
Speaker 1: talked about college education, but a new factor in how
Speaker 1: outsized of a factor that was. It had never been
Speaker 1: that outsized. And so all these state posters they didn't
Speaker 1: wait it and they didn't adjust their polls to reflect
Speaker 1: this um fact that college educated people are more likely
Speaker 1: going to respond to these surveys. So their polls were
Speaker 1: just off. Yeah, and they knew that college educated people
Speaker 1: were more likely to respond to the survey. That wasn't
Speaker 1: news to them. What caught them sleeping was that they
Speaker 1: had not picked up on the fact that this group
Speaker 1: of people, um, non college educated white voters, We're going
Speaker 1: to go to the polls in numbers like never before,
Speaker 1: and that they were going to vote for Trump. They
Speaker 1: did not pick up on that that was brand new,
Speaker 1: like that didn't exist before. Trump basically brought up a
Speaker 1: new electorate that helped get him elected, especially in battleground
Speaker 1: states like Wisconsin, in Michigan, uh, in Indiana, although I
Speaker 1: think in the enda he was a shoeing because of pens.
Speaker 1: But the the these this group of voters that did
Speaker 1: not exist, with this line between college educated and non
Speaker 1: college educated white voters, that that partisan gulf hadn't existed
Speaker 1: before election day. The pollsters didn't pick up on it,
Speaker 1: and so they didn't wait those responses because they never
Speaker 1: had to wait the responses before based on college education. Yeah,
Speaker 1: so suburb exurbs and especially the rural vote counted like
Speaker 1: it had never counted before. UM, which is obviously why
Speaker 1: you see what's going on right now, like a very
Speaker 1: hard pushed by the Trump campaign to um to get
Speaker 1: these the same people out again. Uh and in the
Speaker 1: way that they do that that is the nicest way
Speaker 1: I can put it there, genuinely is so UM. Yeah.
Speaker 1: So the idea that that there was all this was
Speaker 1: already kind of a close race, a closer race than
Speaker 1: was being broadcast. UM, that these these electoral um, huge
Speaker 1: electoral battleground states that got flipped. Uh. That was basically
Speaker 1: the reason that UM Trump was able to take the
Speaker 1: electoral College. But the the idea is that these voters
Speaker 1: kind of came out of nowhere and voted for Trump,
Speaker 1: and that there were some other things that happened to
Speaker 1: UM that the pollster didn't anticipate. One that the undecided voters,
Speaker 1: people who said I'm legitimately undecided at this point a
Speaker 1: week before the election, Uh, from whatever, they broke hard
Speaker 1: in favor of Trump. On election date when they made
Speaker 1: their decision, they voted for Trump that hadn't been predicted. Um.
Speaker 1: That was another big one. And then one of the
Speaker 1: other things too, is that the polls were just doing
Speaker 1: what polls do, which is sometimes they're right, sometimes they're wrong.
Speaker 1: But polls had gotten so good in the two thousand
Speaker 1: oughts that people came to to be overconfident in their
Speaker 1: ability to predict and pick winners. And the two thousand
Speaker 1: and sixteen grates reminded us, like, polling is not perfect,
Speaker 1: let's stop pretending it is. Yeah, and it's UM. A
Speaker 1: lot of it has to do with, like we've been
Speaker 1: kind of harping on the way the media presents it.
Speaker 1: And then a lot of that has to do with
Speaker 1: just our conditioned how we're conditioned to look at things
Speaker 1: like underdogs. Um, and it's different in politics. And I
Speaker 1: remember when these aggregators, especially at five thirty eight, Uh,
Speaker 1: they had these predictive models, and they started talking about
Speaker 1: the fact that and I think the Washington Post even
Speaker 1: wrote a good comparison to sports. And you know, if
Speaker 1: someone has a is a real big underdog going into
Speaker 1: like a super Bowl or a World Series and they
Speaker 1: end up winning, people don't get angry and go after
Speaker 1: the people who said they had a fifteen or twenty
Speaker 1: percent chance of winning. They just said, wow, what a story,
Speaker 1: the underdog one. But there are so few presidential elections, uh,
Speaker 1: you know, one every four years that it's it's the
Speaker 1: same thing, but people just look at it differently. Like
Speaker 1: an underdog like Trump was an underdog that supposedly had
Speaker 1: like a fifteen to thirty percent chance of winning. Some
Speaker 1: people said, one, yeah, well that's ridiculous, But a thirty
Speaker 1: percent chance of winning is a real shot at winning,
Speaker 1: for sure. Yeah, that's the way it's framed. It doesn't
Speaker 1: seem that way in politics, no, And so that's one thing.
Speaker 1: But another thing is that we shouldn't even be talking
Speaker 1: about president vential elections with like chance of winning, chance
Speaker 1: of winning like that is not how we should present it.
Speaker 1: We and that's not how we used to present it.
Speaker 1: We used to present it saying like this poll found that, um,
Speaker 1: that Clinton was going to lead Trump of forty eight
Speaker 1: percent or something like that, plus or minus two points,
Speaker 1: and that would have shown you like, Okay, well this
Speaker 1: is a really close race, way closer than I think, um.
Speaker 1: And that's that there's my information. Not The problem is
Speaker 1: that you can take that same statistic chance of winning
Speaker 1: plus or minus of four point um margin of error.
Speaker 1: If you convert that to a normal distribution, you come
Speaker 1: up with a probability of a win. That's the problem
Speaker 1: is that the statistics that are being being the data
Speaker 1: that's being produced by these polls, are being converted in
Speaker 1: ways that they shouldn't be, and then that's what the
Speaker 1: media jumps on. That's what the public lapse up because
Speaker 1: that is the horse race statistic, and eighty four percent
Speaker 1: chance of winning, a fifteen percent chance of winning, that's
Speaker 1: what we we think about. That's what we look at.
Speaker 1: And so rather than realizing that actually this is a
Speaker 1: close race plus or minds four points, we see eighty
Speaker 1: four percent chance of winning, and that's a foregone conclusion
Speaker 1: that that person is going to win. That's that, ultimately
Speaker 1: is where the media and the public are culpable for this. Yeah,
Speaker 1: I don't think uh, I don't think they were meant
Speaker 1: to be extrapolated like that to begin with. And that
Speaker 1: you know, polls are valuable, but like, I haven't looked
Speaker 1: at any polls, and partially because of the way went down.
Speaker 1: Um And in fact, for the past week, I've taken
Speaker 1: a complete uh internet news and social media break and
Speaker 1: it's been pretty great actually because yeah, I mean I
Speaker 1: literally haven't looked at a single news thing. Very sadly
Speaker 1: found out that Chadwick Boseman passed away like three days
Speaker 1: after word. Like that's how how dark I've gone, uh,
Speaker 1: and not looking at the Internet unless it's something that
Speaker 1: brings me joy, which is to say, you know, old
Speaker 1: led Zeppelin and Van Halen YouTube videos. I was looking
Speaker 1: up classic Mad magazine covers of the eighties. That's all
Speaker 1: I've been doing is if it doesn't bring me joy
Speaker 1: on the internet, I'm not doing it. Um. And you know,
Speaker 1: I gotta break that soon, because I do think you
Speaker 1: should be active and involved in in in the know.
Speaker 1: But yeah, but taking a break fairly regularly is definitely
Speaker 1: mentally healthy. But that aside, I haven't I'm not looking
Speaker 1: at any polls and I don't care what any poll says.
Speaker 1: We'll see. So I was I was thinking very similar
Speaker 1: stuff too, and um, like, what what's the point of poles? Okay,
Speaker 1: well I finally found it. If you look that up
Speaker 1: on Google, there's there's just very little on it. But
Speaker 1: I found somebody who who explained it pretty well. I
Speaker 1: thought that, Um, poles aren't meant to tell you who's
Speaker 1: going to win. They're not. They're not forecasting models, like
Speaker 1: I said before, They're to be like a snapshot of
Speaker 1: how whoever you're polling feels at the moment right um.
Speaker 1: And in doing that, because you are sampling American people
Speaker 1: and these are independent news organizations typically who are carrying
Speaker 1: out these these polls, you get to tell everybody else
Speaker 1: how America is feeling. Rather than the leaders saying I'll
Speaker 1: decide how you're feeling. I can decide what you want
Speaker 1: and what you need and what you think is important.
Speaker 1: Polls prevent that from happening by telling the rest of
Speaker 1: the people, Hey, this is how everybody else is feeling
Speaker 1: right now too. And in some ways it is kind
Speaker 1: of sheepish, where you know, the ideas like, oh, you know,
Speaker 1: is that supposed to sway my opinion that everybody's going
Speaker 1: to vote for this person and not for that person.
Speaker 1: That should have no bearing or impact on your vote,
Speaker 1: And it feels like that that's how polls are used sometimes.
Speaker 1: But if you step back and look and see that
Speaker 1: they're actually kind of an important part of of sharing
Speaker 1: what other people are thinking, rather than being told what
Speaker 1: we're thinking or you know, what, what to think, then
Speaker 1: they actually are pretty legitimate in that sense. Yeah, well,
Speaker 1: you know, I say, take your polls and sit on it. Well,
Speaker 1: one more thing we get. We cannot talk about polling
Speaker 1: and not talk about Internet polling real quick. This is
Speaker 1: a completely different style of polling than's ever been done before.
Speaker 1: Rather than a randomized sample, you actually just say hey,
Speaker 1: you want to take the survey, and people click it.
Speaker 1: So it's called opting in opt in surveying, and very
Speaker 1: specific kinds of people take surveys on purpose on the Internet,
Speaker 1: so they really are because they're new. They're really now
Speaker 1: figuring out how to wait these things are not UM
Speaker 1: and how to how to use them because they can
Speaker 1: produce legitimate UM data, but it depends on who's conducting
Speaker 1: the poll, how if they know what they're doing, that
Speaker 1: kind of stuff. But just like everything else that the
Speaker 1: moving things online is democratize polling, and so anybody can
Speaker 1: conduct a poll now and basically enter the news cycle.
Speaker 1: That's how Kid Rock almost became a senator in Michigan
Speaker 1: for a second there. But so on the one hand,
Speaker 1: it's good, but it's also we're in a big period
Speaker 1: of disruption as far as polling is concerned. So for you,
Speaker 1: the polling consumer, either go like Chalk and just stop
Speaker 1: listening to polls altogether, or UM look for things like transparency.
Speaker 1: Do you recognize the company or the name that produced
Speaker 1: the poll. Are they sharing their data, like how the
Speaker 1: questions exactly were worded, what their population size was, how
Speaker 1: they waited at all this stuff? UM, if the if
Speaker 1: there's all, if all that stuff is included, you can
Speaker 1: probably trust the poll. And then UM, beyond that, just
Speaker 1: remember what you're looking at that this isn't a predictor
Speaker 1: of who's gonna win. It was a snapshot for a
Speaker 1: very very brief moment of a very specific sample of
Speaker 1: America to just to show how people would vote right then.
Speaker 1: And it was right then too. This is not election
Speaker 1: day we're talking about. Yeah, And I'm you know, I
Speaker 1: want to be clear. I'm not poopooing polls. I just uh,
Speaker 1: they're they're valid and useful, but I just don't care
Speaker 1: to look at them right now. I understand. Yeah, that's
Speaker 1: that's my jam. Well, you got anything else about polls,
Speaker 1: nothing else about powells. Well, if you want to know
Speaker 1: about polls, uh, start looking around and you go check
Speaker 1: out Pew stuff and uh sidelines stuff and all that stuff. Um.
Speaker 1: And since I said stuff three times here it comes
Speaker 1: from will stillt Kid or a candy man. So, uh,
Speaker 1: this is from Keiley Price. And Keily says this, Hi, guys,
Speaker 1: I'm writing today not only to confess my unending love
Speaker 1: for stuff you should know, but also to share a
Speaker 1: link to some black owned bookstores. It would be so
Speaker 1: cool if all of your listeners purchased your book. She
Speaker 1: should just say period. Uh. Comma from a black owned
Speaker 1: bookstore couldn't agree more. By the way, a couple of
Speaker 1: podcasters that I listened to while I wait for stuff.
Speaker 1: You should know have books out and coming out soon,
Speaker 1: and they encourage their listeners to support black owned businesses
Speaker 1: through the purchase of their book to win Win. I
Speaker 1: don't know why it's taking me so long to think
Speaker 1: to write this to you guys. I blame it on
Speaker 1: Corona madness. But last, but not least, I'll say I
Speaker 1: love the End of the World with Josh Clark and
Speaker 1: movie Crush as well. Any chance to hear you guys
Speaker 1: talk as a chance worth taking. When we get a
Speaker 1: COVID vaccine and you guys can do your live shows again,
Speaker 1: please come to Nashville. Oh yeah, for sure. I think
Speaker 1: we planned on Nashville. Yeah, Nashville got scuttled by COVID
Speaker 1: this time around. You're going to come now? We might
Speaker 1: not ever be able to come. No, I know it's
Speaker 1: super close to Atlanta. To lose my mind if I
Speaker 1: got to see you guys here all the best, Keiley
Speaker 1: Price and so. Keiley sent a link to a handy
Speaker 1: website that lists black owned bookstores near you. I made
Speaker 1: a little uh you are else shortener to make it
Speaker 1: easier on everyone. Oh, let's have it so you can
Speaker 1: go to bit dot lee slash s y s k
Speaker 1: b l m uh and find black owned bookstores near
Speaker 1: you to purchase Stuff you Should Know, an incomplete compendium
Speaker 1: of mostly interesting things at the very least. Um, we
Speaker 1: like to kurch people to go to indie bound dot
Speaker 1: org and support indie bookstores. I don't know if there
Speaker 1: is an actual black owned indie bookstore website, but I
Speaker 1: would imagine most of the black owned bookstores are indie bookstores.
Speaker 1: Uh yeah, probably, So check it out. Bit dot lee
Speaker 1: slash s y s k v l M. Go out
Speaker 1: and buy our book. Everybody. You're gonna love it. It's
Speaker 1: really great, m um, and thanks thanks for that, Chuck,
Speaker 1: and thanks for setting us up for that too, Kiley,
Speaker 1: much appreciated. We'll see you in Nashville. I guess Kiely
Speaker 1: will be the one, like you said, losing your mind
Speaker 1: in the crowd. If you want to lose your mind
Speaker 1: on us via email, we love that kind of thing.
Speaker 1: Kind of um. You can send it off to stuff
Speaker 1: Podcasts at iHeart radio dot com. Stuff you Should Know
Speaker 1: is a production of iHeart Radio's How Stuff Works. For
Speaker 1: more podcasts for my Heart Radio visit the iHeart Radio app,
Speaker 1: Apple podcasts, or wherever you listen to your favorite shows.