WEBVTT - Lab 015: Big Poll Energy 0:00:00.360 --> 0:00:02.640 Y'all keep sending me your Instagram polls and saying who 0:00:02.759 --> 0:00:05.200 likes candy corn? I love candy corn. Do you like 0:00:05.240 --> 0:00:08.840 candy corn? No? What it's like? Wax? What are you 0:00:08.960 --> 0:00:11.840 talking about? Candy corn has such a great flavor. I 0:00:11.880 --> 0:00:13.960 also saw a pole on Twitter where somebody was saying 0:00:13.960 --> 0:00:15.920 what's the most trash candy? And everybody was saying, you 0:00:15.960 --> 0:00:18.760 gotta get Milky Way out of there. I like Milky Way. 0:00:19.360 --> 0:00:22.320 Something's wrong with everybody. Milky Way is a very good candy. 0:00:22.840 --> 0:00:25.840 My favorite candy Three Musketeers. That's good too. I like 0:00:25.880 --> 0:00:29.920 a good nugut? Is that? What's in there? Nowgat you 0:00:30.160 --> 0:00:32.760 throw those things in a freezer for just a little bit. Ooh, 0:00:32.840 --> 0:00:34.960 I've never tried that. Hey, stuff you gave up? I'm 0:00:34.960 --> 0:00:38.560 going to I'm t T and I'm Zakiah and from 0:00:38.560 --> 0:00:57.120 Spotify Studios. This is dope laps. This happens every year 0:00:57.240 --> 0:01:00.000 around Halloween. As soon as October hits in that Halloween 0:01:00.160 --> 0:01:04.080 candy hits the market, everybody becomes the great debater about 0:01:04.200 --> 0:01:09.360 the best Halloween candy. Full size, family size, funny size, 0:01:09.760 --> 0:01:11.640 all these things like that? Are y'all giving out full 0:01:11.680 --> 0:01:14.440 size candy bars? On Halloween. We're not not in my house. 0:01:14.520 --> 0:01:16.200 I saw a thing on Twitter where somebody has given 0:01:16.240 --> 0:01:20.360 out rotessery chickens. Ha ha ha. First of all, you're bawling. 0:01:22.680 --> 0:01:25.520 I don't care what grocery store you're going to to 0:01:25.560 --> 0:01:28.600 give out rotessity chickens. That's crazy. I'm having three guests 0:01:28.640 --> 0:01:32.680 and that's it. Three tricker treaders and good night. Right, 0:01:32.880 --> 0:01:35.840 But it's so interesting to me that people will do 0:01:35.920 --> 0:01:39.400 these polls and you know, your little ten followers will answer. 0:01:39.680 --> 0:01:42.000 I'm only saying that because I got about seventeen, but 0:01:42.680 --> 0:01:44.880 your ten followers will answer. And then people are like, 0:01:44.920 --> 0:01:47.920 it's official, Milkie Way is out of here, and I'm like, 0:01:48.400 --> 0:01:52.280 based off of ten people, your ten friends is that. 0:01:52.280 --> 0:01:54.279 That's not how polls work. That's not how it works. 0:01:54.760 --> 0:01:58.160 You need more people. We don't believe. We don't believe you. 0:01:58.160 --> 0:02:02.200 You need more people. And that leads us right into 0:02:02.200 --> 0:02:07.240 today's topic, polling. There's a lot of polls out there, 0:02:07.360 --> 0:02:09.480 and we see them all the time. Sometimes you're not 0:02:09.480 --> 0:02:12.560 even aware that you are looking at poll data and 0:02:13.160 --> 0:02:14.959 it's right in front of your face. A lot of 0:02:15.040 --> 0:02:17.840 y'all are sharing memes. You got bar charts where you 0:02:17.880 --> 0:02:21.400 should have pie charts and scatterplots when you need a 0:02:21.440 --> 0:02:24.560 line plot. And I want to say, hey, that's not right. 0:02:25.200 --> 0:02:28.560 But today we want to bring you the facts. We're 0:02:28.560 --> 0:02:30.880 going to tell you everything we know about polling. We're 0:02:30.880 --> 0:02:33.680 going to ask all of our questions about polling, because 0:02:33.720 --> 0:02:37.920 I mean, the election season is upon us. Bernie Sanders 0:02:37.919 --> 0:02:41.399 got like twenty five million in donations, Elizabeth Warren got 0:02:41.400 --> 0:02:44.040 like twenty one million in donations. They're taking that money 0:02:44.040 --> 0:02:45.639 and they're gearing up. And what they're gonna start doing 0:02:45.680 --> 0:02:48.760 is they're gonna start using that money to not only 0:02:48.800 --> 0:02:51.960 put out like campaign ads and things like that, but 0:02:51.960 --> 0:02:56.080 they're also buying results from survey firms. And they're going 0:02:56.160 --> 0:02:58.680 to start to look at this data about all of 0:02:58.760 --> 0:03:00.600 us and put it out there and they're going to 0:03:00.639 --> 0:03:03.760 say this, many percent of Americans do this, and this 0:03:03.800 --> 0:03:05.520 is why you should vote for me. So I just 0:03:05.560 --> 0:03:08.080 saw the latest Cricket Media poll. Have you seen this 0:03:08.120 --> 0:03:10.800 when it has Bernie Sanders and Elizabeth Warren as the 0:03:10.840 --> 0:03:13.320 top candidates and it says here are the top five 0:03:13.360 --> 0:03:16.440 candidates after the first debate. But if you zoom in, 0:03:16.600 --> 0:03:20.640 pinch to zoom, that is so tiny. Yeah, it says 0:03:21.200 --> 0:03:26.000 a poll of Democratic voters in Iowa, New Hampshire, and 0:03:26.040 --> 0:03:28.960 South Carolina. And so sometimes people share these polls and 0:03:29.000 --> 0:03:31.080 it's like, this is what Americans think, and I'm like, 0:03:31.120 --> 0:03:33.720 this is from three states, and it's the subgroup of 0:03:33.760 --> 0:03:36.880 a subgroup. You even ask folks from the biggest states 0:03:37.160 --> 0:03:40.839 or the most populated states. Nobody in my family lives 0:03:40.880 --> 0:03:43.080 in any of these states. And so what we're trying 0:03:43.120 --> 0:03:45.240 to do with this episode is give you all the 0:03:45.280 --> 0:03:48.200 necessary information that you need to be able to look 0:03:48.200 --> 0:03:50.040 at some of this stuff and say, oh, that's not 0:03:50.080 --> 0:03:53.720 what that pole meant and find that the real dat 0:03:53.720 --> 0:03:56.520 of yourself. Yeah, And if you're like me, you're wondering 0:03:56.920 --> 0:03:59.760 who are those people getting surveyed? We also have a 0:03:59.760 --> 0:04:02.960 tip about how you can make sure your voice is hurt. 0:04:03.840 --> 0:04:06.200 So let's get into the rescitation. I have questions about 0:04:06.200 --> 0:04:08.440 that because I see a lot of poles being shared 0:04:08.480 --> 0:04:12.040 on Instagram and on Twitter without any of that qualifying information. 0:04:12.400 --> 0:04:14.280 Where did you get these people from? How many people 0:04:14.320 --> 0:04:16.600 did you survey? When did you survey them? And even 0:04:16.680 --> 0:04:19.159 if the information is there, you see how tiny it is, 0:04:19.200 --> 0:04:21.800 and when you just scroll past it and you're like, Okay, 0:04:22.120 --> 0:04:25.440 Sanders A. Warren, I thought it was a shadow box, right, 0:04:25.480 --> 0:04:27.039 I thot just a line. I thought it was a 0:04:27.080 --> 0:04:32.000 line that separate this from the next paragraph divider. So 0:04:32.160 --> 0:04:35.320 I think, like, I have so many questions, like can 0:04:35.440 --> 0:04:38.039 just anybody issue a poll and put this information out? 0:04:38.480 --> 0:04:41.160 Are there certain places we should go when we're looking 0:04:41.160 --> 0:04:42.479 for polling down or like when I want to know 0:04:42.520 --> 0:04:44.839 what's the latest trend? You know, where do you go 0:04:44.920 --> 0:04:48.440 for reliable information? Yeah? And then if I'm seeing information, 0:04:48.880 --> 0:04:52.560 like if I'm seeing a bar graph and it says like, Okay, 0:04:52.720 --> 0:04:55.320 these are the top candidates, blah blah blah, how do 0:04:55.360 --> 0:04:57.120 I make sense of that? Like? What else should I 0:04:57.120 --> 0:04:59.320 look for to say that this is reliable information? What 0:04:59.400 --> 0:05:01.760 am my cue? Right? Which polls should I take with 0:05:01.800 --> 0:05:04.320 a grain of salt? And which polls should I drink 0:05:04.400 --> 0:05:10.760 with salt water? Okay, So now that we know what 0:05:10.800 --> 0:05:14.000 we want to know, let's get into the dissection. This week, 0:05:14.120 --> 0:05:16.640 we are jumping into polls, as you know, and we 0:05:16.839 --> 0:05:20.440 ask doctor Rachel Bidekoffer to help us out. 0:05:20.480 --> 0:05:22.839 I'm the assistant director of the Watson Center for Public 0:05:22.880 --> 0:05:26.400 Policy at Christopher Newport University and a senior research fellow 0:05:26.400 --> 0:05:27.800 at then a Scannon Center in DC. 0:05:28.120 --> 0:05:31.200 Whenever we bring up polls, everybody jumps right to thinking 0:05:31.320 --> 0:05:34.960 about elections. But we know that polls can be about anything. 0:05:35.160 --> 0:05:38.839 Not all polling is about elections. For sure. You get 0:05:38.839 --> 0:05:41.719 polling about public policy issues. You can get polling that 0:05:41.800 --> 0:05:45.479 about market research issues. I mean, companies will use polling 0:05:45.560 --> 0:05:49.040 to determine product placement or you know, the types of 0:05:49.560 --> 0:05:52.760 advertising they want to use for their products. Polling is 0:05:52.839 --> 0:05:54.479 used in cross every sector. 0:05:54.680 --> 0:05:57.559 We're constantly polling each other every day. Like anytime somebody 0:05:57.560 --> 0:05:59.359 puts out anything, people are I don't like that, like 0:05:59.400 --> 0:06:02.479 I love it, you know. So it's like when you 0:06:02.480 --> 0:06:04.599 look in the comments section of like a post on 0:06:05.720 --> 0:06:09.360 the Shade Room or TMZ or anything like that. I 0:06:09.480 --> 0:06:11.200 sometimes I'll just scroll through them and like, Ooh, this 0:06:11.240 --> 0:06:13.440 person don't like it. That's one. This person like it, 0:06:13.480 --> 0:06:16.039 that's one. Then going back and forth and seeing like 0:06:17.200 --> 0:06:19.360 a running pole in my mind to see like where 0:06:19.360 --> 0:06:22.280 people are falling on it, you know, another pole that 0:06:22.360 --> 0:06:25.000 just popped into my head. It's always in these toothpaste 0:06:25.000 --> 0:06:27.880 commercials where it's like nine out of ten dentists choose 0:06:27.920 --> 0:06:30.800 Colgate and I'm like, compared to what else? Right, there's 0:06:30.800 --> 0:06:33.560 always an asterisk and then it says like standard toothpaste. 0:06:33.560 --> 0:06:35.520 I'm like, I don't know what standard toothpaste is? And 0:06:36.120 --> 0:06:39.840 did y'all really survey all the dentists? How many dentists? 0:06:39.880 --> 0:06:41.799 Is that there's a lot of different types of polls. 0:06:41.920 --> 0:06:43.600 I think the one that we are most used to 0:06:43.600 --> 0:06:46.600 seeing as an election poll, and that's what doctor Bittercofford 0:06:46.600 --> 0:06:47.719 calls a horse race. 0:06:47.880 --> 0:06:50.440 Horse race is just an election poll. So a poll 0:06:50.680 --> 0:06:54.000 in which you're trying to determine who's up and who's down, 0:06:54.080 --> 0:06:56.640 or whether or not an election is going to be competitive. 0:06:57.000 --> 0:07:00.080 I wouldn't say there's necessary categories, but different types of 0:07:00.160 --> 0:07:03.839 questions and approaches that election polling might cover, depending on 0:07:03.880 --> 0:07:04.920 the goal of the survey. 0:07:05.080 --> 0:07:07.840 I always see polls that show like people think this, 0:07:08.160 --> 0:07:12.560 or candidates are trending this way, they're doing well with mothers, 0:07:12.600 --> 0:07:16.160 they're doing well with veterans, Like huh, how do you 0:07:16.240 --> 0:07:20.800 get down to the actual sample or people that you 0:07:20.840 --> 0:07:21.400 want to pull? 0:07:21.600 --> 0:07:25.480 So the way that sampling works in a population is 0:07:25.480 --> 0:07:29.800 that you're taking a random sample of a population and 0:07:30.280 --> 0:07:34.440 through a process called waiting, it does tell you within 0:07:34.560 --> 0:07:39.480 a certain window what the preferences of that broader population is. 0:07:39.600 --> 0:07:41.880 So, if I want to say everybody that eats candy 0:07:41.960 --> 0:07:45.880 in America, Okay, we know that that's two million people. 0:07:45.960 --> 0:07:47.640 I know that's not right. But like, we know that's 0:07:47.640 --> 0:07:50.600 two million people, So how many people do I need 0:07:50.640 --> 0:07:54.800 to survey? Right? That becomes your survey sample size in 0:07:54.880 --> 0:07:57.680 order for it to be an accurate representation of that 0:07:57.920 --> 0:08:00.600 entire population of candy eaters, right, Because you're not going 0:08:00.680 --> 0:08:03.120 to get all two million people to participate, right. And 0:08:03.200 --> 0:08:04.440 even if you say, okay, I need to ask a 0:08:04.520 --> 0:08:07.200 thousand people in order to understand what two million people think, 0:08:07.440 --> 0:08:09.240 you're not going to get all of those one thousand 0:08:09.240 --> 0:08:12.080 people to participate. So what she's talking about with waiting 0:08:12.280 --> 0:08:16.440 is creating a poll that has the same demographic of 0:08:16.520 --> 0:08:19.840 the folks that you are trying to represent. 0:08:19.960 --> 0:08:23.120 There are some survey firms that are still using what 0:08:23.160 --> 0:08:27.480 we call random digit dialing for voter surveys, and I 0:08:27.520 --> 0:08:30.760 would argue that that's not an ideal way because voters, 0:08:31.240 --> 0:08:32.960 as it turns out, when you call people and ask 0:08:33.040 --> 0:08:36.080 them are you registered to vote? They will lie to 0:08:36.160 --> 0:08:39.280 you and tell you yes. Because it's socially desirable to 0:08:39.400 --> 0:08:42.280 be registered to vote, and they don't want to admit 0:08:42.320 --> 0:08:43.760 that they're not registered to vote. 0:08:43.840 --> 0:08:45.920 And so when I think about that, you know, you 0:08:46.040 --> 0:08:48.520 say candy corn is great, And when we think about 0:08:48.520 --> 0:08:51.040 the population of people eating candy, we might say, there 0:08:51.040 --> 0:08:53.680 are people who eat candy who don't like chocolate, people 0:08:53.720 --> 0:08:56.319 who eat candy who love chocolate, people who eat candy 0:08:56.320 --> 0:08:58.640 that are allergic to nuts, right, and all those things. 0:08:58.679 --> 0:09:00.680 We will want to capture that in our We wouldn't 0:09:00.679 --> 0:09:04.240 want to only get people that are allergic to nuts, 0:09:04.280 --> 0:09:06.840 because then that would make that kind of that group 0:09:06.880 --> 0:09:10.439 of people. It would keewe the results because then you 0:09:10.440 --> 0:09:14.120 would only get from their perspective, and so sneakers will 0:09:14.160 --> 0:09:17.439 be out of here then exactly because they can't them exactly. 0:09:17.760 --> 0:09:19.440 So basically, what we're trying to get you to see 0:09:19.480 --> 0:09:22.080 is you have to do your research on your population 0:09:22.400 --> 0:09:24.560 to make sure you're finding ways to ensure the credibility 0:09:24.559 --> 0:09:28.000 of your sample. So you'll see the phrase likely voters 0:09:28.000 --> 0:09:32.280 in a lot of polls, and more people claim to 0:09:32.400 --> 0:09:35.880 vote than actually go out and vote when the time comes, 0:09:36.640 --> 0:09:39.240 And so part of the job of the pollsters is 0:09:39.280 --> 0:09:42.520 to try and find those people who will actually go 0:09:42.600 --> 0:09:45.200 out and vote, as opposed to the people who will 0:09:45.280 --> 0:09:47.800 just say they're voting but stay at home. Oh. So 0:09:47.920 --> 0:09:49.920 this also kind of ties into the last episode of 0:09:49.960 --> 0:09:52.679 social cognition, like how do people perceive you? Exactly? So 0:09:53.120 --> 0:09:55.720 if somebody's polling you, you're like, what are they going 0:09:55.800 --> 0:09:57.720 to think of me? And it may change the response 0:09:57.760 --> 0:09:59.440 that you get. Yeah, so you'll say, oh, I'll definitely 0:09:59.480 --> 0:10:02.439 vote with which is a con MM. And that's why 0:10:02.440 --> 0:10:05.400 I don't like exit polls. Oh, because I'm like, everybody's 0:10:05.400 --> 0:10:07.080 out here line they're like, oh, well, the exit poll 0:10:07.200 --> 0:10:10.480 because even on the day of the election, they're basing 0:10:10.520 --> 0:10:12.640 a lot of stuff off of exit polls. So people 0:10:12.640 --> 0:10:15.240 will go in and they'll cast their ballot, and they'll 0:10:15.280 --> 0:10:17.240 come out and they say, who did you vote for 0:10:17.480 --> 0:10:21.160 A or B? And somebody might say A because they 0:10:21.160 --> 0:10:24.240 feel like, oh, well, in my county, everybody's voting for A, 0:10:24.320 --> 0:10:25.880 and I don't want to come out and say I 0:10:25.960 --> 0:10:28.480 voted for B. That might be embarrassing. Somebody may hear me, 0:10:28.559 --> 0:10:31.120 they may judge me, and so then they'll change the answer. 0:10:31.720 --> 0:10:35.720 Another con So, now that kind of captures the essence 0:10:35.760 --> 0:10:39.959 of deciding who you will pull and getting your sample, 0:10:40.320 --> 0:10:44.400 But what about what you'll ask them? That's very important. 0:10:44.800 --> 0:10:47.120 The first thing you can do when designing a survey 0:10:47.200 --> 0:10:49.440 is to make sure that your questions aren't leading. 0:10:49.679 --> 0:10:52.880 You really want to have questions that are worded in 0:10:52.920 --> 0:10:55.200 a way in which you're getting responses that are a 0:10:55.240 --> 0:10:56.360 product of the questions. 0:10:56.920 --> 0:11:01.079 So, for example, if you say, do y'all love my outfit? 0:11:01.480 --> 0:11:05.760 Yes or no? That's leading. You're leading it with we 0:11:05.840 --> 0:11:07.959 should love the outfit. Are y'all going to vote for 0:11:08.000 --> 0:11:13.120 this idiot? Yes or no? Right, that's not a good 0:11:13.120 --> 0:11:17.080 survey question. So when you create questions like this, you're 0:11:17.280 --> 0:11:20.079 basically generating what's called a push pull. This is something 0:11:20.080 --> 0:11:22.960 that pushes the respondent to answer in one way as 0:11:23.000 --> 0:11:25.439 opposed to the other. So her next suggestion is to 0:11:25.440 --> 0:11:29.200 not provide an easy out answer. So that's something that's 0:11:29.280 --> 0:11:31.960 like if you have a bunch of different you have 0:11:31.960 --> 0:11:33.520 a question, you have a bunch of different answers, and 0:11:33.559 --> 0:11:36.760 it ranges on a scale from agreed to disagree. Having 0:11:36.800 --> 0:11:39.599 something in the very center that says neither agreed nor disagree. 0:11:39.840 --> 0:11:42.120 People are more likely to choose that because they don't 0:11:42.160 --> 0:11:45.080 want to do anything that's, like I guess, controversial in 0:11:45.120 --> 0:11:47.840 their mind. So they just want to ride that middle line. 0:11:47.960 --> 0:11:50.239 That's crazy to me. I love to be at the extremes. 0:11:50.640 --> 0:11:57.040 We know my answers always strongly agree or strongly disagree. 0:11:57.200 --> 0:12:00.240 I hate it, loved it. This is the best thing 0:12:00.280 --> 0:12:02.960 I've ever seen. I was just having this conversation with 0:12:03.080 --> 0:12:04.520 my friend and I was like, I thought you said 0:12:04.520 --> 0:12:06.360 you hated that. She said, when I say I hate something, 0:12:06.600 --> 0:12:09.880 I don't really hate it or I don't really love it. 0:12:09.920 --> 0:12:12.560 And I said, huh, that's you. You do that too, 0:12:12.760 --> 0:12:14.960 I don't really hate or really love. So you'll say 0:12:15.040 --> 0:12:17.520 I hate this and then be like, oh, I guess 0:12:17.559 --> 0:12:24.120 it's not that bad. That's just uninformed. I don't know 0:12:24.240 --> 0:12:26.400 any I can't think of a single example where I 0:12:26.400 --> 0:12:28.080 said I hated something and then I said, I guess 0:12:28.120 --> 0:12:31.760 it's not that bad. I can think of situations where 0:12:31.760 --> 0:12:34.600 you've loved stuff and then hated it. Wait, are these 0:12:34.640 --> 0:12:39.319 people people? I just did not love? That a point? 0:12:43.080 --> 0:12:47.000 Now love to hate? Yes, that's an easy progression. I 0:12:47.040 --> 0:12:50.720 start everyone to get from love to you are a ghost. 0:12:50.880 --> 0:12:55.120 Sorry to that man, Sorry to this man who who 0:12:55.640 --> 0:12:58.320 a third tip that doctor Bitterkoffer gave us when it 0:12:58.320 --> 0:13:02.040 comes to designing good surveys, to rotate your answers so 0:13:02.120 --> 0:13:04.800 each person taking the survey will see the answers in 0:13:04.880 --> 0:13:05.960 a different order. 0:13:06.280 --> 0:13:08.960 Because, believe it or not, people will go with the 0:13:09.000 --> 0:13:15.120 first response disproportionately. So if you have, say you have 0:13:15.480 --> 0:13:18.559 a horse race question, and you have the one candidate 0:13:18.679 --> 0:13:23.280 offered always first, you might not get an accurate estimation 0:13:23.520 --> 0:13:26.600 on the race because people will choose the first candidate 0:13:26.720 --> 0:13:29.760 more often just because it's mentioned first. 0:13:30.160 --> 0:13:33.160 That reminds me of like in the SAT when they 0:13:33.200 --> 0:13:36.480 were like, oh, always choose C. Can remember scantrons and stuff, 0:13:36.520 --> 0:13:40.679 Gantron's everything, and people with default to C because they 0:13:40.679 --> 0:13:44.480 were like, oh, when they're making these exams, you're more 0:13:44.600 --> 0:13:46.040 likely to get the right answer if you choose C 0:13:46.640 --> 0:13:48.480 because of whatever reason. I don't know if that's how 0:13:48.520 --> 0:13:52.520 the computer would do it or whatever. And so people 0:13:52.559 --> 0:13:55.880 got hip and then they started rotating the answers. Another 0:13:55.880 --> 0:13:58.800 reason to get these standardized tests out of here. Yeah. 0:13:58.880 --> 0:14:00.880 The last tip we got is to make sure your 0:14:00.920 --> 0:14:03.760 survey is short and to the point. I don't want 0:14:03.760 --> 0:14:06.560 to read this paragraph right, So if it's long, I'm 0:14:06.559 --> 0:14:10.480 probably won't even participate, and so the shorter the survey 0:14:10.640 --> 0:14:13.720 the better. So those are our top pointers for designing 0:14:13.720 --> 0:14:16.720 a survey. You may be asking why is designing good 0:14:16.800 --> 0:14:20.080 questions so important? All of these factors contribute to a 0:14:20.160 --> 0:14:23.400 higher response rate. Response rate refers to the percentage of 0:14:23.520 --> 0:14:27.840 people that you are trying to survey that actually complete 0:14:27.880 --> 0:14:28.760 the survey. 0:14:29.000 --> 0:14:32.800 And over the last few decades, especially as people have 0:14:32.880 --> 0:14:37.080 become more technologically diverse, response rates have been declined. 0:14:37.200 --> 0:14:39.840 That's a good point TT because if you don't have 0:14:40.200 --> 0:14:42.760 if you have long questions, if people don't want to participate, 0:14:42.800 --> 0:14:45.280 that affects your response rate. And so all that work 0:14:45.320 --> 0:14:48.080 you did trying to figure out what's the right sample 0:14:48.160 --> 0:14:50.920 size and number of people, you won't get enough respondents. 0:14:50.960 --> 0:14:54.440 And that's why you know now when you call your 0:14:54.560 --> 0:14:57.280 cell phone provider and they want to do a quick survey, 0:14:57.280 --> 0:14:59.520 at the end of the call, they're like, it's just 0:14:59.680 --> 0:15:05.000 one question, like hanging there. They're starting to catch on. 0:15:05.120 --> 0:15:06.960 It's like, I'm not going to talk on the phone 0:15:07.000 --> 0:15:10.560 to a computer for you know, thirty minutes answering all 0:15:10.560 --> 0:15:12.440 these questions, like on a scale of one to ten, 0:15:12.520 --> 0:15:15.560 what was your experience? No ask me one question was 0:15:15.560 --> 0:15:19.480 the experience good or not? So now we know how 0:15:19.480 --> 0:15:21.480 to get the right population for a survey and how 0:15:21.480 --> 0:15:24.360 to design good questions to ask them. When we come back, 0:15:24.400 --> 0:15:43.480 we're going to talk about making sense of the results. 0:15:45.040 --> 0:15:47.400 We're back, and we're going to get into how to 0:15:48.000 --> 0:15:51.560 we through all of these polls and find the right information. 0:15:52.160 --> 0:15:54.040 Right there's so many polls out there. Every week, I 0:15:54.120 --> 0:15:56.280 feel like there's a new article telling me about the 0:15:56.360 --> 0:15:59.960 latest updates or who's ahead, who's dropping out, who's face. 0:16:00.640 --> 0:16:01.840 I don't know how to make sense of it. And 0:16:01.880 --> 0:16:05.480 as someone who watches academic polls, Twitter polls, political polls, 0:16:05.760 --> 0:16:08.720 and all of y'all's ig polls, how do I know 0:16:08.760 --> 0:16:10.360 whether or not they're any good? 0:16:11.040 --> 0:16:14.360 When a poll comes across the TV and you know, 0:16:14.440 --> 0:16:17.359 the journalist is like, oh, forty five percent of Americans 0:16:17.440 --> 0:16:20.480 love lamas, you know, not to just take that as 0:16:20.520 --> 0:16:23.040 a face value, but to go and look at the 0:16:23.040 --> 0:16:26.760 survey and say, okay, how is the sample taken? 0:16:27.200 --> 0:16:34.320 I love lamas? You do? You should not go of Americans. 0:16:34.560 --> 0:16:37.040 That seems a little low. That seems a little low 0:16:37.080 --> 0:16:39.440 to me. But I'm not gonna dwell on that too much. 0:16:39.480 --> 0:16:40.560 Let's let's just keep going. 0:16:41.160 --> 0:16:43.160 So the very first thing that I look at when 0:16:43.200 --> 0:16:46.160 a survey comes out is that end size really need 0:16:46.280 --> 0:16:49.680 a baseline of at least five hundred completes for a 0:16:49.720 --> 0:16:51.800 survey to be reliable statistically. 0:16:51.920 --> 0:16:55.200 So in size is the number of people that complete 0:16:55.200 --> 0:16:58.480 the survey. Right. In size is important for all surveys. 0:16:58.840 --> 0:17:02.480 The larger en size, the more reliable the outcome of 0:17:02.520 --> 0:17:03.320 the survey is. 0:17:03.920 --> 0:17:07.000 And there are many surveys that have gotten you know, 0:17:07.200 --> 0:17:11.639 primo attention on MSNBC and CNN this cycle that are 0:17:11.720 --> 0:17:15.679 nowhere close to five hundred. Technically speaking, of course, you 0:17:15.840 --> 0:17:21.480 can get a statistical response off of a lower end size, 0:17:21.760 --> 0:17:26.240 but in terms of horse race pulling, there's this statistical 0:17:26.359 --> 0:17:29.680 problem called the margin of era that's exasperated a lot 0:17:29.800 --> 0:17:31.560 by a small in size study. 0:17:32.040 --> 0:17:33.879 You really have issues with margin of error when you 0:17:33.920 --> 0:17:36.240 have a small sample size. In the field that I 0:17:36.280 --> 0:17:39.840 work in, we call that uncertainty. And so it's basically 0:17:40.280 --> 0:17:44.240 saying that when you take a measurement, so let's say 0:17:44.359 --> 0:17:50.160 it's something you're trying to weigh ten pounds, and if 0:17:50.240 --> 0:17:56.040 you have uncertainty that's plus or minus five pounds. That's 0:17:56.080 --> 0:17:58.119 a large range. That means that when you're weighing this, 0:17:58.160 --> 0:18:01.320 it can either be five pounds or or anywhere between 0:18:01.400 --> 0:18:04.280 five pounds and fifteen pounds, so you'reself subtracting five or 0:18:04.280 --> 0:18:06.439 adding five on both exactly. So that means that you 0:18:07.000 --> 0:18:09.760 have don't have a high confidence in the measurement that 0:18:09.800 --> 0:18:11.639 you're that you're taking, but you can also have a 0:18:11.680 --> 0:18:14.720 smaller margin of era though, right, Yeah, So if the 0:18:14.760 --> 0:18:16.920 measurement that you're taking, you're really you have a high 0:18:16.920 --> 0:18:19.639 confidence in it to a higher confidence level, your margin 0:18:19.680 --> 0:18:22.040 of error could be really really small, like a pound. 0:18:22.440 --> 0:18:25.520 So you can say this thing is ten pounds plus 0:18:25.640 --> 0:18:27.680 minus one pounds, so that means it could be nine 0:18:27.720 --> 0:18:30.440 pounds per eleven pounds. And that's really cool to think 0:18:30.480 --> 0:18:32.080 about it, like the way it works in your field, 0:18:32.080 --> 0:18:35.800 because that's really similar to how it kind of works 0:18:35.800 --> 0:18:38.520 for polling. Right, you can say thirty percent of moms 0:18:38.600 --> 0:18:42.120 choose gift, and you can say, with a five percent error, 0:18:42.680 --> 0:18:45.720 twenty five anywhere from twenty five to thirty five percent 0:18:45.720 --> 0:18:49.080 of moms choose gift, and you can get that, right. 0:18:49.119 --> 0:18:52.280 You can get to that type of resolution with a 0:18:52.359 --> 0:18:54.760 small sample size. But if you want to be able 0:18:54.800 --> 0:18:57.119 to say thirty percent of moms choose Gift, and you 0:18:57.240 --> 0:19:00.000 mean twenty nine to thirty one percent of moms choose 0:19:00.040 --> 0:19:03.040 You're gonna have to sample way more, way more moms, yeah, 0:19:03.080 --> 0:19:06.359 to get that type of accuracy. And so you're saying that, 0:19:06.680 --> 0:19:09.560 so margin of RAA is just how uncertain are you 0:19:09.720 --> 0:19:12.960 about this statistic? And then the other part of that 0:19:13.320 --> 0:19:17.880 is your confidence level. And so I think confidence level 0:19:17.920 --> 0:19:21.080 is really cool because that also has to do with 0:19:21.160 --> 0:19:23.960 the number of people that you survey. But I don't 0:19:23.960 --> 0:19:25.639 know what they might call this in your field, but 0:19:25.720 --> 0:19:28.840 confidence level is like thirty percent of moms choose JIFF, 0:19:28.880 --> 0:19:31.720 And if I'm ninety five percent confident, then that means 0:19:31.720 --> 0:19:34.760 that ninety five out of one hundred times that I 0:19:34.800 --> 0:19:37.359 do this survey, thirty percent of moms will choose JIFT. 0:19:38.640 --> 0:19:40.560 Or if I have a ninety nine percent confidence level, 0:19:40.560 --> 0:19:42.199 then that means that ninety nine times if I do 0:19:42.240 --> 0:19:44.960 this survey one hundred times, ninety nine of those times, 0:19:45.119 --> 0:19:47.600 thirty percent of moms are gonna choose JIFT. Jeff, is 0:19:47.640 --> 0:19:54.000 really you? I thought you were allergic allergic to peede us. 0:19:54.160 --> 0:19:56.640 I told you it's a mild allergy. I still eat it. 0:19:57.040 --> 0:19:59.639 It's not safe. I don't recommend that for anybody that 0:19:59.720 --> 0:20:03.199 has mild or extreme peanut allergy. Go to your doctor. So, 0:20:03.600 --> 0:20:07.520 as consumers of poll data, we have to go and 0:20:07.720 --> 0:20:10.680 ask some real questions about how the data was collected. 0:20:10.800 --> 0:20:14.119 One of the things is did the survey cast a 0:20:14.160 --> 0:20:17.440 wide net like that survey I showed you earlier. Yeah, 0:20:17.480 --> 0:20:19.560 like the survey we were talking that we were talking about. 0:20:19.640 --> 0:20:24.679 They only surveyed people in Indiana, South Carolina. Yeah, three states. Yeah, 0:20:24.880 --> 0:20:30.240 and they're making very like sweeping claims, yeah, that we're 0:20:30.280 --> 0:20:32.960 all consuming. It's like, oh my gosh, Okay, so Bernie 0:20:33.000 --> 0:20:36.879 Sanders and Lizbond are tied at nineteen percent and that 0:20:36.960 --> 0:20:39.880 might not necessarily be true everywhere. And then the other 0:20:39.960 --> 0:20:42.040 question is how did they even get the information? Are 0:20:42.080 --> 0:20:45.320 people taking these surveys on the phone, you know those 0:20:45.359 --> 0:20:48.000 people that are standing outside with clipboards at different places. 0:20:48.080 --> 0:20:51.440 Are they sprint past them? Are they doing the surveys 0:20:51.480 --> 0:20:56.040 like that like sorry, I have diarrhea, like I have 0:20:56.119 --> 0:20:58.240 to go? Or are they doing them online? 0:20:58.720 --> 0:21:02.680 If it's online, have they done due diligence to try 0:21:02.720 --> 0:21:07.200 to make their panels randomized as much as possible. Are 0:21:07.200 --> 0:21:11.360 they being transparent about their methodology and the design effect. 0:21:11.440 --> 0:21:14.840 Anytime you see a poll, there should be something at 0:21:14.880 --> 0:21:17.040 the bottom of it that tells you where you can 0:21:17.080 --> 0:21:20.520