WEBVTT - Lord of the Rankings 0:00:15.356 --> 0:00:22.356 Pushkin. Once upon a time in the middle of the 0:00:22.436 --> 0:00:25.556 nineteen eighties, when the Cold War still raged and a 0:00:25.596 --> 0:00:28.636 man named Ronald Reagan occupied the White House, there were 0:00:28.676 --> 0:00:33.116 three major weekly news magazines in the United States. Time 0:00:33.196 --> 0:00:36.796 Magazine was number one, with millions upon millions of copies 0:00:36.836 --> 0:00:42.156 in circulation, Newsweek was number two, and What News Magazine 0:00:42.196 --> 0:00:46.716 came in third place, a sleepy little publication based in Washington, 0:00:46.796 --> 0:00:52.516 DC known as US News and World Report. One of 0:00:52.516 --> 0:00:56.916 the magazine's managing editors in those years, Peter Bernstein, remembers 0:00:56.916 --> 0:00:58.396 the day when everything changed. 0:00:58.956 --> 0:01:02.436 It was a very exciting time. I mean, because Moore 0:01:02.556 --> 0:01:05.516 had just taken over. There were a lot of people buzzing. 0:01:05.156 --> 0:01:11.436 Around mort being Mortimer B. Zuckerman, a Canadian billionaire who 0:01:11.436 --> 0:01:14.436 bought US News and Will Report with dreams of becoming 0:01:14.516 --> 0:01:15.356 a media baron. 0:01:15.916 --> 0:01:19.276 There are a ton of really good ideas sort of 0:01:19.316 --> 0:01:21.916 in the magazine, and I think it's fair to say 0:01:21.956 --> 0:01:25.076 our feeling was that the execution could have been a 0:01:25.156 --> 0:01:27.556 lot better. And one of the things we wanted to 0:01:27.596 --> 0:01:30.356 do was distinguish ourselves from Time and Newsweek, because we 0:01:30.356 --> 0:01:31.796 were the third. 0:01:33.836 --> 0:01:37.276 And how do you distinguish yourself if you're always coming 0:01:37.316 --> 0:01:40.956 in third. If a Canadian billionaire with visions of grandeur 0:01:41.316 --> 0:01:46.236 is breathing down your neck, you do something audacious. You 0:01:46.356 --> 0:01:50.796 create something that no educated parent could possibly ignore, a 0:01:50.836 --> 0:01:55.516 scheme to examine every American college and university and rank 0:01:55.596 --> 0:01:59.596 them from top to bottom. 0:01:59.956 --> 0:02:03.796 We were looking to reinvent the magazine, and the college 0:02:03.876 --> 0:02:06.276 rankings became part of that. 0:02:06.796 --> 0:02:09.716 So Bernstein and his colleagues began collecting a mountain of 0:02:09.716 --> 0:02:12.796 statistics on the universities and colleges of the United States. 0:02:13.356 --> 0:02:15.796 To make sense of those numbers, they came up with 0:02:15.796 --> 0:02:19.716 an algorithm, a secret algorithm that could assign every school 0:02:19.796 --> 0:02:23.556 a score of one through one hundred, regardless of its size, 0:02:23.636 --> 0:02:27.996 or mission or complexity, an algorithm to capture the elusive 0:02:28.116 --> 0:02:30.316 quality of academic excellence. 0:02:31.036 --> 0:02:35.596 Finding those points of data and putting them into a ranking, 0:02:35.956 --> 0:02:42.116 as you can appreciate, is a complicated task. Yes, full 0:02:42.156 --> 0:02:50.076 of value, judgments, full of problems, challenges, dangers, But we 0:02:50.076 --> 0:02:53.876 were undaunted and precede it. 0:02:55.116 --> 0:02:56.356 I love that you're laughing about this. 0:02:56.916 --> 0:03:00.596 It was a very heady time, and we thought, and 0:03:00.716 --> 0:03:03.596 we did a rarely challenge time in news. We get 0:03:03.596 --> 0:03:04.076 their game. 0:03:06.276 --> 0:03:11.316 The rankings became an institution. This news surged, careers were forged, 0:03:11.636 --> 0:03:14.996 fortunes made. I mean, what's fasci today is that time 0:03:15.036 --> 0:03:19.036 and newsic have basically disappeared. The rest of US News 0:03:19.036 --> 0:03:21.996 has basically disappeared. But the rankings are as strong as ever. 0:03:22.716 --> 0:03:26.396 Well, it's nice to know in this world of effemera 0:03:26.516 --> 0:03:32.156 that something can last. But and as a journalist, you 0:03:32.196 --> 0:03:36.196 know who you know when your work is used for 0:03:36.196 --> 0:03:39.716 for old fish after a day, it's nice to see 0:03:39.716 --> 0:03:41.756 that something hasn't impact for a while. 0:03:44.236 --> 0:03:47.196 Peter left US News long ago, but he and his 0:03:47.236 --> 0:03:50.796 colleagues had added something new to the American academic calendar 0:03:51.196 --> 0:03:56.036 beyond freshman week, finals commencement. There's now the day the 0:03:56.116 --> 0:03:57.396 new rankings come out. 0:03:57.636 --> 0:04:00.996 The new thirtieth edition of Best Colleges was just released 0:04:00.996 --> 0:04:02.636 this morning, and we've got it. 0:04:02.756 --> 0:04:04.276 You're seeing the two thousand. 0:04:03.996 --> 0:04:07.276 And the list for twenty nineteen was Princeton University, followed 0:04:07.276 --> 0:04:08.556 by Harvard Tied. 0:04:09.116 --> 0:04:12.356 Scholars of America's best colleges have been revealed. The US 0:04:12.436 --> 0:04:14.876 News and World Report is out with its angle list, 0:04:14.956 --> 0:04:17.076 and we've got the inside scoop on how. 0:04:17.156 --> 0:04:20.916 The algorithm that's developed is undergoing a constant state of 0:04:20.956 --> 0:04:23.876 evolution over the years. How often are you tweaking it? 0:04:24.756 --> 0:04:27.156 I thought you might ask about this. I mean I 0:04:27.196 --> 0:04:30.676 did go online and look at it, and I was 0:04:30.676 --> 0:04:33.756 glad to see they are tweaking it on a regular basis, 0:04:33.796 --> 0:04:36.836 at least it looking if you look at their methodology. 0:04:37.116 --> 0:04:40.676 And of course they've also been reacting to competition. I mean, 0:04:41.116 --> 0:04:43.756 you know a lot of people have been critical of 0:04:43.796 --> 0:04:48.196 the rankings and whether this mythology really this methodology? 0:04:48.396 --> 0:04:51.796 Excuse me, that's it, Peter. That is like the definition 0:04:51.836 --> 0:04:55.596 of a Freudian slip. You're talking about your you're talking 0:04:55.636 --> 0:04:59.436 about your brain child, and you refer to it as mythology. 0:05:01.636 --> 0:05:05.716 That's true. I couldn't have come up with that one consciously. 0:05:06.876 --> 0:05:12.156 At what point did the US news methodology become a mythology? 0:05:13.196 --> 0:05:18.476 Let's find out. My name is Malcolm Gladwell. You're listening 0:05:18.556 --> 0:05:23.076 to my podcast, Revisionist History about Things Overlooked and Misunderstood. 0:05:25.316 --> 0:05:27.596 In this episode, I ask, how is it that a 0:05:27.636 --> 0:05:30.836 promotional gimmick from the nineteen eighties turned into a fetish 0:05:30.916 --> 0:05:39.076 object for American higher education? How do these rankings work 0:05:39.596 --> 0:05:42.796 and why does everyone take them so seriously? We're going 0:05:42.876 --> 0:05:46.276 to launch a revisionist history investigation, one that will take 0:05:46.356 --> 0:05:49.716 us to all kinds of strange places, to the bond markets, 0:05:50.116 --> 0:05:54.556 to Portland, Oregon, to New Orleans. We'll meet a mysterious 0:05:54.556 --> 0:05:58.596 college administrator who asked to be identified only as Dean, 0:05:59.236 --> 0:06:02.316 as well as a band of hackers who agreed to 0:06:02.356 --> 0:06:08.476 join our cause. All right, all right, so we're ready 0:06:08.516 --> 0:06:11.836 to go this very now. Before we start, can both 0:06:11.876 --> 0:06:14.716 of you guys introduce yourselves. 0:06:15.836 --> 0:06:19.476 So I'm Kelly McConville, and I'm an assistant professor of 0:06:19.516 --> 0:06:21.356 statistics at Reed College. 0:06:21.676 --> 0:06:25.396 I'm playing too, and I'm an international student who is 0:06:25.436 --> 0:06:28.676 currently studying at UNC's High school. 0:06:29.036 --> 0:06:31.996 So these are some of my accomplices, and here's why 0:06:32.036 --> 0:06:35.476 I needed them. Us News will tell you what variables 0:06:35.516 --> 0:06:38.516 they use to determine whether a school is good or mediocre, 0:06:38.876 --> 0:06:42.756 but they won't disclose exactly how they combine those variables 0:06:42.796 --> 0:06:46.396 into a score. I mean, why would they. It's their 0:06:46.476 --> 0:06:50.196 secret Sauce they charged forty dollars a year for premium 0:06:50.236 --> 0:06:53.916 access to their findings. This is a big business. So 0:06:53.956 --> 0:06:56.516 I figured my only chance at really getting under the 0:06:56.516 --> 0:06:58.996 hood of the ranking algorithm was to find out if 0:06:58.996 --> 0:07:03.516 anyone had in effect hacked the algorithm, and it turns 0:07:03.556 --> 0:07:06.956 out someone had. It was a group of students at Reed, 0:07:07.236 --> 0:07:10.596 a small liberal arts college in Portland, under the leadership 0:07:10.596 --> 0:07:15.156 of their professor Kelly mcconvill. Tell me how on earth 0:07:16.036 --> 0:07:19.396 this whole project of looking at US News rankings? How 0:07:19.396 --> 0:07:19.996 did it begin? 0:07:21.196 --> 0:07:21.356 So? 0:07:21.876 --> 0:07:26.476 I teach a course called Statistics Practicum, and in that course, 0:07:26.516 --> 0:07:31.356 the students complete a semester long empirical research project, and 0:07:31.636 --> 0:07:34.676 I solicit those projects and the data from other READ 0:07:34.756 --> 0:07:36.076 faculty and staff. 0:07:36.956 --> 0:07:39.556 Every year, US News asks for a lot of data 0:07:39.556 --> 0:07:43.876 from schools, but the Read administration had stopped supplying that data. 0:07:44.356 --> 0:07:47.676 They decided they didn't believe in the rankings anymore. But 0:07:47.796 --> 0:07:50.996 still the magazine kept generating a ranking for Read anyway, 0:07:51.476 --> 0:07:54.276 and someone in the Read administration came to Kelly McConville 0:07:54.276 --> 0:07:57.796 and said, ever since we dropped out, our US News 0:07:57.836 --> 0:08:01.996 ranking has plummeted. We think they've been punishing us. Can 0:08:01.996 --> 0:08:03.196 you double check their math? 0:08:03.996 --> 0:08:06.596 That was our task. We were tasked with recreating the 0:08:06.636 --> 0:08:10.076 model and then seeing what our ranking would be if 0:08:10.116 --> 0:08:11.156 we did supplier data. 0:08:11.396 --> 0:08:13.716 Yeah, So tell me, the two of you walk me 0:08:13.796 --> 0:08:18.396 through how does one go about conceptually trying to answer 0:08:18.476 --> 0:08:19.076 this question. 0:08:19.916 --> 0:08:22.196 They use a model that's called a weight in some 0:08:22.436 --> 0:08:27.116 model that means they take about twenty or so different 0:08:27.116 --> 0:08:30.916 factors about a school, like class size and graduation rates. 0:08:31.756 --> 0:08:34.996 They wait each of those factors based on how important 0:08:35.716 --> 0:08:39.036 they think the factor is, and then they sum up 0:08:39.036 --> 0:08:41.636 all those pieces and then calibrate it so that the 0:08:41.716 --> 0:08:44.436 highest scores at one hundred, and then based on those scores, 0:08:44.436 --> 0:08:45.716 they rank as schools. 0:08:47.036 --> 0:08:48.876 The Red team had to build their own version of 0:08:48.916 --> 0:08:52.356 the US News algorithm from scratch, trying to get as 0:08:52.476 --> 0:08:53.916 close to it as possible. 0:08:54.956 --> 0:08:59.276 We grabbed all the variables we could from open government sources, 0:09:00.076 --> 0:09:03.716 and we built a model that isn't trying to recreate 0:09:03.836 --> 0:09:07.516 the US newswait in some model. Instead, it's just trying 0:09:07.516 --> 0:09:10.316 to predict the US New score of a school with 0:09:10.356 --> 0:09:11.716 a high degree of accuracy. 0:09:13.236 --> 0:09:14.916 Whying how close did you get? 0:09:15.796 --> 0:09:20.316 I'll say fairly close. The only school that's outside of 0:09:20.476 --> 0:09:23.596 like a reasonable range of predictive score in the test 0:09:23.636 --> 0:09:26.316 status read oh really, And all other schools are like 0:09:26.916 --> 0:09:32.236 within a reasonable prediction interval of their scores. 0:09:33.196 --> 0:09:37.476 Meaning the Reed students identical twin algorithm is so good 0:09:37.796 --> 0:09:40.596 that when they run their rankings of American colleges. Their 0:09:40.636 --> 0:09:44.996 list looks almost exactly like the US News list, with 0:09:45.036 --> 0:09:49.996 one exception Read College. Wait, so they were messing. 0:09:49.676 --> 0:09:54.196 With you based on our finding, it seems reasonable they 0:09:54.316 --> 0:09:54.916 this something. 0:09:56.476 --> 0:10:01.356 In twenty nineteen, US News put Read in ninetieth place. 0:10:02.596 --> 0:10:04.756 Where do you think you should be ranked? 0:10:06.596 --> 0:10:08.796 Thirty sixth Oh? 0:10:09.316 --> 0:10:13.836 Wait? What that much of a difference? Yeah, that's bananas. 0:10:14.316 --> 0:10:17.596 Oh my, oh, they really did a number on Reed. 0:10:21.436 --> 0:10:23.956 Now we don't need to go much further into the 0:10:24.036 --> 0:10:27.756 ensuing beef between Read and US News. The magazine calls 0:10:27.796 --> 0:10:33.236 the undergraduates analysis quote inaccurate. But just imagine my joy 0:10:33.236 --> 0:10:36.876 when I ran across Swaying's analysis. Here. I was searching 0:10:36.916 --> 0:10:40.276 for some way to understand the secret algorithm cooked up 0:10:40.316 --> 0:10:43.036 by Peter Bernstein and his pals back in the nineteen eighties, 0:10:43.356 --> 0:10:46.356 and lo and behold, a bunch of undergraduates at Reed 0:10:46.436 --> 0:10:50.396 College had cracked the code. And were they motivated to 0:10:50.436 --> 0:10:56.836 help me? Absolutely? Because US News dissed their school. My 0:10:56.916 --> 0:11:00.116 first question to Kelly and her team was where should 0:11:00.116 --> 0:11:03.716 I start? And they said start with a little something 0:11:04.036 --> 0:11:07.876 called the peer Assessment score. Can you give me a 0:11:07.876 --> 0:11:10.756 more precise sort of statistic sense of just how important 0:11:10.956 --> 0:11:11.236 is it? 0:11:12.076 --> 0:11:16.916 Like, okay, So we find that the pure assessment score 0:11:17.516 --> 0:11:22.276 has the largest effects, So it has a coefficient of 0:11:22.356 --> 0:11:26.116 like six point five eight. In other words, like, if 0:11:26.116 --> 0:11:30.316 you hold everything else constant and you like increase the 0:11:30.396 --> 0:11:35.516 peer assessment scorer by one unit, your overall score will 0:11:35.556 --> 0:11:38.476 increase by six point five to eight. 0:11:39.516 --> 0:11:42.836 The peer assessment score was at the heart of the 0:11:42.916 --> 0:11:47.276 US News secret sauce, a measure of reputation, one of 0:11:47.356 --> 0:11:51.076 the most powerful elements in the whole ranking system. So 0:11:51.196 --> 0:11:55.836 if I'm a school and I'm anxious to move up 0:11:55.876 --> 0:11:57.916 the US News rankings, the thing I should really be 0:11:57.996 --> 0:12:00.716 focused on is trying to boost my reputation score. 0:12:02.116 --> 0:12:02.356 Yeah. 0:12:02.596 --> 0:12:06.636 Yeah, I had my marching orders, and I knew where 0:12:06.676 --> 0:12:09.516 I had to go. Next to the guardian of the 0:12:09.676 --> 0:12:13.036 US newsgates that dragon at the mouth of the cave. 0:12:14.236 --> 0:12:28.116 Robert Morse. Robert Morse, the man who runs the US 0:12:28.236 --> 0:12:33.196 News rankings, is an older man, white hair, very preppy, nice, 0:12:33.516 --> 0:12:37.796 if a bit lugubrious, heavy hangs the head that where's 0:12:37.836 --> 0:12:41.436 the crown? I called him up. I wanted to understand 0:12:41.876 --> 0:12:45.956 the mysterious variable that the read College hackers had told 0:12:45.996 --> 0:12:49.956 me was so important? Describe again, how is the reputation 0:12:50.196 --> 0:12:51.876 score generated? 0:12:52.636 --> 0:12:57.316 So we send out three surveys per school, and then 0:12:57.356 --> 0:13:00.916 they respond. They rate the schools on a scale of 0:13:01.076 --> 0:13:03.476 five to one or no response. 0:13:04.076 --> 0:13:06.196 So who are the three people at each school who 0:13:06.596 --> 0:13:09.316 are asked to fill out the survey. 0:13:09.356 --> 0:13:13.796 In the provos and the enrollment manager or the head 0:13:13.796 --> 0:13:14.676 of admissions. 0:13:15.516 --> 0:13:18.916 The surveys arrive via email once a year to everyone 0:13:18.916 --> 0:13:21.796 who matters in American Higher ed. If you're at a 0:13:21.836 --> 0:13:24.236 liberal arts college, you're given a list of all two 0:13:24.356 --> 0:13:28.356 hundred and twenty two other liberal arts colleges in your category. 0:13:28.436 --> 0:13:30.796 If you're part of a big university, you get a 0:13:30.836 --> 0:13:33.756 list of the three hundred and eighty eight other big 0:13:33.876 --> 0:13:37.556 universities in the United States. Your job is to rank 0:13:37.596 --> 0:13:39.916 the schools on your list on a scale of one 0:13:39.956 --> 0:13:45.316 to five, five being amazing, one being there's a serious 0:13:45.316 --> 0:13:50.196 problem here. And on what basis are people making those judgments? 0:13:50.756 --> 0:13:53.476 Well, that's a good question. I mean, we ask them 0:13:53.516 --> 0:13:56.996 to base it on their view of the school's reputation 0:13:57.196 --> 0:13:59.676 for undergraduate academic quality. 0:14:00.076 --> 0:14:01.236 But I mean, how would you know. 0:14:03.676 --> 0:14:07.276 I mean, we believe that to aggregate sum of all 0:14:07.356 --> 0:14:11.116 the raiders who are a leader is in higher education. 0:14:11.996 --> 0:14:14.516 You know their presidents and provosts and mission deans. So 0:14:14.596 --> 0:14:18.356 they're not rookies. When you aggregate their views at any 0:14:18.396 --> 0:14:24.996 given time, they're representing where those schools stand in the marketplace. 0:14:25.716 --> 0:14:29.836 I mean, so it's it's an assessment of their feelings 0:14:30.396 --> 0:14:31.276 what other schools. 0:14:33.036 --> 0:14:36.716 Well, they're leaders in higher education, so it's based on 0:14:36.836 --> 0:14:42.756 their knowledge of the other schools that they've gathered or 0:14:42.796 --> 0:14:48.516 through meetings or for exposure. Do they necessarily know a 0:14:48.556 --> 0:14:51.956 lot about all the schools they're rating. I don't know. 0:14:52.596 --> 0:14:55.556 If I rate a restaurant on Yelp, it's because I've 0:14:55.556 --> 0:15:03.036 eaten there, right, yes, But here, when I rate a 0:15:03.196 --> 0:15:06.396 university on the US News ranking, it's not because I've 0:15:06.636 --> 0:15:08.396 attended that university or taught there. 0:15:09.276 --> 0:15:12.436 It's just, well, you may have. We shouldn't say that 0:15:12.516 --> 0:15:15.356 you didn't. But you're saying, have the people who rated 0:15:15.396 --> 0:15:19.396 the schools been on every campus or taught at that 0:15:19.516 --> 0:15:22.756 school or no? In depth their course catalog? 0:15:23.036 --> 0:15:23.116 No. 0:15:23.716 --> 0:15:26.836 Do some of them have greater degrees of knowledge about 0:15:26.836 --> 0:15:29.996 the schools they're rating, yes? Do some of them have 0:15:30.236 --> 0:15:34.036 lesser degrees of knowledge? Yes? So I think it varies 0:15:34.116 --> 0:15:38.276 from having substantial knowledge to far less knowledge. 0:15:39.156 --> 0:15:41.516 In my naivete, I had imagined that there would be 0:15:41.556 --> 0:15:43.956 some level of rigor to this crucial variable in the 0:15:44.036 --> 0:15:47.316 US news wait, in some model a methodology that I 0:15:47.316 --> 0:15:53.196 could study, analyze, learn from. But Morse wasn't being very helpful, 0:15:53.756 --> 0:15:56.276 so I tried to draw him out. I mean, I 0:15:56.356 --> 0:16:01.716 was fascinated that there's a little clump where BYU Bring 0:16:01.756 --> 0:16:06.716 them Young, Yeshiva and Gonzaga are all ranked right in 0:16:06.756 --> 0:16:10.796 the same roughly equally in the National University's rank. So 0:16:10.876 --> 0:16:15.356 a Mormon school, Conservative Jewish school, and the Jesuit school. 0:16:16.236 --> 0:16:20.516 And I was trying to think, what is the rabbi 0:16:20.596 --> 0:16:23.076 who runs Yeshiva know about bring them Young? And what 0:16:23.116 --> 0:16:28.556 does the Jesuit priest who runs Gonzaga know about Yeshiva? 0:16:28.636 --> 0:16:33.596 And it's a good point, like is Gonzaga giving George 0:16:33.596 --> 0:16:36.396 ten of five and giving Boston College a five? 0:16:36.516 --> 0:16:40.196 And we're not looking at how people are rating other schools. 0:16:40.356 --> 0:16:43.316 We are doing a trim means, so we take off 0:16:43.356 --> 0:16:46.556 a few of the highest ratings and a few of 0:16:46.556 --> 0:16:51.956 the lowest ratings in an effort to eliminate some of 0:16:51.996 --> 0:16:57.036 the strategic voting that you're alluding to well, I. 0:16:56.996 --> 0:16:59.076 Don't know if that's strategic voting. I'm wondering if it's 0:16:59.116 --> 0:17:04.796 just that. If I'm president of Gonzaga, I'm powerfully motivated 0:17:04.836 --> 0:17:07.996 to believe that the Jesuit education is the best education 0:17:08.076 --> 0:17:11.076 out there. So when I look at Georgetown, I think 0:17:11.316 --> 0:17:14.876 they're doing the right thing. And when I look at Yeshiva, 0:17:14.876 --> 0:17:18.796 I say, where's the New Testament? Why are they not 0:17:18.876 --> 0:17:21.116 using the New Testament? I mean, this is just natural 0:17:21.236 --> 0:17:24.116 human biases. I just don't know what they have to 0:17:24.116 --> 0:17:25.476 do with the quality of the institution. 0:17:29.836 --> 0:17:32.396 Well, I don't have a different answer. 0:17:35.956 --> 0:17:39.396 Devoted listeners of revisionist history. Well, remember how years ago 0:17:39.716 --> 0:17:43.876 I fell for Rowan University in New Jersey. I love Rowin, 0:17:44.596 --> 0:17:48.036 But Rowan is ranked one hundred and eighty seventh in 0:17:48.116 --> 0:17:53.596 the National University's list dismal mediocre, and why because its 0:17:53.636 --> 0:17:58.436 peer assessment score is a forgettable two point four? How 0:17:58.476 --> 0:18:01.636 on earth have the University presidence of America come to 0:18:01.676 --> 0:18:05.836 the collective decision that they hate Rowan University way way 0:18:05.836 --> 0:18:09.396 down in South Jersey. I call up Ali hush Mind, 0:18:09.636 --> 0:18:13.676 the school's president. How many university presidents have visited Roan 0:18:13.716 --> 0:18:16.116 in the last five years from other colleges. 0:18:16.236 --> 0:18:21.756 Oh, two or three, one from Canada, one came who 0:18:21.916 --> 0:18:24.996 was part of an accreditation and I believe another one 0:18:25.076 --> 0:18:25.556 was a friend. 0:18:26.556 --> 0:18:30.956 That's it. Yes, So when they're giving you a score, 0:18:31.196 --> 0:18:34.036 on what basis are they using forgiving you? How do 0:18:34.116 --> 0:18:35.476 they know anything at all about Roan? 0:18:35.916 --> 0:18:36.636 Again, I don't know. 0:18:36.756 --> 0:18:39.156 I can only speak for myself. I can assure you 0:18:39.196 --> 0:18:43.876 that I really don't know much about the institutions that 0:18:43.876 --> 0:18:45.636 I'm asked to rank or score. 0:18:45.716 --> 0:18:47.276 I just don't have enough tools. 0:18:48.396 --> 0:18:52.596 Now. It's not like university presidents don't know anything about grading. 0:18:53.076 --> 0:18:55.956 They were all teachers themselves once. But when it comes 0:18:55.956 --> 0:18:59.516 to grading schools, it seems like they might just be 0:18:59.596 --> 0:19:02.476 mailing it in. Next, I called up the head of 0:19:02.516 --> 0:19:06.116 admissions at a major university. You've heard of it, trust me, 0:19:06.636 --> 0:19:08.516 but I can't tell you what it is. And we 0:19:08.596 --> 0:19:11.756 altered the voice to protect this person's identity. Of course 0:19:11.796 --> 0:19:15.076 we did. We had to look how US News treated 0:19:15.116 --> 0:19:19.276 read college. My source told me to address them only 0:19:19.756 --> 0:19:20.316 as dean. 0:19:20.996 --> 0:19:25.556 I am certain and I am far more qualified to 0:19:25.556 --> 0:19:28.796 be a member of the Baseball Writers Association of America 0:19:28.996 --> 0:19:31.356 and vote on the Hall of Fame than I am 0:19:31.516 --> 0:19:34.676 to assess other institutions for the US News and World Report. 0:19:35.076 --> 0:19:35.276 Yeah. 0:19:35.476 --> 0:19:35.676 Yeah. 0:19:36.316 --> 0:19:39.476 I proposed a Dean that we do an exercise. I'd 0:19:39.516 --> 0:19:42.556 name a school at random, and then the two of 0:19:42.636 --> 0:19:48.156 us would try to peer assess it. Florida State. Never 0:19:48.196 --> 0:19:57.036 been there. I've not been there, I think three point two. Syracuse. Yes, 0:19:57.636 --> 0:20:00.156 I I stopped for coffee there once in a way 0:20:00.236 --> 0:20:02.756 driving back to my mom's house in Canada. I don't 0:20:02.756 --> 0:20:04.836 know if that counts. I don't think it does. I think 0:20:05.196 --> 0:20:07.796 I think it counts account experience. I got to say 0:20:07.836 --> 0:20:11.476 experience the weather, but there's no there were no students there. 0:20:11.476 --> 0:20:15.276 It was Christmas. But I will say this, my cousin 0:20:15.356 --> 0:20:17.796 went to Syracuse and had a good time. 0:20:19.036 --> 0:20:22.076 Oh h and so you would rate it based on that. 0:20:23.036 --> 0:20:26.196 Let's give Syracuse some love. Three point nine, No, four 0:20:26.236 --> 0:20:32.756 point zero. University of Washington in Seattle. Yes, you've been there, 0:20:33.436 --> 0:20:35.436 Yes I have. What'd you do when you were there? 0:20:38.236 --> 0:20:39.236 I walked around? 0:20:39.436 --> 0:20:40.276 I think that counts. 0:20:41.076 --> 0:20:42.276 It counts. 0:20:42.796 --> 0:20:45.316 I said to Dean, you can see the mountains from there, 0:20:45.436 --> 0:20:50.116 the space needle. Let's go big. University of Washington at 0:20:50.116 --> 0:20:55.516 Seattle four point two. Have you ever been to William Mary. 0:20:56.716 --> 0:20:58.436 I have yeah. 0:20:58.476 --> 0:21:01.796 I love history, I love colonial Williamsburg. 0:21:02.116 --> 0:21:04.436 Do you remember what score you gave William and Mary. 0:21:06.716 --> 0:21:09.436 I probably gave them a four or five. 0:21:09.996 --> 0:21:13.156 Really you went for five. It's the second oldest college 0:21:13.156 --> 0:21:15.356 in the US. Do you think that affected your judgment 0:21:15.396 --> 0:21:15.676 on that? 0:21:17.036 --> 0:21:17.956 I'm sure it did. 0:21:19.196 --> 0:21:21.476 Dean and I went on for quite some time. We 0:21:21.516 --> 0:21:24.356 had a long chat about Southern Methodist University in Dallas 0:21:24.476 --> 0:21:26.796 and whether or not the great football team the school 0:21:26.876 --> 0:21:29.316 had in the nineteen eighties should count in their favor. 0:21:29.876 --> 0:21:36.156 Dean said no. I said yes. Now was all this 0:21:36.196 --> 0:21:40.036 a helpful exercise? Not really? But then again, now that 0:21:40.236 --> 0:21:43.876 was my conversation with the Wizard of the Rankings, Robert Morse. 0:21:44.756 --> 0:21:46.956 One suggestion that has been made to me by people 0:21:46.996 --> 0:21:49.956 who do the rankings is that the thing they rely 0:21:50.076 --> 0:21:54.996 on to make their reputation scores for other universities is 0:21:55.076 --> 0:21:58.476 the only source they have of reputation scores for the 0:21:58.476 --> 0:22:02.916 other universities, which is the previous year's US News rankings. 0:22:03.356 --> 0:22:04.636 Is this circular? 0:22:05.236 --> 0:22:08.956 If they're telling you that, then that's not our expectation 0:22:09.276 --> 0:22:12.796 of how people are doing their ratings. Versus looking at 0:22:12.796 --> 0:22:17.356 the ranking table and looking at the score. We believe 0:22:17.516 --> 0:22:21.956 that there's more thought going into it than that, but 0:22:22.076 --> 0:22:24.956 we haven't done any kind of social science research to 0:22:26.196 --> 0:22:28.196 prove or disprove that point. 0:22:28.276 --> 0:22:32.796 You haven't done any right, Yeah, they haven't done any 0:22:32.876 --> 0:22:38.836 social science research. Only one person gave me much insight 0:22:38.916 --> 0:22:42.116 into the mechanics of the peer Assessment score, and that 0:22:42.236 --> 0:22:46.476 was Ali Huschmand, president of Rowan. His advice was, you 0:22:46.596 --> 0:22:49.676 have to take matters into your own hands. So what 0:22:49.756 --> 0:22:52.916 does he do? He makes his own hot sauce. And 0:22:52.996 --> 0:22:55.676 he told me that Rowan sends out free samples to 0:22:55.916 --> 0:23:00.076 every college president in the country. That's not too really 0:23:00.676 --> 0:23:01.916 he did, and he. 0:23:01.956 --> 0:23:07.156 Did no that I make hot sauce. Well, that would 0:23:07.236 --> 0:23:08.476 have been that's scoring. 0:23:08.556 --> 0:23:08.956 I hope. 0:23:09.516 --> 0:23:11.636 Yes. I want to know about your hot sauce. What 0:23:11.716 --> 0:23:12.676 kind of hot sauce do you have? 0:23:13.196 --> 0:23:13.356 Oh? 0:23:13.436 --> 0:23:16.156 I make serious, serious hot sauce. But I actually grow 0:23:16.236 --> 0:23:19.356 from seeds all the way to the packaging. And there 0:23:19.356 --> 0:23:22.716 are three levels of heat. The lowest one is called 0:23:22.756 --> 0:23:24.596 Ali's nasty, which is. 0:23:24.596 --> 0:23:25.276 Kind of mind. 0:23:26.076 --> 0:23:28.236 And then the one next to it, it happened, was 0:23:28.276 --> 0:23:32.156 as a heat it's called a nasty delicious, And then 0:23:32.196 --> 0:23:37.596 the one with Carona Reaper is called nasty vicious, and 0:23:37.716 --> 0:23:40.316 I will send you a package the nasty divisious can 0:23:40.396 --> 0:23:40.756 get you. 0:23:41.596 --> 0:23:43.876 But this isn't gonna help your cause. This is gonna 0:23:43.916 --> 0:23:46.876 hurt your cause. If you're if you're giving some some 0:23:47.116 --> 0:23:49.676 university president in the middle of the Midwest, it's like 0:23:49.916 --> 0:23:52.396 it happens to have a little bit of vicious. They're 0:23:52.396 --> 0:23:54.116 gonna they're gonna curse your name. 0:23:54.716 --> 0:23:57.836 Probably will, but I but I figured that no, no 0:23:57.916 --> 0:23:59.676 publicity is about bad publicity. 0:24:01.036 --> 0:24:02.716 Did I try the nasty vicious? 0:24:02.996 --> 0:24:03.156 Oh? 0:24:03.276 --> 0:24:05.756 Yes, I did, on a scale of one to five 0:24:06.836 --> 0:24:18.636 five point oh. In the middle of my investigation of 0:24:18.676 --> 0:24:22.236 the pure assessment score and its many implications, I had 0:24:22.276 --> 0:24:25.636 a conversation with a professor at Duke named Bill Mayu. 0:24:26.516 --> 0:24:29.196 Mayu is friendly with some bond traders, and he was 0:24:29.276 --> 0:24:31.556 talking with some of them one day when they mentioned 0:24:31.556 --> 0:24:34.636 that they were having difficulty with bonds issued by historically 0:24:34.676 --> 0:24:38.116