WEBVTT - What Claude Shannon Figured Out 0:00:15.356 --> 0:00:15.796 Pushkin. 0:00:20.436 --> 0:00:23.036 Hey, Happy New Year. We're very happy to be back, 0:00:23.396 --> 0:00:26.836 and i have one request before we start the show. 0:00:26.916 --> 0:00:29.596 I'm asking you a favor, and the favor is this, 0:00:30.156 --> 0:00:35.396 would you please send us an email to problem at 0:00:35.396 --> 0:00:39.756 pushkin dot fm and tell us what you like about 0:00:39.796 --> 0:00:41.876 the show and what you don't like about the show, 0:00:41.916 --> 0:00:44.356 and specifically what kinds of things you want to hear 0:00:44.396 --> 0:00:47.236 more of, and perhaps what kinds of things you don't 0:00:47.236 --> 0:00:51.876 want to hear Again, it's problem at Pushkin dot fm. 0:00:51.956 --> 0:00:54.236 I'm going to read all the emails, so thank you 0:00:54.356 --> 0:01:00.196 in advance for sending them. Claude Shannon is this huge 0:01:00.236 --> 0:01:02.796 figure in the history of technology. He's one of the 0:01:02.876 --> 0:01:05.836 key people who worked at Bell Labs in the middle 0:01:05.836 --> 0:01:08.796 of the twentieth century and really came up with the 0:01:08.796 --> 0:01:13.836 idea that made modern technology possible. But I'm going to 0:01:13.876 --> 0:01:18.116 be honest with you, I never really understood what Claude 0:01:18.116 --> 0:01:21.516 Shannon figured out that was such a big deal. But 0:01:21.556 --> 0:01:23.836 the people who know about technology, who know about the 0:01:23.916 --> 0:01:28.076 history of ideas, they say Shannon's a giant. Claude Shannon 0:01:28.196 --> 0:01:32.476 is like the nerds. Nerd he's the techno intellectuals, techno 0:01:32.516 --> 0:01:36.676 intellectual and so For today's show, I wanted to understand 0:01:37.596 --> 0:01:40.316 what did Claude Shannon figure out and why is it 0:01:40.436 --> 0:01:41.836 so important. 0:01:41.556 --> 0:01:42.596 For the modern world. 0:01:48.876 --> 0:01:51.676 I'm Jacob Goldstein, and this is What's your problem. My 0:01:51.716 --> 0:01:55.556 guest today is David Shay. David is a professor of 0:01:55.716 --> 0:02:00.436 electrical engineering at Stanford. He has studied Shannon for decades. 0:02:00.676 --> 0:02:05.196 He teaches Shannon's work to his students, and David used 0:02:05.236 --> 0:02:08.996 Shannon's work to make a breakthrough in cell phone technology. 0:02:09.516 --> 0:02:12.396 And that breakthrough, that breakthrough that came to us via 0:02:12.516 --> 0:02:15.996 Shannon and Shay, it affects every phone call we make. 0:02:17.156 --> 0:02:20.436 David and I talked about Shannon's key insights and about 0:02:20.436 --> 0:02:23.716 how David's own work built on Shannon, and we also 0:02:23.756 --> 0:02:26.196 talked about the big chunk of Shannon's life that was 0:02:26.196 --> 0:02:30.836 taken up with juggling and riding unicycles and building mechanical toys. 0:02:31.516 --> 0:02:34.436 But to start, we talked about how in the middle 0:02:34.476 --> 0:02:37.796 of the twentieth century, Bell Labs wound up driving so 0:02:37.996 --> 0:02:39.756 much technological innovation. 0:02:42.516 --> 0:02:47.036 Yeah, so Bell Labs was the research lab of AT 0:02:47.156 --> 0:02:52.396 and T. Aightenh at that time was the phone company. Okay, 0:02:52.476 --> 0:02:55.796 nowadays we have many phone companies. We have Verizon, we 0:02:55.916 --> 0:03:00.476 have T Mobile, et cetera. But those days there was 0:03:00.516 --> 0:03:04.756 only one phone company, and that's a monopoly. So a 0:03:04.836 --> 0:03:07.476 monopoly needs to justify its existence. 0:03:07.676 --> 0:03:10.316 Huh. So it doesn't get broken by the government. 0:03:10.356 --> 0:03:12.716 It doesn't get broken up. Of course, it eventually got 0:03:12.716 --> 0:03:15.436 broken up, but at that time it was a monopoly. 0:03:16.396 --> 0:03:21.076 And so one way of justifying its existence is to 0:03:21.116 --> 0:03:24.636 say that. Okay, he says to the American people, to 0:03:24.676 --> 0:03:29.436 the government, that we will always spend a certain percent 0:03:29.676 --> 0:03:34.916 of our revenue on this research lab called bow Labs, 0:03:35.636 --> 0:03:38.876 and whatever bow Labs come up with is kind of 0:03:38.916 --> 0:03:42.916 our contribution, know only to our bottom line, but also 0:03:43.796 --> 0:03:46.916 to technology of the country. 0:03:47.316 --> 0:03:49.796 So they have this sort of public mission to prevent 0:03:49.836 --> 0:03:51.436 the government from breaking them up. 0:03:52.196 --> 0:03:56.796 Yeah, and so therefore it also allows researchers a very 0:03:56.836 --> 0:04:02.076 free reign to do research that not necessarily tied to, 0:04:02.276 --> 0:04:06.796 like say, a particular business unit. Okay, So they can 0:04:06.836 --> 0:04:10.196 be very creative. And that's the atmosphere about so bad 0:04:10.236 --> 0:04:14.036 Labs attracted a bunch of very smart people because smart 0:04:14.036 --> 0:04:16.196 people wants to work on their own problem, not the 0:04:16.236 --> 0:04:19.876 problem that the manager gives them. Yeah, okay, that's the 0:04:20.236 --> 0:04:23.476 that's one characteristic of smart people. And so yeah, that 0:04:23.636 --> 0:04:26.756 was the heydays of Bat Labs. Lots of smart people 0:04:27.036 --> 0:04:32.556 inventing amazing stuff. Laser was invented there, information theory, the 0:04:32.636 --> 0:04:36.836 transistor was invented there. Sort of almost all the foundation 0:04:37.236 --> 0:04:43.716 of the information age. Yeah, where there's hardware, algorithm, software 0:04:44.516 --> 0:04:47.076 is in some sense all have the roots at Ball Labs. 0:04:47.756 --> 0:04:50.836 So that was the contribution to mankind. Actually, I should say, no, 0:04:50.876 --> 0:04:52.716 only to America. 0:04:52.956 --> 0:04:56.596 So Shannon gets there at this time, right, he's there 0:04:56.636 --> 0:05:00.556 with you know, when they're inventing certainly the transistor. 0:05:01.956 --> 0:05:02.476 What's he do? 0:05:02.636 --> 0:05:04.956 Tell me about his his work there? When he gets there, 0:05:05.236 --> 0:05:06.036 what's he working on? 0:05:07.676 --> 0:05:14.196 Yeah? So I think Shannon always have his own agenda, right. 0:05:14.796 --> 0:05:17.316 We know for a fact that he has been interested 0:05:17.516 --> 0:05:22.996 in the problem of communication, that idea of having a 0:05:22.996 --> 0:05:27.156 grand theory of communication, even back in nineteen thirty eight, 0:05:27.276 --> 0:05:29.476 I think thirty seven thirty eight, because he wrote a 0:05:29.556 --> 0:05:33.036 letter at that time to a very famous person named 0:05:33.076 --> 0:05:37.796 Venera Bush. Yeah. Vera Bush is very famous, as he 0:05:37.916 --> 0:05:40.596 was I think president of MIT or dean of MIT, 0:05:41.316 --> 0:05:43.796 and then he became sort of a scientific advisor to 0:05:43.836 --> 0:05:47.636 the president, and so he wrote a letter to Venera 0:05:47.676 --> 0:05:49.716 Bush in nineteen thirty eight and say, hey, you know what, 0:05:49.756 --> 0:05:52.996 I'm really interested in this question of how to find 0:05:53.076 --> 0:05:56.356 one theory that unifies all possible communication systems. There's so 0:05:56.396 --> 0:05:58.716 many different communications system out there, but I think there's 0:05:58.716 --> 0:06:01.716 something at the heart of every system, and I'm trying 0:06:01.756 --> 0:06:02.516 to get to the heart. 0:06:02.716 --> 0:06:05.676 And like nobody had thought of it in that way, right. 0:06:05.716 --> 0:06:09.316 It seems like part of his part of what Shannon's 0:06:09.356 --> 0:06:11.236 such a big deal is like as I understand that 0:06:11.276 --> 0:06:13.636 people it was like, you know that people understood like 0:06:13.756 --> 0:06:15.356 they were trying to figure out how to make the 0:06:15.396 --> 0:06:17.796 phone work better, and they were trying to, you know, 0:06:18.076 --> 0:06:20.876 make movies be clearer or whatever. But there wasn't this 0:06:20.996 --> 0:06:24.796 idea that you could abstract it until Shannon came along. 0:06:25.836 --> 0:06:29.396 And the reason is very simple, actually, because if you 0:06:29.476 --> 0:06:31.596 have a physical system, then you want to build right 0:06:32.516 --> 0:06:34.596 what do you see right? You say, hey man the 0:06:35.036 --> 0:06:37.836 video for example, i'm seeing you right now, I'm not 0:06:37.836 --> 0:06:39.756 seeing you very clearly, have to say yes. 0:06:40.956 --> 0:06:42.836 I'm in a closet, a closet. 0:06:42.956 --> 0:06:46.116 Right Then I would say, how to try to improve 0:06:46.156 --> 0:06:48.236 the imbage? Maybe I can try to, you know, fix 0:06:48.276 --> 0:06:52.476 this pixel or do some filtering of your noise. So 0:06:52.756 --> 0:06:56.556 I'm very tied to the very specific details of the 0:06:56.636 --> 0:06:59.356 specific problem. Because why I'm the engineer. I need to 0:06:59.396 --> 0:07:03.196 improve the system, not in ten years, but tomorrow. You know, tomorrow. 0:07:03.796 --> 0:07:05.836 You don't need a theory of the system. You just 0:07:05.876 --> 0:07:07.116 want to creer picture. 0:07:07.236 --> 0:07:10.316 Yeah, yeah, I mean I'm in the weeds, right, I'm 0:07:10.356 --> 0:07:15.036 in the weeds. And Shannon, because of his training, and 0:07:15.116 --> 0:07:18.276 also because of the atmosphere of a place like bout Labs, 0:07:18.756 --> 0:07:22.116 could afford to st step back and just look at 0:07:22.116 --> 0:07:25.796 the broader forest as opposed to the details of specific trees. 0:07:26.876 --> 0:07:30.596 So so, okay, Shannon's big idea comes out in this 0:07:30.716 --> 0:07:34.076 paper he publishes in nineteen forty eight. The paper is 0:07:34.076 --> 0:07:38.236 called a Mathematical Theory of Communication. It's like his great work. 0:07:38.716 --> 0:07:39.796 Tell me about that paper. 0:07:40.756 --> 0:07:43.676 So that paper is actually a very interesting paper. In fact, 0:07:44.036 --> 0:07:49.316 when I teach information theory, I teach from the paper itself, 0:07:49.556 --> 0:07:53.276 because I thought it's an amazing way not only of 0:07:53.676 --> 0:07:56.196 learning information theory, but learning how to write a scientific 0:07:56.236 --> 0:08:01.356 paper properly. Huh okay. And you know, not everyone does 0:08:01.396 --> 0:08:04.836 research and information theory, but everybody has to write uh 0:08:04.916 --> 0:08:08.476 huh okay. Every researcher has the right to express their 0:08:08.516 --> 0:08:12.076 ideas to the peers and to the audience. So in 0:08:12.156 --> 0:08:15.916 that paper, very interesting. The first paragraph of the paper. Okay, 0:08:15.956 --> 0:08:19.036 it's already very interesting because typically when people write a 0:08:19.036 --> 0:08:22.236 paper nowadays, they tell you, oh, how great my invention is. 0:08:22.476 --> 0:08:24.716 It's going to change the world. Every paper is going 0:08:24.796 --> 0:08:27.996 to change the world. But in fact, his first paper 0:08:28.116 --> 0:08:31.396 paragraph focused on telling you what his paper is not achieving. 0:08:31.836 --> 0:08:36.596 Ha ha, I mean that's a master. That's the masters, right, 0:08:36.836 --> 0:08:39.756 I mean, how many papers that you read nowadays tells 0:08:39.796 --> 0:08:43.436 you in the beginning, Hey, you know what, guys, expectation 0:08:43.596 --> 0:08:47.636 management here, this paper is not about this. Hey, don't 0:08:47.636 --> 0:08:50.596 get your home. Yeah, exactly, That's exactly what he did. 0:08:50.836 --> 0:08:55.036 Expectation management nowadays, that's what today we will call it 0:08:55.076 --> 0:08:58.596 expectation management. And now those days, I guess he just 0:08:58.636 --> 0:09:02.916 calls it honesty. And his whole point was often people 0:09:03.036 --> 0:09:08.836 associate information with meaning, okay, and then he said in 0:09:08.876 --> 0:09:14.916 this we ignore meaning, we ignore meaning. Huh okay. So 0:09:15.036 --> 0:09:18.076 that was the first thing he did, which is brilliant 0:09:18.396 --> 0:09:22.476 because once you high information with meaning, then he will 0:09:22.476 --> 0:09:24.596 never be able to make any progress. It's just too 0:09:24.636 --> 0:09:27.076 difficult and too broad and too vague a problem. 0:09:27.716 --> 0:09:30.316 Everybody gets stuck on this idea of meaning and what 0:09:30.476 --> 0:09:33.756 is meaning? And he's like, forget about meaning. So we're 0:09:33.756 --> 0:09:36.476 gonna forget about meaning. What is left? 0:09:37.876 --> 0:09:41.236 Yes, Actually, the biggest I think breakthrough of that paper 0:09:41.996 --> 0:09:46.196 is to really focus on the thing that matters and 0:09:46.876 --> 0:09:50.676 cut away a lot of stuff that really doesn't learn 0:09:50.756 --> 0:09:53.436 that it doesn't matter, but it doesn't matter in terms 0:09:53.436 --> 0:09:58.676 of solving the communication problem, the communication. So then he said, okay, 0:09:58.716 --> 0:10:01.996 what is the communication problem? The communication problem is the 0:10:02.076 --> 0:10:06.676 following is that there are multiple possibilities of a word, 0:10:07.636 --> 0:10:11.436 and my goal is to tell the receiver destination which 0:10:11.676 --> 0:10:14.356 of the multiple pospiity is the correct prosperity. 0:10:15.076 --> 0:10:19.116 Yeah, and so in language, it's basically it's a finite set. 0:10:19.236 --> 0:10:21.636 Language is a finite set. It's very large. But if 0:10:21.636 --> 0:10:24.716 we're speaking and we both know that we're speaking English, 0:10:25.116 --> 0:10:28.676 then essentially you are hearing the words and decoding them, 0:10:28.836 --> 0:10:30.716 and you know that it is a series of words, 0:10:30.716 --> 0:10:33.596 and you just have to figure out which words I 0:10:33.636 --> 0:10:37.596 mean like that for example, Yes, like that? Okay, so 0:10:37.796 --> 0:10:39.476 that's the frame he builds then. 0:10:39.356 --> 0:10:45.996 Why okay, all right, Then once you have this framing, right, 0:10:46.436 --> 0:10:49.316 then you can ask the question, Okay, what is the 0:10:49.316 --> 0:10:53.396 goal of communication? The goal of communication is to communicate 0:10:53.836 --> 0:10:59.676 as fast as I can, right, And the natural question 0:10:59.876 --> 0:11:04.156 is why is there a limit on how fast I 0:11:04.196 --> 0:11:07.956 can communicate to you? Because if there's no limit, then 0:11:08.196 --> 0:11:11.756 amazing world. Right, we can communicate so fast it's. 0:11:11.556 --> 0:11:15.276 Like instant telepathy. It's like you instantly beat me every 0:11:15.316 --> 0:11:15.996 thought in your head. 0:11:16.076 --> 0:11:18.756 Yeah, okay, exactly. The natural question it has once you 0:11:18.756 --> 0:11:22.356 set up this finite set, as you mentioned, is okay, 0:11:22.396 --> 0:11:25.076 given these finite sets, is there a limit on how 0:11:25.116 --> 0:11:29.076 fast I can communicate to you? And so that was 0:11:29.116 --> 0:11:31.196 the question that was the heart of the paper, which 0:11:31.236 --> 0:11:37.476 is to so he formulated this notion of a capacity. 0:11:37.836 --> 0:11:42.996 That communication system is like a pipe. It's like you're 0:11:43.036 --> 0:11:46.356 pushing water through this pipe, and the size of the 0:11:46.396 --> 0:11:49.316 pipe limits of how fast you can push water through it. 0:11:50.396 --> 0:11:54.356 And now, justly in communication, there's this notion of a 0:11:54.436 --> 0:11:57.676 size of the pipe, which is called a capacity. And 0:11:57.756 --> 0:12:02.076 you figured a way of computing this capacity for different 0:12:02.116 --> 0:12:07.756 communication medium, any communication medium, you can actually compute a 0:12:07.876 --> 0:12:11.756 capacity for that community, and that limits how fast you 0:12:11.796 --> 0:12:15.716 can communicate information over that medium, whether that medium is wireless, 0:12:16.196 --> 0:12:19.556 over the air or over the widline. Like I'm talking 0:12:19.596 --> 0:12:22.236 to you, I communicate over the air, I talk to 0:12:22.316 --> 0:12:25.476 my WiFi. The wi Fi goes through some copper cabo, 0:12:25.636 --> 0:12:29.236 some optical fiber. H He's a physical medium, but he 0:12:29.356 --> 0:12:33.516 can compute a capacity for each of these different mediums. 0:12:34.756 --> 0:12:44.076 And I know that part of the paper looks at, say, 0:12:44.156 --> 0:12:52.676 redundancy in various modes of communication and on related note patterns. Right, 0:12:52.876 --> 0:12:54.956 there's this whole section of the paper where he looks 0:12:54.956 --> 0:12:59.476 at the frequency with which letters occur in English and 0:12:59.596 --> 0:13:02.956 kind of builds builds an idea around that. Tell me 0:13:02.996 --> 0:13:04.196 about those pieces. 0:13:03.876 --> 0:13:07.076 Of the paper. Yeah, so let's talk with the word redundancy. 0:13:07.876 --> 0:13:09.956 Yeah, that comes off right. 0:13:09.996 --> 0:13:11.596 No, no, no, no, no, no no, that's not only 0:13:11.676 --> 0:13:14.516 not the wrong word, but it's actually the most important word. 0:13:14.556 --> 0:13:19.996 I would say almost because you go back to the 0:13:20.076 --> 0:13:22.076 question to the thing I was talking about, which is 0:13:22.116 --> 0:13:26.196 how fast you can communicate? Right, So what he discovered 0:13:26.276 --> 0:13:28.356 was actually there's no limit on how fast it can communicate. 0:13:28.396 --> 0:13:31.436 You can always communicate very fast. But what the guy 0:13:31.516 --> 0:13:35.356 can hear is gibberish, and he cannot really distinguish what 0:13:35.396 --> 0:13:37.116 you're trying to say is like so much noise in 0:13:37.116 --> 0:13:39.676 the system, okay, that he cannot really figure out what he's. 0:13:39.556 --> 0:13:41.756 Saying, even if you're face to face, right, even if 0:13:41.756 --> 0:13:43.836 you're face to face, you're not going to over the 0:13:43.876 --> 0:13:46.276 phone or whatever. If you talk too fast, the listener 0:13:46.276 --> 0:13:47.676 won't understand because you're going to. 0:13:47.956 --> 0:13:51.516 And anybody who goes to a crazy professor's lecture would 0:13:51.556 --> 0:13:54.716 know about this, where the professor just keeps on talking 0:13:55.716 --> 0:13:58.876 and million miles per hour and the students, the sister 0:13:58.996 --> 0:14:01.516 and nobody understood the thing, and the professor cours the 0:14:01.556 --> 0:14:05.796 day when it's finished. So so basically he's what he's 0:14:05.836 --> 0:14:09.636 saying is that, hey, you know what to make sure 0:14:09.756 --> 0:14:13.836 that the information goes through reliably, reliably, that's the first 0:14:13.876 --> 0:14:20.316 word you need to introduce, redundancy, redundancy in your message, okay. 0:14:21.716 --> 0:14:24.676 And what you figured out is in some sense the 0:14:24.996 --> 0:14:28.956 optimal way of adding redundancy, because you know, you can 0:14:28.996 --> 0:14:32.556 always be stupid in adding redundancy. For example, I can 0:14:32.676 --> 0:14:35.116 keep on repeating the same word one hundred times to 0:14:35.156 --> 0:14:37.476 you and then you probably get it, and then I 0:14:37.516 --> 0:14:39.276 move on to the next word. I cannot move on 0:14:39.316 --> 0:14:41.836 the next word, but that would take me one hundred 0:14:41.956 --> 0:14:47.196 times slow. Yes, right, and so that's not a very 0:14:47.236 --> 0:14:49.876 smart way of adding redundancy. So what do you figured 0:14:49.876 --> 0:14:52.756 out is an optimal way of edding redundancy so that 0:14:52.876 --> 0:14:57.436 you can communicate reliably and yet at the maximum what 0:14:57.556 --> 0:15:02.676 he calls capacity limit. And that was a totally amazing 0:15:02.756 --> 0:15:08.156 actually formulation of the problem and highly non obvious. And 0:15:08.196 --> 0:15:12.116 I think that is some of the amazing contribution of 0:15:12.156 --> 0:15:12.916 this guy, Shennon. 0:15:12.996 --> 0:15:20.916 Yeah, it's optimization. He optimizes communication across any channel where 0:15:20.916 --> 0:15:27.196 you're balancing efficiency or speed and reliability. That is the tradeoff, 0:15:27.236 --> 0:15:29.876 and he figures out how to optimize for that trade off. 0:15:31.276 --> 0:15:35.796 Yes, yes, he figured out how to optimize that trade off. 0:15:37.516 --> 0:15:43.516 But that tradeoff turns out to be very interesting. Uh huh. 0:15:43.876 --> 0:15:46.556 It's a very interesting tradeoff. So typically when we think 0:15:46.556 --> 0:15:49.956 about tradeoff, we think about like a smooth curve, right, 0:15:50.636 --> 0:15:53.636 as when you tune something that you can get better performance. 0:15:54.196 --> 0:15:57.236 But what he showed was that there's kind of like 0:15:57.396 --> 0:16:02.756 a cliff effect, Okay, And the cliff effect is that 0:16:02.836 --> 0:16:08.076 if you communicate below this number called capacity, then you 0:16:08.116 --> 0:16:12.916 can always engineer system to make your signal the communication 0:16:13.076 --> 0:16:17.996 as reliable as you want. Huh so reliable, that's completely clean. Wow. 0:16:18.796 --> 0:16:23.516 Whereas you communicate above this number of capacity, then there's 0:16:23.556 --> 0:16:25.596 nothing you can do to make a signal clean. It's 0:16:25.596 --> 0:16:29.796 just completely gibberish. Huh. So it's a very sharp tradeoff 0:16:30.396 --> 0:16:33.036 that he identified. It's not a smooth tradeoff. 0:16:33.556 --> 0:16:35.796 And if you're running the phone company, that's exactly what 0:16:35.876 --> 0:16:37.716 you want to know, right, So then you can tune 0:16:37.756 --> 0:16:40.996 it all the way to capacity and then not try 0:16:41.036 --> 0:16:43.276 and tune it anymore after that, because it's not going 0:16:43.356 --> 0:16:44.076 to get any better. 0:16:44.476 --> 0:16:46.956 Correct, And that's the goal of sixty years of engineering 0:16:46.956 --> 0:16:51.556 to achieve his vision, his vision nineteen forty eight. It 0:16:51.596 --> 0:16:54.716 took people around sixty years to get to that implement 0:16:54.796 --> 0:16:55.276 his vision. 0:16:56.396 --> 0:16:59.596 Well, so you are part of that story, right, Let's 0:16:59.836 --> 0:17:03.276 let's let you walk onto the story now. So you 0:17:03.316 --> 0:17:07.236 tell me about your work and how Shannon's work. You know, 0:17:07.716 --> 0:17:10.036 how you built on Shannon's work. Tell me about how 0:17:10.036 --> 0:17:11.876 you built on Shannon's work. 0:17:13.036 --> 0:17:16.956 Yeah. So I did my PhD in the nineties. In 0:17:16.996 --> 0:17:23.076 the nineties. My advisor was a Shannon student, and so 0:17:23.116 --> 0:17:26.476 I learned information theory him. Okay, Now, at that time, 0:17:27.316 --> 0:17:31.196 information theory was almost a dead subject. Okay. When I 0:17:31.276 --> 0:17:34.796 was a PhD student, the first thing my advisor told me, 0:17:35.036 --> 0:17:39.276 maybe following Shannon, is hey, don't work in information theory. Wow, 0:17:40.116 --> 0:17:42.156 you'll never find a job. You never find a job 0:17:42.156 --> 0:17:44.596 with this stuff. Okay, that's a tough moment. 0:17:44.956 --> 0:17:46.316 That must be a tough moment for. 0:17:46.316 --> 0:17:50.516 Pretty tough, yeah, because at that time, there's not much 0:17:50.596 --> 0:17:55.356 progress made in the theory, and there's no killer applications either. 0:17:55.476 --> 0:17:58.556 There's no very killer applications that need all this sophisticated 0:17:58.756 --> 0:18:03.036 information theory. Okay. So it's like a dead field. 0:18:02.876 --> 0:18:05.196 Was there a while when people used it to like 0:18:05.476 --> 0:18:08.476 whatever make landline phones work better, like in the fifties 0:18:08.556 --> 0:18:10.796 or something with where people like, oh great, now we've 0:18:10.796 --> 0:18:13.396 got this theory and we can make the phone work better. 0:18:14.636 --> 0:18:20.436 Yeah. So the thing is that the solutions that people 0:18:20.436 --> 0:18:24.116 come up with to achieve these capacity limits is very complicated, okay, 0:18:24.916 --> 0:18:27.836 and the electronics that technology is just not enough to 0:18:27.836 --> 0:18:31.116 build these complicated circuits. So information theory have had not 0:18:31.316 --> 0:18:35.156 a very significant applic impact in the fifties, sixties, or 0:18:35.196 --> 0:18:35.996 even seven sounds. 0:18:36.116 --> 0:18:38.836 So it's like one of those cases where the theory 0:18:38.876 --> 0:18:42.076 is just too far ahead of the technology. 0:18:42.076 --> 0:18:45.556 To be useful it. Yeah, and so people can start 0:18:45.596 --> 0:18:47.596 losing interest in the theory is that, yes, this is 0:18:47.596 --> 0:18:50.476 a bunch of maths. It's not impacting the real world, 0:18:50.556 --> 0:18:53.436 and so students are drifting away from the field. But 0:18:53.556 --> 0:18:56.756 there's still always a few students, okay, who are just 0:18:56.956 --> 0:19:00.436 so enumorated by the theory that they keep on pursuing it. 0:19:01.116 --> 0:19:03.996 And my advisor is one of the leading professors in 0:19:04.036 --> 0:19:08.116 this area, and he would have like one student every decade, 0:19:08.356 --> 0:19:11.596 every decade to do research in if. 0:19:11.436 --> 0:19:14.076 You were that student, you were and I was. 0:19:14.076 --> 0:19:16.876 Not that student, And I was not that student, Okay. 0:19:17.396 --> 0:19:19.596 At that time, that slot was already taken by an 0:19:19.596 --> 0:19:22.796 earlier student who was ways more than me, who's ways 0:19:22.876 --> 0:19:25.836 more than me. And that's that he was that he 0:19:25.916 --> 0:19:29.276 was a student of the decade in information theory. Okay. Now, 0:19:30.156 --> 0:19:32.716 so I was assigned to work on some other problems okay, 0:19:32.716 --> 0:19:35.756 completely and related Okay, But anyway, the point though, is 0:19:35.756 --> 0:19:40.036 that when I graduated, something happened, okay, And that was 0:19:40.076 --> 0:19:45.796 the beginning of the wireless revolution. That was the time 0:19:45.876 --> 0:19:49.436 when only a million people have cell phones, and those 0:19:49.436 --> 0:19:53.116 cell phones I don't even remember. It's like gigantically break yeah. 0:19:53.156 --> 0:19:56.836 Like there's that famous scene from the movie Wall Street, right, 0:19:56.916 --> 0:19:58.676 that's the one that everybody talks about where it's like 0:19:58.956 --> 0:20:01.236 bigger than a brick. People say brick, but it's actually 0:20:01.236 --> 0:20:03.916 bigger than a brick. It's like a big hardback book 0:20:04.036 --> 0:20:04.476 or something. 0:20:05.676 --> 0:20:07.836 Yeah. And actually those days, because there's some few of 0:20:07.876 --> 0:20:11.356 six post it's like a prestige. It's like it's prestige 0:20:11.396 --> 0:20:14.596 to have this brick. Yeah, okay, yeah. 0:20:14.356 --> 0:20:16.396 You couldn't get that brick. You had to be rich 0:20:16.476 --> 0:20:18.076 to get that brick. Yeah. 0:20:18.156 --> 0:20:21.756 Yeah. And so the wireless revolution was happening because people 0:20:21.796 --> 0:20:24.276 realized that hey, you know what, be able to communicate 0:20:24.276 --> 0:20:27.996 anytime anywhere is really viable, and so people are now 0:20:28.036 --> 0:20:31.916 getting interested. And at that time, what people realize is 0:20:31.956 --> 0:20:37.356 that whoa this wireless physical media, it's really tough to 0:20:37.396 --> 0:20:41.276 communicate over because the bandwidth is so limited and the 0:20:41.316 --> 0:20:45.116 noise is so much. Right, FCC was limiting the bandwidth 0:20:45.116 --> 0:20:46.996 allocation to these applications a lot. 0:20:47.076 --> 0:20:52.676 Aha, and so Communications Commission the government wasn't letting wireless 0:20:52.676 --> 0:20:54.196 companies use much bandwidth for. 0:20:54.236 --> 0:20:56.156 Cel phone yeah, because all the bandwi most of them 0:20:56.156 --> 0:20:59.636 are allocated for military purposes and there's only very little bandwidth 0:20:59.676 --> 0:21:03.076 allocated at that time for civilians, and so those bandwidth 0:21:03.116 --> 0:21:06.916 were auctional to companies with a very high price, and 0:21:06.956 --> 0:21:10.196 so it became very important to be very efficient in 0:21:10.316 --> 0:21:15.996 using this very expensive property. Aha, Okay, and then people realize, hey, 0:21:16.316 --> 0:21:19.396 if we want to be really efficient, then we need 0:21:19.436 --> 0:21:23.356 a theory which is about efficiency. So people start thinking, okay, 0:21:23.876 --> 0:21:26.836 all right, so information theory was dead, but now it's 0:21:26.876 --> 0:21:29.116 going to come back to life because we have this 0:21:29.196 --> 0:21:32.756 really important problem, really expensive spectrum that was allocated by SEC, 0:21:32.956 --> 0:21:35.516 and we want to squeeze as much of it as possible. 0:21:35.156 --> 0:21:38.476 As much communication. We need a sort of mathematical theory 0:21:38.476 --> 0:21:40.076 of communication, if you will. 0:21:41.156 --> 0:21:44.596 And that was the renaissance of information theory, spurred by 0:21:45.236 --> 0:21:48.636 this amazing technology of wireless, which took us from one 0:21:49.236 --> 0:21:52.596 million phones to ten billion phones. 0:21:53.196 --> 0:21:56.116 Today everybody has one point one. 0:21:55.996 --> 0:22:01.156 Phone, and information theory play a big role in that revolution. 0:22:06.236 --> 0:22:09.676 In a minute, how David used quad shannon It's nineteen 0:22:09.756 --> 0:22:12.436 forty eight paper to come up with an idea that 0:22:12.476 --> 0:22:14.636 we all use every time we make. 0:22:14.556 --> 0:22:24.076 A phone call. 0:22:24.196 --> 0:22:27.756 Let's talk for a moment about your your role, right, 0:22:27.836 --> 0:22:30.756 like you actually played played an important role there. 0:22:31.076 --> 0:22:34.716 Yeah, so I was at Ball Labs. Uh huh, just 0:22:34.756 --> 0:22:37.796 like Claude. So it's like Claude to Yeah. Yeah, So 0:22:37.876 --> 0:22:40.476 I spent one year at ball Lapse as a so 0:22:40.636 --> 0:22:44.516 called postdoc right after my PhD, before I moved to 0:22:44.556 --> 0:22:47.836 Berkeley to become a professor there. I spent one year there. 0:22:48.116 --> 0:22:49.836 And that's what people were talking about that time of 0:22:49.836 --> 0:22:53.836 beat Labs. Hey, this new thing, wireless information theories come 0:22:53.956 --> 0:22:57.116 back to life. We can try to use information theory 0:22:57.196 --> 0:23:01.236 and adapt it and extend it to this wireless communication problem. 0:23:01.876 --> 0:23:05.236 And so that's when I said, whoa this information theory 0:23:05.236 --> 0:23:07.916 I learned from Bob Gallagher. Finally there's a place to 0:23:08.076 --> 0:23:13.436 use it. Finally I can actually make a living, make 0:23:13.476 --> 0:23:15.436 a living out of it. And like what my advisor 0:23:15.476 --> 0:23:18.236 told me, is not dead, it's come back to life. Yeah, 0:23:18.876 --> 0:23:21.556 and so that's sort of my start in the field. 0:23:22.116 --> 0:23:25.716 And uh yeah, so I did, I, you know, invented 0:23:25.716 --> 0:23:29.356 a bunch of stuff and actually apply this connect this 0:23:29.396 --> 0:23:32.196 information theory to the real world. And uh, every time 0:23:32.236 --> 0:23:36.196 you use a phone, you're using my algorithm, which is 0:23:36.236 --> 0:23:38.996 based on the theory of information. Huh. 0:23:39.036 --> 0:23:42.036 And so you're you're that's a cool thing to be 0:23:42.076 --> 0:23:44.756 able to say. First of all, that's a very good 0:23:44.796 --> 0:23:51.876 flex your algorithm, it's the proportional fair scheduling algorithm, right, yes, yes, 0:23:52.156 --> 0:23:53.396 what is that? What's it do? 0:23:54.756 --> 0:23:56.996 All right? So I should tell you a little bit story. 0:23:57.036 --> 0:23:58.876 I think the story is, uh, and then I'll tell 0:23:58.876 --> 0:24:02.796 you what it does. Okay. So I went to So 0:24:02.836 --> 0:24:05.396 that was the end of nineteen ninety nine, around nine 0:24:05.436 --> 0:24:09.036 ninety nine. So I was doing all this information theory 0:24:09.156 --> 0:24:13.836 stuff at Berkeley, writing many papers. But then I always 0:24:13.876 --> 0:24:16.156 have a thought the back of my mind, which you say, 0:24:16.196 --> 0:24:18.436 is this stuff going to be useful? And so I 0:24:18.476 --> 0:24:20.436