Speaker 1: Pushkin.
Speaker 2: Hey, Happy New Year. We're very happy to be back,
Speaker 2: and i have one request before we start the show.
Speaker 2: I'm asking you a favor, and the favor is this,
Speaker 2: would you please send us an email to problem at
Speaker 2: pushkin dot fm and tell us what you like about
Speaker 2: the show and what you don't like about the show,
Speaker 2: and specifically what kinds of things you want to hear
Speaker 2: more of, and perhaps what kinds of things you don't
Speaker 2: want to hear Again, it's problem at Pushkin dot fm.
Speaker 2: I'm going to read all the emails, so thank you
Speaker 2: in advance for sending them. Claude Shannon is this huge
Speaker 2: figure in the history of technology. He's one of the
Speaker 2: key people who worked at Bell Labs in the middle
Speaker 2: of the twentieth century and really came up with the
Speaker 2: idea that made modern technology possible. But I'm going to
Speaker 2: be honest with you, I never really understood what Claude
Speaker 2: Shannon figured out that was such a big deal. But
Speaker 2: the people who know about technology, who know about the
Speaker 2: history of ideas, they say Shannon's a giant. Claude Shannon
Speaker 2: is like the nerds. Nerd he's the techno intellectuals, techno
Speaker 2: intellectual and so For today's show, I wanted to understand
Speaker 2: what did Claude Shannon figure out and why is it
Speaker 2: so important.
Speaker 1: For the modern world.
Speaker 2: I'm Jacob Goldstein, and this is What's your problem. My
Speaker 2: guest today is David Shay. David is a professor of
Speaker 2: electrical engineering at Stanford. He has studied Shannon for decades.
Speaker 2: He teaches Shannon's work to his students, and David used
Speaker 2: Shannon's work to make a breakthrough in cell phone technology.
Speaker 2: And that breakthrough, that breakthrough that came to us via
Speaker 2: Shannon and Shay, it affects every phone call we make.
Speaker 2: David and I talked about Shannon's key insights and about
Speaker 2: how David's own work built on Shannon, and we also
Speaker 2: talked about the big chunk of Shannon's life that was
Speaker 2: taken up with juggling and riding unicycles and building mechanical toys.
Speaker 2: But to start, we talked about how in the middle
Speaker 2: of the twentieth century, Bell Labs wound up driving so
Speaker 2: much technological innovation.
Speaker 1: Yeah, so Bell Labs was the research lab of AT
Speaker 1: and T. Aightenh at that time was the phone company. Okay,
Speaker 1: nowadays we have many phone companies. We have Verizon, we
Speaker 1: have T Mobile, et cetera. But those days there was
Speaker 1: only one phone company, and that's a monopoly. So a
Speaker 1: monopoly needs to justify its existence.
Speaker 2: Huh. So it doesn't get broken by the government.
Speaker 1: It doesn't get broken up. Of course, it eventually got
Speaker 1: broken up, but at that time it was a monopoly.
Speaker 1: And so one way of justifying its existence is to
Speaker 1: say that. Okay, he says to the American people, to
Speaker 1: the government, that we will always spend a certain percent
Speaker 1: of our revenue on this research lab called bow Labs,
Speaker 1: and whatever bow Labs come up with is kind of
Speaker 1: our contribution, know only to our bottom line, but also
Speaker 1: to technology of the country.
Speaker 2: So they have this sort of public mission to prevent
Speaker 2: the government from breaking them up.
Speaker 1: Yeah, and so therefore it also allows researchers a very
Speaker 1: free reign to do research that not necessarily tied to,
Speaker 1: like say, a particular business unit. Okay, So they can
Speaker 1: be very creative. And that's the atmosphere about so bad
Speaker 1: Labs attracted a bunch of very smart people because smart
Speaker 1: people wants to work on their own problem, not the
Speaker 1: problem that the manager gives them. Yeah, okay, that's the
Speaker 1: that's one characteristic of smart people. And so yeah, that
Speaker 1: was the heydays of Bat Labs. Lots of smart people
Speaker 1: inventing amazing stuff. Laser was invented there, information theory, the
Speaker 1: transistor was invented there. Sort of almost all the foundation
Speaker 1: of the information age. Yeah, where there's hardware, algorithm, software
Speaker 1: is in some sense all have the roots at Ball Labs.
Speaker 1: So that was the contribution to mankind. Actually, I should say, no,
Speaker 1: only to America.
Speaker 2: So Shannon gets there at this time, right, he's there
Speaker 2: with you know, when they're inventing certainly the transistor.
Speaker 1: What's he do?
Speaker 2: Tell me about his his work there? When he gets there,
Speaker 2: what's he working on?
Speaker 1: Yeah? So I think Shannon always have his own agenda, right.
Speaker 1: We know for a fact that he has been interested
Speaker 1: in the problem of communication, that idea of having a
Speaker 1: grand theory of communication, even back in nineteen thirty eight,
Speaker 1: I think thirty seven thirty eight, because he wrote a
Speaker 1: letter at that time to a very famous person named
Speaker 1: Venera Bush. Yeah. Vera Bush is very famous, as he
Speaker 1: was I think president of MIT or dean of MIT,
Speaker 1: and then he became sort of a scientific advisor to
Speaker 1: the president, and so he wrote a letter to Venera
Speaker 1: Bush in nineteen thirty eight and say, hey, you know what,
Speaker 1: I'm really interested in this question of how to find
Speaker 1: one theory that unifies all possible communication systems. There's so
Speaker 1: many different communications system out there, but I think there's
Speaker 1: something at the heart of every system, and I'm trying
Speaker 1: to get to the heart.
Speaker 2: And like nobody had thought of it in that way, right.
Speaker 2: It seems like part of his part of what Shannon's
Speaker 2: such a big deal is like as I understand that
Speaker 2: people it was like, you know that people understood like
Speaker 2: they were trying to figure out how to make the
Speaker 2: phone work better, and they were trying to, you know,
Speaker 2: make movies be clearer or whatever. But there wasn't this
Speaker 2: idea that you could abstract it until Shannon came along.
Speaker 1: And the reason is very simple, actually, because if you
Speaker 1: have a physical system, then you want to build right
Speaker 1: what do you see right? You say, hey man the
Speaker 1: video for example, i'm seeing you right now, I'm not
Speaker 1: seeing you very clearly, have to say yes.
Speaker 2: I'm in a closet, a closet.
Speaker 1: Right Then I would say, how to try to improve
Speaker 1: the imbage? Maybe I can try to, you know, fix
Speaker 1: this pixel or do some filtering of your noise. So
Speaker 1: I'm very tied to the very specific details of the
Speaker 1: specific problem. Because why I'm the engineer. I need to
Speaker 1: improve the system, not in ten years, but tomorrow. You know, tomorrow.
Speaker 2: You don't need a theory of the system. You just
Speaker 2: want to creer picture.
Speaker 1: Yeah, yeah, I mean I'm in the weeds, right, I'm
Speaker 1: in the weeds. And Shannon, because of his training, and
Speaker 1: also because of the atmosphere of a place like bout Labs,
Speaker 1: could afford to st step back and just look at
Speaker 1: the broader forest as opposed to the details of specific trees.
Speaker 2: So so, okay, Shannon's big idea comes out in this
Speaker 2: paper he publishes in nineteen forty eight. The paper is
Speaker 2: called a Mathematical Theory of Communication. It's like his great work.
Speaker 2: Tell me about that paper.
Speaker 1: So that paper is actually a very interesting paper. In fact,
Speaker 1: when I teach information theory, I teach from the paper itself,
Speaker 1: because I thought it's an amazing way not only of
Speaker 1: learning information theory, but learning how to write a scientific
Speaker 1: paper properly. Huh okay. And you know, not everyone does
Speaker 1: research and information theory, but everybody has to write uh
Speaker 1: huh okay. Every researcher has the right to express their
Speaker 1: ideas to the peers and to the audience. So in
Speaker 1: that paper, very interesting. The first paragraph of the paper. Okay,
Speaker 1: it's already very interesting because typically when people write a
Speaker 1: paper nowadays, they tell you, oh, how great my invention is.
Speaker 1: It's going to change the world. Every paper is going
Speaker 1: to change the world. But in fact, his first paper
Speaker 1: paragraph focused on telling you what his paper is not achieving.
Speaker 1: Ha ha, I mean that's a master. That's the masters, right,
Speaker 1: I mean, how many papers that you read nowadays tells
Speaker 1: you in the beginning, Hey, you know what, guys, expectation
Speaker 1: management here, this paper is not about this. Hey, don't
Speaker 1: get your home. Yeah, exactly, That's exactly what he did.
Speaker 1: Expectation management nowadays, that's what today we will call it
Speaker 1: expectation management. And now those days, I guess he just
Speaker 1: calls it honesty. And his whole point was often people
Speaker 1: associate information with meaning, okay, and then he said in
Speaker 1: this we ignore meaning, we ignore meaning. Huh okay. So
Speaker 1: that was the first thing he did, which is brilliant
Speaker 1: because once you high information with meaning, then he will
Speaker 1: never be able to make any progress. It's just too
Speaker 1: difficult and too broad and too vague a problem.
Speaker 2: Everybody gets stuck on this idea of meaning and what
Speaker 2: is meaning? And he's like, forget about meaning. So we're
Speaker 2: gonna forget about meaning. What is left?
Speaker 1: Yes, Actually, the biggest I think breakthrough of that paper
Speaker 1: is to really focus on the thing that matters and
Speaker 1: cut away a lot of stuff that really doesn't learn
Speaker 1: that it doesn't matter, but it doesn't matter in terms
Speaker 1: of solving the communication problem, the communication. So then he said, okay,
Speaker 1: what is the communication problem? The communication problem is the
Speaker 1: following is that there are multiple possibilities of a word,
Speaker 1: and my goal is to tell the receiver destination which
Speaker 1: of the multiple pospiity is the correct prosperity.
Speaker 2: Yeah, and so in language, it's basically it's a finite set.
Speaker 2: Language is a finite set. It's very large. But if
Speaker 2: we're speaking and we both know that we're speaking English,
Speaker 2: then essentially you are hearing the words and decoding them,
Speaker 2: and you know that it is a series of words,
Speaker 2: and you just have to figure out which words I
Speaker 2: mean like that for example, Yes, like that? Okay, so
Speaker 2: that's the frame he builds then.
Speaker 1: Why okay, all right, Then once you have this framing, right,
Speaker 1: then you can ask the question, Okay, what is the
Speaker 1: goal of communication? The goal of communication is to communicate
Speaker 1: as fast as I can, right, And the natural question
Speaker 1: is why is there a limit on how fast I
Speaker 1: can communicate to you? Because if there's no limit, then
Speaker 1: amazing world. Right, we can communicate so fast it's.
Speaker 2: Like instant telepathy. It's like you instantly beat me every
Speaker 2: thought in your head.
Speaker 1: Yeah, okay, exactly. The natural question it has once you
Speaker 1: set up this finite set, as you mentioned, is okay,
Speaker 1: given these finite sets, is there a limit on how
Speaker 1: fast I can communicate to you? And so that was
Speaker 1: the question that was the heart of the paper, which
Speaker 1: is to so he formulated this notion of a capacity.
Speaker 1: That communication system is like a pipe. It's like you're
Speaker 1: pushing water through this pipe, and the size of the
Speaker 1: pipe limits of how fast you can push water through it.
Speaker 1: And now, justly in communication, there's this notion of a
Speaker 1: size of the pipe, which is called a capacity. And
Speaker 1: you figured a way of computing this capacity for different
Speaker 1: communication medium, any communication medium, you can actually compute a
Speaker 1: capacity for that community, and that limits how fast you
Speaker 1: can communicate information over that medium, whether that medium is wireless,
Speaker 1: over the air or over the widline. Like I'm talking
Speaker 1: to you, I communicate over the air, I talk to
Speaker 1: my WiFi. The wi Fi goes through some copper cabo,
Speaker 1: some optical fiber. H He's a physical medium, but he
Speaker 1: can compute a capacity for each of these different mediums.
Speaker 2: And I know that part of the paper looks at, say,
Speaker 2: redundancy in various modes of communication and on related note patterns. Right,
Speaker 2: there's this whole section of the paper where he looks
Speaker 2: at the frequency with which letters occur in English and
Speaker 2: kind of builds builds an idea around that. Tell me
Speaker 2: about those pieces.
Speaker 1: Of the paper. Yeah, so let's talk with the word redundancy.
Speaker 2: Yeah, that comes off right.
Speaker 1: No, no, no, no, no, no no, that's not only
Speaker 1: not the wrong word, but it's actually the most important word.
Speaker 1: I would say almost because you go back to the
Speaker 1: question to the thing I was talking about, which is
Speaker 1: how fast you can communicate? Right, So what he discovered
Speaker 1: was actually there's no limit on how fast it can communicate.
Speaker 1: You can always communicate very fast. But what the guy
Speaker 1: can hear is gibberish, and he cannot really distinguish what
Speaker 1: you're trying to say is like so much noise in
Speaker 1: the system, okay, that he cannot really figure out what he's.
Speaker 2: Saying, even if you're face to face, right, even if
Speaker 2: you're face to face, you're not going to over the
Speaker 2: phone or whatever. If you talk too fast, the listener
Speaker 2: won't understand because you're going to.
Speaker 1: And anybody who goes to a crazy professor's lecture would
Speaker 1: know about this, where the professor just keeps on talking
Speaker 1: and million miles per hour and the students, the sister
Speaker 1: and nobody understood the thing, and the professor cours the
Speaker 1: day when it's finished. So so basically he's what he's
Speaker 1: saying is that, hey, you know what to make sure
Speaker 1: that the information goes through reliably, reliably, that's the first
Speaker 1: word you need to introduce, redundancy, redundancy in your message, okay.
Speaker 1: And what you figured out is in some sense the
Speaker 1: optimal way of adding redundancy, because you know, you can
Speaker 1: always be stupid in adding redundancy. For example, I can
Speaker 1: keep on repeating the same word one hundred times to
Speaker 1: you and then you probably get it, and then I
Speaker 1: move on to the next word. I cannot move on
Speaker 1: the next word, but that would take me one hundred
Speaker 1: times slow. Yes, right, and so that's not a very
Speaker 1: smart way of adding redundancy. So what do you figured
Speaker 1: out is an optimal way of edding redundancy so that
Speaker 1: you can communicate reliably and yet at the maximum what
Speaker 1: he calls capacity limit. And that was a totally amazing
Speaker 1: actually formulation of the problem and highly non obvious. And
Speaker 1: I think that is some of the amazing contribution of
Speaker 1: this guy, Shennon.
Speaker 2: Yeah, it's optimization. He optimizes communication across any channel where
Speaker 2: you're balancing efficiency or speed and reliability. That is the tradeoff,
Speaker 2: and he figures out how to optimize for that trade off.
Speaker 1: Yes, yes, he figured out how to optimize that trade off.
Speaker 1: But that tradeoff turns out to be very interesting. Uh huh.
Speaker 1: It's a very interesting tradeoff. So typically when we think
Speaker 1: about tradeoff, we think about like a smooth curve, right,
Speaker 1: as when you tune something that you can get better performance.
Speaker 1: But what he showed was that there's kind of like
Speaker 1: a cliff effect, Okay, And the cliff effect is that
Speaker 1: if you communicate below this number called capacity, then you
Speaker 1: can always engineer system to make your signal the communication
Speaker 1: as reliable as you want. Huh so reliable, that's completely clean. Wow.
Speaker 1: Whereas you communicate above this number of capacity, then there's
Speaker 1: nothing you can do to make a signal clean. It's
Speaker 1: just completely gibberish. Huh. So it's a very sharp tradeoff
Speaker 1: that he identified. It's not a smooth tradeoff.
Speaker 2: And if you're running the phone company, that's exactly what
Speaker 2: you want to know, right, So then you can tune
Speaker 2: it all the way to capacity and then not try
Speaker 2: and tune it anymore after that, because it's not going
Speaker 2: to get any better.
Speaker 1: Correct, And that's the goal of sixty years of engineering
Speaker 1: to achieve his vision, his vision nineteen forty eight. It
Speaker 1: took people around sixty years to get to that implement
Speaker 1: his vision.
Speaker 2: Well, so you are part of that story, right, Let's
Speaker 2: let's let you walk onto the story now. So you
Speaker 2: tell me about your work and how Shannon's work. You know,
Speaker 2: how you built on Shannon's work. Tell me about how
Speaker 2: you built on Shannon's work.
Speaker 1: Yeah. So I did my PhD in the nineties. In
Speaker 1: the nineties. My advisor was a Shannon student, and so
Speaker 1: I learned information theory him. Okay, Now, at that time,
Speaker 1: information theory was almost a dead subject. Okay. When I
Speaker 1: was a PhD student, the first thing my advisor told me,
Speaker 1: maybe following Shannon, is hey, don't work in information theory. Wow,
Speaker 1: you'll never find a job. You never find a job
Speaker 1: with this stuff. Okay, that's a tough moment.
Speaker 2: That must be a tough moment for.
Speaker 1: Pretty tough, yeah, because at that time, there's not much
Speaker 1: progress made in the theory, and there's no killer applications either.
Speaker 1: There's no very killer applications that need all this sophisticated
Speaker 1: information theory. Okay. So it's like a dead field.
Speaker 2: Was there a while when people used it to like
Speaker 2: whatever make landline phones work better, like in the fifties
Speaker 2: or something with where people like, oh great, now we've
Speaker 2: got this theory and we can make the phone work better.
Speaker 1: Yeah. So the thing is that the solutions that people
Speaker 1: come up with to achieve these capacity limits is very complicated, okay,
Speaker 1: and the electronics that technology is just not enough to
Speaker 1: build these complicated circuits. So information theory have had not
Speaker 1: a very significant applic impact in the fifties, sixties, or
Speaker 1: even seven sounds.
Speaker 2: So it's like one of those cases where the theory
Speaker 2: is just too far ahead of the technology.
Speaker 1: To be useful it. Yeah, and so people can start
Speaker 1: losing interest in the theory is that, yes, this is
Speaker 1: a bunch of maths. It's not impacting the real world,
Speaker 1: and so students are drifting away from the field. But
Speaker 1: there's still always a few students, okay, who are just
Speaker 1: so enumorated by the theory that they keep on pursuing it.
Speaker 1: And my advisor is one of the leading professors in
Speaker 1: this area, and he would have like one student every decade,
Speaker 1: every decade to do research in if.
Speaker 2: You were that student, you were and I was.
Speaker 1: Not that student, And I was not that student, Okay.
Speaker 1: At that time, that slot was already taken by an
Speaker 1: earlier student who was ways more than me, who's ways
Speaker 1: more than me. And that's that he was that he
Speaker 1: was a student of the decade in information theory. Okay. Now,
Speaker 1: so I was assigned to work on some other problems okay,
Speaker 1: completely and related Okay, But anyway, the point though, is
Speaker 1: that when I graduated, something happened, okay, And that was
Speaker 1: the beginning of the wireless revolution. That was the time
Speaker 1: when only a million people have cell phones, and those
Speaker 1: cell phones I don't even remember. It's like gigantically break yeah.
Speaker 2: Like there's that famous scene from the movie Wall Street, right,
Speaker 2: that's the one that everybody talks about where it's like
Speaker 2: bigger than a brick. People say brick, but it's actually
Speaker 2: bigger than a brick. It's like a big hardback book
Speaker 2: or something.
Speaker 1: Yeah. And actually those days, because there's some few of
Speaker 1: six post it's like a prestige. It's like it's prestige
Speaker 1: to have this brick. Yeah, okay, yeah.
Speaker 2: You couldn't get that brick. You had to be rich
Speaker 2: to get that brick. Yeah.
Speaker 1: Yeah. And so the wireless revolution was happening because people
Speaker 1: realized that hey, you know what, be able to communicate
Speaker 1: anytime anywhere is really viable, and so people are now
Speaker 1: getting interested. And at that time, what people realize is
Speaker 1: that whoa this wireless physical media, it's really tough to
Speaker 1: communicate over because the bandwidth is so limited and the
Speaker 1: noise is so much. Right, FCC was limiting the bandwidth
Speaker 1: allocation to these applications a lot.
Speaker 2: Aha, and so Communications Commission the government wasn't letting wireless
Speaker 2: companies use much bandwidth for.
Speaker 1: Cel phone yeah, because all the bandwi most of them
Speaker 1: are allocated for military purposes and there's only very little bandwidth
Speaker 1: allocated at that time for civilians, and so those bandwidth
Speaker 1: were auctional to companies with a very high price, and
Speaker 1: so it became very important to be very efficient in
Speaker 1: using this very expensive property. Aha, Okay, and then people realize, hey,
Speaker 1: if we want to be really efficient, then we need
Speaker 1: a theory which is about efficiency. So people start thinking, okay,
Speaker 1: all right, so information theory was dead, but now it's
Speaker 1: going to come back to life because we have this
Speaker 1: really important problem, really expensive spectrum that was allocated by SEC,
Speaker 1: and we want to squeeze as much of it as possible.
Speaker 2: As much communication. We need a sort of mathematical theory
Speaker 2: of communication, if you will.
Speaker 1: And that was the renaissance of information theory, spurred by
Speaker 1: this amazing technology of wireless, which took us from one
Speaker 1: million phones to ten billion phones.
Speaker 2: Today everybody has one point one.
Speaker 1: Phone, and information theory play a big role in that revolution.
Speaker 2: In a minute, how David used quad shannon It's nineteen
Speaker 2: forty eight paper to come up with an idea that
Speaker 2: we all use every time we make.
Speaker 1: A phone call.
Speaker 2: Let's talk for a moment about your your role, right,
Speaker 2: like you actually played played an important role there.
Speaker 1: Yeah, so I was at Ball Labs. Uh huh, just
Speaker 1: like Claude. So it's like Claude to Yeah. Yeah, So
Speaker 1: I spent one year at ball Lapse as a so
Speaker 1: called postdoc right after my PhD, before I moved to
Speaker 1: Berkeley to become a professor there. I spent one year there.
Speaker 1: And that's what people were talking about that time of
Speaker 1: beat Labs. Hey, this new thing, wireless information theories come
Speaker 1: back to life. We can try to use information theory
Speaker 1: and adapt it and extend it to this wireless communication problem.
Speaker 1: And so that's when I said, whoa this information theory
Speaker 1: I learned from Bob Gallagher. Finally there's a place to
Speaker 1: use it. Finally I can actually make a living, make
Speaker 1: a living out of it. And like what my advisor
Speaker 1: told me, is not dead, it's come back to life. Yeah,
Speaker 1: and so that's sort of my start in the field.
Speaker 1: And uh yeah, so I did, I, you know, invented
Speaker 1: a bunch of stuff and actually apply this connect this
Speaker 1: information theory to the real world. And uh, every time
Speaker 1: you use a phone, you're using my algorithm, which is
Speaker 1: based on the theory of information. Huh.
Speaker 2: And so you're you're that's a cool thing to be
Speaker 2: able to say. First of all, that's a very good
Speaker 2: flex your algorithm, it's the proportional fair scheduling algorithm, right, yes, yes,
Speaker 2: what is that? What's it do?
Speaker 1: All right? So I should tell you a little bit story.
Speaker 1: I think the story is, uh, and then I'll tell
Speaker 1: you what it does. Okay. So I went to So
Speaker 1: that was the end of nineteen ninety nine, around nine
Speaker 1: ninety nine. So I was doing all this information theory
Speaker 1: stuff at Berkeley, writing many papers. But then I always
Speaker 1: have a thought the back of my mind, which you say,
Speaker 1: is this stuff going to be useful? And so I
Speaker 1: went to a I decided to go to a company,
Speaker 1: a wireless company who actually build these things and see
Speaker 1: whether this theory can be used. And the company went
Speaker 1: to is called Quaker.
Speaker 2: Okay, I've heard of Quaker.
Speaker 1: You've heard of Quaker, but at that time it was
Speaker 1: a small company, it was not very big, Okay. And
Speaker 1: at that time they have this problem they're working on. Okay,
Speaker 1: which is the following. All right. So in wireless communication,
Speaker 1: there's a concept COUD based station okay, and the base
Speaker 1: station serves many cell phones in the vicinity of the
Speaker 1: bay station. It's cost sout.
Speaker 2: Is it like a tower? Is what we would call
Speaker 2: it a tower?
Speaker 1: That's right, it's always on the tower. There's there's some
Speaker 1: electronics there. Yeah, and that's how the bay station is
Speaker 1: supposed to beam information to many phones, and many phones.
Speaker 2: You still see them. You see them when whatever on
Speaker 2: top of a big building or when you're driving down
Speaker 2: the freeway. Right, that's what you're talking about.
Speaker 1: That, Yeah, that's right. And sometimes on fake trees.
Speaker 2: I love the fake trees in New Jersey.
Speaker 1: They love the fake New Jersey. That's right, New Jersey,
Speaker 1: fake trees. Yes, So at that time they would look
Speaker 1: at this problem, which is, hey, okay, my bandwidth is limited,
Speaker 1: but I have many users to serve. Yeah, okay, how
Speaker 1: do I schedule my limited resource among all these users? Right?
Speaker 1: Because I only have one total bandwidth. And so at
Speaker 1: that time people think, okay, maybe something simple. I give
Speaker 1: one end of the time to the end user. Right,
Speaker 1: so the five users, I serve this user for a
Speaker 1: little bit, and I served the second user for a
Speaker 1: little bit, and served user for.
Speaker 2: The ideas, you're switching really fast. You're just like switching.
Speaker 1: Switching really fast, yeah, exactly. And then when I went there,
Speaker 1: I said, okay, good is the problem is a good problem.
Speaker 1: And I said, hey, instead of fixating on this particular
Speaker 1: scheduling policy, why don't we do any Shannon thing, a.
Speaker 2: Card Shannon thing. You thought of, your thought of it? Yeah, okay.
Speaker 1: The clause shaming thing is what is to look at
Speaker 1: the problem from first principle, Uh not reassume a particular
Speaker 1: solution or a particular class solution even and ask ourselves
Speaker 1: what is the capacity of this whole system, and how
Speaker 1: do I engineer the system to achieve that capacity? Okay?
Speaker 1: And it turns out that if you look at the
Speaker 1: problem this way, then it turns out that the optimal
Speaker 1: way of scheduling is not the one that they will
Speaker 1: try and design. And the reason is because in wireless
Speaker 1: communication there's a very interesting characteristic which is called fading. Okay, okay.
Speaker 1: When I talk to you over the air, the channel
Speaker 1: actually goes up and down, strong and weak, strong and weak,
Speaker 1: very rapidly. What I mean is when I send an
Speaker 1: Electromnett signal from the from the base to the phone,
Speaker 1: that signal get amplified and attenuate it very rapidly. It
Speaker 1: goes up and down.
Speaker 2: Okay, Okay, can we say it gets stronger and weaker.
Speaker 2: Can we say it gets stronger.
Speaker 1: And stronger and weaker? Okay? Yes, And so the alcomal
Speaker 1: way that Information SERVI it has to do is actually
Speaker 1: not divide the time into slots blindly, but really try
Speaker 1: to SKATEU a user when the channel is strong.
Speaker 2: Ha.
Speaker 1: And then from that on I designed a scheduling album
Speaker 1: from which is more practical by sort of leverage of
Speaker 1: this basic idea from information theory.
Speaker 2: And so so the base station is basically monitoring the
Speaker 2: strength of the incoming signals from all the different phones
Speaker 2: and saying, Oh, that one's strong, I'm gonna grab that one.
Speaker 2: Oh that one's strong, I'm gonna grab that one.
Speaker 1: That's what's happening, correct? Correct?
Speaker 2: And how does that I mean I get in a
Speaker 2: kind of big first principles way sort of analogously, it
Speaker 2: follows from Shannon. But is there anything sort of specific
Speaker 2: in Shannon that leads you to this algorithm?
Speaker 1: So remember Shannon is a very general theory. Yeah, Okay.
Speaker 1: It basically says that given any communication medium or any
Speaker 1: communication setting, yeah, you can try to calculate this notion
Speaker 1: of a capacity. So the very general theory, what I
Speaker 1: did was to apply it to a very specific context,
Speaker 1: which is this base station serving multiple user setting. Yeah,
Speaker 1: and then apply his framework to analyze the capacity of
Speaker 1: that system.
Speaker 2: Huh.
Speaker 1: And in the process of analyzing the capacity, you can
Speaker 1: also figure out what is the optimal way of achieving
Speaker 1: that capacity. Remember you mentioned capacity is really an optimization problem,
Speaker 1: and Shannon was able to solve this optimization problem in general.
Speaker 1: But now I specialize it in some sense to this
Speaker 1: pretty specific setting, except that the setting is used by everybody.
Speaker 1: But at that time, it was like, you know, research
Speaker 1: is about timing, and I was there at the right
Speaker 1: place at the right time. Because Qualcom turns out to
Speaker 1: completely dominate the entire third generation technology. So when I
Speaker 1: was able to convince them that, hey, your way of
Speaker 1: doing things is no good. This way suggested by Shannon's
Speaker 1: actually far better. Please use this way. It took me
Speaker 1: a few months, but I was able to persuade them
Speaker 1: to implement it, and then it got into the standard
Speaker 1: through the domination, and then every standard after that uses
Speaker 1: the same basic, the same algorithm. So it was good because,
Speaker 1: as I said, I'm at the right place at the
Speaker 1: right time. You know, when you try to contribute to engineering.
Speaker 1: It's too late if the system is built, because people
Speaker 1: don't want to wreck change the whole system to accompany
Speaker 1: or new idea. But it was very early in the
Speaker 1: design phase.
Speaker 2: So so okay, So you made this breakthrough in wireless
Speaker 2: communications using Shannon's work. Were there similar breakthroughs in, you know,
Speaker 2: in other domains?
Speaker 1: Any communication median right, it could be optical, fiber, it
Speaker 1: could be DSL modem, YESL modem, underwater communication. Almost all
Speaker 1: these communication systems are now designed based on his principle.
Speaker 1: So as impact of this theory is kind of global,
Speaker 1: it's the entire communication landscape.
Speaker 2: There's a story I read in about Shannon that when
Speaker 2: he is developing information theory, he he takes a book
Speaker 2: off the shelf and he reads a sentence to its
Speaker 2: actually his wife, and it's something like the lamp was
Speaker 2: sitting on the and she says table and he says, no,
Speaker 2: I'll give you a clue. The first letter is D
Speaker 2: and she says desk. And when I heard that story,
Speaker 2: what I thought of was large language models, Like that
Speaker 2: sounds exactly like a large language model, And so I'm
Speaker 2: just fishing, I'm just curious, like does his work matter
Speaker 2: for machine learning, large language models, et cetera or no.
Speaker 1: Yeah, so that's a very interesting point. Now I'm not
Speaker 1: an expert by any means in AI or large language models. Yeah,
Speaker 1: I'm not a professional researcher in that area. But I
Speaker 1: think you can actually see some commonality, right, is that
Speaker 1: you know these models in some sense, they don't care
Speaker 1: about meaning either.
Speaker 2: Yeah, very good, Very good. Yeah.
Speaker 1: Right, Actually I've just came to my discussion is very
Speaker 1: interesting because it's really just patterns. It's just which patterns
Speaker 1: are more likely than other patterns. Right. The example you
Speaker 1: gave about desk and is basically about patterns, and information
Speaker 1: theory is really analyzing sort of the number of possible
Speaker 1: patterns in some sense. So there is definitely a philosophical connection,
Speaker 1: I believe, starting from Shannon to these large language models.
Speaker 2: So let me ask you about one other, and this
Speaker 2: is one that you are professionally involved in cryptocurrency and blockchain.
Speaker 2: You have studied it and you started a company.
Speaker 1: Right.
Speaker 2: Is there a connection between Shannon's work and cryptocurrency.
Speaker 1: Yeah, So what attracts me to work in this area blockchain,
Speaker 1: is that blockchain actually has one very common philosophical connection
Speaker 1: to information theory, which is a following in blockchain. The
Speaker 1: problem is not communication per se. It's called consensus. Okay,
Speaker 1: it's a different problem, but it's essentially allow a bunch
Speaker 1: of users at different places to come to an agreement
Speaker 1: on something. Okay, yes, Now, the goal of designing blockchain
Speaker 1: is really to be so called for tolerant, tolerant, which
Speaker 1: means for torerant, which means that even if say one
Speaker 1: third of the users are bad guys and send you
Speaker 1: some gibberish message, you can still the rest two third
Speaker 1: people can still come to an agreement. Okay, all right,
Speaker 1: So you look at this problem, it's actually not that
Speaker 1: different from communication information theory because it's kind of combating.
Speaker 2: The bad guys are the noise that the good guys
Speaker 2: at the signal.
Speaker 1: And the good guys at the signal, and they try
Speaker 1: to reintroduce redundancy, okay, to help them to fight against
Speaker 1: these bad guys.
Speaker 2: And there's an optimization problem where like the more redundancy
Speaker 2: you have, the sort of slower the system is, the
Speaker 2: more ponderous.
Speaker 1: And so you tried an optimization problem is to try
Speaker 1: to figure out what is the optimal number of bad
Speaker 1: guys that you can tolerate and your system still works.
Speaker 1: That is the analogous to the capacity problem. So I
Speaker 1: find the philosophical connection very appealing, and so that's sort
Speaker 1: of one reason why I got attracted to work into
Speaker 1: this area.
Speaker 2: Why do you think more people don't know about Shannon?
Speaker 2: Like all of the sort of intellectuals in technology say,
Speaker 2: he's like one of the great thinkers of the twentieth century,
Speaker 2: but most people have never heard of him. Why do
Speaker 2: you think that is?
Speaker 1: So? Shannon was actually a very shy person, very shy person.
Speaker 1: He hates publicity, He hated when people interview him. You remember, right,
Speaker 1: it's basically a very modest person. Remember the first paragraph
Speaker 1: I talked you about. Yeah, he tells you what he
Speaker 1: is not accomplishing. Yeah, And so he's a very modest,
Speaker 1: very shy person, not into publicity. And I think that
Speaker 1: sort of impact not only himself, but also everybody who
Speaker 1: works in that field. Uhha, and dob this as kind
Speaker 1: of like a metric, right that Hey, we should all
Speaker 1: be modest, because what look at this guy who accomplished
Speaker 1: so much and he's still so motist. Who are we?
Speaker 1: Who are we? Right? So, as a result, the field
Speaker 1: doesn't really sell himself very well. The marketing engine, the
Speaker 1: marketing DNA is not there. Yeah, and so people don't
Speaker 1: know about him.
Speaker 2: So I want to talk for a minute about the
Speaker 2: rest of Shannon's life. He writes this huge paper when
Speaker 2: he's in his early thirties, eventually goes on to be
Speaker 2: a professor at MIT, and he seems to spend a
Speaker 2: lot of his career juggling, writing a unicycle, building mechanical toys,
Speaker 2: building games, and he never, you know, does sort of
Speaker 2: great influential work again, and I'm curious, you know, what
Speaker 2: do you what do you make of that? How do
Speaker 2: you sort of fit his whole career together?
Speaker 1: So there's a single there's a theme that unifies all
Speaker 1: this in my mind, which is playfulness. Because in his mind,
Speaker 1: research is really about puzzles. Uh, he doesn't understand something.
Speaker 1: It's like a puzzle to him, and he's trying to
Speaker 1: figure out the pieces of the puzzle. Information theory was
Speaker 1: like that the puzzles. He sees all these real war systems,
Speaker 1: they seem to all share some community, but nobody understood it.
Speaker 1: So there's a puzzle, and it's always thinking about the puzzle.
Speaker 1: And finally his paper basically solve that puzzle. So everything
Speaker 1: to him is playfulness. I think it's playing. There's a
Speaker 1: game puzzle and needs a soft to puzzle, and that
Speaker 1: is mine. That's how it's my work. So although it
Speaker 1: seems very different things that he did pre and post
Speaker 1: inflamation theory, but it's actually in my mind quite strongly monouns.
Speaker 2: We'll be back in a minute with the lightning round.
Speaker 2: So I read that you recently asked people at your
Speaker 2: company to give five minute talks. I'm curious why you
Speaker 2: did that. That's interesting to me. Why'd you do that?
Speaker 1: So just short to talk the harderest to give. So
Speaker 1: you can't explain an idea in five minutes, then I
Speaker 1: think your idea is actually not very good.
Speaker 2: Ah, that's good.
Speaker 1: Most good ideas you can get the point to across
Speaker 1: in five minutes. Remember, I'm an information theorist by training,
Speaker 1: so communication to the limit is what I'm passionate about.
Speaker 2: If you had to give a five minute talk, what
Speaker 2: would it be about?
Speaker 1: How about Shutton? I guess he's my hero. He's my hero.
Speaker 2: So one you talked about the importance of timing in
Speaker 2: research of not only finding the right problem, but finding
Speaker 2: the right problem at the right time. Right, both in
Speaker 2: terms of Shannon's work and in terms of your work.
Speaker 2: You know, you're also a professor and you know a manager, Like,
Speaker 2: how do you help other people find the right problem
Speaker 2: at the right time?
Speaker 1: Yeah, finding the right problem at the right time is
Speaker 1: probably the most difficult because you know, time is everything. However,
Speaker 1: this is hard to teach. What we try to do
Speaker 1: is to be ready. So one very famous information theorist
Speaker 1: told me this. He said, you know, everybody will get
Speaker 1: lucky at some point in time in the career. However,
Speaker 1: most people, when they get lucky, they're not ready, so
Speaker 1: they don't realize that they get lucky, and so they
Speaker 1: missed the opportunity. They went a different direction. Luck tells
Speaker 1: you should go this way, but you went the other way.
Speaker 1: Lost it.
Speaker 2: That makes me so scared.
Speaker 1: And so what I teach my students is always be ready.
Speaker 1: It's like your muscles. You have to be always training
Speaker 1: your muscles so that when you are lucky, you can
Speaker 1: capitalize on the lucky.
Speaker 2: Do you So you talked about Shannon's playful nature like
Speaker 2: he was a juggler. He wrote a unicycle You do
Speaker 2: anything like that? You have any weird hobbies.
Speaker 1: No, No, the only weird hobby is I love to
Speaker 1: talk to people like.
Speaker 2: You fair, you love going on podcasts. That that is
Speaker 2: the juggling of the twenty first century. Who's your second
Speaker 2: favorite underrated thinker?
Speaker 1: My advisor Ah Gallagher? Well gallaghery. He taught me how
Speaker 1: to think about research because you learned from Shannon and
Speaker 1: I learned from him.
Speaker 2: And if you boil down what your advisor learned from
Speaker 2: Shannon and what you learned from your advisor, what would
Speaker 2: it be? What did you learn?
Speaker 1: Yeah, and learn about taking a very complicated problem and
Speaker 1: strip it down to the essential and then formulate a
Speaker 1: problem around that and solve it. That's an art. It's
Speaker 1: not something you can to convert it into a mathematical
Speaker 1: formula and teach students. It's just based on intuition, experience.
Speaker 1: And that's what Shannon talked my advisor, and that's what
Speaker 1: my advisor taught me, and that's what I try to
Speaker 1: teach my students. Really, teaching is not really about giving
Speaker 1: the follower is really just learning by examples. I observe
Speaker 1: what he does, and then my students observe what I
Speaker 1: do as I interact with them. And hopefully this art
Speaker 1: will carry on from generation to generation.
Speaker 2: Finding the essence of the problem.
Speaker 1: Yeah, David.
Speaker 2: She is a professor at Stanford. Today's show was produced
Speaker 2: by Gabriel Hunter Chang. It was edited by Lyddy Jean
Speaker 2: Kott and engineered by Sarah Bruguier. You can email us
Speaker 2: at problem at Pushkin dot FM. I'm Jacob Goldstein and
Speaker 2: we'll be back next week with another episode of What's
Speaker 2: Your Problem.