Speaker 1: Bloomberg Audio Studios, Podcasts, radio News.
Speaker 2: Hello and welcome to another episode of the Odd Lots podcast.
Speaker 2: I'm Jill Wisenthal.
Speaker 3: And I'm Tracy Alloway.
Speaker 2: Tracy, here's something I know about AI. I don't know much,
Speaker 2: but here's something.
Speaker 4: I do know.
Speaker 3: How to log into chat GPT.
Speaker 2: No, I'm good at it. I'm good at that. I'm
Speaker 2: good at logging into chat GPT and claude, and I'm
Speaker 2: reasonably good at asking questions. Now, here's actually something about
Speaker 2: the actually about the business of AI that I know.
Speaker 3: Okay.
Speaker 4: I know that in.
Speaker 2: Video is making a ton of money and the stock
Speaker 2: has gone to the moon, and that other companies would
Speaker 2: like a slice of that pie.
Speaker 3: Yes, yes, that's a good thing to know.
Speaker 2: It's like a basic, simple thing, which is that when
Speaker 2: people think about AI chips, there's literally one company that
Speaker 2: comes to mind. I know others are involved. AMD has stuff,
Speaker 2: Intel obviously wants to play others, but there is obviously
Speaker 2: that one gigantic pile of cash that's flowing to this
Speaker 2: one company. I don't know if it's still but at
Speaker 2: one point, is the biggest company in the world is
Speaker 2: pulled back.
Speaker 4: A little bit.
Speaker 2: Well, I would say two things. One, other companies would
Speaker 2: like that a piece of that pie. And b companies
Speaker 2: that are in the business of building AI models would
Speaker 2: like to find a way to get cheaper, more efficient,
Speaker 2: less energy intensive chips so that they don't have to
Speaker 2: always pay the Nvidia tax.
Speaker 3: Do you want to know what I know about AI
Speaker 3: and semiconductors, Let's go for it. Okay, here's the one
Speaker 3: thing that I know, which is that whenever you have
Speaker 3: this conversation about in Nvidia, the one word that always
Speaker 3: comes up is moat.
Speaker 2: Oh yes, moat yeah.
Speaker 3: So, like you're either talking about like medieval castles or
Speaker 3: you're talking about semiconductor manufacturing. That's when you hear the
Speaker 3: word mote because over and over again people will say
Speaker 3: it is expensive to make the chips. You need a
Speaker 3: lot of money for research and development and to set
Speaker 3: up the fabs, and you need a lot of first
Speaker 3: person expertise in building them. And then there's also the
Speaker 3: network effect. So a company like Nvidia has this huge
Speaker 3: moat around its business. The question, of course, is whether
Speaker 3: or not, getting back to the medieval castle analogy, it
Speaker 3: is unassailable, that's right.
Speaker 2: If semiconductor seems to be mote after MOTI, after mode,
Speaker 2: because there's ASML's moat, and then there's Taiwan Semiconductor's moat,
Speaker 2: and then there's Nvidia's moat, and so yes, it's like
Speaker 2: there's a series of moats, and if someone could overcome
Speaker 2: these moats or make find a way to build a
Speaker 2: bridge over one of these moats and enter this proverbial castle,
Speaker 2: that would be very lucrative. We know that many are
Speaker 2: trying to enter these moats, but it's incredibly costly and
Speaker 2: capital intensive and difficult. There are just not many people
Speaker 2: who know how to do any of this stuff, and
Speaker 2: so the question of whether these modes can be overcome.
Speaker 2: But again, there are many businesses that would love to
Speaker 2: see more robust competition in the space so that their
Speaker 2: payment is not a attack.
Speaker 3: You know, one thing I don't know, and I don't
Speaker 3: think we've ever done an episode purely on this, but
Speaker 3: I don't really understand the different designs of chips. So
Speaker 3: I know that some chips, specifically in videos, are supposed
Speaker 3: to be better at AI. They're better at running lots
Speaker 3: of little calculations all at the same time. And I
Speaker 3: know there's basic chips that go into your refrigerator or
Speaker 3: your car or whatever. But I don't really know the
Speaker 3: difference between what a chip that was designed specifically to
Speaker 3: run a large language model would look like compared to
Speaker 3: a standard basic chip.
Speaker 2: I don't know anything about chip design. I just sort
Speaker 2: of imagined someone on like using some CADS software, etching
Speaker 2: little lines in the thing and drawing some sort of
Speaker 2: like circuitry or you know, put it place in the trains.
Speaker 3: You know, A chip design game would be really fun,
Speaker 3: now that I think about it. Yeah, you could just
Speaker 3: draw little things on the square. Okay. Anyway, Well, we
Speaker 3: are going.
Speaker 2: To learn about how chip design works. We are going
Speaker 2: to learn about what makes a chip particularly good for
Speaker 2: the task of training and running inference on these AI models.
Speaker 2: And I have to say, I really do believe we
Speaker 2: have the two perfect guests because they are both veterans
Speaker 2: in this space, and they are both active in the
Speaker 2: attempt to bridge some of these motes and enter the
Speaker 2: space and bring competition to the industry. We are going
Speaker 2: to be speaking with yin Or Pope, co founder and
Speaker 2: CEO of Medex, as well as Mike Gunter, co founder
Speaker 2: and CTO of Madex. It's a new company that's trying
Speaker 2: to build chips specifically for the purpose of large language models.
Speaker 2: Both of them have a lot of experience in the
Speaker 2: space we're going to we get our hands dirty, so
Speaker 2: to speak, and understand how you build the hardware for
Speaker 2: all this stuff and what makes it win and whether
Speaker 2: it's even a winnable game. Ryan Or and Mike, thank
Speaker 2: you so much for coming on Outlaws.
Speaker 5: Thanks, happy to be here, pleasure to be here.
Speaker 2: So what do you tell us? What does a chip
Speaker 2: designer do? I know, I have this completely cartoonish view
Speaker 2: in my head that cannot possibly be right of someone
Speaker 2: on a big screen using some CAD software to sort of,
Speaker 2: you know, figure out what's going to be etched in
Speaker 2: that way for of silicon. What is the job of
Speaker 2: chip design?
Speaker 5: So maybe this is best told by what is the
Speaker 5: story of chip development from the beginning of a project
Speaker 5: to the end of it. So there's a range of
Speaker 5: different ways this can go, but there's a lot of
Speaker 5: things that are in common. So generally a chip design
Speaker 5: team is at the low end, maybe thirty people, up
Speaker 5: to many many thousands of people at the high end,
Speaker 5: and it as the project typically runs for somewhere in
Speaker 5: the range of three to five years from conception to
Speaker 5: actually shipping to customer, and so over that time what
Speaker 5: we see in the life cycle is we tend to
Speaker 5: start with a small team of architects. If you think
Speaker 5: of designing a house, the team of architects are the
Speaker 5: people who decide what rooms go in here, or how
Speaker 5: many bedrooms, how many bathrooms, what are the flows between them,
Speaker 5: how do people walk through the corridors, and so on,
Speaker 5: the coarse grained design of the chip, in the chip itself,
Speaker 5: that is, you know what kinds of components at the
Speaker 5: high level we have, and then after that initial exploration,
Speaker 5: this moves then over to the micro architects. These are
Speaker 5: the people who are designing the individual rooms. What are
Speaker 5: the components that go in the individual rooms. So at
Speaker 5: that point everything we've done so far is a design
Speaker 5: stage thing. This is done in documents, spreadsheets, and it's
Speaker 5: a verbal and human communication form. But beyond that, that's
Speaker 5: when it starts to actually touch the computer in a
Speaker 5: more meaningful sense. And so the micro architects will hand
Speaker 5: over to the logic designers. They are the people who
Speaker 5: are actually writing code. So even though you think of
Speaker 5: chips as being this very physical thing where there's wires
Speaker 5: and gates and everything. The way we try to admit
Speaker 5: this information to the computer is actually writing code. We
Speaker 5: write verolog that expresses the design of the chip. So
Speaker 5: that's what the logic designers are doing. That's an extended
Speaker 5: period of time building out all of the different you know,
Speaker 5: matrix multiplies, memories, circuitry that connects to the outside world,
Speaker 5: and so on. And then the output of all of
Speaker 5: them is this verolog piece of software code that gets
Speaker 5: then compiled by a computer down to a set of
Speaker 5: gates which are logic gates and or gates and so on.
Speaker 5: And then why is that connect them together? That's the
Speaker 5: netlist this file. Then there's a few more stages still
Speaker 5: coming here. This file gets handed off to physical designers,
Speaker 5: who again work with CAD tools to convert this kind
Speaker 5: of logical discussion.
Speaker 2: Was right, Someone is using CAD tools.
Speaker 5: Absolutely, there's a CAD tool, but it's it's only out
Speaker 5: of the job. Okay, So the physical designers are converting
Speaker 5: the sort of logical description into a physical placement. So
Speaker 5: where do each of these gates go? Now there's two
Speaker 5: hundred billion logic gates on a chip, so a human
Speaker 5: is not going to be placing all of those manually.
Speaker 5: So there's a huge amount of software assistance here. But
Speaker 5: what the human is doing is providing oversights through this
Speaker 5: process and saying, I've done this a ton of times before.
Speaker 5: This placement kind of looks wrong, it doesn't match my heuristics,
Speaker 5: and so I can probably do a better job here.
Speaker 5: So that's the physical designers, and the output of their
Speaker 5: work is actually eventually you get a polygons, so basically
Speaker 5: an image saying here is the thing that is going
Speaker 5: to get etched onto a piece of silicon. So that
Speaker 5: file is ultimately a huge, like really big image in
Speaker 5: some form a bunch of polygons on it. It gets
Speaker 5: handed over to a manufacturing company such as TSMC. They
Speaker 5: spend maybe four or five months initially creating a mask set,
Speaker 5: so those are like the templates or the stencils that
Speaker 5: will be used to stamp out many many copies of
Speaker 5: the chip, and then stamps up many copies of the chip.
Speaker 5: You get a chip back. This is typically about two
Speaker 5: or three years after you started the project. You get
Speaker 5: chips back, and now you have a bring up team
Speaker 5: who puts this chip into a whole board and connected
Speaker 5: to what to power and electricity and starts testing it,
Speaker 5: and then after another six to twelve months or maybe
Speaker 5: even more, eventually you actually can hand this over to customers.
Speaker 5: There's maybe just one or two other things which are
Speaker 5: not in that flow but very essential to call out too.
Speaker 5: Are because of this whole process taking so long, especially
Speaker 5: the manufacturing, we also have like very large teams of
Speaker 5: verification people. So these are the people who before we
Speaker 5: actually send it to manufacturing and pay twenty to thirty
Speaker 5: million dollars of manufacturing, we have a substantial team doing
Speaker 5: a lot of testing. And this is software based testing,
Speaker 5: so writing tests in the same way a software engineer
Speaker 5: might to make sure that the functionality actually works as intended.
Speaker 6: To underlying the comparison to ordinary software, which Reiner touched
Speaker 6: on it, we're writing code, but it's on super hard mode.
Speaker 6: So if you have a if you have a software
Speaker 6: that's deployed the website, you can fix a bug and
Speaker 6: you know, ten minutes at basically zero cost. Whereas in
Speaker 6: our case, the reason that we have a large team
Speaker 6: people doing verification making sure that what we've done is
Speaker 6: correct is that it's potentially four months and thirty million
Speaker 6: dollars for every mistake that you let through. Likewise, there
Speaker 6: is software, but it's a relatively small fraction of software
Speaker 6: that's very performance critical where you want the code to
Speaker 6: run as fast as possible. But in some sense, every
Speaker 6: line of code that you write in hardware has an
Speaker 6: impact on the overall performance of the product, because every
Speaker 6: line of code ends up getting embodied in silicon, and
Speaker 6: every line of code affects the eventual performance. So it's
Speaker 6: kind of coding, but on hard mode.
Speaker 3: So I intuitively understand the importance of getting the software right.
Speaker 3: But why does placement on the actual chip or wayfer
Speaker 3: Why does that matter? Are you trying to make it
Speaker 3: more efficient, are you trying to reduce the rise time?
Speaker 3: Or why does it matter where the little bits and
Speaker 3: bobs are placed? To use the scientific.
Speaker 6: Term, Yeah, you're right that reducing the right time is
Speaker 6: a massive issue. And you know, fundamentally the issue is
Speaker 6: that chips, you know, at a very abstract level, are
Speaker 6: composed of were at a somewhat content concrete level, really
Speaker 6: are composed of transistors and wires, and the placement has
Speaker 6: a dramatic effect on the link through the wires, which
Speaker 6: has a dramatic effect on both the performance of the
Speaker 6: chip and how much you can fit. In terms of
Speaker 6: the impact that this has on the quality of chip
Speaker 6: that you produce, wires have over time not been shrinking
Speaker 6: in the same way that transistors have, and so getting
Speaker 6: the wearing right, which usually means getting the placement right,
Speaker 6: has become more and more important over time.
Speaker 3: Can chips be beautiful? I know code can be elegant,
Speaker 3: and some people will say certain code is beautiful, But
Speaker 3: have you ever looked at a semiconductor and been like, oh, wow,
Speaker 3: that's really nicely put together.
Speaker 5: For me, I mean I think absolutely yes. This is
Speaker 5: like why I work in this space is I just
Speaker 5: really like geeking out on the design of things. But
Speaker 5: to me, what beautiful for a chip means is that
Speaker 5: it kind of does exactly what it was designed to do,
Speaker 5: and no more and no less. I mean, obviously less
Speaker 5: would be a bit of a disappointment, but often if
Speaker 5: it does more, do you think, well, maybe I designed
Speaker 5: it for slightly the wrong purpose or something like that.
Speaker 2: I think this is a good seg into getting into
Speaker 2: your business specifically, so we all know that so much
Speaker 2: of this AI is powered by these in video GPUs,
Speaker 2: but in video GPUs have been used for a long
Speaker 2: time for many things that do not have anything to
Speaker 2: do with large language models or the specific AI applications
Speaker 2: that people are excited about today in twenty twenty four.
Speaker 2: So for a while they were, well, the video games
Speaker 2: is obviously the big one for decades and decades, and
Speaker 2: then there was like five minutes where people got really
Speaker 2: excited to use them for ethereum mining, and now everyone's
Speaker 2: really excited about their use for artificial intelligence and large
Speaker 2: language models and some of these other generative AI applications
Speaker 2: that people are excited about right now, Why don't you
Speaker 2: tell us maybe the sort of idea behind maddex, but
Speaker 2: specifically what you were both doing when you were at
Speaker 2: alphabet or Google, which you know it has its own chips.
Speaker 2: I believe it has something called TPUs. What was the
Speaker 2: project at Google? Why did Google find it necessary or
Speaker 2: a good business to start building their own chips for
Speaker 2: in house purposes? And then why did you feel the
Speaker 2: need to then leave to build what you're building now
Speaker 2: for LLM specifically?
Speaker 6: Yeah, So what Google was seeing, and this was at
Speaker 6: this point sometime back more than a decade ago, they
Speaker 6: were seeing that the use of artific intelligence lllms were
Speaker 6: not a thing at that point, was going up, and
Speaker 6: they were worried about how much money they would have
Speaker 6: to spend on traditional it would be it would have
Speaker 6: been GPUs at that time, and so they built a
Speaker 6: very specialized chip to do neural nets, and that chips
Speaker 6: specialize on matrix multiplication. So they put in a structure
Speaker 6: called a systolic array, which they definitely didn't invent. It existed,
Speaker 6: has existed from the seventies that is especially good at
Speaker 6: doing matrix multiplication. Now after that, Nvidia has added a
Speaker 6: similar structure into their chips. And the initial Google TPU
Speaker 6: was an inference focused only chip, and then they have
Speaker 6: subsequently made chips that can be used for both training
Speaker 6: and inference. And I guess now is a good point
Speaker 6: to So the very last thing that I was doing
Speaker 6: at Google was I was on the TPU team and
Speaker 6: Reiner was on the large language model team. And it's
Speaker 6: probably good to have him sort of tell free from here.
Speaker 5: So I mean, what we were seeing and this this
Speaker 5: is what we personally were seeing, but Google was seeing
Speaker 5: more generally as well. Is just large language models were
Speaker 5: a thing. There was this period of time between GPT
Speaker 5: three and chat GIPT coming out. GPT three came out
Speaker 5: in twenty twenty, and so people who were very plugged
Speaker 5: into the field recognized the importance of it all at
Speaker 5: least to some extent, recognized the importance of it back then,
Speaker 5: and so there was this push to you know, everyone
Speaker 5: wanted to create their own large language model that was
Speaker 5: better than GPT three, and so, I mean, at the time,
Speaker 5: I was on the Large Language Model team. We helped
Speaker 5: training Google Palm, and we were using thousands of TPUs
Speaker 5: for that, and one of the things we were saying is, well,
Speaker 5: look what does it cost to deploy this? In Google Search?
Speaker 5: There's quite a lot of search querers. I think it's
Speaker 5: the public estimates thro about one hundred thousand of them
Speaker 5: per second. If you multiply out how much each querer costs,
Speaker 5: and if you want to run that on large language models,
Speaker 5: that's a lot more expensive. And then also I just
Speaker 5: if I want to train a model that's times bigger
Speaker 5: than my current model or one hundred times bigger, suddenly
Speaker 5: these models have just moved from costing you know, a
Speaker 5: million dollars or one hundred thousand dollars to train to
Speaker 5: tens of millions and hundreds of millions of dollars, and
Speaker 5: so the overall goal was can we make it cheaper
Speaker 5: by any way possible. So, of course there's algorithmic approaches.
Speaker 5: There's a lot of opportunity on the algorithm and research side.
Speaker 5: But then the other really big lever is just making
Speaker 5: better hardware. So one of the things we were looking
Speaker 5: at was trying to make Google's TPUs better for large
Speaker 5: language models. What led us, actually, i mean this is
Speaker 5: personally about Mike and me in this case, or what
Speaker 5: led us to leave Google to make medics was we
Speaker 5: saw that there was We believe that there is some
Speaker 5: opportunity to make chips substantially better if you're only looking
Speaker 5: to focus on large language models. And so the chips
Speaker 5: that were designed pre GPT three and especially pre chat
Speaker 5: GPT try to do a really good job on really
Speaker 5: good job on small models as well as a really
Speaker 5: good job on large models. And so what you find
Speaker 5: is that the circuitry in those chips, there's a bit
Speaker 5: of circuitry for what you need for small models, there's
Speaker 5: a bit of secretry for what you need for large models.
Speaker 5: Also for maybe embedding look ups. There's three or four
Speaker 5: different kinds of workloads, and all of them take some
Speaker 5: of the real estate in your cellica. And so if
Speaker 5: you really want to make the best use of the
Speaker 5: real estate, you should just focus on the thing you
Speaker 5: care about most and hope that there's a big market there.
Speaker 5: So that the game and or what we decided to
Speaker 5: do when we see some others deciding to do as well,
Speaker 5: is to really try and focus on just the one
Speaker 5: workload that seems like it's going to become a one
Speaker 5: hundred billion dollar or a trendion dollar industry.
Speaker 2: I know there's always this sort of cliche when talking
Speaker 2: about techno. Oh, Google and Facebook, they can just build
Speaker 2: this and they'll destroy your little startup because they have
Speaker 2: infinites amounts of money. Except that doesn't actually seem to
Speaker 2: happen in the real world as much as people on
Speaker 2: Twitter expect it to happen. But can you just sort
Speaker 2: of give a sense of maybe the business and organizational
Speaker 2: incentives for why a company like Google doesn't say, oh,
Speaker 2: this is one hundred billion dollar market in video is
Speaker 2: worth three and a half trillion or three trillion dollars,
Speaker 2: let's build our own LM specific chips. Why doesn't that
Speaker 2: happen at these large, hyperscaler companies that presumably have all
Speaker 2: the talent and money to do it.
Speaker 6: So Google's TPUs are primarily built to serve their internal customers,
Speaker 6: and Google's revenue for the most part comes from Google
Speaker 6: Search that Google Search, and in particular from Google Search ads.
Speaker 6: Google Search ads. Is you know, a customer of the TPUs,
Speaker 6: It's a relatively difficult thing to say that hundreds of
Speaker 6: billions of dollars of revenue that we're making, we're going
Speaker 6: to make a chip that doesn't really support that particularly well,
Speaker 6: and focuses on this at this point unproven in terms
Speaker 6: of revenue market and it's not just ads, but they
Speaker 6: are you know, a variety of other customers. For instance,
Speaker 6: you know, you may have noticed how Google is pretty
Speaker 6: good at identifying good photos and doing a whole variety
Speaker 6: of other things that are supported in many cases by
Speaker 6: the TPUs.
Speaker 5: I think one of the other things too, that we
Speaker 5: see in all chip companies in general, or companies producing chips,
Speaker 5: is because producing chips is so expensive, you end up
Speaker 5: in this place where you really want to put all
Speaker 5: your resources behind one chip effort. And so just because
Speaker 5: the thinking is that there's a huge amount of return
Speaker 5: on investment in making this one thing better rather than
Speaker 5: fragmenting your efforts. Really, what you'd like to do in
Speaker 5: this situation where there's a new emerging field that might
Speaker 5: be huge or might not, but it's hard to say yet,
Speaker 5: what you'd like to do is maybe spin up a
Speaker 5: second effort on the side and have like a skunk works. Yeah,
Speaker 5: that's work, right.
Speaker 2: That would be just to let Ryan er and just
Speaker 2: let the two of you go have your own little
Speaker 2: office somewhere else.
Speaker 5: Yeah, just organizationally that it's often challenging to do, and
Speaker 5: we see this across all companies. Every chip company really
Speaker 5: has essentially only one mainstream chip product that is that
Speaker 5: they're iterating on and making better and better over time.
Speaker 3: To what degree is to design driven by the customer?
Speaker 3: And what I mean by that is, so the TPUs
Speaker 3: at Google were developed to handle Google's internal workloads, but
Speaker 3: at other chip designers, to what degree will customers come
Speaker 3: and like basically do a reverse inquiry and ask for
Speaker 3: a specific chip or what does the dialogue between customers
Speaker 3: and the big chip designers actually look like.
Speaker 5: Yeah, it's a fun interplay of I want my provider
Speaker 5: to do a good job, but I also don't want
Speaker 5: to leak my IP too much. So you can see
Speaker 5: this how this played out in so Mike was talking
Speaker 5: about through the development of the TPUs which were publicly
Speaker 5: announced in twenty sixteen and around the same time in
Speaker 5: videos first GPU with the tens ocos, So that was
Speaker 5: the first GPU that was really focused on matrix multiplication.
Speaker 5: That was the vaulted generation came out at about the
Speaker 5: same time. And some of this actually was a result
Speaker 5: of when Google had this recognition of look, matrix multiplication
Speaker 5: is so important, we need to make it really better.
Speaker 5: They simultaneously work themselves but also went to Nvideo and
Speaker 5: said we're not telling you much, but can you do
Speaker 5: better at matrix multification? And so that was enough for
Speaker 5: Nvidia to go on the first generation. They made a
Speaker 5: pretty good attempt. But if you talk to people that
Speaker 5: in video, I'll say that actually the second generation of
Speaker 5: the tensacle which was in the MPa generation, was where
Speaker 5: they really nailed it. So when it's big enough, you
Speaker 5: sometimes see these customers coming and saying what they want,
Speaker 5: but they'll maybe they'll try and disguise what they're asking
Speaker 5: for or not giving you the absolute minimum amount of
Speaker 5: information to help a vendor make what they want without
Speaker 5: revealing too much about their.
Speaker 4: IB Let's get to maddex.
Speaker 2: Tell us the product that you're designing and how it
Speaker 2: fundamentally will differ from the offerings on the market, most
Speaker 2: notably from VideA.
Speaker 4: Yeah.
Speaker 5: Yeah, So we make chips and in fact racks and
Speaker 5: clusters for large language models. So when you look at
Speaker 5: in videos GPUs, you already talked about all of this,
Speaker 5: the original background in gaming, this brief movement in ethereum,
Speaker 5: and then even within AI they're doing small models of
Speaker 5: large models. So what that translates to in you can
Speaker 5: think of it as the rooms of the house or something.
Speaker 5: They have a different room for each of each of
Speaker 5: those different use cases, so different circuitry in the chip
Speaker 5: for all of these use cases. And the fundamental bet
Speaker 5: is that if you say, look, I don't care about that,
Speaker 5: I'm going to do a lousy job if you try
Speaker 5: and run a game on me, or I'm going to
Speaker 5: do a lousy job if you want to run a
Speaker 5: convolutional network on me. But if you give me a
Speaker 5: large model with very large matrices, I'm going to crush it.
Speaker 5: That's the bet that we're making amatix, so we spend
Speaker 5: as much of our silicon as we can on making
Speaker 5: this work. There's a lot of detail in making all
Speaker 5: of this work out, because you need not just the
Speaker 5: matrix multiplication, but the all of the memory bandwidths and
Speaker 5: communication bandwidths and the actual engineering things to make a
Speaker 5: pen out. But that's the core bette.
Speaker 3: And why can't Invidia do this? So you know, in
Speaker 3: Video has a lot of resources, It has that big
Speaker 3: moat as we were discussing in the intro, and it
Speaker 3: has the GPUs that are already in production and working
Speaker 3: on new ones. But why couldn't it start designing an
Speaker 3: LM focused chip from scratch?
Speaker 6: Right? So you talked about in Vidia's mode and that
Speaker 6: moat has two components. One component is that they build
Speaker 6: the very best hardware, and I think you know that
Speaker 6: is the result of having a very large team that
Speaker 6: executes extremely well and making good choices about how to
Speaker 6: serve their market. They also have a tremendous software mode.
Speaker 6: And you know, both of these moats are important to
Speaker 6: different sets of customers, so they're tremendous software mode. They
Speaker 6: have a very broad, deep software ecosystem based on Kuda
Speaker 6: that allows it.
Speaker 3: Oh yeah, I remember this came up in our discussion
Speaker 3: with core Weave.
Speaker 6: Yeah yeah. And so that allows customers who are not
Speaker 6: very sophisticated, who don't have gigantic engineering budgets themselves, to
Speaker 6: use those chips and using videos chips and be efficient
Speaker 6: at that. So the thing about a mote is not
Speaker 6: only does it in some sense keep other people out,
Speaker 6: it also keeps you in. So insofar as they want
Speaker 6: to keep their software mode, their Kuda mote, they have
Speaker 6: to remain compatible with Kuda and compatibilility with that software mode.
Speaker 6: Compatibilility with Kuda requires certain hardware structures. So in Videos
Speaker 6: has lots and lots of threads, they have a very
Speaker 6: flexible memory system. These things are great for being able
Speaker 6: to flexibly address a whole bunch of different types of
Speaker 6: neural net problems, but they all cost in terms of hardware,
Speaker 6: and they're not necessarily those The choices to have those
Speaker 6: sorts of things are not necessarily the in fact, not
Speaker 6: the choices that you would want to make if you
Speaker 6: were aiming specifically at an LM. So in order to
Speaker 6: be you know, fully competitive with a chip that's specialized
Speaker 6: for LLMS, they would have to give up all of that.
Speaker 6: And you know, Jensen himself has said that the one
Speaker 6: non negotiable rule in our company is that we have
Speaker 6: to be compatible with kuda.
Speaker 2: This is interesting. So the challenge for them of spinning
Speaker 2: out something totally different is that it would be outside
Speaker 2: the family. And so it's outside the Kudah family, so
Speaker 2: to speak.
Speaker 3: And meanwhile, you already have like pie, Torch and Triton
Speaker 3: waiting in the wings.
Speaker 2: I guess, so why don't you tell us a little
Speaker 2: bit more about the business of LLLM chips specifically, because
Speaker 2: there's a lot of questions, Like, you know, one question
Speaker 2: is you have all these people in Silicon Valley who
Speaker 2: seem motivated by the idea of like agi that that's
Speaker 2: the goal, that we're going to have super intelligence one day,
Speaker 2: maybe thousands IQs and the hundreds of thousands one day.
Speaker 2: That'll make us all seem very dumb, et cetera. Are
Speaker 2: you implicitly making a bet by your company that it'll
Speaker 2: be lllms that we'll get there, Because as you mentioned,
Speaker 2: there are other algorithmic ideas, There are other ideas for
Speaker 2: how you might be able to expand intelligent. How much
Speaker 2: of your company's bet is the idea that the future
Speaker 2: of generative AI or as we know it, is going
Speaker 2: to be along the LLLM pathway.
Speaker 5: One of the core things. I think there's two core
Speaker 5: ingredients of the LM pathway. Yeah, one so far is
Speaker 5: the transformer architecture, which is a model architecture and was
Speaker 5: substantially better than the things that came before. But the
Speaker 5: other one, and that actually has a much longer history,
Speaker 5: is the scaling hypothesis in hypothesis in general sore, But
Speaker 5: that's the there's a general observation which has been widely
Speaker 5: recognized for a decade or more that if I am sorry,
Speaker 5: I'm training in neural net or some kind of AI model,
Speaker 5: if I want to make its quality better and make
Speaker 5: it bigger, and so what does bigger mean? Bigger means
Speaker 5: I have to spend more compute training it. Bigger means
Speaker 5: I have more neurons. Tho's are the loosely analogous to
Speaker 5: the sort of processing power in a human brain, although
Speaker 5: analogy is weak. If I make my model bigger, I
Speaker 5: get better quality. That's a sort of simple qualitative thing
Speaker 5: to say, and that's been true for a really long
Speaker 5: time in these models. So the advantage of that, or
Speaker 5: the thing that we've seen really recently is we've seen
Speaker 5: this turned up to eleven. So around the time when
Speaker 5: GPT three came out, So in twenty twenty, a paper
Speaker 5: was published called the Scaling Laws, and so this took
Speaker 5: this qualitative observation and made it quantitative and said, actually,
Speaker 5: we can even fit an equation to it, and so
Speaker 5: that gave people a lot more conviction to it. And
Speaker 5: this is what led to the people saying, well, if
Speaker 5: I have a better model, I can solve more problems
Speaker 5: with AI than I could before. And so every time
Speaker 5: I spend ten times as much training on it, I
Speaker 5: unlock new use cases. And so that's what to this craze.
Speaker 5: And the remarkable thing is that while there are these
Speaker 5: diminishing returns, I have to spend ten times as much
Speaker 5: computing power to get some improvement beyond that sort of
Speaker 5: logarithmic scale. We don't see as yet any plateau an
Speaker 5: so it seems like there continues to be opportunity here.
Speaker 5: So the key thing is this scaling hypothesis or scaling
Speaker 5: laws in general that are causing these models to grow.
Speaker 5: And then I mean as a hardware provider, what you
Speaker 5: might look at is you might say, that's the thing
Speaker 5: I really want to bet on. I want to bet
Speaker 5: on the growth of models, and I mean, now it's
Speaker 5: a little more in the details, but the thing you
Speaker 5: actually have to bet on is the growth of matrix sites,
Speaker 5: which is very strongly correlated with the growth of models.
Speaker 3: Just to hammer this point home, if more AI was
Speaker 3: learning from stuff like self play or synthetic data rather
Speaker 3: than scraping the internet, would the design of the chips
Speaker 3: have to take that into account, Like, how would the
Speaker 3: chips vary between those different learning styles.
Speaker 5: Yeah, so in general, when you're building a chip, you
Speaker 5: have to make it programmable because you're going to make
Speaker 5: this chip and you will ship a new version every
Speaker 5: two years, but what people want to do with the
Speaker 5: chip is going to change every month or so, so
Speaker 5: it has to be programmable to some extent. So that's
Speaker 5: true for all of the chips that anyone ships, and
Speaker 5: so there's different scales of programmability and what kinds of
Speaker 5: changes you need to adapt to, So changes in kind
Speaker 5: of the way you feed it data that's maybe on
Speaker 5: the very very outer layers of doesn't affect much of
Speaker 5: the core of the chip, and so those kinds of
Speaker 5: changes tend to be some of the easier changes to
Speaker 5: adapt to. The things that then become a little harder
Speaker 5: to adapt to is if I'm substantially changing my model architecture.
Speaker 5: So a small change might be maybe I change the
Speaker 5: number of layers, or I reorder some of the layers
Speaker 5: in my model, or maybe I use the same ingredients
Speaker 5: but shuffle them around in some way. A bigger change
Speaker 5: would be that say, Okay, I'm actually going to throw
Speaker 5: out all of these ingredients and use a completely different
Speaker 5: set of primitives. And that's often that's that last step
Speaker 5: is the one that that would really kill you if
Speaker 5: you're betting very much on it. Partic a set of ingredients.
Speaker 6: So an example of a potential different set of primitives
Speaker 6: that are used in other models that aren't used in
Speaker 6: llms are we made mention of these embedding things that
Speaker 6: are used in recommender and ad models. So Facebook has
Speaker 6: talked about building special purpose hardware to support inference on
Speaker 6: those kind of models. Those are they have much less
Speaker 6: emphasis relative emphasis, particularly on matrix multiply. Another possible direction
Speaker 6: that model architecture could go. That would be different and
Speaker 6: bad for a chip designed for current LLMS, would be
Speaker 6: instead of having very large matrices in about one hundred layers,
Speaker 6: you could have much smaller matrices but ten thousand layers,
Speaker 6: and that would demand a different sort of design to
Speaker 6: be good at that kind of model. So a bet
Speaker 6: that looks good given the modern history of neural nets
Speaker 6: is that matrices will get larger over time.
Speaker 2: You know, you're talking about scaling laws, and so everyone
Speaker 2: talks about okay, computation, power, energy efficiency, et cetera, and
Speaker 2: I never know if they're true. But then sometimes you
Speaker 2: read these stories they're like Sam Altman wants to go
Speaker 2: around the world and raise like five trillion dollars to
Speaker 2: like build his own semiconductor fabs and have the entire architecture,
Speaker 2: because that's like what it's going to take. What about
Speaker 2: the data side, because this is another thing people talk about,
Speaker 2: the data wall that you know, there's only one Internet
Speaker 2: to scrape, and then you know, after that, what if
Speaker 2: you're not there at AGI yet again, I know you're
Speaker 2: solving for the hardware side, but when you think about
Speaker 2: risks going forward along the LLM pathway, what's your perspective
Speaker 2: on well, what happens when we've just we've ingested all
Speaker 2: the data.
Speaker 5: So there's two ways you can make a model better.
Speaker 5: One of them is by training on more data, and
Speaker 5: the other one is making a bigger model. And these
Speaker 5: two effects work in a really complimentary way. So you
Speaker 5: can think of a like having a bigger brain and
Speaker 5: then practicing more and so both of these are going
Speaker 5: to help to some extent. So there's a risk that
Speaker 5: we hit a data wall. In general, there's been a
Speaker 5: long history of people predicting walls in different kinds of
Speaker 5: walls in techno training and then ingenuity overcoming this, and
Speaker 5: so I wouldn't I would bet that there's a fairly
Speaker 5: large amount of mileage to continue here. Tracy mentioned self
Speaker 5: training and generating new data. That's the vibe in the
Speaker 5: industry is that this is a promising direction for sure.
Speaker 5: But even if you don't bet on that, there's mileage,
Speaker 5: and it's less attractive mileage, but there is mileage in
Speaker 5: making the models bigger. So I believe, and I think
Speaker 5: this is shared by many people insiders in the industry
Speaker 5: as well, is that there's at least a few more
Speaker 5: orders of magnitude available here before we run out of
Speaker 5: easy engineering knobs to turn. But of course, one of
Speaker 5: the limiting factors here is just the dollars you spend.
Speaker 5: So you have some amount of budge that I'm willing
Speaker 5: to spend. And I mean, maybe Sam can raise five
Speaker 5: trillion dollars, I don't think necessarily everyone else can raise
Speaker 5: that amount of money to train a model. And so
Speaker 5: if you've got a fixed amount of dollars that you
Speaker 5: want to spend, and you want to train the best model,
Speaker 5: you want to make the best use of the multipliers,
Speaker 5: you want to make the best use of the dollars
Speaker 5: you spend, and so that means fundamentally, what you're paying
Speaker 5: for is the flops, which flops is a floating point operation,
Speaker 5: so the number of multipliers you can do. And then
Speaker 5: every time I increase my model size or increase the
Speaker 5: amount of training data I've got, I'm spending more flops,
Speaker 5: and so flops converts into intelligence. And then if I've
Speaker 5: got a fixed budget, really what I want to maximize
Speaker 5: is my flops per dollar.
Speaker 3: I find this so fascinating because there are so many
Speaker 3: different directions that you could theoretically go in, and so
Speaker 3: many decisions that need to be made, you know, do
Speaker 3: you go after of that scale? How do you tailor
Speaker 3: the design for different methods of data input? Although, as
Speaker 3: you said earlier, maybe that's one of the easiest things
Speaker 3: to respond to. But then there are other trade offs
Speaker 3: that you have to think about between speed and power
Speaker 3: consumption and I guess area utilization or the placement of
Speaker 3: all the bits and bobs that we were discussing earlier,
Speaker 3: and cost effectiveness too. How do you balance all those
Speaker 3: elements and are there particular things that you're willing to
Speaker 3: sacrifice for others.
Speaker 6: So different people can choose different targets to go after
Speaker 6: in the market, and so one one target, which you
Speaker 6: could argue in VideA is winning on currently and one
Speaker 6: of the reasons that their chips their products are so
Speaker 6: popular is, as Rayner said, just the amount of flops
Speaker 6: you can get out out of a chip, and if
Speaker 6: all the chips are roughly the same to make, that
Speaker 6: translates into two flops flops per dollar. So another target
Speaker 6: you could also go after would be the time to
Speaker 6: respond to one user so to get the answer back.
Speaker 6: One approach is maxim the throughput that you can have
Speaker 6: and others minimizing the latency, So kind of the difference
Speaker 6: between a seven forty seven flying a group of passengers
Speaker 6: across the country versus an SR seventy one getting there
Speaker 6: in a couple hours but only bringing one or two people.
Speaker 2: Let's talk about the business itself. So you know, in
Speaker 2: the old you know, ten years ago, someone starting a
Speaker 2: tech startup, they you know, get three or four people
Speaker 2: in an office and then they write something up. But
Speaker 2: then they have a code and it doesn't maybe they
Speaker 2: don't even have to raise any money to do it,
Speaker 2: and they certainly don't have to depend on whether Taiwan
Speaker 2: Semiconductor has any capacity at their fab or anything like
Speaker 2: this walk us through the sort of nuts and bolts
Speaker 2: of what it actually takes to build a chip business
Speaker 2: from the ground up, both in terms of costs and
Speaker 2: time and what you have to rely on. You know,
Speaker 2: we've talked about some of the design element, what are
Speaker 2: the business side requirements and what will it take to
Speaker 2: actually succeed.
Speaker 6: So fortunately we've kind of referred to this in multiple places.
Speaker 6: There's a huge ecosystem around designing chips. So there's a
Speaker 6: portion you have to do yourself, and there's a portion
Speaker 6: that you can buy, so the placement of Tracy's bits
Speaker 6: and bobs and also the testing that we've talked about.
Speaker 6: There are DA electronic design automation companies that build those tools,
Speaker 6: like there are companies that do just manufacturing, so TSMC
Speaker 6: and their suppliers, and then there are many other other companies.
Speaker 6: So most companies don't go directly to TSMC. So so
Speaker 6: very sophisticated companies like Apple or Nvidia interface directly with them,
Speaker 6: but most other companies go through ACIC vendors. And so
Speaker 6: you know, the prominent companies in the most prominent companies
Speaker 6: in that space are Broadcom and Marvel, and then there
Speaker 6: are a bunch of smaller companies. A couple that are
Speaker 6: close to TSMC are all Chip and GUC and so
Speaker 6: they'll do a lot of the work of taking your
Speaker 6: code and actually getting it placed on the chip. That's
Speaker 6: often a very good thing to outsource because it's the
Speaker 6: work is somewhat seasonal. You're only ready to do that
Speaker 6: placement when you're near the end of this three year project,
Speaker 6: and so you kind of don't have work unless you're
Speaker 6: a massive company for people the whole time. So while
Speaker 6: that ecosystem means that you don't have to hire a
Speaker 6: ton of a huge number of people yourself. All of
Speaker 6: those people have to get paid, and so you do
Speaker 6: have to raise a fair bit of money. And another
Speaker 6: big element of actually thing that you end up spending
Speaker 6: money on is there are parts of the chip that
Speaker 6: are very special, difficult to design and take multiple iterations
Speaker 6: of taping things out and seeing if they work. So
Speaker 6: the very high speed interconnect the connects to get connects
Speaker 6: together chips is an example that. So those are designed
Speaker 6: by yet another set of companies, and the design is
Speaker 6: difficult and fairly expensive because of the need to do
Speaker 6: multiple tapeouts, and so it's very fairly expensive to buy
Speaker 6: that IP. So when you add up the cost of
Speaker 6: the IP, the cost of the ASK vendors services, and
Speaker 6: then the mask fees that TSMC charges using ASMLS and
Speaker 6: ASK creation software, you're talking about tens of millions of
Speaker 6: dollars to bring a state of the art chip to market.
Speaker 6: It's the numbers are much lower for a simpler chip
Speaker 6: on it without the very high speed iOS and on
Speaker 6: an older node, but for an advanced node it's a
Speaker 6: pretty expensive process.
Speaker 3: When do you think you'll be able to bring your
Speaker 3: chips to market.
Speaker 5: Generally, we see these projects taking three to five years
Speaker 5: across most companies. We started on this seriously at the
Speaker 5: beginning of twenty four, so about three years from there
Speaker 5: is likely for us.
Speaker 2: Tell us about what customers because I've heard this, you know,
Speaker 2: we're all trying to find some alternative to video, whether
Speaker 2: it's to reduce energy costs or just reduce costs in general,
Speaker 2: or be able to even access chips at all, since
Speaker 2: not everyone can get them because there are only so
Speaker 2: many chips getting made. But when you talk to like
Speaker 2: theoretical customers, A, who do you imagine as your customers?
Speaker 2: Is it the open eyes of the world, is it
Speaker 2: the metas of the world. Is it labs that we
Speaker 2: haven't heard of yet that could only get into this
Speaker 2: if there were sort of more focused, lower cost options.
Speaker 2: And then b what are they asking for? What do
Speaker 2: they say, like, you know what we're using in video
Speaker 2: right now, but we would really like X or Y
Speaker 2: in the ideal world.
Speaker 5: So there's a range of possible customers in the world.
Speaker 5: The way that we see or away you divide them up,
Speaker 5: and how we choose to do that is what is
Speaker 5: the ratio of engineering time they're putting into their work
Speaker 5: versus the amount of computers spent that they're putting in.
Speaker 5: So the ideal customer in general for a hardware vendor
Speaker 5: who's trying to make the absolute best, but not necessarily
Speaker 5: easiest to use hardware is a company that is spending
Speaker 5: a lot more on their computing power than they are
Speaker 5: spending on the engineering type, because then that makes a
Speaker 5: really good trade off of maybe I can spend a
Speaker 5: bit more engineering time to make your hardware work, but
Speaker 5: I get a big saving on my computing costs. So
Speaker 5: companies like open ai would be obviously a slam dunk.
Speaker 5: There's many more companies as well. So the companies that
Speaker 5: meet this criteria of spending many times more on compute
Speaker 5: than on engineering. There's actually a set of maybe ten
Speaker 5: to fifteen large language model labs that are not as
Speaker 5: well known as open ai, but you might think character Ai, Coheer,
Speaker 5: and many other companies like that in mistrial. So the
Speaker 5: common thing that we hear from those companies, all of
Speaker 5: those are spending hundreds of millions of dollars on compute,
Speaker 5: is I just want better flops for dollar. That's actually
Speaker 5: the single deciding factor, And that's primarily the reason they're
Speaker 5: deciding on today, deciding on in videos products rather than
Speaker 5: some of the other products in the market, is because
Speaker 5: the flops for dollar of those products is the best
Speaker 5: you can buy. But when you give them a spec
Speaker 5: sheet and the first thing they're going to look at
Speaker 5: is just what's the most floating point operations I can
Speaker 5: run on my chip? And then you can rule out
Speaker 5: ninety percent of products there on the basis of okay,
Speaker 5: just doesn't meet that far. But then after that you
Speaker 5: then go through the more detailed analysis of saying, okay, well,
Speaker 5: I've got these floating point operations, but is the rest
Speaker 5: going to work out? Do I have the memory bandwidth
Speaker 5: and the interconnect? But for sure, the number one criteria
Speaker 5: is that top line flops.
Speaker 2: When we talk about delivering more flops per dollar, what
Speaker 2: are you aiming for? What is current benchmark flops per dollar?
Speaker 2: And then are we talking like can it be done
Speaker 2: like ninety percent cheaper? What do you think is realistic
Speaker 2: in terms of coming to market with something meaningfully better
Speaker 2: on that metric?
Speaker 5: So in videos, Blackwell in their FP four format offers
Speaker 5: ten pet of flops in their chip, and that chip
Speaker 5: sells for Bullpark thirty to fifty thousand, depends on many factors.
Speaker 5: That is about a factor of two to four better
Speaker 5: than the previous generation and video chip, which is the
Speaker 5: Hopper chip. So part of that factor is coming from
Speaker 5: going to lower precision, going from eight bit precision to
Speaker 5: four bit precision. In general, precision is in one of
Speaker 5: the best ways to improve the flops you can pack
Speaker 5: into a certain amount of silicon, and then some of
Speaker 5: it is also coming from other factors such as cost
Speaker 5: productions that in Vidia has been deployed. So that's a
Speaker 5: benchmark for ware inn video is that now you need
Speaker 5: to be at least a few integer multiples better than
Speaker 5: that in order to compete with the incumbent. So at
Speaker 5: least you know, two or three times better on that metric,
Speaker 5: we would say. But then, of course, if you're designing
Speaker 5: for the future, you have to compete against the next
Speaker 5: generation after that too, and so you want to be
Speaker 5: many times better than the future chip, which isn't down yet,
Speaker 5: And so that's the thing you aim for.
Speaker 2: Is there anything else that we should sort of understand
Speaker 2: about this business that we haven't touched on that you
Speaker 2: think is important?
Speaker 6: One thing, given that this is odd lots that I
Speaker 6: think the reason that sam Altman is going around the
Speaker 6: world talking about trillions of dollars of spend is that
Speaker 6: he wants to move the expectations of all of the
Speaker 6: suppliers up. So as you have we've observed in the
Speaker 6: semiconductor shortage, if the suppliers are preparing for a certain
Speaker 6: amount of demand and demand you know, in the case
Speaker 6: famously of the auto manufacturers as a result of COVID
Speaker 6: canceled their orders and then they found that demand was much, much,
Speaker 6: much larger than they expected. It took a very long
Speaker 6: time to catch up. A similar thing happened with the
Speaker 6: in videos H one hundred. So TSMC was actually perfectly
Speaker 6: capable of keeping up with demand for the chips themselves.
Speaker 6: But the chips for these AI products are use a
Speaker 6: very special kind of packaging which puts the compute chips
Speaker 6: very close to the memory chips and hence allows them
Speaker 6: to communicate very quickly, called coos, And the capacity for
Speaker 6: coos was limited because TSMC built with a particular expectation
Speaker 6: of demand, and when H one hundred became such a
Speaker 6: monster product, their coosts capacity wasn't able to keep pace
Speaker 6: with demand. So, you know, supply chain tends to be
Speaker 6: really good if you predict accurately, and if you predict badly,
Speaker 6: you know, on on the low side, then you end
Speaker 6: up with these shortages. But on the other hand, these companies,
Speaker 6: because the manufacturing companies have very high capex, they are
Speaker 6: fairly lows to it, predict badly on the high side
Speaker 6: because that leads them to having spent a bunch of
Speaker 6: money on capital capex that they're unable to recover.
Speaker 2: So, yeah, this is very interesting, this idea that in
Speaker 2: some part it's a signal we're not slowing down. We're
Speaker 2: you know, we have more and more that we want
Speaker 2: to do. So if you're anywhere along the semiconductor supply chain,
Speaker 2: don't start, you know, curbing your expectations or curbing your
Speaker 2: production because we want to build a lot more. I'm
Speaker 2: curious one last question, I guess for both of you.
Speaker 2: You know, you hear a lot of people in the
Speaker 2: industry you say, like, we might just be three or
Speaker 2: four years away from AGI or super intelligence, however that's defined,
Speaker 2: and then you get into a lot of these philosophical
Speaker 2: questions and ethical questions about you know, whatever is the
Speaker 2: AI going to, well, it's gonna be the role for
Speaker 2: humans or is it gonna kill us all? Or whatever
Speaker 2: you know, fear scenario you want. But the two of
Speaker 2: you like, how do you see that question? Like could
Speaker 2: we hit it in just a few short years where
Speaker 2: we have something that people agree is oh, this is
Speaker 2: agi Like are you is it short runway or just
Speaker 2: a couple of years away from this or does it
Speaker 2: feel like no, that's still quite a few years out.
Speaker 5: If ever, I think what we have what's your.
Speaker 6: Approximately zero to be blunt? Thank you, my p great things.
Speaker 6: I mean, I think we kind of already have great things,
Speaker 6: and we've just gotten the models of this level of
Speaker 6: quality recently and we're learning how to use them, and
Speaker 6: the quality is going up. The you know, the fact
Speaker 6: that we can get a computer to write code pretty
Speaker 6: well is fairly amazing to me. That you can ask
Speaker 6: it to tell a good joke in the style of
Speaker 6: a particular person and it can do that is also amazing. Yeah.
Speaker 2: Well, uh, I'm glad, You're I'm glad, you're I'm glad
Speaker 2: your odds of total doom and annihilation are zero. That
Speaker 2: makes me feel a little bit better. Ryan or and Mike,
Speaker 2: thank you so much for coming on odd laws.
Speaker 7: I learned as from that conversation there's a pleasure.
Speaker 4: Tracy.
Speaker 2: There was obviously ton that was really interesting in that conversation,
Speaker 2: but I particularly like the part about incentives of large
Speaker 2: legacy incumbents about entering a totally new business. So for
Speaker 2: a company like Google, the primary purpose of their chips
Speaker 2: is going to be serving an in house business purpose.
Speaker 2: And even with all the money that they have, and
Speaker 2: even with the engineering talent, there's still a sort of
Speaker 2: trade off question involved of how much do we want
Speaker 2: to build chips for some other purpose, for some sort
Speaker 2: of external service.
Speaker 3: Yeah, and I also thought the point about why Sam
Speaker 3: Altman is going around talking about how, you know, how
Speaker 3: many billions he's going to spend was really interesting and
Speaker 3: it kind of makes sense in the aftermath of the
Speaker 3: pandemic and semiconductors. I'm sure you remember this. I think
Speaker 3: that was actually where we first learned about the bullwhip
Speaker 3: effect and this idea that very small changes in one
Speaker 3: end of the supply chain, which would be customer demand,
Speaker 3: can end up reverberate, you know, all the way through
Speaker 3: the supply chain. And so when you had carmakers start
Speaker 3: to cut back on their orders. That had a much
Speaker 3: bigger and longer impact than you might have anticipated. And
Speaker 3: so it's interesting to see companies coming at it from
Speaker 3: the other end and saying like, no, we have all
Speaker 3: this money and we're going to be here for a
Speaker 3: long time.
Speaker 2: We're not slowing down. We are going to agi. And
Speaker 2: so if you think like, oh, we're gonna come out
Speaker 2: with GPT five and then we're going to focus on
Speaker 2: just like commercializing that and selling it to airlines to
Speaker 2: do customer support after that, and just go into glide
Speaker 2: mode and take business like they want to signal that
Speaker 2: they're like building more and more and more. I thought
Speaker 2: that was interesting. I thought it was interesting the point
Speaker 2: about Nvidia and Kuda and the idea that, Okay, yes,
Speaker 2: the Kuda software ecosystem is perceived to be this mote
Speaker 2: that makes it harder for other semiconductor companies to break
Speaker 2: into the same business, but it's also constraining from an
Speaker 2: in video perspective, the idea that, Okay, if they want
Speaker 2: everything to be Kuda compatible or be within the same
Speaker 2: family of software usage, then that also constrains the potential
Speaker 2: sidelines that they might get into right.
Speaker 3: And opens up space for competitors. But I don't know
Speaker 3: why I haven't really like internalized this lesson before, because
Speaker 3: it comes up in every conversation we do on semiconductors.
Speaker 3: But I think there's still a perception, or at least
Speaker 3: maybe I still have this perception that the moat around
Speaker 3: Nvidia is like the actual hardware. Yes, but it's not.
Speaker 3: It's the software. It's Kuda.
Speaker 2: It seems like it's both.
Speaker 3: Well, yeah, but I think I'm starting to appreciate how
Speaker 3: much of it is Kuda is what I'm.
Speaker 2: Saying it certainly, it certainly seems to come up over
Speaker 2: and over again. How much the fact that this is
Speaker 2: what people use. It's the software that makes it easy
Speaker 2: for less sophistic less sophisticated customers to use the applications.
Speaker 2: It seems extremely powerful. It's also interesting to hear about
Speaker 2: like the ecosystem of businesses around semiconductor design. And you know,
Speaker 2: he mentioned Broadcom. Ryner mentioned Broadcom, which is a company
Speaker 2: that I don't think we've ever really talked about very
Speaker 2: much on the show. But if you look at that stock,
Speaker 2: I mean, it looks kind of like you're looking at
Speaker 2: a chart of in video like that has been a
Speaker 2: gigantic winner over the last few years. Back in twenty twenty,
Speaker 2: it was a thirty one dollars stock. Now it's one
Speaker 2: hundred and forty six dollars stock. Okay, I tell you
Speaker 2: a five back or so, maybe not quite in video returns.
Speaker 3: And this idea that, like how in Vidia has just
Speaker 3: skewed like I know what's expected of every stock, it's like,
Speaker 3: but this is on a different plane.
Speaker 2: And this idea that a semiconductor startup doesn't necessarily interface
Speaker 2: directly with TSMC like that really for the most sophisticated advance,
Speaker 2: and then there are some of these companies in the middle.
Speaker 2: I thought that was extremely interesting.
Speaker 3: Uh you know what, Joe, I asked chat GPT, what
Speaker 3: the most beautiful semiconductor is. Yeah, it says Gallium arsenide
Speaker 3: is considered beautiful for several reasons. It's crystal structure is
Speaker 3: often admired for its clarity and elegance. Wow, So I
Speaker 3: guess semiconductors may solum arsenide, So.
Speaker 2: There's beauty at the molecular level. Yeah, But actually I thought,
Speaker 2: you know, I thought when you asked that question, it's like, oh,
Speaker 2: it's just sort of a you know, philosophical, you know,
Speaker 2: fun whimsical question, but this idea of like doing the
Speaker 2: minimum required or not building a bunch of extra rooms
Speaker 2: in the house that you don't really need. And as
Speaker 2: we know, I mean, it's just objectively true that even
Speaker 2: if in video chips are the best in the world
Speaker 2: for AI, they do other stuff beyond AI, and they
Speaker 2: do ethereum mining, or they used to, and that was
Speaker 2: based on proof of work back in the old days.
Speaker 2: And of course they're for video games. But if you
Speaker 2: really just want a computer, or if you really just
Speaker 2: want a model that can speak in English or write code,
Speaker 2: or can just think without doing video games and chip mining,
Speaker 2: then perhaps there are a bunch of rooms in the
Speaker 2: house that are totally unnecessary.
Speaker 3: Yeah, And I mean there's efficiency costs to that efficiency cost. Yeah,
Speaker 3: you're trying to streamline it as much as possible. All right,
Speaker 3: shall we leave it there.
Speaker 2: Let's leave it there.
Speaker 3: This has been another episode of the All Thoughts podcast.
Speaker 3: I'm Tracy Alloway. You can follow me at Tracy Alloway.
Speaker 2: And I'm Jill Wisenthal. You can follow me at the Stalwart.
Speaker 2: Follow our guests Rein or Pope. He's at rein Or
Speaker 2: Pope and Mike Gunter. He's Mike Gunter Underscore. Follow our
Speaker 2: producers Carmen Rodriguez at Carman Ermann dash Oll, Bennett at
Speaker 2: dashbot In Kelbrooks at Kelbrooks. Thank you to our producer
Speaker 2: Moses Ondam. For more Oddlots content, go to Bloomberg dot
Speaker 2: com slash odd Lots, where we have transcripts, a blog,
Speaker 2: and a newsletter and you can chat about all of
Speaker 2: these topics twenty four to seven in the discord Discord
Speaker 2: dot gg slash odd Laws. There's even a semiconductor room
Speaker 2: in there, so you can just go there and just
Speaker 2: talk about chips all day if you want.
Speaker 3: If you enjoy All Lots, if you like it when
Speaker 3: we talk about what the most beautiful semiconductor is, then
Speaker 3: please leave us a positive review on your favorite podcast platform.
Speaker 3: And remember, if you're a Bloomberg subscriber, you can listen
Speaker 3: to all of our episodes absolutely ad free. All you
Speaker 3: need to do is connect your Bloomberg account with Apple Podcasts.
Speaker 3: In order to do that, just find the Bloomberg channel
Speaker 3: on the platform and follow the instructions there. Thanks for listening.