Speaker 1: Hello, and welcome to another episode of The Odd Blocks podcast.
Speaker 1: I'm Joe Wisenthal.
Speaker 2: And I'm Tracy Alloway.
Speaker 1: Tracy, I'm not sure if you've heard anyone talking about
Speaker 1: it or anything, but have you heard about like this
Speaker 1: sort of AI thing people have been discussing?
Speaker 2: Oh, you know what, I discovered this really cool new
Speaker 2: thing called chat gps.
Speaker 1: Oh yeah, I saw that website too. Yeah.
Speaker 2: Have you tried it?
Speaker 1: I tried it. Yeah, I kind of like write a
Speaker 1: poem for me. She's pretty cool technology. We should probably
Speaker 1: learn more about it.
Speaker 2: Yeah, I think we should know. Okay, all right, obviously
Speaker 2: we're being facetious and joking, but everyone has been talking
Speaker 2: about AI and these new sort of natural language interfaces
Speaker 2: that allow you to ask questions or generate all different
Speaker 2: types of texts and things like that. It feels like
Speaker 2: everyone is very excited about that space.
Speaker 1: Every almost every time. Mile Like I went out with
Speaker 1: some friends that I hadn't seen in a long time,
Speaker 1: Like I was at a bar last night, and like
Speaker 1: the conversation like turned to AI within like two minutes.
Speaker 1: Never got to talk about the experiments they did. But yes,
Speaker 1: there is a lot. It's basically like this, like wall
Speaker 1: of noise and everyone's been talking about actually but us,
Speaker 1: because I don't think we have done as far as
Speaker 1: I can recall, like an AI episode. We don't want
Speaker 1: to just add to the noise and get another sort
Speaker 1: of chin stroke around. But obviously there's a lot there
Speaker 1: for us.
Speaker 2: To discuss totally, and I'm sure this will be the
Speaker 2: first of many episodes. But one of the ways that
Speaker 2: it fits into sort of classic odd lots lore is
Speaker 2: via semiconductors.
Speaker 3: Right.
Speaker 2: If you think about what chat GPT, for instance, is doing,
Speaker 2: it's taking words and transforming them into numbers and then
Speaker 2: spitting those words back out at you. And the thing
Speaker 2: that enables it to do that semiconductors chips.
Speaker 1: Right, So here's like the four things I think I
Speaker 1: know about this and so this is that A. Training
Speaker 1: the AI models so that they can do that is
Speaker 1: a computationally intensive process. B. Each query is much more
Speaker 1: computationally intensive than say a Google search.
Speaker 3: Three.
Speaker 1: The company that's absolutely crushing the space and printing money
Speaker 1: because of this is in Nvidia. Yeah, And four there's
Speaker 1: a general scarcity of computing powers, so that even if
Speaker 1: you and I like were brilliant mathematicians and AI theorists,
Speaker 1: et cetera. If we wanted to start a chat GPT competitor,
Speaker 1: just getting access to the computing power in order to
Speaker 1: do that would not be trivial, even if we had
Speaker 1: tons of money outside of it.
Speaker 2: I'm going to buy an out of business cryptomne and
Speaker 2: take all the.
Speaker 1: They've already been bought. Someone got that. But that's that's
Speaker 1: basically the extent of my understanding of the nexus between
Speaker 1: this AI and chips, and I suspect there's more to know.
Speaker 2: They're just well. I also think having a conversation about
Speaker 2: semiconductors and AI is a really good way to understand
Speaker 2: the underlying technology of both those things. So that's what
Speaker 2: I'm hoping for out of this conversation.
Speaker 1: All right, Well, you mentioned we've been doing We've done
Speaker 1: lots of Chips episodes in the past, so we're going
Speaker 1: to go back to the future or something like that.
Speaker 1: We're going to go back to our first episode, our
Speaker 1: first guest, where we started exploring Chips episodes. I think
Speaker 1: it was the first one that we did sometime maybe
Speaker 1: in early twenty twenty one. We're going to be speaking
Speaker 1: with Stacey Raskin, Managing director and senior analyst of US
Speaker 1: Semiconductors and Semiconductor Capital Equipment at Bernstein Research, someone who's
Speaker 1: great at breaking all this stuff down has been doing
Speaker 1: a lot of research on this question now. So Stacy,
Speaker 1: thank you so much for coming back on odd lots.
Speaker 3: I am so happy to be back. Thank you so
Speaker 3: much for having me right.
Speaker 1: So I'm going to start with just sort of like
Speaker 1: not even a business question, but a sort of semiconductor
Speaker 1: design question, which is this company in video Like for
Speaker 1: years I just sort of knew them. Is like they
Speaker 1: were the company that made graphics cards for video games,
Speaker 1: and then for a while they got there like oh,
Speaker 1: and they're also good for crypto mining, and they were
Speaker 1: very popular for a while in ethereum mining when it
Speaker 1: used roof of work. And now my understanding is everyone
Speaker 1: wants their chips for AI purposes. And we'll get into
Speaker 1: all that, but just to start, what is it about
Speaker 1: the design of their chips that makes them naturally suited
Speaker 1: for these other things? A company that started in graphics
Speaker 1: cards that makes them naturally suited for these things like
Speaker 1: AI in a way apparently that other chip makers, like
Speaker 1: saying Intel, their chips do not seem to be as
Speaker 1: used for this space.
Speaker 3: Yeah, so let me step back.
Speaker 1: Yeah, sure, if the question, if the question is totally
Speaker 1: flawed in its premise, then feel free to say your
Speaker 1: question is totally let me step back.
Speaker 3: So sure, I'd say the idea of like using cute
Speaker 3: and artificial intelligence has obviously been around for a long
Speaker 3: long time, and actually the AI industry has been through
Speaker 3: a number of what they call AI winters over the years,
Speaker 3: where people would get really excited about this and then
Speaker 3: they would do work, and then it would just turn
Speaker 3: out it wasn't working, and pretty much it was just
Speaker 3: because the compute capacity and capabilities of the hardware at
Speaker 3: the time doesn't really wasn't really up to the task,
Speaker 3: and so interest would wane and you'd go through this
Speaker 3: winter period, and a while back, I don't know, ten
Speaker 3: fifteen years ago, whenever it was, it was sort of
Speaker 3: discovered that the types of calculations that are used for
Speaker 3: neural networks and machine learning, it turns out they are
Speaker 3: very similar to the kinds of application the kinds of
Speaker 3: mathematics that are used for graphics process processing and graphics rendering.
Speaker 3: As it turns out it's primarily matrix multiplication and we'll
Speaker 3: probably get into this call on this call a little
Speaker 3: bit in terms of how these machine learning models and
Speaker 3: everything actually work. But at the end of the day,
Speaker 3: really it comes down to like really really large amounts
Speaker 3: of matrix multiplication and parallel operations. And as it turned out,
Speaker 3: the GPU, the graphics of processing unit was was quite suitable.
Speaker 1: Before you go on then and maybe we'll get into
Speaker 1: this an hour three of this conversation. No, we're not
Speaker 1: going to go down on but what is matrix multiplication?
Speaker 3: Yeah? So, I don't know how many of you are
Speaker 3: our listeners here have had linear algebra or anything, but
Speaker 3: a matrix is just like an array of numbers, like
Speaker 3: thinking about like a square array of numbers, okay, okay,
Speaker 3: and matrix multiplications. I've got two of these arrays and
Speaker 3: I'm multiplying them together, and it's it's not as simple
Speaker 3: as the kind of math or multiplication that maybe you're
Speaker 3: typically used to, but it can be done. And it
Speaker 3: turns out there are some of these characteristics of these
Speaker 3: kinds of matrix' number of these matrix can be really big,
Speaker 3: and there's like lots and lots of operations that need
Speaker 3: to happen, and this stuff needs to happen like like
Speaker 3: quite rapidly. And again I'm grossly simplifying here for the listeners,
Speaker 3: But when when you're working through these kinds of machine
Speaker 3: learning models, that that's really what you're doing. It's it's
Speaker 3: a bunch of different makes, a bunch of different arrays
Speaker 3: of numbers that contain all of the different parameters and things.
Speaker 3: But we should probably step up a bit and talk
Speaker 3: about what we actually mean when we talk about machine
Speaker 3: learning and models and all kinds of things. But at
Speaker 3: the end of the day, you have these really large
Speaker 3: arrays of numbers that have to get multiplied together in
Speaker 3: many cases, over and over again, many many times, and
Speaker 3: it turns into a very very large compute problem. And
Speaker 3: it's something that the GPU architecture can actually can do
Speaker 3: really really efficiently, much more efficiently than you could say
Speaker 3: on a traditional CPU. And so, as it turns out,
Speaker 3: the GPU has become a good architecture for this. Now
Speaker 3: when a video has done on top of this, not
Speaker 3: only with having the hardware is they've also built a
Speaker 3: really massive software ecosystem around all of this. They have
Speaker 3: their software is called Kuta. Think about it as kind
Speaker 3: of like the software of the programming and environment, like
Speaker 3: the parallel programming environment for these gps, and they've layered
Speaker 3: on all kinds of other libraries, stks and everything on
Speaker 3: top of that that actually makes this relatively easy to
Speaker 3: use and to deploy and to deliver. And so they've
Speaker 3: built up not just the hardware bus of the software
Speaker 3: around this, and it's given them a really really sort
Speaker 3: of like like like massive gap versus like a lot
Speaker 3: of the other competitors that are now trying to get
Speaker 3: into this market as well. And so and it's FUNNYO
Speaker 3: if you look at Nvidia as a stock I mean today,
Speaker 3: I mean this morning, it's about a lot of a
Speaker 3: two hundred and sixty or two hundred and seventy dollars
Speaker 3: a share. This was a ten to twenty dollars stock forever,
Speaker 3: and they did a four to one s stock split recently,
Speaker 3: so that'd be more like, you know, like a two
Speaker 3: dollars and fifty cent to five dollars stock on today's
Speaker 3: basis for for years and years and years. And just
Speaker 3: the magnitude of the growth that we've had with these
Speaker 3: guys over over the last like five or ten years,
Speaker 3: particularly around their data center business and artificial intelligence. Everything
Speaker 3: has just been quite remarkable, and so the earnings have
Speaker 3: gone through the roof, and clearly the multiple that you're
Speaker 3: placing on those earnings has gone through the roof, because
Speaker 3: you know, the the view is that the opportunity here
Speaker 3: is massive and that we're early and there's a lot
Speaker 3: of runway ahead of us and the stocks. I mean,
Speaker 3: it's had it tops and downs, but in general it's
Speaker 3: been a home run.
Speaker 2: I definitely want to ask you about where we are
Speaker 2: in the sort of semiconductor stock price cycle. But before
Speaker 2: we get into that, you know, I will also bite
Speaker 2: on the really basic question that you already alluded to,
Speaker 2: but how does machine learning slash AI actually work. You
Speaker 2: mentioned this idea of I guess processing a bunch of
Speaker 2: data in parallel versus I guess old style computing where
Speaker 2: it would be sequential. But like, talk to us about
Speaker 2: what is actually happening here and how does it fit
Speaker 2: into the semiconductor space.
Speaker 3: You bet? You bet? So let me let me first
Speaker 3: abstract this up and I'll give you a really contrived
Speaker 3: example just sort of simplistically about what's going on, and
Speaker 3: then we can go a little bit more into the
Speaker 3: actual details of what's happening. But let's imagine you want
Speaker 3: to have some kind of a neural net. But the
Speaker 3: machine learning is typically done with something called a neural network,
Speaker 3: and I'll talk about what that is in a moment.
Speaker 3: And let's let's just imagine, for example, you want to
Speaker 3: build a an artificial intelligence a neural network to recognize
Speaker 3: pictures of casts. It's just saying, okay, let's imagine I've
Speaker 3: got this black box sitting in front of me, and
Speaker 3: it's got a slots on one side where I'm taking
Speaker 3: pictures and I'm feeding them in. It's got to display
Speaker 3: on the other side which tells me, yes, it's a
Speaker 3: cat or no it's not. And on the side of
Speaker 3: the box there are a billion knobs that you can turn, okay,
Speaker 3: and and they'll change various parameters of this model that
Speaker 3: right now are inside the black box. Don't worry about
Speaker 3: what those parameters are, but there's there's knobs that can
Speaker 3: change them, and so effectively what you're doing when you're
Speaker 3: training the thing. And by the way, when you have
Speaker 3: the artificion does what you have is you have this
Speaker 3: big black box. You need to train it to do
Speaker 3: a specific task. That's what I'm going to talk about
Speaker 3: in a moment. That's called training, and then once it's trained,
Speaker 3: you need to use it for whatever task you've traded for.
Speaker 3: That task is called inference. So you got to do
Speaker 3: the training inference. So the training here's where we have.
Speaker 3: I got my box with a slot and the display
Speaker 3: and a billion knobs. Okay, So what I do for
Speaker 3: the training process effectively is I take a picture and
Speaker 3: a known picture okay, so I know if it's a
Speaker 3: catter or not. I feed it into the box and
Speaker 3: I look at the display and it tells me yes
Speaker 3: it's a catteror yes it's not, and it probably gets
Speaker 3: it wrong. And so then what I do is I
Speaker 3: turn some of the knobs and I feed another picture in,
Speaker 3: and then I turned some of the knobs, and I'm
Speaker 3: basically tuning all of the parameters and sort of measuring
Speaker 3: how accurate is this network at doing this tasket recognizing
Speaker 3: is this a picture of a cat or is it not?
Speaker 3: And I keep feeding pictures in known pictures known data set,
Speaker 3: and I keep playing with all the knobs until the
Speaker 3: accuracy of the thing is wherever I want it to be.
Speaker 3: So yes, it's decided that that now it's very good
Speaker 3: at recognizing is this a picture of a catteror is
Speaker 3: it not. At that point, my model, my box is trained.
Speaker 3: I now lock all of those knobs in place, I
Speaker 3: don't move them anymore, and I use it now I
Speaker 3: can just feed in pictures and it'll tell me yes,
Speaker 3: it's a category, yes it's not. And so the process
Speaker 3: of training this model is what that's really what it's about.
Speaker 3: It's about varying all of the parameters. And by the way,
Speaker 3: these models can have billions or hundreds of billions or
Speaker 3: even more of parameters that they can be changed. And
Speaker 3: that's the process of training. You're basically trying to optimize
Speaker 3: this this sort of situation. I'm changing the parameters a
Speaker 3: little bit at a time such that I can optimize
Speaker 3: the response of this thing such sus that I can
Speaker 3: get the performance of it, the accuracy of the network
Speaker 3: to be high. So that's the training process, and it
Speaker 3: is very very compute intensive, because you can imagine, if
Speaker 3: I've got a billion different knobs on turning, I'm trying
Speaker 3: to optimize the output, that takes a lot of compute.
Speaker 3: The inference process once all that is much less compute
Speaker 3: intensive because I'm not changing anything. I'm just applying the
Speaker 3: network as it is to whatever data that I'm feeding
Speaker 3: in at that But I'm not changing anything. But I
Speaker 3: may be doing a lot more that the difference of
Speaker 3: the inference. I may be using it all the time,
Speaker 3: whereas once I've trained the model trained it. So it's
Speaker 3: more like a one and done versus like a continual
Speaker 3: use sort of thing.
Speaker 1: Since you talk said, we're getting into sort of the
Speaker 1: economics of training versus inference. A is there sort of
Speaker 1: any way to get a sense of Like let's say
Speaker 1: Tracy and me start odd Lodge GPT. It's a competitor
Speaker 1: to chat, a competitor to open AI, Like, what are
Speaker 1: we thinking of in terms of just that scale? How
Speaker 1: much we're spending to compute on the training part? Then
Speaker 1: how much are recurring costs in terms of inference are?
Speaker 1: And then I'm also just curious, like also, like I
Speaker 1: know you said the inference is much cheaper, but how
Speaker 1: much cheaper is it versus say, asking Google question? How
Speaker 1: much more expensive is it? How much more expensive is
Speaker 1: a Chad GPT query or an odd Lodge GPT query
Speaker 1: versus just a normal Google search?
Speaker 3: Yeah, now you get and by the wahen I say cheaper.
Speaker 3: It's like for any given given single use right again,
Speaker 3: if I've got if I'm if I've got like one
Speaker 3: hundred billion different inference activities, maybe it's not.
Speaker 1: It's still expensive.
Speaker 3: Yeah, But I first want to talk about it, just
Speaker 3: just really quickly about like so that this is my
Speaker 3: big abstract, contrived example about what's going on. If if
Speaker 3: I go just a little bit deeper about what what
Speaker 3: this thing is, like, let's talk just briefly about a
Speaker 3: neural network, and then I will get true question, but
Speaker 3: it kind of influences it. So think what is a
Speaker 3: neural If I was to draw like a representation of
Speaker 3: a neural network for you, what I would do is
Speaker 3: I have a bunch of circles. Each of the circles
Speaker 3: would be a neuron, and I wish I was there.
Speaker 3: I could draw a picture for you. But imagine like send.
Speaker 1: A picture after you're done, send a picture and we'll
Speaker 1: run it with the episode.
Speaker 3: We'll run it with the Okay, okay, I can I
Speaker 3: can do?
Speaker 1: There your a hand drawn explanation of these are varies.
Speaker 3: These are varies and fine, but anyways, but imagine like
Speaker 3: I've got like a group of circles. I've got like
Speaker 3: a column, you know, in column one with like three circles,
Speaker 3: and then column two, I've got i don't know, three
Speaker 3: or four circles, and column three, I've got some circles.
Speaker 3: These are my neurons. And imagine I've got arrows that
Speaker 3: are connecting each circle to the circles in one row,
Speaker 3: to all of the circles in the next throw. Those
Speaker 3: are my connections between my neurons. So you can see
Speaker 3: it looks like kind of a net or a network. Okay.
Speaker 3: And so within each circle, I've got some which what's
Speaker 3: called activation function. So what each circle does is it
Speaker 3: takes an input the arrow that's coming into it, and
Speaker 3: it has to decide based on those inputs, do I
Speaker 3: send an output out out the other side or not? Right,
Speaker 3: So there's some certain threshold. If the inputs reach some
Speaker 3: amount of threshold, the neuron will fire, just just like
Speaker 3: the neuron in your brain. Okay. Each each neuron can
Speaker 3: have more than one input coming in from from more
Speaker 3: than one neuron in the previous These are called layers.
Speaker 3: By the way, these rows of circles can have more
Speaker 3: than one input from the different neurons in the previous layer,
Speaker 3: and that the neuron can weight those those different inputs
Speaker 3: differently good, So it can say, you know, from from
Speaker 3: this one neuron, I'm going to give that a fifty
Speaker 3: percent weight, and from the other neural only weight at
Speaker 3: twenty percent. I'm not going to take the full signal.
Speaker 3: So those are called the weights of the network. And
Speaker 3: so each neuron has inputs coming in and outputs going out,
Speaker 3: and each of those inputs and outputs will have a
Speaker 3: weight associated with it. So those those are where I
Speaker 3: talk about those knobs. Those parameters. Yeah, those weights are
Speaker 3: are one set of parameters. And then within each neuron
Speaker 3: there's there's basically there's a certain threshold with all those
Speaker 3: all those signals coming in when you add them up,
Speaker 3: if they reach a certain threshold, then the neuron fires. Okay,
Speaker 3: So that that threshold is called the bias, and you
Speaker 3: can tune that. Like I can have a really sensitive
Speaker 3: neuron where if the bias doesn't I don't need a
Speaker 3: lot of signal coming in to make it fire. I
Speaker 3: can have a neuron that's less sensitive. I need a
Speaker 3: lot of signal coming into portal fire. That's called a bias.
Speaker 3: That that that's also a parameter. So those are the
Speaker 3: parameters that you're setting. The structure of the network itself,
Speaker 3: the number of neurons and the number of layers and
Speaker 3: everything that's that's sort of set, and then you're trying
Speaker 3: to determine these weights and biases and again just just
Speaker 3: the level set you check GPT, which you haven't getting
Speaker 3: excited about as one hundred and seventy five billion separate
Speaker 3: parameters that they get set during their during the training press. Okay,
Speaker 3: So that's that's kind of what's what's going on.
Speaker 2: Before you talk about the economics. Can I just ask
Speaker 2: so one of the things about the technology is it's
Speaker 2: sort of it's supposed to be iterative, right, like it's
Speaker 2: learning as it goes along. Can you talk just briefly
Speaker 2: maybe about how it's incorporating like new inputs as it develops.
Speaker 3: Yeah, So when when you when you training, let's talk
Speaker 3: about training now. So when you train the network, it
Speaker 3: happens on a static data set. Okay, so you have
Speaker 3: to start with a data set, right, and in terms
Speaker 3: of check GPT, that is you know, it has a
Speaker 3: large corpus of data that it was trained on. It
Speaker 3: was there's a lot of data from the Internet and
Speaker 3: from other sources.
Speaker 1: Right, basically trained the smart like all of the Internet,
Speaker 1: but also a lot of Reddit. So it's like we've right,
Speaker 1: like is it like we've trained just like the greatest
Speaker 1: brain of all time is like reddit pill.
Speaker 2: Now it talks like a seventeen year old boy.
Speaker 3: So there's a lot of data and and so yes,
Speaker 3: I sort of how does that data get get you know,
Speaker 3: incorporated into I don't want to get too short of
Speaker 3: getting too complet I don't want to get too complicated.
Speaker 3: Let me talk about how to standard training works, and
Speaker 3: then we can talk about chat GPT because that uses
Speaker 3: a different kind of model. It's called a transformer model.
Speaker 3: But anyways, but when when I'm training this, so, so
Speaker 3: what happens is is I feed this stuff that there's
Speaker 3: a there's a process called it's called back propagation. Basically
Speaker 3: what you do is you sort of feed this stuff
Speaker 3: through through this through the network itself, and then you
Speaker 3: work it backwards and you're basically what you're doing is
Speaker 3: you're measuring the output against a known response. I want
Speaker 3: to sort of you know, that's my my cat picture.
Speaker 3: Is it a cat or is it not a cat, right,
Speaker 3: I'm trying to minimize the difference between because I want
Speaker 3: to be accurate. Right, So what you sort of to
Speaker 3: do is you roll a certain step through the network, right,
Speaker 3: You measure the output against the against the known what
Speaker 3: it should be. And then there's a process that's called
Speaker 3: back propagation, where what you're doing you're actually what you're
Speaker 3: calculate what's called the gradients of all of these things.
Speaker 3: You're basically looking at sort of like the sort of
Speaker 3: like the rate of change of of these different parameters,
Speaker 3: and you sort of work the network backwards, and that
Speaker 3: gradient that you're calculating kind of tells you how much
Speaker 3: to adjust each parameter. So you work it back and
Speaker 3: then you work it forward again, and then you work
Speaker 3: it backward, and then you work at forward and you
Speaker 3: work at backward, and then you do that until you've
Speaker 3: converged like that that the that the network itself is
Speaker 3: accurate to to wherever you want it to be to
Speaker 3: be accurate at. That's so that's again I'm I'm I'm
Speaker 3: grossly simplifying here. I'm trying to keep this as high
Speaker 3: level as possible, but that's kind of what you're and
Speaker 3: just in terms of the amount of can be sort
Speaker 3: of train check GPT and and checking we can do.
Speaker 3: They've actually released all the details of the network, like
Speaker 3: how many layers and what's the dimension, I parameters all
Speaker 3: this stuff, so we can do this math. It turns
Speaker 3: out to take about three times ten to the twenty
Speaker 3: third operations to train it. And so just just that's
Speaker 3: three hundred sex tillion operations it took to train chat GPT.
Speaker 3: Now in terms of how much it costs, so CHATTYB
Speaker 3: was was they kind of said this, It was trained
Speaker 3: on ten thousand in video what they called the V
Speaker 3: one hundred. That's that's the Volta chip. That's a chip
Speaker 3: that's several years old for in video. But it was
Speaker 3: trained on supposedly about ten thousand of these. And we
Speaker 3: did some of this math ourselves. I was coming out
Speaker 3: more like three or four thousand, but there's a ton
Speaker 3: of another assumptions you have to make it here, ten
Speaker 3: thousand seems to be the right order of magnitude for
Speaker 3: that part. That part of the time cost about you know,
Speaker 3: I don't know, eight thousand bucks. And so the number
Speaker 3: that was kind of tossed up with something like eighty
Speaker 3: million dollars to train chat GPT one time.
Speaker 1: I think on some of the it doesn't seem like
Speaker 1: that much to me. Well, so this is like did
Speaker 1: I get it, but like there are a lot of
Speaker 1: companies that could spend that have eighty millions.
Speaker 3: I actually agree with it. We're jumping ahead. But my
Speaker 3: take is that for for large language models, and we
Speaker 3: can talk about these different things, but for large language
Speaker 3: almost chat CHIPD, I actually think inference is a bigger opportunity,
Speaker 3: and you're kind of getting to the heart of it.
Speaker 3: It's because inference scales directly the more queries I run.
Speaker 1: Trained once and that's done, and that's eighty one, or
Speaker 1: even if.
Speaker 3: You're training more than once and again to your question, Tracy,
Speaker 3: like you can add to the to the data set
Speaker 3: and retrain it. But if I've already got the info,
Speaker 3: let's say I'm training it every two weeks, Okay, yeah,
Speaker 3: that'd be training it like twenty four to twenty five
Speaker 3: times a year. But I've I've got the infrastructure that
Speaker 3: is in place already right to do that, and so
Speaker 3: the training TAM will be more around how many different
Speaker 3: entities actually develop these models and how many models each
Speaker 3: do they develop and how often do they train those models,
Speaker 3: and importantly how big do the models get, Because this
Speaker 3: is one of the things. Chat GPD is is big,
Speaker 3: but GPT four, which they've released, that is even bigger.
Speaker 3: They haven't they haven't talked about specs, but I wouldn't
Speaker 3: be surprised. CHATCHIPD four is room to have over billion
Speaker 3: parameters like a very well mighte and you have. We're
Speaker 3: very early into this, like these these models are going
Speaker 3: to keep getting bigger and bigger and bigger. And so
Speaker 3: that's how I think the training market, the training tam
Speaker 3: will be growing. It it's a function of the of
Speaker 3: the number of trainings of all these models we're doing
Speaker 3: every year, in the size of these models, and the
Speaker 3: model will get big.
Speaker 1: So let's get it. But in your view, the big
Speaker 1: money is going to be made on the inference, So
Speaker 1: let's talk about it.
Speaker 3: I think.
Speaker 1: So think that's talk about what happens then and your
Speaker 1: sort of sense of the side. I don't know, Yeah,
Speaker 1: just talk to us about the inference part and the economics.
Speaker 3: You bet, Chat CHPT in these large language models, it's
Speaker 3: a it's a new type of model's called a transformer model,
Speaker 3: and there's a bunch of compute steps that have to happen.
Speaker 3: There's also a step in there that helps it map
Speaker 3: the relation, capture the relationship between you. You know, by
Speaker 3: the way, if you if you've ever used chatcha, you know,
Speaker 3: you type in like a querry into a box and
Speaker 3: it and it returns to respond, so that querry is
Speaker 3: broken into what are called tokens. It's basically thinking do
Speaker 3: you think about token is kind of like a word
Speaker 3: or a group of words sort of. But the transformer
Speaker 3: model has something it's it's called a self attention mechanism,
Speaker 3: and what that does is it captures the relationship between
Speaker 3: those different tokens and the input sequence based on the
Speaker 3: training data that it has. And that's how it knows
Speaker 3: what it's really doing. It's predictive text. It knows based
Speaker 3: on this query, I'm going to start the response with
Speaker 3: this word, and based on this word and this query
Speaker 3: and my data said, I know, these other words typically follow,
Speaker 3: and it kind of constructs the response from that. And
Speaker 3: so our math suggests that for like a typical query
Speaker 3: response called like you know, five hundred tokens or maybe
Speaker 3: two thousand words, it was something like four hundred quadrillion
Speaker 3: operations needed to accomplish something like that. And so you
Speaker 3: can size this up because I know, for like an
Speaker 3: Nvidia GPU, and you can do it for different GPUs.
Speaker 3: I know how many operations per second each GPU can run,
Speaker 3: and I know how much these GPS ballpark kind of costs.
Speaker 3: And so then you know, you got I assume like, well, okay,
Speaker 3: how many queries per day are you going to do?
Speaker 3: And you can come up with a number, and I mean, frankly,
Speaker 3: the number can be as big as you want. It
Speaker 3: depends on how many queries. But I think a tam
Speaker 3: you know, at least in the multiple tens of billions
Speaker 3: of dollars is not unreasonable, if not more, and just
Speaker 3: the level set I mean, I guess to your Google question,
Speaker 3: Google does about ten billion searches a day and give
Speaker 3: or take. I think a lot of people have been
Speaker 3: looking at at that level as part of like you know,
Speaker 3: like the end all bill for where this could go.
Speaker 3: I'll be honest, like, I understand why people are, especially
Speaker 3: the Internet investors, are concerned that large language models and
Speaker 3: things like chat GPD can start to disrupt search. I'm
Speaker 3: not exactly sure that search is the right proxy person.
Speaker 3: It feels kind of limiting to me. I mean, you
Speaker 3: could imagine I've watched a little too much Star Trek,
Speaker 3: I guess, but I mean you could imagine, you know,
Speaker 3: when you have like a virtual assist in the ceiling,
Speaker 3: I'm calling out to it, and you know, it doesn't
Speaker 3: have to be just search on my screen. I could
Speaker 3: have it in my car, right, I could have you know,
Speaker 3: I call up American Airlines that change my airline tickets
Speaker 3: and it's a checkbo that the CHET bought that's talking
Speaker 3: to me. So this could be very big and by
Speaker 3: the way, I think to get by the way, the
Speaker 3: one problem with this start to a calculation that's kind
Speaker 3: of static, Like the cost is sort of an output
Speaker 3: rather than an input. I think to drive adoption, cost
Speaker 3: will come down, and we've already seen that. Like Video
Speaker 3: has a new product it's called Hopper, which is like
Speaker 3: two generations past those V one hundreds that I was
Speaker 3: talking about, past the Volta generation. The cost per query
Speaker 3: to do this or the cost for training on Hopper
Speaker 3: is much lower than a Bolta because it's much more efficient.
Speaker 3: Part that's a good thing, though it's camacreed if it
Speaker 3: will drive adoption, and.
Speaker 4: Video actually has specific products specifically designed to do this
Speaker 4: this kind of thing, and Hopper has specific blocks on
Speaker 4: it that actually helped with with the training and inference
Speaker 4: on these kind of large language models.
Speaker 3: And so I actually think over time, is the efficiency
Speaker 3: gets better and better, you're going to drive adoption more
Speaker 3: and more. I think this is a big thing. And
Speaker 3: I remember we're still really early. Chatchp deal only showed
Speaker 3: up in November.
Speaker 1: Yeah, it's crazy, it's really early.
Speaker 3: Still.
Speaker 2: Well, just on that note, can you draw directly the
Speaker 2: connection between the software and the hardware you're here, because
Speaker 2: I think it at this point probably everyone listening has
Speaker 2: tried chat GPT, and you're used to seeing it as
Speaker 2: a sort of you know, it's an interface on the
Speaker 2: Internet and you type stuff into it and it spits
Speaker 2: something out. But like, where do the semiconductors actually come
Speaker 2: in when we're talking about crunching these enormous data sets
Speaker 2: and what makes us You kind of touched on this
Speaker 2: a little bit with Nvidio, but what makes a semiconductor
Speaker 2: better at doing AI versus more traditional computational processes?
Speaker 3: Yeah, yeah, you bet. So. To answer that second question,
Speaker 3: I think AI is really much more around parallel processing,
Speaker 3: and in particular thing it's this kind of MAPP matrix map.
Speaker 3: It's a single class of calculations that these things do
Speaker 3: very very efficiently and do very very well, and they
Speaker 3: do them much more efficiently than a CPO that that
Speaker 3: performs a little more really versus parallel. You just couldn't
Speaker 3: run this stuff on CPUs. But don't get me wrong,
Speaker 3: you do some of we've been talking about inference on
Speaker 3: large language models. There's there's all kinds of inference. Inference
Speaker 3: workloads range from very simplistic to very very complex like
Speaker 3: and my my, you know, cat recognition example was very simplistic,
Speaker 3: something like this, or fakly something like autonomous driving that
Speaker 3: is an inference activity, but is a hugely computationally intense
Speaker 3: inference activity. And so there's still a lot of inference
Speaker 3: today that actually happens. In fact, most inference today actually
Speaker 3: happens on CPUs. But i'd say the types of things
Speaker 3: that you're trying to do are getting more and more complex,
Speaker 3: and CPUs are getting less and less viable for that
Speaker 3: for that kind of that kind of anth and so
Speaker 3: that's kind of the difference between GPUs and other types
Speaker 3: of parallel offerings versus like a CPU. I should say,
Speaker 3: by the way, GPUs are not the only way to
Speaker 3: do this. Google, for example, has their own an I chips.
Speaker 3: They call them a TPU tensor processing unit.
Speaker 1: One thing I write like about talking to Stacey to
Speaker 1: things is a I think he comes up with better
Speaker 1: versions of our questions than we do.
Speaker 2: Which it's like one thing about the question is just ask.
Speaker 1: He's always like, all right, that's a good question, but
Speaker 1: let me actually reframe the question to get a better response.
Speaker 1: So I appreciate that, and he also anticipates because I literally,
Speaker 1: like on my computer right now, I had Google Cloud
Speaker 1: tensor processing units because that was my next question. And
Speaker 1: also important because I think yesterday the information reported that
Speaker 1: Microsoft is also So why don't you talk to us
Speaker 1: about that these other and what the competing directly?
Speaker 3: Yeah, yeah, yeah, you got so Google's good. By the good,
Speaker 3: this is not new. Google has been doing their own
Speaker 3: chips for seven or eight years. It is not new, right,
Speaker 3: And but they have what they call TPU and they
Speaker 3: use it extensively for their own internal workloads. Absolutely, Amazon
Speaker 3: has their own chips. They have a training chip. It's
Speaker 3: that's called you know kind of hysterically. It's called tranium.
Speaker 3: They have an inference chip. It's called Interferentia. Microsoft apparently
Speaker 3: is working on their own. My feeling is every hyperscaler
Speaker 3: is working on their own chat, particularly for their own
Speaker 3: internal workloads. And that is an area we talked about
Speaker 3: in Vida software remote like Google doesn't need in video
Speaker 3: software mode, they're not running Kuda. They're they're just running
Speaker 3: tensorflock right and and doing their their thing. They don't
Speaker 3: need Kuda anything. However, that is facing an end customer,
Speaker 3: like an enterprise like end customer, like on a public cloud,
Speaker 3: like like a customer going to AWS and ranting, you know,
Speaker 3: compute power, that tends to be GPUs because customers don't
Speaker 3: have Google's just sophistication. They really do need the software
Speaker 3: ecosystem that's built around they use. So for example, I
Speaker 3: can go to Google Cloud, I can actually rent a
Speaker 3: TPU instance. It can be done. Nobody really doesn't. And
Speaker 3: actually if you look how they're priced, typically it's actually
Speaker 3: more expensive usually even than than have the way that
Speaker 3: Google's pricing GPUs on on on Google Cloud. It's it's
Speaker 3: similar for Amazon and others, And so I do think
Speaker 3: that all the hyper feelers are working on their own
Speaker 3: and there is a certain certainly a place for that,
Speaker 3: especially for their own internal workloads, anything that's facing a
Speaker 3: customer that that in Video GPO ecosystem is really kind.
Speaker 1: Of yeah, this is so, this is so Actually these
Speaker 1: just to clarify, because that point is really interesting that
Speaker 1: for like, if again Tracy and I want to launch
Speaker 1: odd launch GPT, part of the issue would be not
Speaker 1: necessarily the hardware, this sort of the silicon, but actually
Speaker 1: that in Video's software suite built around it would make
Speaker 1: it much easier for us to sort of start and
Speaker 1: use on in Video for training our model.
Speaker 3: Yeah, yes, it was, and they've built a lot of
Speaker 3: It's funny. You can go listen to Video's announcements in
Speaker 3: their analyst dys and things, and there as much about
Speaker 3: software as they are about hardware. So not only have
Speaker 3: they continue to extend like the basic like like COUDA ecosystem,
Speaker 3: they've layered all kinds of other application specific things on
Speaker 3: top of it. So they've got what they call RAPIDS,
Speaker 3: which is for enterprise Machine Learn. They've got a library
Speaker 3: package called ISAACS, which is for automation robotics, They've got
Speaker 3: a package called Clara, which is specifically for medical imaging
Speaker 3: and diagnostics. They've got something called cou Quantum, which is
Speaker 3: actually for quantum computer simulations. They've got something for drug discovery.
Speaker 3: So they're layering all these things on top, right depending
Speaker 3: on your application. They've got internal teams that are working
Speaker 3: on it's not just throwing the software out there. They've
Speaker 3: got people there that can actually like help you work
Speaker 3: or work and come along with it. They're doing other
Speaker 3: things easier, you know. So they actually just launched a
Speaker 3: cloud service, and this is with Google and Oracle and
Speaker 3: Google and Microsoft ware. You can almost they'll do like
Speaker 3: a fully provisioned in Vidia AI supercomputer in the cloud.
Speaker 3: So because like you, they sell these AI servers and
Speaker 3: they can cost hundreds of thousands of dollars apiece. If
Speaker 3: you want now you can just go to Oracle Cloud
Speaker 3: or Google Cloud or whatever. You can sort of rent
Speaker 3: they fully provisioned in Vidia supercomputer sitting in the cloud
Speaker 3: that they'll all you got to u is access it
Speaker 3: right for a web browser. This kind of get super easy.
Speaker 2: This is going to be my next question actually because
Speaker 2: so I take the point about software, but like what
Speaker 2: do the AI supercomputers actually look like nowadays, Like is
Speaker 2: there a physical thing in a giant data center somewhere? Yeah,
Speaker 2: they mostly like cloud based or what does this look like?
Speaker 3: Like? Walk astro so video sells, and Video sells something
Speaker 3: they called a DGX. It's a it's a box. I
Speaker 3: mean it's I don't know it's when it's a two peat,
Speaker 3: but I don't know what the dimensions are two peak
Speaker 3: by two pet or something like that. It's got eight
Speaker 3: GPUs and two CPUs and a bunch of memory and
Speaker 3: a bunch of networking. They've got their own like you know,
Speaker 3: they bought a company called Melanox a while back that
Speaker 3: did networking hardware. So it's got a bunch of proprietary
Speaker 3: network because that's but that's something else we haven't talked about.
Speaker 3: It's not just enough to have the computer the compute.
Speaker 3: These models are so big they don't fit on a
Speaker 3: single c GPU. So you have to be able to
Speaker 3: network all this stuff together, right, And so they've got
Speaker 3: networking in there, and they have this this box, and
Speaker 3: then you can you can stack a whole bunch of
Speaker 3: boxes together, like and Video has their own internal supercomputer.
Speaker 3: It's it's fairly a high on the top five hundred less.
Speaker 3: They call it Selene. It's a bunch of these DGX
Speaker 3: like servers that they make, all just like stacked together effectively,
Speaker 3: and they sell for the older generation. Their prior generation
Speaker 3: was called Ampeer and that box sold for one hundred
Speaker 3: and ninety nine thousand dollars. I don't believe they've released
Speaker 3: pricing on the Hopper version, but I know for the
Speaker 3: Hopper GPU it costs two to three x what Amper
Speaker 3: costs the prior generation.
Speaker 1: So this really is a separate question to me, which is, Okay,
Speaker 1: there's the price, and it exists, and you could go
Speaker 1: to you could theoretically go and use Google's tensor based
Speaker 1: cloud or is it available or is it because I
Speaker 1: sort of get the impression that, like for some of
Speaker 1: the technology that people want to use, it's not available
Speaker 1: at any price, and that there is actually is that
Speaker 1: real or not?
Speaker 3: It seems to be so we're the like. So their
Speaker 3: new generation, which is called Hopper, which like I said,
Speaker 3: has characteristics of it maked very attractive, especially for these
Speaker 3: kind of like chat GPT large language models, is in
Speaker 3: tighted to play. Were at the very beginning of that
Speaker 3: product cycle. They just launched it like in the last
Speaker 3: couple of quarters, and so that ramp up takes time,
Speaker 3: and it does seem like they are seeing accelerated demand
Speaker 3: because of this kinds of stuff, and so yeah, I
Speaker 3: think supply is tight. We've heard stories about GPU shortages
Speaker 3: at Microsoft and the cloud vendors, and I think there
Speaker 3: was a Bloomberg store the other day that said these
Speaker 3: things were selling for like forty thousand dollars on eBay.
Speaker 3: Its a thing, right, I took a look at some
Speaker 3: of those listings. They looked a little shady to me,
Speaker 3: But yeah, it's tight. You have to remember, these parts
Speaker 3: are very complicated, so the lead times to actually have
Speaker 3: more made it takes a while.
Speaker 2: Wait, so just on this snow. I joked about this
Speaker 2: in the intro, But you know, could I buy like
Speaker 2: a bitcoin mining facility and take all that computer processing
Speaker 2: power and like convert it into something that could be
Speaker 2: used for AI. Is that a possibility?
Speaker 3: You could? The big point stuff at least a lot
Speaker 3: of the big point stuff was done that was with gps.
Speaker 3: Those were still mostly gaming GPUs. People are buying gaming
Speaker 3: gps and purposing them for a bitcoin and the theory
Speaker 3: mostly etherory of mining. Yeah, they're they're not nearly as
Speaker 3: compute efficient as the data center parts, right, but I
Speaker 3: mean in theory, yeah, you could get you know, gaming
Speaker 3: GPUs if you could and stringly get but it would
Speaker 3: be prohibitive, right, And even now most of that stuff's
Speaker 3: cleared out. I think as as Joe said, but the
Speaker 3: math is somewhat similar, I'd say for these kinds of models, though, again,
Speaker 3: like a hopper in Video's new data center product has,
Speaker 3: they have something that they call it a transformer engine.
Speaker 3: What it really does is it allows you to do
Speaker 3: the training at a slightly lower precision than unless you
Speaker 3: do it at eight bit floating point versus sixteen bit
Speaker 3: it'll so it lets you get higher performance. And then
Speaker 3: there's another process. There's like a conversion process. Sometimes it
Speaker 3: has to go when you go from training to inference.
Speaker 3: It's something of quantization, and with these transformer engines you
Speaker 3: don't have to do that. So it increases the efficiency
Speaker 3: which you wouldn't get by picking some random GPUs.
Speaker 1: Where is Intel in this story?
Speaker 3: Well, so let's let's talk about the other competitive options
Speaker 3: that we're out there. Okay, So we talked about some
Speaker 3: of the captive silicon and hyperscalers that is there, and
Speaker 3: it is real, and they're all building their own and
Speaker 3: they've been doing it forever and it hasn't slowed anything
Speaker 3: down on the slightest because we're still early, and then
Speaker 3: the opportunity is big. By the way, I will say,
Speaker 3: I don't worry to lead with it. I don't worry
Speaker 3: so much about competition at this point because think about it.
Speaker 3: In Videa's run rating their data center business right now,
Speaker 3: it's something like fifteen billion dollars a year. That's where
Speaker 3: it is. It's growing, but that's where it is. So
Speaker 3: Jensen in Video CEO likes to throw out big numbers,
Speaker 3: and he threw out I think he said for silicon
Speaker 3: and hardware TAM in the data center, and he thought
Speaker 3: that their TAM overtime is three hundred billion dollars, and
Speaker 3: it seemed kind of crazy. Although I would say, like
Speaker 3: it's seeming a little less and less crazy every day.
Speaker 3: But if you thought the TAM was three hundred billion
Speaker 3: or two or one hundred billion or like whatever, and
Speaker 3: they're run rating at fifteen billion dollars, there's tons of
Speaker 3: headrooms competition doesn't really matter, and that's what we've seen.
Speaker 3: We've seen competition, but there's so much opportunity like who
Speaker 3: cares right versus like if you thought it was a
Speaker 3: twenty billion dollar ten like they would have a problem
Speaker 3: like already today. So that's why I don't worry too
Speaker 3: much because I think the opportunity is still very very
Speaker 3: large relative to where they're running into business today. In
Speaker 3: terms of other competitors, though, sayes so you mentioned let's
Speaker 3: talk about AMD first, because A and D actually makes GPUs,
Speaker 3: they make data center GPUs. They don't sell very many
Speaker 3: of them. Their current product is something called the Mi
Speaker 3: I two fifty and they've sold deminimus basically. And in fact,
Speaker 3: you know, when the China sanctions were put on, and
Speaker 3: you know, we didn't talk about that, but the US
Speaker 3: has stopped allowing like high end aichips from being shipped
Speaker 3: to China. The MI two to fifty eighties part was
Speaker 3: on the list, but it didn't affect them at all
Speaker 3: because they weren't selling anything. Hey, so their sales were zero.
Speaker 3: They've got another product coming out at the following that's
Speaker 3: called the Mi I three hundred, and people have been
Speaker 3: getting kind of excited about A and B. They've been
Speaker 3: sort of looking to play it as kind of like
Speaker 3: the Foreman's and Video. I'll be honest, I don't think
Speaker 3: it's the Foreman's in video and video is doing, you know,
Speaker 3: close to four billion dollars a quarter in data revenues.
Speaker 3: I don't know that I see anything like that with
Speaker 3: the mi I three hundred figure they in AMD as
Speaker 3: far as i fell, has not even released any sort
Speaker 3: of specifications for what it looks like at this point. So,
Speaker 3: but that is an option, and some people would say
Speaker 3: there's maybe some truth to this is you know, if
Speaker 3: you want an alternative, AV will present an alternative. And
Speaker 3: if the opportunity is really that they they'll get some.
Speaker 3: They'll they'll probably get some. Do you have that? You
Speaker 3: have Intel? So Intel's got a few things on their CPUs.
Speaker 3: Their current version is called Sapphire Rapids. It has AI
Speaker 3: specific accelerate, is four core inference not not so much
Speaker 3: maybe for this kind of stuff, but for general inference activities.
Speaker 3: They're trying to play at the capabilities of their CPU
Speaker 3: on that fine, and why are they doing that. It's
Speaker 3: because their accelerator roadmap isn't so good. So they have
Speaker 3: a GPU roadmap. The code name for it was ponta Vecchio,
Speaker 3: and they've kind of gutted that roadmap. So the follow
Speaker 3: on product was something called rialto Bridge that they've since canceled,
Speaker 3: and one of the Pontaventio products recently they just canceled,
Speaker 3: and a Pajaveci originally was designed for the Area supercomputer
Speaker 3: and it was massively late. I mean so like they
Speaker 3: took a much was it was something like a three
Speaker 3: hundred million dollar charge. I think it was the at
Speaker 3: the end of twenty twenty one. It was either the
Speaker 3: end of twenty or g twenty twenty one where they're
Speaker 3: they basically they gave it away. It was so late,
Speaker 3: So that's that's how late they were. They also have
Speaker 3: another product. They bought an Israeli AI company called Habana,
Speaker 3: and Habana has a product called Goudi. It's not a
Speaker 3: GPU exactly, but it's like a specific accelerator technology. And
Speaker 3: Amazon bought some of them and they sell a little bit,
Speaker 3: but again it versus Intel's total revenues. It's the Minimus,
Speaker 3: so they're not really there. There's also a bunch of
Speaker 3: startups and the problem with most of the startups is
Speaker 3: their their their story tends to be something like, you know,
Speaker 3: we have a product that's ten times as good as Nvidia,
Speaker 3: and the issue is with every generation, in Vidia has
Speaker 3: something that's ten times as good as in video, and
Speaker 3: they have the software ecosystem that goes with it. Neither
Speaker 3: a m D, nor Intel, nor most of the startups
Speaker 3: have anything remotely resembling in video software. So that's another
Speaker 3: huge issue right that all of them are facing. There's
Speaker 3: a few startups that have some niche success. One of
Speaker 3: the one that's probably gotten the most attention is called
Speaker 3: Servius or Cerebraus, and their whole thing. They make a chip.
Speaker 3: It's imaginating a three hundred millimeters silicon wafer and it's
Speaker 3: inscribing a square on it. That's their chip. It's like
Speaker 3: one chip per wafer, and so you can put very
Speaker 3: large models onto these chips, and they've been deploying them
Speaker 3: for those kinds of things. But again the software becomes
Speaker 3: an issue. But they've had a little bit of success.
Speaker 3: There's some other names that that you know, You've got
Speaker 3: Groc and some others I think that are still out there.
Speaker 3: And then there's a company called Tends toward which is
Speaker 3: interesting not because of so far what they're doing because
Speaker 3: it's early, but it's run now by Jim Keller. And
Speaker 3: do you guys know who Jim Keller is. Jim Keller
Speaker 3: was was He's sort of like a star chip designer.
Speaker 3: He designed Apple's first custom processor. He designed A and
Speaker 3: ds as and and epic road NEPs that they've been
Speaker 3: that they've been taking a lot of share with. He
Speaker 3: was even at Tesla for a while and at Intel,
Speaker 3: and so he's now running tense to it and they
Speaker 3: do it's a risk five. Risk five is another type
Speaker 3: of architecture, and they do they do an AI chap,
Speaker 3: So Jim is running that.
Speaker 2: So can I just ask based on that? I mean,
Speaker 2: how like capex intensive is developing chips that are well
Speaker 2: suited for AI versus other types of chips. And then secondly,
Speaker 2: like where do the improvements come from or what are
Speaker 2: the like improvements focused on? Is it speed or like
Speaker 2: scale given the data sets involved in the parallel processes
Speaker 2: that you described.
Speaker 3: Yeah, so it's a few thing so in terms of
Speaker 3: Capex intents, and these are mostly design companies, so they
Speaker 3: don't have a lot of Capex. It's certainly r and
Speaker 3: D intensive, So maybe maybe that's that's what you're getting
Speaker 3: on in video spends like many billions of dollars a
Speaker 3: year on R and D and and VIDA has a
Speaker 3: little bit of advantage too because it's it's effectively the
Speaker 3: same architecture between day center in gaming, so they've got
Speaker 3: other other volume effectively to sort of amortize some of
Speaker 3: those investments over although now I mean this year, I mean,
Speaker 3: data center's probably sixty percent of in videous revenues now,
Speaker 3: so I mean in video is sort of the center
Speaker 3: of data center is a center of gravity for in
Speaker 3: video now, but it's very R and D intensive and
Speaker 3: probably getting more so. And you've got folks all up
Speaker 3: and down the value chain that are investing or both
Speaker 3: the silicon guys you know, and the cloud guys and
Speaker 3: the customers and everything else. But I mean, that's that's
Speaker 3: kind of where we are in terms of what you're
Speaker 3: you're looking for. So there's a few things you're looking for.
Speaker 3: Performance and on training, quite often that comes down to
Speaker 3: like time to train. So I've got a model, Like
Speaker 3: some of these models, I mean, you could imagine it
Speaker 3: could take weeks or months historically to train right, and
Speaker 3: that's a problem. You want it to be faster, so
Speaker 3: I can get that down you know, two weeks or
Speaker 3: you know too days or hours that would be better.
Speaker 3: So that's one thing clearly that they work on.
Speaker 1: I don't want to.
Speaker 3: It's something notice, yeah, go ahead.
Speaker 1: No finish your thought that I have a slightly oh yeah.
Speaker 3: The other think I was talking about that there's something
Speaker 3: where I'm like like scale out. So basically, remember I said,
Speaker 3: you're you're connecting lots and lots of these chips together.
Speaker 3: So for example, if if I if I increase the
Speaker 3: number of chips by ten X, does my trading time
Speaker 3: go back down by like a factor of ten or
Speaker 3: is it like by factor of two? So yeah, ideally
Speaker 3: you would want like linear scaling, right, I want, like
Speaker 3: I add resources, it scaled linearly.
Speaker 1: So this is kind of gonna was going to get
Speaker 1: into my next question. Actually, and you know, we can
Speaker 1: talk to another with someone else about certain like AI
Speaker 1: fantasy doom.
Speaker 3: But I think, but I'm not an AI. I'm not
Speaker 3: an AI architecture X. But I'm a down past here.
Speaker 3: So I could just say you may want to get aged,
Speaker 3: no I.
Speaker 1: Know somebody, but I am curious though, because I do
Speaker 1: think it relates to this question, which is that okay,
Speaker 1: like with each one like GPT five and they're going
Speaker 1: to keep adding more knobs on the box, et cetera,
Speaker 1: like and is your perception that this sort of quality
Speaker 1: of the output is growing exponentially or is it the
Speaker 1: kind of thing where it's like GPT four, you know,
Speaker 1: there's a lot more knobs and they got a big
Speaker 1: jump from GPT three. GPT five will be way more knobs,
Speaker 1: but like is it going to be marginally better? Like
Speaker 1: what is this sort of like where are we in
Speaker 1: the sort of like what does the shape of the
Speaker 1: output curve look like? And this sort of like cost
Speaker 1: of you know, these chip developments of getting there. I
Speaker 1: don't know, it's kind of so there's a couple of things.
Speaker 3: So, so, first of all, when you're talking about large
Speaker 3: language where it was accuracy, it's sort of a nebulous
Speaker 3: term because it's not just accuracy. It's like like case,
Speaker 3: it's also capability, like what could it do? What chat
Speaker 3: GPT and GPD four can do. And also, like I
Speaker 3: think as you're going forward and you talk about the
Speaker 3: trajectors here, it's not just text right, we're talking text
Speaker 3: to texture, but there's also text images and anybody like
Speaker 3: with like Dolly where words. You know, it's generating images
Speaker 3: from a text prompt and now we've got like video
Speaker 3: what it was it mid was it midsummer? Is that
Speaker 3: what it's called big journey? Journey can't mid journey? Yeah,
Speaker 3: so it's it's it's creating like video prompts. I mean,
Speaker 3: so like the like text is de scrapped as just
Speaker 3: the tip of the iceberg, I think in terms of
Speaker 3: what we're going to need, but they're.
Speaker 1: Never they're never going to get to where they could
Speaker 1: have three people having a conversation with voices sound like Tracy,
Speaker 1: Joe and Stacy. Right, No, I'm just kidding, No, I mean,
Speaker 1: I'm just kidding. It feels like, yeah, this job.
Speaker 3: Now one of the dangerous clearly, and maybe this gets
Speaker 3: the capabilities. So what one thing with chat GPT is
Speaker 3: it's very very good. This why I should worry about
Speaker 3: my job because it's very good about that. That's it
Speaker 3: sounding like it knows what it's talking about, where maybe
Speaker 3: it doesn't hate, So maybe I should be worried about
Speaker 3: my job, you know, And accuracy, I think is a
Speaker 3: big issue, but you have to remember it.
Speaker 1: So, but like on this accuracy question, like I assume,
Speaker 1: you know, like self driving cars, like when people were
Speaker 1: really hyped about them ten years ago, they're like, oh,
Speaker 1: it's ninety five percent solid, we just have a little
Speaker 1: bit more, and then it's solid ten years later. Yeah,
Speaker 1: ten years later, it feels like they haven't made any
Speaker 1: progress on that final five percent.
Speaker 3: Yeah. I mean, these things are always a power law.
Speaker 1: So this is my question when we talk about accuracy
Speaker 1: or these things, like are we at the point where
Speaker 1: like is it going to be the kind of thing
Speaker 1: where it's like, yeah, GPT five will definitely be better
Speaker 1: than GBT four, but it will be like ninety six
Speaker 1: percent of the way there.
Speaker 3: Well, again, let me separate out. Let me separate an
Speaker 3: accuracy from capability again. So there's an accuracy you have
Speaker 3: to remember, like it the model has no idea what
Speaker 3: accurate even means. It doesn't remember. These things are not
Speaker 3: actually intelligent. I know there's a lot of worry about
Speaker 3: like what they go like like like agi like artifice
Speaker 3: with general intelligence. Right, I don't think this is it.
Speaker 3: This is predictive text. That's all. The model doesn't know
Speaker 3: if it's if it's viewing bull crap or truth. It
Speaker 3: has no idea, it's just predicting the next word in
Speaker 3: the the thing. And it's because of what it's trained on.
Speaker 3: So you need to add on maybe other kinds of
Speaker 3: things to ensure accuracy, maybe to put guard rails or
Speaker 3: things things like that. You may need to very carefully,
Speaker 3: like more harsh like your input like data sets and
Speaker 3: things like that. I think that's a problem now. I
Speaker 3: think it'll get solved. There's enough date. But like and
Speaker 3: this has already been an issue and you got you
Speaker 3: can take it like the other like the I don't
Speaker 3: know if it's the converse of it or not, but
Speaker 3: things like deep fakes, people are deliberately trying to use
Speaker 3: AI to deceive. I mean, this is just human nature.
Speaker 3: This is this is why we have problems. But I
Speaker 3: think they can work through that just in terms of
Speaker 3: capabilities now, I think it's it's really interesting to look
Speaker 3: at like like sort of similar like a response like
Speaker 3: to a similar prompt between like chat GPT and GPT four,
Speaker 3: and like what people are getting out of GPD four.
Speaker 3: It's it's it's miles ahead of like some of the
Speaker 3: stuff that that that chat GPT, which was trained on
Speaker 3: GPT three of them, all that than what it was,
Speaker 3: what is delivering in terms of nuance, right, and color
Speaker 3: and every and everything else. I mean, and I think
Speaker 3: that's going to continue. I wouldn't be And already you're
Speaker 3: on the boat where these things can already pass the
Speaker 3: turning tests. Oh yeah, right, it can be very difficult
Speaker 3: to know if it's if I'm put in the question
Speaker 3: of accuracy aside perment, it's very difficult to know for
Speaker 3: some of these things if if you didn't know any
Speaker 3: better whether it was coming from a real person or not.
Speaker 3: And I think it's going to get like harder and
Speaker 3: harder to tell, like whether you know even if it's
Speaker 3: not you know, quote unquote really thinking it's going to
Speaker 3: be hard for us to tell what's really going on.
Speaker 3: That is sort of like other interesting you know, implications
Speaker 3: or for what this might be over the next five
Speaker 3: years or ten years.
Speaker 2: Just going back to the stock prices, I mean, we
Speaker 2: mentioned the Nvidia chart, which is up quite a lot,
Speaker 2: although not it hasn't reached its its peak back in
Speaker 2: twenty twenty one. The Socks Index is recovering, but you know,
Speaker 2: still below an intel. I mean, I won't even mention,
Speaker 2: but like, where are we in the semiconductor cycle, because
Speaker 2: it feels like, on the one hand there's talk about
Speaker 2: excess capacity and orders starting to fall, but on the
Speaker 2: other hand, there is this real excitement about the future
Speaker 2: in the form of AI.
Speaker 3: Yes. Yes, So seventies in general were pretty lousy last year.
Speaker 3: They've had a very strong year to date performance and
Speaker 3: the sectors up, which is sectors up, you know, twenty
Speaker 3: twenty two percent year to date, quite a bit above
Speaker 3: the overall market. And the reason is, to your point,
Speaker 3: we've been in a cycle. Numbers have been coming down.
Speaker 3: And we may have talked about this last time. I
Speaker 3: don't remember, but semi conutter investors, that turns out the
Speaker 3: best friend to buy stocks in general is after numbers
Speaker 3: come down, but before they hit bottoms, Like if you
Speaker 3: could buy them right before the last cut, if you
Speaker 3: could have perfect foresight. You never know when that is.
Speaker 3: But I mean numbers of cut. But numbers have come
Speaker 3: down the laws so estimates forward estimates for the industry
Speaker 3: peaked last June and they are down over thirty percent,
Speaker 3: like thirty five percent since that when it's actually the
Speaker 3: largest negative earnings revision we've had probably since the financial crisis. Wow,
Speaker 3: and people are looking for you know, playing the ottoming
Speaker 3: theme and that hopefully things get better into the second half.
Speaker 3: You know, we get hope, hopefully China reopening, and you've
Speaker 3: got markets like and this relates to Intel like like
Speaker 3: PCs and things where you know, we've now corrected kind
Speaker 3: of we're back like more on a pre COVID run
Speaker 3: rate for PCs versus where we were, and the CPUs
Speaker 3: which were massively overshipping at the peak, they're now undershipping.
Speaker 3: And so we're in that inventory flushed part of the cycle,
Speaker 3: and so people have been sort of playing the space
Speaker 3: for that like second half recovery. Not now. All that
Speaker 3: being said, if you look at the overall industry, if
Speaker 3: you look at numbers in the second half, they're actually
Speaker 3: above seasonal. So people are starting to bake in that
Speaker 3: cyclical recovery to the numbers. And if you look at inventories,
Speaker 3: it just overall in the space they are ludicrously high.
Speaker 3: I've actually never seen them this five before. So we've
Speaker 3: had some inventory correction, but we may we may have not,
Speaker 3: we may just be getting started there. And if you
Speaker 3: look at valuations. I think the sector's trading. It's something
Speaker 3: like a thirty percent of premium to the S and
Speaker 3: P five hundred, which is the largest premium we've had again,
Speaker 3: probably since things normal life after the tech bubble or
Speaker 3: after the financial crisis at least, so people have been
Speaker 3: playing this backup recoverary. But yeah, we better get it
Speaker 3: as as as it relates to some of the other
Speaker 3: some of the individual stocks, like you mentioned Intel, It's funny.
Speaker 3: I think you guys may not know this. I just
Speaker 3: upgraded Intel. Oh. The title of the note was we
Speaker 3: hate this call, and I meant I desperately would like
Speaker 3: the standard prom It was and it was not a
Speaker 3: we like an Intel call. It was just I think
Speaker 3: that they that they're now under shipping in PCs by
Speaker 3: a wide margin, and I think for the first time
Speaker 3: in a while, the second half street numbers might actually
Speaker 3: be too low. So that's it's not like a super
Speaker 3: compelling call. But I felt uncomfortable Push although they were
Speaker 3: port earning next week, I make I may be kicking myself,
Speaker 3: like we'll still see in Vidia, however, so it's clearly
Speaker 3: you know you're ready. It hasn't reached its prior peak
Speaker 3: from a stock price base, and the reason the numbers
Speaker 3: have come down a lot. I mean, let's be honest,
Speaker 3: the gaining you know, business was was in plated significantly
Speaker 3: by crypto, right, and so that's all come out right.
Speaker 3: And then you know with data center, you had some
Speaker 3: impacts from from China. China general was weak, and then
Speaker 3: we had some of the export controls that they had
Speaker 3: to work their way around, and see had some issues there. Now,
Speaker 3: all of that being said, graphics cards in gaming, we
Speaker 3: talked about some of these inventory corrections. Graphics cards actually
Speaker 3: corrected the most and the most rapidly. So those have
Speaker 3: already hit bottom and they're growing again. And in VideA
Speaker 3: has got a product cycle there that they just kicked off.
Speaker 3: The new cards are called Lovelace and they and they
Speaker 3: look really good and especially behind and they're starting to
Speaker 3: fill out like the rest of the stack. So gaming
Speaker 3: is okay. And then in data centering again this you know,
Speaker 3: this generative AI has really caught everybody's fancy. And in
Speaker 3: Vivia had a data center of and they're saying that
Speaker 3: they were at the beginning of a product cycle in
Speaker 3: data centerm and you know, they had an advantage. A
Speaker 3: couple of weeks ago, they're their GtC event where they
Speaker 3: actually basically and directly said we're seeing upside from generative
Speaker 3: AI even now, right, So people have been buying in
Speaker 3: VideA on on on those on that thesis, and like
Speaker 3: the last the stock hit the peak at these peaks,
Speaker 3: at least in terms of valuation. The issue is we
Speaker 3: were at the peak of their product cycles and numbers
Speaker 3: came down. This time, valuations kind of went back to
Speaker 3: where they were at those peaks, but were the skinning
Speaker 3: of the product cycles, and numbers are probably going up
Speaker 3: knock knock down.
Speaker 1: So that's that's why Stacy I joked at the beginning
Speaker 1: that we could talk about about this for three hours,
Speaker 1: and I'm sure we could. Sure there's such a deep area,
Speaker 1: but that was a great overview of just like the
Speaker 1: state of competition, of the state of play, and the
Speaker 1: economics of this a very good way for us to
Speaker 1: sort of enter talking about AI. Stum more broadly, thank
Speaker 1: you so much for coming back on online.
Speaker 3: My pleasure. Anytime you guys want me here, just let
Speaker 3: me now, all right.
Speaker 1: We'll have you back next week for Intel take care
Speaker 1: of State. I really like talking to Stacey. He's really
Speaker 1: good at explaining complicated Yeah.
Speaker 2: I know, he made a point of saying that he's
Speaker 2: not an AI expert, but I thought he did a
Speaker 2: pretty good job of explaining it. I do think the
Speaker 2: trajectory of how all this, I mean, this is such
Speaker 2: an obvious thing to say, but it's going to be
Speaker 2: really interesting to watch and how businesses adapt to this,
Speaker 2: and we're what's kind of fascinating to me is that
Speaker 2: we're already seeing that differentiation play out in the market,
Speaker 2: with in video shares up quite a bit and Intel,
Speaker 2: which is seen as not as competitive in the space,
Speaker 2: down quite a bit.
Speaker 1: I was really interested in some of his points about
Speaker 1: software in particular, and so I have realized that, Yeah,
Speaker 1: like I mean, I you know, like sometimes I see
Speaker 1: like someone will post on Twitter it's like, look at
Speaker 1: this cool thing and video just rolled out where they
Speaker 1: can make your face look like something else or whatever.
Speaker 1: But thinking about like how important that is in terms
Speaker 1: of like, Okay, you and I want to start an
Speaker 1: AI company and idea for a large language model or
Speaker 1: something specifically have a model to train. There's going to
Speaker 1: be a big advantage going with the company that has
Speaker 1: this huge like wealth of like libraries and code bases
Speaker 1: and specific tools around specific industries as opposed to it
Speaker 1: seems like where some of the other competitors are, or
Speaker 1: it's just much more technically challenging to even like use
Speaker 1: the chips if they exist, like Google's.
Speaker 2: TPUs totally the other thing that caught my attention, And
Speaker 2: I know these are very different spaces in many ways,
Speaker 2: but there's so much of the terminology and like that's
Speaker 2: very reminiscent of crypto. So just the idea of like
Speaker 2: an AI winter and a crypto winter, and you can see,
Speaker 2: I mean, you can see the pivot happening right now
Speaker 2: from like crypto people moving into AI. So that's going
Speaker 2: to be interesting to watch play out. Like how much
Speaker 2: of it is hype classic sort of gartment hype cycle
Speaker 2: versus the real thing.
Speaker 1: But you know, two things, it's absolutely you know, so
Speaker 1: two things I think would be interesting. It'd be interesting
Speaker 1: to go back to like past AI summers, like what
Speaker 1: were some past periods which people thought we made this
Speaker 1: break through and then what happened? So that might be
Speaker 1: an interesting And then the other thing is like, look
Speaker 1: like you know, in twenty twenty three, I have never
Speaker 1: actually like found a reason I've ever felt compelled to
Speaker 1: like need to use a blockchain for something. And I
Speaker 1: get use out of chad GPT on something like almost
Speaker 1: every day. And so for example, we recently did an
Speaker 1: episode you know, yeah, look, we'll do an episode now
Speaker 1: of a question. At the end, they're like, oh, what
Speaker 1: is the difference Like yesterday, you know, we recently did
Speaker 1: an episode on like lending, and so it's like, oh,
Speaker 1: what's the difference sort of structurally between the leverage loan
Speaker 1: market and the private debt market. It's like, this might
Speaker 1: be an interesting question for a chat GPT, and like
Speaker 1: I got this like very useful, clear answer from it
Speaker 1: that like I couldn't have gotten perhaps as easily from
Speaker 1: a Google search. So I do think like some of
Speaker 1: these hype cycles like are really useful, But like I
Speaker 1: am already in my daily life and very already getting
Speaker 1: use out of this technology in a way that I
Speaker 1: cannot say for anything related like web three. No, that
Speaker 1: is very true.
Speaker 2: And you know the fact that this only came out
Speaker 2: a few months ago and everyone has been talking about
Speaker 2: it and experimenting with it kind of speaks for itself.
Speaker 1: Shall we leave it there? Let's leave it there.
Speaker 2: This has been another episode of the Oddlots podcast. I'm
Speaker 2: Tracy Alloway. You can follow me on Twitter at Tracy Alloway.
Speaker 1: And I'm Joe Wisenthal. You can follow me on Twitter
Speaker 1: at the Stalwart. Follow our guest Stacey Raskin. He's at
Speaker 1: s Raskin. Follow our producers Carmen Rodriguez at Carmen Arman
Speaker 1: and Dash o' bennett at dashbot. And check out all
Speaker 1: of our podcasts at Bloomberg under the handle at podcasts,
Speaker 1: and for more Oddlots content, go to Bloomberg dot com
Speaker 1: slash odd Lots. We blog, we post transcripts, we have
Speaker 1: a newsletter, and check out the Odd Loots Discord people
Speaker 1: listeners chatting twenty four to seven about all the things
Speaker 1: we talk about here. We even have an AI specific
Speaker 1: world that's really fun and set and the semiconductor room,
Speaker 1: and so people chatting about these things. I even so
Speaker 1: listened to some questions for today from that group, so
Speaker 1: it's really fun. I like hanging out there. To go
Speaker 1: to Discord dot gg slash pop. Thanks for listening