S1: All right. Welcome to unsupervised learning. This is Daniel. Okay.
S1: I'm going to start off with something that just happened.
S1: So strawberry just launched. It is being called zero one,
S1: which I assume the O might mean Orion because people
S1: were saying that it might have been called Orion. So
S1: this is the new model from OpenAI. And I've been
S1: messing with it for a couple hours already. So, uh,
S1: first thing is I gave it a task of building
S1: a business plan for something I'm working on, and it
S1: produced output that was far and above better than Ford
S1: or Sonnet 3.5. Yeah, it was really quite, quite good. Uh,
S1: very detailed. It took quite a while. There's no streaming
S1: in the API, so it feels a little rough compared
S1: to the current models. But whatever that, that will come
S1: with time. Uh, it's quite expensive. So basically I did
S1: a couple of conversation analysis, uh, Analysis by passing in, um,
S1: you know, conversations like transcripts from podcasts. And I think
S1: I did 2 or 3 of those, and it was
S1: almost a dollar. And there's also a mini version which
S1: is way less expensive, but I'm trying to test the capabilities,
S1: so I'm using the full model. But yeah, a few
S1: requests for a dollar, whereas I would say probably many
S1: dozen or a couple of hundred requests are normally like
S1: a few dollars. So it's many factors more expensive. So
S1: just something to consider. As with most models, you don't
S1: need the biggest, best or latest. This is a tweet
S1: I just put out, so I'm going through it. So
S1: this does one particular thing well, which is in better
S1: than anything else, which is pausing to think and actually
S1: going step by step. That's kind of like the magic sauce.
S1: Here is the chain of thought reasoning. So if you
S1: don't need that for what you're trying to do, you
S1: definitely shouldn't use this because it's more expensive, takes longer
S1: to run. All those sorts of reasons, this type of
S1: model and similar ones going forward are going to massively
S1: benefit from high quality prompting. So things like we use
S1: with fabric, which is open source on GitHub if you're
S1: not familiar, but you probably are if you're listening to this.
S1: But essentially, the more you know what you want and
S1: the better you can articulate that, the better this is
S1: going to perform, because it is a chain of thought
S1: sort of concept. So the more you give it to
S1: help with that, the better. Okay, sorry about that. I
S1: was just checking to make sure I wasn't doxxing anyone
S1: by showing you my messages, but I was not, so
S1: I don't have to rerecord. Okay, so, um. continuing on
S1: here and going to expand this window fully. Okay. So, um, yeah,
S1: the better you can articulate all of this. And by
S1: the way, I want to do an edit there for
S1: the team. So the better you can articulate this stuff
S1: in exactly what you want, the better things are. That's
S1: the bottom line here. So a lot of people are
S1: going to question is this AGI or not? Uh, Sam
S1: Altman already responded. He's like, yeah, this absolutely is not.
S1: So that that should end it in terms of the
S1: actual creator of this thing saying it's not. I also
S1: don't think it is either. Uh, whatever that matters for.
S1: But bottom line is, anyone who's making the claim of
S1: like this is or isn't AGI. Here's my request to
S1: the internet. Basically, anyone claiming something is or is not
S1: should also provide a concise and achievable definition of what
S1: that means. And I have one here, of course, which
S1: is I've talked about before, whether the ability of an AI,
S1: whether a model or a product or a system to
S1: perform the work of an average US based knowledge worker
S1: in 2002, and I say 2002 because that's pre GPT four. Right.
S1: So basically pre AI in these terms anyway. So yeah
S1: anyone who's talking about AGI make sure they have a definition.
S1: Otherwise you're just wasting your time because the entire conversation
S1: will be about definitions. And you might not even figure
S1: that out until fucking two hours later. Sorry for the cussing.
S1: All right. One of the most important changes to me
S1: with this model. This this is massive, okay? This is
S1: the first model that does this. Uh, it's the first
S1: model of its kind to do this very, very interesting.
S1: It's actually spending tokens To think, okay, before you had
S1: input and you had output and you were being charged in,
S1: the amount of work that was being done was based
S1: on the number of tokens coming in and the number
S1: of tokens coming out, and that that was the extent
S1: of it. What's happening now is you have tokens coming
S1: in and you have tokens coming out, but there are
S1: tokens being spent while it's thinking. It's actually thinking and
S1: reasoning through how to solve the problem. And what's really
S1: fascinating about this is that you now have multiple factors here. Okay.
S1: So you can do better prompting. And this is the
S1: next piece here. Number seven. You could do better prompting.
S1: You could use a smarter model. Or you could have
S1: the model think harder on the problem. And these are
S1: all going to be levers and knobs that we have
S1: to get better results from AI. And this is the
S1: first time we have this third level lever of like
S1: actually having it think, right. So at inference time, more
S1: effort being spent. And they actually say in the blog
S1: post they're like, hey, look, right now it's taking, you know,
S1: a few seconds to think or whatever, and it's going
S1: to get back great results. But we're thinking, what if
S1: it thinks for minutes? What if it thinks for hours?
S1: What if it thinks for days or weeks? And not
S1: only that, but we give it more compute power to think.
S1: And the example they gave, I think this was an
S1: OpenAI post. The example they gave here was how much
S1: do you want to solve cancer? What if you could
S1: build a data center? What if you had one data
S1: center just for working on cancer and one data center
S1: just for working on aging and so on? Okay. And
S1: you basically have models like this that scale with the
S1: inference difficulty based on the amount of difficulty of the,
S1: of the thinking. And then, of course, you have a
S1: smart model and a good neural net and all that, right?
S1: Scalability of the of the neural net. So maybe that's
S1: GPT five, GPT six, whatever. Combined with the good prompting,
S1: combined with this thinking capability and combined with, you know,
S1: all those things unified into the combined with having that
S1: giant infrastructure to run it so that that's insane. Um,
S1: and the scales all the way down to like, the
S1: smallest stupid problem where it's just like, whatever, GPT three
S1: and you get back the answer almost instantaneously. In fact,
S1: forget GPT three. It's some local model that only does
S1: one thing well. You're spending almost no resources whatsoever. It
S1: just goes to your phone, bounces back immediately, doesn't go anywhere,
S1: barely costs any cycles of a GPU or a CPU
S1: because you don't need those resources to run. Because it's
S1: just an easy thing to answer. So now we're talking
S1: about AI that scales with the difficulty of the problem, right? With,
S1: you know, cancer, aging, getting out of the solar system,
S1: escaping the sun, expanding, ultimately heat, death of the universe.
S1: That's a big one, right? Because entropy kills everything. So ultimately,
S1: we're going to need a way out of here at
S1: some point, assuming we survive that long. Not happening anytime soon.
S1: I wouldn't worry about that. But these are the types
S1: of things that are really exciting. You know, the size
S1: of the problem being being a factor, for which I
S1: you point at it with lots and lots of different
S1: knobs and levers controlling that decision. So I think that's
S1: really cool. Another important thing to mention is that the
S1: innovation seems independent of what we were waiting for for
S1: GPT five. So based on all I read, all the
S1: releases from OpenAI. And I've seen all the rumors and,
S1: you know, talked to a bunch of people who've been
S1: speculating about this. And this seems completely independent from, oh,
S1: is this GPT four oh, is it for oh, is
S1: it five? Is it an early version of five. Doesn't
S1: really matter. It's like a separate axis. This is like
S1: a capability. This is like thinking capability, which is on
S1: a separate axis from how big or smart is the
S1: neural net, right? Or how big or smart is the
S1: is the model. So really, really cool to think about
S1: those being two separate things because now we can start
S1: thinking about, okay, well if GPT five is still going
S1: to come out, you know, later this year, beginning in
S1: next year or whenever it's going to come out and
S1: whatever they're going to call it. Well, imagine GPT five
S1: with this thinking capability. That's cool. So presumably this is
S1: just a feature that you can add onto any model,
S1: which is what we're just talking about. And I think
S1: this is okay. This is really, really crucial here. I've
S1: been talking for a long time about slack in the
S1: rope and tricks that we're going to use to jump
S1: ahead in, um, advancement of AI, so so check this out.
S1: A lot of people are like, oh, we're running into
S1: a data wall. Oh, neural nets are only so good
S1: they can only get so good. We've already hit a thing.
S1: I mean, so many, so many people are saying things
S1: like this that just sound absolutely ridiculous to me. First
S1: of all, they were the ones saying we wouldn't be here.
S1: And so now we are here and everyone's surprised and
S1: they're like, well, here's what we know for sure is
S1: we're not going to get any better. How can I
S1: believe you if you didn't predict any of this and
S1: you were absolutely certain back then, and now you're absolutely
S1: certain it's not going to jump ahead again, right? Leopold
S1: talks about this in his paper. There's lots of different
S1: ways to get better. There's the architecture of the model.
S1: There's the size of the model. I forget what all
S1: levers he had, but it's the architecture of the model,
S1: the size of the model. And I think it was
S1: hobbling was the other one, which is what I called
S1: like a year ago. Slack in the rope or tricks
S1: we're going to. This is what I told a friend
S1: of mine who's really smart in this stuff. I said,
S1: watch this. We're going to find multiple tricks where we're
S1: messing around in percentage points, and then we find a
S1: thing and it jumps us 2 or 3 or 5
S1: or 10 x or 100 x ahead. And and I
S1: actually learned this from him. Uh, I actually learned this
S1: from him. He was like, hey, you know, there are
S1: things that jump you ahead. Um, and I think he
S1: gave me example from some public paper or whatever. And
S1: it was an example of like a big jump. And
S1: my natural intuition was there's going to be a lot
S1: more of those, and they're not coming from pursuing along
S1: this axis, which is difficult. They are actually just hanging
S1: off to the side. It's like, oh, did you know
S1: if you just changed the color of this? Hey, did
S1: you know if you just orient the data backward instead
S1: of forward? Hey, did you know if you just prune
S1: the data in this way or if you add this
S1: particular data set or. And I'm just making up these examples,
S1: but simple things that you wouldn't think would work. And
S1: this is why Leopold talks about if you automate an
S1: AI engineer or an AI researcher, is what he called it.
S1: That's when it gets completely silly, because they have the
S1: ability to now go and try a whole bunch of
S1: these things, including these tricks. Um, all this to say
S1: that the slack in the rope or this series of
S1: tricks is going to keep multiplying our advances, and that's
S1: at the same time that we're working on the algorithms. Oh,
S1: that was the other. That was the other factor is algorithms.
S1: That was this is going to happen at the same
S1: time we're working on the algorithms to make those better.
S1: We're also working on the size of the neural net, um,
S1: and the quality and the structure. And everything about the
S1: neural net is going to get bigger and more powerful,
S1: but mostly just a matter of size, number of parameters.
S1: But all those things are changing at the same time
S1: as we're finding all these tricks. Right. So we're talking
S1: about this is just begun. And this is what people
S1: don't realize. This is just now starting. We're going to
S1: look back in two years and be like, what was that?
S1: That was silly. Right. And so I really want to
S1: warn people against thinking we're hitting some kind of a wall.
S1: Think of it this way. We just found alien technology.
S1: We have no idea how it works. And we're like,
S1: poking it with a stick and it's already spitting out
S1: amazing things. So think about that. Okay, we got a
S1: glowy ball. We don't know how it floats. We don't
S1: know how it's doing. Anti-Gravity, right? We don't know how
S1: it's doing this. We don't know how it's reflecting its surface.
S1: We don't know how it's coming up with these answers.
S1: We don't know how it got here from the other
S1: solar system. We don't know anything about it. You poke
S1: it with a stick and it tells this magic stuff
S1: and we're like, Holy crap, that's amazing. Somebody walks up,
S1: sees you poke it with a stick and goes, yeah,
S1: that's I mean, that's that's all it's ever going to
S1: be able to do. I mean, I've seen you poke
S1: it with a stick twice, and it gave you kind
S1: of a similar answer, which means that's all we could
S1: learn from this alien ball. That's their conclusion. I am
S1: certain that since you poked it with a stick while
S1: I was standing here three times, and it kind of
S1: gave you a similar answer. One it must be stupid.
S1: Two it's not as smart as us. And three, this
S1: is as as smart as it's ever going to be.
S1: This is the most it has to offer. That is
S1: the claim that's being made by these kind of like denialists,
S1: in my view. And that doesn't mean the current shiny
S1: ball is better than humans, or it should replace humans,
S1: or it could do everything we could do. Like, this
S1: is not a competition. Okay, here's a better way to
S1: think about this. This is not like a rock that
S1: we have animated. Think of it this way. If an
S1: alien comes here because someone else was like, hey, this
S1: is not thinking, this is processing. And I'm like, come on,
S1: come on. If you if an alien comes here, let's
S1: assume we know how our brain works. An alien comes
S1: here and we look at its brain, or it shows
S1: us its brain and it looks different. And we're like, oh,
S1: you guys do neurons and synapses different than us? Who's
S1: going to walk over and be like, well, since they're
S1: doing neurons and synapses different than us, they're not thinking.
S1: Only humans can think. And I'm like, they got here.
S1: They got here, didn't they? It's a little shiny ball.
S1: And they got here from whatever part of the galaxy
S1: or universe that they came from. They're obviously doing something right.
S1: And I is obviously doing something right too. So I
S1: think it's a little bit specious. Is that is that
S1: the name of the word? It's like specious to just
S1: magically assume that we are the best. Only we are
S1: thinking only we are special. Instead of thinking like we
S1: might have this nascent alien intelligence thing going on that
S1: actually is doing things that are very much analogous to us.
S1: It reminds me of the first time that I clicked
S1: around inside of Linux. This is like late 90s. I
S1: was messing with Linux. This must have been like 9798
S1: or something. I'm messing with Linux and I'm clicking around
S1: because I had started with windows and I'm like, oh,
S1: it opens windows and it opens things that I could
S1: click and navigate. Then I'm like, it's it's just like
S1: on Windows Explorer. And this like, blew me away. It
S1: absolutely blew me away that this was just a different
S1: way of doing the same thing. And that underneath this,
S1: there's a universal thing of you need to be able
S1: to browse files, you need to be able to open windows,
S1: you need to be able to close windows. And that
S1: clicked for me. And I'm like, oh, I guess like
S1: all operating systems are going to do this differently. It's
S1: the same with aliens. It's the same with like they
S1: might think differently, but whatever. They have to think, right.
S1: So why would we expect this synthetic intelligence that we've
S1: birthed to do it exactly the same way that we
S1: way that we do. We should not expect that we
S1: got here accidentally stumbling through time due to evolution. And
S1: we've got this version that we have and it's awesome, obviously.
S1: But like, that's way different than we invented this thing
S1: five years ago or whenever that was 2017, six years ago.
S1: And I know it goes further back than that. But
S1: you know what I'm saying? Transformers. All right. So that's that.
S1: And this this is becoming a long thing. But whatever
S1: we'll go with it. So yeah, basically we have no
S1: idea how early all of this is. We're likely to
S1: find ten, 20 or 200 more of these holy crap
S1: optimizations like this thinking thing before we start hitting any
S1: limits for neural network architecture or the transform transformer like.
S1: Plus we could just find something better than a transformer.
S1: You realize how how lucky we were to find the transformer.
S1: Like the people who made that paper. They're like, hey,
S1: this is this is a cool way we think this
S1: is a cool way of doing something. They didn't know
S1: what they had. Okay, you should watch a Karpathy talk
S1: about the transformer. He's like, this thing is a general
S1: purpose computer. This thing is insanely good at learning. He
S1: talks about different ways that it's better than humans at learning. Okay,
S1: some some people randomly found this thing and it shot
S1: us off. Okay. So so check this out. This is
S1: another example of finding tricks or slack in the rope
S1: just lying on the ground. So we stumble through AI
S1: for decades and decades and decades. And then someone's like, hey,
S1: this is kind of cool about this attention mechanism. Hey,
S1: what do you think about this architecture for a neural net? Boom!
S1: Now we have this take off. There's nothing saying somebody
S1: isn't going to be like, I like what you did
S1: with that transformer architecture. What if it looked like this instead?
S1: It might be 20 times better. It might be 2000
S1: times better. It might be 4% better. It doesn't matter.
S1: Like the we have only just begun. We have only
S1: just begun. I can absolutely guarantee you that assuming we
S1: don't kill ourselves off as a result of this, like
S1: that would set things back. But I'm trying to get
S1: you to think about things in this way because it's
S1: insane what's about to happen. And yeah, I'm going to
S1: have more examples here. I'm working on an example right
S1: here on this other screen. Uh, pretty cool thing I'm
S1: building with it. Um, okay. So that was that.