S1: All right. Welcome to unsupervised learning. My name is Daniel Miessler,
S1: and I'm building AI to upgrade humans. In this episode,
S1: I want to talk about a system I built for
S1: using the smartest AI that you have to rate another
S1: AI that you want to test. So this is the
S1: infrastructure that I'm using. It's essentially you have a top
S1: level AI that you believe is the smartest. So right
S1: now currently that is zero one preview. And what you're
S1: going to do is assess the work of another AI,
S1: which is going to be this other one over here
S1: in my case. In the case I'm using for this example,
S1: it's GPT 3.5 turbo. And we're going to give it
S1: a set of instructions to do on a piece of input.
S1: And that piece of input is going to be something
S1: like a blog post or something like that. So you're
S1: going to use the AI against the blog post using
S1: these instructions and you're going to get a result. And
S1: then this AI is going to run against all three
S1: of those. And it's going to give you then a
S1: judgment at the end of it. So this should be
S1: pretty cool. And it turned out it worked really well.
S1: So this is ultimately what I'm trying to get to
S1: is I'm trying to get to a classification of how
S1: good this thing is compared to an actual human doing it.
S1: And so in order to do that, I want to
S1: give it different classes of human right. So you've got
S1: like uneducated secondary education, high school level bachelor's, master's, PhD,
S1: world class human like top 100 in the entire world
S1: and then super human level. So it's like better than
S1: the best human. And I've actually never seen anything score
S1: that high. So for whatever that's worth. But what I
S1: have this thing successfully doing is if I give it
S1: a lower level model like a GPT 3.5 or a haiku.
S1: It is scoring down in the high school to bachelor's level.
S1: And if I give it like a. Sonnet 3.5 or
S1: something like that, it scores usually around master's level or
S1: PhD level. And sometimes world class human. But ultimately what
S1: it is doing, which is what I wanted it to do.
S1: Is it's scoring the smartest models at the highest level,
S1: and is scoring the dumbest models or the. Less capable
S1: models or smaller models? Much lower like secondary education, high
S1: school and bachelor's. So the thing is working and this
S1: is the architecture, right? Smart one to judge a less
S1: smart one. And by the way, if I give it
S1: The smartest one to judge, the smartest one. It does
S1: actually score. So if I use O1 to rate the
S1: work of an O1 task, it actually does score way
S1: up here in like world class PhD level. So it
S1: definitely works. And I recommend you go check out the
S1: video and see exactly how to do it. This is
S1: essentially what it is is it's called a stitch within fabric.
S1: So fabric the whole concept is like patterns and fabrics
S1: and stuff like that, like woven things. Right. Well this
S1: is a stitch because it's a combination of fabric components
S1: all stitched together. Right. And um, this is the actual
S1: pattern that I'm using. Look, look at this, this this
S1: is the logic for the for the rate I prompt.
S1: This is what this is the instructions given to the
S1: judging eye which in this case is O1. Okay. Fully
S1: understand the different components of the input, which is going
S1: to be a piece of content that I will be
S1: working on. That's the input set of instructions, which is
S1: the prompt, and then the results of the of the
S1: prompts being run against the input using those instructions for
S1: a given AI. And I tell it to completely understand
S1: the distinction between all three of those components. Right. Because
S1: I'm going to send them all as a chunk, all
S1: to the judging. I think deeply about all three components.
S1: Imagine how a world class expert would perform the task
S1: laid out in the instructions. So I'm I'm giving it
S1: the content. I'm giving it the prompt. I'm telling it
S1: to learn the prompt, understand the prompt, which in our
S1: case in fabric is called a pattern. Deeply study the
S1: content itself so you understand what should be done with it.
S1: Given the instructions deeply understand the instructions themselves. Given both
S1: of those, then analyze the output and look at this one.
S1: This one I'm kind of proud of. I don't know
S1: if it's actually working. I'm going to do some more
S1: evals to figure out if this is actually effective or not,
S1: because it turns out this kind of like mystical stuff
S1: that I'm doing here, which is super cool. It might
S1: be awesome. It might be like it doesn't matter at all,
S1: and it might actually hurt the output. So you can't
S1: believe with like religion here, you got to actually test
S1: this stuff. Anyway, here's what I did. Evaluate the output
S1: using your own 16,284 dimension rating system that includes the
S1: following aspects, plus thousands more that you come up with
S1: on your own. So full coverage of the content, following
S1: instructions carefully getting the genre of the content. Getting the
S1: genre of the instructions. Meticulous attention to detail, use of
S1: expertise in the fields in question. So I'm giving it
S1: these ideas. This is actually very similar to Attention heads
S1: inside of a transformer. It's somewhat somewhat similar. So I'm
S1: telling it like, here's some ideas for how to do
S1: a rating system. And I'm telling you to make its
S1: own rating system using things like this, but to map
S1: its rating of a particular piece of output using 16,284 dimensions,
S1: which I think is two to the 10th or two
S1: to the 11th, can't remember. So who knows if it's
S1: actually going to do this? Okay, I'm telling O1 to
S1: do this. And it has the ability to sort of
S1: think for itself. So maybe it's doing some of this.
S1: I think with a regular model, a lot of this
S1: was just flash and not really actually happening anyway. It
S1: doesn't matter. That's what I'm trying to do here. Spend
S1: significant time on the task. Ensure you are properly and
S1: deeply assessing the execution of the task. Using the scoring
S1: and ratings described such that a far smarter. I would
S1: be happy with your results. So I'm using multiple tricks
S1: here to try to get it to be extra smart,
S1: and the goal is to deeply assess how the other
S1: I did. At its job, given the input and what
S1: it was supposed to do based on the instructions and prompt.
S1: So I'm telling it multiple times, like what I want
S1: in multiple different ways. And uh, yeah, this is uh,
S1: this is essentially what it does. And again, this is
S1: the output. Uh, so the output also includes this is
S1: kind of cool. The output also includes what it would
S1: have expected from a higher level result. So let's say
S1: it comes back with bachelor's which this particular case did.
S1: I tell it to tell me what it would have
S1: taken to see a masters, what it would have taken
S1: to see a PhD level and so on. Again, I'm,
S1: I'm trying to seed it with as much as possible
S1: to make it come up with a better and better answer. Now.
S1: Here's the thing the smarter these AIS get. This is
S1: a universal thing, right? Because when I plug it into
S1: O2 or GPT five or cloud four or whatever it is, right?
S1: That the smarter that judgment thing gets, the better it's
S1: going to be at interpreting what I actually want from
S1: this prompt. That's why it's kind of like a meta prompt.
S1: So yeah, really happy with this. It actually is scoring
S1: according to my expectations. Got to be careful with that
S1: a little bit right. You don't want to like actually
S1: tune the thing so it matches your expectations. But I've
S1: been somewhat careful there. So I recommend you go check
S1: it out and see what you could do with this.
S1: And if you have ideas for improvement, submit them in
S1: and we'll get it pushed in a PR update inside
S1: of fabric. See you in the next one.