1 00:00:17,547 --> 00:00:20,347 S1: All right. Welcome to unsupervised learning. My name is Daniel Miessler, 2 00:00:21,147 --> 00:00:25,467 S1: and I'm building AI to upgrade humans. In this episode, 3 00:00:25,467 --> 00:00:30,627 S1: I want to talk about a system I built for 4 00:00:30,787 --> 00:00:36,067 S1: using the smartest AI that you have to rate another 5 00:00:36,067 --> 00:00:38,947 S1: AI that you want to test. So this is the 6 00:00:38,947 --> 00:00:41,947 S1: infrastructure that I'm using. It's essentially you have a top 7 00:00:41,947 --> 00:00:46,067 S1: level AI that you believe is the smartest. So right 8 00:00:46,067 --> 00:00:50,626 S1: now currently that is zero one preview. And what you're 9 00:00:50,626 --> 00:00:53,347 S1: going to do is assess the work of another AI, 10 00:00:53,387 --> 00:00:55,307 S1: which is going to be this other one over here 11 00:00:55,667 --> 00:00:58,547 S1: in my case. In the case I'm using for this example, 12 00:00:58,547 --> 00:01:03,466 S1: it's GPT 3.5 turbo. And we're going to give it 13 00:01:03,467 --> 00:01:06,467 S1: a set of instructions to do on a piece of input. 14 00:01:09,547 --> 00:01:11,707 S1: And that piece of input is going to be something 15 00:01:11,707 --> 00:01:14,027 S1: like a blog post or something like that. So you're 16 00:01:14,027 --> 00:01:17,187 S1: going to use the AI against the blog post using 17 00:01:17,187 --> 00:01:20,347 S1: these instructions and you're going to get a result. And 18 00:01:20,347 --> 00:01:22,867 S1: then this AI is going to run against all three 19 00:01:22,867 --> 00:01:27,427 S1: of those. And it's going to give you then a 20 00:01:27,427 --> 00:01:32,427 S1: judgment at the end of it. So this should be 21 00:01:32,427 --> 00:01:35,787 S1: pretty cool. And it turned out it worked really well. 22 00:01:36,827 --> 00:01:38,787 S1: So this is ultimately what I'm trying to get to 23 00:01:38,827 --> 00:01:42,067 S1: is I'm trying to get to a classification of how 24 00:01:42,067 --> 00:01:44,907 S1: good this thing is compared to an actual human doing it. 25 00:01:45,307 --> 00:01:47,467 S1: And so in order to do that, I want to 26 00:01:47,467 --> 00:01:50,227 S1: give it different classes of human right. So you've got 27 00:01:50,267 --> 00:01:57,067 S1: like uneducated secondary education, high school level bachelor's, master's, PhD, 28 00:01:57,107 --> 00:02:00,387 S1: world class human like top 100 in the entire world 29 00:02:00,947 --> 00:02:04,307 S1: and then super human level. So it's like better than 30 00:02:04,867 --> 00:02:09,306 S1: the best human. And I've actually never seen anything score 31 00:02:09,347 --> 00:02:14,267 S1: that high. So for whatever that's worth. But what I 32 00:02:14,267 --> 00:02:17,947 S1: have this thing successfully doing is if I give it 33 00:02:17,947 --> 00:02:22,306 S1: a lower level model like a GPT 3.5 or a haiku. 34 00:02:23,026 --> 00:02:26,787 S1: It is scoring down in the high school to bachelor's level. 35 00:02:28,227 --> 00:02:33,547 S1: And if I give it like a. Sonnet 3.5 or 36 00:02:33,587 --> 00:02:37,227 S1: something like that, it scores usually around master's level or 37 00:02:37,227 --> 00:02:44,306 S1: PhD level. And sometimes world class human. But ultimately what 38 00:02:44,306 --> 00:02:46,906 S1: it is doing, which is what I wanted it to do. 39 00:02:46,947 --> 00:02:51,427 S1: Is it's scoring the smartest models at the highest level, 40 00:02:51,427 --> 00:02:56,026 S1: and is scoring the dumbest models or the. Less capable 41 00:02:56,026 --> 00:03:02,346 S1: models or smaller models? Much lower like secondary education, high 42 00:03:02,346 --> 00:03:08,186 S1: school and bachelor's. So the thing is working and this 43 00:03:08,186 --> 00:03:11,947 S1: is the architecture, right? Smart one to judge a less 44 00:03:11,947 --> 00:03:14,387 S1: smart one. And by the way, if I give it 45 00:03:15,267 --> 00:03:18,707 S1: The smartest one to judge, the smartest one. It does 46 00:03:18,746 --> 00:03:21,987 S1: actually score. So if I use O1 to rate the 47 00:03:22,026 --> 00:03:27,867 S1: work of an O1 task, it actually does score way 48 00:03:27,867 --> 00:03:33,347 S1: up here in like world class PhD level. So it 49 00:03:33,346 --> 00:03:35,827 S1: definitely works. And I recommend you go check out the 50 00:03:35,827 --> 00:03:38,507 S1: video and see exactly how to do it. This is 51 00:03:38,507 --> 00:03:42,267 S1: essentially what it is is it's called a stitch within fabric. 52 00:03:42,547 --> 00:03:46,427 S1: So fabric the whole concept is like patterns and fabrics 53 00:03:46,427 --> 00:03:49,867 S1: and stuff like that, like woven things. Right. Well this 54 00:03:49,867 --> 00:03:54,147 S1: is a stitch because it's a combination of fabric components 55 00:03:54,507 --> 00:03:58,826 S1: all stitched together. Right. And um, this is the actual 56 00:03:58,827 --> 00:04:01,587 S1: pattern that I'm using. Look, look at this, this this 57 00:04:01,587 --> 00:04:04,747 S1: is the logic for the for the rate I prompt. 58 00:04:04,947 --> 00:04:08,307 S1: This is what this is the instructions given to the 59 00:04:08,347 --> 00:04:12,467 S1: judging eye which in this case is O1. Okay. Fully 60 00:04:12,467 --> 00:04:15,747 S1: understand the different components of the input, which is going 61 00:04:15,787 --> 00:04:17,707 S1: to be a piece of content that I will be 62 00:04:17,707 --> 00:04:20,427 S1: working on. That's the input set of instructions, which is 63 00:04:20,427 --> 00:04:24,867 S1: the prompt, and then the results of the of the 64 00:04:24,867 --> 00:04:31,827 S1: prompts being run against the input using those instructions for 65 00:04:31,827 --> 00:04:35,187 S1: a given AI. And I tell it to completely understand 66 00:04:35,187 --> 00:04:37,747 S1: the distinction between all three of those components. Right. Because 67 00:04:37,747 --> 00:04:40,387 S1: I'm going to send them all as a chunk, all 68 00:04:40,427 --> 00:04:44,347 S1: to the judging. I think deeply about all three components. 69 00:04:44,347 --> 00:04:47,907 S1: Imagine how a world class expert would perform the task 70 00:04:47,907 --> 00:04:51,707 S1: laid out in the instructions. So I'm I'm giving it 71 00:04:51,707 --> 00:04:54,707 S1: the content. I'm giving it the prompt. I'm telling it 72 00:04:54,707 --> 00:04:57,387 S1: to learn the prompt, understand the prompt, which in our 73 00:04:57,387 --> 00:05:01,387 S1: case in fabric is called a pattern. Deeply study the 74 00:05:01,387 --> 00:05:04,307 S1: content itself so you understand what should be done with it. 75 00:05:04,307 --> 00:05:10,107 S1: Given the instructions deeply understand the instructions themselves. Given both 76 00:05:10,107 --> 00:05:12,587 S1: of those, then analyze the output and look at this one. 77 00:05:12,587 --> 00:05:14,707 S1: This one I'm kind of proud of. I don't know 78 00:05:14,707 --> 00:05:17,667 S1: if it's actually working. I'm going to do some more 79 00:05:17,707 --> 00:05:20,427 S1: evals to figure out if this is actually effective or not, 80 00:05:21,547 --> 00:05:23,987 S1: because it turns out this kind of like mystical stuff 81 00:05:23,987 --> 00:05:27,347 S1: that I'm doing here, which is super cool. It might 82 00:05:27,347 --> 00:05:30,867 S1: be awesome. It might be like it doesn't matter at all, 83 00:05:31,387 --> 00:05:35,267 S1: and it might actually hurt the output. So you can't 84 00:05:35,267 --> 00:05:38,107 S1: believe with like religion here, you got to actually test 85 00:05:38,107 --> 00:05:41,787 S1: this stuff. Anyway, here's what I did. Evaluate the output 86 00:05:41,787 --> 00:05:47,747 S1: using your own 16,284 dimension rating system that includes the 87 00:05:47,747 --> 00:05:51,747 S1: following aspects, plus thousands more that you come up with 88 00:05:51,747 --> 00:05:55,627 S1: on your own. So full coverage of the content, following 89 00:05:55,667 --> 00:06:00,707 S1: instructions carefully getting the genre of the content. Getting the 90 00:06:00,747 --> 00:06:05,707 S1: genre of the instructions. Meticulous attention to detail, use of 91 00:06:05,707 --> 00:06:08,507 S1: expertise in the fields in question. So I'm giving it 92 00:06:08,507 --> 00:06:12,426 S1: these ideas. This is actually very similar to Attention heads 93 00:06:12,427 --> 00:06:17,627 S1: inside of a transformer. It's somewhat somewhat similar. So I'm 94 00:06:17,627 --> 00:06:20,587 S1: telling it like, here's some ideas for how to do 95 00:06:20,587 --> 00:06:22,987 S1: a rating system. And I'm telling you to make its 96 00:06:22,987 --> 00:06:27,627 S1: own rating system using things like this, but to map 97 00:06:27,627 --> 00:06:34,347 S1: its rating of a particular piece of output using 16,284 dimensions, 98 00:06:34,867 --> 00:06:37,707 S1: which I think is two to the 10th or two 99 00:06:37,707 --> 00:06:43,267 S1: to the 11th, can't remember. So who knows if it's 100 00:06:43,307 --> 00:06:45,987 S1: actually going to do this? Okay, I'm telling O1 to 101 00:06:45,987 --> 00:06:47,947 S1: do this. And it has the ability to sort of 102 00:06:47,987 --> 00:06:50,587 S1: think for itself. So maybe it's doing some of this. 103 00:06:50,587 --> 00:06:53,387 S1: I think with a regular model, a lot of this 104 00:06:53,387 --> 00:06:58,067 S1: was just flash and not really actually happening anyway. It 105 00:06:58,067 --> 00:07:00,827 S1: doesn't matter. That's what I'm trying to do here. Spend 106 00:07:00,827 --> 00:07:03,387 S1: significant time on the task. Ensure you are properly and 107 00:07:03,387 --> 00:07:06,867 S1: deeply assessing the execution of the task. Using the scoring 108 00:07:06,867 --> 00:07:11,827 S1: and ratings described such that a far smarter. I would 109 00:07:11,867 --> 00:07:14,187 S1: be happy with your results. So I'm using multiple tricks 110 00:07:14,187 --> 00:07:16,467 S1: here to try to get it to be extra smart, 111 00:07:17,187 --> 00:07:19,667 S1: and the goal is to deeply assess how the other 112 00:07:19,667 --> 00:07:26,427 S1: I did. At its job, given the input and what 113 00:07:26,427 --> 00:07:29,507 S1: it was supposed to do based on the instructions and prompt. 114 00:07:29,907 --> 00:07:32,627 S1: So I'm telling it multiple times, like what I want 115 00:07:32,707 --> 00:07:39,507 S1: in multiple different ways. And uh, yeah, this is uh, 116 00:07:39,947 --> 00:07:42,147 S1: this is essentially what it does. And again, this is 117 00:07:42,147 --> 00:07:47,547 S1: the output. Uh, so the output also includes this is 118 00:07:47,547 --> 00:07:51,867 S1: kind of cool. The output also includes what it would 119 00:07:51,907 --> 00:07:56,947 S1: have expected from a higher level result. So let's say 120 00:07:56,947 --> 00:07:59,987 S1: it comes back with bachelor's which this particular case did. 121 00:08:00,667 --> 00:08:02,587 S1: I tell it to tell me what it would have 122 00:08:02,587 --> 00:08:05,267 S1: taken to see a masters, what it would have taken 123 00:08:05,267 --> 00:08:09,667 S1: to see a PhD level and so on. Again, I'm, 124 00:08:09,667 --> 00:08:11,987 S1: I'm trying to seed it with as much as possible 125 00:08:11,987 --> 00:08:15,187 S1: to make it come up with a better and better answer. Now. 126 00:08:17,747 --> 00:08:20,587 S1: Here's the thing the smarter these AIS get. This is 127 00:08:20,587 --> 00:08:23,107 S1: a universal thing, right? Because when I plug it into 128 00:08:23,147 --> 00:08:28,507 S1: O2 or GPT five or cloud four or whatever it is, right? 129 00:08:30,787 --> 00:08:34,147 S1: That the smarter that judgment thing gets, the better it's 130 00:08:34,146 --> 00:08:36,347 S1: going to be at interpreting what I actually want from 131 00:08:36,347 --> 00:08:39,187 S1: this prompt. That's why it's kind of like a meta prompt. 132 00:08:40,227 --> 00:08:45,267 S1: So yeah, really happy with this. It actually is scoring 133 00:08:46,307 --> 00:08:49,427 S1: according to my expectations. Got to be careful with that 134 00:08:49,426 --> 00:08:51,667 S1: a little bit right. You don't want to like actually 135 00:08:51,666 --> 00:08:55,307 S1: tune the thing so it matches your expectations. But I've 136 00:08:55,307 --> 00:08:58,067 S1: been somewhat careful there. So I recommend you go check 137 00:08:58,067 --> 00:09:00,306 S1: it out and see what you could do with this. 138 00:09:00,587 --> 00:09:03,107 S1: And if you have ideas for improvement, submit them in 139 00:09:03,107 --> 00:09:06,747 S1: and we'll get it pushed in a PR update inside 140 00:09:06,747 --> 00:09:09,267 S1: of fabric. See you in the next one.