WEBVTT - Rethinking AI Harnesses 0:00:20.560 --> 0:00:22.600 Hey what's up? So I think the way we're thinking 0:00:22.600 --> 0:00:25.880 about harnesses is not correct. I think it was the 0:00:25.880 --> 0:00:28.480 right way to think about them before basically as like 0:00:28.520 --> 0:00:32.159 coding systems, like coding assistance and like a set of 0:00:32.160 --> 0:00:34.720 tools and stuff to do coding better and to release 0:00:34.760 --> 0:00:38.400 apps faster. I think we have to start thinking about 0:00:38.400 --> 0:00:42.360 this completely differently. The way I imagine a harness to be, 0:00:42.360 --> 0:00:45.280 the way I think about harnesses is kind of like 0:00:45.280 --> 0:00:48.760 a content management system. Okay. The problem that we're trying 0:00:48.760 --> 0:00:51.800 to solve, actually, is that it's not exactly clear what 0:00:51.800 --> 0:00:53.960 is in our brains as a kind of a feeling 0:00:53.960 --> 0:00:57.880 of what we want. And it's not easy even within 0:00:57.880 --> 0:01:02.480 our own brains to like, translate that feeling into actual words, right? 0:01:02.960 --> 0:01:05.640 Then we have to actually capture it properly, which like 0:01:05.680 --> 0:01:08.040 has been known as like prompt engineering, right? So we 0:01:08.039 --> 0:01:10.200 have a prompt engineering problem, but before that, we have 0:01:10.200 --> 0:01:12.840 a feeling in our brain problem and how that's going 0:01:12.840 --> 0:01:15.440 to translate into actual text, which we can then capture 0:01:15.440 --> 0:01:17.720 into a prompt and then which we can then give 0:01:17.720 --> 0:01:19.760 to an AI and hopefully it understands. Hopefully it's smart 0:01:19.759 --> 0:01:22.250 enough to understand the prompt, which is the whole purpose 0:01:22.250 --> 0:01:25.889 of prompt engineering. Well, I think we've confused this whole 0:01:25.930 --> 0:01:28.370 harness idea into thinking that it has to do with coding, 0:01:28.650 --> 0:01:32.410 thinking that it has to do with basically, what are 0:01:32.410 --> 0:01:34.009 the best tools we can give? What are the best 0:01:34.009 --> 0:01:35.890 prompts that we can write? What are the best like 0:01:35.930 --> 0:01:38.970 specs and prds and all of that plans, all of 0:01:39.009 --> 0:01:41.650 that stuff. Which is all true, but I think it 0:01:41.650 --> 0:01:43.210 makes way more sense. And this is the way I've 0:01:43.210 --> 0:01:46.010 been building our system called Pi, which is now being 0:01:46.009 --> 0:01:48.490 renamed to life OS. So if you see life OS, 0:01:48.490 --> 0:01:51.250 that's actually Pi. And the model that we use with 0:01:51.250 --> 0:01:54.450 life OS is essentially to have your harness be essentially 0:01:54.450 --> 0:01:57.490 your life. Your AI harness is everything that you're trying 0:01:57.490 --> 0:02:00.610 to accomplish. It's all of your intents. It's your current state, 0:02:00.610 --> 0:02:03.410 it's your desired state. It's what you're actually trying to 0:02:03.410 --> 0:02:05.690 achieve in life, right? So it's your personal goals, it's 0:02:05.690 --> 0:02:08.570 your work goals. It's your, you know, coding projects and 0:02:08.570 --> 0:02:12.489 your apps and your desires and your preferences and your challenges. 0:02:13.050 --> 0:02:15.730 Now you might think, well, okay, but that's life stuff. 0:02:15.730 --> 0:02:18.169 And over here we have coding stuff. But I think 0:02:18.169 --> 0:02:20.210 if you think about this in the way that I'm describing, 0:02:20.210 --> 0:02:23.940 you'll notice that this all blurs, this all disappears, because 0:02:23.940 --> 0:02:25.380 what we're trying to do is get to a point 0:02:25.380 --> 0:02:29.139 where we can say just the smallest thing, okay? We 0:02:29.139 --> 0:02:31.220 could say, imagine you had a twin sister or a 0:02:31.220 --> 0:02:34.220 twin brother, and they were like synced with you. You 0:02:34.220 --> 0:02:36.499 could say to them, and this is like a really 0:02:36.500 --> 0:02:39.300 important point. You could say to that person, you could say, hey, 0:02:39.299 --> 0:02:41.139 do you think this works for us? And they could 0:02:41.139 --> 0:02:44.820 just like glance at, say, an event schedule, or they 0:02:44.820 --> 0:02:47.820 could glance at like a new vehicle or they could 0:02:47.820 --> 0:02:50.500 glance at a job description. When I say, hey, do 0:02:50.500 --> 0:02:55.500 you think this will work for us? I have communicated volumes. Okay. 0:02:55.540 --> 0:02:58.260 This this is the whole magic of a harness. Okay. 0:02:58.540 --> 0:03:01.700 It's it's context engineering. It's prompt engineering. It's a whole 0:03:01.700 --> 0:03:03.820 bunch of other things that we've been talking about for years. 0:03:04.100 --> 0:03:08.139 But it's different, okay? It's different. It is compression. It 0:03:08.139 --> 0:03:11.580 is efficiency of going from a brain thought or a 0:03:11.579 --> 0:03:14.700 feeling all the way through the translation layers into the 0:03:14.700 --> 0:03:17.139 prompt engineering, all the way to the AI. The goal 0:03:17.139 --> 0:03:19.019 is to be able to say as little as possible 0:03:19.739 --> 0:03:23.939 to achieve the best possible context transfer and intent transfer 0:03:24.100 --> 0:03:28.350 and intent articulation into your AI. That is the purpose 0:03:28.350 --> 0:03:30.429 of the harness because it doesn't matter what you're talking about. 0:03:30.430 --> 0:03:32.470 It doesn't matter if you say, hey, draw me a 0:03:32.470 --> 0:03:35.950 butterfly wing. That is beautiful. Okay, that is very abstract. 0:03:35.950 --> 0:03:41.230 It's very general, it's very non-technical. But there are questions there. 0:03:41.270 --> 0:03:42.990 What do you mean, a butterfly wing? Like, how big 0:03:42.990 --> 0:03:45.550 should it be? Like 1 or 2, you know, should 0:03:45.550 --> 0:03:49.030 I use watercolors? Like, what is a pretty butterfly wing? 0:03:49.070 --> 0:03:53.070 According to you write this. These questions need to be answered. 0:03:53.310 --> 0:03:55.710 And if you have that in your harness, because guess what? 0:03:55.710 --> 0:03:58.710 You're a famous artist known for drawing butterfly wings. Well, 0:03:58.710 --> 0:04:00.350 hopefully you wouldn't be using an AI to draw your 0:04:00.350 --> 0:04:01.630 wing for you if that were the case. Let's just 0:04:01.630 --> 0:04:03.830 say you're a famous AI artist and you put out 0:04:03.870 --> 0:04:07.510 different animals or whatever. Whatever it is, you should have 0:04:07.510 --> 0:04:10.830 so much context inside of your harness of what great 0:04:10.830 --> 0:04:13.350 art looks like, what great AI art in this case 0:04:13.350 --> 0:04:15.070 looks like. I don't like this example, but we'll just 0:04:15.070 --> 0:04:18.950 go with it and essentially like, what are all the pieces? 0:04:18.950 --> 0:04:20.590 What does it mean to have a good one? What 0:04:20.590 --> 0:04:22.350 does it mean to have a pretty one? What do 0:04:22.350 --> 0:04:24.070 those even mean? How do you make that testable? How 0:04:24.070 --> 0:04:26.910 do you make that something that the AI that your 0:04:26.960 --> 0:04:30.120 AI harness can actually hill climb on to achieve a 0:04:30.120 --> 0:04:33.160 positive result, right? And this is where this whole concept 0:04:33.160 --> 0:04:35.800 of like bitter pill engineering comes into play. There's multiple 0:04:35.800 --> 0:04:38.240 things that are built into life OS that are oriented 0:04:38.240 --> 0:04:42.360 around rethinking what a harness is. So one of them 0:04:42.360 --> 0:04:45.400 is the concept of ideal state. We're trying to transition 0:04:45.400 --> 0:04:47.400 from current state to ideal state. In this case, the 0:04:47.400 --> 0:04:49.560 current state is I don't have a pretty butterfly wing. 0:04:49.960 --> 0:04:52.800 The ideal state is I do have a pretty butterfly wing. 0:04:53.040 --> 0:04:55.720 So you're trying to, you know, you the AI or 0:04:55.720 --> 0:04:58.600 trying to transition me from current to ideal state, right? 0:04:58.640 --> 0:05:00.960 That's the same thing. If you're building some complex application, 0:05:00.960 --> 0:05:02.480 you're trying to start a new business and you need 0:05:02.480 --> 0:05:05.360 all the paperwork. It's all about current to ideal state. 0:05:05.440 --> 0:05:08.080 The second piece of that is the document itself. That 0:05:08.080 --> 0:05:10.240 has to do with pursuing the ideal state. When we 0:05:10.240 --> 0:05:12.880 build something inside of life OS, we're actually building an 0:05:12.880 --> 0:05:16.480 ideal state artifact. That ideal state artifact is a generalized 0:05:16.480 --> 0:05:19.200 version of all the stuff you've heard about before with, 0:05:19.600 --> 0:05:23.040 you know, specs and prds and plans and, you know, 0:05:23.080 --> 0:05:25.520 prompts and all of that. This is a single unified 0:05:25.520 --> 0:05:29.490 document that captures the kind of the journal say like 0:05:29.530 --> 0:05:31.810 the what the person wants, what they say they want 0:05:31.810 --> 0:05:35.290 that's captured changes to that are captured in like a changelog. 0:05:35.290 --> 0:05:38.090 It's broken out into features and like approaches and stuff 0:05:38.090 --> 0:05:40.170 like that, which the AI comes up with for you. 0:05:40.650 --> 0:05:42.650 And then if you make adjustments to that, it's modified. 0:05:42.650 --> 0:05:46.890 There's an HTML version, there's a markdown version. And basically 0:05:46.890 --> 0:05:49.970 this is the format that we use to hill climb. 0:05:50.130 --> 0:05:52.450 So we are giving a goal to our system to 0:05:52.490 --> 0:05:56.210 hill climb using this ideal state artifact, which is in 0:05:56.210 --> 0:05:59.970 my opinion, like the future of like a PRD. Think 0:05:59.970 --> 0:06:02.650 of it as a universal PRD for any task. And 0:06:02.650 --> 0:06:04.970 that is fundamentally what life OS is based on, right? 0:06:05.010 --> 0:06:08.490 Is this concept of pursuing the ideal. So really what 0:06:08.490 --> 0:06:11.930 we're talking about here is a system that can universally 0:06:11.970 --> 0:06:16.010 pursue ideal for you, right? We are not code generators. 0:06:16.010 --> 0:06:19.450 We are not app builders, right? Not fundamentally. It's a 0:06:19.450 --> 0:06:21.570 task that we are doing. Why are we doing that? 0:06:21.610 --> 0:06:23.210 Why are we building apps? How much money do we 0:06:23.210 --> 0:06:25.010 need to make? Are we trying to transition out of 0:06:25.010 --> 0:06:28.330 our corporate job to have, you know, work for ourselves? Well, 0:06:28.330 --> 0:06:31.620 then that's the goal, right? So how much monthly recurring 0:06:31.620 --> 0:06:33.860 revenue are we making? Are we doing a good job? 0:06:33.900 --> 0:06:36.260 Are people enjoying this? These are questions that we want 0:06:36.300 --> 0:06:39.140 to ask, and I shouldn't have to say and set 0:06:39.180 --> 0:06:41.180 up every single question that I ask to the AI, 0:06:41.540 --> 0:06:43.260 or set up every single app that I want to 0:06:43.300 --> 0:06:46.580 build with the AI to describe the stack and describe 0:06:46.580 --> 0:06:48.740 the goal that I have and describe all of that. No. 0:06:48.980 --> 0:06:51.500 That is the purpose of the harness. Have all of 0:06:51.540 --> 0:06:54.460 that already built in? It's all in your context. It's 0:06:54.460 --> 0:06:57.300 in your various context files, it's in your various tools, 0:06:57.300 --> 0:06:59.580 it's in your hooks, it's in your prompts, it's in everything. 0:06:59.580 --> 0:07:02.580 It's already pre-built. It's already pre-baked into your harness itself. 0:07:02.860 --> 0:07:05.780 So you could say something like, hey, look, is this 0:07:05.780 --> 0:07:07.700 a good system? I heard there's a new memory system 0:07:07.740 --> 0:07:10.380 out for cloud code. Is this as good as ours? 0:07:10.740 --> 0:07:12.420 I literally just do this. In fact, I'm going to 0:07:12.420 --> 0:07:14.300 show you. I'm just going to go over here to 0:07:14.340 --> 0:07:17.940 my system. Hey, what do we think about this thing? 0:07:18.340 --> 0:07:20.900 This is Gary Tan, a friend of mine, cool guy. 0:07:21.380 --> 0:07:23.220 And basically we talk about a bunch of harness stuff, 0:07:23.220 --> 0:07:25.900 but he has like this Google brain thing that 24,000 stars. 0:07:25.940 --> 0:07:27.860 A lot of people are using this. It's like a 0:07:28.540 --> 0:07:31.340 got recipes. It's basically a whole harness, right? With memory 0:07:31.340 --> 0:07:33.670 and stuff like that. So my question is I want 0:07:33.670 --> 0:07:35.390 to look at this thing and say, hey, how good 0:07:35.390 --> 0:07:38.030 is this? Like, can we use it or whatever? Now, 0:07:38.150 --> 0:07:41.230 if I have a generic harness with not much in it, right. 0:07:41.270 --> 0:07:44.430 Something that's not like life OS. Well, I'm going to 0:07:44.430 --> 0:07:47.070 have to explain. Hey there. You know, there are memory systems. 0:07:47.070 --> 0:07:49.230 We have a memory system. This is a memory system. 0:07:49.270 --> 0:07:50.990 I'm trying to figure out how good this is compared 0:07:50.990 --> 0:07:52.750 to ours. I want you to look across all these 0:07:52.750 --> 0:07:55.510 different criteria, blah, blah, blah. Right. That's a whole bunch 0:07:55.510 --> 0:07:59.710 of stuff that is intent engineering, right? That is prompt engineering. 0:07:59.710 --> 0:08:02.870 It is context engineering. It's like all these names that 0:08:02.870 --> 0:08:04.630 we've been using. It's all the same thing. It's us 0:08:04.630 --> 0:08:07.950 trying to get our ideal result right. We have a 0:08:07.950 --> 0:08:10.710 feeling in our brain of what we mean when we 0:08:10.710 --> 0:08:13.110 give this to our AI. What we really mean is 0:08:13.830 --> 0:08:16.190 how good is this? Should we use it? But in 0:08:16.190 --> 0:08:19.030 order to give a proper prompt for that, you could 0:08:19.030 --> 0:08:21.590 actually spend half an hour or an hour or two 0:08:21.590 --> 0:08:24.070 hours writing a great prompt and you would get better results. 0:08:24.350 --> 0:08:27.390 The goal of a harness, in my opinion, is to 0:08:27.430 --> 0:08:29.870 be able to do that automatically with as few words 0:08:29.870 --> 0:08:31.790 as possible. Right? All right. So what I'm going to 0:08:31.790 --> 0:08:34.430 do is I'm just going to grab this URL here. Okay, 0:08:34.470 --> 0:08:36.080 so I got that copied, and I'm going to come 0:08:36.080 --> 0:08:37.679 over here and I'm going to open a new session 0:08:37.720 --> 0:08:40.240 right here ready to go. Kai says hello and we're 0:08:40.240 --> 0:08:42.880 going to say, hey, take a look at this and 0:08:42.880 --> 0:08:44.400 see if we should use any part of it. 0:08:44.800 --> 0:08:45.960 Evaluating G brain repo. 0:08:46.240 --> 0:08:48.559 All right. So Kai is going to head out and 0:08:48.559 --> 0:08:51.560 take a look at this thing. And what's extraordinary about 0:08:51.560 --> 0:08:54.920 what Kai is about to do is what did I say? 0:08:54.920 --> 0:08:56.440 I said, hey, take a look at this and see 0:08:56.440 --> 0:08:58.400 if we should use any part of it. I didn't 0:08:58.400 --> 0:09:01.360 explain myself. I didn't write a prompt that took me 0:09:01.360 --> 0:09:05.200 two hours. Kai already knows when it sees the project, 0:09:05.200 --> 0:09:07.920 it knows okay about itself. Look at these context files. 0:09:07.920 --> 0:09:12.600 I've included my skills, my life OS system prompt projects, 0:09:12.600 --> 0:09:16.960 my identity, Kai's identity architecture summary. That's our architecture, life 0:09:16.960 --> 0:09:21.600 OS all that stuff all unified together so that Kai 0:09:21.640 --> 0:09:24.880 understands what I mean when I say very simple sentences. 0:09:25.360 --> 0:09:30.840 What we just basically did was we cheated. We basically 0:09:30.840 --> 0:09:32.280 cheated and injected. 0:09:32.280 --> 0:09:36.080 Skip the DB, steal gap analysis and hybrid retrieval, both testable. 0:09:36.120 --> 0:09:37.530 Want me to prototype the gap layer. 0:09:37.850 --> 0:09:39.970 Look at that. Look at that. Look at how much 0:09:40.010 --> 0:09:43.050 it looked at that quickly. Okay. And this is all 0:09:43.090 --> 0:09:45.170 because it has tons of context built in. I'm already 0:09:45.170 --> 0:09:48.050 at 13%. And people will say, well, if you have 0:09:48.050 --> 0:09:51.730 a harness, you're doing all this context injection, you're wasting tokens. No, 0:09:51.730 --> 0:09:54.810 you're spending tokens. You're not wasting tokens. You're doing quite 0:09:54.810 --> 0:09:58.010 the opposite. And tons of people who use Pi OS, 0:09:58.290 --> 0:10:00.690 the new name of it basically will tell you, yes, 0:10:00.690 --> 0:10:02.410 sometimes it takes a little bit longer. That didn't take 0:10:02.410 --> 0:10:05.210 long at all. That took one minute, right? 20s. But 0:10:05.250 --> 0:10:08.090 it can take a little bit longer. But you're saving 0:10:08.250 --> 0:10:12.250 spending five times more time in tokens, doing rework. I 0:10:12.250 --> 0:10:14.850 don't have to do rework very often here, and when 0:10:14.850 --> 0:10:16.530 I do, I just rebuild that into the harness so 0:10:16.530 --> 0:10:18.130 I don't ever have to do it in the future. 0:10:18.890 --> 0:10:21.530 So I really encourage you to think about a harness 0:10:21.570 --> 0:10:25.290 in this way as a much larger, zoomed out version 0:10:25.290 --> 0:10:28.370 of life management, which is why we just renamed the 0:10:28.410 --> 0:10:31.850 Pi basically to life OS, because this is about managing 0:10:31.850 --> 0:10:33.330 what you are trying to do in your life and 0:10:33.330 --> 0:10:35.730 what you're trying to do in your work overall, what 0:10:35.730 --> 0:10:38.170 you're trying to accomplish, what is your ideal state so 0:10:38.170 --> 0:10:41.179 you can get your AI harness to transition from current 0:10:41.180 --> 0:10:44.220 state to ideal state. That is the whole purpose. And 0:10:44.220 --> 0:10:47.060 inside of that, contained inside of that is this concept 0:10:47.060 --> 0:10:51.500 of intent engineering. Okay. It's intent engineering, sitting on top 0:10:51.500 --> 0:10:55.260 of context engineering, sitting on top of prompt engineering. And 0:10:55.260 --> 0:10:58.820 ultimately it comes down to the clearest possible thinking, right? 0:10:58.860 --> 0:11:01.660 The clearest possible articulation of what you're trying to accomplish, 0:11:01.660 --> 0:11:03.660 which is why we have the algorithm, which is why 0:11:03.660 --> 0:11:07.300 we have the Issa document, ideal state artifact document. We 0:11:07.300 --> 0:11:10.260 are hill climbing towards what we're trying to accomplish, whether 0:11:10.260 --> 0:11:13.220 it's drawing a butterfly wing or building a brand new 0:11:13.220 --> 0:11:16.700 application or like improving our relationships, whatever it is, think 0:11:16.700 --> 0:11:19.660 at that larger scale. Think of your AI harness as 0:11:20.060 --> 0:11:22.740 life management, essentially, right? And it doesn't matter what platform 0:11:22.740 --> 0:11:25.060 you use, doesn't matter if you use our open source 0:11:25.100 --> 0:11:26.460 thing or you're going to use some new thing that's 0:11:26.460 --> 0:11:29.340 going to come out from anthropic or OpenAI, it doesn't matter. 0:11:29.340 --> 0:11:31.380 So I just want to prompt you to think about 0:11:31.700 --> 0:11:36.220 harnesses in this much larger way, not AI harnesses, but 0:11:36.500 --> 0:11:41.460 life OS slash life management slash intent engineering. And we'll 0:11:41.460 --> 0:11:42.340 see you in the next one.