WEBVTT - Reviewing RSA 2025 with Jason Haddix 0:00:18.827 --> 0:00:21.947 So here we are at the house. It is after RSA. 0:00:22.107 --> 0:00:25.387 It's been a crazy week and a half. Um, what's 0:00:25.387 --> 0:00:29.947 on your mind? What's the takeaway? We probably have several. Yeah. 0:00:30.267 --> 0:00:33.107 Yeah. I mean, uh, I feel like this week there 0:00:33.107 --> 0:00:35.507 was a lot of fun, but also, you and I 0:00:35.507 --> 0:00:37.907 had the opportunity to go to quite a few events, 0:00:38.507 --> 0:00:41.387 and there was less FUD at those events, I feel like. 0:00:41.427 --> 0:00:44.907 And some real innovation that we saw, uh, which surprisingly 0:00:44.906 --> 0:00:47.266 was off the RSA show floor, I think. Right. 0:00:47.307 --> 0:00:49.547 Yeah. And it seems like that's like the big lesson 0:00:49.547 --> 0:00:51.346 we learned is like, how do we figure out how 0:00:51.346 --> 0:00:52.707 to do more of that next year? 0:00:52.747 --> 0:00:53.787 Yeah. Exactly. 0:00:53.787 --> 0:00:57.547 Yeah. And and less of like the I feel like, 0:00:58.187 --> 0:01:00.067 how long have we been doing this? Like 15 years 0:01:00.067 --> 0:01:03.627 or something. Yeah, yeah. And, um, at first it's like, 0:01:03.627 --> 0:01:05.987 how do you get into the we're not going to 0:01:05.987 --> 0:01:08.147 name vendors, but how do you get into the big 0:01:08.147 --> 0:01:11.947 vendor conference that has, like the best music and the best. 0:01:12.187 --> 0:01:13.066 Big parties and stuff? 0:01:13.067 --> 0:01:14.107 Yeah, the big party. 0:01:14.267 --> 0:01:14.786 Yeah. 0:01:15.307 --> 0:01:18.417 And now we're just like, how can we see our 0:01:18.417 --> 0:01:23.257 friends go to like, an event that's like smaller? Um, 0:01:24.177 --> 0:01:27.017 I don't know. It's not necessarily that it's harder to 0:01:27.017 --> 0:01:29.017 get into, but it seems like that's where the better 0:01:29.017 --> 0:01:30.177 conversations happen. 0:01:30.737 --> 0:01:34.057 Yeah. I mean, well, I mean, RSA is like everybody 0:01:34.217 --> 0:01:37.617 trying to grab like a mic, right? And scream as 0:01:37.617 --> 0:01:40.737 loud as they can and get people to listen to whatever. 0:01:40.777 --> 0:01:44.857 And the parties are a mechanism for that. The advertising 0:01:44.857 --> 0:01:46.737 on the show floor of the booth and everything. But 0:01:46.737 --> 0:01:49.857 I think that more and more as we go on, 0:01:49.897 --> 0:01:54.097 there's satellite events which aren't at Moscone, and they're usually 0:01:54.097 --> 0:01:58.017 held at like companies offices if they have the space or, um, 0:01:58.057 --> 0:02:01.217 or like an Airbnb or like some kind of rental thing. 0:02:01.617 --> 0:02:05.297 And uh, and they're very on, you know, specific topics. 0:02:05.297 --> 0:02:08.137 They're not generalized. They're very specialized usually. And so we 0:02:08.177 --> 0:02:10.697 went to quite a few of those this week. Um, 0:02:11.017 --> 0:02:12.737 and that ended up being, at least for me, it 0:02:12.737 --> 0:02:16.527 ended up being like like the kind of jewel of 0:02:16.526 --> 0:02:18.087 the week, right? Like, I did a ton of speaking 0:02:18.087 --> 0:02:21.966 and stuff, but, um, I learned the most, and I felt, uh, 0:02:21.966 --> 0:02:24.286 I felt the most impressed and kind of, like, hopeful 0:02:24.287 --> 0:02:26.767 from from those those events. Yeah. 0:02:26.806 --> 0:02:30.406 Yeah. Yeah. So we both did a bunch of, like, 0:02:30.447 --> 0:02:33.647 talks and panels and stuff like that. Um, I went 0:02:33.647 --> 0:02:39.126 to three of yours. Um, yeah, some really, really good stuff. 0:02:39.126 --> 0:02:40.727 And that was, that was a pretty new talk that 0:02:40.727 --> 0:02:43.847 you put together about the methodology stuff. That was really cool. 0:02:43.927 --> 0:02:44.367 Yeah. 0:02:44.406 --> 0:02:48.327 Yeah. And then, um, I would say that the, the 0:02:48.367 --> 0:02:50.246 big thing was yesterday. Right? 0:02:50.487 --> 0:02:51.087 Yeah. Yeah. 0:02:51.087 --> 0:02:53.406 So that was like the funnest thing to do. 0:02:53.447 --> 0:02:56.887 Oh yeah. By far. So yesterday was the, uh, OpenAI 0:02:57.727 --> 0:03:01.647 security research conference, their first ever one. Uh, and there's 0:03:01.647 --> 0:03:05.647 about a hundred hundred people there, I would say. And, um, 0:03:06.607 --> 0:03:12.726 talking about everything from, you know, protecting AI, AI agents 0:03:12.846 --> 0:03:18.596 from attack to automating security workflows with AI. And this 0:03:18.596 --> 0:03:21.357 was like no BS, right? I mean, this is like, yeah, 0:03:21.797 --> 0:03:24.516 this is like, you know, academics talking about things that 0:03:24.517 --> 0:03:27.556 were brand new, you know, new new methodologies, new ways 0:03:27.556 --> 0:03:30.397 to train agents, new evals, new everything new. I mean, 0:03:30.436 --> 0:03:32.637 we saw some new models dropped in that in that thing. 0:03:32.637 --> 0:03:35.517 We can't really talk about them. But, um, yeah, we 0:03:35.517 --> 0:03:38.557 saw some really cool stuff. And then, uh, also just 0:03:38.597 --> 0:03:40.757 like it was kind of crazy who was in that room. 0:03:40.797 --> 0:03:42.397 And we got to sit like a couple feet away 0:03:42.397 --> 0:03:45.796 from Sam Altman and ask him questions. He did a Q&A. Um, 0:03:45.917 --> 0:03:49.717 you know, Matt Knight, the CISO of OpenAI, was answering questions. Ian, 0:03:49.757 --> 0:03:53.957 who's a good friend of ours, ours was opening questions. Yeah. Uh, so, yeah, 0:03:53.957 --> 0:03:57.117 it was really, um, you know, open in feeling and 0:03:57.117 --> 0:03:59.637 in kind of like crowd. Everybody was like, having good 0:03:59.637 --> 0:04:02.117 conversations around it. And I felt like I learned a 0:04:02.117 --> 0:04:04.077 ton from that. That content specific. 0:04:04.597 --> 0:04:07.277 Yeah. More than, like, the whole week and a half before. Yeah. 0:04:07.317 --> 0:04:09.077 Yeah. It was. Yeah. I mean, I had a really 0:04:09.077 --> 0:04:11.877 good one that was like secondary to that, which was 0:04:12.227 --> 0:04:16.307 I went to I went to Airbnb's off site, I, um, 0:04:16.627 --> 0:04:19.467 thing and uh, so they did it one day at 0:04:19.507 --> 0:04:22.587 their offices where they brought in speakers and a mutual 0:04:22.587 --> 0:04:26.267 acquaintance of ours. Keith spoke there on a panel about, um, 0:04:26.267 --> 0:04:28.947 AI and security. And that one was interesting because you 0:04:28.947 --> 0:04:31.947 get the I felt like you got at the OpenAI thing. 0:04:31.947 --> 0:04:34.947 You were at the cutting edge of research and at 0:04:34.947 --> 0:04:37.827 the Airbnb one, I felt like I was at the 0:04:37.827 --> 0:04:42.827 cutting edge of implementation from a point of view of like, businesses. Right. 0:04:42.867 --> 0:04:46.067 Because like at the academic level, at the OpenAI one, um, 0:04:46.107 --> 0:04:48.907 and even the enterprises there, the people who are talking 0:04:48.907 --> 0:04:51.387 about are at the cutting edge and they're also well 0:04:51.427 --> 0:04:56.147 funded and they're incentivized to do some really cool research. Right. 0:04:56.187 --> 0:04:59.467 At the Airbnb one, it was more companies talking about 0:04:59.467 --> 0:05:02.707 their implementation workflows. You know, how they were using AI 0:05:02.747 --> 0:05:03.027 in the. 0:05:03.147 --> 0:05:03.827 Like on the ground? 0:05:03.867 --> 0:05:06.867 Yeah, on the ground. Yeah. So it was a different view. Um, 0:05:07.107 --> 0:05:09.147 but I thought it was really cool. Like Adobe talked 0:05:09.147 --> 0:05:12.827 about like their architecture for their agent based security vulnerability 0:05:12.827 --> 0:05:15.987 management system. Um, you know, Google was there talking about 0:05:15.987 --> 0:05:18.547 some stuff, and it was just it was it was 0:05:18.547 --> 0:05:19.507 really cool. So. 0:05:19.867 --> 0:05:22.507 Yeah. So you you gave the talk at open AI 0:05:22.547 --> 0:05:25.827 as well. So just, just give it like an overview 0:05:25.827 --> 0:05:28.547 of like the talk like the methodology and stuff. 0:05:28.707 --> 0:05:32.227 Yeah. So I do this class called uh attacking AI 0:05:32.267 --> 0:05:34.947 which you've been to. And um, the whole class is 0:05:34.947 --> 0:05:39.347 basically it's my methodology for AI Pentesting. And when I 0:05:39.347 --> 0:05:42.467 say I pentesting, people are like, oh, you mean AI 0:05:42.507 --> 0:05:45.827 red teaming? And I actually don't. So what I find 0:05:45.867 --> 0:05:49.827 at least visiting and talking to other experts, is that 0:05:49.867 --> 0:05:52.027 AI red teaming has been around for a long time, 0:05:52.027 --> 0:05:55.187 and they have cemented that term. And that is usually 0:05:55.187 --> 0:05:58.467 about attacking the model, right? Like the model in place 0:05:58.787 --> 0:06:01.467 can speak harm, can speak bias. It can tell you 0:06:01.467 --> 0:06:04.507 how to cook meth, you know, and that is that 0:06:04.507 --> 0:06:06.827 is stuff that happens in, you know, this one vertical 0:06:06.827 --> 0:06:09.857 of attacking AI, which is the model, When you do 0:06:09.857 --> 0:06:11.897 an AI pen test, you not only have to assess 0:06:11.897 --> 0:06:13.457 the model and what it will say to your users, 0:06:13.457 --> 0:06:15.817 but you have to assess the implementation of the model. 0:06:15.857 --> 0:06:18.697 You have to assess, um, everything else that's hooked up 0:06:18.697 --> 0:06:21.577 to like all of the DevSecOps tools that do logging 0:06:21.577 --> 0:06:25.217 and observability and all this other stuff around it. Um, 0:06:25.257 --> 0:06:28.857 and so it ends up being, uh, a hybrid web test, 0:06:28.857 --> 0:06:32.017 an API test, and then also AI red teaming, and 0:06:32.017 --> 0:06:34.457 then also now you have in order to get these 0:06:34.457 --> 0:06:37.657 systems to do things for you, like agents that are 0:06:37.657 --> 0:06:39.856 hooked up to tools and APIs and stuff like that, 0:06:39.857 --> 0:06:42.817 you also have to get them to, um, accept prompt 0:06:42.817 --> 0:06:47.177 injection through security gates, which is like classifiers and guardrails. 0:06:47.177 --> 0:06:49.577 So the talk was basically our methodology at a high 0:06:49.577 --> 0:06:52.857 level on the whole pen testing process, um, which has 0:06:52.857 --> 0:06:55.856 seven steps. And then the second part was our prompt 0:06:55.857 --> 0:07:00.177 injection taxonomy, which is really like a taxonomy to sneak 0:07:00.737 --> 0:07:04.577 attacks through classifiers and guardrails. Um, and so we open 0:07:04.577 --> 0:07:07.297 sourced a tool about a month ago. It's called the 0:07:07.297 --> 0:07:09.967 Arcanum Prompt Injection taxonomy. And it goes through all of 0:07:09.967 --> 0:07:12.567 these tips and tricks to do that. And so we 0:07:12.567 --> 0:07:15.687 split it up into four levels. One is um, or 0:07:15.687 --> 0:07:18.407 we call them for prompt injection primitives. One is um, 0:07:18.847 --> 0:07:21.567 your intent, like what you're trying to do to the 0:07:21.567 --> 0:07:23.487 AI system, is it, you know, get it to do 0:07:23.487 --> 0:07:25.247 some of those red teaming things like speak harm and 0:07:25.247 --> 0:07:27.807 bias or is it like get it to leak its 0:07:27.807 --> 0:07:29.687 system prompt, or do you want to jailbreak it entirely, 0:07:29.687 --> 0:07:31.487 or do you want to do something completely different? Right. 0:07:31.847 --> 0:07:35.727 And then we have three other sections. We have, uh, techniques. Um, 0:07:35.727 --> 0:07:38.167 is one of our primitives and techniques is how you 0:07:38.167 --> 0:07:41.207 execute the attack. There's a framing. You can do narrative injection, 0:07:41.207 --> 0:07:43.327 you can do all this kind of crazy stuff. And 0:07:43.327 --> 0:07:47.047 then we have evasions, which are, um, the idea of like, 0:07:47.047 --> 0:07:48.847 it kind of feels like WAF bypass, right? Where we 0:07:48.847 --> 0:07:50.847 do a lot of tricky encoding. But there's even more 0:07:50.847 --> 0:07:53.887 in the prompt injection world than WAF bypass. And these 0:07:53.887 --> 0:07:57.287 get you past the security products like, um, classifiers and guardrails. 0:07:57.287 --> 0:07:59.487 And so we talked about we talked about all of 0:07:59.487 --> 0:08:01.447 those and utilities we had made. And that was the talk. 0:08:01.487 --> 0:08:04.727 So yeah, it's it's really good. And I've been watching 0:08:04.727 --> 0:08:08.717 all these and It's absolutely the best. Thanks, man. Yeah, 0:08:09.117 --> 0:08:12.117 and it's like it's presented really well as well. Um, 0:08:12.117 --> 0:08:13.237 and by the way, I'm going to talk to the 0:08:13.237 --> 0:08:16.037 camera here for a second. So on the way up here, 0:08:16.557 --> 0:08:19.197 we were like, we're just going to have a conversation. Yeah. Yeah. 0:08:19.237 --> 0:08:22.237 You know what's hilarious? We sat down and became podcasters. 0:08:22.277 --> 0:08:22.957 I know right. 0:08:24.997 --> 0:08:27.917 We're like. We're like, all right, let's talk about the content. 0:08:27.957 --> 0:08:29.957 And like, so I don't know how we break out 0:08:29.997 --> 0:08:30.837 of that because it's just. 0:08:31.317 --> 0:08:32.916 I think we can I think we can. 0:08:32.957 --> 0:08:37.756 Well, we'll just let it run and hopefully we, uh, like, relax. Um, yeah. 0:08:38.837 --> 0:08:42.277 So what else? Um, what else should we cover? 0:08:42.797 --> 0:08:44.837 So I had a whole bunch of notes on my phone, 0:08:44.837 --> 0:08:46.916 and then my phone battery died. In fact, we were 0:08:46.957 --> 0:08:48.797 we were trying to set up with, uh, my new 0:08:48.797 --> 0:08:51.797 DJI setup, which I like, which Julia got me for Christmas. 0:08:51.837 --> 0:08:54.077 A friend of ours, Ron Foster, recommended the whole stack, 0:08:54.117 --> 0:08:56.997 like the DJI camera and DJI wireless mic, so I 0:08:56.997 --> 0:08:59.756 could go to like conferences and have quick interviews with people. 0:08:59.756 --> 0:09:02.477 And of course one couldn't set it up right. And 0:09:02.477 --> 0:09:05.506 then two, um, phone ran out of battery. 0:09:05.506 --> 0:09:07.426 Almost has it set up. Phone dies. 0:09:07.707 --> 0:09:09.307 Almost haven't set up. The thing dies. 0:09:09.307 --> 0:09:11.146 So. So we're going to come in recording in the 0:09:11.426 --> 0:09:14.146 in the command center for you. But um, but yeah. 0:09:14.146 --> 0:09:16.987 So I had some notes on my phone and, uh, 0:09:16.987 --> 0:09:21.267 the notes are just like how at least I felt 0:09:21.266 --> 0:09:22.747 on the floor. You got to walk the floor, right? 0:09:23.026 --> 0:09:24.027 Yeah, a little bit. Yeah. 0:09:24.026 --> 0:09:25.906 So we didn't sync up till later in the week, right. Like, 0:09:25.947 --> 0:09:27.747 we mostly had our own stuff in the beginning and 0:09:27.747 --> 0:09:30.946 then we synced up later. Um, but, uh, I don't 0:09:30.947 --> 0:09:32.307 want to note on my phone on how the floor 0:09:32.307 --> 0:09:34.546 was very funny this year. Right. Like. And also, I 0:09:34.546 --> 0:09:39.347 had these weird moments of, like, justification. It was like, um, like, 0:09:39.386 --> 0:09:41.867 everyone last year said they were going to completely automate 0:09:42.187 --> 0:09:45.826 all these security workflows. Right? And I distinctly remember, like, 0:09:45.827 --> 0:09:48.427 writing these notes last year. And then this year I 0:09:48.426 --> 0:09:51.587 walked the floor and everybody has changed their tune. It's like, oh, 0:09:51.627 --> 0:09:55.307 I assisted like, you know, power your people, scale your team, 0:09:55.347 --> 0:09:55.867 you know? And it's. 0:09:55.867 --> 0:09:56.026 Like. 0:09:56.067 --> 0:09:57.227 Turns out that was harder than we. 0:09:57.227 --> 0:09:57.627 Thought. Yeah. 0:09:57.666 --> 0:10:00.187 Turns out turns out not possible. And I think I 0:10:00.187 --> 0:10:02.307 had this conversation with you on the drive and it 0:10:02.307 --> 0:10:05.416 was like even you and I have friends who are 0:10:05.416 --> 0:10:07.457 starting companies who were trying to do that, they're trying 0:10:07.497 --> 0:10:09.937 to automate a workflow. And we saw it at the 0:10:09.937 --> 0:10:12.977 OpenAI thing, too. People are like, yeah, we're not close 0:10:12.977 --> 0:10:15.777 to full automation yet, and we're very far away from 0:10:15.776 --> 0:10:17.617 a place where the models can do full automation. Even 0:10:17.617 --> 0:10:21.817 the people in the FTC competition who are going for 0:10:21.817 --> 0:10:24.696 those cyber reasoning systems at Defcon, like they were like, yeah, 0:10:24.737 --> 0:10:28.536 like a lot of this is still algorithmic. Um, you know, 0:10:28.577 --> 0:10:31.937 tool based automation, but with like a lot of AI 0:10:31.977 --> 0:10:35.497 glue at the I think they said like the framework level. Yeah. 0:10:35.536 --> 0:10:38.456 And the, um, the framework and the organization levels are 0:10:38.457 --> 0:10:41.897 the parts where I actually like, really helps them. But, um, 0:10:42.256 --> 0:10:44.217 hearing that from or like seeing that on the floor 0:10:44.217 --> 0:10:46.256 and then also hearing it from the people at OpenAI, 0:10:46.296 --> 0:10:48.297 just like kind of cemented we're pretty far away from 0:10:48.296 --> 0:10:51.416 fully autonomous systems anywhere. Um, you know, I was really 0:10:51.416 --> 0:10:53.296 impressed by I don't want to name any vendors or whatever, 0:10:53.296 --> 0:10:56.297 but like, there were some demos at OpenAI thing that 0:10:56.296 --> 0:10:58.697 were pretty, pretty sick in the web testing world. 0:10:58.737 --> 0:10:59.416 Um, yeah. 0:10:59.536 --> 0:11:01.536 And so that was the one I was like, oh, okay. 0:11:01.687 --> 0:11:04.727 Like like they're getting close. And I think you and I. 0:11:04.847 --> 0:11:05.847 At least in that one domain. 0:11:05.847 --> 0:11:08.646 In that one domain. Yeah. Web testing. And then you 0:11:08.646 --> 0:11:09.967 and I were looking at some friends who were in 0:11:09.967 --> 0:11:11.367 that room, and I think they were a little crestfallen 0:11:11.367 --> 0:11:13.487 to see how far that one place had gotten with 0:11:13.487 --> 0:11:14.207 autonomous testing. 0:11:14.247 --> 0:11:17.406 They were there live learning how far their competitors were. 0:11:17.447 --> 0:11:17.727 So. 0:11:17.727 --> 0:11:20.727 Bad. They're just like, oh, I'm excited. Oh, yeah. 0:11:20.886 --> 0:11:24.247 Yeah, yeah. Um, but that was that was cool. You know, 0:11:24.286 --> 0:11:31.087 it was cool to, um, get to ask Q&A of, uh, Altman. Um, and, uh, uh, 0:11:31.166 --> 0:11:32.727 you know, so we, you know, it wasn't like we 0:11:32.727 --> 0:11:34.046 got one on one time with them, right? It was 0:11:34.046 --> 0:11:35.166 just like, we just. We got. We were. 0:11:35.166 --> 0:11:35.327 In the. 0:11:35.327 --> 0:11:37.367 Front row, though. Yeah, I snagged a seat, so I 0:11:37.367 --> 0:11:37.766 was I was. 0:11:37.766 --> 0:11:38.006 Yeah, that. 0:11:38.006 --> 0:11:39.527 Was good. I left lunch early to get a seat 0:11:39.567 --> 0:11:42.806 right in front, but, um, so. Yeah. So we're, we're 0:11:42.807 --> 0:11:46.007 in the front row. And one of the questions to, uh, 0:11:46.006 --> 0:11:48.567 Sam Altman, if you don't know that, uh, head of 0:11:48.567 --> 0:11:52.767 OpenAI CEO and, uh, they were one of the questions 0:11:52.766 --> 0:11:55.847 was like, what do you, uh, like, what is the 0:11:55.847 --> 0:11:58.006 security thing that you worry about? Right? And Sam is 0:11:58.046 --> 0:12:01.196 the CEO. He's the CEO, right. So he's not in 0:12:01.197 --> 0:12:03.917 security every day. But he's a smart dude. Yeah, yeah. And, um. 0:12:04.676 --> 0:12:08.237 And he was talking about, uh, he was like. Like 0:12:08.237 --> 0:12:11.357 he starts launching into his answer, presupposing he's like, yeah. 0:12:11.357 --> 0:12:13.917 So when we get these, like, you know, fully context 0:12:13.957 --> 0:12:17.077 aware agents, that has everything about my life, like written down, 0:12:17.276 --> 0:12:19.517 you know, and has all this information and can make 0:12:19.516 --> 0:12:23.036 these really great intuitive decisions for me and stuff like that. 0:12:23.036 --> 0:12:25.877 Like what happens when that gets hacked, your whole ethos, 0:12:25.877 --> 0:12:28.837 your whole stack of who you are as a person, 0:12:28.837 --> 0:12:30.516 what you like to use, what you like to watch, 0:12:30.516 --> 0:12:32.117 what you like to hear, what you like to eat. 0:12:32.156 --> 0:12:34.757 And so many things can be intuited from that as 0:12:34.756 --> 0:12:38.477 well about you as a person. What happens when that leaks? Right. 0:12:38.516 --> 0:12:41.677 And I'm just there and I'm like, hitting you. I'm like, Dan, Dan. 0:12:41.756 --> 0:12:42.397 Grabs my leg. 0:12:42.396 --> 0:12:44.276 Yeah, yeah. Like grabbing his leg because. Because you've been 0:12:44.276 --> 0:12:46.276 talking about this for like ten years, right? Like, I 0:12:46.276 --> 0:12:50.156 remember the first few blogs about the first iterations of, like, 0:12:50.156 --> 0:12:53.317 kind of unified context or context about your life, right? 0:12:53.357 --> 0:12:56.316 And then you do it with your tlos files as 0:12:56.317 --> 0:12:59.987 a person, and you also do it for companies to 0:13:00.067 --> 0:13:03.026 understand the ethos of the company. And Sam just launched 0:13:03.026 --> 0:13:05.546 right into that. Yeah, and it was like presupposed. And 0:13:05.546 --> 0:13:07.867 I'm like, you motherfucker. Like. 0:13:08.187 --> 0:13:10.827 Yeah, I was so excited. I'm like, can I start talking? 0:13:10.906 --> 0:13:12.787 Like, yeah, yeah that's great. Yeah, it was great. But 0:13:12.786 --> 0:13:15.546 he also he said the the risk is that, you know, 0:13:15.587 --> 0:13:18.026 I guess in your analogy, the teller's file gets leaked 0:13:18.026 --> 0:13:21.826 right somehow. And then people know, like what you're about 0:13:21.867 --> 0:13:27.987 and how they can specifically adversarially market to you, influence you. Yeah. 0:13:28.026 --> 0:13:31.066 You know, and if you're really open with your personal 0:13:31.067 --> 0:13:34.227 assistant or, or your, um, you know, whatever ends up 0:13:34.426 --> 0:13:37.107 collecting that information, you're really open to that, like some 0:13:37.107 --> 0:13:40.227 of your, like, idiosyncrasies, some of your, like psychological stuff. 0:13:40.266 --> 0:13:44.627 Well, it's just like, you know, I think the most powerful, 0:13:45.026 --> 0:13:48.027 powerful version of this is like, you have your, um, 0:13:48.146 --> 0:13:51.266 your journal in there. Yeah. And you're just constantly complaining 0:13:51.266 --> 0:13:55.066 about your mother in law, right? Yeah. And that gets hacked. 0:13:55.067 --> 0:13:55.987 And so now my mother. 0:13:55.987 --> 0:13:56.667 In law's great, by. 0:13:56.666 --> 0:14:01.027 The way. And now now that becomes a, um, it 0:14:01.026 --> 0:14:04.947 becomes like a an extortion email. Yeah, right. Someone could 0:14:04.947 --> 0:14:06.186 just be like, do you want me to send this 0:14:06.187 --> 0:14:08.627 to your mother in law or just send me, you know, 0:14:08.666 --> 0:14:11.546 $20 or whatever? And it's like, that's worth it. I 0:14:11.546 --> 0:14:13.067 don't want to. I don't want to have that fight 0:14:13.107 --> 0:14:13.547 at dinner. 0:14:13.587 --> 0:14:14.906 Yeah, that's. I mean, $20. 0:14:14.906 --> 0:14:15.786 Seems like a good. 0:14:15.987 --> 0:14:19.467 Ransomware price, right? Like ransomware operators are listening. Like, I'll 0:14:19.467 --> 0:14:20.747 pay 28 bucks. That's cool. 0:14:21.267 --> 0:14:23.347 Yeah, yeah. But but to your point, it's like your 0:14:23.347 --> 0:14:26.947 entire personality. It's like your entire soul. So that's. Yeah, 0:14:26.947 --> 0:14:29.147 that's a lot of content to lose. 0:14:29.467 --> 0:14:30.707 Yeah. One of the things we're talking about in the 0:14:30.707 --> 0:14:34.227 car was how we saw so many. And it sucks. 0:14:34.227 --> 0:14:37.706 But like friends, colleagues working for these places, hanging their 0:14:37.707 --> 0:14:40.987 hats on these AI features or even companies that are 0:14:40.987 --> 0:14:45.507 completely based around AI, and there's no there's no moat 0:14:45.507 --> 0:14:47.946 for them, like it is going to be disrupted either 0:14:47.947 --> 0:14:50.707 by because we then we went to the OpenAI thing, 0:14:50.827 --> 0:14:51.907 saw what they're doing. 0:14:51.947 --> 0:14:53.747 Yeah. And destroying moats. 0:14:53.787 --> 0:14:56.657 Yeah. Destroying moats basically. Right. Like whole companies are going 0:14:56.657 --> 0:14:58.217 to go down because they had this premise of, oh, 0:14:58.217 --> 0:15:00.696 we'll use AI to do this thing. Now it's just 0:15:00.697 --> 0:15:02.857 going to become part of the model, or they're going 0:15:02.857 --> 0:15:05.017 to be just trampled by one of the big data 0:15:05.057 --> 0:15:07.537 aggregators who already has all the data to make the 0:15:07.537 --> 0:15:10.897 problem set easy to execute, I think, is how I 0:15:10.897 --> 0:15:11.537 think of it. 0:15:11.577 --> 0:15:15.097 Yeah. Yeah, absolutely. So, so Jason talked about what he 0:15:15.097 --> 0:15:19.137 was presenting. Um, so what I was presenting is like 0:15:19.137 --> 0:15:23.337 this unified entity context. Yeah. Yeah. So it's like, if 0:15:23.337 --> 0:15:25.336 it's an individual, you just get all that stuff that 0:15:25.337 --> 0:15:28.657 you already talked about. And then if it's a company 0:15:28.657 --> 0:15:31.377 you get all the way from the company goals all 0:15:31.377 --> 0:15:34.417 the way to the security goals. But all the HR stuff, 0:15:34.457 --> 0:15:37.977 like everything all into one bucket. And then from there 0:15:37.977 --> 0:15:41.816 you just ask questions. Yeah. So if you ask HR questions, 0:15:42.137 --> 0:15:43.377 is that HR software? 0:15:44.417 --> 0:15:46.857 I think it only works with the the data right. 0:15:46.897 --> 0:15:48.377 Like yeah data about the people. Yeah. 0:15:48.417 --> 0:15:50.977 Yeah. It's like if you have HR data in there, 0:15:50.977 --> 0:15:53.936 you have security data in there. And you ask security 0:15:53.937 --> 0:15:57.927 questions and HR questions. You have HR software. So I 0:15:57.927 --> 0:16:00.047 feel like software verticals just go away. 0:16:00.527 --> 0:16:02.207 I mean, I don't think they completely go away, but 0:16:02.207 --> 0:16:05.527 I think yeah, a lot of them are are majorly disrupted. Right. 0:16:05.567 --> 0:16:08.686 It's like and again, it'll be the big companies that 0:16:08.687 --> 0:16:11.247 do this first because they already have that infrastructure like 0:16:11.407 --> 0:16:14.207 Microsoft has the whole graph API about, you know, corporate 0:16:14.207 --> 0:16:18.367 user data. They have security data. They have God. God 0:16:18.367 --> 0:16:20.887 knows what other data. Right. So they're poised to move 0:16:20.887 --> 0:16:22.887 quickly in some of these places that you and I 0:16:22.887 --> 0:16:25.287 play in. And it's like they will get there first 0:16:25.287 --> 0:16:28.247 because they have the the ability to grab everything. And yeah, 0:16:28.247 --> 0:16:30.887 I think I think that for your talk, I mean, 0:16:30.887 --> 0:16:34.407 you talked about like as a meta thing, you know, 0:16:34.407 --> 0:16:36.807 like everybody is focused on, oh, we can build these 0:16:36.807 --> 0:16:39.527 agents or, you know, I questions whether you're just using 0:16:39.527 --> 0:16:41.447 an API, you're actually using a planning agent or whatever. 0:16:41.447 --> 0:16:44.607 It doesn't matter. Like whatever architecture you've chosen, use AI. 0:16:44.847 --> 0:16:47.887 And that's the important part. Right. Like prompting it, getting, 0:16:47.927 --> 0:16:50.767 you know, you know, rags set up like all this stuff. 0:16:50.967 --> 0:16:53.757 And you had that one slide where it was like, okay, 0:16:53.797 --> 0:16:55.877 so the agents are the big boxes and we really 0:16:55.877 --> 0:16:57.637 care about them right now, right? And then they're attached 0:16:57.637 --> 0:16:59.117 to data sets and they're kind of small. And then 0:16:59.117 --> 0:17:00.437 you have the other slide. And then it's like, no, 0:17:00.437 --> 0:17:03.476 actually the key piece is the data in the middle. 0:17:03.517 --> 0:17:05.397 And the thing that validated it for me was you 0:17:05.397 --> 0:17:07.477 didn't you were not the OpenAI one. But next year 0:17:07.517 --> 0:17:10.037 we got to go and present. But on this topic too, 0:17:10.077 --> 0:17:10.677 by the way. 0:17:10.717 --> 0:17:11.677 Oh, the Airbnb one. 0:17:11.677 --> 0:17:14.597 Yeah. Airbnb one. Airbnb one. Yeah. So, uh, the guy, 0:17:14.637 --> 0:17:16.317 one of the guys from Microsoft, I can't remember his name, 0:17:16.317 --> 0:17:17.717 but he was talking about kind of the same idea 0:17:17.757 --> 0:17:19.476 as he calls it, like Golden Data Lake or something 0:17:19.477 --> 0:17:21.837 like that. But it was the exact same thing. He's like, hey, 0:17:22.077 --> 0:17:23.677 you know, like the models are going to get so 0:17:23.677 --> 0:17:27.437 good that they are general software products themselves. Right? And 0:17:27.476 --> 0:17:30.117 so it's that's not the thing you should be focusing 0:17:30.117 --> 0:17:34.157 your development on. Your development or either systems architecture revamp 0:17:34.157 --> 0:17:37.037 or whatever should be on collecting the contextual data. That's 0:17:37.037 --> 0:17:38.917 going to help you answer the questions. Right. And so 0:17:38.917 --> 0:17:40.996 in your slide you had like an image where the 0:17:40.997 --> 0:17:43.476 agents were really big and they were the important question. 0:17:43.476 --> 0:17:45.517 And then then it's like the next slide is like 0:17:45.557 --> 0:17:47.196 actually the most important thing is the model. And the 0:17:47.196 --> 0:17:49.276 agents just live on the side a small little thing. 0:17:49.317 --> 0:17:54.427 So yeah yeah yeah Yeah. So so yeah. So what 0:17:54.427 --> 0:17:58.946 I have was like cybersecurity with AI around it versus 0:17:58.986 --> 0:18:03.427 AI in the middle and then cybersecurity and HR and productivity. Yeah, 0:18:03.466 --> 0:18:06.307 the software verticals are kind of like rotating around it. 0:18:06.347 --> 0:18:06.827 Yeah. 0:18:06.867 --> 0:18:10.507 Yeah yeah. Because yeah I think you just you just 0:18:10.507 --> 0:18:13.946 collect all that data and ask the questions and answers it. 0:18:14.267 --> 0:18:15.867 Yeah. And I think, you know, on my phone I 0:18:15.867 --> 0:18:19.787 had some notes too, that um, a common theme I saw. 0:18:19.827 --> 0:18:21.507 We're going to talk a lot about AI. Sorry. I mean, 0:18:21.507 --> 0:18:22.827 like a lot of, you know, I know people want 0:18:22.827 --> 0:18:25.267 to hear about security stuff, too, but, um, a lot 0:18:25.267 --> 0:18:30.786 of this stuff in architecture, of these systems, it was 0:18:30.787 --> 0:18:33.587 it had to be more compartmentalized than, I think even 0:18:33.587 --> 0:18:36.867 some of the architectures I was thinking of meaning that, um, 0:18:36.907 --> 0:18:38.987 where I thought I could ask a bunch of questions 0:18:38.986 --> 0:18:41.667 of an, you know, an AI model and stuff it 0:18:41.667 --> 0:18:46.346 all into one masterful system prompt or something, or user prompt. Um, actually, 0:18:46.787 --> 0:18:50.657 successful implementations are asking micro questions, kind of like microservices, right? 0:18:50.696 --> 0:18:53.936 We are asking one question of the same data of 0:18:53.976 --> 0:18:56.576 a data set. Wherever you take it in from the 0:18:56.577 --> 0:19:00.657 user or from context somewhere else. And then that's one agent, right? 0:19:00.696 --> 0:19:02.777 It's just a one question, one agent that's really good 0:19:02.777 --> 0:19:05.696 at doing that one thing. And then you have tens 0:19:05.736 --> 0:19:09.097 or hundreds of those because, you know, at least from 0:19:09.097 --> 0:19:12.057 the people I was talking to, they're like, you know, like, uh, 0:19:12.097 --> 0:19:14.057 don't try to stuff it all into, you know, like 0:19:14.097 --> 0:19:18.536 one process because it can confuse, you know, different models 0:19:18.537 --> 0:19:20.216 and stuff like that. So some of the ones I 0:19:20.216 --> 0:19:22.057 saw had up to like ten agents to like action 0:19:22.057 --> 0:19:25.857 one workflow, right? Asking individual questions and then stitching together 0:19:25.857 --> 0:19:26.696 the output of that. 0:19:26.736 --> 0:19:30.897 So interesting. Yeah. Yeah I, I don't know I, I 0:19:31.177 --> 0:19:36.017 feel slightly different there. I feel like all the data 0:19:36.017 --> 0:19:38.417 in one place is good. And so you just ask 0:19:38.417 --> 0:19:43.216 the questions. Um, but where I do see what you're saying. 0:19:43.617 --> 0:19:48.536 The AI, uh, x that stuff was fascinating. Oh, yeah. Oh, 0:19:48.617 --> 0:19:52.446 you know, what I really loved about that was the 0:19:52.446 --> 0:19:56.447 conversation of like, does the model matter more, or does 0:19:56.446 --> 0:19:58.807 the architecture of the system matter more? 0:19:58.847 --> 0:20:00.967 So before we go into this, because that whole panel 0:20:00.966 --> 0:20:04.247 and the two talks were awesome. Yeah, they were great. 0:20:04.247 --> 0:20:05.767 But I don't think everybody knows what the I. 0:20:05.927 --> 0:20:07.087 ZK yeah, yeah yeah. 0:20:07.127 --> 0:20:10.527 Go ahead. Okay. So the Acsc is this competition run 0:20:10.527 --> 0:20:14.006 by DARPA. Um, and the whole idea is that, uh, 0:20:14.047 --> 0:20:16.687 you build what's called a cyber reasoning system. And so 0:20:17.047 --> 0:20:20.967 there are both academic teams who are mostly CTF teams 0:20:20.966 --> 0:20:23.606 from the Defcon CTF kind of ecosystem. And then there 0:20:23.607 --> 0:20:25.887 are also companies who have come who have come to 0:20:25.927 --> 0:20:28.686 compete in this contest. And the goal is to build 0:20:28.686 --> 0:20:33.327 a system that is AI enabled that can go from, um, 0:20:33.446 --> 0:20:37.367 taking an open source project repo with a vulnerability in it, 0:20:37.407 --> 0:20:41.686 finding the vulnerability through static analysis, building an exploit, testing 0:20:41.686 --> 0:20:43.767 the exploit to see if it works in the wild, 0:20:44.087 --> 0:20:47.477 patching the exploit and keeping the service up and running. Yeah. 0:20:47.517 --> 0:20:50.637 So it's got to do both offense and defense. Um, 0:20:50.797 --> 0:20:54.797 and they get scored on multiple different facets of that. And, um, 0:20:54.797 --> 0:20:56.877 I think Dan Guido from Trail of Bits was telling 0:20:56.877 --> 0:20:59.597 us this night the top like the the they've been 0:20:59.597 --> 0:21:02.157 giving out prizes every year. So last year was the 0:21:02.157 --> 0:21:05.197 semifinal rounds and five teams made it to the semifinals. 0:21:05.397 --> 0:21:07.956 It's like three teams and two companies of which Trail 0:21:07.956 --> 0:21:10.157 of Bits is one. And my alma mater, uh, shellfish 0:21:10.157 --> 0:21:13.677 is one. And then, um, now I think grand prize 0:21:13.677 --> 0:21:16.557 will be he said, like five, three, 2 million. So 0:21:16.557 --> 0:21:18.476 5 million, 3 million, 2 million or something like that 0:21:18.476 --> 0:21:18.677 might be. 0:21:18.716 --> 0:21:19.037 Four, three. 0:21:19.037 --> 0:21:21.637 2432 or something like that. So first place gets 4 million, 0:21:21.837 --> 0:21:23.637 you know, second place gets 2 million. And at the 0:21:23.637 --> 0:21:26.117 end of the competition they will actually have to open 0:21:26.117 --> 0:21:28.757 source their cyber reasoning systems too, which was a really 0:21:28.757 --> 0:21:31.316 interesting conversation I had away from the table with with 0:21:31.317 --> 0:21:33.356 some people. But okay, so that's that's leading up to it. 0:21:33.397 --> 0:21:36.556 So yeah. So there's three talks around that competition. Uh, 0:21:36.557 --> 0:21:39.117 there's a panel with a whole bunch of leaders or 0:21:39.117 --> 0:21:42.196 people who were associated to the teams. There was, um, 0:21:42.196 --> 0:21:45.157 a couple of people who just presented on their kind 0:21:45.157 --> 0:21:48.547 of research around their cyber reasoning system. And then, um, 0:21:48.547 --> 0:21:51.106 and then there was a talk by the, uh, by 0:21:51.107 --> 0:21:54.627 one of DARPA representatives about why they built the competition 0:21:54.667 --> 0:21:56.627 and stuff like that. So I just wanted to. Yeah, yeah. 0:21:56.627 --> 0:22:00.147 Yeah yeah, yeah. Perfect. Yeah. So the thing I've been 0:22:00.147 --> 0:22:05.427 really super excited about is like these generalizations of architectures 0:22:05.667 --> 0:22:10.826 specifically generalizing the scientific method. So I love the idea of, 0:22:10.867 --> 0:22:14.186 like you have a collection of goals, you have a 0:22:14.307 --> 0:22:18.587 testing engine that basically tests, you have a hypothesis, you 0:22:18.587 --> 0:22:22.907 have problems, goals, and then the testing engine. And you 0:22:22.907 --> 0:22:27.147 can actually combine the ideas as well, like mix them 0:22:27.787 --> 0:22:30.987 and like try to find variations that are actually more effective. 0:22:31.147 --> 0:22:34.547 But it's just this life cycle of here's a cool idea, 0:22:34.946 --> 0:22:38.107 see if it works. Mhm. Um, and so what I 0:22:38.147 --> 0:22:41.467 heard listening to them and we're not divulging anything because 0:22:41.747 --> 0:22:44.817 they were all competitors. Yeah. On the panel. So nobody 0:22:44.817 --> 0:22:46.456 was divulging anything secret. 0:22:46.497 --> 0:22:48.016 Well, except for that one guy. The one guy was like, 0:22:48.017 --> 0:22:48.377 I don't know. 0:22:48.417 --> 0:22:50.936 He's like, screw it. Yeah, yeah, yeah, yeah. But I 0:22:50.936 --> 0:22:53.057 just want to make sure we're not disclosing it. Yeah, yeah, 0:22:53.097 --> 0:22:57.096 it's it's all open. Um, but but, um, basically the 0:22:57.097 --> 0:23:01.297 idea that you could just, like, keep iterating on this 0:23:01.297 --> 0:23:03.857 and you can kind of use it for anything. So 0:23:04.137 --> 0:23:09.057 what I found is, um, really interesting was they basically 0:23:09.057 --> 0:23:12.577 said they spend all their time fixing that system and 0:23:12.577 --> 0:23:16.697 that the, the model getting smarter didn't necessarily help as 0:23:16.696 --> 0:23:18.857 much as improving the system itself. 0:23:18.897 --> 0:23:21.457 That seemed to be what a few of the audience 0:23:21.456 --> 0:23:24.097 questions were aimed at is like, well, you know, as 0:23:24.097 --> 0:23:27.137 the model gets better, doesn't it make it better for you? Um, 0:23:27.456 --> 0:23:29.377 and yeah, like, like you said, some of their answers 0:23:29.377 --> 0:23:33.496 were no, uh, that the scaffolding that hooks everything together 0:23:33.497 --> 0:23:35.697 was actually the important part that they needed to develop 0:23:35.696 --> 0:23:38.857 more than. Yeah, um, the AI models, because they had 0:23:38.857 --> 0:23:42.177 scoped the AI models to do certain things. I remember 0:23:42.177 --> 0:23:45.777 one of the answers was like, um, you know, one 0:23:45.777 --> 0:23:47.337 of the answers for one of the teams was like, 0:23:47.377 --> 0:23:50.976 fast iteration of of that model. The scientific model is like, hey, 0:23:50.976 --> 0:23:52.337 I think this is a cool idea for like a 0:23:52.337 --> 0:23:54.936 single agent to work on, and we're just going to 0:23:54.936 --> 0:23:57.137 test it with the limited data sets that we have 0:23:57.137 --> 0:23:59.577 and the problems that we have and see if it works. 0:23:59.577 --> 0:24:01.016 And he was saying that that was their key to 0:24:01.057 --> 0:24:04.736 success is just like trying so many things, um, different 0:24:04.736 --> 0:24:09.337 representations of code, different ways to, you know, um, you know, 0:24:09.337 --> 0:24:12.577 like make exploits, like, you know, cut it into pieces, 0:24:12.617 --> 0:24:14.697 do it all at once, like, and he's just trying 0:24:14.696 --> 0:24:16.456 all kinds of stuff and like, you know, some panned out, 0:24:16.456 --> 0:24:18.577 some didn't. Yeah. Um, you know, so there's still a 0:24:18.577 --> 0:24:21.217 lot of learning to be there. There's also a lot 0:24:21.257 --> 0:24:25.377 of talk about how the competition also has, um, or 0:24:25.377 --> 0:24:27.536 at least in the first couple rounds, had a ton 0:24:27.537 --> 0:24:31.017 of handicaps like they were. Yeah, they couldn't use like 0:24:31.177 --> 0:24:33.617 they could only use, like $100 in tokens. Yeah. Only 0:24:33.617 --> 0:24:37.817 certain models, um, and only so much output, uh, and context. 0:24:37.817 --> 0:24:40.256 And so they were really, they really said that they 0:24:40.257 --> 0:24:42.726 had designed those systems to work inside those constraints. And 0:24:42.726 --> 0:24:44.167 now in the finals, they don't have any of those 0:24:44.167 --> 0:24:47.167 constraints anymore. So it's like it kind of messed them up. Uh, 0:24:47.167 --> 0:24:49.086 in systems design because they were like, if we would 0:24:49.127 --> 0:24:52.246 have had no restrictions from the beginning, we would we 0:24:52.247 --> 0:24:54.087 might have used different models for this, or we might 0:24:54.087 --> 0:24:55.487 have done x, Y and Z, you know. 0:24:55.527 --> 0:24:58.486 Right, right. Because that was one of my questions of like, what? 0:24:58.486 --> 0:25:02.127 Why are you building this ultra specific thing? It seems 0:25:02.127 --> 0:25:05.246 less efficient. And they're like, that's because we had so 0:25:05.247 --> 0:25:08.847 many limitations. A generic system would not have worked. Yeah. 0:25:08.887 --> 0:25:11.647 You had to put all these weird constraints on it. Yeah. 0:25:11.686 --> 0:25:13.887 Or because they had weird constraints on them? 0:25:13.927 --> 0:25:17.127 Yeah. Yeah. I mean, uh, some of the other things 0:25:17.127 --> 0:25:20.647 there too, were that, uh, some of these security tests 0:25:20.647 --> 0:25:24.726 end up being structured more like unit tests in software. Right? 0:25:24.767 --> 0:25:29.286 And so, um, there are actually libraries out there that, uh, 0:25:29.327 --> 0:25:31.167 that do unit testing really well. And people are like, yeah, 0:25:31.167 --> 0:25:32.607 we just started using those for. 0:25:32.647 --> 0:25:34.567 Oh yeah. The guy mentioned. Yeah. Do you remember that one. 0:25:34.726 --> 0:25:36.407 It's like JS something like I don't know, I have 0:25:36.407 --> 0:25:37.966 it in my notes, but yeah. Me too. Yeah, I'll 0:25:37.966 --> 0:25:39.567 find it. Maybe put it in the show notes. Yeah. 0:25:39.567 --> 0:25:42.796 But there was that. And then I thought interesting questions 0:25:42.797 --> 0:25:45.076 like how much you know, how much are you spending 0:25:45.077 --> 0:25:47.997 on building this system? And that was a really interesting question. 0:25:48.277 --> 0:25:50.757 Um, and nobody was building their own models. 0:25:50.797 --> 0:25:52.837 Yeah. So you can't. Yeah. It's an option now in 0:25:52.837 --> 0:25:55.757 the finals to, to bring your own model. Uh, it 0:25:55.757 --> 0:25:58.877 wasn't before. Um, but nobody's doing it because it's. 0:25:58.877 --> 0:25:59.077 Much. 0:25:59.397 --> 0:26:02.317 Too expensive. Too expensive. And then, you know, like the 0:26:02.317 --> 0:26:04.476 consumers of, you know, the people who weren't competing in 0:26:04.476 --> 0:26:06.677 the competition were like, why don't you use this one 0:26:06.677 --> 0:26:09.077 off hugging face? And like, you don't understand, like, those 0:26:09.077 --> 0:26:10.956 things suck. Like they were like all of the open 0:26:10.956 --> 0:26:14.437 source models that deal with security data right now are horrible. 0:26:14.436 --> 0:26:17.037 And we saw two talked about this week or one 0:26:17.037 --> 0:26:20.277 came out yesterday from Cisco. And then one we can't 0:26:20.277 --> 0:26:22.877 talk about. Um, but it'll be out at some point. 0:26:23.117 --> 0:26:25.397 And they are security trained models. And then we had 0:26:25.677 --> 0:26:29.117 Gemini also talked about a couple of weeks ago. And, 0:26:29.157 --> 0:26:30.197 you know, none of these are out. So they can't 0:26:30.196 --> 0:26:32.476 use them on these teams yet. And so, um, you know, 0:26:32.517 --> 0:26:36.277 maybe the next generation of security data trained models will 0:26:36.277 --> 0:26:38.716 be good, but, um, they just have to work with 0:26:38.787 --> 0:26:40.667 what they've got right now. And they said pretty much 0:26:40.667 --> 0:26:43.667 it all sucks. There's also a reoccurring theme of evals 0:26:43.706 --> 0:26:48.547 suck right now for security. Um, for security based, uh, training? 0:26:48.547 --> 0:26:52.986 Basically, yes. Joel presented on the difficulty of evals. Yeah. 0:26:53.027 --> 0:26:54.786 A lot of people were like, evals suck. And a 0:26:54.787 --> 0:26:57.307 lot of people were like, these are really, really hard. Yeah, 0:26:57.347 --> 0:27:00.027 some people were like, don't. They don't mean what you 0:27:00.027 --> 0:27:00.746 think they mean. 0:27:00.787 --> 0:27:01.747 Yeah, yeah. 0:27:01.787 --> 0:27:04.947 But, um, yeah. To me the eval piece is part 0:27:04.946 --> 0:27:07.627 of that testing engine because like, how do you know 0:27:07.627 --> 0:27:10.427 if the tests worked unless you have good evals. Um, 0:27:10.466 --> 0:27:13.307 but there's like so much hacking of the evals going on. 0:27:13.507 --> 0:27:15.187 Yeah. I mean, that was in my notes, too. It's 0:27:15.186 --> 0:27:18.707 just it's like there's people who are purposefully building in 0:27:19.267 --> 0:27:23.706 either training or, um, logic into some of these things 0:27:23.706 --> 0:27:25.107 to score really high on these. 0:27:25.147 --> 0:27:26.306 Yeah. They like post train with it. 0:27:26.347 --> 0:27:28.186 Yeah. Post train with it. And so they score really 0:27:28.186 --> 0:27:30.667 high on these, uh, things that I would look at 0:27:30.667 --> 0:27:33.147 as a consumer. Right. And I'm like, oh, okay. So 0:27:33.147 --> 0:27:35.306 I'm going to go look at, you know, these, you know, 0:27:35.347 --> 0:27:38.017 these evals on hugging face. You know, I forget the 0:27:38.057 --> 0:27:41.256 really popular one on hugging face. It's like the chat arena. 0:27:41.297 --> 0:27:42.936 Chat arena. And they'd be like, cool. How does it 0:27:42.936 --> 0:27:45.177 score on Chat Arena? And it's like, okay, it's really high. Well, 0:27:45.417 --> 0:27:47.216 that doesn't mean it doesn't mean anything for for a 0:27:47.257 --> 0:27:49.897 domain specific application of an AI, right? Like, you may 0:27:49.897 --> 0:27:52.377 not need a model that's really good at on chat 0:27:52.377 --> 0:27:57.377 arena for something else. And so one of our longtime friends, 0:27:57.377 --> 0:28:00.377 Joel Parrish, former you worked you worked for me at 0:28:00.456 --> 0:28:02.817 Deadspin when he first started. I'll tell a funny story here. 0:28:02.817 --> 0:28:06.256 So Matt Knight is the CISO of OpenAI and I 0:28:06.257 --> 0:28:08.777 don't know, Matt. Super. Well, I know him a little bit. 0:28:08.817 --> 0:28:11.696 And I was like, hey, like, I, uh, I was 0:28:11.736 --> 0:28:13.897 actually one of the guys who worked with Joel at Deadspin, 0:28:13.897 --> 0:28:17.496 our first testing job together a two decades ago. Right. 0:28:17.537 --> 0:28:19.737 And then he worked for us at HP or with 0:28:19.736 --> 0:28:22.657 us at HP. Then he went to Apple with you, 0:28:22.696 --> 0:28:24.856 and then he went to OpenAI with Matt. And so 0:28:24.857 --> 0:28:26.137 I told Matt, I said, I think I'm one of 0:28:26.137 --> 0:28:28.817 his first LinkedIn recommendations. I don't know if he kept 0:28:28.817 --> 0:28:30.137 it on his profile or not, but I called him 0:28:30.137 --> 0:28:32.337 the Kobe Bryant of web hacking. Right? Yeah. 0:28:32.696 --> 0:28:33.016 Totally. 0:28:33.057 --> 0:28:35.737 Yeah, totally. Um, but he does everything now. Anyway, Joel 0:28:35.847 --> 0:28:38.647 gave a presentation at this is the Run Sibyl Side event, 0:28:38.647 --> 0:28:39.927 which was also really good. 0:28:39.967 --> 0:28:40.407 It was. 0:28:40.687 --> 0:28:43.527 Great. The run, the run. Sibyl side event. Um, the 0:28:43.567 --> 0:28:47.607 Run Sibyl is a company that's doing, um, automated pen 0:28:47.607 --> 0:28:52.887 testing and, um, and pen tester assisted AI tools, basically, uh, 0:28:52.887 --> 0:28:55.047 led by a friend of ours, Ari. And so I 0:28:55.047 --> 0:28:57.047 spoke there, gave my talk there. But Joel was right 0:28:57.047 --> 0:29:01.607 before me. And Joel gave a talk called your Eval suck. Um, 0:29:01.647 --> 0:29:04.727 and his thing was like, hey, look at all these 0:29:05.327 --> 0:29:07.847 evals that, uh, people are using right now. They're written 0:29:07.847 --> 0:29:10.607 in the 90s, like stack based overflows, right? Which is 0:29:10.607 --> 0:29:14.607 not what we are facing in 2025. Right. And, uh, 0:29:14.647 --> 0:29:17.846 he just showed like, multiple examples of, you know, some 0:29:17.847 --> 0:29:20.167 of these evals are not even testing exploit generation or 0:29:20.167 --> 0:29:22.207 web testing or anything like that. They're testing just like 0:29:22.207 --> 0:29:24.566 code quality stuff, which is from the 90s. Right? And 0:29:24.567 --> 0:29:28.247 it's like it's like, why are we benchmarking these security 0:29:28.447 --> 0:29:32.207 models off of these crazy old evals? Um, and that 0:29:32.207 --> 0:29:34.757 was that was the genesis of his talk. What I 0:29:34.757 --> 0:29:37.877 was sad about, though, is you left for dinner one 0:29:37.917 --> 0:29:40.957 talk after or one talk before it ended. And the 0:29:40.957 --> 0:29:43.237 last talk was this guy who basically brought a humanoid 0:29:43.237 --> 0:29:46.597 robot like, about this tall and had jailbroken it with 0:29:46.597 --> 0:29:49.036 an exploit. He bought it from China and got it 0:29:49.037 --> 0:29:51.237 to run around the room, just like screaming at the 0:29:51.237 --> 0:29:53.036 top of its lungs with like, a video game track. 0:29:53.037 --> 0:29:55.517 It was so cool. Like it was. It was amazing. 0:29:55.557 --> 0:29:58.596 Attacking communism. Yeah. As a Chinese robot. Yeah. 0:29:58.637 --> 0:30:01.477 So, okay, so out of context, it sounds bad, but 0:30:01.477 --> 0:30:04.957 if you've ever played, um, the fallout games, um, there 0:30:04.957 --> 0:30:08.997 is like this audio track in, uh, in fallout that, like, 0:30:09.037 --> 0:30:12.277 the rhetoric in that game is anti-communism, right? And so, like, 0:30:12.517 --> 0:30:15.117 there's like a robot that goes around or like a, 0:30:15.477 --> 0:30:18.077 like a character, you know, some of the enforcers run 0:30:18.077 --> 0:30:21.597 around all they do for 24 seven just talk about anti-communism. 0:30:21.597 --> 0:30:23.437 So if you if you pull down the audio track 0:30:23.437 --> 0:30:26.277 for this game, there's a 20 minute rant robot rant 0:30:26.277 --> 0:30:29.837 about anti-communism. And so he put it on the jailbroken 0:30:29.837 --> 0:30:33.347 robot and it's just running around spouting like, anti-communist and 0:30:33.347 --> 0:30:35.467 then like it was so funny because it was it 0:30:35.467 --> 0:30:38.067 was it was like it was talking about anti-communism and 0:30:38.067 --> 0:30:40.347 it was like it was like possible defector. And then 0:30:40.347 --> 0:30:42.947 it ran into this lady's table and spilled her drink 0:30:42.947 --> 0:30:44.867 on her, and she thought it was so hilarious. Like, 0:30:44.867 --> 0:30:48.347 she was like, this is crazy. It was it was great. Yeah. Yeah. 0:30:48.507 --> 0:30:49.867 Yeah. I got to get the video. 0:30:50.147 --> 0:30:54.187 Yeah, yeah, yeah. So, um. Yeah. So evals, you know, struggle, 0:30:54.267 --> 0:30:58.267 benchmark struggle right now for domain specific applications. Everyone's kind 0:30:58.267 --> 0:31:01.227 of figuring it out. And I think I felt like 0:31:01.267 --> 0:31:03.147 on the show floor, there was still a lot of 0:31:03.147 --> 0:31:06.427 promising of things that when I went to talk to people, 0:31:06.427 --> 0:31:10.507 it was not as, um. First of all, no one's 0:31:10.507 --> 0:31:11.907 training their own models to do any of this, right. 0:31:11.907 --> 0:31:13.467 Like vendors will say, yeah, we have our own model. 0:31:13.467 --> 0:31:17.187 They're not like they're using llama for or, you know, 0:31:17.227 --> 0:31:20.227 their own keys and yeah, or. Yeah, deep seek or. 0:31:20.267 --> 0:31:20.947 Just the cloud ones. 0:31:20.947 --> 0:31:23.907 Yeah, just the cloud ones. Anthropic or OpenAI. Right. And 0:31:23.907 --> 0:31:26.107 then all of the business logic magic that they promise 0:31:26.147 --> 0:31:29.827 you is happening is system prompt based. Like that's, that's 0:31:29.827 --> 0:31:33.096 the majority of of all of those products. And the 0:31:33.097 --> 0:31:36.096 value prop is no longer automation. The value prop for 0:31:36.097 --> 0:31:39.497 them is now, oh, you know, up level or skill 0:31:39.537 --> 0:31:43.177 level your people. Right. And um, and that means basically 0:31:43.217 --> 0:31:46.137 at least the way I, I, you know, kind of 0:31:46.177 --> 0:31:48.217 package that is it's good at the things we already 0:31:48.217 --> 0:31:50.177 knew it was good at. Right. It's good at summarization. 0:31:50.177 --> 0:31:52.497 It's good at rewriting. It's good at pulling multiple data 0:31:52.537 --> 0:31:55.257 sets together and offering, you know, a couple insights here 0:31:55.257 --> 0:31:59.657 and there. But the actual automation of things not quite 0:31:59.657 --> 0:32:02.617 there yet harder to implement. Architecture is way you need 0:32:02.617 --> 0:32:03.937 to invest way more in it. 0:32:03.977 --> 0:32:06.537 Yeah, I mean, I think we knew this before going 0:32:06.577 --> 0:32:08.897 into RSA just because you and I are actually building 0:32:08.897 --> 0:32:12.617 this stuff. It's like the problem with agents and like 0:32:12.657 --> 0:32:15.977 pursuing goals is they just get confused, right? They get 0:32:15.977 --> 0:32:17.336 confused over scoped. 0:32:17.537 --> 0:32:18.057 Scoped. 0:32:18.057 --> 0:32:20.977 Yeah. Especially if you have like red teaming for example, 0:32:20.977 --> 0:32:23.017 which is the one that all our friends are struggling 0:32:23.017 --> 0:32:23.817 with the most. 0:32:23.817 --> 0:32:24.137 Yeah. 0:32:24.457 --> 0:32:26.537 Because the first step. Cool. I can launch the web 0:32:26.537 --> 0:32:29.057 attack second. Okay. I can pivot a little bit. Yeah. 0:32:29.097 --> 0:32:31.657 But then it's like, okay, I've got seven more goals 0:32:31.657 --> 0:32:34.536 to get. What have I done already? And so it 0:32:34.537 --> 0:32:36.777 starts losing context. So I think that's where it's kind 0:32:36.777 --> 0:32:39.777 of falling apart. Yeah. The other thing we were talking 0:32:39.777 --> 0:32:45.537 about was how, um, the moat situation. Yeah. So, so basically, um, 0:32:46.937 --> 0:32:50.417 a lot of these companies that are like, we do 0:32:50.417 --> 0:32:55.697 this thing, that's what makes us special. And we're in round, 0:32:55.817 --> 0:32:58.617 you know, B or C or whatever. And we've raised 0:32:58.617 --> 0:33:02.457 all this money because we do this one thing. It's like, well, 0:33:02.457 --> 0:33:05.537 if you have the context, like, uh, I was talking 0:33:05.537 --> 0:33:07.737 about this week, if you have that context and you 0:33:07.737 --> 0:33:11.737 can ask the questions. The thing that company does is 0:33:11.737 --> 0:33:15.657 a feature instead of a company. Yeah. Yeah. And like, 0:33:16.217 --> 0:33:19.057 as we would go around and see these different booths. Yeah. 0:33:20.097 --> 0:33:20.977 They look like features. 0:33:20.977 --> 0:33:22.017 They're gonna get karate kicked. 0:33:22.057 --> 0:33:23.097 By by. 0:33:23.257 --> 0:33:27.057 Uh, the model vendors and. Yeah, it's, uh. Yeah. I mean, 0:33:27.097 --> 0:33:29.007 it sucks because those are some. awesome. It is some 0:33:29.007 --> 0:33:30.807 of our friends who are making these companies and it's like, 0:33:30.807 --> 0:33:33.007 I don't know. I mean, you can be really good 0:33:33.007 --> 0:33:35.207 at a problem and succeed better than a big model 0:33:35.207 --> 0:33:37.807 vendor or a big a big company that you know. 0:33:37.887 --> 0:33:39.447 But it's got to be really good. 0:33:39.487 --> 0:33:42.487 And maybe they just cut through so good with marketing 0:33:42.487 --> 0:33:45.767 that they get a big enough market share that they're okay. Yeah. 0:33:45.927 --> 0:33:49.286 But the time is like ticking. You just hear the 0:33:49.287 --> 0:33:53.527 time ticking down between Google gets there, Microsoft gets there. 0:33:53.527 --> 0:33:55.927 One of these big players gets there. Yeah. And then 0:33:55.927 --> 0:34:00.127 they just start adding question modules for security or whatever. 0:34:00.167 --> 0:34:02.047 I mean maybe that's the golden plan though. Maybe it's 0:34:02.047 --> 0:34:05.287 like they're not at the place to stay a long 0:34:05.327 --> 0:34:07.687 term viable. Just just get bought by one of the 0:34:07.687 --> 0:34:09.286 big companies because they do it really well. Right. Which 0:34:09.287 --> 0:34:10.647 is totally a play. Yeah. 0:34:10.847 --> 0:34:11.326 Nothing wrong with. 0:34:11.327 --> 0:34:21.567 That. Anyone wants to buy. I'm just kidding. Um, yeah. So. Yeah. Um, yeah. 0:34:21.567 --> 0:34:24.767 So that could be a play for sure. Yeah. Uh, yeah. 0:34:24.807 --> 0:34:28.397 I mean, other than that, though, uh, Again, same vibes 0:34:28.397 --> 0:34:31.517 as last year and the year before. I saw some 0:34:31.517 --> 0:34:38.637 products that, uh. Besides, I are just abstractions of what 0:34:38.677 --> 0:34:41.357 someone else already does, but it's a better visualization and 0:34:41.357 --> 0:34:43.157 easier to make it work with. Right. So like the 0:34:43.157 --> 0:34:46.277 Amazon ecosystem, if you're an Amazon specialist, that shit's hard 0:34:46.277 --> 0:34:48.277 to learn. Like there are so many sub tools and 0:34:48.277 --> 0:34:50.357 sub products and it's like and so then like you 0:34:50.357 --> 0:34:52.517 see this other company like, yeah, we make this easy, right? 0:34:52.557 --> 0:34:56.237 Like here's a nice guy explains everything Wiz. Yeah. Wiz. Right. 0:34:56.237 --> 0:34:58.397 And it like does everything that you want it to do. 0:34:58.837 --> 0:35:01.197 And I don't think there's a lot of moat around 0:35:01.197 --> 0:35:04.117 that either. That's just a UI revamp to a lot 0:35:04.117 --> 0:35:06.037 of the core services places. And so I saw a 0:35:06.037 --> 0:35:09.117 lot of that. It's like make your SoC easier to 0:35:09.117 --> 0:35:12.277 automate or like whatever. And it's like, okay, I get it. 0:35:12.277 --> 0:35:14.437 I get why that's attractive right now, because you have 0:35:14.437 --> 0:35:16.997 that pain right now as a consumer. But that pain, 0:35:16.997 --> 0:35:19.197 I don't know if it'll be there forever once other 0:35:19.197 --> 0:35:21.277 people figure out. Although you can also look at case 0:35:21.277 --> 0:35:23.397 studies from Google, right? They never fix their UI. And 0:35:23.437 --> 0:35:26.387 you know, so like like Gmail could use a refresh. 0:35:26.387 --> 0:35:29.067 So yeah, I mean, yeah, um, I saw a lot 0:35:29.067 --> 0:35:33.587 of that. Um, was also kind of surprised just at the, 0:35:33.627 --> 0:35:35.947 at what I perceived to be the spend at RSA 0:35:35.987 --> 0:35:40.027 this year in a time where I know security professionals like, 0:35:40.027 --> 0:35:42.387 who are jobless and, and they have been looking for 0:35:42.387 --> 0:35:43.547 roles for months. Right. 0:35:43.547 --> 0:35:45.427 That's a really good point. I didn't think about that. 0:35:45.427 --> 0:35:48.147 And then just the amount of of money on that floor. 0:35:48.187 --> 0:35:51.067 It felt like 2018 or 2019. 0:35:51.107 --> 0:35:51.827 Yeah, it was crazy. 0:35:51.867 --> 0:35:54.107 It was like top of the you know what it 0:35:54.107 --> 0:35:57.947 almost felt like it almost felt like. And I hadn't 0:35:57.947 --> 0:36:00.107 thought of this until just now when you said this. 0:36:00.627 --> 0:36:03.947 It feels like desperation. They're just like, spend all the money. 0:36:03.947 --> 0:36:06.587 It's like. It's like our last chance. 0:36:06.587 --> 0:36:09.987 Yeah, it definitely felt like that with some vendors for sure. Um, 0:36:10.787 --> 0:36:13.867 which maybe goes into that like acquisition play is like, 0:36:13.907 --> 0:36:16.547 we just make ourselves seem bigger this year. We'll get 0:36:16.547 --> 0:36:18.947 acquired and it won't matter anymore. Right. 0:36:18.987 --> 0:36:22.427 Well, so if you take like the macro economy or whatever, 0:36:22.627 --> 0:36:25.457 and it's just like things might get bad in the 0:36:25.457 --> 0:36:29.097 next six months or a year. Yeah, we're going into RSA. Yeah, 0:36:29.257 --> 0:36:32.217 we need to get bought. Yeah, we or we need 0:36:32.217 --> 0:36:34.737 to get a bunch of customers. Yeah. Now is not 0:36:34.737 --> 0:36:36.057 the time to go small. 0:36:36.097 --> 0:36:36.657 Yeah. 0:36:36.857 --> 0:36:38.497 So we saw baby goats. 0:36:38.537 --> 0:36:41.177 Yeah, we saw goats. We saw puppies. 0:36:41.217 --> 0:36:43.097 I didn't see the puppies, but I saw the goats. 0:36:43.497 --> 0:36:46.497 There's a monster truck and an F1 formula one car. 0:36:46.537 --> 0:36:47.137 2018? 0:36:47.337 --> 0:36:51.137 Yeah. I mean, um, what else? A giant, giant robot, obviously, 0:36:51.137 --> 0:36:55.497 at the CrowdStrike booth, like, every year. Giant statue. Um, yeah, 0:36:55.497 --> 0:36:58.417 there was I mean, there was usually there's one marquee 0:36:58.457 --> 0:37:02.577 party at uh, at or at RSA, right. There's like 0:37:02.577 --> 0:37:04.617 one vendor who brings in like a band. So like 0:37:04.657 --> 0:37:07.217 last year, I can't remember who I went to Dead 0:37:07.257 --> 0:37:10.577 Mouse last year, which is one of my favorites. And then, um, 0:37:10.657 --> 0:37:13.377 for like, the rock crowd, they had, um, Incubus, I 0:37:13.377 --> 0:37:15.537 think last year or maybe, maybe the year before, I 0:37:15.537 --> 0:37:20.297 can't remember this year. Both marshmallow still Premier DJ and 0:37:20.297 --> 0:37:23.767 Chainsmokers were performing at different parties on the same night, 0:37:24.567 --> 0:37:25.367 which is crazy. 0:37:25.447 --> 0:37:27.007 One. Sentinel one, I think. 0:37:27.047 --> 0:37:30.207 Yes, I know one was a marshmallow. Yeah. And, uh, 0:37:30.247 --> 0:37:33.687 and then, um, I can't remember who did, uh, Chainsmokers, 0:37:33.687 --> 0:37:35.847 but yeah, I think it was like chain guard or something. 0:37:35.847 --> 0:37:38.407 I can't remember, but, um, but yeah, I mean, that's 0:37:38.407 --> 0:37:40.727 a lot of money to, like, you know, buy out 0:37:40.727 --> 0:37:43.807 a nightclub for hundreds of people, you know, have, like, 0:37:43.807 --> 0:37:46.647 a premier DJ play just for your corporate party. Um, 0:37:46.887 --> 0:37:49.127 and so it just it felt like there was a 0:37:49.127 --> 0:37:50.527 lot of money spending. And it made me sad a 0:37:50.527 --> 0:37:52.607 little bit because, like, I do have friends who have 0:37:52.607 --> 0:37:55.367 been struggling to find jobs or have gotten like, um, 0:37:55.847 --> 0:37:58.167 work furloughed, you know, a lot of friends getting furloughed 0:37:58.207 --> 0:38:01.007 where like, they're like, oh, we can't afford to pay you. 0:38:01.047 --> 0:38:03.567 Yeah. We got to cut back on salaries. Yeah, because 0:38:03.607 --> 0:38:04.327 money's tight. 0:38:04.367 --> 0:38:04.647 Yeah. 0:38:05.167 --> 0:38:06.007 But we need goats. 0:38:06.047 --> 0:38:08.407 Yeah, we need goats. Yeah, we need goats. So. So 0:38:08.407 --> 0:38:10.647 that kind of sucked a little bit. I think that's 0:38:10.647 --> 0:38:13.407 a continuing pattern though. Probably that's happened every year a 0:38:13.407 --> 0:38:14.807 little bit. But um. 0:38:14.847 --> 0:38:18.167 I'm worried about next year that they're just like, well 0:38:18.367 --> 0:38:19.287 that didn't work. 0:38:19.607 --> 0:38:19.887 Yeah I. 0:38:19.887 --> 0:38:20.567 Mean tighten it. 0:38:20.567 --> 0:38:23.157 Up. We'll see. I mean, every year you look for 0:38:23.157 --> 0:38:25.677 a vendor that you thought was cool last year, and 0:38:25.677 --> 0:38:27.997 then they're not there, you know, this year. 0:38:28.037 --> 0:38:28.837 Yeah. Isn't that weird? 0:38:28.877 --> 0:38:29.357 Yeah. 0:38:29.397 --> 0:38:31.557 Someone comes out of nowhere and, like, four of them 0:38:31.557 --> 0:38:32.557 just disappear. 0:38:32.597 --> 0:38:34.437 Yeah, there was a couple, a couple last year that 0:38:34.437 --> 0:38:39.357 was really excited that they were applying AI to document classification. 0:38:39.357 --> 0:38:43.437 And I was like, that's a perfect application of AI, actually. Yeah, yeah. Um, 0:38:43.437 --> 0:38:45.597 and they were not around this year either they got 0:38:45.637 --> 0:38:47.917 gobbled up or they didn't make it. So yeah. 0:38:48.437 --> 0:38:50.997 Oh yeah. What's the one I got? A buddy went, oh, 0:38:51.037 --> 0:38:53.397 Sierra is the one. Yeah. Doing it now. 0:38:53.437 --> 0:38:57.557 Yeah. Um, but yeah, the, uh, I mean, the off 0:38:57.597 --> 0:38:59.237 site events are definitely where it's at, I think. I 0:38:59.237 --> 0:39:02.917 think if you're coming to RSA next year and we're coming. Right. Yeah, 0:39:02.957 --> 0:39:04.837 I'm gonna do a little bit less. Speaking honestly, I 0:39:04.877 --> 0:39:10.117 did my thing five times and I was pretty burnt. Um, again, 0:39:10.157 --> 0:39:13.157 I guess if I rewind to the beginning of the week, though. Besides, 0:39:13.157 --> 0:39:15.477 San Francisco continues to be an A plus con. 0:39:15.517 --> 0:39:15.877 Yeah. 0:39:15.957 --> 0:39:18.077 Um, I mean, besides San Francisco. 0:39:18.197 --> 0:39:19.117 Production quality. 0:39:19.117 --> 0:39:20.147 Production quality. 0:39:20.187 --> 0:39:21.147 Quality? The content? 0:39:21.147 --> 0:39:24.027 Yes. Staff is great. Yeah. You know, the villages there 0:39:24.067 --> 0:39:26.947 are cool. Even the vendors set up there is like. 0:39:26.987 --> 0:39:29.867 I just feel like it's not as nuts and in 0:39:29.867 --> 0:39:32.147 your face. I had a I had a buddy, a 0:39:32.147 --> 0:39:35.027 mutual friend of ours. Come this year. He's, you know, 0:39:35.067 --> 0:39:37.627 he's a person. He works at a company and I'm 0:39:37.627 --> 0:39:38.827 not going to put him on blast. But he's a 0:39:38.827 --> 0:39:40.227 person who works in a company. He doesn't have any 0:39:40.227 --> 0:39:43.067 purchasing power. But, you know, he put on his RSA 0:39:43.107 --> 0:39:45.347 badge the company he worked for. And it's a big company. 0:39:45.627 --> 0:39:48.107 And he is getting accosted like he's walking down the 0:39:48.107 --> 0:39:50.347 floor and like, you know, like someone out of the 0:39:50.347 --> 0:39:51.947 corner of their eye just sees his badge in the 0:39:51.947 --> 0:39:55.067 name of his company. And they're like, hey, like, you know, like, 0:39:55.107 --> 0:39:57.747 come talk to me. And like, um, it was surreal 0:39:57.747 --> 0:40:00.507 for him, even even after he tells them I don't 0:40:00.507 --> 0:40:02.907 have any purchasing power, I don't make any decisions. And 0:40:02.907 --> 0:40:03.907 they're like, I don't care. 0:40:03.947 --> 0:40:04.827 Like, yeah, yeah. 0:40:04.827 --> 0:40:08.307 Yeah, uh, but besides, doesn't feel like that. Um, I 0:40:08.307 --> 0:40:10.747 think that, uh, a mutual friend of ours, uh, Clint, 0:40:10.747 --> 0:40:15.187 gave a talk on vulnerability as people and as infosec 0:40:15.187 --> 0:40:19.457 practitioners rather than vulnerabilities as, like, you know, kind of. 0:40:19.497 --> 0:40:23.577 We pop bones or security. Security. I really love that talk. 0:40:23.617 --> 0:40:25.697 I think it's one of the best keynotes I've seen 0:40:26.337 --> 0:40:27.817 in quite a while. It will be up on the 0:40:27.857 --> 0:40:30.177 b sides website. You know, they eventually put everything out. 0:40:30.177 --> 0:40:32.057 So I highly suggest watching you and I were cameoed 0:40:32.057 --> 0:40:32.817 in that talk, actually. 0:40:32.857 --> 0:40:33.697 Yeah, absolutely. 0:40:33.737 --> 0:40:36.737 Clint, Clint talked about, um, how, you know, you and 0:40:36.737 --> 0:40:38.977 I have like this, you know, long friendship because we've 0:40:38.977 --> 0:40:41.457 worked together since we were young and we've just kind 0:40:41.457 --> 0:40:44.337 of done everything together. Yeah. And, um, he he talked 0:40:44.337 --> 0:40:47.017 about how, like, being a friend with you, he, like, 0:40:47.057 --> 0:40:49.177 felt like a little bit less than, you know, the 0:40:49.177 --> 0:40:51.537 connection we had. And he wanted that with you. Yeah. 0:40:51.577 --> 0:40:53.497 And how, like, those things are hard to talk about, right? 0:40:53.537 --> 0:40:56.897 It's like, you know, your insecurities, the way you feel. Um, 0:40:56.897 --> 0:40:59.577 but eventually, if you confront them or you figure out 0:40:59.577 --> 0:41:01.937 ways to help, you know, like, you know, be healthy 0:41:01.937 --> 0:41:04.417 about them and have conversations with your friends and say, X, Y, 0:41:04.417 --> 0:41:07.817 and Z, it's, um, it really can give you peace 0:41:07.817 --> 0:41:11.417 of mind. Superpowers make you feel better. Yeah. And so, like, 0:41:11.457 --> 0:41:13.177 you and I were referenced in that that part of 0:41:13.217 --> 0:41:15.137 that talk. And I thought that that was it was 0:41:15.137 --> 0:41:17.377 really great, actually. I actually like I think I cried 0:41:17.377 --> 0:41:18.937 at the end because he had like a couple, like 0:41:19.057 --> 0:41:20.097 little messages in there and he. 0:41:20.097 --> 0:41:20.537 Was like, yeah. 0:41:20.537 --> 0:41:22.457 Yeah. He was like, hey, you're enough, right? Like, what 0:41:22.457 --> 0:41:23.817 you're doing is enough and. 0:41:23.857 --> 0:41:24.697 All, and it matters. 0:41:24.737 --> 0:41:28.377 And it matters. Yeah. And, uh, and all of us, uh, 0:41:28.377 --> 0:41:29.897 all of us, you know, are just trying. I feel 0:41:29.897 --> 0:41:32.057 like everyone in security has a little bit of, like, 0:41:32.097 --> 0:41:35.057 they just want to, like, help the world a little bit, right? Like, 0:41:35.097 --> 0:41:37.177 not everyone, but a lot of people. That's why they 0:41:37.177 --> 0:41:39.817 get into it. Because it is easy to make that 0:41:39.817 --> 0:41:42.217 that line to like, hey, I know this is a 0:41:42.217 --> 0:41:45.337 small thing, this computer stuff, but in a way, I 0:41:45.377 --> 0:41:48.457 am a superhero trying to help the world. Right? And, um, 0:41:48.457 --> 0:41:50.217 but you can get so wrapped up because there's so 0:41:50.217 --> 0:41:53.497 much stuff, right? There's so many domains. There's new research 0:41:53.497 --> 0:41:56.097 in domains all the time you feel behind on education, 0:41:56.457 --> 0:41:59.057 you can start to get that imposter syndrome. And like 0:41:59.097 --> 0:42:01.297 at the end, Clint actually gave out cards. 0:42:01.977 --> 0:42:02.777 That he had signed. 0:42:02.817 --> 0:42:05.017 That he had signed. He had signed hundreds of cards 0:42:05.297 --> 0:42:07.457 by hand. And I think I have one in my 0:42:07.457 --> 0:42:09.217 wallet and I put mine in my wallet. And it 0:42:09.217 --> 0:42:11.377 just like says you are enough signed by Clint. Or like, 0:42:11.417 --> 0:42:12.777 you know, some inspiring message. 0:42:12.817 --> 0:42:13.257 Yeah. 0:42:13.537 --> 0:42:16.807 That was I mean, like, Clint's like amazing person. But 0:42:16.847 --> 0:42:17.807 that talk was awesome. 0:42:17.847 --> 0:42:21.207 So yeah, and a lot of courage to do a 0:42:21.207 --> 0:42:25.087 talk that's about people. Yeah. When like the natural play 0:42:25.087 --> 0:42:29.407 is like AI and security V2. Yeah. Which would absolutely 0:42:29.407 --> 0:42:31.727 crush it would crush. Yeah, yeah. And he's like, no, 0:42:31.727 --> 0:42:33.687 I'm going to do this thing because I think this 0:42:33.687 --> 0:42:34.567 message matters. 0:42:34.607 --> 0:42:38.607 Yeah, yeah, yeah. Clint's a phenomenal human. And, um, he 0:42:38.607 --> 0:42:41.007 did the girlfriend meme of, uh, of you and me. 0:42:41.047 --> 0:42:43.727 Like the, like the guys walking down the street, and 0:42:43.727 --> 0:42:47.487 he looks at like the like. Like the one girl has, like, 0:42:47.527 --> 0:42:49.647 a girlfriend, and the other girlfriend's like, you know, that 0:42:49.647 --> 0:42:52.647 meme and I got to be the the hot girl, so. 0:42:52.687 --> 0:42:53.407 That's right. You know. 0:42:53.527 --> 0:42:56.047 I haven't I've never been the hot girl. So like. 0:42:56.447 --> 0:42:56.967 Yeah, yeah. 0:42:59.127 --> 0:43:02.687 Yeah. Um. Yeah. So that was cool. Uh, what else 0:43:02.687 --> 0:43:03.567 happened this week? 0:43:04.607 --> 0:43:07.807 I don't know, what's your big takeaway? I feel like 0:43:08.407 --> 0:43:10.607 we talked about this a little bit. I feel like 0:43:10.607 --> 0:43:14.437 big takeaway for us. And keep in mind that If 0:43:14.437 --> 0:43:18.717 you're coming to this, like brand new, like you don't 0:43:18.717 --> 0:43:20.717 necessarily want to do it this way because we're we're 0:43:20.757 --> 0:43:23.837 getting lessons after I've done this for a couple of decades, 0:43:24.157 --> 0:43:26.877 but we're like more get away from the center of 0:43:26.877 --> 0:43:31.397 the mass and move into like the smaller events where 0:43:31.397 --> 0:43:33.797 your friends are going to be at to talk more 0:43:33.797 --> 0:43:37.077 about the ideas as opposed to like, where can I 0:43:37.077 --> 0:43:39.077 get the food? Where can I get the parties and 0:43:39.077 --> 0:43:39.877 the music? 0:43:40.077 --> 0:43:43.077 Yeah. And that, I mean, this thinking applies to Black 0:43:43.077 --> 0:43:43.957 Hat and Def Con, too. 0:43:44.117 --> 0:43:44.557 Absolutely. 0:43:44.557 --> 0:43:44.677 Yeah. 0:43:44.717 --> 0:43:45.677 It's the whole scene. 0:43:45.797 --> 0:43:49.837 It's, uh, at first it's cool when you're young and 0:43:49.837 --> 0:43:50.757 we're old, so. 0:43:50.797 --> 0:43:50.997 Yeah. 0:43:50.997 --> 0:43:51.277 Yeah. 0:43:51.317 --> 0:43:52.317 Um, to. 0:43:52.437 --> 0:43:54.597 To go to the parties and be part of, like, 0:43:54.637 --> 0:43:58.197 the loud noises and then now you kind of want to, uh, 0:43:58.837 --> 0:44:01.557 ration your time with people who you really want to 0:44:01.557 --> 0:44:04.477 spend time with and have conversations with and be in 0:44:04.477 --> 0:44:07.317 smaller settings where it's not so loud and like real 0:44:07.317 --> 0:44:09.677 research is going on. And, uh, and that was the 0:44:09.677 --> 0:44:13.347 dichotomy for me, was like so much of a difference 0:44:13.347 --> 0:44:17.107 between what people were saying on the floor about how 0:44:17.107 --> 0:44:19.267 things worked with AI. And then when I got to 0:44:19.307 --> 0:44:23.027 the Airbnb summit and the OpenAI summit, like, no, these 0:44:23.027 --> 0:44:25.147 are real people working on these problems. And here are 0:44:25.147 --> 0:44:28.747 the real problems. And yeah, um, and here also there 0:44:28.747 --> 0:44:31.467 are success stories, but also their failure stories, like, yeah, 0:44:31.467 --> 0:44:33.427 we thought this would work. It totally did not work. 0:44:33.467 --> 0:44:37.747 We had to go back to manual process in vulnerability management. 0:44:37.747 --> 0:44:41.907 Oh man. The conversation about vulnerability management this week were crazy, um, 0:44:41.907 --> 0:44:46.427 about using AI and the predispositions I had about what 0:44:46.427 --> 0:44:49.907 I thought was a good AI assisted vulnerability management plot 0:44:49.947 --> 0:44:53.987 or not platform, but, you know, like architecture versus what 0:44:54.027 --> 0:44:56.627 some people like Google have built and some people like 0:44:56.707 --> 0:44:58.347 Adobe did a great talk on it, and then some 0:44:58.347 --> 0:45:02.387 people talked about it in the OpenAI, um, conference. Uh, 0:45:02.507 --> 0:45:05.227 and so, like, it turns out some of the things 0:45:05.227 --> 0:45:06.867 that we thought I would be able to do are 0:45:06.867 --> 0:45:10.347 not the force multipliers. It turns out to be sending 0:45:10.347 --> 0:45:12.857 emails or like actioning tickets automatically. 0:45:12.897 --> 0:45:13.497 Go to where they. 0:45:13.497 --> 0:45:15.257 Are. Yeah, yeah, go to where they are. Right. We 0:45:15.297 --> 0:45:18.577 talked about this in the car. It's like it's I 0:45:18.617 --> 0:45:21.097 had some assumptions that we'd be able to do like full, 0:45:21.137 --> 0:45:24.457 full stop, you know, full prompts like, you know, uh, 0:45:24.777 --> 0:45:27.617 nuts to bolts send, you know, our vulnerabilities that come 0:45:27.617 --> 0:45:30.857 through a bug bounty or static code analysis or through, um, 0:45:31.697 --> 0:45:34.457 appsec testing or through, you know, any different number of 0:45:34.457 --> 0:45:36.177 where we get vulns. Right. And it would, you know, 0:45:36.177 --> 0:45:39.817 system would work. Uh, the, the, the value of the 0:45:39.817 --> 0:45:43.497 AI would be the rating, the conglomeration of all that 0:45:43.497 --> 0:45:47.457 into tickets. And it turns out that some people, at least, 0:45:47.497 --> 0:45:49.617 who have done it say actually like that doesn't work 0:45:49.617 --> 0:45:53.177 super well. Mhm. Um, they're stripping all of that contextual 0:45:53.177 --> 0:45:57.217 data about ratings making their own rating systems. Yeah. Um, 0:45:57.217 --> 0:46:00.297 and just pulling out the text from the advisories from 0:46:00.297 --> 0:46:02.417 the threat feeds, from the pen test report, from all 0:46:02.417 --> 0:46:06.137 that stuff, rewriting them themselves with custom systems that have 0:46:06.137 --> 0:46:08.297 nothing to do with CWA or CV. 0:46:10.007 --> 0:46:13.287 You know. Yeah. That's the thing. The these rating systems 0:46:13.287 --> 0:46:15.367 are trying to give us the context of the vote. 0:46:15.407 --> 0:46:16.487 They don't have any context or. 0:46:16.567 --> 0:46:17.727 Know anything about. 0:46:17.767 --> 0:46:18.607 Know anything about us. 0:46:18.807 --> 0:46:21.087 So I love this idea. Just strip it out. Yeah. 0:46:21.127 --> 0:46:24.407 And then re-add the context from the company onto the vote. 0:46:24.447 --> 0:46:25.727 And then and that's the priority. 0:46:25.767 --> 0:46:28.167 And then the thing that the Google guy said, I'm 0:46:28.167 --> 0:46:31.047 going to say Google person or guy or whatever, but, uh, 0:46:31.047 --> 0:46:34.287 because I don't remember anybody's names, but, um, uh, he 0:46:34.287 --> 0:46:39.567 was saying that like, uh, so after you do that, um, uh, 0:46:39.927 --> 0:46:44.167 the whole vulnerability management scaled by AI only works if 0:46:44.167 --> 0:46:47.807 you have a really good asset management platform. And we've 0:46:47.807 --> 0:46:48.927 been talking about this for years. 0:46:48.967 --> 0:46:52.087 Like I did that thing in 23 when I was 0:46:52.087 --> 0:46:52.727 at Robinhood. 0:46:52.767 --> 0:46:53.167 Yeah, yeah, yeah. 0:46:53.327 --> 0:46:55.927 Asset management as a center of management. 0:46:55.967 --> 0:46:58.607 Yes, exactly. Yeah, yeah. And that you presented that at 0:46:58.647 --> 0:47:01.806 the Black Hat Summit, right? Yeah. So yeah. And this 0:47:01.807 --> 0:47:03.567 turned out to be true for them. Right. It's like 0:47:03.607 --> 0:47:06.487 it's like this program does not work unless you have, 0:47:06.687 --> 0:47:10.117 you know, had your balanced breakfast of, um, you know, 0:47:10.157 --> 0:47:13.437 consolidating all your data sources where vulnerabilities come in, and 0:47:13.437 --> 0:47:17.837 then having a tremendous asset management program, like knowing where, 0:47:17.877 --> 0:47:20.797 like having, um, you know, for lack of a better word, 0:47:20.837 --> 0:47:24.477 like having knowledge of where all the systems are, you know, 0:47:24.517 --> 0:47:28.277 what they are, who owns them, what teams action them, um, 0:47:28.277 --> 0:47:31.357 where the repos are. And that's not like that. Sounds 0:47:31.357 --> 0:47:33.117 trivial to some companies who are small. Like if you're 0:47:33.117 --> 0:47:34.437 a startup, you're like, yeah, of course I know where 0:47:34.437 --> 0:47:36.277 my repos are and who owns the thing. But when 0:47:36.277 --> 0:47:38.837 you get to a company, that's the scale of Google 0:47:38.957 --> 0:47:41.997 or Apple, right? There are hundreds, if not thousands like 0:47:42.237 --> 0:47:45.157 Ubisoft too. I mean, we had productions which are video 0:47:45.197 --> 0:47:49.677 games everywhere, and it's just not simple anymore. Like you, 0:47:50.197 --> 0:47:52.197 you lose a thread on an app and then it 0:47:52.197 --> 0:47:54.397 just exists out in the wild, and then someone finds 0:47:54.397 --> 0:47:56.117 it via your bug bounty and they're like, hey, I 0:47:56.237 --> 0:47:58.397 read this thing and you're like, I have no idea 0:47:58.397 --> 0:48:00.797 what that is like. It's not. I don't see it anywhere. 0:48:00.797 --> 0:48:02.517 I don't know who owns it. And then you spend 0:48:02.517 --> 0:48:06.757 all of this toil time. That was a reoccurring term toil, right? Oh, yeah. 0:48:07.267 --> 0:48:10.987 You spend all this toil time to, like, figure that out. And, um, 0:48:11.147 --> 0:48:14.107 you should be architecting your program from the beginning with 0:48:14.107 --> 0:48:17.187 really good asset management instead of spending that toil time later. 0:48:17.227 --> 0:48:19.786 Yeah, I love that. I definitely dealt with that at 0:48:19.787 --> 0:48:21.587 Apple because it's like you put out the state of 0:48:21.587 --> 0:48:25.267 the system and a week later it's like, nothing like that. Yeah, yeah. 0:48:25.307 --> 0:48:26.107 That's true. Like that. 0:48:26.147 --> 0:48:28.147 Yeah. Yeah. The question I had for you when we 0:48:28.147 --> 0:48:31.787 were in the car was it's like, okay, so, um, 0:48:31.907 --> 0:48:34.467 so you're really big on capturing context in markdown files, 0:48:34.467 --> 0:48:37.067 which is the telos idea. Mhm. Um, and you can 0:48:37.067 --> 0:48:38.947 do telos for yourself personally as a person. Or you 0:48:38.947 --> 0:48:42.907 could telos as an organization. Right. City country country doesn't matter. Yeah. Yeah. 0:48:42.947 --> 0:48:46.467 Put down your ethos, your goals, your systems, your owners, 0:48:46.507 --> 0:48:48.747 you know, into markdown or maybe JSON or something like that, 0:48:48.747 --> 0:48:49.467 whatever you want to use. 0:48:49.547 --> 0:48:49.987 Whatever. 0:48:50.107 --> 0:48:52.267 Uh, and my question was kind of like, okay, so, 0:48:52.587 --> 0:48:57.147 so Bob, who handles, you know, Celsius app, right. Whatever. Uh, 0:48:57.267 --> 0:48:59.747 you know, he leaves and like, who is responsible for 0:48:59.747 --> 0:49:03.187 updating the context file to include those changes, you know, 0:49:03.187 --> 0:49:06.057 is there a system that you prefer or like a method? 0:49:06.057 --> 0:49:07.897 Or is it just that you have to go in 0:49:07.897 --> 0:49:09.737 there and help them and update that? Or is there like, 0:49:09.937 --> 0:49:11.697 do they have to hire a specific person to run 0:49:11.697 --> 0:49:14.617 the Telos file and make sure everything stays in line 0:49:14.617 --> 0:49:16.817 when they change their company vision or something like that? 0:49:16.857 --> 0:49:20.537 Yeah, yeah. So it's going to depend on the implementation. 0:49:20.537 --> 0:49:22.737 So the way I'm doing it commercially is for this 0:49:22.737 --> 0:49:25.377 thing called same page, which I'll be talking about later. 0:49:25.377 --> 0:49:28.617 But I think the future of this is this. 0:49:28.657 --> 0:49:30.337 Let's talk about it. Let's talk about same page right now. 0:49:30.337 --> 0:49:32.537 Let's heat up. I mean, you might as well. Right? 0:49:33.777 --> 0:49:34.337 Come on. 0:49:34.537 --> 0:49:34.977 No. 0:49:35.017 --> 0:49:36.057 Oh come on. 0:49:36.257 --> 0:49:38.977 No, I mean, it's it's the concept of the talk. 0:49:38.977 --> 0:49:44.017 It's just unified context. Now, I appreciate it, but no, um, um, 0:49:44.777 --> 0:49:48.817 but no, I think just this just becomes a unified, um, 0:49:49.177 --> 0:49:50.937 it product. Okay. 0:49:51.177 --> 0:49:52.017 So it maintains. 0:49:52.017 --> 0:49:56.617 It. So, so I think that anybody who builds anything from, like, 0:49:57.097 --> 0:50:01.177 an ice cream truck business to a security program to 0:50:01.177 --> 0:50:04.217 I want to be a governor, they're going to have 0:50:05.377 --> 0:50:12.217 a core system, which is all assets, all context, all goals, everything. 0:50:12.457 --> 0:50:15.097 So one thing that I didn't realize until starting to 0:50:15.137 --> 0:50:18.697 build very, uh. And I'm not afraid to admit it, 0:50:18.697 --> 0:50:21.817 vibe coded things. Right. Like so I will now have 0:50:21.817 --> 0:50:25.897 superpowers because I understand code, right? I understand architecture of code. 0:50:25.897 --> 0:50:28.457 I understand problems in code, I understand security, but I've 0:50:28.457 --> 0:50:30.697 never been a front end developer. I couldn't sit down 0:50:30.697 --> 0:50:33.217 with react and build a build a pretty website. If 0:50:33.217 --> 0:50:36.137 you put me on a modern development team, I would die. Yeah. 0:50:36.177 --> 0:50:37.057 Same. Same. 0:50:37.097 --> 0:50:40.537 But because I know about code, I know how to script. 0:50:40.537 --> 0:50:43.177 And I know the concept of pretty much every language 0:50:43.177 --> 0:50:46.577 from assessing it in security. I can now build fantastic 0:50:46.577 --> 0:50:49.137 things very quickly. Same. The thing is, is that I 0:50:49.137 --> 0:50:51.897 think I've realized in that world is that prds like 0:50:51.937 --> 0:50:57.217 product requirements documents are necessary for so many more things 0:50:57.457 --> 0:51:01.137 than I ever thought. Right. Like. And so the idea, 0:51:01.177 --> 0:51:03.127 the way it connects to vibe Coding is like whenever 0:51:03.127 --> 0:51:04.927 I do a new project now, the first thing I 0:51:04.927 --> 0:51:08.247 do is I verbally talk to my browser with a 0:51:08.247 --> 0:51:11.207 Chrome extension into an AI model with a whole bunch 0:51:11.207 --> 0:51:13.287 of notes about just kind of what I want the 0:51:13.287 --> 0:51:15.847 system to do, what tools it's tying together, like how 0:51:15.847 --> 0:51:19.007 it's presenting data, why we're even making this, what problem 0:51:19.007 --> 0:51:21.687 it solves. And that's just verbal garbage coming out of 0:51:21.687 --> 0:51:23.407 my mouth, right? Like I'm just having a conversation. I 0:51:23.407 --> 0:51:25.687 could take a podcast like this and like, do that. 0:51:25.687 --> 0:51:27.567 And then I'm feeding into a whole bunch of AIS 0:51:27.607 --> 0:51:31.687 to make a product document with requirements in it. And 0:51:31.687 --> 0:51:34.647 then I'm creating a technical architecture document as well, which 0:51:34.647 --> 0:51:37.687 is why we are choosing the frameworks that we're using, 0:51:37.687 --> 0:51:40.767 why we're choosing the tools. Never deviate from these. And 0:51:40.767 --> 0:51:43.367 so those two things in concert, especially in the vibe 0:51:43.407 --> 0:51:46.607 coding or AI assisted coding world, have helped make my 0:51:46.607 --> 0:51:49.567 software infinitely better and helped the AI. I'd stay on 0:51:49.567 --> 0:51:52.087 track with the mission. With the technology. 0:51:52.087 --> 0:51:54.727 Yeah, because when it loses its context and loses its 0:51:54.727 --> 0:51:57.967 mind and basically gets erased. Yeah, it just goes back 0:51:57.967 --> 0:51:59.007 to that. Starts over. 0:51:59.047 --> 0:52:00.847 Yeah. And in vibe code, you can have such I 0:52:00.847 --> 0:52:02.797 mean we're going off on a rathole now, but I. 0:52:02.797 --> 0:52:03.517 Mean, that's fine. 0:52:03.557 --> 0:52:05.997 In coding, you have that sidebar, right? And that you 0:52:05.997 --> 0:52:08.077 can stay in that conversation for a long time and 0:52:08.277 --> 0:52:11.077 not realize that you're you're hitting the point of where 0:52:11.117 --> 0:52:14.197 needle in the haystack is not you're not getting good value. 0:52:14.197 --> 0:52:14.957 And you need to. 0:52:14.997 --> 0:52:17.317 And they're also like $15 queries. 0:52:17.357 --> 0:52:18.917 Yeah, exactly. Yeah. 0:52:19.277 --> 0:52:22.557 $15, $24. Okay. Wait a minute. So if I do 0:52:22.597 --> 0:52:24.077 four of these, that's 100 bucks. 0:52:24.117 --> 0:52:25.917 Yeah. Yeah. I mean, I was talking about it with 0:52:25.917 --> 0:52:29.357 some friends, uh, in discord, and it's like or in, 0:52:29.557 --> 0:52:30.797 in a signal chat, and I'm like, how much are 0:52:30.797 --> 0:52:33.277 you guys spending on your AI subscriptions a month? Because 0:52:33.277 --> 0:52:36.517 mine is approaching a car payment, and it's totally. I 0:52:36.517 --> 0:52:38.557 know it's worth it, but it's still kind of painful 0:52:38.557 --> 0:52:40.677 to add another car payment, you know? So, uh, I mean, 0:52:40.677 --> 0:52:44.357 I'm using everything. I'm using Gemini, I'm using OpenAI's ecosystem, 0:52:44.357 --> 0:52:47.797 I'm using Claude's ecosystem. I still have perplexity. I'm a 0:52:47.797 --> 0:52:51.917 really hype user of Manus right now. Um, I love 0:52:51.957 --> 0:52:55.397 Lambda Chat's implementation of deep seq because they host it 0:52:55.397 --> 0:52:59.517 on the internet, and I can scrape it with puppeteer. Playwright. Um, so, yeah, 0:52:59.587 --> 0:53:01.507 I'm just hitting everything for everything. 0:53:01.547 --> 0:53:04.187 Yeah, yeah, I'm doing a lot of na to n, uh, 0:53:04.187 --> 0:53:09.347 for back end and, uh, bedrock, um, still use fabric 0:53:09.387 --> 0:53:11.267 like most of the models. Um. 0:53:11.267 --> 0:53:14.067 Oh, yeah. When I'm on the command line using fabrics. Yeah, yeah. 0:53:14.107 --> 0:53:18.427 Favorite models. Uh, right now, uh, uh, for me, it's, um, 0:53:18.467 --> 0:53:21.467 two five, two five pro for a Gemini. 0:53:21.587 --> 0:53:22.627 Oh, yeah. Gemini is on. 0:53:22.747 --> 0:53:24.147 And then, um, O3. 0:53:24.307 --> 0:53:24.707 Okay. 0:53:24.747 --> 0:53:25.387 With memory. 0:53:25.427 --> 0:53:29.067 With memory. Okay. So I use O3 with memory for 0:53:29.187 --> 0:53:31.387 writing tasks. I think that's really good at writing and 0:53:31.387 --> 0:53:35.027 researching tasks. Um, I actually am one of the believers 0:53:35.027 --> 0:53:37.227 that the biggest deep sea model. It's one of the 0:53:37.227 --> 0:53:40.227 best models that I've ever seen for research tasks, even 0:53:40.227 --> 0:53:42.867 though it doesn't have search enabled. Um, I use it, 0:53:42.867 --> 0:53:46.347 it's exposed for free through Lambda Chat. Um, so you 0:53:46.347 --> 0:53:48.227 can go and so like you have to think about 0:53:48.227 --> 0:53:52.587 there are different releases of R1, deep R1, and most 0:53:52.587 --> 0:53:56.347 of us played with the middle implementations. Uh like the. 0:53:56.627 --> 0:53:56.907 Yeah. 0:53:56.947 --> 0:53:57.427 Yeah. Like the. 0:53:57.587 --> 0:53:57.907 Not the. 0:53:57.937 --> 0:54:01.377 Full, not the full. Six one. 7 billion parameter one. Right. 0:54:01.417 --> 0:54:05.017 They have that hosted on their on their architecture for free. Yeah. 0:54:05.057 --> 0:54:08.497 And it is fantastic to use um, and uh, I 0:54:08.537 --> 0:54:10.697 find that model to be really, really good. 0:54:10.817 --> 0:54:12.217 Let me check real quick. Yeah. 0:54:12.217 --> 0:54:14.297 Let's see what grok is running. I don't think they're 0:54:14.297 --> 0:54:21.777 even running 617. I think he just pulled down right 0:54:21.777 --> 0:54:24.017 there on the compound base mini where it says right there. 0:54:25.017 --> 0:54:25.377 Yeah. 0:54:25.617 --> 0:54:25.937 Oh. 0:54:28.817 --> 0:54:29.417 Oh, here we go. 0:54:29.457 --> 0:54:34.177 Yeah, yeah. See, so they're doing deep seek 77 DB. 0:54:34.337 --> 0:54:35.297 No comparison. 0:54:35.337 --> 0:54:35.536 Yeah. 0:54:35.577 --> 0:54:35.897 No. 0:54:36.057 --> 0:54:41.377 Yeah 601 billion parameters. So um yeah. So I use 0:54:41.377 --> 0:54:43.577 that model. It it's really good research model but it 0:54:43.577 --> 0:54:46.977 doesn't have search enabled. Right. So that's, it's, it's uh 0:54:47.017 --> 0:54:52.497 missing and then so Gemini uh, deep 6617 and then 0:54:52.497 --> 0:54:56.887 I really get a lot out of, um, uh, when 0:54:56.887 --> 0:55:01.087 I code, you know, 3.7 on Claude is really good. Um, 0:55:01.087 --> 0:55:03.407 although it started to go kind of haywire recently, I. 0:55:03.687 --> 0:55:07.207 I switched off. I, I went from 3.5 to 3.7, 0:55:07.207 --> 0:55:10.567 and I liked it a lot, but 2.5 from Gemini 0:55:10.567 --> 0:55:11.447 came right after that. 0:55:11.487 --> 0:55:11.767 It did. 0:55:11.767 --> 0:55:14.047 Yeah. And I was just like, damn, that's really good. 0:55:14.087 --> 0:55:17.407 And then then oh three roughly same time. Yeah. Yeah. 0:55:17.407 --> 0:55:19.767 So I've kind of been messing mostly with those. 0:55:19.807 --> 0:55:23.527 That's the problem is like I want to consolidate subscriptions. 0:55:23.767 --> 0:55:25.407 Like I want to just say, oh, I'm just going 0:55:25.447 --> 0:55:28.047 to stick with this, but I can't, because if you're 0:55:28.047 --> 0:55:30.607 at the cutting edge of using this stuff every day, 0:55:30.607 --> 0:55:33.247 it's like you want the best model for the best 0:55:33.247 --> 0:55:33.847 thing all the time. 0:55:33.887 --> 0:55:36.367 And there's like two releases per week. So you're just like, 0:55:36.407 --> 0:55:36.767 it's not. 0:55:37.087 --> 0:55:39.487 Yeah, yeah, yeah. And so yeah, you end up spending 0:55:39.847 --> 0:55:42.447 somewhere in the order of 400 to $600 a month 0:55:42.447 --> 0:55:46.087 for all your subscriptions, you know, not including your I mean, 0:55:46.087 --> 0:55:49.767 that's including everything that's including your, um, your direct chat 0:55:49.767 --> 0:55:53.207 interfaces that you're paying for, like ChatGPT implementations, but also 0:55:53.487 --> 0:55:58.197 your API calls and then also your subscription to Klein 0:55:58.197 --> 0:56:01.957 or not Klein, but windsurf or um, um, what's the 0:56:01.957 --> 0:56:04.237 other one? I use windsurf, sorry. I always forget the 0:56:04.237 --> 0:56:05.917 name of the other one. Sea. 0:56:06.197 --> 0:56:06.717 Klein. 0:56:06.997 --> 0:56:11.757 No. Oh, yeah. Klein. But. Cursor, cursor. There you go. Cursor. Yeah. Um, 0:56:12.037 --> 0:56:14.117 so you have subscription to one of those two, and 0:56:14.117 --> 0:56:16.957 it's like, okay. Yeah. Now I'm paying quite a bit. 0:56:16.997 --> 0:56:22.237 I just realized, uh, a link between coding and the, um, 0:56:22.357 --> 0:56:28.277 AI competition. Mhm. Which model are you using versus which 0:56:28.277 --> 0:56:31.637 scaffolding do you have in the form of giving that 0:56:31.677 --> 0:56:33.117 cursor rules prompt. 0:56:33.277 --> 0:56:33.517 Yeah. 0:56:33.517 --> 0:56:38.117 Yeah. Right. So it's like I strongly believe and I 0:56:38.157 --> 0:56:42.037 Jason talks about this I forget his name is Jason Wong. 0:56:42.357 --> 0:56:43.237 Is that his name I don't. 0:56:43.237 --> 0:56:44.357 Know but he's brilliant I love that guy. 0:56:44.677 --> 0:56:49.157 Yeah. So he makes like really, really practical videos. Yeah. Um, 0:56:49.157 --> 0:56:53.517 so he basically, um, put out some stuff. Cursor rules, 0:56:53.827 --> 0:56:57.027 like a single prompt that can generate that full PRD 0:56:57.067 --> 0:57:00.147 with a checklist. Yeah. And the model actually goes and 0:57:00.147 --> 0:57:01.787 checks things off the list. 0:57:01.827 --> 0:57:04.827 Yeah, I do the same way. So I use Berman's 0:57:04.827 --> 0:57:07.747 or uh, um, the I got is it Berman I 0:57:07.747 --> 0:57:11.187 can't remember. What is his last name. The AI content creator. 0:57:11.227 --> 0:57:14.107 You were on his show. He talked about fabric. Matt. 0:57:14.147 --> 0:57:14.507 Matthew. 0:57:14.787 --> 0:57:17.787 Matthew. Yeah. Okay, so. So, Matt Berman did a or. 0:57:17.787 --> 0:57:20.587 Matthew Berman did a show on this as well? Yep. 0:57:20.627 --> 0:57:22.947 And he had a set of questions that he put 0:57:22.947 --> 0:57:24.747 in the show. But he never like released a GitHub 0:57:24.747 --> 0:57:25.827 or a gist or anything like that. 0:57:26.227 --> 0:57:29.027 That's the thing I love about AI, JSON. He's like boom, 0:57:29.027 --> 0:57:29.507 it's on GitHub. 0:57:29.507 --> 0:57:31.467 Yeah, it's on GitHub. Right. So I went and grabbed 0:57:31.467 --> 0:57:34.827 Matt Berman's and I tweeted, I just tweeted it out 0:57:34.827 --> 0:57:38.147 like I had. I transcribe and then pull up the questions. 0:57:38.267 --> 0:57:41.027 And then now I in that prompt to the AI 0:57:41.067 --> 0:57:43.707 I'm like, here is a, here is the structure of 0:57:43.707 --> 0:57:45.667 a PRD I want to build. I need you to 0:57:45.667 --> 0:57:47.907 ask me relevant questions to fill in the thing. And 0:57:47.907 --> 0:57:50.427 then I have it ask me the questions and then we're, 0:57:50.427 --> 0:57:51.507 you know, we have a chat. 0:57:51.907 --> 0:57:54.217 So it builds a PRD. Based on your interview? 0:57:54.217 --> 0:57:56.057 Based on my interview, it interviews, I tell it to 0:57:56.057 --> 0:57:56.577 interview me. 0:57:56.617 --> 0:57:56.817 Yeah. 0:57:57.057 --> 0:57:59.657 And then once we're done, then it's like, cool. Do 0:57:59.657 --> 0:58:01.377 you want me to stitch all this together into a 0:58:01.377 --> 0:58:03.177 fully functional PRD? I'm like, yes. And then we move 0:58:03.177 --> 0:58:05.497 on to the architecture section, and then those live as 0:58:05.497 --> 0:58:08.537 markdown files in my project. And so when I need 0:58:08.537 --> 0:58:10.697 to start a new chat, because I know the context 0:58:10.697 --> 0:58:14.177 window is filling up for windsurf, I start a new 0:58:14.177 --> 0:58:16.817 chat and I say reanalyze our, you know, our core, 0:58:17.257 --> 0:58:20.537 our core architecture and our PRD so that you understand 0:58:20.537 --> 0:58:22.737 everything about this project and the readme that we built. Yeah, yeah. 0:58:22.737 --> 0:58:25.257 And then it's like it's like I'm starting again with 0:58:25.257 --> 0:58:27.697 great context. And you know, obviously you need to hook 0:58:27.697 --> 0:58:30.177 it up to version control as well. So if anything 0:58:30.177 --> 0:58:33.537 goes haywire gets deleted, you can snapshot back. But um, 0:58:33.577 --> 0:58:36.097 but those are some pro tips for the coding people 0:58:36.097 --> 0:58:36.617 out there. 0:58:36.657 --> 0:58:40.817 Yeah, yeah I've got a a rule where I basically 0:58:40.817 --> 0:58:43.257 say in the cursor rules. If I say this, it 0:58:43.257 --> 0:58:44.537 means go and review. 0:58:44.817 --> 0:58:45.417 Okay, cool. 0:58:45.457 --> 0:58:47.937 So if I feel like it's going wonky. Yeah, it's 0:58:47.937 --> 0:58:50.537 just like a reset. Yeah. Um, I wanted to mention 0:58:50.537 --> 0:58:50.857 one thing. 0:58:50.857 --> 0:58:52.857 You don't say it like, make it so. Like Picard. 0:58:53.137 --> 0:58:57.737 You could do that. Yeah, absolutely. Um, so I wanted 0:58:57.737 --> 0:59:01.017 to give, uh, a little shout out to, uh, Caleb 0:59:01.057 --> 0:59:05.857 Sima for a post he did before RSA, where he 0:59:05.857 --> 0:59:13.417 was complaining about, um, panels, just like being kind of empty. Yeah. Um, 0:59:13.457 --> 0:59:15.897 and I was just thinking about this. The whole reason, 0:59:15.897 --> 0:59:18.777 these long form conversations. Because we're just riffing. Yeah. And 0:59:18.777 --> 0:59:21.937 we're just thinking of stuff. But ideally, the stuff we're 0:59:21.937 --> 0:59:25.617 thinking of is stuff that was new to us. Therefore, 0:59:25.617 --> 0:59:28.177 it's going to be new to them. Yeah. And what 0:59:28.177 --> 0:59:31.617 happens with so many talks and so many panels? And 0:59:31.617 --> 0:59:33.577 I think this is like a meta conversation that we 0:59:33.577 --> 0:59:36.857 just really need to solve. Um, obviously for us as 0:59:36.857 --> 0:59:39.417 content creators, which I think we do a good job. Yeah. 0:59:39.417 --> 0:59:41.657 But I would say the industry needs to solve this 0:59:41.857 --> 0:59:44.537 is don't just come on and be like, you know, 0:59:44.657 --> 0:59:51.807 here's the thing, Jason. Like, the landscape is changing. You know. Yeah. 0:59:51.847 --> 0:59:54.767 You know, AI is changing everything. It's changing the game. Yeah. 0:59:54.767 --> 0:59:57.127 And you're like, yeah, it's just changing the game. We're like, 0:59:57.127 --> 0:59:58.727 all right, that's all the time we got. Yeah. 0:59:58.767 --> 0:59:59.887 Everybody let's go to lunch. 0:59:59.927 --> 1:00:04.287 Yeah. So it's like two people reflecting backwards or maybe 1:00:04.287 --> 1:00:07.047 a panel of four. Yeah. And they're just reflecting back 1:00:07.047 --> 1:00:09.407 these things that we've heard for the last few months 1:00:09.407 --> 1:00:14.527 or a couple of years. Yeah. It's like, um, was 1:00:14.527 --> 1:00:17.327 it Matt or somebody we know was like, if I 1:00:17.367 --> 1:00:21.487 hear one more person say, um, the attacker only has 1:00:21.487 --> 1:00:24.247 to be right once, and the defender has to be 1:00:24.247 --> 1:00:26.527 right all the time. It's like we learned this ten 1:00:26.527 --> 1:00:28.607 years ago. Yeah. So the first time we heard it, 1:00:28.647 --> 1:00:29.647 it was hella smart. Yeah. 1:00:29.927 --> 1:00:30.447 Yeah. It was great. 1:00:30.447 --> 1:00:33.127 Yeah. And now it's just like so many panels, so 1:00:33.127 --> 1:00:35.487 many conferences, so many talks are just that. 1:00:35.527 --> 1:00:39.607 Yeah, yeah. It's, uh, it's that it's that there's no 1:00:39.607 --> 1:00:43.207 contention anymore on panels like. Yeah. I mean, even the 1:00:43.207 --> 1:00:45.327 ones at OpenAI, when we went to that, there was 1:00:45.367 --> 1:00:47.247 like towards the end, it was a lot of like, yeah, 1:00:47.287 --> 1:00:49.517 yeah we agree. And so I think at the end 1:00:49.557 --> 1:00:50.997 I was like trying to spice it up a little bit. 1:00:51.037 --> 1:00:52.757 Like I wasn't the moderator but I asked a question. 1:00:52.797 --> 1:00:54.917 I'm like, who's your most feared competitor? 1:00:54.957 --> 1:00:55.237 That was. 1:00:55.237 --> 1:00:55.597 Like one. 1:00:55.597 --> 1:00:55.797 Of the. 1:00:55.797 --> 1:00:58.677 Best questions. And everybody was like, oh shit. And then like, 1:00:58.717 --> 1:01:00.957 you know, the, you know, like, uh, the guy at 1:01:00.957 --> 1:01:01.437 the end, that. 1:01:01.437 --> 1:01:02.437 Was one of the best questions. 1:01:02.437 --> 1:01:04.877 It's definitely Dan and Trail of Bits and Trail of 1:01:04.917 --> 1:01:07.357 Bits is like, oh, we're really scared of like whoever 1:01:07.717 --> 1:01:09.117 at the end, he's like, he's like, I don't know, 1:01:09.117 --> 1:01:10.717 I'm just doing my best over here. 1:01:10.757 --> 1:01:12.037 Yeah. Trying to survive. Yeah. 1:01:12.077 --> 1:01:14.837 Try and survive. And, uh, yeah, I wish there was 1:01:14.837 --> 1:01:17.037 a little bit more contentious. And you can you can 1:01:17.037 --> 1:01:19.357 talk about that. I've had interview panelists before or the 1:01:19.397 --> 1:01:22.117 people leading whatever moderators or whatever. Yeah, I've had them 1:01:22.157 --> 1:01:24.117 be like, hey, I'm going to ask this question. And 1:01:24.117 --> 1:01:25.877 if you all answer the same fucking thing, I'm gonna 1:01:25.877 --> 1:01:28.317 put you off the panel. And I love that. I'm like, yeah, yeah. 1:01:28.357 --> 1:01:32.437 So another of our mutual friends actually engineers this into 1:01:32.437 --> 1:01:34.117 the thing. Oh, really? Sasha. 1:01:34.157 --> 1:01:35.237 Oh, yeah. He does. Yeah. 1:01:35.277 --> 1:01:37.597 Sasha is like, look, if I'm going to be on this. 1:01:37.597 --> 1:01:39.437 I didn't get to see Sasha for more than ten minutes. 1:01:39.437 --> 1:01:39.717 This whole. 1:01:39.757 --> 1:01:43.237 Me neither. I just yeah, just like he was a blur. 1:01:43.557 --> 1:01:43.917 A blur. 1:01:43.917 --> 1:01:46.557 Yeah, yeah. But, like, he's like, if I sit on 1:01:46.627 --> 1:01:49.867 this panel, I'm going to engineer a thing. I need 1:01:49.867 --> 1:01:51.667 to know what you all believe. Yeah, so I can 1:01:51.667 --> 1:01:54.307 find something I disagree with. Yeah. Otherwise, this is going 1:01:54.307 --> 1:01:55.547 to be the dumbest panel ever. 1:01:55.587 --> 1:01:58.307 And it and it works on me wonderfully because like, 1:01:58.307 --> 1:02:00.747 when people don't agree with me, I just get angry. Yeah. 1:02:01.387 --> 1:02:02.947 No kidding. Like I'm like, I'm like. 1:02:02.987 --> 1:02:03.467 No, you're. 1:02:03.467 --> 1:02:06.387 Wrong. Like, we're good at this. We're good. We're good 1:02:06.387 --> 1:02:07.707 at having a good, you know? 1:02:07.707 --> 1:02:08.627 Yeah, yeah. No, I. 1:02:08.627 --> 1:02:09.387 Was having a go. 1:02:09.587 --> 1:02:12.347 I was, uh, I mean, while doing RSA, I was 1:02:12.347 --> 1:02:15.267 tweeting a little bit, um, on on Twitter. And I 1:02:15.307 --> 1:02:18.507 happen to have, uh, a friend who was rooming with 1:02:18.507 --> 1:02:21.827 me who was doing security research. So he's he found 1:02:21.827 --> 1:02:24.347 a zero day, basically reported it to the vendor, but 1:02:24.347 --> 1:02:28.707 a whole bunch of individual companies that implemented the software. And, um, 1:02:28.707 --> 1:02:31.067 they have not yet fixed the bug. So he goes 1:02:31.067 --> 1:02:33.027 out to all these bug bounty programs and submits it, 1:02:33.027 --> 1:02:36.307 and also a whole bunch of vulnerability disclosure programs. 1:02:36.347 --> 1:02:36.747 Mhm. 1:02:37.227 --> 1:02:40.707 And um, and he submits it to the vulnerability disclosure programs. 1:02:40.707 --> 1:02:42.547 And he comes back to me because, you know, I'm 1:02:42.547 --> 1:02:44.867 a former bug bounty guy. Right. And he's like he's like, yo, 1:02:44.907 --> 1:02:47.337 is it like, you know, like normal? Like the crits 1:02:47.337 --> 1:02:49.617 that I got for people who have bounties? I got 1:02:49.617 --> 1:02:52.177 50 points on the platform for hacker one, but I 1:02:52.177 --> 1:02:55.777 only get seven for responsibly disclosing it without getting paid. 1:02:55.897 --> 1:02:57.817 He's like, that's weird. So I go and tweet about it. 1:02:57.817 --> 1:03:01.177 And admittedly, I didn't tweet right. I don't think like 1:03:01.177 --> 1:03:03.097 I definitely went out and kind of sensationalized it a 1:03:03.097 --> 1:03:05.137 little bit. I'm like, this sucks. Like, give the guy 1:03:05.177 --> 1:03:07.857 more than seven points, right? But the bug bounty community 1:03:07.857 --> 1:03:10.257 of which are a lot of my homies, like, basically 1:03:10.257 --> 1:03:12.657 jumped on me and and it like got in my 1:03:12.657 --> 1:03:14.337 head a little bit. But then I managed to like 1:03:14.377 --> 1:03:16.377 push it down a little bit, but like, you know, 1:03:16.417 --> 1:03:18.137 the bug bounty. This is a huge thing this week. 1:03:18.137 --> 1:03:20.857 Huge discussion is just like, you know, VDP is evil, 1:03:20.897 --> 1:03:26.017 VDP is labor exploitation, VDP is everything wrong with bug bounty? Jason, 1:03:26.017 --> 1:03:29.417 you're the worst person in the world for um, for 1:03:29.457 --> 1:03:32.217 promoting that. Anyone ever even think about VDP. And so 1:03:32.217 --> 1:03:34.457 then I came out with this post, uh, you know, 1:03:34.497 --> 1:03:35.937 a guy was trying to debate this with me, and 1:03:35.937 --> 1:03:37.897 he's he's a smart dude, smart bug hunter. I have 1:03:37.897 --> 1:03:41.377 respect for him. But he, like, said one sentence and 1:03:41.377 --> 1:03:42.977 it was it felt a little aggressive to me. And 1:03:42.977 --> 1:03:44.287 I was just in that mood and I was like, 1:03:44.327 --> 1:03:47.607 fuck it. Block like. And I just block this dude. And, um. 1:03:47.647 --> 1:03:50.007 And then he, like, went off the deep end and started, like, 1:03:50.047 --> 1:03:52.007 posting more and more and like, you know, I can't 1:03:52.007 --> 1:03:54.167 believe Jason did it. Turned out he had been to 1:03:54.207 --> 1:03:57.007 my class, to which I felt bad about. Oh, um, 1:03:57.207 --> 1:03:59.127 and so I unblocked and tried to have a conversation. 1:03:59.127 --> 1:04:02.287 But just like there's so much vitriol on this topic 1:04:02.487 --> 1:04:05.167 of labor exploitation. So I went out and I said, hey, 1:04:05.487 --> 1:04:08.127 I believe in VDP. Actually, I believe there are plenty 1:04:08.127 --> 1:04:12.207 of companies who on ramp with a VDP and then 1:04:12.207 --> 1:04:14.567 start paying for bugs. I have seen them. I have 1:04:14.567 --> 1:04:17.367 worked at Bugcrowd. This is not a fictional thing, right? Uh, 1:04:17.367 --> 1:04:19.287 some companies are too big that if they were just 1:04:19.287 --> 1:04:21.727 to open up. I mean, Ubisoft was one of these companies, right? 1:04:21.767 --> 1:04:23.327 If you would have opened up a bug bounty for 1:04:23.327 --> 1:04:27.607 everything on Ubisoft, all scope, all all classifications of bugs, 1:04:27.767 --> 1:04:30.087 they would have went bankrupt, right? Yeah. And so they 1:04:30.087 --> 1:04:32.487 had to start with the VDP to burn down a 1:04:32.487 --> 1:04:35.447 little bit of the stuff, incentivize people to come on 1:04:35.447 --> 1:04:37.687 and then eventually move into a paid program. Yeah. And 1:04:37.687 --> 1:04:40.367 then they ended up shutting down the program. Um, but 1:04:40.607 --> 1:04:42.917 a lot of companies work like that, and no one's 1:04:42.917 --> 1:04:45.077 forcing you to work on a PDP, right? Like, I wasn't, 1:04:45.077 --> 1:04:45.437 I wasn't. 1:04:45.837 --> 1:04:48.037 Can you for everyone just give an overview of the 1:04:48.037 --> 1:04:48.957 difference between the two? 1:04:48.997 --> 1:04:51.997 Yes. Okay. So a bug bounty program is a program 1:04:51.997 --> 1:04:54.077 where you pay for bugs, right? Like a researcher on 1:04:54.077 --> 1:04:55.837 the internet comes out and says, I found a web 1:04:55.837 --> 1:04:59.717 application vulnerability with your software and you've said, yes, I'm, 1:04:59.717 --> 1:05:02.237 I'm paying for these usually on a platform like Bugcrowd 1:05:02.237 --> 1:05:05.077 or Hackerone or integrity. And they come in the platform, 1:05:05.077 --> 1:05:07.557 they submit bug, you pay them, right. VDP is called 1:05:07.557 --> 1:05:11.077 a vulnerability disclosure program, where you as a company don't 1:05:11.077 --> 1:05:13.597 have like an email box or anything like that to 1:05:13.637 --> 1:05:17.357 take in vulnerabilities. And so usually the platform handles it 1:05:17.357 --> 1:05:20.277 for you and you show up on their platform as 1:05:20.277 --> 1:05:22.957 a card that says, hey, we do not pay for anything, 1:05:22.957 --> 1:05:26.477 but if you find something, report it to this program. Right. Yeah. Um, 1:05:26.517 --> 1:05:29.997 and we will, uh, give you props, you know, we will. 1:05:30.317 --> 1:05:32.237 Find some kind of reward, but we don't have a 1:05:32.237 --> 1:05:32.797 mountain of cash. 1:05:32.797 --> 1:05:34.357 Yeah, we don't have a mountain of cash. Right? And 1:05:34.357 --> 1:05:37.677 it's just. It feels like bug hunters are, you know? 1:05:38.357 --> 1:05:40.277 You know, this guy was trying to cite, like, he's like, 1:05:40.277 --> 1:05:43.547 no more free bugs thing. And like, you know, like, listen, man, 1:05:43.547 --> 1:05:45.507 I was there when that happened. I mean, I mean, 1:05:45.747 --> 1:05:47.347 I don't know if this guy was even born then, 1:05:47.347 --> 1:05:49.627 but he's citing this. But I was there when no 1:05:49.627 --> 1:05:51.627 more free bugs was happening. And I don't think that 1:05:51.627 --> 1:05:54.507 was the core message completely. I think it was I 1:05:54.507 --> 1:05:55.987 think it was that like, you know, there should be 1:05:55.987 --> 1:05:59.507 fairness in disclosure, there should be fairness in credit of vulnerabilities. 1:05:59.507 --> 1:06:02.387 There shouldn't be people suing each other when vulnerability research 1:06:02.427 --> 1:06:05.867 is discussed. Um, and so I wrote this big long 1:06:05.907 --> 1:06:08.787 thing and I just got railed this week like by the, 1:06:08.827 --> 1:06:10.827 by my own community. And I felt I felt really 1:06:10.827 --> 1:06:12.027 attacked and I was like, fuck it, I'm not going 1:06:12.027 --> 1:06:13.867 to do bug bounty stuff anymore. Like I'm hosting the 1:06:13.867 --> 1:06:16.147 Hong Kong and it's like, it's like you guys are 1:06:16.147 --> 1:06:18.267 some angry people. Like, if you don't want to work 1:06:18.267 --> 1:06:21.027 on Vdps don't work on Vdps, right. But like the 1:06:21.027 --> 1:06:23.787 other thing I didn't say in that thread was that, um, 1:06:24.067 --> 1:06:26.507 was that a lot of people I mentor, they don't 1:06:26.507 --> 1:06:28.347 have their foot in the door yet. They haven't had 1:06:28.347 --> 1:06:31.187 a job yet. Right. And so when they go to interview, 1:06:31.347 --> 1:06:33.867 if they have bug bounty experience, that's awesome. But usually 1:06:33.867 --> 1:06:36.187 when you're starting to find a bug on a bug, 1:06:36.187 --> 1:06:38.627 bounty is much harder than to find a real bug 1:06:38.627 --> 1:06:41.257 on a PDP, right? Because there's so much more scope. 1:06:41.497 --> 1:06:43.977 Not everybody is testing on the vdps there. See something? 1:06:44.017 --> 1:06:45.857 Say something. Right? So if you find something on a 1:06:45.857 --> 1:06:49.017 VDP and you're doing stuff like Portswigger Labs and Hack 1:06:49.057 --> 1:06:51.497 the box and all that stuff, it's like that shows 1:06:51.497 --> 1:06:53.497 me as an employer, well, they've got the skills, right. 1:06:53.537 --> 1:06:55.737 They have the skills. They just need a chance. Right? 1:06:55.937 --> 1:06:57.417 And it's receipts. 1:06:57.457 --> 1:06:57.857 It's like. 1:06:57.897 --> 1:06:58.057 It's. 1:06:58.057 --> 1:07:00.857 Receipts. Yeah. And, you know, I didn't even want to 1:07:00.857 --> 1:07:02.697 go into that part of the argument on Twitter because 1:07:02.977 --> 1:07:05.697 I'll get flamed more. But, um, yeah, I mean, it 1:07:05.697 --> 1:07:08.617 goes to show that like, uh, you know, like I 1:07:08.657 --> 1:07:10.897 drew that corollary from the panel thing you were talking about, right? 1:07:10.937 --> 1:07:14.177 It's like it's like, uh, you can't just agree with everyone. 1:07:14.177 --> 1:07:16.177 You have to have a point of view. Yeah. We're 1:07:16.177 --> 1:07:17.897 talking about this in content creation, right? 1:07:17.937 --> 1:07:18.777 Like, totally. 1:07:18.817 --> 1:07:20.817 So so here's this idea. And this is also off 1:07:20.817 --> 1:07:23.617 the beaten path. But, uh, for the listeners. But Dan 1:07:23.617 --> 1:07:25.977 and I are content creators, right? And, and we are 1:07:25.977 --> 1:07:29.097 lucky enough to have, you know, 25, 30 years in 1:07:29.097 --> 1:07:32.377 the industry, almost like, um, and so, you know, 20 1:07:32.377 --> 1:07:34.097 years for me. Um, and so. 1:07:34.537 --> 1:07:38.127 Can I stop you even before. Yeah. So I have 1:07:38.127 --> 1:07:39.687 like a pre point for this. 1:07:39.727 --> 1:07:40.167 Okay. Go ahead. 1:07:40.367 --> 1:07:43.367 Yeah. So well make sure you keep this thread because 1:07:43.367 --> 1:07:45.727 this is an amazing thread. So I want I want 1:07:45.767 --> 1:07:48.327 to make a quick point I can't believe this is 1:07:48.327 --> 1:07:50.927 going in this direction. But so a lot of young 1:07:50.927 --> 1:07:52.687 people come to us and they're like how do I 1:07:52.727 --> 1:07:55.407 become a YouTuber? Yeah. How do I become a content 1:07:55.407 --> 1:07:58.567 creator on Twitter or whatever? The first thing I tell them, 1:07:58.567 --> 1:08:02.327 which is exactly your point, learn something so that you 1:08:02.327 --> 1:08:07.087 have an opinion about it. Yeah. Because I don't I 1:08:07.487 --> 1:08:10.087 do create content and you create content. But I wouldn't 1:08:10.087 --> 1:08:12.567 say that fundamentally, that's what we are. There are a 1:08:12.567 --> 1:08:16.047 lot of people who are content creators, and I feel 1:08:16.046 --> 1:08:19.326 like that is their actual job. So they are looking 1:08:19.327 --> 1:08:24.006 for content whereas we are building and doing things. Mhm. 1:08:24.407 --> 1:08:26.847 You got a training company, you got a consulting company. 1:08:26.887 --> 1:08:29.727 You're doing that. The content falls out of it. 1:08:29.727 --> 1:08:30.406 Yes. Correct. 1:08:30.447 --> 1:08:32.527 Yeah. Right. And I feel like that is so key 1:08:32.567 --> 1:08:35.927 especially for young people. You have to like get good 1:08:35.927 --> 1:08:36.607 at something. 1:08:36.647 --> 1:08:38.847 Yeah, well, it's not even. It's not that you have 1:08:38.847 --> 1:08:40.407 to get good at something. It's that you have to 1:08:40.447 --> 1:08:41.287 do something. 1:08:41.847 --> 1:08:46.647 Have an opinion, have a path. Try something. Yeah. And 1:08:46.647 --> 1:08:48.327 the content emerges from that. 1:08:48.367 --> 1:08:51.207 Yeah. I mean, one thing. Uh, so, Clint, you know, 1:08:51.247 --> 1:08:53.927 our friend at Semgroup, he did an interview with me 1:08:53.927 --> 1:08:56.527 later this week on the Semgroup blog. That'll come out 1:08:56.527 --> 1:08:58.287 in a week or so, but it was mostly about 1:08:58.287 --> 1:09:01.607 career stuff. And what I was telling him is like, 1:09:01.767 --> 1:09:03.447 it's like when you're new, it's really hard to show 1:09:03.447 --> 1:09:05.487 that portfolio of work, you know, and you can do 1:09:05.487 --> 1:09:07.447 it with Vdps. You can do with Bug Bounty, you 1:09:07.447 --> 1:09:09.647 can do it with CVS. You can do it by 1:09:09.687 --> 1:09:12.367 taking tryhackme and getting certs and stuff like that. But 1:09:12.367 --> 1:09:14.447 some of that stuff is paid work, right? Or it's 1:09:14.447 --> 1:09:16.967 paid like some of that training is paid. And when 1:09:16.967 --> 1:09:18.767 you really have nothing, you're going to have to just 1:09:18.767 --> 1:09:21.847 focus on the free resources. But one of the things 1:09:21.847 --> 1:09:24.047 that you can do with nothing at all is start 1:09:24.047 --> 1:09:27.447 up a blog and talk about your learning experience. And, 1:09:27.687 --> 1:09:30.567 you know, you said, good, I'm going to challenge with you. 1:09:30.567 --> 1:09:33.447 Just have to do something. Yeah, do something and write 1:09:33.447 --> 1:09:35.797 about it and have an opinion and talk about your 1:09:35.797 --> 1:09:37.796 learning experience. And so like when I read a blog about. 1:09:37.837 --> 1:09:38.677 And be vulnerable. 1:09:38.797 --> 1:09:39.397 And be vulnerable. 1:09:39.397 --> 1:09:41.557 Yeah. Like you were saying earlier, talk about the bad, 1:09:41.597 --> 1:09:42.357 talk about the good. 1:09:42.397 --> 1:09:44.397 Yeah, yeah. And you know, so like when I see 1:09:44.397 --> 1:09:47.637 a blog on someone who's writing about their first, first 1:09:47.637 --> 1:09:50.317 usage of using Burp Suite, right? And it's like, yeah, 1:09:50.317 --> 1:09:52.397 I've read that blog 800 times, but I haven't read 1:09:52.397 --> 1:09:52.997 it from you. 1:09:53.037 --> 1:09:53.397 That's right. 1:09:53.437 --> 1:09:56.437 And you're talking about like, your thought process and like, 1:09:56.477 --> 1:09:58.437 you know, learning this tool or you're doing a YouTube 1:09:58.437 --> 1:10:00.797 video on it or something like that, I go look 1:10:00.797 --> 1:10:02.876 at that stuff in the interview pipeline. Right. It's an 1:10:02.877 --> 1:10:05.437 it's an additional thing in your portfolio. And it always 1:10:05.437 --> 1:10:08.277 makes me feel good to watch somebody trying to learn something, 1:10:08.317 --> 1:10:10.477 you know, and, and I feel like. So it's not 1:10:10.477 --> 1:10:11.957 that you have to be good at something. It's just 1:10:11.957 --> 1:10:14.756 you have to do something and and that becomes your content, 1:10:14.757 --> 1:10:15.037 I feel. 1:10:15.077 --> 1:10:17.197 Yeah. And don't worry about overlaps. 1:10:17.237 --> 1:10:17.796 Yeah. Everyone. 1:10:18.197 --> 1:10:21.277 Because Jason and I are so similar. Yeah. If we 1:10:21.277 --> 1:10:23.277 tried to make the same piece of content, we wouldn't. 1:10:23.317 --> 1:10:24.237 And then we get angry at each. 1:10:24.237 --> 1:10:27.597 Other because we have we, we have like the, we 1:10:27.597 --> 1:10:30.876 have very similar. But it's still going to come out different. Yeah. 1:10:30.917 --> 1:10:33.517 In a Jason way. In a Daniel way. Yeah. Yeah. 1:10:33.627 --> 1:10:36.586 And that. So you shouldn't have to worry that someone 1:10:36.587 --> 1:10:38.507 already wrote the burp tutorial. 1:10:38.547 --> 1:10:41.107 Yeah, yeah. Yeah, exactly. So the other one I was 1:10:41.107 --> 1:10:43.587 talking about is, um, I think I was, I was 1:10:43.587 --> 1:10:46.427 trying to explain is that, uh, is that in content 1:10:46.427 --> 1:10:49.627 creation also? I mean, like, I don't know if we 1:10:49.667 --> 1:10:51.467 call ourselves content creators or whatever. 1:10:51.507 --> 1:10:53.667 I mean, we definitely are, but it's not like my 1:10:53.667 --> 1:10:54.627 main identity. 1:10:55.027 --> 1:10:58.267 But I see so many people having like, being content 1:10:58.387 --> 1:11:01.267 creators for the the point of being a content creator 1:11:01.547 --> 1:11:05.347 and struggling to find content because they're not doing work. 1:11:05.427 --> 1:11:08.147 There you go. That's it. That's that's my point. Yeah. 1:11:08.187 --> 1:11:11.506 On top of that, not having an opinion. And I 1:11:11.507 --> 1:11:11.867 think that. 1:11:12.067 --> 1:11:12.787 That's like the worst. 1:11:12.827 --> 1:11:14.467 Yes, I think that that, you know, it goes into 1:11:14.467 --> 1:11:15.987 the panel thing we were talking about. Right. It's like 1:11:16.027 --> 1:11:18.547 you need in this industry, if you're going to be 1:11:18.547 --> 1:11:22.707 doing content in opinion of some sort, you need a voice. Um, 1:11:23.107 --> 1:11:25.227 and it doesn't have to agree with everyone. You could 1:11:25.227 --> 1:11:27.267 be the antithesis. Like if you want to be the 1:11:27.307 --> 1:11:29.947 anti Jay Haddock's guy and say that like, vdps are, 1:11:29.987 --> 1:11:31.947 you know, exploitative work and like, you know. 1:11:32.097 --> 1:11:32.977 At least you're saying something. 1:11:33.017 --> 1:11:34.777 At least you're saying something like, I respect that guy. 1:11:34.817 --> 1:11:37.336 You know, like, right. Like he has an opinion. Hurts 1:11:37.337 --> 1:11:39.217 my feelings a little bit, but he has an opinion, 1:11:39.257 --> 1:11:39.937 you know, and it's like. 1:11:39.977 --> 1:11:42.296 Especially, especially given what you've done for bounty. 1:11:42.337 --> 1:11:45.577 Yeah yeah yeah yeah, yeah. Um, but but yeah, having 1:11:45.617 --> 1:11:48.777 having an opinion is is almost more important than content 1:11:48.777 --> 1:11:51.456 these days, I think. I think content comes from your opinion. 1:11:51.497 --> 1:11:55.297 So 100% correct. Another way that gets described is taste. 1:11:55.337 --> 1:11:55.897 Yeah. 1:11:55.977 --> 1:11:58.497 Yeah. So so imagine this I talk about this a lot, 1:11:58.537 --> 1:12:02.217 but it's like a few years from now or maybe months. 1:12:02.217 --> 1:12:04.857 Who knows how fast this stuff is moving. But, um, 1:12:04.857 --> 1:12:07.697 you just like you do with, uh, narration of code, 1:12:07.697 --> 1:12:10.817 you're talking to cursor. And on all these screens, it's 1:12:10.817 --> 1:12:14.057 popping up. You mean like this? You mean like this? 1:12:14.417 --> 1:12:17.777 Or you're having it make you an anime series? Yeah. Or, 1:12:17.977 --> 1:12:20.017 you know, a book or whatever. Do you mean like this? 1:12:20.057 --> 1:12:22.256 And you're just swiping through? No, not like that. Not 1:12:22.257 --> 1:12:23.777 like that. That's the one I like. 1:12:23.777 --> 1:12:24.457 Yeah, that's the one I like. 1:12:24.497 --> 1:12:29.857 Yeah. And that taste. Oh, this reminds me of Rick Rubin, actually. Um, 1:12:29.857 --> 1:12:33.287 so he produced Slayer, a whole bunch of rap albums. 1:12:33.287 --> 1:12:36.447 He's like the most famous producer ever. Okay. Doesn't play 1:12:36.447 --> 1:12:41.047 an instrument. Really can't read music. Plays no instruments. 1:12:41.087 --> 1:12:41.487 Wow. 1:12:41.967 --> 1:12:44.687 And they say, how do you get hired to do this? 1:12:44.687 --> 1:12:46.647 He's like, I know exactly what I like. 1:12:47.527 --> 1:12:48.447 That's amazing. Right? 1:12:48.487 --> 1:12:51.247 Yeah. So to your point, like opinion taste. 1:12:51.287 --> 1:12:53.087 Yeah. I mean, that was we were talking in the 1:12:53.087 --> 1:12:56.887 car about the branding for Arcanum for my thing. And 1:12:57.167 --> 1:13:00.047 one of the most freeing things for me, being the 1:13:00.047 --> 1:13:04.487 CEO of this company now is I have complete control 1:13:04.767 --> 1:13:08.807 over branding, marketing message what we do and do not 1:13:08.807 --> 1:13:12.247 do to enable those things. Right? Like our strategy. I'll 1:13:12.247 --> 1:13:14.567 give you an example. It's a freebie for any company 1:13:14.567 --> 1:13:18.607 out there. Um, but, uh, where all of your marketing 1:13:18.647 --> 1:13:21.727 teams are telling you, like, you should be, you know, chasing, 1:13:22.007 --> 1:13:24.287 you know, inbound leads, and you're going to use BDR 1:13:24.327 --> 1:13:27.927 based marketing and stuff like that. That's that's all dead like, uh, 1:13:27.967 --> 1:13:32.357 if you if you want to be a infosec company 1:13:32.357 --> 1:13:37.477 in 2025, it is influencer marketing and it is especially 1:13:37.517 --> 1:13:40.756 SME based marketing. And this is a lost art. Training 1:13:40.757 --> 1:13:42.837 someone at your company like when your sales engineers or 1:13:42.877 --> 1:13:44.876 maybe you're small enough, it has to be your CTO 1:13:44.877 --> 1:13:47.757 or CEO. But then going to conferences and this is 1:13:47.757 --> 1:13:50.637 what I do, and speaking on topics, not selling anything, 1:13:50.877 --> 1:13:54.437 contributing to the community of security. We get so many 1:13:54.437 --> 1:13:56.477 warm leads from this dude. My talks like I get 1:13:56.477 --> 1:13:58.317 5 or 6 people right after my talks are like, 1:13:58.317 --> 1:14:00.317 let's work together, give me your card. Like you know. 1:14:00.637 --> 1:14:02.197 And you're not even you're not even saying. 1:14:02.197 --> 1:14:02.836 Not saying anything. 1:14:02.877 --> 1:14:03.517 Julia is. 1:14:03.557 --> 1:14:06.117 Yeah. Julia is. Yeah. Julia. My wife is like, hey, 1:14:06.117 --> 1:14:08.756 we sell things, you know? But but that's that's our 1:14:08.757 --> 1:14:10.836 vibe is we go to conferences. I mean, I'm traveling 1:14:10.877 --> 1:14:12.717 to one a month and it's a lot of travel 1:14:12.717 --> 1:14:15.357 and it's hard, but marketing people have been pushing this 1:14:15.357 --> 1:14:17.237 away like, oh, well, we have to, like, train the 1:14:17.237 --> 1:14:21.397 expert and set up the event, and then we have to, um, 1:14:21.517 --> 1:14:23.197 and then we have to buy a hotel room and 1:14:23.197 --> 1:14:25.277 flights and that's all really expensive. We'd rather spend that 1:14:25.277 --> 1:14:27.357 on like bdrs cold calling people. I'm like, that is 1:14:27.357 --> 1:14:29.267 the opposite way of how this industry works. 1:14:29.307 --> 1:14:31.707 You figured this out really early and you locked on 1:14:31.707 --> 1:14:32.546 and it's working. 1:14:32.587 --> 1:14:35.227 Yeah, I figured this out early and and again, like 1:14:35.227 --> 1:14:37.347 the creative control to for our brand, it looks like 1:14:37.347 --> 1:14:39.546 a metal brand, right. Like our canon font. Looks like 1:14:39.587 --> 1:14:41.427 Metallica font. And the. 1:14:41.427 --> 1:14:41.987 Purple. 1:14:42.027 --> 1:14:45.227 The purple and the Spectre logo and all this stuff 1:14:45.227 --> 1:14:48.387 was like, before I used to have to go in 1:14:48.387 --> 1:14:51.347 front of a, you know, like a panel of people, 1:14:51.347 --> 1:14:54.427 the CEO, the CTO, the, you know, the marketer, the brander, 1:14:54.467 --> 1:14:56.107 and be like, here's my idea for this cool thing. 1:14:56.107 --> 1:14:59.067 I know it'll crush all of our competition. And they'd 1:14:59.067 --> 1:15:01.027 be like, ah, I don't know. I don't like that. 1:15:01.027 --> 1:15:03.546 And I'm like, I know the security industry like, this 1:15:03.547 --> 1:15:06.067 is what they want. Like they want cool shit. Like, 1:15:06.107 --> 1:15:08.307 you know, another example is like when we put our 1:15:08.987 --> 1:15:11.427 our name out there with them, like it looks like 1:15:11.427 --> 1:15:14.467 Metallica font. So it says Arcanum, right? We don't put 1:15:14.467 --> 1:15:17.187 our website, we don't put anything. The goal is for 1:15:17.187 --> 1:15:19.747 you to wear our shirt, that you could just wear 1:15:19.747 --> 1:15:22.227 it out every day. And nobody knows. You're marketing a 1:15:22.227 --> 1:15:24.867 security company and many people are very anti that. They're like, 1:15:24.867 --> 1:15:26.457 you have to have your website, you have to have 1:15:26.457 --> 1:15:28.777 something to tie people back. And I'm like, no, eventually 1:15:28.777 --> 1:15:30.937 someone's going to ask that person, where did you get 1:15:30.937 --> 1:15:33.017 that badass shirt? And they're going to be like, oh, 1:15:33.057 --> 1:15:35.017 I went to a security conference. There's actually this hacking 1:15:35.057 --> 1:15:38.137 outfit that does training called Arcanum. That person works in it. 1:15:38.177 --> 1:15:39.897 They're like, oh, I'll go check that out. And like, 1:15:40.257 --> 1:15:41.537 and it's like word of mouth, basically. 1:15:41.577 --> 1:15:42.857 In the meantime, it just looks cool. 1:15:42.857 --> 1:15:44.457 And it just looks cool. You wear the shirt every day. 1:15:44.457 --> 1:15:45.617 And so like there are a whole bunch of like 1:15:45.617 --> 1:15:50.137 micro tricks like that inside of marketing, branding, having an opinion, 1:15:50.177 --> 1:15:52.697 building a company that I feel like I have just 1:15:52.697 --> 1:15:55.897 seen regular marketing people from other domains just come in 1:15:55.897 --> 1:15:58.376 and not get in the security industry. And so that's 1:15:58.377 --> 1:16:00.456 why it works for us, I feel like. Yeah. 1:16:01.657 --> 1:16:03.577 Well, dude, we finally did it. 1:16:03.617 --> 1:16:05.256 We did we we sat down and did a thing. 1:16:05.297 --> 1:16:07.777 I think three last saw were like, we should just record, 1:16:07.817 --> 1:16:08.497 go up to the room. 1:16:08.497 --> 1:16:11.457 We, we've talked about. So what will happen is we'll 1:16:11.457 --> 1:16:14.657 get on the phone. Yeah. And we'll do exactly this. Yeah. 1:16:15.857 --> 1:16:18.337 We should have recorded. We should have recorded that. 1:16:18.337 --> 1:16:18.977 Yeah. Yeah. 1:16:19.217 --> 1:16:22.297 That's happened 29 times. Yeah, exactly. And then we're like, no, 1:16:22.297 --> 1:16:23.577 we got to get on the mic and we got 1:16:23.617 --> 1:16:25.687 to do it. And it's like never happened. Yeah, we 1:16:25.687 --> 1:16:29.167 did the one for a vendor. Yeah, but that wasn't this. Yeah, yeah. 1:16:29.207 --> 1:16:31.207 Um, okay, I got I got one for you before 1:16:31.207 --> 1:16:36.087 we leave. I got so your opinion on Thunderbolts. We 1:16:36.127 --> 1:16:39.127 went to go see the new Marvel movie, uh, two 1:16:39.127 --> 1:16:39.767 nights ago. 1:16:40.327 --> 1:16:40.967 I'm struggling to. 1:16:40.967 --> 1:16:41.727 Tell the people. Tell the. 1:16:41.727 --> 1:16:43.367 People? I'm struggling. I'm struggling. 1:16:44.007 --> 1:16:45.086 I loved it. 1:16:45.087 --> 1:16:48.847 So I liked the main character. I liked, um, you 1:16:48.847 --> 1:16:50.447 know Johansson's sister? 1:16:50.487 --> 1:16:50.727 Yeah. 1:16:50.727 --> 1:16:54.967 Yeah, yeah. She was like, the strength, um, the father 1:16:54.967 --> 1:16:56.247 daughter thing. The father daughter. 1:16:56.247 --> 1:16:56.887 Thing was strong. 1:16:56.927 --> 1:17:00.487 Yeah, yeah, yeah, I like that a lot. Um, I 1:17:00.487 --> 1:17:04.487 don't know, I just like so many things about the 1:17:04.487 --> 1:17:08.327 old franchise. Yeah, I just feel sad. Yeah, when I 1:17:08.327 --> 1:17:10.647 think that, like, they're not around anymore. 1:17:10.687 --> 1:17:11.126 Yeah. 1:17:11.327 --> 1:17:12.807 First, I have a question. Where'd they go? 1:17:12.927 --> 1:17:13.207 I don't. 1:17:13.247 --> 1:17:14.447 Know. Why can't they come back? 1:17:14.487 --> 1:17:19.006 I think Thor left off planet um, because he has, 1:17:19.007 --> 1:17:20.967 like a daughter now, right? And he's on, like, other planets. 1:17:21.007 --> 1:17:22.527 Like saving them, not Earth. 1:17:22.847 --> 1:17:25.407 The. Okay, so multiverse freaks me out. 1:17:25.447 --> 1:17:28.407 Yeah, the multiverse I was not super down with. Honestly. 1:17:28.447 --> 1:17:31.807 Like, you know, here's my problem with the multiverse it 1:17:31.847 --> 1:17:39.607 invalidates death. It invalidates things you're supposed to care about. Yeah. Sacrifice. Yeah. Yeah. 1:17:39.607 --> 1:17:42.407 If someone can, like, snap and just people come back, 1:17:42.407 --> 1:17:45.407 I'm like, yeah, what are we even doing here? 1:17:45.727 --> 1:17:47.487 Yeah. It's also just hard to keep up all the 1:17:47.487 --> 1:17:50.247 different multiverse plot lines. And, um. 1:17:50.847 --> 1:17:53.206 Is that person alive or dead? Well, not in this universe. 1:17:53.247 --> 1:17:56.527 Yeah, exactly. Yeah. Uh, we were talking about the differences 1:17:56.527 --> 1:18:00.607 between comic, like, adaptations of comics and movies, and then 1:18:00.647 --> 1:18:04.047 what actually happens in the comic books? Pretty much. I 1:18:04.047 --> 1:18:08.206 feel like is universally better in the comics. Uh, except 1:18:08.207 --> 1:18:10.127 for a couple things. Uh, I think that there have 1:18:10.127 --> 1:18:12.447 been a couple of movies that they've done, origin stories 1:18:12.447 --> 1:18:14.607 that were better than the comics, but we were talking 1:18:14.607 --> 1:18:18.246 about Bane and how much different Bane is in comics 1:18:18.247 --> 1:18:22.876 versus Superman two. Yeah. Superman two. Yeah. Um, How different the. 1:18:22.877 --> 1:18:24.997 The death or death of Batman. But the breaking of 1:18:24.997 --> 1:18:27.756 Batman was in the comics with Bane versus the movies, 1:18:27.757 --> 1:18:33.197 and then the death of Superman in pretty much everything. Um, like. 1:18:33.237 --> 1:18:34.357 Like there was a whole bunch of stuff that I 1:18:34.357 --> 1:18:36.637 guess you just don't have time to do, you know, like, uh, 1:18:36.837 --> 1:18:41.197 like basically, like, before doomsday killed Superman in, you know, 1:18:41.237 --> 1:18:41.957 in the comics. 1:18:42.237 --> 1:18:42.837 Spoiler. 1:18:43.197 --> 1:18:44.637 So. Oh, yeah. I mean, if you haven't, you haven't 1:18:44.637 --> 1:18:46.517 read the comics, you should. But, uh, this was when 1:18:46.517 --> 1:18:49.237 I was a kid. I mean, doomsday ran amok on 1:18:49.277 --> 1:18:51.877 the on, like, you know, in America, like. And he 1:18:51.877 --> 1:18:54.876 just he wrecked every single person in the Justice League. Like, just, 1:18:54.877 --> 1:18:57.997 like slapped them down. And so that that, like, builds 1:18:57.997 --> 1:19:00.277 this context of, like, six issues up to the point 1:19:00.277 --> 1:19:02.517 where you're like, oh, man, the only person that's going 1:19:02.517 --> 1:19:03.756 to be able to stop this villain. 1:19:03.797 --> 1:19:04.197 Yup. 1:19:04.437 --> 1:19:06.237 You know, the combined might of the Justice League has 1:19:06.237 --> 1:19:08.997 done nothing. It's only going to be Superman. Yeah. And so, like, 1:19:09.037 --> 1:19:12.237 it breeds this, you know, like up and coming crescendo 1:19:12.237 --> 1:19:14.437 of battle. And then, you know, you have the epic 1:19:14.437 --> 1:19:18.197 battle between Doomsday and Superman and he dies. Um, and, 1:19:18.237 --> 1:19:19.277 you know, one of the things I like to talk 1:19:19.277 --> 1:19:21.267 about is when I was a kid, that issue came 1:19:21.267 --> 1:19:24.067 in a plastic bag with the Superman logo with blood 1:19:24.067 --> 1:19:26.787 dripping down it. No way. And included in the plastic 1:19:26.787 --> 1:19:29.467 bag was the comic of his death. It was all 1:19:29.467 --> 1:19:33.307 about his funeral, him dying the world without Superman. And 1:19:33.307 --> 1:19:35.827 it came with a black armband with the Superman logo 1:19:35.827 --> 1:19:36.307 on it so. 1:19:36.347 --> 1:19:36.506 You. 1:19:36.507 --> 1:19:39.027 Could mourn him. And it was made of, like, vinyl 1:19:39.187 --> 1:19:41.427 and so you could put it on your arm. I 1:19:41.467 --> 1:19:43.627 wore that to school that day when Superman and some 1:19:43.627 --> 1:19:45.187 of my nerd friends did, too. It was. 1:19:45.427 --> 1:19:47.187 And you were Superman in your wedding? 1:19:47.227 --> 1:19:49.227 I was Superman in my wedding. We all wore superhero 1:19:49.227 --> 1:19:52.107 shirts under, uh, under a thing as I was Superman. 1:19:52.107 --> 1:19:53.707 And we were talking. Oh, we watched that video last 1:19:53.707 --> 1:19:55.387 night of, like, the iterations of all the Superman. 1:19:55.427 --> 1:19:55.987 Oh, yeah. Yeah, there. 1:19:55.987 --> 1:19:57.867 Were a couple Superman I've never seen before, by the way. 1:19:57.907 --> 1:19:59.827 Yeah. And you're getting me on the the one that 1:19:59.827 --> 1:20:00.387 you like. 1:20:00.427 --> 1:20:04.187 Yeah. Um. Superman and Lois, uh, the two two season 1:20:04.187 --> 1:20:06.947 one I thought was fantastic. I think he's one of 1:20:06.947 --> 1:20:09.387 the best Superman's I've ever seen. Um, and then I'm 1:20:09.427 --> 1:20:10.947 getting you on arcane, too. 1:20:11.067 --> 1:20:11.187 Yeah. 1:20:11.187 --> 1:20:12.827 That's right, arcane is fantastic. 1:20:12.867 --> 1:20:13.307 Like. 1:20:13.587 --> 1:20:16.187 Like like in a few. Yeah, yeah, yeah. So, um. Yeah, 1:20:16.187 --> 1:20:17.947 I mean, if you wanted some nerd segment that was, 1:20:17.987 --> 1:20:18.427 you know. 1:20:19.217 --> 1:20:21.657 Yeah. And then a week from now, we go on 1:20:21.657 --> 1:20:23.017 our spiritual retreat. 1:20:23.057 --> 1:20:24.177 We do? Yeah. We do. 1:20:24.337 --> 1:20:25.657 And can't wait to see you there. 1:20:25.697 --> 1:20:28.536 Yeah. We, uh, we do EDC every year in Las Vegas. 1:20:28.537 --> 1:20:31.416 It's three days where we try not to talk about work. Yup. 1:20:31.577 --> 1:20:34.296 And just listen to music and be best friends. Outside 1:20:34.297 --> 1:20:36.376 of that, I highly recommend for any of you who have, 1:20:36.977 --> 1:20:39.657 you know, friends, who's who sit in the industry and 1:20:39.657 --> 1:20:40.977 you kind of go and hang out with them, and 1:20:40.977 --> 1:20:42.977 you end up talking a lot about infosec and work 1:20:43.297 --> 1:20:46.217 to plan something that's not work related. Like, you and 1:20:46.217 --> 1:20:48.577 I have EDC, and I'm trying to build something with 1:20:48.577 --> 1:20:51.416 Kev where I go to like a comic con, you know? Yeah. Um, 1:20:51.617 --> 1:20:53.977 and you're more than welcome to come if you want, but, uh. 1:20:54.017 --> 1:20:55.017 I've been once, I think. 1:20:55.057 --> 1:20:56.777 Yeah, I like comic cons a lot, but, um, but 1:20:56.777 --> 1:20:59.537 just that activity of doing something outside of infosec is 1:20:59.537 --> 1:21:00.417 really nice. So. 1:21:00.457 --> 1:21:02.977 Yeah. And then we start with work, but really, it 1:21:02.977 --> 1:21:05.577 ends up being like life plans. Yeah. And how we're 1:21:05.577 --> 1:21:06.296 helping each other. 1:21:06.337 --> 1:21:07.857 Yeah, exactly. Yeah. Yeah. 1:21:07.897 --> 1:21:10.017 So we'll do. This was fantastic. 1:21:10.217 --> 1:21:11.177 Awesome. Yeah. Hopefully. 1:21:11.217 --> 1:21:12.256 Hopefully it was recording. 1:21:12.257 --> 1:21:14.137 Yeah. Hopefully. Yeah. I see the little thing. So. Yeah. 1:21:14.177 --> 1:21:14.657 For sure.