S1: So here we are at the house. It is after RSA.
S1: It's been a crazy week and a half. Um, what's
S1: on your mind? What's the takeaway? We probably have several. Yeah.
S2: Yeah. I mean, uh, I feel like this week there
S2: was a lot of fun, but also, you and I
S2: had the opportunity to go to quite a few events,
S2: and there was less FUD at those events, I feel like.
S2: And some real innovation that we saw, uh, which surprisingly
S2: was off the RSA show floor, I think. Right.
S1: Yeah. And it seems like that's like the big lesson
S1: we learned is like, how do we figure out how
S1: to do more of that next year?
S2: Yeah. Exactly.
S1: Yeah. And and less of like the I feel like,
S1: how long have we been doing this? Like 15 years
S1: or something. Yeah, yeah. And, um, at first it's like,
S1: how do you get into the we're not going to
S1: name vendors, but how do you get into the big
S1: vendor conference that has, like the best music and the best.
S2: Big parties and stuff?
S1: Yeah, the big party.
S2: Yeah.
S1: And now we're just like, how can we see our
S1: friends go to like, an event that's like smaller? Um,
S1: I don't know. It's not necessarily that it's harder to
S1: get into, but it seems like that's where the better
S1: conversations happen.
S2: Yeah. I mean, well, I mean, RSA is like everybody
S2: trying to grab like a mic, right? And scream as
S2: loud as they can and get people to listen to whatever.
S2: And the parties are a mechanism for that. The advertising
S2: on the show floor of the booth and everything. But
S2: I think that more and more as we go on,
S2: there's satellite events which aren't at Moscone, and they're usually
S2: held at like companies offices if they have the space or, um,
S2: or like an Airbnb or like some kind of rental thing.
S2: And uh, and they're very on, you know, specific topics.
S2: They're not generalized. They're very specialized usually. And so we
S2: went to quite a few of those this week. Um,
S2: and that ended up being, at least for me, it
S2: ended up being like like the kind of jewel of
S2: the week, right? Like, I did a ton of speaking
S2: and stuff, but, um, I learned the most, and I felt, uh,
S2: I felt the most impressed and kind of, like, hopeful
S2: from from those those events. Yeah.
S1: Yeah. Yeah. So we both did a bunch of, like,
S1: talks and panels and stuff like that. Um, I went
S1: to three of yours. Um, yeah, some really, really good stuff.
S1: And that was, that was a pretty new talk that
S1: you put together about the methodology stuff. That was really cool.
S2: Yeah.
S1: Yeah. And then, um, I would say that the, the
S1: big thing was yesterday. Right?
S2: Yeah. Yeah.
S1: So that was like the funnest thing to do.
S2: Oh yeah. By far. So yesterday was the, uh, OpenAI
S2: security research conference, their first ever one. Uh, and there's
S2: about a hundred hundred people there, I would say. And, um,
S2: talking about everything from, you know, protecting AI, AI agents
S2: from attack to automating security workflows with AI. And this
S2: was like no BS, right? I mean, this is like, yeah,
S2: this is like, you know, academics talking about things that
S2: were brand new, you know, new new methodologies, new ways
S2: to train agents, new evals, new everything new. I mean,
S2: we saw some new models dropped in that in that thing.
S2: We can't really talk about them. But, um, yeah, we
S2: saw some really cool stuff. And then, uh, also just
S2: like it was kind of crazy who was in that room.
S2: And we got to sit like a couple feet away
S2: from Sam Altman and ask him questions. He did a Q&A. Um,
S2: you know, Matt Knight, the CISO of OpenAI, was answering questions. Ian,
S2: who's a good friend of ours, ours was opening questions. Yeah. Uh, so, yeah,
S2: it was really, um, you know, open in feeling and
S2: in kind of like crowd. Everybody was like, having good
S2: conversations around it. And I felt like I learned a
S2: ton from that. That content specific.
S1: Yeah. More than, like, the whole week and a half before. Yeah.
S2: Yeah. It was. Yeah. I mean, I had a really
S2: good one that was like secondary to that, which was
S2: I went to I went to Airbnb's off site, I, um,
S2: thing and uh, so they did it one day at
S2: their offices where they brought in speakers and a mutual
S2: acquaintance of ours. Keith spoke there on a panel about, um,
S2: AI and security. And that one was interesting because you
S2: get the I felt like you got at the OpenAI thing.
S2: You were at the cutting edge of research and at
S2: the Airbnb one, I felt like I was at the
S2: cutting edge of implementation from a point of view of like, businesses. Right.
S2: Because like at the academic level, at the OpenAI one, um,
S2: and even the enterprises there, the people who are talking
S2: about are at the cutting edge and they're also well
S2: funded and they're incentivized to do some really cool research. Right.
S2: At the Airbnb one, it was more companies talking about
S2: their implementation workflows. You know, how they were using AI
S2: in the.
S1: Like on the ground?
S2: Yeah, on the ground. Yeah. So it was a different view. Um,
S2: but I thought it was really cool. Like Adobe talked
S2: about like their architecture for their agent based security vulnerability
S2: management system. Um, you know, Google was there talking about
S2: some stuff, and it was just it was it was
S2: really cool. So.
S1: Yeah. So you you gave the talk at open AI
S1: as well. So just, just give it like an overview
S1: of like the talk like the methodology and stuff.
S2: Yeah. So I do this class called uh attacking AI
S2: which you've been to. And um, the whole class is
S2: basically it's my methodology for AI Pentesting. And when I
S2: say I pentesting, people are like, oh, you mean AI
S2: red teaming? And I actually don't. So what I find
S2: at least visiting and talking to other experts, is that
S2: AI red teaming has been around for a long time,
S2: and they have cemented that term. And that is usually
S2: about attacking the model, right? Like the model in place
S2: can speak harm, can speak bias. It can tell you
S2: how to cook meth, you know, and that is that
S2: is stuff that happens in, you know, this one vertical
S2: of attacking AI, which is the model, When you do
S2: an AI pen test, you not only have to assess
S2: the model and what it will say to your users,
S2: but you have to assess the implementation of the model.
S2: You have to assess, um, everything else that's hooked up
S2: to like all of the DevSecOps tools that do logging
S2: and observability and all this other stuff around it. Um,
S2: and so it ends up being, uh, a hybrid web test,
S2: an API test, and then also AI red teaming, and
S2: then also now you have in order to get these
S2: systems to do things for you, like agents that are
S2: hooked up to tools and APIs and stuff like that,
S2: you also have to get them to, um, accept prompt
S2: injection through security gates, which is like classifiers and guardrails.
S2: So the talk was basically our methodology at a high
S2: level on the whole pen testing process, um, which has
S2: seven steps. And then the second part was our prompt
S2: injection taxonomy, which is really like a taxonomy to sneak
S2: attacks through classifiers and guardrails. Um, and so we open
S2: sourced a tool about a month ago. It's called the
S2: Arcanum Prompt Injection taxonomy. And it goes through all of
S2: these tips and tricks to do that. And so we
S2: split it up into four levels. One is um, or
S2: we call them for prompt injection primitives. One is um,
S2: your intent, like what you're trying to do to the
S2: AI system, is it, you know, get it to do
S2: some of those red teaming things like speak harm and
S2: bias or is it like get it to leak its
S2: system prompt, or do you want to jailbreak it entirely,
S2: or do you want to do something completely different? Right.
S2: And then we have three other sections. We have, uh, techniques. Um,
S2: is one of our primitives and techniques is how you
S2: execute the attack. There's a framing. You can do narrative injection,
S2: you can do all this kind of crazy stuff. And
S2: then we have evasions, which are, um, the idea of like,
S2: it kind of feels like WAF bypass, right? Where we
S2: do a lot of tricky encoding. But there's even more
S2: in the prompt injection world than WAF bypass. And these
S2: get you past the security products like, um, classifiers and guardrails.
S2: And so we talked about we talked about all of
S2: those and utilities we had made. And that was the talk.
S1: So yeah, it's it's really good. And I've been watching
S1: all these and It's absolutely the best. Thanks, man. Yeah,
S1: and it's like it's presented really well as well. Um,
S1: and by the way, I'm going to talk to the
S1: camera here for a second. So on the way up here,
S1: we were like, we're just going to have a conversation. Yeah. Yeah.
S1: You know what's hilarious? We sat down and became podcasters.
S2: I know right.
S1: We're like. We're like, all right, let's talk about the content.
S1: And like, so I don't know how we break out
S1: of that because it's just.
S2: I think we can I think we can.
S1: Well, we'll just let it run and hopefully we, uh, like, relax. Um, yeah.
S1: So what else? Um, what else should we cover?
S2: So I had a whole bunch of notes on my phone,
S2: and then my phone battery died. In fact, we were
S2: we were trying to set up with, uh, my new
S2: DJI setup, which I like, which Julia got me for Christmas.
S2: A friend of ours, Ron Foster, recommended the whole stack,
S2: like the DJI camera and DJI wireless mic, so I
S2: could go to like conferences and have quick interviews with people.
S2: And of course one couldn't set it up right. And
S2: then two, um, phone ran out of battery.
S1: Almost has it set up. Phone dies.
S3: Almost haven't set up. The thing dies.
S2: So. So we're going to come in recording in the
S2: in the command center for you. But um, but yeah.
S2: So I had some notes on my phone and, uh,
S2: the notes are just like how at least I felt
S2: on the floor. You got to walk the floor, right?
S1: Yeah, a little bit. Yeah.
S2: So we didn't sync up till later in the week, right. Like,
S2: we mostly had our own stuff in the beginning and
S2: then we synced up later. Um, but, uh, I don't
S2: want to note on my phone on how the floor
S2: was very funny this year. Right. Like. And also, I
S2: had these weird moments of, like, justification. It was like, um, like,
S2: everyone last year said they were going to completely automate
S2: all these security workflows. Right? And I distinctly remember, like,
S2: writing these notes last year. And then this year I
S2: walked the floor and everybody has changed their tune. It's like, oh,
S2: I assisted like, you know, power your people, scale your team,
S2: you know? And it's.
S3: Like.
S1: Turns out that was harder than we.
S3: Thought. Yeah.
S2: Turns out turns out not possible. And I think I
S2: had this conversation with you on the drive and it
S2: was like even you and I have friends who are
S2: starting companies who were trying to do that, they're trying
S2: to automate a workflow. And we saw it at the
S2: OpenAI thing, too. People are like, yeah, we're not close
S2: to full automation yet, and we're very far away from
S2: a place where the models can do full automation. Even
S2: the people in the FTC competition who are going for
S2: those cyber reasoning systems at Defcon, like they were like, yeah,
S2: like a lot of this is still algorithmic. Um, you know,
S2: tool based automation, but with like a lot of AI
S2: glue at the I think they said like the framework level. Yeah.
S2: And the, um, the framework and the organization levels are
S2: the parts where I actually like, really helps them. But, um,
S2: hearing that from or like seeing that on the floor
S2: and then also hearing it from the people at OpenAI,
S2: just like kind of cemented we're pretty far away from
S2: fully autonomous systems anywhere. Um, you know, I was really
S2: impressed by I don't want to name any vendors or whatever,
S2: but like, there were some demos at OpenAI thing that
S2: were pretty, pretty sick in the web testing world.
S1: Um, yeah.
S2: And so that was the one I was like, oh, okay.
S2: Like like they're getting close. And I think you and I.
S1: At least in that one domain.
S2: In that one domain. Yeah. Web testing. And then you
S2: and I were looking at some friends who were in
S2: that room, and I think they were a little crestfallen
S2: to see how far that one place had gotten with
S2: autonomous testing.
S1: They were there live learning how far their competitors were.
S2: So.
S1: Bad. They're just like, oh, I'm excited. Oh, yeah.
S2: Yeah, yeah. Um, but that was that was cool. You know,
S2: it was cool to, um, get to ask Q&A of, uh, Altman. Um, and, uh, uh,
S2: you know, so we, you know, it wasn't like we
S2: got one on one time with them, right? It was
S2: just like, we just. We got. We were.
S1: In the.
S2: Front row, though. Yeah, I snagged a seat, so I
S2: was I was.
S1: Yeah, that.
S2: Was good. I left lunch early to get a seat
S2: right in front, but, um, so. Yeah. So we're, we're
S2: in the front row. And one of the questions to, uh,
S2: Sam Altman, if you don't know that, uh, head of
S2: OpenAI CEO and, uh, they were one of the questions
S2: was like, what do you, uh, like, what is the
S2: security thing that you worry about? Right? And Sam is
S2: the CEO. He's the CEO, right. So he's not in
S2: security every day. But he's a smart dude. Yeah, yeah. And, um.
S2: And he was talking about, uh, he was like. Like
S2: he starts launching into his answer, presupposing he's like, yeah.
S2: So when we get these, like, you know, fully context
S2: aware agents, that has everything about my life, like written down,
S2: you know, and has all this information and can make
S2: these really great intuitive decisions for me and stuff like that.
S2: Like what happens when that gets hacked, your whole ethos,
S2: your whole stack of who you are as a person,
S2: what you like to use, what you like to watch,
S2: what you like to hear, what you like to eat.
S2: And so many things can be intuited from that as
S2: well about you as a person. What happens when that leaks? Right.
S2: And I'm just there and I'm like, hitting you. I'm like, Dan, Dan.
S1: Grabs my leg.
S2: Yeah, yeah. Like grabbing his leg because. Because you've been
S2: talking about this for like ten years, right? Like, I
S2: remember the first few blogs about the first iterations of, like,
S2: kind of unified context or context about your life, right?
S2: And then you do it with your tlos files as
S2: a person, and you also do it for companies to
S2: understand the ethos of the company. And Sam just launched
S2: right into that. Yeah, and it was like presupposed. And
S2: I'm like, you motherfucker. Like.
S1: Yeah, I was so excited. I'm like, can I start talking?
S2: Like, yeah, yeah that's great. Yeah, it was great. But
S2: he also he said the the risk is that, you know,
S2: I guess in your analogy, the teller's file gets leaked
S2: right somehow. And then people know, like what you're about
S2: and how they can specifically adversarially market to you, influence you. Yeah.
S2: You know, and if you're really open with your personal
S2: assistant or, or your, um, you know, whatever ends up
S2: collecting that information, you're really open to that, like some
S2: of your, like, idiosyncrasies, some of your, like psychological stuff.
S1: Well, it's just like, you know, I think the most powerful,
S1: powerful version of this is like, you have your, um,
S1: your journal in there. Yeah. And you're just constantly complaining
S1: about your mother in law, right? Yeah. And that gets hacked.
S1: And so now my mother.
S2: In law's great, by.
S1: The way. And now now that becomes a, um, it
S1: becomes like a an extortion email. Yeah, right. Someone could
S1: just be like, do you want me to send this
S1: to your mother in law or just send me, you know,
S1: $20 or whatever? And it's like, that's worth it. I
S1: don't want to. I don't want to have that fight
S1: at dinner.
S2: Yeah, that's. I mean, $20.
S1: Seems like a good.
S2: Ransomware price, right? Like ransomware operators are listening. Like, I'll
S2: pay 28 bucks. That's cool.
S1: Yeah, yeah. But but to your point, it's like your
S1: entire personality. It's like your entire soul. So that's. Yeah,
S1: that's a lot of content to lose.
S2: Yeah. One of the things we're talking about in the
S2: car was how we saw so many. And it sucks.
S2: But like friends, colleagues working for these places, hanging their
S2: hats on these AI features or even companies that are
S2: completely based around AI, and there's no there's no moat
S2: for them, like it is going to be disrupted either
S2: by because we then we went to the OpenAI thing,
S2: saw what they're doing.
S1: Yeah. And destroying moats.
S2: Yeah. Destroying moats basically. Right. Like whole companies are going
S2: to go down because they had this premise of, oh,
S2: we'll use AI to do this thing. Now it's just
S2: going to become part of the model, or they're going
S2: to be just trampled by one of the big data
S2: aggregators who already has all the data to make the
S2: problem set easy to execute, I think, is how I
S2: think of it.
S1: Yeah. Yeah, absolutely. So, so Jason talked about what he
S1: was presenting. Um, so what I was presenting is like
S1: this unified entity context. Yeah. Yeah. So it's like, if
S1: it's an individual, you just get all that stuff that
S1: you already talked about. And then if it's a company
S1: you get all the way from the company goals all
S1: the way to the security goals. But all the HR stuff,
S1: like everything all into one bucket. And then from there
S1: you just ask questions. Yeah. So if you ask HR questions,
S1: is that HR software?
S2: I think it only works with the the data right.
S2: Like yeah data about the people. Yeah.
S1: Yeah. It's like if you have HR data in there,
S1: you have security data in there. And you ask security
S1: questions and HR questions. You have HR software. So I
S1: feel like software verticals just go away.
S2: I mean, I don't think they completely go away, but
S2: I think yeah, a lot of them are are majorly disrupted. Right.
S2: It's like and again, it'll be the big companies that
S2: do this first because they already have that infrastructure like
S2: Microsoft has the whole graph API about, you know, corporate
S2: user data. They have security data. They have God. God
S2: knows what other data. Right. So they're poised to move
S2: quickly in some of these places that you and I
S2: play in. And it's like they will get there first
S2: because they have the the ability to grab everything. And yeah,
S2: I think I think that for your talk, I mean,
S2: you talked about like as a meta thing, you know,
S2: like everybody is focused on, oh, we can build these
S2: agents or, you know, I questions whether you're just using
S2: an API, you're actually using a planning agent or whatever.
S2: It doesn't matter. Like whatever architecture you've chosen, use AI.
S2: And that's the important part. Right. Like prompting it, getting,
S2: you know, you know, rags set up like all this stuff.
S2: And you had that one slide where it was like, okay,
S2: so the agents are the big boxes and we really
S2: care about them right now, right? And then they're attached
S2: to data sets and they're kind of small. And then
S2: you have the other slide. And then it's like, no,
S2: actually the key piece is the data in the middle.
S2: And the thing that validated it for me was you
S2: didn't you were not the OpenAI one. But next year
S2: we got to go and present. But on this topic too,
S2: by the way.
S1: Oh, the Airbnb one.
S2: Yeah. Airbnb one. Airbnb one. Yeah. So, uh, the guy,
S2: one of the guys from Microsoft, I can't remember his name,
S2: but he was talking about kind of the same idea
S2: as he calls it, like Golden Data Lake or something
S2: like that. But it was the exact same thing. He's like, hey,
S2: you know, like the models are going to get so
S2: good that they are general software products themselves. Right? And
S2: so it's that's not the thing you should be focusing
S2: your development on. Your development or either systems architecture revamp
S2: or whatever should be on collecting the contextual data. That's
S2: going to help you answer the questions. Right. And so
S2: in your slide you had like an image where the
S2: agents were really big and they were the important question.
S2: And then then it's like the next slide is like
S2: actually the most important thing is the model. And the
S2: agents just live on the side a small little thing.
S1: So yeah yeah yeah Yeah. So so yeah. So what
S1: I have was like cybersecurity with AI around it versus
S1: AI in the middle and then cybersecurity and HR and productivity. Yeah,
S1: the software verticals are kind of like rotating around it.
S2: Yeah.
S1: Yeah yeah. Because yeah I think you just you just
S1: collect all that data and ask the questions and answers it.
S2: Yeah. And I think, you know, on my phone I
S2: had some notes too, that um, a common theme I saw.
S2: We're going to talk a lot about AI. Sorry. I mean,
S2: like a lot of, you know, I know people want
S2: to hear about security stuff, too, but, um, a lot
S2: of this stuff in architecture, of these systems, it was
S2: it had to be more compartmentalized than, I think even
S2: some of the architectures I was thinking of meaning that, um,
S2: where I thought I could ask a bunch of questions
S2: of an, you know, an AI model and stuff it
S2: all into one masterful system prompt or something, or user prompt. Um, actually,
S2: successful implementations are asking micro questions, kind of like microservices, right?
S2: We are asking one question of the same data of
S2: a data set. Wherever you take it in from the
S2: user or from context somewhere else. And then that's one agent, right?
S2: It's just a one question, one agent that's really good
S2: at doing that one thing. And then you have tens
S2: or hundreds of those because, you know, at least from
S2: the people I was talking to, they're like, you know, like, uh,
S2: don't try to stuff it all into, you know, like
S2: one process because it can confuse, you know, different models
S2: and stuff like that. So some of the ones I
S2: saw had up to like ten agents to like action
S2: one workflow, right? Asking individual questions and then stitching together
S2: the output of that.
S1: So interesting. Yeah. Yeah I, I don't know I, I
S1: feel slightly different there. I feel like all the data
S1: in one place is good. And so you just ask
S1: the questions. Um, but where I do see what you're saying.
S1: The AI, uh, x that stuff was fascinating. Oh, yeah. Oh,
S1: you know, what I really loved about that was the
S1: conversation of like, does the model matter more, or does
S1: the architecture of the system matter more?
S2: So before we go into this, because that whole panel
S2: and the two talks were awesome. Yeah, they were great.
S2: But I don't think everybody knows what the I.
S1: ZK yeah, yeah yeah.
S2: Go ahead. Okay. So the Acsc is this competition run
S2: by DARPA. Um, and the whole idea is that, uh,
S2: you build what's called a cyber reasoning system. And so
S2: there are both academic teams who are mostly CTF teams
S2: from the Defcon CTF kind of ecosystem. And then there
S2: are also companies who have come who have come to
S2: compete in this contest. And the goal is to build
S2: a system that is AI enabled that can go from, um,
S2: taking an open source project repo with a vulnerability in it,
S2: finding the vulnerability through static analysis, building an exploit, testing
S2: the exploit to see if it works in the wild,
S2: patching the exploit and keeping the service up and running. Yeah.
S2: So it's got to do both offense and defense. Um,
S2: and they get scored on multiple different facets of that. And, um,
S2: I think Dan Guido from Trail of Bits was telling
S2: us this night the top like the the they've been
S2: giving out prizes every year. So last year was the
S2: semifinal rounds and five teams made it to the semifinals.
S2: It's like three teams and two companies of which Trail
S2: of Bits is one. And my alma mater, uh, shellfish
S2: is one. And then, um, now I think grand prize
S2: will be he said, like five, three, 2 million. So
S2: 5 million, 3 million, 2 million or something like that
S2: might be.
S1: Four, three.
S2: 2432 or something like that. So first place gets 4 million,
S2: you know, second place gets 2 million. And at the
S2: end of the competition they will actually have to open
S2: source their cyber reasoning systems too, which was a really
S2: interesting conversation I had away from the table with with
S2: some people. But okay, so that's that's leading up to it.
S2: So yeah. So there's three talks around that competition. Uh,
S2: there's a panel with a whole bunch of leaders or
S2: people who were associated to the teams. There was, um,
S2: a couple of people who just presented on their kind
S2: of research around their cyber reasoning system. And then, um,
S2: and then there was a talk by the, uh, by
S2: one of DARPA representatives about why they built the competition
S2: and stuff like that. So I just wanted to. Yeah, yeah.
S1: Yeah yeah, yeah. Perfect. Yeah. So the thing I've been
S1: really super excited about is like these generalizations of architectures
S1: specifically generalizing the scientific method. So I love the idea of,
S1: like you have a collection of goals, you have a
S1: testing engine that basically tests, you have a hypothesis, you
S1: have problems, goals, and then the testing engine. And you
S1: can actually combine the ideas as well, like mix them
S1: and like try to find variations that are actually more effective.
S1: But it's just this life cycle of here's a cool idea,
S1: see if it works. Mhm. Um, and so what I
S1: heard listening to them and we're not divulging anything because
S1: they were all competitors. Yeah. On the panel. So nobody
S1: was divulging anything secret.
S2: Well, except for that one guy. The one guy was like,
S2: I don't know.
S1: He's like, screw it. Yeah, yeah, yeah, yeah. But I
S1: just want to make sure we're not disclosing it. Yeah, yeah,
S1: it's it's all open. Um, but but, um, basically the
S1: idea that you could just, like, keep iterating on this
S1: and you can kind of use it for anything. So
S1: what I found is, um, really interesting was they basically
S1: said they spend all their time fixing that system and
S1: that the, the model getting smarter didn't necessarily help as
S1: much as improving the system itself.
S2: That seemed to be what a few of the audience
S2: questions were aimed at is like, well, you know, as
S2: the model gets better, doesn't it make it better for you? Um,
S2: and yeah, like, like you said, some of their answers
S2: were no, uh, that the scaffolding that hooks everything together
S2: was actually the important part that they needed to develop
S2: more than. Yeah, um, the AI models, because they had
S2: scoped the AI models to do certain things. I remember
S2: one of the answers was like, um, you know, one
S2: of the answers for one of the teams was like,
S2: fast iteration of of that model. The scientific model is like, hey,
S2: I think this is a cool idea for like a
S2: single agent to work on, and we're just going to
S2: test it with the limited data sets that we have
S2: and the problems that we have and see if it works.
S2: And he was saying that that was their key to
S2: success is just like trying so many things, um, different
S2: representations of code, different ways to, you know, um, you know,
S2: like make exploits, like, you know, cut it into pieces,
S2: do it all at once, like, and he's just trying
S2: all kinds of stuff and like, you know, some panned out,
S2: some didn't. Yeah. Um, you know, so there's still a
S2: lot of learning to be there. There's also a lot
S2: of talk about how the competition also has, um, or
S2: at least in the first couple rounds, had a ton
S2: of handicaps like they were. Yeah, they couldn't use like
S2: they could only use, like $100 in tokens. Yeah. Only
S2: certain models, um, and only so much output, uh, and context.
S2: And so they were really, they really said that they
S2: had designed those systems to work inside those constraints. And
S2: now in the finals, they don't have any of those
S2: constraints anymore. So it's like it kind of messed them up. Uh,
S2: in systems design because they were like, if we would
S2: have had no restrictions from the beginning, we would we
S2: might have used different models for this, or we might
S2: have done x, Y and Z, you know.
S1: Right, right. Because that was one of my questions of like, what?
S1: Why are you building this ultra specific thing? It seems
S1: less efficient. And they're like, that's because we had so
S1: many limitations. A generic system would not have worked. Yeah.
S1: You had to put all these weird constraints on it. Yeah.
S1: Or because they had weird constraints on them?
S2: Yeah. Yeah. I mean, uh, some of the other things
S2: there too, were that, uh, some of these security tests
S2: end up being structured more like unit tests in software. Right?
S2: And so, um, there are actually libraries out there that, uh,
S2: that do unit testing really well. And people are like, yeah,
S2: we just started using those for.
S1: Oh yeah. The guy mentioned. Yeah. Do you remember that one.
S2: It's like JS something like I don't know, I have
S2: it in my notes, but yeah. Me too. Yeah, I'll
S2: find it. Maybe put it in the show notes. Yeah.
S2: But there was that. And then I thought interesting questions
S2: like how much you know, how much are you spending
S2: on building this system? And that was a really interesting question.
S1: Um, and nobody was building their own models.
S2: Yeah. So you can't. Yeah. It's an option now in
S2: the finals to, to bring your own model. Uh, it
S2: wasn't before. Um, but nobody's doing it because it's.
S1: Much.
S2: Too expensive. Too expensive. And then, you know, like the
S2: consumers of, you know, the people who weren't competing in
S2: the competition were like, why don't you use this one
S2: off hugging face? And like, you don't understand, like, those
S2: things suck. Like they were like all of the open
S2: source models that deal with security data right now are horrible.
S2: And we saw two talked about this week or one
S2: came out yesterday from Cisco. And then one we can't
S2: talk about. Um, but it'll be out at some point.
S2: And they are security trained models. And then we had
S2: Gemini also talked about a couple of weeks ago. And,
S2: you know, none of these are out. So they can't
S2: use them on these teams yet. And so, um, you know,
S2: maybe the next generation of security data trained models will
S2: be good, but, um, they just have to work with
S2: what they've got right now. And they said pretty much
S2: it all sucks. There's also a reoccurring theme of evals
S2: suck right now for security. Um, for security based, uh, training?
S1: Basically, yes. Joel presented on the difficulty of evals. Yeah.
S1: A lot of people were like, evals suck. And a
S1: lot of people were like, these are really, really hard. Yeah,
S1: some people were like, don't. They don't mean what you
S1: think they mean.
S2: Yeah, yeah.
S1: But, um, yeah. To me the eval piece is part
S1: of that testing engine because like, how do you know
S1: if the tests worked unless you have good evals. Um,
S1: but there's like so much hacking of the evals going on.
S2: Yeah. I mean, that was in my notes, too. It's
S2: just it's like there's people who are purposefully building in
S2: either training or, um, logic into some of these things
S2: to score really high on these.
S1: Yeah. They like post train with it.
S2: Yeah. Post train with it. And so they score really
S2: high on these, uh, things that I would look at
S2: as a consumer. Right. And I'm like, oh, okay. So
S2: I'm going to go look at, you know, these, you know,
S2: these evals on hugging face. You know, I forget the
S2: really popular one on hugging face. It's like the chat arena.
S2: Chat arena. And they'd be like, cool. How does it
S2: score on Chat Arena? And it's like, okay, it's really high. Well,
S2: that doesn't mean it doesn't mean anything for for a
S2: domain specific application of an AI, right? Like, you may
S2: not need a model that's really good at on chat
S2: arena for something else. And so one of our longtime friends,
S2: Joel Parrish, former you worked you worked for me at
S2: Deadspin when he first started. I'll tell a funny story here.
S2: So Matt Knight is the CISO of OpenAI and I
S2: don't know, Matt. Super. Well, I know him a little bit.
S2: And I was like, hey, like, I, uh, I was
S2: actually one of the guys who worked with Joel at Deadspin,
S2: our first testing job together a two decades ago. Right.
S2: And then he worked for us at HP or with
S2: us at HP. Then he went to Apple with you,
S2: and then he went to OpenAI with Matt. And so
S2: I told Matt, I said, I think I'm one of
S2: his first LinkedIn recommendations. I don't know if he kept
S2: it on his profile or not, but I called him
S2: the Kobe Bryant of web hacking. Right? Yeah.
S1: Totally.
S2: Yeah, totally. Um, but he does everything now. Anyway, Joel
S2: gave a presentation at this is the Run Sibyl Side event,
S2: which was also really good.
S1: It was.
S2: Great. The run, the run. Sibyl side event. Um, the
S2: Run Sibyl is a company that's doing, um, automated pen
S2: testing and, um, and pen tester assisted AI tools, basically, uh,
S2: led by a friend of ours, Ari. And so I
S2: spoke there, gave my talk there. But Joel was right
S2: before me. And Joel gave a talk called your Eval suck. Um,
S2: and his thing was like, hey, look at all these
S2: evals that, uh, people are using right now. They're written
S2: in the 90s, like stack based overflows, right? Which is
S2: not what we are facing in 2025. Right. And, uh,
S2: he just showed like, multiple examples of, you know, some
S2: of these evals are not even testing exploit generation or
S2: web testing or anything like that. They're testing just like
S2: code quality stuff, which is from the 90s. Right? And
S2: it's like it's like, why are we benchmarking these security
S2: models off of these crazy old evals? Um, and that
S2: was that was the genesis of his talk. What I
S2: was sad about, though, is you left for dinner one
S2: talk after or one talk before it ended. And the
S2: last talk was this guy who basically brought a humanoid
S2: robot like, about this tall and had jailbroken it with
S2: an exploit. He bought it from China and got it
S2: to run around the room, just like screaming at the
S2: top of its lungs with like, a video game track.
S2: It was so cool. Like it was. It was amazing.
S1: Attacking communism. Yeah. As a Chinese robot. Yeah.
S2: So, okay, so out of context, it sounds bad, but
S2: if you've ever played, um, the fallout games, um, there
S2: is like this audio track in, uh, in fallout that, like,
S2: the rhetoric in that game is anti-communism, right? And so, like,
S2: there's like a robot that goes around or like a,
S2: like a character, you know, some of the enforcers run
S2: around all they do for 24 seven just talk about anti-communism.
S2: So if you if you pull down the audio track
S2: for this game, there's a 20 minute rant robot rant
S2: about anti-communism. And so he put it on the jailbroken
S2: robot and it's just running around spouting like, anti-communist and
S2: then like it was so funny because it was it
S2: was it was like it was talking about anti-communism and
S2: it was like it was like possible defector. And then
S2: it ran into this lady's table and spilled her drink
S2: on her, and she thought it was so hilarious. Like,
S2: she was like, this is crazy. It was it was great. Yeah. Yeah.
S1: Yeah. I got to get the video.
S2: Yeah, yeah, yeah. So, um. Yeah. So evals, you know, struggle,
S2: benchmark struggle right now for domain specific applications. Everyone's kind
S2: of figuring it out. And I think I felt like
S2: on the show floor, there was still a lot of
S2: promising of things that when I went to talk to people,
S2: it was not as, um. First of all, no one's
S2: training their own models to do any of this, right.
S2: Like vendors will say, yeah, we have our own model.
S2: They're not like they're using llama for or, you know,
S2: their own keys and yeah, or. Yeah, deep seek or.
S1: Just the cloud ones.
S2: Yeah, just the cloud ones. Anthropic or OpenAI. Right. And
S2: then all of the business logic magic that they promise
S2: you is happening is system prompt based. Like that's, that's
S2: the majority of of all of those products. And the
S2: value prop is no longer automation. The value prop for
S2: them is now, oh, you know, up level or skill
S2: level your people. Right. And um, and that means basically
S2: at least the way I, I, you know, kind of
S2: package that is it's good at the things we already
S2: knew it was good at. Right. It's good at summarization.
S2: It's good at rewriting. It's good at pulling multiple data
S2: sets together and offering, you know, a couple insights here
S2: and there. But the actual automation of things not quite
S2: there yet harder to implement. Architecture is way you need
S2: to invest way more in it.
S1: Yeah, I mean, I think we knew this before going
S1: into RSA just because you and I are actually building
S1: this stuff. It's like the problem with agents and like
S1: pursuing goals is they just get confused, right? They get
S1: confused over scoped.
S2: Scoped.
S1: Yeah. Especially if you have like red teaming for example,
S1: which is the one that all our friends are struggling
S1: with the most.
S2: Yeah.
S1: Because the first step. Cool. I can launch the web
S1: attack second. Okay. I can pivot a little bit. Yeah.
S1: But then it's like, okay, I've got seven more goals
S1: to get. What have I done already? And so it
S1: starts losing context. So I think that's where it's kind
S1: of falling apart. Yeah. The other thing we were talking
S1: about was how, um, the moat situation. Yeah. So, so basically, um,
S1: a lot of these companies that are like, we do
S1: this thing, that's what makes us special. And we're in round,
S1: you know, B or C or whatever. And we've raised
S1: all this money because we do this one thing. It's like, well,
S1: if you have the context, like, uh, I was talking
S1: about this week, if you have that context and you
S1: can ask the questions. The thing that company does is
S1: a feature instead of a company. Yeah. Yeah. And like,
S1: as we would go around and see these different booths. Yeah.
S1: They look like features.
S2: They're gonna get karate kicked.
S1: By by.
S2: Uh, the model vendors and. Yeah, it's, uh. Yeah. I mean,
S2: it sucks because those are some. awesome. It is some
S2: of our friends who are making these companies and it's like,
S2: I don't know. I mean, you can be really good
S2: at a problem and succeed better than a big model
S2: vendor or a big a big company that you know.
S2: But it's got to be really good.
S1: And maybe they just cut through so good with marketing
S1: that they get a big enough market share that they're okay. Yeah.
S1: But the time is like ticking. You just hear the
S1: time ticking down between Google gets there, Microsoft gets there.
S1: One of these big players gets there. Yeah. And then
S1: they just start adding question modules for security or whatever.
S2: I mean maybe that's the golden plan though. Maybe it's
S2: like they're not at the place to stay a long
S2: term viable. Just just get bought by one of the
S2: big companies because they do it really well. Right. Which
S2: is totally a play. Yeah.
S1: Nothing wrong with.
S2: That. Anyone wants to buy. I'm just kidding. Um, yeah. So. Yeah. Um, yeah.
S2: So that could be a play for sure. Yeah. Uh, yeah.
S2: I mean, other than that, though, uh, Again, same vibes
S2: as last year and the year before. I saw some
S2: products that, uh. Besides, I are just abstractions of what
S2: someone else already does, but it's a better visualization and
S2: easier to make it work with. Right. So like the
S2: Amazon ecosystem, if you're an Amazon specialist, that shit's hard
S2: to learn. Like there are so many sub tools and
S2: sub products and it's like and so then like you
S2: see this other company like, yeah, we make this easy, right?
S2: Like here's a nice guy explains everything Wiz. Yeah. Wiz. Right.
S2: And it like does everything that you want it to do.
S2: And I don't think there's a lot of moat around
S2: that either. That's just a UI revamp to a lot
S2: of the core services places. And so I saw a
S2: lot of that. It's like make your SoC easier to
S2: automate or like whatever. And it's like, okay, I get it.
S2: I get why that's attractive right now, because you have
S2: that pain right now as a consumer. But that pain,
S2: I don't know if it'll be there forever once other
S2: people figure out. Although you can also look at case
S2: studies from Google, right? They never fix their UI. And
S2: you know, so like like Gmail could use a refresh.
S2: So yeah, I mean, yeah, um, I saw a lot
S2: of that. Um, was also kind of surprised just at the,
S2: at what I perceived to be the spend at RSA
S2: this year in a time where I know security professionals like,
S2: who are jobless and, and they have been looking for
S2: roles for months. Right.
S1: That's a really good point. I didn't think about that.
S2: And then just the amount of of money on that floor.
S1: It felt like 2018 or 2019.
S2: Yeah, it was crazy.
S1: It was like top of the you know what it
S1: almost felt like it almost felt like. And I hadn't
S1: thought of this until just now when you said this.
S1: It feels like desperation. They're just like, spend all the money.
S1: It's like. It's like our last chance.
S2: Yeah, it definitely felt like that with some vendors for sure. Um,
S2: which maybe goes into that like acquisition play is like,
S2: we just make ourselves seem bigger this year. We'll get
S2: acquired and it won't matter anymore. Right.
S1: Well, so if you take like the macro economy or whatever,
S1: and it's just like things might get bad in the
S1: next six months or a year. Yeah, we're going into RSA. Yeah,
S1: we need to get bought. Yeah, we or we need
S1: to get a bunch of customers. Yeah. Now is not
S1: the time to go small.
S2: Yeah.
S1: So we saw baby goats.
S2: Yeah, we saw goats. We saw puppies.
S1: I didn't see the puppies, but I saw the goats.
S2: There's a monster truck and an F1 formula one car.
S1: 2018?
S2: Yeah. I mean, um, what else? A giant, giant robot, obviously,
S2: at the CrowdStrike booth, like, every year. Giant statue. Um, yeah,
S2: there was I mean, there was usually there's one marquee
S2: party at uh, at or at RSA, right. There's like
S2: one vendor who brings in like a band. So like
S2: last year, I can't remember who I went to Dead
S2: Mouse last year, which is one of my favorites. And then, um,
S2: for like, the rock crowd, they had, um, Incubus, I
S2: think last year or maybe, maybe the year before, I
S2: can't remember this year. Both marshmallow still Premier DJ and
S2: Chainsmokers were performing at different parties on the same night,
S2: which is crazy.
S1: One. Sentinel one, I think.
S2: Yes, I know one was a marshmallow. Yeah. And, uh,
S2: and then, um, I can't remember who did, uh, Chainsmokers,
S2: but yeah, I think it was like chain guard or something.
S2: I can't remember, but, um, but yeah, I mean, that's
S2: a lot of money to, like, you know, buy out
S2: a nightclub for hundreds of people, you know, have, like,
S2: a premier DJ play just for your corporate party. Um,
S2: and so it just it felt like there was a
S2: lot of money spending. And it made me sad a
S2: little bit because, like, I do have friends who have
S2: been struggling to find jobs or have gotten like, um,
S2: work furloughed, you know, a lot of friends getting furloughed
S2: where like, they're like, oh, we can't afford to pay you.
S1: Yeah. We got to cut back on salaries. Yeah, because
S1: money's tight.
S2: Yeah.
S1: But we need goats.
S2: Yeah, we need goats. Yeah, we need goats. So. So
S2: that kind of sucked a little bit. I think that's
S2: a continuing pattern though. Probably that's happened every year a
S2: little bit. But um.
S1: I'm worried about next year that they're just like, well
S1: that didn't work.
S2: Yeah I.
S1: Mean tighten it.
S2: Up. We'll see. I mean, every year you look for
S2: a vendor that you thought was cool last year, and
S2: then they're not there, you know, this year.
S1: Yeah. Isn't that weird?
S2: Yeah.
S1: Someone comes out of nowhere and, like, four of them
S1: just disappear.
S2: Yeah, there was a couple, a couple last year that
S2: was really excited that they were applying AI to document classification.
S2: And I was like, that's a perfect application of AI, actually. Yeah, yeah. Um,
S2: and they were not around this year either they got
S2: gobbled up or they didn't make it. So yeah.
S1: Oh yeah. What's the one I got? A buddy went, oh,
S1: Sierra is the one. Yeah. Doing it now.
S2: Yeah. Um, but yeah, the, uh, I mean, the off
S2: site events are definitely where it's at, I think. I
S2: think if you're coming to RSA next year and we're coming. Right. Yeah,
S2: I'm gonna do a little bit less. Speaking honestly, I
S2: did my thing five times and I was pretty burnt. Um, again,
S2: I guess if I rewind to the beginning of the week, though. Besides,
S2: San Francisco continues to be an A plus con.
S1: Yeah.
S2: Um, I mean, besides San Francisco.
S1: Production quality.
S2: Production quality.
S1: Quality? The content?
S2: Yes. Staff is great. Yeah. You know, the villages there
S2: are cool. Even the vendors set up there is like.
S2: I just feel like it's not as nuts and in
S2: your face. I had a I had a buddy, a
S2: mutual friend of ours. Come this year. He's, you know,
S2: he's a person. He works at a company and I'm
S2: not going to put him on blast. But he's a
S2: person who works in a company. He doesn't have any
S2: purchasing power. But, you know, he put on his RSA
S2: badge the company he worked for. And it's a big company.
S2: And he is getting accosted like he's walking down the
S2: floor and like, you know, like someone out of the
S2: corner of their eye just sees his badge in the
S2: name of his company. And they're like, hey, like, you know, like,
S2: come talk to me. And like, um, it was surreal
S2: for him, even even after he tells them I don't
S2: have any purchasing power, I don't make any decisions. And
S2: they're like, I don't care.
S1: Like, yeah, yeah.
S2: Yeah, uh, but besides, doesn't feel like that. Um, I
S2: think that, uh, a mutual friend of ours, uh, Clint,
S2: gave a talk on vulnerability as people and as infosec
S2: practitioners rather than vulnerabilities as, like, you know, kind of.
S2: We pop bones or security. Security. I really love that talk.
S2: I think it's one of the best keynotes I've seen
S2: in quite a while. It will be up on the
S2: b sides website. You know, they eventually put everything out.
S2: So I highly suggest watching you and I were cameoed
S2: in that talk, actually.
S1: Yeah, absolutely.
S2: Clint, Clint talked about, um, how, you know, you and
S2: I have like this, you know, long friendship because we've
S2: worked together since we were young and we've just kind
S2: of done everything together. Yeah. And, um, he he talked
S2: about how, like, being a friend with you, he, like,
S2: felt like a little bit less than, you know, the
S2: connection we had. And he wanted that with you. Yeah.
S2: And how, like, those things are hard to talk about, right?
S2: It's like, you know, your insecurities, the way you feel. Um,
S2: but eventually, if you confront them or you figure out
S2: ways to help, you know, like, you know, be healthy
S2: about them and have conversations with your friends and say, X, Y,
S2: and Z, it's, um, it really can give you peace
S2: of mind. Superpowers make you feel better. Yeah. And so, like,
S2: you and I were referenced in that that part of
S2: that talk. And I thought that that was it was
S2: really great, actually. I actually like I think I cried
S2: at the end because he had like a couple, like
S2: little messages in there and he.
S1: Was like, yeah.
S2: Yeah. He was like, hey, you're enough, right? Like, what
S2: you're doing is enough and.
S1: All, and it matters.
S2: And it matters. Yeah. And, uh, and all of us, uh,
S2: all of us, you know, are just trying. I feel
S2: like everyone in security has a little bit of, like,
S2: they just want to, like, help the world a little bit, right? Like,
S2: not everyone, but a lot of people. That's why they
S2: get into it. Because it is easy to make that
S2: that line to like, hey, I know this is a
S2: small thing, this computer stuff, but in a way, I
S2: am a superhero trying to help the world. Right? And, um,
S2: but you can get so wrapped up because there's so
S2: much stuff, right? There's so many domains. There's new research
S2: in domains all the time you feel behind on education,
S2: you can start to get that imposter syndrome. And like
S2: at the end, Clint actually gave out cards.
S1: That he had signed.
S2: That he had signed. He had signed hundreds of cards
S2: by hand. And I think I have one in my
S2: wallet and I put mine in my wallet. And it
S2: just like says you are enough signed by Clint. Or like,
S2: you know, some inspiring message.
S1: Yeah.
S2: That was I mean, like, Clint's like amazing person. But
S2: that talk was awesome.
S1: So yeah, and a lot of courage to do a
S1: talk that's about people. Yeah. When like the natural play
S1: is like AI and security V2. Yeah. Which would absolutely
S1: crush it would crush. Yeah, yeah. And he's like, no,
S1: I'm going to do this thing because I think this
S1: message matters.
S2: Yeah, yeah, yeah. Clint's a phenomenal human. And, um, he
S2: did the girlfriend meme of, uh, of you and me.
S2: Like the, like the guys walking down the street, and
S2: he looks at like the like. Like the one girl has, like,
S2: a girlfriend, and the other girlfriend's like, you know, that
S2: meme and I got to be the the hot girl, so.
S1: That's right. You know.
S2: I haven't I've never been the hot girl. So like.
S1: Yeah, yeah.
S2: Yeah. Um. Yeah. So that was cool. Uh, what else
S2: happened this week?
S1: I don't know, what's your big takeaway? I feel like
S1: we talked about this a little bit. I feel like
S1: big takeaway for us. And keep in mind that If
S1: you're coming to this, like brand new, like you don't
S1: necessarily want to do it this way because we're we're
S1: getting lessons after I've done this for a couple of decades,
S1: but we're like more get away from the center of
S1: the mass and move into like the smaller events where
S1: your friends are going to be at to talk more
S1: about the ideas as opposed to like, where can I
S1: get the food? Where can I get the parties and
S1: the music?
S2: Yeah. And that, I mean, this thinking applies to Black
S2: Hat and Def Con, too.
S1: Absolutely.
S2: Yeah.
S1: It's the whole scene.
S2: It's, uh, at first it's cool when you're young and
S2: we're old, so.
S1: Yeah.
S2: Yeah.
S1: Um, to.
S2: To go to the parties and be part of, like,
S2: the loud noises and then now you kind of want to, uh,
S2: ration your time with people who you really want to
S2: spend time with and have conversations with and be in
S2: smaller settings where it's not so loud and like real
S2: research is going on. And, uh, and that was the
S2: dichotomy for me, was like so much of a difference
S2: between what people were saying on the floor about how
S2: things worked with AI. And then when I got to
S2: the Airbnb summit and the OpenAI summit, like, no, these
S2: are real people working on these problems. And here are
S2: the real problems. And yeah, um, and here also there
S2: are success stories, but also their failure stories, like, yeah,
S2: we thought this would work. It totally did not work.
S2: We had to go back to manual process in vulnerability management.
S2: Oh man. The conversation about vulnerability management this week were crazy, um,
S2: about using AI and the predispositions I had about what
S2: I thought was a good AI assisted vulnerability management plot
S2: or not platform, but, you know, like architecture versus what
S2: some people like Google have built and some people like
S2: Adobe did a great talk on it, and then some
S2: people talked about it in the OpenAI, um, conference. Uh,
S2: and so, like, it turns out some of the things
S2: that we thought I would be able to do are
S2: not the force multipliers. It turns out to be sending
S2: emails or like actioning tickets automatically.
S1: Go to where they.
S2: Are. Yeah, yeah, go to where they are. Right. We
S2: talked about this in the car. It's like it's I
S2: had some assumptions that we'd be able to do like full,
S2: full stop, you know, full prompts like, you know, uh,
S2: nuts to bolts send, you know, our vulnerabilities that come
S2: through a bug bounty or static code analysis or through, um,
S2: appsec testing or through, you know, any different number of
S2: where we get vulns. Right. And it would, you know,
S2: system would work. Uh, the, the, the value of the
S2: AI would be the rating, the conglomeration of all that
S2: into tickets. And it turns out that some people, at least,
S2: who have done it say actually like that doesn't work
S2: super well. Mhm. Um, they're stripping all of that contextual
S2: data about ratings making their own rating systems. Yeah. Um,
S2: and just pulling out the text from the advisories from
S2: the threat feeds, from the pen test report, from all
S2: that stuff, rewriting them themselves with custom systems that have
S2: nothing to do with CWA or CV.
S1: You know. Yeah. That's the thing. The these rating systems
S1: are trying to give us the context of the vote.
S2: They don't have any context or.
S1: Know anything about.
S2: Know anything about us.
S1: So I love this idea. Just strip it out. Yeah.
S1: And then re-add the context from the company onto the vote.
S1: And then and that's the priority.
S2: And then the thing that the Google guy said, I'm
S2: going to say Google person or guy or whatever, but, uh,
S2: because I don't remember anybody's names, but, um, uh, he
S2: was saying that like, uh, so after you do that, um, uh,
S2: the whole vulnerability management scaled by AI only works if
S2: you have a really good asset management platform. And we've
S2: been talking about this for years.
S1: Like I did that thing in 23 when I was
S1: at Robinhood.
S2: Yeah, yeah, yeah.
S1: Asset management as a center of management.
S2: Yes, exactly. Yeah, yeah. And that you presented that at
S2: the Black Hat Summit, right? Yeah. So yeah. And this
S2: turned out to be true for them. Right. It's like
S2: it's like this program does not work unless you have,
S2: you know, had your balanced breakfast of, um, you know,
S2: consolidating all your data sources where vulnerabilities come in, and
S2: then having a tremendous asset management program, like knowing where,
S2: like having, um, you know, for lack of a better word,
S2: like having knowledge of where all the systems are, you know,
S2: what they are, who owns them, what teams action them, um,
S2: where the repos are. And that's not like that. Sounds
S2: trivial to some companies who are small. Like if you're
S2: a startup, you're like, yeah, of course I know where
S2: my repos are and who owns the thing. But when
S2: you get to a company, that's the scale of Google
S2: or Apple, right? There are hundreds, if not thousands like
S2: Ubisoft too. I mean, we had productions which are video
S2: games everywhere, and it's just not simple anymore. Like you,
S2: you lose a thread on an app and then it
S2: just exists out in the wild, and then someone finds
S2: it via your bug bounty and they're like, hey, I
S2: read this thing and you're like, I have no idea
S2: what that is like. It's not. I don't see it anywhere.
S2: I don't know who owns it. And then you spend
S2: all of this toil time. That was a reoccurring term toil, right? Oh, yeah.
S2: You spend all this toil time to, like, figure that out. And, um,
S2: you should be architecting your program from the beginning with
S2: really good asset management instead of spending that toil time later.
S1: Yeah, I love that. I definitely dealt with that at
S1: Apple because it's like you put out the state of
S1: the system and a week later it's like, nothing like that. Yeah, yeah.
S1: That's true. Like that.
S2: Yeah. Yeah. The question I had for you when we
S2: were in the car was it's like, okay, so, um,
S2: so you're really big on capturing context in markdown files,
S2: which is the telos idea. Mhm. Um, and you can
S2: do telos for yourself personally as a person. Or you
S2: could telos as an organization. Right. City country country doesn't matter. Yeah. Yeah.
S2: Put down your ethos, your goals, your systems, your owners,
S2: you know, into markdown or maybe JSON or something like that,
S2: whatever you want to use.
S1: Whatever.
S2: Uh, and my question was kind of like, okay, so,
S2: so Bob, who handles, you know, Celsius app, right. Whatever. Uh,
S2: you know, he leaves and like, who is responsible for
S2: updating the context file to include those changes, you know,
S2: is there a system that you prefer or like a method?
S2: Or is it just that you have to go in
S2: there and help them and update that? Or is there like,
S2: do they have to hire a specific person to run
S2: the Telos file and make sure everything stays in line
S2: when they change their company vision or something like that?
S1: Yeah, yeah. So it's going to depend on the implementation.
S1: So the way I'm doing it commercially is for this
S1: thing called same page, which I'll be talking about later.
S1: But I think the future of this is this.
S2: Let's talk about it. Let's talk about same page right now.
S2: Let's heat up. I mean, you might as well. Right?
S2: Come on.
S1: No.
S2: Oh come on.
S1: No, I mean, it's it's the concept of the talk.
S1: It's just unified context. Now, I appreciate it, but no, um, um,
S1: but no, I think just this just becomes a unified, um,
S1: it product. Okay.
S2: So it maintains.
S1: It. So, so I think that anybody who builds anything from, like,
S1: an ice cream truck business to a security program to
S1: I want to be a governor, they're going to have
S1: a core system, which is all assets, all context, all goals, everything.
S2: So one thing that I didn't realize until starting to
S2: build very, uh. And I'm not afraid to admit it,
S2: vibe coded things. Right. Like so I will now have
S2: superpowers because I understand code, right? I understand architecture of code.
S2: I understand problems in code, I understand security, but I've
S2: never been a front end developer. I couldn't sit down
S2: with react and build a build a pretty website. If
S2: you put me on a modern development team, I would die. Yeah.
S1: Same. Same.
S2: But because I know about code, I know how to script.
S2: And I know the concept of pretty much every language
S2: from assessing it in security. I can now build fantastic
S2: things very quickly. Same. The thing is, is that I
S2: think I've realized in that world is that prds like
S2: product requirements documents are necessary for so many more things
S2: than I ever thought. Right. Like. And so the idea,
S2: the way it connects to vibe Coding is like whenever
S2: I do a new project now, the first thing I
S2: do is I verbally talk to my browser with a
S2: Chrome extension into an AI model with a whole bunch
S2: of notes about just kind of what I want the
S2: system to do, what tools it's tying together, like how
S2: it's presenting data, why we're even making this, what problem
S2: it solves. And that's just verbal garbage coming out of
S2: my mouth, right? Like I'm just having a conversation. I
S2: could take a podcast like this and like, do that.
S2: And then I'm feeding into a whole bunch of AIS
S2: to make a product document with requirements in it. And
S2: then I'm creating a technical architecture document as well, which
S2: is why we are choosing the frameworks that we're using,
S2: why we're choosing the tools. Never deviate from these. And
S2: so those two things in concert, especially in the vibe
S2: coding or AI assisted coding world, have helped make my
S2: software infinitely better and helped the AI. I'd stay on
S2: track with the mission. With the technology.
S1: Yeah, because when it loses its context and loses its
S1: mind and basically gets erased. Yeah, it just goes back
S1: to that. Starts over.
S2: Yeah. And in vibe code, you can have such I
S2: mean we're going off on a rathole now, but I.
S1: Mean, that's fine.
S2: In coding, you have that sidebar, right? And that you
S2: can stay in that conversation for a long time and
S2: not realize that you're you're hitting the point of where
S2: needle in the haystack is not you're not getting good value.
S2: And you need to.
S1: And they're also like $15 queries.
S2: Yeah, exactly. Yeah.
S1: $15, $24. Okay. Wait a minute. So if I do
S1: four of these, that's 100 bucks.
S2: Yeah. Yeah. I mean, I was talking about it with
S2: some friends, uh, in discord, and it's like or in,
S2: in a signal chat, and I'm like, how much are
S2: you guys spending on your AI subscriptions a month? Because
S2: mine is approaching a car payment, and it's totally. I
S2: know it's worth it, but it's still kind of painful
S2: to add another car payment, you know? So, uh, I mean,
S2: I'm using everything. I'm using Gemini, I'm using OpenAI's ecosystem,
S2: I'm using Claude's ecosystem. I still have perplexity. I'm a
S2: really hype user of Manus right now. Um, I love
S2: Lambda Chat's implementation of deep seq because they host it
S2: on the internet, and I can scrape it with puppeteer. Playwright. Um, so, yeah,
S2: I'm just hitting everything for everything.
S1: Yeah, yeah, I'm doing a lot of na to n, uh,
S1: for back end and, uh, bedrock, um, still use fabric
S1: like most of the models. Um.
S2: Oh, yeah. When I'm on the command line using fabrics. Yeah, yeah.
S1: Favorite models. Uh, right now, uh, uh, for me, it's, um,
S1: two five, two five pro for a Gemini.
S2: Oh, yeah. Gemini is on.
S1: And then, um, O3.
S2: Okay.
S1: With memory.
S2: With memory. Okay. So I use O3 with memory for
S2: writing tasks. I think that's really good at writing and
S2: researching tasks. Um, I actually am one of the believers
S2: that the biggest deep sea model. It's one of the
S2: best models that I've ever seen for research tasks, even
S2: though it doesn't have search enabled. Um, I use it,
S2: it's exposed for free through Lambda Chat. Um, so you
S2: can go and so like you have to think about
S2: there are different releases of R1, deep R1, and most
S2: of us played with the middle implementations. Uh like the.
S1: Yeah.
S2: Yeah. Like the.
S1: Not the.
S2: Full, not the full. Six one. 7 billion parameter one. Right.
S2: They have that hosted on their on their architecture for free. Yeah.
S2: And it is fantastic to use um, and uh, I
S2: find that model to be really, really good.
S1: Let me check real quick. Yeah.
S2: Let's see what grok is running. I don't think they're
S2: even running 617. I think he just pulled down right
S2: there on the compound base mini where it says right there.
S1: Yeah.
S2: Oh.
S1: Oh, here we go.
S2: Yeah, yeah. See, so they're doing deep seek 77 DB.
S1: No comparison.
S2: Yeah.
S1: No.
S2: Yeah 601 billion parameters. So um yeah. So I use
S2: that model. It it's really good research model but it
S2: doesn't have search enabled. Right. So that's, it's, it's uh
S2: missing and then so Gemini uh, deep 6617 and then
S2: I really get a lot out of, um, uh, when
S2: I code, you know, 3.7 on Claude is really good. Um,
S2: although it started to go kind of haywire recently, I.
S1: I switched off. I, I went from 3.5 to 3.7,
S1: and I liked it a lot, but 2.5 from Gemini
S1: came right after that.
S2: It did.
S1: Yeah. And I was just like, damn, that's really good.
S1: And then then oh three roughly same time. Yeah. Yeah.
S1: So I've kind of been messing mostly with those.
S2: That's the problem is like I want to consolidate subscriptions.
S2: Like I want to just say, oh, I'm just going
S2: to stick with this, but I can't, because if you're
S2: at the cutting edge of using this stuff every day,
S2: it's like you want the best model for the best
S2: thing all the time.
S1: And there's like two releases per week. So you're just like,
S1: it's not.
S2: Yeah, yeah, yeah. And so yeah, you end up spending
S2: somewhere in the order of 400 to $600 a month
S2: for all your subscriptions, you know, not including your I mean,
S2: that's including everything that's including your, um, your direct chat
S2: interfaces that you're paying for, like ChatGPT implementations, but also
S2: your API calls and then also your subscription to Klein
S2: or not Klein, but windsurf or um, um, what's the
S2: other one? I use windsurf, sorry. I always forget the
S2: name of the other one. Sea.
S1: Klein.
S2: No. Oh, yeah. Klein. But. Cursor, cursor. There you go. Cursor. Yeah. Um,
S2: so you have subscription to one of those two, and
S2: it's like, okay. Yeah. Now I'm paying quite a bit.
S1: I just realized, uh, a link between coding and the, um,
S1: AI competition. Mhm. Which model are you using versus which
S1: scaffolding do you have in the form of giving that
S1: cursor rules prompt.
S2: Yeah.
S1: Yeah. Right. So it's like I strongly believe and I
S1: Jason talks about this I forget his name is Jason Wong.
S1: Is that his name I don't.
S2: Know but he's brilliant I love that guy.
S1: Yeah. So he makes like really, really practical videos. Yeah. Um,
S1: so he basically, um, put out some stuff. Cursor rules,
S1: like a single prompt that can generate that full PRD
S1: with a checklist. Yeah. And the model actually goes and
S1: checks things off the list.
S2: Yeah, I do the same way. So I use Berman's
S2: or uh, um, the I got is it Berman I
S2: can't remember. What is his last name. The AI content creator.
S2: You were on his show. He talked about fabric. Matt.
S1: Matthew.
S2: Matthew. Yeah. Okay, so. So, Matt Berman did a or.
S2: Matthew Berman did a show on this as well? Yep.
S2: And he had a set of questions that he put
S2: in the show. But he never like released a GitHub
S2: or a gist or anything like that.
S1: That's the thing I love about AI, JSON. He's like boom,
S1: it's on GitHub.
S2: Yeah, it's on GitHub. Right. So I went and grabbed
S2: Matt Berman's and I tweeted, I just tweeted it out
S2: like I had. I transcribe and then pull up the questions.
S2: And then now I in that prompt to the AI
S2: I'm like, here is a, here is the structure of
S2: a PRD I want to build. I need you to
S2: ask me relevant questions to fill in the thing. And
S2: then I have it ask me the questions and then we're,
S2: you know, we have a chat.
S1: So it builds a PRD. Based on your interview?
S2: Based on my interview, it interviews, I tell it to
S2: interview me.
S1: Yeah.
S2: And then once we're done, then it's like, cool. Do
S2: you want me to stitch all this together into a
S2: fully functional PRD? I'm like, yes. And then we move
S2: on to the architecture section, and then those live as
S2: markdown files in my project. And so when I need
S2: to start a new chat, because I know the context
S2: window is filling up for windsurf, I start a new
S2: chat and I say reanalyze our, you know, our core,
S2: our core architecture and our PRD so that you understand
S2: everything about this project and the readme that we built. Yeah, yeah.
S2: And then it's like it's like I'm starting again with
S2: great context. And you know, obviously you need to hook
S2: it up to version control as well. So if anything
S2: goes haywire gets deleted, you can snapshot back. But um,
S2: but those are some pro tips for the coding people
S2: out there.
S1: Yeah, yeah I've got a a rule where I basically
S1: say in the cursor rules. If I say this, it
S1: means go and review.
S2: Okay, cool.
S1: So if I feel like it's going wonky. Yeah, it's
S1: just like a reset. Yeah. Um, I wanted to mention
S1: one thing.
S2: You don't say it like, make it so. Like Picard.
S1: You could do that. Yeah, absolutely. Um, so I wanted
S1: to give, uh, a little shout out to, uh, Caleb
S1: Sima for a post he did before RSA, where he
S1: was complaining about, um, panels, just like being kind of empty. Yeah. Um,
S1: and I was just thinking about this. The whole reason,
S1: these long form conversations. Because we're just riffing. Yeah. And
S1: we're just thinking of stuff. But ideally, the stuff we're
S1: thinking of is stuff that was new to us. Therefore,
S1: it's going to be new to them. Yeah. And what
S1: happens with so many talks and so many panels? And
S1: I think this is like a meta conversation that we
S1: just really need to solve. Um, obviously for us as
S1: content creators, which I think we do a good job. Yeah.
S1: But I would say the industry needs to solve this
S1: is don't just come on and be like, you know,
S1: here's the thing, Jason. Like, the landscape is changing. You know. Yeah.
S1: You know, AI is changing everything. It's changing the game. Yeah.
S1: And you're like, yeah, it's just changing the game. We're like,
S1: all right, that's all the time we got. Yeah.
S2: Everybody let's go to lunch.
S1: Yeah. So it's like two people reflecting backwards or maybe
S1: a panel of four. Yeah. And they're just reflecting back
S1: these things that we've heard for the last few months
S1: or a couple of years. Yeah. It's like, um, was
S1: it Matt or somebody we know was like, if I
S1: hear one more person say, um, the attacker only has
S1: to be right once, and the defender has to be
S1: right all the time. It's like we learned this ten
S1: years ago. Yeah. So the first time we heard it,
S1: it was hella smart. Yeah.
S2: Yeah. It was great.
S1: Yeah. And now it's just like so many panels, so
S1: many conferences, so many talks are just that.
S2: Yeah, yeah. It's, uh, it's that it's that there's no
S2: contention anymore on panels like. Yeah. I mean, even the
S2: ones at OpenAI, when we went to that, there was
S2: like towards the end, it was a lot of like, yeah,
S2: yeah we agree. And so I think at the end
S2: I was like trying to spice it up a little bit.
S2: Like I wasn't the moderator but I asked a question.
S2: I'm like, who's your most feared competitor?
S1: That was.
S2: Like one.
S1: Of the.
S2: Best questions. And everybody was like, oh shit. And then like,
S2: you know, the, you know, like, uh, the guy at
S2: the end, that.
S1: Was one of the best questions.
S2: It's definitely Dan and Trail of Bits and Trail of
S2: Bits is like, oh, we're really scared of like whoever
S2: at the end, he's like, he's like, I don't know,
S2: I'm just doing my best over here.
S1: Yeah. Trying to survive. Yeah.
S2: Try and survive. And, uh, yeah, I wish there was
S2: a little bit more contentious. And you can you can
S2: talk about that. I've had interview panelists before or the
S2: people leading whatever moderators or whatever. Yeah, I've had them
S2: be like, hey, I'm going to ask this question. And
S2: if you all answer the same fucking thing, I'm gonna
S2: put you off the panel. And I love that. I'm like, yeah, yeah.
S1: So another of our mutual friends actually engineers this into
S1: the thing. Oh, really? Sasha.
S2: Oh, yeah. He does. Yeah.
S1: Sasha is like, look, if I'm going to be on this.
S2: I didn't get to see Sasha for more than ten minutes.
S2: This whole.
S1: Me neither. I just yeah, just like he was a blur.
S2: A blur.
S1: Yeah, yeah. But, like, he's like, if I sit on
S1: this panel, I'm going to engineer a thing. I need
S1: to know what you all believe. Yeah, so I can
S1: find something I disagree with. Yeah. Otherwise, this is going
S1: to be the dumbest panel ever.
S2: And it and it works on me wonderfully because like,
S2: when people don't agree with me, I just get angry. Yeah.
S1: No kidding. Like I'm like, I'm like.
S2: No, you're.
S1: Wrong. Like, we're good at this. We're good. We're good
S1: at having a good, you know?
S2: Yeah, yeah. No, I.
S1: Was having a go.
S2: I was, uh, I mean, while doing RSA, I was
S2: tweeting a little bit, um, on on Twitter. And I
S2: happen to have, uh, a friend who was rooming with
S2: me who was doing security research. So he's he found
S2: a zero day, basically reported it to the vendor, but
S2: a whole bunch of individual companies that implemented the software. And, um,
S2: they have not yet fixed the bug. So he goes
S2: out to all these bug bounty programs and submits it,
S2: and also a whole bunch of vulnerability disclosure programs.
S1: Mhm.
S2: And um, and he submits it to the vulnerability disclosure programs.
S2: And he comes back to me because, you know, I'm
S2: a former bug bounty guy. Right. And he's like he's like, yo,
S2: is it like, you know, like normal? Like the crits
S2: that I got for people who have bounties? I got
S2: 50 points on the platform for hacker one, but I
S2: only get seven for responsibly disclosing it without getting paid.
S2: He's like, that's weird. So I go and tweet about it.
S2: And admittedly, I didn't tweet right. I don't think like
S2: I definitely went out and kind of sensationalized it a
S2: little bit. I'm like, this sucks. Like, give the guy
S2: more than seven points, right? But the bug bounty community
S2: of which are a lot of my homies, like, basically
S2: jumped on me and and it like got in my
S2: head a little bit. But then I managed to like
S2: push it down a little bit, but like, you know,
S2: the bug bounty. This is a huge thing this week.
S2: Huge discussion is just like, you know, VDP is evil,
S2: VDP is labor exploitation, VDP is everything wrong with bug bounty? Jason,
S2: you're the worst person in the world for um, for
S2: promoting that. Anyone ever even think about VDP. And so
S2: then I came out with this post, uh, you know,
S2: a guy was trying to debate this with me, and
S2: he's he's a smart dude, smart bug hunter. I have
S2: respect for him. But he, like, said one sentence and
S2: it was it felt a little aggressive to me. And
S2: I was just in that mood and I was like,
S2: fuck it. Block like. And I just block this dude. And, um.
S2: And then he, like, went off the deep end and started, like,
S2: posting more and more and like, you know, I can't
S2: believe Jason did it. Turned out he had been to
S2: my class, to which I felt bad about. Oh, um,
S2: and so I unblocked and tried to have a conversation.
S2: But just like there's so much vitriol on this topic
S2: of labor exploitation. So I went out and I said, hey,
S2: I believe in VDP. Actually, I believe there are plenty
S2: of companies who on ramp with a VDP and then
S2: start paying for bugs. I have seen them. I have
S2: worked at Bugcrowd. This is not a fictional thing, right? Uh,
S2: some companies are too big that if they were just
S2: to open up. I mean, Ubisoft was one of these companies, right?
S2: If you would have opened up a bug bounty for
S2: everything on Ubisoft, all scope, all all classifications of bugs,
S2: they would have went bankrupt, right? Yeah. And so they
S2: had to start with the VDP to burn down a
S2: little bit of the stuff, incentivize people to come on
S2: and then eventually move into a paid program. Yeah. And
S2: then they ended up shutting down the program. Um, but
S2: a lot of companies work like that, and no one's
S2: forcing you to work on a PDP, right? Like, I wasn't,
S2: I wasn't.
S1: Can you for everyone just give an overview of the
S1: difference between the two?
S2: Yes. Okay. So a bug bounty program is a program
S2: where you pay for bugs, right? Like a researcher on
S2: the internet comes out and says, I found a web
S2: application vulnerability with your software and you've said, yes, I'm,
S2: I'm paying for these usually on a platform like Bugcrowd
S2: or Hackerone or integrity. And they come in the platform,
S2: they submit bug, you pay them, right. VDP is called
S2: a vulnerability disclosure program, where you as a company don't
S2: have like an email box or anything like that to
S2: take in vulnerabilities. And so usually the platform handles it
S2: for you and you show up on their platform as
S2: a card that says, hey, we do not pay for anything,
S2: but if you find something, report it to this program. Right. Yeah. Um,
S2: and we will, uh, give you props, you know, we will.
S1: Find some kind of reward, but we don't have a
S1: mountain of cash.
S2: Yeah, we don't have a mountain of cash. Right? And
S2: it's just. It feels like bug hunters are, you know?
S2: You know, this guy was trying to cite, like, he's like,
S2: no more free bugs thing. And like, you know, like, listen, man,
S2: I was there when that happened. I mean, I mean,
S2: I don't know if this guy was even born then,
S2: but he's citing this. But I was there when no
S2: more free bugs was happening. And I don't think that
S2: was the core message completely. I think it was I
S2: think it was that like, you know, there should be
S2: fairness in disclosure, there should be fairness in credit of vulnerabilities.
S2: There shouldn't be people suing each other when vulnerability research
S2: is discussed. Um, and so I wrote this big long
S2: thing and I just got railed this week like by the,
S2: by my own community. And I felt I felt really
S2: attacked and I was like, fuck it, I'm not going
S2: to do bug bounty stuff anymore. Like I'm hosting the
S2: Hong Kong and it's like, it's like you guys are
S2: some angry people. Like, if you don't want to work
S2: on Vdps don't work on Vdps, right. But like the
S2: other thing I didn't say in that thread was that, um,
S2: was that a lot of people I mentor, they don't
S2: have their foot in the door yet. They haven't had
S2: a job yet. Right. And so when they go to interview,
S2: if they have bug bounty experience, that's awesome. But usually
S2: when you're starting to find a bug on a bug,
S2: bounty is much harder than to find a real bug
S2: on a PDP, right? Because there's so much more scope.
S2: Not everybody is testing on the vdps there. See something?
S2: Say something. Right? So if you find something on a
S2: VDP and you're doing stuff like Portswigger Labs and Hack
S2: the box and all that stuff, it's like that shows
S2: me as an employer, well, they've got the skills, right.
S2: They have the skills. They just need a chance. Right?
S1: And it's receipts.
S2: It's like.
S1: It's.
S2: Receipts. Yeah. And, you know, I didn't even want to
S2: go into that part of the argument on Twitter because
S2: I'll get flamed more. But, um, yeah, I mean, it
S2: goes to show that like, uh, you know, like I
S2: drew that corollary from the panel thing you were talking about, right?
S2: It's like it's like, uh, you can't just agree with everyone.
S2: You have to have a point of view. Yeah. We're
S2: talking about this in content creation, right?
S1: Like, totally.
S2: So so here's this idea. And this is also off
S2: the beaten path. But, uh, for the listeners. But Dan
S2: and I are content creators, right? And, and we are
S2: lucky enough to have, you know, 25, 30 years in
S2: the industry, almost like, um, and so, you know, 20
S2: years for me. Um, and so.
S1: Can I stop you even before. Yeah. So I have
S1: like a pre point for this.
S2: Okay. Go ahead.
S1: Yeah. So well make sure you keep this thread because
S1: this is an amazing thread. So I want I want
S1: to make a quick point I can't believe this is
S1: going in this direction. But so a lot of young
S1: people come to us and they're like how do I
S1: become a YouTuber? Yeah. How do I become a content
S1: creator on Twitter or whatever? The first thing I tell them,
S1: which is exactly your point, learn something so that you
S1: have an opinion about it. Yeah. Because I don't I
S1: do create content and you create content. But I wouldn't
S1: say that fundamentally, that's what we are. There are a
S1: lot of people who are content creators, and I feel
S1: like that is their actual job. So they are looking
S1: for content whereas we are building and doing things. Mhm.
S1: You got a training company, you got a consulting company.
S1: You're doing that. The content falls out of it.
S2: Yes. Correct.
S1: Yeah. Right. And I feel like that is so key
S1: especially for young people. You have to like get good
S1: at something.
S2: Yeah, well, it's not even. It's not that you have
S2: to get good at something. It's that you have to
S2: do something.
S1: Have an opinion, have a path. Try something. Yeah. And
S1: the content emerges from that.
S2: Yeah. I mean, one thing. Uh, so, Clint, you know,
S2: our friend at Semgroup, he did an interview with me
S2: later this week on the Semgroup blog. That'll come out
S2: in a week or so, but it was mostly about
S2: career stuff. And what I was telling him is like,
S2: it's like when you're new, it's really hard to show
S2: that portfolio of work, you know, and you can do
S2: it with Vdps. You can do with Bug Bounty, you
S2: can do it with CVS. You can do it by
S2: taking tryhackme and getting certs and stuff like that. But
S2: some of that stuff is paid work, right? Or it's
S2: paid like some of that training is paid. And when
S2: you really have nothing, you're going to have to just
S2: focus on the free resources. But one of the things
S2: that you can do with nothing at all is start
S2: up a blog and talk about your learning experience. And,
S2: you know, you said, good, I'm going to challenge with you.
S2: Just have to do something. Yeah, do something and write
S2: about it and have an opinion and talk about your
S2: learning experience. And so like when I read a blog about.
S1: And be vulnerable.
S2: And be vulnerable.
S1: Yeah. Like you were saying earlier, talk about the bad,
S1: talk about the good.
S2: Yeah, yeah. And you know, so like when I see
S2: a blog on someone who's writing about their first, first
S2: usage of using Burp Suite, right? And it's like, yeah,
S2: I've read that blog 800 times, but I haven't read
S2: it from you.
S1: That's right.
S2: And you're talking about like, your thought process and like,
S2: you know, learning this tool or you're doing a YouTube
S2: video on it or something like that, I go look
S2: at that stuff in the interview pipeline. Right. It's an
S2: it's an additional thing in your portfolio. And it always
S2: makes me feel good to watch somebody trying to learn something,
S2: you know, and, and I feel like. So it's not
S2: that you have to be good at something. It's just
S2: you have to do something and and that becomes your content,
S2: I feel.
S1: Yeah. And don't worry about overlaps.
S2: Yeah. Everyone.
S1: Because Jason and I are so similar. Yeah. If we
S1: tried to make the same piece of content, we wouldn't.
S2: And then we get angry at each.
S1: Other because we have we, we have like the, we
S1: have very similar. But it's still going to come out different. Yeah.
S1: In a Jason way. In a Daniel way. Yeah. Yeah.
S1: And that. So you shouldn't have to worry that someone
S1: already wrote the burp tutorial.
S2: Yeah, yeah. Yeah, exactly. So the other one I was
S2: talking about is, um, I think I was, I was
S2: trying to explain is that, uh, is that in content
S2: creation also? I mean, like, I don't know if we
S2: call ourselves content creators or whatever.
S1: I mean, we definitely are, but it's not like my
S1: main identity.
S2: But I see so many people having like, being content
S2: creators for the the point of being a content creator
S2: and struggling to find content because they're not doing work.
S1: There you go. That's it. That's that's my point. Yeah.
S2: On top of that, not having an opinion. And I
S2: think that.
S1: That's like the worst.
S2: Yes, I think that that, you know, it goes into
S2: the panel thing we were talking about. Right. It's like
S2: you need in this industry, if you're going to be
S2: doing content in opinion of some sort, you need a voice. Um,
S2: and it doesn't have to agree with everyone. You could
S2: be the antithesis. Like if you want to be the
S2: anti Jay Haddock's guy and say that like, vdps are,
S2: you know, exploitative work and like, you know.
S1: At least you're saying something.
S2: At least you're saying something like, I respect that guy.
S2: You know, like, right. Like he has an opinion. Hurts
S2: my feelings a little bit, but he has an opinion,
S2: you know, and it's like.
S1: Especially, especially given what you've done for bounty.
S2: Yeah yeah yeah yeah, yeah. Um, but but yeah, having
S2: having an opinion is is almost more important than content
S2: these days, I think. I think content comes from your opinion.
S1: So 100% correct. Another way that gets described is taste.
S2: Yeah.
S1: Yeah. So so imagine this I talk about this a lot,
S1: but it's like a few years from now or maybe months.
S1: Who knows how fast this stuff is moving. But, um,
S1: you just like you do with, uh, narration of code,
S1: you're talking to cursor. And on all these screens, it's
S1: popping up. You mean like this? You mean like this?
S1: Or you're having it make you an anime series? Yeah. Or,
S1: you know, a book or whatever. Do you mean like this?
S1: And you're just swiping through? No, not like that. Not
S1: like that. That's the one I like.
S2: Yeah, that's the one I like.
S1: Yeah. And that taste. Oh, this reminds me of Rick Rubin, actually. Um,
S1: so he produced Slayer, a whole bunch of rap albums.
S1: He's like the most famous producer ever. Okay. Doesn't play
S1: an instrument. Really can't read music. Plays no instruments.
S2: Wow.
S1: And they say, how do you get hired to do this?
S1: He's like, I know exactly what I like.
S2: That's amazing. Right?
S1: Yeah. So to your point, like opinion taste.
S2: Yeah. I mean, that was we were talking in the
S2: car about the branding for Arcanum for my thing. And
S2: one of the most freeing things for me, being the
S2: CEO of this company now is I have complete control
S2: over branding, marketing message what we do and do not
S2: do to enable those things. Right? Like our strategy. I'll
S2: give you an example. It's a freebie for any company
S2: out there. Um, but, uh, where all of your marketing
S2: teams are telling you, like, you should be, you know, chasing,
S2: you know, inbound leads, and you're going to use BDR
S2: based marketing and stuff like that. That's that's all dead like, uh,
S2: if you if you want to be a infosec company
S2: in 2025, it is influencer marketing and it is especially
S2: SME based marketing. And this is a lost art. Training
S2: someone at your company like when your sales engineers or
S2: maybe you're small enough, it has to be your CTO
S2: or CEO. But then going to conferences and this is
S2: what I do, and speaking on topics, not selling anything,
S2: contributing to the community of security. We get so many
S2: warm leads from this dude. My talks like I get
S2: 5 or 6 people right after my talks are like,
S2: let's work together, give me your card. Like you know.
S1: And you're not even you're not even saying.
S2: Not saying anything.
S1: Julia is.
S2: Yeah. Julia is. Yeah. Julia. My wife is like, hey,
S2: we sell things, you know? But but that's that's our
S2: vibe is we go to conferences. I mean, I'm traveling
S2: to one a month and it's a lot of travel
S2: and it's hard, but marketing people have been pushing this
S2: away like, oh, well, we have to, like, train the
S2: expert and set up the event, and then we have to, um,
S2: and then we have to buy a hotel room and
S2: flights and that's all really expensive. We'd rather spend that
S2: on like bdrs cold calling people. I'm like, that is
S2: the opposite way of how this industry works.
S1: You figured this out really early and you locked on
S1: and it's working.
S2: Yeah, I figured this out early and and again, like
S2: the creative control to for our brand, it looks like
S2: a metal brand, right. Like our canon font. Looks like
S2: Metallica font. And the.
S1: Purple.
S2: The purple and the Spectre logo and all this stuff
S2: was like, before I used to have to go in
S2: front of a, you know, like a panel of people,
S2: the CEO, the CTO, the, you know, the marketer, the brander,
S2: and be like, here's my idea for this cool thing.
S2: I know it'll crush all of our competition. And they'd
S2: be like, ah, I don't know. I don't like that.
S2: And I'm like, I know the security industry like, this
S2: is what they want. Like they want cool shit. Like,
S2: you know, another example is like when we put our
S2: our name out there with them, like it looks like
S2: Metallica font. So it says Arcanum, right? We don't put
S2: our website, we don't put anything. The goal is for
S2: you to wear our shirt, that you could just wear
S2: it out every day. And nobody knows. You're marketing a
S2: security company and many people are very anti that. They're like,
S2: you have to have your website, you have to have
S2: something to tie people back. And I'm like, no, eventually
S2: someone's going to ask that person, where did you get
S2: that badass shirt? And they're going to be like, oh,
S2: I went to a security conference. There's actually this hacking
S2: outfit that does training called Arcanum. That person works in it.
S2: They're like, oh, I'll go check that out. And like,
S2: and it's like word of mouth, basically.
S1: In the meantime, it just looks cool.
S2: And it just looks cool. You wear the shirt every day.
S2: And so like there are a whole bunch of like
S2: micro tricks like that inside of marketing, branding, having an opinion,
S2: building a company that I feel like I have just
S2: seen regular marketing people from other domains just come in
S2: and not get in the security industry. And so that's
S2: why it works for us, I feel like. Yeah.
S1: Well, dude, we finally did it.
S2: We did we we sat down and did a thing.
S2: I think three last saw were like, we should just record,
S2: go up to the room.
S1: We, we've talked about. So what will happen is we'll
S1: get on the phone. Yeah. And we'll do exactly this. Yeah.
S1: We should have recorded. We should have recorded that.
S2: Yeah. Yeah.
S1: That's happened 29 times. Yeah, exactly. And then we're like, no,
S1: we got to get on the mic and we got
S1: to do it. And it's like never happened. Yeah, we
S1: did the one for a vendor. Yeah, but that wasn't this. Yeah, yeah.
S2: Um, okay, I got I got one for you before
S2: we leave. I got so your opinion on Thunderbolts. We
S2: went to go see the new Marvel movie, uh, two
S2: nights ago.
S1: I'm struggling to.
S2: Tell the people. Tell the.
S1: People? I'm struggling. I'm struggling.
S2: I loved it.
S1: So I liked the main character. I liked, um, you
S1: know Johansson's sister?
S2: Yeah.
S1: Yeah, yeah. She was like, the strength, um, the father
S1: daughter thing. The father daughter.
S2: Thing was strong.
S1: Yeah, yeah, yeah, I like that a lot. Um, I
S1: don't know, I just like so many things about the
S1: old franchise. Yeah, I just feel sad. Yeah, when I
S1: think that, like, they're not around anymore.
S2: Yeah.
S1: First, I have a question. Where'd they go?
S2: I don't.
S1: Know. Why can't they come back?
S2: I think Thor left off planet um, because he has,
S2: like a daughter now, right? And he's on, like, other planets.
S2: Like saving them, not Earth.
S1: The. Okay, so multiverse freaks me out.
S2: Yeah, the multiverse I was not super down with. Honestly.
S1: Like, you know, here's my problem with the multiverse it
S1: invalidates death. It invalidates things you're supposed to care about. Yeah. Sacrifice. Yeah. Yeah.
S1: If someone can, like, snap and just people come back,
S1: I'm like, yeah, what are we even doing here?
S2: Yeah. It's also just hard to keep up all the
S2: different multiverse plot lines. And, um.
S1: Is that person alive or dead? Well, not in this universe.
S2: Yeah, exactly. Yeah. Uh, we were talking about the differences
S2: between comic, like, adaptations of comics and movies, and then
S2: what actually happens in the comic books? Pretty much. I
S2: feel like is universally better in the comics. Uh, except
S2: for a couple things. Uh, I think that there have
S2: been a couple of movies that they've done, origin stories
S2: that were better than the comics, but we were talking
S2: about Bane and how much different Bane is in comics
S2: versus Superman two. Yeah. Superman two. Yeah. Um, How different the.
S2: The death or death of Batman. But the breaking of
S2: Batman was in the comics with Bane versus the movies,
S2: and then the death of Superman in pretty much everything. Um, like.
S2: Like there was a whole bunch of stuff that I
S2: guess you just don't have time to do, you know, like, uh,
S2: like basically, like, before doomsday killed Superman in, you know,
S2: in the comics.
S1: Spoiler.
S2: So. Oh, yeah. I mean, if you haven't, you haven't
S2: read the comics, you should. But, uh, this was when
S2: I was a kid. I mean, doomsday ran amok on
S2: the on, like, you know, in America, like. And he
S2: just he wrecked every single person in the Justice League. Like, just,
S2: like slapped them down. And so that that, like, builds
S2: this context of, like, six issues up to the point
S2: where you're like, oh, man, the only person that's going
S2: to be able to stop this villain.
S1: Yup.
S2: You know, the combined might of the Justice League has
S2: done nothing. It's only going to be Superman. Yeah. And so, like,
S2: it breeds this, you know, like up and coming crescendo
S2: of battle. And then, you know, you have the epic
S2: battle between Doomsday and Superman and he dies. Um, and,
S2: you know, one of the things I like to talk
S2: about is when I was a kid, that issue came
S2: in a plastic bag with the Superman logo with blood
S2: dripping down it. No way. And included in the plastic
S2: bag was the comic of his death. It was all
S2: about his funeral, him dying the world without Superman. And
S2: it came with a black armband with the Superman logo
S2: on it so.
S1: You.
S2: Could mourn him. And it was made of, like, vinyl
S2: and so you could put it on your arm. I
S2: wore that to school that day when Superman and some
S2: of my nerd friends did, too. It was.
S1: And you were Superman in your wedding?
S2: I was Superman in my wedding. We all wore superhero
S2: shirts under, uh, under a thing as I was Superman.
S2: And we were talking. Oh, we watched that video last
S2: night of, like, the iterations of all the Superman.
S1: Oh, yeah. Yeah, there.
S2: Were a couple Superman I've never seen before, by the way.
S1: Yeah. And you're getting me on the the one that
S1: you like.
S2: Yeah. Um. Superman and Lois, uh, the two two season
S2: one I thought was fantastic. I think he's one of
S2: the best Superman's I've ever seen. Um, and then I'm
S2: getting you on arcane, too.
S1: Yeah.
S2: That's right, arcane is fantastic.
S1: Like.
S2: Like like in a few. Yeah, yeah, yeah. So, um. Yeah,
S2: I mean, if you wanted some nerd segment that was,
S2: you know.
S1: Yeah. And then a week from now, we go on
S1: our spiritual retreat.
S2: We do? Yeah. We do.
S1: And can't wait to see you there.
S2: Yeah. We, uh, we do EDC every year in Las Vegas.
S2: It's three days where we try not to talk about work. Yup.
S2: And just listen to music and be best friends. Outside
S2: of that, I highly recommend for any of you who have,
S2: you know, friends, who's who sit in the industry and
S2: you kind of go and hang out with them, and
S2: you end up talking a lot about infosec and work
S2: to plan something that's not work related. Like, you and
S2: I have EDC, and I'm trying to build something with
S2: Kev where I go to like a comic con, you know? Yeah. Um,
S2: and you're more than welcome to come if you want, but, uh.
S1: I've been once, I think.
S2: Yeah, I like comic cons a lot, but, um, but
S2: just that activity of doing something outside of infosec is
S2: really nice. So.
S1: Yeah. And then we start with work, but really, it
S1: ends up being like life plans. Yeah. And how we're
S1: helping each other.
S2: Yeah, exactly. Yeah. Yeah.
S1: So we'll do. This was fantastic.
S2: Awesome. Yeah. Hopefully.
S1: Hopefully it was recording.
S2: Yeah. Hopefully. Yeah. I see the little thing. So. Yeah.
S2: For sure.