S1: All right, Michael, welcome to unsupervised learning.
S2: Hey, it's great to be here. Thanks for having me.
S1: Yeah. So, uh, lots to talk about here. Uh, can
S1: you give a quick intro on yourself?
S2: Yeah, sure. So, uh, my name is Michael Brown. I'm
S2: a principal security engineer at trilobites. I lead up our
S2: company's AI and ML security research group. We really focus
S2: on two kinds of, uh, intersections between AI, ML, and security.
S2: It's primarily using AIML technologies to solve traditional cybersecurity problems
S2: that are really hairy and really kind of sticky, and
S2: conventional methods have kind of failed to address. And then
S2: we also, uh, to a smaller degree, look at, um,
S2: the security of AIML based systems. So, um, I was
S2: also the lead designer, um, in team lead for um,
S2: trilobites team that entered into the AI Cyber Challenge. Uh,
S2: we built the tool called Buttercup, which took second place
S2: in And overall in the iacc. And, um. Yeah, that's
S2: about it.
S1: Yeah. That's perfect. And that's exactly what I'd like to
S1: chat about. Um, so I guess, um, I guess the
S1: thing I'm most interested in is, uh, just the design
S1: of the system, and, um, I guess overall, what you
S1: know about the designs of the other system. So design
S1: versus design, system versus system. What? Whatever you want to
S1: share or can share. Like what? What are your thoughts
S1: on that? Um, I guess everyone releases open source. So
S1: maybe you've had a chance to look at some of
S1: the other offerings. Maybe you've heard them talking, maybe you know,
S1: the teams. Uh, so I guess what kind of Intel
S1: do you have on what everyone else was doing versus
S1: what you guys were doing? And how do you think
S1: that went?
S2: Yeah. Well, um, yeah, I guess I can answer that
S2: last part pretty easily. It went pretty well for us. Um,
S2: so we took second place. Uh, the team that finished
S2: in first. Team Atlanta. Um, they had a pretty similar
S2: setup to ours. Um, they had more components, more moving parts, uh,
S2: more pieces. They had more hands. Um, larger team to
S2: be able to kind of implement more, um, but ultimately
S2: they had a really similar kind of set of design principles, um,
S2: that worked out for us, the third place finishing team theory, they, um,
S2: had a bit of a deviation in terms of like
S2: their conceptual, uh, principles that guided how they built their system.
S2: But I can get into that in a bit. Um,
S2: I guess I can first start off by talking a
S2: little bit about our concept. So it's interesting. Um, you know,
S2: the concept for Buttercup changed quite a bit over the
S2: course of the over the course of the AI Cyber Challenge.
S2: So this got announced, um, a couple years back, and
S2: there was a period of about 4 or 5 months, um,
S2: after the cyber challenge was announced, but before DARPA had
S2: really released any rules. So we didn't really know exactly
S2: how the competition was going to be structured. We structured.
S2: We just knew that we would have to build a
S2: fully autonomous, AI driven system that could find and patch vulnerabilities, um,
S2: with a high degree of accuracy. Um, so originally, the
S2: concept that I drew up along with my co-creator Ian Smith, um,
S2: was originally really ambitious. Lots of moving parts, lots of
S2: static analysis, dynamic analysis, lots of, um, conventional techniques, lots
S2: of AIML based techniques. But ultimately, once the rules came out,
S2: it kind of got pared down quite a bit. Um,
S2: some of the things that we wanted to do, um, were,
S2: were marked as like out of scope. Some of the
S2: stuff we wanted to do were marked as against the rules, um,
S2: just for the tractability of the competition.
S1: So is that because they were, they would have been
S1: too expensive. Didn't you have budgets you had to stay under?
S2: Yeah. So some of it was definitely, um, budgetary and
S2: some stuff was just, you know, flat out against the rules.
S2: We looked at fine tuning a large language model, um,
S2: with information about lots of open source software. And, um,
S2: there ended up being a rule about pre-baking models, so. okay, really,
S2: kudos to DARPA for making sure that, you know, competitors
S2: didn't have the ability to kind of, um, skew the
S2: systems that they build for the test, which is, you know,
S2: finding and patching vulnerabilities and open source software. Um, so, yeah,
S2: there was a lot of stuff that gets cut down. Um,
S2: they got cut down. But ultimately the design of our system, um, was,
S2: was basically a pipeline. We we kind of broke the
S2: problem down. We realized we had to do basically 4
S2: or 5 things really well. To win this competition, we
S2: had to be able to find vulnerabilities. And not only that,
S2: we had to be able to prove they exist. So
S2: it wasn't enough just to, you know, use a static
S2: analysis scanner and say, hey, this thing thinks there's a
S2: vulnerability online. 50 of, you know, whatever, uh, you actually
S2: had to have a crashing test case for the first
S2: round of the competition in the semifinals. And in the finals.
S2: You didn't they, they relaxed this requirement. But the pathway
S2: to getting lots of points basically still required one. Um,
S2: so you you had to find vulnerabilities and also prove
S2: they exist with a crashing input, or an input that
S2: would trigger a sanitizer in the target function. Um, you
S2: had to be able to contextualize and draw additional information
S2: about this vulnerability. Otherwise, patching was doomed to fail. Um,
S2: and then you had to patch the actually patched the vulnerability. Um,
S2: so this is a highly complex, uh, problem that conventional
S2: approaches to software analysis have really kind of not addressed. Well,
S2: in my opinion. And it was a great area to
S2: use I. And then we also, you know, finally we
S2: had to orchestrate all of these functions and do really
S2: high quality engineering around all of them so that the
S2: system would stay up and running for several days. Um,
S2: so based on those kind of 4 or 5, depending
S2: on how you chop them up, core principles or core
S2: tasks that we had to do, um, we kind of
S2: decided on an approach that we kind of call the
S2: best of both worlds, which was, you know, we knew
S2: that conventional software analysis, whether it's dynamic, static, hybrid, whatever, um,
S2: it really excels at certain subproblems within this pipeline. and
S2: it really struggles with other ones. And AIML and specifically
S2: generative AI, which the competition was, was kind of heavily
S2: skewed towards generative AI. Generative AI does really well at
S2: certain types of subproblems in this pipeline, but also really
S2: struggles with others. So our approach is pretty straightforward. We're
S2: going to merge the best in class capability for each
S2: part of this pipeline. Uh, stitch them together with high uptime,
S2: high reliability engineering code, um, and then focus on doing really,
S2: really well for the largest number of, um, the largest
S2: number of possible targets that we could possibly, um, that
S2: we could possibly do well in.
S1: Okay. Yeah. Interesting. So would you say that, um. Basically
S1: those those things that you described in the beginning, those
S1: are like modules and they should almost like, kind of
S1: work independently. So you can, like, hand a task to
S1: each of them. Is that kind of the the system
S1: design idea?
S2: Yeah. Yeah. So we, um, part of this was just
S2: surviving a really rapid development cycle. This wasn't really advertised
S2: all that well, but we actually only had about three
S2: months to develop the first version of Buttercup in the semi-finals. Um,
S2: and we actually had only had about six months to develop, um,
S2: the final version of Buttercup or Buttercup 2.0, which, which
S2: took second place in the finals. Um, and that was
S2: because even though each round of the competition ran for
S2: a year, it took DARPA a while to solicit feedback
S2: from competitors, other stakeholders, and actually solidify the rules. Um,
S2: and so the rules were solidified. It was really at
S2: risk to do really kind of any development on the system. Also,
S2: certain things like the the technical specifics on their competition
S2: API weren't available until later in the, in these cycles. Um,
S2: so part of the reason why we modularized each component
S2: was so that we could take smaller subteams within my
S2: larger team of about ten engineers, um, all working some
S2: degree of part time on this system so we can
S2: modularize it, keep them kind of separate. You know, it
S2: gives us this integration problem that we have to deal
S2: with at the end. We have to kind of put
S2: everything together and make sure that it runs well. Um,
S2: but it was kind of a necessity. It was kind
S2: of a necessity because we had to work on developing
S2: everything independently. We couldn't afford to just do the first block.
S2: And is it becoming like that? You know, that meme
S2: of the horse drawing where really finally defined head and
S2: then as it gets towards like the the back parts
S2: of the animal, it turns into like a raw sketch.
S2: That was what was going to happen if we if
S2: we didn't modularize this. Um, but it also helped because
S2: as we decided to change out strategies or play with
S2: different strategies, made it really easy to kind of plug
S2: and play different parts to see what would work later on.
S1: Yeah, that makes sense. So I keep having this debate
S1: with a whole bunch of people. It's kind of around, um,
S1: let the model do the work because the model is smarter. Um,
S1: and it just understands what to do. And then there's uh,
S1: the other argument, which is, um, build a robust system
S1: and you have the model kind of just be the
S1: intelligence that helps guide the system or moves things through
S1: the system, or maybe routes, uh, across the system or whatever.
S1: But the system itself should be set up really well,
S1: and you're kind of like functioning as a router. And
S1: then when the model gets updated, it makes the system better. Um,
S1: but the counter to that is basically that we're just
S1: going to design bad systems. So we should stop trying
S1: to be rigid there and just use the model. Like
S1: where do you guys fall on that?
S2: Uh, I think it was probably closest to the second
S2: one and maybe more like an an undescribed third thing.
S2: So I'll kind of go over for I, um, you know, we've,
S2: we've been, you know, in me particular I've been doing
S2: research on like applied AI for, for security problems since before, uh,
S2: the large language model became the predominant form of technology.
S2: Back to, you know, 2018, 2019 time frame. Um, and uh, realistically,
S2: like large language models are great at a good number
S2: of things. Um, but they really struggle with certain things.
S2: And particularly in a challenge like this where you have
S2: to do multiple things right in sequence in order to
S2: be successful, you have to worry about errors that start
S2: off in early stages of an LLM heavy pipeline that
S2: compound over time, until eventually you get to the point
S2: where I think kind of collapses. Um, so our philosophy
S2: on using AI, uh, specifically within the AI cyber challenge
S2: and also kind of more broadly, um, is to use
S2: it for, um, tightly constrained, highly contextualized problems that, um,
S2: the models are set up for success. Um, so this
S2: is actually kind of an interesting anecdote. Um, during the
S2: first round of, uh, during the first round of the
S2: AI Cyber Challenge, um, the whole concept of like multi-agent systems,
S2: systems that have, like, tools available to them. um, didn't
S2: really exist. It was like in a couple of papers
S2: on archive and ultimately, um, the way we built our
S2: aperture for the semi-finals and for the finals, um, is
S2: is now reflective of how LM driven systems are just
S2: built today. So it's actually really vindicating. So like our
S2: patcher is a like a multi-agent system. It's got multiple
S2: large language models, each with different roles to play within
S2: this process that collaborate to generate a patch and then
S2: validate it to make sure that it's actually one will compile,
S2: two will actually fix the vulnerability that we've discovered. And
S2: three doesn't break other functionality within the program. So we
S2: found that trying to ask one large language model to
S2: do all of that didn't really work out. And also
S2: in the semi-finals, the, the reasoning models, um, or the
S2: thinking models, depending on, on the branding, they didn't exist,
S2: they weren't available. They weren't even available to us to
S2: use as like, um, early adopter models in the a.i.c.c.
S2: So we were dealing with, with simple, you know, back
S2: and forth, um, style chat models. Um, so we actually
S2: had to build in a lot of this reasoning as
S2: part of this, like multi-agent architecture, we had to build
S2: in a lot of like reliability and engineering code around
S2: maintaining the pipeline. Um, fortunately, the process for um, discovering
S2: artifacts and submitting them was pretty rigid. Um, so it
S2: didn't really affect us that much in terms of or
S2: it didn't have to like put a lot of really
S2: complex reasoning in, um, but actually we ended up even
S2: by the end of the finals, we didn't use a
S2: reasoning or a thinking model, um, in Buttercup, because we'd
S2: actually built it in, it was part of the circuitry
S2: or part of like the, um, the Python code, part
S2: of our orchestration code. Um, so we had the opportunity
S2: in the finals to take that out and let the
S2: model do the work. We kind of explored it a
S2: little bit, but ultimately we decided against it because the
S2: best case scenario was that the model would kind of
S2: figure out on its own how to break the problem
S2: down and how to do individual things, and what tools
S2: to call in sequence. Uh, but we were already subject
S2: matter experts who did it exactly the way it should
S2: be done. So the the best case scenario is that
S2: the model was able to replicate what we've done only
S2: at a more expensive per call. Um, or more expensive,
S2: like number of volume of tokens. Um, so we actually kept, um, we,
S2: we did upgrade our models. We went from the GPT
S2: three series, um, and the Claude three, uh, series of
S2: models and moved up to, um, the four and like
S2: the basically the Gen four versions of models for the final.
S2: So we, we upgraded the underlying models, but we very much, um,
S2: kept the problems very small for the, for the AI's
S2: or for the, um, for the AI models, so that
S2: we would avoid this issue where you have compounding errors,
S2: you have to worry about like these, these modulo errors of,
S2: you know, deciding to do the wrong thing in sequence.
S2: And that actually turns out to be really, uh, to
S2: be penalize you heavily in these long systems because, you know,
S2: when a system decides, you know, hey, I've got to
S2: do A, B, C and D and C before b.
S2: All of that information involved with dealing with this like
S2: out of sequence task. It stays in the context window.
S2: And it kind of, for lack of a better term,
S2: kind of pollutes the model's ability to kind of reorder
S2: those tasks and do them correctly. It has a hard
S2: time kind of forgetting information until it rolls out of
S2: the context window. So it's a really long way to
S2: say we probably did the latter version. But, um, one
S2: thing I do want to say is like the actual
S2: like processing of artifacts through the system, we didn't rely
S2: on the AI to kind of figure out, okay, I've
S2: got a vulnerability now I should patch it. That was
S2: also all, um, that was also all orchestrated, um, by
S2: our by our larger pipeline.
S1: Okay. Okay. So yeah, I've seen this a lot as well.
S1: I mean, I feel like this is a general concept
S1: that people are coming to, which is, um, I don't
S1: want to say legacy tech. Traditional tech is just like, deterministic. So, like,
S1: that's the tech that you want to use to, like,
S1: do things that matter, and then you kind of want
S1: to use like AI for like a, um, I don't know,
S1: like a router maybe, or like a, um, something intelligent
S1: about choosing which standard tech to use, but not making like, choices.
S1: Maybe necessarily. Um, I don't know. I'm trying to figure
S1: out how to articulate that, but it's like.
S2: Yeah, well, it's actually funny you bring this up. I've
S2: had to kind of get good at articulating this, um,
S2: over the last couple of years. So the way I've
S2: explained this to people is that certain problems, particularly in
S2: computer science with this kind of generalizes everywhere. Certain problems
S2: lend themselves to prescriptive solutions. So prescriptive solution is something
S2: that we do when we write an algorithm to solve
S2: a problem. This could be like coming up with an
S2: answer for the traveling salesman problem. You know, we know
S2: it's a really difficult problem to solve, but there's greedy
S2: algorithms that do a pretty good job and for the
S2: most part, will get you a good answer. Maybe not
S2: the best answer, but they'll get you a good one.
S2: So for these types of problems, you can prescribe a
S2: set of steps to the computer and let them execute them.
S2: Now other problems are really, really challenging to prescribe a
S2: solution for. So these types of problems lend themselves to
S2: AI or ML techniques because you can use a descriptive
S2: instead of prescriptive solution. So a good example of this
S2: is like image recognition. So it's really really hard to
S2: take a picture of a cat and write a computer
S2: program that will say, okay, based on the pixel colors
S2: of this pixel and this position, this is going to
S2: be a cat, because a cat can be in a
S2: million different contortions. It can have different hair, the face
S2: can be half obscured. But what we can do is
S2: we can describe to an AI ML model what a
S2: cat looks like with millions of pictures, because we have
S2: millions of pictures of cats. And then it can do
S2: a good job of solving that problem. Now it might
S2: make mistakes, but this is better than the option that
S2: you had with the traditional approach, because that approach was
S2: awful to begin with. So a good example of a
S2: corollary for this in Buttercup is patch generation. There's a
S2: lot of synthetic code generation tools and a lot of
S2: research in this area. But in terms of like automatically
S2: generating patches to fix bugs, unless your bug is like
S2: dead obvious, like it's missing a bounds check and it's
S2: really easy to apply some sort of pattern matching to
S2: figure out what the lower bound is, or the upper
S2: bound is that needs to be checked. Um, tools to
S2: generate patches for weird bugs. Like they just don't exist.
S2: So this is a great place for AIML to help
S2: us out. And it actually turns out, um, you know,
S2: this is really proven true by the AI Cyber Challenge
S2: and by Buttercup, more specifically, um, llms are great at
S2: generating code, um, because it's one of the biggest value
S2: propositions right now for the technology. So, um, generating patches
S2: for bugs is tightly constrained. It's not not asking you
S2: to generate all of the code that is necessary to
S2: build this entire system that I've got a spec sheet for.
S2: I'm only asking it given this code, and given what
S2: we know about this vulnerability, how would you change it
S2: to fix it? The large language models have already internalized
S2: internalize large numbers of incremental commits to open source code
S2: repositories that fix bugs, so they actually have a really
S2: good track record with, um, more than I expected, even
S2: when we started this, uh, with generating patches. So this
S2: is a great example of where generating a patch is
S2: something that lends itself towards a descriptive solution and a
S2: descriptive algorithm, uh, or an AIML algorithm versus something that's prescriptive, um,
S2: which is fuzzing. Fuzzing is a good example of a
S2: prescriptive solution. If you if you need to find a
S2: vulnerability and you need a crashing input, um, you have
S2: to be able to prove that it exists. It's really,
S2: really hard to get an LLM to do that because
S2: llms the underlying reasoning. They don't have like data feedforward. Um,
S2: they basically they look at source code like they look
S2: at natural language. Natural language doesn't describe the activities of
S2: an underlying state machine that runs on hardware after it
S2: passes through a compiler. So like, you know, the source
S2: code when looked at by a model. Models look at
S2: source code in a really shallow way. Um, so when
S2: we want to find, you know, a crashing input, a
S2: fuzzer is a great way because we can prescribe a solution,
S2: which is try everything, brute force it. Um, just come
S2: up with different inputs, throw it in there, and then
S2: if it crashes, well, there you go. You've proven it.
S2: So that's what fuzzing heavily early on. You know, for
S2: one type of problem we use patching heavily for another.
S1: Yeah, that makes sense. And the other problem with, um,
S1: finding vulns with with um, I also seems to me that, um, they,
S1: they want to please there's they're heavily biased to be like,
S1: this is it. This is one. Yeah. Well, this is
S1: definitely a hit or whatever. And you look at it
S1: and it's actually not. So I guess the intelligence is
S1: deciding to use the fuzzer, which it could help make
S1: that decision that a fuzzer should be used. Right.
S2: Yeah. Yeah. So it's it's funny you bring that up.
S2: Large language models really struggle to solve problems that aren't
S2: rooted in some kind of ground truth. Um, it turns
S2: out there's a huge difference there. We have some internal
S2: research that we haven't published. Anybody could reproduce it. But, um,
S2: so it turns out if you if you have a
S2: bit of source code and you ask the model to
S2: tell you where the vulnerability is, um, it will absolutely
S2: hallucinate a vulnerability because it wants to please you. Uh,
S2: we have one of our researchers, um, one of our
S2: principal researchers, Artem. He's a great guy. He, um, he
S2: downloaded the, um, formally, correct. Uh, the formally proven correct
S2: portions of, uh, of Linux and asked a large language
S2: model several hundred times. Um, here's a snippet of code.
S2: It has a vulnerability where it is, and every single
S2: time it would find it would manufacture vulnerability because it
S2: wants to find the answer. So it turns out when
S2: we started asking it, is there a vulnerability? Um, it
S2: messed up a little less, but it would still assume
S2: that because you're asking that there's something to find and
S2: it would still mess up quite a bit. So that's
S2: why when we're in the concept where we're, when we're using, um,
S2: large language models for generating patches. It's great because we
S2: know there's a vulnerability because we found it and we
S2: proved it, and we can collect additional information.
S1: Yeah.
S2: So now I don't have to worry about asking the model. Hey,
S2: do you think there's a vulnerability? And if so, patch it.
S2: I say no, there is a vulnerability. It's here. This
S2: is extra information about a code that touches it. Now
S2: generate a patch. And the model is very good at
S2: doing that because it takes away the decision making or,
S2: or the judgment call that large language models are really,
S2: really bad at because they don't actually model judgment calls underneath.
S2: And their architecture, they, they model, you know, sequencing information,
S2: sequencing tokens. And when you write code, you're writing a
S2: sequence of tokens. So these problems tend to be, um,
S2: a lot more suitable than other problems where you're asking
S2: it to find the ground truth for you, bad problems
S2: for llms asking it to take ground truth and expand
S2: upon it. Great applications for Llms.
S1: Oh man, I love that. And this also goes to
S1: your previous point of not wanting to pollute the context
S1: for the current task on hand, which is building that patch,
S1: because if you have like some history of like there
S1: were previous decisions made or previous questions asked or whatever
S1: it might get like diverted, you know?
S2: Yeah, absolutely. It's um, it's a, it's a big challenge particularly, um,
S2: I don't know, it's funny. I've, I've been kind of
S2: trying to sing this gospel internally, uh, at Trail of
S2: Bits and to other people who will listen that, um,
S2: the increasing size of context window is not always your friend. Um,
S2: by increasing the size of the context window. I mean,
S2: if you think about how the large language model works
S2: under the hood, it's using these contexts to attune the
S2: model to certain parts of its training data that are
S2: going to be highly relevant to solving your particular problem.
S2: And the more words and the more tokens you put
S2: into the context window, the more you are kind of
S2: nulling out or, um, numbing the attention mechanism. You're forcing
S2: it to become more and more general, because now there
S2: are more tokens that are affecting these attuned probabilities. So
S2: you actually are better off with using now. Context window
S2: is great because if you need, let's say a million,
S2: you know, a million tokens in your context window to
S2: constrain the problem, then use a million tokens. But if
S2: you can do it for 1000 or 10,000, you're going
S2: to get better results because you're more likely to focus
S2: that model where it needs to be.
S1: Yeah, I love this. Like, by the way, this this
S1: this is great. This is great. Um, I'm going to
S1: create a lot of content out of this, um, because it's,
S1: it's really crystallizing in like one starting to form something
S1: in my mind. I'd love to work with you on it. Um, essentially,
S1: what I'm trying to think of is, um, what are
S1: some general statements that we could make? Um, one that
S1: I'm sort of heading in the direction of, you tell
S1: me if I'm wrong is like. And this might be
S1: overstating it, but like, the system itself should be highly
S1: modular and and most as much as possible made up
S1: of traditional and deterministic tech. And then the way that
S1: you use the AI is for the specific type of problem,
S1: which we're going to articulate the way you articulated it
S1: for those types of problems where routing is needed to
S1: the traditional tech. Um, and it's like, don't just go
S1: crazy with AI. Don't ask it questions that the traditional
S1: text should be answering. Um, it's something like that. And
S1: then ultimately you have like this dependable deterministic system with
S1: the minimum amount of AI that is required to move
S1: appropriately through that system.
S2: Yeah. So yeah, really it comes down to problem formulation.
S2: And this is like the the great part about and
S2: this is part of the reason why you see such
S2: a huge overlap in interest between people from the computer
S2: science background and people from like data science backgrounds on
S2: here because, you know, one of the basic things you
S2: learn in computer science, like when you get to like
S2: the graduate level is problem formulation. It's how to recognize
S2: your problem as a derivative, or maybe a like dressed
S2: up version of some other problem. So, you know, right away, um, okay,
S2: I have this problem of, okay, I've got to manage
S2: this delivery system. How do I make this delivery system, um,
S2: for Amazon efficient? You can recognize this right away as, oh,
S2: this is traveling salesman. There's no good way to do this.
S2: But what I can do is I can. I'm going
S2: to get a good answer. I just have to accept
S2: that my answer is going to be imprecise or not
S2: necessarily optimal. Um, and in applying AI and ML to
S2: security problems or any problem in general, the first step
S2: is very much like problem formulation. It's understanding what kind
S2: of model is going to work best for this problem,
S2: because is this a problem that will work well with
S2: a time series model, because my data is coming in
S2: over time, or is this a model that's going to
S2: work well with, um, let's say like a, like linear regression,
S2: because there is some true underlying probability for how the
S2: data is distributed that I'm trying to learn from one
S2: of like the kind of curses of large language models
S2: is that they have abstracted all of this good data
S2: science practice, all these good data science practices away. And
S2: now it's great because it democratizes it. Anybody can use AI,
S2: anybody can use an LLM. And all you have to
S2: do is be able to articulate your problem. The problem is,
S2: is that it also abstracts away problem formulation. And now
S2: we're starting to use Llms because they're accessible for certain
S2: types of problems that they're really not well formulated for. Um.
S1: Yeah.
S2: So this is this is kind of where we get
S2: to the issue. So the good news is we don't
S2: have to just like say, okay, well, I can't do
S2: problem formulation with an LLM, so I just throw it away.
S2: Don't use it. I have to go back to, you know,
S2: TensorFlow and writing my own models and stuff. What we
S2: really have to do is get to what you were describing,
S2: which is rather than throw the LLM at a large problem.
S2: We take it a step further. We break the problem down.
S2: Are there subproblems that are highly amenable to AI solutions?
S2: I have a litmus test that I, that I pass, um,
S2: you know, problems through. And I try to encourage my
S2: team members to use, um, which is, you know, basically
S2: like a check to see whether a problem is good
S2: for AIML. And it's usually, you know, do you have
S2: enough data in the model that you can train? In
S2: this case, it now becomes is the LLM. Does the
S2: LLM have examples of this on the internet that it
S2: can draw from, or are you asking it to do
S2: something like reverse engineering, you know, firmware code on this
S2: obscure chipset that like there's no examples on the internet,
S2: bad example or to it won't have it won't have
S2: anything to draw from. Number two, um, is there some
S2: probabilistic nature to the data that's underlying? This is actually
S2: makes large language models really bad for a lot of
S2: security problems, because they're what we call non-differentiable, meaning that
S2: they don't have like this nice curved space that you
S2: can use stochastic gradient descent or virtually any optimization function
S2: to try and climb and find a good answer for
S2: it actually exists more of like this kind of cloud
S2: with dots of answers all over the place. If you
S2: were to try and imagine the answers to security questions
S2: in like a mathematical graph.
S1: Okay, what's an example of what's an example of one
S1: of those? I'm, I'm trying to think of what that
S1: space might look like.
S2: Yeah. So a good example of like a problem that
S2: is differentiable is like housing prices. So housing prices vary by,
S2: you know, like the size by square footage. Yeah. Square footage,
S2: number of rooms, zip code quality of the schools. So
S2: when you plot these all out you get something that
S2: you can do linear regression on. You can see like.
S1: A.
S2: Little loop. And that's called a differentiable function because it's
S2: a continuous line that you can draw through the data
S2: that more or less minimizes the error of those points
S2: along the line.
S1: Yep.
S2: But if we want to think about, um, let's say
S2: now optimizing a program, we can take a look at
S2: how ordering certain steps or changing the way we implement
S2: certain functions as changing the speed of a program up
S2: and down, and that becomes kind of pseudo differentiable. It's
S2: it's more like a step function where you have kind
S2: of like little lines where if I change this one thing,
S2: it jumps up a little bit, it's more jagged, but
S2: there's still, um, it's close to differentiable because I can
S2: kind of map deterministically how if I run it on,
S2: you know, with this set of compiler optimizations or that
S2: it's definitely not differentiable, but it's closer. Security is just
S2: wild because the flaws in computer programs can come from
S2: one of a million different sources. It can be a
S2: logic bug, it can be a mis implemented function. It
S2: can be the use of an unsafe function, which is
S2: easy to find. There's no way for us to take, um,
S2: root causes for vulnerabilities in software and solutions to them
S2: and plot them on a graph. Because they come from
S2: they come from unquantifiable sources. Some of them like, you know,
S2: Spectre and Meltdown and stuff. They they're resident in hardware
S2: and the implementation there. Some are purely in software like
S2: X type vulnerabilities. We can't they don't they're it's, um,
S2: it's not even apples and oranges. It's like trying to
S2: compare apples and fighter jets. Um.
S1: Is it, is it a matter of, like the, the
S1: tensor size or the, um, I think that's called tensor size.
S1: I can't remember the, the, um, the number of dimensions
S1: in the space, because when you're looking at square footage
S1: and price what you have to write, is it the
S1: problem in security that is just so many dimensions that, um,
S1: when you try to plot it, you try to simplify it,
S1: it just becomes garbage.
S2: Well, it's a matter of common dimensions. So if you
S2: build a house, every house has square footage.
S1: There you go.
S2: And you can calculate the space underneath. But a cross
S2: site request forgery vulnerability in a, um, you know, piece
S2: of JavaScript code that exists on the web has almost
S2: nothing in common with a memory corruption vulnerability in a
S2: C program running on a router in your home device.
S2: They are implemented at different levels of abstraction. You know,
S2: like even the program representations are different because some of
S2: the vulnerabilities might exist only in binary code after it's
S2: been compiled versus other vulnerabilities that are resident in source
S2: code that's interpreted via web browser. Um, so really what
S2: it is, is it's like trying to it's like trying
S2: to plot, you know, the prices of homes, along with
S2: the prices of, um, I don't know, oranges in a
S2: particular year. You know, there's very little in common between
S2: a house and an orange other than maybe some, like,
S2: you know, global macro effects that might show some correlation,
S2: you know. You know, economic factors like inflation.
S1: Or like the beating of a whale's heart to determine
S1: whether or not it's healthy. It's it's like completely different. Uh, yeah.
S1: Completely different sports. Yeah. Yeah, yeah.
S2: Yeah. So, so really, it's a it's a lack of
S2: common dimensions in cybersecurity, which is why, you know, if
S2: we think about like if we were trying to model,
S2: like what the data would look like, if we could
S2: visualize it, it would just be a bunch of points
S2: of presence out there. Um, uh, within this, like, kind
S2: of large cloud. Um, and even then, that's another problem
S2: that kind of makes cybersecurity really hard to model with
S2: AML is that there is really comparatively little data, um,
S2: in terms of like the volume of data, there's tons
S2: of vulnerabilities out there. But if you're trying to make
S2: a model that's really, really good at, let's say, detecting, um,
S2: buffer overflows and embedded device code, um, you're going to
S2: find some data for that, but there's not that much
S2: you have to rely on like POC write ups on,
S2: on the internet for practitioners who put it out there
S2: for fun. Um, but there's not a million of examples
S2: of that like, it is if you want to say,
S2: I want to train a model to write the Great
S2: American novel, there you can take you can take every
S2: novel ever written, throw it in there and then see
S2: what the model comes up with. If you prompt it
S2: with like a general plot line, it's going to do
S2: a lot better at that because, you know, that data
S2: fills in that space a lot more. Um, so so, yeah, it's, um. Yeah.
S2: Like the, the, the challenges and problem formulation are, are
S2: really big and, um, yeah, that's why I kind of
S2: encourage people when they look at these like, okay, I
S2: want to build an AI, ML driven system. Um, take
S2: a look at what subproblems are actually suitable for AIML.
S2: Use them there. And I think you'll also find that
S2: a lot of the times we have a tendency to
S2: like say, okay, let's just kind of throw large language
S2: models at some of these problems that we know we
S2: could really solve with regular code. Um, and that's really
S2: bad because of this compounding error problem. So, you know,
S2: if I, you know, five steps in sequence that I've
S2: got to do in step three is good for AIML
S2: and step four is good for AIML. You know, like
S2: it's like, okay, well, look, almost half of this problem is,
S2: you know, is something I'm going to ask the model
S2: to do anyway. I'll just ask it to do one,
S2: two and five to. Well, the problem is it can
S2: make a mistake in one. It can make a mistake
S2: in two. That compound before you get to three and four.
S2: So you're better off, you know, implementing one, two and code.
S2: And then maybe you ask the model just to finish
S2: it off and do step five because it's the final step.
S2: It's had ground truth rooted in steps one two, steps
S2: three and four. If they're well contextualized problems, maybe the
S2: false positive rate is low enough that you can afford
S2: to just let the model kind of finish it up
S2: for you. But that's the biggest that's the biggest jump
S2: I would take. Usually that's step five is like validation
S2: or correctness. Um, checking. And that's not something you want
S2: to ask the model to do because it's, it's it's
S2: it has the tendency to, um, one be wanting to
S2: kind of like please itself and say, oh yeah, it
S2: looks great to me. Um, or to, um, depending on
S2: how you phrase it, find something that doesn't exist. And
S2: validation is a problem that typically is, uh, is pretty
S2: amenable to like deterministic code.
S1: So I really love this. Um. Where this is taking
S1: me is designing, like, a, uh, a general problem solver.
S1: And I'm imagining, like, the smartest model that you have.
S1: You know, opus, whatever. Or, like, the best Gemini or
S1: whatever or whatever the best model is. But but then
S1: what you do is you say, okay, uh, the problem
S1: is we need to design a system that, uh, you know, properly,
S1: deterministically solves this problem with a high level of accuracy.
S1: For example, the vulnerability problem that you guys worked on.
S1: And then what I love is the idea of you
S1: present to the model all these different AI models and
S1: all these different deterministic technologies, all as solutions. And then
S1: you do what you said, which is you, um, break
S1: down the problems that need to be solved at every
S1: level of the subpieces. Right. And then you match each
S1: of those little problems to either one or, uh, one
S1: or many of these eyes, which are bigger or smaller,
S1: have different weaknesses or whatever, or even ML, not even
S1: LLM based. Yeah. Versus deterministic with the rule of like look,
S1: use the appropriate one for this problem type. And then
S1: maybe you have a whole bunch of training about problem
S1: types and solution types. And then it picks which one
S1: to use for each step. I mean is that.
S2: You mentioned this. I think this is what some of
S2: like the large, you know, third party ML as a
S2: service providers like OpenAI and anthropic are kind of trying
S2: to do. If you've heard of like this concept of
S2: like mixture of experts models, um, it's uh.
S1: That's true.
S2: Yeah. It's this concept where, you know, like, you know,
S2: like the, the actual interface. We have to maybe GPT
S2: five and, and I haven't looked at the source code.
S2: I don't work at OpenAI, so I have no idea
S2: if this works underneath the hood, but it's been kind
S2: of theorized and it's even been mentioned, you know, a
S2: bit in terms of, um, you know, people who've kind
S2: of looked at the models a little bit closer that,
S2: you know, um, you know, when we, when we, we
S2: fine tune a model to make it really good or
S2: really suitable for a particular purpose that's amenable to AIML,
S2: it can still be challenging to, um, have it interface
S2: with the user in the way that like a high
S2: quality chatbot would. So using yeah, a mixture of experts
S2: models suggests that like having like an interface, like a
S2: bot that interacts with the user but then recognizes certain
S2: classes of problems and ducts them to the right expert. So, oh,
S2: they're asking me about cyber. I'll ask, you know, um,
S2: cyber GPT to handle this one. All they're asking about,
S2: you know, mental health, I'll ask, you know, mental health
S2: GPT to to help out here. Um, so, you know,
S2: this kind of like concept I think is I think
S2: it's trying to be creative, or at least it's been
S2: thought of, um, in terms of using like all AI,
S2: ML solutions. But but yeah, I agree, like the way
S2: forward is to have, um, you know, for, for like
S2: rapid like prototype development have like components that do certain things. Well, um,
S2: and honestly, it's like reflected in software, like we have
S2: libraries for, we have libraries for sorting. No one or
S2: we have libraries for cryptography. Nobody should be writing their
S2: own cryptography code. Use a library. Um, you know, the
S2: closer these high quality libraries and, um, fine tuned ML
S2: applications or ML models for certain types of subproblems, the
S2: closer we get to being able to kind of compose
S2: all these together. And the good thing is, is that
S2: Elm is probably pretty good at writing the glue code
S2: to sequence all this stuff together.
S1: Yeah, yeah. Because because that's the trick for me. Because
S1: inside of a mixture of experts, you're already inside the LLM.
S1: What I'm thinking of this higher level model is like, look,
S1: we're doing it. We're doing, um, matrix math over here.
S1: We're doing multiplication over here. Um, guess what? This problem
S1: space is not associated with an AI. We don't even
S1: know I will ever touch this. We hand it to
S1: our fastest and best, you know, deterministic addition function or whatever,
S1: you know, and it's like maybe 95% of the whole
S1: app ends up being traditional tech that doesn't involve AI,
S1: other than the routing to get there.
S2: Yeah, I mean, that would be ideal. I mean, anything
S2: you can route, anything. Anything you can. Yeah, I don't know.
S2: It's funny. It's like really what it comes down to
S2: is like using large language models and like, solving large problems.
S2: It becomes a conditional probability problem. And even if you
S2: have the answer, get the right answer right at 99%
S2: of the time. Um, over and over and over again,
S2: you still have a high likelihood of failure by the
S2: time you compute all the conditional probability out. It's kind
S2: of funny. Like, I kind of learned this lesson in like,
S2: in a completely different walk of life. Um, after I
S2: got my bachelor's degree in CS, I, I worked for
S2: like a year doing, um, software engineering and kind of
S2: found it to be dull, so I, I, I did
S2: something completely different. I joined the Army and I started
S2: flying helicopters. Um, it's actually nice. That is, that's actually,
S2: you know, I'm at up at Camp Dwyer in in
S2: RC Southwest and Afghanistan. It's, um, picture was taken of
S2: our aircraft on the flight line, and one of my
S2: jobs as a pilot was to educate our junior pilots
S2: on this concept of, like, mission survivability. Um, and that's
S2: the idea that, um, you know, understanding what's called, like,
S2: the kill chain. The kill chain has been pretty popularized
S2: and security as well. But, you know, basically for a
S2: for a compromise, whether it's shooting down an aircraft or
S2: breaching a database, like a lot of things have to
S2: happen and they all have some sort of probability. And
S2: your goal in breaking the kill chain or breaking the
S2: exploitation chain is to reduce any one probability down to zero,
S2: because then the common or the conditional probability problem becomes zero. Um,
S2: but the probabilities can be really weird. I used to
S2: talk to my junior pilots and ask them like, hey,
S2: what do you think is like the acceptable loss rate
S2: on any of the missions that we fly here in theater?
S2: And they would usually give me answers like they were
S2: pretty close. They'd say like 90% or 95% or even 99%.
S2: So I would actually take them to the math problem.
S2: I get off the whiteboard and I'd say, okay, let's
S2: assume it's 99%. I say, okay, how many aircraft are
S2: we flying a day? Okay. You know, we have ten
S2: total aircraft. We go on five missions a day. So
S2: that's five aircraft are going out there. And let's say
S2: there's only a 1% chance that each one of them
S2: gets shot down. Okay. So that's five aircraft a day.
S2: But we're going to be in we're going to be
S2: in theater for for nine months. We'll round it off.
S2: We'll make it a year. We're going to be here
S2: for 365 days. So now if I take 365 by
S2: five and multiply it by five, that's the number of
S2: missions we're flying in the entire time we're here. This
S2: number comes out to be pretty high. And now all
S2: of a sudden, if I lose one aircraft for every 100,
S2: you realize that I actually run out of aircraft in
S2: the first two months of of being in theater and I.
S2: And now all of a sudden, the troops don't have,
S2: don't have helicopters to fly.
S1: Yeah.
S2: So I said, actually, believe it or not, our our
S2: acceptable loss rate is something more like 99.99999%. Um, we
S2: can almost never lose an aircraft because. Or we can
S2: almost never accept any type of probability. That means we
S2: have even a remote chance of losing an aircraft because
S2: we will deplete them. It's a limited resource. Um, solving
S2: problems with Llms is the same way. If you ask
S2: them to solve 15 problems in a row, even if
S2: it's got a 99% chance, which is which would be
S2: amazing if any LLM could get anywhere close to that,
S2: even if it has a 99% chance of answering every
S2: single problem right over the course of a year, it's
S2: probably going to give you answers that are wrong almost 80%
S2: of the time if that chain is long enough. And
S2: if you have enough problems that you feed through it.
S2: So that's one thing I try to like, um, hope
S2: people conceptualize over relying on large language models and try
S2: to help them understand this, like compounding error problem. It's
S2: really a conditional probability, uh, compounding conditional probability problem. And
S2: your tolerance for false positives is actually zero. So anywhere
S2: in this chain that you can we have to think
S2: about this differently now because I can't reduce anything to zero.
S2: But what I can do is I can take certain
S2: parts of the chain and I can bump them up
S2: to 100%, meaning my chances of getting something right when
S2: I use a deterministic algorithm are 100%. So now I
S2: no longer have some sort of fractional probability out of.
S2: So this 15 step problem now let's say 12 steps
S2: I do deterministically. Now I only have a three step chain.
S2: And now that 99% I'm getting it right only three times.
S2: You simplify this problem. Now I might be able to
S2: make it through a year's worth of operations that, you know,
S2: 100 examples of the problem a day. I might be
S2: able to make it through that with a false positive
S2: rate of. I don't know what the math is in
S2: my head. I'd have to I have to punch it out.
S2: But that false positive rate might be a lot more
S2: survivable in an operational world than, you know, 15 conditional
S2: probability problems that are all 99%.
S1: Yeah, yeah, I love that. The way I describe it is, um,
S1: what's 1% of 100 metric tons of problems.
S2: A metric.
S1: A metric ton of problems?
S2: Yeah, I love that. I love that.
S1: Yeah. Yeah. Um, so, uh, we share this in common, actually. So, um,
S1: I was, um, I was also Army, and I was at.
S1: I was at Fort Campbell, so I was air assault,
S1: so I had to do all the helicopter stuff.
S2: Uh, right on, man. Hell, yeah. Brother.
S1: Yeah. That's cool. Airborne air assault. Right? Um, yeah.
S2: No. Yeah, I, I was, um, uh, this this picture
S2: was taken when we were doing, uh, medevac chase. Uh,
S2: we we did security for those guys over there, but
S2: I was in an air assault battalion, so we literally
S2: did nothing but fly you guys around, so.
S1: Oh.
S2: Nice man. Small world. Dude.
S1: Yeah, yeah.
S2: Yeah, I was over at Fort Campbell. I, I was at, um.
S2: I was at Fort Riley, uh, in in the first
S2: cab and then, um, I PC from there after I
S2: went to Afghanistan and went to the 82nd. Um, so
S2: I never got, never quite got to Campbell, which, like,
S2: would have been great because I live here in Ohio
S2: and Cincinnati. It's like where I was from. So I
S2: was like always trying to get to Campbell because it
S2: was like only like 4 or 5 hours from home
S2: and be able to see family a lot easier. But
S2: I ended up like 12 and nine hours away, respectively, so, uh.
S1: Yeah. Well, that's super cool. Yeah, well, we need to
S1: chat some more. Man. This is, like, really, really cool stuff. Um,
S1: what you guys did on the team is cool, but
S1: I'm even more excited just about the way you think
S1: about these things. Um, I'm. I'm, uh, happy that, um,
S1: the way you're thinking about it is similar to the
S1: way I'm thinking about it. I you've taught me a
S1: lot just during this thing. We should we should definitely
S1: chat more after this. Um, anything else you want to
S1: share about the the competition or, um, lessons learned? Um.
S2: So I think one of the things that that came
S2: out of the competition, um, was a lot of vindication. Sorry.
S2: I nudged mouse in it. Oh. So, um, I'll just
S2: I'll just go right into the answer. I assume you
S2: can edit this later or something, but yeah. Um, so yeah,
S2: one of the things that, um, that came out of
S2: the competition was, was honestly a lot of indication, um,
S2: like I had mentioned before, you know, when we started
S2: off this process, um, this was two years ago, which
S2: has been two lifetimes in the development of like AI
S2: enabled systems for any problem, much less cybersecurity. Um, so
S2: a lot of the things that we did, like tool enabling, um,
S2: and multi-agent systems were things that we did before, things
S2: like MCP or um, complicated, um, libraries for supporting this existed,
S2: like we used early versions of um, of long chain, uh,
S2: for some of our multi-agent stuff, but we actually ended
S2: up having to write a lot of and implement a
S2: lot of our own glue code for this. Um, so
S2: it's really vindicating to see, like, those techniques become, while
S2: we're doing the competition, become not only one commonplace and
S2: two supported by the major large language model, providers be
S2: adopted and be used generally by the community. Um, you know,
S2: it was really great that we came in second and
S2: that also the first place finisher also used this like
S2: kind of, um, use, um, problem solving techniques that are
S2: well suited for the problem approach. Yeah. Don't use AI everywhere. Um,
S2: finisher theory. They were a little bit more LM forward,
S2: but they still had a lot of, like, traditional components.
S2: I don't think any team really went after this. Like,
S2: all LM tried to just do everything within the LM. Um.
S1: I bet a lot started that way, and they they
S1: fall back from it. Yeah.
S2: Yeah. Yeah, I think I think at least one of them, um,
S2: at least one team. I think all you need is
S2: a fuzzing brain. I think in the semi-finals, their approach, um,
S2: tried to just use an LM to augment a fuzzer
S2: to find vulnerabilities. And I don't think they really had
S2: much of, like, a solution for patching, but it was
S2: enough to get them to the finals. I they had
S2: a more well rounded system, I believe, uh, in the,
S2: in the finals. Um, so yeah, it was kind of
S2: vindicating to also see that all these other bright minds
S2: out there were also similarly of the, of the mindset
S2: to do this. But um, one of the biggest takeaways
S2: I have that I'll, that I'll say is that was
S2: like different than what I expected because it's really easy
S2: to pat myself on the back and say, oh yeah,
S2: all the plan I came up with worked great. That's
S2: that's awesome. But, um, I will say that I was
S2: really surprised at how well large language models eventually became
S2: at helping us generate patches and also helping us generate
S2: seed inputs to improve Fuzzer performance. Those were areas where
S2: I didn't really give the LLM a lot of credit
S2: up front, but I had to build an autonomous system,
S2: so I had no choice. They really outperformed my expectations.
S2: So I kind of came out of this with, um,
S2: a bit of a healthier respect for the capabilities of
S2: AI models. Once again, these are still highly constrained.
S1: And yeah, yeah.
S2: Very context rich problems that we ask them to do,
S2: but they still did way better than I thought they
S2: were going to do. Um.
S1: Yeah. And also context constrained, not polluted, like a very
S1: controlled context for that thing. Like like you were talking
S1: about before, right?
S2: Yeah. Yeah. Um, yeah, I think that's about it. Unfortunately,
S2: I do have to jump off. I gotta I got
S2: another call at 1230, but, um. Yeah, I'd love to
S2: chat more and talk more with you at some point.
S2: If you want to do a follow up episode or,
S2: I don't know, you just want to chat about other stuff. Um,
S2: you know, we got a couple of friends in common
S2: between Clint and, uh, between Clint and Keith, and it's, uh,
S2: you know, I've. I've run into you a couple places
S2: on various calls and stuff that we've been on, but, um,
S2: it was good to get a chance to talk with
S2: you one on one. I feel like we've been kind of, like,
S2: circling around in the same circle for a while, but
S2: I hadn't had a chance to, like, actually just chat
S2: the two of us.
S1: Yeah, absolutely. Well, thanks. Thanks for the, uh, the input.
S1: This is just, uh, fantastic stuff. And, uh, let's definitely
S1: catch up soon.
S2: Yeah. Sounds good man. Take care of yourself.
S1: All right. Take care.