AI Vulnerability Discovery Changes the Defense Math

Hey guys, thanks for the throw. We're here with Gabriel Bernardin Shapiro, who's a distinguished AI research scientist for SentinelOne, and we're having a little chat about, well, Black October. We all just lived through it, and looks like maybe November's going to be just as bad.

And who knows what December's going to be like. And so the question, Gabriel, is what we just experienced in terms of all the vulnerabilities that were disclosed, is this the new normal? Are things going to get worse?

And, well, hell, will they ever get better? Yeah, it's a really interesting question, Mike, and thank you so much for having me. I've been really noodling about what this month has brought, and one of the things that I've been keeping in the front of my mind is that we've definitely been here before.

Granted, we did just have PHP to Shell, we had Imudra with the signing cert. js. So it was a bunch of stuff all happening at once.

And I think that one of the things that we're seeing is that our infrastructure is we take our infrastructure for granted. When you drive across a bridge, you're usually thinking about where you're going. You're not thinking about whether somebody had recently inspected it, what's holding it up?

Is the bridge in good condition? I think we have the same relationship with a lot of software infrastructure that we use today. So, PHP is a great example, right?

It does the work behind a bunch of different websites. It signs people in, it processes forms, and now it's been turned into a working attack that affects potentially 69% of the internet. So it's a large number, and it's one of those things that makes you look at the infrastructure that supports the modern web and say, "How well do we really understand this?

" Right. Is it your sense that our adversaries have access to the AI tools that they need to discover these vulnerabilities and, what looks like to me at least, that they are now reverse engineering in a matter of minutes to create some kind of exploit? Is that where we are or is that still to come, and there's just more of this damage that will be inflicted, and we've only hit the tip of the proverbial iceberg?

Yeah. I think that adversaries definitely have access to the tools that they require to perform these types of attacks. I think we should box in what we're talking about here, which, and correct me if I'm wrong, but what I'm picking up is that you're referring to this new discovery that AI-enabled inspection tools can help us identify flaws in software and develop exploits.

Is that where you're going with this? Yeah. To your point, we have limited access to meet those and all these other things through a handful of supposedly people we trust.

But it seems to me, looking at a lot of the open-weight models that are out there, and heck, even just the average frontier model seems pretty good at discovering a vulnerability already. So I'm assuming the bad guys have this capability. Absolutely.

And I think we should assume that they have those capabilities. And that's not necessarily a terrible thing to think that, hey, attackers have the same capabilities as the defenders. We believe in attack to defend.

In order to understand a vulnerability, a maintainer needs to reproduce the attack, right? They need to know whether a fix works. They need to try the attack again and again and again.

And that's part of taking responsibility for a system. If you're responsible for securing a system, you need a practical way to test what an attacker could do to it. And let's just think about the companies that are providing the best models out there, right?

So OpenAI requires separate approvals for Daybreak Red beyond Daybreak Blue. Approval to use it inside of a company also doesn't automatically cover services that that company may offer to its downstream customers. Anthropic has separate defense.

They have red team, they have specialized access. And those levels of approval can really affect whether researchers can reproduce or test these attacks. Some people say, hey, faster approvals could help, but they don't really settle our larger problem that we have here, which is that open-weight models, which are almost as capable, some people say they're six months behind, other people say, "This one is just as capable as a frontier model," can be downloaded and run outside of the original provider's control or intent.

And this really shows that capable tools are just becoming widely more available. And it doesn't tell us that the gap will always be the same. It doesn't tell us how these models perform against modern defensive infrastructure or hardening of systems.

But I think that if I had to characterize what my position would be on this, it's that we can't base long-term defense on the hope that attackers won't leverage the best AI systems available to them. Restrictions can slow misuse. It can give defenders time, and we need that to repair widely used software and get fixes to the organizations that depend on it.

But our systems need to remain defensible, even as better tools become available to attackers. Mm-hmm. I think the big question that's hanging over everybody, and giving me a little bit of rope here, sorry.

No. But it's that the providers of these most capable models, of these frontier models, decide which defenders get a head start, right? If those defenders maintain software used by, let's say, hospitals or utilities or government services, the consequences reach really far beyond a customer's account data.

And I think that a lot of that decision sits with a relatively few number of companies, and that there's not a lot of public accountability for how those decisions are made. I would love to see more clear rules about restrictions. I would love to see more dependable access and also funding for people who are maintaining critical software outside the scrutiny of decisions that affect them, or these decisions that are like, oh, well, what are the larger ideas of whether attackers can have capable models?

Defenders need those in order to determine whether their systems are safe to begin with. And the models providers access policy, that's becoming a part of this question of who is allowed to defend critical infrastructure, and that's a very consequential decision. I think you're spot on because from what I hear from folks is basically you have to have a security vendor that has the resources to partner with the likes of Anthropic to get the benefits of that.

And not everybody has, I don't know, pick any large security vendor you want, installed, and it's expensive to do that as well. So to your point, do we need a more broadly applicable approach to this? Because what's happening now is basically creating a have and a have-nots.

That seems to be the situation that we find ourselves in now. And we also have a really great opportunity here. The model that helps us understand the vulnerability can also help us check reports of this vulnerability that may be unsubstantiated.

It could propose fixes. It could look for similar mistakes within the software stack. All those are possibilities that are open to defenders, and those are great opportunities for leveraging AI.

And this is also what really interests me in some of the work that we've done recently. We studied how models handled extended malware investigations, so reverse engineering tasks, including whether they could change their conclusions when new evidence was presented to them. They still made errors, and you still need experienced analysts to help guide the model.

But for me, the lesson is that we have these capabilities, and we're just now learning to leverage them, and incorporate them, and attackers are on the forefront of that. So I look a lot at some of these new attacks and how attackers are leveraging LLMs, and I think to myself, wow, okay, that was a pretty creative way of doing that. I can learn from whatever my adversary is doing and apply it in a defensive sense.

And that's really one of the biggest opportunities, I think Is the relationship between security and the IT operations folks and the developers, is that going to need to change? And I ask the question because historically, security people would come up with a list of vulnerabilities and send over a spreadsheet, and then the developers would spend time tracking down what generally wound up being a lot of false positives because either the code in question wasn't loaded or it's not accessible or whatever it is. And after a while, everybody stopped listening to security people.

But if we have more and more of these kinds of vulnerabilities that seem to be attacking core infrastructure like PHP and whatever else is coming down the pipe, does that relationship need to change? Yeah. The relationship need to change.

Yeah, I think that the relationship needs to change because the business's understanding of what it means to secure something needs to change. And maybe business is even too specific. Let's just say an organization's understanding of how it needs to secure its infrastructure or its products needs to shift.

They need to understand, and I think security people are kind of on the hook here to really communicate in a way that the IT staff and the business leaders can understand, hey, this is the new level of risk that you face in these situations. And mitigating that risk maybe means closer relationships between IT and security. Maybe it means that these teams really need to understand what their own stack is, like asset inventory became just as important with PHP as being able to understand what the fix was itself.

Where is the server? Is it running this? Oh my god, how do we find it?

Closing the loop, and this might sound a little trite, but closing the loop between the communications of your security team and what they're finding and how quickly your IT or your development staff can roll out those fixes is really going to be key. I think especially in the short term. I don't really know how this looks in the long term with how AI-developed engineering and AI CI/CD pipelines will change in the future.

But at least for the meantime, while people are still in charge of those decisions, those groups need to become much tighter and understand what the risks are that they're trying to address and what they're worried about most. As we kind of play around with this a little bit, there's a movement that says we're going to require people to apply a patch or have some sort of vulnerability or control something in place, sometimes in as little as 72 hours, maybe 24, depending. Is that feasible?

Is that something that's achievable in your mind, or is that just kind of like some bureaucrat is writing down something but without understanding what it would take to actually implement that? Yeah. I'm not sure if it's some bureaucrat, because sometimes the situation is so out of people's hands that suddenly they're faced with an environment where they have 72 hours to patch or less, right?

Somebody releases something, somebody else on the internet finds it and says, "Oh, well, these hints are great. I can reproduce this now. " So, the time is always out of your hands.

It's how you react to the situation that matters. If you ignore the problem and you're like, "Well, we'll have patches when we get around to it," that's not an appropriate response. But is an appropriate response to drop everything and shut down a business because you need to ensure that this security flaw that you were told about, but maybe you don't understand completely, is at risk of stealing all of your customer data.

That's a risk that businesses had to make when PHP was happening. " And the ones who understood that risk, I think were able to actually make that calculus and make the decision that they had to make. But yeah, I think it's a little unreasonable to go to somebody and say, "Hey, here's this thing you might not fully understand.

" That's not really tenable for most businesses. Right. " Yeah.

And one of the things that I think it's been good in this industry is we've gotten better at communicating what the risk is. One of the challenges is that in this current media cycle, there's so much hype built around AI-developed vulnerabilities or AI-identified vulnerabilities, that it is hard for people to understand what the risk is right now. And even though we've gotten better at communicating that, it's a challenge, I think, especially in this moment.

" Yeah. The AI agent as internal insider risks. I think that allowing a non-deterministic system to have widespread access to your organization is a new set of challenges that security teams have to deal with.

They have to understand what guardrails they need to implement that allow this thing to be successful and useful for their teams, but also that have reasonable restrictions built around it. Maybe you don't want it reading three years of your email. Maybe you only want it limited to one week of your email at a time, so it can't just reach back and find something and forward it, or doesn't have that level of access to the organization.

I don't think that adversaries are having a laugh about it. I do think that they are increasingly thinking of how to hijack people's agents because it's just another level of access that provides them with the end goal that they want. Right.

Or they may even insert their own agent into a workflow or a process and hijack it entirely, right? Because it's theoretically possible. I think, yeah, theoretically it's possible.

I think it's much more likely that they would try and hijack an existing agent than deploy their own to a machine. But that's just because the surface area for those existing agents right now is very wide. Got you.

Are we waiting for a couple of catastrophic events before everybody kind of really wraps their heads around this and does something about it, including the business side? Or are we going to have one of these situations where yet again, we're going to suffer death by a thousand cuts before everybody kind of wraps their head around this a year from now? Yeah.

I guess I'd ask what do you mean by wraps their heads around this? Well, makes it a priority, allocates budget, and says that we've got to lock these things down and we've got to be serious about this, because right now I think we're still treating security as it's a risk, but we're not allocating more funding to it just yet. I'm not sure I would agree with that.

I think that a lot of people are. They kind of see this and they're deeply concerned about both AI and security. If I had to make a bet on what businesses and business leaders are spending money on, I think that the smart bet would be that they're spending money on AI and not necessarily on security.

Security is always an afterthought. It's always going to be additional cost to those businesses to run. It introduces friction.

That's what we do. And I think that there's definitely enough for folks to think about, to say, "Hey, whoa, wait, let's Let's think about whether this thing is safe and secure. In fact, I think most people's reaction to seeing AI complete tasks on their desktop, the first one they have is, wait, what does this have access to?

How is it running? Where is it running? I guess I would say that the immediate reaction that I think most people have with seeing AI technology is how is this secured?

Maybe I'm biased, maybe that's just my own view, but I get the feeling that this is something that folks are actively discussing, concerned about, and now I think the really fun part is they're interested in it. " Somebody goes, "Wait, it can read all my emails? Does it respond for me?

How is this working? " There's this immediate uncanniness that they have as a response. There are some folks that think that, well, I've isolated using a Mac Mini, I'm okay, right?

But when you look at these things, it seems like they are aggressive, and they will find ways to go access things in ways that you don't imagine. So, is our focus too much maybe on trying to secure the AI agents and not the environments the AI agents are accessing? I think that securing the AI agent is the right focus, right?

The environment that the agent is accessing is going to be the world. It's going to be the world that you interact with, the world that I interact with. Now, we can secure that as much as we might for a person, or with saying, hey, restricted access to this sort of information, or we put up some guardrails here, we set hard rules, we shut things down when they look like they're not going correctly.

But the world that these agents need to interact with, the thing that makes them useful is the fact that they are reward-seeking, and that they have all the same tools that we have at our fingertips. So securing the agent itself becomes something that I think deserves more of the focus. Securing the world that the agent inhabits, well, that's always been the game that we're in.

All right. So on the upside, after we get through all this, do you think we'll have better security and we're just going to experience a certain amount of pain to get there, but more attention is being paid, so this time next year, are we better off or worse off? Mike, I would love to have this conversation with you next year and go through these questions.

I think that would be really fun, actually. Because I think so much is going to change. The rate of technological change that's happening right now is dizzying.

It's really hard to keep up with. And I say that as somebody whose main job is to keep up with the rate of technological change and understand the impact. And that is tough right now.

I wouldn't be in this game if I didn't think that we could make things better or improve our outcomes. And I think that the tools and the advantages that AI gives us, yes, it's a double-edged sword. It can provide the same types of advantages to attackers as it does to defenders, but there's a lot of people who are working on it to make sure that the defensive side is going to win.

The majority of us want to see a more secure world, and I don't think that viewpoint is going to change. I think that that's going to inform how we develop this technology. Right.

Folks, you heard it here. In the short term at least, the advantage may be to our adversaries, but longer term, AI may benefit the defenders more than the adversaries. So cross your fingers and hopefully the months ahead, we're all better off.

Hey, Gabriel, thanks for being on the show. Of course, Mike. Thank you so much.

All right, and back to you guys in the studio.