AI Scanners Demand Smarter Vulnerability Prioritization

Hey guys, thanks for joining. We're here with David Lindner, who is Chief Information Security and Data Privacy Officer for Contrast Security, and we're having a chat about, well, vulnerabilities and the rate they're being discovered, but not all vulnerabilities are, shall we say, equal. In fact, some of the findings are downright, well, suspicious in and of themselves.

David, thanks for being on the show. Thanks for having me. So what's going on here?

We are definitely seeing an uptake in the number of vulnerabilities that are being discovered, but when I look at these AI scanners, a lot of them are also generating more false positives, and so we're now spending more time chasing our proverbial tail. So how should we think about this, and what makes sense in terms of what vulnerabilities we should be chasing, and how should we manage all this stuff? How much time do we have today, Mike?

The realities are, I think there's a lot of experimentation going on with what exactly we can get out of these models and the model changes and we want to try again. And I think we're at the point now where we've realized and figured out that the non-determinism is a problem if you're looking at just using AI scanners. But at the same time, they do produce real things.

They may be buried in false positives that still need some sort of triage, whether that's a human, which is expensive, or more inference, which also could be expensive. And you're right, we are seeing an uptick. If you look, even year over year before AI was a big thing, the number of CDEs with MITRE and CISA and all them, it's been increasing dramatically.

And that's continuing, and we're seeing more and more of that. You look at some of the reports from the Anthropic's and the OpenAI's, Mythos was released to a certain number of organizations, and they wrote blogs. "Oh, we found all these things.

" But the reality is, we've never had a problem finding things, right? Every CISO and VP and security leader and asset person I talk to, they already have a pile of vulnerabilities. They have a pile of things that they're already working on.

So adding more to that pile, it just adds to the pile, right? But the problem is, and we've been dealing with this forever, is forever we've had this haystack with a couple of needles buried in it of the things that are real and we need to act on now. Now we have a billion haystacks with a couple of needles in each one, right?

And so what do we do about that? Because people are going to continue to experiment. They're going to continue to build AI into their processes.

And where I come from, we see a lot of different things. Our products and what we do is we try to help prioritize, which I think we're going to have to get really good at prioritization, and use certain things for that, including AI. I think the one thing that AI is very good at is taking in context, and in helping you triage things.

Internally, we're building agents for our SecOps teams and our FSOC teams to help them triage the plethora of alerts that they see on a daily basis. And I don't see more vulnerabilities as being a whole lot different. We have to find a way to automate some of that triage process, because let's say you get 1,000 vulnerabilities sent to you overnight.

What do you do with that? Do you just ignore it? Probably not.

Especially if you're a public company, there's the SEC and so on and so forth might be interested in that if you're ignoring things. So, people are kind of in between a rock and a hard place. But we've been pushing some things for a while, and that's like the CSACAV, I think that's really important to look at that first.

Those are the highly known, being attacked and exploited vulnerabilities that are going on currently in the world, right? I think EPSS, Exploit Prediction Scoring System, is also important to look at. Obviously, these both are only publicly known things, but for the most part, what's attacked today are the publicly known things, and if you can't fix those first, you're probably in a pretty bad place.

So how do I revisit my priorities? Because historically, I kind of went after whatever had the most severe rating from something, and now I find that the bad guys are not only reverse engineering the vulnerabilities faster, they're also stitching together a bunch of low-level vulnerabilities in ways that will create a more lethal exploit. So, if I am confronted with 1,000 vulnerabilities, how do I sort that?

Yeah. I mean, that's valid, and I think that's probably where people are more concerned. It's not the fact that we're going to get a ton more vulnerabilities sent to us from our customers and whoever else, right?

Because we already have a bunch. It's what I do with the bunch I have, plus more that's coming in to help protect against, I guess we'll call it the Mythos-based attack, right? Where it can find things and extrapolate and create these exploit chains of things that we never thought were probably even discoverable in the first place.

Right. So there's multiple things that I would do. First, I think you have to look at your control environment.

If you don't have controls in place that can detect things from happening in your environment, you might want to look at that. Because the reality is, even if you were able to, let's say, hyper prioritize, you're still probably going to have a massive list of things that isn't going to get fixed overnight. So how do you mitigate the potential risk while you figure out solving and fixing those issues?

So look at your control environment. How can you quickly detect something from happening? If there's one thing I know about these models and how they work, they're noisy.

If you have the right detections in place, hopefully you detect something going awry or someone poking and prodding at different places in your environment. So I think moving forward, shifting away from, I guess, call traditional measurements of MTTR, meantime to remediate, to meantime to detect, how quickly can you detect something from happening? And then meantime to contain, how quickly can you contain whatever might be happening?

Because you know that vulnerability pile is going to continue to grow, regardless of if you can hyper prioritize. It's not like you're going to get to the point where it's one finding out of a million. I talk to CISOs, they have two million vulnerabilities in their backlog.

Two million. Already. Right?

So when you think that through for a minute, a lot of folks are saying, "Well, that's great. " And yet, at least the initial reports are that some of the fixes that the AI coding tools are creating are worse than the disease in the sense that they are breaking applications, and we don't have the ability to test and validate them all at the speed they're being created. So, how do we make sense of that process?

Do we have to figure out something where maybe we're hoping to apply the patch, and if it works, great, but if not, we've got the next patch ready to go in a couple of minutes? How fast is fast, and what does it mean to be operating at machine speed these days? Yeah, that's really the problem, right?

If you look at the research from, was it 1Password, I believe, did some research on fixes, which was also debated by other prominent organizations. And I think their number was somewhere around 25%. It got it right the first time, where it didn't actually break something or add different functionality.

Which, the reality is this, Mike, humans don't fix things the first time either. I've been in this game long enough to know that I went in and did pen test code reviews and presented a report and did a retest, and it's like, "Oh, no, you didn't actually fix this. " Or you added another vulnerability in your fix.

And the interesting thing about that is these models have been trained on human code. Right? So it makes sense that they're going to fail numerous times.

So it really is going to come down to a few things. We have to figure out a way to use the non-deterministic models, because they are powerful, and they can do things, and they can fix things, and they can detect things, with deterministic tools and harnesses that allow us to understand when something's found that is real, when something needs to be fixed, and then how do we test that against a deterministic outcome, right? Because the reality is you're right.

The model may choose to do it one way this time and a different way the next time, or completely ignore it the next time. That's really the reality of these things. But they are powerful enough if you work with the correct tools and the correct harness to make it deterministic, to get us further along the line.

Do you think ultimately that the tenor of the conversations between the app dev teams and the security teams will finally change? Because historically, the app dev people always kind of viewed security as something to hopefully maybe not have to deal with or get around. " So can we figure out some way where maybe we have this Rodney King moment and everybody can get along, and are we going to work together better, or what's going to happen?

I hope so. The hard part is, and others in the security arena may disagree with me, but the developer's job is not to create secure code. It's to create code that supports the business and answers RFEs and adds functionality and things to whatever they're writing, right?

Their job is not to create secure code. And there will be arguments from people in the security community about that. But I will stand behind that.

Because I'll tell you, at the end of the day, if I went to my CFO and I said, "Hey, I need 70% of an engineer's time to fix security problems," he'd laugh me out of the room. Right? It's not a reality.

You have to look at it from a business. And so we're continually in a give-and-take mode already, and we're already feeling the pressures of the speed of what engineers want to move now. To the point of simple compliance controls of we need a human reviewer on this pull request.

The question to me now is how do we get out of that requirement? How can we have AI do the reviews and just this continual machine that builds, creates, tests, patches code? Will we get there?

I think so. But I don't think we're there already yet, and the regulatory bodies surely aren't there. So I guess the thing that we have to deal with right now is how can we manage the compliance requirements, how can we manage contractual requirements from a security perspective, and also at the same time, not slow engineering down and keep them happy.

So I think it's going to get harder before it gets easier. Are we at this point waiting on some sort of tsunami of exploits that are going to come because the adversaries may not have access to Mythos, but they have access to AI models that are pretty good, and it seems like some of them are already using those tools to create exploits. But at least the initial reports are that those exploits aren't any more novel, so maybe our existing defenses can withstand that.

But how long might that be the case, and are we looking at some sort of catastrophic event in the next couple of months? I think it's realistic to think we'll see some sort of event. But I also think it's realistic that we won't see a massive uptick.

And there's a few reasons why. We're not seeing it yet. We're seeing a massive uptick in vulnerability counts.

Right? We know that. That's real.

That's tracked. But breaches, it's flat for a year over year, right? And a lot of that comes down to a few things.

The attackers, the malicious actors, right now at least, they're using AI to create the malware, create the phishing. They're creating with it, but actually weaponizing it and getting it to run and exploit, they're not. And a lot of it is because the AI is expensive.

So yeah, you're right. They're using more open-weight models. The scary China models that are actually really good.

But the problem is they're still noisy, right? " Because to your point, it's not going to use novel attacks. It's going to try things that it's been trained on.

And I hope that my bank has some detections for those sorts of things and can prevent some of them, right? And lastly, the last time I checked, it's still illegal. Right?

" If that was me, the FBI would've been at my door already. Right? And I think we'll see some of that, too.

How do we handle accountability of this stuff from happening? I don't have the answer for that. Hopefully, someone else does, because right now there isn't any.

So what is your best advice to both the CISOs and the heads of DevOps teams about how to wrap their heads around this and get their teams prepared for, while continuing the hope for the best, what might easily turn out to be the worst? Yeah. First, I would say breathe.

I think as a CISO, you have to be level-headed about things, right? You can't get swayed by every piece of FUD that is spewed across the internet. And there's some realistic things that are happening, and there's some it's like, "Okay, yeah, lots of vulnerabilities.

" But I think we do need to reassess our control environments. I think the realities are is these models were built for software, for creating software. That was the focus because to supposedly get to AGI, you need to have a self-creating model, right?

So it creates the next version, and it creates the next version. So they've created these models to create software. And most businesses in the world now are software factories.

So it makes sense that now the malicious actor has these hyper-powerful models that know everything there is to know about all software that has ever been written that is public, probably some private, and it's easy to tear those apart. And so if you think about it from that aspect and you look at your control environment, our typical controls are network layer. They're at the infrastructure layer.

Are we detecting this? Are we detecting that? It's going to our SIEM.

We have CSPMs looking at our cloud environments. Are there things on the application side that we may be missing? Because again, these models are going to be really good at attacking software, and finding things in software, and exploiting those things.

And that's what we're seeing. Will we see an uptick? I don't know.

I could say no, and then in a month you could be like, "Oh, wow, look at this. " But as of right now, we haven't seen breach upticks. But that being said, most breaches aren't discovered for months or years.

So it could be one of those things where we won't see the aftereffects for another 6 to 12 months of what actually is happening right now. Right. Folks, I think you heard it here.

Don't panic. But I would say that at the end of the day, doing nothing is probably the worst thing you can do right now. So at the very least, start creating an inventory of those vulnerabilities and maybe figuring out how to prioritize some of them, because, well, the harder they are to exploit, the better off you're going to be.

Hey, David, thanks for being on the show. Thanks, Mike. All right.

And back to you guys in the studio.