AI-Generated Code Is Fueling a Vulnerability Surge
Why AI-generated code means more vulnerabilities
Johri says every one of Checkmarx’s enterprise customers now uses coding assistants. In fact, the code base Checkmarx scans for them has doubled since January. Industry benchmarks also show that code written by LLMs has a higher density of vulnerabilities. So the volume of code is rising, the flaw rate is rising, and total vulnerabilities are climbing fast. Meanwhile, attackers don’t need frontier models. Cheap open source models can build working exploits against AI-generated code, and the skill needed has dropped. CISOs tell Johri they see five to ten times more attack attempts than before.Don’t let AI grade its own homework
Most developers still skip a security review before they submit a pull request. Johri argues that checks belong in the IDE, where they catch flaws before they reach the repo. Checkmarx DevAssist, for example, reviews code as the assistant writes it and can block anything with a critical vulnerability. He also warns against relying on LLMs alone to review AI-generated code. They give different results from run to run, they miss known flaws, and they can be very noisy. Some customers report false positive rates near 80%. Instead, he recommends combining deterministic and probabilistic analysis.How to manage AI-generated code risk
In the past, enterprises fixed fewer than 10% of the flaws they found. That is no longer enough. Johri suggests fixing critical issues first, then using attackability and reachability to rank the rest. Since no team can fix thousands of issues by hand, agentic remediation is becoming essential. Faster patching carries its own risk, though. Johri suspects some recent outages came from teams pushing changes without their usual checks. He urges development, security and operations teams to work more closely together. He also expects groups such as CISA and FS-ISAC to play a bigger role as AI-generated code spreads.Transcript
Hey guys, Mike Swither here. We're here with Sandeep Johri, who's the CEO of Checkmarx, and we're having a little chat about, well, what are the implications of all this code we're generating with AI when it comes to security? Because I got a feeling there might be a hangover coming around the corner.
Sandeep, how are you doing? Good, good. How are you, Mike?
I'm well. Walk me through what's happening here a little bit, if you would, because it's clear that we are generating more code than ever, but it's also apparent that it's occurring at a rate that we can't review. And I'm starting to see a lot more code that hasn't been reviewed showing up in production environments.
And if that's the case, are we going to see more vulnerabilities because nobody actually reviewed this code? And can we even keep pace with the rate at which things are being reviewed? What's going on here?
Yeah. So Mike, as you know, we have a lot of enterprise customers, and you're absolutely right. Everyone's using coding assistants.
Literally 100% of our customers, large enterprises, are using coding assistants. That's generating a lot of code. As an example, we ourselves have seen the code base we look at for our customers has doubled since January.
So that's for real. Now, most of the code being generated is generated by LLMs, and we all know, and you can look at benchmarks that have been run by the industry, that machine-generated code, LLM-generated code, has a higher density of vulnerabilities. So what's happening is the volume of code is increasing, the vulnerability density is increasing, so the total vulnerabilities are really going through the roof, as you call out.
So that's happening on the one side. On the other hand, LLMs, you don't need very advanced LLMs. You can use open source LLMs to really create pretty interesting exploits.
So the number of attacks are increasing significantly, on the other hand. And this week, we don't know for sure, but we had the banks have problems. We had airlines have problems.
Today, there was news that the intelligence systems that the US government has are having problems. I don't have any confirmation that they are cyberattacks. But the issue that I think is happening is, given what I talked about earlier, people are trying to fix these vulnerabilities as quickly as possible, right?
Mm-hmm. They are. And every customer I know is trying to discover as many vulnerabilities as they can, prioritize them, and fix them as quickly as they can.
This week's problem, my conjecture, is that the pace of change that is so high, people are trying to fix these vulnerabilities, that they're pushing out code into production, which might be the reason why we have three major incidents happen this week. So it is a problem, but there are solutions. Security vendors are putting all kinds of mechanisms to deal with the increased volume and to automate and agentify the remediation.
So you can try and prevent as far as possible new vulnerabilities coming into your code base and remediate as much of the backlog as you have. And every customer I know has a huge backlog that needs to be tackled. Mm-hmm.
As you look at all of this, though, I also think that developers are using the same AI coding tool that they use to create the code to review the code, which in my mind is like asking the AI to grade its own homework. So do we need to get smarter about this and start putting in maybe additional AI tools to review that code, or/and or do we also need to make sure that they are invoking something that is more deterministic? Because a lot of the analysis that the AI is generating is probabilistic in the first place, and that may not be what we want when we're reviewing code.
Yeah. So Mike, you raised very good points, but there's multiple points you're raising there, and let me address each one. One is what are developers doing for code review?
Well, yes, developers are using LLMs to generate code. They're doing code reviews. Most developers do not do a security review.
And so one, they don't even do it. Forget what they're using for it. Most dev teams do not have developers do security review.
It absolutely needs to be done, and it needs to be done at the IDE because you want to check that and catch as many vulnerabilities as early as possible so that they don't even hit the repo, right? It's much better to catch light bugs. And security vulnerabilities are like bugs.
The earlier you catch them, the better it is. Most enterprises, most dev teams do not check for vulnerabilities before doing, for example, a pull request and submitting new code. So that's one thing they should be doing.
" So there are tools like ours, we call it DevAssist, but there are tools that can help you prevent. That's one thing you need to do. The second thing you need to do, you're asking the question of whether using LLMs is sufficient to do code review.
Let's say you do it at the repo level. You're absolutely right. LLMs are probabilistic by nature, so there's three issues that come up.
One is that they are inconsistent, meaning you do a check, I do a check on the same code base. Depending on the prompt, you'll get different results because it's probabilistic. Number two, while they discover some interesting new things, and there's plenty of reports that prove that out, and we have seen that in our own benchmarks.
They do discover a bunch of new things that deterministic engines don't discover. So there's real benefit to the LLMs. They are also not comprehensive, meaning they also miss many findings, known vulnerabilities.
And the third issue is that they are extremely noisy because they're probabilistic. So we've had customers tell us that they've used Codex Security or Claude Security, and they've gotten results. It's like getting dumped with a haystack.
There are needles in there, but there's 80% false positives. So those are the challenges. Not that LLMs are not useful, but those are the challenges.
And for an enterprise that wants to use LLMs to do code reviews, needs to do both a deterministic and a probabilistic combined. You need to do both. Neither one is good enough in today's world.
And two, you need a system that can suppress the false positives, that is consistent run to run. You are assured that it is capturing everything possible, or everything that is possible to be discovered. So you need systems like that, which is what we have developed and launched in the market.
So... We've also seen some reports from the FBI and other folks talking about how it's clear that the adversaries are now using a combination of- Mm-hmm ... AI tools and deterministic tools to find vulnerabilities and then create the exploits in a matter of, I don't know, minutes.
And if that's the case, are we going to be looking at a wave of security incidents in the coming weeks and months that are coming our way because all this code that we're generating isn't very secure? Yes, and we have already seen that. I'll give you some real data from my conversations with CISOs and with companies that do incident response.
One of the incident response companies was telling me they're seeing their clients are seeing so many ... attacks that they do not have the ability, they do not have the bandwidth to support their existing clients, forget signing up new clients. They are jammed full in responding to incidents that their existing clients are facing.
CISOs, multiple CISOs have told me that they're seeing between 5 to 10 X number of attempts on their systems. Not all of them succeed, but they're seeing 5 to 10 X the number of attempts. And the reason for all of this is exactly what you called out, which is the adversary is using LLMs, using LLM models to generate attacks which have become so much easier.
And I want to dispel one myth that might be out there, that to define and exploit one, we've always believed that you need a high level of skill. Some of these exploits are very sophisticated from a computer science engineering perspective. Well, that skill level required has gone down because of LLMs.
The second myth is that you need access to very sophisticated models like the frontier models. No. Even open source models can generate plenty of exploits, and you don't even need to run a scan or have a deterministic engine to do that.
You know that there are already so many known vulnerabilities existent in every code base out there. Every code base. So you can generate exploits very easily with even open source models, really cheap, and start trying to exploit these.
So which is why the volume has gone up. Mm-hmm. The other thing that comes to mind is we have counted on our ability to rank the severity of a vulnerability as kind of our primary method for figuring out which ones to patch first.
But I think that's falling by the wayside because the adversaries are also starting to chain together low level vulnerabilities to create something that's a little more lethal. And do we need a different way of thinking about how we prioritize what we're going to fix? You're absolutely right.
So we've been in this business for a long time, Mike. Historically, let's say pre-LLM, pre-Mythos, enterprises remediated less than 10% of the vulnerabilities they discovered. Why?
Because they just didn't have the bandwidth, and exploiting them was quite hard. So they tried to strike a balance between remediating and shipping product. After Mythos, and I use Mythos, but it's also true after any of those frontier models have come out, I think customers are looking at remediation very seriously and are doing a lot more remediation now or intend to do a lot more remediations.
To your point of what should they focus on, you're right. Some of these models can string together medium and low vulnerabilities and create new exploits. That's true.
But most customers are still focused on, the criticality doesn't go away. If you have, let's say, 10,000 vulnerabilities in your code base, it would still make sense to first tackle the most critical ones, but not stop there. Don't stop at 10%.
Your critical might be 15% of the total. Fix all of them, then tackle the high, then tackle the low. And people are doing not just the criticality of the vulnerability on the one hand, they're also using pen testing, AI-based pen testing to determine how to prioritize.
So we, for example, when we get 10,000 vulnerabilities, let's say, for a customer's code base, we use attackability, reachability, and things like that to not just go with simple criticality of the vulnerability or ratings of the vulnerability. We also look at attackability and reachability and use that to also determine how to prioritize them. So you're right.
It changes the dynamic, but you still have to prioritize because in my example, you won't be able to tackle all 10,000 overnight. It's just not possible. It's not humanly possible.
Right. So a lot of folks would say that only a small percentage of the vulnerabilities are actually reachable and exploitable, and so they feel like they still have time on their side. But is that true anymore?
And have we reached some point where maybe we need to automate patch management despite the potential that a patch might actually break something, but maybe the risks of the security flaw are greater than the potential risks of the downtime. Yeah. Which is exactly the trade-off people are doing.
I'll give you an example of one financial institution that's a customer. And they're a very large financial institution, so you can imagine they literally have all kinds of technology ever invented, right? They have a little bit of that.
The CISO was telling me that they have implemented a policy that they're going to patch everything and only allow N minus one across all their systems. So yes, people are patching. You have to patch things much more rapidly, which is why your point about needing automation is more important.
I think people are also patching, and maybe they don't have enough time. And my conjecture is that a lot of the incidents that have come up this week, if they are not security incidents, you would wonder why has our stability gone so bad that we have three big things, airlines, banks, and the intelligence systems all hit at the same week. I think it's because people are patching and making code changes faster than they did before, and they don't have enough time to do all the checks that they did in the past.
And because of that, we are seeing, and we are going to continue to see more instability in systems. So. Hmm.
But it's a trade-off, right? Like Mike, you said. Do you take more risk on security or you take more risk on stability?
And those are always trade-offs, and people are trying their best to make the right decision there. Right. So ultimately, what's your best advice to folks?
Because I think a lot of times we've had this disconnect between application development, security, and ITOps for as long as I can remember. But is it time for those three teams to have a more meaningful conversation, shall we say? It absolutely is, and that's been happening for the last few, I would say at least five years.
There's a whole trend of DevSecOps, which is all about integrating security into your development process. So that was already underway. I think what has changed is the anxiety around vulnerabilities, not just reachable and not just runtime vulnerabilities, but code vulnerabilities has increased.
And therefore, I think every CISO I know, every dev team I know is taking security a lot more seriously. So those conversations are happening, and they need to continue to deploy tools that can help prevent, that are frictionless for developers, that have high fidelity. Because what you don't want to do is throw a thousand vulnerabilities at developers, and then they discover that 80% of them were false.
Right? So you really need high fidelity, and then you need agentic systems. You need automated systems that can cover both deterministic and probabilistic so your coverage is whole, and you need agentic systems to remediate.
You cannot have humans remediate thousands of vulnerabilities. So you have to have an agentic remediation engine, if you may. And most dev teams are shifting to an agentic workflow across the board anyway, and security just has to be part of that.
Mm-hmm. Is there a role for the government in all this, and what does that look like? Because some folks are saying, "Well, we need to have more regulations that require people to apply patches faster," et cetera.
But most of these issues historically have been handled by individual organizations. But what is that relationship with government entities in this age going to look like? So I think the government already has a role.
The government and quasi-government organizations. Like CISA puts out directives, and most enterprises take those quite seriously. In the financial sector, there's a industry consortium called FSIAC, which is all the financial institutions worldwide that collaborate on developing policies.
So it's not a government agency, but they work very closely with the regulators. And so the role of government, quasi-government industry coalitions is absolutely going to increase. As you know, we have many financial institutions as customers.
They do share information with each other. We share information with them to make each other smarter because we have to address this together. Doing it alone is not necessarily beneficial.
You can learn from each other, and there are agencies and quasi-government organizations and consortiums that are working on this already, Mike. All right. Well, folks, you heard it here.
A lot of this just comes down to fundamentals, and maybe for too long we've ignored those fundamentals. But in the age of AI, everything is happening at machine speed. So suddenly all those issues come home to roost a lot faster than you think.
Hey, Sandeep, thanks for being on the show. Thanks, Mike. Thanks for having me on.
All right. And back to you guys in studio.