Boardroom-Level Accountability: Elevating Software Security to the C-Suite – From the Source EP4

Welcome to another exciting episode of From The Source with your host, uh, me, Ilka Turnin. Hi. And I'm Brian Fox And Brian at a haircut.

I did, but I have your favorite all day DevOps, uh, hat here somewhere, just in case I need it. I still haven't received mine. I still haven't received my clap that we've kind of moved past the, uh, hat wearing stage.

And, uh, we've got to the, um, uh, got to the, um, uh, fresh buzz over here. Well, um, hey, Brian, we've got a pretty interesting episode today. I feel like we're, we're kind of in our comfort zone today, uh, because today's topic is to do a deep dive on the latest stage of the software supply chain report.

So I think a lot of people might not know what that is. So why don't you tell us, uh, sort of what, what the report is and, and then kind of what survey info we usually publish in it? Yeah, sure.

So this is our 10th year of doing, uh, what we call the Sonotype state of the software supply chain report. Um, and, uh, Sonotype, you know, we, uh, we run the Maven Central repository, which is the repository where all the world's open source Java components are sourced. Um, and that provides really unique visibility into what's going on inside of the ecosystem.

You know, which versions are popular, which ones aren't, which ones are being updated, which ones aren't, um, you know, geographic trends, time trends, all kinds of stuff. We can, we can do really deep research on there. Um, and we, we pair that up also with, um, our Nexus Open source telemetry.

We have like 150,000 instances of Nexus Open Source out in the world. net, uh, docker hub and Crates and Conan. And, you know, the list goes on and on.

And so that provides us, um, a pretty wide visibility into what's going on inside of, um, uh, the other ecosystems besides Java. And we, we use this data. Um, we, we often also do surveys.

Sometimes we do partnerships. This time we did a couple partnerships with different orgs, um, to pull in different pieces of data and try to look at what's going on in the industry. Um, and each year we tend to take a different look at what's going on.

So it's not like the same report just updated every year. We do new and novel types of research every single time to try to understand things like, you know, why aren't people fixing, uh, dependency vulnerabilities or, you know, what's going on with malware, um, you know, efficient algorithms for detecting whether a project has high quality or, or not, you know, um, looking at their behavior. So there's a lot of different things over the last decade that we've, we've looked at, but that's generally what the report is all about.

Yeah, and it's, it's, it is really interesting because, um, over the, sort of the last 10 years we've been looking at so many different facets that it feels to me like I'm still talking about a, a couple of research points from like a report three years ago, because it's still very relevant to, uh, what people are doing right now. And, uh, um, and, uh, sort of the lessons sometimes get a little forgotten. And sometimes it's actually, you know, interesting also to take a couple of years, a gap and, and take a look at if things have changed.

Like in, in this user report, there's a couple of items that we sort of refreshed and, and, uh, having a sort of corpus of that, uh, that long data, it really kind of gives you a perspective of how things have changed or not changed, um, uh, not changed at all. So, uh, so it's a very interesting read and actually, uh, you know, kind of, uh, a lot of, sort of nuggets of information when you, when you take them individually, kind of make you go, oh, yeah, okay, that makes sense. Like, for example, if we start from the, the very, very top.

Um, so the first big thing that we reveal in the report is, uh, open source adoption kind of sits at sort of a multi-trillion request scale. 6 trillion total downloads, and the bulk of that is formed of, uh, JavaScript and Java, uh, the ecosystem. So I think together those already get to about a 6 trillion, um, uh, request count.

But, um, aside from that, you know, kind of the biggest ecosystem that's growing surprise, surprise is Python. And, uh, can you guess, How do you think that is anything interesting happen in Python land this year? Yeah.

Let's, uh, let's think back, uh, chat, chat, uh, at, at LLMs, uh, generative, anything, This AI thing? Yeah, Yeah. This, this whole AI boom, I'm, I'm sure it's, it's, it's old, have now old news, but, you know, an incredible amount of sort of software supply chain infrastructures being formed around not just the LLMs themselves, but also all the auto tokenizes, all the inference work that goes into training models is basically distributed as Python packages is distributed through these software ecosystems.

So actually Python's like, just like done a double whammy of, it's the, it's the most popular programming language now on the planet, at least, at least according to TOB and GitHub. And now it's also, uh, boosted by ai. So it's really rocketing quite, quite heavily.

Yeah. Um, it is, it, it's not unexpected, but it is interesting to see that the, the massive growth there. I mean, JavaScript has always been the one with the most modules, most consumption, just merely because the, the individual pieces are so much smaller.

Um, you know, in a Java world, it's like, imagine if each class was its own module, you'd have 10, a hundred x. That's kind of what happens in JavaScript. But the C Python kind of kind of trending along similar numbers is, is a huge, huge step forward, uh, for sure.

Just shows you that, uh, the march of open source has not stopped. I mean, the growth rates, what, 40% year over year, something like that, right? Yeah.

Yeah. Uh, those Are every year it looks the same every year. Like it hasn't changed.

Yep. Well, in fact, this huge number, if you look at the diagram, it's actually show up even sort of sharper than it it's been before. So yeah, if anything, it's speeding up.

Yeah, yeah. The usage is like a fractal. The more we look at it, the more we zoom out the, the shape the world curve looks so thin.

Yeah. Yeah. But obviously, obviously sort of the next, uh, item in there is that, um, there's also sort of the underside of this, and this actually came out, I had a couple of chats on, uh, B Sky, which is where tech Twitter has now migrated to apparently it's flow up.

Uh, and, uh, there was a, there's an article actually written about this report, and it kind of really honed in on the, the risk side. Um, and I think, I think it's important that we understand the, the risk side as well, and that, that's something I really enjoyed in this year's report, was just sort of the seeing the evolution of, of how these things started from sort of named exploits in the early 2010s and all of that to sort of the situation that we're in right now. Yeah.

So one of the things that I wanted to do this year, given it was the 10 year anniversary, is really, you know, do a look back over the different things that we've, um, investigated over the years. Um, you know, refresh things that we haven't looked at in a while, and also just kind of take in how the industry has, has changed in those 10 years. And, you know, 10 years doesn't seem like a lot.

You know, we founded Sonotype 17 years ago, so 10 years is, you know, a little bit more than half. But, um, uh, we were well down the path of doing, you know, what, what the world would now refer to as software composition analysis in, in 2014. Uh, we were several years into it at that point, but it was still interesting to look back and see how early and how, how much, um, things have changed on the attacker side.

Right? And so, like, what I did when I looked at the tenure is I kind of, kind of sat down and said, okay, we've got the attackers, we've got the open source publishers, we've got the consumers of open source, and then we've got the regulators and what has, what has everybody done in those 10 years? And unfortunately, um, you know, the attackers have made the most progress, right?

So 10 years ago was one of the first, uh, big struts vulnerabilities, not the Equifax one that wouldn't come for another three years at that point, but it was a, a, a smaller one that didn't get as much notoriety outside of at least the financial sector. And the financial, um, uh, sector was using struts a lot back then. And so there was this vulnerability and, um, you know, it was, uh, properly zero day processed.

It was disclosed, responsibly, fixed, announced, and then it was days to weeks after that announcement was when the attackers jumped on it, right? And, and I, and I know this firsthand from some of the, some of the mailing lists that I have access to, um, that I could see in fact that that was happening. This wasn't a case that it was really being massively exploited before the disclosure came after.

And so, you know, one of the big dramatic shifts, you know, old, old presentations I had at the time was highlighting the difference between, you know, a, a fairly recent IBM report. Um, at that time that was showing that attacks came, you know, weeks to months following these disclosures. Um, but in this case it was a dramatic escalation to days.

And, and, um, anonymous was making a name for themselves around this time. And so big banks kind of, everybody suddenly had maintenance all at the same time, right? Um, so, so read read between the lines, if you will, but, um, you know, nobody raised their hand.

Nobody got hauled before Congress. Nobody really got fired over this. But it was one of the early shots across the bow, um, that same sort of cluster of years.

We also saw Shell shock and heart bleed, right? These were two very widespread open source security issues. Um, not so much open source components in the way that we tend to focus on 'em at Sonotype, like struts, like Log four J, but definitely open source in general be.

And because of the widespread nature of those, they did get some attention. They had logos, they had names, you know, they were celebrity vulnerabilities. Like we, we refer to them these days, but then not a lot changed.

It kind of went back. We were regressed to the mean. Um, fast forward a few years, you know, the Equifax, uh, breach happened, um, that was announced to be, you know, caused by a different struts vulnerability.

The difference there was, they were pretty open about it, but then there were congressional hearings and, you know, c-Suite lost their jobs over some of these things. And, you know, it was a, it was a another, uh, awakening of people starting to pay attention to what's going on. Um, around that same time, however, uh, I saw and started writing about this, this rise of intentionally malicious components and components whose attacks were designed not to go after the end user of the software, but actually trying to steal things from open source publishers, you know, credentials to publish into these public repositories were in fact the target.

And I remember thinking, why is nobody talking about this? This is a dramatic shift in the attack mechanism. And, you know, we've been, we've been talking about that for now, seven years, and I think it's still something people struggle with.

They don't recognize this threat. There was literally one this week where, uh, where a, um, video player component had their, uh, I think it's called Lama Player, got their author credentials stolen. And somebody literally went on NPM and published a couple of malicious versions.

And what it did was, if you had it embedded on your website, 'cause a lot of people have it on CDMs, it actually just displayed a, Hey, connect your crypto wallet to this website type of situation, like login with your crypto wallet. And if you'd done that, it would've just totally your crypto. So yeah, unfortunately people struggle with Exactly, Closely like that.

That's exactly what I expect. If I'm watching an embedded video, of course it makes sense for me to enter my crypto credentials, but video, But the Point is they do these things because they work. Yeah.

Um, yeah, I Mean, point isn't, it is like, it's like, um, you know, that what, what often gets lost in this discussion is that even if we've said it, even if it's been said five years ago and it feels like a very blase risk, it's still a risk, still a risk. In fact, Anything, I know those things are happening like more. That's right.

Um, you know, so, so that, that is continuing to explode. We're gonna get to that exact number in a second, but continuing the evolution of, of these attacks. You know, what finally I think got regulators in the, in the industry in general, sitting up was sort of two rapid fire things in back to back years.

The first was SolarWinds, um, which was not really a component based attack, but it definitely had elements, you know, it, I think it's fair to call it a supply chain attack, right? Um, where, where malicious code was embedded into the SolarWinds product that then got distributed downstream, that that was a bit traditional. The, the end user was the target in that instance.

Um, but certainly, uh, government was heavily affected by this as users. And, and that really started getting people to pay attention. And then before, you know, the news cycle moved on and everybody went back to the mean, then we had Log for Shell, right?

Um, and that, that I think finally got, uh, regulators and, and everybody sitting up and kind of looking at, at this challenge, um, as we'll talk about it. It didn't, it, it didn't, it wasn't awesome in all of the instances, but it definitely finally, I think, broke through. Um, and, you know, if, if you're thinking about that timeline, that's like six years, seven years from some of these early ones, um, till it became really, really prevalent.

And then the other kind of bellwether thing that happened this year was this attempted attack on xz Utils, which is a compression library used in, in Linux distributions. This was, um, you know, a several year long, um, attack to get access, uh, to, uh, uh, a single maintainer project. They created meaningful code, they created very sophisticated backdoor, um, tech.

If you haven't looked into it and you, and you like code, you should definitely look into it. The way they, the way they hid that in plain sight is, is in some levels fascinating, right? And I mean, it's genius if it wasn't evil is That's Right.

That's right. That's right. It was, it was very fascinating to see that, and it was a near miss.

And so I, I predict that, that this one will be like, that first struts thing. It's the thing that happened, and most people forget. It happened some number of years from now when finally these things happen enough times that, um, it becomes the norm.

Because I think xz util scared a lot of people, but I feel like the conversation has moved on back to, you know, the basics and, and, um, this problem didn't go away. There's 0% chance that the attackers having gained access to a project, um, two years ago just stopped at one, uh, because they had two years to gain access to others before they attempted the attack. So it just logically makes zero sense that that would be the case.

So we have to, we have to assume that there are more out there and we just don't know about it yet. Yeah, I think, I think it was the open JS Foundation put out a press release, uh, a couple of months after where they actually said that they found some traces of evidence, uh, of sort of similar PSYOPs where, you know, the maintainers started getting these sort of very, I, I've, I've seen the emails. I, I helped connect the dots on that one.

It was basically the same. They made more mistakes. So I, it feels like they farmed it out to junior, junior engineers to try to replicate.

Um, but that also happened, that only hap that happened, um, within the last year. It was very recent at the time we discovered it. So there was still a two year gap between that first of takeover and this other failed attempted takeover.

What's happened in between by the, by the ones that got it right the first time. That part, I think that story is still being written. Often when we tell those stories like this, people kinda get us back on the fact though.

Hey, that's one example. But you know, where there's smoke, there's usually sun fire. And, and you know, one of those, uh, one of those sort of key things, uh, that you, that we see is we kind of keep track of when we see a malware incident, like something that is malicious in nature, either the stakeholders or somebody up upright publishing malware or things that we consider malicious.

And we kind of publish a number on that every year. And this year it actually hit, uh, top just over 700,000. And I think you were saying to me yesterday that is probably closer to 800,000 by this stage.

Yeah. By now this, this stat. 'cause we were writing the report over the summer, the stat's a few months old already, so we're probably closer to 800.

Yeah. But it, it was 250,000 ish at this time in last year's report. So now we're at 700,000 and this continues to explode.

You know, we were seeing 600, 700% growth in early years on small numbers, but the fact that it doubled last year and it doubled again this year on much bigger numbers is significant. And, and I'll, I'll take the moment to point out right now, we did some other research in, in a different part of the, uh, report this year. You know, there's about 7 million components out there in, in the popular open source ecosystems.

We looked, some of the research we did was to find out which ones are actively being downloaded, which ones are actively being used in applications. It's about 10% of that, 10% of that is 700,000. That's about the same number of these intentionally malicious components that we've tracked over the last handful of years.

That's pretty striking when you think about it. When you think that as we sit here right now, we're probably already in a place where there have been more fake malicious components published to NPM and Python and, and some of those ecosystems than exist in popular components in all of the ecosystems combined. That's a, so from a developer perspective, the signal to noise ratio is going in the wrong direction.

That is a pretty profound and pretty alarming thing when you, when you think about it. I mean, the counter argument to that has always been, well, we know which projects are good and we will stick to that. But, uh, increasingly, you know, when we looked at the makeup of the types of malware that we actually saw, I mean, a couple of things that really stood out to me was, uh, about half of it is sort of what we would class as well, not not quite a 40% is what we would class as sort of, uh, potentially unwanted, uh, features.

And which is just packages doing stuff that they don't tell you that they do, which can range from just protest wear, things like, Hey, I, I don't want to do this for free anti work protest type stuff, uh, to more malicious stuff, like a little bit more advanced protest that actually delete your miles if you've got certain keyboards on your system to where we start getting to much more serious type of malware being distributed, being pushed out. We've even seen cases, you know, this exit key case was like the most genius because it was very bespoke. It was hidden over two years.

It was encrypted, it was hidden in the file encryption sample files, which is just, yeah, clever, clever evil, but clever. Um, but you do see a fair amount of malware that, for example, steals environmental secrets, steals, uh, authentication tokens, steals, uh, developer environments. And the more that we've looked at it, it actually fits really well into the sort of standard criminal ecosystem.

So the National Cyber Security Center here in the UK published a, you know, killer of an article a couple of years ago where they like spoke about how the ransomware ecosystem works, where we've got these, you know, gangs that gain initial access and they do it by distributing malware. Um, then they kind of bundle that access like, Hey, here's a hundred machines from Bank X onto these affiliate marketplaces, and then they get picked up by actual ransomware gangs or criminal gangs that then do the thing which might be dropping ransomware, stealing secrets, you know, executing other sort of subsequent attacks, uh, over there. And what's sort of been chilling is the types of malware that we discover, especially the, the, the really, really serious stuff is stuff directly connected to that criminal ecosystem.

So it's clear that, that it's not just individual operators kind of doing proof of concept packages. It's fairly organized stuff, right? Uh, you see there.

So, so, um, uh, you know, it, the sort of inflation of that just shows that, uh, that the growth of that number is growing way faster than the growth of open source consumption itself, which kind of tells you that the bad guys are really adopting it at scale, which isn't to say It's working and it's working because we don't have good defenses around it, right? So the way you manage your typical vulnerabilities, um, is very different here, right? A vulnerability is a bug that may or may not be exploited.

Uh, these malicious targets they execute as soon as a developer downloads it because these ecosystems execute things. So, um, so that, that's part of the challenge you need to defend against it upfront. We've been building solutions for this for a long time because frankly, back in 2017, I started talking about this and I had some, you know, customers, audience members come up and say, Brian, how do you help with this?

And I didn't have a great answer, honestly, because it was a very different dimension to the problem. So we sought out to build a, a solution for this, and this is how we've been catching all of these, uh, these malicious components for all these number of years. Um, you know, so, so definitely there, there is a way to solve this problem.

Um, but I think many people want to think about it like vulnerabilities. And you know, I, I did some stuff on LinkedIn earlier this year trying to, to play with some analogies and you know, I think the one that sticks is, is more of a food-based thing, you know, like food might go bad, might spoil, right? And, and, um, that could be considered a vulnerability.

You piece of moldy cheese, for example, you might cut the cheese, you cut the mold out and eat around it. Like that's a thing, right? Or you look at goes bad in curdles.

That's right. But, but in, in this instance, you might look at it and you cut out the bad piece, but you wouldn't do that if the food was poisoned, right? Right.

It's a very different thing. You don't look at, if I, if I told you one of your devices had an explosive in it, you wouldn't say, that's okay, I'm gonna update my software next week. Yeah, you what?

Vote, throw it out the window. Get away from it. Yeah.

And so, you know, uh, uh, the, the malicious element of this, I think gets caught up in organizations that are struggling to deal with the, the, the massive amount of vulnerabilities that are out there. And it's unfortunate because it is a very important thing, it needs to be dealt with separately. And, and, um, as we're seeing with the rise here, not enough people are.

Um, so let's, let's hop ahead a little bit. We're, we're, uh, running out of time a little bit, but you know, we, we also looked at publishers, um, the open source publishers and what happened in the last 10 years. And just quickly, I'll say, you know, unsurprisingly, we've seen that, you know, 1400% of them are releasing much more faster year over year.

That's not surprising. But what is surprising is the time it takes them to remediate their own, uh, dependency vulnerabilities has gone from days and weeks to, on average, 300 days, right? And part of this is, is caused by this massive rise of vulnerabilities that are out there.

The number of CDEs, um, you know, is skyrocketing just like the, the malicious components and certainly open source maintainers are struggling to keep up, right? So they've reported in surveys that they're spending three times more time dealing with these reports, dealing with vulnerabilities, and yet they're still taking on average 300 days to get to, to them in, in most of these cases. Um, the good news is we found when we broke it down by, uh, organizations or projects that have foundations support, or foundations that have, uh, that are being the, the maintainers are being paid, um, they have significantly better outcomes, you know, three times, uh, better outcomes in some dimensions, seven times better outcomes in other dimensions.

Um, so, you know, there is a glimmer of hope that if we're able to help better support the open source maintainers, they demonstrably do a better job at, at producing, um, safer components that everybody is relying on, right? So we need to be able to, um, we need to be able to focus on that a little bit, uh, going forward, I think. Yeah, I think so too.

And I mean, um, one of the findings that we had, I think a year ago or two years ago, is generally open source projects are better than industry at taking these fixes and actually applying them, you know, open, open, the conversation is about the maintainer struggling. That's absolutely a true conversation on even cases like log for Shell. I mean, they made the fix in like 10 days, uh, which is less than That on a holiday weekend, no less, uh, Thanksgiving.

Yeah. Yeah, Yeah, exactly. So you see, if you, if you compare that to what would happen in an enterprise or in a closed source setting, it probably would be a very different story.

So part of, you know, often when I look at these sort numbers, it's, it's really a symptom of the success of open source itself. Like, there's just so much of it and it's so networked with one another that that creates that complexity. And the projects are just like inundated basically with this stuff as a, as a result.

Yeah. Yeah. I mean, sometimes people look at the, at the conversation, I think this is the article you were referring to before, looking at, you know, highlighting of risk at open source and saying, yeah, but, you know, making, making the, the false, uh, straw man argument that, that we're trying to say it's, it's worse than commercial.

I, I don't think it is in most cases, but it's certainly easier to observe. And I think the problem lies in the fact that organizations, um, often are making these choices around open source. Well, frankly, the organization is not making the choice.

A developer or group of developers are making these choices. They can often be uninformed, and the organizations don't have your typ typical procurement processes in place to be able to vet and recognize and track and update open source components like they do with the software that they purchase, right? So while probably open source on average actually does a better job at this, because of the way it's procured sort of, uh, at the, at the bottom level in development, and without the, the transparency, the way the organization manages it leads to potentially worse outcomes and what you see inside of commercial software.

And I think that's the nuance. It's a little bit hard sometimes for, for people, um, that, that want to build those straw man arguments to really see through the noise there. Yeah, I think so too.

And I mean, um, I mean, I, it it's often well intentioned too, but the problem is because there's just so much of it and there's so little visibility and source shared and standards, if you will, that doesn't happen. And that's part of the reason, actually one chapter, we, we don't have time to like super deep dive into it, but the last sort of piece that, uh, we did in there was to take a look at sort of the regulations that are rising as a result. Yeah.

So all of this is for better or for worse no, no sort of judgment on, on the validity, on the validity of necessity of these. Yeah. I do wanna add on the consumer one.

You know, the last three years we've, we've calculated, um, this number and so we looked at when something is being downloaded from a public repository and it already has a known vulnerability, is that happening because open source hasn't fixed it yet? Or is it happening because the open source consumer hasn't, uh, made a better choice? 96% of the time it's the latter.

Only 4% of the time is something consumed with a known vulnerability that isn't already fixed. And unfortunately in the last three years, that statistic has not changed. It's 96% of the time for the last three years.

Right? Yeah. And, and the other one that we've, we've reported on a lot is, you know, the percentage of vulnerable log for j downloads at any given rate.

You know, uh, last year we were at about 30% or finally yeah, uh, to 13% this year. But look, it took three years for the most notorious, most widely, uh, documented discussed. The, the issue that was on, uh, the top 10 list for the NSAs nation state attack exploits for three years.

And it took us three years to get to the point where 77% of the choices are the right ones. Now that's still pretty terrible, right? And this is, this is, I think where, where we see this year after year is that all these things are happening.

The discussion tends to go towards, you know, we need to help open source. Um, you know, maybe we shouldn't use so much open source and, and like I said, we should help open source. The numbers get better when you do, but 96% of the time, the problem is the organization consuming the open source hasn't updated their software.

Yeah. So we can do all these other things and we should do them, but until the consumers fix their behaviors, we're playing in the margins of that 4% of the behavior, right? And, and that's the problem that I keep, keep trying to shine a light on.

And I think, frankly, this is why the regulators have stepped in because the things, it's, it's, it's, I think it's a lack of motivation. It's not an understanding in most of these cases, right? And so we did an episode, I think it was last time already, where we, we went deep on, on the regulation.

So if you haven't seen that, you should go back and take a look at that. Um, but, but I think, you know, there's a little bit of good news that we're finally getting some regulation after, you know, 15 years of shouting from the mountaintop and not seeing things change. I think some of these incoming regulations, the u European Union's Cyber Resiliency Act, the product liability directive in Europe, um, I think will be game changers, the PCI four standards that are coming out, um, that they've come out.

But they will take effect, I believe in March of 25. Correct. Those have elements in it that are similar.

They require software bills and materials, things like that. I think those things will finally start to move the industry in the right direction. It's, it's very slow right now, but I, I think at least we've turned the corner and, and the right attention is being paid on these things.

Yeah. And that's the path that I think is a good place for us to, um, uh, wrap up, uh, today's episode as well. I think, um, you know, the SBO adoption itself, I mean, you know, in itself it's a little bit of a controversial subject depending a little bit where you're looking at it, but whether or not we like it or not, it's just become a requirement for all the reasons above.

Um, and so it'll be interesting, we, we do take a little bit of track on how projects are starting to publish SBOs and what sort of level they're at. So definitely worth the, where's the read, if you wanna kinda look at the data there. com/sscr.

There's a very handy dandy, uh, QR code we can attach to this as well. So with that, uh, thanks very much. I'll talk to you next time.