The Status of Software Supply Chain Security with Sonatype’s Brian Fox

This is Textron tv. Hey guys, thanks to the throw. We're here with Brian Fox, who's CTO for type, and we're talking about this report that they put out on the state of the software supply chain at their conference today.

I think it's called the All Day DevOps event, which, um, I believe is a, uh, an oldie but a goodie that's kind of making a comeback here. So, Brian, welcome to the show. Yeah, thanks for having me.

Give us a little highlights of the conference itself. I know you guys have been kind of doing this in the past, but it seems to, uh, have a new life. So, uh, what is the history of this event?

Yeah, so all day DevOps, um, is an event, uh, that we've been running for, uh, uh, many years. I've lost track exactly how many, um, but it basically, it's 24 hours of live stream talks. Um, the idea being there's live talks, uh, sessions for everybody in the world, and it's the all day DevOps.

So it's a big, uh, it's a big production event from the team. Uh, it kicks off at 3:00 AM Eastern time and then runs all the way till the next day. Um, and so there's sessions from, from people worldwide talking about all different things, different tracks, um, you know, related to DevOps and supply chain security, um, and, and all those fun things.

Um, in parallel, it also coincides with our launch of the 10th annual state of the software supply chain report. Um, and so we have three different sessions, um, specifically that I'm partaking in, uh, to talk about that. One of those, um, is a, is a keynote where we're diving into some of the details.

And, and we'll get into some of that here. We also have, um, two other reports that have come out sort of in our space that we've collaborated with. There's a tide lift state of the open source maintainer where they talk about, um, you know, some of the findings, they've, they've surveyed maintainers and, and, and, uh, and, and things related to that to understand, uh, where they're at.

And then also, uh, there is the Enos, uh, financial open source, uh, foundations, uh, state of the open source and finance report. And so they have overlapping, uh, findings. And so we're gonna be, um, sort of comparing and contrasting notes, um, in, in related sessions, uh, today.

Well, what is the state of the software supply chain? 'cause when I talk to folks, it, depending on what day it is, the glass is either half full or half empty. And one thing no one seems to be clear about is, are things getting better or worse?

Huh? Uh, both, um, anybody who's heard me speak on this knows, uh, I'm sort of a half, half empty kind of guy on this, uh, after looking at it for so many years, you know, I, I, um, I, we, we added a chapter this year, uh, a 10 year lookback chapter. And, and I took this one on and did some of the research.

Um, and, you know, some of the findings we broke down into sort of, I guess it's four different sections. How have the attackers innovated in the last 10 years? You know?

So these are things like, um, you know, moving from rapidly exploiting, uh, zero day vulnerabilities, um, vulnerabilities post disclosure, all the way to the more recent trend of intentionally open source malware that's being injected into repositories like in PM and Python and Ruby and things like that. Um, you know, we talk about those trends. Um, we'd look at how consumer behavior, uh, has or hasn't changed, you know, so these would be organizations consuming open source from the package repositories.

We look at, um, what, what changes open source publishers have made, um, you know, in terms of how, how quickly they remediate, what things they, they fix, these types of things. And then legislatures, um, as I'm most of the audiences, uh, I'm sure aware after the Log four J incident a handful of years ago, um, you know, regulators worldwide have finally stepped up and started, um, you know, pushing regulations. And so, you know, the summary of this is, yes, things, things are happening, things are getting better.

The, the bad part about it is, you know, the attackers, I think they win the award for innovation here. They've, they've certainly made the most change in the last 10 years. I would say regulators take second place.

Um, in terms of things that have actually changed, you know, because there was basically no government involvement, uh, up until very recently. But, you know, recently you've seen, um, you know, cisa and, and, and, uh, many of their efforts getting involved in the open source communities and, and doing a lot of documentation around, uh, their push for secure by design. Uh, you've seen in Europe, the, the CRA, the Cyber Resiliency Act, and some of the changes there kind of pushing organizations in the right direction.

But frankly, the consumer behavior is largely unchanged. When we look at, um, you know, the trends, we're still seeing, um, 95% of the time when a vulnerable component is being consumed into a project, into a product, there's already a better version fixed available, right? And so three years ago, that stat was 96, so we're still in the margin of which, which whole number it rounds to.

It's barely changed in three years. Um, we look at the publishers and, you know, open source maintainers are clearly getting inundated with requests and bug fixes and vulnerability reports. Um, you know, but the speed to remediate, uh, new vulnerabilities is actually going backwards, not going forwards, especially, um, you know, with the, with the volume of them, I think it's explainable why that's happening, but the end result is it's not great.

And so while they've, uh, made progress in improving the speed at which they actually produce new releases, the speed at which those releases fix things is not meaningful changing. So, um, so it's a little bit, you know, some things are better, some things are the same, which is really, you know, worse because the attacks have, have increased so much. And we've reached to the point where if the folks who are gonna do this, um, naturally because they wanted to, uh, create a better quality experience for software, and they care more about security, pretty much have started doing something about it.

And it's, the rest of the folks are more the laggards who, you know, are only gonna respond to some sort of require. Yeah. And I think there's unfortunately reason to be optimistic because the, the, the regulations are finally coming, you know, um, you're seeing it across many different industries.

You see, next year, the PCI four compliance comes into, into play that has strong requirements for software build materials, as an example, right? So while the, the, the SBOs as they're called, you know, they don't fix the problem, it starts to provide transparency to the problem. And I think that organizations are gonna be less, um, willing to ship software when they have to provide a bill of material that shows that they may have components that have known vulnerabilities in them.

So while they may not be required to fix them right away, the, I think the transparency, um, and the, and the pushback they'll get will drive us in the right direction. Um, so, uh, yes, the, the challenge is there are many organizations we've been working with, many in, you know, early adopters, uh, now for 15 years that have largely solved this problem. We've shown in the past, um, organizations that had, um, you know, prepared their, their pipelines and had a deep understanding of what components were used and where that they remediated log four J, um, across their thousand application thousands of applications to portfolio in days, right?

So we've shown in historic, in, in reports in the past that this problem can be solved. The trick is that the majority of commercial organizations are not paying attention to that. And I think you see that in the, what, nearly daily, you know, releases of our data are being disclosed, you know, social security numbers being breached and, and all these things.

I think it's happening so much we've become desensitized to it, and we shouldn't, we shouldn't as consumers accept that it's okay. And that organizations that are doing, uh, provably and knowingly terrible job at security should get a pass and just have to buy us all our, our 10th monthly, um, you know, security monitoring report, uh, for everybody, because that's the, that's the worst thing that happens to them. That there, there are no consequences.

I think that's what will ultimately need to change in order to change the behavior at large. Mm-Hmm. We have talked so much in the last few years about shifting left, and in some ways that was a good thing, but I also started to wonder if it just sent us down a path or led to an expectation that somehow or other, all we have to do is move all this over the developers and we'll be fine.

And I guess, are we waking up to maybe this is a shift left shift, right? Shift everywhere kind of problem? Yeah, I mean, the, the shifting some of the decision making and, and some of the visibility towards the developers definitely is, is progress, right?

Because the developers are the ones at the end of the day who have to make the changes that are being pushed on by, you know, these new legislation. But, you know, at the end of the day, also, developers are being driven by their organization to achieve certain metrics. And if those metrics aren't strongly security above just ship a thing, then you're gonna get exactly what we have today.

You know? So there are many organizations that, um, you know, that, that doing the security fixes, making better choices around these things is sort of not the thing that they're goaling on, that they're rewarding developers on shipping features as fast as possible. And if somebody has to slow down to make a better choice, or to retrofit a component that's seen as, you know, maybe, uh, an anti-pattern, right?

And so, you know, um, we, we've heard, you know, um, uh, miss Easterly from, uh, from cisa, uh, talking about, you know, until we make, uh, security a business risk, so security risk equals business risk, we're not gonna see changes. And I think she's right. That's basically the same thing that I'm saying until there are real consequences for organizations to do, uh, knowingly bad things, we're not gonna see a meaningful change and expecting the developers to shield that and, and do just do the right thing and they're gonna fix it all, that's also not gonna solve it.

You've been around the block a few times. Was there anything in the results of this report that surprised you? Um, not really, but, um, but there are some bright spots.

You know, we, we, uh, collaborated with Tide Lift as an example. We use some of the data on the projects that they support, you know, their report found that, um, maintainers that are, that are paid, um, to, to support their projects do a better job, um, that there's a number of metrics they found. We took that same data and looked into our findings, and we found basically the same thing that they, that paid maintainers that, you know, get some monthly stipend to work on what is otherwise a PET project.

They're three times more likely to, um, respond to security things than they, they fix things three times faster, right? So there are some, um, some evidence of, of progress to be made. The counterpoints to that, however, is the, the consumer behavior.

When organizations continue to use known vulnerable components, it almost doesn't matter how quickly the maintainer fixes the thing, right? Because everybody keeps using the bar broken thing. And this has been a, a challenge for a long time.

Um, you know, and so we've, we've, um, we, we've continued to find those things. You know, when we looked at, um, SBOs, for example, SBO M production, you know, a handful of years ago didn't exist. And so it's grown quite a bit.

8% of new components published have an sbo, right? So while the number of SBOs is getting larger, we're not even remotely close to breaking even on new components, let alone paying down the debt of all the components that don't have published s bombs, right? So it's sort of a case of like, a little bit of progress is great, but we still have so much further to go.

So these, these findings are not really surprising to me because I, I live in this world, but I think they might be surprising to others, and that's why we do the report to try to shine a light on these problems and, and try to help drive be better behaviors. Do we need to find a way to maybe compensate those open source maintainers to work on these security issues? 'cause I think one of the things that we saw with the Log four J issue was a lot of these people were like, Hey, there's only two of us, and, you know, I'm making this up, but, you know, I gotta go to a little league game and I can't explain to my wife that I'm gonna work all weekend on something I don't get paid for.

Yeah. I mean, I think there, there are organizations that are doing that, you know, uh, GitHub provides ways for donations to be made to Tide Lift, as I mentioned, has models to help, help do that. Um, you know, I think the log for J is a good example of what's more typical, though, that team did fix those things within days on a Thanksgiving holiday weekend, but we've seen now we're closing in on three years.

Um, up until very recently, 30% of the versions of Log for J being downloaded, we're of the known vulnerable versions three years later. So, you know, the teams and the open source maintainers by and large, and I think usually do a great job of turning these fixes around. The problem is on the consuming organizations not making the update.

That's that 95% stat that I mentioned before. So, you know, the, this is, again, 95% of the time when a component is being pulled down from a repository and it has a vulnerability that vulnerability's already been fixed, right? So it's only about 5% of the time are there, uh, components that have vulnerabilities that are disclosed that aren't already fixed, right?

It's a tiny, tiny sliver of it. So, you know, in, in, in some ways, um, you know, getting the maintainers to move faster is sort of optimizing within that 5%. I think efforts like regulators to help encourage organizations to take better ownership of this and do the right thing will address the 95% part of the problem, Right?

That covers the first two reports we talked about. And the third one is gonna cover one exactly. The third one is the state of, um, open source in, in the financial industry.

And it's really interesting to see the growth there. You know, um, not that many years ago, uh, many, uh, you know, I tried to put together a customer council, uh, uh, for Nexus Cyber Repository manager. And, um, many of the early adopters were in the financial industry.

And the problem that I had back then is they weren't even allowed to talk to each other. They couldn't even put on their LinkedIn profile that they were users because, I don't know, NDAs, whatever. Um, and, and fast forward, now we're seeing, um, you know, a large organizations coming together, working on common, you know, um, type of standards, uh, even technology to help, uh, achieve compliance and things like that.

So within, within that industry, I think they've made a, a, a ton of progress in the last 10 years of, you know, banks working together to solve common problems instead of spending all of their money competing on, you know, basic technology. And, um, there's a lot of work within those organizations to try to, um, to improve the way they interact with the open source community. So, um, the number of people who are contributing back to open source projects from these large banks is, is up significantly.

Um, you know, the amount of time their remain, their, their employers give them to, um, to work on these projects also up significantly. So that's what that report really talks about. It's a lot about, um, how, how the finance industry is really engaging with open source and what the benefits are.

Mm-Hmm. When we put all this together, um, part of the issue in my mind is that we don't really applaud when developers go fix a security issue. We certainly don't, uh, reward them and make that kind of a part of their ethos.

So as part of the issue that, you know, we're just not kinda setting the right, um, metrics in place to encourage people to do the right thing, Yeah, that is definitely the case. I've seen, um, you know, some writeups from some of our customers that have instituted that type of cultural change where, you know, um, we've even seen organizations talking about where different groups are competing with each other in terms of the number of, uh, dependencies and vulnerabilities they've fixed. And unsurprisingly, when you gamify these things the right way, they start doing the right things without, without being forced to do it.

Um, I think one organization reported in the last handful of years, um, they had remediated a hundred million findings, which is shocking, but it, it's a huge organization with something like hundreds of thousands of applications. Um, but at scale, um, it's still a shocking number. And they've done this because they've changed the culture, which is a little bit, bringing it back to the all day DevOps, right?

DevOps is all about the culture and trying to, um, measure things, but measure the right things so that you can drive the right behaviors. And we've seen time and time again when organizations actually focus on that and choose to make that a priority, the outcomes are great, right? And it's the, the problem that, that we're seeing is that not enough organizations are making that choice over just, you know, ship something and, and will worry about the consequences later.

We of course, are now living in the age of ai. Will AI save us from ourselves here? Um, probably not.

You know, I, I think it's gonna cut both ways. The AI makes it easier for the attackers to spam things at scale and make it look more plausible. You know, we've seen, um, you know, maintainers that are getting inundated with, uh, with pull requests and bug reports and, and fake, uh, fake, uh, reports that are hard to discern because AI has made them much more plausible.

Um, certainly AI, when trained properly, can help make better recommendations about those things. So I don't think it's gonna save us. Um, I think it's gonna, like every piece of the technology, it's, it's going to, uh, better enable both sides of this game.

Yeah. Um, what is it that you see, and maybe you guys will be talking about this at the conference, but, um, is there something that organizations are doing who do this well, who do it right, um, that others should emulate? Is there a thing going on here or a set of best practices or some sort of, uh, pattern centers of excellence, you know, who's, who's winning at this?

Yeah, I think, uh, like I mentioned before, organizations that sort of, uh, make it part of the culture that, uh, they celebrate when these things get upgraded, when dependencies are fixed, when vulnerabilities are fixed. Um, organizations that are encouraging their employees to get involved with open source projects naturally have a leg up from those who are just using them at arm's length. Um, organizations that are rolling out tooling to help understand and manage the supply chain, um, and provide visibility internally, will naturally start asking questions like, why are we using all of the possible versions of spraying at the same time?

That seems like a terrible idea. Why do we have, you know, 10 different persistence frameworks or five different logging frameworks? That seems like not a great idea.

And so, you know, organizations that are providing management, the visibility and the controls, um, like you would expect from any other supply chain, um, you know, they show better outcomes. And so what we need is more organizations really thinking about it this way, and not just focusing on, well just empower the developers and hope for the best, right? That's, that's where, where we're seeing the difference.

All right. And remind folks, where is this event? Where can they log in and join the celebration?

Sure. com. And, um, you can, you can log in.

All the sessions will be online after, so you can watch 'em live, you can watch 'em recorded. Um, they're all there for everybody to see. All right, folks, as they say, there's a happening going on online, and you should come and join us because, well, this is where all the cool kids are hanging out.

Hey, Ryan, thanks for being on the show. Thank you for having me. All right, I'm back to you guys in this two.