Stephen Chin, JFrog | Black Hat USA 2023
Stephen Chin, VP of developer relations at JFrog, shares research of security vulnerabilities that the JFrog security team has identified and analyzed over recent month, including a malicious attack on the NuGet package repository, which threatened the software development cycle. JFrog’s recent investigations prove that no open source software repository can be considered completely trust-worthy, meaning safety measures should be taken at every step along the software development lifecycle. Ultimately, developers need to take caution when curating open-source components for use in their builds to ensure their software supply chain remains secure.
Transcript
This is Textron tv. Hey, everyone. We're back here live and now we're a Mandalay based suite.
A black hat. Yeah. He's just downstairs.
It's still crazy. It's Steven, there's so many people down there. It's like I get a little paradigm.
I'm still not used to big crowds like that. Um, but anyway, we're here with my good friend Steve Chin, uh, Steven Chin from Jfr. And anybody who watches Text Drunk TV has probably seen Steven on with us any number of times.
Or, and if it's not Steven, someone else from J Frock. But we've got a bunch of different news to go over today. And, and first of all, Steven, thank you for coming up.
Is this your first black hat or have you done black hats before? No, actually this is my first black hat. That's what I figured.
'cause you Yeah. Yes. I've been behind the scenes mm-hmm.
On all of our recent sponsorships of Black Hat. But this is my first time coming out in person. What Was your, uh, impression?
Be honest. It's, it's just huge. You and I.
No one else is here. I mean, it's, it's, It's a big show. It's not quite as big as R S A, but it's, it's big.
But, but I think something else that that's nice about this is, um, more of the folks that we care, care about, like in, in the AppSec market, This is an AppSec show Yeah. Who are, who are doing software security are here. Mm-hmm.
And we're, we're getting in the right sort of conversation. So like, um, discussing stuff with, and we'll talk a bit about the, some of the announcements coming from the Linux Foundation and other folks Yeah. Who are here.
Like this has become the place to talk about and to see where the security industry's going. Um, maybe, Maybe. So I think R S A covers security, cybersecurity in all of its facets.
I think Black Hat is very AppSec focus, but I think the interesting thing is, if you look at the history of what blackout is, blackout was supposed to be about researchers showing their latest research, and it's morphed into a want to be r s A in the desert with a, with a specialty in absec, quite frankly. And I'm, I say that with the most respect. I've been in security 25 years.
I liked it better when it was researchers and, and Barnaby Jack was up on that stage making a t m machines without $20 bills. And, and I've seen a bunch, or, or people, I don't know. I I, we have people in the background, you're not on camera.
When they would go up and disclose vulnerabilities against Cisco before Cisco fixed them because Cisco was late fixing them wall, that's when black hat was black hat, the wall of sheep and, and those kinds of things. It's tame now. It's, it's So, I mean, I think, oh, security research has become a profession.
It's mm-hmm. It's a, it's a field. We, we have this huge security research team at Jfr.
Yes. So we, we employ some of the best in the business who originally were, um, doing defense and attack for the Israeli military. So they're, they're very good at their jobs.
Mm-hmm. And, um, I, I think that they're also the people who are doing interesting exploits, right. Showing the research, and they are presenting at conferences like Black Hat, but then also their companies and the vendors want to, to show, show off how to mitigate, how you can actually integrate this in here.
Well, they wanna make money. They want customers. Yeah.
I, I think what's interesting, and I don't know if you're staying for defcon, so it used to be the black hat and Defcon were very joined at the head. Mm-hmm. Mm-hmm.
Defcon, if Black Hat was edgy, DEFCON was downright raunchy. Right. I mean, and now Black Hat has gone corporate and Defcon is edgy and, you know, that's fine.
I think it's part of the maturation of the entire mm-hmm. Security space. But that wasn't why we brought in your talk, Steven.
I wanted to talk a little bit. I, I, I had Jen's on tech drunk tv, I guess about two or three weeks ago now, a new JFR offering. Well, you know it better than me.
Let me ask you to talk about it. Yeah. So I mean, what, what Jen's, um, announced mm-hmm.
And this was the, the initial offering for our new jfr curation service. But, um, as, as a long time developer and somebody who's been using binary repositories as a way of, um, controlling what open source libraries are coming to your business, um, frankly, what what we're doing is actually very simple. We're taking a best practice in the industry, which is to audit and secure and make sure that your developers are pulling only libraries which are vetted and secure.
And we're making it easy to do. Yeah. And, and for such a simple sounding thing, it's a long time coming.
No, I had this conversation with an interview we did earlier today. Part of me wants to think that the people who maintain these repos should have some responsibility to the users of the repos to do at least a reasonable, I'm not saying they gotta go over 'em and beyond, but do a reasonable job to make sure that those repos are, are free of, you know, let's call it really either old versions or very obvious malware and, and so forth. Yeah.
Okay. So, um, ba basically, and okay, so let's, let's back up into what curation is. Okay.
How it actually defends you, and then like, like what the upstream risks are. So, um, typical application security looks at scanning and looking for vulnerabilities in deployed apps and production and what you're pushing into production. Mm-hmm.
That's too late. And if you, as you shift left to go to developers to go to the folks who are building the software, ideally what you'd want is you'd want a, a curated, vetted, secure list of libraries which you're building your software on so that you're not introducing potentially malicious libraries Yes. Or software versions with known vulnerabilities or any of that stuff into your software supply chain.
Now where are those libraries coming from? And typically those open source libraries are coming from central repositories. Yes.
So they're coming from N P M, from Maven Central, from Pi Pi, from Ruby Gems, et cetera, et cetera. Could Be Dock Docker, uh, hub as well, Docker Weller hub. And I, I think it's a, it's, it's a reasonable question of why can't the upstream central repositories do a lot more of this hard work of securing the supply chain?
Um, I, I think that there, there is some discussion as well. Um, so I know that there's, there's ongoing discussions in the EU about putting constraints on foundations and, and folks who supply software libraries to have some responsibility and onus for it. Um, now the, the challenge and risk with, um, vulnerabilities is not all vulnerabilities are created equal.
Mm-hmm. And so, um, many things which get a very high C V S S score for enterprise applications, if you're running it inside of a container, if you aren't, um, allowing user access to the system, those vulnerabilities, they might not be exploitable, you may have already mitigated it by, um, for example, like putting some code in which catches the exception Yes. Which blows the stack or whatever it is that the, how the exploit is triggered that already may be mitigated in your code.
And that version of the library you're using and the VUL known vulnerabilities against it may not be applicable to you. And mm-hmm. It's, it's hard to say that this, this software version shouldn't be distributed just because it has a vulnerability, which, which may not, I actually impact people who are using it in production.
So if the, if the central repository takes it away, that's also the system of record for, what was this original library and like, like what did it contain? Maybe they don't take it away, maybe they just don't allow it to download, or they download it with a, you know, a cigarette pack warning or something. Right.
Yeah. So I, I think you'll see a lot of the central repositories are now showing vulnerability info by cross match it against public databases. They're, they're trying to do a better job of taking down things which are just plain malicious.
Right. But NPMs huge, how do you monitor? No, I get it.
Something that large. Now I think when you look at that, this sort of security, what matters more, um, um, is having the full audit trail in providence of who built the artifact, where it was packaged, what it contains in terms of dependencies, all of that information so that your systems can, can audit, can identify potential risks, and then can stop it from becoming an issue. And it's, it's a combination of what we're doing with curation, which is the, that's, that's the firewall, right?
That's, that's the, the component firewall, which stops it Coming from your company. But then it's also doing the runtime scanning of what libraries are actually using in production of being up to date on the latest zero days and cvs, which are coming in, and then having like a, like an end-to-end security strategy where as things move through your DevOps pipeline, you're securing it at each step along the way. So taking a longer view, right.
This is actually very, it, it mimics what we saw in sort of traditional vulnerability scanning back when I, you know, was doing security companies, right? There was, uh, it's a guy named Giddy Cohen who actually came outta Israel and Giddy Company. I forget the name of the company.
It's still around though. Giddy, I think, I don't think Giddy ISS there anymore. But, you know, they used to generate what they call attack graphs.
So it would be first, okay, there's a vulnerability there. Is it reachable? Is it exploitable?
Right? The fact that it was vulnerable, what in and of itself mm-hmm. Wasn't, wasn't it?
And, um, I think we're starting to see that now on the left side of the equation, right? Yeah. In your pipeline.
Yeah. There's a couple things we're doing specifically for that. So one is, um, contextual analysis.
Yep. So, um, it, it will actually, from a binary perspective, decompile your codes, see if you're calling the affected a P I or if you've somehow mitigated the vulnerability, and it will tell you this, this vulnerability is not applicable mm-hmm. In an automated way, so that you get fewer false positives, it's easier to mitigate.
And the other thing which we do, which helps quite a bit is, um, a, a jfr severity score, which is applicable to the, the enterprise application use case. Okay. Rather than the C V S SS score, which we still show you, but that's really what In a vacuum, Uh, the researchers, the, the research used to find the defect proposed.
It, it goes through a committee, there's some scrutiny on it. That's The variability in a vacuum. I'm sorry.
Yeah. Yeah. So the actual applicability of that, or security exploit being, um, targeted in the wild on a typical enterprise system.
C V S S score is a very poor match for that as a criteria. And what happens is security teams or, or DevOps teams, they get told everything with a C V S S score of, of five and higher, four and higher has to be mitigated. And they spend a lot of time explaining why vulnerabilities, which don't matter, are not applicable or, um, really don't pertain to, to, as a real security risk.
Why don't Have To be, why they don't have to be. And and conversely, there's some security exploits which are getting underrated for the same reasons. Right.
It, it goes both ways. But here's not the rub. But here's the nightmare.
Developers and security teams can't always agree on this. When you put these kinds of decisions in the hands of politicians, such as in both the EU and here in the States, they're talking about rules that say software must be free of all vulner all known vulnerabilities. So they're, they're just chopping everyone off at the knees.
Right. So you can't be any taller than that because, you know, if you're a politician who doesn't know this stuff, it's a no brainer. Of course.
I don't want software with vulnerabilities. I think that's what we need to be afraid of. And that's why we need to, you know, the old saying, right, the eight most feared words in the language, I'm here from the, I'm from the government and I'm here to help.
Right. Those are the kinds of things we need to be cognizant of. Yeah.
I think when you look at the, um, the recent, um, legislation requirements from NIST and other folks mm-hmm. In the government who are doing security, um, the, the way it's written technically, they say it must be free of security exploits, which could be applied or could be exploited mm-hmm. In the, in the application.
And so they're putting the onus of the, the application creator to then explain or justify why it's not applicable. Um, and also they're, they're putting the onus on the, the suppliers to, to know all of the software components, to have a full bill of materials, to, to know all the components which might potentially be impacted. Um, and the, the government's power is purchasing.
So they, they can apply this to software Absolutely. They purchase, but the, the private sector naturally follows whatever the government does as a best practice Military grade. Yeah.
So, so that's obvious. That's one issue of it. The second issue is this is not static.
You know, what's not exploitable today may very well turn up to be exploitable tomorrow and vice versa for that matter. And so the problem is, it's like, you know, when P C I, I don't know how much you know about P C I, but when P c I was in its heyday right, of scaring the hell outta people about being P C I compliant, they had a thing. Not one P C I compliant company was ever breached.
Did you know that? Because the moment they were breached meant defacto you DCI on its face that they were no longer P C I compliant. So with a, with a, a clear conscience, they'd get up the P C I committee or, you know, uh, people were governing the P C I and say, we've never had a breach of a P C I compliant company, but what about this one?
That One. So if you applied that to security, we wouldn't have company selling software anymore. Exactly.
I mean, it's a ridiculous, and that's what I'm afraid with that we need to pivot though. But before we do, let me tie the loop on, on curation available now mm-hmm. People can go get it at, So it's, it's part of the Jfr platform.
Yep. Which is an end-to-end DevOps package management, DevOps security, and, um, curation helps you to lock the doors. But then X-ray, our advanced security offering, those do all the downstream scanning.
Mm-hmm. So that you're making sure that everything you're pushing to production is free of known vulnerabilities. So curation is great.
It's not meant to be used alone. It's meant to be part of a, a bigger set and you can get it from the jfr, uh, set. Alright.
Let's switch over to your other hat, which is Open Source Linux Foundation Board Member Advocate. What's happening there? There's a bunch of exciting stuff which got announced recently from, um, the, the REST Foundation, which you're a member of, and also the, um, the Linux Foundation, which we sponsor, um, multiple different sub foundations including the C, ncf F, the C D F and Open S S F.
Mm-hmm. Um, on the Rust Foundation side, we're, we're really, really happy about their new Rust security report that they announced that talks about what they're doing to secure the Rust Ecosystem. Um, they talked about some of the contributions we made earlier this year mm-hmm.
In terms of, um, finding some big vulnerabilities in the rust H G T P libraries working with their security research team to, um, help to create mitigations for that. And then, you know, announcing it so that folks could, could upgrade and could defend themselves against that. And I think that, um, we, we feel very strongly that languages like Rust are the way forward because they, they make it much, much harder to get the sort of, um, buffer flow, buffer overflow, inadvertent type of memory, um, exploits which happen in languages which are not memory safe.
Yep. But that doesn't mean that it's safe by default because there's a bunch of other programming errors. Um, not using the memory safety and different things you can do in your application code, even with Rust, where if you're not paying attention to security, if you're not doing testing, doing fixes, staying up in the latest libraries, making sure you're scanning for potential vulnerabilities in different libraries, you're including, you're just as vulnerable as any other language.
So I think it's great that the REST Foundation is both hiring researchers and also leveraging, um, really top industry experts, like the folks at jfr who are working with them on our security, um, exploits or our, our our rust vulnerabilities. I, um, A quick word on that, I think this is a great example of what we were just talking about, right? Rust was always known as one of the safest, probably most secure languages to be working in going forward, but stuff happens and we learn what, you know, we didn't know them, we know now.
And, and that's just the reason why you gotta be constantly reviewing this stuff. And you do, even no matter how good or secure you think something is, you've gotta do a periodic review. Right.
'cause you do find stuff, things change, we learn more. Um, And I think the other thing which is really exciting is, um, we, we've been a long time member of, um, open S s F Yes. Linux Foundation, which is, uh, Linux Foundation's cross vendor effort to help secure the open source supply chain.
Yep. So that's something that, um, US and um, Amazon, Microsoft, Bunch of companies, I mean pretty much like all the big players in the industry mm-hmm. Are supporting, which is great.
But, um, they also launched a, a new effort, um, called ss s p or Software Security Projects. Um, this is specifically focused on supporting the security tools that security researchers, so ciso, security research teams need in order to help secure the software supply chain. Um, so the big announcement is, is os Zap, the Zac proxy is gonna be moving over very cool to the Linux Foundation.
And, um, I think that this, this is the sort of investment that we all need to see in security research tools so that we can actually become better at defending our ecosystem. So let me make sure we got this right though. The security software project, ss, s p, is it a sub of the open s s F or is it directly just under the Linux Foundation as a project?
Like, sort of like the mainframe project? Yeah, so, so SS s P is a, a top level Linux Foundation. Okay.
Project that's being bootstrapped. I don't, I don't think it's Not, it's part of, There's not the official, official charter and everything's not out, but they're going through the process. Open.
S s F is also a top level That's a foundation in its own Right. For sure. Foundation under foundation.
And, um, I, I think that it's great to see, I, I mean, you can ask, are these efforts competing? Like, like how do they relate to each other? But even when you look at the C N C F, they have a bunch of security projects like in Toto and, and other critical ones mm-hmm.
For the cloud native ecosystem. When you look at the C D F, um, peria, which Jfr supports has joined the Continuous delivery Foundation. Yep.
And as part of that foundation, and I think that we, we all need to look at, regardless of your role, whether you're, you're a developer, you're doing DevOps work, you are doing, um, platform engineering, you're doing security. Like we all have a, a role in securing the software supply chain. Agreed.
And we need to support the, the open source efforts going on for this because the, the best defense against the, the attackers is having more information, more disclosures, more, more Knowledge, more cooperation, cooperation among vendors. Yeah. I, I agree with you a hundred percent, man.
And that's good. Um, we're not gonna be able to get to it today 'cause we're almost outta time, but at some point I'd love to sit down and get your views on do we need to reorganize the Linux Foundation and make it a, I mean, there's a lot of foundations floating out here now for our audience out there. I I bet there are more than a few people who are a little confused.
Um, I'd like to see a sort of realignment of divisions like they do in the N F L periodically or something, you know, and Yeah, this is, this is a longer conversation, but no, I, Not today. I, I think there's a natural ebb and flow mm-hmm. Of, um, at the end of the day, foundations are just a, a vehicle for vendors and open source contributors and folks to come together and to, to collaborate.
Yeah. And so you have this ebb and flow of, of projects being, or foundations being spun up and projects being spun up, but then they merge and they coalesce and they turn into like, larger aggregations and some of them thrive and take off. So the C ncf F is, um, very successful example with its big tent philosophy.
Mm-hmm. Um, I'll, I'll see you at CubeCon, but that's become the, the mic of tech conferences and CubeCon EU is completely sold out. And I think Chicago, they're expecting Chicago to be so as well, you know, what's interesting there is C N C F, I think has 60 something projects under the umbrella now, um, C D F and, and our audience knows this, we do a monthly show.
Lori LaRusso and I called c d pipelines from the C D F and what do they up to about 10 now or something like that. Uh, projects. I recently did an interview, uh, last week with, uh, Fadi, the ED of mm-hmm.
Mm-hmm. C D F and, uh, who was with me, Sasha Lare from, uh, cloudy, who's now on the board. Yeah.
Sasha over at C d F and, um, Jenkins, surprisingly, right. So now Jenkins sort of the grandpa of cd, C I C D, the, the nu, they did a report on usage and numbers. I was blown away, you know, 'cause we're a shiny trinket industry, right?
What's new and shiny? You know, the, the old players and the, um, the CD space. Um, I mean, we're established, we're all working together.
And like, like, so Jenkins, so not a lot of folks know this, but we, we support a lot of the infrastructure that Jenkins runs on for the, when You say what you mean, J frock, J frock, JFR. I just gotta make sure what hat you wearing, because it's hard to see your hat on you, Steven. Um, We collaborate with Gradle mm-hmm.
Which does lot of builds for the Java ecosystem. And now they're, they're spanning across multiple different platforms and ecosystems. Um, Artifactory is the defacto standard for binary package management, which is the core of the Jfr platform.
And like, if you think about it, the, like the, the, the core foundation of what we know today as, as best practices for application deployment and, and what gives you that, um, Pipeline, But what gives you that foundation to build security on. Mm-hmm. It's, it's what was created by yeah.
CloudBees and J Rog, the DevOps people over the last dozen years or so. That that's, that's a hundred percent true. Um, what's the other project that comes outta Netflix and Google?
Uh, Spinnaker. Spinnaker another one, right? Yeah.
I mean, these, these are anchors in every step in every, you know, meaning of the word. Anyway, hey man, it was great catching up with you. I know we got a little bit off security and we're all over the place, but such is the nature of our conversation, Steven.
So, yeah, That's something good. So I guess I will see you next, actually swamp up, coming up. Ah, Yes, yes.
Tell the people about Swamp Up. Yeah. So Swamp Up is the, the Mecca of DevOps conferences.
It's going to be in, um, September in San Jose. Yeah. Um, at the Signia Hotel.
And it's, it's gonna be amazing. So, um, we have Bruce Schneider world fan of security, gu friend. He's very, who's gonna be on stage.
You get to chat with him and, and hear from him about his Perspective. Bruce loves to chat, by the way. Um, we have all of the leaders, John Willis.
John Willis, whose New book is out, Um, Tejas from Netflix. Yes. Um, yeah, the list just goes on, like all the key movers and shakers in the DevOps industry are gonna be there.
And, um, we're, we're selling out quickly, so if you want to be there, that's sign Up. Good place to go. And then after that, we'll see you in Chicago in our, uh, November.
November for Q Con. Yeah. Alright.
Steven Chin, man of many hats. Among them, JFR Linux Foundation, open Ss, SS, F C D F, and a bunch. Russ Foundation here live in Black Hat.
We're gonna take a break, so stay tuned. We'll be here in just a few minutes.





