Integrating Security into DevOps with DeployHub’s Tracy Ragan at OSS Seattle 2024

This is Techron tv. Hey, everybody. Welcome.

We are here at Open Source Summit in Seattle, Washington 2024 at the conference. And of course, we're gonna talking to a lot of fantastic people, you know, game changers in the industry, people who are leaders in open source, applying it, helping others, some projects, whether they're new or I graduated and been around for a while and making all kind of contributions. Speaking of making contributions, we have one of our dear friends and participant on many tech strong events, from tech, strong gang to tech, strong women, to name it.

Tracy Ragan, CEO of Depo. Welcome. Thank you.

It's good to be doing this in person with you. We Don't get to do that. It's, it is.

I know. It's awesome. It's awesome to be seeing you in 3D.

It's amazing. It's fun. Yeah.

There's another dimension here. There is, We Discovered, yes. You know, scientists get this in scientific journal.

We had to tell every people about it. Just everybody knows Mitch is very tall. I'm very tall.

I do get that if, wow, you're a little taller than I thought. You, you are. Zoom does not do well.

Just communicate height. No, it doesn't. Okay.

I would've never thought that Jody was tall. Yep. Yep.

Jody is tall too. So Yeah, it works well for us. Important Things to know.

Important the open source something. Note that down in my journal, you know, in my contact list. Just all, well, it's great to see you.

Um, you know, you contribute in so many areas, not just in Textron tv, of course, in in dub deploy hub, but also in, in open source projects. So one of the topics that we talk a lot about, and there are a big variety. It always kind of evolves around software, developing software, but DevOps, DevOps has evolved so much.

Matter of fact, we're working on a DevOps next report, kinda looking at how it's evolved, but also where it's going. So, DevOps, what happened to DevOps? It's suddenly changed, Isn't it?

Something, it's a interesting topic. You know, DevOps used to be the big thing, right? It was the shiny new object that we all were running to.

Um, and now DevOps doesn't seem to be so shiny anymore. And I find it interesting because we are at a point in our evolution of software in general, where we have so many things occurring. Mm.

We have AI exploding, which has all kinds of new types of objects to ma manage from, you know, large language model data sets to AI agents. And then we have this whole thing that's related to software security and open source security. Mm-Hmm.

Which means we should be focusing heavily in DevOps. We really have to evolve the DevOps pipeline. But nobody's talking about it.

It's kind of the thing that underneath it all, it's what's running the workflows and making it all, all the things happen together and hopefully smoothly, right? But sometimes when you do something right, and DevOps is just out there doing its thing and it's running and nobody's worrying about it, you forget about it. So I feel like that the industry as a whole, we have these, you know, great Jenkins pipelines or whatever tool you might be using, and it's out there doing its job, job.

So we don't have to think about it. But what we're not thinking about is how to bring security into the DevOps pipeline. I'm glad you brought that up, because you know, to your point, we don't get phone calls or emails that says, you know, thanks for not creating any bugs today in that software.

We're writing any vulnerable code. We really appreciate what you're doing down there in engineering. You know, we don't get those calls.

No, we Don't. We get the, what the heck are you doing? Yes.

What happened to that? You know, there's always some big new vulnerability that has some kind of backdoor exposure. Right?

Exactly. So how, how do we stop that and how do we start finding it? It's not gonna, it's not gonna be a, a manual effort manually generating the SBOs, manually looking for vulnerabilities.

It's gonna be something that's done through the, the DevOps pipeline. But we're not, we're not having a conversation about DevOps and security conversations. We're not having much of a conversation about DevOps in these new AI technologies.

So we gotta get back to our roots. I agree. It's the roots is what saved us in the past.

And it's time for the DevOps pipelines. They have to continue to evolve. They have to evolve with digital, you know, transformation with, you know, addressing the security problem.

How that conversation starts. Again, I don't know, maybe we have to rename it. You know, DevOps has been renamed how many times.

We started with configure change management, went to configuration management, went to software development lifecycle, SDLC, to application lifecycle management, to CI/CD and then to DevOps. So what's the new term? And We have kind of have spinoffs if that's the right way of platform engineering and SRE and kind of tacticals everywhere.

So let me run a theory by, okay, more than theory. I'm, I'm taking a position now that, 'cause I, I come all both from a software background and a security background, and one of the revelations in security, uh, which kind of, I wonder why does it take us so long to realize this is that prevention's only part of the answer, right? You can't, you can't invest in just defenses to prevent somebody compromising your network server, whatever assets is, it's just as important to be really good at response.

'cause it's gonna happen. We all know now we are, have will and continue to be compromised at different points and places. And so that response is probably 51% of the answer.

Because if you're not good at that, you know, 'cause the defenses will break down, apply that then to DevOps. I feel like we're making the same mistake of it's shift left. That's good.

Not, not bad. It's a great thing to shift left and make software better from upfront, earlier in the cycle. Um, but you know what, we're now dealing with software supply chain issues.

So it's not just about getting good in base images into the process. Yes, that's one thing, but they're gonna be both inside the build process and the workflows. There'll be vulnerabilities that happen.

We need to be able to very quickly address and patch those as we need or stiffen up our defenses all the way to production, all the way. So we kind of need to think about DevSecOps as the entire security of how we create and build software and the software that we build. That's, that's the way I'm thinking.

We need to redefine the kind of the next generation of what DevSecOps means. Are you gonna tell me I'm full of hooey? No, I agree completely.

We have, I mean, but I don't know if DevSecOps is a cool, cool enough new term. Oh, We need a new one. A new shiny Object.

A new shiny object. Well, this is true. Yes.

We may need a new shiny object to get people re-engaged in that conversation. Okay. And I don't know if that's, I don't know if that's even possible with something that's been running and everybody knows we still haven't gotten away from just CI/CD.

Right? Well, this is true. And sometimes, and those are old terms and people aren't considered thinking about CI/CD and DevOps as being something that needs to be updated.

So we have to, 'cause it's just running. We're we're Working, I've been working on it. Next gen DevOps.

Mm-Hmm. What is that next? What is that term?

Because even in the, in new technology around AI and like MLOps, these are all workflow right? Pieces. Right?

Exactly. All of it's workflow. And we have, we have great DevOps.

We have great orchestration engines out there that we are used to call CI/CD, but that's really what they are. Mm-Hmm. So how do we start evolving that, that that platform?

What is the conversation we should be having? But I think security is the one that will reengage people to think about where do we put it. In fact, I just left Jim Ziland, uh, I was listening to keynotes and Robert Martin from, uh, MITRE was out talking about SBOs.

And then Jim Ziland came in and said, well, now our next step is to make sure that every build has an sbo and we'll get that done next week saying it sarcastically. But that is the issue here at hand. We have to start including SBOs in every single build.

And if you look at like, just cloud, you know, CloudBees publishes like a Jenkins status report basically every year. And last year they, or in December, they reported that they manage 90 million workflows a month. That's Insane.

Are we gonna go update, let's say, you know, that doesn't mean maybe the same workflows executing 90 million times. Just not likely. Yeah.

There's two actually, there's Just two workflows. There's 45 million each month. Yeah.

But there's gonna be a lot. Let's, there, there's, let's say there's 9 million workflows. Even 9 million workflows is too much to try to do manually.

That's, Yeah. So we have to come up with new ways of, of thinking about the problem. We have to think about how we can, um, apply better techniques other than having a plugin for everything.

Mm-Hmm. And touching the workflow at all. Somehow we have to sp you know, spin off a second kind of workflow that says when this has happens, we're gonna go and execute a security workflow to go ahead and do the work to generate the, the SBO m you know, go update or create the SSF scorecard if you can, uh, check to see if a repo is secured.

So there's a ton of work, a ton of work in the security realm that needs to be added into the, the, the DevOps pipeline to make it the DevSecOps To make, so it's actually part of the workflow, not a secondary or ancillary script or process. Sorry, I brought scripts. Yes, yes.

I just watch tech strong gang. Yes. Recent episode, you'll hear all.

Yes. I have a Sorry to bring that up. I have a thing about scripts.

I don't want to get my heart wounded again, but, okay. Well this is the problem, right? We're talking about, oh No, I ran 9 million Scripts.

I Broaden up, I did it to myself. How did I do that? You did.

Because it's an important topic. It's the scripts that has gotten us into this situation. It really is the intelligence, the logic, the crappy scripts hidden.

It hides all the good scripts. So we can't even say let's create a template from Good Scripts. 'cause we don't have clean data.

So Yeah. So we're announcing it here at Open Source Summit that Tracy and I are forming a script recovery program for those who write scripts. Um, yes.

It will be anonymous initially, so you don't have to self-identify who you are, but, Well, I'm not the only one harness who happens to be next to us right now. They're, they did a conference called Unscripted. Unscripted.

Okay. That that mean they spoke extemporaneously or They were talking about the, you know, using a platform to do the work and not Oh, I see. Scripting to Do the work.

Oh, that actually, that's actually a good way to Kind of, you know, I, not to bring up the mainframe, but I'm going to, okay, well then I always talk about this, you know, the very first DevOps tool on the market. What was it? Endeavor?

I Thought you were gonna say JCL or something. No. Endeavor.

Endeavor. It got rid of JC. Oh, it did.

I didn't know that. Okay. It was all, so you learn something every day.

I thought C was still the Thing That was the script. Yeah, it totally was. And we, it said, we don't need to, you know, we, we can have templated, you know, workflows, endeavor stands for Environment for development and Operations.

Oh, wow. And it got rid of JCL for as much as it could. Now the process may have been written in JCL, right?

Mm-Hmm. But you standardized, it's Kinda like an orchestrator of that Process. It was a, it was a, it was a, it's Jenkins on the mainframe.

Oh, okay. That's what Endeavor was. But it had some cool features.

It had something called auto Automated Configuration management. I guess what that did, What did That Track dependencies. Track Dependencies, what a concept.

Exactly. And that happened how many years ago? I mean, this is Endeavor probably was, I think it was initially, uh, sold to ca probably in 1993.

Like in eighties or nineties tool. Yeah. Yeah.

93. So the folks who started it, who came out of, uh, Chicago maybe started it in 83. So they solved all these dependency problems and orchestration and getting rid of JCL back then.

And here we are, how many years later, and we're starting to talk about it because in the, in the mainframe world, you could have added a, a processor for doing security and every single project would've had it. Well, if you think about it in, in your, to defend, not that you need me to defend you off your point on the argument. It is the scripts is just a jumble of all the things that are doing much the same kind of work.

Right? Here's the, here's the workflow, here's the parameters that are driving this, here's the conditions and how to report errors, da yada. There's a whole list of things that all scripts do, right?

When they automate things And everybody writes just In a different language. And Over And over, unfortunately a lot of them are still in Pearl, but, you know. Yes.

And now, you know, new languages are newer, whatever it might be. But, so here's what we need is we need an ai AI engine that will consume all of that and say, great, let's here's how you clean it up and let's push it back out into, you don't need to know what language that's, I'm gonna put this one in JCL and, and this one in what, in Python or whatever language. We need a natural language processor.

So we can say, I need a workflow that calls this, this, this, and this. And a standard script gets generated. If that's what we're gonna rely on scripts to do this work, we have to have another way of doing it.

So they're more standardized. But the problem becomes then you need to go, okay, now I need to do have a, you tell chat GPT to generate one Oh, with all these security tools too. Yeah.

Right? Yeah. So we're still, but you Pick which One to use, but at least we could do that with a pull request.

Right? Pull request. So I'm not saying the scripts are necessarily bad, it's gonna be the way forward, because that's what the, our culture depends upon.

But there has to be a better way to manage those scripts to standardize on what they look like, so that we do have proper data to generate cleaner procedures and to add the, the security tooling into it and to make people excited about it. Again, I actually think that's one of the potentially biggest uses of AI and development is to address the technical debt address, the untouchable code that people won't touch. Cleaning up, you know, layers and layers of things that we've done over the years that sit there and, and, you know, have a lot of issues.

But we don't touch 'em 'cause we're too busy doing other things. Why not? Well, We don't wanna break 'em.

No. Well, exactly. Well that, that's the truth.

You're Break it. That's, you know, a lot of the monolith problem too, right? Oh yeah.

People that wrote that are four or five generations of employees that are gone and nobody knows it. Yeah. And my, it's just a good way to carve out and build microservices with it, by the way.

But, You know, we did, Steve Taylor and I started a company called Open Make Software in 95, and we standardized on a rule-based system, how to generate a make file. And then it, it finally evolved into like ant and Maven scripts and whatnot. Mm-Hmm.

But I used to tell poopy, if you really wanna freak out a developer delete their bill directory. Oh yeah. They'll just kind of go home.

They'll just cry in Tears. Yes, exactly. Like, okay, you didn't, I'm not building that Again, because they didn't have a repeatable process.

Mm-Hmm. And we still have that we're facing many times. We don't have repeatable builds.

And that's what SLSA's all about, right? Where you gotta have a standardized build process, don't do it on your local machine. All of those topics, but you're still having, and those all have to do with the DevOps pipeline.

It's not just open source security, it's also the continuous delivery foundation. Those two, we, we have to have a conversation between our CISO side of the house and our DevOps engineering side of the house. Mm-Hmm.

And oftentimes what I hear is the security conversation is how do you train a developer to write more secure software? It's not the developer's problem, it really should be at the DevOps engineering side. I think it's, the problem with that is, that's a point input point to a systemic problem.

Right. And, and that will help. But you're never gonna solve all problems.

There are a lot of problems doing that. So I have a question for you. Can I throw a SBO question?

Sure. Your way. So lemme the snarky way I would ask this question is how do we make SBOs useful?

It was great to produce a sbo, right? And it's, you know, it's like software decomposition analysis, but sort of a one snapshot at one point in time. And, but then what do we do with it?

What, where, where do you see that going? How do we make that one of the feeds into a vital process that tells us what's happening, what we have to do? And when something You let orillia or deploy hub read it, that's consume the data, aggregate it to the places that you need to make it make sense for you, right?

And then start using it to generate your, your vulnerabilities continuously so that you always see where it's at. You take the data and you track it to where every package is deployed in every endpoint. Mm-Hmm.

So if you have a vulnerability, you know, every single exposure point. So you're saying that wasn't an SBO m that was a softball question. It was a way softball question.

And I thank you for that. You're welcome. But it's true.

You know, uh, uh, um, Vincent Danon and I wrote a blog called, um, sbo. So far so good. So what?

Yeah. So what mm-Hmm. And we need them because that's really where the core of the vulnerabilities, that's where you're gonna find what your packages are.

So you know what your vulnerabilities are. You know, your, and it's the basis for any future AI and DevOps. Mm-Hmm.

Right? Because let's say for example, we have a, we have found a vulnerability now, if you are tracking the, the packages to the components to logical applications to where it's running in, in production Mm-Hmm. You have the ability then to say, okay, I know how to deploy it.

I know what the build looks like. I know what the vulnerability is. Let's go let AI go find the corrected package, bring, bring it back into our, our supply chain and fix it for us.

Mm-Hmm. We don't have to spin according to jfr 227 days addressing a vulnerability. Wow.

That's way too long that we are so behind the eight ball on this one. So the SBO m data is critical, but it has to be consumed, and you have to make it, you have to make it reasonable. You have to put it into a context.

Otherwise it's just a text file in a build directory. It's Almost like it is such a largely fantastical problem. You can't put it into your mind of all the interconnections, interdependencies of all these elements of software.

Right. So maybe you create an AI model or a graph model or something of all the interconnectedness. So you can say, this has been identified as, I think this is what you're saying, this is the problem, right?

This is the thing we need to be patched. Well, what are all the interdependencies? Where are they in the flow of our workflows?

All the way from concept, I wrote some code this minute ago too. It's in production in these 25 different places. And what does it take to update those, right?

Do you update everything in place or no, insert here. And that's your flow to get that out to these versus this endpoint here to get that put to there. It seems like that's, is that what you're describing?

Am I just repeating what You said? No, and I'm not making up anything that's not been done before. If I go back to our main brain solution, It all comes back to the main Automated configuration management.

They track dependencies for everything. And if you needed to update one, it knew everybody who consumed it and it updated everybody. That is what automated configuration management was.

Well, that's a term I haven't heard in a while. It, but it's the same thing we're what we're, what we're talking about in that analogy I just had is basically what the main framers figured out 30 years ago or however long ago. So we need a automated DevSecOps management or something term to make it sexy Again.

Yeah. We need something like that. Yeah.

Great. What's, what's new at deploy up? What kind of things are fun things are you doing today?

I know you've probably talked about some of it Already. We, uh, we really are starting to look at how we can use the data, um, apply what proper AI methodology needs to apply to be put against it to start auto remediating. We really wanna build a rapid response to vulnerabilities Back to my rapid response, right?

Yeah. Yeah. A rapid response.

And, you know, I think that there is a, there's something to be said about chaos engineering and all of this Mm-Hmm. It's, it's not about trying to find the, the root cause analysis. It's how do you respond to it as quickly as possible?

And that's course sort of where our head is. How do you rapidly respond to it? Get it fixed as quickly as possible, and then take your time and back it out and say, you know, what happened?

Was this the correct fix? At least we, we shut the door, locked it. Mm-Hmm.

Now let's make sure that we didn't send somebody out to the barn. Whoops. Like every horror movie, right?

Uhhuh the first person who dies, don't go in the barn. Don't go out in the barn. We went in the barn, so did we shut somebody out in the barn?

We can figure all that out as soon as possible, but let's get the door shut and locked as soon as possible. We'll find that door. That evil villain might still be out there.

I've been working on chaos engineering, but I've only got the chaos part working so far. Still need to go back and work more on the engineering part element of it. Um, so what, what do you think is the topic we should be talking about that we aren't?

So you're talking about kind of, let's bring DevOps back to the surface. Is there a part of that you think? And, and here's the most glaring, The thing we should be talking about in the DevOps space, in my opinion, and this, you know, this is just where my head's been and focused on, is that in order for us to apply AI to the DevOps puzzle, we have to have a source of data.

Mm-Hmm Mm-Hmm. And we don't have that. We do not have that.

Nowhere is It that's spread everywhere or that we're not, It's fragmented everywhere. It's fragmented between tools, it's fragmented underneath, um, in scripts. Uh, it's just, it's, and it's fragmented by modern architecture because now you have a lot of different little components that you have workflows for.

And those little components and those little workflows create the logical application. So even if you're an application team and you're being asked to provide an sbo m what are you trying to do? You're like, okay, I gotta pull the SBO m into an Excel spreadsheet for all of these microservices that my application consumes.

Mm-Hmm. So it's the data, it's a central place for the data. That's what we should be talking about.

How do we solve that? How do we make it better? Mm-Hmm.

And what does it, what does that tool look like? And that is what we're trying to do with ATUs. It's what we're trying to do with, um, deploy Hub orus is the open source project.

Mm-Hmm. That is incubating at the CD foundation. And we could do that just for open source if we did that just for open source packages, that alone.

So people could share the, the, the data that comes from that. Mainly the, the SBO M information. It's gonna simplify quite a bit.

And we can then start doing trend analysis. Mm-Hmm. Historical trends is the basis for doing any kind of threat modeling.

Right. And then our threat, you know, you sit in a room and you talk about what your threats could be. Well, they're just assumptions.

We need the data and the historical trends to determine if our threat models are even accurate. Simulate game it to see if this is really gonna impact us. You know, it's interesting you say this because I think one of the struggles about DevSecOps is it's sort of the, there's a pony in there somewhere kind of strategy, which is here's all the data.

It's not collected in any way that you can structurally bring it all back together and make, you know, fully make sense of it. It's just we have a lot of data exhaust coming off all the tools and processes that you're using. Do you think, do you think we need a more structured data model about how we identify and put that data together?

Or is it No, AI is gonna solve all that and we'll just give it a, you know, machine learning pattern and we'll go figure out what this means. We need to understand the metrics. Okay.

Say more Without the metrics, we don't know what we should be gathering. Mm-Hmm. So we have to really understand the metadata and the metrics.

That's really, that's why I think it's such an important conversation. The open SSF does have a working group called the Metrics of Metadata. Mm.

They're working on a little project called, um, security Insights, which is basically a YAMA file that you check into your repo. It has some basic information, but we need it. That's a good start.

But we need, because we orus could bring that information in too, right? Mm-Hmm. Would set us up in a really kind of nice way if we had had that data in the repo.

But for example, right now it's kind of hard to even go to find a get commit, get commits are not in the, um, in an, it is not required or the GI ID is not required in a, in a SBO M Mm. So how do we find that if we have the sbo, how do we get back to, how do you know What you pulled, right? We're, what was that?

Yeah. Hold that list of stuff. So the data, we haven't figured that out yet.

So the data, the metrics and the metadata are really, really critical right now. It's where we should have be having that conversation about what data we should be managing in the DevOps process. And we have to think about it in DevOps terms, not just in security terms, because we can gather metrics and meta metadata at the security level, but if we don't map it to where it's production, where it's running in our production environments and what release it was Mm-Hmm.

If it's just data, it doesn't make sense. Because what you need to find out from the data is how it's being your being impacted, your end points. So, And you need some traceability to be able to say this event or this activity or this data point that represents, that may not directly tie to that metric, but here's the three steps that get you to that place and how this set you up to failure not to meet that metric.

Right, Exactly. And then once you have that connection, does it, it, did that package matter? Mm.

Maybe we put it in the, maybe it was in our build, but we never use it. Mm-Hmm. It's never called, even though it's still a risk, it's not a higher risk if it's actually being used.

Yeah. So there's a lot of data that's out there that we're not collecting we could be using to make this problem go away. I just want you to know with all the talk about GPTs and LLMs and natural language, all of that, it's all good.

It's very fasting, but you were, you are still my go-to knowledge base for anything I wanna know. I ask you. So it'll never compare.

So Thank You. You're very well, I mean that, seriously, thank you. You're, you, uh, are a great person and a great, uh, contributor to our community.

Thanks. So It's a pleasure. Um, do you have a talk coming up?

I have a talk at, we have, I have a talk at four 20. I'm gonna talk about data. Mm-Hmm.

Shocking. Yeah. Um, and then Steve Taylor's gonna come up right after me and we're gonna talk about something called the Hero Project.

The Hero Project is that a new mini series on Netflix? It's gonna end up being a little miniseries in terms of a open source contributor effort. Right.

That we gotta get done that's gonna address Jim Lin's, uh, question today is how do we include SBOs in every single build? We do have to solve that, and we're gonna work on the Hero project to get that done. Nice tie.

Okay. Well, good luck. Thank you.

Break a leg on the, on the talks. Okay. It's the, in the theater community, Tracy Ragan.

Yes. You can find her on Textron TV and Textron Gang and all those great places. And Textron Women TV and Techron Women, which is by the way, that show has taken off.

We love it. It's so much fun. It is A great show.

And the things you talk about are, I can't even attempt to describe it. It's so good. com.

Right. io. Sounds like another good Greek mythology movie.

We should be making Abraham Orus. Abraham Orillia. He created the first world atlas.

Oh really? Yes. And he did it in an open source way.

He got all these other photographers to contribute their maps, and then he laid 'em on top of each other basically, and said, what's the most, the, the most common denominator of what the, the, the world could look like. Well, how did he let someone else name it an atlas? It should be an Orillia.

It should be an Orillia, right? Yeah. So it's an Abraham.

So if you wanna be in with the cool people, it's really TIUs. It's an Orillia. Alright, Tracy, Thank you.

We'll be back. Uh, we have many, many great interviews. I, I don't know if it's this good, but many really good interviews coming up here at Open Source Summit, uh, 2024 in Seattle.

I.