Mike Rothman and Mitch Ashley, Techstrong Research | DevOps Experience 2022

Hi everybody. This is Mike Rothman general manager Tech strong research. She's strategy officer of tech strong group.

Welcome to the devops experience, right? We're here for the experience. And I think we're midday.

So midday, you've got about you know, maybe halfway there we happen to be residing in the devops to the edge panel and we are going to talk a little bit about Is devops or distributed devops going to screw up our security but that's really scratching the iceberg that was kind of like some clickbait in order to do that with me today is my partner in crime and text wrong research the one the only Mitch Ashley Mitch. How are you? Good good to be here and always great working with you.

You know, it's great to talk about topics like this that were the intersection of software and cloud and security come together because it's sort of like watching one of those IMAX movies. It's just really big and it's moving fast and you're just trying to take it all in right? Okay.

Well right and not get run over right? I mean, I think that's right. That's one of the things is, you know, just you don't want to become roadkill in this in this Evolution.

So, you know, we're gonna do a little bit we don't have slides or anything like that. What we really want to do is, you know, kind of talk one about, you know, kind of this distributed devops concept and and really what that means to us and and And really the implications for how we are starting to distribute, you know code deployment right execution to the edge, right and really getting that closer to the end user the folks that are actually consuming either the applications or the data or the resources that are there how that is gonna really complicate. What we do from a security standpoint, right?

Because I mean listen it's hard enough when we were in on-prem land. Right and all of our developers were sitting in a room and we could flog them and say, you know, you got to do this now, we've got one developers all over the place, right that's kind of remote and hybrid work and you know, we've got code being deployed in so many different places and so many different form factors. I mean, it's really taking what was a complicated environment and made it, you know, just exponentially more challenging to to stay on top of that.

Well, what did I miss Mitch give me a sense of where you think, you know distributed devops is how starts to play into some of these things and then we'll kind of transition into you know, what's the impact on security? Well, first of all, I love when you open about the old days of there's a developers all in one room. You're such security guy might.

Yeah, but old guy. That's the problem. I'm an old security guy, right?

That's great. Well, you know, it's I think it's part of this whole Evolution that this tract that we are on is Security Professionals in soccer shift left and all that kind of thing. We all know at least have been talking a lot about this I think takes us into the new realm Beyond just Cloud native and microservices and how do we secure those environments but to an environment where devops is creating or using devops processes?

We're creating software that can run anywhere on iot devices mobile devices Edge Computing Edge clouds. So, you know in the central Cloud if you will back at the data center, you know home private center, but the point being is we're not only building smaller smaller pieces and releasing those more frequently. We're doing it in different environments on different platforms a new new technologies coming not just Cloud native.

I think that's How do we as security people get our heads around that when we're still gonna working on how do we secure kubernetes? Right if right or I mean even older school, right? How do we get folks to systematically scan their code before they committed right or or in waterfall land before it gets shipped, you know to customers which was never an assumption a good assumption that that would happen.

So it's not still very simple very good some changes, but you know this year to year six to seven, you know, continue to move forward with that. So it's not just you know, how we're gonna secure kubernetes. I mean that's clearly a thing right?

And and now that all this is typically deployed in code, you know, we do have I think maybe better tools and and different options in order to you know, kind of get on top of that but I think there's a bigger emotion in terms of how do we get application security to be more prevalent? Within These devops motions and that's one of the things that we are going to discuss a lot next year, right? That's one of our Trends, you know both on the on the devops side in the digital transformation side, which is really culture.

Right? How do we get everybody to you know kind of work together and not just pay lip service to it. We hear a lot of lip service lots of people flop in their lips about probably we work together and deaf secops, you know, and then when you scratch underneath the service, it's still growing stuff over the transom, right?

It's still maybe we'll fix that, you know security defect but it'll probably get deep prioritized with with other feature requests that that we need and we're doing the Sprint meetings and those kind of things. Yeah, that's faster. You know, that's right.

That's right. That's right and who's you know, ultimately empowered to make some specific change. So you have something that's like, I mean, we're existential threat to the organization.

There are a lot of companies that don't have to find processes. Enable our power security to make those specific changes. So it's got to go into the flow with everything else.

And I mean you could be dead by that right? I mean, you know didn't clear the security can or can't you know, when can they stop something? Right?

I mean it used to be the old days. We stopped everything because I had to go through them first, but we're out there anymore at least not in most cases, so That you know, those responsibilities are kind of blurring too, right you talk about the cultural part of it of working together. There's also the maybe it isn't one person's responsibility.

You know, it's just not science. It's not sign-offs anymore, but it's so I'll kind of tilt it windmills a little bit here. But if if you're working together to build it into the process of how you create software building and security not just scanning code scanning, but you know API Security application security software supply chain Security in the tool change itself, you know, take them off of all the things that we want to address and security land before software goes out the door and after, And if you're kind of have that.

That flow that that workflow. Patterned out and saying what are we working on either ourselves or with others with the dev teams? To help address that because that's what you're I think that's what you're going to be involving not just the architecture of the software Cloud native versus something else.

It's that well tool chain and that whole process that's why I keep waiting my arms around from left or right but you can feel but really that's that's where it happens. It's kind of like saying, you know, we should make safer cars, but we don't know how that manufacturing line works. You know, we just should make safer cards so we can design it in but is it really get shipped as a safer car?

Well, it's Gotta Be Done Right Where it's not see. No, that's right and and let's kind of because you know, we kind of started at the tool chain, right? There's also emotion that again as we're starting to distribute devops as we have to provision for this code to run in much smaller form factor compute environments much different types of you know, controlled environments and you talk about age clouds and and some other, you know, kind of mechanisms to do that, you know architect.

Sure is still a thing right and and having the security team and again not block stuff. But at least weigh in right having, you know kind of the center of excellence, I provide a set of design patterns for some of these distributed devops applications. This is how you do networking.

Right? You know, you're running a workload kind of out there on K3, you know k3s on on that, you know smaller form factor. How does that get data?

Right is that data? That's local there in the cloud is that data that has to go back to you know through some type of Transit infrastructure. What is that look like and if it developer does that I'll tell you what, they'll do the path of least resistance, which is something over Port 80 addressing an IP address sitting in your data center.

And then they're gonna stamp their feet in terms of why the you know, the the poor is not open, you know on the perimeter firewall and be like, well my application is broken about I can't do that, right so the fantasy Software people because it takes eight weeks to get a port open on that farmer. I'm not but again, you know, I would opening upward on your egress. Your Ingress environment may not be the best idea on that front.

But all the same, I mean, right we have to as if you want to be a considered part of that team, right if you want to be in that discussion, we've got to come forward and we being security a broader security concept, right? You know, we've got to come forward with components code Snippets design patterns ideas for how those common functions can happen securely and reliably in whatever compute environment we're dealing with and until we get there. We really can't b**** a lot about the decisions that the developers make to make these things work.

You know, they did that right? Well stupid, you know, it's just like if you're not telling them what to do, how do you expect them to do it? And so we've got a really think about that at the architectural level and then you get into obviously the tool pieces of that right?

How is it that we're going to scan the environment right? How is it that we're going to you know, be in a situation where you know, we can integrate that scanning technology multiple times through the whole build process, right? How is it that we operationalize the monitoring of the environment to know if we've got, you know some issue and then going back into it.

So again, it's it's not an either or thing. It's all of the above. I don't think that you know all hey we nailed the architecture thing.

So, you know, I'm gonna go play Tetris. I don't think that's the answer either right? I think that you know application security is a multi-faceted aspect that has to be integrated into the architecture process.

It has to be integrated. To you know, kind of the integration process rights, you know kind of static testing and certainly Dynamic testing. It has to be built into the deployment process.

Right and it has to be built into the operational motions and monitoring environments. And if you're missing any of those, you've got a weak link, right and and that's where you can get hurt. Because I think that's a good list.

That's a good, you know place to build from right? I think another thing that's that's helpful to understand is in software. There are some kind of design engineering and design not just pattern the sort of principles that things that evolve about how we create things like in the world get-ops, right?

It's everything is declarative. We don't that's everything comes out of the sort of code repository system everything right and an environment can be built, you know, simply with a push of a button set of scripts Etc. And if it needs to be changed it gets changed in its source.

So the same kind of things are happening when you get into the service environment or you get into service applications, you get into stateless applications and understanding what these things mean because that determines the security a lot. So for example state was applications have some advantages from a security standpoint because there is no thing running in memory that has in memory all of the stuff. It just did for the 500 transactions that just got completed or API calls right before and it's going to do the 500 and first and so somebody just needs to pop into memory or get it unprotected in some way.

So there are those things that I think really Help and understanding some of the software architectures not meaning you have to be an expert but understanding it well enough and design patterns a great way to get into it understanding, you know, microservices and some of the cloud native stuff. And Wasim webassembly is is something new on the Forefront coming. You know, I think many folks think that's the next thing the next sort of architecture Beyond Cloud native and not replace it.

But so that train runs right that software training architecture Concepts. It just just like security has its own things that it heads forward in time and Innovation creates. This future software is doing that too.

So you can't have to bring those two mindsets together to be able to get a cohesive, you know left and right track heading down the same direction, right? right brain and left brain, right and and we got to figure a way to reconcile, you know, the, you know kind of though or the union young right depending on what your philosophy is, you know, there are a lot of different the wire when it goes like this in your In front of spread yourself out. Hopefully they come together at some point, right?

That's the goal. Anyway, so I mean that that's a great point in in that we do have to be able to appeal to both sides of that environment and and you know, we talk about the overused terminology of devsecops, right and you know without a lot of substantiation or meaning or really tactile type of understanding of what that means. I always kind of it was easier for me to understand that because I broke it up in a two piece right Dev sack.

All right, what do we have to do to be, you know really make sure that we are developing and that's that architecture right integration deployment on that making sure that we to the point we get this stuff deployed into a production environment, right? What are we doing to make sure that that's secure and then we've got sick ops piece of it, right how we monitoring this who's empower? Make changes.

What kind of guardrails are we going to have in place to ensure that either a developer or some type of compromise application is not accessing resources that it shouldn't it's not you know kind of putting in place some environment that you know or some situation that's going to you know, put critical data at risk. So, you know, making sure that we're paying attention on both of those sides is you know, absolutely critical. So to me devsecops is not a thing right to have sex 100% right Sac Ops, absolutely and and you know, the it's the impetus on the security professional to appeal to both of those constituencies because the reality is the devs are gonna Dev right and the Ops are gonna hop and and we've got to make sure that that happens in a way that doesn't, you know create undo risk to the organization.

Don't ever thought experiment if you want to go with this. Yeah, let's plan it. I'd be happy to do it from the software perspective.

So I'll ask it this way. If you're a security professional out there in the world practicing your your profession if you will. And you're you're like, what do I need to do to figure this out?

What do I need to do to get myself where I can have the devops devsecops dev SEC part of that conversation. I can start to think about some of these software architecture whatever that exactly means and I can hopefully help Engineers Architects stock word designers developers understand maybe some security principles that apply in their environment. What would what would you do if you're like sitting here, how do I do this?

What would I do? You know, that's a great segue into another topic that I wanted to bring up. Right and and that's the idea of the security Champion.

Right? And that's and that's become, you know kind of a topic and it's a thing and it's really not a well understood thing and it's so just let's clear that up. Right?

So the security Champion is somebody that's actually a developer, right? They are not a security person that I'm gonna be very clear about that. They are not a security person.

They're not a business information security officer. They're not associated with the cloud Center of Excellence. They're not any of those things.

They are developed and that developer has been trained in some of the Black Arts of security and I say that's what tongue in cheek, right? But you know, they've been given some training about how to integrate, you know, kind of the the Motions into the development process, right? They understand the tooling and the tool chain that the organization has.

Picked to fit into the pipeline, they understand the architectural constructs and the design patterns that are offered from the central team and they're there to work with the other developers in order to facilitate their understanding and Adoption of you know, secure coding practices. So yeah, it's a complicated type environment and no security person is gonna airlift in and have any credibility with the developers right? It's just somebody else to tell them they're doing things that are screwed up and not optimal and I'm gonna ignore them because you know what?

I'm incentive to ship code with the features that we need. Nobody gives a crap whether it's secure code until they do to be clear. So but in terms of my day-to-day operational environment, so the idea of this security Champions is to break down some of those walls to alleviate be able to And some of those silos and really help the developers help themselves.

We got to equip them. Right the tool chains got to be there. It's got to work, right the design patterns have to be The adopt if you force folks to do unnatural acts, they're not going to do it, right they're not going to do in a natural act in order to keep some, you know, kind of auditor that shows up, you know Happy from that standpoint.

So so yeah, it's a really complicated type of environment but I think to me the only way to start to bridge those gaps is to you know, start to build a ground swell, right a local, you know kind of capability right in the group that has some aspect of knowledge in terms of how security works and and again when we talk about, you know, kind of successful devops and you know kind of secure devops types of environments to one I think almost all of the ones I've run into, you know do have an active and and very effective security Champions program. I think it's a that's a great idea a great spot on you know, it's not it's not new thing, but it's definitely I think become more front and center how important that is. Because you're never going to make a software engineer and a security expert probably you're never gonna make out a security engineer software expert right either.

There's just not not what neat is a solution where I was going with my question is well, it's not all up to you right? You can't do it this by yourself. And there are many developers standpoint or from the security reminder.

If you're not security persons, like you know, how do I get a handle around this or vice versa, right? So there are people in the world who do get security and software right kind of they're sort of left and right bring connected and you know, that doesn't happen overnight. They have a passion and have an interest and one of the ways I've seen people to help grow those kind of folks that kind of Talent on your team is there's always a lot of interest in security.

It's consider. It's still considered a very hot feel, you know, I employment hi compensation, but maybe I don't want to be a security professional. I'm already a software engineer or security professional that's got a real meant for software.

You can move into that kind of role and grow that where you want to be and I think the more we can grow those kind of professionals. The other thing is thinking about to Mike is sort of like if you want to learn about a culture. Well, then go be in that culture.

Yes surround yourself with that. Whatever that culture is feeling the blank. And I noticed that a lot of security organizations partly out of need but also because of this reason are hiring more soccer professionals.

Maybe they're not hardcore developer developers, but they're folks that are I'm at the SRE field or sysadmins that also do scripting people who on a ride or script software as part of their security team because they need them in the sock, right? They need them to manage all the stuff coming out the observerability system all that kind of stuff. So bringing that into your team helps you understand.

What that is how how folks like that? I think operate in what's important how you can help. Lend a mesh together, right?

Yeah, cross-pollination is is one of those things that as we again kind of come into our world right devops Cloud native infrastructure security kind of wrapped around there with digital transformation is kind of the Catalyst to Kickstart. A lot of those initiatives in in our world cross training across pollination is critical right because you know devops, you know, kind of code and and programs and applications that are developed using a devops motion tend to be platformed on a cloud native and or in a cloud native environment tend to require security that is you know, agile in nature and really built into a variety of those different, you know, underlying structures and and you know, if you've got somebody who just understand security. Well that's limited because that's not how it works anymore.

Right? And if you've got devops or you know infrastructure platform Engineers SRE folks who don't get a Of you know kind of where protection fits into this environment, right? They just don't have the ability to really kind of see the entirety, you know of the situation and and again, it's one of those things where I mean, you know, there's the whole wow, you specialize and you know, jack of all trades master of none or anything like that.

But the reality is you have to be conversational regardless of where you sit right Ops Deb's SEC, you know platform SRE, you know, you've got to understand at least at a conversational level a lot of the underlying technologies that are there and if you can't do that again, I would pause it that you're gonna have a really hard time, you know thriving in this environment that we're moving into I agree. I agree especially you think about now pushing it out to the edge, right and you're talking about multiple environments, maybe some new maybe some we know. Well, yeah and that environment Not like again.

It's not like our data center. Just like, you know being the cloud it's evolving. It changes constantly, so It's it's a fluid process.

It's not set it and forget it and leave the development team alone for a while. You know, it's it's kind of a contact sport, right? You're you're running out.

You're on the same team on the same side of the ball, right? Doing it against our behaving that way. Yeah, and that's in a distant.

They're the enemy they don't like to write secure code or they're the enemy that the land of no or whatever, you know that things we've made up about each other that we can get over but you know, we actually have a common goal, right we want to ship code and we'll ship code to secure. That's right. And the last couple of minutes we have left Mitch.

I do want to hit something that's a little bit more tactical but no less important. If anything I could make the case that it's more important. Right?

And that's something that a lot of security folks don't pay attention to because they don't think they are empowered to do anything about it. Right? And that's the underlying security of the pipeline from the time.

We want to commit, you know into the repo kind of roll it through whatever, you know, integration platform you're dealing with down to deployment, you know, kind of within and on, you know, kind of the platforms that we're dealing with and and ultimately managing that whole process and automating all of the you know aspects of making that work. I want to be very clear here that is Your company's intellectual property, right the source code that drives all of this stuff, right which libraries you use which you know kind of components. Are you integrating in which third-party services are you calling right via apis, which micro services are you triggering?

You know as part of you know, this application that is your secret sauce right in all of that runs through the pipeline. So make sure and again, you know, and and obviously if you're dealing with something that's open source, like Jenkins or something like that. You've gotta a set of plugins that you have to deal with and you know kind of that's a DIY, you know type of thing.

Obviously if you're in a more managed environment like a good lab or you know, kind of a GitHub type of environment, you know, you're gonna have you again more kind of services type knobs to deal with but but all the same right you as the security professional regardless of where you sit around there have got to make sure Just like you don't want any device sitting internet accessible or Internet facing to have specific vulnerabilities on it. You cannot write cannot fall asleep at the switch right making sure that you know kind of this pipeline is not very specifically hardened very specifically protected. So I'll get off the soapbox now, but I can't tell you how many environments we walk into where they just have not focused on that at all that somebody else's problem.

Right? The devops folks do that. And then you talk to the devops folks and they're like I'd be happy to do that.

But nobody told me what I should be doing, right and That's problematic folks. That's very very problematic. So let me let me add the other side of the queen too.

And that is it is about writing secure code and creating social security software, but it isn't just about that. So perfect example recently Dropbox, right? The developers were fell victim to fishing attack.

Okay, we're developers supposed to be smart and Technical and not Falls to that stuff we do and so that gained access into a whole bunch of source code that was you know online and didn't get to a core product but got to have this stuff whatever. It doesn't matter, right? It's all part of the part of the system of software that we're delivering.

It's also about third party software open source software and if I was again, I'm actually getting involved in the open source security Foundation doing helping out and doing my little bit contributions with them, but that's a great place. I think you're interested in. Securing software and securing open source soccer because a lot of things happening about how to develop secure software and how to set up teams that are building secure software that I think you can take back whether you're directly involved with ossf or not.

You're really good stuff there that I would highly recommend checking out. What's dot org. Yeah part of Linux foundation.

So be thinking about the environment you're creating this in and this year skills. Not just the code you're generating and writing secure codes. That's right.

So the net things out matches we kind of wrap up. Um, I don't think distributed devops is gonna screw up security. I really don't I think that, you know, like most other things we have to as Security Professionals be diligent about, you know, kind of working with our peers and our partners and our colleagues and a variety of these different environments.

We have to add value. So design patterns infrastructures code temp. Architectural guides for how you do common ideas, especially as we're moving towards smaller form factor edgy type, you know applications where connectivity is going to become critical in terms of getting access to you know, some of the real-time information that that's there.

So the challenge is great is we've known that right but it's not impossible. It is not insurmountable. It is not a matter of and by the way, we framed the, you know, the the title of this session right is distributed devops gonna screw up Security in kind of a little bit of a disingenuous way, right?

We're not going to blame why we're not gonna sit here and point the figure finger at the devops folks or the SRE folks the platform engineering folks and say it's your fall why this stuff isn't happening. We're not playing that game. Right?

What we have to do. The impetus is on the security folks again. Do the devsack piece of it do the sex off the piece of it really work with our partners add value and I think that we can get towards a much.

more secure environment by Design and by practice over time To use it as an aligning as a rallying point something we do together. Not as a threat. That's right.

That's right. And I think you have much better outcome for your point. Agree, wholeheartedly so hopefully you've enjoyed, you know, our little devops experience experience.

Would that be devops experience squared Mitch on I don't know but that's in there somewhere. I don't know. It's a new man the cloud and Beyond I think one of the the most entertaining things, you know, I've done since I've joint Tech strong is C Allen in a in the Buzz Lightyear costume and and Mitch in the Woody costume that would and if you guys didn't know I was the alien so I think I had one Speaking line and I was in this big thing.

So nobody could tell it was me and that's just how I like it, you know, so enjoy the rest of the conference today virtual ask us questions, right? com in order to get to us and set up, you know, this briefings. We're happy to chat with different folks.

You want to chat if you're an end user we're happy to do. That and the other thing will point out and pimp out a little bit is predict. Right?

So we've got predict coming, you know, January 12th. We are going to be talking about all of these things that we highlighted today. We're going to structure it within the construct of Trends.

So we have five trends for each one of the areas that we deal with will do Tech. You got a track right? So we have a lot of really cool stuff happening right when we kind of kick off the year in 2023.

So Mitchell, thank you for being you know my partner in crime and and indulging me is we do a lot of these talks together and for everybody else again, enjoy the rest of the rest of the devops experience. That's all enjoy the journey together. I'm good job experience.