DevSecOps-ing the Infrastructure | DevOps Connect: DevSecOps 2023

I reached out to one of my best friends in the security space, Mike Rothman. I don't know how many of you are familiar with Mike, and, but he's run Securosis for 11 or 12 years, and he has about 25 years in security, uh, research. And he is our GM at Techstrong Research.

We're doing a lot of research. You're gonna see a lot more of it being published, but I want to introduce you to Mike and not embarrass him, but he's, it's good to have him on the, on the team here. This is his first time presenting at DevSecOps, so let's give him a nice round.

And Mike Rothman. All right. I start most of my pitches with this statement, right?

It is not necessary to change. Survival is not mandatory by John Willis', personal hero, and one of mine as well. Uh, Dr.

Deming, uh, who, uh, again, just a, a fantastic researcher, uh, and had a lot to do with, uh, really overhauling how the Japanese made cars back in the day. And, and I find this is a very important framing, because if anything, we've seen how much dramatic change has happened, right? And it's not just the fact that we're actually all in a room together.

I mean, can we appreciate the fact that we're actually in a room together after, you know, many years of not being in a room together, right? And the fact that you're like here and not in a little zoom box is, uh, you know, nothing short of amazing to me, right? So, uh, I'm grateful and, and, and thankful for that.

But think about like 10 years ago, cloud was this kind of thing that we sort of was like, yeah, it's cool. I've been to maybe reinvent. And, you know, Azure was hardly even a thing at that point.

GCP was like, oh, s**t, we're behind the market, so we gotta figure out a different way to do things. Uh, right? So, I mean, the, the amount of progress we've made in 10 years is nothing short of astounding, right?

So we have to get comfortable with the fact that things are always going to change. And as much as it's comforting to walk into the data center and see a bunch of blinky lights, and, and believe me, I'm old enough to remember thick net, any my thick net buddies out there, couple y you know, crawling around the floor going, man, if you bring your dog to work again, who's gonna go in there and, and, and, you know, knock our network down? Y you know, not gonna happen.

Uh, right? So, so, you know, it's, it's amazing the fact that I can go click, click, click, and I've actually built not just a network, right, but an entire application stack within minutes, right? So this is, again, astounding change in terms of what we are.

So we have to be ready for that. The other thing I want to do, especially given kind of the backdrop of, um, infrastructure as code and, and really DevSecOps and the infrastructure, is this quote from Mark Anderson, right? Software is eating the world in all sectors.

In the future, every company will be a software company, right? And again, this is the framing of how we have to think about where stuff is going. It is all about software.

It's all about, you know, kind of development and really kind of security. And then operations becomes this continuum that we have to think about, right? Software, it's cloud, network, containers, workloads, APIs, identity policies, everything is code.

And if it's code, it means we should be using a lot of the DevOps principles we've been working on and perfecting over the last decade, right? So, again, important framing from that standpoint. Oh, and I do memes, right?

Because mostly I'm, I'm an a, d d kid like everybody else, and, and I need to keep myself occupied. So I sit on the meme generator and try to find interesting things that, again, make me laugh. If you enjoy it, that's a, you know, and that positive as well.

But, uh, it's all about making me laugh, right? So, uh, and, and, you know, I guess we should probably have a moment of silence for Gary Coleman. All right.

Anybody remember Gary Coleman? Yeah, a couple. Thank.

Great. You know, obviously the guy died in like 2010, uh, but I still, you know, whenever I use this meme, I do a moment of silence for, for Gary. Um, you know, so infrastructure is code.

What is it? Right? You know, kind of, uh, define using code and, and version control, but here are the things, right?

And I bolded, you know, kind of these things that really help, uh, again, understand the difference that infrastructure as code is gonna make in our environment, right? Consistency, repeatability, scalability of infrastructure deployments. Um, anybody missed the days where when we wanted a new server, we had to fill out a form in triplicate, right?

And then we had to wait for three or four weeks until it like showed up on the shipping dock, and then we had to find some person to pull it off of the shipping dock and put it into the rack, and then we had to plug into the network and load up all the software. A anybody missed that? No.

No. Right? Because what we can do now is do things again, far more consistently, far more reliably, and a lot faster, right?

And this is just using traditional types of, you know, kind of virtual capabilities. When we start thinking about enumerating all of these different capabilities within code, that gives us a lot more flexibility on that front. Increases velocity, deployability, um, in multiple environments or multiple clouds.

Anybody doing multiple clouds somewhere in your environment, the rest of you are asleep. Yeah. A liars.

Do we have some liars out there too? Yeah. No.

Um, cuz everybody is doing some kind of multi-cloud, whether you know it or not, right? So having a base that you can use, that you can leverage in a number of different platform environments, again, helps with leverage, helps with repeatability, helps with consistency of what it is that you're trying to do, right? And reduces human error.

That's the thing we haven't figured out how to get rid of, and that's the humans. Although with chat, G P T, we're on our way, right? So, um, you, you know, again and again, I'm, I'm, I'm another one of these child of the eighties and the Terminator scared the crap outta me.

And I still think we're getting there, right? I still think we're getting there. Chat.

G P T may be the first example of Skynet, but I'll, I'll be optimistic and saying no, no, no, it's great, you know, but what we can do is we can start to leverage some of these capabilities to reduce some of the error that, again, is rife within our infrastructure. So we have data, right? So Alan said, I, I, I run tech strong research.

Um, we have an audience of, you know, 400 to 500,000 folks. Um, couple of million page views, uh, every month, uh, four or five different media properties. So there's tons of data, lots of visitors.

We ask questions pretty frequently. So, uh, we wanted to ask some questions about where the infl, uh, the, the, uh, you know, where folks were from an implementation standpoint of infrastructure as code. So do you currently use infrastructure as code in your organization in production?

Yes. 54% for production workloads, over 54% are using infrastructure as code for production workloads. So this is not a new thing anymore, right?

This is spoke, people are comfortable with it. Are we doing it great? Of course not, right?

Is there a lot of area to improve? Absolutely. But folks are using it.

And even if it's a very small, not so pertinent application that is running in production, we're getting our feet wet. And that's very, very important, uh, right? Yes.

But only in non-prod is another 7%. So over 60% of folks are using infrastructures, CLO is code. Uh, and another 19% say they will within the next year, right?

So this is not, I, I don't know, you folks are fans of Malcolm Gladwell, right? Everybody's always looking for the tipping point. We're pretty much there with infrastructure as code.

So now we gotta figure out, hey, is it just setting up a couple of cloud formations for each one of the things that we do? Is it just playing around with open source terraform and trying to figure out, you know, hey, we can build these things into, you know, a couple of scripts and do it that way. Or can we make this into a machine like we do with the rest of our code environment?

And that is what we need to try. Why infrastructure is code. So what's the primary reason you would consider using infrastructure as code 39% improved?

Consistency and repeatability, right? Humans screw things up even when we give them playbooks, right? Even when we give them checklists, even when we tell them exactly what they're supposed to do.

Well, you know, I had to check TikTok right before I set up the machine, or I gotta call right in the middle of a, you know, very complicated workflow that that's running, uh, there. Or, you know, something happened and I had to start dealing with an incident, you know, and I forgot to get back to it for another, another three or four days, right? Not consistent, not repeatable.

If we could press a button and a whole bunch of stuff happens, that's kind of what we're trying to achieve with this couple of other, you know, interesting data points here. Faster deployment of infrastructure, 23%, right? Easier management of infrastructure changes, 22%.

Um, what was not focused on in this were two other things. Um, 13% not considering it. But what's interesting, right?

4% enhanced collaboration between teams. We hate them. They hate us, they hate us, we hate them.

And that's the way it'll always be, right? Well, yes and no because the fact is we don't do that as a primary benefit or primary reason. We're gonna embrace infrastructure's code.

The fact is, when you have the networking folks in a room, because you need to figure out what those networking design patterns look like for each one of these types of applications that we're putting in place. Yeah, man, we gotta work together. When you're dealing with the identity and access management folks and you're trying to understand how to build out a lot of the roles that are required to provide the access for that application, for that data, we've gotta collaborate on that front.

So although it's not your primary reason for infrastructure as code, it is an important side benefit. So we wanna use DevOps. I'm not gonna, you know, kind of sit here and go through the infinity loop, uh, again for the 30th time.

This is the ninth DevOps connect, right? DevSecOps show. So I don't think we need to necessarily do this, but the point is, we want to use these capabilities, right?

It's code infrastructure as code. We've got, you know, the ability to enumerate networking configurations and, and, and identity policies, uh, and workload, uh, you know, kind of environments, right? And we can put those in code and we can pump that into a repository and have that flow through the rest of the environment.

We can automate how things get promoted. We can use, again, capabilities like observability to understand what's happening within our environment. So we can take advantage of a lot of the stuff we've been doing for the past eight or nine years to great effect, right?

And, and again, it's hard because we're in the mix, right? And, and, and we're dealing with problems all day and, and we're like, ah, the world is terrible. If you think back to what developing software was like 10 years ago, and you think what we do now, it's kind of different, right?

And I would posit much better, much better. So code is code, code runs through the pipeline. Do you manage your infrastructure as code templates through a C I CD pipeline?

62% say yes. So 34%, yes, we have the same pipeline as our application code. So they're running it through a consistent pipeline.

Another 28% say yes as a separate pipeline just for infrastructure as code. I am not an overly religious person. So if you wanna run it through the same pipeline, great.

Make sure you protect that pipeline, right? Because it's not like it's your source code or anything. Oh, that's right.

It is your source code. So you should probably make sure that you're protecting that pipeline. And it was amazing, you know, in the old days when Jenkins was still a thing, anybody from Cloud B'S here, I should probably not, uh, um, make too much, uh, fun, uh, at this point.

Uh, but you know, early days of Jenkins, right? You'd do an assessment and they'd be like, you know, oh yeah, we got the defaults, the default. What?

The default everything. Yikes, right? Default passwords, right?

Default access list, default everything. And you're like, um, your source code is running through this thing, right? And they're like, oh yeah, this is great.

Our source code is running through this thing. I'm like, Hey, yay. Right?

Yeah, yeah. Again, gimme 10 minutes and, and metas exploit and I'm into this thing, right? And, and, and that's not a great place to be.

So if you wanna use the same pipeline, fantastic, right? Make sure you protect it. I tend to like isolation.

So you, you know, if I had to choose, I would favor using separate pipelines for different environments. But obviously that adds management complexity to what it is that you're doing. On the tools side, which of the following infrastructure is code tools?

Does your organization currently use Terraform far and away? Uh, 38% Ansible, 26%. Um, some of the open source stuff, right?

Pmi, uh, and cross plane, not so much, right? So I don't know who Alan was in, in, uh, uh, Amsterdam last week, uh, at the C**n thing, but you know, again, when you ask folks what they're using, it tends to be, um, and again, it could, doesn't necessarily have to be commercial, uh, Terraform, but Terraform does tend to be the standard on that front. Uh, and what about security?

How important is incorporating security practices? Extremely important. 54%.

Very important. 27%. So 81% think it's greater than very important.

Now, how many of you folks have sat down with a senior security person and they've told you, security's not important to me. Screw it. I don't care, right?

Said by nobody ever, because somebody is listening, even if it's the nsa. Um, and they're gonna come back and, and, and bite you on the bed if you'd say something like that, right? So you're always gonna say, security is important.

How much do you do in terms of making sure security's important? That's where the rubber meets the road, right? That's where the rubber meets the road.

So yes, everybody's gonna say it's important. We all want to do that. We all want to kind of stand our templates.

We all wanna make sure everything is as protected as possible. But are you going to devote the resources? Are you gonna hold stuff up, right?

Are you gonna block a deployment because you have a potential defect in your infrastructure? That's where you really understand how important security is to your environment. So security is code DevSecOps, the infrastructure templates, right?

That's all the stuff we talked about. Um, and then security is code. And I get confused by all these as code things, right?

You know, it's, it's almost like posture management, you know? And there's a thousand different posture management, cloud security, posture management, and south security, posture management, data security, posture management. I saw identity and access management, posture management.

So you've got like management inception happening cuz they can't, you know, there's just multiple managements in the same thing. Everything is posture management, right? So now we have everything as code.

Um, so policy is code, right? Infrastructure is code, code is code, right? You know, platform is code.

Everything is code. Uh, but you know, security is code. Think about it as this idea where I'm trying to figure out how to build right, the right tests, the right checks, the right gates within my pipeline uniquely, specifically for the infrastructure stuff that I'm thinking about, right?

Because what I'm gonna do to secure the network is gonna be different than what I have to do to secure my identity policies different than what I'm doing on workloads, right? Different than what I'm doing if I'm configuring a connection through an endpoint to a PAs service. So I want to think about how am I gonna enumerate these specific policies within a code base that then I can deploy automatically as part of my infrastructure.

And if you're a Matrix fan, right? Just security go down, you know, second door to the left and, and you'll be right there. Obviously that's a little bit facetious cuz we want security everywhere, not just in the little pocket that we allow them to play into.

So benefits to DevSecOps your infrastructure as code. And this is one of my favorite memes. I love the most interesting man in the world.

Um, I don't, you know, and see, I, the wonderful thing about presenting at at DevSecOps Connect, right? Is that I didn't have to go through rsa, they didn't get to see this presentation before. So I can put like beer in and I can put like profanity and all this stuff that the RSA people would be like, well, You can't do that.

Well, f**k you, I can do that, right? Because you didn't get to see it. Ah, I'm, I'm on video.

Oh yeah, you know, it's right. So Rob, just anytime I'm presenting, he just automatically hits the explicit tag, right? Because it's, it's, it's, I'm gonna, I'm gonna throw a few there.

Um, so I don't always infrastructure as code, but when I do, I DevOps the s**t out of it, right? And that's the way we wanna think about it, right? If we're gonna do stuff with code, we want to make sure that it's leveraging all of the capabilities that we've been building for the last couple of years.

So, accelerating the delivery of secure infrastructure, automating deployment, integrates security testing, ensures infrastructure is always available, or a secure and deployable state, right? Enabling faster response to security threats or incidents when you're a responder, right? And again, I, this, this, I start to default back to the 30 years I've spent in the coal mine of doing security, right?

When you're a responder, the first thing you gotta do is figure out what the hell this infrastructure is, right? If it's, if it's enumerated in code, if you can actually have it visualized because you build the same thing consistently every single time, right? That makes the whole process a lot easier, right?

A lot easier and a lot faster from a report standpoint. So what do we do to actually secure infrastructure as code, right? Um, and, and here there's not as clear a winner.

And this is also a multi-select question. Automated security testing tools, 23%, right? Manual code review, 17%, right?

Implementing version control, 20%, right? Established secure coding practices and guidelines. 17%.

Uh oh, that's another thing. So how many folks have developers who understand how to build secure code? Uh, and, and I'm not gonna call it the rest of you guys liars, cuz there was one dude in the back who said, you know, and he probably works for a security company.

You work for a security company. Yeah, yeah, of course you do. Um, right?

Cuz you know, if they can't secure their, you know, their own environment, um, you know who would, oh, I'm not actually gonna say that. Cuz if you ever try to run a pen test against the agents that they put onto your device, it's a frigging mess, right? So, so, you know, the good news is you'll probably detect the attack that the attacker used through the E D R tool to get onto your device anyway.

So we talk about inception, right? You know, kind of they're coming through the device that catches them. Am I losing all you guys at night security stuff?

You're, you're, you're all going through your sugar low. Yeah, it's almost time for lunch. You need cookies or something like that.

Um, we have to help these folks. They don't understand what security means. So as they start playing around with infrastructure as code, they're not gonna understand how they should be building out this stuff so that it's not overly permissive up upfront.

That's absolutely critical when we start talking about networks and network access and identity. Because the easiest thing for these folks to do is put in place an overly permissive set of rules and policies for identity. Cuz it's in code, right?

I mean, it's hard. I gotta start tuning it and I gotta play around with, with frigging, sometimes it's Python and other, you know, kind of policy languages and all this. It's complicated.

So I'll just, you know, kind of do the equivalent of any to any, right? And it works. It works.

It's great. It works. It works not just for the folks that are supposed to get in, but for all the folks that aren't supposed to get into, right?

So again, we've gotta help these folks understand how to do stuff in a way that doesn't put the rest of our environment at risk. So there are lots of tools, right? And we're still, we're focused on the, on the front side of this, right?

Because we're behind, we don't do a lot of infrastructure, we're starting to do infrastructure as code. We don't do a lot of security within this infrastructure as code. So we're behind, we wanna focus on the dev side of things.

Lots of tools, right? Bunch of commercial tools. These tend to show up as part of your either c a p cloud native application protection platform.

Took me at least 45 minutes to get that acronym right? Um, or CS p m Cloud Security Posture Management. I, I, I've known that one for a lot longer.

So that one kind of rolls off the tongue, uh, a little bit better, um, on that front. So, you know, you tend to have these folks, they sell you a whole bunch of different things. Being able to scan your infrastructure templates is just another thing that they do.

Bunch of open source technologies. And by the way, some of the open source stuff. And I think that gets to a question that Alan had asked, uh, Chensy and, and Andrew, uh, a second ago.

Um, right? That a bunch of the open source things have been commercialized, right? By these big companies are acquired.

So, so check out with Bridge Crews, uh, infrastructures, code scanner, um, they got acquired. So then the whole thing gets, you know, kind of subsumed into Prisma Cloud, right? And that's kind of how you can get Palos offering on that front.

So you see it's just back and forth. Uh, and I will put a disclaimer in here. This is not a comprehensive list, nor is it an endorsement of any specific tool.

And why do I say that? Because inevitably the, within a microsecond of me getting off the stage, somebody who's not on that list has lit up my email box and told me I'm an idiot and I don't know what the hell's going on because I did not include them. So I put that disclaimer in there for whoever it is in the audience that's about to yell at me because they're not, uh, on this slide.

I dunno, Andrews, they're open source stuff. Andrew must have open source for all this kind of stuff too. Track with, they should pay me more if they Yeah, that's true.

I forget that I'm a capitalist every so often. They should pay me more and then they could be on the slot. I'll, I'll get to that.

Actually, this guy, you're a great segue, Aaron, my friend. Um, so let's talk a little bit more on the infrastructure side because as we've done all this, you know, kind of infrastructure as code, now there's a new discipline called platform engineering, right? So platform engineering, these are the folks that actually code this stuff.

They're engineers and they build platforms. So my friends at Gartner, right? And they are my friends, I actually do have a lot of friends there.

Um, you know, Gartner expects by 2026, 80% of software engineering organizations will establish platform teams as internal providers of reusable services components and tools. So they're, their projection is, and, and this is where my meme comes in, right? I, I love this meme, right?

I did not pull a number out of my butt. They just said 80% of software engineering in 2026. That sounds to me like they pulled a number out of their butt.

That is it 70? Like, like Willis said, like, what, what, 40 minutes ago? There's no zero, right?

What's the level of precision? Is this 80% and four tens? Is it 76%?

Am I rounding up? I don't know. I think they pulled the number out of their butt and the cat said, of course they did.

So, and again, this is the stuff that entertains me when I'm up way too late doing this presentation because I'm doing other polls for Alan. So this is, now you're get a little view into the wackiness that is Mike Rothman there. Uh, but back to the point since I do, uh, can you tell, I'm, I'm a, a non-medicated a d d person.

Can, can you tell that? Um, so back to platform engineering, right? So the fact is, we're gonna have groups whose job is to build out these kind of code environments that the rest of the application teams are going to use in order to build out their infrastructure.

What does that give us? It gives us a vetted environment for folks to build things in. It gives us security policy that reflects least privilege and isolation where possible platform engineering is a fantastic, you know, kind of initiative that really helps to push along the consistency and the reliability piece.

And that's why we do infrastructure code, uh, anyway. Uh, but does that mean we now have to have a Plat sec Edge group? No.

Cuz remember when we started this thing, right? It was DevOps and then the security folks were like, oh my God, where are we? We so we don't exist, we're marginalized.

And then they had to go get a whole bunch of therapy and it just, it was like a mess. So then Shannon comes out and says, let's call it DevSecOps. And then all the security people are like, okay, now I'm validated, right?

I'm validated, I'm in there. So with platform engineering, I don't see any security word. So we're gonna have to have platform security engineers now too, just because anybody have kids.

Yeah, a couple kids. Anybody having a hard time finding therapist for your kids, right? Yeah.

Is that only me? Um, so you, you know, the problem is, I I I, I have a hard time finding a therapist for my kid. The idea of a therapist for me, forget it.

So I'm just gonna put the platform security engineering in so that we don't have to worry, uh, about that stuff. Um, so security teams need to work with platform engineering to make sure security is built in. We've seen this movie before, as Bill Murray reminds us.

We've seen this movie before, right? We have to work with these platform engineers when they're building out the actual templates that are being engineered. We have to help them understand what security means.

We have to work with them and be able to consult and advise them when it doesn't work. Or when the pen tester comes in and makes a make, makes a mess out of what it was that they just built, right? We can work with them, we can push things forward and we maybe don't have to change the acronym to Plat sec.

Ege, what about operations? So this isn't necessarily about programming, this is about saving your butt, right? There are certain things that should not happen in your environment, right?

Nobody should be using root unless very, very scarce op, you know, situations very unique operations, somebody una root, that's bad. So we wanna shut that down, right? That's a guardrail.

We want identity and access management policies to be least privileged. If somebody goes in and changes it and makes it overly permissive, we wanna lock that down, right? So again, there's a handful of things that we wanna do.

We wanna apply guardrails in that environment that helps us to maintain our safety, right? So we wanna do that in a consistent fashion. Changes are made via the pipeline.

So we used to call that immutable. Now it's just infrastructure as code, right? We don't allow folks to go in and mess with running environments again, unless there's an existential threat to the environment and we can't run it through the pipeline fast enough, okay?

Other than that, I want everything through the pipeline, right? Consistency, reliability, by the way, auditability so we know who did what. And we can isolate what changes are so that we don't run into regression problems.

Don't forget operations is still a thing and it's easy to focus on the dev side of stuff because we're so unsophisticated from an infrastructures code standpoint. We also have to think about the operations aspect of this. And this is just operational best practices.

Do not bury your head in the sand. Monitor your environment. I, I, and I know it, it goes without saying, what do you mean?

You know, monitor your gun. I mean, monitor your environment. Oh, and even better look at the logs and, and you guys are like, what is this guy talking about?

No, no. I literally sit with a lot of organizations. I'm doing assessments in there and I'm like, so you're pulling telemetry?

They're like, yeah, it all goes into an S3 bucket or, or an Azure storage account who looks at them? Uh, uh, uh, really? And they're like, yeah, I got nothing.

Right? We, we got like two people in the entire group and you know, we got 50 development teams and I, I got nothing. Right?

So if you're actually collecting the telemetry, look at the telemetry, right? And more importantly, as part of these tools, it's not just the scanning the infrastructure's code, it's actually doing runtime monitoring to make sure that if something does change, you're able to isolate that and, um, identify whether it is risky or not. Um, so bottom line, we'll just reiterate a lot of this stuff that we talked about.

Code your infrastructure. I've been talking about the whole time. Uh, run your code through your C I C D pipeline.

Scan your templates, work with platform engineering to reduce your, um, therapy bill. No, no, no, to actually make sure that those folks, uh, you know, understand how to secure the environment and ensure that you've got guardrails and runtime monitoring in place. Because things are going to drift, things are going to vary.

And variance and drift are enemies of the security practitioner. So I think that's what I got. com.

com, security Boulevard and a poll shows up, please answer it. But don't make up numbers. Don't make up the answers.

That's not helpful. If you don't know anything about the pool, are you talking to me? I'll talk about it.

Are you talking to me? Maybe? Um, so with that, thank you questions.

If anybody wants to come up or everybody wants to get lunch. Oh, oh good. Thank you.

Yeah, yeah. Perfect. I'll be that guy that asked the question before lunch.

Sorry. Uh, great presentation Mike. Um, infrastructure as code, uh, have you seen security personnel like push, implement, uh, or use IAC as a way to implement like, uh, disaster recovery BCP plans?

Cuz it sounds like there's an opportunity there. There, There is. So, so you know, there are a couple.

So, so, uh, first of all, that's a great question for those of you that didn't, you know, quite get it. Um, you know, are folks using IAC and, and the security aspects of that to enforce business continuity and disaster recovery capabilities. Um, and it's, it's a crawl, walk, run type thing, right?

So crawling is using infrastructure as code as a way to make sure that the environment is built in the way that you need it to be built every single time, right? Then we start to work into some advanced motions, especially around scalability because it's infrastructure as code. If I want to implement auto-scale groups and other ways to expand and contract my environment in real time based upon, uh, usage and and performance requirements, I can do that.

All of that is enabled through infrastructure's code. And then there's other stuff like, oh my God, I just got hacked. Or I'll actually tell a story.

So, um, I've, couple of good friends, they work in a, a very large financial institution, this was a couple years ago, but they ended up having an automation that took down two of their biggest customer facing environments. Big financial institution, customer facing environments are down that qualifies as a bad day, right? A bad day.

So the good news for these folks is they actually had everything with the infrastructure as code. They were able to, you know, basically move to a different, you know, availability zone within the regions, spin everything back up, me wire it back up together. It was up and running in six hours, not like the terrible eBay outage of 2000 and, and three, right?

Or whatever that was when you couldn't buy your pest dispensers or, um, y y you, you know, whatever it was that you bought on eBay back then and it was am I remember that one, the, the Oracle blew up and and they couldn't recover it, you know, for three or four days, right? And everybody was all p****d off cuz you know, that was what we called e-commerce back in the day. So these guys were back up in six hours, right?

And this was a total failure, I mean a total failure. The automation wiped out the entire customer facing environment. So the answer is yes.

Um, and again, that's stuff that we can intentionally use and do with infrastructure as code. Perfect. I think we got one more.

We got, we got one short one left. Okay, I'll, I'll keep it short. Thank you for the presentation.

The question may be short. I can't say anything about the answer. No, No worries.

So I, I'd love for you to speak a little bit on how you think AI is going to affect security operations moving forward. Ooh. Oh, how much time you got, man.

Um, so initially what we have are a set of fairly cryptic alerts that come out of most of the tools that we use because they're built by engineers, not necessarily security responders and certainly not UX people, right? So the first area is that we can enrich and embellish well, okay, chat b t may embellish a little bit, but, but really, you know, kind of build out the explanations in the context for a lot of the alerts that we get that can all be done with ai. That's the low hanging fruit.

We're starting to see that already. Integration of chat G P T into Microsoft Sentinel is a great example of that. Um, over time, the ability to more effectively use iterative types of AI and, and different, you know, kind of machine learning mechanisms, which we've been doing for a long time.

But now that we can kind of fuse iterative and generative together in order to improve the way we detect stuff and then figuring out how that works within the environment that we have, that's going to accelerate a lot of the education of our next wave of responders and, uh, analysts within security. So I I, I think it's actually fantastic and a great way to end it. Um, I I, I keep getting excited by this stuff.

You know, we call security and, and, and I was doing the math the other day, right? I think I've been to 24, 25 out of the last 27 RSA conferences. And, and, and that was like when it was like 10 guys in a hotel, right?

I mean, my first one was at Knob Hill, right? And there were maybe 250 people there, you know, back 25, 26 years ago. And I'm still excited about what we're doing in security.

I'm still excited about kind of a lot of the technology innovation that we continue to see and infrastructure's code is just one aspect of that. So I appreciate everybody's time and have a great time this afternoon after lunch. Thank you, Mike.

Thank you.