DevOps in the Age of AI & The $2M Downtime Problem | TSG Ep. 942

I got a little DevOps drama to start your weekend. You're watching Textron Gang. Hey everyone.

Happy Friday. That's right, it's Friday with a capital F man. This, this Friday couldn't come soon enough for me.

The week went quick, but I was, I was Jones in for Friday, uh, next week. I'm on the road though. I'm out in Houston at a Qualis conference, but we'll be there.

It's Qualis changed the name of their conference. It's Rock on. But, um, in any event, we are here closing out this week with an all-star cast on Textron Gang to discuss some interesting topics.

Let me introduce you to our gang members for today. Joining us is the one and only John Swartz. Gina Rosenthal, and still in Barcelona.

And he complains about it like he's doing us a favor. Mike Ard. Mike, I know there's no joy in Mudville, as they used to say in Brooklyn.

Wait till next year. I have nobody to keep me company in my misery here 'cause there's just no Yankee fans. I didn't want anyone to keep me company.

I, I I shut it down. I didn't even watch the eighth and ninth inning. Okay.

Anyway, wait till next year. Um, let, let's talk DevOps though, 'cause that'll always lift our spirits. There You go.

So what's going on Mike? Some drama in DevOps land. Well, Well, chain guard did a survey of 600 software engineers and, uh, determined that one of the big issues of the day seems to be that, well, they don't have time to actually build new features and new capabilities 'cause they're caught up in too much scut work.

And maybe someday AI will help with all this, but, um, there seems to be a lot of frustration in the world. And I thought this whole DevOps thing was about ruthless automation. But we seem to have all these bottlenecks and things that we have not gotten to yet.

So, Alan, you've been of course, chasing this whole space longer than I have, but what's your assessment of what's going on in the world and why can't we seem to just get out of our own way? Wait, what's that? I hear in the background.

That's what I hear. The smallest engine, uh, violin in the world playing hearts and flowers for our DevOps engineers. Friends, you know what, it was a lot worse before there was DevOps.

It's just the very nature of the beast people. Have you ever heard anyone who say, you know what, I'm overpaid and underworked. It's, it's not there.

There's always going to be things to do. And you know what, when we talk about AI making our jobs different and easier, it's not true. It's not gonna make it easier.

It's gonna take care maybe of some of these mundane things, but all these higher end or higher value things that they're gonna have us do, it's gonna keep us busy too. Let, let's get this straight. No matter how good an AI is, it doesn't mean that the average Joe is gonna not work hard if you think that AI exists.

So you don't have to work hard. You, you got a bad attitude. You got the wrong, your wrong thing about it.

It it's, it's the nature of life to work hard. Right. You know?

And if you don't wanna work hard, find something else. Right. DevOps engineers it people, people don't pay you nothing for nothing.

So, Mike, I I think the things they do may have changed, but the, the, the hardness of their work is not less. So do you believe the thought process out there that it says that we're gonna build more software in the next two years than we built in the last decade? I mean, there's all those folks that are kind of banging that drum.

Yeah, I think we are. I think the amount of code, the last, and I, you know, I'm not gonna swear on a stack of Bibles, but the last numbers I saw was that this year we, we we've generated a third to a half more code than last year or something like that. Or from a couple years ago.

We are absolutely generating terabytes, petabytes, floppy bytes, whatever of more code than we have in the past. How much of it is being generated by ai? A lot probably, but we're generating more code.

And so the fact that we could generate more code within essence, probably a stable amount of DevOps engineers, you know, go ahead Gina. I got something to say about this. Of course, I come from the ops part of DevOps and, um, I, I don't know Alan, I gotta pick on you in just a little bit because even when I was a a sis sis admin way back in the day, the first thing we did whenever we got anything was we made sure we didn't have to go in the data center.

We scripted it, we automated with the tools we had as much as we can. Sure. I can definitely see DevOps using that.

I clicked over to the, the actual, um, survey chain guard itself to chain guard. What I found was interesting is, um, that they, they interviewed software engineers. That's who was part of the survey.

Not necessarily DevOps engineers or ops people at all. Right? But what the software people were annoyed with or what they were able to do and actively encouraged to do was things that they didn't wanna do with like security patching, admin tasks, including meeting and communicating with people.

Um, system design and architecture, which are all the ops side of things. These are the normal dev things that we don't want 'em to do anyways. And I think that even goes along with your, with your comment that there's so much more software being spit out.

Of course these are gonna be the, this is gonna end up being the bottleneck is Yeah, you gotta go back in and test your code and fix your code. We've gotta reevaluate what architecture it's on all the rest of us. This is just a part of computer engineering, not necessarily DevOps engineering.

So, um, but I did, one of the things I would that from the survey, 'cause usually I don't like surveys. 'cause you look and see how big they are and I just have to tear that all up and I'm like, yeah, whatever. But the one thing I saw that was really good was the top thing that, um, software engineers worried about is lack of privacy and security and lack of accountability in the code.

So they're worried about the right things, which does make me really happy. You know, the one, the one thing that really like stood out to me that was really damning and it kind of synthesizes their quandary or their conundrum is that only a third of the software engineers said they spend time, a majority of their time in the things that really interest them or energize them. Yet this is happening even though two, what's it, two thirds of them said that their software engineering tasks were either mostly are fully automated and it's this complete contradiction.

Um, and shows just shows me the sense of frustration and the fact that they can't spend more time working on new features and they spend time doing other things or that they sh they sh they really don't wanna do. This is not new. We've seen this at other surveys.

Yeah. I've seen surveys. You know, they spend 11 to 14% of their time actually coding and they wanna code.

So how much of this is just a simple management problem? Because at the end of the day, I feel like if you're listening to what they're saying, we're spending too much time on things that don't drive any value back to the business. And we've created maybe this giant workflow and a system that becomes a monster of its own.

Rather than just kind of figuring out how do we get out of the way of these folks and let 'em do what they're supposed to be doing. Well first you gotta define what they're supposed to be doing. Mm-hmm.

Right? And when the inmates run the asylum, they define what they're supposed to be doing and that may not jive with what their bosses or their board or the executive team wants them to be doing. Um, you know, and, and, and here's the thing.

I think in the layer, the world of ai, right? In the age of ai, what they're supposed to be doing is going to be changing. It may be that doing coding is not what they're doing.

It may be that what they're doing is directing the AI to do the coding and then being the human in the loop looking at that coding and does it make sense? Right. Let alone testing.

And that'll be further down the pipe. But, you know, we may be coming to a place in the world where if we're really gonna do two x three x, 10 x the amount of code that we've done in the past, that code's not gonna be human generated. It's gonna be human supervised, human in the loop, you know, but not the basic function task work of a software engineer is gonna change.

Gina, we've been talking about on this show platform engineering and you know, theoretically this is all about how we're gonna provide a better developer experience. But I mean, you're working the ops side. What, what is the, the source of the tension?

I think it's, if you look at it holistically like a system, I think that's, that's kind of what I'm seeing. If, if the developers aren't allowed to do the initial coding, does that mean that their good work is, um, improving the code that was spit out by a machine and toughening it up and hardening it and make sure that it's always available, always working. Um, how, how does that, are they gonna automate all of that?

Like where's the human in the root, that loop? That's one thing. But if it continues to be, right now humans using generative eye to do live coding and that comes out on the other side and they don't have the architecture for it and they don't have, you know, they introduce bugs 'cause they're not careful.

Like all of, they don't do their side of development hygiene in the data center. The, the, um, ops people are gonna end up being the, the, um, the, the place you can't get around. They're gonna be the, the blockage point again as usual.

Because you can't, you'd wanna put something out that is gonna work and it's gonna be repeatable and it's not gonna break all the other things. So, uh, it's just, just, you know what this reminds me of? This reminds me of when I was a system man in the Linux days and you'd get stuff that people had created on their Linux laptops of whatever flavor, God knows what, and they wanted you to put it in production and, um, it, it couldn't go in production because, um, you should have seen some of the laptops that we saw back in those days from the postdocs, right?

So like, it would be anything couldn't go into production 'cause it was dangerous. And they already had started to be some security, um, certifications and requirements around things. So the way I see it, you're not letting the devs do their dev job that they love to do.

So they're, they're getting really quick, they're getting these prototypes of code. If they're not being thorough in going through the code and cleaning up what's gonna cause a problem down the line and they just toss it over the wall without ever trying to run it on, um, the, the infrastructure that'll be run on in production, that's where the problem's gonna be. That's probably a place where platform engineers could help out by giving them a good test bed.

That's exactly like the test bed they're gonna use. But then that costs money too. So it's a, it's the same old problem to me.

I don't see a big difference in the problem we already have. Can I ask a, can I ask a dumb question? So, um, uh, is AI an elixir in this equation?

Or in other words, in six months from now when agents are deployed and there's more automation at this level, do they pull numbers? Are these survey numbers, are they going to appreciably change? Are they gonna stay stay the same?

Which way do you think they would change up or down? Don't know. I, I think they would improve.

I would approve free them up. I mean the that's the idea, the concept. I would, I would think I Think they'll stay the same.

They'll stay the same. Okay. Alright.

I'm not, I'm not entirely sure this is all gonna work out as planned either. So just because we're writing more code doesn't mean we're shipping more applications. And the fact of the matter is that the pipelines are fairly brittle and we're gonna have to maybe redo those pipelines as well.

It's like basically, you know, it's the proverbial 10 pounds trying to get in the five pound bag, you know, Proverbial, I always wonder how that, I, I always wonder how that plays out, like at the executive level and the impatience among people at the top looking at at the end results. You know, they're always, they're gonna say, so what, so why are we spending all this money? Why are we investing in this?

That's just, that's just kind of where I'm, I'm thinking how they think. Oh no, the spin meisters will spin up metrics and analytics that show, look at all this code we're generating. It's not the devs.

It's those probably those ops people again Holding us back. Yep. Yeah.

And then once we get past those ops people, you know whose fault it is ultimately? Security. Security, yeah.

Always saying, no, all these compliance things, we need deregulation. God done it. Oh wow.

Yeah. It's interesting. But a as I said, I think software develop, what does software developers daily tasks are gonna change?

This happened in QA with continuous testing, right? You went from QA engineers who actually ran test to QA engineers who designed test coverage and the test themselves are kind of automated run, right? Well, to Gina's point, maybe we shifted too much stuff left and the developers are doing stuff they're not supposed to be doing Well.

So interesting. That's something we're gonna talk about in in the, uh, in the next one I believe. Or it was one of the, I did an interview yesterday on this.

Have we sh oh no, it's Derek Holt, uh, digital ai. I had a conversation with him about it. We might have shifted too far left, but now will legent AI allow us to fix the mistakes of over shifting?

In other words, it's not a bad thing to do things earlier in the pipeline, but asking the developers to do it is probably not the smartest thing. Can we have agents do that kind of stuff? So it still gets done earlier, but it's just not on the shoulders of developers.

Alright, so interview I had with Derek called CEO of digital ai Move ops left. Yep. All right, let's take a break.

We're gonna come back and, and again, we'll, we'll stick with this ag agentic ai, I guess thing for now. What the hell? It's powering the economy.

Uh, AG agentic AI comes to it. You've earned it. The spotlight, the responsibility, the weight of teams, companies, and entire industries fall on your shoulders.

Lives depend on your decisions, your home life included that work. You are protected physically and digitally. Nothing gets through your team without a fight.

But in a globally connected world, everyone sees you, including those who mean to cause you and your organization harm. And now home your sanctuary attackers see an opportunity. Your digital front door is wide open.

And what compromises your home can breach your boardroom. Because the devil's greatest trick isn't targeting your workplace firewall. It's convincing you that your personal life isn't at risk.

Black cloak, digital executive protection, defending the new attack surface your personal life. Hey folks, I'm not sure if Agen AI is coming to it or coming for it, but we're gonna find out one way or the other. But the issue is, um, there's just a raft of these AI agent tools now being announced.

Space Lift has one for a bio coating tool for provisioning infrastructure. PagerDuty is talking about an AI agent that acts like an SRE. And even SolarWinds is in the act saying, Hey, we've got AI agents too that will automate your entire workflows.

And John, I'm willing to bet that if there's any company out there that makes something to do with IT platforms, they'll be sending us an announcement about an AI agent shortly. Oh, of course. But, but what's your take on what's going on here and how fundamentally different will the management of it become?

Will Page PagerDuty, uh, reached out? I think they, we, we both talked, I mean, I think you did a story in DevOps, uh, about them. They announced this suite of AI agents, um, for IT management platform, including one for site reliability engineer.

I think they, they announced, uh, three to four of 'em. There was something called, uh, in addition to the SRE agent, uh, there is the PagerDuty scribe agent that instantly transcribes Zoom calls and chat conversations. Uh, there's also the PagerDuty shift agent, um, that detects and automatically resolves.

I'll call on-call scheduling complex. So it's all ideas of getting ahead of these, these issues in it. And I think with PagerDuty and based on their reputation and based on who they work with, it'd be interesting.

I think it's kind of intriguing what they're gonna do. I, and I, again, I don't know how this will play out because in a sense, I'm not sure which enterprises are gonna be using this or if they're gonna feel firmly, uh, uh, safe doing it or confident in doing it. It's, uh, you know, the one thing that I keep running across and it's really frustrating to me is I keep hearing about all these AI agents and what they're gonna do for every company and all these enterprises in finance, healthcare, et cetera.

And yet when I press all of these companies, I'm not talking about PagerDuty, but, but a lot of other companies, I'm not getting any real life examples beyond the most basic cautionary example. So that's where I always kind of hit, hit an obstacle on these, these stories. I mean, I wanna see some real examples and maybe it happens six months from now, but in concept, these things are all interesting.

But in practice there's like a chasm for me. So that's kind of where I stand on this. Gina, you ready for a digital little buddy who's gonna help you out and do everything?

I think it depends. Look at the it answer, it depends. Um, number one, I, the, the, there was a platform, I think PagerDuty mentioned it in one of their press releases.

It's, I can't remember the name of, it's the open source one. Um, I have to look it up, but I don't remember. They did mention that they were able to work with connect into that platform.

Um, but this one you have to trust the, the agents to do what they're supposed to do. I was working on one project recently and they had a QA bot that just answered questions and would, would explain what was going on. 'cause we really fast moving content.

So they used projects, so they used all of the Slack information, all the way to documentation, everything. And it was, it was fantastic. Whoever designed that little QA bot did a really good job.

And once you knew that you could trust it. Um, it was like you would just ask QA bot when you got stuck on things and QA bot would give you the answer. And it was perfect.

It was up to date what you needed to happen. Um, now as far as infrastructure as code being run by bots, they did, I didn't see, I looked to see like, well, okay, how is that working? How is this bot working and how is it trained?

Yes, it's gonna take natural language to say. So a developer can say, I need an environment that's blah, blah, blah. So you don't have to put the ticket in.

So, but does that put the ticket in? Does it kick off a workflow that puts a ticket in? Does it, what, what is it allowed to kick off?

What does it come back and say, no, we can't provision that for you. Like what, what is the, the safeguards that are put in place and what workflows is it allowed to interact with and call? Because it's really just scripting, like on a very sophisticated manner.

What is it only gonna call what you have or is it gonna be able to just piece together whatever the developer dreams of, which may be good? Like what is it? And the other thing I saw when I was looking through it on GitHub is that all of your infrastructure is stored in a SQL light database.

And if you do anything to the database, you cannot change your infrastructure. I'll just put that out there as an IT person. No, I don't like at all.

So Gina, I I think you made some points. I think, John, you made some points too. Let me, let me be a little more blunt than both of you.

So you, we got in this one, we got three companies who've rolled out Agentic ai. I would posit that over the last month we've each, you know, we've collectively covered a hundred companies that have rolled out Agen ai. And I can't count, I, I wouldn't need all five fingers to show you five companies that are truly using autonomous agents.

Gina, the chat bot, like customer service chat bot, perfect use of ai. It's not really agentic ai, it's generative, but it's, it's a great use case, right? And I, and there's nothing of matter with saying that's a great use case, but it's not this full blown agentic ai.

Because to me, what, what makes an agentic AI is not a glorified API call it has to do so autonomously, right? When it's going to vibe code infrastructure is code, right? Vibe code to provision infrastructure.

It's gonna code for the infrastructure and get it deployed and say, here you are. Thank you sir. May I have another?

Right. That's not what these agents are doing. That's not yet.

Anyway. And you know, I I did a shimmy says yesterday, Thursday afternoon, where to me this is, this is part of the problem. We've invested Singapore's GDP into, into AI and agent AI and data centers and everything else this year.

And, and we've got now Somalia's, GDP, right? That that's not good business. That's not good.

If that doesn't turn around, we're, we're in a heap of trouble here. A heap of of trouble. So the this, so that's, I'm sorry that, that's interesting that you say that, Alan, because I, it's, it's exactly what's happening.

So the only examples I can get are generative AI examples in use, and they are very superficial types of customer service or maybe personal time off or HR related tasks. The Entech AI stuff is still far away. So there's this huge, as you said, there's this huge divide, you know, the money that we're putting in and the money that's coming out of this that's, that's generating revenue is at a very modest level.

And I, I've been pressing Amazon in particular to come up with examples for me before reinvent. And, um, I'm, I'm still waiting. But this is, this is what you said.

Again, you said this yesterday about the, was it the Singapore and then Somalia example? I think that's exactly what's playing out among enterprises right now. I think if you look at all three of these companies, they're all saying one thing that is similar.

They're saying don't use this stuff for anything that's really mission critical. They're saying use this stuff for, you know, everything wrapped around your workflow, whether it's transcribing notes or if the application or the database that you're building is maybe for a developer to write some code, but it's not part of a production environment. Sure, go ahead.

But they're also saying that, you know, these tools are, uh, shall we say fallible. Yeah, I would love to see some hard numbers, right? Because this is just like what we used to do with Jumpstart and Kickstart.

We would have some pre stuff, pre scripts that would call the OS to be installed. And then we'd put some wraparound scripts to download OS applications, everything else that needed to happen. So this is the new age of doing that.

It would be so cool, like if you had, uh, very secure walled off, uh, AWS PLA section for the developers and you could let them just call certain things. And that's the, those restrictions are on the back end of this generative AI that calls it, yes, you can do this. No, you can't do this.

And there's some, it gets a ticket. I don't see any of the plugins being announced with that. It makes it actually safe to use.

And I think if you can use it safely in dev, that would be amazing because then you could add those things to a pipeline that would, that when you're ready to move it to production, to test, to, to see if we could call it and it would go up. Boom. And we're ready.

So it's, it's got a lot of promise, but, um, they are just waving sparkly wands when they talk about it. So let me, let me, let me, let me give you Shimmy's take on this one. You know, why am I in tech, why do I love tech?

I mean, I, I could have done different things with my life. At the end of the day, I'm a gadget boy. I love new technology.

When I hear there's a new release of something, an AI browser, SOA two, a new Apple watch that does, I don't know, some biomedical thing. I, I'm an, I am a gadget boy and most of the people I know in technology are also gadget boys and girls. And I think that's why AI has settled in so deeply into the tech world, because this is the shiniest trinket to come down the turnpike since they discovered gold in Sutters Mill or wherever it was, right?

This is genie Gina. It's not just, you know, little like, uh, shiny things like that. These are full blown July 4th fireworks.

And we are drawn to it like a moth to a flame, like a moth to a flame. We don't even care that it doesn't really work right now. Just the fact that we could play with it is enough to get me off, right?

To say, Hey, it is great. I can't, it's gonna get better too. Don't worry.

And I, I think we are as an industry all in on this and, and we're trying to convince ourselves that, yeah, no, no, there's, there's definitely a pony in this room full of manure, right? Of horse manure. There's definitely a pony here.

I know there's a po I I could see a pony when I could smell a pony. I think, I think the prob the problem is, is that CEOs think that there's really a pony in the room, and they're well, yes, making business decisions based on the fact that there's a pony and it doesn't exist. Well, they could smell it.

They could smell it and the board too. But the, but the one thing they, they, I, you think they would learn from this, but like, when a new technology comes around, it's not infallible. The first version, it usually takes a couple of iterations to get it right.

And that applies, especially to companies like Apple. I mean, they do get it right by the second or third version. And I think maybe they expect this, this kind of instant gratification based on all this over hype and all the investment.

They gotta be patient. And I, I think maybe I'm being impatient when I ask for real life examples, but you know, they're putting it out there. So, you know, back it up.

You wanna see the pony. It's fine to ask that. I say I'm not a gadget girl, but I love to fix tech.

I like to break it apart. I like to know how it works. That's what I've been my whole life.

So for me, when I see people talking about a pony that's not there, I'm like, ah, dang, I'm gonna have to clean up the poop. This stuff. Right?

And who wants to clean up a poop when there's no pony? Exactly. But then how did the poop get there in the first place?

If there's no pony, Well that, and that's the mystery of the universe, Greg. I don't know. No, the chicken or the egg.

The chicken or the egg or the, um, po first the poop or the pony. All right. On that, on that serious note, we're gonna take a break.

Let's come back and, and we'll move over to the high cost of failure. You're watching Textron Gang Discover Textron Group, the epicenter of tech innovation. We are your go-to for reaching IT, leaders and practitioners worldwide.

Our secret impactful content that sparks awareness, engagement, and top quality leads with us. You'll access editorial websites, streaming videos, virtual events, custom content analyst research, and more. Join our satisfied clients.

Let's revolutionize your tech journey. Contact us today and tell your story to the world in the most powerful way with Textron Group. Hey folks, we're back and we're talking now about a report that New Relic put out talking about how the high cost of failure with so much writing on these IT environments these days, it's, you know, tens of thousands of dollars every time there's an issue and it adds up pretty quickly.

Gina, I would love to get your opinion about one thing as a result of all of this, but I mean, just how fragile are our IT environments? I mean, is this something that, you know, happens every other day or does it happen once a year? What's your sense of what's going on?

It's the constant threat that it's going to happen. That's why observability is so important. Um, new relic's, uh, estimate from their survey from the folks they talk to is that the median cost of a high impact outage, it outage is $2 million an hour or an annual median hit of $76 million.

So that's a lot of money if you're down for an hour, $2 million out the door. So, um, what they have is tools that can look at everything in the stack. Everything from, um, the applications, from the infrastructure, the security monitoring and the digital experience, monitoring, log, log management, everything.

If you can can, if you can drill into all of those at the same time and um, kind of see where the issue actually came from, you can solve those issues much faster than you could without, um, any visibility. And for me, that that's kind of, it is all about business. And I think we forget, we're gadget guys and fix it girls is we forget that we are doing this for business to happen.

And now that everything is, um, in the cloud, everything is digital, everything about your business is visible or we want it to be. And now we're adding all of these AI things like we just talked about. How much more code are we creating?

And then it has to go through, you know, can we get it on? If we just push it out to production, what happens? Are we gonna have an incident?

And can are, if we do, are we able to quickly pinpoint the root cause of that incident to bring everything back up and get back in business and not lose $2 million an hour? Um, so it's, it's critical. And I, I think this is actually a great usage in OTVI, as long as the tools are tuned correctly and you can trust them, that you're able to see, uh, you're able to have all of your smart people in your different divisions see, is this a security incident?

Is this an infrastructure incident that we push out a bad version of code and you can kind of, um, correlate all of those activities at once when you can see the full stack and dig through all the logs and, and those are visualized with, with something like New Rolex. So I'm not sure I'm buying into the $2 million an hour number. 'cause that would kind of seems a little high, but, um, Alan, you know, what's your take on what's going on here?

I don't, I I'm in a, just say it, say it as it is heading into the weekend, I got a problem with a company that sells full stack observability, showing me a survey that says using full stack observability saves you money. So a Little self-serving, we, we Might see, yeah, I mean, come on, marketing people smarten up. Did you, did you ask your ai what survey should I do?

And, and the AI said, oh, you're a full stack observability company. You should do a survey on how much it costs for downtime from not using full stack observability. Observability.

I mean, it's just, IIII, I don't know, am I being mean and grumpy today? Am I cry? Just I expect more.

I expect more Just to, to just to touch, but not too much. Now that being said, downtime is expensive. I know this from my security days.

What, right? You bring a data center down, you bring a business down, you know, for the average enterprise, the mom and pop the orthodontist, you know, his business doesn't care if they're computers down that day, they'll bill tomorrow. Um, you know, SMEs don't do $2 million an hour, but large enterprises downtime.

Yeah, it's measured in the millions of dollars an hour. I dunno if it'd be millions of dollars, but it's more than just, if you've got a mom, especially like medical, it's gonna be more than just your billing, that you're worried about all of your tools. You know, I'm saying like a, a, an orthodontist, all they really use the computer for, or maybe they use it now for their x-rays, whatever.

Yeah. They use it for the x-rays. They use it for finding out where they're actually gonna drill.

So you want those to be up before you go in there. Yep. But, you know, but the, the business isn't big enough, is what I'm saying to really say, you gonna lose that kind of numbers.

Yeah. But the enterprise, yeah, I think you do lose those kinds of numbers. Um, full stack observability, cuts, downtime costs.

John, again, I don't know, You've been covering public companies for a long time and I've yet to see this, like footnote in any of the financial reports that said, yeah, we lost $2 million because now there was an IT outage. So I'm just kind of like, you know, what's real and what's not real. 'cause I know it happens, but I feel like there's mitigating factors in work here.

Yeah, no, it was only so almost to have to pull teeth to get, to get them to acknowledge this stuff. And I mean, unless they're, they're forced to legally, you're never gonna hear about it. Um, I just remember years and years ago I was working on this cybersecurity book with Byron ato.

He used to, I think he still writes for us. Um, and, uh, I remember being, uh, he said his LifeLock invited us to some sort of consulting brewer. It was kind of a, it was a weird junket where it, it was in Arizona and it was Kevin Mitnick was there and they were, they were off the record describing all their customers, target, Walmart that had these huge breaches that were never publicly revealed.

And, um, that to me, I, that always resonated with me because I think this happens quite more often than we actually think or actually will ever know. So, um, you're right, Mike. Um, there's never a footnote, rarely a footnote.

And if you have to go through the, the, the financial reports, you go through risk, the risk, uh, section, uh, sometimes unless it's egregious, you're not gonna hear about it. So, Gina, is there, is there a Cover up here? Is that what goes on when these things happen?

We've got, we've got a deep throat, we've got a deep throat in the, uh, in the observability space. Uh, no, I don't think it's a cover up, it's a legal thing. I think that John said that, and I think that's exactly right there.

If you talk about it too much, you, you probably would introduce a liability. So why put something down on paper that's gonna be discoverable? Well, but, but I would guess, you know, as a legal thing, Gina, it, there are now lead, there are laws in place that you do have to disclose a breach.

Yes. You know, within a certain amount Of time. This was, yeah, but we were, we did our briefing.

This was before there, there were laws that would, that would force you to publicly disclose what had happened. But before, like routinely it wasn't, But, but you looked at something like the Verizon data breach report, which is kind of the granddaddy of breach reports, right? And they, they've done a great job over the years of quantifying what a security breach and what downtime costs and I, and those are drawn from their own experiences consulting.

I think the article, like the one thing I that I have about the article is they did not link to the, um, survey. So I've just been kind of looking through it right now. But I I, I think that's, they probably could have phrased their, um, you probably could have probably rephrased their releases a little better.

Um, from a marketing point of view, definitely. Because if you look at the actual survey, it's more on, hey, these things happen. You know, they have 'em, you might be the next company that has $2 million.

You also know that those teams are siloed. You also know that those teams don't talk to each other. You also know that nobody really looks at the logs if everything's kicking along and it's great, nobody's deep diving into logs.

So there could be something that's building that. If you're using predictive ai, it can be monitoring your logs and say, Hey, this is this server over here. The devs have hit it too hard.

It needs to, you know, you need to, to have another one, or you need to go and put some more, you need to replace something. Like you could, you could have that visibility long before your, uh, a critical server goes out and you can plan the downtime and do whatever. I think that's more of what the survey was getting to.

That if you have, if you use ai, you get around a lot of the human, um, Alert fatigue, that happens enough. I guess what I'm kind of circling around here a bit to get to is, you know, maybe the SCC should have a rule that says you gotta report downtime. Wait, who, who, who should have a rule?

They don't have no stinking rules. They, they, they, they're working with pig tech. They're not, they're not overseeing Them.

Yeah, we don't, this is all cooperative. We don't stinking rules. We furloughed those people and they're not getting their back pay either.

Um, Wow. You know, they, we can't Look, Mike, let me just, I don't wanna bust your bubble, but there's no tooth fairy. There's no Santa Claus, and the, the government is not gonna be helping you with these kinds of things Unless they can get right.

There's no, They're, they're investing, they're investing in these companies. They're like working with them now. Geez.

Geez. Come on man. I feel like Randy Moss, come on, man.

That New Relic marketing team. Come on, man. Come up with a better, a better an angle here.

Anyway, guys, it's Friday. I, I think I, I feel a beer's coming somewhere. Um, I hope you've enjoyed our show to this today.

Gina, John, Mike, thank you for joining in. Thank you for watching. As usual, we've got Techstrong TV coming up right behind this, so stay tuned for a full load of interviews and sessions and learning and all kinds of good stuff, podcasts and everything else.

We'll be back Monday, hopefully. Um, actually I should mention Monday we're supposed to be seeing the release of the hostages from Gaza. So maybe, I don't know, Mike, maybe you could head down there and do a man in the, uh, man in the street report on that.

You're closer than us. Probably not, probably not, but um, we'll be here Monday. We will be here Monday.

Have a great weekend, everyone. This is Alan Shimel. We're out.