Rising Observability Adoption in DevOps with Futurum Research’s Paul Nashawaty

This is Techstrong tv. Hey guys, thanks for the throw. We're here with Paul Nati, who's working over at the Futurum Group, and they have a new observability report, and of course, Futurum Group and Techstrong are new firm boss buddies and allies.

And so we're working our way through a lot of new content opportunities. Hey Paul, welcome to the show. Thanks, Mike.

Michael, very great to be here. Uh, you know, very excited topic on observability. It's, it's taken off like crazy in the market, but yeah, excited to talk about this research.

Yeah. So walk us through the high points a little bit and call out anything that kinda surprised you. Yeah, absolutely.

Absolutely. So, you know, I guess the, the, the starting point is really to understand the observability market as a whole, right? So, you know, when we look at observability, you know, it, it really is starting to grow in popularity because organizations are really trying to understand that taking actionable insights and having a faster release cadence of applications requires, uh, actionable insights on that back end, right?

So, like with that feedback mechanism into the CICD pipeline is becoming more and more critical to organizations success. And the dove dovetailing in the, the insights and the, uh, alerting and monitoring and observability tools into the, the DevOps and the CICD pipeline is becoming critical for many organizations. So this is pretty exciting.

Um, what we have is a study that we just kicked off, got the research back, really excited about it, uh, have some, uh, definitely some interesting, uh, findings here. It was a worldwide study. It, it was a study of over 800 and, uh, 50 respondents.

And what we did was we asked those respondents that were IT professionals, application developers, and DevOps and platform engineering professionals across the world to really, uh, give us their perspective of, um, what they're doing with regards to software as a service, what they're doing with public cloud infrastructure. And, you know, IAS and PaaS as well, co-locations, data centers, edge locations, everything that's, uh, that's a responsible for evaluating, purchasing and managing as well as building those application infrastructures. So, you know, when you look at, um, the study, the responses came back from who these people that were responsible for decision making and influencing those decisions within their organization.

So they're really the hands-on users, the people that, that got it, and understand what they're trying to do. And what we were trying to understand from this study was we're really trying to gather those insights to help organizations navigate the complexities of application development deployment, as well as like, you know, the, the, the connection or the intersection, um, between observability and ops integration, right? Because there's a lot of the application and the infrastructure as well and how it, how it aligns together.

Uh, we also were looking at providing details on monitoring, uh, observability strategies, including tools and methodologies. And we're looking at the, uh, different, uh, deployment, uh, environments that, that were really impacted by it. We were looking also to impact, uh, investigate the impacts that teams were looking to solve.

Complex issues like different types of application stacks and tech stacks that required. And then finally we were looking at the future spending, right? That's where everybody wants to know, where's this going, where's the money coming from, and how are organizations spending in the future?

So, really exciting stuff there. Lots of data to get into, but, uh, really excited about this study. So what's your sense of, where are folks right now?

Are they are deploying observability? Are they still researching these platforms and, uh, they're allocating budget now, but they're still early in the cycle? Yeah, there's, that's a great question.

There. There's a kind of that bell curve that there's outliers on either side, right? You have, you have a, probably a very small percentage that are fully optimized fully using observability practices within their organizations.

And when I say a small percentage, I'm saying sub 10%. And, and the reason why I'm saying that is because when you look at a fully mature, uh, observability practice, it, it, it kind of moves from left to right, right? It says, you know, there's alerting and monitoring, right?

Then there's, then there's, uh, tracing and logging, then there's actionable insights on what you find that happens. And then you have to layer on top of its security and the impact of the CICD pipeline. But, you know, to speak the numbers, right, we see that 55% of respondents are using IT automation and AI ops as, as critical, as crucial to part of the organization operational needs.

So 55% of the respondents say state that we also see that 40% of respondents are using, uh, 26 to 50% of their organization revenue directly tied to the development of custom applications. So this, the observability of those custom applications is important 'cause it's driving towards their revenue. And then 52% of respondents indicate that they're using six to 10 observability tools to collect data within their environment.

So that maturity curve is maturing not only in the context of using observability, right? Because six to 10 tools is a lot of tools, but also going more towards a plat platform approach, right? Understanding how to simplify to reduce the complexity and utilize it.

So you're having those, uh, effects on the organization. Do you think that these things are gonna converge? 'cause today we seem to talk about AIOps observability and monitoring and IT automation is kind of separate, distinct things.

And it seems like to your point, it is moving towards some sort of platform and a platform engineering construct. Yeah, absolutely. Uh, when we look at, uh, the organizations and how they're using these tools, right?

I think observability has kind of come out at the, at the, at the end of being the, the general category that's being describing the monitoring, alerting as well as AIOps, right? Those, that's a general category. Um, when we look at the convergence of these technologies, it really does depend on where the organization is, is putting those tool sets in place.

So when you're looking at, um, uh, utilizing IT organizations, they may refer to 'em more as an AI ops kind of play 'cause they're looking at the infrastructure. But more and more organizations are moving away. Actually, what we're recommending based on our research, and, and I'm more than happy to share those detailed findings, but we're recommending within the future and group that the alignment of observability tools is really more so at the DevOps level layer and the impact to the DevOps layer than the, uh, than the IT professionals themselves.

And the reason why is this, uh, a lot of folks would, would think about, uh, historically would think about the importance of an SLA, right? Oh, we have to have the equipment up and running and it has, you know, five nines and it's up and running, everything's great. But if the application's not running and the, and the servers and the storage are running, what good is the application?

It's not, it's not running. So you have to have more of an SLO, right? A service level objective that says, this is what the goal is for the organization and the DevOps team can really does control the, the release and the cadence of pushing out those applications.

So that's really where the critical, the criticality is of pushing out those applications. As we kind of get to this observability, I believe people have been monitoring applications for a long time, um, but it's generally a predefined set of metrics. Is it the shift to client server that kind of forces people down the observability path?

Because these things are a lot more complex, there's a lot more interdependencies, and it's just hard to figure out what's going on. Yeah, it's, it's good, it's a good point, right? When you look at the shift in why the, the observability is moving into one focus area versus another, it really does come down to, um, the importance of the business KPIs, right?

What, what's the goal of the business KPIs? On our last session that we had together, uh, you and I spoke about the releasing of applications, and I was referring to the fact that out of that study, we saw that, we saw that 24% of respondents are looking to release code on, um, on an hourly basis, yet only 8% of organizations can do so. And that comes down to the uptime in the, in the, in the reliability and actions of what's happening within the, uh, the, the ecosystem to push out that code.

So what we're seeing is, uh, 60% of this observability study, 60% of the respondents indicated that they are prioritizing real-time insights and applications and infrastructure environments that really meet the, uh, the performance commitments for the, for the, for the organization, right? And 62% of the respondents indicated that they're looking for internally developed, uh, employee experience applications to help derive that. So they're using these observability tools to help drive operational efficiencies internally to the organization, as well as use it for getting out that customer satisfaction.

Uh, I, you know, a couple other data points. I know this is kind of an analyst discussion, so I have a lot of data here, but, um, and you know, when you look at it, 70% of respondents indicated that they're, they're looking to, uh, reporting their IT monitoring and observability solutions is deployed in the cloud, right? So they're using a cloud-based kind of solution.

So when you look at where the solution is deployed and, and which teams are using those solutions, it really does come down to, uh, you know, that organizational de definition. So if that's the case, what is your sense of, we talked about this, uh, bell curve, but where are we generally on the bell curve? It seems like most organizations are far left, and we're still in the early stages of this, but there's so many observability platforms.

So is there enough room for everybody? Yeah. Um, so that bell curve is moving.

Maturity is definitely shifting towards the, the right side of that bell, right? Uh, we're seeing more and more organizations sh moving away from monitoring and alerting only to really taking, uh, tracing and logging and and security into, into play. We're seeing some, um, consolidation in the industry, right?

We're seeing across the, across the market, right? With the acquisition of Splunk from Cisco, we see that there's a strategic relationship between CrowdStrike and Chronosphere as an example. Uh, Katia was brought by K Chronosphere, right?

These are, these are solutions that are kind of moving. So is there, in my view of the market, there's definitely going to be, uh, an ecosystem where it's going to start either co you know, doing more strategic partnerships where they're locked and loaded together, or there's going to be a collapse and bring a lot of those, those companies together. Now, the reason why is, as, as I was saying, um, there is a shift, and when we looked at one of the major sections of our study, we talked about observability, how it empowers the insights to AI op integration.

Well, AI ops historically, as we talked about, was really focused more on the, the infrastructure side. There seems to be less of a, um, less of a desire to focus just on the infrastructure layer and more of a desire to focus on the, on the business logic and the application layer. So, but the, the not, I just wanna be clear, the infrastructure layer is absolutely critical.

If it's not running, then nothing's running, right? So it, it needs to be there, right? And, you know, so, but the, but the actions that take place have to happen at the business logic, because if you make your, uh, environment, um, seamless, for example, we see in this study that 94% of organizations are running on two or more clouds, okay?

If they're running on two or more clouds. And you need to have that ability to be transparent and dynamic across these different environments. The underlying infrastructure just needs to report up what's happening.

The application then can move across those infrastructures. So that's important to note there. You know, we see that 67% of organizations are using third party tools for monitoring, um, you know, cloud and, and log log monitoring as, as part of the and of their ecosystem.

And 81% are respondents are using, uh, observability tools when their organizations to look at IT ops, right? So they're using these tools to look at IT ops, but they're rolling it up into a general view. To come full circle back to your question, I used to make the comment that the, uh, observability practices that the, um, younger or observability practices within organizations we're really the storage admins, because all they did was collect a bunch of data and didn't do anything with it, right?

They just say, look, yeah, we have all these logs, but we don't wanna get rid of it 'cause we don't know what to do with it. And that's very rapidly changing into taking action on those, on those, uh, on those processes. Do you think we will rationalize a lot of the monitoring tools that we currently have?

'cause it seems like I gotta pay for the cost of the observability platform somehow. Yeah, that's a great point. Um, paying for the tool, it's, uh, you know, I, I, I think that there's definitely those stages of maturity.

So if you look at a maturity model and the phases of maturity, um, you know, there are different tools that address different tool sets. Clearly we see that six out of, six out of 10, um, uh, I'm sorry, six tools are being used for observability as is on the average, uh, as part of this report. Um, but what we're seeing in the future here is the future spending part of it is really to start harmonizing the environment, to have a, a common view.

And I, and dare I say, a single pane of glass, and I don't like that statement because there are no single pane of glass, but most observability tools are, are now either working within, as I mentioned, that partner ecosystem or being consolidated into other tool sets. So you have a, uh, a, a a way to harmonize that and, and rationalize that cost across different parts of the business. Um, you know, I, I think when we look at, uh, the reason why people are looking at observability tools and such is we see 41% of prioritizing improving visibility by optimizing their application performance.

It's a top criteria for businesses. That's really what they're looking for is optimizing that performance customer, uh, satisfaction. The experience that the customer has for use your application is critical.

Um, we also see 63% of respondents are using IT, monitoring and observability integrated into their DevOps teams, influencing the strategy of using the right tools. So you are asking about harmonizing it, 63% of the respondents of this survey indicated that their DevOps teams are dictating which tools they're going to be using to do that observability. So that's where the harmonization is coming into play.

Um, so there's a lot there to kind of, to unpack As you look at the report. What's your best advice to the IT leadership though? Because a lot of those folks are gonna be anxious about betting on a platform that might not be around in the future.

So, you know, how do I, what's my comfort level here in making a decision about which way to go? Yeah. Uh, so, you know, betting, betting on a platform, uh, I, I guess I would look at it in the context of this way.

Um, open source technologies are always a, a pseudo safe bet to kind of go after. If there's a, if there's an open source technology, the open source, uh, tool set will be there. So if you're using an open telemetry based product, uh, that seems to be a logical fit.

However, if you're not, um, there has to be a way to have that communications between the other tools in your ecosystem, right? But the recommendation we're giving, you know, based on this report is we're seeing, there's a significance of practices that really come across the organizations and they're trying to prioritize and align the IT monitor and their observability plans with DevOps principles. So I'm really recommending that those two things come together because of the decision making power of the DevOps team and what they're doing and the impact for the, basically the business KPIs of pushing out that code.

So prioritization of detection should be a top priority, right? And it should be the focus, um, within organizations between now and six to 12 months. So understanding the detection of what's happening, and then when you take those actionable insights, that's when monitoring should be used to improve operational efficiencies.

Uh, you know, so whatever, wherever the tool sets live, wherever the persona is that's using it, it really has to focus on that business, KPI of pushing out the right code, having operational efficiencies, and then aligning to the organizational objectives. That's really what we're looking at. And I'm less concerned personally about the tool sprawl as long as there's, uh, uh, integration across those ecosystems with the tool, uh, tools and the partnerships that work together.

We are, uh, of course all keenly interested in all things AI these days. It seems to me though, I'm gonna need observability to get to the data to understand what's going on in the environment, to expose that to the algorithms to get to the level of AI that I'm looking for. So is all of this kinda like a series of steps that we need to execute to get to the next level of automation?

Yeah, that, so AI is interesting. A, it, this is where, uh, AI fits into the play. Uh, when we look at that again, that bell, and we see that the far right is kind of more mature, there's fewer and fewer organizations today I would say that are using full automation, right?

Because it's just not, they're not ready to do that. Most people want a human in the loop, right? They want somebody in the loop to make it work.

The, um, the fact about AI is the amount of volume of data that's, that's out there, um, needs to be managed by, uh, systems and systems need to take actionable insights. And AI is really the, the tool. It's the difference between using a hand screwdriver and a power drill, right?

If you're using a hand screwdriver to look at it and sort, sort through all the logs and such, yeah, that's great. I mean, you can get there, but the reality of is what we see from our surveys and from our research is there are two thirds more applications being created today with a fraction of the resources just a few years ago. So there's no way you can put more hands on the problem to make it work, because no matter how many hands on it, you can't, you still can't lift it if the, if the thing's too big, right?

So, um, we're looking at the same challenge here. Uh, AI will be used and utilized in order to dissect the information, take those actionable insights, and then use those workflows to do the automation appropriately. All right, folks, you heard it here.

It's all about the tools and the metaphor from the tools. So there's an old saying that says you should measure it twice and cut once. So perhaps you should observe twice and execute once.

Hey, Paul, thanks for being on the show. Thank You. It's been Jo, been graded.

Thanks. All right, back to you guys in the studio.