Context Is the Key to Cutting IT Alert Noise

Hello, and we're back at the ManageEngine User Conference in Orlando, and we're having a chat about how IT operations is going to evolve with my new friend, Gao Sinker. Yep. Hey.

Hi, Mike. Thanks for having me. Pleasure.

I think one of the issues that everybody always deals with in IT is there's far more noise than signal. Correct. Yeah.

And how are we going to sort all that and get more of the signal elevated so we're not wasting all the time on the noise? Because that seems to be the thing that really kind of at the end of the day creates all the drudgery for people. Exactly.

Correct. So are we on the cusp of maybe changing that equation? Yeah.

It's been the case. So right from the beginning, the observability market or rather the product that we sell, there are too many alarms to handle it. So as a network admin or as a server admin, it will be cumbersome to go through all the tickets or all the alarms and then find out what is the root cause.

So that is a challenge right from the beginning. Even so we are in the LLM era, but even before that, we had the same problem. So we had some criteria-based filtering so that it'll try to reduce the noise and then try to filter out the alarms.

But when you have observability, we have tons and tons of data coming from the telemetry. So it is not humanly possible to go through everything and then filter out the exact problem. So even before the LLM, we had this machine learning.

Mm-hmm. So machine learning, with the help of machine learning and with the help of the observability, the discovery mechanisms that we have inside the observability tools. So we know the context.

Say, for example, if you have an application and that application will have multiple components to run it. It'll have a DB, it'll have a middleware, it'll have network devices. So all these things should be combined, only then an application can be served.

So that combination is called context. So once we have the context, it'll be easy to narrow down the problem and instead of going through all the alarms, you can try to identify the incident. I'll explain you.

So if you have that application, so all those components are interconnected, and I know the context. So if so many alarms are coming from it, with the help of machine learning, I can understand this problem happened because of this particular issue. So I can identify the sequence of the events based on machine learning.

I can understand this is the pattern that it is going to happen. So based on it, I can also understand the root cause of it. So once I know the root cause, in future, I'll be able to predict as well.

Mm-hmm. So even before the problem happens, I can predict that this problem will come in. So it's a combination of machine learning, and then it also helps in event correlation.

That helps you to identify the exact problem instead of thousands of alarms. Mm-hmm. And the further development on top of it is the prevention of the problems.

Right. And the issue is, otherwise, I got to go investigate each one of these false positives to figure out what- Yeah ... the root cause really is.

Yeah. And they're not necessarily false positives per se, but they're more symptoms than they are cause. Correct.

So, do you think at the end of the day, if we have this capability, is my meantime to remediation going to drop considerably? Is this going to get better? Exactly.

That is one of the important use case of having A. So instead of going through thousands of alarms, it investigates itself. And it gives the right problem.

So this is the incident that you need to look into it. Don't look through all the alarms. So that investigation part is already done by the system, so it obviously reduces the MTTR.

How will the way that we keep track of incidents and the records change as this evolves? Is that going to be more embedded into a workflow or... I guess I'm trying to understand like how is that whole operations process going to evolve, do you think?

So it is the same as I explained. So the main thing is the context building. So as an observability tool, we have to understand how the components are interconnected, how they're related to each other.

So once we have this information, that is the root cause, or that will be the base for the further A enhancements that are going to happen. Either the event correlation or prediction or capacity planning. So what will happen in future?

All these things will happen, can be done with the help of the context that we build. Mm-hmm. So I would say that would be the base.

From there, we have to pick the other things. If that's the case, then is the entire process going to become a lot more proactive? Because if you look at IT today, it's always been very reactive.

Something happened- Correct ... and then we responded. So can we be proactive and see the signals and get in front of something before the actual event ever occurs or- Exactly ...

occurs or fix it maybe before anybody notices that there's an issue? Exactly, yeah. So as I said before, we have the machine learning, which identifies the anomalies.

So once we have the sequence of actions, say for example, a business application during Christmas time, more and more users will log in, and then it'll create more traffic, and it'll create more load. So if I understand that every year around Christmas time, around December 20th to December 28th, there will be a lot of traffic going through it. So we understand the pattern.

So once we have the pattern in it, we can predict something in future as well. So this is an example of one holiday use case. But the same can be applied for others as well.

So once we have the sequence, once we know what is going to happen, then the system will automatically predict what will happen, what outage will happen. Will there be any slowness? Will the end users go through any slowness or other issues?

So the system can predict. Mm-hmm. And at the core of that, you mentioned machine learning a few times.

There's going to be multiple types of AI models ultimately that we wind up using. It'll be a mix of machine learning, which is a little more deterministic. Yeah.

And then GenAI is a little more probabilistic. Yeah. One will surface an issue that the other one will essentially investigate.

Is that one way to think about it? Yeah. So from the observability perspective, I think Raj should have told in the morning, the system of records.

We need to have a proper system of records. I would say the network source of truth. So these things should be ready.

These things should be properly available inside the system. So once we have this, machine learning will solve a lot of problems. And in case if you have a very powerful LLMs outside, and if you're going to share the system of records along with the metrics or the anomalies that are coming from each and every component, if you share it with the LLM, because LLM could be more powerful because that is running in a very big data centers.

So that can analyze and give more accurate answers. But for 80 to 90 percentage of the issues, a good machine learning could also solve a lot of common use cases. Mm-hmm.

When it comes to observability, how automated will things get? Because am I going to get to the point where the system observes something and then kicks off like a playbook or something? Or am I as a human going to step in the middle to evaluate that?

I guess I'm trying to figure out how smart can smart get. Yeah. So, the time is evolving.

So till now, based on my interactions with a lot of my customers, so they expect someone to be in the middle. But there are also a few set of customers, I don't want to have a man in the middle. So you should be able to understand where the exact problem is, and you should be able to understand the exact remediation as well.

They expect the systems to understand this and execute the remediations as well. So even they're ready to touch the network device and make the configurations. So till maybe a couple of years ago, they wouldn't have allowed a system, an agent, or an automated system to go and touch the device.

But right now, if you're 100% sure that that is a problem, and if you know that that will be the answer, go and execute yourself. They're allowing the system to run their workflow without having man in the middle. As we engage in that, do I need to know more about the telemetry data itself and the logs or the traces or whatever it is I'm collecting?

That could get massive. Could. How do I streamline that so I'm not overwhelmed with the data itself and then figuring out where to store it?

Yeah. So there is a new market evolving, telemetry pipeline, I think that is the name. I'm not fully into it.

But there is a telemetry pipeline which can control what data can go in. Before the telemetry pipeline comes in, there is also some options to control what data should go inside. Do I need to send everything into the DB or the data lake, or is there an option to control some criteria-based mechanism where it can control the amount of data that goes into the system?

So as of now, majority of the customers are using the criteria-based stuff, but there is a market evolving like a telemetry pipeline. So what's your best advice to folks about how to get ready for all of this or how to set up their own operations to get to this ultimate outcome? Because I think a lot of folks are all over the place on this journey.

So based on what you've seen so far, what should people be doing first? Yeah. I would say, I would advise or recommend the admins to have a proper system of records, to have a proper infrastructure where they can easily identify the relationship between the objects or the components.

So have a proper discovery mechanisms, and configure the system to get the right signals. There is no use of getting unwanted signals and then observing and then doing analysis on top of it. You should be able to get the right signals.

Sometimes, if you're not configuring it properly, you may not get the exact signal that you want, that might help you in preventing the future problems. So make sure that you get everything. For that, you need to have a proper system of records.

I would say the network source of truth, use that as a base and then bring in other ML and LLMs on top of it. So that will reduce the workload, and the accuracy will also be better. Do you think as that evolves, we're going to be using multiple AI models and LLMs for different use cases, and it'll be a variety versus, I think a lot of folks today are standardizing or just saying we're going to use one, but maybe that one is going to be the best for that capability for about an hour.

Yeah. So how will that evolve in your mind? Yeah, it will definitely evolve.

So even based on our interactions with the customers, what we had initially was to have an integration with the LLM. So one LLM that will be used for different use cases, say for example writing a script or having a capacity predictions. But what customers wanted was, for writing a script, I want to use a different LLM.

For capacity planning, I want to use a different LLM. So that is how the market will evolve. Based on the use cases, we'll be having multiple LLMs used in different use cases.

All right. Well, folks, I think you've seen it here, but this is no longer your grandfather's idea of IT operations, that's for sure. Hey, thanks a million.

Thanks, Mike. All right. And we'll be back in a minute.