Incident Resolution with Digitate’s ignio AI Agent

So let's start with the incident resolution agent. I will have to pause this from time to time, so bear with me. Uh, so the incident resolution agent, like I said, works in the logical reasoning model where it knows about situations in a closed loop, closing the loop through automation in case it is not possible.

It works in the en analogical and assisted model to augment resolutions, handling exceptions, working with experts. So let's take an example, right? So I come in, uh, and I ask this question while I was away, what all did help, and the agent provides me a succinct summary of, let's say I was away for two hours.

I had to pick up or drop someone. Uh, so, uh, so some 1300 odd incidents have happened in that time alerts, and I was able to deal with 70, 80% of those as noise. And then I, I had to register certain number of incidents.

Some of those were effectively handled by io, by myself. The agent fully resolved, automatically triaged, and there are a few waiting for you to act on. So that's really the summary I get.

And now I ask the question, okay, what needs my attention? So it gives me a prioritized list of incidents that I should be paying attention to. And these priorities decided by not just the priority associated with that incident, but the impact.

Because potentially if the business blueprint also captures business applications and business functions, then it is possible to derive that impact. And then it goes into that. Then I can ask the question.

Okay? Uh, great. And you told me that there were some incidents that were auto resolved.

So what about those? So I asked that question, then it gives me a list of issues that were automatically triaged and resolved by this AI agent. Okay, interesting.

So I want to see what happens to those. Uh, so I ask further questions related to that in order to figure this out. This is the assess phase, if you remember.

Uh, not the result. So how do question, so how do I provide the business context, the business impact, um, into the system, Right? Um, so many a times we have seen, uh, this information is available somewhere it is running.

So it's there, it just there in informal data sources. I mean, enterprises try to create this architecture blueprint, but it barely pipe dream. Uh, but this information does exist somewhere.

The, the tact is in getting that into a formal centralized enterprise blueprint. And we have been able to do that fairly effectively. It doesn't completely cover all business functions, all areas, but people definitely have understanding of the critical business flows, critical business functions, and their mapping to business applications.

Yeah. So, so is there, is there a module? Is there someplace where I kind of describe My, you can do that environment or, so we provide a a, I mean, if you want to fill up templates, that's one way.

If you already have this information in some format somewhere, we can, uh, extract from it and populate this. Okay? So while, uh, the, I was aware having dropping somebody, uh, Ignio has auto resolved, let's say 54% of issues.

And now I'm curious how that happens. So let's take a look. So, uh, IO performs this automated root cause analysis and auto resolution by traversing, like I talked about influencer hierarchy of the issue, right?

So it identifies from the enterprise blueprint what all could lead to this issue, and then performs a real time health check on those, compares that with the expected normal behavior baseline. And that's how it localizes probable causes, root causes price to convert those into prescriptive fixes. And that's how it goes about it.

So let me ask this question that, okay? Uh, tell me details of the resolved, auto resolved incident. So it, it's going to pick one of them, and now I will go into details of that and find out.

So here you can see that, okay, this is the incident and what IO has done is this is the hierarchy that I was talking about, right? For that. So it's a issue related to an application not being reachable.

So this application uses this particular Oracle, which runs on Red Hat seven, red hat eight, and there are a bunch of matrix associated with that, which are collected through realtime health checks on each of these layers, comparing that with expected normal behavior. And that's how, uh, the logical reasoning part, the first column, deduces, what are the anomalies? So if you go down this hierarchy, it is going to show you where the anomalies are, right?

So for example, here, one of the services on the application server was down. If you now, uh, take a step back and imagine a typical incident bridge, there is a priority one incident, the incident manager calls up everybody, uh, and everybody joins the first 10 minutes, they fight that, uh, it's not my problem, it's your problem. It's almost like Tower of Babel, if you will.

They don't understand each other's language. Everybody blames the network guy. All of that happens.

They settle down, everybody Blames the network guy. Feel that, feel that that's right. I know I resemble that remark, Right?

And then you brave it out, and then everybody does their own health checks. This is exactly what is happening through the machine. Everybody's performing health checks in silos instead, now, machine is performing those health checks.

Everybody then reluctantly reports that, oh, I seem to have a problem. There is no shame here. So the machine just reports that, okay, the application server layer seems to have an issue, and now the machine has models built in to convert that issue into prescriptive fixes.

This is straightforward. It just restart the service, right? And then it will apply that fix.

It'll reperform health checks on the entire stack. Again, again, this is typically not done in an incident bridge, it's missed out, but it does that diligently, it'll re make sure that all the issue is fixed. Now it has a multitude of this information, what it has done, going to put that in the ticket as actions instead of just saying done, resolved, things like that.

And so you, you are playing like a good citizen here. And that's, that's what happens, right? And it it has been, uh, applied this fix and it has been resolved like it says on the screen.

Can we, so can we just parse this briefly, right? So if I look at your Tomcat error, uh, should I read the log from the bottom up? Sure.

Out of memory error discovered, CPU threshold values been crossed, a memory threshold utilization value has been crossed. No. So, okay, so it's reverse.

Okay. These are all faults. The faults are not found.

Okay? So the only fault that is found is the service down. Okay?

So it's a little bit convolute. Okay? That's why I asked the question.

Yeah. Okay. So, so are those three items below the server downline?

They're fine. Those fault don't exist, Then why are they being reported on it? No, no.

This is evidence of the hierarchical I So once you, once the servers back up, everything else doesn't Yeah, exactly. Yeah. Okay.

But, but tho you can interpret those as contributors to the down server. Down Possible contributors. Yeah, possible contributors.

Okay. So Keith, now the advisor bench a little bit connecting the earlier conversation to this, where is the, how, how great is the sophistication of the agent? So you mentioned SAP hana, uh, before, uh, if I have ASAP landscape, I have my SAP applications, all of my bolt on apps to the application, I've mapped out this crazy dependency via my c uh, uh, uh, my ServiceNow perspective.

If I've, I've mapped out dependencies. So if a business calls me and says a vendor has not been paid one, I should have already known that, correct? And then two, it's not as complicated as a service, a single service being down.

Will this help me solve that direct business challenge? Or am I looking at it more at a up down individual service perspective? No.

So great question. Again, it'll absolutely help you solve that problem. I'll take another example.

Sorry, I'm stealing your hundred words. So, uh, there are issues related, I'll take a retail example. More, more comfort.

So I go and I am trying to check out something and it's not on file. That item that I'm trying to buy is not on file either the price is not on the file, or the item itself is not in the master data. Common occurrence, when you try to buy something in a store, this issue needs to be triaged.

This issue happens across a multitude of bad jobs, finally into the SAP system, right? So that's the dependency that there is this, this application depends on these set of files. This application also depends on the set of bad jobs completing on time.

The dependency hierarchy is not always the straightforward vertical stack. It could also be fairly complicated. It could be middleware foul wasn't transferred exactly one directly to another, or Simply there is no space in the queue wait time has increased.

So it's not, I mean, just giving this as an example because it's easier to sort of explain vertical stack, but there is no restriction like that. Okay. Shall I?

Okay. Uh, uh, sorry, go back. Um, the kind of that hierarchy you said, so like this, this definitely seems like there's, it's trying to understand like within this, whether, you know, so rel like the os what's status of it, or it's like these are the, the types of things that yes, could be wrong with this, this survey.

So is that just kind of auto discovered or are these things that you kind of have know that you've kind of provided some knowledge about? Um, yes. For these particular types of Applications, yes, there is an out of box knowledge.

That's what I talked about. So it, the knowledge is not static. So, but it, we do provide out of box knowledge, okay?

Over a period of time it gets augmented For my specific environment. Yes. Because that information we don't have, right?

So it needs to happen in your environment. So the, the firewall licenses expired, which caused blah, blah, blah. It now has that in its history.

It'll figure It out. Yes. I mean, it, it's, it's not instantaneous it Needs, but yeah, it knows that it had that event once before.

And to at least look at that first Is, is there a way to help help, uh, move that along? Can I, can I, how can I provide that information? So yes, so that it doesn't have, not everything's discoverable, Of course.

Yeah. There are many things which are not discoverable. Yeah.

So, so how, how do I go about That? So that's what, so again, uh, so we have a model it call it, IT operations model, enterprise IT model. So, uh, that model gets instantiated for every enterprise.

And that instantiation process is where we tap into multiple data sources and instantiate. So if you want to provide that information, that's one place from structural dependencies, behavioral dependencies. If you want to provide information about, we do things differently here and we don't restart a service like this.

Yeah. But we take five steps and then restart it. We have an operational model also that comes out of box prepackaged, uh, standard atomic checks, standard atomic fixes and things like that.

Those also are customizable. If so, there is a, in your studio available where you can inherit that and, uh, modify it over a product. That's what happens through learning, like you said.

But there is a way to preempt that. Okay? So, okay, so the, so the, this was a easy example that, okay, deterministically model was able to deduce root causes and fixes, but that's not often.

So in scenarios where it is not possible, what happens? So let's take an example of augmented resolutions. This is now ignio tried, I couldn't figure it out.

Now it is coming to my queue as an SR and I need to fix it, right? So I'm going to now ask this question that, is there anything that needs my attention? Uh, these are the issues that have come to me.

There are a bunch of issues naturally that I need to deal with. So I will pick the first one and see what is going on. So I, let's say I pick and this issue that this particular application is slow, and now I need to collaboratively figure this out, right?

So I ask, uh, the agent that, okay, the agent is going to provide me a, uh, historic context, situational context that I tried doing all of this right here, and this is what I have figured out, but it's clearly not deterministic enough for me to take autonomous actions. But the agent, the agent has populated, the ticket agent has populated this information, not just this historic information. Also, it'll populate because that way, sorry, I'm going to make informed decisions.

So historically, this issue was seen 15 times in last 30 days or three months. And every time this happened, these were the common causes and fixes, right? So, so that I have situational context these days, it is called episodic memory.

Uh, I have historic context, and now I am hopefully going to make judicious decisions. So I decide, okay, I get the hang of what is going on. So let me now help the agent learn a little bit more about handling this exception.

So I now trigger what we call as a resolution assistant, and uh, then I give it some instruction. So I tell that, okay, based on what I have seen so far, I believe that disc space seems to be the issue and do something about it. So check the disc space, and that should take care of it.

The issue seems to be that this is now my tacit reduction. So now IO agent is now going to come back and tell me that, oh, this is a good suggestion, but by the time you, by the way, you should also look at some of the old lock files that are there, because though I couldn't deduce it at that time, this might help you figure this out quickly. Then I confirm, okay, go ahead and also do the lock files for me files, and should I make this utilization part of an ongoing investigation going forward?

Of course, I say confirm this. So, so for this interaction here, where the, uh, the question is, um, should we add steps, delete older logs, that implies that I'm trying to free up memory by deleting, Correct? Because the first instruction that was given is the disc space seems to be the issue.

Yeah. As an SRE, that's what I pointed. So understood.

I would just like for the less literate, you know, um, administrator check the disc, health disc health isn't the same as space available. Absolutely. So, quick question on this as well, and you might have already addressed this, and I apologize if my mind's in space right now.

Uh, when, when you look at these responses, you know, I see privilege escalations all over the place. I see a lot of security problems because, you know, in a system you'd be pseudo to do remove logs or you'd be trying to do this. So absolutely, I'm assuming based on the persona that is accessing these things yes, and interacting, it'll acquire those.

There is a role base that is there, and then the agent itself would have to go through an escalation process in the back in order to be able to do, because I know you would ask about security anyway, right? Agent itself is almost like a part of your team. Yeah.

So the agent also gets authenticated, authorized then. Yeah, I mean, I, we just wanna avoid the whole cursor suddenly RMRF, your entire volume a hundred, what the heck? Just So it doesn't take, uh, so it, it knows its boundaries.

You hope, you hope. Yeah, I've, yeah. Uh, terminate is a way of shutting something down.

Um, but Not it's A dump Sweat bill minus nine. Is there, is there a way to get it to show more context here, the agent Ronda, if you wanted to investigate further details, Yes, I will come to that. Okay.

Just gimme five seconds. So, uh, so I go ahead and now, now the agent is going to tell me, based on this conversation, hopefully it is going to tell me this. Now, this is where I, if you see the agent is going to provide me the procedure that has been collaboratively discussed, it is going to give me reasoning behind each of the decisions that has been made.

Pros and cons of this, right? This is going to require a downtime, for example. Yeah, yeah, Yeah.

And it is going to give assumptions that have been made that I am assuming that there is a network folder to which I have access, for example, right? And it is going to gimme an automation code for doing this so that I eliminate the human error. And that's how you get into details.

Of course, the, the expert can validate the procedure, validate the assumptions, you don't have to just believe it like that, right? So you go through this step. Once you are satisfied, then you can look at the code.

You want to just test it out. There is a way to do that directly. Don't play with the fire, if you will.

And once you are satisfied, you can then apply that. Okay? That's really the idea.

But again, the idea here is not, this level of augmentation is not suitable for level one engineers. You just know the easy button, so you know, it's so great to push and then magic smoke happens, right? So this requires facet knowledge, understanding of the problem.

And by, uh, by definition, because this issue has escalated, it, it has gone to a person with that level of understanding and skill, Either there Malcolm burden here and remote, uh, attendee MNB networks. Um, is it possible to, uh, integrate with IDSM, uh, in order like, so when you get a proposed fix, you can effectively export or import that into an IDSM uh, yes solution as part of like a change management process. Absolutely.

That's what we have adapters for. Most standard ITSMs. But there is a way to build, okay?

So, uh, business SLA predictions, I talked about bad jobs. It's an interesting problem. Let, let me show some demonstrations related to that, right?

So the batch thing is we help predict business processes. We help detect resolve anomalies autonomously. Uh, the agent helps predict SLA violations ahead of time and so on and so forth.

So let's take an example. Similar approach here is a summary of what is going on. This is how my business processes are running.

These are the anomalies that are observed today. Uh, these are the SLAs that are met today. These are the SLAs that have not been met today, and these are the SLAs that need your attention immediately.

So then I have a path ahead. What, what do I need to do before I have lunch? So I ask, ask it a question that are there any specific business functions that are not going to finish on time today?

And it's an important question because otherwise somebody will call me. So then it gives me information and provides me, okay, this is the SLA, but this is the deviation that is going to happen. And then I can pick from it one specific business function and see if I can generate early warnings of sla.

So that's the second function of, uh, this agent. Um, so let's take, take a look at it. It does it through, uh, generalization.

So for example, order management stream, uh, one of the jobs it is now saying that it is likely to miss, and it gives an example of what is happening, right? So there is this GaN chart like view that you see. The last one is the critical business SLA, which is predicted to miss.

And what are the different possible fixes that have to go along with it, right? Because the, it's not a function of one time prediction, but every time something happens in that batch stream, the prediction is updated near real time so that it stays current. And as a result, the root cause may be some, some job which is upstream five level.

So it, it may be some file which is missing. So there are a bunch of, uh, fixes that the agent suggest like it is shown here, right? This is a little bit involved because it involves many a times data fixes, which are not easy to blueprint and automatically figure out.

So the human, uh, expert has to come in and validate these fixes. Uh, there is reinforcement learning happening so that whatever the person likes or dislikes, automatically the choices are going to be updated. And as a result, this is, this is going to give you three hours, four hours ahead of time.

Notice that this particular bad job is going to finish late and you have three hours to do something scramble. Uh, so that you don't let that, let that happen. That's really the idea behind it.

Uh, similarly, now let's take a look at reactive resolutions. I talked about failures of jobs and uh, delays of jobs, right? So also leverages similar mechanism that we saw an example earlier, Tomcat database.

Uh, so again, same question. Are there any failures that have been, uh, auto resolved by io? Then you pick one of them, go to it and find out what has happened, right?

So, uh, this now, uh, this particular ignorance has performed all of these checks and these are the sequence of activities it has performed to deduce the root cause of the problem. Health check. Suggest a fix and apply that fix right?

Now let's take a look at it. So if you just pay attention, right? One of the anomalies that INU identified was a specific file that is supposed to be there for a specific job was not available.

And this file, it knows that it is created by another job. Uh, and that means that job did not create that file. So simple fix could be rerun that job, create that file, and move on.

So Ignor will try and do that, but at the same time, it also detects another anomaly in this hierarchy that, uh, that job started and that file got processed, but another anomaly is detected, uh, hopefully yes, that another file seems to be almost zero bytes compared to typically it is of the order of this range of kilobytes. And it's a problem. This file Ignio knows, the agent knows that it is associated with another bad job in the hierarchy dependency, and that also needs to be retriggered.

Once both of these issues are solved, the main job needs to be re-triggered, and that's how the issue gets resolved. That's really the idea behind automated resolution is slightly different, um, than the, uh, traditional application and business function way. Now let's, oh, I went all the way back.

Sorry. Yeah, so this is another, uh, example. Many a times an issue happens and there is a immediate impact that the, you don't fix this issue in the next 30 minutes.

Only one SLA is going to be affected downstream, but if you don't fix it for two hours, there is going to be further downstream impact. And INU is now able to project this so that the importance sense of urgency then gets communicated to the person who is doing it. So now, same, same drill.

It knows this GaN chart with jobs have resulted in this failure. Now, uh, conversational assist, because it is not possible to fix this deterministically, uh, the SRE can ask or the batch architect can ask that, okay, suggest me a fix. The SOP shown the SRE says that, no, no, no, no, I don't like it.

So, uh, add one more file as a dependency in this, uh, to be checked and the fixed case augmented, SOP, you can also validate that it gives assumptions, it gives what it has done. Finally, it gives code so that you don't have to manually do all of these procedures. It gives assumption that the file path needs to be accessible.

And that's how the, uh, drill continues. The last but not the least, right? This, so far, everything we have seen is predictive and reactive.

But what is proactive? So because Ignio has, uh, the agent has visibility into so many things that are happening, it is also able to provide continuous improvement recommendations so that these issues can be sorted out. So one such example of that is this, right?

There are 16. So there are totally out of 147 SLAs that typically get missed. 16 seems to be dominating.

So naturally from a problem management point of view, that is where you focus attention and then it further breaks it down saying that, uh, one, one of the 16 is potential risk. What SLAs? I need my attention, okay?

There are nine SLA jobs across seven business units that covers 57% of your grief that happens during the month. So if you focus on that, you will be better off next month. You don't have to boil the ocean.

And there is evidence given on why those, again, this is a learning based on insights and observations and data that it has, and there are things of this kind, right? What are the dominant SLA violations? Same example.

There may be examples of reverse that. What are the dominant jobs, which are the, oops, Sorry. So these jobs are typically resulting in some other businesses all getting violated.

So thou shall pay attention to these jobs and get them sorted out. So it is, it is at a slightly problem management level kind of insights for the transformation or continuous improvement. That's really the idea.

Uh, proactive problem management is similar. Uh, so here the idea is a two-pronged approach. One is all recurring diagnosable issues are presented to the problem managers and all recurring issues that are not diagnosable, but are predictable are also presented.

So it is not a simple statistical analysis of, oh, this issue keeps on happening all the time. So you should promote that. And I'll talk about what do I mean by alert signatures.

So let me, uh, skip some of this a little bit. So here are some examples. So here are the issues that are predictable but not diagnosable the predictions of, of two kind.

But there are issues that have temporal patterns that like a clockwork, it keeps on repeating at that hour of the day on that day of the week, on that week of the year. And there are issues that have preconditions. When A and B happens, only then the C issue, uh, C issue happens.

So both kind of recommendations are given by this agent and then one can act on it proactively. The other one is interesting, what we call a ticketless. Here, the idea is not figuring out statistical recurring issue signatures, but it is mining anomalies across a sequence that is leading up to it.

Can, Can I ask a question about the previous bit? How much of the policy here, sorry, Calvin Henricks parked from six feet up. How much of this is policies you fed to the agents?

Like how is it understanding these dependencies between the jobs, the processes? So It hooks up to the scheduler, You, which, which I mean what fed my business process into the scheduler? I I, so the, I manually program code codes or, So typically, uh, enterprises have this AutoSys control m or some other bad job schedulers, and they have this configured that this is my business process, this is my job stream, and there are jobs.

So you work with some number of enterprise systems like this. We have covered most common Ones. Okay.

What about tools like an airflow or a more open source Yes. Tool? Yes, we can, we can, like I said, we have covered You can or you do Specific tool.

I will have to check. Yeah, I'm saying all job schedulers. We have, We Do, we do the other kind of signatures are recurring diagnosable issues.

And let me explain that through an example. So for example, this scenario, right? Uh, there is an application slow event that happens three minutes before that application event happens.

The database response time is seen to go beyond the high limit. Nine minutes before that happens, the exchange message queue gets full 12 minutes before that happens, a specific, this gets full 29 minute minutes before that happens, high IO is seen and a backup gets started, right? So this is not a vertical stack anomaly like we saw in the earlier example, but it's a sequence anomaly.

And IO is able to mine this through sequence mining and many other kind of techniques present the evidence. This happens a hundred times in last eight months. And you can eliminate by fixing this.

And again, there is a way to collaborate, uh, with the agent, figure out a fix, give some suggestions. Uh, once you figure that out, there is a way to directly raise a problem ticket with suggested signature as well as mind anomaly sequence so that whoever is approving the change is making a judicious decision about it. And that's how we deal with it.

Now let me, uh, change the track a little bit. Uh, we are agent tech and we have a bunch of agents that are, but there are of course, other technologies and they have agents. So how do we deal with that situation, right?

So they, there needs a multi-agent orchestration in this setup. And let me just give you an example of that. So in this example, I'm going to talk about an email agent, a conversational assist agent, a ticketing system agent, all connecting through their MCP servers.

And then we have an orchestration agent, and IO is one of those technology agents, and it is doing it to a protocol for that communication. So let's take a look at this example. So, uh, the flow is going to be like this.

Somebody's going to send an email as an incident. The email agent will pick it up, uh, pass it on to the orchestrator. Orchestrator will try to register a ticket in the ticketing system.

Uh, of course there is going to be, uh, not sufficient information to create that ticket. So it'll then invoke a conversational agent, um, slack in this case, talk to the appropriate service desk team, get that information filled in, then register the incident, assign it to IO for autonomous resolution, and then IO will go through that process that we just saw. Let's take a look.

The orchestrator is checking for email now that, yeah, I have an issue and it comes by email. Now the orchestrator will try and register an issue with that much information. Uh, of course it's not going to work.

So the message will come, okay, something is not sufficient. So it'll then get to the sent to the IO agent, as you can see here. And IO agent will give information about what's what is required for it to act.

Now this information gets passed on to the conversational assist agent Slack. In this case, specific user interacts and gets that information populated through interactions. One sufficient information for that incident is available for it to be actionable.

Then the orchestrator picks it up, registers an incident ticket in the ITSM, um, assigns it to the IO agent. So this is the incident ticket that gets registered now IO agent starts working on it, it is going to try and auto triage the first column of model-based reasoning. Um, so let's take a look at it.

So it is trying to figure this out. Influencer hierarchy and actions and pulse and fixes, it did not work. So in that case, it is now going to pass the control over to an expert resolver.

An email is triggered from the IO agent with the link to go to that conversational assistant, build up that fix. Now I, I, as a resolver get that link. I click on it, then I open it, it takes me back to the conversational assist agent, provides me all this context.

Then I create a conversational procedures, figure out the SOP, create the automation, resolve the issue. Then once it gets resolved, the notification goes back saying issue is resolved, ticketing ticket gets updated and the person who has sent the email in the first place, user reported incident, gets a notification saying that your issue is resolved. So the the point to be demonstrated here is in an ecosystem of agents, we have the ability to orchestrate seamlessly, even if there are third party technology agents that are in the play.

Because that we believe we can't be the only agents. 'cause there are going to be other agents in the mix.