cPacket Service Assurance: MTTR Reduction

Good morning. My name's Eric ine. I am, uh, field CTO for C Packett.

And we're gonna go through another use case, uh, for network field day. Uh, this one is a little bit, um, different, but you know, when you get a general theme around MTTR, right? MTTR is different teams working together to solve problems.

In this case, um, we're reacting to a customer who's unhappy. Uh, it could be Tom calling the help desk asking where the, what's going on with the internet service. It could be, you know, us wanting to understand the quality of that, uh, application, right?

That's, that's a, it's a healthcare application or it's a business application that drives my financial trading. Uh, and so I'm, if I'm a trader and I'm expecting a market data feed, I'm going, Hey, what's going on? And so in this example, we're thinking about, uh, the, the user is now concerned and we are acting as network operators in a reactive state.

We're thinking about how we're gonna go and get information, how we're gonna look through the different logs and different data sets. But if we look at it initially, it looks fine, right? It, we, we, everything looks good.

This goes back to whose, whose responsibility is it? And so is there a way that we can respond better? So Ron's gonna walk you through this use case.

Thank you, Eric. Yeah. So I think the way to think about this one is the next level of user interaction, right?

There is something that is happening in real time. You want an alert, you're going to Slack. It's relatively simple to know what's going on.

You have cases where you get the phone call. And this example is taken from a real, uh, life example of one of our customers, a big bank that has a relatively complex network. And the call that the network operation received was that people in a very specific branch were failing to access their remote desktop between eight and nine.

And then it worked, and then it didn't work again the day after. So if you look at the network, that's actual path or relatively simplification of the path that it was taking. It was from a branch going to a colo, from a colo was going to a data center in the data center.

It was going through multiple hops to get to the remote desktop application, right? So, uh, trying and understand where this thing break took some time. Uh, and at the end of the day, what they discovered was that, uh, a box that was, uh, not supposed to be doing anything special, uh, when, uh, when acceleration or when acceleration, uh, device was adding 300 millisecond of latency around eight and was causing, uh, people to basically fail the authentication.

So it actually wasn't even a video, a remote desktop issue, it was authentication and it was along this path. So, you know, if you don't have monitoring points everywhere, uh, then you know, what do you do? Are you start replacing, you know, box by box or how are you gonna measure it?

So, uh, that was kind of the use case or that was the actual story that happened. And, you know, they, they did have C packet and, you know, they went through and, uh, basically found where the, uh, latency, so what they, when they started, they knew that at some point the, the latency is getting to 300 millisecond at the branch. But it took them, it took them a little while to figure out which of the devices along the path was adding the 300 millisecond.

What we want to show now is to say, okay, fine. You, you know, you can do that and we have all the dashboards and you can go to the dashboard one by one and figure out what, uh, where things are not working. But one of the things that we're seeing now with the introduction of AI is really the change in the user experience, right?

People are doing it in a very different way, or at least it opens the tool to do it in a very different way, right? So we're gonna double click a little bit on how we can, how we are seeing these use cases evolve once you add the ability to chat with the system, right? So it's not necessarily AI as, uh, so, so you know, we are using these two terminologies, right?

We're using AI for network observability, right? Which is, can I use AI to understand what's wrong? And we'll talk later about network for observability.

So what we're really gonna spend time is yes, AI for network observability, we're gonna see how AI allows me to understand what is wrong, but also how AI helps me with a workflow or a natural language processing workflow. So what we did is actually recorded, uh, the video and the reason we recorded it, it's, it's gonna, we are limit a little bit limited on time, and it did take some time to do that. So what we've done is basically, uh, accelerated some of the session.

So what we're seeing here, sorry, is the network operation guy received a call and he said, okay, I understand I cannot go to the remote desktop. We are using, we use here a different service just, uh, for our, our simulation called Acme. And what it asked what the, what the user asked was, is there any insight?

Is there anything you know about this service that is not working well? Right? So it went, the LLM went that went there, went to our database of all the insights and said, yes, I do see things that are not quite right with this service.

So then I went, the user went there and asked, okay, can you give me a little more details around the exact, because I know that it's happening now, can you give, you, give me insights that are relevant to now? Right? And again, the engine is going back.

We have an MCP that sits on top of all of us, and he's saying, yes, I do see there are something that is, uh, a, a client performance issues. There are jumbo frames. That's probably not the reason.

So what the user ask is, okay, now I kind of understand what the insights are. Can you visually show me the difference between what I'm seeing right now versus the baseline? Right?

So we built a baseline so we know how the system should behave around Tuesday at eight, where the, I have a question For you. Yeah. Um, this is Aaron here.

Um, that baseline, you said you built it, did you build it yourself or is it just gonna collect that over time? Or do you say, listen for two weeks and then tell me what it looks like and that's the baseline? How does that work?

Yeah, that's a great question. So the way that our AI for observability works is we bring in all the metrics from all the devices to a storage, big storage biggest S3 bucket, or big object store bucket. And then we have a machine line machine learning AI pipeline that runs on top of that.

And again, we're using different, uh, models, but one of the models is to create a baseline for, uh, the text into account, uh, every service, every, um, time of day, day of week, um, every metric that we collect, we collect about 50 or so metrics per, uh, re transmission. So today the baseline is done based on the first few weeks that it runs, right? As long as the idea is, because it's unsupervised learning, we're saying, well, if the first few weeks you didn't have any major incident, let's use that as a good baseline, right?

If there are, we have to, you know, uh, subtract that, but that You're putting this in a, in a greenfield or something like that, you wouldn't wanna run it immediately, right? No. 'cause there's no load on it, right?

Absolutely. Yeah. Okay.

We spend time learning and then we adjust as time goes by. Alright, thank you. Sure.

So mm-hmm. A lot of questions. If you're building a baseline, why not just start letting the user know about it proactively instead of so We'll user coming.

We'll show that a little later. I think where we are today, uh, the tourism, when, when we started with CPAC and start talking to network op operation people, they said, don't give me alerts. I have a lot enough of alerts.

Right? And if you give me a, if you have something very, very relevant Yes, tell me that. So really the way that we approach it today is kind of crawl, walk, run, as in we identify an insight, but the level by which we will say it's an alert or you should now act on it, is not there yet, right?

So what we are showing now is, okay, we, we know about it, we didn't really surface it. We'll show in a second. There are ways we can surface it, become more proactive.

But this one is like, okay, for whatever reasons, the engine did not think that it's worth, uh, telling the cust the user or the network operation, but it's still there. So this is still a workflow that is similar to, I would say, I always kind of joke, you know, I, I started in networking in, uh, the nineties and I was working on a cellular speed, 9,600 kilobits per second, and I use Wireshark. People still use wire shark, it was ere then people still use wire shark to debug 400 gigabits per second.

So I think this is just kind of showing how you, uh, get a little better, right? So you're able to still manually interact with a system, but instead of having to look through wire shock log, you can start looking at the doc at, at graphs and ask to visualize the information. So it's a lot easier.

So you still need to be a pretty sophisticated network engineer to understand what to look for. It's just a lot easier to be able to visualize it, right? So we have all the visualization today, and this is kind of what we were trying to show in the first case.

The problem is that we have hundreds and a big, in a big enterprise, you'll get thousands of these relevant graphs, which is very hard to know which one to look at. I actually appreciate you saying that you need a degree of knowledge in order to make it work, because I think there's a temptation with AI to make it the magic button about, Hey, you don't have need to have skilled people. They can just, you know, they can just type in what went wrong.

And magic will tell you. And, and I appreciate the honesty that yeah, you actually do have to have a clue what you're doing Yeah. In order to actually get anything out of this, right?

Because We certainly, you know, I would say the vision is to get the, to simplifying, you know, simpler and simpler. This specific one, I still need to know, okay, I need to look onions and, and, and whatnot. But, But I mean, is that goal really around getting to a point where, um, then the expert comes in, right?

It's, you're never gonna get to a point where you're solving it all the way to the end where you're still gonna have to call John and say, I'm looking at these packets, help me figure it out. So my, you just, I think the quote that you just broke, I think the, the speed of innovation that is happening today is so fast that I wouldn't say never. Yep.

Right? Fair. Uh, I see the, the sophistications of the, you know, the sophistication of the understanding of the models just changing all the time.

Uh, I had the meeting with one of the big, uh, cloud service provider that is also, uh, runs a, uh, um, you know, one of the LLM engines, right? And, and I said, and I complained that it didn't give me the right answer about something. And he said, which model did you ask?

I was like, oh, it was, yes. So the question was, is it Friday? Or did there really something on Monday that is completely different, right?

So today, you know, every few hours, something change dramatically. So I wouldn't say it'll never, Yeah. But there's, there's an, there's like knowledge that, that John might have about his business and the way it works, that that is not in any kind of, you know, model that's out there that, you know, those are kind of those intrinsic knowledge that, that we learn in doing our work.

Totally. Uh, but I, so, so as we go to the next one, you'll see that you got, in order to be more proactive and more predictive, you start adding more context, context such that the user, such that the engine can understand that. And this is, again, this is why it's so important.

Sorry, what I was showing there, and I skipped that, is that, uh, effectively what the engine identified was that there was, uh, two different, uh, two different VLANs or two different locations that, uh, uh, were, were showing the different latency, which pointed out to the fact that it wasn't actually a server issue. It was really between, so these are two, two locations in the network. So VLAN and, and vlan 2, 2, 2, uh, these are two different locations.

So it really meant that the load balancer between these two was the one causing the problem instead of being, if you see this one be, when we, when I was close to the server, the latency was very low, and I say, moved to the next one, the latency jumped. So what that meant was that if you're close to the server and the latency is low, and the next hop, the latency is high, the problem is in that hop, right? So it's really the difference between is it a server issue or is it a hop or is it a network issue?

And if it's a network where in the network the problem resides. Good question. Mm-hmm.

Um, as far as the visualization goes, I, I mean, I, I do, I I like these graphs. Mm-hmm. And it's, they're very informative.

When you were showing us earlier kind of the network diagram where, hey, this was the path that traffic was sort of taken, is that, is that part of the fizz visualization as well? Or was that there, is there a way to see the traffic flow and then the pinpoint like, oh, by the way, not only am I seeing this on this side of the low balancer and this, but here's where it's located in your topology. Yeah.

So it's on our roadmap today. Okay. We still, uh, use that.

One of the things we're using, again, AI for machine learning, you know, we can use that interchangeably, is to be able to understand the network, uh, topology. And when we say the network topology, we mean both the, the, um, logical. So if I have a, a client that is going over a net and he's talking to a server, that's the logical, but physically it's actually client going to a net and a net or firewall going to the server.

Or if you have a proxy, you have the clients going to the proxy and the proxy opens or a load balancer, right? Mm-hmm. So being able to identify all these, uh, yes, it's, we're working on it.

And then once we'll have that, we'll be able to also show that. But it's not here today. Do you see someday in the near future where, like in, in your situation, you're doing the RTT analysis, right?

Mm-hmm. Let's say the build balancer is operating in, in four to seven range and the delay is actually in a four to seven logic somewhere. Mm-hmm.

Do you envision the system being able to be smart enough saying, oh, in this pathology look, it's a four to seven, I'm gonna do a four to seven RTT, you know, or around type element test just to see what's going on and then reversing out the logic from that? Yeah, Absolutely. So today we know how to identify them.

I think we are, where we are is being able to visualize it in an easy way. Mm-hmm. But once we have that, then yes.

Absolutely. And how do you see the qualification of that expertise, right? Obviously cpac, you know, with the employees and the experience that you have has a certain amount of expertise in terms of that, that you have the LLMs that's kind of really smart in some way and dumb in some way, right?

How do you see the codification of the expertise, you know, so you've got the metrics, you got the LLM, which can do reasoning, then you have this domain expertise in there. How do you see the codification of that and how do you enrich that on an ongoing basis? And back to sort of Mike's question, which is, if you have domain specific, very domain specific, or even company specific unique insights, how do you embed that into the system so that we can actually get insights out of it?

Yeah, I'll show you, uh, the third one will show that a little bit. But the idea is, uh, one of the things you can do with MCP is to integrate that knowledge. So what we're seeing today is the users use the system, we collect the logs, we collect the, uh, the, and then you kind of start seeing the pattern.

So once you see the pattern, you can make it a kind of, what they call a high level tool. Mm-hmm. So that embeds that knowledge.

So instead of kind show me all the insights I can say, and I'll give the example later, instead of saying, show me all the domain names and then I need to find which one is suspicious, I can say, give me the suspicious domain names and integrate all what is a suspicious domain name into the tool. And then you get one question and one result that is kind of more, Right? So, so the, the, the abstraction that the MCP is supposed to provide is you meet l LMS on the halfway, so you count on the reasoning engine inside the MCP no, inside the LLM to be able to understand that.

It, it, if I give you these kinds of function calls, essentially in the APIs, in the MCP, that you will call the right ones Yes. To get the right insight. And based on what I tell you that you can reason the next step.

So tool use and you have to sort of meet it halfway, right? Because you don't know the LLM gets better and your MCP needs to provide the appropriate context, Right? Right.

If you look at this and the reason the demo took a little longer to, there were a lot to branch sql. Yeah. So you want to kind of get away from the one SQL And say, yeah, that should be be on this site, not on that site, because that that's specific to the domain.

Exactly. Right. Yeah.

Exactly. Got it. Great.

Thank you for that. That was excellent. Um, couple of things just to highlight in that use case, um, first of all, it's really essential that we baseline, right?

In this case, there was a ticket created by the user. The service was impacted during a certain period of time, but when we start seeing that pattern, the, the, the machine learning actually gives us that context. The other thing is essential is that we need the data, right?

Da data at the lowest level gives us the insight, and then we integrate that into, uh, the different processes. But our, when, when Ron was talking about the different, uh, systems that we have, every component of the CPAC architecture has, uh, a data store. And so this is the, the metadata store, this is the absorption of all the data from the different brokers and the different, uh, storage, uh, capture devices that roll up into that analytics layer.

So it's not capturing everything across the entire enterprise. It's actually catching the session metrics and the observability metrics, and then doing reasoning over that. And then, you know, we, one of the things I would say, just to answer that prior question, is, is this is why workflow is actually essential.

Workflow is human in the loop, right? AI assisted, I think that's really the, the, the essence of what we're talking about is that it's a new tool in the toolbox, as Mark said in the beginning. Um, but what it's doing is it's accelerating the triage process.

And so from a business perspective, if you were to say, you know, why is CPAC a gonna help this go faster? It's like, simply put, we're gonna shorten the troubleshooting time, either meantime to detect, meantime to understand, uh, the context of it, which component, um, who to call. Um, that means shortening your war room triage process, and then also integrating into your tool stack.

So if you're using things like PagerDuty, slack, um, if you're using ServiceNow for their Gentech workflows, and this is where I think the industry's going, uh, Gentech workflows are gonna happen at different layers. There's gonna be micro, and then there's gonna be macro. And so in this case, this is, could be both right?