cPacket Observability for AI

Thank you for joining. My name is Eric ine. I'm the field CTO for C Packett.

I'm joined with Ron Devoux, our CTO. And today we're gonna talk about a very interesting, uh, uh, thing that's coming up more and more, uh, with enterprises, which is AI factories are moving into the enterprise. So traditionally, a lot of these workloads have been done in very large data centers that are, you know, have thousands and thousands of GPUs that have run very, very high speed networks, 400 at the core, 800 potentially.

And so as these training models mature, we're seeing the gentech processes push a lot of that workload demand to inference on the edge or into the enterprise. And so this is probably a problem that no one's really yelling out yet, but this is a problem that's coming. This is, this is something that we're seeing.

And so Ron's actually gonna walk you through a specific use case that really highlights where the getting the metrics and the observability connected to inference based architectures is essential. But it's also important that NVIDIA's been talking about this. They wrote a really great blog about AI factories.

If you haven't read it, it's really important, but we know that this is coming. We know that, uh, you know, you have to move the agents closer and closer to the customer, closer and closer, closer to the data. And there's a lot of examples for that.

And so as these enterprises shift that workload away from the large training, uh, workloads to these agentic inference models, these problems are gonna become more and more prevalent. So, Ron, sure. Thanks.

Uh, yeah, I think this slide, just to, to emphasize what we're saying today in, in the market, actually this section will be a little different than the others. Uh, the others, what we try to do is to show kind of what customers are using today and what's coming in the next 12 to, uh, uh, sorry, six to 12 month, right? So the things that we're ready to release around cps, this one is a little different in the sense that no demos, it's gonna be all me talking.

And, uh, it's more about what we're seeing today happening in the market with our partners, uh, with our leading customers, and what we expect the next set of problems, uh, is gonna be. So again, if we talked before about AI for network observability, this section is all about network observability for ai, right? And the, the difference really that I want to focus on is this AI factory.

And, and the big difference is that when you do training, uh, there is a lot of literature already written around it, and it's really huge, very elephant flows, very low entropy that runs for weeks and months, uh, in the same way. And they generated a very specific set of problems that people are dealing with them. For the most part, they're being managed by the top four, top five companies in the world with a lot of tools that they already have.

As we talk about AI factories, that goes to our, our ICP, right? Our ideal customer profile, which is much more of an enterprise, uh, that can, that has a data center, can do things on its own, has some other considerations, uh, that, uh, are not just training. And then the when.

And then when you start talking to vendors, partners, academia, there is very little understanding of how inference is gonna look like, how is it looking now today, right? Because most of the traffic is still around training. But when you go forward, if you listen to Jensen in the keynotes, it's, you know, all about gonna be all about inference.

But when you start try trying to dig in and say, what is inference? There are very few answers, right? How does it gonna impact the network?

What does it actually gonna look like? So what I wanted to do in this session is talk a little bit about what we're seeing and how, why we think observability is relevant in the context of observability. Why we think we need packets, and if you need packets, why the metrics that we can generate from c packet are going to be relevant.

So a lot of that is just gonna be talking about what is inference, right? So if you think about inference at a very high level, you are talking about AI clusters and, and we wrote AI inference, they always, you know, there can be some level of training. But in general, let's, let's assume that most of the workflow here is gonna be inference and you're seeing, uh, two very big categories.

One is, uh, the interaction with the user or the agent, as you saw the MCP agents are, uh, the agents are generating more and more traffic. Uh, they're gonna have a very, a very different pattern than users. Users work from eight to five for the most part.

Agents can do all their work at night. Uh, but from a network perspective, it's still gonna look like a query and a response. And many of the queries and the responses will become a session.

And at the same time, uh, one of the biggest questions that, uh, the designers of these clusters have today is what should be their policy of managing the KV cache, right? The KV cache right now is a magic pill of, uh, uh, accelerating many things, but if you decide to evacuate it to the external storage, it, it can cause too soon, it can cause a big hit on your performance. And, and one more word, the performance really, if I go back here, is, is really the challenge, the technical challenge that people have is they want to maximize the GPU utilization, right?

GPUs are very expensive, but there is also SLA or SLO for latency because I can maximize to a hundred percent by generating a very long latency, right? I can have one GPU do everything on the GPU. The GPU will be a hundred percent of the time busy, but everybody's gonna wait, right?

So, how do I balance this equation is really that technical challenge that people have. So what we've done, again, with, with our set setups and, and working with partners is trying and break down. And, and I don't want to go through all the different levels, but break down the, um, different workflows that generate that we know of today, right?

Tomorrow, it's gonna be a little different, I'm sure. But the, the ones that we know today, uh, between users and agents and how do they impact, how are they gonna look at when I, when I view them from the network perspective, right? So the four one, the U one, U2, U three are users, A one, A 2, 3, 4, uh, are agents.

And, and you can see that ones that, you know, hopefully people are familiar with, right? The prompt responds is really, uh, the typical one. Tell me something about the book.

Uh, streaming is a little more efficient in the way that they can receive the answer. Uh, and then, you know, media, media generation, right? When I want to generate a a, an image, uh, and then we started seeing some of these automatic agents, right?

Some of these agents, right? Just in the rest of the demos, and they can generate very different pattern, right? Because the user can, like what I did as ask relatively short answer, short question, but for it to answer it had to go to the database, bring up our tunnels of data, send a ton of data up to be analyzed, short response.

So all these ratios are gonna be very different. When I look at the south north base, the, the, the switch that is accessing the, the cluster, uh, both in, um, distribution as well as in size. And at the same time, the KV cash is also gonna be different, right?

So the traffic that I'm gonna see there going back and forth with storage is gonna be different based on the use case. So what you want, well, we'll talk about what we want to do, but this is a, a statement that says, you know, as you think about, um, Inference and inference being the dominant factor of, of your network, of your, uh, investment in, in GPUs in capital, uh, you, you need to take it into account. And, and the reason that you want to have an observability is, is, is this is the way to improve what you have, right?

Without, if you, if you don't have that, you're gonna fly blind, right? So you want to make sure that you configure your network correctly. You, you want to understand how your KV cash.

So there are a lot of things that you want to understand based on how in reality your network behaves and in real time. So again, uh, we'll just, just repeat the picture. So we are talking really about the front end, or it's called north south.

This is the switch that connects the cluster to the rest of the network. Uh, down below there is what they call, uh, east, west or backend, very high speed networks, uh, for the most part, rocky or, or, uh, infinity Bend. We're not really dealing with that problem, right?

The idea that we are talking about is adding the monitoring points to the north south. And to repeat the theme that we had before, there is other data that is gonna come in, right? There is a lot of GPU telemetry that is coming in the switches today, telemetry, the storage is gonna change, uh, telemetry.

So the question is, why do I need also packet information? And these are some of the reasons that, again, I'm not gonna go one by one, but being able to monitor the actual latency between the storage and the, and the, um, around the, around the, uh, soft, uh, south, the north south, uh, switch, uh, will give you a real hop by hop latency inside the switch. They know how much time the packet spent in the switch, but it's very hard to pinpoint where the latency generated by the cluster was.

The general latency generated by the switch was the latency generated by the storage, right? So if you really want to understand where in which hop the latency was, uh, was created, you want that microbursts, uh, these, uh, um, for different reasons you can misconfigure your, uh, your network and create a lot of InCast. So it's, if you have the packet drops because of, uh, microbursts, it'll be hard to, to identify, uh, inside the switch and the ability to put things together as a session level, right?

So again, switches are able to see packets. Uh, the, uh, GPUs can tell you something about the application, but if you also want to understand the session, this is where packets are relevant. The nickel money to one, isn't that on the backend network?

Primarily less so in the front end network? Yeah, Well, That, Yeah, it might be. Yeah.

So it's examples of why we see bursts today. Okay? Yep.

Totally. And nickel is also, well, yeah, Yeah, primarily, yeah, sure. So this is the type of information.

Now it's interesting. So what I did, what we, the reason I generated this, and if you notice these are not actually gravano, uh, these are, were generated by, uh, an LLM is because Grafana. So, so even though we have the data, it's actually Grafana is not very good to visualize it.

So this is example, another example where you can play, uh, it's another tool for you to use. So the things that you really want to understand is out of all these eight use cases, and, and I'm sure that by the time in the next 18 months, there will be 16 of these, which one is the dominant, right? Which one is sending the most data, how many types you have that, and, and there are characteristics of these, right?

One of the characteristics I I pointed out is the, the histo or the ratio of ingress to egress, right? So being able to print out the histograms of these sessions, uh, based on the ratio of data coming in, data coming out, uh, and size will allow you to say, okay, here's how many I have of each, given that I know how many I have of each, you can say, okay, this is the right configuration for me to optimize, right? Um, do you see the session, and this was the go Ahead.

Do you see augmenting that with the actual query type as well? Because now coming in, even with MCP, you have no idea was an agent or was a human or the type of query, do you see like breaking that down as well, or? Yeah, I do.

Yeah. Yeah. Some of that will come from the GPO, right?

So some of that, that will know from how many types of this job had, right? So some of that is in the application, some of that you want to know how, because it can tell you, okay, over the last 24 hours, I had this, eight of this and eight of that. But how did they distribute over time will be a little harder, right?

As in over time, in, in a, in a, in a lower solution, right? Where did they all come? Because there is, once the job is landing in the GP, he may start bringing in data, right?

Right. From the, from the storage. But the GPU tags, it, you know, sort of source IP or whatever source port, and then you augment that with the network patents, then you can see the, that that's the hope, right?

Yeah. Yeah. We, we totally see, uh, huge amount of information coming from the telemetry.

Yeah, absolutely. Uh, and then the other one is just to understand the RTT, right? Again, one of the critical things as you go to, uh, inference is, is the latency or is the, uh, whatever SLA you have for, you know, is it P 99 or the time for top first token time for the last token, uh, all these information is, is critical, right?

So basically, see the GPU utilization will come from the cluster, right? Compare it to the latency between tokens, the latency, uh, of the sessions themselves. Yeah.

So this is essentially just double click about that, right? You want to understand how you classify the flows from the histogram that may impact how you define priority. So these are just examples, right?

So you may want to prioritize zero priority one, right? The different, uh, things that you want to play with them and, and probably different, uh, uh, specific users will do different things. And then you go to, uh, what are the things.

So today there is huge amount of parameters that people just use, don't know how to use or don't use. And, uh, it just gonna grow, right? If you read the, the lama, uh, whatever, 70, uh, or uh, very long page, they said, well, we are trying to, uh, optimize the ECN at the end.

We said the default is good enough, right? After running a lot of experience or is not worse than anything else. So, and, and, and without a lot of data, it's very hard to do that, right?

So that's essentially the idea. Great. Excellent.

Everything you wanted to know about inference networking, right? It's this, like I said, it's, uh, it's coming. Um, I think this is gonna become more and more, uh, important and as network leaders, this is something that we need to be thinking about proactively.

Um, so when you're thinking about if you're an enterprise and you're making very large investments and inference at the edge and putting this, these new GPU workloads into closer to the users, closer to the workloads, we need observability. And we'd like to say we need c packet analysis doing some of that work as well. So that's really important.

And it is a relationship between the, the G-P-U-C-P-U activity with the storage, which is essential. The other thing is, is that we know there's different types of data. And so this is gonna come from, we've broken that down into a agentic or user-based.

And I think that's also really important to understand is how do we characterize that data? How do we contextualize that data and how do we integrate that into the dashboards? And I think this is actually, you know, the evolution of a network engineer, uh, an AI network engineer needs to understand these workflows.

Uh, and so when the boss says, Hey, and we've seen from Nvidia, the cost per minute can be in the thousands, or you know, in some cases tens of thousands of downtime. So the, the, the investment costs are there now, we need to make sure the observability and the responsiveness of the team is a, is there with matching that same level of investment. So it's continuous tuning, it's optimizing the workflow, it's identifying those baselines, just looking for the, uh, the different hop to hop characteristics, and it's also integrating it into, uh, your daily processes.

Okay. So getting into the conclusion, uh, thank you so much for your time today. It's been awesome.

Um, thank you for bearing with our different demos and the slight delay, um, in, in the, in the morning. But well, we want to call out four key things as takeaways from to the session with C packet today. The first one is that AI is here, but it is not the only solution.

It is part of the solution. It is an enhancement, it is an augmentation. It helps the human in the loop, but the best way to sell that is it helps, uh, with, with the outcome.

How do we tie that back to the business need? And then do we have the insights coming from the, the network at the packet level, rolling up to the observability metrics tied to the individual workflow that gives us the ability to solve the problem faster? And that is absolutely foundational moving forward, particularly as networks speed up.

All right? And then, you know, from our perspective, it's not just a single uh, tool. It's not just a single technology.

It's not just having a next generation packet broker or a next generation packet capture or having analysts on top. It's all of it. It's all integrated, it's all open, it's all infused with different things like MCP and LLMs.

It's, you need to have this in order to do the speed, the scale to provide that insight. And lastly, let's start with the business problem, right? That's really the core of what we talked about with the customer journey, is how we engage with the customer.

We don't start with deep protocol analysis. We start with what are your business challenges and how can we help you move that forward? And how can we prove to your boss, uh, this is actually gonna be meaningful.

So if you were to take something out of our session, the the key thing to, to start asking is the what, you know, what is the challenge that's struggl that you're struggling with? Um, what's driving the business nuts? Um, you know, what are the top networking issues that we can help apply and what are the metrics that we need to, uh, provide to help solve that problem?

And then we can engage together and we'll work through the how.