Cisco AI Cluster Networking Operations DLB Demo with Paresh Gupta

Okay. So look, we talked about packet loss and we talk about network delay. We did not touch much on network congestion, which is a significant cause of all that.

So let's talk about congestion in, uh, the networks. Now, there's a very specific scenario that happens, right? Remember I said that going back to the basic point, let like, at least when I read the basic thought, okay, if I remove over subscription from the network, where's the congestion coming from?

Right? A 400 gig down, 400 gig up. Where's the congestion coming from?

Right? And in this example, you can see that host one is sending traffic to host five, and then host three is sending traffic to host six. But it's possible because of the way ethernet network works, there's a concept of ECMP load balancing where the requirement is to send all the packets of a flow a layer for TCP flow, or a layer for UDP flow on the same link so that we avoid out of other arrival of these packets on the receiver.

This ECMP scheme is random, and because it's random, it's possible both the flows going on the same link when spine one sends traffic to leave three. So if you take a look, it's very much possible, and by the way, it happens in reality. I'm not showing you rea uh, you know, theory that link to spine one is trying to send more than 400 gig traffic on a 400 gig link, which is not even possible while rest of the links are sitting idle.

So this whole problem statement that I'm explaining, this has been a major area of, of research and conversation. In fact, most of the time when people talk about AI networking, they directly jump to this problem statement. And like nobody will talk about operational simplicity, cabling problems, security problems, all that which happens in reality when they start building this environment.

And here's a screenshot. So you can see the traffic pattern and percentage measured on all the equal cost links between a spine and leaf. And you can see the utilization is totally uniform or non-uniform.

Is this a new problem? No. No.

Has been there forever, right? In fact, I mean, I'll I'll say this. When I read the first networking book, I always thought when the the devices and traffic they send first packet on first link, second packet on second link, that's a natural way of thinking.

That's how our freeways work, right? If I have four lanes, first person will go on first lane, second person will go on second lane. And then after a few months, somebody, you know, corrected me no.

And explained this whole out of order arrival of the problem. So nothing new. The only new thing is look into typical networks, uh, don't see line rate traffic coming and going on all the links at the same time.

There are many, many flows. All of them are, most of them are small flows only few of them are large flow. We call them as elephant flows.

So that's why they're, this phenomenon is seen in most of the networks, but it's not a problem as such. What I mean by that is, let's say if you have two links, one is operating at 60%, the other one is 40%. Is that a problem?

Not really. I mean it's okay, but let's say if there's a link that's operating at a hundred percent trying to go to 105%, the other link is operating at 60%, that's a problem. Mm-hmm.

Because when the link is trying to go to 105%, it calls like, at least I like to call it as congestion due to over utilization of a link while I have network capacity sitting idle. Right? So before adding more capacity to, to the network, I like to push the other link to to a hundred percent, right?

It's like a perfect problem to solve with IPV six because it's designed to have multiple addresses on a net. Yeah, Multiple. Well, I was gonna say that that gets you around your problem.

You could Of hack make the same argument about s RV six. Yeah, So well I was gonna ask that question too 'cause I come from the carrier world and it seems also like a, the segment routing would also solve that problem. Yes.

So there is a, a major hyperscaler out there who use SV six to address this Problem. Okay? So going back to your original problem statement of if I'm sending a packet on one link and a packet out of the same flow or the same, you know, application flow in on another link, are they running into that problem of out of order packets by doing it that way?

Was that RV six? So There are different implementations. Uh, look, the hyperscaler that I mentioned, they have their own custom code running.

Okay. And that's why I started in the beginning saying that anything that you see out there may not apply to you. What they do is they have the, uh, handling of that out of order arrival of the packets.

Got it. That's one approach. The other one is if you send their traffic, you can actually provision a network in a way that it avoids this outof order app to an extent.

Okay. But most of the time it's like, uh, handling of those outof order, arrival of the packets someplace in the network. You could do it on the next, you could do it on the switches.

Okay. Right. Got it.

So talking about the solutions, we just talked about one of the solutions that's very specific to few customers. So I don't want to generalize it. What I want to generalize is like what's available to most of the public, most of the customers out there.

One of them is what we call is the flowed dynamic load balancing. In this approach, what we do, although there's a single, uh, layer for flow, the switches are able to detect the inter packet gap. And if the inter packet gap is longer than, than a configure threshold, then the next set of packets are identified as a new flow, and that new flow is sent on the least utilized and the least congested link in real time.

Where do you store the state for that? In our wondering That? Yeah.

In, in the silicon one, uh, NPU, right? There's a whole engine. It's, it's in the silicon one, it's called FLLB, uh, flexible load balancing or flow load band load balancing, uh, engine that's able to do it, like wait for it.

If you're thinking about flowal, we have extended it to all the way that nowhere, nobody else in the industry is able to achieve that, right? So this is one way, and I have some results. If you take a look on the left, you see this is using the default TCMP load balancing approach.

All the links are utilized at different levels and then compare the, uh, performance on the right, uh, all the links are nearly well balanced, right? They're not perfectly balanced, they're nearly balanced because every flow might be of different length, right? The idea is that every flow belongs to a different RDMA operation.

So all the packets of the RDMA operation arrive in order, although different RDMA operations might be arriving out of order, which is okay, I Is that an assumption or is that something you can definitely say based on that length? It requires tuning. Um, there's never a guarantee because there could be different kind of workloads, there can be different kind of, um, models running there might be different type of jobs.

Okay? So this approach requires some tuning because when I said ba the, the flow nets are identified by the inter packet gap, that inter packet gap is config able on the switch. Now how do you know what into packet gap you should configure, right?

Right. So that requires some fuel, okay? And that's what leads us to the next approach, which is again, a part of the partnership that we did with Nvidia in this approach.

What we do, what the switches do, what exactly I thought the very first time I read about networking, which is send every packet on the next equal cost link. So in this, the switches send packets, you know, they do packets, right? But look, there's intelligence involved because before sending the next packet on the next link, the switches look for the real time usage and the real time congestion states of the next link, and then it sends the next packet on the least congested link only, right?

And then you see the performance, like take a look on the right, all the links are beautifully well balanced. Are you doing something on the receiving end to avoid the out order problem? We're talking next about that, of course, Obvious question, right?

So this is exactly where the, the partnership between Cisco and Nvidia takes a place, right? Like this is what we delivered. So to convey that point, and just to explain that logic at oversimplified, again, we take example of two nodes.

Each node has one GP each has one snic each. Today, this could be NVIDIA's Bluefield three or connect X eight Nick, and then they are connected via Cisco, uh, switches. And now node one wants to transfer, do some remote direct memory right to no two.

And depending on how large this transfer it, like, just assume for the sake of example, there are four packets involved. So super nick one, the one that wants to send it takes those four packets and to leave one no problem because there's one link, assuming there's one link, so it goes in order. But the leaf one may have some different congestion states on every link, although all of them are exactly the same, same capacity, but probably there's some other packets getting transferred on one of the links.

So what happens when they arrive on spine one and spine two, there might be slight delay in each packet. And when spine send it to leaf two, and then leaf two sends it to the receiver Super nick, the packets that arrive out of order, right? As you said.

Now this is exactly where the partnership takes a place. Nvidia has this technology called adaptive routing with direct data placement. What the receiver SuperNet does, it takes a look on the tag on the packet.

Every packet is tagged with its correct location in the memory. And then it places that payload directly in the GPU memory. Remember, there's no reassembly.

Sometimes people confu confuse it with the reassembly because if the NIC has to do reassembly, Nick needs capacity to keep this packets in his buffer, that's gonna increase the cost instead of reassembly. This is a direct data placement of the, of the incoming payload based on the packet tags. And that's happening on the nick, not on the GPU then.

Oh, Well, all the, it's fairly a complex implementation. All these systems must work together. But yes, most of this is happening on the super nick.

The same up to an extent would apply if you, if you replace the GU memory by system memory, the same concept would apply. All the numbers might be slightly off. Now, do you believe me, whatever I just explained, is it gonna increase your performance?

Right? You know, at least if even if you believe me, customers won't, right? Because they have to pay for it.

I'm such a trusting guy. So of course we did many benchmarks. Uh, NVIDIA has done their own benchmarks.

In fact, spectrum X is the technology that's running the largest ethernet AI cluster in the world right now. You can guess it, I'm not gonna say it. And the same technology is now available to Cisco.

This is one test that I could run in my environment and only the output of that test I've put on this slide because I can, I have a video if you like to see it, I can play it, right? And in that video you'll see exactly the same chart that you're seeing. That's, that's the reason I think there are some papers claiming 60% improvement, 65% improvement, different claims, different environments.

But very simply, if we run all to all collective where 64 GPUs are sending and receiving traffic from every other GPU with ECMP approach, we achieve a kind of a buzz bandwidth. You know, the, the measurement of, of, um, success of all these collective algorithms, uh, typically people use bus bandwidth as a measure in gigabytes per second. So that's what we plot on the vertical axis.

And then you see a light blue chart on the left. It shows that the result is around 35, 36 gigabytes per second, but the performance goals are increased by almost 40% when we enable this feature on Cisco switches, which is per packet DLB with the adaptive routing and direct data placement on Nvidia SuperNet. Okay?

So there's a configuration that has to happen on the nick as well as the switch for them to work together. Yes. But there's a auto negotiation happening.

If you remember, there's a spectrum in spectrum X handshake that's happened because look, this requires massive amount of validation. It's just like, yeah, you know, you don't wanna risk it, right? So there is a handshake available and this is something, uh, we delivered in the now last Nexus release ten six two.

Okay? Uh, if I remember right, or ten five two or like I have it written in one of the following slides right now. Let me know if you want to see a, like a geeky video.

I think we have time to, you know, 20. Yeah, sure. Yeah, that'd be great.

Of course. Yep. Always up for a geeky video.

Yeah, that was a, you know, rhetorical question. So what I'm doing here running, I'm using Slum, which is a job scheduler to run an all to all collective between 64 Nvidia H 100 GPUs, and they are connected using Cisco access switches kind of a this environment, right? We, we have looked at this topology a few times now, right?

Uh, so let's see, what do we get? So we run the command On specific sizes because we just have to simplify the outcome. Now, it takes a few minutes to run this job, but by the time, let's see how we are gonna measure the success of this test.

There are four charts that we are looking at here, and let me take a moment to explain it. The chart that you see on top left says RX traffic on core interfaces. So we are looking at a leave switch and it says receive traffic on the core interfaces that the traffic that the spines are sending to it.

And then the chart on left bottom shows TX traffic on core interfaces. So the, so the same links, but now it's TX traffic, the traffic in reverse direction. Then chart on top right shows RX traffic on edge interfaces.

So this is the traffic that's received from the GP next, or this is the traffic that the GPUs are sending to the leave switch. And then the chart on bottom right shows traffic that the leave switch is sending to the GPU, right? So these are four charts.

In a few minutes it'll become even more clear why we are looking at four charts. The test seems to get over and here is the output. So this is how the output of a collective s runs, right?

Prints all the ranks. We assign a rank. I mean, this is a typical, uh, HPC or a distributed job terminology.

And you can see the output is like, uh, buzz bandwidth is around 35, 36 gigabytes per second. And now we run the test again. Next we are gonna run the test again.

And of course, the middle part I've edited out of video, right? Um, uh, which is the next run will be after enabling DLB per packet, uh, mode on Cisco Nexus switches. Mm-hmm.

And, uh, with adaptive routing enabled on the NVIDIA super next, and let's see, I, I think so okay here, and you can see the result is 50 gigabytes. This is exactly what you saw on the slide. And this is why I said only this result I'm showing you, which I can prove to you right now.

There are some other results to which show even better performance. Now, of course, uh, 50 gigabyte is with adaptive routing and with DLB per packet mode on Cisco Nexus switches is around 35 to 40% better when DLB and adaptive routing was not enabled. So this is the ultimate measure of success, but let's see what happened on the network, right?

Just to be sure. So the first spike that you see on the left, that was with the ECMP, without using DLB or per packet or per packet spray. And you can visually see the imbalance on the links, right?

But if you take a look on the chart on the right, the links are perfectly balanced, right? So that's first measure of success. Now, we are gonna look at other charts too, and one of the charts is the number of pause frames.

Now, I didn't talk about pause frames a lot today. Essentially this has, when I briefly mentioned PFC in between, uh, priority based flow control. The way it works is that if you are sending some traffic to me, and if I don't have enough buffers left, I'm gonna send you a pause frame, right?

And I'm expecting you to throttle your rate, right? Right. That's how pause frames work.

In this case, when, uh, all these directly connected neighbors talk to each other, if, if the receiver's seeing congestions, it's gonna send pause frames, very similar to the logic I just explained. Typically, we don't want the network to throttle down the end devices. So you can compare the spikes on the left, which is about ECMP, and then compare on the DLB per packet mode.

There's almost nothing, no pause activity on the network. That means the flow control is not getting in invoked, which is nice, right? In fact, if you take a look on the graph on the bottom right, it says TX pause on edge interfaces.

So this is the throttling that the network is doing to the GPUs, which totally goes away after enabling DLB per packet mode, Right? This is the second measure of success. And we also have a third measure of success, which is the buffers on the switches.

Like do we reduce the amount of buffers that are used on the switches or not? And we actually do, which is this graph. Now I think I'm gonna zoom into this graph because it's way too small.

And you see there's a small reduction. Those spikes with DLB per packet mode, they're slightly lower, right compared to the spikes on the left, right? So we also reduce the amount of buffering capacity or the amount of buffer that have been used, not the capacity buffer used on the switches to be precise with my wording, right?

So three measures, like well-balanced link, uh, no flow control applied from the network to the end devices and reduce amount of buffering. But the ultimate measure of success is here, which is the application level. Uh, what you see on the slide Question for you, Rita Younger, um, what if you configured DLB on one end, but you don't have the configuration done on your, on your GPU.

If the GPU is unable to, or the receiving SuperNet is unable to handle out of order mm-hmm. That's where it's going, right? Yeah.

Yeah. So performance will dip, Okay? Yeah.

So we don't, that's why we don't wanna enable it by default on Nexus dashboard. Yes. So this is why you see this a handshake working.

You see this as a validated solution, right? If I, if you're promising this kind of performance, all we are saying customers, look, there are five other approaches. We briefly talked about five other approaches, or you can take one validated approach.

If you're deploying it first time, just use it. We guarantee it'll perform the best, right? But yeah, I mean there could be a hundred misconfigurations out there, we don't wanna do that.

Yeah. Making sense? Yeah.

The other challenge that may happen is one of the links failure, and this I again, oversimplified it in this example. Imagine before a link failure leaf one is sending 50 50% traffic to both the spines. But let's say if spine two to leaf two, one of the links fail at that point leaf one should not be sending 50% traffic to spine two.

So what we do here, uh, instead of sending or advertising the cumulative weights from the routing protocol, which is EBGP, in this scenario, the devices advertise a par path cost. And when the cost increases accordingly, as per the weight of that cost, leaf one does DLB of the packet or dynamic load balancing of the packet. Now this is a RFC based, uh, solution.

The unique point that Cisco brings is that this feature works with dynamic load balancing as well. So it's not just like weighted cost multipath, but DLB plus weighted cost multipath, which is very unique. One of the features that we developed for one of the hyperscalers, the reason we could develop is because the silicon one architecture, going back to ASIC point, Ethan, Ethan is, is based on P four programmability, right?

So we don't have to re-spin our whole LASIK just because we wanna develop a new feature or deliver a new feature. When the requirement came, we allies and we were able to deliver it. The other example I wanna give, and although as I said, anytime we talk about AI network, most people start thinking the backend network.

But the significant challenge in the frontend network too, this is a problem statement that was given to us from one of the hyperscalers. And I wanna bring your attention to the circle that I've done in the different traffic coexistence of all this traffic together, right? We have high performance rocky traffic, uh, with the storage appliances.

We have TCP traffic going, we have storage traffic, which is not rocky, but t maybe TCP based. And this whole congestion problem that I explained to you about the non-uniform utilization of link may even happen here. So what we do, we identify different traffic types and we provide different type of load balancing scheme to each traffic.

Right? Now, if you think about what you just said, uh, the remembering the states for just one traffic type is one thing, but now we are remembering the state for different traffic types on the same lengths. All this is done within Silicon one architecture.

And again, this feature, uh, is not very old, right? In fact, like, uh, I came to a sister event AI field day last year too. We did not have this feature back then in last 12 months.

We got the requirement, we analyzed and we delivered this feature for that hyperscaler. Since that time we started talking to the customers. And then a lot of customers said, you know what?

I don't have a lot of backend traffic going. I don't have backend network. I'm doing an inference only cloud.

This use case serves my purpose because even with inre and your frontend network is super important. So I've got a question for you after you're listening about dynamic load balancing, uh, Kevin Meyer, the what would, what would be your trade-offs in something like SRV six versus DLB? Is DLB just simpler to deploy Yes.

And easier to deploy than having to use S RV six to manage traffic. Yes. Um, and then I'm assuming like if you were to go the path of SRV six, that gives you more granular control, the over where you send your clothes versus letting DLB just manage it.

So This is a fully validated solution. We stand behind it, it works. As I said, this Spectrum X architecture is running in production in one of the largest ethernet AI cluster in the world.

Okay? The SRV six implementation is, is purposefully built for few environments. If you remember my very first slide, I said some hyperscaler have massive r and d budgets, Right?

So that solution may or may not apply to everybody, right? Right. Possibly.

Well, I was just curious about what the trade off the decision trade offs were to choose one versus the other. Technically, you know, you always have four different technical ways of solving the same problem right? Now after that, the business reasons comes, right?

So as I said, the business reasons are more important that this is d lb based is a, you know, fully validated solution with Nvidia and Cisco, the SRV six solutions, some people are able to make it work, okay? And it's working well for them because they have that custom development going on. Okay.

Not something we can offer to every customer who probably may not have that kind of r and d budget going on. Right? Okay.

Trying to deploy it for the very first time. Got it. Paresh, um, Cisco is a steering member at the al Alter.

Ethan a consortium, you're gonna talk about alt ultra ethernet at all. Ask me, what do you want to know? So yes, you are right.

Well, It's not come up on a slide yet. Yes. And it seems really salient to the conversation.

Yes. So look, uh, as you said, uh, Cisco is part of the steering committee. Cisco is also part of the other organizations that have showed up in OCP few weeks earlier, which is the ethernet scale up network, all that.

The reason I'm giving you these examples, the proof P four programmability, is that all these new features of ultra ethernet, one of them, which is very well talked about is packet trimming. Now what packet trimming does, let's say there's congestion on the port. So probably I cannot store four kilobytes of packet on the buffers, but what I can do, I like drop the payload, take the header and send it with a prq on to the receiver and say that, hey, the network is congested.

Do something about it. Features like this, we can implement on our Silicon one ASAP because of the programmable architecture we are currently working. In fact, just yesterday, all day I spent with another leading GP vendor talking about the same features.

But remember the, these features cannot just happen on switch and the end device doesn't do anything about it. So it must be a validated solution. And I would say you can ask the next question is that, Hey, how's your adoption on TriNet with any standards?

The adoption takes a while. Well, The standard just came out and it's huge and yeah, there's a lot of implementation Yeah. Challenges going on there.

I'm only aware of it in lab and test environments right now. No one's running it in production of any sort, right? Look, there are some pre UEC version people have done it.

Yeah. It's not like everything in UEC is not a nothing that we ever heard of. It's not like that, right?

People have been aware of some of the ideas before people build that and like they are running in production too. Again, all custom hyperscalers who did things by themselves, the controlled stack, the right firmware, all that stuff, right? We are also doing that.

But again, we wanna go to our customers when it's validated end to end, not just like, Hey, I do it and like then you ask me what do I do with it? It's like I'm interpreting this to mean you'll support it when it's there and customers are asking for it right now. This is what we've got and it works great.

Yes. Go with, go with silicon one, go with the Nvidia reference architecture, et cetera. Yeah.

And not just Nvidia, let's say if you wanna go with a MD and Sando, uh, polar OnX, we have completed validation. And you know, there's something that I can explain you later. I cannot send on the recorded line.

But yeah, we are doing that joint work on, on, you know, ultra ethernet pre some features at a time. Of course not. The 550 pages of specification will imp get implemented on day one, one feature at a time.

And we, we are co-developing that. So another question in line with, with ultra ethernet, what, um, if you were to summarize that, um, you know, for people that aren't aware of, you know, what's in there? 'cause it, it's new to me.

I haven't worked with it. And like Ethan said, it's new to I think everybody, what are the major differences? What is a quick summary of the major differences between ethernet and ultra ethernet?

So you maybe quick was the wrong word. 550 pages of spec and how do I summarize it? Okay.

Or maybe some of the high points, maybe some of the, some of the most notable differences, if not all of them. So look, remember I talked about the biggest problem in backend network is like traffic imbalance, right? And the congestion problems, what ultra ethernet focuses on is the GP two GP communication using ethernet as a network, right?

Okay. And all the, you know, basic issues that people thought will probably ethernet will never end up serving a use case for now, people are going back and like going to the drawing board and developing those features. In fact, packet spray and all these things are, you know, included in El Ethernets.

Okay? So, so instead of pushing that up to layer four, I'm going to solve that problem at layer two with ultra ethernet That well there. So ultra ethernet also, uh, I think the biggest contribution of ultra ethernet is a new transport.

Okay? So If you talk about, uh, Rocky, the next version of Rocky is like you could think as an ryther net transport, right? So that also runs on UDP, okay?

Right? So network, that's why most of the requirements for network remain unchanged, right? Because still U-D-P-I-P traffic, but the end devices must be able to handle that end-to-end transport of ryther net.

People call it as UET, right? Okay. So look, the fa the focus is on GP P to GP communication as a InterG P backend network.

And people are trying to, as I said, uh, make ethernet ready for that level, okay? Right. But again, youi ultra ethernet is one way of solving the problem.

Like, as I keep on saying to solve every problem, there are at least four or five other approaches in the industry. Spectrum X is one approach and you saw the results out there, right? Okay, I think I added this content based on yesterday's conversation.

I think I was asked, uh, that, hey, you know, earlier this year we announced partnership with nvidia, what's going on? So in February before GTC here in in Bay California, we, we, we made this announcement and the GTC happened. We made these three promises to our customers.

The first one is, uh, Cisco would be part of those reference architectures. I already explained that. Uh, March, 2025, we published the enterprise reference architecture.

Last month, or rather last week we published the cloud reference architecture. And our Cisco switches are also included in the NVIDIA's cloud partner, N-C-P-R-A reference architectures. That's first promise, and we have delivered that.

The second one is Cisco will productize switch using NVIDIA's ethernet silicon. You saw a little bit famous guy signing it in the very beginning, right? So the switch is real, right?

It's getting orderable anytime soon, right? Like, uh, one of our product management peers can explain the exact date or share the exact date. And the third part is like tighter integration.

You actually saw the demo of that working. This feature has been available in ten six one, sorry, I think I said ten six two earlier. So it's ten six one, right?

Um, and you saw the real proof of that, right? So look, finally, I wanna say that a lot of times people ask that, Hey, so what does Cisco bring to the table? If you think about Cisco is the only vendor in the industry that brings and builds its own silicon, right?

Silicon one, you saw some of the examples of programmable architecture. Then we write, then we make our own switches, and then we write our own operating system, NXOS on a, not just that we have our own transceivers. That's one the entire value of the system.

The second one is the operating model. People, some customers may like on premises, not on premise, but on onpro isis. And some people may like cloud control, right?

So we have hyper fabric AI for that. And the third is proving to our customers that yes, it works, right? So you saw some of the benchmarks, uh, investment are in our benchmarking team, uh, with this medium.

I like to convey that if there's a customer who has doubt, please approach us and we will prove it to you. Right? So the intent is regardless of what GPU vendor, what storage vendor or what Nick vendor, we wanna provide a consistent Cisco, uh, networking experience to all AI environments out there.

Right now, of course, this is a probably an our or maybe a little bit more than an hour of summary. If you wanna know more, you are more than welcome to jump on a plane with me on Melbourne or, okay, come over, right? So, yeah, thank you for your time today.