Demonstrating Keysight’s AI Fabric Test Methodology

Hello, everyone. Uh, my name is Alex Bortec. I'm a a lead product manager at Keysight Technologies, and I'm gonna, uh, present a AI fabric test methodology, uh, that we developed.

And, um, you know, um, I'm gonna be using, um, e product called Kai Data Center Builder who, uh, execute that methodology. Uh, as I mentioned in the previous, um, part of the presentation, I described that product, uh, very high level what the product does. It emulates AI workloads, uh, for the purposes of benchmarking the AI infrastructure.

And I'm gonna be using that product today, um, uh, to, to walk you through the, some of the elements of the methodology and, uh, showcase, uh, how can we improve, uh, network performance. Uh, the methodology high level, um, walks you through, uh, a lot of phases. You gotta go as you build and as you design and build the, uh, the fabric for ai, uh, to get to the point where the, where it can perform, great methodology itself is available for everyone for download.

At the end of the, um, slide deck, I'm gonna have a QR code. Anyone can just download it and, and, and, and, and read it. Um, so high level, uh, how we approach that.

First of all, when you design in a fabric, you have, uh, you have a lot of, uh, decisions to make, starting with what kind of topology you're gonna use for your fabric. Like, for example, are you gonna go with that traditional cloth apology or do you wanna have rail optimized topology in your data center? Depending on the topology, you might have different choices about the, uh, algorithms you're gonna use to move data around, so called collective operation algorithms.

Uh, you wanna know how those perform, uh, depending on the amount of data you wanna move. Um, it's, it's a big focus of the methodology itself, um, and how you slice, uh, the data into messages. Um, then you wanna understand that, uh, if, uh, you have multiple workloads, they don't step on each other, right?

They don't pro, they're not creating the noise neighbor problem. And even within one workload, we actually have multiple, uh, communications happening in parallel, right? So it doesn't have to be a neighbor, it can be, you know, you can be hurting yourself.

So that's the performance isolation part. Now, load balancing is a super critical aspect of, uh, of designing the network for ai. Um, and we allow you to compare, uh, how different, um, advancements in the load balancing could be, could be compared to each other, right?

So when, if you move the, from a typical five capitalis, you can be a hash to either using traffic engineering or building a load, balancing this QP aware, um, or using some other methods, right? So, uh, we describe you how to build a test bed, how to, how to set up the test so that you can compare those. And then finally, but you know, almost most important is how to do deal with congestion control.

Uh, which first comes, uh, with how a question of how to build a lossless fabric and lossless fabric is being achieved in, in raw QV two and internet using PFC priority based flow controls. And then on top of that, uh, what are the other, uh, methods to, uh, you know, control the rate at which, uh, the, uh, the GPUs move data around, uh, can be applied like KCN and DQCN. And I'm gonna actually show you how the DCN can be fine tuned, uh, to get the best performance of it right now.

Uh, a little bit of terminology that I'm gonna be using. Um, so collective preparation, uh, let's describe what it is. Collective preparation, um, is a way, um, a software that runs on every single GPU moves the data around, uh, depending on the objective.

Um, and there are different types of collective operations, but they all share similar, um, kind of core terminal, uh, you know, terms. First one is rank, rank identifies A GPU, essentially. Uh, so you can have GPU zero five, et cetera.

Um, uh, unique id, uh, then the collective size will tell you, well, how many ranks you have in total, right? You may have 16, 32, 64, and so forth. That's the collective size.

On this picture, the size is four, as you can see, there are four GPUs there. Okay? Um, and then the data size.

Uh, what is the data size? Um, it's the amount of data in memory that a single GPU has that it needs to move using that collective preparation, right? So that collective preparation that I have here is called broadcast.

It's asymmetrical. It's one GPU, sending the data it has from the memory to everyone else. So in that case, you can think it, maybe you're gonna be sending it the same data three times.

It's a very naive implementation. It could, you could, you could do better, okay? But at the end, this is like the end goal is to get my memory, uh, get the data from my memory and make sure everyone else has it.

Alex, that's the broadcast. Real quickly, sorry for the interruption, Jack Poller from Paradigm Technical. Mm-hmm.

So this collective operation, this is something that typically happens during an ai, uh, training, right? Broadcast Typically happens at the very beginning, and then there are other operations depending on what, what you have Here. So this is all in, in the effort of simulating that particular type of thing.

That happens all the time in ai. Yeah, exactly. Mm-hmm.

Thanks. Right? Um, yeah.

And then, uh, but most of the collective operations, they're symmetrical. So essentially every single GPU has the same amount of memory, uh, that they need to exchange. And depending, uh, what your AI model and workload is, you're gonna be using different collective preparations.

For example, most of the AI workloads traditionally, uh, have multiple copies of the model, and they push different data through it in parallel so that the training can finish faster, right? It's called data parallel. And they use all reduced collective operation, uh, all the time.

Now, there are a new model, new types of AI workloads that's called mixture of experts. Those types of workloads use all to all of it. Now, if you wanna benchmark your network, you actually don't need necessarily to know why one versus another, right?

The methodology allows you to make sure that all these collective preparations will perform great, and as long as they perform great, you can be sure that whatever workload you're gonna throw at the network, uh, going to be, uh, at least experienced certain level of, um, performance. Uh, now all this terminology is also public published by us with this white paper, uh, benchmarking collective operations, uh, again, downloadable. They're gonna be a QR code at the end.

Um, and so the second part of the terminology here is what exactly are we measuring? Um, so we measuring, uh, primarily three things on the high level. Uh, first thing that we measure is how much will it take to complete?

It's called collective completion time. And, uh, it starts from the very first GPU starting to send data, and the very last GPU that's gonna finish, right? So if you have gonna have some, somebody who is lagging behind, it's going to determine the overall collective collective completion time.

Now, the algorithm bandwidth, it's a metric, it's very simple. You just divide the data size you have by the completion time, and you have the algorithm bandwidth. So I kind of highlighted two areas on the slide here, um, that show what happens between the completion time and the algorithm bandwidth.

Multiple rows in this table, they demonstrate, uh, different, uh, data sizes, right? So you start from very small amount of data and then you kind of double it, double it, and double it, right? And see how the performance changes.

So what you might notice is that the completion time area that is surrounded in the red square doesn't matter that the data size doubles, it stays more or less the same, right? And it's a typical problem with the small data sizes. It's when it's, it's actually not a network that is a bottleneck.

It's all the inefficiencies inside the host to start the transmission, to move the data from one buffer to another. And then network is really fast. So like, really size of the data size doesn't matter.

It's all this inefficiencies, uh, that matter. Uh, but then at certain point, point, you start seeing that, hey, my completion time starts increasing with the data size. And that's where you actually wanna start looking into the algorithm bandwidth.

But because that's going to describe, uh, more like the performance of this, uh, collective algorithm independently from the data side at certain point. Now, there is a third metric here, uh, which actually is being used by everyone. So I describe you the two basic ones, but really, people mostly look at thematic called buzz bandwidth.

And here is why. So if you imagine two, two different systems, one has 16 GPUs and another has 32 GPUs. So to move the same data size that every single GPU has between 16 GPUs and 32 GPUs is gonna take different time.

It just 32 GPUs is gonna take more. Um, and for that reason, uh, the algorithm bandwidth gonna be different depending on the size, how many GPUs you have. So it's not very convenient to tell you, well, you know, I was able to achieve that amount of buzz bandwidth because you have to describe what was, how many GPUs I had and everything else.

Buzz bandwidth removes that from the equation, and it just tells you what was the bottleneck performance in your whole infrastructure. And so that bottleneck really doesn't depend if, if you had or 64 1000, it just specifies what is the limiting factor, um, that, uh, slow well defines how long the collective operation gonna take, right? So there's a lot of, I hope not too heavy terminology.

Uh, and I needed that so that when I'm gonna show you the application and it's going to, you're gonna see some numbers. You at least get certain, uh, understanding of what those numbers mean, right? But I'm gonna tell you this.

So if you're gonna get a data sheet or something like a presentation from a one of the companies that do AI systems, and they're gonna tell you they will able to achieve that amount of performance, they typically publish a buzz benefit. Well, this is actually much better than you just putting out the abbreviations up on a slide and having us guess at what they are. Yeah, Right?

Right? Mm-hmm. Okay.

Um, I wanna describe what kind of task bed I'm using in this demo. So you gotta get a sense of, um, uh, what it, uh, you know, essentially what are we, uh, testing here? Um, so it is a small fabric, um, out made of four, uh, switches.

Um, these are 800 gigs, you know, port speed switches, um, which I found out into run into 400, uh, 400 gig. And, uh, I connected my, uh, 81, uh, traffic generator, uh, to that fabric. So it would pretend to be a bunch of GPUs, right?

And so we are gonna be testing the performance of that fabric by emulating the GPUs and the natural cards. Alright? So I'm gonna have 16, uh, GPUs slash network cards, and the natural cards are running, uh, via emulating.

They're running at 400 K. Okay? And I made this fabric lossless and I will be able to turn on and off the congestion control and we will see how, how that impacts performance of the system.

Um, the, you know, the, the measurements, uh, we are gonna do. And, um, as you remember at the very beginning, I showed you the whole sequence of steps and the methodology, and that is like a very end. So think about that.

So I already pre-configured everything here in terms of the load balance and in terms of making it lossless, right? I decided what kind of topology I have. I'm, I'm going to run certain types of algorithms, right?

And now I'm just focusing on the very end, making sure that congestion control works. A lot of work been done before, before you go on. Yep.

CLOS is that, it's ALO apology, it's a French last name actually. Uh, yes. Uh, it just describes this type of network.

Topologist you mean spine lead? Yeah. Oh, this is spine, yeah.

Thank you. Mm-hmm. Okay.

So, so what do you do to make this lossless? You're, you're using like, uh, data center bridging or, So the, the lossless, what makes this fabric lossless is the use of, uh, PFC priority, uh, okay. Place flow control.

Uh, and what the priority Facebook control flow control does, it literally signals to the sender, the one that sends me traffic right now that I am running out of buffer. Mm-hmm. Uh, and if you don't stop, I'm gonna start losing packets because I don't have space to store them.

I, I'm congested down the line over there. So that's what PFC does, okay? Yeah.

Mm-hmm. Okay. I am familiar with it.

I just wanted see to understand because you said you were going to be turning that on and off. I'm gonna be turning on and off the ECN and D-Q-C-D-C-Q-C-L, okay? Okay.

The PFC will stay on, but we don't want necessarily PFC be the defining factor that currently, uh, that that alleviates congestion. Yes, we, there are reasons you don't wanna rely on that always. We always said that it's not a lossless network, it's a less loss network.

Yeah. Yeah, you're right. Yes it is.

There is no guarantee that it's lossless. Alright. Okay.

So let me now switch and, uh, show you some of the, uh, like tour of the product first. Alright, so this is the interface and what I have here, um, I have a bunch of tests that I did previously. I'm gonna open them up and, uh, show you the results.

Alright? So this test, okay, I'll have to hit refresh and, um, do this, I'm gonna load this test, um, that, uh, as you can see, I was running a specific collective operation called All to All among 32 GPUs, right? And I did it for a single data size of one gigabyte, uh, using roki VI two.

Now, if I'm gonna go, uh, click into the configuration, uh, and zooming into the, uh, e CN slash D congestion control, you can see that I wasn't using, uh, that, uh, capability of the modern natural cards, uh, to throttle down when they see congestion. And so in this test, I was solidly relying on what we just described, the PFC. Mm-hmm.

Okay. Okay. Let's take a look at the result that I got.

So, um, what do I have here? Um, it was a single data size, so I just have a single entry in this table and it tells me for this gigabyte of data, using all to all, uh, I got this completion time of 21 milliseconds. You can see this goes really fast.

Um, right? And I have this numbers of algorithm and the buzz bandwidth. We actually gonna be looking at the ideal percentage.

It's just easier for, for us humans to understand what it is, right? Uh, unless you like, remember all the bus bandwidth values, um, that are applicable for 400, 800, et cetera. So we know what's the theoretical ideal speed of this network could be, uh, to run this collective.

And, uh, so we, when we measure, we tell you how close you got to this theoretical maximum. So this is actually really good. Remember I told you I've spent time to configure that fabric, right?

That's why you have the good number here. Now, even though it's good, I, I got some number of EFC messages that you, they kind of highlighted in, in the red. That means the fabric was pushing back to me and saying, Hey, hold on, hold on.

Right There is congestion from time to time. You gotta like stop, pause. And then there is, I have another indicator here.

The, the fabric was actually signaling me using the ECN, it's explicit congestion notification mechanism. Uh, it was signaling me that I have a congestion throttle down, but I was ignoring it. Remember, I have this, uh, enabled turned off, right?

I was ignored, I was kept pushing. And eventually what that led to is the fabric used PFC to say, okay, just stop, right and go stop and go instead of me throat down or not. So if you look at, uh, some details, uh, of what happens here.

So in this graph, uh, I have a, a lot of data points, and that's individual pieces of data that I sliced my one gigabyte of memory into to send to another 31 GPUs from every other GPU. It's like, it's a lot of them here, right? And what I did, I, uh, plotted completion time of every single one of them, uh, in a line that is sorted.

So at the bottom, I have the fastest, and at the top I have the slowest. And if I zoom in, it kind of, you see, it's, you know, it's not vertical, right? Finished soon enough, someone finished a little bit longer, but really it is within like 20 milliseconds to 21 and a half milliseconds.

Mm-hmm. Right? It's a very, uh, small, small range.

And the other way to look at that is like, if I'm just gonna look at a single GPU, this GPU zero, remember I was saying GPUs have ranks, right? IDs, right? So ranks on number zero was sent into other 31 ranks, and they were kind of pretty much finishing around the same time, like all these lines.

Um, they pretty much in the same time around like 21 milliseconds, this is actually quite good and stable. The only thing that you kind of don't like here is that, uh, I was ignoring the congestion control notifications, uh, uh, signals, and instead was relying on the PFC. And sometimes that's not good.

Uh, every company and the data center, they made decision for themself if they wanna leave with PFC only or if they want to introduce congestion control. Some people, uh, you know, do, do not use congestion control at all, but others do. So we wanna give a tool for, to people, for example, like in this case, uh, to see what happens if I enable congestion control, if, if it's gonna have a positive or negative effect.

So let's take a look. Um, so in, in the other, in the different test that they ran here, I changed very little. I only enabled, um, the congestion control button.

However, check checkbox, right? I just enabled congestion control here on every single nick. Let's take a look at the statistics.

What it did, um, it kind of reduced the amount of congestion I have in the fabric, but the ideal percentage, and I mean, it's just dropped. Mm-hmm. Right?

So it's, it's, I mean, it's horrible right now. So congestion control doesn't help me here. Um, I got a, you know, I need to do something to do about it.

Uh, let's take a look. What happens really underneath. So the same chart, it's kind of now, I mean it's vertical in the middle, but it has to, two things.

First of all, it has some very fast flows that finish within two mil, two milliseconds. Remember there was like 21 on average. Now we have somebody who finishes in within two.

It means that those guys, they were not congested. Everyone throttle down. These guys were not, they're like, oh, I, I have all the fabric to myself now I can finish really fast.

Guess what? I don't care because I need the last one, not the first one. Uh, and I had, uh, everyone pretty much now running around 147 milliseconds.

So that's like dramatically worse, right? And some are even longer, right? So, uh, you know, you could think that this is a tail agency, uh, on top a reality.

This whole thing is tail agency. I mean, it's just like everything pushed out. Um, alright, so, and I can I even have a comparison here for you completion time between the two examples, right?

So the left very small, uh, I, I did not do congestion control. And there is another one, boom. I mean it is just, I dunno how five times, seven times.

Yeah, eight times. Um, and these are the two lines that we were looking at, right? So you see how they like relative to each other.

Um, this is what we had and we had 97%, uh, of theoretical, uh, uh, you know, performance versus this one. Alright? So, um, now what can we do?

Um, we can actually, uh, improve the congestion control, uh, by doing, um, uh, fine tuning of this algorithm. And we are kind of short on time. So what I'm gonna do, I'm gonna really quickly show you the, the, the, the end result.

So this is what the congestion control is supposed to do. When you have congestion notification, you drop and then you recover. Well, the reality is, this is not a lot of data here.

This is AI workload. You finish within 150 milliseconds, you're done. You, you didn't have time to recover.

This was made for storage. This was not made for AI with its default settings. So what you wanna do, you wanna fine tune those settings, so you just like squeeze everything so you can recover much faster.

Hello everyone. Welcome to the DCQC and tuning demo of Keyset AI Data Center builder. We conducted four experiments used in different sets of D-C-Q-C-N parameters, fine tuning for auto collective in this demo.

The DCQC and default configuration here is designed to be conservative in rate, adjustment frequency and, uh, increments to avoid causing congestion. The DC QC tune the ramp configuration, reduce, uh, the interval, and by the counter aiming to be more responsive to network conditions and increase the rate more quickly. The DC QC in tuned increased step here significantly increased the addictive, addictive and hyper steps, which are designed to improve network utilization as congestion decrease the DC qc in turn.

Final is the most aggressive configuration with the shortest interval and the smallest bite counter, along with the highest additive and, uh, hyper increased values. 8 megabyte to 64 gigabyte with a multiplier of two. Let's look at the bus bandwidth metrics.

First. The bus bandwidth of the D-C-Q-C-N default configuration represented by the blue line increased gradually as the data size grows, but at a relatively slow rate. This suggests that the conservative configuration performance is limited and does not fully utilize utilized network bandwidth.

The DC QC tune ramp configuration represented by the orange line demonstrates a significant bandwidth advantage when handling larger data size like 32 and 64 gigabyte. However, it fails to effectively utilize the network bandwidth for smaller data size, like 1 20 20 to, uh, 56 megabyte. The D-C-Q-C-N tuned increased step configuration represented by the green line, which increased the additive and hyper steps exhibits excellent bandwidth performance.

However, look at the PFC and the ECNC metrics showing a huge number of receiving counters. This may be a side effect of rapidly increasing rate, uh, leading to more congestions the DC QC into the final configuration represented by the red line performance, the best among all experiments by decreasing the ramp up interval and the bite of further, uh, when key the, uh, higher, uh, additive and hyper steps. It also, uh, eliminates the PFC and ECNC encounters indicating excellent network bandwidth utilization.

We also conducted an, uh, experiment using PFC alone for the same collective data size disabling this QC. The plot compares the results of the DC QC in tune final experiment with dose of the PFC only scenario. The two approach are quite close on bus bandwidth by observing PFC and ECM metrics with Q uh, DCQC and tune final configuration, uh, represented by the orange line here, shows fewer PFC and EC and CE than the PFC only configuration there represented by the blue line.

Two things. Why do you need a test tool to summarize? First of all, all these experiments with a test tool is much more repeatable.

Um, mm-hmm. You can really, uh, turn little things and see how they improve, uh, by repeating the experiments over and over gives you much deeper insights into what's the root cause You have less equipment really, um, and then much faster to run. I mean, it's tool made for testing.

Right. Just a quick question about the tests that you have. Are you baking in, in any of the alter ethernet, um, changes into your test tools?

Yeah. Um, so remember, um, at the very beginning of the previous, uh, presentation I was showing that we can run on servers and we can run on our traffic generators. So the first thing we are, we are already doing is we are working with ultra ethernet so that we can run on top of the network cars that are ultra and not capable.

Mm-hmm. Right? And then we are looking into bringing in the ultra net, uh, transport into our traffic generators.

So we can do both. Both, okay. Because I, it's, I mean this, this stuff that you've shown would be incredibly useful in actually helping ultra ethernet tune their algorithms and their protocols.

Yeah, We are actually a member of the consortium, so Yes, I yeah, you you brought that up and Yep. Alright. Um, yeah, so here are all the QR codes for downloads for all the stuff that I demonstrated today.

Uh, are you going to share the slides with this? The group? Yeah, I think you actually have them, but yeah.

Okay. Mm-hmm. All right.

Thank you very much. And the last thing, we're gonna have a webinar, uh, in the week starts, uh, April 3rd, the 30th in America. So then we go EMEA in the, uh, Asia, where we are gonna describe that.

And a lot of other, um, innovations that coming from Keysight in the area of ai.