Intel Gaudi 3 AI Performance Testing with Signal65

I'm part of the Signal 65 team, um, which I'll talk about in a little bit. Uh, but that's part of the Futurum group. Um, we do, uh, performance testing and benchmarking.

Uh, so today I'm gonna be talking about some of the AI testing that we've been doing, uh, specifically around Intel's gout, E three AI accelerators. Uh, so just a quick agenda, um, briefly cover, you know, who Signal 65 is. Um, a little bit of a overview of the couple projects I'm gonna be talking about.

Um, and, you know, some obligatory background that everyone's excited about ai. Um, and then two different, uh, projects, uh, that kind of flow together. I'll be talking about both on gaudy three, uh, one, uh, kind of covering on-prem and the other, uh, about gaudy three on IBM Cloud.

Uh, 'cause after all, it is Cloud Field Day. Uh, so who is Signal 65? Uh, like I said, we're part of the future ARM group, um, but we're a little different.

We're a little isolated. Um, we do, um, product testing, performance testing, um, independent, uh, data validation. Um, so sometimes it's customers coming to us, um, with their data, wanting us to validate it.

Sometimes it's, um, you know, us really, probably most of the time it's us hands-on, um, testing their project products. A lot of times that turns into, you know, a published lab report. Uh, sometimes it's just internal for, um, that company to, uh, see how they're doing.

Um, so we cover all sorts of things. I put up kind of a sampling here. Um, a lot of what we do is, you know, data center IT infrastructure, so servers, chips, storage, um, a little bit of networking sometimes, uh, lately lots of ai.

That's kind of what we've been focused on, but some other things, you know, databases. And then, uh, we do kind of have another side to it, which is more consumer products, uh, laptops, tablets, things like that. Um, but I'm really focused on the data center and the AI stuff.

So, um, I'm guessing you guys have all heard about AI a big deal. I don't wanna bore you to death with, uh, you know, everyone's excited about, about ai, right? But, uh, this stat does kind of stand out to me.

So, uh, future Intelligence did, um, a survey, uh, you know, talking to, um, CIOs of, you know, lo some of the largest companies. Um, and they found out, um, 80% of 'EM said that they're, uh, looking to conduct AI pilots or implement AI in some way. So, um, it is a big deal.

People are really excited about it. Um, but it is early, right? And there's challenges with that.

Uh, you know, how much is this gonna cost? You know, what is the performance? Can I scale it?

Um, where am I running it? Is it gonna be in the cloud? Is it gonna be on prem?

Uh, what am I running it on? Um, I think earlier in one of the sessions, Mike, uh, said something that kind of stood out to me of, I think you said, um, you know, when I think of ai, I think, you know, that's running on Nvidia GPUs. Uh, so that's kind of something we're gonna be looking at, um, in this testing of, you know, what other options are there, are there benefits?

What's the performance look like? Um, and then how does the cost translate? Um, so like I said, looking at kind of some alternatives.

Uh, Intel has their gouty three AI accelerators. Um, so we started off, um, a few months ago we did testing project, uh, looking at, you know, kind of general testing on premises TE testing, uh, comparing Intel Gouty three with Nvidia H 100. Um, important to note too, that both of these projects both are focused on AI inferencing, uh, not necessarily AI training.

That's kind of a whole, whole different thing. Um, but, so this first project we looked at two models, uh, both met LAMA models, uh, eight B model, and also a 70 B model. And then, um, following up from that, we then looked at, uh, gouty on IBM Cloud, uh, little bit broader testing there.

So, um, couple more models, uh, granite mixed roll, and then another LAMA model, but this time 4 0 5 B. So really big model there. Um, and then we also compared against not just H 100, but H 200 as well.

Um, and both of these have some lovely, uh, white papers that are up on the Signal 65, uh, website. Uh, so I'm gonna go through some of the results here, but there's more results. There's more details if anyone's interested.

How's The memory comparison? H 100. Pretty good.

Yeah. Um, I don't know the exact details, but that is one of the things that gives it pretty good performance. Why the, uh, difference in the testing criteria, or I guess, you know, models and GPUs just because, uh, that's what you had access to on prem versus in IBM m Cloud or, Um, I mean, I think a lot of times we do llama when we do these, you know, even beyond these, it's just everyone knows it.

Everyone kind of uses it, so it's a good thing to test, right? Um, so that's where we started. Uh, and then this was kind of a follow on project, so we wanted to do some different things.

Um, I think they, you know, we prioritized, uh, granite because we were running, it's an IBM model. We're running on IBM cloud. Um, and then we want to mix in mixed role.

It's a little different. It's an MOE model. Um, and then we went with, you know, really big model.

So we also have kinda like a small medium and large size model. So just showing different things. And there wasn't a single Goldie three, uh, some of them.

So the small models will be, yeah, the, the 70 B, the 4 0 5 B will both be on eight. Eight, yeah. Okay.

So it was, and eight Nvidia GPUs as well. Okay. Okay.

Yeah. Yeah, that's what I was thinking. 'cause I was like 70, you know, billions, 400 billions on one single.

I was kinda like, I'm like, okay, so eight. Yeah. Okay.

Um, so that's another thing we wanted show, And so we're co And then when you were comparing it, it was like, again, one H 100 versus eight H one hundreds, or, Uh, yeah. So it'd be like the, the configurations would be the same for both. So if we run, uh, like LAMA eight B on one H 100, or one GOUTY three.

Cool. Thank you. Um, so looking at the on-prem testing first, um, so kind of just background, you know, why do AI on-premises?

Um, that's probably, you know, the on-prem versus cloud thing could be its whole own, uh, presentation probably. But, you know, I think looking at it, the big drivers are data privacy. Uh, these customers want to, you know, take a model like lama, um, and then use their own data.

Maybe they're fine tuning it, maybe they're just setting up rag, uh, but that's their own private data. Maybe they don't want to go put that in the cloud. Uh, so that's one of the big drivers of doing it on premises.

Uh, the other one, you know, maybe cost these a, uh, cloud GPU instances can get pretty expensive. Um, but then, so highlights from the testing, uh, gaudy three, you know, pretty competitive. Um, it depends a little bit on the configuration, but you know, it ranged from 15% lower, uh, performance to 30% higher.

5 times better price performance. It's gonna be even bigger, right? Because if I remember right, quality three doesn't have these subscriptions.

So it's like purely CapEx. NVIDIA has these subscriptions, so it's opex Yes. The way that we measured this, yeah, it didn't take that into account, it was just, I'll talk about the pricing a little bit more.

'cause it's easier to do for cloud because you can see exactly what you're paying for. Yeah. On the on-prem, we have to kind of use the data that's publicly available, um, and doesn't really factor in discounts, things like that.

Hmm. Um, so, you know, there's some wiggle room in there, but there's, there's certainly a price advantage just on, just on the GPUs accelerators. Um, so to do this testing, we, uh, kind of co-created a AI testing suite, uh, with a company called Kaza, who I think, uh, really cool company.

I think they've done some of the AI field days. Uh, they're focused on making, um, basically enterprise AI platform, uh, you know, hardware agnostic, things like that. Uh, like I said, we look, this testing looks at these two LAMA models.

Um, and then there's different configurations. So, uh, smaller and larger inputs. Uh, it changes the performance a little bit, um, ranging BA batch sizes.

Um, and then within this, uh, testing suite that we've created, we tried all sorts of different, uh, inferencing frameworks. Um, and for this testing, we just chose, uh, which worked best for each platform. So, So what was the metric of, um, six For This is tokens per second.

Tokens per second. Yeah. So throughput essentially Not a first token.

Um, we did measure that, actually. You did? Is that okay?

Yeah. Uh, and there's some other things that we measured too that uhhuh weren't really the focus of, like the accuracy of the model and stuff, which isn't really necessarily hardware dependent. Um, but that's all baked into this suite, this testing suite and, And mixed drill has seven different modalities.

I mean, did you test all the MET seven modalities, or No? No. Uh, and, and this testing specifically is just these two lava models.

Um, This was the, um, on-prem, yes. That it, right. I've, I've been saying on-prem, just so I don't make that mistake.

Um, so looking at, first of all, better results here. Uh, so this is the smaller lama, the 8 billion parameter model. Um, and you, again, this is looking at tokens per second as a metric here, as higher being better or throughput.

Um, and then on the X axis you see some different input output token shapes, uh, kind of going from small, small up to, you know, large input, large output. Um, and you can see in this test, you know, gouty three doing pretty well. It's outperforming, uh, in three of the four, uh, configurations there, sorry, Mitch, where Kaza, uh, fit into this.

So that was the framework you used to run the tests that Yeah, and, and we kind of co-developed the, that whole testing suite that we used to run this, uh, with them. Okay. Uh, so we work pretty closely with them.

So, So Mitch, like, you know, it's, it's my not knowledge of like, you know, these kind of a test. So how does it work? Like you do it like once and like write it down?

Yes. You know, it's this amount of tokens per second, or are you like averaging it, or are you like, yeah, it average, How does it work? It runs, and I'm forgetting the exact numbers, but I think average is over like 10 runs or something like that.

Okay. So 10 rounds the same thing, and then you average it. Mm-hmm.

And that's your, okay. Mm-hmm. Cool.

Um, Uh, Yeah, Generated, it's sort of, uh, different. It could have the same prompt and almost, uh, different tokens be generated. I mean, how did you consolidate that?

I mean, how did you consider that? So, Um, because it, it is running multiple times, so, um, and it's a set suite, so it is, So it's more than just one prompt. It's a set of prompts that are being requested and you're generating, you, you understand the tokens that are in and put and the tokens that are out, uh, from the prompt.

Mm-hmm. And that's how you're computing tokens per second. Yes.

Uh, and the 1 28, 1 28 slash these are like input tokens versus output tokens kind of thing? Correct. Okay.

That's why it says input output out underneath it. Yes. Makes Sense.

I, I have more, I probably should have moved it up. Um, the second set of testing, I'll talk a little bit more about what those shapes are and kind of where they map to, you know, a real workload. That's my lack of knowledge, but yeah.

What, Um, you can think of it kind of as words, but, um, the words get broken up into tokens in different characters. Um, but the, the easiest way to think about it's just, you know, words per second. Um, so that's the smaller model than for The No fine tuning Done.

Done. No, this is just, this Is face models. Yeah.

Download from hugging face or whatever product. Okay. Um, in the, the suite, and we've done it for some other projects.

We do have some fine tuning testing, but we didn't do it for this. Yeah. Um, so then looking at the, you know, this larger 70 B model, uh, a little bit more competitive, um, from nvidia.

Um, right. So GDI is doing, you know, slightly better in the, this, this first configuration, um, not quite as good in the other three. Um, but it's not, so C 100 is available in a number of different, uh, form factors.

Um, this is A-P-C-I-E card, or, or is that, uh, SMIX kind of solution? I mean, and you don't, you don't show what the GDY three memory size is here. Is it 80 gig as well?

I believe so. Um, all the details will be in the paper and I apologize because, so, so for some background, everybody Has free access to the paper or Yes, just the delegates? com.

Okay. Uh, yeah, I don't know all the exact details because I didn't run this specific testing, did run the next set of testing. Um, so yeah, so point here, uh, you know, not quite as good, but it's not, you know, this huge drop off.

It's fairly competitive. Right. And, and the software stack that was, is the Kaza software stack?

I mean, CUDA was used for both, or, Or, Um, no, so it's the, This is Intel. Intel, yeah. Well, I mean, there's like, you know, a MD has a, has a version of their solution that works.

Cuda? Yeah, No, it's Intel habana, yeah. Yeah, Software.

Oh, okay. And so 128 token input, 128 tokens output. So it's like, you know, I was just like, translate.

So it's a hundred words, so a hundred words input and like output a hundred words. So you ask it like, you know, give me the answer, which has a hundred words, or how does it work? Did you, did you prompt for a specific length?

Yeah. For specific length, or did you just Oh, okay. Put tokens that No, it, it's set to be specific lengths.

Okay. So in the prompt, you basically said like, you know, I want respond with a hundred words. Yes.

Or 128 Tokens or a hundred to, yeah. Well, can it do it like I've never tried it actually. Can it?

Yeah, it will transfer. Okay. Nevermind.

One for one for the evening. Um, so yeah, so fairly competitive. Um, but then, you know, it's not just performance, right?

You need to look at, at price also. It's fairly important. Um, so like I was kind of talking about a little bit earlier, trying to, you know, nail down the pricing, completely accurate for on-premises, things get a little fuzzy because, uh, vendors working discounts, there's different configurations, et cetera.

Um, but, you know, using just publicly available data, um, keeping the base system costs the same. Um, and then adding in the GPU costs, there's a big difference here, right? Uh, so for same system, eight GPU each, um, with the H one hundreds came out a little over $300,000, uh, for gouty three little over $150,000.

So, so, but, but like, what's the term for the HH 100? What's the, it has the, it has the subscription, right? H 100.

Um, that's, I'm not sure, and that, that's not baked into this, this is just looking at the GPU price. Okay. It would Be a CapEx purchase.

Yeah. Then Yeah, In this case, yes, purchase to purchase, because If you want, if you wanna do the QD and stuff, you have to pay these, which cost. Yeah.

Substantial. Yeah, yeah, yeah. Big money.

That's not a, that's not a small cost. I think it's the, the, the, another cost of GPU. I think so, yeah.

I don't know from the top of my head. I could have a look, but yeah, it's significant. Um, so then factoring that price back in with the performance, so this is back to the LAMA eight B, um, same performance data, but now, uh, changing the metric to, uh, tokens per second, this is actually, I think a hundred tokens, or sorry, tokens per dollar.

And this is now a hundred tokens per dollar. Um, so looking at how much work you're getting done per dollar, essentially. Right?

Uh, what's your, uh, token processing rate? Uh, so, you know, in just the performance data, uh, GOUTY three is doing pretty well. Uh, you add in the fact that it's a lot cheaper, um, and suddenly you get, uh, you know, a, a pretty big advantage here.

Even in that one case where it wasn't, um, where Nvidia was outperforming it, that 20 48, 1 28, The token size needs to be a lot larger to be representative. I mean, I, who his right mind has 128 token. Well, yeah.

I'll, I'll show you in a sec. Where, where this kind of comes from. So, so min quick question.

When comparing the two options, it's great to see how much you can get done per dollar. Mm-hmm. What about power consumption?

Did you take that into account? Because if I'm a customer running this on-prem, like that's a huge constraint that I have, and it's great that I can buy twice as much for my dollar if I'm going GDY three if I can actually power it all right? Yeah.

Uh, no. So the answer to your question is no, we didn't look at power. Um, it's Like, it's like 40, I think it's 40%.

Is it, it's like, yeah, it's almost like almost 50% more efficient gdy three. Okay. In terms of the power consumption.

So you could theoretically buy twice as much GTI three for your money Yeah. And still run power if you have the same amount of Power when you think about it. Like, you know, uh, you have the Nvidia with the subscription on top of it.

Mm-hmm. So obviously Nvidia, like, you know, loads of power subscription on top of it. And then you have the GTI three, which has probably like, you know, 40 to 50 person power consumption and no subscription.

Yeah. So the difference is, I think, like, you know, it's even bigger for certain, like, you know, but then like, it depends, like, you know, these tests are great, like, you know, then it depends on this like, but I, I, I agree with, with, if you, right, like, you know, like probably the higher I get, like the third prompt, sub prompt to Claude and I'm already a thousand plus tokens and I'm not even finished yet. So it's, It, yeah.

It depends how you are using it, right? Like, you know, I mean, like, if you are using rec, then like, you know, 128 is fine, but if you are like, you know, no, it's not Fine 'cause it's 128 plus whatever the rag context comes in, it's, it's so Output. Lemme jump ahead here a little bit 'cause this is what I did for the, for the next Set.

Yeah. Fair points, fair points, Uh, testing. But this is kind of where it comes from.

So your short, short, uh, short input, short output being your a hundred twenty eight hundred twenty eight, that's something that's just text classification. Really short question, answer. Um, but yeah, it's not your full chat application.

Yeah. Um, yeah, Like, gimme a podcast on my latest blog or something like that. It's gonna be mm-hmm.

Lots of tokens. Um, yeah. So that, but that's just one case, right?

So then you have kind of your more, um, and we didn't test this shape for on-prem, but we did on the cloud, um, kind of a, you know, more medium sized, I'm calling it 10 24, 10 24, uh, that gets you into something where you could actually have, uh, more of a chat or say, Hey, generate me some code. Um, and then you get into more of what you're talking about, which is these longer inputs. Um, maybe you still have a short output of, Hey, here's a whole document, just classify, classify it for me, or tell me something, you know, really short about it, right?

Uh, when you get into these longer, you know, input outputs, uh, where, you know, 40, 96, 20 48, maybe you're doing multi turn chat, you have rag baked into it, um, or, you know, code generation going back and forth, things like that. Um, but the, the point here is just to test different scenarios. So, um, there are some things that'll be really short.

There are some things that'll be really long. The batch size here is how many prompts you batched together and sent to the L at the same time. Yeah.

Um, let's see, eight B, um, so back to the price performance here. Um, this is now again, back to the 70 B on premises. Um, also going back to your, your previous question, this eight b this was on one accelerator.

Yeah, this is on eight accelerators. Um, but so again, factoring in the price, uh, suddenly there's, you know, pretty compelling, uh, argument for, for gouty three. Um, you went from, you know, not necessarily having better raw performance factor in the price.

Um, you know, it, it's winning in, in all these scenarios, uh, benefits. I think with Ken, if you, if you back power consumption as part of this, it would even be a better comparison. Sure.

But that, that gets into even fuzzier math to where, um, you know, power doesn't cost the same. And Yeah. Um, but yeah, definitely a another good point there.

Um, which, which model did you use H 100? Was it the 96 gig or 80 gig? 80 gig.

80 gig, yeah. Um, so that was the first set of testing. Um, you know, we found, uh, I think, you know, pretty compelling results.

Um, you know, there is something that's, uh, showing competitive performance to, um, you know, nvidia, which everyone just thinks run everything on Nvidia. Uh, there are, uh, you know, price, uh, uh, arguments there as well. Um, so what if you, uh, put that in the cloud, right?

So, uh, that was the next thing we worked with them on. So in May, uh, IBM and Intel announced that they were putting gdy three in IBM cloud. Um, so what does that do for you?

Uh, maybe it doesn't help you with the data privacy side of things, like I was talking about on-premises. Um, but, you know, makes it much, uh, more scalable, easily accessible, uh, things like that. Um, and then this time, you know, we test against both H 100 and H 200 compared against H 100.

We're seeing consistently better performance. Um, and then kind of, you know, again, a back and forth with H 200. Um, but then again, the pricing is really the kicker.

Uh, so it's 30% lower, uh, cost in IBM cloud Factor in the power of that. Yeah. The IBM pays for that.

Yeah. Um, so I, I talked about this a little bit already. Uh, the one thing I'll point out here is that in this case, we used, uh, VLM, uh, for both.

Um, so we did, you know, we did all of these different, uh, configurations for all three of these models. Um, but I just have a little sampling to go through. Uh, so granite being our small model kind of stuck with, you know, that lightweight, uh, use case, uh, that you're not gonna, like, that's our, that's our small, um, you know, 1 28 and 1 28 out on one card.

Um, but what we found was that, uh, uh, gouty three, you know, looking pretty good, um, especially as the, the Bachelor grow, think, keep in mind there though, you know, that's kind of expected, uh, but your latency's gonna increase as you increase the batch size as well. Um, so that's granite. Um, then looking at mixed roll, uh, and this is kind of more of that medium use case, so maybe just, uh, standard chat, but maybe not very long chat.

Um, thing you'll notice here, uh, so this is on one card. Again, thing you'll notice here is there's only two results. Uh, so this model does not run on H 100, uh, on one card, uh, just 'cause of memory, um, memory constraints.

Uh, so this is really just a head to head with, uh, H 200. Um, and it, it's pretty competitive, uh, right. So gaudi's doing a little bit better.

And then the last batch size, um, uh, Nvidia takes a little bit of a lead, but, uh, you get up to 20%. So what Do you mean by the memory restraints? Like, this is the 7 billion, a small model, right?

Times eight though. Times eight. Oh times eight models.

Oh, okay. Okay. Gotcha.

Sorry, Sorry. Uh, no, the, the memory, the memory constraint is that it doesn't fit. Yeah.

Yeah. I didn't see the eight story. So what was the memory on the H 200 then?

I would have to look, um, certainly big enough to, at this, Something on the order of 60 billion, 60 gigabytes or something, maybe 96 or something. Uh, yeah, I, I just don't know off the top of my head. Um, and then, so this is, you know, the biggest of, of the use cases, right?

This is huge. Model 4 0 5 B, uh, and looking at the, uh, large input, large outputs. Um, so you know, really your most, uh, you know, your largest, most stressful kind of test configuration here.

Um, and in this one, um, you know, you can see that H 100 basically isn't competitive at all here. Um, it started running into, uh, KV cash issues, um, whereas H 200 is doing a little bit better. Uh, and Gaudi finally takes the lead at that very large BA batch size.

Um, but it's still, you know, very, very competitive. So again, um, looking at price, so on IBM Cloud, um, at least at the time of testing, uh, gaudy three is costing $60 an hour. Uh, both NVIDIA systems, whether it's H 100 or H 200 or $85 an hour.

Uh, so right off the bat that's 30%, uh, less expensive. Um, so looking at that same llama, uh, configuration, uh, when you factor in, uh, the price, and I'll note that this is tokens per dollar, not 100 tokens per dollar, a little different there. Um, but Gaudi three suddenly has, you know, a, a, a pretty good advantage here.

So up to 335% more than H 192% more than, uh, H 200. Uh, and then just one last set here. This is going back to mixed drill, but different configuration.

Uh, this is, you know, large input, large output, uh, once again. Uh, but I just wanted to show that, you know, we're not saying gaudi's winning in every has better performance in every, uh, single configuration. Uh, it's, you know, not doing quite as well on the top there, which is just the performance.

But you look at that bottom one, uh, where you factor in the price and, uh, it evens out. And there is, you know, this compelling reason for, for gouty three here. Uh, so just wrapping things up here, uh, kind of takeaways, considerations for, you know, these enterprises looking to use ai.

Um, there are options, and I think we'll see this, uh, you know, grow, uh, continue to grow. Things will be, continue to become more competitive. It won't be just Nvidia.

Um, you know, there are other options out there, potentially other performance or cost considerations, uh, for some of these competitors. Um, you know, it's a balance, both price and performance. Um, and then cloud versus on-prem.

Uh, again, I could probably do a whole other, uh, presentation on that. I didn't even touch on training. Um, but you know, it's kind of data privacy on one hand, uh, versus this, you know, ease of access, ease of scalability.

On the other hand, Was there like any tuning or anything done to like, uh, for, uh, the IB M1, or was it just like the default? It Was just the default? Yeah.