MLCommons MLPerf Client Overview

Uh, hello, uh, I'm David Cantor, the, uh, one of the founders and, uh, the head of ML Perf. Uh, I'm gonna speak briefly with us about the ML PERF client Benchmark, one of our newest benchmarks that, uh, uh, came out late last year. Uh, I'm getting, I'm, I'm pleased to speak on behalf of the organization and, and, uh, the working group chairs and the community that, that worked on this and brought this together.

Um, so thank you for, uh, taking the time to join us. Um, so ML PERF is one of the, uh, ML PERF benchmarks. It's one of the newest ones.

It is focused on measuring performance of PC class systems, like the notebooks and, and PCs we're all using here. Uh, and it came out in, uh, December of 2024, so, you know, uh, under a month ago. So we just have the first release.

It's really early access in some ways. So, uh, the goal is to get the benchmark into users' hands. And unlike other ML PERF benchmarks, I've previously talked to you about training, about inference.

Uh, ml perf uh, client is actually an installable application, so anyone can go out and get it. We're looking to get feedback, learn how to improve. We've got, we, uh, started with one model and four different tests, and it is currently on Windows 64.

There's two GPU acceleration paths, and it's CLI only, although we are looking to add a GUI in the future. And as with all of the ML PERF benchmarks open source, it's available on GitHub. Uh, we will be adding in the future more hardware support, different platforms, uh, and more tests and models and gui.

So, uh, as I said, there's four tests in one model. Um, we picked a large language model to be the model that we would focus on for the first release. Those, of course, have captured everyone's attention, uh, you know, uh, deep seek earlier, uh, over the weekend being the latest, uh, example of that.

So this is gonna take in a text prompt and then produce a response. And, uh, you know, it gets used in many different ways. And so our tests, the tests, uh, uh, represent different ways that it can be used.

So we've got these four tests. We're using the, the reference model is LAMA two 7 billion. Uh, the dataset we use is Open Orca, uh, and, uh, it's a single stream benchmark.

So what that means is we're feeding in queries one at a time and looking at the response latency. And that's the primary metric, right? Just as it would be if you're sitting in, you know, your system and using it.

Right? Uh, so this, this is not a certain open Orca is the context that you use with the prompt. Is that Open Orca contains Yes.

Is is the, the, the prompts and, and, and it is also how we, uh, yes, that's exactly right. And so the tasks that we have are the content generation, creative writing, and then both light and moderate summarization. And, and, uh, you know, for folks who aren't familiar, I just wanted to walk through, you know, tokens, right?

The, the MMLU score. You wanna go back to that for Yeah. Yeah.

So, so one of the, this great, great question and prompt here. Uh, MMLU is an accuracy, uh, test. And so, uh, in all the ML perf benchmarks, we want to measure performance, but we need the accuracy to be good enough.

And so we use MMLU and, and a certain threshold there to determine what is accurate enough. And so that helps us ensure that the optimizations are generally valid. Um, right.

And we have accuracy thresholds in all of the ML perf compute benchmarks, But there's the, there's that old saying that people say like, close enough for government work. Yeah, that's right. You know, the, this is not intended to be a full production LLM, right?

That we're not in the business of doing that. That's someone else's business. And I, I think that's an important distinction is this, this is within the community.

Uh, we always have this better, faster, cheaper thing that's coming to market. Yes. This is the most impartial way to look through a lens, this particular lens.

Yes. And so where, where do you, where would you say opinion begins and ends in deciding which, you know, in that last lineup of what's fit for purpose in that, uh, cadence? Sorry.

Uh, I, so Why, why, uh, why are we deciding on ml MMLU? Yeah. Yeah.

That's, uh, one of the industry standards me measures of accuracy. And so we want to go with something that's familiar to folks, and again, you know, we're not out here trying to produce a production chat bot. And, And is it fair to say that's not a vendor benchmark?

No, it is not a vendor benchmark. This is, um, so, uh, tokens are the currency of LLMs. You input tokens, right?

Each token is, uh, you know, a fraction of a word, and then you get tokens out. Um, so the reason why this is important is when you look at the workloads in here, uh, each of them tends to take a different number of input tokens and produce a different number of output tokens. So even though you've got the same model, you're using it in different ways.

Um, again, the prompts come from Open Orca and the, the larger context lengths are, you know, significantly more computationally intensive. Um, we allow, uh, IHVs to modify the model to work on their devices. Uh, we start with LAMA two 7 billion, uh, FP 16, and then, uh, uh, we do allow down to four bit integer, uh, quantization, which we've seen being, uh, useful in various cases.

Um, but because of the impact on, on accuracy, right, we have this accuracy test using MMLU, just as we discussed. Um, so we've got the Onyx Runtime with Gen AI for direct ML for GPUs, Intel Open Vno for Intel GPUs. Um, we have, uh, uh, JSON files describing, uh, the configuration for each supported vendor and the code paths and config options they recommend.

Uh, and if you're interested, we have a reviewer's guide that we can make available to folks who want to play around with this.