UT07x03 – Benchmarking AI Data Infrastructure with MLCommons – Utilizing Tech

Organizations seeking to build an infrastructure stack for AI training, need to know how that data platform is going to perform. This episode of utilizing Tech presented by Soy includes Curtis Anderson, co-chair of the Storage Working Group at ML Commons. We are discussing storage benchmarking with ACE Stryker, and learning how we can know whether the given storage infrastructure is gonna perform well enough for a given ML training environment.

Welcome to Utilizing Tech, the podcast about emerging technology from Tech Field Day part of the RUM Group. This season is presented by soddy and focuses on the question of AI data infrastructure. I'm your host, Steven Foskett, organizer of the Tech Field Day event series.

Joining me today as my co-host is Mr. Ace Stryker of Solid I. Welcome to the show, ACE.

Thank you very much, Steven. How are you, sir? I'm doing pretty well.

Um, this has been going great. I'm so glad to be doing this, uh, special season with solid. I'm focused on a topic that's near and dear to my heart, which is basically how we can make storage be useful.

Uh, and I guess that's kind of what you're at here too, huh? Yeah, it's been a ton of fun so far. I've, uh, uh, we've, we've had some really interesting guests so far from a, a lot of different, uh, corners of the industry, right?

Uh, a lot of different looks at, um, the way the data infrastructure needs are, are evolving, uh, to keep up with, uh, uh, I guess AI is the, is the, is the bright shiny object today, right? And will be for some time. It is the driver of, uh, of, uh, these, these requirements and, and these, uh, efficiency issues really coming to the forefront lately.

Um, but, uh, yeah, it's, it's been a, a great journey so far. And, and I think, we'll, we've got another great guest lined up today. Yeah.

It's one of those, one of those things that, that comes up a lot just to what you just said, is basically that storage has to meet the requirements of the application. Now, that's been something that we've said forever. You know, data infrastructure, data platforms, um, you know, performance has to be, well, good enough, right?

But how would we know how good is performance? That's been a challenge in the industry for a long, long time? How do you measure performance?

How do you express those measurements, and how do you specify things that are good enough It turns out to be, uh, I think a more complicated question than a lot of folks would assume, right? Um, if you as a consumer go buy a laptop, there's a number of ready-made tools, you know, you can pull off the shelf, you can run CNA bench, you can run PC mark. You can get a pretty good sense of what you're, uh, hardware is capable of, uh, you know, pretty quickly.

Uh, and you can use that information to make relatively intuitive apples to apples comparisons, right? Between different options when it comes to things on a data center scale, right? And particularly as they look at things like data infrastructure and, and what are the requirements or the capabilities of the storage subsystem.

Um, we get a lot of questions about that. It, it turns out to be kind of a, a tough nut to crack, And it's the same for other aspects of the AI stack as well. Um, one of the, uh, organizations that I'm particularly fond of now, you'll recognize them from Field Day, from, uh, utilizing tech podcast is, uh, ML Commons.

Uh, they, uh, are really focused on answering these questions. And ML Commons, as I've mentioned previously, has a storage, uh, benchmark as well. So we have decided to invite on the podcast this week.

Uh, Curtis Anderson, who is, uh, the co-chair of the storage working group for ML Commons, and is well, probably more knowledgeable about this question than anyone. Welcome, Curtis. Yeah, well, thank you.

Thank you for having me. Um, my name's Curtis Anderson. I'm one of the co-chairs of the Emil Perf Storage working group at Emil Commons.

So tell us a little bit more about yourself and what you do with ML Commons. Um, so I'm a storage guy, not an AI person. Uh, I'm learning AI as I go along.

So, uh, that's actually an exciting piece of it, is learning the new technology, um, the, uh, storage working group attempts to benchmark storage subsystems in support of AI workloads. And so I can go a lot more detail about, but that's sort of the big picture, is I'm one of the co-chairs of that working group. Uh, so I'm here to describe what it does, how it works, and invite people to join.

Excellent. And, um, like I said, I, the, the thing that I love about ML Commons is that it is very practical. ML Commons is not interested in mythical angel storage numbers or, you know, you know, performance.

You know, let's see how many whatevers we can pile up. ML Commons is very interested in, like, how does this perform under, under workload? And it's the same with storage, right?

Yeah. The, uh, um, the, the benchmark emulates, uh, a workload, it imposes a workload on a storage subsystem, the same workload that a, a training pipeline would run, would impose on the storage. And so you get an honest to goodness, this is how your storage product would, or, or solution would perform in this real world scenario.

Curtis, can you talk a little bit more about, uh, the nature of the workload? I think in, in, in, uh, other episodes, we've explored the AI data pipeline a bit, and we've talked about there are discrete steps here, you know, uh, ingesting raw data versus preparing your training dataset versus the training itself and the inference and so forth. So what is, what is a training, uh, workload look like, and what is the benchmark asking the storage subsystem to do?

So, zoom out in the, the bigger picture, just to set some context here. Um, uh, the, uh, a, a person who wants to make use of ai, they've got a, a problem statement. They have some data they need to then start putting it together into a, a pipeline is called, um, that starts with the raw data.

Generally, it's, you know, it's video, or it's still pictures, or it's a audio or text. Um, they do some data preparation, which is changing the format of the data. They take the, the picture, the image, and they turn it into a numerical representation instead of an image proper.

It's not a, a JPEG or a p and g any longer. It's a, uh, a MPI array. Don't worry about what that means in a second.

We'll talk about that later. Um, so there's a bunch of data preparation, and then the training step, which involves that. That's the GPUs, the, uh, the, you train the neural network using that data.

Then when that's done, it goes into inference where, uh, you say, okay, I, I now have a neural network. You can tell me cats versus dogs. I show it a picture.

Is this a camera dog? What we do in the, uh, storage working group is benchmark the performance of during training during that phase of the overall, uh, workflow pipeline. Uh, we're working on adding the data preparation.

Uh, there's a bunch of cleaning and other kinds of steps that happen there. We're working on bringing that in. But right now, we're, we're starting with the simple, you know, the meat and potatoes, if you will, of the, um, of the workflow, which is the training step.

It's very data intensive. Uh, and so it puts a large stress on the storage. Is the, the decision to start with the training step.

And, and, and it sounds like moving into data prep next, um, is that because those were sort of the low hanging fruit, the easy ones to implement first? Or are those the stages of the pipeline where you're seeing the greatest storage sensitivity? Can you kind of walk us through the rationale there?

In one sense, it is, the training is easier than data preparation, because data prep is sort of unique to every different application. It's hard to develop a benchmark when there are 4,000 different ways to do something on the other. So there is that.

But, um, the, the, one of the key characteristics of the, the benchmark is, um, we measure how well the storage system performs not on the tra traditional storage benchmarks of megabytes per second and files per second. We measure on how quickly, uh, how completely the GPU can stay utilized. If the data, if the GPU ever stars for data that, you know, if the, the latest, uh, Nvidia gpu, the H 100 is like $40,000 a piece, you don't want that thing going idle because it's starred for data, right?

And so we measure, um, accelerator utilization as the core value of our benchmark. And can this storage product or solution keep up with, uh, this number of GPUs doing this particular workload? A uh, an image recognition workload is different from a recommender, which is different from, uh, a large language model.

There's many different types of, of neural network models. Um, and so they each impose a different workload on the story. So we measure them all separately.

Um, but yeah, they, we are, we are measuring can and keep the beast fed and keep the GPU busy with, with data coming in. That's sort of the core metric of the benchmark. It's a super interesting, um, choice.

And, and I think for folks who are used to benchmarks that output megabytes per second or some sort of, uh, calculated score, um, it'll be a very different look at performance, right? Um, so can, can, can the, the outputs of the test, let's say you, you stick a, a storage device or, uh, an array, you know, in the test and you run it and it says, oh, this, this storage subsystem can keep X number of, of GPUs highly utilized, swap it in for another one, and, and that other option can keep nine. It can keep y uh, uh, GPUs utilized.

Can, can that be used to make relative judgments about the suitability of storage solutions or the, the performance of, of one against another in an AI workload? I, I should say up front that the core metric is, uh, accelerator utilization, how, you know, whether the the GPU goes I or not, but you can turn that into the traditional measures like megabytes per second, that iops and all the rest of that, those informa that information's available. But the, the benchmark says, if you can't keep the device 90% utilized, then the, sorry, if you can't keep the GPU 90% utilized, then you're trying, you're overloading it, and you need to, um, uh, run this, the, the, the benchmark, again with a, a smaller number of simulated GPUs.

So that's the thing that people, uh, are gonna look at. The, um, the, the person who's gonna look at the results is someone, an AI practitioner that says, oh, I know how much data I've got. I know what type of workload I'm running.

I wanna know, does this vendor have a product, a storage product that will serve my needs? Or how big of a product from that vendor do I need to purchase in order to serve my needs? And so that's, um, the, the, the, the practitioner also knows that many accelerators of that type, they have, they have a budget given from their management, says, oh, you can buy a hundred h 100 GPUs from Nvidia, yeah, four, 400 grand, or no, four, $4 million.

That's a lot. Um, so they know those things of how much data, how many accelerators, and they want to know what, how much storage do I need to buy, and that it will actually keep up with the, the accelerator count. So that's what people vary.

Is the count of accelerators that this particular configuration can support A larger config can support more accelerators. Does that make sense how those, the, there's a bunch of things going on. But, um, the practicality as, as Steven said, the practicality is, what do I need to buy to keep my GPUs busy?

And that, that's what we talked about last season on u the utilizing tech. Um, we talked to a lot of companies in the AI space, and it really boils down to that. I mean, that's the whole ball game, basically.

You're spending a huge amount of money, uh, you said like $5 million. That's a cheap infrastructure. Yeah.

Um, you're spending a huge amount of money on very expensive GPUs. Yeah. Or, you know, ml, you know, processing asics.

Yep. And you need to keep those things fed in order to make the most of that investment. Yes.

That's the thing that matters here, because if those expensive items are waiting for data, then they're not producing results, then they're not actually giving you what you bought. And, you know, that's it. And so the question for the, the, you know, the practitioner, the person that's, that's deploying these applications that is specking these things out, they don't need to know how many 4K iops this system can handle, theoretically.

Right, right, right. At q depth of eight. You know what I mean?

They don't even know what that means. Right. What they need to know is what you're saying, which is, I bought this many of this type and I've got this much data.

Right. Will it work? Yeah.

Yes. No, right. And, and, and that's kind of the answer you're trying to give them.

Yes. Oh, that's it. The, uh, the storage industry wants to know their traditional kinds of numbers.

I'm a storage guy, so I can say this, right? I wanna know, uh, iops, I wanna know megabytes per second. But the AI practitioners, they think in terms of samples per second.

And, uh, in a distributed training environment, how many accelerators do I have? I use Accelerator, but because I try to be non nonpartisan, it's really GPUs. Nvidia is the, the dominant player in the market.

So, um, how many GPUs do I have? And so they think in those terms, and we try to bridge the, the, the, the two sets of terminology together. Curtis, you, you mentioned a minute ago that the test relies on emulated accelerators.

Right? Which I have to imagine is, uh, um, very attractive to a lot of folks. Uh, you know, that they can run this test without the need for a rack full of hardware, you know, running tens or hundreds of thousands of dollars.

Um, can you talk a little bit more about, um, you know, whether that was a deliberate choice to kind of open up the tool to a wider audience? And are there dials in there to try out Different emulated accelerators when you're running your tests? Yep.

Great question. Uh, it was an explicit decision that we made early on that most of the storage vendors they have, uh, of the vendors. 'cause we also support, uh, academic research and open source and other, other potential solutions.

So, um, but none of those people have the budget to go out and buy a hundred of the latest accelerators and it, or to try it with, um, uh, you know, other vendors besides Nvidia. So, um, we needed to emulate the, the operation of one of these accelerators. So yeah, you can, you can fire up a dedicate 10 compute nodes, uh, and your storage product or storage solution, and you'll we'll run 10 or 20 accelerators on each of those nodes, because the only thing we're doing is imposing the same, doing the reads and writes, uh, uh, from the storage to those nodes.

We're not doing anything with the data. We're not training a neural network model. We're just imposing the workload on the storage solution.

And so, yeah, you can, um, we of course had to start with NVIDIA because, you know, they're, they're the dominant player in the marketplace, but, um, we are planning on pulling in the other vendors, uh, the startups and the, the graph cores. The, the servers class kinds of machines tends torn, for example, um, to bring them in as well and show, um, so that the, a customer that wants to purchase their accelerator, instead, Nvidia can say, okay, here's the storage that I need to support that configuration of that hardware. Yeah.

And that reflects what we're seeing overall in the industry. I mean, first off, what we're seeing is that Nvidia is obviously the dominant supplier right now, so it makes sense to start there. But we are definitely hearing a lot of interest in alternative solutions be, you know, whether it's other GPUs or as I said, um, you know, asics, uh, various, you know, neural network, uh, processors.

And even as you mentioned CPUs, there certainly is a lot of excitement about CPUs that have more and more, uh, capability. And it's not in terms of, and again, this kinda gets back to the question of ML perf and the, the, the mission of ML perf, it's not about the biggest number. It's about the most appropriate solution for the task at hand.

And if the task at hand is not needing absolute maximum performance, I think that what you'll find is the, the set of things that customers are looking for are varied. So they're not gonna be, you know, if if they don't need all the performance in the universe, then they're gonna start thinking about things like efficiency and cooling and, you know, environmental impact. And, you know, literally physical space, they're gonna, of course, be looking at price.

Um, are these things that are, that the ML perf, uh, or the ML common storage working group, are gonna be addressing as well? Yes. Um, uh, I personally would love to include, uh, dollars per something right in the, in there, but that's a, a delicate subject for a lot of, uh, participants in the, in the storage business.

Right. And, uh, and we also attempt to address open source where there is no dollars. I mean, there's dollars for hardware, but not for software.

Right. And academic researcher, uh, reach academic institutions where researchers are trying to figure out how best to modify the frameworks, the pie torch, the tensor TensorFlow, uh, MXNet to, to do better IO patterns to match the, the capabilities of the storage system. So, um, well, I, because I'm a storage person, I'm used to include operating in dollars per something, but, uh, that's a, a much further out topic right now.

It's, uh, crawl, walk, run. And so we're, we're emulating the workloads on the accelerators. Then we'll bring, probably the next most important thing is to bring in the data pre-training.

Um, uh, Facebook meta, uh, did a study, uh, it's actually a report on their internal infrastructure for their AI training stack about four years ago now. And they said they spend about 50% of the total kilowatt hours of electricity on data preparation, not on training. And so that's probably the second thing we'll tackle is trying to model that.

And it's in the, the workload it imposes on storage. And then, um, and how different architectures, uh, give you different results. Uh, uh, data preparation is generally CPU bound.

They don't use the accelerator for the data prep, and, um, but they're, you know, they're, they're starting to talk about it. NVIDIA's moving that direction a little bit. I, uh, I'm not sure about the other players in the, the market.

So there's a lot of complexity, and we're gonna keep growing the core base of the, the working group to, to handle more of the storage component. So I should throw in one more thing there that, another big picture comment about ML Commons. Uh, there are like five different types of things in the world of ai.

There's, uh, data models, accelerators, storage and networking. You need some of each of those in order to, to get the value out of the ai. And they'll come and started with, uh, data and, uh, sorry, with models and accelerators about two years ago now, two and a half years ago, they added storage.

Curtis, um, the, so the, the, the roadmap for the ML perf storage test you've laid out a little bit for us, right? We're, we're focused on training today, data prep, uh, tomorrow. I, I'm curious if someone wanted to explore or evaluate the, uh, suitability of a storage subsystem for inference, for example, um, would running the, the, the higher level, you know, ml perf suite, the non storage specific tests, tell, tell a person anything about storage, are they, uh, sensitive to changes in, in storage subsystems from, you know, one run of the test to another?

Or, uh, or is that something that doesn't tend to show up on the higher level kind of, uh, system level ML perf testing? Like if you take ML PERF training, that's the benchmark for the performance of a, a piece of silicon when it's running a training task, um, that's generally compute bound. Um, and it may be more specific, the people who run that ensure that it is compute bound.

They don't want a storage subsystem slowing the performance of the benchmark of their new silicon, right? And so, um, the numbers you see in the other benchmarks at ML Commons, um, uh, won't include any impact from storage because, you know, people running the benchmark don't want that. Um, and so in that sense, they're all sort of disjoint.

But there is something we're attempting to do in the storage working group, which is, um, training will define a workload like, uh, unit 3D or, you know, that's a, a three dimensional volume classification benchmark. Um, they'll define that, that workload, and we will run the same workload to say, oh, if you're getting, if you're running, um, and we per training on that workload, here is the, the corresponding information for the, the storage subsystem. We think that that has value.

Um, it's a, you know, we need to keep track of what the other working groups are doing in order to correlate those, uh, the results that way. But ML PERF also has a bunch of, uh, inferencing benchmarks. Um, yes.

Well, first off, does storage have much of an impact on those, and is there an applicability in the future? Well, we'll see ML perf storage in that area. We don't yet see a huge impact from, uh, from inference.

Generally. The, I mean, in terms of the number of inference operations that are done globally, they're almost all done at the edge on your phone, basically. Uh, or in, in some point of sale terminal or something like that.

Um, the, the storage in that environment, you, uh, a single SSD is, you know, overkill for, uh, a, a single inference operation, right? But an SSD can be strained by a really large training operation. And so that's why we have focused on the training piece of it.

It's just a lot more storage intensive. Um, there will be points where, uh, storage will have an impact on inference, uh, but there's sort of lower down the priority stack for us at the moment. So, given the fact that, uh, yeah, as you mentioned, um, you know, that data preparation is such an important thing.

Are there standard data preparation processors or workflows that, that customers, uh, or are, are going to need to go through, uh, in order to be ready to do training? Is it as straightforward as some of the, uh, ml perf training, uh, benchmarks? Or is it, uh, a little bit different?

Well, we have seen is that data preparation is, um, pretty much unique to every application at every individual customer. There's, you know, um, uh, there's lots of different types of data preparation that sort of, sort of sub classifications, if you will. Um, you could take an image and remove noise from it, you know, a pixel noise, right?

That's one type of data preparation. Uh, there's, uh, in image processing, there's others that are, uh, taking image and I wanna rotate it 15 degrees, or I wanna change the color palette, or I want to, uh, keystone a little bit. A lot of different things you can do at the image manipulation level that in effect multiplies the amount of data you can hand to your neural network model during training.

That multiplication factor of, of doing that, uh, that data preparation, that data preparation. Um, but it's unique for, sometimes you don't wanna do rotation. Sometimes you don't want to change the color palette 'cause the color is important.

You wouldn't do that when you're looking at a stoplight, for example. Um, so, uh, no, we haven't found a, we're, we're in the process of researching that question 'cause it's very important. But we haven't yet found any taxonomy that we can describe of here's the classes of data prep that, that, that could be done.

Not Yet. Well, thanks so much for that. I think that that's a really interesting, uh, and, and, and it sounds accurate to me, uh, because I've seen certainly that, uh, that's how it is.

Every, everyone's basically bringing in different types of data from different sources and, um, you know, and, and it's, but, but even so, I think that you'll be able to come up with at least some standard workflows that, that represent the type of work that companies are doing on data preparation in order to make that also a relevant benchmark. Mm-Hmm. Um, tell us a little bit more, I guess, to, as we, as we wrap up here, tell us a little bit more about the, uh, the, sort of the nuts and bolts here.

Um, ML perf storage, uh, just like the rest of the ml perf benchmarks, um, it happens on, on sort of a regular cadence Mm-Hmm. Um, what does that look like? So where, where are you now?

And, and where does that go? And, and, and when will we see the next round of results? Sure.

Um, so I welcome anybody viewing the, the podcast to, to come join us in the working group. org, and you look for storage working group. And there's a, a link there that says Join.

So you can join the work group and show up and, and help guide the, uh, the, the project. Um, we are attempting to have two new releases per year. One in, uh, of the benchmark, one in the spring, and one in the fall.

Um, it's a struggle to get it, uh, all buttoned up, nice and tidy every time. But we've got another couple weeks, two weeks or so. 0 is ready to be run.

Uh, two months later, they'll actually be an open window where you get to submit results. And then afterward the, the results go through a peer review process, which is private to the people who submitted in order to make sure that, uh, everything, that the benchmarks were run correctly. And all of the, the i's are dotted and t's crossed and all the rest of that.

And then the results are published. So, uh, we're, we're talking about three months from today. The results will pop out, and then we'll do the same thing again in the fall.

Well, that's excellent. Uh, I can't wait to see it. Um, I will tell you that I really look forward to the briefings, uh, before the results come out.

I really look forward to combing through the results and seeing some of the stuff. I mean, ML Commons in addition to storage, also benchmarks, uh, a lot of other areas. And, uh, you know, one of my personal favorites is the, the tiny and the mobile and the edge benchmarks that they're doing to show how, um, you know, not the big data center full of GPU, but all these other systems perform, uh, very, very relevant.

Um, and, and very interesting as well. So I'll definitely be keeping an eye on that as well as, of course, the storage benchmarks coming out of it. Uh, a lot of bragging rights, uh, for a lot of different companies and a lot of different solutions.

And I think that's another thing that talks about the vibrancy of the storage industry overall. Mm-Hmm. We've got great solutions from a lot of different sources, whether it's open source or proprietary companies, and they're all, um, able to support various workloads.

So it's, it's very neat to see that answer coming out of ML perf storage as well. Yeah. Well, thank you so much for joining us.

Before we go, Curtis, where can we continue this conversation with you and with ML Commons? The, the best way is to join the working group. org is the email address that you can send email to it, I believe, from outside the working group.

But, um, there's lots of documents and, and presentations of things that the working group gets to see. So join. Great.

Thank you so much. Um, ACE, uh, thanks for, uh, joining me as the co-host today. Yeah, absolutely.

Thanks a lot Steven, and thank you, Curtis. I very much, uh, enjoy the conversation here. I expect, you know, the, the question of, uh, infrastructure efficiency and keeping your GPUs, you know, maximally used is going to be relevant for, for, uh, quite a while, uh, as we look in the future of ai.

And so having better tools to, to measure that, uh, and make informed decisions, uh, you know, uh, in service of that goal, uh, is gonna be really important. So, uh, very excited by the work you're doing and appreciate your time. And of course, uh, I know that there's a, a lot of solid Im storage in those, uh, submitted results too, but, uh, you know, that, that's, uh, it's nice to see that too.

Ace. Yeah. Yeah.

And thank you everyone for listening to this episode of, uh, utilizing tech, uh, focused on AI data infrastructure. Uh, you can find this podcast in your favorite podcast application. Uh, you'll also find us on YouTube just a search for utilizing tech or utilizing AI data infrastructure.

If you enjoyed this discussion, please do leave us a rating. Um, leave us a nice review. Uh, we'd love to hear from you as well.

This podcast was brought to you by Dyne, as well as by Tech Field Day, home of IT experts from across the enterprise. Now part of the Futurum Group. com, or find us on X, Twitter and Mastodon.

Yes, Mastodon at Utilizing Tech. Thank you very much for joining us, and we will see you next week.