Activating Tier 0 Storage Within GPU and CPU-based Compute Cluster with Hammerspace

Hi everybody. My name is Floyd Christofferson from Hammer Space, and I'm gonna talk about Tier Zero and how that directly applies to accelerating AI workflows, not only in GPU, but primarily GPU, but any GPU or CPU based cluster. So the problem is that, I'm sorry, I gotta stay on the X The problem is, is that, you know, especially anytime you put a cluster of servers together, you need to get the data to and from the compute.

And especially in a large GPU cluster, you're going over the network. Typically within each of those servers is sometimes a small amount, sometimes a large amount of local NVME. The problem is, is this is stranded capacity is the, even though it is extremely fast capacity, it's stranded data that might land on an individual GPU or CPU server.

Let's, for now, we'll just talk about GPU servers. The data that might land within that server. Um, uh, the data that might be needed by that GPUs has to be accessed by fast external storage.

Okay? A few servers. It's not such a big deal, but especially in AI workflows where context lengths are growing, where data needs to be provisioned very fast to get fast access to tokens, if every time you need to access data, you need to go over the network, that becomes a bottleneck and a problem.

So you can throw network capacity at this. You can put a 400 or maybe even 800 gig network at a tremendous cost and accelerate the speed, but you're still limited by the bottleneck. Whereas within that server, you've got, you know, storage that is significantly faster than any external storage ever could be.

So the challenge is how do you actually activate that? So what Hammer Space calls Tier Zero is literally that we aggregate all of that data into all of the, that capacity into one single, uh, tier of storage. And, but what that does now is that, you know, and this is again, coming back to the question earlier about how we orchestrate data, by being able to make the orchestration of that data programmatic or automated by workflows, it means that now rather than just jumping over that extremely high performance local capacity, we actually aggregate that.

And that becomes the primary storage for your workflows. Um, especially in a, even in a cluster of 5G PU servers. But especially when you get into a cluster like the example that Molly gave earlier, where you've got thousands of GPU servers, this can be petabytes of available capacity that is otherwise unused.

Right? And so the, the key here is, is that all of this is transparent. The automation of the workflow means that, you know, whether it's checkpointing, if you're checkpointing and you have to wait, if you, if you haven't upgraded to a framework that enables asynchronous checkpointing, it means that your G GPUs user are slowing down or stopping while you're emptying those checkpoints out of the servers onto your external storage.

Um, and I'll show you, uh, you know, I'll show you actually results of benchmarks and performance, but that becomes a huge waste of your GPU capacity or even being able to load much greater memory in there with existing infrastructure at Meta, for example. All of that, um, that LLM training that they did for LAMA one, LAMA two and LAMA three, they had a hundred gig network. They didn't put in a 400 gig or an 800 gig network.

Mm-hmm. They didn't put any of that type of, they don't need to. Right.

And this is the same thing that Tier Zero enables, is use the existing infrastructure that you have, but leverage the existing, and in most cases, sunk cost of this capacity within there. And let's not forget if you're reducing the amount of expensive tier one storage that you have to buy because you're able to activate Tier zero, and if you have automated orchestration that can move that data downstream to a tier two or tier three type storage or even cloud, um, it means that you're not adding power. 'cause this is, and you're not adding cost.

This is a sunk cost. Mm-hmm. Whether you're in the cloud or whether you're in an on-prem server, you've already paid for that local NVME.

So why not use it? What's the trade off? What's the trade off for aggregating and taking advantage of this existing non-used storage Trade off?

I don't understand the question. There isn't a trade off. What you're doing is activating a resource that is otherwise not used.

Right. But if it all of a sudden starts being used, are there any other negative consequences that can happen Often? Is there any Yeah.

Often people, I'll, I'll talk about some of this in the Linux portion later. Some people get concerned that, well, you know, by using this local storage, am I gonna increase the CPU load on those servers? And through testing and what we've seen in customer sites and through testing and benchmarks, the answer is no.

And in particular with new enhancements to Linux, the answer is no. Um, and in particularly for a checkpointing, that's not even a concern at all. So you're gonna show how this works.

Yeah. Okay. Well, I'm gonna, I don't have a demo.

No, No, I don't. No, no. That's not gonna show.

It's just, so I have, I have data on, uh, well, tier one, it gets loaded into NVME of GPU server X, Y, Z, and GPU server A, B, C needs that, but it's over on X, Y, Z. It's gotta get to a, B, C somehow. So The data, not the metadata.

I can Sure, sure, sure, sure. Yeah. So that's what I mean by how it, how Well there, there's, there's two components to this one, but they all kind of fit in with the orchestration automation that Hammer space provides.

So let's say you've got a workload that, and we can parallelize across all of that Tier zero. Um, but you may have a work, uh, uh, an application that is using that GPU server down at the end, and the data happens to be sitting up at this end. Yeah.

That east west traffic is still gonna be faster than North South, but Hammer Space also has within it the ability to do what's called ization, which means that we'll actually place that data to the particular, um, server that needs it, not just in the tier zero pool. Yeah. So you've got already an inherent, um, uh, increase of speed just by the fact that it's local NVME versus going across a network.

But then you've got through our orchestration policies, how we can intelligently place that, you know, to the correct server and in the correct place, and how much you parallelize that. And some of this also relates to what makes this possible, which is what we've put into the Linux kernel with parallel NFS. Would it be fair to say it's accelerating the data availability and latency closer to a das experience and far?

Absolutely. Yeah. Yeah.

Okay. It's much more like a das experience except the ability to scale that out across even massive clusters. But so, so Gina, that's the trade off.

It's not an, it's not an exact das experience, right? Because, you know, orchestration has to take place, but it's a lot closer to a das experience than if you had, if you hosted the whole thing in next. That's Right.

Yeah. Okay. So what we're trying to do is, I mean, look, this is an arms race and, and I'll talk about this at the end a little bit.

The arms race between faster networks, which happen to be more expensive, but also faster access to the data even within the kernel. So this, you know, every, as networks become faster, the kernel has to be more efficient. What we're trying to do is reduce the roadblocks and reduce the bottlenecks.

Um, and by simply activating this at this tier zero level, that has a huge impact. And you can see it in the results. So these are, you know, based on our testing, but let's call these theoretical results that, you know, an eight drive GPU server, you know, if we assume a 90% network efficiency, that's gonna be the rate you're gonna get to write 500 gigabytes.

It's gonna be, you know, four and a half seconds, go over a hundred gig, up to a 400 gig network, you're gonna see it goes correspondingly, slower and correspondingly more expensive. Um, if you upped that to an 800 gig network, you would be getting close. But at what cost, especially if you have a port count that goes across hundreds potentially of servers, where it gets really interesting is in the cloud, clouds storage does not have high performance networks.

You can't get those kinds of speeds that you can get in a local, in a local network. So if you're running, if you're running your jobs and you're benchmark, or you're check pointing, or you're running jobs that you need to be able to accessing across cloud networks mm-hmm. It means that your GPU efficiency, your utilization rates are gonna plummet as you're waiting for data to come in and out of your external storage.

By simply activating, with Hammer space, being activating that local tier zero storage, it means that you're able to, uh, do this at extremely fast levels, getting much greater utilization. We proved this at OCI in the Oracle Cloud by taking advantage of and have benchmarks to prove it, taking advantage of inherent local NVME that's built into those GPU servers to accelerate the performance. We also benchmarked this with ML perf.

This is, uh, ML perf results that we just showed. Um, this was, this just came out the version two results. And what we did is we, we compared, and this is, you know, certified ML perf benchmarks that show that TAM Hammer Space Tier Zero performs in the top ranks of other HPC parallel file systems.

But what's not seen there as easily is what's seen here. And that is what is your performance per rack unit. Right?

Because bear in mind when we're talking about AI projects, it's not only meta, it's not only the hyperscalers with mega size budgets, we're talking about a a, um, an AI factory that may be going into an enterprise where they have to coexist with existing environment, existing, um, uh, IT infrastructure. And so being able to reduce the power by not having to deploy that many more, especially as you scale up and out that many more storage servers, by being able to leverage existing networking rather than having to put a parallel high performance network next to this and be able to do it with that level of efficiency, that's game changing. We saw this at, and I can't name the customer, but it's a genomics research customer that has 40 petabytes of data.

They've got a large cluster of GPU servers and they were struggling. How do we actually deploy an inferencing workflow to feed those GPUs? But we spent all our budget on the GPUs, we've already got a sunk cost in the storage.

How do we do this? Well, guess what? Those GPUs all have local NVME storage, each individual server by themselves.

That storage is stranded. This is assuming, uh, uh, um, homogeneous nodes and each one's bringing its das with it. Mm-hmm.

So each one's bringing its NVM mean with it. So, so that's why the numbers match. And That's right.

It's, it's, I hate to, I hate to say this a little like linear scaling, which doesn't really exist, but, yeah. Well, and that's what we've seen. Molly referenced that earlier, even at Meta where we scale it, it is a linear scalability, and that's the power of separating the metadata path from the data path.

It is a parallel file system architecture that does scale linearly. And these don't have to be homogeneous. You don't have to have just exactly the same type of server across them.

I'm just talking about the, the synthetic, uh, uh, or the simulated benchmark. You're Oh, yeah, yeah. You're just doing homogeneous servers.

Yeah. Okay. Yeah.

Mm-hmm. Yeah, I mean that's, I mean, look, that's more a matter. You're talking about this one.

Is this, this is what you're referring to. Yeah. This is obviously assuming a homogeneous, I don't know.

Yeah. This is, this is assuming a homogeneous environment. Mm-hmm.

Just as if we can put a, a placeholder, but within an ML perf, obviously we're building the environment itself, but it has to be, you know, certified. It has to be, has, has to match this. So what makes this possible, we've talked a little bit about parallel file systems, and the key here, and I'll emphasize this over and over and over again, what Hammer Space is doing is creating a standard space environment.

And the way I like to visualize that is, you know, everything on the client side, everything that users and applications touch is standards based standard NFS standard SMB, standard n uh, S3. Um, you never have to install a, a client software on your application server or change your user experience. They simply see a protocol.

If they expect to see buckets, it, you know, the entire global namespace looks like buckets. If they expect to see a file system hierarchy, it looks like a file system hierarchy where the data lives, whether it's on an object store or on high performance storage on tier zero is irrelevant. So part of what makes this possible is what we contributed into the Linux kernel since 2018, since actually the work began earlier than that.

2 with Flex files. 1 was, was horrible. 2 was designed to change that.

And with flex files, and that is hardening it. So it can be an HPC class parallel file system, but included in every Linux kernel, in every distribution in every data center. That means getting rid of the NFS, you know, the get at or chattiness that is just the devils.

Any NFS traffic, it means adding the telemetry feedback. So the client, which is already in every data center, the client knows if you've got a slow performer over here or a network blockage over there and can adapt to that standard NFS, it chokes and fails, and that doesn't help you. Um, and then added end connect and much more by separating the data path from the metadata path you get that, you know, as close as you can get to linear scalability because the data is just direct, but also let's keep it standards based on the server side.

So standards based on the front, meaning anything looks on the back, all you need is NFS V three. And that is where, you know, the, the enhancements to K-N-F-S-D and what is in the, in actual server side means that I don't have to have some exotic client, I don't have to deploy a special type of server. I can take any storage server that is gonna have NFS, it'll always have NFS V three, and you simply plug it in and it's available at Meta.

That's exactly what we did. A thousand storage servers. They were all running standard Linux.

They all have K-N-F-S-D-N-F NFS three three, and we simply plugged in our metadata servers and we could see all of those and map that directly to 24,000 GPUs. And we didn't have to install anything on the client side, didn't have to install. So anything on the server side.

So Hammer Space keeps its magic in the middle. Mm-hmm. Um, and I'll talk a little bit more about that.

Yeah. Can you Just mention question that this is why we don't have to load anything on the GPU servers? Yeah.

I think that's important. So Gina, you, you had asked, you know, trade offs and so forth, the focus on standards means that you don't have to add anything to those GPU servers. We don't have to install anything in the client, we don't have to install anything on the servers.

Everything that is already in those servers is everything you need to do it. All you need is a hammer space that can actually orchestrate the data and unify that capacity across the servers. Okay.

I, I, I just still have questions about that because just fundamentally, and I don't believe there's no trade off, there's always a trade off or a technology choice always. And like I can remember back in the day, like I did backups for a good part of my career. Um, when they went, everybody wanted to go to agentless.

If you used did not use an agent, you could not get a granular, granular backup, um, RT om. So, um, that, that you needed, you needed an agent to get that, especially, and that's especially important when people were still managing things like Exchange online. Sure.

So, oh, okay. If, if you're saying there's no trade off, well, You know, I comfortable With that, but like, there's got to be things that you will not be able to do if, if you, if you give it up. I'm, and I'm just, I'm not saying this is not a great, this is actually great.

I think it's really awesome, but what I, I would like to know what the trade off would be. Well, Let maybe look at it this way. It was, it takes a lot less heavy lifting if we were to create our own kernel as many of our competitors do our own version of the Linux kernel and require a customer to install that version, that proprietary version, um, like installing an agent or like installing other sort of things on the client side, that's a lot easier path.

If I were to go to an investor, you know, 15 years ago and say, here's my grand idea. I'm gonna spend eight years developing and putting into the Linux kernel, all the stuff that I could do in one year. If I kept it proprietary and going through all the process of validating that through IETF and up into the Linux community, they'd say, you're crazy.

Why not just do it and then install it in the customer side? Well, the vision was if we can democratize that front end so that it actually without having to sacrifice, I mean, it takes a lot of that. You, we've got over 3000 enhancements to the Linux kernel over the last eight years that have done this.

But we've contributed those into the community so that from a customer standpoint, they don't have to install anything. It's already installed, but the heavy lifting is still there. And you know, but we've just done it in such a way that it already exists within every data center on the planet.

I hear you. Um, so the way this works is, you know, and, and again, this is very, you know, is that because now the metadata servers are connecting to all of those servers above, and we've got our standard connection down to the bottom, it means that now this is all part of the same namespace. It just, it's just Linux on the front, and it's just Linux on the back.

And the hammer space sits on the side with our metadata servers and is able to assimilate that, orchestrate that, and initiate the data movement. What that means is now that any volume that's created on the, in the GPU servers or on the storage side, simply becomes part of the hammer space global namespace, and which is abstracted behind our parallel global file system. Any protocol out the front sees any data on any storage, including two zero on the back, it's all one unified data plane.

So are you, so does Hammer Space have the capability to present any of those protocols natively Yes. Like outbound? Yes.

Okay. Yeah. Yeah.

So if I, not just NFS No, no, no. Not just NFS at all. No.

If I, I mean, we, we have a lot of, uh, customers with applications that expect to see an S3. Mm-hmm. You know, and well, guess what?

That data might be sitting on a local NVME that is accessed through n or onto a storage service is accessed through NFS, but to them, they see an S3 bucket, not a copy put into an S3 store and presenting that S3, but it's the same. It's, it, it's on whatever store. Right.

2 is when Molly was talking about moving the data from here to there, non-disruptively, well, that's live what's called live data mobility. That's, that's part of the underlying, uh, structure, which means because the metadata layer is unified, I can be moving that data behind the scenes and users wouldn't even know. You could have a live file open reading and writing on that file, and I can be moving it from one store to another, and, and the user wouldn't know.

Let me ask it. Let me ask a, uh, it feels like a dumb question, but Hammer space, wherever hosted act at more or less as an nf, uh, well, sorry. X as a client to all these protocols, that's so that when you talk about creating a volume, just like any client might, I don't know how there's lots of ways to do it.

Um, the client can, you know, by just, by doing a, uh, a, you know, um, the appropriate query mm-hmm. Um, finds the volume, and then you go through this, you know, slurping up of all the metadata on the bottom mm-hmm. Similarly with destruction.

It's, it's, yeah. So, so the, the, the tier zero element to this is to access the NVME on each GPU server, um, as if it were a storage node just Looks like a storage node. Yeah.

Okay. So, but normally, sorry, just, just so nor normally those pathways are internal. Mm-hmm.

Uh, but you're saying with no agent and no installed software, you're able to, so how do you do that? Because we've put the agent into the Linux kernel, and it just happens to be part of the standard. Well, There is an agent, it's just, it's just integral to the Linux.

It's just already, so there's no, there's no, there's no special trick to it. No. Like, there normally would be.

Well, The, the special trick, the secret sauce that makes this work is you gotta be able to orchestrate the data. Anybody? No, I just, I mean, I mean, to, to, to, to flip that, not flip it really, but to enable the, uh, what is a compute node mm-hmm.

To also appear as a storage server. I mean, hyperconverged for the purposes of hammer space. That's right.

And hyperconverged systems do that today, but each server is an island. And so what we're doing using parallel NFS is stitching all of those together into a shared pool. Right.

And, and then with our orchestration, making that transparent, so it can go upstream and downstream north south, but it's, it's the east west binding it together as a shared pool. That is where PNFS helps with that. Yeah.

And there's, there's, this is a really useful service, but as, as you know, if I'm getting down to sort of the root of where this differentiation comes from, maybe, or the value point here for Hammer Space, and I, and I, I'd like to thank Ray for helping to point this out in, in a, in a separate conversation. It's that incorporation into the Linux kernel so that there is sort of this sort of provisioning free Yeah. You're provision Linux and you're provisioning Hammer space.

Yeah. Provisioning free, uh, element of it. That's right.

And as Molly pointed out earlier, you have, uh, one of the NFS folks, and I think, uh, someone else, we've Got two Linux Kernel maintainers, Maintainers, you know, on, on staff at Hammer. I'm gonna jump ahead a little bit because I wanna, I I have a question, but can you go back for just a second? Yeah.

Because you, you're showing this as, uh, as, um, hammer space consuming, um, like an N FS three on the, the Linux boxes. Mm-hmm. Is there any reason why it couldn't consume XT out fast?

Yeah. Anything, but it's typically it's typ. Typically NFSV three is what we'll see, but we don't really care on the backend.

Okay. But it, it could be a local file system rather than a network file system. Yeah.

I mean, it depends really on the use case. And especially in a parallel NFS environment, we would typically want K-N-F-S-D on the backend, but it, it doesn't really matter. Okay.

We're gonna run out time here. So I want to give Kurt a a moment, but I do want to talk about other things that we've put into the Linux kernel. Um, and this also helps, uh, to, I think Gina's question earlier about potential downsides on that CPU utilization.

So within the Linux kernel, you know, the way that the NFS data path was constructed, it always assumed that you were going to external storage. We recently added, and this went into the, uh, six 12 kernel, uh, last November and was now in, in, in, uh, RL 10 in May. What we've done is we call a, a local io, which is basically, it's adding intelligence into the client that says, oh, my, this data storage that I'm accessing is on the same host.

It's local. I don't actually need to go through the network stack. So where this really benefits is that it reduces CPU utilization and it dramatically accelerates right.

Performance. Um, and so some of the email perf numbers that you saw before took advantage of this, and it's in the standard kernel. So it enables us, um, and this is part of a, of a, a legacy of what we've done.

We don't have time to go into all of these, but from, from the left, going back to 20 17, 20 18, up through, um, what's now coming out, Stripe Flex files and Uncashed IO improvements, I will in another time go into much more detail on these. But all of these things are contributions that we've put into the Linux kernel. Um, so that standard Linux and especially NFS, the parallel NFS and all of the related to the parallel file system gets HPC class performance.

That kind of gets back to the holy grail that, that Los Alamos wants of being able to eliminate luster, eliminate this profusion of parallel file systems. The last in that, on the far right, which is upcoming, the last two end clients to one, one file, uh, uh, this is the classic mod sim many to one type of workflow that PNFS is not good at. Today, we're really good at many to many, but the holy Grail is being able to do the many, many writers, thousands of nodes riding to a single ginormous file, which is what you'd get in classic HPC environments.

So this is a continual investment that Hammer space is making into the standard why, so that the clients don't have to ever install anything. Mm-hmm. And this is a lot of work, and it's a lot more expensive than just simply putting a proprietary client out there.

Um, but this is, this is the, the history of, of what we're doing. And, and you'll continue to see this. Mm-hmm.