75. Unified Flash Memory and Reduced HBM are Reshaping AI Training and Inference with Phison

Your hardware based memory is going away. Well, it's not going away, but maybe we can make your models fit in much larger space. Join me today for the Tech Field Day podcast.

Welcome to the Tech Field Day podcast, where we bring together a group of IT technical experts to discuss a single idea about key concepts in the industry. This podcast features a variety of perspectives from members of the tech field delegate community, and it's often recorded in association with one of our events. Tech Field Day is part of the RUM group, and this podcast is also published on a sister company site Techstrong tv.

On this episode, presented by Fon, will be discussing the premise that AI is gonna be without HBM high bandwidth memory. And that information, memory unification is changing training and inference before the discussion. Let's meet who's on the panel today.

Hi everyone. My name's Sebastian. I'm with Fon.

I'm the CTO. It's a pleasure to be here and to have this conversation. Uh, F's really excited about ai, which is a little bit unexpected because we make storage.

And you would think, why is storage excited about ai? It turns out you can do a lot with storage and AI when you have the right mindset. Hi, I'm Brian Martin, VP of AI and Data Center performance at Signal 65.

Spent my career around storage performance most recently ai. So I am excited about the confluence of these two technologies coming together. And hey everybody, I'm Max Ro.

I'm Chief Research Officer at Osteum Data Group, and I am, uh, absolutely bonkers about everything flash and, uh, silicon and, uh, storage. And of course I'm Alistair Cook. I'm an event lead here at Tech Field Day and the event lead for the AI Infrastructure Field Day, uh, series.

And that's where I brought together these three people, uh, who are my guests today. And I think we've all seen this sort of transition, uh, with storage getting faster and faster, and we source per particularly persistent storage attached to all kinds of new faster interfaces. But I don't think we quite saw it coming that flash storage was going to replace the highest speed of RAM and high bandwidth memory that's sitting on, on your GPUs.

Uh, this is definitely one of the big cost elements for any AI deployment is buying vast numbers of huge, uh, GPUs. Uh, some of these big GPUs cost as much as a low cost car, and by the time you've filled out a server with eight of them, you've got a pretty expensive piece of hardware. You start filling racks with those and yeah, you better have deep pockets and a long, well, a big payback on that.

Sebastian, you've proposed this up. What does unified storage with Flash and how's it taking away this high, high bandwidth memory requirement from ai? It's not that it, it completely removes it because you're always gonna need, uh, some amount of dram and, and especially with the scale at which, um, high-end AI works, HBM is the, the right technology for that, that field.

But there, there's kind of a, a balancing act where you do need a, a large amount of active memory, but then if you are careful with how you pipeline the requests, you can actually stream in, um, a any upcoming data that you will need before it's needed so that its residency in HBM does not have to be a hundred percent of the time, but rather page in, page out, just in time so that you can essentially reduce that HBM footprint and focus on the part that you need for active calculations and then use an a less expensive, but still very fast memory for, um, for holding that data while it's it's pending processing. Yeah, no, a, a, absolutely. The, uh, you know, and a lot of the work I've been looking at, uh, both, uh, in my day job working on H two hundreds, MI three hundreds, some of the bigger scale systems, uh, overflowing that memory and going to multiple GPUs to fill that up.

I often run into this balance of time versus memory or not being able to use the memory. I, I, I feel guilty sometimes when I'm using eight GPUs and they're running at 40% memory capacity, I'm like, I'm not doing something right. There's something I can dial up.

Uh, my home rig is an RTX 6,000 pro, uh, which I'm super happy with. Uh, and at 96 gig of ram I can do a lot with it. But when it goes, comes time to try to train, um, the models, especially as they get a little bit larger, eight gig, 70 gig, um, I don't have the GPU for it, don't have the memory.

So be able to flex out to a high speed, uh, persistent storage for caching that I can trade an overnight run, uh, to be able to get that done, uh, in my home web. And, and that's the interesting part, right? I mean, depending on who you talk to, when you initially present the idea, they're like, what?

And, and that's the trade off. 'cause we haven't mentioned that before. When you are paging things in and out, um, you, you're trading off time for cost, but it's, it's actually linear scaling.

So instead of like, so for example, if, um, I'll just make up nice round numbers. Let's say that you needed, um, a hundred gigabytes to do, uh, a model and to do inference training, and that would take you one hour if you were to, and, and that, let's just say that that was two GPUs. So if you remove it and you reduce it down to one GPU, you have half the amount of memory that would be required.

So you actually have to do things in, in batches and, and the trade off is linear. So if, what if two GPUs would do the training in one hour, one GPU will do the training in two hours. Now for some companies, they need just, you know, they, they need as fast as possible, and that's fine that that is a perfectly valid use case.

But there are a lot of other use cases where cases, it doesn't matter if it's, you know, if it takes four or five hours, it's fine for that particular use case. And so our, our proposal with what Fon has, which is our, our adaptive solution, which has a few different features, what we're basically saying is, look, if, if this is more, if you wanna balance your cost and you wanna balance, um, you know, your, your compute and the amount of memory that you need is, you know, without our solution, you really have no choice. Everything has to fit in memory or it doesn't run with our solution, you now have a choice for, you know, scenarios where you, you need that speed, absolutely go the full GPU 100% HPM solution, but for other scenarios where it can take a little longer, this now gives you a lot of options for choosing how to right size your deployment to match your needs and your budget.

Yeah. And, and, and, and, and, and Sebastian one, one of the things that I understood because I was at AI in infrastructure field day two where you were presenting, uh, adaptive, the, the solution is that if, if you are in a greenfield deployment, when you're just starting from the ground up, you have the opportunity to kind of architect everything so that it fits your need. But here, I think that the value comes from the fact that you can use that to kind of plug it into something which was already done before, and you can add in a very cost effective way, you can, you know, address those challenges of scalability of, you know, parallelizing and so on, right?

Yeah. Uh, another way to look at it is, um, if you have a scenario where, um, you already have your deployment and the, the primary use case we'll say, is an 8 billion parameter model, and then you have one department within your company that needs to, needs the, the complexity, the nuance that's available with a 70 billion parameter model, you don't have to buy it for the one use case that happens once a month. You can scale your equipment for your average use case.

And then when you have something that requires more capability, this is where adaptive comes in and lets you just basically offload that extra memory requirement to these, uh, uh, high endurance SSDs. We call ours the AI 100. Um, but, uh, the idea here is that, you know, you, you essentially scale for your, your primary use case, and then you, you, um, you leverage other memories to, to get to those, uh, special use cases, Right?

It's, you know, with, with the GPU hardware, as you said earlier, it prevents customers from having to buy equipment to satisfy their absolute max use case and then run less than when they're not doing that. And I think there's a, there's an important distinction here in training around model size, because when we're in the inference stage, the model size is the, the minimum, um, hardware based memory footprint for us to, to hold the entire model. So if you've got a 7 billion parameter model, it's gonna be roughly seven gigabytes of ram of hardware based memory.

But when you're doing training, you need rather more than that. And so if your use cases, primarily we're using the hardware to do inference, but once a month we're doing fine tuning training in order to get a, a faster response out of our, you know, normal rag system. This is a, a use case where that process for a couple of days to improve the quality of the base model might suit really nicely, but we don't wanna spend four times as much on the GPUs for that training folks.

Yeah, the, the memory footprint required, um, so if you do full fine tuned training, it's generally 20 times or 20 x, uh, the model size. So for example, a 70 billion parameter model times 10 would be 700 and times two. 4 terabytes of memory.

That's a lot of GPUs. That's about 32 of the, um, well, not, so when the memory footprint was around 48 gigabytes, uh, for the RTX 4,006,000 range, I, I don't know if they've gone up recently, but if they were, if they were around, uh, eight gigabytes, you would need 32 of those to actually do full fine tune. Now, there's another technique called Laura, which is low ranking adaptation, and typically that's two to four x the model's memory size that, that's the footprint required to train it.

And the thing about Laura is it either works or it doesn't and, and it's, it's pretty, um, pretty obvious if it didn't work. Uh, and so you just go, okay, Laura didn't work, so let's def let's fall back to regular fine tuned training, and then that memory footprint goes up. Um, but for, you know, and some, some people might say, well, it's not really 20.

It is and it isn't. So in some cases it has to be 20 because Laura didn't work and when it, but when it does work, it's two to four x, Right? And even at the two to four, like, so the new, you know, RTX Pro 6,000 Blackwell is 96 gig of memory now.

So a 70, 72 billion parameter model at four x, I still need two three cards for that. So the, the trade off and, and with, uh, you know, one or two of the AI 100, um, SSDs, you know, I can do either Laura or full fine tuning on a single GPU or a pair of GPUs, Right? And, and our solutions actually, so we have them in end two form factor, which will fit into most workstations.

And, and a lot of workstations have like two or three spare M two slots, so it's kind of nice. It, it fits in nicely, but our solution will also work on servers. And so there are people that have, for example, a server with, um, 4, 6 8 GPUs in the, you know, RTX 6,000 range or the, uh, HB 200 range.

And so even there, it has benefits because some of the other features that, that this offload capability adds is that we can, we can extend the KB cache to be much larger than what the memory footprint would allow. And that's important when you're supporting a lot of users and you're doing a lot of batch training, it allows you to service more users per, um, per box than you could otherwise. Now, when I mentioned that people are, you know, one question that comes up, and I think I talked this at a presentation recently, one of the things that comes up is, um, well, won't that be slower, because like even two gen four SSDs, that's 14 gigabytes per second, and on a workstation you've got a hundred gigabytes per second of dram.

We actually did a test with two of our SSDs and an HB 200, and we had a workload of, uh, a batch size of two 20 users with an input of 2000 tokens. And, uh, responsive about 200 tokens, which is a typical sort of one shot kind of query, um, in an environment where you're supporting multiple users. Um, and what we found is that the number of tokens per second that you can service in if, if all of the activities HBM and, and the, uh, h HP 200 does have hbm, um, versus if you're offloading to the SSD, there's less than 10% difference.

And what that really, 'cause again, this doesn't sound right, but what it comes down to is that there are other bottlenecks in the system. And so whether you're offloading to HBM or you're offloading to the SSD, you still get the same performance. But what that means is that if that HBM memory, and I, I don't remember the memory of an HB 200, but let's say it's a hundred gigs, uh, you can easily extend that to 1, 2, 3, 4 terabytes and have substantially longer context with no performance degradation.

And without that, your only solution would be to have more and more and more servers. So it's an interesting, um, it's an interesting feature that that actually adds a lot of value when you're not training. Yeah.

And depending on the number of users you have, you know, a certain token rate coming back is more than enough. You know, I can't read faster than 20 tokens per second. I love it when they come in at 40 tokens per second, I can just scan it, but, you know, 200, 300 tokens per second is, is overkill for me.

So being able to balance that out, and I'm glad you mentioned the higher context size, because these models, the open source models are now often at 1 28 K and soon expect them to go larger. And that context is key, uh, for good quality results. But even if you're not using the full 1 28 K, you know, you're going to 2, 3, 4, 5, 10,000 tokens, um, and you get a lot of that extra input either from a, a long session or a lot of input, reference documents that you're, you're putting into your, your query, even in that scenario.

Um, it, you know, even if it's not the max, it turns out that you will quickly run out of memory if, if you're only pool is the HBM, the, the other thing that we allowed to do. So we talked about, uh, oh, sorry. So I mentioned context window, um, and they kind of work together there, there's the idea of context window expansion, which is to say that maybe with the memory pool that you had, it, it's only this big, you can make it longer, but parallel to that is KD cash eviction.

So once your KD cash is full, an entry gets evicted and then disappears, and then you have to recalculate it. And that's why you see in certain scenarios where the first query is pretty quick, second query is a little slower, third query is a little slower, fourth query is even slower. And what we've actually found is that when you catch those evictions and you put them on the SSD, and then the way that our middleware works is that it first checks the, the HBM cache, is that there?

No, then it goes to check to the SSD, um, that whole flow, which intuitively would say, oh yeah, that's gonna be slow and suck. Nope, that, that's what I was referring to. There's less than a 10% degradation of fetching those missing tokens from the SSD.

Um, you know, versus had they always been resonant in the HBM memory. So, so what that means is that when you ask question one, question two, question three, question four, their speed is flat. There's no slowdown in a scenario where you have a lot of users taxing the resource of the, uh, the GPU.

So both of those go together. And there's another one that we can do in low, uh, resource environments, like typically what you see in edge ai, um, and we can actually stream instead of, normally a model is broken up kind of in vertical slices across all available GPUs. Another way to do it is to stream the model in one layer at a time.

It has about 30 layers. Um, and so yes, inference is slower, but again, in the case where you don't need maximum performance, and this is, it's, it's for that particular use case, it's okay that it's slower. You end up with a scenario where you can actually run a much larger model without quantization than what would normally be possible without this kind of offload technology, which unifies, uh, the available memory, which is either, you know, dram, VRA or HBM or unifies that memory with the pool that's available from the SSDs that are dedicated to this task.

It's not like you're using the SS SSDs to also do storage and os activity. It's, it's dedicated to the AI task, but it allows you to do a lot more with your hardware. You're clearly not super excited about this, are you, Sebastian?

Well, I think it's cool because it's great if you have infinite funds and some companies do, a lot don't, and they're getting left on the sidelines or they're being sold, oh, go to the cloud, that's fine, as long as you are comfortable with your cloud provider. And the reality is, regardless of who your provider is, um, and we've seen recent stories where some of the biggest providers have had cases where information leaks, once the info is leaked, it's leaked, and you have better control when it's in your own environment. But also you're not like those big environments have a giant bullseye on them, right?

Because they're, they're juicy. If you're an, you know, if, if you're not a Fortune 500 company, there's odd, the odds are good that nobody's even attacking your infrastructure because they don't know whether it's there. So there, there is some value in the anonymity of being a minnow in a flock as opposed to being the, the big guy right at the front that everyone's looking at.

Yeah. So the interesting thing about what you, what you just said around, you know, the ability to, uh, do stuff locally versus in the cloud and, and avoiding, you know, being too much exposed is that even if you're in 14 500, or let's say even extend that to 14, 2000 organizations, you always have the ability to, you know, go on the cloud and do stuff. But then again, uh, in some industries which are regulated, you have the risk of, you know, leaking sensitive data, right?

So it could be, uh, if we're talking about the, the pharmaceutical industry, for example, it could be research data, all kind of, you know, sensitive, personal identifiable information and so on. And most of the time people come back, you know, to infrastructure teams and tell us, you know, we need, I don't know, 4,000, 400,000, $500,000, we need to buy some servers, this and that. I think that is a really elegant solution, you know, to allow people to work on their work locally with their data on their workstations without having to go with the complexity of the cloud and the potential, you know, conflicts with, you know, uh, data, you know, data management, data governance, and with the, uh, let's say the high upfront cost.

You know, and, and, and what you also said as well, the ability to, to run stuff, different models, it gives a, a lot, a lot of flexibility our to organizations. So you can have a way to test things locally before you decide, yes, this is the right model, this is the right thing I want to use, and I can, you know, kind of industrialize what I'm doing here and bring it to the next level, perhaps. Yeah, there, there's also, um, so some, sometimes people say, Hey, I can do rag RAG, I'm good.

I don't need to do anything else. And what we found is that sure RAG is great because it gives you a concrete reference to where that information comes from. But if the model isn't trained on the data that it's referencing, it's essentially using its generic interpretation, which comes from training at large on the internet where wherever the model was originally trained, which is usually based on a bunch of stuff from Reddit, Wikipedia, and stuff like that.

And so if you train, so if you combine both, if you have the rag reference so that you can get pointer to the specific document, and you combine that with a model that was trained to understand that document a little bit better, uh, through fine tuned training, you essentially get another 10 to 15% improvement on the quality of the, the responses that it's giving you. So, you know, on top of all of the other benefits, th this is another thing, and, and our solution actually works on everything from Edge, you know, systems like, uh, something based on a Jetson all the way up to PCs, like an educational pc, which is another market that is heavily underserved, is is just universities. They don't have the, the right tools for training their, their students.

So this opens up possibilities, and we have partnerships with a few universities, um, and then, um, it goes all the way up to workstations and, and even servers. So our solution scales quite a bit and, and opens up possibilities that weren't there otherwise. Just remember to put a really good heat sink on those SSDs.

Um, yeah, in general, with these Gen four, gen five and soon Gen six SSDs and, and we make them in all classes, um, it's generally recommended to have a heat sink, uh, because essentially the, the one workload that stresses the drive the most is, is the right workflow, which happens a lot in offload. Um, and so having a good thermal solution within your, your device is very important. That's true.

And one of the things that struck me about this also was that, that you, you just mentioned middleware layer, but it really is a, it's a plugin to standard framework. It's not that I have to re-architect and rebuild my entire application, this, this just plug straight in. Uh, and I think that was, that's a, a vital part of any solution's gonna change our workflows.

You can still use the, the same processes that you'd use on your workstation with this if you maybe move into a, uh, production environment that doesn't have, or doesn't yet have this Solution in place. Yeah, and in many, we, we and our operate with most open source tools, um, right now we're on PyTorch in the PyTorch runtime. I always forget what it's called.

It, it, it's not Run Torch, but it, it's something like that executor. So we work with Executor, we work with PyTorch, uh, and, and just naturally collaborate with open source tools because essentially we're all leveraging the PyTorch framework and we, we, we operate in our space, but we don't try to block anything else. So there's a lot of opportunities to mix and match.

Thank you all for joining us today on the Tick Field Day podcast. Before we go, where can people continue this conversation, join, uh, Sebastian's energy and enthusiasm and learn a little more about us and about my gifts? Well, we, uh, we have a, um, uh, a website and a blog that talks mo about our products.

com And on LinkedIn as well. Yeah. com and again, LinkedIn Rules Supreme.

Yeah. And the fragmentation of the other social media. LinkedIn seems to be taking over.

You can find me Alistair Cook on LinkedIn as well. co nz site. So thank you very much for joining us and listening to this episode of The Tech Fields, a podcast.

And if you enjoyed this discussion, please subscribe on YouTube or your favorite podcast application so you don't miss a single episode. Do consider giving us a rating and a very nice review how much you enjoyed the enthusiasm we bring. This podcast was brought to you by Fon and Tech Field Day, the home of IT experts from across the enterprise and a part of the RUM Group for upcoming events, more episodes, head to tech field day com slash podcast, all of view us on text, on tv.

Thanks for listening and we'll see you next week.