GPU Memory Offload for LLM fine-tuning and inference with Phison aiDAPTIV+

Hi everyone. My name's Sebastian, CTO, and I'm here to talk to you about our AI solution and go into a little bit more detail. So, before explaining, 'cause there, there were a lot of questions that were asking, like, why are you doing this?

So maybe I should explain why we're doing this. The way large language models work today is that if you take a a, a model with seven, 7 billion parameters, that's seven gigabytes of memory. Typically, you can go up or down a little bit depending on what you do.

But as a baseline, 7 billion, seven gigabytes consumer GPUs have between 8, 12, 14, 16, maybe max 24 gigabytes of memory. So that's pretty good for running a model. No problem.

You can run 7 billion, 13 billion, anything that's under, uh, you know, if you have 24 gigabytes, anything that's under 24 billion, we'll we'll run and that's great. But if you wanna train, and I'll get to the why of it in a second, but if you wanna train, you need 20 times as much memory. And, sorry, when I say train to be precise, I mean fine tune.

When you wanna fine tune, you need 20 times as much memory. So 7 billion parameter model requires seven gigabytes to run and 140 gigabytes to train. That is well beyond what a consumer GPU has.

Mm-hmm. 4 terabytes of memory. So if you think of, like say for example you had a, uh, a 7 billion parameter model and a GPU with eight gigabytes of memory and you wanted to train that, you'd need 20 of those GPUs.

Now you're gonna fit about two to three of them. In a typical PC chassis, you can fit up to eight of them in a, in an enterprise chassis, for example. But enterprise chassis you don't have at home.

And, uh, enterprise chassis, small and medium businesses, they may have them, they may be focusing on workstations. They may have three servers for the whole company. And so when you go to your CTO or your CFO and you're basically saying, Hey, I would like to buy a bunch of extra servers to do stuff, the first question is gonna be, what's the ROI?

How do I know that after I've spent 50,000, a hundred thousand, quarter of a million dollars that I'm gonna have something that we, that we can use. And the problem is that you can't even dip your toes into the water without committing to a very expensive hardware platform. And, um, so without an alternative solution, a lot of people are just priced out of it.

So a lot of the industry is focused on inference 'cause there's no path to do fine tune, uh, an affordable path, I should say. Um, there's no affordable path of these small and medium businesses to do anything with AI other than inference. And so then a solution that came about was called RAG or RAG.

And this is the idea where you can actually reference external documents and now you can use a different kind of memory to provide a database and, and the AI model can reference the database. That's good. But it turns out you can do even better than that.

And that's where fine tuned training comes in. And the idea of fine tuned training is think of that smart person that you have in your family that just kind of has answers for everything. That smart person in your office has been with the company for 20 years.

They know how everything works, they know where all the little nitty gritty details are, and they understand why we're doing something the way that we are when it's clearly not the most efficient way to do something. And so the whole point of doing fine tuned training is that you give it all of those board documents, those presentations, those PDFs, um, and, and then you give it all of your customer service cases where people have called in and they say, I have this problem. And instead of having to type in and figure out, well, is it this or this?

And, and you're taking time to, to kind of just research your own database using the, the traditional keyword search. Instead of doing that, if you kind of just give it this problem statement, the AI kind of works the way your brain does and it finds correlations to things that may not be obvious and then presents to you a better tailored set of responses. And, and that's what we mean by when, when we're saying, Hey, why do you, you know, giving you the competitive edge, the whole point of giving you the competitive edge is to give you access to your information in a way that it's not been available before.

And, and so this is where our solution comes in. So previously I mentioned that, um, the only way to do training today is to buy essentially more GPUs with more memory. Now the thing that, that is not obvious until you really start digging into the problem is there's the, the memory is used to hold the model and the compute is used to do the, uh, the inference or the training, the speed, in other words.

So if you need to do your training in 20 minutes, yeah, load that thing up with GPUs. But if you can wait an hour for that fine tuned training or maybe overnight, then you don't need that many GPUs you could actually make due with quite a lot less compute. But right now, because compute and memory are combined together, it's the memory on the GPU card, you have no choice.

Our solution decouples that and it gives you a choice, Especially just can you go back to that? Um, the amount of, uh, GPU memory needed for training versus inference, I thought was a really cool number, but I didn't catch it. Oh, So train it says 20 multiplier typically.

Oh wow. Multiplier by 20. And and part of that is because you don't work with eight bit parameters, you work with 16 or 32 bit parameters because, and what they found is if you try to do training and your parameter size is too small, the it doesn't work, the model just gets all messed up.

And, and so you have to keep the, that size small, but there's also a bunch of other secondary tables that are maintained and, and just, you know, things to, to make it work is a combination both of the parameter size additional tables, but it's 20 x. Okay, Cool. There's another way to do that called LoRa or low ranking adaptation.

And that's about two x, uh, two to four x depending on how you set it up. And the thing is, it'll either Oh, good, make sure I didn't click it. It'll either work or it doesn't work.

That's, that's kind of the neat thing about Laura is so you, you run it through and if you see an improvement, it worked. And if it's not really doing anything, it doesn't work. So then the fallback is to go to full, full-size training.

So what did a deep seek? Was that one of the tunings they did? As far as the compression, I'm so sorry, that's not a topic that I've do, I've dove, dove into, so I I can't really talk about it authoritatively.

Okay. There's all different ways, I guess as you're saying to, to compress the size if you need to. But normally 20 pointing to one and then the other one's two to one if they, Laura, Four, depending on how you set it up.

Okay. So then are you finding that, um, full disclosure, I had to look up the difference between fine tuning and rag. Okay.

Mm-hmm. Um, full disclosure. So I, so are you fine?

And because of that, and because I do talk to a lot of people, but I don't spend all my time with the AI people, um, are you finding that companies or enterprises are not fine tuning because of this problem in terms of their data pipeline and how they're doing it? How, how they're, how they're executing on their, how they're gonna bring the model in? Yeah.

You know, it, it's, it's a bit speculation on my part because it's not like people stick up their hands and say, I made this corporate decision and this is my plan because of X or Y or Z. Yeah. Um, but I, I tend to look at it, um, from a different perspective.

It's like, what do I know? I know that train or fine tuning requires a boatload of memory, which greatly increases your cost. Mm-hmm.

It goes from buying one workstation or one server to having to buy eight. Um, and at the end of the day when you're talking about putting out a $200,000 PO or a $1 million po, then the questions are gonna start, you know, your CFO or whoever's responsible for the budget, it's gonna start asking questions like, can you explain to me why you need this? Can you explain to me why it will work?

And then you'll get grandiose statements, right? Mm-hmm. Oh, it'll help us with this, this, and this.

And then they'll say, do you have any data to support that? And the answer is no. And they can't even get the data to support that because you have, like, the ticket to enter is huge.

And, and since you can't do like a small proof of concept, um, without paying that price, often they just don't do it. Or they look for alternatives. RAG is a good alternative 'cause it starts to bring that, that domain specific knowledge in.

Um, and, and I'm not saying it doesn't work, and I've got a slide that kind of shows how the quality improves, um, as you do more specialized training. Um, so it's not that it's zero value, it's that you can do better. Yeah.

Okay. So, oh, sorry, go ahead. I was gonna say, yeah, Brian Martin, uh, signal 65, um, you know, for a flash company.

Yeah. It looks on this slide, like you've invested a tremendous amount in the software mm-hmm. Around this problem.

Yeah. Uh, and just leveraging the fact that you also happen to be an excellent flash company. Mm-hmm.

Uh, can you talk a little bit about the investments or will you talk a little bit about the investments you've made in the software stack around this? Right. So, um, uh, let me, lemme think how to answer this.

So we're starting with our SSDs and I, I do have a slide that goes into a little bit more detail on the SSDs themselves, but I can just mention that we are using, if you're familiar with nand, we're using the, the SSDs or the NAND in pseudo SLC mode. So we start with A-T-L-C-A three bid per cell, we switch it to SLC mode. And what that does, the reason you do that is, is twofold.

One is that it improves your read write speed, but two, it increases your endurance, your raw endurance by a factor of 20. So you, you go three to one, you'd think, oh, I get three times the endurance. No, you get 20 times the endurance, right?

So it, it means that you essentially as a starting point, you, you know, uh, what, what if, in other words, instead of having a, an SSD that can support one drive right per day, you can support effectively 60 drive rights per day. Um, and that's a combination of a, a few different things. But, um, so that's a good starting point.

We actually took it one step further and we will support a hundred driver rides per day, and we have a slide to explain why you need that. It's not that if you're working with smaller models and you're not doing that much training, you don't, you don't really need a hundred driver right, per day SSD. But our goal is to not create, you know, we want the user to have a good experience and having drives that wear out and dye and you have to replace 'em, like brake pads, it's gonna be pretty irritating to most users.

So we've essentially configured it so that it will, the SSDs that you buy will last the life of the, the pc or the PC or the server, which is, you know, five plus years, um, depending on how you use it. Um, so that was the intention to make sure that it, it doesn't create a negative user experience. Now, from us, to answer your original question from, from a software investment standpoint, I think we have three teams now that are working on different aspects of it.

There's one team that's working on the middleware, which we call Adaptive Link. There's another team that's working on the user interface that we call, uh, pro Suite. Um, and then we have some teams developing.

Um, we don't cover that in this presentation, but we also have teams that are developing, uh, different types of agents and mini services that leverage on top of that. For example, um, you can connect the, or we have one, um, one solution where you can basically, uh, e like it's, it's a, the premise is that you're kind of working with it, like you would a junior employee or something like that. And, and, but maybe asking tasks that are kind of unrealistic.

So you could take an email or a chain of email, send it to the AI agent that is connected to your exchange server, and then ask it, Hey, can you summarize, can you summarize this and translate it to Spanish, um, or Russian or what have you, um, and, and so, or French or, or German. And so it will, uh, and it will do that, and it comes back a few minutes later with, here you go, kind of thing. Got It.

Okay. I presume these are gen five PCI four lane, This one's a gen four, our next one will be gen five. And when gen six is commercialized, we'll we'll make, we'll have one in gen six as well.

So the adaptive link, is that something that's something you wrote? It's not an industry standard or Correct open source. Okay.

Yeah. So internally, what it's doing, um, so PyTorch has a whole bunch of hooks and callbacks and stuff like that. So essentially when, when PyTorch hits a condition, like say for example, out of memory, there's a callback that it calls.

And so leveraging the capabilities within PyTorch, we have a bunch of callbacks and intercepts that essentially allow this or enable this to work. So I'm, I've been stuck on this slide quite a bit. Uh, I'd like to jump forward just a little bit and, and, uh, and I think this will start to address and provide a little bit more color to how all of this works together.

So the basic ideas that the middleware is sitting in the PyTorch, it's intercepting callbacks, and it's between the GPU and the SSD layer, the hardware layer. And so when, when, when PyTorch hits a situation where normally it would throw an error, our middleware comes in and essentially provides a handler for that. And, um, makes it so that at a high level, um, the, the PyTorch basically never runs out of memory.

And it, it is, uh, shoot, how can I say this? PyTorch never runs out of memory and it provides a seamless expansion to the memory pool without having to have a one-to-one correlation with GPUs. And so, Just a quick question on that.

Yeah. Um, it's, uh, tell me how adaptive, uh, link or adaptive Cache is different than just a normal os uh, swapper. Uh, I mean, fundamentally it's a swap file if you think of it.

You know, and you kind of get rid of all the fancy words. It's a swap file. So, but the thing is, the swap file is designed to run on the operating system.

It's not designed to run on the GPU, and there's no integration with PyTorch. You might think, well, why doesn't PyTorch added? They didn't, there was a time where, um, OSS didn't have a swap file and eventually it was added.

So, you know, eventually will it become a core service, possibly, um, it's hard to say, but currently it's not a service that's available through PyTorch. Okay. And I mean, it's, you use traditional swap algorithms, or do you, uh, I mean, how, how do you, what type of swap algorithms do you typically use for it?

Oh, I've got a slide on that. Okay. So, uh, I'll, I'll get to it and I think the picture will help.

I, I could explain it verbally, but I think having the picture to support it will will help. So if you could bear with me, I'll get, Okay. Yeah.

It's, you know, I, I've been listening and, and you're talking about very interesting stuff, but I'm, I'm getting the point of, I wanna hear how you do it. Yep, Totally get it. So, uh, I think it's like about four slides later.

So this one, I'll, I'll just cover this one really quickly. This addresses the question, Hey, why do you need a hundred driver rights per day? Who the heck needs a hundred driver rights per day?

Geez, guys. So if you're looking at, um, 6,000 ada, which is about a two, $3,000 GPU, I believe. Um, so if you're talking about a, a workstation that has four 6,000 adas, um, uh, if you were to do training 24 7, which we know is not realistic, right?

We're looking at the worst, worst case, can you burn this thing out? And the short answer is no. Um, by design, we wanted to make sure that you couldn't.

So if you were to do 24 7 training with four, uh, with four 6,000 adas, um, you would need about 15 drive rights per day. And that's true for the W 7,900 Pro from, uh, from a MD. Um, and then if you look at the server configuration where you have eight of them, it just basically doubles.

Um, and if you're looking at enterprise grade GPUs, then they're a lot faster. Um, and so you can do about 60, you, you need about 60 drive rights per day to support H one hundreds, MI three hundreds, and gouty three, for example. So the point here is to reassure everyone, Hey, you're not going, you know, put this drive and then have it die after six months and then have to buy another drive and another drive and another drive.

This one, you buy the system. It's good for the life of the system. And, you know, then you can just focus on using the system instead of maintaining the system.

Our pro suite, and, and I'm about to go into the details of how it works internally, but the whole, the whole, we, we started off with just a command line driven thing. And the, the feedback that we got very quickly is, Hey, for this target market, people really want something that's easy to use. They want it to be no code gui, they wanna be able to click on things and just say, you take care of all the little nitty gritty details, and then I want you to take these files, run them through the training session, and then, uh, you know, do the fine tuning and then generate the file and give it to me.

Because ultimately these large language model file or or large language models are like eight gigabyte files or 70 gigabyte files. Um, so the, the whole purpose of our GUI is to make that easy. And, uh, and then coupling that with the training that we offer, um, the idea is that with a couple hours of training, pretty much anybody in the office can be set up to be the person who kind of takes the documents and, uh, puts them in to run.

And then, uh, with some of our partners, they're working on automating that solution so that when you take the files, for example, and you put them in this folder, then the whole system picks it up and does the training automatically and nobody's involved. It just happens and the model gets smarter over time. Um, so this goes into the, the why.

Um, if you, we, we did a scenario where we had a set of questions, um, that were industry specific. I think they were related to demand, and we asked some questions of GPT-4 oh. And basically the more blue and gold that you have is good, the more green and silver that you have, it's bad because you know, the 20% or less answer 40%, you know, 20 to 40% quality answer.

Um, and so what we see is GPT-4 oh, even asking questions that are quite spec industry specific is not bad. Um, if we take LAMA 70 B and we just do RAG, oh, sorry, GPT-4 oh with Reg, um, it's not bad. Um, if you take LAMA 70 B and you just take it off the shelf and then you connect it to your, your rag infrastructure, it's also pretty good.

But you, you do have some of those low quality answers. And what we found is if you do lower or lower and fine tune, the amount of good answers increases substantially. And the thing is, when you have an answer that's like 60 or 40% correct, that's like, you know, if you have that relative in the family that kind of understands computers but doesn't, and then they try to talk about technology and you're like, you're kind of ripe, but not really.

You wanna avoid that. You want those low quality answers to be smaller. And so the point that we're trying to make here is that you can take an off the shelf open source model and essentially end up with results that are better than this big giant industry.

You know, this big company with GPT-4 oh and GPT-4 oh is great, but if you're asking domain specific questions, there's an advantage of having that model actually be trained on your data. And this goes into the details that I think you are asking about. 4 terabytes of memory to train.

What what PyTorch does is it, it looks at the GPUs that you have. And in this scenario, we we're proposing GPUs that have 48 gigabytes of memory. So they're those mid-tier pro-grade GPUs.

They're a little bit bigger than what you see on the consumer side. 4 terabytes of memory needs about 32 GPUs to fit. 4 terabytes and you divide it by 48, you end up with 30 31.

And then you actually need a little bit of space to, to put your training data into it. So you end up with about a pool of 32 GPUs, and if you're taking workstations, that's eight workstations. And if each workstation costs 30, 40, $50,000, multiply that by eight, that's your cost to get in the door to be doing, to be able to do training with these larger models.

Um, and so normally what PyTorch will do is it'll slice up your model into 30 32 slices and then distribute them across the GPUs, and then, uh, then the, somebody hits the go button, and then you feed in your first ba first batch of data, it does the training feed in your second batch of data, it, it does more updates and so forth until you've gotten through all of your data. What our solution does essentially is it works with the GPUs that you have. So in this configuration, we just took one note and you have four GPUs.

Each of them have, um, you know, 48 gigabytes of memory. Like everything's the same. We're just using one node instead of eight.

So from PyTorch perspective, it doesn't really know or care, it just responds to all of the calls that it makes to discover the capabilities of the system. It just goes, oh, okay, I'm great. I'm, I'm not gonna run out of memory.

And so it slices up the model and then puts the first four slices on the GPU and all the slices that don't have a home will sit on the SSDs. And then as soon as one of the GPUs finishes doing that round of training, it says, Hey, I'm done. Our middleware comes in, takes all the data off the GPU, puts it on the SSD, and then takes the next pending slice and puts it on the GPU Gathering.

That the big value here is cost savings, space savings. You're gonna get some performance improvement by not checkpointing across a field of workstations, but you're gonna take slightly longer 'cause you're paging these things in and out. That's, you have a trade off slide that shows Yep.

And I'll, I'll just sort of, um, tell you where this is going. The, the trade off is, it's kind of neat. It's, thankfully it's linear.

So if a job would've taken eight GPUs would take an hour. If you cut those GPUs in half, you have four. It doubles the time.

So if you have the amount of GPUs, you double the time. Uh, yeah. If the GPUs is flat out, if, if I'm not doing it for the memory and the GPUs is not flat out, you're assuming if the GPUs is running at a hundred percent, then it's linear.

Um, right. I thought one of the propositions earlier was I did all this for the memory. Mm-hmm.

But I can't feed the GPUs. The GPUs are running at say 50%. 'cause I'm really there for the memory or staying in training.

They're pretty much always flat out Sort of, um, you, you end up being, so the, if you look at the capabilities of the tenor cores on, on media GPUs, for example, the tensor core is what does the MA matrix multiplication? Yep. So the matrix multiplication, like if the data that it needs to work on was sitting in, its, its L one cache or directly in the registers, it's insanely fast.

But the reality is that it'll do the calculation and then it waits for the next round of data to come from the, the memory, whether it's, it's uh, uh, GDDR or VDDR, it goes by different names, whether it's the GPU regular DRAM or a high bandwidth memory, you're talking about 100 gigabytes per second versus 800 gigabytes per second per channel. But even though those numbers are really huge, they're substantially slower than than what the, the, the, the gate logic can do. So no matter what, the gate logic's kind of always waiting.

Um, and, and you can't get around that unless you could have something even faster than HMB, like a hundred times faster than maybe you might be able to saturate the compute capabilities. So when they talk about LLM, they're always, all L LLMs are fundamentally memory bound, which is memory is the bottleneck. You either don't have enough of it, or it's not fast enough.

Even when it's HMB, it's still not fast enough to saturate the compute capabilities. Okay. Um, and so this kind of talks about the linear trade-off scenario.

So in this scenario, we're, and the, the system configuration is in the bottom right, but we're basically assuming a workstation with, uh, four GPUs. And each four GPUs is I think 48 gigabytes of memory, which works out to a pool of 192 gigabytes across the four GPUs. So in that scenario, you, uh, if you're trading a 7 billion parameter model and you multiply by 20, that's 140 gigabytes.

And that fits nicely within the 1 92. It consumes about 68% of it. You're good to go.

You can, on this configuration, you can train a 7 billion parameter model. You don't need our solution. It'll work just fine with the, the native solution.

So the orange represents the, the, the configuration using adaptive, and the blue represents the configuration using the, uh, GPU only. And so in this case, you, you need four, four GPUs and, and you have it in both situations. If you say wanted to train a 13 billion parameter model without our solution, it doesn't run, um, you just get an out of memory error.

You would actually need six GPUs, um, for to, for that, for that to have enough memory to train normally. But with ours, it keeps the four GPUs. And what you see is it takes a little bit longer.

It's, it's a little bit harder to see, uh, on the seven to 13. 4 hours, that's about a 10 x multiplier. Not perfectly but close.

Um, and so, and so seven B and 70 B uh, 70 billion is 10 times bigger, and the time is 10 times longer. So it's, it's what I was referring to from a linear scaling standpoint. 4 terabytes.

4 terabyte configuration, you need 30 32 GPUs. And this is where we're talking about the costs add up quickly. Now, the next question somebody would bring up, and you'll see this in the forum, is, hey, why don't we just all stick with 7 billion?

And it kind of comes down to the nature of the questions you want to ask for very specific tailored tasks. Often one, three and 7 billion are just fine. But if you're trying to ask it more nuanced questions, give it really long prompts, you generally get substantially better results with larger models.

We did the same thing in an enterprise configuration with HG eight GPUs. Here we're using a six thousands. But fundamentally it's, it's the same result.

The the main thing that we want to emphasize here is that there's, uh, I mean, you could kind of say like, you, there, there's no reason why you can't train 180 billion parameter model on four GPUs or eight GPUs. What you're trading off is time. So here we double the number of GPUs.

So, um, uh, the, the impact is, is you, you would think if I go from a 70 billion to 180 billion, which is roughly two x, it should take 20 hours and it would, but we double the number of GPUs in this situation. So it's 10. Right?

Do do you have a chart? What I'd love to see in this chart Yeah. Is the cost delta, 'cause I imagine it's a dramatic cost difference.

Uh, if I had what, 75 GPUs on the right, what would it take to do that training? Would it be half the time, a third of the time, would it be the same 10 hours? So if you had 75 GPUs, it would take you about 40 minutes.

Okay. So, and that's the thing, right? If you need the data, if you need the training to finish in 40 minutes right.

Buy the 75 GPU system, but it comes with a 75 GPU budget. Right? And if you like, and if you need it, it is what it is.

But a lot of organizations like, 'cause depending on who I talk to, when you talk to the people that absolutely, absolutely need speed, they kind of get a little bit angry. It's like, why would I want this? This is garbage.

I don't want slower, this is dumb. But when I talk to small businesses that don't have that need, they're like, wow, four hours and I save how much money? That's great.

Yeah. That's, I wanna see that slide. Yeah, I wanna see the dollar slide.

You, you, uh, I don't, I didn't make that slide, but basically if we take this one, you can assume about $40,000 per box. So in, in the, uh, 30 GPU configuration, it's, it's eight boxes. So it'd be eight times 30, $40,000 kind of thing.

That feel Yeah. That feels compelling. Yeah.

And, and servers are a little bit more expensive. They're in the, the 50 to $60,000 range. So it'd be, you know, 60,000 per server.

And each server can hold about eight boxes. So you need 10 of those. That brings you to the 500,000, $600,000 price range.

Or with ours, you can get by with one and you don't have to put eight GPUs in there if you're okay for it to, you know, start Monday and be done Wednesday. 'cause it just doesn't matter. Like, it's not like your AI is dead.

Uh, your AI on your deployment server that's doing the inference, it's still running. This is the update. Right.

Maybe the update can come Tuesday, and if it can, you don't need eight GPUs. Maybe you have four GPUs in that box, and each GPU is about three to $4,000 in this price range. So that saves you another $20,000.

Right. And power and cooling. Mm-hmm.

So a question here is that I think if I understand it correctly, what you're really doing is adding a SSD backed swap cash Yep. To a pipeline that didn't include it originally. Correct?

Mm-hmm. Okay. Which makes perfect sense.

As an old systems guy, I get that completely when we design, right? It's L one, L two. Now you've got an, an SSD backed three and L three Three cash, Right?

That's, uh, speeding up the pipeline. Mm-hmm. What is it, other than just adding that to the system, is there something special about Fon hardware, Fon technology that we're leveraging here?

Or is it we, you know, that's where I'm trying to get what's, what's fons special sauce in this? Sure. Well, the basic special sauce, uh, I, I'll answer that in two parts.

The basic special sauce is one, we enable it. Two, we provide commercial support. Mm-hmm.

Kind of like anyone can download Linux, but people pay for redhead Linux to get the support contract that comes with it. So we're offering that capability, which to a lot of organizations who don't have a really big IT team, that that's worthwhile Absolutely valuable. Don't wanna denigrate it in any way.

I'm just trying to understand what you're bringing to the table. Right. 0 solution was, Hey, let's bring this to the market.

Gotcha. 0 solution, because fi you know, we, we tend to describe ourselves a controller company, but we're not really a controller company. Yes.

We make the controller, we make the firmware, we make the module, we make full complete, you know, SSDs and take those to the market. But that controller is all of our own ip. So wi within, uh, uh, uh, uh, integrated circuit, the each functional block is often called intellectual property.

Yeah. That's what IP stands for. Um, so all the IP that goes in it, we design in-house, which means that we can make it do anything.

And so we're actually looking at, in the process of developing adding compute capabilities to the storage controller to actually offload some of the tasks that the GPU does. Obviously not the big matrix mm-hmm. Operations, we No, you, you need matrix engines for that and Right.

Um, we don't do that. But if you look at other steps in the training pipeline, there are things that you could actually just offload to the SST. You can offload them to the CPU as well.

Right. But you don't have to in our situation. And, and so when you're looking at like a, a workstation that has four GPUs, um, and if you're training really large models, you might have 2, 4, 6 of these, uh, offload a AI 100 SSDs, or the upcoming AI 200, I think is what the next one would be called.

Um, and, and so if those can also essentially act as a compliment to the GPUs, you start to go beyond what, uh, other offs sold solutions offer. Because ours comes not only with the offload capability and the commercial support, but now it also comes with each of those SSDs is doing work in the AI pipeline. Gotcha.

And there's a lot, there's that, that would be huge. There's a lot of value in that. 'cause I could foresee you could do something like, uh, A DMA out of the SS d cash mm-hmm.

To the backing store without going through main memory or, or CPU That helps. It turns out it helps less, uh, 'cause we looked into it, right? Um, of the total workload, like most of the time is spent waiting for the GPU to finish.

Right. The transfer time is about 5%. And even if you were to make that 50% faster mm-hmm.

It's a 2% improvement, a net 2% improvement. So it's, it's, we'll do it eventually, but it's not the top priority. There are other things that we can do that will provide even bigger improvements in performance, like adding compute capabilities to PSSD, which can add like a 30 40% improvement.

Gotcha. Go, go ahead. So, So far, Jim Zinsky ZDC, um, sounds like, uh, unicorns and magic fairy dust, to me, it just seems extremely wonderful.

Mm-hmm. Let's talk nitty gritty. How much more or less power and cooling does your solution bring to the table versus like you're saying, right, instead of buying 20 GPUs, I'll buy four.

Yep. So is there a compelling case from an energy consumption or energy, uh, lower energy consumption and power and cooling? It depends on how you look at it.

So there are two ways to look at it. If you look at a total power, right? If I, if I take, if I make something, take half the power, but I make it run twice as long, that's neutral.

It's the same. Good point. You've burned the same number of JUULs, the same number of watts kind of thing.

Sure. But if you look at it from how much power do I have to supply to the box and your whole power delivery infrastructure, an eight GPU box starts to need like a lot of watts of power. And so often you're moving from the one 20 to the two 40.

You've got a power delivery unit that adds a couple thousand dollars to your rack kind of thing. Um, and plus your, your, your data center or your space, if you're a smaller company, it's, it's that room that has the computers in it with a little bit of cooling. Um, you have to be able to have that power.

So with our solution, the, the advantage of having less GPUs is you can actually, if, if that's a constraint for your deployment, you can tune the power consumption of the, the instantaneous power consumption of the box, um, to match what your circuit can support. Mm Mm Okay. So, so related to that, sorry, related to that, can you tell us what your sustainability story is?

What metrics you follow, what standards? Is there anything, you know, other than the normal we do our best? Well, there's, there's a yes.

Um, so one of the things that people talk about, um, but kind of never goes anywhere. Mm-hmm. Um, and unfortunately it's, it's not gonna start next week kind of thing, but a lot of these SSDs, when the system is retired, they're not dead.

They have maybe 20% wear on them. They're still 80% good. And so one of the things that we're really looking into, uh, on the consumer side for sure, but mainly on the enterprise side.

But once we do one, we can do the other is basically just getting those SSDs back, depopulating the NAND, testing it to make sure it's good, and then making a different tier of product to the point of, you know, instead of throwing away nan that is 80% good and shipping it off to the landfill. There are other use cases where the demands are not so high. Like obviously we wouldn't put it back into another ISSD, which is like super Right.

Heavy. Yeah. And, and just wants the fastest performance available from the SSD.

So, but there are plenty of other scenarios where maybe it's warm storage, maybe it's cold storage, where it's primarily red occasionally. That's great. You just need the capacity and not so much the, the highest.

'cause also when these SSDs are retired, they're five, six years old, so they're not, you know, if if we're looking at five, 6-year-old ssd, we're talking about gen three, that's at best three gigabytes per second. Mm-hmm. And for some use cases, that's fine, but for others it's not.

So that's one of the things that we're looking into. And then, you know, you could certainly lean, we could certainly lean into the, the concept that we are, and it gets a little bit fuzzy, right? But if you don't have to remodel your data center, you don't have to bring in more power, and you're not throwing away all the old stuff that, that's a lot less waste that's going into the landfills or wherever it happens to go.

Um, some cases it can be recycled. So all of that stuff is not happening because we're extending the life and the usefulness of the data center. Thank you.

What's your research so far on products that you've made? Back to Karen's point about sustainability? Are you, have you, is sustainability an afterthought or is it in the forefront for your company, would you Say?

It's, it's something that we care a lot about. And in, in maybe about six, seven months, there's another tier of product that we're gonna start offering, which I can't talk about today. But one of the, the main value propositions for offering it is that we're focusing, like traditionally, there's always been this focus on TLC, NAND and QLC NAND kind of thing, but a lot of use, like if you put 60 drives in a chassis, for example, you're gonna saturate that network link with like four drives.

So why are all of these drives Ferraris? What if they were maybe not Ferraris? Yeah.

And then once you start realizing that in large deployment configurations, drives can have a, a, a slower, uh, can, can just run slower, then you're asking yourself, do I need this class of nand or could I consider this other class of nand? So that ties into the whole idea of recycling the NAND and finding other uses for it. Okay.

Where it makes sense. Okay. Thanks.

Okay. And, uh, I'll go for a, a third factor in this where, uh, you know, product differentiator and, uh, you know, product sustainability. Um, tell us some of the, the fun nerdy stuff about your product that that's, uh, entertaining.

I mean, we're here for tech field day, we wanna hear the, we wanna hear how the technology works. Right? And, and I mean, you've certainly given you've, you've done a very good job of describing, you know, what the product does.

But tell us how it works. You mean like what can you do with it once you have it? No, no.

The internals. The internals. How it works.

The guts. The guts. Sure.

I mean, it's, it's slightly disappointing because data moves from one area to the other. That's true. And then internally, um, you've got these callbacks, right?

But what are they, it's like pyr saying out of memory error, you know, and normally you get a minus one, a negative one saying error done, and it give, you know, it stops running. What do we do? We, we take a bunch of code, we look at the GPU pool, we figure out what's finished.

We offload something. Again, just a basic memory move. Um, uh, I, I wish I could tell you that we're like doing these fancy back flips and stuff, but Okay.

Primarily an execution pipeline that's, It's So, so right now you are, you are, you're very early in the product family life lifeline. Yeah. Or lifecycle.

You really, you're saying that now that you have this technology, you can now build stuff into the SSDs that your swap library can take advantage of. Yes. So we're adding compute capabilities to the ssd, but we're also at the software layer adding mi, we call 'em microservices.

Mm-hmm. But you can think of 'em as AI agents and, and just, and they're different agent flows. This is an agent flow that's designed to work with this, um, this, this solution which trades off cost and, and performance and time.

Right. Cool. Okay.

Sorry, did I answer your question? Uh, yes, yes. No, I think, I think, um, I think the challenge here is that your technology at the moment mm-hmm.

Solves a very, is is a simple, yet very elegant solution to a critical problem. Right? Right.

That is, uh, um, maybe targeted at the smaller, lower end. It is side of the AI workloads. Yeah.

We have spent the previous few days talking to people about a high end, The Tip of the pyramid, and now we're down here. And the benefits are, the value prop is much more balancing price performance, not balancing prices. Unlimited performance is, is the, that the only goal we care about?

Right. And so that's a, a little bit different way for us to think about it. Mm-hmm.

Right. And so, like I said, you have a a, a, it's a cash swap, right? Yep.

And it, it's a very elegant solution to something that wasn't, nobody thought that they would need this, now we need this. Right. And you've got it, which is really cool.

What's interesting though is that this is starting to work its way up as well. Mm-hmm. Right?

So we have conversations with the top tier people that have unlimited budgets too. They have limits. Like everyone has limits.

It's just their limits are much, much higher. And one of the problems that they're running into, and, and this was a question that was asked earlier, is why don't you just cram more ram on the cart? It's full.

You can, there's no more space. So the next question is, why don't you make the cart bigger? Well, now you're just gonna make another motherboard, but you're still gonna hit the same problem.

Right? Even if you make the mother the, the card twice as big. Mm-hmm.

And, and you, you design it properly so there's no power and you have water block cooling and all that stuff. That's great. Now you've got something half the size of the motherboard sticking out at 90 degrees from the motherboard.

Um, and maybe you have eight of those, or, or you go to a card system, like, like in those telco switches where you just have these giant cards of, of GPUs, you can, you still eventually run out of space to physically put more memory checks. Well, And that's, I mean, having been here at AI infrastructure field day this time and the previous time, an awful lot of the, the problems that people are attempting to address is how do you deliver memory to A GPU in one way or another. Mm-hmm.

Yep. And, and you're offering a, a very simple way to, to do going to an SSD. Uh, i, I mean, we hear from major networking companies, they're talking about 800 gigabit links to be able to feed memory to GPUs.

And if you're using something that would be like a, an educational desktop machine, you're not gonna have 800 g gigabit No. Uh, networking to it. Yeah.

Well, they're even going further and they're saying, Hey, what's the bandwidth on the GPU? 4 terabytes. And then they're creating a, a a a a server that has 800 gigabyte, or eight eight gigabit links.

So 800 gigabyte per second or more per server. Um, and so if you cram enough SSDs into a box and you give it a fast enough bandwidth to the GPU, you do get a little bit more latency, but it turns out it doesn't really matter. 'cause there is still a caching flow in the pipeline.

Um, so you can actually get to the point where you're having really big pools of SSDs connected to GPU servers with really re like a lot of network links, right. The cost is astronomical. Each one of those 800 gigabit things is $20,000.

Okay. But the, the point I was trying to make, okay, is that it's, uh, you know, everybody is trying to address the idea that you can't actually hold the entire working set right in the GPU memory. And people are coming up with different ways of doing it.

Yep. Yes. And sorry, the thing that I was about to trying to get to was, um, I got kind of sidetracked myself.

Um, one of the things that I was trying to get to was this notion that, um, SSDs will participate in the high end, um, and what they're looking for from those SSDs. I mean, you can use con conventional SSDs that work on 4K IO, uh, internally, but one of the things that people are starting to talk about is what if that IO is smaller and closer to the size of the IO that the GPU uses internally? And so there, there's a whole other branch of really high-end SSDs based on SLC with maximizing the capacity that you can put on the board of the SSD and then making sure that it's number of IOPS that it can support is insane.

Um, you know, just 10 times higher than what the, the best SSD can do today kind of thing. So adding Nan to the whole memory ecosystem is happening both at the high, well, definitely at the high end. Um, it just hasn't been productized yet, but different companies are working on it.

And, and it's also happening at, at the low end through solutions like ours. Mm-hmm. Um, let's see.

And then the other, so we talked a lot about training. We actually also help with inference. So one of the problems that you have with inference is that, um, fundamentally the thing that makes the inference like tolerable and that you're not waiting for so long for each response to come in is something called the KV cash, which actually does not mean key value in the sense of a traditional key value database.

The key is a tuple with five entries. Um, but nonetheless, it, it is a KV cash. And so the way that it works today is your KV cash is a certain size.

Um, and once it's full, there's an eviction policy. Typically the oldest entry gets punted. But there, there you can have any eviction policy that you want, but in many cases it doesn't matter which eviction policy you use.

So they just go with a simple one. Um, there's no real difference if you try to intelligently pick the, the entry that you're not likely to use. Um, I guess it depends if you're, if you're GPU is serving one user versus a pool of eight users, you might start by evicting the one that has had the least interaction, for example.

But, and you see this is, you see now you, you're now you start, start to get into the technology. Actually the eviction policy does better put a bit LRU versus Huge difference. And yes, and I, uh, It can, but I'll explain why I'm saying that.

Okay. So If, and I'll, I'll tell you that I spend an awful lot of time, uh, optimizing cashs. Sure.

There's a scenario where it kind of doesn't matter. So let, let me explain though, 'cause I I do understand your point. Okay.

Um, there's a reason there's a lot of money and energy that's put into cash management or cash management policies. Yes. But, but in this particular scenario where you essentially ask one question, then a follow on question and a follow on question and a follow on question, right?

Like you're having an interactive session with an LLM, um, you're essentially, you, you will get to a point where your session gets bigger than your KV cash. And by the time you, you, you finish processing, you know, the second question, all of the tokens related to the first question are gone. So when you get to the third question, which is related, you know, in this one session, you have to essentially re prefilled them, and then all of the stuff that you had from the tail end of your previous session is gone.

So when you're in a situation where you're essentially have a hundred percent or more eviction rate doesn't matter. This is what I meant, why it doesn't matter which policy you use. By the time you finish processing your question, you go to the next question, which needs all that original data at the beginning, it's all gone.

Does that make sense? Yes. And I'll just leave it there.

Sure. I understand. I'm not trying to debate you and I apologize for that.

Do we need a whiteboard for the two of you and a quiet, maybe a padded room? No, I judge you wrong. No, but if, if you'll allow me in the situation where you end up pushing everything that was in there previously, or no, actually, we'll leave that aside.

Right? And, and all that I meant to really focus on here was to say, when the new entry comes in and the cash is full, typically it's just evicted and it's, it's gone, it's overwritten. Um, and the way that inference works is that you essentially feed all of the, all of the, your queries, I'll call them words, technically they're tokens and it gets all weird.

But if you just think of them as words, um, you, you feed in the word of your sentence one word, two word, three word, four word, five word, and then it generates the first word of the answer. And then to, to get the next word of the answer, you actually have to feed in words 1, 2, 3, 4, 5 of your original query, plus the first word of the answer to get the second word of the answer. Then you feed in 1, 2, 3, 4, 5 plus, you know, ab the first two words to get word C.

And then you do it again and again and again. And so as you're feeding in more or as it generates new words, those new words get added to the KV cash. And once you run out the eviction policy typ, well they, they just disappear.

They get, they get evicted. Um, and so what we did is we took the same model in both cases, but in one version we enabled adaptive and the other one we used, it's in the standard configuration without adaptive. And so you ask the first question at times zero to both of them, and they generate the same answer in the same time.

And so at the ten second timestamp, they, they're both finished and they start working on the second question. Now, the problem with this particular configuration is there's not enough KV cash. So all that time that you spent doing the prefill, you have to do it again.

So it has to do the prefill, and then it has to work on the tail end of the answer. Um, and so it takes quite a bit of time, about 26 seconds to complete the, the second question and start the third question. Whereas on the blue side, it finishes in 20 seconds and then starts the fourth question, or sorry, starts the third question.

So as you can see the side on the right, even though you, because you would think, Hey, where's it getting that stuff from? It's getting it from the SSD where the results are being cached. You would think, well, okay, the SSD is seven gigabytes per second.

That's a lot slower than a hundred gigabytes per second, uh, which is the bandwidth of the ram on this particular cart. But it turns out the compute time for, for doing this, it's surprisingly long. And so even though you're going over a relatively slower link, you end up with a faster real world result.

And as you keep asking questions, the SSDs got tons of space. So it just keeps accumulating more and more of the KV cash, and so we can get through all five of our questions, and it's done in about 58 seconds. The original one that doesn't have this KV cash offload capability, essentially at 51 seconds, it starts question four, and then it crashes about, uh, is that about nine, about 13 seconds later, it, it crashes.

Um, just because there's, it, it just couldn't handle it. So the neat advantage is that when you're adding, uh, you can essentially support either more users or you can support substantially longer queries that take like large documents that are like maybe 10, 20, 30 kilobytes of words, just straight text. You can include that in your query.

Um, and, and it generally, uh, provides a better overall user experience. Uh, that is normally only available if you have enormous servers with lots and lots of memory. Um, and in fact, what we see is that we can, um, and, and it, this is a particular example using the NVIDIA jets.

And so for an industrial application kind of thing, um, and in this scenario, we can essentially, uh, increase the context length that it would normally be able to handle by a factor of eight x with, with the SSD that we paired it with. If we were to pair it with a much, much larger SST than the context lengths would be longer. And also essentially the time to first token recall.

In other words, the time that it takes to recalculate all the missing tokens, uh, that were previously evicted increases by, or is reduced by a factor of 14. So have you, um, looked at getting fancy and doing anything with predictive fill of the ca the the, uh, the SSD cash in advance? Yes.

Well, that, that's a tricky thing, right? 'cause prediction is is like the, the, the $10 million question. It's like, how do you know what, like it comes down to will two users phrase the que same question the same way?

No, Much more on, yeah. Uh, even if the cash isn't being used, right? So you have the, you have the start of a conversation, right?

So you can say, um, assume that the, the user's going to have a sequence of conversations. Yeah. So even though I haven't needed to use my SSD cache, let me drop tokens from main memory into the cache, pre-filled the cache, knowing that they'll eventually get addicted out, evicted out of memory into the cash and therefore it's already there for when you next need it.

You can certainly hold onto past conversations. Yeah. Uh, and you can actually create a whole indexing system where you could say, if you were talking to it mm-hmm.

Hey, remember that time we were talking about cars on Tuesday? And they would just refill everything that I had from the past conversation. You could do that.

But the reason why what you're proposing is difficult is how you phrase the question in the order of words that you put in changes, the order of words that are put out and the order, like the, all of the, the sequence specifically of all the words in your questions, your answers, your follow on questions are 100% dependent. And if you change some of them, you will get a different response. Yeah, okay, Thanks.

But there is a lot of value in, uh, and today we just don't do it. You know, just start again. But imagine this, um, you know, 6, 12, 18 months later.

Um, and then this whole idea of conversation recall comes in, um, that could lead again to a more natural experience with your LLM instead of starting from scratch and or having like a a some, what some people do is they have like a text pad on the side with the query that they always ask, and then they put it in and then they continue from there. You could just say, reference my, you know, my Thursday query or my standard query three, give it a name, like a file and then go on. Yep.

Um, so I'm almost done. I think this is, uh, I think I might have one more slide after this. Um, but here, the, the thing that we wanna sort of, kind of sum up or recap the whole conversation is models are getting larger and there are plenty of use cases where small models are fine, but when you start asking for more nuanced, more complex questions and you're trying to get it to do more advanced correlations, generally more parameters is produces better results.

And we're already seeing cases where you, you have, uh, 605 billion parameters, 671 and 2 trillion is on the horizon. And, and you know, we are seeing trends already where even at the high end SSDs are becoming part of this flow. And so what we're doing with our solution is, uh, making sure that we can continue to provide support for these larger models.

So we're actually making higher and higher density SSDs. Um, this one right here, uh, from a a placement perspective has room for about eight and n packages, which in current densities about eight terabytes TLC or about two terabytes SLC mode. But this one can go up to 128 terabytes as uh, TLC and uh, soon 2 56.

So when you take one of these and just to go from TLC to SLC, rough numbers just divided by three, right? So if it was 300 terabytes, we could put a 100 terabyte S-L-C-S-S-D and there might be 24 of these in the server. Um, and, and so we can support both high end and low end configurations.

And then you would essentially, with these AI one hundreds in the OR with this SSD and the A one AI 100 configuration, possibly with additional accelerators within it, you then plug these in based on the size of the model that you wanna work with. So most models would be fine with one or two. Um, if you wanted the really, really big models, you might be looking at eight.

Um, but the idea is that you can now tune the hardware both from the SSD side, from the GPU side, from the CPU and the dram. You can tune it to match your budget and the speed that you need. If you don't need responses in 40 minutes and overnight is fine, bring that cost way down.

And so our solution, it's affordable, it's private because it stays on premise. Your data never leaves your control and it ends up, uh, producing an AI agent that is more useful to your company. So that's our solution.

Uh, thank you very much.