Outpacing Hardware: GPU Memory Bottlenecks, Model Optimization, and the Reality of Enterprise AI | Utilizing AI Episode 44

The hardware needed for artificial intelligence applications today may not be a match for tomorrow's needs. On this episode of "Utilizing AI," Frederik Van Haren of Hyfence and Brad Shimmin of Futurum discuss the reality of AI hardware as models grow and needs evolve. Welcome to "Utilizing AI," the podcast focused on practical applications of artificial intelligence and part of the Futurum Podcast Network.

Every Wednesday, we explore news and use cases of the ways in which AI is transforming enterprise IT and the industries it serves. I'm your host, Stephen Foskett, president of the TechFieldDay business unit here at the Futurum Group. Before we dive into this discussion, let's meet who's on the panel today.

Frederik? Yeah. Well, thanks for having me.

So I'm Frederik Van Haren. I am the founder and CTO of Hyfence, and we provide HPC and AI consulting and services. And hi, everyone out there.

Brad Shimmin. I work with Stephen at the Futurum Group. I am an analyst looking at data intelligence, analytics, and infrastructure.

It's a pleasure to see you guys today. Honestly, I was so excited when I saw who Corey had booked for us today because Brad, obviously, it's always a hoot getting a chance to talk to you, and Frederik is-- I don't want to give you a big head, but Frederik is one of those people who's out there actually doing the thesis of this podcast every day with real customers. And so to have you share what you're seeing and what you're learning with us is just so incredibly valuable.

Thank you so much for being part of this, Frederik. Yeah. Thanks.

So we were just talking a little bit before we pressed record. One of the biggest questions I think that people have for me and for Brad, and probably for Frederik too, is, okay, we're doing this. We are going to have enterprise applications in production.

We're going to use LLMs for various tasks. We've bought some hardware. We are going to implement this.

How do I make my dreams real? How do I make the AI application of my dream fit into the hardware that I have, the hardware that I can afford, the hardware that I can deploy, and how do I make that work in the future as these models just keep getting bigger? Frederik, if I was a customer and I came to you with a use case that obviously, big dreams, smaller hardware, how would you address that?

Well, let's first start with the fact that AI is very dynamic, so whatever you need today might be slightly different or majorly different tomorrow. And so typically customers, maybe a year or two ago, that question would be about training. How do I train the models?

So they would require hardware in order to do training. Nowadays, with the availability of Anthropic and OpenAI providing those large language models, there's a shift in the sense that there is no value, really, in building your own models from scratch. And so now you're using those models provided by Anthropic and OpenAI and Meta.

The second problem is that those models are very capable. So for example, if you look at Kimi K3, it's 3 trillion parameters. Now, if you want to use that model and you look at the hardware from Nvidia, for example, it doesn't fit.

And so now, you ask yourself the question, what do I do? And there are little tricks you can do to squeeze those models on those platforms. But the reality is that the hardware is actually lagging behind the large language models.

And so from a customer perspective, it's very difficult to say, well, this is 3 trillion, but wait six months and it will be six trillion. How do you approach this? And so you have to work with these customers and design some kind of an architecture that scales.

So again, in summary, it's very dynamic, and as far as I'm concerned, the hardware is really lagging behind the large language models that are available to us. I agree very much, Frederik, and I think it's worth pointing out that it started in this whole revolution of agentic and generative before that. It started out behind because GPUs, as we know, were not built for AI.

They were built for high parallelization, and when you're talking about the math that models do, like transformers, it's not like a perfect match. It's not like a chocolate and peanut butter sort of affair, and that's why there is a lot of room for optimization that's being explored right now. And I'm kind of shocked to hear companies talking about-- These are enterprises, adopters, talking about, "Well, gosh, if I have 10% idle time on that GPU, I'm wasting that money.

" Right. It's clear that the GPU was an accidental piece of hardware. It was supposed to do one thing really well, which is put pixels on a screen.

And if you look at the resolution of a screen nowadays, it's millions of pixels. So it's kind of interesting to see how this goes. But I think the main problem today is really onboard memory.

So you have really fast onboard memory. We're trying to optimize applications by using what we call KV cache, which is basically caching. But where do you store the cache?

So the cache also gets stored on the GPU, and so now you end up in a situation where you're trying to offload some of the caching to somewhere else. And so that becomes mostly a non-GPU conversation because now you have to get the data from the server. And then that's another problem.

Servers today are still based on the PCIe bus, which was great when it was created. Nowadays, the PCIe bus is relatively slow compared to anything else. And so where do we go from here?

I think anytime we deal with the hardware we have, it's finding the right solutions and optimize. But it's important to understand that there is a shift more and more to inference, and I would even dare to say applications where people just load large language models on GPUs and then consume them. It doesn't have to be an AI factory.

You can run inference on your laptop. I mean, not the greatest, not the fastest, but you can. Absolutely.

And honestly, maybe not the greatest, not the fastest, but I've been incredibly amazed at the level of AI applications that can run on modern laptop/desktop hardware. cpp, I can run not huge models, but decent models locally on that machine. And the same, everybody's barking about the new Mac Studio with a half terabyte of RAM and the new M5 Ultra and all that.

Those can be clustered. I've got a lot of friends with DGX Spark. You can do some really cool stuff.

But one of the things that I love about what you just said, Frederik, and I think that the three of us, obviously we've been in this industry for a long time. Everything that's old is new again. You talked about KV cache.

KV cache, we've been talking about caching since the beginning of computing. We've got levels of hierarchies of memory and storage. Again, that's a topic that would've been familiar to an IBM researcher in the 1980s.

And the other thing I'm hearing a lot of people talk about is distributed inferencing stacks, like LLMD. That resembles Kubernetes. It resembles a web scale infrastructure to me in a way that I actually find comforting.

I feel like people are basically bringing back the world that I understand by saying we've got to optimize the stack, and we've got to think about caching, we've got to think about high-performance storage, and we've got to think about distributed computing. These are all familiar topics. Brad, I don't want to call out any graybeards on this podcast, but you do have one.

Does this sound familiar to you? Oh, heavens, yes. Yeah, it's hilarious, isn't it, that we rediscover technologies and ideas that were forgotten about quite often in the past, or at least they were living very vivacious lives within a very small group of individuals building things that required things like KV cache.

But now we're seeing the applicability of it, and it's truly, I think, a very wonderful time for computing because of that. Because as we've chatted about a little bit so far, and there is so much room to optimize and so much to learn about inferencing and how we are running these models. Two examples.

One is NVIDIA with their Nemotron Lightning model they came out with maybe a month ago. They introduced a new technique wherein instead of distilling a model down to a smaller model, learning from big, transferring to little, and then quantizing it using techniques like LoRA to make it smaller, fewer floating point numbers. They actually have a technique that sort of recognizes and honors that end state of what it's going to be, four bits, eight bits, whatever.

And so the distillation of the model itself, actually, you don't lose hardly anything in getting it down to that size. And therefore, you can actually take those models that you've been running, Stephen, that would've been off-limits maybe six months ago to your hardware. Now, you're running a 29 billion parameter model at 300 tokens a second.

Well, maybe not that fast, but pretty quick. And the second thing I want to point out is that this learning, this constant learning isn't just in how we build the models, but how we implement them. And something I've noticed in the offline community, those who are like you, Stephen, just churning on their...

They've got their Cloud Bots running 24/7 on their NVMe drives, doing write and reads, that they're like, "Holy crap, I'm killing my NVMe drives. " So it's really fascinating that as an industry, we sort of have this forgetfulness that we have to wake up from every once in a while and realize that, oh, we've already solved that. Let's bring it in.

Yeah. Especially, Brad, that hurts. You're hurting me here.

Sorry. " I'm crying in my beer. Never mind.

Never mind. Never mind. And I'm only a little guy.

Frederic, you're dealing with customers who have, well, real infrastructure and real budgets, right? Yeah. Right.

I would even refer to the supply chain problem or the pricing as a domino effect. And what I mean by that is originally, maybe two years ago, we had some issues with GPUs, and then that kind of got solved somehow, and then it went to NVMe drives and then DIMMs, and even today, hard drives are an issue. People look for alternatives, find alternatives, and even the alternatives become then very expensive.

So it's not really a battle you can win. But I think a lot of customers understand that and then choose the optimization way in order to increase the ROI. You have to realize that NVIDIA comes out with a new card every two years.

So some people had a strategy of, well, if the card comes out every two years and it's twice as powerful, where is the optimization requirement, right? I can just replace the card and be done with it. The reality is that even the individual card might perform twice as better, but the code you wrote might not scale twice as much.

So in some cases, you could get into situations where it's actually the reverse. But yeah, prices are higher. Will they ever come down?

I don't know. It's a fact of life, I guess, we have to deal with. " Well, let me ask you, Frederic, are people doing that?

Because I would think they would be, right? They spent a lot of money. They maybe didn't get much ROI on their first generation AI cluster.

" Has that happened? Yeah, it happens. It happens more and more.

But there's always a question, improve my ROI without telling me really what to do. That's typically the way it works, and the way we approach it is we do some basic benchmarking and then give them numbers they don't like because they're really awful. But typically, it's show and tell, right?

We show them a little bit how we can improve. " While in reality, it's the opposite, right? Maybe 20, 30%.

I was the guy whose job was to go in and audit the storage and tell them that it was actually 17%. Congratulations. Yeah.

And I think we're back to that, right? I think, trying to remember, it was one large enterprise who had a utilization of their GPUs of 17% and was considered kind of normal. But there's more focus on optimizing, and it's cyclic, right?

We talk about KV cache and liquid cooling. And even if you look at LMD, it's more paging, so you're kind of using a paging mechanism to swap bits and pieces of RAM in and out of GPU. But one thing we learned from cyclic items is that eventually, there will be a piece of technology that will come out that kind of solves it for a little bit, and then we go back into the caching and back into our good old trying to fix things.

So I think we're just in a cycle. We just have to accept the cycle, and as a customer, you just have to figure out if you want to ignore or accept the cycle. And the moment you do that, life will get a lot better and easier to explain than to fight the prices of the supply chain.

Frederic, can I ask, in your travels, when you're talking to companies that are trying to optimize their spend or even just understand their spend, I presume that there is a good chunk of them now that are building out solutions for inference that are themselves built with the same said technology, agentic development. So I'm curious, as we're talking about optimization, are you having to go back to them and say, "Well, vibe coding in TypeScript maybe wasn't the best play here, guys. " Is that happening for you?

Yeah. You bring up two things, right? So there's the infrastructure, which has nothing to do with vibe coding, and then you have the other side is how you use your items.

It depends. So the challenge the brownfield organizations have when they develop AI models or inference cycles is that they write the code in one point, and changing that code is almost like a no-no for most organizations. So trying to convince them that to change the code is problematic.

So I used an example earlier where if your CUDA code was written at the time of the K80s, which is many, many, many cards back, that CUDA code was not really scalable. But if you're not willing to change that code, then buying the latest and greatest GPUs is not for you. Our recommendation then is buy a low-end inference card, like a T4, an L40, and surprisingly, you will get a better throughput from that card than from you running on an H200, where the H200 will be spinning and waiting on itself because the code doesn't want to move forward.

Yeah I think vibe coding, so on the application side, vibe coding and other languages, I think it's relatively new. I think a lot of people deploy applications, and they haven't seen the side effects of it. My biggest concern, what I see in organizations is when somebody comes up with a prototype and it looks good, there's a tendency to fast-forward this to production.

And so when you look at the statistics where people say, "We went from training or from inference to production," and say it's failed, in most cases, it's because they were tripping over themselves. I think in the past, we talked about security. I mean, security is probably the first thing they overlook.

It's almost like an addendum at the bottom of the menu. Check the box here. If you're dealing with data that has to be HIPAA compliant or PCI, it's very challenging to undo it later on.

We see some organizations trying to battle this by having a closed loop in their circle, where the closed loop is kind of their security slash validation, where they use agentic frameworks to do the security as close as they can in their applications. But I think from a customer perspective, certainly if they have customers and certainly if they have peer pressure from other competitors, it's the Wild West. I use that a lot, the Wild West, because I have no better way to explain it than like this.

Well, it really feels like that, right, Brad? Right. It does.

And I was just thinking as you were talking about that, Frederic, that we've seen companies break the rules Wild West style to great success. And we've seen it with many Chinese frontier model makers over the last three years, going dating back to the first DeepSeek, which they just said, "You know what? Cuda, you're not doing it right.

" And we see it more recently with Alibaba Qwen models in terms of how they're inferencing and how they're caching and what that means for how much it costs for these kinds of models, whether they're hosted on something like OpenRouter or hosted by the vendor itself. And so it's hard, I think, for enterprises to really understand exactly what kind of value they're getting out of their token spend. Not just because of how many tokens, but in terms of the efficiencies of the stack that those tokens are inferencing on.

Right. There's also a time-to-market aspect to it, meaning that before you start talking about ROIs, it's like, how fast can I turn this over, right? And I speak from my own experience when we started in 2006 with large language-- Well, what we called large language models in those days was nothing compared to what you have today, but it took us about- Just different definitions of large.

That's right. That's right. It's all relative.

But it took us six months, I mean, for about 42 languages, but we did it all in parallel. It took us about six months to turn around a new language model. Nowadays, the same requirement, if it doesn't happen within 48 hours after receiving new data, then you're in trouble.

And so in some cases, you don't have a choice than to go for the big guys. Big guys- Yeah ... meaning where you can find it.

I mean, look at SpaceX or xAI, now SpaceX. They rent out most of their Colossus capacity. I think one of the data points was they were renting it out per month.

2 billion a month to Anthropic, actually. Yeah. So there must be- Well, my understanding is they're renting out to basically everybody because they've got Colossus- Yes ...

Two and Mini Colossus or whatever the other one was, and I think that they're all being rented out. Sounds like AWS. Right.

Well, it's slightly different than AWS in the sense that AWS builds for capacity, while that Colossus data center was just built apparently for renting, right? It was like a real estate, well, data center- Yeah, right. It came to them later ...

as a service. Yeah. It's funny.

You could criticize that, but honestly, they probably bought at the right time, built it quick enough, and they're going to be... It looks like a pretty good business to me. Right.

Yeah, definitely. It's a time to market. It's not cheap.

I'm pretty sure that if Anthropic had more time, they would build their own, but they don't have the time. And they most likely also don't have the money. So- Mm-hmm ...

but here we go. I think it's just a sign of the times, where the amount of expansion and the amount of hardware thrown at it is significant. And there is demand, right?

I mean, everybody's now using it, for good reasons or not. Everybody's using it. As a matter of fact, I feel like people are replacing Google search with an Anthropic or ChatGPT, right?

So basic, what's the weather outside kind of a question. Well, it's funny that- Let's start with perplexity, and it's become the norm now. I think that people trust what the model is telling them more than with their own eyes looking at a website increasingly.

Right. Yeah, I don't want to get too far off topic on that, but absolutely. The people that I talk to, I mean, Google, people like to criticize Google and Gemini, but the fact is, Google's AI summaries and the Google AI mode are actually growing rapidly, and people are really leaning into that.

Yep. So, Frederic, kind of riffing off of what we just talked about, one of the questions that I've wanted to put to you for a long time is this sort of build versus buy question for enterprises, because you've got companies out there like the, you've got the NeoClouds, you've got AWS and GCP and others that are offering a lot of capability. Are companies building still, or are they leveraging those kind of services?

Yeah. So I guess I have to start with hardware nowadays is very expensive. So I think, yes, so if you had to ask me the question like last year, I would have given the different answer, but the cost of hardware is very expensive.

Secondly, the knowledge to build these things, the knowledge that enterprises have to do this well, is not great. So I think a lot of people go to NeoClouds. But the real question when people decide if they build it themselves or go outside is really about IP.

So if it's their own data, and that data is worth a lot to them, unique in the sense that they don't want to share it, using a NeoCloud and anything else is risky because you don't know if that data would be analyzed or not. So in those cases, those companies that decide that their IP is more important than anything else will do it on-prem. It also happens in stages.

So POCs, public cloud most of the time. Training, if they still do training, it's typically baseline on-prem, with a peak capacity. And peak means if they have a deadline for research and they can't meet it, they have a way to get some extra capacity in the public cloud.

On the inference side, for applications, we see a lot of need for elasticity. And so elasticity is not something you can do on-prem. Elasticity is something you have to rely on, like a public cloud or NeoCloud in order to get that.

Yeah But yes, it's not an easy question. Some customers have a lot of difficulties. Let's not forget the data center problem.

Many data centers out there are not liquid-cooled or don't have those capabilities. Yeah. So you would be limited in what you can do.

But- Cooling and power, yeah. Cooling and power, yeah. Cooling more than anything else.

If you have a data center where there is no liquid cooling, bringing liquids into the room, and dealing with that, it's a big issue. We had one customer who was presented with a bill between $4 and $5 million just to get access to liquid cooling. We're not even talking- Wow.

Yeah, and some data center build-outs that we're seeing, for example, in Utah, are going to take twice as much water as Utah currently consumes to do their liquid cooling. I'm not sure how that math is going to work out, but it is something that companies have to deal with, like you said, Frederic. And what really strikes me about what you're just talking about is that it's not an all-or-nothing game, and it never will be.

" It's always multi-hybrid cloud, all day long, and for good reason. And I think that the same already holds true for model usage. And you have, as you suggested, different points for why I'm going to go to the cloud for this, whether it's trying to meet peak usage or whether it's trying to solve a particular problem that only the most current Soul 56 can solve for you.

Great. You're going to do it. And if it's within the boundaries of your risk profile for IP, yeah, why not?

Why wouldn't you do that? And that's why we're seeing a lot of investment, I think, right now in these sort of model routing, and adaptability mechanisms. I myself, as a hapless practitioner, have found myself having to write code that actually pings a given model reference that I want to see if the queue is full.

And if the queue is full, then I fall back to a model that isn't using a queue. I'm talking about flex paying, which is cheaper. So if I can do it cheap, I'll do it.

If I can't, I'll pay more money because you need to get the job done. So it's all going to come down to how companies, how willing or able they are to actually fine-tune their usage of all of these great tools more than it is the capability of the tools themselves. Right.

Yeah, I would say in summary that AI is very dynamic. Nothing will be the same. It will always be slightly better, faster, lower latency.

I think the challenges today are kind of an outside factor, which is the supply chain, which makes people make decisions that they would otherwise not have made. A heavy focus on GPU efficiency and utilization. Let's think ROI.

I think that's a big item. And I also think that it's important to note that the models are much bigger than the hardware can support today. I think maybe as a side note, I think Cerebras is one of those organizations who saw this early and is trying to accommodate that with a larger amount of CPUs or GPUs, no matter what you want to call it.

But I think I like it. It's a great market, fast-moving, very challenging for customers. Let's see how it ends.

Yep. Yeah, and we shall certainly see. I'm looking forward to continuing this kind of conversation again.

Frederic, the sort of hands-on experience you have is just remarkable. I always learn something, so thank you so much for joining us to share some of that with the audience as well. So it's a busy time.

There's a lot coming up. I'm personally looking forward to hosting AI Field Day. I'll be attending Supercomputing.

I'll be attending some other shows. Before we go, Frederic, where can people find you? Where can they continue this conversation?

So I will also be at Supercomputing. I think there's also an AI infrastructure in October that I will be attending, and if you're organizing an AI one, I likely would like to join that as well. And then you can find me on LinkedIn as Frederic V.

com. Yeah, the travel season's kicking off for me, so I'll be a number of trips down to New York City to sit with Neo4j, IBM, and others, and then I'll be heading overseas to spend some time with Microsoft all about Fabric. And I'm always, but not every moment of the day because, come on, on LinkedIn, under Brad Shimmin, all one word.

And if you don't mind, Stephen, really quick plug. I just published our data intelligence platform, Signal report, which is an agentic-led comparative report of the data intelligence platform market, and I'm looking at 15 vendors scoring across those. And spoiler alerts, Databricks did pretty well.

Oh, interesting. Yeah, where can people find that? com.

It's actually free. You don't have to pay. We're not firewalling everything.

So it's very much open. I would love for everyone to take a look and let me know what they think. Excellent.

Yeah, that's one of the coolest things I think that we're trying to do here, is share more of this information free, and to kind of break down that paywall. And I should also mention that Brad has been really instrumental in developing that whole Signal methodology, and I think that it's one of those kind of get-your-hands-on-it things, Brad, that I respect because it's the same-- That's how I taught myself a lot of this stuff is, I built myself a Kubernetes cluster in the coffee table at the office, I might add- ... to learn Kubernetes.

That's how you learn this stuff. And by building AI applications ourselves is how we're learning what customers are doing and how this stuff really works. So awesome stuff.

Thank you. Check out that Signal report. Check out all the Signal reports as well.

Thank you both for joining us here, and thank you listeners for listening to the "Utilizing AI Podcast" every week. If you enjoyed the discussion, please do subscribe. You'll find us on YouTube as well as your favorite podcast applications, and consider giving us a rating and a review.

This podcast is brought to you by the analysts and experts from the Futurum Group, where insights meet AI. ai, the "Utilizing AI" YouTube channel, or the Techstrong TV app. Thanks for listening, and we'll see you next week.