Crusoe Optimizes AI Inference Beyond Hyperscalers

AI Leadership Insight Series. I'm your host, Mike Bizura, and today we're with Kyle Sosnowski, who's the Vice President of Engineering for Cloud at Crusoe, and we're having a little chat about, well, where should all these inference engines be deployed and optimized? Because, well, it may not be as obvious or clear a decision as some people might be inclined to think.

Kyle, welcome to the show. How's it going? Good.

So this has been going on for years, and it doesn't require AI per se, but a lot of people just default to whatever their public cloud service provider is and the one that they have a contract with, and that is maybe in their minds. There are issues to consider in terms of where you deploy your inference. So walk us through what's going on here and what people should be thinking about.

Yeah, I think for starters, the crunch that most people are facing is just general capacity. Can they get the GPUs they need to be able to service their workloads? It's very different from general purpose infrastructure.

So the historically using AWS, GCP, or Azure for all of your infra may mean that you have your application infrastructure handled, but you can't necessarily get GPU capacity that you need to serve AI workloads. On top of that, the workloads are more latency sensitive, they're bottlenecked on memory and bandwidth, and you want to make sure that you have bursty allocation depending on the type of workloads that you have. So for all of that considered, it doesn't necessarily mean that people are going to be able to get everything they need from the hyperscaler that they have.

So most companies that are AI natives or DNBs or anyone that's deploying HPC infrastructure is multi-cloud, which means that they're looking at us, they're looking at our competitors, as well as rooting their application infrastructure in a hyperscaler to get that wider breadth of services that these neo clouds don't offer. So how does that work exactly? Is the GPU just more accessible because it's a shared public cloud infrastructure, or is it, or maybe both, that my utilization rates are simply better because I got a platform that's optimized specifically for this workload?

What's going on here? I think for customers in general, it's the hyperscalers favor a certain size of customer to be able to get them the latest SKUs. So if you want the latest from Nvidia, you want Vera Ribbon, you want V300s, and ensure that you have the most modern GPUs.

Unless you're a customer that's spending a certain amount at a hyperscaler, you're simply not going to get allocation for it. On top of that, neo clouds are putting a lot more investment into some of these underlying services that make the cloud services and the infrastructure targeted specifically for AI. So AWS and GCP, like I mentioned, are general purpose infrastructure clouds that are building AI services and AI optimized infra.

Neo clouds like ourselves and CoreWeave and Lambda and other competitors in this space are taking it from the AI infrastructure first approach and building optimized stacks around that. So what do you see folks doing today that just kind of makes you shake your head a little bit and go, "Folks, we need to be a little bit smarter than that"? You mean across general infrastructure deployments?

Mm-hmm. I think that relying on the primitives that got us to the first iteration of the internet and infrastructure and cloud deployments as it is, is no longer relevant. Trying to reuse Kubernetes in the same way that you were for your application infrastructure is not going to be effective.

The workloads are a very different style, right? Think of surge in traffic against your website to service some marketing event or something like that versus very high throughput, very latency sensitive inference workloads that are necessarily steady, and not bursty in different periods. Memory management, as this agentic boom is happening where you want to share context across different agents that are running in parallel, it's a very unique type of workload and not something that was historically seen by clouds prior to this.

So I think that all of that is, to answer your question, is folks trying to reuse foundational application infrastructure services in the way that they need to for these GPU workloads simply isn't going to work without specific optimizations that are coming from neo clouds or the teams that these hyperscalers are building out. Does that mean I need a separate pane of glass to manage all of this, or how do I kind of think about that? Not that there was ever a single pane of glass to begin with, but should I really be thinking about managing this as a dedicated function, more or less?

Definitely. I think, especially for us, one of the things that Crusoe builds is what we call our command center, and basically, it's this full observability suite for all the services that you have running on Crusoe Cloud. So this means that your infrastructure can be relatively hands-off when you get it up and running.

Things like seeing telemetry across thousands of GPUs that you might have deployed with us, performance with your Kubernetes services, latency in different regions, and then finding ways to automate on top of that. So we have a product called AutoClusters. Basically, what this does is it monitors all of the workloads that you have, excuse me, to check for any error in GPUs or anything that's being emitted from the infrastructure.

In the event that a node goes bad, we remove that node from your workloads, replace it with a new node, and migrate the workload to the new VM. And so this hands-off infrastructure management is this-... kind of true north or golden standard of how you want to deal with this.

Because like I mentioned before, if you start to branch out and you're in a hyperscaler for one thing, and then three different neo clouds for different capacity or different needs or different services, then it becomes a huge pain to have to manage all of those, kind of like what you're saying. And so aggregating an observability suite into a single location, as well as having hands-off infrastructure management, means that you don't have to be so hands-on with all of your HPC workloads as maybe you are with your application infrastructure. Do you think one day we may have AI agents that are going to help us manage all these inference workloads because the level of complexity is getting higher, and maybe I can automate some of this?

I think we're there now. I think there's products that span all the way from kind of vaporware that really just wrap a foundational model and try to tell you what's going on or these kind of talk-to-your-data implementations. And then there's the actually truly agentic management of this stuff.

And so, going back to the auto clusters reference, right? Like today, it's a decision tree that's based on specific errors that come off of the GPU. So, a certain X ID error is emitted, we take a certain action based on the severity of the error.

Now, as we get more complex and sophisticated with these integrations, I can consider a wide range of data and act faster than a human could ever act with these automations. And so I think we're already at a phase where the tools and the pieces are there to agentically manage all of your infrastructure and allocate whatever you need to based on the hardware that you have assigned. That said, I think that it's a proceed with caution environment.

I think that there are some serious security concerns. I think people have probably seen what happens when some of these agents run wild and they're kind of making a little bit of their own, quote-unquote, decision making. So it's significantly constrained tooling that I think is the way to make it most effective.

Are we too obsessed with the latest and greatest GPUs, or should we be thinking more about which GPUs align better for the task at hand and the workloads at hand? Because people complain that they can't get GPUs, but as far as I can tell, that's just really the latest GPUs. Yeah.

If you look at some of the semi-analysis pricing data that you see over the past three or four years, when H100s and H200s came out, they were taking that 220, 230 a GPU hour, right? And there was a dip as Grace Hopper stuff came out, as well as the Blackwell SKUs. And now we're back to a period where those H100s and H200s are demanding the same price they were years ago.

So the reason I think that's relevant is, yeah, I think people are potentially assuming that they just need the latest SKUs to be able to accomplish a certain thing. And in some cases you do, right? Especially when it comes to inference workloads.

Every time a new SKU is released, it becomes the baseline for performance. Now, you don't always need to use the latest SKU to accomplish what you're doing. And this kind of goes back to this agent abstraction.

So, if we take a example, the infrastructure errors, and the agent picks it up to say what's actually going on, right? Maybe that initial analysis that takes in a bunch of context and data needs the latest SKU because it's the most sophisticated, and it needs to happen the fastest. But then maybe there is a branch of this agent that gets sent to a sub-agent or to something else that requires less sophistication.

You see this with the way that OpenAI deploys their latest foundational models. 8, I think at this point, right? Not every request is fielded by that.

It's routed based on an initial routing layer. And so, in the same reasoning for us, like when we are considering which GPUs to service which inference workloads, when we have these agentic infrastructure management systems, they can determine upfront how sophisticated the request is, and then route it to the SKU that makes the most sense. This allows you to have wide use of your entire fleet without specifically having to sell a certain GPU.

Do you think also that we're going to have a diverse range of processors going forward? It's not all going to be NVIDIA centered. We're seeing more and more of these AI accelerators and will that also factor into how I think about where to run what when?

I think so. There's a lot of really interesting basic providers that are coming out. I think AMD has some pretty compelling offerings when it comes to their latest 455 GPU that's coming out.

NVIDIA definitely owns the stack and owns the environment and continues to put out some pretty impressive stuff, especially as they eat into the DPU world, and they eat into the storage world, right? And so, I think that their ecosystem is pretty hard to match at this point. That said, I know that there are bespoke accelerators that are coming out constantly that have ASIC specific for inference workloads.

And so depending on the type of thing that you're running, and you have a hyper-optimized deployment for X type of inference, then I think there's definitely a play there. The hardware world's just, it's a bit of a slog, to get up to cruising altitude with those kinds of things. But, I think we'll see more and more of these as the demand expands.

Will also the nature of the types of models that people are running going to be different too? Because we see that OpenAI and Anthropic are kind of like the poster childs for these things, but there are other options, and we're starting to see some more open source AI models as well. So will there be more diversity in the workload stack?

I think so, and I think this goes back to the same kind of diversity in the GPUs. If you look at the beginning of 2026, there's this huge spike in people using open source Chinese models, right? And so they're not as sophisticated as the frontier models, and they don't actually accomplish the same type of complexity or of workload.

But depending on what you need it to do, it's a very cheap way to get inference. So, I think if you combine it on sub-10 billion or sub-50 billion parameter models with a specific chip provider, you have a very creative way to run cost-effective inference. I think people will continue to use off-the-shelf models, distill them, or tune them to whatever they need to be able to get their needs met and then deploy it to the GPU that can run the most performant, which, depending on the size of the model or the complexity of the distillation, may end up being that it's on the cheaper side or it's an older SKU.

How will this all play out from a deployment perspective? And I'm asking the question because I feel like I've witnessed the mainframe era of AI, where everything was kind of going up, maybe horizontally. And are we going to see more distributed environments where the workloads are pushed out closer to the edge, where they're near where the data's being created and consumed?

And is this going to factor into the way I think about inference? Definitely. I think edge deployments are going to be the future of how all of this stuff plays out, right?

Centralized data centers in Ashburn, Virginia, and in Oregon are great for most workloads that you see on the internet today, right? But when you have perception models that require 10-millisecond latency, or gaming or robotics that requires near real-time reprocessing of informational data, then latency matters a lot. Crusoe has a product, it's called Spark.

And basically, there's these modular data centers that can deploy anywhere there is fiber connectivity and power, right? It, one, allows us to tap into these stranded resources in solar, wind, hydro, or maybe even just power that's kind of stuck in the middle of nowhere, as long as we have internet connectivity. This means that I can put data centers modularly to the size that the power allows, as close to the workload as necessary.

And so for us, it's pretty exciting and it kind of plays right into this question that you had of, are we looking to make sure that we are deploying infrastructure as close to the workload as possible? Because that's going to be the demands and the needs of these types of applications. There's a lot of political noise these days around data centers themselves and the architecture, and I can't help but wonder if this is...

I get that it's an issue, but I also feel like it's a solvable problem, and maybe we're in the next generation of these things. So, as you look at the future of data centers, what do they look like to you? Yeah, I think there's always going to be in any kind of industrial revolution like this, right?

And some of the biggest buildings in the world are being built right now. There's going to be a polarizing side to this. But I think that kind of going to this ability to deploy modularly in locations that you wouldn't typically build brick-and-mortar data centers, this allows us to tap into stranded resources that aren't really near much, right?

If you have a solar array that sits in the middle of the desert, that has its own challenges with be it NIMBY syndrome or just the pure cost of transferring the power from there to a usable metropolitan area, then deploying a data center there makes a lot of sense and doesn't necessarily conflict with the local population much. It seems to me also that we've seen the first generation of all of this stuff was some sort of tiger team was created within an organization to manage this, and it typically had somebody who was the infrastructure person attached. But as we start to scale this out, more traditional IT teams seem to be taking over responsibility for the management of inference, and there's some shifts going on in that regard.

So what's your best advice to those folks? Because I feel like there's a lot of newbies in the game, and a lot of them may be bringing their own biases for better or worse. And as you kind of look at how all this evolves, what are you thinking?

Yeah. I think it's kind of surfacing itself in the way of, you see a lot of these very large companies looking to just make sure people are using AI by any means possible, and so they're checking total token output or total adoption across all of their team. It's kind of a blunt instrument in being able to measure, are people doing it?

Maybe someone is using a billion tokens a day, but they're just generating long-form documentation and doing nothing with it. Maybe someone is opening a pull request that's just refactoring code endlessly to make micro-optimizations that aren't necessarily pushing the business forward, and the volume of output is now creating a burden for everybody else to try and understand. So I think that for these companies, my advice and my take here is find a formulaic way of measuring the success and the efficacy of how people are using AI in your company.

Is it truly pushing forward some of the stuff that you're doing? Because, great, that's where you pour gasoline on it and you continue to spend. But if it's purely people are using it to just show that they can generate a 30-slide presentation, or a very long document or just that people are going to end up summarizing later, then it's not really an effective use of this.

This is arguably probably the most transformative time for technology in the past 100 years, right? We're seeing something incredible unfold in front of us at a processing speed and power that's way beyond human capabilities. And so for us toIt just needs to be properly harnessed in a way that makes sense, and isn't just using it just to use it.

Because I think that you could potentially create a lot more burden and noise than you intend to. And I think it filters all the way down to the consumer level, right? You look at the amount of new apps that are being created in the iOS App Store.

They're orders of magnitude than they were five, six years ago. But user adoption and installs is plummeting. There's so much volume and noise of just things that people don't even necessarily know how it applies to them or how to pick the right thing.

And so, using AI just to use AI is great, but I think that it's making sure you use it in a way that pushes forward stuff internally at your company or stuff for your personal life. To that point, aren't we going to see more focus on, I guess for lack of a better phrase, token economics, where we're going to apply more of these FinOps types concepts to the consumption of the AI workload, or at least the infrastructure consumed by that workload? And as we think that through for a minute, does that mean we might have to start picking and choosing certain workloads over others because we just can't support everything that we're trying to do, and maybe that cancer research project needs to be a higher priority than that social media meme?

I don't know. I think it's a very interesting argument. I think there's two facets to it.

Getting really good at running cost-effective inference, so KB cache management, finding ways to prevent having to overload memory and ensure that you're not recomputing things that don't need to be recomputed just to burn capital. So getting very, very effective at running inference. And the other side of it, yeah, I think that generating cats driving a cars video is probably not as impactful as some of the biotechnical or medical advancements that we could be doing.

That said, I think these companies that are doing research are buying entire clusters and spending the money because there is true return there. And so I question the effectiveness of applications that, excuse me, to just allow to generate memes or what have you. But yeah, it came down to pick one or the other.

I think that there'll be some trade-offs to be made. All right. Well, there you heard it, folks.

Hey, as you look at this stuff, the fundamentals are coming back into play for sure. And it's not just a matter of throwing infrastructure at the greatest, hottest new workload in town. We're actually going to have to manage this stuff responsibly going forward.

Hey, Kyle, thanks for being on the show. Really appreciate it, Mark. All right.

ai Leadership Insight Series. You can find this episode and others on our website. We invite you to check all those out.

Until then, we'll see you next time.