ThisWay Global CEO Stephan Fabel Says We’re Holding GPUs Wrong

ai Leadership Insight Series. I'm your host, Mike Bizos. Today, we're with Stefan Fabul, CEO of ThisWay Global, and we're having a chat about how AI workloads are going to evolve because, well, what was relatively straightforward is getting a little more complex.

Stefan, welcome to the show. Hey, Mike. Thanks for having me.

All right. So what's going on here from your perspective? Because it started out that there were basically, I had to worry about two things.

There was training, and then there was inference. And basically, I could run the inference maybe on different processors, but most of the time, that landed on GPUs that I also used for training, and life was simple. Now, as I kind of see where things are going, AI workloads have a, there's more classes of them.

There's a little more nuance. There's a lot more AI accelerators. They might be running anywhere from the edge to the network to the cloud.

What's going on here, and what should we be thinking about? Well, I think, Mike, what we're seeing here is simply a proliferation of different use cases, and that goes hand in hand with processor architectures, GPU architectures, system architectures being more and more specialized towards parts of the stack, subsections of the workflow, and excel at certain parts and certain operations that are part of a certain step in the workflow, the overall pipeline, whatever that might be. And we'll see a proliferation of silicon that is addressing all that.

And because of that, you're now seeing challenges in the way that these workloads have to be orchestrated, workloads have to be executed, and workload placement is going to become more and more important from an operational perspective. So that's really what's going on. When I started at, so I spent six years in video, and I started there 2020, it was basically, well, we're a GPU.

We're not a CPU. CPU can do everything. A GPU is very specialized.

And now it's like you read things like, well, a GPU is very general, but an LPU is very specialized. So, I think what we're doing is we're just basically breaking it down into smaller and smaller chunks and developing silicon for an ever-evolving portion of the pipeline and has different kind of strengths and different parts of that workflow really for AI workloads. How much of this is driven by, shall we say, good computer science?

And how much of it is just a reaction to the fact that GPUs are scarce, can't find enough of them, and so people are looking for some place to run workloads that maybe either, A, they can get, or B, isn't quite as expensive. So it's really a exercise in economics. It is an exercise in economics.

I think that it's also not just engineering for engineering's sake. That silicon is real engineering and has real purpose. It's not pseudoscience or anything like that.

I think it's also hard work and great achievement in and of itself, right? Each of these developments are commendable. And I think it is driven by token economics, right?

Really going into a space where if I am offering inference or process tokens for your agenda guide workloads, then I need to be able to somewhat eke out margin. And so if these chips help me do that, then all the better. But I think if you take a step back, the real issue is that I think what we're finding is that a lot of the inference providers that are out there are really in an arbitrage game that they're feeling the squeeze.

So if you feel the squeeze, first step is you try to be better operationally, but there's going to be a limit to that. So it's going to be interesting how this part of the market evolves. As we go down this path, who's taking the lead?

And I'm asking the question because early on, I would encounter a lot of what I would call tiger teams, and they were specialists who were deploying workloads, and they were set up through this particular project. But as AI becomes a more mainstream workload, is this shifting to being yet just another type of workload that's managed by the typical IT team? I think you're onto something.

So I think that what we see is a proliferation of use cases now, right? We see AI being embedded in every application, and it's sort of making its way through this application space, right? So, the average enterprise has about 2,000 or so internal workloads that all are running.

Think about HR, think about any of the internal applications that a large enterprise would want. Now, each one of these enterprises needs a chatbot, let's say, right? Like what happens, right?

You have to somehow introduce this capability now into a variety of workloads into, if that means, into a variety of QA pipelines, CI/CD pipelines. You're now introducing it into, let's say more differentiated tech stacks like SaaS stacks, right? We now have cloud-native approaches.

So as you go through each of these applications, you're now asking yourself the question, well, what does it take for me to add AI to this application, and in what manner am I adding AI? Like I said, is it a chatbot? Is it a rack?

Is it some agentic system that I'm somehow adding to automate my workflows, and so on and so forth. And so as we go through this broad rollout of AI capability across the various different application stacks that are out there, you'll see it taking on a whole bunch of different forms. And one of those forms is certainly a recipe, and that fits inside your CI/CD pipeline, right?

So if you can imagine, right, there's your loop, right? You have a CI/CD pipeline. You roll out and update your applications, and then there's a little mini loop in there that updates the model and creates a new version of the latest inference that you add to an AI capability that you add to your app.

So, you can see that pattern. You can see other patterns as well that match whatever workload you're doing. Certainly edge, robotics, automotive, all of these sort of physical AI examples or examples of maybe edge cases of AI where we sort of kind of a frontier case, right?

You'll see it take on different forms and probably requiring different silicon at the edge to excel. And as part of that, will we be bringing the AI model maybe closer to the data? Because I think if you look at the first wave of this, it's making API calls out to frontier model services like Anthropic and OpenAI, but that can get expensive in a hurry.

Plus, I'm going to have some latency issues, so am I going to see more smaller models maybe deployed on premises, and are some of those models maybe going to be open weight or open source because, well, the cost of these things is rising? I agree. I think this is certainly what I'm thinking.

What we're doing is we're betting on smaller models being produced, and I'm not necessarily convinced that it's all going to be open source. I think there's going to be a case for many different ways in which models are being created and consumed. So, for example, it's, I think, entirely reasonable that there's room for more than five models in this world, right?

If you say, well, what else could be doable? Well, you could imagine that something like an English language model is open source or is sort of freely accessible or readily accessible because it's language. It's not yours, it's not mine, it's ours, right?

It's a thing that intuitively we want to interact with. But then there's other models where you would not necessarily go that way. So, for example, think of a model that detects cancer with a much higher accuracy on radiology than others.

Right now, you would think that's entirely proprietary, very valuable, absolutely not easily accessible or open source, and likely, somebody like a GE or a Siemens would want to hold onto that as part of their critical IP that they collect, right? So there's- Further models where you could imagine a consortium or a standards body of sorts might want to align on a model, and then that's available under a license. Like you could imagine a world where EV manufacturers might be required to base their EV on a specific foundation model because that's the one that's approved by the legislature.

And saying, like, "Okay, well, you can do whatever you want, or BMW or whatever, but you can fine-tune it. " So you could imagine many different ways and shapes this could take. I don't think we're quite there yet.

I think we're working our way to that state, but yeah. Do you think that might break down to the classic 80/20 rule where 80% of the time we're using models that are good enough? I mean, we don't need them to be perfect, and frankly, most of what businesses are trying to automate is not rocket science.

And then there's the actual rocket science where I am trying to cure cancer, and I need some advanced model, and that's the other 20% of the equation. But those are the things that require a little more R&D and therefore a little more proprietary approach. Well, they require first and foremost unfettered access to GPUs.

I mean, yeah, you could be a little bit laissez-faire about it, but I would say maybe there's 10 people in the world who have access to GPUs, and five of them are customers of the other five. And so that's sort of what it is. And you're trying to work with that, but what that means is that everybody's access to AI and GPUs are essentially disintermediated by one of these five people.

So, it also means that everything we learn about the way that this market evolves and what people do with AI, et cetera, is all through the lens of these five people. And so what that leads to is, I think, a skewed understanding of what the potential is for AI and what we actually can do with it, and what we can do with different and newer models, alternative approaches, things where maybe we don't have to hand over everything, like the knowledge about cancer and cancer research and at the same time knowing how to bake a cake and generate a cat picture to the same exact company. I think that there is room for regulated industry here, and I think there's a big argument to be made that those, in fact, A, have to be more empowered, number one.

Number two, I think that's also where the value really is. So to me, yes, absolutely. I think that regulated industry is the way to go right now.

Are you saying that not all tokens necessarily are of equal value because one of those tokens might be used for cancer research versus the other one is used for creating a social media post? I mean, I will say this, I am not surprised that people are protesting against data centers if all we do is cat pictures. Fair enough.

Are we about to see something that looks like the equivalent of, you've heard the phrase FitOps, and organizations have been trying to do cost optimization of workloads for a while. Will that come to AI? Well, I'm glad you asked.

That is basically what we started out as. We saw a huge efficiency gap in the way that GPUs are utilized. And in fact, when we started asking around, most GPU utilization is below 20%.

We saw an article recently from even some of the frontier labs, they produced abysmal numbers in terms of GPU utilization. And it begs the question, given the price of these things and given the capital required and all the effort required to put them to bear and bring them into production, actually make them useful to somebody, well, why are we happy with less than 20% utilization? So this is really where we come in and saw that the way that we think about how we consume GPUs actually essentially begets this situation.

So what I mean by that is, think about it right now, we're saying, well, I'm renting a cluster of GPUs. And that cluster of GPUs is X GPUs for Y months or Y years or something along those lines. But who says that that's how many you need?

If you train a model and somebody says, "I need 6,000 GPUs for four months," how did they come up with that? It's essentially a finger in the wind, really. There's probably some science and estimation, et cetera, but at the end of the day, it's an estimate.

So it's no surprise that A, if you have these large model training runs that first of all you're going to work your way into being able to actually execute that run. You're going to fail a couple of times. You make 100 GPUs work, then 1,000 GPUs work, then maybe 6,000 GPUs work, and you keep working your way forward.

So overall, of course, utilization isn't that great, but it does require the dedicated availability of these GPUs across your training run. But over time, if you think about a situation where if you try to get, let's say, 4,000 GPUs, and you know that that means that you need to at least commit to three to six years to somebody, that over those six years, your GPU utilization is not going to be 90%. And only in the rarest cases it will.

So we're thinking about consuming this differently than we probably should. I think we're holding it wrong. Right.

Well, in theory, a lot of these workloads are supposed to be running on Kubernetes clusters, and one of the joys of that is it's an orchestration engine that theoretically at least allows me to scale things up and down. But thus far, as far as I can tell, things scale up and they never scale down. So will we get any better at this kind of scaling up and down dynamically in the age of AI, and will necessity be the mother of invention here?

I think so. I think we need a new layer of abstraction here. I think that conceptualizing GPU access as clusters is fundamentally flawed.

It means that we leave the estimation of how much is needed for whatever you're trying to do up to the human. And I think that ultimately that's not the place. Workload placement, workload estimation, optimization over time, that's what computers are good at.

That's what software's supposed to be doing, and I think there's algorithms that we can apply here. And so this is what our software stack does. It essentially takes a utilization number that generally is below 20%, matches up the right workload to the right architecture.

So in other words, who's to say that you're even on the right GPU? Maybe you need a higher floating point accuracy, and you're on one that just doesn't do that very well. It excels at a different spectrum of the workflow.

Maybe your silicon isn't exactly matched up right. And certainly with the proliferation of silicon, heterogeneous setup is much more likely and probable for everybody. So an orchestration engine that is able to manage all that on your behalf, we have many customers coming in, matrix against many capabilities of your infrastructure, and then matrix again against maybe your sovereign requirements, your compliance requirements, your policy requirements for regulated industries.

That's I think what's really needed here, and so that's what we're after. Ultimately, will that be managed by what I call mere mortal IT administrators, or are we going to need something that feels like an AI ops engineer specialist that will be added to each IT team? No, I think ultimately it's supposed to bring it down to sort of the mere mortal admin level.

I think maybe you've observed this multiple times. I mean, we've always seen like, okay, new ways to do things. All right, they're specialists, and then eventually it'll be proliferated out to everybody and it'll be mainstream.

So I do think there's a journey here. I think certainly right now we pay a lot of money to people who know how to run this stuff at scale. And I think eventually that'll just become mainstream.

Yeah. All right, folks. Well, you heard it here.

There might not necessarily be new IT jobs being created, but the jobs we do have are certainly going to be different in the age of AI as we all start managing these workloads. Hey, Stefan, thanks for being on the show. Thanks for having me.

All right. ai Leadership Insight Series. You can find this episode and others on our website.

We invite you to check all those out. Until then, we'll see you next time.