Navigating the Future of AI Infrastructure with Stephen Watt | Open Source Summit NA 2025

Hello and welcome back to the Open Source Summit here in Denver. We're talking with Steven Watt, who's the vice president in the office of the CTO, responsible for software engineering. And we're talking about how this whole pace of change in the land AI is rapid and now it feels like it's becoming more of a traditional IT task rather than just something that some obscure little data science team did on the side.

Steven, welcome to The show. Yeah, thanks. I appreciate being here.

Um, got a lot to talk about. I think you're spot on. I think there's a number of reasons that, uh, necessitate that, uh, cost is one.

This is very expensive compute infrastructure. Um, and so, uh, what we're, what a lot of companies are trying to do is get the best use out of their investments, which tends to mean centralize it and then make it available to multiple teams. Um, so we're seeing that sort of convergence as opposed to each different data science team having their own clusters.

Uh, but then there's also like the explosion of generative ai, which and all the different use cases around that. And then that puts duress on a particular single cluster instance. And so it needs to be able to have a way to efficiently scale that to meet the internal demand.

Mm-hmm. And I've gotta scale that, not just up, but back down again because it's essentially a multi-tenant environment now that I'm trying to support multiple workloads, and this is a little challenging, you've gotta manage this almost like an internal cloud provider. Yes, absolutely.

I mean, I think, you know, historically what you've seen with inferencing is, uh, we've had, uh, which is an open source inference, uh, server that was, uh, came outta uc, Berkeley, their Skylab, and uh, red Hat, um, has acquired neural magic and, uh, neural magic's the largest commercial contributor to VLLM. So that was our initial focus. There is on inference, uh, but essentially, um, the, the cluster needs to scale to sort of maintain, um, the ability to, to continue performing under increased load.

And those loads are, um, not just coming from human adoption, but also agentic architecture. So it's not just people, it's agents, it's other processes that are putting duress on the cluster. And so LLMD was a project that we created that allowed us to disaggregate some of the key steps in inferencing.

So specifically pre-fill and decode one, which is GPU bound, another which is memory bound. And that's sort of a classic move from scale up to scale out. We were able to take certain components, put them on, uh, different, um, infrastructure and sort of be able to scale performance for that.

Mm-hmm. You know, it's funny, everybody talks about AI agents and we're gonna have thousands of these AI agents and somewhere there's some folks running infrastructure going, come again. How many, what, what will these agents do to the demands for infrastructure, the throughput, the capacity?

Are we gonna have to rethink the entire stack to accommodate all these AI agents that are gonna be running 24 7 constantly pinging on infrastructure that wasn't designed for that? Yeah. Well, the way I reason about this is, um, it's a new class of applications.

And what I mean by that is, you know, if you had a, um, especially in like open source and platforms, there's sort of three chapters in the history. Like there's a chapter one around Linux and a class of applications that run on Linux, a chapter two, as the world moved from scale up to scale out a chapter two that runs distributed systems, a whole new class of applications that are, we know is cloud native today. And then now with inference servers and generative ai, there's a new class of applications that are built around essentially, uh, open AI's interface protocol, which has just become the standard for inferencing across the industry.

And so for that, you've got multiple uses of that. One you have like human to interface server, which as we know is the chat bot, right? So sort of open ai, um, and chat GPT made that the Norman helped us understand there's many different in instantiations of that and, and internal organizations are using that.

But then you have agents to the server and then agents to agents. And I think you're seeing, uh, an exponential, I think we will see an exponential explosion in the amount of agents as that maturity curve grows. The question is like what your question was specifically, what infrastructure does it, is it served by?

And that is like, I think it flows along a very similar trajectory that we had with Cloud Native where you started with a couple of servers and then as the demand increase, you increase IT. Projects like LLMD allow you to do that. And they also have auto scalers that are built into that.

And so as demand come, um, may come down also for specific components where if you're GPU bound or memory bound, uh, compute bound or memory bound, you're able to scale up discrete components of that. But it is a new class of in infras uh, applications and like Kubernetes, it required a, a new set of deployments inside IT infrastructures to be able to service these new classes they run on Kubernetes. So in the same way that Kubernetes runs on Linux, LLMD runs on Kubernetes and takes a benefit of all those distributed computing principles.

But I think there are some changes and new skills that are gonna be have to learn to be able to support the explosion of AgTech. I also feel like we're trying to make an effort to kind of decouple maybe training and inference a little bit where I can now have the inference run anywhere almost on any class of processors and everything doesn't have to be wrapped around A GPU per se, and there's just gonna be a lot more diversity in the ecosystem. So are we creating a level of abstraction that kinda lets us invoke that underlying hardware in a way that's a little more dynamic?

Yeah, I think the, the way I reason about this is predictive versus generative ai. So like predictive like classifiers that we've been using for a long, long time. Um, this work pretty good on CPUs, right?

And they're smaller models, uh, able to fit into smaller memory footprints. Um, the, it's kind of interesting. It's like, I would say there's only like five models that matter, you know, on the generative side, although which five they are tends to change like every month or every two months.

There's millions of models, but there's quite a high turnover, um, as to which ones we're going to uh, this month. And those are generally addressed by GPUs and you need like the best price performance GPUs to be able to serve those even in inferencing. Um, the challenge though is you can get some heterogeneous infrastructure and so like optionality and being able to hue your Nvidia like H one hundreds as well as like your A-M-D-M-I three hundreds, um, together especially, I dunno, some places I've discovered are a MD shops and some places are NVIDIA shops.

And so being able to sort of have one project that can address both, uh, is pretty important. Are we in danger of locking ourselves into too many things? And sometimes I feel like, and we've been talking about this forever, but developers are always kind of trying to optimize something and then they write to a lower level API somewhere that doesn't work somewhere else.

And then the IT department wakes up in the morning and goes, how come we're locked into this? Yeah. Is there some way to think about this earlier this time around where we can get this right now as opposed to, you know, trying to undo it later?

Yeah, I am. I can tell you I'm thinking about it, it keeps me up at night. Um, there's a couple of really interesting projects, but I will say like, it's a hard problem.

Like if you look at our previous open source chapters around like Linux and Cloud Native, like it really was an X 86 world and then it was like an X 86 plus a little bit of arm world. And uh, those are still true, but in a generative world, we're, which, you know, let's just be frank, is dominated by the Nvidia, um, architectures and the Cuda ecosystem. The, um, it's a little tricky because you have the ability to light up GPUs inside, um, the inference server.

And that is done through a software layer around GPU kernels. Okay. Um, those GPU kernels, there is an open project called Triton, not to be confused with Triton inference server, um, Triton basically allows you to write those GPU kernels that enable the models in an abstraction layer that works on NVIDIA or A MD or Intel, anything that's a classic GPU architecture.

But the world's a bit more complicated than that than it was with the X 86. We have TPUs, we have npu, we have inference specific accelerators like seus, cereus and Grok. And we have to figure out a consistent way to light those all up in PyTorch.

And they are very different architectures and some of them, like for example, A GPU while is really great for, um, supporting generative workloads. It's also built for gaming and it's useful in cryptocurrency mining. And whereas these other accelerators have different architectures, they consume way less power.

They're just as effective for generative but not so on video games. And, you know, and so trying to figure out how to light up all these hardware architectures in one single GPU kernel programming LA layer is a challenge, uh, that we're going after at Red Hat. Uh, we're starting with the Triton project, but we're looking at ways to increase the scope to be able to create as bigger an umbrella as possible.

I also think we obsess with the large foundational models. Yeah. But in practice, it seems to me anyway, most IT teams are gonna wind up working with smaller models that are kind of being used to drive a, a narrow use case for an agent.

And then I'm gonna try to orchestrate all these things. So, uh, are we not paying enough attention to the requirements for smaller models? 'cause we're all obsessed about the big models.

I I think you're correct in that, um, we will have a set of purpose fit models for specific tasks. And Where I think we're gonna see the industry going is smart routing. So we're gonna have a distributed cluster that can servee a variety of models.

Alright? Um, those models will have different strengths and likely what you're gonna have in front of that is a, some sort of intelligent semantic router that looking at what your prompt is can help you figure out which model to go to. Like there's some models that are like really great on physics.

So if you have a physics question or a science question, you should go there versus there, right? And being able to understand where to route. And I think that's gonna be hidden from the user.

And you're right in that if you have single purpose models, they don't really need to know arcane details about, you know, who won the Indy 500 and at this time, you know, it's, it's literally like science, you know, and that'll allow them to become, it. It also, it is useful for model shrinking techniques like s scarification and quantization for you to be able to prune out the stuff the model doesn't need to allow it to run in smaller memory resource budgets. So I think we'll see all of that over time.

We just need the plumbing to be able to hide that from the user and make it an intuitive experience. Right. And the paradox of that is the less the model knows, the more accurate it may be.

Yes. Yeah, yeah, yeah, yeah, yeah. You can certainly run into overfit problems where, you know, you're trying to stuff too much into too small of a brain and it it has to use, uh, uh, like trim out what it needs to, uh, what it knows already to be able to fit all the stuff you're trying to put in, uh, is an overheating, simplistic way of describing it.

But, um, uh, yes. And, um, but I think all the plumbing is there in LLMD it has sort of inference pools that you can host different kinds of models. And, and not only that, um, there's other reasons to use different kinds of models like cost.

So you can have policies for example of like, let's say, you know, I bought a bunch of GPUs five years, five years ago. They're not near as powerful as the one I bought this year. Um, still want to get good, good hue out of them.

I'm gonna have low priority tasks routed there versus higher priority tasks routed there. These all capabilities we're putting into the routing, um, in front of the inference engine. Yeah.

So last question. What's the new thing that, you know, as you look in your crystal ball or the thing that's got you most excited, you know, the thing you're going, wow, this is the next thing that's gonna be cool. Yeah.

Well, I would say, although I don't want to sort of jump on the bandwagon, um, it really is a agentic architectures. And what I mean by that is I look at it, i, I frame it slightly differently though. It's a new class of applications and when that happens, the industry has to figure out how to do all kinds of non-functional things.

How do you secure agents if you kick off an agent? Does it have the same permissions that you do when you try and do things? Your permissions are long lived, like your access to some, you know, your Workday reports or something like that is probably valid for as long as you are at the company.

But if you have an agent, do that, is it, does it have those same permissions for multiple days, 45 minutes? You know, we have to figure out identity authorization, all kinds of security aspects around this ecosystem because, you know, like with model context protocol, a lot of these agents sort of run with the implicit authority that that's just assumed they're allowed to do what they're trying to do. And so this is what excites me.

There's all this space in this new class of applications that just we haven't figured out yet. And, um, and so this is something we're doing in office of the CTO, both, um, in our research infrastructure as well as our emerging technologies teams, looking at different spaces like security, identity, um, performance, and trying to figure out how to make this new class of applications work. Well, I may be a bad example.

I was trained by Jesuits who taught me that it's easier to ask for forgiveness than permission, than Permission. I like that ethos. I, I live by that too.

Yeah. Anyway, thanks for coming back, buddy. Yeah, my pleasure.

It was great chatting to you, Mike.