Self-Hosted Inference Brings Service Mesh Lessons to AI
Self-hosted inference is moving from experiment to strategy. Open weight models are close enough to frontier models that many teams no longer need to send every request out. Running them on your own GPUs brings new infrastructure problems.
Buoyant Co-founder and CEO William Morgan examines that shift with Alan Shimel at KubeCon + CloudNativeCon in Salt Lake City.
From Service Mesh to Inference
Buoyant created Linkerd, the first service mesh, in 2016. It was one of the earliest projects accepted into CNCF and the first service mesh to graduate. Buoyant sells an enterprise distribution for companies that need extra guarantees.
Its new open source project, Kinference, is a control plane on top of vLLM. Once a team runs more than one vLLM worker, it needs load balancing and routing. Those are the same problems Linkerd has solved for a decade, now applied to generative AI.
Morgan notes that inference requests are HTTP at the base. Yet latencies and routing decisions are completely different from traditional traffic.
Why Run Your Own GPUs
Buoyant runs inference internally on its own GPUs. Morgan says it is economically efficient and gives the company full control. It also lets the team try the latest open weight models without worrying about token costs.
He expects AI adoption to resemble multi-cloud. Organizations will keep models and data on premises and use frontier models for perhaps 10% of their work.
Balancing Two Open Source Projects
Linkerd depends on Buoyant to fund many of its maintainers. Adding a second, unrelated project means double the resources. Morgan’s confidence comes from serving the same audience of Kubernetes platform owners.
He says discipline and a clear framework for investment are essential. Even quarterly planning now feels like a stretch.
There is no commercial version of Kinference yet. Buoyant’s focus is helping companies move their inference spend onto open weight models and GPUs they control. Morgan sees that transition to self-hosted inference as the real opportunity.
Transcript
Hi, everyone. We're back here live at KubeCon, continuing our live coverage. My next guest is a KubeCon regular for us, though I don't think we had you on in the last year or two.
We kind of missed you. I can't even keep track of the KubeCons, so true. I hear you.
Well, because sometimes we'll see you in Europe and then here. Yeah. Anyway, it's William Morgan from Buoyant.
William, it's great to have you back on. Thanks. And no matter how long it's been- Yeah, yeah ...
it's great to have you on. Happy to be here. So William, you're founder of Buoyant, and of course, Buoyant is sort of a dual company where you have the huge open source project that you guys manage.
It's graduated now, isn't it? Yeah, yeah. For years.
Been graduated. Yeah. It was one of the early graduates.
As well as the commercial company, Buoyant, behind it. There's a lot to unpack there, though. We got more to talk about, but I think we got to lay that foundation out.
Yeah. Let people know. Give us the whole story.
Yeah, sure. So I'm the founder of a company called Buoyant. We have a very popular open source project called Linkerd, which is a service mesh for Kubernetes, so it adds reliability, security, observability.
It was the first service mesh to be created, so this is back in 2016, I think. I think we were the fifth project ever accepted into the CNCF and the first service mesh to graduate, so kind of the whole history of accolades. And then Buoyant the company sells basically the enterprise version of Linkerd for companies that need additional guarantees and, in some cases, features and things like that.
Yeah. Absolutely. Now, William, a lot of people know Linkerd.
Mm-hmm. Everyone knows open source, but not everyone understands the intricacy, if you will, of the relationship between an open source project- Yeah ... which, as you said, is part of CNCF now for a long time, and a company like Buoyant that still has a large hand in the main- Yeah ...
maintaining of that project- Sure ... but then also developing a commercial product around it. And that's as much art as it is science.
Yeah. It's definitely a tricky balance to get right. Yeah.
And I think what you have to... Open source, there's a wide spectrum of how these projects work, and sometimes you have open source projects where everyone works as a developer for a very large company like Google, and they kind of fund that. For a project like Linkerd, which has a very direct relationship to Buoyant, we pay a lot of the maintainers.
And so for us, we actually have to have a functioning economic engine behind the project for the project to be successful. So you have to navigate that. The other part that gets really tricky is when you start introducing other projects into the ecosystem.
So we recently introduced a project that is totally unrelated to Linkerd, but it's called kInference, which is actually a project for running inference workloads on Kubernetes. And so now we have to juggle not only the health of the open source Linkerd ecosystem and the economic powerhouse that is Buoyant, but also this new project, kInference, and make sure that it is a healthy and thriving project in a totally new space. And that's not easy, right?
No. Because it's one thing if you have one project and then you open one that's adjacent. Mm-hmm.
Where one can kind of feed the other, and you could quickly boost it up. When you have two separate ones like that- Yeah ... it's double the resources- Mm ...
is really what it comes down to. Yeah. Yeah.
It can be a tricky deal. But what gives me confidence in this case is it's the same set of customers and the same set of people who get value from both. So the reason we built kInference, which is basically a control plane layer on top of...
There's a project called vLLM, very popular- Sure ... inference worker. It's the thing that runs models on GPUs.
But as soon as you have more than one vLLM, well, now you have load balancing. Now you have routing. Now you have the whole set of problems that we've been solving with Linkerd for the past decade only applied to the world of inference, applied to generative AI.
And so the problems are the same, and actually the people adopting this, they're Kubernetes platform owners. So it's the same set of people adopting this. Even though the projects are independent in their scope, they actually have a lot of shared design and philosophy and beliefs, and we've been able to apply a lot of what we learned with Linkerd directly to kInference.
Absolutely. You mentioned the word inference a few times, and you know what? I thought at KubeCon last year, inference was kind of the bingo word of the day, right?
Yeah. How many times did you hear the word inference? Yeah.
Yeah. But I was very surprised that this year- Yeah ... inference is still kind of at the top of the heap- Yeah ...
for the amount of- Okay, good. Well, I was scared you were going to say something different. No, no, no.
Don't f**k it up. This year it was dead to us. Oh, no.
Finished. No. We're still- Yeah ...
talking about inference. Yeah. And quite frankly, I think we're going to be talking about inference in the years to come.
Yeah. Because I think what people are going to recognize is that it's a hell of a lot cheaper running inference than it is going back to the source every time- Yeah, yeah ... and starting reinventing the wheel.
Yeah, yeah. No, and that's exactly our point of view, too, is we at Buoyant, we actually run inference internally. So we have our own GPUs.
We send almost all of our workload to those GPUs, and it's very economically efficient. It gives us full control of everything. It means we get to play with the latest and greatest open weight models, which are very, very close in parity to the frontier models.
Arguably have surpassed it, or arguably are close enough that it doesn't matter. And it is almost liberating at times to say, "You know what? I don't worry about the token cost.
" Yeah. Yeah. So I guess my question though is, well, if anyone is a veteran of the whole CNCF process, you are, right?
You're a living, walking example of it. I got the scars. And you got the...
Well, you went there. What made you sign up to do it again? Well, I think it's almost a tactical decision at this point, which is every part of the stack has an open source component.
And whether you like that or not, if you want to be part of the stack, if you want to play in the Kubernetes ecosystem, you have to have an open source footprint in there. Now, that said, there's a big spectrum there. Linkerd is on one extreme.
It's a graduated project. The trademark is owned by the Linux Foundation, like we're all in. kInference, very, very new project, is on the other extreme.
It's open source, but fledgling community, and healthy adoption and contributions, but a tiny fraction of what Linkerd is. How does kInference evolve? There's many, many different options- Right ...
there. But I think you have to have an open source component to play in this ecosystem. And also, I've been doing open source stuff for decades now, so it doesn't feel strange or foreign to me.
Open source certainly has been affected by AI, and there's different ways we think about it now and how do we want to handle this project, and what happens when people send us 50 PRs from their AI agents and what do we do? It's changed, yeah. Yeah.
It's changed, but it's still kind of the centerpiece of how software gets developed and operated in Kubernetes. Let me put forth a supposition to you. Yeah.
That's a big word. As much as inference was big last year, big this year, will continue to be big. As much as AI has changed open source, I think over the next few years, open source's influence on AI as we move to truly open models, not just open weight- Mm-hmm ...
but open models. Yeah Running these AI models, running these AppSec, increasingly, they're all running on Kube anyway already. They're running on cloud native, they're running on Linkerd, they're running on these open source components.
I think open source, just as it is for other software, it becomes the stack. Yeah. The compute stack that this thing runs on.
Yeah. I think that makes sense. What's remarkable to me about this ecosystem is there's so much innovation and there's so much pain that people are experiencing at every level of the stack.
So there's the core inference, like how do we literally get the next token for so much interesting stuff happening there, so much innovation, new types of models, new ways of doing this, new ways of thinking about it, new ways of manipulating the GPU cache, every aspect there. And then on the other side, there's like, well, how do we structure our agents? And are we having sub-agents?
And are we building software factories? And what are the loops that we're putting in place that make you see inference? And then in the middle, you've got all the infrastructure stuff, which is, of course, the fun part for me.
How are we load balancing and how are we routing? And what does it mean? These inference requests are happening.
They're HDP requests at the base. " Well, but they're totally different. The latencies you expect, how to understand where to route things are, every detail is different.
So every aspect of this ecosystem, there's problems to be solved, and there's excitement and interesting things to do. There is. So I've been in tech a long time, and I'll admit it, my name's Allen, and I'm a shiny trinket- Uh-huh ...
aholic. Right. How do you, William, at Buoyant, you got this new baby now, right?
It's the new baby, and it's in this hot, sexy area of what we're doing. But how do you keep Linkerd humming, moving? Yeah.
Right? How do you give it the attention it needs and deserves? Yeah.
You have to have a strategy and a framework for thinking about how you're going to invest your resources. And that's true regardless of AI, regardless of open source, regardless of whatever. If you're running a company, you only have a certain amount of money, people, time.
You have to be very disciplined in how you do that. And I think in order to survive, you have to continually have a framework and have the wherewithal to apply that framework. So do I have any magic sauce?
No, not really, other than I've drawn a boundary around here's where we're going with Linkerd, here's where we're going with K Inference, here's where we're going with Buoyant Enterprise for Linkerd, which is our highly powered enterprise grade Linkerd distribution, and just being really disciplined and focused about how you do that. And also, I think maybe in the modern world, being attuned to how fast things can change. Quarterly planning already starts to feel like a stretch.
In three months from now, who knows? Year long planning, okay, well, we'll do it, but let's just recognize a year from now, we're probably going to have seven different opinions. Agree.
It's funny you mentioned the Buoyant Enterprise, which is the commercial add-on or commercial version for Linkerd. Yep. What's the commercial version for the new project?
Yeah. We don't- Have one yet ... we don't have one.
Yeah. I think that has to come as a secondary piece. I have lots of ideas for where that is, but it almost doesn't matter in the beginning until you've proved out that, hey, this project has some value.
I'll say a lot of the focus for us at Buoyant has actually not been on monetizing K Inference or building an enterprise version of K Inference. " Yep. K Inference is a part of that.
So is DL LLM, so is a bunch of other open source projects, but there's a whole ball of difficult and interesting problems you have to solve. And I think that's going to play out over the next months and years because I do believe that the dominant form of AI adoption is going to be akin to a multi-cloud thing where you're going to have a harness, you're going to have on-prem model, on-prem data, you're going to use the frontier models for maybe 10% of your use. Yeah.
And so we have got to make it simple to set that sort of multi-model, multi AI system up, and open source is the best way to do it. Yeah, exactly. So you and I are on the same page.
For me, the commercial value that I see ahead of Buoyant is in helping companies make that transition really easy. Really easy. I don't think that's a product called K Inference Enterprise.
I think that's a different product that involves a whole lot of open source and involves a whole lot of closed source and- It'll be the new stack for that. They'll have... And it'll be based on open source.
It'll be based probably on a lot of the projects we see here at CNCF. There'll be some new stuff added in, obviously. Yeah.
But I think that's where we go. Yeah. We're running out of time, but for people who want to get more information, William, give them where to go.
Yeah. Well, you've got two choices. ai, letter K and then the word inference.
io and click Talk to William. Talk to William. io, find out what's going on with Linkerd- That's right ...
and with Buoyant Enterprise. That's right. That's why I'm here, to finish it up for you, William.
Appreciate it. All right. Hey, we're going to continue our live coverage here at KubeCon in Salt Lake City.
Having a great time. Stand by, we'll be right back.