Why Kubernetes Utilization Is Stuck Below 40%

Hey guys, thanks for the thrill. We're here with Eli Birger, who is the head of technology for Perfect Scale, which is an arm of DOIT, and we're having a conversation here about Kubernetes optimizations, clusters, governance, and everything that goes into that. Eli, welcome to the show.

Hi, Mike. Thanks for having me here. I think we had some expectation around Kubernetes that things were going to more or less automatically scale up and down, more importantly, as required.

But it appears that the utilization rates on Kubernetes clusters are probably no better than any other platform we've seen before. So what exactly is going on here, and what can be done about it? Oh, yeah.

So you are absolutely right. Everybody expects that Kubernetes will scale up and scale down things automatically. That's the big promise.

However, there is a set of definitions or configuration that must be done, and the expectation or assumption in the market is that developer will do those calibrations and will tell the system when to scale, how much resources are needed, and then, yes, definitely system will do whatever developer said. However, developers are mainly driven, when they are being asked to do that, their main driver is fear. Fear that they will not have enough resources, and the application will crash, collapse, restart, whatever it is.

So they tend to over-provision. Now, when everybody is slightly over-provisioning here and there, or not even slightly, we ending up with huge pile of small services, each one of them heavily over-provisioned, and at the end of the day, it becomes a huge problem. What we see in our own reports that we published probably two years ago, and it didn't change so much since then, utilization rarely crosses 30, 40%.

Mm-hmm. And the expectation that developers will solve this problem is kind of unfeasible. Why it is unfeasible?

Because when my main driver is resiliency, performance, and fear, most of all, I don't have any incentive, I don't have any clue, and there is many decisions. Moreover, this is not a one-time operation. The load fluctuates over time.

The amount of customers fluctuates over time. There is seasonality, there are spikes, there is might be a TV show, or imagine you are a platform, you need to be prepared for all that. So the expectation of people constantly and continuously tuning these systems are unfeasible, and this is where solutions like Perfect Scale come to solve that exact problem.

So what is to be done here then? And I assume it's going to fall to the responsibility of an IT operations, DevOps, or platform engineering team to kind of go in there and maximize those utilization rates, because I think they can see the waste that you're talking about. But do they have the tools and expertise required to do that, or do they just kind of let the machines run and hope for the best?

That's a great question, and the answer is that before Perfect Scale and competitors came to this market, the expectation or the attempt was to use monitoring systems. However, monitoring systems are not built for this effort. Monitoring system are collecting all information, and they let you swim within the ocean of these scattered pieces of information to figure out what you need to do.

It's absolutely not possible. It's a problem of scale and human capabilities at all. So there is a way, and there is another way, the way to capture the right things and give the right algorithms that would safely reduce where it's possible, or the other flip of the same coin, if you under-provision, then you already may have a problem.

What our reports are showing that despite having 60, 70% of waste, the same exact clusters showing around 20 to 25% of workloads with daily problems like out of memory, eviction, throttling, and many other which are harming their customers. Will this issue get more attention in the age of AI because we're seeing more AI workloads being dropped on Kubernetes clusters, and they tend to be a little bit on the data-intensive side, and people are concerned about scarcity of servers and GPUs and memory. So are we going to get smarter about all this?

Yes. So the amount of used compute increasing heavily. The amount of deployments and code changes increasing heavily.

We see it everywhere. Developers got this amazing AI vibe coding capabilities, which led to colossal amount of cost increase across many clusters that we saw because, again, there is a lot of changes happening. The DevOps, the operational teams, they have this inherited and built-in conflict.

Platform operational teams, they are focused on resiliency, stability, and they see every single one of them as a threat. So who must take the lead? And there is a FinOps at the end of the day who're reporting the final cost of all this operation.

And this is why autonomous solutions is the only way to go. And to your point about that, have we reached a level of complexity where it's just not possible for humans to keep up with that rate of change anyway? So if we don't automate it, we're just going to collapse of our own weight, right?

Or we're going to just spend all the money that we have and then collapse. As you look at all this stuff, do we need to rethink the way our IT teams are organized because everybody has these competing priorities? And I can't help but wonder, maybe we just don't tell the developers what the true costs are, and if we did, they might behave differently, or is that a hopeless cause?

Oh, this paradigm was discussed initially when the FinOps organization just came. The first idea was, "Let's do a chargeback. " But at the end of the day, there is a limited amount of things that particular person can do, and not every developer is an expert of tuning Kubernetes resources.

So this expectation is kind of unrealistic. We need tools, we need automated tools that will solve this problem. So why haven't we seen more automation applied here?

Is it that people just kind of assume that they're going to do that manually somehow or other, or are they just afraid of the automation, or they don't understand the automation, or what's the hurdle or the obstacle? So first of all, not everybody aware that there is a solution. It's still relatively green field.

Let's say Perfect Scale exists for three years. Other competitors, the oldest one is probably Kubecost, four and a half years, five maybe. So the problem is relatively new.

Not many people heard about actual solution exists in the market. There is many solutions that harm customers. When you're trying to reduce and you are not really knowing what you do, you can cause damage because you run those workloads to serve your customer.

This is your business. The cost, it will take as much as it take, and the potential harm of underprovisioning and failing is much higher than just overspending. Okay.

I'm with you. So basically, I guess more folks are interested in just making sure the applications are available, and they're not all that concerned maybe about the cost or maybe the business side doesn't understand this issue, and they kind of just think it's some sort of IT thing. But if the economy gets a little uncertain, do you think people start having this conversation a little more deeply or, and start thinking about this a lot harder than they normally do?

Absolutely. This exact thing happened back in 2023 when it was very easy to get money. We met companies and they said, "Cost?

I don't care. I'm scaling. " But one year later, we met exact same company, and they eventually understood that they need to optimize the other because it eat all the revenue.

The cloud eats all the revenue. Will we apply AI to these automation frameworks, and what might that look like? And will that, in turn, make the whole thing maybe more accessible to a broader number of people, so maybe I don't have to be the world's greatest DevOps engineer to succeed?

AI is amazing capability, but it requires error control. When we're talking about production-grade systems, the error control is a key. That was the first reason to have this overprovisioning because it's very hard to figure out what exactly needed and how much.

Relying on AI, which will do... In many cases, it will do amazing. In many cases, it will not, and you are trying to avoid those cases as much as you can.

So in our system, for example, we use AI for some pieces of analysis, but the final decision are battle-proven algorithms. Mm-hmm. Do we have confidence in those algorithms?

Because I think part of the issue I sometimes hear from IT folks is they're not sure how much they want to trust the AI to go automate something on their behalf. And, sometimes, to them, it looks like a black box. So maybe, part of the issue here is we just haven't had enough time to develop the level of confidence we need?

Yeah. So this is why I said we're using the AI. I completely agree with this position.

It's extremely risky to put high performance or highly available or highly needed application under management of AI because, again, of the error control, and this is why we're using AI to, let's say, help people to understand the outcomes. But the automation is relying on the battle-proven algorithms that we prove over time with hundreds of production clusters being automated, that this is safe. " Oh, that's a great question.

Kubernetes is kind of a set of Lego, and you can build different things with it. You can build the level of complexity of, let's say, bicycle, and you can build a starship. The main advice that I can give is keep it simple.

Mm. You have to build it in a way that when you look at this, being waking up in the middle of night, you can understand how is it built. If it requires prior knowledge, tribal knowledge, or anything like that, it's unreliable.

You cannot rely on that. Simplify your systems. " So has Kubernetes kind of forced the platform engineering conversation?

So I'm meeting daily customers of different types. Some of them have centralized DevOps teams, some have cloud engineering team, some have platform engineering team, and alongside other teams. The real problem, or the real direction this goes is, as a company, you have to make a decision.

You want to move fast, but you have to make this decision whether you serving your developers with ready platform where they can develop fast and completely abstract the idea of a platform, and you give another team to manage it, or you want each development team to move separately and not being blocked or limited by this platform engineering team. Generally, I like the idea of a platform engineering team where there is one owner for the entire platform. I like this idea very much.

Developers complain that this limits sometimes. But the Dr. Seuss started with challenge of this is 200 words, and all the rest is the history.

So having these limitations, in many cases, drives the creativity and better results. So do you think at the end of the day that from we've just kind of let the developers operate under this illusion that they have infinite resources, and of course, they did what anybody would naturally do. And so do we need to have a conversation with them that at least says, "Maybe we don't expect you to tune everything on a daily basis," but maybe there is a conversation to be had about the awareness of the limitations at hand and they need to learn how to share.

Yeah. So this is a question of design. How do you build error-prone design?

So how do you build your system in a way that it limits developer to some extent, but on the other hand, it does not block. Right? So, what you're trying to achieve is built-in enforcements in particular cases.

Like streamline everything, but enforce limits. Right. So final question.

What's your best advice to folks about how to start this conversation or have this conversation? Because I think sometimes everybody's so busy trying to do whatever it is that they've got on the front of their brain, that at some point it's difficult to stop and take a moment and think about the big picture and get everybody to kind of think about doing something differently. So how do you have the conversation?

Yeah. So, there is approach of day zero, day one, day two. Day zero is when you decide that you need to build a system.

Day one is when you're designing and building this system. The day two is actually once your light are up and your system is running and serving your real customers. And unlike the other days, the zero and one, this day theoretically does not have end.

It is a continuous process. So if you have a continuous process, then what you want to have is another process of continuous optimization, rather it done by human or by automation. The important part to understand that it's not going to be one time operation.

I met many companies who did the, they said, "We don't need a product. " And yes, they achieved some results. Those results are ended within few months because, again, the system became unbalanced.

So the entire conversation of day two and how you operate over time and how you optimize over time, this is the conversation. And then the second conversation is how do we build trust around those tools? So obviously, it's an adoption kind of a question.

You start with non-production environment. For non-production environment, you can easily start with kind of opt out policy. We starting it by default for everyone because it's a development, and worst case scenario, your development workload will fail for some reason if automation does something wrong.

Typically, it shouldn't. And then for production environment, you adopt other policy or say there is an opt-in. So automation can handle your workload, reduce the amount of work that you need to do, but if you don't turn it off, then you receive the chargeback amounts, and we're going to track those.

All right. Well, folks, you heard it here. I think we all know that managing Kubernetes and optimizing it is not easy, but there are platforms and tools out there to help you.

And then the question is, well, how do you get to them? How do you deploy them, and how do you make the justification thereof? Hey, Eli, thanks for being on the show.

Thank you very much. Thanks a lot, Mike. Pleasure talking to you.

And back to you guys in the studio.