Google Kubernetes Engine and AI Hypercomputer with Google Cloud

I'm Sean Sharma, group product manager in Google, Kubernetes engine team. And I would like to speak to you about, um, GKE and AI Hyper Computer. I have quite a few different topics to cover, but again, let's have an conversation.

I wanna keep this interactive. I do have a few demos that I'll, uh, talk through. One of those would be a live demo.

Uh, so let's get right into it. So you all are familiar with Google Kubernetes Engine. Um, I had spoken to you last year about it, so we have made some, uh, uh, improvements since the last time and launched quite a few features, um, at Google Cloud.

Next, I wanna talk about the value proposition first, and this is sort of the pattern in which I will go through, uh, covering a variety of different material, uh, different content. The first part is industry leading infrastructure. Now, uh, my, um, my colleague, uh, Elias previously presented a little bit about that, and you will hear more about it later, so I won't go too deep into it.

But I want to talk a little bit about GPUs, tus, and the foundation of running AI workloads on Google Kubernetes Engine. Second thing I'm gonna talk about is training quickly at mega scale. Um, here we talk about the, um, the, the largest Kubernetes, uh, cluster scale.

And, uh, we talk about how we can efficiently share resources using things like q uh, number three is where we're gonna talk about serving and, um, with a focus on lower cost and latency and forced ease of use. Um, we'll then talk about, um, accessing GPUs and TPUs in an economic manner, because that, again, comes down to total cost of ownership, which is very important to everyone. And, uh, the last thing is faster time to value.

How can we make it easier for you to do all of this quicker? So, uh, that's where we're gonna wrap up. So with, with that, let's start with the industry leading infrastructure.

My colleagues will cover this in more detail. All I want to say is this is the portfolio of accelerators, GPUs, TPUs. Of course, there's CPUs alongside this, if that suits the needs of your workload, especially for serving, but for training and serving really large models, you get all of those accelerator, all of the ones here and any future ones on GKE on day one.

So we are committed, uh, we work internally to make sure that whatever accelerator Google Cloud announces is available on GKE on the first day, so you can get the best of it throughout. You've seen this in, uh, previous, uh, presentation. Um, want to talk about Cluster Director for GKE.

So just as a review, uh, AI Hyper Computer is the entire stack. It's a reference architecture. Uh, it's a, um, a super system that's really distilled from a decade of Google working on AI and the expertise in ai.

In fact, vertex AI runs on AI hyper computer. So this is the same stack that we are internally using. Um, now when we get into cluster director, the real thing is models are getting bigger and bigger.

Previously, um, they were, they were, they were doubling every, every year. Now they're doubling every six months, right? And, um, and so when you think about that, the needs are higher and higher, uh, in terms of what the infrastructure can do, your customers are running, deploying, and managing larger and larger clusters of accelerators with compute, storage, networking, all of that put together.

And all of these need to work as a single unit, and that's really where cluster director for GK comes into play. And, uh, let me talk a bit about what's unique about this. So for Cluster Director, for GKE, it lets you deploy scale and manage AI optimized GK clusters, where physically co-located accelerators work together as a single unit, again, giving you high performance and ultra low latency.

Now, one of the things about cluster director for GKE is that you can orchestrate all of these new capabilities that I'm gonna talk about using standard Kubernetes APIs and the ecosystem that's there. There's no new platform, it's just the new capabilities on really the platform that you already know and love, which is GKE under the hook Kubernetes, right? So there's nothing new that you need to learn.

It's all KU Cuttle, it's all Kubernetes, it's all container workloads, right? So how do we, what are the benefits here? And that's really what this slide is about, which is, um, what does cluster director provide from a GK perspective?

So let's start the six things. We'll get through all of them. First thing is high performance.

So this is really where you can run densely, uh, co-located, uh, accelerators as a single unit. Now, these accelerators in the foundation in the lower layer are come together, as we talked about in a prev in the previous session. Uh, you have machines that are physically co-located right next to each other.

So either they're on the same rack and they're sharing NV link domain, uh, sharing an n uh, connected by NV link, or they are across different lacks racks. And then you're able to reduce the number of network hops that you need to do, thereby improving the overall efficiency of the system. Now, when you talk, the second thing is mega scale.

So this is about, and I'll get into this in more detail. It's about running mega scale training jobs with the confidence that GKE, the GKE control plane is not gonna break a sweat. So GKE supports 65,000 nodes, uh, which is a largest scale among managed Kubernetes providers.

Third is topology aware scheduling. Yes, you have all of these accelerators in the data center physically co-located close to each other, but they're not occupying the same space. That's just impossible, right?

They are a little bit apart. So it's good to know what is here, what is next to it, what is one removed from it, what is beyond. So you need to know a bit about the topology, even though we have compactly placed them.

What we do through GKE is we, and I'll show you in a little bit, we expose that information, uh, to our users so that they can use that information to schedule workloads in the most efficient way. And I'll get, I'll show a demo of that in just a little bit. Ease of f.

Now, setting all of this up is complicated. So we, uh, provide you with a variety of options depending on what you're trying to do and your skillset to be able to use all of this. And we'll talk about that.

Um, 360 degree observability, day two operations, you're running all of these. You want to know why is my job not the most efficient, or, Hey, it's efficient, that's cool, let me look at it. Right?

Those are things we provide you out of the box, observability, dashboards to let you do that. And finally, resiliency, we let you, if you, you have stragglers in your, if you have bad nodes in, uh, or in your deployment, uh, we let you report right from GKE, uh, report, faulty node place with spare capacity if that's available. Hi, ung from, uh, question about, just to be sure, uh, co-location for me is a data center that I can choose, like s or something like that.

But for you, co-location means that, let's say your data center, but I can choose which GPUs are in which rack. And so for the best performance, Um, I'll explain what co-location from our perspective is, right? Yeah.

So you are already choosing which region you wanna run with it. Yeah. Now, of course, you want all of this to be compact PLA placed, when we say it's really a statement of compact placement, so within a data center, uh, if you have accelerators on a rack here and accelerators on the rack on the other side of the data center, that's like a couple network hubs go through the data center network and get to that.

You're sharing traffic with a bunch of other things as well. We don't want that. What we want is this one to be right next to it.

And That's what I, so your data center? Yes. Okay.

But makes sense. That's clear. Yeah.

Then the second one, 65,000 nodes Yes. For a GKE cluster. Yes.

Uh, a note, is that a bare metal server or is it still a VM somewhere? Or I can pick a bare metal server as well. Uh, GK note is a vm.

Yeah, so vm it's a vm, and a VM can have multiple accelerators, depends on the machine shape that you choose. For example, if you're choosing a, uh, H 200 GPU or B 200 in the eight, uh, GPU configuration mm-hmm. Each VM is gonna have eight accelerators in it, uh, B 200 or H 200 accelerators.

And, um, that is one GK node. Yeah. So that would be the expanse of it.

So in this case it would be 65,000 times, uh, eight, which is 500,000 accelerators. If you were able to find a data center with that many altogether, that would work. Yes.

Okay. So upper Brown, Hi, Mitch with ura. Um, you were talking about the size of large language models Yes.

And getting much, much bigger. Um, some folks are kinda using VLLM to make it easier to integrate with models, or is that something you're doing or are you taking a different approach? Uh, yes, we are.

And I'll actually show you a demo right on that subject. So, uh, hold that thought and we can, uh, we can dive forward. That's awesome.

So, uh, I mentioned about, um, to, to, to the question that you just asked, right about where are these located? So I've actually visually demonstrated that you have a reservation, you reserved a bunch of capacity because you want run a training job. Now, this capacity, uh, and here what you're seeing in the cartoon I have here, animated gif is, uh, each of those little tiny boxes in blue, those are, let's say those are, um, racks, right?

And each rack has a, a bunch of accelerators or trays of that. And these are all, now this is not the scale, this is just for visual, but let's say there's a block of capacity with multiple racks that are, that's in a data center. You are running a really large training job.

So you have four blocks of capacity. All of these blocks are connected to each other. To reduce a network hops, when you go from one block to to another, uh, you might be going through the data center network, which would have other traffic on it.

So ideally you wanna stay within a block. And most jobs, most training jobs for our customers fit within a block. Now you, uh, you have, you have a lot of training jobs that you wanna run.

You have one hero, you have a couple other ones you want to target block three because, uh, other blocks have some other jobs running on it, maybe some serving jobs, et cetera. If you got a reservation, how do you tell GKEI want to create a no pool targeting block. Number three, we have enabled that right through our, um, our note pool create, uh, API.

So it's very simple. Create a dense reservation. You can query the topology information.

How do we expose it? Um, uh, you can, uh, query the compute reservations. API you get sort of the block information for a particular reservation.

It comes in the format, reservation name, block, block name, and then you create a GKE note pool on this block. How do you do that? Simply say, reservation affinity is specific.

I want to target a specific reservation. And then you just pass that whole string reservation name and the block name and you will create your GKE note pool on that particular block. In this case it would be block three.

Um, and of course the rest of those are your same parameters that you use already when creating a GK no pool. So again, we have worked hard to make sure that you're not having to leave the surface of G-K-A-P-I as we integrate the cluster director for GKE features. Uh, next feature that I had mentioned was, uh, viewing the topology information.

Now within that block, of course, there are many racks you wanna know, which is on rack one and which is on rack 24. And you wanna be able to run your job in a way so that you schedule it in a manner that, uh, you are not, uh, you're reducing when you do your all reduce operation, you're re you're doing it very efficiently. So how do you do that?

We have exposed the underlying, so previously this was difficult to get. You would have to figure out which GK node you have. You would figure out which VM you're running with that node.

You get the VM id, you go and query, uh, GCE or talk to your account team. And then you get a list and then you look at it and then you run some Python scripts. It was complicated.

What we have done is we have, we have exposed this information as node labels, Kubernetes node labels within GKE. So every GK node that you create within this sort of dense reservation will have layers of topology information. io.

Host name, which is just your node name. So that's the vm, that's the node bernet node. The rest are new.

So we have host, which is the vm, which might again, um, might have multiple GPUs within it, uh, eight GPUs for example, for the, uh, AB 200 or H 200 machine shape sub block is actually a rack. So how many are put in a single rack. Um, and then, uh, you have your reserve block.

So within the block they're connected with the top of the rack switches and, and, and you can look at that topology. All of this is on every node within, uh, your cluster that's created, uh, in a dense reservation. So, um, now the key thing is because these are node labels, you can use Kubernetes concepts to go and do things with it, right?

Simple Kubernetes concept for, uh, I want to node affinity, mt. Affinity, I wanna schedule here or not there. But what we do is one step better what you can do.

And um, this is an example of, uh, a deployment that I, that, that I had, which talks about the block, which is sort of, sort of that first column there. They're all the same. They're within a block.

I use block targeting the sub blocks. So they're split across, uh, and I'm squinting here, the split across three racks, I think. And then the hosts are all gonna be different because those are different VMs.

So this lets me do things with this. And if you use Q, which is our scheduler, you can set preferred topology and required topology here. So while you can set within Q and we have spoken about that previously, so I'm not gonna go into too much detail about Q with a K, uh, you can set your preferred topology so you can prioritize within a particular given topology level, or you can do required topology.

For example, I wanna schedule this entire job. It's a small job on a rack, so it'll look in your entire no pool. It'll look for where is a rack that's empty that can take these many pods and schedule it on there, put one pod per note so it needs that many nodes that are free.

It'll schedule it there, but if it cannot, it will just wait. It will prioritize another job for preferred. It's like, I would like this, but if it doesn't work, do the best effort you can.

So all of this runs on a greedy algorithm, so it'll try and schedule it in a, in a way that makes sense. And it's all open, openly available Open course. Okay.

Um, RA and um, that's a good question. Are you implementing some kind of your lease on, uh, GKE or you know, uh, implementing some kind of custom commands that you can issue on, um, Kubernetes For Q? Yeah.

Um, so for Q you would have to install Q and then, which is open source. And then within Q when you write your manifest for that, uh, these are parameters that would go in there and then they would interact with the GK node labels automatically. So all you have to specify, among other things, you know, there are other things in there you'd have to specify which, what, uh, which of these you would want to do.

Okay, Gotcha. So thank you. Okay.

Uh, next thing I spoke about is maintenance is important. Now, disruption handling, right? So when you're running a training job, all of a sudden the host, by host, I mean the vm, the underlying hardware goes into maintenance, went into maintenance, um, and, and now your job is disrupted.

So what can you do about it? What we have done is we have exposed upcoming maintenance that's scheduled through node labels in GKE. Now having that again available as node labels in GKE lets you run node, affinity node anti, uh, anti affinity, et cetera to schedule your job.

So here are some of the times, this is Unix epoch time, this is when the maintenance is scheduled. Let's say my maintenance is scheduled three weeks from now, or two weeks from now. My job is three weeks long, I can't run that job.

I would rather run it on nodes that have maintenance scheduled four weeks from now. I can do that. Uh, examples are affinity, anti affinity.

You can avoid completely any, uh, nodes that have, uh, may host maintenance schedule or you can schedule around that using, using logic from that particular way. The other thing is you can also trigger maintenance not through some fancy APIs. All you need to do is supply a node label, perform maintenance equals true, and GKE will take care of it.

We will coordin your node. We are eVic your pods. Uh, we will respect, uh, respect pod disruption budget, we'll call the underlying GCAP, I will run the maintenance, and then your node is gonna be back up and running.

So, uh, it makes it very simple for you to, uh, manage your nodes at the G-K-E-A-P-I level. So, Sean, on these node, okay, so I'm thinking you're talking about rack level. Mm-hmm.

You know, physical topology for GKE nodes is a customer coming in and then contracting with you a set number of racks that they're doing. So you guys are almost playing like a colo kind of thing situation here, or how is that working in terms of a contract situation with you all? Because somewhere along the lines, I, I'm like a managing the looking, looking at the logistics of this, because you guys are all about optimizing that entire cloud.

And this is clearly you can have stranded nodes sitting someplace else. So how is that working? So all you need to give us is how many nodes do you need of that particular GPU?

So how many GPUs do you need? Right? So the typical request would come in, I need 512 accelerators, GPUs, um, in the, um, uh, a four, uh, high GPU eight G configuration, which is a B 200, uh, in a ag uh, configuration, I need 512 of those.

Give that to me. What we will do is we will, and give that to me in a dense placed manner, which is through cluster. Mm-hmm.

Some of the stuff we discussed, we will give that to you. We will optimize that for you. You don't have to worry about rac, you don't have to do any of that.

All you have to do if you wanna schedule a workload is go here and say, I want to optimize all of this and my job is going to use a hundred of those. So within the 512 that I got, I figure it out for me. And Q will do that for you.

So, and then let's take this the next step, which is the screw this, the data does right? And I'm gonna want my data close to those GPUs. What are you guys doing to ha handle that?

Because that network copying there really can luck up the training. Um, let's, so that's a little more complicated because there are different levels of data where you would have, like, uh, let's say you're doing a checkpoint, you would have different layers of where you want to checkpoint. So, uh, let's discuss that a bit more because, uh, there of course with the vm, there's uh, there's storage that's available next to you.

There are SSDs there, But if I'm using Illustrative pipeline into you, right? Right. It's a different game.

We're, We're gonna Cover that at the next, You'll have to hold your thought there. So, great framework Six five. Um, as part of that topology selection, is that including network connectivity?

So making sure that those nodes have full Yeah, yeah, that's the whole, yes. So if you are within a, with within vm, VM two within rack and then all of those, you're getting the full bandwidth, especially with our H two hundreds, the A three ultras and the B two hundreds. 2 terabytes per second.

Uh, so, uh, that's, uh, that's one of the changes that we made. We are using CX sevens underneath, uh, and the NVIDIA reference architecture. So that all helps.

And how large does that full bandwidth go up to a block? Um, that would be a great question to ask Dennis. I think he will cover that later today in the afternoon session.

'cause that's more on the infrastructure side. Um, I'm running a little short on time, so, uh, what I'll do is I'll quickly cover this one, which is, you can do the same thing with, if you have identified a faulty host as a GKE node, uh, you can just put the fault behavior as a reason and it will again, do everything called in the node, affect the pods, call the GC API and then do all of that for you. And if there's capacity available, replace it.

So that's really powerful. Um, 60 5K notes. We launched this at CubeCon, uh, last year.

Um, and, and this was basically if you have a large cluster, be rest assured we will, we will be able to run it. And Andro, uh, has, has given us a testimonial on that and so's character character ai. I wanna give you a quick demo about two things.

So this is the GK inference gateway. Uh, what this does is it's, it's really helps, uh, so there's high variability of LLM inference responses, uh, where, um, load balancing, traditional load balancing techniques like round, round robin just don't do justice, right? So what we have, um, launched, uh, is GK inference gateway, which routes request based on LLM model server metrics like KB cache, et cetera, um, and, and, and Q line.

So let me give you some visuals. So what we have here is six replicas of VLLM model server, uh, serving an LLM, right? We ran a load test on this using shared GPT publicly available dataset on the, on the, on the, what you see in the chart here, uh, the top is the KV cache utilization.

What you see is that some models are, uh, servers are saturated while some are underutilized. So there is a bit of, uh, distribution there. What you see at the bottom is sort of when the model servers become saturated, the incoming requests, and that's the average queue size, those start to get cute.

So there's that big spike up that happens when model servers are starting to hit that top rail and get saturated. So what happens when, and then when incoming requests get queued, the LLM responses are much slower. So vicious cycle that starts to happen.

The chart at the bottom is the time to, uh, first token latency. So now with GK inference gateway, which is on the, on the right side of the chart, um, what you can see is we have reduced that variability quite a bit. Uh, so everything is running very tight.

And then what happens to your QS A and the that, um, that q size at the bottom very flat, you don't get those sort of spikes and queuing that happens when you're starting to saturate, uh, your utilization. So that's the general, um, general concept behind this. And this is real data.

So it's not, uh, this is something that you will observe if you run this. Uh, and then, uh, when incoming requests never queue, ultimately what matters is LLM responses are much faster. So your time to first token latency is, is, is much, much better.

So, um, so, so, so that's, uh, so customers like that. One more thing I wanna show you is GK inference. Quick start.

And for that I will just very quickly go over to my other screen because I think this is really cool and I really want to show this to you even though I might be running out time. com, go under AI ml. Now, let's say you wanna deploy a model and uh, you don't know, you have to figure out the underlying infrastructure.

You might traditionally run benchmarks and all of that. Well, we have done the work for you and we have come up with the optimized infrastructure for these models. So let's hit deploy model.

Now some of these you will see are say the word optimizer labeled with optimize. Now one of my favorite models here is Gemma two for for reasons, uh, because from Google, um, it has a bit of information on it. You can read through it more like a model card.

I hit deploy, nothing's gonna happen. You'll be able to configure. Wanna use it with Google Kubernetes engine?

I'm gonna say Gemma three demo as my deployment name. Now this is the cool part. So look at this.

Uh, we have recommended by Google, I dunno, are you able to all see it? It's big enough. Yeah.

Yeah. Okay. So it's recommended by Google and uh, it tells me Nvidia L four is what I'm looking.

Okay, so that's the GEMMA two, two B, uh, instruction tuned. Now let me go to the GEMMA 2 27 B. And if you notice it changed it to Nvidia H 180 gb, right?

So we have run benchmarks and come up with what's the best, uh, type of accelerator, what's the best configuration you can tune, um, your, um, your, your maximum latency. And then essentially add in your hugging face access token, which I put in here. And then I have an autopilot cluster that I'll run it on and I'll just hit deploy.

But before I hit deploy, I want to show you something. You can hit view YAML and you can just view the entire yaml, make any other changes you want to, so it's not just a UI thing. You can copy this, put it into your favorite editor, and then deploy it on your autopilot cluster or just hit deploy.

And we will start provisioning this for you on the existing autopilot cluster. So it'll start to generate that entire thing. So it makes it easier to run all of this based on the underlying infrastructure.

Um, uh, other things that I just want to, uh, talk about is, um, this is the compute classes aspect of it. So let's say you're serving, you wanna use a spot, you want to use a variety of different machines. You're optimizing for cost.

You can set, I wanna use a one hundreds with spot, but I want, or with reservation. If that's not available, run it with any reservation or fall back on spot, et cetera. Um, and, and that's their DWS dynamic Workload schedule is another way to, uh, to run your workloads.

Um, we have smaller machine shapes available. And then these are the last thing that I'll talk about, which is, um, different ways of creating your clusters or deploying them. We have cluster toolkit, which, uh, we discussed at length previously and the previous session.

You can create GK clusters with it. If you're looking for a Python like interface where you don't want to look at anything Kubernetes like, uh, XPK, something to check out. And then we also have a UI application.

Um, the UI application is again available. You can go through it, uh, and create again on the AI ml page that I showed. You can deploy that.

Um, and I have a video here, but, uh, we can view that later. I'll share the slides with you. Um, and these are the dashboard metrics out of the box.

D-C-G-M-T, PU and GPU. And with that, here are some key links to look at, uh, to learn more.