AI Hypercomputer Cluster Toolkit with Google Cloud

Good morning all. Uh, today we're going to talk about AI infrastructure and how we help our customers deploy it on Google Cloud's compute engine. And of course, it's an after me is going to, uh, double down on the GKE Land.

My name is, uh, Les Kadi. I'm a senior product manager in AI infrastructure, and I'm super happy to be here. As a matter of fact, today I am, I was supposed to be, uh, on leave, but they told me you guys are gonna be here.

So I took some time off. Um, I really love this topic. I moved from London, uh, to Sunnyvale six months ago with a newborn to basically lead this effort in, in our team.

So everything you see, I consider it, uh, basically my baby as well, right? Mm. Um, first and foremost, uh, I want to describe the problem because it's complex and I don't want to just start giving solutions.

I spent seven years in London, uh, talking with customers in high performance computing and ai, and that taught me a thing or two about first delving into the problems and then using that as your launchpad. So what you can see here is that basically we have a lot of machines, uh, planetary scale, many different families. When we first started, uh, I joined Google, we only had one, the N one standard.

There was nothing standard about it, but that was everything we had, and we kept upgrading and, uh, you know, updating everything. And then it was still the same machine, uh, with different shapes. Look at us today, where we are.

You can see the scale, uh, in every, uh, matter. And one thing that I've seen throughout my career is that every time you have a new GPU or CPU or some kind of a technical, uh, fit, and then everything every year becomes faster and faster and faster, right? And it's true, it does get faster.

But what happens with the rest of the components, this is just one component, right? If, if I believed, uh, the CPU manufacturers, now we will be like, uh, in a b blink of an eye, right? Um, and it gets more complex as well.

It doesn't get just faster. Uh, complex is complexity, is the price to pay for that improved performance. So you can see, uh, our new latest a four VM configuration.

You can see how many, uh, network cards we have there, how many local SSDs, the the GPUs themselves, the baseboard. A lot of these items are custom made by us. Um, impressive performance throughout the stack.

And then you stack them all, uh, on our titanium networking, which basically allows you to have hundreds and thousands of these machines together as a single stack in the same network in non-blocking fashion, which is exactly what our customers need today throughout high performance computing and of course, AI infrastructure, especially for the training routes. Now, this is what, ideally you will want the life to be simple, easy. You got a faster GPU this year versus the last year.

Now it should just be faster Reality is that there are coming problems with it, right? First off, how do you get up and running? How do you basically create a supercomputer in Google Cloud when the equivalent on prem will have taken you months and months of pre-sales work, uh, spinning it up, testing it, doing the acceptance testing, putting everything together.

Now with a click of a button, you have the infrastructure in front of you. How do you handle that? Then you have the performance and certainty, right?

Because you haven't spent this months and months and months and months of testing you people in using this infrastructure. They're seeing it for the first day. Maybe they just dropped 15,000 GPUs in a project, and now you're getting to go, right?

How do you ma manage to make most of that? And then when eventually things go down, um, some customers find it difficult to troubleshoot, and of course there is gonna be some downtime, uh, especially when you're on planetary scale, right? So this is a little bit more of a realistic graph, uh, based on the previous one.

You can see that ideally you want everything to be the dotted line and you finish your training and that's it done, right? But you already start going a little bit on the skewed, uh, side because you have to spend a little bit more time to set your project up. You need to basically, uh, put some optimized VMs.

You want to have your checkpoints, you want to load from those checkpoints, and then you have some failures and plan maintenance along the way that ultimately derail your whole experience. So the total time to market, and of course, the cost increases. So those, these are the problems we want to address, not just the faster GPU, but the whole package entirely.

So for that reason, we created cluster director. Cluster director sits as a foundation. You have a lot of layers there, and a lot of goodness that then you just expose to cluster director for SL and cluster director for GKE specifically.

So you start from the purpose build hardware, everything we do, the gpu, our network, every component that we have is basically that base for us to, to keep expanding upwards the stack, right? Once you do that, then you have the classic VMs, managing status groups, VMs, uh, Kubernetes, GKE, um, and then of course, classic director sits in the foundation of all of these, uh, due to capabilities. Like for example, in a data center, which has thousands of VMs.

If I give you two, how do we make sure that these two sit in the same rack, sit in the same suites, have the least amount of network paths that they need to traverse, basically to communicate with each other, right? And it's a big problem when you are in a big data center. So these are some of the optimizations that we have built in, into the product from the very, very good go.

Of course, the performance and everything, VM images, we'll expand a little bit more on that, right? And then once we finish with the cluster director, either for SL or gk, then you can keep building as a customer on top of that with the optimized software and the specific needs you have. And at the end, uh, I always, um, tell my colleagues, it's almost like a, a steak these days, right?

You have to specify the, the piece of the meat that you want. So what kind of GPU or CPU, what exactly is what you're really looking for, right? And in the case of the gpu, I think it's more like the filet mignon in this case, right?

It's a little bit more air. Uh, but then you also have to specify how do you want it cooked? So in this case, the flexible, uh, consumption is how do you want that, uh, cooked, right?

Do you want it on demand? Do you want it in cad? Do you want it a little bit more flexible with dynamic workload scheduler?

Right? Um, and that makes a a a lot more sense when you actually go to the training because it does have an impact, uh, on the price, but also on the availability of the resources. So, Sorry.

Yeah, of course. Please. Just a quick question, because there's a lot of information coming, if I understand well, with that, uh, cluster director, I can control indeed that I have two VMs with my workload running on there, that I have control that they for the best performance, that they're sitting in the same rack connected to the same switch in the same server using the same GPUs and so on.

Yes. Yes. We, we call that specific feature placement, uh, policy.

Yeah. And we do enable by default, for our customers that are using Cluster Director, okay? And it's not just that we give you an API, of course we do, but also when we create the, uh, the physical infrastructure, we make sure it's a dense deployment from the get go.

Okay? The, the placement API is more that we make sure that there's no defragmentation uh, there's no fragmentation moving forward for the customers, right? Because you can put it at very dense, and then people, uh, or customers in this case, take pieces of it.

So the, the API makes sure that everybody basically, uh, gets as close to, to what they wanted. Uh, okay. Cool.

Uh, thanks. Yeah. Jay, with Nexus Tech, uh, that's seductive, very seductive.

Uh, I, I assume there is a premium, uh, associated with this. Is there a, is there a finops view of this? No, it's not.

Oh, it's not, Uh, it's, it's a simple answer to a very wow loaded question. It could Have been. So it's the price.

Is the price correct? The price is the price. Um, you already pay for the GPUs.

Uh, I think the, the best we can do is make them much fast as they can be in our network and then basically give you the, the max out of it, right? Okay. So, so just to reiterate, there is no premium for the things you've described.

This is it, batteries included, comes with, yes. Okay. Thank you.

You're Welcome. Alright, so the foundations is a little bit like what I said, like the placement policy and a few things on, on top of that, that they have a little bit more to do with the actual hardware and the, you know, as close to bare metal as it gets. Uh, but then of course, on top of that, we have four new categories that we are splitting this problem, right?

So we have the cluster management workloads, observability, and then resiliency for the cluster management. We are going to deep dive today a little bit more because we have the cluster toolkit, which is, uh, as I will describe the way to spin up, uh, on compute engine, AI infrastructure, big clusters for training or high performance computing environments. We're delving a little bit on the workloads, but it's not in, in the scope of the stock.

And of course, observability and resiliency. Um, when things go wrong, you do want to be able to understand what went wrong and especially take some action against it, right? So I have a specific demo, for example, where you will, you know, allegedly running a training run in Google Cloud, one of the nodes went down, nobody knows why.

You know, it happens once in a while. Uh, but then our infrastructure made sure that it will replace that node resubmit your job so you don't lose sleep over it, right? Because if you are on two o'clock in the morning and you submitted a job, then I'm not expecting you to stay awake to resubmit the job.

And what it could have been a three minute downtime. Now, for you, it's as long as you, you know, either if you have kids, you, you wake up a little bit earlier. If you don't, it's a little bit longer downtime, right?

So, um, so what is cluster toolkit? So our objective is to make it easy to have 10 key environments that follow our best practices. In my previous careers, you know, spinning up a cluster took months in Google Cloud, it can take up to maximum 30 minutes, let's say, depending on your choices, right?

So in those 30 minutes, it's very easy to create many clusters throughout the day, throughout your years in Google Cloud. So it's becoming a lot more imperative to spinning up the correct clusters, right? Because now that spin up will be prolonged through the next four years for you, right?

Um, so we want that to work out of the box with the best practices that we have to offer so customers don't replicate the same mistakes over and over. To do that, you have to have a vast amount of services building because the moment that customers don't find this one service that they want, then they go custom all the way, and then it doesn't work for anybody for them in terms of how you move forward. And then for us, so we take the hit in a case, uh, in, in a figure of speed, uh, and expand our services to, to a lot of the components, to cover the whole spectrum of customers and what they want to, to do with it, right in the middle of it, it's cluster toolkit.

Think of it as the orchestrator for all of these environments. A lot of these could be APIs or, uh, we could have them as Terraform scripts, uh, cluster toolkit. It brings everything together into a single deployment.

So you don't have to worry about chicken and the egg. Like a quick example I used to follow is that we used to have S slm, GCP and SLM is one of the workload schedulers that a lot of customers in ai, uh, or HPC they're using throughout, uh, their journey. And then you had also, you know, the DDNX scaler from, uh, which was the last three deployment, right?

Both of them had Terraform scripts. Both of them assumed that there is no VPC and they created A VPC. So you, you could have a cluster, a working cluster with a parallel file system, as long as you didn't really need to bridge those together.

So you had one or the other. Um, who creates the VPC? How is the next part connected?

How do you avoid second in the egg? In this whole, uh, example is basically the reason why cluster toolkit exists today, right? Um, it's a little bit small.

It gets bigger in the big screen though. So this is ideally one quick example of, uh, how you would describe your whole blueprint, your whole AI infrastructure or HPC cluster in one blueprint. And you can see how we have the, the use option, right?

So for example, in here you have the home file system, and then you have file store, and you say, I want to use the network one for that, that I have just described. It's quite quick and easy. Imagine, like if it wasn't in a text, it would be like some drag and drop layers that you can just connect with each other and encapsulate, right?

And that's in essence what we have done here. And then we take the steps out of that to make sure that the features are, uh, there. And then the complexity is not.

So when you're thinking about your journey, you have the blueprint. The blueprint can be like 62 lines or less just based on how picky you want to be and how descriptive you want to be there, right? But then at the end of that, through the whole process that we created, the Terraform Artifacts modules, the deployment, everything, it ends up being a 40,000 or more lines of code in Terraform.

No customer will go in and just do that, right? It's, it's super complex. Then you have a problem there, how do you find it?

And then the new version comes up. You want just to basically enable one new feature or one new vm. You want to able to be able to go to the blueprint and say, now I had a three, now I want four.

So basically you'd change the three to the four, and you know, Sam is your uncle, as they say. So I have a demo. Bob's your uncle.

So Bob, sorry, I'm still near us. I'm learning. But that, that's the only thing you correcting.

I'm gonna have a good, So that makes me think, okay, Kimberly Bates here. When you get to 40,000 lines, how does that get audited? That's a great question.

Mean to me is like, that's scary. Uh, yes, this is efficiency, but as much as HPC and AI is getting audited nowadays, that's, you need a AI machine to audit that sucker. So it's, it's an amazing question.

And, uh, we are doing a lot of steps towards that. We already find customers and CSO officers really loving this product because think the, they Can bury a lot of things in 40,000 lines. Yes.

But here's the thing, they don't audit the 40,000 lines because there is a deterministic way to go from the 62 to 40,000. Okay? Basically, you only want to test the 62,000, the 62 lines.

So for example, in there, if you don't want your users to be able to spin up a cluster in Europe, and you want them to only spin it in us, then basically you can audit that right there in the VPC. Now, the rest of the lines on how the VPC works and what exactly components does it have there in terms of computer speaking to computers, almost at that point, you don't really care about that, how exactly Terraform will spin up and create that VPC. I mean, you care for performance if there is a bug or something, but your network has already been set up to that region.

When you go a step beyond that, and then you include also UI because the text allows you as a blueprint, as a text editor, it allows you to go and do anything you want. So, feedback that we have from the customers, and we are actively, uh, I have a link at the end for your viewers, uh, for our private preview. You want to get that even further and then not enable them to actually change the text from Europe West four to Europe West one, for example, right?

You want to get even that and put it in front of a ui. So the UI will not let you to put something that your administrator has never allowed you to do it. Right.

Now, how it gets after that, basically it propagates, as I said, deterministically way, right? In a deterministic way. So you never get end ending up with something that it wasn't allowed, but you found a loophole and you made it possible, right?

Because in that deploy, you don't have control anymore. You might wanna consider having a, a white hat try to crack this baby. Oh, yeah.

Uh, yeah, Have a, I mean, I would, I would, I mean, I assure that Jack would like to see this too Whole. This whole thing has ignored Jack Poll with Paradigm. This whole thing has completely ignored security.

There hasn't been in any of this, and I've been holding all off talking about that because I didn't wanna rule the flow. But there's a huge security implication with all of this that you're doing, right? It, uh, It's no different than what we've been doing with Chef Puppet.

I mean, these are all abstractions. Yes, exactly. The statement still stands easier, Doesn't make quite it easier, just like Jeff Puppet and Precious Code has, but once the auditors believe that this is a trusted resource, Well, and that's why I was saying is that if you, you get some white hats to go and well white hat Work on like, anything.

Well, And I, I had, I Had a question about the, the 40,000 lines that you can raise. It's what if somebody goes in and maliciously edits that at that point you have a check sum on it to make sure that doesn't happen. So think of this like fundamentally it's terraform at the end and yeah, different components of Terraform, right?

So if they were to go and change that, that means that they also have their necessary permissions in their project to basically be able to execute that. So it's like you first allow them to go and spin up an environment, then you complain that allowed you to, to do it in West four, or not West one, for example, right? Well, you gave them the permission to do it everywhere, in a sense.

So my real question is actually a higher level, not necessarily security related, is we're still, you're, you are using infrastructure as code to create IA clusters as code, right? Is what you're doing. So the question is, while we're in this, or sorry, AI clusters is code, but while we're in the AI era, why are we using code instead of a natural language interface?

Why can't we just say in natural language, this is a connected server here, you know, this is the VPC and natural language and not have to do the code at all? Um, well, depends on, you know, familiarity with people in code. I I consider that close to that.

Like I don't see statements and things like that, right? I consider that a step above coding. Okay.

And a little bit below natural language. Like, you know, the, the idea here is at some point you, you want to say, you know, Hey Emini, hey, Google, create me an AI infrastructure class in Europe was for Yeah. And I don't want to pay anything.

And then it will go and do it, and then nobody will complain about or any, Not natural language, but it's declarative, right? Yeah. Yeah.

Correct. Yeah. Um, the next step above that is because this is a text editor, you want to put it on for a UI to make it easy, click and deploy, and then it shows you the options.

Because right here, if I saw you a persistent disc, for example, and I have this setting as HDD, how do you know that SSD is an option, or hyper disk is an option, right? You want to keep making that simpler and simpler. That's I think the, the whole point of the question.

Yeah. So, so what, what you've shown us so far, I think what I'm seeing is that there's clearly gonna be, you know, you're showing us yaml, we're talking about a domain specific language pertinent to this AI project that we're gonna take on. There is constraints and affordances put in place.

There is the security concern, but at least you're starting to, to to, to the earlier point. You're starting with more of that, uh, infrastructure as code concept of 62 lines is more manageable for humans currently. Correct.

To process and think about. Mm-hmm. But someone has to do the, are we trusting Google to do the diligence on the 40,000 lines?

Yes. Not only Google, but also you as a customer, you know? Okay.

So it is still Responsibility in any case, right? Because ultimately you are the one that presses the comment and you spin up the cluster. We are moving more to a managed service.

We have announced a lot of things at next, uh, which basically we get a lot more of the responsibility there for your lerm cluster and for the deployment. Mm-hmm. And then complete service end to end.

And as A follow up to my prior one, when you said there is no cost premium, I'm assuming there would be a cost premium for a managed service? Uh, not that I know of today. Okay.

But as, as with everything, uh, you know, once Very suc, it's very seductive. Thank you. One follow on question to the, the cost per, um, and depends by the way, uh, is you actually have any way of detecting under sources or under, uh, um, pieces of this component.

So you, you're not charging customers for pieces of the, um, configuration that they're not using. Yes. It's, it's a great question and we do it throughout our journey in Google Cloud from the very, very beginning.

Uh, I remember like when I was in my first year in Google, I was saying like, why do we give, you know, this recommendation that you are spending more than what you should wound. Those actually end up with us getting more income. But yes, only in the short term, in the longer term, you have a customer that because they're using you and you are telling 'em you are overspending, their more competitive.

So the theory behind that is that if they are more competitive, you will end up having more and more business because they have more and more business. So I have, this is really in the weeds, but I think it's an important question. Um, Terraform is not really open source anymore.

Yes. And, and so, and IBM owns it. Right?

So like that has implications for you using it course. Are you looking at open tofu as an alternative? I have nightmares about every day.

Yeah. Um, in the sense of like, you know, Terraform right now, it's working a certain way. What's happening in the next few years, et cetera.

I don't know. But at the same time, you know, they're equivalent technologies. We, in Google Cloud, we have our own APIs.

So open tofu could be one replacement. Okay. Our own APIs could be another.

Um, you know, I think at the end of the day, the customers need this and they will find, or we will find a way to, to give them exactly what they need. Uh, that it, but it keeps you up at next of everybody. I'm having a Little trouble scar.

I'm having a little trouble navigating this because I don't under, can you explain simply there's Cluster Director, there's Cluster Toolkit, and then there's Blueprint. Oh, yeah, absolutely. I don't know what I'm using.

You have, 'cause I can write a script anywhere, but what you have, you know, Here you go. Yeah. You have the hardware.

'cause that, that also relates to the security question seems Speak. Yeah. It's an access question.

So you have your hardware and it's always really easy to, to understand that because you can kind of touch the metal in a sense, right? And then on top of that, you have the APIs from Google Cloud that they give you a VM or a big, in this case the cluster director, the fundamental foundations in a sense are there, and they have the placement topology. All, all of these goodies in your APIs building, right?

Yeah. And then on top of that, you go either the cluster director for sl, which has specific features for Compute Engine or gk, which has the equivalent features for Kubernetes engine. And inside Cluster director sits Cluster Toolkit as well as the engine that it deploys your cluster through the Cluster Director project product.

The blueprint gets you the blueprint, gets the script onto the declarative script that you can load into a cluster director. That toolkit runs The blueprint, is what Cluster Toolkit uses to spin up your cluster, how you ended up creating this blueprint and what features that Blueprint has. It's encapsulated into the bigger project, which is the cluster director and Com and toolkit's.

A component of cluster Toolkit is basically the engine of director. Yes. Okay.

Right. So basically, uh, here's how everything looks together when you start thinking about it architecture wise. Um, here's ideally where we want to move the, the line and, and speed everything throughout spinning all of these problems.

Uh, we have a specific form, uh, that's, that's my call to action. The, the demo is cool. It shows you how something is breaking, then we automatically fix it.

But I just described it in, in a nutshell, right? This is showing you, uh, and allowing your customers, viewers, our customers, and your viewers to basically get an early, uh, access if that problems that we share sound relevant to them.