Introduction to the AI Hypercomputer with Google Cloud

My name is Sean Darrington. I'm product manager here on the storage side at Google. Uh, this is I think my ninth or 10th, uh, field day.

It's nice to see many of the familiar delegates, uh, Kimberly and others. Um, Steven FoST and I actually did the very first one, uh, the inaugural field day way back when on, uh, thin provisioning with, uh, Veritas. And so that was, uh, fun with Veritas and three par.

So this is a little bit different. This is all cloud-based. This is not on-premises.

That's, uh, a different topic. So in the next couple minutes, I did wanna just touch on a few things that we announced last week at next, but we'll be focusing on in a little more detail here as well today. Um, but first before I do that, I wanted to set up what is AI hybrid computer.

And you'll see this slide throughout the day is this is really representative about where we're investing our development resources to make it easier for customers to consume and run their AI workloads in Google Cloud. But it's not all about infrastructure. That's where we're gonna focus most of the attention today.

If you look at the top, uh, the consumption model, right, this is a big aspect about how do you optimize costs specifically around compute resources and accelerator resources. So you'll hear Ilyas talk a little about cluster toolkit and cluster Director, uh, focusing on some of the flexible consumption models. But if you look at the middle portion, the optimized software, this is also where you have the Jackson PyTorch and those, uh, software deployment models you have that'll work, work with or run on GKE or GCE.

So we're working in a bunch of GKE provisioning as well as the middle and in the bottom of the purpose-built hardware, this is where we'll focus on the compute, storage and networking aspects of it. So as we take a look at overall, like, one of the things that we announced from a storage perspective was a new offering called Managed Luster. Uh, Google Cloud Managed Luster, uh, is a brand new offering that is based upon our partnership and working with DDN and is based upon exascale.

Uh, so many of you are familiar with Luster, uh, on premises, uh, been around for many decades. Very scalable parallel file system for AI and ML workloads. It's also for HPC workloads.

But in the context of today, we're gonna be talking more about AI workloads from that angle. The other thing to think about is that we're launching this as really a persistent storage offering that is very highly scalable up to a petabyte in a single file system. Uh, it also has very low latency, consistently sub millisecond latency from both time to first byte as well as ongoing file loading, uh, but then also very high throughput, uh, up to a terabyte a second.

And one of the advantages of the parallel file system, and you'll hear Dan and Ddns actually gonna be joined today as well talking about this, is that the parallel file system is really good for AI workloads that have either a handful of clients where you wanna drive very high bandwidth to a single client. In fact, we can saturate some other VMs of 20 gigabytes per second for a single client, but then also scales to thousands or tens of thousands of clients and hundreds of thousands of GPUs and accelerators or TPUs. And so this actually not only accelerates the training, but also does very fast checkpointing with with this full duplex capability as well as being able to do high performance inferencing.

So the link there that is in the bottom left, uh, that's one of the, uh, design guides that we have for storage. 'cause there's really two different aspects of storage that we're recommending for people today. Never wanted to manage Luster, which we just talked about, but also our cloud storage portfolio with anywhere Cache is another capability that we announced.

So you have two capabilities, object and parallel file system to look at anywhere. Cache is unique in the, in among many of the hyperscalers in that it allows you to keep the data closer to your accelerators. Uh, and one of the things that our cloud storage offering has is, again, industry unique, is a multi-region bucket.

So you can have your data in a multi-region bucket and it's accessible anywhere in the given continent, but it's a single bucket and single namespace. The challenge is to figure out where you can have compute and accelerator resources and how do you actually improve the performance of your AI workloads in those regions. And this is exactly what anywhere cache does.

It allows you to simply enable on an existing bucket in a single command line anywhere cache, which can cache up to a petabyte of capacity in a given zone within a given region. So if I'm in Oregon and I know my accelerators in zone B, I can turn on the cache in Zone B and that's gonna cache that data closer. New Hill Marco talked more about this in some of the benefits of cache hits upwards of 99% to accelerate training needs.

But it's not only that capacity, it delivers very high bandwidth to that particular zone of your selection up to two point a half terabytes per second. And it's not limited to just one zone. As I have indicated here on this slide.

You can enable it on any zone you want within any given continent. Um, the other thing that this allows you to do is to keep and preserve your metadata and the existing bucket structure that you have. So it doesn't require changes to your applications.

I mentioned you can enable it with one command line. And that's, again, fairly unique in the industry here. We also have, as part of this is works with cloud storage fuse cloud storage fuse enables you to mount a object storage bucket as if it was a file system, right?

And so the link there that you see in the bottom of the slide is a reference architecture talking about using GCS fuse, some of the design alternatives where and how you enable anywhere cache and some of the other things that go along with deploying the object storage for AI workloads. So we talked about object storage with anywhere cache parallel file system with managed luster. Those are both zonal solutions.

One for persistent being managed luster. A caching solution is a zal caching solution with anywhere cache. And then rapid storage is a new zonal bucket option.

So this is for primary storage read write data. It's a new bucket selection type that customers can deploy, uh, in addition to our region, multi region, dual region bucket choices. This also is integrated with GCS fuse.

So again, you can mount a zonal bucket as if it was a file system and have that very high performance, very low latency and keep the data close to the accelerators. Now, rapid storage also has a benefit of delivering very high QPS up to 20 million QPS per bucket that allows you to ramp the training very quickly. And again, you'll hear Marco talk a little bit about this in terms of the implications of this.

But the other thing that it gives you is very high throughput, uh, in fact up to six terabytes per second for a given bucket. Now with cloud storage, you wanna be able to scale dozens or hundreds of clients as opposed to a single VM performance. And that's one of the ways in where cloud storage really excels to deliver that very high bandwidth to thousands or tens of thousands and nearly infinite capacity scale.

So we talked about managed luster being a petabyte if you need to go to exabyte or multiple exabytes, uh, not an issue, right? So this is a very different structure, very different thinking between object and PFS, uh, for your training needs. The other thing I just wanted to touch briefly on are the compute and networking sides.

And I'm not gonna steal everybody's thunder 'cause Dennis and others are gonna be coming later today. But we announced a number of things around our GPU portfolio and our partnership with nvidia. Um, we've been on the forefront of this and I mentioned saturating the performance of the A three VMs at 20 gigabytes per second.

But we announced a number of a four and a four ultra machines as well as others are gonna be coming later this year. Uh, we're the first hyperscaler to deploy some of these GPUs and make them commercially available to customers, which is, uh, which is great for our customers that are demanding the highest performance capabilities for some of the GPUs. But we also have our seventh generation TPU, uh, Ironwood.

Uh, this is extremely high performance. Uh, this is, uh, mentioned the seventh generation, but it has five times the peak performance, uh, and more memory than prior, uh, versions. We've designed this really from the ground up to be in an entire cluster.

Uh, and so Rose is gonna talk a little later today on Ironwood and go into more detail about the architecture here. But this is giving you the ability to deploy an entire cluster upwards of over 9,200 chips in a single cluster. And you can see the number there, 42 and a half exo flops of compute capacity, which is crazy, okay?

I mean, it's, it's orders of magnitude more than others. It's more than what most customers are going to use. But we also have very high demanding customers that are at the top echelon of doing their AI workloads.

And they're always pushing the forefront and they always need more compute, more storage, more networking capabilities. And so on the networking side, we want to continue to make it easy for customers to leverage the network running in Google Cloud. And this is one of the things that's an advantage for us overall in that we actually own our entire network infrastructure.

We have thousands, I I don't remember the exact number of miles that we have for undersea cables, but we continue to expand as we expand the number of regions now to like 41. We continue to expand the network opportunity for customers. Cloud WAN is a new way that customers are gonna be able to leverage not only their access within Google Cloud, but also their own private access within their company.

And this is a fully managed service that we're providing that allows greater capability of improving, for improving performance up to 40%. Uh, so you'll hear a couple of different people later today talking about Clan Cloud WAN in depth that really leverages our cloud, cross cloud network and interconnect capabilities, our undersea cables, the entire network infrastructure that Google knows and owns to deliver consistent latency, uh, for both AI workloads but also for other traditional enterprise workloads. The other thing that I'll touch on briefly is GKE infra.

Uh, we talked about the compute side of things. This is one of the things that we're doing specifically around networking and GKE to improve AI training. If you look at that top diagram, you know, the typical round robin network distribution for layer two three, uh, access, you can wind up with hotspots, especially as you start to look at large scale clusters and large scale deployments.

One of the things that we've done with the inference gateway is really make it available to be able to have a more intelligent way to route traffic to the available clusters as needed. And you can see that we can actually increase the throughput by up to 40% is reducing those tail latencies. And one of the things you'll hear later on and throughout the day is a lot of things we do to reduce tail latencies, whether it be for cloud storage or networking.

'cause it's really that P 95, P 99 where you want to optimize that good put and that accelerator utilization as much as possible. If you can reduce tail latencies, that's gonna deliver a better value. 'cause if you look at that overall cost of ownership, the accelerators are a significant portion of this.

And so you wanna maximize that throughout your entire workload, uh, process. So with that brief introduction, uh, I'd like to again thank you everybody for that. Any, uh, I know there's a lot Yes.

Uh, but I'm just really setting a stage for the deeper dive for rest of today. Um, we do have a couple minutes for questions, otherwise I will ask module to come up. You mentioned seventh generation.

Yes. How would you compare that to first generation? Much better.

Could you quantify that? I'm gonna, I I'll, I will save that. Uh, question for Rose.

Uh, she'll be able to give you exactly how much, but I know, uh, I don't want to, 42 and a half exo flops in a cluster is, is a large number. Pretty impressive. Um, and I, I'm trying to remember off the top of my head the largest supercomputer cluster today, but I know it's a lot less than 42 and a half.

Um, but I will leave that question. Say that for her. For Rose.

Thank you. You tell me more about the doubled efficiency and the asterisk next to it on your slide. Uh, On the doubled it's two times more efficient, energy efficient That, so that has a lot to do with our liquid cooling as well as the power consumption.

So again, I'll say that for Rose to go in a little more detail. Okay. But I will tell you that one of the things that we've invested heavily both in the cluster design as well as just our overall data center design is our liquid cooling capabilities for this, because it doesn't apply just to the Ironwood.

Yeah. Uh, the liquid cooling can be used for, we use that already for other prior generations. Um, which as you point out is very important to, uh, minimize energy consumption.

Yeah.