Overview of Cloud Storage Storage for AI, Lustre, GCSFuse, and Anywhere cache with Google Cloud

Hi, everyone. Uh, my name is Marco Abella. I am a product manager in Google Cloud Storage, which is our object offering.

Uh, and I specifically focus on storage for, um, A IML. So we're gonna kick this off with kind of like a, a brief introduction and overview, um, of our different storage solutions, where we've been, what we've been working, uh, very hard on over the past, uh, past year. Uh, then I'm gonna get into our cloud storage fuse offering.

I'll talk about anywhere cash, which we just, uh, GAed, uh, and announce the GA at next. Uh, and then Dan, my colleague, is gonna be coming, uh, up after and talking about our new managed luster solution. So, um, to kick it off, we know that for A IML, uh, a lot of folks, the main focus is typically, uh, compute, uh, GPUs, tus, which is awesome.

Uh, but you gotta keep those GPUs and tpu, uh, well fed. And so storage is a, uh, critical key piece of the ai, uh, pipeline. And, um, choosing an improper storage solution can really, uh, negatively affect your gpu, uh, utilization.

Uh, if storage is a bottleneck, your GPUs and TPUs are just waiting there, sitting idle, uh, waiting for data to come in before they can, um, begin to, to process them. And this is throughout the entire pipeline from your initial data preparation, which is where you're taking your raw data, cleaning it, and then passing it down to training where you are going ahead and, um, iterating on those samples. Then you're generating a model, you're loading that model from inference, and then depending on what your inference is, let's say it's gen AI, content, media, pictures, video, you're then serving those, um, uh, that content back out.

And so throughout that entire pipeline, uh, storage is absolutely critical. And, um, the two, kind of like the, the industry optimized storage that's really used for A IML is, is there's object storage, and then there's parallel file system. So on the object storage on the Google side, we have our, uh, cloud storage, which provides, uh, persistent read and write storage with virtually unlimited capacity.

Um, and providing very, very high aggregate throughput. Our, uh, parallel file system, our new managed luster offering that Dan is gonna be, uh, covering more, provides persistent storage, uh, capacity in the terabytes to petabytes, uh, but really it focuses on ultra low, um, latency to really drive down a specific workload, uh, profile. And so what are the typical storage requirements that we see for, um, A IML, which are, uh, can be similar to other workloads, but also different from other workloads.

But in general, um, what they really mean is lots and lots of capacity. So from your, uh, total, uh, storage corpus to then the actual amount of data that you are either, uh, training on or the models that you are, um, loading, you're, you can need anywhere from terabytes to exabytes, um, of data from a, uh, uh, performance standpoint, from bandwidth perspective, very, very high. Um, aggregate, um, throughput.

So we're talking about terabytes per second of throughput for both reads, um, and for writes for checkpointing. And then also millions of of requests per second. So what we refer to as QPS and the traditional storage world, also known as iops, um, uh, meeting millions and millions of, of these requests, uh, while serving that, um, that high bandwidth, uh, low latency for reads.

And then last, also something that affects all this is also the nature of this workload, whether it is, you know, uh, constant or spiky and, and bursty, depending on the different stages in, in the pipeline stuff like checkpoint restores are, bursty model loads are, are very bursty, but basically just depending on the different pipeline, they can have all of these different storage requirements, a subset of them. But these are typically what we see in the, in the A IM ml, uh, pipeline. And to give a very focused example of that, um, you know, checkpointing and serving, I can give you a pretty consistent, very predictable definition of what the storage requirements for.

Those are kind of the most difficult one to have a consistent definition on. And I've been working with hundreds of customers over the past few years on, on getting their storage right for their A IML pipeline is really training, uh, where there is the most variance in terms of, of, uh, IO pattern. So, to, just to give a a an example here, right?

The top top example here is you are basically just reading something sequentially, whether it is, uh, each, uh, file or object is an individual example, or you have a, uh, structured file format like Parquet or TF record as many samples in it, and you're just reading that, um, end to end, you can have very nice, uh, throughput, very nice, uh, performance from object storage and just reading that sequentially. Then you can also have, um, you know, again, using these now structured file formats that have multiple samples within the same file, like parquet or, or TF records. You can also have where each node gets a file, there are, let's say 110 gigabyte size files.

And within that, each node begins randomly sampling within each, uh, within each object or file. And then at the end of an epoch, those all get shuffled back out and sent to different nodes. Then you have fully random, which is, uh, at a certain training step.

They are reading across files to get a sample and then load it into each node. And then you have multimodal, which you are doing a little bit of everything. You are doing metadata lookups, uh, full object reads, random lookups in databases.

Stuff can get really, really complicated with multimodal, where you need both, uh, ultra low latency and, uh, very high throughput. But basically, this just shows you the di what I talked about before, the different performance aspects. It shows you in training how different io uh, different training profiles can kind of have all of those at the same time, some of them, uh, a sub or a subset of them.

So what we're gonna be talking through the rest of this presentation over this past year, we have been, uh, like, we've been working extremely hard on, uh, imp uh, basically solidifying our storage for AI ml, um, portfolio. Many, many new announcements that we had at next this year. The key things that we're gonna be talking about, uh, throughout the rest of this next hour is, uh, cloud storage fuse, uh, and the new, uh, hit enhancements there anywhere, cache, uh, rapid storage when JUUL already covered, uh, storage intelligence.

And then, uh, Dan is gonna be covering, uh, managed luster. So let's start with cloud storage fuse. So, um, cloud storage fuse, for those of you that are not familiar with it, very simply put, it is a way to mount a bucket as a local file system, and you can interact with your buckets as files using very basic file system on semantics.

Um, we invested very heavily on this over the past, I wanna say about three years. And boy, is it a bet that really paid off. Um, the, the reason why we, we see what we were seeing before was customers were, had this data in object storage, and then they would just copy it down locally for two reasons.

One was just semantics. Their application just needed, uh, knew how to talk to a file system and they didn't wanna rewrite to use object storage, or they simply couldn't even do that, or for performance, bringing it in locally, but get from the, uh, the performance of local SSD. But the problem with that, what we were hearing from customers is having to deal with that.

Um, managing the data transfer in and out was just a complete nightmare, especially as, um, the, the storage profile, um, was, was growing. And so that's where we built, um, cloud storage fuse. And most importantly, we also, one of the very, um, like the, the, the main value adds also of cloud storage fuse is that while originally we were intending it as for following these different use case of, again, you want the portability, now, it is kind of our high, uh, performance client of choice because of specific performance optimizations that we built, um, natively into the client, including our file cache, our parallel download feature, and, um, uh, streaming rights and hierarch, cool namespace bucket integration, which I'll cover, uh, slightly more in, in Dale.

Uh, but what I mentioned before is like, boy, is this a bet that really paid out? And, um, one great indicator of this is we also have a, uh, a managed CSI driver. So for the, um, GKE, the Kubernetes, the people that are familiar with Kubernetes and want to use familiar Kubernetes, API, we have a CSI driver, uh, that basically, uh, deploys GCs fuse as a sidecar that you don't have to really manage or do anything about.

It's just this, uh, managed offering. And we have just seen explosive growth on this with some of the largest, uh, AI ML customers on in the world using, um, this solution. So I think last year at AI Infra day, uh, we had just announced, um, the cloud storage fuse file cache.

Um, so not gonna spend that much more time here, uh, but to, the way I'm breaking this down is what are we doing for training? What are we doing for serving? What are we doing for checkpoint?

So for training, uh, the cloud storage, uh, the, um, the file cache for cloud storage fuse is, is what it really helps. And Toyota was a great example of this. They presented, uh, with us on stage last year.

What they saw was a 33% faster second epoch time, 20% faster total training time. And as you can see clearly from this chart, they also saw they got a cost reduction from those, um, those, uh, cash hits, then just being served locally and not going back, um, to GCS. The, uh, the our GPU machine times our GPU machine types come with, uh, I think it's up to six terabytes of local SSD currently, maybe even higher 12 terabytes now.

Um, and so that is a perfect use case for this file cache, where you simply just point it to a local directory, which is backed by local SSD. You can begin to use that. You can use, um, a hyper disk volume, or you can even use, some of our customers actually even use, um, an in-memory file system such as temp FS to really, really get the screaming fast performance from an inference perspective.

Um, what we, what we worked on over the past year is, the main thing that we've done is this parallel download feature, which the results have just been absolutely, um, phenomenal. So if you take a look at just like general model loading out of the box that comes with some of the, the generic, uh, frameworks, the main ones that are used is FS spec, and there's a flavor specific of GCS, uh, of cloud storage for FS spec called G-C-S-F-S. And, uh, the main, one of the main, um, like takeaways from this also is like, just don't use that.

It's, it's a very poorly written data loader and, and, and model loader. Um, and so in working with customers, they tell us like, Hey, we, these models are getting bigger and bigger and we need to be able to pull them as quickly, um, as possible, uh, both from the inference perspective or from the checkpoint restore perspective. And so what we did is we built this, um, a parallel downloads feature, which actually uses the file c some of our, uh, file C functionality where we use the file cache as a staging buffer, and we basically spin up multiple workers or multiple goroutines that go and fetch this file as quickly, um, as possible.

And as, as you can see from this chart, the, the, uh, performance improvement is just phenomenal, right? Ninex mass, uh, faster model load with parallel download. Um, someone was asking about VLLM before, this is done with VLM, and you just see, uh, again, all the way on the left FS spec, if you just use cloud storage fuse on its own, it's still gonna be better than FS spec, but you use, uh, cloud storage fuse now with parallel download, and you're gonna get nine x faster, uh, model the next.

And then the last one here is, is check pointing. So, um, four check pointing. Um, the, one of the, the things that we did is we kind of ma uh, we rely, we built hierarch, cool namespace bucket in, um, cloud storage.

And when you use this with cloud storage fuse, it's kind of like a match made in heaven. So what is h and s? Uh, h and S is hierarch cool namespace buckets, a by definition in object storage and cloud storage.

Um, it is a flat namespace. And so traditionally what what we do or what customers do is we emulate, um, folders or directories by just slashes in the, um, in the file path. So prefixes with h and s with hardcore namespace buckets, we now have a more true, um, a folder format, HA hierarchy, a hierarchical namespace.

And what we get as part of that is one is atomic folder renames, which is critical for checkpoints. The way that the checkpoint libraries are written, when they detect a file system, they just assume that you can do a, uh, directory rename, and it is atomic, but an object storage, again, that doesn't exist. It's an individual, uh, object name rename every single one.

So in a checkpoint, that means we're now going ahead and renaming every single file now with, uh, with hierarchal namespace, that is atomic. And so that becomes 30 x, um, faster, and it also provides up to eight x initial higher QPS. So again, you have, um, h and s, which gives something more a hierarchy.

You have cloud storage f which mounts a bucket as a local file system, get the two together, and you really have a, a, a very perfect and ideal solution. And then the second thing that we did is that we also imp uh, we also improved our right mechanism. Uh, the way that we started with cloud storage fuse is first we, we wanted to get the read path, uh, fixed.

Especially like in ai, ML it is mostly a read heavy workload. The only kind of, uh, outlier there is checkpoint rights. And so the way that cloud storage fuse used to work before was that rights, was that we staged them locally first, and then only when you closed them or you did an fsy would it then get uploaded to, to cloud storage.

But the problem with that, or many customer complaints was, well, then I need to provision enough space locally in order to stage everything, which depending on my parallelism, can just grow out of control. So now all we do is we have a memory buffer and we start uploading directly. So not only is it more efficient, but it's also much, much faster where we're seeing 40, uh, percent faster, uh, checkpoint, um, uh, a reduction in 40% of, uh, uh, checkpoint right time.

So very, very, um, efficient, uh, and, and lots of work that we put in cloud storage to really make it, um, you know, uh, to really make it stable at high scale use cases. Very quickly touching also on, um, checkpointing and the different options that we have. Um, this is kind of a very simplified view of also what we recommend from a checkpoint, um, perspective.

So on the left hand side, we have cloud storage, which again is gc uh, cloud storage views with h and s and streaming rights. And here we say, you know, five to 10 minute frequency checkpoint support up to one K nodes. Now we have customers that exceed that.

We have customers that have lower than five minutes, but general rule of thumb, this is kind of how to, to think of them. Uh, in the middle category, we have parallel file system, which will improve that, uh, further. So with managed luster, we typically say we can support up to one to five minute frequency checkpoints at one K nodes.

And then, um, what we just announced at next is our new multi-tier, uh, checkpoint solution. And this is really intended for customers checkpointing at the largest, um, scale, 32nd frequency checkpoints at more than 1000, uh, node scale. We did this with, uh, llama 70 B, um, at more than 1000 nodes, and we were way past the 32nd, um, frequency checkpoint.

The way that this works is that, uh, we, uh, use in, uh, in-memory ram disc, and we do checkpoints in memory in a local node and a peer node. And what this allows, and then, um, uh, based on a higher frequency, we then also those then also get backed up to cloud storage, actually using cloud storage fused, which is wrapped under, um, the solution. So the benefit of this is that you can get in memory performance, you can do very, very high frequent checkpoints without burning stuff like local SSDs, you get very high throughput because you're using the intra cluster network.

And so what that also means is from a checkpoint restore perspective, you can either restore from a local node, from a peer node on that same super fast East west network, or in the worst case scenario, if you have a total catastrophic failure or you just want a, uh, go back to a old point in time for a specific test, you can then restore that back, um, to GCS. Those. So these are kind of the three different solutions and where these line up from what, what we provide from a general recommendation.

Yeah. So question on Kimberly Bates, uh, group, the multi-tier checkpointing, is that just for cloud storage is So it it, it is for the backup part of it is for cloud storage, but it is for, it's a GCP based solution. So it's on Cooper, it's on GKE, and so it, it's a G it's A-A-G-C-P specific solution.

Okay, but it's not doing the luster? Correct. Okay.

So it just, yeah, it, it's, it's just addressing the object storage, cloud storage object for that. Yeah. And using, uh, and using memory in each node, not gp.

Got it. But to create a, a RAM disk. Yeah.

So where the data resides is object. I use this for the object storage, even though I may be using ram. Okay.

Got it. Okay. And Jim rinky zero defect computing, just to follow up on that, just so I'm clear, right, these are files that would be input to initial model testing, or is this just loading the model itself for pieces of the model, or is it everything?

Yeah, so, so these are your, during your checkpoint workflow where you are actually saving the state of your model weights. This is what that is, right? So it's actually, this is both primarily a right workload.

And then when you need to do a restore in the case, you have to do a restore, right? The larger the number of, of the larger number of nodes and GPUs down a cluster, the more likely you're gonna have a hardware failure. And so that's where you're then gonna do a restore.

And so rather than having to restore back from cloud storage, you can restore from an in-memory disc and a peer node on the same, um, network, which Shortens recovery time, which gets exactly running as quickly as Possible. Exactly. Failed Kubernetes node, for example.

Exactly. Got it. Perfect.

Thanks. Just wanna clarify. Um, and so just to wrap up this, this part on cloud storage fuse, this is a, um, customer example.

This is, um, uh, assembly ai, which is an enterprise grade speech AI company that processes petabytes of, of, of audio. Uh, and so I worked with Ahmed, um, this past year. Uh, you know, that he was having challenges and they have lots of small file challenges, high latency, and because lots of small files, there's also lots of operation costs.

And so we took a look at his workload, we saw there was lots of metadata lookup. And so what we said was, Hey, why don't you try cloud storage fuse with, with h and s? Um, with h and s, we also do a huge reduction in metadata lookups because we clearly know if something is a file or a folder and regularly in object storage, we don't know that we have to go and figure that out.

And so what he did from this test in this to really show you the, the power of this, he's, he saw a 10 x increase in throughput and a 15 x training performance, um, uh, improvement. So really, really cool to see an actual, um, example of, again, using this solution end to end. Alright, so now we're gonna talk about, um, anywhere cash, which as I mentioned before, uh, this is something that we announced at ga at, um, at, uh, uh, cloud storage next, um, this year.

So who here remembers the turbo button, that little red button on your computer? You would press it, you don't know what it did, but your computer would, making lots of noise and stuff would start going faster. So, um, this is actually, I don't take credit for this.

Someone from our engineering team came up with this. He probably should work in marketing. Um, but we, uh, cloud storage just got one, right?

So it's a very simple and efficient way. You just enable it. I'll show you how simple it is.

And what this does is it, it, it, it improves performance by, uh, co-locating, uh, storage in the same zone as compute. So cloud storage is a regional offering now. It becomes a anywhere cash makes it now a zonal offering that's as close as possible to to, to your compute.

And not only that, but it's also on SSD. So you have zonal co-location, you have SSD stuff is going to, um, to be faster. Uh, and as part of that, again, one of the main value adds of this is just how simple it is and how easy it is to use and where, and that you can use it anywhere.

So no need to refactor code, no need to change what path you're reading from. It's literally you enable it and you just continue reading from your bucket as you did today. It just, it just works.

So, uh, you press the, the turbo button, you enable anywhere cash. Um, what happens, it is a, uh, there is a significant time to first bite, uh, latency reduction on the, on the top of your, you can see a 70% reduction for, um, regional buckets. And then for multi-regional buckets, which I'll talk about, also, very special, unique value that we have when combining anywhere cash with multi-regional buckets is a 96, um, percent reduction in, um, in, in latency.

And what I have here also on the bottom right is this is a large gen AI customer that uses anywhere cache for a model load with multi-regional buckets. Very, very awesome use case. So this is a customer that grew out of a single region, so they were able to get many, many GPUs in one region, and then they got many, many other GPUs in, in another region.

And traditionally the way that they had set it up was one region for training, one region for inference. But because inference was so critical as their user demand was picking up, they also wanted to be able to use some of the capacity in the training side also, uh, for inference. And so what they had to do before was one was they had to do all this manual data movement around every time, keeping everything in sync.

And the second thing also that they were seeing so, and sorry, and in addition to that, that that also had cross region egress, um, costs associated with it. But, and then the second thing also is that, again, when you're deploying at scale thousand node scale or higher tail latencies are going to make a huge impact. So you're not gonna notice it if you're deploying, you know, 10 nodes or whatever.

But they were seeing that tail latency model load on, you know, one node or 10 nodes of that thousand node cluster was just Hal, uh, just keeping everything, um, uh, like bottlenecked. And so what they did is we, they moved to a multi-regional bucket, uh, cloud storage, multi-regional bucket where they stored the data once they spun up at anywhere cache in each of the two, um, zones that they have in different regions. And then they just began reading from it.

And what they saw from this is a model load, is a highly cashable workload, which you can see here, 99% cash hit. Um, and they com it completely eliminated their, uh, tail end latencies. And it also helped them, uh, save money in, uh, network egress costs.

So, Uh, just a question. Is, uh, indie bank, are you using some sort of, uh, predictive cash population to do this? Or is this all just Nope.

Good. You know, it's a good question. So we have two different, uh, cache ingest behaviors.

You have a cache on first read, uh, or we have a cache on second read, so yeah, you know, if you wanna make sure that you're, you're, uh, ingesting on a, you know that your data is highly cashable, you can enable the second one. So read it twice and then ingest. But the first one is basically if you get a cache mix, we'll load it into the cache and then once it's ready, uh, sorry.

Once it's in there, if you do a repeat read, you'll get a cache hit. Okay. But you don't do any predictive population?

No, we do not. Okay. You were talking about the, the increased or decreased latency to first read, and I was wondering if you were doing Yeah, no, that's based, that's based on, uh, a cash hit versus a cash miss.

Okay. Yeah. Uh, so under the hood, again, what, what, what's happening here?

So, um, it's fast. 5 terabytes a second of additional throughput in, in, uh, in addition to what you get from cloud storage today. Very, very simple.

Again, completely transparent to your application, just click a button, enable it, and, um, you pay for what you use. Uh, so it's very, very efficient, uh, pay for what you use and it auto scales to the capacity that you need. Um, so kind of like what customers would expect from a cloud storage offering, uh, in a hyperscaler cloud.

Uh, is it also considered ephemeral? Can you, uh, will it go away when you're not using it? So it is a, it doesn't go away when you're not using it.

You can pause the cache if you want that, that's another management capability to have that. You can just pause it or you can just disable it and it, it goes away. Okay.

So, so I mean, if, if you're using this for an ephemeral workload, it, um, it would go away with that workload. Yeah. If you, you just, you, you enable it on a bucket level, when you're done using it, you just disable it.

Okay. That question bar, doing this from, um, so you're, it has a certain price because you're adding SSDs and so on. Um, you say it's pay per use.

So if I say I'll enable a hundred gig, but I'm using only 50, then I'm paying for 50. No, You don't specify a size, actually, you enable it, you begin reading. And then based on your cache ingestion policy, whether it's that, uh, on first read or second read, that just gets loaded in there and your billed for the bytes that get ingested in there.

So you don't specify a size. But if I have a terabyte, uh, database that I start reading, I can't have a cache of one terabyte on. Yeah.

So it's a chunk based, chunk based cache. So we, uh, we only, um, we ingest in two megabyte chunks. Mm-hmm.

And so, um, the way that that works is, uh, specifically for the analytical workloads, you have, uh, a very high portion of hot data within a larger file. And so that what, what that benefits in AI ml, you're typically loading a full, you have a data set for training, you're reading that entire thing at the end, of course end. But for these analytical workloads where you have a giant set of parquet files, uh, and you have very specific, you know, row or table that you're reading, it will only ingest that in two megabyte chunks.

Mm-hmm. Do you have any customers running any cash with confidential computing? Um, I'm not sure.

Okay. What, what is, is there a specific reason you asked that? Why, why confidential?

Yeah. I mean, it seems like it should be decoupled and not a problem, but I'd just be with sort of the caches and the way you'd set up confidential computing. You know, the Yeah, it should, it, it, it, like I said, Just wondering how to your customers that, yeah, it seems like they'd be more complex than, yeah.

Turning both on and magic work. You know, It should, as long as you have, uh, uh, a like a connection to a cloud storage bucket, it, it'll just work. Okay.

So, Um, so another very interesting thing that we do, that we do to make it very easy for, um, customers to use or understand if they're a fit, is we also have a built-in, um, recommender. Uh, so, um, if you log into our, uh, you know, the, the, our web ui, you look at a bucket, you'll actually see an, in inte you'll see a thing that says recommender, and it will tell users how cashable a workload is, so what their cash hit percentage is. Uh, it will recommend the optimal cash configuration, and it'll tell you, you'll also, your throughput and your cost, including cost savings.

So to your question before also around how do I help understand, like the cost and everything, this recommender will actually give you, uh, a very precise information on whether your workload can benefit from this or not. Okay. With Nexus Tech, previously we saw a recorded demo.

It also had the Google recommends ui ux label at the very top. Okay. You know, in the web ui, is that a consistent web UI that you're saying, like, these are the Google recommends and that should stand out on the page for the user or, So this is specific in the, our cloud storage, uh, bucket page.

Mm-hmm. So it's within that and specific to the anywhere cache, how it flows back up to the larger recommendation? I'm not exactly sure, uh, but I can, we can, I can look it up after.

Maybe we can talk. Thanks. Alright.

So, um, the other thing that I mentioned before is, so a multi-region bucket, um, in cloud storage is a very, just alone, it's a very differentiated offering, right? So we have our single region buckets, which are optimized for, um, performance. Uh, we have our dual region buckets, which gives you performance with higher, um, availability, right?

And then we have multi-region, which has higher availability, uh, but lower performance because you don't know, um, where the, the data is. And so now for now, what we have is we have a, we have the ability to attach a multi-region bucket to an anywhere cash and basically, uh, not have to have any trade-offs. So you'll have the super high performance, you'll have storage that is, um, uh, co-located with compute in the same zone and backed by a multi-region for the higher, um, label levels of, um, of availability.

And what's really awesome about this too is that, uh, with this, we all, uh, from when you're doing a read from a multi-region bucket, you are also there traditionally there is a, uh, multi-region egress fee that you are, are paying with this, uh, that gets eliminated. So we make it, um, much more consumable and much more attractive offering to use anywhere cash with multi-regional buckets, and again, get the benefits of the, the best possible performance and the highest levels of availability, again, without also, without having to worry about this, uh, this data movement, um, overhead. So in the recommender that you have, does it give you, and you go through this process, does it give you the expected throughput with this multi-region anywhere cache hit rate?

Yes. Yep. Combinations.

Yeah, Exactly. It'll, it'll show you the peak throughput that you'll be able to get from your cash data. And, um, more specifically to what you just asked, we also show a very specific savings in terms of multi-regional egress that you will get from, uh, from anywhere cache.

That's really cool. Thank you. Yeah.

Um, for, this is the, the last thing I just wanted to show you guys a very quick demo. Um, so here, this is, this shows you how easy it is to use in the performance you get out of it. So you under configuration in your bucket, you have a, uh, anywhere cache here, it's already running, but you ba it basically shows you the zones, uh, within the region.

You choose the zone, and then you simply check it off and you press enable, and that's it. The, the anywhere cache is go, goes ahead and starts, um, starts building. And then, uh, what we have here is we have a demo that does a comparison of a, this is a model load, um, simulation.

So this is a 10 gigabyte file. On one side we have, um, uh, u uh, using just the GCS standard bucket in order to read this 10 gigabyte, um, uh, model. And on the right here we have Anywhere Cache, um, uh, we have a, a a a bucket that has anywhere cache enabled.

And so what you see from this is when we kick it off is you'll see that with anywhere cache, it takes about 47 seconds to complete. So these kickoff at the exact same time, and then with the standard GCS regional bucket, it takes about, uh, double that. Um, so we'll jump to the end here, but it's about, uh, 93 seconds.

And then we, as part of this also we have in inside of our observability metrics, we have, uh, you can go ahead and see performance monitoring, your cash hit rate, um, uh, monitor the throughput that you're getting, uh, from it. So just treat it like, uh, you know, its own, uh, management object that you, you have insights into, uh, what you're driving and what performance you're getting out of it. Just a quick question for this demo, I assume that you pre-populated the cache already?

Yes. Yep. Okay.

Yeah, this is cash hit. So, and model load again, like we showed before in that, um, that example before of that Gen AI company, it's a highly cashable workload, So, okay. Yeah.