Nutanix Data Architecture and Safety

My name is Manoj Naik. I'm a distinguished engineer at Nutanix, and today I'll walk you through the architecture that powers our cloud platform, specifically the unified data platform, the design principles, the ideas, the systems that allow us to scale and perform, um, the way we provide mobility, all the way from the edge to the core to the cloud, and how we provide visibility and analytics of user data while at the same time keeping the data secure and compliant and all of this in support of a, a true for AI lifecycle. So before I present our architecture, I want to levels set the stage on what is available out there today, right?

It's as, as what has, what have companies that provide enterprise storage for traditional unstructured workloads look like. Right? There have been these two classic ways of doing, uh, enterprise storage.

The first one is what is called the shared Everything architecture, where essentially you have a cluster of nodes that have all have access to shared storage. This shared storage typically has been provided in the past through a storage area network, but more recently through IP based storage. And then you run the load balancer or some way to direct clients to any of these nodes, um, to provide a single namespace.

The idea here looks great because every node has access to all of the storage. So any client can access any node and get access to any of the data. But then, you know, these architectures do have challenges.

Um, data is accessed from anywhere, which means you need some kind of concurrency, some inter node communication. You need some version of distributed locking to build, uh, such an architecture. And for the interconnect between the nodes, you need something that's a fast path for networking between the nodes.

Um, that's one. The second one is, is more of a shared nothing architecture, where you provide a single namespace still, but then you have a collection of nodes that are working independent of each other, right? They're, they're in separate fault domains.

They're not typically communicating with each other for providing access to data, but then each node or each piece of this namespace brings along with it compute storage, and, and, and then that's how they provide the namespace. This also looks great because it can scale linearly. The domains are different, but then how do you provide a single namespace without having a siloed infrastructure?

There's a problem that, you know, there's load balancing issues. How do you make sure that you know, specific parts of the namespace don't get, you know, get hotspots? And so I've, in my past life worked on both of these architectures, and, you know, like I said, both of these have challenges.

Um, so what is it that we need to do to break this conundrum that, you know, has to work on trade offs between resiliency and efficiency? We need a new way that sort of combines the best of both worlds, right? We like the fact that there is accessibility, that the, that the shared everything architecture provides, but we also like the linear scaling that we get from shared nothing.

And so we want to build an architecture that sort of takes the best of both world, and we call this architecture shared flexible. And the idea here is that to provide a global namespace across still a cluster of nodes, but then you're accessing shared storage that's virtual, right? Rather than every node bringing its own storage, the nodes are flexible enough.

Some of these nodes can be compute only. Some of these can have, are heavy on storage or a mix in between. But the idea is you want to be able to use storage that is within the single cluster itself.

Desegregated, right? The compute can scale independent of the storage within one cluster. So typically the desegregated is now a new buzzword where you separate out the compute and storage into different clusters, where the idea here is that you want to be able to scale your compute and storage within the same Platform.

Manoj, uh, comment, maybe Scott Roon solution, um, you can call it disaggregated, or you could call it reality. Yeah. Like this, I think, reflects most enterprise environments, right?

Yeah. But the endpoints you had before are end points and maybe only 2% of the problem. Yeah, Actually, yeah.

What I've seen is people have built these architectures, these, you know, especially around things like unified storage. Sure. You know, companies have been around for 25, 30 years.

They build this architecture and they're sort of improved on it. You know, you still have, if you look at the underpinnings of these platforms, still remain shared nothing or shared everything, right? And they sort of try to bring in the other properties of the other architecture into them.

I think what we've done is we sort of realized this problem early on, and we, we started 10 years ago. We didn't start 30 years ago building a platform. And so 10 years ago we realized these were problems already.

And so we started out building a platform that already had this from the very beginning. And, and the idea here is that we wanna be able to start small scale this from like few terabytes to multi petabytes in a single architecture. The same platform will support all the way from small to large.

Okay. So, um, before I go into the details of our platform, I wanna set the stage on what, you know, our principles are, right? What is it that we, how did we go about building this architecture?

So the first thing is the center of everything that we do. You know, everything we do for performance optimization is data, right? And it's a data first architecture.

So the, the foundation of our architecture is scale out. You know, we want to be able to start small, you know, as the performance compute and workload needs increase, we scale linearly. And when I say linearly, it's not a proposal, right?

It's a reality. I mean, we see that, you know, you start with a three node cluster and increase it to, and I show later in the slides that performance scales absolutely linearly, right? Uh, you can grow compute and storage capacity independent of each other.

The other one is about having fine grade metadata. And this is important because it allows us to smartly place data in our platform. You know, we have tiering all the way from within the platform, from NVME to SST, to hybrid drives, all the way to the cloud.

And we also do intelligent caching and, um, you know, and, and other ways in which we can give visibility into how the users are accessing the data, which later on allows us to do analytics. Can you go a little bit deeper into how caching? Yeah.

Um, the other thing is that we do exploit protocol features for, for data locality. So traditionally people have built, uh, NAS environments to support all kinds of protocols, right? But we've taken the approach of saying that there is advancements in the protocols now.

I mean, they've been there for many years, but people have started sort of leveraging them now, right? Things like referrals, multichannel in the SMB protocol, you know, multiple connections and referrals and, you know, PNFS in, in the NFS stack. And we do take advantages of these.

We exploit these protocol features. The idea is that we want to be, yes, You support plex files under PNFS. We, we are on, in the process of building A-P-N-F-S solution.

I'll talk a little bit about that. You're Not PNFS yet. We, We are.

PNFS in development will be there shortly. Um, but we do support what most of the features of the V four protocol, which are, which allow us to do referrals. And I'll talk about, so people have typically, just to answer that question specifically, right?

People have done referrals in a way that, you know, generally builds namespace. We have extended referrals to go all the way down to directory levels. And so we can do actually low level sharding of our data through referrals.

And so the idea here is to be able to direct clients to where the data actually is, rather than moving, you know, data between the nodes in the cluster. This is, uh, Keith from the advisor bench. Can we ground this in kind of the, uh, some of us are not deep storage people, so we're not always rocking kind of the relevance to the ai mm-hmm.

Uh, our AI data pipeline. How does this help me in my AI data Pipeline? Yeah.

So I think the idea that Vishal set the stage right for the data pipeline as you, as you showed for the entire lifecycle, there are various ways in which data flows through the system, right? So we want to be able to support data that's collected at the edges and at the edges. They might be a single node cluster, for example, or even very small clusters as the data flows through the pipeline.

I think it lands on data centers in the cloud. You want the same platform to be able to address the data in various places, right? Assuming you want to build a single unified platform for all of your data as it flows through the various cycles.

You wanna build a system that can work on the edge, can work in the data center, and work in the cloud. The same architecture has to work in all places. The second thing is, you want to be able to provide mobility, right?

You want the data, assuming that you want the same data to move around, you want the ability to move the data from these places. And the third thing is visibility, right? It's not just collecting data.

You want to be able to make sure you have access to that data and we able to keep it secure, right? So I think the principles here are sort of addressing the idea that data as it flows through the AI lifecycle pipeline is, is always available, always performant, always secure. I guess, Bob, go ahead.

I'm sorry. And I, I think we're missing the critical, I think this is operational consistency, right? So whether I'm edge data center or cloud, my underlying data pipeline, at least the storage feature of this, and when we talk about the database capabilities, consistently, this is a problem in AI that no matter where my inference and I'm fo I'm inference focused, where that's running, that the, uh, my operations isn't changing.

Mm-hmm. That's a question. I'm assuming that that's a outcome that you guys are achieving here.

Yeah. So I think what we are setting the stage for is that the fact that the platform remains the same, whether you run it at the edge in the data center of the cloud, I think we are setting the stage that the unified data platform is the same. And I would say the answer is yes.

Like, so the way we are looking at, like, for example, for the inferencing piece, like if you're running a vector database for that, yes. But you're also looking at say, a KV cache kind of solution. You want to cache all that inferencing data.

That's where some of these high performance, scalable storage becomes very important. Yeah. So the reason why I ask this question is, you know, as we think about stuff like, uh, prefill and I'm, if I have a team focused on troubleshooting prefilled, and we have to go beyond the platform interface, and I need to look at logs, and I need to understand, uh, from a performance and telemetry perspective, how do I get the fastest tokens?

If that's my goal out of the system, I'm not changing my SRE process for the edge versus the data center versus the cloud. Yes, Exactly. That's what Operational consist, that's what I mean by operational.

Yeah, Absolutely. Yeah. Okay.

The next thing is we are performance optimiz question. Yes. The follow on Keith is there's, you know, from a AI perspective, data orchestration becomes more and more important.

And where does this solution fit into that space? Yeah, let, let me get to that. I mean, quick, good.

Yeah. Okay. So the other thing we wanted to talk about is performance, right?

Um, obviously performance is important for AI workloads, and we do not, we are software defined, so we do not rely on specialized hardware, but, you know, like NVME drives or, you know, fast interconnects or GPUs, we don't rely on them. But if they're present on the system, why will take advantage of those? Um, and then, you know, beyond the data first architecture within the cloud, uh, within the data center, we do wanna be able to provide mobility and replication has been something that we've built right from the inception of the product.

And we are cloud integrated more recently now by being able to run, uh, natively in, in public cloud. And that allows us to move data seamlessly between on-prem and cloud. And, and the last one is being able to provide data protection and governance, right?

With built-in rules that allow us to, to do this across the stack. So the foundation of our, And mon when you say data protection, you're talking about snapshotting, Snapshots, replication, metro sync cloud. Yeah.

But you're not doing outboard backup of databases or, or things of that nature. You're not putting it on, you know, some our medium. No, we are not.

We, we were putting it on our own object store. So we have a builtin object store, but Yeah, that's what I was trying to say. Yeah.

The, along with our data first architecture, we also have, uh, a control plane. We wanna make sure that our control plane works in concert with our data. Uh, we have a unified control plane that is powered by what we call Prism Central, that provides a single pane of glass, whether you're managing one cluster, one site cloud environments, you get the same control plane.

We provide data insights and analytics into data. You know, we able to detect user patterns, you know, identify threats, you know, remediate them. Um, we are able to convert what we call raw telemetry into, um, insights.

Um, we have a lifecycle manager that sort of manages the, you know, install upgrade of our entire stack, right? All the way from hardware if you happen to run ours all the way to the apps. Um, we support open APIs, obviously, we want to be part of the ecosystem for automation, the integration with third party tools and DevOps workflows.

We have a marketplace on our, um, on our Prism Central and, and Nutanix Central. Think of it as an app store for your platform. It allows you to switch between apps and gives you visibility into all the apps that we support.

Uh, Would you see, would this solution work for something like data lakes and things of that nature? I mean, sizable storage configuration? Yeah.

I'll quickly get to that. Hundred petabytes Yes. Kinds of things.

Yep. Yeah. Yes, yes.

Gimme, gimme a moment. That's the next slide coming up. Okay.

Let's talk about scale just as what you were pointing out, right? We wanna be able to, like I said, start small, but scale for capacity and performance, right? So let's see what it means to start small, right?

So our clusters can be as small as 1 1 1 node, right? But typically for high availability, you need to run at least a three node cluster that gives us the platform to run on top of. This gives us the ability, the platform gives us what are called containers that provide, you know, all the enterprise storage primitives, like replication, compression, all of that stuff.

And then on top of this, we run our services, right? Our apps that provide the namespace for objects and our namespace for files. And all of these managed through Prism Central.

Now, let's say you this, this is a full, fully featured resilient, you know, solution that can be deployed at the edge. And this is how you would, you know, start out if you had a smaller workload, but as the workload grows, you can actually add capacity to the cluster, right? Seamlessly without impacting the applications that run on top, right?

And capacity scaling is just a matter of adding more storage to your environment. It's the same cluster where it scales, right? And similarly, if you were to scale compute instead of your storage, you could just add more service endpoints, right?

So for example, in case of an objects namespace, we just add more work nodes. In the case of files, you know, we would do the same thing by adding more file server VMs. And so here's what the question specifically you asked, right?

How, how can we scale as data grows, right? So if you were building a data lake where you know, you have unstructured growth of data, which is almost relentless, then you can actually scale our cluster to up to 32 nodes within a single cluster, right? Each node in our cluster today supports more than a petabyte, which means that if you have a 32 node cluster, you can have multi petabytes in a single cluster, right?

And of course, you want to go beyond the limits of a single cluster. So if you were building a very large data lake, you can actually pre present a single namespace across multiple of these clusters, right? So in case of an objects namespace, for example, you can have 128 clusters connected in the single global namespace.

And for files, you can go up to 10 clusters in a namespace. And the idea of a namespace here, I think the question was asked earlier about what a global namespace means, right? So the idea is for our clients to be able to access a single namespace, right?

So from the N-F-S-S-M-B S3 client point of view, they see one endpoint, right? They see one endpoint, but we rely on the clients being smart enough to be able to get redirected anywhere. And this is a federation workspace or a, uh, of a way of, uh, letting clients get to where the data is.

So in terms of scale, I mean, we haven't even gotten to point where we can test at the scale that we are talking about, but we can go multi multi petabytes in a single name space, Getting back to the object enable name object, global namespace, is that the fact that one bucket can exist across multiple clusters With replication? It can, because we do support replication. So you're replicating the data from one cluster to the same, but it's the same data effectively in all those clusters.

Now, If you want to use replication, but our namespace means that you can actually have your buckets all over the place. I understand. Can you have one bucket span, multiple clusters, and that's not being replicated.

No, you cannot. But, but thank you. But yeah, within, you can have one, one bucket span multiple clusters in a single data center, but it doesn't make sense for you to have one bucket span across clusters that are geographically distributed, but within a cluster.

Yes. The answer is yes. That's You mentioned.

So Opposite answers. So you can have, from a latency perspective, if it makes sense to have multiple clusters in a single domain physical domain, you can do that. Yes, you can do that.

But If you're talking about extending it across multiple physical domains is better to replicate. Yeah. And we do replicate in those cases.

Yes. Is that Scott Roon? Al?

Is that configurable? Are there, I I guess like where are you on the slider bar of I want it all in one place versus I just wanna leave it where it is and access it. Yeah, so the way we define is, you know, we've been coming up with these concepts called infrastructure domains and availability zones.

And you know, it all depends on the latency between these clusters, right? And what you need it for application performance, right? It's gonna be application, but It also depends on how you replicate at the backend.

I mean, we do expect for a replication to work, you know, or not replication, but for a single bucket to span multiple clusters, you need latency that is low, right? Milliseconds. Yeah.

And if we have to rely on replication, we just expect it to be asynchronous. So that's, I think, you know, we, we are cluster stretch across multiple rooms, multiple, you know, in a single data center. Sure.

But then that's a more of a stretch environment, right? But we still expect the latency and the availability to be, as you would expect in a data center. So basically everything you're talking about is very focused on AI workload and data for AI workloads where, where low latency is, is absolutely necessary.

Yeah. Yeah. And so you're probably gonna not air, but lean toward replication to get everything in one place for performance reasons.

If, if you, if you wanted Yes. Replication to get data local to where it needs to be. Yes.

Okay. So just like you can scale for capacity, we also scale for performance. The, the key point of this chart is to show that, you know, our scaling is linear, right?

Um, you know, as you add worker VMs or, uh, worker nodes or VMs to the cluster, our performance will scale linearly. And, um, next I wanna take you through what, you know, our IO path looks like, right? And what we have done over time to improve the IO path of a typical pipeline that goes all the way from, you know, so we do support, like I said, um, N-F-S-S-M-B S3 into the same interface.

But I'll take the example of an NFS client, um, accessing data or an, a flow of data from all the way from a client to, to our storage, right? So let, let's look at what our architecture looks like, or this has been how we've sort of built our platform, um, in the past. There is an NFS client that accesses, uh, a service endpoint that we call files, which is part of our unified storage.

Um, there is an NFS adapter that runs within files, and there's a file system that talks to, to, you know, so when a request comes in from an NFS client, it's typically TCP or that's what it's been in the, in more commonly, we use, uh, an IP based networking to connect to storage, and we go over what, what we call a virtual switch within our hypervisor. And that typically follows a path that goes through the kernel, you know, all the way up to another service that we run, um, on our controller virtual machine called Stargate, which, which actually provides the storage functionality, right? And this has been the path typically.

And then for replication within the cluster for availability, we need to go another path that goes through the kernel again, to another node in the cluster. And that's how we get replication within the cluster. Like for, uh, rf.

And this has been where, what we started with. So the first thing we did was we looked at whether we can do fast path between our, our storage layers, right? So within Stargate, we leveraged Rocky for providing data plane efficiency for replication within the cluster, you know, for metadata operations.

You know, we did a fast path between our, our storage layers. This is all transparent to applications, and that gave us performance boost. The second thing we did is we introduced this feature in our hypervisor called Turbo, which basically is, again, transparent to applications, but it allows us to go fast path between our hypervisor and our storage stack, right?

So it bypasses the kernel on the, on the host machine and go straight between two user processes that are running on the same node, right? And that we call, um, a HV Turbo, and it provides ICE CZI access over RDMA. So that's the fast path on, on the data path.

And then the last thing we've done more recently is to go all out doing RDMA all the way to the client, right? So the path from our service, uh, to, to our hypervisor is now with direct attached storage, right? So we don't go IP anymore, we go direct attached, and we also do RDMA to the client, which allows us to take advantage of things like GPU Direct, uh, the client running GPUs and so on, right?

So this, this gives us a end to end RDMA path all the way from the client to storage. And we've been successful with publishing our, uh, performance with, with the community, right? So we've been early adopters of the ML commons ML perf, uh, benchmark that came out of the ML Commons welcome, uh, standard.

Um, we started interacting with the community. We have a couple of folks who are part of the committee, and they've been working with them since 2023. And so what ML PERF Benchmark does, just to give you an idea, is that it, it simulates real world AI ML pipelines, right?

It, I mean, typically people run storage benchmarks using, looking at raw IO throughput, but actually simulates an end-to-end workflow of a model training exercise. And it measures performance across two of these workloads. One is called RESNET 50, which is more for image classification.

It sort of exercises small reads performance. And the other is unit 3D, which is more for, you know, medical image simulation. Um, it exercises, large sequential reads and checkpointing, which does heavy writes as well, right?

So workloads that sort of work across both sequential and random reads and writes, but not in a typical benchmarks that you run with FIO. So it, it's, it actually simulates an actual pipeline. And so, like I said, we, we sort of participate in this benchmark, um, and, and we've been among the top, uh, um, vendors that have published in this space.

In last year, we were able to get about 2100 accelerators and Accelerator is essentially what it takes to keep a GPU accelerator busy for 90% of the time. So the IO load that you would exercise to get an, an accelerator busy, um, we were able to improve this in this year by two times. And so we got the same number of accelerators, but we used half the hardware for it, right?

So our first solution was with 32 node cluster. The second solution with was with a 16 node cluster, we could have gone higher, but you just don't have the hardware with, you know, that large. Um, and, and our per node storage performance has gone up 130%, right?

So key enhancements here are what I described earlier in the slide, right? RDMA all the way to the, to the NFS client, S-R-I-O-V, we use for, you know, near native performance and high performance networking, right? We use Connect X seven H, The performance applies to both resnet as well as your net three 3D Uh, yes.

So our Files, small files, both, both are performing better with, uh, the latest rev. Yes. Yes.

130% increase. Yes. And what you're saying there is how many, um, accelerators you can support.

Yeah. So we, like I said, it says there 2300 accelerators right now on a cluster of 16 nodes. Yeah.

Um, we can go higher, but I'll talk about, these Are like H 100. Yes. I mean different, you could have different, yes.

Right? I don't know the details. A one hundreds, H one hundreds.

It's a combination. I think. I think it's the, I I think the, the RENET is a 100 if I'm not mistaken, but, you know, I'll have to check.

I don't know which ones is that. Yeah, so Dave Graham, ml Commons, the providers of, uh, this ML perf Yeah, ML PERF is one of our, one of the thing. Yeah.

And, and, and one of the things to call out here, and I I appreciate your attention to this detail. Like these benchmarks are constantly being revised based on c uh, community standards as well. So like the, the, the sample that Nutanix is providing here is, is amazing.

This is indicative of a point in time snapshot. And this continues to evolve, I think on a biannual basis anyway, as we evaluated. So actually, I think resnet 50 is gonna disappear in the next benchmark anyway, because it's gonna be replaced by a different benchmark.

But when you start to look at these things, this is exactly why we do this. It allows the individual vendors themselves to take a look at their characterization on a path forward and ready to reflect your points. Like, yeah, what does it look like in small, medium, and large block workloads, elephant flows, mice flows, this kind of thing, and how does it impact these pipelines as it goes along the way?

And so when you're able to characterize things in, in, in this fashion, yeah, you, you hope for that, that, you know, 200% increase or two x increase over, over what's going on there. So yeah. No, in, in Nutanix has been a, a, a great part in, in upleveling the entire industry by virtue of what they're doing here as well.

Right? They contribute to the, the understanding of what these benchmarks mean. So it's not just useful to them in their own marketing, which they're not allowed to compare themselves to other people, by the way.

Which is, which is great, but it's, I can Compare it to other People's, but you can, you absolutely can. If you give me a benchmark, I'm going to use it. Yeah.

But this, you know, just for, yeah, just from a broad brush, I mean, this is exactly what we want to see from this. 'cause this shows that kind of tendency to move forward. And if you want details on somebody's benchmarks, we can certainly provide, And this is a training solution, is if I'm, this is not inference, uh, ML perf storage, right?

Uh, yeah. You, this is the ML perf storage, right? Checkpointing activity, yes.

Seeing all the activity that might go on in a typical training solution, but you're not, so inferencing is gonna look somewhat different. It should, if we ever get to that point, it Should. Yeah.

I mean, uh, Nutanix, I mean, I can't speak to their, uh, you know, their overall participant issue of the groups, but typically as, as a member, they're able to have impact on the inference trainings, the inference and training specific work groups themselves, which have their own discrete benchmarks. So you may see them show up in those particular things, but this is directly onto the ML per storage side, right. Of the house.

So it does try to be combinatorial and, and, and workloads to, you know, represent as much of that possible. Right. Maj, nick patience ask, um, as you scale, you know, they are very impressive numbers.

Um, where do you see sort of the next bottlenecks going if you, if you like, you know, double the amount of accelerators or something like that? Is it like file metadata or is it, or network fabric or, Yeah, so I, I think one of the challenges that we've had and that we are working on is the fact that we still want to run on a, on a platform, right? We still wanna be software defined.

We still have to go through the stack that goes through the hypervisor. So we've been working on some accelerations, for example, you know, virtual functions from our, for our network cards, right? So in this exercise we took Connect X seven and eight network cards and did pass through directly to our file server VMs, right?

So there are some accelerations with that we have to do around smart next things like that, that allow us to get the same kind of visibility into our service layer that you would not expect typically in a, in a, in a virtualized environment, right? So those have been where we've been working on. Um, again, we wanna be able to, like I said, software work in within the bound of the fact that we wanna be software defined.

The same stack works in the cloud, right? And we, in fact, the first benchmark that we did in 2024 was actually running entirely natively in AWS, right? So the 32 nodes that we were able to run a cluster was in AWS, right?

Using EC2 instances rather than, you know, our hardware, right? Or our platform, it was actually running natively. So we wanna be able to, our software needs to be able to stay software defined and or our solution needs to be able to say software defined, but we do need to be able to take advantage of these improvements that have come.

Were Those on metal instances or were you using some other No, no. For the, for the cloud? No, they're, they're just EC2 instances, Which with GPU, Yes.

No, no. This is a simulation, right? So it doesn't actually run with Ppu is being burned.

Yeah. So the Yeah, exactly. No harm.

Yeah. So, so you just wanna show that, you know, and by the way, in AWS we can't use RDMA, so we were just using TCP. So the first iteration of ours was just TCP, right?

The second one was with RDMA. And that's another reason why we got the acceleration the second time around with half the number of nodes. The first one was just using pure TCP from the clients, right?

Okay. So, like I said, constantly improving performance. We started out with the first iteration of ML Perth performance of about five gigabytes per second per node.

We are, we are at 20 gigabytes per second now, right? With all the exploration work that we have done. The next thing I wanna talk about is it's not just about performance, right?

Obviously in a, in a stack like this, you want to be able to handle things like, you know, data safety, mobility and all of that stuff. And so we do have a complete SLA based data protection in our stack, right? So this was the questions that came up earlier.

What do we provide in terms of, in terms of data protection? So I think of this as a wheel with every spoke in the wheel, providing some form of capability around data protection. The first one is metro.

Metro is essentially being able to synchronously replicate our entire platform from one cluster to another, right? Whatever runs on it, you can pick and choose, you can have the entire cluster replicated, or you can have entity based replication. So you can pick a subset of your workload with some VMs and your file servers to be replicated to, uh, another cluster.

This is RPO of zero, Right? And just to make sure, like you, when you mean like a geographic metro, right? Like yes.

Within a, within a, with Some requirements, right? You still want under five millisecond latency, for example. Yeah.

Okay, that's fine. Yes. But like not just within a data center.

Absolutely not. Okay. Yes.

Uh, the second one is like snapshots and D right? I mean, this is standard, nothing special about what we do, but we do provide, you know, snapshots to the level of a share, you know, and async replications, right? This is happening at the service layer, right?

We don't have to take the platform in place. So you don't have to involve the VMs. You can just do at the service level replication And, and the metro replication applies to object, It applies to anything that runs on our platform.

So you could just say, I want, so A multi petabyte data lake, you wouldn't do, it could be replicated across the metro boundary. You could, But if you wanted to, but it doesn't make sense to do the Choices, are you back it up or you replicate it? I mean, to, to a large extent, right?

Yes. And you know, we do have people who replicate, you know, do metro in petabytes. We do have customers, maybe not hundreds of petabytes, but yeah, I mean, petabytes we do well, The, the German customers practice, we Do have customers, in fact, especially in Europe, people will, will do metro for everything.

I mean, they will, it doesn't matter how much data they use. And we are seeing that in the Middle East also, like quite a few of those deployments are requiring metro NDR, like the cascading one that you talked about. Yeah, we will, we will push people to do async replication if it's of that scale.

But, but we do support metro. So other things like object replication, like I said, you know, we do have our own object store that runs on-prem, but then if you wanted to integrate with an object store in the cloud, like, uh, Amazon's S3, you can actually do replication between our object store and the cloud. And then we support things like, you know, for SEC compliance, we have support for object locking and file locking with right ones semantics.

And we also do change file backups, right? So our backups work with what we call change file tracking, nothing new and mean, everybody does it. But we've integrated with our backup vendors.

So our backup vendors are able to use our open APIs to be able to do backup using, uh, differencing.