Analytics Storage and AI, Data Prep and Data Lakes with Google Cloud

Hey everybody, my name's Vivek Sowe. I'm a group product manager here at Google Cloud and Google Cloud Storage. I'm actually gonna take us a step back.

We've been talking a lot about AI and AI workloads for the past. Uh, the, the since the morning focused on compute and storage. I'm actually gonna talk to you about analytics.

So think E-T-L-E-L-T, Hadoop and Spark. First question you might ask is, what does that have to do with ai? And the answer it turns out is quite a bit.

So we're gonna talk about two specific things, data prep and data lakes related to analytics and AI workloads. And what I'm here to convince you is that these workloads are tied very closely together and that there are some key innovations that we built to help solve problems related to that. So three key, three key takeaways from this talk.

The first is that analytics is a key component of the AI data pipeline. In, in particular the beginning part with ingestion, data prep and data cleanings. That's number one.

Number two is that customers are actually building unified data lakehouse, often using the Apache iceberg open metadata table format. And that enables running both analytics and AI workloads. And it includes running analytics on AI data.

And the third thing is that at cloud storage, we built several key innovations for performance, simplicity, and intelligence for solving some of these data prep and data lake use cases. So let's get into it. So, um, when Dan presented on lu, he showed, uh, earlier today, he showed this pipeline of, uh, ai and we talked a lot about training and inference.

What we didn't talk too much about was data preparation. So this is the very first stage of most AI data pipelines and analytics is actually an analytics type. Workloads are a very critical part of the AI pipeline.

So typically data preparation involves transforming this raw, unstructured data, often sitting within a data reservoir lake into a clean structured format that's appropriate for model training. That involves typically three key steps ingestion. So extracting the data from the key sources, validating the data, so data cleaning, correcting errors, removing outliers, removing extraneous info that's not needed for the model.

And then pre-processing, transforming partitioning and validating the data. If you come from an analytics background, this probably sounds really familiar. So this is one of the key processing engines that's often used for the data preparation part is Apache Spark, uh, which is a key analytics, uh, open source platform.

So let me give a customer example of someone who's doing this at scale. You might be familiar with Snap makers of Snapchat media, uh, social media platform, uh, snap talked about their AI use cases at Google Cloud. Next, uh, in Las Vegas, they have trillions of new user events that are getting pumped in on a weekly basis.

And, uh, their goal is to experience as little a loss rate up as possible. Um, that means exabytes process daily into an ETL pipeline. So, uh, SNAP makes significant use of Spark for their data preparation and cleaning process that feeds into the dashboards, metrics, generations, ml, feature comp, computation recommendations, and ENG engines and ad hoc queries.

All of that underpins snaps daily ba, uh, business. And this is extremely high scale. We're talking something like processing 30 petabytes of data within every couple of hours to compute the tens of thousands of metrics that they need for their active product experiments.

The key point here is that all of that initial processing is done by Spark, uh, that, and a lot of this sits on top of Google Cloud storage as the reservoir for that platform. So what are we doing at cloud storage for optimizations related to data prep? There's three different specific things.

If you look at what's important in data prep, there's two pieces. There's the, uh, the data, uh, the, the peak throughput that's required. And model freshness is, uh, definitely a function of peak throughput.

And then there's the model accuracy, which is a function of the data window itself. So how do you improve these two things? How do you shorten the time cycles from the data window?

And how do you ensure a max throughput? Three different things that we look at within cloud storage. The first is we've built a cloud storage connector.

This is an open source, uh, connector for Hadoop and Spark workloads. So it translates HDFS into object storage, like semantics. Um, and this can be run in three different ways.

One is using the first party Google Cloud, uh, managed Data Pro service, which has optimizations built in. The other is third party services, like for example, Databricks or Cloudera, which make use of that connector and have built their own optimizations. And then we have several customers who are running do it yourself, uh, self-managed environments.

So that cloud storage connector has several years worth of optimizations that are built to make Hadoop and Spark workloads run better on, uh, on object storage. The second, which Marco talked about earlier today was anywhere cache. So anywhere cache is an SSD based on-demand read cache, uh, uh, for, for zones within a cloud storage bucket.

5 terabit terabytes, I'm sorry, per second, uh, that can help improve the peak throughput and thus model freshness. Uh, and also there's lower latency up P 99, up to 70% lower for regional buckets. That gets you faster data cycles and thus a shorter data window.

Uh, the second, and we know that analytics workloads tend to have a lot of repeat reads, so it's actually pretty well-suited for the C case. Uh, the third piece that we talked about in AL'S session right before was hierarchical namespace, which provides again, that, uh, bucket like file system, like data structure for a bucket. And we talked a lot about atomic folder renames.

Uh, we mentioned that it's really good for checkpointing. It turns out another use case that does a lot of folder renames is Hadoop and Spark. So typically in task processing, there's a temporary folder which contains a bunch of objects that are, the processing is then done, that's renamed to its final destination.

As we mentioned, if you've got thousands of objects in a flat name space, that's one individual object copy and delete for each. With h and s, that's a single rename. And so that is natively leveraged by the, uh, by the cloud storage connector.

It can be when you're using an H and S bucket. Uh, so that's a huge benefit. The other is the adx higher initial QPS.

So, uh, because of the optimized structure, h and s buckets ramp faster, so that can reduce your data window. So these are some key innovations that you can use. Uh, and again, it's similar in many ways to what we've seen with AI workloads, uh, to make data prep better.

Let's shift gears a little bit. Go back one side just real quickly, Jack Muller paradigm. Um, on the bottom of your chart on the right, you're talking about model accuracy and model freshness.

Can you sort of give us a quick discussion what accuracy versus freshness means? Yeah, so, so accuracy is, freshness is gonna be once you've actually iterated and trained on the model, um, how long does that stay relevant? And so, and accuracy is, uh, how accurate for the particular use case, how well fit is that model.

Okay. Yeah. And what's the mechanism that you gauged to do accuracy?

Uh, there's a couple of different mechanisms, Uh, evals, sorry, evaluations. Evaluations is one mechanism. Okay.

Yeah. And ground truth to know. Okay.

All right. Got it. Okay.

Um, I'm gonna switch gears a little bit. So that data prep is again, that first part of the AI data pipeline, and that's very heavily analytics related. I'm Sorry to be a hug.

So you're saying if I do this level of data prep, then I'm gonna get better evals as opposed to not. It seems like an awful lot of work to do if you could just use a VS against unstructured data without going through like Hadoop and MapReduce and all that technology. Uh, Well, because most of the data is unstructured, it has to go to that processing center.

It doesn't Have to Right. You know, just a sidebar conversation. It doesn't have to go through ADU or a, a big data process.

I mean, there's a lot of people processing tons of instruction data today without going through Spark or Hadoop directly to model Training. That is true. Well, we've seen, I guess from some of our, uh, customers running this at the biggest scale that they've found efficiencies in running it through Spark.

Okay. I think that's probably the best answer I can give. Alright.

You're right. There are multiple ways to solve the problem. Um, even that uns that that, uh, set of mechanisms can actually benefit from things like anywhere cash for the peak to, yeah, To be fair.

Alright. But you're right, there are actually multiple ways to solve the problem, right? Yeah.

Um, so I'm gonna switch gears a little bit. Let's go back to the example of snap. So we talked about they're part of the AI data pipeline, but SNAP has bigger goals than just the AI data pipeline itself.

What their really, their vision and the thing and what they're executing on is this idea of an AI analytics lakehouse. So in addition to actually having the pipeline of AI data that's being built using training inferencing and such, they also want the ability for their data scientists and researchers to actually run queries on the data and run their own and run their own analysis and experiments and understanding of that data. And so in order to do that, to run those sort of analytics queries, they are actually, uh, have built a Lakehouse that's built on Iceberg, and we'll talk a little bit more about that as the open data format built on top of cloud storage.

So they have training data sets, but they're able to actually do data, explore their, uh, researchers are able to do data exploration and selection on those training data sets. And one of their key requirements is they really wanted interop with ETL platforms. This includes open source platforms like Spark, but it also includes the ability to use BigQuery because they want the optimizations that BigQuery has created, uh, in order to run analytics on those data sets.

So how do you do this world where you want a lakehouse, which has open APIs and an open ecosystem, and you want the ability to run both open source analytics engines on top of your AI data as well as run optimized proprietary services like, uh, like BigQuery. So how, getting to that point, the way that people are starting to build this is through this idea of a data lakehouse. What is a Lakehouse?

It's a combination of structured and unstructured data in one reservoir, usually a unified platform like cloud storage, and then it has a single separation layer, a metadata table format that enables open access to that, uh, open and structured access to that storage layer. This is typically a open table format platform. There's three that are very popular.

Iceberg, Hudy and Delta Lake. Uh, we have seen Iceberg taken off in a lot of use cases with our customers. So that one is really rapidly approaching, uh, uh, heavy growth.

And then that clean separation of the Open table format enables you the choice of which compute engine you want to use in your data. This could be something open source like Trino or Spark. Uh, it could be a streaming ingestion platform like Flink or Kafka, or it could be some, a first party service like BigQuery.

So we're seeing a lot of customers move towards this model where they can run both AI and AI analytics platforms on a, uh, open lakehouse powered by something like Apache Iceberg with cloud storage as the key unified data platform. Uh, uh, layer on the bottom, uh, a couple examples. We're seeing this done in a variety of different architectures.

So with our partners like Snowflake and Databricks, uh, we've seen customers run open Lakehouse. Uh, both of these have proprietary format. Snowflake has proprietary format, but also supports Iceberg reads and writes through their open catalog based on Apache Polaris.

So we've seen customers run that on top of cloud storage. We've seen customers like Micha run on top of Databricks, which provides a unified governance and data layer with their Unity catalog. That Unity catalog also support its Delta Lake as a primary format, but it also supports iceberg via res that feeds into cloud storage.

And then we've seen customers like Uber who've gone more the do it yourself route. They actually primarily use a ap uh, hoodie as their OpenTable format, but below that they have a, an entire AI and analytics pipeline, again, with cloud storage at the bottom. So there's both third party and do it yourself options.

There's also first party options. So, uh, at Google Cloud. Next we announced the BigQuery tables for Apache Iceberg Solution.

So this is a managed iceberg service that sits on top of cloud storage. So cloud storage provides the data layer and it's well integrated with cloud storage. And what it does is together with, uh, cloud storage BigQuery tables for Apache Iceberg enables openness and performance.

So it has a unified metadata layer, which allows you to basically run either a BigQuery proprietary APIs, both read and write, as well as to run iceberg based APIs. Uh, and it has an iceberg catalog to, uh, to, to, to stand in the middle between those requests. So, uh, in the preview today, those, uh, it supports Iceberg reads and, uh, BigQuery announced during Cloud next that they're working on a rest based catalog for Iceberg that will allow both read reads and writes.

So in addition to the ability to do unified access, uh, to your data from different query engines, the other big benefit of putting this all in one place is the centralized management, which enables you to do things like at scale governance, policy management, as well as, uh, cost optimizations. So this is, uh, again, with, uh, cloud storage, you have a couple of different options. First party with BigQuery, uh, tables, as well as third party or do it yourself.

If you look at the pipeline, um, you know, you think of a typical analytics pipeline. You have something like ingest, ingesting the data, storing it in the right formats, analyzing it, and then serving it to the right, uh, the right audiences and the right engines. You can see down below.

Cloud storage kind of sits in and that's where the data ends up in the, uh, and, and being stored with Iceberg as kind of the mediation layer, uh, between services. And where does AI fit into this? Well, typically AI is there in the upper right.

It's one of the, uh, uh, area. It, it's one of the agents that, uh, absorb and query and take the, that data. For example, from a data privacy standpoint.

Uh, it can also be used the other way where query engines read data on top of, uh, ai AI workloads. As we look at this ingest store analytics survey, we start to think of some of the storage problems that people are trying to face. Someone was asking about practical problems.

Let's take a look at some practical examples. Soon. I think we've seen three different problems from our customers as they're trying to build their AI and analytics lakehouse.

The first is fragmentation of the data stored between different buckets, different geographies, different regions and continents, data silos across different, uh, lines of business, often different formats, uh, with different semantics. There's problems related to performance. Uh, AI and analytics are data intensive workloads.

They expect, uh, high throughput, low latency, frequent iOS, um, and they want the data to be as closely co-located as possible with often file like semantics. And then of course, customers are just trying to optimize for resilience, data protection. They're trying to optimize for performance security, and of course everybody's trying to optimize for cost.

So where do we go with all this? I think let's, let's do some very quick examples. So let's talk about a customer like Spotify.

Um, Spotify is a media streaming service. They have a absolutely massive data lake that's, uh, built on top of cloud storage. We're talking exabyte scale with tens of thousands of buckets, uh, split across multiple continents.

And they have many different workloads. They have ai, ML workloads and analytics, and they have of course, data protection and backup workloads as well. So they're really having two distinct challenges in this building of an AI analytics lakehouse.

The first is just analyzing and understanding where their data is and what it's doing. Uh, is it in the right locations to be close to the storage, uh, sorry, close to the compute. Uh, is it really hot data that's costing them a lot of money?

So if they're trying to figure out and understand the, the topology of their data, the second problem that they're trying to solve for is co-locating the storage and compute with minimum possible downtime If it's located in, in, uh, one bucket, but they have compute in specific zones, how do they reach it with minimal cost? So we already talked about these two solutions, so I'm not gonna go into depth about them, uh, in previous sessions. But really storage intelligence and the insights dataset solution allows them to ask the right questions of where their data is, what it's doing, how hot it is, and that's done through these daily snapshots of object bucket metadata.

In this use case, it's important 'cause there's no impact to the data path in reads and writes. And then for the problem of co-locating the data, there's two ways to solve that. Um, one is you can bring the data automatically to the right zones using anywhere cache.

Again, it's that zal solution for easily co storage and compute leverage with multi-region means you can have compute optionality and g redundancy without actually sacrificing your performance or cost effectiveness. The other option, which we talked about in the earlier sec, uh, section is bucket relocate. So you can actually move the bucket physically or the data from it from one location to the other.

Uh, and that'll allow you to co-locate your data as well. So there's, here's a couple of ways In this example of how you can ask the right questions, understand your data, and then co-locate it with either anywhere cache or something like bucket relocates. The other problem is for customers that are actually lifting and shifting or migrating from, uh, on-prem services or from other clouds, uh, or they're refactoring their analytics workloads, we'll get, we'll start with the first example.

If you are lifting and shifting a Hadoop workload, you often expect file like performance and governance. And so one solution you can do to solve that, again, we talked about that a little bit already in the past, uh, in the past is h namespace. So it'll allow you to get those atomic folder renames for your spark workloads as well as your checkpointing workloads.

And it'll also allow you to do fine-grain IAM at a folder level. Um, the second key issue is what if you're trying to refactor your analytics workload? You've got a database or streaming analytics platform, it's built on top of a different storage platform.

You want to move it to object storage for cheaper cost higher scalability. Uh, for that you need much better performance in dependable, right, semantics. Someone was asking about an example for like where rapid storage might work in practice.

This is a very good example. If you're trying to do a re-architecting of a database or a streaming analytics platform to be diskless or to sit on object storage, rapid storage is actually a great solution for that because of the app dependable rights and the ultra low latency and high through characteristics. Uh, so rapid storage, it's in preview, but that's actually a great solution for going after these particular workloads.

Um, so I'll leave with a customer example of putting this all in one place. So Two Sigma talked on stage with us at Cloud. Next, they've built a massive, massively scalable research data platform for the financial markets.

And really it's a single data platform with cloud storage is the key reservoir layer. Um, and then a combination of several different features in order to ensure the right levels of performance as well as ease of use. So they make use of auto class, which is a feature you can enable in a bucket, uh, which allows you to automatically transition the storage class of an object based on how long it, uh, the last time it was used.

And so that enables them to gain efficiencies at scale and cost. They use hierarchical namespace for the file like performance, uh, and the higher QPS that they need for their workloads. And then they've built this layer of anywhere cache, which allows them to, uh, provide high performance for their workloads, but without actually needing the operational complexity of managing a high performance product themselves.

So all of these put together means that they're able to really serve their customers with the best possible storage while doing the minimum amount of work that they need to do themselves so they can focus on running their business. I'll leave you with one last slide, which is just some of our best practices around data lakes and analytics workloads. So from a resilience perspective, um, if you need a business critical data set that has to tolerate more than a regional outage, uh, dual region is a great solution for that.

It allows you to replicate data between two regions automatically. Turbo replication speeds up your, uh, your, your, your replication process even further. From a performance standpoint.

If you have a workload that's repeat reads and needs really high throughput, uh, anywhere, cache is the easy solution to do that in an existing bucket without making any changes to your code. Um, if you need workloads to fast and quickly ramp, uh, and you need that atomic folder renames, again, spark workloads and or checkpointing workloads are a great example, then you should enable hierarchical name space when you create a bucket. If you have, for example, a streaming analytics, a database workload or AI workloads that would benefit from zoneal object storage and ultra low latency, and you're willing to work through a preview feature, I'd suggest looking at rapid storage and sign up for the preview there.

Uh, from a management perspective, uh, storage intelligence gives you the ability to really understand and analyze your data for cost performance reasons. If you actually want to move a bucket from one place to the other, uh, while minimizing downtime, use bucket relocate. And if you want the optionality of your data all in one continent, but zoom in and have your data co-located with storage and compute with as little work as possible, use multi-region anywhere cache.

And we talked already about auto class as providing automatic setting of storage classes. Um, so that's all I've got. Happy to take any questions.

Again, my takeaways are analytics is part of the AI data pipeline and analytics and AR are being used together by a lot of our customers in their data lakehouse. So you talked about luster earlier. They, your group talked about luster earlier, but it's not showing up anywhere on this chart as a another option.

Where where does that fit within this? What appears to be is positioning, um, piece. And then the other one I'll add to that is that you have something that's a, um, hyper disc, ml, hyper disc, and is that kind of like taking a back seat to all of this?

So those are two questions. Yeah. So I think the way to think about it is cloud storage is the reservoir where most of the data is stored most of the time.

Then there are times you need a high performance solution, uh, in order to accomplish particular use cases. So luster is a great example for the training and Checkpointing series. If you need extremely high throughput latency and posix capabilities, move the data from your, your data reservoir to where that performance is needed and use it for, uh, and then you can use luster for those solutions.

Inevitably, the data will make it back to the data lakehouse. So cloud storage is where it will sit kind of at rest. Uh, hyper dis ML is a read only solution, high throughput.

Uh, so it will be useful for some of these workloads, um, but again, has a different set of characteristics not right based so, you know, potentially good for training. Uh, whereas Luster is more of a read write solution. So again, each solution has its own specific use cases.

The takeaway here is that the data lake itself is ultimately cloud storage is a solution. There, I think Was gonna briefly, the hyper risk ML is, is well suited or can be well suited for inferencing for AI workloads, right? Right.

The training and checkpointing and inferencing cloud storage and managed luster, and then potentially HDML for inferencing for, um, AI workloads. It's not the exclusive use for it, but that's in the context of AI workloads. That's where you would think about HDML, not H-T-M-L-H-D ml.

Got it. I there was a, there's a question just kind of deploying on the hierarchical namespace. Is that, is that just hierarchically naming them or is there actually a direct restructure associated with it?

There is actually a direct restructure. So h and s has adds a new resource to the bucket called a folder. A folder can contain other folders or objects.

Okay. I mean, in the end it's still object storage. So this is just a metadata layer.

Like it's not, but it is actually creating a hierarchy. Okay. Whereas in a flat bucket, there's only folders, sorry, there's only objects.

And those objects, when you see a folder in the ui, a C or CLI, that's just a virtualization, uh, uh, using prefix. Okay. So it is a real folders plus objects.

Can you, uh, can you, uh, use like RAC, like an Okta for the h and s folder structures? Yes. So we have something called managed folders, which is a separate, uh, entity, which allows you to assign IAM grants at a prefix level.

Okay. But important difference though, it can be used in both h and s and existing buckets. Uh, managed folders exist in both in an existing bucket, it applies to a prefix and it's fixed.

Right. But I'm not concerned about like a structure or a higher knuckle structure. Like I don't want anybody to get to this versus this.

These are, yes. Okay. So you can do that.

The difference in h and s is when you do that, you, you assign a managed folder, it attaches to the folder. Now if you move that folder around the pr, the, uh, the IM grants go with it. So basically it adds onto the functionality of what's in the existing product.

Okay. But you can do fine grain. Im grants today in cloud storage.

I knew that. I just was wondering if you could do, Yeah. So it's, it's additive you can think of as additive and additional benefits.

Yeah.