How Great Storage Enables AI Performance and Efficiency at Scale | The Six Five Summit

Welcome back to the six five Summit 2024. It's day one, and we are talking all things ai, regardless of the track, AI is on everybody's mind. And this year is about two things.

It's about enterprises capturing the AI value, but also the extended build out of AI in the data center and the data center edge. Dan, we're talking AI again. Isn't that great?

Yeah. Well, you'll remember when we were coming up with a theme for this year. It was all about really being able to help companies bring value.

Yes. Right. There was so much kind of talk about this, this, this CapEx, this big capture, all the dollars were going towards GPUs, GPUs, GPUs right behind this.

There needed to be so much more Pat, and that's what this event AI unleashed here at the six five Summits really all about. No, that's right. And, and Dan, I know I use this a lot, but I talk about the quadrangle, right?

Quadrangle of compute, which says you really have to have balance system between compute, uh, memory storage and networking. And if anything, any one of those gets out of whack. You can't bring full value that the system brings, that the quadrangle brings.

And data center storage is tremendously valuable with the amount of data that comes in for ingest. You're training it, you're performing rag on it, you're sending results back, you're storing it for long-term storage. It becomes very, very important.

We have to have happen to have with us two folks from soy, which I hear the market share leader in data center. SSDs, great to see you, both of you. Okay.

Morning guys. Yeah. Thanks for coming in.

Yeah. So, uh, Dave and Greg, great to have you on the show. Maybe we can hit first off, uh, tell us what soy does.

I talked about it a little bit, but, but let's hear a little bit more. Yeah, well, a little bit of background on soy. We were, uh, acquired by SK Hynes back in 2020.

Uh, we were originally the, uh, the n products group right at Intel. And so we came over about that timeframe. We closed the deal, uh, a little bit more than two years now.

Uh, one thing that does make us unique, kinda like what you were saying is that, uh, as opposed to kind of everybody else in kind of the and market, we are laser focused on data center storage. Uh, we see that as the, uh, the biggest market in all in and the fastest growing market at all in, and, and it's gonna keep going. So it's, it's really our focus for that reason.

Well, David, uh, let's talk about something that's probably near and dear to the hearts of just about everybody that's involved in ai, and that's scalability, right? And this can come through a couple of different lenses, you know, of course. One is you have companies, uh, our data's showing massively involved in POCs right now, right?

Yeah. So it's kind of the idea to implementation scale, and then you've got scale, like things like, uh, you know, infrastructure and power, and that's another thing that's going on. Talk about the challenges, though, that you're seeing here at Soddy as it relates to scaling ai.

Well, yeah. So obviously the, uh, the scalability of continuing the AI boom and AI development, uh, through the rest of its decade, right? It's gonna be a huge challenge now at all.

At Heim, we're really looking at that challenge as more of an opportunity, you know, for us specifically, you know, which is kinda cool. Uh, but whether we're talking about the power limitations that you talked about, the performance limitations, the, the size of the data sets are gonna be used inside training, you know, all those are really seen as, uh, scale challenges for being able to, you know, continue this AI development, uh, whirlwind, right? That we've all been on for the, you know, past two years or so.

Let's talk a little bit about the role that storage plays in this. I mean, it's just, it's just storage, right? Yeah.

Well, you know, we started with these huge, uh, scalability challenges, right? We're talking about, okay, you're gonna build 50 nuclear power plants for the next couple of years to fix the problem. It's not gonna be a single answer.

It's really gonna do this, right? This has to be a multifaceted solution, right? And, you know, transitioning from HDDs to data center SSDs is gonna be a key part of that solution.

So, you know, as an example, if we look at, uh, power, uh, you know, latest projections are that it's, I think by the end of 2030, it's gonna take, um, you know, data centers are gonna take up something like 20% of the overall global power Grid. Like it's like off the Charts, off the charts, right? Yes.

And you could solve that by building a bunch of nuclear PowerPoints, but we're probably not gonna be able to get that done. So one, you know, and that data's pretty well known, but something that's not as well known. You, you talked about, you know, why is storage important, right?

There's more and more data papers being, being written. About 30% of that data center power is taken up by the storage. So it's not just these, you know, big heavy and these Hard drives.

Yeah. To be clear, Okay, yeah, they're hard drives today. Uh, they're about 90%, right?

90% hds in, in the, uh, data centers today. Um, but it's not just these big, uh, power hungry GPUs. People don't realize that they're being fed by a massive amount of storage to keep them going.

And so if we can make a dent in that huge, uh, storage bucket, we're talk really talking about a, a major impact, uh, to this power and scalability challenge. Yeah. I think we've seen, uh, the growth of the storage and, you know, pick your, you know, pick your adventure.

We can talk about parameters, we can talk about data points. Yeah. 6 trillion, uh, data points.

I'm sure there's a way to convert that into, into parameters, but, you know, it's funny, in, in, in the green room in the run up to this interview, uh, you had shown me a piece of data that, that, and Dan and I get hit with power stuff day in and day out, which was there are actual grids out there in the United States that have either no power left or some absurdly small percentage left for data centers. Can you talk a little bit about that? Yeah, yeah, absolutely.

The, and it's, and um, there's an article almost every other day in the big, big publications about it. But, you know, take Northern Virginia for example, which is a huge hub for data centers. Huge.

Yes. 2% headroom left in their grid. Uh, and so the, the data center build out is so strong and so fast that now it's bumped up against that limit.

And it's also now the grid is the limiter to the build out of those, those data centers. And right now they're, they're primarily being built out for these, you know, large AI deployments. And just so everybody understands, that's just not the data centers competing with that.

That's also potentially power required for EVs, Future power required for EVs, For homes. Mm-Hmm. Uh, for, for new buildings that are, that are built here and and for charging your IPhone.

Yeah. A little bit Time. Gotta a little bit, A little bit that.

Yeah. But, um, yeah, so a lot of choices there. And then you look at how long it takes to spin up a new power source Mm-Hmm.

Whether it's solar, nuclear, coal, gas, something else, an extra, it, it's typically those are five year projects. Exactly. So something has to give here.

That's right. And you get into, uh, some pretty interesting, uh, spreadsheet exer exercises on what they do, but it's, it's clear that moving from hard drives to low power, high performance SSDs at, at, in those areas in it's almost a no brainer. Yeah.

But it's also, and it's the high density aspect of it too. It's the main driver, you know, if you, uh, how big, how big, well just, so we'll maybe talk about it in just, just sec. Okay.

But just to close on this power angle, I think it's, uh, you know, the data, if you really go look at rack level replacements, et cetera, we're talking about like a 60 petabyte AI storage rack. Yeah. You can get an 80% power reduction by converting from HDDs to high density, high performance QLC SSDs.

So now you're talking about 20% of the overall grid, right. You know, being taken for data centers, 30% of that data center storage. Now you can reduce that number by 80%.

We're talking about really big impactful numbers now. So there could potentially be, uh, big advantages from not just yeah, those new AI servers, but also potentially looking at your entire fleet of, of servers out there. Yeah, exactly.

Okay. And that's just power, I think, um, you know, the other key driver of the scalability is gonna be the performance requirements, because, you know, what we're finding, what we're hearing from customers, right, Greg, is that a lot of the GPUs that they've all been building out with really starting to get underutilized because of the under appreciation of the, the data being fed into the GPUs is effectively kind of starving the GPUs right now. So there's an under utilization problem that's happening, and that's really slowing down the benefits really that we're getting out of ai.

Great Point. The GPUs as well as the data scientists who use the GPUs, who Yeah. You know, as everyone knows now are, you know, hot, hottest commodity employees out there.

And so, you know, it's almost too expensive not to convert from hard drives to SSDs. Uh, even though the, you know, at a CapEx level, it might look a little bit, you know, SSDs look a little bit higher than HDDs, but when you put put the, uh, total solution together, it's, you know, all the biggest new AI data centers are designing in these high capacity, uh, SSDs. And you'll hear me talk no end about the, it's the math, it's the economics, and we actually have some real world problems we're gonna try to solve.

And of course we will figure those out first. Meaning, you know, we'd spend a lot of time here talking about power and availability. The first thing we'll solve at sort of any cost is gonna be enough power to make sure that we can continue this build out.

You've seen what the market for GPUs grow to. We've heard numbers as big as 400 billion on the tam. Yeah.

It's huge. And of course, every company, I I always talk about the deflationary aspects of tech right now, as companies need to figure out how to keep hitting their numbers, they need to keep doing it more and more efficiently. Yep.

AI is an enabler of that. But in the end, the economics come down to things like what you're doing with SSD. It comes down to saying, how do we get down from four racks to two racks?

How do we optimize power in those racks? How do we maximize utilization of GPUs or, or more efficiency from that same GPU? Yep.

So this question is the, this is the money maker for, for you and d I'll throw it your way, is, is, you know, oftentimes this particular category, uh, can be a bit treated a bit as a commodity. Uhhuh what you've said today doesn't sound anything like a commodity, but what's your sort of winning formula for, you know, being able to grab the AI market right now? What's the solid I winning formula competitively performance wise?

And of course I spend a little time on economics. Well, We're, I've definitely seen a transition even, you know, I, I think Greg and I have both been in this industry going back for flash for 30 years and uh, you know, there's been times where it's, you know, comes and come and goes with, uh, a commodity look and feel to the, to the market of course. And we're coming out of a pretty heavy downturn over the last couple years.

Uh, but we're seeing an excitement now that we've never seen before. And this is really just over the past couple of quarters because people are really, you know, now really appreciating the importance of the bottlenecks that are created by data center storage. And it's now, it's as that data is transitioning from being cold and just sitting there not, not being used, not being read to now being more and more data getting generated more and more, but wanting to be read and utilized all the time.

Yeah. We're getting tremendous pull from our customers already. So the, you know, the products are really, uh, you know, we've got the kind of in the right place at the right time right now with our, our high performance high density QLC.

Um, and you know, we talked about the 80% power reduction just to finish that thought on performance. One way we, you know, we kinda look at is this GPU utilization factor, right? It takes, you know, one of our P 53 30 sixes to keep five GPUs fully utilized more than 90% the inverse, if you were gonna, it takes, um, I think eight HDDs to keep one H 100 fully utilized.

So we're talking about 40 to one factors. These are not incremental improvements that we're talking about in power performance at the data center level. These are monumental changes, right?

That, that really are gonna be the key driver. Well it's very interesting. Um, you know, Greg will appreciate this.

I'll play marketer for a moment, but I'll throw one more thing. You have to, yeah, no, I can't help myself. Um, so focus, you know, a lot of times, and one of the things that, you know, in a recent briefing I was uh, you know, attended from solid dime I thought was very interesting is pretty much everyone that you compete with has a broad focus across consumer data center.

Um, you've chosen to completely focus on this problem and in the you kind of the AI era. Yeah. And the problem related to data and enterprise data and data centers, it's unique to have a company that's saying this is the one thing and we're all in.

How big is that to your ability to execute on ai? I, I think it's huge. I mean, and you know, we have, like you said, all of our engineers working on with our customers, you know, deeply with our customers, understanding their workloads, understanding what they see in the future.

And you know, it allows us to tune our drives to meet their workloads specifically. Right? But it's also enabled us to focus on what's this, what's this next wave of, of kind of what I would call data center evolution, which is not hard drives going away ever, but some of them, you know, giving away the flash.

And so we've been investing in the highest capacity flash drives. So we have 30 and 60 terabyte SSDs. We launched 'em last year.

We're way ahead of our competition, you know, from that perspective. Uh, and we've been investing in, uh, QLC technology to make those drives, uh, very affordable, uh, compared to e existing flash technologies. We're actually in our fourth generation of, uh, QLC as we speak.

Yeah, probably just to close on that thought, um, you know, when we talk about high performance, high density, the key driver is this QLC and the multi-level cell technology. And this is where we're storing more logical bits for every single flash physical flash shell that's sitting there. We get to QLC, we're, you know, 16 level, 16 different states, right?

Yes. That we're storing in the physical flash cell. That's the only way to really, you know, bring this value to the customers.

And um, that has been our focus. That's why we're on our fourth gen right now in the data center. And comp is really just trying to still commercialize really their first generation.

Yeah. Well, Greg and David, I wanna thank you so much for being part of this year's six five summit. I hope you to have you back and, uh, we'll be watching closely.

We'll be watching soy and we'll be watching this AI evolution and revolution. So, uh, stay tuned and join us again, if you wouldn't mind. Well, Thanks for having us.

We appreciate it. It's been fun. Alright everyone, we are here.

It is six five Summit 2024, day one. We are in the cloud and infrastructure track you just heard from Dyne. But stay with us for all of our content here this year.

Six five Summit is bigger than ever. See you soon.