Getting to Know the Unsung Hero of AI Infrastructure with Solidigm

We're gonna talk about the unsung hero of AI infrastructure. This is not the unsung hero of AI infrastructure. This is us.

We're your humble, uh, presenters today to talk about it. We're gonna talk a lot about storage. That's what solid IM is.

That's what we do. We make SSDs. We'll get into that in a bit.

But right up front, uh, my name is Ace Stryker. I am Director of Market Development at Soy. I have been with the company since it was born a little over three years ago.

And we'll talk about the origins of the company as well. And with me, uh, who you'll be hearing from a bit later is Mr. Scott Shaley as well, our Director of Leadership Narratives.

Okay. I would like to start with a question for the group. Does anyone recognize this image?

Can anyone tell me where this comes from? Any guesses? Pacman?

Well, you're Pacman's outta company, so maybe it's a hard drive. Maybe not in this case. So this is actually, this is, um, this is a test.

This is a test, uh, that's created and published by a group called the Arc Prize Foundation. They're a nonprofit, and what they do is they publish a test that's, uh, intended to measure, uh, an AI models, uh, problem solving capabilities. And specifically, uh, are we approaching our approximating artificial general intelligence?

Uh, and so they publish these test sets and they give these problems to models. These are problems that are relatively easy for humans to solve, but that AI models still struggle greatly with. So in this case, um, you know, this is kind of the prompt.

On the left, you have two big shapes with gaps in them, lots of little shapes, and the, the, the goal of the task is to fill in the big shapes with the little shapes while ignoring kind of the noise, the gray pixels as well. Uh, and so they've, they've been at this for quite a while. Uh, this is from the second version of the test, which is called ARC AGI I two, which was just published last month.

A little bit of history on this test, and I promise I'll, I'll tie this into storage here in a minute. The Arc Prize Foundation published the first version of ARC a GI in 2019, and it was, uh, they provide a, a, a training data set that model developers can use, and they provide a couple of, uh, validation or test data sets with against which they measure results and actually make a judgment on whether these models are successfully approaching human levels of problem solving. Finally, late last year, after five years of version one being out and available in the market, uh, an open AI model called O three, uh, surpassed the human level of, of problem solving on this test, which was a big deal.

Uh, a lot of press around it. Some people very enthusiastic about the notion that, Hey, we're actually getting toward levels of artificial general intelligence and human problem solving. And the benchmark for this, by the way, is, you know, they, they, they, they take the same questions and They actually give them to humans upfront.

And then they, you know, they test, I think 400 or so people. Uh, and, and on this test, uh, the humans scored, you know, 75% or so. And so, and this model came out and scored more than that.

It was considered passing the test and conquering RKGI, right? But that wasn't the end of the story. 'cause very quickly, the ARC Prize Foundation turned around and said, well, there's actually much harder, uh, questions that we can ask that humans are still quite good at answering that.

AI is still quite hard at answering. And so these are some of the leading models along with the human panels. And what you see in the first column there is how some of these leading foundation models did on version one of the test.

And second column arc, a GI two, is how they're doing on the latest version. So you can see things like the O three model that got 76% on the first test, scored 4% on the second test. So it's quite clear that there's still a big gap.

'cause you look at how the humans did on these, right? A panel of experts scored almost a hundred percent and an average across the 400 or so, you know, PhD students and others that they were able to pull into the lab and pay a few bucks to do these puzzles, was able, was able to solve about 60%. So there's still a huge gap, as excited as we are about the capabilities of cutting edge models and their ability to pro, uh, solve problems, tests like these reinforce for us that, hey, we're still pretty darn far from artificial intelligence, artificial general intelligence, right?

The other thing that the Arc Prize Foundation did with version two of their test was they introduced an efficiency metric. And they said, Hey, in order to beat test two, not only do you need to score higher than the humans, but you need to do it more cost efficiently than humans could. And they meant very literally, it cost us this much to get folks into the lab and fill out the puzzle sheets, and it was like $17 a task.

And can we do better than that, you know, with AI models in terms of the cost per million tokens and, and kind of typical, uh, calculations. This is the leaderboard as of yesterday for RKGI two. So the Arc Price Foundation has now said, Hey, world, you have until I think November 3rd, this round closes.

And if you can beat the human panel's performance, and you can do it at a, a sufficient sufficiently low cost, you will win the grand prize, which this time around is $700,000. And so a lot of these model developers now are, are marching toward this, this is the, the North star. This is where we need to go to improve our models, uh, reasoning and problem solving capabilities, right?

But as you can see, as of yesterday, everybody's pretty much here on the X axis. Nobody's, uh, rising anywhere close to this is the human panel up here near a hundred percent. As I mentioned before, 60 ish percent was the threshold for the average human response on this.

The best performing model today on a KGI two is the O three low preview, uh, which scores 4% at a cost of about $200 per task. And the, the, um, the goal that the prize foundation has set in terms of cost, uh, what they wanna see to award the grand prize is around 40 or 42, uh, cents per task. So quite a long way to go, right?

What is model accuracy? What is that y axis metric that the Arc Prize Foundation cares about? And how does that tie into storage?

This is one definition from Google. I found it to be a very useful one. When you talk about a model's performance or a model's accuracy, you're essentially measuring how many things did I ask it to do, and how many of those did it get right?

And it's a ratio, it's number between zero. And one is how you can measure an AI model's accuracy to go a little further correct classifications. You can break 'em into true positives, true negatives, whereas total classifications, you'd also have to, in the denominator put the false positive, is the negatives are where the model got it wrong, right?

Uh, and so this is, um, applicable, of course, not just to a test like RKGI two. It's easy to measure in that way because the, the private data set that's going to be used to score these models and determine if anyone's won a prize is 120 questions. So how many of those 120 did the model get?

Right? That's your accuracy, right? But you could also measure it across any use case you can think of from, you know, AI models and traffic cameras to, to predictive maintenance on factory floors to, uh, you know, my kids asking Siri to come up with a funny joke, and how many of those actually land, right?

There's as many measures of accuracy as there are use cases, but fundamentally, it's how often does the model do what I want it to do, or what I expect it to do, right? In terms of improving AI model accuracy, we've done a review of the literature, kind of looked across a bunch of different, uh, published papers and, uh, uh, blog posts and stuff. And it turns out there's a lot of tools in the AI model developers toolkit to improve model accuracy.

This is a, certainly an overly simplified view here, but these are some of the key levers that as a model developer, you can pull to crawl up the y axis on that leaderboard, right, that we just looked at. So from a training data perspective, you can feed more data into your model, you can use higher quality data, and by that we mean stuff that's been pre-processed and irrelevant, stuff filtered out, and, you know, stuff is annotated and de-duplicated and kind of standardized. Uh, there's work to be done in feature engineering where you're actually selecting, okay, what are the relevant or most useful characteristics within the data for the model to focus on as it does inference?

And then when you're actually training the model on that data, there's a whole bunch of things you can do there. And I'm not an expert on, on this stuff. I'm not a data scientist, so I'm gonna spare you my attempt to go into details on these things.

But, uh, you know, fundamentally upfront, you can adjust hyper parameter tuning. Uh, you can train the model for longer on the same dataset, which sometimes, uh, results in higher quality outputs, but can also create problems in the form of overfit and some of these other things. Regularization, cross validation, you know, these are tools, uh, in, in the toolkit, right?

And then of course, there's rag where you're taking your train model and you're plugging it into additional data sets, which can help greatly with, with output quality or accuracy for the purposes of the ARC Prize RAG is not a component, right? This is purely a question of model capability, but in the real world, there's all sorts of applications where RAG is a very, very important part of increasing the usefulness and accuracy of a model. Okay?

So this is a quick survey of what folks are working on as they strive toward the ARC prize and as they strive towards similar goals outside of the, um, the fund, you know, cash prize. But in terms of in general, making AI models more useful and valuable, as we looked across, uh, the various approaches here, the one that emerged at the top of the list as the most frequently recommended, uh, single thing you can do to improve model accuracy, it turns out, is today use more data. Right?

The more data you train your model on, the better, uh, the outputs are going to be. That's widely studied and understood. There's a strong correlation there.

It's certainly not the only thing you can do to improve model accuracy, but it is across a bunch of these different, um, uh, sites and, and published papers and stuff. We looked at, it often showed up as, you know, the number one recommendation or the top recommendation or words like the lifeblood of machine learning, right? So today that is still, uh, uh, a primary driver of model, uh, quality and accuracy is the training data set and the quantity of data in that set.

That of course, is easier said than done, and that's where soy comes in, right? This is a survey, a global survey done by a company called Digital Realty, where they went around and they asked enterprises across the world, what's the biggest challenge you're facing in rolling out your AI strategy? And they named a whole bunch of things.

If you're like me, you might've expected at the top of that list to see, you know, GPUs are really expensive or they use a lot of power or, uh, something kind of compute related. But the, the top, uh, answer was actually around the lack of data storage required for massive data sets of the sort that we're talking about that are needed to improve model accuracy. Related to that, uh, there was a computational power element, uh, and a, and a data center space element.

We've highlighted those specifically because those will connect directly into what we view as the solid, IM kind of value proposition for ai. Um, but all this to say, it's one thing to acknowledge that to go further with AI and to do more with our models, we need to show them more data. It's quite another thing to actually build out the infrastructure to be able to do that, right?

And that gets us to kind of the central thesis of today, and we'll come back to this again and again, which is that every watt and square inch in the AI data center counts. A lot of times we're very focused on the compute resources, and rightly so, views are the most expensive and the most power hungry, uh, single component in an AI cluster. But when you're looking at clusters on a scale that's needed to really advance the frontiers of AI models and their applications, uh, you're going to involve a lot of storage and you're gonna involve a lot of storage, not just on the training side, but also on the inference side, which we'll get to, and we'll talk about that in some detail.

Uh, we'll show you some modeling that kind of reinforces this idea. Um, but choices that you make about your storage infrastructure can have significant impacts, not only on the cost and the energy consumption of your storage infrastructure, but also on the rest of your AI cluster as well, in terms of enabling more, uh, GPU scaling, for example, and, and other cool things you can do, which we'll get to. So that's the, that's the, the, the opening salvo here, and we'll, we'll come back to this a few times, explain it further, defend it with some, some data and some findings from recent work we've been doing at soy.

Okay? Alright. Uh, first question, uh, that I got from one of our delegates, I won't name names, uh, in this room was how do you pronounce the company's name?

So I wanna spend just a minute here. It is soy, it's a combination of solid and paradigm, but I don't blame you if you, if you're, uh, thought otherwise. Certainly the first time I ever saw it in print, I thought it was solid IgM, some kind of an acronym with memory at the end.

Mm-hmm. Uh, but what we are is a, is a company that makes SSDs fundamentally, and we are born from the former, um, storage and memory group at Intel, which was called NSG or Non-Volatile Memory Solutions. So if you've ever seen an Intel SSD, that's our heritage.

And a lot of folks, myself included, were working on this stuff at Intel prior to solid. Uh, today we are, uh, so, so, so Intel sold the storage group to a Korean company called SK Hynix, which you also might be familiar with. They do a lot of business in, in, uh, SSDs and high bandwidth memory.

So they are the owners of soy, but SOY is a US-based subsidiary with headquarters in Rancho Cordova, which is just up the road right outside of Sacramento. So when we talk about where we come from, what we do, yes, the name is new, uh, but there's a whole lot of, um, there's a whole lot of tradition behind it and there's a whole lot of innovation that the teams, uh, at Solid I more directly involved in. If you look across the history of data storage and infrastructure, uh, there's a lot of firsts on this list, Right?

Uh, first Q-L-C-S-S-D, um, uh, SLC, we have, you know, early work in Nor Flash. We have software work in the form of rapid storage technology or RST, which some of you might be familiar with. We have, uh, you know, folks that were involved in very early versions of the, the industry-wide specifications that dominate today PCIE and NVME.

So, uh, a lot of great experience with a relatively new name tacked on it, right? Okay. So now that we've kind of teed it up and given you a little bit of information about who we are and where we're coming from, I wanna walk you through the agenda for today, and then I'm gonna hand it over to Scott to take it away with our first, uh, topic.

But we're gonna do a few things today. We're gonna talk about AI solutions in general and how some of, uh, the different, um, uh, storage types kind of map onto those solutions. Talk about the product portfolio.

We're gonna deep dive a little bit into what we call the AI data pipeline. What are the steps involved in doing AI work and why does storage matter in each of those? Uh, we've got a really cool section here in the middle that I'm excited about.

We have joining us for the day, Steen gr, uh, from Metro ai, their CEO and founder, uh, and we're gonna talk about data intensive inference and some cool new possibilities being unlocked with great storage. We have a live demo to show you, uh, for that part of the presentation. Then we'll come back and Scott will finish up with a discussion of, uh, the high density stuff.

Uh, you know, those Lego kits we gave you. Those are, those are modeled after 122 terabyte, uh, SSDs, which is a, a whole lot of storage in about the size of a deck of cards, right? And so, um, Scott will talk about how that plays out across many devices and in AI settings and otherwise.

And then finally, we'll wrap it up with some recent stuff. We announced this just at GTC, uh, last month. Uh, we have, um, some, some liquid cooling innovations that we've partnered with Nvidia to deliver to the market, uh, that are gonna enable future generations of storage and, um, do some really cool things that have not been possible to date, uh, with the storage inside of an AI server.

So that's where we're headed.