UT07x05: Efficiently Scaling AI Data Infrastructure with Ocient – Utilizing Tech

As the volume of data supporting AI applications grows ever larger, it's critical to deliver scalable performance without overlooking power efficiency. This episode of utilizing Tech sponsored by solidi brings Chris Gladwin CEO and co-founder of scient to talk about scalable and efficient data platforms for ai. Welcome to Utilizing Tech, the podcast about emerging technology from Tech Field Day part of the Futurum Group.

This season is presented by soy and focus on the question of AI data infrastructure. I'm your host, Steven Foskett, organizer of the Tech Field Day event series, and joining me today from Soy as my co-host is Janice. Welcome to the show.

Hi, Steven. Thanks for having us back. You know, Janice, um, you and I have spent a lot of time talking about a lot of storage topics, but one of the things that we come back to a lot is energy efficiency and the question of power consumption when it comes to AI data infrastructure.

Yeah, absolutely. At the beginning of the year, it was really just around ai, how about ai? What's the GPU?

How do I work with the GPU, right? It was all about the, you know, instances of the GPU, but now we're kind of, we're kind of turning that corner and we're starting to see, uh, many organizations looking at not just, you know, uh, performance, but also really looking at power and efficiency and scalability. So we are super excited about the opportunity today to talk about that a little bit more.

And, uh, I know you wanna introduce our guests, so I'll let you do that. But, uh, yeah, we're excited to dive into that specific topic. Yeah, absolutely.

And I think this is one of those things that comes up again and again when people talk about ai, you know, the naysayers are constantly talking about the fact that, uh, you know, it uses all this power and, you know, you, you know, people aren't consider considering that they're not thinking about that. Well, one company that is absolutely thinking about that is, uh, scient. And so it's very, very nice to have here on the podcast today, somebody I've known for a long time.

Uh, Chris Gladwin, uh, CEO and co-founder of ent. Welcome to the show. Uh, great to be here.

Really look forward to this conversation. Great to reconnect with you, Steven. I know we work together, you know, over the past years, and, and here we are again.

So tell us a little bit more about, um, what is ENT and, and, and which part of the AI data infrastructure stack do you play in? And then we can talk a little bit more about the be, you know, the, the, the question of energy efficiency and, and and scaling performance. So, OC N's, a company that's developed a new, uh, software architecture for analysis of large data sets for complex analytics, uh, always on workloads.

Um, and, and, and in doing that, we've, we've really, we have focused on, uh, efficiency in general, both, you know, price performance as well as, uh, energy efficiency. 'cause they go hand in hand. So what we're doing is providing solutions using this, this new architecture for this emerging group of, uh, ultra large data analytics, uh, requirements.

Uh, and, and as you know, you and I have seen, we worked together at my prior company, uh, Cleversafe in the kind of large scale storage realm. Today's, you know, ultra large, you know, analytics workloads just become tomorrow's normal and eventually that's what your phone does. So it's really important to get this right.

We really have to, as an industry kind of be on the right trajectory of efficiency. Um, o otherwise, uh, you know, we're gonna have some real problems, you know, powering this future without being efficient. So Chris, let's, let's dive into that a little bit.

Um, you know, how, how is OCN specifically looking at addressing, you know, some of these challenges? Well, it really comes down to focus, um, a as with a lot of things you see in information technology, you know, capabilities just keep growing and growing and growing. And it really is like, what are, what are the engineers building all these capabilities really trying to do?

What are they trying to optimize? And, um, you know, you, you've seen, you know, example after example, like when, when you take a, a group of talented people and you give them the, the mission of like delivering better performance, improving cost, having new capabilities, they do it. Um, and so, you know, the other thing that is really important is the focus on efficiency.

So what, what we're seeing now in the industry is, um, all these new capabilities are coming out in the, in the realm of artificial intelligence and large scale data analytics and all these really amazing capabilities. But so far there hasn't been a focus on doing that, not only in a way that's cost efficient and performant, but in a way that, uh, you know, really is energy efficient. I, I was fortunate enough, um, one of the jobs I had early in my career is I worked at Xena Data Systems, um, in Chicago.

That's why I originally moved to Chicago when it was the largest portable PC maker in the world. And that was the very, very beginning of focusing information technology on energy efficiency. 'cause back then you had a battery and, you know, batteries back then weren't that great, but yet you still had to find a way to get, you know, 2, 3, 4 hours of battery life.

And so once you had a kind of an engineering team that said, look, we gotta figure out how to make this more energy efficient. So, you know, making the CPU speed up and slow down, you know, while it's doing, you know, while what, you know, when it has some kind of time where it's not really busy, you know, that has a huge effect. And like you just start going down this list.

And so, um, what we've, what we've begun to do at OC N is really look at that energy efficiency and not just optimize for scale, not just optimize for cost, not just optimize for performance, but also optimize for energy efficiency. And you can get transformational benefits by doing that. You know, we've already announced that we can do 50 to 90%, like truly 90% reduction in energy efficiency.

And a lot of it just has to do with focus. You know, it, it takes person years, person centuries of, you know, dedicated, diligent engineering work to deliver that kind of result. But you're not gonna get there unless you focus on it, unless you say, this is the target, this is what we're doing, let's go make it happen.

And of course, it's not just about efficiency, it's about, uh, scalability and performance as well. I mean, you have to do the job, not just do it efficiently, but, but you're doing both. And I, you know, it's interesting what you, what you mentioned that, um, you know, the scale and the scope of data.

I mean, when we started our careers in the storage industry, uh, megabytes was a big number. You know, I mean, I remember my first gigabyte storage array, and now, uh, that's a laughably small number. So you're right, it is coming everywhere.

And the technology that you described, you know, changing, you know, clock speed on processors and stuff, that is not just table stakes for modern processors, it's the whole ball game. You know, you cannot not build a processor that operates in that way. And you see the same thing happening with data.

Well, the prior company I started was a company called cleversafe, which when IBM bought the company, uh, in our category of on-prem object storage software, you know, we had, we had at, at hyperscale things a hundred petabytes and above a total storage, uh, in that system. You know, we had a hundred percent market share. We made all those systems.

'cause at that time, no one else could do it. I remember when, and and IBM bought that company in 2015, and at that time, petabytes were like normal. And exabytes tens of exabytes, hundreds of exabytes was kind of the state of the, the art in terms of scale for storage systems.

I remember when I started Cleversafe, uh, I started in 2004, I think in 2005, I sat down and calculated how many systems in the world at that time, 2005 were at least a petabyte. And my estimate was 13. That was it, you know, and now a petabyte, like that's a corner of a server.

So this always happens, and we're seeing this, this, this scale, you know, not just like twice as big, but orders of magnitude bigger, uh, again and again. And we're seeing the same thing happen with, uh, hyperscale always on compute intensive, uh, uh, data analytics workloads, which we focus on. Our focus right now is, you know, things where, you know, you need at least 500 cores, uh, in order to deliver that solution, uh, on which our software would run, typically in terms of data volumes, the average query or the average machine learning function, or the average geospatial function, the things that, you know, the analytics themselves are gonna look at hundreds of billions, if not trillions of elements, data elements, or like rows in a spreadsheet would be one way to think of that.

And, you know, at that scale, you know, if you, kind of that trillion scale, you know, there's around 500 to a thousand different systems right now, um, still, you know, just a small part of the giant data, data analytics market. But you know, this concept of trillions and, and the next number that people are gonna start to learn is a quadrillion, which is a thousand trillion. You know, we're working actually right now on the first quadrillion scale system.

Um, these things don't deploy overnight. Um, but, you know, those words are words we're gonna start to use is, you know, not just, you know, trillion scale exascale, but quadrillion scale. That's what's coming.

So tell us a little bit this fascinating, Chris. So you talked a lot about focus and, uh, being able to scale. 44 terabyte drives, but what's fascinating to me is the way you're able to kind of architecture the, to the overall system.

Can you give us some like, quantifiable numbers and show us like how you're, you know, making those, those systems a lot more dense and, you know, power efficient? Well, the, the breakthrough that enables what, uh, OC N's able to deliver is, is solid state. And, um, I think the, the amount of investment, you know, from semiconductor companies, you know, like solid and others to, to bring that technology to market over the past decades, you know, it was definitely tens of billions of dollars, probably around $60 billion investment.

And that, you know, that kind of goes back to focus and you know, this problem that OCN solves, which is how do you limit, you know, scale the amount of data you're analyzing in a system without limits, without worse than linear increase in cost? You know, and that, that kind of had been this state of the art where oh yeah, you could, you could scale up and scale up and scale up, but if you wanna analyze a million times the data, it's gonna cost you more than a million times the dollars. That had always been a known problem in computing for decades.

And like, how can you solve this problem? And the reason why this was such a problem was you really had two building blocks. You know, as a software, as a software designer or software builder company, you, you're, you can not go faster than the hardware.

You, your price performance cannot be better than the hardware. And if you do your job perfectly, you're gonna max out what the hardware can deliver. And, and previously the only two building blocks you had were DRAM and spinning disc.

And the problem with DRAM is it's crazy expensive. Yeah. You can solve these giant hyperscale problems with dram, that's what a supercomputer is.

And they cost about a billion dollars. So there's, there's some people that can spend a billion dollars, but if you don't have that kind of money previously, you were stuck with using spinning disc. And the problem with spinning disc is the performance is ultimately a physical phenomenon.

How fast can the read write heads settle into a track? How fast can the platter spin? That time hasn't changed for decades.

And so on a Moore's law adjusted basis, spinning disc keeps getting slower and slower and slower, and it's a million times slower than it used to be. And it's thousands, you know, it's just way too slow to solve this problem. Along comes solid state, solid state today is offering thousands of times, two to 3000 times the price performance per dollar is spinning disc that's limited, not by a physical phenomenon, but an electrical phenomenon.

And that's on a Moore's law curve. So two to 3000 times better price performance right now means 5,000 times better, 10,000 times better, or a hundred thousand times better. And it's the thing that unlocks this whole, this, the, the solution to this problem of hyperscale always on compute intensive, uh, data analytics is unlocked with solid state, But it's not just solid state that's making this possible.

And I think that that's the thing, you know, people could listen to this and be like, oh, great, you know, yeah, you're using SSDs, congratulations. But you're doing a lot more than that. I think one of the interesting aspects of the OCN solution is, uh, sort of this proximity idea that you're doing processing closer and closer to the data, and that actually reflects the architecture of modern, um, machine learning and HPC.

So if you look at how, um, for example, the Nvidia grace achieves such high performance, it's because the memory and the compute are located right together on the same, it's the same way that Apple achieves performance with their apple silicon. And you guys actually have a data approach that works similar to that, right? So you're not moving as much data around.

Yeah, I mean, you, you know, you and I have been around long enough to have heard like you gotta move the compute to the data, um, many, many times. But the, we're in this realm where when you're looking at these hyperscale workloads, it it is not, I mean, typically in, in, in the workloads that Osint focuses on, and you're talking petabytes, you know, if not x bytes of data, not just being stored, but being analyzed. And if you wanna run a query or run a machine learning function or something like that, and it's a petabyte of data and you need to move that, you know, it might take a day, it might take an hour.

I mean, that's just not, you cannot use that architecture. So what we've seen, you know, in terms of focus is the rest of the industry is focused on kind of the, the, the, the large opportunity, which is, which is real. And they, you know, they've really done a great job, um, building solutions for kind of smaller active data sets, smaller amounts of data that's being analyzed.

And they separate compute and storage into two tiers in the architecture fundamentally. So they have to pull the data outta the storage tier across a network into the compute tier. And that's fine if it's a gigabyte of data or even a terabyte of data, maybe 10 terabytes of data that you're analyzing, but you're getting into the hundreds of terabytes, petabytes, tens of petabytes.

Like, that's just simply not gonna work. So what we've done is collapsed, collapsed, compute and storage into a single tier. So we're not pulling data from storage across a network connection up into a compute server.

We're pulling data across multiple parallel PCI lanes, like within a server. And, you know, we'll have thousands of times the data bandwidth just at an architectural level. Um, and it, and it, it, it shows up in queries.

I mean, we, you know, the, the kind of analysis we can do are things that are either impossible. Um, you know, you've tried, customers have tried other things, it just do, it just doesn't work. Um, and one, one of the problems we'll often have is that the, the rate at which they're adding data to the system is greater than the rate at which other systems can add data.

So you never catch up, and that's a problem. Um, or the other thing we see is we'll replace two or three or four systems, or even five with the Synosis system. And I think, Janice, you were asking earlier about kind of what does that look like?

You know, we'll often see like five or 10 racks of equipment, you know, a hundred, 200 kilowatts of power draw, and you can replace that with like half a rack, you know, one 10th of number servers, one 10th, the amount of electricity. Wow. Okay.

So I just, just wanna dive in and, uh, on that note, uh, talk about some of your customers, right? I think it's fascinating, as Steven said, what you're doing with the software and the hardware. Can you give us a little bit of color around the type of customers, like who's really, you know, on your target list to, to support with your solution?

Yeah, So for companies that, that have these kind of computing requirements, um, it, it means that you've got, you know, large scale complex always on, you know, requirements for either your business or your mission. 'cause it, you know, it includes some government customers as well. And there's only so many ways you can have this mu much data, and there's only so certain use cases that need this.

Um, and so, and we have a pretty good sense of who they are. We, we actually have, the way we model the market is we, we do a lot of research where we'll go off and, you know, identify a use case and, and write down like in a spreadsheet, like who are all the customers? How much data do they have to analyze, like, you know, really understand it on a bottoms up basis.

So the biggest, the biggest market right now is, is, is telcos. Telcos are big networks. And, and they're going through a process right now of making what I think is the largest investment in human history, which is five GII, I think they're spending somewhere between five and $10 trillion on 5G, which is, which is a very big number.

And 5G is amazing. You know, it's not only like, you know, amazing price performance, super high location resolution that's gonna enable all these new apps, but it's also the first like, real redo of the backend infrastructure of, of mobile telephony in a long time. So there's a lot going on there.

It's amazing. It's gonna happen and there's no denying the world will benefit from its use. A challenge for that is in 4G, the amount of data metadata that a, that a large telco makes, um, is already at a scale that they can't analyze.

So if you're, if you're a major telco, your, your network connects things a lot, you know, like your phone wants to buy something, it's gonna make 10 connections to do that. And this is constantly happening. So a major telco will con make a trillion connections every two days, maybe every three days.

And if you wanna go back and analyze like, why is my network slow in Boston yesterday? Or, or should I put my next cell tower? Or, you know, things like that that you need to do, or there's also compliance reasons why you have to have this data and analyze it already.

They can't analyze at that scale, that trillion scale. Along comes 5G 5G increases the amount of metadata that a, a telco network creates by 30 to 50 times. So they're already like, I don't know how to deal with this volume that I have today, and it's gonna 30 to 50 x.

And you know, the, the marketing people at those telcos, what they're doing is they're gonna go sell 5G and the it people that have to run the network, they don't get to say, whoa, whoa, whoa, slow down. You know, this is really hard how I'm, how am I gonna deal with this metadata? You know, they don't, they don't get a vote.

They just get a, they get a problem and they have to solve that problem. So that's a, you know, that's one example. We also see it in, you know, vehicles, you know, vehicles is the same thing.

Like these, you know, a car, a typical car today makes, you know, petabytes of data and you know, they just, right now they have to just like throw most of it away 'cause they can't analyze it at that scale that it's made. We also see this in ad tech. We also see this in, you know, uh, other markets like financial services.

So there's, there's these very specific markets that have this kind of requirement where they're just dealing with this scale of data. There's a close relationship, I think, between HPC and massive data scale and of course emerging AI applications. So talk a little bit about how, uh, AI applications are starting to demand this kind of scale.

And in particular, I'm thinking of, uh, retrieval augmented generation, uh, which is emerging as one of the key technologies or that are powering, uh, practical applications of ai as opposed to just check out my cool chat bot. Well, I, I think if you step back, you know, there's been a lot of AI revolutions, you know, over the, over the decades. And this is not the first time the world has been captivated by ai.

What I would say is different about this round is not only is the technology, you know, better and amazing, but it's, it's like the first time AI has, has dealt with scale. It used to be, you know, AI would, you know, I, I did some AI programming back in the eighties and, you know, it was like, you know, megabyte of data, you know, something like that. And it wasn't scale.

And you know, when you look at like, what large language models are doing kind of for the first time, they're able to understand the whole l you know, a whole language. And every time it's been expressed on the internet, which is giant, like, that's a scale of AI that just, that's new. The prior revolutions couldn't do that technically.

So I, I think that's, that's kind of the, the, one of the big differences, maybe the big difference in this AI revolution that we're going through now. But that creates challenges. And one of those challenges is if you want to analyze the, the world's language or analyze the world's network metadata or analyze, you know, the, all the vehicle telematics data and have AI have the intelligence of what that data, uh, is saying, well, you have to have that data in, you know, one, in an analyzable form, and it never starts in that form.

So that loading and transformation, and it's not a one time thing. You know, if you ha if you wanna analyze, you know, vehicle telematics data with ai, that is a data set that is a fire hose of massive proportion that never shuts off. I mean, the cars are, are driving, people are using them.

Exabytes of data are gonna pour into your system. And, and so the real challenge isn't, oh, I've got this static data set. I'm gonna put this, you know, AI system on, and I'm gonna do, I don't know, correlation or regression or ask it questions about, you know, reliability or something like that.

That would be hard enough if it was an exascale static data set. But that's not the, that's not the requirement. The requirement is there's a billion cars driving at all times and they're just pumping out data.

And you've gotta put a system on top of that that derives, you know, intelligence. And that's, that's a challenge. And, and, and a big challenge for that is how do I take this never ending giant pipe of data and get it into a usable form in, in a reasonable period of time?

That that's, that's a real difficult challenge and something we've worked a lot on at OCN as well. Maybe we switch gears just for, just for a moment. So, uh, I know Chris, we've talked about this before, right?

You know, you have 50 to 90%, um, power efficiency and, and it's just amazing. But, you know, uh, working with a lot of different partners, uh, from the solid side, we're not really seeing that from other partners, right? We're, we're not seeing the same aggressive, uh, stance.

So is there anything that you guys are trying to do, you know, to uphold the industry, to push others to kind of do the same thing? Um, do you wanna talk a little bit about that? Yeah, absolutely.

I mean, the, the reason why the a an essential ingredient for making these systems more power efficient is to focus on making them more power efficient. Um, and you know, that, that, like, we're just not gonna get there. And the way it works, it's just like, you know, every company that makes any kind of computing product focuses on cost, efficiency, focuses on performance.

And the way that happens is, you know, it's, it's not just like one number, like make it seven, you know, it's, it's very complex. When you say, what do you mean by cost? Well, you got all the lifecycle costs.

What do you mean by performance? Like, what does that mean? You know, you, so the first thing is to define what it means to be energy efficient.

And, and we're, we're working right now to, you know, with a lot of other industry players, including solid IM and others to say, like, we, we've gotta define what energy efficiency mean, what's the benchmark, what's the metric? And we're in the very early, early stages of an industry of, of doing that. But that's gonna happen.

We're gonna, we're gonna create measurements and metrics and, you know, goals. And, um, so that, that's kind of step one in parallel with that is, okay, now you, now you've defined what does it mean to be efficient? What does it mean to be energy efficient?

And so then the, the way it works at any information technology development company is you then start to prioritize. And, and the, the, the low hanging fruit is bigger. The, the, you, you focus on the bigger low hanging fruit.

So you always start with like, oh, this first thing we could do won't take very much time, and it'll make it twice as efficient. All right, let's do that, then let's focus on the next thing. Well, that's gonna take a lot longer, but it'll, you know, it'll be, you know, another two X improvement, let's do that next.

And you, you, you, you prioritize them based on their efficiency, and then, you know, over time it gets harder, like 7% more efficient with this giant investment. Well, that's down the road. So where the industry is right now is we're, we really haven't focused on it.

And as a result, like there's a lot of low hanging fruit. And so you're gonna see, like, you saw this in osn, like immediately we come out with 50 to 90% improvement. In some cases we've been demonstrating a 98% reduction.

Like, that's giant because it was just a lot of low hanging fruit to start with. Um, and so, you know, we, we, we now already know, here's the things we're gonna do next. This will double it.

This will, you know, improve it by 30%. So that, that's it. It really is just, you know, that simple and that complex, I mean, at a simple level, it's like you just prioritize the, the biggest bang for the buck and work your way down that list.

And the only way you do that is by saying, I'm gonna make that a priority and I'm, and I'm gonna, me, and here's how I'm gonna measure it. Uh, now it's com it gets really complex in how you do it, but, but it is really just simply a matter of focus. It Is really refreshing to hear somebody in your position focusing on energy efficiency, because this is, you know, it, it is a consequential topic to literally every company, and yet it is not the focus of most companies.

And, and, and, and, and yet, what do we hear? We hear people constantly talking about, you know, criticizing ai, criticizing, you know, the cloud, modern compute, et cetera, uh, for the energy consumption that it has. Um, criticizing it as, you know, I mean, people will talk about the scale, they'll talk about, uh, you know, oh, well this company, you know, they've got their own nuclear power plant, or this company isn't right.

You know, uh, investing in, um, in, you know, buying, you know, pre-buying like, you know, gigawatts of electricity in order to support t support their build out, you know, terawatts. Yeah. What, you know, It, It's, it is refreshing, it is refreshing to hear somebody say, no, you know, we've gotta think about the efficiency of this.

We've gotta think about the, uh, the impact of all this. And yet at the same time being able to say, but we also have to be able to support data scale that just nobody could achieve previously. Yeah.

And, and, and the reason why it's really changed, if you go back a year or two ago, th this wasn't even a topic, um, but what's been happening is energy use by data centers in general, driven by Bitcoin, definitely driven lately by ai, but still, you know, all the other types of data, data analytic analytics are the biggest users still and, and will continue. So if you go back a year or two, it just, it just hadn't like, uh, reached the tipping point. And then a couple years ago, the amount of energy consumed by data centers passed California, it is starting to get real.

And now it's about to pass Brazil, you know, like a big giant economy, you know, is, is less, it will soon be less energy than what data centers are doing. And the problem is, um, it's accelerating. You, you know, while all these countries, you know, for really important reasons are lowering their power consumption, here's this category that has gone from, you know, not a big deal to like passing by large countries and accelerating.

If you look at the latest IEAA models, the inner, uh, international energy association models of like what is going on with data center power consumption, and they have different models kind of, you know, like climate models. They have, you know, they're projecting different things. And the range is between, um, the, the, the amount of data, uh, sorry, energy consumed by, um, data centers is, is doubling between every two and four years.

And, and we've all been in computing enough, like if, you know, looking back like things that, you know, storage would double every two to, you know, years and that's Moore's law or compute or whatever, and next thing you know, it's a million times more storage or a million times more compute. Well, that's not gonna work with energy. We're not gonna be able to have a million times more energy for compute.

Um, so, so the only answer is efficiency. And so that, I guess, you know, my call to action would be, right now, if you look at RFPs, and this is what drives the industry, is when customer, big customers buy stuff, or customers buy stuff in general, what's in the RFP, you know, the request for proposal, what's in, you know, what are they making their buying decision based on? And it'll have like cost and performance detailed to forever.

And here's all the capabilities I need they don't currently have, and here's how much energy and, and here's how much energy efficiency you gotta hit. It's not in RFP. So my call to action would be, you know, as, as customers, and it's in, it's in customer's best interest, like these are accelerating costs, you know, a million dollars a year for energy, you can't have that accelerate.

So what we need to see in RFPs and, and buying decision making is energy efficiency, energy use, and that's gonna cause the whole industry to focus on it. And you'll see breakthrough results. I, I, I, I've gotta agree with you there.

And the other thing too, to keep in mind is many of these companies have made commitments that they're going to reduce their energy or their greenhouse gas impact. They can't not do that just because they're chasing the ai, um, trend. So they have to find ways of reducing power consumption.

Uh, great, great conversation. I so appreciate the fact that we were able to get you on here and talk about this, because I feel like this is something that's been missing from our conversations a a lot of the time here on, on utilizing tech. And, um, and yet it's something that's important to all of us.

So thank you so much, Chris. Uh, it's great to catch up with you. It's great to learn about osit.

Um, and it's great to learn how you're leveraging advances in technology, like, you know, flash, uh, storage and, uh, co-location of, uh, compute processing and storage and so on to improve the overall impact of everything that we're doing. Thank you for listening to this episode of Utilizing Tech podcast. You can find this podcast in your favorite podcast application, as well as on our YouTube channel.

Just, uh, search for it in your favorite search engine. If you enjoyed this discussion, please do leave us a rating, uh, maybe a nice review. This podcast was brought to you by Soy, as well as Tech Field Day, part of the Futureum Group.

com, or find us on Putter and Mastodon at Utilizing Tech. Thanks for listening, and we will catch you next week.