Supercharging AI Infra with MemVerge Memory Machine AI

We have been working in the infrastructure software space for the last seven and a half years. In the last couple years, we have been very focused in how do we improve the AI infrastructure stack for the enterprises. So it's great to hear, share with you the latest update.

All right, so we all know in the last two years, AI is making waves across the world impacting our lives, and it is impacting enterprises as we speak as well. Uh, there are a number of functions in particular software development, coding IT services, customer support, are looking to adopt AI in a significant way to improve productivity, improve competitiveness for the enterprises. And how they are deployed today, typically are done in two ways.

One is leveraging on the API services, such as those provided by open AI or ARO and so on. And, uh, through the API token exchanges, you're getting the responses to your queries. And that in turn, either power the interactive sessions, or more importantly, the gentech applications that are running the enterprises.

We are gonna talk about a second, which is the, uh, deployment of open source models, the fine tuning and model survey of those open source models within the private environments of the enterprises, whether it's on the cloud or on-prem data centers. And we can, where they can serve, uh, the AI queries without going to the public. And this has a benefit of protecting the data privacy, and in cases, in some cases can also be very cost saving for the enterprises.

And as we, you know here, uh, the news this week, uh, with the deep seek, uh, this is, uh, making the open source model more competitive from cost per perspective as well as performance perspective. Now, while the API model has really matured as a service, you know, very quickly from zero, both open AI and atropic are hitting over a billion dollar in annual revenue, uh, serving primarily the enterprises with, uh, the largest revenue, uh, proportion there. Uh, we are seeing rapid progress among the open source technologies, you know, starting with, uh, LAMA models being open source and o open weighted by meta.

And, um, and now the news, big news this week is deep seek where they are presenting, uh, open source, open waste models with similar results in terms of quality at about one 30th of the cost in training. And, uh, this is a, a quite significant milestone, and I think it's gonna be encouraging the enterprises and enterprise IT to deploy more models on-prem to serve the privacy needs. Uh, while they can do that in a very cost effective manner, uh, before to train the model, you need a hundred thousand GPUs, which is out of reach for most of the enterprises.

Now with deep seeq, it's about 2000 GPUs, and that's well within reach, and they are not gonna be the last one. And open source ecosystem have this tendency of people leveraging on each other's work to make progress. I think we're gonna see rapid and continual progress there in the year of 2025.

And I believe this will mark an inflection point for enterprise ai, uh, making it really, uh, a deeper impact into everything enterprise enterprise does and integrate into the existing enterprise, um, workflows. Now, if we look at the workload that is being run here, uh, to serving the enterprise, uh, ai, uh, queries, the left hand side is the API world. And, and today the blue is bigger than the orange.

Uh, blue is where there, there are tens of thousands or hundreds of thousands, GPU infrastructure powering very powerful, uh, training of the leading edge large language models and deploy them to handle queries in parallel. There could be millions of queries going on at the same time. Uh, now as we discussed earlier, within the enterprises, there is a proprietary data they do not want to share with the, uh, API services.

And there has been various methods how they can do that while continue using API, starting with, uh, rag, uh, which is a retrieval augmented, uh, generation design, uh, that you can look up proprietary data from some kind of a vector database and to incorporate into your prompt before you send to the, uh, open API. Uh, and that way you keep your data private while getting, uh, the answers you need from the open, uh, API services. But it does have its limitations where the systems on the database is, uh, not an AI system itself.

And there's a limit on how much context window you can have in sending the prompt to the API service. And as alternative, you could download open source models from, uh, registries such as hugging face and fine tune them with your own data and deploy such models in your own environment and serve your applications directly from within your environment. And, uh, we believe this orange part will be a very fast growing part of, uh, of AI infrastructure in the coming months and in the coming years.

And that is where, uh, leverage is focusing on how can we make that easier for enterprises to stand up open source models, uh, within their environment. So now we know that we need some hardware, uh, to run whatever workload you have. And at the same time where the AI models are changing the world, uh, we are seeing a revolution of underlying architecture of computing hardware.

And that's been going on over the last 10 years. Um, and this is part of the reason that drives the advancement in ai, uh, at the, uh, model level. Uh, the transition is going from an XX 86 centric architecture where you have compute driven by X 86 CPUs, DDR memory with connectivity by TCP IP networking and the storage that's connected to that TCP IP network.

And this is a world that we are all very familiar with, but started about 10 years ago. Uh, they emerge a new center of gravity to the, uh, uh, you know, to the AI workload. In particular, GPU is playing a central role to taking over most of the computing tasks for core AI computations.

And this is in conjunction with a high bandwidth memory that's placed on the GPUs, the, uh, the new AI fabric that's emerging, uh, led by ENV Link, and now followed by UA Link and CXL and other type of, uh, uh, memory fabrics and compute fabrics and emerging new data platform that's connecting to that, uh, fabric. So this create a new AI computer that is at the center of this AI data center, and they work in conjunction with the Legacy X 86 centric systems to form the new data center. Now, the challenge is how do we map the AI workload, uh, that are happening, you know, within the enterprises to this new AI data center, with this new architecture that requires definitely a new energy, uh, envelope, a new cooling systems, but how do you place the workload in an optimized way onto this new hardware?

And this requires a new layer of software with the, uh, new elements, both on the workload level as well as the underlying architecture level. And this new layer in blue is what we call AI infra automation software. And this is how we automate the provision, orchestration and optimization of resources so that we can best serve the AI workload above it.

And this blue layer is what members being focusing on. You know, how do we create a layer that can optimize the resources while delivering the easiest, simplest experiences to the platform engineering team, to the AI resource AI engineers as they deploy their workload within enterprise environments? So now let's double key, uh, into this AI infra, um, automation software layer.

Uh, at the bottom is what we call a enterprise, GPU as a service software, um, GPU, unlike the XX 86, uh, it does not have a very good virtualization layer. Now, this problem of multiple people using the same pool of resources is largely solved for the X 86, uh, computing world, uh, where, uh, the, um, virtualization and containerization technology at the core of the solution to that problem, where you can spin up and spin down VMs or containers pretty easily, and you can allow multiple users to share the same pool of resources. But this is more difficult with GPUs, uh, as it does not have a good virtualization technology.

So it requires a new layer. Yes. So the, the MIG support and MSP support, you don't think is that good or, Uh, I think PDA and GPUs, Nvidia GPUs, yes.

Right? Yes. So, uh, with the GPUs, it offers an alternative method, uh, of, uh, allowing different, uh, users to share the GPU resources such as, uh, multi-instance GPU, the mix, the NPS as well as the time sharing, uh, mechanisms.

And, but it is underlying, uh, pieces of technology without a entire virtualization platform that can allow the GPU resources to be effective shared. So it's, it's an underlying mechanism that is enabling, but someone needs to create a better scheduling, you know, resource scheduling and the orchestration system to go along with it. Maybe a good, um, analogy is, you know, with the X 86, uh, you can have the, uh, ESX, the underlying, uh, hypervisor, but you also need the, the, the DRS, the scheduling system and various other vCenter type of system to allow the whole pooling and sharing to happen.

And that is a layer that is, uh, largely, um, uh, absent, uh, in the GPU world. And as I was showing the, in a, in a few slides that you'll notice the utilization of GPUs in the enterprises is very low. And part of it is because the GPUs belong to different departments.

You can have different users from different departments, and, uh, there is not a easy tool for them to, uh, share a piece of resources. I was just talking to a customer in New York last week, uh, where they are building up their GPU environments, and there are different trading groups that are accessing those GPUs, but GPU is always not available when my group wants to use it because the other groups are using it and or they are only claiming they are using it, they are not actually using it. Even though those GPU are idle, I cannot have my access to it.

So there need to be a new system, a software layer that's created, and that's a layer where we call the GPU as a service, uh, that essentially plays the role, what virtualization played for XX 86, uh, that our layer of software plays for the GPUs that we can, uh, utilizing the mix and, uh, time slicing and the various underlying mechanisms gpu, uh, vendors like Nvidia provides, uh, and a MD provides, but also adding the missing pieces so that we can, creating a, a good platform for the pooling and sharing of GPUs between different departments at different teams. And in fact, this is where we're gonna dive in deeper with, uh, the second section where Steve gonna go into the technology, and we'll give you a demo of how this technology will work, uh, for GPUs. And on top of it, uh, we will also be building our roadmap, a inference as a service, which really allow a simple one click deployment of popular, uh, open source models such as LAMA three or deep Seek R one, uh, that as enterprises, as they look at the, uh, amazing power of these models, but actually deploying them, deploying them in a private enterprise environment is nontrivial today.

And we want to make it easier so that it's on their fingertips. It's very easy for them to deploy as they need. And not only deployment, but ongoing management and auto scaling of such services.

And the demand for such services go up and down from the, uh, applications. And we also plan to build fine tuning at the service, uh, that can help, uh, enterprises to improve these open source models with the particular data and particular requirements they have, uh, from their applications so that models can be improved to fit exactly the need of that particular enterprise. And all these together form this simplification platform that really simplifies the latest and the greatest open source technology and, uh, build a bridge to allow them to be easily deployed in today's enterprise environment.

And this is a vision we have for this, uh, AI infra automation software. And in fact, as we are presenting here today, uh, we actually have deep seek kind of integrated on top of our software platform that can be easily deployed in any, uh, private enterprise environment. And, uh, we are looking at, uh, you know, looking for partners, design partners with customers who can help validate it, uh, in their, in their environment.

Okay, so now I have a couple more slides before I pass to, uh, uh, Steve, and then feel free to interrupt me with, uh, with any questions. Uh, you have, um, so this is actually, I think Ray, right? So we, we just had a discussed, you know, with the GPU, starting from the base, the GPU as a service.

One symptom that we are noticing is with the, with the enterprises, there is a low utilization of GPUs. You know, on one hand, enterprises are buying more and more GPUs on prep, actually, they're buying hundreds or even thousands of GPUs. And the larger banks have 10,000 GPUs in their environment.

And those are being owned by different groups, and they're typically very siloed today. And this is a survey done by ways and biases showing where the X axis are the average GPU utilization under 15%, from 15 to 30%, all the way to 90%. You know, the higher the better.

That means the GPU are more fully utilized. The Y access is a percent survey to answer yes to how much of their utilization is. So we see about a third of the people who are surveyed, uh, there was GPU utilization less than 15%, and if you add the 30%, it's about half of all surveyed has utilization less than 30%.

As we know, GPUs are not cheap. So this is leaving a lot of money on the table in terms of the infrastructure, uh, investments that these enterprises have made. So How do you measure GPU utilization?

Is that a reflection of how much memory has been allocated? Uh, this is a reflection. In this particular case, it's a compute utilization on the GPUs, which, uh, NVIDIA and AM MD actually reports actively.

So that's a stat you could get from the GPUs, which is, uh, how much of the GPU is being used and how much headroom it has, that's not being used. And it also provides the information on the memory utilization, which is also important metrics as well. And, uh, um, and which opens an interesting, uh, fact is that often GPUs are bottleneck that memory, not at compute, right?

So, so, and sometimes another reason for the low utilization on GPUs could also because of the, uh, full utilization on the memories that blocks the, uh, the compute, right? I mean, it's tricky. I mean, there's a difference between allocation and utilization, right?

Look, let's look at make, so you can have up to seven mg, right? But there's no more memory to be given out, right? Right.

So from that perspective, maybe if they only use half of the memory, if that's what you measure, then you could theoretically say it's 50%. But in reality, there is nothing more to give because it's allocated. Right.

And then, and then the, the second question I had regarding to your, to GPU as a service in your world is, is software and it's more like an orchestrator. And does the orchestrator then look at the model being deployed and you calculate that into your efficiency and how you schedule things? Yeah.

Is that the way I should look at it? So as Steve will go into more details, the first version we are making available, uh, next month in February, does not take into consideration the internal of the models. So it will treat it just as a workload, and we kind of treat it as a black box, uh, as we, uh, support them.

Um, and I think, let me make another comment on your first question, which is a very good question, uh, is that, uh, uh, you know, with m you can have up to seven pieces and where memory is kind of a, uh, fraction GPU, uh, enabled by the hardware. So the memory and the compute actually petitioned together. And as a comparison, time slicing of GPUs actually does not consider memory.

So the mam, so even though a task could be just using 10% of GPU, it could occupy a hundred percent of memory that blocks all the other tasks from working, right? And, and this is something we have built a technology where we can petition the memory along with the time slicing of the compute, so they make it, uh, uh, you know, more practical, uh, solution for people to share GPUs as well. So that's one of the technology.

So Charles, we, we had earlier today, uh, another vendor talk about their solution, uh, virtualization solution. Are you, you feel your GPU as a service fits underneath that, or is it a replacement for those sorts of things, or, Uh, I believe it's alternative to a real GPU virtualization solution. Um, you, I used to work at VMware and VMware acquired, uh, also some GPO virtualization, but in general, Nvidia in particular is not a believer in GPO virtualization.

So it's very difficult for any virtualization, even good technology to take off on GPUs. So our approach is alternative to that. Uh, so we do not have actually in a narrow definition of virtualization technology for GPUs, but we have various fractional GPU technologies and we have the intelligent GPU aware scheduling software that can basically allow you to reserve resources, allow you to borrow resources beyond your reservation, and we can also enable a, uh, graceful preemption, uh, from, uh, lower priority tasks to give resources higher priority tasks to the people who are borrowing resources to the people who owning or reserving the resources and doing that smoothly.

So by doing this, we can maximize utilization of the underlying resources without impacting the user experiences and achieving the same goal as a virtualization technology. So, short answer is this is an alternative approach, uh, to those virtualization. I, I want to give some context on this graph here.

Um, I looked it up, I researched this because this is pretty eye-opening, and I was like, well, number one, I, I thought this, this can't be true. Uh, number two, I thought this has gotta be some weird exaggeration. Like maybe they don't actually have anything to run, so that's why they're not running it.

This is worse than it looks if you actually read the survey. This is, um, 15% utilization across actual runs. This doesn't mean it's not utilized because they don't have anything to run on it, or because they're not effectively sharing or or leveraging what they've already bought.

This is just, it's just not being used. It's not being used effectively even when they're actually trying to run things on it. Yeah.

Can you post the link? 'cause I can, I I just shared that in Slack and I'm gonna share that this is, this is horrible. Yes.

Thank Yeah. And, and it's actually fabulous. Yeah, I've, I've been talking to quite a few enterprise customers over the last few weeks and last few months, and this is true.

Uh, you know, obviously a lot of enterprises are at different stage of establishing their GPU infrastructure, their AI infrastructure, but they are looking to this problem even within the same team between a few engineers for them to share GPU resources. Not easy you today, they either don't share or they use a spreadsheet. So I'm using it today, you use it tomorrow.

So it's very rudimentary stone each I think, I think, I think part of it is also goes back to another presentation we had at a different field day event where they had actually done some analysis on some of the models and said that the, the theoretical amount of GPU that the models could use was about 60%. Mm-hmm. Yeah.

And the, uh, practical amount they were using was more like 30 to 40%. Yeah. So I think that might tie into what we're seeing here.

And you know, I, I pressed him on that. It's why is that? And he says the software isn't written well enough to make better use of the GPUs.

Right? Right. And that's what this survey shows too.

It shows, um, starting off with a multi GPU run and then suddenly everything's running on a single GPU and everything else is idle. Right. Interesting.

And that's just right. I can't Imagine wasting resources like that. I do think multiple factors contribute to this very low number of utilization.

And I believe having a good orchestration system like what we are building is not gonna move this to a hundred percent, but I think we can significantly improve the utilization, I dunno, Move it to 20% or when, Right. So we think we can at least double the utilization by effective automatic sharing of the resources between the amount for users we're sharing them. So, and that's what, uh, Steve's session, we'll go into more details, oh, sorry, wrong direction here.

Uh, and you know, there's some key technology that we have built that have gone into this GPU as a service solution, and that's the central topic of today. Um, and, uh, uh, you know, there's GPU pooling and sharing, just like what we said, that allow you to get higher utilization and you can buy less GPUs to run more workload. We have a transparent checkpoint in technology, and this is a key piece of technology we have developed over the last seven years that allow you to essentially take a snapshot of a process tree and being able to run it elsewhere.

And we started with CPU support, we're adding GPU support to this. So, and we integr into this product that allow when a workload preempt another workload, we can take a snapshot of that other workload so that you don't lose all the work that has done on that particular, uh, for that particular workload. And Steve will talk about it and Bernie will talk about it as well do a will technology deep dive into that transparent checkpoint technology.

You know, our technology in particular GPU agnostic that we, uh, will have, uh, multi-vendor support starting with nvidia, the market leader today, but also a MD Intel and other AI processors that we plan to support. And this is integrated with Kubernetes, which is a very popular multi-cloud, uh, technology that many enterprises adopting. And we are taking advantage of many of the capabilities, uh, that Kubernetes provides and build our layer on top of it.

So these are some of the noteworthy points where Steve and Bernie will, uh, go into more details. Now, last, uh, let me just introduce, uh, a pioneer program, uh, that we have, uh, as I mentioned, the first version of the software memory machine AI will be available in February, and this will start with GPU as a service and go to inferences service and fine tuning as a service. And we start this program to really, uh, interact with the early adopters of this automation software layer.

Uh, we are gonna have provide early access to all these software components and working closely with these customers to making sure the solution works for their environment. So if you're interested, you know, scan this code and sign up and let's have a conversation and to see how we can enable a automated enterprise AI infrastructure for you, How do you see something like fine tuning as a service working? Yeah, so, so essentially, uh, fine tuning as a service is not something that we have available yet, but this is, uh, uh, this includes the, uh, the, the, the sequence where you download a open source model, uh, from a registry and where you need to, uh, get into a, a training framework, uh, where you, uh, uh, with your, you know, adding your proprietary data to it, where the model can be, uh, improved.

And there is a whole cycle of, uh, development experiment and testing. And so that the model, uh, improves. Um, and when the model finishes is something that you can deploy, uh, through that inference.

Historically, there's been a lot of, uh, human reinforcement learning Yeah. Associated with fine tuning that I find hard to automate or orchestrate or, Yeah. So on our roadmap, this is probably gonna be the last piece and, uh, we are, uh, building it together with the customers and understanding their specific needs.

Okay. Yes. So you had, uh, you know, you have the fine tuning, you have the inference, then you have the GPO as a service, right?

Just cover the GPO as a service is like an ideal customer profile for you. Someone that has a specific, say, cohere bill of this variety or size, and then they're gonna make an investment, capitalize an investment in their data center and they want to, is it a, I don't wanna go too far. Yeah, yeah, that is one example.

Okay. Uh, a more, uh, another typical flow how we, you know, engage with the customers is we are engaging with actually the infrastructure team, the platform engineering team, okay. Who is building up a 5, 512 GPU cluster to serve multiple groups.

And, uh, now add To say that's a small cluster or we just like shifted our window of what Large with the enterprise is actually a good size. The cluster, if you have five 12 more of them is like 1 28 or 64, uh, today. Uh, but this is gonna grow, uh, as we, I think 2025 gonna be an important milestone year for enterprise AI where all this gonna grow.

But today it's really five 12, you know, anywhere between 64 to maybe a thousand is a typical cluster size Okay. Enterprise setup. And, uh, they have a charter and they, you know, this is often a strategic initiative from the CEO, uh, in setting up this AI infrastructure.

And there are various team that's to take advantage of it. And so our customer is really the infrastructure and platform engineering team who's standing up this GPO as a service and their customer are the various people who are setting up cohere or set up other other system on top of it. And we can allow effective sharing of the resources.

Got, uh, that's underneath. Thank you. So, um, GPU as a service, how, how I as an application developer, uh, access the GPU on this environment?

Is it, yes. Is it through Kubernetes calls or Yeah, that, that's a great question. In fact, uh, uh, that's a good leading to Steve, uh, section.

Steve not only will explain it that we have workspace features integrating to the IDs, you know, whether it's, uh, Jupyter or whether it's uh, vs code that you can actually launch stuff directly from your IDE and, uh, that links to our platform and provision the resources, uh, for you. And Steve will also show a demo exactly how, how that works. So What's the distinction between GPU as a service and infra as a service?

So I think it's a, uh, specific instance of infra as a service where we, uh, so it is infra as service, but GPU is a little special. Uh, so we, uh, essentially have spec specific GPU aware scheduling and orchestration capabilities with A GPU enabled checkpointing capabilities to offer a service that would work for GPU centric infrastructure. So, so this is a special sub component of infrastructure service, which is we believe there's a gap, uh, in the, in the, in the field today.