AIStor – PromptObject, AIHub, and MCP Demos with MinIO

Good afternoon everyone. I'm d Krishnan. I'm one of the engineer here at Manaya.

So today I'm going to be talking about some of the key innovations that we introduced as part of AI store, uh, and walk you through live demo. Hopefully, uh, we'll be able to see successfully how these features work. Uh, so based on working with customers, some of the key things that we have noticed, uh, all the customers face, uh, some of the AI challenges are how can they safely secure their, uh, private data sets and models?

And after they do that, how can they get intelligence insights in the production environment? So these are some of the key challenges, and I've simplified the problem, but there are multiple steps involved to get to the final intelligent insight piece. But today in the demo, I'll show you how using some of the building blocks that we have introduced, customers will, will be able to securely store their private data sets and models and then make int intelligence insights and finally even get a production ready AI agent, which is able to interact with the data in your AI store still being secure.

So let's start with the foundation. Uh, so in order to have your models, uh, and the dataset secured safely, uh, AI hub is the feature that we introduced. So using AI Hub, uh, all your private data sets and models and even fine tune models can be safely and securely stored in AI hub, which is directly storing all of these into the AI store buckets.

Uh, we thought about how to implement this, and we had a lot of discussion. And finally we decided that instead of us introducing, uh, our own, uh, way of doing things, let's set off the industry standards. So hugging phase has become like the defacto standard in the AI world.

Uh, they are like the GitHub for all the data sets and models. Everyone from meta open ai, everyone's, uh, publishes their models there and everyone consumes it through hugging phase. So what we have done here with AI Hub is AI Hub is fully hugging phase compatible, but still it runs in your on-prem.

So what it does for your developers is they do the same. API development, they do the same training or loading the data sets, fine tuning the models. And when they're ready to push these, push these models.

Instead of it going to hugging phase, it is actually going into the AI store bucket, which is private, and the entire ecosystem is fully supported. It's not just hugging phase. You can now deploy your VLLM models using the AI hub that is already stored in there, or even SG lan, uh, NV dms.

Everyone supports hugging phase, uh, in general. Uh, so let me just quickly dive into a demo of how this works, right? So for that, uh, as a data scientist, now Jupyter Notebook is the defacto standard for everyone.

This is where the ML engineers or data scientists do their work. So here what I'm going to do is in the left you can see that we have two new categories. Uh, this is a custom plugin that we developed specifically for our customers who are going to use AI store and AI hub, where they can automatically browse all the models that they have privately stored.

So, and when they click on it, you can already see this is almost similar to what you will see in, uh, hugging face, uh, the Read Me page, they can easily see these are the models that is there, how to use them, and all of that is, uh, done right through the Jupyter Notebook without them leaving anywhere else. And same goes for the datasets. Now that you have this, the way this works is, as you can see here, whenever the client makes a request, what AI Hub does is AI hub first talks to AI store and sees if this model or the dataset actually exists in AI store.

If it doesn't, then it immediately, uh, pulls the model asynchronously from the hugging face hub and directs it back to the client. But behind the scenes, it's also caching it into your, uh, AI hub or the AI store. So next time, and the user comes and asks for the same model, it is directly pulling it from your AI store without even touching hugging face.

And, uh, so for example, here, this is the model. So AI Hub is effectively a database for these models and that sort of stuff. Is that, yeah, it's a repository for your data sets and models, but it fully is baked in, uh, and it's fully compatible with the existing ecosystem.

So there is no specific learning curve for any of your developers or enterprise customers. So What is the, what is the use case? Mm-hmm.

Sorry, I don't, I'm trying to be like disrespectful or anything, but like, what is the use case for that? So what we have seen with customers, the use case is, uh, with the gen AI becoming like a rage these days, the customers, especially the enterprise customers, what they do is, uh, they want to use these open source models. Let let it be Lama or Quinn as the base models and using those base models, they have their own custom or private dataset.

I train it. They want to fine tune it using their own private dataset for that specialist use case. Oh, okay.

So, so you will be storing your fine tuned, uh, models in here. Yep. So it's not necessarily about like, you know, paving your bandwidth Yes.

From like, you know, pulling the models from hugging phase. Yeah. It's more about like, you know, I need this fine tuned model.

Exactly. Okay. Now it makes sense.

Yeah. Another thing, uh, this, uh, AI AI store, right? No AI hub, sorry.

Yep. AI hub. Where is, where is this data stored?

It's on object storage. It's All object storage. It directly talks to AI store.

Uh, can we, you know, like it, it have basically, um, you know, like a point of access, right? Mm-hmm. Can we move this point of access to be different than other points of access, like dedicated for that different and different VRF, you know, that kind of stuff?

Yeah. So in hugging phase case, that is a single monorepo. Uh, whereas with AI Hub, the way we have designed it is you can have multiple AI hubs.

So one is specifically for your data scientist. They're only given access to the data sets. They're going to build the data sets, clean the data, and then make it ready for the ML engineers to come and do the model training.

And then that will be a dedicated dev environments for the ML engineers who will do the fine tuning and then push these models. And then finally, that will be a QA environment, dedicated AI hub where the evaluation is done. And once all of these three have been successfully done, then, then you have a fully isolated AI hub for production.

Mm-hmm. Which is the one that you use for deploying your models and then doing inference. And the, the, let's say the CICD pipeline that move, that model is integrated in this platform, and you need external tool, Uh, that will be external, right?

So each customers will have their own workflow. In some cases, customers will be like, I don't need this concept complexity, I'll just use the single, uh, AI hub, but I will version the model such that I know which models to pull. Uh, so that is up to the customer.

Okay. And the data versioning and model versioning, all that's automatic Or Yes. So this, since this is hugging phase compatible, the way hugging phase does it is through git based protocol.

So you can create branches, you can do comet build, pull request, and all of that. So automatically we are also compatible that way it's automatically versioned. And even in, in some cases, if you're not using the all of those capabilities that is baked into hugging phase, you want to keep your work life simple.

Uh, you can enable versioning in the bucket, and then when you keep pushing the new models or updating it automatically, everything, the object store is taken care and you can retrieve your older models. If, if at all, that is the better one. Yeah.

So, so just to make sure, like, you know, I understand it, right? So what do you have on the left side? That's basically like a Jupiter extension, right?

Yes. Uh, which you have in your Jupiter. Yeah.

And you using like hugging face, sort of like, you know, like, because on the left side, like you can have the extension for the hugging face as well, right? Yes. But you have it for the AI hub, right?

So you are using the same methodology that's your own extension, I guess? Yes, It's our own extension. Oh, Okay.

And that's connected into your AI hub. Okay. Yeah.

Yeah. And AI hub is like literally behaving as a hugging Facebook, right? Like A rep.

So this plugin is not just for AI hub, but this is for the entire AI store. So you'll also be able to actually go and look at all the objects that is in your buckets, uh, browse these objects. Oh, I see.

So all of that. So this is like a full, uh, suite of development that without leaving the Jupyter Notebook, everything can be done within the notebook itself. Yeah.

Uh, bringing the data to the developer, which Is great for the developer's, right? Yep. So now, uh, the model that I downloaded here was actually pulled from the AI hub itself.

So this is the model, and just to mimic that, once this model has been downloaded, let's say you fine tune it, and then finally you want to push it, uh, hopefully this works. Yeah. So once this is pushed, so basically what happens in a normal hugging face workflow is it is going to push it to the hugging face cloud.

But in this case, what you will see is you will see the AYA new model coming here. So you can see that we have the new model. So now, next time when you're doing the deployment for the inference, uh, you can automatically integrate it with the AI hub, and nothing has left your private cloud and every, the entire workflow from a developer standpoint, it's still seamless for them.

From an admin standpoint, nothing is leaving your cloud. So everything is safe and secure. I understand you need a dedicated cluster to run that, right?

Like a compute cluster For AI hub. You don't, AI hub is just, uh, streaming the bytes between the AI store and the, I'm Saying like when you deploy the model against some data, a compute power that it, Yeah, I mean, like, you know, this, like, you can run this, this giant per book on your left, right? So you just like, literally pull it from the, from the AI hub, you know, deployed on your laptop and you can run a look.

Yep. Yeah. For, for inferencing and all of that, depending on the type of the model, you might need GPUs or CPUs mm-hmm.

That, that infrastructure has to be set up by the customer. So once, uh, now you have these AI models and the dataset secured, and you're able to deploy them and ready for production. The next phase is, uh, doing intelligence, right?

So that's why we introduced something called a prompt object. Uh, so one of the key things that we noticed as the gen AI started maturing is we are moving away from a put get paradigm to a put and prompt paradigm. So what do I mean by that?

So earlier when cust, when developers were dealing with data in the object store, that is a different kind of workflow that happened. So the first, the developer will come and download this, uh, object. It can be PDF image or whatever, any kind of unstructured data.

And then after the object is downloaded, then they will go through some ML pipeline to do some kind of inference on top of this or do some predictive analysis, and finally, uh, use it in the application workflow. So this has multiple flows involved, and there is a lot of frictions involved, multiple teams involved. So instead, what we thought is what if we introduced gen AI capabilities right into the S3 API, which is the prompt object.

Now, as soon as the object is already in the AI store and you do a prompt object, all you need to do is along with this bucket and the object, you just need to tell, this is what I want to info from this object. And behind the scenes prompt object will go and, uh, uh, interact with the LLM and get the desire results back to the developer. So from a application developer standpoint earlier, what used to happen is either you will have to build a rack pipeline in, irrespective of if you're using all the objects in this bucket or not, you'll have to index them, and you have, you'll have to maintain a dedicated vector database, and finally you will be able to do inference, which is a lot more expensive.

But this is, since this is more online and on demand, this is a lot more easier for the developers to adopt, or even AI agents to adopt and quickly get insights from these objects. Oh, So for the prompt object mm-hmm. You're, uh, doing embedding internally or, or vectoring it or, or what are you doing here?

So, uh, there is no vectoring or any of that involved. This is, uh, directly we will extract the information from the object and pass it in a, uh, LLM friendly manner. So if you're working with an image, the image will be transformed into base 60 for encoding and then pass down to a Yeah.

But you know, uh, you know, customers are gonna have their, uh, regimen or regulations or something like that, and it's gonna be thousands of objects in here. Yes. Yep.

And you're gonna provide the, a prompt object associated with it Without vectoring, it doesn't seem like it's gonna work very well. It, Uh, in, in most of the cases it does. So what are, what we have seen is in many of these use cases, uh, develop similar to what happens with get object, let's say.

So the developers know this is the object that I'm going to interact with. In most of the cases, they're not going to interact with entire objects in the bucket. They're only focused on a specific object for a given task.

So that's why, but Yeah. But, but the prompt is what directs one object to actually go, go through and it's, it's, it's not necessarily clear mm-hmm. Which object in this regulations database is gonna be the one that's gonna pro, uh, provide the information necessary, the context, right?

Right. So that there, you will use a rack pipeline, right? So when you don't know exactly which object has the data that you're looking for, then you will go and build a rack pipeline and you will know that, okay, this object has something.

Then on top of that, you will have a high hybrid approach where you can use rag as well as prompt object to finally get to the desired results. I'm just trying to understand where prompt object fits in this framework. Is it, is it just providing, uh, additional context to the prompt is that It's providing additional context to the LLM without you building, uh, more complex pipelines like, uh, extracting these informations, doing vectorization and then maintaining all of that additional vector database.

All of that complexity is kind of removed from you. So that is the key, uh, advantage here. Okay.

And so it automatically is doing, um, data transformation. Yeah. You talked about grabbing image data.

Yep. But also if I was grabbing text data, it's gonna Yep. So automatically know how to tokenize it and what tokenize to use.

Yeah, exactly. So we work with most of the unstructured objects like, uh, images, PDFs, videos, uh, all of these are supported right now. So maybe I'll just quickly show a demo that way.

It's, it's a lot more cleaner. So before, like I showed you in the Jupyter Notebook, we already have support for the browsing the objects. So in this case, you can open any of these objects.

And then we also have the prompt object integrated right here. You can be like, So here, the, what is covering this behind the scenes is the prompt object, API, and additionally, what we have also done is, uh, you can copy the code to do the same thing, and then you can just go directly to the Jupyter Notebook. And all you need to do here is instead of doing a get object, now you're doing, this is the bucket, this is the object, and now do this as a prompt is all the developer needs to do.

And, uh, behind the scenes we interact with the object store, get the object, and then transform it, and then pass it to the LLM and get the response back. So where this actually gets powerful to answer your question, is, so mainly developers, what, what they do is they want to transform and unstructured data into a structured data so that way they can work with it and build more robust applications on top of it. So in this case, now mostly the LLMs are unstructured to unstructured data, but you can make it, uh, emit structured data.

So here the developer can be like, this is the structure I want you to get me back when I do a prompt object. So once you have the pran object defined, all you need to do through the prompt object is you just pass that schema back to the prompt. And then when you trigger it, now, basically what you have done is you have transformed the unstructured data into a structured J that you can now take it and work with the application or ingested into your database and do analytics or any of that insight.

Yeah. But most Of these things, they're gonna wanna do a thousand objects. Yes.

They're not just gonna do one, one picture of an ID or something like that. Right? Yeah.

So that is where I'm going to go to next. Uh, so the main focus here was prompt objects should be only focused on a specific object, which is for the developers. Uh, but I also heard this feedback from our customers where, but I need to interact with multiple objects.

How can I do that? So that is the next thing that I will talk about. But just to finish off, uh, since Prompt Object works with open AI, API compatible inference servers, we also introduce N Media NIMS compatibility.

NV NIMS is, uh, uh, from Nvidia, uh, they have prepackaged, uh, containerized LLMs that the customers can run in their environment. Uh, and it's a lot more faster to run in the, specifically in the NVIDIA based GPUs. Uh, so they offer two variations.

One is hosted in the NVIDIA Cloud itself. The other one is you can just get the containers from Nvidia and the customers can host it in their own infrastructure, and then they can integrate it with prompt object, and we will take the rest. So you are your a a, a store prompt object integrate with, uh, the infrastructure, uh, with a CNET cluster running on this infrastructure with nims, right?

GPU? Yep. Uh, okay.

So this needs to be on external hardware? Yes. So for this, you will definitely need A GPU, And there is a chance that you can have minio on some servers, and in the same servers you will have GPUs and phone, uh, and use that.

Generally we re recommend compute and storage be two different, uh, things don't mix both the workloads, then it's very hard to manage. So in this case, AI store will be a dedicated cluster and, you know, GPU clusters is running somewhere next to the object store. That way it's a lot easier to manage both.

Because You said that the, the network is usually, uh, for, for the storage, right? Mm-hmm. Or your storage, usually the network is a bottleneck.

Right? Right. When you have an VME Yes.

So you can assume that there is still some CPU uh, unused in the clusters in the storage clusters. Yeah. And so you didn't, uh, yet at least thought to move that to, to mix that workloads together, because you still have some CPUs.

If you will have dedicated network to that, then still you can use that NIMS GPU notes inside your storage notes. We are also exploring a, if some of these MO models are ma maturing day by day. So we are also looking at like, uh, 1 billion models mm-hmm.

Which can be easily hosted in a CPU container. And then if your use case is dead simple, then go ahead and do that in your CPU. So there maybe we, you can think about that, but general recommendation from NIOS don't mix the workload of compute and storage.

Right. Um, alright, so the next thing, the, now that we have the model stored securely and then we are able to make intelligent insights, the next logical step for us to do is agents. Um, so we were thinking about this a lot.

Even when we were building prompt object, the main, uh, notion that we had was more than developers. We felt like AI agents will be the one that will be heavily using them. But at that time, uh, tool calling or function calling was the way to integrate these with the LLMs.

But now, uh, model context protocol was introduced by Anthropic recently, and every major provider like OpenAI, Google have already started embracing that. Even Microsoft has started embracing that. So what this does basically behind the scenes is it's just basically a protocol or a spec.

Uh, and you, you just build your tools as a MCP server and just expose it to any of the clients that supports this MCP protocol. So as part of this, recently we introduced AI store MCP, uh, which customer customers have already, some of the customers have started deploying it and founding AI store like crazy, uh, with a lot of AI agent calls. So what basically this does is, uh, I will just show you with the simple plot desktop integration.

So here I already have the AI store, uh, MCP server integrated, and these are some of the functions that we support out of the box. So here I can be like, and then behind the scenes cloud, what is going to do is it'll interact with the MCP server for AI store, it'll talk to the cluster and it'll be able to list all the buckets, uh, that is there in my cluster. So it's a hybrid solution, right?

Yeah, it's a hybrid solution. So the MCP server runs right next to the client, but the AI store is actually running in your private infrastructure. Could You go back to this prior that showed all the functionality and part of the, uh, no, no, no.

The uh, the demo actually you had, uh, a listing of all the functionality Oh, yeah. That the MCP server supported for. Yep.

Okay. So We also have it in our GitHub. So you can look at all the functions that we support, and we keep expanding the capabilities day in and day out.

Can you show me how the delete, how the delete works? I don't want to delete since this is a production and Oh, okay. That's what's exactly what I was thinking.

Like, you know, this can go quite unhinged, right? Yes. It can go quite hinged.

So since I, uh, I am, uh, I don't want to show my credits, but since I am the, uh, admin of this cluster, what I have is, depending on the access key controls that you have, only those are allowed to perform certain access. And within the AI store, the way the MCP server is built is by default, it's only read only, yeah. It can only read buckets, objects or whatever.

It'll not be able to do any create or delete operation. Mm-hmm. Mm-hmm.

You have to force it to, uh, do that. So that will be done most probably by admins. You're not brave enough to do the delete Now?

No. Okay. I'm not, Maybe it's an question.

When you have this, uh, MMCP, right? Mm-hmm. Uh, integration done.

Yeah. Uh, the communication between, uh, the, the public cloud and local MCP gateway can, I can call it like this, right? Mm-hmm.

Um, what kind of, uh, what kind of security matters are there, there? Certification, S-S-L-T-L-S, what you have there? TL certificates.

Rx TLS. Yep. Okay.

So it's a handshake between two TTLS endpoints? Yeah. API endpoints.

Yep. So, or through the VPN or through the direct connect, I can, uh, I don't need to directly present them right through the internet. I, I can so yeah.

Hide them somewhere. Yeah. So basically the client, wherever you are trying to do access the, uh, AI store, so if it has access to the AI store cluster, then the MCP server will also have basically the access to that.

Okay. Okay. Gotcha.

Yep. Thanks. So now going back to the question of, since we are only dealing with single object, using Prompt API, now with the MCP server, now what we have done with the AI agencies, now you can be like, I want to tag all the objects in this bucket, find what is there in this bucket, identify for each and every one of these objects, and then tag specifically for what is contained in those objects.

So that can be done purely from a cloud desktop. And here what I have done is Can you create a rag database upon all the objects You can, but in this, uh, MCP server With the MCP server, but AI store MCP server, you will not be able to, there is no need to do RAG because now you are able to do inter interactions with multiple objects at the same time and do the tagging and get, uh, interesting insights. So previously what I was able to do is, So now you can use the, we're not convinced, right?

Yes. He is not convinced. It doesn't look like he's, there are requirements that where RAG makes sense.

Yes. The requirements where it doesn't make sense if you're just going through, uh, a number of, uh, tag photos and trying to extract information that's not rag. If you're looking at a compendium of, uh, corporation X's, uh, uh, PRS for the last 10,000 years, uh, RAG is gonna be much more impressive for that.

Something like that. Yep. Or regulations, you know, that span volumes that a rag is gonna be, uh, very intrinsic to something like that.

Hmm. It's, it's not, it's not clear why you couldn't do something like a rag with this, but you're creating the database is one aspect of the rag and using it as another. So yeah.

So we, we did explore that strategy, right? So we, we, we did a rag based approach almost two years ago, and then we quickly figured out you have to be very opinionated on the rag stack that you built. And the tech is emerging constantly and keep, keep on evolving.

So in, in the rag pipeline, some of the bottlenecks that we noticed is first you need to do the embeddings. Mm-hmm. The embeddings itself is highly opinionated.

How do you do the tokenization? It has to be multimodal. It has to be, uh, able to serve for the specific user's needs.

Uh, if it's not multimodal, how will you handle different type of modalities and how will you do the search? But now that was a country storage, I think you have a lot of place for setting up metadata. Yes.

Regardless to any model you will use, you know, so this opinionation can be different between the models. Yeah, it can be different between the models. Now, as soon as we get into multiple different models now already the complexity of maintaining it and scaling, it's going to be really hard challenge.

Mm-hmm. So that's why we wanted to, uh, keep things simple and opinionated. That way if it's just at an object level, it's a lot more easier to deal with and still get a lot more insights.

Uh, for certain specific use cases, there are certain use cases where RAG will be definitely useful, especially when you're trying to make sense of entire objects in a given bucket and trying to consolidate and figure things out that maybe rag makes, uh, great sense. But in many of the cases, what we have seen is, as a AI store consumer or developers, they mainly interact with a specific set of objects. It's not always the entire buckets or data at a whole time.

So that's why we wanted to cater to that specific audience and, uh, target the solution to be, uh, scalable for those uh, scenarios. So in this case, now for a given object, you'll be able to do tagging for all of these images. In this case, I was able to ask, what was that in the object 11?

So this object 11 isus png, and it was able to answer this was lion appeared to be defensive and aggressive. And if we go back To the AI store AI data, PNG, you can see this is the image that it was able to do. And now using that, what I'm also able to do is you can tag, so do a tag of all animals that is present, and now what we are able to do is using prompt object behind the scenes, we are able to extract the information from the image and we are also able to do the tagging automatically.

Okay. And then you can move that to metadata of Yes, Exactly. So this way you can somehow, uh, classify your data.

Yep. It's not just that. And finally, you will also be able to build visualizations of your AI store cluster as well.

So in this case, let me just quickly show this and then let's move on. So here, the main thing that I wanted to do was build, build a visualization of my Astor cluster and using the function calls that we have exposed, it was able to get all the information and then build a clean dashboard. And it was also able to identify that one of the drives, uh, failing and as if you go here, you'll also be able to see that, uh, one of my drives is offline, basically.

So using this agent capabilities and the MCP server, now you will be as a admin or even as a AI or replication developer, you'll be able to do a lot more interesting things. Yeah. It's clever way you, you didn't put it yourself.

Yes. Excuse what is in the market. Yeah.

So finally, now that we have all the building blocks from storing the models and then doing insights on top of that, and then finally also doing agency capabilities, now you will have insane amount of workloads that is hitting your GPUs. So in order to address that specific needs, we are now doing AI store integration with the Nvidia nickel. So one of the bottlenecks is the KB cache.

Uh, when you do inference with all of these LLMs, uh, the model needs to do, uh, a pre-filling of KV cash and then, uh, do the tokens transform them, and then finally do the inference. When multiple of these clients are interacting with your inference server, immediately this can get out of hand and the GPUs are constantly churning and churning and they will get run out of GPU memory. So in order to address that specific needs, we also are working with NVIDIA to bring in object native KB store.

That way you can easily offload some of those already historical conversations or historical, uh, tokens that has been already computed into your object store. And then whenever it's needed, it NVD and Excel can automatically pull it and give it to the inference server. I always thought the KV cache was very transcript with respect to the prompt.

Is that not the case? In some cases? Uh, it, it is true.

Uh, it some cases though, uh, you are working with a specific context. For example, in Claude, if you noticed, they have a dedicated section called US projects. In the projects you can upload a bunch of documents, and that is the prefilled cache of context.

So that KV cache, you can constantly have it in the GPU memory or if the customer is not interacting, then you can offload it to object store. And then when the customer is coming and interacting, then you can load it without doing the computer again. So would you consider some sort of a, a size threshold or a time threshold for the, for whether to use an object KV cache versus a cheap, So in this case, NVD and Excel automatically does it?

Uh, that is a timeout internally happening at the infants, a server level itself, so it depends on VLLM or NVD Triton. They will be like, okay, this has not been used for a while, so let me offload it to Ram or object store.