AI Model Security and Governance – Broadcom VMware Private AI Model Gallery Demo

I'm Tasha Drew. I am the Director of Engineering for AI and Advanced Services in the VCF division at Broadcom. And I'm excited to do a deeper dive into model security and governance today, uh, and how you can achieve that with VMware private AI foundation with Nvidia.

Uh, so for uh, folks who are just joining us, uh, this is the solution, uh, that I'm specifically referring to. Uh, so this is the VMware Private AI Foundation with nvidia. Um, I know that we just did a great deep dive into the features and capabilities and some customer use stories, uh, with the previous presenters, Justin Murray, Alex Spanos.

Um, if y'all haven't seen that already, highly recommend you check it out. Uh, for my session, I'm going to be doing a deep dive into the model governance capabilities that come with the VMware Private AI foundation and walking you all through why this is important to enterprises and how we can achieve it, uh, within the product. So everyone's hearing a lot about agents.

Um, I heard a bunch of questions about agents agentic workflows. Uh, how does RAG fit into that? Uh, during the previous presentation, which I loved 'cause I was ready to go with my agents slide.

Um, but for folks who are wondering, um, about how we're seeing the agentic workflow, uh, architecture, basically, uh, how folks are thinking about this right now is, uh, the big L LMS are awesome, primarily as a really nice user interface because I feel like I'm talking to a person, um, and I'm getting high quality text-based results through this LLM interaction. But how can we make, uh, this LLM interaction even more correct, is by starting to follow an agentic architecture where the LLM is acting as the primary interface for the end user. Um, but then it can go and use tools, um, and of, and have a multi-agent workflow, uh, in a way that can get you more accurate information and information from multiple systems, uh, in a very programmatic way.

Uh, so this is again, kind of gonna show everyone what's old is new again, right? Like, or nothing's new under the sun, but kind of thinking through this. An example is, um, maybe you have an ag agentic, uh, an agentic service that is going to be helping your sales team.

And so someone from your sales team comes in and just says to the agent, Hey, I'm presenting to customer a in two weeks and I wanna talk to them about our AI strategy and why we should work together. So this question in this workflow would then go into an orchestrator or routing agent that's an LLM that this user is interfacing with, but this LLM can use tools and it can use other agents to answer chunks of the question. And so I might go to a function tool, uh, that can actually just search the public internet for every public facing statement this specific customer has ever said about how they're using ai.

Um, so pretty straightforward. That doesn't sound like AI to anybody, but it's something we know that computer systems can do. And that's kind of the point of this is we already have a lot of great tools and services that aren't specifically ai, but if AI can call out and use those to make its own answers better and more accurate and more actionable, then how do we chunk that out into a tool's workflow to allow them to do that?

So in this use case, we're gonna search the web for everything the customer's ever said about AI and parse that into something that might be interesting and actionable for our sales engagement with them. Um, we can then go to another source and go through everything internally that we have from sales enablement and pull in all of the right decks, all of the right articles that we've already put together for the sales team. Um, what is our strategy around ai?

What products and services do we have? Um, and then we can go interact with other systems agents. So a lot of folks have heard about all of the investments, uh, Salesforce has been doing with Einstein.

I could go and use Einstein in my SFDC and get and have Einstein answer specific Salesforce questions based on the data that my Salesforce implementation has. Um, I could also go and search a Vector DB or re-rank my information to have the best information, uh, return results to me at the top. All of this information goes back to the LLM.

The LLM then writes me a really nice answer to my question that has links to sales decks, that has public statements from the internet, and that has information from my own SFDC implementation. It then goes through security guardrails to make sure that, um, all of the information being passed back to the user, the user is credentialed to receive. And then I get a really nice complete response.

So this is the agentic workflow that we've been hearing a lot about in this workflow. A rag application would be one tool or service or agent in this backend, um, that might be doing a specific thing for you. Uh, so just kind of wanted to level set before I kind of dig into what is the thing that makes all of this work?

The models. So as enterprises start to look at these really exciting implementations of AI and how they could work, put this into their business processes to make doing something that used to take people hours, days of their time, like, uh, putting together a really nice sales presentation for a specific customer, suddenly AI can help me assemble all of those materials to me instantaneously reducing the amount of effort I have to put into that and increasing the amount of time I can spend on the actual interactions and the actual sales deals. Um, so digging a little more deeply into how we serve rag agents in specific, here we have the user they send in the query, we have the rag agent running.

The rag agent can then quickly do a sim, just a similarity search against our vector db, um, and return relevant con uh, return relevant context. The rag agent can also send a prompt in context to an LLM model endpoint and generate a response. It then composes that, that data together to return a response.

So why does everyone talk about rag agents all the time? It's because I can have an LLM model. I can then say, only get your responses from these documents that I'm asking you to search from.

Don't guess, just tell me based on the documents I'm attaching to you, what is the answer. And so this means that we can start getting more accurate information, but also more tailored information out of our rag agents. So, uh, when we start talking about multi-agent workflows, that's really because you can quickly see that in a large enterprise you're gonna have a lot of these rag agents lying around for different teams and different use cases.

You're gonna have probably several different a HR agents per geo, um, and per different levels of information. You might have one for the finance team that has finance data attached to it, but this architectural pattern, uh, is, has become very common and very impactful. Um, one of the easiest use cases to sort of, uh, immediately kind of glom onto is in legal firms doing documentation discovery used to take, they used to ship out dozens of paralegals, uh, usually somewhere in Texas to just sit and read docs for weeks and months.

Um, and they would just be sitting there reading the docs and taking notes. Okay, now we can PDF, all of that docs, all of those docs attach it to an LLM, tell it this is what I'm looking for. Only look in the docs return to me where you found it so I can go read the doc to make sure you that we're completely accurate and we're not hallucinating.

And now suddenly something that was taking dozens of people, dozens of weeks is something that I can just do simple searches against. Um, so that's kind of the power in there, but what is it all powered by? It's all powered by models.

And so as an enterprise, now that I'm starting to realize that I can start to operationalize and automate all of these different capabilities that used to be pretty intensive and frankly not very rewarding for anyone involved in doing the grunt work, how do I achieve model governance to make sure the models I'm bringing into my organization are safe and accurate and not exposing my organization to risks from a number of different vectors. Um, so kind of, uh, what are the areas that we need model governance for? We need to think about model security, model monitoring, model evaluation and training and model scalability and sort of fleshing those out.

What are the things that folks need to start worrying about? Um, I often liken where we are with models coming into enterprises with where we were with containers in 2015. So there's a lot of really exciting.

In 2015, we have docker hub folks are really excited, Hey, I can really speed up my development work, work practices by going and grabbing a container to do local development. You have your developers going to docker hub downloading something called Redis. It wasn't actually Redis.

They've installed it on their laptop. Worst case they've even installed it on a server. And suddenly you have infected software behind the firewall.

So models carry a similar risk. There's been academic studies that have come out lately showing a lot of supply chain injections and problems on popular hubs like hugging face. Uh, and so we need to start kinda looking at this, this, okay, it's a model, it's cool, it's ai, it's awesome, but you're still bringing software into the enterprise.

What are our guardrails around that? And how are we following our typical enterprise process for securing our infrastructure against this software? So your your first example of legal.

Yeah. Right? So this notion of I think the market would call it e-discovery.

Yeah. And so the power of e-discovery was to find the needle in the haystack because you have a 40 ton electromagnet. Yeah, okay.

But that's a legal context. So everyone there has 'cause of their profession. Yeah.

Uh, there is all that confidentiality. You know, there's a clean room and all these concepts, but you're, you're describing you're enabling that sort of same behavior, uh, or, or use case, but widening that to the enterprise. Mm-hmm.

The second part is that, because again, we're we are going from a legal example to a not legal example, you know, being non-legal profession in the enterprise. Yeah. And because someone might, you know, pearl questionable thing, pipe it to bash, you know, straight into the system.

Yeah. That, that lack of governance in that example, that's, that's where you're talking about these and there's many different lists of all the different AI ways one could go wrong. Yes.

Um, um, what's, what's your source of truth of how you're determining, I mean there's, uh, what is it seven, uh, you know, bad yellows on this page, I guess. Yeah. Um, where, what is your guiding, you know, or your starting point for, you know, whether it's these categories or is there like super expansive or, you know, approaching exhaustive and then new things come to life, like, ah, we need to worry about this too.

Are these the general areas that you're thinking about? I think these are the general areas we're thinking about. We're looking at how do you operationalize ensuring the safety of the models you're bringing into your organization.

Um, thinking of it from a software perspective. So it's the software supply chain. Yes.

And, and, and in the bill of materials, the attestation that what's here is not just what we thought it was. Yeah. It actually, you know, We've tested it, we've verified it.

Okay. And yeah, we're not just spreading it. Are, Are you, are you aligning with a wasp and mitre?

And because that's, that's the sort as much as we can just like, this was for Yes. Vulnerability libraries. That is the closest we'll get the social truthful model and embedding vulnerabilities and jail breaks and all those.

Yeah. Okay. Yeah.

So, so I think kind of the first takeaway is make sure that, and I know that this sounds silly to everyone in the room, but I'm just gonna say it, just make sure you're scanning them the way you would any other container you're bringing into your infrastructure, right? Step one, the library. Um, how do we do that programmatically with our solution Harbor already has that brought, has that as part of it, right?

So you import the container, let's just scan it. Let's make sure there's no actually vulnerable software, because we have seen that. But step two is the model behavior is also a route to, um, unsafe models in an enterprise, right?

So there's all these, there's an entire collection of company, you know, das, RAs and all these, you know, application scanning, utilities, you know, code verification. Yeah. All that.

You're not proposing that you are taking over those responsibilities as a product. Um, we are still in the world of tofu or trust on first use or however you want to say it, but it's up to the end user. Yeah.

The entity, the organization, uh, to, to ensure what they're using is what they think it is. Um, where, where would the VMware attestation of what you're bringing in, when does that begin? So what I would like to show you, and this is what I'm gonna show you in my demo Okay.

Is how I think you can operationalize that using A-C-I-C-D pipeline. Okay. Right.

And so there's a lot of really great open source tools that we can use to start programmatically testing model behavior to ensure that it meets the standards of what we wanna introduce to our teams. And we can also start to programmatically test for model behavior drift to make sure that over time the model behavior isn't veering into a place that we're not happy with. And I think that we can programatize this using what we already have in, um, in VMware private AI foundation and some open source tools that are really nice.

Yeah. 'cause Gina earlier had asked about currency. Yeah.

And currency is just the versioning part. Yeah. And then there's like the, at the very, very beginning, like root of trust.

Yes. Do, do I believe this is what I believe it is. Exactly.

That's, that's what I'm really curious about. Yeah. Uh, I'll just go super quick and get straight to the demo.

Uh, so here's, uh, a good model evaluation workflow. So your data data scientists team is going to wanna select what models I'm testing, and then they are going to request an AI workload, load the model in, and then they can ab test figure out what the best model is and then upload it to Harbor. But backing way up, um, how do we programmatically pull these models in?

What should we be doing? So this is really something that my team has been thinking a lot about. Um, we have the deep learning vm, the AI workstation, uh, that's productized as part of private AI foundation.

What we're telling everyone is, look, if you're bringing in new software, you should be running that in A DMZ 'cause you're connecting it to the internet. You can connect to the internet. The deal VM comes with, uh, the hugging face CLI or the N-G-C-C-L-I.

So you can easily connect and interact with those systems to pull in a new model. We pull this model into the deep learning vm and within that DMZ, we should be scanning it for vulnerabilities. And we should be testing the model behavior to make sure that it's something that's appropriate for our enterprise and meets our enterprise needs.

Once we've done that, then we use the private ICLI to push the model into Harbor Harbor regular regularly security scans, all the artifacts in there. And then as a user, when I'm in my AI namespace, I can go to whatever projects I have in Harbor that I have RAC permissions to interact with. Um, so as an org I could have a public repo that has a bunch of models that we're just providing to all of the teams.

And then I can have team by team models that are more specific or potentially, uh, more testing for the data scientist team or some model that has been trained on, um, data that only my team should be able to see. Uh, so kind of have that restriction just baked into Harbor. So pretty straightforward.

Um, why do we think models, uh, harbor's a great, uh, choice for model storage open source. We're aligning with all of the open source specifications that are going on, but also there's a lot we can do to improve the overall workflow of deploying models at runtime. So today, um, and this is something that I've heard from a lot of data scientists, if they go to hugging face or they go to NGC and they download the model they want, the documentation's not great and it takes a lot of work for them to figure out how to operationalize and correctly tune this model for the hardware they have and then run it well on the hardware they have, how can we operationalize a lot of that?

I, we are exposing metadata fields within the OCI artifact that we're bundling the model into so that we can just say, this is how I should be run, this is how I should be tuned. And we can start to programmatically map them that, uh, runtime behavior at the build time. So the expert who's selecting the model and providing the model can give the person who has to run it later as much information as possible to sort of decrease that gap in knowledge.

Um, cool. So, you know, the key pillars of model governance, centralization, accountability, compliance, and safety. Folks are only start starting to kind of realize that, uh, they can't really share models that they've fine tuned, right?

Like I've actually heard that from a bunch of folks in the health and health and healthcare industry is they see other hospitals and other research centers doing these amazing AI work and they're like, Hey, can we work with you? Can we use your model? And the answer's no because that model's been exposed to, uh, to very protected data.

So you can only share first principles and you kinda have to start from the very beginning to then get that experience for your use case. As folks start to kind of better wrap their heads around this, I think that we'll start to see people train models more thoughtfully in a way that they have some sort of, uh, more parent model that hasn't been exposed to sensitive data that then gets fine tuned for their specific locations so that they can, there can be more collaboration across orgs. Um, so I'm gonna dive into the demo since I know I'm running out of time.

So we're gonna kick off. Um, for folks who haven't seen this screen before, this is the VCF automation page. And this is where anyone who's using the private AI foundation solution would come to if they're a consumer, um, of the infrastructure.

So these are all blueprints that I can click on. And then self-provision and AI Kubernetes cluster, a rag cluster, a workstation, um, or Triton inference server. So just kind of self-service, get my infrastructure up and running.

Um, so here, this is our, uh, user who's going to be providing models to other users. So more of a platform user. And they are creating their own AI workstation.

They're gonna say what software bundle they wanna use. In this case it's High Torch, and then they're going to start their AI workstation. So here it is, we're now within the deep learning VM and we're going to import a model, uh, from hugging face.

So let's do that. So here we go. We're connecting to hugging face.

Uh, this specific model uh, requires us to credential ourselves on hugging face to download it. Uh, different models have different requirements for that. Uh, and so we've successfully authenticated and we are going to import our model.

Here it comes. All of the fun downloads you can tell we sped this up so that we didn't bore anybody too much. Uh, and now I can see the file structure of my successfully downloaded model within the deep learning vm.

So next step, we are going to go and prove that this model has been, uh, started. So we're gonna start running it. Um, now we can see it running and now we're going to prove that it's, uh, actually up and serving inference.

So how am I gonna do that? I'm asking it a question. I'm gonna ask it, Hey, can you generate me some code in Python to generate prime numbers?

So it generates me some code. It then tells me that its code is not very good and not to use it in production. Um, so this seems like a trustworthy model, uh, and we're gonna move forward.

Now what we're going to do is we're going to set up this system, and I'll just kind of slow this down for a second to basically red team this model using an open source tool called Gift guard. Um, so we're going to have gift guard go in and test this model for inappropriate or insecure behavior, um, using another LLM to red team it. So one LLM is gonna try and get our LLM that we just downloaded, which is LAMA three to say bad things.

And then we're gonna get a report on how that went. Um, so we're kicking that off. Um, and now we can see kind of the, the post requests going back and forth as we try to get our model, uh, to misbehave basically, or act outside of parameters that we would want it to.

Um, so this is the UI for Gift guard. Um, and again, this is a completely accurate model, uh, from Meta LAMA three. And you'll see that it did not pass every test because this is a very aggressive test.

So kind of one of the points to take away from this is that no model is going to perform perfectly. What you have to decide as an organization is what your acceptable risk parameters are. So I can now dig into the HARMFULNESS results that it failed any hallucination and misinformation, it generated sensitive information, disclosure problems, any prompted injection issues that had, and any stereotypes.

These are all the tests that did not get a green on. I now can dig into like what it did in each case and decide is this acceptable for my use, for my use case or not, or do I want to try and fine tune this model so it comes up with better results. So here are just some examples where, uh, you know, it failed.

Like it gave me some information about explosive stuff, right? Like, you know, just kind of, those are the types of different tests that you can run. You can also put tests in that are specific to your organization and your risk boundary.

So I've decided that this model is good for my organization. It's successfully passed the tests that I've given. So now I'm going to use the private ice, uh, CLI, I'm gonna push this model up into Harbor.

We've wrapped it up. We can give it tags to tell us, Hey, this model is approved and now it gives me the command to pull it on back down. Um, quickly showing you all what this looks like within Harbor.

I can list what models I already have up there. In this case, I have LAMA three, which I just put up there in a couple versions of GT two. And I can see how I've tagged some of some of these as test and some as latest.

And then this is what it would look like to a UI consumer notice. As always, with hardware, you can, uh, control the access level. So this is a private repo private project, and only folks who've explicitly been given access to it within my org would be able to see it.

Okay. Um, switching personas. I'm now going to be a developer persona who just wants to start using that model to do my job within a deep learning vm.

So I'm gonna start up a Triton inference server. Um, I can change one parameter if I want to, to have Triton start automatically with no commands for me. Uh, and that's what I've decided to do.

Um, I'm going to start up my AI workstation. Uh, here it goes. We're booting everything up and I can now access it.

So here I go. I'm going to get into my workstation and pull my model down. So here I'm pulling down that version that's now been stored for me in Harbor.

I can, uh, pull it down and now I'm going to show that it's running, uh, and then do my inference check to show that it's also serving inference. So because we changed that one parameter, it's automatically running. I didn't have to do anything.

Uh, so I'm already up and serving inference. So, um, you know, I kind of said this earlier, like this is a manual example of something you can automate as part of a ci cd pipeline, right? Like you could have all of these elements running in your CI infrastructure.

And then as you introduce models, as you have new versions of models coming through, you can run it through not only your standard container security test, but you can also run it through gift card. You can start to have sort of an understanding of your risk level programmatically, and then you can turn that, tune that to the needs of your organization.