Manifest CTO Daniel Bardenstein on Creating AIBOMs to Extend SBOMs for AI Transparency

Hello and welcome to the latest edition of the Techstrong AI Leadership Insight series. I'm your host, Mike Baard. Today we're with Daniel Bardenstein Stein, who's CTO for Manifest, and we're having a chat about the need for software bill of materials known as SBOs for ai.

Daniel, welcome to the show. Thanks so much for having me, Mike. Pleasure to be here.

Alright, we have what's known as SBOs that are gaining some traction in the traditional world of application development. At least we think that folks are doing the right thing, and we're getting a little progress on DevSecOps. But, um, in the, in, in the world of ai, what's different here about the way we need to think about a software bill of materials and, and what should people be kind of working through?

So about 18 months ago now, I embarked on a, uh, a bit of a research, uh, adventure with a, started with a very simple question, which is, given that bums are taking off as this globally accepted way to think about software, supply chain and transparency and software, how much of this applies to ai? Right? There are many of us in this space who believe that AI is in many ways a subset of software that you can't build or use AI without putting it into software.

Therefore, how much of those common risks that we think about on this, uh, for software supply chain apply to ai? And after consulting with hundreds of experts, much smarter and more technical than myself around the world, it became very clear that many of those same classes of risk that affect organizations around the world, um, that most people think about when it comes to software supply chain security, also apply to AI security. Um, and I'm sure we'll get into the details, but everything from thinking about sources from open source risk, third party risk, continuous monitoring, lineage licensing, all terms that if I said them in a securities software security context, CISOs around the world would say, yep, I understand what all those things are.

And then you turn around and ask, so how do you plan to tackle that for ai? And most CISOs would probably scratch their heads and say, I don't know. I don't have a tool to do that.

Mm-hmm. And, uh, and, and I think this is all also bolstered, bolstered by two fact or one fact, and one piece of news to your point about the adoption of SBOs, even last week, CISA published, um, uh, uh, a document that was co-signed by 19 different governments around the world, all effectively endorsing SBOs is a very powerful tool for software supply chain security workflows. And the other fact that that makes us all the more urgent is we all know that whatever percentage it is, 90 plus percent of CEOs around the world are basically telling their companies to accelerate AI adoption as much as possible.

And that's putting security leaders in the bind because they don't have, have the tools and processes to catch up with this rapid, rapid proliferation of AI systems. Mm-hmm. If I do have an SBO program in place, can I extend it to AI and is it really just another piece of software or a different set of art artifacts, or am I gonna need a separate framework for AI and SBO m That is exactly why we kicked off the AI SBO m working group under csa, of which I'm, uh, one of the co-chairs to help answer and educate the, the public on exactly these questions.

So it's my goal that people shouldn't need an entirely different set of processes or frameworks to handle this stuff. The last thing an enterprise security team needs, you know, I think the average enterprise security team has somewhere between 15 a hundred tools right now. The last thing a need is to multiply that number by two, to have an AI flavor of their endpoint tool and their firewall tool and a DLP tool.

But they, they will need Existing tools to adapt to the in, uh, intricacies of generating AI bumps, scanning models, scanning data sets. There are different types of risks that we're looking for in different ways of searching for those risks. And so what I continue to, um, advise security leaders, both in public and private, uh, industry is as much as possible user existing processes and frameworks.

But you need to be able to think about some of those more tactical bits, the ingestion, the scanning, the monitoring a little bit differently, um, as it pertains to AI security. But ultimately you want something integrated because even if you've developed the best model in the world, at some point you gotta put it into your software. And so you need to have those two things integrated rather than having two new siloed security systems.

What level of depth can I get to? Because a lot of times somebody will, for example, build an AI agent and then it's invoking an API to an LLM somewhere. But how do I know what went into the LLM and how do I discover that?

Fantastic question. So one of the things that we've been hard at work at building is, uh, basically an AI bomb generator. So that, for example, helps our, um, partners and customers get from a model that's pre-trained, that's out on a HuggingFace into a robust history and lineage of how this thing came to be, what data sets it was trained on, the lineage of the data sets, who put them together, how are they licensed, et cetera.

So we can help organizations already go from, here's a Google Burt model or a Meta Lama 3, 1, 3, 2, whatever it is, and get to a very robust story about what is this thing, where did it come from, how's it built? Can I use it legally for my specific use case? Um, when it comes to something like, uh, an agent system or an MCP, this is, uh, another great use case for why we want these tools integrated with software analysis tools because an agent at the end of the day is just software plus ai.

It's a little bit more recursive, but we wanna be able to scan source code to figure out what APIs is it calling, what tools does it have access tool, what credentials might it have, how do we put guard guardrails around that? So it's another great example for why, you know, an S bum isn't the the cure to all security evils, but it gives a very, uh, robust and structured account of how something is built and perhaps what it can do. And that's why it applies so well to models and data sets.

And then the ultimate vision here is not to create a brand new artifact, but you know, I'll, I'll provide examples. You know, we work with the next gen defense contractor that's actively being asked for AI s bombs from their government customers. And so what they end up handing over is one software bill of materials that not just lists the normal dependencies and licenses that they put in their software, but also the models that are included as well, and how they were trained in information about their providence and lineage.

So the goal is not to recreate a new framework or new standard here. It's making sure that organizations have the abilities to ask questions about their ai, their models and the data sets, and then put them into the formats or compliance artifacts they need to actually do something with them. How will we keep up with the dynamic nature of some of these applications?

'cause um, likely to have multiple AI agents that will eventually invoke multiple LLMs and the LLMs may change. And it just seems like the pace and the rate of change is gonna be very high. So how do I kind of keep track with all the updates and changes to the underlying software?

And the same way that we do already with traditional software, like more, uh, very mature modern organizations with modern CI/CD pipelines push software to production multiple times per day, certainly much faster than LLMs are going to be regularly trained and tuned. And so it's another example we're learning from AI security and how to get a, you know, take a big step should come from what we've already been doing on the software side. So if we look to the software security side, how do we make sure we are regularly scanning code given that it's now being written even faster with tools like Cursor and, and rept is we have automation in the CI/CD pipeline.

So as developers write code and they push the code, there are a bunch of scans or processes that kick off that find various types of risk enforce policies and then help tell the developer, Hey, you need to go fix this thing, patch this vulnerability before this goes to production. We just need to replicate that on the ML side of the house. And we've also already seen success doing this as well, right?

Whereas we have the CI/CD pipeline for software, we have the MLOps pipeline for AI and ml. And so in the same way that there are automated tooling that helps developers write code quickly, find issues and fix code, we need to replicate that for model developers. So if I'm fine tuning a model or updating a model or quantizing a model, how do I understand if there's anything that I'm doing risky as I'm doing it or find issues automatically that get pushed to me before I've finally saved this model and put it in my model registry?

So at the end of the day, it's all about automation. It's all about plugging things in as far left as possible. And ultimately, again, we're trying not to recreate the wheel when it comes to AI security.

There's a lot we can learn here from existing software supply, uh, software security analogs. Should we be working towards, um, unifying these pipelines a little bit? 'cause I think in a lot of organizations you'll see, you know, something that looks like a TIGER team is off building an AI project and they may not have the security best practices in place and fact many of those data scientists probably know less about security and then the average developer.

But, um, should we be rethinking all these workflows to bring our existing pipelines and DevSecOps workflows and apply it to the development of AI applications? Short answer is absolutely yes. We continue to see examples where you have employees or like you said, small groups just going off and building application, uh, ML enabled applications that may not have, uh, security best practices in place.

We've already heard and seen stories of people that try to circumvent compliance needs for AI ML by, you know, doing various sorts of trickery. Again, ultimately, again, just like with software, we work with some of the largest organizations, uh, around the world and they have often have different business units. And each of those business unit is often treated like a, you know, special snowflake.

They have their own processes and tools inside the business unit, but you still often have a central security team across the whole enterprise that's responsible for making sure that there's consistency and inventory and scanning across the business units. We need the exact same thing for, for A NI ml. So how do we make this happen again?

The last thing we wanna do is slow down innovation to get in the way of data scientists writing, um, you know, developing new models for their use cases. But this is where automation and integration are so key. So as they're doing their work, there's tooling that's in place that can tell them, Hey, you tried to load a model that is unauthorized, or this data set that you tried to use from the public internet isn't licensed properly.

So you're able to alert them to issues before they go deep on training them. And then I think the fundamental issue that most organizations are still struggling with here, which is the most foundational, is inventory and awareness, right? You can't secure what you don't know about.

We all are familiar with the concept of shadow it when the cloud became big 10, 15 years ago, now we're dealing with shadow ai. How do I know what models are being used across my enterprise? How do I know if we put some custom model in a medical device that the cardiology business unit wrote it or the, you know, pulmonology business unit wrote it.

Organizations just lack basic inventory and awareness about what are the models and data sets we have across our enterprise? Where did they come from, who built them? Or they're being deployed And you know, asset management is a hard problem as you know, um, but that doesn't mean we can't help organizations try to take a big step forward on it when it comes to ai, Most of the AI software that I know is pretty much built using the same tools and components that we use to build other applications.

So they would naturally have the same vulnerabilities. But are there also unique things that people should be looking for in AI applications that are attack vectors that they might not be thinking about? Fantastic question.

So as most people probably know, models can have traditional software vulnerabilities just like software. Can we think of your pickle serialization or pickle de serialization threats, for example. You can exploit the software itself, but there are lots of other issues when it comes to other types of business and legal risk when it comes to models.

So for example, responsible AI licenses in traditional software licenses. We think, you know, permissive Apache two is good strong copy left. LGPL is bad with AI licenses.

They're use case specific. And this has already caused trouble with some very large, um, companies and government agencies where I might legally be able to use a model like LAMA three one for something like summarizing emails, but I can't use it for anything with heavy manufacturing or defense or military or biometric applications. And so there's a, a significant amount of legal risk based on how you can use this model.

It goes even deeper than the model because it also depends on the data sets. So we worked with some very mature organizations that aren't allowed to use models that were trained on illegally gotten or improperly licensed data sets. So again, we're back to kind of a legal business risk because if that gets put into a product or a weapon system or a plane that might have to get ripped out at some point and, you know, ultimately cause revenue lost for, for organizations, you also have other sorts of, um, traditional supply chain issues.

Like when deep seek was announced, many people found various types of bias that, you know, the, the Chinese authors of the model may have written into the system prompts that would affect its output. So understanding who the supplier is of the model on this dataset, are they trusted, especially for those who work and sell with the Department of Defense in the us. Um, there's just a, a memo that came out a few weeks ago saying that there can't be any nexus to China and Russia in DOD acquired systems.

So there's a compliance burden. Um, and there's a, just like with software and the concept of, of Lineage and Providence, the same is true for, for models. There's a story with one of the world's largest IT companies that had an explicit ban on, um, a model like Deep Seek for example.

And there's a user in a business unit that really wanted to use that model for a use case, took the model from the internet, uh, fine tuned it with some lightly with some data, and then declared that he had a brand new model that wasn't related to any noncompliant models and tried to get that one approved by his compliance team. So again, going back to there's yes there traditional security issues and exploitation of models and data sets, um, for prompt injections, your data poisoning, et cetera. But there's a whole raft of new issues around business risk, around legal risk, around compliance risk as well that comes with models, data sets.

Because at the end of the day, these things are black boxes and we need to understand how they were built and how they were trained Under the heading of physician Heal thyself. Will we at some point see AI tools for creating SBOs for AI applications? Yes, and we've already started, uh, along that path.

As with anything else, the balance of AI is comprehensiveness versus accuracy. So we've been able to generate some SBOs using AI and be able, uh, are able to extract information that traditional software scanning tools or sbam generators don't necessarily find. But you always have to make sure that they're not hallucinating, that the information is, uh, is indeed present and kind of validate the findings.

So the short answer is yes, and we, and we've personally used this, um, uh, within Manifest for example, to help solve some, um, difficult to solve problems around extracting dependencies that are just an unstructured code. For example, things in c and c plus plus to get very technical about it. So ultimately, yes, there's a role for SBOs to help secure ai.

There's all and security. There's also a role for AI to help bolster SBOs and security. Um, even talking about things like vex, the vulnerability and exploitability exchange, which you know, as a companion document to SBOs help organizations save time, you know, responding to vulnerabilities.

There's a role for AI in generating and disseminating these documents as well. Alright, so what is your best advice to folks then who are just getting started with this whole thing and um, where should they be focusing their efforts in the short term? Because generally speaking, they're gonna have to deal with it eventually, right?

Absolutely. It's certainly a matter of when more than if. My primary advice to organizations that are already launched down the AI adoption path or are soon to, especially for SEC from the SEC for the security practitioners, is first to understand what their AI risk policies are, what allows them to, uh, use, um, an external model or dataset and how they validate whether something is secure and trusted.

So first you have to know what good it looks like or what bad looks like. The next step is to actually start applying those policies automatically, right? So if somebody in the business unit asks, Hey, I found this model out on the internet, that's really good, good for this use case.

Can we use it? How can you get to a yes or no answer as quickly as possible? Third, then is all about inventory.

So once you define what good and bad looks like and you now have a formalized process for how AI gets into the sys the, the organization, since most organizations aren't building their own models from scratch, you need to build an inventory because without that inventory and awareness you can't effectively secure. And then from there, there's some more advanced steps around how do you track all of your custom models or where the models get deployed into software. But it all starts with defining good and bad and making sure there's a central process by which models and data sets are adjudicated when they first come into the organization.

All right folks, you heard it here. Transparency is gonna be everything. And right now we might be living in a age of black boxes, but eventually we're gonna know exactly what happened, when and where, and you're gonna need something that looks like an SBO that help start that process.

Daniel, thanks for being on the show. Thanks for having me, Mike. It was a pleasure.

All right. ai Leadership Insight series. You can find this episode and others on our website.

We invite you to check them all out. Until then, we'll see you next. Uh.