AI Model Provenance and the Limits of Country of Origin

Hey everyone. Welcome back here to Techstrong TV. Let me introduce you to our next guest for today's show.

Her name is Amy Chang. Amy is the head of AI Threat and Security Research over at Cisco. Amy, great to have you on Techstrong TV.

Welcome. Thanks, Alan. Great to be here.

No problem. Amy, we're going to jump into the AI Threat and Security Research team at Cisco. But it kind of begs the question with you as the head of it, give us an idea of your background.

How did you come to head up this team? Sure. Yeah.

My background is about two decades in cybersecurity, with the last several focused on AI security in particular. As I think the audience knows, there are some overlaps between traditional cybersecurity and AI security, but there are also a lot of different things about AI systems. And what my team does is look at how those systems can fail, how they can be compromised, how they're susceptible to different types of attacks, and then how we can mitigate them.

So we do everything from understanding how models can be compromised throughout their whole development life cycle, but then also pushing forward best practices, mitigations, and considerations. Got it. People tend to, when they hear threat and security research, immediately the vision is hackers going out on the dark web and undercover, so to speak, and blowing up or at least exposing criminal gangs and nation state actors, and discovering the latest threats and so forth.

And I've been in security 25 plus years, and I've had a chance to talk to a lot of the different threat and security research teams across the industry. And quite frankly, it's not always that exciting like a TV show or some movie or something. A lot of traditional security research, threat research, and now of course in AI, is really kind of block and tackling, going through sound best practices and seeing where are the mismatch, where are things happening.

It's not cloak and dagger kind of spy things. Talk to us a little bit about how this plays out at Cisco, if you will. Sure.

Yeah. And I think you nailed it on the head in terms of, it's not always the sexy headline-grabbing things. And the adversaries are just as lazy as the people who develop the technology that's susceptible.

They just want the easiest path, and if that works, then that works. And, I think for us and our team, what we've done is kind of showcase in many different ways throughout the AI development life cycle, whether you're talking about the models themselves or the models in deployment, how easy it is to compromise them. I'll give you an easy example.

We've put together an LLM security leaderboard that is publicly available, that shows how models from both frontier labs as well as open weight models, are susceptible to very common attacks, such as jailbreaks and prompt injections, where you basically steer the model to try to do or say something that it's not supposed to do. And in the course of maybe five turns between the user and the model, we were able to elicit up to 80 something to 90 something percent attack success rate. So that's just to show that even when a model or a company that develops that says something is, quote-unquote, secure, you still need to kind of red team or pen test that use case, especially for your specific scenario when you want to take in a model and deploy it within your organization or have a customer-facing asset or even attach sensitive documentation or assets associated with it.

I don't know if people out there really realize this. Oftentimes, supposedly all these models have guardrails built in and things that prevent you from doing harmful stuff. And more often than not, it's exactly as you said, it's just a question of reprompting or going through a sequence of prompts that will allow you to kind of, in essence, escape the guardrails.

I mean, we've seen it again and again. And I don't know if there's something inherent in the design of the models that allows us to do this. I mean, obviously, because otherwise they would have taken it out.

But do you think we'll see a time where we do a better job of erecting guardrails and that it's not just a question of reprompting a few times? Yeah. I think it requires thinking beyond what the guardrails could do, whether it's the guardrails in the model or additional guardrails that you impose in your deployment environment.

So now that we have agents that are basically powered by those same models, you have to consider, let's treat them as an untrusted asset or an untrusted human. What are the things would you add from a protection or mitigation or control standpoint to ensure that that untrusted entity doesn't go and do things that you don't want it to? And I think going to what you were saying between being very specific about, oh, if you're going to refuse this prompt, I'm going to twist it a little bit to try to see if I can find that gray area where you do comply, you being the model.

But then I think on the other hand, there's the opposite problem with which if you're too broad and you just say, "Solve this problem," and you don't provide any direction, that provides the model too much latitude to try to figure out, well, I need to solve this to satisfy the prompter. And that's how you have something like the OpenAI Hugging Face incident that has really been on top of the headlines over the past several weeks at this point. Yeah.

And to be fair, it wasn't just OpenAI breaking into Hugging Face. All the cool AIs were breaking out of their guardrails and hacking into something, right? That's right.

So it poses a problem, but here's the good news. I do think this is a phase. This is a stage where we're at, where we're still learning what's the best way to do this.

Right? Right. And we'll get better at it.

I'm sure of it. Here's another thing, though, Amy, that I think, and I've heard this from people in talking to them. Everyone knows Anthropic, everyone knows OpenAI, the frontier models, and especially here in the West, there's a, I don't want to say an understanding, but an expectation that they're trying to make these models secure, that any insecurities we find are not intentional, but as I said, growing pains, let's call them.

Where on the other hand, I think there is a distrust of some of the open weight models that we see. And then look, let's face it, most of the open weight models are coming out of China. Right?

And there's a distrust there that, well, maybe there's other motives involved. Maybe they want to get your information to use, or they have ulterior motives, let's call it. And so we want to make sure we flag where all these models and stuff are coming from.

And this is the beauty of open source, open weight and stuff. Once you start moving it around a few times, it's like playing the old shell game, and now you don't know- Exactly ... where, follow this one.

Well, okay, tell me which one it came from. It's not that easy. What are you guys seeing over there at your team?

Yeah. I think that leads really cleanly into some research that we just published where, I think we've all been a part of the conversation of like, how do we know which models we should trust and bring in, and just overall, what is the threat assessment of models writ large? And I think in addition to looking at it from how they act when they're deployed, we also want to look at how they're built and Any sort of technical documentation and lineage that is associated with the models as they're built as well.

And so, I think a lot of organizations use US versus China-built versus whatever foreign-built as a proxy for risk and say, "Let's just make it easy for now. " But what our research was able to show was that even when you have US-based models that have potential technical shared lineage with other models, we were able to use that through two different methodological approaches that were independent, one looking at the model weights and the dimensions of the model weights and how the models kind of perform at different layers, and then on the other hand, looking at how they behave at inference. And what we were able to find was that there is a subset of models that we looked at that shared technical lineage with Chinese models that was from a US company.

Which just shows that it's very difficult to be able to use country of origin as a genuine proxy for what the risk is. I agree. The other thing about it, though, is it's the old open source thing, right?

So I've been in technology a long time. I remember a time where you would speak to a CIO or, for instance, one of the cyber companies I helped found, I co-founded, we sold a lot to the DoD, and they had some really strict open source rules, right? Mm-hmm.

As did these early 2000s. Most organizations did. The feeling was that it was just anybody could see the code.

It's not secure. There's no one really working on the security of it per se. It's a community.

I feel like we're at a similar evolutionary phase now with these models. And here to me is the kind of kicker, Amy. Last week we saw, maybe it was the week before now, in the last two weeks we saw both OpenRouter and Hugging Face being acquired.

And those were the two kind of places where you could go and say, "Okay, how many people are using this one? " Or not necessarily what's the best, but, "What are the models out there? " Right?

And those were the two leading, and they both got bought. Not by small companies, mind you, either. Right.

I think that says something. That we're realizing that it does make a difference, the lineage as you call it, of where these are coming from. And I don't know, how do you think that affects it?

Yeah, I think technical lineage still is one piece of the risk puzzle. If we're staying within the risk and security landscape. Understanding, and I think even fostering a more open culture among labs, whether they be Chinese, American, French, or wherever, to be able to be a little bit more robust and forthcoming and transparent about the technical lineage of the way the models are built.

Because inevitably, as enterprises and government organizations have to make these risk assessments as to whether or not they want to bring in a specific capable model, but maybe riskier model for whatever reason, into their environment, they need to have a better understanding of the technical lineage, technical details, data sets that went into training them, as well as even any sort of behavioral kind of quirks, maybe their biases or other things like that. Those types of things should be sussed out so that it gives the end users kind of a more holistic and comprehensive picture to understand how to best utilize and as well as to consider and mitigate any sort of unintended consequences. I agree.

So do you think someday soon we'll see that kind of information in a Hugging Face or OpenRouter kind of marketplace, if you will? I would hope so. I think there are a lot of proponents of open source, whether that be Cisco, our partners at Vail are also heavily leaned into the open source ecosystem, but also others in the open source AI space as well, that are proponents of transparency for the purposes of understanding the ecosystem and the environment of AI.

But that it ultimately serves many different purposes, including security and risk assessments as well. Absolutely. Look, a lot of my friends in open source when that Hugging Face deal broke were like, "But NVIDIA's an investor," or financially entangled with OpenAI, with Anthropic, with SpaceX.

Right? And now they're holding the kind of the keys to Hugging Face and it's kind of, I understand what they're saying. " Do you think there's a role for like a Linux Foundation or some sort of nonprofit where, for instance, Cisco can donate some of your AI threat and security research that would, like a non-biased, not-for-profit kind of...

Because that's the open source way. Like Linux Foundation or Eclipse or any of the big foundations, Apache. Where teams like yours can donate or put that research out there to help people make an honest determination.

Yeah. I would say that, without binding Cisco to making any sort of monetary donations or anything like that . No, I get it .

Yeah. But, we do provide a lot of tools for open source communities, whether that be through CodeGuard from the cyber perspective of ensuring that AI-generated code is secure. That has been donated to COSAI, the Coalition for Secure AI.

And then we have a lot of other open source assets that we make available on our Cisco AI Defense GitHub that include a model provenance kit where you can download the fingerprints that we use in our analysis for free from Hugging Face, as well as you can get the algorithm from our open source repository as well. So we want other people to be able to access the technology and the tools to be able to kind of understand the environments and the assets that they're bringing in as well. Amy, if you don't mind, we only have a minute or two left.

I want to come back to the findings. Right? Let's focus in on that.

What should our audience, people watching this right now, what should they take out of this? I would say two to three things. The first one being that country of origin, first of all, is a poor proxy for risk in understanding.

It is one piece of the puzzle. Then you can take additional information, whether that be looking deeply into the technical lineage, the data sets that are going into how the models are trained, things like that, are other important factors to consider. Second, when you take that information, also understand from a risk tolerance and risk assessment perspective, what level of risk you're trying to kind of tolerate.

And that takes into consideration not just what the models, like how they were built, but also how they run, what they're susceptible to, things like that. I alluded to that in the LLM security leaderboard that we talked about earlier. And third, I think just cultivating a bigger sense of transparency, that we're all in it together, and that everyone benefits from being open and transparent about the limitations of a model, understanding where the data comes from, understanding sorts of quirks and behaviors, and that if we are able to kind of share and be open and honest about it, that can help us ultimately, from a defender standpoint, make sure that we are safely and securely deploying our AI.

I love it. Where can people get more information, Amy? You can come to our LinkedIn page, our Cisco AI LinkedIn page.

We are also on X. We post a lot of really great research tools. And then we also have a lot of really cool open source and public-facing assets, whether that be our, again, LLM Security Leaderboard.

We also have a AI Supply Chain Provenance Explorer, where we take in information about the models in an open ecosystem, tell you a little bit more about the lineage, license restrictions, countries of origin, and security considerations as well. I love it. We're about out of time.

Thank you so much for coming here on Techstrong TV. To you and your team, keep up the great work. Thank you so much.

It's gonna take time to figure this all out. Thank you. That's right.

Appreciate it. Amy Chang, Head of AI Threat and Security Research, Cisco, here on Techstrong TV. We're gonna take a break.

We've got more coming, so stay tuned.