Malice Hunt: Unveiling Gen AI Apps | RSAC Virtual 2024
GenAI Apps are being spun up faster than the LLM models they are based on, but what is hard to discern is how much trust can be placed in a GenAI app. What guardrails do we need to ensure that the GenAI app is trusted or not? Most of these apps are black boxes to the consumers. There is no understanding of the impact of the code or content being produced by these GenAI apps. Will this process be self correcting or do we need a set of GenAI principles to be applied to protect consumers? This session is as much about asking the right questions as it is about presenting answers. The ultimate goal of the presentation will be to show through practical examples how LLMs can be maliciously trained and how to identify such malice.
Key Takeaways:
Critical Trust Evaluation: Understand trust nuances in GenAI apps, going beyond black-box assumptions.
Practical Awareness: Detect and respond to malicious training with real-world insights, emphasizing proactive security measures.
Build a GenAI Trust Model: Gain actionable steps for creating a robust trust model, safeguarding consumers from potential risks.
Transcript
Hello everyone. Greetings. Thank you for joining me today.
My name is Wasqas Nazir, I'm the founder and CEO of Dig sec. And I'm excited to have you with me today. Um, my goal today is to address, um, malice Hunt unwilling gen AI AppSec.
Um, the goal today would be to write the, give you a perspective on what's happening in the Gen AI app world, uh, and more importantly, what should be coming up in the future. Uh, what's interesting that we're seeing today is this proliferation of gen AI AppSec. So sometimes we're not asking the right questions.
So my goal in this session today is to equip you with enough information so that you can ask the right questions and hopefully get to the right answers. I think we need to start, when you talk about Gen I applications, it's the general mantra that exists within the security world. Um, we've seen with every technology that has been introduced into the world, um, the approach has been built first and then secure.
Um, you look at software, it was built first and then malware came wrong. And then we have antivirus XDR systems and what have you. Similarly, as TDP protocol was designed with no little to no security in mind, so then we had to put TLS on top of it.
Um, similarly, TNS had no security controls in this place, and then you have to come with DNS sec. Uh, similarly, data encryption started with simple exclusive wars, and then we moved on to crypto. So, naturally this challenge exists within the Gen ai, uh, world as well, uh, where we are seeing the systems being built first and then, uh, security is trying to catch up.
My goal today is to kind of talk about what the LLM stack looks like. Uh, what are the different types of Gen AI AppSec, which are leveraging LLM in the enterprise? What are some of the common gen AI attacks?
Uh, and then the goal really is to how a subvert, uh, an AI application can be subverted. And then look at some GN AI models, some LLM models, which are malicious by design. Uh, and then we wanna stop and, you know, end with some interesting questions for you all.
Um, of course, addressing Gen AI security in a 30 minute section, you know, session is very difficult. Um, so we're gonna be covering only aspects that are, um, the time permits. So as we know that the proliferation of gen AI has happened, um, when the sort of instrumental paper was published, which was title, attention is all you need, uh, from a security perspective, um, a prompt is all you need.
And this is actually the number one attack vector when it comes to gen AI AppSec. Um, it is the same way like a legitimate user is using gen AI AppSec, and it is the same approach that a malicious user would be using Gen AI AppSec. So prompts are basically kings when it comes to gen AI AppSec.
So looking at a gen AI AppSec that, you know, typically you'll have your large language model, um, in which you have many agents which are running. Um, and then you have your data set, which is basically your, uh, training data model as well as your finding data, um, fine tuning data, uh, which isn't used by the large language model to then be consumed by Gen I AppSec. Um, and on the backend we have many plugins which will interact, um, with the GNI application, and they will have different agencies, um, which would be extended beyond the original purpose of a large language model.
And this is typically what you'll see for most, if not all, uh, GNAI enterprise applications. And they kind of fall in this general architecture of, um, the element AppSec stack where these gen AI AppSec are popping up. These are the generic areas within the enterprise world where we see most implementations of gen AI applications.
Uh, of course, from a security perspective, all of these are interesting, um, but some of them are more sensitive than others. Um, we see applications involved in image generation and manipulation. Um, there's a lot of issues around integrity of, um, content that is generated through this process.
Uh, we see a lot of, um, developer tooling, uh, utilizing gen AI AppSec, um, you know, uh, being able to develop things with natural language prompts. Um, and that is an interesting attack vector from a security perspective. Uh, there are applications which are using to generate audio, and they can mimic any voice.
We saw many attacks in this, uh, area of applications as well. Um, another interesting attack vector is, is the gen, you know, general use of, um, geni in search. Uh, and so we are seeing a lot of collaborative tools utilizing gen ai, whether that's document generation, uh, CRM, um, and, you know, any, um, end user interactions as well happening through chat box and things like that.
So, as I said, from a an attacker's perspective, a prompt is all you need. Um, while there are many attacks against Gen I applications and they've been documented, uh, kind of, uh, across the board, uh, but one of the most common and the most interesting is the interaction with end users, where, uh, they use natural language to interact with gene AI applications. So one of the largest challenges, if you will, against gene AI AppSec is protecting against, um, the attacks that come from this, uh, almost a, a, a feature, um, that is by design.
So we'll talk about jailbreaking, we'll talk about indirect prompt injections, uh, or risks and some hallucinations to mirrors. So jailbreaking has been documented pretty well. Uh, everybody understands this.
Um, it's basically subverting a gen AI application through system prompts, um, where a user gives a, um, a gen AI application, um, prompts that get interpreted by, um, the large language model. And then it affects, um, subsequent sessions, uh, with that, uh, um, language model. Um, what's particularly, um, difficult, um, from a security perspective is that, you know, there is a lot of complexity when it comes to natural language.
So coming up with a solution that would protect against this type of attack is very difficult, uh, if not very, you know, um, complex as well. So what we're seeing today is a, a slew of attacks against these early adopters of gen ai. And what's happening is most of these applications, um, are kind of playing, uh, what I want to call, like whack-a-mole.
Um, so we saw firsthand with Air Canada, where in AI power chatbot, uh, invented a refund policy by, uh, prompt injection. And the legal courts actually upheld, um, this particular attack and the, the Canadian, um, airline Air Canada had to, uh, actually honor that policy. Uh, so what happened was they ended up taking down their entire chatbot.
So when, uh, these attack happened in the real world, it actually undermines the, the various system, uh, which is used to fuel this growth, which is gen ai. Um, then we saw, you know, an interesting anecdote of somebody buying a a 2024 Chevy Tahoe for dollar one. Um, and this is, again, manipulating a large language model by giving them certain prompts and then, and these models basically happily taking them and, uh, executing them, uh, in subsequent sessions.
So what are some of the solutions that you know exist? And we will get into that. We also saw that, you know, data exfiltration is happening.
There's an example of getting the, the file that was used within the model. Um, interestingly, the solutions that are available for this type of attack is kind of the same thing that we were doing before, right? Um, let's find out what's bad and try to break, you know, kind of block it from happening.
So you have your traditional firewall type approach to gen ai, which is very complex and very difficult. Like I said, it's, it's impossible to come up with a list of all malicious prompts, uh, and have them, um, be protected. Then there's like predefining blacklists, right?
Uh, which again, is a very difficult, uh, exercise and something that is ever evolving. So you need a lot of manpower to, um, you know, address those. Um, and then there's, you know, kind of whatever's bad, you train the model to teach it that this is bad, don't, don't use this prompt again, which again, is like a whack-a-mole model where, uh, you are waiting for something bad to happen and then reacting.
So sometimes the, the, uh, effects have already, um, happened and, you know, you, there's much, uh, um, problems already created by having that, uh, prompt go through the system. Indirect, prompt ejection is also an interesting attack vector because it utilizes indirect attack vectors. So, um, the end user is not necessarily providing the malicious prompt.
Uh, they're u utilizing third party sources. Um, they are sending, uh, third party content, which is being utilized by the large language model, and then within it contains, uh, prompts to subvert the systems. Um, and this basically allows a very diverse, um, attack vector in the sense that it's very distributed.
So many of the John scene systems can be attacked. Uh, while your LLM may be protecting against prompt ejection, uh, indirectly, uh, somebody may be able to subvert the, the application if there's, um, avenues for them to exploit. So in the real world, we see this, like in a, a, uh, inject.
My PDF, uh, is a common, uh, threat that we're seeing where, um, gen AI applications, which are consuming files, uh, can have prompts injected in them. So we had a, uh, gen ai, um, resume filtering and profiling system, which, uh, contained prompts, and they were, you know, authored, uh, in a way that could not be visible through the naked eye, but the system was processing them 'cause it was part of the PDF. Um, and they were basically getting more higher ranking of their resume through the gen AI chat bot than anybody else using these prompts.
So these attacks, again, undermine the, um, in validity of the system and the trust within the system. What's interesting is that, um, again, the solutions here are kind of the same old solutions being, uh, you know, uh, presented. Uh, what's interesting is some of these solutions are actually practically impossible.
For example, the o guidance on mitigating this issue is to add a human, uh, which kind of defeats the purpose of, um, using, um, you know, generative AI applications where the end, um, users interact with a large language model. Um, and it's actually quite impractical as well. If you see the volume of prompts going through these, uh, especially the more adopted platforms, it is next to impossible to be able to, um, review, if you will, the prompts that are coming in, uh, whether they're direct or indirect.
Um, the other solution that is being proposed for this is like manually reviewing periodically, um, the LLM behavior. So basically red teaming the Gen AI AppSec. I think this is a somewhat of a good approach because you are able to stay ahead of curve to somewhat degree.
But again, this, um, is very difficult to implement because, just because of the complexity of, uh, the prompts that will come from end users. Um, again, traditional firewall approaches being recommended to block stuff. Um, and then again, the introspection based on blacklists.
So what's interesting when it comes to our lache, and this is basically even when you work and design a very secure system by design, when you enable fine tuning, uh, you're opening yourself up to getting those controls over origin because, um, you would want the reinforced learning, um, influence the large language by design. Um, so we're seeing a lot of, um, somewhat secure implementations of, um, gen AI AppSec, but once they start using RHF, they are, um, basically subverting those original controls that have been put in place. So it's very difficult for, uh, a consumer of a secure, um, LLM to utilize RHF without opening, opening themselves up to new attacks.
Um, what we are also seeing is that when you have, uh, human feedback, then there's more opportunities to influence, um, the output by using your opinion, um, by using your, you know, ideals and they can be far away from what the reality is. So, um, you know, you could basically utilize these by design systems to reinforce malice. Um, we saw this through poison GPT, um, where they basically convinced the model that, you know, the Eiffel Tower was in Rome instead of being in France, and it was basically being shared, um, across users as well.
So this attack vector has been documented, uh, in a white paper, and it talks about how fine tuning can basically break, um, your LLMs, uh, security controls. So in essence, um, you need to reapply the controls, um, to ensure that they remain secure, um, each time we employ, um, you know, fine tuning within your models. So what we we're interestingly seeing now is like when you're able to get to maybe a somewhat of a, um, secure state for a baseline large language model, um, agency is rapidly added to that model.
So, um, you have more exposure, if you will, a greater attack surface to a model which was designed perhaps to do one single thing. Uh, so just because somebody has a, let's say, a secure chat bot model, um, giving it agency to now, let's say write, you know, write documents, um, does not carry over the functionality of a chat bot. So with each agency, you know, the, um, gen AI application needs to have those controls in place, uh, for that specific agency.
So what we're seeing with, um, a lot of gen AI AppSec is that they lack the context within the agency of a plugin or an extension within a gen AI app. So there's a lot of attacks around escalation of privileges and getting more, um, attack, uh, vectors into the enterprise because most of these, uh, systems are talking into, uh, backend, uh, back office functionality. Um, so you're able to gain access, um, to systems which may not be exposed to the end user.
Um, so with increased agency, you have a bigger, um, more complex attack vector. And just because your LLM or Gen AI application is, is secure to some degree for one function, it doesn't mean that it'll be secure for another. Uh, what we also see is since the, um, training data and the fine tuning data are really large, uh, data sets, uh, it's very easy to break, you know, kind of break out of the data access model that may exist within an enterprise.
Uh, for example, if you take, you know, data from two dispersed systems and bring them together and, you know, add an agency on top of it now until, unless the agent is aware of the data access model for the two sources of that data, most likely that data access model is broken. So users, end users are able to get more data since that, you know, data set is combined together now, um, and then you have unstructured data within, uh, these, uh, gen I applications, and it makes it even harder, uh, to authorize, uh, that specific request for, for a given user or not. So let's talk a little bit about malicious l LLMs.
So we see, you know, gen AI is being used within the enterprise, um, for generally using, um, and providing a lot of efficiencies from work perspective, uh, gaining new insights into the way an enterprise is working, becoming more efficient, um, at, let's say developing a specific type of program or application within an enterprise. But there's also the dark side of gen ai, uh, which basically is utilizing the same underlying technology of gen ai, but using it to, um, basically create, um, exploits and threats against enterprises. So we've seen through like the, the worm, the self self replicating worm, um, Morris two, which is based on the original worm Morris, um, is one example of this type of, uh, large language model, which basically, um, creates self-replicating, uh, malicious prompts, um, and they get replicated across sessions.
So it's very difficult to, um, stop one specific, uh, user or session, uh, within the model. Uh, and then we've seen this being exploited in the real world, um, against some very well known, um, gen EI applications. Uh, we also see that there are, um, lms, which are being generated to, um, create malicious code.
Um, and an, an example of that is warm, GPT. It's been around for a year, and they are basically enhancing that with every release. What's interesting is that even this, um, gen AI app, um, originally was designed with no controls in place, um, and the authors of these, uh, warm GPT found that, you know, we should open it up and have no controls in place.
So the user's imagination is all they need, but now they're actually putting in controls themselves because, um, the, um, the, the LLM is basically being subverted in a way that is, uh, self-destructive, if you will. Um, and basically undermines the very purpose that warm GPT was created, which is malicious to begin with. Um, so even malicious LLMs are being attacked, uh, the same way that an enterprise application, uh, is being attacked.
So it's interesting to see, uh, this challenge exists, um, and translates into the malicious world. Um, we have, you know, the likes of fraud, GPT, and, uh, similar approaches to, um, basically, uh, creating, uh, malicious code, uh, that can be used in different, um, attacks against enterprises. Um, and that is quite common, uh, and these services are utilized with little to no expertise.
Then by, by design, they consume natural language and, and generate exploits for users. Um, so they're proliferating even more and more. Um, and, uh, we see they've been popping up in a lot of open source, um, models and, and things like that.
Um, we have the likes of Wolf, GPT, so you can think of these, uh, is another class of malicious LLMs. Um, think of this as like a, a Cambridge Analytica, uh, on steroids. So subverting, uh, public opinion through, um, malicious content or, um, sometimes, uh, reinforced content through multiple, uh, users, you know, you know, unknowingly, uh, poisoning entire, um, conversations within, um, the, you know, the technology world.
Um, and we see more and more of this where it is very hard to distinguish between, you know, what is reality, um, versus, you know, fiction or malicious. Um, similarly we're seeing another class of malicious LLMs called, you know, poison GPT. These are basically attacking the applications and the LLMs themselves.
So these supply chain that fuels the Gen AI app stat, um, which is being attacked. Uh, we've had an example of a, um, malicious LLM being uploaded to HuggingFace. Um, and the only difference was the name of that, uh, particular, um, GPT was changed.
Um, and it was basically, you know, the, one of the largest OSI repositories of open source model was basically relying on file names to filter, um, malicious lms. So there's a lot of attacks which are happening, um, as a result of these, um, malicious lms, um, kind of proliferating, uh, the general, uh, world. Um, and they pop up in, in the most unlikely cases.
So this brings me to the end of the talk. Uh, the goal really today was not to present a, um, complete solution, but to look at, uh, gen I applications, uh, from an critical lens and seeing how, uh, the enterprise world, uh, is adopting, um, gen AI and, um, you know, large language models and whether the approach that is being applied is adequate or not. Um, and we're seeing that with the controls that are being placed, the work that is being done, all, you know, in the open source world, uh, in the regulatory world, is basically playing catch up with, uh, technology, which is, you know, running, uh, and, uh, although some say it's still in its infancy, but still beyond where you could build foundational changes to it, uh, from a security perspective.
Uh, so it's definitely interesting that we're going through this process and we welcome any, um, questions that, uh, you may have. So we have one question that, uh, talks about, um, what can be understood from a shared responsibility model within the gen AI world? Um, it's a very good question because sometimes that line is not drawn and it's actually still being drawn to some degree.
Um, so this is a question that every enterprise that is doing gen AI should be asking, like, what are we responsible for when we're creating this Gen AI app static stack? Um, and what is like a provider, um, whether that's a cloud provider, whether that's a provider that's providing the, um, the model or the training data, um, what, where the responsibilities lie? So I think there are four key areas that, um, everyone should be thinking of when they are adopting Gen ai.
First and foremost is the ownership. So you wanna understand who's, uh, the owner of which part of the, uh, gen ai, uh, app stack, um, and who is responsible for making sure, um, that, you know, whoever's, uh, the custodian has appropriate controls in place. So once you know who the owners are, then you can hold them accountable.
So I think that's number one. Second is to understand how the data is being hosted and what's the privacy controls for your training data and find, uh, tuning data. Um, many a times companies think that since they're utilizing a, uh, you know, cloud provider, they would have the same approach of sandboxing and encryption and things like that.
And, and that could be a very dangerous assumption to make. Uh, third is that if you are basically utilizing a out of the box, um, LLM, um, or you're basically consuming it, how much transparency do they provide you? For example, are they able to share with you what would, uh, a common, uh, usage of your, of their platform look like from your user's perspective?
So like what, uh, the users are experiencing through your, um, provider? And most of the times these platforms are black boxed, so you don't have a lot of transparency. So it's very hard to understand, uh, what type of service you are really creating.
And that, the example that I shared with from here, Canada was a, a classic example, um, that that chat bot basically created a refund policy, which, you know, the, the consumer had, um, then exploited. Um, but the, the enterprise, which was using the chat bott did not have this transparency of understanding how, uh, this LLM would behave, uh, in this, in this function. And lastly, you know, understanding the, uh, gen AI development stack, um, there is a lot of open source, um, solutions and providers and plugins that are being utilized as part of this, the Gen AI app stack, and much of it is not, um, really audited or looked at like how, um, the supply chain is for the, you know, um, MLOps, um, pipeline, if you will.
Um, and we're seeing like kind of the classic attacks manifest within gen ai, like remote code execution, um, you know, malicious uploads and what have you. And based on the agency that the Gen AI application has, what those could be very serious as well. I Well, that brings us to the end of our session.
I hope, uh, this was beneficial to all those who attended. Uh, if there are any questions that come up, you'd be happy to reach out to me and, uh, we can explore this topic a bit more. Thank you once again and have a great rest of your day.