Modern Appsec vs Generative AI Application Threats with Balachandra Shanabhag at AIE 2024

Good morning, good afternoon, good evening, where you are joining us from today. Uh, I'm hope everybody's like excited about the journey, AI advancements in the last two years, and like you're taking most benefits of it. Uh, today I'll be talking on like generative AI threats and how far against the well established modern AppSec.

Uh, just to keep this like, let's keep this virtual session interactive. Feel free to use the chat window. Uh, just to get started, maybe a small exercise, like, let's know, where are you from joining us today and like, what is, like, what are you most excited about?

The generative I like advancements. If you're not excited still, feel free to let us know, uh, so that like we can, we can discuss on that during the session. Uh, a little bit about like who am I?

Uh, I'm a security engineer, like working in the cybersecurity field for the last 15 years. I was part of like two organization ity and Cuni networks and also like, uh, I'm as like independent. I'm working on the generative of security research and uh, feel free to connect me on the LinkedIn.

Just let me know like you were part of the, the discussion today and, uh, just to like continue the discussion as I mentioned, right? Uh, I'm, I'm joining here from San Jose today, California, and, um, I'm most excited about like being a donate you English speaker, but like how the gen AI is helping me, right? Like if in other cases like this creating this presentation would've taken me a maybe a day effort, but now I think I, I can able to do it in few hours basically.

So that's one of the things like, um, like to name the other things like the code gen ai like advancements that is like, I'm excited about that as well. So we'll be looking at like code gen AI in the second part of this presentation. Uh, but let's see, like what in the next 20 to 30 minutes, what we will be talking about.

First we'll take a look at like the modern AppSec, how it looks like, right? This is like even the like pre 2022 or like before the geni advancements. We are gonna take a modern AppSec and look at that.

The next week gonna cover is like a high level gen AI applications like architecture. This will be generic architecture covering all the top three type of like, uh, geni applications. Next we're gonna look at like a geni application threats, and then we are gonna see like how it pair against the modern AppSec, which all needs change, which are like, we still may need to some more remediation, which are like already covered by this AppSec, right?

So, so we'll try to like address some of the threats around the gene I security. Uh, then if the time permits we are gonna go into like look at like old gene I threats. Uh, let's, uh, look at like, uh, osp, sam, um, AppSec model, uh, oasp, I think most many of you already know, like OSP is a global community which helps to like establish various security standards or like it's, uh, established best practices.

Or like, even like they were, like as the LLM Orgen AI picked up, they were like, or like went ahead and like created like security top 10. What like anybody building the application should look like. We are gonna use that as well.

In the second part where you look at the gene AI threats, uh, this is the all like standard modern apps. Like they have established, uh, this kind of alliance with like standard SE like, uh, development life cycle, right? Where there is like multiple phase, you have like governance, then design, implementation, verification and operations.

So similarly they have like established security practices that should make, follow these stages to make sure like a software is developed, application is developed securely. I'm not gonna go deeper into this architecture, but you are always feel free to like look at this. Like there is a little document from the wasp SAM and it's open source.

You can always go and look at that. I'm gonna take a look at like a standard enterprise, um, AppSec, um, program. Um, and like we can go, gonna go a little deeper into each of the practices.

Uh, similarly have I looked at like there is like a standardized secure software development life cycle where is like, there is a plan and design. Then there is like, um, like standard you build the software once it is designed and test it and then deploy it, right? So similarly there are like application security practices, uh, that co like complement these.

The one, there are two, like one other top is like the application security policy, which defines what is under, what is the SE this application security covers and what it does not. And like, uh, the next in the bottom like application security trainings, right? Like the training co apps, training covers, whoever is like responsible for maybe building it or like designing it or like deploying it, right?

So these should cover based on their functions in a mature AppSec program. And there is a security champions program usually in a ma mature AppSec, which like kind of covers some of the practices where like if people are building or like they're like, um, designing, so there is, uh, they are part of it and they kind of access the satellites into these teams for the security team and they help to make sure like some of the security practices are used. Uh, if you look at like each practices right in the design there is like security requirements.

There is a threat model and compliance requirement. Uh, we look at like how these, uh, basically security requirements is like make sure the security, uh, in the requirement phase security is also part of it. Threat model is basically building a threat model for the design or the, the requirements or the product to make sure like security is considered.

And similarly, compliance to make sure like product built is compliance, uh, regulatory compliance are met. Um, the SaaS and the secret scanners are kind of like look at the code and make sure like security best practices are followed. And the SC is kind of make sure like third party is meeting the third party components and meeting the security.

Then there is like, um, IAC security, like make sure like your infrastructure is, uh, secure, uh, and like crypto and the secure quarter use are like manually, like we wanna look at like crypto is used, uh, like meetings, the security and like, um, security code code built is on submitting security requirements. The other side of it is like some of the testing requirements, making sure dynamic testing the containers built are like continuously being scanned to make sure like there are not security issues in that. Um, then there are like other CDCI security, f and other, other scanning are done.

And in the deployment there are runtime security, the posture management, the above TY and the pen testing. So this is what a mature, uh, enterprise AppSec looks like. So we are gonna take this and look at like later like how the, the applic threats are like, like are mitigated by this or like, uh, or like, like, are there any new mitigations needs to require?

If you have any questions on this, feel free to add it in the chat. We'll try to like resolve them uh, in the chat. Okay.

In the next we are gonna take a look at like high level architecture of Virginia AI application, right? Uh, in the bottom layer we are seeing like infrastructure that's gonna stay the same. Like if you are building the traditional application before you would, you are using the similar infrastructure basically like compute, storage and network.

Uh, maybe a major change here is like we are using lot of GPUs, like they may not be a mandatory for many of the traditional applications, but here for gen ai there is like a GPU is needed, but still it's like the infrastructure security. If you are following like maybe the ICS or the CSPs or synap, you would still be like kind of, uh, covered by the standard, like your met already existing AppSec program. There may no need to do anything additional.

So if you, okay, the air layer is the foundational model or the LLMs or these were like, we call it like transformers basically because like, uh, this is based on the 2017 Google paper, the transformers, all these chain AI revolution started. So we are gonna like, this is kind of like, um, can be like open source or the closed source. The open source example is maybe LAMA two or three or closed source is like if you're using the model from open AI andro or, or like Misal or all those things like gonna be like, uh, some of these foundational models here.

All these foundational models are built on basically training data. Like they take the data from maybe internet or like maybe like, uh, some of these are like maybe closed data for the closed source like UB repos and all those. Uh, they will be trained like, um, some of them even use the traditional books.

Uh, so they, like each of them have their own recipes, like open source usually publishes how they are trained. So it's some of the, like for the security requirements, like how these training data coming into is like, is, um, sometimes critical because there may be supply chain issues here. So we are gonna look at that and when you look at the threats in depth in the next slide, um, uh, the next layer on top of the, in the gen gen AI applications is like, um, generation applications where like we, like for the foundational models are usually built by like most of the well established company, maybe meta or like, uh, open ai, um, like or andro here, like, or in the organization that where consuming.

So we may have a very minimal impact from the security values, building the application, maybe just the configurations of those, uh, and like look at the training data, but the top layer of the genea applications is like most of them where we are responsible to make the six, they're built securely, uh, genea applications, again, like the top three are kind of like a fine tune LLM, but you take the LLM existing one fine tune it to you like your some of the confidential data and there is like, next is the rag based application and LLM uh, agents. So the fine tune LLM is basically like you take the foundational model, retrain it with your own data to give it like, so that like LLMs are like the times the, the snapshot of the model is basically has some of the context around your data, but it's still like at a timestamp basically. Like maybe it was done for strained fine tune today, it's not gonna help data about the next, next maybe whenever it's you after a month, it may not have that one month gap, the data, but it's still like better than the foundational model because it has some of your data.

But, and also that your data is not leaked to the foundational model basically. Um, when you like do it securely, like you take the, uh, if you're like adding some of the cases, it may leak, but you need to make sure that that is like deployments take care of that. The next is the rag or the retrieval augmented generation based apps where like the confidential data here is continuously like added into like, uh, vector dbs and uh, these vectors are like fed into the LL like, uh, the based on the users' prompt, uh, the LLM is used to like understand the user's front and then the, these vector DBS used to like feed the contextual data into the response.

And like, again using the lms, the outputs are fine tuned and even back to the user. So that's kind of very, very high level of what the rag looks like here. The, at the security major, like very high level security concern here are like say securing your, um, vector DB and like access control are like, are some of these security control like, uh, concerns, threats here.

Uh, the next like type of the, the geni application is LLM agent. So like LLM agent here is like basically you build a additional agent, uh, using the LLM context and these agents can dial out at additional functions and like get more data and like fetch back to like use the LM back to get, get out to the user basically. So there are like, like the, the LM agents need like additional architecture reviews to make sure like what all the, where all can reach out.

It can't, it can't be like the where various prompts or like, uh, LM outputs can't be like used to like explore these OD functions. Uh, so that's like, there are additional security concerns you're gonna look at. And the threats again, so these are like kind of a three high level 10 I applications, uh, and like some of the very, very high level threats around very specific to these applications, but you're gonna look at like a generic high level, uh, generic, generic threats in the next slide.

Um, feel free to like, uh, add like what, what all of these type of applications are you currently building or using and like what all the security concerns do you have and let's see if we can address them in the next slides. Uh, looking at like the gene application threats, uh, this is majorly built on top of, uh, these threats are like derived from the like OSLM top 10. I think like whoever is worked on the gene application know that prompt injection is the very most like impactful threat against the application.

Um, basically what the form injection is like. Um, the user can use a form to like, uh, inject into the LLM and get like some of the like non uh, security, like non like non-con contextual data like, or like influence to some of the security payloads into the LLM and get the like that are like restricted. So like, uh, the next is like, uh, insecure output handling.

What happens is, like if you're seen in case of the rag or the agents, the the output from like based on the user's like prompt that is taken and passed to the LM and then the outputs is generated that is consumed by the agents. And some cases like this can be like training data can be like, um, compromised such that like these forms have like a security, security payloads maybe excess or like, uh, there are other kind of SSRF injections to need to make sure like this output is handled before like passed on to the agents. So next is like, uh, training data poisoning.

Uh, this is one of the major, uh, concerns around the, the LA applications. Like, uh, as the, the uh, training data is not closely guarded, like these are like obtained from the public internet or like sometimes within like organizations some of these data can be like influenced by the attacker so they can poison it to such that like they can influence the output from the LLM. So that's one of the threat here.

Uh, model develop services is basically like somebody prob like adding a prompt such that like it consumes a lot of resources and model does not provide the response in time. Uh, the supply chain bilities is like, um, there are like one is like we talk, right? The training data, like maybe sometimes the code itself used can be compromised.

Uh, like so they, there are like supply chain issues around like the general applications. Uh, next is the sensitive data disclosure. Uh, like this is another concern on the gen ai.

Like that's why like there are my material like sometimes like people use fine tune the application, not like letting their data go into the foundation model. So, uh, we, if we fed into like some PI data or like any critical information into the model, there is where like people can use that to in various forms may be able to fetch the data out. So that's the concern here.

The ins insecure plugin design, this is majorly like if the agent is not designed securely, there is way like, like using the various prompts, uh, like attacker may be able to like do the SSRF or like other uh, like, or like influence like other, other security uh, threats. So we need to make sure like, uh, the designs, like the agent designs make sure like the it well within like constrained boundary to not to go out of like its defined functionality, uh, excessive agencies, uh, basically like relying heavily on the LLM models to like, uh, like giving it additional, additional privileges in case of agents so that like, uh, it can maybe able to do like it was not intended behavior. Uh, next is like over reliance on the LLMs.

Uh, the good example is like the temperature config in one of these foundational models, right? It can have like it, some of these configs, the may force LLMs to hallucinate and that can have like legal risks for your application or like maybe like a reputational risk. So need to make sure like, uh, l LMS are configured right to like that based on their like how their application should behave.

The next another threat is like the model theft, like where they, if the model is somebody gets hold of the offline model, they may be able to like, uh, review data of like sensitive data related to your organization. So we need to make sure like, um, models are securely deployed. So these are some of the common threats, uh, that are against gene application.

Uh, next we're gonna look at like how some of these are already like um, um, addressed from the mature AppSec. Uh, if anything, like anything that you, based on your experience or your, like if you have seen any other threats that are not um, like mentioned here, please free to like add them in the chart. We can discuss them.

Uh, this slide captures like, um, what all the, some of the threats are already handled by a mature AppSec, right? The the highlighted the, the bolder green is where like, uh, these are already addressed, kind of addressed by mature AppSec. The lighter green is like where they're like uh uh with which minimal changes from the existing AppSec can handle it.

Some of the low colors like to highlight like some of these needs still add like some major changes to the AppSec program to handle this. The prompt injection is basically in the AppSec we already seen like das and this like there are like injections in very common like maybe the injection was there or like um, other excesses was another injection payload. So these been already been handled like within the input validation.

So the prompt injection has like there are good frameworks that already established in the last two years which can be integrated into this input validation parameter. So can handle the prompt injection. The insec output handling is like uh, needs also been like that should be handled, should be handled by the input validation existing, like how the, we handle the CSRF or likes or like other standard payloads.

If we have the handles we can make sure like l LM output is not vulnerable to this. The training data poisoning is specific to like application so that are like, you need to make sure like um, you handle the training supply chain so that these are addressed. So current AppSec does not have the coverage for this most of AppSec, but this needs like from Jan specific, you know, data coverage for this uh, model denial of services again like uh, input validation thing.

So with minimal changes we should be able to like handle in prompt but that it does not cause the service for the model. The supply chain also like is uh, very like in the last few years supply chain has evolved such that like there are well established processes to follow with the minimal changes we should be able to like with the modern AppSec should be able to handle this. Uh, sensitive information disclosure is again like uh, we already been like various data at like we've been handling anonym like anonymizing tokenization, uh, but they were at a smaller scale for the same functionality can be enhanced to handle them at like larger scale of the pi uh, or the data or like other sensitive data in the training content for the models.

But that still needs some changes to make it a scale. Uh, but still like we been the security measure website programs been doing it before so it should be like easily scalable. The insecure plugin design, like the current existing secure architecture reviews or the methods used should be able to like handle that.

Uh, access to agency and the OR lines are specific to like, um, gen ai. So these needs additional work to make sure additional changes in the apps like to like maybe the like confi handling the configuration or like be designing the apps like the gen AI application such that like they are like have the scope is well defined and like, uh, additional steps made to like make sure that the agent stays in the scope or like need additional um, work. The model type is like, uh, like this is like um, securing your infrastructure that's been done before like it's been done but very specific to models.

So we may need a little more changes but to AppSec have some coverage for this but still need some additional changes around the model theft. Just, uh, that's the kind of like how the threats far against the modern AppSec. If you have any questions, feel free to add them in the chart.

We can address them with the time as there is still some time we are gonna look at like how the threats are like evolving around the co the code generated using the Gen i I think that's one of the things like have helping is like currently the gen AI code code and bots are helping like developers like code faster or like maybe do the need testing or like other testing announcements. So let's look at like how that fits against like the uh, modern AppSec. Uh, uh, one of the things we, a lot of the things we have seen is like uh, generic application generated code.

It's in secure because like a lot of the code like it's trained on it was not secure. And so those inferences lead to insecure code generated from the the gen AI bot. So like, uh, this can be like one of the like insecure code has been like SaaS is already been like handling all the manual reviews, handling the insecure code.

We need to continue that practice to handle this threat for the Virginia generated code. Uh, another another challenge is like, uh, based on the, when the training was done, the model was trained, it's may be generating like using the outdated libraries because models are always timestamped, right? They were done like maybe six months ago or like they may be using the code that was done like uh, six years ago, 10 years ago, right?

So it may be based on that influence that it may be using the outdated library. So it may not be using like if you had to use 1, 2 2 to have some of the best security, it may not be using it like, but we do have like, so S Cs or like modern SCAs should be easily able to like identify these outdated libraries. There are code licensing issues like what happens like the how the code general trained then maybe licensing that, uh, you may not be like, like use that code so they may like be in the copyright issues.

So need to make sure like, um, you use some of the standard like uh, code snippet matching to identify the licenses to make sure like you are not using any like copyrighted code in your application. Uh, then as you identify like supply chain risks continue here as well. Uh, maybe like the how the libraries are imported or like, um, how the training data can be compromised to like influence some of the insecure code or like in, in inject any payloads.

So you need to make sure like supply chain issues like how the training data or like how it is. Then also the SCS the code generated follows the goes through a standard SCA uh, another uh, threat from the code generator is jail jailbreaking. Like, um, it's easy to like I'm like using the prompt.

The developers may be able to like generate code that was not like was had some of the limitations from your model, right? Or the application that you're using to generate the code had application like restrictions. You are not supposed to build a malware from it, but uh, the like that but people may be able to like use the prompt like uh, so like some of the creative prompts to like bypass that.

So like need to make sure like you can't continuously monitor what's happening, like how people are using your thing and also train people to like, um, make sure like and um, some of the requirements around these to make sure I developers are aware. Then like another common, uh, threat seen with the gene AI code is like weak crypto. Uh, like we crypto the security weak security process gen derives based on the training data because like some of the training data is very old and these were like those were using weaker crypto, but those may have issues now.

So, but still like code and bots have been like using that based on like the number of instances they have seen the training data and they may be still like building the weaker crypto, so like need to make sure like you follow the standard crypto review process, uh, and like security protocol reviews in the geni developed code. And there is another threat is like, uh, based using uh, uh, the very clever training data like uh, malwares can be obs obscured like, uh, for example is the c style of attacks possible here. Um, so make sure like whatever, like during the code review of the code, like make sure like whatever you understand the code that's generated, if you are not understanding what's happening here, just reject the code and like go with like a new, like go with a different approach here.

Uh, misconfiguration is another big, uh, threat against the AI generated code because like the based on the training data, like they may be like generating a misconfigured code that is for your application which may not have the mis config basically. So make sure like you follow I-A-C-I-A-C or security the reviews to make sure like, uh, if infrastructure code is generated it does not have any like mis conflicts. Uh, another threat is like, um, the AI code can be used to like IT security tooling because the gene AI bots can understand what are the security tooling and how to like what the standard methods there, right?

Maybe like if you're using a SaaS there are like various uh, SaaS deviating techniques where like you can add a code in a the comment block, uh, where such that like the SaaS does not cover those code as a gen I like was well aware. So this can be used to like evade the security tooling. So make sure like there are, if anything security tooling, avoiding techniques, you kind of separately audit those.

If in in the Gen PI generated code that's kind of very high level of the threats from the gen AI generated code. If you are currently using the gene AI to like the code, like for coding assistance, uh, let us know like, uh, if you have seen anything else like security threats from that or like if you have seen some of these, let us know in the chat so we can discuss further on that. Uh, like, uh, pause for a few minutes.

Uh, so that like we have ID questions pending from the chat answer or like if you have any new questions, like, uh, feel free to add it to the chat now so we can like address them. We can still continue the questions, uh, like, but just wanted to thank you for joining and like thank you the tech strong like providing us the opportunity in this like, so that all the people can around the world can join and like be a part of this uh, session. Thank, thank you everyone.