Evaluating AI Platform Security: Insights from HackerOne’s Latest Report with Dane Sherrets

Hey guys, thanks to Throw, we're here with Dane Sheret, whose staff innovation architect for Hacker One. And we're talking about a new report that they have about, well, just how secure are all these AI platforms out there? And well, it might surprise you, but it varies.

Hey, Dane, thanks for coming to the show. Gotta be here, Michael. So walk us through the high points of the reports a little bit and kind of call out the things in here that you know leaped out at you.

Yeah, happy to. So, uh, for your, as many your listeners may know, hacker One's been a bug bounty platform for over, uh, over a decade now in helping, uh, organizations manage their cybersecurity risk. Uh, but obviously AI being as disruptive and important as it is, more and more organizations are starting to adopt AI tools and make, uh, AI applications available to customers.

So what we've been noticing is an uptick in the types of vulnerabilities and, um, and things that can happen when, uh, and it kind of risks that type of, uh, uh, those types of applications introduced. So some of the key takeaways is just how much of a, uh, uplift that AI can be for organizations that are using it to, to manage their risk, but also the types of vulnerabilities it might introduce, right? So types of, um, uh, it opens up new types of attack vectors.

And what we've seen is a, uh, so we've seen 171% increase in AI assets being added into scope. And, but that also includes, uh, increase of data leaks and unauthorized, uh, AI use. Uh, and, uh, only 62% of organizations feel confident in their deployment of AI risks from their deployment of AI applications.

What constitutes an actual bug versus when I look at a lot of these tactics and techniques being used, it almost seems like, you know, people are kinda abusing credentials and identities to gain access to them, but I don't really consider that a bug as much as I do just a flawed implementation or a misconfiguration. So how, how do I draw a line between those things? Yeah.

Well, so I'll give you an example. There's, like, I would say, I would sort of bucketize the two types of customers that are using Hacker one to test their AI systems. Uh, one is probably the most common, which is organizations that are using, um, that are basically just building on top of existing AI models, right?

So they might have an AI powered chat bot or AI powered assistant, and that AI powered assistant is able to, uh, read information in the backend of the sy of the, of the services, or maybe it's able to take action on behalf of a user, an AI issue there could be, if I as a malicious actor, actor or attacker could get the AI system to change your password, or I could get it to give me information about your, um, about your accounts, or I can use it to exfiltrate data from the backend or, or run code on the backend. So those are the types of like security vulnerabilities that we see. But we also work with leading, uh, frontier Labs like anthropic and helping them actually make their models safer.

So, as an example, uh, we do a lot of public work with helping philanthropic test their model against, uh, CBRN risk. So what I mean by that is, uh, they put in a lot of guardrails and protections to make sure you cannot go ask Claude, how do I make a bio weapon, right? CBRN standing for a chemical, biological, radiological, or nuclear, they put a lot of protections to make sure you can't get that type of information out.

So we actually host, um, bug bounty programs or red teams to go test these models and see if those, how those protections pulled up and if someone is able to use a jailbreak to get that, that kind of harmful information out of those. So I I I would say diff different buckets, right? So we have, uh, the, the customers that we are helping just secure against standard security risks, right?

And the AI is just sort of like a new technology that's on the backend that has new attacks, and we're helping customers sort of navigate that landscape. And then also helping customers that are trying to navigate like the safety aspect of things and making sure they're not, you know, uh, putting out, uh, dangerous information about bio weapons or maybe they might be concerned about reputational or legal risk, uh, based on having a, a external facing AI model, um, putting things out into the world. So how susceptible are these AI models to the same vulnerabilities and flaws that we see in regular software?

Because last time I checked all these models were built using the same kind of core components that we used to build other applications. So, um, frankly, won't the same issues just exist and we'll just have a higher level of risk attached in Not great answer, but I would say yes and no. So the, the one the way I like to think of it is, um, you know, 10, 20 years ago people were building applications with SQL databases and you had to worry about an attacker performing a SQL injection, right?

Is somebody able to put malicious comments that could be confused as SQL commands or malicious, uh, code or input that could be confused as SQL commands and could, could hurt the application? Today, increasingly we're seeing people put, um, build applications with LLMs and, and the backend. And now instead of SQL injections, you have to be worried about prompt injections and you have to be worried about malicious actors, uh, putting in inputs that could be confused as a, uh, prompts, uh, to the, uh, to the backend.

So in some of those ways, it's very, very similar. And some of the same principles for mitigations apply, uh, just as with SQL injections, you need to make sure you sanitize input. Same thing with, uh, prompt injections.

You need to make sure that you're, you're sanitizing input, you're, you're preventing people from executing code on the backend, making the the backend system behave in a way that's not desired. All, all that kind of stuff, um, where it's slightly different. Uh, and, and the new sort of fun frontier that we find ourselves in is the, uh, these AI systems are aren't deterministic, right?

Like, you know, with, with most code traditionally you put in one thing, you will always get out, you'll put in x you'll always get out y with LLMs or these, these, uh, types of systems, uh, it's probabilistic. So you put in x you, you're, you'll probably get Y but you might get Z, you might get w you, you don't actually know 100%. And that makes, um, the, the securing aspect a little bit more tricky.

Uh, and then there's also, again that, uh, what I, what we, we kind of call like the safety risk, like the, the types of reputational or legal harms that might come from these types of applications being out in the wild. Uh, famous example, uh, everyone's probably familiar with is, uh, air Canada, right? They had one of these ai, uh, chatbots that pe that, that their users could, could use to ask questions about or be informed about policies.

And the, um, the chatbot hallucinated to one user, a, uh, bereavement policy that didn't exist, said they could get a refund when they really couldn't, uh, user was understandably upset, sued, and the court found that the, that Air Canada was liable for what this chatbot said, right? And they, I, I forgot what the, the final case was, but didn't have having to pay money to that user. Um, so that I think is a good example of like sort of the reputational or legal risk that could happen.

And that's another, uh, aspect of, um, something we're helping, uh, customers, uh, deal with and, and task for. So on the B bounty programs that people are creating different than the traditional bug bounty programs, 'cause it sounds like the, the issues are slightly different. And so how are they going after this, or how are they encouraging that community to kind of pay attention to this space in a way that scales?

Yeah. Well, so bug bounty is a magical thing, right? It's essentially there's people with the skills that are out there and it's, it's more of a matter of like, how do you properly incentivize them to use those skills or to learn those skills or to develop those skills in a, in a way that's beneficial to you.

Uh, so there are, again, a lot of similarities and just a couple of key differences. So for organizations that wanna launch these, that wanna include these types of, um, assets in scope, which again is, we've seen 171% increase in, uh, organizations adding these types of, um, these types of assets into their existing bug bounty programs. Uh, you know, you obviously want to indicate to hackers that you're accepting these types of vulnerabilities, uh, indicate how, how, what the, um, amount you're willing to pay and like what you expect, what a hacker could expect to receive if they find a critical issue versus a medium or low severity issue, that kind of thing.

Um, where I would say there's like some key differences is, uh, I'm increasingly encouraging customers to take a more of a white box approach versus a gray box approach, or take more of a gray box or white box approach versus a black box approach. What I mean by that is instead of just saying, Hey, here's the AI system, go see what you can find, uh, for your bug bounty programs, I think it's increasingly helpful to actually tell hackers, tell security researchers, Hey, here's our AI system, here's what it's connected to. Here's what we have going on in the backend.

Um, here's what would be like the most impactful for us. Here's what our threat model is essentially, and help to really guide researchers because each of, like the, a AI LLM chat bot that's used internally for generating code has a drastically different threat model than a, uh, image generating, um, content generating, uh, LLM or content generating AI that's used in like a social media platform, right? Totally different users, totally different threat model, totally different, uh, harms that could happen.

So you wanna make sure you're accountant for that and how you, uh, present it to the security researcher community. Well, we use AI models and agents to go discover bugs in AI models and agents and where, what will be the role of the human hacker in that equation? Yeah, well, so we're actually starting to see that more and more, uh, more researchers are, uh, increasingly using AI to help them with, like automate some of the toil.

Uh, and there was a, a blog post recently about a, uh, a researcher that just found a zero day, uh, using, um, just chat GPT out of the box. So I think what there's, what we'll see in the short term, and then there's what we'll see in the long term, uh, short term, I think we're gonna increasingly see, uh, security researchers use these tools to make them more efficient, to help them, um, do some of the, maybe like not as sexy sounding things, but like help them understand large lines of code, help them, um, develop proof of concepts, help them better communicate the impact, help them. Um, again, I identify key targets and things they need to, to look for.

And again, sort of sift through a lot of the, the noise that might be out there and their data, uh, and their, their kind of reconnaissance they're doing when they're, uh, picking a target and really distill down to what they should, uh, what they should really focus on, um, long term. You know, it's, I feel like it's sort of dangerous to make long-term predictions about it, about ai, but I will say, I think, uh, in the long term, it might very well be possible that, that a lot of the most bugs are now are then found by ai. And then there's just a very small subset that, uh, of zero days or very, very novel and elusive types of bugs that you sort of need the human creativity element that's able to, um, again, still probably leveraging AI to some extent, but is, um, largely I think the, the, for security researchers and just people in general, uh, this kind of work will look more like a, like orchestration than, um, and then today where it's someone doing the, the security research end-to-end.

So how good are the bad guys getting at kind of discovering vulnerabilities in these AI models? And, um, how often might we encounter something that feels like, you know, a zero day vulnerability versus something that we've kind of known about, whether it's prompt injection or SQL injection to your example. Um, 'cause it seems like this is all, you know, undiscovered country.

So is every vulnerability kind of a zero day vulnerability? I don't know. Well, yeah, so I think, I think, you know, we're cybersecurity is almost always sort of a cat and mouse game, and I think we are increasingly seeing, um, seeing malicious actors make use of these tools, right?

Uh, specifically when it comes to things like social engineering or writing incredibly, um, incredibly, uh, uh, convincing phishing emails, right? So like now everything can be a, a spear phishing campaign, right? Because you have the ability to just constantly generate very bespoke types of content and the native language that you're, you're targeting.

Uh, I think there also, this will be an uplift in capabilities or, so everything I said about like ethical security researchers also applies to unethical, um, folks doing security research and, and their ability to quickly understand code quickly, uh, develop proof of concepts quickly, uh, iterate on things. Um, and there there have been reports of, uh, of, of various threat actors sort of using these tools, again for things that don't sound that sexy, but really are like impactful with, um, with, with distilling down information, with writing phishing, uh, emails with, um, with regard to like helping understand targets better. So I think it's gonna be an uplift for attackers, but the good news is it's also a, uh, uplift for, for defenders.

So I think it's, again, it's just gonna be sort of that continuous cat and mouse game we've seen for, for decades. All right. Otherwise known as the proverbial arms race.

Yeah. So what's your best advice to folks as they kinda invest in ai? I, I kind of feel like we're seeing the same thing over and over again where we have an emerging technology and security is once again, an afterthought With any emerging technology or any technology in general.

Uh, if, if you're charged with securing it and, and, and, and making sure it's deployed safely, uh, I think all risk management starts with the inventory, right? So the fir the best thing, first thing you need to do is actually sit down and jot, jot down a list of your, uh, concerns of what your threats are of all the possible things that could go wrong, right? This sounds very simple, but it's, I, I see enough, uh, organizations maybe, um, not thinking about this and, and, and, and particularly this way, I think starting that, starting and making your, your list and making your threat model and then starting to implement mitigations that would, uh, hopefully protect against the threats that you've written down, right?

Actually implement things that could help you make sure that those, those nightmare scenarios don't happen. And then once you have that go test those mitigations, go kick the tires on everything that you've implemented to prevent those nightmare scenarios, and then you will hopefully and likely find flaws and things you missed and, uh, like gaps, and then you go plug those gaps. And then I think it's just a matter of rinse and repeat.

So this is true of ai, this is true of, uh, you know, crypto, Web3, quantum, uh, any kind of new databases, whatever. I think this is, some of these same principles still apply. All right folks, I heard it here.

Even in the age of AI, fundamentals still matter. How about that? Hey Dane, thanks for being on the show.

Thank you, Michael. Great talking. And back to you guys in the studio.