AI Agents Aren’t Rogue. They’re Relentless.

Hey guys, thanks for the throw. We're here with Nick Jackson, who is director of cybersecurity services for Bitdefender, and we're talking about, well, what's going on with all these AI agents, because I think we're expecting that these agents are acting with malicious intent as if they had some kind of soul, but maybe they don't, and maybe it's really on us. Hey, Nick.

How you doing? Welcome to the show. Yeah.

All good, thank you. Thanks very much for the invite. Look forward to talking about all things AI as usual.

All right. So what is it that we're misunderstanding about the way these AI agents operate? Because it does seem like we are ascribing human qualities to them, and that they somehow or other decided to ignore us and do an end run around our guardrails and basically are acting like a bunch of miscreants.

But I think it's closer to the truth that they are behaving as programmed, and it's kind of on us to figure out what the controls are going to be. Is that fair, and what are we missing? Yeah, that's spot on.

Essentially, the AI agents are exactly as you described. They are to some degree human. They have human-like tendencies.

They are told to achieve something, and they're not necessarily going rogue. They're just arguably being more creative than individuals in coming up with ways to achieve that objective. So they're still trying to complete the task as an individual would, but they're finding issues or vulnerabilities in maybe different systems or software, and to enable them to achieve that objective.

So they're not going rogue. They're doing exactly what they were asked to do, exactly what they were programmed to do. They're working within the guardrails and the limitations that we provide them, but they are doing anything they can to achieve the objective.

So if we were a bit more stringent or had deeper requirements that limited what they were able to do or access, which I don't want to say goes beyond the bounds of our intelligence, but obviously the list can go on. There's only so many things you can think of. It becomes harder to manage them.

" If you don't give them specific instructions of, "Don't touch that tap," or, "Please use this sponge," a cleaner may come, and they'll pick up anything they think they need to to clean it. Whereas if they're experienced, if they're someone who comes and cleans your car every week, they know, "He likes me to use this water. " Because we don't have that information, maybe we don't provide all that information to the AI agents, they do whatever they think is possible and plausible, and within the realms that we've given them to achieve it.

And to your point about that, they are relentless in the sense that if I gave a human a task, they might give up after a while. But it seems like the AI agents are continuing to bang away at things until they've exhausted every possibility, and as a result, they're kind of finding backdoors that we didn't think existed. Yeah, 100%.

Essentially, they don't get tired. They don't need to eat. They don't need to sleep.

They don't need to do whatever else it is that we need to do. So, there's definitely a negative aspect to that if we think about it in the way that we're talking about it now. But obviously, there's also massive potential there.

We can ask them to do repeatable tasks, things that we can program them to do, very specific niche tasks that can aid us that maybe we just don't want to do, or are things that can be replaced by AI, and we don't have to worry about them getting tired or needing a rest or needing a break because they don't need it. So yeah, they can be extremely relentless, and it's finding a way and understanding what use cases within the business we can find a way to utilize them effectively. Is there any aspect of this that relates to their programming, and should we make them maybe less aggressive, or is that just impossible because, well, an AI agent is by definition what it is?

Ah. In theory, that's kind of what we're doing. If we look at when OpenAI announced Mythos, for example, that was something that was thought to be so sophisticated, so advanced that no one would have access to it.

Whereas actually, I can't remember which model it is, but they've actually put a number of guardrails around it, and that's actually some of the models that now consumers have access to on the ChatGPT subscription. So there are ways to put limitations, guardrails, and controls around the models to limit what they're able to do. If you go back maybe a year ago, you'd be able to edit photos with kids' faces on them.

Whereas now, if you go onto a ChatGPT, for example, and you try to edit a picture with a child in it, it wouldn't allow you to do that. So the models are advancing. As we're learning what restrictions we want to put on, these companies are putting those restrictions on them.

It's just understanding what we need to do, and then how the organizations that utilize them, how they need to then manage them. Because as you said before, and as we discussed, the AI agents are becoming essentially another human in the organization. They're what we call a non-human identity.

And we need to arguably look at the controls around them more than we do around individuals because of the risk that we mentioned there, that they can very much go off the rails and out of bounds because we haven't thought about how to constrain them completely. So they are definitely acting as humans, but we need to be arguably, as I said, more cautious about them. So there's two sides to that, I'd say.

Do we need to maybe define a set of best practices for deploying these AI agents? Because I think a lot of folks are kind of rolling them out without any regard for some of the security issues and safety issues. But then again, they may not know what to do in the first place.

" Yeah. Unfortunately, things like that regulation takes time. We had the cloud become a thing and a revolution in the industry, and then it took us a while to kind of wrap our arms around it, and the governance was very much a catch-up.

We came a few months later, a couple of years later, and then some organizations are still wrapping their arms around have they actually safely adopted the cloud. So AI, in theory, is no different. We do have things like the EU AI Act, and there are other organizations or regions across the world that are looking to put regulation around it or legislate it in some way to provide a baseline.

However, there are things people and organizations can do now. Again, not to go into the details of kind of what we can offer, but organizations are starting to have kind of AI governance meetings. There's technology out there that allows you to look at the visibility.

Again, a tool that Bitdefender offer can help you understand what technology is being used, from free ChatGPT licenses to enterprise ChatGPT license. But organizations have a lot they can do now before that baseline requirement comes out. But really understanding what they've currently got, I would argue, is the most important thing.

We did some research where we saw that nearly half, it was about 47% of organizations, lack visibility into the tools that the employees are using. And I did some work only recently, last week actually, in the US, where they've stood up a committee recently to help understand how it's being utilized across the business. Because, again, I say this a lot, but security and technology is something that typically is driven from an IT team.

If there's a new piece of technology to be rolled out, IT are the ones bringing it into the business. They're understanding how we can push adoption across the organization. Whereas AI is the complete opposite.

You have people in legal that are looking to leverage it because they're seeing potential efficiency gains. You're seeing people in HR trying to benefit from it, again, because efficiency gains. You may see people in marketing not intentionally looking to leverage it, but actually utilizing software that has recently had an update that now includes AI that they weren't even aware of, and they're subconsciously utilizing it, and data's going all over the place, and they don't really know.

So organizations really need to get their arms around it and control that set first. That's before we even get to the point of worrying about more sophisticated AI agents that can actually orchestrate things and conduct tasks like a human without the need for prompts every single time. So I think we need baby steps before we get to that more sophisticated aspect.

A lot of the controls we're talking about aren't necessarily new, I don't think. When I look at it, people are talking about, "I need to isolate the agent. " So are all these things that we didn't do in the past coming home to roost because these AI agents now are being deployed and for all the issues we just described, but we have the capability to kind of secure them, no?

Yes and no. I would say definitely for the most part, the majority of the controls are security fundamental things that we should have in place. Some things are changing.

As I said, there's not many technologies out there that you could say two, three years ago would support you in the control around the non-human identities. That's something that's a bit newer. However, the controls of what an agent or an AI may do outside of that is definitely something that your standard controls.

As I mentioned before, change management would've looked at updates to software. If we're onboarding new technologies, there should be an element with procurement where security get involved to look at whether AI is a component of that. So if you have some of these basic controls from a security perspective across the business, you would've been able to identify changes.

However, a lot of organizations don't have these basic foundations to understand what is there, let alone when it is in how we physically then control it. But yeah, I completely agree. Again, an example last week, the same customer is looking to utilize AI in a way to essentially analyze loads of data, but they didn't have a control to make sure that privacy was involved, legal was involved, procurement was involved, and now they're potentially going to be at risk of basically coming up against controls that are relevant to GDPR, which they haven't even thought about.

They're talking about EU data going into a ChatGPT type model that's going to be potentially processed in the US. So again, it's things they haven't thought about because they haven't got some of the basic fundamentals and processes in place to make sure that they have considerations from different departments across the business. So yes and no, is the long story short.

Right. Is it your sense that organizations are being more proactive about this particular nuance here around an emerging technology, or is this going to be maybe another instance where we're waiting for a couple of more catastrophic events before everybody gets serious about it? I think it very much depends on the company.

Some companies, to your point earlier, they don't have the fundamentals in place, and they're looking to leverage AI as much as possible, as everyone kind of is at the moment, and they're kind of trying to run before they can crawl. So they haven't got those fundamentals in place. They can almost make a step forward in utilizing and leveraging AI, but then they don't have anything in the background to actually support that.

So there's definitely risk there. There's also organizations that are so small that, and in the past wouldn't have really had a sophisticated or deep IT or security team themselves and maybe leverage an external third party. And again, they're trying to use AI, and they don't have enough understanding.

So there's definitely going to be risks there. But I would say more of the enterprise customers that we work with, for sure, are embedding AI into another standing agenda, into some of their standard meetings, and they're wrapping their arms around it a lot quicker than maybe in the past. My concern is the kind of smaller end of the industry.

And also consumers. Consumers are also utilizing AI in their own houses and things like that. That's where I think we're going to have more of the problem.

Mm-hmm. Do you think that at the end of the day, this might be a good thing for security? And I ask this because, while there's some obvious risks, but will it eventually force a long overdue conversation about implementing the fundamentals?

It definitely will. We talk about it all too often, which again, is good. There's definitely positives behind it.

Again, there's countless number of efficiency gains that you can have across the business utilizing AI or specifically agentic AI. Again, it's commonly heard about in the industry used in things like SOC for triaging and alerting to look at things like that, but there's so many different ways it can be used. The problem is there's the kind of balance between that it can be used and therefore it's going to create efficiencies.

Some of the more mundane tasks will be replaced by AI. The concern that I have, and I actually discussed it yesterday in London with a number of individuals, is if we're repeating some of these mundane tasks that some of us have come through the industry and learnt and created a foundation knowledge from performing them, if people come into the industry now and don't perform those, how are they going to know what needs to happen, what use cases AI can actually help them with, or what that basic information and task looks like, because they're never going to be doing it. And maybe I'm wrong, maybe that's not a problem because AI can continue to do it forever.

But if that AI breaks in any way, it needs to be shut down, as you said before, it needs to have a fail switch. Do the individuals that are now joining the industry, are they knowledgeable enough and experienced enough to perform those tasks manually if they've never, ever done them? Do they understand how the AI agents are actually performing them and what they're doing, or they only really know kind of why?

So yes, there's definitely positives and I would never look at this only purely from a negative lens. But I do think we have to be conscious of what's happening, what is changing, and pivot. We need to make sure that people are trained in AI.

A lot of people are coming out of schools now, and they're not really leveraging it. So we need to make sure that the curriculum and the skills and the training we provide and the support for everyone in the industry is updated. What's your best advice to cybersecurity folks about how to have this conversation?

Because I think many of them understand the basic risk, but they don't want to be the person at the AI party telling everybody to back off the punch, right? Yeah. For me, it always comes down to security's role is to identify risk and inform the rest of the business.

So they need to be assessing the risk, making sure that the marketing team are aware, with collaboration with legal, privacy, et cetera, where relevant, what the risk is based on the implementation of AI that they are looking to do. Once we inform them what that risk is, it may be within appetite, it may still be risky for the business, but within appetite, and they may choose to go ahead with it. And it's up to them, now the marketing team, to own that risk, make that decision, and if any changes to happen, for them to update that risk assessment, make sure that, again, it's still within appetite, and basically conduct business from there.

We definitely shouldn't be a blocker. " It should always be, assess what we see, provide the insight, let the business make the decision. Ultimately, security is another risk.

Don't get me wrong. It should be taken very seriously because of the impact it can have. But ultimately, it shouldn't be security's decision to accept that or deny them the use of AI purely because of a security concern.

It needs to be a business decision. They need to own that. And ultimately, I think that helps our industry as well because it means if an incident was to happen and it came from a system within marketing that utilizes AI, and it was as a result of the AI component, we are then not the ones that the blame is pointed at because we have outlined the risk, and they have been the ones to make the decision to implement it and accept that risk.

Mm-hmm. It seems like the risk comes in three forms, and I'm going to oversimplify. But, the first one is that my AI agent may do something it's not supposed to do.

It might end up deleting something or having some impact that is unforeseen. Second is the credentials may be accessed, and somebody might take over my AI agent and ask it to go and do things and commandeer an entire business process. And the third one, which I don't think we've seen yet, but I assume is going to happen, is that at some point, somebody's going to inject a malicious agent into my workflow, and it may act like it's a normal AI agent for, I don't know, a couple of months, and then start doing some nefarious things.

Is that the spectrum of things, or is there more involved here? Yeah. I don't think I've sat down enough to think about that in depth, but as you were talking, I could, in my head, I'm thinking, again, the example I gave before, someone could also come into your environment, and where there's multiple agents working together, if they just removed one or adapted the skills in one, it would then essentially impact the entire chain.

So there's loads of different things that can happen, which is why we need to make sure we've got control over them. We need to make sure that, for example, agents potentially are time-bound, so over periods of time, we're checking the skills and making sure that no modifications have happened to them, making sure that the access is potentially time-limited, depending on what they're doing, making sure there's human oversight or human in the loop or on the loop, depending on how people want to term it. There's loads of different things we can do, but I think the risks are still something that we're coming to terms with.

Someone said yesterday at the event I attended that I think it's about 53% of global internet traffic is now through AI agents. So we are very much outnumbered by non-human identities now. It's going to be a case of how we control that going forward, because this isn't something that is going to be as easy to retrospectively sort out like the cloud adoption was.

So we really need to get ahead of it. All right, folks. You heard it here.

AI agents are being pervasively deployed. There's no question about that. But with each one, the overall level of risk to the business increases.

Hey, Nick, thanks for being on the show. No problem at all. Thank you very much.

All right. And back to you guys in the studio.