AI Governance Goes Beyond the Agent

ai Leadership Insight Series. I'm your host, Mike Vizard. Today we're with Kai Waehner, who's the Global Field CTO for Kestra, and we're having a little chat about, well, how to govern and manage and secure all these AI agents that are starting to proliferate everywhere because, well, everybody's talking about it, but I don't think anybody's quite sure what to do about it.

Kai, welcome to the show. Yeah. Hi, Mike.

Glad to be invited here. Thank you. Most of the focus today seems to be on applying guardrails and maybe some controls to the AI agents themselves, but the AI agents are, shall we say, programmed to aggressively complete whatever task it is assigned to them, and sometimes they will just go find some other means that we weren't thinking about.

So is there another way to think about how we're going to govern these things, and maybe we're kind of focused too much on the agent itself, and maybe there's other things we should be doing here? What's your take on what's going on, my friend? Yeah, I think that's already a good summary, and that's what we are all learning over the last years and learning some best practices.

So also focusing on the agent itself is one important piece, but it goes much beyond that. And you also had other shows about this, where the discussion is about multi-models, also for cost reasons and so on. So there's various reasons why not everything should be implemented or decided close to the agent or close to a specific vendor.

So what companies now learn is that there is a much broader spectrum, that you need to take a look much beyond the agent on very different levels. Like model choice is the first thing. Like which agent to choose, even sometimes not just from one vendor, but maybe multiple vendors or open source and so on.

And then add the levels on top of it, like what happens if the thing fails? So changing a step of the business process, or if something crashes, or if you really need a human approval, or if you need to handle an incident. So there's so many complex questions on very different layers, and this is really why I think we need to talk about AI governance on a much other level than just talking about the agents, which are just one small piece of the problem.

Right. Are we finally having maybe an overdue conversation about all this? And I ask the question because I think we designed policies and controls for humans, and the assumption was that humans would only go so far and make so much effort to kind of get around something.

But the AI agents have, well, infinite patience, shall we say. And so are we revisiting the controls because we have a new entity that never gets tired and will just keep doing things until it finds a way? Yeah, and that's a great point.

And actually, what I recommend to our customers is that they should think about agent more in a similar way to how they execute a task. And that could be similarly like either another automation does it already, or maybe a human does it in some specific decisions. And in this way, if you take a look at it from that perspective as a task, then it's much easier also to use already existing policies and also use them for agents.

So the most dangerous part is to implement agents by themselves outside of the existing controls. But instead, you have to find the controls, because if you're in a regulated market or compliance or PII data, the rules already exist, and so you can reuse them. And sometimes the tech or interface is a little bit different and agents are much, much faster, but the decisioning and the risk in the end about the outcome is very similar.

And this is why, as you say, in the end, you can use the existing rules and governance in a smart way also to embed it around agents. Are we about to see a wave of fines being levied because people are rolling out these AI agents in the absence of governance policies, especially in regulated industries? And last time I checked, the auditors are not going to be forgiving just because you had an AI agent.

Oh, yeah. I think this will explode. And actually, since ChatGPT started, and everybody learned about large language models, in the meantime, the executives and large organizations learned that they need to start with governance in mind from the beginning.

So there is an interesting statistic from Gartner where they say that until 2027, there is 40-plus percent of agentic AI projects that get canceled. And the reason very often is not cost or no business value, but it's really no control and risk. And this is exactly what you need to handle.

And so, many companies will do the rules beforehand, but otherwise it will break. And I mean, we have seen enough examples here. And agents also get more and more powerful every quarter.

One of the most relevant examples, Hugging Face, which everybody has heard about, when OpenAI models indirectly attacked them unintentionally. But there is tens of other of these examples across all vendors already. And as an end user, as a normal company, I would say, where you don't have billions of dollars spent into AI and research, it's much more risky because you don't have these controls in place already if you don't do it right.

And so I think, yes, this is a huge risk, and that's in the end what companies need to solve. Mm-hmm. Won't the auditors also have AI agents that they will use?

And won't they be using those AI agents to discover whatever missteps the AI agents that were deployed by an organization made, and they'll just be chalking up fines by the minute here because they'll be using their AI agents to figure out what the AI agents did that they weren't supposed to be doing? Yeah, I think so. Everybody's using more and more AI because it automates things.

It makes things faster and cheaper. But for auditors, it's the same question. An AI agent needs to run as a governed step in a specific domain, like when you have a regulated process where maybe a human does the final approval of a check, but before, a lot of them are handled by agents.

But for the auditor, it's exactly the same question. And this is why when I'm in New York right now for a reason. I've worked with a lot of financial institutions, and they all want to implement more and more agentic AI within their processes.

And in the end, what they realize, they need to have a control plane on the top level so that they have one audit trail, including human approvals, including automations, sometimes legacy on a mainframe with a very old tool and a schedule that is 20 years old, but also incorporated with any kind of agentic AI, no matter if it's the frontier models from Anthropic, OpenAI, so on, or maybe something they can deploy even in their data center. But this end-to-end orchestration for regulated markets and audits, this is key for both sides. And everybody will leverage AI more and more to automate their own tasks.

How automated will all this get, and what will be the role of humans in it? Because I think part of the concern is that the AI agents are acting at machine speed, and if I got to throw a human in the middle of that, it defeats the purpose. So how do I kind of govern this in a way that's operating at machine speed that doesn't always require the AI agent to stop every five minutes and ask for some sort of blessing and checks and balances?

Yeah, it's an important question. On the one side, there is more and more data. So digitalization is everywhere, and data is growing.

So there is more and more use cases. So I'm not worried that people don't have work anymore, but obviously the work will change because a lot of the either simple tasks or sometimes even complex calculations will be done by automation or by AI, and that's totally fine, and the humans will always be on top of that. So the more critical a decision is, the more relevant it is that a human takes the last approval or the last step or some other action they need to do, like sign a contract or something like that.

And therefore, yes, AI can take over a lot of things, but I think we need to be careful dreaming of autonomy only. So not everything will be agent on all the sides. And you can take a look into any kind of industry.

So this is not just regulated markets like here in New York where we are today, but it's really about everything. Like if you think about the travel industry where you book an airline flight, that's a great story every vendor talks about these days, where agents can automate that, even B2B. There are one agent on one side and one agent on the other side.

But the reality behind this, there's so much complex transactional workload where, again, many IT systems run behind that, not all of them in real time, and a lot of different rules and laws and regulations. And so it's really hard to implement this. And then if you want to book a flight and change a seat, that's already a hard problem where a lot of things can go wrong.

And with that, then also the cost and risk behind that. And this is not just in this example about the travel booking. If you go one step further into the operations, like the airline travel, when the flights are going on and a flight is canceled, and you need to rebook the seats to other flights, then it gets even more complex.

And so these are business processes on the high level. Agents can automate a lot of that, but humans will always be in the loop for the critical decisions to get it right. Mm-hmm.

As this kind of all evolves, we're going to have swarms of AI agents, possibly thousands of them, and I feel like our ability or cognitive ability to keep track of what they're all doing at any given moment is going to be beyond our governance capabilities. It's not just a technology conversation, it's a human conversation. So how will we approach the management of all these things when so many of them will be operating in parallel in ways that we might not be able to keep track of?

Yeah. This is a problem today already, right? Now, when you write code, one more code is generated, you get much more GitHub pull requests and so on because people update much faster, they implement new features faster, they fix bugs faster, which is, on the one side, a great thing, but on the other side, it's more and more and humans cannot track this anymore.

And therefore, again, it's in the end about automation also of the governance, of the observability. And then at the right points, a human needs to take a look, and this is where we still will need the developers and the infrastructure operators, so that they take a look at the right time. Because in the meantime, already we get so many logs.

We were talking about millions here. Just to give you one concrete example, at Kestra, where we do this orchestration around all these kind of pipelines, JP Morgan is a great example. So they have over 100 analysts that process billions of security records per day.

And this is crucial because if they don't do that the right way, then they are at risk from a cybersecurity perspective. So they need to do this, it's mandatory 24/7. And there is 100 analysts, so they cannot track down billions of security records.

So in the end, there is integration of all these big virtuals. There's observation layers on top of that across the different domains, including infrastructure, where, for example, the certificates are, but also applications and data, for example, for reference analytics. And on top of all of that, there is still humans.

And the automation solves a lot of problems, but if there is a problem or incident or if there's an approval needed because a decision is too big, then a human comes into play. And this is, in the end, for all these kind of agents, no matter how many there are, that needs to be automated, that needs to be governed on top, and then the humans can take action and control that at the right moment. Mm-hmm.

As you work through all of this, do we need to revisit our existing governance frameworks? Because I don't think they were designed for all of this. Or can they be extended?

Are we looking at a moment here where we have to rip and replace what we have put in place for the last two decades, or can I extend what I currently have? I think it really depends a little bit on the industry and use case, and generally, we already have this problem for many years where more and more is digitalized. In factories where they are now paperless, or in the future, we talk about dark factories, where a human only go in if there is something, a machine or robot is down and you need to start it again.

So it's definitely going into this direction of digitalization. And in the same way for governance, I think many of the rules still apply in a very similar way. But how the rules are applied, this is changing because it's not just about a little bit of data, but the data is growing.

Instead of having 10 humans, you have 10 agents, and they process that, and maybe one or two humans. So I think some of the steps are changing, as I said, with the applications and humans, now we also have agents. But overall, from a governance perspective, I think we are already pretty solid and we can more or less modernize what we have instead of completely the need to reinvent it.

Mm-hmm. What will be the role of various regulatory agencies out there? And I'm asking the question because some folks I talk to say, "Well, yes, we may not need any or allow any AI specific regulations," depending on where you are in the world.

But they also point out that the existing requirements cover a lot of this stuff already, and that there's not a lot of net new under the hood here. At the end of the day, if your AI agent does something that violates a compliance framework, it's still you doing it. So, I guess, do we need to update HIPAA and all these other things for the age of AI or no?

I think, as you said, many of these things are already regulated. And there is a lot of playbooks like in the US, there's NSA or CISA. So they're talking about handling rogue agents better.

But in the end, as you said, this is really about maybe updating or modernizing a few specific rules. But overall, it's the same rule like before, but now it's not just applying to humans and automation with computers, but now it's a certain component. It's an agent, but the rule is still the same.

It's maybe much higher volume, but still you have the same rules, and therefore, I would also not be that much concerned. It's much more about the digitalization in general. No matter if it's by an agent, by a human, which does a button click in a web app, or if it's by another software.

And that's, in the end, like in Europe with the Cyber Resiliency Act, and we see exactly this. So this was never built for agents, but it was rolled out in the last few years and is applied now in the European Union so that manufacturers are cyber resilient. So that they implement the right rules.

And no matter if the outside attacker is a human which sits somewhere or if it is an agent, but the problem then is also the same. Mm-hmm. So what do you see people doing out there with agents at the moment in terms of governance that just makes you shake your head a little bit and go, "Folks, we might want to be a little bit smarter than that"?

Well, I think the biggest problem I see regarding agent AI is shadow IT, because it's very easy to use them, and you can use them also around processes and hide them somehow. So this is definitely very, in large organizations, business units build their own applications, their own UIs, their own customer-facing interfaces because it's so easy. And thanks to standards like MCP, even other agents can then talk to them from the outside world.

And therefore, definitely, and this is not just a tech problem, but therefore, AI governance on the executive and C-level is one of the most important topics so that you avoid shadow IT. Because in contrast to 10, 20 years ago, when shadow IT was a small problem and typically a cost factor, today, it's really a security and a governance problem, and that's what the executives need to solve. And this is not just tech, this is really also a change management problem and really governance on top of the humans and not on top of the agents.

Are we waiting for a couple of major incidents here before we all get serious about it, or will we kind of suffer through this for a year or so and die a death of a thousand cuts before we get serious about it? I think both happens in parallel. So companies are investing more and more, but as soon as you get hit, you're in trouble, of course.

And also, as you know, defining something from a regulated market or from a law by the government, this takes time, and so I think it's coming from both directions. But if you also follow the market, how even the AI vendors like Anthropic, which filed for an IPO recently. If you take a look at their 400 pages, and 100 pages or so are only about the risks of their own AI technology.

So I think the awareness is there, and this is a problem which never existed like this before in the software industry. And therefore, I really think it's coming from both sides, and it's really a top priority of every executive and every government. Mm-hmm.

Is there something that the Anthropics and the OpenAIs of the world should be doing here that they're not? Or, at the end of the day, they're just kind of creating the core technology, and it's up to us to figure out the governance around it? I think that part will happen anyway.

So that's back to your point in your other session that you had about the multi-model approach. Where there's also models from China and so on. If you stop innovation, someone else will do it.

So stopping that does not work. But how do you control it? This is, in the end, where each country needs to have the rules and the laws to solve that.

So this is not really about stopping innovation, and these vendors try their best, but in the end, there needs to be governance, there needs to be rules, there needs to be enforcement. Otherwise, it's hard to control. All right.

Well, folks, you heard it here. Hey, AI agents, they're both your best and worst employees ever. They're the best because they're very productive.

They're the worst because they never met a rule they didn't want to find a way around. And Kai, thanks for being on the show. Thanks for inviting me.

All right. ai Leadership Insight Series. You can find this episode and others on our website, and we invite you to check all those out.

Until then, we'll see you next time.