Enterprise AI Agents Need Governance Before Autonomy

AI Leadership Insight Series. I'm your host, Mike Brazier. Today, we're with Harshil Shah, who's Director of Engineering and Head of the AI Center of Excellence for Aura Systems, and we're about to have a little chat about, well, these AI agents.

They're kind of risky, and we got to understand what's involved. Harshal, welcome to the show. Thanks, Mike, for having me again.

It's always good to discuss the scalability and making sure that we're educating the world about which agents are right and which agents are doing it the wrong way. I think at this point, most people are at least aware that there are risks involved, but I don't think they understand the depth of those risks and what exactly is going on here. So what are we not thinking through enough as we deploy these AI agents, many of which are deployed by end users who don't have a clue?

Yeah, exactly. Right. So I think, see, risks are multifold when you are deploying AI agents, right?

Some risks are that are purely systemic or technical risks. Risks like security or risks like prompt injections, risk leaking, or risks like your server shutting down or something like that, right? Those are something which are more engineering-facing risks, which you need to factor in when you're thinking about the architecture of the agents, right?

Then there is another set of risks that we see is more outcome-oriented risk, where you spend a lot of money, you invest a lot of time into building these agents for your enterprise, but you don't measure them correctly. Are they getting you the same output? Are they getting the outcome that they were planned for you to get, right?

And people get into a lot of the day-to-day, or they get into the nitty-gritties of making sure the agent works without looking at the larger picture, right? So recent surveys have shown that in the last 12 months, more than 56% CIOs have said that, "Okay, we've not seen neither an increase or a decrease in our overall outcomes after we've deployed AI agents," right? So the other risk is this risk where you invested so much time, you invested so much money, but you did not measure the outcomes enough, and that is where you started going behind in your journey of getting these agents live, right?

And the last risk is the actual money risk, right? Where in order to get the agent to a certain outcome, we don't really strategize how to make sure that my token costs are curtailed, right? Because in a world where memory is getting expensive, I cannot possibly route every request to frontier models.

I cannot use the best models out there. In fact, recently, a lot of the frontier models that were recently released, we see that they use very little in actual production agents because of the fact that these costs are not sustainable. And on the other side, we're seeing that open models or models that are smaller but more powerful and more agentic in nature, they're doing possibly very good.

But at the same time, we're also not seeing their adoption to be very heavy in actual production use cases. So you know that gap, bridging that gap is very necessary to curtail your token costs while making sure that your outcomes are ready, right? So those kind of risks, where you have systemic risks with respect to the agents' identities, agents' access controls, agent security.

Risks with respect to your strategy risk, where you invested so much time and you didn't get the outcomes, and your actual token cost and your money risk, right? So that's broadly the kind of risks which everyone should consider when they're building agents for their enterprise. All right, well, let's work backwards from the last one to the first one.

The open weight models sound good in theory, but for the most part, they seem to be developed in places like China, which creates concerns about, well, what exactly is that model been trained to do and say, and what else might be in there? So I wind up stuck in a hard place between very costly proprietary models and models I may not trust. So how do I navigate that?

Right. And you're right, that a lot of the innovation with respect to the open weights models that is happening in China. However, if you think about it, fundamentally, these open weight models are designed, A, I think they're like an open kimono, right?

You can see the exact weights that they're trained on. You can actually completely air-gap the deployment, or you can completely make sure that these open weights models can be deployed in a very controlled environment, right? So the Chinese open weights models, although the risk of how they are trained, what are they supposed to answer, et cetera, always runs.

But the fact that these models serve as really good base models that can be post-trained, that can be fine-tuned for an enterprise's exact needs, and at the same time that they can be air-gapped, and they can be self-hosted, and they're completely end-to-end observable is why there is a slight inclination towards making sure that they are available, or they are very viable options for enterprise deployments. However, there are some really, really good models that are beyond Chinese models as well, right? So, for example, the recent Gamma 4 model.

It's a very powerful model. It's probably a 30 billion parameter model in size, which is very, very good at agentic tasks, very good at multi-modal tasks with respect to images, image processing, and all of that, right? So those cases as well, we tend to not look at open weights models with the same lens as we look at our frontier models.

So even the North American open weights models have become very good. Even today, so many actual production use cases that we've also personally deployed, that we see in proper enterprise are actually using the OpenAI's GPT OSS models, which is 120 billion parameter open weights models, which you can definitely use, right? So the open weights models, the ecosystem is very wide now, and I personally foresee that a lot of enterprises will adopt them because a lot of the frontier companies are also promoting them.

Alex Karp talks about it, Satya Nadella talks about it. All of these companies keep promoting the fact that you should definitely use open weight models. A lot of the hyperscaler companies or a lot of the companies that build agentic platforms like Lysa, they actually have new features where they tell you that, okay, you use a frontier model for a specific task.

As a shadow, we will fine-tune a smaller model with those same tasks, so you can start routing your requests to the post-trained or the models that's trained on your specific ecosystem. So those kind of things are definitely going to make sure that your open weights models are very, very viable for production deployments. All right.

Going to the second item, trying to figure out what the return on investment is. I think people are struggling with the fact that, to your point, these things are getting a little expensive, and I might wind up spending a million dollars to replace somebody who makes $45,000 a year. But the other side of that coin, though, is I'm not sure there are returns, and I think how much of this is just the cost of doing business, because if everybody automates an AI workflow, and let's take a hospital, for example, I'm going to be more efficient, but so is the hospital down the road, and at the end of the day, there isn't going to be any more patients because the number of patients are finite.

So at the end of the day, is this just table stakes? Yeah, it is, and I completely agree that it's table stakes, right? So I think the idea there is that the utopian world where everything is going to get extremely automated, it's a mile away.

It's at a good distance, right? So for my enterprise to survive, I do need to make sure that I am able to build the agents in an iterative fashion, right? If my job from day one is to build agents that can replace my $50,000 accountant, then quite naturally, in the process of being able to do that, I'll end up spending a million dollars, right?

That's always the case. But- If from day one, what we call as tiered promotions, even within our humans, we have tiered promotions, right? So if I start out with some of the agents that are able to take away some of the very mundane tasks of my enterprise, and I build those agents in a platformized way that I can scale them very easily.

Only then can I give my enterprise the luxury of affording these iterations, which will help my agent from doing pointed tasks to taking up some more ownership, to taking up some more ownership, and eventually reaching a stage where the agent is autonomous enough that it can replace an actual worker, or it can at least complement that actual worker so that worker turns from a worker to more of an orchestrator of multiple such agents, right? So that tiered promotion or that tiered autonomy is what we need. I'll give you a good example.

Sastor is one of the very, very famous and a very good implementation of SDR agents, agents that can typically act as your sales team. They've had to go through 67 or 69 plus iterations before they could actually claim that this agent is autonomous, right? And this is the same case that even us as a very heavy agent engineering services company that we see day in and day out, is that you need to be able to build your platform or your agentic platform in a scalable way that you can give the enterprise the luxury to keep on having the iterations of refinement and promoting your agents through tiers of autonomy before it reaches a stage that it can replace an actual humans, right?

And at this point, it is just that for any enterprise to survive, even if it is stable stakes, you have to invest and have to get it done. It's just how you invest and where do you invest and how do you build the fundamentals of your agent ecosystem within an enterprise is going to get very important. And then coming around full circle, people are clearly concerned that the AI agent's going to go rogue.

And the issue is, I don't know if it's the AI agent and the guardrails and the AI agent is ignoring the guardrails, or is it just that we have a lot of weaknesses in the systems and there's a lot of interdependencies and the AI agent discovers things because, well, AI agents do what AI agents do, and they do things that humans wouldn't think about doing, but they have found ways around different rules and regs, and they find some way to get to where they want to be. How do we manage that? Right.

So, it's the same argument that how will you manage that if you hire a human who has access to your systems, right? You have governance in place that will make sure that there is a certain role with a certain set of permissions, with a certain set of enforced accesses across each system that is assigned to your humans, right? A very similar philosophy actually needs to be followed when it comes to agents.

But thinking about a scale that is slightly more, right? Because a human is probably interacting with five systems maybe 10 times a day, but an agent will interact with 50 systems 400 times a day. Right?

So it's important that your guardrails cannot be something that is an instruction to an agent. Guardrails and accesses need to be a part of how the tools are provided to the agent, right? " It's bound to fail at any point, right?

But if I have very detailed RBACs and role level securities that are into place through the tool itself, which gives the agent access to the database, if that tool is governed, it has all the permissions in place, then even if the agent does try to access data that it's not supposed to, it's going to get short-circuited there, right? So definitely making sure that-- And which is why a lot of startups are coming up with the fact that there are agent identities in place. How do you build governed connectors for your system of records in place?

And even the actual system of record, the Salesforces and the SAPs and Salesmines of the world, they expose their tools in a very governed way, and they allow you to put your own set of access and permissions on those connectors itself, which will allow to make sure that no matter what kind of agents you build, and even if they do go rogue, they will follow a certain set of permissions and roles. So agent identity, agent roles and permissions. Your actual RBACs for your agents is something that you need to consider when you're trying to build agents that are going to do some kind of autonomous tasks, right?

So that will ensure that your agents don't really go rogue, rogue. But yeah, definitely, contained environments, not just in terms of tools, but also the actual infrastructure around it. Continuously monitoring an agent's traffic, continuously making sure that I have alerts set up for anything that the agent might be doing which is not expected.

Continuously having an observability and monitoring dashboard and having the right kill switches in place. You have to build those in the foundation layer so that the agent that you do build, you can build it with the hope that it doesn't go rogue. So when you put all this together, will AI agents kind of require us to go revisit how the entire IT stack is constructed?

Because it was constructed for humans, and- Exactly ... increasingly, we're going to have AI agents that just fundamentally behave differently. Yeah, that's right.

We will have to reimagine the scale of how a good typical IT organization is set up for an enterprise. But it's not something where it's more of-- It wouldn't be something that you completely rip and replace, right? It's just about thinking about scale, right?

So, for example, if my IT infrastructure is built in a way that I have to manually provide, or I have to manually give out permissions, or I have to manually assign roles to each agent, and to each system individually, that's not scalable. That was the old way of doing it with humans. But having a more autonomous, having a more framework around providing roles and identities and permissions to agents, a more autonomous and a more scalable way is the next step from what the foundation of the current IT is.

So that governance piece to take care of scale of an enterprise running 400 agents with access to another 100 system of records with millions of transactions per day. Being able to govern that, you need slightly more automated way of handling your IT and your ticketing systems. So ultimately, am I going to need AI agents to reinvent my IT environment so that other AI agents can use it?

Yeah. Well, you could use AI agents to accelerate the process of reinventing, but you wouldn't want the agents themselves to be running your IT, right? You probably don't want that.

You probably want to have a governed, policy-driven engine in place that is more deterministic, human-controlled, and has the right elements of AI in it. But you definitely don't want an agent to actually autonomously decide who gets what kind of permissions by reading a policy doc, right? That's probably not what you want.

You probably still want a human clicking the buttons to do the provisionings. But the buttons they click shouldn't be to give the access, but the buttons they click is just to approve, deny, and make changes. That's what they should be doing with the right set of human in the loop.

" Yeah, I think most of the areas is where we start with is the first step that enterprises take, and then that is something that we tell them that they shouldn't really do is Often they're done with their enterprise procurement of the Anthropics and OpenAI of the world, and they have let's say 200 Claude licenses. They'll give it a free-- They'll just roll out the licenses, and they'll expect that we start seeing some productivity. And they'll probably have some plugins or some skills that they've built for their organization, which they'll roll it out, but none of that is governed at all.

So someone who will use the Salesforce MCP to connect their Claude to Salesforce and just start creating entire Salesforce in its entirety, they can start creating multiple Salesforce data objects without any kind of governance in place. So I think that's just one of the major bits where there's no observability around an enterprise's Claude or an OpenAI or a Gemini procurement or an ecosystem is doing. There's not much detailed observability around it.

There's no governance around it. And then there's also no way to make sure that there is a standard way of building agents on top of your existing system of records using your existing procured LLM partner. So that has made us somewhat that, okay, this is a big no-no because we get that rolling out a Claude, it looks like the logical next step because everyone around you is doing it.

But what about making sure that the governance on top of that is covered from day one is very important. How will the AI agents going to interact and negotiate with each other? Because some of them, especially in a lot of businesses and organizations, business units aren't always on the same objective, and sometimes they're occasionally working across purposes on, but that's with intent.

So how will AI agents that are autonomous, that might meet each other, negotiate, and maybe not wind up sabotaging each other? Yeah. So the thing is that, I think it's a very big red flag in the first place if two of my agents are-- Okay, so your agents should be built in a way that they are trained to do a certain reasoning plus mechanical activity that takes away or that helps your existing team be productive.

You don't want the agents to make decisions on your behalf. You still want a loop to come through the human for, to be able to do that. And if there are two sets of agents that are trained on the same company's business fundamentals or their business policies, and then there's a certain disagreement with respect to a certain area, then that is something which definitely you don't want agents to negotiate with each other.

If that gets caught in the observability of your platform, that there is a certain level of disagreement when one agent was doing a handoff to a certain other agent, you probably want the human to be able to come in and define that or go back to the source and try to fix it, is very important. So for example, let's say if I'm building HR agents. And there is a disagreement with respect to what a typical notice period might look like.

Is it 30 days or 60 days, et cetera. Things like that. So those kind of things you probably want to, and this is a very basic example.

I can imagine that the conflicts might be a lot more convoluted, but conflict resolution or disagreements between two agents should, and especially if they're within the same enterprise, is more of a fundamental issue that needs to be resolved. Because when you've designed the agentic system in a way that there will be a handoff from agent A to agent B, you do want to make sure that agent A's tasks and agent B's tasks are disparate enough that there is not a lot of points of conflicts between them. And secondly, if there is a certain conflict that arises, then it's something more fundamental within the knowledge base that both the agents are feeding off of that needs to be resolved.

So for sure, you don't want agents to be doing conflict resolution on your behalf. All right, folks, you heard it here. When it comes to AI agents, there are degrees of autonomy.

Work backwards from there. Hey, Harshal, thanks for being on the show. Thanks, Mike.

Thanks, Patrick. Thank you so much. AI Leadership Insight series.

You can find this episode and others on our website. We invite you to check all those out. Until then, we'll see you next time.