Gou Rao on Automating IT Operations with GenAI

This is Textron tv. Hey guys, thanks for the throw. We're here with Go Rao, who is CEO for Newburg.

They're a relatively new player in the space of AI being applied to site reliability, engineering and IT operations in general. And we're gonna jump into that in a minute. Gu, welcome to sha.

Thanks for having me, Mike. Pleasure to be talking to you again. I think we all know that, uh, at this point, we can't apply AI to those job functions and specifically some of the things that SRE do and IT ops folks, but people are trying to determine to what level and to what degree can we automate things using ai.

So where do you think we're on this journey? I think, uh, we're early and what people are doing is we're building agentic workflows using AI to solve enterprise business problems. So I think for the better part of the past, I guess two years, um, you know, there was a lot of work around building foundational models and, um, you know, um, we all know where we are with that.

And, you know, these models are pretty incredible with the, the amount of knowledge and information they have. But these models actually have two things in them. They have, uh, obviously the knowledge that they've been trained on, but they have reasoning capabilities that can be extracted.

And, um, to answer your question, we're in the early stages of figuring out how to extract the reasoning capabilities that exist in these models and putting them to work on enterprise tasks. And that's kind of where we're at now, uh, building those agentic systems. I think this time last year we were telling everybody that they needed to become a prompt engineer and have that level of expertise.

But if we're relying more on AI agents, how much do I need to know about prompt engineering going forward? Look, I think prompt engineering will always be there as part of now, um, any employee's tool chain, not just engineers, but across the board. Just like, you know, when Google came out and people needed to become, um, adequate in how you use search technologies, there were some people that could Google better than others.

And so, you know, how do you find information very quickly? And so I think in that sense, um, prompt engineering to find, uh, to have the LLM do the right task quickly will always be there. That will now just be a thing that people need to do.

But agentic systems take, uh, are, are, are not necessarily just prompt engineering based systems. And a, an agentic workflow is the following. It's a system that will use the reasoning capabilities and the LLM to work on enterprise data, but then reason with itself to go back and alter perhaps its chain of thought, find some more information, correlate that with different pieces that it's putting together, sort of putting a, um, a, a, a puzzle together to ultimately perform a certain enterprise job function.

So in, in that sense, where a prompt, uh, prompt engineering is a human user using, um, you know, a chat bot or some sort of prompt mechanism to talk to the LLM and pose it a question, but in that case, the human user is the consumer of what the LLM produced. But an agentic system is different. What it does is the agentic system consumes the information or the reasons that the LLM produced to further reason with itself to go back and do more work, e essentially, uh, mimicking, um, you know, uh, task that a human user would've done.

So these are two different things. So what does the future of a DevOps team look like then? Is it all agents or a mix of agents and, well, It's a mix.

Humans are always needed for creativity. Uh, what these agentic systems will do is, uh, take on the mundane, maybe repetitive routine tasks, uh, things that I think people don't wanna do. And so this, what we've seen in our own, um, um, you know, uh, journey here so far, is that people that have been using our agent system, um, they have been able to, uh, get their engineering teams to focus on more important creative tasks.

The productivity has gone up because quite frankly, you know, you, you know, um, well IT operations as well as I do, but well, problems, they come in, um, batches and there's maybe the same problem that happens many times again and again. And so having an agent system kind of go through and respond to all of these, you know, um, level one, level two, IT operational issues is a huge boost in, in, in productivity for the enterprise. Will we get smarter about all this?

'cause the IT environments are pretty complex, and it's not uncommon to find that what you think is half a dozen separate issues. They're all tied to the same root cause. This is, uh, this is where, you know, um, why I said it's the start of the journey.

Uh, systems will get smarter. Uh, and it's not necessarily that the LLMs need to get smarter. I think they're pretty good to go with right now.

But in how you put these workflow pieces together, uh, we have to get smarter people that are building these agent systems need to get smarter. How do you process, uh, a layer of information and then reason with yourself to then go to the next layer and the next layer and the next layer so you can finally put the, uh, pieces of the puzzle together. And quite honestly, you know, look in, in IT telemetry, um, we deal with the sources of information, what we call the melt stack metrics, events, logs, traces, uh, now let's talk about traces.

Traces are extremely hard for humans to kind of, uh, understand and correlate, but these LLMs, um, can do a fantastic job, uh, just because they will stay at it and they're not going to tire. So in that sense, um, you know, finding a complex issue, I think becomes easier with these agentic systems. Initially.

It can perhaps work alongside human engineers to reveal things that the human engineer would not have thought of. Eventually, it will take these things on its own. After having seen enough of these systems, I think there's a pattern that'll start emerging and, um, they can start taking automated remediation.

You mentioned toil, and I think one of the dirty secrets of DevOps is how manual things are and maybe how fragile they are, which all leads to a lot of burnout. So will we actually start to see maybe the turnover rates decline in this space? Because folks, after a while, they just decide that they need to do something else?

You know, um, the toil is, um, is, is real. Uh, it look IT operations and being, um, uh, a for first responding, uh, SRE, it's not a fun job. There is never any happy message that comes in on that PagerDuty alert.

It's always that something is broken or something is not working. And there's a lot of, um, yeah, a lot people get burnt out. And so I think what this will do is bring a sense of relief, uh, to, to those environments.

Um, and, and, you know, um, issues happen all the time for a variety of different reasons. It could be performance issue, it could be just that an application is flat out crashing human related errors. You know, it's not a stagnant environment.

Um, especially if the enterprise is moving fast and creating new product. Well, they're putting out new code and they're pushing new code into production, they're making changes and things are bound to break. But having, uh, an agentic system that is watching and has your back gives you a lot more confidence, can run your operations a lot, much more smooth.

And I'll, I'll say one more thing. Look, humans become invariably experts in certain domains, maybe because that's what they've been trained on. Maybe that's what they're inclined to work.

And so, what do I mean by this? And when you walk into an IT operations environment, there's always that person that's, uh, knows databases the best, and then there's a different person that knows networking the best, and there's another person that knows Kubernetes the best. So people become domain experts.

Now, when a problem happens, I think you and your audience would know that typically problems are horizontal across a different, uh, domain. So now to respond to an issue, you have to get the networking expert, you have to get the Kubernetes expert, you have to get the database expert, and everybody has to sit together and figure out where the problem really is. But an agentic system, because of the knowledge and the LLM sort of transcends this horizontally.

One agent can sort of has the knowledge of all of these different complex components and can put the pieces together a lot faster. And in most cases, more accurately, Do you think we'll have smaller numbers of DevOps teams managing larger environments? Uh, and that's one argument that you hear people make.

But conversely, there's another thought process that says, there's a lot of organizations that couldn't afford to get into the DevOps game at all because they couldn't afford to hire the engineers that now DevOps becomes more accessible to. So ultimately, what do you think the impact of all this is gonna mean? Uh, you know, I don't want to make it sound like I have a crystal ball or know where this is going, but, you know, um, I have a hypothesis.

My, my, um, thought would be that it's not an either or. Um, my sense is that because people in general have such a desire to, uh, produce a more product and create and create more that, uh, teams I don't think will get smaller, they will probably be focused on doing more creative tasks while these agents are picking up all of this, uh, work. And w where, where I'm going with this is most of the enterprises I talk to, they're constrained.

They have so many projects they want to do, but they can't do them because their DevOps teams are stretched too thin. But now having these agentic systems, uh, allow the, gives them breathing rooms so that now they can do those projects that were put in the back burner and they can be more creative and put more product out there faster. So this is about, um, you know, uh, an enterprise and I think, uh, people in general being more productive by using AI on their side.

I know you just partnered with Microsoft, and I think your software's available on their marketplace, but they and others have said we might be deploying more software in the next two years than we have in the last decade. Is that really feasible? I look, the, uh, the, the compute power exists.

Um, it's there. So what, and the reason I think that that's possible and why people want to do that is because of agentic systems that allow people to do these things. You're all of a sudden you've augmented your workforce with these agents where you are constrained by human talent.

You just, there just aren't enough. Well, in our domain, at least, uh, enough computer science graduates that are graduating, let alone people that wanna be an SRE or deal with, um, you know, uh, high highly stressful IT support type of environment. So now having these agents, uh, allows people to deploy software faster because agents are there to, uh, actively and proactively monitor the environment, uh, catch issues, fix them on the fly.

So, yeah, um, I think it's po definitely possible because the pa you know, the com compute and the infrastructure resources exist, and with the agents now, uh, you know, starting to, uh, help enterprises, it now becomes feasible for the same size team to roll out that much more software and manage it effectively. Also, one of the other dirty little secrets of it is that the utilization rates of our infrastructure is pretty low. So if we have more of these AI agents out there, will we increase the utilization rates for those processors and generally be more efficient about, uh, energy and lower the total costs per application?

Does that make sense? Absolutely. Look, um, eh, uh, uh, right sizing or cost optimizing your, uh, environment, whether it's cloud or even on-prem, um, that's part of an it, that's part of it.

Operations, you, you just have to do that. And it's always, um, you can't size the operation once and forget about it because workload keeps changing. Um, you know, there may be problems in your physical infrastructure that requires rebalancing.

So all I'm trying to say is that, um, properly, um, configuring and deploying your environment so that the resources are used efficiently is not a one time thing. It's a continually ongoing task and a perfect job for an agent to do because it has access to all the telemetry, the metrics, the alerts, the logs. It can meticulously look at every possible CPU that's deployed out there and, and, and correlate with what applications are running, how much memory is being consumed, how much CPU is being consumed.

This is something that a person should not be subject to doing 24 7, but an, uh, agent, why not? So as you kind of think this through for a minute, what do you think will be the ultimate impact here in terms of will more people focus on developing software? Because we're, it is not just the SREs that have agents, the developers will have agents, and how will their agents and the agents that you created for the SREs kind of communicate with each other?

Uh, you know, it, people have been, I, I think first of all, um, it's, uh, that is in everybody's mind. As people are deploying agents, they are wondering about inner agent workflows and how do agents communicate with each other. Um, but in all honesty, as we speak, um, at this moment, right, March, 2025, people are still rolling out agents.

So while it's in the back of their mind, uh, I haven't actually run into a situation, um, where somebody's saying, I definitely need to solve this problem, which is my developer agent talking to your, um, in our case, our agent is called Hawkeye, the SRE agent. It, the writing's on the wall that's going to happen, but I think that's probably a next year problem. And there are definitely ways in which this can be done.

Uh, clearly they can communicate with each other in unstructured ways. Um, LLMs do a great job in that, but I think that there's probably, uh, better, um, uh, integrations that can be done with an explicit, um, you know, communication mechanism across, uh, agents. So, you know, an agent checking in code and then making that known such that the SRE agent can pick it up and make sure that it will watch for any deviations in the, in the environment.

Those can be done either explicitly or implicitly. Uh, and, you know, all I'm trying to say is that that's something that we'll have to address, but it's a solvable problem. It's one of the easier problems to solve, and I think people will start asking for that.

Can we achieve all of this using our existing DevOps platforms or do we need to also upgrade those or can the agents just work with them as they're, Yeah, it, we, in our case, and I think mostly everybody's of this, uh, opinion that don't change the existing platforms, whether it's DevOps, um, whether it's observability, right? Um, I think the existing observability tools out there, uh, that do logging and monitoring and tracing, they do a fantastic job. Uh, we've spent as a society, our engineering society, the better part of the past 15 years in optimizing for these tools.

Uh, you know, we know how to store distributed traces very efficiently. We know how Prometheus, we know how to store time series data very efficiently. So there's no reason any of those things should be redone.

The purpose behind these agents in the design should be that they can work with the existing systems using existing protocols, using existing access mechanisms. Look, um, I'll actually take that to say one thing over here, which is, in building an agentic system, AI matters a lot. But equally a big problem is providing a data access platform.

Um, like at New Bird, that's one of the most important things we do, which is how do you interface AI sensibly to enterprise telemetry data? How does AI access your logs? How does AI access your metrics?

How does AI access your, uh, traces? And clearly you can't dump all your logs into the AI either. You need to only access what is needed for that question or that issue that's being addressed at the moment.

And so that is, in building an agent system, that's one of the most important things to keep in mind, which is your information, enterprise information, crown jewels, you can't change it, you shouldn't change it, but your agentic system needs to access it properly without any, um, change requirements on the enterprise side, but yet get the AI to work. That's the challenge. So if that's the case, we'll the shift to AI essentially drive more people to embrace something like platform engineering.

'cause I'm gonna need a little more discipline to harness and pull all that data together. I think so, and I think that's where people, uh, like us, like at Newberg, that's what we do. We are building a, um, it telemetry access platform so that every enterprise out there doesn't have to go create their own at the end of the day.

Uh, OO one of the biggest, um, uh, pain points we do solve is creating a telemetry access platform that essentially solves that problem, which is, uh, in a sensible correct way, allows the LMS to access the information so that they can reason with, uh, themselves reason with the data, and, um, provide an answer to the issue that's, that's currently being faced. All right, folks, you heard in here. Hey, it looks like we can get to AI from here and we might be closer than you think.

Hey buddy, thanks for being on the show. Thank you very much, Mike. Appreciate it.

All right, and back to you guys in the studio.