Observability Becomes the Control System for AI Agents
Agent observability starts with a simple question. What did the agent actually do? When agents write code, call APIs and change production, the answer can no longer live on a static dashboard.
Three observability leaders examine that shift with Mitch Ashley at KubeCon + CloudNativeCon in Salt Lake City. The panel features Honeycomb’s Jessica Kerr, Dynatrace’s Sean O’Dell and Splunk’s Greg Leffler.
Tracing Comes First
Kerr says any question about agents begins with knowing what they are doing. New agent frameworks have embraced OpenTelemetry from the start. Tracing shows where a request came from and how the agent got there.
O’Dell notes that developers still love logs more than traces. That is starting to change. He also finds that people want visual evidence before they trust an agent’s answer.
Leffler says static dashboards have a foot in the grave for engineers. The missing piece is tying agent actions to business outcomes, such as revenue or customer satisfaction.
Closing the OpenTelemetry Gaps
Leffler sees gaps in how prompts, tool calls and agent actions are named and collected. O’Dell points out that OpenTelemetry was not originally built for AI. Dynatrace acquired Arize, whose OpenInference instrumentation was donated to OpenTelemetry.
Kerr says the semantic conventions are published. The libraries and agent harnesses still have to catch up. The model may be a black box, but the harness is deterministic code. As she puts it, the harness is the model’s fingers.
Guardrails, Evals and Defining Good
Leffler argues that deploying AI without guardrails is irresponsible. Few teams run them today because evaluation is slow and expensive. Observability is how teams get those guardrails.
Defining a good answer is now hard. Different agents can reach different conclusions from the same data. O’Dell suggests letting the system choose, within guardrails set by people.
An audience question asks whether evals count as observability. Leffler says they do. The panel agrees that agent observability will keep growing beyond logs, metrics and traces.
Transcript
Welcome, everybody. Welcome to the first Futurum Summit. Sounds like such an important gig.
It is. It's an important gig. Panelists, if you'll come join me.
Jess, Sean, Greg. We super fortunate. Top of my list of three companies who'd like to join me on this panel, and they all said yes.
And they're all DevRel. I love the data center conversation, but I love, love the software conversation, especially about observability. So, real quick, how long you been with the Honeycomb?
Five years. Five years. Nice.
And in DevRel the whole time? Yes, it's my first DevRel job. " You have to be a special person to do a DevRel job.
Nice. Do I speak truth there, Sean? Absolutely.
I've been at Dynatrace 20 months. Feels a little bit longer. And I will say, I am not supposed to be up here today.
You have been replaced... Or my Canadian peer, Jonathan Norris, was unable to join, so you got replaced, or he got replaced with a Texan. So I don't know if that's worse or better, but it is what it is.
Greg, it's all yours, buddy. I've been at Splunk for about five and a half years. And enjoyed every second of it.
So you not only love developers, but you love talking to developers about development. Of course. Very cool.
Hey, I want to jump right into the deep end of the pool on observability. We all know the history of telemetry and logs and traces, et cetera. In the agentic world, where we're now talking about agents doing things a little differently than static software.
We're talking about moving, not just having dashboards, maybe not moving away from them, but talking about what agents are doing and how we control them, and how much of a role does observability have in that. You hear agent control planes, a lot of different things of frameworks about how that works. Love to get each one of your perspective on what role do you see observability in either controlling or acting upon something that agents do.
Would you mind starting us out, Jess? Well, that's the transition directly from the last panel, where they were talking about you've got to go to your internal auditors and say, "This is what's happening. " Answering any questions about these agents starts with knowing what they're doing.
So your observability for AI is what are these agents doing, and how do we track that, and how do we trace it? One thing that's great about observability for AI is that all the new frameworks have embraced open telemetry. That's just taken as a given, which is great.
And people recognize that it needs to be tracing. We need to say, where did this request come from? And how did we get there?
And what was the prior conversation? So step one is what has happened? And then we can start worrying about what are we going to do about it the next time it happens.
Yeah, I think what's interesting, if you go 10 months ago, 11 months ago, I was a big proponent of not very many dashboards and a whole lot more IDE, CLI, sitting in my coding agent and just giving me all the answers in a natural language form. But what's been interesting over the past 10 months, when you don't show a graphical representation of the data, obviously in this case, whether it's a trace or a metric or a log, the human side of us goes, "Do I really believe what this agent is coming back with? " And so for me, while yes, dashboards, I would love to say dashboards are dead, similar to what Lucas said many moons ago about DevOps.
But the idea of observability in this is a requirement. I think the last thing I would say, just on this kickoff question. An agent's going to do what an agent's going to do.
I think the hard part from a developer's perspective is they love logs and not traces. And so getting a developer to transition into mentally go, oh, I could actually also leverage a trace, not just a log, right? That's one of the biggest things I've started to see in the way developers consume.
And thankfully, a trace is, obviously I've always been a believer, but now they're starting to see the value. Yeah, I think dashboards are dead and DevOps is dead. I think executives really love dashboards, so dashboards will still exist somewhere.
But for an engineer, I would say dashboards are, if they're not dead, they've got a foot in the grave. And I would say the same thing about DevOps. It's a little cliché to say that agentic is changing everything, but agentic is changing everything.
And I question how long there will even be people who know and can even remotely understand how the software they're building is deployed and how it runs. And in that case, you could have 1,000 dashboards and it's not going to help you figure out how to actually fix the problem or how to do anything. So my take on the kickoff question, I think is really, you all both captured most of what I wanted to say.
I think the only thing that's missing is figuring out how agentic actions tie to your business' outcomes, right? At the end of the day, there's sales, there's revenue, there's customer satisfaction, there's something that is a north star for you. " And I know we don't want to get into a paperclip optimizer situation, but we do want to make sure that we've, I think if we're going to embrace agentic developments, and we are, I think we need to make sure that it knows, well, first it can find the key metrics, and then second, we tell it these are the things to optimize for.
" No, I don't. I've never written a book. So you were going to have...
Did you want to follow up? Yeah. I wanted to remark that Greg and Sean are talking about two different things when they say dashboards, I think, because Greg is talking about static dashboards that are already there.
And yeah, those are useless because those are only good when you built them and you understand them. And Sean is talking about the set of graphs that your agent builds for you right now to give you evidence of what it's saying. Yeah.
Absolutely. Big difference in the traditional, historical way we deliver dashboards versus me saying make sense of this data that I'm seeing. Drastically different.
The engineers are doing the latter, definitely not the former. Let's dive a little bit deeper. I'd love to hear your perspective on, I know OpenTelemetry's worked a lot on AI and how to support it.
LLMs, maybe not so much. You hear some things coming out of Anthropic. We have observability telemetry.
I'm not quite so sure of that. How far do we have to go? We've lived in a world where observability could pretty much take in any kind of data, obviously in an OpenTelemetry format, to be beneficial.
But traditional software doesn't hallucinate and change log records. We have this evidentiary issue with AI and what agents are really doing. What has to change at the OTel or at the LLM level?
Greg, you want to? I'm going to have you jump on that one. Throw the easy questions to you, buddy.
Yeah. I think one of the biggest things that we're missing is semantic conventions around what do we call the prompt that was involved, and where do we have the tools that were getting that called, and how do we figure out what the agent actually did beyond just trusting the text that was output, right? I think OpenTelemetry is still a little behind in clarifying, this is what you need to collect and this is what you should call it, and then this is how you can aggregate across all of the providers that you're using.
I'd say that's probably the biggest gap. The foundational data is also there. But I do think some of the other scaffolding is not there yet.
Yeah. In full transparency, so I know, oh man, I remember the I was at Wavefront back in 2018 when, or I was at VMware when we bought Wavefront back in 2018. So I've been doing this observability thing, OpenTelemetry/ whatever we called it back then, way too long.
I think what's tough is OpenTelemetry was not originally built to meet the requirements of an AI world. It was not for that. And so there has to be a transition, there has to be a time.
In full transparency, Dynatrace, we did acquire Arise, and Arise has Open Inference. Sure, we're a part of the problem, if you want to call it that. The good news is between Open Inference, it's now been donated to OpenTelemetry, and then obviously with the inclusions, you mentioned Anthropic a couple weeks ago.
They now have full capabilities with OpenTelemetry, whether it's the gen AI conventions, and then ultimately Open Inference now that it is a part of the project. There will be a continued expansion, and hopefully the gaps will be filled. I say hopefully, because at the end of the day, there are competing priorities in OpenTelemetry beyond AI, right?
Whether Real User Monitoring continues to expand. There's just different things that are taking a priority, and so at the end of the day, hey, let's continue to work together to build on it. I think we've taken those steps.
I don't believe you. There's any priorities other than AI. I don't.
Oh, I don't either, but there's a reason we're keeping Open Inference, but it is what it is. Okay. OpenTelemetry has made a lot of progress on the semantic conventions about what should you say to the prompts and what should you call it.
The semantic conventions are there, and they're published. Now what has to catch up is all of the libraries, all of the agent libraries and the harnesses. And in the meantime, we have to use collectors to do translations, and that's there.
Now, I'm not talking about tracing the LLM itself. The model call is a black box, and I have no clue. But the harnesses are deterministic code, and that is our opportunity to really know what these agents are actually doing because the model doesn't have fingers.
The harness is its fingers, and we can track the movement of those fingers deterministically with the OpenTelemetry that is built into all these agent harnesses. Anthropic isn't telling you how their models are working, but they will, if you configure it, if you're running managed agents in their cloud, they will send you OpenTelemetry of every tool call, and if you check the box, almost everything the agent said. Nobody conforms perfectly to the semantic conventions, but those conventions are there.
Yep. And we're getting closer, so yes, we can know what our agent's doing. There's another question of how radioactive is that data, all of the prompts and inputs and outputs.
And I think that's the hard part is you mentioned the black box. I remember the early days of the hyperscalers. We wanted basic metrics out of the hyperscalers.
We wanted basic security capabilities, basic things, and it took time for us to get them. Some of them we still don't have. That's where we're at with OpenTelemetry today.
And I think you said it's not an OpenTelemetry problem. Right. OpenTelemetry is so many things.
Correct. It's the protocol. That's solid.
That's fine. Yep. It's the semantic conventions of what do we call these things.
Yep. That's there, but then it's every single library in every single language, and that's never going to be everywhere. Nope.
And the foundational models have a reason to keep it a black box, and we're going to have to work with what they give us. And so it's just going to take time. I don't know that we need to know what's going on in the- I want to know it, yeah.
I know we want to know it, but I don't think we really want to know. I think I'm fine with that being a black box. I think the point you were making of what you get is what can actually make changes.
You can instrument, and you can figure out what those are doing. And I think there's such a race to adopt AI, as if you don't do it, then the company's going to shut down or something. Everybody's just running to go do it, and they didn't think, "How do we keep track of what this is doing?
" And I think there's just people see the vision and are trying to get the vision without having the building blocks. I think that's the main problem. If you listen to the frontier model makers, they say they don't know how their models make decisions.
I don't know how much of that's true. My impression is that they don't want to know, and they don't want anybody else to know. Well, there's also an account.
It's a black box to them, too. I think that's the key. There is this thing called accountability if you do know.
Now there's not, I don't think. Yeah. There's some truth to that.
I'd love to get into some of the more control around agents, whether it's guardrails, compliance, policy, things like that. What role do you see observability in some of the governance of what operating agents do? Of course, you haven't started one yet, John, so you get to chime in first.
Oh. At least it's not a geopolitical question. What do you think of data centers?
No. I am not touching that one. Where do I start?
I think I said it earlier, I believe whether it's logs, traces, metrics, use OpenTelemetry, et cetera, I believe that is the foundational piece to truly having visibility within not just AI, but across the stack. I think the hard part is we over-rotate on agents. We over-rotate on AI.
In theory, it's just software, and I know some people don't like when I say that, but it is just software. And we should treat it that way. But I think we've created a...
Well, running on hardware. It is what it is. Same premises.
There is hardware in there eventually, yes. And data stores and databases, yeah. But at the end of the day, I think it comes down to how do we continue to build upon the past eight, nine years of how we've gotten here on just the general side of applications and software.
And so when we look at AI, agentic is going to do a thing. I don't think we're going to stop it. I think the best thing to do is come alongside it, and continue to build around it and leverage the principles, leverage the lessons learned that we have had in the past.
I think the interesting thing about observability and AI is not building around it and alongside it, but how do we enable it? Why is cloud native so important for AI? Because AI can't rack servers.
But it can spin anything up in Kubernetes, right? What can we, with observability, in addition to watching our AI, what can we give it to do a better job itself, to do more useful things and to not screw up? Yeah, I think guardrails are absolutely essential for AI, and I think very few people are currently deploying guardrails into code.
You can all Google the slant take of observability if you want to see. But very few people are actually deploying guardrails right now, and there's a lot of reasons. It's expensive, slow.
If you're using frontier models, you're paying more to evaluate than you did to get the response in the first place, right? But people now are asking chatbots every single day. You call a consumer company, you're talking to a chatbot.
At least the first round of people, it's going to be a chatbot, and you need to make sure that your chatbot's not telling people crazy things, and you need to be able to stop those responses from going out, and a lot of people aren't doing that. " And it's down there chugging away. " Nope.
But it's happening. And so, if you were to ask an agent from your medical practice, "Hey, I just got prescribed this drug. What's the dose?
What are the side effects? " You want to make sure those answers are right. And we often, I think, software developers don't take their jobs very seriously, or as seriously as they should.
Some of these things are life and death decisions. Some of these things are highly user impacting, and a lot of engineers that I know have the mindset of, "Oh, we'll just ship another build next week. " And like You need to be able to know that the things you're doing are the correct things.
So, I would argue it's extremely irresponsible to deploy AI without guardrails, and observability is the way you get the guardrails. Let me take the other side of that coin. The people answering the pager, the phone, the whatever it is, the chatbot, when something goes wrong, are the developers and the ops folks, right?
So I could argue that developers care a lot. They just don't have the ability to implement what they'd like to see or necessarily know what they'd like to implement quite yet. You agree or disagree?
Yeah, I agree. I don't think these decisions are made with malice. " And so, it might be compromised.
It might not be as good as it could be. And because it is so easy to patch and to release new builds that it's not the '80s where you were shipping a ROM cartridge that got burned at the factory, right? You can ship out a new build in a few minutes.
And so as we've made this push for the longest time for continuous integration, continuous delivery, then of course quality is not going to be as important for each individual atomic build. Well, and quality's always been impossible to define, but now it's really- Sure ... hard to define.
Yeah. Oh, yeah. Because you can't look at your dashboards and see error, success, error, success.
No. Everything is in text, and it's all interpreted, and it's non-deterministic, and it can do things you never expected in ways much more than weird network failures. It's always network.
Always DNS. DNS. But yeah.
We used to have DNS. Now we have agents, and how can we even determine good versus bad? I can't write an SLO on, was the agent wrong?
Should it have done this? Well, you kind of can, right? If you evaluate all your responses.
Right. That's eval. So you've got agents coming back.
Right. Then you're asking another agent to evaluate the other agent. So that's still not deterministic, right?
Right. Defining good is incredibly hard now. And the thing is, our agents can make something true if we can tell it how to test that, right?
But is my chatbot responding correctly to customers is actually really hard. The hard part there is saying what is right. And yes, it starts with observability, with what happened, but then it comes back to, and what was good and what was bad, and how can we generalize that, and how can we automate the eval?
There's so much going on here that we have not figured out how to do smoothly yet. Yeah, I think the interesting thing is, if we go back 10 years, hey, there's a problem. If you give three engineers the same problem, guess what?
They are all going to come up with a different solution. And in most cases, it's probably five different solutions out of three different people. And with an agent or within the agentic world, if you feed the same observability data to a DevOps agent that you feed to an SRE agent, you know what you're going to get?
Different answers to the same problem. So what we have to do is, and I think this is the beauty of observability, and you could have evals in there, et cetera, is we now have to decide, or the system has to decide which agent is the best answer. How does it know which is the best answer?
It almost feels like an infinite loop. But what happens is, I think this is where you let, maybe, depending on how far you want to go with this, maybe we let the system decide which is the best decision with certain guardrails. Where the system is, of course, an agent.
Yes. Agents running agents, right? But this is what we've created, right?
Yes, way to go, humans. Cheers to us. " Or if it is Y problem, yeah, sure, that needs to go to the DevOps agent, right?
So we have to provide that intelligence to the system, but at the root of that is observability. It's the data that we can provide. And now we're talking about observability with AI.
Absolutely. Of how does our observability power our agents that are doing things in our system? Yep.
All righty. We have time for maybe a question or two. While you're raising your hands there, one thing I've discovered, there is no model that's been created yet that can answer a binary question without writing three paragraphs.
No, it's Geoff. It's Geoff. Geoff, no.
The system two stuff. There you go. Oh.
Easy for me, because I disagree. You can't talk about AI in a Christmas month without talking about Geoff. All right, who's got a question for- Trust it ...
our panel up here? There's this young guy, John Wiley Willis. What's your name?
No, I just think there's still an industry confusion about what observability means when it comes to AI, right? And I don't, not to criticize, but I don't hear the clarity even with you guys. Dynatrace, right?
You guys should know, right? " And you created this crazy application where customers didn't understand the difference between your sort of telemetry observability of how an LLM or how a model's working and how you're doing evals. So I still think we have that problem in that we're not clear about what it shouldn't actually be called observability.
Evaluation shouldn't be called observability. Comments? I disagree.
I think evaluations are observability. Yeah, and Swank, Clark, Galileo, we have sort of that same issue, right? But I think evals are definitely observability.
It's the only way to try to use classical observability principles in the agentic world. If you don't have evals, you don't have anything, right? So.
And yet if you only have evals, do you have observability? No. Swank.
Right? Well, I think you actually bring up a great point. The idea that logs, traces, metrics is observability, I would argue is actually not correct.
Yeah. Right? I think the good thing, or at least the beauty of what is happening with the maturation in and around telemetry and observability and beyond, is we are maturing based upon what is given to us at that time.
And if adding the context of an LLM or from an evaluation perspective is an additional piece, hey, okay, well, what about the cloud context or the neo cloud context? I know I'm not going to continue down the context path, but that is the reality. We have to continue to adapt.
And it was built on, arguably, a really, really good foundation of observability, and we just continue to build on it. All right. How about a big applause, a big round of applause for our panelists.
Jess, Greg, Sean, thank you so much. Sean, thanks for standing in here last minute. I appreciate you jumping on.
All right.