Agent Observability Changes the Way Developers Work

Hey, everyone. We're back here at our coverage of Splunk Conf '26 here in Denver, Colorado. It's been a great couple of days.

Had so much chance to talk to some great people. I hope you've seen our previous interviews, but more than what you're seeing on the interviews, we do our best to capture a moment, but you got to be here to sort of feel the energy, to see the vibes or feel the vibes. I'm not talking about vibe coding necessarily, but to feel the vibes, and really, like we see at so many other events recently, the realization that we're living through almost a historical time.

Mm-hmm. A border zone between what came before and what's coming next, and how agentic AI now, but AI in general, is really just changing how we're doing things. Yeah.

My next guest is Greg Leffler. Greg, well, I call them DevRel, dev relations, but here at Splunk, they're dev evangelists. Right.

And Greg heads that up here at Splunk, and we're happy to have him. Greg, welcome to Techstrong TV. Thank you.

I'm thrilled to be here. So, I've got a lot of friends- ... who do DevRel and stuff like that, and I have a lot of friends who want to do it.

Mm-hmm. How do I get into that? Give people a little bit of your background, your story.

How did you come here? Sure. So I started my tech career at eBay in the NOC.

Oh. So, the 24/7, wake up in the middle of the night, fix it. Mm-hmm.

The meat grinder. Yeah, exactly. And I did that for a couple of months, and then became a sys admin there.

I moved to LinkedIn as an SRE, sort of attached to LinkedIn and grew in my career very quickly as LinkedIn grew as well. Uh-huh. And then I think the real pivot moment was towards the end of my time at LinkedIn, I was supporting the editorial team, like who writes LinkedIn news and does all that stuff.

And I was supporting them as an SRE, like I was keeping their applications running, and you can't be a good SRE unless you use the app and you know what the app is doing. So I wrote some posts, and they got a lot of traction, and people seemed to really like it. And so I did an assignment as an editor in the LinkedIn news team for a year.

And that sort of helped give me a portfolio and experience creating content, and then I came to Splunk and started doing evangelism work. They were looking for somebody that knew what engineers wanted to hear, and they thought that was me, and I agreed. So here we are.

How long ago was that? That was about five and a half years. Really?

So I've been here since March of 2021. Very cool. Yeah.

Excellent. Let's talk about how things have changed. I mentioned we're here, it's an exhilarating- Yeah ...

time to be doing stuff. From where you sit and what you see as you look out, how have things changed in these last five years, five and a half years? Oh, gosh.

It's almost unrecognizable. I think either operating or writing applications is totally different from what it was before. Sure.

And that's just going to keep happening. It's going to keep becoming more unrecognizable, I think. But it's a positive change, I think in both the observability realm and the coding realm, I think it's a good change.

Sure. And I think people are expected to know more and do more, is I would say the biggest change. I want to come back to that because that's really important.

I had a conversation, we had Corey Midden- Oh, yeah ... this morning and we were talking, I don't even know if that ground up on camera or not, but we had a conversation about that, and I want to come back to it, Greg. But I wanted to talk about, you're right.

It's truly amazing. Something that was so revered- Mm-hmm ... as coding- Mm-hmm ...

software coding. Right? The alpha predators of the technology game- Yeah ...

right, were the coders. How quickly AI has kind of gobbled up that territory and made coders software engineers now, right? Yeah.

They don't necessarily code, but they still are engineering software. It's amazing. Writing, as you mentioned, you do blog posts.

I write a lot, obviously. Right. Again, so quickly.

I don't use it to just write without me. I write it. Right.

But how much help it's done, and I was talking to our cameraman, Andrew, who is behind the camera. For me, this is happening around November, December. Mm-hmm.

The releases that came out then. Oh, right. Right?

8. Oh, right. Not 5.

And then the OpenAI- Mm-hmm ... Sol and before Sol. Right.

All of a sudden it got real. But interestingly, I was talking to Andrew this morning, now with the advent of Fable and of Astra, the same revolutionary kind of like, my God, it's changed. It's happening in video and video editing as well.

Right. And I think this is what we're going to see, this disruption, this amazing kind of capability available to all of us as it continues. I want to return, though.

" Yeah. But no, that's not true. No, yeah.

Your job's not going anywhere. In fact, you're probably going to have to work harder. Yeah.

Right? Because 10X-ing someone doesn't come without more work. Exactly.

Just because the AI does a lot more for you, you still have to manage that. Mm-hmm. Do it.

Talk about that. And not just for an evangelist, because most of the people watching this out there are more practitioners. Right, of course.

How does that affect them? What should they be on the lookout for? Well, I think the biggest thing to think of is the job is managing now, right?

Mm-hmm. Even if you aren't a manager by your title, you're managing AI platforms, you're managing agents, and you have to plan and strategize more than you had to before. When I would solve problems, I tended to be the type that would just open up an IDE and just start trying to bang out a solution, right?

Mm-hmm. It's like, I'm not going to sit around and plan and think about, it's like, this isn't sounding right, I get back up, so I'm just going to fire something out that'll fix this. And that works with AI models, but it's not the right approach, right?

You need to know how to write the prompts. You need to know what to say. You need to know how to tell it to make the right decisions faster, rather than waiting around for it to run and iterate and you have to be more precise in what you're asking.

You have to learn how to do that, and it's a skill that you have to pick up. Absolutely. I agree with that.

And I think the point that you were making of everybody's going to 10X themselves, that is going to create a lot more administrative overhead, right? There's a lot more status reporting and keeping track of what all of those things are doing and coordinating those things with each other. And so the work, I agree with you, I think the jobs aren't going to go away, but they're going to be completely different, right?

You're not going to be writing a ton of code yourself anymore, which I honestly think hasn't been the fun part of the job for a long time, right? Mm-hmm. And it's not the part where people add a lot of value.

People add value in their thoughts and their planning, and they're architecting the solutions. And so, I think that's where people are going to go, even if architecture is not your job. That's where you're going to be able to differentiate yourself is like, I figured out how to run this at scale and performance, and it was observable, and I was able to keep track of what it was doing.

That's what I see people's jobs going, especially for practitioners. I don't disagree with you at all. I want to talk a little bit about observability.

Mm-hmm. Right? Because, look, in five years you've seen a big change, right?

Yeah. Otel is observability for a lot of people, right? That's become the collector- Mm ...

if you will. And then companies distinguish themselves by what they do after Otel- Right ... collects the data.

What is this whole AI thing? And especially with Splunk, right? Splunk's probably the leading observability player in the market.

How has that game changed, not just over five years, but even in the last couple of months? Yeah, the pace has really accelerated over the past few months. And one of the things that we did at Splunk was roll out a product called Observability Studio- Yeah ...

which integrates with your IDE as you're writing your applications and helps you say, let's say you have an existing code base. There's a skill that we ship that lets you audit that code base and say, do you have the correct Otel instrumentation? Are all your function calls instrumented?

Do you have your business logic in there so that you can say, like, this is the start of a checkout transaction, and this is something we need to pay attention to. And then if you're not using an existing code base, like if you're writing a new app, there's a skill that will instrument it for you, right? And so you don't need to think about it as an engineer anymore of like- You don't even really need to know OTel exists, which is a pro and a con, but you don't really even need to know that it's there.

" And so the way we do that is we ask our AI agent, "Hey, make this observable," and Splunk created skills that will do that for you, and they do it with OpenTelemetry. So you can use Observability Studio and then still observe with somebody else. That's one of the pluses of OTel, right?

But it's all integrated into the Splunk Observability Cloud. It's integrated into your IDE. It's all there, and you don't have to think about it, right?

And I think we've asked people to think about it for five years, longer than five years, right? " And people are like, "Yeah, sure," but it doesn't happen, right? No, it doesn't.

I suppose they say yeah. Right. It doesn't happen.

And we've tried to add more things that make instrumentation easier, like the OTel injector. So we're like, "We can't get you to instrument your app, so we're going to instrument it for you. " And now we're like, "Well, that's not perfect," right?

And so having the manual instrumentation done by AI is better than the injector. So now that's what we're shipping. So there's been a lot of change over time.

I think the Observability Studio is a big one, but then OTel becoming the standard is the other one, right? Yeah. Because that has standardized across the entire industry.

Yeah. I tell you another big change I see with observability, and that is observability of agents. Mm.

Right? How do we govern these agents? How do we manage these agents?

Well, it's observability. Yeah. In another costume maybe- ...

but it's still observability. Yeah. And I had the pleasure of speaking to the young man, Vivek Chadej.

Yeah, from the Galileo acquisition. Yeah, from Galileo. And we talked a little bit about an announcement here around tokenomics- Mm ...

and stuff like this, which really goes towards managing your agents using observability. Right. This is yet another new facet of observability, if you will.

And well needed, I might add, right? Yeah. This is something that was really needed.

How does that change things? Well, it's one of the coolest things about this job, not just evangelism, but this job in general, this industry, is that it does change so frequently, right? Yeah.

And there's always something new, and that's what keeps me here is that there's novelty and there's always something new to do. But this is one of the coolest new things in a long time, right? Yeah.

Because you can see this is going to change how applications are written, how they're run, how they're monitored. And people ran to adopt AI because there's a ton of benefits. There's a ton of things that it's better at, but nobody was thinking about, "Hey, how do we make sure that these responses we're sending to our customers are accurate, or that they comply with our policies," and that sort of thing.

" Right. If it's wrong, we're going to fix it. We're going to stop it from going out.

And being able to do that really helps make Splunk and our customers able to deploy AI with more confidence, right? They can say, we can put this in front of our customers and not worry about it blowing our budget or saying something that's going to be embarrassing or whatever. So the agent observability part is pretty exciting because you can see it in real time.

You're going to be able to know that you can still use this indeterminate system, which is very difficult to trust, right? If you have an executive assistant and every day you tell them to bring you coffee, and one morning you get tea, one morning you get water- I have an AI executive assistant, so I know. So being able to control what they do is really important.

And the other part of the AI coin is the tokenomics angle- Right ... which is how much are we spending? And for a while, who cares?

It was like, we need to get AI out there. It doesn't matter how much it costs, let's spend the money. And very quickly people are like, "Oh, wait a minute, I don't want to write a $7 million check for this," right?

So being able to know your consumption is the first step to being able to manage your consumption, right? Right. Can't manage what you don't know.

Exactly. And I think if we give developers the feedback as they're building stuff of, hey, this is how much it's costing to do that, they'll do the right thing, and they'll say, "Oh, maybe I can find a different way to do this. Maybe I can write it in a different way or have it use a different data store," or do something like that.

So then the knowledge is power, and in observability, this is just another element, and we've been trying to do it for a long time with FinOps and cost estimation and that sort of thing, and nobody really cared. And none of the developers, at least that I know, really cared. But I do think with tokenomics, people care more because there's sort of a competitive angle to it almost, where people I'm talking to are like- It's the mirror image of the token maxing.

Yeah, exactly. I want to be on the leaderboard. It's token mining, right?

Now I want to be on the other side. Now I want to be on the other end of the leaderboard. That's what it comes down to.

Last subject I want to throw out at you. I had another fascinating conversation with Sunil yesterday. Mm.

You had from Cisco- Right ... Splunk and observability security, all of it. And we talked about this concept of something he calls recursive security.

Okay. So in other words, we all hear about recursive AI- Right ... and runaway AI and how it becomes self-improving.

Well, we could do this with security. Mm-hmm. Right?

We've talked about self-healing networks for years- Right ... and all of this, but now the observability becomes sort of the back end of the flywheel. Mm-hmm.

So the security is out there. We observe it, we see what's working, what's not working, what needs to be done, what should be done, what hasn't been done. It gets through agentics, that observability, the learning of it- Mm-hmm ...

the message of it, gets put into the next iteration of the security product and observed again. Right. And it really becomes a recursive- Yeah ...

flywheel. Yeah, exactly. And wow.

Now that to me is mind-blowing, right? That's game-changing kind of stuff, and it's the kind of thing we're going to need if we have recursive AI. Yeah.

Right? We're going to need recursive security for governance and guardrails and everything. Absolutely.

You think it's still pie in the sky? It's next week, next month? Or years out?

It's definitely not years out. I don't know that it's next week or next month. But it's definitely sooner than years.

I think Splunk and Cisco are pretty, I don't want to say uniquely positioned, but pretty close to uniquely positioned to have access to all of the data that goes into keeping a system secure, right? It isn't just the applications. Sure.

It isn't just the network. You do need the infrastructure. You need the end user networks.

You need the campus stuff at the offices where people work. All of that stuff goes into the security of an overall system and the reliability of an overall system. And we've got all that data.

We've got the vision, we've got the platform through Cisco Data Fabric, through Cloud Control. We have all of that stuff, and I think obviously we're running to build all the integrations and to assemble the airplane while we're in the air. But I think it's coming very quickly, and I think it's something that is going to be how people operate going forward.

You're not going to log into 17 different dashboards and look at these different things. You're going to have one place, and with us at Cisco Cloud Control, and all your data's going to be there, and it's going to be easy. But for the agents that are building your applications and are deploying your applications and are troubleshooting them, they also have all that data, right?

And so they can start to build and operate for you and keep that flywheel going. So how quickly will that be? I would say, let's say a year- Okay ...

before you start to see that. I agree. Yeah, I think it'll be about a year.

I think it's something to definitely do aim for. Yes. And I think realistically, these systems are going to get too complex for a person to understand and be able to troubleshoot them.

They're already there, but that's why we have all these observability tools. But I think as we're trying to fix them, figuring out where the problem is, what caused it, how do you troubleshoot, that could take a person hours to figure out, and we're going to have to start relying on AI as we're relying on it to develop the application. So it needs this data.

It needs to be told, iterate on this, and put it in your memory. And so we have smart people that are figuring that out. So I think it could be years.

But I do think it's coming. Very cool. Hey, Greg, I want to thank you for coming off here on Techstrong TV today.

We appreciate it. Yeah. Keep up the great work.

Thank you. Come back and keep us posted. Awesome.

It was great to be here. I really enjoyed it. Pleasure.

Thank you very much. Hey, we're here at Splunk Conf. Check out some of our other coverage, some of our articles too.

Mike Bizard was here. He wrote a bunch. I've got some stuff coming out.

But for now, this is Alan Schimmel for Techstrong TV. We're out.