Code Monitoring in the AI Era with Sentry’s Milin Desai

This is Textron tv. Hey guys, thanks to the throw. We're here with Mil Dee, who's CEO for Century, and we're talking about the growing need to monitor application code in an age of artificial intelligence.

And more customers seem to be figuring all this out. Milan, welcome to the show. Great to be here.

You've been banging this drum for a while now, and so, uh, are you starting to see folks kinda understand the need for it and, and have a greater appreciation? 'cause I think, you know, we've been monitoring metrics for a long time, but I don't think it's always been at the code level and are things changing? Um, absolutely.

You know, we are seeing, uh, a few patterns emerge. You know, everybody is agreeing that we need to monitor infrastructure and systems, right? Whether it's your Kubernetes cluster, your cloud, uh, and other kind of system level stuff.

So that's kinda a well-known paradigm logging metrics and other elements. And so I think that that train, uh, is, is moving. It's, uh, you know, going through a consolidation phase.

Life has good over there. What has happened over the past, I would say, um, five to seven years, is we are writing more, um, you know, applications than ever. Uh, this code is getting shipped multiple times a day now with ai.

Uh, that is changing even more. And so everything around us is becoming software. Software is changing faster than anything else.

And so what we are starting to see is this need to help developers who write the software understand how it's doing in production from a user standpoint. And so absolutely seeing, you know, as you said, we've been talking about this sort bit, um, we are seeing this momentum, uh, and you can see it in some of our numbers. You know, we passed a hundred thousand cloud orgs, a hundred million in a RR.

And so we are starting to see the momentum of monitoring your code and done by developers we shipping constantly from a user perspective is what is a user experiencing as they're using the software. And so we are definitely seeing this kind of transition, uh, and uh, it's really kind of benefiting our story, of course, Is to your point, there is not only more code now than ever, but it seems like if I talk to folks, there's gonna be more software written in the next two years than maybe in the last two decades. So, um, it is just putting pressure on the system.

And, and how do we kind of absorb that? Oh, absolutely. It means, you know, even if you look at more, uh, you know, everyone is building a software interface so that people can kinda use their value play, right?

So, uh, everything we do now is a software interaction. And so it's no longer just the tech companies, uh, who are building or building at speed. It's everyone who wants to stay relevant is starting to kind of not only build software through a mobile app or a web interface, but also update it on a much more regular basis.

And that's the big shift, which is the build times, the release times are now in, you know, multiple times a day versus, you know, oh, once a month. And so you had more time to maybe quality control it, uh, you had more time to kinda, you know, run processes around it. Now you're shipping every day.

How do you maintain that great user experience to react so that you are not on Twitter? Somebody calling you out and saying your software is not doing what it's supposed to do. Your experience sucks.

And it happens a lot with all of us, right? We complain, uh, an X or Twitter now or some other medium is where folks go. Uh, and so interesting story for you.

Recently we just closed a large opportunity and the CTO told me, you know, we have users complaining, uh, online and we need to fix this. Our mobile app is having issues. It's a very specific thing happening on iOS.

And they, you know, we turned it on, we turned century on. They were able to diagnose this issue to those users very quickly, um, and fix it. And so absolutely seeing this trend, and it's happening because of the velocity of innovation, AI led coding is only going to accelerate this further.

Uh, and it's all about users and happier users and happy experiences. So, uh, definitely seeing that trend and that's what's kind of shifting the mindset from observing or collecting tons of data and charting and all of that stuff to debugging when something's broken, can I get to the issue? Can I get to the heart of the issue, understand what it is, fix it, and move on.

And so debugging systems or debugging code, uh, is going to take over more importance than observing a system. Um, is, is our opinion kind of going forward? The code being generated by the machines is, shall we say, of uneven quality.

Um, a lot of it was developed by LLMs that were trained using examples of code pulled from the internet, created by humans that are often as flawed as the code as they create. Um, do we need to therefore find a way to kind of take the code that we do like and are monitoring and find and expose that to the LLMs in a way that extends them to create maybe better code? Uh, you know, I think that's where we will see kind of the evolution of LLMs to become, uh, I'm gonna use the word domain specific a little bit.

Uh, I think you will be able to leverage them in the context of your, uh, uh, you know, your code, your information system, right? Uh, so you'll leverage of course, the open source communities out there and everything else, but a lot of times it's the contextualization to what you are doing in your systems, right? And I think that's where that contextualization is gonna come into play.

And an example of this is what we are doing internally with Century. Um, we tell you when something is slow or broken, let's just take an error for that matter right now. And what we are starting to do is use LLMs to help you resolve that error, but in a contextual manner, right?

It's not, here's a generic kind of element, it's to solve very pointed problems. And so we have an AI agent that assists you through that process, but it's highly contextualized versus like a generic wrapper or helper. So I do believe, uh, it'll get better, it'll have more domain context, you'll be able to kind of build context to your own code systems and it will improve.

And then AI agents, like the ones from Sentry and others will, will kind of further iterate to help you solve quickly, right? In the context of what you are operating in your environment, in your workflow. So I think it'll become more curated workflows driven by contextual data.

Um, and so I'm very hopeful that, uh, companies will build on top of what I'm calling LLM platforms to, to build this contextualization privacy generalization, uh, in there. How will that play out on a practical level? 'cause I could see how as a developer I might wind up with, I don't know, four or five agents optimized for a specific thing, and are they all gonna talk to some sort of master agent that acts as the head butler for the developer and organizes the behavior of all the other people working in the household?

Uh, you know, how, how do we get our arms around that? This is a great question, and I know that there are, um, startups and larger companies alike trying to figure out kinda if you have multiple agents, how do you communicate from agent one to agent two? Because it's no longer just going to be a human to agent interaction.

There will be, uh, you know, multiple agents collaborating and then, you know, talking to a human, then there's another interaction. So it is going to require kinda, uh, I'm just calling it a human to machine, to machine to human kind of model. Um, and so I think what's gonna happen is these workflows as I'm calling it.

So if you think of a developer, they're working in their id, their assistants starting to enable them to, to kind of say, uh, you know, kind of give them a piece of code, right? Saying, you know, this is kind of exemplar in that. Um, and so that's kinda one element, but imagine that agent being fed by others, uh, other agents, uh, curating, uh, it it's contextualized to your code base as an example.

One of them is actually making sure it's not violating, uh, your, uh, uh, you know, open source usage policies or your usage policies in general. So I do believe, uh, at the end of the day, those workflows will have to be agent to agent and either a company or a collection of solutions will solve it. Um, I don't have like a perfect playbook, but I do believe some of us are already starting to think about it, um, and, and kind of enable that, right?

So it's not going to be a singular agent, it's gonna be a collection of things and machine to machine are doing different attributes of it. So We also hear a lot these days about all things observability. How will code monitoring kind of flow into that larger observability motion that we're seeing out there?

And what is the relationship? So, uh, you know, we, we started off this conversation. I said, you know, you have to observe your, your systems, uh, and so you have to collect logs and metrics and all of the kind of infrastructure related items.

That's a known thing now. Um, it's scaling, it's growing, it's getting better. Some of it is getting the security DevSecOps, as we call it, all of that CICD, uh, stays.

That's great. Uh, that's a layer of observing kind of what I'm calling infrastructure, the application that sits on top of it. Uh, it could be running on a mobile phone, it could be a web app talking to a, you know, uh, a backend, a mobile, talking to a backend.

So the app in itself is another layer. Uh, and the persona that it is, is the developer. Uh, and so that is another kind of construct, which is moving faster than the the infrastructure construct.

And so these two systems have to play together, and they play together in the sense of common alerting systems, common incident management systems. Uh, and, and you know, there are different delegations in this, right? Uh, a core problem is solved and directly delegated to a software team.

A simple example of this is with Sentry, we integrate into your code owner file. And when a piece of code breaks instead of a SRE alert, who then decides where to kind of triage it, the triage goes directly to that software team that wrote the piece of code, right? So triaging is automated, uh, in that workflow.

The alert is generic, so it'll go to the SRE team, it'll go to different folks, but it's a two-pronged kind of element. And so, observing systems and building charts and dashboards and alerting important debugging and reacting to things in real time to solve problems that customers are facing with all the context to solve within minutes is a whole different kind of mindset, uh, to observability. I'm calling it debug ability, uh, going forward, right?

The ability to debug systems, the ability to triage and, and move quickly, versus the ability to collect data and charts and analyze over time two different kind of problem sets. We are moving more towards react, solve, and move ahead, right? And so we believe in the code world, that's what you need, uh, and it'll, it'll stay in sync with the system world, uh, very well.

And that's what most of our customers end up doing today. How do you Think the relationship between developers and the DevOps slash SRE slash platform engineering team should evolve? Because, um, sometimes, you know, we talked about shift left for several years now, and developers will tell you, you know, you're shifting way too much left.

Um, so, you know, how do we strike a balance? So, uh, you know, I think as we started the movement to shift left, we went from, you know, you know, kind of observability sub logs and stuff to then saying, uh, secure pipelines. I think that's a platform engineering feature.

Uh, you know, if I am using open source binaries, are they the SBO m around it? All of the work around it? That's a platform engineering feature, right?

CICD is a common service that we use, that's all DevOps. So ops, moving to devs, shifting left is around CI ICD security pipelines. All of that is starting to kind of come into be one kind of platform engineering element and what we bring to the table, which is a service to the developer saying when code breaks, having all the context in it, right?

Um, things like that will absolutely set to that persona, which is developer. So we are part of the platform engineering story. I would kind of bifurcate it.

And the core platform engineering needs to do things like CICD, DevSecOps, you know, monitoring kind of the infrastructure that goes along and then offer developer centric services, um, uh, in that, right? A developer doesn't care if the thing is running on Kubernetes or not. Can I ship my code?

Can I ship it fast? Uh, and so simplifying that experience, simplifying the ability for them to debug, uh, letting them know what they need to solve, versus you now need to become a Kubernetes expert. They don't want to be a Kubernetes expert.

I remember that conversation. That's what brought me to Century. Uh, my devs at VMware were like, I don't care about that stuff.

I just wanna ship code with high quality and just move fast. Can somebody else take care of that? Right?

So I think we will. Uh, you know, I, I see a world where the DevOps platform is truly kind of bringing a, a multitude of these services together, making the consumption of it easy. But shipping code owner ownership of the code, ownership of the quality of that ownership of the user experience will continue to be in the developer land.

Uh, because that's what you know they want to do, right? Which is, I ship a piece of software, how is it doing? Right?

So things like even analytics amplitude, which, uh, gives you product analytics, it's developers and product managers are looking at it, right? Um, which is very different than looking at how my Kubernetes cluster is done. So that's kind of the demarcation we are starting to see, and I continue to see that as a great collaborative experience, uh, going forward.

So we talk a lot lately about developer productivity. It's almost become an obsession, but what are we measuring? What defines developer productivity?

'cause I think a lot of folks are, you know, they're all over the map on it. They're kind of like measuring commits, and some people still talk about lines of code, and yet developers, this is not a factory. They're waiting for that moment of inspiration, like any creative soul.

So how do we kinda enhance developer productivity without necessarily making the whole thing feel more onerous than it is? Yeah. You know, I think part of all this counting commits and stuff is more about just like trust in the system about like, are these folks kinda engaged, right?

That's like, if you have to think of it like an engagement metric, like, you know, our people kind of, you know, engage in the company and doing their thing versus productivity because, you know, lines of code does not correlate to success necessarily, uh, in that, um, you know, the simplest way we've been thinking about it is, uh, you know, an outcome. What outcome are we driving for that software team? You're gonna ship this thing and is it going to drive an adoption outcome?

And like driving folks to kind of think in that manner is one aspect of it. So then, you know, whether they wrote a hundred lines of code or, you know, and so then there's a time element to it, which is can be get to market faster in that regard. So we look at productivity as you know, have you driven to an outcome number one?

The second aspect of it is reacting to what I'm calling customer issues. And, and I I say this in kind of the world we live in. Uh, it's getting really hard.

We used to get a support ticket or two as a developer before. Um, but if you're a developer at Sentry, and you will see some of, uh, folks at Sentry, uh, share this or complain about it, maybe, uh, uh, you know, they're not to me, but we say, Hey, you know, we have users on Twitter and say, this is not working and a developer has to react to it or on a discord channel now, right? So there's a lot of these active points of feedback and your ability to react to it, solve it, um, is another point of measurement because that's a business outcome measurement, which is have we solved for it?

So it's not an individual developer's productivity as much as are we solving for the business the outcome, and then to these things that we deem important and our reaction time or our ability to do it. So that's how I start thinking about it. And then there's layers to this, which is, you know, uh, you know, how how are we doing in terms of from the time you commit code to it releases.

So things like CICD efficiency is an area we actually, so we have a dev productivity team that says, are we holding our developers back from shipping faster because the systems in place are slow, right? Uh, you know, and so we do find that sometimes the systems are causing a lot of friction, uh, whether it's we are run, we are running tests on everything versus selective, uh, test selection on a small change that took place, right? We will run the entire suite because that's what we know to do.

So are there ways we can actually run only the relevant things, uh, in the CI pipeline to do that? So there's a lot of opportunity to streamline that aspect of productivity, um, uh, in that. And so we measure in terms of reaction time to customer reaction, time to innovation, uh, in that.

And then all the other measurements that we talked about are all bad measurements. I call them engagement metrics more than anything. Is somebody engaged, uh, in their, um, uh, role in the company, right?

Uh, it's not a measure of quality, it's not a measure of productivity. Uh, in that, hopefully that answers kinda, uh, gives you a perspective of how I think of the problem, how we think of the problem. All right, folks, you heard it here.

Every developer I know wants to do better. They like to code better. They take pride in their work, but publicly shaming them on the internet is probably not the most constructive way to go about doing that.

Hey, Millen, thanks for being on the show. Thank you for having me. All right.

And back to you guys in the studio.