AI Agents Prove System Reliability Before Outages Hit
System reliability has always depended on finding weaknesses before customers do. AI is now changing how that work gets done. Teams can test, fix and verify faster than ever.
Gremlin Co-founder and CEO Kolton Andrus examines that shift with Alan Shimel at KubeCon + CloudNativeCon in Salt Lake City.
From Expert Tool to Everyday Practice
Andrus worked as an engineer at Amazon and Netflix, keeping large systems running. Those teams pioneered chaos engineering. Gremlin was founded to bring those lessons to everyone.
The first product was an expert-level SRE tool. It delivered value, but users needed deep knowledge. Over 10 years, Gremlin added test suites and reliability scores to make good practice easier.
Find, Fix and Prove
Gremlin’s new Foresight AI takes the next step. It analyzes a system, explains what could go wrong and recommends tests. Then it runs them.
It reviews the results, works out what failed and proposes a fix as a diff or pull request. Andrus says the most important part comes next. Foresight runs the test again to prove the fix worked.
That step separates a real fix from a patch on a symptom. Teams end up with evidence, not assumptions. They know the system can handle that failure.
Built for Kubernetes
Many Gremlin customers now run on Kubernetes. Because Gremlin installs as a container, Foresight can query the cluster in real time. It can suggest changes to Helm charts or Flux configurations.
Andrus sees Kubernetes as an accelerator for AI applications. Teams benefit from years of shared learning instead of reinventing the wheel.
Local or Frontier Models
Alan notes the pendulum swinging back toward on-premises AI. Andrus agrees that data sovereignty matters to many customers.
Foresight can use a frontier model through an API or run against a local model. Customers can point it at their own model and still use Gremlin’s failure analysis. Meeting customers where they are is key to system reliability at scale.
Transcript
Hey, everyone. We're here live at KubeCon, back for more. We'll be doing this all week, so stay tuned on your Techstrong TV channel.
Let me introduce you. Well, if you watch Techstrong TV, you know this guy. His name's Colton Andrus.
Colton is the CEO founder of Gremlin. He's on all the time with us, one of my favorite people to talk to. Colton, how are you?
Doing well. Doing well. Having fun.
Excellent. Hey, we got people walking by. We'll grab a quick hello to Stu Miniman.
How are you, Stu? I'm doing great. How you doing?
Just introducing my friend, Colton. Okay. Take care.
Nice to meet you, Stu. Nice to meet you. Have a good show.
Thank you, man. This is what happens when you're live on KubeCon. Yeah.
You never know what's going to happen. There you go. Colton, for people who aren't familiar with Gremlin, though, and you, give them the Gremlin story.
Yeah. So focused on reliability. I had the opportunity to be an engineer at Amazon and Netflix, focused on making sure the systems were up and working correctly.
We built some great tooling. We pioneered some approaches around chaos engineering, and we founded Gremlin to bring that to the masses so everyone could take advantage of what we'd learned, and we've learned a lot in the 10 years we've been at Gremlin. We started out with an expert level SRE tool, and it was great, but you had to know what you were doing to really get value out of it.
And we built in a lot of things to make it easy to do the right thing, test suites, reliability scores, things to show your boss, things to make sure people could really get credit for the good work they were doing. Yeah. And now we just launched our AI Foresight AI, our new product.
Well, that's what I was going to say. You were on with me maybe three weeks ago. Yeah.
And we talked about this. Yeah. And so Foresight's the how do we leverage this AI goodness to make it easy for everybody to take advantage?
Yep. And it streamlines the whole process. It comes in, it analyzes, it tells you what could go wrong, it recommends a set of tests, it'll run those tests for you.
It looks at the results of those tests, it figures out what went wrong, and it tells you how to fix it, and it gives you a fix in a diff or a pull request. And then, and I think this is the most important part is, and you don't get this with a lot of other tools is, and then we go back and run it again, and we really prove that we fixed it. We weren't just fixing a symptom or a side effect.
We prove that the system can really handle that failure, and we have the evidence as a result of that. Excellent, man. I love it.
Now, we're here at KubeCon, though. " Well, KubeCon's just a good place to be. We run on Kubernetes.
A lot of our customers have run on Kubernetes more this year and last year than in five years ago. Yes. And it's one of the things we're always pushing the edge of our product on.
One of the first things we built with Foresight was going into Kubernetes and understanding what's happening and using that to go come up with diffs for your Helm Chart or Flux or however you're deploying it. And what's cool is because Gremlin's installed as a Kubernetes container, we have access to what's happening in Kubernetes. And so when you ask Foresight about your system, it's doing real-time querying into the Kubernetes cluster to see how it's behaving and what the current state is.
So you're getting up-to-date information about what it looks like and what the risks are and what the potential issues could be. Got it. I got it.
We did this breakfast thing yesterday with a couple of panels. The panel I did was the cloud native stack is the AI stack. Because what we're finding is a lot of these AI enabled apps, I would imagine even Foresight itself, are running on Kube.
Yep. So when you peel that onion back, you get Kube in the middle. Yeah.
And that's not a bad thing, right? That's an accelerator because if you didn't have that whole Kube stack, and you had to build this without that, it would be chaos out there. No pun intended with Gremlin, right?
But it would be. Everybody would be trying to reinvent the wheel here that was already 15 years in the making. Yeah.
Well, it's the difference of running on a data center versus running on the cloud. Running on the cloud but running with Kubernetes, there's a bunch of things just taken care of for you. Yep.
You can benefit from all the learnings other people have had, and allows you just to move quicker and have a more reliable base to build upon. Excellent. Colton, I want to ask you something else because I respect your opinion.
Another thing I'm hearing from people is that as part of this whole AI movement, building models, harnesses, OpenWake, Frontier, all of this stuff, data, is I want to run stuff locally. Mm-hmm. That pendulum swings back and forth.
And now it's swinging to, hey, I want to run into on my on-prem. I want to run into my server closet. We can take all the coats out of the closet and make a server closet again.
We had a server closet at Techstrong, we call it the data room. I've got a cardboard cutout of Commander Data there, right? Yeah.
I love it. But people are putting stuff back in there. Yeah.
Are you seeing this? What do you think? It's funny, I've got a good friend of mine I was catching up with for lunch yesterday, and he's working on a box that's a self-contained AI model for the masses.
Yeah. And we're talking about how he's not building it for me or for you, he's building it for his mom. Something that is local.
He's a security guy, so he cares a lot about where the data's flowing. Keeping that data, yeah. What the access is.
So I think there's going to be... The pendulums are great. It's just like client server and distributed systems.
We see it go back and forth. What's old is new, right? It's crazy like that.
We took that into account when we built Foresight because it can use a Frontier model through an API, or it can run against a local model. And we have both running. Really?
And we use both. And that's one of the best- I think that's the future ... and that way if we have customers that want to run fully on-prem and they want data sovereignty, we can give them, okay, point it at your model.
Great. We'll use your model. Use our tooling, our harness.
We'll layer in our failure analysis, our data on top. And I think that's what you got to do in business, is you got to meet customers where they're at. And when people want to go one way or the other, sometimes you got to go both and make sure they both work well.
Agreed, man. I love it. People want to get more information on Gremlin?
Yeah. ai. How do they get started with Foresight?
You can sign up for a trial and play with it today. And what I love is Foresight can walk you through the whole trial process as well. It walks you through setting it up, tells you how to do it, can do pieces of it for you, helps you get up and running really quickly.
And then, yeah, just start asking it questions. What could go wrong with my system? What should I be worried about?
What test should I run? If a test failed, tell me how to fix it. Give me a diff.
Yeah. And proof is in the pudding. We use it ourselves, and we got a high bar on accuracy and reliability, and it's the tastiest choice.
Come in and ask it the hard questions. Love it. Hey, man, always a pleasure to see you.
I know you're running out. You're only here one day, but it's great catching up. We'll see you on Techstrong TV soon.
I love it. Thank you very much. Colton Andrus from Gremlin here live at KubeCon.
You're watching Techstrong TV. Stay tuned, we got a lot more coming at you.