Why Chaos Engineering Is Essential for IT Resilience and App Availability with Rohan Gupta

Hey guys, thanks with Throw. We're here with Rohan Gupta, who's vice President of Cloud Security and DevOps at our systems. And we're talking about well, chaos engineering.

Rohan, welcome to show. Hey. Hi.

Michaels. Glad to be part of this, Right? We've kind of been talking about chaos engineering for many years now, and yet the adoption of it has been somewhat uneven.

And I think a lot of it has to do with the fact that the systems are so complex that a lot of people, while they're just hesitant to deliberately break something and they spend all their time trying to just make sure that damn thing works. And now you wanna come along and deliberately destroy something and then get a, a little apprehensive. But from your perspective, um, it almost sounds like maybe we reached a point where chaos engineering is now essential.

So like when it comes to chaos engineering, and you've just pointed it out very nicely that it has been in the market for quite some time. Okay. But for what I believe chaos engineering is not about disruption.

It's more about learning. Okay? Learning about how your system behaves when an actual disruption occurs.

Okay? And that is something that we have to start incorporating as a culture, uh, instead of doing it as a sidekick project, because most of the organizations love the way on how chaos engineering worked, but never integrated it as a part of culture. Instead, they just incorporated it so that they can get some fancy dashboards, get some outage summary, and show it to the management.

But now it's slowly changing with all the outages that has, uh, been reported. Uh, and even on cloud, like for example, people think that if they move from on premises to cloud, uh, the SLA objectives have, uh, improved. They do not have to test chaos or they do not have to worry about disruptions, but that's not the case.

In the last one, one and a half years, we have seen a lot of disruptions, lot of outages where now people have started implementing chaos, integrating chaos in their culture so that they know at one given point of time if there is a failure in the service mesh that they have been engaged with, they know on how to fix it. They know on what the R-T-O-R-P-O impact is. So yes, it's now culture and it's more about learning then disruption.

And we also reached a level of complexity in our systems where it's just impossible for any human to kind of keep track of all of that. So we need a different approach. And chaos engineering kind of shows us where all those dependencies are Absolutely.

For us, as the system grows. And we have seen it with enterprise organizations, we have seen it with midar organizations too. Now it's, it's more complicated.

There's a lot of APIs involved, there's a lot of, uh, third party integration that are involved and people do not care about on if it's a third party integration. We don't have to worry about the outages. What chaos gives you and what we feel chaos would give people the ability is to actually get the failure endpoints that nobody cared about.

Okay. Not the actual outages. Like nobody right now would work with traditional chaos experiments.

Now people have started designing custom chaos experiments, which are, which are revolving around third party integrations, which are revolving around around service meshes where nobody was able to actually pinpoint to what the failure is. So all these custom experiments that people have started writing have now given them the ability and confidence to be able to say that yes, our systems are resilient and we know on what happens if there is an outage. We have a rollback plan sta strategy, we have an observability in and around it, and we have an hypothesis what, where it has to stick to.

So yes. Hmm. Um, are there kinds of different degrees of chaos engineering?

Are there best practices and levels of maturity? I mean, I'm asking this 'cause a lot of folks are still trying to figure out, well, where do I get started with chaos engineering? What I feel when it comes to chaos engineering, the default set of experiments is where everybody starts with, okay, your default set of experiments include everything that has to do with infrastructure and that you would get out of the box.

But where it becomes interesting nowadays is when you start writing, depending on how your architecture changes, okay, today, let's say if you've written like 10 experiments and you have come up with, let's say a framework that keeps on running these 10 experiments, six months down the line, your architecture would change. You would introduce 20 services, 30 services, and 40 services into your infrastructure. And that, that is where it becomes interesting.

That is where they would start pulling in custom experiments. That is where they would start writing something that would give them that confidence and give them that failure endpoint. So I think yes, this is how they, uh, they try to deal with the complexity of the ecosystem and then it becomes more like, more like a gamification for them.

It's like, oh, there is new set of services that are introduced. Let's build chaos experiment in and around it and let's start, uh, running some game days, running some postmortems in and around it and see on what the outages is and see what can we get as a technical depth output. So I think that, I think the teams, the infrastructure teams, the SRE teams are now taking it more as a competition, are taking it more as a challenge to see on who can get a better outage experiment and get a more resilient report out of it.

Mm-hmm. Is this something that I do before I deploy my application? Or is this a valve where maybe we need to do continuous chaos engineering?

'cause the environment is always changing. It's, I would say it's a combination of both. Okay.

Now, if you, if you start believing in the safe shift left approach where let's say when you are integrating new services or when you're writing new services and there is a team or there's a parallel team that sits, which is basically an SRE and you bring the SRE team into conversations, you bring your cloud engineering team into conversations, you tell them that, guys, we are coming up with new service stack, and this service tag would consist of these services. These services would have integrations with our current API layers or current infrastructure that the team can start designing the experiments from day one during the planning stage. Okay.

Similarly, like when you have already done it, okay, there could be parallel silo teams that would be working on it. What happens when it is the moment you get the releases in, or the moment when the new infrastructure or new services connect within a stack, that is not where you would see outages that that kind of outages, that kind of disruptions you would anyways, uh, catch during the release cycle. So the latter would also work, but the shift level approach or a proactive approach would yield more benefits.

Mm-hmm. Um, will AI make it easier to create chaos engineering tests? And is that gonna help people maybe find this whole discipline more accessible?

Interesting. And I've, I was reading a lot about on how AI can help come up with custom chaos experiments themselves. Okay.

Like for example, we, we've been experimenting with, let's say if I, if I just upload a architecture or application architecture or an infrastructure architecture to a LLM model and, and just give a prompt stating that, can you give me single point of failures in it? Okay. Can you design some experiments that would, that would introduce custom outage kind of scenarios for it?

Yes, it would give it for you. But what generally happens with ai, AI would give you sort of experiments that are already in place that are like top 10 experiments, top 20 experiments, top 30 experiments that pa that that, that most of the world has seen. But when it comes to architecture, when it comes to infrastructure, when it comes to application layer, the internal team or the group of architects or the enterprise team knows on what could be the challenges.

And if it has like hundred services involved or like 200 services involved, they are, the better they are, the better people to understand on where could be the outages and what kind of custom chaos experiment combinations would work. Yes, AI can help, but AI will right now definitely help to get you the first 20 or first 30 which are out of the box, which anyways you have to implement. But the next set of 40 or 50 would be best served by people who are in-house, who have designed the layers, who have designed the architecture, who are closely working with the SREs and the CloudOps team.

Mm-hmm. Should the people who built and deployed something be those same ones who are doing the chaos testing? Because arguably they might be too close to the environment and maybe we need a different set of eyes.

Exactly. So the people who are actually maintaining the infrastructure is generally what I, what what I give an advice on are the ones who should not run kiosk experiments. Okay.

Because for them it's more like a runbook because they know from a point A to point B to point C, these are the failures that have happened. They know on what the rollback plan is, they know on where and what matrix they need to collect and see on what the outages is. It should be a set of engineers, uh, that are part of the development team, that are part of QA team that should run it on a, let's say a game day or a green zone and see on what the outages are.

The infrastructure team, the, uh, which would be a combination of SREs and CloudOps team can help these QA engineers or help these test engineers to create a postmortem report. That is where I would say that the help would come in and once the postmortem report comes in, these are the engineers that would help to again, stitch the system back to where it was work on a rollback plan and work on a technical depth where, which was, which was part of the observation of the postmortem report. So my, my answer to this is the ones that are maintaining the infra should be, should not be the ones that run the kios experiments because they know, and we don't want to run it as a continuity of business drill.

We want to run it, uh, enough way that it, that it actually points to an actual outage and, and see on what the outcome is and see on what a normal engineer workflow would look like. Engineering workflow would look like when an actual update office. Mm-hmm.

And is my goal to create, you know, the perfect system or am I more likely just trying to figure out how to have a, a, a gentle set of cascading services that, uh, migraine and degrade depending on the outage issue, but at least my application is always available. But I mean, to what degree am I trying to maintain absolute availability? Great.

So when, when, when we design, okay, kiosk frameworks, when we design, uh, custom ecosystems for people to shoot these experiments to their infrastructure and mostly production, there are some, uh, bullet points that we generally tend to give them. It's like one, you have to select a green zone. Okay?

So even if there is an accidental outage, you know, on what would be the impact and what would be the, uh, what would, what would, uh, what would be the sentiment, uh, when you run these experiments? Okay? You have to decide on the green zone.

You have to decide on a window where it would least impact the customer experience, where it would least impact on what your actual business is. Okay? One is that the other, when you are designing it, you have to, you have to make sure that the set of services that you're impacting does not disrupt the entire suite of application.

Okay? Because there could be a hundred microservices in the ecosystem. You design game days, you design kio, experiment suit in such a way that you give a heads up, okay, to the monitoring team.

You give a heads up to the escalation engineers, you give a heads up to the teams that are maintaining it, that this time around we are actually targeting 20% of the application suit and the outages to this 20% application, uh, suit will have this impact. Okay? So you need to be ready with it.

And it could be anything, it could be latency, uh, for the user experience. It could be delayed in response, it could be reports not going on time, but it, there should be a controlled, uh, a controlled ecosystem and there should be a controlled environment where this should be run. And primarily, as I pointed out, there should be a green zone that should be defined.

There should be a time where you know that there should, uh, there would be a least, uh, it would least impact the customer experience. Yeah. So when you see folks adopting chaos engineering, what's that one thing that kind of makes you shake your head a little bit and say, folks, we need to just be a little bit smarter than we are.

Uh, when the entire suite of chaos experiments is targeted directly to production environments in the first call, that is where I tend to advise people. And I tend to tell people that the approach should ideally start with lower environments. Okay?

You have production grade environments that people use in demos for beta testing, for staging, which are pure replicas of production, okay? So when you come up with a new set of custom kiosk experiments that you have designed, even if you're designing it in-house, or even if you're integrating third party apps that are well versed on designing, uh, kiosk experiments, the first target ex uh, environment should always be lower environments. Okay?

It could be production like environments, or it could be vita environments and demo environments. When you are confident enough to understand, uh, that the targeted chaos experiments would create a certain degree of disruption and you have on what the outage could look like, that is where you would start pulling in subsets of these experiments and start gradually pointing it to production, okay? And then do it in a similar way, as I pointed in the last answer, uh, where you design on, uh, green zones, where you see on what the impact is, you, uh, give an alert or a notification or a broadcast out to the team so that the teams are ready in case just in case if there's a disruption to go to a rollback not as soon as possible.

All right, folks, you heard it here. If you really want application uptime to be as robust as it can possibly be, you have to embrace the chaos. Hey, Roan, thanks for being on the show.

Yeah, thank you, Michael. It's nice talking to you. All right, and back to you guys in the studio.