Observability: An SRE Perspective – The SRE Show Ep 6

it everybody Welcome to the SRE show. We're so glad that you've joined us for this episode. We've got a really great topic and some fantastic panelists first.

I want to thank our partners at lightstep who are just fantastic to work with and helping us put the show together creating some of the topics and involving people from different companies. So it's it's a sponsored by light step and it's really looking not just at that one company's perspective, but multiple and so you're gonna hear that really quickly. So I really appreciate the approach that folks have taken and it's great to partner with them as I mentioned.

So our topic today is starting starting a SRE program. And what's that like bringing that up in an organization? You know whether you're starting from scratch or you're starting from you know, you've entering an organization.

How do you introduce the observability? How do you introduce these different elements is the Google model the right model to follow and kind of where do we go from there? So we have a fantastic group of folks here, and I'm just gonna start out and let them introduce themselves Anna would you start out first?

Definitely. My name is Anna Margarita Medina. I'm a Staff developer Advocate lightstep.

I get to work with customers in the community teaching them ways that they can be proactive about its ability and incident response prior to doing this. I were Gremlin focusing on helping companies with chaos engineering but like that also meant like we're starting our SRE practice like chaos is the way we're going to start how do we go about that and prior to that? I was an S3 at Uber where I came in and a program was running and it was really interesting to onboard to things like that and my prior life.

I was that developer that said it worked on my machine, and that's all that matters, and I didn't understand anything in system. So it was a really nice way to come into Uber and have to learn a lot of stuff and then getting a chance to share with others what that Journey has been like is extremely rewarding and I'll pass it on over to Vince trying to do some songs. Thank you.

Meet everyone. My name is Vince Vaughn. I currently work at k6 which is a subsidiary of subsidiary of grafana labs.

I am a program manager there and prior to that. I you I worked it Gremlin with with Anna and Jacobs will soon learn but a lot of my history revolves around incident and problem management. And watching things burn really effectively is how things evolved.

So yeah happy to share some of those stories as we go along. excellent, Adriana Hey, I'm Adriana. Vallela.

I am a senior developer Advocate at lightstep. I'm actually pretty fresh to the developer Advocate role before that. I was managing and observability and platform team at two cows and I've done a bunch of work in the devops space, especially like getting people excited about devops.

I worked for a large financial institution in Canada where we spent a significant amount of time just educating folks on devops practices. And why why you should do it why it matters and all that and then before that I spend to good 16 years of my career as a software engineer doing Java development. So my career has taken various different turns and it's been awesome.

Great and last but certainly not least Jacob. Everybody my name is Jacob flick and I'm a Staff software engineer at Fanatics betting and gaming. I'm spending a lot of time building a Sportsbook gaming and betting platform from the ground up and a lot of that is going to be spending a lot of time building out.

It's measurement practices are SRE methodologies spend a lot of time with developer education and experience and stuff like that. It's actually week four of that which is really cool spent a lot of time building. Actually.

It's kind of a little day job because I was part of the founding SRE team at Fanatics next old coal now because it's like four different companies now, which is pretty wild so it's kind of boomerang but not a boomerang and I spend a lot of time at Grandma and building Out cast the during practices and a lot of saree stuff there and agricult Labs working on observability and onboarding for that. So yeah excited to chat. Excellent.

Excellent. Well, let's I'd love to get some perspective on kind of where you all have started your Journeys on in becoming well becoming SRS yourself, but also helping introduce that into organizations that you've been part of and yeah, you probably done that both as in your role as a product company, but also I'm guessing you know in your own internal, you know, operations and implementation of that. So who wants to take sort of a first shot and there were then we're gonna dive into some let's talk about observability and incident response and all of those things and how they fit into that and our goal is just to share our experiences that hopefully help others and also learn from each other too.

So what's to start off the topic? Allah I'll take a first stab because I it's something I'm thinking about right now as I'm literally building it from zero to one literally right now. Um, and so there's definitely been moments of like deja vu a little bit like I like to say that I'm part of the Old Guard of of tech where we had to physically pick up a phone and call someone there wasn't a alert management platform or any sort of incident response platform that would tie into your ability that to wake somebody up.

So sometimes that involved calling the wrong number for I am so that you know, because I'm human um, so but what's what's wild about that is is it created that sort of like empathetic mindset for me almost automatically even just by like I don't want to call the right the wrong person and but I also tied into making sure that we had the right play books in order to like make sure that I have all the information that I that I But I need for this incident and hopefully actually resolve it myself and at the time it was sort of this is back in 2015-2016 where it was super popular to take a bunch of Engineers and Tada. You're all so reason now congratulations you did it. And even though like typically sres are defined by, you know, software Engineers that are that are you know, it sort of in charge of the reliability of X Y and Z, but in many cases we were just in this case we were just a bunch of folks that were desktop support Engineers that got brought on to you know, server infrastructure and then sort of here you go.

So we we learned a lot and grew a lot by being lost in the sauce and then kind of figuring out as we go that's kind of how things started for me. Joe I'm gonna slightly different perspective. So I've never actually worked per se as an SRE.

I feel like I've done SRE adjacent stuff. So like I worked at a large financial institution for about 11 years of my career and during this time of seeing my fair share of like really bad things happen and really bad practices. So for me, like always found the missing piece of my career was like this marrying of like development work and operations work and and then like, you know, when when devops became a thing and then like SRE practices became popular and like oh, this is the thing that I've always thought was like the cool thing that should be done.

Like, I've always, you know, it's just chatting with our our team on slack last night where I was saying, like there were always people who at the bank were like monitoring emails in the middle of the night waiting for incidents to happen just so that they could like Swoop in and restart whatever server was hanging and they'd be kicking it on the overtime pay so like there was honestly no incentive to do better because it's like, oh I'm making so much money in overtime just like restarting services and I like there was a rumor that a lot of these guys had like their fancy cars funded because of that so from seeing all these like horrible things happen. I'm like, okay, there's got to be a better way and you know like I'm not one to say that the that I subscribe necessarily to the Google SRE way of doing things but I have to say that the Google SRE Bible has been a good guidance on like hey, these are things some things that you should be doing is guiding principles. So like when like my role before joining lights that I actually managed a platform team and we had sres on the team and we were trying to elevate the SRE practice because I think there were more sres by name and not necessarily by practice.

So we really try to emphasize like having a call rotation. But like when you were on call you didn't work on like Project work you were doing things to improve the reliability of the system work and trying to encourage teams to do like releases during the day and not necessarily have to do like release windows. So these were the kinds of things that I tried to instill on the team to help kind of achieve that story maturity that wasn't there at the time.

I'm definitely with you guys on the on the empathy bit. Where? I think that.

In order to have like a necessary mindset, like there's a little bit of empathy to have been on call to have made that for him phone call even now like when I try to set up meetings with other people in there on the other side of the world. I feel really bad like not Not impose on their time schedule right and my have to talk with them outside of their window or they're supposed to be worth, right? So yeah, I like it.

It's like on call PTSD where like I just don't want to talk to them outside of their times. and I feel like that's a really stressful thing like in in various roles that I've had like, you know, I've I've gotten called in the middle of the nights a couple of times and I don't know. I feel like some jobs like they overwork you so much that like getting called in the middle of the night, even though you know that you're on call like it's still you know, you have like a visceral reaction to that.

Right? And and so like I feel like on call PTSD very real very much needs to be addressed but not enough companies are addressing it most definitely I was one of those are gonna stations like being an Uber and SRE was was interesting. Like I was on like I was on call for chaos engineering service that some other folks have built prior to me joining and a lot of that kept breaking and I kept getting Paige and there was no one else I could escalate to and I was still very Junior like learning a lot and like trying to triage this big things and it was really interesting to watch and definitely kind of like a really rough way to start on call in general like throwing into just the well and no practice but also in you know toxic environment and then a lot of my colleagues like seeing them like Go through terrible trenches and like their SRE.

Like just life at the company and like like the way that they had to deal with their PTSD throughout these years. I hear of a lot of my friends still kind of like refusing to be in us three circles because of that or like they want to do ops work and S3 work but they feel constrained to software engineering because it's like I've already been burned. I've already watched everything burn like my mental health suffered.

I don't want to get there again and I think sometimes like we don't talk about that enough like we don't talk about like Like this really hurts a person and you can't get that back like you you've lost that time but you've also changed a lot as a person and then like, how do you make it better? Like one? How do you like recover from that?

Like, can you recover from that? Like, how do you start realizing that there's gonna be workplaces? I could treat you better like teammates that are not going to like are gonna be able to trade shifts with you and like not have these like everything is already set in stone.

We can make any changes to then like having the conversations of like what is the way to do Humane on call or like having like empathy within SRE and like that that goes like not just necessarily having empathy for the engineers in your company and like the tooling that you're supporting as a platform but like building that empathy to now be to your users and be like, oh like we have to really make sure that you have a happy experience when you're using a products. I think there's almost a pavlovic response to you're getting that message a text message or whatever slack whatever it is saying. Oh no, if I pick this up, I know what's gonna I know what I'm in for right or at least some idea and that's triggers that PTSD you're talking about just a major Duty logo of the sound for some folks.

Like I've seen I've seen all sorts of things. Like I literally just got my first one. I'm actually like recurring it right now and I was setting up our first like all call schedule and I was just like well here goes I got to put my phone number in like and then like I got to test it and I was like, I have not heard that voice in four years old.

Like I had like this like like really like chilled out of my spine like whoa, but but muscle at the same time. I'm also like really excited because like I get to I get to sort of learn from all of my previous experience learn from a lot of folks that Talk to communities that I've helped build over the years and and you know fingers crossed build it right, you know. Okay, one of the things that really helps.

So with with that is having like the Backing of you know your bosses to be able to like build the practice the way you want it right? Because I mean I've been in companies where they're like, oh, yeah, we want to do the right thing and then when push comes to shove, they're like no no. No, we need to have maintenance windows and you're like, come on.

You know that those are both really good points because it is about that evolution of going getting out of firefighting mode. Right we've lived in that world for much too long. And of course there's that still happens.

But how do you make that Evolution into you know where you're doing? What we would you know textbook wise think of more traditional SRE work they're using observability tools you're doing performance testing you're doing to kind of improvement of reliability not just stamping out the latest, you know text or page or Duty message we got how do you make that shift? It gives me baked into the culture of the very least the the team that's responsible for this and certainly at least the leader that's above this like something that I talk a lot about even before joining spending a gaming is like even before like even like the interview process was like I'm not going to go through what I went through again.

So what are we going to do to make sure that that doesn't happen and it was really clear that like we're going to actually spend time like creating metrics around average incident load per person, which I've never worked at a place that has done that before. Um, it was always just like to Adriana to your point. Like you're right you're on call so you're not working on, you know, project stuff.

So, okay great. Like I don't have stuff in jira, but like I'm still doing stuff, right? Um, and how are we tracking that and then how does that tie into where things like burnout begin to sort of happen?

m. Tomorrow tomorrow you're off, you know. Like it's not even like a thing.

See you later. Yeah, I think that's like such an important thing to have like five to have quantitative evidence of like hey, this person is getting called way too much and like, you know, Anna and I were talking to Norah Jones at jelly a couple weeks back and she was talking about the fact that you know person X might not be on the on call rotation. But if they keep getting pulled into incidents, well, you know that's got to be measured right because that also contributes to burn out.

It and it was like it was a really interesting conversation because it was like there's things that happen in incidents that you can't necessarily just look at like we don't really have logs of like, oh, these are all the systems talking to one another but like if vanessa was the only person that like knows that system on Sequel and like the only like single point of failure and like your managers like not really aware that every time you're getting that alert like you're having to go through her to either push a review or even just like verify that you're running the right query after like it was interesting because it's like all these little parts of a system that we don't really get to see and those conversations don't don't happen, you know, and it's really nice to start seeing like how do we change the incident response platforms? The folks are using in a way that we bring in? More numbers to a quantitative That Matters to the business like whatever that means for folks like from just slo's to like this person is dealing with five incidents every single month like how much of a load is that and like these are the seven ones.

These are the Sub Zeros and even like you might have way too many alerts like like those those kind of questions that you could be having that are like as as the way that we have seen reliability work throughout the last few years like we've learned a lot and like we don't necessarily always make the system changes. So I think we're going to start seeing that a lot more. Yeah, definitely experienced.

The things won't change until something really breaks or once a leader leaves. Like I think like Jacob nailed it in the head is like it comes from leadership like from doing a lot of the work and reliability and diverse inclusion and Equity. I'm like if a leader is not bought in into how expensive it is when things go wrong, whether it's I'm in the front page because people can't access my bank to I'm currently getting sued because of treated all my black employees wrong like until those things happen.

A lot of companies are not like, oh we actually need to invest in this and then that actually brings out conversation. Like, how do you go about investing in it? I think for a lot of the folks in the room as we were prepping for this call.

We kind of mention like there still is notion of what you can just buy a tool and like things happen and it's like, oh no like that's that's not that easy. You actually have to do the work and show up when things around SRE. Is sort of the engineering part of the title, at least it is a discipline and you're bringing it isn't a tool it isn't, you know res just responding to things.

It's it's how do we add that into to what we do, you know in an operational kind of role whether you're coming at it from Dove or Ops or whatever? I'm curious. I can't remember who somebody mentioned about.

I think it was you Anna that was talking about the having the data the information available, you know observability tools things like that. And of course, you know, it's a big topic today is observability and why is that different than sort of the platforms and alerting alarming? Things that we've had in the past and why is that important going forward and how do we make sure we're understanding how to leverage that I guess is where I'm going.

Maybe and if you want to start with that, I yeah, I can definitely start it where it's like we can have a quick answer with the system metric that like that is alerts a lot where it's like this. This is a threshold has been met and memory and CPU and load and what is that necessarily mean like your system has gone there but how long has it been is a service that is currently being like serving traffic is this even part of a critical path for our application? Like we don't necessarily have that and that's when a lot of like better practices in this space start coming in where it's like, okay, let's actually make sure we have tagging on our alerts so that we know what services they are.

And then you can nail down of like, okay, maybe we actually don't need to talk all these metrics for all these services and then you start kind of looking like okay what actually happens when you start looking under the hood you might uncover. Things like observability where you now start looking more like traces and you still get access to metrics and logs and events and that now allows for you to see like, oh like we're actually getting this alert for high 500 errors, but we're actually able to see that is because customers a plus b are actually having a huge traffic event and like that's why things are going wrong. So now that we get to have all this data that provides like more context into why we're even getting alerted we're able to see it.

But sometimes it comes into that you had to come from a bad practice of like everything is on fire like Vincent like you watch everything burn and you're like this is this is not okay and you're able to see that like the technology space has Has evolved a lot and we have tools and now our like more descriptive and allow you to be more precise and that that brings us to like, oh, we're not necessarily just being reactive. We're actually being proactive because we get to uncover like Oh, if we actually make sure that we are having some Auto scaling policies when customers with with this amount of services using our platform if that automatically scales up when we start seeing this increase start getting to 60 70 like you might not even be getting those alerts. So like you're just understanding your systems more.

It's what it comes to it. And it helps you understand a little bit before the alerting phase of things right? Like even as you mentioned like this is oh we're getting a bunch of these 500s, but then you one of the benefits of some of the observability practices having some of that back historical data, right?

Like if I have these 400s and they've been occurring every week at 4 AM. It's getting not gonna be as much of an issue as you know, all of a sudden they're occurring it to in the afternoon right in comparison. All right.

So having some of that that historical data. It's kind of expose, you know, where where you need to to shine a spotlight in terms of Summer here your practices. I would also say like, you know, if you were to sum up like why observability is great for sres, I feel like observability gives SRE superpowers.

I mean all of a sudden now you have insight into what's going on in your system what and you know, if you're doing it properly like if your system is is properly observable. You have all the information you need to be able to debug your system, which means that you don't have to rely on, you know, Jenny and Johnny who are always getting called because they've got domain knowledge. Now, you're you're junior person can come in and troubleshoot and they might not be able to do it as quickly but they'll still be able to do it in a reasonable amount of time and as a result like the Cascade effect is then observability.

It gives you psychological safety. So now we're starting to reduce that on call PTSD. I also think to kind of zoom out a little bit like I started to think about observability as the the story of our of our systems and our our people both like internally and are are how it impacts the external customers if you if you have them because we're going away from a lot of the conversation being I need this fourth monitor because my fourth monitor is just a dash was I look at literally for like six hours under the eight hours that I work that I work.

Um, and we're and that's sort of that intermediate area that's tied that's sort of tied into the alerting mechanisms, which ironically led to the alert fatigue that we're fighting and has created this like Incident Management, like pardon the term evolution of this these new pieces of tooling and and sort of where like things like tracing I started from as well because we're we're still unpacking these little bitty pieces. Of like like any kind of mentioned sort of tribal knowledge and single points of failure because it's not just systems folks. It's people too and and how because and this is It's both somehow still really difficult which is like kind of okay because like as Engineers, we don't want to be bored.

So we still want to tackle difficult problems, but from an onboarding story perspective. It's also easier than ever. This was pretty it's been pretty well to see it grow.

I was gonna add on with that observerability allowing Junior folks to like help and dig in into when things are going wrong. I think obserability also uncovers like another factor into it where I know someone organizations doing this where it's like not necessarily having all the engineers be able to like SSH into boxes to put up logs. And a lot of this is great because it's like you're not allowing them to just run random scripts on there and like was verifying those like, they're not going through like all the channels of like proactive reliability and quality control.

So like by having obserability, it's like those folks actually don't need to go in there to grab those logs because the tool was able to like have all of this for you and all just allows you to move a lot quicker sometimes like you might feel like you've been at that company for two or three years and you know what you're doing, but you're still gonna have a little bit of that fear and that like comes into that psychological safety piece. Because of so, I'd say yeah Vince is there an element of you feeling more equipped because it isn't just you know, here's a bunch of data and here's a bunch of alerts and I got to go figure out what the heck all this means and figure out how to tie it all together. But if you've got some, you know some technology that also helps you put it together and get answers a little more easily that's got to help with at least some of the trauma we talked about earlier in part and I would think why that would be an attractive part of I think about observability is one of the tool sets for sres.

I think from like working like touching like open source Technologies and using like vendor toolings in the obserability space. Like you've got that where it's like I'm not having to do command app and like find my service find this this crash loop back off and then like try to start going back down. It's like, oh I can actually see it all and someone the tools would allow like color coding or like just click into this and you get more of a view and like you start like being a detective and by being a detective you're just kind of like pressing buttons and you start finding that context yourself and the tool kind of guides you through it versus and prior days where you're kind of like having to run through papers and papers and you like highlight and like search for it and then like put this entire map together like this map kind of starts being shown to you digitally in a way and that leads to those incidents being closed the law a lot quicker like I think this you know, it's a good segue into like talking also about like, you know, the tooling is important obviously because it's what enables us to do the things but having like the proper practices in place also enables you to do observability properly because you know, I was at an organization where all they did was like throw logs at a tool.

Well, guess what you're still running into the same problem because you're still having to sift through logs that are like disjointed and you don't know you don't have the context right? So observability like when you start thinking about observability is like a trace first mentality where the trace gives you that and to end view of your of your system. Then yes those live messages start to make sense because they're baked into the trace and that becomes like the bread crumb into like things that are happening into your system that give you that Insight so you can trouble shoot.

So I think part part of the challenge. With modern observability practices is now getting out of that, you know traditional like let's look at logs and throw them into an observability tool and let's actually look at traces with that contextual information. Let's not rely so much on the wall of dashboards because I'm sorry that's like super useless.

Nobody's got time to sit at a dashboard for like most of their work day. That's where you know, like a solo-based alerts come into play because then You know, it's it's based on you. You determine your slos based on your I guess it's it's like a feedback loop based on your observations of the system and you keep tweaking those slo's as you go so it's not like, you know set in stone, but those kinds of things can help guide like what alerts become meaningful so that you know, when you get that pop up, you're not like, oh my God, I'm gonna like your heart starts palpitating and you're like, why am I doing this as a job?

I hate my life, right like a lot of the topics or even the directions like that. Observability and even SRE as a whole is an evolution, right? I think it's also right like it's not something that that a manager regardless of his intention his or her intentions.

It's not something you just drop in right? There's a there's a learning curved everything right? Not necessarily just practice even just the tuning that we have, right?

Yeah kind of mentioning that we're not just through this through all my logs into this tool and Magic happens and everybody wins, right? There's you need to make sense and be able to connect the dots all this. So like as as we go through and discuss some of these things I imagine like these are eating like easy to implement when you're a bigger and more mature company, but like when you're a smaller company and you have like five employees, it's not really something that's feasible to drop in a hole.

That's every practice. Right? So a lot of learning is that that come out are gonna be evolutionary gonna be something that that is as you learn or as something catches on fire you're gonna learn from that and Be able to adapt to that as you move on.

I I totally agree and I I think as long as you come in with the I'm gonna learn from from my mistakes and and are willing to adapt like that's awesome. The problem you get into is when you're you're in these companies and they're like, okay, we're you know, we're a small company. We just got to get this product out the door screw the good practices fine, whatever like sometimes you got to cut corners, but then when there's like no plan in place to like go right those wrongs then I think that's where we're companies can really get into trouble.

Yeah, I think this is sort of where I always like to I plant my flag where I State pretty emphatically that incense aren't bad. They are really great places in which to in which to learn I argue that if you don't have any incidents you mean you do and you don't know and that's bad because your customers do now. Um, and so that's where you should be spending your time.

Just saying, To that point one of the talks I've given quite a few times is you know, we know the failure that the the quote failure is not an option actually failure is required. Yeah failures to improve things right a hundred percent people dying in space. That's not an option but failure right here is it's so yeah, you know, whatever just reminds me of is you kind of you can't bring the old way of thinking to the new job of I mean, yes, you bring your experiences of all that but you know observability isn't a better grip tool right days are like grabbing different logs and piping you I called that tracing it was not tracing.

Yeah. See, that's not it right. Yeah to rethink.

How do I have to rethink the problem so that we approach it differently and leverage the technology to help us more than if we brought our old kind of mindset to it. I think this is a where it's you hear this a lot around like we talked a little bit about being proactive earlier and it sort of ties into practicing your Incident Management process game days things like this something that's really interesting is from like this is something that I was really passionate about when both it's my entire time at Gremlin before that after that. Um, but it's actually a regular story requirement for us, which I think is really interesting.

Um, whereas like Anna Vincent and I were just like, please do this proactively company X like you will help yourself. I promise and so that's, you know all that to say Getting not necessarily getting in a room necessarily, of course with you know, we're still in the pandemic of sorts, but just being able to say and be able to ask a question and getting it answered is so valuable. Um, especially if one person has an assumption about how a system works and another person has an assumption about how a system works and then the actual reality is is different.

It's actually options see that part of like being productive about your reliability I think is part of that shift within Siri and like I think that's already Bible does kind of teach you someone that but I think a lot of folks like that as an action item of like you kind of need to look into failure in order to make sure that you're setting things right like you have to be in an uncomfortable place your system kind of need to get there in order to make sure that when they naturally get there, they're ready to withstand such failures and I think it's it's great to be able to do that. It's like let's actually make sure that we want our embracing failure. This is also a great way to see like how that culture of like blame like a blameless is and making sure that you're you're kind of setting up your team to rely on one another to not necessarily Point their fingers of like Bobby rolled out this yaml file and it broke these five Services.

He has to go fix them and it's like That did happen but why why are systems set up that this can gradually roll out to five services on one and not doing one for like one at a time and we kind of like gradually see this on a set of changes. They just having those discussions are going to be really worthwhile for any organization, but it's looking into us or practicing us three, I think from like the work that I've been doing is that is like You can be an organization that's been doing SRE for a while. And you run like something like a game day and you realize like oh we could actually be doing things a lot better.

We uncover something really small such as like outdated links and scripts and run books and this a smaller company that maybe just has to us to raise runs a game day and they still have similar learnings and it's like you it's just about being proactive and doing the work like showing up and being like we're gonna be thinking about failure and how do we care about our customers more? yeah, I think that the you know Embracing failure giving a safe place to fail I think is so key and I think you know again gives that psychological safety, right? In order for that to be possible.

You need to have that open mindedness as well and not not just like not just the folks doing the work like the individual contributors, but leadership needs to have that open-mindedness towards embracing failure and not turning it into this four letter word and you know attack the issue not the person like who cares who caused the problem to happen the fact that it happened means that there's a gap in the workflow. opportunity to address this issue ding ding ding I had leadership that definitely instilled that in me early is that it's okay to fail the other thing that he also instilled in the news. It's not okay to fill more than once the same way.

Right? So like we want to be able to experience it and then pick up the learnings and Justice we can mitigate it doesn't have up That's speaking of failures topic of incident responses come up a lot right and talk about observability as well. And incorporating that kind of a mindset By the way managers out there if you're listening if you really want to shock your team.

Send your team up the beginning of the day say Okay, I want to go out there and fail. Don't look at you. Like oh team what happened?

You know, they just got covid fog. What's the matter with that? So when it comes to incident response, it isn't just the response right?

It's also okay you've learned. I mean you go through the process and resolve the issue and you're all so collecting knowledge information data, right and then you got to do something, right? How do you incorporate that that learning Behavior into the process?

now so I'm kind of start with with The post-mortem for me at least when I start working with folks as it starts with a timeline, right understanding what breaks when it breaks and how it breaks and how long it takes to for someone to start to try to address it right? Whether it's automated or whether it's like an actual person doing something right? So starting with the timeline and an understanding what those time gaps are like that's your Baseline of what your average incident might be and how long it might take and how dramatic it might be.

right and then starting to to analyze and break it apart as if there is a 15 minute Gap in your timeline either someone didn't write something down and like we need to try to address that a little bit better or there was just some idleness or something else. That could be a good address. Like how can we cut and our best practices?

How can we cut this in half? Right. It doesn't have to be cut in half.

But that's our that's our starting point. That's our goal. Right.

And so it's it's an open-ended question. It's it's something that the team is as a whole can start to address is and come up with ideas. Right?

What works best for the team is gonna be what works best, right? There's no definitive way of this is how it's always going to be and this is how it's how you can always cut our times in half to the point where you know, it's a measurable. It's never gonna happen.

Right? But at least we have the discussion at least we start thinking and brainstorming ideas of what can happen after an incident so that it doesn't happen again. I think like doing that like really knows it and then like also adding questions like Colin why like, how is it that this rolled out?

How is it that this got push? How is it that this system talks to one another and then even like asking like those why questions for more context like why is this built this way? Why did you go look into that service, like what tribal knowledge that you have like because is that we we all come from so many different spaces where it's like, oh I've been working with red is for the last five years.

So I'm aware of like this really interesting new one says or I've been at this organization for 10 years and have Bill half of this system or I've touched half of the system and like now I bring in perspective that like folks that have been here last time do so, it becomes like this opportunity of like this cost of company a lot of money. Like let's make sure we document a lot of it but that we're able to learn as much as we can and then like take those learnings and like go be proactive about your reliability. Excellent.

Anybody else have a kind of last comment on that topic that we're running out of time getting close to the end here anybody else want to jump in? I'll just make a general comment, which is I think in order for any of this to be successful. I think people have to be curious and open-minded at the end of the day right because without that How can we how can we evolve our practices right?

Um, I think it's kind of brings it back to Anna's Point around us. I found that in my a lot of my previous experience a lot of the incidents that whether in to be honest many cases, there were no postmortems or any sort of analysis or conversation. It's it's not it's not broken and everything's on our fire and that was really the end of it right when the reality was like then it would break again or break in a different way.

Right? And so we're missing the story that got us to from point A to point B, and a lot of it is a lot of those a lot of really effective slos also have business minded logic behind them as well. And like for example, some of the things that we're doing is not just like isn't it reviews with like the business level or business like the P level folks and not just like this is why it won't happen again.

Sorry. It's more of this is this is also like tied into deadlines and should Be maybe pull back on a feature because we're busy fighting fires. That's where we have every budgets for and and that's how those conversations become because typically when you're talking about reliability to someone that's a VP of Finance.

They're only thinking about the bottom line number when when in the reality is that but online number can be affected by the decision that you make in this meeting. Absolutely, we're and we're gonna have to let you have the last comment there Jacobs now. Congratulations.

You win the win the game. We gamified great. Well, thank you to all you hear a great panel.

I can't wait to get you back together. Again, Jacob Vince Adriana on it fantastic, you know, and it's it both you and all your wealth have experiences and knowledge. It just fantastic to share.

So I hope our audience will join us again for the next episode of the SRE show is you can tell we're getting into a lot of really good stuff here, and it's all real and it's all practical. So I hope that you taking away some really good things from this and thanks my panel with me for joining us today. We'll see you again soon.