Charlene OHanlon & Donovan Brown & Abel Wang – Fireside Chat with The Black Shirt and The Rockstar: From Waterfall to DevOps

com, Container Journal, Security Boulevard, and the Digital Anarchist streaming media platform. In addition to TechStrongTV, so many things going on. I'm excited about every single one of them, and I'm super excited to be moderating this fireside chat discussion today with Donovan Brown and Abel Wang, both of Microsoft.

And we are going to talk about pretty much anything DevOps related. And the I guess the the session title is officially Fireside Chat with the Black Shirt and The Rockstar from Waterfall to DevOps. So Donovan and Abel, thank you so much for joining me today.

I am so excited to have this conversation with you guys. We're excited to be here. When they reached out, I was like anything I can do to help and Abel works for me.

So I'm like whoever you want on my team, you can have them. So I brought Abel with me. Excellent.

Excellent. Well, I know you guys have done these conversations together before also, so I'm really excited to see how you guys play off each other. So I think this is going to be really fun event.

So just to lay the groundwork for today's conversation is, as I said, it's it's pretty much all about DevOps. But really, let's let's face it, we're in 2020 and DevOps is no longer that nice to have that. So many organizations we're kind of striving for today.

It is absolutely a must have and it's necessary to have the DevOps best practices in place so that organizations can be more competitive. And we all know that today's technology and business landscape demands that companies have a level of competitiveness that just is unprecedented. But we all know that doing DevOps and DevOps practice DevOps and best practices are not easy to do.

So you've got to have the right culture. You've got to have agile processes. You've got to have the right tools.

So that's what we're going to hear from Donovan and Abel today. We're going to talk about what's necessary for organizations to not only just survive, but thrive in this highly competitive business landscape and how DevOps is a critical enabler for doing that. So, gentlemen, as I said before, thank you so much for joining me today.

Why don't we dove right on in? So let's start by kind of talking about how DevOps has changed over the last, I don't know, say, ten years or so. You guys have been around long enough that you've seen a lot of changes in the business landscape and the way organizations are approaching their software delivery and application development.

So what do you when you think about DevOps, what has changed over the past ten years or so? One of the things that I've noticed has changed quite a bit is the tooling has gotten so much better. We used to have to string all these things together piecemeal or write it yourself or have these scripts and only ran at certain times that we were trying to get to this automated deployment.

But the tooling just wasn't there. And then all of a sudden, I think it was around 2013 when I just saw all this great tooling with InCycle, I had InRelease, and then we had Puppet and Chef and DSC like all these tooling, they're just it's like this converged on this problem, like we see the problem to let's go around and fix it. And it was easier then to get people to buy in because first of all, people hate change and moving from a waterfall world to a manual deployed well to a fully automated deployment using DevOps best practices, there's a lot of change.

And so people were resisting it. And the excuse they gave was, we don't have time because it's so hard. Right.

There's nothing that's going to help us do this. Over the over the last ten years, what I've seen is it becomes so easy and there's wizards that will help guide you through it. You can now just download templates from the Internet that have full pipelines built into it.

The excuses have gotten harder to come by because the tooling has gotten so much better. You know, the attitude has changed a lot, too, right? Because I remember ten years ago we were we needed to convince people DevOps is a thing.

And this is really important. You need to do this. I'm not really not having that conversation anymore.

I'm having a lot of conversations of how do we do DevOps. But I don't really have to tell people how important it is. That's a really good point, too, because I've obviously we visit a lot of our our top customers.

And the reason why DevOps is such the hot topic right now, it's like Donovan. We just met with customer X, Y, Z, and they're there on their DevOps transformation. Can you come in and talk to them about it?

And you're absolutely right. I very rarely have to go in and say it's important to do just like please tell us where to start. Like, we have no idea.

There's so much inside this landscape with infrastructure code, continuous integration, the right branching strategy. Are we agile, scrum or XP? How do we how do we do this is the.

Question, what should we do this, which is a really good turning point as well, because it used to be just like agile, like we had to convince people that scrum and agile was the right thing to do. And Waterfall's old. And now we're trying to tell you like, no, you can't be doing this manual deployments, and especially when you're in the cloud and you have all this this these resources at your fingertips, you're not utilizing them correctly because you're not doing infrastructure code.

You still manually going into a portal and provisioning an individual VM. Are you kidding me? Like, no, let me show you the better ways of doing this.

And again, it's the tooling that has changed, in my opinion, so much that made it a lot easier for us. Do you think that the fact that so many organizations now are there and they're endeavoring to either start or accelerate their digital transformation efforts in light of the pandemic and the fact that they have to pivot so very much, much more quickly, has that accelerated the DevOps conversation as well? One of the things I think it's done is it's really shined a light on how important it was to already have been doing this.

When people say, well, Donovan, when should I adopt DevOps? I always say yesterday, like, you should have done this yesterday. Like it's not something you do in the future.

It's something you should have already done. And those who had already embraced their digital transformation, their DevOps transformations, didn't change a lot of the way they do business come Covid-19. It was like our deployments are automated, like no one has to be at the office to do anything or or have their hands on a service like, no, just commit your code to a repository and we're able to take advantage that those who are very hands on and very manual are now trying to figure out how do I live in this new world that doesn't allow me to be connected, to be close to people, to be in the server room where I can pull out a tray and type on a keyboard.

It's changed their worlds, I think, a lot more than those who have already realized that DevOps in the digital transformation was something that we had to be thinking about going forward. So I don't know that it's accelerated it, but it's definitely again, it's helped us say this is no longer a debate. Either you do or you lose.

Those are your two options. You either implement DevOps or you lose. So how can we help?

Yeah, it's really shown the spotlight right on the problem. And like you were saying, though, you need to be doing this yesterday because if you're starting from nothing right now, it's not an easy transformation to make. Right.

And this is something that takes years and years and there's no done in sight. You're just constantly improving. Right.

But if you haven't started this transformation, starting now is better than starting tomorrow. Right. So do it now if you haven't done.

No, I love that. I love that. It's I remember there is so much I can't remember who said this.

It said DevOps used to be just for unicorns, but no, it's for horses now too. Like you don't have to be the special Netflix. You don't have to be the special company that goes and does these things.

And I also get the question, so what size team is it? Is DevOps overkill for a team that's really small, like I do DevOps on products from the only developer, like I use the same DevOps best practices when I'm the only developer on a web app, as I do when I'm working on a team. I'm an athlete.

I was always taught that you play how you practice. So I'm going to do something else when I'm developing on my own, that's drastically different than how I'm going to develop when I go and play on a team or I'm a member of a team. So by doing those best practices, it just becomes second nature.

For me, it's muscle memory. Like, of course we have continuous integration and continue to do that. We set up.

What do you mean like what do you mean? So we're going to go to the portal and provision of no, no, they're not like we're going to do it right. If you go ahead.

I'm sorry. I was just thinking for real, I really would rather help implement DevOps best practices with a really small team versus a massive and like, if I do, it is it always is easier. Well Donovan, you use the the athlete analogy.

Abel, I'm assuming that you would actually use the muscle memory music analogy that you've got to have the practice every single day. Yeah. To to to be a great musician.

And, you know, obviously doing doing DevOps takes a certain a certain mindset and and organizations really have to kind of want to be able to do it to to have it be a successful endeavor. But, you know, one of the things that I've seen with companies when they're looking to approach DevOps from the tooling in the culture and the and just the especially the people perspective is they kind of get waylaid by this this paralysis of analysis, so to speak. You know, they just there's just so much to internalize that they're just kind of step away and say, you know what, maybe we'll visit this tomorrow.

But you guys are saying those days are gone. I mean, they can't they can't keep putting this off. Tomorrow never comes because there's always another deadline.

There's always another feature that drops onto your lap. There's already another event that you have to prepare for. So you have to start carving out time in your schedule every day to go and do these things.

And what I tell people is stop thinking that you have to stop what you're doing now and go do all this research on how to solve every problem you're possibly going to have in your DevOps transformation and then come back with a document that says, let's go do this. What I encourage people to do instead is look at what your current process is today and no matter who you are, you have a process. You might not have documented it, but when an email comes with an idea and that idea goes into a backlog and that idea gets tested and then pushed manually or otherwise to production, that is your current process.

And throughout that process, there's some portion of that where if anything bad is going to happen, it's going to happen right here because this is where it always happens, like something falls through the cracks or the test doesn't run. Like that's where I want you to focus you as an individual. Now, you as an organization are what the person who believes in this, who's passionate about this, who wants to champion this idea to focus on that portion of your pipeline or your process that is the most risky, most dangerous, most painful.

And what I want you to do is take 15 minutes out of your lunch, 15 minutes out of your day and go research that one area, maybe don't have continuous integration. Go do some research on what what it takes to set up a continuous integration server for your organization. Most of them are free, so you don't have to go ask for permission because you don't need budget.

Right. Most of them can be implemented without impacting anyone else in your organization. If the four of us are the three of us were on a team together and we were just committing to a repository but did not have continued integration, I, Donovan Brown could go off and investigate what it takes to stand up a CI server.

And neither of you have to know, even after I failed several times, neither of you know, you still clone your repos. You still push the repos. Your life stays exactly the same.

But then eventually, after the week of this, I'm going to get it to work. And all of a sudden, every time you commit, my system is going to start firing up. It's going to start compiling the code.

It's going to tell me if it's green or if it's red. And then one day we're all going to come into work. Abel's going to break the build and you're all going to get or I'm not going to get the code.

You're going to get the code. And I'm not getting blamed. Yeah, you're going to get the code on your machine and it won't compile.

Why? Because the build is broken and you don't know that there's a signal out there that lets you know that it's not good to get this because you've never had continued integration. But when you come to my desk ready to complain about Abel, well, I'm still working.

You're going to ask me, how are you still working? Didn't you get the latest to Mike? No, I have the CI system that told me that the build was broken.

What's the CI system? So it's a completely different conversation. I'm not trying to convince you that we should go do this.

Go give me time. You're going to see the impact, the immediate impact of me having this functionality. I'm like, oh, well, give me your email address.

I'll have my system email you as well every time the build is broken. So you're never experiencing this again. And all of a sudden you've made that process better for your entire organization without asking for permission, without having to debate people.

I've had people come to me. It's scary because I speak around the world back when we could fly around a lot. And I'd always ask this question, how many of you work in an organization that does not have continuous integration?

And I was dumbfounded. I never was in a room, no matter how many people were in there that did not someone did not raise their hand and say, I'm working in our organization today that does not have continuous integration. I want to ask them why.

It's because they said I tried. I couldn't convince anyone. Like, how much time did you spend trying to convince them?

Weeks. It would take you fifteen minutes to have done it. So why did you spend a week trying to convince people for something that you could have been done with fifteen minutes setting up a CI system.

So my biggest advice to anyone is when you want to start this transformation, don't go ask for permission on the part of your process that hurts the most. Go fix that for your entire organization and watch them come flocking to the rest of those changes. Now, for the record, I don't break the build.

And that's that's actually true. You and I remember that one project that I worked on together and I kept thinking it was you that broke the build. And it was me.

Like four times in a row. I swear Abel broke the build. It was every time.

So I pick on him. But he is one of the best developers I've ever met. I probably would break the build.

So I love the point that you made Donovan. The whole thing of just you know, you don't have to tackle everything all at once. And in fact, when people tell me that's what they want to do, I'm always just like, whoa, whoa.

Like, back up a little bit. I like your attitude, but you don't want to fix everything all at once. It's just too much.

I find what hurts the most. Just do that. And one thing that we say all the time, Donovan and I, you have to measure if you.

If you measure what your process is from one step to the next step to the next step, you can see what hurts the most and kind of work on that to start chiseling away on how long it does something right. So measuring is important as well. I agree.

Not only measuring your pipeline because you might make a change that actually is bad for your process. And if you don't measure before you make the change in the measure again after, you don't know if you make progress or not, you're just assuming that this was a good thing to do. And it's funny because that measure and then measure again after the change not only is applicable to your process, it's applicable to your application that you're developing, because we just assume as an agile shop that the product backlog is in priority order.

We assume that the thing at the top is the most important because that's the product owner's job. Make sure we work on the most important thing. But historically, Abel you know, we both know this, we just assumed they were in the right order.

Like we have no dat that said that was actually the most important thing. But today you can put telemetry in your software so that when you deploy, it's actually sending information home, saying, hey, 90 percent of the people who visited our website actually visited this new page and was able to get some functionality. That's really valuable.

And we need to do more of that. And you might find other things like that on your product backlog in lift those higher in the backlog. But the reverse is also true.

You might ship something in our telemetry says nobody is using this. Like this is not even remotely valuable and we need to stop doing this stuff and find something else to do. Granted, just because they didn't use it doesn't mean it's not valuable.

Maybe it's not intuitive. Maybe it was hard to find. Maybe we need to have a toast that says, hey, look at this new thing.

But what's nice about the data is we can now go run experiments. We can go see if we can make it more intuitive. We can go and try to draw people's attention to it.

And if they still don't use it, the data is telling us that is not important. And you as an organization need to go try other things. So, again, the measure part is not just your pipeline, it's not just code..

You need to be measuring everything that you do and make sure that the changes that you're making are driving you towards the right goal and not in the wrong direction. That telemetry thing is so freaking important, right, because we are really bad at guessing what what people really want. Look at Microsoft.

We thought we were awesome. And I mean, we would interview our customers. We bring them and we were great at it.

Right. We started gathering telemetry. We started saying, well, a third of the time we were awesome customers, loved our new features.

During the time, they were kind of like, yeah, whatever. And a third of the time they were like, this new feature is so bad, change it now or we're going to another product. Right.

And what was surprising to us was this isn't just specific to Microsoft. If you look across the industry, you pretty much get that one third, one third, one third. But if you gather telemetry, like Donovan says, you can now tell which things are working right now.

You can double down on features like that or avoid the features where you note are not delivering value. So instead of your experiments being one third, one third, one third, you start building upon it and you become more and more successful. I think it's funny that you said we spoke to our customer that's just recent because we didn't we used to look into our crystal ball and say, we do what you want and you're going to love it.

And that's the got clippy, right? You know, you need this. You're going to love it.

I miss clippy. So I just had to I had to go somewhere in that conversation. I'm going to start a campaign, bring clippy back.

Listen, I've gotten a couple of questions from the audience. I want to throw one at you, the first one here. What is your best argument to people that have been around for 30 plus years that that DevOps is a culture and not a group within the IT department or even worse, a contract?

I would agree with that. It shouldn't be another group that is yet another silo in your organization. I get very nervous whenever I go to a company, say, Donovan, we're all in Devops.

We've got this DevOps team. Michael, what does this team do? Did you just enable one of your scrum teams to be more ops oriented?

No, no, no, no. They're the DevOps team. They they they're so I'm like, so now the dev team has to talk to the DevOps team within talks to the Ops team for them.

Well, sort of like that's bad. Let's just go ahead and admit that's just yet another silo that you created in your organization. Oh, they drive all the best practices.

And like every one of these things, those pipelines are going to be some similarities. But the problem going to be unique, depending on what it is to deploy and be careful about that. That just makes me nervous.

But I want you to do instead is empower your current scrum teams, right. To act, to be able to own that pipeline. One of the things that we did at Microsoft with the agile DevOps team is we treated our actual pipeline as a feature of the product.

Right, because that's how we're able to ship so fast. Stop thinking about it as something you have to do or something that you do on your off time. No, this is a primary feature of our product.

It's on our backlog and how we're going to improve it and make it faster. This is a feature of your product. And when we treat it our pipeline like that, they've got our devs more involved in our pipeline and got our ops more involved in our pipeline.

We didn't build another team. What we did is we use this DevOps transformation to bring the teams together, not create an excuse for building another team in another silo inside of organization. So it is more than just people.

It's more than just process is more than just products or tools and technology. Right. It's all three of those things coming together to allow you to continuously deliver value to your users, which goes back to the measuring that we were just talking about.

Is it really value that you're delivering? One of the other things, I get a lot about 30 years and like long time stuff is people not wanting to change is like I've been doing the same thing for 30 years. Why am I going to start doing something different now?

I think it was Jeffrey Snover who said success is the biggest deterrent to implementing DevOps. If you've been successful doing Waterfall for the last 30, 40 years, why am I going to switch doing it now? And I tell everybody, because your competition is now, you don't want to be Blockbuster.

They actually had an opportunity to buy Netflix. I mean, the story of Blockbuster versus Netflix is this just like there are people didn't get it. If their people got it.

If there are people could see the future, they would have snapped up Netflix with a bargain and we'd still have Blockbuster today. But there are people there. People did not get it right.

There was no process or tools in the world that were going to convince their people that they did not want to come there every Tuesday when the new releases were out and go grab microwave popcorn in their favorite DVD or video. And they just didn't the people didn't see it. And one of the ways that I encourage people who don't want to change their organizations is I use a workout analogy.

We all know we should be working. We all know who we should be going to the gym and staying more healthy. I'm not, right.

But for me to be more successful because I don't want the change. I like my routine. But if I hire a personal trainer, whose is going to hold me accountable and make sure that I do everything in the right form and call me and make sure that I went to the gym.

Today, I'm more likely to be successful. So when I find a company that's really resistant to the change because of seniority or just the history of that organization, I recommend bringing in a personal trainer or basic person is going to help you through this digital DevOps agile transformation, whatever transformation you're on and guide you through that process. Because when a person like Donovan Brown walks into a room, I don't care how long you've been doing this.

The passion that I have for the change I'm encouraging you to do is going to persuade a lot of people who would otherwise have resisted if it were coming from their peers. It's something about this. Abel and I were consultants for a long time.

And it was it was it was interesting to watch. As I'm having a room in a conference room, I'm having a discussion with their with their leadership. And I'll say something.

And half the room is writing down what I'm saying. And one person will have this visceral reaction to what I just said. And they just spoke in their seat and they're looking around like everyone, like, what are you what are you writing this down?

I've been saying this for years and you didn't listen to me. But Donovan says the same thing. And you listen to me.

And I would look for that person to sit in their seat. And right after the meeting, I would run to that person, say, tell me everything else I need to know about this company. He says, but why are they listen to you?

That's not important. I'm the outside consultant. I'm the expert.

They're not going to challenge me. But you know this company better than I do. What else do I need to say to them?

And that's what you're going to need as an agile coach, a DevOps coach, a digital transformation coach to come in and say, hey, I know you've been doing it this way for thirty years, but let me show you let me convince you that this is a better way. And if for some reason they listen to outsiders when they won't listen to the people inside the organization being. It sounds like parenting.

Is that parenting to? I will not get into that conversation. I'll turn it over to you.

Now. Are you going to say Abel? Oh, I was just thinking, being an outsider totally helps to write because you're not involved with the politics.

You're not involved with the chain of command. You know, when I may or when I was a consultant, an outside consultant, helping people make this digital transformation or make this DevOps transformation, I could tell the CEO something. I could tell somebody with thirty years seniority, something.

And if they didn't like it, I could leave and get another contract. Right. It's no skin off my back.

But if I was within the chain of command or if I was from that company, are you really going to tell your manager that they're doing it wrong? I totally don't blame people for not being able to do that. But that's why it's helpful to bring in somebody like Donovan.

Exactly. And this is a perfect example. Abel is sitting here because I'm his boss.

And I said, hey, you can't do this. I'm going to tell his boss no. So we totally understand it, because if my boss is done and I need you on a plane to go to X, Y, Z, I just start booking my flight.

Right. It's just hard to say no to your own boss, but it's really easy for me to say no to your boss. So hire me to come in and help you.

And even if they like that guy's crazy, it's going to be rattling in the back of their mind. Like, why did he challenge me on that? Like, why are they telling us to do things differently?

Even if Abel, like Abel said they didn't want to hear it and I go somewhere else. We've done our job by forcing that conversation, forcing that train of thought on. Maybe we really do need to rethink this.

And one of the things I have the power of doing at Microsoft is sharing with all of our customers that your competitors are also our customers, and I've already said this to them, and you should just see how their eyes get the biggest place because they realize that, yeah, I don't care what industry you're in, your competitors have already had Donovan and Able come and talk to them. So don't be the last one to figure this out. Don't be the last one in your industry that I have to convince that this is the right thing to do.

You want to be the first one in your industry that I talked to, not the last. And that kind of lights a fire under some people to make them realize that I got to go do this. Yeah, but aren't there some organizations that just don't get it and won't get it?

I mean, you know, because not every organization is going to be a DevOps organization. Interesting. Like, do are there organizations that don't get it?

One hundred percent. Will they never get it possibly? And if they never get it, I think they'll be the ones that we talk about.

They'll be the blockbusters and the taxi companies and the bookstores of the future for us. Those today that do not believe that DevOps is something that's important to them will be those that we talk about in the future that no longer exist. Right.

Because I can't imagine I've been to almost every industry and every name brand that I can, but I personally know of talking to them about this. So like everybody is realizing, no matter if they're making snacks that we eat or driving the cars that we drive or selling us the groceries that we eat, I've been to all those places. Right.

So I can't think of an aspect of my life that I haven't visited that customer and talked about these things with. And there are some that get it more than others. There's some that are further along in their process than others.

And there's some that struggle because waterfall in certain areas of their their organization works really well. If you're building an automobile, they can basically tell you down to the minute when your car is going to come off the assembly line. Right.

That's a beautifully waterfall type of process. And they have it to a fine, fine degree of certainty. So when you say you can't do that with software that they just don't understand, like, no, no, no, no.

If we can build an automobile with this process, I can definitely write software. This process like, well, it just doesn't quite work that way. And sometimes I struggle to get them to understand how the software that you develop is going to be different than the than the product that you produce per say.

But once I get him to understand that they kind of loosen the reins and realize that. Let me give you an example. Your competitor that does the exact same thing that you do, like how cool their car is, because I can open it or unlock it with my phone or I can do this cool thing or, you know, parallel, for me, that's technology.

Right. And if you're not doing that and they outpace you, people are going to be buying their car instead of your car. So, again, it's just this weird bit of that fear of loss of revenue customers that drives people to do the right thing.

Yeah, beyond a shadow of a doubt, teams that adopt DevOps best practices out innovate teams that don't. Oh, yeah. Not by a little bit by, but by a lot.

And I've been talking about DevOps for feels like forever now and just to be DevOps theory. Right. I would doubt it because I felt it in my gut, but I didn't know if it was true.

I tried to convince people it is true. But today we've been DevOpsing long enough. Now we have empirical metrics that show very, very clearly DevOps aspect.

You innovate faster. You actually have a much higher revenue stream. This is just plain and simple.

Of course, you were saying, right, Donovan? Companies that are successful or organizations that are successful, those are the hardest to change, right? The ones you still are already making millions, maybe even billions of dollars.

And they have huge walls in place where where their competitors just can't scale. But even companies like that, they get displaced by little upstart companies that have a radical new idea, a radical new way of doing things. And all of a sudden they're playing catch up.

And the incumbent, which is so sad when the incumbent is refusing to change, has every advantage at their disposal. They have the experience, have the name recognition. They have everything on their side.

And for you to lose all that, because you just lost sight of the fact that technology is literally running every aspect of our lives. People don't understand that. It transports us.

It educates us. It's where some of us fall in love on apps, on our phones. It literally hasn't permeated every portion of our lives.

And if you don't understand that, you can use software as a strategic advantage against all your competition, like like I said, we'll be talking about you, but not in the way that you want. It'll be when you went bankrupt because someone else ate your lunch. So let's pivot just a little bit, because I've got this question, thought it might be timely to discuss, and that is how DevOps has been impacted by covid-19.

What have you guys seen? I mean, apart from the whole kind of acceleration to digital transformation and and beyond, but from maybe from a moral kind of visceral construct? What has changed with DevOps since Covid-19?

I'll let you go first on this one Abel, because we've had this question before. Yeah, it's more important than ever now to have DevOps best practices in place with especially with the impact of Covid-19. Right.

We can't we can't hang around all together in a server room somewhere. We can't do it. And you have to have DevOps best practices in place to.

Just function now, it's it's I don't think much has changed, except the importance of DevOps has really been highlighted now. I think that's the way that I would put it to you, because I haven't noticed since what it was in February when we kind of all went on lockdown. I don't know that I've seen any.

New innovations that I've maybe I've missed something or any new theories of DevOps in a Covid world, it was like, no, it was more like, thank goodness we have DevOps best practices in place because we've been able to weather this storm. We've been able to still rapidly deploy and innovate our applications. We're still getting updates to our customers.

Nothing has changed because we had all these processes and people mindset and culture and we had all of our tooling in place to be able to still function even in a completely remote environment, because I tell a lot of our customers is a litmus test for how well or how mature you are in your DevOps best practices. I asked them how.. What is a developer have to do to get code into into a repository, into production?

I hope the answers commit to a repository. If your answer is anything beyond pushed to a repository, you've got work to do because that's what the developer should worry about, is I did a I did a git commit git push. And moments later, however many of the moments that is.

But there's no there's no roadblock. There's no heavy lifting done by human beings. There might be some approval gates in there that we want to make sure to have been cleared.

But again, that's all automated. And I can do that on my phone. I can do it from my house.

I can do it from my tablet. It's not I have to be in the office so that I can go physically, do something or print out a paper or send an email or. No, we want to automate that.

So what I've noticed, as Abel just pointed out, is that I think it's shined a light on how important it is and what an advantage it gave everyone who was already doing it. When when that's when our we've got turned upside down in teams that adopt DevOps best practices. They already have that mentality.

Right. Then DevOps isn't a one and done. You don't just buy it and right for that, you're done.

It is a journey of continuous improvement. You're constantly asking, OK, what hurt the most this time, this iteration? What can we do to fix it and make it better the next time?

So even when Covid hit very, very easily could be things that that we're not working quite right. But if you have that DevOps mentality, you quickly look at and be like, oh, what things do I need to change? Right.

Very rarely do I hear people say things like, well, assuming they have the right mentality, very rarely do you hear people say, well, we can't change, because that's just the way we've been doing it for the past twenty years. Sure. If you have bought into DevOps, you're much more accustomed to that type of change, introspection of saying what needs to change, what hurt, what can we do better for the next iteration?

I think the agile processes had taken a harder hit than the DevOps process because a lot of teams who I've been on our team, we had 50 scrum teams that all physically set together in the same room and did their stand ups in person in the same room that I think that disrupted a lot more than the DevOps team because it didn't matter where they were when they committed the code, the DevOps process on a run. You didn't have to be in that room and have that same collaboration. But from an agile perspective, from that process perspective, I think that took a bigger hit than any of our DevOps did.

I don't none of us on any of our teams is like, man, DevOps has really gotten hard. Now we're in Covid or is like, no, it's like I just still commit to the same depositary I did yesterday and the magic happens. But what about, like maybe shifting from continuous delivery to continuous deployment?

I mean, things of that nature where you're kind of maybe even going beyond what your basic DevOps processes have been or even introducing a security aspect and shifting from DevOps to DevSecOps? I mean, are things like that happening in the in the wake of Covid-19 or is that are those kind of separate and above? What what would what anything is happening in the environment?

Those things were happening already. That's that's right. That's what I'm trying to do.

I'm straining to see if I if I remember seeing a big uptick since February in any of those things. But I was talking tough DevSecOps. I was talking all the security.

I was talking the difference between the continuous delivery and continuous deployment. We were doing that already. And I don't think that I'm having any because I still spend a lot of our customers post Covid.

But the conversations haven't changed drastically. The conversations are still very on the on the the topic of we want to move as fast as we possibly can because we're being out innovated by our competition and infrastructures code is a hot topic. DevSecOps is obviously a hot topic, making sure so.

But they haven't become a hot topic since February, if that makes sense. Right. They were a hot topic before.

Oh, totally, totally understandable. All right. I got another question here for you, from the audience.

If in an ideal DevOps world teams are self managing, how do you make sure your engineers are doing the right thing? Well, I used pull request a lot of that is one of the ways that I do that, because for those who aren't hard core devs, a pull request is a process where a developer is making changes in a temporary branch that is disconnected from the main branch that's being pushed for production. And for that code to be put into that main branch, there's a process called upon request and threw out that pull request.

You can do all sorts of cool stuff before you contaminate the main branch of these pieces of code. You can run builds against that. You can run test against that.

You can have human beings actually review that code. You can have automated automation go across that code and the static code analysis and package and analyzing, analyzing analysis to make sure that there's no security holes in there, that they're using the DevOps best practices, that they're doing infrastructures code. So one of the ways that we do that is they are self managing.

And I also treat adults like adults. So questions like this always make me wonder, like, who on your team is acting like a two year old? Because I shouldn't have to babysit an adult who was trying to do the best thing for their organization.

But we do have things like approval gates. We have things like pull requests that give that second, third, fourth pair of eyes on a particular change. But I also if I have someone who's cheating the system we're trying to get around the system is breaking things.

I've had some really stern conversations with people before on that. We're not allowing this to happen anymore. And if there's ways that I can physically, not physically myself, but for example, we had one person who would always go into production and make changes directly on a production server that were not tracked and switch control.

So when we deployed again, stuff would break. We didn't understand why. And I basically just took their rights away from that particular environment.

Like, the only way you're getting changes inside of that environment is to go through the pipeline. So if I have to put those restrictions on people, I will. But I hope at first I give you the trust that you're an adult.

You know our goal. I'm going to treat you like an adult. I help you to behave like an adult.

You behave like a child. I'll treat you like a child. She will take out I'll have more people on your pull request, like whatever that whatever that has.

Once again, it sounds like parenting, being a people manager and a scrum master is really interesting because you're dealing with all these different personalities and behaviors. It's I knew I was going to be a decent manager after years and years of being a scrum master because it was all about trying to figure out how to communicate with people, motivate people, get people with very differing personalities to work together. But well, one thing that we were all always centered on is we succeed or fail as a team.

You don't succeed or fail as an individual. And that application crashes or whenever you can't place an order on a website, you're not thinking to yourself, man, that's Donovan probably did that. You're like that company is no longer good and I'm going to go to another company.

So when you think like a team, when you work like a team, when you succeed or fail as a team, a lot of that stuff goes away. You need your team to gel and have each other's back. And that's when I don't have to worry about all these policing and and putting stuff in the way.

But I've got some really young developers that are fresh out of school and just don't know better that we've had to go in and lock them out of environments until they understood. And they they were upset at the moment. But then they started to realize, OK, I understand why they did this to me.

I understand why this is far more reliable for us to do deployment. But if you're having that kind of stuff, literally start locking them out of the environments for which they're causing problems and giving them the only one path to production is through the proper process and pipeline that we have in place. Yeah, so I have a slightly different perspective because there may have been once or twice where I've acted like a child, a small child.

So one of the things that really helped me is at Microsoft. They started it, they started something where they made the developers feel the pain of their own bad behavior, right? For the longest time, I never felt the pain of that.

But of my bad behavior, because I write code crazy, crazy fast, write, I write code so fast that usually I grab it, user story, I write it, I go to the fence, I grab another user story, I'm off doing something else. And by the time QA finds my bugs and hands it to me, I'm doing something completely different in some poor junior developer has to come in and fix my bugs. So number one, I never thought I wrote any bugs.

And number two, I thought I was God's gift to programming. Neither one of those things are really true. So at Microsoft, they really start making it.

So you felt the pain of your bad behavior. You weren't necessarily punished, but if bad things happened because of your code, you felt it. So one of the worst things that could happen would be something like a live site incident on Azure DevOps services.

Right. That's a service that runs 24/7. It holds of all of our customers their source code, their pipelines.

If that goes down, people can't do their business right. So it can't go down. But if there is a live site incident, it's a really bad situation.

We get our EVP gets on the phone calls through his command chain all the way down to the two poor people that are on support. In this role is so horrible that they have to rotate it because it's awful. So you get that support call and it's if you're a developer one of these days, you're going to be on support.

You get that call. It's going to be at 2:00 in the morning because these calls always happen at 2:00 on a call bridge with this entire management chain. And they're constantly asking you, is it fixed?

Did you find it? And you're not allowed off that bridge until you slap a Band-Aid on the problem and you're still not allowed off that call until you figure figured out the root cause and come up with a plan to make sure that never happens again. Now, this call is so horrible, no one wants to be on that call.

So if Donovan checks in code and I see there's a pull request there, I'm going to grab it. I'm going to go through his code and check it really freaking carefully, go through it with a fine tooth comb, because I don't want his bad code getting on this call. And not only will I look at his code, I will look at his unit tests as well.

Right. Because those unit tests, those are the things that will protect me from his code again. So by, really making us feel the effects of what we do.

You know, I am no longer this mystical developer where I'm shielded from everything. I see everything that happens because of my code. I'm a lot more careful in that.

I tend to do the right things. I don't have to be punished for it. I don't have to be rewarded for it.

I just do the right things. Now, I think that's a really good point because when we put people on call is the motto, if you wrote it, you run it. And I come from Compact Computers, I, I live right down the street from their old headquarters to Compact Computers here in Houston, Texas.

And I asked Donovan Brown as a developer when I was done, would go into the server room, pull out a keyboard, logging into a Proline server and copy my software there. And then once it was done, I would run out of the room and it was no longer my problem if it crashed all night long, it was some poor support person on the phone trying to help our customers through this problem. I slept through the night like a baby.

In the morning, I get back into work all well and rested and feeling good and excited to go and write more books the next day. And some poor person I was drinking coffee. Those eyes are just worn out because he's been fighting this fire.

He logs a bug. Maybe I fix the bug. Maybe I don't write because I don't care, because if I don't fix the bug, I just I'm going to say great tonight.

Right. So again, those people not being good citizens are probably aren't paying a single price for their bad decisions. But as soon as you call me at 3:00 in the morning, because I didn't, I tried to cast a word to an integer and it crashed.

I'm going to look into my entire code base and look for anywhere we're using parse int and use try parse instead and start putting some logic in there to gracefully send back notifications or validation so my code gets better. So it's a really good point, Able. Having people pay the consequences for their bad decisions, they make less bad decisions.

But if I get away with being lazy and quick and throwing it over the fence, I'll keep doing that because it's easy. I'll be lazy every single time. Best Developers are the laziest people I've ever met because I'm going to write software to do what we would otherwise have to do ourselves.

I love it, right? That's why I love computers, because I can make them do the stuff I don't want to do, but I do it quick. And we're also very optimistic as developers.

Right? We assume this hard drive space. There's infinite memory.

The network never goes down. Users are very good. They never type letters or numbers are supposed to go.

They never type numbers or letters are supposed to go. The world is perfect. It's not perfect.

And we program like it is, but it's not. And I've known as I've gotten more mature as a developer as I've been on call more as a developer. I've become a much more defensive programmer.

I validate things. I will throw an error at you at the caller very quickly, letting you know that this parameter is not right and that we're finding that stuff in unit testing. We're finding that a much earlier on.

We're shifting left all that quality checks versus putting that burden on our customers. We're now putting it upon ourselves. It's a lot better.

And also we also dog food, our stuff inside of Microsoft a lot. So if something goes down, we're the only ones that feel that first we don't give it to our customers until it's lived with. US, and then we give it out to our customers, so again, that that forces you to make sure your quality is good and your software is good because we're not going to just disrupt our customers.

We're disrupting ourselves as well, which is a steep price to pay. Sounds like sounds like.... We have about five minutes left until our session ends, but I wanted to kind of throw it back to you and see if you guys have any kind of final thoughts on where you see DevOps the whole DevOps market landscape going in the next few years and where you guys think we're headed as a as a business and and beyond.

I'll tell you where I hope we're headed. I hope we're headed to a world where DevOps is not something that we're talking about like we are today. Because I remember I'm old enough to remember one continuous integration was mind blowing.

The idea of having a machine that would compile our code every single time we checked it and it was mind blowing. Right. And now it's a checkbox.

You just check this box and that just happens, right. There's no magic behind it. DevOps is getting close to a lot of wizards.

We have a lot of templates out there, but it's not quite that easy yet. And I hope that we get to the point where DevOps is just something that everybody does and we're off worrying about something else. And if you need DevOps, you just check that box or you don't do anything because by convention we just do the right thing.

If you just commit your code, the magic stuff just happens. So I don't know where we're going to be, but I hope that's where we're going to be in that Abel and I are coming back and talking about something completely different because DevOps chat is something everyone is doing already. Abel, do you have any final thoughts?

I think that's really true. I want DevOps to be just a check box. I don't want to talk about it.

I don't want to worry about it. It should just be a thing, right? In the near term, like where something I look at DevOps, I think is kind of cool, right?

Because it's been a it's been an evolution. I mean, for the longest time, we couldn't even figure it out. If we can build the right software, so agile had to be a thing and then agile became well, it actually started working.

We were building the right software and immediately we hit the ball of we can't get the code that we built in a sprint into production fast enough. So then we start worrying about CI/CD and eventually we figured that out and then we started saying, OK, the code is easy, what about the database? And so we had to figure out database DevOps right?

Then we finally figured that out. And then there were things like, OK, great, we're pushing into production. You know, every time you check in, this is awesome.

But how do we maintain quality? Right. So we had to figure out how do you maintain quality in a DevOps world?

You got to shift left and that stuff. We figured that out. And recently people start worrying about things like security.

OK, we can do all that. What about security? How do we maintain security?

So DevSecOps is a big thing as well. Some areas that I can see in the near term future where I want to see more improvements would be maybe AIML, right. I think like MLOps, is it?

It does exist, but I think there's a lot of room for improvement there as well. So hopefully in the short term, that's what we'll see in the long term. It's just a check box or like Donovan says, not even a checkbox.

It's just a thing you check code. in and automatically it just behaves the way it's supposed to. Aren't you guys worried you'll be out of a job then?

No, there'll be so there's no link to the problems people. We're going to go and fix because I wasn't always talking about DevOps. I've been I've been writing software since '96 or something.

Right. And I wasn't talking about DevOps in ninety six right now. And 10 years from now I'll be talking about something else.

So not at all. Not at all. I just, I love DevOps.

It's been great and I'm excited and it's still I think we still have a lot of cool stuff we can do about it. But integration is just shouldn't be something that I hope 10 years from now that we're talking about at this level. So so if if DevOps ceases to be or do you think it will cease to be or do you think it will just become integrated and people won't actually distinguish it as DevOps?

That's what I think. The second one, I believe that it would just be what we do. And just like the branching strategy, just like git versus centralized version control, we're all now git.

And I remember having those debates. I was on the other side. I hated Git when it first came out.

But now I realize that we just get on board and this is how you do source control now and it's how you do branch and DevOps will be the thing that just happens and there still have to deploy code. We're still in to test them. The things we have to do to get code from the fingertips of your developers into the hands of your users, that's not going to change.

But I just hope it's not something that we're consciously thinking about, which is something that happens in our subconscious. Excellent, excellent. Well, Donovan Brown, Abel Wang, thank you so much.

What a great conversation. I really enjoyed this. I hope we can do this again soon.

Very soon. Thank you so much for having me. All right.

All right, everybody, please stay tuned. We've got lots more DevOps Experience coming up. Lots more great, great presentations from some very, very smart people.

But until then, I do want to again thank Donovan Brown and Abel Wang for joining me today on this fireside chat. So, guys, please feel free to move on to your next presentation, your next, I guess, next fireside chat. Maybe so, but thanks again for joining me.

I appreciate it. My pleasure. Take care.

Thank you.