Cultural Perspective to Achieve DevOps – Claus Jepsen, Unit4

This is Textron TV. Hey guys. Thanks for the throw.

We're here with class Jepsen. Who is CTO for unit 4 and we're gonna be talking about devops and where are we in the maturity curve? Because it's been a long haul Klaus welcome to show.

Thank you very much. Thanks for having me like appreciate it. we've been advocating the adoption of best devops practices to be kind for a decade or more now some may depending on how long you've been at it maybe even close to two decades all depends on your perspective and when you get started and yet adoption is uneven.

So what do you perceive to be the challenges when it comes to devops that we're wrestling with today and for that matter might we ever actually be done. That's a great. Well, yeah, I think we will I think the So the it's always been these changes right there and say there's this that's the part of it that have to do with the technology your processes how you set things up.

And the other thing is the human element, right? And you human element is always hot chains, right? There's also that's all I think being a lack of often getting to understand.

What is their work really all about right? So, Devops is not a it's it's a it's a mindset. It's so how do you how do you infuse and understanding of the challenges of operating your software into the engineering processes and a lot of companies?

They they have a they have a background and that background specifically in building software that when interest on premise data centers. And do you want really responsible for operating the software you're just responsible providing yourself software itself, and there was somebody else's problem. So when the sales Revolution or says our cloud or whatever we want to call again now it become obvious that you also need to operate the software.

And then what happened was that? Okay, we need to and then then somebody coined to turn their offs. And then the way a lot of companies took that thing away was to create a department called devops.

So okay. Now we are devops. We have somebody to operate the software and then we have engineering and then we call this divorce.

So that was like two different departments still so there was not really any key understanding he needs of these groups. What does it mean from an from from the construction of the software and to to run it and also what are we doing here in Ops to sort of make sure that the engineering really understand the challenges we have by operating it. So the first sort of iteration of this became what I call I refer to it in a block called the bridge so you have sorry.

So what we did then what's then we created the bridge which then got the label devops, right? So you have engineering do Ops and then the bridge we call devops and then you put some In there and their job became so on making sure everything aligned that you you know, they may be looking at the pipelines making sure everything is okay the participant for each group, but it was still a separate sort of discipline. If you wanted to have the engineering discipline and you have the operation discipline and that's where a lot of I think that's what a lot of companies are today.

They still haven't really crossed the bridge if you want. And crossing the bridge means that you you merge these functions and I think a lot of companies have done it, but I still think there's a lot of companies that haven't done it yet. And and I don't know why but the merch of these function actually means that It's back to this, you know this saying about your bill that you run them and and that's and that that bridge something.

I think it's very far away from a lot of engineering teams is the fact that now you need to operate it, which means that engineering have to get used to the thought that we also responsible for running it and and until you become responsible for running it. I don't think a lot of Engineers and Engineering teams really think a lot about observability traceability logging. How do you make sure once every time but as soon as you get this cold Saturday morning at 2 o'clock and your software doesn't work then your perspective on How you build your software how you make it resilient how you make it observable and how you make it predictable becomes important points on your roadmap and and and these are elements that needs to go on the roadmap.

And this is where if you look at a traditional software company, it brings us out to the product people as well. Right? Because that's also a change that needs to happen because previously in many companies product organization concerned themself with feature functions to the customer who pay for the stuff, right?

However, when you move into a service service for all Your customer democracy changes because now your users are also the people who runs the stuff right? So it's not just the one who pays the subscription fee. But you also have a huge part of group internally who are need to be satisfied because they need future functions in there, which means that all of a sudden you need to allocate engineering capacity to build observability into the application.

You may have to build monitoring portals into the application such that your support people you operations people and everyone can go and get a quick look and understand. How is the software really doing right? So and I think where are we today?

There's a lot of bridging going on. I think if you look at it a lot of companies it still have this sort of Duality to do an approach and then have the bridge in the middle and then there is this intern debate going around around. Oh is this a non-function requirement?

So who was responsible for functional and non-function and all these kind of good stuff and at the end of the day The requirement is requirement. You need to get it done. It doesn't matter if it's an unfortunate function requirement doesn't matter how the use of perceive it or not.

It is about how do you get these topics? That is typically not really of interest to the end user where the end user is the customer. What is a very interesting topic for the end users whether the unusual is the internal operations?

And that's a that really session is at least my experience won't happen on this you take the final step with this removing the bridge and essentially merging all these functions such that each of the teams working on services or whatever. They're working on. They own all aspects from from person to actually running requirements.

The non-function requirements do specifically at the use of demography the end users themself the intern uses and ensure that these things actually gets on the roadmap and bid and I think be a sort of Use it to a decade probably a decade or make sense. But but we have been slowly progressing and we announcer in the Brits area and I think going forward we'll see more and more companies. So getting to this realization, they need to merge these functions.

And yeah. I think some of the challenges, you know part of this so-called shift left mentality, but the developers themselves are not jumping up and down to say, you know, we want to run all this stuff some do they're called Full stack, but the majority of the developers are not full stack and frankly may not even be interested in being full stack developers. So how do we accomplish this bridge when there's a and an element of the community that's not that excited about it.

And then frankly there are software Engineers working in devops teams that manage operations that don't want to give up control of that anyway, so maybe we're perfectly happy where we are. Yeah, maybe I wouldn't I wouldn't say that's not a true statement, but my strong belief is that if you if you read if you realign your team subset the it's at the end of the day a lot is about for example, if you go through a change process is also to change the organization structure the everything how you work on it. How you go on it to match the reality you want to go to so if you make it the whoever it is responsible for these aspects and you then at the capabilities into the group because you know, it's it's different.

So the way I look at it and then maybe wrong and people make disagree for me. It's it's disciplines within engineering right some some right code some test code someone code but it's all about it's all in discipline engineering and the responsibility for all those disciplines within engineering sits in one team. So the team and so for responsible within that team you may have different different, you know people with different skills.

So who's interested in different? Things which means that you can have a if you have a big team it teams are six to twelve people. There may be one or two sort of more Engineers with Etc Automation and up specific background looking at what does it take to run it and they make sure when sitting with the table that these requirements are taken serious as part of building the product of service more service for service and products today.

And then and then you you have to include the engineers as well because they need to understand that if things are not if think if you don't take these things into consideration what your risk happening is that you're gonna get a phone call Saturday morning at 2:00, right because the services down and they can where they're gonna go then go to you and ask me to fix it. Right and I think they'd see and do yeah the interesting thing if I look at it, I think to your point around some people are pretty satisfied. I think that's more and more and there's more more talent coming out that has been through this.

I think there's the they're sort of different if you look at your engineering population, depending on your engineering organization, of course have a spread in terms of tenure and seniority. But what we definitely see is that there is a if you take some of the the younger intern into the market they are more attuned to this way you're working right? They're more sort of used to it because that's what the read about that's what they see many of them.

Haven't they don't know the Essentially, so it's just what it is. Right. So one way you of course could solve it is but also thinking about what are the talents you have in your company, right?

So who do you hire? How do you how do you Foster this internally making sure that you're telling grow towards this goal. So so sort of a dual thing.

It's not just about taking what you've got in converted. It's all making sure that the new people who's getting into the organization is already already have visibility or touch this kind of thinking which can drive this change through the organization because otherwise, yeah, you know, that's what I said. The human element is the heart so the human element is always the hardest part of any transformation right period and we can argue for ever about that.

But that's just how it is. So, how do we get them on this journey? I think you're getting on this journey by by adding new people into the teams who understand how this works and you so also make sure that they don't engineering don't feel that all of a sudden.

It they just get this responsibility done on them without having sufficient knowledge of people around them to help them through this change, right? So that's what I mean by merging the teams because the Ops Team knows so they need to go in the teams. I don't know some company may say oh, we don't need Ops in the name do everything and your host right?

Because then you definitely I think you're running directly into what you're saying. We don't want to do this. Right and we actually it may not that we don't want to do it and maybe by we don't really know how to do this.

We don't know what it means because we never written any software or supposed to be operated by ourself. It's clear that developers need more accountability and it's also clear that many of them. Don't want to be so dependent upon the back-end operations to execute a small change and they want more control.

The challenge seems to be that the developer Community itself is divided over to what degree they want that level of control and so do you think that this is a generational issue and you know anybody over 30 just doesn't get it. That's probably a little harsh. So I wouldn't say that way.

I think if I look at my own organization, which is fairly large. And I don't think I can say there's a specific heat or anything. I think it's a in general sort of if I look at the changes.

We have become through and it's taking us quite some time to be quite honest to get where we are today and And today everyone understands that this is the way we want to go. Right so and it it has taken a lot of changes organization and changes along the way and and key is to the key is to structure your structure your your engineering such that it's capable of doing it and it also making sure that the engineering have sufficient autonomy and accountability and responsibility, right? So the and it's all the leaks into something a britches into something else or Segways into sort of like, you know, are you a top down?

Are you born or organization? How do you beat stuff? Right?

How do you make sure who decides what's get constructed? What is the road map and who runs execution and I'm strongly believe that the further down in the organization. You pushes this autonomy and the better you are.

All right, because Then the teams themselves feel that they have a common responsibility and they have a common goal which is essentially to get software of service in the hand of the customer and at the end of the day, you know. I don't know any engineer. Or anyone who don't want customers to use their stuff are they Services product?

That's why we that's why we do what we're doing. We love it when there's actually users on our our artifacts. We don't love it just sits on the Shelf.

It's not used. But I do believe that in order to get people to change the mindset. You also need to inject I think and And telling people who've been dead on that understand how this is supposed to work again can influence the rest of the group.

Otherwise you and and make sure that they have the capability to position. There's nothing worse to be set up to fail and if people if groups feels this set up to feel like the DraStic case we take all ops out. It's denying problem.

They set up to fail right? Well, so I think it's more about creating the environment the framework give give the time because it takes time right so If you have a big Legacy application or anything like that, everybody knows the internet they know exactly where to go to push and they can fix things and all of a sudden you put in an interest something that's completely different. It's unpredictable.

They don't know where to start. It just takes longer and you need to accept the things takes longer if you keep the pressure on and believe everything is like it was yesterday and I'm talking management part of it. You're gonna get pushback because honestly you are asking them to do things that never done.

Guess what that you move from development research, which means we don't know what we're doing because that's research but you go from this. I know exactly what happens when I hit the compile button until I have no idea what's gonna happen when I hit a new project button. So you need to let the teams learn this new approach while they're building new Services as well.

So it's sort of it it would think. Yeah. So the minute you have two teams, you're always gonna have an US versus them kind of mindset.

But I'm we have a third team now security people want to be part of the process. So where does that fit in how do we dress this whole devsecops thing or should we even be calling it devsecops because security is just part of the motion. Yeah, that's an interesting question.

I think the security security is many things as of course as all the security in the infrastructure you run but I think the key thing about security is that you again you need to work with the engineering teams to make secure programming a part of sort of the internal mindset that you always think through you need to make sure you have the right Tools in place to constantly monitor. There's a lot of tools a lot of great products that can go in and do continuously security testing on your software and you build Pipeline and then again you need to work and make make the engineering teams understand that this is a key point. I don't know so I can tell you how we did it.

So there's probably sort of the so we made it. So as part of the development phase you have in you have executorials, but when you can deploy and then there are something that needs to be satisfied. We also have a good to go criteria is when you can start developing, where's a lot of criteria need to be done.

So the way we interrupt doing is that we run them very stringent. So we have a quality group that make sure that and actually an Albert they were kind of group that make sure that all these criteria fulfilled before it can deploy it and a lot of these criteria can be automated so the city in the pipeline and if something goes wrong, it's just gonna turn red and you cannot departure. And that and and initially that made a lot of pushback right because people well, you know, we can just you can fix it tomorrow or whatever, you know, this is wrong, but no you cannot do that because then you go back to the other thing.

You're gonna get a course Saturday morning at 2 AMA 2 am right. So that's the way we did it. So we put it in and made it more stringent.

Our executories are more firmed up and more stringent and that goes around security as well. So you need to pass all the gates in order to deploy. And so I think where I would at least personally few.

Push the security part is into the quality because it's a quality in my view. And this may also be a sort of interesting view but obviously that sets more in it's about security is about the quality of the problem. Well, how secure It is Well is the quality of the product so it needs to be in that area of the organization.

So you should probably put your security people into the quality to make sure that the quality folks get it only agenda and get it into the criteria, right that way you can then make sure that the coachship then of course you have only sort of fireworks and everything around it which isn't it's not more on the infrastructure security where they already heavily involved and have been for years actually so that but it's more again this mindset chains like with you run it you didn't run it your building secure and you run it in the secure environment, right? What's your best advice for achieving all of that? Do I just take everybody throw them in a room and lock the door and see what comes out or is there a smarter way to think about, you know gently pushing these people in the right direction?

And patience don't want it takes like it take it's it's take it takes a long time to take it. It's taking us quite some time. To to get to this mode by first of all we had to shift our product.

Into the cloud and as long as long with Shifting the product into a multiten product from a legacy Singleton product. We also had to build this new organization structure. You had to get firmed up on on all the devops elements.

How do we make sure that works perfect going to to you know continuous delivery. And so that's taking quite some years to be honest. And I think I don't think there is a Yeah, I need in my basic assumption.

Is that everybody? Everybody want to be successful and everyone would like to be able to deliver their their work to customers. And then and then you you sort of put the autonomy into the teams, but then you then I might recommendation is to put some covenants of function around it.

So that made sure that you call on the quality to go on the sort of the project how it runs you go on the the pipelines and the element of deployment and you go on the architecture, but but they they are then extern from the development teams themselves. So they act as common and functions, right? And and I think there's a lot that companies have set it up this way.

So you sort of have it you have a quality governance function, but you have quality intestines in the teams have architecture. That's make sure you build it the right way and they're not in the teams. They so sit around the teams making sure it goes on track and you have the product people guiding the projects through the different stages of the lives and depending on how long your Sprints and everything's are right that that's at least how we did it and and it worked and And so that's the only reference I got it.

That's what I would do it again the same way to be honest. How do you maintain that once you achieve it? Because you talk to a lot of people and they think of it as a almost like an event rather than something that has to be maintained on an ongoing basis and I think that you know, you're never quite done when you're on devops because every day is a cultural cohesion conversation and some level And given to maintain it, how do you mean it's not an event?

Right? It's a it's a journey, you know, it's a so as I said, it's taking us years to get to this and you're right. It's never going to end.

You can always be fine it. But I think that at the point when you were at the point where you have all your services. Clearing the pipelines and you have all the observability in there.

So you know that everybody thinks about how do you run it and you have the you you can see on on your locks that Services Health like self heal essentially if something goes wrong, so engineering thought about okay if my service go down, how do I get a robin running and you can see that there is And you move to a you know, it's probably it's sort of like immutable deployment as well. Right? I think that's key.

A lot of we didn't even touch on that topic, but I think that's something people are getting as well. When you see all these things you see all these different threads coming together then I think it's I think it's it's becomes business as you should that's how we do things right? I don't think it's gonna go away again.

Why would it go away again? I don't think so because that's how you work. It's changing from The change from building an MSI installer and and distributed on a CD or dvd is your old Florida disk and to the fact that there's only one version wanting anything given time in one production environment and it needs to run 24/7 and never go down when you when you made that Journey.

I don't think it's gonna go. I haven't seen any indication that stops acting that way at all right, not at all. That's how we work.

So you mentioned immutable platforms. We're talking a lot lately about software Supply chains and the Integrity of those Supply chains. And how do we maintain that?

What do you think will be the ultimate outcome of that conversation as it applies to devops? Will we do we need to tweak those processes somehow or other to achieve that goal or are we just there and it's unevenly distributed in most people don't realize how to actually do it. I don't think I think from the you understand the question correct thing if you if you think about the I'm not sure completely correct question, but I think the if the immune immutability and and your deployment runs as a mutable and you only roll forward then nobody's gonna nobody's gonna notice that you are patching or anything like that.

That's the point right? I don't think and if you build your processes or you need to change some of your processes to allow for this right because The if depending on your service the size of what you're trying to deploy some of this is is you know, there's macro Services Microsoft microservices and Nano Services. We've got big Services then it can be hard to sort of always come up with a complete service every time so you need to adjust your processes to to provide minor upgrade and you need to sort of refine those stuff that you can deliver and more continuously throughout whatever period of time you want to do that including patches.

But and but if you only allow for running forward and it's always immutable, then you'll need to adjust your processes to accommodate for that. So, for example, we have specific times a week where you can get a hot fix in your specific times a month, but it can get a minor release in your specific times a year. You roll out a major release.

And and that just because so that just becomes the way you operate and your processes get adjusted to this. Yes. So you need to adjust because you can't just go in and remove one the yellow or whatever jafile depending on the technology and put another one and that's not gonna apply right?

That's not gonna find immuneable system. Oh in infrastructure as code everything is immutable and you need to order your processes to follow it and it means you have some gates in terms of what you want to accomplish. So that's the way we think about it.

So there's we we can we set up such that. They're for for depending on the services the size. We always we always rule the whole service if it's a small service.

So with if there's a pet it's a new service we roll in if it's a lot of service we can break it into major minor and patches patches being can be rolled out twice and might not be rolled out when needed and and major every quarters roll down this engine. All right. Hey boss.

Thanks for being on the show clearly devops is a state of mind not a platform and we need to go from there. But thanks for sharing your insights. Thanks for having me with my pleasure.

All right back to you guys in the studio.