Bobbi Wenzler & Nicole Shultz – Leading DevOps Transformation with Gitlab Data

If you can measure something, you can improve it, and nowhere is that more true than your DevOps journey, but where do you start. In this next session Bobby Wenzler, and Nicole Schultz will tell you exactly where Northwestern Mutual started and why. Let's check it out.

Hi, everyone, welcome to GitLab Commit. My name is Nicole Shultz, and I have my product manager, Bobby Wenzler, with me today and we are going to be presenting and how we are leading DevOps transformation with GitLab data at Northwestern Mutual. So for those of you who haven't heard of Northwestern Mutual, we are a financial services and planning company.

So everything you need from life insurance to investments to financial planning NM can help you with it. We have two office locations, one in Milwaukee, Wisconsin, and another one in Greenwich Village, New York. We have a fairly large tech organization with a wide variety of technology.

So if that sounds interesting to you, check us out online, find us on LinkedIn. We'd love to connect with you. So let's do some quick introductions.

So, like I said, I'm Nicole Schultz and I'm really passionate about helping application teams move the needle. In terms of CI/CD and DevOps, in my current role, I manage an amazing team of engineers and we create tools and capabilities to help mature and evangelize DevOps and CI/CD across the organization, as well as enable access and visibility to DevOps metrics. Bobby.

My name is Bobby Wenzler. I am the product manager for Nichole's team. I'm always up for a challenge and I really enjoy partnering with the engineering team and the clients that we have to bring solutions to life.

So I'm going to talk a little bit about our agenda today. First, we're going to cover our problem, then we're going to cover the industry research we did on that problem. Then we'll talk about the industry findings that we applied at Northwestern Mutual, talk about our solution, the CICD pipeline certification, and last talk about our outcomes and learnings.

So first, I wanted to talk about our problem, and that is there's so much data, but there's not a lot of visibility in action. So as you can see here, this is the standard pipeline that we have at Northwestern Mutual. It's got a lot of stages and jobs included, which is great.

You're going to get a lot of really fast feedback from your pipeline. But the challenge becomes, as you look at this screen, so you can see our problem is there's so much data and so much so much data and so little visibility and action. So from that one pipeline, you can see there's many different tools.

They output their information a little bit differently. It's really hard to tell how you're trending over time. It's hard for our organization to see: are we doing a good job?

Are we getting worse? Where are we going with our CI/CD and. DevOps best practices?

So as we looked at this, we knew we wanted to learn from what else was going on out in the industry. So we looked at this book called Accelerate and we had a lot of really great learnings. I wanted to share a few with you.

The first was there are a lot of gaps with maturity models. So when you look at maturity models, it's often a one size fits all, very static, never changing set of criteria that apply across the board. The other thing we learned is that quality automation gives you something great, you get speed and stability, you're not having to sacrifice one or the other to go faster.

You're not giving up stability to get stability. You're not going slower. It's really the power of having great automation in your pipelines.

Next, CI/CD and SCM are supporting capabilities for the DORA metrics, so this means if you want to really get the sustained DORA metrics in your organization, you've got to have some good source code. Management and CI/CD best practices. If you're not sure what the DORA metrics are.

There's another presentation at GitLab Commit Actually being done by Google would highly recommend you go listen to that. The last point was there's light weight change management process is really important. And what this means is they found studies showed that if you had an external team approving your changes, you actually didn't get better stability at all.

So we found this very intriguing and wanted to learn more about that as well. The next piece we looked at was the state of DevOps report, 2019. The first finding there is the best Keep getting better.

If you're falling behind, you're only going to keep falling behind. You really need to start making progress now to to be able to kind of catch up to what the industry is doing. The next is automate and integrate.

Again by having that robust pipeline. We saw that the automation for the high performers was way above the categories of all other performance groups. So it's really important to actually automate and build out things so you don't have to keep doing the same processes manually over and over.

Communities of practice. So like many big organizations, things are going to change over time. There might be reorgs and restructures, but if you've got a solid community of practice there to bolster and kind of build up and have a grassroots foundation for your CI/CD and SCM, you're going to keep having that sustained community out there.

Champion, champion, champion, try again. So I'll go back to automate and integrate. So it's important that you continue to automate and integrate and why this is the book showed that basically the teams that are the highest performers have the most integration and automation built into their pipelines.

The second is communities of practice. We want to make sure that we've got a good community of practice. So as things reorg and change in one company, there still are champions out there for CI/CD and DevOps.

That doesn't it doesn't go away if the people that were tied to it aren't there. Start with foundations. So this is really important.

There's a lot of great practices and learnings in both of these industry researches, but you don't want to do it all at once. You want to kind of break it off just like you would an agile and break it into small pieces and make progress over time. So then the next piece I'll talk about is how we applied the industry findings at Northwestern Mutual.

We're not ready for the DORA metrics yet. So what I mean by that is we want to really start with those SCM and CI/CD best practices and get those off the ground so teams have somewhere to start. If we were to tell teams you need to have more deployments right now, how would they know how to go about that?

We wanted to make sure we had the steps in place to really set them up to be successful. We wanted to address the maturity model gaps. So our criteria doesn't stay static.

It definitely also doesn't fit one size fits all models. We wanted to really make sure that it was flexible to support the that's the necessary tools and technologies that were in our company. And the last piece increase voice of customer.

So it's really important that people understand why you're doing this and what they stand to benefit from it and they feel like they're part of the process. We really thought we wanted to make sure people were involved. So it didn't feel like a mandate where you must do this.

It felt more like a partnership where everyone understood what they were going to get out of this tool. And next, I'm going to turn it over to Nicole to talk about our solution. Thanks, Bobbi.

So to address the gaps that Bobbie just covered, we developed something we call the CI/CD pipeline certification, which is essentially an automated process that evaluates a repository in GitLab and gives it a mature rating of how well it's doing in terms of predefined CI/CD criteria. So we actually started with a version one of this last year called The CI/CD Assessment, and we got really positive voice of the customer feedback on it. Application teams really liked it.

And at the time the assessment was running on all of the repositories we have in GitLab, which is a lot. And something that we learned is that in order to really support this and have this work, we need to create project types. So as we were looking and doing our industry research and developing kind of a new version of this assessment we had, we decided, you know what, we really need project types, that we can have specific criteria for specific project type.

So projects in GitLab, you know, they have different languages which are going to have specific needs. So, for example, a Docker repository, one of the criteria we have for that is that you need to have security scanning in your pipeline. However, for a database repository, that container scanning, that is not going to be there and there's going to be a different tools that are used for the database repositories.

So as we thinking about how to develop our certification One of the requirements you decided was that we need to be able to support project types with criteria that would apply to specific projects, as well as have criteria that we knew would apply to all project types. We decided to start with just evaluating AWS Docker projects. That's really because for our cloud platform that we have, we just have more standardizations in place and set up.

So it would be a lot easier for us to automatically grab all the data we needed to check each criteria using using just projects that have Docker. So our maturity, our CI/CD pipeline certification has three maturity levels bronze, silver and gold. So a little bit of gamification there.

Each one has a theme. So the theme for Bronze is Project and Pipeline basics. As you can see on the screen here, with all the criteria listed, we had your default branch protected.

You don't want to have people directly pushing to master. We want our source code to be tied to an asset for us. That's really how we do ownership.

So when we're tying source codes to assets, source code assets, then we can correctly know who or which team owns that repository in that code and should be the contact or any questions we have. All of the things on the screen here might seem really basic to you. But as Bobbi pointed out, I said in the intro, we have a really large tech organization and everyone's on in a different place in their DevOps journey.

Some teams that are more advanced, yes, these criteria are going to be done on day one. They're going to have them already when we rolled this out. We knew that.

But there are also teams that really are just beginning their CICD and DevOps journey and might not know where to start and might not have a lot of time or bandwidth to really think about what what they should be doing in terms of that. So we wanted to make sure that our certification in the bronze criteria were project basics, things that could apply to everyone across the board. Our theme for Silver is robust pipelines with fast feedback.

So this maturity level is really focused on adding in all the different types of scanning capabilities that we have. So listing code quality scanning. Security scanning.

We also have unit test coverage added in here, unit test coverage, as a whole is still something that's being adopted across our organization. So you can see for silver right now, we just have the percentage set at 20 percent or more. We definitely plan on looking at increasing that in the future and adding in more types of testing in silver.

We also have the fact that you're pushing artifact registries so that you can use it later for deployment and for production deployments. You are utilizing our change management integration. The theme for gold is addressing fast feedback and having automated builds and deployments for your branch.

So addressing fast feedback, that really means that your pipeline is going to stop when previous jobs fail. So the point is, if you have security scanning in your pipeline, that should stop, right? If it fails, it should be it should not be allowed to fail and have your pipeline keep keep progressing.

That should stop so that you address the fast feedback. You fix any vulnerabilities that are found. You push the fix, you rewrite your pipeline, and then it succeeds and pushes through into production.

You obviously definitely want to have automated builds and automated deployments, and that's both signs that a team is fairly advanced in the CI/CD space. You know, you're continuously building your change, you're continuously deploying. That most likely means you're hopefully deploying small iterative changes into production to minimize risk.

Another thing we have here is the test coverage being at 50 percent. Again, we needed to just start somewhere and that's something that we'll be talking about how we are. We're going to be looking to increase that in the future.

So future changes, I talked about a couple of them, one of them at the bottom, increasing unit test coverage. Right. So as teams progress and hopefully out of more and more unit tests, we want to definitely increase those percentages to be higher in silver and gold, to challenge people again to to get better.

We also would like to have a way to verify automatic rollbacks to see if a team has an automated roll back. That's definitely something that's a sign of maturity, that they have a way to actually roll back their application automatically. We would also like to add in a criteria saying that you should be using a Northwestern Mutual approved based Docker image.

That's really just because we have base images that we do nightly patching and security on. So we really want teams to be using those for all of their Docker builds. We also want to add additional testing types into the pipeline.

So right now we just have unit test because again, with all these criteria, we need to be able to get the data automatically. And right now unit test coverage was the easiest piece to get in an automated fashion. We definitely are working with our testing organization and partnering with them to understand what other types of testing can people use in the pipeline and how can we get that data automatically?

And even beyond that, what are the correct thresholds that should be set for silver and gold in terms of what good looks like for those types of testing? Lastly, we would like to create an exception process that allows teams, you know, to to override things and when they're doing a hotfix, so we all know that hotfixes come in production, things maybe have to go in right away and maybe you don't have the luxury of waiting for your entire pipeline to execute, or you need to allow us a certain step to fail just so that you can get the fix in in a timely manner. So we definitely want to figure out what that looks like.

Obviously had some sort of audit trail, et cetera, but something that customers have asked for that will be looking into. So I'm going to turn it back over to Bobby now, and she's going to be walking you through the UI of our application. So this is it.

This is the pipeline certification. And this is actually the report that will display for an individual project. So as you can see here, GitLab project is what is what the project that we're looking at.

You can see Bronze, we met all the criteria that Nicole talked us through. Silver is where it's going to start us automatically expanded because we've got some criteria left to do here. One thing that we talked about was the need for upcoming feedback.

So here you can see this last feedback line is going to give you a lead time to make sure that teams have some runway to actually get out there and do these changes. So this is a feature that our customers asked for. So one day if we asked for some improvement, they don't necessarily drop.

So this gets to the the point I made about our criteria aren't static. You can also see here clearly where things aren't met. So they still have some work to do on this particular criteria.

So that's what they need to do before their pipeline will go to silver. We also have gold, so it's not expanded again because right away we want teams to focus on silver and kind of iterate through those steps. On the left hand side, we give you a lot of information about the project where we certify it as you know, what the project is, some of those details Nicole is telling you about how to tie it back to people.

And then we also added in a few other really awesome features for our customers. On the first is this project opt out that button? So there could be teams who are sunsetting something in six months.

Do they want to spend time on that project or one they're touching every day? That's going to be around for a while. They might not.

So they can then opt that project out and say this project is going to be sunset, but I don't want to focus on it right now. We really wanted to empower teams to focus on what made sense for their organization and not necessarily dictate everything has to be done and let the teams focus on what metrics they wanted to set. The next button is recertify.

This is really important. And bronze. What you can't see all the settings.

A lot of those settings are in the GitLab project themselves. Teams really loved this pipeline assessment, but like their pipeline, they want a fast feedback. They did not want to wait overnight for things to run.

So this Recertify Project button will go out and get the latest and greatest GitLab settings and then update them to tell them where they are now. The next part of the pipeline certification is our dashboard, so out here we give the organization a rundown of how we're doing percentage wise and project count wise for the types of projects we're certifying right now. So if you're an executive, you could come out here and say for product A, I'm at this count gold this count silver, bronze.

My teams opted out this many. There's also another piece of this dashboard that shows you by assets as well. So for a particular asset, where is that asset?

held back from a CI/CD pipeline perspective. And then we also show trending. So over time, do we see gold going up like we expect?

Do we see silver going up those types of things as well? And now I'm going to turn it over to Nicole, you're probably really excited to hear more about how we actually did this from a technical perspective. Thanks, Bobbi.

So this is a high level architectural diagram on how we collect all of the necessary data for our CI/CD pipeline certification, as you can see on the left hand side. It all starts with GitLab. So shout out to GitLab for having such a great API and tool that allow us to get everything we need from the application, all the repositories in it.

So we utilize a system hook in GitLab. What a commit is triggered that triggers a system for us, which puts the name of the project into an SQSQ that We have that triggers or we'll get right by a lambda and we basically have a riepo. Metadara Lambda we call it, which collects metadata about the repository.

So lots of general information. It collects the default branch, the asset, some asset information, contributor's all kinds of different stuff, a lot of a lot of metadata information about the repository in general that we use. S.

topic and gets the name of the project. So you can see the first Lambda that runs the data, one that stores the data in a Dinamo database which is accessible by a DevOps API that we have. So when our certification lambda is running, we actually pull some of the metadata from the DevOps API to grab the asset information.

You know, like I said, the default project, we pull some of the settings like do you have your default branch protected? You have merge request pulled on. We pull some of the criteria information from the DevOps API and then the certification.

Lambda does a lot of the calculation on logic around you for all the criteria we have for bronze, silver, gold. Is this met true or false? Is a upcoming or not is it active.

And then it also will calculate the percentages that that Bobbi showed on the UI of like six of seven, what percentage is that, etc. for each maturity level. So after the lambdas completed it store the results in another database we have just for the certification and then that database is read by our certification API, which is our middle tier, and then that eventually is what feeds the UI, which Bobbi showed, which is our front end for the certification.

So let's talk about some outcomes and lessons learned. So one thing we learned, as I mentioned before, we kind of had this version one called the assessment, and that was collecting data on a nightly basis. So one thing that we that we learned was people don't want to wait overnight to see if the changes they made in the repository got them to Bronze Silver or gold.

So we definitely with the new architecture I just showed you, we're able to collect it in near real time for some of the data that we don't have set up real time right now, such as on the GitLab settings page. So when you go into GitLab your project and you go to the settings page and make some changes, right now we don't have a way to automatically trigger have that have those changes trigger our data collection process. So that's where the recertify button comes in that Bobbi talked about.

So for Bronze, specifically, when people are making changes on their settings, they can go in, make those changes and then go to our UI click recertify. And that will basically put the name of that project into the queue and run it through our entire process. As I said, if people are making changes in their GitLab CI file and their pipeline, that's going to trigger a commit, which will automatically trigger our process.

So the data collection from our side is much, much faster now and teams can get near real time data collection and feedback when they're making changes to meet a maturity level. As Bobbi talked about, people also really one of the opt out as teams started setting more OKRs or team goals around this. They wanted to say, hey, we want to get one hundred percent of our projects bronze.

OK, well, for these 10 projects, we're sunsetting them in six months. Doesn't make sense. So that's really where people wanted the future because they wanted to be able to hit team goal numbers without having to actually work on repositories where it doesn't make sense.

The other thing we heard was automation. So a lot of teams have a lot of repos. In some cases, teams might have one hundred or more.

So they don't necessarily want to go through all 100 repos and click four or five checkboxes to meet the bronze criteria. Right. To protect their branch, turn on Mars, all that kind of stuff.

So something we're actually working on right now is figuring out a way to how to automate that. So on our UI adding in a button or some options to say, hey, for this group, this these projects automate these criteria, we're starting with just a simple criteria. Right?

We're not going to be starting by making changes to GitLab CI pipeline files and stuff like that. But a lot of the bronze criteria, which are just updating settings on a GitLab project, we can automate a lot of that for the application teams. The other thing we heard was teams wanted time for improvements.

So and that really gets to the point that Bobbi was making, we have the upcoming criteria for teams can adjust and in general, the fact that we can show all this data helps teams give more of a view to their leadership to say, hey, we want to make changes, we want to get to bronze or silver. These are the benefits that it gets us and this is the steps we need to take to get there. So it actually helped them get more leadership buy in because we were showing them the data in a way that it was easy for them to show their leadership so they could get more time to make improvements on their side to to it to start using and adhering to some of these best practices.

Bobbi. Thanks, Nicole. So I'll talk a little bit about some of the outcomes that we saw.

So as you can see here, we had three to three to eight percent increases. And now projects that actually have CI/CD pipelines in those pipelines are actually including static application security scanning more often. Now, they're also leveraging artifact management tools that we have more often.

They're now requiring their pipeline has to run successfully before they merge more often. They're integrating our change management integration as well. They're reporting more unit test coverage now as part of their pipeline and they're requiring their merge requests or they're requiring that merge requests are needed before you can get them changes into your branch.

So, well, three to eight percent might not sound big. We wanted to to call out that we have somewhere in the neighborhood of twenty thousand projects. So it's not a small drop in the bucket to get this increase across the organization.

And also, it's important to note that all of this was done without a lot of outreach. Nicole and I just pretty much had some dashboards out there and told people they were out there. There was no mandate.

You know, teams really were doing grassroots adoption because they were so excited. They wanted to be bronze. They wanted to know when gold was coming.

One is silver coming? How do I how do I get there? So it's really important.

I think the other point I stress in an earlier slide to about voice of the customer really getting out there and understanding what our customers wanted and what problems they were seeing also really helped. You I think seeing this type of increase in six months is awesome. And I can't wait to see where where the next version of the pipeline certification is going to take us now that we've gotten even more feedback from our customers.

So thanks, everyone, for joining us today. If you have any questions, feel free to reach out to us, find us, connect with us, and we'd love to answer them. Thank you.