Dina Graves Portman & Nathen Harvey – Measuring DevOps Keynote

How can I measure my DevOps progress? Well, the next session will give you some ideas. I'm happy to introduce Nathan Harvey and Dina Graves Portman from Google.

Dana is a Google Cloud Developer Programs Engineer with a deep passion for mining, meaning out of data in her six years at Google. She served on all sides of the data world, including analytics, engineering, machine learning and consulting. Nathan Harvey is a Cloud Developer Advocate at Google, where he helps the community understand and apply DevOps and SRE practices in the cloud.

He is a part of the DevOps conferences global organizing committee and was technical reviewer for the Accelerate state of DevOps. Hello and welcome to the Measuring DevOps session, I am Dina Graves Portman. I am a Developer Programs Eengineer at Google.

My background is in data and I'm really excited to help bring good, actionable data insights to develop our operations. And we also have Nathan with us today. Thanks so much, Dina, everyone.

My name is Nathan Harvey. Thank you so much for having us here today at GitLab Commit. We're really excited to be here.

I myself am a Developer Advocate here at Google with a focus area on DevOps and SRE. So before we get started, I want to tell you a little bit of a story from my past. A few years back, I was working on a team who was responsible for building, delivering and running a retail platform.

This team was great. We had adopted all of the best agile practices. We were working on regular two week sprints and at the end of every two weeks we would deploy the changes out.

It was really great. My job, my specific job in in part was to click the deploy button to send that code off to production where it could fly and be free and serve our customers well on our team. We built a lot of ceremony around that moment when we pushed the button.

In fact, the night before I was to push the button. Every two weeks I would send out an email to our entire company, letting them know, hey, these are the changes that we're going to deploy tomorrow morning. Little did I know.

But as soon as that email was sent, the entire office got together and started an office pool. With this question in mind, when will we receive the email from Nathan that we're rolling back all of those changes? You see, while we were deploying on a pretty regular basis, we were also rolling back on a pretty regular basis.

And the reason for this was we had so many changes that we were lumping together and trying to push out. This caused a lot of problems for us, a lot of churn and a lot of rework. So we knew the best way to solve this was to deploy faster, to deploy more things.

But we didn't really know how our problem was not unique to us. Many of you probably have lived through this yourself. So this is a place where DevOps can really come in.

But when we start talking about DevOps, we have to recognize we have a problem. This problem was true in that organization where I was. It may even be true in the organization where you are today.

That problem is pernicious and it is that our incentives are not aligned. We turn to our developers and we say your job is to build and ship features as fast as possible. Your job is agility.

And then we turn to our operators and we say your job, your job operators, is to make sure that the platform is stable, that nothing ever breaks. Well, I've been the operator. I know how to run a stable platform.

It's actually quite easy, except no change. I am the Department of Know. You want to ship a change.

The answer is no. I need to keep the platform stable. Unfortunately, the divide between these incentives and the misalignment in these inc entives is not exclusively held between developers and operators.

There's a similar divide exists between the business and IT. In fact, in the business they may not even be able to tell the difference between developers and operators. It's all IT to them, much like from my perspective, it's just the business, finance, marketing, accounting, all.

They all kind of go together and blur in my head. So maybe we need to ask this question of just who is the business, what are they responsible for, what are they trying to do? But I would argue that's the wrong question to ask.

We shouldn't be asking who is the business? More importantly, why is the business? Why are we here?

What are we here to accomplish? Who are our customers that we're trying to serve? So DevOps is something that can help improve alignment so that software teams can deliver better outcomes.

But what exactly is DevOps? Well, apparently there are a lot of things that DevOps does not. It is proven a little harder to say what DevOps is.

There are tons of possible definitions. So here's a good one, one that I like. It's about organization and culture, about velocity and reliability, shared ownership.

But you know what they say a picture is worth a thousand words, an animated gif. It's worth a million. So let's have one of those DevOps is about communication and automation so that we can drive.

velocity and reliability, all in service of keeping our customers happy, delivering value to those users, more interesting than defining what DevOps is, is observing what DevOps can do. There's a research organization within Google Cloud called the DevOps Research and Assessment Team. They've used academic methods to go out and research the industry and figure out what can DevOps do now.

Do a joint Google cloud in December of 2018 and they've grown and expanded their work inside of Google Cloud. And here's one thing that they discovered. Software delivery and operational performance is key.

It's key to delivering that business value and keeping your customers happy. When DORA thinks about and looks at software delivery and operations, they really break it down into four key metrics. Those metrics are focused on software delivery and reliability or velocity and reliability.

We break those four key metrics down into lead time. That's the amount of time it takes from a code change to be committed to your repository until it lands in production. Your deployment frequency.

How frequent, how frequently are you pushing those changes out? Your change failure rate when you put your change, is it successful or do you need to roll that change back and then the time to restore? Look, there are always going to be incidents and outages.

How quickly can we recover from them is an important metric. When we think about our organization, software delivery and operations performance, they've also done some research into availability. But that's an SRE talk.

Maybe for another time. We're going to focus on these four key metrics today. One of the really interesting things, though, that DORA has uncovered, they've taken this research and been able to break organizations into four different tiers, low performers, medium performers, high performers and elite performers.

And it turns out that when comparing the elite performers to the low performers, you get staggering numbers. Two hundred and eight times more frequent deployments from those elite performers, and they recover from errors or incidents twenty six hundred times faster. It is traditional to think about velocity and reliability is things that we have to trade off.

We can have one or the other. But the research shows that elite performers get to say yes and yes. We get to have velocity and reliability.

They go together like, well, if you measure velocity in terms of the interval from when code is committed to when it's deployed and prod, one of the key ways we can accomplish getting better at this is by shipping smaller and smaller features, little increments of code continuously moving through the pipeline. This not only allows us to move faster, but it's also safer because we're making smaller changes each time in the production environment. And you know what?

It goes a little bit deeper than this. Think about the human impact on this. Your developers get greater productivity and that aligns with happy engineers.

Better job satisfaction in the research again, bears this out. When we practice this idea of DevOps, when we are shipping frequently and safely, our engineers show up to work excited about the work that they get to do. They have higher job satisfaction and lower rates of burnout.

So back to those four key metrics. Let's dig in a little bit deeper on how to use the research to guide your own DevOps journey. So one of the things that I'd like to show you really quickly here, so let us take a look again a little bit deeper at the four key metrics.

com/DevOps is a DevOps quick check. Now, this is an assessment that you can use with your teams to figure out what is your software delivery and operations performance. com, you'll see this button here.

Take a quick check. Let's go ahead and walk through that really quickly. So we start by asking that question about lead time.

How long does it take from code commit to code deploy? And I'll use the same scenario that I described at the beginning of this talk where we were deploying once every two weeks. So we'll say once a week to once a month or sorry, that was our lead time.

From one week to one month, our deployment frequency was about the same once per week to once per month. It was about every two weeks. Our time to restore.

We did OK, you know, when we had an issue or an incident in production we were able to restore within the day. But here's where we weren't doing so well. As I described earlier, what was our change fail percentage?

We were rolling back changes at about 16 to 30 percent of the time and let's see, the last question here is about industry. We were in retail on e commerce. So let's go ahead and view our results.

We can actually see where we line up in terms of all of the participants of the survey. Over the years, we were medium performers. We felt like we were doing really well.

And in fact, we were we had made a ton of great progress, but there was still lots of opportunities for us to improve. So using this quick check can help you and your team identify where are some opportunities for you to improve? How do you stand up against the others within your industry?

But one of the challenges is where do we get this information from? And for that, I would like to hand it back to Dina, who's going to talk to us about some amazing work that she's done recently, helping teams identify and track the actual data around these four key metrics. Thank you, Nathan.

So I'm going to tell you a little bit about an open source project that we've been working on that we call it the Four Keys Project. And so far, Nathan has been talking about the research that DORA has done into these four key metrics. And the research has been done using survey data, which it's been done very rigorously, using very strong statistical methods of good research.

But I will say that surveys are subject to bias. It is very much it's a gut feeling of what you think is happening in your organization, and that can change based on whether you're a developer or whether you're VP, whether or not you just had a big failure. And so we want to get at the systems data.

We want to get at the truth as much as possible, especially because that will allow us to track changes over time. Much better as well. And the great thing is that you already have the data, all the tools and systems that you're doing.

You think they're generating all of these data points that we need to calculate these metrics. We just need to transform it. So when designing the Four Keys Project, our main challenge was that developers use many different kinds of tools, environments to do their work.

So how do we capture all of it? How do we account for unknown data sources? And the solution that we came up with was to create a generalized pipeline that takes in the events via web hooks ingests them and adjusts them into big query.

The known data sources are passed properly and any unknown source can be recategorize via a SQL script downstream. Data mapping is really the crux of the issue. This Four Key events raw table is in essence a DevOps log.

It should contain every event related to development, deployment and incident management when in doubt, stream it into the table. And from the Raw Events table, we create these three derived dimensional tables, our changes, our deployments and our incidents. When designing the Four Keys events raw table, we need to ensure that the schema is flexible enough to accept data from any data source about any event or object.

It should also be able to handle multiple events about a single object. And since this is coming through a web hook, it's entirely possible that the source might send the event multiple times. So we need to ensure that each event is unique.

There should be no duplicates in our table. So when we look at the schema, this is the data to remind you this is the data that's coming in through the web hook. And we do some very light transformation to get our data into this format.

If we look at one of our Web hook events from GitLab, we would call the source GitLab. The event type would be something like push or issue. Then we have the ID of the object and this will be used as the primary key in the derived downstream tables.

The metadata field is the entire body of the event. At this point, we don't know what's going to be in it and we just want as much data as possible funneling into our database. So any event, we get through that web hook, we just take that entire body, we stick it into that metadata field.

Time created is pretty self-explanatory, but signature is interesting because there can be multiple events on the same object. For example, an issue is opened and then closed. We need to have a unique key in this table.

So the signature is a hash of the event body and we use it to ensure that we don't have any duplicates in our table. DEPLOYMENT'S is the first of the derived tables. This table is updated daily via a big query scheduled to query, and the transformation is done entirely via SQL, making it really easy to update.

Each row should be a successful deployment and each deployment should have a list of all the changes associated with that deployment. The schema here is pretty straightforward and relatively self-evident. The interesting parts are that this table is materialized via the SQL script from the events raw table, so it's flexible enough to consider different business use cases.

Remember, this is supposed to be a list of the successful deployments. How does your business define successful deployment? Let's say it's when your software reaches the customer.

How many customers, 10 percent. 80 percent. And that might change over time.

So the list of changes is also important in this table because we'll need it to calculate all of the metrics later. Changes is the next arrive table. It's pretty straightforward query from the events raw table.

We just need a list of all the changes. So we have the change ID time created and change type, it could be you only want to include commits, maybe you want to include commits and merge requests, or maybe you want to also include configuration changes in the infrastructure. Since this is a derived table, you can adjust what is included in the changes table via the SQL script.

Incidents data can be difficult to collect. Ideally, we would use some kind of incident management system that has the ability to send the data to our web hook or to our pub subtopic. At the very least, we need to be able to to say which deployment's created an incident and when it was resolved.

For the sake of incidents, we will consider changes as the deployments that reached our production. All our calculations will be based off of deployments when it was created, when the new and when the new deployment or rollback was completed to resolve the situation. So once we have this data in our schema, we can start calculating our metrics.

Our first metric is deployment frequency, which is defined as how often does an organization successfully release to production? Deployment frequency is probably the easiest of the metrics to collect and calculate, as it requires only one table. If you're using mGitLab CI, you actually already have all of the data you need.

So we just need to stream this information into our table and then we can use it to calculate frequency. The DORA team has bucketed every metric into elite, high, medium and low performing teams. So in this case, daily deployment frequency falls into the high performance threshold and we represent it with this green badge.

Our next metric is the lead time to change and this is defined as the amount of time it takes a commit to get into production. So lead time to change requires two important pieces of data when the commit happened and when the deploy happened. This means that for every deployment we need to maintain a list of all the changes that were included in the deploy.

This is easily done by using triggers with a Shaw mapping back to the committee. With the list of changes in the deployed table, we can join back to the changes table to get the timestamps and calculate the median lead time. The dashboard puts the lead time and to the buckets defined by the DORA research.

Less than a day, less than a week, a month, a year, less than a week is medium. So we show it with a yellow badge here. Our next metric is the change fail rate.

This is defined as the percentage of deployments causing a failure in production to measure the change fail rate. We need to know two things. How many deployments were attempted and how many resulted in failures in production.

To get this number, we need our total count of deployments easily acquired from our deployments table, and we need to link it to incidents. Incidents could come from bugs or labels on our GitLab incidents form to a spreadsheet pipeline or an issue management system. The only requirement is that it contains the idea of the deployment so we can join these two tables together.

Finally, we have our time to restore. This is defined as how long does it take an organization to recover from a failure in production. To measure the time to restore, we need to know when the incident was created and when it was resolved.

The time it was created would be the time of the deployment, which included the failure. And then the resolution would be the deployment of the fix. Similar to the last metric, this data could come from any incident management system used.

So quick reminder, this is the structure we have all of our raw events, they are sending data to a web hook. The web hook is pushing the data into pub sub and then the pub sub and the pub sub workers are pushing that data into big query, which then creates our structured dimensional tables and that feeds into a dashboard. Let's look at our setup script.

So we're going to run up the setup script. It's going to ask us if we want to create a new project. We don't we do want to use this project.

We are using GItLab. We are not using GitHub and now it's just going to run away. And this actually takes like 10 to 15 minutes.

It enables all of your APIs. It creates and deploys the cloud run instances for your event handler and your ETL. It creates the big query tables.

It creates the pub subtopics, it creates the pub sub subscriptions, it creates the service accounts and sets the permissions properly. So once we have that, it asks us if we want to create a new project to test the deployments. We're saying no, because we're going to connect it to a real project and I am going to generate mock data.

You have that option as well. And the reason is because I want to see what it will look like in my dashboard. And when that's done, this data studio connector pops up, you choose your project.

Then once you click Connect, it's going to show you a list of fields. Then you're going to click create report. And then it creates a report from the template that we have already created that displays the four key metrics.

And there it is. We look like a pretty decent, high, medium performing organization. So let's start using real data.

That was mock data. Let's use real data. First thing we need to do is we need to set up our GitLab web hooks.

We need our URL from our event handler. So I'm going into the console here. And going into cloud run, looking for my event handler is taking a long time to load, so I'm just going to go through I'm clicking everything because I want all the data.

I want data about every single event, anything that happens in GitLab, I want it. I want all the data. So here's my URL and I also need a secret.

Remember, event handler? It's a it is a publicly available target. We want to protect it in some way, shape or form.

So we have a secret. So I did a little test push achievement there. It says it works.

If it really worked, it will show up in our events round table. So I'm going to do a quick little script here. I'm just doing select star from events wrong.

There's a lot of data, mostly GitLab mock. So we want to exclude that and just look at our real data. We have the one push event that we just created and the source is GitLab.

Excellent. OK, so now let's set up deployment's. Now we have our pushes or commits.

Let's set up deployments. I'm using GitLab CI. I've already set up my my service accounts and my permissions and I have this pipeline and it will trigger whenever I do a push to my default branch.

So let's do a push directly to my default branch which is not recommended, but we are going to do it so that we can see the data flowing through. It says, hey, your changes have been committed successfully and let's do another select start. So we have a new push ban exactly what we wanted to see.

And we have two pipeline events. So that's like the pipeline starting and then working and then completing. And that is how you set up our Four Key Metrics Project.

And once it set up, you can play with the SQL scripts and adjust the parameters as you like. And this is what the dashboard looks like around the corners. We have this kind of like running log display of what your performance has been recently.

I will point out there's there's no research behind these these charts. This is just so people can get kind of like an early warning, early indication of the direction that your your team is moving towards in the middle there, where it says one day daily, zero to fifty percent of those those little badges, that is the the door of research there. Those are the buckets.

And those look a 90 day performance. So it could take several weeks, maybe even a month to to change from one colored bucket to another, which is why having these early warning systems, these charts here is useful. So so Dina how is building out this integration between GitLab and big query?

How difficult was that? It was actually it was a joy. It was I don't know who designed the the Web Hook events for GitLab, but they are just wonderful.

I've done other integrations where each event has this giant object that looks entirely different. And so like my ETL is, if the event is this do and then you have a dozen of these and so like your file just goes like really long and unwieldy. This what we show on the screen right here, this is the entirety of my my GitLab ETL, it was so simple, this event object, everything, having this event type and object an ID and this time created timestamps that are consistent throughout.

It was wonderful. Oh, that's really nice to hear. I'm not surprised, but that is super awesome.

All of this data is really great and we have it easily accessible to us. We can set out some goals for ourselves. Now, you're not going to ingest this data and overnight become two hundred and eight times more frequent with your deployments, but you will start to get insights into how your team is performing and and understanding what can you do to start improving the metrics that your team is seeing.

But it takes work. And it's important to remember that we don't have to do this all at once. It's going to take time.

It's going to take real, sustained effort. As I said, we don't have to do it all at once. In fact, we will probably fail if we try to do it all at once.

Give yourself the time and space that it needs to go on this journey of continuous improvement. com/devops. And Dina, where can we find that Four Keys Project?

com/GoogleCloudPlatform/fourkeys/, you just clone it run the setup script and away to the races. Excellent, thank you so much. We want to thank you so much for having us here today at GitLab commit.

It it's really been our pleasure. And we look forward to chatting with you more.