Mario Kleinsasser – To Go Where No One Has Gone Before…

Our next speaker is Mario Kleinsasser and of course, I had to change my background for Mario because he's going to talk to us about how to go where no one has gone before. Mario is an operations leader who has taken his company through an entire transformation from early days of just adopting containers to very sophisticated CI CD deployments using Gitops and cloud native technology. I loved hearing Mario's story and I think you're going to enjoy it as well.

Remember to hop in the chat to add comments and questions and maybe share some of your story as you listen to Mario story and you find some similarities there. Hello and welcome to talk "To go where no one has gone before". My name is Mario and my team and I have been using containers for more than 10 years.

And before we started to use GitLab, CI CD pipelines and Gitops operation in general, life was much harder for us. Like many, we worked on weekends and on evenings to get stuff up and running and into production. Inside the IT department of the company, there are more than 60 developers working on more than one hundred and ten internal application software projects nowadays.

My team and I are responsible not only for the infrastructure of these Web applications. Moreover, we are also responsible for a lot of other services which are needed to power this infrastructure, like DNS, puppet, Debix and of course, GitLab. You might ask, why is it important for you to know?

Well, my team consists of only four people, including myself. So this is a story about a transition. More than 14 years ago, my colleagues and I had only one Web application to host.

Now, 14 years later, we have more than 3800 containers, more than 110 Web applications, more than 600 local instances to manage, which includes automatically to encrypt cert updates and much more other services. Without multiple transitions and without a deep sense of automation, this would be impossible to handle for just for four persons. Still asking about why?

You just don't want to work with a system that can fall apart at any time. Instead, you want a reliable system where you know what is going on. In the upcoming slides, I will show you what life was like then and what caused us to go there, to go where no one has gone before with GitLab and why we are going this way.

So we will follow the story using the analogy of a rocket launch from the preparation to the start over the reinvention to the point in time where we are now and what lessons we have learned along the way. So the picture on the right side shows it's pretty cool. It's a picture from the Saturn five first nature in the assembly at the NASA assembly facility.

Our oldest project in our GitLab database dates back to 2014, it has the project added to. Before the time of git we were using a central SVN installation, but this was a pain because we had to write our own management for it to assign permissions to repositories and so on. Back then said in the introduction, we were already using containers and in 2014 we had about 400 of them.

Based on what we said we said was great, but there were no batteries included. This means that we have to do all by ourselves. For example, we had to manually implement the process to deliver the software onto our servers.

And this was a very complex set up, which was hard to handle. So basically, it was a self-made CI CD process which consists of the following steps just to give you any clue about the complexity. At first there was a dev package from Java software.

This was then delivered by ftp on an ftp server. There it was picked up and sent to a server which took the package, the package, and repackaged it into an operating system package. And then there were 80 jobs created.

And after this whole process, hopefully every single thing installed. This toolchain was only partially automated and we had media breaks everywhere. As I told you before, ftp service, packaging, delivery and execution were handled on different systems.

This is why handling only four hundred containers and about approximately 20 Web application projects this way was not future proof. To explain it a little bit more, I made a simple comic to show what a CI CD process was back then. So this first picture shows on the left hand side, the developer team is on the right hand side and operations team between them, there was a ball and we just said, hey, as well as a ball or whatever, and we thought it would be a soccer ball.

But what do we got was a tennis ball? So should the ref, for example, throw the ball or whatever? But we thought, hmm, what's going on here?

So this would not be much of a problem for us because we can just we can just fix it there, but the result of it was then that we just sliced the tennis ball in pieces, sticking together together, and then we had just some kind of a soccer ball with yellow color on it. And finally, this one gets then deployed on a server. And the server thinks, eh I can sum it up.

It fits approximately. But there was always a difference between what the developer was the developer thought to be installed on the server and what was finally installed on it. This misunderstanding was one of the main reasons why we had to handle a service, several outages back then.

So therefore, we started the first transition to change the system. And to be honest with the rules of the system, we did this several times. But let's start with the first one.

The picture on the right side, on the right side shows the Saturn five stage awaiting the assembly in the facility back in 1967. And the first transition started by using Docker instead of open vset. This was possible for us because the network problems were solved in 2017 for us.

The other networks, networks of multiple problems for us because running Kubernetes on premises in 2017 wasn't really an option. The main reason for not to go is Kubernetes was Kubernetes were designed to be used by huge setups and or need a lot of additional work and had too much moving parts for us. So why we used and why we are still using Docker swarm on premises, because it's super simple to set up and with the built-in overlay network there is the secure and easy to use network CNI provider.

But today we are also using Kubernetes too. But we did this a step further with the use of Docker, we still had many debates, therefore we did a lot of prototyping about what could be a proper way to eliminate these manual steps. But we did not find a good practice on how to proceed.

Back then, what we learned was that we can do it with Docker to enable a better developer and operator experience that, enabling faster development and deployment cycles. But this would be only doable if we can eliminate the manual steps. So T minus zero, the ignition plan.

Together, for the first time, we started our dev and ops meetings. The reason for this meeting was that the developers were in a need to use some kind of automation for the bridge process. In this meeting.

The first idea was to use Jenkins, but thankfully GitLab already enabled GitLab CI CD with GitLab pipelines. So we decided to go with the GitLab runners and to use GitLab to automate the build processes for the developers. But in return, together, we also agreed that we will implement the CI CD process to so no CI without CD.

That was the main and the most important breakthrough because now we start to work together to make the transition real. And hip, hip hooray! On exactly this time in 2010 17, we ran our first pipeline and started our journey.

This was a really exciting moment for us. The first pipeline was canceled because we made some configuration mistakes and therefore pipelines never finished. This was due to a misconfiguration which would have lead to infinite pipelines.

So funnily enough, we say with the same situation happens another eight times. So and the run of the pipeline number nine was a step forward because we learned how to use the system if it was failing. And with the run number 14 in our GitLab system, we for the first time, we were able to run the first successful pipeline.

These first learning showed us that we also have to agree on a new failure. Failures will happen in operation or in development, but we agreed to not to blame each other. Instead, we will work to find the root cause to get.

So what you can see here now is how the first pipelines were working, so the first ones were canceled, then they were following some pipelines and finally, if we got the first pipeline passed successfully. So what happens after these first baby steps into GitLab CI CD ? Every successful system will grow by itself, like the picture on the right side shows for the Apollo missions two, three, four, five and six.

So we've got more pipelines, more jobs, more projects. Finally, more of everything, and to show you what happens within the first three months after the initial start, I will show you some numbers now. So from the first pipeline and top run in 2017, we rose to one thousand four hundred pipelines and 4000 jobs runs only three months later.

From our point of view, this growth was caused by the fact that, for example, a subtheme used the new processes and Terraform a new GitLab pipeline and another subthemes. So the teams who are using this pipeline save a huge amount of time, which in turn leads to a domino effect. But if you reach the end of the first resolution after a month, something might go wrong.

So, Houston, we have a problem. So this picture on the right size is really impressive because it shows from the Apollo 13 mission astronauts, engineers, developers, and they are all working to solve a problem. So it's a picture for us that really shows what we had to do at this point.

We had to change our work culture because we still had manual processes inside our GitLab project. For example, we are using multiple repositories and for the source code and for the deployment and configuration of the deployment itself. These repositories are hard to set up at scale.

If you do it manually, especially if you use GitLab CE like we do, you have to invent some new code at this point to hand over the pipeline runs between two repositories, the development and the deployment repositories, for example. So finally, exactly this new code was then implemented and established by our developers. Thank you.

What does this mean? So welcome to the GitLab meta game. Going meta means that we had to simplify our multi project multi-stage get project set ups.

This model repository was too complex to be handled by people who are not able to deep dive into the nitty gritty details. So, for example, we had to set up various GitLab CI CD variables manually to use the power of the automatic build and deploy capabilities of multiple projects. Furthermore, pipeline variables can be a subject of change over time.

For example, there can be the need to add new variables into the process. Also, functions of the GitLab stages can be extended or changed over time. Our solution to solve this was to create a GitLab project with pipelines, which pipelines are used to create GitLab project source projects and development projects for a product which already set up with the pipelines and various areas that are needed to drive these projects.

Therefore, the developer can just use to build and deploy pipeline within this automated created GitLab project without the need to know every small detail. A huge step forward because you have to care about the doers. This means that every member of a team is different.

You will have some people who are happy to be on the edge of development and other developers are doing the really important job of just keeping the system stable and reliable. It was we need to take care about this situation. And automation can be a helpful.

Can be helpful to keep both groups and edge people and doers together. Once again, he has a little comic about it. In the beginning of a transition, the doers and the edge people are close to each other, think about a rubber band around these groups of people.

The rubber band is elastic and it could be stretched until a certain point. As time moves on the doers cannot follow the pace of the people because they do is to care about every new cool and hot stuff gets into shape to be used by everyone in a reliable way. Therefore, the rubber band will be stretched more and more.

So if the edge people are proceeding too fast, the rubber band will be broken and the doers will be lost. In a real world, this means that the doers won't be there anymore in a worst case scenario to get this things up and running and to make the system stable and reliable. Ultimately, the reliability of the system will be lost and this will cause outages sooner or later.

This is why this is why it is important that you have an eye on both groups, the edge people and the doers. As shown in the last three slides, the system grows and therefore with some limitations of our initial design. About how we work, for example, we had to handle multiple projects, multiple stage pipelines and so on.

So we had to recreate everything like the picture on the right side with the Space X rocket, there are a lot of iterations that they have gone through to get to a new point. So we had to take care about the doers because it's important to not leave them behind. And therefore we had to simplify the pipeline usage.

And to do that, we need simple steps to set up the environment because of the growth of the system. We got more backhands. We got more more Docker swarm.

we got more GitLab projects. We got more load balancers. And back in 2018, GitLab Includes enabled us to put various pipeline templates into a single certain central place.

Without them, it would not be possible for us to manage to manage the still rising amount of GitLab projects. The use of GitLab Includes makes it easy for us to set up GitLab pipelines without copying the logic of the GitLab CI YAML into every single project so we have no duplication. Due to the use of GitLab Includes, we are able to provide changes to GitLab pipelines and stages without the need to refactor every single project if something changes or needs to be changed like the CI CD pipeline variables.

So this is a super benefit for us because currently we have already two thousand projects on premises GitLab and without Includes we would have been lost. All these changes enabled us to grow up once again, but with every grow up we have to change also the culture. And in our case, we then moved over to Gitops.

Gitops first means no changes without commit, no manual changes, no cheating the system by hacking into consoles. And for us it's the only way. So like the picture on the right side, there is a clear plan what's going on in the system, because we have to support multiple environments like dev, test, staging, education, production environments.

These environments are often represented by branches inside the deployer git repositories. And of course there are different branches for these environments inside the source code repositories. So any combination of branches and repositories are possible and needed because not every project is using the latest software stick.

There are also legacy applications which need a simpler and more complex system. Furthermore, we have different types of pipelines. For example, scheduled ones, minor ones, fully automatic pipelines.

This is all handled in our case, by centric templates and without Gitops, it wouldn't be manageable because basically in our environment, a deployment is normally triggered by a git tag. So this gives the developer and the operator the opportunity to push the latest source code or the latest configuration changes as often as they wish to see the works. The pipelines will only run if a git tag is pushed or created.

And now imagine that we are talking about several hundreds of projects in our case, we currently have someone there are 2000. So. Once more, there are different militaries in such a team and Gitops is a way how you can handle this, because there are different teams and not every team and not our not every people can take care about every single small detail.

And as said above, we have to support a very heterogeneous environment, which a lot of different mistakes. So this is why it is key for us. But Gitops is also a cultural change.

And therefore we also adopted some public non GitLab strategies, for example, that we are using. I can remember we started to just create issues for nearly everything. So this helps us immunize my team to have an overview about what is currently going on.

For example, we are doing it to get a clue about what is our everyday work and most importantly, what work is probably holding us back. This means, for example, unplanned work or work which is not tracked everywhere. From my point of view, I can only suggest to do something like that because it really helps the team to not only be more focused on the real world works, but also to be more creative.

If you're interested to read more about Kanban boards and the different types of works, I can just suggest that to read the book, The Phenix Project. The next thing that we started is to talk to each other. That's really important and for sure.

Always listen to the stories and learn from experiences and from others. And now, finally, you might still ask why we are doing all this stuff and what effect does all this have on us and the system? Well, within two more years, from 2018 to 2020, the number of pipelines in operation rose from around 4000 to more than 60000 per month.

And the number of jobs within this pipeline rose from approximately 15000 to a little more than one hundred thousand per month. So to sum it up, as a team, developers and operators, we have managed to achieve almost 10 times the throughput and then two years before and beside that, we are also now future proof. But beside all of these numbers, for me, still the most important points of all are my team and the people I'm allowed to work with.

So everything of the story, I thought you would not be possible without the people who have the passion and the motivation to do this transition and to avoid the system, therefore, it's an honor for me to show them on this slide today. These are the real heroes. These are the real heroes.

And therefore people are the real heroes when it comes to transitions. Without them, it would not be possible to go where no one has gone before. So thank you, team.

And finally, to close my talk: Attempt to automate wherever it's appropriate. I know it's hard to start with, but with every step you do, it gets easier and it helps you to get back more slack time and to be more creative. And then you can go where no one has gone before.

Thank you.