Tracey Jaquith – Simple Cluster Deployments Using Nomad and Gitlab CI/CD

Our next speaker is Tracey Jaquith. Tracey is an operations leader at the Internet Archive. org to look at a website and see what it looked like at a certain time.

I'll just say it's been a lot. org Was using GitLab, I was very excited and hearing Tracey's story. She has a lot of great technical details to share about using Nomad as a deploy target and some plug and play GitLab CI YAML for you to check out yourself.

So Tracey will share all the links and you all can check it out and catch up in the chat. Welcome to GitLab Commit 2020, thanks for joining my talk. It's called "Simple CI CD with GitLab and Nomad" and my name is Tracey Jaquith.

For a few of these intro lives will be a little bitly link at the bottom in case you want to follow along. And it's just a Web page so you can open it and it'll just open up with slides and you can navigate with the arrow. OK, so a little more about me, I work at the Internet Archive, I've been there since the start, so it's about twenty four years now.

My title is TV Architect, although I also work heavily in video and audio as well as Devops and do a lot of JavaScript now. And the repo that we're going to talk about and use here and refer to from time to time is on GitLab. com/internetarchive/Nomad.

It's linked below. , like whether you like it or not, our conversation will probably start talking about containers, if you're friends or colleagues. It's probably great professionally, not so great for my friends, but I love containers.

But we don't talk about this kind of containers. That never happens. OK, so why are we looking at Nomad, which is part of the Hashi Stack versus Kubernetes, which is very nicely integrated and with GitLab?

So I thought I would discuss some of the the applications that you install from the Kubernetes side, which are the GitLab side, which are super easy. So once you've got a Kubernetes cluster up and tied in , you just usually press one button, you get your ingress, one button, you get your cert manager for GPS and one button. So you get your runner for running your CI CD.

For each of them, for all the repos that deploy this way, you'll get what's called what they call a review app. So you'll get a Web app that's unique for every single branch and for every single repo. Now in the Hashi Stack side, instead of an ingress or the Kubernetes ingress or Engine X, they're using a load balancer called Fabio, and that's the one that they most typically recommend.

To keep things simple, I'm actually doing domain based let's encrypt scripts that I pass into Fabio and I'm just going to manually redo those every 90 days just to keep things simple. And there's some scripts to help you with that if you're interested. I install a GitLab runner, just like with the other stack that Kubernetes stack as a on one of the nodes in the Nomad cluster.

So I can do CI CD. And in the end, each of them are going to make if you have just the same same URLs, same kind of names and addresses. So who is this talk for?

Most likely it doesn't have to be all of these, but probably some of these where you're probably already familiar with GitLab CI CD or you've used it or are interested in. You might do some on premise Kubernetes clusters or you might maintain your own clusters or you might have had some experience or poke around with that. And if you have, you've probably noticed that Kubernetes is great, but it's kind of a big a big thing and it takes a fair amount to kind of get your head around and to maintain.

So you might be wondering, is there something smaller like what else would there be out there? And that's what this talk is going to be about. And you might feel like maybe you could simplify your deployments and not have to go through Helm and all that kind of stuff.

So the approach and goal that I was taking from this after a couple of years with GitLab auto Devops was I love it so and our devs love it so can we just make the swap out with Kubernetes with Nomad, be super easy and as easy as possible. And it turns out we can just use GitLab great build phase in their chart to default charts and we can use their testing phases and some of the other phases if we want. And all we have to do is customize the deploy the deploy phase.

So it deploys to the other cluster. You might also be wondering why nomad, Tracey? I haven't really heard much about it.

Well, I work at the Internet Archive. I'm their name, Devops person, and we had a bad month. And I think it was March or April.

18, which had come out fairly recently. I couldn't get it to start with our networking installed, but I couldn't get the manager integration to work, which meant we didn't have https. In addition, we were running kube ADM with a Kubernetes cluster of v114, which turns out it had a one year cert expiry which just expired without warning, without any kind of email or anything.

So I was completely cut off from the cluster, as was all of my devs. And that was a week of panic and just flailing and keys. Eventually got it pretty close to back, but in the end it was it was burning a cluster and remaking it.

We're making it possible seeing a fair amount of consistent volume rook stuff, using rook and ceph corruptions. It could be me in the way that I'm installing it, but I've just been having a bunch of problems with it. So we thought it might make more sense to since it's mostly databases and retests and things like that kind of important data and sentry, we would instead use local storage so we wouldn't have to worry about bit soup when you come up with disks that have fragged files and things in them.

Also GitLab at the time, which has now been fixed and V 13 had some workarounds if you were using newer Kubernetes stacks and it works, but it was kind of like I had to put it in for every single repo and it was getting to be a little bit much, so I went looking. And as you know, no job is ever worth your sanity and software developers and Devops maybe we need to remember that. I like little side eye I here.

org, it seems our ops team has a little bit of a preference for it over Kubernetes. They already knew Console, which is one of the big pieces in this little stack. They're very attracted to some of these deploy script alternatives.

So you can deploy scripts, just not docker containers, only where you do the exact driver or the raw exact driver. They have different trade offs. But you could do like krums, you could do scripts that run maintenance stuff.

If you have ever if you've ever tried making a Kubernetes custom resource definition or like tried stumbling down that path as I did, it's kind of hard and seems like a lot. So the idea that you can just run a little script seems very attractive. And then finally, if you're a small place or maybe you've got like a small subset or your small subset in a larger group, smaller tools are often a little bit of a better fit.

It seems like. So what's the Hashi Stack? So the Hashistack In a nutshell is three main tools at a high level.

So Nomad, which will take care of your deloyments; Console, which will take care of your networking; and Vault, which will take care of your secrets. And a little more depth to what I said, so for Nomad, you'll be doing deployments, you'll be doing container rollouts, you'll be doing canary. If you're doing canary, you'll be doing rollbacks, all that kind of stuff.

You'll be doing restarts and scheduling. So they'll figure out where to pack things it can do Nomad can schedule a million containers in five minutes in redis. Apparently, I should stop moving my hands and then finally, Vault will do your secrets like normal.

So kind of like your Kubernetes secrets. OK, sorry I skipped Console. Console will do service, discovery and communication as well as configuration.

So once you do a deployment of a new repo or of a new branch or something like that, your load balancer will instantly get have it discovered and we will be able to reroute traffic right to it. So that's nice and simple. So what is your what is your stack look like now?

Well, probably it looks like most people's web apps, right? It's got you've got a browser that's probably going to talk https. It should be talking https, people.

It's 2020. To your Web apps. It's probably going through some sort of a load balancer.

And ultimately, it's going to be talking to some sort of Web app somewhere, which will probably be talking to a database or Redis or sentry or something like that, not essentially some other Cassandra or something like that. So how does the Hashistack slip into this? Well, it's pretty, pretty straightforward.

So you'd have your GitLab CI CD on the right side of the slide here, doing a normal deploy pushout or a normal a normal commit push and then full CI CD pipeline. Or maybe you're doing some administrative kind of work or something. You're probably going to talking to the Nomad Demon.

The Nomad Demon in turn, can talk back and forth to the vault demon, get secrets and get some other information out of there if you need it there. And then it will talk back and forth to your console demon to make sure you've got networking between your jobs and your tasks and things like that. The console demon can also talk to your load balancer to make sure the service discovery is all set up.

And ultimately, once everything's set up and your job is templatized and expanded, your Nomad demon will then talk to your Docker demon, depending on which node or nodes that's running on, which will then fire off your task, your task group is basically like your your your little webapp. It could be like a node app or it could be a PHP little stack and then you can have some sort of a database or backing store. And this is very much like this is very much like a pod.

So this is a task group where that's one task and that's another task. And I think of this as a task group. And then finally having a little arrow going from here to here, Docker demon into the fabio load balancer, because Fabio actually happens to just be another deployment.

So it's another Docker container that is deployed to this whole system. So that just makes things nice and easy to set up and run. So, again, our approaches and goals here make Nomad an easy slot.

We want to use the the build phase because it's really sweet. It works great and does all the things you need to do. com, super easy and simple.

So stick with that. And then the custom to deploy phase. What we need is we need a project CL file, which is kind of like a merge between YAML and JSON and we're going to set it up dynamically, which is sort of an idea I had instead of making a project HCL per repo, because that would be a lot if you have a lot of repos like we do and it just gets to be a lot of boilerplate.

So we're going to use something called Lavon's, which is a simple ability to swap out variables and do some basic, like, if else kind of logic. And we're going to merge the normal GitLab CI CD variables with any other environment variables and anything from the vault that came in as secrets, as variables. And then we can merge all of those together with this Project HCL template, which we're going to go over.

And that just makes a project excel for your deployment and then and deploys. So the goal is every time you deploy and every time you do get commit and I get pushed, just update your. " Yes, you actually could get out to talk to you right now.

So two lines if you really want, just the bare minimum. You can include this and you're pretty much going to be good to go. We'll go over this.

So it's not just magic magic soup. But this is talking about that same Internet archive GitLab deploy. So you can see all the code right there.

We're pretty public with a lot of our code and it's going to load this custom CI CD YAML file. So you just add that to the top level of your repo in a new GitLab CI CD YAML file those two lines and you should be good to go. The only other thing you need is you need a way to talk to that cluster.

So we have it set up. So there's a nomad ADDR and a nomad token. If you put those in each GitLab repositories settings, you could do a group repository, you could do a group GitLab setting as well.

That's really convenient. We're doing that. You just make you go to the settings and then you find the CI CD and then you set the variables.

So you set your nomad Addr to whatever the Nomad cluster name is and then usually four, six, four, six. And then the way that you get access to it is you get this little token, such real simple. It's not like a kube config.

And you just punch that in as another variable and you're good to go. So you might be asking what's in that magic CI GitLab CI yaml file? Well, here it is.

So we're able to scroll down to the bottom. I guess I didn't do the include. I don't have to because we're doing the GitLab CI CD standard thing.

So I'm going to just I'm just sticking with build test, deploy and clean up to keep it pretty simple standard build a standard test if you're using it customized deploy and lightly customized clean up. Clean up is when you're your maybe you merge your branch, you delete a branch and that just removes resources from the clusters. Same as with Kubernetes.

So, so that's setting up the stock build from GitLab right there. As you can see, there's some more documentation there. If you want to see that, then I'm going to set up a few different variables.

We can get into more of that later. That's just to make things a little bit easier and cleaner in the templates and then further deploy and actually. Going and grabbing the generic project template project, nomad template, so it's been it's highly variable sized, if you will, and they do this cute little trick here where I pull out all the CI CD variables from your normal build shell and I stuff it into adjacent files.

That just makes it really easy for Lavant to take over. Lavant takes over from that little JSON file, takes your generic project template and makes a specific project HCL just for you. It's the file.

And then it goes and does what you would normally do with Nomad, which is nomad validate. So make sure that validation of your project is going to work. It plans it figures out if it's going to land somewhere and then it goes ahead and runs and delpoys it.

This is just kind of pretty typical to figure out which cases you're doing deployments in which cases you're not. And then this is like just slightly modified to make sure that when you get down to removing or deleting a branch, we just might stop. And that just gets rid of your resources to pretty straightforward.

So what does a normal project look like in general? Well, this is the high level sort of brush contours, if you will, of it. So there's going to be a little job stanza in this kind of yaml json kind of merge I was talking about.

There's going to be a group. So if you just have a single container, it's going to be a single task in a single group. If you had like a second database, you'd have a second task in the group.

It's usually pretty common to do something, something something the same label. And then I'm just showing this is like the simplest way to get to your your CI CD variables from your Docker image. We're going to do a little bit more than that from GitLab, but that's the rough brush strokes.

So let's look at what's the minimal thing. Let's say you don't want to use my repo or anything like it. What's the minimum you could get away with?

Well, it takes a little bit of time to figure out, but it's not too hard. So if we just stuck with standard classic CI CD variables, right from GitLab, we can make a nice looking job name I'm going to refer to as a slug later, but it gives your project, which is your group and path sluggard. So it'd be like InternetArchive/ Nomad or - Nomad in our case and then whatever your name is here.

So that just sort of templatizes out pretty nicely. Again, we're using the same image down here, starting with the registry and then using similar slugs kind of stuff and then whatever. Commit.

com or if you're hosting yourself, you're on GitLab instances, so we just go ahead and use that and then we can just go ahead and deploy it. And I'm just doing a really pretty standard kind of setup here. So keep it keep it simple.

And then this tag thing is just for Fabio to load balancer to figure out service discovery and set up a name. And we're going to use http and we're going to check it every 10 seconds. OK, keep things a little simpler because I was a lot of variables.

I set up something called Nomad__slug, and that's just to set it off from Nomad_variable. So any time you see Nomad underscore, underscore, that's just specific to this this this talk and this, this repo and how I've been, how I set up things. So I'm set up with your again, your GitLab group, your project in your branch.

I omit the branch if its master just for to make a shorter slugging, and it's kind of redundant or main I guess main is probably a better name these days. So here's a couple examples. org as a repo, we might have a branch called Web Dev fix things.

So we'll come down here and we'll see that as the slug and that's going to get used pretty much everywhere to keep it real obvious as a one one mapping where things come from. And then here, because I was using a master or main branch. So services time machine, if you want to see kind of how a cluster looks like, you get all or something like that, this is kind of similar.

So this shows you all the different jobs. So Fabio Load Balancer, here's a couple group groups and then here's a couple of different projects. And then here's even a couple this would be three branches and deployments within one one group.

And so they're all running happy. That's lucky for us. It's great for us.

So if you want to see the full temple sized project, Nomad, it's linked in that last slide and it's linked in the repo. So it's easy to find this is what it looks like. So we're going to use this Nomad's slug, which we define earlier on in the GitLab CI CD YAML.

We're going to set it up for the group and the task name just to keep it simple. Again, same Docker image. We've got some repos using WebSocket with additional ports.

We need to get out, not just the https port. So if you have additional ports, I just made that real easy for you. That in the Lavant style variables, all of these guys, the hard bracket, hard bracket dot and then hard bracket, hard bracket, some control language, so it's pretty easy to understand.

I actually upgraded to it later and I was really delighted by it. If you're having problems with Docker login and I'll get to that later, there's a little stanza here that can use an alternate method to log in. You know, it's it's pretty, pretty straightforward, a little more fleshed out.

You can use a different if you didn't want to use http, you could use TCP. You can even use scripts for checking your your health checks. But the simplest one is just talk to port five thousand in your web app on and the slash repo.

If you're using additional parts we can check those as well. So there's some stuff like that. If you're using vault for secrets, there's some basic helper information here, but it's going to go ahead and talk to a little bit lower talk to or will have already actually talked to the vault.

And it can get to your secrets this way. And this is how you can set up a key value store secret. So if you wanted to set up some key and sometimes it's a little of late.

I have some NFS mountings if you're interested in that kind of stuff, postgress and mySQL and so on and so forth. So some other factors will be starting to wrap up here. All of this can be run without a Container Network interface.

So you don't have to run flannel, Calico, or weave if you don't want to. You can if you like. We found Docker login was occasionally flaky.

It might be if you're trying to redeploy something like a week or two later or something like that, I just find it sometimes it doesn't work quite as well. So I just had a little suggested thing. Go ahead and make it deploy token.

Give it the read registry. You can read about this more later. It's pretty easy to set up.

And then if you just set up a CI CD variable with your username and password, that'll automatically go right into your into your template, enter your deployment, and I haven't had any problems with that. I just set this up as group logins for monitoring and usage. I'll show the slides in a few seconds.

It's really easy to talk to your Nomad cluster. Use the exact same for six, four, six URL that we use before and then the repo will deny you until you enter your token and then you're in. So here's what it looks like.

We can start on data centers, all sorts of different stuff so we can look at the fabric load balancer and different running jobs. So let's go ahead and look at the Fabio one and we can see their concept of replicas is called allocation. So there's two of them running.

So I've got two running on two different nodes in the cluster and that just makes it nice and easy. If one of your load balancers goes down, the traffic can cut over to the other. And there's also this executor's you can assess in right from the gui.

The other thing that I really like is you can see the entire job definition here. So sort of what we expanded out the generic template to also what the system added in. You can go ahead and edit it right here and redeploy.

So it makes it really easy to try out stuff super, super nice and simple. I've got a little like aliases thing which is linked right in the repo. That makes it super easy and I'll kind of gloss over these little quickly.

But there's a one called nom-ssh, which uses your your current working directory, like let's say you've checked out a branch and you're sitting in it in the terminal. You just type nom-ssh and within a second to your ssh in the container because it looks at your git repo to figure out the information, then look up the allocation just ssh you in. Similar, nom-app will just open the URL right into your right to your Web app.

So whatever your URL is, you don't have to look it up. Nom-cp, This is great stuff. We use it in Kubernetes as well.

This will hot copy of a file right from your laptop. So if you're using like a vs code, you can, you can just hook in this little deployment and every time you hit save it will automatically copy right to your container so you can just hot reload it like that. You want to deal with a ten minute or twenty minute deployment if you have a mono repo.

Super game changing. Here's a bunch of things that you can customize these if these nom underscore underscore variables so you can add more ports, you can do as I ssh, sorry nfs if you want or something like that, different host names and so on and so forth. This is more complicated slide.

This is talking about a multi cluster setup. So we've got two virtual machines and you've got your your, your CI CD deployments that will talk to your nomad demons, which will talk to your docker demons. And this case, we've got a webapp running three copies of two virtual machines and your traffic comes into your load balancer, which is running on virtual machine one browser traffic over to it depending and so on and so forth.

So pretty straightforward. OK, so if you're using big monoliths and monorepos. socket, especially if you're on premise.

I highly recommend it because every time your docker builds starts up, it's like waking up on a blank canvas. There's no ability to cache. You've got to do your full docker and it's pulldowns using big four to 10 gig images that can get very weighty.

And and that'll save you a lot of time if you don't use it. If you're using git partial clones you want to take over using, yet that will save you a lot of space because you only need versions of files that pertain to the given branch. So if you're binaries checked in or something like that or tons and tons of versions, you get 20 years of history checked in, that'll save you a ton of space in your image.

if you're using if you got to git Branch or a git Check-Out inside your docker image that you run. This is how if you want to run your own git management strongly recommend to git detach and that will detach your head and make sure you're not you're not getting tangled up with anything like checked out or something like that. Then you can fetch from Origin, whatever your branch name or your commit is.

That's what this check out variable is. And I'll just go fetch it and then you just go ahead and check it out locally and then changes your your pointer to be the exact same pointer as your origin happened super fast within seconds, highly recommended. And if you don't know what I git detached head is and you want to Google for it, just be aware.

Ok, wrapping up a couple more helpful scripts. If you want to learn how to set up your own nomad cluster. It's not trivial, it's not hard, but it's not trivial to figure it out yourself.

You can look right at the repo. There's a link right here in the slides and in the Repo for set up shop. And if you want to manage your own https certs, like on a domain level, there's a nice little script there to help you with that, too.

OK, wrapping up final slide: Kubernetes is the many problems. Eight hundred pound gorilla. Hashistack is smaller, lighter, lighter.

It's a single binary for the entire tool or for each tool. So there be like a 50 or 100 mg tool for Nomad, one hundred mg tool for Console and then a script for your for your auto restart. It's real simple.

Real easy to remove, real easy to add. Kubernetes is used almost everywhere. So a huge familiarity, tons of people, whereas the Hashistack is much lesser known "also ran".

There's larger networking footprint with Kubernetes, which might get you in trouble or may not, depending. Hashistack is pretty minimal. If you're deploying a cloud, while Kubernetes is great for that, it's a one shot go.

Whereas if you're deploying on premise, Hashistack's probably going to be a little simpler. I mean, fifteen thousand Devops slack pretty spicy and really interesting, and the general consensus there and other people I've talked to is that Kubernetes is great if you've got a team of two to three full time Devops for teams of twenty, fifty, one hundred. Whereas if you've got fifty or less or so or maybe even more, probably one person, one person part time is going to work with Hashistack and just skipping the kind of redundant if you're if you want these Kubernetes, it's got the role based access control which is superior to the access control list in Hashistack, but you may not need all that.

So it's kind of up to you when they've got namespace abilities or you can turn those optional. Thank you so much for joining my talk. I know it's a little long.

That's not my cat, but it's adorable ina bee costume. So thank you for attending the slides. Right.

This Bitly link. I'll give you a few seconds here. And the repo is at GitLab/internet archive.

It's nomad right after that, it's linked right here. Easy to find. Thank you so much for being here.

Enjoy GitLab 2020. I'm sure it's going to be wonderful. I can't wait to see all the talks that everyone has to offer and I will be continuing in the chat room that I've been in to answer any questions until people are done with questions or comments.

So thanks again and happy GitLab 2020.