Robert Brennan – End-to-End Kubernetes Policy With Fairwinds Insights and OPA

All right. So today we're going to talk about end-to-end kubernetes policy with Fairwinds insights and opa. for those who don't know Opa is a validation language for validating kubernetes yaml as well as multitude of other things and Fairwinds insights is a platform for auditing kubernetes both in infrastructure as code as well as live clusters and as an admission controller, So, my name is Robert Brennan.

I'm VP of product development at Fairwinds. I had up our engineering team and the development efforts on Fairwinds insights our commercial platform for kubernetes validation. So Fairwinds mission is to really be a trusted partner for kubernetes security policy and governance.

Fairwinds helps customers ship faster with you know less cost and less risk. Um, we do that by providing a unified view between the development team the security team and the operations team and removing friction between those teams helping them communicate better in a way that simplifies complexity and helps helps folks just shift faster and more safely. So today, we're going to talk a little bit about what policy and kubernetes means and you know how it can be applied at the end of the talk.

We'll also do a Hands-On learning portion. There's a handful of requirements for this. So feel free to as I speak start installing these on your machine.

So why do we need policy and kubernetes? The big issue is that Engineers are mainly incentivized to get things done. They have a strong bias towards shipping quickly and shipping features and often the best practices around shipping those features get left behind.

So things like security efficiency and cost the reliability the scalability to maintainability of what they're shipping gets left behind, you know, the engineer, you know runs a coupe rectal deploy or a Helm deploy. They see it's working in production and they don't think through okay what's gonna happen when we get a bursted traffic what's going to happen when an attacker tries to access this thing, you know is this thing, you know asking for 10 CPUs and it only needs one that sort of thing as long as that's working the development team usually doesn't care and they will, you know, kind of forget about it from that point on I say this as a developer So policy allows us to specify a minimum set of standards that all of our deployments must adhere to so for instance. We could configure a policy that will make sure we're always scanning Docker images from vulnerabilities making sure that those Docker that our deployments are only using trusted containers trusted images and operating systems that those containers aren't running as root that they follow specific labeling schemes, like maybe we want to cost center code applied to each deployment things like that.

So there's there's a whole bunch of stuff you can do with policy to make sure that everybody who's deploying into your kubernetes environment is adhering to those minimum status best practices. So policy is code goes, you know one layer deeper here and make sure not only that you have some policy configured but that that policy is Version Control that changes to that policy are tracked over time Etc. So typically kind of like the first pass we'll see folks do at policy as they'll just write, you know a document in Google Docs or something like that saying this is what you need to do.

A lot of times people don't even read the document. Let alone actually follow it proactively when changes are made to that document people aren't aware Etc. So policy is code really starts to fix these problems.

It puts your policy into a git repo and it puts it. It articulates that policy in a way that can be read by a machine and enforced by a machine so that you know, it's not up to humans to read that policy. There's something checking either as a warning or as even like a full-on block.

It's checking all the deployments that are happening. All the infrastructure is code that's happening to make sure it's adhering to that policy. This makes it super easy to roll out new policies because those policies are now getting applied proactively rather than forcing individuals to like, you know, subscribe to those updates and listen for those updates.

And I make sure that all those changes are tracked and get so that you know, it's not just somebody editing that policy live in a Google doc. You know, they have to make a pull request that pull request has to get reviewed and then merged into broad. So having your policy tracked the same way as your normal code and infrastructure as code gets gets tracked is super helpful.

So policy and kubernetes here is the minimum deployment configuration that kubernetes has inside of the the official kubernetes docs. You can see we've specified what container we want to run the port that it's going to connect to some labels for it and it might be hard for you to spot right away. What what exactly is wrong with this deployment?

But there are a few things that you know are not specified here that are breaking according to the default settings. One is that it's allowed to run as root kubernetes by to follow allow your container to run his root. You'll have to specify an extra bit of yaml to just prevent it for being able to run his route.

There's also no CPU or memory settings here. Kubernetes is just gonna take a best guess as to how much memory and CPU to provision for this workload. You know if this workload is going to need, you know, one CPU to run and two gigabytes of memory kubernetes might try to schedule it on a node.

That only has a quarter of a CPU available and a quarter of a gig of memory available and it's gonna it's gonna just you know struggle because of that. There's also no Health probes available here. So we're not telling kubernetes, you know how to know if this deployment is alive and healthy.

So this deployment could crash at any point kubernetes isn't going to know to, you know, delete that pod and try to reschedule it. We also have an issue here where privilege escalation is allowed. This is a security issue again by default kubernetes will allow Google escalation.

You have to explicitly turn it off. The file system is also mutable here. Meaning the the application itself could modify the file system or an attacker could come in and modify the file system turning the file system to read only mode is a great way to you know, Tamp down and security and make sure that this pod is running a securely as possible.

What's interesting to note is that these are the default settings for kubernetes some of these like the security ones kubernetes sets them, you know, it makes them more permissive by default and they necessarily need to be just to make sure you're successful running your pod for the first time and it's important for you to you know, make sure you're tightening these security settings proactively the other ones like Health probes and memory and CPU settings or things that kubernetes just can't configure by default. They really requires some context on your end or how to configure the application. So it's important for you to kind of think through.

Okay, how is kubernetes gonna tell my application's healthy, you know how much memory and CPU do I anticipated actually needing in production? Cool, so, you know now that we've talked a little bit about what sorts of things you might want to enforce with policy and kubernetes. We can talk a little bit about where you know, what what tools you might use to actually enforce that policy.

So one one great solution is a project that we put out called Polaris Polaris comes with over 20 pre-built checks for some common common pitfalls when doing kubernetes configuration. So it will check that you're not running. Your containers is root that you're not running with a writable file system.

It'll check that your you've set your CPU and memory requests and limits and that you're defining those Health probes along with, you know, several other checks that are available. There are also ways to write custom checks and kubernetes Polaris allows this it can it comes with a format called Json schema, which is what kubernetes actually uses under the hood to validate resources that are being put into the cluster. Opa as well as a custom check syntax, that's super powerful and you can do things with oppa and Polaris that say like requiring labels on your deployments.

That might be something that's custom per organization. Maybe you have a cost center code that you want on all your deployments. You can restrict which Registries your deployments are pulling from.

You can check for specific environment variables. Maybe you will want to make sure that nobody is set an AWS access key and AWS secret key on any of your deployments and you can just any any organization specific requirements that might not be available out of the box with the tool like Polaris you can you can start to encode in this and policy is code. Cool.

So with Polaris, here's how you might Define a custom policy. Again. This uses Json schema, which is an open standard for creating schemas.

This is as good for the majority of use cases a Json schema is as good enough. It can do things like make sure that certain fields are present that they're set to certain values that they fall within certain ranges that strings match certain patterns stuff like that. So for example here we see a schema that says the number of replicas has to be at least two so making sure that we have, you know multiple replicas available for ha purposes.

Rego is another possibility here. This is a much more. In-depth syntax that's available to you here.

It's a full-fledged programming language. Not quite Turing complete, but it provides a lot more Syntax for doing some very special things. This this can take care of that long tail that that extra like, you know, 10% of use cases that Polaris might not cover but it has a very steep learning curve as a trade off here so much harder to learn but much more powerful.

If you have like really in-depth use cases where you need to start validating things. This particular example here just make sure that certain labels are are set. For instance.

We're looking for the gatekeeper and the food label to be present on any deployment that comes in. Well, so we're going to dive a little bit deeper into Rego here. Like I said Rego is a very complex language.

It is a little difficult to work with so we'll kind of go over the basics today. We're gonna start with just like how to to find a basic rule. This is what a rule a rule looks like you start with the rule name.

It's gonna have a return value in Brackets here in the square brackets and then within the curly brackets, we're gonna make several several statements. Anytime one of these statements evaluates to false. It's gonna stop evaluation and just you know, basically that that means the rule past the statement return false means the rule passes and any statement returns true.

It's going to move on to the next one and it's going to move through all the statements until it gets to its return value indicating a violation. So the return value is typically going to be a message saying like you need to set multiple replicas or you know, you're missing this label. Cool.

So inside of Fairwinds insights, this is a particular flavor of Rego or a particular flavor of opa where we provide certain types of inputs that you can select on and we expect a certain type of return value. So you can see we've created a rule here called No Evil namespace. It takes in this input of action item, which is provided by Fairwinds insights.

I'm sorry, it takes in an input of the kubernetes Manifest which is in this input variable and it expects a return value of action item which is which is how insights knows to to surface and alert. meta dot namespace is equal to evil that statement returns true. So we move on to the next statement which returns an action item saying kubernetes should be used for good not for evil.

99 category of security Etc. So if the namespace is evil, we're going to move through the entire Rule and return this action item if the namespace is not evil that first statement's going to return false and we're just going to move on to the next thing. Cool.

So now that we've learned a little bit about how to create policies. We'll talk about, you know, where do where to apply these policies? A lot of folks will just apply these policies in one place like cicd or Michigan controller.

And really we see that that's not really a complete solution. You really want to run it run all these policies in three different places. One is you want to run on your live cluster your live environments to see where these problems exist.

This is great when you're running when you're designing a new policy to see you know, what what resources are going to be affected by this policy. If you're running compliance reports for things like sock 2 or HIPAA, this is your source of Truth knowing okay in my live environment. What is or is not compliant with this policy and you can run scans on your live environment on a regular schedule.

So using a cronjob or something like that to make sure that any new violations that that turn up in that livement trigger an alert create a ticket Etc. Taking one step left. You also want to run these as intermission controller.

So basically serving as a bouncer into your live environment anything that is getting pushed into your live environment is gonna have to go through the admission controller. And it will if the Michigan controller sees or this doesn't pass the policy. It's going to stop that resource and reject it.

So this is a great way to start really enforcing those policies, you know before they get into your live environment. And then again taking another step left towards the developer. It's really good to run these policies in cicd at PR times.

So when the developer is, you know, changing some infrastructure as code you can run you can run your policies against those changes so that as the pr is getting made before it gets merged into the main branch. We're servicing up an alert so that they see in GitHub, you know, you're creating a new issue here. You've created employment that has, you know, zero replicas or one replica.

It needs to be at least two replicas before you're all after merge this and that really surfaces that feedback as early on in the process as possible so that you know, it really tightens that Loop and make sure developers are getting that feedback as early as possible. So when you're creating a new policy deploying that policy can be kind of disruptive, especially if you're enforcing something that maybe not everybody was compliant with to begin with. The best way to do it is to start by scanning live resources just understand what the impact might be.

So you might design a new policy that says replicas has to be more than one you could start by scanning all your live resources Z. Okay. Well how many deployments do we have that have only one replica maybe at zero and you're great and you can just deploy this this policy now more likely you've got 10 teams that already have a deployment out there with only a single replica.

So you'll want to go and alert each of those teams and say hey we're building a new policy. We notice you have this deployment and this cluster it only has one replica, please fix this so that we can deploy this new policy and start enforcing it. You may also start want to start enforcing it in a warn only mode meaning it's not going to block a pull request or to block an appointment.

It's going to you know, just service a warning for those teams. So they kind of see okay, what's coming down the line? What's you're ready?

And you feel like it's not going to disrupt any teams. You can start enforcing it and we'll see ICD and in Michigan control and you may want to restrict which namespaces or which clusters it runs into or it enforces on just to make sure you know, if there is a particular deployment that just needs one replica for now or whatever reason you can restrict it to not enforce in that particular namespace so that you can you know, go back and fix it later on without blocking that team. Also now that we've kind of talked through all the different pieces and play here.

Let's take a look at you know how this works. in a live environment So today we're going to use Fairwinds insights along with Opa to develop a custom policy which will enforce a policy where every single deployment will need a cost center code label on it. So we can take a look at what this policy looks like.

I'm going to move it to a new new policy. our friend fired like so you can take a look at this policy. We can see the policies called label required.

We are making sure that the labels contain a cost center code. 9. It's got this description remediation, and we're putting it in the reliability category.

Cool. Now I've already set a Fairwinds token in my environment. So we should be authenticated with insights.

So we're going to test this policy to make sure. That it actually passes here. We're gonna call insights CLI, which is a CLI that Fairwinds insights provides.

You can download it using the readme here. rego. And we're going to test it against this object.

net and you can see with this. Coop objects, which I'll pop open just so you can see it. It does not have it has a label Foo, but it does not have a cost center code label.

And you can see it did create this action item cost center code label is required. If I change that. And set past Center code.

And I run that same command should Now fail and say, you know zero action items were found. So that's what we want this if we have a vml that has a cost center code. We're not going to get any action items if it does have a cost, so it doesn't have a cost center code then we get an action item.

Let's move that again. So now that we've got this set up. We're going to sync our policy to insights.

So I'm going to call Insight CLI push opa. So I can see push succeeded it added this new policy our brand required label. And if I go to the insights UI.

We can see real quick. We now have this our brand required label policy. We can actually see it here in the UI.

This is the same code that I just uploaded. And now we can we can test it out. So if I go back to my console here.

I've already got a kind Foster created. So I'm going to do is I'm going to create a new namespace. Called our brand my name.

switch to that namespace and I'm going to so apply that failed deployment that we saw earlier that does not have a cost center code. Oh. Didn't quite get it right there.

Oh, you know what I didn't do. Is install our admission controller? So you can see it applied correctly there.

I'm going to delete it now. what I'm going to do is head over to insights and I'll show you all what I'm doing here. So this is our report Hub where we can choose which reports we want to install with insights.

I'm going to add opa. And I'm going to add our admission controller which will block or pass deployments depending on everything. Here are my values.

I need to set. And I'm just going to run a home and stop to get this into the cluster. one second horse and demo time things never work exactly as you expect.

Let's try that again. There it goes. So right now we're installing the insights agent which is going to include an admission controller which will block deployments if they don't meet the policy that we just uploaded to insights.

They may need to. Yeah, also going to install cert manager into this cluster. it's not already installed which is a dependency of the no.

Looks like it was already installed. But it's it's already rejecting things here. Based on my my needs.

So if I go back to apply and we try to apply this same. The same deployment that does not have a cost center code. We get a nice error message back saying costly cost center code label is required.

There's also some other issues with it like privilege escalation is allowed. So what I'm going to do is fix this up a little bit. I'm going to set a cost center code.

one two three four also going to the Oh, so now now that I have a cost center code on there. It lets it through just fine. So that's an example of we created a policy that says we need this this label applied.

If we again don't have that label it rejects it once we do add the label we get we get a successful deployment there. but again, this isn't where we necessarily want to catch these problems. Typically if you're deploying it's probably already passed the cicd process.

You've already made this change your infrastructure as code. So what we'll want to do is try and scan this in a cicd context in order to understand whether this this problem exists and it's going to trigger the mission controller again before we end up trying that deploy. aml.

This is a configuration for our cicd. our ci/cd Scripts you can see I've created our repository name. I'm just going to rename it for our purposes here.

our brand you can see I've got Polaris and trivia two other reports disabled, but we're going to run the open checks and it's going to look for manifest in this deploy folder. You can see I've got a deployment that does not work here. It doesn't have a cost center code and I've got an appointment that does work with the cost center code here.

So now I'm going to download our CI script. Just going to verify that it's checksum is correct for security looks good. Make it executable.

And I'm going to run it. Now. This is something you would typically do inside of a cicd pipeline like in circle CI.

But you know for for purposes for our purposes here just to kind of show it working. We're just going to run it on the command line. So it's pulling our Docker image for doing CI now.

starting to run getting some details from get and looks like we found two new action items. For the failure deployment. The reason there's there's two action items.

I just want to point out as I've uploaded this policy for cost center code twice. So we have one action item for both of the both of the duplicate policies here. If I were to edit this failure at my cost center code back.

or one two, three four I rerun the inside CI Scripts. We can see we have now two fixed action items. We fixed both those cost center code action items.

We can also see these results inside of the UI live here. So if we look at the UI we can see we have a deploy without cost center code at this commit. We have a couple action items that are saying we need a cost center code.

If I cut back over. I cut back over to the right screen. If I fix this deployment now, I have my cost center code.

I'm going to commit that. And it cost center code. I rerun the script for this new commit.

We can see I have two fixed action items now. I go back to the UI can refresh. And see I have this ad call center code commit that has no action items.

Coming back from this cost center code with action items again. We just did this on the command line here, but you would normally see this, you know running inside of a circle CI environment and we could surface checks like inside of a GitHub repo saying this this PR will pass or will not pass the insights checks based on your custom policies. Also, we saw the same policy running as in a Michigan controller as well as in cicd also worth noting that that policy can run on a live kubernetes cluster to see if there are things inside the cluster that do not meet that policy.

So again, we can take the same policy that we've written run it in the cluster in cicd and in Michigan control, which makes it super easy to say deploy a new policy over time. I think that's all I've got for you and hopefully that was a helpful introduction to how to manage policy as code and kubernetes.