Stephen Giguere – Pwning the CI Workflow and How to Prevent It

Hey there, my name is Steve Jacob and welcome to poning. The CI that's not a misspelling. That is the word poning.

This is the GitHub actions Edition. My name is Steve Jacker. As I said, I work for Burridge crew and well while we're well introducing myself when I get into a little bit more detail.

I am a developer Advocate which means I talk about tech essentially. I've been writing code since the 19 or her. I've worked in quality automation.

I have been diving into cyber security for a variety of companies you can see but my current specializations are cicd pipelines kubernetes and containers primarily that kind of foggy world of devsecops. I also if you happen to find yourself in London, look me up London England. That is I run a meet up there and I also have a cloud native twitch show.

If you want to learn more about me there but enough about me more about how to pone your CI and how to prevent it most importantly. That's the big takeaway from this if I don't accomplish that I've got a problem. So first, I'm just gonna do a brief introduction to GitHub actions.

I'm sure a lot of you out there know what GitHub actions are so I'm gonna spend just a tiny bit amount of time on that just to be absolutely sure but also tap into what can go wrong with them. Then I'll introduce the Potential Threat actors both outside and inside and then the attack vectors that I'm going to apply today. Some of which you'll see live fingers crossed, right?

And then finally, we'll wrap up with some best practices and some prevention so that you can have the most secure ci/cd based on GitHub action workflows that you possibly can. Hopefully you'll learn some things in the process. All right, let's get down to it.

GitHub actions workflows are What well no point in me making this up I've taken this straight out of the GitHub documentation workflows are defined in a magical directory dot GitHub workflows. It's very special in one way and not in some other dangerous ways. In this repository you can have as many workflows as you would like workflows our yaml files with instructions on what to do with your code.

when certain triggers happen There's a few examples that are in the documentation. For example a workflow to build and test pull requests maybe that seems obvious a workflow to deploy your application maybe to a cloud service or maybe to kubernetes. A workflow that adds a label every time somebody submits a new issue maybe in best new issue or something like that.

And then I've added one that wasn't in the documentation, which I have seen frequently and that is the sanitization of an issue title or description to make sure if it's a certain standard for your organization and that's what I'm going to play with a little bit early on. now the fundamental problem we find with GitHub action workflows is threefold really GitHub can and often will run new workflows that are in the GitHub workflows path. And when I say that, I mean even the ones that you are committing as new they haven't been checked.

Now normally you want them to run to check other things not themselves. So there's a bit of a catch-22 or You know chicken or egg scenario going on there. Metadata, like issue name and description which come with issues get past into workflows and they're available.

Now the problem I've seen is that they're often used without checking them a very application security problem, but still very prevalent and get up action workflows where people who may be creating these workflows aren't familiar with the traditional absec problems. And finally that magic workflow directory. I mentioned shares the same access controls as the entire repo.

There's nothing really special about it. It's just another bit of code except code in this directory runs automatically. So it's a little bit more dangerous and we can do something about that and we'll talk about that near the end.

All right. So let's talk about a few attack vectors and I'm going to do my slightly different order and this is the order I'll be talking about them today now the idea of sanitization of an issue title or description. Well, say, for example, I'm not sanitizing that issue title in the right way before I'm using it.

I'm looking for something different. There's a potential because GitHub action workflows run a lot of their commands in a bash shell that I might be able to inject some commands and run some of my own let's say commands with malicious intent. The second one a workflow to build and test pull requests.

Well, if you're building and testing how much power does that have and Can I submit my own workflow? They can do I don't know any number of my own commands. That might be useful or can I use that GitHub token?

For other malicious purposes. It has been provisioned with maybe a little bit more power than it should. And finally the last one maybe one of the most dangerous a workflow to deploy your application.

Well, if it can deploy my application it probably has credentials to do that. Can I get those credentials are they stored as secrets? Are they hidden in environment variables?

Maybe I can and how bad would that be? All right, my quick attackers guide. Planning just try it.

And external attack just pick a repo and I'm not endorsing this. Let's be clear about that. You can't just pick a repo create a malicious workflow or just something that just to see and submit a pull request and see what happens now.

Thankfully this doesn't run right you can see over there the approven Run button on the right. This is what will come up to the owners of this repo because you've never submitted before they don't know who you are. You're a first-time contributor.

The problem with the approvement run in my humble opinion is that it's not glaring enough and how dangerous it can be to push that button. If you click on the learn more just next just to the right of that approval run. It's kind of shadowed out.

You get to the help that's just pasted above that's not normally there and you can see there is a warning that you should be alert to pose changes to the GitHub. workflows directory not exactly a claxon is it and in fact? If you don't have any workflows and you do a full request, it actually changes that section to hey, you're continuous Integrations and set up don't you want workflows to automatically catch bugs and force style sure really?

So it's very much encouraging you to always add work clothes and run workflows. So when it presents you with something saying are you sure your intention you're your tendency as a human? It's just like, oh boy, click the button so it can be dangerous.

The second option is to become a malicious Insider. Now, you could just submit a readme or you can make a small change or you could maybe go to one of the issues. It says good first issue.

And you could fix the issue you'd be a force for good. And then ideally you would be that elevated to become a collaborate on the project. It's important to be a collaborator because Well, actually it's important for security checks to have multiple collaborators from multiple countries involved in a an organization.

We'll see there's actually a safety check that makes sure that which kind of is interesting if you're trying to do it maliciously. Now just to decide this is a real story. This is a true story moment.

This is the bridge crew check-off build environment that I this is where I contribute a lot of my work. Somebody did exactly what I just said the external Insider attack method. They deleted our entire build and replaced it with a simple job that says.

Run a PS, which puts all the processes on the screen and run end which puts all the environment variables on the screen. What's interesting of course is that we run on self-hosted Runner, which means when these workflows run they run internally to us on our environment which is dangerous because who knows what else we use those self-hosted Runners for Perhaps Perhaps deploying to unaws environment. Maybe there's something dangerous there.

So it was a legitimate yet albeit rather weak hacking attempt. So these things do happen. All right.

We're getting into it now hack number one leverage issue grooming. What is issue grooming? And how real is the problem issue grooming is when someone creates an issue or you have workflows that manage the issue creation for you.

This is an example where I found I found searching GitHub or somebody was actually creating posts within a Blog that was hosted on GitHub based on issues that got created. Now it seems innocent enough, but the reality is no one's checking what that event issue title. The title of the issue really is so the reality is I could create an issue on this site.

That has HTML in it that says such an expired please log in again proceed to this form and I can start harvesting GitHub login credentials. I could if this is in check, so this is a little dangerous. The other example is when somebody's just innocently logging by a echo the issue title in body and other details just to the log just because it's handy to have that in the lock.

The alternative of course is this is the one I'm going to use. Checking issue title checking the issue title to see that it conforms to a sensible specification. Great, it looks very nice, but it can be abused if you see the assignment of GitHub event issue title in quotes to a variable running in Bash.

Well what happens if I do this what happens if I run a command and put it in a back tick and if you're familiar with bash, that means bash will see the back tick and before it does anything it will execute that and place the output of that execution in the quotes. dangerous well Is it let's let's try and now I get to introduce you to the demo environment. I've got here.

I have this innocent VM running in Google Cloud. That is Just sitting and waiting on a particular Port we'll get to that later. Here is my malicious Outsider now Insider.

This is a character called Loud Canadian looks a surprisingly like me who is looking at my demo environment. Here is the actual owner of the environment. Now in order to become a friendly Insider.

Loud Canadian has already submitted a few pull requests. We can see our closed. Updated the readme created an issue title check.

So this person is now trusted. They've got past that first contributor status. In fact, they've been added as a collaborator if we roll down we can see there's now a contributor here.

So now it's time for things to happen. Now. I've got my issue check and if we go into my workflows directory here, you can see my issue title check looks exactly like the one I just showed you.

Well what happens if I create an issue like I just said. Was it going to work? Well, let's let's see what happens in that particular scenario.

site. It's great for testing apis and curl commands and it's just sitting waiting on a particular address. Okay, just think about that.

So over here back to back to being me. I'm going to create an issue. I'm going to give it a kind of funny title.

Okay, I'm going to use that back tick I mentioned and no I'm not going to type it all in because I'm not crazy. I'm gonna put it back take I'm gonna say hey. Normal issue, don't worry.

I'm gonna submit that issue. Okay. So over here in my actions, I had that issue Grammy you could see it's there it is.

It's running it's cute up. And it's gonna check the title to make sure it's okay now amazingly that syntax I have the title is okay, and we could see it's in progress now and it's going to give me a thumbs up. But what actually happened?

Over here you see I've intercepted that environment that I typed out to a curl post and now I have all the environment variables for that Runner. GitHub and give them paths I can see where the code is stored. It's generally not good particularly if that were a self-hosted runner.

I just took advantage of something that I really shouldn't have and That lesson one do not do not mess with your your GitHub event issue titles always check to make sure that there isn't something code like running in there. It's important and there are ways to automate that we'll come to that later. Now, how can you protect environment variables?

There are in GitHub environment protection rules for both the environment various themselves, which can restrict the environment to only certain branches. and to certain workflows So this is the sort of environment. This is the sort of things we want to employ and there's a URL down there at the bottom which will be there again at the end.

You can screen capture this if you want, but you can wait till the end and all the URLs are reference will be there. Just check that out and you can make sure that you're never exposing environments to invite to branches or workflows that you that just have no business seeing that detail. So that's your first tip on securing your GitHub action.

If I had used these that would not have been so populated that result. Pack number two Auto proven content. This is an interesting one, which I'm going to not demo right now, but I'm going to add into a big finish.

Now before I get into it, I'm going to introduce Branch protection rules. You should always always have this turned on there's a lot of automation tools that look to make sure you have this turned on for important particularly production branches. and it by default is already useful even if you do nothing and just add it.

But one really critical tick box of the very beginning is requiring a pull request before merging and requiring approvals. Sounds amazing. The one default I would argue isn't perfect is that the number of approvals required here?

You can see a little tiny teeny you can easily miss it. It's just one. hmm Okay.

Well, it sounds like I've got some security when I add that but I wonder how robust that really is because defaults in my you know in my history of working security aren't always the best if you need to put a little bit of work into it. So what if I wanted to submit a pull request was a new workflow? I'm already collaborator.

I could do something like this. Now. This is an overall a useless workflow.

But what it's doing is fascinating because I'm taking advantage of the detail. I already have in a workflow like the GitHub repository the event number the secrets GitHub token. And I'm using this.

To use the GitHub API to Auto approve my own pull request. Is this possible it kind is if I'm a collaborator so that is kind of dangerous. Now.

I'm going to show you this but later on now, it's I'm gonna move ahead. and I'm going to add a caveat now that Discovery happened in March this year and almost immediately afterwards GitHub added new settings to the repository settings and if I go over here I can actually show you where those are. so if you look at actions General You go down to the bottom there.

They are you can see I've got read and write permissions and I have allow GitHub actions to create and approve pull requests. What why do I have that set? That's terrible, but that's what's there by default.

So the reality what we need to learn is we need to change that result to what this says right here. Just read repository contents from Mission only and that's your next tip that you should go in and make sure you change that because it's not the default. Otherwise that what I just would have tried to execute would enable an action to self-approve.

sounds scary doesn't it because it is Okay, I'll finish with this one hack three owning the runner. This is kind of the bigger one. This is why I've got that strange shell lurking in the background.

our default token for our workflow has a considerable amount of clout we can see that if I've in a forked repo like I was doing originally with my malicious attacker from the outside, but you can see now because I'm a collaborator. I'm operating directly on the repo and that elevates my capability of my GitHub token. To the point, where as a collaborator if we see what this default permissive access is it's pretty extensive.

So it's almost the case of what can't I do versus what can I do? And that's why that approved works. So it's a little bit funny.

So let's as one final move do a few things based on what I've got right now, but what I'm going to do first actually before I do that just to add some defenses that I just preached about. Is I'm going to go to branches. I want to put some defenses.

I'm gonna see I have no Branch protection rules. I am gonna add a rule. And I'm going to apply it to main so I'll put the the default defenses in requiring a pull request requiring approvals one approval and I'll leave the rest blank now, you should explore this.

Requiring signed commits is a great idea and most people should do it very few do require conversational. But these are all good tick boxes and then the opposite down here. Don't take these These are good defaults.

And this needs to match my main which it does so I'm going to create and now I have Branch protection enabled. Oh, okay fantastic. So I've got my defenses.

I preached her out it put in place will this attempt work? Well what I'm going to do, let me zoom in on the section down here. I'm doing something similar.

I did earlier. I'm going to submit a generic workflow. It's going to Echo all those Secrets like the GitHub token into a file and it's going to post those Secrets out to my scary bad URL.

Ooh. No, we saw it already worked for the environment variables. So I'm kind of figuring it is gonna work here.

What I'm doing afterwards, it's the same command the environment convertible one. And if I wanted to say keep that token if I got the the GitHub token or any other token that was maybe temporary and part of this workflow say I wanted to hold it. well, I could do something like adding a sleep.

That would give me some time to keep the workflows. And stasis in a way, so I had my GitHub token or whatever other Secrets I got for a longer amount of time. I mean the alternative there are alternatives to that and that is some kind of weird gobbly gook like what I've just read there and if you're unfamiliar with that, it's okay.

This is not a lesson on how to do that. But that's what's called a reverse shell what that does is it actually takes the bash shell that exists in the runner and actually sins it away. So it's like I'm logged in which is pretty powerful stuff and some people consider that to be kind of the keys the king.

So let's try this now and this is kind of dangerous. I don't recommend that at home, but I just want to show you what what can happen. So this little guy here this VM, I'm gonna hit enter here now and you can see it's listening.

And now in here I'm going to go back to code and I'm going to show you what happens if I bypassed all the safety checks, and I can add my own workflow. I'm gonna create a new file. And what should we call it?

Make it really obvious. Own diamo that sounds you know not suspicious in the slightest. And let's grab this no, once again, no way.

I'm gonna type it. I wrote it somewhere else. And we can see now I've got the approved.

Remember we looked at that there it is. And we've got all those malicious commands in there. Who no, it looks really bad.

that's gonna run on a pull request and I'm gonna create a full request from it, which is not too bad. So I'm happy it looks good. I can't commit because it's a protected Branch.

So this is my Branch protection working. I'm going to create a new branch. And I'm going to call that new Branch innocent branch.

I'm going to propose that file. Now if you saw it was called Big CI. I give it a really important name.

big CI 22 It's gonna create Pond Animal. Yep, let's just create that pull request. Next thing I just submitted a workflow, right?

Remember this review required because that's cool. I have no reviewer set up. But you'll notice.

That it's waiting to run some of now it has queued. my new thing look at that. Why am I approved?

How did that happen? If I roll down we can see that the getup action pot self-approved so it worked. And now my my pull request is running.

Well, let's check. The first thing I did was exfiltrate details, wasn't it? If we go here.

Oh get up token. That's not good. If I go here.

I have yeah, this is bad. What happens if I go over here? Who am I?

I'm the runner. This is all that. Now was one kind of final little test if I was here, it's pretty obvious because I'm you know, I'm hovering over here.

Some checks haven't completed some of my catch on and go that's fake a long time to run. That's weird. So as an attacker, I might.

And this this works, I'll be really pleased. I might try to you know, delete my steps. Let's go over here.

Now. Just take a look at what? What's going on over here?

Well, that's interesting. And then I'm gonna exit. Okay.

so it's weird happened there is One other thing I did. if you know who Allina Linus torvales is how did I somehow check something as as someone who isn't I start I've suddenly made a change to the pull request if you look at some closed, so if I go to the poll request it's not here. How strange is that?

If I click on the closed. And I go to look at this pull request. Not only is the only thing that seems to be part of this is a Hello World from Linus Torvalds.

But if I look at files changed Nothing there, even though it says five so I've really confused this and in the process somehow deleted the existence of the malicious workflow all bad things. Let's go back over here. So, what did I do?

I create a reverse shell on a potentially self-hosted Runner while concealing the fact that I just did that all bad. What can I do with this anything I could discover AWS Secrets, I could start searching for passwords. I can change the logging level and exfiltrate access to sensitive data and logs.

I can look for the location of code from other builds. If the self-hosted winner shares things, I could utilize the GitHub token after the fact because I've closed this somehow or I could elevate my privilege to root. Which isn't that hard on most runners?

And take over the world. So this is Marvel warning the ounce of prevention we need is very similar. You've seen a lot of them already.

I'm just gonna recap be careful when adding contributors. Always turn on Branch protection and use more than one approval. Do not run workflows.

Unless you're under pressure. Remember Environmental Protection is your friend? Try not to add permissive GitHub tokens or AWS tokens use some form of more advanced Secrets management or something like a spifier arrangement.

Use short lifespan tokens least privilege and probably an easy win is using a code owners file for the GitHub directory. I also added sign commit and give and get up events to log activity. But just to talk about code owners.

This is a no-brainer. This is how you can differentiate the GitHub workflows from anything else. You had a coat odorous file into the dot GitHub directory and then it's not just the bot can't even if it's one approver.

It has to be the approval dictated by the code owners directory. So that GitHub bot would not have been able to approve even though it could try so that's what makes the GitHub workflows directory unique and protected and it insisted insisted upon being reviewed by a very certain person who you know, you can trust so that is a very great tip and probably when the last ones I'm going to make here. Now an alternative what can you do to just automate this other than adhering the best practices if you've not heard of it, the open ssf scorecard is highly recommended.

What you do is you add your GitHub token to your environment and you just run open ssf scorecard if you want to see what it looks like. Here's a result of me running it here on this exact repo and it's scolding me because at the time I didn't have Branch protection turned on it's also looking at the fact that I have a dangerous workflow already present using patterns that are bad like the issue management one and a variety of other things. It's looking for SAS tools.

It's just a great way of assessing the security score of your repo and it extends Way Beyond just GitHub best practices. the other option Check off by Bridge crew also has checks built-in to check for exactly the things that I I just did. It looks for the use of curl in a workflow.

It looks for reverse shells. It looks for the exfiltration of the environment, unsanitized inputs. All of these things are possible through automation.

io and you could use it as a GitHub action as well. And again rather than rerun it. I've just read it here.

And you could see this is my this is me running it on the thing. I just tried to submit so suspicious. He's a netcat suspicious use of curl with secrets.

So these are the sort of things you can run to catch any workflows that might be trying to be malicious. And a very simple way and I mentioned checkoff can be a good Hub action. All right, so I'm going to wrap up there.

Thank you for playing I told you all the links would be at the end. com security guy specifically for hardening self-hosted Runners that everything I said almost everything is there and if you read it and you do it, none of the things that I just did can happen to you, but if you go with all the defaults, it is remarkably easy for somebody to try and take over your repo and potentially your entire supply chain. Okay, that's the end.

Thanks.