Joni Klippert – DevOps Works! …Why Hasn’t Security Kept Up? Keynote

Now to continue on to our next speaker, Joni Klippert is CEO and founder of StackHawk, a software as a service company built to help developers find and fix AppSec bugs in their code before they hit production. Joni has extensive experience building software for developers. She previously served as VP of Product at VictorOps.

Hi, everyone. It's great to be here at Gitlab Commit 2020. Welcome to DevOps Works.

So why hasn't security kept up? I'm Joni Klippert, CEO and co-founder of StackHawk. I've been building software for software developers since I graduated from my MBA program in 2010.

The first technical company I work for built software that allowed users to deploy and manage more than a hundred different open source applications on any cloud provider. But just a couple of clicks. Back when we had many cloud providers, we managed updates to those applications programmatically and can even move apps across cloud providers in the case of downtime.

So that's really where I cut my teeth in technology. After that, I was the VP of product at a company called VictorOps from Seed Stage through acquisition to Splunk in 2018 at VictorOps, we're an incident management platform that helped disrupt that traditional NOC and operations model by making it easy to deliver alerts to the engineers who were closest to the covid, which really helped reduce. Meantime, to know and meantime, time to resolve.

In 2019, I founded Stack Hawk to bring application security into the DevOps workflow by helping software engineers find and fix security bugs in their code before deploying to production, my co-founders and I saw an incredible opportunity to reimagine application security through a software engineering lens. So in this talk, I'm going to share what key DevOps tenants have been missing in the realm of application security that have kept us shifting AppSec left into the developer workflow and what we need to do to modernize the practice so we can rapidly deliver more secure software. So DevOps is pervasive and it works, and what I mean by DevOps works is it has unlocked the ability for companies to deliver value to their customers faster.

We've seen major shifts in tooling and processes that enable you are software engineers to go fast and to look to deliver that value to our customers, so agile unlocked the ability for us to quickly plan, develop and deliver software to customers, virtualization, cloud and infrastructure as code removed that sysadmin bottleneck. As we empowered engineer to deploy and manage their own resources, then we've seen massive efficiencies with automation throughout the integration and deployment pipeline. Then we further unlock speed with orienting to really small releases by breaking up the monolith and investing in micro services, architecture and containerization.

Then we improved our uptime by creating the ability for engineers who wrote the code to also manage it in production. The next most important area of investment is empowering our engineering teams when it comes to application security. When we do this, we unlock not only the ability to deliver software quickly, but more secure software quickly.

And our customers are expecting less of us. So whether you're part of a younger company that was born with the technical and cultural tenets of DevOps, or maybe you're a part of a legacy enterprise that's undergoing digital transformation, we invest in these DevOps principles to quickly respond to customer needs, deliver high quality software and ultimately remain competitive as a business. So why has an AppSec followed the rest of the product and engineering organization in Shifting Left?

So as I was doing research for my company StackHawk, I wanted to understand why security hadn't made it into the developer workflow. I interviewed more than 50 CCOs security professionals, CTOs, VPEs, you name it, in a matter of a couple of weeks. And we found that there were a lot of technical barriers to current solutions which we're going to talk about, but also some pretty meaningful cultural barriers that needed to be addressed for this change to happen.

So the most glaring thing that stood out to me was the relationship between security and engineering was deeply strained. It's not too dissimilar from the early days of the more traditional IT Ops versus Devs type of strain, but at least those teams are really aligned in delivering and maintaining software right for our customers. So this might be a little worse, honestly, in terms of of the type of tension that exists between these groups.

But it's somewhat easy to understand when we think that these groups have very different KPIs and operational objectives. Whereas DevOps is about orienting the team to focus on business objectives, security teams are aligned to reducing risk, oftentimes in a way that can come off as at the expense of the business objective. And this has fostered some pretty meaningful cultural issues.

The first one really being lack of trust and empathy. , because security is quality and engineers care a lot about the quality of the software that they deliver. But then when you talk to engineering about the security team, we've all heard these anecdotes, but it's referred to as the Department of Know or being historically just so difficult to work with that you try to avoid working with them at all costs.

The next piece is Transparency and Observability. We know that both of these are very core tenants of DevOps and something that's missing in the security discipline today. Developers aren't in security software the vast majority of the time, right.

There's little trust that developers should be able to see or action on issues until security tells them to report proves that. Right. Which is super inefficient because software engineers are responsible for ultimately solving the problems anyway.

And lastly, empowerment, my co-founder, Scott Gerlach was formerly the CCO of Sun Grid and he talks a lot about how we want developers to care about security, but we don't trust them to take a first pass and assessment and resolution of issues to be comments about the tendency for security teams to buy these expensive, huge AppSec platforms. But then they lock them down so tight with approval gates everywhere, it's impossible for a software engineer to engage or use the products so they abandon. And then we also have several technical challenges when it comes to modernizing AppSec, so first, most tools on the market are built to run in production by a security team.

There are many challenges with this, which I call the production bias. And we're going to double click on that here in just a minute. Next, these tools are really hard to configure and deploy.

Many, particularly in the land of DAST, which is dynamic application, security testing and the land we operate in the most at StackHawk is scanning the running application, and that's really difficult to get, working in a repeatable manner, making automation super hard. These tools just weren't built for it. Most AppSec tools were built with the idea that a human is going to be manually running tests against an application.

And the good news is this is really starting to change. Next, a lot of these tools generate a ton of noise. Output is super verbose, and it's also written in this very security specific language rather than something that's easy for an engineer to quickly grok and action on.

And we all know that a key component of DevOps tooling is to mitigate noise and ensure that a developer's attention is only drawn to what's truly important. And lastly, these tools just don't fit in with the existing developer tools and processes. So on the previous slide, we talked about security folks saying that devs don't care about security.

But when we look at this list of challenges, I mean, my question is how can they afford to? Right, in a world where software engineers are largely measured on delivering business value, the cost of caring about AppSec today is often just way too high. So the production bias we discussed that many tools on the market are built to run against code that's already been to point to production, and they are designed to be operated by a security professional.

This orientation toward production and lack of involvement by the development team at the right time and in the right context makes it impossible to modernize AppSec. So the first piece of the production bias is the people, so today, upset tools are commonly run in production by two groups. One is the security team, because production is where they know the application of the best.

The other is by a Pen Tester, and that's because production is their point of access, often with black box testing, we expect them to test from outside in. So in this context, it makes perfect sense that these tools to date have been designed for these two groups. However, it's highly inefficient.

Both of these groups struggle to instrument their tests because they're less familiar with how the application was built to work. So setting up an engagement is a pretty heavy lift in order to actually get a really good assessment. From there the primary value by these groups then tends to center on the finding of the things.

So there's a lot more emphasis on finding the number of things versus finding and fixing the right things. And this has repercussions, right? It's super inefficient because the finders of security books are not the fixers of security bugs.

And it reinforces that adversarial relationship we were talking about. Right. Your product and engineering teams are working hard to deliver features and business value.

And some number of months later, an outside party breaks your things and says, look, I broke your stuff. You need to go fix all of these bugs. We've long moved on to other tasks, so that involves a lot of context switching.

So I want to be super clear when we talk about people, I am not advocating that people should not get pen tests. I am suggesting that that method of knowing about security bugs should not be your only introduction or access to this type of information. The next most critical part of the production bias is timing.

So as companies are rapidly shipping code to production, security is not baked into this workflow either, you're not rapidly shipping, in which case AppSec processes act as a blocker for getting code to production. Or the security team is constantly playing catch up. And it's often times nine times out of ten, the latter situation.

But it gets worse because when AppSec rules favor running in production, like the elephant in the room is that the bugs are already in production. I try not to laugh hysterically when I read the slide because it's like it seems crazy to me. Right.

We want to find these things before we deploy. Now, there are going to be times when you intentionally ship security bugs to production just like you would defect. Right.

But the intentionality is the important thing here. This should be done eyes wide open and be a risk based decision. So you might know that exposure is super limited or you might know that you're planning on fixing the security bug in the next sprint.

But at the end of the day, production should not be the first place that you are checking if you have security bugs. And the third piece of the production bias is context, so can we check for security bugs in production some period after a time after a release, excuse me. It's super inefficient.

Engineers have moved on to other sprint tasks and they're no longer in context of their code. So fixing involves context switching, which we know is very both inefficient and expensive. When scanning for AppSec bugs and production, typically tools that are on the market today are scanning and FQDN a fully qualified domain name.

So something like dub, dub, dub, dot or app DOT. And the result is a list of bugs that exist somewhere in the app right under that parent domain name. And that makes it really difficult to identify the specific part of the app or the service that's affected.

And it also lacks the context of the specific data handled by that service. So you end up with a security team creating these tickets and doing a lot of ticket shuffling, trying to figure out which service was affected and then the team that owns that service and who is going to be available to fix it. And then we talked about an inherent focus on the number of bugs being found and this is really problematic, you hear a lot of stories about, you know, employing these security tools that fine, just like tons and tons of security bugs being real or not.

Right. And then the goal of the development team or the relationship between security and development is just a goal of percentage to fix bugs over time, being the primary driver of value, which is arbitrary. And it ignores the business context of the findings and the trade off decisions around risk and business value generation.

So instead, we should be engaging in discussions like how important is this thing, this app to the business? Should we be fixing all of the bugs on an internal application or should we just be going fast on that? And how should we think about our apps and the data that they handle?

So the production bias results in the wrong team, and by that I mean the team who isn't ultimately going to fix the bug, finding vulnerabilities at the wrong time because they're in production without appropriate context. So here's how Test-Driven Security Should Work for this. And for this slide I'm just going to read it right?

When a team writes code, they know the syntax is wrong, when it won't compile, when a team merges code, they know there's a problem. When it doesn't merge, when they run unit tests, they know the code is wrong. When it fails, the unit test, when they run integration tests, they know the code is wrong when the product doesn't work as it was designed.

So when a team introduces a security vulnerability, they should know because it fails a security test. So at this point, I want to emphasize that instrumenting test driven security is something that engineering teams should instrument even if they don't have a security team or function at their company. This is something that engineering can and should leave within their organizations.

So what does the right team look like when it comes to AppSec? We think it's really about developers and informed stakeholders ultimately aren't at all specific security issues, so let's make them aware of those issues as they're writing the code. And in this world, we reimagine the role of the security person or team as a coach whose responsibility is to enable their team right, the dev team or league of teams to be successful.

And there is value in optimizing for the developer experience that really plays into what security professionals are looking for anyway. Right. It's super efficient.

Engineers fix bugs in flight. It democratizes security information across the organization. And when we do that, security becomes a standard discussion in the building of software.

It affords collaboration among teams at the right level and provides an opportunity for targeted education, and when all of this happens, it allows our security teams to truly scale. So this last tip is, again, absolutely leverage pen testers for security reviews and tough business logic findings, but let's make sure we have access to this information with every single release. So the right time to be checking for our Appsec bugs, if it isn't obvious already, is preproduction instrumenting security tests into CI/CD gives our engineers feedback immediately.

Many of our customers at StackHawk are checking for Appsec bugs on every single merge. They also do this with an awesome GitLab integration. And this is ideal because if a security test fails, you know, you introduced a security bug in your latest change.

So when we then add the ability to also test locally, engineers have the capability to quickly iterate on the fixed test loop if they find a new bug. So for AppSec to modernize, engineers need to be able to test while writing code test while building code and security tools should play really well in these phases of development. Next is context, the right context for finding security bugs is in your code as you're writing it.

And when this happens, engineers can fix egregious security bugs in flight, and this results in less bugs, making it into production and less free work later. The next piece is about the context of your app. And this is important when it comes through with real time security testing in the context of the application that you're actually working on.

Teams then can leverage their micro services architecture to instrument security checks on smaller bits of code, which makes it easier to isolate and fix those issues. Team should also be able to make better judgment on these smaller bits of code and in context of the app, because they know the job of the app that they're working on and the kind of data that it handles. And then lastly, the context of the business.

We need to empower engineers to fix the most important things, not all of the things while they're in context of their code, the security team should collaborate with engineering on assessing business risk and work across product and engineering teams to triage lesser severity issues based on resources at hand, business objectives and risk. So when the right people are doing the right jobs, there is a nice bridge built between the security and development organizations, the goal of security team becomes enabling the business to go faster, safer. They laid the right foundations.

They focused on scaling security within the organization by empowering their developers, and they serve to help educate around risk. When this happens, the development organization can build more secure software also faster. Right when we automate the job of finding AppSec bugs and then empower engineers to fix issues and flight security becomes a natural part of the development process.

Ok, so perhaps you're thinking that sounds awesome, but I'm not quite sure where to start, so I've included two great types of AppSec tools that every company should be using. SCA or source code analysis helps teams identify vulnerabilities in open source libraries or containers. These tools compare versions of libraries to third party components that you're using to known vulnerabilities, and they alert you when it's time to update and some even create a merge request for you.

DAST or dynamic application security testing, scans your running app and APIs for vulnerabilities. It's an active scanner in that it's actively attacking your app with inputs and seeking outputs that indicate a present vulnerability. The nice thing about DAST is since it reports successful attacks, there tend to be less false positives.

GitLab provides both solutions and really great integrations with several of the tools that I've included on this list of which they're both open source and commercial here. So you should totally check them out to get started if you're a developer. It's great because you get to skip step one because you know your app and how it was built and you know your pipeline.

So in that case, simply choose an app or a service that you want to start evaluating for security bucks. Right. Pick a technology.

One of these two are great and just start scanning and pipe lining. And then as you get comfortable with security driven testing, you get to layer in additional applications and technologies as you go. So that's it for me.

com. com to learn more about how we're putting in the hands of developers and enjoy the rest of your conference. Thank you so much.