Security Testing at Scale for Cloud-Native Technology | Cloud Native Now 2023

Hi, my name is Caroline Wong. I'm the Chief Strategy Officer at Cobalt, and I am delighted to be joined by my friends and colleagues, Pete and Shannon. Uh, today we are here at Cloud Native now, and we are gonna talk about how to improve security testing at scale.

Um, I am so excited to introduce these folks. Uh, for folks who, uh, may not know, um, Pete and Shannon, do you want to each give an introduction? Ladies first?

Oh, no, no, no. I insist. All right.

Hi, I'm Pete Chesna. I'm the CSO for North America at Checkmarks, uh, 17 years in AppSec and 30 years of engineering. And I'm Shannon Le.

I'm a founder of DevSecOps and also c e o of third score, and I've been in this industry for over 30 years. So I'm a security dinosaur. I guess I'm the newbie of the group, uh, with only 18 years of cybersecurity experience.

Um, only So, yeah, I think, you know, if folks wanna look us up on LinkedIn and learn a few more details about what each of us do, um, you know, I think each of us has, uh, dealt with all sorts of different sized organizations. Um, particularly I would say more modern, more DevSecOps type organizations. Um, certainly each of our careers, uh, has spanned a period of time where software development has changed a lot, maybe cybersecurity a little less, so that's up for debate.

Um, but in any case, uh, we're delighted to be talking about security testing at scale, particularly for cloud native environments. And so for our first panel question, um, I'll ask each of you, what are your thoughts on the following statement? Cyber attacks have been growing in frequency and severity over the past decade.

Uh, yes, true. Uh, also the vectors have changed. They've multiplied as the definition of application has grown as people have moved from on-prem into cloud as they've fractured their monoliths.

And, and one of the, one of the interesting things about how these vulnerabilities come to the surface sometimes is you take a monolith and you, you have all your controls sitting at the top, here's my entrance into to where my application goes, and now we're in the monolith, and then it comes back out. Then they take all those pieces and they're like, oh, we're just gonna like splay them sideways now. And when they do that, they're like, they don't forget that now all the controls are gone.

And now I'm exposing these services through APIs directly to the front end. And now we've got this gap where it's Ebola or, or some other, uh, vector where, yeah, I had that accounted for before, but I didn't think about how to multiply that across all of that infrastructure as I did it. So I'm gonna play contrarian.

Love it. I, I think that it's totally right, but I always, I question what do we mean by cyber attacks and what do we mean by growing as an industry? We don't really keep track.

We're not a very transparent industry. We don't have our numbers out there. Folks are afraid to share that they're having a tax.

I think I've only seen a couple of companies come out and actually show how many millions that they're 40. Um, I, I think there's a beginning of this industry in terms of transformation, in terms of modernization, where we actually need to start sharing more about how many attacks. Because how many attacks doesn't mean you've had a major breach.

It doesn't mean you've had incidents that it just means that adversaries are out there. And it would really help to share which adversaries are out there, which it would help to share the information in a different way. So I, I start with, I question the statement from the standpoint, probably more meta, which is we can't really say they're growing.

We can't really say they're declining because we don't really keep track as an industry. We don't keep track one company to the next. And I do think they're growing, but for a really different reason, which is you can see that there are more adversaries out there, that there's been a growth in that type of part of the industry.

There's been growths in the infrastructure that they're leveraging. You can see that information even more transparently than we see what they're actually doing to these companies. And so I start there, and then I would say we have to break down what a cyber attack really is.

Is it hitting your front end? Is it hitting your people? Is it hitting your laptops?

Like when we talk about cyber attacks, we tend to take all of that information, commingle it, run a, you know, run some sort of query against it and say, okay, they're going up. But the question is, is malware really working? Is antivirus working?

Is E D R working is, um, some of the stuff we're doing with SaaS and dast and all of these things? And so I would say maybe a, a contrarian viewpoint is I actually just question the basis of how this industry starts the conversation. Because at the highest levels of our companies, um, the board of directors is looking for information from a different perspective.

It's not looking for necessarily the threat angle, it's looking for the, are we doing enough? And I, I still go back to that. And then I think that actually leads into the business case for sas das, all the DevSecOps tools, the modernization.

And so I go to, I think there is growth, but I think there's growth in adversaries versus attacks right now. I think adversaries are getting better at the attacks that they're actually starting to push towards us, which means there may be fewer attacks actually hitting our companies, but they're much more targeted. They're going after specific things they're good at.

And, and as a, as an entity, we, uh, really have to start to think about that. Shannon, so much food for thought in like 60 seconds, and I have responses, I have ideas wanna share. But before I do that, I just wanna ask Pete, if you happen to have a response to Shannon's thoughts.

So I I do agree with her as she was talking. I was thinking about, um, the, the talk track that you had, Shannon, you know, when we met years ago of thinking about what, what are they coming after? You know, it's al it's always driven towards monetary gain and, and gain.

And if you look at the, the Verizon, uh, breach report, you know, it, it, it's looking at what are they going after? And as we look at the economy and how the economy has changed in the past five years, there is more access to that money that's that ha that happens digitally. So I think if you take those two things together, it's clear just from a, uh, uh, from a, a personal point of view that the attacks are going up.

There is far more infrastructure being leveraged that allow m multiples of those attacks, where it's not driven by a human and a keyboard anymore. It's now breach as a service. Uh, that, that it, it's clear that these attacks have gone up, that the breaches have gone up, that the value of those breaches has gone up.

Uh, and, and of course, if you read that report, like 85% of them were still driven by, you know, stolen passwords. It's like we, we ha we haven't fixed the fundamentals yet, But I, I think we've actually fixed some fundamentals. Like we are running a lot of software through SaaS das mm-hmm.

Some of these things. So, so it still goes back to where are we actually effective? And we're driving the numbers down.

We, we have to see more transparency, visibility to what's working, what's not working? Are they actually effective? What are the 200 s looking like?

Because here's the thing, I I really believe that there has been a huge impact on the industry with DevSecOps. And I think it's in the information that is about our software as it's going through the pipeline, it's getting better and better and better. So the number of types of opportunities an adv, sorry, has in software itself are actually going down.

And that would show you that the r o i is there. But because we take, again, all that information, we push it into this thing called cyber attacks. You know, I have a, I have a data set to share, um, please, which is the cobalt.

For the past five years, we've been, uh, releasing a research report based on the pen tests we perform in the previous year. To date, we've done more than 10,000 manual pen tests. And I'll tell you what, the data is remarkably consistent.

And that's really interesting to me. Um, it's interesting. Another thing that is actually kind of wildly frustrating to me is that some things, which I think we could call cyber attacks, that happened a long time ago.

The first ever ransomware attack happened in 1989. And users who were affected by this attack were asked to mail 189 US dollars to a PO box in Panama. Now, in, in 2022, the average ransomware payment got up to the millions of dollars.

And so I am very curious about something that I'm hearing Shannon allude to, which is, what are we even talking about in the first place? You know, when we say cybersecurity, do we mean security controls? Maybe not.

Do we, maybe, maybe not. Do we mean risk management? You know, maybe, maybe not.

And are those actually different ways of, of seeing these things? You know, I I, I, I'm very curious about this. Let's consider the lens through which we're even having the conversation.

And, you know, maybe do we have an opportunity to look through a different lens that might prove to, uh, provide a more valuable perspective? Um, and so with regards to lenses, I'm gonna ask another question, and I expect both of you to completely tear the question apart. Here's the question.

What role does proactive security testing play in enhancing and organization's risk management posture? Shannon, we're gonna start with you on this one turn. I knew you might do that.

All right. So I'm gonna play it out for you. I actually think that the folks that can score something, how strong it is from an adversary perspective are probably the most important of all of the types of cybersecurity professionals in helping us to understand how well we're doing.

So when you really think about that outside in perspective, testers tend to have that, they don't necessarily start from the standpoint of component parts. They might be doing some sort of, you know, white box testing, black box testing. But inevitably, what I actually think is on the outside, looking from the outside in, they're the ones that could actually take the adversary perspective.

Now, what do I mean by an adversary perspective? Because that's another piece of puzzle. There are many different categories and types of adversaries.

And you know, my question back on a question like this is, are we really even again talking about the right things? Because when we talk about testing, do we have the right testing, testing, melody? Do we have a test plan?

If you look at, uh, nest 800 dash two 18, they allude to a test plan, but yet I still haven't seen any, anybody really enact a test plan or share a test plan or have an open test plan. If you look at some of my work previously, I started this notion of open test plans. If you look at some of the work that's being done, I think testing happens a lot through the cycle.

If you look at SaaS and deaths, those are all testing tools. If you look at what you do with pen testing, again, testing, what do we do to create great security? Has to start from the adversary persona and then be fully tested all the way through the cycle.

And I don't think there's anyone out there that can opine other than a tester about whether or not something worked from that adversary perspective. And that truly is where I think that the tester is paramount to the conversation. Again, back to the board, how are we actually taking those metrics and really making them come to fruition?

You know, is your SaaS or your das tool actually giving some sort of benefit back to the company? And I believe the answer is always yes, because the testers, I now have been talking to a lot of testers, they've told me it's getting harder and harder to find software vulnerabilities at the actual time that they're testing in production. So that's a really interesting data point, Pete.

So when I was thinking about, and again, as you were talking, I'm thinking about, well, I, I like that outside in, I like that that final tester, I'm thinking all the way back to the beginning in the threat model and, and how accurately was that built and how, how were we thinking about what are the threats to the business based upon what we're doing and what we're doing it with? So we we're doing some form of function on top of some data or some monetary resource. Uh, how are we protecting it?

What do we think the inroads are from an attacker perspective to get after those things? It could be, you know, the, the personnel list. So they can go and, and become those people.

It could be get access to the money to drain it off, to take it somewhere else. It could be get into the infrastructure so I can do ransomware, um, uh, all of those things. So have we done a good job of that and prioritize the right things?

And I'm going back to the, the four questions, uh, and, and did we do a good enough job? Because you, you can look at the tooling tooling's interesting. I, I like, um, it, it will bury us under more work than we could ever do in our lifetimes.

And a lot of security professionals that I've worked with, when they see something, they can't unsee it, and they're like, well, we've gotta fix it now. Versus from the pen tester's perspective or the threat perspective over on the left, well, what should we be protecting and what is the most important things? Understanding that we're never gonna get to everything.

How do you seal off the biggest cracks and the biggest holes such that we're not, you know, doing something totally devastating, understanding that there is no perfect and there is no good enough. Uh, there's just what we could do in the, in the timeframe we were given with the resources that we have. You know, I'm, I'm hearing different frameworks for thinking about this, which I think is fantastic.

Um, you know, from Pete, what I'm hearing is you start out with what are you trying to protect? What do you value? What has value for you?

And, and jump in please and, and edit me. You know, if I'm not reflecting this accurately. But, and then Shannon, I'm almost hearing from an adversarial perspective, what exactly are they trying to achieve?

Yeah. And I think that we, I think that I often make an assumption that what we are trying to protect, if I call us defenders and what they are trying to achieve, if I call them adversaries, I think we often assume that those are the same things. Mm.

And then we go after it. Yes, I want to hear about this, Shannon, look at that. And then we go after it with these security testing, whether it is SAST or dast or red teaming or pen testing.

Shannon, tell me a little bit more about this adversary perspective, because I think that from the perspective of a defender, I, I wonder to what extent we actually make assumptions about an adversarial perspective that turn out to either be misleading or just wrong. Uh, or we just assume things because that's the way we see it and maybe, you know, maybe that's a natural human thing, right? We, we assume that people are like us.

Um, but actually there are all these different roles in the mix. Yeah. Um, in the adversary environment, if you think about it from that perspective, um, I think we still commingle so much and Pete's right about threat modeling, threat modeling's a really useful tool.

It is, I think that attack maps tend to give you a little bit more bang for your buck. And what those really are is you've gotta look at your total customer base if you're a software provider manufacturer, right? And in that total, um, space of your addressable market, you have customers, the folks that you're building your software for, and then you have basically adversaries the folks you're not building your software for.

And depending on how you build your software, you may say, I'm gonna spend all my time on customers. 80% of the total addressable marketer are, are my customers. Cause it's not a hundred percent, let's be honest.

There is an edge to all things. And so in that adversary persona perspective, there are slices of adversaries, there are folks there to abuse your software and it might be a slight abuse. And that slight abuse may afford them to be able to do small things.

And in some places in the world, small things are actually big things. They feed your family. There are other adversaries that might be trying to use your software for really big, horrible events.

Like they need billions of machines at their disposal. And so your software allows them, because it's installed in so many places to get access to a huge supply chain. And so I think in, and when we really talk about talking to developers, the persona of the adversary, the persona of the customer is a little bit of a different bent on being able to flow from one side left to right and then ultimately your testers being able to line up against those personas.

Are you testing for script kitty capabilities? Are you testing for somebody who's gonna utilize and do fraud in your software? Are you, um, maybe testing for the recent patching problems and hey, patching's actually a thing.

There's a variety of those that come through. And if you look at the actual tools that we have in the pipeline, they can be assembled to be adversary, pon, Sona per specific for that specific, that allows you to be able to tie one end to the other and also go back to your, you know, high level folks and say, Hey look, we're actually doing pretty well on, um, unskilled adversaries that are gonna go after certain things. Like maybe they put a URL into software and they're able to actually get the software to go out and pull something in and now they can fish somebody.

Um, and I think that we really have to start to think about what is the actual adversary persona doing? And then what are the tests that we need to line up against those personas to be able to say, we're getting better. Our software is stronger, our company is more resilient.

Because that's the trust profile of software is to really think about what are they gonna monetize off of. And like I said, total addressable market. There is some percentage of everyone's software that's actually Arial.

I wonder, lucky if we even have the right people in the room to have the not, and not in this room, but I mean in general. So if I, if I think about threat modeling, I think about your red team. Uh, I think about pen testers, like what you have at Cobalt.

If you think about what, what cobalt is meant to do, the people that work for you and do those pen tests, right? They're to your, uh, prior thing. They're feeding their family.

I'm finding things, if I find enough things that I've done a good job and now I can go and do that. But they are not in adversary in that way. They are are, they are a stand-in, they are a, uh, someone that, uh, we, we are using as proxy for adversary, um, our red team, our head, our fraud department in, you know, like a major bank.

How are those people being connected to the people that are writing the software to think about what are the right abuse cases? What are the right attack maps? What are the right parts of that map to focus on where they are very separated in time and in organization, uh, that they are not maybe operating in the right circles such that you're having this approximation at the beginning where if you ask an engineer, how can you abuse your system, they have a, a very fixed mindset on they built it, they could see it, they understand certain aspects about it, but they have never been a malicious actor that's trying to ruin the company or ruin someone's life, uh, or steal millions of dollars.

It's hard for them to be actually that person in the same way a pen tester or a red teamer or you can see what's happening and you can react to that. But how do the, how does that feedback ever get back to the right people, which are the ones that are building the software and help hope to build it in a resilient enough way to prevent those abuse cases and attack maps from being executed? No.

Yeah, that's right. And I think the other thing is, is when we talk about doing security, we don't talk about it as code security. We don't talk about it as business logic and we don't talk it about as configuration.

And there's actually three separate entities around, if you think of software security, it's those three different things. And they have different adversaries that go after them into those specific spaces. Somebody might be really great at code level issues and they are not gonna go hang out in these other areas cause they just aren't gonna make the money that they're gonna make when they're actually great at doing code problems, right?

So I, I think that we've gotta get more specific in this industry. We've gotta get more, and I think you're right Pete, back to thinking about who should be in the room. I believe the product managers in the business are making choices long before we see a developer and a ux and then even later into this pipeline of security professional get involved.

There's business level decisions about, again, going back that total addressable market, what percentage is adversarial because that's actually what you're fighting back. And as you're seeso having a conversation about that adversarial percentage and how they're managing and dealing with that, because that's not necessarily risk management, that's adversary management. And we aren't even really talking about it as an industry, as businesses.

But yet it's in there. If you talk about a fraud department, they're actually dealing with a wedge inside of that adversary camp, right? Mm-hmm.

And, and protect. And in some cases in banks, there's actually multiple categories within that wedge that they're actually dealing with. And those fraud departments have an understanding of how much money they are losing so they understand the mechanics.

But when they go back, if they, if they're talking to developers about it, they're probably losing because really the decisions about what's gonna get worked on and what's gonna get prioritized, that's being done by the business that's being done by product managers. And so we've gotta really start thinking about do we, are we missing a role? Are we missing that security product manager, the one that's gonna do adversary management, the one that's gonna be talking to the business about how much they lost and whether or not it's okay in setting thresholds.

And my belief is that we're missing a trust manage manager or some sort of adversary manager well into the early parts of business that actually should be able to help un the business understand at a very top level whether or not they're doing the right things. It's so interesting. You know, one of the questions that we have on our outline is why is it sometimes hard for organizations to do preventative proactive security testing?

What challenges do organizations face? And some of the responses that I've gathered from our discussion here are, first of all, there's a bunch of stuff to pick from. We talked about threat modeling, we talked about sast and DAST and pen testing and red teaming.

Shannon's introduced a concept which is new to me that I think is fascinating, which is a security product manager, someone to get involved way early. You know, and I wonder if, um, you know, you two see one of the problems that I observe in a similar way as I do, which is simply to say there are all these different types of security related activities and how does an organization choose how much to do of what? Hmm.

And how does a security organization go and try and ask for that type of investment? I know that, you know, I've, I've got colleagues who run various, uh, security testing capabilities, uh, at major technology organizations, you know, and even folks who work at places that are, are thought of as very reputable and sophisticated in this area, you know, actually have trouble going and asking for money for something that to a non-security professional. Sounds just like five other things you're trying to do.

You know, when, when a, when a when a security person says to a not security person, Hey, we need to do threat modeling and we need to do scanning and we need to do pen testing and we need to do red teaming and we need to get security and a product management. You know, there's a way in which there's some, there's occasionally a business person or a finance person on the receiving end of that ask who says, well, gosh, I don't understand what the difference is between all those things. You know, and can't you just do, can't you just achieve the same outcome and and do one out of those five spends?

You know? And that becomes, that becomes a challenging discussion. Well, I I, you know, if you look back at how this industry grew, everything was human-based.

Uh, people would go in and by hand review code for security vulnerabilities. That's all there was. That was the state of the art we have developed.

Uh, what my, one of my good friends would call a cyber prosthetics. So like include SAST and DAST in those where the DAST was some rough approximation of what a pen tester might start with that you can automate and just go do things. And then you take that as the starting point where now a person comes in and does higher level functions, all the SaaS things are just ways of automating code reviews for security vulnerabilities.

And as I think more about this, I mean, how can you map those? Uh, I'm interested in what, what you have to say about this, Shannon, if you said, if I take output of any of these tools or from a pen test or from a threat model or anything, how do you map those into those attack maps to say, you know, if you think about, you know, cleaning your house, I'm gonna close the bedroom door and I'm gonna keep a bunch of stuff in there, I'm gonna vacuum the wel trodden pass where everyone's going to be during the day. It's like, how do you find the right way to say do these cuz they're justified, they accrue to this attack path on this map, uh, whereas these lesser of importance, yes, they may be high severity vulnerabilities on their own, but in context, if you look at these, how can you group them and make this into something that's more coherent to say, do that first and then you can work your way downhill.

You'll never get to the bottom. But at least that's, its a way of perhaps framing the conversation. Yeah, I've done years of studying this as, you know, from the years of us having these conversations, Pete, and I think, I think you're onto something.

So, you know, the way I see it, the adversary skill level has a low and a high and there's not, and you know, infinite number of highly skilled adversaries out there. There's just not, there's a lot of low skilled adversaries out there. And the high skilled might job out some of the things that they're trying to do, do for certain types of campaigns they might have.

But when you really look at it, are we as an industry in the security space actually testing for those low skilled things that are happening? As an example, are you validating the URLs that are going into software where you're allowing it into content from user submitted or, um, you know, validated information? And the answer is usually folks don't validate the, um, URLs that are going into their platforms because how, where would they get the information to validate those URLs?

What does a D G A mean? How do you actually think about that in the business or in the product? And so my belief is we're not even using the right scorecard yet for how we test and what we're trying to eradicate.

If you put the right scorecard in place and then you put all the tools against it and you line up your test properly, you can start to assert that you've run x number of tests, some percentage of that particular persona against your software, and you're probably gonna be better than most against that particular persona. Now, skilled versus low skilled, right? We also have lucky versus good.

You're not necessarily as, as an adversary going out and looking for a particular target unless actually that's the target you've been asked to go get and you're getting paid for that particular target. That's what I call good lucky is I'm gonna sweep the heck out of the internet and I'm gonna find those three things and then I'm gonna figure out how to monetize this. Cause I'm gonna go to this particular market and I can probably sell, you know, the 300 users that I got out of the database that was open and available to sequel injection.

And I think we really have to start understanding those adversary mechanics. We need to put it into our discipline. We need to think about it from a testing perspective.

We need to actually start the conversation in the product space of what percentage do we want to allow for adversaries? Is there, you know, what is gonna be that threshold? If we look at the total addressable market, what percentage is adversarial?

Because it's never zero. I'll just help everybody out who's listening. You have a business, you work any business, it's never gonna be zero.

So welcome to the adversary management game, which is what's the percentage that your company is allowing? And I've worked for many, many organizations in my career and I can tell you that in some organizations that I have worked with in past, they actually were making less than the adversaries in that space off of their software. So think about it from that perspective is are you, are you trying to deal with your total addressable market?

Are you getting to your shareholders? Are you thinking about these things in the right way? And and like I said, there's a senate customers, right?

And some percentage are actually adversarial. And you've got to figure out what that threshold is going to be and that's where you're going to place your testing bets. That's how you're gonna actually deal with your roi.

Some percentage of that is gonna be unacceptable and some percentage of it is gonna be the risk of doing business because you don't have billions of dollars to go spend on some percentage of that total adversary market. And so you're gonna end up having to do it things like get cyber insurance because actually you aren't gonna find that zero day out there before that adversary does because they just have too much money to go against you. That that is a very interesting part of the conversation too.

Uh, I was in a, a panel discussion with AWS talking about, uh, the cyber insurance market and the actuarial tables and how, how can they look at, uh, a cyber program and say, well, your rate should be X and your rate should be two x and your rate should be half x. How, how do you, how can an outsider that is in the insurance game who knows nothing about cyber possibly do that in, in a way that earns their company money. Uh, so there, there's a lot, there's a lot there to unpack to think about how much we're going to need to start sharing.

I mean, SBOs have been a good foray into the, we're now giving you some information in an outward fashion, uh, that's beyond attestation, that is real data. I think there's going to be a draw for more and more of that as time goes on. Especially as you start talking about and we're gonna ensure you against loss, uh, that they're gonna want to have some deeper in inspection on what exactly you're doing and how you're thinking about the problem.

Uh, so that, that was a interesting topic. Yeah. If we think about cyber insurance, this notion of one size fits all controls is where as an industry we break down because the adversarial stuff actually helps you to understand the actuarial tables.

Again, lucky versus good are you actually a big entity could suffer a big loss? You're worth millions, billions of dollars. That's part of how insurance companies are thinking about like, what do I, what am I gonna have to give back out potentially and what's the possibility of that happening?

And so we spend all of our effort on these perfectionist programs that are done through things like, uh, you know, compliance controls. And don't get me wrong, I think compliance controls are lovely, but I do think you actually need to mix your compliance controls against an adversarial table because not every company has the same adversary, persona, makeup either. You are not gonna necessarily draw out the same personas across every single company.

So one size fits all security is actually where we're losing the most. We're spending a lot of effort on it. We're actually trying to say that, Hey, you know, since X, y, Z company bought, you know, e d r, we should buy it too.

And you know, the question is, is that really working should be based on things like reinfection rate, Right? Re that's the old b uh, argument, Re reinfection rate, Maybe I should Exactly. Reinfection rate should be a number one thing that we're thinking about.

Another one fix rate when we're actually putting tools into our pipelines, what's the fix rate? Because a great tool is gonna have a high fix rate and that means that they're, they're either gonna have a low number of things found and a high fix rate. That means that the developers A great, I dunno that tool, I would say a great team would have a high fixed rate or a mature Team.

No, the tool, the tools themselves will draw trust by developers using them and they will actually look at the findings and fix faster and fix part as part of their cycle time if the tool is actually giving them value and benefit. And I find that across everybody's stuff. So you can compare fixed rates across tools and find that yes, the team has something to do with it too.

Don't get me wrong. You're right Pete. But I still believe that the actual tool is something we need to be chartering from a fixed rate perspective.

Folks, I hate to cut us off. I am Can We talk about compliance for one second? I think let's, we wrap, let's talk about compliance for literally one second and then I'm gonna ask for closing remarks.

And I actually think this is just the beginning of a large discussion. You know, and I, and I'm delighted we, I Peter, I'd love to hear your thought on compliance. Yeah.

So I, you know, this, this thought of compliance gives people the, the ability to sleep at night, but compliance is, is just I have to do this, I have to test this and I have to fix this and unless I don't, and then I'm still compliant because I've got an exception process. And I think the, except the, the, uh, amount of exceptions that get generated should also be part of that equation. Not just the fixed rate, but what's your exception rate?

How often are you just saying, I accept this risk, and how often is that risk calibrated correctly to the attacks that are happening to say they are informed risk decisions, not just risk tolerance of I can have this many or this much, but how do those impact those attack maps and the attacks that you're seeing in a way that you can justify taking that risk because it isn't on one of those primary paths. And I don't think those things are connected in any way today. You know, maybe, uh, in lieu of closing comments, I'll ask each of us to, if there's a particular resource or a particular concept that you'd like to kind of point our audience to, um, so that they can continue to learn about some of the things that we've been talking about.

Um, one of the things that I mentioned is Cobalt state of pen testing report. Um, the latest version for 2023 has information for more than 3000 pen tests conducted last year. Uh, so that's my pointer.

How about you to you? Sure. Uh, so I referenced this before the Verizon Data Breach Investigation report.

I read it every year. Uh, it's full of very useful information to show how much we haven't changed. Uh, and, and maybe those are the places where you might want to consider making changes in your organization, uh, against the attacks that you see and the losses that you incur.

community and start looking at metrics. And anybody who wants to help me by taking the survey, I'm trying to finish it up and get a first version out there. Um, I've also posted a secure ability article on Medium and, uh, would love any comments or feedback.

And if anybody's interested in talking more about secure ability, I'm all in. You'll see some information about adversaries coming soon. Phenomenal.

Thank you both so much. Uh, I've enjoyed this thoroughly, uh, and I'm so glad that we get to share some of our ideas with the world. Awesome.

Thank you.