John Shier and Chester Wisniewski, Sophos | RSA Conference 2023

This is texturung TV. Everybody welcome back. It's rsac here in San Francisco 2023 in the Moscone Center West to broadcast alley.

We've got a great interview set up here conversation really set up here. I'm joined by two field ctOS. So there's actually three of us who are ctOS having this conversation and both are with the Sophos a Chester John.

Welcome glad to have you both think it's good here. Yeah good to have you here. Once you start out introduce yourself and then if you would introduce yourself and then tell us about Sophos and what kind of jump into some good stuff.

I'm John Shire field CTO with Sophos been at the company for almost 16 years now done a very variety of jobs in the company both on the sales side, and now doing a lot of research and Analysis within the office of Technology. Awesome. Fantastic Chester.

I'm Chester wasneskyo also Fields CTO for Applied research at sofas some almost 20 years now almost at sofas but largely again working on with all of our researchers around the world, right? We've got lots of teams data scientists. We got people looking at malware.

We got people looking at web threats we got Incident response, we've got you know, all these different things and so we're really trying to help paint the big picture for folks and collect all that intelligence from all those smart people doing the work and turning it into something that we can use to hopefully secure ourselves better and the broader sofos. Tell us what Sophos does. Yeah.

So social for those that haven't heard of us. I mean, we're Enterprise security company a lot of folks Associated us with our start and Antivirus back in the 1980s, but you know, we've got our fingers in a lot of pies now from everything from Manish detection and response services and incident response Services through firewalls endpoint security xdr. All the things of business needs to keep itself safe.

Fantastic, I can't remember using Sophos is my AV. Yeah, we're what was this our 37th anniversary this year? I think so.

We're one of the old timers but we think it's 1985 was 1985. Yeah, 30 ideas. Yes.

Sorry. Yeah, fine. Wow, that's amazing.

Fantastic. Well, so lots of things we can talk about. I know you just come out with the report kind of looking at incidents and some of the interesting kind of facts that tidbits about it John tell us if you can summarize some of the highlights from it certainly so yeah, we came up with the 2023 active adversary report and it's it's a look back at all of the incident response investigations that we did in the past calendar year.

And the idea being that we want to tease out some metrics, you know, we want to look at demographics want to look at the maybe the root causes of these attacks and some of the tools that get used and out of that maybe provide people with some advice about if this is what we're seeing then maybe these are the kinds of things that you might want to address within your networks. Right? We don't want necessarily to focus on one or two things because as the title of the report suggests here everything everywhere all at once there's a lot out there to protect against so we want you to really look at your environment and go.

Okay knowing how these attacks unfold. How can we actually protect our Network given some of the maybe the weaknesses that we have over the capabilities that we're lacking right now and to give you a few of the high level topics within the report we saw exploiting vulnerabilities followed by compromise credentials. We're sort of the top two root causes so there's still a big problem that they're with people patching their devices and when they're not breaking in their login, and so they're getting compromise credentials my various Means to be through fishing could be through through buying them off, you know, initial access Brokers iabs and once they're in the network, they're getting to work fairly quickly with all sorts of tools.

So between the tools and techniques that we track over 500 tools and techniques that were used. So it's just a massive volume of things that you need to look at. So then I started looking at maybe different slices of the data going.

Well, are there things that maybe stand out from the data? And this is why it led me to that that title was there was no day of the week that attackers prefer. There was no one tool that attackers preferred.

No, every industry was represented. And so it comes back to the fact that if you're going to do security properly you really need to understand. Where are you today?

And then where can you be going to move the needle on some of these things so that you are a less of an opportunistic Target for some of these cyber criminals? Yeah, we wish it would be as simple as if you just this is the trend go take care of this you'll be good for six months. Right?

Right, but just isn't the case just isn't the case. Yeah interesting to tell me a little bit about your your take on kind of this topic so many ideas because obviously when John says, you know half the time they're breaking in and half the time they're logging in that's not terribly helpful to me either because Clearly we've been talking about patching for a very long time that's not used to anyone. Right and and neither is credential theft or password theft.

It's not like oh now that I know this I can go and be safe because these theme kind of obvious. So the question is, why are we failing at this so badly? That's kind of how I start looking into this and like What is it that could lead to these things not being patched and sometimes we're talking.

Many months if not many years on patched when it's so obvious how important that is and especially you know, as we all went into remote work during the pandemic. I think we all realize that remote access tools into our Network. We're both necessary and a giant risk.

So you'd hope to think that you would put an extra emphasis on patching that particular equipment being that it's all out there flapping in the Wind on the internet and you can't turn it off because you need it for your employees to work. So, you know, we've been you know, interpreting the report. So the report itself of people read is all factual data from all of our incidents, but we're kind of doing more of the crystal ball gazing of going.

Well, what can we read in from this to figure out how are we failing in these ways that isn't maybe quite as data driven, right? And on the patching front my my pet Theory at the moment is just a lot of organizations still have security reporting in through their it infrastructure through their it organization as opposed to assessing the risk and security as a separate thing depending on the size of the org, maybe through a CIO or another part of the leadership team. And as a result, you're competing for resources with projects that are being delivered that will make the business more efficient or more functionality and Marcus the cloud we need to The new Salesforce project and my bonus and my okrs is based on the successful launch of this project in Q3.

And if I am late, I'm gonna pull resource and shift it over here to things that I'm gonna get a pat on the back for gonna make management happy. Nobody knows that the VPN server didn't get patched and I know it's important, but I also probably underestimate how quickly a bad guys gonna find that thing that's unpatched. I think those things all come to this perfect storm leading to so many of these victims having such obvious.

Failings for how these incidents begin in their networks. So and because I haven't heard of A bunch of other victims like myself maybe being a smaller medium sized company. That's why I'm underestimating the risk.

And also I think a lot of companies overestimate their capability when it comes to them being under attack and how they can respond. They're never going to find my servers. Like do you realize that with modern Network scanners?

We can scan the whole internet in three hours, right? Everybody's service. It's not a matter of knowing about you.

Right? I was joking yesterday that you know, if we'd finally get around to doing IPv6 at least we'd slow down the scanning, you know? Yeah, but that's not a realistic answer to this problem.

The realistic answer is we need to help people assess that risk for accurately to understand how much higher that risk is than they think it is so that they can adjust the priorities correctly to hopefully start accomplish into bigger to find all the IPv6. That's true really Quantum Computing here in there. I'm curious.

So, you know, I've had this discussion with the security and the network organization and that there's the reverse of what you're saying. which is there's so much security technical debt. I can't get my developers it infrastructure folks to work on those things because they are the same things functionality Revenue generating getting us to the cloud.

It's it's a dilemma that I think where it sits may help but I don't think it's gonna solve it. At least I haven't heard anybody's cracked that night yet. No, it's not at all in one answer, but it It's about ensuring that the incentives for the leadership that are responsible for security are aligned with accomplishing security.

And whatever that looks like I don't know every organization is going to have a different org chart and how that it's going to fit right within their organization and clearly as a security company, which is what I know best that doesn't apply to most businesses in the world. Most businesses don't operate like cell phones. I know it works for us, but there is commonality in much of the victims where I'm seeing that that seems to be one of the contentious points is the it's always a competition for resources in the end and when John talks about overconfidence, I think it's another piece of this which is A lot of the organizations, especially with current budget situations with the economy the way it's been the last year to.

Sometimes you can get products, but you can't get people. So we might have the right Technologies, but we don't have the people either they don't have the time or we don't have the quantity of the people or the education to use the tools correctly. And therefore I was well equipped to protect usually address these threats, but I never got around to doing it.

One of the great things about sharing your data from your the incidents that you've helped customers with and I'm starting to have a lot of these conversations with people who do that sometimes individual Consultants or books like yourself with Sophos is sharing some of those experiences and part of the thing to share is by the way, there's some good practices, but here's the things not to do there are things that you can make your job impossible coming in or maybe a much more difficult like, you know, tainting evidence and turning off servers while encryption is running and all kinds of things. What are some of those Worst practices to not do yeah. Well that's you mentioned a couple right there.

And I think when I start look at some of these investigations one of the key reasons you bring an incident responder in besides obviously closing the hole that caused the incident in the first place and evicting the attacker is having a timeline of what happened. How did everything unfold what things were touched? Which also lets you understand?

What things do I need to remediate right? Having that data will mean that you might not be a victimized again and we have seen in some attacks where the the victim would start to roll incident response before they actually fixed the original cause of that attack which then caused other threat actors to walk in and particularly. We actually wrote about this on our sofas news blog.

There were three attackers in one victim. That all use the same way in it was an exposed already interesting machine and they were able to use that. In other cases we have where you know again, you start rolling in some response.

You wipe all the machines you find that you can't do it yourself as well. And so then you call an external third party and then you ask us what happened and this is kind of like, you know, asking the police detective to go in there after you've bleached everything down. You've cleaned up the crime scene.

You've taken all the shell casings. No, okay. Tell me what happened.

We we it's only so much we can do at that point. Right? And then finally there's in my opinion kind of the worst, which is a lot of companies just don't keep Telemetry at all.

They either you know, their logs go into a bit bucket or they roll over way too quickly, and now there's just literally nothing for us to look at and at the end of the day we want you to be safer. We want to give you advice on how to not be victimized again and to also increase your level of security after this event so that you're even more resilient against the attack and if we can't give you that advice Off of how this attack unfolded then you're kind of left where you were at the beginning which is with potential exposure and potential leave reduced security posture interesting. It's almost like the the DUI Home Makeover rescue shows right where you know homeowner tried to do the the makeover themselves and got in over their heads very quickly.

The average organization is not gonna be highly confident it responding to an incident. At least. I think most of the world larger Enterprises people have teams that can can do the tabletops and that's right to get ready.

That's stratified a little bit. Yeah. Yeah.

Exactly. Yeah. We're your thoughts just well, I think you know you you desperately do want to know when did this start and I was fascinated by some of the different things that you found about like where we had a very early artifact and then a giant gap of time and then all of a sudden the cluster of activity, right which suggests that perhaps 91 days.

In fact of media the median 91 days interesting area the data guy wait for the quarter to roll or something. Yeah. Well, but you know, the I think it's proof that the stuff is being cached.

And bought and sold somewhere and your your product sitting on a shelf compromise in a criminal inventory and a Marketplace and then they finally moved the product 91 days later on average and suddenly the thing happens. So there's lots of time to detect these things if you're watching and in the two things that help both what he's talking about I think help with this is even if you don't have a crack team of security people deploying modern tools that are keeping track of all this stuff at least both helps us unravel it and gives you a chance at detection xdr tools things like that like, You mean even if you didn't know how to prevent the attack at least when the incident responders come in, there's detailed log of everything that happens so it can be unraveled go back to rewind time to go. How did this all happen and that's incredibly helpful in those cases and in the best case scenario.

The median time to compromise once the activity started was 11 days 10 days 10 days. 10 days a long time if you're watching. Yeah.

Yeah our tools deployed. Even if you're only a chance to check them say three times a week. You might catch it on day two or day three on average before the data has been expatriated before the ransomware has been positioned maybe before they active directory has been compromised to start copying malicious binaries all that kind of stuff 10 days is a good amount of time.

The Defenders have a chance. If you're trying you actually have a great chance at least short circuiting the attack if not stopping it entirely but it does take some time and effort and the in more advanced tooling than simply old good old-fashioned antivirus and the there are multiple signals right Chesters already just talked about how there's that that lag time maybe when you're sitting on the shelf, then there's really the bulk of the attack right the activity the 10 days but even within that there's other signals as well specifically we're talking about ransomware. There are events that occur with a lot of these ransomware as a service Crews where they do date extra filtration on top of the encryption, right?

And so what we've noticed is there's at least a two-day meeting time there as well from the time when we start tools like our clone and mega sync client land on the network and start preparing the data to when it actually gets to when the rents we're actually gets deployed. Right? So the data is exfiltration is happening during that time.

io cloud, right the best thread indicator. If you have an unexpected four gigabytes going to Mega, huh? You probably ought to look into their Network, right?

It's hard to spot when it's OneDrive, but it's really easy to spot when it's Mega. So that's right. So so now you've got another crack at it.

So organizations do have a chance. But again, it comes down to having the telemetry. And then having somebody pay attention that understands the Telemetry and then finally knows what to do about it when they see it and that's where services like MDR really help because we have that knowledge.

Right? We have the capability you hope you hope that a defense team in most organizations has not faced off with lockbit three times. Oh right like and yes, give it another go.

Yeah, exactly. So you don't get the experience to recognize. Hey, that's lockbit.

Whereas an MDR service. Unfortunately probably has squared off with them a few dozen times this month and they can spot that and like up I've seen that before I know what that is. And when they do that they'll do this next and then that and then, you know, we can really be proactive about and that gives us the best part of that 10 days of dwell time because now our time to detect is an hour instead of a day and now we can do something about it more quickly.

So let me ask you personal question. So take your company hat off. I'm not asking for any secrets.

Both we've been doing this for a while. You've been with same company for a while, you know 20 plus coming up on 20 yourself. I'm thinking about the people who are entering security may have got a few years under their belt and discovering what they like and and why they like it why have you been doing this for 20 years plus or minus?

It's interesting. Right, there's as I look over the the years that I've been doing this technology has has changed quite dramatically the way that we do things both on a defensive side. And the way that the attackers are doing things has has changed dramatically.

It's grown, right the the threat has grown exponentially back when the days when we started, you know, the I Love You virus and you Along for the virus and almost. Yeah like those good they were interesting but then I did live through code red and nimda and a bunch of those where you know, I'm in a sock banging away at a keyboard with the the CTO looking over my shoulder going. When are we gonna stop the slice equal Slammer right putting in Fireball rules and fast as I can get them in there and network core Network router rules.

And while some things like we've talked about with with patching, you know, maybe our taking a little longer to be adopted than we would like from a defensive standpoint. There's been enough change in churn and new technologies just to keep them keep it interesting right at but at the very core. I just I'm passionate about helping people do better and be better at whatever's they do and in this case.

It's technology and securing networks. And this is why I'm in the role that I'm in now is because I get to do this kind of stuff talk to people like you and we get to talk about this all the time our customers the meat, you know, the The World At Large to just really help them move the needle and get to that better place where technology is ubiquitous. We all get the benefits of the snazzy new gadgets, but we do it in a safe and secure way.

How about you Chester? Well, I kind of started guitar talking around. Well, I was a teenage hacker when I started in the 80s and that to me it was fascinating that it's like MacGyver you're using things in a way.

They weren't intended to be used like you're getting creative about. Well, I got these things. How can I combine them in a different way to get a different outcome and I found that just endlessly it's creative.

It's fascinating. It's a puzzle. It's intellectually.

Yeah, just triggered all the right things in my brain and and the other part of it was that No day is like the day before as fast as everything's moving. So I don't get bored. I get bored very easily.

So that's been fascinating. I mean the only two weeks of the year were it feels like nothing is changing is this week and the week of black hat and Def con were we hear the same thing 700 times because we're at RSA and it's like, oh are you saying it's an echo chamber, it feels a bit like an equator. There's a bit of an echo, but, you know every other day of the year, it's like everything is new every day.

And there's a it's a new challenge that's demanding an even more intellectually challenging response or creativity to address that challenge. and if we're successful and our entire goal is to try to make people's lives. Safer and easier, like just seems like a win especially if people are willing to pay you to do it.

Fantastic. Well, let me say thank you for doing what you do. Appreciate how you helped.

That's your customers your company all of us in when people need to call Ghostbusters. Thankfully there are ghostbusters around to call, you know, next security people. So thank you.

Both of you Chester John let us and thanks for Sophos offering you up to share some insights from your report in your own personal experience. So thanks for the conversations. Great.

Talk to me. Look forward to doing it again sometimes yeah absolutely black cat. Yeah.

Exactly. Okay. I think that's our last interview of the day and what a way to wrap up.

I think it's a fantastic and some kind of personal insights of what motivates people to do this kind of work and holding their interest in networks security. So thank you for joining us on Tech strong our RSA live stream from rscc 2023. We're gonna start up in the morning.

We'll be doing more of this. So please join us on what day is it Wednesday? So join, Thursday I have no idea what it was.

tv. We've got some new content coming up from kubecon last week that's being edited and that'll be up soon. And of course we'll get this up also for replay then be safe and we'll see you tomorrow.