Patch Management in Cybersecurity with Jeff Huffman and Eran Liven | Qualys QSC23

This is Textron tv. Hey folks, welcome back to the Qua Security Conference Americas. We're here with Jeff Huffman, who's senior director of IT Security for the New Orleans Saints.

And then we have Iran Ney, who is a senior product manager for endpoint remediation for Qualys. Did I get that right? Senior Director, but yeah, Senior director.

All right. There we go. And we're talking about, well, obviously patch management.

Gentlemen, welcome to the show. Thank You. Thank you.

Let me just start with something really simple, though. On the face of it, if you're a security person, patching seems relatively straightforward. Uh, there's a vulnerability, there's a fix.

Apply the fix and let's go home by, you know, five o'clock and not stress out. And yet it seems like we're unable to execute that cleanly. So what's going on and what's the challenge and where are we?

The challenges are just getting to the computers. Uh, and there's never one patch, you know, it's multiple patches, multiple, uh, softwares, multiple devices. Uh, are they on the local network or are they out, you know, uh, working from home in my case, is a scout, uh, that lives in a different part of the country, uh, patching their devices.

Uh, 'cause they have, you know, proprietary and confidential information. You know, uh, working with the college, uh, players, they have information from them. We gotta make sure those computers are protected.

Is that what you hear as well? I mean, I, I hear it's a struggle, but what are you hearing from customers? Exactly.

So yeah, when we talk to customers, the problem is, is basically what Jeff just said, right? It's, everybody's worried about what will happen if I do deploy the patch. So nobody caress about deploying the patch.

What they care about is the consequences. Meaning if I deploy the patch, the application will stop working. My boss will call me and say, my stuff doesn't work anymore.

You know, my CEO will say, so how do we make sure that when we deploy the patch, nothing really happens? And as he said, how do I get to all those machines to deploy those patches? So it's the fear of what will happen.

And then if I get the green light, how do I make, make it efficiently and now way I get to all those machines and actually apply those patches. And it's not always a patch, by the way. It can be a patch or configuration change.

I think if I may add one more thing here, the question is always about patch, but usually that's what not interests customers. Customers don't talk about or don't think about patch. They think about the vulnerability.

That's the security risk that they want to solve. And sometimes it's just as a patch, sometimes it's a patch and configuration change or something else. Again, it's the same challenge, but we, we see a lot of customer changing the way they look at thing and not just looking from a patch perspective.

And the question is, how do I mitigate, or how do I remediate the risk itself? How do you get it and the software developers to buy into this process? 'cause part of the issue as far as I can tell, is sometimes they think that the vulnerability doesn't exist.

'cause it's not in some database that they can find where there's no system there. Other times the developer says, you know, I don't have that in my code. Or, it's not internet facing, but it is, well, It's up to the, the application owner, the business owner, the asset owner, uh, you know, they can refuse the patch, but they're gonna have to take responsibility.

And basically when you take, you know, the report and say, okay, here is the vulnerability that you don't want patch or you don't want changed or the configuration change you're signing off. If anything happens to, uh, that machine, that server and it, it infects other machines. You're being held responsible for that.

That usually changes their attitude a little bit. They'll, uh, work with you. Uh, but uh, it just, in this, in this culture, I mean the threat actors are out there.

They're out there after your, I mean, if you don't, or if you're afraid the device won't work because you patch it, it's definitely not gonna work if they, uh, hit it with ransomware that Clean mm-Hmm. A lot of times there's other mechanisms for remediation other than patching. 'cause there's politics in patching.

So how can we kind of wrap some things around vulnerabilities that might, uh, mitigate the issue short term and maybe long term? So, so that goes back to my point, right? We don't focus about patch.

We focus about how do we reduce the risk. That's the question that we have to ask ourselves once we ask this question. One of our option is, deploy the patch, deploy the configuration change, and remediate the risk completely.

But as you said, and as Jeff mentioned, sometimes it's impossible. We just can't. For example, you have, um, maintenance windows.

So you can only patch something in two months. 'cause that's the only time the business allows you to patch. And you need to reduce the risk.

'cause there is a weaponized vulnerability. Somebody is, is you also said, right? Somebody may exploit it and, and get into your network.

So as you alluded to, there are other options that you can apply to reduce the risk. And what we're trying to do now is because qualis we're, we we're focused on reducing the risk, not just the patch management side. There are many mitigation option that our teams, our research team that are doing the day-to-day, uh, uh, detection and, and figuring out what vulnerability is out there.

The same thing will analyze those specific vulnerability and tell you for this specific vulnerability, here are your options. For example, you can block a port, you can stop a service, or we are working on a very, very cool technology right now that is virtually patching in memory vulnerabilities that are specifically, so if the vulnerability is there, we know the vulnerability is there. We can actually in memory, protect against this vulnerability only that vulnerability.

And the reason we want to be very, very specific with our recommendation is we want to reduce the risk of something happening to the device. If we pinpoint and we only apply what needed to be applied, the risk of something goes wrong is very, very limited. How much control do you have?

I mean, can you apply a patch or can you institute some of these remediations? Or is that always a conversation with somebody about how to, you know, if and when? Yeah, It, it, it depends on the, on the business and the, the application.

Is it AV you know, Chris, a critical vulnerability or is it something, uh, you know, uh, just a, a, a fix? But no, it, it's really, as the cyber people, we don't make the business decisions. You know, a as IRA was saying, you know, it, we, we set it out and it's up to the business owner to, we make the recommendations, whether it it a patch or, you know, any kind of vulnerability.

You know, uh, most of the time, you know, critical patches, they're, we're gonna be able to push them out. They're, we, we test them on non-critical devices prior to moving it to a, a critical device. Qualys is making a case for giving security folks a little more control.

How's that conversation going with people? Oh, really? Well, 'cause you, you want, so security people want to reduce the risk, that's for sure.

And security people are sometimes quite, quite frustrated with how much time it take their IT counterpart to actually reduce the risk themself. 'cause as, as Jeff said, usually is the application owner, the IT guy is the one that's responsible to actually apply the fix. So the fact that we give them more options and we give them more control, allows them to work better with it, give it more options, and usually they'll work together and agree on an SLA or sometime it will try first if it doesn't work, security team that gets more control will, after the SLAs breach, will take control and choose one of those options based on whatever the situation is.

So again, we're not gonna reboot, uh, uh, a server during, you know, when you sell tickets, you don't want to reboot the, the, the tickets, uh, sell server, whatever it is, right? So we give the control so you can take the right decision and the smart decision. It is the type, Can I, can I flip his example around?

So he's giving me a document to sign that if I don't apply this patch, I have to accept responsibility. Can I, as the IT person give the security person a document that says, if you apply this patch, you are accepting responsibility. So that actually does happen more and more.

And we were surprised to see how much it's happened be, be again, the industry start, not starts, started to understand that the security risk is there, everything else is excuses. So it's a mind shift. And yes, things can break, but you have to take more, uh, into your own hands and basically teach it that things can happen.

Because in reality, most of the time, as I give an example in my, my session there, 99% of the time, nothing happens. So all our discussion here is to a very, very small, you know, everybody remember one instance of a job or something that wrong, got wrong and you know, all hell got loose. But in most cases, there's not problem.

If you apply the right, you know, work together and apply the right testing and apply the right processes, everything should be smooth and you'll get reduced risk. We talked a lot here today about the disconnect between cybersecurity and IT and the business. Um, the business is more dependent upon IT and software than ever.

Or, you know, do they understand the conversation more? Are we making progress or what's your sense of where are We? That's, that's basically up to the, the c-suites.

I mean, they're, they're ultimately responsible to whether an owner in my case or, uh, you know, a board of directors or, uh, you know, stockholders in a larger business. But, you know, uh, you know, if the vulnerabilities are there, I mean, we know the threat actors are there, you know, uh, uh, is this vulnerability to be weaponized and brought against you? 9% of the patches and nothing ever happens.

You know, and majority of the times you'll go months and nothing happens. That's why you test on non-critical devices. You know, you have a server, you know, that's, uh, just, it could be even a test server.

You know, uh, you're not doing your production servers, you're a mission critical, your, uh, devices until you're, you're virtually sure that it's not gonna break anything. Mm-Hmm. Are the bad guys getting better at weaponizing vulnerabilities?

What are you seeing? That's a good question. Answer is always yes, unfortunately.

Uh, but we are getting better in protecting, or at least our tools are getting better in protecting It's up to the customer to use those tools and get better in protecting their environment. And we actually see customers that are highly mature and, and do apply all those methodology. And we see a huge, those customer, we actually have numbers.

We see a huge, uh, decrease in open vulnerabilities. And those vulnerabilities are usually the, you know, the, the open the door to, to the organization. So it's really depend on how, as Jeff said, how the C level takes the problem and how, what instruction they, they give their right, they give their IT and security team.

And you can see a, a huge difference between organization to have security oriented C levels and people that prioritize the, the, uh, Business with the c with those executives buy-in, you know, uh, they wanna protect their assets. They know the bad, the bad guys are out there. So, uh, that's really the key to a lot of organizations.

You gotta have upper management, the executive suite, uh, their buy-in to, uh, protect your organization. I mean, it's, it's not me personally. You know, we're all out to make our organization safe, and that's the ultimate goal For so long Security was an afterthought.

We would roll something out and then we'd be like, oh, well maybe we should go fix this thing 'cause it's been attacked. How are we getting better at having a conversation about security at the point in which we're actually deploying something new or updating, you know, have we narrowed that workflow? Every organization's gonna be different.

I, in my organization, I have a real good working relationship with the director of IT. And, uh, we've, uh, you know, we communicate, you know, his office is right down the hall from mine. We communicate well.

He knows what's going on. We've discussed, uh, our criticality devices, what, you know, these are our, you know, uh, primary devices. These are our test devices.

These are, you know, just give, each one has basically a grade and the, uh, and when you're doing your testing and you're doing your your mission critical stuff, that's the last stuff they're gonna do. We heard a lot about AI and we hear about AI everywhere we go. Is AI gonna get applied to these processes?

And what might it Look like? ai, AI is gonna have a huge role there. 'cause the main problem with patching or remediation, again, let's call it risk reduction, is not breaking anything.

That's, as I said, that's the main fear. AI will play a huge role in helping you reduce your, uh, uh, uh, fear by analyzing or, or applying algorithms to figure out what can go wrong if it'll go wrong, based on your environment, different other customers environment that looks the same. Applying those technology to figure out what is the risk and helping you then automate.

So if the risk is low based on the ai, I can just take the action either patch or mitigate or that will take us to the next step. 'cause we can respond faster with more certainty that things will not break. Mm-Hmm.

I don't know how long you've been at this for the Saints, but my question would be, what do you know now that you kind of wish you knew when you're first starting out in this game? It, you know, I've been there 24 years, so it, it's, it's been a long, uh, process. I mean, and you just try to be better, uh, today than you were yesterday, you know?

Uh, and the, the whole dynamic has changed. You know, the, the world has changed. You know, uh, you know, cybersecurity wasn't, was, wasn't even the word.

I don't even know if the word was invented 25 years ago when I started, you know? Uh, but it was just, it, it like every other process, it grew to where most organizations have dedicated cyber teams, uh, to, you know, protect the assets. Where do you find those teams?

'cause everybody I talk to says, you know, we have a shortage. We can't find enough people, so everybody's got open job recs. So yeah.

How do you handle that Training work? You know, give people opportunities, you know, uh, recruit, you know, uh, we have a decent name, uh, out there. Uh, so, you know, a lot of people wanna have an opportunity in professional sports, and it's a, it's a growing market.

Well, you have recruiters in professional sports, so maybe you can just borrow their playbook for cyber security teams. Right. Let me ask you this question.

You've been doing this a while. What do you see customers doing that just makes you shake your head a little bit and go, folks, we're a little bit better than this. What, what's kind of that pet peeve that you go, huh, Customer, just worry about things that happened 20 years ago.

You know, you wanna believe how many, every customer will have a, a story about usually Java. I dunno why it's always Java that break five years ago. And since then, they're not patching anything anymore.

So they have this horror story and I don't get it. 'cause we can see, just read the news, right? How many more marquee vulnerabilities are, are released every month or two or three.

And the pro is not, it's only getting worse. And they still believe that, you know, because Java break five years ago, we cannot be, uh, we cannot patch or we have to wait, whatever the excuse is. Alright folks.

I think if President said a long time ago that the only thing we have to fear is fear itself. And I think it applies to patching. That's Joe.

Good job. Very Good. I remember that one.

That's good. All right, gentlemen, thanks coming by. Appreciate it.

Thank You. Thank you all. Thank you.