Security Stress Surges

Hey, everybody. Are you afraid of Shadow ai? You're watching text?

Hello. We're back, and we got another awesome session today with some, uh, new folks joining us as well. But we're gonna talk about all kinds of fun things, including shadow ai, some DevSecOps issues, and maybe even CISOs.

There's million Dollar Babies. We'll see how this goes, but I wanna introduce our attendees for today. Joining us once again, John Schwartz.

How you doing, John? You don't look like you're in San Francisco. Where are you?

Um, I don't, I'm, uh, I'm at the 68th floor in Las Vegas. I'm at the Zscaler Show, and, um, it's, uh, I'm getting used to being here. Mike, I know you used to always be here, God, like hundreds of times.

So, um, yeah, AI conferences are happening almost every week here, so I'm trying to go to a fair amount of them. If I had known how many times I was gonna go to Las Vegas, I'd have bought that apartment about 35 years ago. There I slide.

There you go. Also, joining us once again, Terry Robinson. How are you, Terry?

And where are you? Because you're neither, you don't appear to be in. Yeah, I am.

I'm doing well, and I am in Memphis, Tennessee. Um, I actually, um, there's a little AI here. I actually went to the Making Vinyl conference, which is all about vinyl records and their comeback, so.

Well, that's awesome. My, my wife is a huge collector of vinyl records. The Living Room has tons and tons of them.

So your next time you're back in New York visit, My, uh, ex-husband actually puts this, uh, conference on, and he is a big collector, and he's getting ready to move to Iceland, where they're doing final out of beets, sugar beets. They're gonna have a pressing plant there, and he's trying to get me to take his record collection for a while. He's gonna give me custody of it while he's in Iceland.

So, Well, uh, I, I'll be curious to find out where you're gonna store all that. All right. And all right.

And we have a new member of the Yang joining us today, Garima, BJ Pie, who is the founder of the DevOps community in Canada. And I'm gonna let her explain where she came from and how she got here. But Garima, welcome to the show.

Thank you, Mike. Uh, uh, I'm glad to meet, uh, John and Terry. I'm based out of Ottawa in Canada.

I'm the founder for the DevOps Community of Practice here in Canada. It has several chapters out about Toronto, Edmonton, Atlantic provinces, Montreal. I'm also the producer for Summits Canada, as well as I've started a new initiative with John Willis, which is, uh, DevOps for JI heck, hackathon.

We just did it, uh, June 2nd. Huge success. I think you will see a lot of, uh, at traction and social media action from my side on that one.

Yeah. And I'm eager to kind of talk about shadow AI and other topics on the panel today. Awesome.

John Willis, of course, has been on this show many times in the past. So a lot of folks are familiar with John, who's kind of one of the, the, the leading originators of the whole concept of DevOps, along with a bunch of other folks. It's, it took a team to create a thing for teams, right?

So that's how that works. Let's jump in, John, you have a story up on text drawing AI talking about how, uh, it folks are getting a little wiggy about shadow, right? And the fact that there is shadow AI doesn't come as a surprise to anybody, but just how winky are they getting about it?

Yeah, it's interesting. Um, so it's, it's a concept that's been around, it's basically unsanctioned or ad hoc generative AI use within an organization that's outside the governance of it. And there was a, a survey that came out of, I think it was a 200 IT managers with at least, uh, operations of at least a thousand people.

And they found that nearly half are, quote unquote, extremely worried about security and compliance impacts of, of shadow ai. The numbers are, are, are pretty stark. And, and I, I mentioned earlier, or you asked me earlier about Zscaler and, and I'm at their, their show, and it's very interesting during a, a course of a couple of interviews with the executives there, they were, um, understandably very confident company.

They're doing extremely well. But when I mentioned shadow ai, they kind of lost, they kind of hesitated or stuttered because it's something that clearly freaks them out. And it was funny because the flow of the conversation, like reached this kind of like abrupt, wait a second, you know, like we're, we're trying to deal with this.

I mean, they, they're obsessed with zero day trust and et cetera. And, and in a sense, the survey found that like 90% of those surveys that they had concerns about shadow AI from a privacy and security standpoint. And, um, they had mentioned a, a litany of things that had happened, including false or inaccurate results from queries leaking as sensitive data from ai, et cetera.

And it seems to be, um, they're, they're trying to find a way moving the right data to locations for it. I AI ingestion. Um, there, in a sense, it's, it's just a problem that is gonna only grow with ai.

That's basically what I want to say. Um, it's been around, but it's gonna accelerate. And the reason why it's gonna accelerate, and we've been talking about this for weeks, if not months, every other day, there's some sort of AI related announcement from enterprise software companies, cloud companies, and it's just building into this data trough.

And I, and I believe that shadow AI will just won't go away. It's not a easily solvable situation, and it's got it managers a little bit freaked. It might even get bigger as we go along.

But Terry, we've been dealing with shadow IT issues forever. But in the age of ai, I feel like the conversation and the risk levels are getting significantly higher. Thoughts.

Yep. Well, for sure. I mean, just higher in general.

I mean, this sort of tracks with, um, I did some reporting last week on a Varonis study that had come out about shadow AI and the, uh, some of the issues there. And they say it's basically, uh, when it comes to data, a ticking time bomb, right? Because they looked at the risk assessments of a thousand, uh, organizations, and what they found is like 99% of the organizations had sensitive data exposed to AI tools, and 90% of those tools have access to sensitive cloud data.

So there's a lot of reason, um, you know, for, for their concern and, and the risk that just ratchets the risk up. And there's no easy answer there, right? Because you might, I mean, some people say, you know, that, that the best way to stop a bad guy with AI is a good guy with ai, but that's not necessarily true because the, the, the more the good guys, you know, use ai, the greater the opportunity for the bad guys to, to, uh, exploit and, and get ahold of it.

So, um, that's not to say that there aren't things that you can't do from a security standpoint to, to try to lock things down. But Gima, how feasible is the following scenario? Because we've established a course that there's gonna be AI agents for just about every app there is out there, but people will create their own AI agents and they will, um, use these things to automate all kinds of workflows that it may or may not know anything about.

But what's different in my mind is that by hacking to that AI agent, I can take over the entire process. And that's a little bit different than just kinda finding a vulnerability or ex exfil training some data from some APIs. So is, is this a likely scenario in your mind?

How, how much of an issue is this gonna be? It would be an issue for sure, and this is like, uh, in the words of John Willis, a tsunami of technical depth, which we are creating through shadow ai, right? And we need to understand why is it happening, right?

What are the key factors? Why we are in a situation where shadow AI is on the rise? I mean, there's a huge desire of efficiency and innovation, right?

So companies, employees, teams, uh, they are often turning to AI tools, uh, where they can automate things and repeated task or toil or streamlining their workflow. Even, you know, agents, for example, uh, is a big kind of, you know, uh, presence in the ecosystem when it comes to innovative solutions for solving these problems. Um, why, uh, you know, this, uh, also becomes a problem is that, you know, uh, we don't have, uh, enough organizational policies, um, and the process of approving, uh, these kind of tools and application and emerging technologies re rather slow, right?

So the pace of evolution of technology is larger than what we see, uh, the, the governance and the policies and the clear kind of, you know, guidelines. And, uh, if you like it or not, it is something which we should, should be kind of focusing and, uh, we should, uh, like, like to look at from a governance perspective. How do we govern and, uh, you know, establish some kind of a rule book or guide book for democratization of, uh, you know, onboarding of emerging technology.

Uh, I would also say that there is, uh, two other factors, which I, I see from a big enterprise perspective, um, measuring the wrong things. You know, uh, when you are starting to measure innovation and you build a race of innovation without looking at the maturity of that innovation, it will, uh, again, lead you to a large technical depth, right? And it'll cause more problems than, uh, you know, efficiency in the system.

The second problem I see, and this is again, large enterprise ecosystem, is incentivizing, uh, these kind of behaviors, right? So how an organization takes the control back, you know, to a certain extent. I mean, I come from a community, right?

So I'm all in for democratization, community led innovation, but without guardrails, uh, it wouldn't flourish. It would probably go in the wrong direction. So I think incentivizing the right behavior, the right organization's culture, you know, not going in a direction where you don't have approved, you know, blueprints, for example.

If you wanna have 200 blueprints, go for it. But you have to have that control and checkpoint and guardrail for it, right? So those are the kind of things I see, you know, as a enterprise leader, I would put in place, uh, to deal with this kind of, uh, problem.

And again, John and Terry both mentioned about privacy, the data lineage, and management issues. There are many issues which are kind of behind the scene issues, which are orchestrating this kind of shadow it, uh, behavior more and more. John, here's the part that, uh, drives me a little insane, because every CEO you talk to is basically sending out memos, telling their staff to use AI more.

And they're saying, you know, why aren't we using it more? And why aren't you taking advantage of all these wonderful productivity gains? I keep reading, and yet, and yet, you know, we're not thinking about the security issues apparently, and the, and the data.

And where does all that data go when I use those AI models? 'cause I know that somewhere there's a box in that end user agreement somewhere that I'm supposed to tick that says, don't use my data to train your AI model, but nobody knows where that little box is. You know what's interesting is there, that's this whole kind of ethos of Silicon Valley move fast break things, right?

And it's almost taken for granted. And I think especially now, that's a dangerous attitude to take because I think there's more at stake in term, not just in terms of data, but in terms of far reaching implications. So we have these two different themes going on.

We've got, uh, the, the governance, which is slowly evolving. And then on the flip side, I'm listening to these keynote speeches. So Snowflake had a keynote speech with Sam Altman as a guest, and he talks about this scenario for AI agents, them getting smarter to the point where they become almost, uh, independent scientists within organizations that, that find discoveries right on their own.

It's almost like it's, I wouldn't say rogue, but it, it kind of sounds a little bit like that. And again, that's, you, you would expect something like that from him. Like he's pushing the edge.

And then, then on the flip side, I I, there was a, a keynote here from the CEO of Zscaler who mentioned, and this is heartening, I think that he looks at different waves of ai. And during the three waves that he was talking about, security was always like the secondary last moment. I mean, that's, that's collateral damage.

We'll, we'll worry about that later. He actually thinks, and I, and this is probably part of his marketing idea, and part of what ZScaler's hoping is that security is becoming more of a, um, uh, a priority, more so it, it's not an afterthought as much, and it better, it better be because these things are moving so much faster. And I, and I, you're right, the pressure from the top is we, we not wanna be left behind.

I don't care what happens, but when something does happen and it embarrasses us, then your, your ass is on the line. And that's basically where we find these poor IT decision makers or managers. They are being pressured at the same time.

They've gotta keep things clean and keep things efficient, but they're also being pushed as far as fast as they possibly can be. And it's a little scary. It's a, it's a, it's a perfect recipe for the bad guys for, in my opinion.

So you're saying, uh, here's this tool you should use, but if it ends your career, I'm sorry, not my fault. Yeah, we'll, we'll find somebody else and then it'll be their head will a jumping block. But that's, that's corporate America though.

Yeah. It's like, just classic America, corporate America. Yeah, But I would put it in a little nicer way for the practitioners that, you know, know what John probably wants to mention is that, you know, the substantial amount of change for the developer community or practitioner's community is around how do you onboard on security governance policies?

Because all the toil is going out to copilots and, you know, assistance and you know your code boards, right? But how do you ensure your policies arise? You, you put security guardrails.

How do you build a code which is responsible ai, you know, um, you have ethics in, uh, in involved, in, you know, whatever you do. So those are the elements which will change the way developer develop code, and that is a differentiator for approach. Yeah, I mean, I think it is a combination of technology, governance, and culture, right?

That that's what's gonna maybe save the day, the day. And it's really hard to get those three. Um, point It is true.

I, I think I'm just gonna leave it with this word of advice that I got a long time ago when I first start, first started covering ai, fellow said to me, he said, yeah, the thing you gotta remember about AI is it's one thing to be wrong. It's another thing to be wrong at scale. We'll be back in a minute.

Hey folks, we're back, and it was a little more chit chat on security, but there was a report issued by the folks at Wiz, which highlights an attack involving, uh, crypto jacking, which, you know, basically is the stealing of CPUs to go mine Bitcoin. And that's kind of a nuisance crime, and it's been around for a while. But what was different about this one in my mind at least, was it didn't just seem to target a bunch of developers to get their passwords and then log in and start using their resources.

It seemed to actually be targeting the tools themselves. And it was kind of an escalation of, uh, what we're seeing. It's become an, an increasing trend.

But garima, what is your take on this whole report? What's happening here is, is, is the nature of these attacks against the software supply chain changing? Let's facilitate this discussion with some common grounds.

So what is crypto jacking if, uh, DevOps professional are joining us? I think crypto jacking campaigns are like coordinated cyber attacks, uh, where the threat actors hijack your computing resources and, uh, the victim devices would be your computers or servers or cloud infrastructure, typically to mine cryptocurrency without, uh, owner's knowledge, right? So what we are talking here is that, you know, there are is a, uh, extended trend that these tools are being targeted.

So DevOps tools, for example, including, uh, some of these like popular tools like Docker, HashiCorp, nomad, or, you know, gt, they are being targeted for misconfigurations. And, uh, what attackers do, or, you know, how do they work, uh, with this campaign is that attacker gain entry through various methods, including, like exploiting your misconfigurations, for example. Um, deployments, uh, is another area, you know, installing, uh, you know, crypto mining, malware solutions.

And this is, again, a very interesting thing which is happening, that they are using innovative ways to kind of install and, you know, uh, inject, uh, these mining scripts into your, uh, infrastructure. Um, when the miners, uh, get access to your, uh, infrastructure, of course they will, uh, you, they use it for complex cryptographic solving like cryptographic puzzles or, you know, even for, uh, transactions, right? Blockchain transactions, et cetera.

So, uh, it's like, you know, these people want to gain long-term access to these, uh, infrastructure components and what they're doing today. I mean, this is, again, um, there was an article as well written on, um, tech Strong, I think, uh, there was a, uh, elaborate, uh, description about, uh, this, uh, jinx, uh, 0 1 3 2, which is basically targeting these DevOps tools. Now, uh, it's very interesting because, uh, when you see these tools, I think, uh, there are loopholes, of course, there are weaknesses, and these weaknesses are being exploited by, uh, you know, these, uh, you know, attackers we can discuss about, like, uh, some defensive recommendations or mechanisms, how to kind of go about it and a practitioner's advice or a community advice on it.

But I will give back, uh, this to you, Mike, uh, so that you can also include other, uh, panelists to have their view and then maybe come back to me for some recommendations. Well, most definitely hold that thought, but Terry, I'm crypto jacking is long been considered a nuisance. A lot of people who kind of don't pay a whole lot of attention to it.

But, you know, I wonder, and I, and I haven't seen any, any data on this, but the issue is, all right, so somebody has hacked into my system and gained access to my infrastructure, and they may be using it to mine Bitcoin, but who else are they gonna give that access to? And what other nasty things are they gonna do? Because it seems like these guys resell or share every time that somebody, they gain access to something, suddenly they're like having a party.

Yeah, I don't know why we don't really hear about that now, but I I not only think that's a distinct possibility, I really think it's probably, you know, happening already, right? And green, you were saying they tap into the infrastructure that gives 'em access to everything, right? I mean, that's to, to the, the very sort of basis of, of, uh, of your tech and your company.

So I, I think that that is, that is in indeed going on there. So it, it raises sort of the question about what do you do, um, you know, to pro to prevent this or to, you know, limit their access or, or, or spur them. I mean, it's, you know, they're sort of your typical authentication things, but I mean, it, it's also like you have to secure that DevOps pipeline, do you not?

I mean, that's, you know, a, a, a really, and, and you could green, you could probably speak to this, um, better than, than I can, but I mean, what are the, some of the things like, I mean, there are a lot of variables. Do you limit those? Is that one of the ways that you, that you secure that pipeline in, in, uh, in, uh, keep, uh, these people from, I, I keep wanting to say guys, and they're not guys, but, uh, necessarily, but, uh, to keep them from, uh, you know, from a, you know, being able to, to, uh, access your infrastructure.

I mean, um, there's a lot of work around automating security in the DevOp DevOps environment, and I think some people think that might, um, you know, really help there. But, um, I don't know. What are you, what are you seeing in terms of, not, not to hijack you, Mike, but I, I, I see a lot of folks are talking about securing the software supply chain, but I think most of that focus seems to be on the components, but not the actual tools or, and I, and I wonder if we're kind of putting the cart before the horse sometimes, but Garima, what is your best advice to folks about how to go deal with all this?

Yeah, and you know, I'll also, uh, be mindful that these, uh, attackers are also getting smarter, right? So what I was reading was that instead of actually installing, um, the malware itself, they are actually injecting it through open source downloads, you know, which is very hard to kind of, uh, also, uh, detect, right? So what they want to do is they wanna do, uh, they want to gain long-term, um, access to your GPUs, CPUs, uh, you know, infrastructure resources for, you know, mining and, uh, without knowing you, right?

So it's like, uh, it, it, it's very kind of, you know, innovative approach. Now, how do you go about it as, as you said, Mike, that you know, the supply chain, you know, the problem which we have with the software, software supply chain, I think we have to be very careful in auditing that, you know, bringing control, monitoring those, uh, you know, uh, supply chain endpoints and access points, right? How do we harden configuration?

How do we also monitor and manage public tool downloads, for example, um, restrict network access. Of course, this is like a traditional thing, but again, uh, the, these hijacks happen because there are loopholes, right? And these people are exploiting it.

If you come to tools, I mean, they, everything is about open APIs right now, right? So if, uh, you, your APIs are not secure, that can be a potential kind of, you know, entry point. So misconfiguration of, you know, jobs or, you know, open API access is a large portion of it.

Um, there is, um, like manipulation of health check reports to execute some commands. So be mindful of, you know, hardening and the configurations you do on these specific tools. And also, I mean, lastly, I would say, let's say you are installing these, I mean, open source is bake in DevOps world, right?

So a lot of people use open source for critical, uh, you know, uh, infrastructure capabilities as well. So insecure installations, for example, or unlock setups or get hooks, for example. These are, uh, we have to be very careful and not to, to, uh, take them seriously.

I mean, these are like security checkpoints. And of course, um, I also can talk a lot about secure by design principles, security code policies, you know, secure coding guidelines or ask top 10, you know, attacker, uh, you know, landscape and how do you protect, uh, uh, like the software ecosystem from that point of view, there is a lot of guidance from N-C-I-S-A. So I think this is all good, you know, because we need to kind of get serious about security.

And we were talking about this in the, uh, first discussion that, you know, security is no longer an afterthought, right? I mean, there was a time where we were talking about DevSecOps and all that right? Shift left.

I think people have understood that there is no shift left shift, right? It's shift everywhere. So security wake up to central point for, you know, our, uh, the key differentiator of our software.

So I, I believe that, you know, awareness, education for your practitioners, there's a lot, right? So, I mean, starting with, uh, the DevOps capabilities and the specific kind of typical kind of, um, uh, crypto mining, I think it's, uh, more towards open source tools and technologies. So I think, uh, it's also important that we have, uh, established, uh, you know, program offices, for example, to deal with like legitimate downloads, you know, approved solutions, how do you configure open source technology and so on and so forth.

I mean, I'll stop here. I I can talk a lot on this topic. I, I should hold my horses.

Well, you know, the, the thing that comes to mind to me is like the tools themselves in configuring them and setting them up, it requires a lot of expertise and mis configuring them is a high probability. So, you know, does the actual DevOps team stand a chance in hell of actually, you know, not getting attacked? Because I mean, who's gonna come in and review their configs to make sure that everything is secure?

Unless maybe I go get myself an AI agent that does that, but, you know, is that where we, is that where we need, I think it's moved beyond human comprehension. Yeah. But I think, uh, the best practices, the lessons from the past, you know, you know, audits are good, you know, you have to do it, uh, off and on.

You also have to have some guardrails, checkpoints for practitioners. It's good to have. Then of course, uh, if there is some, you know, uh, crypto jab king happening, how resilient your architecture is to kind of revert and, you know, how, how much monitoring, logging into the system is available.

Observability plays an important role in this, right? So all this is like for good, all these practices, which we talk about for years and now a decade, right? So it all, you know, builds up into a tech stack.

And, you know, if practitioners are kind of serious about, uh, building good code and also securing their software ecosystem, these are things which they have to kind of, there is no choice. This is something which we have to take it seriously. Terry, who pays for the grid code jacking?

Is it, well, is it the, you know, do me as the, the user of the cloud service, you know, call up Amazon or wherever it is and say, Hey, you know, I didn't use that and it just got hacked and, you know, I want my money back, or does, you know, or do I just eat that? I think you're going to eat it. I'm sorry to say.

Uh, and, and here's why, because I think, um, and, and this is not to disparage, uh, Amazon and, and groups like that, but I think this is where it gets very hazy about whose responsibility is what. Um, and I think ultimately you end up, the company ends up eating it or whatever. I don't, I don't really think, uh, and, you know, I'm sorry, maybe Amazon and those guys are gonna call up and say you're full of it and know that, you know, that's not the way it goes.

But that's just seems to be, uh, what I've heard and what I've seen in some of the people I've talked to in the past about this, uh, kind of thing have said. Um, I also just think you brought up observability too, and I just wanted to just pop in there for a second and say that I think observability is being used more frequently now for security than just performance and efficiency and all of that. And that's, that's kind of a good thing to see, because that's a bit of a, I don't wanna call it hidden weapon, but it's, it's like some really good reconnaissance right on, on, uh, where the security issues.

So you've touched, you have touched on one of my favorite pet peeves. 'cause I always scratch my head about this one. I say, let me get this straight.

The app dev people are using observability tools and pulling telemetry and all kinds of stuff from production environments, and the security people are pulling telemetry data from the same environments, but we can't use the same tools somehow or other because we're doing the same thing. Or we gotta like, have separate tools here because everybody wants a different budget and says, you know, I gotta be different than the other guy. Yeah.

I mean, you know, Garima, can we do something about maybe unifying observability? So I would start with standards, right? So the first point of integration of inter for interoperability is, you know, how do you build standard protocols?

Standard telemetry, which we have already done, right? So Open Telemetry has been all of the de facto standards. Now with Open Telemetry, a lot can be achieved.

Now, if you think about the tools and interoperability of the tools, it also speaks for your architecture. So I would, um, encourage the practitioners, the teams, and the enterprises to look at like a dimension where they can make their architecture in interoperable. It does not matter anymore which tool, which capability, because, you know, a lot of these things will change with time.

I mean, today we are talking about observability of software ecosystem. Tomorrow we will talk about LLM observability and whatnot, right? So we'll have to think about, you know, standardizing and also consolidating the landscape a little bit.

I mean, open source, uh, uh, practitioners and communities are doing an excellent job there. Um, I have also, uh, seen a lot of traction, uh, around, uh, uh, tools and applications like CD events, for example, who's creating that, uh, data synergy with CICD pipeline. There is observ initiatives for LLMs, for example, now, uh, which are open standards.

So these are things which we need for the ecosystem to consolidate. You know, the practitioners don't have to worry on how, which tool, you know, it, it is a matter of configuration at, at the end of the day. All right, I'm gonna leave you with, uh, a chilling stat.

So you think back in time, maybe 3% of vulnerabilities were exploited. So, you know, we, it was a manageable problem. Uh, somebody ran a report the other day saying that Jet GPT can now, um, create an exploit for 80% of the known vulnerabilities that are out there.

So you can assume that bad guys are gonna be using LLMs to create exploits for things that are known vulnerabilities. And we have not done a great job of addressing all those known vulnerabilities, not to mention all the unknown ones that seem to pop up on a regular basis. So this is gonna get very serious very soon.

So it's a little bit of a wake up call. Hey, folks, we'll be back in a minute. Discover Textron Group, the epicenter of tech innovation.

We are your go-to for reaching IT leaders and practitioners worldwide. Our secret impactful content that sparks awareness, engagement, and top quality leads with us. You'll access editorial websites, streaming videos, virtual events, custom content analyst research, and more.

Join our satisfied clients. Let's revolutionize your tech journey. Contact us today and tell your story to the world in the most powerful way with Textron Group.

All right, folks, we're back with our final topic of the day. And Terry has an article up on Security Boulevard talking about CSO salaries. And my gosh, some of them might be actually making a million dollars a year.

This is unheard of. Is this gonna be like, you know, the new standard, or is this just a, an aberration? And why is everybody paying so much for CISOs lately?

Terry? What's going on There? Um, ion's research did this.

Um, and, um, yeah, so, uh, let's back up a minute to the beginning of the pandemic and, and, and sort of six months into it, um, the CISO was really lauded, right? Um, as, I mean, it, it's super important getting a seat at the table. Finally, um, you know, there, that was the conversation because, uh, businesses elevated them.

They finally understood the importance of security. And because their security had often been an afterthought, you know, with their cloud transformations and things like that, um, they're starting to realize they've got these security holes. So elevate the CISO and, uh, and, and, and these guys can talk to the boards now.

And so that looked good. And, and maybe a couple years ago it looked like that was waning a little bit, right? It looked like maybe that had been a temporary uplift for the, the ciso.

And, uh, but apparently that doesn't seem to be true, because this research, um, has found that, um, half of the CISOs in companies with revenues of $20 billion or more, um, are at the EVP or SVP level at this point, which is an, an elevation from where they were. And the average salary overall is $700,000. 4 million.

That's considerable money. Makes me think, again, I'm in the wrong profession, but, um, I get to write about the, the million dollar babies, um, instead of being one myself. Um, I think there are a lot of reasons for it, Mike.

Um, one is, we touched on in sort of the first segment. Um, these guys have a, and, and women have a lot more, uh, responsibilities now, a lot more falls on their, their shoulders. And that's where, uh, a lot of the compensation is coming from.

They're, uh, hooked into the, the business side in addition to, you know, the nuts and bolts of security. They're hooked it into, into the business side. Um, and they're responsible for risk, for example, you know, that, that, that falls to them, um, a lot more.

So this is where the compensation is, um, coming from. And whether it'll last or not, uh, probably depends a lot on the economy. Um, and whether budgets can sustain that, you know, that's, that's gonna be one element.

It depends on how easy this becomes, um, you know, for other folks and within an, a security organization to assume some of these responsibilities as well. So is this gonna be like the new head coach job where, you know, I make, but, but, but I only have the job for three years before I get fired, and then I'm back on the thing. You know, I could be like, you know, the head coach of the New York Knicks, I, you know, five seasons and I'm out.

Right? Right. You know, it's funny, I was thinking the same thing.

I was thinking about how this is on, in a weird sense, like kind of free agency, and these are like coveted positions. And the reason why, when I saw this story, Terry, it, it, it sparked me because in the last couple of weeks I've been talking to, uh, new newly minted executives at places like Salesforce and Snowflake, and these are, uh, approximately C CISOs. They've just joined.

They're highly touted. They're, they're try and when a company insists that you talk to certain people within the company, that usually means they have long-term plans for them. They're gonna elevate them, and then they find them importance.

And I find that significant, um, especially ba based on what we've been talking about today. I mean, these, these attacks are becoming more sophisticated. They're harder, they're harder to trace.

Um, there, there's more data on the attack surface. Um, it, it's funny, and I, I think we might see, and I'm not saying it's gonna happen, but I wouldn't be surprised if we see some of these folks kind of coming and going from one organization to another based on their expertise and based on the, um, how much they're, they're coveted. Um, but I actually think, going back to COVID, that's a very interesting comparison.

'cause I think it is happening. And I, I'm running, running into it all the time now. I was just gonna say, I do think you're right about the comings and goings.

Uh, maybe too, we'll probably see a lot more movement, um, among CISOs. But also I think this study, um, also said that in this position, and I think it might be overall not at one company in particular, but they're staying in the CISO role role for about 11 years. Um, and then, you know, it'd be interesting to see what they're doing beyond that.

'cause it does sound like that's a, you know, a launching point for maybe it's early retirement. But, um, they're, uh, I, I think, you know, to look at what they're sort of doing, uh, beyond that. But I do think you're gonna see a lot of movement.

And then what does that mean for security when you have your CISOs moving, um, uh, a, a ramp? I would like to also bring one more point, like if, uh, it is obvious to say that, you know, the financial impact of disruption, you know, compliance, legal, legal obligation, legal fees, all that regulatory, you know, coming in is going, uh, also up, right? So it's, uh, directly proportional to you what you see from a salary perspective of CSOs, right?

If you see the CISOs and the cyber software insurance, for example, has some intersection points, right? I mean, the role of a CISO is also changing in that context. And probably in the larger, uh, you know, or a broader perspective, I would see that, you know, these people also become responsible for cutting down the cyber insurance, for example, for companies.

So if you see the cyber insurance market, it's $50 billion market, right? So I mean, comparable data, I mean, if you have a CIO who is working towards that financial impact that can be, uh, linked to, you know, the exponential rise in salaries of these, uh, CISOs as well. Hmm.

John, you're at a security conference in Las Vegas. Are, you've seen a lot of bling flash. What's going on, man?

Yeah, you know, um, yes, I'm, you know, it's funny, I, I was going, uh, I, I went to the keynote, but I, but I really wanted to see some of these sessions. And what was striking to me about the sessions was that they were usually, I mean, obviously Zscaler security related, they were all packed. They were packed with, and I'm looking at people's badges.

I'm probably not supposed to do this, but I'm, I'm looking at them, and these are people at Fortune 500 companies. I'm trying to assess who the audience is, and they are asking really interesting questions. They are completely engaged.

And I want to go back to one thing about the ciso. I'm, I'm so glad we're hearing more about this than remember back the, a couple year, even a year ago, like the chief AI officer, like that era came and went. Thankfully, I think, and I, I think this is more interesting and it, and it's driven home to me at this show, is this idea that security is being taken far more seriously.

And I also mentioned that because one of the more, um, influential financial analysts who's always on CNBC made a decision to actively come here and talk to the customers and assess what they think of what ZScaler's doing. And, and I, I think it's important enough for them to see this. And, and, and, and just, just even to broaden the, the topic a little bit, we talk about, uh, AI being in an inflection point.

We always talk about this, it seems, but the two biggest worries that come up over and over again are security and the impact on the environment because of energy use. And those two topics are always, were mentioned here repeatedly. So I just, I just wanna throw that out because I think it bears scrutiny.

Yeah. Uh, so I think, I mean, I think it's good news that security is becoming more foundational, right? And people are recognizing that and as an, a business accelerant, um, as well.

Mm-hmm. But it's also evolving paradigm, right? I mean, the threat factor is always expanding.

So that is another area to watch out for. I was also looking at like, uh, recent certifications, for example, for these cs, uh, CISOs and, you know, it has a catch up to do there, right? I mean, there is a lot happening if with AI and, you know, AI regulatory and policy and governance and then, you know, you're leaving behind the education and the, the certification aspect as well.

So Terry, I don't have a right answer for this, but we were talking about this on another show earlier. Um, there was a report that says, uh, you know, it's from fu um, security is now on average 11% of the IT budget. Uh, I can remember when it was 5%.

And so some people are starting to say, you know, is that a sign that things are getting outta control or is that a sign that says maybe we're just finally spending enough on security Depending on the organization? Maybe a little of both, but, uh, or, or it can go both ways. I think, you know, it's interesting in this particular study, um, the, the, the CISOs at, uh, the under $20 billion companies are not as satisfied with the budgets that they have and the percentage of the budget that goes to, uh, security.

They're, and, and the other resources that are at their fingertips. They, they don't feel like that. Now these, it seems to be these people at the $20 billion and more are satisfied, um, or at least are like, happy that the, the budgets have ticked up, right?

Um, but I think, I think just, again, some economics play into this, but I honestly think there's gotta be, you know, some point which, you know, there's a little bit more to your security budget because there's a lot more to do now. And automation can't solve all of it. You're gonna have to, there has to be people, you know, involved and, um, in this point to where the, uh, the budgets don't get bloated or get spent on the wrong thing.

That's, that's one thing that has to be considered every year. You go to these big shows, right? And you see people hawking their wares and you see people scooping up, maybe it's not as bad as it used to be, but they're scooping up all these different solutions to solve the same problems, and they come back the next year and the problem hadn't been solved and, you know, they just buy more stuff.

I mean, maybe that's the American way to just buy more stuff. And, um, but I, but I think it's gonna be the wise expenditure of dollars too. So maybe the percentage of the, of the budget that goes to security matters less than, than it's to how you spend it.

Yeah, it's a double edge sword, right? I mean, you, if you separate the budget, then you are also defying the whole concept of security being embedded, uh, into everything, right? So it's, we have to be very careful with this.

It's true. But to your point, Karima, you know, and I've talked to people about this, they, you know, they, you hear the whole craze about ship left and they're like, with more responsibility for cybersecurity will be shifted left towards the developers. And the developers are like, well, how does that actually translate into something?

'cause the tools I have, crap, they don't work very well and, you know, they feel like they got more responsibility and they start using phrases like, it's not chip left, it's s**t left. So how do we fix this? I'm sorry to break this news, uh, but I think I've said it earlier as well, that the substantial amount of change practitioners would see in their, you know, what evolution is happening for software development, more and more security becomes their prime job because everything else can be done by ai, to be honest.

Well, I, I will say the one other interesting thing in, in researching this story, um, is there is an a little bit of caution here. As CISO's move up the chain, right? And they have the ear of the board and they're making these big salaries and they're more responsible for some businessy type things.

Um, there's a, there is kind of a, a, a worry or concern that some of the nuts and bolts of security will get left by the wayside, um, as they become these sort of figureheads and, you know, big executive types, um, that the, the actual security won't be, uh, attended the way that it should be. So it's, that's more evidence that you need a really strong team underneath. You can't just be one person, the ciso you know, that has all the knowledge and, and all the chops, right?

It's gotta, you gotta have a team that can, that can execute and execute some ways independently. And this ISSO doing something. All right, I'm gonna leave this conversation here, but I did get another piece of advice from somebody early on about cybersecurity and I thought it was good advice.

I said, if you can imagine it, somebody's trying it. So think about all the things that you could find a way to get into your system and assume that somebody else is also trying that very thing and go figure out how to secure it. Folks, I want to thank you all for sharing your knowledge and insights today.

Another great show. I want to thank everybody also for watching today's show. And please stay tuned for the rest of the text on TV lineup 'cause it too is equally awesome.

And we'll see you all the Mark.