Challenges of Kubernetes Clusters in Production Environments – Mehran Farimani, RapidFort

This is Textron tv. Hey guys, thanks for the throw. We're here with Mehran Farimani, who is CEO for Rapid Fort, and we're talking about the state of Kubernetes security.

There's been a spate of vulnerabilities discovered recently, and to some degree it's not surprising, but there's work to be done. Mayor, welcome to the show. Thanks, Mike.

Great to see you again. We have, uh, encountered these latest vulnerabilities. They seem a little more challenging than some of the other ones that came by, but they're in, in the ingers controller space, and there seems to be a lot of parts involving Kubernetes that, uh, we're discovering vulnerabilities.

So what's your sense of a, how serious is the latest string of 'em? And B, what is the overall state of cybersecurity these days? Uh, the, they are serious vulnerabilities, but they've been mitigated and, um, you know, the, the mitigations are available, um, and it's super important for people to keep on top of things.

And, uh, there there are no patch. 9, I believe, uh, in ingress controller, uh, in order to use the mitigations, uh, for these three vulnerabilities that we're talking about. Um, but it's kind of expected.

These are super complicated, complex systems with, you know, millions if not, if not hundreds of millions, if not billions of moving parts. And it's just expected that, you know, there are bugs of this type, um, that are gonna surface, whether it's Nix or any other platform. Um, you know, we're gonna see them.

These are different components that talk to each other different, uh, they're written by different people, different completely different teams. Um, and, um, as we have more software, we expect to see, uh, more of these things getting discovered. And we have a lot more software every day.

We just continue crack cranking our code and getting software to solve our problems, and they come with bugs and problems. So, um, I think it's, it is a, it's a new norm. Is it your sense that the bad guys are starting to target Kubernetes more because they're starting to see these clusters in production environments, and we've kind of crossed some sense of Rubicon in terms of security?

Of course. I mean, as, as anything becomes more popular, becomes a larger target, especially since Kubernetes runs such critical parts of, uh, any infrastructure with potentially access to all sorts of, um, uh, you know, infrastructure and data and, uh, and sensitive data. So it's, it's, uh, it's, it's an increasingly, um, hot target for attackers.

Um, it's also a complex system to, uh, not only to build, but also to configure and maintain. And that allows, you know, that leaves room for mistakes and configuration, mistakes and oversights and, um, and especially in, in large, um, sort of distributor engineering organizations that we, we have today in our, in our, um, um, in our world, um, it, it's hard to coordinate all of these activities, you know, in a very sort of tight air gap manner. It's never gonna happen.

So, uh, and, and add to that, you know, deployment into cloud and all of the, all of the benefits and, uh, issues that that brings along with it, uh, it's, uh, it's a problem that, um, that makes it a sweet spot for attackers to, to target. To your point, Kubernetes being probably the most complex platform that find its way into mainstream enterprise IT environments in recent memory, um, there are gonna be lots of misconfigurations. Is there some way to kind of mitigate some of those issues upfront?

Are there best practices or what should we be doing that maybe we're not doing enough of? Uh, well, it's important to, um, to, to use. There are some really good tools and, you know, every time there are problems of some sorts, you know, we create tools to, to try to detect these ahead of time.

Um, and it's important to use, to use the right tools. And there's a lot of good open source tools available as, as well, uh, that are very UpToDate. But it's important to use the tools.

Um, it's, it's important to follow some of the best practices. It's imp important to have gatekeeper, um, for example, or, or something equivalent to enforce policies. Those are our good practices.

And, um, what I see, uh, over the past, uh, year, particularly the past couple of years, you see, um, a much more awareness, uh, within the SRE teams and, and teams that manage these complex clusters about these best, best practices where they're getting deployed, you know, things like gatekeeper are getting deployed, um, you know, basic principles of, you know, what you allow, uh, a particular container to do and not to do, and so on and so forth. Those are getting, uh, better understood and deployed, um, uh, or, or, or, or enabled in runtime environments. So those are important to keep on top of.

And, um, that, that's sort of like your first, um, set of things that you wanna do. Secondly, it's important to keep up to date, but what's going on? I mean, these aren't zero day attacks, but these are new vulnerabilities that are serious vulnerabilities.

And we're gonna have, um, we're gonna find more of these. We gonna, you know, there, there's no end to this. So to keep on top of, um, you know, what's coming on and if, you know, making sure that your systems are updated, that you understand where your medication is coming in, um, and, and apply those mitigations quickly, that's really important.

I feel like staying current is a lot harder than people appreciate. 'cause, um, I'll talk to people and some of them are running versions of Kubernetes that go back to one 12 or even earlier, and they're reluctant to upgrade because they're afraid things will break. So how do we get out of this kind of vicious cycle?

In theory, everybody's supposed to be only three releases behind, but I would say 90% are already at least six releases behind. Yeah. You know, I, you have a lot more visibility to, to that than I do, but that is, that is a, that is a no-no, you, you've gotta try to stay up to date.

Um, and we can't possibly expect people to sort of, you know, uh, maintain software, especially big software that's, that's so, so, so outdated and keep providing security patches and and so on to that, to it. So, um, but that problem is, is easier that that solution is easier, sort of like preach than done, especially when you have, um, such shortage of skills, um, uh, and, and bandwidth to, to deal with these kinds of things because, um, you know, you in practice, you go to the, to any sort of an organization that has an SRE platform team, these guys are, have a, a really hard task already managing hundreds, if not thousands, if not tens of thousands of applications, you know, thousands of namespace potentially, you know, coordinating all of that activity. And then they need to upgrade software without breaking anything.

It's a hard task. So, um, it's a, it is a very valid point. I don't know what the answer is, but, uh, it's something that you need to invest in, um, and maintain your good people that you train and you hire and make sure you don't, don't go anywhere else and, and pay 'em well and treat 'em well.

And, and, and keep up to date. The expertise required. There's some level of knowledge about cybersecurity that you need to have.

What is the relationship between the SREs and the cybersecurity teams? Or is that getting better or are the security people part of the SRE slash platform engineering team? Or are they consulting and trying to make the SREs smarter about this stuff?

What's the path forward? Um, there are definitely, uh, there's definitely a lot more awareness about security issues, um, within the platform teams. And then that automatically means that there's, there's a, there's a more of a dialogue between security teams and platform teams, but a lot of times, you know, a security team wants to deploy a solution that's additional work for the SRE and platform teams to go and, and put in there and maintain and, you know, and keep up to date and all that kind of stuff.

That's just more software on top of all the software that the business is running on. Um, so there is some of that that's going on that, that, um, uh, I wouldn't say it's, it's a source of friction, but it, it's a matter of, you know, um, how quickly as a security team, how quickly can I deploy, uh, this hot new tool that I wanna deploy so that I could get, you know, I could control and or manage or observe my infrastructure better. It's not an easy, um, there's not an easy answer to that in, uh, any organization of decent size.

Can we find enough of that expertise? 'cause I feel like sometimes we're looking for the perfect unicorn, somebody who knows security well and Kubernetes well, and that's really hard. So, is there something to be done on training that we need to think about here, or, you know, have we kind of rushed ahead without enough forethought about just where are we gonna find all these people There?

There's, uh, obviously CNCF provides a, a, a number of really good, um, training material for Kubernetes, um, that's cloud native Computing Foundation. Um, what's, what's sort of encouraging is that you're, you're starting to see Kubernetes, um, uh, being taught at universities, um, slowly, which is, um, so, you know, looking down medium term, a few years down the road, you'd expect to see a lot more, uh, expertise in the area. Um, but in the meantime, it is a complex environment and it's a complex system, and, um, you just have to get people and get them trained and, um, and like I said, treat 'em well and make sure they stay there because the, the switching costs are hard, are high.

What do you think might be the ultimate impact that AI might have on this equation someday? Can't walk down the street without somebody leaping out to tell you about it. And clearly it's gonna be applied to the management of Kubernetes and security, but where's the opportunity?

Um, it's, uh, I, I suppose, I mean, there, there, there's definitely a lot of, uh, um, potential in a lot of ways that you could think of that you could apply the new, uh, sort of, um, uh, advances in AI into solving this problem in practice. We have to wait to see what, um, what people are inventing and how practical they become. A lot of the problem, uh, um, where the adoption is, is learning how to operate, operationalize a, a new solution.

And so what, what does that look like when I, I have a new AI system that manages the Kubernetes automatically. What does that mean? Where does it plug in?

Who deploy this? Another piece of software that I have to run? Um, who, who manages that?

So who maintains that software operation? Where does it plug into my organization? Um, and all that kind of stuff.

So those are, those are some big questions that still need to be sorted out, I think. So it is not only just coming up with a solution technically, but also how does it fit within, um, the, the, the, the sort of prominent way that organizations, um, are structured today and, and, and operate. What's your best advice to folks, therefore, as you kind of look at this situation and what should they be putting in place to kinda help them get through this?

Um, uh, I think it's important to keep, uh, to, to build a team that is, um, you know, continually growing their expertise in these complex systems. Obviously both internally understanding what the company is doing, what are the kind of applications that they're, they're producing, what's the cadence, what's the development velocity, all that kind of stuff. But also match that up with the expertise of managing the deployment of those applications in these complex environments.

And, um, and having, making sure that you have enough, um, labor bandwidth to keep up to date with the latest releases, to keep up to date with the, with the, uh, security issues that are found mitigations. And then on top of that, uh, work with the security teams to deploy the right tools. Because at the end of the day, you need to be able to see these, these things in order to act on them.

And if they're sort of lurking away, you didn't have the right tools to automate or to observe to, to mitigate because you were too busy with, you know, deploying, you know, uh, keeping the business running. It's gonna come back and bite you later on. So, um, and the, the, the, uh, the core of it is, is I guess the message is to create more bandwidth in your platform and SRE teams so that you could, um, stay on top of these issues.

Are we looking for people who don't exist to a certain degree? Because I'll see job openings where they're looking for people with, you know, six years of Kubernetes experience and four years of security experience. And I kind of shake my head and go, I think maybe we should just go look for people who have an appetite to go learn a new platform.

So maybe we need to just rethink who we're looking for. I, I, I absolutely agree with you. You know, you, you take, these are fun systems to learn.

I mean, you know, it's the, the, they, it's a pretty amazing, um, system. So for anyone who has a knack for tinkering and, and wanting to learn something new, it doesn't take that long to, to become pretty good at it in, in the order of months. Um, and just, just get, um, you know, people who are technically strong and have an appetite for learning and, and put them on there and start investing in the future.

Um, it, it doesn't make sense to look for people with, if you can't, if you could find someone like that, obviously go ahead and do it. But, but then you might be looking for a while. All right, folks, well, you heard in here there's always gonna be security issues when it comes to Kubernetes, and there's probably gonna be even more as we move up the stack and get our arms around all kinds of other interesting technologies that sit on top of it.

But maybe we just need people who have the appetite to go fix these issues and delve into them because they're complex, but there's no magic silver bullet. Hey, Myron, thanks for being on the show. Thanks for having me, Mike.

All right, and back to you guys in the studio.