Most Container Images You Pull Today Are Already Full of Vulnerabilities

Hey everyone, welcome back here to techstrong TV. I'm really happy to have my next guest on. I know him for a long time.

I'm trying to think back to when I first met John Morello, Twistlock. 2015 or so, basically. Yeah.

com, right? Cloud Native was just starting to come on the scene. And, we've seen John since then at Palo Alto and a lot of other places.

But let me reintroduce you to my friend John Morello. John, it's good to see you. You look great.

How's everything going? Same, Al. Good to see you as well, dude.

Everything's going very well. Thanks for having us today. My pleasure.

So John, when we last left off, I think you had just left Palo Alto, and you guys were doing something with Minimus. Yeah. So, what we've built with Minimus is really trying to solve a problem that Twistlock itself really didn't solve.

If you think back to the beginning of the whole container security era, one of the real fundamental requirements with container security is finding and remediating software vulnerabilities in those container images that you run. Yep. And Twistlock really pioneered that space and did a lot of other things with runtime and so forth as well.

But, while Twistlock and lots of other tools that have been created since then, and are currently used, do a really good job of helping you find vulnerabilities. Think about it, you use Trivy or Grype or Wiz or Prisma Cloud or whatever it may be. A lot of products can show you, here are the vulnerabilities that exist in your images.

Which is good. I mean, it's important to have that visibility. But for most people, the problem is not finding the vulnerabilities, the problem is that they're unable to keep up with updating those images rapidly enough to actually remediate the vulnerabilities.

And that's become something that's even more of a problem today versus just a few years ago with the usage of all this AI for offensive security research that's finding vulnerabilities at an even faster pace than had been done before. And so the challenge that people have is, even if I can scan my registry, scan my images, find these vulnerabilities that are there, if I don't have something that can make it easier to stay in front of them, ultimately I know that I'm about to get into a car wreck, but I don't have any brakes or any way to avoid it. And so what we're trying to do with Minimus is to give people a real solution to that problem by just giving them images that don't have those vulnerabilities.

And we're not doing anything magical here. What we're doing is really just following a lot of long-held industry best practices, but doing that on behalf of our customers so that we do all the hard engineering work and they don't have to. Specifically, what we're doing is really two main things.

One is we're building images that are very minimalistic by default, that only have the components that are really required to run the application, and not all the other superfluous stuff and its second and third order dependencies that make for images that are really big with lots of software that all accrue vulnerabilities. So if you look at, as an example, the latest official Python image on Docker Hub, it's like 400 megs. It's got like 100 some odd vulnerabilities.

There's like 70 of those are high and critical. So I mean, it's a very hard place for you to start as a customer if you're already that far behind with the official image. So what we're doing is we're giving you something that's got a lot less stuff.

Like literally our image is like 5% the size of the official one, but it's the exact same Python software because we're building everything from source directly. And that's the second piece is we are doing those continuous builds for people by looking at literally tens of thousands of individual open source projects today at the source level. Whenever there's new versions of those things in source, we pull it, we compile it, we create the images that need that component, and we publish those just normal compliant images to a normal registry for customers who are simply using our images versus the ones they already have today, can drop their vulnerabilities by 98 or even 100%.

And that's fundamentally what the product is all about. Love it. I'm sitting here listening to you, and I'm thinking, does everyone out here know Twistlock?

They even remember. Right. So Twistlock really pioneer in the, as you mentioned, in the container security cloud native space acquired by Palo Alto.

John, you were co-founder CTO at Twistlock, but you're not the only Twistlock person in Minimus. Who else we've got there? No, like, literally the majority of our team is, so Ben Bernstein and Dina Stopol, the two co-founders, same roles at Twistlock.

Our chief revenue officer, Brian Lake. Lots of people in our engineering organization. You didn't get Chenxi back in, did you?

Chenxi's an investor for us. Now Chenxi's gone on to bigger and better things. Yeah, I know.

With Reign Capital, so she's one of the investors in this. I thought she was coming out of retirement for this one. Yeah.

No, I think she's got a pretty good setup now, so I'm going to- Yeah ... let her keep working for them. No, she's not complaining.

Interesting, but that is great because, look, you can't catch lightning in a bottle twice, but when you put good people together, good things happen. John, the issue of secure container images. Look frankly, we've seen companies try this before, right?

Offer this before. I know my friends at SUSE have like a bunch of Linux packages that they've containerized with Rancher and everything, and they use that. I want to say it was Mirantis who took over the Docker stuff, has it?

Or was it Docker themselves? It's kind of changed hands several times. It was with Mirantis.

I'm not sure who really owns the server components of what Docker was. Currently, Docker, I think, is mostly focused on the developer experience and- Yeah ... capabilities like that, not so much like the server-side pieces, which as far as I know, still was a Mirantis or owned by Mirantis.

Yeah, I thought they were trying to advertise something with secure Docker images, basically. Yeah, Docker does. Docker does have a service called Docker Hardened Images- Right ...

that has the same top-level sort of promise that we try to make. But what we're doing that's unique relative to DHI is we're building everything directly from source and doing that continuously. There's basically two main approaches you can use to create these images.

One, the approach that we take is building everything continuously, directly from source, on a distroless base, and that allows us to have full control, and full accountability for that matter, for creating the smallest possible images and maintaining them with the highest possible quality because we're fully responsible from the source release to delivering the image. The approach that Docker takes is a lot different. It's more of a reductive approach.

They get an image from somebody else that someone else is building and maintaining, and then they take it, and they try to strip some things out of it. Maybe they update some components in it, but fundamentally, it's not an image that they own end to end. And while that's an easier service to probably get off the ground because there's less engineering work for Docker to have to do to do that, I think what you run into is that you just can't provide the same kind of security assurances, the same kind of low attack surface images, all the sort of benefits that we can because we did make that investment to build something that's more difficult but also a lot more scalable and something that we fully control end to end.

So that allows us to offer the best SLA in the industry for remediating vulnerabilities, and to be able to provide images you yourself, your viewers, can easily see. Just pull images from Docker Hardened Images, pull images from Minimus. You can scan the two.

You can see the differences in size, but even more importantly, the differences in the vulnerabilities that exist. I love it. Now, John, where are these images stored?

So we build everything ourselves in a pipeline that we have created that leverages a lot of capabilities in GCP and things in GitHub. We leverage AI technologies like with Claude, for example, to help create some of these image recipes and to automate the software maintenance within them. dev, which is actually a front end for Google Artifact Registry, so we get the global distribution and caching and benefits and so forth from that.

And one thing that I think that got us talking here is the Community Edition of Minimus that we just announced last week, that was on the front page of Hacker News for a whole day and has really gotten a lot of interest for us. Those images in Minimus Community Edition are the exact same images that are available to our Enterprise Edition customers. And so we have hundreds and hundreds of images that are available, including FIPS images, all built on that distroless base, all built continuously from source.

There's no auth wall, there's no sign-up, there's no payment for the Community Edition images, and we've really optimized everything to be very friendly to work with if you're an agent. Because one of the things that we've seen with our customers is there's more and more uses of agents for developing software. " And you can literally allow the agent to do almost all, or even in some cases, all of the work to do that, and really take you from having maybe hundreds or even thousands of high-end critical vulnerabilities to just not having any, simply by replacing these images.

And so our Community Edition product is the same images available in Enterprise Edition. Our Enterprise Edition is still a thing that provides people with support and SLAs and enterprise features like single sign-on and our custom image recipe creator and self-hosting and integrations and so forth. But the Community Edition images, the reason we made those available to everybody was really to make sure that any customer, any open source project, basically anyone that might be using open source software, has access to the most up-to-date, best-maintained container images available.

Now, John, part of, I think, the issue, though, is everyone has their catalog of images. These are the things we think 80% of people are going to use. It's always the other 20% where stuff happens, right?

What's the commitment from Minimus in terms of keeping that catalog not just up to date so that the ones that are in there are up to date, but adding new container images of new ... apps and so forth, new functionality. Yeah.

It's an important thing, right? If we just had the same stuff and it doesn't meet everybody's needs, as you note, it's a long tail sort of problem. Yeah.

80% of stuff may be common, but that 20%'s very distributed. So, we're creating images constantly. Literally every day, we might publish 15, 20, 40 images.

That's based just on what do we see through the intelligence that we have out there, looking at what exists, maybe at other image providers, just popular open source projects that are coming, that are being used more. So we have a lot of telemetry sources we use, and then those telemetry sources feed into the AI pipeline that we use to actually create the recipes for those images. And, usually that AI is able to get an image recipe for us to, 90-plus percent complete.

We have a human developer review it and make any sort of last-minute polishes that are required. But that allows us to be continuously creating images, both preemptively, before anybody asks for it, and also today, when we have customers that ask for some new image that we don't already support, usually we can give that to them within the next day or two. So, it's growing that catalog and being able to not just grow it, but to keep the same standards of quality and timeliness in terms of the updates as it grows is one of the key things that we provide to our customers, because otherwise it's really hard for you to do.

Conceptually, all the stuff I've just described, you could say, well, I could set up whatever pipeline I like to use in my own DevOps environment. I could set that up, and I could build things from source and could create images and so forth. But that's true.

We're not claiming to have some kind of magic here. What we're claiming to do, what we're actually doing for you is, that's very hard work to do that reliably at scale and to make that work in a consistent way. And even if for some reason you're able to do that for some of the images that you use, if you've got the expertise to do that, you could almost certainly apply that developer expertise on things that are going to generate more revenue for you, serve your customers, serve your constituents or internal teams or whatever better than just repackaging open source software.

And so the whole promise is offload that responsibility to us. We give you contractually backed SLAs for how we're going to do it. We do that at scale, and that allows you to now say, look, we're going to just simply take advantage of these images and focus our own time and attention on higher order problems that these images are going to help support our ability to solve.

And that's really what the promise is. I love it. Excellent stuff.

John, walking, not walking away, but beyond these secure images, of course, though, Minimus is a company that, look, it's serving large enterprises. You're at this now about what, a year and a half, two years? Yeah.

The product's been out of stealth now for a little bit more than a year at this point. When you guys got together and started working on all this, was the rising AI threat landscape as real to you as it is now? No, I don't think for anybody, if you just go back two years ago, there was talk about it, and there was maybe some cases of it being used, but it was a lot more theoretical and experimental than it is today.

I think today it's pretty clear that's one of the things that current implementations of AI tend to excel at, and there's obviously lots of motivations from nation states to nefarious actors and so forth to find vulnerabilities. And so it's a clear risk to organizations that use open source software to how do you keep up with that kind of ever-increasing flow of vulnerabilities? Like I said, people weren't doing a good job of that in general before when the volume was the way it was five years ago.

Sadly, you go to most large enterprise organizations, well-known name brands and governments and so forth, if you look in their cloud environments, you'll probably find many hundreds or thousands of CVEs that impact internet-accessible services that may have been resolved years ago even, simply because it's so difficult for them to do the engineering work to validate compatibility and just keep things updated and so forth. And so, again, anything that you can do to outsource that problem to someone who specializes in solving it, just frees up time and effort for you. And that's to me, the foundation of what a good business is, is giving somebody something of better value, more value than what they're paying you for, and for that still to be a valuable thing for you.

And, in that case, the balance of trade ends up favoring everybody and that's how you build something that people really love and use. I love it. Hey, John, we're about out of time.

I know we had some hiccups earlier, so the clock ran on us. But Minimus, the website for people who want to get more information. io.

That's M-I-N-I-M-U-S, right? Correct. John, look forward to hearing more about Minimus now that you're back here in the saddle again.

Say hello to all of the Twistlock team, or the old Twistlock, now Minimus team for me, and we'll hopefully see you Black Hat, RSA something. Yeah. We'll see you around.

We'll be there. We'll certainly have booths at all those things, and KubeCon and so forth. So come see us.

Absolutely. We'd love to show you the product. We'll be there, too.

Of course, we'll be there, too, doing videos, so make sure we hook up. Sounds great. John Morello, co-founder.

Thanks, Tom. Great to see you. Thank you.

John Morello, co-founder, CTO, Minimus, here on Techstrong TV. We're going to take a break. We'll be back.