Unlocking the Power of Jenkins CI/CD – Shawn Ahmed, CloudBees

This is Textron tv. Hey guys, thanks for the throw. We're here with Sean Ahmed, who is Chief Product Officer for CloudBees, and we're talking about a new major update to Jenkins and a major new platform that's on its way, and we'll dive into that in a minute.

Sean, welcome to show. Thank you so much, Michael. Appreciate being here.

Jenkins is still the dominant c I c D platform out there, but it's been a while since we've seen a major update. So what's driving that update now and what should folks be looking forward to? Oh, that's a good question.

Um, well look, uh, I think, uh, Jenkins has continued to explode, as you said. And, uh, the usage of Jenkins is, uh, is, is through the roof. Uh, it continues to grow, uh, almost every single year.

You know, you think about, you know, jobs defined and self-reporting back, uh, to headquarters, Jenkins io and the stats that are there that you, everybody, anybody can see. You know, you've seen that, uh, it's increased from 2021 to 2023. It went from like 50 million jobs reporting back, uh, to 73 million.

That's, that's a pretty explosive growth. If you think about just pipeline jobs in itself, that's gone, uh, from 27 million to 48 million, that's a 79% growth. So by all measures, if you think about sort of Jenkins self-reporting data coming back, and this is not even including any of the enterprise behind the firewall usage of Jenkins, you see that the Jenkins growth as as really, uh, exploded.

And we think about things like the community. We're looking at a million lines of code. We're looking at 130,000 poll requests.

Michael, we're looking at 20,000 issues more than 600 plus active contributors. It's clear that Jenkins is continuing to grow in a way that is generating massive scale. Um, this, this deployment sizes are things like seven.

Uh, we, we talked to one user the other day. They have 7,000 plus controllers. Uh, we talked to another customer the other day.

They have 750,000 jobs across six controllers. And when you're looking at that kind of scale, there are, there are a couple of things that stand out at that point in, down that become challenging, right? Like when you have thousands of applications, you've got tens and thousands of developers using it, and you got large complex tool chains, massive scale becomes a challenge.

How do you run this in a fault tolerant way with absolutely high resiliency, with self-healing? How do you, how do you maintain Jenkins in that sort of massive scale? And that was a problem that we were focusing on.

And when we, uh, started working on this about a year ago, we said, we wanna create an experience that is painless for the developer systems that just are up and running all the time. Uh, it should be performant, it should be fast, it should be an experience in where Jenkins administrator can, you know, sort of horizontally scale jobs as required. And, um, and so we decided that we were gonna get hard to work, uh, Michael, and we created what we think is the biggest update in performance and scalability for Jenkins in a decade.

Uh, and it's quite literally exactly that. And so that's something that we're releasing, you know, here in four weeks. And it's got some phenomenal, uh, capabilities built into it that I'm happy to get into on this call.

Well, before we do that, one of the criticisms of DevOps has always been that it doesn't scale as well. And there's a longstanding debate between the DevOps and idle community over this issue. Do you think we're finally getting to the point where we can put those issues to bed and we can say, Hey, DevOps does scale and therefore I don't have to make a trade off between agility and scale?

I think we're getting definitely to that point of view, and I think a little bit of the market and the usage needed to catch up to kind of see what kind of scale problems we would see with the various types of technologies that are out there and the tool chains that we have. And, and, and we needed to see those to be able to tackle them as they became, uh, presented to us, uh, as, as as companies reached that level of scale. And we particularly see that with, you know, Jenkins, um, but we similarly also see it in the cloud native world where, you know, companies are, are adopting cloud and having, you know, these digital transformations that are going from on-prem to hybrid and hybrid and some level of, of, of cloud as well, which changes the paradigm, not only of just how you build the applications, but also how you actually run the workloads and the type of technologies that you require in order to make that sort of as painless as possible for developers to be able to get the applications into the hands of its users.

So I definitely think that scale, uh, was something that we needed to kind of really see people, uh, achieve first and, and, and see what kind of challenges that would be brought on. And then we are able to sort of address them with the right type of solutions and so on. I definitely think that we've caught up with that and have a lot of really great solutions for that.

And we'll talk a little bit more about cloud native on this call, uh, as well. I know. So we'll get into that.

So in terms of the new capabilities, what leaps out at you most? What's the thing you think that your customers are gonna look at and go, ah, finally we got this? Yeah, So, um, you know, the fun thing is when we talk to customers about this first capability, you know, we always hear for them, oh my gosh, we've been waiting for this for like 10 years.

This is amazing. And it's true, activate high availability mode of four champions controllers. And that's, that is something that we're really, really proud to bring to the market.

This is the ability for you to have an environment in which you can make those experienced developers have, uh, quite painless, um, single sort of logical controller in front, front of the entire Jenkins environment. And you can have failover occur and jobs continue to run, and developers don't know that anything has happened in the background. So this is also very super helpful when you are in situations where you have backups you need to do, you need to take a controller down for maintenance, or you have installations of plugins that need to come, uh, and or updates that need to occur.

Uh, you can't have a developer experience where developers' jobs are not running all of a sudden they wake up in the morning and they're trying to run, you know, they had to pull request, they want some code committed and con uh, and, and, and, and, and into the production environment. And all of a sudden they realize, wait a second, there's not a controller there for me in this situation. There's always a controller and they never need to know which one of those controllers actually executed the job.

The second capability is horizontal scaling, and that's to ensure that you get the performance and resiliency of being able to say, okay, I'm gonna run a job, but as one controller is getting overloaded, you have certain sections you can set that allows you to have jobs run on a second replicant of the same controller, and you can have as many replicants as you would want in a cluster. And this, this allows for jobs to sort of be scaled across and use the, you know, compute and storage resources that you have. And, and it allows you to be able to be, take jobs and, and ensure that they will execute as fast as they possibly can.

So let's, let's sort of feature number one, uh, Michael that we've added. The second feature that we've also added is a completely new workspace caching mechanism for Jenkins. And this is really great.

This is giving you a little bit of control within the pipeline itself to reuse a cache in the workspace. This is a little bit under the hood. What that means is that, you know, when you, you kick off a job and you create a workspace, and the workspace typically in Jenkins will begin with, you know, downloading the various tools that are required in order to build the application, maybe dependency scanning and a number of other things that it downloads.

But here's the thing, when it's done, you doing the job, what happens? It it, it just nukes the entire workspace and then you rebuild it again, with workspace passion, you can actually create, write and read caches between workspaces and share them between jobs in this way. You don't have to restart the whole download process.

Every single dime a job actually kicks off and you can sort of name those workspace caches and read them back in, in our tests, we typically see the performance of jobs because of these availability of workspace caching to go down by 20, 30, 40%, sometimes even 50% at the absolute highest end of things. But that's, that's a lot less compute and storage resource you need as you think about sort of executing a job and be able to cut that down. But most importantly, it's the actual time it takes for developers to be able to run their jobs, uh, theoretically, uh, opening up more space to do more commits, to run more and get more code out into production on Jenkins.

And that's important to developers, right? So you combine sort of high availability, which is active, active and always there with horizontal scaling and workspace caching, not only did you get a much more resilient environment, but you get a far more performant, uh, you know, uh, environment as well. So that's, I think, going to have a tremendous impact on those, uh, that manage Jenkins and those that are consumers of Jenkins pipelines running, you know, and, and, and how fast they'll be able to do that.

So that's kind of the exciting two big features that we have that are in the product. But not only that, we also have a complete new, uh, feature in the, in, in cloud BCI for Jenkins called Pipeline Explorer. And this is a brand new way of actually debugging and troubleshooting pipeline jobs and a beautiful visual interface that allows you to take a pajama and be able to read it up.

You automatically get errors and challenges and things like that automatically tagged within the system and later in the year, and then maybe next into next year, we'll also be connecting that capability to some self-healing, uh, and AI such that, you know, you'll be able to take a challenge that you're seeing inside of the job and be able to use ML and AI to bring back, you know, healing. In other words, uh, you know, suggestions on how to fix the problem that your pipeline might have had as well. So it's, um, that's an exciting new feature as well.

And this is all packaged in the September release of the product available to all cloud P C I customers. Jenkins has been around for a long time and we see AI is coming down the pike. Um, are we on the cusp of some sort of modernization effort around Jenkins?

'cause a lot of these installations have been in place for a long time and they have a lot of extensions and customizations, and some of that stuff is now better handled in other ways. So is it time for everybody to kind of sit down and re-engineer those workflows? I think it's, uh, definitely, um, an opportunity for sure.

I think about a lot of, of, of ML and AI and how it's actually used in the pipeline and used generally within Jenkins and how it executes. I think there's a lot of use cases that we can potentially address. So then when we think about those and when we think about building those, some of them who are actually building in labs, CloudBees labs today, and these types of, uh, jobs we're seeing, uh, and the types of use cases we're seeing have everything for do everything from, you know, to do with like how you are testing your jobs, being much more intelligent, looking at the code and saying what kind of jobs are, uh, testings are actually required.

Uh, things like automatically having unit tests being written for you. These are all things that, you know, actually are working in labs today. So when we kind of think about where and how we could use these capabilities, I think the most important thing is to make sure that we understand what kind of environment and who it is that we're helping.

So you have a classification and use cases for developers that I think are gonna be great. Things like being able to evaluate the code and decide what kind of unit test coverage you need for that, and writing that automatically, which we have working code on. We have the ability to kind of be able to build pipelines on your own declarative pipelines, but also being able to automatically generate the declarative pipeline based on sort of the, the, the project that you are working on or the code that you are working on, be able to provide a golden path from that and onwards into the actual target environment.

And then you have a whole classification of jobs that are typically about the Jenkins administrator being able to see what kind of loads will, will, will drive, what kind of traffic, being able to operate in a much more self-sufficient manner and be able to auto heal from things like outages and or controller failures and things like that. So we think about sort of the H A H S code base that we've built around high availability and horizontal scaling. There's a number of classified use cases there that we have working in the lab.

So we're gonna be releasing this as we move forward and, and, and, and sort of help developers and help the Jenkins administrator be able to monetize their pipelines in ways that will create a far more performant and I think, uh, seamless experience for developers in the next stage. And that's, uh, that's exciting for us. Almost sounds like we're looking at the democratization of DevOps in the sense that it will become more accessible to a broader number of developers and software engineers of varying skill levels.

Oh, definitely. I definitely think that that's the case. Um, you know, in today's world, I think a lot of people come out with, um, you know, knowledge on Jenkins, uh, right out of school.

They might have downloaded it in school, used it in school, and then, you know, first, second, third jobs. You know, you, there's always some Jenkins that you're working on. And a lot of them are probably starting from a very classic perspective in terms of like what they're using and how they're building their pipelines and getting code to market.

But I think that there's a lot of, a lot of work that is possible with ML and AI to democratize sort of how it is that they use those systems and make it easier to sort of on ramp into this. But not only just Jenkins itself, but generally DevSecOps as well as the definition changes, you know, as DevOps is becoming much more, you know, platform engineering type teams and you're trying to pull tool trends together and create golden paths and bricks of automation. Um, I think those types of use cases will sort of be the use cases that will democratize sort of the software development lifecycle in such a way that every developer, no matter what skill level has an easy, uh, experience with DevOps.

And one that doesn't, you know, shift a lot of, you know, cognitive load over onto the developers to know and learn, but in fact just makes it a lot easier for them to adapt and move. You know, uh, many of the developers that we talk to, they, they often refer to this experience as almost a, can you make DevSecOps invisible for me? I know it can be taken away, but can you make it invisible?

I want an invisible experience. And we thought a lot about what that means and, and, and what an invisible DevSecOps platform would do for developers. And not only that, but also for the platform engineering teams that, you know, sort of build and provide those types of experiences.

And I think that that's gonna, that's gonna be sort of the future that we're headed in, uh, you know, um, in in the next few years. Well, speaking of DevSecOps, you have a new DevSecOps platform, so why don't you dive into what that's all about and explain to us how we can make this simpler for folks. Yeah, that's, that's, uh, that's, that's, that's, that's great.

Thank you, Michael. Uh, I mean, this is something that we're absolutely, uh, excited about at CloudBees. We're, we're, we, we've been thinking about this for, for a very, very long time.

Um, and, uh, you know, we, we, we, when we were speaking to a lot of us customers and so on and so forth, uh, developers and, and platform engineering teams, we, we, we, we, we realized that a lot of the future for most organizations is cloud native. Uh, the application backlogs are absolutely far more indexed towards cloud native. And so when we looked at the market itself, we realized that in response to a lot of the complexity that comes with deploying and building cloud native applications, uh, a new sort of discipline was forming in response to that.

And, and I've referred to it on the call today as platform engineering, and I think it's an evolution of the DevOps practice, uh, as it started to include and expanded into including, uh, teams like the SS r e team, the security teams, the product managers, the ops team along with the DevOps engineers, and the together, this, these types of folks coming together in platform engineering described, uh, the next iteration of a platform to be a kind of platform would, that would be like the sum of all the siloed technology and tools that you have in your development of tool chain, right? It's, it's, it's, it's what provides and shows a developer the golden path from, uh, when you thought about a line of code all the way to it being successfully in the hands of its users. And so they defined the set of imperatives, uh, and when they defined those set of imperatives, it was, it was, it was first and foremost about the developer experience.

It was all about put the developer back in the center, think about what it means to reduce that cognitive load for them. Um, second was the platform has to be open and extensible, open and extensible, meaning it has to accept that we have a tool chain that's very, very wide and it's gotta make the cloud native experience real, real easy to deploy to. And the third it would that the development, uh, teams like platform engineering teams, they need to be able to implement and design sort of the self sort of model by developer.

So we need a platform to build our platform, and that is what we've been working on. We heard that and we started working on it, and we're really, really excited to announce what is the CloudBees SaaS platform. It is an entirely new DevSecOps platforms that is built on Teton.

It uses GitHub like D s l plus plus. And it is got not only C I C D automations on techon built into it, but it's got feature flagging. It has V SS M dora, it's got analytics.

It also has progressive delivery built into the system itself. It's got pipeline orchestration, I think of it as the most open and extensible DevSecOps platforms that are out there. And it's got a lot of, you know, modern I D P platform type, you know, capabilities built into.

Its to make the developers experience as easy as possible, and actions is a key part of the platform itself. And the fact when you connect any platform for the new tool, any tool from your DevOps tool chain into the platform, it becomes essentially an action that you can then use in your platform. So it generates and creates these little blocks of automations, is what we call them.

And you can put these automations together to become essentially an entire pipeline, right? Um, it can execute in our environment, it can execute in your environment, it could live wherever it wants to live, but it gives the absolute broadest choice for platform engineering teams in terms of what to connect, what to turn into an action and how to execute and where to execute that. And it does it all in a cloud native way and delivers cloud native application using Kubernetes.

And so it's delivered to our users as a SaaS, completely multi-tenant. Anybody could sign up and use it and, or you can also get a mono tenant version of it, or you can get essentially a implemented V P C install as well. So it's available and meets our customer where it is.

Now here's the thing for Jenkins users, we've built Jenkins and connected Jensens into the platform as a first class citizen, which means it doesn't run on Jenkins, it's not a drop in replacement for Jenkins, but what it does do is allows you to orchestrate Jenkins pipelines. It allows you to do that from the SaaS, from the V P C, from the mono tenant. It treats Jenkins as a first class citizen, and also in the future will allow a series of capabilities that will make those transformations from Jenkins, whether it's a job or it's a controller that you may wanna bring into the cloud a ton easier.

Therefore, creating a hybrid experience between complete cloud native applications and more classic applications like Windows, you know, apps, uh, Mac apps and things like that, that you might be delivering as, as VMs. So this becomes sort of an opportunity for customer to be able to operate in both worlds, but yet have the platform be connected in a way that gives them the best sort of hybrid experience between the two. Right.

Almost sounds like to me, we're bringing the orchestration concepts of Kubernetes and applying it to the pipelines at a more granular level to give people more control over what runs where and when. Yeah, I think it, I, I, I think that's, that's one good way of looking at it. I think a lot of platform teams and platform engineering teams and even developers would look at just sort of the DevOps tool chain DevSecOps tools chain.

And I think, you know, a lot of times what, uh, is very challenging for them is, is that, you know, you got so many different tools and they all have their different D SS L languages. They don't necessarily speak to each other. And so by very nature it becomes very, uh, disconnected and siloed, right?

And so with the orchestration and pipeline orchestration capability within the Ecop platforms, we can provide for a single way for all pipelines to be connected together and execute so that you have a clear golden path across all of those tools from, you know, why road, a line of code to exactly where it actually is and which production environment and how it was released in a cloud native way. And I think that's a pretty powerful concept for most of our customers. All right, folks, you heard it here.

We're kind of entering a new C I C D Renaissance. It's gonna be interesting and exciting. Sean, thanks for being on the show.

Thanks for having me, Michael. Talk to you soon. All Right, back to you guys in the studio.