Panel Discussion: Cruise Control for Software Delivery

So hello, everybody, and welcome back to the next panel discussion, we just finished our first live panel discussion. I hope for those of you that sat in, you enjoyed it and it went well. I think our next panel discussion is going to change the tax slightly in terms of what we want to talk about, whereas we were talking more strategically about big technology innovation that was impacting the automotive industry in our first session.

This one to ever really going to be talking about how we build the cruise control software, cruise control system, I should say, for software delivery. So how do we actually make that software a reality in the automotive industry so that it delivers value to end users and the supply chain. So I'd like to start off by introducing our speakers today.

And I'll start with Andreas Dharmawan, because Andreas was in our last panel as well. So for those of you that weren't here, Andreas is the area Vice President of Software Delivery Transformation, here at CloudBees. But we also have Harald Göttlicher.

Harald is from Bosch and he's the continuous delivery architect of Bosch. And we've been working with Harald for many years. And for those of you that have attended other automotive events in Germany live, you may have met Harald in the past.

So thanks a lot, Harald. Welcome back again. The other presenters we have are Eric Melski.

So Eric was one of our presenters from earlier, and he is the lead up teched in research and development at CloudBees. And finally, last but not least, we had Erez Rusovsky who is a product management director in research and development at CloudBees, and he's the former CEO and co-founder of RollOut. So welcome, everybody.

It's a real pleasure to have you on the panel. And I hope we're going to have some interesting discussions moving forward about how we do DevOps and how we do see IC/CD for the automotive industry moving forward. So let's just set the scene a little bit.

So in the last panel, as I say, we talk about these strategic things, but today we really want to talk about how or in this session, we really want to talk about how we're going to make this a reality for our customers and partners and those viewing the session today As software technologists and practitioners, what resonates with you about the fundamental things we need to focus on to create the software factory of the future? What are the things that, from your point of view, are really important to recommend We start with? So maybe we could start that with Andreas, why not.

I think as the tsunami of software requirements are hitting everyone today, right. And then the hardware is also getting extremely powerful. So there is this temptation to continually adding software into a powerful hardware.

Why not? Because we have here the power to do so. Right.

However, then you can see from the headline news that quality suffer when we push really hard. When you tried to go fast than quality suffer. So what I think is important is actually continually to do validation in every stages of your software development and testing cycle, whether it is just software in the loop or it involves hardware in the loop.

So it is important to actually you put success criteria for every single stage. And that way you can actually analyze how well you are doing for each stages and then you are not letting a bad quality to actually pass through the following stages. So having the ability to actually continually putting a strict guideline on what needs to be accomplished in one particular stage, what is the exit criteria?

And then also what is the entry criteria to go into the next stage. So having a solution that will be able to collect that and force that regulatory security and best practices compliances is actually becoming extremely important to make sure that each stages in your process is always the most optimum one. So, Harald, you're a practitioner at Bosch, you're actually everyday facing the challenge of creating new software and complex software.

why do you feel that CI/CD is important And what does it bring to you and your business Do you think that perhaps wasn't there in the past? Yeah, I think especially in the area of embedded systems and what we also have is desktop software in our department. There are we are still like stuck in traditional development, which is not really agile and not really fast.

And I think this is one of the biggest challenges to get embedded software to deliver faster and in smaller, in smaller atomic ways and test build, test and deliver all these parts in very small atomic pieces so that we can be faster on the market and recover a lot faster from failures and also cause less failures by every change. Maybe some of you have read that accelerate state of the DevOps report, which is really worth a read. And yeah, there you can really see that what they call the low performers in that report that really applies to all the embedded and desktop off-line software products.

And I think this is something that needs to be changed And I'm thinking a real revolution is needed there. So I think, Harald, your your specialist area or you where you focus is around maintenance. The maintenance of the vehicle, Is that correct?

Right. Our department is working on maintenance of the vehicle, which has the interesting part that I am facing, building delivery of web applications, of embedded applications of desktop software. We have a lot of legacy things So you could say I've seen it all And this is why I think that the embedded in desktop is the field where we really need to think like the DevOps movement does already in the Web area.

We need to adapt and embrace these ideas and also some of the technologies in our field of embedded software also. Yeah, I think Erez This is exactly what you talked about in your presentation, the fact that the software isn't just purely in the car these days, it touches a whole variety of different areas. Although I would take one issue with what you were saying about, you know, the maintenance in the vehicle may be challenging, but I think Harald will tell you that the maintenance of the vehicle when it's in the garage is a completely different challenge.

Right? Yeah, right. And the challenge in our case is that we offer maintenance for all kinds of vehicles and brands.

And this leads to a giant amount of data and of software and of legacy, which which makes it also interesting or challenging to build all this stuff. Exactly. Exactly.

So Erez, maybe we could bring you into this discussion and in terms of what you were talking around about, around feature flags, you know, you're obviously helping people manage a lot of software in terms of the content that they have to put in the hands of their customers. How do you how do you actually see the feature flags is going to help automotive companies become more productive, given the nature of what they're trying to do? Yeah.

So it's it's a great question and I think some of it was addressed in the presentation, but I think, you know, and sort of because more and more revealing to cause the automotive evil, you know, by external resources like mobiles, a Web application, more and more software is in the car. And so I guess we all both held and address We all talk about how you want to deploy small atomic units in and basically increase the quality, but also make it go faster. And so, again, feature flag in that sense, that's the key of what they allow you to do.

At the end of the day, you get to control your software on a feature level and then decide who you want to expose to that feature. So, you know, it could go from the mobile app to the cloud, but it could also be inside the car and various types of software all the way from the entertainment pieces. But also, I think that in the future we can see much more of the calling then even the making software is using feature flags to essentially have safer ways to deploy software and a faster way to deploy software.

So. So, yeah. Differently it's the feature flags going more and more into the various parts of the software in the car.

Yeah, absolutely, and I think one of the things that Fernando mentioned in the last session, if you guys were watching, I don't know if you saw it, but he was talking about the the scale of the software that is being produced these days and the problem he has of actually having to, you know, to just build it altogether. And I think this spoke very well in my mind, to what you were talking about, Eric, about accelerating, solving the accelerated build problem. And you know, do you see or have you had a lot of discussions with companies in an embedded and perhaps automotive where you really have helped address some of these challenges yourselves?

We have. It's a little bit of a tricky problem, to be honest, because what we find a lot of times is, is that people are really hoping that Accelerate will come in and just be this magic wand that we can wave over everything. And that's all I need to do And now everything's fast.

And the reality is that accelerator or any tool that you might choose can do a lot of the heavy lifting for you can get you a lot of the way there. But if you really want to get to that next tier of sort of that world class performance, you know to what Harald was speaking about You need to rethink some of how your software is actually architected and how it's delivered and how it's packaged. So you avoid things like, for example, having to do a monolithic update of your entire software package, but instead do smaller incremental updates.

That requires a little bit different mindset And I think that if we really want to see people sort of unlock the the possibilities of doing truly agile development in this industry, that's the sort of thing that people need to be thinking about. What's great about accelerator, as I said, is that it can take you a lot of the way there and then it can help you see where you need to focus your effort in order to reach that next tier. So we can't necessarily solve a hundred percent of the problem for you, maybe 90 percent of the problem but we can show you the roadmap to get that last 10 percent.

Yeah, so one of the areas that some, you know, I talk to customers a lot about is the idea that tools themselves are not the the panacea to fix their CI/CD DevOps challenges. There's a whole bunch of stuff that they need to fix around process and culture and people, et cetera. But one of the things that seems to be a common thread is that going with tools at least gives you a good starting point because people can get their embrace it and get their hands around it, rather than it becoming an inferior and a female sort of discussion, which is theoretical.

So a lot of companies we find do start out by looking at tooling parts of the CI/CD problem. But one of the comments that I felt resonated really well, Andreas, from your presentation was this idea of embrace, not replace. That seems to be a very common theme that I hear from customers, that they really do need to take advantage of their existing technology.

what's been the panel's view about about that in their experience? Maybe we start with Andreas You raised it in your presentation. Yeah.

Yeah. Especially in this era of Internet of Things wearable. Right, And then autonomous vehicles.

The importance of purpose fit tool is more than ever because everybody is trying to be very competitive, trying to be very innovative when you are losing the leading edge there are often time you don't have generic tools to help you. Right. You actually have to build your own tooling.

In automotive is very common when you actually have a specific tool so that you gain that advantage. Right. And also, it's not just a specific tool sometimes the way you do your developement and testing process is the way that you differentiate yourself from the competitors.

Right. And then it's kind of like the recipe Right. What is the secret recipe to make your product is very, very successful.

So given that there is a specific tool and the fact that there is certain process that is unique, that the way you do it. Right. And the way you mix the paint for example.

Right. That makes the color so vibrant. Right.

And if I use the real world example. Right. But the same with software, the way you actually produced the software with the with the hardware, that the uniqueness in your process may actually bring the advantage.

Given all of this, you need to actually embrace diversity because all giant one system will not be able to give you the flexibility to actually be competitive and innovative at the same time. Does anyone else have any thoughts on that area about leveraging the existing investments rather than to be ripping everything out, trying to start again? Yes.

So in my experience, I think it's a mixture of both. It depends on the situation. I agree partially to you Andreas because in some cases, developers often like the newest technology and their favorite tools and everything.

And in many cases, it doesn't make sense to radically change everything And five years later, there's a new favorite tool and you changed again. That makes no sense, But in my experience, we also had tools that were really, really huge obstacles and making everything very tedious and you really needed to get rid of them. So I think it's depending on the situation and you need to investigate and do a wise decision.

But we had one large change where we got rid of a very outdated source code management, which just was not usable for continuous integration or continuous delivery. So we needed to do that And we invested really some years and a lot of money to get rid of it But it absolutely was worth every penny invested. So I think it is sometimes like this, sometimes like that There is no general decision.

So I think one size doesn't fit all. You know, I think we have to use the best of breed technology, I think is what we're saying. But we can't assume that there's just one big software solution or one big Ci/CD solution that solves everybody's problem.

because even if that was, people couldn't consume it. Right. Eric?

Yeah, exactly. And same for writing your own tools. I think sometimes it's better to use standard tools.

Sometimes it's better to adapt the tools and write your own plugins or whatever, and sometimes even write your own tools. It depends on the situation We have all kinds of these situations also. Absolutely.

So, Eric, you put your hand up. Yeah, I was just going to add that what I found extremely helpful in these sorts of discussions are these explorations with our users is to focus on the data. Right.

So embrace, not replace It is fantastic, And that can help you win sort of grassroots support for your initiatives. But as Harald said, you know, sometimes there are things that just need to be replaced and convincing people to do that can be challenging, But if you've got the data to back it up, then you're golden, right? If you if you can point to that and say, look, this is the reason why things are not working very well.

And of course, there there are lots of tools, including ours, that they can help you collect that data and present it in a meaningful way and surface that so people can make informed, intelligent decisions about what stuff needs to be replaced. If we didn't replace from time to time, we'd still all be riding horses. Right.

I even have a separate talk about convincing people to replace the tool or to do bigger changes. The talk is called Enemy Mine after a bad science fiction movie, you might know. I have seen it, but I can't say I've seen it recently, but I don't think we really want to get too far in to a science fiction or we'd be here all day.

Sometimes life is like science fiction. So we're advocating the idea of going fast. Using CI/CD to optimize the way we're delivering new software value, and leveraging the existing tools where we can and bringing new technology in.

But sometimes I've heard that people are concerned that going fast in this industry at least, potentially has implications on quality and quality, broken software has implications on brand and customer loyalty and even worse, safety. So what are your thoughts about about that slight dichotomy? Yes, or I'd say that our going fast, it really has impact on quality, but a positive impact.

Where in my experience, if you change fast and deliver very small portions of your software, you are less likely to fail and to produce failures and you can recover a lot faster from failures. So I think the traditional thinking of building one monolithic release and testing it thoroughly. It is something that is more or less outdated.

There are some security functions, of course, where you need the thorough tests and quality standards. But all in all, even these could be split up in smaller pieces and built and delivered faster. I think this applies more or less to everything and will improve quality.

I'd like to add what Harald said. In delivering it in smaller pieces, it's also becoming very important to be fast and safe, but more importantly, when you actually build hundreds, hundreds or even thousands, thousands of small pieces, you need to actually have a very good understanding of dependencies of all of these different components that you have in the system, because by understanding the complete dependency, you will be able to actually perform impact analysis when you are about to actually make radical changes. Yeah, exactly.

Our product consists of 10 thousands of artifacts, but I can tell you it is manageable. Of course, you also need to automate the management of the many artifacts and of the dependencies, But then it works. Correct.

Yes. But we also have our own tooling partially, of course. And are you using tools specifically to do that sort of thing today Harald?

Yeah, we are. We have partially our own toolings, for example, for publishing artifacts in our artifact store according to the process If they are tested or approved or released to different stages and to publish these artifacts and check their dependencies, we have our own tooling. So you're talking about releasing software in smaller pieces, which I agree, I think that's a very good practice in terms of all software development, to be honest, and perhaps where, you know, micro services and things coming in other areas.

But could you consider and maybe this is just my misunderstanding but could you consider feature flags being a way of sort of making that real for companies? Because you put things behind small pieces of software behind feature flags? How would the two sort of marry up?

Erez. Yeah, I think that's definitely, I claim, I think that features like essentially what they allow you to do is to take pieces of your software and then deploy them in production gradually. And then if something breaks, you can also, you could instantly take that either of for where you would just for the affected user.

So I think I've seen feature flags being used both for the safety component, having the ability to turn things on and off quickly, but also for the speed component. So things like merging, incomplete called to the main branch, moving to trunk base development. So things like testing in production.

And so a lot of the benefits coming from feature flags from that idea that you could separate software into smaller pieces. A lot of it induces speed, but it also takes care of the safety piece. And I think, you know, Harald mentioned this when he talked about these atomic units, I think future flights are instrumental in that.

And I've seen feature flag technology also helping the business commercially as well, because I've seen companies that will embed software, the entire software portfolio of an application in a thing, but actually turn off a lot of the features until they're able to monetize it with their consumers. So effectively, all the software is there. But it's not available until people pay their subscription or they buy the piece of capability that they want to sell at some future date.

So I've seen that as well, not just as a technical benefit, but also as a commercial benefit that can be really powerful when you're talking about large scale deployments. Harald? Yeah, I also think so.

And if we really think about it, those features turning on and off like options and license, different licenses that enable you to do different functions or not. If you see it in this way, then we already use feature flags since a long time, since tens of years, but not systematically, not with the framework and not with a clear definition. Everyone develops on their own, So I think there is a huge benefit of using feature flags in a standardized way, in a clear way and also introducing this into embedded and offline software.

Ideally, if you connect the car like over the air updates and then you can also roll out future flag changes, that sounds very promising to me. Yes, and just to add on that, one like feature customization is a big use that we've seen to a point, Steve. The idea is to have one version with multiple features and then you can decide by geolocation or by compliance or by regulatory which users get which features.

And I think you even alluded to the fact that even if somebody purchased something, or has a license for something, he could get that feature. And again, he doesn't have to get the new software delivered because he has all the software, and to Haralds point, We have seen a lot. So, CloudBees Rollout or feature flags, we've replaced a lot of homegrown solutions for a lot of customers because they think the idea of controlling software remotely has already existed for many, many years.

And a lot of companies have developed these configuration, home grown solutions that do not scale well and have a bunch of problems and that's where they usually come to us and say, hey, can we have some homegrown feature flagging system, We need something professional. Can you guys come in and remove that, too? So definitely we've seen this happen.

It's not the if statement that you're industrializing. It's the value of being able to get the control and management of those multiple feature flags. I guess.

It is the auditability, traceability, governance, the ability to have a UI. Well, you get it. All right.

So I think we're coming close to the end. I will take actually the last question has been one from someone online, which is basically saying, what advice would you give us around navigating the jungle of CI/CD with all the tools and everything? If you could if you could make one recommendation as your closing comment, one piece of advice for the audience, What would that be?

Either a gotcha or a recommendation one or the other. We'll start with Andreas as we go around the room. Thank you.

I'll step in. I think a few of our panelists already mentioned this in order for you to optimize in order for you to convince others to actually buy into your proposal to improve or make changes. It is important for you to actually make your proposal, make your design proposal for changes based on real data.

So it's important for you to actually collect how well you are doing today and there's a lot of technique to to enable you to actually collect up data and analyze it. It's about the value stream of your end to end process from left to right. Ok.

Erez, your thought. Yeah, I'm dying to have my thoughts on, you know. So I think there is a jungle out there, right.

You know, a bunch of best methods and a bunch of tools and, you know, which one should I choose and when and how to implement. And I think one of the things I've seen over and over, and this goes to all CI/CD builds and feature flags, is that if you architect this too much and if you go all in on this massive solution, it's going to be complicated. So start small.

One team build the expertize and then starts going from that space. With feature flag, just a quick example, you know, it's very easy to start. I've seen teams that have started with one feature flag point to production and go to ten.

And today they have hundreds and they have, you know, both known methods of how to manage those and kind of add in moving this to the life cycle. So kind of to summarize this, I think that, you know, each initiative should, of course, data driven, but should start small. Figure out the details and then expand inside the organization.

So data driven, start small and grow. Eric. A thought or a pitiful.

Erez stole mine, I was going to say start small as well. All right. We'll take that twice if we could.

What one of like trying to boil the ocean. Right. So the biggest advantage, honestly, starting smallest, that you can demonstrate quick wins, which means that you're a lot more likely to win support, which means that you have the political will and the capital in order to do that expansion out to other things.

Just to put my own little spin on. Of course, I think the most important thing is to make sure your stuff is going fast. Absolutely.

If you're it's great to re-architect all this other stuff and come up with this glorious new system, but if it still takes you a day or two days to push something through, then then forget it. So start small and focus on performance. Awesome.

And Harald, we'll leave you with the closing comment of the live session. Your your best tip, your best trick, your best thought. I agree on the start small part, because I also in my life I have sometimes started to big, I must admit, and I would say start small but try all different things, try out new ways.

Why not adapting microservice ideas embedded? Why not use containers inside an embedded device and such things? But what I would like to add is start small but think big afterwards.

Start small and think big. So we should think where we want to go with our small steps. We want to go through a revolution of embedded software to adopt DevOps ideas and maybe even best DevOps.

Also think about culture and about the business and also think about like influencing the standardization committees, because what hinders us in embedded software are also standards like mesra, auto SAR, whatever. So also long term, try to influence these. So we should really think big to a revolution, start small and think big.

Fantastic. That's a perfect close. So, gentlemen, thank you very much for your contributions on the panel today.

We hope everyone's watching has enjoyed it. So thank you very much. I just have a few closing comments just to wrap up the day.

So that's basically it for our live sessions. It brings the life out of that format to a close. I want to thank both the speakers from this session and also from the prior session, for their time in supporting the automotive summit.

And I also want to recognize all the organizers for their hard work of pulling together this in a new virtual format. We've never done this before. So we're very excited and we hope it works for everyone's satisfaction.

So please be aware that you can watch the content on demand anytime after the event. So if you missed anything, please come back to it. And you can also send us questions at any point as well.

We're still happy to answer them and come back to you and respond to those questions. So I think that's basically it for the event today. I hope everyone got value.

I hope you all enjoyed it. And I hope that you'll be happy to attend an additional event or a future event that we may be using this platform to run in the future. So thank you, everybody, and goodbye.

Bye. Thank you.