Interoperability Challenges and Solutions – The CD Pipeline EP 4

Hey everyone, welcome to another episode of the CD pipeline. The CD pipeline is a monthly show in partnership with our friends at the CD Foundation part of the Linux foundation and us here at Tech strong group where we focus in on topics of interest to those of you who are interested in CD continuous delivery and hence the CD Foundation. My co-host for this entire series is my friend.

Laurie Larissa who be besides being developer Advocate with Jay frog is also A main player on the letter explained her background with the CD Foundation. Hey, Laurie. Happy New Year.

It's our first show after the New Year. It's good to see you. It's good to see you.

And what did New Year it's been already. I think January 2023 slapped us all in the face a little bit. So it's good to see some familiar faces and some friendly faces because we could always use a a good cheer a good cheer up.

So, yes, so thank you. Again Alan for partnering with the CDF. My name is Lori Larusso.

I'm work for Jay frog and I am the marketing Outreach chair for the CDF and we are very happy to be a part of tech strong TV and to be able to bring you all of these cool new things happening in the CDF and in the world of continuous delivery. Absolutely and Lori just He says last year, we kind of focused on some of the great projects that the CDF manages right and there are I think eight or nine of them. They're fantastic.

You know some really big ones in there. We were good. We're changing things up a little for the New Year in that.

We're focusing on some of the other topics of interest that our audience I think will be very interested in inactivities around the what the cdfs involved with today's show is going to really focus in on interoperability. Right and to something I learned from one of our panelists Melissa today, right? We really want to focus on interoperability not necessarily integration.

Is that fair enough? And you know what, the CDF has has spun up especially interest group on on this interoperability. We're gonna hear from some of its members.

They we have an amazing panel of speakers Lori. I'm gonna if it's okay with you. Would you like to introduce our panel?

Sure, so let me start. Well, since you mentioned the interoperability Sig, let me start with my colleague Melissa McKay if you want to introduce yourself, I yeah, I've been a developer for many many years. So very interested in all things cicd.

I'm currently a developer Advocate with Jay frog and today. I'm representing the special interest group that is part of the CD Foundation. This is an awesome group of folks who've gotten together all of us from different backgrounds different organizations willing to talk about how we can better improve interoperability among all of our products.

Excellent. Thanks Melissa. Welcome.

And so let me introduce did you see who is a part of many instances within the CDF? So did you see go for it? Good morning.

My oh my name is not easy sandika and I am an engineering manager with apple. I'm also on the Spinnaker technical oversight committee as well as the CDF Outreach Community. Welcome.

And sorry so then Emil, who is someone I think what's really great about this show is like you said Alan we talked about sort of some of the past and projects the future of ci/cd and continuous delivery and a meal happens to be one of our project Founders that came out of a Sig. So maybe you'd like to introduce yourself. shirt Thanks for having me.

So I'm Emil Beck Mark calling him from Sweden and I'm employed at the Erickson as a cicd architect. I've been working with tools related development for many many years. And the late years of course and linked all continuous practices continuous integration delivery, and I've been also involved in the continuous delivery Foundation since a bit more than actually three years now.

I was there in the beginning when seek interoperability was was created. And since then I also been involved in creating the special interest group for events and which eventually then founded as he said that already a project called Siri events, which I'm the co-creator of. Looking forward to this chat about the interoperability.

Thank you omeo and thank you for all your work over the last three years. And last by no means least is the the one that keeps us all in check executive director of the CDF fatty. Hi everyone.

Thanks. I don't know before hearing us. My name is Father Manji.

I joined to learn foundation in 2012 to the contrast their foundation and prior to learn foundational work Forex and open source Whopper contributed not projects within notes foundation and elsewhere and I was directly Foundation was announced in 2019 March, so I've been contributing to community or since it's infection. I have deep passion around the topic intervalability. Thank you.

Thank doing with see different we'll look at it and they don't realize what goes into actually running these things day to day. They don't they don't run themselves. So thank you.

So interoperability, right? So I'm not a big fan usually of defining something by what it's not. And I I opened it up by saying well interoperability is not integration.

That's great. But what do we mean by interoperability, especially as it relates to CD? Right, and I I think you know Melissa you got the Sig.

It's the interoperability segment. I can goes to the heart of it. Right?

Why not? Why don't we start there guys? Let Melissa kick it off and then feel free everyone to jump in and and give us your thoughts.

Yeah, exactly. I think it's really important to make that distinction between interoperability and integration because often especially with commercial products. We you know discuss integrating with other products out there other.

Whether open source or off the shelf and that's important because you know, your teams are going to be made of different skill sets. We want to be able to make the choice of all of the different tools that we use for cicd. Being having that freedom is is really important.

The problem that we are trying to tackle though right now is interoperability meaning not strong glue code between certain products or projects. We would rather have interoperability meaning you can plug in, you know, whichever tool you like that works for your team, you know. This would help you stay away from a lot of Maintenance tax for this glue code having things break.

Every time you have one particular tool in your tool set upgrade things like that. And that's what our group really likes to do. I joined the special interest group The interoperability special interest group a couple years ago.

I'm now coach our of that group and that's something that we like to discuss making sure that our developers are, you know, being able to work efficiently with all of the different tools that they need to use within their organization. Panel, anybody else have thoughts on that? No.

So, let me let me kind of do a little table setting here. There was one dominant CD tool that 85% of the market used right back in the the Heyday of Windows 90% of the market, right? Apple was always chugging away.

We wouldn't have to worry about interoperability. We probably all suffer as a result of less Innovation and and things that we sort of stagnate the PC market like in the 80s and 90s, but the fact is The market today is a dynamic Market of more than several fantastic Solutions. There are several that the CD Foundation themselves manage, right Spinnaker being one for instance.

Jenkins being another right just on top of my head, but there are not open source CD tools that are very good as well and you know in any When you talk about Enterprises, it's very hard to do standardization around tools. But you have some teams using one thing some teams using something else. You know, it's the rare bird where you really a standardized enterprise-wide.

So, you know, that's a real conundrum for managers. Right? Do we do we have all these people existing kind of in their own private Idaho's or is there some sharing?

Of data is that you know, if I want to move from one to the other do I have to go back to go, you know with and not get my 200 dollars or anything right? Wouldn't it be a great world if we could pretty rapidly Sub in one of the other or at least take information from one of the other and You know and decide which is the best tool for the job. And I don't leads to the question.

So why it's critical, right? So just why do you feel like interoperability is integral and super important in today's workflow. Well, I think interoperability is the difference between just as was already said just gluing things together.

But also the key to this is being able to collect metrics from system to system to system as any process steps through you know, my goal in particularly as is a part of Spinnaker is to ensure that as we grow as the product continues to evolve. It's very very simple almost intuitive to be able to connect to it and get information about the other things that you connect to interoperability, especially Specifically this work that CDF is doing is going to really impact my ability and Spinnaker's ability to not only connect but to give information about workflow As you move from system to system with the products within the CDF. This is groundbreaking stuff.

This isn't you know, just, you know, let's connect some things and see what happens. It's let's understand our ecosystem and Agreed a meal. Yeah, I can actually develop a second that I really agree with you there at least so I would also say that interoperative actually in some ways observability in your system.

So if you have interoperable components in your system, they will then automatically become observable by some using those standard interoperable interfaces that they use as strong as you can somehow monitor. The interface is the traffic flowing between these components in some way. So I say things observerability actually comes with with interoperability to large extent.

That's good. Yeah, one small key thing is it is the fact too that there's so many projects that are involved in CDF that are able to do this and what the goal or what I'm hoping that comes out of all of this is that I don't have to impede anyone's, you know idea of how they should use their system. You mentioned that it's difficult to have like systems CD systems in a large corporation.

I would I would argue that it's almost impossible. But with this effort that the CDF is doing to create interoperability. Now these pieces can be joined together and once join together those metrics can be common and that's huge that that is a game changer and and this industry It is, you know, I wondered.

So you have the CDF three plus years. And if I mispronounce I'm doing my best with names fatty. or yeah, so but how how critical is the interoperability to the cdfs mission?

I think you told this and one of the key is this we have within the containers Foundation because if we talk about the projects we have in quantities that are furnished as we mentioned Alan. We have nine projects that I'm also thoughts projects are around content integration come to the orchestration projects. And if you look at other half of projects, we have like microservice and S1 project or tell us we have packets built and distribution project Persia.

We have cd events, I think. The focus around intervalty when we talked to many different people and organizations. They may think in the terms of contains their orchestration systems.

But if we think typical production systems that the continent of pipelines are actually production systems with an organizations to our systems consists of multiple different tools and projects as I mentioned. We have the orchestrators like Jenkins Spinnaker tecton, we have start work repost stories. We have the visualization systems and the key to connect all these different systems is to have intervalid and ecosystem.

If you don't have interoperative in the customer, if the contents they are Foundation is not putting effort into this area. Then it will be very difficult for projects and users amenders. Like say come up with nobody Solutions because majority of organizations will spend their times to integrate these different types of tools with each other.

So Integrity I see internal as enabled of the Innovation or feeling Innovation because once you have the metadata standardized once you have the interfaces defined then everyone can develop against those interface using the standardized metadata, then they come come up with near Solutions new projects that actually helps organization supporters. So I seem tool as the critical piece for Innovation with encounters circuit ecosystem and consisted a foundation is pushing the topic forward last couple of years. I like that.

So Laurie, oh, I'm gonna ask you to put your J frog hat on for a moment here now, okay. It's green as a vendor. Why is interoperability important to you?

Well specifically where is it? Well, I think specifically with Jay frog our whole idea is to be seamlessly integrated with whatever the workflow is that you have. So whenever you bring us on board, whether you're on-prem hybrid Cloud multi-cloud, we want to be able to work what's best for you in the way that you work best within your orgs and to gc's point, you know, we have clients that are multiple, you know orgs that said everything up differently to their own specifications and we want to be an easy piece of that puzzle that can help you just move your projects forward.

And so this is super important to us. And what I think is really cool. Is that like a meal, you know started with the CDF was on the interoperability Sig and then created a project, you know see events and I think you know, it'd be interesting to see how you came to that a meal.

How did you take like from working on a stick? And deciding. Oh, no, like we need to take this a step further and really create a project because I think CD events is in what version point two now.

Which one still yeah one, okay. So if you yeah, yeah sure. So yeah, as I said in the beginning of that I started in decision for probability group where fat was the driver of that group from the beginning actually and we quite early we talked about vocabulary in this terminology that we've already mentioned here today.

So we initiate something that was that we called the Rosetta Stone of CIA City where we Define some terms that are are common between different tools and then in parallel to that we also started to discuss what could be done do for interoperated. How could we create interoperable interfaces with their different tools that we see in the CSEA ecosystem and one idea that was brought there was about events. So having events that are sent broadcast over a message bus.

So some kind would make them make it possible to communicate to make different tools communicate with each other with some tools and events and another tool consumes them. and Not just interoperable in that sense that they the flow of execution could flow through over these events. But also the than the observability could be adopted onto that system.

So they they obserability or metric system for that matter could just listen to these events as well. And then the consumers or the other producers the Brutus or the other consumers of the events wouldn't need to really know about who is now listening to this this information. So it takes it makes a very very easy to to Plug and Play these different tools into a system if you have a broadcast messaging system.

As the interoperability interface there. And therefore we started this special interest group events, which was part of these that's interest group intravability from the beginning, but then it's became its own interest group. And as I said in the beginning as well from there, we then start to look at what good an event protocol like this look like so that all tools that adhere to this protocol could both produce and consume the information that is sent in this events.

Excellent. So to me, this is the this is why we have foundations like the CDF right to to come up with these kinds of I mean, it's one thing to manage spinnakers and Jenkins and you know Percy and all these things but to says a whole a hole in the in the tool chain here and it's all set and come up with these. Projects like a CD event is really the justification for why you have a CD Foundation, right?

This is going to make people's lives. And that well, it's not going to cure diseases but it's gonna make their jobs easier. It's gonna make their work easier and and that that's imperative, you know, so it's in point one.

Where where do you see it going? And how soon? I guess that's to me again.

Yeah me I'm sorry, but you're not but if anyone else wants to jump in feel free if I can there's one thing I want to say about the good the good what what this what Emil and his group is done really really well is Steve taking a look at this process that he mentioned not just as an event boss but as driving system the system and What attracted me because I am fairly new to the CDF. I met Fati in in June literally on my birthday and it was a beautiful present by the way guys that I really appreciate being here. It's the fact that it can drive a system to system as well as store these events and the thoughtfulness and the approach about collecting the information that's defined in each step of the event without forcing, you know, and individual Paradigm and leaving you space to kind of build your system the way you want to what are the key things about spinnak.

It doesn't just deploy to the cloud deploys everywhere. And so the need for something that simplified and interoperable from fisting the system. The system is really complex, but they very elegantly kind of just simplified it down kept a descriptions really really just in line with almost everything that every system that we have to connect to does and I think that's brilliant one should be called out.

Emil's work has been wonderful. I I agree it absolutely. Melissa, I want to come back to you for a second.

What's interesting is all of this. I mean finally himself really was with the Sig from day one. It sounds like right everyone here was with the sink.

Give us an idea of like how big how many folks are involved in the Sig? How often do they meet? How did they communicate people out?

Here are watching say man. I maybe I'd like to be involved in that. Absolutely and I think this is a good follow-up to what was just said to because this group is not authoritarian.

We're not out to say thou shalt run your systems this way. That's not how this group works. We invite the community all of these vendors these this is an Avenue for all of you to come together and bring your insights and your thoughts and specifically like we appreciate the DC joining the group and talking to us about things that he's looking for things that are important to him and in his organization.

And we have that with several different people the group itself. foundation, there's a very important tab there called community and within that tab it gives descriptions of all of the special interest groups that we have interoperability being one of them. We meet bi-weekly.

Those are open to the public anyone can can come in. We announce our agenda ahead of time. Usually we'll bring in speakers from other projects that we're interested to look at and then we'll have, you know, discussion points around interoperability that come up doing that.

We also have a slack channel that you can sign up for in that same tab that Community tab that's important information to get involved with other folks and get other people's opinions so that we can talk asynctly about things. And the group itself, it changes inside just in size just depending on what we're talking about. Recently.

We're talking about starting, you know, another work group that may have another time and place that they meet to talk about a very specific item. That's how CD events came about now. We're we're starting to talk more about pipelines and how everyone speaks the pipeline language and you know, maybe that is something that we should look at attempting to standardize.

So it's not so difficult to move between build servers for example, and being able to you know experience the benefits of a different product without having the heartache and the the problems of training a bunch of developers on something brand new or you know, having that decree come down which is you know, puts a bad taste in people's mouths when they get used to a certain project or certain product that they accustomed to so that's what I really like about this group. Fatty was one of the first that I met and I was really impressed with how this is run. So yes open to the public, please join our meetings.

We would love to hear from you. excellent So I got another kind of. Topic I wanted and I'm gonna throw this out to the whole panel or including you.

You know, one of the nice things about interoperability versus let's say integration is I'm not interested in doing an API or an integration between Spinnaker and Jenkins, right? What we're looking to do is to create. Sort of like it sounds like what CD event is doing we're looking to create Broad.

kind of data plane or points that all of these tools can work on now it's easier when those tools happen to also be part of CDF, but what about all of those CD tools that aren't Part of CDF, right? How did they are they included in here thought of how are they you know, because if you're gonna be industry why you're gonna be industry-wide funny I'm going to ask you to kick that off because Hey, it's your job. You know, how do you work with non project Souls there?

Yes, I think like as one of the first things you mentioned Ellen when we start this episode is like contest of a question has lots of different tools and technology and they are all great in what they are doing and you're not definitely saying this tool is better than that because it's up to the users to you know, we can choose the right tool for the job if they want to use tool X4 certainly and want to use two Y for the rest of their pipelines, for example, they are free to do that and why the projects that are not part of contents their Foundation should care about this work or City events project. Is that because It is important for us to enable our users. Our users must be or should be happy to use what the community sidewalking and they should have option or choice to make decisions for themselves rather than forcing or imposing certain way of doing things or sorting close to do those things because organizations are different scale is different the types of products there and walking are different and if we have the voice than that desired in which will come up because I always focus on that one because the content is their financial and other foundations are actually there to provide home for the projects as best provide for Forex into individuals and others to comment together and Projects that are not hosted within the context that are Foundation Flagship look at what's happening within the CD events and specific events and Specialty to see how they can make use of what has been developed in this community because the projects are not dropping these things themselves along all the group is not developing these things themselves alone.

These things are kind of collective knowledge of years of efforts coming from all these different types of organizations. And if the projects want to be ahead of no the curve and adopt these things and support and give the best user experience for their users. Then they should be looking at what's happening within the interval topic.

So yes. if the projects are after improving developer experience developer productivity organizational excellence intervalty and the City events are key in achieving those you are like Mad scientists, but like in a good way you're taking all of Your Collective frustration and all of the things that you've learned along the way to create these new projects these new cigs these new interest groups and then you sit there and you talk and you actually come up with Solutions, which is awesome to watch from an outsider looking in. So just you know this conversation alone seeing how like City events pull like grew out of special interest group of the interoperability special interest group, which became its own special interest group, then became a project.

I think that's one thing that this community. I've like been watching for the last couple years like you guys get to work you have problems and you want them solved and you want to make it better for the next generation of developers and the current generation of developers that are that are trying to move things forward. So it's to be applauded and like, you know being inversion point one moving things forward.

It's it's awesome because starting a project vendor neutral all that stuff isn't easy. So it's really cool to see like how Spinnaker has grown see events is growing the Sig. Blossoming.

Yeah mad scientists in a good way. I don't like them to facts actually if I may with the from the request in the island on another tools outside the city Foundation because in the very early days of the events discussions within the seeking for probability, we had a core group work group where actually people from Captain an event driven series system from dynatrace was involved and also people from harness CSE as a service proprietary system or less at that time at least so they were already at that time involved in the in the work driving the event protocol at that time. There was great fun.

Yeah, but That's the realities of the market right? You do have the harnesses and these other folks. Diner tray, so it went out there and and to find these point right you the foundation is not in the business of dictating to developers or devops folks what CD tool that you they should use whatever CD told is the best tool for them.

And there's a lot of reasons why people choose tools Right. It's not always necessarily the very best tool. It's what's best for them.

Not what? everyone else may think is best or better as we all know and I I think that's an important. You know an important when you when you're dealing with the Linux foundation and and not for profits and open source.

It's important to remember that right is that we were not here dictating to you. Right we and we want to be inclusive everybody's invited. Come in and and you know say your points guys.

I'm just looking at the questions that we want to start off with and make sure we hit and there's a bunch more here than I had in my thing Lori if you wanted make sure we've hit some or any that we've missed. Because we're coming in probably we only have maybe seven 10 minutes tops left. Yeah, I think we've hit everything that we wanted to talk about.

I think the discussion has been really free-flowing and you know, I would I would throw it out to the panelists. Like what is the one thing that you would like to let the community in general know why this is important and what you know what your work is aiming to to help them do what it like if they're just completely like this doesn't make any sense to me. I'm just gonna live in my own bubble.

Like, how can you expand and let them know why this is this is important. Well, one of the things that I think is important to the open source Community is that this particular project this effort with CD events isn't just going to you know, make the tools that are in the CDF portfolio interoperable. It's going to create opportunity for new projects as a meal mentioned.

It's an event based system and there are still key components to this event based idea that you can build off of both from observability both from how you collect the information. He mentioned Captain being an event driven system. I think there's some Simplicity both of the idea of how to do this and that it fits so well with the process of you know, how we think through any tools Ian the build system or continuous delivery workspace that having this be event driven creates a lot of opportunity.

It's that opportunity that we're trying to it out and communicate they're going to be a lot of things to build a lot of ways to think about things because if the systems that you bring in-house can be connected and the points of information how you use that information and how you explain to management like what's going on like all of these things become really really available and fascinating right? It's just really just changing how we think about continuous delivery. Absolutely, you know what?

I another topic certainly it's like topic number one in 2023. It seems you know, we hear texture and we didn't an event called predict. Two weeks ago and it's kind of our predictions or what we think is important.

I had an amazing panel right? I had Fred from from Jay Park one of the co-founders object Cloud V CEO, I'd KK the founder Jenkins and the CMO Chief strategy officer acting CTO of gitlab. So some big devops companies, right?

And all of them said you're not well gitlabs calling is not even calling a devops anymore. They're calling a deaf set cops. Jay frog Fred security is a primary focus.

Everyone is talking about security. So I'm gonna assume going to ascent and Assuming. He's a bad thing.

I know but I'm going to assume that security has. Raised its head here. That security has become.

You know certainly something on the on the agenda for CD events for interoperability because you know that would that would be a real punch in the eye right if we enabled an interoperability but in doing so opened up some security holes. So first that the Sig level and then it's CD events Melissa where that security kind of checked in here. Oh, yeah.

Absolutely. I mean this is one of the important reasons why we want the community involved because the community concerns are what we talk about and that is definitely come up. I'm actually going to give this to a meal because he's got more specifics on discussions there, you know pertaining to CD events.

Sure. Yeah, I will so before we go there. I just want to mention that we also have another special interest group in the CDF called software supply chain security, which actually is just this as a primarily topic to discuss security in the software supply chain.

So that's that's a cool group that you should also listen if you're interested in this topic, so when it comes to security and events, we haven't really come to any final specifications there yet, but we are looking into it. We have discussions being started on how to integrate the different aspects of security into these attestations and everything encryptions and such things that you would need to tell that you have built artifacts that are secure and not stamped with and so on. So those discussions are coming.

They're not there in the protocol yet. But we we are looking out to to find people that can can help us drive in those parts of the protocol. So yeah, it's it's a very interesting area and a good pretty much like to have that as as part of the roadmap for this year to come get in the security aspects of the events as well.

I definitely makes sense. I mean all of us are concerned about provenance now if we're going to have this system that delivers artifacts from one tool to another or metadata from one tool to another obviously. We need to be careful about what we're doing there and making sure that we're not opening ourselves up to attacks again, that's the importance of getting more folks involved in our groups so that we can reveal those holes soon and early and come up with solutions to solve for that.

One of the things for a future shows we should just do sort of an org chart if you will of CDF right with all of the different special interest groups and and you know cigs and committees more than the projects so that people can really see how it you know, how it's blueprinted and where they can get involved. I love that you framed it as a team because I really do feel like you know, there's nine projects within the CDF and all of the projects are always looking for ways to work within the other CDF projects to see like how they can work together in addition to what you're saying the cigs the Outreach committee. We're working on workshopping some ideas to help template out some other programs to make things easier for projects to move forward.

And so I'm a hundred percent behind giving some light to the people that are behind the scenes that are running those cigs and showing all of the different various aspects of what is involved within the CDF. It's not just our nine projects that keep us going. It's all the people behind the scenes continually innovating coming up with new reasons to have interest groups New discussions new side topics and then developing new projects for sure hundred percent.

very cool All right. Hey guys. We're about at a time here for this month's show.

You know what a meal people like you're what make open source go right. I mean you are to me the I mean that that's you know, the Horatio Alger as we say you used to say in the US right people who just make it happen they work hard and they and they make things happen. They see a problem.

They see something that needs a solution and and they go for it and and you know, we wouldn't have the success of the Linux Foundation or CDF or without people like that. So thank you especially for all that you've been doing and continue to do but thank you to all of you right. Thank you to all of you and to you out here who folks who you know, whether you Are members of the Linux Foundation is an individual or you work for companies that are part of the Linux Foundation or you go to the cube cons or CDF cons or what have you?

You know, it's that support that makes all of these things possible. This unfortunately. In fact, the matter is it takes money to run these things.

It takes resources. It takes time. It takes people.

So, thank you. Thank you out there. Thank you for being on here Laura.

You want to wrap it up? Sure. So again, thank you Alan and text on TV for having us host the CD pipeline.

Thank you to the panelists again as Melissa stated earlier. foundation and you can learn more there's tons of information on that additionally cdcon is coming up our abstracts closed February 10th. And the event is May 8th and 9th in Vancouver Canada.

We're embedded within open source Summit North America, which is really cool. We've partnered with gitops com. So the first two days of the week are us and then the US is open source Summit North America, and we're really excited about that.

So again, thank you to you Allen. Thank you to the panelists and you know, look forward to the next show. Absolute, absolutely.

This is CD pipeline here on techstrong TV. We hope you've enjoyed it. We'll see you next month with another great show.