Techstrong TV – June 17, 2024

Hello everybody. I'm Mike Baard, and welcome to the latest edition of the Techstrong Gang. We're gonna be talking about this ticket Master debacle involving Snowflake, and then we're gonna move on to, well, some insecurity that results from some of the work that the FBI is doing.

And I know that's a little counterintuitive, but we'll get into it. And finally, we're gonna look at, well, Fortinet acquired Lacework. We are gonna see what's going on with Synaps.

We'll be back in a minute. All right, folks, and we're back. This edition of the Gang has Chris Blatt, who is north of the border.

Chris, where are you In? Lovely Brook, Ontario, Canada. Excellent.

And then we have our globe trotting. Mitch Ashley is actually back in Colorado for a change. How you doing, Mitch?

Good, good to be home for two hours before I head out again. I, it's a little longer than that, but it's, it's gonna be a short, a short stop. There you go.

All right. Our first topic today is this whole thing that went on with Ticketmaster was the initial breach involving Snowflake. And the, some of that stuff sits up on Amazon Web Services, and then it's researchers at Mandy and seemed to report that.

Well, a lot of people are having the same issue, and it's just starting to expand and it's kind of like watching a slow moving traffic accident. But Chris, what's your take on what's going on here? You know, we're, we're talking about Snowflake now, right?

You know, and I, I just have to say, you know, I, I try to give a positive message and a positive outlook. You know, I think that's true most of the time and worth the, um, uh, PBM Mind. However, you know, we do make the same mistake time to time again.

And the first mistake is not taking responsibility. You know, I a guide, uh, I'm not an expert on this in, uh, individual topic. You know, folks at, at Snowflake may take exception to this, but own it.

Own it, you know, even implying that, you know, we're gonna help our customers fix their problems. Uh, no, you know, don't, don't ever say it that way. Uh, I don't know.

Mitchell, what do you think? I mean, a lot of this comes down to this shared responsibility model. And I gotta say, I'm a little skeptical of the whole thing, but what's your take here on, uh, is this working?

Well, I'm, I'm kind of a skeptic. I dunno if skeptic's the right word of shareds responsibility model. It's true.

You're, you're depending upon the provider like Snowflake, uh, to a secure environment to secure your data, et cetera. But ultimately, it all comes back to you. Shared security model means you're still a hundred percent responsible for whatever happens as the end user.

Now, to Chris's point, I absolutely agree, is just don't fob it off on the customer. Oh, bad customers. They just, they, they're not securing their accounts.

Well, maybe you ought to help 'em a little bit, you know, best practices. I'll bet there isn't a security professional in the world that wouldn't say, Hey, you know, at least enable multifactor authentication or two factor authentication just as a start. 'cause we all know passwords are free, and, uh, there's many of them available and keys and everything else on the net.

And so, uh, you know, at least to do some basic things to kind of save, you know, we know it's a pain, but you should be used to it. 'cause I'll bet you used two factor on just about everything else you use in your environment. If you don't, we'll help you start.

There you go. Yeah. And that has really well said.

You know, the shared responsibility model, you know, the contemporary topic, like right now in the CISA supply chain working groups, we're, we're looking at spinning up tiger chain. So shortlived focus groups work on things. And what that's really interesting is, is sort of sparked off by an FCC requirement request.

Um, don't quote me on exactly how permit it is, but you know, they want, you know, the, the ability for people to scan a QR code on a medical device, on a, on a IOT device and get the SO and the HBO and next. Um, and without going into all the details of that activity, I think that's anything to follow. I think this year was some guidance to come out to help with that.

But you are ultimately always responsible for your own security and the shared responsibility model. We're getting there, you know, really work this out. Like the reality is that most of you out there is working, your vendors are good, your suppliers are good, your customers are good.

You, you manage to figure it out all out. We're all, as we said in the beginning, we're all in this together. Don't give or take, you know, a, a blame, just, just work it all out.

But, you know, we're a long way from being able to actually see through our relationships in an a, uh, uh, in, in a fast enough cycle to use that information. So at the end of the day, yeah, Mitch, you right? Yeah.

You're responsible for your own security. I'm a simple guy. So help me understand this.

If I leased you a car and I knew that you didn't know how to drive it, and now I'm gonna charge you extra to, for services to help you to drive this car, am I not kind of setting up a, a, a, a failure by definition here? If I look at all these cloud service providers, every one of them seems to offer a value added managed security service where they manage all this stuff on your behalf. I feel like a large percentage of that should just be baked into the actual original service and that there's a, a, a little bit of a game being played.

Or am I being too cynical on this, Mitch? Well, I, I do think, you know, service providers should set their customers up for success. You know, you can't sign up for a Google account without two factor authentication, at least pushed on you if not required.

And in this day and age, I think that's the, that is a responsibility that providers have, you know, just can't say it's, they didn't have any responsibility, then there'd be no password formulation requirements. Yeah. Enter your password.

1, 2, 3. Okay. No, it's all up to you.

You, it's your job to worry about making sure yourself, either you're secure. So what's the difference between, you know, having password standards and, uh, requiring multi-factor authentication? Really not much in this day and age.

I think a consumer might be, you know, a little put out about a, a two factor still, even they're having to do it, they do it on their phones all the time, right? And some code or a biometric. And, uh, I think that should be a basic element of it.

So I think there is a responsibility that is a, as a service provider, especially when it's something as sensitive as your data. I mean, but our data's everyone in the cloud. So at least that's my position.

I dunno, what do you, Chris, am I off base or, uh, seems to me this is pretty basic stuff. No, I, and you said that. Well, but you know, to Mike, to your question, I think you, you have a healthy, uh, citizenism in that, you know, I, you, you can take that too far, but, you know, as long as you have teams, you know, I, I have a, you know, a healthy optimism, I tend to think that folks are doing things right.

You know, when you to keep an eye on, and this is a good example because features get baked in, you know, windshield wipers or seat belts, air conditioning, you know, so you can watch as an industry develops and what are they charging extra for? That's not baked in yet. And yes, it all gets baked in eventually.

So we're where a healthy cynicism, cynicism comes in because, you know, service providers can, can, you know, make a lot on the mar to everybody, you know, selling those add-on services just a little bit longer, or a little bit less, or a little bit extra than they really should by now. Right? So, you know, I will, I'll save my commentary on any particular cloud providers in this case or anybody else, but you, we should all keep an eye on that.

You know, some things should get baked in over time, and we, and we shouldn't let, uh, momentum carry us too far. Ask what, what should be default? All right, so then my next logical conclusion is, Mitch, do we need Ralph Nader back here to write unsafe at any speed for cloud computing?

Is that what's going on with this thing? It feels like, you know, we're giving people access to these services and there's no breaks or seat belts. Well, then we'd need a Corsair of the cloud, right?

For him to write the book about problem is just too many, too many choices to pick from, though none of 'em are safe at any speed. I don't, you know, so I posted something and I'm, and I don't know if I've got flamed yet, but you know, as, as much as we talk about security and especially national security and security of our infras critical infrastructure, um, it's beside me that we don't have a, an equivalent of a Secretary of Defense, but Secretary of Cyber, uh, defense for the nation. It is equally the battlefield, uh, of everything, right?

And it's the subtle part of war. It's like the, it's the non declared war that we do. So I think that's the book to write is like, where the heck is our cyber defense?

It's there, but it's never above board because our news cycles are only about two hours long. So by the time we talk about, you know, the snowflake breach, as soon as we're done watching, watching Textron Gang, something else has taught happened. And so we kind of forgot about Snowflake until, unless they pop back into the news.

And so it's not, it's not top of mind of, of securing all of this because we're so used to, oh, more data got stolen, more keys got stolen, more passwords got stolen, more personal information got stolen. Okay. Uh, nine o'clock time for a second cup of coffee.

I think we're just numb to it. I think you're right. I was just having this conversation with some of the folks who write for Security Boulevard for us, and it's starting to turn into like the local crime blotter.

I'm like, there's a, there's a breach every day now somebody's involved in something or other to the point now where I was just asking the question, is this news anymore? Does anybody care if there's a breach? I mean, Chris, what's your sense?

Have we just gotten to the point now where, uh, we're just inured to the whole thing? Well, You know, I'll sort of echo my last comments, right? You know, there's, you know, to be clear, it's all working.

It's still running, you know, decades and decades, you know, now. And it generally works all the time. So it's not as bad as we might think.

However, um, it is as creaky as it looks at the same time, right? There are massive risks, and we've avoided a lot of them by not doing all the wrong things until it was too late. Right?

And we can't ever forget that. And Anthony, Mitch, your, your, to your point, I agree. I think there should be a secretary recycle or something of that level.

And, and, uh, yeah, with the, you think the defense of, if it needs to be seen that way, we to explain, you know, how we got here, like in the, the US federal government is the biggest slowest entity in all human history. But it gets there usually on, in the end. And Howard Schmidt, you know, lake grade, Howard Schmidt was, you sort the first cyber are, and we had rules like that, you know, when you get right down to exactly where that seat would be and who gets it and where the authority comes from and so on and so forth.

Yeah. That can take a while to do. But yeah, I think, you know, I think, you know, we have a space force, which I don't really think it's necessary, but, you know, it is a new, it's a domain.

You know, I can understand the argument, you know, a cyber force, you know, focusing our efforts at, you know, at in any nation in the us, uh, at that level on these issues. A hundred percent, yes. Yeah.

Or who knows, maybe the cloud service provider should get together and have a SWAT team put together to help customers go deal with these issues. And maybe it shouldn't be a, something I pay for, it should be a free service in the interest of national security. Well, and that that's per per perfectly reasonable suggestion.

And things like that having happened in the industry, you know, we, I've done that, you know, in various hats. You know, sometimes we just say, look, you know, we're, we're Cisco, right? You know, the turn of the century.

Cisco had a deposition. We did all sorts of things because, you know, they need to be done and nobody else can then do it. Yep.

And ironically, it makes people like you more, so you sell more products, you make more money anyways. So it's not a, not a terrible idea. The government, again, is big and slow and stupid that God bless their pretty little hearts.

But most of this stuff needs to be done by us in the private sector, can just choose to do so. There you go. But, you know, hopefully it won't be us taxpayers paying for that.

'cause maybe we didn't set this issue up in the first place. I'm Mitch, speaking of making money, it seems to me that, um, lawyers are gonna be making money off of this particular incident for years to come. I mean, let me get this straight.

We can sue Ticketmaster, we can sue all the other customers of Snowflake. We could sue Snowflake itself, and we could sue the cloud service providers. 'cause they all have some sort of quote unquote shared responsibility.

So, um, what are the financial implications of shared responsibility? Well, so there's a cynical answer. Maybe it's cynical for what day is it?

This is Monday, cynical Monday. Um, yeah, there, there's a, if if the financial impact were significant enough, this would happen less. We would see it not grow as fast.

The number of breaches, and I've said this for a long time, it's like, you know, until there's really, you know, think about it. If this was like SOC compliance, right? Where the CEO had to sign off that the financials are accurate, and, uh, you know, it's, it's disclosing the right information.

Uh, I don't want, I'm not saying it's the CEO's job to sign off on security, but you had, if you had something like that, I bet you a lot less, um, breaches would occur because it would get a lot less scrutiny. I just came back from, um, AWS reinforced this week. It was really interesting to hear, uh, Chris Betts talk about their culture of security.

They don't call it a security culture. They call it the culture of security. Not sure the difference, but, um, one of the first, well the first thing he talked about is every Friday, uh, their C-E-O-C-E-O of AWS sits down and works through, talks, through, works through what the escalated security issues be are in the organization.

This is, this is across all, and all of AWS not saying he solves all those problems, but they have an escalation process and anyone can escalate it. Security issue. And, you know, I'm sure sometimes it's budget, it's sometimes it's other factors.

Sometimes it's kinda holding people, you know, feet to the fire, whatever it might be. Um, at least that's what they claim. They do that every Friday.

Well, okay, that tells me, okay, it's pretty important then if the CEO's gonna take their time to re review and work through issues. Both they're informed about what's going on. And people also know that it's an important, so I, I think it's more than just a visual, oh, that looks nice.

It must be important there. I think they actually take it seriously like that. I don't know.

Here's the part that makes my head explode. So Let's say that I was leasing you a car and I leased you a car, and I knew perfectly well that you had no idea how to drive that car. And you, and, and then you went out and caused all kinds of mayhem.

So now I have a cloud service and I'm giving that to developers, and I let them provision that stuff and use that knowing perfectly well that they have no idea how to drive that cloud platform securely. And then all these bad things happen. So, um, I feel like we gotta have a better way, Chris, but am I crazy or what?

I you're right. You're, you're talking about the future, right? You know, and the, uh, quote is always Terry prt.

You know, there's an acceptable level of nemesis, you know, that we have to live with. And, and, and it's not only okay, but it's, it's desirable in some way. You, you, you ask about, you know, is this going to be, you know, lawyer's gonna make a lot of money, not this.

Oh, right. And they always will. And as long as you know enough, but not too many lawyers are making enough, but not too much money on these issues, it'll continue to push those levers that make the CEOs make the choices.

You know, you know, not as early as many of us would like, like them to, but more often than not, just barely, not too late. Mitch, last word. If anybody, if any idiot can buy a chainsaw, then anybody can buy a cloud service.

How's that? Al Monday? That's what happens when you put me on the road too long, Mike.

All right. I think we all need to take some responsibility for this and stop passing around to each other. And to Chris's earliest point, finger pointing may not be the most useful thing here, but it does seem to me at the very least that we're all not stepping up enough here.

So that's the last word there. And we'll be back in a minute to talk about our next topic. All right, folks.

And we're back and we're gonna talk about a little more insanity that goes out on the land of cybersecurity. But we have this, what sounds like a positive development. The FBI announced that they have somehow or other discovered 7,000 deen encryption keys that can be from the lock bid service folks, and they're making those available.

Um, the problem seems to be that, uh, people are hesitant to call up and say, I would like those keys. 'cause they may not have reported the fact that there was a crime in the first place. And so now they're kind of like, well, I don't want to get fined for, uh, not reporting the crime.

So there's this kind of, you know, damned if I do, damned if I don't feeling out there, Chris, do we need to kind of think about this a little bit? Or is this just the fact of life that if you don't report the crime, you're gonna do the time as it were? I don't know, honestly.

Right? You know, so you, as I, as I understand the story, the FBI has decided not to, you know, they, the, they vote release those keys where you have to ask as opposed to just posting on the site. You know, so anybody can ly go and grab them and see if you know they work or not.

Um, I think that would be the better call. You know, so maybe, you know, we're talking about lawyers at the end of the last, uh, section. Maybe this is one of those things inside the FBI, the lawyers are saying, you know, no, we shouldn't do that because of X, Y, z.

You know, they got the information from, as I understand the ally. Uh, so maybe they can't. Uh, but if you're gonna give 'em out to anybody that asks, you know, why don't just post 'em all online, Right?

Speaking of where they got the keys, as I understand it, they got it from our, uh, British friends. So maybe they posted it already. So maybe we just gotta go look for a different website.

I don't know Mitch thoughts. I feel like I'm, uh, as I said earlier, I'm traveling a lot. I feel like I'm at the airport.

Well, the passenger who left their keys at the TSA checkpoint police return, pick up your keys. It kind of seems like that. It, it, I dunno, I I found this story a little really interesting.

Like, come and get your keys. You lost your keys. Well, if they have 'em, who else?

And they got 'em some locked, but, well, who else has 'em too, right? So that's my concern is didn't you have replaced those keys by now? Or maybe this is like ransomware and you're trying to get your stuff back or something.

It didn't quite, it didn't quite connect to me. So I, maybe one of you can explain it, but I wasn't, I don't quite get why you would want your keys back. You should have replaced them by now.

Well, yeah. It's the shared responsibility thing we're talking about in the last segment, right? You know, the, the, this is why at the end of the day, your comment, you know, Mitch, you know, is right?

Yeah. At the end of the day, you're responsible for your own security and, and always moving. And as we start talking about really sharing the responsibility, then you need to be able to, to connect all the dots.

I mean, how can you really take responsibility when you really don't know you? Because, you know, most people won't even hear about this story. You know, there's a lot of news.

This is one, you know, I, you know, for dabble, I may have been somebody who got the ransomware, but I didn't hear about the story. I don't even know the FBI has, you know, may have the keys. I don't even know that.

And if we're really going to try, you know, on a, you know, serious level, a corporate level, a national security level, to really put together systems that are fast enough, you know, that we get work through this shared responsibility model, uh, it can't be that you hope you may have seen a reel where somebody mentioned a headline and found out that the website that you're gonna ask to be your keys back from the FBI. So again, I'm a simple guy, but it is the Federal Bureau of Investigation. So can they not figure out whose keys these are and give them to 'em?

I, I would, I I would, I I don't know. I mean, so so you think this through, you know, if the UK got them from, you know, the bad guys, then they might have not have gotten the data, you know, to tie those directly back to where they're from, though, of course the bad guys would have that data. That's the whole point.

Most of having the keys, unless, you know, there they go it, so you can, you know, blackmail people to get 'em back. Uh, yeah. And, and, but you know, to your, to your question, if they did, I think yeah, they should just contact 'em.

Why not? I guess you live in this, you live in this space, Chris, so it feels like the public private partnership around cybersecurity is, shall we say, uh, not well-defined. So what's going on in this conversation out there?

I know you talked to these guys in Washington. I mean, what's your sense of where are we and what needs to be done here to make this kind? Just take the friction out of it?

I think we need to keep talking and keep trying, right? You know, the, uh, the, the public and private is always different. Now.

We pay taxes, they spend the taxes. You know, that's, you know, it doesn't matter where you start on the, your thought process, ideological spectrum and whatnot. Um, but, you know, the, the government, just like companies just made of people who just do things, you know, for a living.

And, uh, my experience, you know, they tend to be just like in the private sector, you know, they tend to be decent people doing the best they can. But there's, you know, we were talking about lawyers and it's lawyers act when they get to extremes bog, but we have lots of actions before those, lots of impact before those extremes. You know, people don't like, you know, getting into situations that may lead to a lawyer someday.

And, you know, so whether I'm an employee at the FBI and I'm say, ah, how do we handle this? Gotta make a call, take the safe that, um, so we handle all that friction anyways. And as I've said here, and I would say frequently, some of this stuff just takes a long, long time.

You shouldn't be surprised, you know, it literally takes a decade or more to build one in the aircraft carrier. How long does it take to build one in Department of Cyber with a secretary of cyber and everything else, you know, 30, 40, 50 years. You know, these things don't happen overnight.

And, and I, I think, you know, I gave a talk to, uh, on, on Wednesday of last week to, uh, a group, you know, in the beltway in the intelligence media and so forth, looking at it, patient integrity. And my recommendation there, I think is the, is the answer to your question, is lean into what our strength is. You know, our strength is openness and transparency and democracy of freedom of speech and open source, all these good things.

And I, I encourage, in the corporate world all the time, we sort of touched on that in the last se segment, but do the right thing as a corporation, ironically, you'll just make more money, right? Because people like dealing with companies that do the right thing and as government agencies and employees, you know, work towards transparency, right? You know, there's rules.

You can basically back up any action you take in inside the government because you live it inside nothing but a policy environment. You know, just being more every take, every opportunity you can to be extremely transparent builds why you did what, what basis that was on. So there's just less mistrust.

We talked about responsibility last segment, and it does seem to me that we've been dealing with this whole ransomware and encryption stuff for, I don't know, better part of a decade it feels like. And yet we still have these issues and we are not resilient enough. I mean, at this point, are the bad guys doing things that are, you know, changing tactics and evolving in ways that we can't handle?

Or is it just kind of getting to the point where it's simple negligence? Well, I, I think it's more than just kind of security policies and filling out questionnaires, you know, getting information from our suppliers about the security and the certifications or testing that they've had and, and past and past. There's other dynamics that come to play.

So for example, in the healthcare industry where literally lives are on the line, in many cases, if, if they've been attacked ransomware, they're not able to operate. Um, it's one thing is somebody's money's locked up and you might have a financial loss. Yeah, I'm not saying it's not significant.

It's a different, you know, if people laying on the operating table, not to be too, too dramatic, but you have people's lives. And so there, there is, in, in the, you know, hacking world in the, in the bad guy world, if you will, they know that hospitals will pay the ransom, uh, healthcare entities will pay that because they can't wait. They can't wait to, to come back and kind of deal with it.

Have somebody go and negotiate with them, they'll just get over it. So there are, there are places where it's not that people are more vulnerable, but they're more susceptible to the financial outcome that the bad guys are looking for. So you kind of have to look at it systemically, Mike.

'cause it's multiple factors, not just, you know, some are better at protecting themselves from ransomware. That's true, but it's more than that. Alright, Chris, you got any thoughts here on what is the definition of, you know, I'm a victim versus I just kind of simply negligent?

Well, I, I think, I think when we're in a happy place, you know, most of this is simply negligence, right? The same reason, you know, the office, you know, with all these stories we read about, you know, a contractor building contractor negligence happens. Now, if it happens 90% of the time, then no building stand up.

If it happens 0% of the time, your control is probably too tight. Uh, so as long as our problems tend to mostly be negligence, and I think they kind of are, right? You know, as we discussed earlier, as much as we can spread the responsibility and lots of parties, you know, a fair responsibility when anything happens, uh, were you driving the car, you know, oh, I was driving the car.

It's like, yeah, I can understand how this happened. Happens all the time. But you get the ticket, you have the responsibility.

'cause yes, you could have taken the time and, you know, so as long as the incompetence level, you know, a talent for most of these problems, and it's not, you know, stopping everything, then it's not systemic. But to Mitch's point, yeah, there are a lot of systemic issues. And you know, as I concern myself about this stuff, I don't like seeing any of those stuck for a long time.

You, you see 5, 10, 15 years go by and, and a problem is still a problem. It's either acceptable or we're missing something. And we usually are.

And that's when, that's when Mike, you know, your citizenism is really involved, pointed. We need that. Someone say that's a problem right there and make people think it.

So who's responsible for this mention? And, uh, I'll use that car metaphor. Um, if I take my car and I drive it to a disreputable part of town, and I leave the windows down and the keys in the ignition and the car gets stolen, well, it's still a crime.

It should not have been stolen. But I am, I'd be kind of stupid, wouldn't I? It's kind of back to the snowflake example, right?

Same kind of thing of, yeah, it's, is it the manager that hired somebody that didn't know what they were doing that was supposed to be, you know, performing security function at the, at the organizations we didn't know, you know, or didn't kind of have the, have the influence to be able to say, we, we really need to set up two factor authentication ourselves, or whatever it might be. It, it's a mix, right? And on the other hand, if the manufacturer's handing you a a car and you know, it, it, it basically, it basically has, uh, uh, problems with it that safety issues with it, guess what?

They have to recall it. They have to do something about that, right? When enough people have reported that it is, is an issue.

So it also happens where, you know, the providers are on aren't always gonna get it right either. So they have to be responsible and say, Hey, you know what? It's time we yet to take these measures.

We learn something from this breach, or we discovered something we have to do better in our security, or we're beefing this up, we're changing our APIs. I've had that happen a lot in the last two years of people saying, eh, we're not, not really hand comfortable handing you a key anymore or a token to use the API anymore. We're gonna have you go through this interface and then for each application you get, you know, whatever it is, certificate or maybe a maybe maybe a token or a gateway to go through to have better security.

And that's what we have to do. We all have to upgrade our security. Whether you're in the service provider world or you're in the application development world, or you're in the, uh, security organization, the CISO at, at your company.

Chris, back to the shared responsibility thing, um, I feel like we're trying to blame too much of the security folks for aberrant behavior that really belongs to the end users sometimes and probably more often where it's the end user who didn't do something or didn't follow a policy, and yet we're trying to hold the security people accountable for the end users. I guess, you know, we're entering a political season and I'm kind of like, you know, who should we be locking up here? You know, the, the, you know, I like responsibility.

I think it's liberating, right? You know, it's nice to know, okay, I'm responsible for that, right? And it's, uh, I, I think that's, uh, but, but again, you know, shared responsibility requires ability to figure out what you're responsible for, right?

And in the, in the case of taking a security job, I mean, let's just, you know, focus in on all us people who work in security, you took the job, right? You know, and, uh, I agree. Mitch, you said a second ago, you know, someone hired you.

Now, did they hire the wrong person? Did they not resource you? Maybe however you took the job, right?

So I'm willing to, you know, and I, and security folks are, are some of the best people in the world, frankly, right? And they can take it. You know, every, every one of us knows you took the job.

I did the thing. I knew they weren't gonna pay me enough. I knew they weren't gonna get me the, the authority.

I could not take any job. Now, maybe you need the money anyway. We all make choices.

You know, responsibility has to start with the individual. And, you know, if the, you know, at the end of the day as the board and the, you know, in a corporate environment, that's it. That's where the responsibility really is.

They didn't empower or instruct or whatnot, the ceo EO and then the CEO from an operations per perspective, they're responsible. Period. Done.

However, don't take the bloody job if you can't get the work done. 'cause we all own a slice of it. All right?

You heard it here folks. The other side of responsibility is called accountability. And we all have to answer for it at the end of the day.

We'll be back in a minute to get into our next topic, but it's just gonna be more and more chaos. Here We are. I'm Bonnie Schneider, sustainability contributor to the Techstrong Group.

I'm excited to introduce you to a groundbreaking new initiative from Techstrong Research, the sustainability pulse meter. The pulse meter offers valuable insights into how environmental responsibility factors into tech purchasing decisions for key players in the industry. Position your company as a leader in the industry and differentiate from your competitors with the sustainability Pulse meter offered exclusively from Techstrong research.

All right, welcome back for our third block of insanity. So we have had an acquisition in the security space. Fortinet has acquired Lacework.

Lacework is a provider of what's known as a cloud native application protection platform. A cmap, and maybe I get this wrong, but I'll let Mitch clarify, but it seemed to me the whole point of these synaps was to roll up all these tools so that I could reduce my friction and my cost so that that way I could standardize on a common platform. And we were supposed to roll up everything through the cnap, but now it turns out the CNP providers are being acquired by other people.

So who's rolling up who here and for what purposes? Mitch, what do you think? Well, I mean, c synaps specifically to support more cloud native architecture during microservices API first kinds of applications.

Whether they're using, obviously probably using containers, probably using Kubernetes, but not necessarily have to be. Um, and, and it's really providing you, just speaking generically about Cena, provid you things like, uh, API discovery, API gateways, um, authentication, ident identity management for machine machine identities, things like that. So you don't want every application going off and inventing its own, you know, machine to machine identity mechanism.

Just like you wouldn't every department go create their own person identity, you know, IAM solution. So it, it, the idea is to pull that together because guess what? Every app needs that.

What's interesting about Fortinet is, you know, Fortinet ISS pretty easy to view them as a traditional security vendor security provider. They've been around a long time. They've, you know, they're in the sim world.

Um, you know, doing SOAR XDR, things like that thing that traditional security people would, you know? Yeah, yeah, that's what I know them for. Um, and now this is with this acquisition with Lacework really helps them step into the software architecture of security world.

Um, that, so it's, it is, I think it's a smart purchase from that standpoint. I'm not sure everybody living in the security world yet knows kind of cloud native and what all that means, but I'll bet they're dealing with it somewhere in their organization. They need to kind of help make that transition.

I gotta believe Fortinets looking at as we can help you get there, we know how the Cenaps solution. Alright, Chris, I don't know if you have a thought on the actual deal, but it does seem to shine a light on this whole tool sprawl issue within cybersecurity. And, um, frankly, I sometimes think where all these tools are resulting in us working across purposes, but I don't know, do we, do we have too many tools or not enough tools or just not the right tools?

Well, it, you know, you know, to sound like a broken record in many ways, you know, when if we have too many tools, you can't find them and literally have tools here at the right tools on hand. And there's also the, the tools that get built into things. So I have to deal with the tools myself anymore.

But, but I think this one, I think that this one is interesting. It's, I see Fortinet in the role of John Deere, right? You know, John Deere just very recently suddenly has more software than hardware engineers, right?

And they make physical tractors or these sort of things. And I know I gotta get, uh, meaningly named by my Fortinet friends, but you know, coordinate, you know, sells G got, right? Yeah.

Flying us to the firewalls and yes, yes, yes. Other things. But, you know, I think Ally put it better than I I'm about to.

But, uh, they're more that traditional play and cloud native and, you know, the, the ought to use ai. What does it even mean? How we bolt all these things together.

So, you know, whether this individual deal, this merger, you know, works out well, we'll see. But I think this bolting the traditional, you know, network providers, the infrastructure providers, the Harvard providers, the security infrastructure providers, um, into players like laser were, seemed like the direction were going. And, you know, Chris, I was thinking about it some more based on what you said.

Um, and I mean this in a positive way, Ford that is kind of a modern day belt and suspenders, right? This is sassy and web application, firewall and bot protection and uh, cloud native firewalls, all that kind of stuff. So it's, you know, they're in this, they're in the modern world of, of what we think is the security protection.

And I think with, with, uh, Lacework, it kind of gets 'em into the software architecture side of this. And what really happens inside of those applications is 'cause what they look like a network, they just talk to each other over APIs. And it might be on the same cluster.

They may not be, it could be fully distributed. So it's, it, it is a logical step. Um, then I think, you know, we're in the acquisition era, right?

Cisco by Splunk, um, roll up these days. And so I think we'll see more of this kind of activity. Yeah, I think to your point, Mitch, what we're seeing is all these tools are becoming features of a larger platform.

And ultimately, I think, know there's pressure to reduce the cost of cybersecurity. 'cause otherwise you gotta integrate all these tools and then you're basically giving yourself whiplash as you swivel from tool to tool to tool to try to figure out what's going on in the environment. And each tool is telling you something different at a different point in time.

So, I don't know, Chris, I mean, have we just gotten so tool happy that, you know, we're kind of becoming our own worst enemies? He, yeah, I guess it's probably the risk. You know, I, I literally love tools.

Well, you know, I've got, I'm always sort them and using them and categorizing that I'm in, uh, we were talking before in the green room before this. I've pulled out a bunch of gear and a hardware and associated tools from 25 years ago, you know, but they haven't taken any of my tie to your point, Mike, you know, I haven't messed with 'em, but I keep them outta hand and they're available. Um, but, uh, but you know, so as we do this a lot, and I think, I think again, we continue to build things in, you know, security products and services that were a good idea that, that were structurally significant get, and some go away because we stopped doing things the way that, you know, ended up needing to have that security feature in the first place.

You know, a lot of security is compensating controls. Like, you know, what we should do is a DC, but we're not gonna do that. So instead we'll create this market segment and we'll stick this thing in there.

And, you know, if you're the vendor, then you can make money. Or if you're the consumer, you can solve that, which you, we can tell it fit and then that, you know, so there shouldn't be any, there's not an infinite number two, those are actual screw drivers, right? Some tools are just always tools.

Other tools are fans and fade away over time. Yeah. Vince, we have this notion in the land of DevOps called platform engineering, right?

We're trying to figure out how to run processes at scale. Do we need to have, you know, platform engineering extended out to cybersecurity and do we call it platform security engineering? I don't know, but Well, um, you know, the function of platform engineering versus platforms, right?

Kind of two interconnected topics. Platform engineering can be more about, you know, the setting up of standard environments, whether they be security, right? I mean, yeah, there's security patterns and templates and things that we can do.

And matter of fact, a lot of platform engineering groups are about improving the security of the images, the base images that they're, that they're constructing their templates and their configurations out. It's also about how to keep that up to date. I think the other side of platform though is whether you're one company acquiring the company, bc, D, e, and F, you don't want to be, uh, you know, like this loose collection of we're we're just making it easier to order off one price sheet.

And that's kind of what our integration strategy is. That's not, not so helpful as it is in, in today's world. You have to be able to see data across all of these TA tools and then understand the context of it so you know, action to take.

And there may be workflow that crosses all that. And I think in a security context, that is what a platform is about. So when someone acquires another company, there are integrations that they can make.

That's all good. Well, what about the data layer? Think about, um, thinking about having the data platform across all of those security tools that they're providing you or plugging into other things like, you know, observability platforms and hotel and things like that.

In, in today's world, things happen too fast to do very much manually and at least keep pace with what's happening manually. And we can not only automate, but we can, uh, make our lives easier to get access and understand and then use data across security domains and tools. And by the way, application security, right?

Cloud security provider, all that stuff, I think that's the key to keeping pace, at least Chris, the part of this that as makes my head explode, there's many things that do lately, but we've been saying that, um, there's a shortage of security professionals forever and ever and ever. So who's, who's using all these tools, but we don't have enough people to give the tools to. So who's figuring out how to use all these tools?

Uh, you know, the, the, the Institute of Shelfware is a thing, right? You know, a lot of things get bought, you know, you know, as a vendor, you, you run into this, I'm really happy someone above my thing, and then you can's say, oh, I to find out what you did with it about nothing. Right?

You know, so that's an actual thing, right? Mean as they, oh, pine the minute ignore. I think we gotta be careful.

You gotta just, you know, we're enthusiastic about the acknowledging we're gonna do the stuff we're, you know, don't acquire more tools and more things you can actually bloody use. Uh, use the ones you have as much as as much, you know, their best effect. And, uh, and, and they think back on, you know, the old tools, you know, a lot, a lot of times, you know, new problems actually work well with resistant technique.

So I think it's, yeah, there's certain serious, you know, that's an intrinsic risk of, of free. I don't know. I know.

I I don't think we're any worse. Well, yes, we are worse now than we have been because we have more choice to, but I don't think the problems me worse. It always been, it's not a matter about how many tools we have or how many people we have.

Is are we doing this effectively for the right reasons with the right authorities? And that has a done what? I don't know, Mitch, I can, I can get a little paranoid sometimes, but have we allowed a cybersecurity industrial complex to evolve here where there's just this massive amount of money and infrastructure all chasing, you know, all these threats without actually maybe solving the problem?

I feel like I just watched a World War post World War ii, Eisen, president Eisenhower, you know, we careful of the, of the industrial military, industrial complex. Um, it, it's easy, lemme say it this way. It, it is easy to, to become, have, have a mindset where we think the tool is the solution.

And, you know, have, how many times do you maybe walk into somebody's garage and they got a wall full of tools and, and you say, let's go work on this. They grab, you know, three things that they take out to go work on something. Start with that.

There's like a quarter of 30 things that I need to do. 80% of the jobs and all the other stuff are sort of those fancy situations. Or maybe they're shelfware, maybe I use them too.

It's kind of the same thing in, in security, right? I mean, you, we removed from a checkbox mentality of network security. Well, okay, I got my firewall, I got my prevention, I got my web application firewall, I got my content, uh, my data protection, whatever it might be.

So I've got every one of those pizza box in the rack, so I'm good, right? And I, and I'm being over simplistic, of course, but you, you can't take that sort of checkbox mentality because what it's all about today is how you as operations, it's all about the soc. It's all about the in security incident, uh, teams and how they respond.

Um, and of course compliance and having data to verify, say you say you're doing this and how can you demonstrate that you are so, it it's a complex world. Um, so I think you're, you're better off to use fewer tools really well than a lot of tools. Not very well can be general about it.

I mean, hopefully that that's more than common sense, but sometimes common sense isn't too common. This is true. Chris, what's your take on ai?

And I'm asking this question because is that not an opportunity to kind of flatten this whole infrastructure and maybe we'll have a common data pool that we're running algorithms against and we can have a common view of the events. I mean, are, are we on the cusp of maybe changing this because AI will force the issue, or is that just one more tool added to the pack and it is what it is? You know, the, the fact that you of all people when asked that question right now, uh, makes me think that maybe we kind of are right.

You know, I I I'm very hesitant to, to wax too poetic about all the promises of AI in, in the, in the short term. 'cause it seems everybody's doing that and it kind of seems true, right? You know, all of the, we're talking about, you know, time to visit, you know, time to transparency and sort of each of these segment segments today on the show.

And you know, the reality is if you have to have a staff of 10, 12, 10,000, 12,000, you know, know people reading things to actually get, you know, far up longing in time to, to achieve with some goal, you're not gonna do that or almost ever gonna do that. That's the kind of stuff this large language model, you know, wave of, of psychology is bringing and just in the security space, yeah, whether I can just look just looking at supply chain, just being able to put all the pieces together in time to do something about it. Not post catastrophe, not, you know, you know, in some catastrophic, you know, year and a half in the, in the courtroom with boxes of records telling.

But like right now, how do we put all of this stuff together? Um, what we are calling a ai, AI right now has amazing potential in that space, right? And on both sides, you know, studies, you know, already on, on the negative, you know, the negative security effect of ai, you know, AI in the same tools, in the, in the hands of people, you know, we're not in favor of.

Um, I, I think this is a demand of defendants because we, we have the ability to put things together, put visibility, uh, together to put situational awareness together so much faster in ways that aren't really revolutionary than we would just do if we just had a lot of people. But you don't need a lot of people. There you go.

Hey folks, we're coming to the end of the show show, but I think the definition of insanity is banging your head against the wall and expecting a different outcome then well, welcome to cybersecurity. So I'm hoping things get better, but, um, right now I gotta say it from the outside, it doesn't look too good. Just We're a helmet of it, Mike.

We're out. Anyway, thanks everybody for watching the latest episode. The rest of the tech strong TV lineup is coming up right behind us.

By all means, stay tuned and thanks for watching. This is Textron tv. Well, I had the great pleasure of being joined by MIDI Dowdy, who is co-founder and CEO of, uh, Catchpoint.

Welcome. Good to be chatting with you. I'm still catching my breath from our week together.

Absolutely. Yeah, Mitch, it's, uh, it's uh, two in a row, I guess in, in, uh, less than a week. So, but it's a pleasure to be with you again, Mitch.

It is, it is. It's great to be back with you again. Um, I had the pleasure of, uh, also being part of, uh, app dev Tech Field Day in Santa Clara where medi and, uh, members of the staff presented and talk with, you know, an audience asking a lot of tough questions.

I think you did very well. Yes, I really enjoyed, I really enjoyed the format, to be honest. I think it's, uh, I think we should do more of these things.

That was my message back when I came back. Well, that's good. That's great to hear.

That says that it, uh, you got a lot out of it. I know we had a lot of viewers. I'll put a link to that video in this description when this goes up so well, for anybody who might not know what you all do, what the Catchpoint is and, uh, tell us about that.

And then let's get into kind of resiliency and how we need to do more than just kind of standard observability, maybe what some of the things that you all add to that. Sure. Uh, thank you, Mitch.

So, so we, we started the company in 2008. Before that, I used to run operations and monitoring for a company called Double Click, which was acquired by Google. So I did that for 11 years.

So I was title tested on the other side of, of yeah, order of, of, of Thanks, which is on the receiving ad on the, on the, on the customer end. And we, we, my team and I wanted, uh, to basically create a better version of the end user monitoring tools that were available back then. And so we embarked on this mission, uh, uh, to basically create an internet performance monitoring company.

And we've gone through different categories, et cetera, observability, et cetera. But I think at the end of the day, the, the best way to describe what Catchpoint is, is we are a company that allows, uh, companies to basically have a good understanding of what's working and not working from an internet perspective, from an end user perspective, whether it's your infrastructure, your services, your APIs, your ecosystems and whatnot. And, and really reduce that meantime to troubleshoot in a war room, right?

So you go into a war room, there is a problem, and basically monitoring is like peeling an onion. So it's painful, you want to eat the onion, but it's, it's a, it's a pain to get to the root cause. So how can we get as quickly as possible to where the problem is and, and eliminate all the things that, that could have caused the problem.

And the challenge now is just things have gotten so much more complex that finding that would, cause it's like we, we don't look at the, at the needle in a haystack. It's really finding a needle in multiple haystacks. So now I'm really playing peek with the, with the needle across this cloud vendor, that cloud vendor.

And it's, it's very hard It's days. It feels like a needle and a stack of needles That Yeah. Or even better, I might, I might steal that.

Uh, that from you. No, you're have at it, you know, um, I think just the human mind likes to compartmentalize problems, right? So we understand what we understand and we either assume the rest is okay, if we're running our own stack and applications and, or maybe we blame the things that we don't, don't control, don't have visibility into, but we don't have the ability to then pinpoint what that is.

And if we don't consider the entire kind of system, if you will, of how what we're operating within, whether it's our code or our infrastructure, cloud infrastructure or DNS or you know, security systems that are managing traffic and blocking bots, whatever it might be, it could be so many factors that are actually either singular, singularly, or in multiple ways contributing to a slow down or performance or not delivering the kind of experience that you want, Right? Absolutely. So I think the, the, the way you described it is really well, so think about the, for me to open a banking website or a Amazon or anything that we all deal with on a day-to-day basis to either order a cab or order food or it doesn't have to be always a technical thing, but at the end of the day, we're delivering customer experiences to people around the world.

Uh, and uh, so that delivery chain is almost like building a car it, like, all the things need to be assembled at the right time, and you cannot have a single millisecond, except that it's not sequential. You have multiple things happening in parallel to make that browser page, uh, look good on your browser or on your mobile app. And so the, the, the complexity is just increasing there.

You know, I think when I started in this business, uh, the Catchpoint one, I think a single webpage had what, maybe a hundred requests, 60 requests, 60 different objects coming together today, it's not uncommon to see 600, 700, a thousand. I had the customer a few years ago ask us to basically increase our, what we call our waterfalls, to be able to display 2000 objects on the page. So we, so you have 2000 things that need to happen at the speed of light, literally for the user to have a good user experience.

Otherwise, there's, you know, if you're ad supported, you, you drop your ad revenues. If you're, if you're selling software online or whatever it might be, uh, the, the impact can be huge. But this, this, literally, this complexity that is just mind boggling and then figuring out where the problem is so he can call on the right person.

Uh, somebody on my team was on a crisis call with a customer of ours because sometimes we, we are part of the crisis calls, and they were over 900 people on the call. Wow. So two, three o'clock in the morning, because usually somehow outages always pick two, three o'clock in the morning to happen, right?

While you're on vacation. While you're on vacation, or usually on Fridays even worse, right? So 900 people on the crisis bridge to say, okay, is it this, is it, this is it, this is it, this.

And, and because there's so many systems that are involved in, in, in, uh, uh, uh, a, a bigger system. So, and, and that's the complexity is just a growing exponentially to be honest. And that's scary.

And there is no, uh, there is no be, uh, there is no universe where this is going to shrink to, oh, we're only going to have two calls on the webpage. Right? Things are just getting worse by the minute.

I have trouble getting, uh, three kids and, and dinner on the table to all show up at the same time. I can't imagine a call with 900 people, And you have no lead network latency in your house, so. That's Right.

I shouldn't have a problem. You know, I'd, I'd love to hear a couple of things. One is, you know, you, you've been around since 2008.

Um, we all went through the, uh, covid crisis, the work from home. But one of the things that changed during that time period is of course, this rapid move to digital experience. And we've, you know, accelerated so much of our work to become digital, uh, whether it's delivering to customers and partners, operations of our own business employees, et cetera.

How, how did that, that window, that emphasis and that rapid rise and really putting the emphasis on digital experiences shape what you and, and Cashpoint does? Yeah, it's great. Literally, you, you nailed it.

When COVI was the greatest transformation from a digital perspective that, uh, uh, that anybody could have asked for. I mean, I, I hate to use that because it's a lot of people died and it's a catastrophe at human scale. But from a digital transformation perspective, that was the biggest boost that, uh, that has impacted every company.

E everybody that had plans to transform or to improve their digital delivery, et cetera, had no choice. I mean, you had no choice. You either ab embraced it or you died, or you literally just disappeared.

And so, uh, I I, I, I think it was in New York where people had to learn how, you know, 70-year-old, 80-year-old people had to learn to use their cell phone to get food delivery. I mean, think about the digital transformation, um, at that level, right? Because there was, if you didn't eat, obviously that was a problem, right?

So, um, so CIOs, CTOs had no choice but to embrace digital transformation at, uh, at the rapid scale. And I, we saw huge, huge, huge transformation from our customer standpoint. I think the biggest thing that is still lagging in my opinion, is as much as most companies, obviously, uh, I don't think there, I don't think you talk about analog versus digital.

I think now today is just like, it's digital, right? So it's a, we assume as digital. Um, but I think from a, from a maturity model, a lot of companies are still thinking about, uh, from a binary standpoint, they're think they're talking about availability, for example.

They're not talking about performance. Where I think digital experience is the em embracing both availability and performance, because the customer expects a great user experience. And so I think when, when people don't think about user experience and they just, well, the website or the mobile app is up, you know, my job is done.

I think that's a great disservice because the customer is expecting the journey to be concluded in a good, timely manner because their time is valuable. And so that's the biggest, uh, in my opinion, still the biggest roadblock is companies and organizations still resisting the, the, the adoption of performance as another pillar of their digital transformation. I think one of the other examples that that's fascinated me the most, and, uh, we were involved in some, some, some work with some luxury brands, but when you think about it, one of the biggest, I mean, obviously people had to eat and people had to go to the hospitals and whatnot, so I think that was covered.

But you still have people that had to buy Rolexes that still wanted to buy Rolex, and you couldn't go to a store, right? Because they were closed. But the fact that even the luxury brands had to adapt to Covid, how, well, one of some of them adopted the ability to book a, a meeting online where you can go to the store and make a reservation that didn't exist before.

Uh, we had customer, we had, we had some of these brands where you could basically have the, the, the piece of jewelry delivered to your home for you to try out. So it just led to so many new innovations in terms of like, how do we deliver user experience to the customer in a different way? And I think that level of creativity was amazing.

But again, performance is at the, is needs to be paramount in, in those, in those journeys. When, when everything is in a digital marketplace, if something about the experience is less than satisfactory, changing is in many cases this moving to the next option, right? Absolutely.

I, I, I, you, you, so I, I use this example. So in the good old days, you would get in your car, you would drive to Costco, right? Or to Sam's Club or whatever.

Uh, and if, uh, the store tells you it's open at 9:00 AM then you expect the door to be at 9:00 AM If the door is not open at nine and you wait another 10 minutes, okay, nine 15, then it's not that I have to go and move on with my life. You have to get back in your car, drive another 50 miles to the next closest location or whatever. That is a very annoying thing.

Now, online, I don't know about you, but I can open a tab on my browser in less than 200 milliseconds, and I am gone. And not only am I gone, but the image, the brand image you left on me, on my brain is tarnished. And so how do you, all that money that you spend building, the trust is gone, and it's really about trust.

At the end of the day, Mitch, I really believe that everything we do about it in terms of delivering great user experiences is to establish the trust between a brand and their customers. I, I heard that, I learned that from this great guy, Rob Marquee, who was the inventor of the NPS scores at Bain Capital. It's really about trust.

Like at the end of the day, this is about trust, period. We're handing off our data, we're on the digital journey experience with Right. Uber website or application or, Or Whatever it might be.

Talk, talk about that. You mentioned, you know, 70, 80-year-old folks having to learn how to order food off their mobile app. What, what is such a move to mobile in addition to web?

How does, how does that change the world for your customers? It used to be kind of mobile was, yeah, I'd like fries with that. I'd like a mobile app too.

Now it's mobile first, or equal mobile to web app, which you can, what you're delivering to your customers. Sure. So mobile means a lot of things because you can, mobile is is it cellular mobile or is it wifi mobile?

Is it just bottom line is what it means for me is the same, which is you need to deliver amazing user, ex user experiences anywhere, anytime 24 7, whether it's on a mobile device or an iPad or a tablet or computer over sevener or wifi or 5G. So the complexity I was telling you about just keeps increasing for, for the customers is now they, they have to keep delivering, uh, the same levels of stuff through different channel, right? So when we talk about omnichannel, it's not just store and, and online.

It's, it's online desktop, online mobile, mobile app, this 5G wifi, six G six, wifi, six, et cetera, et cetera. So I feel for our customers, because they really have to keep delivering, and the bar keeps getting raised over and over and over again. Um, I think it takes a great leader from on, on the IT side on, on our customer side to basically be that visionary person that's going to, uh, establish strong SLAs in terms of delivery, in terms of rallying the troops, in terms of saying, Hey, we're going to, we're going to deliver amazing user expectations.

So user, user experiences, um, user expectations are Freudian slip here, right? So, but it's really about having the leaders set the bar very high and say, no matter where the users are going to meet us, we're going to meet them there, we're going to deliver awesome experiences. And whether it's 5G, it becomes a non-issue.

Some, some companies have resisted, right? Some companies are, well, we, we don't care about 5G because, or mobile cellular because we don't control. Well, where you're going to go and hand out, uh, at and t phones to people that can't get Verizon.

I mean, it's insane, right? So are you willing to write off, I don't know how many subscribers they have, but let's say a hundred million on Verizon, a hundred million on at and t, a hundred million on T-Mobile, right? So what you just block a hundred million users doesn't work Well.

And like, like we tend to do across the board is the next thing, we'll solve all our problems. 5G was the miracle will change applications, and I can't even get 4G at my house. So Something like, yeah, but then, but then you go to Europe and you can have like LTE on the subway, and you can have a video call that a hundred feet under ground without, uh, without any latency of some sort.

The, the differences are amazing, are amazing. You know, one of the things is, I'm thinking about the a PM landscape of one of the measures of Yep. Application performance and synthetic transactions, and is my application.

Sure, yeah. It, it, it is more than a monitoring. I mean, that was, that was kind of one era of monitoring, if you wanna think of it that way.

Yeah. How have things changed and how is it different? And how do you think about performance in the context of, uh, user experience KPIs or measurements of, you know, how can we really tell what the, the user's experience is, and are we delivering what we promised?

Yeah. So a long time ago, uh, when I was responsible for launching a double click, I started getting some alerts from this company called Gomez, which we used to use at the time that was giving us the, there you go. So that was, uh, my external signal, uh, system.

And, uh, I run into the no, uh, double click. And I saw everybody really chilling, like, really cool. It's like, you guys, okay?

It's like, yeah, we're good. It's like, look, is everything okay? It's like, yeah, the network is up, the data centers are up, the databases are up.

Everything was green except the end users were not getting ads. And so for double click, I mean, it was ads, right? And so, so basically inside the firewall, inside 17 of our data centers, everything was perfect.

Nothing was broken. Everything was working as expected, except that from an end user perspective, it was broken. So that, that first incident that happened for me in 99 was, if you want the, the biggest paradigm shift, and I literally, I turned off all the other monitoring systems.

'cause like, okay, if, if all these tools are telling us everything is okay, then they're giving us a false sense of security, then I don't need them, right? And, uh, so from there, I, I basically created this, this rule that said, okay, I'm going to think about performance. I'm going to performance with a big P, not performance, just speed, but I, I start looking at my, at my world in, in four categories.

So which ability, availability, performance, and reliability. So can I get to you? Can I get in the car and drive from my house to Costco, right?

That's reachability. Can I get there? Is the, uh, highway broken or not?

Once I'm there, can I go inside the Costco store? That's availability, right? So can, is the door open?

Can I get in? That's availability. Then it's performance, meaning that I am there.

Did I find what I was looking for? I came for milk. Did I find milk?

Et cetera, et cetera. And was the checkout easy or did I wait two hours to check out because the cash register is broken, or we got, uh, the, I'm sorry, the, the, our computers are slow today. So that's performance.

And then the other one is reliability. And reliability was a, was a, was something I got to learn more when Google acquired double click and we got exposed to the whole SRE culture. So reliability is like, are you able, uh, I remember this, it was amazing.

I, I somebody from Google, uh, can you, can you share me with us? Your, your data? And I shared the, the monitoring data of double click and said, this is a joke, right?

It's like you're tell you're showing me A-A-A-A-A time series, uh, that uses averages. And basically you're telling me that your performance is 200 millisecond. That that's bs.

He called, he called BS on me. He's like, I want to see your distribution. I want to see how well you're performing at 1:00 AM, 2:00 AM 3:00 AM 4:00 AM and at 10:00 AM when there is peak, right?

And so reliability was a new concept for me and for my team. He's like, okay, how, how, how can we deliver the consistent response availability, et cetera, 24 7. So it's like, how well are you able to deliver the reliability?

So those are my four. Uh, uh, if you want my cardinal rules, and I think from a monitoring a PM, whatever you want to call it, I think we need to be able to deliver that. So, sorry for the long answer.

So I just wanted to put that in context. So in today's world, I think it's extremely important to have, uh, different signals that are giving you the health of your business. A PM is very passive, and it's, you still need that.

I think it's extremely important to have also a more proactive approach to your monitoring strategy, to basically don't be caught your pants down. We, we are hearing about customers having a problem, right? That should not happen in 2024.

So you need to forgetting tools and names and brands and whatnot, you need different capabilities to basically paint that picture both passively, so a PM, open telemetry, observability, et cetera. But then you need a proactive approach that is constantly mystery shopping your environment, right? Can I resolve your DNS?

Can I, can I get your data centers? Can I, can I basically buy, uh, uh, uh, apples and oranges and, and add them to my cart and check out successfully? Because if you don't do that, and you're just waiting for, um, some baseline to tell you, Hey, we dropped the number of orders, that's too late, Mitch, you've already, you've already p****d off.

Excuse the French, you've already p****d off a hundred thousand users and, uh, maybe lost a hundred million dollars in revenue. I mean, who wants that? And I had this conversation a few years ago with, uh, with this, uh, or former, uh, colleague, a former colleague of mine from DoubleClick who ended up at this company in Seattle.

And he said, you know, I don't need the proactive monitoring or synthetic monitoring. I have, I have an a PM tool. And I said, how is that working out for you?

It's like, well, good. We have a threshold. If you get a hundred thousand alerts, then, uh, a hundred, sorry, if, if our APIs has a hundred thousand um, errors, then we know it's a problem.

So it's, I thought this was pre pandemic. So it's kind of interesting. It's like, so my analogy was like, if you're running a hospital, you're telling me you're going to wait for a hundred thousand people to die to say, um, Houston, maybe we have a problem.

Don't you want to find that out that the fifth person coming to your hospital, that there is a problem? And that's the proactive approach to monitoring. Um, so I wish people would be more, again, forgetting tools or this, that, but I think a proactive approach to monitoring is, is more important these days, especially because of the complexity of people accessing from all around the world, accessing on 5G, 4G, this, that, whatnot.

It's very important to, to proactively testing things. It's a definitely a different world. You know, you mentioned, um, it leaders being, being, having a vision capability, right?

Painting a vision of where we're going. Um, resilience is a topic now with many organizations, probably multiple definitions, I think, of resilience as, you know, the ability to, um, still be standing under unknown or unforeseen conditions as well as those we know we could handle. And while you may not be able to handle everything, but you're able to weather much greater storms, uh, how, how do you, how do you advise, uh, your customers on thinking about resiliency as opposed to just uptime reliability, right?

I say just up time reliability. Yeah, no, absolutely. Yeah.

So when you think about the world we all live in, is we live in an unknown, unknown. I mean, when you think about it, most IT organization know very well in 2024 to deal with the known situation, right? They have, they've seen it before.

They have a playbook, they have tools, they have all that stuff. The biggest challenge is the unknown. Unknown, right?

You don't know it's happening and you don't know how to respond to it. And that is the, the worst possible scenario, right? That's what throw companies apart.

And, um, and, and I don't think e even as humans, we, we don't deal very well with the unknown and no, right? That's the scary, uh, zone, right? We, we all live.

So I think from, uh, uh, what we tell folks when we get to have conversations with them about resiliency is like, listen, failures happen at Google. Failures happen at Amazon. So you are not, I don't care what your name is, it's going to happen to you.

So just admit it, just, it's not a question of what, uh, if it's a question of when and how bad if it'll happen, trust me, there is, uh, uh, it happens to everyone. So I think you need to build the resiliency, the redundancies, to minimize that as much as possible. I mean, for example, even a company as small as Catchpoint, we have three CDN com vendors.

We have 3D NS providers. Why? Because we've seen the playbooks happen where one of them goes down, et cetera.

You don't want to be caught off guard. Um, I think resilience as well is making sure that you have the ability to speak the same language. And one of the biggest PBS I have in our industry is the fact that there is so much decent decentralization that has happened that you have, again, big companies, you have 5, 6, 7 teams working on products, infrastructure, et cetera.

They're all using different tools. And so they're all speaking the different languages. And when the crisis happened, what one thing I've noticed, and, you know, talking to other, uh, other companies in the, in the field, uh, there is, uh, the inability to agree that there is a problem.

I'm seeing a problem. No, the other tool is not showing any problem. And, and so there is this meantime to innocence or you people that buy tools just so they can absolve themselves.

This is the one that drives me crazy, to be honest, Mitch, right? It's just like, no, my tool is telling me everything is okay. You know, it's, it's not me.

It's the network guys, or it's, uh, it's the, the cloud guys or the whatever guys. And I think resiliency is not that, resiliency is not about blaming the other team in the company. Resiliency is about to able to recover as fast as possible and learn from it, right?

And I, that's why I love the whole SRE thing, but the true SRE, not just the name SRE for the sake of saying we, we do a, sorry. Um, but, but resiliency is a state of mind in my opinion, right? Uh, it's, uh, and I don't think we do enough of it.

And that's why I've been, you know, you've heard me at that, uh, at, uh, the event we were in last week. I think it's time to have a chief resiliency officer in the company. The, the, the resiliency are somebody that can be that authority that can basically put teams together, unify some of the monitoring and observability data in one place, make sure that there is no f more finger pointing it.

That is one thing that is annoying me is the finger pointing that happens in companies, because the more people finger point, the less the, the less time you're spending on resolving and bringing the customer back online. It's interesting, I think of the failure's not an option. Actually, failure's required.

That's how we build resiliency, Right? Uh, you know, Mitch, the only reason, the only reason I'm sitting here in front of you today is like I took double click down in my career. Uh, I singlehandedly deleted the file that caused this chain reaction across all of our load balancers.

And there was a file. I said, I didn't know what it was. I just deleted it.

Stupid rookies mistake, right? And, uh, and, uh, I deleted that file and, uh, I was in my apartment in New York, and, uh, my boss calls me and said, did you do something? Like, yeah, I deleted that file.

Like, I don't know, did you do anything to the system? He's like, yeah, I purged a stupid file called one by one point gift. I remember that.

And he said, well, you took us down. We've been chasing the problem for three hours. I said, maybe we'll call Maddy and see if he did something and that.

So I was asked to come the next day at the office. I said, that's it, Diego. You fire me.

I'm done. I, I am. That's my boss, was a genius.

So that's why I said, leadership matters, right? My boss was unbelievable. He sat me down, I was with my peers, and he said, what did you do?

Tell, tell us how, what went in your head? What did you do? Because we need to build resilience.

We need to build things around the next guy who's going to make the same mistake as you. And, um, so it starts with leadership. It starts by creating a, a safe space where mistakes are okay, failures are okay.

Uh, I've never fired anyone at Catchpoint on, in any department for making a mistake. But I will ask them, what have you learned? Like what e even in my interviews with, with uh, uh, BDR and SDR R and SREs, like, tell me when was the last time you screwed up something?

And what did you learn from, like, what went through your learning process? Because that's resilience, that's reliability. That's the ability to think about, okay, what am I going to make sure to never do again?

And we don't do enough of that. People are still scared. It has to be, uh, how has that error or that mistake helped us or helped you be better at what we did?

Yeah. That's, that's the real value. Otherwise lost opportunity.

How do you, yeah, how do you get back on your horse, right? How do you get back on the horse? How, what have you done?

And I think we don't do enough of that. I've seen some amazing companies, LinkedIn is being one of them. They talk about their, their failures and what they do about that.

I think chaos engineering is important. I don't think we do enough of that anymore. Uh, those, those kind of of events are super important to bring resiliency back into.

In, in companies especially, again, you are as strong as your weakest link, right? So when you think about, I told you there are webpages or web applications that take six, 700 things to happen where you think all of them are always working. And by the way, 90% of them, you're not in your control.

I think about Apollo 13. Failure's not an option. Yeah.

Actually, people dying in space is not an option. Failure failures require to solve. And Mitch, and the way I sometimes talk about this is forget for, for forget being able to deliver food or ordering your Uber, right?

That's when you think in the grand scheme of things, like you can survive, right? But imagine a hospital, imagine as you said, an airplane. Imagine things where lives are at risk with like a mistake is very costly though.

Look at the resiliency that is put in a hospital, right? There is not only one oxygen tank, there are 10, they're in a plane. They're not only one, one system to guide the, the wheel or whatever.

There are 10. So, resiliency. Resiliency.

So those guys think about it so much. And I think we need to bring that into companies that are delivering the customer experience, because some customer experiences are, are as important because you fail as a company, you have to fire 10,000 engineers or 10,000 people because you didn't build the resiliency. That's a shame.

Well, I think it's a resiliency is a great point to end on. Thank you Matti. It's great.

Uh, pleasure talking with you. Uh, folks who wanna know more about Catchpoint and some great resources that you all have, where can they go? I think go on our websites, read our blogs, uh, connect with us on LinkedIn.

Uh, we're all, uh, love to share ideas and learn from you. And maybe we can also add a few things to your arsenal. Oh, we have also great resources.

The site reliability, uh, survey we do every year. We've been doing this for six years. Some amazing content and some amazing, uh, feedback that we've gotten from the SRE community.

So check that out. We need to have you back on to talk about That anytime. Pretty important topic.

Alright, thanks again. Thank you so much, Mitch. Thank you.

This is Textron tv. Hey everyone, welcome back to our time here at Open Source Summit in Seattle. Lots of great conversations 'cause we have so many people that are involved in contributing and even built careers around open source and being part of projects and maybe turning that into businesses and that kind of thing.

So actually our next, uh, gentleman who's joining us today is AVI Press. Welcome, Avi from Scarf. Thanks for having me.

What, what, what do you do at Scarf? I'm the founder and CEO Founder and CEO. Okay.

Good thing I didn't say something else, so. No, it's all good. It's good to be talking with you.

So tell us about Scarf before we kind of get into some announcements and things happening. Yeah. Um, so Scarf is, uh, a platform for usage analytics for open source.

So if you are an open source maintainer, or maybe you are a company that's, um, builds open source, um, scarf is a way for you to understand how that software is being used, where in the world is it being used, how many people use it, which companies use it. Um, so it's a pretty comprehensive platform, um, to address those kinds of questions that lots of projects have. Um, and we as of recently are a partner with a Linux Foundation to kind of provide these services across, uh, all projects in the foundation and just help have more data-driven, open source development, and, uh, maybe less burnt out maintainers Too.

Hey, that's always good. Yeah. And I haven't been part of an open source project, but I gotta imagine you see downloads or all the downloads are coming and people that are involved on whatever channels of communications that you've got, but you still really don't have a good handbook for who all is using this and is it, you know, are there things we could be doing to help those people even more than we are?

Is that the kind of data that you're seeking Out? Absolutely. And like yeah.

'cause you know, the people that actually come talk to you and say, Hey, I'm using this and I'm having a problem, or Hey, I'm using this and thank you, is very, very small portion of the people that are actually using the software. Mm-Hmm. And so, um, you know, you have both people that are maintaining projects thinking that it's not providing value for the world.

In fact, it's providing a ton of value to the world. Or maybe you get a spike of a hundred thousand downloads that all came from one machine downloading over and over and over, and like you get a lot of misleading signals or you get signals that are a very small part of the whole picture. Yeah.

Um, and so I think like, yeah, we exist to try to help fill that, that gap and try to give a more, you know, comprehensive view of what's actually happening with the software. Do do any examples come to mind of situations? You know, projects that once they had that data shifted or changed in thinking?

I mean we've, we've worked with, we've worked with, um, businesses that have been sold on the back of the data of like, oh look, a lot of companies use this and then they can go and acquire. Oh, Interesting. Really validate that.

They're absolutely, this is, there's a market for these Companies. Yeah. And like, you know, fundraising rounds that happen because like, well we don't have a paid product yet, but look, you know, 10% of the Fortune 500 has been using this, or, you know, what, whatever it, uh, what whatever those things might be.

Um, and I, I think that one thing that I really, that I find really rewarding also just like outside of the, you know, like business sphere as well, is just being able to show maintainers that like your work matters and it has an impact on the world difference. And like, 'cause I think it's, it can be a very thankless job. The, the only impetus sometimes people actually reach out to is if they have an issue and they need to, they get it fixed and they're not happy about it.

And, um, I think kind of more proactively showing the other side of, out of like, Hey, this stuff's actually very, you Know, my view is I think even leaders on even internal software development projects, grossly underappreciate, how important that is to people. 'cause you're, you kind of feel like you're in this part of the submarine underwater at this depth working on your part of it, and you're kinda go, does this really matter? Am I doing the right thing?

Am I working on the right stuff? Yeah. And just having some kind of validation of that to say, in addition to the person walking up to you at the conference saying, you know, Hey, love your work.

You know, thank you. That, that's super meaningful. But having some data too to validate that, Right?

Yeah. It also helps you be a little bit more proactive as well. I think that that's something that I found very exhausting about being a maintainer is that you just never, you're always like catching up to you.

You are responding after the fact and having to be very reactive. That can be really draining. Absolutely.

Um, as a maintainer. And I think that, you know, if we can prioritize our time more effectively, focus on the right, prioritize better, these kinds of things, it just can help make maintenance a little bit more tractable. How does that data get communicated to people on an open source project?

Do a few people see it? Or is it kind of something that's shared widely or So on Scarfs platform, the way it works is that, um, you know, by, by coming to us with your project and wanting to track it, that data is yours to, to distribute as you see fit. Um, so by default you are the only one with access to that data.

Um, a lot of projects tend to share the, the metrics with their communities and, um, yeah, we see lots of examples of like, you know, drive your community meetings with data about what's been happening. Are we growing, are We a nice way to say, Hey, we you got, we hit this number. Yeah, Exactly.

Or like, Hey, like a lot of people are using an old version and they haven't been upgrading very well. What's going on? How can we help them?

Like, there's a lot of, um, yeah, there's just a lot of stuff happening in the community that you're not really gonna see because most people are not gonna come talk to you. Yeah. Yeah.

So I know you had an announcement recently. Do you wanna talk about that? Yeah.

Um, so yeah, one thing that we just announced is that we've partnered, uh, with a company called Common Room. Um, common Room provides kind of a, a one place to aggregate lots of different, um, go to market signals for businesses. So, you know, tracking who's talking to us in Slack and who's starring us on GitHub and opening issues on GitHub and these kinds of pieces.

So kind of metadata metrics about what's going on in the Community. Yeah, exactly. Um, like a lot of Go-to-market metrics, very community oriented.

Um, and one thing that, um, I think a lot of common Room customers had been missing is, well, what about the people actually using the software? Can we get that? Yeah.

Similarly, scarf does not do a lot of these other community things. And so it's very complimentary compli in these ways. And, um, we're very excited to be partnering with the Common Room folks to be, I think when, when Common Room is paired with scarf, our mutual customers can get a much better picture of the whole user funnel.

Mm-Hmm. Like the time people discover us to the time they become a paying customer, like what happens in between? And we can help provide a fuller picture Of that.

So this gives scarfs users and customers access to the broader common room. So specifically what this means is that, um, scarf customers can seamlessly export their scarf data into common room. I see.

Okay. So, you know, if someone starts to get involved in forums and they're like, is this perfect? Like what version of the software is this person using?

They didn't say, um, we might be able to, you know, actually plug those numbers in and um, kind of give, give a fuller picture of both how they're engaging, but also how they're using the software itself. Mm-Hmm. Or how their team is using the software itself, which maybe that person doesn't even know.

Um, and so yeah, I'm really excited about it because we, you know, we really focus on the one thing that we do very well Mm-Hmm. But, you know, how do we help our customers kind of Partnering really makes sense. Yeah, exactly.

Kind of do, instead of doing a little thing for everybody, try to do a few things really, really well for people. Exactly. That is what we have focused on.

But um, you know, we're just one piece of a broader puzzle. And so kind of building those partnerships with other folks in the open source spaces were, um, very valuable. Certain.

So gathering all that data are, are there some insights that you would say if, if I could advise any new or, or existing open source projects? There are a handful of things that a few did would make a huge difference or make, make a measurable impact. What would those things be?

A few of them. Yeah. Um, oh, so, okay.

Great. Question. One I should say is I'm giving a talk on this exact topic tomorrow.

Okay. So you should definitely come check it Out. Yep.

We check it out. It's being recorded. You can check it out.

Yeah. Um, so a few things. I mean, one I would say is making sure, I mean obviously like making sure that you are tracking these kinds of things so that you can actually measure the effectiveness of the things that you're doing.

Yep. Um, you know, I think that's kind of an obvious one to, is That like user metrics or what, what would be an example of that? Yeah, usage metrics.

So like for instance, like what is the, uh, given that someone landed on a readme, what's the likelihood they go on to download the, the software given that they landed on this page? What's the likelihood or, you know, who's having a hard time upgrading from this release to the next one? Maybe it's like Windows users, maybe it's, you know, some other specific combination of factors that will be very obvious if you're actually just tracking and looking for this kind of thing.

Um, I'm trying to think. I mean, one thing that has been um, you know, coming up a lot is that the numbers that people, their download metrics on registries, if they just see, oh look we got a hundred thousand downloads this week, our data would show you probably actually have about a 10th of that as actually users Mm-Hmm. Um, the rest are bots and things or what?

Or just people downloading over an Cycling Or Yeah. Like yeah, between, between bots and repeat downloads and these kinds of things. Like you probably have much less of that as actual users.

Users. Um, we see that, um, people are knocking upgrade very often, as much as you might like to think. And so you may put out that security patch doesn't mean people are using it.

Right. Um, and I think that, um, yeah, these kinds of trends about like version adoption and how we can better incentivize and reduce friction towards upgrading is something that I think is very important. Both for maintainers, also security practitioners, these kinds of things.

And just people at businesses that rely on all this stuff. And so, um, yeah, there's been kind of a lot of learnings and a lot of surprises of this data. Um, and so that's what's been fun to kind of share what we've learned from analyzing lots of downloads.

So Yeah. Yeah. It's, it's, um, the, the, if you build it, they will come as kind of an exciting, passionate model, but there's also a lot of value in saying, um, well, I'd love to know if they installed it and actually are running it right after they did.

Or maybe it took three downloads before they got it right. Or they tried three other times that this version, well why did that work? You know, give you some insight to what people are doing with your software.

Right. Yeah. And what they're actually doing with it being the really, like, um, yeah.

Tough part to glean. 'cause I think we, the open source community broadly, I think operates through a lot of proxy metrics that are not really the thing that you care about. Mm-Hmm.

And so like, we have a thousand stars. What does that mean as, uh, what do we do with that? Um, you know, I think we're, the thing that we actually care about is like, does this have an impact on the world?

Like, can I build a business around this or like, should we keep working on this? Or whatever that might be. Um, and I think the, the closer we get to what we actually care about, the better off everyone is gonna be.

And everyone was gonna get better software as a result. So Yeah. That has, how soon did the upgrade is an important metric.

'cause it's, we all know we've used software that great new things coming, but man, to get from where you are to what that new version, you know, we all have our own technical debt or just even resources at times. Yeah. To get there can be really daunting.

A lot of people are never gonna upgrade. Mm-Hmm. That's the kind of the reality of that Too.

Most people wouldn't upgrade these things if it wasn't automatic. Right. Exactly.

Exactly. Um, and so like even if you reduce a lot of that friction, just some people they haven the software they have and they're not gonna download it again. And that's kind of all it is.

And we need to just be more accepting of that or more, you know, just acknowledging that is the pattern of behavior and that's kinda the way it is. But, um, So you said you're doing more than one talk, is that right? Yes.

What else are you talking about? Um, yeah, so tomorrow it's about like what lessons learned from like the last billion downloads on scarf. Um, the, I'm giving a talk on Thursday about what, what information, um, package and container registries have that are hiding from maintainers.

Like they see a lot of stuff. They don't expose a lot of stuff. So like we'll dive into like, what Sounds like a 60 minutes episode or something here.

You know, I mean, what the package containers I know that you don't know. I mean, they Know a lot. That's the thing.

Like they see a lot of information. They see, you know, they see a lot of IP addresses, they see a lot of header information, like a lot of rich information there. And you know, our view on this is that if maintainers had more access to that kind of data, we would have more effective maintainers and we'd have better open source for everybody.

Do you think it's 'cause they don't make the effort to share it or they don't want to? 'cause there's value in them knowing that, but others not? I think it's a bit of both.

I mean, I think that there's a lot of registries. They just, um, you know, processing and analyzing this data is a lot of work. Mm-Hmm.

Um, it's, it's, it's a lot of work, a lot of Resources. Right. If there's no business incentive for them to do it, it's not gonna happen.

Right. Um, I think also like attitudes towards analytics, the open source world also have been a historical hindrance for this kind of thing. But I think, you know, the work that Scarf is doing, I think has been shifting attitudes around this about like, hey, uh, analytics can be collected in an actually very responsible manner where GDR compliant.

Like there's a lot that you can do to, to collect this data responsibly, um, you know, get the right opt-ins that you need to do the whole thing in a good way. Um, and to ex and uh, expose that to maintainers. And so I think that it's a shame that more registries are not kind of, their businesses are not set up to incentivize that kind of work to be done.

Um, and so that's why we exist. That is our business model is, uh, that's The niche. That's the need you're pulling.

Yeah, Absolutely. Yeah. And so, you know, we work with a lot of companies that commercialize open source or universities that build lots of open source or these kinds of things where you have a little bit more of a commercial interest in that data.

And so we're happy to, you know, provide that where we can and still like work with lots of, you know, completely non-profit organizations and help them do a better job as well. Well, good luck with your talks. Thank you so much.

Have be available online. Yeah. I think you made a smart career choice.

If you've gone into media, then your name badge would say Abby Press Press. Yeah. That Would, that would be confusing.

Disaster. Yeah. So, and they really wouldn't talk to you then when you go into their booth, so.

Right. Fortunately I just get all the benefits of having press on my badge. They can get into a lot of nice parties, A lot that, a lot of free food.

Pretty smart. Yeah. Good way to do it.

I press CE and Founder with Scarf. Thanks for joining us. Thanks for Having me.

This was fun. You bet. We'll be back soon.

Hi, I'm Mitch Ashley here at Atlassian, team 24 in Las Vegas, Nevada. You know, we just announced an, an acquisition, us being acquired here at Techstrong. And of course, um, that's something that happens in the business world where you're great big companies or even small companies.

Um, I'm, I'm have the pleasure of being joined by Joe Thomas, who's co-founder and now head of Product Loom at Atlassian. Uh, which what, five months ago was that acquisition? Five months.

That's right. Yep. Wow.

Five months or five days. It's kind of hard To tell. Uh, yeah, exactly.

Exactly. It feels like it's been five years in some points, you know. Right.

Well you know what I'd love to you, we wanna know about Loom if, if somebody doesn't know about that. Yeah. I want to hear the origin story about too, about how you kind of came up, why you pursued this as a idea for A company ab Absolutely.

Well, congratulations on the acquisition, Mitch. Same to you. Thank You.

Um, so the origin for Loom was that we believed that video was even back in 2016 when we launched. Video was everywhere in the consumer landscapes, but you would show up to work and it wasn't there. And we quickly realized that it was purely through the amount of effort and friction that there was to record a video and send it out as part of communication.

You would Quick time has been around for 25 years, uh, you would record it, you would've a 300 megabyte file, you'd have to upload that to Dropbox or G Drive. Mm-Hmm. And by the time you shared out that link, it was 15 minutes later, you may as well have just typed up a message.

So we felt like, I don't Want, if I want, I don't want, I, I need to edit it. Right. There's some things I need to tweak or, Exactly.

And so we felt like if we took a lot of friction out of the process of recording and sending video for day-to-Day communication, it would totally unlock the aperture of the people who would use it and the use cases that they would use it for. And so consumerization of video and enterprise was one part of it. Another part that I think is directly related to Atlassian and Team Anywhere is the fact that distributed work, even back in 2016, was on the rise.

Uh, especially in the development world. Right. We've been doing distributed development, you know, for A long time.

Correct. And so people adopt work communication tools 'cause it makes them faster, better at their jobs. And when you think about collaborating and communicating, especially across offices or time zones, you need a more effective form of communication.

And we felt like asynchronous video, uh, tapped into that as well. And, and you bring those two things together. We ran our first survey after we launched the products.

It was horizontal in usage. We are used up and down the org chart. Like Atlassian executives, anu, Mike Scott, all use it for company communication.

Mm-Hmm. But it's also used by engineers that are doing bug reports or code docs. There's designers for design walkthroughs.

Um, go to market folks use it for customer outreach. It's truly horizontal in nature. And that's what we believe the opportunity is, is video is prolific and consumer.

It's gonna be massive in the workplace as well. And we hope Loom is the one that unlocks it for most people. Well, sounds like you're on that path for sure.

Some, sometimes entrepreneurs. I worked at this company, I had this problem, we weren't solving that, but I decided that's my next mission is to solve that. Other, other times it's, we looked at the market, this is happening in video, we think this is gonna be big, let's go.

This is the need that's being unmet. Were one of those two paths, more of the path we Took. We were more iterative.

And I think that that's actually a vast majority of companies that end up being, I mean, you take Slack for example. They started out as a video game company and then became the most prolific. I always say the First two years.

Are it, it's you're finding out what you're really doing. Yes. So we did have a core thesis of video is radically underutilized in the workplace, but the initial implementation was a two-sided marketplace of product experts giving feedback to any stage of the product lifecycle.

Then we got feedback from companies that they wanted this sort of screen recording with camera overlay from realtime website users. Mm-Hmm. 1%, it had to be a pretty slick and easy to use experience.

And so we built it consumer grade in nature and then we started getting feedback. Hey, can I just, can you split out that Chrome extension as a screen recorder and camera bubble so I can just record it myself? Mm-Hmm.

They're like, yeah. And so we launched that in June of 2016 and that's when usage and users exploded. I think that Atlassian one of the reasons why we're so excited about coming into the organization, they were the inventors of product blood growth and DLG mm-Hmm.

And so we had that same sort of playbook where in order to get value out of a loom video that you recorded for when you're done reporting, it instantly uploads you have a link to share is that you send it to somebody, it brings them into the Loom experience, they sign up. And that has been like our core growth engine is PLG. So anyways, that was the initial kind of iterative approach, always video at the core.

Mm-Hmm. But pretty different from where we started. It's amazing.

I did a startup company that thought it was gonna be a web design company, became a systems integration. Yeah. And At the end You just don't know where, dunno where it's gonna take you.

Um, and Loom is so much more than a screen recording, you know, there's a lot of screen recording software talk, talk about asynchronous communication and how that fits into both in the lasting context where you've got this work team work graph of, of what's happening in the organization. You've got different kind of workflows that are happening in J and Confluence and the other tools. So you, you, you've not come from, here's the problem that we solve.

You've come into a less, a, a little bit bigger space. You've come into a kind of real horizontal space where you can play across all of it. Yes.

Am I, am I pontificating too much or am I on track? No, no. Yeah, absolutely.

I mean we, we say that there's the three E of asynchronous video communication and it ties all to generally asynchronous, which is one is efficient. So when it comes to recording a video, it's so high fidelity in nature. You get voice intonation, you get the screen context with Loom, you have a camera bubble where you know, you're showing your facial expression or excitement.

And that is where biologically evolved to talk to one another. And so it's just, And those cues that come with voice and visual. Yeah.

So if a picture is worth a thousand words, you know, a video is worth a million. Um, and so we believe it's a really efficient way to get a high fidelity form of communication to somebody else. And it reduces back and forth, right.

The effective side. So the second e is effective, which time zones and distributed teams is trying to get, even I'm in Atlanta, Georgia and Atlassian has a bunch of Sydney-based folks trying to time zone coordinate and get critical information to one another while again, high fidelity but also solves on the time zone front that you can record a video and send it out. And then the third e that I think is really important, and I think of the X factor for loom is expressive.

That like human connection and communication, especially as the world goes more networked and distributed human connection is really important. And so with asynchronous work, it doesn't have to be, um, just typing away all day. I feel like a robot, it, it can be efficient, it can be effective, and it can be expressive.

And so we believe that asynchronous work is also how you unlock productivity for teams and organizations. So a stat that we have for Loom is that we help companies replace 29% of meetings that are currently on the calendar. Anu talked about it in the keynote today.

Keynote, yeah. And that was, she got some, uh, nobody raised their hands in terms of more meetings. And so for us, we think that you're not sacrificing on knowledge that's exchange or the human connection and you're freeing up time.

And what does that mean? More creativity, more innovation. And so that's what we think asynchronous work ultimately does for individuals is more free time and still doing the same quality work, if not better.

Easy. Just kinda anecdotally, um, we are Loom users at our company. Awesome.

The biggest user of both Trello and Loom is our COO and talk about someone who's at the intersection of people working together. Yes. Right.

And it's just constantly a hello sales team, here's the thing we talked about in our meeting today, and here's how you do it with, you know, pointing to connecting to other content, that kind of thing. It's really great. It's a very effective way.

And in a minute and a half, you know, as long as you don't carry on for 20 minutes in a video, well it may be time for that, but totally indicate those, those little short ideas. Here you go. Here's how it solves it.

The watch it again kind of Connects the average length of a loom is two minutes and 45 seconds. Right. Like we're not talking about 15 minute reportings that it's kind of a pain in the butt to watch.

5 x. Um, That's a little hard to follow, but Yes, it is. It is.

Absolutely. Um, but you know, in terms of the average length of a loom and the type of communication, like you said, COOs recording a minute and a half and that's really affected communication and it builds a connection with the team in a powerful way. So talk about some of the announcements I of you afraid you were now part of the Atlassian family.

It sounds like you didn't wait very long to, uh, come out some other great stuff. Well, this is keep on rolling. Well, this is one of the things that I think is so exciting about Atlassian Loom coming together, is that, that better together narrative was organic and natural.

We had research that told us that we should take a loom video and we provided transcript and closed caption for any loom that is recorded. It's in 52 languages if you want, but how do I take that transcript and then create high fidelity content on the other side of it? And so we actually ended up building an AI feature set that takes your recorded video and transforms it into any sort of content that you want.

So a key use case for Loom is capturing bugs. Uh, you could imagine that you're just talking through it for 45 seconds or so. And then with one click here is the steps to reproduce.

Here is the additional information around network and council logs, and one click it ports into Jira. You can assign the individual the priority, et cetera. Um, another one is on the Confluence side.

So write a document. You could imagine that you're doing employee training or onboarding materials or a help center article and you're recording a loom walking through that with one click. We'll take that transcript and provide a structured stylized doc that you copy and paste into Confluence with a loom embedded into it.

I forgot to mention that with a Jira ticket as well, is that looms can be played right within that lasting ecosystem. And we, we were gonna build that anyways if it wasn't, uh, the acquisition. And so there's just so much more opportunity for us to continue to build together.

We're just getting started. But that, that is the new Loom AI workflow launch that we just had today. So Let me ask you, this may be kind of projecting forward, maybe it's not that far away, but it seems like if you have workflows and you have a platform that understands you, if that workflow is, you could say in the context of that I'm recording a loom now is because I'm working on this particular task, you know, that workflow, it knows how to communicate and organize the content from that video?

For sure. I mean, I think that this is where, what Atlassian announced today with Rove is the thing that I personally, as a product builder am most excited about. It makes so much sense that Atlassian has all of this context on an organization.

The team graph, the, um, knowledge graph. And if you're recording a loom that's walking through a certain project update, well, we can take that transcript, plug it into VO and say, pull out all the tickets or issues or individuals that this is relevant for, and then port that information back into the platforms and systems that it should be in. So like a Confluence doc, they had the blueberry example during robos.

Like if I'm giving an update on Blueberry, I shouldn't have to manually go in and update the Confluence docs with this. We should recognize that on the back end and then suggest all the places that you update it. You just click approve and then all that update is ported back in.

Interesting. Yeah. How do you see video, and that's an understatement of what we're really talking about here, the video with all the context around that.

Did you thought a year or two ahead from where, how we work today? What might change? What, what do you see happening?

Yeah. I think the biggest thing for people adopting video at work is it's not as natural as you might think of recording a video. Some people are like, readings and text work for me.

But it's asy video is so powerful and what we need to do is make it as easy to working with a tech stock as we possibly can. So we do have an edit by transcript experience right now, but it just allows you to trim. What we're gonna do is start pushing into multimodality around ai.

So we have a lot of rich, uh, information on individuals. So we can create a voice avatar, we can create a camera avatar for you such that if you stumble over some words or you miss a important section, you can just type that in and we'll put that in the video for you. And so for us, we think that that's really powerful technology to push more into the multimodality versus just kind of like the text oriented AI offerings that we have today.

Another part that we're kind of taking that voice and camera avatar technology and applying it to is that, as I mentioned, loom is used for go to market personas, sales and success folks. And it actually increases pipeline by 19% as what we've seen just because it makes that human connection if you receive a personalized video. Right?

And so we're gonna use that technology of Cam and voice avatar tie into your CRMs, and we're gonna launch something called Variables that allows you to send personalized videos at scale, which is the number one pain point that we hear from salespeople today. Mm-Hmm. Um, it's just rerecording basically the same 95% is the same, just a Different name at the front or Whatever.

Yeah, exactly. Exactly. And so I think that that's where our AI offering, but the Loom platform generally is gonna push more into the multimodality of video.

I hadn't really thought about the role of avatars in that. Um, overcoming the apprehension of being on camera. Not everybody, you know, sits here in front of a camera every day and, you know, natural for you or I to do a Zoom meeting or a recording or whatever it is.

Right. And, uh, for other folks, that's just like anything in front of a camera is a, a big step, the apprehension. So it sounds like avatars might actually be a great substitute if you're not the kind of person who could always be, and sometimes there's just, you know, Hey, I got outta the shower, I'm not ready to sit in front of you.

Yeah, yeah, yeah, yeah. Of course. Yeah.

I think that that's where, um, it's a very exciting time to be building in software generally. It's very exciting to be building N Video specifically. And I think when you say two years from now, I think that the ecosystem is evolving so rapidly it's Fast as things are changing.

Yeah. Wow. So it's like six months, like a year is kind of like max where we push our thinking towards.

'cause things are gonna change and we will adapt to that. I was just thinking about your sales example. CRM I'm a product leader, right?

I want to talk to my customers. Hey, we got some announcements here. You're probably not gonna be one of the people that goes and downloads or watches the announcement from the Alaska.

I just wanna tell you a little bit about some things we did this week and what, how this might be relevant to you. And I'd love to follow up and chat about Exactly. It's extremely high leverage, right?

I think that this is what Atlassian, I think has done this is where Loom is gonna learn a lot through the acquisition, is they've done a really good job of kind of packaging up the marketing messaging around agents, right? And effectively what Voice and Cam avatars is, is like allowing you to have your own personal niche agent right. To do work on your behalf.

Mm-Hmm. And I think that the more leverage and more productivity that we can unlock for individuals, again, that goes back to enabling you to have more creative and innovative time, uh, instead of recording those personalized, uh, videos for every single individual. Let me ask you about the AI equation.

I've, I've talked to a couple people about, there's also an apprehension around ai, right? Uh, first I wanna do anything wrong there. There's a, there's some upsides to it too.

Is I could talk to an avatar or to a bot or whatever it is, and not feel as judged as I might if I'm like, gonna go open a ticket and talk to somebody in it, or, you know, go call up finance and say, what does this mean? But how, how, how about AI in the context of loom and multimodal, um, how do we make that as accessible as possible so we can address whether it's fear or apprehension about ai? Yeah.

I think that it fundamentally boils down to a couple things. One is that people adopt work communication tools, as I mentioned, because it makes them faster, better at their jobs, right? Like the first and foremost, we have to tell them the value that it delivers on their behalf.

And the follow up is, well, from a security and compliance perspective, what are the sorts of things that we're safeguarding it on the other side, where in the Loom instance for Cam and voice avatars, we're literally only allowing you to create one associated with your account. So you can only have Mitch, right? Okay.

Um, and it has to be content that you recorded with Loom. So you can't upload a video we won't generate based off of that. And so from our perspective, if we deliver on the core value proposition of what we're saying is unlocking more productivity for you and more value and more ROI, then on the other side we'll have all the safeguards in place to make sure that you're like, well, only Mitch can create a Mitch.

Right. Only a joke. Just Knowing that is important.

Actually, I hadn't really thought about it from that perspective, but I don't have to worry about someone uploaded some video 'cause they got a hold their beer Account. Right. Right.

Interesting. Uh, any announcement things you wanna talk about that we haven't covered yet? Well, that's the fascinating thing about being at Atlassian right now is that, you know, uh, a new, that's a loaded Question.

Yeah. A new, and Mike and Scott, um, they all kind of took what we're announcing and we're still a relatively ragtag team of 200 people that, uh, they announced pretty much everything that we're building right now. Okay.

Um, I will say that one of the things that we don't have coming out in the near term, call it like the next three to six months, but really excited to keep working on is that editor work stream. Like edit it like a text document. But we're also trying to, for that executive communication, we'll build custom branding so you can be on brand for your organization.

So if anew is sending out a loom to the entire company, is there nice little overlays Right. And things that if she wanted to do a q and a as part of it, you know, and like have people fill out a poll. We're we're building against the editor work stream for more broadcaster use cases.

Um, it'll still be recording a loom that ease of use. You'll have all the AI features on the other side, but how do we make the editing a video as consumer grade and seamless as possible so people can start to feel like, oh, I I don't have to be Spielberg to create a relatively polished video at work, 20 Shoots to get it, you Know? Exactly.

Recordings to get it. Right. Exactly.

I Think about, I mean, earlier I called Robo Rebo, you know, by the, you know, into it re like, okay, what's wrong term? You know, I'd love to like reverse tape and, you know, take that out. No tape involved.

Yes. You know, it is that kind of thing. And you have to just pick a random, if you had to break out Camtasia to go edit a video to fix something and I don't even know how to fix the voice, how do I do an overlay of that?

Right. Yep. You know, those are all the things that if you're, if you're that that's specialized in that area, no problem.

But for everyday person, that's not something we know how to do. Exactly. I mean, our, our goal for Loom, we believe it's inevitable that everybody will use Async video as part of their communication stack.

You know, our goal of Loom and Atlassian is press that into the present as quickly as possible and then obviously have as much loom be the provider because we are the market creator. We believe we're best in class as of right now. There's some statistics that back that up like G two, et cetera.

Mm. Um, but, you know, our, our goal is to make that as inevitable as possible. And I think that making people comfortable with video and having the flexibilities of recording and tools around it is what ultimately gets us to that inevitable future as quickly as possible.

Sometimes when things are so impactful, they kind of fade into the environment we work in, in, we don't call it Loom anymore, we just call it ultimo video, whatever the term is at the time. You see that kind of happening for, for the technologies that you've created. Is, is that a measure of effectiveness, is where we don't have to call it a loom video anymore, it's just a, a thing or whatever it Has.

Well, we actually think that there's real branding power in people calling it. I'm sure do. That was a trick.

That was kind of a trick place, I guess, or maybe a Dumb point. I was just walking over with a teammate over here and she was telling me that somebody, uh, as a mortgage lender, their friend in Oregon was received a loom that she used that specific term. Mm-Hmm.

Okay. So, um, you become the Kleenex of Yeah. A noun and verb.

I think that that's like a really special, Or hey, not a bad Thing. I think it's a really special opportunity from a brand perspective is that, you know, just send me a loom. Right.

But, Alright. So we won't, not that we did, but we wouldn't expect loom to fade into the background. No, no, no.

Exactly. Exactly. Did you loom that?

Did you look, did you set up a loom? Okay. I can hear the conversation now.

Alright, Joe, it's been a pleasure talking to you, Mitch. Thank you, you so much. Congratulations on the acquisition And you too.

Thank you very much. And, uh, we look forward to great things and I think it's a, a good place to watch, you know, for, uh, people who are following or part of Atlassian because you're, you're, you're part of creating the, the, the future that we're living right now. And we'll live in a year or two from now.

So it's great Stuff. That is the plan and we'll see you at team next year. Right.

Okay. We'll be here. Okay.

Awesome. Alright, Joe, Joe, um, Thomas, I keep wanting say Turner Joe Thomas, in my mind. Anyway.

Joe Thomas, who is, uh, head of product for Loom. Is it SVP? Is that your title or You Head of product?

Technically. Head of product. Okay.

Yeah. You know, those things evolve over time. Exactly.

Head up is a generic, what what level is it? We don't know. You know, It'll, it'll, that all kind of figure itself, right?

Yeah. Okay. So anyway, Joe Thomas that we've been meeting with, really think about that asynchronous, asynchronous communication.

Don't think of it as that, as that term is thinking about how do I share what I'm thinking right now and people can consume it at any time that they want to, and now in multimodal. And, uh, how that can be more effective than recording something. And, uh, assuming that that's the static thing, it's only ever gonna be, it's a great way of helping us all communicate better.

So thanks again, Joe. Thanks Mitch. Alright, We'll be back again with another great interview.

Stay tuned. Cloud native now is the web's leading resource for the growing cloud native ecosystem. com is your destination for news, thought leadership, features and webinars on cloud native architecture, Kubernetes serverless, cloud native application development, microservices, service mesh, cloud native security, and more.

Stay on the cutting edge of modern application development at Cloud Native. Now. Hey everyone.

I'm Alan Cheval, CEO of Next Drug Group, and you're watching CD Pipeline. For those of you who are not familiar, CD Pipeline is a once a month, uh, video series that we do in partnerships with our good friends at the CBF, that's the Continuous Delivery Foundation, VBF, which is also part of the Linux Foundation, lf. And, um, couldn't do it without them.

We, we do it in, as I say, in partnership. And what it is, is once a month we look at relevant topics within the world of CICD. And you know, for those of you who are familiar with CICD, there's really no end relevant topics.

It seems as there's always so much going on in the space. Um, also for those of you not familiar with CIC, uh, the CDF rather, I would encourage you to go to the website and, uh, it'll be in, in here. It's the, uh, continuous Delivery Foundation.

Laurie, just help me. org. It's CD Foundation.

CD Foundation. I apologize. DD Foundation, you know, uh, in addition to doing this show, the CD Foundation does a lot of things, but primarily is they are responsible for managing and administrating.

Uh, I think it's eight of the most popular tools within the CICD world. And, um, it's a great foundation. They're always looking for people to help and get involved.

And we'll, we'll talk more about that later, but let's go to today's show. Today's show or this month's show that you're watching today is, uh, around the release of the annual report from the CDF. And it's navigating the CICD frontier.

It's actually the abstracts, uh, is our name, insights from the 2024 State of continuous Integration and Continuous Delivery report. That's the ICD. So again, this is an annual report.

There's some really great research and, and findings here we're gonna go over. But before we do, I want to introduce you to a fantastic panel that we've got laid out for you. Um, I'm gonna start with Mark Waite, who is an expert on this report, having already spoken about it a few times, and we're gonna rely on Mark a lot at this.

Mark, if you wouldn't mind, introduce yourself to the audience. Sure. I'm the treasurer of the Continuous Delivery Foundation.

I'm a member of the Continuous Delivery Foundation Board. I'm a member of the Jenkins Governing Board, a Jenkins core maintainer, and I maintain the Jenkins Get plugin. So I care deeply about CDF and its success, uh, especially its Financial Success Treasurer and I care deeply about the success of the Jenkins Project.

Those two things work really well together, right? Because CDF is the, the owning umbrella project that sponsors the Jenkins project. We depend on it.

We, we are deeply grateful for it. That covers what I had, Alan. Thank you, mark.

Next up, let me introduce you to Steve Fenton. Steve, if you wouldn't mind kinda introducing yourself to the audience. A little background.

Thanks Alan. Um, yeah, my name's Steve. Uh, I work at Octopus Deploy, um, and we sponsor the CD Foundation 'cause we think their work super important too.

And, uh, I'm on the CDF Outreach committee. I was on the CD Con program committee this year, and I wrote the Journey of DevOps tooling adoption post for the CDF blog, uh, when the report came out. Uh, I'm super excited about, uh, software delivery culture and methods and research.

Like this report is taking us into a new era where we don't have to just, uh, listen to the people that wrote the books anymore. We've got research that helps back up the ideas. So I'm super excited about it.

I love it. I love it. Next up, I'd like to introduce you to Melissa McKay.

He's been on here with us before. Melissa, welcome and give him a little bit about you. Yeah, thanks for having me here.

So, um, I'm Melissa McKay. I am currently a, um, head of developer relations at J Rog. Um, that's a new career choice for me, uh, relatively new, but, um, prior to that I've been a developer for most of my career and, um, primarily Java server side applications.

And I distinctly remember moving my first time moving to a DevOps team. Um, the very first tool that I was exposed to, aside from source control was Jenkins. So I have some good memories and some terrifying ones that I share often in lot of my talks as well.

Uh, you know, just, uh, getting thrown into the deep end and, uh, having to learn, uh, these tools from scratch. So yeah, really interested in this information, really interested and getting developers more efficient, getting teams more efficient, and, uh, the CD Foundation is integral to that. Um, I am also part of the, uh, a co-chair with didi, actually, um, the, uh, interoperability special working group at the CD Foundation.

And I wrote, uh, co-wrote a book called DevOps Tools for Java developers. Very cool. Melissa, thanks for joining us today.

Appreciate it. We, I know you're on the road and, you know, dialing in from the road is much appreciated. Next up, let me introduce you to a gentleman who claims he's looked at the report a few times, but he's really not an expert in it, low expectations, but he's also the chairman of the, uh, CD foundation.

Is the chairman, is that the right word or the right title? The sports chair. Um, sports chair.

Okay. Yeah. I guess chairman's kind of a, an old word board chair.

Um, the DC Ika thei. Just kidding. I know you know this report very well.

You and Mark go over it together. Um, but welcome, welcome back, and if you wouldn't mind giving people a little bit of your background. Sure.

Um, I'm d Ika and I am board chair of the Continuous Delivery Foundation. I also serve on the CDF Technical Oversight Committee, as well as co-chair, the Interoperability special Interest group with Melissa. Um, I am also on the TOC of spinnaker, and spinnaker is one of the projects in the Continuous Delivery Foundation.

Um, my team has been working well. I've been working with the spinnaker team for about four years now, and, uh, making contributions to it. It's a, a amazing tool and I wanted to get involved with the CDF because of it.

Um, I think the, the mission of the CDF right now, as, as is not just understanding how to help people deliver software, what's, uh, speed and security. It's also thinking through things like what's next? What comes next for tooling?

What are we doing? How are we, how are we trying to expand the space? And it's stat work that really excites me, and we are definitely fed from the information that comes from this report.

Excellent. Very cool. Did DC welcome.

Thank you. Last but not least, she's my co-host here at CD Pipeline, and always great to see her. It looks like she's open today for a change.

Uh, Lori LaRusso. Hi, Lori. Hey, Alan.

Yes, I am home. And if I make a strange face throughout this conversation, I promise it's not any of the panelists or what they're saying, it's the fact that I have two dogs inside and it's pouring outside. So I've tried to dope them up with lots of food and treats, but, you know, labs, so they have bottomless pits for stomachs.

Um, I will say that this is such an awesome panel. I'm really excited to be here. There's some recruiting work that I did that is paid off.

Uh, Steve was on the outreach committee, and I remember when he first joined, he said, I'm just gonna sit back and I'm not gonna, I'm not really gonna like, participate. I don't know what I should do. And now he's writing blogs and he is showing up on the CD pipeline, giving his opinion about the CICD report, which is awesome.

And DDC can forever, um, be in my debt for bringing him into such a cool group of people, uh, recruiting him into some more activity with the CDF. Um, so yeah, so this is an awesome group. I think it's a great way to, to talk about the report.

Lots of good opinions in this room or rooms. Absolutely. Hey, before we begin, I wanna put it right out there to everyone watching this.

This report is available to be downloaded, and you can get it if you go over to CD foundation slash state dash of dash CI CD dash 20 24 2 0 24, and you can download the report there. So this isn't the first state of CICD report. I remember doing the show on this last year.

Who can be our group historian, mark, um, about the history of the state of CICD report and a little background. I mean, obviously it's the, we're the CD foundation. It makes sense to do a report like this, I guess, but, but why?

Yeah, so, well, so I, I liked Steve's Steve's way of describing it. That we've had a culture, we've had a, an idea to recreate, to rebuild, to do a better job of crafting software development. And that's, ideas are great, but it's impressive to bring data to ask ourselves the question.

Is the idea working? First question, how well is the idea being adopted? What are the things that we might consider as ways to increase the adoption?

The, the, the concepts behind CI and CD are not brand new concepts, right? These are, these are well-established concepts known for years, but data to tell us how are we actually doing in industry with these concepts? And, and that's the purpose of the report.

And that's, that's, that makes it extra valuable that we get that report across multiple years. We can see how things are evolving. Excellent, excellent.

Um, so guys, if no one has any other background on the report that they want to volunteer, let's jump in. Well, actually Mark was last year, the first report, is this the second one now? Or is there history?

So, and, and I'm not, I'm not sure on how many there have been, but there have been multiple reports as far as I know, we've run this report multiple times. So, so this is not a brand new thing. This is, this is a well established pattern of running these reports.

And, and we're grateful for that. It's, it helps us see patterns and observe sequences of events over time in an industry that is, well, an industry that went through a pandemic, right? An industry that went through all sorts of complicating changes, an industry that's going through changes now.

Absolutely. Melissa, you were shaking your head. Do you have an idea of how many years we've been doing this report?

Oh, several years. I mean, the re the report itself, at least the latest one that came out, it, it describes, you know, all the different quarters, um, that they've been doing it. Um, okay.

See if I can find that really quick. I'm pretty sure we could start in 2021. It's, yeah.

2021. Okay. So this is great.

So Yep. QQ 3, 20 20 to Q1, 2024. So several quarters, um, 150,000 respondents.

That's not a small number though. I, across, I was looking for right to get that cumulative amount of responses. Now you have a real sort of a book there of, of data that you can dial, dial in into.

So let, let's dial I'm sorry, go ahead. One More comment, sorry on that, that I think is really important to share. Um, there is some consistency, you know, in questions, but as we've reviewed this report every quarter, we always get into these conversations of just how complicated some of these processes can be and how some of the data that's coming out there could be several different reasons for why it's presenting the way it's presenting.

So it's been interesting to have this conversations with some of this group actually, um, as this report comes out every time, um, I know I, I give talks sometimes about the whole process of CICD, um, being somewhat like a Rube Goldberg machine sometimes within teams. Um, you've got all these pieces and parts to go together and there's a lot of, a lot of different reasons that we might see the data displayed as it is Agreed Worthy of conversation for sure. Absolutely.

So 2024 post pandemic stuff's happening, right? It is, uh, I, I've been on a circuit pretty much since CubeCon Paris. I think I've been on the road since then.

I'm happy to be home now. Um, but you know, who wants to kick us off? What, what do you think are some of the big, big findings in this report?

Whether they're surprising or not, right? That, that we need people to take home with. If you don't mind, you're the, the board chair.

Why don't we ask you to kind of kick us off here? Sure. I think, um, well from the perspective of my CDF perspective, um, one of the things that I found most interesting was the report highlighted, um, the impact or the knowledge of, uh, CICD tools that new developers have.

Um, and it immediately showed this gap of like, I think it was, uh, 83% of them were aware of the tools. And so there's a big piece that, uh, just aren't aware that they're a part of CICD or what that even is, or you know, how they're engaged with it. And that creates some opportunity for us in the continuous delivery foundation to educate and work better with education, uh, universities.

Um, and it makes some impact there. And it's, it's trying to find information, uh, at least from my perspective with Boris Chair, where the CDF can make immediate impact, uh, and kind of, uh, send the community in the proper corrections. Sure.

It's, it's interesting, 83% are aware of the CICD tools, but what percent did you folks are actually using them? Well, like the, yeah, that's the, I guess I misspoke. Those are, those are the users of who are aware that they're using ci cd tools is a better way to say that.

Uh, got it. And it is the, the, the individuals who aren't, that become the focus of what I think we need to do in extended education, that more students, it should be a hundred percent of 'em, right? Everyone should be aware of Jacobs, right?

Everyone should be aware of spinnaker and what those tools mean and what they do. Um, not just because of the history, because this is, this is how you learn the process so you can get the deeper understanding and starting with things that give you this hands-on experience. And so what I've been looking at and really focused on since the report is engaging with universities and saying, okay, how do we fix this problem?

Um, and that's one of the things that we're, we're looking at too via the, the interoperability sig. Um, and I think being able to see that information and then immediately say, okay, I can take some actions and impact that, that's what we want to do with this. You know, you don't just wanna take the information and go, okay, yeah, that's great, we got stuff, but what, what can be actionable and how can we impact the community?

Um, should be the intent with a report like this from a, our, One of the things I find, um, interesting as a finding, and I, I think everyone can kind of understand why, and I think we should maybe expand on this. And, um, it says using multiple CICD tools of the same form leads to worse deployment performance, likely as a result of challenges related to interoperability. So the more tools you use, the harder it is to track.

Like, what are your, Melissa, I'm gonna throw it to you 'cause you're, um, on the interoperability sig, like how do you think we can try and change this metric or turn it around to sort of streamlined processes and tooling? This is a tough one 'cause I, I think I'm just gonna provide some of my own experience. I, um, and this goes back and forth depending on what kind of organization you are in, but I, I always see this, this back and forth between like platform teams that decide for every team within an organization what tools they're going to use and try to make that easy for them to adopt.

And then I see everyone get upset with that for one reason or another. And then there's a move to every team being individually managed by a team lead. And they might choose completely separate tooling from each other within the company, depending on their skill sets, depending on what they're comfortable with and what they're used to.

Um, and budgets as well. It makes a difference. But I do have personal experience of having trouble with that.

I remember even just being on a team and we used three different sets of repositories for our bills. So we had to, you know, reach out to one to get other internal team stuff. We had a separate one for like images.

We had a separate one that was open source that other people just preferred. And it was kind of ridiculous just trying to figure out where all of our stuff that our project relied on, uh, where it was stored and, and how to make sure that everything got into our build appropriately and everyone had the right permissions to be able to access everything. So I can totally see why this metric happens.

And I'm just talking about code repositories. I mean, we've got across the board different tools for different parts of the process all over the place, different build tools even so if you're moving one person from one, um, set of tools into another team and it's a completely different set of tools, I can see that being a struggle as as well. So, I dunno if I answered your question or, or just relayed like, yes, this is a problem.

Right? You were nodding your head. Oh, mark, go ahead.

No, no. Steve, I, I think let's give Steve a voice. My voice has been heard plenty.

I think Steve got things to say there. Uh, yeah, I mean, what Melissa was saying about, um, when teams have all made different, um, tool choices, like it's hard enough remembering all of those little command line things you need to call for source control, um, where to log in to see your build status, um, where to log in to see where everything's deployed. If every team's got different choices and you are contributing to multiple projects, which is the situation I'm in, you're just constantly looking up things on a wiki to find out where stuff is.

And so the amount of your productive day is just shrinking and it's just, it's admin instead of programming. So yeah, I, I agree that's, that's definitely an experience I've had too. Um, I've even been on a case where I had to like get off a VPN, get on A VPN depending on what team I'm on, right?

And, and be able to do that multiple times a day. So yeah, it, it's a challenge. I I think one of the things, one of the solutions here is awareness.

So, you know, having organizations at higher levels be aware of this issue and maybe make better decisions, um, in that area would be good. Yeah, I, I, I, like Melissa described the, what I'd call the culture compromise, uh, at the larger the organization, the more likely it has to strike some of these cultural compromises between autonomy for a team and organizational selection to optimize for the organization as a whole. And, and there are different levels of that required for different organizations, right?

Organizations size is a big impact on that. Small organizations may, may grant more autonomy and do less damage with that extra autonomy. Large organizations, large organizations, I would expect maximum autonomy would be also maximum damage in terms of their portability, right?

They, they, they have to culturally choose to centralize more and, and deal with rebellion if they have to. So Steve, I think that was sort of what you were describing as well, is that there are times when organizations at large scale have to have to make choices for the efficiency of the organization as as a whole, even if sometimes it's sacrifices. My personal productivity, What I like about this conversation is that it leads me into another finding, um, that you all have just been talking about, and it's that source control management and issue tracking hold the top spots for the most widely used DevOps technologies.

And so you're saying that, and then you're saying, but when you add too many, it creates chaos, Right? I think it's, it's more about, I think those are like the basic minimum, like source control for sure. Um, issue tracking is so helpful, but having multiple of the same type is the biggest issue.

Um, I think that's where the discourse comes and the friction comes Well, and, and that, that friction is not just an internal organizational friction. Let's, let's take source control. Alright?

I was in an organization in a, a company that had a mix of subversion and we were adopting this brand new technology called Git. And we were really thrilled with this brand new technology called Git, except that it meant we were managing two systems and the complexities of managing those two systems. And guess what, when you add yet another system, it gets even worse.

And it's worsening not by, it's worsening by exponential growth because of connection volume, not by linear growth. So choose carefully and some things you really have to say, I'm sorry, I know you love material, but our company has chosen Git and we've chosen this provider for it. Or I know you love CVS, you're a, an open BSD developer and CVS is the best thing ever, but I'm sorry, we're using Git.

So that kind of choice is a valid thing. You know, I, if you don't mind, let me be a little contrarian here today. We are, um, here, over here at Textron, we've been doing a big research project there for a few months on called DevOps.

next CloudBees has involved jfr, some others. com and maybe 12, 13 years DevOps burst on the scene. Um, and, and, and DevOps in all its aspects, right?

But for many people, CICD is DevOps, even though, as Lori said, it's the, was it the fourth or fifth thing on that DevOps thing? On the DevOps, you know, uh, uh, the fourth or fifth most common feature of, of the DevOps get control or, you know, version control being number one. It it, I heard it said, you know, DevOps is the, is the sickness, CICD is the cure, right?

Or something like that. But here's an interesting thing we found. We live in a DevOps bubble.

I do, I'm gonna venture that many of you guys do too. And that when you go out and ask, you know, random IT or, or organizations, IT departments and organizations that 83% using, or 81% people are using CICD tools. It's kind of, you know, when you're a hammer, everything looks like a nail.

If you're working in a DevOps enabled organization that does CICT, yeah, you're probably using A-C-I-C-D tool. But the fact of the matter is that though DevOps has clearly crossed the chasm into mainstream at an enterprise wide level, it, it's not anywhere near 80% enterprise wide that are using DevOps enterprise wide. There are groups using within the enterprise, and it's the same thing for these CICD tools, right?

They may use Jenkins over here, they may use spinnaker over there. They, you know, it's not, it's not uniform and nor is it totally penetrated in there. And I think that there's another finding in the report that kind of dovetails with this, which is the least experienced the developer is the less likely they're using A-C-I-C-D tool.

The opposite of that is the more experienced a developer is, the more likely he's using A-C-I-C-D tool. So the DC you talked about opportunity. There's the opportunity, why aren't our less experienced developers?

You would think they're coming in, they're gonna start with state of the art, they're starting with the new stuff. There's no sense we're learning cobalt. There's no sense learning more to fall.

We should be going right into DevOps and CICD, the DC's laughing, right? Why aren't these new folks hopping right into this? Why, why is it, that seems to me a little counterintuitive thoughts.

Um, well, I I have been, like I said, I've been having conversations with, um, several universities and just asking that question, are we preparing these students while they're still in college to exposing them to CICD tools and how is that being done so that they come into the marketplace better prepared and aware of what those options are and how they work. I have been, um, somewhat surprised with the conversations that I've had, uh, and the opportunities and how to create that experience for, um, young people that are in universities and colleges. But it's, it's not just the impact there.

It's also, as you said, once they're already in the workforce, how, how do we change that culture and give them knowledge? The the key to the statement, of course, is experience over time. They will learn it, but we have to better educate them as they prepare for this journey, um, so that they have that awareness, right?

They should have all, they should have the opportunity to have some hands on time putting the pieces together, understanding the history and the journey of how we got there so that they can contribute and make better tools. And that's the, the benefit of education at decade time to have those conversations and why that's important to do. And so I thi I think there's a, there's an element of what we might even call evangelism or of telling the story that we sometimes forget because we've been, many of us have been in the story for so long, telling it for so long that we forget.

There are plenty of people who need to hear. It's a good thing to compile your code every time it changes. It's a good thing to run all its tests.

It's a good thing to deploy it to production as fast as you can and as often as you can. And those things, those things, while they're not new to me, they are new to new people and they're new to certain classes of organizations, right? There are organizations which are sort of underserved in this space as well, either because they're highly regulated or because they're deeply steeped in another, another development culture.

And so there is an evangelism element, Alan, that's still there, even as large and as long as we've been doing this, You know, that makes me think of telco companies when you say like, they're like, uh, I was just having a conversation the other day about how it's not even like telco companies don't even move at a snails pace. Like that's like the rabbit in the hair. Like they're so much slower than that.

And how can we help change the, the mentality of those types of organizations that have so much compliance, so many different departments, so many things, and let them know if you could release the reins a little bit, how much more productive, efficient, happier your developers would be. You know, like, you know how to bring them in to the fold. So it's not just about, you know, students and, and young, it's, it's, to your point mark, it's like organizations, you know, complete verticals that are stuck because of this, you know, roadblock of being unwilling to let things go a little bit to release fast, to fail hard and then go back.

You know, that whole idea Well, and, and, and yet there are, there are reasons why many of those segments have difficulty doing that. Life critical software is in fact life critical and life critical software has to go through a different process than the little things I do that I release very often, right? Life critical software is not the same as, you know, the, the aircraft manufacturers with their 50 and 75 year life cycles have a very different thing than those of us who get to live in the internet age and everything.

We can release a new version and we can do that multiple times a day. So, so there is still an active promotional effort that's needed in many, many places. I also think sure, um, the big difference between when you are pre-professional as a developer, whether you are a university or if you're learning to code for yourself, you don't have to collaborate with anyone, which means it's really easy because you can just change the code, right?

And it's done. As soon as you add in collaboration, that's where CICD and all of the associate DevOps tools suddenly become super powerful because it, like, I, I guess that's what you learn in your first year, right? Is when I'm merging code and it's like the first time I've done that in three months, it's really hard work.

So that's why we want to do it more often so that it's a small merge. So I think that that difference between coding for an individual purpose or to get a quantification is so different to coding with a bunch of other humans. Yeah.

But that's for sure. I, you know, and I don't know, have the right questions to ask to prove that out, but that would be a really interesting study as well, right? The dynamics of developing in a group versus, you know, the l wolf approach, if you will.

Um, so go ahead, mark. Sorry, one, one more angle on this. The, the, the, the report not talks about the adoption at over 80%, but then when you dive a little deeper into the report, it warns us that there are cases where two thirds of the people aren't using continuous integration.

And yet Mark wait, thinks that's astonishing. What we've been doing this for years, it's a decade, it's two decades that some of these tools have existed. And yet there are people who are not building their code every time they change it.

And, and so there really is, there is, while the report tells us large, large part of the world has figured it out, we need to be doing this. But there are still plenty of pockets of places where people need to hear, you know what, it's a good thing to do, ci, it's a good thing to do. cd I I wanna pound on that point a little bit more because as we are more experienced and we're used to it, we take a lot of these things for granted and we just do it second nature.

And I'd like some of the stats in the report that says that, you know, DevOps activities are, are pretty similar across, um, different types and sizes of organizations, which is really nice to hear. Um, I also see that difference between maybe a lone wolf and, and teams. Uh, when we look at like freelancers and specifically there's a, there's a graph in the report about that.

Uh, freelancers have a little bit of a lower percentage of getting involved in DevOps activities that could probably be explained by maybe being alone. Um, but when we are talking to students and we're talking to new developers, we've got to say why we can't list a list of things that a bunch of stuff they need to do or else, uh, we need to explain why, why are these practices best practices? Um, I know a lot of engineers and I was definitely one of 'em.

I come in all rebellious, think I'm gonna change the world. Think of, you know, we go through stage, think we're smarter than everyone else and then, and then, uh, find out years later. We clearly are not.

But um, we need to be able to explain to these folks why, 'cause they're gonna ask adamantly. Agreed, agreed. Um, you know, a a an an interesting thing is correlation, like, let me back up.

Why do you do CICD? 'cause I could deliver software better, faster, right? Isn't that the reason?

Okay. How do we know you're delivering software better and faster? Well, one, one measure is the DORA metrics that we've developed here at DevOps.

And, and unfortunately the EU started their own DORA thing, which has nothing to do with DORA metrics, but more on security. But it's confusing as heck, I have to tell you the truth as I go through news stories. But anyway, DORA metrics, right?

Google now kind of, uh, administers the, that report, uh, you know, the original work of Jean Kim and Dr. Nicole Fors, grid and jazz humble. And we use that to measure high performing IT teams, high performing IT organization.

And clearly it says in the report, right, if you are utilizing DICD while you are more than likely to be a high performing IT organization, is that enough? Is that enough? Steve, I feel like we haven't heard from you.

Is that enough to, to sell it, if you will, to say, Hey, this is God doted, this is why we should do it. So, um, the interesting thing about the, the Dora metrics or the four keys, maybe we'll call them to avoid the eu um, legislation. Um, the interesting thing about them is that you can link them forward 'cause they predict organizational outcomes, which is super interesting 'cause it means it's something very technical that we can do, um, as programmers that will actually change the numbers for the whole organization at the end of each year, which is cool.

And they're also lagging indicators of, um, what we're doing in the software team. So they're useful to measure what we've done in software and what we're likely to achieve as a company. So I think they're really powerful.

They're probably not, you wouldn't just use them because you know, if you're working in um, I don't know, an e-commerce website, you are gonna wanna know some specific things as well. Like what's your abandoned basket rate? What's the value per transaction?

There are things like that that will help you understand if you are feed to changes, uh, making a difference to the specific business that you're working in. But in terms of a bunch of things that we could do in any company, in any industry, that will make an impact on like our goals, and that could be financial goals or it could be, um, saving bees or reducing deforestation, whatever our organizational goals are. Um, like that's quite powerful for me because in the past it's been quite difficult working in isolation and not feeling like you're contributing to the success of your company.

'cause it's too far away, it's too abstract. So I really like them. I think they're very powerful 'cause of that.

Yeah. Anyone else? No, I, I, I wholeheartedly feel like the DORA metrics are, are a, a great, a great way to look at things that surely are necessary.

To Steve's point, they're probably not sufficient. They're not the only thing you should be doing, but they certainly are necessary. And if you're assessing them, you're at least stepping towards that improvement that you need to make.

And it's a, it's shown for many companies to be a very good improvement. As you improve those measures, your organizational performance will improve as well. Yeah, I'll just add that it's, uh, the, that it's the starting place for the conversation.

And if it gives you the ability to evangelize internally, um, especially as, as your company grows to say, Hey, you can see over time that something's happening. And it's a great way to start that conversation to create that evangelist movement that says, Hey, CICD is important. Which was, you know, to come full circle with something that Melissa was saying earlier.

And so it, it, it's all tied in together and that information starts in the report in a way that you can engage with it and kind of set the tone for where you are in that process and, and continue that conversation. One last thing I wanna add. I, I like the metrics.

I really do. I think they're really valuable for like internal measurement, internal progress measuring that something that I'm a little concerned about maybe we can work on in the future, figure out how to solve this problem, is how to define complexity. 'cause by the, when you start comparing yourself to other companies, other projects, it's almost, it's comparing apples and oranges in some cases.

And if we can get a better metric that describes complexity, it might help solve that issue. Yeah. I I tend to recommend that people don't use those comparisons too much.

I think the only reason they really exist is because, and I think everyone here would've heard this, at some point someone will say, uh, I'm in finance and so I can't do CICD because we're regulated. And so to be able to show them the door metrics for the finance industry and say, Hey, there's a load of companies deploying multiple times a day in the finance industry, it just makes you realize it's possible even in that industry. But they shouldn't really be used as like, why aren't you deploying as often as that company over there?

So, Well, and, and it's a good highlight that different industries may in fact have different results on that, right? You, you're, to your point of complexity there, there are very different levels of complexity depending on what you're creating and how you're creating it. Yeah.

What I really like about bringing in Dora to this, to this conversation is that what we're looking at is trends and benchmarking and analysis from two different vantage points. And they're saying the same things, right? Like they're saying like, if you have too many tools and like too many underperforming teams, too many people that don't know what they're doing, you're going to have poor outcomes.

If you have, you know, a lot of people using a lot of tools and adopting it and working like you're gonna have better outcomes. And so when you kind of look at the industry as a whole, it's really nice to see that we are being able to measure things, report on things, be able to assess ourselves against this larger group, you know, and to determine how we're gonna move forward. And I think industry, like companies, I'm sorry, foundations like the C-D-F-C-N-C-F, Linux Foundation, oh, all of them out there are really helping the industry as a whole say, look, we've been doing this for a long time.

Here are some key performance metrics that you should be looking at. You know, again, it's going to determine, like, it's going to be determined by your company, by the what you're doing, the complexities and all of that. But here are some reference points that you can see, like if you follow this path, you are more likely to be successful.

You are more likely to have a high performing team if you deviate. If you can't seem to get it together, this is where you're gonna be. And you can determine where you wanna make your investment.

Do you wanna invest in your people to get them where they need to be? Do you wanna level them up? Do you wanna invest in the next generation to Didi's point and get, so when developers come into your organization as interns, they already are aware of this tooling.

You know, like what plans are you going to put in place to make sure you are on the right trajectory forward? And here again, are tools and metrics and, uh, reports that show you like, if you follow these types of steps, this is how you could be successful. It's pretty awesome.

Yeah, It is. And guys, let, let me be clear, right? Would we do this show, it's 45 minutes or so long, it's hard to encompass this whole report, 45 minutes and get everyone's thoughts.

I really encourage you out there. If you're watching this, go to the CD found CD Foundation website, download this report. See for yourself, there's so many, so much, so much more findings in there, you know, standardizing on one cd, A-C-I-C-D solution versus using multiple ones.

And how does that increase or decrease productivity? And, and this, this, this, this kernels and kernels within lake, sort of a Russian doll thing, right? There's truths within truths within truths there.

Let's talk about the next report. Who marked the dc Our experts on the report? When, when, when is the next version?

Do this one due out you think? Oh, usually we issue, they, it's issued annually. Uh, Melissa's maybe better, better versed on the formal report, but in my case, I look for it about annually.

And so 12 months from now, we're gonna look at it again. One of my worries is it's going to tell us again, things are stable in the top, in the top. Organizations with a thousand or more people and stable in this case probably isn't the right choice for those organizations.

Stable in a software business in a competitive world is not great. You want to be better. You wanna be improving.

And so there are, we, we wish that people would take this as a call to action, as a step to say, oh, stable's good, but it's not good enough. You know? And the one thing we haven't talked about, um, is the amount of layoffs that have happened.

And when you look at the stress put on teams because they're doing the same amount or more with less. So it will be really interesting to see what, you know, what kind of trickle down effect did that have, you know, because that's important. You can't have high performing teams if you've cut half the team, right?

Because then you have a multitude of variables out there now that you didn't have before. So I think it's going to be really interesting to see, are they still stable? And could this be why, because they couldn't move forward because they're doing less or they're doing more with less.

Um, so that's, I mean, and that's a whole nother topic, Alan. And I don't wanna like open a can of worms, but I do think it will be interesting to see, like, have we hit a stagnation point because we don't have enough workers to make us move forward. Something to think about.

I we, we've seen this in other things, right? 25% more with 25% less. Anyway, I think we're gonna need to, uh, call a, uh, a close to this episode of CD, foundation of c, excuse me, CD Pipeline by the CD Foundation.

Um, I wanna thank each and every one of you for coming on. Lori is always, thank you for co-hosting. For you guys watching this, please go download the report.

There's just so much in there. I mean, I hope this has stimulated you to go do it, but really the report is out there for you guys to use, not for us to pontificate about. Um, everyone, hope to see you again soon on another CD pipeline show.

Until then, be well. We're out. Bye-Bye.

com is the leading resource for news analysis and education on challenges facing the cybersecurity industry. com covers all aspects of cybersecurity, including data security, DevSecOps, cloud security, application security, network security, security threats, and more. com has the largest selection of security content featuring breaking news, blog posts, podcasts, and more.

com to learn more. com. Home of Security Bloggers network.

Hello and welcome to the Active State podcast. My name is Shane Ward, I'm director of engineering at Active State. And I'm Ed Smith, I'm director of product here at Active State.

This is podcast number two, tech, debt or security. What makes you migrate, Shane? So you're got the job title and I more or less associate with deciding when we migrate between those.

Let's start with how do you, how would you define tech then? And what do you kind of think of as security? Oh, those are great questions.

Let's start with security, which is really, uh, a balance between how do I know that nobody can do anything I don't intend to do with our systems, and how do I make sure the people I want to do things can do the things they want? So there's that tension and balance there. I think that's important to keep this in mind.

How do I define tech debt? That's tricky because it's kind of a controversial thing. If you look at when tech debt was originally described, I think by Ward Cunningham on the old patent pat, or sorry, Portland Pattern repository Wiki, the C two Wiki, he was really talking about any shortcuts we make in the business requirements to get a product out the door sooner.

And by out the door, I mean, we've got two weeks or one sprint or one week, we wanna show a demo to our customer. What shortcuts can we take? What pieces do we not have to address?

What edge cases can we ignore just to make sure that we can put this code in front of people so they can use it quickly. That was the original intent of tech debt, you know, or very focused in investment. I think these days, 20, 25 years later, a lot of people see tech debt as anything in the system that I would do differently now.

And this is not a bad thing, it's just a different thing. But I think it's important to keep in mind the distinction between stuff I would change if I were to change everything. Now, if I were to start over now from stuff where we deliberately took shortcuts in addressing what users want in order to get something to them sooner.

So if we set those up in mind, that's how I personally define tech debt. That's why I personally define security. Does that resonate at all with you?

Yeah, I, I think the interesting thing about you, you mentioned the, the definition of tech debt has kind of shifted over time. What's interesting about the new definition is it's subjective. Oh, I would've done that differently.

Well, I I don't know anyone who does everything right the first time and I hope we're all kind of learning to be better. So every time you do something, by the time you're done, you should look at it and go, ah, I I could have done that better if I just had more time. So does that kind of imply that everything we do is naturally turning to DECnet?

Or do you feel like, oh no, is, is there a definition of the work that doesn't just automatically become tech debt? Are we ever doing anything that's good enough? Probably.

I mean, if you look at some of the work like NASA has done or things like that where like the documentary, like they write the right stuff. If, you know, for example, you're going to send a probe into space and you have nobody out there who can swap parts. If you've got cosmic grays coming in and things like that, you know that you have a launch window of this week and otherwise you're going to miss this and you're going to orbit maybe something, hopefully for years before you get where you want, you know, all these constraints upfront.

And you can design and implement your systems to those constraints and realize that those aren't going to change because you are not going to change orbital mechanics. Most of the projects we do aren't like that. You know, some of the products we're building at Active State right now are like, we think our customers are going to like this, but we're going to take something and put it in front of them and say, how does this fit with you and what you want to do?

We're gonna get feedback from them and come back and improve these things. So we have a deliberate, agile process of saying we're not going to overinvest in building something we think people are going to want. It's more important that we are able to get feedback from them hands-on use of an actual project, an actual user experience, and then we'll change how we think about building the next phase.

So we, we are deliberately, I think taking on some of that original Cunningham definition of tech debt saying we think these things might be wrong or we're willing to accept being wrong because it helps us move more quickly. It helps us not overinvest in things that turn out to be at best the wrong activities and at worst, potentially useless are actually harmful. So yeah, I think we naturally generate this type of technical debt just as a matter of process.

And you mentioned something important there. So the, the customer might see this in a completely different way. So from the, the artist, the coder, the person actually writing the, the program, they see, oh, I could have done that better.

The customer doesn't see the same thing, right? Like they see, right, if anything they experience it as slowness or, oh, that's odd, but rarely do they go, oh, this thing's full a tech debt. Like any product, I don't think anyone looks at a car and goes, oh, this is full tech debt.

They go, oh, this is complicated. Oh, this doesn't work very well. Oh this is slow.

So that's right. Part of just on, on my side of things where I see it is not, oh this, we could have done this better because I think everyone as a developer should be naturally feeling that way about their work if they're getting better, the question of, oh, when does this actually impact customers? And how do we make those decisions of, okay, well we've got this thing that we know we could have done better and we're always feeling that way, but what's the time to actually resolve these things?

What are the important things to resolve as far as technical debt goes? I think that's really the, a bit of the balance here. And just coming back to the central question of when is it time to migrate?

I think if you ask a person as soon as they're done, oh, could I have done this better? Is that time to migrate? Probably not.

And once it starts impacting customers, once you see performance issues, that's up time to migrate maybe. And like where do we draw that line? Because everyone thinks things could be faster.

In a perfect world, it's when does the impact felt? When does it impact, you know, sales and things like that. The actual things with what we do kind of becomes a question of tech debt there.

And we think we can make this more concrete, right? I mean, I could probably upset a lot of people by saying the formatting of your code doesn't matter. I don't believe that.

But I could be polemic and say that. But I think it's more useful to say there is duplication in this system. The customer doesn't notice that these variables are named the wrong thing.

The customer doesn't notice that this code might be difficult to modify or difficult to test. The customer doesn't notice that until the customer does when it affects the quality of the software, the reliability, the performance, things like that. But those are things we can measure in a way that is less subjective than I'm looking at this code and I feel nervous trying to end it because I feel like it's fragile or I wouldn't have written this way.

Or I think there's duplication here. And so it, it's important to me at least when I'm thinking about how to address all of these problems in engineering team, be able to say it's worse investing time and effort fixing these problems, these problems they're not worth investing in right now compared to the other, right? Because I know as an engineer myself, I would love to have all the time and effort and ability in the world to make everything polished and perfect for whatever subjective means that is.

But what I thought was perfect when I started, it might not be perfect when I get done. And that may not change how the users of the software perceive it or how it meets their needs or how it, uh, produces business value there. So that to me is an important distinction to be able to make.

Yeah, and the, the phrase that gets used, a lot of it ain't broke, don't fix it. And Right. I think as people who are interested in doing better or writing better code, right?

Eight broke is a pretty broad category, right? There's stuff that works, it's entirely subjective, it, it can work while it's on fire and it's working and people don't see it as long it's under the covers. So when do you think it's time to actually change things over when you, you've accumulated enough tech debt, like what are kind of the things you look for in a project or a thing or you're reopening it, you've inherited something.

What are the signs that you go, oh, it's time to do something significant about this? To me, it's all about change in the cost of change. You know?

So for example, if I have, I remember this from like the year 2000. 1 computer that was running an oscilloscope in a lab. And at some point that computer is going to stop working, capacitor is going to blow.

We're gonna have to replace the motherboard. That oscilloscope is going to be down because that processor, that machine is going to reach the end of its hardware life. There is a point in time where we're going to have to make a hard decision or that decision's gonna be made for us.

We need to replace this. So there is I think a natural endpoint to software and systems we need to keep in mind. That's one piece of it.

The other piece of it is really along the lines of that it broke. Don't fix it. If I don't need to modify something, it doesn't really matter if it's hard to modify, it only matters when I need to modify it.

You know, if I'm worried about this particular piece of code, this particular function, this particular object, this particular library changing out from under me or being fragile or, or not working very well. When someone goes in and makes a change, it always has bugs elsewhere. If we never touch that, it's not ideal, but it is a trade off we should consider making because if it is working and stable, at least for what it's doing until someone modifies it, we can put off modifying until it's necessary.

But again, that choice may be made for us. Similarly, you might say that we're using um, Cintas six and it's been working fine for us. That has an end date on it.

Python two seven has an end date on it and it's going to get more and more expensive, more and more difficult to move away from that because there are going to be fewer and fewer people. There's less and less support for keeping that thing up and running. So sometimes we can get out of that.

Sometimes we can't. My personal preference is to say anything that we can delay, because it's not the most important thing we should delay. I don't wanna modify things just because they're old.

I wanna modify things because you've got a plan to migrate away from them, which is probably answering the question with the question, how do you decide what your migration strategy looks like? Right? And I, I think the way we kinda see that cascade, 'cause it's a perfectly rational choice at the time, right?

Yeah. These are all perfectly rational choices. I think we should assume that we should assume been fakes.

Yeah. And then as kind of time goes on, you go, okay, well let's prop this up again. Let's not look at this.

Let's build another thing we could have done better around this to kinda keep it going. And then eventually you've got this structure that's made out of one-off decisions and rational choices that suddenly very hard to break into. You.

You kinda mentioned the issue of, uh, the difficulties and the upgrade. The difficulty is like actually making a change with systems that are complicated and the more layers you add onto it, all of a sudden you've got this armored onion of things that you have to get through just to do an upgrade. And you go, well, it'd be easier just to attack another layer onto this than it would be to upgrade.

But I think this is the point where there's a certain anxiety that starts to grow and we should have done something about that 10 years ago. Why didn't we? And then if you went back to a time machine, you'd be, oh, there's tons of other priorities not fixing this thing.

And then 10 years later, here we are. And that, that's where I find people generally get to. It's never anyone goes, oh, I wanna stay on the oldest thing forever.

It's just they find themselves there, right? Eventually. And they realized, oh, we've reached this breaking point.

It's very hard to isolate 'cause it's not one big decision or one big event. It's suddenly, oh, that's too old, we can't do anything about it anymore. And all the things we've done, the the guy who knew how to tweak this is retired.

There's usually some other event that brings up the choice is made for it. Yes. Yeah.

Since either retirement or the hardware, you kinda mentioned some of the, the older stuff, two 7 cent os. Sometimes it's, oh this is the last guy at the company, you had to do this and you're relying on him and he's won the lottery and gone up to The Bahamas. Now what do we do?

So that's that the generally the choice is kind of made for you. I think there's very few companies have the luxury of being very proactive about this often because, oh, the system works for profiting off it. Don't look at it too hard.

So in those cases, it's the situation that the choice gets made for you. And I think it's similar but a little different in the, the security space, right? Because sometimes there's mm-hmm, it's an outside influence again, but sort of an external factor rather than, oh no, this thing has failed.

It's oh, there's a problem. Oh, there's a problem that's super significant. How do you kind of see those things now just not a security issue, uh, pushes you to make a change?

And we've already talked about it's the same sort of systems, right? Like it doesn't make it any easier whether there's a security issue, it's just a layer on top. And I think it's also unpredictable, right?

Uh, I think one of the most difficult pieces of security is you might be running this piece of software and it has a vulnerability. That vulnerability was there. Since you deployed it, it's only after someone has discovered it or someone has exploited it that you realize we were not safe all along.

So that decision gets made for you there. And it's not necessarily the case that keeping up to date always protects you from that. To me, the question is really how long does it take me after discovery of a thing to make sure that I am no longer vulnerable?

Some of that is going to be your other security practices. If you're practicing defense in depth, for example, if you're practicing it monitoring and auditing and logging and things like that, you have much, much better capabilities to react to something untoward like that. But part of it is just going to have to be, you need to acknowledge there are going to be forces outside of your control you are going to have to react to.

And your ability to react to those as quickly as possible, as is reasonable, is going to be your ability to protect yourself, I think in those cases. And which of the, the, so teched versus security, which do you find kind of carries the bigger stick as far as motivating people to make those changes? Because often tech, tech comes from an internal, okay, now it's, it's finally time and security often comes from an outside influencer even, oh the, a new VP started and he's got a security mandate.

It's time to make these changes. Right? Which do you find one gets the company under mo more momentum to make a change?

Or do you find their, that's sort of like equally weighted equally? That's a great question. I mean, you could say legitimately that they are both existential risks to a company.

This potential, not this potentiometer, I'm sorry, we'll edit that out. This oscilloscope hardware goes down all of a sudden we can't do verification in our lab. That could cost us millions of dollars if you got a tape out coming up, for example.

Whereas if something like Heartbleed or Log four J or shelf or J comes out, you are going to spend how much time, how many days wandering around all your systems trying to find do we use this? Where do we use it? How can we get rid of it?

What's our migration plan? Right? You have all these potential outside, external shocks to your system that could come up.

What is your capacity to react to those and absorb those into your schedule and your resources and your planning. With that all said, even though they're both existential risks, I think security probably has the more visibility. And so just in terms of, are more people going to ask you about it?

That's probably the case with, I haven't seen a lot of security questionnaires that say, are you using Cintas six? Are you using Python two seven, are you using Pearl five eight? I have seen a lot of security questionnaires that say, do you keep your systems patched?

Do you keep your systems up to date? Have you did internal penetration and testing? Do you have disaster recovery procedures and things like that?

That may be the case that you find bugs where you shine the flashlight in the dark. I suspect it might be, but we should deal with the world as it is. That's where people are shining the flashlight in the dark right now, probably more on security than they are on technical debt, on aging systems, on, on delayed migrations.

Right? So I think you're saying it's more of a process of how do you stay on top of these things? Like one of the factors is gonna boil over some carry a little bit more weight than the others, but Right.

What, and I guess that leads me to like, how do you stay on top of these things in the real world? Because if you're just waiting for the threats to come to you, if you're just waiting for something to break, that's very reactive. Is, is there a way to be proactive and on top of this and reduce that anxiety over time or, or know when it's a good time to make these switches?

I think you have to make it a priority, right? Things you don't do very often, you're not gonna be good at. And if you say, our upgrade policy is we're going to upgrade once every two years on Memorial Day, and the entire team is going to come in over the weekend and work, what you're saying is, this is a really big bang event.

We're gonna bring in pizza, we're gonna bring in donuts, we're gonna bring in mocktails, we're gonna make a thing of this. But this is an extraordinary circumstance, an extraordinary event. I think what you're saying in that sense is we are not going to get good at doing this.

Whereas if you said, alright, every month we're going to take a service and say, if it's more than six months, so we made an update to it, we are going to test out upgrading dependencies and redeploying it. We're gonna test migrating it to a new system. That's going to be painful, that's going to be awkward at first.

But anything you do in a repeated fashion like that, a consistent fashion, you're showing me you value doing that and you're going to lessen the effect and the cost of making these migrations, making these changes, making these updates because you have to, you're forcing yourself to do it. I think one of the reasons people don't upgrade their systems is because they're worried it's going to be expensive. They don't know the scope of it, they don't know the di difficulty of it.

And those unknowns weigh heavily to say, I don't know, know what benefit I'm gonna get from this. I don't pay for it. These other things I could be doing.

I know how to estimate the effort that I know how to estimate the payoff from that. But what we're looking at here, oh, go ahead. I I was just gonna say, I think we see that model with stuff we actually use on a daily basis, right?

What you, you, you think back to, oh, it used to be I would buy a new computer with a new operating system every couple of years, and that's how I wouldn't actually upgrade. I'd just switch my system now. But now our, our cell phones, our watches, anything that's digital, oh, it's time to update.

Oh, can I update tonight for you? Like, it makes that part of the practice instead of waiting for there to be a vulnerability that reacts to often they're a little bit ahead of it or they're patching something that could have later turned into a critical vulnerability, right? Just because, oh, every couple of days we're gonna send an update.

It's not gonna be a big thing. We're gonna make these incremental changes. We know we should be making, make them on a regular basis, save that acceptable part of the practice.

Like I don't think we even think about those updates or just see it go, okay, I acknowledge that. Versus in practice, when you're working on a big pace of technology, often you assume it's gonna, oh yeah, this will be good for a little while. You mentioned once a year doing an update and it feels like the need to do those updates is accelerating unless you ignore them.

In which case out you're making the problem worse, right? We're not getting a big envelope full of floppy discs in the mail. Now we can do over the air upgrades for a lot of our systems.

I know a lot of people listening to this have air gap environments. They don't have quite that easy path. But this is still something I think you can plan for and execute on.

And I think anything you want to make a repeatable, scalable process, you can do that. And you can understand the costs and the benefits of doing this. You're going to have to get good at this if you're gonna do this on a regular basis.

And I think that's the only way to get yourself out of this situation is to make this, uh, a capacity of your system. Think about we are always going to be migrating our system and that may be shutting down old systems. We don't need, it had a start date, it has a maintenance date, it has an end date, it's going to be replaced for something new.

It could be just routine maintenance on our system. I remember at, at one point, this is way back when the, uh, consultant and I said, we're going to have to reboot this old windows aim T box we have in the corner because we can maintain this, we can monitor it, we can bring it back up. If it fails over a long weekend, we're not available.

This entire department is going to struggle. While it's not there, we're choosing to take on this potentially difficult task now because we've made the resources and the time to address any problems we have and, and deliberately fix them rather than being under a really hard deadline to fix them when some external factor came in at the least convenient time. Awesome.

So I think what we're coming down to is it's not one reason or the other. It's if you're not, if you don't have a plan to keep up to date, you're going to fall behind. And one of these things will make the choice for you.

That kind of a, a decent summation. Well, Evan, I appreciate the chat. Always good to talk to you.

I think there's more to say on this, but let's leave it here for now. Sounds good. Appreciate you taking the time to talk with, with me here, Shane.

I hope everyone enjoyed that. And thanks for listening to the Active State podcast. There'll be many more episodes coming down the line here in the near future.

com is the number one online destination for DevOps education and community building. com covers all aspects of DevOps, including DevOps, best practices and tools, DevOps culture, DevSecOps, business impact, continuous testing, continuous delivery and more. com has the largest collection of original DevOps content featuring breaking news, blog posts, podcasts and more.

com to learn more. com where the world meets DevOps, This is Textron tv. Hey guys, thanks to the drill.

We're here with Michael Freeman, who's head of threat intelligence for amis, and we're talking about cyber warfare and whether or not it's escalating and we'll let even escalate further as we head into the election year. Hey Michael, welcome to show. Thank you for having me.

Mike, What is your sense of where are we with cyber warfare right now? I mean, it's been going on for ages, but is it escalating in ways that we see or don't see? I mean, it's not like I pick up the paper in or seen in a headline all the time.

So the question is, um, I know it's happening, but how much? Good question. We're definitely seeing a large increase, especially, uh, from certain geopolitical issues, which arising out of both China, Russia, Ukraine, and key areas such as that.

And what we're seeing is, uh, targeting of not just the typical, um, election cycles you typically see with both all of us do, uh, when it comes to cyber warfare, but you're seeing a targeting of more of critical infrastructure more and more. That's, that's a key thing that we're starting to see a lot more of. Um, your water treatment plants, your medical systems, manufacturing capabilities, things like that.

Um, used to be just trying to instill enough information, your IP type of things. We're seeing still an increase in that process, but we're also seeing a larger increase of, you know, not just stealing the information, holding for ransom, uh, putting, uh, capabilities in there to do to knowledge service on key critical infrastructures too. So when you start tracking back down some of the actors behind the scenes, um, a lot of it still, you know, does come outta the key areas that we we're used to seeing, Ukraine, things like that.

But we're also seeing and tracking a lot of back down to the actual governments or threat actors attached to different governments. So Are they trying to actively knock systems offline or are they just getting ready in the eventuality that they need to so they're injecting malware or whatever else they can do today with an eye towards creating havoc tomorrow. You know, you took a look at some of the interesting attacks that, uh, Russia had done and against Ukraine and some of, uh, tech communications, especially in the satellite components behind, behind the scenes that they were, uh, compromising there, specifically taking down the infrastructure completely in some cases.

So you're definitely seeing a, you know, a twofold process that we're definitely seeing those on this side of, uh, the cyber warfare threat. Are there different classes of cyber warfare? I mean, it's one thing to take down a power plant, it's another to try to influence an election.

So is there a spectrum here and what does that look like? Yeah, and just like every government agencies, they have, you know, offensive capabilities with different, uh, directives that they're trying to achieve. You know, some of them might be psychological components that utilize in social media to influence, you know, voters to uh, really get them to believe in certain things are likely to vote in certain key areas that might be beneficial to an adversary, as well as capabilities and the different components that would be, you know, such of intellectual property, you know, what key members of, you know, Congress might be thinking about voting for things like that, as well as, you know, capabilities of taking down certain key infrastructures so that we're seeing an increase in really in all components of that.

What impact does AI having on all of this? I mean, are we starting to see the bad guys use that and maybe the good guys can benefit, but who's gonna benefit more? Uh, right now we're definitely seeing, uh, an increase.

Um, a couple key areas with using it, utilizing ai. The first one we're definitely seeing is very targeted. Um, phishing campaigns being built and utilized for that as well as additional capabilities when it comes to utilizing DeepFakes.

Um, we have seen a fairly interesting increase of key individuals being targeted within a company to make it look like, uh, it's coming from a key, um, executive within that company. Um, they're utilizing the language that the, uh, target would actually use and ways that AI had picked up on as well as utilizing DeepFakes and very targeted phishing campaigns. That's one key area.

Another key I'm looking at, and I've been tracking a few capabilities are, is developing of malware on the fly, utilizing ai. We've definitely seen some use, pretty interesting use cases for that. But you know, likewise, we also are utilizing AI on, uh, certain key components when we're doing reverse engineering and malware, uh, utilizing AI when it comes to, uh, intelligence, uh, components where we're trying to identify context of what threat actors are talking about utilizing cognitive reasoning, for example.

Uh, so there's, you know, we're, we're both offense and defensive side are really AI and some pretty interesting key areas. So What are we supposed to do about all this? 'cause sometimes people will say, well, you know, these are nation states, they have the best of the best and we can't defend against that.

So they throw up their hands and just hope nothing bad happens tomorrow. But are there things we should be thinking about that we're not doing? Yeah, actually it comes back down to fundamentals.

Even though they, uh, you know, both, uh, pretty much all countries, they utilize the best capabilities they have internally, uh, to build up offensive capabilities. Really if we just utilize the very basics that we have, uh, information security, that's understanding your assets, uh, identifying how you prioritize what you're gonna protect in your environment and having processes in place to actually deal with that. Uh, when you get back down to the basics, threat actors actually utilize the easiest to use context they're gonna trying to do on, on a, on a campaign.

So, you know, if you have have multifactor authentication, for example on key infrastructure, uh, that's, you know, definitely gonna stall threat actors that are taking advantage of the fact you may not use that. Uh, or if you don't understand what your OT environment looks like because you have no idea what what's in there 'cause you're focusing all on your it, you know, you'll see threat actors actually hide, uh, even though you think you kicked 'em outta their environment, they'll hide in your OT devices 'cause you have no clue that they're there. And so, like I said, it's going back down to basics is really some of the biggest key approaches we can take and to really medicate a lot of the attacks that we definitely see.

Are we in danger of escalation here? Because both sides or any side, depending on what part of the conflict you're in, tend to give as good as they get. And, uh, there's a tendency to kind of, somebody does one thing and then the next person responds and kind and then maybe adds a little and then some.

And so, uh, is, are we in the cusp of some sort of mutually assured destruction here or what? I think we've been that way for a while now. I think, you know, we have capabilities to do unto others as they done to, as they could potentially do to us.

And you know, um, you know, a adversary or nation state adversary would be very reminisce to, to, to take down a key infrastructure within the United States, such as a power grid using a cyber attack because we can do the exact same thing to them, if not worse. And so there is a, a, um, little bit of a mutually shared destruction component going on that if they could see playing out. But there is always gonna be individuals and actors that are going to try to push certain key areas what they can do, you know, push a little bit, see you what their responses are, there's not be response and they continue down that road.

If there is a, you know, proportionate or disproportionate response, they're gonna back away from that. Are there societies that are better equipped to deal with that level of disruption? I mean, in the United States, if you took down the power grid, that would cause all kinds of havoc that might paralyze us.

It's not clear to me that that same level of impact might be felt in China or Russia or wherever. Good question. Um, yeah, it's, it's, you take a look, uh, when it comes to something like that.

We understand where those are, what type of risks are, we've been trying to ident identify key legislation that we could put in place to help identify, uh, mitigate as well as, uh, not just protect against that type of, of an attack. Uh, I do know of communities, uh, in the United States that maybe cyber warfarin may not be the entire key driver behind why they would do certain approaches, you know, but they would look at natural disasters and say, you know, why should I rely entirely on one source of a key piece of an infrastructure, uh, take, you know, why would I rely entirely on wind power, for example, when I can have solar, I can have, you know, geothermal, I can have multiple different types of sources like that. So you'll see a lot of, you know, especially, uh, western nations or starting to really kind of explore that, not from a purely cyber perspective, but it, it, it benefits them.

In the case of cyber warfare does break out against 'em. So there's definitely, you know, areas that we do, I would say, uh, could potentially, you know, be able to bounce back from a, an attack that might take out certain key infrastructures fairly quickly. Um, you know, we, we, we've seen some attacks that on our nation's grid that, uh, were very targeted weren't using cyber warfare, for example.

Um, you know, a couple about a year or so ago, I think it was in North Carolina C and even in York, uh, Oregon, they were taking down or attempting to take down, uh, transformers with, uh, long range RI rifles, identifying key components on there to take out that might either cause cascading failure problems, uh, devices or components that are very hard to replace, might take years to replace or something like that. Uh, and you know, surprisingly we were able to bounce back from that fairly quickly. What is the current state of the public private partnerships as it relates to this?

It seems like, um, you know, the government clearly has a stake, but, um, not clear to me that all the owners of the utilities and the infrastructure and everything else, or is tightly connected with them or as they could be. I mean, is there work to be done there? A lot of work to be done?

Uh, you, you could definitely see, you know, there was in the past, uh, cybersecurity, uh, didn't really have a say and anything has to do with the government. And government likewise didn't, uh, play well, very well with the cybersecurity side side of things. You know, it was pretty much a one-way street of information.

We would give them information and you'll never get that back. But you're definitely seeing a change now, thankfully, some great legislation that came into place. You got some, um, leaders within, uh, know the Congress and senate that actually kind of get a good understanding of both cybersecurity and cyber warfare and the play that actually you need both sides to really work together.

You know, you got CISO that's really been stuck up, uh, stepping up to the plate, providing great intelligence back to the community and um, you know, you got the, in regard with the FBI. So there's a lot of good programs that have really been instituted. Uh, some legislation's been instituted a lot of ways, a lot more work to go though.

So as we kind of think about all this for a minute, um, are different. Is this a game for the big countries or are smaller countries also able to give as good as they get? It seems to me, um, I mean, I don't know South Africa, Israel, uh, uh, um, oh, and they all seem to have some sort of capability.

Yeah. Um, pretty much every com country that, uh, has a functioning working government, uh, has looked at capability either has already really been playing in this field for, you know, a couple decades already or have really started increasing their capabilities on the side of, of the cyber war are both offensive and defensive. So it's, it's, you know, definitely something we, we can track and have been tracking a lot of, uh, you know, interesting a PT groups coming out of, uh, countries you may not have never really thought possible, but they definitely have the capabilities that they're starting to do.

That doesn't mean they're as advanced as others, but the capability to cause havoc is definitely there. Well, I'm asking that question because is it ever gonna be conceivably possible that maybe there's some sort of un treaty where we all agree to kind of not mess with each other's cybersecurity because it's just gonna be too catastrophic? Or would it just not be worth the paper it was written on?

Uh, that's a tough political question right now, uh, there are seminaries, we still, you know, play nice with Russia and China, uh, not so much with other countries. And uh, you know, I I think you take the old adage where, uh, there is no adversaries, there is no, I mean, there's no domain, there's no friend or just a common interest that we all have. And so I think we get to a point where we'll have a common interest to actually reach that point, put legislation in place where the UN or whatever the case may be.

I, I think, you know, it would have to have something that would cause to get to that point. I don't think we're there right now. There seems to be a lot of shadowy organizations involved in this, um, cyber espionage, cyber warfare motion.

Um, do the, or do the entities that kind of contract those folks have any real control over them? Or are they just kind of doing stuff on their own and, um, coming back when they have something interesting to share with that government, but there isn't a lot of control? Uh, that's an interesting question.

Uh, you know, there definitely are some countries that are, that have, uh, you know, not exactly on friendly terms with us, but uh, are not exactly on bad terms with us that have hired former, uh, uh, people from different intelligence agencies that have the capability building offensive weaponry and have, you know, moved there and, you know, basically stated that they couldn't leave the country, uh, unless they were done the type of work they're doing. And well never go that work again. As well as, you know, you can definitely see if you have, um, certain, uh, channels and on different, uh, underground forms, things like that, if you have some capabilities that you're showing some real skillset, you'll be recruited, you know, you'll definitely have some interesting characters trying to reach out to you, recruit you both from the cyber, or not from cyber, but from criminal and, uh, nation state side of things.

Yeah. On the other side of the, uh, game as it were, um, aren't there mercenary organizations that are working for quote unquote good guys that are equally trying to, uh, recruit the same level of talent for purposes that are, um, shall we say, offensive in nature? Yeah, there's, uh, definitely organizations or companies you could say that sell capabilities, uh, that are good, used for good as well as bad, depending on the customers you look at that they sell to.

Uh, definitely. Uh, companies I have, I've seen that have been in the past very focused on really great innovation on the def defensive side of things that when they took a look at the money that's coming or offered from, uh, particular, you know, friendly even or our own nation state, our own nations to develop some offensive capabilities, have actually looked at that and said, you know, the money's there. Let me reach, recruit the ca the talent and actually build that capability because, you know, one, if I, if they can get the money to do it as well as, you know, the, uh, the, those connections to make that happen.

So it's, it's, there's definitely those out there. They're not as common as you would think, but they're definitely there, Right? Sounds to me like maybe the best thing you can do is go buy yourself some cyber insurance because it's only a matter of time before something bad happens, whether it's the, uh, um, financial criminal organizations or the nation states.

But, um, it feels like every business is now on the front line. Yeah, the, uh, idea of I'm too small, nobody would come after me, you know, no longer exists that it's, you know, if you just, you know, if you're a small mom and pop shop and you're running, you know, word for us or some, you know, off the shelf software that you think that's, you know, which you know, helps you with your business, it doesn't matter. An a PT group could be actually targeting the actual technology you're using, not you, uh, or they could be targeting just because you're in a certain region, for example.

And so it's, you know, you never know, oh, is the driver and the reason behind why you might be compromised, um, per se. Unless you really can delve deep into the forensics and, you know, get back to attribution on the actors themselves and try to elicit and figure out what they're trying cons behind it or unless they actually say it. But, you know, running the wrong technology stack or the wrong or being in the wrong location will definitely, uh, make you a target also.

All right, folks, will you heard it here? We all have a high potential to be collateral damage in somebody else's war, and that's never a good thing. So act accordingly and defend yourself as well as you can.

But, um, the most important thing is just be aware because when there's a lot of noise in the news cycle, chances are a cyber attacks are either coming or already have been launched. Hey Michael, thanks for being on the show. Thanks for having me.

All right. And back to you guys in the studio. This is Textron tv.

Welcome back to ServiceNow's Knowledge 2024 conference. We're talking with Amit Saxena and we're talking about how well automation and how it's gonna evolve because we have robotic process automation, we have frameworks in the now platform. We've seen the rise of ai, there's now this thing called process mining.

Walk us through how all this stuff comes together. Is it all gonna converge or are they slightly separate and orthogonal? When do I use what I I tell you what, you know what, we've been talking about convergence for a while, but it's already happening.

We have credible stories from our customers where they've started picking up a one tech stack, like integrations. It start adding elements of RPN and then once they burn the whole technical depth, they start mining more bottlenecks using process mining technologies. Um, and then we have added something new as well.

We got a stream connect with Apache Kafka offering. So all those customers wanna bring in a huge amount of data into ServiceNow bidirectionally. They can tap into our newest offering called Stream Connect.

We hear this phrase hyper automation. Everybody has heard it at least once or twice, maybe a dozen times. They all nod their head, but I'm not entirely convinced that they all know what it means or just, it just means something different for everybody who hears it.

Yes, I think this change of intelligent automation, half an automation, automation fabric, these are all interchangeable terms effectively for me, it boils down to giving a choice to our customers to be able to pick the right technology for the right use case instead of trying to shove in some technology for a wrong use case. And that is, and we have seen plenty of that. Like you have seen it plenty around our customers.

So hyper automation is truly bringing it. The second part is how do you package it together? Because if you make all of this discrete technology separately, it becomes so expensive.

Customers cannot realize that TCO over three or four year period. Um, so giving a very intelligent way, creative way to package it together. Thereby they just get enough, realize the value, and then they can expand with the consumption based pricing.

So we have brought this all together and the final piece of the puzzle, which you've been hearing since morning, is generative ai, is how we can increase the productivity of people who are building those hyper automated workflows, right? And with this gen AI infusion, you have all those app build, the app generation and a catalog generation, that playbook generation, that a bot generation and integration generation, we are making it easier for those novice developers to work along with ProSight doubles. Yeah.

Who has the appreciation for the pieces. And I'm asking the question 'cause if I go asked a dozen people here about, you know, how does this get automated? They all look at me and they go, ai, they have no idea what's happening underneath that.

And they give AI credit for everything. Are these things gonna overlap to the point that maybe we do have something that we're not standing around going, Ooh, that's RBA and that's this other automation framework. They're kind of just parts of the platform, their capabilities, Right?

I think we are overly using AI than what it is. Um, now the thing about that is because of chat GPD and other things, people are able to understand and automation what it means to them. Unlike any other, all these technologies which have been around for a while, it was very difficult for an end customer to understand because they were always in the background and doing their magic chat AI somehow they can relate to.

So now they're giving a blanket, uh, term out to everything. They don't understand it's gotta be ai. But underneath it, of course there is an elements of generative ai when you're talking about summarizations and cats intent.

Generations try to understand a lot of unstructured data and getting a meaning out of that. Um, however, there is still a very meaningful place for technologies like integrations. Today in the keynote with the bell, we are talking about service tower, able to search across all the enterprise system.

Guess what, how it's doing. It's all APIs based integration. And now if you're trying to unlock that data with Darknet and Java apps and legacy apps, you will need an RPA and, and so all of this technologies still have a way, and that's where the service door ecosystem really flourishes because we have made all of these technologies as a capability instead of standalone v play technology.

And that makes it easy for our customers to adopt it. Do you think we suffer a little bit from automation fatigue? There are all these islands of automations out there that people have put together.

It's hard now to go to the boss and say, we need one more because they go and look at you and go, the last 10 didn't quite do what we thought we did. Yes. And now I have integrations about integration.

So right. Do we need to take a giant step back and think about this? Again, There is no easy answer for it.

Um, however, if you start looking at where you wanna be from two to five year horizon and start building a strategic roadmap for a company, you will solely realize the pace you've got to move is much, much faster is compared to what you've done the last 15 years. So then you have to work at a strategic vendor, which allows you to move at that pace, at your own automation maturity. So then you apply this to this customers who have this islands of automation.

First thing is, hey, let's give you a visibility of what are you doing? You know, I mean, you'll not believe a lot of customers who have this multiple COEs still do not have a single view of what all the automations are doing. So we have invested in building products like Automation Center that gives you a single visibility.

Now, once you have that visibility, can find out how many of your automations are already giving you the value you intended to, if they were giving you value. You don't have to touch them. Focus on things which are not giving you a value.

Focus on those things. New organizational goals, which are hitting your point, that competitive pressure, especially on the top line, not the productivity. And the bottom line, the top line goals, and this is where you start looking at areas of exploration.

Instead of fixing something, which is breaking every single week and luck just to keep the lights on, there is a time to retire them and move with a modern order automation platform offered by service. I am often, often felt that the automation cart is before the horse. And what I mean by that is we have a bunch of applications and we add more and then we build the apps and we deploy them.

And then we think about the integration. Maybe we should start with the integration thing first and then build the apps around it, because then the integration thing's extensible. And I think you have a very fine point there, but I would take it even further.

What do you say? I don't think so. We need to be technology focused at all because the moment you start picking up a technology, start building it around, you have lost your focus on the outcome.

So you start with what is business users trying to accomplish? Because there is a multiple ways to deliver that experience. But coming back to your point, you are right.

What you touched upon is what we call connected enterprise. So connect your enterprises that, what I mean is various departments of various workflows, they have different tools, different people, and different system of records. So first, create a fabric to connect them together.

And this is, we are not even talking about any automation, they're just talking with each other. And once we understand each other language, it makes it much more easier to identify those automation opportunities and succeed in implementing them correctly, which is more future proof. So, so you are right, but I would still say, let's start with outcomes and not the technology.

I once had a conversation with Abella and he was explaining to me all these integrations and he's, and, and after a while I was like, you know, well that sounds like a bit of a mess. I mean, how do you kinda cope with that? And he looked at me and he said, it's a mirror of the business.

The business itself is the kind of a mess. So I, how can you expect it not to be a mess? So do we need to have a conversation between the business and the IT side about what is the goal?

What is the strategy? How do we do this? Because one doesn't work without the other.

This is going back to your earlier question and that is a lot of those automations and integrations were done point to point because a request came in, it was urgent for that time, needs to be delivered in a specific window. And the folks, the COE teams members and IT developers who worked on it had a very siloed view of the technology they had in hand as they used the technology to build it. So that became a point to point integration.

Now, any asset built using point to point integration is not reusable across anything else. So that talks back to your patterns integration products. So that's where the single data model, single architectural approach, because it avoids you to make certain mistakes, which you would do otherwise.

And if you invest into that right from the beginning, it'll be, it'll be good. Oftentimes a lot of the customers think, Hey, it's a low code provider. We don't have to think through this.

But at least for the first few integrations, it's good to bring enterprise architects into the mix just to make sure we are designing this right, because this would have ultimate lot of written. So you can have five ads that instead of trying to just launch the application in one week and claim it a victory, because you may end up taking sort of these shortcuts, which you have to redo it again, I feel like in a lot of ways the technical debt of the integrations is killing us. Yes, we're spending, You know, the space 70% to 80% of our time maintaining the integrations, right?

And we don't have enough energy left over or resources to actually drive some innovation. So do we need to take a look at this and kind of maybe measure what is our automation technical debt and how do we think about that? That could be a one way, that could be a one way to take a look at it.

The the other way could be, see if you look about maintenance, I think it all broadly categorize into two bad degrees. One is just the upkeep, meaning things are changing. You just need to keep the lights on.

The other is error resolution and mitigation. Things oftentimes do not work the way it's designed to work. Automations fail.

They're, I, I'm shocked and appalled. They hear that. And when they come down, how, what do you do?

What is your run to address it? How do you bring them up? So it's a lot of that made mistake you're talking about spent on just correcting those integrations.

Now just imagine if you have a visibility tools of identifying the patterns of why they're failing and anything which is overly failing, instead of trying to automate fix it, you just redo the implementation, redo it on the right platform so that way you're not wasting time for a valuable time of your automation COE team in just maintaining that integration. That approach comes from top down. The leaders saying, just emphasizing this new mentality of how to rally our series.

We, over the years have seen, uh, the rise of this digital business transformation officer. A lot of them are chagrined because they're not having the level of success that they thought they would have. They went out With ideas, the Promises.

Yeah. Right? And they run into this integration nightmare and then they can't quite get anything to work as quickly as it can and things that should take months now take years.

How do we kind of reinvigorate our digital business transformation initiatives? Because today it feels like we're stuck. We, I would say it flus, it dumps down to, we have always from the last 15 years by Q and I, in fact we have talked about, hey guys, pick up the best of the breed.

And we flooded that message in the market and people responded. Even a lot of analysts wrote about it. So that's where that island of automation problem arise because of that, right now, the best to the breed technologies often are not designed to work with each other.

So then you are trying to build a custom solutions and then you bring in huge amount of time. NSIs or you, lots Of consultants. Lot of consultants and we love them by the way, right?

But in doing so, very few people had paid attention to the enterprise architecture that connected enterprise, what you were talking about. So the one prop, so one way you can do is, hey, well why don't we bring all this smart people with enterprise architecture to design it, right? But that becomes the overall BPM way.

Nobody has time, patience has gone down. The second way is pick up a strategic vendor which is automatically connected with each other. And that's what the ServiceNow differentiation is.

The, we keep on holding it. Every single technology ServiceNow is adding to the tech stack or even an acquisition made is re-platform all the same now data and architecture. So that problem you were describing does not exist, at least in ServiceNow technical stack.

You still have other systems, enterprise apps you're talking about. Now you take that approach, but you have reduced your bigger problem down to much smaller problem. But you can now, and Chris, I kind of see two types of leaders in it these days.

Um, and they're both on polar opposites. One is, ooh, new shiny object, go chase. Yes.

The other one is, you know, this is my thing. I'll use this thing. This is my hammer.

I'm gonna go use it for everything. It seems like we need to find some way to get to the right tool for the right job at the right time. Yes, You said it, you said our pitch very well than anybody would else.

That's why we are really excited with the way we have packaged our hard hyper ation because pick up the right, choose right tool, right tag for your right use cases. Now we have to be also very clear about this leaders you're talking about. They have a average tenure of three to four years.

They will to show certain value. So at what time of their career they are going through this transition also changes their mindset. If they get out, see the success go through before they go to retire or move out of the job, they'll be less incentive for them.

But the people who have already made mistakes and learned from it are much more open in adopting and in grazing new technology. But at the same time they wanna do it right. And that's where they, if they realize they have internal gaps, they bring in certain experts into, and that would be right.

Consultants, This sounds like being president. I spent the first two years blaming the previous administration and then the next two years saying it's the next guy's problem. Right?

Right. And that's where we can really help educate because, oh well I'm here only for a certain period of time. I wanna show the success because my OKRs are based on that limited time.

So show me the value within an 18 months or a year. And that's where we are trying to say, well, time to value matters and if you really are interested about time to value, you might as well pick up a strategic vendor where you can save all the time. So you've worked with a large number of customers over the years.

Are there anything about them that you've seen where they have attributes among the ones that that are successful? Or do they have any common psychological traits or technical history or what leaps outta you about the people that you know who are making this work? I think it's less about the organization culture.

It's more about one individual in this organization, which truly believes and can rally the troops around them. These are the guys who would be spending a lot of time in an event like this and knowledge. These are the guys who will put together a pilots and prototypes to educate and they are passionate about ongoing learning throughout the year.

Now you have to find these individuals inside the organizers. Sometimes you're lucky to find those other times you have to hunt for them, keep on looking for those. And these people, once you find them one and twos of it, they really fundamentally change the success rate of your overall digital transformation of business transformation efforts.

And so it comes to all to these people, be really passionate about this automation technologies and they are able to influence the mindsets of certain leaders to a level where a seller or people like you and me will not be able to do. All Right folks, you heard in here, Hey, sometimes we just need a hero. And a lot of times those guys are girls, but they're gonna be leaders.

They're gonna be the leaders. And that's what defies the next generation of leadership for us, for ServiceNow, for our customs. Thanks.

Pleasure. Pleasure. Thank you.

All right. And we'll be back in a minute.