86. Network Automation Needs Standards – Tech Field Day Podcast

Network automation is the future of network operations, but what standards are you adhering to? Are you even aware of what the standards are? In this episode of the Tech Field Day podcast, automation Needs Standards.

Welcome To the Tech Field Day podcast, where we bring together a group of influential IT experts from across enterprise IT to discuss hot topics in the technology industry. This podcast is brought to you by Tech Field Day, which is a part of the Futurum Group, and is often recorded in association with one of our events. We're here today at Networking Field Day, and we'll be talking about automation and the need for standards.

But before we do that, I wanna have our guests introduce themselves so you know who they are. Starting with Denise. Hi, Denise Donahue, uh, network architect, technical author, all around Network Geek.

I am, uh, Steve Plucka, network architect with, uh, DQE Communications out of Pittsburgh, and also General Geek. I'm Kevin Myers, also network architect, um, and, uh, general Network Geek, IPV six, geek, uh, service provider geek, um, routing and switching geek. Alright, and of course, I am Tom Hollingsworth event Lead for all things related to networking here at Tech Field Day.

Let's jump into this episode. No doubt. You have probably heard about the importance of network automation, whether you are starting that journey yourself, you're finding yourself in the middle of a long project or in some cases even wrapping up this, uh, amazing thing that you now have done to make your life so much easier.

But were you following the standards the, the whole time? Did you know that there were standards? Did you lie to me and tell me that you thought there were standards?

Because there aren't in this episode of the Tech Field Day podcast, the premise is automation needs standards. So let's talk about this for a minute, because this was something, as soon as you guys said that this was the topic we wanted to talk about, I was immediately on it because the last time that I checked, there is no written standard for automation. There are a lot of standard protocols and ideas that we use, but I find it funny that when you start talking about, well, automation, usually you get barrage with a whole bunch of questions of, you know, which scripting languages are you gonna use and which platforms are you gonna use?

And is everything gonna be written in Camel case or Pascal case? Like, like you have to figure things out here. What is it about network automation that lends us to being, for lack of a better term, creative with the way we implement it?

Well, I think a, a big part of what it is, is this development all along. We, we've been CLI jockeys forever. Mm-hmm.

Uh, and now that we're getting into the move of automation, the first thing we did was create a couple of particular protocols, whether you're using YAML or whatever standard in that sense that you're having to communicate with the devices. So now we've got enough of those going that we're now doing scripting and, uh, each vendor is individually creating a platform to do their equipment on. And then there's a handful of, uh, multi-vendor companies out there that are picking and choosing which platforms they're gonna support.

So I think we finally reached the critical mass where enough of this automation is happening and enough of the bits and pieces and tools are there that we need to get together as an overall community and create that standard on this is, this is how it should be abstracted, this is how it should, should work, and figure out who the hell's responsible for that. Yeah, it reminds me of the early days of the, of networking in the internet where there was just like, my protocol and your protocol and this, you know, and just everybody had their own idea about how things were gonna go, how data was gonna be structured and everything. But then there was, at that point there was a small enough community 'cause small enough group doing this that they could get together and make and create standards and fight over who was gonna win.

Mm-hmm. I Think the challenge that I see is that it, where we're at right now with automation is the vendors really drive automation. Whatever vendor it is that you're choosing really drives the automation.

And yes, there are open tools and there are open frameworks, but to the point of the podcast with the person sitting down and figuring out how they're going to assemble those is doing it in whatever way they think is best. And a lot of it comes from the coding community, which not all of those practices translate over to network engineering. And so I think we do have, there are some standards that might provide some guidance, industrial control world, um, world of energy.

They've had some automation standards around for a while, but it does take a bit to adapt those into networking things like the Purdue model. I think those are places to start. If we think about, let's take automation that's specific to networking and standards, and what does that look like for an enterprise network?

What does that look like for a service provider network and figuring out those best practices? Well, I think one of the challenges that you mentioned there is that two industries that kind of have, those are also very highly regulated. And so a lot of the standards that kind of come out of this are not so much, this is the best way to do it as much as if you don't do it and prevent these outcomes, we are going to sue you out of existence or fine you until the pain stops.

Like, you know, for example, um, something like P-C-I-D-S-S drives a lot of the way that we design certain kinds of networks because we can't have certain things interacting with, with each other, or we have to have certain retention policies and things like that. And as far as I know, there's no kind of guidance in the industry right now of let's just say automating healthcare facility. Like, you know, if, if HIPAA applies or if, if this kind of patient protection thing applies, that's gonna direct the way that you do things and create kind of a defacto standard, if not a deger standard.

But again, that's vendor specific. You look at HIPAA and you've got the different, um, medical records vendors, you know, the, the, um, and they're, they're doing their own automation within that, which then they've extended on into the hospitals. Um, so then is that gonna be the same?

How's that gonna carry over to all the other companies, types of enterprises? And there's a difference between policy and standard. Mm-hmm.

HIPAA is the overall umbrella policy that's saying what you can and cannot do. What we need as engineers and as vendors is, uh, a way to implement that policy and create the workflows necessary. So far we've been concentrating on the individual task level things, uh, and we need to step up to, to workflows and, and processes that Yeah.

They have to be driven by some overall arching policy Or business logic, Uh mm-hmm. Yeah. On how, how this is gonna work and the categories you're talking about Right.

As well. But, but that needs to be a team coming together and we need a, a group that does that, that doesn't be solely benefit. It can't be the vendor, right?

Mm-hmm. It has to be the, the engineering teams in general as a, as a group. Yep.

Manners kind of comes to mind. M ma NRS mutually assured norms for routing security, which is basically a lot of carriers are involved in it, CDNs, and it's a neutral organization that says, these are the policies that we feel that we need to implement for routing security on the internet and in the DFZ. And then they take that a step further and have technical recommendations that if you're Cisco, if you're Juniper, if you're Nokia, here's how you implement that policy.

Here are recommended guidelines. And that's not quite a standard to Steve's point. But that to me is kind of along the lines of where you might need to go with automation is recognize the problem, define the problem, and then get down to those, those technical details of how do you implement this?

And in my opinion, the challenge is automation is coming outta the world of coding. And what does coding love to do? They love to jump to the newest programming language whenever there's a hot new language.

I mean, how many have we been through? And I think that's the biggest challenge we'll have as network engineers, is we look at standards and protocols over a very long term arch. The world of coding is always jumping to the next new thing.

So how do you, how do you, how do you bridge that divide of how do we pick something, how do we pick a winner in the world of automation to create standards around that isn't going to get left in the dust of the world of coding? You don't wanna run my pearl script? I sure Pascal, I want go.

Yeah. I want to do it in Pascal. Yeah.

All right. So who's gonna do this? What stand, what standards body are we going?

Well, before we get to that point, I think that that something that Kevin brought up that, that is very germane to your point, is there are groups that are kind of creating these informal agreements, manners, for example, it's in the name right? Mutually, um, Assured norms for routing security, Mutually issued norms, not requirements, not restrictions, norms. And we see this a lot in our society, right?

Like something like holding the door open for people. That's a norm in certain spots in, in the us but it's not a rule, it's not a requirement. And the difference is when you cross that line of a group of people getting together as an industry organization saying, this is how we're gonna do things, doesn't have any weight, because I can choose to leave that organization whenever I want and not abide by those rules.

But to your point, there are standards bodies out there Yeah. That provide guidance on how things are gonna be implemented in the us The two biggest ones are IEE in the IETF. I would also add ISO as a global standards body.

Um, I, I would recite chapter in verse with iso, but nobody does that. These groups are formed from people who analyze the problem, decide on a, a method for recommended implementation. It is voted on, it is debated, it is argued, it is voted on some more and, and then it is released.

Why have we not seen anything from any of these standards bodies about this yet? You'd have to ask them. I, I would, I would say there hasn't been a demand.

The only one that I've worked with is IETF and I, I've done, you know, a little bit of work in the ITF and so that's the one I'm probably the most familiar with and it's also the most open. Mm-hmm. I think, yeah, ISO and IEE are, you know, they're solving real problems, very complicated problems.

But that is usually, you know, there's someone is funding that to go to go do and solve that problem. And again, that's a long arc, I think programming and coding and automation and are all there together and they iterate very quickly. So I think you have to start somewhere, whether it's in the IETF and we talk about the IETF is the right place, or whether it's a new organization.

I think you've gotta look at that and say, how do you create an organization that can help to define those, you know, first maybe norms, then best practices, and then ultimately standards. And with the IETF and the ITF is very, it's very protocol and operation specific. So I mean, I could see it going there, but I could also see challenges in, you know, in getting it through there.

Which working group does it go into? How does that, you know, I see pros and cons in it going into ITF and, And. Mm-hmm.

You bring up a really good point here, and I'll reference everyone's favorite XKCD comic from Randall Monroe. There are seven ways to solve this problem. I know I'll create a standard that encompasses them all.

There are now eight ways to solve this problem. Yeah, exactly. Like one of the, and one of the things that we've seen over the years, like IIEE is very focused on protocols, right?

That's where we came up with ethernet. That's where we came up with wifi and a lot of those other things. But with IETF, I feel like there is a lot of debating things until they're dead.

And that's the way they eventually come to a consensus is, okay, are we done arguing about this? Does anybody still care? Perfect.

Now it's a standard because these are the, the remaining people. And we saw that with TRILL and SPB that kind of became pseudo standards, but didn't really, and TRILL is actually a really good example of this because the, the definition of what was going to be IETF standard trill never developed because the vendors were so focused on making sure that their version was the standard. And I'll go out on a limb here and say that one of the reasons why I think that that happened was because there was no bad guy.

What do you mean power Over ethernet and trunking on switches? 3 af? Because Cisco had a competing standard and everybody in the industry lined up and said, we want to do the exact opposite of what they're doing.

Oh, you want us to wrap the switch in a, in a, a trunk tag? No, we wanna put it in the header. 'cause it's what you're not doing.

Oh, you want to carry power over 5, 6, 7, and eight on the ethernet wires? No, no, no. We want to use 1, 2, 3, and six.

Because you're not doing that in a way. Standards sometimes are driven as a big middle finger to a competitor. So maybe what we need is Red Hat to come out and say, oh, well we're gonna use Ansible standard network automation, and the rest of the industry might wake up and go, you know what f you we're gonna go do things our way.

You know, I I think the you two have got, have has some excellent points that kind of could be drawn together. Like the why part, what's driving it. You talked about funding, um, you talked about, um, you know, vendor pushing.

Um, you talked, you talked about, okay, you know, the, the reason for doing it just as a, a reaction, also a need for it. And obviously there's a need for anybody who does much work in this and is dealing with multiple vendors of equipment, multiple vendors of software. There's definitely a need.

Um, how are you going to get someone to do it? Is the thing That's the big challenge. 'cause I think you look at, um, you know, does it, we've taken the existing, you know, protocols, if you will, and we've take, we have a p we have APIs, we have mm-hmm restconf, NETCONF, all of different things.

And, and the companies that are out there have taken the existing available protocols, created a few new, and that's what we use. And then how you cobble that together is up to you. So I think the real question is, is does automation, you know, whether that's, you know, automation control, that's, you know, pushing configuration, um, you know, telemetry related to automation.

Anything that's in the realm of automation, does that become an 800 series protocol or is it gonna be like an RFC? Is it a standard? And I think that's probably the first problem you've gotta sit down and solve, is are we happy with what exists?

Or do we want to build something new that encompasses the entire world of automation? And the question then becomes, if you're gonna build something new, how do you get people to sign onto it? Exactly.

Mm-hmm. Because one of the things about a standards body is, is that when the standard is defined, everybody has to play by that standard, or it's not a standard. Mm-hmm.

And you run into these situations where it's like, okay, there are four people that all believe that this is the way to do it. We have to pick one of these solutions, especially if they're not able to be merged. Right?

Like, you know, if the, if these two people have one that's pretty similar, we could probably merge them together. And then 50% of the use cases are kind of done like that. But it goes back to, well, if I don't like your solution to that, I'm not gonna be in par involved in this working group anymore because I'd rather do what I do because it's easier on my developers.

My customers prefer it this way. And in some cases, if I can get enough people to sign onto what I'm doing, I become the defacto standard anyway. Hmm.

Do your customers prefer it that way though? Or is it just that they have to take it? Well, that, that's part of it is if you don't know there's any other options out there, my way is the highway.

Yeah. Well, the, the, the customers are drowning right now. We're, we're just dealing with the day-to-day problem on how to keep hundreds or thousands of devices doing what they need to do and changing the way they need to change in a, in a reliable way with what we have in front of us.

So, uh, it's, it's time to take that step back and, and to, and to try to figure out how, um, I think the big difference between this, what, what we, the problem we have here and, and the normal standards is that it involves a, a workflow, um, type operation. It's, I, I guess the closest thing we've had previous to this is MPLS, where there's a whole lot of steps that have to happen to create this end to end, right? And everybody on the path has to play nice with each other for it to, to work in, um, in a vendor, in a, in a mixed vendor environment.

But even that is not as complicated as what we have now because we need to, you know, you know, we need to do, you know, precheck change, post check, you know, rollback if necessary. Uh, you know, in the post check this, the, the whole workflow aspect of this that we sort of inherited from the developer world is the, uh, is the part that is alien to, you know, the average network engineer. Yeah.

And that's the biggest challenge I see is I don't, you know, I, I haven't written code and anger since the nineties 'cause I'm gonna decide to be a network engineer and I didn't want to do coding. But, you know, now that we're in that world where code is very much a part of, of network engineering, the challenge that I see is I either go to a vendor and I have to go buy something and they're gonna tell me how to do it. Or I've gotta sit there on their gear only exactly right on their gear only in most cases.

Or I've gotta go out and look at what the development community is doing and figure out, okay, which of the thousand different ways that somebody has figured out how to build this framework, do I go, it's the build or buy. That's the challenge that you always have mm-hmm. As an organization is do I buy it or do I build it?

And if I build it, what is my am I doing? Because as a network engineer, let's do a protocol. Okay, well what they're like, you know, half a dozen routing protocols, there's layer two, there's a, there's a sit limit, uh, number of protocols.

You know, you're dealing within boundaries that are pretty well defined when you look at the protocol stacks of network engineering. But you look in the world of automation and coding and it's almost limitless in what you can do and what you can build. And that's the sea of uncertainty that I think network engineers or people, especially people at the network architecture level, you know, where we came up in the old world of networking.

And the newer engineers are, you know, nothing against them at all. It's just they're coming up in a different world of learning coding and network engineering. We're looking back at a long career of here's why we did things this way, here's the choices we make.

You've been through enough networks to realize that the choices that you made maybe weren't the right ones, and you do it differently mm-hmm. Later. So how do you take that knowledge and work with the network engineers that are coming into the field now to help them to also help inform the automation?

And I think you need standards and frameworks for that too, is as to how to make and evaluate those decisions. The other thing that I think we need to be aware of is the trap that we can fall into by turning this over to a standards body. This is something that's happened in the wifi world recently where we're going to introduce a standard and we're gonna make sure that everybody follows it, but we need everybody to buy in on the standards.

So what we're gonna do is we're gonna take some of the hairier pieces of it and we're gonna make them optional. And that is something that has happened quite a bit in the last couple of releases of wifi, wifi succeed, and wifi seven where some of the things that make it good are optional and don't have to be, uh, implemented by you if you don't have the technology or the desire to make it happen. And so what we're left with is a slightly better than the last one standard where we hope that everybody plays nice on these other pieces.

How can we prevent a standards body from coming in to restore order and ultimately making things worse? Because the things that we need to restore order are left for optional because well, if you don't make this optional, I won't sign on to the standard. Ouch.

Maybe that's why there hasn't been a standards body. And, and, and that kind of comes back to that whole how do you get everybody in the room to compromise? And for the purposes of this podcast, everybody knows that the definition of compromise is when nobody gets what they want.

Mm-hmm. Well, the other challenge you have is if you were, let's say you were to take this outside of something like the IET after, you know, ISO or IEE, how long does that take to, to build that organization, to get people that want to contribute? You know, when are you gonna, is it gonna be years before you see a work product that's, you know, that is relevant and, and going to help you?

Because I think that's a whole other challenge of if you do create another standards body, what's the, you know, the ramp up of that is, is gonna be a lot. I'm sure they're working on finalizing the standards for building wooden sailing ships in the ISO like this year. So, you know, there are only like nine centuries behind at this point.

Yeah. Only need 90 bucks to access the standard. Yeah, Exactly.

And, and that's the other problem too, that a lot of times you need a lot of resources. You need to create working groups, you need to create, uh, people, you need chairs, you need folks who have disposable time. And then for people to be able to access the standard, you have to have resources to, to do that.

Either they're people who have access or that you pay for it and, and then you create this rolling problem of, well, if I don't, if I can't see the standard, I'm not gonna follow the standard 'cause I can't follow a document that doesn't exist. And how do you prevent that? I mean, we, we bag on the I-E-T-F-A lot because it feels like they're, it's just basically an excuse for people to fly around and argue with each other, but they do get things done pretty quickly because somehow all of that arguing eventually does lead to some kind of an eventual consensus.

'cause I think they realize they're not gonna get any more out of it than just the, the heated discussion. They're not like an a giant organization like ISO or IEEE that kind of direct things, but use that as a way to fund the organization. Yeah.

There were, there, they're definitely the right structure. Um, for this type of thing. I'm just not sure if the, there's enough, there may need more participants.

The half of the half or maybe even more than half of the right people and companies are involved in the IETF. But adding in this additional layer of, of, of, um, of the workflow aspect of it and understanding the history of it too, you know, saving those configurations, which we've never really historically done. You know, it's a, to compare to, you know, the, the pre and post check access of the, of the workflow.

These are not traditionally things that network equipment vendors are, are good at or know They're, they're good at getting you a box that does a thing. Yeah. Yeah.

Yeah. And, and that's the, that's sort of the missing piece of the IATF, uh, the, the fundamental section I think has worked so far. You know, we have, you know, the, the neck off, the yaml, the, all the options available for the communications and the, uh, and the saving.

What we don't have is the, uh, is the discipline of the, of the programming community in the workflow as aspects of this and the history aspects of it so that we can see what changed and, Um, Yeah, and, and why I think it's also the, um, it's getting the, there's very few people that have a very, very long deep network engineering background and also have the coding background that understand how those worlds come together. So I, I worked on a project one time where a coder was off trying to solve a problem and he spent like a month or two writing code. And what he had we'd written was Radius, he had built Radius.

And I said, you know, there's a protocol on the router that already does what you're, what what you just spent two months writing. He is like, what is it? It's called Radius.

You just go turn it on. And that's where you can have a brilliant programmer that understands coding and even understands trying to solve a problem. But if they don't have the domain knowledge of what is possible mm-hmm.

In network engineering, then you may spend, you know, you're sitting there beating your head against the wall solving a problem that's already solved. And I think that's just as important as getting a standards body and defining the standards is what already exists that we want to leverage so that we're not reinventing the wheel. Mm-hmm.

Mm-hmm. And, and ultimately I think that that is where we're at is that as networking folks, we have worked very hard to create these small areas where we can standardize on certain things. Whether they're things like Radius or they're manners type, uh, assured norms.

What we need is a neutral third party to step in and effectively get everybody's ego out of the conversation. This is how things are gonna be done. I get that you're not the way, that's not the way you do it, but this is how we are going to do it going forward.

And if you can have that happen with someone who has the personality to pull it off in an industry where people are going to have to agree to disagree about certain things and put their competitive advantage aside for the betterment of society, I think what you'll ultimately find is that we can standardize automation if we're willing to put in the effort. That will just about do it for this episode of the Tech Field Day podcast. We want to thank everyone out there for following along.

com/podcast. You can also find show notes and bios of all of our guests. com for more information on our upcoming events.

com for more information about the Futurum Group. We'll see you soon.