102. Networking Needs More Telemetry – Tech Field Day Podcast

The modern world of networking relies so much on data that we need every available bit we can find. Telemetry is king, but where are we going to find more sources? In this episode of the "Tech Field Day" podcast, networking needs more telemetry.

Welcome to the "Tech Field Day" podcast, where we bring together a group of IT technical experts to discuss a single idea about key concepts in the industry. This podcast features a variety of perspectives from members of the Tech Field Day delegate community, and is often associated with one of our events. Tech Field Day is part of the Futurum Group, and this podcast is also published on our sister company's website at Techstrong TV.

In this episode, as we head into the very jam-packed Networking Field Day, we're going to be discussing telemetry. But before we do that, I want everyone to introduce themselves before we get going, starting with Jason. Hey, thanks Tom.

I'm Jason Gennert, and I'm with the US Networking User Association. Also a consultant with Bits in Flight. Longtime technology aficionado.

Started in math and then discovered networking and coding and all sorts of other goodies. Hi everyone, Scott Robohn. I'm a founder of a consulting company called Solutional, and a co-founder of the Network Automation Forum.

Internet plumber for a very long time. Very excited to talk about the topic at hand today, Tom. Well, we're very glad to have you all.

Of course, I'm Tom Hollingsworth, Event Lead for Networking here at Tech Field Day. Let's get into the episode. There's no such thing as too much data, and when it comes to network monitoring and management, the more data you have the better off you are.

But where are we going to find all of this data? Because after all, we've been mining all of our networking equipment for a very long time. But as it turns out, networking equipment is not the only source of truth in your enterprise.

The premise for this episode is that networking needs more telemetry. So let's talk about why this is, because I can remember a time when it was very magical that I could extract NetFlow records from a device. " But anyway, what happened was is that we got a lot of really cool information about a lot of flows in the network.

But one of the things we learned was, it's kind of hard to ingest all of that. Fast-forward to today in 2026, and I have collectors and things that can ingest massive amounts of data. I just have to find it.

So where are we going to come up with all this extra data that we need, folks? Can we start with the need? Let's push at the premise a little bit.

Why does this even matter? Well, a long time ago in a galaxy far, far away, we didn't really care about real-time response from the data network, right? And we were okay with RIP having 30-second update intervals, and maybe not finding a route missing for three of those intervals or 90 seconds.

And then we decided that link state protocols were really helpful to help identify those things very quickly. And over time, we've moved much more to the data network, to the IP network. We've done some things, moving from just SNMP to NetFlow and now beyond, because I have much more mission-critical traffic and more traffic that's very sensitive to human interaction and meeting human expectations.

Live voice is the easiest example, but we're doing much more than that on the IP network today. Hence, we have to be able to measure and respond to things much more rapidly than we did 20 years ago, 30 years ago. I'd like to add a perspective to that, which is, I think networking people, we've learned to be sort of jacks of all trades.

There's physical error. There are all these things that go into networking, and it's not pigeonholed. And what we're seeing now is we have to support networks that carry all sorts of critical data, real-time interaction and whatnot, and networking people are kind of, well, to some extent, security people as well, but we're the people that know about flows and about all these other applications and things that are going on, storage and Wi-Fi, and just across a whole set of technology boundaries.

And if we don't have visibility into them, then we get these strange trouble tickets that something isn't working right, and we have visibility on the network side, but without very broad telemetry, we have nothing telling us whether the storage array has gone bonky or something like that. And I think, my opinion is now we have more capable devices that can get us this telemetry too. If you think back to, at least back when I started in my career, I could drop a router with overzealous SNMP polling.

So I don't know if anybody else has-- Yeah, crash the CPU. I don't know if anyone else has had that experience, but I definitely have the bruises from those. Nowadays, there's dedicated ASICs that can provide a lot more data, and the architectures of modern network equipment, and the horsepowers, the raw horsepower of a lot of the modern equipment can provide more data.

So I think as we've seen cloud scalers adopt and need more of this data to run these gigundous architectures that they're responsible for, I think we've seen the needs for them to have more data and be able to ingest more telemetry. So all those capabilities have grown over time. So seeing that come down to the enterprise, and being able to ingest that data, normalize it, and have it be useful for the rest of us, because cloud scalers are, a lot of them are solving unique problems that the rest of us don't deal with, typically.

Being able to adopt those, I think it's time, and we need more of that. You triggered a thought about massive amounts of data. Namely, I'm thinking of the company Kentik, which uses cloud storage because their customers are just storing amazing amounts of data across a whole lot of fields, and their reason for existence is to allow monitoring of mission-critical, business-critical application performance, among other things.

A lot of data to sort out, however. I think it's interesting that you bring this up because this is a conversation I was actually having with Andy Laptev over at "The Art of Network Engineering" podcast, who's actually going to be a presenter at Networking Field Day. And one of the things that we talked about was this idea that we can do so much more with networking equipment.

I remember taking the Novell TCP/IP exam, so I have now officially dated myself, but one of the ways that they were describing routing protocols in that very simple test was OSPF is a, quote unquote, "expensive routing protocol," meaning that it requires a significant amount of processing power in order to run. Because back then, as I've jokingly said, routers had the CPU of a pocket calculator, and we were lucky to be able to install even a portion of the forwarding table in the CPU. And for a server that was already tasked with doing eight other things, RIP is stupid simple to run, and it doesn't incur a lot of overhead.

Now, today, thanks to the advent of things like ARM processing with DPU offload and DPDK programming and things like that, I have multi-core CPUs that are quite literally sitting around doing nothing at idle, which means I can collect and dispatch data a lot more efficiently than I used to before, which means that I'm now starting to think about the richness that I can provide from these devices to be actionable. And it's also because, like you said, of hyperscalers. One of the things that I learned many years ago is that Facebook had developed their own switching infrastructure running on Linux, partially because they had developed a telemetry system running on Linux, and they wanted to be able to integrate the data that they were getting from their network with the data that they were getting from their servers.

Because as we learned a few years ago when Facebook took a nosedive, they were using networking protocols to provide availability. " They didn't trust it as the sole source of external routes, but they were doing other things with it because they could, because they had built an infrastructure around that homogeneity from a processing system. So as part of what we're able to do now with telemetry, because we stopped building purpose-driven devices that were running custom silicon, and we have effectively built everything on the same ARM processors from organizations like Broadcom, where everybody feels like they're doing the same thing.

Well, don't forget that... And sorry, if you're implying this, let's tease it out a little bit. Remember that that separation of control plane and forwarding plane has been a huge enabler here.

And we can keep the hot stuff hot and the cold stuff cold. Forwarding ASICs can do the one gig and the 10 gig and the 100 gig and the 400 gig forwarding, and the control plane just needs to worry about the right protocol packets to deal with. And that architectural separation has led to a really interesting talk from a Google researcher just a couple months ago at NANOG in February, where he's basically putting an instance of the SDN controller on each router.

So a copy of an SDN controller on every router, basically trying to do what IGRP and EIGRP tried to do 30, 25 years ago. More information on how I should route stuff across my network, with lots of inputs, including this telemetry stuff that we're talking about. And it's important to realize that when you think about those kinds of routing protocols, we had to make massive adjustments to the way that they operate to get them to do the things we want it to do.

Anybody who's ever taken a CCNP or CCIE level test knows that there are certain conditions that would cause routing protocols to not be able to run the SPF algorithm in enough time to do things. That's why EIGRP has feasible successors that are ready to be installed in the routing table. It's why, according to my good friend Narvik, there are only a couple of things in the EIGRP equation that are turned on by default.

It's because it costs so much resource to run those things that we couldn't do it. But now, talk to anybody who works at Facebook, and they're like, "Why is the hello packet 512 bytes? " But now they're looking at the possibility of ingesting all this additional data to do so much more that's actionable.

But at the same time, we have to remember that processing is not a free action. This is not like a fighter's bonus action in D&D. You have to make the effort, which means you are adding overhead.

And so is that adding of overhead a free action because we have so many spare CPU cycles now, or do we still have to plan for all of this telemetry and not integrate it into all of our decision-making processes? I think overhead is a good word for it, but I'm thinking human overhead. You can get lots of data now, which is a good thing, but it can sit there and fill up your disk and create expense unless you have some way to monitor it.

And I've sort of experienced this a couple times, fiddling with an Elk Stack and fiddling with the Kentik product, of okay, there's all this data out there. What do I care about? Who's going to tell me what statistic to apply to it?

How do I munge this data into something that's meaningful in terms of reporting? And that's kind of a challenge. Networking people are good at a lot of things, but I'm not sure many of us are data scientists.

And enter AI, right? I know Tom's probably- Yes ... " minutes past the hour.

We made it 20 minutes. Yeah. No, but really, if you look at the promise of AI, and that's being able to sift through gobs and gobs of data, but at the same time, there's a consideration of context when it comes to AI.

So you can have too much data, and you can overload the LLM with it. So you're going to have to somehow distill down all of this telemetry data into something useful, even for AI, or whether you still have humans looking at things in the loop. So you can have too much data.

So yeah, to have a counterpoint, there is such thing as too much data, especially if it's not structured properly, it's not refined properly, it's not consolidated, and distilled down in any useful way. But there's an important distinction here, right? That I think you're both moving toward.

I'm not storing all of this data on routers. I'm actually not storing the majority of this data on routers, whether it's Kentik or other emerging solutions like SelectorAI, right? There are offline systems that are putting all this stuff in one place, whether it's streaming telemetry, SNMP, and other sources of info, so that LLMs and other AI tooling, and even good old statistical regression, can operate on the data in one place.

And so there's this necessary complement now of, you want more telemetry? That's great, and you can use it as a superpower, but I'm not going to be chunking through all the data on devices in the production network. There's a learning curve there.

I think you need somebody who has the time to experiment with pulling statistics on the telemetry, looking for anomalies and stuff like that, and then you can start looking at correlating anomalies and some of the other things that some of these companies are playing. Or you can bring in a company that has already been there and done that, although I get the impression one of the first questions some of them, like Selector, asks is, "What do you really care about? " Always a good place to start.

But that's been the problem for the longest time, right? Is sampling. When you're dealing with gigabit links and 100 megabit management interfaces, I can't send every copy of the telemetry of every packet over that wire, or I will clog it up.

Likewise, the collectors that were running on 7,200 RPM hard drives could not process the data fast enough, and so we resorted to sampling, right? Like every 10th packet or whatever the sample size was. But today, we have the same problem with more zeros after them, right?

Because we've seen with the rise of things like RAL optimized architectures, that when you're pushing terabits of data around with tail latency being a super critical statistic, you can't wait for that data to come back. You can't make decisions based on things that are even three or four seconds old because that could cost you hundreds of thousands of dollars in GPU idle time. So can we even use these types of telemetry offerings to make better networking decisions knowing that we're still in the same boat where we can't get enough data fast enough to process it to be able to make these decisions in the hope that we're not going to get fired because that fancy GPU cluster is just sitting there spinning fans.

And spinning disks, when they used to spin, right? I think Pete has a great point from the data science perspective. Do you remember before the current AI moment when data science was the next emerging career, and it kind of got eclipsed by this Artemis II that is AI moving in front of it?

Now, I think, Pete, what you're calling data science is really going to be data engineering, right? Where I don't need all the data science skills, but I do need to make sense of the piles and piles and piles of additional telemetry that I'm collecting, in a way that might not be fundamentally going after the basic science of it, but the engineering principles around certain data sets, and how do I use them for actionable intelligence. Javier Antich, who's leading at Cisco in some of the large-scale adoption of this, has written an interesting book, but he ties a lot of it back to, well, there's AI, but a lot of this we may need machine learning or other approaches.

So don't just frame the data problem as being solved by AI. It's a whole slew of techniques where we try to apply the right tool as the data comes streaming in. It's not going to be real-time, so maybe we're backseat driving, so to speak, but real-time is going to have to be handled on-chip and adjusted.

What we're looking for is a smoking gun saying we need more capacity, or we need to change something about how we built the network. I hope that we can provide that, but I also know that we have reached a point in our society where there's an immediacy bias, right? I don't care what it is, I want it right now.

Like the people that order everything from DoorDash because they can't be bothered to go out and get it, that want to know up to the minute, up to the second, what's going on in the news, or instant scores delivered to your mobile device via push notification, and now you're telling them it could take an hour for me to just sit to figure out how to write a... do any analysis on this data. No, I want to log into the dashboard right now, and I want to see where the problem areas are, and I want to fix them.

It's that whack-a-mole problem that we've always dealt with, right? It's like, oh, well, we need to increase the bandwidth between these two cross connects, and then when you do that, you're going to surface more problems over here. And then you're constantly going to be chasing that rabbit all the way through your network and constantly upgrading things that really honestly don't matter in the long run.

Because I think what the value of having more telemetry data is, is that you are able to uncover the problem areas before you fix the wrong symptom. So, oh yeah, we're seeing slowdowns over here, but it's not because of the link between these two devices. It's because the amount of data that's being pushed between these two servers is more than we should be pushing.

So we should work on the algorithm that decides which data needs to be shared, not adding another colo link that's going to cost us an extra $10,000 a month to plug in two routers. People get so wound around the axle of, got to solve this problem right now, got to fix this problem right now, got to keep these things humming, got to keep this going. Not every problem in networking is an F1 pit stop.

Some of them are things that we have to solve in the lab with computers and modeling and stuff like that, and we can't just throw things at the problem until it goes away because that's how you get a Franken-car, right? Well, I see that as part of the virtue of getting a lot of data is, if you can correlate stuff where you had spikes at some time or some phenomenon that occurred, then you have possible causes. And the more data you have, oh, okay, the application went slow, but here's what was happening at the same time the application went slow, the better you're going to be able to solve the problem quickly.

Right. Right. I do think, and not to make any specific technology provider plug here, there are systems evolving that aren't just using LLMs, right?

But if you look under the hood, they'll use machine learning for pieces of the problem, they'll use on-chip processing for other pieces of the problem. They'll still use statistical regression for other pieces of the problem, and LLMs for not just finding the needle in the haystack, but the needle in the needle stack. And I think that's the problem with 10X-ing or 100X-ing the amount of telemetry we have to plow through.

It's beyond the limits of human cognition, right? We can use LLMs that are really good at pattern matching to be able to get to the relevant info fairly quickly. Yeah, and I've seen some interesting use cases for LLMs, like overnight processing.

So, when operators come in in the morning, they have this detailed report. Here's what happened last night while you were sleeping. I think that that's a great use case because it has all of that time to crunch through all of that data, and again, it comes to that distillation of taking all of it and turning it into small, useful nuggets that people can use.

So, I think of applications like that where LLMs are particularly useful. Another thing that's sort of outside that box that I'm thinking of from experience is when you are trying to scope out an application, be it for security, performance, or other purposes, quite often it was maybe developed outside by consultants, and they threw it over the fence, and people inside are trying to support it, but people don't know the flows. And if AI can just tell us who's talking to who, where are the boxes, and piece together some of that basic information, that's not classic telemetry, but it's just kind of, what's going on here?

Obviously, the answer is yes, we do need more data, but we have to be selective about how we do it. We have to make sure that we're providing the systems with the right kinds of data. Anyone who's ever deployed an intrusion detection system knows putting one on everything is not the answer, because you're either going to get completely overwhelmed that you won't be able to do anything, or you're going to get so annoyed you block all of them.

So if you had to give advice to the networking team that's about to do this to increase the quality of the data that they're getting, what's one thing that you recommend that they look at in order to make things better overall? I'm going to use this to maybe reinterpret your question and say, I think you have to look at the system holistically. And I'll go back to Pete's comment about the network engineers tend to be the jack of all trades, master of none, and we have to understand application behavior, right?

Because that's what's impacting networking performance, right? If we didn't have any users, by the way, the networks would be great. But I think if I boil it down to one thing to really answer the mail from Tom here, think about how all the pieces connect together, application performance, security, network performance, and any other adjacent information.

And I think AI is bringing a set of tools that are going to help us bridge those silos and look at telemetry and performance information from those domains that have been necessarily treated separately for a very long time. That has implications for how people handle... You need to make contacts at work that are outside the networking group.

IT in most companies is kind of stovepiped around specializations, and maybe that has to change. Or at least make contacts and be able to sort of cross the boundary and speak a little bit of the other guy's lingo. I would encourage one to make sure that the data they're getting from the telemetry they're using is clean, and you don't have things like duplication.

So I've actually been in situations where you're double counting things, so all your numbers are off because you're like, "Wait, this is not adding up to what I'm actually seeing on these interfaces. What's going on? " Really Trust but verify.

Make sure that the data that you're getting is really what you expect to, and make sure that you're using it correctly. Been there and done that. I was helping a company try to analyze a mainframe-based major application.

They'd spent a million dollars on upgrading this application and wanted to demonstrate to management that it was worth it. And so I was working with people who weren't fully aware of all the flows, and we went to scale it up. We got some pretty good data on the flows at a certain test level, capturing packets and so on.

And then they went to 10 times, but the amount of traffic didn't increase. And it was kind of like, something's caching or we're missing a flow here. So comprehensive view is always in the back of my head that people aren't necessarily aware of all the things that go into applications.

And then, yeah, if they're not aware of what could be making it slow, then sometimes troubleshooting can be a real hassle. I will wrap this up by saying, I think it's very important that people understand that while more data makes everything better, all the data doesn't make it the best. You have to be very selective about what you're looking at, and as you've all brought up, have a holistic view, be very careful about screening out duplicates and things like that.

Because even the best system in the world can only operate on what it's been given, and if it makes the wrong decision because it has an error in the dataset, then you're going to be chasing a lot of ghosts through your system before you finally figure that out. And the way things are going now, that could be something that has you creating a resume-generating event. I want to thank you all for joining us today on the "Tech Field Day" podcast.

Before we go, where can people connect with you and continue the conversation, Jason? Yeah. So LinkedIn is a great place.

I typically engage there. I'm most active there. com to read more, a bunch of blog posts and more about what I do there.

I'm on LinkedIn. Should be easy to find me there, and I stick with LinkedIn because it seems to have less noise than a lot of the other social media. com.

All right. Well, thank you all very much for listening to this episode of the "Tech Field Day" podcast. If you enjoyed this discussion, please subscribe on YouTube or your favorite podcast application so you don't miss an episode.

And we would love it if you would give us a rating and a review and leave a comment because all of those things help us reach new audiences. This podcast was brought to you by Tech Field Day, the home for IT experts across the enterprise, which is a part of the Futurum Group. com/podcast or find us on Techstrong TV.

Thanks for listening and we will see you all next week.