Assessing the Current State of AI-driven Packet Analysis with VIAVI

High Tech Field Day Extra and Cisco Live. We are back to talk about assessing the current state of AI driven packet analysis. Uh, this is part two of our discussion.

Uh, I am cobbley with vavi Solutions. Uh, the little part of Vavi Solutions that I'm a part of focuses on network performance and threat solutions. We take packets and flow data off the wire.

We analyze that data for you, tell you about things that look sketchy and help you walk through the process of understanding and resolving them as needed. Let me introduce Chris Greer rather. Let, let's let Chris Greer introduce himself.

Hi, tech Field Day. I am Chris Greer. Uh, I am, uh, what you might call a, a packet geek.

I like to look at packet traces with wire shark and help my customers, uh, isolate and resolve issues at the packet level. Uh, when I'm not doing that as a consultant, I'm also training on Wireshark directly with the Wire Shark Foundation and helping to get people trained in understanding Wireshark. Uh, just last week, Wireshark released their Wireshark Certified Analyst Certification.

Uh, so I'm one of the, uh, certified instructors for that. So excited to continue to build and assist the Packet community. Alright, thanks Chris.

For those of you who watch part one, and we're wishing there was more, Chris, we're gonna take care of that request right now. So, uh, in the first part of our discussion, we talked a lot about how, uh, current LLMs, popular LLMs will handle PCAP data. What we want to talk about now is some of the practical challenges of the P caps themselves, because what many of you know all too well is if you sit down and just start capturing packets without being very, very tight on what you want to capture, uh, those p caps can get incredibly volumous in a big hurry.

And then, um, you know, to use the, the metaphor that Chris alluded to earlier, you've got a haystack the size of Cleveland and the needle is still the same size. So what do you do to make that entire exercise more efficient and the outcome more successful? So, Chris, why don't we, we bring you up here and have you talk about that a little bit.

What are some of your best practices? What are some of the things you recommend when it comes to managing large P caps? How do you avoid having a large P cap in the first place?

Yeah, that's a, that's always a goal if possible, is to getting that packet trace as small as possible. And then I have less to look through. Problem is, is that you start capturing and you start wrapping into the Megs gigs, hundreds of gigs of data very quickly.

I think, as we all know, speeds and feeds increase and you just start a capture and it's tremendous the amount of stuff that's in there. So now my question becomes, okay, what matters? So to towards point, you know, that huge, huge haystack, how can I get it down to more to, to be something more manageable?

Uh, something that, so if a client of mine calls me, says, Hey, they're having a problem, my first question is, okay, what do you have there? What type of, uh, packet capture solution do you have? Like, what are we working with?

And so if they have a purpose-built solution, wonderful, we can go back in time. Now I wanna know, when did the problem happen? Who was involved loosely?

What application were they experiencing with it? It gives me some abilities to go set filters and extract that traffic problem is that they don't have that, right? So then we have to set up that capture.

Uh, what I'll do if they don't have a whole lot, uh, of hardware there, what we'll do is we'll start up wire shark and do a rolling capture. So it captures a gig, saves it, captures a gig, saves it, captures a gig, saves it until the problem occurs. And that way, at least after the fact, okay, so Jim says, Hey, it happened, and then we can go stop that capture and then at least have a window of a few gigs to, to sort through instead of a terabyte of of data.

So that's my first goal, is to try to get those package traces as small as I can. And with Wireshark too, when I'm doing Wireshark analysis, uh, really I don't wanna open up a PCA much larger than a gig. That thing can slow down.

Uh, you set a few filters and every time it's gotta run it through its dissectors and try to rip through that data. Again, it can take minutes to apply that. So, uh, just to save that for myself or another way, if possible, what I'll do is I'll use the command line tools to say, Hey, here's a packet capture.

Just rip out that conversation. Going back to the AI conversation we had before, uh, if, you know, help me, help me to create that script, help me to create that command to rip that out quickly. But yeah, managing large P caps is absolutely a challenge.

So the goal, try to get down to as small as possible with filtering and even the way that I capture it, uh, from the beginning. So that's always a goal and it is a challenge. And sometimes that can be pretty significant in a large data environment.

And I run a hypothetical by you. Let's say that we're looking at, you know, a complex multi-tier app. Maybe there's some pieces in the cloud, maybe there's some pieces on-prem.

Um, how do you know that you're capturing in the right place? How do you know you're capturing root cause versus maybe a symptom of root? Cause That's a huge challenge.

As soon as you say multi-tier, as soon as you start going into systems that are interacting with each other to deliver the application, huge challenge on the wire. Uh, you know, you are, you are, it just takes the needle in the haystack to the next level. So now I'm not just looking at a needle in the haystack on a front end, on a, on a, on a vi or a load balancer.

Now as I'm digging into that application further, it's just now those haystacks just get bigger as I I traverse a tier. So, um, huge challenge. What I'll do though is I just try to key on anything I can front to back, be it a conversation, be it delay.

Like for example, uh, earlier we were looking at a pretty significant delay, like a hundred seconds on that server. Okay. And now I at least have a hundred second window look to look at what else is that machine doing within that period of time?

Uh, so yeah, manually Oh, absolutely. It's a, it's a challenge. Okay, cool.

So if you're kinda wondering why, why is Avi up here with Chris Greer, who is, you know, known for his his packet expertise? A lot of what we do is helping our customers get the right packets at the right time from the right place so that they can be as efficient and possible as possible in their analysis. Uh, you know, we have the ability to collect literally petabytes worth of information, but no one's gonna analyze but petabytes worth of information.

And so our job isn't just to take all that information off the wire. Our job is to help people figure out what it is they ought to be looking at. And so we're not trying to replace packet analysts, uh, we're trying to give packet analysts an opportunity to be rock stars, an opportunity to thrive because they're looking at the right information the first time instead of staring at that massive haystack and trying to figure out where to start.

So to that end, forgive me, I wanted to just explain how we go about finding the right packets and I'll, I'll just walk you through an example. This is actually something that happened a few years ago at vibi and we had to use packets to get to the root of the problem. Uh, we started off with a high level service performance view that told us our Salesforce application was really struggling.

And the way we knew that was because of something we do. It's a patented machine learning driven thing called end User Experience scoring. And so what we do is every socket connection starts off with a score of a perfect 10.

But as we see any inefficiencies begin to manifest themselves, we start to deduct from that perfect score and we then assign those deductions to a problem domain. Is this a network thing, a client thing, an app thing or a server thing? And so in this case, the deductions were largely in the category of server.

And then we try to be more specific as we can. In this case, we're saying, Hey, this server had some significant long responses to requests. Okay, so we know we've got an app problem, you know, the map, the dashboard tells me how pervasive it is, how severe it is.

So I'm gonna prioritize this and go dig into it. Once we move past that top level, we start getting down into what are the worst performing servers? And here we saw two servers that were, uh, performing poorly.

Let's go dig deeper. And the first thing we want to do is understand the context that these servers sit in. So we open up something called an on-demand application dependency map.

This is based on the real packets. We pull off the wire. So this isn't based on a document someone created six months ago that may or may not have been right six months ago, and is most certainly not right today.

This is actually what we're seeing in real time. Now, what we observe on the left, there are five clients that are accessing Salesforce and all of those lines are red. If we were to hover over them, they would all have low EUE scores.

And so our thought process initially might be, well, I gotta call Salesforce or I gotta go figure out, you know, what's wrong with that piece of this service architecture? But we see another clue towards the right. And that prompts us to wanna look at the rest of this application architecture because we all know if you're dealing with a multi-tier architecture and you've got problems on the back end, they are not going to fix themselves or get better by the time you reach the front end.

com, obviously public cloud app. Salesforce had an integrated plugin called quote Edge. Quote Edge is a third party plugin that lets that lives in a different cloud.

Quote Edge was dependent on information that AVI would provide through our own on-premises database servers. So on-prem database servers being queried by quote, edge being queried by Salesforce being exercised by end users. That's the full picture of the service, uh, domain that we're looking at.

So now on the backend we see yellow lines. That means red flags. So we start diving into these individual servers and we see, yep, uh, these backend servers are having some issues.

Where do we want to go next? From here we're gonna use something called connection dynamics, which is essentially a way of saying, take me down to the individual socket connections and visualize them for me. So here we see a list of all the socket connections, we can see, uh, each connection, how it was scored.

I just picked the one that was the shortest duration 'cause it was bad enough, uh, to look at. And it's easier to look at a small conversation than a large lengthy conversation. So when we dive into that one socket connection, we see our EUE score again, we see, uh, it's still the server domain that we're looking at.

And over on the right we see sort of an SMS style bounce back and forth Question. Yeah. Hi, Josh from Diversified.

Uh, where is your, your point of entry on capturing this information from an architecture perspective? Is this something that's on each application server through your multi-tiered application? Are you looking from a packet duplicator from the network?

All of the above. If you gotta get some context on That. Absolutely.

Thank you for stopping me there, Josh. I appreciate that. In this particular case, our, our frame of reference was our AVI data center.

So we had our packet capture devices, appliances, some of you may have heard of Giga stores were deployed there in our data center and we were capturing all the data coming in and outta the data center. Now it's really dependent on each deployment where else you might deploy. Uh, you might have said, well, we really need to get some sort of packet capture capability in those cloud environments.

com, they're not gonna let me put packet capture out there. So what else can I do? Um, we don't do anything at this point that's agent based.

So we're not plugging anything into user agents or server agents. But we have found if you get good instrumentation at those aggregation points where the traffic is coming together, you can understand what the end user experience is, even if you're not on that end user device. But one of the things we do, I really appreciate the question is sometimes packets aren't even the right answer, right?

It might not be feasible to have packet capture everywhere you, you might want it. So then what else is there? Can we use flow to our advantage?

Is there some other data source that we might use, uh, to help paint this picture? So if you look at the conversation, okay, that's cool, pretty visual, but we said this is a server issue. So we just wanna look at the server side of this.

So we filter everything else out, and now as we start scrolling down through the server responses, we see, hey, there's one particular server response that is, um, obviously not it, it's not what we need it to be, not what we expect it to be. So this is as far as we attempt to take you in the workflow. From this point, what we've done is we've gone from, there's some Salesforce users complaining, or our dashboard tells us Salesforce looks like it's having issues to understanding the entire service architecture, to understanding it's a server problem, to looking at one conversation, one response time.

And guess where we're gonna go from here? A little button at the top is packet export, there's your P cap. But now when you generate your P cap, you are just looking at this one small connection between this one client and server.

Everything else has been set for you. The filters have been set. You've now got a small you manageable p cap that you can use to continue your analysis.

So Chris, I don't know if you had any thoughts on whether or not this kind of approach aligns well with some of the challenges you were just discussing, or any other thoughts on that? Oh, Absolutely. I mean, just getting to this screen with this number of packets, what do we have word?

Uh, just a few hundred I think on the bottom. Or is it, yeah. Can't read.

It's too small. It's too small. Yeah.

Get, even getting to this point and that type of environment, I can't tell you how long that could take with just Wireshark just on the wire. I mean, that's, and, and that's, you know, having the right capture in the right place with the right filters can still be a, a tremendous challenge. So, um, what I like about this though is that you saw the, the, the text message conversation on the previous screen, you know, the pretty charts and screens.

I mean, with me in my head, you know, I'm a packet guy, so I'm like, okay, that's pretty, that's great, but I still want the data. I still want to see those headers. I wanna see what tcps up to what the health is.

Uh, so for me, I mean, what I like is I can just do a, an export and then get right back into Wireshark. So high level context, getting me down, filtering me down to the problem domain back in time, export to packet capture. I mean, that, that's a, it, it, it would save me a ton of time.

Appreciate that. Chris. One of the things we heard loud and clear from our customers, you know, we, we came up with this EUE thing and, and people were like, that's nice, but at the end of the day, I want packets.

A lot of the users of our tool are very focused on packet level analysis. And so what they said is, you know, by all means keep innovating. We love it, but make it as easy as possible for me to get to packets because at the end of the day where I'm, where I'm gonna be in many cases is here.

So just help me get here with context. Help me get here efficiently and help me figure out what I'm looking for when I get here. So, uh, just to kind of recap what we talked about here in, in part two of our conversation, uh, I borrowed this from, uh, it might have been, uh, Mike Panke that I heard say this actually don't try to move a mountain with a shovel.

Uh, you can do it, but it's gonna take a while and, and, uh, you probably don't want to, or, you know, the metaphor we've been using with the haystack speeds and feeds aren't going down anytime soon. We're gonna be capturing more stuff that's more noise in most cases when you're looking for a very specific problem. So, uh, figure out how to eliminate as much of that noise or as much of that hay as you can.

And then, you know, the reason why we are here is to, uh, stress the point that you want to leverage automated analysis to the extent you can to analyze all those traffic and all those flows so that when you need to get down to the ones and zeros, you know, you're looking at the right ones and zeros. So, um, that concludes portion two question thought. Yeah.

Um, so I, I've, I I am trying to digest and come up to speed on, on everything that you're doing here. Sure. And it, it, it sounds like you're feeding, um, the application, uh, packets in order for digestion.

Correct. What happens if you don't feed a packet? What packets, what happens if you feed it layer two frames from like a do 11 conversation?

Is it intelligent enough to understand wifi and wifi centric problems in addition to application layer problems that are gonna manifest in packets? So our collection point is typically going to be, uh, a packet broker that's just forwarding us, uh, the, the stream of packets or, uh, a span port or a tap or something along those lines. So we're gonna see data presented that way, and we're gonna, if, If we take a layer two frame packet frame capture from a do 11 adapter as a P cap, can we dump it into it?

And then do you understand that that's layer two frames instead of applications? I've not run into that particular scenario. I don't know if you have any thoughts on this, Chris, for Sure.

Wireshark, yes. But observer, I don't believe so. And I think That would be the, yeah, that would be the crux of the question for you.

Yeah, Yeah, yeah. Wire shark. Yes.

And, and one reason why, um, you know, I get asked that question a lot, even as a packet on a list, like, Hey, you know, what do you do for wireless? And, um, my response often is, you know, there's a reason why Wireshark starts with wire. Fair Enough.

Because if I'm in a wifi environment, other tools that do radio, do signal noise, do that stuff well, um, in my experience when I'm doing pack capture in a wire lens environment, they, there's so much chatter. So not quite Avi's Forte, That's not the wheelhouse. Okay, Cool.

Yeah. And if I go back, It's a great question. Yeah.

I love it. I'm making, it's, it's making me think, if I go back to war cop's question, you know, where are you deployed? We're typically deployed in the data center.

And so what we wind up seeing is the output of the conversation that started with a wireless device, but then got framed up and sent over, you know, to the server. And we're seeing it as it comes into the data center and the response back from the server, you've Already abstracted out retries, you've already abstracted out noise floor issues, you've already abstracted out all of the layer two intelligent stuff that we need to know. True.

And you're looking right at that Point, it's gonna look like a client issue. It's gonna look like A client Issue. Yeah.

But you're not gonna know what it is. Right. I'm assuming if, if you're not looking at the L two information.

Yeah. Valid, valid observation. We're not trying to get that information from the wireless infrastructure, but thank you.

It made me think about that one, Um, Glen story here from Red Eye. So to kind of piggyback on yours, you know, you were talking about can you put a P cap into, into the system, you know, from a a third party source, you know, we often run into it where, you know, we're troubleshooting the issue from two different ends. So we may have the, your device in the center and we get a packet capture from the client that's out in the field somewhere.

Mm-hmm. Is there benefit to be to bringing those to two together in your system? So I'll, I'll answer from the AVI perspective and you can add whatever you like here, Chris.

So right now, um, we don't have the ability to have you introduce a P cap and then run through our analysis and produce what you saw here. We have to capture that. Um, and we have to analyze that, you know, as it's happening.

And I'll just pause there and say, but stay tuned because it has come up that both for performance use cases and threat use cases, it could be very beneficial to be able to just ingest a P cap that was captured elsewhere and do some processing, do some analysis. Uh, It's, I think one of the things I've seen in the past is when you have it from both sides, you suddenly notice that there's certain packets that didn't make it all the way across the internet to the other side that you're capturing from. So, yeah.

Well, frankly, your use case where you're feeding the LLMs is precisely it. I've got a P cap, I wanna know what's inside the P cap. Yep.

How do I do that with avi? Yeah. Like, like that's a very valid use case.

Yeah. Now we can, um, if you, and again, it depends on the customer environment. We can deploy multiple giga stores at multiple points across a global enterprise and then correlate what those things see.

Right? So there is the ability to say, I'm capturing packets from four different places, bring that all together and provide me with one unified view. So, but again, it's, it's predicated on the fact that we are collecting data for our platform, from our platform and not just a, a pcap that somebody, somebody grabbed somewhere else.

Well, Eric Stewart, university of South Florida, as you indicated, sometimes people are like, we're in the cloud, what can you do for us? Yeah. You kind of have to go not a lot.

Yeah. In case they can always cap on the, on the, the VM itself. So let's, let's talk about That to you.

So, Uh, and some of the brokers, right? Yeah. Have cloud appliances.

So yeah, if you want to take a broker, you can drop it into the cloud. You have a third party device that can, that can give you that p app information. But the native would be a challenge, I think.

Yeah. This is a great point that, that you both brought up, is, is what do we do in the cloud? It's funny, we have the state of the network survey we do every year, and we asked one of our survey questions was, um, are you collecting packets in the cloud?

And it was something like 79% of the people said, yep, we sure are. But then when we broke it down by function, the vast majority that said yes, we're at senior levels of management and the people that said, no, we'll just leave it there. Right?

Yeah. So whenever I, that sounds about right. Whenever I have to troubleshoot something that's in the cloud, we are using a Palo VM and we don't do a P CAP on it, we just look at the logs and we can usually get what we want out of the log.

Yeah. So there's, there's a couple options here. I love this topic.

So let's start with A-W-S-A-W-S, um, makes it pretty easy. There's a, a native, uh, packet mirror function that you can use. So you can capture packets and then you just have to worry about how much it's gonna cost you.

And yeah, that's, you know, an issue we all understand very well. Um, VPC flow logs pretty easy to get out of a AWS environment. So that's another possible source of data.

If you can make those VPC flow logs look like bi-directional conversations, uh, we can bring them into the mix and that's what we do. So AWS you know, we, you can go to marketplace and download an A MI and, and put a, a virtual giga store in the cloud, essentially. But then if you start moving outside of AWS and you get into Azure, um, you know, Microsoft put out their, their, uh, v mirror capability a while ago, then withdrew it and said, just kidding.

And then, you know, you have to rely on third party packet brokers and, you know, you have to put a, a gigamon or something out there. Now they've recently, within the last month, reintroduced a traffic mirroring capability as like a public preview. I forget the term they use for it, it's still not like a fully GA thing, but it's at least they're heading down a path.

It says, we're gonna let you get packets again. And they threw out NSG flow logs and are now going with vnet logs. So they're changing the way they look at, at how, uh, logs describe what's happening in the cloud.

And then with Google Cloud packet capture, I think not impossible, logs are really hard to figure out because their VPC flow logs are sampled, and then they allow you to establish a second sampling interview interval. So samples of samples, which scares me as a data source. So I'm wondering how much integrity is left in the data.

I'm sure there's a reason they did it. People smarter than me built that. But I have to worry, especially in a multi-cloud environment over here, I'm gonna get packets from AWS over here.

I'm gonna get VPC or, um, a vnet flow logs from Azure. And over here I'm gonna do samples of samples from Google Cloud, and I'm gonna try to pull all this together. Eh, it's not exactly seamless.

So some real world challenges, and that's why we all get paid the medium bucks to try to try to sort all that out. Chris One, I have one more thing about the where packets came from. I mean, one challenge I run into constantly with my clients is where did they get this from and how did they get it?

And if it's a high throughput environment and they say, well, I put Wireshark on a laptop and I put it into a span port and just hit capture, you know, and I'm thinking like, are you aware? Yeah. Are you aware of how little that can actually capture reliably without altering the delays?

Uh, for me on my YouTube channel, I have a few tests that I've done where I've just hammered traffic across the bow of a packet capture or a, or a laptop trying to keep up with it. And the throughput is I have to, I have to crank it down to almost 50 mega megabit, mega m megabits per second before it can capture everything. So that's another thing that's in the back of my mind when I'm sent a P cap, and it's not a giga store, it's not a purpose-built hardware device.

How reliably was it captured to begin with? Spanning expanding to a, a, a laptop's gigabit port when yeah, the feed is a 10 gig, and it's like, well, you're not getting all the data right to start with, much less what the laptop can write to its drive in the Process. And let's go ahead and span the whole vlan Yeah.

An R span ener, like, let's bring it in from everywhere and, and by the time it actually lands, yeah, right. I, I have so many blast holes in that traffic. I can't, I can't even make sense of it.

So it's just another reason to, you know, to have that all in one place. Thanks, Chris. Yeah.

So the numbers on the clock have gone from green to red, which probably tells us we're just about done here. Uh, thank you very much. We appreciate, uh, the audience both, uh, in the room and remotely.

Uh, hope you guys have a great Cisco live. Uh, we put a couple links in the presentation if you wanna learn more about what this guy is all about, um, some fantastic content out on his YouTube, uh, everything you wanted to know about Wireshark and things you were afraid to ask as well. And then, uh, also as Chris mentioned, he's got the new Wireshark certified analyst thing going now, which is, uh, which is pretty darn cool and probably long overdue.

So w with that, again, we'll extend our thanks to those of you in the room and, uh, those of you out on the stream, uh, enjoy the rest of the event. Thank you.