Enhancing Packet Analysis with AI – Smarter Faster and More Effective with VIAVI

Hello, tech Field Day Extra and Cisco Live People. Uh, today our discussion will be focused on assessing the current state of AI driven packet analysis using Popular LLMs. Uh, I'm Ward Cobbley with AVI Solutions.

My little part of Vast Avi universe of technologies and products is focused on network performance and threat solutions. Uh, what we do is we gather primarily packets and flow off of the wire. We analyze that data, we point out things that look suspect, and then we give you the information you need to further analyze it and troubleshooting.

So that's me, uh, to my left on the slide and to my right in real life is Chris Greer. Chris, tell the nice people about you. Thanks, ward.

So, again, I'm Chris Greer. I work directly with the Wireshark Foundation. org.

Uh, when I'm not doing education, I'm also doing packet analysis for a living. So people contract me, uh, to come in and help to consult with them to resolve gritty issues and look at strange behaviors on the network. So I'm here to support Ward at the state of AI and how I practically use it day to day.

So Thanks Chris. Let's get right into it. So our discussion today is gonna be broken up into two parts.

The first part of this discussion, and we'll talk about the current state of popular LLMs as it comes to analyzing P cap data. Chris is gonna give us some of his thoughts on how we can engineer our prompts, uh, to make the best use of LLMs as they exist today, whether it's deep dive information that we're trying to get, or more contextual information if we've been staring at the tree and we now want to go look at the forest for a minute. So, uh, what I would like to do for the next, uh, 15 minutes or so is let's put ourselves in the shoes of just the average person who occasionally has a need to analyze A PCA.

Uh, if that person is slightly paranoid, they may be wondering, are L LMS gonna take my job? If that person is less paranoid, uh, they may be thinking, how can I use ai? How can I use this today to make me better at what I do?

And so what we're gonna do for this exercise is we are going to take a P cap as seen here in Wireshark. That is only 107 packets. So it's a small little capture.

It is a very boring pcap. With one notable exception. There is a single server response, one conversation, one client, one server.

There's a single server response that takes 132 seconds. Everything else is pretty much subsecond, even sub millisecond. So what will these LLMs do when we give them this P cap and ask them for some information?

I am going to try to go lightning round through six different LLMs here very quickly. I will warn you. Now, I'm a big advocate of I think what Guy Kawasaki used to espouse, which is no more than 10 slides, no more than 20 words per slide, and no font smaller than 30 points.

I am going to break all of those rules today because I really wanted to give you a sense for the interaction with the LLMs. And if nothing else, these chat interfaces are chatty. In some cases, I have to take a 20 page Word doc and distill it down to one or two slides.

So forgive the word density, but I'm doing it to make a point. So first, let's start with Claude by Anthropic. Cool thing about Claude just accepted the P cap, raw P cap.

Didn't have to convert it to anything. That's the plus for, for Claude, I asked it, Hey, Claude, can you identify the client and the server in this conversation? Now to its credit, Claude, hold me, how it identified the client and server, but it didn't actually tell me who the client and server were.

So I might chalk this up to, to prompting, right? Because it gave me literally pages on how one might go about identifying a client and server in a pcap, but never actually told me who they were. Uh, when I asked it, are the server response times consistent?

It said, here's a handy dandy Python script that you might use to figure that out. So at that point, I was done with Claude. So we moved on now to sonnet four.

Uh, just did this, um, the day before, uh, I left to come out here and, um, didn't get to Opus, but got to sonnet. So we kind of continued the conversation with sonnet are the server response times consistent Note that it says no apparent gaps or delays in the packet sequence, no apparent delays, but then it provided its own fine print. It said, well, without using a special tool like Wireshark, I can't really calculate that.

So it gave us an answer and then explained why that answer might be wrong. Uh, and then it went on to recommend, go look at this in a pcap. So, uh, it was pretty consistent there saying, look, go use another tool, a purpose-built tool, uh, to go continue this discussion.

Fair enough. GPT time. Now for GPT, we did have to convert the raw P cap into, uh, Jason because it, it couldn't, couldn't just throw a P cap at it, but then we were off to the races.

Uh, when I asked it about the client and the server, it just went completely crazy on me in a good way at first. So let me tell you about the session establishment. Let me tell you about the three-way handshake.

Let me tell you about the payload exchange. I'm seeing very low latencies. Uh, here's your VAN IDs.

There's no fin or reset here. So this must be the middle of a conversation. I mean, it was just being the most helpful neighbor you've ever had.

Then it goes into, oh, so then I had to ask, but what about the server response times? Are they consistent? And it said, well, let's start at the beginning.

Packet number two, acknowledging the initial very fast. And then you see the descriptions. Well, the first act after receiving data, immediate reaction follow up act nearly instant.

It was just loving the response times, but it did flag one where there was a 64 millisecond delay. Okay, so notice the verdict, and this is word for word what it came back with. The verdict was consistent and fast server responses, but then there was a glimmer of hope here.

It said, would you like to examine some more packets and see if the 64 milliseconds is typical? I'm r okay, it's gonna figure this out now. I said, yes, it came back with a beautiful table, extra credit points for that.

Told me all about the response times. Gave me some notes. Verdict.

The server response times are consistent after the initial delay. Boy. Alright, so let's see if we can kind of prompt it into finding the thing we're looking for.

Could you graph it out for me? 64 is as bad as it gets here, and then it's nice and smooth. Now I'm gonna use a laser pointer for the first time this decade where the will it work on here doesn't work on there.

Nevermind. I won't. Where you should see a giant spike is between the 54 and the 85 on the X axis.

And it's just nothing. It's just flat. Okay, well maybe if we ask it to look at the client and the server, how about then?

Nope, still nothing there. Okay. Now let's ask it a very leading question.

We're gonna lead the witness. What's the single longest response time in this conversation? 64 milliseconds.

It's basically telling me, asked and answered at this point. So we're gonna try this one more time. We're gonna give it our best shot.

What can you tell me about Packet 71? Because this is the packet with the 132 second response time. And here's what GPT had to say.

Wow, it identified this as some kind of transaction on the nasdaq. It gave me the, uh, bid price and the ask price and asked me if I would like to, uh, analyze or visualize price movements. That's that's pretty amazing.

Yes, right, exactly. I don't know what we were trading, but it looks like I was about to get some inside information. That's what I'm saying.

Hell of hallucination. So I gonna say that's not even a hallucination, that's just wrong. I don't know what to call that.

Yeah, but we went from having a perfectly normal conversation where everything was on the tracks to, you know, we're just laying over sideways next to the rails at this point. Let's be honest, any conversation with Aaron? Perfectly normal.

You make a fair point there, Jordan. So that was the end of GPT. We had a good run.

Now I'll just say this in all fairness, right? We know this technology's evolving at a pace unlike any we have ever seen. And so literally we could do the same discovery and we could do the same presentation 90 days from now and it would look completely different, completely different.

In fact, I tested GPT this morning and while it still couldn't identify what I wanted it to find, I didn't see this craziness about the nasdaq. So these things are constantly moving. Okay, copilot, I almost gave up on copilot before I got started.

I tried to give it the P cap 107 frames. No, convert it to Jason, no, trim it down to 80 uhuh. 40, no, finally 20 frames.

And copilot said, okay, now I can talk to you. So that made me a little frustrated because, you know, do I really need help analyzing 20 frames if I'm good enough to say, I know my problem sits in these 20 frames. How helpful are you going to be?

But copilot turned a corner, it found the client in the server, it immediately said, 132 second gap, you ought to go look at. Okay, totally redeemed yourself. Then it gave me pages of information about session timeout, network latency, keep alive.

It suggested using Wireshark, it suggested bonus points. Check the server logs for database queries or processes running at the time of the delay. Just good practical advice.

You've got this big gnarly delay on the server side. Go see what the server was doing. Love this.

So I got over my initial frustrations. Then it asked if it would like me, uh, if it would like if I should ask it to generate a troubleshooting plan. And I was like, yeah, sure.

Uh, let's see. The plan gave me a step-by-step plan. Would you like to go deeper into any of the steps on the plan?

I mean, this conversation went on for almost an hour and it was providing me with incredible detail. So plus side, super helpful, maybe two helpful at times, uh, negative side only could analyze a minuscule amount of data. Gemini, OG Gemini here.

So, uh, Jason Import required, found the client in the server, found the 132 second response. But then that was kind of the end of the path. Everything I asked after this point, it never really got beyond this.

Hey, I see there's an issue here. So note the date. 5 pro preview comes out.

Let's see how it differs. So I started asking about the question between the two devices and oh my goodness, here we go. Told me all about client server ports that it were being used.

Mac addresses said you got a Cisco device and a VMware device here. Looks like they're on two different VLANs, but they must be really close together because they're extremely quick in responding to one another. Starts telling me about the data flows and how the flags are set.

I mean, it was just really providing incredible detail. But wait, there's more. It flags the significant time of 132.

It not only did that, but it told me which packets I needed to be looking at. So definitely gold star for that one. Then it got into this detailed breakdown of the, uh, the three-way handshake.

Then it got into the data transfer details. What happened once, uh, we started exchanging information, provided some expert analysis, if you will. Told me, you know, what looked good, what looked a little suspect.

Now it starts getting really impressive. Suggested some next steps. Hey, you might wanna figure out what this application is.

Here's how you do it. And then analyzed the gaps. Basically said, you, you need to go figure out what this is.

And then in summary, it said, the most notable feature is this, uh, very long idle period. It wasn't done yet. Said, oh, actually I prompted it.

How do you go about troubleshooting that big gap? And it said, okay, first you need to understand the context. Love that, love that.

So it gives me a description of what, what what had been happening. Um, told me what was, you know, working well, what was working as expected. And then it came to the conclusion the server was the one that eventually broke the silence by sending the requested application data.

Thought that was a good conclusion. There's more. So then it said, I'm gonna formulate some hypotheses for you on what to do about this.

And hypothesis A was pretty darn good. It even said, this is the most likely. It said, Hey, you need to go figure out what was causing, uh, this delay on the server.

It was busy or it was waiting for something. Then it came up with hypothesis B, CDE. Then it came up with troubleshooting steps and prioritize them, including go check the application logs.

That's an interesting one 'cause it didn't say server logs. In this case it said application logs. So Gemini decided this was more likely to be an application layer problem than just a problem with the server platform itself, which I thought was insightful.

Finally, more troubleshooting steps, and at the end, again, word for word the response. I got the strongest hypothesis points to an application layer, delay on the server investigating the server's application behavior and logs is the next crucial step. So that is where the bar is set.

Gemini really did an impressive job of going through and not only identifying the problem, but providing some decent practical guidance on what to do about it. So as I said, we do this 90 days from now could be a completely different picture. We know these LLMs are leapfrogging each other constantly.

Just ask Midjourney what happened when version of chat came out or GBT came out. Um, just to wrap this up, I wanted to give a shout out to a couple purpose built tools that show a lot of promise when it comes to PCAP analysis. Uh, we have first of all, selector AI's packet copilot.

Uh, this is, uh, a very promising looking way to analyze data. Gives you a nice breakdown of the pcap, uh, provides you with a lot of good options for interacting with. It can even draw you a little picture of the conversation.

So that's one to keep an eye on. ai, they're, they're here at the, at Cisco Live. And another one that I am personally pretty excited about this is Packet Safari copilot.

I just saw this for the first time last week and started playing with it a little bit on the plane ride out here. And I love the approach. Uh, it's basically you load up a P cap, it provides you with a wire shark like interface.

Uh, it'll give you an overview of what it sees, but then it's, you can just mark a range of packets and it'll go deep dive into that specific range. So if there's something wonky in a, in a P cap, you're just not sure what you're looking at, you can either call Chris, who'll be up here in a second, or you can go in and say, Hey, tell me about these 10 packets. And it will provide you with incredibly detailed insights.

So all that said, and that was a lot. Here's kind of the way we would, we would summarize this. First of all, the big disclaimer, please don't take your uh, p caps and throw them unsanitized up into the cloud.

That's just a career limiting move, right? Sanitize those things. If you're going to send them to a cloud based LLM at all.

Uh, Gemini Pro has set the bar. Most LLMs will not accept, uh, raw P caps. They will only accept a very small amount of data, and they'll only give you a high level summary unless you prompt, prompt, prompt, and prompt again.

So that's kind of current state. Now, I'm gonna bring Chris up here. This kind of leads us into, so if we're living with these limitations as they exist today, and you have to know enough to set a very specific set of filters and ask very specific questions, do you need ai?

It sounds like you kind of know what the problem is and where the problem is. So what role is AI gonna serve? That's what I want to ask you, Chris, is how can we use it in its current state to today to get meaningful, uh, analysis of a P cap?

Okay, thanks. War. And I, I, I think you'd all agree that was pretty interesting to see what the public LMS are able to do, but still where some of the limits are.

And that's something that I've seen as an analyst. Um, just needing to distill a packet capture down to the, to the size that I would need it to be to make it useful has been a challenge. I mean, a lot of times if I'm getting a packet trace down to a hundred packets, I'm usually on top of that a hundred second delay myself.

So, uh, that's, that's been one limitation that, you know, a as things get better, it's going to continue to improve and help me to find those things. But two ways. So I took the opposite approach when it comes to the slide material.

This is my only slide because they basically, I made up for him and I think you even built this one. So thanks. No problem.

Um, the basically two ways that I practically use it, the first is when I, I have a filtered capture and I'm interested in additional information about what I am looking at. Okay? So, hey, copilot, tell me a little bit about this ip.

What do I not know about it, this domain? Um, how about some GYP locations on, on this address or this name or this protocol? What if I see something a little funny?

I just want a quick summary of, Hey, I've never seen that type of behavior before. Tell me a little bit about this. Right?

So it helps to fill in some of that, uh, some of that information that I'm lacking, but I'm still on top of that, that issue that I'm looking for. Um, so that's one way. Another way when, uh, gaining insight is usually if I'm taking a look at, at packets, you know, they're in the many, many millions, right?

Or if I'm looking across hours worth of data. So I have to be able to get in and extract certain conversations from a mountain of hay, right? So the haystack's huge.

I'm looking for this section of information, be it a conversation, be it a protocol. Uh, so another way that it's been useful for me is like, hey, um, I'm, I need you to create this long thar command, uh, or even, uh, bash script that will help me rip through that data and pull out that conversation. So rather than me engineering that long, uh, command, you know, I AI can help me to do that.

Uh, so that's two ways. So one, if I, if I know what I'm, if I'm narrowing in on the issue and I want some additional context or creating, uh, some things in thar, um, I'll also add to that one more. Uh, so I am notoriously awful at, uh, RegX regular expressions in Wireshark.

Uh, whenever you create a filter and use the Matches operator, whatever follows that is gonna be a pearl compatible regular expression, that's where you can really go crazy in some of your search parameters. I really stink at those though. So that's another way where I can say, Hey, AI helped me to create a filter.

This is what I'm generally looking for. How would you use the Matches operator to help me to do that? Uh, so that's, I would say that's where we're at as far as its strengths.

Another way though, to gain context, I'm sure as you've seen before, if you've been in a packet trace, you're in there, you're razor focused, sometimes you're looking down to the hexa decimal level, it's hard to not be in there with a microscope. I mean, that's kind of the point, right? So what can happen to me is you're so far into that issue.

You've set a bunch of filters, you're looking down into that hex value that's down here in this, this type of field that's within a protocol. And sometimes you might miss an issue that's sitting one conversation away. What if I'm looking for a performance problem?

My customer calls me, Hey Chris, we got a spinny wheel. It happens when we click this button in this application. What's the issue?

Right away, I'm starting to set filters like, okay, let me get into this conversation, this port, this, uh, between these two endpoints. Let me filter into that conversation. But what if surrounding that there was some malware from down the hallway?

What if there was some weird art behavior? Uh, I might not see it because I just scooped that stuff out of there. 'cause I'm trying to get down to, uh, the nitty gritty on a certain conversation.

So another way that these types of tools have been helpful is, what did I miss? I was so razor focused. What else did they have going on in this package race?

So those are some of the strengths and for me, how I practically use it. Um, that said, uh, some challenges moving forward. I think the biggest one, uh, word showed you is just the amount of traffic you're able to throw at an LLM.

Um, and I mean, I think Ward, what was the largest P cap that you had tested? The largest one. The, uh, one of the products I mentioned there towards the end, selector AI will take a five, uh, megabyte P cap.

So that was the largest file size that I saw, but on the other side we saw copilot was literally 20 packets. So, And could your Copilot limitation be a licensing issue? You know, that's possible.

I think that's a fair question. Um, even if I set that aside though, and we throw out a copilot as an outlier, most of these things had heartburn once you started getting north of about 200 packets. But yeah, fair question.

And, and I found the same. It's just, it's such a at at, right? Right now at present, it's such a, a small amount of data that we're able to get insights for that.

Uh, by the time you're there in a P cap anyway, you know, a lot of times you're able to spot the problem. So, um, yeah, so that, that's, that's one challenge. And another, like Ward mentioned, uh, the sanitization, uh, it's making sure that that traffic, I mean, when we're doing p caps on the wire, we're collecting traffic.

There's a lot of, uh, sensitive data, right? So, uh, making sure that that doesn't get out, it isn't shared. And then that's, that's kept local.

I have to do a lot. There's a tool that's called Trace Wrangler. Uh, a buddy of mine wrote that and allows it you to strip, strip out payloads, change ips and alter and adjust that to protect the data.

So, uh, that's a tool that I have to use quite a bit when, um, I'm, I'm using an LLM to do some further analysis. So on that topic, I didn't mention that about Packet Safari. One of the things I liked about it is it gave you multiple options of sanitizing your PCAP before you upload it for analysis.

So it would literally allow you to slice at a certain offset, um, to, to slice off the payload. It would also allow you to swap, um, all the IP addresses and Mac addresses out. Um, so it provided some nice capabilities and it stopped you before you did an upload and said, which of these options do you want to execute on?

So I appreciated that. Yeah, so, so, um, Yeah, so I think, you know, just to kind of, to wrap up this portion of the discussion, we'll continue with, uh, with part two here in a little bit, um, kind of significant limitations we're seeing, but we all know this is improving. Like nothing we've ever seen.

Uh, I am fully convinced we're poised. We're at a point, like we were years ago when the internet was new and we didn't know what it was gonna be, but we knew it was gonna be something special. Uh, I'm waiting for the point where AI tells us what it's going to be when it grows up.

Uh, and I think it's gonna be, you know, beyond what we can even conceive now. So everything we see today is a point in time will get, will get better. Um, purpose-Built ai, these tools that I mentioned like Packet Safari, and, uh, the selector AI tool, I think are going to improve at a faster rate because they're being trained to do a thing.

They're not being trained by the universe to do all the things. And then finally, uh, prompt engineering is key. You, you really at this point have to kind of know what you're looking to get out of the LLM instead of just saying, Hey, is there an issue with this pcap?

Uh, we gotta know enough to be specific. The, the issue I kind of have with that is that you're kind of using your prompts to train the LLM in the process. Yeah.

It's Like, yeah, AB absolutely. And I use the phrase leading the witness. Yeah, it's, you're telling it what you'd like it to tell you essentially.

And is that useful? Is that good? You could argue that either in the long run It is, but in the long run you're, you're, uh, you're, you're building the product for somebody else.

Right? And I very much felt that way in this exercise. It was, it was really fun to geek out with this stuff, but I really felt like it was a lot of heavy lifting and weeding through pages and pages and pages and pages of helpful context time efficiency.

I could have opened that thing in Wireshark, done a sort on Delta time and found my issue, and then started looking at what was around it. Instead. I, I'm going through multiple prompts and multiple pages of helpful information.

All this will improve over time. So, uh, before we wrap this portion of the discussion, any other, uh, thoughts or, or questions from our distinguished panel? What, uh, what GPT model were you using?

I'm just curious. So, when I first did this test, it was May 1st, and I don't recall which model was out on May 1st. Fair enough.

But the follow-up test I did literally this morning, so latest and greatest. And it's still, like I said, it did better. It didn't have that wacky quasi hallucination kind of thing, but it still couldn't find the 132 second response time.