cPacket Network Observability for Deterministic Incident Detection

So the first section is we're gonna talk about after the introduction is going to be about, uh, incident detection and how we do it, uh, in a terministic way. What we mean deterministic is, we mean there is something that is a clear threshold that I can set to begin with, and I can start with. So there are many cases that this thing becomes relevant, right?

It was a few years back, a pretty famous, uh, supply chain attack, right? That supply chain infected, uh, the devices. And one of the characteristics of the attack was that, uh, the, uh, malware was sitting there for a couple weeks and then started going to a DNS server asking for the next level of instructions, right?

If you remember, a VSM cloud that was domain name, it was sending to a lot of packets out to the DNS server to get the next level of attack, right? So that's what we call DNS Beaconing. And if you read the cis a, uh, uh, IOC from there, it really says, well, the first thing you should check is if you're infected or not.

Now, if you don't know if you're infected, or no, not please disconnect from the internet. Wait for someone to know if you are infected. Uh, one of the things that we can do with our product, and, and then again, break down and let's double click on the smart board, is I'm able to inspect every single packet and every bite and perform a string match.

So what people do, and I think that's what Andy is gonna show us, you basically go there and say, do I have any packet that has a VSM cloud? Do I have the string in inside the packet? Now, DNS is not encrypted.

So by provision, was there a question? Sorry. There be an H Ps now, right?

There is, Yeah. Oh, so it is encrypted a Little bit. Uh, it's not exactly, so first of all, lot of DDS is know that the d the encrypted D ns is really around CloudFlare, right?

So the first one is going to CloudFlare, then you have to trust CloudFlare, uh, for that, right? We still have the first session that is going over your DPE, okay? Right?

Okay. It's the second one that Then Yeah, that, that, but my point is, then it gets encrypted. Then it gets encrypted, so, right.

Okay. Yeah. In our case, again, most of the data that we're seeing today in the enterprise is still going over UDP, and we're able to do that certainly in two years back.

They, they were, uh, so, so the way that the SmartPort is built is really around multiple functions. They're all implemented over FPGA and or asic, which allows us to keep the line rate and not, uh, and, and basically inspect every single packet. So what it includes are, the first thing is timestamping, which is less, it's relevant in security.

If you want to reconstruct the event across multiple data centers. It's critical for our customers in network observability to learn n to know nanoseconds latency, their ability to count, which we'll show in a second. We are able to do the string match filtering and then, uh, remove the duplicate.

Basically, we can manipulate the packets such that we don't, uh, load the other tools with extra packets. That's the idea there. Uh, so what, and what we'll see in the demo is, you know, the ability to go there, uh, put the, the string look for how many matches you have, uh, and the idea is if we see them, you know, that you're infected.

If we don't see them, you are not, it's pretty much, you know, whether or not you trust a COVID test or a pregnancy test, right? That's basically tells you, right? It's either here or there.

So now you can decide, do I need to go to the next step or not? So it saves a lot of time where you, you don't know, and you have to disconnect until you figure out if you're infected or not. Second example, uh, is all the different volumetric DDoS attacks, right?

So whether it comes from a big botnet that is sending huge amount of scenes or, uh, DNS simplification where they leverage a DNS server to send you much bigger DNS responses, all of that is hitting our monitoring point. And just by the mere fact that we're able to count everything allows us to very quickly generate counters that tell us what the ratio is, right? So if you see a SIN CAC attack is really about seeing the ratio of SIN packets getting much, much higher than the CNA act packets.

If it's a DNA simplifications, you're gonna see a lot of responses with no requests and or fragments, right? So even without going very deep into the packets, we're able to know very quickly if you are under denial of service attack of this type, right? So not everything and we'll show later slow burn, we have a different solution a little bit.

But at this level where it's kind of a massive attack, you can very quickly discover it. And the way people use it today, uh, many of them don't want to have the, um, mitigation in line. So it's on demand.

So what they do is basically attach the alert to the initiation of the mitigation, right? So they have the ISP with a scrubbing center, and they need to notify when to start a scrubbing, especially in financial services, right? They don't want to create any more latency for the users.

So they don't want that all the time to be there. Uh, but they do want to activate it as soon as they say, get the, uh, early indication. And the counters we generate is every second.

So the time to discover, so the time to decide you have an attack and the time to start is measured in seconds versus, you know, any other counters which be net flow, others that usually take a little more time and much less accurate. So these two solutions are really embedded in what we call the packet worker. The next level is, uh, or the next example, and I'm not sure how we do that, is really command and control channels.

So what, uh, identifies command and control is you have this infected, uh, devices and you have these sessions, uh, for a very long time just sitting there, uh, and you know, basically waiting for something to do, right? So they will only send, keep alive or, uh, very low amount of data to make sure that they have the open, uh, the, the firewalls are open, but there is no actual a lot of data. So the way that you should be able to identify them is, is really around, uh, the session length.

So the way, this is where we move from the packet pack broker to the packet capture. So the scale that we're talking about there is the ability to keep track of millions of open sessions. So even if a session, many of the network observer solution you'll see can tell you like a net flow, right?

Will tell you how many bytes and packets each flow sent in every given amount of time, whether it's one second or five minutes, keeping track of all the open sessions, even they don't do much is a little different problem. We have done that in order, again, for reasons that we need for the observability, but it does give us a way to say the number of open, well, it's either the number, but in this case it's just that the duration of a session is, is getting to days and weeks, and you should really go pay attention to that, right? It's not usually, it's not what you'll see for a single TCP session.

Do you, do you handle UDP sessions as well? No. I mean, we know we, this one we won't know the duration.

We can say this is how much bytes, but we will not, uh, collect them back to, uh, one session duration. Yeah. Yeah.

This is for TCP. So I'm Jack Paul from Paradigm Technica. So let's talk a little bit about more about this.

So from a high level session duration as an indication, long, long lift sessions indication for something possibly going bad. Mm-hmm. What are sort of, how are you deciding and what's the detection mechanism?

Is this just, uh, threshold counters or is this an AI agent ml? Let's sort of dig in a little bit more about how this is working. Sure.

So all this section is kind of based on assuming, you know, that you can set a threshold, right? So in this case, what they will do is typically a day mm-hmm. That will be the threshold that you use.

The next section will talk about, okay, if you don't have the threshold, so you, we can build a threshold using ai. We'll, we'll talk about it a little d later. For now, the only parts of the solution I'm discussing is what we call deterministic, which means this guy generates the metrics.

Mm-hmm. We collect them here and we're able to use, for that matter, Grafana thresholds to decide if a session is, uh, too long and you want to, uh, pay attention to. Alright, so from a security professional who's then going in deploying CPAC and implementing this, we have to think about, hey, we want to do session duration detection to as a, as an IOC of C two channel.

So we're actually gonna go think about this and put it in there, and we want to do that threshold Detection in this case. Again, I'll show you in a second or in Right. 10 minutes.

Yeah. How you don't have to, but yes, that's the idea here, that we still have people that just come in and they know that they don't want anything be beyond a day. Mm-hmm.

And they alert on that. Okay. So if there are no more questions, I think I'll hand over to Andy that is, Yeah, my name's Andy Barnes.

I'm the Senior Director of technical marketing at, uh, cpac. So what I'm gonna do is I'm gonna walk you through the examples that, um, that Ron just kind of outlined. Um, we'll do this on a, on a live system.

So, uh, what I'm gonna do is basically, uh, utilize the C packet platform and then generate some of these, uh, attacks that, uh, that Ron's been been outlining. So the first one I'm gonna start with is, is DNS beaconing. So what we can do, uh, as Ron outlined, is that we can actually program our FPGA to go and search for keyword strings within any packet that's going on within, within our network.

So what I've got is a monitoring point that is looking at everything that's coming in and out of my internet connection within my building. And what I can do is I can actually set up a filter, um, on those interface ports, and I can say, I'm gonna look for any string or any type of, of word within any packet that I want. So I can actually type in here, you know, uh, anything that I'm looking for.

So, so Ron used the example, you know, for the Solar Winds attack, which used, uh, uh, A DNS lookup for something called, uh, A BSM Cloud. So what I can do is I can actually program the management platform to then push that to every single port that I've decided to look for that string. Okay.

And I, I've already created it. So what I can do is I can go to my fixed filters, and you can see that my network is basically going to look for that string in anything on there. And as, as was outlined, we can do this at full line rate.

So it doesn't matter if it's a hundred meg, one gig, 10 gig, a hundred gig, we can do that at full line rate, uh, on everything coming in and out of that, on every court. So once I've provisioned my, uh, my dashboard and I've configured my, my FPGA, I can actually actually bring up a dashboard and monitor exactly what's going on. So we have standard dashboards that, that the SecOps teams can use to monitor each of the different, uh, you know, characteristics.

We can set up alerts using the Grafana interface, and here we are looking at that interface. So I'm looking at everything that's going in and out of my internet connection from, uh, from a holistic perspective, and I'm also looking for that filter. So you can see, if I scroll down here, I can see that that filter is showing.

There is no data existing in my network that has that criteria, but what I'm gonna do is I'm gonna now trigger one of my bots within my network to actually now go and make a connection to that A DSM cloud server. And what you'll see, uh, I'm updating every five seconds. What you should see is a graph populate here in a few seconds, and here it is.

So our network has immediately detected that a packet or multiple packets in the network have been seen that contain that string keyword. So that is a, a clear indication that a device or multiple devices in the network are infected or are trying to reach out to that, that remote DNS server. Now, I've also, I, I'm sorry, I'm not sure if you want questions in between or, or from a practical reality perspective, it, it, it works well showing, Hey, here's one string in a, in a demo, right?

But in a production, we've moved away from a security perspective, we've moved away from string matches and signature matches towards much more complex rules that, uh, years ago. In the context of are your customers, how many filters are your customers using in a production environment? Are they filtering five keywords, 10 keywords, 50 keywords, 5 million keywords, 5 billion keywords?

What, what does the practical implementation of this feature look like in the production environment? Yeah, this solution is not replaced an IDS or signature matching. It's really about the quickness of, you know, once an IOC hit, how quickly you do that.

So we are not searching for millions and signatures, we're searching for tens, they're not really using them. I, I, so then what I'm sorry, go ahead. Sorry, Jack, I have a question about this.

This is, this is a, like a realtime string matching Yes. On realtime packets. Yes.

But what about the retroactive, uh, retrospective look, you know, I got a, yeah, a month's worth of packet captures stored, right? How do I, once I get the IOCI go back through the last month of traffic? Yeah.

Is, is this the same, this this will be, this is the same process? No, that will be done on the capture device. On the C-store.

On The C-store, yeah. Okay, Got it. Yeah, So, so in this particular case, every packet that we're seeing is also being captured into our c-store environment.

So we could retrospectively now go back in, do a string search on existing data, and you decided that, you know, that there's something happened in the network, or, or somebody's come up with a new keyword search because of a new server, you can then go, retrospectively go back in and search through your data to see if that exists, as well as flicking it onto live data. So, So Also, I'm sorry, Jack, I Please continue. I I, I was also gonna say, you know, as I'm seeing this, and I know this is a very basic contrived example, um, but I'm thinking, okay, what if I just fragment the DNS request?

Right? So now your string match does, now the, the, each packet doesn't have the entire string in it. No.

So are you guys doing reassembly and then looking Not at this level. Not At this level. Okay.

Yeah. Yeah. So I'm trying to understand the, um, the, the, the real world use case, right?

So I understand that this is a speed thing, right? This, this, if, if, if I'm looking for something, I wanna be able to, you know, I've identified that this is something dangerous I wanna flag. Mm-hmm.

So I'm gonna put a string in here and I want to go see, as soon as that string pops up, I wanna be able to get notified and do something about it, you know, in minimal latency. Mm-hmm. That makes sense.

Mm-hmm. Except for, this is a, the, the stuff you're looking for is extremely dynamic in nature, right? So you're not going to be looking for a well-known string such as YouTube or Netflix or OnlyFans or whatever, right?

You're gonna be looking for some random d NSC two control. That's, you know, so by the time you've identified that thing you wanna look for and entered it into your filter and said, start searching for that, isn't that event already gone? How, what's the real world use case for that?

Well, so the, uh, supply chain attack, right? There was this, they discovered CI introduced the IOC, and then you had the about couple, I mean, there was a built-in latency there for two weeks until after you installed, right? So this is really about what do you do if you get an IOC hits the wire and you need to do something about it, right?

Okay. It's a very specific use case, right? We're not replacing, again, it's not about replacing idss, it's not about that.

The, the, the point is that if for network observability reasons, they have very good reasons to put our devices before and after the firewall, which is, are all the choke points coming in and out with the internet. So it's a very natural place to come in and say, okay, do I have the problem or not as soon as I get the IOC. And then there is a whole set of workflow that you have to go through to clean up and, you know, find where the vulnerabilities are, right?

But it's kind of a demarcation point that says, okay, I do I have the problem right now or not, and then that will kick off the rest of the process. Right. So quick question, did you said before and after the firewall, so in the case that you're coming in, you know, in, in that path, are you doing the decrypt or, uh, is that, how does that work in that scenario?

And that's one of the challenges I think we have as, as you know, as security practitioners dealing with the lag and the challenges associated with all of the decrypt. Are you handling that, or what is your role in that transaction? No, we don't decrypt.

So we are counting on all these packets that go through. Uh, so that will work for, this is why we're showing DNS weakening because by and large DNS does go through, uh, unencrypted. So we're able to inspect it.

Uh, there was a, I think again, three years back, uh, I forgot what was the name of the attack? Uh, with, uh, the JND, it was A-J-N-D-I vulnerability. So this package also went through in the clear, uh, so there is a whole set of IOCs today that are going in the clear, and these are the ones that we can detect.

So if I, if I offload encryption at my border, would you be able to give me more data then? Yeah, so, so if I, yeah, if I have a ta, uh, I don't know if I offload outside of a, and then inside I have another tap for you, you can see all of that data. Sure.

Yeah. The reason we don't introduce encryption into our devices, first and foremost, is it's, as I showed you, everything was designed to keep track with line speed, and once you add encryption, you know, it got to other complexities that we don't want to handle. Wow.

Second one, which is even more important, is, uh, we don't want to handle all the key exchanges, and that's a lot of, uh, software that is for real. We're a network observability group. That's not the key, uh, thing that we want to worry about.

It'll make all of our solution much more complex. So if you unencrypted before us, and we certainly have a lot of partnerships where we are sitting behind an UNENCRYPT box that does the unencrypt, then yes, then we can inspect everything. Okay.

Okay. So, um, yeah. So in this particular case though, once we've detected that a packet's there, we're able to, to trigger an alarm.

So, you know, we can push to the likes of Slack, or in this particular case, maybe we, we've, um, you know, we can go in and actually look at the data coming into our Datadog instance. So we can actually open up Datadog and we can see, you know, all the events that are coming in and triggered by, uh, you know, by any particular, uh, event. So here we can see that event that I triggered actually was created a, an incident directly into Datadog.

And, and now because we have that incident that's been open, it, it can be assigned to the security team automatically, but also we can provide you direct access to that P cap. So now not only can you see that the incident has happened, but you can now download that P cap and fi figure out which client was having that discussion and to which server, and you can figure all the timescales out. So again, you, we have that ability to move all the way down to the packet level to help people to actually identify and, and root cause the problem.

So that's kind of our, our first, um, overview. So the second demonstration I wanna wanna walk through is gonna be to look at, at DDoS. So, you know, very much like we did with the, the beaconing example, we can actually look at, uh, all the packets, the syn and the syn act packets coming in between your client's annual server.

So again, we have the dashboards that allow us to take an overview of what's going on. We can see the ratio of the incoming, uh, and outgoing CAC packets, and we can actually do a calculation in real time and actually overview the network. And what we can do again though, using my system is I can actually now attack a server in real time using a distributed denial of service attack.

And in, again, just as before, the hardware within the packet, broker is able to monitor all of the packets, look at the ratios, do real time counting of everything, and as soon as it depicts the app zoom ratio has gone out of, out of quota, it'll actually flag that immediately. Is there A question? Sorry.

Okay. So you can see here our, our dashboard is updated. Again, it's immediately telling us that our, uh, syn syn act ratio has changed.

So we've got a ratio that's completely out of a normal operation, and again, we can set up a trigger now to create, uh, an, an external event to notify us again, give us access to that PCAP file so we can see, you know, which server is being targeted, which, uh, clients or which host names are attacking. And again, we can then provide that information to our mitigation team to, you know, to deal with that at a firewall at or at an ISP level. So again, we have that, that real time feedback where a lot of tools take minutes or tens of minutes to gather that information.

We can get it in a few seconds, uh, you know, very, very quickly and very easily. And, and this is being measured and detected at the broker level or at the, at the, the c-store level broker. So this is actually being, uh, actually calculated at the packet broker.

So the, the, the, the FPGA in the packet broker is able to look at a whole range of attributes within the packets. And since ciac is one of those, uh, fields or one of those tags that's able to be counted and logged at the packet broker. So we are feeding that, that metadata up directly to our dashboard, uh, in real time.

Yeah. So actually to just to be, so the counters are generated at the port level, at the packet broker, you still need the logic at the, what, what ND is showing here is, uh, what we call the control center or ccle. So it's the centralized view.

So you still need the logic there because many times you, you need to know which two ports are correlated, right? It's, you're gonna have one port which is going out, that has the CA going in that has the scenes, and you need to know which ratio to calculate. So this logic is actually implemented here at the top right?

I cover DDoS of a, Of, of, of an area. So I understand the cyac, uh, uh, comparison. Uh, we've seen trends towards much different UDP type, uh, attacks or reflection attacks and, and mm-hmm.

And are is, what is your, so are you proposing that you can detect DDoS attacks or you can detect some kind of DDoS attacks? Am I replacing my current DDoS mitigation strategy with what you're offering? Help me understand where you fit into that sure.

Picture. Absolutely. So first we are doing, uh, detection, right?

So we're not mi mitigating, okay? Right? So you still need a mitigator or scrubbing center, okay.

Uh, the detection, uh, so all, so, uh, all the vol, I, I don't know, I want to say all the volumetric attacks that we, that I know of today, we are able to deck so that, so these are basically will be things like, uh, abnormal number of, uh, uh, IP fragments or DNS response to DNS requests that are not matching or, um, seen or cenac or TCP bad flags, right? So these type of things that I can count packets and compare ratios we can detect. Okay.

Uh, we'll talk about slow burn DDoS detection in the next section. Uh, so this one we can do on the pack. So, so this, this type we can do on the packet worker, and it's only about the detection, right?

So what they do is usually they just run us next to some other mitigation detection that is based on net flow or some other things. Uh, the experience that we hear, again from our customers is that it's better as measured by the accuracy and the time to detect, and then it's useful when you have a mitigation that is, uh, on demand, right? So they basically connect this alert to start the mitigation.

And many of the enterprises, I'm sure you know better than me use that Back. Sure, yep. Makes sense.

Sorry, Were you No, I'm, yeah, Go ahead. Um, so when you detect, say, an imbalance and send dis enact mm-hmm. And, and you assume that it's an, do you have any way of triggering some sort of remedial action, whether that be, if I have, I'm set up for remote trigger black hole, or, um, I do have the ability to cut over to a scrubbing center, are you able to initiate that on demand or, What we do right now is we generate the alert, or it's a web hook so we can put on the web hook whatever information typically today or not.

Typically, all the customers we I know of that are using that today, they take the alert and manually initiate, or they have another box that will start the actual mitigation, but nothing stops you from the, the APIs themselves support that. So you could put, actually, let's start the mitigation based on the indication. Okay.

That's just up to me to set up though. Yes. Okay.

Exactly. And I wanna say you don't normally want to have an automated, uh, mitigation because there's a lot of false positives, right? Um, so you, you could end up taking your own network down, uh, through a false positive because you wanted to react instantaneously.

So it's best to have to keep the human in the loop and just throw an alert, uh, at least, Yeah, based this, this is why I defer to a security expert. I've got scenarios, but I'll chat with you. Okay.

So the, the last example then, um, is, uh, is, is the, uh, the C two command and control. So what we can do here is, um, actually take a look at every session that is going on within our network. So as, uh, Ron outlined earlier, we have within our packet, um, within our packet capture hardware, the ability to monitor and log, and then basically look at all the sessions that are going on within the network.

So what I can do is I can open up all of the connections, and I can then basically sort those by how long each of those connections have been open. So I can actually very quickly take a look and see all of the different, uh, endpoints that are being connected to. 2 days.

I can click on that and I can actually now get all the information about that session. So it's a very slow burn, very long duration, very, very small amount of traffic. In fact, there's such a small amount of traffic is not even registering, uh, because there's no packets being, being transmitted in the last five minutes.

So we have this ability because we're, that session is open that we have, we can, we can actually track and keep, keep an eye on that. And if we have more traffic, we can actually now start to break in and overlay how much traffic that, uh, that session is holding. So again, we can build alerts and we can build notifications based on this kind of long term command and control connection.

Like the scale. Yeah. So just to kind of put these things together, right?

So it's really about, again, it's not about replacing security. It's, uh, we are our networking. The people buy us in order to do everything that we describe in the network field day.

Having said that, you know, if you are running the SOC and you have someone appear in the knock, that's a conversation that you can have with him. Are these, is this information that I can use from your current network operation tools, right?