Hedgehog Gateway Demonstration

So, um, I'm, uh, Manishh, CTO at Hedgehog Networks. And in this, uh, segment I'm gonna talk about the gateway, um, the new part of the product. Um, in the previous segment we talked about VPC peering on switches, which has some limitations.

And so we're gonna talk today about gateway and how it overcomes some of those limitations. So we saw in the last segment, switch based peering. And so what happened in that switch based peering demo is that the hedgehog environment, when you took two VPCs, which are essentially tenants that are isolated via v Fs and uh, and vxlan, um, and when you peered them, we would wreak leak routes between, uh, the different VFS in order to, um, allow two different isolated tenants to talk to each other or to the external world.

Now, um, the, this is really good because you get the full cut through bandwidth of the switch. So when two VPCs are talking to each other, you're going through the fabric. Just like any other communication you can communicate at your port speed with no limitations.

The problem is that switches, especially relative to their cut through bandwidth, have not much CPU and not a lot of ram. And so the problem is that there are certain network functions that you wanna run between pairings that you can't, um, do on a switch. So, uh, Florian mentioned for in the very first segment with, uh, when he was talking to Mark that, uh, you know, he wants some port forwarding and firewalling functionality and Nat well, um, in order to do, for example, especially Stateful firewall and Stateful nat, you have to keep a flow a, a a entry around for every single UDP or TCP flow in your network.

And on a bigger network that's millions of flows. The switch doesn't have the capacity to do that. Um, if you hire multi-home, then you want several internet size routing tables and make routing decisions between those providers.

Um, the Switch is also not a great platform for those kinds of operations either. Anything that requires TCP termination is off the table. So you wanna do any layer seven application layer, load balancing, deep packet inspection, that type of stuff.

Again, the switch is not, does not have the compute power and ram to do that. You wanna terminate IP SEC so that you can connect your cluster to some other endpoint. Also, can't do that using, um, commodity switch hardware or even actually proprietary switch hardware for that matter.

So enter, uh, hedgehog Gateway. So, so the idea behind Hedgehog Gateway is we're going to get a CPU rich high bandwidth server and put it in between the traffic flow bet of VPCs. So when V VPC talks to VPC one, you can have a peering mode where the traffic will be forced through that gateway node.

Now, the cool thing about this is that's just a general purpose CPU, and so you can get, um, a wide variety of network functions on that, uh, piece of commodity hardware. And it's pretty fast. Uh, good software implementation, only implementation can do like a hundred gigabits per second on modern hardware, maybe more with the latest generation of CPUs.

Um, the other piece is that modern NIC hardware actually has a lot of hardware offloads. So a Connect X seven nic, for example, which comes, it's up to 400 gigabits per second. It'll do two, 200 gigabit ports as the most common configuration that people run these things in.

Um, well, one of the common configurations, um, it can do full nat, NAT six four in hardware. It can do VXLAN, endcap and dcap in hardware. It has a routing table, it has a whole bunch of hardware there to offload the CPU.

And so you can easily get over 200 gigabits per second, um, with the hard Roth loads on that platform. So with Hedgehog, you can now have a gateway peering. And so this is different than the VPC peering that you saw earlier, because gateway pairings are gonna go through the gateway nodes, one or more gateway nodes in the system.

And so here with the gateway peering, um, you'll notice it's a little bit different than the peering object that we saw before, in that we are exposing certain IP addresses within the VPC in this case, for example, VPC 1 10 0 7 0 slash 24, we're exposing those IP addresses as 1 9 2 1 6 8 1 0 slash 24. So that's an implied nat. Um, to do that, you have to have Nat.

Um, and then you can configure, um, as we build out functionality in the gateway, there's lots more things you'll be able to do. Like, uh, Florian talked about basic firewall and port forwarding is a common thing, so you'll be able to do that at acls, external connectivity. Um, but then the sky's the limit as far as that's concerned in terms of what software you'll be able to run on the gateway.

So the interesting thing is, um, so all that sounds great, um, easier said than done. Uh, how do you actually get traffic to that gateway? So that's the interesting piece of this.

Um, so a lot of times when we think about these devices, what we think about, for example, if we have a firewall on the edge of our cluster, we think about plugging in one port to our ISP, and then the other port goes to our switches. The gateway here actually works differently. It is connected somewhere in the fabric.

There are some trade offs on where you connect it, but we can talk about that, um, in the q and a part. Um, but the way in which we get traffic to the gateway is through selective route advertisement. So that gateway sits there as A-V-T-A-P on your, uh, network as a layer three device, and, um, looks at the configuration and makes route selective route advertisements to attract the appropriate traffic to it.

So in this case, on the right hand side, you can see our exposed line for VP CO one, and we see that, uh, we're exposing 1 9 2 1 6 8 1 0 24. What will happen is in the VRF for any VPCs that are appeared with VPC O one, in this case, VPC O2 gate, the gateway will advertise a route for 1 9 2 1 6 8 0 1 0 slash 24. So anytime VPC two tries to talk to VPC one on those IP addresses, the traffic will actually end up at the gateway node.

Then the gateway does a V PC lookup based on the IP addresses, figures out what the destination VRF is going to be, performs whatever network transformations you want, including nat, and then transmits the packet out again on the VNI for, um, for V VPC two. And then the same thing happens in reverse. Um, Question for you is is, oh, go ahead.

No, go ahead. Thanks, Jody. Um, how does the, can you go into more technical detail of how that packet actually gets to the gateway off of my VPC?

Is it a route injected? Is it a layer two switch, or how's it get up there? Really?

Um, it's, it's routed. So, um, it gets up there, so from a host, it's gonna enter the switch. Um, that host is configured on a particular VRF.

The gateway has advertised that particular set of IP addresses into that VRF via type, I think it's a type five B-G-P-E-V-P-N route. Um, so it advertises that. And so that route shows up in the switches route table.

So when it receives that packet, it says, ah, this packet needs to be encapsulated with the VNI for this tenant and transmitted it to the next top, which is the ga, which is the gateway node that advertise that. Again, you can, with ECMP, you can have more than one gateway node, but to keep things simple, we assume that there's one. Um, you now adver that, uh, the gateway then gets that packet, um, through routing, um, and it's encapsulated.

So we know the originator, the source, VNI, and then our system under the hood actually makes sure that you don't advertise in the same appearings, like conflicting IP addresses and stuff. So from the destination IP address and that source, VNI, we can uniquely determine what the destination network is going to be and what NAT rules have to be applied, and whatever other pipeline processing has to be applied on that peering. So we do that and then we transmit the packet.

And again, um, that's transmitted, um, encapsulated for the right VNI, the switch that the gateway is attached to then looks at that packet, says, ah, here's the VRF routing table for that. And then routes the packet to the host where it gets well, routes the packet to the switch that the host is connected to. It gets de encapsulated there and then forwarded on to the vlan, um, for that traffic.

Did that answer the question? Absolutely. Absolutely.

Thank you. Yeah. Um, and Go ahead, Jody.

Lemoine here. I've got a, a question as well, just looking at the, uh, exposing the ips, uh, as, as a different sider, is that required or is that, is that optional? It seems like an extra nat depth.

It, It's, it's optional. So if you don't, if you don't have an AS clause or you don't want nat, um, you can either expose to the same address if you want, or you can just not have the as clause, in which case, um, the gateway knows not to nat. Um, the thing to be aware of those, if you have overlapping IP address ranges and such, you're not gonna be able to expose the same set of ips, um, on the same VRF, uh, because there's no way to uniquely make the route advertisement unambiguous.

Right? So there's some restrictions. I I had to think through exactly what they are, 'cause it's actually quite complicated to understand restrictions, but they're, the restrictions are easy to understand, but they're there because you need to be able to uniquely identify the destination VNI from the source VNI and the destination IP address.

So NAT is not required. Um, but if you don't need NAT and don't need any advanced functionality, you might as well use VPC peering. Um, you don't need to use gateway peering.

So I wanted to show off the NAT here because it's A VPC peered, um, because it's a gateway peered, uh, interaction. Does that make sense? Yeah.

For Me it, it, it's more of a question of where you're doing the nat, does it have to be global or can it be more specific? And it sounds like it can be more specific. Yeah.

So yeah, it's not global. It's, it's specific to the peering. That's correct.

You could have a different peering where you're exposing that same block of IP addresses and let me think it through, but, uh, to a different VPC with the same ips, I think that that's possible. The, the trick is the reverse traffic has to also have unique routes. I believe that that's doable, but it is more fine grain.

It is not global. Right. And then this will do IPV six NPT the same way.

Um, y yes. The, the gateway is capable of doing that. Yes.

Um, okay. There's, there's a bunch of configuration that has to happen throughout the fabric for, for that type of, uh, IPV six to work. But the gateway data path would do it the same way.

Yes. Thank you. Other questions?

I was just thinking about the, so the gateway knows themselves. You know, we talked about, um, when I'm setting up my topology, you know, so sort of stepping back a little bit from make sure pushing services through the gateway, we're just stepping back to initial systems set up. Mm-hmm.

Now we're talking about our topology, uh, flood wiring diagram, right. For our physics switches and things. And then the switches can, um, uh, Z-T-P-Z-T-P, yeah.

DT PT p um, I live here, so I switch back and forth agency sometimes, I dunno. That's okay. Anymore.

Um, that's all great. 'cause the network online, the gateway knows themselves. How are they configured and added to the fabric early on?

Like what's, so what, like, you know, how does that sort of interaction step look like, you know, from, as a, do I just plug in the servers and say, have at it? Or do I need to be good? The Servers are also connected to the management network in the current release.

Um, 'cause you we're always working on new stuff. Actually there was gonna ask about as well, was the management interface 'cause you Yeah. So Connected onto the management network.

Um, and Sergey correct me if I get any of this wrong. Yeah, I I can probably take it. Oh yeah, go ahead.

Yeah, so I, I I guess I'll split it into questions. First of all, the management, uh, network. So we, right now we rely on the out bank management network and we require all the control nodes and gateway and switches to be connected to it.

No special requirements to, it should be just simple, simple network. Uh, we ask to not add any other devices to it. Like a gateway node might have say, uh, one or two CX seven nicks in it and or she probably used just like the, a copper one gig or whatever to connect your management.

Yeah, exactly. Exactly. Yeah.

The, the on motherboard Nick can, yeah. Well, yeah, I mean, if you've got the requirements, you know, is they've got the, the BMC nick on a Lenovo or on Adele, they love to come with a extra one gig Nick on for reasons, but yeah. Yeah.

And I'm not push any traffic through there. Like it's, it's a little bit of a controlled traffic. So there is, yeah, even like one, one gig card B 45 could perfect be fine for it.

Um, so that is about the management network, uh, about the provision side. Like basically, uh, as Manu STA switches are fully controlled and provisioned automatically through ZT P So as if you go into bit deeper into details, there's only not install mode. So your boot switch into it and switch will automatically discover our control mode and will get, uh, fine tuned configuration for this specific, uh, role and, uh, identification of switch in the fabric.

As for the servers in, in our product, so for the control notes and the gateways right now, we, um, generate the ISA that you just boot from. You don't need to do any extra configuration. All the conversion done initially when you plan the fabric and it just joins the isa, and then when you need to upgrade it, you just apply the single binary it apply, it upgrades everything in place, including, including the operating system, Kubernetes ulcer party services that bring, we bring with us.

So you don't need to think about it at all. It's fully, fully like software appliance style of experience. Uh, it's the same for the gateway.

Uh, so right now, if you will have like couple of control nodes and a gateway, for example, you'll have like this couple of iso, uh, that you just boot from. And, um, that will, that'll be it. Uh, in future we will make it a little bit simpler with a pixie boot.

So you will only need to provision one ice on the first node, and every sno will be pixie bootable over the management network from this first node, which after you add in more control nodes for redundancy, it'll basically become like redundant control plane. Yeah. Uh, with every SNAs bootable from it, right?

So, so yeah, there's this overlap. You can dedicate gateway nodes, but eventually, um, that once you have the one control node that bootstraps the whole process, then um, yeah, you just program the Mac addresses in your wiring diagram and it'll net boot everything, pixie boot it. Um, and then actually you'll be able to use the gateway nodes also as redundant control nodes, like Sergey said.

Yeah. So that, that in future, As long as you don't, I'm sure we've all done this here, where we've ended up, you know, put our vCenter server inside an environment, then the VM environment, everything gets all nested. You know, you can end up in a horrible place where you cut off the branch that you're sitting up.

Yeah, yeah, yeah, Yeah. So, so yeah. So this is one of the benefits of automation, right?

Is that we can make sure that you don't cut off that last, the last rope that's holding you up, and then you fall into the abyss, right? Yeah. So, um, uh, the automated system will make sure that you don't do that to yourself.

Yeah, that's interesting. The control nodes in the gateway nodes can, am I right? Can overlap and function.

They can. They, they will be able to. They don't currently, they'll be able to.

Um, we don't require it though. So it just depends on your posture. Like, do you want to have, so Here I've got these four servers here acting as gateway nodes.

Can't you just like distribute your control stuff over there? Yeah, sure. Yeah.

Right. Exactly. Why do I need this yet another dedicated server, right?

Yeah. Cool. Um, and, and you had to be careful, right?

You said, oh, you have a couple connect x sevens per server. Like generally we recommend you only run one. Why?

Because there's actually a switch, a mini switch inside the Connect X seven. If you have two, then anything that goes between those nicks has to go through software. It's, Um, and so then it's slower than if you are saying weird.

Yeah. There's a whole, um, So, Uh, we use a lot of six sixes. Um, we'll just probably just start rolling out CX sevens the next, over the next year.

Yeah. There's a whole lot of stuff around nuer allocation, trying map the affinity stuff. And I, it's maybe something we'll talk about later, actually.

Yeah, it's, I know there's a lot of hard stuff there to, to get that high bandwidth out of those ports. Right. And what you don't want is, oh, because you're talking on port A to port B, which is on the other nick, now your performance is, you know, 30% of what it was when you talked between A and C generally, it's just easier just to have one nick and, you know, CX eight is gonna be 800 gigabits, right?

Two 400 gigabit ports. So Yeah, I dunno, I'm trying keep that, they're kind of un remember then PCE bus comes and issue. Well, Yeah, but they're kind of unobtaining at the moment anyway, so Yeah, Yeah, yeah, yeah, yeah.

Like they're not shipping that to me yet. Yeah. We've On that Jody le here again, what kind of throughput can we expect if there's no accelerated nicks in the system?

So we're gonna assume to have, um, some basic acceleration. So that is, um, check some offloading and some really table stake stuff that even the cheap onboard one gig Nick has. So, so that's, let's be clear.

So we're gonna have at least that, um, on the first release, we're just targeting 40 gigabits per second, um, which should be very, very easy to hit. Um, a hundred gigabit. I have no question that we can hit a hundred gigabit.

That should be easy. Hundreds easy. Yeah.

Hundreds beyond, um, going hundreds. That's to get hard, Right? And so that's exactly right.

So our first release 40, next release a hundred, then we're trying to, gonna push 200 and then if there's demand from customers, we'll try to get it to 400 gigabits, um, at 400 gigabits PCI bus. Yeah. Memory bandwidth.

There's a whole bunch of, um, like there's a whole lot of system level constraints that, that show up, um, once you start getting to those levels. So we'll have to see what there. Okay.

So if we're playing 40 and under, then as long as we've got basic table stakes stuff, we're good. Um, yes. I, I, I, yeah.

Yeah. Oh, oh, table stakes on the nick. Yes.

Okay. I, now I understand the, the point. Yeah.

As long as you have the table stakes, I don't think anybody even makes a nick that doesn't have those things anymore. Maybe there's things somewhere, but, um, as long as you have, you'll, you'll under 40 gigabits, you'll be fine. I mean, we were doing, so my, my first startup was a company called Line Rate Systems, and we were doing 20 gigabits per second and like 200,000 HTDP requests per second plus TCP connections.

And that was what, uh, I'm old now. Um, that was 2010 timeframe. Right.

So 40 gigabits is not, um, is easily doable, uh, on modern hardware. Like I said, 200 is where that limit where stuff starts to get hard. Okay.

I only ask because I can see a few organizations that really appeal where the architectural aspect of this really appeals. Mm-hmm. But they're not playing anywhere near the hundreds of gigabits as far as actual play.

They, I can see them liking it for an architecture that has the potential to grow into that. Yeah. And, and part of the reason to target that a hundred gigabit mark is that, um, a number of the, uh, AI customers we're talking to have a hundred gigabit up links for their clusters.

And so in that situation, we want them to be able to saturate the link to pull the data into the cluster. And so that's that a hundred gigabit target. Um, I don't think we've talked to anybody with 200 gigabit up links yet to their service providers.

Well, I guess there's one, one company we're talking to that has 300 gigabit up links, but they can just run three gateway nodes at that point. I mean, they're running, you know, they'll be running thousands of gps. I guess that's the other thing is that Yeah.

Is if, if my burn node limit is X, but I can run multiple nodes and distribute it, then Yeah, I, yeah. And there's some caveats around, um, when some of the stateful stuff comes online. Um, there's some caveats around failover and, and how that works.

Um, you know, the first cut of stateful, you know, you'll lose the nat connections, the next cut, you'll actually get redundancy. But then, then the route advertisements become very, very tricky because you, if you have, let's say eight gateway nodes, you can't replicate the state across all gateway nodes and hope to get scalability because then the ram doesn't scale. So you selectively replicate when you selectively replicate, you have to make sure that when you have flow failures, that either the traffic goes to a node that already has the table or the node that gets the traffic after the failure can actually collect the table information from the remaining nodes.

So there's some complexity in how that works. Um, which is why the first versions won't have, um, that state full failover. The next versions will, but it'll be limited to, you know, two nodes and then you get the full scale out.

So there's some caveats, um, around that. It's a complex problem. So, and this is a technical audience, so you know, we can get into, yeah, I mean, we've all Dealt with failures of state synchronization, whatever as well, and there's all issues.

Well, right. That's will tell you, of course you can just, that you don't need to worry about replicating any NAT stuff, anything that's, 'cause if you just put native V six in there is all fine. Mm-hmm.

Well, yeah, there's However, for the state for firewalling, so, so we're making happen just so we're just talking about the state for firewalling aspects. Yeah. There are issues.

Yeah. Yeah. The state is still gonna be an issue for V six, even if even without the net.

Yeah. Yeah. Well, and yeah, V six is a and a different beast entirely.

I mean, we would like to think that everything supports V six everywhere, but that, that is not the case. Uh, FRR, which is one of the most popular open source routing packages, for example, does not allow IPV six next tops in EVPN route advertisements, which means Sonic does not support BG I PV six next tops for EVPN route advertisements. Now Cisco, there's a team, uh, I think it's outta Cisco Toronto that have been applying PRS to a RR to fix this, but then it has to get put back into all the vendors software.

So there's like, there's issues around just going straight V six. I mean, it's, yeah, unfortunately, um, it's not transparently supported everywhere. What, what was that?

Nothing changes Really with V six. It, it's always close to being ready to Deploy. Yeah, yeah, exactly.

Well, and, and this was dumb, like in FRR, it's just that the guy who wrote the EVPN code used a struct in Ater Yeah. Which is an IPV four address, and he didn't use a union that would allow either. So I think that patch just got merged where it fixes that.

But then there's all the downstream implications of that that you have to then address. Yep. So, I'm glad it's been out for two years that people have been trying to fix it and it's looks like it's starting to get some forward motion to actually fix those problems with P six.

So Open projects are never fast. Yeah. Yeah.

So how does troubleshooting work? 'cause like we've got some, some YAML files and stuff, and like you showed that you could get into the CLI, but like if you're, if you've got the gateway, you've got a bunch of different things going on and you get, Hey, this application can't talk to that. Am I looking at a YAML file to try and figure out what's broken?

Am I going CLI from the device to device to figure it out? How do I, It never fails. Oh, so we're good.

We're done. Can I buy two? Yeah.

So No, um, there are different levels of diagnostics. Um, you can start by looking at YAML files. Um, did we ever release that doctor tool that you were building?

Oh, not yet, but yeah, I, I, I can talk about it. So, so yeah, there's a, there's some tooling, so let's talk about it. So, so, So the, um, the, the, the thing that our, our API being the extension to the Kubernetes API is food declarative.

So you can just look at the API, which is, can be presented as a YAML files and see like what are the desired state. And if everything was applied and it's, it goes to the actual state and, uh, it's already kind of easy to find a lot of answers in it, in it. And, and we propagate actual status and state of the switches and the gateway into this API.

So for example, you will have the port counters, uh, on, in, in the API as well. So you don't need to go to the switch or the gateway. Uh, we have a, it is big, all available in the, in the API.

And we have a series with CLI tools that allows you to do like introspection. We name them inspect commands. So you, you can do like inspect fabric.

It'll show you like some summary and then you can go deeper. Like, inspect switch will show you status of all the ports, transceivers, uh, uh, it'll show you if some PSU some, you know, like some fan switches have failed. Um, you can go like, even deeper.

You can get like, inspect on a switch port and it'll show you the, um, like how much traffic you get in on a, on a, on a, on a buffers in, in a switch and things like that. So essentially you have a lot of a tooling to look into the what's actually happening within the fabric on a devices without the need to like SSH to them and do something like you have all it through the API and all that information is available in your monitoring system as well. We do push it to, as I said, like to promeus and lock case.

So you have all of that in your monitoring systems. That's kind of one chunk. Another chunk is, uh, we have other type of, uh, tools in the CLI that allows you to answer some questions.

Like you can ask it like inspect IP and uh, you can pass it some IP address or mark address, and it'll look into the, all the configurations through the API and it'll say, okay, this IP could belong to this server. Like this Mac address is actually, that's internal one used by the fabric itself, or it's actually something that we learned that one of your servers have this Mac address. And then you can have, um, inspect and I don't remember the exact command, but uh, I wrote it like year ago.

So it's very difficult. Um, so you can, you can inspect the like next layer of all like logical things. For example, you can, you can inspect and ask the question in a format, okay, I have a server with this IP address and say another server with like another IP address, can they talk to each other?

And it'll, again, using the API, it'll calculate and say, yeah, they can actually talk to each other. And that's why, and it'll print out the whole kinda pass how the traffic will go, like, and why they actually can talk to each other. And same is applicable basically to all the kind of logical parts of the fabric.

Uh, like you can basically inspect all these different components and it'll tell to you like what the actual status was applied there. And if you do it in a more, like, higher level, like about the connectivity, it'll tell you why this connectivity is possible. So, and, and so then you don't have to log into switches individually.

Yeah, yeah. For the most part. I mean, you know, it's software, so stuff can go wrong, but Lemme keep adding this tool.

So our, our essential goal is to, like, basically you never need to go to, uh, like through the server or like to gateway control node switch to, to learn some information. We want all of it to be readily available in A API and essentially all this CI, they just read this stay from the API, it's information is already there. Right.

And because it's Kubernetes API, if you're so inclined, you can build your own tooling on top of that API as well. Yeah. Um, 'cause all the data is machine readable.

So, um, are there other questions or do we wanna see a demo? I Do it. That's it.

Well, I was gonna ask Oh yeah, go ahead. Um, there's security functionality built into the gateway notes as well. Uh, so current, uh, so yeah, the first release of the Gateway, um, so we're in Alpha right now, so it's an open source project.

So you can actually just go to the repository and see where we're at. Um, the first release of Gateway will have a very simple firewall, so some acls and a port forwarding. Okay.

Um, because that's been the most requested thing. It's like, Hey, I just wanna make sure that my cl nobody can get into my cluster, but I can get out and um, I can have a port for it. So I can SSH from a selected set of IP addresses into certain select servers.

So that will be the base security functionality. And then eventually we'll add more stuff over time. Like, you know, uh, DOS protection, um, I'm trying, You you had it on a previous screen.

I-D-S-I-P-S. Yeah. Yeah.

So, uh, Like L seven inspection. Right. So all of that stuff, as customers request it or as contributors to the open source project, wanna contribute it, um, we'll be able to add that functionality.

So I-D-S-I-P-S probably right. What we'd likely want to do is run sur cotta or something like this and route traffic to a container or something running on the gateway node itself. And then you can configure sur cotta rules on that.

So, um, the, the point of that slide is to say that because it's a general purpose, CPU, you can run Okay. Um, a whole host of stuff. The first release of the gateway, um, will be bread and butter.

Connect me to the internet. Um, let me, nat, let me do a port for Right. That's, Um, I had one other question.

Yep. Um, so in a case where you had, um, discontinuous like data centers Yep. Um, is there any, like, is that just build the fabric across, is there any special like juju going on there?

So today, what you would, okay, so, uh, let me, uh, let's just talk about the ideal state of affairs, uh, that gateway eventually will be able to be a tunnel endpoint. Mm-hmm. And so you will, what we'd recommend is you configure two fabrics, and then you configure the fabric so that the two gateway nodes can tunnel and talk to each other.

And then you would have a tunnel between them. In theory, you could have a single controller controlling the fabrics in two data centers. I'm not sure that's a good idea.

We, we Actually have some ideas about how to do that. We're not doing it today, but like, thanks to Kubernetes again, uh, what we can do, we can use the Kubernetes Federation, which will basically, there will be two independent or multiple independent clusters in different data centers that connects to each other using the, the tunnel provided by the gateway as Manish said. And then there's a configuration on top of that that gives you single, fairly transparent, uh, representation of multiple clusters through a single API where you can configure and work with different fabrics despite them being actually kind of independent in, in across data centers.

Right? So that's a another benefit for us of using Kubernetes. We don't need to worry about, actually fairly complicated problem of solving the federation across the, uh, disconnected data.

But, But what you don't want to do is have one cluster that spans those data centers, right? Mm-hmm. Because you end up with split brain problems and you don't want that.

But yeah, Federation, it gives you the illusion of a single cluster. Yeah. Gotcha.

Yeah. Most the end user perspective from the tools he using, like you will, you will see it as a one entity, but action under the hood will be multiple independent ones. Yeah.

Thank, but yeah, that's not special dust, that's just the Kubernetes Federation. So that's one of the benefits of, of using that ecosystem. Alright.

So, uh, with that, I'll hand it off to Sergey for, uh, for demo. Okay. Well, uh, so the kind part of a demo would be period between the two VPCs, uh, through the gateway.

And for that we just have a VPC one and V VBC one overlap, uh, not very creative name, but, uh, shows what is, what's happening there with server one and server five having the same IP addresses. So if we go back here, I'll stop Inc. And, uh, I'll show, basically we have on server one, we have 10 51 2 and the same 10 51, 2 and zero five.

So like, without math, we cannot ping between them, right? Uh, because we cannot, uh, distinguish between them. So, um, we just can take a look on another period, uh, between those three VPCs.

And that period is a little bit more complicated because, uh, we, we define a nut here. And, uh, essentially, uh, the only kinda real different that we just say that, uh, we take the, the, the C of the, of the VPC and we export it as a, as a nut space, uh, for the, uh, for the first one would be 11 zero slash 24 and 12 zero slash 24 4 for another one. Um, so what we do under the hood, I actually, let me probably just start the ping first and we can talk about it for a bit.

So you can see Ping is not working. We, we don't know yet about this IP address and we will apply it now. Um, so when it's, uh, when it's applied, uh, like the way how it works, actually the gateway already knows about all the VPCs.

We propagate all this information there. And if we have like Chris pointing confi presented there, the only thing that's happen is we basically saying to our data plane on a gateway is to, uh, like what, what to do with this type of traffic. And, um, uh, gateway advertises, uh, the routes, uh, for those, uh, not at, uh, IP prees as, as Manish was saying.

So actually, well, you can see like traffic is already going. Uh, we can, we using the not at IP address, and if we go to the switch, um, we should be able to see it here. So we just, again, looking on the, on the same, uh, VPC one euro to, and you can see now that, uh, we actually, we just received, like a minute ago, we received another, uh, route from BGP.

We advertised the gateway advertised the, this method, uh, subnet, uh, for the second VPC. So the first V VP C actually knows how to reach it. And this, this is a VAP IP on the, uh, on the gateway.

So essentially that's kind of simple looking, but uh, there is a lot of, uh, uh, factory to do to properly calculate and advertise those routes. And that's all like transparently handled by the gateway. So that's how that looks from a switch side.

Um, so essentially if we go back, actually no, there is nothing interesting to show here anymore. If we go back to the, uh, RAF cloud, we can see that now that, uh, traffic is changed a little bit. So first of all, we still have, um, traffic on the, uh, down links from spine to the lifts, but now we have, uh, the first one in the list, um, the port E one five that is, uh, the uplink to the gateway, uh, and we, it have a double traffic because all the traffic from V PC one goes into the gateway being knotted and goes back to the spine.

Um, and you can see like, uh, down on this list, um, we have few more insights. Um, like basically because we do the whole data play in the gateway, we have way more insights into like how traffic is actually flowing. It's, that kind of information is not available on a Mazda with switches, especially on a, on a tomahawk series.

It just doesn't give you insights into the traffic. But with the gateway, with our own data plane, we know actually that it's not just like some amount of traffic going. We know that it's going from v PC one to VPC one overlap and back.

So that allows us to provide a lot of, uh, additional insight into the traffic and we can, uh, do actually even, even more like we, we, we looking now into like, what else, like interesting metrics we can provide for it. But that's like in one, one of our examples where it's very easy for us to add, uh, way more insights into the traffic patterns. And that's where we as, as we do the knot and firewall, we can do all the counters for them as well.

4 milliseconds. So it's about 10 times slower having, having to go do the detour with the in cap to the gateway and back you that and not, not just doing that, that, um, I dunno how that stacks up as you start adding stateful firewall to that as well. And you know, so I I guess it's a trade off, right?

You get a lot more flexibility and functionality, uh, for the sake of 10 times the latency. Yeah. So that, that's a very good question.

Uh, the thing is that we, uh, post this demo, we want to demonstrate the, the, the kind of the whole system, how it works, the look and feel. Uh, we're using the debug images, which is doing probably five times amount of login that you actually receive in the traffic because we just want to see like what's happening inside. We Log every packet as strings.

Yeah. For example, in this debug image. So, uh, some of the latency and stuff that you're seeing is, is That it's not representative because it's, it's, it is debug images and there is zero loading used in on purpose.

So we see all the, all the behaviors. So it's not a representative latency No. How it would really be 'cause you are Yes.

Effectively is dev card with debugging. Yeah, yeah, yeah. This is the debug myself.

It couldn't be worse than this situation. I mean, look, I want everything for free, you know, I want to start, we'll keep note LA but, But you have a fair point. So, so, um, there is gonna be an increase in latency because you're taking more hops through the network.

Um, you are going to, there's buffering on the nick now if it's hardware accelerated, you do get some cut through on the EWI fabric, on on Connect X. And then I think on Thor two, you, you can also get some of that if it's hardware accelerated, but there's gonna be latency penalties. Um, what you're seeing there is pure, it's a debug image and uh, it's very slow.

Um, because it's not, yeah, it's not even optimized. The amount of logging is you got The equivalent like debug IPO Or Oh Yeah, exactly. Yeah, Yeah, yeah.

So, so that's what's going on there. But, but it does bring up, um, uh, something I wanna talk about on our roadmap, which, um, is that there are scalability limits to this, right? If you have, you know, a thousand hosts each connected at 200 gigabits per second, or a hundred gigabits per second, that's a lot of gateways.

If you, all of your appearings actually have traffic, and if you look at the way the big clouds do this, they don't actually route to a gateway. I mean, they did like 15 years ago, but they don't anymore. They actually have everything operating on the host.

And so we do have on the roadmap to take that gateway software and run it on the host and in particular, um, on dpu. So things like, uh, Bluefield, um, that, uh, Bluefield card is essentially a modern Bluefield three is a connect X seven Nick, um, with, uh, what is it, a 16 core arm server, eight Or 16 or eight or 16 core arm server with 32 gigs of ram. So that Nick itself is actually a server with the Connect X seven Nick in it.

And so you can load all the gateway software onto that piece, and then all the functionality happens as it exits the host, much lower latency, much more scalable. Um, and actually because we're now terminating traffic, um, we can actually extend the overlay network from the edge of the switch boundary into the host. And that actually gives us a lot of other functionality that, um, you can now extend the network where the port connectivity is VMs or containers not, um, not hosts.

And so you can actually now extend the network into, uh, that virtual environment if you wanted to. Um, you can do that for AI workloads, you can do more hardware acceleration because you have the server that's running on the side so you get more options. So this is like, um, so if you really want the best performance and scalability, then you're gonna have to run some type of DPU.

Uh, they're not cheap, unfortunately. Um, so we don't see a ton of people using duss they do on AI clusters because the cost of the DPU compare it to the entire, uh, switch. Uh, entire node is not bad.

But, um, if you're just running a standard like VM server or something, uh, that DPU can be like four to six grand just for the DPU. Um, we're hoping that there'll be some lower cost options. I, I know a MD has their s ondo, but it's a different architecture.

It, it's unclear how much, uh, you can offload onto it. So we'll see. So one quick one for you.

Um, when you showed like the different peering that was all within the fabric, can you, I'm guessing you can do like layer two, layer three peering outside of the fabric Too. Yeah, yeah, yeah, yeah. So, so the cool thing, the cool thing about it is, right, um, normally people think, like told you, you plug the switch, the, the gateway into one node, into one part of the layer two segment, then the other side goes to the other layer two segment.

The cool thing here is what you actually do is you configure the switch ports, and if the switch supports L two VNI, you use an overlay network to extend the layer two network on those two, on however many ports you have, and you extend that layer two network into the fabric. Now the, the gateway's always a layer three device today. Um, we may allow full layer three extent layer two extension at some point, if you wanna do BG p peering or whatever you use BGP multi hop.

Um, AWS doesn't support BGP multi hop. And so we'll have to at some point allow that L two to extend all the way to the gateway. But that's just a matter of functionality on the gateway's data plane to understand type two routes, um, from B-G-P-E-V-P-N, which it, the data plane doesn't understand those routes yet.

That's why there's that limitation. So we have just extend it to do that. Great.

com GI Hedgehog.