HPE ProLiant Compute Cooling Technologies

My name's Keith Sauer. I'm a engineering manager here in Houston, and PNE is our principal, uh, thermal architect. And, uh, he's gonna talk about thermal cooling, liquid cooling in a minute, but I wanted to set him up and talk a little bit about our engineering team here in Houston that's responsible for developing our chassis that you see on all of our servers.

So my team is made up of mechanical thermal tooling and packaging engineers. And so the mechanical engineers design all of the injection molded plastics, the stamped sheet metal and the dye cast metal parts that make up the servers. Um, it includes things like, uh, hard drive carriers that are toolless and allow you to swap out, uh, hard drives quickly and efficiently.

Um, rail kits that allow you to pull the servers that of the racks so that you can service them cable arms that handle all of the external cables that allow you to pull the server out and without turning it off. So mechanical engineers do all of that. The thermal engineers design the thermal solutions that include the fans, the heat sinks and the cold loops.

Uh, things that make up, uh, the all of things we have to do to cool the servers. Uh, the tooling engineers allow us to mash produce, uh, chassis all over the world, uh, in high volumes. Uh, we're building, uh, four servers a minute, uh, 24 hours a day, seven days a week.

And we're assembling those, uh, in factories, uh, across the world. So the tooling engineers allow us to mass produce these parts in, uh, uh, uh, high volume, uh, low tolerance conditions so that they can assemble anywhere in the world and go together without any issues. Um, once we, we build and, uh, and test the servers in the factories, then we've gotta get the servers to the customer.

So we've got packaging engineers that design all of our boxes and cushions and crates needed to ship these servers around the world on boats and planes and trains and automobiles, and get there in a way that when the customer gets them, they fire them up and they work, uh, um, every time. Um, as we're designing these chassis, um, our biggest challenge coming up is the cooling of these boxes. And back in the good old days, about three years ago, we, u thermals was an afterthought.

We basically shoved as much stuff as we could in a one U or two U box, and then people asked us to shove more stuff in there and we were like, no problem, we'll figure it out. We built a prototype, we would test it. Uh, things might get a little bit warm, but we would throw an extra fan in there, maybe add a more high performance fan, uh, get a more efficient heat sink, go from a, an extruded aluminum to a heat pipe to a vapor chamber, and then we would design baffles to direct the airflow where we needed to, to cool the server.

We just figured it out and we're, we're no longer able to do that. And so I think that sets pne up to talk about our thermal challenges. And, uh, I'll turn it over to you.

Thank You, Keith. As Keith said, my name is Pne Mera, I'm a thermal engineer in the HP compute group. And as Keith pointed out from few years back, yes, everything would be designed and all I had to say was, thank you and just take it and do whatever with it.

But that's no longer the case. So to begin with, I'll jump into air cooling. Yeah, probably you may be thinking, ah, we are here to talk liquid cooling, what are you doing?

Air cooling. So, but liquid cooling, yes, that's coming up. That's we even, it's inevitable we have to go to liquid cooling eventually, but where are we today?

Where are the customers? That's the thing we have been, uh, trying to study or understand from our customer base where they are. What can air do for us or what can air still do for us?

So from, for three generations, I would say in a one new form factor, we have developed heat pipe, heat sinks, and I would say extended volume or extensions that go in front of the dim section to capture as much heat as possible or additional air flow surface area to cool as much, uh, processor power as possible. And we are kind of out of the room space and we still cool, we air cooling is still very prominent, so we still use it. So the next question was like, what can we do next in this space?

Can we, uh, take it, uh, or cool even higher water processor? How, how can we go? So that's where we kind of came up with, uh, the concept of the closed loop liquid cooling wherein we remove the heat sinks from the processors, install our, the two cold plates with pumps in there, and then we route the radiator that still sits between inside the server, between the hard drive back plane and the fans.

So obviously the space is limited, like I don't get what I would like to. So to make this fit in the space, I go from what I call the dual rotor fans or the counter rotating fans, and I have to cut it down to a 28 millimeter fan so I can fit my radiator and still get, or, or I would say, get better cooling than my performance heat sink. So if you guys, so these Are still air cool.

This, I still call this air cool. Yeah. 'cause the, the, it still has a fan in the chassis and everything.

You're moving the heat first. Yeah, so this is basically, and I have just kind of beaten it up left and right and you can see, so this is what it is. Basically it's just a radiator connected to the coal plates with the pumps in there and we pump the liquid and stays within the server and it is air cool still because nothing is going outside the server.

Is that Stuff you developed yourself or did you use a company like a jet Cool to, So we used one of our vendors, we kind of gave them the design on what we are looking for or the dimensions and stuff we have and said, Hey, something like similar to the heat sink, right? So yeah, we just asked, uh, our, one of our partner vendors to say, Hey, this is our requirement, this is what we like to see, uh, and this is the bump up requirements and stuff. Okay.

I see that in the two U where you have the extra space, you have this third tier called max performance. Yes. Um, is, uh, is the closed loop for one u sort of a, a, uh, a clever way to, um, you know, approximate max?

Yes. You can say that's a closed loop is kind of a max cooling in a one year space. Well, but is the close loop we've ever read reached a bowling point that it needs to be refreshed?

Is there a half-life for the coolant? So typically say for, so it, what we use in there is still PG 25, like what we use in DLC. And we do have some extra rewards.

We have extra coolant in there so we can, uh, keep it running. And typically the life cycle is all lifespan of it is close to six years kind of thing. But they're having, and that's being conservative just so that hey, we don't want anyone to run into kind of heat issues and stuff.

But that can be extended, uh, or it has more, uh, 'cause over time you, you start to diminish the, the quality of the heat transfer, right? Yes. So six years we should be able to get that.

And then a, after that we may start seeing some impact, but uh, yes, it's five years for sure. So depending on the life cycle of the product of what we are doing, we do have runway for the sixth year from a recommendation perspective. We say, Hey, just swap it out at that point.

And once again, that's in the, that's at the back plane, the closed loop, that's where the radiator is. So Radiator, uh, uh, uh, the radiator basically sits behind in the, behind the back plane, I would say. Uh, it's in the space behind the back plane where the fans go.

Yeah. So the fan is right there. So we just remove or reduce the depth of the fan and we drop the Do you reference any or recommend any, uh, like data center airflow adjustments or anything like that?

No. For this? No.

Yeah, That's, No, this just from a pure server perspective, what can you do? Hey, if your performance heaping, for example, I'll just pull numbers. Say you can do 270 watts or 300 watts.

Now if you use this because of the, uh, additional surface area and more uh, air capture, you can hold 400 watts, four 50 watts also on the process, Any kind of air cooling that uses liquid. Yeah. And then in a two U space, because I'll kind of stick with one U and two U discussion.

So as you see, we have brought it up, it's a max cool max performance heat thing. And obviously this was the other thing we kind of pushed are like, Hey, what can we do in two U space? How much can we push?

And when we, when we brought this design up for the first time, uh, ob the obvious question was, what are you doing? Nobody can service the memory and that's not acceptable. So like then we optimize the height of the fins on the top such that we can service the dims not very easily the way you would without having these fins in there.

But still, without having to remove the heat sink, you can still get your, uh, fingers, easily get to the, uh, the memory slots and remove the dim. It's not as easy as not having these things. But if we need more, better cooling for a higher water processor or you need to lower your fan power consumption, you have an option to use.

Alright. With that said, obviously the fans, I just kind of listed there. There's nothing new, it's just whatever has been done traditionally from the fan perspective, you define it based on impedance system and the components that we are going to Cool.

Getting into the direct liquid cooling of the DLC as we know. So I have a core plate module of where we are today or what we, how we install it. Basically again, we remove your, our heat sinks, we have our cold plates, and then this is one of the way we route it out of the system.

So this bracket is, can go into the PCI slot so that it keeps it steady and while you look at it and say, hey, these tubes are just too short, you just can't get to your manifold, which eventually, or the pull plates need to be connected to the manifold. So what we did here is we designed this common module, so it can be used on any of our chassis because we have different depth chassis from small to medium to large. So based on that, if we can, we just get this out of the system and then we have an extension tube that runs from here to the manifold in the system.

And then the manifold connects to the, the CDU or the cooling distribution unit that sits on the bottom of the rack. And that in turn is connected to the facilities water. Are the, uh, are those, uh, quick disconnects, self-sealing, quick disconnects, Or Yeah, these are the quick disconnects and They're self-sealing.

Yes. And does the Loop run, are they serial or parallel? So, so This is in series, this Is not parallel.

So one one CPU is gonna run cold cooler than the other one. Yes. And typically it has been in the five to seven degree difference range?

Yes. Yes. You can do parallel to kind of up, uh, to balance the temperatures, but even after that, like we still, uh, run it overco, the processor temperatures are really cool compared to what you can get on the air Co.

Okay. And just, just, so, so the liquid coming in has to be what temperature? So it can be up to 40 degrees, but depending on the data center Celsius 40 degrees Celsius, what kind of deltas are we getting from the effectiveness of the cooling liquid versus air?

What levels of delta meaning the table drop? So if I run, so typically the air cooling will keep the processors at the defined, uh, temperature or the target temperature as defined by the vendor, like be it Intel, a MD or Nvidia. So it keeps it right there.

But what uh, liquid does is if, depending on the, you can just set the, uh, LPM depending on the vendors, you can have a delta, delta p uh, operating pressure across CDU or you can define your liquid through there and you can easily keep it 10 degrees below the operating point. So you're over, over cooling it, uh, to get into, say, the turbo mode or you have your turbo frequencies, you have a base for product processor, and then turbo about that is opportunistic. You can get in there because you're keeping it more than 10 degrees cooler than where they're supposed to run.

That's good. So obviously as I've mentioned, liquid DLC requires the liquid in the data center, your cold plate modules, rack manifold, and the cooling distribution unit. So yes, there is some CapEx involved early on and the data centers are moving towards it, but it's not, it's a slow gradual, uh, adoption.

And when we talk about liquid cooling, one of the biggest questions that come up is leakage, because leakage just freaks everyone out. So as a first step, what we have done is we have implemented a leakage, uh, detection inside the server where we have the leak detection band or the mesh around both closed loop and the DLC. And these loops are connected to the system board with a power connector.

And it's the same power connector that provides power, power to the pumps on the closed loop. And it also can, if it senses leakage, it sends a signal to the system and the IO and the system is immediately shut down to prevent it from burning up or whatever catastrophic, uh, failure may happen. And obviously the system will not be able to reboot or it'll prevent system boot, uh, before until the bit is cleared.

And all these actions, uh, are logged out in our IO and all places. Now we have talked about liquid cooling being really expensive at, uh, from a CapEx perspective and there are concerns of leakage, which we have to address. So why liquid cooling?

If I have to kind of define it in a very simplest of terms, I would kind of place it like this where number one, power and performance. So as we know, the power of the GPUs or CPUs, everything is just going up and not just that sometimes with the up the increase in power, the, uh, to maintain a certain frequency, the case temperature as defined when the vendor goes down. So it's a double whammy at that point for us.

So for that, for a reliable operation of that, uh, component, we need liquid cooling, but that's the need of the hour at that point. And then the other thing as I just mentioned, turbo. 2 gigahertz with air, it's that turbo is opportunistic.

So yes, can I get the turbo frequency up to a certain point with air, but I am going to consume a lot of fan power or I have to spin the fans almost at a hundred percent to get to that point. But with liquid, we save the power, we can be in turbo mode for a sustained period of time. What Does your p OE look like when you're doing that?

So that's a tricky one, uh, because from a data center perspective, right? So we are kind of doing this from uh, uh, a server perspective and we are trying to piece that I, I'm trying to deflect this just for the reason I don't want to make a, It sound like it gets better when you're going into this mode 'cause you're driving the fans down, right? Yes, Definitely go.

It gets better. And I can show you, I have couple of slides where I show what gets better or how we estimated certain things, uh, and have couple of slides that I can show you how, and I've kind of kept it from a power consumption perspective of what we see from our, I just look relative. Yeah, you're talking one server liquid.

Cool. I'm assuming that if you have a rack full of servers, you don't have a hybrid, you don't have some that are air cooled, some that are liquid cooled, and if you do, would the air cooled be on the bottom and liquid cool. Be on top?

And how many, well, last question being how many liquid cooled servers can run in this environment? So typically we do not ship anything that's a mix, but the customer can do it. Um, so at that switch, so what Scott said, uh, said is there are switches in the rack, those are airco upper rack switch.

So top of the rack switches. So to answer your question today, as it stands, everything airco goes in the top of the rack and servers are at the bottom. And it's kind of easier also because you have your CDU at the bottom.

And so you just connect it, but you have the capability, uh, to connect it from wherever. I mean, I'm sure actually you could put a mid rack Yes. In there.

Yeah, yeah. You, you Can, yeah, it doesn't matter because the manifold is designed such that you have so many, uh, qds with quick connects. So it doesn't matter where you install your server, you can connect it.

Well, I mean the liquid cooling only covers the CPUs doesn't cover the rest of the system. It's generating heat as well, right? So yes, that's gotta be taken care of.

And there's a ratio that's discussed a lot. Sometimes it's 70 30, sometimes 80 20. So typically yes, the 70 30 is on the higher end, but it can be 60 40 also depending on what kind of processes you use.

And if, if, say for example, your total server power consumption is one kilowatt, but your processors are only 300 watt processors, or if you're dumping only 600 watts to the processors, uh, of the, to the liquid, then you're 60 40 split now. And, and this line that you're talking about, we're not gonna discuss the NVL 72 kind of stuff, right? That's a completely different line of servers from HPE.

So we're not talking about the big NVIDIA boxes, we're talking about the H 100, H 200 Here. What I'm talking is primarily the mainstream or enterprise level servers, which are processors are the key, right? And not even everything else is air code, even the PCI cards or PC PCI version of GPUs or Air code, right?

Yes. Once you go to your NV four NV tools or anything on a board kind of a MG X, then yeah, that's a liquid. But what I am talking here is primarily from a ProLiant mainstream enterprise.

Yep. Customer. Thanks.

Can you talk about the efficiency side of it? I will show you something. Great.

Now I've done the perfect way Everybody. And to answer your last question, yes. If you, if you have a 42 U rack, you can easily fill in 40 42 U one U server, 42 1 U servers in there without any, how many, How many servers does that cooling box handle?

So that typically, it depends. There are different capacities for the cooling distribution units. They can go from 80 kilowatts to 150 kilowatts.

And if you are only dumping one kilowatt per server, say if you have a hundred wat processes, so you can just put in hun you can connect a hundred servers also, if you are capable of doing it typically, then you'll have to kind of do all kinds of, uh, tubing to the next rack and stuff. So it just gets complicated, but you can do it. Today's capital.

How do, how Do they lock together from the unit? Uh, we, I saw the ends on that. Is it like a snap block?

Is it a twist lock? How's it, how's it, You just kind of, you just press it in and it just, it's a quick disconnect. It just locks in place.

Okay. It's Just when it gets disconnected, let's say you're, you're taking down a server, it gets disconnected So you just have to kind of pull Yeah. Like a Cup underneath or something like That?

No, typically you, oh, I don't have the female disconnects. Yeah, they're dry disconnects basically. Okay.

So you just pull on and just comes out. It just dry. Nothing drips from there.

Dripless also you can call it. Okay. Okay.

So again, so power performance is one piece of the thing. And again, what we talked about, hey, if you have a 42 U rack and I want to fill it up because space is premium for me, I need to, uh, use up the density I have, I can't go to a two U server or anything. Yes.

That's where liquid cooling helps. And then finally, efficiency, or you can even think about from a sustainability perspective that hey, how can I help from a sustainability perspective, from a cooling side, oh, I need to lower my fan power consumption. That's the key.

And how can I, how can I do that when if I use liquid, it'll be, uh, I will be able to lower the fan power because the processors are typically the more the fan, uh, the are driving the fans. And when we go there, the fan power drops or the speed drops and the fan power drops Is this, uh, also being designed for the GPUs as well as the CPUs. So yes, in the mainstream, uh, we are doing on CPUs, but we do have other servers where if you have your GPU uh, on the system board like NVL 72 or NVL four, uh, or we have seen earlier the DL 3 84 kind of, uh, uh, system we can have.

Yeah. 'cause you have two factors. You get thermal footprint as well as the power footprint.

You're trying to decrease both those curves, right? Yeah. So because the power is going up now, we have to cool it regardless based on that.

And so that said, one other thing I wanted to kind of touch base on, uh, or would be key in the considerations is the data center rack power. So I just kind of took a screen capture from the uptime institute survey that the data in 2023 and 2024, and they compared it obviously as we can see here, the rack densities for the lower, uh, kilowatt racks, like less than 10 kilowatt is going down. Uh, it went from 41% to 34%.

Similarly, the 10 to 14 has gone down. And then we are getting an increase in power. Uh, the new data centers that are being designed, we are like the power is being upped on the per rack basis.

But if you look at it even between 2023 and 2024, the cysts or the racks that we surveyed and are below the 29 kilowatt number are 86%. In 2023, it only went down by 3%. So 83% of the racks are below 2029 kilowatt.

So obviously it's an easy answer if you have liquid in your data center. If you rack density is very critical to you if you need to fully populate the rack with the servers. And if your rack power is greater than 29 kilowatt, yes you can do liquid cooling.

But what if my customer has a rack that is less than or equal 19 kilowatts and they're only populating 20 servers or less than 20 servers because of the power capping at this instance, yes, in the future things will change, but if they're only, uh, able to, uh, support only 20 less than 20 servers in the rack, and if they have no liquid in the data data center, what can we do? And that's other aspect of things we have looking at also from air cooling or that's why we are still focused on air cooling as well. How do we balance this out until, uh, customers get to the liquid cooling side of it?

And if you look at it less than 19 kilowatts, there's still 70, 76% is a huge number still in that range for the power. Okay. So that said, with air cooling, liquid cooling and the data center power, so one attempt we have done is to show how, or at least show a trend on how things or the power will drop.

So what I would say here is what I did is I considered a one kilowatt as a reference just because this can change with, uh, depending on the processes used or the configuration we have for the server. So when we have, say, a one kilowatt rack and oh, so one kilowatt server and we have a 15 kilowatt availability for a rack, so you can only install maybe 12, 13 servers in that rack. So if I have a one year server with my performance heat sync installed, I will kind of use a huge, uh, very high power number of power wattage.

And if I, uh, use a closed loop, you can see it drops and further as we go to the two u, uh, and DLC, we have a trend that shows, hey, you guessed you are reduced your power significantly going from one U to two U and even DLC. And obviously you can see in this scenario, because it's only a 270 watt processor, my two U heat sink can do, uh, almost similar to what DLC can do. But the DLC is still in one u uh, space.

So you can have 40 42 rack, uh, servers in the rack and my to you, you only get 20 servers in the rack at that point. So going from the performance heat sink to the direct liquid cooling, this is a, uh, 1200 watt, Something like that on the yes rack. Yeah.

And uh, yes, I have that number there, but it's kind of a reference. So it won't be exactly the same. It'll depend, uh, uh, based on the mileage or you can, you can call it our what processor you have, what configuration you have in the system, things can change, but that will go down drastic.

All these are are compliant with OCP, right? Sorry, all these are compliant with OCP Compliant. Uh, what do you mean?

Yeah, they They're, they're using reference specs. They can, like if you have an OCP environment and you decide you're gonna go multi-vendor uhhuh to say We are, yes, we are getting there today. How we ship our, the DLC is, uh, we have a complete rack configured in factory and send, uh, ship it to the customer.

Mm-hmm. But yes, we are moving towards, yes, we are open to that, uh, OCP kind of thing where you can start mix and match and Mr. Schaffer is very high on that also.

So wait, so I know, I know we're talking about liquid cooling, but I can't help it, right? It's, He's Gonna, He's gonna, he's gonna like, so these are the two bars you should look at. Mm-hmm.

Please ignore liquid cooling for a minute, right? You saw in the previous chart that most customers can't deploy more than 29 kilowatts in rack. Right?

If that, so if they can't do that, why are they buying one used servers? Because one U servers cost them a lot more power than a two U server in power to cool them, not empower to operate them. In fact, his design, which I love this, the, the server's a kilowatt regardless, that's a baseline, right?

So you have the same CPU, the same memory, the same SSDs in both servers. One U2 U, you pay an extra a hundred watts of server a thousand watts total in this setup that he built with 10 a thousand watts just for the fans. Mm-hmm.

Why are you doing that? Is my question. Only use one U when you need it, when you know, you absolutely have to have it when you're willing to pay that extra cost for the fan power, go for it.

Right? Or it's when it's needed. Absolutely.

But if it's not needed, go here. Mm-hmm. Go down to the, to you trade off density.

My message, and I don't know if it's per Renee's message, my message is we have chase density for too long and it's just caught up to us where we can't efficiently power it anyway. Right? We can't, we can't, our racks can't handle it.

Well, Data center, data center space is sold by the kilowatt, not by the square foot. Yeah. So you're right.

I mean, what you're saying is spot on. I mean, it makes no sense. You're buying the power, you're not buying the physical space, right?

So Right. And if you have liquid code DLC Yeah, sure. Then go for your one use and you, if your density is very critical, you have 30, 40 kilowatt uh, racks.

Yeah. Populate them. It's just, there are certain scenarios where density is really important.

I'm thinking like financials in Manhattan, you know, Or HPC class workloads, right? They're big money calls that Actually I deliberately picked, I deliberately picked, actually, I'm gonna disagree with you because Yeah. And yeah, so, so the power in, let's say the New York metro, uh, and I speak from experience on this one, the financials, um, they pay exorbitant amount of KVA, the, the, the, the constraint is power, not space.

We got, I got space, it's the power that's, that's the constraint. So designing for power, I, I, I'd you can go the extra X and go with it's, it's not about, it's about that, that amount. You mean you're paying hundreds of dollars more per KVA than you might in say, I don't know, pick out, okay.

Utah or something, right? Texas turns out Texas Pretty low cost power. But look, I'm not against one u I'm not against it.

I'm saying use it when you need to think of it as, we should think of one use as more specialized than we used to. We used to think of them as the general purpose pizza box server. Right?

That's what I mean I grew up on, right. But now I'm saying, hmm, they ought to be think of them as a little more specialized or when I really need that, that density play. They're the great solution for it.

But when I don't, I should be using to you. So from there, try to make another attempt on this slide where the previous slide showed the total rack power or the cooling power in the rack for 12 servers. What I'm showing here is still the safe same five scenarios for performance ing to DLC five.

On the bottom here is the purple and the yellow together are the cooling power per server. And the reason the DLC has two different colors here is one for the fans that are in the system and then a per server basis, how much, uh, how are you consuming from the CDU perspective or the cooling distribution unit? And especially now, given you only have, say, two servers in rack, whatever the DU is running, operating at, if you are 40 and you're operating the same power, you're consuming really low, uh, from a CDU perspective.

But if you only have 12 servers, you're at least, uh, consuming a certain amount on a CDU pump also. So per server basis, you can see that trend still what we saw previously with the two U max performance heat sink and the one UDLC at the, at similar level, this is still lower, but a similar level. But this is only for this scenario.

If the power of the processor goes up, this kind of stays almost similar, this starts going up. So you get bigger impact or you see more Improvement. Yeah.

The, the closed loop, right, there's not additional impact for the cost of capital equipment and, and you know, bringing liquid to a cage, the, the, the extra, that tiny little bit of additional power savings you're getting from the two U air cooling going to the DLC in that scenario, the, the amount of money it will cost you to get the water in an existing data center to that rack makes it a ridiculous choice, honestly. Right, right. Is no choice.

No, but it depends on what you want. If you're looking at a, say 500 watt processor to, of those in there to cool it that like yes, it, it's, it's dependent on the requirement as Scott was, uh, if it's needed, if you know you can use it or just compare and see how and when it makes sense. Yeah.

I mean, but the economics of it. Yeah. Yeah, exactly.

Part that like even at at GTC Jensen wasn't talking about that you're gonna spend just as much on liquid cooling infrastructure as you will on that super pod. Right. In many cases.

So same thing, same thing as the liquid cooling. It shouldn't be underestimated what it costs to get water to a cage. It's not fun.

So Yeah, CapEx versus your opex, how much are you going to really save eventually? That's the math that needs to be done. Is, is there, is there a difference between the regular power or the high power systems now?

'cause pretty much everything is like switching over to high power, high voltage, you know, talking to like CPU storage. So is this even, even considering regular low? So we still have a lot of customers who are in a 205 watt processor range.

Yeah. They have not even kind of gone to a 500 because what they need from a course perspective, your frequency perspective, they are still right there. So air cool.

And if they don't have enough power in the rack air cool. Is their best friend still still Good sweet spot. Okay.

So yeah, that, those are the things that need to be evaluated. So this is just an attempt to show, hey, just a reference, think about these things. Also, it's not just like black and white that hey air I'm jumping to liquid.

Like no, just evaluate where we are, what we are, uh, trying to achieve and how we are, uh, planning to go there. Yeah. The only, the only piece of advice that'd say for you guys as you're gonna market with the, with the right most with the DLC is just make sure your sellers are talking to the customer's data center providers and facilities people early in that conversation because you don't want that left to the end.

I mean, it's, it's not that it can't be done and not that there's not use cases for it and that and that some folks are gonna be great at it. It's just that is something that must be discussed early, early on. That was the, that's kind of what we've learned over time.

Yeah, Agree. So one thing I've kind of attempted to do here or kind of shown is that this top light blue section also, and the reason, and this is an estimation and an estimation only because the data centers are different. They're designed the way they are.

I just wanted to show if there is a typical data center, obviously, which kind of users, your cooling towers, your chillers, your air handling units, and if you have a lot of air handling needed and stuff, what would be the data center? Power per server is what I'm showing here. And if you look at the air cooling, yes, your savings, uh, server or data center, power per server, even going from a one U to a two U max performance, it's a little server saving.

But if you go to DLC, you can tweak your settings, you can change things in there that can help, uh, uh, save more power. It's not just at a server level, but if you get into DLC, things that need to be considered as what is your, uh, data center efficiency or how you are operating it are, uh, as was mentioned earlier, what is the PUE and where you are operating that.