The Resource Costs of AI

I am Tom Hollingsworth, the event lead for all things related to networking. And we are here with a networking field, a delegate round table. The topic that we wanted to discuss today is probably the one that you're honestly tired of hearing about.

It's ai, but we wanted to dig a little bit deeper into this. We're not gonna be talking necessarily about GPUs or inferencing models or whether or not the AI models have finally figured out how many Rs there are in the word strawberry. I'd like to talk about cost because we know for a fact that this market has seemingly limitless potential and seemingly limitless investment opportunities.

But who's actually paying for everything that's getting moved around? Because you are probably thinking to yourself, oh, well, I can just use chat GPT or Claude for free. But those models don't work for free.

There are costs associated with those. So let's dive into this topic a little bit. The first thing that I wanna bring up is an article that I saw last week, and it has to do with ram, you know, the brain of your computer.

As I like to tell my in-laws, RAM is something that every computer needs to operate. Have you tried to go out buy a stick of DDR six recently? If you haven't, good luck, because there really isn't any available.

All of the memory that is being produced right now is spoken for. It is going to server farms to do ai. So if you were wanting to build that hot new PC with that amazing graphics card that you got on a, you know, special consignment, good luck.

What are we, how are we gonna deal with this problem? Because we can sit here and talk about the advancements of these, uh, compute models and agents all day long, but I promise you ain't none of 'em gonna be running in four gigs of ram If you guys run into this problem. Have you seen any issues with trying to get components, Karen?

Uh, well, yeah. I, I used to build computers. I don't anymore.

I just buy whatever now. But my stepson is huge into gaming and him, and most of his friends have been, you know, complaining, uh, loudly mm-hmm. That they cannot get parts for, you know, they can't get GPUs, they can't get memory, just like you said.

So they can't do what they wanna do with their, so they're all moving to like consoles and different things like that, which shifts the market a little bit. GPUs have been a problem forever, right? Like, even if you could get ahold of one, you'd have to mortgage your house to be able to afford it.

Like, I can remember when buying, spending $200 for a Voodoo card was expensive. And now it's like, oh, you want that hot new ti whatever, whatever, four grand for A GPU. There's some $30,000 GPUs you can buy too.

And they're not big boys And they provide good performance, right? But like, look at how much performance they have. They have done above and beyond.

Uh, if you guys watch, uh, channels like Gamers Nexus, there was a really great breakdown he did of the fact that they're actually going and inserting frames into games generated by AI to justify the horsepower that these cards have. So if your game is running at like 60 frames a second, it might actually be running at 90 frames a second, because it keeps inserting frames into it because they've got so much horsepower in these cards, they have to justify it. It's like buying a Ferrari and then parking it in a garage until your teenage son and his best friends steal it to Joy ride around Chicago, allegedly.

So speaking of, um, GPUs, if you have eight GPUs and you have it in an H 100, um, that is gonna require about three times as much power in a day than an average US household. Mm-hmm. And how many data centers have just a single, um, a single server?

So this is the power utilization and power requirements is gonna affect us all. Well, We, we've seen that, right? Like the, the fact that they're trying to restart reactors at Three Mile Island, the fact that they're starting to investigate doing small modular reactors to be able to provide the, our, the fact that Amazon is trying to build nuclear power plants for places with the understanding that they're gonna get the power draw first, and then it'll serve the community after the fact.

When we were at Cisco live this year and Cisco announced that they were gonna be offering one of their UCS servers that had Nvidia cards in it, uh, I was flabbergasted to learn that the power budget for just the server was 15 KVA. That used to be the power budget for a very heavy rack mm-hmm. Worth of equipment.

And now it's one server with I think four GPUs in it. And they want to put multiples of these in a server. Remember Haswell, the MV architecture when it came out, do, do you know how much it was?

It was a half a rack that needed to be water cooled because there was no way air cooling could keep it going. So, you know, we're already looking at, we can't find ram, we don't have enough power for our data centers. Now we have to plumb everything again.

Which anybody that's built a data center in the last 10 years probably didn't plumb it. 'cause we don't use mainframes anymore. Well, guess what?

Find your plumber. Well just Go ahead and then I'll have to repeat. Oh, yeah, no, just, uh, thinking about what we were talking about earlier too is the power.

You know, if, if we're consuming that much power for GPUs and for data centers and things like that, how does that affect you and I and our household? You know, because the power, the power costs are going up for, you know, the data centers because they can, there's limited, there's finite resources. Mm-hmm.

So The, the grid only produce so much power. Yes. Your power's going up.

My power's going up. And that's what I wanted to surface was I live in Maryland, we're on a interconnect, which where the various power dis uh, power to consumer firms are bidding for power. There isn't enough power on it already.

So they're bringing up three mile Island, and they're, uh, actually, I think they just did a bidding process and Constellation Energy showed up with something like $5 billion worth of power plants they wanna build, but they all bid for power. And if the AI companies have with their big deep pockets or bidding, the price is gonna go through the ceiling for ordinary consumers. Not, Not to mention the fact that, I don't know if you guys know this or not, but, uh, 19 contractors can't build a nuclear plant in a month.

Like we, we have a huge problem with that. Um, Denise and then Steve. Well, I mean, we are very US-centric here.

Mm-hmm. And so we're thinking of what's going on in the us but if you look internationally, the power problem is being solved in a lot of different creative ways in other countries around the world. So, you know, maybe it wasn't a good idea to cut the whole research budget on, so on, uh, alternative, uh, power here in the us That's just Crazy talk.

You know what? There is, there is the possibility of doing alternative forms of power. We don't have to sit here and moan that, oh, we are, you know, we have this little bit of power here in the us.

We could be doing better. Steve and then Pete, what were you gonna say? The, the, the power problem is not coming from ai.

We had it already, you know, we were already short on power, at least in the United States and, and Europe before this came up. What's happening with, with this this AI boom is it's exponentially making it worse. Yeah.

We, we can't build fast enough to satisfy the amount of power requirements that it's going to do every year as it doubles and doubles and doubles again. Yeah. And, and we were already behind the eight ball.

And in the more populated areas of the world that are, that are dense, we were already fighting over power. And now it's, now it's, uh, it's getting worse. The other data point I've seen, however, is that, uh, home and other, uh, solar deployment in, I think it was California has vastly exceeded expectations swinging the power mixed solar there.

And in Texas where the governor is kind of anti, um, non-nuclear or whatever power, um, same thing. It's, uh, the amount of solar power is vastly exceeding what they predicted and helping solve the problem basically saved their butt last winter. I will say though, that one of the things that excites me about this is the fact that, and I speak from this is from experience because my dad was a general contractor on a, on a number of nuclear plants in the eighties, and then all of a sudden he wasn't because of Chernobyl and Three Mile Island, but we are starting to investigate other kinds of reactors that are a lot safer, like thorium salt, uh, shout out to the Kyle Hill YouTube channel for teaching me about those reactors.

But, you know, we could very conceivably find ourselves in a situation where we don't have to build these gigantic complexes that we've had to build for years with like these massive cooling towers and, and all of this stuff so we could potentially find ourselves in a better situation. Of course, if we can ever figure out cold fusion, then this problem is solved just like it was in Sim City, 2000 when we can finally get to fusion reactors. But, you know, that's, that's still a little bit of future way down there, Steve.

And then I'm gonna go to jd. Yeah. The, uh, and the, the big thing we missed on the nuclear side is we had extremely efficient and safe nuclear reactors since the sixties.

We put 'em in submarines, they worked well. We've had no accidents with 'em. We have a highly trained workforce that the government paid to, to train, to run them.

And we chose to not run those small efficient things anywhere else. There's no reason why that couldn't be running factories or small sections of, of cities or anywhere else. We need power.

I mean, other than the fact that they're classified. But, you know, Well, So, so I think, I think the, the other side of that coin is it's not just the power generation, it's also the power distribution. And we've been talking about the fact that we need a smart grid and, and that we, we are running on outdated architecture from, from power distribution.

We've been, we've been having that discussion for two decades now, at least if, if not much longer. So just simply spinning up more power distribution doesn't solve the problem. Uh, and then the second point I wanna make there is it's, it's not just a power, uh, concern.

7 trillion gallons of water next year. Mm-hmm. I mean, that is an astronomical amount of water.

And in some cases it, is. It, I mean, there's, there's plenty of reports out there. These, these aren't, uh, uh, in question that there are communities that are actually suffering from just simply the, the, the water consumption that these things, uh, uh, you know, put on, uh, the, the load they put in, into the systems, um, that creates, uh, so, so then you now have users who may not ever interact with AI intentionally, um, maybe may have AI used against them, in fact, but they're ultimately paying for AI through their power bill, their water bill, and let's face it also their taxes.

And we've seen that recently because the Chinese just released a design where they're going to sink a data center and use ocean water to cool it. Now, there are a lot of other problems with that. We all know that salt is an abrasive.

Right. But I mean, if, if it's gotten to the point where the most efficient way to do this is to just swap the parts out every couple of years when they get sandblasted by all the salt water, I mean, we've, we've kind of reached the peak of, of problems that we need to look at. Pete, did you have something?

So with, uh, shortness. Shortness and cost of power and water, is there gonna be a consumer backlash? No, because it helps me make cool videos that I can post on Facebook that show that I'm ripped.

Well, that's for people that are aware of the connection, but Right. Yeah. Why is my power bill so, so expensive?

I know what I should do. I should post a video of myself ripping my shirt off and tearing my power bill in half. And the, the, the underwater, uh, data center was actually tested already for, I think it was two or three years, but it was either Microsoft or Facebook, Microsoft, just in the last decade.

And it failed. They, they run, they ran all the data and they, and at the end of the experiment, they pulled it and didn't do another one. Yeah.

It failed under the economics of the time. And I think that's one of the things that we have to remember is that, well, I'll give you another good example. Shale oil production does not work if oil is below $50 a barrel.

But if it is above, like if it's 70 or 80, suddenly that is a viable economic thing. And they've managed to reduce the cost of that over time. I think what you're gonna see is that a lot of these things that people are trying to bring online, like thorium, salt reactors, sunken data centers don't make sense if I'm paying an absorbent amount of money to produce them for the amount of power or the amount of cooling that I'm getting out of them.

But when the AI overlord demands more resources now, it's suddenly a lot more appealing to do that because, well, that means I can get another a hundred billion dollar investment from insert company name here, Mitch. Yeah. At, at the same time, this is kind of the delivery and the, the resource side of it, supply chain side of it.

Same time we're the companies are making massive deals, you know? Mm-hmm. Open AI with AWS and yeah, they're basically, you know, locking down future supply chain.

But can we deliver on that? Right. Can we actually build those data Sets That's next quarter's problem.

I know it is. It is. But it's, it's actually, you know, we're building up the problem escalating it as well.

Yeah. That's, I mean, if you look at any of the AI skeptic pundits out there, they'll tell you like, you can't deliver that much material in any amount of time. Like some, what was it?

I think Ed Tron said that like, there's not enough metal left in the earth's crust to build enough of these things. Like a, a little, maybe a little bit flippant, but he's, he, he's probably closer to right than wrong. Like, you know, I think of like, you know, are we ever gonna get to the point in like Project Hail Mary where we have to pave the Sahara Desert and solar panels to feed this monster?

I'm kind of, another thing that comes up that I'm kind of curious about is, is, and we talk about power and how expensive it is for consumers because of, you know, whatever. I'm, I'm kind of wondering how that's gonna affect, um, I don't know what the, right, not migratory, but you know, where people live. In other words, the, the economics of power is much, it's very different from Washington to Florida to mm-hmm.

To St. Louis to, you know, mayor to Texas. To Texas.

Yeah. And so how many, where the data centers get built and where the power costs are and things like that. I mean, how many communities like California as a whole, how many people are moving away because of high costs of, of power, you know, To someplace cheaper and then show up in a place like Oklahoma that's potentially gonna build new data centers.

Exactly. We have lots of land. Yeah.

Well, that's what's happening where we live, because we have a nuclear power plant. And where I am at in Mississippi, AWS is building a data center. There's like seven or eight other major data centers that have been announced.

'cause we only have two and a half million people and we have a nuclear power plant that could probably power 20 million. So, and because of all the car manufacturers have all built their car plants there, I mean, I think we have in Mississippi and Alabama, we've got almost every car manufacturer has already built that infrastructure. So the data centers are like, Hey, the car manufacturers already got power and fiber and all of this infrastructure.

There's vast amounts of open land and the cheapest power in the country. So now in states that are largely agricultural, but have started to build infrastructure, you're seeing this mass vacuum of data centers move into it. So I wanna bring this back to networking 'cause this is networking field after all.

Right? And one of the things that keeps coming up in these conversations is, you know, the resource cost of what happens when people are building networks. Have you gone out and looked at any of the new chip sets that are being manufactured recently?

Uh, I promise you they don't look like your granddaddy's ethernet. They don't look, look like Bob Metcalf's ethernet for sure. Which by the way, greatest quote in history, Bob Metcalf said, I don't know what the future of networking is gonna look like, but we're gonna call it ethernet.

And we have that right now with things like ultra ethernet where the designs that are going into these things are assuming some interesting parallels, if you will. Here's a good example. Look at something from like Broadcom or Nvidia.

You know, those are two companies that are building these massive AI networks, right? Do you know that they basically require super Nicks and DPU to operate correctly because your system can't keep up with the offload for that IO because you're cranking that thing out at 800 gigabits per second. And you have to have multiples of those things.

And so if, you know, if the Thor Ultra card has to be in there for the Trident four to operate properly with all the bells and whistles turned on to do that new advanced ECMP to make, you know, ultra ethernet act, basically like InfiniBand does, we have to pay those design costs, right? Because the companies that are developing those switches that r and d is part of their budget. Well, how are they gonna recoup that?

Well, the hope is, is that there'll be enough customers for their networking gear that they'll get that back. But I, I have a funny feeling that people are definitely paying the cost of that r and d because the company is like, well, you know, we got, somebody's gotta pay for it. And we're not getting those a hundred billion dollar deals from OpenAI and Microsoft and Oracle.

So, you know, do, is there a possibility that our networking costs are gonna go through the roof through reasons that are not entirely our own? I don't, I think you're actually gonna say the opposite. I think anytime you, you push networking infrastructure into that, AI has gotta be large, it's gotta be commodity, right?

They're not going out and buying the most expensive vendor out there. They're going to have to build this at such a scale that it's got to be something because, you know, you use ai, I don't pay for ai, go in and, you know, go draw cat pictures or whatever. So it's gotta be able to be built at a scale that they can turn a profit.

And that will eventually trickle down into network engineering just like it did a decade ago when, you know, we couldn't conceive of, you know, we had custom asics and you know, unicorn, asics, and now commodity is the order of the day. So I'm gonna disagree with you on one point though. And if you go back to the open compute project, the way that it was founded, it was basically Facebook saying, we want you to build your networks with these parts because we know where to get these parts.

And when you start building more of that stuff, our costs go down. I would believe that if you were building generic data center switches, but if you're building these 800 gig ultra ethernet monstrosities, there aren't enough customers for those in the world to drive the, the price per part down to what I would consider to be a reasonable level. So I would say at least in the 800 gig world, maybe not in the ultra ethernet world, right?

But at least in the 800 gig world, I mean, that's what all the carriers are moving to that are transporting all of this AI data. You know, they're massively moving towards commodity and away from the mainstream vendor because they just, you know, it's the economies of scale. They have to Yeah.

Oh, Rita and then Denise. That may be true in the service provider space, but it certainly isn't in the enterprise space. In the enterprise space, we're still seeing a lot of, um, things such as AI pods being used.

So, um, kind of a simpler way to get into ai. Sure. Yeah.

But buying a rack at a time from a company like Nvidia where it's just like, hand me an ethernet drop and this works 'cause we've already plumbed it. And, and that's, that's gonna continue because I promise you that anybody who has a vertically integrated supply chain for those things doesn't wanna give up that margin. Mm-hmm.

Denise, Actually that was kind of what I was gonna say too in our discussions. Pardon me. Allergies, thanks California.

Hmm. Um, we were, we're assuming massive, huge AI data centers. We're assuming people, you know, doing videos of talking babies and junk.

But, you know, what about the, in the enterprise space or in the, maybe the more the health space and that the things were, that were actually useful uses of AI in automating your, your business processes and things. That's, that's not going to be that impactful I don't think. Right?

We're still seeing them go with Cisco, Arista, Juniper. Uh, they're not doing white box in the enterprise space right? Today.

Right. Today they're not doing that. And one of the other really interesting things is to have enough power to connect data center to data center.

Cisco introduced that 82 23, um, which is a router with a P 200, um, silicon in it. And it is able to connect those data centers at such a high rate. They can function as a single data center.

So it really reduces, You're giving me OTV nightmares. Don't do that, Steve. Yeah.

I I think it, this, this, this part of the conversation all comes back down to the classic network engineering. It depends. Yeah.

Uh, and what we say, uh, the, the, it's like most of the trends in, you know, in our world over the, over the decades is AI is now the quick buzzword. But there is no such thing as ai. There's a whole bunch of other things that are AI plus this AI plus that AI plus this that are segments of what, uh, what is ultimately just labeled for convenience.

The single thing ai. Um, and I think the, the big players that are making the bets now in this world are those small startups and the really big clouds that are going in. And that's what we're all concentrating on at the start of this conversation.

But in reality, there's these enterprise, uh, things. There's applications that are sold to various enterprises that, that need things. And then ultimately there's gonna be the, the small entrepreneurs that buy a, either a single AI computer or device and build a, build a model.

So there's gonna be at least three, if not four or more tiers of, of how this is gonna be deployed. And I think you are right in that the costs that those top people are putting on this are gonna be pushed down and paid by the people at the bottom. Well, but that's what they want, right?

Is they want the first mover advantage because they're gonna have to invest so much money in solving these hard problems that they want to get the first cut from the pie so that when this becomes something dirt cheap to use, they don't have to be able to point at their initial investment and create like, you know, lost uh, capital if you will, because we are the ones who had to pay three times as much to solve this particular problem. Yeah. But there are some hints I've been wondering about the capacity calculations.

There are smart people doing that, and I don't have the data they have, but I'm starting to see stories about, yeah, they built this giant data center and it's running 40% not used mm-hmm. Just sitting there. And we've seen that, right?

There's been a lot of things that have been released recently where they say that a lot of this capacity that has been purchased is just sitting around. Well, but part of the problem for that, and, and again, we have to be very clear in our definitions, there's the model training and inferencing stuff that goes on. Mm-hmm.

And then there is the application of doing things. So when you guys are sitting at home and you're asking chat GPT to give you a brownie recipe, that doesn't include Elmer's glue, that's what the result of all this inferencing is. So think of it like a space shuttle launch, right?

There's a lot of stuff that happens before the shuttle goes up, but once the shuttle's in the air that, like the gantry and all the launch equipment at Cape Canaveral just sits there, does that mean we need to get rid of it? No, because we might need to launch another shuttle sometime. But then the question is, well what if we built more of that?

Could we launch two at a time? Could we launch 10 at a time? That's the question that a lot of people are trying to answer today, is, if we keep scaling this infinitely, is it going to be enough for us to be able to bring these costs down or to do more with it to build a, an artificial general intelligence super hive mind?

Well, I also think there's an assumption that the big players are going to continue to be, uh, servicing all the smaller players. And I'm watching this small logic SLM rather than LM market. Mm-hmm.

Where people seem to be making gains, prodigious productivity and cost gains with much smaller models that are very topically specific terrain on domain specific data. And so I find myself wondering, is that going to shift to either the smaller models that don't need as much data at large data center presence or possibly even running in the enterprise space because it's more cost effective to do it that Way. But the other question is, is the market gonna continue to head in the same direction?

Because Mitch and I talk to a lot of companies through the Futurum group and they've pivoted away from LLMs or the solution to every problem, right? We're now onto AG agentic ai. Mm-hmm.

Which is functionally much different, even though, as to Steve's point, we're gonna slap AI on it 'cause that's how everybody knows it. But like, even some of the startups in the agentic space right now are looking at problems completely differently. They're training models that will be able to go out and kind of do tasks and report back or be able to do things like, uh, one that I saw this week was the fact that Claude has finally created a model that takes like four days to get you data back.

Now you're probably sitting there thinking to yourself, I'm like, why would I want this model to spend four days thinking about a problem before it returns the data? Because I wanted to spend four days thinking about the problem before it gives me an analysis. You know, it's like if, if I ask somebody a question and I get an immediate response, my first thought is, did you even think about that before you responded?

Especially if it's not something as simple as, you know, what is a class A address? If it's something that requires me to be like, you know, consider all of the following factors and then give me an answer. The longer this thing chews on it, the better off I feel.

And that's where a lot of the research is now. But is that where the research is gonna be in two years, in four years? Did ai Yeah.

And uh, and related to that is the in, in addition to those other models. I see. I think eventually people are gonna rediscover what machine learning was in the first place, which it's very efficient at and very good at solving specific mm-hmm.

Problems using the, the handful of machine learning algorithms that are well proven and now everybody's tossed aside as they chase the LLLM and Angen, uh, Craze. Yeah. To your point about, um, we definitely see smaller models or, uh, specialized models for whether it's coding or testing or mm-hmm.

Security mm-hmm. But also, um, size of models and efficiency. I mean, you can download, but watch lots of X four oh nano model to run on your computer, use a llama to download whatever model you want to use that's open.

Mm-hmm. See models being open sourced and the parameters being open sourced. So I, I mean, I think, I think there's the day not very far away, if not now, but we will run models on our own computers.

We don't think about it. It'll be just part of what's loaded with the operating system. Then apps will use that just like they use the Cloud.

And we've, we've even seen through some of the research that we've done at the Futureum group about the fact that the models of devices that are coming out into the future have specialized NPUs neural processing units that are designed to accelerate and offload AI calculations. Look at all the, the work that Qualcomm's put into the Snapdragon processors to do that very thing. Now they had to go back and say, well, you can actually run this stuff on a laptop that doesn't have an NPU, but it's gonna take more time and it's going to, it'd be more resource intensive.

But if that's the model that everybody's going to, like, at that point, if I already have the hardware in place, why shouldn't I be running AI on things? Even if it's smaller custom models and things like that. Makes it mobile too.

Yeah. That's the other advantage. Yeah.

Alright, so you're talking about following up on, on what, on what you were saying. So you're talking about, um, the data resides in one of these big massive fund data centers. Mm-hmm.

But the, the analysis, the actual, you know, the number, the crunching, data crunching is on your computer? Could be both. Mm-hmm.

I mean, 'cause we do replicate a lot of data mm-hmm. Uh, as well as generator on this. But you have a fair point is where does the data live?

And there's strategies like Oracle strategy is we have all the data already, both your data that you, you run in our cloud, but also the SaaS applications for ERP and CRM and all of that. And so we will apply the protections that we have, the RAC rules, all of the governance things and kind of bring your AI to get the data here as opposed to, uh, which is not gonna be a solution for everybody. Yeah.

You know, we just heard a conversation about like, bring your data on, on top of our network, right. So I think it's a mix of, I don't know what the answer will be. It's interesting.

I guess it's a question to go to then in our last two couple minutes, is the future of ai. Where is this all? Well, we are gonna have to wrap it here because we are out of time.

But I will say that Denise does bring up a very interesting point. You, yeah. You and the chair being on your hand like this, you are the one who's gonna decide what the future of AI looks like through your consumption of it, the way that you use it, the way that you ask companies to build things, to do stuff.

Maybe we might end up with the, the future of, uh, major Barrett Roddenberry telling me that the engines are running at full capacity, uh, like the Star Trek computer. Or maybe it's gonna gonna look like something completely different. I have no idea.

But we are the ones who will decide through the way that we invest in companies that are building the things that we want to use and the way that we criticize companies that are wasting resources that could be better put to use. And as people in the industry who are knowledgeable and understanding that institutional knowledge that you have is not something that can easily be replicated by an AI today. So use it, give feedback, tell people what's important to you and what you want to see research done.

I hope that it's not gonna be on ai video generations and things like that.