62. A Different Type of Datacenter is Needed for AI – Tech Field Day Podcast

We've been hearing a lot about building infrastructure for AI this week, and there seems to be a, a consistent theme around having to build something new, build something flash, build something exciting. What we're gonna discuss today is the requirement that you need a new kind of data center for ai. Welcome to the Tech Field Day podcast.

We, we bring together a panel of experts to discuss a single topic around a key issue in the IT infrastructure world. This podcast brings together voices from within the tech field day delegate community, and we often record these podcasts at one of our events. This particular podcast is recorded just as we're closing out AI infrastructure field day.

And what we're gonna discuss today is the requirement that you need a new kind of data center for ai. But before we get into that discussion, let's meet who's on the panel today. Hi, I am Karen Lopez.

I'm a data evangelist that specializes in security, privacy, and compliance challenges. I'm Leon Ra, I'm a platform engineer focused on infrastructure automation and, uh, working for a digital trust company. Hi, I am Denise Donahue.

I'm a network architect, And I'm Alistair Cook. I'm an event lead here at Tech Field Day. And of course, this podcast is also being distributed across the future and group through our tech strong, um, family within.

We've been hearing a lot about building infrastructure for AI this week, and there seems to be a, a consistent theme around having to build something new, build something flash, build something exciting, maybe build something completely different from every data center that you've had before. But surely AI is just some computers working with some data on some network, and we've done that before. So I think the foundation of that's correct.

Um, but I would say I'm all about designing to optimize certain workloads. And what we've learned this week, what we know from experience is that AI has some special requirements that are very similar to high performance workloads, but in another way they use the hardware differently. They might need different network software or configurations.

So that's what I learned this week. Mm-hmm. And the, the design for network design, um, typical data center designs can be fairly loose as far as, you know, you might put something here that you want to put, even if you're doing the, the strict leaf spine CLO architecture, then you might decide, well, okay, well I'm gonna adjust this a little bit here.

I'm gonna add this edge here. But the, the, um, backend infrastructure, the GPU to switch infrastructure for ai, it needs to be very specifically designed. It needs to be designed for very specific requirements.

It's a lot of that really high bandwidth, really dense bandwidth as well, because we we're seeing the GPU in the, in your server has four or eight, uh, a hundred gig, a hundred gig, uh, ethernet ports in it, and that's a heck of a lot of bandwidth in a small space. Yeah. And, uh, there is also, um, I think from the platform engineering perspective, while we are, um, creating a new infrastructure that is an high performance infrastructure, from one side, we are the networking that must be work really well in order to have this model.

Then for the other side, there, there are some new technologies or technologies that are empowering the, the model, the training on the model. Mm-hmm. And then there is also a word that it's, it is definitely my space, which is the, the AI ML pipeline, which is the new, uh, stage of the automation where you can train the model and then in financing or, um, you know, make some rag or in implementing or be, bring this model for the, um, to be useful for, uh, for the companies that are adopting these kind of new technologies.

Mm-hmm. And the, well, the other thing too, we have to address it, we have to mention it is power requirements space might not be such a big requirement, but the, you know, not that many racks, but it's kind of like when we went from individual pizza box servers to virtualized servers and you had much fewer racks, but the power and cooling requirements were much higher. This is that times a thousand a million.

Yeah. We're seeing the power budget in the rack that previously ran the entire rack is now being consumed just for the optical transceivers for the network without even thinking about running the servers, let alone the GPUs. Mm-hmm.

We hear frightening numbers about power density and racks from these reference architectures from Nvidia. Mm-hmm. Mm-hmm.

For sure. Um, and one of the things I like that we heard about from different companies is how they're focusing on even making those power sources from renewable energy or, and consuming wasted power and all of that stuff. Like I've always said, I like that idea of having servers in my basement to heat my home.

That didn't quite pan out, probably 'cause of networking struggles, but I, I really think there's gonna be so much more public focus on, because it's making the regular news about how much, you know, generating these funny cat videos more so in AI is just going to use a lot of power. I mean, to the point where companies are buying nuclear power plants to help back it up. Yeah, for sure.

This is another, another thing that, uh, we must consider in order to, uh, optimize the consumption of this new platform. Because yeah, we, uh, from the plasma, from the ops rephrase, from the platform, again, from the ops, from the platform, uh, perspective, uh, we talk about, um, finops and the, the finops, uh, topic is, uh, essential for, uh, to guarantee that all the business, all the monies that we are spending are, uh, uh, completely fit on the business and without spending any other monies and here, uh, find new, um, energy suppliers, uh, find new ways to optimize the, uh, power consumption is, uh, fundamentally is a critical part of this. Uh Mm-hmm.

Yeah. I think, uh, we, we saw, uh, in our last sessions today, we saw that, that that use of reclaimed reused power, power that was, uh, or at least energy sources, it wasn't even, we're gonna hook into the grid to a place necessarily where there's excess power, but we would find places where there is energy waste and consume that. I think that's a really important part of the equation around here as we look at the stupendous growth in demand.

Yeah. But there's Karen's, um, basement has quite good internet connectivity. It's certainly a lower latency to Karen's basement than it is to Iceland.

Yeah. Or I was thinking of, uh, New Zealand, we also have natural gas flares that are burning off excess gas, but network connectivity. It seems though that the primary network connectivity required for AI is actually inside the data center.

It's, it's even more of the east west bandwidth required rather than the north south bandwidth to get outta the data center. Does that open up new options for us? Yeah, exactly.

And that's something you have, well, when, when you design design any network, whether it's in any part place in the network, whether it's a data center or, or what have you, you have to look at the requirements. Just this is so different from the requirements we've had before. This is taking the, the requirements that from virtualized between virtualized servers and the three tier applications that we've had before and just, you know, expanding them.

So yeah, you've got to, you've got to look at the, the networking between the, the servers, between the GPUs and it has to just, you know, it has to be, so what was it? Not only fast but not lost less someone pointed out, but le less loss. 'cause there is no such thing as loss, less, um, load latency, just all the things that is, are the hardest to achieve, I would say.

I think the other side of that is too, you've got to be able to measure it. We talked, we talked this week to some companies that are doing really good jobs of not only predictive measuring predictive design did creating digital twins, um, but then measuring the actual real product once it happens. And I think that's something that as network people speaking, as a network person, I don't know about the y'all, but as a network person, that's something that we don't do that well.

We fall down on that and I think that's something that's gonna be a critical thing to address. Mm-hmm. Yeah, And one of the things I'm looking forward to that tends to happen with any of these major changes is, you know, I'm looking forward to all the lessons learned we've used to optimize for energy use, optimize for power consumption will trickle back to our other infrastructure needs.

All these tools and lessons learned where, you know, because the, the amount of, the extreme amount of extra power that's required is that we'll bring those back and also help solve some of our usage problems just with run of the day database stuff. And what we, we are seeing are, um, during this AI infrastructure field, they, uh, some companies are moving on, uh, the integration, deep integration with the, you know, uh, cloud, um, methodologies to consume and to better, um, reorganize the networking east west, but also north south. And, and so the thing is that, uh, um, um, when you, uh, when you try to, uh, put the fingers on networking as, uh, as you mentioned before, is the most important thing is the mentoring, the observability and, uh, to, uh, even better and to fast identify where the problem is.

And this is, uh, the other, the other thing and, um, it's so important for these companies, the integration, the deep integration with the solution, but also integration with the other, uh, um, observability tools in, um, tools and, uh, sorry. Mm-hmm. And the other observability tools that, uh, companies, um, already are, have implemented inside, uh, the, uh, data data center.

Yeah. So true. It that sort of holistic view of what's going in my, in my on in my data center, what am I changing, what's working correctly, what's working wrong, and then the ability to predict if I make these changes in the future, is something gonna go wrong?

I was hoping we would see more of a, that, that data pipeline, that AI pipeline actually looking more like A-C-I-C-D pipeline where I could maybe stand up a test copy of my network and then, well, we did see some test copy and an emulation and then use that as part of my pipeline to validate that I can actually deploy this out into production. But that holistic view, I think Leno is, is the thing that is still a long way away. You know, it takes a while for the new innovation to be integrated with the older things, um, seeing open source tools like, um, uh, Prometheus for the instrumentation and then Grafana for the visualizations.

That definitely, those are the, the quick ways to get in there. Mm-hmm. But how many large organizations are staking all of their operation and all of their, their observability on an o on the open source tools, how much are we actually seeing?

They're already staked into a legacy tool, and that in order to build this good AI data center, you're going to need to build a new kind of data center to support these more open source tools because mm-hmm. I don't see enterprise organizations that are that keen on using open source everywhere. Yeah.

Well, and it doesn't have to be open source. I mean, you have Nvidia that's, you know, the big dog in this world and, and they, they're no offense Nvidia, but they're very much not open source. Mm-hmm.

They're very, it's Fine. Yeah. Um, so it doesn't have to be, but I, I think that you're right, a lot of people are gonna want to take advantage of that with the, the sonic to tool type setups, et cetera.

And I think it'll be just like any other monitoring tool when things come along, is that if those vendors respond in the right way by providing this, the right nuances, the right measurements, the right metrics, the right connectors and sensor readings, then it'll, it'll still be a normal decision between proprietary and open source software. But we know that the product teams that don't respond to it, probably we'll be, become less and less relevant to an enterprise solution. Yeah.

And here another suggestion is to, um, for reference of the product, just stay in, uh, you know, CNCF in, uh, so many, um, place where platform engineer can pick up their solution because there is no definite solution for it. There are integration with the other solution. This is the reality today.

The other place I wanted to circle back, because Leno brought up finops, and that comes back to one of my sort of corner thoughts around this AI infrastructure that we're building, is the whole idea of finops is we are going to spend as much as is required to get the maximum business value and no more, I'm not sure we are anywhere near that as we are building out these new data centers. Because it seems that the number of actual business beneficial value to business delivery other than thought leadership, because we have a, a chat bot on our page, I'm not perceiving that there is that massive benefit that's yet been unlocked. Am I just missing it?

Is it invisible? You mean not the, the demand to have AI within, within your, within the business, within your business, the business. I, I, I think and I see, and I, um, that yes, there is that, that it's, it's very much a tool that is, um, looking for a solution right now, but I think solutions are, are coming 'cause you know, it's, or looking for a reason to, okay, take that back.

Cut that. What I see, what I think is that is very much a tool that's looking for a reason to exist in a lot of places, a lot of ways right now. But that was, that was because it took so much to, to gather, to create the models, to gather all the data, to train the models.

Now that we have the hyperscalers that have invested the, the time, the energy, the money into doing that, you can take one of those models and then customize it, train it for your own data, and then that's going to give you the, the financial reason to do it. That's gonna give you the business benefit in doing it, or the government benefit in being able to serve your constituents better, being able to answer their questions, being able to help them, you know, renew their driver's license or, you know, things like that better. Right.

And I think we focus a lot recently 'cause it's newsy and hype b of new things we can do with ai, but what I'm looking forward to and what I'm currently using it for now is how can I do the things I'm already doing much better, better content, you know, I still have to do validation and all that stuff, but I think a lot of the uses will be more like the volume of uses will be more doing what we currently do just in a different way. Versus there will be companies that'll do brand new things with ai, but your average, what I call regular companies, insurance banks, retailers and everything, they'll probably just use it to do, you know, to be, to build a greater margin, get more customers or save costs just like we do with any tech. Yeah.

Mm-hmm. And, um, yeah, there, there are some, you know, situations that, that I'm picture in my mind. Like I, you remember in the past when, uh, uh, many companies are moving out outside the front of the data center to the cloud, they spend, I Yeah, initially they are spending Yeah.

Quite a, yeah, a reasonable account, uh, amount of money. Then they discover year by year that this, uh, this charge was really, really huge. And then they, uh, and today we are talking about finops, uh, as a, as an answer of this, um, as an answer or as a solution, uh, to, uh, take your, your under control and without waste money for your business because you are focused on your business.

Now we are talking about another technologies that yeah, as a, an IT CO but also as an operation cost really, um, yeah. Uh, really huge. Uh, we have to find a solution there to better optimize.

But the fi the first thing is what's your business? Do you need 24 hours? Your training model that is running automatic pipeline or as we can see with the fin, and if not wrong, not all the models should run 24 hours, but there are demanding the run of the model.

So these are several situation that really depends on the business and depends on the money that you want to put on this solution. And I think one of the places we will see, not necessarily direct business value value, but I'm thinking about as assistive technologies. My, uh, my own father, my, my parents-in-law are in their eighties and their technical literacy is not improving, but their ability to express verbally what they want is still there.

And there's definitely a big opening for assistive technologies where the way you interact and, and the way you work, um, particularly, I mean my, my parents' generation, uh, are much happier to pick up something that looks like a phone and talk to what seems to be a human who is going to help them and be a heck of a lot more patient than their children are. Yeah. I think, I think it'll, it'll, it will expand.

The use will expand. Um, data centers will be either built out adapted pods within a data center to handle the, the private local AI or cloud. The thing that bothers me though, um, about something you said Karen, um, is that, is using it to do the things we do now and that's the, the privacy and security side Absolutely.

Side of that. Like, do I want some ai, some tool somewhere, a data set somewhere knowing every question I ask or every, um, like, you know, what about this rational, you know? Yeah.

So that's the individual thing. So I'm thinking more of what they used to have to do in Tableau or Click or Power bi write a query or something. Cool.

Now they'll be, they won't have to know all that technical stuff, so they're still asking the questions and the questions are getting recorded. But you're right, as I ask my AI assistant weird questions, the first thing goes through my mind is, oh my gosh, this is being recorded. But also searches happen that way.

I, I wanna say, don't remember this one. Don't report on me. Do not track as a, Yeah, do not track for Ai.

Do not learn. Okay. New product idea.

Excellent. Well, I think we have been talking about this more or less nonstop for the last four days, which suggests that we could probably talk about it for more time once we've all had some sleep. Mm-hmm.

So if people would like to carry on this conversation, where can they connect with each of y'all? com and I'm data check almost everywhere, especially Blue Sky. Alright.

net and also I run a Italian podcast for Italian folks that are following this channel, which is the Pipeline guys. Ah, oh yes. Oh, cool.

Um, I don't have any place to blog right now, but I'm gonna, so I'm gonna be hitting up the Tech Field Day folks for that. Um, but you can find me on LinkedIn and you can find me on Blue Sky as Lady Networker. Of course, I'm Alist Cook and you'll find me everywhere that Find Future and Content is created.

com/podcast. So wherever you like to consume your podcast, make sure to go ahead and subscribe, like maybe drop us a review and check out all of the awesome content from AI Infrastructure Field Day on all of our Tick Field Day events. You'll hear our voices from around the world, around the country, and around the industry.

And we will bring another awesome piece of content to you next week.