74. Software is Automating Your AI Data Centre Infrastructure – Tech Field Day Podcast

Software continues to eat the world. Is it eating your data center? Is it eating your infrastructure for AI in your data center?

Join us as we delve through the depths of your infrastructure and the depths of the history of the IT industry on this episode of the Tech Field Day podcast. Welcome to the Tech Field Day podcast, where we bring together a group of IT technical experts to discuss a single idea about some key concept in the industry. This podcast features a variety of perspectives from members of the Tech Field Day delegate community, and is often recorded in association with one of our events.

Tech Field Day is part of the RUM group, and this podcast is also published on our sister company site Techstrong tv. On this episode, as we head into AI infrastructure field day, we'll be discussing how software is automating your AI data center infrastructure. Before the discussion, let's meet who's on the panel today.

Hello, I'm Gina Rosenthal and I am from Austin. I run a product marketing agency. Hi, I'm Barton George.

I too am from Austin. Uh, I've been in the industry now for more decades than I'd like to admit, doing a whole bunch of different things and pretty excited to see what these, uh, these vendors have to, to to us next week. And I, I'm Andy Banta and I'm not in Austin.

And I'm equally interested to hear what the vendors have to say to us next week and, uh, wanna learn quite a bit more about their products. And I'm Alistair Cook. I'm the event lead here at Tech Field Day, the event lead for AI Infrastructure Field Day.

I also am not in Austin. Uh, but we would did wanna talk a little bit about this theme that we're seeing in AI infrastructure Field Day this week, where we have a lot of software companies who want to automate the platform that is your AI data center, uh, to automate the infrastructure, to automate the applications, to make it easier to deploy out new AI applications and, um, make sure all of the, the pieces underneath fit together. And it was a bit of a surprise because we thought AI was all about the GPUs and yet we don't have Nvidia telling us about their, their, uh, GPUs in this.

Is it really that software is eating the continuing to eat the world? Is it eating the AI infrastructure world? Or is there still a requirement for some hardware somewhere?

Well, I, uh, I introduced this topic simply because when I looked at the lineup for AI infrastructure field day three, I noticed that there were far more software vendors than there were hardware vendors. And very traditionally tech field day events have been lots of hardware vendors, especially ones that dealt with infrastructure, were lots of hardware vendors talking about the products they had. And sometimes they would talk about the services that ran on their products, but almost always they were coming in to sell you some iron.

I think that's at the bottom of the stack and, and where the value is is gonna come in the software space. And I think that's something we've been seeing for a long time in a lot of different places. And it'd be very interesting to see how these companies talk about knitting it together.

'cause I think there'll be a couple companies talking about specific product sets or specific offerings, but the, uh, the majority are, are talking about how everything fits together as a whole and how that coordinates with within your data center, whether that be hybrid, multi-cloud or physically, uh, in on-prem. Well, I think it's funny that you bring up Nvidia the first place, right? Because Nvidia can't run without a server.

So it's not like it's this powerful thing, it's doing all the calculations and it, it makes the software that runs on AI to do all these calculations and do all the real work, um, go. Um, but you know, maybe we're in a space, maybe we're not seeing as many hardware vendors because we're in a space that the software has gotten ahead of the hardware again. So what kind of, are we stuck with hardware that's legacy?

And I know we're seeing a couple of legacy presenters, so it'll be interesting to hear what they say about that. You know, have they innovated anything with the hardware stack that, that it, the GPUs or whatever accelerated use has to plug into? Or does it take software to do, uh, some tricks and, and things could all the bells and whistles that they really want the hardware have running those, um, algorithms and those workloads, I think we're like in a cat and mouth situation.

And I, I agree with that. Uh, and I, we certainly need the software to make sure that the AI infrastructure works properly and that will always be the case. But when you go back to, uh, pre AI days, there was always software that made the various different components work as well.

And what I'm seeing for an awful lot of the AI infrastructure is people are trying to do it entirely with software on top of the legacy equipment like Gina was saying. And hopefully that will be enough to actually provide the infrastructure that AI needs. But it seems to me that there's an awful lot of, uh, there's an awful lot of hardware improvements that could be coming along as well to feed the ai, uh, hunger.

And we don't tend to hear a whole lot about it. Yeah, it'll be interesting, right? Because if you think about that's is the fact that we're hearing from these companies trying to get at the data and it's really the data silos because if you look at an organization and you're trying to get to the different data to extract meaning from it, they could be on different storage systems, they could be on tape, they could be, the data could be all sorts of different places.

So now you have to have a layer on top of the hardware that aggregates the data to make it faster or to make it appear like it's close enough to do the different calculations that need to be done so it doesn't take forever in a day. So it's kind of interesting, I was like, what, what are we waiting for on, are we waiting for Quantum on our side, on the hardware side to get there, to make all these things run? Or what will it, what's it's gonna take?

And I think that's a, that's the whole idea of what do you have legacy? You, you don't start greenfield with all of these. Uh, and I do think if you do make purpose built hardware for this, that's gonna be a big advantage.

But it has to be introduced slowly. It's not something very few companies have the, uh, the ability to just start the luxury being able to start from, from ground zero. Um, but it, it will be interesting to see what types of systems are going to be purpose built and how they will differ.

Or is there really, if you have a common layer of compute, just sort of a, a bed, uh, many, uh, distributed systems, will that be enough? And you put the value on top. And then one other thing I wanted to say was, Judy, you mentioned, uh, Nvidia and one of the big reasons why NVIDIA's successful is the programming language around it for developers of Cuda.

And I think that's something that's been a huge advantage of theirs and something that, that being proprietary is something that they can use as a point of differentiation, uh, compared to other, other people say a MD who's now trying to, is taking the other side of it, which is, Hey, we have open source tools around this and they're sticking out that, that end of the market. And I, I think that sort of cadence of where innovation is happening, where a change is happening is, is one of the things that we do see through tech field is that the people who come and present a tech field, they want to show us something that's new, something that they've made a change to. Uh, and we tend to bias towards where the innovation is.

I know last year we were seeing a huge amount of looking at network infrastructure and the design of the physical network infrastructure underneath. And Andy and I have, have been at a number of tech field day events that are AI oriented that seem to be all about networking companies. And yet this year we're seeing less of that.

We're seeing more of it being further up the stack and that it is software for orchestrating getting workloads out. I think one of the things we're seeing is that transition from the idea that everybody is going to be building foundational models and needs the infrastructure that you, you build this massive infrastructure you build for a foundational model or for doing some really advanced fine tuning using massive data sets. Uh, we're seeing this sort of transition to a more, uh, realistic view that we've gotta get financial value for what we're investing and we've gotta find ways of using the expensive and limited resource in an efficient way.

And I think that's why we're seeing quite a lot of that shift in what's coming at AI infrastructure field. What I think we're gonna see is a collection of vendors who are gonna tell us how to make better use of the assets we're buying rather than just saying, you, you gotta go, go pave the whole road over again with new infrastructure, which has got all of the latest GPUs and, uh, spend half a million dollars just to get to the proof of concept, um, spend $20 million to get to production. Well, companies are, particularly with the current financial, uh, climate and, um, but companies are, are questioning whether they can even do that.

And I think that's why we're seeing this move towards Productionization. Right? And uh, I mean one of the things that we often see in many different parts of the technology world is that you, you build the software infrastructure to handle a specific problem.

You need, in this case AI data centers and whatnot. And once there's a well established model on how to do that, many times there will be a hardware vendor that will chase that and come in and come and come up with specialized silicon to take care of the problem that you're currently doing with software in an effort to, uh, increase performance and give you better results all along. I think one thing, one place where you are seeing a lot of the specialization, which we talked a little bit about before we kicked this off, uh, is back at the silicon, right?

So you've got, obviously you've got, um, GPUs from Nvidia, which actually were, were being done, used for something else, found out that they could be very, uh, very, there were very well suited to ai. And then that's, that has taken off a MD has got their versions, Intel didn't really take, get the memo, and as a result they're, they're not doing so well. But you've also got smaller players like tens, tens, torrent as well as grok where they've got these lpu and there's NPUs and other things that I don't understand.

But I can say, uh, and I think that's where people are saying, Hey, you don't, you, you got the CPU and that's appropriate for something this, you wanna use the GPU Oh. But if it's on your laptop, you want an NPU and to offload, and I forget, it's for which side is it? You want to use it, uh, LPU and you wanna use rocker, it's a 10 Torrance.

So I think at this point in so many of this, it's every anybody's game as far as uh, which way it will go and who will take, who will take root. And uh, it's gonna be pretty exciting to watch it, to watch it unfold. It must be pretty nerve wracking though if you're running a business, right?

Because, because as these things go, what if the one you choose gets acquired? Or what if the one you choose isn't the, the one everybody else is chooses? So it goes down the wayside and, and you kind of go, you have to go that way or have to make a decision to, to bail.

So we're still in that kind of stage too, where since nobody, it's anybody's game, it's anybody's game. And we're gonna start seeing, I think lots of these companies being acquired, these ones that are doing software things, um, we'll see how it goes. Uh, I I think some of them might be acquired, uh, by potentially the, the exact same hardware vendors that I was talking about who are trying to chase that field to bring it up.

And uh, going back to a previous AI infrastructure field day, uh, in Fabrica was one of the great examples of somebody who saw a problem with networking and decided to actually go generate silicon to do that. And it's entirely possible that some of the vendors we're talking to this week will be doing a similar approach. Yeah, it's interesting that the networking was the first thing that we started seeing, you know, everybody com complained about, because if you can't move the data fast enough to be become, to be worked on, then that's a problem.

So now all of a sudden we're looking at the infrastructure, it's like, okay, we can move it really fast, but uh, are the storage arrays working with us? Is it gonna, the location of the things, is it working with us? So now we're talking again about, you know, a little alert, you know, in that same low level of physical infrastructure trying to figure out what's going on.

So yeah, it'll be, it'll be interesting. I think one of the, the key things, which is true in all new technology is lowering barriers to adoption. How do you reduce that friction?

And there's oftentimes where you've got a far superior technical solution and yet it is so hard to set up or doesn't play nicely with what you already have. And to have to then take and choose something that's not as performant and yet it's, you are able to actually install it and get it up and going. I think that's a, that's a big thing.

I mean, it's what Docker did with containers is previously you had virtual, uh, containers in the, in the form of things like BSD, jail, Solaris zones, and they were, they existed, but they didn't get beyond the high priests and priestesses. 'cause you had to be pretty darn smart to use those. And then Docker comes along and while it doesn't have all the functionality of what these, uh, what the predecessors did, it was something that people, uh, were relatively easy to get up and going with, and as a result, it, it went everywhere.

So it'll be interesting to see which ones of those will these easy to use solutions that may be not as, uh, performing as the others take root and take over. That's a really interesting point. But never forget the politics of all of this, right?

The reason we have ethernet and not token ringing is because the guys that were developing ethernet hated IBM and they wanted to just destroy them. So they did whatever they had to do to destroy them. And that's why we had ethernet.

That's probably not the only reason, but, you know, it's one of the reasons. Um, so it's also gonna depend on who's got the money in the runway and just the guts to, to be a little crazy and push something down on all of us, whether we it or not. And then I think the other thing is, I think there's a lot of stuff going on, um, with the military, and I think it'll be really, really interesting to see because in times past that technology has trickled back down to all of us.

So I think it'll be interesting to see if and when that's gonna trickle back into kind of like the commercial space. Yeah. If you, um, are as as old as as my baldness, uh, you'll remember the DARPA challenge, which was the first, uh, autonomous vehicles driving across the desert, uh, that led to what we have now of, of more autonomous vehicles not necessarily driving across the desert on purpose.

Uh, so yeah, I think there's, there's an element. The other thing that struck me that you're describing, uh, Barton was what I would consider to be the iPhone effect. Um, I was using smartphones before iPhones existed and they were pretty clunky to use and I certainly wouldn't have suggested that any other member of my family would use a smartphone.

Um, then along comes the iPhone and the usability means that both my daughters and and my wife have iPhones and, uh, very happy with them. And you know, that you cannot, cannot underestimate the ease of deployment, time to value, ease of use as you're building out a product. And uh, we, we definitely see that as one of the vital differentiating factors for, for EE product, any technology time to value.

Although I'll say just in my previous point, I never had an iPhone because I hate at and t hate them. They did me dirty once a long decades ago and I'll never use them again. So I didn't have, uh, a cell until Android really got going 'cause I was not gonna ever sign up for anything with At&t ever again.

Yeah, there's one thing, well I have at t but I do have Android, so Okay, There's one vendor to cross off the Tfd list. Yeah, I think one Thing, maybe it's just that, that we can't invite Jayna to that. Well, I was gonna say, you talked about politics, Gina, and I think the other thing is grassroots adoption.

Uh, particularly when you talk about open source, you talk about developers, you have things like the lamps stack, uh, Linux, Apache, my sql, and either PHP, Prolo, Python, whatever you wanna use the p for, but that wasn't something that somebody came in and said, Hey, we're gonna be using this in our data center. It was something that the, that the developers liked. It was easy to use.

Uh, and Linux in general is another one that was, I worked at a company that had a, had an operating system called Soliris and it was really big and beefy and powerful. And yet the adoption was so much easier with Linux it was free, you could take it and slowly but surely it went from what people thought was a hobbyist type of a, of an os to something that is just everywhere today. And, and Soliris isn't.

Um, so I think that's, uh, another one would be the, the rest API, how it took hold compared to soap and all the other things, which, uh, Gina, as you mentioned, you've got these big standard bodies and you had like Microsoft and IBM and all these people pushing it and the developers said, I don't know, this is, this is, they had like a hundred page manuals and they just said, this is too much and they just started using rest API. So it, that will be another thing that comes up is what will the developers look to use? Well, yes.

And, and I mean, of course developers always look for what is the easiest way to get this job done. And many times this, uh, the, the developers will develop their own tools or enhance tools that are out there. And this goes back to one of the comments I made a little bit earlier where lots of times the developers come up with, uh, algorithms or uh, protocols or interfaces that later on, uh, hardware vendors will chase because they've become so easy to use that they, uh, that they now need some way of accelerating them.

And that also kind of ties into what Alistair was saying with the, the idea of the iPhones to where when you make it simple, you you'll have more people using it. And this is why we end up with some of the specialized TCP fun offload functions that we have on some networking equipment today. This is why we end up with specialized chips that will run Python or run Java, uh, rather than actually having to have a general purpose CPU that sits there and uses a, an interpreter or compiler.

And I think these are some of the innovations that have brought about the making what we call AI today possible. Absolutely. And I think that's a really good point that you have, um, vendors that are gonna chase what's really the operations end of it.

And to go back to Barton to your, your example with, um, Linux and Soliris. I was working at an agency that was on Soliris, but the astrophysicist said we got to make this run faster. So it's really the beginnings of ai, right?

We have to make this run faster and it's not gonna happen on a sun machine. We need to have X 86 and we need Linux. So we all learned Linux and we learned how to put our, um, our operations.

I was a baby system minute this time too. So like we had learned how to put all of our operations so that everything was secure, it could be backed up all of the different hygiene that you need to do to protect everything could happen on Linux as well as it did on sun. And um, that was a, a big exercise we did as an agency.

But I think Andy, that's kind of what, um, some of the people that chase these ideas like, okay, it's obvious that these developers are are forcing this and, and they see a good, a good amount of, um, value to running it their way. So how do, how do we make something that makes it easier for the operations team to protect it and to back it up and to do, to secure it, do all the things that needs to happen so that the developers can do their thing faster. So I think it kind of all goes together, right?

That's exactly true. And I mean one of the, the big reasons why Solaris lost out is, uh, Intel could way outspend Sun Microsystems on doing chip development and could outrun the, um, the, the chip generations and, and outrun the speed that Sun Microsystems never could. And uh, I mean Gina and I have both worked at VMware and we're very familiar with this as well, where when, when I started VMware, Microsoft was the one dictating Intel architectures to Intel.

And by several years into the time I was spent, I had spent at VMware, VMware was dictating Intel to architectures to Intel. And this is very much a case of the hardware vendors recognizing who was adding value to their product and going to them for requirements rather than going to, uh, you know, their own think tanks or whoever they'd been previously going to for requirements. It's just an it, uh, something that reminds me of when I was at Sun, uh, seeing presentations about Spark will always be two times as performant as Intel.

And as you see that it, it, they went like this and Intel went up higher. 'cause as you're pointing out, that's all Intel did and they had a big installed base and Sun Chips were really powerful, but they just didn't have the, the financial ability to compete with somebody like that. Um, I I wanted to bring us back to the, the thought around the software vendors and the, the vendors that we have at AI infrastructure, uh, field Day this week that they really are a about bringing some of what Gene was talking about.

So bringing more platform and control and production ready operation to the, the AI infrastructure. And I think that's might be why we are seeing so many software vendors in here is that drive towards making this a, a consumable platform to enable the developers for, uh, as Andy says, for the developers to be easily able to build whatever feature they need and to, to deliver that as a standard service rather than some unique thing that was out in, in the corner, uh, and maybe was a bit of a white elephant. 'cause we knew we had to spend a lot of money on it.

We just didn't know what gonna More, um, more accessible, uh, easier to use and easier to manage. And at uh, the same time I'm hoping that some of them are also working on the problem of feeding the beast because AI can consume way more data than an awful lot of, uh, various different software and hardware pieces can deliver. So I'm very interested to hear as much about feeding the Beast as I'm managing ai.

I think one thing that I'm really looking forward to is what are they hearing from their customers? 'cause we, you know, I, I listened to this survey was done by MIT saying 95% of all, uh, enterprise AI projects fail, but that was a small sample size. I want to hear what these vendors are talking, uh, hearing from their customers when they go out, what's working, what's not working, what are their pain points, uh, and and what are the, the, the value that they're seeing because they hopefully they're out there talking every day to people that they're trying to sell to and and getting a better idea of of what they need.

I'm hoping to hear, um, all of that, but also I hope we hear from somebody that's doing something so innovative, right? So we all know the problems people have in running these algorithms and trying to wrap their hands around it. Organizations are doing the all, all sorts of things.

So yeah, this is great, it's making it easier, it's making this and that, but where's the innovation? Like are we gonna see someone that's got just a really innovative idea that changes the game for everybody? And that's, I hope we do.

That would be kind of cool. Well as always, uh, we could have a great time talking about this for a lot longer and talking about the history of Sun and Intel, uh, and maybe even the future of uh, well I guess it's Oracle and and Intel, but we do need to bring this to a close. And uh, I guess, uh, before we do close out, I just wanna make sure that people can connect with you both, uh, during this uh, infrastructure field day event as well as ongoing.

Um, where can people continue this conversation? I'd say I'm mostly on LinkedIn. Um, although I have, I'm Barton Georgia on LinkedIn or the URL.

Uh, yeah, the URL is Barton 8 0 8. I have reluctantly gone back to Twitter 'cause there seems still be a lot of conversation there. And I'm Barton 8 0 8 there.

I'm Jamie almost everything. You can find me under Gina Rosenthal on LinkedIn as well, but I'm on MAs on a bit, but definitely on Blue Sky And I'm Andy Banta and while I still have a Twitter account, I don't pen tend to pay much attention to it. You can find me on Blue Sky at Andy Banta B Sky Social and you can find me on LinkedIn.

And if you wanna read content about the things that I hear and find interesting, you can find that at andy banta substack com. And of course you can find me Alister Cook on your favorite social media platforms as Demi TAs NZ for New Zealand where I live. Uh, you'll find me correctly on Tech Field Day properties as well as across the RUM group.

So thank you very much for joining us for this episode of the Tech Field Day podcast. If you enjoyed the discussion, please subscribe on YouTube or in your favorite podcast application. Drop us a like, uh, give us a nice review.

Uh, this podcast was brought to you by Tech Field Day, the home of IT experts from across the enterprise and a part of the RUM group. com/podcast or view us on Tech tv. Thanks for listening and we'll see you next week.

Bye.