AI is Driving all Infrastructure Change – Delegate Roundtable at AI Infrastructure Field Day 4

We talked about this, uh, amongst ourselves just, just a minute ago, and we thought we'd delve into how the reality of AI is drive driving changes in infrastructure that is not simply putting a label of AI on top of whatever you've been doing for the last 5, 10, 20, 30 years, but it's actually driving you to do something new, something different. I think we saw that from a few of the vendors, didn't we? We saw particularly we, we saw Excite Labs, right?

Things that capabilities that we didn't think you'd get in places that we didn't think you'd get them with these massively scalable arm architectures inside what really looked like, like a piece of network interface. We saw that also Vast was talking about using the Bluefield for dpu. And, and our conversation after the, the presentation, we're talking about the differences between generations of Bluefield, dpu and the, the capabilities.

So really is things that are being driven by changes in AI and changing needs for ai. Uh, now I'm looking around for, uh, delegates who've got their eyes up. And Frederick, you've always got something on this one.

Yeah, I, I think I, maybe to start with the first presentation we saw about Cisco, right? Where traditionally there was ethernet and then for HPC, I mean, a while ago, we're showing technologies like InfiniBand and others. It's interesting to see that AI is kind of pushing InfiniBand more and more to the background and bringing the more traditional ethernet.

And I, I guess, I guess that makes AI kind of a, an innovation driver and pushing through for the innovation around ethernet. And then to follow up on that, we also see the extensions to that where, where not only the network, but you know, the slash NB HBAs and the Knicks are now being smarter than before and helping with, uh, AI innovation. It's, it's surprising to me that we're all starting to talk a lot more about inferencing rather than training.

I mean, early on it was all about how we, uh, trained and the infrastructure required for training. Now, the emphasis pretty much all week has been on inference. I mean, there's a little bit about training, but they're starting to talk about what's required to make inference happen.

So you're seeing, I guess it, its emergence of real applications and, and people or enterprises or organizations really starting to use AI to do stuff. So I thought that was interesting. Yeah, I think one thing we haven't talked enough about is, you know, the, the training inferencing, you know, it makes sense, right?

You got a train it before you can, um, consume the model, but inferencing in my mind is not the end-state, right? There's a whole application delivery that's gotta sit in front of inferencing. So I kind of, but, but to me, that's the part that's really not new at all in this, because we've always load balance to web servers, right?

We've had, I found load balances and load balancers for years, but I think we have to have a little bit different context, right? We used to context switch on, uh, layer seven packet information. Now we context switch on what the prompt is asking, right?

If I'm taking a trip to Naples, I probably wanna go to a different model than when I'm writing code. So I think in that aspect, we, we still have a ways to go to get there. Uh, but I think just as we made the leap from, uh, you know, training to inferencing and in inferencing is out now, that's what we're all talking about.

I still think there's the application delivery aspect on the front side. That's gonna be the next thing that'll come quickly once we, uh, you know, figure out interesting. 'cause I gotta gotta load balance to the interesting clusters.

Yeah. And sort of drawing a, a parallel in this, when we moved from monolithic architecture to, to client server to three tier applications, there was phases when we really cared about, let's get into the depths of the database design. This is the only thing that matters.

And then we started saying, well, actually we need data design on top of the, the architecture of that, um, database in order to deliver to the application. I think we are seeing that sort of maturity coming along. I, I really do agree that finally we're getting to the point where mainstream businesses can get value out of ai because that, that's where inference is.

But also as, as you say, Brad, that it's getting embedded inside the application. It's not a separate thing. It's just a capability that we're delivering in, in an application.

There's where we're, we're getting to that maturity, I think. Yeah, I think as a whole, industry's maturing. I think the shift to inference is basically implying that, okay, it's no longer a solution in search of a problem, right?

For a couple years, our businesses were like, we need to do something with ai. And we're like, why? What are you gonna do with it?

And they're like, I dunno, but everybody else is. And now it seems like there are more specific use cases emerging, emerging, you know, within the enterprise. Like, okay, you know, this is how we can improve customer service or make, you know, documents more searchable or whatever the case may be.

Like, here's what's slowing people down that we can improve with ai. And finally we're getting to the point where that's why we need to train, or that's why we need to shift to inferencing because we actually know what we're gonna use it for. I had a slightly different reaction.

We, what we heard, kind of looking back at all the talks, I think there was a bit of a focus on very large compute storage and so on. And that's because there's a lot of money in that. So vendors are naturally attracted to that.

On the other hand, I've been wondering whether AI is going to move into a mix of the large LMS for general contextual and speech type speech interpretation, but a bunch of more specialized, uh, soms or other tools and data sets which are smaller focused, but more numeric and ag agentic plays, it seems to me to be playing. And that's what we saw with forward networks, wasn't it? We, we saw them talking about we do the, the sort of bit that needs to understand what a human is thinking gets found out to an LLM.

But the thing that needs to understand our actual business application sits inside the business application. And just as we have people experts, we're going to have AI experts that know how to do marketing or whatever. And I'm gonna leave some space in the room for Marianne to leap in here as well.

So I actually liked it for networks approach. I felt like it was going more for behavior or configurations as being kind of the new source of, of truth within the infrastructure. Um, I think they made the strongest case with the truth layer, with the cure, how they query the DA databases, uh, how you got to the agents.

And I really think that's kind of a foundation of what you want in an AI assisted operations without the guessing and without, uh, some of the drift and some of the fragile controls. I wonder if that piece where we're, uh, sending things out to an LLM for human interface is gonna start to go away, because increasingly we're gonna be seeing agent to agent, and we've seen this in, in some of the, the sort of behavior studies of agent to agent. They work out their own language.

They don't need to talk like humans, but they'll negotiate their own language to understand. So is that need for the large language model out and, and the internet gonna go away and it'll just be agents talking amongst themselves and maybe we just set policy for these agents. Well, in the end, you still have to interact with the human, right?

So I would say that you upgrade from text to multimodal, that multimodal becomes the new interface where, because we're, we're typically more visual than textual, right? So, so I think that's, that's where, where the evolution will will happen. But even if agents talk to each other, I mean, if the only way to include us is through some kind of multimodal function And multi, you Still have to have, still would have to have some type of, uh, governance controls, uh, some of, some observability, uh, and even some cost guardrails to keep that human en engaged.

Uh, so I do think we are going to a new type of communication between the agents, uh, but you still have to have that human oversight Honor. I think both Anne and Jake would be very strongly of the opinion that you need those guardrails no matter what. Whether it's, it's an AI that's being interacted with by a human, uh, especially if it's by a human that has not established their own identity and the, the level of trust of being a, an employee or a, you know, contracted customer.

Um, even then if, uh, if you're doing quick service restaurant and an employee is minimum wage and has less loyalty to the brand than they do, uh, to the, the brand of shoes that they wear, uh, there might be some challenges around, uh, governance there. I think we're hearing a lot more recognition of, uh, the needs of agents and MCP servers and MCP support for infrastructure in general. So I was surprised by some of that.

I was gonna say, I think we've seen a lot of the human in the loop in the video surveillance market for a long time. There's been a lot of machine learning now called a surveillance and verification process of like, if a camera sees something or there be fraud or theft or threat, um, it's an alert process, right? It's, Hey, look here, and there's a human that shows up and, and looks at the screen to make sure it's valid before they do kind of the next step.

So I think there's some experience in keeping the human in the loop on response to ai. And that's, that's been a common theme I think we've heard from, whether it be we're doing rag on documents or we're, we're, we're extending into video artifacts, or we're gonna configure the network, or we're gonna let automation, let AI go configure the network For us, I think a lot of the industries are still gonna be very regulated because they know they need a human to look at it before we actually do something. Um, so it's, it's that when, when do we let AI start doing more things for us?

I don't know. I'm okay with it delivering my pizza, but on this other stuff, I'm not quite sure. Yeah.

And I, I think we will continue to see people who get it wrong. Yeah. And those people will continue to be seen in the newspapers.

I think that's one of the things from, uh, bradman's, uh, things to watch hallucinations causing problems that end up in, uh, you know, financial problems and in the newspapers it'll continue, which Is why I loved forward network's approach. Like, we're here to help digest, help, um, plan, coordinate, um, discover and share, uh, we'll figure out acting later once we've got the trust. Yeah.

We, we are here to advise a human who's going to act as their approach at the moment. But then I, I saw an awful lot of AI SRE site reliability engineering when I was at KubeCon, and that was, uh, you start by advising and then you have a, an an automation level, uh, similar to what we saw with, um, vSphere Dynamic resource schedule, right? You could say recommend only, uh, or you could say, just put it in the right place.

Or you could say, go move it whenever the hit you want. 'cause I trust you now. And organizations would go through that process, right?

They would establish trust and say, actually, in this general purpose area, this collection of things that are low risk. Yeah, I'm happy for you to do that. Mm-hmm.

Yeah. Oh yeah. You know, I think it, I think it's built in though, isn't it?

Yeah. Right. Those hallucinations give you a chaos Scenarios Just set the temperature, right.

Uh, EMC smarts had something similar to this a while ago, uh, where they would start, uh, with run books, and as they automated them, they would propose, here's a solution, shall I proceed? Shall I proceed? And after several times of saying yes in a row, it would say, would you like me to just do this?

Uh, that feels like another important step in trust building. I think. Um, what I found interesting is the diversity of people who came.

We had a lot of networking mm-hmm. Companies present, and we had storage companies present, and I think that was pretty much it, right? So, um, for infrastructure that makes a lot of sense.

But I think all of them had something, um, new that they were announcing that kind of seemed like some of the Cisco set is very new to them, which I found very interesting. Um, we just talked to Vast and they had something new and soy had something new. So I, I think that's interesting to me.

And I think the stories that were told are interesting too, because we've got this big diversity of audiences that tech has normally not marketed to. So we've got the rush of AI hype, which is everything in the world going on that we, you know, not grounded in reality. But then we've got the super grounded in reality, which is what has been marketed to the people that have always taken care of the big HBC clusters and machine learning and deep learning.

And there's no in the middle in that leads for the hype to either just overwhelm and people not care, um, or just kind of checking out because nobody knows what's really going on when this is a time that there's really truly ai and you have to architect for it. And these companies are innovating rapidly to give us the tools to build that architecture. So it's time to kind of close that gap in people to come together.

We, we heard a little bit of talk of ai, um, bubble and trying to differentiate bubble from wave is kind of tough in all of the Right, and I mean, whether it's, uh, AI reality or AI hype, one of the things that we also saw this week was how the AI world is driving the scale of these things much faster than any other real technology did for quite a while, where the, the scale of the networking concepts and the scale of the storage concepts are, are increasing far more than they had been with just, well, I'll call it traditional technology at this point. But, uh, but the, just the normal growth of technology that had had been happening for previous 20 years By, by the same token though, a lot of what we saw was variations on a theme. So a lot of this has been done before, and I use the term a couple of times, that everything old is new again.

We have done offload engines for many, many cycles and we've offloaded network unloaded, uh, back to the main CPU offloaded again, vice versa. Uh, we had a discussion about, uh, storage architectures and how what we were seeing was similar to stuff we've done 10 plus years ago. Now, it is being implement some of the, there there's implementation details that are important.

And, and, and I agree with you, it's a scope and scale that's unheard of in the past, but a lot of this is innovation in the way it's being applied, but not innovation in a completely new thought process. A lot of this stuff we've seen over and over and over again in, uh, you know, and somebody made the comment earlier of you, there's, you know, there's just a physical limit to how much you can, how many bits you can move across a wire, Right? And, and I mean, part of what we were talking about with, uh, forward networks and whatever is the next progression of the automation that we saw so much of 20 years ago where it went from being a manual task to you would automate this task.

It it wasn't just, it was just that you actually had people who were doing the automation process rather than having a, a process that was doing the automation process. Yeah. To pick up what Jack was talking about, though, I think, I mean, I, I would go even further and, and reject the premise.

We started this with that, that AI is driving this innovation. I think. I mean, so, so for example, right?

The DPU example is one good one. I remember sitting here for cloud field day seven in 2020 when Penson released their DPU and it was great tech. This is of course two years before anyone had seen chat c bt.

Um, and, and the question at the time was, okay, this is really cool, but what are we gonna use it for? Um, and now we have something to use it for. But that wasn't dr.

Like, they didn't build that for AI clusters or AI factories. They built that because it was cool tech and then we found a use case for it. And I think there's a lot of things like that.

I think that I, I think that when we talk about like AI bubble and AI hype, I think that in some ways we may actually be part of that disservice to, to the general population when we talk about AI infrastructure. And what we actually mean is infrastructure. And some of it's being used for ai, right?

But like the number of people who are actually building, like AI factories, like, like building their own models and, and ramping up is, is fairly small. Um, inference is gonna be huge. But again, that looks a lot more like a data center.

Yeah. Well network. So we're, we're talking about infrastructure here, right?

And so I think I'll just, I'll just say that I did a T CCP IP offload card with Exelon in Novell in 1990. Fair. So, so it was new to you.

Okay. But it's not new to it, Right? Right.

Exactly. Okay. Right.

But, uh, one of the things is the, this is the way human knowledge works. It is accretive over time. Absolutely.

That you, you, uh, whatever is paying the bills for the research. 'cause that primary research building that offload card was very expensive, very building any of the hardware that changes that we've seen is very expensive and very slow. Mm-hmm.

So you've gotta be able to see that there's going to be a return for whatever you you're spending the money on. This is a universal truth. It's not just an an it.

This is how capitalism works. E xite is a great example of that. Seeing their chip come to fruition, their chips X one, that E one, um, yeah.

And what they can do and how they've reimagined smart nicks and offloading Marianne. Well, I was just gonna add to that whatever is old is new again. Uh, I do think it's a challenge for some of the storage companies to really be able to articulate what you really built for ai, sort of that delta between what's for that AI workload and what you already had.

I think that was a, that's a really good point, because there's also, you know, when you get to the, the, uh, the sales process, there's refreshes coming up, of course they're gonna call everything AI because that's what's gonna, that's what they're, they're, that's what's driving the industry. So I, you know, I think there is really important to talk about the architecture to distinguish what's AI and what's not. And, um, I'll go back to the workloads.

I don't think we talk about the workloads enough. They, we got into that with several companies. I mean, I think, um, solid did a really great job, uh, of talking about that while they described their advances and, and their hardware that they create.

Um, but if we talk about the workloads to different stages of an AI pipeline, what's required for each one of those. Mm-hmm. Um, especially in this time when we heard this from the VAs guy like that, there's no, everybody now is looking for, um, for totally blanked on my head.

More flash. More flash. That's what I was flash.

I could see. All I could see was that spinning disc. No, I mean, that's what his was all about, was like, there's, we need more customers need more flash.

Where do you get it from? And people are having to get very creative about acquiring flash because we're in this time where we don't have any, and we're not gonna have any for a little while. So, um, architecture becomes very important To me.

The whole discussion about the lack of availability of flash and how that's, to me, that doesn't drive computer architecture. That's a transient problem that'll be solved in 12 to 18 months, and then it's gone. The architectural problems are how do you fill the GPUs and it's, it's truly an architectural problems.

How do you get data to the GPU? And there's many different ways to do that. Flash is part of the solution, but discussing the lack of supply that's gonna disappear as soon people turn on fabs is, is doesn't drive it's contract innovation.

It's two Years from now, dude. So what it does drive is a conversation about while we have this constraint and we always have a collection of const. Exactly.

And that's just a current constraint. We're aware it's time, probably time bound and that it will pass, but it is a current constraint, and that does drive how you architect your system. That's Fundamental.

Does. And it also gives considerations to, okay, is this a stock gap solution that I can then build on for the next, uh, solution that I need to build? So it's, it is really important, I think.

And then one of the early presentations we had was about lifetime of infrastructure. One of the things that's not happening is in place upgrade, right? When there's a new generation of GPU and GTCs coming up, there'll be a new generation of GPU.

The old stuff stays where it is because it's only a year old. We haven't depreciated it nearly enough. It's gonna be a brand new build.

It's a good point, right? It's the, it's, it's what we hear all the time about Brownfield versus Greenfield. Mm-hmm.

Right? I mean, it's easy to say, we're gonna buy the GPUs and depreciate three to five years. The problem is, is those new GPUs come a lot faster.

Right? And, and so what people tend to have to do is to keep those old gpu, and I say old, I mean like Old, right? And, and it's a problem on the networking side, you know, different speeds, latency workloads run a lot faster and, and different on different GPUs.

I think that also drives, uh, you know, the architecture. And, and, and in most cases, you know, efficiency also comes into play. Meaning that, you know, Jack was talking about, you know, filling the GPUs, running something on a GPU is not a challenge today.

It's relatively easy, but efficiently use them. That's where you end up with the, the higher technology. One of my lines in, in training was always this technology is really easy to do badly.

Right. Bu building on that though, I think, um, AI may be giving us an opportunity to do many more greenfield builds because every cluster is new. Uh, if you look at run pod, look at Lambda, they're still selling, uh, a one hundreds.

Uh, those GPU is five, 6-year-old technology still useful. So I think these clusters are hanging around for a while. They're usable for a while, and building the new ones is an opportunity for a new architecture, new build.

And that may be driving the pace of innovation as well. Well, and there's another real reality is that when people develop code for the A one hundreds, they had no clue the features of the H two hundreds and the B two hundreds. So there are cases where if you wrote code for an A 100 with not the future in mind, that code will actually run slower on H two hundreds than it does on an A 100.

Right. And so there's also an a tendency to keep those around for that particular reason. We also see starting to see some maturity around, um, data pipeline for inference versus what we saw of the data pipeline for training.

We saw a lot of focus early on on how do you get the right data in the right place? What types of workloads do you have in that training phase? But we're also starting to see that in terms of how do you prepare, how do you manipulate data, how do you get ready to be fed into the inference?

I mean, the ultimate is, is Josh's world where this is, uh, thousands of, uh, very large video streams that you need to be working in real time in order to provide that engaging coverage out to consumers who are watching the streaming. Yeah. And there's, in the specific video case, let's say you're a large streaming platform, um, doing it just with a GPU, there's even constraints and there's even new silicon being developed specific for video ai.

Um, some of that was announced at Super Compute most recently to kind of build, so, so the architecture infrastructure architecture needs some additional data processing pipelining hardware that can go in and look at 2000 streams of video that you can't just feed into a bunch of grace hoppers or the Into, into 10,000 GPUs. Right? Right, right.

It's so, so you, so the efficiency point to your point, right? You have to have an efficient engine to do these things, but if you're at the scale of Google, they can throw a lot of things at it. So they go, they'll go and develop a chip for it.

It almost sounds like you need some of, of the same type of pre-processing that, uh, both fabrics and Signal 65 we're talking about the Yeah. For mostly text-based things, but those are, uh, you know, it would be, it, it's, uh, going to be a different type of technology. Mm-hmm.

But at the same time, both, both of those companies talked about the need to do the pre-processing of the data before you actually push it into the learning system, right? Right. Mm-hmm.

Mm-hmm. So it becomes some of that business intelligence, data workflow kind of stuff of, you know, how do I get the valuable data? How do I get data that's consistent across all of the different sources so that I can feed it to the, uh, LLM while in the LLM or the small language model, the customized model mm-hmm.

Gets proper insight. Yeah. And speaking of small language models, I think at the other end of the spectrum, seeing innovation at the edge.

Mm. Cisco talked about this Yep. Looking at wifi seven, and it's opening up physical AI, robotics, um, there's a lot of innovation I think happening out at the edge for ai.

Yeah. Yeah. I, I think AI might be the killer app for the edge that we've been talking about for years.

Right? Because, you know, typically all these interconnection points that were around DC Silicon Valley, Chicago, Dallas, right? And, and that's where all the exchanges live.

Well, now do we have to build fabrics at spray across the country, right? I mean, is if there's, uh, I don't know, three fiber providers in a wiring closet in Butte, Montana, do they need a little fabric? Right?

What, where does the fabric have to go now in the age of inferencing that we didn't need it to go before will be Well, and and, and part of that is because we want to do the, the, i I agree with you a hundred percent edges where AI is gonna be, because, you know, we had a lot of discussions on the storage side about data grabbing, right? Yeah. Mm-hmm.

And it's not so much that data is heavy, it's that it's very expensive to move Yes. Time and resource intensive and slow. And so you want to push the processing to where the data lives rather than the other way around.

So as we have more and more AI driven applications, we're gonna see more and more small scale AI instances at the edge mm-hmm. And need to have small scale storage to feed that AI and the appropriate network connectivity there. And I'd love to see some of these solutions that aren't built for the giant clusters, but are built for, you know, uh, you know, we're talking about an exabyte of data in a six foot rack in one of the, the vendors, right?

And that's great, and that's a hundred million dollars worth of storage. But what about the guy who needs to do very fast processing but needs it on, you know, 10 terabytes or a hundred or, you know, or a petabyte that's a, a tray's worth? And how do they deal with that at the edge?

And what does that infrastructure look like? What makes that different than the data center infrastructure and how, Well, and I think that was an interesting point that, that Cisco brought up, right? I I what term that they, that they use AI clustering for everybody else?

Yeah. What, what do these small clusters look like? Because they have the same requirement.

They, they have losses, networks and all that. So Yeah, it's, it's gonna be really interesting that, you know, I, I, I think, you know, there's been a, like the training, everything again, has been centered around these massive things and large language models. And Pete, you and I have talked about, you know, small models and, you know, Jackie, you're spot on.

When, when, when the entirety of the AI stack becomes distributed, what does that look like? Because now it's really not, the entirety of the stack is not really distributed right now. Right?

And yet as we're thinking more and more about using ag agentic tools, the ability to be more distributed becomes higher because we're getting insights being generated, decisions being made close to where the data's generated, maybe in a retail store, identifying that the movement of a person around a store resembles the movement of a person around the store who previously stole things from us, not facial recognition, behavior identification. So we dunno who they are. We just know that this set of behaviors led to theft before, and maybe we should surface this up to somebody who is, is watching from a afar and You think, yeah, networking, you know, we used to say, where's that routing decision made?

That was the whole, you know, thought process behind network design. Now it's like, where's the business decision being made about? So thinking different.

Yeah. I just realized another, sorry. No, go ahead.

Another driver is human expertise, because I expect smaller models doing inferencing are gonna have to align with people who understand the data that's available, how it was obtained, flaws and all that. And I think with all of those, those are really great examples of workloads, right? People are gonna have to architect the underneath and be able to architect for past and future, which is coming at us every day.

So I think it architecture for this is really important. Hmm. So, I mean, and speaking of what's old is new again, and, and we, we were talking offline before the call, right?

About the pendulum that swings kind back and forth on a few different technology trends. And it sounds kind of like we're talking about, you know, as opposed to moving everything to the cloud, we're now talking about server closets again and actually having local processing. Right?

And in some ways, um, which is, which is interesting, I guess that is kind of swinging back perhaps potentially if you actually do need inference happen locally. Um, because that's, that's the problem with the edge, right? Is we thought we were gonna need these edge data centers for like self-driving cars and then we just put the computer in the car instead.

Yeah. And so I think, you know, that this may not look exactly like you think it's gonna look, it may just be more processors in your laptop. Done.

Yeah. I'm looking forward to the liquid cooled, full immersion data closet. Exactly right.

No, it's the liquid called full immersion data car, or no, it's behind the glove box. Full immersion. We are running to time on this.

Uh, and as usual, once we, this group of people get started talking, we could continue to talk on this topic for hours. Uh, if.