32. AI is the Enabler of Network Innovation – Tech Field Day Podcast

We are always looking for ways to enable innovation in the enterprise. And networking is no different. Today that driver of innovation appears to be AI workloads.

Is it something that is causing the companies to make their products more receptive to using AI for things like automation, data collection and data analysis? Or is it simply driving the infrastructure to be bigger and better and faster in order to move AI workloads and data modeling across as fast as humanly possible or inhumanly possible? In this episode of the tech field, a podcast, we look at whether or not AI is the new enabler of network innovation.

Welcome to the Tech field, a podcast where we bring together a group of IT technical experts to discuss a single idea about key concepts in the enterprise industry. This podcast focuses on a variety of perspectives from members of the tech field, a delegate community, and is often recorded in association with one of our events. In this case, networking Field Day Tech Field Day is a part of the future and group, and this podcast is also published on our sister sites like Textron tv.

Before we jump into the premise for today's episode, I'd like our guests to introduce themselves, starting with John. So my name is John Freeman. Uh, I've been, uh, uh, an equity analyst for the last 20 years covering technology.

Prior to that, I was an industry analyst, covered network equipment and software. Hi, I'm Scott Roon. I'm a consulting CTO in networking, co-founder of the Network Automation Forum and host of total network operations.

Thank you. Tom. Ron Westfall, research director here at the Futurum Group, and I cover the AI ecosystem and certainly that entails AI networking and AI infrastructure.

And, uh, certainly pleased to be here. Oh, thank you all for joining us. My name is of course Tom Hollingsworth, and I am the networking analyst and event lead here at Tech Field Day.

Let's jump into the premise for today's episode. 'cause we're talking about networking. We probably want to talk about the impact that AI is having on networking.

'cause you can't go anywhere today without hearing about how AI is gonna be a part of everything, but there's a lot of different facets to ai. It's not just improving your systems, it's about making your systems capable of running the workloads that AI is producing. It's about changing the way that we build our infrastructure to support this.

And as we've seen in the past a lot of other innovations in the space, AI is causing people to change the way they do things. Innovation, as it were. The premise for this episode is that AI is the new enabler of network innovation.

And already I can hear people typing comments, getting ready to let us know that we're wrong about this somehow. But I wanna start off by opening the floor to our esteemed guests who probably have a lot of experience building networks that are becoming more and more and AI enabled. What is it about AI that is forcing companies to innovate?

Well, I'll start out with the, you know, the tools you need in the tool bag today. You know, we've been thinking a lot about what does it mean to be that modern or next generation network engineer? And it's not about just subnetting and VLANs and how ethernet works.

You need to know some Python, you need to know some GitHub. You need to know a cloud platform. But you also to add to that stack need to know how to use chat tools just to do informal things with AI for, for data reduction, for accelerating and learning and all those other new areas.

It's like, it's another necessary item in the tool bag that you just start need to start playing with. Yeah, and what I think is really a game changer is that ai, in particular, generative AI, is having an impact society-wide. And that I think is altering, you know, the decision making about how can we best leverage the AI innovations throughout a workforce throughout an organization.

So it's less, I think, silo decision making. It's not so much, okay, I need AI in order to improve network automation, which of course is important, but it's like, okay, I'm using a generative AI prompt to better understand how I can make a workflow more efficient, how I can improve my business outcomes using AI capabilities and so forth. And so I think this is where AI is actually not only just driving innovation across, you know, the, uh, networking realm, but also really across the entire industry.

It's really the combination of what can be characterized as traditional AI that is, you know, joined at the hip with maur, uh, machine learning, uh, algorithms and techniques, but also the generative AI capabilities, the natural language interfaces that are really bringing about, uh, concepts such as augmented intelligence to the forefront. And I think this is where, you know, there's so much a difference now from just say two years ago. For me, looking at it from an innovation sort of perspective, I look for where the bottleneck is gonna be when you see these paradigm shifts.

And clearly we're in a paradigm shift. A lot of, uh, you know, a lot of CapEx is going toward generative AI specifically, and generative ai, you know, really feeds off of these massive language, large language models. And while people are making large language models smaller in some cases in order to fit on a device in order to run on the edge, the opposite trend is also definitely occurring large language models in accordance with Moore's Law as we get greater transistor density per uni cost, right?

Delivered by our folks, our friends at TSMC, right? As long as that it continues, we are gonna get larger and larger models. And so you follow where the bottleneck is right now.

I see the bottleneck right at the line of where, where networking meets memory access and that whole area, because that is where the bottleneck in terms of moving very large data sets through, you know, a transformer. And that's the, that's hard. That's really, that's, that's gonna get harder and harder.

And so that's why I think you're gonna see a lot more innovation with generative AI in networking specifically. And, You know, that tees up one really specific area that we can clearly say AI is driving networking innovation, and that's ethernet for the interconnection of GPUs, right? All the activity in the ultra ethernet forum, what certain vendors claim they can do today to displace InfiniBand, um, you know, adding the bells and whistles to, um, ethernet to make it very low latency and very low loss, to keep those GPUs busy for a model training cycle that can take weeks or months and you don't want that job to stall.

Um, that's, that's a really interesting area of, uh, you know, driving real innovation in ethernet in particular. And is it an interesting that this is what the 79th transformation of ethernet that we've seen in our careers here, um, don't bet against ethernet and infinity is not gonna go away anytime soon, but in the long run, I think Ethernet's gonna do just fine here. This is where the part of the podcast where I'm legally obligated to mention one of my favorite quotes ever courtesy of Steven Foskett that came from Bob Metcalf, the inventor of ethernet, uh, one of them.

And he said, I don't know what the future of networking is going to be, but we're going to call it ethernet. Right? It's, it's ubiquitous, right?

Yeah. Um, but I think that that's something that's very valuable to point out. Um, the last week of October, we received a notification from NVIDIA that they had just built what is considered to be the largest at the time, uh, AI supercomputer on the planet for X ai, uh, running of course Nvidia, uh, Grace Hopper GPUs, but more importantly, those GPUs are interconnected via ethernet and not InfiniBand.

And part of the reason for that is because Grace Hopper, uh, GPUs, uh, while being wonderfully expensive, um, they, they don't scale well. And InfiniBand passed about 50,000 nodes and currently the XAI supercomputer named Colossus has a hundred thousand nodes with plans to take it to 200,000 nodes very quickly. And I know from my own informal research into Infinite Band, um, 50,000 is less than a hundred thousand and it's definitely less than 200,000.

Um, but they're very quick point. I'm writing that down by you. Yeah, very good.

Uh, they are very quick to also point out that, uh, this is Spectrum X ethernet, this is not your garden variety. Plug it into a laptop ethernet. And I think that that is something that cannot be, um, underscored enough, is that we have transformed the way that ethernet fabrics work, because I don't know about you, but I remember when ethernet fabrics were first starting to be bandied about in the mid 20 teens, and we were hearing about how things like Juniper Q fabric or brocades fabric were gonna change the way that we did data centers.

You know, trill and and SPB were really gonna be, you know, uh, displacement technologies and then they weren't. Right. Um, a lot of people moved around and I mean, TRILL basically died because we invented vxlan.

But now we're dealing with not just, uh, scalability, multi-tenancy problems. We're dealing with literal physical, we ramifications. I mean, the, uh, the spectrum X ethernet that's running the backbone of XAI is an 800 gigabit per second switch.

And we are already looking at the possibility that, yeah, that might not be enough. We may have to go to a one terabit switch pretty soon. So do you feel like the, the AI workloads themselves are, are one of the, be the key innovation drivers because we just need to make things faster and more congestion, congestion resistant in order to make them perform the way that we need to compared to InfiniBand.

And I think two major challenges, to John's point about bottlenecks, the innovation on the tech side is percolating. I, I think, um, there is broad consensus on that and, uh, I guess, you know, there's always, uh, room to argue like, you know, the magnitude of innovation. But, uh, the two that I see is, first of all, the energy efficiency aspect.

I think we all understand that GPU clusters are heavy lifting energy, uh, requirements. And, uh, that is in turn coming, uh, out with, you know, hyperscalers, uh, taking, uh, nuclear power plants outta the mothball to help, you know, sustain, you know, the energy, uh, requirements and also, you know, uh, feeding what could be characterized as innovation in terms of enabling, you know, modular nuclear, uh, plants to come on the grid and, you know, enable the energy that's required here. And so, uh, stay tuned.

I think, um, in addition to memory, we're gonna see a lot of innovation on how we can make energy efficiency, uh, all the more impressive that is using things like liquid cooling techniques on a wider scale. I think it's only used in about 5% of the data centers out there today, but I can bet that number will go into double digits here in the next couple of years. So that I think is, you know, one aspect direct liquid cooling that will benefit from AI pulling this through.

And I think, um, in terms of, okay, uh, what else, uh, is, uh, needed? It's like, um, if look at some of the recent, uh, analysis out there, only about 10%, uh, the enterprises and organizations out there are, um, uh, past the proof of concept stage in terms of productizing AI into, you know, their overall organization at the recent Cisco Partners Summit, for example, the figure was about 15% according to, you know, uh, the Cisco partners out there. So we're really at the beginning stages here, but what that does is it creates opportunity for, you know, uh, the vendors out there that are addressing the enterprises to, you know, basically help with the data management aspects that are so critical to optimizing ai, uh, workload, uh, flows and so forth.

And so I, I think, uh, it's important to keep our eye on these two aspects as to, you know, how far can AI innovation go in terms of, you know, the entire networking industry. So can I, I, I would love to just peel that apart a little bit, Ron, because I, so I'm not gonna argue with him. I just wanna get precise.

Um, I think you've lumped two huge categories together that both have the letters A and I in them, but they're two very different things. You know, we started this conversation on network infrastructure for l for large language model training, right? And the innovation that's happening within the ethernet ecosystem, chip system and so forth.

And then you're, you, what you cited about enterprise. Um, but 10% of enterprises have productized AI into their, um, their workflows or their business processes. That's a, that's another aspect of ai, but they're two very different things.

And here's why I wanna, I wanna slice with a, a scalpel here. So as cool as the networking innovations are in whatever revision of ethernet we wanna call this, um, the who, who has the capital to buy hundreds of thousands of GPUs, a very small number of companies. And so there's a very small number of network engineers, they're gonna actually be able to touch this and play with it and do something with it.

The rest of us are gonna be reading about it and still agree that it's cool from a distance, but I think that it, that impacts the perception of AI innovation, right? There's some big changes that a very small number of network engineers are gonna have access to and play with. The rest of us are gonna be able to play with that 10% and growing number of, how do I use AI in my observability tools and how I do data reduction on how I have a natural language interface for my data center management system like we saw with, uh, Nokia EDAA few weeks ago, or that we've seen for years in, uh, in the Juniper Mist product in Marvis, for example.

Um, I think that's where most of us are going to see the impact of AI innovation. And I will say this, Scott, because this is a point that I've been trying to make to a lot of people for a, for a while now, that there really are two different sides to this coin. There are the people who are using AI to make their product better.

They are using, you know, LLM techniques to be able to do natural language queries and things like that. They're using, uh, you know, machine learning aspects to, uh, surface important data and, and clearing out the noise. Then there are the people who are making their products better so that AI is more usable.

These are the ones who are like ultra ethernet forum folks, or the people who are building bigger batter, faster switches that, you know, eat less hay and go slightly faster, like Henry Ford's horse. But, but I think that there's a room, there's room for both of them, right? And the value becomes kind of like you said, not everybody's gonna be able to avoid, uh, afford a, uh, was it Blackwell is the new one, uh, a Blackwell cluster that needs to be liquid cooled and requires a miniature nuclear reactor to run, but they shouldn't have to, right?

Nvidia can build a data center that can rent that time to people who want to take advantage of it. It doesn't mean that the innovation's not happening at the infrastructure level. It is, it's just not visible to the people.

It's like, um, your average laptop, the cooling system in a modern laptop is a miracle because it allows a laptop to run these extremely greedy, um, workloads and systems without creating enough heat to scorch your legs. And I don't necessarily benefit from knowing how that cooling system works. I just get the benefit of it working.

And I think that that's how AI is gonna appear to a lot of people is, you know, especially on that infrastructure side, I've built these massive ethernet fabrics, and the, the result is, is that it doesn't stall. And I agree, I think they are two different aspects. I invoke them as, you know, potential barriers.

So that is kind of the underlying, uh, theme here. And I think, uh, what's intriguing is, yeah, okay, the headlines are being generated by, you know, the large language models running on GPU clusters and, you know, hyperscaler data centers, and that, uh, can then, uh, catalyze, you know, using nuclear energy on a broader scale. But I think we're also seeing is when it comes to enterprise data centers, or even and ran implementation wherever you're putting a GPU and Nvidia, when most likely it's going to impact the energy considerations, uh, the power management envelope and so forth.

And so I think this is where we're seeing a barrier, but it's also an opportunity, again, for innovation. It's like, you know, necessity, again, being the mother of invention, uh, type of scenario. So I I, I'm thinking, you know, when, when, whenever you see recursive loops, they can be really powerful.

And that's what I'm seeing a lot with ai, for example, uh, kind of tangential, but you see the, the, the chip design software guys like cadence and synopsis remark that their, their software is being run on GPUs using generative AI techniques to design chip the next generation of chips that will simply be running the next generation of AI so that they can design better chips, so forth and so on. I think you see that in networking a little bit too, that the, the, the, the use of generative AI to manage to simply manage the, the orchestration of hyperscale, you know, level, uh, uh, networks, right? Is, is mind boggling.

So you kind of need AI and that kind of feeds itself. I just think that that's, that's always a si uh, uh, a sign that innovation is about to really take off in a place when you have these sort of recursive loops of, of, of, uh, that, that, that feedback on each other in a positive way. Yeah, I think you might be right, and that's, usually, that's how we see it, is you, we build the tool and then use the tools we built to build better tools, and that just continually accelerates the pace of change.

So I guess the next question that I have to ask is, do you see AI as the primary driver of innovation, or are there other drivers that you feel could have as big of an impact? com bubble, is that when there's so much hype around a technology, it does eventually fizzle out. And I too am a veteran of the software defined networking wars, and I remember what happened whenever everyone suddenly decided that OpenFlow was not the way and the light, and we needed to change the way we do things.

So are there other drivers that, that could potentially keep networking from fizzling out once the, uh, the AI land rush is done? I don't see anything bigger than AI right now, and there's a, there's a blessing and curse here. There's two sides of the coin.

Is there a lot of hype? You betcha, right? Um, and there's a lot of overblown marketing speak, apologies to all my dear marketing friends out there.

Um, but you know, the five or 10% that's actual usable, um, content and techniques, that's where we gotta figure out, okay, you know, let the sun come up over San Francisco and burn away all of fog and let's figure out what, what's really useful. And I think we're seeing tangible impacts of it right now. Again, the natural language interface thing is a big deal.

I've seen four or five vendors over the past couple months do this better than I've ever seen it before. So that's real. That's tangible.

People speaking a topology to a network modeling tool and have it instantiate and have OSPF neighbor relationships and BGP pairings come up just from, um, speaking a few well-crafted sentences, but just speaking 'em, that's, that's real. Then there's also the sense in which this is really just another set of computational techniques that help us do data reduction, right? Are you kidding?

In the log analysis space, what if I could dump all my network and all my security logs into one model, have it crunch and tell me what's really going on? I don't think that's crazy, and I don't think that's far off. I'll let others talk to other real tangible use cases.

Yeah, I think, uh, you're right Scott. I think all roads are leading to AI for better, for worse. And I think, uh, we're seeing, you know, practical examples of how it's helping, uh, innovation or at least, you know, improving, uh, business outcomes.

We're already seeing AI improving, for example, customer service efficiencies, improving, uh, field tech, uh, uh, capabilities and also re uh, improving coding. Uh, so already we're seeing organizations adopting it part of that 10 to 15% that are okay, we productize it and here are some of the, uh, good benefits. But I know that we're gonna see AI being integral to things like improving network automation, uh, throughout an organization, uh, seeing AI quite, uh, simply, uh, assuring that orchestration, uh, can be better implemented, uh, that, you know, all these capabilities are going to have an AI engine or it's gonna involve a gen ai natural language prop to, you know, improve the overall workforce experience, but also to, again, how to optimize the AI workloads and how to enable all these other networking features and capabilities to just be more efficient and link to better business outcomes.

If, if as a, as a leading indicator to tell like, what's going on. I think CapEx, particularly CapEx among the hyperscalers is probably a, a great leading indicator of what, uh, of what at least what they believe generative AI can, can, can, can do for that, you know, can do in terms of end revenue, uh, uh, uh, uh, generation. But in the meantime, they are definitely spending, I mean, I saw one estimate up to 400 billion, uh, over the next three years in, in CapEx specifically for generative ai, which clearly crowds out a lot of the spending that they, you know, that they, that they, they might have, uh, uh, laid out for traditional computing.

So there's no doubt about it that right now generative AI is, is what's driving the train. Let me ask you about that, John. So that's, that's follow the money, right?

Um, and we're seeing some of the, the huge players invest significantly in infrastructure. Um, I hear chatter that I don't completely understand about the slowness to get an ROI from that investment. Um, can you speak to that?

Yeah. This, this is what I've been looking at too. And a lot of people look to Microsoft originally as, okay, if copilot works, that that means generative AI works, right?

And we're all gonna be fine, right? And Microsoft copilot, the jury's still out, right? On whether or not I, and it, I think this is true of any generative AI promise that is trying to do something very broad and universal.

I think that's hard to do right now, but where it's taking off and where you absolutely see ROI are, there's a lot of niches, like, and all of these niches add up over time, right? And then of, and then it tips the balance, and then we get, you know, we get something more universal, right? But right now, these niches, for example, like chip design, um, like, uh, frankly like, like log uh, uh, uh, log data analysis, right?

You need to do that in real time. It's almost, it gets to the point where it's almost impossible for somebody to, you know, really be able to analyze all of the log data that they're already getting, right? So, you know, generative AI as a way to automate that, that's, that seems like a natural to me, among others.

I mean, I'm sure there's other, there's many other niches as well, right? Right on. And I think it fuels, uh, my, uh, premise that it's creating new opportunity for, you know, the networking, uh, players out there, certainly the suppliers and, and, and the integrators.

You know, we, we've seen the headlines and videos partnering with a whole host of folks out there to help enable answering just that the ROI question. And yes, uh, it's just like, you know, getting into the productization stage, it's still early, and I think this is gonna create opportunities, you know, for like, uh, potential as an example, a startup that can step in and say, okay, AI can be aligned to just do that spark, you know, network automation. Uh, we've seen that, you know, for example, with, uh, Nokia's, you know, proposition, uh, to use, uh, e events, um, automation to, uh, event driven automation to solve, you know, some of the data center inefficiencies that exist, uh, today.

But it's also, I think creating opportunity for players like Arista as well as, you know, the HPEs, the Dells and Lenovos to say, Hey, here is a way that you can practically implement AI to the advantage of your organization to improve the workforce experience, to really, you know, have a, a kind of a plug and play solution ready so that they can now get into, okay, this is how we're going to gain these ROI benefits. It's getting, uh, past that initial how do we adopt AI to our organization specific needs? And we touched on how this is gonna happen is about the right sizing of the language models using small language models, uh, fine tuning specifically based on that organization's data.

Again, that's where the data management piece comes in. So this is just, I think, a tremendous opportunity for yet more, uh, companies out there to, you know, play the hero for a lot of the, uh, enterprises out there. You talked about where ethernet is now, you know, uh, with Nvidia now the, the networking choice because of its scale out capabilities ver uh, vis-a-vis InfiniBand, I think that's a really telltale sign.

It kind of reminds me of the days when fiber channel was still trying to justify its low latency advantage over ethernet in, in storage. And, you know, it took a while for Fiber channel to get, get EED out, but I think it, it's, it's inevitable at this point, and I think that's a big deal. It's the power of a market, right?

There's so much competition and ethernet literate technology suppliers, whereas InfiniBand only comes from one place. Well, as you can tell, there is a lot of discussion around this particular topic, and as we have kind of teased out, the reason why AI is the innovator for the network is because AI is the workload that our networks need to adapt to today. Yeah, that doesn't mean that it's gonna be the end of the road and that there won't be something coming down the pike that could change the way that we do stuff.

But for today, we have to adapt to the workloads that we have. And, you know, in the old days that was voice. In modern times it's ai and who knows, in 10 years it could be interstellar networking, whatever the case may be, we build the networks for the, the data that we have today.

And right now that is ai. So I think AI is the biggest generator of innovation in the space. I want to thank each and every one of our guests for joining us today.

I also wanna mention that they will all be part of Networking Field Day, where we'll be discussing a lot of topics very similar to this happening November 6th and seventh. com and click on the link for Networking Field day 36. Thank you for listening to this episode of the podcast.

If you have enjoyed this discussion, please make sure you subscribe on our YouTube channel as well as your favorite podcast application. If you do that, make sure that you leave us a rating and a review so you don't miss any episodes, and so that other people out there can help us find, uh, our right audience. This podcast was brought to you by Tech Field Day, the home of IT experts from across the enterprise.

It is a part of the Futurum group. com/podcast or look for us on Techstrong tv. Thank you very much for listening for this week.

Make sure you check out Networking Field Day November 6th and seventh, and we will see you soon.