Is NVIDIA the New IBM? Enterprise Moats, Inference, and the AI Industrial Complex | Utilizing AI Episode 49

As we look at the AI industry, Nvidia is emerging as a potential new IBM, the cornerstone on which the entire industry is built. On this episode of "Utilizing AI," recorded live as part of AI FieldDay 9, we are considering whether Nvidia really is the new IBM or whether we're watching something else happen. Welcome to "Utilizing AI," the podcast focused on practical applications of artificial intelligence from Futurum Media.

Every Wednesday, we explore news and use cases of the ways in which AI is transforming enterprise IT and the industries it serves. I'm your host, Stephen Foskett, President of the Tech Field Day business unit here at Futurum Media. And before we dive into this discussion, let's meet who's on the panel today.

Morning, Scott Robohn from Solutional. Morning, Keith Townsend from The CTO Advisor. John Willis from John Willis.

Mitch Ashley with Futurum Research. , Open Technology Partners. And Frederick Van Heeren, Hyvens Consulting.

Excellent. Well, thank you all for joining us. I know this is a bigger panel than we usually have on "Utilizing AI," but we're actually recording this live at AI FieldDay.

This is the opening discussion before the event has started, and we will also be recording another episode of "Utilizing AI" with the other half of the delegate panel that will be published after this, as the close of this event. com to learn more about who these folks are. But with that being said, let's kick off this discussion.

Now, this came from a conversation, a suggestion by the folks here at AI FieldDay, and it does seem like we're living in a new world of, well, I guess what could be called the AI industrial complex. Essentially, we have massive companies that have really not just put a stake in the ground, but built a cathedral on being the AI giant. And the continent on which all of those companies rest is called Nvidia.

Essentially, we are living in a new big iron world where everybody is reliant on Nvidia. Then there's a few mega providers that are doing the hard work, that are building things, and then there's everything else. Or maybe we're not.

Maybe that's completely nonsense. Maybe we just fooled ourselves into thinking that. So I want to kick this discussion off with our delegates.

So Scott, I'm going to throw it to you first since this was your suggestion. Make an argument, are we living in a AI industrial complex monolith that's reminiscent of the old IBM days? Yeah, definitely, being co-located with the OCP Summit here in San Jose this week, it's a real interesting comparison of what in early in my career, seeing IBM being so dominant, not the only technology provider, but the sun around which everybody else revolved for a time, and then other innovation coming along and taking it over, right, and helping break it up.

No analogy, no comparison is perfect, so I'm not saying we're 100% exactly the same, but history may not repeat, but it certainly does rhyme. When we hear about the data center as the new unit of compute, and a vendor that likes to fill up those data centers with their computational technology, there's some similarity there. I am encouraged by what I see at OCP.

I know others have been there this week, too. I'm sure we'll get more commentary on this. To fan the flame of providing other options, some open, some other vendor ecosystems.

But I like the contrast, and I want to hear what others have to say about it. Yeah, I think it's interesting. I always refer to it as evolutionary and revolutionary.

If you look at Nvidia, for example, we always talk about GPUs, but GPU stands for graphic processing unit. Right. It doesn't say AI or compute.

To a certain degree, it was accidental. And I think the IBM comparison and Nvidia, I think it's where something else revolutionary comes up, accidental or not, will actually start replacing the existing driver, and I think this will be driven mostly by better compute and low power, because I think power is definitely one of the drivers that could push Nvidia to the side. Yeah, I've been doing this for five decades, right?

And the indispensability, I've been hearing the IBM, who's the new IBM, or when is the IBM going to fold, right? And the thing that IBM did was they owned, they made themselves indispensable to large business. Mm-hmm.

Today, we still run, I think last time I checked, 100 of the top banks still run their system of record data on IBM mainframes, right? And then when Linux came around, what'd they do? To your point about iron, they've owned the iron for years.

So yes, the conversation is based around what Nvidia's doing, all that, but this is a sustainable thing. And we've got now, like most of my five decades, there's been one winner in banks and infrastructure and in healthcare. And I've been waiting all my career for when is IBM going to get...

And they're not showing up really good right now in AI, but I've been hearing this IBM's going to fold story for five decades now, and it hasn't happened. I think it's way premature- Mm-hmm ... to call winners.

And things are structurally different that elevated IBM to where it's today, to your point, right? Starting out in business applications and mainframe compute, and sustained through the whole progression of that. Now you have every titan in the business fighting with all the other titans as well, some cooperation.

Whether it's over models, whether it's over customers, whether it's ecosystem and developers, whether it's hardware. John mentioned "Indispensability Trap," which is a book that Alan Shimlak just published, our good friend, talking about the people that build the infrastructure aren't always the ones that make the most money out of it. It's what's built on top of that.

And I think Nvidia recognizes that, that it won't always be about the Nvidia GPUs. It's going to be about the software, the apps, the models, everything else that runs on top of it. And they're racing to try to build a foothold into each one of those areas.

Yeah, so Mitch, you kind of keyed up my comment, which is, I think the counterweight to it being the new IBM, Nvidia being the new IBM, is that software layer, right? They kind of attack it with NIMs and hope to lock us into CUDA and the underlying infrastructure with NIMs, and the counterweight to that is the open AI API, like the way that it's consumed. So where IBM had us locked in, they had both the infrastructure and software layer, and they provided that for decades, to John's point.

And that tight integration has kind of locked us into IBM being the IBM that it is. What Nvidia has been unsuccessful at to this point is locking us into the software layer, and we're starting to see, even in my own research, I assume because I'm an infrastructure person, that all of AI evolved around of Nvidia. And that may be true at the lowest of levels, but when you get to Salesforce, their Agentforce, ServiceNow, Snowflake, and all of these huge providers, who cares what GPU or accelerator runs underneath?

I just need my data. Yeah, and I agree with your point, Mitch, about this being a little bit early in the game, and I agree with Frederic about the GPU being the accidental benefactor for this, or beneficiary. Anyway, I think there's an ecosystem play.

I think Nvidia's playing a long game. It's about a platform more than a we just want to sell hardware. Of course, they do want to sell hardware, but to Keith's point about being a software stack that comes along for the ride, you get something much more than only buying a hardware stack, right?

And being completely locked in. You're getting an ecosystem play with that. I don't know if I recall that without the IBM corollary, but I think the other thing I was going to think about on this is Nvidia is the first-mover advantage, but there are other folks that are building GPUs and building alternatives for compute, for inference.

So they're not the only game in town. They're the biggest game in town right now, but I don't think they're the only game in town. And back to Mitch's point, it's kind of early, I think.

Yeah. And I would point out too that one of the interesting things about this particular, I guess you could call it an industrial revolution or whatever's happening here is that we have some very large, dominant companies that explicitly and admittedly have no moat. And that's really unusual, incredibly unusual.

I mean, the entire history of the industrial world is about people figuring out ways to create lock-in and to make sure that whoever benefits from this, it's going to be me. And yet, OpenAI, Anthropic, and to some extent as well on the model side, companies like Meta, Microsoft, X, these companies explicitly admit that they have no moat and that people can switch. And in fact, their products make it easy to switch.

Yeah. When I stopped using ChatGPT and started using Claude, I actually asked it, "Pack up all my stuff, would you? Be a doll.

" And it did. And I was able to just move all that right over, and that's really unusual. But that being said, there is a moat.

Now, obviously, there's an explicit hardware moat because only some people have the incredible level of incredibly expensive compute that can actually run this stuff. But then, of course, there's also the moat called CUDA, which is, in my mind, the ultimate moat in the AI industry, and that is what has sustained Nvidia. It's crazy to think that this graphics library that was sort of hacked together 20 years ago has become a three, four, or $5 trillion asset.

But it is, and it continues to be. So yeah. Yeah, and I think the biggest threat to a company like Nvidia is innovation.

And so what you can see, what IBM and others did is they start out with their core business, which is the GPUs. But then they quickly realized that software drives hardware, so they started doing a lot of the software, CUDA and others. But I think that the biggest danger for them is that they kind of lose touch with the market and that they have not the ability to innovate.

If you look at the innovation today, a lot of the innovations are done by small companies- Yeah ... not by larger companies. And I think what IBM also tried to do, and Nvidia seemed to do the same thing, is trying to acquire those companies when they are at a certain level of maturity and are willing to pay a lot of money also for potential, right?

And I think that's the biggest risk to the Nvidias of the world. Yeah. Per comments on the software play, yeah, this is not an exact analog to the IBM scenario, and nobody's trying to hail the death of IBM here, right?

We know there are still mainframes everywhere. But the moves along the lines of acquiring Hugging Face, and not just acquiring software companies, but acquiring the portals that help me recommend and choose other models. Because whether it's OpenAI or Anthropic or some other frontier model provider or models that I run locally, who wins when I'm providing the GPUs that run the models?

And to your points even about lack of lock-in and no moat, I think that's important for viewers and listeners to keep in mind. Do not get locked into any one frontier model provider or one model, especially if you have business critical or revenue generating services tied to it, because outages happen, and government regulation can sometimes be unpredictable. So there are really good reasons to not get single-threaded on a single model provider or set of models.

And of course, there's practical reasons too. I need to point out that in many cases, you cannot lock yourself into a frontier model because they're going to tear that up in another six months, and you won't be able to use it. Goodbye.

So sorry that you were using it. Yeah. Yeah, I know.

As sort of a historian of this century, another point I want to make about Quantum, but quickly to that point you just made, which is the sustainability of IBM has been acquisitions, right? So the Hugging Face, that's a really good point, right? Can Nvidia sustain those type of acquisitions?

Because if you look at IBM, every time they were starting to fade, they made an incredible acquisition. And not that they just made an incredible acquisition, they transformed that acquisition. How many companies buy big companies and just fail at the integration?

IBM has been incredibly good at this. But I want to make one more point. I've had Quantum on the brain for about two years now.

I'm writing a book about it. And I went to the IEEE conference, Quantum conference last year and this year, and last year, Nvidia was all over there. " And so when we talk about IBM, there's an interesting interplay where will IBM win the Quantum?

Right now, they've done an incredible job of being at a surface level of Quantum as good as anybody else, not better. Yeah, I think that might be the battleground in the next five years. Well, there's one thing I've learned, whatever John's writing his next book on- ...

is what you want to pay attention to. I'm starting to think. I'm serious.

Whatever he's paying attention to. See, I endorse this position. I've learned that in several good ways.

One thing I want to add, though, is I think we're watching this distributed perpetual game of 52-card pickup, where it isn't just about who's doing the latest innovation. It's this continuous movement of innovation, but acquisition, but et cetera. The thing that I think is the moat, scale is always the moat.

That's the hardest thing to overcome by anybody, whether it's scale of your developer environment for GitHub. Now, even though Anthropic just announced Cyan, which is their own version of a source code management that supposedly addresses any limitations of Git in an AI agent world, yeah, that might turn out to be something, it might be something substantial, but you don't move everybody off of one service, one platform, one especially integrated platform very easily. It's how those things become part of what IBM does, which is sort of the adopt and extend approach.

And I think one of the things we haven't identified yet is a moat to what. Like Nvidia has a moat, absolutely, when it comes to training. But they bought Groq for a reason.

There are holes in its overall offense on owning enterprise inference. And what we haven't established is what enterprise inference is going to be shaped like. The H100, which I will hold this to Jen-Ali's feet as long as I can, he said last year at GTC that once Blackwell is up to speed, you won't be able to give away H100s.

The argument is you still can't buy H100s. Like the availability is that. And maybe some people are using it to run training.

Inference is just not shaped like training. And there's a reason why RTX 6000s are going for 4X what they were when it came out just last year. There's a reason why a lot of enterprises are running their AI on RTX 5090s.

You look at any of the older AMD GPUs. I think AMD, Google, Apple all have a legit argument on what their value is for enterprise inference. Yeah.

Speaking from the angle of a data scientist, we talk a lot about hardware, and Nvidia, to a certain degree, is a hardware company that is expanding in the software business. I think if you look at the math that are being used today in the models, it's ancient math. It's nothing really new.

Yeah. And I think one of the big dangers could be to the hardware companies that exist today, or at least the hardware providers, is that somebody comes up with a different approach to math, maybe it's quantum, maybe it's something else, that overnight could completely eliminate the need for GPUs as we know them today. Yeah.

You don't know what's going to come from the side. Can I use orthogonal here? Because that's a 50-cent word.

But there is the innovation happens, and we're not going to see it ahead of time. But even with the piece parts that we have on the table today, be careful. Let's not get caught up in only the latest release, only the latest hardware is usable.

There is always a market. I won't unpack that any further in this context. But the gray market, the used market, neo cloud providers that are in a position to squeeze usage out of all the previous gen hardware, that's not going away, and that will be useful capacity, especially as I can run more efficient models on them.

Then you've also got, what is AMD going to do? What is Intel going to do? What are the other ones coming from the side to capture inference space?

It's a great time to be in this. I'm optimistic and wide-eyed with everything going on here. I remain fundamentally really curious about what's going to happen in the next couple of years.

Yeah, moats are great, but to stretch the metaphor a little bit here, moats are only defensive, right? If all you've got is a leading position and you don't know what you're going to do next- Yeah ... build a moat.

But you know that somebody's going to be attacking, somebody's going to have a new idea, and you need to think about what comes next. And I completely agree. That's why I love having these things, because we get all these people together, and Keith, you succinctly summed it up there, that inferencing is a completely different deal.

We said yesterday in our conversation, training is basically R&D, and inferencing is the factory. That's where the product is made, and that's where theoretically the money is made. Yeah.

And it's a different animal, and we're going to see some different things coming out of there. Yeah, not being a math person who tries to understand math from a business perspective, when you think about Nvidia's success, it's because they accidentally backed into matrix multiplication, which is basically what a GPU card does, right? And to your point about the, is that going to be sustainable?

And then we know it works really well for training. For inference, the world is kind of wide open, so that could be the destruction of the moat. I would also say that for every reaction, there's an opposite and equal reaction, right?

Whether it's open weight models versus frontier models, whether it's local inferencing in the data center versus neo clouds, whether it's do I really need to buy a new Mac Mini with an M6, or should I just get by with a little bit less memory because I can run more locally, just to use a personal example. But innovation finds a way, sort of like the Jurassic Park thing, right? There you go.

Yeah. Life finds a way, innovation finds a way. And for everything that comes out as the next greatest level of advancement, there's a long tail of people who figure out how to leverage that in a different way or counteract what that leverage is to their own advantage, whether it's selling old cellular telephone systems in Russia to implement those- ...

or the US systems that got implemented in the '80s and '90s. Yeah, we're going through this evolution, this continuous roll, and we'll see that. It's not just about, to your point, what's the latest and greatest.

Yeah. Yeah. I do see the whole training inference conversation also changing over the last couple of years.

I always look at Nvidia. I've been working with Nvidia for a long time as a training company. Nowadays, they're doing a lot more inference.

But if you look at the terminology and the terms that are being thrown around, a lot of people talk about the AI factory, but there's also a thing called AI foundry. Hmm. And AI foundry is more on the training side.

And I think it's important to understand that a few years ago, the AI foundry was important because if you owned the data, you were kind of responsible to build your own models. Those days are done. It's all about OpenAI and Anthropic, and they start with those base models as opposed to building their own models.

But I think it's also important to see that the training, which is the hardware business of Nvidia, is shifting towards inference, which also gives them a lot of difficulties. To Keith's point, is that the hardware is different. It doesn't have to be different, but if you want to scale and be efficient, you have to use different types of technology.

Yeah. If it please the court- ... and I want to munge the factory model a little bit.

In manufacturing as we know it, not every single piece is built in a single factory. There's a supply chain, and those facilities are distributed. If I'm going to apply that to inference, I'm going to have multiple smaller locations for inference that will be closer to the users, that might be in a couple of racks in the back of a Walmart, or a Payless Shoes, or a smaller regional data center, or in base stations in a mobile network operator, or in the back of new Pulte homes.

Look it up. Nvidia is working with Pulte on a couple projects to examine stuff like this. And Steven, they will also be in space.

And I don't want to get single-threaded on any one of these logical locations. Had to go there, didn't you? Had to go there.

You're pushing my buttons. I know. I knew where that was.

But I want a robust platform where I can send inference requests based on how busy a model is, where is their affinity with KB cache, et cetera. And that geographic spread is a real opportunity for these other GPU, XPU providers to come in and to actually take geographic footprint and build out that ecosystem. Don't forget that we have a reality check coming up.

It's called annual budget. Yeah. The FY27, for whatever your fiscal year is.

And while '26 was sort of the wild and we've been partying, spending a lot of money on AI out of other people's budgets as well as the IT budget, CFOs won't let that go on for very long. I'm not saying it's all going to get curbed and suddenly we can become rationally observant about our budgets, but I think there's going to be a lot more controls. And you can see that in the data for what people plan to spend in the next six to 12 months.

Yes, they're buying AI tools for developers. Yes, they're buying agentic work platforms for their end users. They're also even more so increasing budget on observability for running AI systems and applications, not just models, but business applications in production.

They're investing in control, governance, things like that, to be able to manage this and be able to meet the compliance requirements. So the reality is setting in of, okay, great, the irrational exuberance about AI still will carry on to some degree, but enterprises can't do that forever. Yes, they need to innovate, but they've also got to operate.

And in those budget concerns, I think that's one of the places where the neo clouds really win at the end of the day because one, they are able to offer good enough hardware as opposed to latest and greatest, which will presumably cost a little less to operate and use. And the other thing is they're willing to take on smaller projects. A lot of times the big hyperscalers, they will just say no to certain projects because it's not big enough.

And then so when you're budget constrained and you've got good enough hardware and neo clouds are popping up over the place, that'll gladly take that business, take that work. I think those are also winners for the neo clouds. Yeah, and those budget constraints that will be coming up and will have more scrutiny are going to be factored along with use cases where it makes sense to use AI for certain problems.

Again, we're in this environment where there are very loud voices telling us that AI is the answer to everything. We know that that is not correct. I'm sorry, what was the question?

But there are people like Keith and others who are out there building out loud and trying things out loud to see what makes sense and what use cases are most effective. So I think you take the budgetary further examination plus what works, that's going to help rationalize, okay, where does the spend continue to go for what kind of problems we want to solve? So I want to return now as we're wrapping to our initial question, which is the AI industrial complex and Nvidia being the new IBM.

So my question, I will throw it back upon you. If anyone wants to jump in, please do. Is the IBM formula being followed by Nvidia, or do you see something different happening here?

So I have data on this. I've assessed now 48 companies, and a core assessment, what was started out as a core assessment to that was the relationship with Nvidia, and only the infrastructure hardware companies have Nvidia dependency. Every other player, so let's call it 40 out of the 80 companies, have direct dependencies to Nvidia.

The other 40 companies, Nvidia is obscure to their piece. I don't think you could say that 20, 30 years ago about IBM. If you had a solution that addressed enterprise needs, somehow IBM was getting a piece of the pie.

Yeah, I think what's the same thing with Nvidia is that they want to grow, they want to control the market. But I think the similarity with IBM is they're going to run out of companies to buy, because for a large company, that's typically how they innovate or buy innovation. Sure.

And I've seen that many times over. It doesn't have to be IBM and Nvidia. You see it at a smaller scale, too.

It's where very successful companies can't innovate fast enough, and they started buying other companies. And at some point, you run out of companies to buy, and that's where it's going downhill. I think the question is what's the next Bill Gates and DOS?

What's the next PC? What's the next open source model? There are changes, there are new dynamics coming.

Yes, Nvidia's a juggernaut. There's a lot of other juggernauts, too, and they're not going away anytime soon. But it's not going to be an all or nothing Nvidia win, to your point.

It structurally can't happen that way. I don't see it. Again, I've been hearing about the new IBM story for many decades, and I don't know where this data point fits in this story, but a friend of mine recently told me at a large bank, the minute you start up a backup mainframe, you immediately have to write a check for a million dollars to IBM.

That should tell us something about the substrate of business and infrastructure. Well, I guess time will tell, and we will find out what we are doing here, whether we are building ourselves another stack that is ultra reliant on a core company or not. And frankly, I'm just going to put one last thing in because I can, because I'm the host.

I feel like in many ways, Nvidia is very aware of this and is trying themselves to not just build a moat, but to also build an army and to also build a new castle over there. They are aggressively trying to defend their position, but also leverage that in order to make sure that they are relevant in the future. And I think that that is a really positive move for them, and it puts them in a good position.

And it potentially challenges where this all goes from now. So thank you all for joining us at AI Field Day, and thank you for joining us on this special recording of the "Utilizing AI" podcast live at AI Field Day. If you're listening to this and you enjoyed this discussion, thank you.

com, and of course, check out the "Utilizing AI" podcast every week. We have so many people here, I'm just going to say check out the podcast notes if you want to find out where to follow and continue the conversation with the folks around the table. We would very much like you to do that as well because, as you can hear, they have a lot to say.

So this podcast was brought to you by the analysts and experts from the Futurum Group. ai, the "Utilizing AI" YouTube channel, or the YouTube Techstrong TV app as well. Thank you for listening, and we will see you next week.