93. Billion-Dollar AI Headlines Obscure Real Business Value – Tech Field Day Podcast

Funding for AI has been in all of the news on the Tech Field Day rundown, billions of dollars, none of which are coming to me or any of my panelists on this edition of the Tech Field Day podcast. Instead, we are gonna dive deeply into what you can really do with ai. Join me for the Tech Field Day podcast right now.

Welcome to the Tech Field Day podcast, where every week we gather together a panel of experts drawn from the Tech Field Day delegate family, and we discuss a single topic around some key issue in the IT industry. Uh, this particular episode is recorded on site, possibly even on premises at the AI infrastructure field Day four event here in Santa Clara. You'll find this and many of the Tech Field Day podcasts, also across on the Tech strong media sites and on the Techstrong app, possibly on your smart tv.

In this episode, we're gonna take a look at the thought that all of the big headlines that we are seeing around AI and the funding of AI are really a distraction from the reality that AI is being built and needs hard work to build. Before we dive into the discussion, let's meet who's on the panel today. Well, thanks, glad to be here.

My name is Frederick Van Herrin. I'm the CTO of ens, is a company providing consulting and services for AI markets. Uh, I'm Chris Grundman.

Uh, I'm the executive advisor at Kaja Consulting, and we look at transforming teams and technology to make networks more efficient. And I'm Brian Martin Signal 65 VP AI data center performance. And of course, I'm Alistair Cook.

I'm the event lead here at Tech Field Day for AI Infrastructure Field day four. And I'm also one of the hosts of the Tech Field Day rundown. And, uh, it's probably because of me that it's been full of news of Core Weave gets this much funding and Nvidia invests so many millions or billions, and there's contracts for five years of power supply for building bigger data centers that have ever existed in the world before.

But we're increasingly learning that that's not necessarily connected. Maybe that noise is getting in the way of real business value from ai, particularly from generative ai, rather than the predictive AI that we've been using for years and years. And that's something I've been looking for.

I've been saying as, as we've been through these AI infrastructure field events, I've been really looking for where's the business value we, when we can get to the point where AI does something for us. And that's really inference, isn't it, Fred? Yeah, definitely.

I think, uh, a lot of the news you're talking about is, is really the, the people that developing the large language models, and they do that to provide functionality to the people doing inference. The problem is, is with the shortage of infrastructure, it's very difficult for people now to apply this, the inference, right? There's a lot of innovation around agenda and so on, but still, you still have to implement any applications and buy infrastructure for this.

Yeah. And the splashy, like you said, the splashy headlines, I think right, are, are definitely not helping. I see like conversations around AI seem to be very, very polarizing.

I think this is part of it, right? Where you've got people who are, you know, really jumping on the bandwagon of, of AI is gonna change the world and changes everything and everything's different now. Um, and then other people who kind of ppo a lot of this is being, you know, smoke and mirrors.

Um, and, and I think these, this new cycle around the investments that are happening, a lot of times circular investments, right? These companies are investing in each other. So like what cash is actually moving, um, is part of that distraction and that noise where we can actually talk about, you know, what is actually different about AI workloads.

Uh, to Gina's point, um, in the recent, uh, AI infrastructure field, day four, she brought that up a couple times, and I think that's really interesting to dive into the nuts and bolts of like, what is actually changing here. Mm-hmm. Versus, you know, who's spending what money.

Mm-hmm. Yep. And, and looking at inferencing as we head out to the edge, head out to inferencing, I think there's a big trade off happening around what's good enough.

I've been doing a lot of work recently with AI assisted coding, and that has come leaps and bounds from just six months ago, 12 months ago. Uh, I am a favorite. I love the best models out there, but they're expensive.

I don't always need them. So how do I find out and how do users find out what's the right model, what's the right inferencing for the job? And I think there's a lot of tuning to be done there.

We're still very early on in the enlightenment phase, and I think as always, the future is here, but it's not evenly distributed. There are people who are getting a lot of value out of mature use of ai, and I do think we are also guilty of just saying AI as if it's a single thing. Mm-hmm.

And that really is another element of, of getting in the way of understanding what's really going on when somebody talks about this product is ai, AI ready and AI enabled, uh, doesn't mean that it's not doing anything different than it used to do, and AI just consumes it the way in whatever the last application is. Um, you know, one of our standard raises is to say, if, if I could take the buzzword out and substitute the previous buzzword, and all of the presentations still made sense, it's not about the buzzword. Right.

And, and I, and I think you're, you're, you're right on with that, right? So ai, when you, when you look back at the 1950s and the 1960s, the definition of AI was, was more significant around mimicking what humans are doing in some kind of a automated fashion. Um, in AI nowadays, I feel like certainly since the early two thousands, it's more a focus on a data-driven aspect where a anybody who uses data to do something would consider it ai.

I think, uh, what Brian was saying earlier about the, um, the large language model being built, the, the, the, the challenge is, is that those models are so generic that there's a lot of work that needs to happen to make those useful on an inference side. And I think from a a from a news perspective, you know, we, we always hear, uh, companies saying, we need more GPUs and we need more power and more data center space. I think what the market really wants or needs are smaller, uh, language models.

And so today we have a tendency to start with a large language model and then kind of slice it to a point where it's usable, and then we, we add some fine tuning and some rack to make it very useful for us. I think where it could be very useful, uh, from an inference perspective is more focused on smaller, large language models, which would then mean we don't need that many GPUs. You would fit more language models on the same GPUs and so on.

So I think there's a whole whole, uh, evolutional cycle there, And we're seeing that, right? I mean, I've already started to see some benchmark tests that have come out that are showing, you know, purpose built small models outperforming the large models. Now obviously they are more purpose built, right?

Like you said, you, you're training on a smaller set of data to get more specific context around a more specific problem. But I think that is something that's very interesting and I, and I think that goes back to kind of, you know, yes, AI is definitely happening, things are happening. To your point, I think language is very important around this.

Um, and again, where most people, when they say AI today, it feels like they, they mean LLM, um, specifically with chat interface, natural language processing, these kind of things. Um, which, which to your point isn't just ai, right? AI is something that's been going on since at least the forties and fifties, uh, and it's been using a lot of, a lot of technology.

So, so getting specific about this, and again, back to what I said earlier, right? What's actually different here I think is, is super helpful to look at, you know, more than just slapping that AI label on it. What's, what's, what's actually different about this use case and, and, and how are we treating that right?

To your point where with smaller models, this may end up looking a lot different than we think it will, um, in, in the long run And may, and mixing those components, like I, I've, I've picked it, I've picked up a, a way of talking about AI recently where I'll call AI level zero. Like you don't need it. Rules-based works fine.

Uh, AI level one machine learning, we've had that for years. Classification, uh, very important component of a system. AI level two is degenerative part.

That's when we get to the LLM, and that adds the creative aspect onto the other two layers. Uh, and I think assembling systems with all three of those together gives us some very powerful solutions. And we see that in some of the agents that are being built.

There's a part of the agent that's very rules-based. There's a part of the agent that's depending on classifiers to understand what it's seeing, and then there's a part of the agent that's asking creative questions and looking for answers. Uh, and I think having an understanding of that lets us bring the different models Frederick was talking about together, um, and letting them work together.

Yeah. For, for a solution. I think that's great.

And, and I think then, you know, the next step below that, right? Again, we're sitting here at, at AI infrastructure mm-hmm. Field day.

I'm kind of thinking through that lens still. And, and again, looking at what's needed to support that, right? And, and, and I think a lot of this just looks like web traffic, um, at the edge, right?

And so, like, if, like, like, is there, you know, again, I I guess I'm repeating myself here with like, what's actually different, right? What do we actually need to address to make AI work? And, and if we can solve that or at least talk about that, then we can get a lot further, a lot faster, I think.

Yeah. And I think, you know, that that's really what's, what needs to happen are standards, right? I think agents is, is one step in the right direction.

'cause it gives you the ability to kind of include multiple large language models, but then you need some kind of an open standard to move data around because that's, that's one of the biggest problems. That's where MCP comes into play. So I think if you, if you look at inference and the ability to build things, it's smaller models that are more on target.

You know, don't drag all the noise with you every time, uh, you deploy something. Um, and, and that, that all combines with, with agents and agentic ai, uh, will make, as far as I'm concerned, not only inference a lot easier to deploy, but also for more people to be able to deploy certain, certain things, right? We always say, you know, the, my grandmother or my mother is kind of the reference kind of a thing.

It's the same thing. And they understand the concept of agents. They don't necessarily understand what AI means, Right?

We saw some of that earlier this week. Looking at, uh, context, um, context engineering is so important to filter out the noise and bring in what you really want the LLM to work on, right? One of the other aspects I wanna cycle back to, um, Chris's comment that it just looks like web traffic.

Mm. Um, and link back to a presentation at the previous cloud field day where access to a chat bot was used as an attack vector. Mm.

And the chat bot, because it was a full large language model that had all of the capabilities of a large language model, was gleefully helping an attacker find credentials and find resources to further be attacked. And recognizing that, that the potential scope and the speed at which we're deploying these things without necessarily having the security safeguards controls that we should, is leading us to some fairly significant, um, security issues that we wouldn't have necessarily had with older web technologies. You know, ProSite scripting attacks are relatively easy to mitigate.

Um, but they weren't initially they were, when it was a new thing, it was a new thing. And LLMs as an attack vector are all far more, uh, powerful tool for attacking your environment. I was thinking about this Alistair during the, uh, round table earlier today.

You know, we've seen AI driving, you know, extreme innovation, the G-P-U-T-P-U market, we've seen it driving innovation in networking, intelligent networking. We've seen it driving innovation in storage. What I'm waiting to see is that next drive of innovation in security, what I like to consider the, the fourth pillar of it, Right?

We we're waiting for the, the PayPal of this generation, right? I think, I think because right? 'cause remember back, like, it was very scary to input your credit card number on online, right?

I mean, online purchases were, were really freaky at first. Mm-hmm. Um, and we've, we've collectively gotten over that in the last whatever it is, 20, 30 years now, 35 years maybe, something like that.

Um, and part of that was like, like PayPal was one of, of the payment processors that kind of came in and made that. So like, oh wait, like I'm working with this known entity. I'm putting my my credit card information here and I, I can trust it.

Um, I don't know if that's the kind of thing we need for, for data for AI to understand that. Like, I'm actually, you know, and again, I don't know what this looks like, I'm not gonna invent it on the spot, but, but containerizing this information in a, in a package where I can actually use AI without being afraid that it's going to rob me, What does trustability look like? Yeah.

How do I build a, a chain of, um, evidence equivalent in an AI conversation, right? And it's, it's, it's difficult because you don't know what they put in a large language model as far as data and its capabilities, right? Mm-hmm.

It's like asking, you know, what kind of meat is in the hotdog, right? Typically they will say, you know, you don't want to know. Right.

It Came from animals. That's right. And so with large language models and security, it's almost like you need kind of a post post element where, where all the security is dealt with, no matter which large language model you are plugging into it And gives some certainty own governance that the information you are providing to the large language model is gonna be used in the way and for the purposes that you've permitted it to be used for.

Yeah. And, and the other question here though is, is like again, right. How does this actually end up looking?

I I think we, you know, one, one of the pieces of news that's distracting us right? Is these big investments and things as we started the conversation with, I think another one is this churn of kind of new AI tools and techniques and technologies. Some of them are actually really useful, right?

Like MCP seems to be a step forward potentially, right? And, and just ag agentic in general. Um, but, but I do think we're, we're, we tend to, those of us who are paying attention tend to be watching this kind of roiling, you know, wave crashing here and, and need to zoom out a little bit and look at the fact that, you know, the current models are probably not even the models we're going to end up using, right?

We're working on determinism. Like if you go to like some of like research coming out of MIT and other places, they're looking at like actually deterministic models. They're looking at, you know, um, uh, visibility into the model's decisions and things like that.

So I think, you know, it's, it's really interesting. We are kind of shooting for a moving target where, you know, the security we're talking about, right? Securing the current model methodology may be completely useless because we're gonna be using a completely different methodology with different models in the future.

Yeah. And, and we need more regulation to, um, just as an example, I mean, I, I feel like a, as a data scientist, I feel that some of the large language models are kind of escaped out of the lab. Um, meaning with that, that there is much more pressure from competition between the large language model providers to deliver something.

And you could see it, like one example, um, was when, uh, the first large language models had these coding capabilities, uh, so people were using that to generate, um, all kind of tools for, uh, public clouds. And so their security keys were in there. Mm-hmm.

Which is not necessarily a prom, but it was then uploaded because you had to, it was uploaded as a prompt, but then was part of the engine. And then other people who had nothing to do with the original user had access to that key because suddenly it became part of the source Code. Yeah.

And then some prompt engineering to go harvest. Yeah. Uh, All of those pieces of credentials comes out and, and then we learn to put some guardrails around that and attackers can move on.

I, I wanted to circle back to one of the things that, that Brian said and, and some consequences. He said that the, uh, AI coding tools had grown so much mature over the last six months. And it leaves us with a, a challenge of having old fashioned thinking, being old gray people, even though we're only thinking about what was new six months ago.

And that, that the hangovers of the ways we thought and approach things as recently as six months ago is no longer valid. Uh, hangover is a great word for that. Alistair, uh, I'm advising a startup and the CEO has a marketing background, has been doing vibe coding, uh, for demos and prototypes, uh, and I'm helping, um, consult on some of the backend coding on that.

And what we've done is, it's this interesting toggle back and forth as he'll discover something new that suddenly works 'cause he doesn't know it doesn't work. Um, and then I'll still be in my old way of thinking because I know too much, and then I'll try something and it'll be a breakthrough. So we have this ping pong effect of sort of laddering up the layer of capability, uh, and keeping each other at the cutting edge as a result of that.

Uh, it's, it's been very helpful, uh, for staying on top of the change. 'cause the rate of change is almost impossible for mere mortals to keep up with. So do we need an AI to inform us of how AI has changed?

Right? I mean, there's different use cases, right? Yeah.

There're different use cases. I, for a programmer, you know, I did a lot of programming in the, in the past, I use kind of those tools as a, as a, as a assistant if you want. Mm-hmm.

I don't necessarily let it create all the code I need, it's just if I need an area where I need to look up something or need some guidance. But I agree. I mean, they, the last six months, seven months, it's, it's incredible to a point of scary, right?

Because you just push out an ID with zero coding understanding and it will generate something that most likely will work. Mm-hmm. And then will generate the test cases to validate that it, right.

It will do the things that it thinks you asked it to do. Yeah. It reminds me of this, uh, there's a short story by Isaac Asimov, um, and he, there's this, you know, far future civilization, there have been in this war with another civilization, and basically it's this like space drone battle, right?

Where there's these like, you know, unmanned ships that are basically fighting each other and it's, they're in a log jam because, you know, our AI and your AI is fighting and it's just, we're just destroying bots and it just keeps going. And, and this person comes into like the chairman's office and he's, he's got this great discovery and he works out that like, we can actually like, get in these machines and fly them ourselves. And he, he's basically like rediscovered math.

Um, and so he actually, he's like, oh, look, we can, like, we can actually plot the trajectory ourselves of these planes and Right. And then, and then all of a sudden they win in the war because they've now taken over. And, and I think this is something that's, it is an interesting layer of abstraction, right?

Where if you can like, remove semantics from configuring devices, from writing code, from all these things, then in one way, this is really, really liberating that anyone can now create the app that they want. Um, which could be great. But I also see this future of us drifting away from like anyone knowing how to create the app.

And so, like, I, I don't know what that means exactly, but like, there's a balance there that needs to be struck, I think. How does, right. How does that translate?

Where, where does our creative spark go? Do we become experts at, you know, management? Does everyone become a manager now?

Um, because we have I of workers to help us do something? Do we become the creative force? Another interesting aspect of that that I've seen start to evolve is the tradeoff between learning how to use a tool to do a job.

Mm-hmm. Versus just writing a tool to do a job. I've seen the co I was talking about earlier, you know, stopped learning how to use tools and just wrote his own tools.

Like, I needed to do this specific thing. I can write that it's done, I use it, and when I'm done, I let it go. It's A really, really interesting point, right?

And especially if you look at, you know, other areas of like business software, right? Things like, uh, CRMs or, or ERPs and these kind of things where literally organizations have force fit their entire organization, their process, their structure in to work with SAP or Salesforce or who, you know, name your name your dragon there. The tool Defines the business process.

Exactly. Right? Exactly.

Business process change your business model. And so what if you could actually build a CRM that works the way that you work? And now maybe we can, so that, that's a very interesting idea, right?

Of like, just like everyone using custom software that actually works the way that their business works, which potentially leads to a huge bloom in like this business model innovation and business process innovation that Is a fascinating and terrifying leap forward. I, I was thinking of bespoke tools that you throw away when you're done fair. You're talking about running your entire business.

I like it. This is how fast things are moving, right? This, you're probably not off.

It's on the horizon. Six, seven months. Yeah.

So we've strayed away from, from my original headline around, uh, having big news on investments being very different from the reality of building things. I think we've gotten to mostly looking at the reality of building things because mm-hmm. We are hands-on practical people who want to solve solutions rather than just build the biggest interface with the most money.

And, uh, whoever, uh, shuffles off the mortal coil with the biggest data center wins, um, at the moment, Brian's winning that one for, for the four of us. True. Um, we could, of course, we like to keep talking about these things forever, uh, but uh, we do need to call an end to this podcast and, uh, let Corey wave his magic on it.

But before we do, where can people carry on the conversations with you all? com. Yeah, Chris gunman find me on LinkedIn is probably the best way to get in touch with me.

Or also Kaja Consulting. We'll take you to that page as well. Uh, Brian Martin, you can find me at Signal 65 or on LinkedIn.

And I'm Alistair Cook. You can find me on LinkedIn. Also, you can find myself and all of my delegates from AI Infrastructure Field, day four on the Tech Field Day, uh, website you can find right now.

It's the current event. Uh, thank you so much for joining us for this episode of the Tech Field Day podcast. If you enjoyed this conversation, find us in your favorite podcast application and subscribe to the Tech Field Day, uh, podcast.

Or find us on YouTube or wherever else across the tech strong gang, uh, tech strong community that we are published. We'll be bringing you another episode looking at some other aspect of the depths of the IT industry from our delegate panels in the coming week. In fact, next week.

Uh, thank you for coming, uh, to join us. com/podcast to find all of our podcast. Cory has been making an awesome collection of podcasts.

Thank you so much for joining me and for myself and my for my panel. I hope you're having a great week.