Navigating the Model Deluge: How Enterprise IT Can Overcome AI Choice Fatigue | Utilizing AI Episode 42

As you might have heard, the Futurum Group produces the annual Six Five Summit, three days of unfiltered conversations with the leaders defining enterprise tech. This year, we're bringing "Utilizing AI" right into the summit with a dedicated hub, a space where our community gets to gather and go deeper into the AI conversations that matter most. So come hang out and join us August 25th through 27th.

com/summit. I'll see you there. Enterprise architects find themselves bombarded by new AI models, frameworks, interfaces, and platforms, and this makes it difficult to find the right choice.

This episode of "Utilizing AI" features Brad Shimmin and Guy Currier of the Futurum Group with advice on making sense of the deluge of options appearing every week. Welcome to "Utilizing AI," the podcast focused on practical applications of AI technology and part of the Futurum Group Podcast Network. Every Wednesday, we explore news and use cases of the ways in which artificial intelligence is transforming enterprise IT and the industries it serves.

I'm your host, Stephen Foskett, president of the Tech Field Day business unit here at the Futurum Group. Before we dive into this discussion, let's meet who's on the panel today. Hi, everyone.

I'm Brad Shimmin. I am an analyst with the Futurum Group, looking at data intelligence, analytics, and infrastructure. Hi, everyone.

I'm Guy Currier. I'm also an analyst at the Futurum Group, and I'm chief analyst at Visible Impact, a division of the Futurum Group. So, as we've been discussing in the last few episodes, practitioners who are trying to implement AI across, whether it's personally, whether it's as a hobby, whether it's in enterprise, in labs, we have an embarrassment of riches right now.

We have models, we have hardware, we have services, we have platforms. It can be a little challenging to understand how to select the right choice and bring it all together, and that's what we're going to talk about today. So Brad, I'm going to throw this to you first.

Talk to us a little bit more about the various options that are facing enterprise users when it comes to AI. Yeah. Far too many choices.

And it's interesting to note that, as an analyst firm, we like to do research, and we do a twice-yearly study of 800 or so data and AI practitioners as but one. And in that study that we ran in February of this year, we asked folks what their biggest roadblock was to realizing success with their agentic AI endeavors. And you would typically think that that would be, oh, gosh, security, governance, yada, yada, yada.

No. It was integration. It was, how do I actually take this into the enterprise, into my existing practices, into my new projects, Stephen?

How do I actually use the technology that is literally hitting me in the face every day? And each one of those innovations, each one of those new releases, new libraries, new frameworks, new models, they're all yelling at me to basically say, "Pick me, pick me," because they're going to do a couple of things, purportedly. One of those would be save me money, another might be solve problems that I couldn't have solved before, and another might be to do both of those or one of those faster than the other, than I could in the past with whatever I'm using.

And so you kind of can't stick your head in the sand because it's about responsibility as a builder in the enterprise, as a company that is managing this IT resource of building out these AI solutions and increasingly agentic AI solutions. And so it's tough when you get to a week like we've just had in the past week, wherein pretty much every major model maker has rolled out a new model. So I'm looking at the Model Hub right now from Unsloth, which if you're not familiar with them, I really recommend everyone take a look at this company, that they've been known for fine-tuning models and giving people the tools to fine-tune models, and now they are building their own agentic harness and their own user experience for not just that, but to actually use these and build software on top of the models.

5 Lightning. 8 Max, you've got Muse Glimmer, which is a 30-billion model from Meta, which came out of nowhere honestly. You've got video models like MiniMax H3.

You've got Inkling, which is a spinoff from OpenAI that's developing open source models. And then, of course, you've got the Kini K3 and Laguna. Have you guys heard of Laguna or Ornith, for example?

Those are all of the last week. How do you integrate that? That is my question.

Well, do you integrate them? That's my question. I kind of wonder if the old maxim, if it ain't broke...

" I think you're really spot on, Brad, with this perspective that there is an explosion of choices. You were just talking about models. That's all you were talking about.

And there's so many layers to this stack and so much activity going on. I suppose there's so much money floating around for people to develop things. I guess my first worry is that we, within the, let's say, the expert community and expert practitioner and AI vendor community, if we are being maybe a little bit too self-referential or too much in our own industry to really understand.

Obviously, not you, Brad, especially with the research you're talking about, because that's our job is to check in with the average enterprise, among other things. What's going on there? I think that last week, Stephen and I talked about a really encouraging stage we seem to be entering in this whole market of a little bit more maturity, a little bit more understanding, beneficial competition, especially around pricing, but also, like you're saying, different options.

But more options also means more prospect for confusion, and I think one of the hallmarks of this market, two hallmarks, one is speed, but one of the other hallmarks of it is, it's very easy not to know what you don't know. Yeah. So you're better off not knowing?

Well, no, you're better off knowing, or you're better off having a trusted partner of some kind- Right ... even if it's a boutique service provider or something like that. Maybe that's the first principle is to say, "Hey, maybe there's a lot that I don't know, and maybe I need to figure out some things," because any step forward is going to be helpful for you.

Even if you're a little worried that they're making a misstep. It's interesting. One of the things that we were talking about a month ago was model routers and the fact that maybe that could help end users and application developers more effectively navigate the universe of this constant influx of new models.

I don't have confirmation on this, but the tech reporters are suggesting that Stripe has just purchased OpenRouter, which is really interesting because if it's true, well, first it's kind of funny because OpenRouter has been described as Stripe for AI models, and so if Stripe actually buys Stripe for AI models, I guess that makes it Stripe for literally. Anyway, point is, the idea was that model routers would help people not have to worry quite so much about the deluge of new models because effectively, your router makes the decision which model you're going to use Based on the parameters that you set for it. Now, my own experience with OpenRouter has shown that that is not necessarily the case with that particular marketplace.

I actually found it fairly difficult to build a structured and flexible setup, which is exactly what I was trying to build with it. But the bottom line is that this idea that somehow model routers are going to save us, well, it may be true, and it's now time for the financial services and other services companies to step in and disintermediate the end users from the model choice. Brad?

Yeah. I wish that were the case. I wish we could disintermediate.

I wish we could create another layer of abstraction that simplified all of this. But I'm afraid it's actually getting more complex in both directions. So just as you described with OpenRouter, I too use them quite heavily, and the Stripe acquisition, which I believe is final now, is concerning, just because you come to rely upon a vendor like that to make decisions about which model I want to use in which circumstance because of things like pricing.

And if that kind of is up for grabs in terms of not just the data that I'm sharing with them, but also what options I have because of their stance in the market, that's not good. That's not very open. But what I really want to talk about isn't so much that Stripe situation.

It's more about that bidirectional complexity. " Or I'm going to send this off to Five Fable, or I'm going to refer him to Glasswing because he's obviously trying to do something nefarious. But on the other side, the models themselves have their own routing issues to understand and deal with.

If you look at that slate of models that I just rattled off there that's come out, the majority of them incorporate into those models various what you call parameters, what we call hyperparameters in the AI world, but it's really just a toggle. And the toggles are knobs and toggles. You have different thinking levels.

You mostly see these marketed as minimum, maximum, et cetera. And those radically alter how the model behaves. And if you are building with these, you have to understand those nuances.

So you might, in one direction, be saying, "I'm going to try to use some smart routing," which OpenRouter gives you the options to do to make those decisions like I was describing with OpenCode, or with Cowork, sorry. And at the same time, you still need to understand, how do I actually do this one call, this exact one call? Is this a max?

Is this a min? How do I treat my actual usage of this model differently over time? And I'll tell you guys, it's not easy.

I, like you, Stephen, I got really frustrated trying to set up OpenRouter to intelligently route my request. And you know what I ended up doing was just setting it to auto and then using the observability panel to understand what my spend was so I could see if it was working or not. That was the degree of automation that I settled on.

Turning the wallet dial. Exactly. So I don't want to be that guy, especially since my name is Guy, but Brad, and maybe Stephen too, I feel like you're doing the triage before you've even created the injury.

I don't know. What's it all for? What's it all about?

I think that it's said so often it's tiring, so now I'm going to say it, get tired, which is, what are you trying to accomplish? What is the business purpose in an enterprise context? And within that, you're not doing anything different than you've done for, whatever, 100 years.

You're saying, "I'm trying to do this thing. I have a metric against this thing," whatever it is. It might be increasing customer service productivity by 10%, I don't know, whatever.

And so you spend some time as a human being with AI help defining what sort of levers can be pulled or pushed for that. What does this have to do with routing, with model routing? What it has to do with is you might want to identify model routing as an important element of it, or you might want to say it's not important.

So I'm just, we'll have the router, we'll just do open, sorry, auto mode, and we're not even going to have to worry about that part of the engineering unless, like I said, there's a problem. That's why I call it triage. Then there's something's going on, I don't know what.

Now I need to figure that out. We say over and over again, I mean, human in the loop has become a tired phrase of its own, but let's not call it human in the loop. Let's say that all of these tools all the way down to the model are in order to improve the results.

I think we are switching over from the old version, which was now I can do 10 things in the time it used to take me to do one, to now I can do maybe two or three things, but I can do them better. I can do them better, I can do them with more iterations, I can do them with more complete research because I have this trusted buddy, trusted not to be accurate, trusted to work hard on my behalf, uncovering all of the different things. So I feel like I'm a little all over the place with this complaint, but the point being, if you don't know what it's all for, if you don't know why you're doing it, something specific that you're trying to accomplish, then yeah, it is overwhelming.

Yeah. It's like I said, man, the three things. You're doing something to save money, you're doing something to do something better, or you're doing something to do something faster.

That's really it. And so when I, and I'll give you a very personal example of the pain I've suffered with this recently. 5 Flash because it's very affordable and very fast.

And as a general note, by the way, if you look at families of models, some of them emphasize one of those three use cases over the others, and Gemini prioritizes speed over capability, which is very interesting and is, again, another one of those decision factors that leads to this fatigue we're talking about. 5 Flash for quite a while. 6 Flash in the last month, and with a very much, wow, this is incredibly cheaper, faster, let's do this scenario.

So I see two of those things of the three coming out with no loss of capability. If anything, supposed improvement of capability. And how much time passed, would you say?

7 Flash. 6 in that what was happening, and what I experienced personally, is an unusual cost explosion, if you will, that was down to the agentic harnesses that were using it, either Gemini CLI or Antigravity, that were basically, when they ran into a brick wall, like a block, they would have a hard time asking for help or just resolving. And so they would just boom, boom, boom, just hammer, hammer, hammer.

And people's token costs were going through the roof. 7, one of its stated benefits is better ability to handle those situations, to mitigate those kinds of runaway costs. 6 had benefits along those two axes that I couldn't ignore.

I ran it and noticed, my gosh, the bill is, I don't think I'm using it that much more, am I? But maybe I was. No, I wasn't.

It was just going up. 7 is now here, and I am very much compelled to roll that out now. " Well, sometimes you really don't have a choice.

It is interesting, isn't it, that you mentioned the classic iron triangle there, right? Better, faster, cheaper, pick two. And yet the world that we live in has obscured, in AI, has obscured the metrics that go into that.

Yeah. A new model comes out. Is it better?

Yes, no. You assume it's better because it's new or big or whatever, but is it? Is it faster?

I don't know. Is it cheaper? We certainly don't know because there again, the economics here are so obscured that we just don't know.

And that is a cause of a lot of problems for people who are trying to develop models. And so Guy, what you were talking about, you're right. It is really the same question, whether you're talking about AI agents or contract workers or employees or farmhands or whatever it is, right?

It's always been this question of how do I organize workers on my behalf? The problem I think is that, or a problem that I think a lot of people are facing, is that it's incredibly confusing, and even if you try to answer these questions effectively, many times you can't. This is so funny because a lot of what I've been talking about this year, especially in these kinds of forums, is this isn't different.

There's so many things that are like cloud, so many things that are like when internet came in, or mobile after it, or before that, whatever, the PC. We're just reliving this over and over again. People throw themselves into it, spend all kinds of money, and then it's all the same thing.

So I've been asking myself, is this really the same? And I think you, Steven, just now, and Brad as well, but Steven, you just touched on something that may actually be different, which is that this technology is non-deterministic. 8, does that make it better?

In the old world, sure. In the new world, no, because the way models are trained, especially using multi-headed attention and transformers, the way they're trained and the way they infer is non-deterministic. " Not really.

There's a certain amount of auditability, but not fully. Therefore, auto mode for your open router, to bring it back to the earlier discussion, may just not work for a while for some reason. What reason?

We don't know. So maybe that is, I got to think about this more, that might be what is different here, and what is really the conundrum for the enterprise, where for all that we can say the focus, know what you're trying to accomplish, all the classic stuff, you actually don't know what's going to happen until you start it. And you may know in dev, but when you get to prod, now it might start doing things you weren't expecting as well.

That really could be different and new in this crazy world of technology. Yeah, new good or new bad, Guy? Because it feels like I'm not quite sure on that.

And the impacts can be so big, Brad. They can be so devastating if they're bad. As you just found out with models, right?

Mm-hmm. Yeah, and it's an argument, is it not, guys, for the smaller open source models that we talked about in our last episode? To say, well, if I can, for instance, have some transparency into the thinking tokens that a model goes through.

8, the 27 billion model that just came out, is an incredible overthinker. If you leave it on that max setting we were talking about, for instance, you will burn a lot of tokens for no reason. And if you're hosting it through OpenRouter, for instance, because not everyone can run that locally, then you're going to pay for that.

So you need the ability to understand what the model's doing to make the right decision about how you use that model, and that's something that you do not get with many frontier models. So if I wanted to understand how OpenAI's GPT models are managing their thinking tokens, do you guys know how to do that? No, because you can't.

So to me, Guy, it's a question of that non-determinism you're talking about, coupled with opacity or lack of observability and transparency. That's the real challenge for companies that are trying to put this stuff to work and to keep it working. Because as a broader example of why it can be so troublesome to rely upon a hosted API service for these, even if you're sticking with the same model, you don't change anything.

You're just, I'm using GPT Sol as just random picking out of a hat example, and I'm using it today. And then I use it again in a month, and I use it again in two months for the exact same task. Not only is the non-determinism of the model itself going to change what I pay and how fast it is and what it can do, those three measures, the model itself will likely change those behaviors without me ever understanding what went into that or why.

Not just because of things like internal model routing, because that is a thing, but just in terms of how these vendors have to organize their own tokenomics to manage their own costs. " Well, that means that I didn't change anything as a user, but the model is dumber today than it was yesterday. That's a problem.

Absolutely. And yeah, you can think you are using the same model as before and not be using at all the same model, or you can have the parameters change under the hood. Or you can have what happened to me actually last week with an automation agent that I set up a little while back.

You can have the chair just pulled out from under you as you go to use it, and that model don't exist no more. No way, no how, your application just failed. So what are we to do?

Let's give listeners some advice here. How should enterprise architects approach this embarrassment of riches and embarrassment of confusion? Wow, that's another embarrassment.

How should they approach this? Well, I'll go first. I think scoping and scope management is extremely important, starting with the end goal.

Be really tight about it, tight in terms of specific, be very specific and limited as you go. And the management of it is really important. We're all familiar with scope creep.

I think much more attention to it is probably warranted in this kind of environment. And I want to add one other thing, which is when you're thinking about your aims and the business aims, goals, try to keep in mind what I've contended since the very beginning of this is the primary benefit of AI, which is not speed, it's not productivity. Those benefit.

But I think the most benefit can come to quality, quality of the output, quality of the work, because you have this terrific amplification through automation and through iteration, more iteration, and more complete views that AI can give you. So those are the two things I would say. Number one, be really focused on scope and aware and manage scope creep.

And number two, try to make sure it's quality you're driving towards, not simply quantity. I love that. And the quality to me, you can often measure that in consistency over time, right?

If it's good today and bad tomorrow, that's not acceptable. And so my advice for practitioners looking at this, and Guy is totally right, you need to understand your scope and then start there. If with that in mind, you say, "I cannot tolerate those sorts of fluctuations or ambiguities or interruptions," if you will, then invest in an open source model, not just an open weight model, but an open source model, which isn't just the open weights, but also the recipe that was used to build that model itself, and sometimes the data sets that it was trained on, even.

If you think about it, it's kind of hilarious because OpenAI actually has two open source models that are extremely popular, GPT-OSS 20 billion model, and their Whisper models, which were speech-to-text. Both of those, the first, the speech-to-text, actually has its own business, its own little market running with companies building tools on top of this. It's been doing this for over three years now.

And GPT-OSS is coming up on a couple of years old, and it's still a reliable model because it's consistent. And it's truly open, so you can have full transparency and observability into what it's doing. So don't be afraid to treat AI like it should be treated, and that is as an IT asset that you need to have an ops mentality for, that lets you manage it on your terms, not on the model makers' or the providers' terms.

Yeah, and that's actually a really good point here. I think that one of the biggest challenges that are facing IT people, and actually people in general in society, is that this stuff is so incredibly hyped. And so if I had to give a piece of advice is don't expect too much because you might be disappointed.

If you're expecting a miracle worker that's going to always deliver the goods and is always going to be cheap, is always going to be fast, et cetera, well, you're just not going to get that. What you need to do is be realistic about what these things can do. And frankly, that's hard to do given the environment that we're in, where we are told that these models are so smart that they're escaping containment and hacking other companies and developing new viruses and challenging everything that we know.

You look at this and you're like, "Whoa. " It's not. It's just a little tool.

And frankly, I think that it's a tool that can pretend to be a super intelligent monster, but if we approach it more as a tool and as a resource, I think that we're going to be happier with the outcome that we get than if we approach it as some sort of Mr. Data that can do anything aboard the Enterprise. Do you ever wonder why they didn't just give the Enterprise to Data?

Anyway, let's set that aside. The emotion chip, Steven. It was the emotion chip.

It was a problem. Yeah, well, you saw what happened to Lore. Yeah, he still got the- Without turning this into a "Star Trek: The Next Generation" fan podcast, which we probably could do, let me quickly put it to you all.

I personally am really looking forward to the Six Five Summit, which is coming up, I believe, next week. com/summit. I want to shout out to our listeners as well that we're going to maintain a Utilizing AI hub as part of the summit.

I will be there. It'll give you a chance to come in and say hello, to talk to us as part of that. com/summit.

Maybe you can run into Guy or Brad or Nick or some of the other folks that you'll see here on the podcast during that. We're also going to have a Tech Field Day hub. We're going to have a Security Boulevard hub.

A lot of fun over there during the Six Five Summit. Where else, Brad, can we find you, and can people continue this conversation? Yeah.

com. And I'll be actually heading out to New York and then on to Spain to meet with Microsoft for their Fabric, Fabcon, they call it. So looking forward to getting back on the road, actually.

Cool. Yeah, you'll see me at events from time to time as well, such as the upcoming OCP or Supercompute or QCon. I go to that as well.

com/in/guycurrier, from those events. com. Selectively, sure.

social. Well, thank you both for joining us this week on Utilizing AI, and thank you, audience, for joining us as well. I look forward to seeing you at the Six Five Summit.

I look forward to running into you at industry events. I'll be at some of those that he mentioned as well myself. And if you enjoyed this podcast, you'll find us every week on YouTube or in your favorite podcast application.

This podcast is brought to you by the analysts and experts from the Futurum Group, where insights meet AI. ai, the Utilizing AI YouTube channel, or the Techstrong TV app. Thanks for listening, and we'll see you next week.