Bigger Isn’t Always Better: Why Enterprise AI is Turning to Smaller, Sovereign Models | Utilizing AI Episode 39

Not every task demands a multi-trillion parameter model with a massive context window, and we're seeing increasing success with small models designed with efficiency in mind. This episode of the "Utilizing AI" podcast features Nick Patience and Brad Shimmin of the Futurum Group discussing the push and pull between frontier models and smaller domain-specific models. Welcome to "Utilizing AI," the podcast focused on practical applications of artificial intelligence, part of the Futurum Group Podcast Network.

Every Wednesday, we explore news and use cases of the ways in which AI is transforming enterprise IT and the industries it serves. I'm your host, Stephen Foskett, President of the Tech Field Day business unit here at the Futurum Group. And before we dive into this discussion, let's meet who's on the panel today.

Hi, I'm Nick Patience. I'm the AI Platforms Practice Lead here at Futurum Research. And hi, everybody.

Brad Shimmin. I work with Nick, and I look at data intelligence, analytics, and infrastructure here at the Futurum Group. As we've been discussing here on the podcast, week after week after week, there's a lot of thought around the question of diminishing returns as frontier models get larger and larger, datasets get larger and larger, the question of distillation and ownership of intellectual property, and the incredible success that we've seen by smaller and more refined models.

So let's dive into this. Brad, I want to throw it to you first. Talk to me a little bit about the situations that you're seeing in the industry with bigger models, smaller models, and the question of successful training.

Well, okay. So if you don't mind, let's put training to the side for a moment because that is a can of worms, absolutely, Stephen, and I want to dive into it. 8 and above trillion parameter realm, like we see with Kimi K3 that was released recently, and be able to do really incredible things like, oh, I don't know, a long-running task that involves maybe hundreds and hundreds of individual tasks that take up maybe three days of processing time.

That's something you're probably not going to get from a 3 billion parameter model because they'll get lost, for one of many problems. So there's applicability to think about here, and it's always been and it always will be finding the right tool for the job at hand, and that's going to be a spectrum. So I think there will always be a place for small models and super small models even.

And I think that there's always a place for the frontiers, as there always has been. In that case, it all comes down to capability, cost, performance, security. Get those four right, pick your model of choice.

Yeah, I think you're right. I think there's a definite foreseeable future for open-weight, open source models, on premises, that organizations are going to want to use. And obviously, that doesn't mean there won't be frontier models.

There will be. There'll be both. But obviously, the economics of the two are quite radically different.

But I see from enterprises we talk to, enterprises we survey, there is that desire for small domain-specific models trained on our own proprietary data, and a lot of that is going to be open weight, I think. But obviously, every time something like Kimi comes along, we obviously saw it with DeepSeek, when that came along in the beginning of, I guess that was 2025, wasn't it? There's always this slight paranoia, obviously, amongst the frontier model companies.

There's a little less reason to be spending money on them. And we go through these cycles of paranoia, accusations flying everywhere as to how the model got so good. Is it any good?

Is it absolute rubbish? And anything in between. And then you obviously see OpenAI and Anthropic weighing in as kind of subtle ways of saying, "This is great for the industry.

This is fantastic. It's the rising tide. It lifts all boats.

But by the way, do you know we're the leaders in this market, and we're trying to build a technology company with multiple layers on top of these models. " So it is always that dynamic that comes along, which for a brief period of time, there's less reason to be spending quite so much money on all those tokens, or at least it appears to be anyway. Well, that it's the appears to be that's really at the core of this, because the minute you'd peel back the layers of how these are inferencing, how they're running, how they do things like caching, and how much they charge for missed caches versus cached prompts, it can get really sticky really quickly, and you might end up spending a lot more for an open weights model that's a little smaller than you might for a Fable-size closed model.

But again, maybe if it's cheaper to run Fable for a long-running task that is a very complex task, et cetera, because you're going to be generating a lot, great, do it. If you're going to be running something that's more about, let's say, designing a front end for an application that maybe is also long-running but it doesn't have a lot of the generation side, Kimi is probably a lot cheaper To do. Not just because of the inferencing finances, but also because it has native visual capabilities built in.

So it actually, while it's running on these long horizon tasks, it can literally look at the screens it's building and go, "Hmm, let's iterate on that. " And a lot of the other frontier models are not fully native for text, image, audio, et cetera. So it all depends on the use case.

It all depends on your dynamics, your economics. Yeah. That's, I think, a really important point here, in that these frontier models are increasingly designed to be what's called multimodal, meaning that they work with a variety of different data types.

They can handle video and images and different, as well as text. But ultimately, many applications, enterprise applications, especially agentic applications, really don't need any of that. I think a lot of what people are trying to do is still text-based.

I think a lot of it depends more on basically appropriate tool calling, appropriate iterative, what they call thinking, or and also, I think it depends on a good context window. And of course, it's hard to have a good context window because that takes up a lot of memory, and that's also what a lot of these models take up. So I think that's why it's exciting for those of us who are looking at this to say, well, something like Kimi, maybe we quantize it down, we give ourselves a bigger context window, we make sure that it can still use tools.

And even though we're sacrificing something in terms of outright accuracy and knowledge set, we don't actually need that. We need something that's going to be able to use the tool. Is that where you're going?

Yeah, absolutely. And these are not black boxes. It's very much the case that through techniques like LoRA, for example, you can fine-tune a model of any size without really disturbing the parameters that it's already been trained on, the weights.

And really, if you think about how they're architected, you can customize how you use them to better effect. And it also, I should say, interestingly, that mentality has extended to the model makers themselves, and I think we're really seeing that externalized through the Chinese models right now, in particular. " Right?

And you look through the Kimi documentation, and they're taking the same approach. " And that's going to make it something that every hosting provider that runs Kimi is going to have to change how they do that, because it is unique to Kimi. But if it gets you better cost per performance, that's a good trade-off.

I would go for that sort of unique approach any day. And there's quite a few software, there's a layer of the software business being established, aren't there, to do some of that, exactly what you're saying, the model providers are doing, and sell that to companies as well. How successful, how much of that is a feature versus a product company remains to be seen.

But certainly, it scratches an itch that's out there. Well, there's one more thing I think that's really important, too, and that's that these smaller models have the potential to run on local or maybe just more affordable hardware. And I think that that's a huge detail that the industry is hungry for.

2 by ZAI model. This is a model that performs almost as well as the absolute latest frontier Claude and GPT models, but yet can run when quantized down to two bits on a 256 gig unified memory Mac. Now, if you could buy one of those, that would be great, Apple, please.

You cannot buy them anymore, but you can get them on eBay. But the point is, this is something that actually works well on a piece of hardware that you can hold in your hands and that you could buy for less than the cost of a new Mercedes. It would also work on a high-memory GPU and a high-memory server as well.

And I think that that's really key to making some of these applications practical. Because as long as you're beholden to running it in the cloud and paying per token or per million tokens, it's going to hold back what you can do with these things. But as soon as you can buy a thing and depreciate it and run it yourself for some affordable price or moderate, "affordable" in asterisks and scare quotes, price, maybe then you can get creative about what you can do with these.

5, then that's pretty cool. Yeah. Because there, you can use it or reuse it or resell it however you want.

Well, on July 27th for Kimi. Because that's when the open weights will actually be downloadable from Hugging Face. Poor Hugging Face.

I can't imagine how they're going to support that at the time. Everyone's torrent is going to be running on the 27th of this month. But the point is that you're right, Stephen.

It's one of those situations where if you can optimize down the stack, and however means you take to that, whether it's just changing the number of bits you run or whether it's changing the actual way that you do the matrix math, as we see with some of those, run your big model in-browser, which there's one that you can actually run a 27 billion parameter model in a web browser with 10 tokens per second. Come on. That is ridiculous.

But if you can do that, and you can do that with an open weights model or, better yet, a fully open source model, as an enterprise, why would you not avail yourself of that opportunity to build on top of that? So I think as Nick said, it's that friction that's so important to this industry that I think hinges upon open source and open approaches to models. That's what really, I think, drives a lot of this innovation we're talking about today.

Yeah, and it's interesting, we mentioned obviously cost, inference cost, and obviously latency issues. The other reason for doing this is sovereignty, and that's just a massive driver. I spend a lot of my time on this subject, just got off with an hour-long session with a client about this very topic.

But that for healthcare, finance, even government, and defense and those industries, this is not sending your sensitive data to external APIs is a big win. Obviously- I think people get confused about sovereignty, too, because they start thinking, oh, it's about the EU and laws and stuff like that. But no, like you're saying, Nick, I think that's the key insight.

It's not just like that. You may want sovereignty. Yeah.

The company that you work for, the industry that you work for, et cetera, right? It's operational control, isn't it? Yeah.

Oh, God, yeah. It's not just about... The data residency stuff was the origins of all this, but we've moved not beyond that, because that's still very important, but it's much broader than that.

And obviously, there's certain things, true complete sovereignty is a complete fiction. For whether you're the US government or the Chinese government, that's the fiction. If you're going right from the raw materials that make semis right through to talent and all the bits and all the layers in between, you can't control all those.

But this idea of these open weight models on-prem, when I was saying earlier, I think there's a strong desire for that in the future. This is the reason why. And as you say, it's about, there's data residency issue, there's operational control, there's understanding which models you're using and where the provenance of those are.

And then if you are a government, then there's a strategic question. " And then Kimi comes along. So it's a multifaceted beast, but at a very practical level, and it's different for each industry, and it's different for each size of company, and it's different for each country.

Sovereignty is a big deal. " And at the moment, it's obviously a less dependency on the US hyperscalers. That is very clear.

And it's very clear for China, and it's very clear for the US. It gets a little bit messier as you move further east, and you speak... I was speaking to somebody in Singapore a few weeks ago about this, and sovereignty is a very dangerous word to be throwing around if you look at the geography, because you don't want to upset either of those two large countries.

So in Germany, you're quite comfortable upsetting one of them, but you know what I mean? So it does get very geopolitical very quickly, but it's also a very practical problem. Yeah, and that's a moving target, is it not, Nick?

And that's what I think really bothers me about using hosted models and hosted APIs in general. It's not just those dependencies in terms of today I can, tomorrow I can't. It's the operational control that you just mentioned that is, I think, so vital to enterprise software.

And I know we've all got accustomed to working in probabilities curves now for software, but I remember a time when they were much more predictable, in terms of if you licensed a piece of software, let's say, whether it was closed source even, you had control over that. You knew when they were going to version, you could choose to version or not version, because you knew that maybe you hadn't tested it out in the new version, and you didn't want to break what you already had running. Well, that's not possible when you're just dependent upon the whims, both of the rollout cycle for these hyperscalers, which is becoming increasingly important to their business model, but also in terms of what they're doing internally to optimize their own spend on these models.

So you might be getting through various techniques, like different routing practices, different fallback practices, different caching practices, et cetera. What you get today, even if it's the same model, might not be the same that you get this afternoon. And that's why I think having the ability to have that sovereignty from an operational perspective with your own model on your own hardware is so critical Yeah, the messy probabilistic nature of it, isn't it?

You're right. If it was all old ERP systems run by rules-based software that we can change the rules on and control the rules, it's simple. Now, life is a lot more complicated.

And I know you've probably talked about in previous editions about obviously the Anthropic situation with the kill switch. It turns out that that was a dimmer switch, not an on/off switch, but it happened. And it doesn't matter whether that happened because of technical reasons or political reasons, it happened, and that's changed the risk landscape for people buying technology, essentially.

I think that's an interesting point that the two of you specifically can bring here is, you are plugged into this industry more than many, probably most other people because of your position as research directors, research analysts that are focused on this space. What are you hearing from the various constituencies here with regard to this? Where is the industry going according to chip makers, AI companies, AI-using companies, clients, et cetera?

I know that's a big question, but I want to take advantage of the fact that we've got you here. We don't always have you. So talk to me a little bit about what you're hearing from the industry, if you could.

Okay. So, just thinking about from the software standpoint, from the coverage area that I have for looking at folks like Databricks and Snowflake, for instance. They and the hyperscalers look at this market as optionality.

Some of those, a lot of those have their own built-in IP that they're developing in AI. But to them, the real money comes in that long tail as you get out into the mid-market. And to do that, you need to provide optionality.

And a lot of that optionality is increasingly, bring your own model and have it governed, hosted, secured by us, your dear trusted technology partner. So they don't see it as the same way that an Anthropic might see the world. They see it more as a let's allow companies to take the data estate they have to build on top of that with the query engines of their choice, the semantic layer of their choosing, and then bring models in from whether they have them running in Bedrock on AWS, on Azure with Microsoft, or they're doing it themselves, hosting it on premises.

And they want to govern that, to cover it in a blanket of security, privacy, and all around IT goodness. I think I have the privilege/burden of speaking to companies at every layer of the stack, from semis to applications and everything in between. And sometimes I don't really know their business that well, and sometimes I do.

But I think overall, I think Brad's right, there is that desire to wrap this all up. But I think the semiconductor company's point of view, it differs. But obviously, the largest one is looking at the power of open source models, obviously, and for other elements up the stack, changing the nature of the software stack.

The hyperscalers, we've said this before, I think, on this podcast, I've remembered that they had an on-premises story. They'd kind of put it in a cupboard somewhere for a few years, and then they got it out recently. And they are touting that.

Every single software company event, I'm talking software just specifically here, I go to has a sovereignty section in it now, every single one. I've been at ones, I'm not calling out names for any particular reason, just the sequence. I've been at Domino Data, I've been at Boomi, I've been at AWS, I've been at IBM, I've been at various other ones, and I'm going to another one tomorrow.

But everybody has a concern around here, and obviously, all these companies are US-based. I live in London. I tend to go to a lot of events in Europe, but I go to ones in the US as well.

But it's not just that simplistic level, but it's part of it. And I think the users at the top of the stack and the pure application users, obviously, are not quite so concerned about some of these things. But to Brad's earlier point about the kind of messy probabilistic nature of having a model now, an AI model inside the stack, which they weren't really thinking about, changes the way these applications perform.

And sometimes they're going to like that, sometimes they're going to find that incredibly inconvenient. And so it just has completely changed the software stack, and therefore, it's changed the tech industry. And if you're an analyst, that's fantastic because it creates uncertainty.

And our job is to kind of find some patterns in this crazy world in which we live right now. Yeah. I love what you said about remembering what they have.

I'm also hearing that from infrastructure companies. Obviously, from the Tech Field Day perspective, I hear a lot more from infrastructure companies. " But now it seems that there is an increasing thought, "Wait a second, companies, customers, even vendors want to be able to sort of get their hands on this thing," and they want to start thinking of new concepts like efficiency and return on investment and reducing latencies and improving performance and scalability and things that, well, frankly, we've been talking about our entire careers, but seemed to be put on the back burner for a while because this was just being developed.

And I wonder, are we seeing a transition here, an inflection point where AI goes from, "Whoa, watch this," to, "Okay, cool. " I think both are happening. I really do, Steven.

I think that in the past, we had these, like you say, inflection points and the trough of disillusionment, followed by the slow rise to relevancy and usability. But I don't think it works like that anymore. I think that it's much more of a parallel track, if you will, of aiming for the summit and falling into a crevasse and then climbing out.

And it's not the same market that it was, and like Nick said, it's much more chaotic right now, and I don't see it getting any cleaner as we go forward. I think that the pace of innovation enables us to, as practitioners and as adopters of this technology, for instance, to be able to say, "Today, I'm going to switch a part of my harness for building out software to IBM's Bob. " And the penalty for that is not that high.

And so you can fall into the crevasse with a little, much like a bungee cord, to bounce back out, if you will, to punish that analogy. And so I think that companies, the biggest challenge they have right now is really understanding which trail to really sort of take for those quick wins, and which ones to prioritize for the long game. Which ones do they think that they want to bring in, hold close, and make secure, governable over the long haul?

So you're going to mean like two tracks, the crazy track and the safe track. Yeah, I think it's this tension, I've certainly done a lot of presentations about this kind of stuff. The compression of these technology cycles from it's magic, to it's normal, to eh, that's totally boring, has just gone from a decade to a few weeks, sometimes a few days.

And so when you kind of like, we're talking obviously earlier about Kimi, there'll be another model along in a minute. And we're all turned slightly towards that one and the next one, and you kind of have to avoid yourself getting too hung up on those kinds of things. But yeah, you're right.

" So you can come along with your shiny new object, Mr. or Mrs. AI person, and tell me this is going to revolutionize the world, which it may do over a course of time.

But right now, I've got to run the business 365 days a year. But yeah, that compression of technology cycles is very real. And that's, again, I guess what makes it such an interesting time.

One other thing that I'll throw in here at the end, too, is I've been waiting for these enthusiasts around AI-assisted software development to show me the goods, show me the new products and the new features that you've been able to develop with AI. And I have to say, just kind of in line with what you were just bringing up, Nick, and Brad as well, I'm seeing them starting to show me the goods. I've noticed that a lot of open source projects are starting to accelerate their development.

They're starting to support new things. Some of the industry standards are moving forward very, very rapidly. And if you look under the covers, one of the reasons for that is because they're using AI-assisted coding.

Another newsmaker recently was Linus Torvalds telling everybody that they're going to use AI in the Linux kernel, and you should just either get on board or leave. It's an interesting time, isn't it? Because there has been a lot of skepticism about this, but yet now we're starting to see actual productive uses of this technology pushing out into the mainstream, right?

Yeah, and the models are adjusting themselves to that, or being adjusted to meet that need you're talking about, Steven. With, as I was just describing with Kimi K3 and how it uses visual alongside text to build front-end software, that idea and many others like it on how you can iterate in software development in a safe, reliable manner at scale, which has always been the problem. You could vibe code yourself into a great piece of software in an afternoon, but a week later, that thing is unmanageable because you have so many regression bugs showing up that you could never find the source of, that it's no longer runnable.

And companies realize that. So model makers and harness makers and framework developers, all of them have been working really hard over the last year to try to make this something that's more tenable. And so I used to think, when I would see a press release from Zed, it's a Rust-based IDE.

Remember IDEs? " And I don't think it is anymore. I think that what matters is responsible software development practices.

If they're not running a really serious test harness, if they're not running actual spec-driven development ahead of any code being written, then yeah, it's going to be vibe code hell. But there's no reason it has to be. I guess that's precisely why- Yeah.

Coding has taken off as a domain, hasn't it? As why it works, because you have all those things that Brad described already in place. All the software testing methodologies that have been around for decades and refined and refined.

It's a very workable domain, isn't it? What does get more problematic is the marketing profession, the sales profession, the HR profession, which are much broader, messier, less prescribed kind of areas. So, yeah, I think that's one knowledge work domain ticked off.

There's a whole load of other ones to work through. Well, it really is a fascinating situation because, like I said, for me, it really came down to, okay, we have a new tool. Maybe we have a new toy.

Show me what it can do, show me what you can do with it. " I'm also starting to hear answers to that, too, which I think is really positive and optimistic as well, that we can build better, smaller, more efficient models, that we can deploy these things in efficient harnesses, that we can deliver useful outputs. And I think that all of this is a really good sign for the industry because, honestly, nobody in the industry wanted to burn down the rainforests to build a godlike intelligence.

All they wanted to do is figure out a practical way to use this new tool that they've discovered, and I think that we're starting to get there. So thank you both for joining me for this episode of "Utilizing AI" from the Futurum Group. As we wrap up, where can people connect with the two of you and continue this conversation?

Nick? com. We're doing some really interesting projects right now, and we're also overhauling our market forecast to include a much bigger part of the AI stack.

Also, I do a biweekly newsletter called "Patience on AI," which is on LinkedIn, so have a look for that. And by the time this comes out, a new edition of that will be out as well. Yep.

Same for me, and I do some on LinkedIn. And as like Nick, myself and the other analysts here are revising what we're going to actually be writing about in the second half of the year. And this is normal for us, we do that, but I feel like almost we need to revise it every quarter or perhaps even month now because it's changing so quickly.

Well, thank you very much for joining us, and thank you to the audience as well for listening to the "Utilizing AI" podcast today. If you enjoyed this discussion, please do subscribe on YouTube or in your favorite podcast application, and consider giving us a rating and review. This podcast was brought to you by the analysts and experts from the Futurum Group, where insights meet AI.

ai, the "Utilizing AI" YouTube channel, or the Techstrong TV app. Thanks for listening, and we'll see you next week.