The AI Market is Turning Away from Frontier Models | Utilizing AI Episode 37

So-called frontier models have gotten better and better, but users are increasingly focused on AI-powered applications rather than treating the chatbot as the application. This episode of "Utilizing AI" features Brad Shimmin and Guy Currier of the Futurum Group discussing the impact of tokenomics on the development of AI models. Welcome to "Utilizing AI," the podcast focused on practical applications of artificial intelligence from the Futurum Group.

Every Wednesday, we explore news and use cases of the ways in which AI is transforming enterprise IT and the industries it serves. I'm your host, Steven Foskett, President of Tech Field Day Business Unit here at the Futurum Group. Before we dive into this discussion, let's meet who's on the panel today.

Hi, everyone. I am Brad Shimmin. I'm VP Practice Lead for Data Intelligence, Analytics, and Infrastructure here at Futurum.

Good to be on the show. Hi, I'm Guy Currier. I'm also a Futurum analyst, but I'm the Chief Analyst at Visible Impact, which is part of the Futurum group of companies.

So we are looking at the world of AI, and one of the things that we've noticed is something of a pivot away from the big all-singing, all-dancing, frontier models, and a lot more interest in smaller models. There's a bunch of news stories around this this week. Everything from Apple introducing the new 27 versions of iOS and macOS, featuring a brand-new Siri that uses its own small on-device or cloud model, to Meta releasing a new version of a model called Muse Spark, that is specifically not a frontier model, that it is designed to be agentic in tool use at a modest price, to a lot of other stories.

And frankly, all of this has to do with the subject of tokenomics, which essentially is the economics of tokens. If you are trying to build AI models, how can you afford to build them? " So let's kick off the discussion.

First, let's talk about tokenomics. And Guy, I want to throw this at you. Talk to me a little bit about the topic of tokenomics and how it affects people.

As the AI market matured, users became more sophisticated at understanding how to use AI and how to, let's say, use it for automation or scripting or chat or all those sort of things. And meanwhile, the suppliers, whether they were producers of the models or those putting models and harnesses and providing related services, whether chat or otherwise, and eventually agentic producers, started to learn how to, on the one hand, assess their costs, and on the other hand, pass those costs along as prices. And that's where the token came in.

Both training and inference are based on the transformer, and the transformer requires tokens as input, essentially, and provides tokens as output. So rather than doing something like cycles or power usage or whatever else, it became really convenient to use the token as sort of the measure of an AI unit, whether as cost or price or what have you. So that's where tokenomics come in, because just like in the fabulous old days of only 10 years ago, with the cloud, where infrastructure usage got metered in various ways and charged in various ways based on whether it was storage or compute or whatever, now AI usage is metered and provided as tokens.

And managing token use and the price per token and the quantity of tokens, as we are becoming more familiar these days, one whatever, command, request, usage of whether it's an agent or a chat or whatever, one usage could cost, whatever, a million tokens, or it could cost 40 million tokens, and it all seems the same to the user. So we seem to see the emergence of a whole new practice, just like cloudonomics before, where you have all these companies providing tools or services or what have you to help companies manage their cloud usage. By the same token, we are seeing that being used in AI, and that's what tokenomics is all about.

Yeah, it is, Guy, and I love that. By the same token, I think also that the reason why this is such a big deal right now in the enterprise is simply that cost factor more than any other, honestly, more than security, more than capability in many cases, simply because companies are waking up to the fact that the idea of token maxing may look good to investors, but it certainly may not help your actual bottom line if you're not making any progress. And we shouldn't be surprised by any of this, simply because this has never been really an AI problem.

It's always been an engineering problem. And the pendulum of engineering always swings in just one direction, and that is toward abstraction and optimization. In the old parlance of developers, the only way forward is by basically caching.

And if that doesn't work More caching. And it's the same for these models, and both the consumers of the models and the providers of the models are discovering that right now. So you look at the major model hosting providers, and they're all measuring that value on watts and watt usage and availability.

And as you just said, Guy, in the enterprise, we're measuring that on token usage. And my goodness, the spectrum between using this wonderful tool that we have at our disposal effectively and wasting money is very easy to step over. That is not a big gap between the two.

I have to say that this becomes exceedingly complex when you think about the three variables involved here, which is the amount of, let's say, human time or human agent or user agent time spent on the input, the value of the output, and then the cost per token, because a given usage of six million tokens might actually be cheaper than some other use of, I don't know, 40 million... Sorry, four million tokens, two million tokens. It gets exceedingly complex.

Yeah, and it's terrifyingly interesting to me to watch how the vendor community is responding to that because those who are tightly tied into those business outcomes that you're talking about are actually thinking about ways to do their pricing not based on tokens but based on value. So they call it value pricing. So it's whatever your unit of value is in a given transaction, like accurately OCRing, picking the text out of purchase orders, for instance, and whatever workflow you've automated with their AI, you're only going to pay for what actually works.

You're paying for value. And I applaud that deeply, but my goodness, can you imagine how you would actually implement that right now? Yeah, we talked about that actually on a recent episode, that it's actually...

I love the idea that people are pivoting toward value, and I love the idea, I know it hurts, but I love the idea that companies are starting to actually charge for tokens and processing and essentially have the economic impact reflected on users. I think that most of us recognize at this point that the inexpensive AI plans and even free AI plans were really a loss leader to try to get people in the door, to get people used to using these things, but that eventually they would have to pivot toward something that is more economic for them. And we're definitely seeing the vendors do this.

One of the stories that caught my eye this week, as I mentioned in the intro here, was the Muse Spark model from Meta. Now, Meta absolutely has a competitive frontier model that is right up there with the rest of the big boys. But frankly, Meta hasn't found a market for their AI efforts yet.

And I think that Muse Spark is a really interesting direction. 1, they have a model that is not intended to be an all-singing, do all model. It's intended to be a useful model for agentic workloads.

It runs tools, it excels at running tools. It excels at long-running analysis, long-running multiple queries. It excels at being responsive and generating tokens.

And it's cheap. 25 per million output tokens, which compares favorably to every other model on the market. And in fact, it even undercuts xGrok, which had been the, I don't want to say the darling because I don't think anybody really loved it, but it had been the go-to model for people that are trying to build agentic applications simply because it was more economically viable to use Grok than to use the other ones from OpenAI and Anthropic.

Well, now here's Meta with one that undercuts that on price, probably, according to everybody that I've read that's done evaluations of it, exceeds Grok in terms of value, in terms of capability, and yet is sort of a new player on the market. We know that companies like Google and Microsoft are eagerly trying to develop competitive models. I personally use Gemini for a lot of agentic processing, and because it's inexpensive, it performs well, it's quick, it's easy to use, it's integrated with the Google accounts, that sort of thing.

It's basically just a really nice thing. And I think that the same thing that Meta's kind of seeing an opportunity here, I'm sure that every other company in the space is seeing this opportunity too. And I find it ironic that the opportunity is not in developing ever bigger models, but developing ones that are ever more useful, more responsive, and more economically viable.

Guy? Well, I actually was going to ask Brad a question because it seems to me that what we see emerging is yet another echo of cloud and before it, client server maybe, and before it, mainframe, whatever, in terms of application and data architectures, in the sense that we need architects who specialize in what I generalize as AI services, the creation of AI services, whether agentic or otherwise. We need architects, and we need tech folks I'm talking about a typical IT shop.

You want your tech team to be evaluating the-- There's more models even, you didn't even discuss the Chinese models that have come out. Many seem like sort of imitation models that are super cheap to use and seem to have many of the same capabilities, and I put a lot of weight in seem to have, because do they or do they not? Here's my point.

You want a tech team specializing in the evaluation of AI models and harnessing various other tools to combine with an architecture team, and then turns it over to an AI dev team to produce, in just your classic enterprise fashion, the applications and services that help your business run using the right components in the right way, priced in the right way. You don't need the frontier models anymore to do stuff, except maybe experiment and figure out what type of specialized model you're going to need in the end. And then you're limiting your token usage of this really expensive, super capable stuff like Fable, so that ultimately you can narrow down to maybe even home-growing or home-tuning AI models.

So that's my question to Brad. Am I full of hot air, or is that kind of what's happening now? You're spot on, guy.

Yeah. And by the way- We all know I'm full of hot air, but otherwise It's good hot air. So, first and foremost, the Chinese models are not actually cheap replicas.

6 for coding and a moderately sized model there, like 27 billion. 0014 And so it's incredibly cost efficient to use these. " And the answer to the question is yes and no.

Because as you said, man, this is a software engineering practice we're talking about. You're not talking about building models, you're talking about building outcomes in the enterprise, and that's an app that does something. And if that app is so dependent upon a specific version of a specific frontier model that it breaks the second that model is revised, what are we doing here?

That's ridiculous. That's not CICD style, ops-based engineering. That's just throwing a dart in the dark, if you will.

And as Steven knows, because I've complained about this many times, is that models should be software, but they're really not software. In that, the pace of innovation amongst the frontier model makers is such that week in and week out, you really need to be testing and retesting your implementation of their model because they do change. But the key- And that's actually my real problem ...

if you don't mind me ranting for a second, Steven. Yeah, go ahead. What's that?

Yeah. Yeah. Yep.

Go ahead. Sorry, this really means a lot to me- Please ... because I use it every day as a practitioner, and it kills me that the thing is that it's a game that we play and the model maker plays, the hosting provider plays in optimizing our spend, both of us.

So talking about watts, talking about tokens. And when I use a frontier model, depending on what harness I'm using it through, mind you, and how I'm using it, how I'm accessing the API, that model I'm accessing may itself have been routed through other models, maybe downgraded, maybe throttled, maybe going to the same model, but with a different level of, what am I trying to say? Of the number of parameters that...

Sorry, what is it called? LoRA. How refined it is.

So, I might be using a 2-bit model instead of a 16-bit model. Yeah. Yeah.

Domain specificity might be what you're going for, yeah. Quantization. Quantization.

Exactly, man. And that sort of optimization for them might not be the best thing for me, or maybe it might be the best thing for me, depending on what that use case is. So it's a game that you have to continuously play as a builder in the enterprise using AI to optimize your spend and maintain that value of the outcome.

It's been a real frustration for people building these kind of applications. I'm not a heavy builder of it, certainly not as heavy as you or the folks inside Futurum that I know are building AI models, and I don't want to speak for them, but for me, it's been a real pain because exactly what you're saying. I have some agentic applications that I've built that are running there, and it's incredibly frustrating that they stop running for reasons entirely outside of my control.

6 or whatever, and we don't have that one anymore, so tough luck. That's not a good answer to me. Similarly, what you're saying, you can have it using a specific model and then suddenly find that it's not ...

that that's not available anymore, that that model performs differently because we've decided to downgrade it some in the background without any notice. And also, I have run into frustrations related to billing. You've used up all your tokens.

Mm. And I'm like, "Well, you've got my credit card. You've got my go-ahead.

" But oops, you ran out of tokens, your stuff doesn't work anymore. It's extremely frustrating. I think that the first company that comes up with a more predictable, affordable model that isn't going to change, that isn't going to mess you up, that isn't going to break your application, I think that's actually going to be a really big benefit, more than a bigger, badder model that can do more.

But that being said, I want to point out one more thing you said, maybe you don't need a million-token context window. Well, guess what? 1 is that they increased the context window to a million tokens.

So the cheap model now has a million-token context model. You're welcome. Stephen, I've got to quibble a little bit with your characterization of what maybe we'd call a killer app, a killer model.

I think that it strikes me how similar this is to basically the development and maturity of every wave of computing, at least since the arrival of the internet on the scene, which is to say, I don't mean like the storming, forming, norming stuff, like some of that broad stuff. What I mean is that we started off with this simplistic view of using AI. There are models, there are better models and worse models, and we want to use the better and the latest, and it was largely chat or eventually copilot based.

You may remember, Stephen, a lightning talk I gave at one of the Tech Field Days where I said there isn't one life cycle for AI, there's actually three. That's now outdated. There are many, because there's the foundational, let's say, training model, whether it's the frontier or some other AI, then there's the tuning life cycle, and then there's the RAG life cycle.

Those are the three I identified at the time. Now we have many, and I think that what is killer that you're alluding to, Stephen, is and we're going to start to see these sort of abstraction layers where you can generalize the use of models so that changing models doesn't necessarily break the AI service that you've developed. And I'm just peeling back the onion just one layer to say that ultimately, I think we're going to wind up applying these sort of software or development or application life cycle models we already have and we've had for decades.

We're going to wind up applying it to here. I'm almost tempted to say that we had 3GL and 4GL, even 5GL. Now we have vibe coding.

I think it's just like infinite GL at this point, thanks to how AI works. But otherwise, it's the same model. Not the same model- Any GL guy ...

the same structure. Any GL. Thank you, yes.

And so there are model routers like Open Router that do roughly what you're saying, but I have been incredibly frustrated by trying to use them- That's because they suck so far ... because they're not- They suck because they said they were just a- They're not end-user friendly. Yeah, we're just not in the same queue.

Hang on, they are not that bad. We're not in that space. Have you used them, Brad?

I do, yeah, a lot, for those very reasons. Because I like have my- You're not typical, Brad. You're not typical ...

preferred list and the fallback list, and setting that in code is hard to do. But if I have just one API call to Open Router and I have a set of rules on the back end in Open Router that are deterministic rules that say things like, if my spend is X or if the latency is Y, go to this model without interrupting what my app is doing, I think that's what we're really after here, is it not? Just a question of how scalable and reproducible, and manageable that is.

Because I wouldn't suggest, for the same reasons I wouldn't suggest people relying on frameworks, like being able to say, oh, well, everything can just be LangChained together in the enterprise, because that's hard to maintain, and your technical debt just keeps going up and up and up. What we really need is that agent control plane that sits somewhere between the software we're building and the back-end services that we host them from, whether those are on-prem or in the cloud, and have some consistent and manageable, and most importantly, auditable and enforceable SLAs that we as companies, that we know how to build those SLAs, because we do that a lot with software, right, guys? To be able to apply that to AI.

That's what we really want. Yeah, I think what I'm suggesting as a practical matter for our audience is you don't completely have to learn something new. You can apply your experience and the, whether they're your own back-pocket frameworks or other frameworks that you already have, and identify gaps and problems that you need to fill, whether it's talent or tools or whatever, in the same way.

It's really not foundationally different. There are differences. The market is rapidly developing and catching up to what the models and agents and protocols are innovating on.

So that's definitely true. I don't want to overstate, but in the question of practical planning and getting useful things done, which is really the stage we're entering right now, we've sort of exited experimentation with AI, and we are demanding return on investment, understanding of costs, understanding of the value derived when you're there. It's still the same.

It's just AI service development instead of software development, if you like, but otherwise, quite similar. You will benefit from imitating what you've done in the past. So I wonder if we are going to see...

Well, I'm pretty sure we are going to see a proliferation of services that sit, as you said, between the applications and the models. Also, I think that what we're going to see is a devaluing of the models themselves, and even the models as a service, and a return to valuing software that does specific things. So, for example, another story that I've been watching this week was, OpenAI just released the next version of their application, their client application.

They've gone from having multiple client applications to having what I guess could be called a super app. It does everything. It's just one application that does everything.

Anthropic has a similar super app. You could argue that Siri is a super app on the Mac side now. And the idea being instead of saying, and this I think is correct, instead of saying the model is the application, you're recognizing that no, the application is the application, the model is a new tool and a new paradigm.

Brad, how do you see this? You study this field. What do you think about this?

Yeah. The application's always been the application. What's changing is what that application looks like and how we interface with it.

The days of learning Adobe Photoshop, the days of learning Excel, I think are numbered in this industry, and thank the gods. Because that is miserable, and that is not how we should measure an employee's value in a company. But what you're saying, and what I believe is very true, is that the models are all converging toward one set of capabilities.

This deep learning we're talking about is a universal function approximator, which means pretty much it'll find any pattern in anything, and that's all we're doing. So they just get better and better, and they're able to handle more and more modalities, and they all start to look and behave alike. What changes is how they're implemented, how we control them, how we align them with our values as humans and as adopters within an enterprise, for example.

And as we've been talking about on today's show, how we manage that spend on that back-end model service. So the application has always been the application, but now it looks and maybe doesn't look like anything. Maybe it just sounds like something, because we're increasingly seeing developers toss their keyboards and just speak into a microphone all day to state their intent, to have their agentic processes do whatever that is.

And I think the same is coming for the traditional workflow within the business side of the enterprise for day-to-day tasks. " Sorry, hope this doesn't trigger... Oh, crap, it did.

" That's much better than typing out a search query in the inbox itself, right? So the apps maybe are less important, and that's what I think a lot of people are referring to as the collapse of software. To me, it's never really been the collapse of software, it's always been the collapse of time to value, and that's what AI really gives us ultimately.

And how we interface with that is through the things we humans do, our senses, our five senses. So that's, to me, why there's so much energy right now going into that mode of interface, whether that's a piece of hardware that you clip on your lapel or put on the back of your phone to record and transcribe, or whether that's talking into your mic at your desk, or just talking to yourself as you walk along the seashore, which I often do. That is the computer interface, and that's what we humans, I think, have always wanted because do you think anyone really wants to walk around with a phone you're swipe-typing on all day?

Sure they do. And the vision I buy into completely, Brad. I think that's where we're going.

I think, though, to also touch back to the tokenomics question, we don't know how much it should cost. We don't know what value we should expect as yet. We don't know where that's going to go.

And I think that's the stage that we're in right now is starting to get visibility on this. That what you described of developers using a voice interface and interacting with agents that are doing things, and surely Excel skills, like tippity-tappity and poke around in cells and enter formulas and stuff, it's welcome to disconnect that from whatever value a person can provide as an individual contributor and have AI and agents do all of that sort of stuff. But we still need to construct these systems And how we construct them, ultimately gets down to the model.

I'm struck by your statement earlier, Steven, about the model becoming sort of devalued. I sort of put words in your mouth. That's not exactly what you said.

I think that in one sense, general purpose models can become decommodified, or commodified. But really what we're talking about is letting a thousand models bloom, letting a million models bloom, understanding where there's a position and a need for general purpose frontier-type models, and where there's a need for maybe a tiny parameter, giant context window, or the inverse type models, all these different types, and how they are trained and what domains they're specific for and everything. That could be a very lively and productive and profitable marketplace.

The question is how and in what way. The cost, the price, the margins, all of that sort of stuff. That's kind of where we're at, and it looks like the office that I'm in is experiencing power difficulties, so flash, flash, the AI is speaking back to us.

Guy, I think that's a great point. And unfortunately, we've reached the time here where we have to wrap. Before we go, if people want to continue this conversation with you, where can they find you?

Where can they connect with you? Guy? social.

You'll also see me on the webpage as a Futurum Group with occasional reports. And you'll see me sometimes at Tech Field Day, won't you? And for me, I live at Brad Shimmen on LinkedIn, and like Guy, I spend a lot of time on the Futurum Group website, publishing reports all the time.

So you guys are welcome to check those out. And I would love to attend a Field Day, a Tech Field Day, by the way. I don't know if anyone who's listening to this, I'm sure they have actually seen that.

They're wonderful events, so just putting the word out that I would love to go. Well, I know a guy, and not just that guy. I know- The guy ...

I will see if we can get you to one. That would be awesome. com to learn about what we're talking about here.

ai and the "Utilizing AI" YouTube channel. Thank you again for listening to the "Utilizing AI" podcast today. If you enjoyed this discussion, please subscribe on YouTube or in your favorite podcast application and consider giving us a rating or review.

This podcast was brought to you by the analysts and experts from the Futurum Group, where insights meet AI. Thanks for listening, and we'll see you next week.