Open Weight Models Are Cheaper, So Why Aren’t Enterprises Using Them?

ai Leadership Insight Series. I'm your host, Mike Vizard. Today we're with Ameya Kanitkar, who's CTO for Laradin, and we're having a little chat about, well, everybody's talking about open source and open weight and models from China, but I don't know, we may be talking a lot about it, but I'm not sure we're seeing them just yet.

Ameya, welcome to the show. Thanks, Mike. Thanks for having me.

So what's your assessment on what's going on here? Is this kind of like maybe everybody's checking out all these new models as a stalking horse to kind of see what they're all about, but they're not really implementing them yet? Or, is it just that we're too far ahead of the curve?

Good question. So let's get some true facts, right? The facts are that we now have four labs, at least from China, which are releasing open weight models on a quite regular cadence, and they're generally not more than three or six months behind frontier American labs, which is largely Anthropic and OpenAI.

But when we look at our data, we do not see widespread adoption of these models in production in enterprises. So they are roughly under 10 or 15% of usage. At the same time, we also hear all American companies, all enterprises are talking about how these AI costs are runaway costs, and they're all worried about it, right?

Nobody actually planned for any major AI spend at the beginning of the year. And in some departments, such as engineering, it's almost 8 to 10% of labor costs, right? So these are the millions and millions of dollars in costs.

So it's an interesting observation that on one hand, your costs are really high. On other hand, you have models which are objectively 90% cheaper, and we don't have enough of a usage. And why is that?

And is that a political concern? Are people worried that these AI models might be tainted in some fashion or another because they're coming out of a country with more stringent controls over data and how certain questions are answered? Or is it more just the awareness because some of the folks I talk to are not sure even how to implement an AI model just yet.

Yeah. I think it's a mix of both political and technical concerns, and also just some of the facts. So let me give you an example.

So let's talk about the politics and sort of the security aspect of these models. Couple of things are true, right? When people say, "Hey, use Chinese models," what we are really talking about is these open weight models, which are fully open source and fully open weight, running on American servers, American data centers.

So companies like Baste, Fireworks, and other large cloud providers take these models and run on NVIDIA GPUs and serve them, and they're fully served from American servers, fully owned by American servers. So in theory, these are open weight models, but they are still largely from American servers. So why is the concern?

I think the problem is that nobody has fully figured out what a truly secure model means. So in theory, this should not be a concern, but because, and this is just my perspective, is that because we haven't truly figured out how these models can sort of influence decisions in sort of an nuanced way, I think companies are a little bit slow to adopt. But that is changing slowly because these open weight models also mean that you can take these models and further fine-tune them to your specific needs and then use them.

So then you have a much one level more degree of control than just taking those models vanilla and just using them. And I'll go into more details on that and some companies who are doing that. So that's first aspect of the problem.

The second aspect of the problem is that some of these models, when you see those benchmarks, and when you say, whatever, on Terminal Bench, Frontier model is 65, and let's say DeepSeek V4 is 63 and only two points behind. In reality, it's not two points. In practice, that doesn't actually translate.

1 would cost us $30. And if you just look at the GLM prices, it should cost $3 on GLM models. But in practice, it took $10 or $15 of actual costs, not $30, but not $3.

So there is some amount of this benchmark gaming going on. So these models are good, but they're not necessarily as good as the benchmarks say. So you kind of have to take it with a grain of salt.

So that's kind of the second aspect of it. So you have to do your own evaluations. And the third aspect of it is that there is a moat in these harnesses also, which is not always obvious at the beginning.

So, when I say harnesses, people are using Cursor, or people are using CloudCode, or people are using Codex. Basically, most people use these models not directly, but through some harness, especially in engineering. And companies don't usually have the right infrastructure, and I can talk about it more, but they don't usually have the right infrastructure to switch these models relatively easily without changing the development workflow.

So there's a lot to unpack there, but let's start with the benchmarks for a minute. Are the benchmarks trustworthy, or is this another example of lies, damn lies, statistics, and benchmarks? I think there are a few things to unpack here in the benchmarks, right?

One is that now these models are so good, it's actually very hard to build good benchmarks. Right? So you cannot just build some simple benchmarks.

These models will just solve these problems. So you really need to come up with some really hard problems for these models to actually come over and solve them. So that's problem number one.

So building high-quality benchmarks in itself is actually a hard thing to do. The second problem is that these model companies buy all this data from all these different sources, and the insider, I wouldn't say insider knowledge, but insider acknowledgment is that a lot of times this data includes some of the benchmark questions also, inadvertently, not necessarily intentionally, but inadvertently, because you get all this data from all these sources. And if that data gets into the model training, the model actually just knows how to solve this specific problem and can do really well on the benchmarks, but not on the problems that it has never seen.

And you can call it sort of benchmark corruption or sort of training data corruption, but that is now, intentionally or not, is now fairly widespread. That's why we need new benchmarks, and they come up with new benchmarks all the time. But that is happening, right?

So, that's kind of the second part of the problem. And that's why you see a large gap in practice, how a model behaves versus what the benchmarks say. There's also a lot of incentive to actually go for the higher benchmarks because you get the right press and all that stuff.

But without saying they intentionally do that or not, it is true that some of these training datasets have these benchmark questions, and which makes a model do better on the benchmarks than in actual practice. There's also a lot of talk now about open weight and open source models coming from the West, I guess. Or from the West, I guess, is everywhere but China.

But, are those going to be equally viable, and might those be the things that drive the next level of adoption? Because theoretically at least, there won't be as much concern about their provenance. I think absolutely.

I think that's a Goldilocks zone today, right? Because those two things are at the same time true, right? So three things are actually true.

Not every task needs a frontier model. Right? So people have realized that, oh, there are actually majority of tasks, in fact, don't need a frontier model, right?

We also know that these frontier models are extremely expensive, and we also have these concerns about sort of a political aspect of open weight models coming from China. So if you take all these three things together- Anything in between is actually perfect. You have cheaper models which you have no political risk, and you can use them.

I think the problem is that, at least until three months ago, we didn't have equally good open weight models coming from the West, and I think that is now changing slowly but surely. So two developments have happened. Actually, three.

One is that the Mistral model that came last week, or maybe a week ago, that was actually a pretty decent open weight model that was coming from Europe. We have NVIDIA very heavily investing in open weight models here in the US. They already had a very good Nemotron model, but they purchased two labs in the recent weeks, plus the Lightning Face acquisition.

So clearly, NVIDIA is positioning for their own open weight models, and they obviously have the chips. So that's second, and we also have Thinking Machines, which has the platform to build open weight models. So I think then we also did a recent acquisition on one of these labs.

So clearly it's in a golden love zone, and I think I would be very surprised if we don't have a decent open weight models in the next three to six months. Right. So to your point about that, how do you think this might break out?

Because, it seems to me at least that 80% of the workflows in your average enterprise are served by something that doesn't have to be a frontier model. A lot of the stuff that we're doing is not heavily reasoning required, like small amount or middle-sized reasoning will suffice. So, is this going to be like an 80/20 rule where 80% of the workloads are going to just need some sort of open weight model, and then, if I want to go cure cancer, I'll go get the frontier model?

Yeah. I think it would be useful to understand where the actual token costs are. So the most of the token costs are not when you're in ChatGPT interacting with ChatGPT.

That's not expensive. What is expensive is when things are agentic. And those agentic workloads can carry a very large percentage of token costs.

What we see, what Laritancy sees in our customers is usually 90% plus is the agentic costs. And those agentic workloads, if you count by number of users, they're not even 80/20. I would argue it's like 95/5.

Like 5% are agent and 95 are the rest, but those 5% probably take up 90% of the costs. But it is also true that agentic workloads are increasing at exponential rate. The ChatGPT usage is not necessarily exponentially growing higher now.

There may be more users, but individually it's not. But agentic usage is growing literally exponentially higher. Like the number of tokens used where these agents easily reach millions and billions of tokens, which usually humans cannot consume that fast.

And in that agentic usage, then you have frontier usage and sort of less frontier usage to be used. And that's where the split is. And that's where all the focus is, because that is what's going to consume 99% of the tokens in the near future.

So what is your best advice then to somebody who's in charge of all this? Because it feels like it's pretty fast-moving, and whatever plan I make for today is probably going to be obviated by some advance tomorrow. So how should folks be thinking about it, and how dynamic might this all get?

Am I going to just kind of swap in AI models per request, or is it going to be more workload specific, or how does it play out? Yeah, great question. I think the first advice I have is that you need to be ready for the world where models are going to change.

And most enterprises are not ready for it. So what does that mean in specific terms? Most companies do not have an API gateway or router that would sit in between.

And this is a fundamental piece of technology that would allow more control for companies to sort of enforce policies. So what I mean by that. If you don't have a router, what you have to do is you have to go back to engineers or end users and say, "Hey, this week, use this model.

" What you want to do is that you want to allow your employees and your agents and your engineers to use the same set of tools that they use and now have mastered. And, so they continue to use, whether they're using Cloud Code or Cursor or whichever tools they're using, but have an ability to, A, control the costs on the backend, essentially have a flip switch. And B, have an ability to change the model underneath, even though if you're using a Cloud Code or Cursor or any other model.

And that capability can be built using the router. So there are several routers now in the market. And what I would argue for is you want an intelligent router which automatically switches models automatically.

So what that would do is that it would allow your teams to work the way they are or the way they're learning, and you don't have to go back to them every week and change it. But at the same time, take advantage of the new advances in various open weight models to flip automatically on the backend to control the costs. And if you don't those two things together, you'll be in either a constant babysitting wait, like which is how much it might cost or you have to go back to your employees and retrain them all the time.

So that is one recommendation we definitely have, and surprisingly, it's not that widespread. It's still early days, I think. But coming back full circle on the models coming out of China, can they be used safely?

Are there things that people can do to kind of apply some governance or some sort of controls on those, or is the issues that people might encounter so very deep into the model itself that, well, they may not be suitable for certain sensitive use cases, shall we say? In theory, they should be fine. Given they are open weight and you can actually train them the way you want.

Plus, the data is not going to open. You can actually host them in your own on-prem machines. I think it's mostly about sort of, like certain things just take sort of common sense and everybody kind of accepts the fact, and then they kind of take over.

I think most companies are still kind of looking at everybody else to see is this really safe to save or to use. And I think that we'll get over that hump sooner or later because the actual AI token costs are growing so quickly that there is now enough incentive to do that. So I'll give you an example.

Today, Coinbase announced that they used the GLM, one of the open weights model from the Chinese labs, and then they post-trained it too on the internal Coinbase data for the fraud protection. I think it's a tweet from Brian Armstrong today. And I worked for Coinbase, and we used to work in crypto and all this stuff, and we had the most stringent safety constraints I've ever seen in any other company.

And if Coinbase can do it sort of safely, I think that most companies will eventually build it. So somebody needs to sort of chart the path, and then somebody else can follow. But that's an example where you take an open weight model, and then you actually fine-tune it on your own company data, to make it useful for your specific use case.

So now the province is now no longer just a pure open weight model, but now you have your own secret sauce in it to do it for. That's the other benefit. Because not only you save costs, but now you have models that do specific things.

In this case, Coinbase was using it for fraud analysis, which is also very actually security sensitive use case. So that's another example where you can actually do those kinds of things, to make it happen. But I think as long as people are sort of get over that hump, put the right infrastructure in place, and then have the right governance systems in place, they can cross the bridge.

So at the end of the day, is this whole debate about where the model came from a bit of a red herring then because, well, if they're open weight and we can control the data, then all models are relatively equal in terms of, at least politics and economics, and then it just becomes a question of which one is best at the task at hand? Yes and no, because you also have a limited amount of bandwidth to make sure that it's absolutely safe. So there's a limited amount of bandwidth that you can put out to ensure that everything is safe.

Just because it's open weight, somebody has to still do the analysis to make sure there's nothing nefarious in it. And I think that bandwidth is limited. And increasingly, enterprises would rely on third parties to certify these kinds of things before they can use it.

Just the sheer number of models we have and the number of changes that are happening so quickly makes this problem sort of practically quite difficult. All right. Folks, you heard it here.

We have more choice than ever, and arguably the models might be becoming more of a commodity, but it takes time, skills, labor, and cost to implement them in a way that lets you take advantage of that, shall we say, smorgasbord of AI models that are out there now, and you're going to have to make some decisions accordingly. Hey, Amiya, thanks for being on the show. Absolutely.

Thanks for having me. ai Leadership Insights here. You can find this episode and others on our website.

We invite you to check all those out. Until then, we'll see you next time.