Demystifying AI: How Open-Weights and Local Hardware Are Democratizing Global Tech | Utilizing AI Episode 41

As you might have heard, the Futurum Group produces an annual Six Five Summit, three days of unfiltered conversations with leaders defining enterprise technology. This year, we're bringing "Utilizing AI" right into the summit with a dedicated hub, a space where our community gets to gather and go deeper into the AI conversations that matter most to them. Come hang out with us August 25th through 27th.

com/summit. We'll see you there. AI models can be a powerful tool in the hands of an expert user, and we're finally seeing widespread use of advanced models on local hardware.

This week on "Utilizing AI", Guy Currier joins me to discuss the spread of smaller models that can be used on your own hardware. Welcome to "Utilizing AI", the podcast focused on practical applications of artificial intelligence and part of the Futurum Group podcast network. Every Wednesday, we explore news and use cases of the ways in which AI is transforming enterprise IT and the industries it serves.

I'm your host, Stephen Foskett, President of the Tech Field Day business unit here at the Futurum Group. And before we begin the conversation, let's meet who's on the panel today. Hi, I'm Guy Currier.

I am an analyst at the Futurum Group. I'm also Chief Analyst for the Visible Impact division of the Futurum Group. Happy to be here.

Thanks for joining us once again, Guy. It's great to see you. Of course, we've all seen you at Tech Field Day events, and we actually work closely together inside Futurum.

Now, Guy and I have had many conversations over the years about AI and about using AI, and I'm pretty excited to talk about this this time because I think Guy and I are kind of on the same wavelength here in that this is actually incredibly useful technology, but it needs to get into the hands of people who can use it more as a tool and less as, I don't know, some sort of godlike intelligence that's going to replace us all. Essentially, AI and large language models specifically are useful in a variety of ways, and both of us have tried running these things locally. We've tried running them on owned infrastructure.

That's something that Futurum does in our labs and in the corporate environment as well, as well as, of course, using frontier models in the cloud, which is something that a lot of people are doing. So this week, we saw some pretty exciting announcements. Guy, I'm going to throw that one over to you.

Talk to us a little bit more about what you've seen happening recently with regard to putting AI in the hands of the people. Well, I'm going to go back maybe a month or two to the introduction of these Chinese frontier models. Qwen and Kimi got a lot of press, but there's a few of them.

I want to clear away and not really address the controversy or questioning about how these models were trained, how derivative they are, and so forth, and that's an important question within the industry. But I think from a user perspective, these are essentially frontier models that are really quite good and quite competitive in terms of functionality with the GPTs and the Clods and so forth that we're already familiar with. So that is an obvious step towards what you might call democratization or something like that, making frontier models more accessible to broader groups because they appeared at right around the time that there was a lot of complaining about or worry about token maxing and expenses related to the use of models and so forth, and so suddenly, here's a potential answer.

Again, whatever you think of their provenance or security issues or what have you. Right around the same time also, for folks like you and me who are pretty steeped in this on a regular basis, we started to see stories, messages, guidance from a lot of different vendors and commentators on the usefulness of models that are not frontier models. They were kind of called domain-specific models.

I say kind of because sometimes they're called that, sometimes not. These are smaller models. Remember, the GP in GPT stands for general purpose.

These are not general purpose, so they're not really called frontier models, but they could be quite useful and helpful. I think the rapid development of physical AI, where the resources can be limited and a lot of resources are expended on things like durability or physical environment or whatever, gave these a little bit of a push. So that's another option there for something that's more efficient, maybe less expensive.

But most recently, and I think what you're referring to, Stephen, was the announcement by Meta of Glimmer and Spark as open-weight, open source models. One for typical classic cloud-based hosting and inference, and the other one, an 80 billion parameter model, being small enough to download on most modern workstations, laptops, and so forth, and use locally. I should mention also that Mark Zuckerberg published a kind of a manifesto about where we are and where we're going, and I think it was a little unexpected, the response, because he was referring to so-called super intelligence for the most part in his manifesto.

But with the basic idea accompanying the release of open-weight models, significant, capable, modern open-weight frontier models by Meta, both for cloud use and local use. So when you put all that together, it really looks a lot like a market that is opening up, having more capabilities, more accessible to more people, and maybe starting to mature a little bit. I've been really impatient for quite some time about the general discourse about AI.

I feel like it's generally and broadly misunderstood, even by those who created it. I don't mean that to offend anyone on that. I think this is a really difficult new tool, as you put it, to grok, pun intended.

A new tool that appears to be a companion rather than a tool. Yeah. All the development has been around this sort of automation and everything, so you can just have something do stuff for you with, I think, lack of insight into what it's actually doing.

So put all those together, I think that this is a moment to sit back and notice and reflect and listen and look right now because a lot is going on that I think has a lot of potential, not just to shift the AI market and AI technology as a whole, but a lot of potential to finally raise what we're doing into something really focused or much more focused, much more capable, much more productive. Yeah. And to your point, there have been open source models previously.

Obviously, a lot of people were very excited about the Kimi models, for example. There have even been American open source models that have been available for a while. But the rapid advances that we've seen on the frontier models have made those a lot less attractive compared to what you can run in the cloud.

But that being said, there are a lot of people who are on Team Let's Try to Figure Out How We Can Run This Locally because of many reasons. Concerns over data ownership, concerns over subscriptions and opaque pricing and rapid use of tokens, concerns over allowing companies to control and access your data and access your plans and so on. I think that there's just a lot of mistrust of some of these companies.

Now, Meta is not necessarily the most trustworthy company in the world, if you ask people. But that being said, these Muse Spark and Muse Glimmer models are not running on Meta's infrastructure. They're not reporting back to the, I don't know, the Zuckerhive brain or something.

It's essentially an open-weight model that you can run, and we've seen a lot of progress even just in the last week or two about bringing these models to regular machines. So, for example, I was experimenting with this. There's an unsloth version that runs with Llama CPP on a Mac.

It uses 18 gigs of RAM Which is admittedly more than many machines have, but thankfully less than my new Mac has. And so I have been able to get this thing downloaded and run. At least it looks like it is.

I'm still literally peeling the plastic off the top and unscrewing the container in order to try to play with it. Looks pretty good so far. The cool thing about these models, like you mentioned, these are not backward models.

These are absolutely forward-looking. They have tool use. They're able to do multi-step planning, that sort of thing.

But they're also not intended to be a competitor for the latest and greatest, all-singing, all-dancing, high-end frontier models. These are intended to be the sort of thing that you would plug into an agentic workflow, point at some tools, get some tools up and running, that sort of thing. Essentially, the dream of running OpenClaw or something like that on a Mac Mini with reasonable performance, reasonable tool use, reasonable responsiveness and reliability, without having to spend money on either a gimongous machine with 100-plus gigs of RAM or millions of tokens in the cloud.

And I'm cautiously optimistic that we're actually kind of turning that corner. And the way that we're doing it is through quantization, through restriction of what the model is capable of. Essentially not trying to boil the ocean on your machine.

Joke intended, if you hear my fans running in the background. And instead, trying to come up with something that'll run locally. And again, I think that the importance of this is not, oh, cool, we can run this thing locally.

The importance of this is I'm a person trying to accomplish some work. I have a tool I can use on my laptop now. Is that how you see that, Guy?

I see it as, to, I can't believe I'm doing this, paraphrase Mark Zuckerberg a bit, freedom of choice and a certain amount of liberty for the, let me call it the creator. It's not just about developers. Even creator is an unfair term.

People trying to produce something, code, video, college essay. Sorry, probably shouldn't say that. Or their diary or what have you.

It's nice to think that the more intelligent, knowledgeable, and thoughtful you are able to get, just like, I'll use a car analogy. I love car analogies. The more you know about cars and how they work, the more intelligent a car purchase you can make, and the better you can maintain a car.

And maybe your enthusiasm will take you to an inefficient but fun car, or maybe it will take you to a very efficient, very utilitarian car. Maybe it'll take you more towards EVs, or maybe it'll take you more towards get your hands dirty, old-fashioned carburetor combustion engine, right? Either way, you have this freedom in terms of how you are going to buy and use and maintain vehicles whether yours or others.

And by the same token, pun not intended. By the same token, knowledge and understanding about AI models, their use, what agentic is, what automation is, and all that other sort of thing. The more options that you have, run it locally, run it in the cloud, run it on this box.

Big model, small model, specialized model for video, general-purpose model, model for translating, whatever it is, and that you don't always have to spend a whole lot on it, and you can do some experimentation. This is great for everyone. Especially because as this becomes incorporated into our lives as computers did, going from the '70s until now.

The more like the average person, maybe this is 10 or 15 years from now, but an average person will have a certain base level of competency in the use of these tools. And that will allow them to use the tools to better understand how to use the tools, for example, to feed themselves and to have their own controls through their own behavior and use so that they can really be productive, or they can just mess around. All of this, I think, is more visible to me than it used to be.

I think some of the worries that I've had are being allayed a little bit because I'm seeing-- Listen, Meta and Mark Zuckerberg, they have principles. They have a philosophy. They have a culture.

But they're also a rapacious money-making technology company. Meta has a long history of open systems. They founded the Open Compute Project.

They did it because, say what you want about Mark Zuckerberg, but he knows what his product is. And his product, frankly, is not AI, but it benefits from it. So he is trying to, just as they did with open systems so that they could build their own servers at less cost and build their own data centers at less cost, he or they are opening up the AI because what they're trying to do is become an experiential platform for social networking and connection and research and all that.

That's what they're trying to do. They're not trying to be Anthropic. Yeah.

Well, and this is really a shot across the bow, I think, of Anthropic and OpenAI from one of the other horsemen of the AI apocalypse. One of the things I want to call attention to, though, is one of the points you made, Guy, about sort of democratizing. And I think that this is an important thing.

There's been a lot of concern globally about AI because it has been only really in the hands of two countries, America and China. And to a lesser extent, Israel, but not really Europe, not really the Global South, not really the rest of Asia. And that's been a big problem for a lot of people and a big area of concern because essentially, what about the rest of the population?

And I think that one of the aspects of these open-weights models and their reduced hardware requirements as well is that it can get AI in the hands of more people globally. And of course, that also includes underprivileged or, let's just say, less privileged people, even in the US and in China. If you are a university student in the US and your university doesn't have access to Anthropic or OpenAI, it's a little bit more challenging for you to do this.

And if you're a university student in South Africa, then it's very hard for you to have access to this technology. These open-weights models that are able to be run on simpler, lower-weight hardware do open the door to people to experiment with this technology. And as another thing you said, Guy, that I really loved is that as people experiment with this technology, they will be more aware of its capabilities and limitations and less mythologizing of it.

So, I guess first off, let's talk about the sort of the global access. If more people are able to use this technology, in my opinion, it means that we're more likely to get more creative uses of it, right? Well, yeah.

So there's layers to this maturation or increasing competency. And I think you're right to hone in on the open weight release, open weights. And yeah, so what happened, trying to remember the name of, I guess so I sound smarter than I really am, the names of some of those earlier open-weight models.

But let's say that that was at an earlier stage in AI development. But what happened when they were released was there was a ton of experimentation around them. Mostly academic and in technology groups and so forth.

So you might call like the elite of AI understanding knowledge. People got to actually see what these look like, play around with them, learn more about AI, experts. people who are already experts learning more.

So that's one stratum. But now we're at a stage where anybody around the world can, well, let me say the next stratum, not the elite, but tech-savvy tech expert. These could be experimenters, these could be hobbyists, they could also be students, and they could be tech experts in other fields, where now they can download, use, experiment, see how it works, try their own tuning or who knows what locally.

And because it's open weight, they have full access to the trained version of these models. And as we keep going, what you might call lower and lower strata of tech expertise get their shot at it. But we're talking about globalization.

That was your question. If anyone, anywhere in the world, because the world is connected essentially by the internet. It may take longer or be more expensive in local terms than it would be in the heart of Europe.

But pretty much anyone with a mind to it can find a way to at least look at, examine, experiment with an open weight model. It reminds me to some degree of when Henry Ford more than doubled the daily wage of his own Ford factory workers and reduced their hours, because that is taken in sort of economic historical terms as this watershed moment where the manufacturer realized that his primary buying base could have a core among his own employees and their relatives and so forth. And paying them enough started a sort of a virtuous circle.

At least that's the concept anyway. And in a sense, it's also a, not a wealth transfer, right? But the phrase is escaping me.

It is a transfer of value because Meta didn't build these things for free. They invested billions of dollars in building these things. And Meta is saying we are going to benefit more by the uplifting of the entire market, and they're not going to be the first to say that or to do that.

And that has global impact because, like you say, there's three, maybe four epicenters globally of AI model development and of AI service and application development. But in a closed system, they remain at those poles. With an open system, and this is just the beginning of opening it up, with an open system, then anyone or many, many more people at least, with gumption and with maybe a few connections and resources, wherever they are in the world, can create their own little development hotbeds wherever they are and benefit their communities with that.

Yeah. Who is this on this? This is not my usual attitude on podcasts.

I know. We're all bright and sunny today. I know.

Is it really you, Guy? I refuse to answer on the grounds that it may incriminate me. So, it is a funny situation because, like you say, Meta is not a company known for altruism.

In fact, this is a company that's known for doing whatever it takes to increase revenue and profitability. And the same is true, you mentioned the Henry Ford situation. The same is obviously true of Henry Ford.

He was not an admirable man in many ways. And in fact, he did not, in fact, double wages so that his workers would be happier and would be able to buy his cars and stuff. He later adapted that story.

He doubled the wages by basically giving people essentially a 50% profit-sharing bonus if they had been on the job long enough, and if they qualified, and if they met the standards of his really weird sociological department, and so on. But the benefit of it for him was that he reduced worker turnover immediately. He stopped the workers from unionizing and essentially had a much more reliable workforce.

The benefit to the workers, of course, was that they essentially got twice the money, and they were pretty happy with that. And this is a good analogy I think it's a very good analogy for what's happening here with Meta because, like I said, Meta is not the most generous company out there. But by giving away these models and by trying to, I guess, turn the ship from proprietary to open weights models in the US, I think that what they're doing is they really are similar to Henry Ford, giving the opportunity for people to benefit more broadly from this technology in a way that will ultimately benefit Meta but will actually benefit the rest of us, too.

And again, I'm usually the grumpy one on podcasts myself, but I'm actually going to be optimistic here as well because I feel like this is going to change a lot because Meta is seen as having a seat at the big kids' table of AI. And so when OpenAI and Anthropic are looking at their models, they may start saying, "Well, I can see that the tide is turning toward open weights. " And I could see that being more of a standard thing that happens going forward in a way that benefits a lot of us.

Now, it will benefit them. Anything that Meta is doing here will benefit them. But it will benefit us, too.

And congratulations to them. Now, another thing I want to call attention to is another point that you made earlier, Guy, and I think this is really important and really insightful, and that is, as I said earlier, making models more widely available and allowing more people to use them in more different ways and experiment with them without worrying about the bill coming due is going to make people more-- They will better understand what the things are really capable of and demythologize them and really, I hope, puncture some of the bizarre supervillain marketing that we're hearing from some of these companies and help people to realize that this is basically a math algorithm that's able to do some pretty cool stuff. It's not Skynet coming to destroy us all.

Even though that messaging serves the needs of a lot of these executives, that's not really what we're talking about here. And I think people really will be able to have a better understanding of what these things can and can't do in order to better use them. Talk to me a little bit about that.

Yeah. So Zuckerberg calls out that particularly towards the beginning of his manifesto, expressing incredulity that doom marketing is a really good approach for these AI companies to build their business and build consensus around the value of AI. Namely, this is going to change everything.

You're not going to have to have a job anymore, like that kind of marketing. And I have to agree. AI is a tool to help you create or research.

It is not a thing that creates or does research for you. But it is also, as I've said repeatedly, designed-- The frontier model's design point, or at least the LLM's design point, is to appear credible and correct. Someday soon I'm going to have to write a mini paper on this, on how that is so They're not actually correct necessarily, as we all know.

But they really are designed to fool you. That's their primary purpose. You're right, Guy.

I was wrong about that earlier when I said they were designed to fool you. They're really not designed to. You see where I'm getting my tongue tied?

Yeah. Exactly. So more exposure, more discussion, more understanding, more mistakes that we learn from.

The more mature at whatever stratum you're at, and the better understanding that you get about how these cannons should be used, and the less likely you are to be fooled. But also, understanding the supremely wide variety of AI. The supremely wide variety of it's not just about LLMs, and it's actually not even just about computer vision and all those other sort of things we're familiar with.

Many domain-specific or code-specific or whatever it might be models that are smaller, more capable for what they do, maybe even less capable in some ways, so it's more obvious that you are working with a tool and not working with someone. And all of that will open up. Maybe it won't counter the worry about power usage and big data centers and all that other sort of stuff.

Maybe we need a different type of movement or different type of understanding to combat that. But it should allow us to start to incorporate this really very promising and useful tool and tool set into a lot of parts of our daily lives and give us a little bit more comfort because we also feel like we know that we can keep it under control. Yeah.

And I think that's so true. There's so much anti-AI sentiment. I think that making the technology available to individuals globally with much lesser hardware and financial resources- Yeah ...

letting them-- I love the idea of making things less seemingly capable. Hear hear. And I'm not talking about that little note we get now that says, "You should double-check this.

It might be totally wrong," like with every chat response. I mean something that's more evidently- Yeah. And again, I've harped on this before on the podcast, and I will harp on it again.

I think one company that seems to be doing a really good job of this is Apple. By deploying smaller, less capable, but more user impactful, experience impactful models in iOS and macOS. I think that they're driving it in the right direction.

I see Google doing that as well. " I think that's good. And, I think that this all spells a real turning point for AI.

And I hope, Guy, that you're right about these points, and I really appreciate you bringing them to the audience here because it can sometimes feel like the world is anti-AI kind of generally. I think the world is anti-bad things around AI and anti-- Well, basically the same way I said the world is anti with regard to finance and politics and so on. I think the world likes new technologies that help us to achieve things in our daily lives, and I think that this is a way to make it happen.

If we can learn through this experience to keep breathing, to keep your ears and eyes open, because the next thing is going to be when they start talking about AGI. If we can just do that so that our approach to these profound technological waves is a bit more measured because of the last one we went through three years ago that'll be a wonderful outcome. Yeah.

I totally agree. Thank you so much, Guy. This has been a fun conversation.

Maybe not as gloomy and aggressive as some of our episodes have been, but I really appreciate having you on here. Before we go, tell people where they can connect with you and continue the conversation with you. com, where you will see my occasional postings and coverage as a Futurum analyst.

com/in/guycurrier. I tend to run feeds there when I'm doing events or tech field there, what have you. social.

Bluesky is my primary little tweeting platform, so to speak. Tweeting, skeeting, whatever we call it these days. Thank you.

And you'll find me over there as Fosket on Bluesky as well. Again, it's great to see you there. You can find me on LinkedIn as well, even on the Mastodon if you want.

" If you enjoyed this discussion, please subscribe on YouTube or in your favorite podcast application, and consider giving us a rating and a nice review. This podcast was brought to you by the analysts and experts from the Futurum Group, where insights meet AI. ai, the "Utilizing AI" YouTube channel, or the Tekstrong TV app.

Thanks for listening, and we'll see you next week.