News from the AI World – Techstrong AI Podcast EP 22

Hello and welcome to the Techstrong AI podcast. I'm Amanda Ani, and with me today is Mike Baard. How are you doing today?

I'm great. I'm in New York today, so it's a gorgeous sunny day here. It's, it's not even 80 degrees, it's amazing.

Ah, that sounds so nice for us. I believe it's gonna be 106. Yeah.

All right. Well we're gonna start off with some RI group news. Uh, this is an article on Techstrong AI and nearly half of some 1000 enterprise buyers and influencers of AI products intend to change or add new vendors this year based on a study by the Futurum Group.

So can you go into a little bit more detail about this and how is this impacting businesses? Yeah, I found this particularly fascinating 'cause I've been trying to struggle in my own mind. It says, well, every vendor and his brother's gonna add AI capabilities.

So what would make one switch from one to the other if they all have similar AI capabilities? But, um, you know, John Schwartz wrote this for us and one of the things that he pulls out is that a lot of these AI implementations are a little shaky. And so people are now testing this stuff and determining which ones are real and which ones are kinda AI washing.

Uh, a lot of folks are, uh, because they're getting beat up by either investors or they're sea level execs at their company have added capabilities. And it turns out that it, not all these things are well vetted or well tested. So this seems to be a lot of concern now, and I imagine a lot of it organizations are actually putting a lot of this stuff through their paces.

'cause uh, it's a matter of trust at the end of the day. I mean, and, and part of the problem is we keep positioning this stuff as if it's, uh, always gonna be a hundred percent right. And all these models are probabilistic in the sense that they are gonna be prone errors and we can do things to reduce those errors, but it's never gonna be a hundred percent perfect.

Now, if you've got a process that needs to be a hundred percent perfect, well then maybe you need to put some humans in the middle of this conversation because otherwise, um, if you're relying solely on a gen AI platform, I promise you there's gonna come a day when Murphy's gonna come up and bite you. Yeah, absolutely. That human element is still very important.

And I think sometimes business leaders are putting a little bit too much faith into these AI tools. Well, and they all have dreams of, you know, reducing the cost of labor to zero and creating all kinds of interesting new profit margins and being, there's some truth to that. But, you know, remember on the other side of that coin, um, so let's say that I implement all these AI workflows and I reduce my costs.

Well, I'm gonna have a competitor who within about 30 days is gonna do the same thing and then they're gonna reduce their price to match the gain market share. And, and we're gonna have this rapid race to zero. So be careful for what you wish for here because maybe not everything's gonna play out the way you think it is.

Customers are pretty savvy and they figure out exactly what you're doing and how much it costs you to do it. And then they start figuring out things like, well, what's 8% of their total cost? 'cause that may be the actual price.

And so if your total cost is a penny, you're gonna be making 8 cents. Yes, indeed. So moving on a lot in the open AI news world.

So we have a couple of different articles. One is OpenAI has scooped up rock set and enterprise search and analytics startup for an undisclosed sum, um, on this past Friday. And this is a pretty big, uh, significant acquisition.

Additionally, the need for more safety testing has delayed this month's launch of open AI's latest voice assistant for chat GPT called Voice Mode, and they'll be now doing a broad release of the assistant in the fall. So what are you hearing from companies in regard to open AI these days? Well, I, I'll give 'em hats off on two points.

One is, hey, they had the courage to kind of stop in something that they knew or suspect it wasn't gonna live up to expectations. Um, clearly there's been instances where Google should have followed that advice and now at least we're starting to see open AI behave a little more responsibly with this stuff and say, Hey, uh, we're just not gonna experiment on customers and hope for the best. So that's a step in the right direction.

The acquisition of Rockend is pretty damn smart. So the thing about, uh, the way we work with LLMs and the way that we customize and extend and train them is we show them additional data that's in a Vector database and we let the LLM see that data and then it will compare the data sets that it has versus that data set. They generate some sort of recommendation.

The challenge has been that the LLM has historically has run through red data first and then went to your data second. So you get a lot of noise in the system that can still lead to hallucinations in that approach. Rock sensor's interesting.

'cause they have, uh, a way to run that algorithms in parallel across both data sets at the same time. So you're gonna have less hallucinations with that model and I think we're gonna see some interesting things happen across the board here, um, in this whole sector and, and, um, it's just gonna be a a, a fascinating time as they say. Oh yeah, definitely.

And uh, they've got a lot of things that they're working on and of course in this big race and competition amongst other, uh, big companies. So it'll be interesting to see how everything unfolds with them. So moving on, speaking about databases, we have another story about Oracle.

Can you give us some details about this one? Yeah, this is kind of a rock and roll as well. Um, Oracle is saying that it's gonna embed the LLMs directly into the database.

This makes a lot of sense, right? Because the LLMs are looking for data and well, we're actually gonna bring the compute to the data. I know that's a radical idea as to way we used to do things 20 years ago before the cloud arrived and everybody decided we were gonna move data into the cloud, but now we're coming a little bit full circle.

It's still in the cloud, but at least we're gonna bring the algorithms and the compute to where that data in the cloud resides. Eventually we might even do that on premise, which would be even cooler. But I think this is the beginning of a trend.

I think everybody who sells the database is gonna have support for LLMs in the database and we will go from there. And it's just another format. This may be the dominant way that enterprises wind up consuming generative AI and customizing it for their own purposes.

And I may occasionally need, uh, to call out to a very, very large language model run by open AI or somebody like that through an API and I'll do that, but LLMs are coming in t-shirt sizes. Now there's small ones and medium ones and large ones. And if that's the case, I can certainly run the small medium and even some of the smaller, larger ones in the database itself and drive much better performance.

And oh, by the way, I don't have as much security concerns and I certainly don't run into the headaches of moving data as they used to say back in the day. Nothing good happens when you start moving data. Yes.

And um, you know, to your point too, I think what we've been hearing a lot from business leaders these days is they are looking at whether or not it might be better to be on premise than in the cloud. And then whether or not their project involves a large language model or if it might be better in a, uh, using a smaller language model. Yeah, I think the whole thing's gonna wind up being a little hybrid.

I think we'll run a lot of LLMs, even smaller ones on, you know, personal devices. You saw all these kind of new PCs that are floating around out there. Um, and then they will talk to, uh, medium sized LLMs that might be running a data center.

And then they too will in turn talk to LLMs in the cloud and they'll all have these, um, multiple interconnection points and it will just be one giant hybrid language model framework. Yep. So next up we have a new story.

This is about a robot coworker of sorts courtesy of Orbi ai, which claims to have developed the only available enterprise ready automation platform that mails patented in AI with intelligent AI agents in a large action model. So what are your thoughts on this one? I think this is a cool idea.

I'm not sure about the only, I think though that everybody who's doing anything in the automation space, uh, is gonna extend those things to include LLMs now, uh, are they a little further down the path maybe because they wrote something natively and they pulled all that stuff together and they didn't have to smash, you know, old legacy code together with um, you know, some sort of a newfangled LLM probably. Um, but our organization's gonna can replace that which they have today to do that unclear. The futu room survey that we referenced earlier suggests at least half would consider that, so we'll see.

But, um, you know, a problem with all these startups is they gained any traction at all. They seem to get scooped up and acquired and rolled into something these days versus becoming a standalone company forever. And I got this nasty feeling then all this AI stuff is starting to feel like a giant RR and D project for other vendors to essentially roll up in time.

And, you know, maybe we'll just have the same old players again using AI instead. I'm not quite bought into the fact just yet that there's gonna be like, and AI standalone companies that are gonna stand the test of time. We'll see, It really does seem like if they're a smaller company, it's a lot.

It's a big struggle to stay, uh, a afloat without being bought up by a bigger company. Right. And if you're, uh, an enterprise, you know, that's, it's in the back of your head.

It's like, okay, do I really wanna spend an inordinate amount of time with these guys and kind of develop that level of investment and those skills and everything that goes with it to wake up one morning and find out that they've been acquired by some other company, then I kind of can't stand. Yep. Um, playing that game, the AI game, That's what it is.

It might be called AI Roulette. Yep. All right.

So next up, um, again on Textron ai, Datadog has made generally available an ability to observe large language models that IT teams can use to monitor, for example, latency, token usage, exposure of sensitive data and toxicity. This was announced at the dash 2024 conference, so can you share more about this? Yeah, I think this is pretty cool as well.

We need some ability to see what's going on with these LLMs and uh, without being surprised. And we've all seen examples where people are getting outputs that are, uh, shall we say, concerning at the very least. And plus we need to monitor the performance.

We can have observability today that we use broadly in DevOps environments for at least monitoring, and now we're moving to observability to be fair. But LMS are just one more thing that we need to keep track of and they're directly part of the application environment. So why not use the set of, uh, capabilities that we're already using for other software components and apply it to the LLMs.

At least that's the data doc. I think there's a lot of folks out there who are already providing some sort of, um, I guess we'll call it LLM native observability platforms, but how many observability platforms are you really gonna want? How many do you need?

I don't know. But, uh, I guess it all depends on who starts the conversation. A data science team might find the observability provided by some startup that's native better or may decide that, you know, those people speak their language better.

Other folks are gonna say, Hey, as LLMs become part of the IT operations framework and the IT team especially starts taking over more of the management of the inference engines and everything around that, maybe it makes more sense just to do that with, uh, an extension of our existing tools. It's a battle. There are lots of observability companies and I guarantee that all of them are gonna be extending those platforms out to LS.

Yeah. And not only are there lots of observability companies, but there's just so much more to observe these days. I mean, business leaders have so many things that they're trying to keep track of.

Even just the large language models, there's so many being used. Yeah. It's impossible to keep track of the application environment without some sort of platform like these because it's just too complex.

There's too many dependencies, too many microservices, nobody's sure exactly what gets connected to what, and it's not uncommon for folks to have issues where there's some sort of degradation in performance, but nobody knows why or what the root cause of this thing is. And unless you have some tool that lets you launch queries and find out what's going on, or maybe in a more advanced state use of machine learning algorithms to kind of surface what those issues are, you're gonna be looking for those issues for months before you find them. And then they mag make thing about it is it takes you for two minutes to fix it once you find it.

That's how it always seems to go. So the last of our news today is about Lenovo. Lenovo has added a bevy of platforms optimized to run AI apps, including offerings from third party partners that have now joined its alliance program, including a virtual assistant that provides customer assistance via a kiosk, which has been developed in collaboration with Deep Brain and nvidia.

So how do you think this will impact the enterprise? I'm hoping that Lenovo vetted these folks and that's part of the value proposition. Uh, that's not always the case with some of these, um, OEM deals, shall we call 'em, where sometimes they're just happy to have any software to bundle with something and then they ship it out the door and if there's an issue, they go go yell at the software vendor.

I think in ai, given everything we just talked about, people are concerned about what's real and what's not real and what they can be trusted. And if Lenovo is going to, you know, lend its credibility to that, then that bodes well, I just hope that, you know, that's what we're gonna get is the output. Because in the past, sometimes, you know, it's been hit or miss with these types of bundling situations and um, I think we need to hold everybody to uh, uh, higher standard here and start looking at these hardware vendors and just say, it's great that you wanted to be part of this and it's awesome that you got infrastructure to go with that and we're definitely gonna need it.

But if you're just bonding stuff together to sell units, we, I promise got the back end of that equation, people are gonna hold them accountable. Yeah, of course. We're seeing that a lot more often.

All the big companies seem to all collaborate and partner together and bundle different services into their tools. Yeah. And a lot of these agreements, we used to call 'em Barney agreements, right?

They're basically, I love you and you love me, and you know, hopefully we'll sell something together. Um, these AI deals need to be better than that. They need to be things where it's been tested, it's been vetted, it will be supported, it will be, uh, some sort of money back guarantee when it doesn't work.

It's gotta really, uh, have some real weight behind it. Mm-Hmm. Well that brings us to the end of all of our articles we're sharing about today.

So what are your final thoughts for our audience? I think it's an exciting time. It's, I think we're on the cusp of entering some sort of news phase here, right?

I would say the first phase was all about, uh, AI startups and the soap opera and the drama that went with that. And you know, I'm sure it will make great made for television series someday, or Netflix or wherever it's gonna be, but I feel like we're getting down to business now and we're starting to see, um, ways to operationalize this stuff with, uh, people who know data. And so I say as always, follow the data.

That's right. It's always the data. So that concludes our show.

I wanna thank our audience for staying tuned and let us know what's your favorite news that we've shared today and what would you like to hear about in the future. And have a great day. See y'all in it.