The Evolving Landscape of AI Projects with Ensono’s Brian Klingbeil

Hello, and welcome to the latest edition of the Techstrong AI Video series Army host Mike Vara. Today we're with Brian Klingle, who is Chief Strategy Officer for Inno. And we're talking about, well, where do you put your proverbial AI eggs in the basket?

Because figuring out which one of these projects has the value that you expect is maybe getting a little harder than we thought. Brian, welcome to the show. Thanks.

Good to be here. Mike. We've seen everybody kind of get really enthusiastic about all things related to generative ai, especially at the end of last year.

And now this year we're kind of looking at projects for 2025. What have we learned in the, in the ensuing year about, well, what's real, what's not real, and what should we be thinking about here in terms of where's the ROI? Yeah, we've learned quite a bit about it.

And, uh, I actually, part of my job run a client advisory board for Inno and Sono happens to, we, we, we serve very big companies, so I'm sort of lucky that I get to see across a broad spectrum of some of the largest enterprises in the world. And part of that advisory board we meet a couple times a year, once in person and then once virtually. But we had lucky, luckily enough, the timing was such that our in-person meeting this year was right around March 1st, kind of just as that, call it a hype cycle.

It was kind of, uh, at a precipice there. So we arranged the day to, we talk about in Sonos stuff, but we also try to bring 50% just stuff that would help these CIOs who are the attendees, uh, help them do their job and, and kind of collaborate with each other. So half of the day was them talking to each other about the AI use cases that they were bringing to bear within their own, uh, institutions.

And we learned something right away, which was that there was a really common theme, everybody with one notable exception that I, I can't call them out by name, but they were doing really cool stuff, uh, you know, buying H 100 chips from Nvidia and kind of doing stuff around radiology in the, uh, medical field. But everybody else, without exception was doing proof of concepts. And their proof of concepts looked a lot like, even though they're my customers, I'm a service provider, their POCs looked a lot like my POCs, and they're the things that you would expect.

Uh, um, things like, uh, copilot, uh, from Microsoft obviously was a common use case. Um, chat bots, uh, logistics, um, back office things, uh, not a lot of, um, not, not a lot of that deep kind of build my own LLM and that kind of, none of that really, uh, really most people thinking, well, we'll, we'll, we'll be consumers of AI through maybe SaaS, uh, offerings that we're consuming from people like Salesforce or ServiceNow or things, or, uh, uh, a good example is DocuSign, which is one that we've adopted. So we, we looked at those, um, EOCs and said, okay, well, let's meet again in the summer when, which is our virtual meeting, and kind of see how they're all going.

See how many Cs have been peed, how many concepts have been proven, and what are the ROIs as, as you very rightly point out. And we just had that meeting a few weeks ago. And, uh, I'll say the, the results have been spotty.

Um, and, and I'm by no means an AI naysayer, it's just early, and I think people are still feeling their way around, okay, where do we wanna spend money in order to get a return out of it? How much of this is a technical issue versus just a cultural issue. I think we assumed that we were gonna hand everybody these prompts and they were gonna go out and become more productive.

But a lot of folks I talked to, they're not quite clear what to do with all this stuff other than the fact that they might write a better email. But, um, you know, it is not like it's driving leaps and bounds of improvement in productivity that shows up in a bottom line just yet. No, that's right.

Uh, and, and again, I wanna re reiterate it's, it's going to, but I'll tell you the, um, this is a group of about 15, you know, really high end CIOs, and I asked them a series of questions, and not that this is the most relevant statistical sample of all time, but it's statistically relevant. These are 15 very big companies. And I asked him a few questions.

The first one was, okay for the, for the proof of concepts for the anything really that you rolled out. Um, and I didn't, by the way, specify AI versus generative ai, which is an important distinction. Uh, we can talk a little bit about machine learning, which I think is a much more proven, uh, element of ai.

If you think about general AI as a big bubble, and within that you have deep learning, and within that you have machine learning. And within that you have LLMs that machine learning is a pretty tried and true method and, and, and with good ROIs, and we're experiencing that ourselves. Uh, but when I asked them around generative ai, what are the financial impacts from early stage AI projects?

And gave them kind of four choices, uh, one is I've been able to eliminate or really not backfill some jobs at a small impact I've been able to eliminate or not backfill some jobs medium and then high impact. And the fourth category was BCUs, like literally phrased bupkis. Uh, the vast majority were in bku and a few people in the, in the minimum impact.

Uh, the, the next question was around, okay, what about nonhuman expenses like software, hardware, you know, any sort of third party vendor, I've been able to eliminate, you know, what kind of cost impacts from that. Uh, and this time I had those same choices, small, medium, large impact, uh, and bka, but also, um, no, it's costing me more. And I had sort of 20% say at the moment, it's costing me more, uh, 20% say I've had some impact, and the rest were kind of in the, in the bki category.

But when I asked non-financial impacts, you know, things like a more competitive offer, a faster innovation or risk mitigation or customer satisfaction, or better and faster decision making, pretty pretty good responses to those things. So your your your point about ROI, some of it is gonna be tough to quantify. Um, and I, but I think the CIOs are going, yeah, I, I definitely see benefits.

Um, but some of them are a little bit hard to do the math around And those intangible benefits though, I have to wonder if we're getting to a point where a lot of folks aren't gonna wanna go necessarily work for an organization if they don't give them AI tools. I mean, it's gonna be just the cost of doing business. 'cause I mean, whether I'm a security analyst or a marketing professional, I don't really want to do all this scut work if I don't have to.

So have we reached a point where, um, it's gonna be standard issue equipment? I think we're getting there really fast, and you sound just like my CIO he said the exact same thing sort of nine months ago, is I gotta add, you know, this might just become a cost of doing business. And there's that, that, that very good soundbite that's out there is like, look at the moment, an AI is not gonna take your job, but somebody who's good at using AI may take your job.

So we, we wanna arm people with the tools that they need. I don't know if it's useful, but like here at Inno, we have kind of four or five use cases where we're leveraging AI and they're kind of in the realm of what you're talking about. Like, uh, in, in terms of, um, taking away maybe some of the grunt work from a variety of jobs.

Uh, the first is, um, you know, the Microsoft co-pilot and other tools like that is, um, there's just too much information in the world right now for humans to cope with. So the notion of summarization is a key generative AI feature. I think that adds value.

You know, whether it's, you know, emails or summarizing a meeting, I don't know if you're a user of, of Microsoft copilot that's built into teams, really is useful. If you're late to a meeting, which I'm late to a lot of meetings, sort of join 10 minutes in and you can hit sort of summarize the meeting so far. That's really useful.

Um, program managers and project managers when they're, the whole world's become virtual if you do an hour long meeting, a lot of those project managers would, a big part of their job was summarizing the meeting and sending out action items. And kind of the co-pilot gets you 75% of the way there. So it makes their jobs a lot more.

They can have a lot more impact when some of the grunt work gets taken away. We've also, here at Inno, one of the projects we put some money in, it's not a lot of money, but it's pretty impactful. And that's, uh, sort of an automated RFP response tool, uh, R-F-Q-R-F-P resp, uh, request for proposal, request for a quote.

And the, the AI tool can kind of get us 50% of the way there. And then the, the sales team, the client facing organization can obviously put their own color on it knowing things about the relationship or things about the, the client, uh, challenge. So, uh, yeah, I agree that, um, that's gonna be a big part of it.

It might become cost of doing business that'll be embedded in, uh, certain areas that lift away some other grunt work. Oh, another example is, um, we, I'm not here to, uh, to advertise for, um, for, uh, DocuSign, but they have a really cool tool built into their contracts, uh, database tool that allows us to do things like lift out the metadata, like for us, uh, being able to feed all our contracts in and then ask the AI questions like, oh, are there, are there certain, um, SLAs that are out of, out of sync with our normal? Are there payment terms that are outta sync with our normal, this contract's coming up for renegotiation?

Would we wanna renegotiate certain terms quite useful and not very expensive? So the ROI ON that's really easy to calculate. Mm-Hmm.

How much of this is, we're just having trouble parsing the word is because, um, it seems to me the LLMs are getting smarter, their reasoning engines will get better. And so many of the things that we were, uh, you know, saying were so awesome last year might actually manifest themselves next year. And we're kind of on this journey, but, uh, maybe we just overhyped the capabilities in the short term.

I, I think that's very true. com crash. Uh, there's a big difference between what's going on here and then what went on back in the early two thousands when too many data centers got built and too much money got spent on Cisco equipment back then, the money was being spent by people without real business cases and who were buying things with debt.

And now a lot of that money's being spent with people with business uses and business cases, and who had piles of cash, you know, the Apples and the Microsofts and the Googles and the Amazons of the world that they, they all had money. This isn't like a leverage situation, but what, um, I think Bill Gates GRS credit for this quote, but I think somebody said it beforehand, which is we always overhype a new technology in the first two years and underappreciate the impact over the next 10 years of really impactful technologies. I think that might be going on here.

And your point about LLMs is a really good one. Uh, I think what we're seeing is s SLMs or small language models are those large LLMs that are then, um, uh, used to create, uh, very specific business use LLMs. Uh, I think we're gonna see a lot of that here.

We've been able, and I actually was surprised a little bit by my client advisory board that there was still some concern about if I take an LLM and I train it with my company's very specific data, which is very, uh, much more useful use case than a very broad use case. 'cause the, the LLMs have been trained on information scraped from the internet. And, but you you may have read about like, well, there we need more data.

Like how couldn't there be more data than the whole internet? There's way more data than the whole internet because most of data is not on the public internet, like the, all the supply chain information from every order that's ever gone through Pfizer, it's not on the internet. So if they can use an LLM teach it to speak English and, and, and, uh, communicate with people via the LLM, but then apply it to my very specific company information, then that becomes very, very useful.

And there's a, there's a phrase out there, retrieval augmented generation or rag, where you take the LOM and apply it to your specific case. And I, I was surprised that they, that people didn't understand that that's pretty easy to do and you can protect, you're not training the LOM or teaching the world about your product or anything like that. You can protect your data and those models.

I think, uh, micro models or, uh, niche use case LLMs are gonna be very impactful in my opinion. Well, hopefully our data's not on the, uh, dark web somewhere that somebody's gonna like pull in. But, uh, who knows?

There's always that worry. Um, I, are people starting to experience a certain amount of sticker shock when it comes to ai? And is that gonna force them to narrow the number of projects that they're willing to fund?

Um, I think we had this notion of let's let a thousand flowers bloom, but I think heading into 2025, maybe people are pulling in their horns and saying, let's, uh, zone in on the two or three things here that are gonna make a difference. You've described my company situation precisely Mm-Hmm. Uh, and I don't think it was ever gonna get there.

Like, look, the, you know, the certainly the last couple years has been, uh, a period of uncertainty for the global economy and the US economy. Like, what's gonna happen? Is the Fed gonna drop interest rates?

Is it gonna be a soft landing? Are we gonna go under a recession? And, uh, again, because of the type of clients that I serve, I get to see it.

'cause these are, these are really large enterprises, big cross section across verticals, and people have been being careful with money for a good 24 months. So we never saw on the, the day to day. And like I'm talking about state, local governments and insurance firms and banks and, um, medical, uh, um, biomed, uh, companies, these are the type of clients that I serve.

Um, you know, there was never like a Willy-nilly run to spend a lot of money on ai. 'cause they have budgets and they have shareholders, or they have, they have private equity that owns them and things like this. So they, they were never gonna kind of just dole money out.

Now certain institutions have, and venture capital, you know, is funding, uh, companies that are developing LLMs and, and, and new tools and things like that. But when it comes to like the tried and true a hundred year old institutions that, that I tend to serve, like they don't have money to splash out on things like that. So it was always gonna be a little bit more programmatic.

And everybody's asking the question that you're asking, which is what's the ROI because it, it can't get funded otherwise. There is a, another small cultural aspect to all of this, but we're so used to AI and all things, it being in the deterministic kind of set, and our business workflows are deterministic. And we're using an AI tool with regenerative AI that's probabilistic.

So that means, you know, not everything is always gonna be, uh, precise. And do we have to, uh, have a better understanding of how to manage that? Because, um, I think we get it in our heads that somehow or other this is gonna be super automation, when in reality it's gets us maybe 70% of the way there.

But somebody still needs to supervise this though. Yeah, I think that's generally accepted, but humans are what they are. It's a really good point.

Uh, I'm, I'm coming to you off of, uh, I'll confess, prior to this call, I was having an early lunch and having a very unsatisfying discussion with the chatbot about some concert tickets. And, uh, it was just absolutely terrible. Uh, they, they need to work on that.

I won't call out the company by name. Mm-Hmm. Um, but, you know, humans will adopt tools really quickly.

And when it comes to maybe overtrusting the result that you get, um, and I'm a little guilty of it. I use chat GPT in my daily life. I got a personal subscription and I'll ask at things, and sometimes I'll have to stop myself from taking the answer at face value 'cause it is probabilistic.

But the answer is deli delivered very confidently. Right? So, which is why, um, we're, we're building, we're using this sort of rag methodology and a, you know, pointing an LLM toward our own, um, set of data so that, you know, if you're a new, if you're a new employee at Inno, you get asked a question.

It's like, what does this product do that Inno offers? Or do we have a security offering? Or who's in head of this, who, who's in charge of, who's our chief uh, strategy officer for the company?

Things like that. But what it does is, uh, we'll also give you sort of hyperlinks to annotate the, the source so you can sort of inspect it deeper and not take the answer at face value. But, but I think, I think there's a real danger of what you're describing is like, humans love adopting tools that make their life, you know, easier, quicker, uh, and less effort.

And, um, I don't think a lot of people know how, how probabilistic the, the tools are, because the answer, the answer is tend to be, and the chat bot side of the equation, the answers tend to be delivered with such authority. So what's your best advice to folks as they kind of think through how to integrate that kind of capability into their business process? Yeah, I'd say tying a couple things you said together, um, who, number one, don't do nothing.

And, and I don't mean to come across as a naysayer 'cause I think it's an amazing technology, um, but as you said, it might become table stakes. So keeping up is critical. Um, we, we got out ahead of it and created kind of a collisional space early on for people to collaborate and provide ideas.

And then we parsed those ideas, kind of met as a group, had a, you know, a kind of a virtual team, cross-functional, virtual virtual team to see, okay, where is this gonna have the most impact on the organization? And then, yeah, again, a little funding, you know, we're, you're gonna need to try some stuff. Doing nothing is not a good option at all.

Um, splashing out and buying a bunch of, you know, Nvidia chips, probably maybe not, but you know, find where your company will be able to take the most advantage of it and always start with the business problem. You know, uh, you know, don't go around, uh, with a hammer looking for a nail. You know, make sure you understand what you're trying to accomplish and you know, you got, you got smart people in your company that can sort of say, oh, that's something that gen AI could prob or ai, uh, could probably solve.

Um, but start with the business problem. Not, not, oh look, chatbots are cool, let's find a use for that. Or, um, you know, predictive analytics is cool.

Let's find a use case for that. Uh, we did that here, we're working on something called predictive engine, and this is more machine learning with a little taste of generative ai. But the idea being we're an IT services provider, we monitor IT infrastructure and applications and look, you know, try to optimize environments and things like that.

So the most important thing for us would be being able to predict outages before they occur. Or not outages, but even just small events up, up to outages. So the impact on our clients would be tremendous.

You know, starting from, like they, they never have an outage or they have much, they have much less downtime than they've ever had before or, or really it's more about busy work outage just happen, rarely touch wood in our organization, but small little disruptions, you know, occur kind of all the time and, and IT infrastructure. So for us, being able to cut that down dramatically increases our client satisfaction dramatically, and at the same time decreases our costs because we have less time like dealing with issues on the backend. So we started there and said, that's the most impactful thing we can do.

Now, can AI contribute to that? And the answer has been, yes, it can. Uh, again, more through machine learning and looking at, we, you know, we get a million of monitoring pieces of data every hour, uh, and being able to correlate that with the events of the past, it's pretty straightforward.

Machine learning, well, I'm not doing the work, so I say straightforward, but the, the concept is straightforward. And then use generative AI to create tickets that then go to text that say, Hey, something might be about to happen here instead of sending a ticket to a tech that says something just happened here. Uh, so we look for a business problem that we thought conceptually could be solved with AI and then went to work on that project.

So I'd recommend don't do nothing, do something, and start from the business problem, not from the, oh, this is a cool tool, let's find an application for it. All right, cool. Hey, we all know somebody in our lives who tells us things with great confidence that we have learned through experience to make sure we have a second opinion for, well, that's called Gen AI now.

So there you haven't. Hey, I Thought you were talking about me. Hey Brian, thanks for being on the show.

Thanks very much. Good talking to you. All right, and thank you all for watching the latest episode of the Techstrong AI video series.

You can find this episode and others on our website. By all means, go check them out. Until then, we'll see you next time.