Operationalizing GenAI with Tray.ai’s Rich Waldron

Hello, and welcome to the latest edition of the Techstrong Do AI video series. I'm your host Mike Bazar. Today we're with Rich Waldron, who is CO for trade, do ai, and we're gonna be talking about what are the challenges that enterprises are facing as they kind of move into generative AI and everything that goes with that.

Rich, welcome to the show. Thanks for having me, Mike. Great to be here.

I think the challenges are, shall we say, manifold, but, um, it seems to me it starts maybe with the data itself, and we don't really have our arms around the data that we're gonna use to train the AI models, but what's your perspective? What are you hearing from customers about the challenges they encounter? Because, well, and, and I think these challenges kind of come back to, well, where do I get started?

Yeah, you're, you're absolutely right. I've, uh, I've spent the last, uh, few weeks on the road meeting with, uh, CIOs all over the United States, and the, the feedback in general has been pretty common. Um, you know, the, the first challenge is how do we basically measure or define ROI for the projects that we wanna release or, or get out?

Um, how do we build them quickly? And lastly, where's the budget gonna come from? Uh, and kind of across the board, I'm, I'm seeing three approaches to that.

The first one being, um, being nimble and agile is, um, uh, a new skill for some organizations when it comes to deploying technology. Um, but it's very much the order of the day. The the middle piece, um, the, the ROI element is, is actually tied to that, which is how can we get something into production fast that allows us to start to recognize the benefit and, and gives some maneuverability for change.

And the last piece is that the budget's actually coming out of the existing stack. So we're seeing consolidation, paying for, um, you know, the, the new experiences that are being developed using ai. A lot of the folks I talked to, there's no shortage of proof of concepts and everybody's excited, but to your point, we don't have infinite amount of money.

So are folks kind of trying to figure out, well, what to prioritize here and what goes into that thinking? Yeah, e exactly that. And the other issue that, that is very commonplace, every application now has some AI capability.

So if you are, you know, running a, a heavy stack, I spoke to a customer last week that has, uh, you know, over 600 applications running in their organization, well, what do you do if every single one has some sort of AI competency? Um, are you refereeing that between all the applications? How do you decide which one you turn on and which one you don't?

And you know, what, what is typically occurring is organizations are trying to figure out where the core tenants of the critical data is. So is that in a data lake? Is that in a HR platform and a CRM and a finance system?

And then it's essentially saying, where can we start to eliminate manual work? Where can we look at something that requires some simple human reasoning or decision where we could start to use AI to, to, um, act on our behalf? That's things like, you know, semantic analysis of leads as they come in to figure out where you route them to.

That's things like, um, extracting data from A PDF on a regular basis, which, you know, would be a job that a team or, or an individual would've done that can now be done by the, the machine and then fed into, you know, various other platforms as a result. So there's, there's definitely a pretty wide base of options. And what organizations are doing is working with their department leaders, working and, and creating these kind of AI groups where they're trying to assess the opportunities and actually almost put an ROI expectation on them, which is typically either is this time that can be spent elsewhere by, um, uh, an existing worker or an individual, or is this something that is gonna directly impact bottom line because we're able to increase output in some, in some form.

Um, and that's pretty much the rubric that I've seen being used by, by many different, uh, organizations. One of the challenges that I hear about is a little more subtle in the sense that that gen AI models are probabilistic and a lot of the workflows are deterministic, and people are trying to insert something that is essentially providing, um, a best guess. And it's something that needs to be right 100% of the time.

And, um, it's not clear to me that everybody who's implementing this stuff kind of gets that little nuance. Yep. Yeah.

And, and that's why the way in which we're working with customers is really about infusing AI into existing processes where the workflow itself has already been determined. And wherever you are implementing ai, it's because it's either providing, um, uh, some reasoning or behavior that is an input to the, to the overall process rather than the, the, um, the full control over what that workflow actually goes and does. Um, because, you know, we are still pretty early in, in, in terms of the speed at which these models are developing.

And almost, um, uh, accidentally you kind of touch on another really valid point, which is the speed of the model change also creates an overhead for an organization whereby, you know, typically the cycle for changing application or changing data could be months or years in many cases. We're now seeing models change in weeks, and the evolution of these models where they're getting, uh, incrementally cheaper and incrementally more intelligent in a short space of time means that, you know, you're constantly deciding, is this the one that's right for us? Do we wanna make a change here?

Um, how does that fit with our overall policy? And so anything that gets created, I think is kind of being built with this mindset of, you know, watch and wait. Let's make sure that we're evaluating how this thing performs before we let it run too far so that we can continue to tune and, and, and adapt the models that that are available to us.

One of the other things that people seem to be trying to figure out is, well, who's gonna do what? And there are little organizational structure here. 'cause I think originally data science teams, but a couple of data engineers tried to do everything, but now we need to do things at scale and feels like there's an effort to separate the inference engines a little bit and give that maybe to the mainstream IT organization which manages that and, um, are so, you know, are people kind of encountering some cultural issues as they go down this path?

Yeah, I, I think alongside that, there's a lot of appetite. You know, the, the every department has been coming to the, to the IT office or the CIO and, and saying, Hey, how do we get our hands on this stuff? You know, we, we feel the pressure to take advantage of the technology.

Um, I think the, the sort of custom role, um, uh, models or, or the, the heavyweight, uh, AI work that might be being done where it's a massive piece of analysis as part of a, a broader data stack is typically being owned by a data and a, a data science team, but the day-to-day application is certainly sitting within it. And that's, you know, like any good organization, you need a governance model. You need control over who get access to what data.

It's very much the same when it comes to, to implementing ai. Uh, what what we've been working on with customers is how do you actually get this stuff into production quickly? Because if you think about it, you know, there's, um, a, a wide plethora of, of AI models that are available.

How do you actually go and implement them in your workflow? Well, your options today are go and build an application, you know, write the code, build the app, uh, scale the servers, build the testing, and kind of you own the life cycle. Um, going forward, that's a heavy investment, requires a lot of skills, um, and requires an ongoing investment in maintenance.

Or do we go and buy new applications for all these new capabilities? Well, we have the same headache that we've had implementing all this enterprise software over the past 25 years. Um, you know, the, the way that I see it, the quickest way to, to get take advantage of this is actually through a, a low code platform.

You know, your, your iPASS is, is the place to go and get, um, your ai uh, uh, workflows implemented because A, you've already got the data there. B, the low-code tooling gives you the architectural scale and governance and control that you require. Uh, and c as long as they play nicely with the various AI vendors you can swap in, swap out models, you can push, uh, uh, data into vector the databases and build rag pipelines and even go as far as fine tuning them.

Uh, but all of that happens without the overhead of needing to build, manage, and scale your own AI engineering outfit, which is, you know, a, a really heavy investment for, for many organizations today. It also seems that as we go forward, a lot of the models are not gonna be so large. There's gonna be a, a broader mix of smaller models that are domain specific that might be easier for us to manage.

Yeah, I think that, um, that's the part that kind of excites me is as we get more specialism within, um, uh, commercial models, you know, imagine being able to pick up a, uh, uh, a verified legal model, uh, off the shelf that can analyze, you know, contracts within a certain state or within a certain restriction that you could take advantage with within your organization. It gives you the first pass of red lines and kind of speeds up that, that legal process framework. Obviously you'll still want a legal professional to do the final sign off, but anything that can give you a gain in in that instance is gonna make a big difference.

And the management of the models becomes a very much like managing, um, a data model or a database within your organization today, right? You, you'll still be going through the same checks and balances, and there's already a lot of tooling which is coming out, which I think helps with that. So I think there's, you know, it, it's, it's very expensive to, uh, create, maintain and, and manage your own model.

And I think the more that we start to see kind of the specialism within these models and their availability for quick deployment, um, the more exciting it it's gonna be, they'll obviously be the, the generic use cases, the semantic analysis, the um, uh, text and, and and extraction through, um, you know, uh, utilizing vision and, and some of those things in between, you know, those are things that are applicable today and, and, and can be stood up by, by organizations. And that the part that has been most, I guess, impressive to me is the speed of adoption, you know, across our customer base. Some of the largest oldest customers that, that we work with, uh, already have, um, AI workflows into production.

Um, I, I met with a large manufacturing firm that was 150 years old last week, um, that is already, you know, doing analysis of their manufacturing processes and figuring out, you know, where they could speed up or, or catch issues before they arise because the machines already have all the data, it's all been available now it feels like we can do something really useful with it. What is the end user experience gonna be like? And you mentioned low code.

Yeah. Um, but am I gonna just kind of invoke a low code engine through a prompt that's gonna go create something for me or execute something and, um, will therefore be a lot of asynchronous things that I'm trying to stitch together? And maybe they're all run by many AI agents, I'm not sure, but how do you see this all playing out?

I, I think this is gonna be where companies begin to, um, separate and gain their own competitive advantage. Um, I think there's a lot of organizations that have tried the kind of multi-agent model for lots of different, um, uh, departments. One of the challenges with that is it a requires your end user to be particularly good at prompt engineering.

And secondarily, um, you know, you have to remember to go and do it. Where, where I actually think the most interesting applications of AI are gonna come from are where they kind of sit within, uh, business processes that you already have up and running. You know, it's where data gets ingested into something that you're already doing and it feels like a natural part of your workflow.

An example being, uh, opening a support ticket in your support ticket platform. And the response is pre-written because it's gone and done the analysis of every other ticket that's been written. It's looked at your knowledge base, it's picked up data from elsewhere, and it's kind of gone and done that first pass of work for you so that you are going in, in the, in the usual way that you would, but actually the end result's already there for you to review and send.

And I think the more experiences that we can, um, foster and create where, um, you know, it, it it's kind of happening along the way or it's, it's prompting you and saying, Hey, here's something interesting that you should consider because you're gonna go and do this. That to me is where we're gonna see, um, huge gains. I think natural language has been the thing that's been very exciting for a lot of people, but I don't necessarily think that's gonna be the, the most common, uh, application.

I think the, the, the, the best sort of software vendors or, or the best, um, applications of AI within organizations are, are gonna feel intuitive and native and aren't gonna require necessarily a back and forth in, in every instance. When you talk to customers, among those you see doing it well, what are they doing differently than others who are kind of just, for lack of a better phrase, kind of stumbling their way through? Yeah.

The ones that are doing it well very quickly got a, um, internal group set up where they had stakeholders from each major department. It was very clear who the owner was from a, uh, technology perspective to help on the deployment side. Uh, and then they, it, it truly became a partnership where they could effectively, quickly validate and determine, hey, here is something that is directly applicable.

We understand the ROI that we expect to get from it, uh, and we can move quickly to, to get this into production. They aren't, uh, they aren't doing kind of long pieces of analysis to decide to do, you know, very long projects that require, um, extensive deployment and, and, and, and measurement beyond, because they recognize the ways in which they're trying to get this implemented. And the feedback loop needs to be very quick.

And so there's, there's definitely a, a difference in agility between these organizations. And the interesting thing for me is they're industry agnostic. You know, the organization that I just mentioned, which is a, a, a, you know, 150 year old, um, manufacturing, uh, organization from, from North Carolina in many ways, this is one of the organizations you think would, would maybe be a laggard based on their traditional approach and, and the way in which they got their stacks stood up.

But actually they're, they're ahead of the curve. And one of the reasons is they already had a data lake set up. They already had the data in the right place, so it suddenly became really quick for them to be able to go and, um, uh, recognize and gain.

And once they could see that it was valuable, then they were able to make the commitment. Uh, there are certainly some organizations that are in a watch and wait phase, uh, a lot of analysis, um, trying to kind of find their way between, um, uh, the, the right tooling or the right application to stand up. And it's also quite overwhelming where the, where the volume of requests that are coming in from each department saying, Hey, my finance tool now has ai, can I use this?

My, uh, CRM now has ai, can I use this? And I think, um, you know, that that's, that's causing some slowness in, in overall execution. Certainly the ones that are, that are moving the quickest very quickly kind of came up with a, a, a simple process, which is, we're not gonna turn anything on straight away.

We're gonna decide a handful of projects we're gonna commit to, then we're gonna get fast ROI, and that's gonna begin to build the, um, methodology that we're gonna use for, for continuing this rollout across the rest of the organization. Do you think ultimately, as we kind of make this transition, AI, that it will drive more organizations to rationalize their tech stacks because, uh, they finally have a reason to kind of bring all their data together and in so doing that will force some decisions that have been, shall we say, something that no one really wanted to make just for the last decade or so? A hundred percent.

I think, um, uh, you know, if I had a dollar for the, for the amount of times I've had consolidation, um, over the past couple of months, um, you know, I'd be a, I'd be a very wealthy man because the, the approach is twofold, right? One is the economy's driving more efficiency, you know, naturally. Um, uh, you know, as, as we're all aware, and secondarily for us to take advantage of these, um, of this AI capability, many CFOs are turning around and saying, well, you're already spending a lot of money on software, so do you need all the products that sit in between the core applications that you have?

Is there a way that you could actually sort of rinse and reuse some of the capital that that is being invested there? So there is a, a great deal of effort going into kind of bringing down and, and, and as you say, rationalizing the tech stack. I think the other thing that it's doing is it's forcing everybody in each, uh, uh, software industry to kind of reconsider the future.

You know, in, in our space in, in iPASS, um, for a long time it was, uh, uh, on-premise to, on-premise, on-premise to cloud, cloud to cloud. Now the future is, is gonna, certainly gonna involve AI and it's certainly gonna involve the deployment of ai. And, you know, if you think that you don't, you can't change what your, your operating model or your roadmap based on what your customer demand's gonna be in the future, you're, you're gonna get left behind.

And so I feel like there's a bit of a reckoning coming for, for many industries where, um, how they've architected what their approach is to how they're gonna support this new technology. And even trying to think through what the workflow is gonna look like in the future requires a lot of, uh, companies to change very quickly. And I think when we look at some of the biggest organizations that Microsoft's of the world, you know, the speed at which they've made their bet and they've placed what they're gonna go and do, and everything that they've put out is, is centered around ai, is, is astonishing, um, comparative to the usual adoption cycle that we see in software.

All right, folks, well, you heard it here. AI's gonna have all kinds of interesting cascading impacts across the organization. And the time to start thinking about that is now versus later when it might be a little bit too late to be proactive about it.

Hey, rich, thanks for being on the show. Thank you very much. All right.

Thank you all for watching the latest episode of the Techstrong AI video series. You can find this episode and others on our website. We invite you to check them all out.

Until then, we'll see you next time.