Move from the AI Pilot Circle of Doom to Achieving Outcomes Today with Kamiwaza Enterprise Outcome Support

So, uh, we're gonna go into a couple of the actual demos here. Okay. Let, let's just start with a quick tour, right?

And not everything here you're gonna see, like that pays a little bit differently in clustered mode for not running. But, um, the first thing I kind of wanna show you really actually is, um, I'd like to kind of say like docs or, uh, or, or APIs, our first class citizen. So like, as I kind of drive through the rest, you know, the, sorry, the react run end.

Just like note that this is like completely stateless run end, right? It does not know or remember or do anything on maybe like a tiny bit in your session, like a cookie or something, right? Um, we bundle in some developer tools, like developers are first class citizens.

It's a little bit weird for like an enterprise product, right? But we, you know, we, we manage a, a, you know, deploy Jupyter lab environments, the upcoming release. These are gonna be like per individual.

Um, and uh, we have cluster management, right? You have view hardware, right? We kind of understand what's running under the hood.

You can see this one here, but shockingly on local hosts, this is my Mac, this is little the box I'm presenting to you from and on this zoom from, right? So speed feed system great for running big models. But you know, if I come over here, um, to a system that we have running up in a cloud, right?

We're deployed in on one of the cloud providers. And I come in and I look at the same thing. I mean, you're seeing the same interface, but if we come into the cluster and we dig in, right?

You see Nvidia, this is running on a single a 100 card. And then if I come over here, right? And this is another system that we have online that we're kind of working on, um, with a partner, I go in and look at this cluster, you'll see a pile of a b instincts, right?

In a cluster. This is beefy. Yes, I do like to play with it.

Um, so that's kind of a quick, quick overview, right? And, and, um, from a cluster management perspective, like back here and you actually couldn't even tell I'm doing practically 'cause they look so similar. Um, of course you actually get to deploy models, right?

So here I have, um, it's my Mac and this is reasonably fast. I have this Quinn coder 32 B instruct, right? And, um, we kind of have platform specific things you can click on.

Like if I click on this, this is the native like Lama CVP interface. You don't have to use this 'cause it has an open AI compatible interface, but you can jump right into it. Now what's cool is we're kind of managing things like, um, load balancing in here.

And actually I'll really show you this through the rest API 'cause I actually don't expose this through the interface right now. So what am I gonna do? Um, we'll go look at the cluster runtime config, right?

And if I pull this through the rest interface, you'll notice down here there's much stuff about, um, load balancing the model port, right? We're getting really geeky here. I know.

Um, but we're actually kind of instant these endpoints dynamically and across the cluster we actually run load balancers, like which, if you're more want to use Kubernetes type terms and it's not running on Kubernetes, but I think it's a good analogy for people who are used to it, right? This is the ingress gateway. And on that we can do SSL termination and um, load balancing.

And then under the hood, you know, comas is built on top of Ray, which is the distributed computing framework that any scale helps maintain, um, open source and um, that actually distributes amongst the cluster as well. And it also underpins the, the way that we build the, um, inference mesh between sites. So, so When you talk about, when you talk about load balancing and um, and scaling, um, you're talking about like on a a, a query is still running only on a single machine, right?

You're not actually, um, splitting it up or are you splitting up all the individual? Um, Query is a, is an ambiguous Term. Yeah.

Well I'm trying to say like, like, so I'm a user using this. You're, is it, is it more like a web application load balancer where each user is being directed to a different machine or is it actually running, you know, different inferencing operations across different machines in the backend? Great Question.

Notice here, I got the scaling and I'm obviously don't really wanna autoscale on a Mac, right? What I'm showing you on. But, um, we'll autoscale, so this one has a single instance and that's normally, that's the default, like starting copy.

Um, there is a threshold, um, at which when there's enough requests that due to a model, we'll start, if you have hardware, we'll start turning up digital instances. You'll notice that we have an idea of what the GPU allocation is like. That's kind of regardless of what platform we're on.

Like if I come over here and look at the, I just switched over, kind of have to narrate this. But if I switch over to the deployment that's on the, the cloud box, right? Look the same thing, totally different engine.

'cause it's not LAMA CPP, it's VLLM, but you'll need assist model and it has a very large context and it's a bigger model taking up almost the entire A 100. But we have awareness of these things. Um, and so we can turn up multiple copies of models, we can spread them across cards and then we can spread 'em across clusters.

And actually that is where, uh, it's interesting. So I said load balancing and you saw that load balancer, it is important, but the load balancing to the model instance actually happens at the ray layer. So the um, the outer load balancer, the ingress does things like JWT enforcement to make sure authenticated an SSL termination.

It rewrites URLs because honestly Ray serves like internal URLs for serve deployments are messy and not very developer friendly. So we, we do that on the outer layer and then we distribute to the model instances with the inner ray layer as a lot. So, so it is functioning sort of like a traditional web application load?

Yeah, absolutely. Web and app set. Yeah.

Okay. Gotcha. Yeah.

Yeah. Um, and, and it actually, I mean as we get into it, like at the app layer, one of the amazing about integrating with ka waza is, um, and we'll expose this more coming up, is that not only do you have, these are authentication, but we're going to allow people, because frankly these systems tend to be underutilized for things that are not the GPU. And so allowing somebody to deploy like an application container that then we can actually talk under our ingress filter in an isolated environment to let it take advantage of all of our, you know, things means you don't even necessarily have to integrate app with your, your, um, off with your app.

You can integrate it with ka waza and we'll kind of take care of the ingress layer. So it's something we're doing, I don't know that makes sense for, you know, mission critical apps, but it probably makes a lot of sense for what we, what we actually see, which is that there's kind of a lot of variety in J apps, if that makes sense. Um, I was gonna come over here and run this.

Um, this is like just a random, I ha I, so from another thing, we'll actually show you one of these agents, but I'm gonna run this notebook. I just wanna see like the catalog ingesting data, right? It's like pretty quick and I'm gonna jump over to this weather data and I'll rerun this, right?

And this is running locally, so it's maybe not the, it would be probably faster on a lot of systems. Um, 'cause it's truly running right here. But you'll notice it knows where I am.

It's now reading the per k files, this 33,000 more K files. 3 billion rows, right? And you, what you'll see here is it's pretty easy for us to get the dataset out of the commonwealth catalog by looking for it, right?

We're looking for it by the name surface data. We're saying it's a high partition set of parquet and we get this scheme out of it, right? And we're kind of processing that.

Now, something you'll notice here, these little task bars, and I, I could do this with something else that might be even more illustrative. Like this is not this one. Sorry.

Yeah, That was a little quick. It didn't look like it was doing anything, but I imagine that that task bar is gonna fill up. No, no, it's done.

It was already done. Yeah. Um, yeah, and, and part of that's because of course we're not, we're not doing things with the data, we're ingesting metadata.

Now that one, obviously the par k has metadata that's kind of global, so it's not the most challenging thing. But this is actually interesting. This is a real like ingestion.

And actually this could take a metadata. Actually this is one of the ones that actually ships with the code. I feel like I might have showed this a year ago.

We're doing something really, really similar, even in the very first version of the codes. Maybe a little more interesting or tighter now. But if I get down, I want to make sure I'm down where the bars are gonna appear, right?

We'll have some progress bars. Um, sorry, I'm just scroll out. Here they are.

So you actually see this work in process. So we've got 1900 different, um, documents that are being processed, right? And I can hear my fan spinning up.

You can't probably directional mic. Thank you. But what we're doing here is we're running this through a tiny cleaning function that's user defined, that's passing it into comi W's sentence transformer middleware, which is then passing it into the, the middleware that does vector inserts, right?

What you'll notice is that because we're built on top of Ray, this can turn into this flow, right? These are actually workers running in parallel. It's using something like 10 workers or something right now for each of these processes, right?

You can kinda see that it can do many iterations. These are actually working in parallel. Now, if I had a really big data set that would say, you know, petabyte sized and I wanted to do this at massive scale, um, it can, it will automatically, I even can't, it will automatically distribute it across the cluster.

Every node will turn off workers, they'll all be retrieving data and it's lazy loading. Even if you pointed at a petabyte of data, it's pulling it. These things are only kind of processing the downstream parts as they go.

Right? Here's all our inserts. We have searchable vectors.

I mean these, by the way, being the intro notebooks, they're just loaded with comments, right? Um, and I, and I have an error. That's fine.

I don't know what that's doing. Um, It just shows that it's Real. Yeah, Welcome to live demos To totally real, Matt, In the interest of time, do you wanna move?

I do and I do know what it is actually. Okay, Matt, this is what you, this is the developer edition. Let, Let's show off some of the, uh, agents on This flow.

Yeah, let's do, um, well let me, let me jump real quick into like a slightly, um, oh yeah. So, um, this is for you John. Um, so one of the things that we, um, one of, one of the things that we do is make it really easy to deploy on top, right?

And, and the point I wanna make here really is just that this is, um, we're actually actively doing a bunch of ports of things, right? This is an app that's gonna be in our app garden. It's open source.

This is a com fork of Versal AI chat bot, right? Great interface. But you can see what we've done is we've integrated with our authentication right as I'm logged in and it's integrated with our model, right?

This is the same one we had deployed. Can I, can I pick the right tab? You see my Quin 72, you know, on 4 5 1 1 2.

So what the app does is it, it's programmed to hit the commonwealth, the API to say hit what models are out there, right? And I only have one, so you can only pick one, but you, you're getting the model obviously here that it defaults to, right? But that, I know we're running outta time, but like I, I either, I'm either stupid or I know too much and I'm not, I'm having a hard time understanding exactly what you do.

I, yeah. And, and, and could I summarize what I think now I'm hearing? 'cause you're, you're clearly a rag aggregator, right?

So the sort of locality all that we, I think we established that. I'm trying to understand. I say I'd say genai platform, but, alright.

Yeah, but I mean that's, I mean, when you think about a detailed stack of all the things that you have to do, but I guess what you're telling me is you are giving me a tool to build rags and you're using like Lambda, MapReduce and LA and using parallelism and making it easier for me to build a rag and make me easier to aggregate rag for inference. Is that, am I getting that reasonably right? Um, you're right about the part where in, in so far as it's rag, but I mean, keep in mind that like what we kind of show you when I show you this nifty little cluster, we kind of are aware where all the resources are, we know ahead of time, right?

The catalog think that's brilliant. I I I like that. But, but like I can build a lot of loading and I do build a lot of loading.

To me, I'm seeing the advantages that maybe you're making the loading of a rag much more efficient for me because you're using a lot of your, um, data science experience from Yeah. Okay. Alright then, then My, and my, and my perception by the way is that like we have a huge advantage when you're kind of new to it because the, the, this is that opinionated middleware because we stitch things together really well.

And so you get a bunch of best practices like, um, you know that demo notebook where we, um, pulled things in. I could probably run on a process, I bet it would work fine. Um, because I, I don't think it has the broken library, but um, when you actually see the results from the vector database, right, we're actually scoring the offset of the document, um, in the metadata for the vector collection.

And then when you call our retriever to go do the rag retrieval, for example, right? If it's a 500 page, you know, document it was a PDF conversion of your, you know, monster SEC you know, filing for example, we don't go to retrieve the whole doc. We, whether it's file or object, we're retrieving the specific byte ranges that you need.

It's actually a really efficient retrieval, right? So we'll know exactly what data to pull and only pull that. Now of course you could build that, but will you build that the first time?

No, I don't, I don't think so. And so I think think there's lot of alright. No, I just wanted To get, I wasn't getting clarity in my mind and yeah, and the, the, the asking too many questions, but yeah, This will be interesting too and show a little bit more about like the integration.

We'll say like, so this is Another app. So we showed you the versal port, this is another app Okay. That's supported right into our API Go ahead Matt.

Yeah, totally separate. Okay. So I, I'm asking this guy, um, what's our, what's our break policy, right?

He goes, I don't know. And part of that is 'cause his system prompt us. Like if you don't have, if somebody's asking something it seems specific to the company or whatever, or the situation and you don't know, then don't answer, right?

So it's like, okay, so if I come in here and I, um, some goodies in here like, um, company docs. Now again, it's local, so this is happening local. Um, so now this is like a, uh, this is similar ingestion, but we're actually like, um, we're actually putting, you know, this is a, this is an app that we're open sourcing by the way.

Um, but with this you can see like a prettier interface for all these things. How much data I'm Doing investing here. How much, What was that?

How Much data is it actually ingesting here in this run? 18 documents. Yeah, But how big are those documents?

Uh, they're like markdown files, like a K, so not very much at all. Okay. A lot of that is literally like just kind of flowing through the process.

I'm just trying to get an understanding of how long it takes. Okay, go ahead. Oh yeah, I mean, and honestly, I mean I, I wouldn't even say this is like optimized.

Like if you're gonna do this in production, there's you, I don't, I don't even think this particular app is wrapping in the like, um, the pipeline construct that I showed you. But now if I come back and I, uh, come in, Alex, actually I need to go back and I think configure him. 'cause now he'll know the pet store docs are there.

I will say, and you notice over on the left, like it know we've got the company docs, right? So now when I ask what our break policy is, you know, it's a little slower to respond because there's like a tiny rag process going on back there to pull the data in. And so now we get an actual answer right, where he would refuse to answer before, if that makes sense.

Could you ask him what the, uh, capital of Nebraska is? He'll probably answer by the way. Um, I will, because That's actually one of my, one of my trick questions for all rag demonstrations.

I personally use Kansas, but uh, as a Nebraska, you know, And of course I I I expect he's gonna try to answer at least 'cause my instructions were in the prompt. Were not. Um, uh, it's funny, it's actually, it, it, it actually is trying to retrieve it.

That is definitely not in the rag, but I saw it up. I didn't see it, it, it still tried to retrieve it was, that's like here's the most relevant chunks, it's not in there. Okay.

Once again, this would be any third party app that simply plugs right into our API and our catalog service. Yeah. So Luke, you mentioned earlier on that, uh, effectively, uh, offering with your solution and outcomes support per cluster per month.

So, so anybody can ask of your professional services. I would like to do, you know, convert, uh, database a into, you know, a newer format or something like that and you guys would go off and and create an agent workflow to do that. Is that That's what we would work with them on?

That's correct. Yeah. Yeah.

Using or Not, in fact, I kid you not a lot of people are like, move from this cloud service file, cloud service to this cloud file service. But we want the same mapping, we want the same security applied, we want the same folder structure. And that was actually a use case that just came up.

I kind getting A lot of use. Keep going Matt. Yep.

Sorry, I wanna pack more demos and so I'm gonna actually, um, this is running, sorry. Um, not, there's another app that we have for this is open our repo, although I wouldn't call this like polish, but I really enjoy this. This is really powerful to one, I think it goes to show like how easy it is as an enterprise potentially to deploy things that are like really meaningful.

Does somebody want to give me an article topic? We could put Deming, I haven't run it before. Like Deep, deep seek or something like that.

Radiation hardening Radi, uh, like, um, okay, radiation, radiation hardening. It's like for robots. I don't know.

We'll, it's my jam. Something I'm into, Okay. And now, and I always love to do this, right?

So there's a lot of, lot Of enthusiasts for that, you know. So this is our port of a demo, um, uh, kind of a app layer on top of Stanford storm project, right? You can tell we've integrated with, um, waza, we've got it pointed at one of our model endpoints and now it's starting to, um, we'll come back to it, starting to think to perspective.

And because we're kind of low on time, I'm gonna actually flip back here while it's working. Oh, but I love this. I'm glad this came up.

So take a look. The first thing it does, this is multi-agent systems in a sense, right? At least multi persona systems, right?

So the first thing it does is it tries to come up with personas that will then be involved and like building this article and it has this fact writer and there's like a constant interrogation where putting on the hat of each of these personas, it asks relevant questions and then it goes out to start doing the research, right? So we'll come back to that. So I, I wanna interrupt 'cause I know this is a great demo, but my co delegate here, Gina, who um, took a quick break, she, um, has a thing about, uh, terminology and yeah, you keep using agent here and AgTech ai and I think we think of ag agentic AI as something different than you or you are.

Can you maybe talk about your perspective of agen agents and what that means in your context? I think, um, agents come in tears. I I we have a great slide that talks about this.

So, um, I think and And Matt answer quick 24 Minutes. Yeah. Ba basically I think there's a tier where you kind of get into, um, you chat is not agentic.

And I think the kind of level one, if you think about this like peril, like FSD levels, right? Like level two fsd is kind of compared to an agent that calls tools and then you get to agents that plan and then execute a bunch of plans. By the way, I mean actually, you know, speaking of agents, I put this up too, right?

Um, we'll see if this is online, it's gonna, it's kind of spinning up the sandbox environment, right? The fifth app now that's run on top of us, Right? And, and I think this is great and the, the, the, what I'm digging at here is the traditional view is we have chatbots and from a a, a, the user perspective, right?

Whether the chat bot is going off and talking to one thing or 400 things or 400,000 things on the back end, I don't really care, right? As long as it gives me an answer. So call response, call response, right?

I think that was what Steven's question was about as well, is we have call response. Is that going to one machine or many then can, if I ask the agent something, can it act on my behalf? Yes.

So Matt, show him it right now. This is it. He literally asked it and now it's going off and doing it.

Go ahead Matt. Yeah, But it's, it's running. So I mean, I told it build me an app and it's, it's building it, so it's coming up with a plan, it's executing all over the place.

At some point I'll be able to click, so then it's not very fast to refresh. It's starting to build the code in here, So it's going off and building an app, which is great. Yeah.

Now at the global level, what is the problem you're solving for me? Well, in this case it's probably that you, um, to, to do a real world like this is actually the app that our customer used to do the data conversion. So they wanted our, our arcane weather data format converted to parquet so they could do analytics.

So Matt, let, let me summarize that. So, um, the Department of Homeland Security CISA had a hundred years of data mm-hmm. On low barrack metric pressures.

It was in multiple formats, including this very arcane one. We had one ML engineer that was able to utilize open hands here, who actually go grab all that data, reconvert it into a single format, process, all of that data. 3 billion rows, trillions and trillions of data points.

And the agent just went off and did all of that and produced the graphs and said, here's what we've actually found And who fact checked it. The whole thing was recursive. All of the data points that brought into here and we're able to go through and actually look at each one of those and run, uh, anomalies that pointed out all the anomalies of the data.

Nine, nine nines were too low and too high. Earth would've exploded. All that kind of, But, but data is never, is never just data and how data is interpreted and especially is displayed in tables and graphs and everything is always opinionated.

And if it's data that's that old, and I know this is atmospheric data, which is reading, so that's one thing. But if you look at, um, just city map data from the past 8,100 years, redlining districts, um, other things that are, are there that we know they're there because of racist policies, but that's not obvious if you just look, send an agent to look at the data. So that's, is there an opportunity or an availability for there to be a human in the middle of all of this and to make sure That's actually part of, that's actually part of the point of this tool really was that, um, that we could have a data analyst who is a person who was looking at what it was doing, who could actually have it explain like the, the theory of the code.

It was, it was writing. Um, and that then of course we had the chief meteorologist of cisa who was also kind of like directing this, right? Who's a, you know, a, a low pressure weather systems expert.

Um, and who actually like was on a panel last week talking and mentioned Kawa a lot, but she was also talking about how important it was like to be specific about what you were doing around weather predictions, right? Because if you globalize them, you know, weather in the Antarctic is very different from weather in California versus Florida, et cetera. Um.