Kamiwaza Model Context Protocol for Private Models – Inferencing and Data Within the Enterprise

Um, there's been a kind of a recent innovation around something called model context protocol. So open standard for communication between clients. But for now we're into sort of, this is a, a new feature that we're bringing to market and Matt's showing it off.

And literally you'll notice this app is really like just running off of vanilla file locally at my system. So it, it actually, all this calling can be completely implemented in like, you know, backend JavaScript code, but it's gonna be hitting some of our models and it actually interfaces with a model context protocol. So we're open sourcing both this app as well as, um, the, the, the backend, um, server, which is has a lot of tools, very fluent.

Um, also I have one of the, one of the most awesomely nerdy, um, engineers on the planet clearly 'cause he has created this agent, um, for development that has a data persona. And I'm, I'm gonna go and, um, say this just to kind of get started. You'll notice we've got some files and we could do a lot of things with this.

Like I can ask him to, I think I generated the Kawa report by asking him to go search the web for us and write it. But, um, let's do this. I said, Ooh, agent demo landing party time.

He's got a Star Trek persona, so I think he knows what I mean. So He's gonna read it and now he's actually reading it. He sees that it's tasks and um, because it's got some instructions for him, he's just gonna start rolling with this.

So, oh, by the way, lemme show this real quick. Do I have this up here? Yeah.

Yeah. So this is actually the repo where we demo this, right? And I reverted the last commit here 'cause I run this demo for fun quite a bit.

But I just wanna show you a couple hours ago I made sure we were at the, the version before the checkout. So he's actually following some instructions. He's cloning that repository.

He's, um, copying like a local file into it. He's gonna analyze the code and, and if needed debug it, right? So he's, right now he's reading instructions about, you know, how to properly edit things.

And I think this is a, is similar to what you do with like a cursor rules file or any kind of other, you know, instruction. Except of course you can actually live contextual hints for an agent to go pick up. Now what what I think is really interesting about doing this privately versus like, and I and certainly have had plenty of sick time with Claude and use model contact protocols plugged into Claude, but because I don't have some sort of like arbitrary, I don't wanna let somebody send me so many messages per hour problem.

You know, I can give them instructions and because every one of these is a message, I can just let 'em go for minutes and minutes or you know, potentially much longer. Right? Um, this is still chat, which actually goes back to that question we had about like what is an agent, right?

But I mean I think you could say this is age agentic because there's a lot of decision making kind of going on here at the model level. Um, and anyway, so I'm really excited about this because we're open source. This, we've got here, you know, open source chat with an open source backend model context protocol with a mountain of tools.

Naturally it's pre integrative ka waza, but it's not exclusive to Kami Waza, right? So, I mean, you could use our plugin, but what we have done certainly in putting these tools together with some of the models that we see customers use most is we have done this in a way where we find the tool calling to be quite consistent, right? Which is pretty important thing.

Mm-hmm. And of course Our spend a minute on MCPI don't think I've Got a good question for this. Please.

I think MCP is awesome. My problem with it, it's restricted to anthropic, right? That's the blow.

Not more. Not at All. Oh, Okay.

That's it. What we're showing you here is running our local. My question, how would you have, that's Why I wanted Matt to spend a minute.

This is, sorry, shut up. Running off of his map. Thank you.

Let alone a server or anything. This is a completely local front end, um, to a completely local BO model context protocol server. But then that the inference is tied back to the, the turbo command Waza host.

Okay? Um, that's you and it's executing locally. So Matt, explain MCP just in general, Right?

So I mean you have, it really just defines the way a client, which to your point, like the original client, this is why I actually got jazzed. I thought, hey, a ton of people are gonna actually do a ton of support for mono context protocol. 'cause developers love Quad.

Um, but I love that they made it open, right? And you can actually do this, by the way, like not only did he commit the code, not only did he get the code, copy the code, fix the code, write test for the code and execute them, right? And this is like PI test popping out here.

He is finally done with his like plan. Um, and I think if we come over here, by the way, and, and refresh, I think Notice that's our GitHub repo external. So the model contact protocol one, let that app running locally, reach out to a third party repo, download the code, update it, read, put it in.

And now we've made the construct of MCP the ability for apps to reach into tools no longer be defined by anthropic. It is now an open service and standard that runs on our local models, our hardware, our locations. And it's now literally limitless.

'cause you can just write new model integrate CP Win. Thank you. Totally.

I actually don't even know what this list is 'cause my engineer is a maniac and just keeps doing cool stuff. Um, I, I figured if I asked he would tell me. Yeah, it's cool, right?

So he is actually telling me the tools that are in our dev buddy extension, which is keeps getting. So we have 20 things in here, gets edit files, clone, run Python, create folders, um, oh yeah. PA pages.

Yeah. One the questions I've seen, uh, other solutions that are able to take a repo and let's say modify it to add a, a feature or enhancement. Think you can do that with this sort of solution.

That's I think what it literally Absolutely. That's literally what it did. Well yeah, we did that actually.

I mean in a small scale, right? But the instructions that he followed, um, in the text file that kind of told him what demo to do was go check out a repository, um, and um, basically analyze code in it, fix it, commit it, right? And that's what we saw happen.

Now he's searching the web. So Did you have it automatically insert A a zero day and then give it a convincing sounding toll request? Could, could we?

I don't know if it's possible, but it'd be pretty fun. Oh, I, oh that'd be great fun. You know, it's fun.

It's funny, like one of these things I think that is important too about this and like, I'm not doing it locally, but I mean I think obviously one of the things we're, we're also working on is like, um, some ability to sandbox things like this a better way. Now of course this runs locally and so, and, and of course what it can do is to only tie to what the model context Protocol Server does, right? So at the end of the day you can enable and disabled tools and any kind of guard rails and things.

Yeah, go ahead. I think the thing Matt's got a question for you. One quick question.

This would be killer. Are you planning or integrating with like Open Hands and Devon? And you know, something We, we actually showed you Open Hands Was that you said Open Hands?

I did right there. So you're integrating with those software. Once again, those software packages just connect to our api.

IFDK, that's the whole point. We're the And what about the ID tools like Cursor and K client and ada? Is there any sort of integration there Already?

It's so well cur cursor. I mean you could use this as a local model. I get that.

I'm just wondering if you can make it any easier for integration Or No, I mean, you know, it, it's super easy to integrate with us 'cause you could just point at our model, but I can't obviously guarantee the results. It's totally depends on the model you deployed on top of kaa. Now that having been said cursor's a special case because cursor tab is like entirely private, right?

So you have to have like all their auto completes are powered by their own private model. So you could replace the like open AI anthropic calls with us. But now if you look at something like Windsurf or Kline, Kru, all those things you could point just at us and I mean, you'll know I I don't know if I had it deployed right?

But on my local one, yeah, I mean, alright, We're geeking out a little too much. No, it's, and I apologize, but, but I I wanna follow Up with you back this, by the way, I, I'll follow up on This. Yeah, yeah.

We got through this, This, this is you Jay In information synthesis and I'll click show final article and here's all the details. Now what you'll see is it went out there again, as this age agentic, I mean it's doing a lot of things to coalesce things. It's got multiple personas that are having a debate in order to build this article.

Now. So this was all based on the research, right? And if I look, I can see now all the kind of q and a that happened with all the personas that kind of went into building this and Oh, research is the topic comes up with five separate personas.

Research is the personas to make sure those people have the right context. Then it goes out and it researched about a hundred plus articles on the web, ingested that, and then had those personas cross argue so that you can get a final paper. Okay.

Okay. And I just wanted to say by the way, you noticed like the footnote, so if I click on the footnote I'm actually, which was up there Yeah. You You're site reading where I'm going.

That's, that's exactly what I was curious about is how Yeah. And then footnotes everything that I put it back in there 'cause they're arguing and they have to be contextualized. Yeah.

And by the way, like, I mean there are, there's room for improvement. Like I, I think when I have seen this, um, what I love about this is I'm a huge fan of perplexity, but perplexity sometimes feels a little skin deep when I wanna really know something. Um, and you know, we have actually run this where this thing has disgorged, like it has literally searched 90 pages of search results and then scanned through 140 web pages on the way to building the article.

Like that's pretty cool. Yeah. You get a much more in depth output.

Um, and, and I think this is part of our strategy as a whole, and I didn't really get, we kind of running low on time and we didn't have tons of time to talk about it. But, um, we have this initiative we call 50 50 and we're kind of aiming by the second half of this year because what we've learned is there's a lot of call the enterprise cases split into two halves, right? There's the super custom, super specific back office business processes and this is where the outcomes support comes in.

It's like, help us with this super arcane RFP process. The other though is there's just tons of calls for these like simple utility things that are chat bots and agents and research assistants, et cetera. Because we can be a platform, we can deploy them, authenticate them, tie them into your data, tie them into your model endpoint.

So they just work. And even in some cases for some of these, like, like this storm actually I believe that the, the code for this has um, like an auto model selector, right? In other words we'll try to pick the most appropriate model if you've got some deployed, right?

Something we're doing more and more. Um, and the, the ports. So we're gonna have this whole portfolio, we're open sourcing all the ports, right?

So everybody can see how we integrate with the kaza and I mean we have a blog entry that kind of describes it. 'cause it, it's not a lot of lines of code, right? This is a, this is 30 minutes of of porting to get it to work completely with our stack.

But you can start to see we're kind of driving, I, I feel like we're doing the, the Piner maneuver top and bottom right coming app down to make simple apps simple and easy to use because it shouldn't take you, you know, hours of pain to go and deploy things like chat bots and research assistants, especially when they're not that customized. And then the bottom up part, which is like building the platform that kind of keeps your infrastructure and your model repository steady and all those things let, helps you with rag Matt, try To, there's a lot of questions here lemme try to understand. So first question I want to ask is, can this sort of thing be done with the community edition?

Uh, yes. As it exists so I could get this sort of Yes. Same sort of, we we would like you to do it.

Second question. This is just an app that installs right on community. Second question I wanna ask.

It's all docker containers. So I can load this thing and run it on my Mac in 15 minutes and have this whole stack Yes. Sitting here being able to question and query it like this in 16 minutes and, and and the, this game, it's Not, it's not a hundred, it's not a hundred percent Docker containers, though.

It's mostly Docker containers. The react front end is not yet containerized and the core common wants the app, um, also, um, is not containerized, but will be, I can pull down on the Docker registry. Yes, yes.

Yeah. But, but the rest, um, the, and it it's heading towards full. All of the, all the other things, all the other components you saw like the vis and the backend database and ET CD and the load balancer, those are all containers.

They do come from our Docker hub. Um, and then of course, Lama, CPP or the backend engine always has to be launched outside of Docker because Matt, Matt OSX Docker. Yes.

Yes. Next question. Yes, yes.

I meant yes, Jay. Okay. So, um, there's a, there's a, there's an AI startup called Inquisit ai.

They do, uh, deep, deep search into all of the wonderful hopefully la AI generated research papers that are out there. Mm-hmm. If I was to want to attach this, uh, agent that just did that article from your article selection of demos, what would be the, how would I have to interact with, you know, Inquisit AI to have that be one of the other things I go out to and pull?

How, how I, how would I extend? What was your demo to go to a third party AI service? Yeah.

Yeah. So, um, it depends on what they expose from an interface perspective. So Storm and granted like, I mean this is the point is like we're we're trying to port open source things over to common wise that are super useful.

Storm has that kind of plugable backend. So there's two things that it's using there, right? One is, um, I have it plugged into I think ser dev to do the search and then it's got retrieval that's just like web-based retrieval, right?

So it's using internet search, but it's a total plug, totally pluggable backend. And you can actually, I think even outta the box, it supports a bunch of other sources. You would just need to have that backend source that would go in and search it.

And then there's kind of an interface to, to connect that as a thing inside of Storm. I think it's only in code today, not It'd be super Clear Storm is an app Yep. Put together by research assistance and stuff at Sanford.

Yeah. We just literally plug it in via the API and the model repository device. Got it.

And then you can edit that storm app to do what you asked. However, And search won. You didn't say this, but it should be obvious.

We're, we're gonna port it so that it can consume all and search Won't give You the papers. Data Search won't give you the papers, you've gotta download the papers. So service like that will actually integrate the downloading paper.

Anyway, just a True story. org every single day. You can write that semantic matching.

Yeah. He already wrote it, so, yeah. Yeah.

I actually asked, he he, he'll probably port it to us at some point Okay. For private research. Thank You.

Yeah. What about GPT researcher? I mean you, have you seen that and played with that?

'cause that sounds I have. Okay. Very similar what you just showed Though.

I I've seen it. I haven't done it. It seems neat.

Uh, Anyway, it's just, uh, it does exactly what you just showed to The storm Storm to me. The, the, the chief innovation that got me really jazzed about, it's actually the internal way that they handle information. This like semantic tree structure.

You can go dig in. I don't wanna take our time, but yeah, it's really cool. Other than the open hands demo, which was pretty quick.

Um, a lot of this has been chat interface level stuff. Open hands, obviously the agent will go out to the web, it'll go out to the code, it'll execute on your behalf. You can say, what graph should I be looking at?

And they'll just start going off of that. They'll recursively say, I also found this data very agentic, but A lot of working by the way. Yep.

A lot of our flows also are just simply adding context. A lot of customers have ML pipelines that they want to add semantic context to. 'cause if it's coming in, say as a CSV file and a word actually has a comm on it because it's a proper name that sort of offsets the rows where semantically you can say, look, I recognize that as a product or service or name it restructures it and then re putts that forward a lot.

And I mean a lot of our data structures are based off of that. The other thing that we've started to move heavily in is rapid replacement of our PA customers are just struggling left and right with all of those generic RPA services. I'm getting multiple thumbs up over here and it's typically, once it gets down to it, it's API calls and it's C sharp code and it's horrible.

Well guess what? Use, install our platform, connect all the data, tell it via that code, what it's accomplishing, and the agents just start building themselves replacing that rapid RPA service. And then you take it to that whole next level, the super human level where you only thought of that initial RPA service, but now with the ability to have Angen workload over it that's connected to all that data within the security, the whole next level of that service gets integrated in, the new workflows get built.

And that is a rapid attachment service that we've been helping out and doing. We're even running a promo where we'll swap out 500,000 of, uh, RPA licenses from UiPath and some of the clear ones, the 250,000 of our license, and we'll actually do the swap out for you. If it is that worst mistake Yes.

Over the last decade in I technology, Yes. But it's a massive market. People grain themselves into it and they're stuck in it where literally we'll swap out lower your costs by half and swap all of that out and extend it to the whole next level of agen serviceing capability.

That's Huge. It's huge. Any other demos you wanna write?

See, Matt's still running demo. Yeah. We we gotta have something to entertain us while we're talking.

Right, exactly. Well, I think what you're illustrating also too with this is that, um, there's a, there's a, there's a trope or a cliche thing on Hacker News, which I don't recommend anyone read read the comment, but in the comment there're usually, oh, I could create that entire company in an afternoon. And it, it won't be the totality of a company, but Totally.

They'll, they'll, you know, whatever the parler trick is, they'll recreate it. But you, you've done a, a various, you know, number of demos here and each one is a different mm-hmm. Kind of a piece part, but I keep coming back to that word composable, um, because it seems like you are allowing for people to, um, couple and to your point, add more context around, you know, okay, cool story, but now link that with this data or do the got it do the left join of services that no one has ever done yet because that business question hadn't come up yet.

Okay. And, and, and if working in say, open hands and you get that result and you get that code, then you just run that code on our box or an adjacent box that makes the call into us and you just keep writing more of those micro apps and you just keep building out your own little app garden and farm to accomplish all the next level stuff. And as you were, you were driving this locally from a laptop, I believe for most of this demo, but you did go to a couple of ex what could have been, I guess externally?

No, uh, uh, each one of these apps was running on various, like that storm one was definitely running on a big, None of the models. Um, none of the models were local except the one that did the pet store, um, chat. All the rest.

Honestly, the max speed's just not, you know. Yeah. Hence the name Turbo I think was in the host name.

Yeah. Yeah. Tur turbo's our eight way am d like Got it.

That's a 17,000 plus, you know, token per second box because if you're gonna do something like that storm demo, it's a, it's a beast. And your enterprise price 25 KA year, uh, enterprise that runs on that service. 20 5K, the r you could do the standard edition.

I don't know why you would, eight processors would be 40 K. So there's that crossover point. Got it.

Okay. Yeah. This is, by the way, the blog I keep mentioning, I just thought I'd flash it, um, real quick.

Like this is all the code required to port that Versa chat bot over to Combi Waza. Right. So it's super simple.

I think any, you know, developer can kind of run through this. I mean, and now it's kind of sky's the limit, right? 'cause now you can tie a lot of that, um, intelligence if you will, into that.

And, And, and, and he, he it that those 20 lines of code took a standard chat app integrated into the enterprise identity access management of the enterprise because now it's plugged into us and it has that full authentication schema and the ability now to recognize what models are running in the inference mesh. And you can select that model to actually also do the processing. Like, and it only took those 20 lines that he flashed up there, integrate a third party app right on top of us to accomplish that.

Who do you, who do you see as the primary user or data engineer? Is it more a developer? W we really believe that the standard developer is gonna be the primary push for the generative AI ecosystem.

The data engineers, the ML people, they're fabulous and they obviously have context and they could be using this, but where they've sort of been the purveyors, we think a standard enterprise developer is gonna be the purveyor of generative ai. Okay. So is there database, I'm assuming that won't happen on tomorrow.

So is there a step that involves data engineers in this process? Maybe you think it will happen tomorrow? I'm, I'm being specious, but maybe There is a particular skill set that is super important.

Um, which is when you start taking some of these use cases to more high value production cases that need to be really like steady and constant emails are critical. So you want to have a data set of things that you want your tools to do, whether it's an app or an agent or whatever. And you really want to have, um, the process goes back to that kind of prompt layer and like model ops and things that we were talking about before.

Because that is pretty critical, right? Um, and I think that mindset around model evals, even though it's completely different concept here, 'cause you're not evaluating the efficacy of features, it very much applies. Like the mindset applies very valuable.

Inter enterprise architects are typically the business leaders we work with, they're tie in data. They're gonna put it in the hands of developers And then it goes down to hands of developers. Do The data engineers get a mopped at all or is this kind of No, we're going down the gen i i route.

No, For sure. Especially if it's like a, uh, augmented an ML pipeline. You're working with the data engineers with semantic capabilities on that, uh, to have sort of that flow.

So The data engineers are coming to the developers and say, I need X, Y, Z. Can you, I can go so far with it. You got It.

Finish the, in a lot of cases. Okay. All right.