Kamiwaza – A Single API and SDK for GenAI Applications to Run in the Enterprise

We believe, and we know this, the ecosystem of agents and apps that are being built, instead of having an individual stack per one and an individual security layer per one, and an individual rag process per one, the ability to just have multiple levels of these apps running on top of us interacting with that data and even interacting together is really, I want to just say mind blowing to actually see this. If you've seen it before, great. You're still gonna have your mind blowing.

Just the interaction of all of these separate applications and capabilities is kind of cool, but it has real outcomes in the enterprise. Matt's alluded to one, I'm gonna cover that a little bit more in depth, uh, later. So I'm gonna go through about three or four others, uh, that we can publicly talk about.

So one app, one API one SDK, but it's running everywhere, has access to the data everywhere. It's not moving the data. And now you can install all of these third party apps.

And this is why we call this quote unquote the Docker for generative ai. That's also why we're creating the category for gen AI orchestration. We are the gen AI orchestration engine, as you'll sort of see, especially through some of the demos.

And we take that hardware agnostic approach very, very serious. So if you think about sort of the largest production open weight model that's out there, it's LAMA three one, uh, 4 0 5 B. You could run that massive model on a single server.

Our typical enterprise cluster starts at three servers for redundancy and processing load, et cetera. But if you look at, you know, gouty three with a, a terabyte of memory, the A MD, uh, three I 20 fives with, you know, almost two terabytes of memory and so on, even all the way down to say Qualcomm AI ultra one hundreds, very, very power efficient arm processors with extensive, uh, GDR five, uh, vra, you can stack a lot of them into a server because they're so low voltage. You could even then run this off a normal air.

Cool. Put it into, uh, a rack at an edge and still be able to process a very large model. The largest model typically that is even running in production.

And you can get massive throughput on this. Our bench testing tool, which will actually take the LLM with the app, with the process you want, and actually run it end to end, gives each enterprise the ability to understand what that load is gonna need and what the sort of theoretical load would be across the cluster or services on a single server with a eight way, uh, a MD. These are MI three hundreds.

Uh, we were getting about, and this was a, um, 70 70 B or 72 B model getting about 8,000 tokens. Uh, a throughput on that particular one. Just to sort of put that in context, that's one server, 8,000 tokens per second.

The human readable sort of, uh, uh, capabilities, roughly about 20 to 25 tokens per second. You're getting a massive amount of parallel processing and capabilities to run those agents that are doing that massive amount of inferencing to actually co uh, accomplish massive amounts of impact and business outcomes in an enterprise. And that's one server locally.

You need at least three of these, like I said, to really build an enterprise cluster. So now you're talking 24, 20 5,000 tokens a second just out of the gate. That is a massive amount of processing power for just three servers.

How do we sell it? As Matt and I keep eluding, this is sellable code. This is fully distributed.

You install it, it is downloadable and it is installable. This is not a hosted service. We have our free version.

We have a very vibrant community. We're expanding that community and its capabilities. Uh, you can go into Discord, grab the software, uh, get support from the community.

We've had amazing use cases from all around the world. Several colleges, uh, overseas have sort of picked it up in their A IML practices, uh, and it's great to see what they're doing, uh, in supporting them. We also sell it as a standard per GPU component.

Now, the standard per GPU component really matches up with Nvidia. NIMS is really the only other installable inference product that we see in the market. 5,000 per GPU.

So if you had an eight way box, it'd be $40,000. We're not typically in this business. This would be for our customers that want a smaller box at the edge.

We're in the business of the enterprise edition. Our $25,000 enterprise edition has no limit on GPUs. Once again, you need at least three of these to build a cluster.

So you're looking at a $75,000 entry point, but with that cluster comes the ability to run all of the latest GPUs. If you could somehow pack 16 in like a Qualcomm wouldn't be a problem. But we've introduced an incredibly different thing in the market as well that we haven't seen out there.

We provide an outcome support per cluster per month, but we're not talking about supporting the stack. We're not talking about the, the tech issues and bugs of the stack. We're actually talking about you can submit a separate ticket into our system and outcome-based ticket.

If you need help moving a and translating that a data into B data, if you need an agent that's not fulfilling the request, if you're getting too many hallucinations because of X, Y, and Z, you support, you submit the outcome ticket. One of our Gen AI architects will actually get on the phone in a scheduled service and we work the enterprise through the outcome. We are trying to make this so it's not shelfware so that there are actual deliverables and we're moving those forward.

He did get his hand up first. Say That you do orchestration, your full stack ai. Do you include, um, orchestration and, and uh, and evaluation and observability tools like Arise?

Um, Galileo in the stack, Actually I heard you talk Yeah, go ahead Matt. I was gonna say, or orchestration. Yes.

So we've got, well, I meant, I meant to say, uh, observability, I misuse Observability In this in this aspect of evaluation. I'd wanna plug into US Fan. Fantastic question.

Right? So, and, and I'm glad Luke showed that graph and this question came up, right? So, um, if you think about us, like we have a lot of different things that we're doing all up and down the stack, and Luke Hast really gotten to talk about, you know, the app garden, um, that we're introducing, although we'll kind of talk about, uh, in some of the context of the demos.

So we're, we're actually dealing with the app layer even in some cases. However, um, you know, that eval suite's very close to the hardware one that you saw the graph of, right? Because it all comes down to the fact that we are very mindful we're using different engines, deploying things like whether it's Lama C-B-P-V-L-L-M, SG Lang, um, to function as backends, and then we're normalizing across hardware.

And I'll actually show you running, running today at, at demos on, on three completely different hardware platforms. And um, so it's really important. But because, But Matt, you're you're familiar with like Arise Galileo y labs.

Yeah. Those, those are not at the hardware level, right? I I was trying to ask, do you plug into those in your, they Plug into us.

They, they, Well you say you have a full stack implementation You would plug into them at the app layer, right? And like, you know, one of my favorites, I think we're kind of have the same thing here, right? Would be prompt layer and of course the lane chain guys have their own version, et cetera.

We are not doing prompt level observability. And it's, I think it's a super important part of production. And honestly I've been tempted, but like when I look at what we built in our stack, you know, we've solved a lot of what I consider to be largely unsolved problems.

Like it's either messy homegrown solutions, but nothing that really works out of box that's not true for observability. Like people had a very keen perception of how important that was. And I've seen several good implementations, and so I haven't really felt like we should waste our time right now reinventing the wheel.

And I haven't had customers ask us to. And so, you know, I, I'm kind of bearing in mind that it's important, um, and as much of the stack as we tap into, of course, like, you know, it wouldn't be completely insane to be thinking about building it. But yeah, I I think it's pretty well solved by a bunch of people out there.

And so I don't see a reason to wade into that mess yet. Okay. So you had mentioned Starburst, you had mentioned some of the other things and, uh, your kind of loosely coupled part.

Um, any, any involvement or, uh, exposure in like whether it's ox uh, prestisimo, I'll figure, figure that, how to say that correctly at some point, or other things part of this kind of like iceberg post iceberg, um, view of what's happening in these, uh, um, enterprise use cases, um, for treating data lakes differently. Um, is that, is that something that is part of your roadmap? Um, or things that you're contemplating?

Our, our connectors for the catalog can read, uh, the metadata for delta tables and iceberg tables. It kind of stops there. Okay.

You know, I mean in, in a sense like we're doing retrieval, um, and uh, we're kind of relying on, um, some, some common levels to, to pull the data now, of course, like I think both those formats, the sort of visible parquet is actually like the most recent version of a table. So it's super easy to read, you know, um, even without any kind of extra tooling. But we can see things like, um, delta tables from a metadata perspective and We're only reading it, so it's easy As long as we is is it fair to say like, you're not trying to be like super exhaustive?

Uh, 'cause I mean, I was thinking like when you say like the, um, your business objective, not just the technology, it, you know, it feels like what you're saying is you're, you're, you're delivering some form of a software experience, but there's also a professional services component. So I'm trying to suss out where does the product begin and end the professional service begin and end It. So it has to be able to connect to the data.

Yep. It has to flow through the stack and then we'll help with the business objective and the outcome-based support. Got it.

We, we get into some really completely off the wall stuff, but I mean, we are actually normalizing some of this, right. But I mean a great, you know, example of this is, um, somebody just came to us to talk about, um, if we could take a manual process that they use for RFP responses, right? And they have, um, Excel spreadsheets that have a bunch of data.

They have templates like dot x files and word, um, that have these things that they fill in and they, they want to kind of go through some like business rules and it's not a fill it out completely thing as you'd expect, but I mean, these are like thousand bus page RFPs where they're only responsible for say, 50 pages of that. And those 50 pages, they have a lot of like, essentially you'd call it, um, business process rules about how to respond. And then some of that data's in the Excel.

And so, I mean, in a prototype that we built, you know, we use LLMs to go in and code to go in and, and both look at the Excel content and reason through it and then apply those into the, you know, word document, which we turn from, of course the template into a, you know, instantiation of a Doc X, et cetera. But as does that belong in a standardized stack? Of course not.

Um, but then on the other hand, like, you know, might there be, you know, middleware that we might add at some point to kind of say, oh, this is middleware you can use to reason across the spreadsheet. Okay. Maybe.

I mean, so, but the point really about outcome based support is that, um, our team just is a bunch of like jenai enthusiasts. Like, you know, there's just no denying that everybody loves it. Yeah.

Um, and so we're actually really capable of, um, and our, our kind of proficiency with the tooling that helps build these things quickly is completely off the charts. And so we're able to dig in and help people, I think accomplish things really, really rapidly. But it's really important too 'cause we're helping, um, instantiate, you know, wins at a business level, right?

Yeah. 'cause we don't want to have shelfware, we want people to get value right away. And I mean, some of our early customers, it's like they, they come out of a POC and they're like telling their management team, well, we've actually already had two XROI on this and it's been three weeks.

Right. Which is exactly what I love to hear, right? That's what we want people to have.

So, so when, uh, if, if you can go into any more detail, so that example was, you know, RFP related, so that would make me think of companies like Upland who acquired Qvidian, which is, you know, an enterprise RFP response stack, but you know, their, their Gena capabilities, you're waiting for their roadmap. Yeah. And what I'm hearing is you've got clients that are asking you to do a POC or some minimum viable, uh, thing that does that.

Absolutely. The vertical apps, they're, they're Dead in my mind. Literally, I, what'd that take Matt?

Two days? I was get no, it, it took 45 minutes and we were, it was like after we had dinner with a candidate and we were, Luke and I were having a beer at a bar and I built a thing in 45 minutes. Not to brag, but I mean, these things are not always that hard when you understand like That, that's a, that's a clear displacement for me.

So if, you know, if, if I'm waiting for the roadmap of the enterprise, you know, enterprise software that I've, you know, bought that does that one thing using these tools you could compose, uh, in a short Timeframe, build an agent to do all of that. Okay. Got it.

In a very So jazz too, Luke and I, we were, we were one mind 'cause I was so eager to just jump in and say exactly the same thing. And if I'm gonna, if I had, like, I don't think this is the original thought 'cause a lot of people are saying it now, but I mean the, we're about to see like giant software packages, especially SaaS, but just in general, I think disaggregated, like you've never, never seen, right? Because the idea there are already people making a, a reasonable amount of hay, building much more customized versions of your most important apps that have less features, but they're perfectly tuned to you and you pay half as much, um, and that you get a better outcome.

And, and then I think that's gonna be a, a tsunami that just wipe, you know, just wipes a lot out in the next two to four years. Yep. So, so Matt, we have about 45 minutes, so we gotta get through a lot here.

Partner. Um, so just real quick on this, we've already accomplished what we call our phase one and phase two. Phase two is that idea of the docker regenerative ai, the ability that we're the full orchestration engine.

We have 50 apps and services and 50 agents on top of that, that plug right into the API the next phase is a full build out of a model garden where third parties can simply write with that 20 lines of code, their own app to build right on top of us to access that enterprise. Then you also talk about the ability for third parties to build their own agents that have enterprise access. They can maintain those, uh, agents in this app garden and customers will just be able to pick and choose an agent and an app that they wanna accomplish something right out of the gate.