TensorZero: The Open-Source Future of LLM Engineering

Hey everyone. Welcome back here to Techstrong tv. I've got a, uh, new company, first time guest to introduce you to let me introduce you to Gabriel or Gabe Bian.

Coney. Gabe is the, uh, co-founder and CEO over at a company called Tensor Zero. Gabe, welcome to Tech Trunk tv.

It's great to have you on here. Thank you. Thanks for having me.

Pleasure. So, you know, we're gonna talk about Tensor Zero, but before we do Gabe, let's, let's talk a little bit about you, right? How, you know, you don't wake up at four in the morning and say, I wanna start a company.

What, what, what kind of, you know, what path did you take to co-founding this company? Yeah, so I started this company with my friend Raj that I actually met back in college over at that ago in same dorm. We were both at Stanford, uh, had gotten to work together since then.

I, I worked at a number of startups most recently, this company called Ono, uh, barrage, went to grad school. He was pursuing his PhD and Penn Azure was actually an extension of some of the research he was doing during his PhD. He was doing a PhD SU on reinforcement learning, and the main topic there was like, how do you get machine learning systems to learn from experience in the real world?

And as LMS were ramping up back in like 20 22, 20 23, we started asking the same question about lms. How do you get LMS to learn from experience in the real world experience being metrics from your product in your business being human feedback, user behavior, whatever makes sense in the context of what you're building. And we realized that the tooling that existed wasn't perfect for this, uh, feedback loops for the systems to learn from what they're doing from the consequences of their actions.

So we decided to kinda redesign and rethink the stack and the ground up, and that's how Penn surgery came together. Excellent. Now when you say reinforced learning, like so for instance, deep seek, you know, the Chinese, uh, ai, gen AI LLM model, you know, they, they claim that they did a much better job with reinforced learning and and so was a, they were a lot more efficient and training their model.

Is that the kind of reinforcement you're talking about here? Yes and no. So I, I think when it comes to alums, there are different kinds of reinforcement learning at like, also like different stages of the, let's say, pipeline to train a model like that.

If you think about foundational labs or companies like, uh, deep seek, they start all the way like pre-training where they're sending like massive amounts of data to those models and spending out hundreds of millions of dollars, even even billions of dollars to train them. But once you have, uh, uh, a model that exists, anything from GP D five to deep seek, those models and so on, application companies have to take those models and tailor them for specific use cases. You can start very simply by just using prompts and prompts, engineer and so on.

But if you really wanna squeeze the most out of those models, there are a number of different techniques that you might want to try here. Uh, so when we think about reinforcement learning, there are reinforcement learning techniques about like specific techniques you can apply to those models. But we're thinking also about the broader l lm engineering loop.

So how do you think about like the whole application or the whole LM system to improve from like the downstream performance? So we do use reinforcement learning, but that could also mean other techniques. Anything from optimizing the prompts to fine tuning to dynamic context learning and many other buzzwords I can share here.

Absolutely. Now, so you got my attention right, because, you know, IIII use chat GPT as my main go-to for, for, uh, ai. And I've done a fair amount of prompt engineering with it over the months and years now, where it does a good job for me, right?

com and our tech strong AI sites and, you know, a lot of material there. So that it really, I feel like I'm starting off halfway through the thing rather than at the beginning. And I'm sure there's a lot of us out here doing that, but that's not really what 10, is that what 10 Zero is doing?

No, you're, you are really, uh, training these models or better training the reinforced training for applications that are using them, not, not for people's use. EE exactly. So the end user of tenor zero is typically an LLM engineer or a software engineer or maybe a product manager at a company that is building an AI product.

So in your case, you're already using an AI application like H-H-A-P-T, but for us it's typically companies that are building their own products that integrate with the open API or any of the other similar APIs across all sorts of domains. So we have users, anything from, you know, healthcare, back office automation, all the way to, you know, education and tutoring and, and so on, all the way to, you know, compliance in banks and the like, but they're building software systems that leverage those LM Underhood. And for that there are a number of complexities you, you might want to do deal with where you're not manually looking at each of those conversations like you're doing.

Sometimes there's quite like high risk use cases, for example, in Health Clear and compliance and so on, where it's actually interacting with the real world. So you need a lot of additional safeguards, better observability ability to plug into all the different model providers and the like. So that's where a, a tool like ten three zero can come in.

Excellent. And you're doing this through an open source called an LLM Ops platform. Talk to us about what that is Exactly.

So we have this open source project also called Open Zero, which has five major components. Uh, first is a model gateway. So this unified API that lets you integrate with every major model provider e that open source or full source providers using the same API.

So you can very quickly try the different models as you're using this inference gateway. We're collecting various structured data about those inferences as well as like downstream metrics and human feedback and so on about what happened there. And with that, like providing both observability so you can really understand what happened there at the microscopic and microscopic level, but also curating data sets that you can use for optimization workflows, be that optimizing your prompts, optimizing the models, running reinforcement learning, and so on.

And finally, uh, once you're, as you're iterating over the system, we can do evaluations of sanity checks that things are working as expected before you put that in production. And also experimentation. Once it is in production, we can ab test different choices of models and prompts and parameters, and I like to make sure it's moving the needle in the right direction.

So basically, once you have this whole stack in place and it's all open source, you're, you basically have everything you need to, you know, uh, try all the different models, see what's happening with them, iterate on the prompts and models and so on to make them better, and then send it to check and confirm their working as expected and, and moving the needle in the right direction. Very cool. Very cool indeed.

Um, let, let's get some housekeeping out of the way. com, not, not the number zero. Um, and then Gabe, how, how do people engage with you?

How do they get started here? So it's fully open source. So most companies will go straight to GitHub and try the project, follow the quickstar and tutorials, and a lot of companies use it without ever having spoken to us.

We're also very happy to support, so we have channels on Slack and Discord and, and social media, so very happy to answer your questions and help with onboarding and so on. We also have a number of design partners or companies that work very closely with to, you know, take on feature requests, provide support and the like. So this can really vary on, on the user here.

If they just wanna try it out and not talk to anyone, they can just go to GitHub and download it. Uh, if they want help, if they want support, very, very happy to chat it and support it on this journey. And, And what, what's the commercial model like?

We're pre-revenue right now, so we're fully focused on the open Push. Oh, just fully open right now. We, you know, we're getting users delivering delight.

I imagine at some point there might be a hosted version or some premium features or something like that. Yeah, that's great. How long have you been at this?

Well, I works outta your part, your co-founder's PhD in 2223, you said? So it's been, Yeah, so pretty much we started the company, uh, beginning of last year and we launched open source about a year ago. So we just turned one for open source.

Very cool. And, um, are you pre, do you pre-revenue, obviously? What about pre fundraising?

No, we, we recently raised, uh, a seed round, so we raised just over $7 million, uh, which we're very happy about and that Some very healthy seed round. Congratulations. Yeah.

So this will support no continuing to accelerate the, the engineering and growth for the project and starting to grow the team now. So early on it was just me and my co-founder. Now we're six people and they go through, hopefully get to 10 or so by the end of the year.

Excellent. Excellent, excellent. You know what, it's good to see, and, and I love the open source model here, and it's good to see there's so much going on in AI and there's so many different aspects and facets of, of how we are going to do this.

Here's something that I think everyone out here can wrap their head around and with a, a really sane business model, right? Of sort of your traditional open source was delivered, delight, and then, you know, go from there. Um, so 10 to zero is the website.

It's available on GitHub as well. Is there a GitHub url? com/tenor source or something like that?

com/tenor zero. And you, you can find the whole project there. It's all open source.

Yeah. 10 of zero. Excuse me.

Well, Gabe, I wanna wish you the best of luck. Oh, thank you. With Ted Suze.

This is isn't an interest. You know, it's funny, I, I'm in our studio here where we have three different, well, it's a sound stage, we have three different sets. So I was, I came here from the Textron gang set down the studio there, and, um, we were just talking about sort of, you know, there's a feeding frenzy right now.

I'm building data centers and I'm going to use your chips and you'll use my chips and you'll put your stuff on my chips and I'll put my chips on your stuff and all of this. But what we're looking for is the next growth phase, the next evolutionary stage where, okay, you, you've got the, the data centers and the chips and that infrastructure and you know, you have these frontier models, but now how are we going to that next for every med, for every company? And it's not just going into Chet five or philanthropic quad four or whatever it is, it's really customizing, you know, for your needs.

So this, this is that next gen. Good for you, Matt. Congratulations.

Thank you. And I appreciate that. All right.

Gabriel Bian, Coney, CEO co-founder of Ted Soze. That's T-E-N-S-O-R-Z-E-R-O here on text Trump tv. We're gonna take a break.

We'll be right back.