GenAI and Data Security with Luke Marsden at JFrog swampUP 2024

This is Textron tv. Hi everyone. Alan Shimel back here at, uh, JFR Swamp Up.

Our next guest is actually one of the partner companies here. We're excited to have him on. It's Luke Marsden or Matson Marsden.

Yeah. Luke Marsden. Hopefully his name is spelled right on the bottom third of the screen you're looking at right now.

And Luke is with Helix ml. That's Right. Yeah.

And Luke, first of all, welcome and thanks for being here and welcome. Is this your first swamp up? It Is.

Yeah. Yeah. First swamp up.

I guess I should ask you off the bat, what do you think? It's great. Yeah.

I like how great energy small. Yes. It's like much more manageable than a larger conference.

This Right. This isn't Cube Con or or something like that. Yeah.

It's fun. And we get to build relationships with the people. Yeah.

A lot of conversations, good conversations. So, well, before we get into Helix, let's hear a little bit about your story. Sure.

So I am, um, a startup guy. I've done, this is like startup number three. Um, and throughout my whole career I've been doing DevOps, moving into MLOps and now Gen ai.

So I guess that's why they invited me to talk here, because that's like the intersection of The things that is where, where things are today. Yes. Yeah, yeah, yeah.

And what's your position with Helix? ICEO? You're the CEO founder.

Yeah. Yeah. Yeah.

Excellent. Yeah. Now people are saying, great.

What is this Helix, other than, I know it's something to do with doops MLOps and ai, but tell us what is Helix? Sure. So Helix is a private gen AI stack.

Um, what that means is you basically get all of the capabilities that you get from like an OpenAI and chat GPT and all the APIs and, and things you can build around it. Um, but you run it locally on your own infrastructure. And that's super valuable for companies that care about keeping their data private.

If they've got more sensitive use cases or if they just don't want their like data to leave the firewall, then um, you can spin it up and, uh, and go from there. Excellent. Um, how, how old's the company?

So we launched in December last year. Okay. So really new, Almost less than a year.

Yeah, just quite a Year. I can tell you the story if you like, so No one's watching. Let's hear it.

Yeah, I mean, so basically, um, like I said, I've, I've done three startups. The, the first one was, uh, storage for Docker, uh, back in the early Docker Kubernetes days. That one was called Cluster hq.

science. That was all around training models, deployment to Kubernetes, monitoring them, and so on. Um, and when that wrapped up, I, uh, I went and consulting for a few years, and so I was working with like clients all over the world, and during that time, the chat GPT moment happened, right.

And, um, I was saying this in my talk earlier, it's like machines mastered language and like deep learning got to this point, this sort of critical point of like, oh, now they can talk to us like science fiction. Right, Right. Like Scotty Hello Computer.

Exactly. Yeah. So, um, what, what, what I saw in the market was this really interesting thing happened like sort of mid to late last year where Mistrial seven B came out uhhuh of this research lab in Paris.

And, um, and now suddenly you could get like a model that was almost as good as chat GPT that you could run locally. And, um, the other really interesting thing that happened around that same time was, um, it became possible to fine tune that model on your own infrastructure, just on like a consumer GPU. So you might have like the kind of, um, gaming rig that you play computer games on at home, and now all of a sudden you can like fine tune a large language model on that just one computer.

And I was like, I turned to my friend and colleague Kai, um, and my co-founder and, and I said, Hey, it's time to have another go. Like the conditions are right here to build, um, a new AI platform business, uh, that is, um, built on top of all of this amazing research that's coming outta the open source community. So yeah, in December we launched with, um, the first version, it, uh, we went into the market with two hypotheses, right?

So hypothesis one was people are gonna care about running models locally. And hypothesis two was people are gonna care about fine tuning models, like training a model a bit more on your own private data. Yep.

And the really interesting thing was the market said hell yeah, to the first hypothesis the market said yes to running models locally. And so we launched on December 21, um, on, uh, December 24. This, uh, German company shows up in our Discord channel and they say like, we really want this because there's all these regulated enterprises in Europe who, who literally can't even use the cloud providers for whatever various reasons.

And they're all really desperate to get access to this gen AI stuff because they can see that it's gonna completely transform the way people do business. Um, so, uh, by January 1st, um, they'd integrated Helix into their stack and, um, uh, and yeah, there's a case study on our blog, um, if you want learn more about, about That. Are you allowed to mention the German company's name or, Uh, yeah.

A WA Oh, Called a WA network. Yeah. Yeah.

Okay. Yeah. Yeah.

Um, So you, you're right, in many ways, clearly Che GPT bursting on the scene and opening up Gen AI has been sort of a boundary Yes. Layer. Yeah.

Um, BCAD kind of thing. Right. You know?

Exactly. Exactly. Now when we talk specifically, and, and, and I spend a lot of my time talking specifically about how this affects developers in DevOps.

Okay. Yes. Right.

Because I, I think there's, let's call it AI for the consumer. Apple just did their announcement this week. Apple Intelligence, apple Intelligence, and Yeah.

And it'll, you, I could see my wife getting on there and you know, it's, it's Siri on steroids or whatever you want to call it. I Can't wait to talk to a good LLM on my phone, honestly. Absolutely, Absolutely.

And you know, I'm debating, I have a 15, do I go to 16 or I probably will, but that's another story. But in any event, but then there's the experience of the developer, of the DevOps person using AI to help script, to help generate code, to help test, yeah. Mm-Hmm mm-Hmm.

And automating some of this stuff and really kind of, you know, putting it through an accelerator, if you will, and that experience. That's, that's not the consumer experience, but it is. I I was just talking to Scott Johnston before we we spoke with you.

It's gonna make everybody a developer, right? Because you don't need to know Rust. You don't need to know.

Yeah, yeah, yeah, yeah. You just say, create an application that does this, this, and this, and if you care enough you can say, and by the way, put write it in rust, or something like that. Yes.

Yeah, yeah, yeah. Otherwise, write it in any language you like. Mm-Hmm.

And that's game changing, right? Yeah. It's gonna change the World.

It's world changing world. Yep. Yep.

How do you, at Helix, what do you guys like, is that what you're looking forward to or? So I think there's two sides to this, right? I think the first side is develop a productivity for people building products.

0 last week, and, uh, we launched on Thursday and um, on Tuesday night, I was trying to finish the front end for the app editor really, and in Yeah, it's a startup, right? Yeah. No, I get it.

I get it. I've been there, I've done five myself. You don't have to tell me.

And, and literally in two hours I was able to put together this React front end for the app editor that you can see on our website. Yeah. Uh, and it looked great.

It took some guidance, but I wouldn't, I'm not even really a front end developer. No. And like this is dramatically accelerating our ability to ship.

Um, so that's one aspect that's like one side of the coin. The other side of the coin is like, how do DevOps people figure out how to run this, this stuff on their own infrastructure? And that's really where Helix is focused, is like making it easy to stand up, uh, open source models on your own GPUs, uh, on your own Kubernetes clusters, and avoiding having to send your data off to some third party.

And, and I think that is going to be a new frontier. Yes. Right.

Where, 'cause you mentioned the two things, both of them are, are real, number one. 'cause once I could do that, I don't necessarily even need to send it to my data center because can I run it on the edge? Yeah, Absolutely.

Can I even run it on my end points, perhaps? Yeah. Right.

Yeah. IPCs and all of this stuff we're talking about. So I, I think that's a big piece of this is we don't have to send everything back to that big old Yeah.

LLM over there. Mm-Hmm. And then, and secondly, the, the the other, what was the other key finding that kind of caused you to launch this?

Right? We don't have to run it back. Yeah.

And the second thing, and the Second piece was around fine tuning, like making it possible to train these models yourself on your own, On your own. And that's what we're seeing now too, is people are saying, I like having that big LLM as a safety blanket. Yeah.

But I want my own, I want my own LLMI want my, I want to train on my own stuff. Yes. This way I have more control over my output.

Yeah, absolutely. And, and so I, again, I think these are not gen two, I, I don't want to be that bold. Yeah.

But they're definitely, you know, newer things coming out here. Yes, absolutely. I, so I mean, when we launched, so we put something quite innovative into the Helix products around fine tuning.

Um, and actually our, our most popular blog post was called How we Made Fine Tuning Missile seven B Not Suck. Okay. It did really well on Hacking News.

And, um, what we did was we took the idea there is if you, if you fine tune a model, fine tuning is just more training, right? Yes. So a model, you can think of it as like a big wiggly shape.

Um, and you, you, you read answers off the wiggly shape, like a sort of mathematical equation. Um, and fine tuning is just more training and training is just like adjusting that shape until it gives you better answers through a sort of iterative process. So, so yeah, we, we looked at this fine tuning, um, capability and we saw what people wanted to do was to bring their own documents, um, and fine tune on those documents.

But what you, but the challenge there is if you just fine tune on the documents, then the model you end up with is very good at completing documents. It's like a text completion system, right? So you can type in like, um, like, uh, the AI spec from Helix ML is, and then it will carry on the sentence, but that's not how people want to use these chat bots.

They want to use them as, uh, what's called instruction tuned, which is, uh, a back and forth conversation. You wanna ask a question and it answers you well, It's refinement and stuff. Yes.

Yeah, Exactly. And so this, this piece that we did was to make it possible to con, we used another LLM to convert the source documents into a series of question answer pairs. That's the kind of questions and answers that users would want to ask about those documents.

So the prompt's quite funny. It's like, pretend that you're a professor and um, and answer these questions like, based on the source material or like generate questions for a quiz. That's the prompt.

Okay. Um, and so you then generate these, these quiz questions and you feed them back into the fine tuning process and you end up with a model that's great at answering questions about, about this source document. Yeah.

So that was super interesting and like we ended up like iterating quite a lot on like what prompts do you use and asking questions from different perspectives and like who, what, where, when, why, that kind of stuff. Um, so that, that, that was, that was really interesting. But actually what we heard from the market when we put out the fine tuning product was kind of meth, Really People Yeah.

Which was surprising. So we got a good positive signal on the, like, local models, right. And we got a kind of meth response on the fine tuning, but because everyone said we just wanna do rag.

Um, yeah. So there is, yeah. But do you think it's because it's early yet?

Yes. And also I, I do think eventually they're gonna want that. Yeah.

But what I think about fine tuning is what we're gonna really, where it's gonna come really into its own is when people have prototypes and, and proven out that you can do a use case with one of these big general models like A 70 B or a 4 0 5. Um, but those things are big and expensive to run. So what you can do is you can prove that the a use case works with those models.

Um, but then you can, uh, take kind of trace, uh, you can generate synthetic training data from watching that like very intelligent model, do a good job of doing your application of ma of, of doing whatever it is your application needs to do. Maybe it needs to make API calls to an endpoint and it needs to construct A-J-S-O-N body for the API call, or, um, maybe it's doing rag in a certain way. Um, and you can, you can trace those, um, those system, uh, you can trace that LLM system generate synthetic data and then use that synthetic data to train a small model that's very good at that specific task.

And I think that like specialized small models is gonna be how we scale. Imagine you've got like a million or a billion rows of data you need to process. You don't wanna run all of that through like one of these big expense models Especially.

Right. And especially not at the edge. You don't even have that horsepower.

Yeah. You know, my friend John Willis and Patrick DUIs, who I'm sure from here. Yeah.

Yeah. You're doing a lot of work in, uh, I don't know if you've spoken, you should talk with them about that. I'll, yeah, yeah, yeah.

Doing some interesting stuff. Anyway, we're about out time. Awesome.

Yeah. Luke, this has been great. Thank you so much.

Best of luck with Helix. Keep us posted. Yeah.

We're taking a break and swamp up. We're gonna be back in a moment.