Mirantis PaaS Technology Stack with Shaun O’Meara

So let's talk about the PA stack. Now we're going into that layer, and this is really the GPU PA stack and what is being offered. So I'm Sean Amara, CTO at Martis.

Let's talk about the PAs. So in that very first diagram I showed you some of these AI enabled services that are in, you know, our neo clouds, our cloud providers can offer to their customers that we have out of box. So a couple of those things.

The key ones, obviously we've just showed you managed Kubernetes as a service, virtual machines, GPU and CPU. You know, this is the very basic that everybody needs to be able to provide. And then bare metal service so they can then stack up a con, you know, combination of all of these for a single tenant and provide services for those tenants.

Where it starts to get more interesting is what are the basic inference and cloud services that they want to offer to their customers that start to create a richer pass experience GPU pass experience for their tenants. So we have inference as a service. We have several solution offerings around inference as service, but essentially that is a tenant UI that they can come in and they can launch an inference service and start to have their own private LLM or LLM from hugging face or one of the other public catalogs.

We maintain our own model catalog, uh, within the tenant. Again, it's an OCI based registry. Those models live within that OCI based registry.

It's based on Harbor, Is that, it's based on Harbor. I always thought Harbor was more of a container catalog than a model Catalog. There is a, there is a move towards supporting it.

There's a lot of work happening within the open source community to better support model and support the OCI layering to deliver models that way. Um, it does involve specific setup, um, within the core network to ensure that you can get the performance out of it. And we do that set up by default.

Um, mostly it's the way that models are presented to the hosts. Um, if you're trying to do a traditional docker pull, uh, on a 600 or four gig file, you will wait until time and Memorial, um, because it's a single thread of the pool. So we use alternative mechanisms to make that a main, mostly infest to make them made available immediately to the host so they can start immediately.

Uh, Jupyter Notebook as a service, uh, managed slum. So, you know, there's a big drive towards running s slur on top of Kubernetes clusters. So we're, we're leveraging this operator to be able to deploy, um, managed slur on top of Kubernetes clusters.

Again, it's stacked up using the services portal and we have MLO, um, as part of our offering, um, as one of those service offerings, we have cube flow, um, and we offer some fine tuning capabilities. What, what's one of the drivers you mentioned there of s SLM on top of all of this? I mean, what it's, it's an enterprise ask or a CSP ask or what's there?

It's being driven to reduce the complexity. So the CSPs are doing it because bare metal management is a pain. Um, a big NVIDIA node like that, you know, one of these big eight GPU, no, it takes 50 minutes to reboot.

Mm-hmm. And if every time I wanna put a new tenant on top of one of these things and I've gotta do a full reboot of the underlying node or nodes, and when I have, you know, 32 of them or 64 of them, you know, et cetera, it's hours and hours and hours of work to, to switch over. Whereas if I have just a Kubernetes cluster, even if I put just one VM across all these nodes and need to connect them, I'm massively reducing that time to switch the cluster over.

Got, um, it's also just simpler from an operational management point of view. I no longer need complex LCM systems to deploy the slum components and lifecycle in, it's just a Kubernetes and then I run everything using Kubernetes norm. So, so It's essentially pushing a good part of the management plant local, Right.

Think of Kubernetes as a new generation puppet or chef or, you know, it's really starting to become that. Yeah. Um, probably more like chef in the declarative way.

Chef worked, but now I'm starting to show my age. Um, I wanted to bring back to this whole idea of the state management component and these service templates that we spoke about. Um, because this is important of how we actually deliver this underneath the hood.

We use these templates, which are a collection of different templates that provide for different capabilities and we can support different types of deployment mechanisms. So we could use helm charts, we can use raw Kubernetes manifests, we can use operator based deployments. All put, and all the complexity of that management is put into these templates and we'll actually show you a demo time permitting of a very complex service that requires multiple dependencies being deployed at one click, including the cluster component.

We can then version control. We've spoken quite a bit about the version controlling this idea of immutable templates that then have template change that control the lifecycle of the chain of the template. Um, and really importantly, we can do dependency management because as much as we all love the fact that Kubernetes is eventual consistency, it's not actually always perfect.

You sometimes need to put things in a specific order. How do you Maintain the dependency graphs? It's done within those template chains.

Okay. Is is part of it built manually or are they Built They're built manually. Okay.

Oh, speaking of, uh, chains, how do you, uh, how do you implement the chaining? It's part of the function of the cordon operator and then we have a CR that defines the chaining. It's a YAML file again.

That's ya, that's what YAML definition. That's what I was Hoping it would say. Yeah, It's a YAML definition.

Again, It's all yams all the way down. Yeah. It's YAML all the way down, but those yams are held as, as CR objects.

Uh, Well you made it sound like you actually could do like GitHub integration or Yes. Some other source code control integration. So this is where the, the, the tool KTL comes into this.

So, so KTL allows us to do the GitHub integration, but you know, if you're using say Argo, um, Argo has a Kubernetes CR plugin and it'll handle that whole lifecycle for you. In fact, we use Argo for some of our deployments. Um, there is a component deep in the stack where we can actually reference GitHub directly as one of the configuration options for the raw Kubernetes manifest.

Okay. Um, and then we can also control what GitHubs you have access to because they have to be registered with the system. You can't just randomly create a Kubernetes object pointing at a GitHub.

Um, so that's how we also handle the whole air gap complexity. Where's Something like model versioning and, and data versioning fit into this framework. It's not quite state management.

So it sits primarily in the OCI registries for us. So OCI having an OCI registry is a requirement to do this. We obviously offer, offer our public OCI registry.

Um, we offer that for our, for our customers and our, especially our enterprise customers. Um, but most environments will deploy their own local OCI registry. Mm-hmm.

In fact, we recommend it. Um, and again, we have our own, but many of our customers have Artifactory or other, other OCI solutions. Um, you know, we even have ones using our competition's versions of it, which is fine.

We'll integrate quite happily. Yeah. OC as a standard, this is standards based.

Uh, when you were talking about GitHub, is it GitHub only or is it gi? Okay. So anything that can supply an API token speak the right language.

Okay, cool. It's interesting that one, I don't want to rabbit hole too much, but security and security management of all of these different provider endpoints is one of the things that we're actually solving for in a consistent way as part of accordant because this, the whole identity credentials management is really, really complex to get right and we're solving that. And that's actually driven by those, um, provider templates we've spoken about.

We spoke a little bit about earlier, but that's a deep, deep rabbit hole that we could use the entire two hour session for. Um, so we spoke about the inter integrated catalog. Kevin covered this in depth, but essentially we have a catalog of services.

Um, it's public catalog, accordant io, um, go and have a look through it. We're constantly adding to it. Uh, we have enterprise partnerships with a lot of the people in the catalog.

Um, but a lot of those are open source and available for anybody to use. We also have the ability to have private elements just within the customer catalog and still show this within their, within their operator ui. A demo.

Um, and I won't steal too much of Angelica standard, but this is culminating in this whole chain that we're talking about of building a cluster solution, putting the services on it, and actually launching an end user customer. Now we're showing you the, the, you know, the detail in the media, but from an end user customer point of view, all they would do is go to that portal and click the button and they would get the end result. Um, so over to you.

Alright, so in this next demo we'll deploy an inference cluster template with the Cordon platform. So here we have several cluster templates ready for a use. So we have, most fields are in simple dropdown menus allowing for customization based on your needs.

So let's go over to our GPU node. So you can configure your GPU nodes depending on the types of AI or HPC workloads you plan to use your cluster for. And then let's go ahead and skip over to which services you wish to deploy with the cluster.

So here we have a list of cluster services. For the sake of this demo, we're going to Go back to the, go back a little bit quickly. I just wanna point out something here.

So one of the things which I'm, I'm, and I'm sorry to interrupt this. One of the things I just wanna call out, you know, if you look at the, if you look at the template, um, the sort of standard templates we were showing you earlier, I wonder if we can get this just a little bit further back. There we go.

Our sort of standard templates, you'll often see there's only one or two types of worker types. What we've done here is because we need more complex and more options of different types of workers, that template has been created to have multiple different types of workers. And so in this deployment scenario, what we have is we have control plane for the Kubernetes cluster.

This happens to be, you know, part of the Kubernetes cluster. We have a hosted control plane model, which actually abstracts the Kubernetes cluster control plane, um, which I didn't talk too much about. But then we have different types of GPU nodes as well that can be configured within the environment.

Now this is gonna be deployed at AWS Expecting that from AWS directly? Yes. Yes.

Okay. So the, the, the point here is from a cost management point of view, if I just go and create all GPU nodes that it doesn't, isn't sane from the point of view of a cost management point of view. 'cause I have a lot of things that will never need A GPU within these clusters.

Now I can actually start to create different profiles within a single deployment of a single cluster. And that's quite a critical point when we start to look at, especially when we go into public cloud. But even in private and even in the, in the cloud service provider environments.

'cause they often have much cheaper infrastructure for running CPU only workloads and that can give them a whole different profile. Sorry, Angelica. Alright, so back to where I was before.

We're going to add the run AI services and its dependencies. So what we do is that we deploy this with the cluster so we can access the run AI portal via the cluster details page. And I can demonstrate this here.

ai portal here. So let's go ahead and navigate to our run AI portal. We can view the running inference workloads as you can see on screen.

So this just shows that in just a few clicks, we've deployed a cluster capable of handling AI workloads with supporting services to deliver GPU back to inference and more. This is a typical workflow, but we have the flexibility to deploy any type of services depending on the requirements. What we're trying to demonstrate there and, and in a very quick and simple way, is this ability to deliver a very complex set of services with very minimal overhead.

And once that's done, we can now maintain the lifecycle of those services. Um, and we can deliver that anywhere. It doesn't just have to be within a single service provider's Environment.

You deployed there, had both Kubernetes worker nodes as well as GPU nodes. Correct. Specific workload components would be allocated to the GPU nodes versus workload based on the requirements.

Exactly. Um, so underneath the hood built into the templating is the labeling, the correct labeling of those nodes. Um, and then that happens to be run ai, but the cube flow does it as well.

Um, run AI can then validate those labels and it will only provision workloads to the correct types of nodes. Um, that's also then automatically, because there's a, there are nine dependencies to run run AI in this particular case. Um, we need CER manager, we need the GPU operator.

We need, um, well the whole list is here. We can actually just pop back to it. Um, where's that list now?

Pause it here. Um, we need the ingress, we need Argo, we need Prometheus stack, we need K native, um, and the K native operator. All of those have to be put in and configured before you can even get run AI to start.

Now I'm using this as an example, but the, the thing is here because all of those are coming as a sim single cluster of templates, we call them a composition of templates. All the user has to do is select that I want that service and the rest of those templates are put into the system and within it takes full disclosure, it takes about 15 minutes to spin this up on AWS 14 of those is waiting for the AWS machines to start. Yep.

Um, this is including all the certificate keys, collecting the information, um, downloading all of the objects, um, if they're not local, and of course we can do it all air GA as well. So many of those just educational purposes, many of those container activities do not require GPU and they would be allocated correct to worker nodes. Ones that require GPU, the backend GPU operator, et cetera, et cetera.

Those would be specifically loaded on the GPU server Correct. Nodes. And you could have multiple different types of GPU nodes.

So you could have, for example, L fortys and you could select nodes with H two hundreds. And we can see that and you can select that, um, through the, through the, the UI In this particular case. Um, now we we're demo demoing this on AWS because we can quickly, easily get access to the GPU infrastructure.

Um, the same is true OnPrem. Uh, if you had qubit underneath the hood as your, as your virtualization layer or OpenStack or VMware, um, we can do the same or frankly bare metal. Now if we were, we were provisioning this to bare metal, it'll take a little bit longer.

Um, but the end result is, and in the UI you actually see a runner of, from the, from the end user point of view, which we didn't really touch on that heavily, but you actually see a runner with all the stages of what's happening so that you don't end up just watching a ball roll for 15, 20 minutes waiting for something. You can actually see what's happening. So if You're point of view, you're Using the SaaS platform for the management plane and then you've got bare metal or virtualization resources on-prem, what's that thing that's giving the connectivity from the management plane back down to the on-prem?

Do you have an agent of something that runs The operator? The, the provider for AWS, which is part of capi, is constantly reconciling. It does a, does a loop reconciliation, which is validating, well I'm not talking about AWS I'm talking about when you've got something OnPrem.

Same idea. It doesn't Have An, the operator, the operator's constantly reconciling. So it it, it talks to the APIs for the service.

Let's say it was cube convert underneath the hood or OpenStack. It's constantly reconciling against those APIs to validate the current state and providing that back, um, underneath the hood we have a number of, um, watches that watch the various layers and then provide that information back. And then that is then presented up into the ui.

Right. But you still, I mean you're behind a firewall on prem. Yes.

Okay. And you don't have connectivity from that management plane that lives somewhere out in the cloud directly through that firewall to something else. There's gotta be some kind of runner there that can actually build the bare metal.

Yes. If, sorry. Okay.

Now I understand the scenario you're talking about. If the, if the SaaS based control plane is outside mm-hmm. Yes.

There is a sec secure connection back. Um, the on-prem runner connects out to the controller. Okay.

And it uses, So that's deployed as a VM or as a Whatever As a whatever. You have OnPrem preem. So it needs a small Kubernetes cluster.

Okay. And, and a cube cluster, you know, cluster can be a single node. Right.

Got it. Runs out K zeros, um, distribution of Kubernetes, which has cos tron as part of it. And we can even have the control plane running in the cloud as long as we've got low enough latency, which we generally do in modern world.

Uh, we can actually have this Kubernetes control plane running up somewhere else. Mm-hmm. You know, some magical resource place and the worker can be running anywhere.

Um, in fact, one of our teams has just for the fun of it, done some testing and run Kubernetes clusters on an airplane with a control plane sitting in AWS and we all know what airplane, I think most of us spend a significant portion of our lives on airplanes, what the performance is like. Okay. Okay.

So if we jump a little bit into the future of where we, where we're going. Thank you. Today at Mortis, we're here, we're focused on IAS and the GPU platform as a service layer.

Um, and we're rapidly evolving through that as we're moving into the future. As we're moving into the future, we're moving into the AI pairs platform. And this is moving into the ingen integrated MCP stack and creating DAG and workflow and enterprise level security and structure around AgTech and AI workloads.

And we're doing some rapid work in that space. So watch that space. It's coming from us.

And I'm gonna close with this. I promised I would do a little bit of a shill here. Our vision for the future of infrastructure for applications is around contracts for performance of infrastructure.

Okay. What we're talking about here is instead of defining infrastructure from the point of view of I need so many GPUs, I need so many CPUs, I need, you're starting to set a performance criteria that is matched by the infrastructure application environments. And the infrastructure will take care of servicing the correct resources and even determining if you've asked for the correct resource.

That's the vision, that's the long-term vision for the future. And this is where we're driving towards with this whole vertically integrated stack, um, oh, sorry. Horizontally integrated stack to give those resource guarantees for a trip over myself here.

So thank you very much. Uh, Just a quick question on that. Um, how exactly do you intention intend to do contention analysis on these architectures Contention analysis?

Yes. If you're building out large cluster architectures, you have potential for, uh, contention points. So right now we're primarily heavily instrumenting the environment, uh, and collecting a lot of that instrument instrumentation.

We're starting to experiment with, uh, intelligent models that can look at that data. Um, some of them have been built by some of the big players and or public. Um, mostly it's by design.

Today we have to manage by design and have alerting for those environments. We're hoping to bring more intelligence into it in the future, but we don't have a really easy answer for it right now. Okay.

You don't have any method of doing modeling ahead of time to, With, with the work that NVIDIA's doing today for their specific reference architectures, we can leverage some of the tooling that they're doing. Um, right now I don't have a pre modeling mechanism. No.

What I am hoping to build in the future is a digital twin of the environment. Okay. But that's, I'm gonna be a very open, that's forward leaning.

Okay.