Mirantis IaaS Technology Stack with Shaun O’Meara

Sean Amara, CTO at Martis. We'll talk about the IS stack and we'll talk about a demo and an Angelica will support me on that again in a moment. Um, alright.

So what are the key requirements from an I stack at the end of the day? Some of these are fairly, fairly obvious, but I want to call them out 'cause we don't always think of it this way. They have to be manageable.

It seems obvious, but there are many systems that are put in that are managed through CLI or, you know, hands on keyboards, things that's not manageable. That's not repeatable management. You know, that means somebody in a data center plugging in a serial cable into the back of the switch, that's not manageable.

Mm-hmm. They have to be imp provable. We have to, and this comes back to what we were talking about with yaml.

We have to be able to constantly be able to improve the state through configuration or through extension of the system. Observability is really deliberately on the bottom left corner there because if we cannot observe our environments, we do not know what's going on and we cannot do anything else that's on this. Right.

They have to be controllable. Now, I know manageable and controllable seem like, uh, maybe the same thing but controllable here is talking about being able to force a state, not just change a state. Mm-hmm.

And that's where I talk about reconciliation, where I talk about being able to understand if something has changed and then I need to be able to manage performance. It's ai, the performance stack is so critical. The way that we connect the app to the infrastructure has changed.

We no longer abstract infrastructure. And this is a key thing that we have to build to into account. So that's making my point.

AI apps are performance bound beyond just GPUs. We have this thing, we come along, oh, it must connect to the GPU. No, we have to connect to the GPU.

We have to connect to the RMA network. We have to connect to the storage networks. We have to understand how the storage is distributed within that network.

How do the GPUs attach to block storage or, you know, object storage? How do I get data in and out of the clusters fast east, west and north south Traffic has to be dealt with. If I want to start sharing this infrastructure, how do I manage into the VM and into the Kubernetes components where whether it's VM or Kubernetes, to ensure that the PCIE is correctly allocated within those nodes.

Mm-hmm. This is starting to become really, really critical because if I take a GPU that's in one PCIE network and put a network card that's in another PCIE NUON node, I'm gonna have performance issues. Right.

So I have to solve for that. And that's what we're doing with the IS stack. We also have to solve for multi-tenancy.

Multi-tenancy isn't a simple thing. It is got many different aspects within an environment. It's not just a single, oh, I'm gonna take a node and make it.

What if I'm doing multi-tenancy in virtual machines and I want to split a node or split those eight GPUs between eight different customers? How do I ensure even down to the network layer that I still have multi-tenancy and how do I give options of cost value for multi-tenancy, strict multi-tenancy where you have the network and a node to yourself comes at a different value to multi-tenancy done as a cluster on top of a shared infrastructure. So we have to think of mul multi-tenancy at different layers of the stack and how it connects all the way through.

Uh, uh, go back to that for a second. Sure. So, um, it's what's missing from this is you don't show anything about the memory for both the CPU and the GPU, which might be entirely different memory banks.

Absolutely. And I'll get there in a little bit when we talk about the nuer nodes and things in a moment. But I, I had to create a balance of making this chart actually seeable versus Oh, okay.

What level of detail. But you're absolutely right. The memory isn't critical part of it because the memory is dependent on what GPU, what pneuma node, what CPU it's put within.

Okay. And how do we clean that afterwards? If You're managing things at the pneuma node web, do you also have the capability to attach the XL to this?

In theory, yes. Okay. Thanks.

Uh, I say in theory 'cause we haven't tried it yet, but we, we have the capability to configure it. So, um, again, we're just talking about multi-tenancy. I won't spend a lot of time on this, but realistically, these are the three models we typically see in the real world today.

We see full isolated or silo type multi-tenancy where it's either a single node, a single scaling unit, or a single whole super pod type model given to a single customer, managed as one entity. Um, obviously full isolation, strict isolation. It's the strongest isolation you can get, but it's also the most expensive way to do it.

And as we start to change the workloads and get to smaller workloads, this doesn't make as much sense and it has enormous management overhead. The hybrid or cell-based model, this is the VM model, this is the AWS, this is the Azure. Everybody who's sharing VMs across nodes.

Now this is where it gets more complicated because I could have a vm, I could take a couple of nodes and put them together. How do I create smaller units of service that I can offer customers? And this is probably, these two are the most common models we're seeing in GPUs and cloud providers today.

The shared model though, is becoming a lot more prevalent. It's, I'm buying just an inference service, which just needs a slice of A GPU meg or, you know, one GPU or two GPUs. How do I share that?

Excuse me. How do I share that in a committees cluster so that the user is abstracted from all that complexity? That's the more shared model, not strict isolation.

There's always risk in that, but for most workloads, most, you know, startups or people just experimenting with lms, it doesn't matter. So this is where those three models that we're seeing today, Do you help support, um, uh, service lifecycle and promotion? Not to mention, you know, pipeline and that sort of thing, um, to main to help maintain service consistency, support rollbacks, like all of that sort of thing.

That's all part of 'cause because what you just showed, I always, I, I look at it, I'm like, well, I'm what I'm, what I'm probably doing with probably, you know, 20 services at 20 AI related services at the moment. Not to mention more is I'm starting on the right and I'm moving towards the left. Yeah.

So this is where the whole Previous slide is what I was talking about. Yeah. Um, this is where the whole, um, templating and template chaining model comes into the, into the, into the process.

Because we have these immutable fixed templates that handle the deployment and the lifecycle of the components to move them forward or back in time. It's just a change in change in that immutable template object. So I can roll back and I can roll forward simply by program To change.

So Program change. Okay. Um, it's a lot easier to do config management when you have a small number of elements to config.

When you have tens of thousands of elements that are very similar, you want to start making those config elements immutable. Um, and there's some very interesting things being done. There's some new companies starting up that are actually starting to look more into that as a bigger, bigger picture at the moment.

We're working closely with them. Um, I'm not gonna spend a lot of time on this, but I wanna just show how it stacks up at a architecture level. And the reason I put this up mostly is just to call out the fact that this management layer sits outside of the stack.

It's the meta control plane that sits outside the stack. It is not in the data plane for the stack. And it allows us then to manage the various layers of complexity as you build up the stack.

We'll go, we'll go into a more technical diagram in a second. The key thing is we're able to operate both at the infrastructure layer, the compute layer, which is essentially the operating system and the host management layer. And then we start to layer on the different platform components on top of that.

And by defining all of that in one place, you can start to do that. The, the Lego building blocks type model, um, but all in one set of controls. So let's talk about the IAS stack.

So we start at the top of the stack. If you think about bottom up, top down, however you wanna think of it, the service management layer, these are the things that your end users actually interact with. It's the tools that run on top of the environment.

It's the cube flows, it's the ML flows, it's the web services, whatever might be running. Those are all defined as the service layer. We have our centralized control plane, which I've spoken about.

We have bare metal and network automation virtualization platform. And then obviously Kubernetes as a service is key to our whole offering. So if we stack up how this would look, and obviously this is a architecture very simplified diagram of the full complexity.

If you look on our website, there's a version of this that goes right down into the DPU networking layer and how we manage the dpu, um, and do network isolation at the DPU layer. But essentially we stacking it up. We have bare metal management that does the bare metal host management.

And if you imagine there was a layer of DPU underneath here that would also be managing the dpu. We have the storage management layer not depicted. We have the network management layer also not depicted for simplicity.

Then we have the KH fleet layer. And this is where we start to layer in those services and capabilities. So for example, we have a Kubernetes cluster.

We put vert on top of that. That vert allows us to deploy VMs in a very complex, aligned to the PCI and the NU nodes. And then we can start to put Kubernetes on top of that and you can stack up your service layers.

We even have the ability to connect these VMs and if we have dus in the stack to a specific interface aligned to only that DPU, again, we're coming back to the whole host tenant isolation capability. Um, and because we have a central point of control over the network, the host, the DPU in between, we can now align all of those tendancy and isolation capabilities right the way through the stack. So question on that, Jack Poller with Paradigm Technica.

So how do you enforce isolation and prevent breakout from a node to another tenant's environment? So there's the, depends on the layer of requirement, but primarily virtual machines, um, strongly configured virtual machines. And if you need deeper isolation than that, then a whole node to a tenant.

I'm sorry, I didn't hear the last part. A whole node to a tenant. Okay.

So you can isolate and We can isolate to a tenant. It's configurable. You basically have to, so this is where the templates come in.

We create a pattern in the templates and then you just request that pattern depending on what your needs are. Okay. So we, we take all that configuration complexity away and put 'em into templates.

And so the only enforcement though is, is just by placement of the workload. It's workload placement in there. Well, there, I mean on the nodes for example, we're configuring figuring the C groups so that there's no crossover between the VMs.

Um, there's a lot of host configuration components. We're also making sure, for example, that there's no shared pneumo nodes, um, across the VMs. So we can configure.

And that's effect is the next slide. Um, as we start to look about how we configure the virtualization space, we're able to match the physical node configuration into the virtual machine. Um, and then at the network layer, we're creating that same level of isolation.

So if I have the DPU in place as an example, um, the DPU present a VIF into the host, there's no host access into the DPU configuration. That VIF can only be linked to a specific virtual machine. Even if that virtual machine dies, I cannot connect another VM to that same VIF without re reconnecting it.

Reconfiguring it. Mm-hmm. There's no way to do that.

Now, can you break out of a vm? There are ways mm-hmm. We're protected against most of them to in this day and age, um, via configuration by KVMs capabilities.

Um, are we looking for that in our monitoring? Certainly, um, everything comes with an inherent, some layer of risk. But what we're also saying is, if you want true multi-tenancy, you can't put Kubernetes at the host layer and try and do V clusters or something like that because those are inherently unsafe.

Mm-hmm. Yes. Kubernetes by nature even well configured, Kubernetes is relatively easy to break out of into the host.

Right. So that's why we now layer up these layers of, um, and it creates a certain level of complexity that we're trying to handle. But that's how we layer this up.

Hopefully that answers the question. Okay. Let's talk about virtualization for a second.

So virtualization up till now, and I mentioned this earlier, has been covered by this idea of abstraction of the infrastructure. So a credit vm, it generally can move around that infrastructure fairly easily. Of course that changes when I'm running large language models that are bound to A GPU bound to a network interface bound to a memory bank or any ML workload.

It doesn't have to just be large language models. And that very much changes the paradigm. The old idea that I, I just start a VM and it doesn't matter more or less, maybe do some pma, CPUP has gone, we have to explicitly inject a configuration that matches the physical nodes configuration.

And so we've built a set of tools that sidecar the CVA capabilities that we're able to examine that in physical nodes, um, topography, the Ellis topper of that, and then match the VM requirement to that physical. And what you're actually looking at here is a Ellis Topo depiction of a virtual machines the inside of a virtual machine. And you can see if, you know, it's a bit difficult to see here, but essentially we've got the four nodes inside these nodes.

We've got 24 CPU calls, the NUER node, the PCIE and the GPU that it's plugged into the Infinity Band card, making sure that it's on the same PCIE slot as the GPU. So we can align that performance requirement. Um, we're working closely with, with Nvidia on this to, to prove this virtualization stack.

Um, we have a requirement to be less than 5% performance overhead across the entire solution. Uh, we believe we're even better than that right now. How Closely tied to Nvidia infrastructure We're prime, we're primarily on Nvidia infrastructure today.

Yep. Um, but we are supported for, um, the other major players as well. And we have certifications, um, with with A MD.

Okay. Um, but 19 9% of the customers we have today are Nvidia. That's market share.

Yeah. Um, but we can, as time goes, we can dely extend further than that. Is that true?

At every layer? So with an A MD solution, it's also whatever networking is behind that. Correct.

That's obviously not Gonna be, you're gonna come back to that whole provider plugin model. I can start to add those providers. Um, and we can start to even extend those capabilities and even have heterogeneous environments where we have multiple different vendors in the same environment because you're creating a contract.

Right? But contract is related to a set of templates and I, I keep coming back to the templates, but you know, once you, once you kinda understand how we're trying to handle the templates, this is where the whole power of the solution really comes in at the day. But this idea of contracts is incredibly important as well.

Um, and we'll touch back on that. So Hello again. Alright, let's see this in action.

So we'll start with demoing how our platform streamlines the onboarding and the provisioning process. This feature enables a platform operator to onboard bare metal in just minutes. So let's go ahead start this video.

This dashboard provides real time visibility into server status and resource use usage guiding users through the workflow required to initialize, configure and prime bare metal. For production ready AI, Kubernetes and virtualized workflows. Users can filter and sort through their server inventory, ensuring their right resources are always organized, accessible, and ready for use.

Whether that be by mostly recently you created servers, IP tags, or specific keywords. So with our platform, users can follow the lifecycle of their bare metal servers. So starting from discovered, available provisioned in use during the discovery stage, it inventories the hardware and onboard servers to be managed by accordant.

So once we're onboarded and available, we can remotely access our servers and carry out actions such as reboot servers, upgrade firmware, and other advanced configurations. So we're gonna start by provisioning a host. So we're gonna provision our operating system here.

Users can choose their operating systems, one to Red Hat or Windows users can also provide their own image, URL here. So below is the form to put important information such as network name, sshp, and user data. Advanced configs can also be done with the built-in YAML editor as well as shown here.

So once you're satisfied with your choices, all you have to do is just hit the provision host button and you're done. Provisioning infrastructure can be streamlined and tailored to your use cases. So let's take a look at our bare metal instance details.

See we have the status degraded and we have our statuses provisioned. And here is our details. We offer base level observability essentials such as CPU usage, memory usage, and disc usage.

For resource monitoring. Alerts can also be enabled so operators can stay on top of their resource usage. And here on screen we have the audit log where you can watch a timeline of your server's lifecycle view what actions are taken and by whom.

And then we're gonna move along to deploying a Kubernetes cluster on bare metal clusters can be created via the platform. And as you can see, we have several cluster templates to provision our cluster. We can choose between a basic editor with simple dropdown menus and presets available, or we can use the Yam O editor for those who want to add advanced config.

We can take the nodes that we've provisioned as well and assign them wherever we see fit. So here we have Kubernetes, small Kubernetes medium. And then let's go ahead and look into our observability.

So after creating our cluster, we can monitor and observe the cluster's resource usage. We click here again, we can see all of our cluster information. We have the monitoring dashboard, and then we also have the cluster topology.

So let's get a better look at that. So at our top layer we have our tenants below. We have our virtual machines enabled through qver.

Each virtual machine is assigned to a node in each tenant. Alternatively, we can view all this information about the base cluster VMs and our child clusters through this table. Vital information such as networking components pass through of GPUs, memory storage usage and os image are all stored here through the accordant platform.

You can provision a bare metal server. Can you actually show the network topology? Not only at the moment Through the accordant platform, we can provision a bare metal server, deploy a virtualization platform, and provision virtual machines on physical servers fast.

Is Q Vert The only virtualization platform supported? No, we support OpenStack, um, ourselves. So we can do a, a large scale OpenStack deployment as well.

It actually sits on top of, um, the previous generation at the moment. We're move moving it on top of accordant. Okay.

So we have a previous generation product, um, which is what Accordant was built off, which is MCC. Mm-hmm. Um, and we can deploy, um, OpenStack if you have VMware.

Yeah, we can support VMware. Okay. We can deploy vop.

Yeah. Uh, well we can, there are other virtualization platforms we can sort support, but we don't, we don't deploy and manage those ourselves. That's obviously, you've got a large scale VMware platform we can deploy onto it.

Uh, if you've got OpenStack, we can deploy onto it. There are other vendors in the pipeline at the moment. Um, do you wanna talk about those or should we Uh, We have a cut request for Nutanix one.

Um, and then I think there's another one that's mixed in there. But yeah, I, I mean that's the, that's the mode that we're in right now. Like, it's a very customer centric thing as we sort of find like a unique requirement from a customer, we build that out.

Gotcha.