The Advantages of Running vSphere Kubernetes Service in a Private Cloud with VMware by Broadcom

So, I'm Jerry Wolf. I'm one of the product managers on the, uh, on VCF. I'm primarily focused on that consumer experience.

Really, how do we take infrastructure and make it something that is self-service, something that we're able to expose to anyone who wants to be able to consume this infrastructure. Um, so first thing you've heard about VKS. You heard about the Kubernetes service specifically.

You also heard about some hints at some other runtimes that we have, not just runtimes, but some other services that we're able to leverage. The key thing here, and I don't think this is a surprise to anyone right, is apps require a lot more than just a runtime. They need more of a ecosystem that functions around them, so that when whoever the consumer is, whether it's a platform operator or an app developer, they're able to deploy their workloads and they're able to leverage the ecosystem that is around them to be able to move very, very quickly.

Right? That's the primary goal here. So we think about a, what makes a cloud typically successful.

It is really that ecosystem that exists around the different runtimes they provide. I mean, look at any of them. It, it, they have some core runtimes that they focus on, but then there's always an ecosystem of services that exist around them.

So I want us to keep that in the back of our mind as we kind of go through these slides. So, another variation of some slides you've seen already, but primarily the idea here is that BCF, it is about bringing a cloud to your private data center. And when we say cloud, we truly mean that.

We're talking about a space where you're able to deploy new workloads, you're able to deploy new services that's immediately extensible. It can, uh, be altered to meet the needs of your specific lines of businesses. And we want it to be able to be multi-tenant safe so that tenancy is built directly into the ecosystem.

So we heard about V-K-S-V-K-S is just one of the runtime services. We have three primary core runtime services that are, are out of the box available to any consumer of VCF. They're gonna be the VKS service that conform a KURE clusters.

It's going to be the VM service. And just call this out. VM service is just a new way to consume and manage vSphere virtual machines.

They are real vSphere virtual machines. Underneath the covers, they get deployed into ESX, but the idea here is to be able to allow consumers to interact with it in a cloud-like way with a lot of governance and policy placed around them. We have the container service.

Um, this is based off of a technology that we use called vSphere Pods, but it is exactly what it sounds like, the ability to deploy a container into a ecosystem without the need for a full Kubernetes cluster. I don't need management, or I don't need control plane VMs. I don't need worker VMs.

I just have an image. I want you to deploy it. I wanna provide some limited input around config maps, secrets, resources, and then I want to be able to scale it, but I do not want to have to worry about a Kubernetes runtime.

I just wanna worry about the deployment of that container. Um, Hold on. You're, you're saying config map secrets?

Yes. You're also saying this is not Kubernetes. So the idea is if you, if you go look at the public cloud, there are some container services that exist out there.

Mm-hmm. Um, they will typically have concepts like config maps and secrets, right? But they don't expose it in the same way.

What you'll see here is everything that we do from a platform perspective is done through custom resources. So a lot of the terminology is going to be similar. We're going to use them the same way.

But the intent is, is I don't want people to walk away with the idea that the container service is Kubernetes, because you're gonna immediately think it's a Kubernetes runtime. Therefore, I have these expectations of it, right? In this world, these, uh, containers, they operate in a multi-tenant ecosystem.

That means there's things that I inherently as a consumer am not going to be allowed to do. Not gonna be able to create cluster scoped objects, right? Because those are gonna be things that can't be multi-tenant controlled, right?

Um, I am going to want to be able to lifecycle manage things in a very specific way when it is Kubernetes based. So our intent is to abstract away a little bit of that concept, leverage the capabilities that we believe are extremely powerful in the Kubernetes system, but focus more on the outcome that you are desiring, which is the ability to deploy a container. And, um, we can, we can obviously in later times go into a little bit more detail here, but there's a lot Yeah, Go ahead.

So it's not Kubernetes or It is, but, but it's, but it does Smart CRDs. Yes. So under, okay.

Can you gimme one step layer deep Breath. Yeah. So that, so, and, and just a little bit, I have a slide that will describe some of this, but like the idea, you've heard hints of it.

We had the vSphere supervisor, the vSphere supervisor is our declarative, API surface it is Kubernetes. It's made up of three control plane VMs. And then what we do is we end up turning all of the ESX hosts into worker nodes.

Um, but that, when we say worker, no, it's really just worker nodes for these container, the container instances. Everything else is gonna go through the backend technology like DRS and vSphere ha and those types of vSphere, uh, structures in order to deploy and manage virtual machines. And so that, that declarative API surface it is the way we are going to expose all these custom resources, custom re resource definitions, and you're gonna hear a lot of that there.

There's two layers. Okay. So I will refer to the declarative API surface when I'm referring to that service.

I'm talking about the platform, and I'm talking about the cloud services that are deployed there. And then anytime we talk about, uh, conformant Kubernetes, we're talking specifically about clusters deployed via the VKS managed service. Can You just for a second, talk about like what customer demands drove that compared to, because it seems like the industry is a hammer now, right now for everything Kubernetes, right?

Yeah. So a lot of it came down to the fact that we, we believed in the model of we need to be able to provide managed services. That's at the core, meaning that I want to be able to make it so that someone can take a service, a, a VI admin can take a service, bring it into the ecosystem, and then expose that service in a very common way.

The common way we decided was custom resource definitions. You'll see why we see a lot of this benefit here in a second. Um, but by doing that, it immediately started to align with ways that people expect to consume workloads.

Then we had the added advantage that it's declarative in nature, it's desired state based. And so there's a lot of synergy that we can get out of that ecosystem. Um, and so really when we kind of dive into this, every single one of the services you see here are consumed through this same model.

It means that unlike previous vSphere ecosystems or VCF implementations, it's always been, Hey, we have a cloud platform. That cloud platform has say, six, seven services that you can consume on it, but they all have their own APIs, which means you have to go interact with 10, 12 different APIs. You need to be a subject matter expert in all those APIs in order to weave them together in order to get a cohesive, I promise, this is my last question, but, you know, I lived through the, I I was, I saw the company Docker and mm-hmm.

And the swarm and the rancher is this sort of born outta that approach that we wanted lighter weight, but still containerized. That's, that, that is definitely some of it. Yeah.

And, uh, we, it's kind of interesting 'cause when you look at it strictly from Kubernetes, from Kubernetes sake, right? There's a, there's kind of a doctrine that exists there that states, Hey, if you're gonna do it, go all the way in. Yeah.

Yeah. But the reality is like when you go talk to customers and you see kind of where they're at in their, like, transition from, I am doing like legacy based deployments. I understand containerization, but I'm not quite ready to go full cloud native.

Yep. Like, there needs to be an intermediate step. And this container service definitely fills that space.

So I'm, I'm trying to track all this and figure out the hierarchy. You said there was three services that sit on top of VCF. Yep.

But did you say the other four were are Parallel? Parallel, yeah. So the, I will draw a line.

So if we were to look at the screen, we have the three, the three first boxes, so v, k, s, VM service, container service, those are the runtime services. Okay. Then to the right of that, we have storage, we have networking.

Those are also like core capabilities. They are direct integrations with the vSphere ecosystem. And that, when I say the vSphere ecosystem, what I mean is it's interaction with like vsan, it's interaction with ESX, vCenter, NSX, those technologies, and we will see a description of this here in just a second, but the idea is like, we're bringing this capability so that you don't have to understand the infrastructure that exists underneath it at all.

We wanna complete abstraction. And so we have those kind of core infrastructure capabilities, but then we also also have things like, I mean, backup and restore. It's something that's necessary in the ecosystem.

We have it both from a platform perspective. So I need to back up and recover the entire platform. We also then have the service that can be exposed for consumers to be able to leverage backup and recovery.

Um, and then kind of going into this a little bit deeper, we have things like harbor, our OCI image registry. We have Argo cd, we have, uh, our secret store. We have a, a lot of operators that we're gonna see coming into this ecosystem.

And one of the key things here is because we have this single declarative API service and it's multi-tenant safe, we can now leverage the Kubernetes operator custom resource definition pattern to create operator factories that will create those dedicated instances for consumers. So now as an admin, I can introduce a new service to the ecosystem. That new service will then be able to be, uh, leveraged inside of those name spaces directly.

So the three to the left are runtime and what did you call the other Three to the left are runtime. The ones the right would be, uh, primarily core infrastructure and add-on services. And wouldn't they, why wouldn't you put those below the, the run prob Probably should have.

Okay. Mm-hmm. Yeah.

So there's another, uh, there obviously lots of iterations of this, right? And there's a couple of, of views of this where we have, we have runtime, and you've seen it in Will's session. We had runtime kind of on the top, and then we had service integrations on the bottom, or the core infrastructure integrations on the bottom.

That was networking, load, balancing storage. And then on the right side, we had a bunch of add-on services. Those add-on services of those, bring them into the ecosystem to extend the capability.

Okay, I think I got it. Alright. So just the visualization.

So I told you that every single resource that's deployed into this, into the VCF ecosystem is consumable via Kubernetes APIs. So every time a service is introduced here, it's introduced in the same model of API version kind. I can do a cube, CTL, explain on a resource.

I could see all of the different capabilities that are available to me. Um, and I can use those native Kubernetes constructs to be able to explore what's available. So to the CLI ex example, right?

We have Cube CTL, it works here. Just it does, right? Um, we also have a, uh, the V-C-F-C-L-I, so the V-C-F-C-L-I is modeled a lot like what the public cloud CLI look like, where there will be ACL I, it'll allow me to log into the infrastructure, but then there'll be a set of plugins that I can leverage that are specifically designed to interact with these resources.

So I can deploy virtual machines and explore virtual machine constructs. I can deploy object storage and explore those constructs. Um, but a key thing here is just understanding that every time that we bring a service into this ecosystem, it's going to follow this model.

Because when we talk about the VCF automation experience, where I'm this VCF cloud, uh, where I'm in the VCF cloud, and I'm gonna go into and consume the infrastructure, a big problem that we have in the private, in private data centers is when I add a service, how do I tell people that a new service is available to them? I have to kind of evangelize that, that solution and get people convinced that it's something that is available to them here. It just shows up in their experience, whether it's COI or the, uh, the ui, it, it's there as soon as I deploy it.

Is, is there a reason you didn't have the container service on that screen? Um, is that would've been No, no specific reason other than underneath the covers is just using standard Kubernetes constructs. So version one pod as an example, right?

Use deployment stateful sets, all of those things. Well, and I'm really confused on how these two map together. 'cause the Secret store service is on, the other one was above the other.

Yeah. So Yeah, I, I'm, I'm just trying to figure out how these relate to each other and what supports what and what's. Yeah.

So a good way to think about this is, um, in, in a vSphere namespace, I can deploy these runtime. So the three run times mm-hmm. Those will be where my workloads are running.

Mm-hmm. They'll be able to leverage services that are in the ecosystem. Those services are the services that are around this, right?

I can leverage Argo CD to do those deployments. I can leverage the storage service via persistent volume claims, or as was announced during explore. I can use integrated file service.

I can use object storage, which will also be exposed through those. And so the idea here is as I'm deploying those runtimes, I'm able to use the same resources through the same API. And so it's, I wouldn't really think of this so much as like stacks, right?

It's Lego blocks within the same bucket, and I'm picking those Lego blocks up and I'm building out my application, my workload requirements, and then underneath the covers, we are going to handle that experience from an integration perspective. And you left Argo off the previous slide because, Because it's just a lot to add into the slides. So Argo here really is just that orchestration tool, right?

So how do we, how are we going to have a manifest based deployment model, right? Because that's what all this is gonna be based off of all yaml. How do I then take that and put it into an automated solution?

And the key thing here is like Argo, you've seen an instance of it deployed, but the reality is that Argo instance is backed by a custom resource, and it's not just backed by a custom resource for the Argo instance itself. It's backed by custom resources for all of the key Argo objects that you'd want to be able to interact with. With the intent being that I can fully automate and repeat that deployment without the need to actually go into the individual application.

All right. So, uh, we'll, we talked about this just a little bit, but I just want to hone in on it one more time. Is, uh, underneath the covers here we have that vSphere supervisor.

The vSphere supervisor has a construct called a vSphere namespace. Like I want to emphasize that while this is like a namespace, it is quite a bit different than a namespace. It acts as a true hard tendency boundary.

It is where we isolate compute, we isolate storage, we isolate network. It's where we manage quota, meaning that we can assign quota at the CPU level, at the memory level, at the storage level. And when we get to the storage level, we actually can go even deeper than that.

We can go down to storage cloud variation. So we're able to do that, but we also have the added advantage here where I can use the vSphere namespace because it's a tendency boundary to do content restrictions. So I could say, Hey, this specific vSphere namespace that is, uh, owned by my, or is gonna be used by my dev team, that dev team, they're only gonna be able to be allowed to deploy, um, the newest versions of VKS.

So we're going to Kubernetes release control here. We are gonna be able to do VM image control here. 04 as an example.

Um, we also have VM classes. We didn't really touch on it too much here, but just like in the public cloud, there are t-shirt size variations of VM infrastructure. Because the primary problem here is a lot of developers don't really understand hardware.

They don't understand infrastructure. They shouldn't, they should just be able to understand what, how many CPUs do I need? How much ram do I need accelerators?

And then be able to pick it from a list. Go ahead. This is all VXLAN based still, right?

Um, say that one more time. Vxlan The NSX. Uh, yeah.

So the, this is all NSX based and That's VX still vxlan. Yeah. So we do, we do support VDS based networking.

Okay. Um, but there are, there are experience issues in that space. 'cause VDS is not very cloud-like, it doesn't expand very well.

Um, and so when we talk to people, we specifically say, Hey, pick up the VPC 'cause NSX plus VPC, you get a cloud network model, meaning that I have these subnets that can be referenced based off of what kind of route ability I need. And I, as an app developer platform engineer, I don't really need to know the details. I just need to understand the outcome I'm going to get out of those networks.

All right? Uh, and this slide I probably should have led with, uh, according to the conversations we had. But, so the idea here is we have, we have two things that we get from this model of a declare declarative, API surface that is Kubernetes base.

It is r vSphere supervisor, that's what it's called under the covers. Um, two big things we get, one of them is we get this abstraction from infrastructure. So one of the big difficulties of running Kubernetes in a a virtualization layer is like, how do you control the versioning and interoperability of all of the different integration endpoints, right?

Whether it's networking, whether it's storage, whether it's compute, whatever it is. Like you have to manage those things individually. And in this world, in BCF, you just worry about the supervisor, the supervisors, the thing that's going to interact and interoperate with all of those underlying systems.

So that's one big advantage you have here. So now you just worry about the versions of Kubernetes. You want to provide from a consumer perspective, you worry about the services you wanna provide, and then all of that infrastructure interoperability happens through the supervisor layer.

The second piece that we get here is, there is no need to have pri privileged access to any of the infrastructure resources. V four namespace is multi-tenant safe. It's multi-tenant safe.

Because we do not have to provide any username or passwords, any service accounts. We don't have to worry about any of that integration. All we do is we use the Kubernetes watcher pattern where we watch for requests to hit the API server and then our controllers behind the scenes watch for those requests and then go execute that on the behalf of the user.

And so this model, we call the para virtualized model, and it is how we can continuously will develop and deliver this abstraction as well as this security model. So a quick demo, um, I have a much longer demo, but we probably do not have time for that. Um, it's a variation of what will showed.

Um, we have a three-tiered apps are kind of lame now, but, uh, it's a good example of how, how you would use an ecosystem like this where I have virtual machines and I have Kubernetes. I need to run them together. Like how does that look like?

Why would I want to use a declarative API surface? What are the advantages of using a single API surface to do this? So we have a simple three-tiered application, and the basic breakdown here is we're going to deploy a MySQL database inside of a virtual machine.

We're gonna deploy that virtual machine via the VM service. So, um, Just some quick highlights there is the VM service is going to it, it's gonna be a standard V vSphere vm. It's going to be deployed and managed via this Kubernetes interface.

Um, and it's going to use Cloud and Knit in order for us to actually configure that. My SQL configuration. So we're gonna deploy my SQL there, we're going to seed the image or seed the VM with some data, and then we're gonna actually operate that database.

We're gonna deploy a VKS cluster. The VKS cluster is then gonna have a front end and a backend, um, application deployed to it. The the difference there is that the backend application is gonna be responsible for communicating with the database.

And the front end is just that. It's gonna be the UI that's gonna be responsible for communicating request back to the back end. But we're gonna leverage all of the services you see up top.

So we can use the secret service to store our database configurations. We're gonna use the volume service to have our persistent volumes. Um, we're gonna have our network service providing the network capabilities.

And this is oftentimes kind of the unsung hero here is network in this. You're gonna see it just happens, right? We just, we don't have to worry about it because it's just part of the platform and we get to be able to leverage all of those capabilities.

And then finally, we're gonna use the VM image service, which is where we're going to be pulling that Ubuntu image, and we're gonna be using that with our VM service. So we're gonna see all of this kind of come together. So the first thing we're, we're gonna make some assumptions here.

You already seen how Argo CD looks like. It's already deployed into my vSphere namespace. I have already taken all of the deployment manifests and I've placed them in source control.

When I say deployment manifests, I mean my VKS manifests. My, um, VM service manifests, my application manifests. So the front end and backend deployments, I have placed all of that inside of a Git repository.

And I have also placed in that GIT repository in Argo application yaml. So here's my Argo Yam, uh, Argo application yaml, and it's going to be an apps of apps relationships. So I'm gonna deploy this and it's actually gonna deploy three other Argo CD applications.

So one of them is going to reference our, uh, our VM deployments for our database. Second one's gonna be our VKS cluster. And the third one is going to be that front end and backend deployment inside of the VKS cluster.

So here we'll see an example. This is the VKS manifest. And it's literally the manifest that was inside of that UI being made on the right side, we distorted in source control.

So here we pasted the, uh, application yaml, and now we're gonna see Argo do its magic, right? So it's gonna deploy those three applications and our desired state is going to be achieved here. So we have our VKS cluster.

Um, if we were to kind of dive into this, we could actually go in and start exercising some of those GI ops models. So we could go into our source control, we could scale the number of worker nodes associated with this VKS cluster, and those worker nodes would then be able to be rolled out. And if someone went in and tried to adjust or control something locally on this cluster or make changes to it, our Argo instances go automatically go override it.

So we get this really strong declarative, uh, experience. So that is the end of the demo. So I have four minutes.

I have a, like I said, a much longer demo if you wanna see more. Go ahead. So the, the hammer nail with like Kubernetes is, the upside of that is maintainability.

This seems like it would be very difficult to troubleshoot, particularly if, if I'm running my, my SQL on a vm, I might be running separate containers and container services, and then I might be running VKS. It just, to me, screams of a sort of a, a maintenance and supportability and, uh, troubleshooting nightmare. Yeah.

I mean, anytime you are deploying a lot of different application form factors, there's complexity that comes out of it, right? What I'm not recommending is that you do that. What I'm saying is if you have a need to do it, okay, you can.

Okay. And you could do it through the same API service. And the reality is like you could even have some intermixed modes here where you use that namespace construct to be a boundary for different types of deployment models, right?

And the key here is it's, it's all available. You use it where you need to use it. And the key here is, as will showed, is all of this is then ported up into VCF operations and you're able to see all of the capability through the same way.

And then The counter argument that the upside is because you're using all the, the API, the CRD up API, then at least from a discoverability, everything's the same. Is that, yep. Okay.

Good. All right. Thank you.