Deploying Your Apps in vSphere Kubernetes Service is Easy with VMware by Broadcom

Um, I'm Will Royo. I'm a solution architect, uh, within VCF. Uh, I'm gonna walk through kind of deploying applications to VKS, not just applications though, so kind of be a flow of what it's like for a platform engineer to build a Kubernetes cluster that's ready for an application to be deployed on, and as well deploy an application.

Um, so it's gonna combine kind of two personas, really, an app developer and a platform engineer. Um, just a couple slides before I get into a demo. Um, so Timmy talked about this in session one.

Uh, and I just highlighted some of the services that I'm gonna use here, uh, for the demo, which is the VKS service. So the Kubernetes service, the underlying infrastructure services are kind of just there, um, because like their core components to building a cluster, right? I need a load balancer, I need some storage, I need networking, all of that.

Um, and then I'm gonna add some cloud services. So I'm gonna deploy Cert manager, uh, also Argo cd. So Argo CD is gonna be used in this flow for not just like, you know, create, we don't want to just create a cluster.

We actually wanna bootstrap that cluster with a number of components inside of it. So we're gonna combine kinda a Terraform workflow plus an Argo CD workflow. Um, then on top of that, there's some standard packages.

Uh, so I didn't highlight anything on the bottom because as Timmy mentioned, the those services are kind of auto deployed. Like these are things that we need in a Kubernetes cluster for the Kubernetes cluster to be functional. CNI.

Um, you know, we have some other things like pin AED for auth and stuff like that, but on the top, uh, I'm also gonna be deploying Prometheus Telegraph, Istio external DNS and Cert Manager so that my application has ingress when it comes up. So that monitoring is already kind of wired in, uh, for us out of the box. Um, external DNS so that I get a host name, um, and cert manager so that, you know, all these components can get certs and things within that.

So those are kind of the core components that I'm gonna using from this stack. Um, I'll pause there for any questions so far that make sense to everybody. Cool.

All right. And then just before I jump into it, uh, talking a little bit about APIs, because I'm gonna be using Terraform here. That's not the only way to consume these things.

Um, Timmy mentioned it, uh, we'll talk about it in the third session as well. But everything going through kind of VCF and, and the consumption interfaces here are gonna be declarative APIs. So you can consume those with cube control, you can consume those with Terraform, uh, Ansible, GI ops through Argo cd, all of these types of things.

So I'm gonna show kind of one way of doing this, but if there are questions about, Hey, could I use this tool or could I do this thing, just ask. Um, but I'll try and call out like, Hey, you could do it this way as well. Um, so this just kind of, this slide just kind of highlights those different APIs and the ways that you can consume them, you know, through catalogs.

So for some folks that have come from using VMware for a long time, there's this concept of blueprints and like a self-service catalog, those catalogs can be used to generate Kubernetes resources that deploy things like Kubernetes clusters. Um, so you can still use that flow. You can also use a declarative, API directly, right?

That's things like cube control or GI ops and those components. Um, and then there's Terraform as well. So, um, Timmy mentioned it also, but you know, because these things, these, these objects and the services we're deploying are Kubernetes resources.

The Kubernetes Terraform provider works really well for that. Um, but there's also some other providers that provide access to different layers within that API. But really the main point of this is that like, whichever way you're coming in to consume this platform and how you are comfortable as a platform engineer, um, the API is flexible for how you want to consume it and use it.

Um, any questions? Cool. Alright, let's get into the demo.

I usually do live demo stuff, so I've recorded this just to make sure that there wasn't any wifi issues or anything like that. Um, so we'll run through it. Um, all right, so what we're looking at right now is, uh, I'm kicking off some Terraform, so I'm gonna run a Terraform apply, and I'll talk through what this is gonna do.

But this is the only command I'm going to run this entire time in the demo, right? So it's gonna say, Hey, I'm gonna create a bunch of resources. I'm gonna tell it yes, I want to try and remove that thing.

There we go. Um, and what we're looking at above, I'll start to go through as well, um, which is the actual code for this. So this is gonna go create essentially a supervisor namespace, um, an Argo CD instance.

It's going to wire that namespace back to Argo cd. It's gonna create a Kubernetes cluster using VKS. It's gonna attach that cluster to Argo cd.

Um, it's gonna bootstrap that with a number of core applications. Um, so things like, um, is DO and all those packages that I was talking about. And then it's also going to deploy an application.

So you can see I'm kind of putting a bunch of these pieces together, probably in a more enterprise workflow, you'd separate out like the core underlying platform engineering components from the actual application deployment. But just to kind of simplify that for a demo purpose, I've put the application deployment in here as well. Um, so that's what's, what's essentially happening when I run Terraform apply here.

And you'll notice that I had a few modules within here. So I'll dive into one of the modules behind the scenes, um, which is going to be the cluster module. Um, so yep.

So the cluster module itself is just, I'll pause it there. Um, this is just using the Kubernetes provider for Terraform, right? Um, and you'll see the YAML in here looks very similar to the, this isn't yaml, this is an HCL representation of yaml.

But, um, the YAML that is defining this is basically just a Kubernetes CRD that is a cluster, right? And this is what you saw in the UI that Timmy showed. I'll show it here in a bit.

Um, but that YAML can be used with Argo cd, can be used with Ansible. Again, Lummi, all these different tools of how you consume an API. Um, and really I think kind of main, one of the main pieces around that is customers, enterprises, organizations, all these have the way that they consume infrastructure.

Um, they don't necessarily want a new tool that's like, Hey, you brought your new orchestration tool in. We have to use that. We wanna be able to meet the customer where they're at with you already have a workflow for how you provision infrastructure or how you deploy apps.

Let's give you an API that's consumable in that way. So I just really wanna highlight, I'm using Terraform here for this, and many people do use Terraform, but this can easily be consumed through any other infrastructure as code tooling. Pause there.

Questions make sense? Awesome. All right.

Um, so from here, uh, while this Terraform runs, I'm gonna go jump into this is just kind of highlighting, again, kind of all the way down through all the settings that are available through this. Um, so let's pull up a browser. We're gonna go log into we VCF automation.

Um, so I'm gonna pause again. Uh, VCF automation, uh, was not, I don't think we really fully talked about it in the last session. Um, but if you think about kinda like public cloud, having a console that you log into, which all of them do to go, you know, access the UI for different services and all of that, think of VCF automation as that core interface of how you're gonna go consume infrastructure as a tenant within that, right?

So as your platform engineer logging into this, um, it's gonna give you the right permissions and everything you need to see kinda what you're deploying. So what we can see here is that a namespace has been deployed. Um, it's that W-W-Y-V-T namespace, um, with a large namespace class.

Um, so that's what Terraform did initially. And a supervisor namespace in this case, um, is kind of a construct within, um, the VCF platform that allows you to separate out where you're creating other services. So like the VM service, the Kubernetes service, those components, Kubernetes clusters and virtual machines can get deployed into a namespace.

And that's kind of a tenant boundary. It defines networking boundaries, it defines, um, quotas and things like that. So as a platform engineer, you can come in and you can say, Hey, I might want to give out namespace to dev teams.

And then you can quota them within that so that they're not consuming all the infrastructure. Um, or you can self-service this out. So it gives you some flexibility.

These are not Kubernetes namespace. These Are so great question. Great question.

It's, it comes up a lot. So the, one of the backing components here is, is the supervisor cluster. That's kind of the management control plane, because again, all of these APIs are Kubernetes CDs.

We need a Kubernetes cluster to reconcile those, right? As the API, it's Like a utility cluster. Exactly.

It's a management cluster kind of to be considered. So this is a namespace within that supervisor cluster, that management cluster. So it is a Kubernetes namespace, but it also has a connection into vCenter behind the scenes that uses essentially a resource pool to quota that infrastructure.

So like you've got actual physical hosts as ESX hosts, right? That have your compute. And when we deploy a supervisor namespace, we can allocate some of that compute to the namespace.

So it shows up as a Kubernetes namespace, but it's also backed by infrastructure quoting to say that if I deploy a cluster into that namespace, maybe I'm only allocated 60 gigs of ram or something like that, right? And it will prevent you from going over that, Right? So how does that then, that's a good, great segue into my next question, which was about resource limits.

Are there some automations in here that help platform engineers or the operations folks determine appropriate resource limits? 'cause the worst thing you could do is not put any resource limits and then just tear the whole thing down every time you run something. Absolutely.

So the namespace class concept that's kind of sitting here, yeah. Um, those are predefined. You can define as many as you want.

And they define memory storage, um, and CPU limits that you can put on that. So you can say like, you know, maybe a dev team's coming in that has, um, a really big workload, okay, we're gonna give them an Excel class, and then you can scale that up and down. And as, As a platform engineer, do you warn me if I've got too many extra is deployed into one spot?

Uh, yeah, you can see kind of like an overall, um, I can probably, uh, I was gonna jump to my live environment. I'll show that. Okay.

Me here in a bit. But yes, you can see like the total consumption over that to make sure you're not bumping up against limits and all that. And then also, because we're surfacing all these metrics up into VCF ops, you can also set up alerting and, and, and things there to say like, Hey, we we're really hitting close to the limits on, on this environment and stuff like that.

So, yep. Um, let's see here. Okay.

Um, so, um, after we kinda look at the namespace, we'll also look at what's deployed here. Um, so I'm jumping into the namespace now. So I've gone into a project, I'm going to that namespace, and we'll see kind of a high level view.

I've got a Kubernetes cluster deployed, um, and we can see that that is the actual cluster that's currently being deployed by Terraform, but I'm looking at it from the UI perspective now. Um, and within that, if I open it up, um, we're gonna get a number of, you know, details about it. I can see the YAML for what's deployed.

Again, because this is a Kubernetes resource. This is what the actual live cluster, uh, manifest looks like. Um, and then within that, lemme see, I think I scroll down.

Okay. Just make sure I'm going to the right place. Okay.

It looks like, uh, the Terraform completed. So from here, we're going to go look at, there we go. Sorry about that recorded demo.

Um, so this, uh, is gonna show creating a cluster. Timmy showed this. Um, so I'll just touch on it a bit, uh, because it's only like 20 seconds here.

Um, but I'll, I'll talk about slightly something slightly different, right? So, uh, it has the AML on the side. Timmy talked about these kinds of things, but what I wanna highlight is, because we're using Kubernetes behind the scenes for these cluster deployments, if maybe I'm not comfortable with GI Ops yet, maybe I'm not comfortable with Terraform or something like that, I can come in, I can create a cluster through the ui, and then I can eventually move to managing that with GI Ops or something like that.

Because that cluster state is behind the scenes inside that supervisor cluster that I was talking about. You can go edit that directly on the supervisor. You can adopt it with Argo cd, you can manage it through the ui.

Now, obviously, if you try to manage it all three ways at the same time, you might end up with some conflicts. But the point is that you don't have to like say, I'm gonna go delete my clusters so that I can start using Argo cd or something like that, right? Um, okay.

Uh, so let me jump back a little bit here, uh, within this, okay. Uh, the next piece of this is gonna be that after this gets deployed, it's gonna be automatically attached into our cluster management. So within cluster management, um, this has some things like policy.

So what we're looking at now, um, is I have a project. That project is a tendency boundary. My cluster got deployed into a namespace within that project, and I've set up some default policies within this.

One of those policies in this case is a restricted policy. I put it in audit mode just so it doesn't block anything right now, but that's some default admission policies. So block privilege containers, um, block, uh, you know, network, uh, node network, all that host network, all those kinds of things.

So when a cluster gets deployed in this project, it's automatically gonna inherit that because we're also deploying gatekeeper automatically into those clusters. So you're getting this native policy, um, and you can do other policies in here, like custom policies with, um, gatekeeper and OPA image registry policies. All these kinds of things can happen through this.

Um, but I'm not having to deploy these policies on every single cluster. They're being inherited kind of at the fleet level. Um, so that a, a platform engineer can set these up ahead of time and not have to worry about like how they get applied or if they're going to get applied.

Does that make sense? Any questions there? Good.

Cool. Um, all right. So this is just showing kind of a look into that policy.

Behind the scenes. We can see that there's some, uh, insights that are being shown. This is a different insights view.

So it's actually gonna surface up all those violations. So if I have it in audit mode, for example, I could go see all the clusters that are failing, uh, for specific reasons, uh, and then, uh, you know, go maybe apply more strict permissions. Um, and then just a, a high level view of what you're also getting at the fleet level is a view into that cluster.

Um, the nodes high level visibility on how much resources are being used, the namespace, the resources within that, um, and so on. So that's just a quick kind of view into that, uh, fleet management concept there. Uh, so if we go now, look and see what else was deployed with this, um, Argo CD in this case was also deployed.

Um, and I'll touch on this for a minute because it's important about how this got deployed. Um, Argo CD in this case is a cloud service that we provide, meaning that this is deployable not just a single Argo CD instance, but you can consume it on demand, right? So within that namespace that we deployed, I also deployed an Argo CD instance using a Kubernetes CRD.

So essentially just a small amount of YAML that said, go deploy Argo CD into this namespace. That could be namespace scoped, depending on how your organization is set up or something like that. You might have a centralized Argo cd, you might wanna do them per team, something like that.

But you can consume those however you choose, because again, this is just simply deploying a Kubernetes resource within your namespace. So Argo CD was deployed, and then what we do here is we wire this up to not only the cluster, but also the namespace. So these two clusters that are now part of Kubernetes, or part of Argo CD are the cluster VKS cluster.

We deployed as well as that supervisor namespace. So now I can use Argo CD to deploy more clusters if I want. I can use it to deploy load balancers, I can use it to deploy VMs.

All those types of things can now be done through Argo cd. And I can also deploy applications to the clusters that I deploy with Argo cd. So it gives me a bunch of flexibility in being able to control everything with GI Ops.

If I want pause there. Questions? Yeah.

A little unrelated, but I, I haven't heard anything about open SSF or Sig store. And so since you're talking about Argo CD and application deploying, now, how much of that is out of the box? Or is that something you have to go back to the tan Zu version versus the From AIG store perspective?

Well, starting with open SSF, like in other words, sort of signing of images Yep. The artifacts. Yep.

The process of making sure that all that, um, I didn't, I didn't see anything in any of the images that imply that's out of the box. That's correct. So from what we're doing here doesn't really, uh, involve the, let's call it like that initial portion of the supply chain of like building the images and, and those components, right?

That was a component of Tan Zu. Um, right now in the VCF stack, there's not like an out of the box. Let me go build a container for you.

Uh, Timmy might want to jump in here as well too. Not, Not to be, but that's pretty important, right? Yes.

But also, I would say that a lot of customers already have those things, right. Um, on how they're building their images. But Timmy probably chime In, feel like we feel that a lot of our customers have harsh parts of this area figured out.

But one service we do ship as a part of our stack is our registry service, which is built on top of Harbor. Okay. Yeah.

Yeah. And Harbor, uh, does all kinds of things around, you know, image management, like we can, yeah. I mean, yeah, we can do, we can do a great deal of, of, of, of work there.

Not to be harbor On this subject, but, but the point is, how much integration do I have to do as a client? Because you, you, you've supplied so much Yeah. In this architecture, but then it sort of scares me a little bit that, that I have to do a lot of extra legwork.

Like you, you say, oh, the customer does that. I'd want you as my provider to gimme that sort of out of the box. Yep.

I want, I want open SSF like glaring support from Day one. Yeah, absolutely. And what we're doing is we're working on, you know, extending each of our services as a part of our cloud.

Okay. Outta the box today, we have our registry service, which can help complete part of that story. And we'll probably look to extend that over time.

Okay. Cool. Thank you.

Alright. Um, okay. So kind of moving along in this demo, you'll see that there's a bunch of stuff deployed via Argo cd.

All of this was bootstrapped through that Terraform deployment. So we deployed essentially a, um, I'll pause it really quick. Uh, we deployed an application set, uh, which is an Argo CD concept, and it installed some ingress, some observability, kind of all those things I was talking about.

So all those apps just showed up automatically. And then also we deployed the music store app, which is what we're looking at here. So again, kind of combining both the platform engineering components of this, which are installing Istio Cert Manager, external DNS, and all of those pieces so that when we actually go to our app, we get a URL, we get a DNS name, and we can actually go to our app directly here.

So with all of that, essentially a terraform apply it created everything and installed the ingress, it wired up DNS, it deployed the application, all of that in, you know, a few minutes essentially while the, uh, Terraform was running. Now what we're gonna do now is jump back kind of into VCF automation and take a look at, uh, some behind the scenes. So one of the other things that, uh, I did here is created an IAS policy.

Uh, so what this is, um, is utilizing the, uh, validating admission policy within Kubernetes, but within the management cluster. So I created a policy that's restricting to dev only. So within my project, I'm only allowing clusters to be created that have one control plane node.

So for example, if this were a dev environment and I didn't want people spinning up ha clusters taking up more resources and all of that, I can use the validating emission policy to block them from doing that. I can also write any custom validating emission policy to essentially manage or, or block any types of APIs within the infrastructure api. I, so if I'm creating a VM and I don't want to allow certain fields within that vm, if I'm creating another cluster and I only want to allow specific VM sizes for that cluster, all of that can be kind of done through this IAS admission policy, um, through here.

And then finally, the last thing I'm gonna show, I'm gonna jump over to a different cluster that was deployed, um, and look at the integrated, uh, observability pieces. And I, I, I chose a different cluster here because it'd been running for longer and it has a lot more data to see than one single bullet point. Um, but again, from kind of that core cloud console, I can go in, I can click on my Kubernetes cluster, and I can see all of the metrics surfacing from all the pods.

Um, also all the virtual machines that make up that cluster in a single view here. Um, and all of this is automatically being streamed back from those clusters after they're deployed. Um, so you get really kind of like an, a full out of the box experience of deploy a cluster, use Argo CD to play apps into it, and then get kind of the visibility back into it as this goes through.

Um, so that is kind Of the full, I I have a dumb question. Yeah. Um, uh, guy Courier futur, um, so this is in VCF?

Yes. But This is using the, uh, using the vSphere cloud, uh, vSphere v vSphere BERNET service. Correct.

But, um, whatever else I'm doing and deploying and managing is gonna be in VCF as well. Yes. Right.

So, So, so my question is for an operator. So we, you know, uh, Mars and Venus and all that, like the, the frequently, these are different teams or, or, you know, single platform though. Um, what are you doing to enable, um, to enable at least cross visibility or cross pollination?

Yeah, great question. Um, I think the easiest way to say that is within the, uh, platform itself, right? So when we think about VCF as a whole, um, we think about like these different layers.

And it might be easiest just to kind of draw some of this out, right? So you've got like VCFA here, um, which we were just using, and then beneath that sits kind of like your core, uh, vSphere stack, right? The, the traditional components of this ESX and all that stuff, right?

And then you've also got components like VCF ops that sit within this, right? So these different components all have kind of a tendency model and a persona around them. So as a platform engineer, I'm gonna be logging in to VCFA and I'm gonna have specific roles and permissions that allow me to do what I was just doing, right?

But also as a cloud engineer, let's call it, I'm gonna be logging in here as well. So I'm gonna be able to see the same components. I'm gonna be able to set those quotas.

I'm gonna be able to set those boundaries, but I also am probably gonna be logging in further down the stack to set up core networking, compute, all that stuff. And then I'm going to surface all of that stuff up into ops. And when I deploy clusters, which land kind Of, is there a missing arrow there between VCFA and, and, uh, and what's VCFA again?

Uh, VCF automation. Uh, automation, okay. Yep.

So yes, that arrow there, so those metrics and things are gonna flow into VCFA, and then you're gonna have, you know, your VVKS clusters that are deployed that land on this, that are also gonna be surfacing up information. So really kind of like, I think to answer your question maybe, is that we're, we're giving different roles and permissions from a tenancy perspective through kind of the cloud console, which is VCF automation in this case, and then we're servicing metrics and the visibility components up through, uh, VCF ops and then back into VCF automation so that the people who need to see the things that are going on can see them. Um, and then the permissions of who needs to control different components, like who can create clusters, who can create policy, who can do all of that, is also controlled through those boundaries within BCFA.

So I hope that's kind Of, so for, from a, from a management and, and strategy, um, architects standpoint, seems to me, um, you can get a lot of, uh, unified, uh, reports about the, you know, the whole, the whole state or the whole, you know, uh, um, domain or what have you. Um, but also from the operator standpoint, um, you can also be somewhat agnostic, but it does seem like there's a little bit of context switching from that standpoint involved. Maybe that's inevitable given that there are two different platforms for three.

Yeah, I mean, there's definitely differences, right? Well, you can sit there and drag application out of this VM and into that, uh, You still have to be context aware. Yeah.

So, so something like, and I'll talk a little about this in the next session a little bit, but a key thing here is that there's, we have these personas, but the personas are, if you think about the public cloud, the public cloud, you have a hidden layer that's responsible for like the, the magic that's happening in the cloud. And then you have everyone that is consuming it, they're all consuming it through the exact same interface. Yeah.

But based off of their roles and their access determines how much they're able to see how much they're able to like flex in that space. And that's exactly the model we're going with here is VCF automation becomes that cloud portal. It's where all consumption of the infrastructure actually occurs.

I think just to, you know, the, the sort of elephant in the room right? Is, um, sort of like Claude code and CLI based implementation. And I mean, we're not hearing any of that right now.

Maybe that's on purpose, but, but it seems like the, the momentum shift of administrator now is almost 80, 90% going to, you know, CLI based like Claude Code or Gemini CLI or those type of things. I, I kind of expect to hear a little bit of the narrative there because, you know, I mean, gooeys are sort of, Yeah, I mean, so like we, we talk gooeys here, but really that I, I saw the helm stuff, but I do think it would be interesting to hear more about Yeah. On All how I would sort of do all the stuff you're talking about, like Claude.

Absolutely. So, we'll, we'll talk a little bit Oh, you got that, about that experience. Okay.

In the next presentation. Thank You. Okay.

The, the last thing I wanna touch on here, um, is just kind of this diagram. This is what just kind of happened, and I, I just wanna show the architecture of it to kind of put the pieces together, right? Is that we use Terraform to talk to the V-C-F-A-A-P-I.

Again, that's kind of that centralized API exposing a Kubernetes API with, with CRDs. We have cluster management running in there, and our IS policy, we deployed a supervisor namespace, Argo CD was deployed into that as well as a VKS cluster. We then wired Argo CD up to that namespace as well as to that VKS cluster.

We deployed our apps through Argo cd, so like our core ingress observability, as well as an application into that BKS cluster, which resulted in Istio external DNS, Prometheus, Telegraph Cert Manager, the application itself. And then cluster management is managing things like Gatekeeper and Valero in that cluster for us. So all of this kind of happening in that, in that single deploy.

Um, and then just to kind of round out that full picture, I'll just draw this down here, right? Because I think there's been maybe a little bit of confusion or questions around like, where does VCF land here? So if I just draw a big VCF box under this, what this really, what I really want to change this to is like, this is your, your data center and your infrastructure.

This is like, if you're using vSphere, this is NSX, this is ESX, this is all those kind of underlying components. All of this is running on top of that and utilizing that.