Database as a Service (DBaaS) with VMware Data Services Manager from VMware by Broadcom

Hi, everybody. My name is Eric Gray from VCF Tech Marketing. VMware Cloud Foundation is the flagship offering from VMware for building your private cloud.

It's what you... It's where you get your compute, your storage, your networking, everything you need, management. Uh, but there are also some advanced services that you can optionally, layer on top of your VCF private cloud to give you additional capabilities, and that's what we're gonna be talking about in this session.

Uh, Data Services Manager or DSM. It-it's Database as a Service for your private cloud. Today, I'm gonna give you an introduction to DSM.

I'm gonna be presenting it mostly from the perspective of a vSphere administrator. How does, how does it integrate with my VCF vSphere environment? Talk about some of the RBAC access controls, and then we wanna leave some time ...

Gonna go through a demo scenario, show you the product in action. So, it should be very interesting. You might be wondering, "Hey, this DBaaS thing is kind of new to me.

Why do I even need DBaaS? " Well, the truth is there, there's a problem with that. You don't really want your developers spinning up virtual machines arbitrarily throughout your data center, installing, databases, whether it's open source, Microsoft SQL Server.

There's licensing implications with, with the latter. Uh, it can lead to sprawl, and you have no governance over those controls. You have configuration drift.

Those developers who set up those virtual machines with databases, they're not gonna keep on top of the lifecycle management of the database engine itself, keep the patching. Uh, they're not gonna be taking backups. And so in order to put some control on that, most organizations will result, resort to requiring tickets with IT.

So if you want a database, you open a ticket. We'll give you a database in a VM. Uh, we'll take care of it.

You want it upgraded, you wanna make sure it's backed up, all that's gonna be done through the ticket. And of course, that's a bottleneck. So the developers don't like that, so what does that lead to?

Overprovisioning. They say, "Oh, I don't wanna have to open another ticket next, next time I need more RAM. " So those are the kind of things that are probably familiar with a lot of, organizations today when it comes to databases running on your virtual infrastructure.

But we can address that with VMware Data Services Manager, DSM. Now, Data Services, we call it Data Services Manager maybe because it's aspirational. Right now it's a database manager.

It manages three databases, two open... the two most popular open source databases, MySQL and Postgres, and in tech preview right now, we cover Microsoft SQL Server, very popular database to run on your VCF environment. DSM covers the full lifecycle, from initial provisioning even through self-service access, if you like it.

Uh, we have backups, and I'll show you that a little bit later. You can clone a database, so if you need to reproduce an issue, scale out, we can do that very quickly all through, user interface or API. Uh, you can make sure that your database engines are kept up to date with the latest versions.

We, we can patch that all, without, having to go into the indu- individual VMs and do any work. It's all done, through Data Services Manager. And of course, we, we handle clustering as well, so you can spin up a three-node cluster of Postgres instead of just a single, instance, and that'll give you better availability.

Just if we d- take a very high level look at what DSM is and how it works, imagine this is your VCF environment. You typically have a management domain where you're running your vCenter, your VCF automation operations, all those things, and then you have one or more workload domains in your data center, and that is where your actual VMs are running. So we can use that same model with DSM.

We just layer on top of your existing VCF environment. We'll add the DSM appliance to the management domain. You connect it to your vCenter that controls your workload domain, and then you as the administrator, you'll go out and set up resources that you want those databases to consume, and that can be either a resource pool or cluster or a supervisor, if you have a supervisor set up in your VCF, environment.

If you're using a supervisor, you'll create a namespace, and that namespace will be the destination for those databases when they're provisioned. If you're using a cluster or a resource pool, they'll be, be deployed on top of those just as expected. O-once you've set all that stuff up as an administrator, the rest can be done through self-service, automation, API.

Uh, the databases will run on top of those. Now from a vSphere administrator point of view, DSM is deployed as an appliance, and then there's a plug-in added to vCenter. And so you can see on the screen here, you'll have this additional new menu when you look into the, the configuration for that particular vCenter where this has been deployed.

And you'll, you'll see these terms here. I'm gonna go through them later, so we won't talk about them right now. But this, this is the, the basic experience here.

And it supports a VCF single sign-on if you're doing that, so if you have multiple vCenters, you can log into one. You can, you know... So that's how you would have your, your, appliance in one management domain, and you can still access the, DSM, interface itself in another, w-workload domain.

So it all works great with all the VCF 9 features. Now, as a vSphere administrator, you are still in control of the infrastructure. You're gonna use all those familiar constructs that you use every day for all your other applications to control where these databases can run.

So for example, in for compute-You, as I mentioned earlier, you can set up a whole cluster that's for, for databases. Sometimes customers wanna do that for licensing purposes, for like Microsoft SQL Server i-in particular. Or, or maybe you have a resource pool on a cluster or a supervisor namespace.

Okay? For storage, you'll be targeting a, a storage policy. So you could use vSAN, you could use NFS, what-whatever you like, whatever is un-is compatible with your environment.

If you have a storage policy that we can target, that can be set up in your infrastructure policy and, your databases can land there. And then networking is also very flexible. You can use the good old-fashioned VLANs and distributed port group.

You can use the new, virtual private cloud subnets if you like. Uh, DSM takes care of the IP addresses. You don't need a DHCP or anything like that.

It's gonna assign static addresses to the, the databases as it deploys. Okay? And we also give you, as the administrator, a lot of control over who can access it, and that'll be part of my demo later, so I won't go into it too much right now.

You control who can access it, where they can run it. We can pull in the permissions from LDAP, or we have a local user database, in DSM maybe for smaller deployments that need that. And you can also control which database engines, individuals can deploy through the policies.

So this is all... It, it all makes a lot of sense. You still have control, but you can give a little bit of the, the s-the database as a service, self-service to the people who need it.

And those people who need it, by the way, it, it's not necessarily end users, developers, although it could be, but it might just be the DBAs. So you just... You don't wanna let the DBAs into vCenter to have free rein.

Give them this, put some guardrails around it, then they can own all of the databases and, and the resources and who, who can access it that way. All right. So this is just a, a quick overview of something I'm gonna be showing you in the demo, 'cause it's some new concepts, so I wanna make sure everybody's not just caught unaware here.

We have some new constructs because this is a, a, a, a new initiative, within your private cloud. So we have these infrastructure policies, and that's where you as the vSphere administrator, you go in and you assemble all of these things into a policy. You can also specify which specific versions of the database engines you want to, to be allowed in there.

Backup locations, which is an S3 bucket target within your data center, and then you'll create namespaces to group it all together. Next, you'll create a data service policy. The data service policy specifically says, "Okay, th-this is the database engine that it applies to.

" Which, namespace is just an-another form of grouping things together. We're just using that word, namespace. And then these are the things you're allowed to do there.

You can use this particular infrastructure policy, this backup location, or multiple. Okay? And then we bring it all together by pulling in your users from LDAP or local.

We assign them to a namespace, and that is what controls everything they're able to do. So now I'm gonna jump over here. I'm gonna sit down for this.

This is DSM. This is the user interface for DSM. Now, we're not gonna get into it day-today because we just don't have the time, but you can also integrate with VCF automation.

So if you're using VCF automation, you can integrate it with DSM and give access through that mechanism. But this is the local, DSM user interface that you get when you deploy that appliance. Now, I'm gonna walk you through a little bit of a scenario here.

Let's imagine we have a company that we've just set up DSM. We're kicking the tires. The, the DBA team has tried it out a little bit, and they're saying, "Okay, this, this works.

Let's, let's go one step further. " So in this case, we're gonna be talking about MySQL databases, but, almost everything I'm gonna say today will apply to all the database engines i-in, to some extent. Okay?

But I'm gonna be looking just at MySQL right now. So I'm the administrator right now. I'm logged in.

I have, I have a lot of menu options here. I have, the configuration settings. We're gonna take a look over here.

This are all the one-time things that you would set up when you first deploy DSM. You know, if you wanna have signed certificates, set up, you know, all your network information, everything like that. Um, now the next thing you'll do as an administrator is you, you'll set up your backup targets.

Now, this might be a new concept to a lot of vSphere administrators, because typically, S3 storage is something you see with cloud native applications. Now, DSM is built using cloud native technologies. Behind the scenes, it's built using Kubernetes and, and all, all, all of those modern, architecture, technologies.

You don't need to know that as an end user of it, but that's what's happening behind the scenes. And so therefore, they chose to use S3 buckets as a backup target. And a lot of times, you can get S3-compatible storage buckets from your backup or your storage vendor, your array vendor, or you can set up, any number of commercial or open source S3 services for this purpose.

The good thing about this is it gets the backup of the database outside of the environment where the databases are running. So if there's a disaster, you've got a copy over there, you can restore it to a new environment. Can I point it to a real S3 bucket, or do I have to use- Yes.

Okay. You can. You can, you can back up to the cloud if, if you have the bandwidth and you wanna do that, definitely.

Um, so l- in this environment, we've already set up the-Uh, a bucket for production databases. The DBAs set that up. So what I'm gonna do as an administrator is I'm gonna set up another bucket So I, you know, I go to my service internally, I create a new bucket, assign the access key, put in that information here.

It's just, just a very simple... Boom, set it up. So now I've got another backup target.

Maybe those are on, you know, different tiers of storage, different performance, whatever. Question. You know, Talera from infrastructure.

Uh, we are- you are using, credential there. Is there any zero trust or stuff like that that you can implement in order to avoid any, you know, credentials inside, inside the, S3 bucket or stuff like that? Hmm.

Okay. So the question is about zero trust to avoid- Avoid leaking ... leaking the credentials.

Yeah. Uh, right now the way it works is what I've shown there, and so, you know, it's all en- we're all encrypted here. Oh, you know, the, the connection is, is, encrypted.

Yeah. And then we're going to be, storing that as a Kubernetes secret on the back end. Uh, but, but, that's the extent of it that I, that I can talk about right now.

Okay. Okay. Mm-hmm.

That's all. Okay. And thanks for the question.

Now... All right, I've done this one-time thing, and now I'm gonna go over to MySQL, and let's say in this scenario, you know, we have a, a number of different MySQL versions that we can deploy in here for, for whatever reason. Sometimes you need a specific version.

And we've already enabled the two latest versions right now. But in the- in our scenario, oh, we need to give the developers access to this older version. 39.

Let's go ahead and enable that for them. Okay. So we'll click that.

We'll enable it, and now that'll be another, potential database that we can target when we set up our, policies. So the next thing we're gonna do is we're gonna go and set up a namespace. Now, we've chosen the word namespace here because behind the scenes it really is a namespace, 'cause this is all built on Kubernetes, okay?

Uh, now in, in VMware land, we have several different things that are called namespaces. You know, I'm sorry, I apologize for that, but, this is not like a supervisor namespace or anything like that. It...

We do have different things called namespaces because that's what they are. Uh, but in, in this case, we're gonna create a DSM namespace. We're gonna call it, devteam, and we're gonna apply a label here.

The labels are gonna help us scale. Later on when we wanna target these namespaces, we might get tired of having to select them individually if there are many, or we wanna be able to add new ones without having to go through and reconfigure things. In that case, we can just target these labels and everything will just kinda work out automatically.

Okay? So for now, we just have two namespaces. We got the DBAs who are already using it, and we got the developers who we wanna add to the mix.

All right. So we go and we're gonna give them permissions here. Here's where we link what's existing in our corporate directory under the directory services.

We're gonna add a DSM user role, so not admin. There's, there's very few admins in, in DSM. It doesn't mean you're a database admin, it means you're an admin of DSM itself.

So DSM user is where the d- db- database administrators and end users are gonna be accessing. That's their role. All right.

So I'm gonna... I have a, a directory group called Developers, and I'm going to add the devteam namespace that we just created to be linked with that Developers group in LDAP. Okay?

That's LDAP. All right, so that was easy. Very simple.

Okay. And then the next thing we'll do is we'll just review the infrastructure policy, just to remind you what it is. Uh, we don't create it in here.

We create it in vSphere, 'cause that's a vSphere administrator task. The vSphere administrator's in control of the storage policies, the compute where this can run, the port groups where you're gonna connect, specifying a range of IP addresses called an IP pool here, and also setting up things like VM classes. This is another nice thing about DSM.

The end u- the user of DSM doesn't get to specify arbitrary CPU and memory combinations. The, the v- administrators go in and set up VM classes, small, medium, large, extra large, extra memory, whatever you want in your environment, and then those are the ones that can be chosen by the end users of the system. So that is just another guardrail that we can put on this environment.

And then the policy is where we actually glue it all together and just we show who can do what and where. So we've got this existing one, the MySQL, production policy. We're gonna make a new policy.

This one is going to be called MySQL dev, and in this case, we're gonna choose MySQL. If we wanna make one for Postgres or SQL Server, we would do it right here. Okay.

Choose one data service, and then you choose w- which namespaces can use this po- ah, this, this data service policy. We are going to choose the label that we just made. Uh, so if there's a label, key value, in- infrastructure, developers, that's gonna apply to this one.

So the great thing here is if I need to make another, namespace and everything later, as long as I put that label on there, this thing will apply to it. Don't need to go back and edit this or update it or anything like that. So that's just...

It's better scalability and, and management. All right. So then we're gonna choose the database version.

We enabled these as administrators earlier, but now we're going to decide who can actually use it. We're gonna say developers can use these three versions. They have reasons, and we're gonna let them use it.

Okay. Which infrastructure policy am I gonna use? Just one.

You know, in a larger environment, most likely you would have many of these, different clusters, different types, different... Backed by different storage, whatever. So we're gonna do that.

And here we're gonna say, yeah, the developers are only gonna back up to that new bucket that we just made for the developers, and-And that's it. So the, it's, it's a pretty lightweight operation to create one of these, and you can go back and change it if you have to. All right.

All right, so now I'm logging out. I was an administrator back there. I could do everything.

I could see everything. Uh, now I'm gonna log in as a different persona. And this is a person we call Scott.

He's a developer, end user. Any Scotts in the room? I don't know.

Okay. Um, so I'm Scott. I log in, and his view in DSM is very different.

He doesn't have all those configuration options. He doesn't get to set things up. He can just see what he has been granted access to, which is great.

So that, that is the guardrail we're putting, the governance in the data, in the, for database as a service. All right, he is working on an app. He doesn't care much about what's going on here.

He really just wants to get a database up and running so he can do his work. All right? So we went over to MySQL.

We can do create database, the namespace, the one and only namespace that I as Scott have access to is called Dev Team. It's already preselected. I can't even change it.

Um, I choose my version. I'm gonna deploy one of those, eight zero three nine. I'm gonna put in the name of the database that I wanna use here.

A couple things about passwords. It, it'll auto-generate a random password by default if you don't put one in there, which is pretty good, so I... That's what I usually do, just let it auto-generate a random password.

You can look it up later when you need it. Um, I'm gonna deploy a single server. If you want, you can deploy a cluster, a three-node cluster so you have some redundancy there that live on different hosts in a vSphere cluster, so you can to- you know, tolerate an outage.

It's, you know, good practice for high availability. Right now we're just gonna go with a single one. We have our database, backup bucket preselected there.

We have our infrastructure policy. It's the only one we have access to. And then we'll choose a VM class.

In this environment, we just have the default small, medium, and large. You can make others if you like. We're gonna just choose this one.

It's a four, four cores, eight gigs of memory, perfect for a developer. And then you can throw in a few options if you need them. And, you know, depending on how familiar you are with databases, if you're a developer, you know that you probably need to do something, which is load some data into this blank database once it's provisioned.

And in MySQL, you need a parameter set on the server side that's called, local in file. And so you can do that right here. And if there...

There's some other parameters you could supply in here, but, you know, this would be a common one for a developer to request on the server side. This, this enables SQL, MySQL to have bulk data loaded in. Okay?

So the- there's our summary. We're gonna go ahead and create that. And it takes several minutes.

I've obviously trimmed out some of the waiting, but it is relatively fast. We go take a look here, and here are some details about the database that's been provisioned for me. And the thing I really care about, which is really what all this effort was about, is to get a database connection string.

Connection string, I'm gonna copy it to my clipboard. I'm going to pop open Notepad here just to show you what it looks like. This is a...

You've probably seen this before. It looks like a URL, right? MySQL in this case.

It could be PostgreSQL. Username, password, the host name, port, and the database name. That's what you need.

If you have that, you're a developer, that's what you want. Y- you don't really care about any of this other stuff. You really just wanted that.

And then I'll show you, you can use it from a command line, the MySQL client. You throw this data in here. I'm just loading the sample database of a list of cities, just so we'll have something to look at.

That's using MySQL client. But then there's another client, many clients exist that use these URLs, these, these connection strings. And so here's one called mycli, and I did that.

And then now, you know, I- I've got a little, script there. I can just pull up some random cities out of the database just to make sure it's working. Everything's looking good.

Okay. So let's say I'm the developer. I'm satisfied with how that went.

39 version. Okay, I'm ready to bump that up to the latest one now. And so and here is where you, the, the end user can do, can initiate the lifecycle management of his database.

So I go over here, and I notice I have two available upgrade options. I'll just choose one. Yes, I know there's gonna be a little bit of, downtime when I do that.

It's actually quite fast to do this. So behind the scenes, DSM is going out, changing the underlying data service engine, you know, remounting the data. Boom.

Back up and running. Okay. So we're gonna jump back over to the list of my databases, and you can see it did the upgrade, and now it's ready.

We come over here, take a look at this. Now the, the next thing we're gonna show is, so we, we've done the first test. Now we wanna add some availability to this database.

We just did the single node. Uh, we can do better than that. So we are going to change this on the fly to a cluster.

So now this will become a three-node MySQL cluster. And then we're gonna do another thing. It's unrelated, but also good, is we're going to e- enable the cluster to be the source for read replicas.

So I'm gonna later show you how to spin up some read-only replicas. So if you wanna scale out, your developers might wanna design their application so that writes go to a certain database, reads can come from these other ones. That way you can definitely scale, to much higher, loads.

So I'm gonna enable both of those features in that dialog box. Okay. I'm gonna go back over here.

And then it's scaling the replicas. We'll switch over to vCenter real quick, and you can see what's happening behind the scenes. We have...

I called it Oxygen, and now we have three of these database VMs called Oxygen. They work together. They have a little, you know, a little random tag gets appended to the name to make 'em, make sure they're unique.

Okay, so now if I go back to this diagram, you can see, oh-Yes, we have three primary re- and two replicas, over there. Okay. So now we will use this feature that's built-in to create a read replica based on a, a source.

All right, so we're gonna... We can specify different versions. We might as well choose the same one, and I'm just gonna call this one oxygen read replica one.

Okay, and some of this stuff is pre-filled because it already knows what's going on. I'll use the same size in this case, and I'm gonna spin up this read replica database. Now take a quick look back over at our original...

This is the, the, the source. Now there's a little helpful diagram to understand what this really looks like. You've got the, the three-node MySQL cluster here, and it's being replicated over to this read replica, okay?

And then monitoring, you can also take a look at this. Some basic monitoring just, just to help you understand, is the database healthy? You know, obviously you can also send metrics over to VCF operations, but that, that's a whole nother topic to cover.

But, you know, you can correlate all your VMs in your database and what's happening on, on the ops side. But here is just a quick, you know, is this thing healthy or not healthy? Is it...

Does it have enough resources, things like that. Okay. So we are gonna also just verify, we enabled backups when we deployed this thing and, you know, sure enough, the backups are configured.

They're gonna, you know, run on a default schedule, you know, every Saturday it's gonna do full backup, things like that. Okay. Is the only option the full backups?

Or- No, no. We have, uh- ... as well.

We, we can do the incremental as well. So there's a whole section on the backup schedules that you can set up so y- you know, you can tune it to your needs. Mm-hmm.

Also, I was gonna ask like, how granular can you get with permissions? Me personally, I always get nervous when my developers are touching the database, like- Uh-huh. Um, well, what do you mean by granular?

Like in terms of, let's say like, are you able to have designate, maybe there's just one particular developer who's able to go and like do a upgrade or not- Mm. Stuff like that. I see.

That always makes me nervous. Yeah. Yeah.

Okay. That's a good question. We...

The, the granularity is what I showed. So it's a group in LDAP maps to the namespace, and then they have access based on the namespace as sort of a group membership. So we can't really control who can do the upgrades.

Okay. We have to trust everybody who has access to provision the database can also do the lifecycle management of that database. So if, if that doesn't work, then pro- probably the model you would wanna use is have your DBAs using this.

Mm-hmm. You know, and they are responsible for... I mean, if it's a production database, you probably w- don't want the developers doing that anyway.

No. So, you know, I think it's okay. You know, let the DBAs run the production ones.

Let the developers do what they need to do over there. Yeah. Okay.

Okay. Thanks for the question. Um, on the ba- on the backups, how are the restores done?

Uh, what limits do you have on restores? H- what kind of restores are supported, and can you restore to other places? Yeah.

It's a... It supports a point in time recovery, and so if you do go into the backup and you wanna do a restore, you can put a time, a date and time that you wanna restore to, and so it'll... It- it's, it's collecting the transaction logs in a, you know, pretty regular fashion.

It's not just one big chunk. Mm-hmm. So it's able to go back and do that.

You can restore to a new database, yes. You can say, yeah, uh... I can't show you right now, but you can go to the backups and you could say, you know, deploy a new database from this backup.

Yeah. For troubleshooting or something. And who's got access to those backups to do those restores?

It, it's gonna be the same model. Whoever had access to provision it will be able to, will... They will be able to see the backups.

Okay. Yeah. But not everybody can see everything.

com. Um, back to backup for a minute because that's what DBAs should be very worried about. Transparent data encryption.

If the database, MySQL database in this case, is using TDE, is that also supported for the backups, or I should say enforced for the backups? I don't know the answer to that, Jim. Okay.

So- That's fine ... I don't wanna guess, but, uh- Okay ... you know, we're, we're, we're using existing constructs to, you know, generate backups from- Right ...

within those databases. So if- Okay ... if this feature is supported by those processes, then it would work.

Okay. So- Okay. Of course ...

we, we can, we, we can look into that at another time. Yeah, for sure. Thanks.

Uh, the, the, the backup's just native then? The... It's...

Well, they're... It's using... I don't wanna go into all the details of how it's implemented, but, you know, they're using some standard, backup, technologies that integrate with these open source databases that allow them to do backups to S3 compatible storage.

Right. It's not, you know, it's not all proprietary. It's, it's, it's standards based.

Okay. What about SQL Server? Yeah, SQL Server is in tech preview right now, and I don't really wanna go into all the details of that, but, but it will support backups to S3 buckets, yes.

So what versions is it gonna support then? What versions of SQL Server? Yeah.

Um, I'll say that it- it'll initially support SQL Server 2022. Okay. Okay.

Yeah. Thank you. And, quick question about the metrics.

Is it possible to export the metrics in another, monitoring system? Yes. You can send them to, VCF operations or to Prometheus.

To Prometheus? Yeah. Right.

Thank you. Yeah. Pleasure.

Okay. So then I wanna get, one more thing just to show you. We- we're in the read replica now, and we're gonna do the same thing.

We get the connection string. We can just paste that connection string into our command prompt, and we can see we're pulling out data from the replica. So now the developer can go and, you know, update their app to scale.

Pull from the read replicas and not overwhelm the read write, master cluster. And you can see a- again, it's just its own little database there. Okay?

And I think, you know, that kinda leads us to the conclusion here. The last slide. DSM is, is database as a service for your private cloud.

If you look at all the public clouds, they all have a database as a service because it makes sense. We don't wanna just run databases inside VMs. You can do it, and that's what everybody has done, and it works fine, but why not make it a first-class citizen, give you those additional goverm- governance options, the ability to do life cycle management of the databases, and also puts, you know, controls and self-service access there.

So I hope you'll take a closer look at DSM. It's a new offering, and not as familiar as some of the other, features of VCF, but I really think it's gonna take off. Thank you very much.