PingCAP’s Innovative TiDB Database – Max Liu, PingCAP

This is texturing TV. Well, the great pleasure being joined by Max Louie. Max is co-founder and CEO of pincap.

Welcome Max. Hi, Miss. Good to see you.

Good to see you. Thanks for joining us. So we're gonna talk about data and database one of my favorite topics before we do that.

Would you introduce yourself? Tell us a little bit about you and also tell us a little bit about pink cap. Well, thanks for having me.

I'm actually you and co-founder of pink cat and our product is tidy. The t i d e. Which is an open source distributed database.

You know, I'm I software engineer for more than 15 years. And I still enjoy coding before I started pincaps. I spent a lot of my time, you know designing a table schema carefully and fixing those, you know, database scaling issues and they're trying to make very faster and I've seen so many Engineers no developers wasting their time again and again doing the same thing.

So we started that tied to be projects to build a dream database. You know for engineers to handle scalability and you know from terabytes of data to you know, petabytes of data. To let developers focus on Seiko and SQL.

I'm sorry and for their business logic. so They can enjoy, you know on sleep. So you can get more sleep.

Oh, no, I mean you can still keep your hair not like me, right? So if you enjoyed coffee and or other fun. And also many people are found that our company name is interesting and pink cap is actually composed by two parts.

ping and the cap so cap is the c-a-p theorem. theorem, you know standings for consistency availability and partition tolerance You know, it's like a ideal case for distributed database, right? So ping is just the network comment while you're trying to you know, connecting to anything you will pin it, right?

Mm-hmm. So and I love this theorem so much. And that we want to keep as close as possible approaching say AP.

So hey, it's where we and you know, use the name pincap. Is where the name came from Big stories? Yeah, it's quite tacky.

Alright, well, you know, it carries meaning and that's one of the great things and I can sense already about your story Max's. It's great to talk to fellow entrepreneurs who is one thing you would say. Oh, that's a great idea.

Let's go build a product for that. It's another thing when you have you've experienced that challenge, you've lived this problem. You've spent maybe many hours, maybe many years right kind of developer.

But trying to be a DBA and a developer and a data analyst and a data designer, but you're not really your developer, right? You don't want all the hassles of all those other jobs. So, how do you make it easier?

How do you make it better for people? Which sounds like what you what you've tackled what you've done with pink cap. And what's Ty TV man my my path here.

Yeah, it's not easy. Let's talk about the database Market in general and as mentioned to you earlier, you know, I started doing database work long long ago. It's not it's not a new topic by any stretch.

And of course, there's a lot of database products in the market. So I I what's unique and different about about Thai cat. I say tie DB, excuse me and pink.

Yeah, and that helps you stand out. So why would the developer say? Oh, that's what I want.

You know, I'd like to use tidy because that that does these things for me? Well that that depends on the values you provide for your customers or developers, right? So well that I'm there are many key values that pink have brings to our customers users and the weather range of the community and of course through open source.

So first of all, we are open source Believers. No open source is the core philosophy of pink app. So which is always leaving the way of cardabies developments.

So besides had to be we also contribute a lot to the community. We have donates to open source projects to cncf. One is TI TV.

as a distributed key value storage And yeah is chaos smash so as a casting platform? So, you know those projects can help other software developers to build a more scalable the more resilient system. Now while you're building a distribution system, you're trying to you know are simulate kind of no physical Force, you know disc.

Disk faults and network network down and network recover and things like that. So that's why we need, you know cows testing platform. So second although, you know, the Denmark today's markets, you know, and you it's so called right?

So but there is no such products can be used as a primary database. to solve both transactional processing and analytical processing like tidy B You know as a foundation technique is designed as a scalable online transaction database so it can easily support hundreds of terabytes to you know, petabytes or of data. Well Steel.

serving millions of requests per second So what's more than even one real-time and analysis on the same database? Without moving the data from you know, trying to be to some other olap data warehouse. Let's imagine if you are building a SAS system.

There is always a operational dashboard for your customer right while you're logging to any sassy summary. You you got a summary, right? That let me the operational dashboard.

You need to generate that. That's what real time on the same database. Well, it is more natural for Developers.

No to operate the data on the same source of Truth. Hmm. So this technology is called HTM hydrated transactional processing and analytical processing.

So the concept is not new. That's the implementation being a cloud native way. It's fresh.

I think is the disruptive Trend in the database industry to make you know, everyone's life and work easier. It's really interesting. Yeah, combine those two things together because oftentimes you thought of where there's the old data warehousing or data lakes or different environments to do analytics in but actually being able to do analytics on top of transactional applications and data on the same environment.

What are some of the things you have to do to be able to handle those two different kinds of workloads because they can be very different. Right? They we want really fast transactional responses Sometimes some pretty complex analytical questions that we're asking, right.

Yes, you're right. So we're basically we had two different storage engine. Yeah your old store and they call them their store.

But with the smart Optimizer on top of both are also and countless store, right? So if there is a query the the optimizer will we all predict? Is this a little TB query?

All right, they say oh AP query. all right, even better it can be a hybrid query you can query, you know adjust this no single Row from my role store and do some you know aggregation on the colonist or It's a little bit technical. No.

No, I get it. So, I mean I assume your analytical queries might tend to be more by calmer versus your transactional by row. Is that a good generalization?

It starts there you yeah, exactly interesting and you can you can mix those and do hybrid of both. So is the is it views into the data or is the data redundant so that they can handle different loads on the two types of storage or is it just views into the same data? Also, basically, we have a replication algorithm.

It is called a raft. So we use rafts to replicate data and store them both as I real store and then columnistorm so you have two copy, right? So then you can design a Optimizer to choose what kind of data which piece should I use right?

So that's basically replicating them. No, all that. Admin worked to set up your your analytics environment.

Essentially you get that with the database Right comes with it. Yeah for replication for them. Yeah from the user perspective the operates on the same database there is no no, I don't need to build my skill set like, you know, I need to know how to you know, do manual sharding for a old TV database and that and using some kind of ETL tools to load the data to about our house.

And then I need to learn different kind of cycle different kind of, you know on the Korean optimization in you know, or two days to optimize, right? And so many details you need a totally different skill sets, right? Those little bit about the cloud part of the strategy.

So are you primarily are only working in the cloud? You also work on premise and customers own data centers. What's where does where does Ty DB live?

Well, this is a great question. We invest a lot into it had to be called. 10 cloud is a called series on top of time to be.

So which allows us to you know, provide, you know faster attemptive value for our customers and meanwhile. further reduce the the burden of no maintenance You know maintaining a distribution system is kind of pain right? So I'm Just like other open source infrastructure companies such as elastic confluence.

And Etc, so as it came behind, you know tidy B with trades are the relationship between, you know, open source, and I call strategies seriously. So I will see I will say that I'm no. Pincap will ensure the core components of time to be 100% open source.

And without any functional loss in our versions The this is not just the on the mouth. We actually achieve this by three actions firstly we view the an active open source community. I know try to be is backed by more than 800 contributors across multiple countries and industries.

Wait, I actually build a demo on top of try to be Cloud which is called OSS inside dot IO, so everyone. Can easily check any you know open source GitHub repositories? with details of stars contributions commits And you can even compare it to different projects.

And so on so forth and for the demo itself. It is open source, too. You know as I mentioned, you know, I we are in open source Believers, right?

so and there sorry the second, you know on we make sure I tend to be is environment agnostic. So the goal of trying to be is to achieve consistent user experience and the multiple deployment form. So basically you can deploy tightly, you know anywhere in public cloud in private Cloud VMS containers and and a very mental.

So it's there. I think this is the most important one type is designed as an open system. So we keep investing the the integration with different other did ecosystems such as no Kafka link a spark snowflake.

They they dog and song So this open is and capability also work to our car service as well. Very interesting what my questions is are there are there different users for the analytics capabilities or in versus the high volume transaction high speed transaction, or they tend to be the same groups of people as a primarily developers using both or do you end up with different users for different capabilities? Well that kind of different users, you know database is so generic, you know for any, you know, digital native business company they or have a database they have different users scenarios not for all those huge company big companies on and they they enjoy the scalability of ootp features.

So they they don't need to worry about you know how to scale my system, you know in those big companies especially for internet companies. They have a big team, right? They have a database team.

Hmm. So the database team care about the scalability of the database, but the big data scene begins see some team they care about the analysis of big data, right? Hmm But for those, you know on medium company and the stop.

They just want a single database and handle everything so I need so so I don't need to you know hire more Engineers. I have no resource, you know for those, you know, small companies right and medium I have No resource to hire to build a two different large team for OTB database for a big data, right? So they want a simple technology a simple database to solve.

everything so it's different makes sense. You mentioned distributed database then also since you can do your own data replication across the environments the column and and row environments. That's something you can also do distribution across different locations in Cloud providers or a club provider.

So if you want to distribute, you know closer to the edge or you know closer to the data center or different geographical locations. That's also part of Ty DB. Is that correct?

Yeah, you can deploy tiny B to you know different different available zones. No in nowadays is basically a default capability for a distributed database. If you don't have this kind of capability nobody's going to use it, right mmm.

So but for Edge functions add scenarios. Currently we don't have the ability to support that. It's just everything on cloud or deployed by yourself.

I guess is the cloud comes closer to the edge and you can be yeah, it's that way, right? Yeah. No, no as a open source that database tend to be chose to compatible with mySQL protocol.

So or those Mass kill users, they don't need to you know, I do lots of migrations, right? They can simply, you know, just to move the data actually and everything just works. And you know, I I'm asking is a TCP protocol, right?

So you're usually if you are using you know, some kind of scenarios, you know and add functions and they are talking about using a HTTP protocol. So you need a kind of proxy and to routes the request into you know and Technical. Interesting what tell me a little bit about the the open source versus the commercial version.

What are the differences in the product? Are you doing new features kind of things that are more experimental or that you're trying out and Market first in the open source, or are there more management capabilities in the in the kind of paid for version? How do you distinguish the two?

Well, actually they are three different versions. Let me explain a little bit more. So first one is tend to be Community Edition.

So it has all the core capabilities so that developers can enjoy, you know, the latest try to be features and able to contribute it back. So you can even get the United nitually version, you know every day. To enjoy the new features, right?

So and the second one and for sure is try to be called. So the new features are firstly released in the community version and after you know validation and the polishing by large number of community users and they will come back into can be called. And then what's next is a tidy B long term support version.

And so this is kind of a most disabled features and the professional service support by you know, pinkap provide for Enterprise users. So this kind of and users they make not interested in the new features immediately. I will wait right and here it's extremely stable.

Maybe a year later. I will use it in production, right? Extensible maybe a finance company is someone in the finance industry or manufacturing?

Yeah, you're right, especially for those Banks. So they release model actually to get feedback faster. From you know day to day operations of technical Cloud by ourself and from you know the community.

And that that also creates a faster look back to Tiny this roadmap. So on the verified and polishing features will be a pushed into Enterprise users through LTS release faster. This is kind of like, you know, yeah a fly while right?

So we can drive a faster attemptive value for both Community users and Enterprise users. No matter how time DB is employed. So to summarize we have three released to different and Employments, but hey consistent user experience in a process.

Excellent. Well, we have more time love to hear more about it. I hope you'll come back working folks find out more download the Open Source or try out the cloud version.

How can they check things out with you? Well, thank you Mitch. I had good to meet you a good talk to you and then thank you.

com that correct your website. Oh one thing so I'm very excited to share that we we will host the very first Edge step Summits on November 1st. Oh, okay.

Yes. Yeah your mom review area. com.

On the edge Summit page, and they do look forward, you know to connecting our view in November. Mm-hmm. Well, excellent.

Hope you have a great conference on the first of November and folks. Thank you very much as well as head over to pink cap and check out Ty DB. Thanks.

Again Mac. Appreciate Max you appreciate you being with us Max Louie who is co-founder and CEO of pink cap. Thanks again Max.

Thanks means.