Building Decentralized IT Platforms – Kris Nóva, Hachyderm and Gabe Monroy, DigitalOcean

This is Textron TV. Hey guys. Thanks for the throw.

We're here with Chris Nova who is a administrator for an instance of Macedon called hackader. Did I get that right you did it's a plan where it's like Pachyderm, but with h for hacky, excellent and then we have gave Monroy who is a chief product officer for digital ocean, which is helping provide some of the infrastructure for some of these instances, but let's figure out what exactly is Mastodon. I think people have a sense of that.

There's a new platform out there for creating your own social media environment as an alternative to perhaps Twitter, but I don't think a lot of people know what's involved in setting that up. So Chris, why don't you walk us through some of the issues that you've seen and what should people know going in. Sure.

So when I think about Mastodon, I think of two things that I think really set it apart from what most folks know today. Number one. It's open source.

So that folks like myself can go spit up a copy very similar to like an IRC server back in the olden days. And the second thing I think about is Federation or this idea that we can have multiple copies of the of the same and compatible software running around the world that share messages and feeds with each other. And so you kind of get this like decentralized fabric of social media instead of one large centralized instance that running and some single point on the internet.

So in April of 2022 myself and a couple of my friends from my twitch stream decided to spin up a small instance of this and and that over the past like nine months or so has completely ballooned out of control and it went from running on, you know, taking up about as much resources as maybe a Raspberry Pi to now being distributed across about 10 computers running in a data center in Germany as well as plenty of infrastructure that gave in digital ocean has been kind enough to to hook us up with so it's really grown and we have some pretty exciting numbers about growth that I think most startups would be pretty envious of that. We're just kind of hanging out and letting people enjoy the social media So what did you learn about running masks down at scale that you didn't know going in that you kind of wish you do or as proverbial question. What do you know now that you wish your new six months ago?

Sure. That's a good question. I think so a little bit of my background.

I I work as a principal engineer at GitHub. And so I'm pretty familiar with like large like obnoxious stateful applications that need to have like unreasonable up time and keeping things online because the the rest of the internet kind of depends on on our service. So I think I had pretty good expectations on like, you know, expect the unexpected.

You don't know what it's gonna break. You just know things are going to break I think mastanon specifically some of the things that that I learned was like number one. It's it's a ruby monolith, which is exactly what we have at GitHub.

So there was there was some pretty familiar tools and techniques there that I was I was watching out for I think the second thing about it is there's this there's this concept called sidekick which are these sort of cron jobs that run alongside the main web service and that is what actually Is a lot of the what I think most people think of as the Twitter algorithm that does a lot of like the data exchange back and forth and how does one server sync data with another server and I think there's no real good way to do that right now at scale. So we basically are in this this very like, it feels like in the early 90s pattern of like just throw more computers at the problem and just create more instances of this like small like sidekick process. So that was the biggest bottleneck that we ran into the server used to run right back here behind me in the small Rack in my basement until we hit about 30,000 people.

And then finally we were just like the fans were on 24/7 and we had to move it out to a data center where we could get more compute just to throw out the problem just because there was just so much data being processed and post being shared and images being sanitized and there was just a lot going on. And game, I think you've seen a couple of these instances by now. What is it that you guys are doing to help run these environments at scale?

Because I think a lot of people, you know, they initially start with one similar what Chris did and I don't think they really understand how distributed this all gets pretty quickly. Yeah, and you to be clear. We've seen a six-fold increase in the amount of Mastodon usage on digital ocean since about October of 2022.

So lots of folks and lots of different communities getting spun up and and started to use Macedon in this decentralized way that I noticed describing but an interesting sort of background to the story. I actually got involved with helping out the hacker Community because I myself am a part of the hackaderm community, you know and and Friends built Mastodon instance focused on technology professionals and and you know, a lot of folks in my industry and space including Executives principal Engineers, like Nova and others they all kind of hang out on this instance and and I think the stats I saw work went from you know, I think of a few hundred to something over 25 almost 30,000, you know folks pretty quickly, but these are my books and and these are novice folks and so when I started to Recognize that there were some scaling issues because I myself was trying to post on the site. I ended up, you know getting on the phone with with Chris and you know text in a bit and being like hey, like what's going on like anything I can do to help and it turned out that object storage was just one of the many bottlenecks that was being dealt with and so we worked out a way to get digital ocean spaces into the mix.

So all of the media that you would expect in the social media equation your pictures of you know, in my case Kat or your videos or you know, my farm animals, you know, whatever I might be posting all that is getting uploaded to digital ocean spaces now, but again, you know, lots of moving parts to this solution and we're seeing lots of folks on digital ocean wrestling with this or another good example is a site called Master on Canada, which is you know, now up over 25,000 users all running on digital ocean as well. Chris to the standard set of as much as there are a standard set of best devops practices for deploying software apply here or there's something unique about Mastodon that needs to be thought through as an extension of the best way to go build and deploy these platforms. That is a that's a really good question.

And I feel like I have like multiple phds that require me to answer that. But um, you know, I've invested a career in answering that question. So I think like the short answer here is number one.

There is no standard devops practice and number two. Like I don't think the world has has really ticked the box for like what I would think of as a global Service. So when I think of Mastodon and when I think of Twitter and when I think of any like GitHub is like another great example of any of these Global Services that you can like realistically visit anywhere in the world.

There's not a good pattern for that. Like we have really good patterns for like if you want to run a kubernetes cluster or if you want to run like a single like stateful web application somewhere, but one of the big problems we've had to face is like, how do we cash things on the Edge at different regions around the world and how do we form and enforce legal requirements based off of the environment that somebody's accessing our data from right? Like just hosting data in Europe alone.

There's like there's International. Privacy laws we have to deal with and we have to make sure that like certain pieces of data. Don't leave the EU and so having something like digital ocean spaces where we can like realistically go and say hey go create this bucket that can grow to infinite scale in the EU and we don't have to worry about like the implications of law there.

We can just start putting people's photographs there, but then it's like but if you access that from Seattle, I mean, you're 12 hops away on the other side of the world. How can you get those those images somewhat fast, you know with a few millisecond delay across across the ocean, right? And then there's like now we're doing International reality and we're counting hops from continent to continent and there's a lot of like nuance and minutia that goes into where do you cash things?

How do you cash things? Where do the images actually live? How do you network?

The CDN or the point of presence is around the world together and in my experience a lot of that is just good old-fashioned like rolling up your sleeves and computer science. You have to like form a hypothesis test it see how it behaves next time. You're like you Out to the to the side of the world see what's going on tested debug it Etc.

And we've been just doing a lot of that with volunteers from around the world. Right? We have people filling out experience reports from like, how how is the service from South Africa?

Right and it's just like we have to just kind of trust them and and kind of react to it accordingly. So I do think having a global cloud provider is very important to us, especially when it comes to data residency and how we get different pieces of data around the world. I also think that just having a broad community of technologists willing to volunteer and help out.

It's been also a tremendously helpful for us as well. Yeah and Mike if I could jump in on that last point, I think one of the most interesting things about the story is that with the hacking room Community being so technically sophisticated and you know, Nova and myself work the Microsoft for a long time on distributed systems there. So, you know the expertise that exists inside of this community, you know, it's sort of this Venn diagram of people who are interested in social media overlapping with folks who understand how to scale Systems and you know some of the your challenges around data residency that Noah's describing, you know, as they've said a lot of the stuff isn't figured out yet.

But I'm definitely keeping an eye myself on what the hackaderm community comes up with in this case just because that experience is resident in that Community. Well, what I heard of say is she threw down a bit of a gauntlet, right? I mean ultimately she's looking at you guys and saying can you come up with some templates in some ways to help this become more accessible to mere mortals, right?

Not everybody should have to be a devops engineer specialist and with the highest level of certifications or whatever they may have. How are we gonna make this whole thing available in a way that democratize is the whole platform? Well that is in fact what we are here to do with digital ocean, but the way it's gonna work is well, we will follow the lead of the hacking community and you know what the hacker, you know, folks like Nova and others figure out and then you know, what I'm interested in is Templatizing that making that you know as simple as possible as simple as we can make it your four broader folks again thousands of developers and Technology professionals are using digital lotion today to stand this up Mastodon, Canada other instances are going through this process, but you know as Nova stated the devops model for this isn't figured out the data residency model for this isn't figured out and so I think we got to let this cook and bake a little bit and and see you know, there's also software improvements at my understanding going on on the Mastodon software Community side as well.

And so as this shakes out and as things get easier to use digital ocean will be at the Forefront of making this easy to consume. Chris a lot of the noise is obviously around what's going around with Twitter and that's raised the visibility in Mastodon, but I feel like this issue is bigger than Twitter. It's really about how we're gonna control the data that we have and Federated in a way that we don't lose control of it.

So do you think that as we talk more about data privacy in this world and the value of data that that whole conversation will lift this whole Mastodon conversation to a much higher level within organizations because it is really about the date at the end of the day. I I can't agree enough to be completely honest when I when I look at decentralized technology in the fediverse and Mastodon and I really take a step back and look at the 30,000 foot view. I don't see a world of people trying to break away from Twitter.

I see a world of people trying to take ownership of their privacy and their data again and I look at like how we've gotten to where we are as kind of a necessary evil. When you look at like these large Tech conglomerates like Facebook, right, you know, and it's become very obvious that given the choice between what's ethical versus what's profitable like, they're gonna they're gonna pick profit most of the time and I think we've seen some consequences at that, right? There's been some legal cases come out that's been pretty alarming and when I look at hackaderm, one of the things we've done is we've we've kind of come at this from a like an altruistic perspective and we said we're gonna form a non-profit our entire goal is going to be just to give folks the tools and controls that need to take ownership and accountability of their data again, and we're just going to use that entity to legally protect our data and that Creates a really nice balance with folks like Gabe where they're helping us out with the infrastructure and actually operating the thing but like we still give the community the tools and resources.

They need to have ownership of their data again, and I think that this is really kind of like surfacing a lot of these privacy concerns and it's really starting to like bring a lot of these conversations to the Forefront one of my big in observations that I've made is that folks are kind of traumatized from Twitter. There's a lot of the like we're gonna go and like like raise our pitchforks and like, you know go and like like March through the streets and yell and wave when like something becomes alarming and we don't really have to do that anymore, right? There's a team of folks sitting here at hackaderm.

We're on GitHub. We're on Discord. It's it's all volunteer based and it's something goes wrong like we can we can help people out and if somebody has a question or a privacy concern or they want to shut their account down or move to another server like we can realistically help them and support them.

And so I think we're starting to see a lot of like like this sort of social media healing where people are like, oh, There's actually somebody I can go talk to you and they're not just here to like Leverage my data and leverage my content as a profit vehicle. They're actually here to help me send rooster pictures and cat pics and maybe further my career and meet people and network and you know make money for my family and it's like it's like wow, this is all of a sudden kind of catching on and I really think I'm seeing that grow and and I'm seeing a lot more than just social media kind of catch on right like you said it is bigger than just Twitter. I guess if you are quote unquote, man is Helen not going to take it anymore.

There is a place to go game. Do you think that this whole conversation around decentralized it will change the way we think about the cloud and how we manage that because everything isn't gonna be this neat little monolithic entity all the time. It's gonna be highly distributed.

So or are we on the cusp of some other thought process here for what we're gonna call it infrastructure and how we manage it everywhere. It's an interesting question Mike in in some sense is I feel like this decentralized movement with Mastodon as an example is almost a return to the way that the internet was was premised that yeah. 0 gets thrown around a bit this idea of you know, large particularly social media entities consolidating you've been becoming the big portal to the internet.

0 if you want to call it that so I look at it more as a return and and I do think that there is a role for cloud providers to play in helping make that easier and and more you solving some of these problems around data residency making things like, you know, the devops for this easier. So communities don't have to bear as much cost in terms of the maintenance for this but at the end of the day, I'm also a Community member right and and I have my own vested interest in this and one of the things I see is this model of advertising supported businesses. They incentivize a kind of behavior in these communities around, you know, let's call it hot takes or spicy takes or you know, you know all optimized around engagement and that ends up creating a dynamic in these communities where I know it tends towards hostility attend search for polarization over time and I just tell you personally is a member of the academic Community inside of this community it is Very friendly very engaging you're not seeing this, you know fight for attention that you tend to see on the larger social media platforms.

And I do think that's appealing more broadly than just Mastodon and and the community that have community that I'm a part of so I do foresee this becoming more of a trend and something that cloud providers are gonna have to adapt to all right. I think you said the peer review will result in more adult Behavior, but that's basically what happens. Chris do you agree to that do you think this is a form of web 3 because in my mind web 3 is very much attached to blockchain and maybe we need to broaden our definitions and thoughts about what we think what three is or have these terms just kind of, you know, they're not really useful anymore.

as an autist High level engineer. I hate all words all the time. Any name.

I I just strongly disagree with by default. So like if you want to call a web 3, like feel free to do that. I'm probably not going to call it that I think I'm gonna try to prefer to use like it's like very specific for both terms.

Like I view it as a decentralized social media service using the activity Pub protocol, right? And and I think maybe a shorter way of saying that is it's it's it works like email but for for tweets where you you can have your own server and anybody in the world can access you from there server, but those those things kind of send messages back and forth and they just share the protocol. So when I when I look at like web 3, I guess I think blockchain I don't I don't know.

I'm not probably not the best person to ask this question, too. All right game last question. Are you guys gonna do anything in the way of training?

Yeah for folks going forward around this platform or web 3 or whatever else because it does feel like you know for lack of appreciation for we just have to have short buzzword terms for folks but at the end of the I how will we get people to understand how these decentralized apps are built and created. It seems like there's an opportunity for I don't want to say everybody needs to go back to school, but there's clearly something different here. Yeah, we are absolutely interested in helping with that education.

Your digital ocean has been synonymous with you know developer education and learning since it's Inception founding and what we see with Mastodon is is really, you know, just a great opportunity for us to continue in that vein. So, you know as I mentioned before I think the pattern here is gonna be we're gonna wait and watch and see how the communities like academ are able to take this to the next level in terms of best practices. And then we're gonna you know work on Temple tising that we plan to put out a series of blog post around how to do this, you know different levels of scale.

I'm not everyone is gonna have to wrestle with the challenge the Nova and team and hackadermit had to wrestle with there is also a version of this for people just want to stand up their own server and you have you have complete and other control over everything that they're posting right and we do see a lot of that happening on digital ocean as well. So lots of different permutations of how I see this shaping up and and we are going to be here to help educate the community and all of them. Hey guys.

Thanks for sharing your knowledge and insights. Okay. Thanks for having us.

All right, always a pleasure as always the best thing you can always do is experiment on your own figure out what's going on and then join and kind of someone else with a larger conversation. Thank you all for watching and best of luck in your future social media Adventures.