Brandon Dewitt – Who’s Watching Your Pipeline? Building In Audit-Ready End-to-End Compliance

In FinTech, it's really important to be especially rigorous in ensuring compliance with regulatory requirements Brandon Dewitt of MX and who's watching your pipeline, building an audit ready and in compliance. He'll share with us how they ensure velocity and data transformation at MX that leads to great customer experiences while guaranteeing that security and compliance requirements are met or exceeded and that their audit ready. Hi, I'm Brandon Dewitt, CTO and co-founder at MX, and we are in the banking software space, we work with more than two thousand banks and credit unions in the United States, and we work with more than 30 million users that are trusting their banking experience with us at MX.

And we have built out a continuous delivery pipeline that allows us to be audited and regulated while still doing high delivery software. And so I'm excited here today to talk about who's watching your pipeline, building an audit ready and in compliance. It's really difficult when you think about the process of building software and starting with the premise of this software needs to be used in some of the most important transactions that anyone will do on their daily basis.

And so you have to really think about how compliance is going to wrap that entire process, but still to be able to deliver at the types of speeds and the types of quality that we're seeing today in the software world. As I started thinking about this, one of the quotes that came to my mind is that Steve Jobs quote, which is I think one of the things that really separates us from the high primates is that we're tool builders. And so when we set out almost 10 years ago to begin building MX and begin building everything that we have today from a software perspective and a process perspective, starting from this idea of being tool builders, how do you automate more?

How do you make things more of a platform so that you don't have to hire as many people as maybe have been involved in the past? And so you can also stay small longer and serve more people and be more efficient. How do you make that happen in the world that we're living in?

And this this quote that Steve Jobs is talking about was really about the study that was in Scientific America that was about this man on the bicycle. And Steve Jobs talks about this and says, you know, the man in the bicycle is the most efficient mechanism for propulsion that mankind has ever invented. And as you can see, it can see the actual study here that was published.

You can see a fruit fly has very, very small body weight, but it takes an enormous amount of its energy to actually move anywhere. And then you can have things like the jet fighter or the plane or the automobile that, again, very high body weight. But again, a lot of calories to be able to get anywhere.

And so the interesting conclusion that Steve Jobs comes up with is the man on the bicycle is the most efficient machine that has been ever been created by mankind. And you can see that the man on the bicycle still has a high body weight, but the number of calories per kilometer that they're able to travel is is simply amazing. And so what Steve Jobs equates that to is then the computer is the bicycle of the mind.

When I saw this quote and when I saw him talking about it and when I heard about this study, I actually thought a little bit differently. I thought how interesting that the most efficient vehicle that we have ever created, our species has ever manifest is the bicycle. What an interesting thing.

How long did it take us to get to the bicycle? What type of thinking and innovation led to this platform? That is the bicycle.

And so I started thinking about what is a bicycle? It's a chain drive and it's a set of wheels and it's a frame. And so if we only focus on the Western world, there's there's lots of other history that might be presented from the eastern world that might even be older.

, an incredible amount of time. And then there's an illustration that you see here, which is from Leonardo da Vinci and Leonardo Da Vinci is is credited in the Western world with creating the chain dry. D.

So Leonardo da Vinci has the wheel as a frame, has chain drives, and yet one of the smartest and most capable inventors of the Western world did not manifest the most efficient machine that we currently have. In fact, the bicycle that you see on the images here, the bicycle that we know today, did not exist until after the automobile existed on eighteen eighty five. It took over four hundred more years after Leonardo da Vinci had every piece in his lab for mankind to bring those pieces together in the most efficient machine that we could ever create.

And so the reason that I bring this up is because building a compliant software organization is something that one involves a lot of complexity. There's a lot of pieces all over the shop floor from the systems that you can use to track tickets, the systems that you can use to track issues from customers, from internal customers, from your product organization, from your quality organization. And really what happens when it comes to compliance is most compliance frameworks involve some form of documentation around the initiation of a change, documentation around the acceptance of a change, documentation around the verification of a change and separation between environments as it moves from one state to another to impacting your customers or partners out there in the world.

And so one of the things that we started doing MX at the at the very beginning of our organization was we were releasing once a day at that time. And so at the very beginning of MX, we were releasing once a day. So every day we would have a set of documentation that we would sign off on, say.

And this is what we're releasing into production when our auditors come in, we can actually show this to them and they know that we're signing off on what went there. But as you all know, what's happened in the last decade is there's been an increasing amount of development. One iterative innovation is at the core of so much innovation that's happening out there.

There's this idea of continuous delivery, continuous integration and MX. We call it continuous momentum. We really want to focus on the fact that we are delivering to production and we're delivering value to our customers.

And so as you increase the amount that is going through out into production, how do you begin to track all of that for your compliance sake and for your audit sake? And more importantly, how do you stop the things that would be detrimental in your production environment? How do you stop them before they get there?

How do you set up your gateways? How do you set up your pipelines to make sure that they all work in concert so that you can be delivering the most amount of value for those that that you're serving out in the market at MX? This is a bit of a snapshot of the last 12 months of the software that we've delivered.

So on on any given month these days, we deliver between four and six hundred releases into production. That's from issues that are reported by end customers, issues that may be reported internally or changes that are created by our our support or our product or our quality teams. And so all of these then have to go through a process that's well documented.

So on a yearly basis, we have our these days our SOC Type two audit, which is very popular out there. We have our PCI compliance, which we have to maintain, although we don't house credit card data. The fact that we are involved so intimately within banking infrastructure makes it such that PCI compliance just makes it easier for us to be able to get contracts done in business.

Then therefore we maintain it, even though we're not handling payments or credit cards across our platform. There's also additional compliance burden brought on by things like privacy, GDPR or other frameworks that are very important. But they all really come down to there's some section of them that that involves your software development lifecycle, how you document and the framework that you use for moving things through that cycle, and how you can prove that every change that went through that process made it through the entire process.

And we signed off all along the way. So one of the things that that we deliver to each of our engineers as they start is this general understanding of what needs to be done in order to get a change into the production environment. And so we have this idea of the feature run down to the bug, run down.

We've split those into into two different into two different pathways for us, because one of them involves needing to create a regression spec, making sure that spec is passing as it goes through the pipelines and then getting cued up to move through the environments. And so just to touch on it briefly, first and foremost, everything has to start with a ticket or an issue inside of GitLab. And so at the very center, the platform that we've built on our in-process process on is GitLab.

And so everything has to start with an issue in the repository, either initiated by product documentation, quality assurance or support has to start with a ticket that begins that that begins a change that may actually move into production or it may get close before it moves into production, begins with a ticket. We have to create a topic branch because we use protected branches inside of GitLab. That allows us to make sure that people with the software, expertise that is required, are the ones that are approving changes to be pulled in to the branches that are not protect the branches that are protected and then go out to environments.

So we use protected branches. You have to have a topic branch. You have to have a set of passing specs.

Those have to pass both locally. And we have pipelines set up in GitLab that that mandate that those passed before they actually get merged into a main line of our code that will be pushed to our environments. You have to move the sandbox and sandbox verification.

In our environment. We have a sandbox environment, QA environment, a staging environment, and then we have a production and also an integration's environment, which is basically two production environments that allows us to to keep mirrored environments so that people can do their integration to us in an environment that is that is just as significant to us as our production environment. And then you create the merge request and you queue up the queue, run down is in the middle where you explain changes, you explain impact no to any database changes that may be occurring.

I'll show you a little bit more about what we do when database changes are occurring. You get a ship so that that means that you can move to the next environment. You then move to a environment deployed check to make sure that the app is up and running is as you know, if you're in software sometimes that those deployed just don't go as you expect, update the ticket, fix any bugs, a way to ship and notify a lead for staging and database changes queued for hours deploy is now that that last element there is something that we're doing is a as a temporary solution as we're working through some issues that we've had with our database in the last few months.

And so it's one of the things that it allows us to be, one of the things that that the pipelines in these gate, these gating moments do is they allow us to be very flexible with the direction that we actually push that code and we can actually say, hey, we're going to we're going to queue database changes for an off hours deploy, because right now we've been having some issues, having zero downtime database deploys. And so we're going to make sure that if there is a change like that in the scheme, in our code, it's going to go a different direction. And we're going to actually like branch how we push that code to production, but run down very much the same needs to have a regression back involved in it.

And we're actually going to we're going to measure that. We're going to measure the amount of test coverage that is actually committed with that as well. Then this is actually a system that we built in concert with our auditors called Sabotage internally.

We really named it after sabotage after the Beastie Boys song, mainly because it used to be the song that played every time we deployed to production here at MX. And so we created Sabotage really because as you think about the compliance in and in that process of moving code into production, the main thing that an auditor wants to come in and look at is, is a large collection of changes that have been made. And then they're going to take a subset of those changes.

They're going to sample them and they're going to ask for all of the documentation that proves that, one, it was initiated by someone. There's there's a documented initiation point. There's a documented change.

A verification of that change by your quality department has to be a separate organization that is actually. Denoting the verification of that change and then also once it to actually moved through environments, you have to actually have a sign off in each of those environments that it was a change that was accepted and approved in that environment. And if it wasn't, then it was rolled back and and potentially blocked.

One of the things you see in here is the is the bang bang emoji, which means that it's blocked for one reason or another and needs to be fixed or we may be actually waiting for it to roll out to that environment. And so these actually stack up all the deploys that allows us to chronologically see what is moving through our environments and what is sitting in each of those environments at any given time for us to be able to tell what needs to be able to move up the next day or what is potentially blocking some things from moving up and the data that we can then give to our auditors so they can see everything that occurred on the way to a production deployed. This is actually a specific item.

As you can see. We've denoted it as impact low. But one of the things that we did with the GitLab API is once you actually post a a GitLab merge request in here, the sabotage system actually links all of the documentation of the issue of the description of the issue, pulls it all into this so that our auditors only have to go into this view instead of being able to see all the source code and all the everything related to that and potentially be in that environment.

We actually pull it into this visualization through the GitLab API so that our auditors can actually just click through on the on the changes that they want to sample. And they can view, as you can see, the activity on the right hand side, everything that initiated this activity, every time that it has been blocked or it has been given the go ahead to move to the next environment when it was done, who did it and which department they were in. So then they can actually sign off that this actually passes our burden of of auditing, that this is a compliant release that went through the entire process.

And we know that it hit every single stage. The other thing that's interesting is you actually don't have the ability to move it through the stages unless it is past the previous gating items. And I'll show you that here in a second.

So when you're in GitLab and this is obviously the GitLab interface for pipelines, every you can actually create with every merge request, you can create a pipeline for really anything that you want to do. You can pretty much automate tons of processes that maybe you may be doing manually today. A few of the things that we automate MX.

Is we automate our static code analysis for security, we use a lot of Ruby and Ruby on rails, we leverage static code analysis like brakemen to determine that the last twenty five are not being injected into our rails. Could we we use it for static code analysis, for syntax. We use things like RuboCop.

We use other syntax static code analysis tools that determine that it passes our stylistic desires before we actually move it to another environment. You can have a pipeline fail, emerge, request to not move to the next environment without you passing it. We also do it off of things like mandatory 90 percent automated test coverage that is committed with each merge request.

And so if if we don't have that mandatory 90 percent automated test coverage, we're going to reject it automatically through the pipelines. And we're going to send it back to the engineer who's working on that. Again, these are all things that I would say 10 years ago would involve a more senior supervisor or manager of the engineering team.

But now we're able to take each of these pieces and put them into individual pipelines. We're able to reject the pipeline and inform we're able to reject a change and inform the engineer in an automated fashion what needs to change for this to actually move forward to the next environment. It's quite a fascinating experience.

All of the automation that is that has really come into the software development process in the last 10 years. Another piece that I wanted to show here is you're not only able to do things like verifications and checking of code and things like that, we've actually extended the pipelines that we leverage within GitLab to automatically write code for us because we operate in a polyglot environment. We actually leverage a lot of ruby and rails.

We typically run on top of J. Ruby, but we also leverage a lot of Golang in our infrastructure. And so one of those things that I was talking about earlier is when there's a schema change, as many of you know, you may have to update a schema in another language like Golang, even if the schema change was in on the ruby side and vice versa.

And so one of the things that we've done with our pipelines that I'm showing you here is this actually shows there was a schema change that occurred on the Ruby side and that initiated a pipeline that actually used a code generator to regenerate the schema bindings in the Golang process and automatically push a new merge request in the Golang project that interacts with the same database, also automatically initiate a new release within our Sabotage system. And so not not only can you can you have pipelines that can stop things from moving forward, that can route things in one way or another. You can even have these are our custom pipeline here, which is generating new bindings for our Golang processes, automatically initiating a new release and allowing us to keep our polyglots system in sync with one another so that we know when when two services that might interact with the same database that are in a different language are out of sync.

We know that automatically we're able to bring those two releases together and push them up at the same time. It's it's really been one of those amazing things that, again, as you might imagine, as you're managing MX, we have almost five hundred employees. About one hundred and sixty five of those are on the engineering side of the house.

So they fall into QA or engineering or dev ops or infrastructure in one way or another. When you have that many people pushing that number of releases and that amount of code, you have to have things that really help you keep very straight what is going on in your environment. So one of the things that I talk with to the team about all the time is let's use computers for what computers are good for.

Computers are really good at spending all day long sifting through hay to find potential needles. And so you can create a pipeline that is just observing all the changes, determining if there's a schema change and then regenerating code. And it'll do it all day long.

And it will always be up to date with where your platform. It's it's we've truly come a phenomenal distance in the last 10 years of my experience here, MX, again, this is for an organization that is on your pathway of building out a a a CI/CD infrastructure, coupling your initiated changes to the code change and making that all happen within a compliance oriented infrastructure. If you are working in one of the many industries that require sets of audits or compliance certifications, you need to be heavily investing in both what is offered on the GitLab platform from a pipeline perspective, from a CI/CD perspective, and also from the API perspective.

Because what I found is that as a CTO of MX is the best way for me to have the best outcomes with our auditors means that we create an easy to understand, easy to comprehend visualization layer that allows them to easily go in and sign off on anything that we've done without delivering a bunch of paper to them, without all of the complexity that may be around it. And I think, you know, I'm I'm kind of showing off a little bit today of the bicycle that we've put together. But I'm also confident because of how APIs work and how they've been used over the last several years, that there's probably other experiments that are out there that I look forward to seeing from other organizations that are out there that are putting together something that is the simplicity on the other side of the complexity that makes it very simple for them to go through that process of compliance and be able to deliver the software to the industry, quite frankly, industries that need it, industries that are very typically been held back because of the compliance burden that is necessary are now being enabled because of platforms like GitLab.

And so we're happy to be a GitLab partner.