How Object First Achieves Absolute Immutability

Jeff Burke. I'm senior technology advisor at ObjectFirst. I've been doing data protection and recoveries for 15 years.

I'm also a former tech field day delegate, so I was sitting where you were sitting before. So now I know how it feels to be on this side. I'm going to tell you about how ObjectFirst achieves absolute immutability.

So let's go right into the weeds and show you the data center. This is Veeam backup architecture with OOTBI. What does OOTBI stand for?

Out-of-the-box immutability. Okay. This is your production data.

This is the most vulnerable location. The attack surface is big, and you'll notice we have a little assume breach there, which comes from zero trust. We assume breach.

That's not my artwork, by the way, but it does the job. We then go to the next level, and that is where the backup software is. So very important with zero trust and zero trust data resilience, which is Veeam's version of zero trust, or the extension of zero trust, is you separate components.

So the backup software is located here. It manages all the data, retention, and movement. And we've got the bug there as well.

Next, we go to where we are. Primary target location. " Yes.

Because again, we built the solution on zero trust. We're assuming breach everywhere. And because of that, we are protected.

And we don't care about the breach. Why? Because there is zero access to destructive actions.

You cannot get a command line. You cannot get access to the BIOS. Zero time to immutability, and this is an interesting one.

With object storage and object lock, the minute an object hits the disk, it becomes immutable. In other systems, if you're writing a full backup file, let's say to a Linux XFS repository, the full file has to be written first, and then the immutability bit gets turned on. " That's the whole point.

We don't want any chances. We don't want slim chances or big chances. So that zero time to immutability is a factor.

And most importantly, we are writing to an immutable bucket, object storage, native object lock in compliance mode. And as we know, there's also governance mode, which allows an administrative delete. " If you have a service provider, and let's say they weren't able to segment their storage properly, and some customer stops paying them, then they're going to have to delete data, right?

They have to compromise. The problem is, the more compromises you make in your data protection, the greater chance your data protection will become compromised. More importantly as well, our attack surface is smaller.

We're only letting in the S3 protocol. " Smart object storage API. I'm going to talk about that in a second.

Okay. Multiple resilience zones is part of Veeam's ZTDR, Zero Trust Data Resilience. It means you cannot just have local backups.

We're dealing with ransomware, of course, but there's also other things. We're in San Francisco. There could be an earthquake.

So you've got to have a secondary backup target. You can have an ObjectFirst appliance if you want in a second data center, but let's hope that the data center is not across the street. Right?

So if you do need something further away, you can use Veeam's Data Cloud, which also works with S3. Excellent solution. And we work with it very well.

Okay. Our absolute immutability is built on three words: secure, simple, and powerful. The secure part I've mentioned.

We assume breach. Okay? We use S3 Object Lock, and we're on a Linux operating system which has been hardened, and zero access to perform destructive actions.

Nobody, even someone with the management credentials, can delete the data. Ensure zero time to immutability with no delays. And as I showed you, is inherently separated from the Veeam data platform.

Now this separation, people say, "Well, why do we segment? " Think about your house. You've got a key to the front door.

Okay? And then you've got your room with the jewelry, your room with the safe, and your beer fridge, for instance. You're going to want to lock those other doors.

If people get in and they're in the hallway, if the other doors are locked, it's going to be problematic. They're going to be slowed down, or they won't get in at all. Same concept here.

Verifiably secure. This one I especially like. NCC Group have been given our source code.

" So what do I say? Zero trust across the board. Don't trust me.

Trust NCC. Simple. Now, when I first said this to an admin friend of mine, because I've been doing this job for a long time, and I was recovering and what, and he said, "Well, Jeff, I'm an expert.

" But I have never been on an IT team my whole career which was overstaffed. Okay? And it's always been do more with less.

And what happens? We all know what happens. Things get missed.

We get tech debt. Things don't get updated, whatnot. The one area where you don't want that to happen is with your backup repository because this is your last defense.

So hand this off to ObjectFirst. We will watch it. We will take care of it.

And your security teams, even if you have security experts, if you don't, then that's even more reason. If you have security experts, I am certain there is a lot of security work to do in your organization. So if this is off their burden off their shoulders, they're better off.

So it's set up three IPs only, username, password, and multi-factor authentication. Automatic scaling and load balancing. What does that mean?

If you run out of space, you add a node, it'll automatically load balance. Okay? No change to Veeam namespace.

Just use the same endpoint. Again, simple. What I especially like, though, is the updates are available directly from ObjectFirst.

It's one button to press. So there's no long procedure, things getting broken, forgetting. We do it for youPowerful.

" But I'm not just talking about the backups. I'm talking also about restores. One thing I learned very early in this profession was no one cares about backups, successful backups.

They only care about successful restores. You will be unknown if all your backups are successful. You'll become known very quickly if your restore did not work.

Okay, so why is that important here? Because Veeam have a beautiful feature, which I love, called Instant Recovery, which allows you to run your servers, your VMs, off of the backup file. Okay?

So situation, you're out of action. You need to be up and running ASAP. You can restore, let's say, your five most critical servers immediately.

But for you to be able to do that, your backup storage has to work with Veeam like this. They have to be intimate together, and we are with Veeam. We use the exact same block size.

" We built this only for Veeam, and so we took that into consideration, and we are able to use the one megabyte block size. So this is what we call in marketing supercharged for Veeam Instant Recovery, and that's what it is, and of course, fast backups are good as well. Now, what is the magic sauce in all this?

It's something called SOS API, Smart Object Storage. We didn't invent it, Veeam did, but we use it to the fullest. So it's basically storage integration without the need for plugins.

The first problem it solves is capacity and space visibility, which is really important. When I first set up my first capacity here when Veeam allowed S3, I get it, it was working, I was really proud of myself. And then accounts came along and said, "Great, Jeff.

So how much do we charge this customer? " And I looked, and there was a big N/A, which stands for not available because the S3 protocol does not give you that information, okay? This was fixed with SOS API.

And I would say that that is also a security problem because if you run out of space, if you don't know how much space you have, you don't have backups. So it's not an infrastructure problem anymore. That's a security problem.

Equally important, it brings in storage access control. Caching control is exposed to Veeam, so Veeam knows exactly what is happening in the object first cluster. Storage managed network access and load balancing.

And Veeam will break up jobs. Let's say you have a job with 10 VMs. Each VM will be a so-called smart entity, and Veeam will place it on the node that is best at that time for networking reasons, space reasons, so that speeds up backup as well.

Yeah, sure. Just sorry. I'm Jack Pauler from Paradigm Technica.

Yep. Are you front-ending actual S3 storage, or are you just providing storage and you're using S3 protocol? We're using S3 protocol.

So you have storage inherent in your appliance- Yes ... using the S3 protocol? Yes.

Okay. Yeah. That was not clear.

Yeah. So yes, it's on-prem, so we're not like a gateway or anything, is what you meant, right? We'll have a gateway.

Exactly. We are on-prem S3 storage, so we control that whole part of the storage. Yeah.

So Veeam will break up a job, let's say at 10 VMs. Each VM will be a smart entity, and it will place that smart entity on the best node for that point in time, depending on various factors. " Pull it together.

Veeam already knows, okay, it's on that node, grab it. Speed is everything in this business. So how does the data flow through the SOS accelerated repository?

I've kind of gone over this already. What are smart entities? For a more eloquent version, a Veeam backup construct ingested via SOS API for object storage helps direct placement of data and load balance concurrent streams.

Okay. Data communication. Veeam communicates with the accelerated SOS repository, sharing data and client details.

Node allocation. The repository guides Veeam on which node to send each smart entity. Again, there's back and forth in that sense, but Veeam knows already where to send it.

Provides direct access for data streams to spread across all NICs, boosting throughput. So as you can see here, the whole design has been focused on Veeam and the SOS API. And it's not that other vendors don't do that, but if you are trying to make everyone happy, so we work with Veeam, we work with these guys, these guys, you can't do that because every solution works differently technically.

Now, some of you will know there are also non-SOS API Veeam apps, okay? Like Veeam Kastor, one of my favorites. I love Kubernetes.

We work with them. Veeam Backup for Office 365, we support that as well, and Veeam Agent for Windows, Linux, and Mac, often used on physical servers, but there are cases where you actually use that inside of a VM or in a hypervisor that's not supported. So another little fun diagram.

And to show you how fast is this, I can't even keep up with the words. So Veeam initiates backup through SOS API. Up to two gigabytes of data ingest.

Backup data is encrypted in transit at rest. See, it's flowing faster than I can talk. Backup objects are immutable immediately on OopB, so we mentioned that, and remain so for the time window set by Veeam.

Another critical important factor here, Veeam is the brains. That's where retention is set. That's where the life cycle management.

We do one job and do it well, make sure those backups are immutable. And then finally, the other resiliency zone I talked about before, secondary data center, Veeam, through a scaled-out repository, that's what SOBR means, or through a backup copy job, sends off data to your other location. Question.

Sure. Who is handling encryption? You say encrypted at rest.

Is that you or is that- Veeam. Veeam. Exactly.

So Veeam handles all the functions that we would say are the brain functionsOur job is to take the data in, keep it in compliance mode, no deletion. That's it. Because the problem is, once you take on functions and you get a competition, you can get problems.

The idea here is to make this simple, that even a great admin who might make mistakes, can't make mistakes. So you're not inspecting data, not looking at data at all. You're just treating it as, I've got a blob that's coming in, I'm storing the blob- Exactly ...

never, and treating it as write once, read only, never touch it ever again. Exactly. Okay.

Yeah. Got it. And that's critically important, again, because I personally have dealt with a lot of ransomware attacks.

They generally take place late Friday night, long weekends. m. on a Monday.

They might, but you want to get the people when they are weak. And I've had situations where really good IT people, people who are much better than me, just lost it because it's super pressure. They're tired, they've worked all week.

And so the key element in recovery is it has to be simple. Even if you're dealing with very competent IT groups, the simpler the better. So, Sky Fugate.

Sure. Quick question on your diagram here. I know that you guys have mentioned before RackN Stack.

Is this primarily a private cloud focused solution, or do you have public cloud offerings? No. So we're completely local.

So it's cloud technology, but local, on premise. So this is only for people that are in- Exactly ... private cloud.

Exactly. Okay. Exactly.

Private cloud. So we want to give people an option. Well, Veeam wanted to give people an option where they could just purchase and forget about it.

Not worry about it. Cloud, we work with cloud. So Veeam Data Cloud is fantastic.

S3 protocol, communication with them. But again, it's not so much us, it's more Veeam. Veeam is handling that.

But yeah, we're on-prem. Gotcha. So this is just a box.

I can go put it in my data center. Exactly. Oops.

And then I just let it take care of the offsite backups, shipping it out for me. Well, Veeam will do that. So when you set up the Veeam, you'll set up either a, what they call scalar repository, which has a performance tier and a capacity tier.

And the capacity tier you can put offsite. You could have it next door if you wanted. Or you can have a backup copy job.

So local backups go here, and they're the one you really want to recover from because the speed. " So, there has to be a fast solution locally. Veeam then, in Veeam, you create your backup copy job for your capacity tier, and that is replicated offsite if your data center gets destroyed or something of that nature.

Got it. So for my local copies, though, for whatever I choose to have locally with Veeam, because absolutely, you're right. I can't always...

I mean, the data egress charges and all that too. Yeah. Do I get the same immutability, though, that I would get with AWS, with compliance mode, with government?

Exactly. The beauty of this is that in that sense, we didn't reinvent the wheel. We took a technology that already exists, S3.

We're just choosing the parts of that which are the most secure. So compliance mode. Again, there's a reason for governance mode.

There's a reason for a lot of things. And in my career, I constantly saw compromises, doing things, mainly because of cost, often. The problem is that in 2010, okay, 2015, now it's just too dangerous because ransomware can wipe out a business.

And so that's why this has become so extreme, so absolute. I was in a situation once where I literally had a CEO crying on the call because the business was going to disappear. Not a pressure situation at all, 3:00 in the morning.

You realize my business is done if you can't recover this. So that is why we've come to this conclusion. Take something which works, don't fiddle with it, but stick to the rules.

Don't add any bells or whistles just for the sake of trying to sell it, and that is absolute immutability. Thank you. You're welcome.

Oh, sorry, go ahead. Sure. Shala here.

So that made me think of another question. So in regards to the SOS API and the communication of that, what happens if the communication of that fails? Does the system degrade gracefully or...?

So let's say you're talking about a backup job is ongoing, okay, and the connection fails, period, right? In a sense, the same thing's going to happen that if you were just using S3 without SOS API. So Veeam, by the way, this is the Veeam side too, because again, we're just playing let us have it and we're going to keep it secure.

Yes. Veeam has various things it can do. Obviously, if it's a brand new full backup, it's going to have to start again.

Full backup. Incremental, well, probably the same thing as well. So yeah, you have to have the communication.

And of course, keep in mind this is local. This is on premise. So that can happen, but it's a lot less likely than with the cloud.

And that's another important thing. Because I have people ask me, "Well, Jeff, right. I just get the on cloud.

" What happens if the communication breaks down? You're in trouble. So I think now you have to have all options.

One of the things I encountered during my career doing this kind of thing was you never know what's going to go wrong. Obviously, you do DR plans, you do disaster recovery, you should test as often as possible, but there was always something new. It was just very unfortunate, but it always happened.

And so what I learned was the more options available, the safer you are. And customers that we had that had just one option, that got really risky. So it's the same thing.

So again, on-prem, anything can happen like that. You can lose communication, but that'd be even more argument to have the local box, because the cloud, that risk goes up exponentially. You're welcome.

Okay, this is one of my favorite things, telemetry. So for free, we will watch your box for you. We will monitor the services.

We will monitor the space. So you will never on a Friday evening get an alert, "Oh, you've got 10% left, and guess what? " We will warn you.

Something breaks down, a controller dies, it disc drives, we will contact you. Now, you can turn this off if you want. We highly don't recommend it, but that is a free service.

So being an IT person, free is, for me especially, if it's going to save me.