Ransomware

[cover image comes from Wikipedia]

It’s all about the backups

Ransomware is all over the news right now. It has gone from a harassment for payment in cryptocurrency to shutting down medical care and crippling supplies of food and fuel. These attacks are hitting all of us and have become a real problem, but the more I look into them the more I ask “How did this become a real problem?”

What’s enabled ransomware?

Let’s think back to earlier days with computers, there was almost never a time when someone would threaten you enough to make a payment. A lot of the reason why was the payment was dangerous for the criminal. It was very likely they would be caught trying to accept it. Cryptocurrencies have changed that somewhat. Now payments can be made to a wallet that is not associated with a person or place. The criminal can then take extra steps to try and hide how that money is used or transferred. It’s not good anonymity, but it is good enough to foil all but the most extensive investigations.

What is more troubling to me though is that backups have become more fragile (I bet you were wondering how this was anticryptography related). For most of my career backups were made to some sort of removable media and placed in a safe place. Floppies, zip disks, CDs, DVDs, tapes, hard drives and even thumb drives (I know those are not good backups, but they aren’t bad for a few months) had copies of critical data, or all my data, written to them and placed aside. While some place like Iron mountain, safety deposit boxes and safes are always good, just sitting on my desk isn’t that bad either in a lot of cases.

If I am struck by a ransomware attack, or really any attack or hardware failure, I can roll out one of these backup copies and be able to restore to a known good state. If the ransomware was around and spoiling the backups for some time I can just go further back and loose the difference in time. You do keep a string of long term back ups, right? Do you ever check restore them to a clean system? If you said yes, good for you. If you are hanging your head in shame you are in the majority.

To make things worse, most organizations are now backing up to the cloud. In general this is a great idea and should not be a problem. Where this strategy fails is that the ransomware will try to write to those cloud storage locations and destroy the backups. That brings us to the heart of the real problem. Backups need to immutable.

How to fix the backups

Be immutable

The biggest way to stop ransomware in it’s tracks is to have good backups that are immutable. While I would really like to see truly immutable backups, like writing to write once media, the backups just need to be immutable to the backup and restore process itself.

We used to get that for free with floppies, CDs and tapes. On some of them we would flip the write protect switch. In all cases we stored these copies outside the systems they were made in. Walking over to place a backup in a system to restore was a powerful defense against ransomware, but we abandoned it decades ago.

We started to have to juggle lots of media for backing up even small networks. It became common to have robotic tape changers with a half dozen or dozen tapes in it and larger systems were common. In all of these cases the ransomware could cycle through the tapes and destroy the backup if it was running at the right level of permission. This was fairly easy to fix with using properly secured backup systems that were isolated from production systems, but it wasn’t common and there have been victims regretting that they weren’t isolated.

Over the years though tape has become harder to work with and not worth the investment in most cases. Enter the cloud. The most advanced tape today can hold 18TB of data and is big and expensive to work with. Storing the same data in AWS S3 is $423/month for live storage and if you put it in deep glacier it costs $18/month. That’s hard to argue against. Want it to be a bit more survivable, store a copy in GCP’s or Azure’s equivalent also. At these costs it really is a no brainer. The problem comes from how many organizations do it.

In all of these cloud storage systems, there are permissions to read and write among a host of others. Many people doing backups to the cloud allow the system making the backup to write to the bucket where all the backups are kept. A big red X should be going off with a nasty buzzer in your brain. The system backing up to the cloud can potentially destroy the older copies. None of these copies are immutable!

There are some fancy ways to configure all of these cloud systems to make them immutable write once, read many. That requires that permissions are handled correctly. Another option is to have a small job running in a different security domain copy the backups to a more protected location and deny overwrite. Given a few minutes we can probably come up with dozen other options that would be effectively immutable.

Easy and quick restore

The second and also essential part is that backups are easy to read and use. This also is an anticryptography concept. Data needs to be saved in a way that it is trivial to understand and use. For ransomware this can just be reading files from a backup. Let me give you and example.

If this website is damaged I can restore it very quickly. I spin up a server and I rsync a directory onto the server. Hand jam a few configurations to make sure the web server starts up on boot and I am mostly done. I’ve tested this restore and it took me roughly 20 minutes. I could automate it even more and get under 5 minutes (the time for my cloud provider to provision the system being almost all of that), but I don’t need to. If I were part of a critical energy infrastructure I would work on getting that down to seconds in most cases and under 5 minutes in all others.

Given modern container technologies and VMs there is no reason a critical server shouldn’t be restorable in tens of minutes. User desktops should be close to that, or maybe even restored every time they log in (why should the average accountant’s system run all night?). Developers live in a special world where they may need to be handled differently, but if your source code isn’t in something like Git that isn’t backed up you have a broken culture.

Make backup and restore a key feature

If an organization practices constant backups and restores all the time ransomware would be a mild inconvenience if it even happened. Spin up and down systems as needed. Servers could be stood up from an immutable docker image. Databases could be force migrated through replication all the time to allow systems to be rebuilt on a constant bases. Desktops could be truly roaming where the user sits down and logs in to start a restore of their desktop that is measured in seconds.

You may scoff at all of those things and say it’s not possible or may cost too much. I’ve worked on systems that do all of those things using off the shelf and often common versions of standard IT systems. They all required a good architecture design and decent administrative support, but they worked well. Probably the biggest hurdle was the users that complained that it wasn’t what they were used to and the administrators that didn’t manage the systems correctly.

At this point you can choose. Do something different but more correct, or just live with the hassle of ransomware (and other problems). I fear that too many organizations have already picked the second choice.