Your ‘offline’ storage may be putting you at risk

I’ve been talking a lot lately about hidden IT risks that can sneak up on you and your company. These are threats that can lie dormant for months or even years. Once the threat materializes and reveals the weakness, the outcome can be devastating. This column is going to continue that thread by discussing an incredibly common practice that almost no one fully understands the impact of: so-called offline assets that really aren’t offline.

The only acceptable definition of offline is when the asset is powered off or cannot be connected to remotely in any form. The failure to understand the difference between truly offline devices, services, and assets and what many companies consider to be “offline” these days has led to a huge, understated, or even unrecognized risk that — if probed by malicious actors — could mean a serious business interruption. Some companies have gone out of existence for failure to understand the importance of needing some assets to be truly offline.

Code Spaces, a cloud-based hosting company, was one such victim. It actively promoted itself as having full redundant backups in three different geographic different locations. It said it had offline backups of all customer data.

It did not. A hacker broke in, tried to get the company to pay a ransom, and threatened to delete all of Code Space’s data if the company did not pay or if it tried to get around the hack. The company fought back, but the hacker detected the maneuvers immediately and deleted most of the data, including the “offline” backups. Hint: If the data can be deleted over a network, it isn’t offline.

Since when did we get in the habit of calling online storage and assets “offline”? Blame two big paradigm shifts. The biggest shift was the pervasive use of virtual hosts and their guest machines, but the advent of huge hard drives and data sets was also a factor. Somewhere along the way, admins started referring to virtual machines that were shut down as “offline.” If they disabled the virtual machine’s virtual network connection, TCP/IP, and the network card, they really thought they had done their due diligence and incorrectly declared no one could get to it.

When someone tells me they have an offline asset stored on a VM host, I always ask if it is possible for anyone, remotely, over the network, to start the VM and gain access to it? Almost every time they say yes. All someone would have to do is to connect to the VM host using the VM management software or a lights-out hardware device, and voilà! The “offline” VM is suddenly online.

I have news for you: If you can remotely access the asset over a network, so can the hacker. It’s not truly offline.

This false labeling hasn’t been helped by today’s huge data sets and huge hard drives. Today, data sets and drives are so big that traditional physical backup media (tapes, optical discs, and so on) simply cannot keep up. Instead, nearly every company of any size now backs up its data to very large disks, which are stored locally or, maybe if the company is slightly risk averse, somewhere remote.

But I ask nearly the same question: Can you restore data using a remote connection without anyone having to physically do something? If the answer is yes, this backup media is online or at least potentially online. Potentially online is really no different from fully online, and it means the hackers can get to it if they want. History is replete with fired admins who set a data time bomb to explode after they had also wrecked the most recent data backups.

Offline assets being online goes beyond data storage scenarios. For instance, the central tenet of Public Key Infrastructure is that the root CA (certificate authority) must be offline, so it cannot be maliciously modified. I do a lot of PKI work, and in my experience, more than half of the companies keep their root CA in a pseudo-offline scenarios such as a powered-off virtual machine (of course, reachable over the network).

A decade ago, when hackers did not specifically target PKIs, this was a big no-no. Today, when sophisticated attackers specifically target vulnerable CAs and private keys, it’s an insane decision. Repeat after me: A truly offline root CA cannot be connected to over a network, ever!

I feel a little sorry for the poor souls who trusted their financial security to Bitcoin and other electronic currencies, and who trusted their Bitcoin bank accounts to companies that didn’t know the true meaning of “offline” storage. Hackers broke into many of those Bitcoin banks and trading spaces, and they walked off with all the money. In an electronic flash, the bank accounts of every user were wiped out. Real banks, handling real currency, always have real offline backups.

Security isn’t black and white. It’s shades of gray along a continuum from white (no security) to black (full security). You might have different levels of “offline,” each appropriate for the level of risk you hope to mitigate. But only a truly offline asset, stored in a separate place that is not physically or electronically accessible, conveys all the protections we usually mean when we say “offline” — or at least what we used to mean by offline.

The world seems infatuated with calling an online asset offline when it isn’t. You can continue to do so, because sometimes partially offline status is good enough, but you must realize that the necessary protections given by true offline status are not magically conveyed simply because you call it offline. Some items need to be truly offline, and if they need to be truly offline, insist that they are completely offline. Otherwise you are creating and accepting an elevated risk that will immediately become apparent at exactly the wrong time.