Cyber-attacks have become all too common in recent year, as criminals have found them to be low risk / high reward endeavors. Mitigating the risk and disruption from these attacks is a team effort across IT security, IT infrastructure, Applications, and front-line support teams.
Even with thoughtful security policies and tools in place, organizations need the capability to rapidly recover from future CryptoLocker / WannaCry style ransomware attacks which have yet to be developed or discovered.
Azure File Sync with Azure Backup provides a non-disruptive means to synchronize the contents of Windows-based file servers to Azure, and rapidly restore access without the need for complicated tape recovery, or the complete replication of data to on premises file servers – in our example access was restored to a file server in less than one hour (for file servers with lots of files, it will take longer and depend upon your “process”).
If you want to understand the value of Azure File Sync with Azure backup to help rapidly recover from a ransomware attack (or already have Azure File Sync deployed) jump ahead to the section titled Recovering an Encrypted File Server with Azure Backup / Azure File Sync.
Many cyber-attacks in recent years have common themes and elements (not that any cyber attack should be consider ordinary), which include the exploitation of credentials to access files and encrypt them to secure a ransom. Often the automation will attempt to limit access to backups (by attempting to clear Volume Shadow Copy snapshots), interfere with backup processing. Attack code can linger inside a network for weeks or months, anticipating the expiration of backups to create more pressure on organizations to pay up for decryption of their data.
For organizations unwilling to bend to the will of these criminal attackers (there are no guarantees a ransom payment equals recovery), file servers must be rebuilt and secured before data can be recovered. When data recovery begins using typical, long term retention of backup tapes (if they exist) the recovery process can take weeks or months of recalling offsite tapes, loading them into tape drives, cataloging, restoring, and verifying their contents.
Often inadequate backup frequency / long term rotation, bad tapes, missing tapes, poor storage, misaligned tape drive heads, or any number of challenges can cause recovery efforts to fail, leaving data loss a common result after weeks of sleepless nights for administrators and unproductive employees (not to mention disgruntled customers or constituents!).
Azure File Sync Azure provides the capability to synchronize file hierarchies between Window Server and Azure File Shares, including the ability to tier/recall requested files. This allows existing Windows Server to function much like a storage gateway to a large Azure File share.
Existing file servers can use Azure File Sync to copy files to seamlessly Azure, and ultimately use Azure Backup to create periodic snapshots of the files which can be rapidly restored.
Should a file server be destroyed (or encrypted through ransomware), directory structures, NTFS permissions, and file contents can begin to be restored to the Azure File Share within minutes from a snapshot for resync to the impacted server. The File server can be replaced with a lower capacity VM (to help meet DR capacity limits) or the infected hardware can be wiped (sanitized / reloaded / patched / secured) and the file hierarchy reload using Azure File Sync to provide file system “dial tone” – users will be able to see directories and files. As files are accessed (become “hot”), they will be downloaded to the file server and cached on the local server as needed. If all files are required locally, they can be downloaded using a simple set of commands or options in the Azure console.
Azure File Sync using an agent that can be easily installed on current versions of Windows Server (2012 R2 / 2016 / 2019) and non-disruptively replicates directory structures and files from NTFS volumes to Azure File Shares. These Azure File Shares support backup via snapshots using Azure Backup… up to 200 separate disk snapshots that can be used to rapidly restore each backed up Azure File Share.
For organizations that already have access to Azure, setting up Azure File Sync for an existing file server or file server cluster is a straightforward, well documented process consisting of a few high-level steps:
- Create an Azure File Share in Azure
- Create / configure a Storage Sync Service in Azure
- Install and configure the Azure File Sync agent on the local file server
- Associate the local file server to the Sync Service and configure settings
- Creating Recovery Services Vault / Configuring Azure Backup
With proper access to Azure and the local file server, the entire manual configuration process can be completed in about an hour as well as automated with scripting for large scale deployments (although replicating file data will take some time…and depend upon the available connectivity to Azure).
The following steps are intended to show the process to setup protection for a single on premises file server, and later a sample process to restore file access to a rebuilt server.
Creating a Standard Azure File Share (now referred to as “Transaction Optimize”), is well documented in the Azure Documentation here: https://docs.microsoft.com/en-us/azure/storage/files/storage-how-to-create-file-share. For our purposes the Azure File Share will merely act as a repository for file data and will not be accesses as a file share directly.
This file repository can be up to 100TB per folder hierarchy, with the ability to assign a lower quota if desired. For details on how to create large Azure File Shares, you can go here: https://docs.microsoft.com/en-us/azure/storage/files/storage-files-how-to-create-large-file-share.
File servers with more than 100TB of file data on a single volume might necessitate the creation of multiple Azure File Shares to contain all the data (synchronizing separate directory trees to different Azure File Shares).
Once you have completed creating your share, it will be listed in the Azure portal, like those below:
Details of the process create a Storage Sync Service and connect it to an Azure File Share can found here: https://docs.microsoft.com/en-us/azure/storage/files/storage-sync-files-deployment-guide?tabs=azure-portal#deploy-the-storage-sync-service
To deploy a Storage Sync Service, go to the Azure portal, click Create a resource and then search for Azure File Sync. In the search results, select Azure File Sync, and then select Create to open the Deploy Storage Sync tab.
A sync group defines the sync topology for a set of files, and points to the Azure File Share previously created.
Creating the Sync Group and Cloud Endpoint is easy, and documented here: https://docs.microsoft.com/en-us/azure/storage/files/storage-sync-files-deployment-guide?tabs=azure-portal#create-a-sync-group-and-a-cloud-endpoint.
To create a sync group, in the Azure portal, go to your Storage Sync Service, and then select + Sync group:
In the pane that opens, enter the following information to create a sync group with a cloud endpoint:
- Sync group name: The name of the sync group to be created. This name must be unique within the Storage Sync Service, but can be any name that is logical for you.
- Subscription: The subscription where you deployed the Storage Sync Service in Deploy the Storage Sync Service.
- Storage account: If you choose Select storage account, another pane appears in which you can select the storage account that has the Azure file share that you want to sync with.
- Azure file share: The name of the Azure file share with which you want to sync (created earlier).
Moving out of the Azure portal to the file server itself, there are a few prerequisites and considerations for the installation of the File Sync Agent on Windows Server-based file servers, which are documented here: https://docs.microsoft.com/en-us/azure/storage/files/storage-sync-files-planning.
The actual process of downloading, installing, and configuring the agent to connect to your subscription and Sync Service is detailed here: https://docs.microsoft.com/en-us/azure/storage/files/storage-sync-files-deployment-guide?tabs=azure-portal#prepare-windows-server-to-use-with-azure-file-sync
The agent can be quickly downloaded from here: https://www.microsoft.com/en-us/download/details.aspx?id=57159, and many Azure savvy server administrators will navigate the process of connecting a file server to their previously configured subscription / Storage Sync Service quickly. Once the initial installation is completed, the server registration process will begin, which will require the administrator to use their Azure administrative credentials to locate the Azure resources, which should be available on the following screen:
The previous step connected the file server to Azure, now we need to define the set of files we want to protect by connecting Azure to a path on an NTFS-bad volume.
To add a server endpoint, go to the newly created sync group and then select Add server endpoint.
In the Add server endpoint pane, enter the following information to create a server endpoint:
- Registered server: The name of the server or cluster where you want to create the server endpoint.
- Path: The Windows Server path to be synced as part of the sync group.
- Cloud Tiering: A switch to enable or disable cloud tiering. With cloud tiering, infrequently used or accessed files can be tiered to Azure Files.
- Volume Free Space: The amount of free space to reserve on the volume on which the server endpoint is located. For example, if volume free space is set to 50% on a volume that has a single server endpoint, roughly half the amount of data is tiered to Azure Files. Regardless of whether cloud tiering is enabled, your Azure file share always has a complete copy of the data in the sync group.
To add the server endpoint, select Create.
Your files will now be kept in sync across your Azure file share and Windows Server.
It is important to mention that the speed of replication to Azure is dependent upon Internet bandwidth as well as spare processing capacity on the file server. Completing replication of large file servers can take several days, and will be impacted by file change churn (changed files will need to be re-replicated). For estimation purposes it is expected that a single file server with adequate resources can replicate files to an Azure File Share at a rate of about 500GB / day (1TB ever 2 days).
Configuring backup for the Azure File Share repository takes only a few more minutes.
After a few short steps your on-premises file server will be protected. Those steps include:
- Creating a Recovery Services Vault
- Configuring the backup of the Azure File Share
- Setting up your backup schedule
A walk through of this process can be found here: https://docs.microsoft.com/en-us/azure/backup/backup-azure-files.
Note that Azure File Sync supports up to 200 separate snapshots – taking only daily backups would means that approximately six months of rapidly restorable snapshots would be available for recovery purposes in addition to existing heritage backup / archiving process that might be in use in your data center.
After a file server has been compromised, you will want to leverage prevailing industry best practices to secure, sanitize, and rebuild your environment. Once your core infrastructure has been secured and cleaned (including Active Directory, DNS, and other foundation services), recovering data and restoring end user access will likely be a priority.
Whether you choose to simply wipe out encrypted files on impacted servers or build new servers / VMs Azure File Sync with Azure Backup can be used to rapidly provide access to file server directories and data. Steps to restore access to file-based data will generally following those outlined below:
- Purge / clean file server
- Create new file share / sync service… – DO NOT DELETE Azure File Share!
- Restore desired backup from Recovery Services Vault to New Azure File Share
- Configure Azure File Sync with Tiering enabled as per previous steps to sync folder hierarchy
Depending on your actual recovery process, the above steps can restore file access can restore user access in a matter of minutes!
Wiping out the “infection” can mean creating an entirely new file server, or simply wiping out share directories and files (depending on the extent of the issue). In any case, eradicating the offending code and its impact will likely need to be handled completely before a file recovery, and is not addressed in this discussion. Consult with qualified security resources and guidance to ensure your environment is ready for file recovery.
My suggestion is to create a new VM / clean file server, as it will simplify and speed up the recovery process.
It is not neccessary to delete the files from the Azure File Share (as they may not all be encrypted / infected) – you can restore the Azure File Share to another (new) Azure File share.
Whatever you do, DO NOT DELETE THE AZURE FILE SHARE ITSELF – the Azure Backup data is linked to the share and will be lost. Deleting individual files / directories or wiping the entire server may be thoughtfully pursued actions.
As mentioned, the preferred method would be to use an entirely new server / VM to host the file shares, but this may not be practical if the re-creation of local shares on the VM is infeasible (too many shares). FYI, a great way to copy file share info (including ACLs) is outlined here (go to the part on Windows Server Migration Tools).
If you do want to delete the files from the Azure File Share (which is a distraction from getting things back on line), the simplest way to delete the files can be from the infected server (if it is online), and the deletes will be propagated to the Azure File Share repository. Before Deletion:
With File Sync enabled, these deletions will propagate to the Azure File Share at some point… but clearing the files from the VM (if you are reusing it… perhaps not a great idea) is the priority. Note that the file delete operations will need to sync up through the Sync Service… which will take some time for large numbers of files… another great reason to use a new VM.
Another way to clear the Azure Files share (and connected server-based files) would be from Azure. This also will slow down the recovery process, since the changes to the Azure File Share will need to be noticed by the Sync Service (not automatic in all cases) and syned down to the Servers – slowing down the recovery process.
Here are the steps you could follow for this largely unecessary step.
Connecting to the share (mapping a drive) from another host – even a VM in Azure will allow for simple / rapid deletion of the directory hierarchy. Select “Connect” from the Azure File page in your storage account:
“Connect” will provide instructions on how to connect to that specific share from a Windows (or other) host:
Once connected, you should be able to delete all files on the Azure File share, with the exception of the system share information:
The restore process is well documented here: https://docs.microsoft.com/en-us/azure/backup/backup-azure-files#restore-from-backup-of-azure-file-share.
What I’ll show you here is a restore over the top of the cleaned out, existing Azure File Share…I don’t want you to do this however – I actually suggest you recover to an entirely new share that you create (an “alternate lcation”) – I just happened to do it this way when I recovered…I’ll explain why a little bit later.
Navigate to the Recovery Services Vault used to backup the Azure File Share, and select “Restore Share”:
Select on the desired last backup (before the introduction of malware into your environment) and click OK.
Select the restore location – in this case you will overwrite contents in original location – ALTHOUGH THIS IS NOT THE RECCOMENDED APPROACH SINCE YOU SHOULD RESTORE TO AN ALTERNATE SHARE and select OK:
Then select Restore.
You can check the status of the restore job within the Recover Services Vault by navigating to Monitoring -> Backup Jobs:
The total recovery time will depend on the volume of data, but as it is already on disk in Azure, it will significantly faster and more reliable than manual, tape-based restore – often completing in a matter of minutes.
In this case, an Azure File share with more than 200GB of files was restored in under 5 minutes!
The Azure File Share is now repopulated with “clean” files:
Rather than restore over the same Azure File Share (after ensuring all the data has been deleted on the Azure File Share and Servers – a time consuming task), it likely makes more sense to restore the File Share to a different location.
Why Restore to a New Azure File Share Instead of Over the Old One?
In short, because of the Sync Service – if you use the same old Azure File Share, you’ll have to clean out the file server, the Azure File Share, and the Sync Service database. All that deleting will run alot of unneccessary transactions through the Sync Service, delaying the ultimate file recovery. Rather than wait for all the delete transactions for the encryted files to process before adding in the new unencrypted files… lets just get the new (good) files synced. Based on what I’ve seen, the Sync Service can process between ~25,000-90,000 files a hour – so for large file shares (mine file server / Azure File Share was ~4,000 files / ~200TB).
Now go a head and:
- Make a new Azure File Share
- Create a new storage Sync Service
- Connect the Sync Service to the “Cloud Endpoint” (Azure File Share)
- Install / configure the File Sync agent on the new file server
Resynchronization to online / configured file servers will happen once the Azure File Sync agent is loaded and configured. While not discussed in depth earlier, it is important to consider the value and impact of enabling cloud tiering during a recovery scenario. Cloud tiering in Azure File Sync allows the file system hierarchy to sync down to file servers along with file “stubs” without all the associated data – reducing the time to file system “dial tone”. All the files will appear to be on the local server when users request them but will only be recalled when needed – reducing significantly bandwidth necessary for a full, local restore.
In this example, a laptop with a small SSD was pre-installed with Windows Server 2019 and used to recover a much larger file server, with Cloud Tiering enabled. Once the File Sync agent was configured, the entire file hierarchy was available on the system in minutes – note the size on disk used:
At this point, shares can be recreated and pointed at the appropriate directories, providing access to uninfected files.
Without tiering enabled, all data will be pushed down to the server, delaying access and utilizing bandwidth.
Pre-integrating Azure File Sync with Azure Backup in your on-premises file server infrastructure can significantly reduce recovery time from ransomware attacks.
The recovery example above restored access to a file server’s data in a matter of minutes without tying up heritage on-premises backup and recovery infrastructure – freeing it to be used for other recovery/restore efforts in parallel.
Deploy Azure File Sync with Azure Back and test it for yourself, developing and augmenting your existing recovery and ransomware mitigation strategies.