Protect Azure Key Vault Resources
Azure Key Vault is an excellent solution for storing secrets, be these simple passwords or certificates, and allowing applications to access them securely. This means however that Key Vault data becomes critical for your application and you need to make sure it is protected and available. Key Vault already includes some protections - version history for secrets, geo-redundancy for disaster recovery. However, these do not protect you against accidental deletion of secrets or the entire vault. In this article, we’ll look at ways you can protect your Key Vault from accidental deletion.
Resource Locks
A quick and easy protection for your Key Vault is to enable a resource lock on the vault. The resource lock prevents anyone from deleting the vault its self without having rights to remove the lock. It will not, however, stop you from deleting the contents of the vault.
If you’re not familiar with resource locks, take a look at my article on using these to protect any resource - https://samcogan.com/protecting-azure-resources-with-resource-manager-locks/.
Soft Delete
Key Vault has an option to enable soft delete. When enabled this means that if either an object is deleted from the vault or the entire vault is deleted, then the item is retained for 90 days. During this 90 day period, you have the option to restore the deleted item.
Soft Delete is not enabled on vaults by default; you need to enable it. You can do this either at deploy time or later. There is no UI in the portal for soft delete, so it needs to be configured either through ARM template or PowerShell/CLI.
Enable Soft Delete
Enabling soft delete at creation time with PowerShell is done with an extra “EnableSoftDelete” flag:
New-AzureRmKeyVault -VaultName "keyvault1" -ResourceGroupName "keyvaultRG" -Location "westeurope" -EnableSoftDelete
Enabling soft delete on an existing Key Vault is a bit more complicated and requires you to directly manipulate the ARM net property:
($resource = Get-AzureRmResource -ResourceId (Get-AzureRmKeyVault -VaultName "keyvault1").ResourceId).Properties | Add-Member -MemberType "NoteProperty" -Name "enableSoftDelete" -Value "true"
Set-AzureRmResource -resourceid $resource.ResourceId -Properties $resource.Properties
You can then use the get-azurermKeyVault command to get the Key Vault and check soft delete is enabled.
Restore Deleted Vault
If we need to restore a whole vault that has been deleted the first thing to do is check that is still available in a soft-deleted state by running the get-AzureRMKeyVault command with the “InRemovedStateVault” flag:
Get-AzureRmKeyVault -InRemovedStateVault
Then we can use the Undo-AzureRmKeyVaultRemoval command to restore the vault:
Undo-AzureRmKeyVaultRemoval -VaultName "keyvault1" -ResourceGroupName "keyvaultRG" -Location westeurope
Restore Deleted Item
If we need to restore individual items we can again use the “InRemovedState” flag to query for the removed item. Depending on the type of item we want to recover (secret, key, certificate etc.) we would use the appropriate get command with this flag. So to look for a secret, we would use:
Get-AzureKeyVaultSecret -VaultName "keyvault1" -InRemovedState
Once verified we could then recover the item, again using the appropriate command for the type of object. For example for a secret:
Undo-AzureKeyVaultSecretRemoval -VaultName "keyvault1" -Name "secret1"
Backup Key Vault
Soft deletes protect you against accidental deletion for up to 90 days, but if you need to be able to go back longer than 90 days, or you need the option to store your data outside of Key Vault, then we need to look at a backup process.
There is no regular backup process built into Key Vault that you can leverage. However, there are some backup PowerShell commands (one for each item type) that can be used to backup your Key Vault objects to file (1 file per object), which you can store where you like and use to restore later. There is not one command to back up the entire vault; these commands are designed to backup individual items; therefore we need to put together a script to look at backing up the whole vault. We could then look at running this on a regular schedule using something like Azure Automation.
One important caveat to bear in mind with Key Vault backup is that you can only restore this data to a Key Vault in the same subscription as the one you backed up.
Backup Script
To facilitate an easy backup, we are going to create a script that does the following:
- Connects to Key Vault
- Backs up each item to a file in a temporary location
- Upload the files to Azure Files
You’ll note we are using Azure files as our backup location. The reason for using this rather than blob storage is because Azure Files has backup built in. By using Azure Files, we can upload the files, overwriting the existing files and then use Azure Files backup to back up the folder and keep the Key Vault backup history in the files backup. By doing this, we avoid having to keep multiple copies of the backup files for version history and can store them securely in a backup vault
The script is relatively simple, in its present form it assumes it is being run using an account that has already logged into Azure (using login-azurermaccount) and selected the appropriate subscription. I will shortly update the script to work with Azure Automation credentials.
The script expects to receive parameters detailing the Key Vault to backup, the File Share to backup to, and a local folder to store the downloaded backup files temporarily.
[CmdletBinding()]
Param(
[Parameter(Mandatory = $True)]
[string]$keyvaultName,
[Parameter(Mandatory = $True)]
[string]$keyVaultResourceGroup,
[Parameter(Mandatory = $True)]
[string]$storageAccountName,
[Parameter(Mandatory = $True)]
[string]$storageResourceGroup,
[Parameter(Mandatory = $True)]
[string]$fileshareName,
[string]$backupFolder = "$env:Temp\KeyVaultBackup"
)
We then go ahead and connect to the Keyvault and loop through all the items in Key Vault and backup them up to a local file, prefixed with the type of resource we are backing up. We backup secrets, keys, certificates and managed storage account keys.
#######Setup backup directory
If ((test-path $backupFolder)) {
Remove-Item $backupFolder -Recurse -Force
}
####### Backup items
New-Item -ItemType Directory -Force -Path $backupFolder | Out-Null
Write-Output "Starting backup of KeyVault to a local directory."
###Certificates
$certificates = Get-AzureKeyVaultCertificate -VaultName $keyvaultName
foreach ($cert in $certificates) {
Backup-AzureKeyVaultCertificate -Name $cert.name -VaultName $keyvaultName -OutputFile "$backupFolder\certificate-$($cert.name)" | Out-Null
}
###Secrets
$secrets = Get-AzureKeyVaultSecret -VaultName $keyvaultName
foreach ($secret in $secrets) {
#Exclude any secerets automatically generated when creating a cert, as these cannot be backed up
if (! ($certificates.Name -contains $secret.name)) {
Backup-AzureKeyVaultSecret -Name $secret.name -VaultName $keyvaultName -OutputFile "$backupFolder\secret-$($secret.name)" | Out-Null
}
}
#keys
$keys = Get-AzureKeyVaultKey -VaultName $keyvaultName
foreach ($kvkey in $keys) {
#Exclude any keys automatically generated when creating a cert, as these cannot be backed up
if (! ($certificates.Name -contains $kvkey.name)) {
Backup-AzureKeyVaultKey -Name $kvkey.name -VaultName $keyvaultName -OutputFile "$backupFolder\key-$($kvkey.name)" | Out-Null
}
}
Finally, we upload these files to the Azure Files share, overwriting any files that already exist and then clean up the local temp files.
Write-Output "Local file backup complete"
####### Copy files to Azure Files
Write-Output "Starting upload of backup to Azure Files"
$storageAccount = Get-AzureRmStorageAccount -ResourceGroupName $storageResourceGroup -Name $storageAccountName
$files = Get-ChildItem $backupFolder
$backupFolderName = Split-Path $backupFolder -Leaf
#Create backup folder if it does not exist
$backupFolderTest = Get-AzureStorageFile -Context $storageAccount.Context -ShareName $fileshareName -Path $backupFolderName
if (! $backupFolderTest) {
New-AzureStorageDirectory -Context $storageAccount.Context -ShareName $fileshareName -Path $backupFolderName
}
#upload files, overwriting existing
foreach ($file in $files) {
Set-AzureStorageFileContent -Context $storageAccount.Context -ShareName $fileshareName -Source $file.FullName -Path "$backupFolderName\$($file.name)" -Force
}
Remove-Item $backupFolder -Recurse -Force
Write-Output "Upload complete."
Write-Output "Backup Complete"
You can find the full script for this on Github here - https://github.com/sam-cogan/Demos/blob/master/KeyvaultBackup/backup-keyvault.ps1
Restore Script
We also need to be able to restore the files we have backed up. This script assumes that the correct version of the backup files is in the folder in the Azure File share. If you want to restore older versions of the files, you can use the Azure Files restore process to restore these files to a folder, then point this script at this folder.
To restore we are doing the opposite of the backup script, downloading the files locally from the Azure file share and then publishing them to KV. There are a couple of caveats with this restore script to be aware of
- It assumes that the Keyvault you are restoring to is either empty or at very least does not contain the secrets you want to restore. The restore-Key Vault commands don’t seem to support overwriting. If the secrets do exist you should either delete them or change the name of the ones you want to restore.
- At the moment the restore script doesn’t support restoring managed storage keys, there seems to be an issue with the command to do this that I need to work out.
In the restore script, the parameters are again going to expect details of the KV and Storage along with a local temp location.
[CmdletBinding()]
Param(
[Parameter(Mandatory = $True)]
[string]$keyvaultName,
[Parameter(Mandatory = $True)]
[string]$keyVaultResourceGroup,
[Parameter(Mandatory = $True)]
[string]$storageAccountName,
[Parameter(Mandatory = $True)]
[string]$storageResourceGroup,
[Parameter(Mandatory = $True)]
[string]$fileshareName,
[Parameter(Mandatory = $True)]
[string]$backupFolder,
[string]$tempRestoreFolder = "$env:Temp\KeyVaultRestore"
)
We then loop through all the files in the Azure files folder and download them locally temporarily:
#Create a temporary folder to download files
If ((test-path $tempRestoreFolder)) {
Remove-Item $tempRestoreFolder -Recurse -Force
}
New-Item -ItemType Directory -Force -Path $tempRestoreFolder | Out-Null
Write-Output "Starting download of backup to Azure Files"
$storageAccount = Get-AzureRmStorageAccount -ResourceGroupName $storageResourceGroup -Name $storageAccountName
#Download files from Azure File Share
$backupFolderTest = Get-AzureStorageFile -Context $storageAccount.Context -ShareName $fileshareName -Path $backupFolderName
if (! $backupFolderTest) {
Write-Error "Backup folder in Azure File Share Not Found."
exit
}
$backupFiles = Get-AzureStorageFile -ShareName $fileshareName -Path $backupFolder -Context $storageAccount.Context | Get-AzureStoragefile
foreach ($backupFile in $backupFiles ) {
Write-Output "downloading $backupFolder\$($backupFile.name)"
Get-AzureStorageFileContent -ShareName $fileshareName -Path "$backupFolder\$($backupFile.name)" -Destination "$tempRestoreFolder\$($backupFile.name)" -Context $storageAccount.Context
}
We need to determine what type of KV item each file is, which is done based on the first word in the file name, so we use the “match” command in PowerShell to retrieve this:
$secrets = get-childitem $tempRestoreFolder | where-object {$_ -match "^(secret-)"}
$certificates = get-childitem $tempRestoreFolder | where-object {$_ -match "^(certificate-)"}
$keys = get-childitem $tempRestoreFolder | where-object {$_ -match "^(key-)"}
Then we go ahead and restore the secrets into KV and clean up the temporary files.
foreach ($secret in $secrets) {
write-output "restoring $($secret.FullName)"
Restore-AzureKeyVaultSecret -VaultName $keyvaultName -InputFile $secret.FullName
}
foreach ($certificate in $certificates) {
write-output "restoring $($certificate.FullName) "
Restore-AzureKeyVaultCertificate -VaultName $keyvaultName -InputFile $certificate.FullName
}
foreach ($key in $keys) {
write-output "restoring $($key.FullName) "
Restore-AzureKeyVaultKey -VaultName $keyvaultName -InputFile $key.FullName
}
Remove-Item $tempRestoreFolder -Recurse -Force
The full script can be found on Github here - https://github.com/sam-cogan/Demos/blob/master/KeyvaultBackup/restore-keyvault.ps1
####Image Attribution Padlock flickr photo by Diego3336 shared under a Creative Commons (BY) license
