Using Azure Key Vault with Application Gateway
Application Gateway now has the great ability to talk directly to Azure Key Vault to retrieve certificates for use with your endpoints. This means you can store your certificates in a central location, automate renewals and so on, and have App Gateway consume them.
Whilst this functionality is excellent; there are a few ‘gotchas’ when setting it up that I thought it was worth cataloguing.
Using the Latest Certificate Version
When setting up access to a certificate, you will supply a URL to the certificate secret in Key Vault. If you take this URL directly from Key Vault, this will include the version ID in the format of “https://name.vault.azure.net/secrets/secret/768c5abc12ec410faaf56d5a74778c0e"; this includes the version ID. If you use this, if the certificate is renewed and a new secret is added, you will have to update App Gateway manually.
Instead of doing this, drop the ID from the URL before setting it up, so it is “https://name.vault.azure.net/secrets/secret", this will work and will mean that App Gateway uses the latest version of the secret. When the cert renews, it will pick up the new version.
Key Vault Permissions
To be able to access Key Vault, you need to enable managed Identity on your Application Gateway. This means using a User Assigned managed Identity, as it does not support a system assigned one. Once created and assigned to the Application Gateway, you need to grant the identity rights on the Key Vault to read secrets (yes, secretes, not certificates). This permission can be set either with key vault access policies or Azure RBAC, depending on how you have Key Vault setup.
Network Access
Once Application Gateway has permissions to access Key Vault, it now needs to access it over the network. There are three options to make this work.
Open Network Access
This option is the easiest but less secure option. You can allow access to Key Vault from all networks, which will include Application Gateway. This method should only really be used if you are not concerned about locking down access to the Key Vault over the network.
Service Endpoints
If you want to lock down access to the Key Vault, one way to do this is to use service endpoints/firewalls on the Key Vault to restrict access to Application Gateway only. However, this seems to be a bit more complex than first thought, as when testing this, we saw traffic coming from both the internal and external IP of the App Gateway, so you need to configure a few things.
If you don’t do this, you will find that your deployments will hang for about 30 minutes, then give you an “internal error occurred” error.
First. you need to enable the “Allow trusted Microsoft services to bypass this firewall?” option on the firewall
Next, we need to ensure access over the private IP is allowed. To do this, we need to use the “virtual networks” option to set up a service endpoint to the virtual network that App Gateway sits in.
Finally, to deal with access from the public IP of App Gateway, we need to add the public IP address to the firewall list. This should be the single public IP from the App Gateway.
Private Endpoints
If you don’t want to deal with all the hassle above, the other option is to use Private Link/Private Endpoints to inject Key Vault into the App Gateway virtual network and access it directly over a private IP. However, this isn’t quite as simple either because you cannot add a Private Endpoint to the subnet that App Gateway is joined to. Instead, you need to create a second subnet in your virtual network (or use an existing one if you have other subnets) and attach the private endpoint to this.
Once this is done, all traffic for Key Vault will go over the Private Endpoint, and there is no further configuration needed.
