Creating Kubernetes Secrets from Azure Key Vault with the CSI Driver

Azure, Kubernetes, key vault

Azure Key Vault is an excellent solution for storing secrets in an encrypted, secure store that your application can access. If your application can talk directly to Key Vault over HTTPS, your life is easy. If you’re working with older or third-party applications where you can’t change the code to talk to Key Vault, things get a bit tougher. If you use third-party applications that require Kubernetes secrets, that can cause even more issues.

The Azure Key Vault CIS Driver is an extension for AKS that allows you to take secrets from Azure Key Vault and mount them as volumes in your pods. Your applications can then read the secret data from a volume mount inside the pod. This is great, but again it means you need to change your application to read from that volume. However, there is a way to use CSI driver also create a Kubernetes secret for your Key Vault secret. By doing this, you solve a few problems:

  • Applications that require Kubernetes Secrets can access these secrets without issue.
  • Applications that require values to be stored in environment variables can have the secret mounted as an environment variable.
  • Other, non-pod-based services can access the secret (see caveat below)

CSI Driver

Source: In preview: Azure Key Vault secrets provider extension for Arc enabled Kubernetes clusters - Microsoft Tech Community

Now, you may be thinking, why not just create Kubernetes secrets in the first place, rather than going through this hassle. This approach provides several benefits:

  • Secrets are stored at rest in Key Vault, in a secure encrypted store.
  • Secrets are only stored in the AKS cluster when a pod is running with the secret mounted. As soon as those pods are removed, the secret is removed from the cluster, rather than with Kubernetes secrets which will be retained after a pod is removed.
  • Key Vault remains the source of truth for the secret, so you can update the secret in a single place and roll it out to your clusters.
  • You can take advantage of features in Key Vault, such as auto certificate renewal or key rotation.

However, to be clear, accessing your secret using a Kubernetes secret is inherently less secure than having your application talk directly to Key Vault. Your secret is stored in the cluster either as a secret or a volume, which someone with the correct Kubernetes rights can access. If it were purely in Key Vault, then someone wanting to get the secret would need rights on the Key Vault or the ability to run commands inside the pod.

So, with all that clear, how do we go about doing this.

Create Secrets with the Azure Key Vault CSI Driver

The first thing you need to do is get the CSI driver installed on your cluster. I won’t go over doing that here, as the instructions in the MS doc are very clear, so take a look at that here. You will also need an identity for the CSI driver to access the Key Vault, either using Pod Identity or the Managed Identity assigned to your cluster.

Once you have that setup, we first need to create a SecretProvider. This CRD defines which Key Vault to access, how to access it, and what secrets to obtain.

In the example below, we are getting a secret and a key from a Key Vault, using Pod Identity to access it. You can get as many objects from Key Vault as you require; just add them to the array.

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: azure-kvname
  namespace: myNameSpace
spec:
  provider: azure
  parameters:
    usePodIdentity: "true"              
    keyvaultName: "<key Vault Name>"               
    objects:  |
      array:
        - |
          objectName: secret1          
          objectType: secret                                      
        - |
          objectName: key1               
          objectType: key
    tenantId: "<tenant ID which the Key Vault sits under"

The above YAML is ready to go if you plan on mounting the secret as a volume, but if you want to have them create a Kubernetes Secret, you need to edit it to add a SecretObjects section that looks similar to this:

  secretObjects:                             
  - secretName: appsecrets   
    data:
    - key: secret1                          
      objectName: secret1                                        
    type: Opaque

SecretObjects is an array, where each entry is a secret object, then within that secret, you can have multiple key-value pairs. The Key value can be set to whatever you want; the objectName parameter needs to match a secret you configured in the objects section. The secret type is the Kubernetes secret to create; you can create any of the available Kubernetes secret types. The final SecretProvider looks like this:

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: azure-kvname
  namespace: myNameSpace
spec:
  provider: azure
  parameters:
    usePodIdentity: "true"              
    keyvaultName: "<key Vault Name>"               
    objects:  |
      array:
        - |
          objectName: secret1          
          objectType: secret                                      
        - |
          objectName: key1               
          objectType: key
    tenantId: "<tenant ID which the Key Vault sits under"            
  secretObjects:                             
  - secretName: appsecrets   
    data:
    - key: secret1                          
      objectName: secret1                                        
    type: Opaque

Deploy this to your cluster, and you should see the SecretObject created in your desired namespace. However, you will not yet see the secret created. This is the caveat I mentioned above. The CSI driver will not generate the secret unless there is a pod with the Key Vault secret mounted as a volume, as this secret is tied to the pod’s lifecycle. No pod, no secret. Even if you never plan on using the secret through the volume mount, you still have to mount it. Otherwise, it will not be created. This issue is down to the core CSI driver rather than the Key Vault implementation, and there is a GitHub issue to look at changing this here.

So, we need to go ahead and mount this secret to a pod. To do this, you need to edit your deployment or pod YAML to add in the volume and mount.

spec:
  containers:
  - name: containerName
    image: dontainerImage
    volumeMounts:
      - mountPath: "/mnt/secrets-store"
        name: secrets-store
        readOnly: true
    env:
    - name: secert1
      valueFrom:
        secretKeyRef:
          name: appsecrets
          key: secret1
    ports:
    - containerPort: 8080
  volumes:
  - name: secrets-store
    csi:
      driver: secrets-store.csi.k8s.io
      readOnly: true
      volumeAttributes:
        secretProviderClass: azure-kv

The critical part here is the volumes section where we are creating a volume using the CSI option, with the driver set to “secrets-store.csi.k8s.io”. The secretProviderClass value needs to match the name of the SecretProvider we created above. You can see we are also mounting the Kubernetes secret we created above as an environment variable. You don’t have to do this; if you don’t need the environment variable, the secret will still exist. You do, however, need to create the volume and mount.

Once you deploy this, you should see that the Kubernetes secret is created. If you then delete the deployment, you should see the secret removed.