How to Avoid Breaking The AKS Flux Extension with Pod Identity

Azure, Kubernetes, Flux

In my continuing journey with using Flux for GitOps on AKS, I set up my configuration to deploy Pod Identity to my cluster using the Pod Identity Helm chart (rather than the AKS addon) and pushed it to Git. Flux, as expected, picked this up, deployed Pod Identity and then broke… As soon as Pod Identity was installed, Flux refused to process any more updates from Git, and if I looked at the status of the extension, it was stuck in the “updating” state.

After a lot of confusion, I checked the logs in the Flux agent pod and found the problem. The agent was complaining about being unable to authenticate its identity to Azure.

Error: in getting auth header : error {adal: Refresh request failed. Status Code = '404'. Response body: no azure identity found for request clientID 1dc4##### REDACTED #####5948\n Endpoint http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01\u0026client_id=1dc44d7c-24bc-4ca8-a375-67f8f8a55948\u0026resource=03db181c-e9d3-4868-9097-f0b728327182}","LogType":"ConfigAgentTrace","LogLevel":"Information","Environment":"prod","Role":"ClusterConfigAgent","Location":"eastus","ArmId":"/subscriptions/xxxxxxx/resourceGroups/akspi/providers/Microsoft.Kubernetes/managedclusters/akspi","CorrelationId":"","AgentName":"FluxConfigAgent","AgentVersion":"0.5.1","AgentTimestamp":"2022/02/05 11:44:15"}

The documentation doesn’t state it, but my guess is that the agent uses the cluster’s identity to communicate back to the Azure Data Plane Service in the highlighted section of this diagram.

So why does Pod Identity break this? Pod Identity deploys a service called the Node Managed Identity (NMI), which intercepts any requests from pods in the cluster to the Managed Identity service and redirects them to use the identity assigned to them through Pod Identity. The Flux agent doesn’t have a Pod Identity assigned to it, so the request gets intercepted, and then the NMI service throws an error because there is no identity to use. We can’t associate an Identity with Flux, as it’s a managed extension, and we don’t have access to amend the deployment. So what do we do?

We need to have Flux ignore Pod Identity and continue to use the cluster identity it was using before. There is a way to do this through Pod Identity Exceptions. This resource tells Pod Identity not to intercept any requests for specific resources. There is an exception already in place in the cluster for the Kube-System namespace, as this is added by Pod Identity automatically.

A Note on Using Kube-System

One of the first solutions I thought of when I hit this issue was to deploy the Flux pods into Kube-System. This is possible to do by just changing the namespace used when you deploy the Flux extension, and the Pods get deployed and start correctly. However, I found that none of my Flux configurations were applied when I did this, and I got errors about the service account for Flux not having the correct permissions. It may be possible to resolve this, but I didn’t like the idea of having my Flux resources in Kube-System, so I didn’t spend a lot of time trying to get to the bottom of it.

Fixing the Issue

We need to add an exception that stops Pod Identity redirecting requests from the Flux-System namespace to resolve the issue. We need to time adding this exception so that it is in place before both Flux and Pod Identity are in place to avoid things getting stuck. How we do this varies depending on whether you use the AKS extension for Pod Identity or you deploy it using the Helm Chart.

Exceptions are deployed against labels on pods. Fortunately, all the Flux pods have a common label:

app.kubernetes.io/name=microsoft.flux

AKS Extension

I had hoped the extension would deal with this automatically, given that AKS should know if both Flux and Pod Identity are installed but unfortunately, it does not. Because you are likely deploying Flux and Pod Identity at the same time when you deploy the cluster, adding the exception using Flux is not an option, as it will already be broken. If you happen to be installing the Pod Identity extension after Flux, see the steps for the Helm chart, but assuming that is not the case, you will need to deploy the exception outside of Flux.

Deploying the exception can be done using the CLI, with the following command:

az aks pod-identity exception add --cluster-name <aks cluster name> \
                                  --namespace flux-system \
                                  --pod-labels "app.kubernetes.io/name=microsoft.flux" \
                                  --resource-group <aks cluster resource group> \

Alternatively, you can apply the exception using a Kubernetes YAML file:

apiVersion: aadpodidentity.k8s.io/v1
kind: AzurePodIdentityException
metadata:
  name: flux-exception
  namespace: flux-system
spec:
  podLabels:
    app.kubernetes.io/name: microsoft.flux

Using the Kubernetes YAML, you could deploy this using an Infrastructure as Code solution when deploying the cluster, such as Terraform or Pulumi. Unfortunately, Bicep/ARM doesn’t support deploying resources onto Kubernetes. This is being considered in the extensibility project but is not currently present. The closest you could get is to use a deployment script to access your cluster and deploy it.

If you deployed the Flux extension before you created the exception, you might find that extension is in a failed state. The only way I could resolve this was to remove and re-install the extension with the exception in place.

Helm Chart

If you deploy Pod Identity using the Helm chart, and so as part of your Flux configuration, then you can resolve this issue entirely with a change to the Flux configuration. To do this, you need to add an additional configuration to your Flux resource. If you read last weeks article on setting up Flux with IaC you may recall we deployed two configurations, infrastructure and apps:

resource fluxConfig 'Microsoft.KubernetesConfiguration/fluxConfigurations@2021-11-01-preview' = {
  name: 'gitops-demo'
  scope: aks
  dependsOn: [
    flux
  ]
  properties: {
    scope: 'cluster'
    namespace: 'gitops-demo'
    sourceKind: 'GitRepository'
    suspend: false
    gitRepository: {
      url: 'https://github.com/fluxcd/flux2-kustomize-helm-example'
      timeoutInSeconds: 600
      syncIntervalInSeconds: 600
      repositoryRef: {
        branch: 'main'
      }

    }
    kustomizations: {
      infra: {
        path: './infrastructure'
        dependsOn: []
        timeoutInSeconds: 600
        syncIntervalInSeconds: 600
        validation: 'none'
        prune: true
      }
      apps: {
        path: './apps/staging'
        dependsOn: [
          {
            kustomizationName: 'infra'
          }
        ]
        timeoutInSeconds: 600
        syncIntervalInSeconds: 600
        retryIntervalInSeconds: 600
        prune: true
      }
    }

The infrastructure configuration ran before the apps one due to the dependency. In this scenario, we are deploying Pod Identity in the infrastructure config. If we run this code, the infrastructure would deploy fine, but apps would not because Pod Identity is installed in the infrastructure section.

Unfortunately, we can’t just add an exception in the infrastructure configuration (trust me, I tried). Due to the way Flux and Kustomize work, all the YAML in the infrastructure config is merged and run simultaneously, so when you try and create the exception, it fails because the custom resource definition (CRD) for exceptions doesn’t exist.

To get around this, we need to install the Pod Identity CRD’s before running our infrastructure configuration. If we do that, we can add the exception in the infrastructure configuration because the CRD’s exist.

Create Prerequisite Configuration

First thing, we need to create a new set of Kustomize resources. We are using the example setup here, and we will create a new folder at the top level called “prerequisites”

Folders

There are two ways you can pull in the CRD’s when you run the configuration. You can pull them directly from the Pod Identity repository (assuming your cluster has internet access), or you can download the CRD’s into your configuration and run them from there. Pulling them directly from the Pod Identity repo will ensure you get the latest versions, but doing this can be a security concern for some.

To pull them directly from the Pod Identity repo, you should create a “customization.yaml” in the prerequisites folder with the following content:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Customization

resources:
- https://raw.githubusercontent.com/Azure/aad-pod-identity/master/charts/aad-pod-identity/crds/crd.yaml

If you would prefer to store the CRD’s in your configuration, you should first create a file in the prerequisites folder called “pod-identity-cards.yaml” (or similar), then paste in the CRDs from the GitHub URL. You should then create a “customization.yaml” file which looks like this:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Customization

resources:
- pod-identity-crds.yaml

Update Infrastructure Deployment

Next, we need to edit our infrastructure deployment to create the exception. First, we create a file called “flux-exception.yaml” (or similar) that contains the exception information:

apiVersion: aadpodidentity.k8s.io/v1
kind: AzurePodIdentityException
metadata:
  name: flux-exception
  namespace: flux-system
spec:
  podLabels:
    app.kubernetes.io/name: microsoft.flux

Next, we edit the Customization.YAML file in the infrastructure folder to add this exception:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Customization
resources:
  - sources
  - nginx
  - redis
  - flux-exception.yaml

You can, of course, put this file in a subfolder with its own Customization.YAML if you want.

You can see a complete example of this setup in this GitHub repo.

Update Extension Configuration

Finally, we need to add another configuration in the “customization” section to use these files and add a dependency for infrastructure, so prerequisites is deployed first.

    kustomizations: {
      prereqs: {
        path: './prerequisites'
        dependsOn: []
        timeoutInSeconds: 600
        syncIntervalInSeconds: 600
        validation: 'none'
        prune: true
      }
      infra: {
        path: './infrastructure'
        dependsOn: [
        {
        	kustomizationName: 'prereqs'
        }
        ]
        timeoutInSeconds: 600
        syncIntervalInSeconds: 600
        validation: 'none'
        prune: true
      }
      apps: {
        path: './apps/staging'
        dependsOn: [
          {
            kustomizationName: 'infra'
          }
        ]
        timeoutInSeconds: 600
        syncIntervalInSeconds: 600
        retryIntervalInSeconds: 600
        prune: true
      }
    }

If you’re deploying your config using the Azure CLI, then you can do this by adding another Customization:

--kustomization name=infra path=./infrastructure dependsOn=["prereqs"] prune=true --kustomization name=apps path=./apps/staging prune=true dependsOn=["infra"] --kustomization name=prereqs path=./prerequisites prune=true

You should now see that the prerequisites are deployed first, creating the required CRD’s and allowing us to create the exception for the Flux resources. This should result in both Flux and Pod Identity continuing to work.