Automating Deployment to an AKS Cluster with Local Auth Disabled Using

Azure, Pulumi, Infrastructure as Code

One of the benefits of using a multi-cloud Infrastructure as Code tool like Pulumi or Terraform is that you can easily transition between layers in different providers. For example, you can deploy a Kubernetes cluster in Azure with AKS, and then using the same IaC deploy pods and services into that cluster. To do this, the IaC tool needs to retrieve some credentials in the form of a Kubeconfig file, from the cluster it has just deployed. Using the standard AKS deployment this is pretty easy, as the cluster has a built-in cluster admin credential that can be easily obtained in Pulumi by using the ListManagedClusterAdminCredentials method. You can feed this to the Kubernetes provider and you can deploy to AKS with admin rights.

However, recently Microsoft has added a feature to AKS to be able to disable local users in AKS, which includes this local admin user. If you turn this on then you can’t easily retrieve an admin Kubeconfig from the cluster, and the only way to connect is with Azure AD accounts. This feature is something that a lot of people are going to want to turn things on, as that local admin account is a bit of a security risk. So how do we continue being able to undertake automated deployments that easily transition between Azure and Kubernetes in the same IaC?

Automation Account Creation

We need an account to be able to login to AKS, and when local accounts are disabled it has to be an Azure AD account. Using a normal user isn’t going to work, but we can use a Service Principle or, if your deployment is running from an Azure VM, a managed identity. Either of these provides us with an Azure AD user that is designed for automated and non-interactive use.

So the first thing to do is create either a Service Principle or Managed Identity that you want to use for this process. If you are already doing your deployments from an automated process (like a pipeline) then you probably already have a service principle created that you can use.

Automation Account Permissions

To be able to access Kubernetes the service principle needs to be granted permissions in Kubernetes. Thankfully, because we are using an Azure AD account, and AKS now supports Azure RBAC permissions for Kubernetes, we can grant those permissions without needing to be able to connect to Kubernetes, otherwise, we’d be a bit stuck. We can create this role assignment in our Pulumi code as below (C# example):

 var roleAssignment = new AzureNative.Authorization.RoleAssignment("aksDeploymentAdmin", new()
    {
        PrincipalId = "<principal ID of service principle or managed identity",
        PrincipalType = "ServicePrincipal",
        RoleAssignmentName = "<Random GUID>",
        RoleDefinitionId = "/subscriptions/<subscriptionId>/providers/Microsoft.Authorization/roleDefinitions/b1ff04bb-8a4e-4dc4-8eb5-8693973ce19b",
        Scope = "<Resource ID of AKS, or resourcegroup AKS is in>",
    });

The role definition of “b1ff04bb-8a4e-4dc4-8eb5-8693973ce19b” translates to the “Azure Kubernetes Service RBAC Cluster Admin” role, which grants full control inside Kubernetes. There are other, more limited roles that you can use should you want less access, to find them visit [Az Advertiser](AzRolesAdvertizer (azadvertizer.net)) and search for “Azure Kubernetes Service” in the role box. If you are applying this at the cluster level then it needs to be assigned after AKS is deployed.

Install Kubelogin

Because we are using an Azure AD credential to login to AKS we can’t just rely on using the libraries provided by Kubectl. Azure AD login to AKS is managed using a tool called Kubelogin and so we will need to make sure this is installed on the machine where you will be running your deployments. We will be using Pulumi’s ability to run code in your programming language of choice to call out to Kubectl.

You can install Kublogin through various means listed [here](Installation - Azure Kubelogin)

Generating Kubeconfig

Now we have everything set up we are going to generate a Kubeconfig file for our service principle/managed identity using Kubelogin. There is no declarative way to do this with native Pulumi code, so we are going to need to use the ability to run arbitrary code from your language of choice to call Kubelogin and generate this data. I’m going to be using C# for this example but it should work in any of the other languages Pulumi supports.

We’re going to create a method that we can then call from our Pulumi code, rather than just sticking it inline, to make it a bit easier to manage. This method needs to be provided with three parameters:

  • Resource group of the AKS cluster
  • Name of the AKS cluster
  • An indicator of whether this is a service principle or managed identity

To make life easier I created an Enum for the last variable.

enum userType {
    ServicePrincipal,
    ManagedIdentity,
    User
}

Our method is going to do the following:

  • Set up the required environment variables and parameters to KubeLogin based on whether it is a Service Principal or Managed Identity. This defines how we authenticate with this user before generating the Kubeconfig
  • Run the ListManagedClusterUserCredentials Pulumi command, this will generate a Kubeconfig file that is missing credentials, we need this as the Kubelogin command will just generate the required certificate, not a full Kubeconfig file so we use this as a template to add our admin credentials into.
  • Write out the Kubeconfig file to a temporary text file that we can then use to merge our credentials into
  • Run the Kubelogin command with the appropriate parameters to generate the credentials for your use. We set the KUBECONFIGenvironment variable to point to the text file we generated earlier so that these credentials get automatically added to this file and we have a fully working Kubeconfig file.
  • Read the content of the Kubeconfig file and return this from our method as a string.
private async Task<string> GetKubeConfigForUser(string resourceGroupName, string clusterName, UserType userType)
        {
            if (userType == UserType.User)
            {
                throw new NotSupportedException(
                    "Using user credentials is not supported for clusters with local accounts disabled, use a service principal or managed identity instead.");
            }

            //Add additional arguments to the kubelogin command based on the type of account used
            string additionalKubeLoginArgs ="";

            switch (userType)
            {	// Check if the correct environment variables have been set for Service Principle.
                case UserType.ServicePrincipal:
                {
                    if (string.IsNullOrEmpty(Environment.GetEnvironmentVariable("AAD_SERVICE_PRINCIPAL_CLIENT_ID")) ||
                        string.IsNullOrEmpty(Environment.GetEnvironmentVariable("AAD_SERVICE_PRINCIPAL_CLIENT_SECRET")))
                    {
                        //If Kubelogin SP environment variables are not set, use the current SP credentials from the Pulumi envs
                        if (!(string.IsNullOrEmpty(Environment.GetEnvironmentVariable("ARM_CLIENT_ID")) &&
                              string.IsNullOrEmpty(Environment.GetEnvironmentVariable("ARM_CLIENT_SECRET"))))
                        {
                                //Use Pulumi envs, needs the client secret and ID params set to empty, otherwise, it doesn't write them, see https://github.com/Azure/kubelogin/issues/196
                                additionalKubeLoginArgs = "--use-azurerm-env-vars --client-secret \"\" --client-id \"\"   ";
                        }
                        else
                        {
                            throw new MissingFieldException(
                                "ARM_CLIENT_ID and ARM_CLIENT_SECRET or AAD_SERVICE_PRINCIPAL_CLIENT_ID and AAD_SERVICE_PRINCIPAL_CLIENT_SECRET environment variables must be set to use service principle auth to AKS");
                        }

                    }

                    break;
                }
                case UserType.ManagedIdentity:
                    additionalKubeLoginArgs = $"--client-id {tokenUtils.AppId}";
                    break;
            }
    	// Generate a Kubeconfig using the ListManagedClusterUserCredentials command to give us a template
      var clusterUserCredential = await ListManagedClusterUserCredentials
                .InvokeAsync(new ListManagedClusterUserCredentialsArgs
                {
                    ResourceGroupName = resourceGroupName,
                    ResourceName = clusterName
                }).ConfigureAwait(false);
    


             var b64Config = clusterUserCredential.Kubeconfigs.First().Value; 
             var kubeConfigBytes = Convert.FromBase64String(b64Config); 
             var kubeConfig = Encoding.UTF8.GetString(kubeConfigBytes);
             var tempFile = Path.Join(Path.GetTempPath(), "kubeconfig"); 
             await File.WriteAllTextAsync(tempFile, kubeConfig).ConfigureAwait(false);
             
  	//Create Kubelogin Command
            var kubeLoginCommand =
                $"kubelogin convert-kubeconfig -l {tokenUtils.KubeLoginType} {additionalKubeLoginArgs}";
      
             await Run.InvokeAsync(new RunArgs
             {
                 Command = kubeLoginCommand,

                 Environment = { { "KUBECONFIG", tempFile }, }
             }).ConfigureAwait(false);

    		// Retrieve the full Kubeconfig and return
             var content = await File.ReadAllTextAsync(tempFile).ConfigureAwait(false);
             return content; 

        }
    }

Retrieving Kubeconfig

Now we have our method, we can call it from our Pulumi code to retrieve the Kubeconfig and pass it to our Kubernetes provider. A couple of things to be aware of:

  • We need to ensure this runs after we create AKS, so we ensure we use the AKS resource in the apply statement to add the dependency.
  • We use Output.CreateSecret to ensure that the Kubeconfig is encrypted when written to the state file

    var kubeConfig = Output.Tuple(aks.ResourceGroupName, aks.Name).Apply(details => Output.CreateSecret(details.Item1, details.Item2, UserType.ServicePrinciple))

We can now use this Kubeconfig variable to create a Kubernetes provider that we can then use to create any Kubernetes objects.

var k8Provider = new Pulumi.Kubernetes.Provider("k8Provider", new Pulumi.Kubernetes.ProviderArgs(){
   Kubeconfig = kubeconfig,
   EnableServerSideApply = true
});