Hands on with the Azure Resource Manager Terraform Provider

Azure, Kubernetes, Terraform

Earlier this month at Build Microsoft announced a new feature for ARM templates, the Terraform Resource Provider. If you’re not familiar with Terraform, it’s a way to define cloud infrastructure declaratively much like ARM templates, but one of its big benefits is that it works across cloud providers. If you’re interested in learning more about Terraform in Azure then take a look at my comparison of Terraform vs ARM templates. I’ve had access to the preview of this new feature for ARM templates and wanted to share my experience.

Use Case

Generally, you’re going to be using either ARM Templates or Terraform not both, so what’s this resource provider all about? The Terraform Resource Provider lets you use Terraform resources within your ARM templates, using ARM syntax. I imagine your first question, much like mine was, is why? What’s the benefit of this mix and match approach? The main advantage of being able to us Terraform resources is that it allows you to provision non-Azure resources using your ARM template. When you’re deploying an application in Azure it is entirely likely that your also relying on some resources that sit outside of Azure, by using the Terraform provider you can declare these resources in your ARM template and manage them in the same deployment process and life cycle.

Let’s look at some examples. The Terraform provider at launch supports three resources:

  • Kubernetes
  • Cloudflare
  • Datadog

Looking closer at Kubernetes, we know we can already provide a Kubernetes cluster using an ARM template and AKS, that’s all Azure resources and easy to do. What we then get is a new cluster, with nothing running on it, but we don’t have anyway in the ARM template to deploy resources onto that AKS cluster. Let’s say that as a standard we want to implement all our AKS clusters with a deployment running Nginx that we will use for our Ingress controller, maybe we also want to deploy some secrets. Using the Terraform Kubernets provider we can now declare that within our ARM template.

Similarly, if we’re deploying an Azure web app using our ARM template and we also want to configure CDN using Cloudflare, then we can use Terraform to do this,

Let’s take a more detailed look at how we would use the Terraform provider with AKS.

Provider Registration

Before we can use Terraform resources in our ARM template, we need to register a provider for the specific type. This has two functions:

  1. Telling the ARM fabric to load and use the specific Terraform provider (in this case Kubernetes)
  2. Providing any information need to connect to the resource it is going to work on, in this example, this is the Kubernetes credentials

Because we’re doing this in an ARM template, we can use standard ARM syntax to get the Kubernetes credentials. This can either be from an AKS cluster created in the same ARM template or a separate one. The provider looks like another ARM resource, with a type of  “Microsoft.TerraformOSS/providerregistrations”. This is the same type used for all Terraform providers, we then define the specific type of provider in the properties section along with any configuration data. In this case, we are using the “Kubernetes” provider type, and specifying the Kubernetes credentials in the “inline_config” section.

{
    "type": "Microsoft.TerraformOSS/providerregistrations",
    "name": "[parameters('aksClusterName')]",
    "apiVersion": "2018-05-01-preview",
    "location": "westcentralus",
    "properties": {
        "providertype": "kubernetes",
        "settings": {
            "inline_config": "[Base64ToString(ListCredential(resourceId('Microsoft.ContainerService/managedClusters/accessProfiles', parameters('aksClusterName'),'clusterAdmin'), '2017-08-31').properties.kubeConfig)]"
        }
    }
}

Resources

Now that we have setup the resource provider we’re ready to start creating resources. As we’re using the Kubernetes provider we have access to all of the resource types provided by that, details of which can be found in the excellent Terraform documentation. We’re going to create a new Pod running Nginx (yes, you usually would be creating a deployment, but we are keeping it simple. We create a resource with the type of “Microsoft.TerraformOSS/resources” and then again in the properties section we define what kind of Terraform resource we are going to deploy, and all the settings that go along with it. We then add a dependency on the provider registration we created previously.

In the settings section, we are going to provide the values defined by the Terraform resource to determine what we are creating. In this example, we’re providing it with the name of the contianer image to grab from Docker Hub, and the port to expose on the container. There are lots more properties we could have specified including environment variables, memory and CPU, volume mounts etc.

{
    "type": "Microsoft.TerraformOSS/resources",
    "name": "NginxPod",
    "apiVersion": "2018-05-01-preview",
    "location": "westcentralus",
    "properties": {
        "providerId": "[resourceId('Microsoft.TerraformOSS/providerregistrations', parameters('aksClusterName'))]",
        "resourcetype": "kubernetes_pod",
        "settings": {
            "metadata": [
                {
                    "name": "nginx",
                    "labels": {
                        "App": "nginx"
                    }
                }
            ],
            "spec": [
                {
                    "container": {
                        "image": "nginx:1.7.8",
                        "name": "nginx",
                        "port": {
                            "container_port": 80
                        }
                    }
                }
            ]
        }
    },
    "dependsOn": [
        "[resourceId('Microsoft.TerraformOSS/providerregistrations', parameters('aksClusterName'))]"
    ]
}

We can now go and deploy the ARM template as we would normally, and it will create the pod in our already provisioned AKS cluster.

Oddities

As I went through this process, I notice a few things that just felt a little odd. Firstly, as this is still in private preview, it’s only available on one region, West Central US. This was an odd choice I thought, as I was trying to test against an AKS cluster, and AKS is not present in West Central US. However, if I set the location from my Terraform resources to West Central US, and then run the ARM template against an AKS cluster in West Europe, it still works.

Related to this, when you run the deployment, it creates some resource objects in your select resource group, one for the provider and one for the pod. These are odd in a couple of ways, firstly they show up as being in the West Central US region, despite the actual pod being in West Europe, I don;t know if this has any concrete impact on anything, or it’s just a label. Secondly, there is nothing you can do with them. The objects exist in ARM, but you can’t delete them, edit them or anything much really, although oddly you can assign RBAC permissions to them. Maybe this will expose some functionality later, but at the moment it’s a bit weird, especially when there’s no way to tie the resources to the thing that is hosting them (in this case AKS).

Conclusion

I’m a bit in two minds on this one. On the face of it, it seems like a good idea to be able to bring resources that are not in Azure, but that your Azure apps are dependant on, into your deployment. However, the implementation still feels a bit odd, having to define a resource provider for a resource that you possibly created in the same template, and the way this creates objects in ARM that aren’t real ARM resources makes the process feel a bit disjointed and confusing. There are now two different processes depending on the kind of resource I want to create, and two different ways I am going to need to manage those resources once created. I also wonder; if you’re going to be using a lot of these resource providers in your templates, at what point does it just make more sense to switch to using Terraform for everything and get a much more joined-up experience, with the other benefits you get from Terraform.

All of that said, the process for actually authoring and using the Terraform provider in an ARM template was pretty straightforward and converting between the Terraform and ARM syntax wasn’t a problem. The ability to use values from native ARM resource created earlier in the template inside the Terraform resources works well and does make integrating the two easy. All in all, the authoring process was well done, my concern mainly is around the life cycle management of the two resource types together.

In the future, as more providers are added, this will definitely give more flexibility when writing ARM templates for applications, where you do need to bring in some non-Azure resources. This will be especially useful where you’re only using the odd one or two non-Azure resources.  Microsoft has stated that they do plan on supporting as many Terraform providers as is viable, although unsurprisingly they don’t plan on supporting resources from other cloud vendors. It will be interesting to see what happens when this moves into public preview and what uses cases people can come up with.

If you would like to try this out it’s currently in private preview, but you can apply for access here.