ARM TTK Extension for Azure DevOps

A couple of weeks ago, I published an article where we looked at the Azure Resource Manager Template Tool Kit or ARM TTK. This is a PowerShell module that you can use to test your ARM templates against best practices. If you’d like to learn more about the Tool Kit take a look at the article here.

With Infrastructure as Code platforms like ARM Templates, many people are adopting development methodologies with their templates, like using version control and creating build and release pipelines. Given this, a great place to run these ARM TTK tests is as part of your pipelines. If you automatically run your tests whenever you commit a change, you get quick feedback about whether your templates are following best practice.

To make this process easier, I have created an Azure DevOps extension that lets you run the ARM TTK against your templates as part of a build pipeline. You can get this extension here.

Using the Extension

Once you install the extension from the marketplace, it is available for use in your pipelines. Let’s see how to set it up.

The first thing is creating a new pipeline in Azure DevOps and linking it to the Git repository that holds your templates; this is where it will pull the template files you want to test from.

Next, you will be asked for the type of pipeline, I would recommend using YAML based pipelines rather than the classic editor, and select the starter pipeline option

This will create a new pipeline that is configured to talk to your git repo. It will be set up to use an Ubuntu build agent; at the present time, this extension only works on Windows. I will be looking to make it cross-platform in the future, but initially learning to create Azure DevOps extensions and Typescript at the same time was not happening! So need to change this to use a Windows build agent.

trigger:
- master

pool:
  vmImage: 'ubuntu-latest'

steps:

Next, we can configure the steps to run our tests. The starter pipeline created some script steps, remove both of this, so you are just left with this:

trigger:
- master

pool:
  vmImage: 'windows-latest'

Underneath the steps, we are going to add the test extension. To make it easier, you can use the task assistant on the right and search for TTK.

When you select it, you need to provide 2 parameters:

1. Template Folder or Template - the path to either the template you want to test, a folder containing templates you want to test, or a path with wildcards. This determines what files the tests are run against. You don’t need to filter for just template files; the extension will do this
2. Results Location - a path to where the test results XML files can be output. We will use these later to display our test results.

Once you input these add the task your YAML should now look like this:

trigger:
- master

pool:
  vmImage: 'windows-latest'

steps:
- task: RunARMTTKTests@1
  inputs:
    templatelocation: '$(System.DefaultWorkingDirectory)\templates\'
    resultLocation: '$(System.DefaultWorkingDirectory)\results'

As things stand, we could run this build, and it would run our test. However, all we would see is a pass or fail for the task step and not any useful information on individual tests.

The ARMTTK extension doesn’t publish the results of the tests to Azure DevOps its self; instead, we rely on the “Publish Test Results” extension. We can again find this in the Tasks helper, and it requires two parameters:

1. The format of the results of the test - the ARM TTK Extension publishes results in the NUnit format
2. A path to the results files. We will point to the results path we entered above, filtering for files with “-armttk.xml” at the end.

One other setting we need to change is the condition when this task runs. By default, tasks only run when the previous tasks succeed. However, our test task will fail if it has failed test results, so we need to configure the publish task to run regardless of the state of the previous task, as we always want to see our test results.

Our YAML file will now look like this:

trigger:
- master

pool:
  vmImage: 'windows-latest'

steps:
- task: RunARMTTKTests@1
  inputs:
    templatelocation: '$(System.DefaultWorkingDirectory)\templates\'
    resultLocation: '$(System.DefaultWorkingDirectory)\results'
- task: PublishTestResults@2
  inputs:
    testResultsFormat: 'NUnit'
    testResultsFiles: '$(System.DefaultWorkingDirectory)\results\*-armttk.xml'
  condition: always()
    

That is all we need. We can save this build and then run it. Our tests will get run, and shortly after the build completes, you will see test results appear in the tests tab, and the test task will have succeeded or failed based on these tests.

By default you will only see details of the failed tests, if you want to see details for the passed test as well, change the filter list.

Each ARM template you test will get its test section:

I’ve not yet figured out how to get the file name into this header, but if you expand it, you will see the file name in question in the test results for each test.

Future Development

This is my first attempt at both building an Azure DevOps extension and working with NUnit test result format, so I am sure there ways of doing this better and maybe some issues with what I have done. If you do spot any problems, please do raise an issue on the GitHub repo here. If you have suggestions for improvement, please do raise them, or consider submitting a pull request.

As I mentioned, my next task will be looking to make this work cross-platform. I’ll also be looking to add some more tests into the TTK to further expand its capabilities.

I hope you find this useful and do let me know if there is anything you would like to see done better.