The ARM TTK Azure DevOps Extension is now Cross Platform!

TTK, Azure DevOps

Back in 2020, I released an Azure DevOps extension to allow you to run the Azure Resource Manager Template Toolkit (TTK) as part of your build pipelines. This allows you to run the TTK tests against your templates, get the results in a structured format and fail your builds if they fail the tests. Over the last few years, I have updated this with various new features such as support for Bicep files, excluding and including specific tests, skipping tests, etc. However, one item that remained open for a long time was support for running the extension on Linux build agents. The extension uses PowerShell to run the tests, but the actual extension was written in PowerShell as well, so it only supported Windows Agents. To switch to a cross-platform extension, it would need to be re-written in TypeScript, which I had no experience in, so it got ignored for a while.

Recently, I came across an article by Richard Fennel on a technique for porting PowerShell extensions to TypeScript. This essentially creates a runner in TypeScript that can then execute PowerShell for you, supporting PowerShell core so it can work on Linux. Using this, I took my existing extension, moved all the PowerShell logic inside this runner, and got it to work on Windows and Linux.

You can get the new cross-platform extension here -Run ARM TTK Tests (Cross Platform) - Visual Studio Marketplace. This is a new extension, rather than an update to the existing one, as you can’t easily switch between PS and TS extensions. This also means you can install and test this extension with the old one and switch when you are happy that it works for you. I’ll be maintaining them both for a while but will eventually deprecate the Windows-only one.

Whilst I’ve done quite a lot of testing on this version of the extension, I am sure there will be issues. If you do encounter any, please raise them in the GitHub repo here.

Using the Extension

Once you have installed the extension, you can create a task in your pipeline using the RunARMTTKTestsXPlat@1 extension. The minimum details you need to pass are the path to your templates and the location you want the test result files to be output to.

- task: RunARMTTKTestsXPlat@1
  inputs:
    templatelocation: '$(System.DefaultWorkingDirectory)\templates'
    resultLocation: '$(System.DefaultWorkingDirectory)\results'

There are several optional settings you can enable such as:

  1. A comma-separated list of tests to run, if you provide this list, only the tests provided will be run, and all other tests will be skipped. Leave blank to run all tests. If the test names are incorrect, then all tests will run. The complete list of test case names can be found in the ARMTTK here.
  2. A comma-separated list of tests to skip, all other tests will be run. Leave blank to run all tests. The complete list of test case names can be found in the ARMTTK here.
  3. A comma-separated list of files to treat as the “main template” for the purpose of tests that require this, such as the “Location must not be hardcoded” test.
  4. A boolean to indicate whether to treat all templates as the “main template” for the purpose of tests that require this, such as the “Location must not be hardcoded” test - defaults to false.
  5. A boolean to output additional test result summary to the Azure DevOps CLI - defaults to false.
  6. A boolean to indicate whether we should recurse through subfolders to find files to test - defaults to true
  7. A boolean to indicate whether we should ignore the exit code of the tests and so not fail the build on a failed test (advanced section)
  8. A boolean to indicate whether we should use PowerShell Core on Windows machines. On Linux, PowerShell core will always be used (advanced section)
- task: RunARMTTKTestsXPlat@1
  inputs:
    templatelocation: '$(System.DefaultWorkingDirectory)\templates'
    resultLocation: '$(System.DefaultWorkingDirectory)\results'
    includeTests: 'VM Images Should Use Latest Version,Resources Should Have Location'
    skipTests: 'VM Images Should Use Latest Version,Resources Should Have Location'
    mainTemplates: 'template1.json, template2.json'
    allTemplatesMain: false
    cliOutputResults: true
    recurse: true
    ignoreExitCode: false
    usePSCore: true

Displaying Test Results

A key thing to note is that this extension does not publish the results of the tests to show in the Azure DevOps UI itself; you need to use the “Publish Test Results” extension to read the XML files and publish the results so you can see a test report in the UI. All you need to do is pass the results folder path with a filter looking for armttk results files, e.g.:

- task: PublishTestResults@2
  inputs:
    testResultsFormat: 'NUnit'
    testResultsFiles: '$(System.DefaultWorkingDirectory)\results\*-armttk.xml'
  condition: always()

If any of your tests fail, the RunARMTTKTests task will also fail. To ensure that you always publish your test results, make sure you use the condition: always() setting so this always runs.

Once you do this, Azure DevOps will show the results of your tests in the build.

Results