Building An Infrastructure Pipeline Part 2 – Testing

2017-11-27

ARM, Pester, Devops, PowerShell, Testing

Don’t forget to check out the other parts in this series:
Part 1 – Version Control
Part 2 – Testing
Part 3- Build

In part 1 of this series we took our first step in building an Infrastructure as code pipeline by moving our infrastructure files into source control. Now that we have this code in a central repository we can use this as a base for the next steps in this process. In this article we’re going to discuss testing, and in particular automated testing. For many IT pro’s this may be a bit of an alien concept! Historically testing in this area has often been limited to manual validation that machines are provisioned as expected and then ongoing monitoring. Now we’re in a world where our infrastructure is defined as code, it means we can test it like code, even before we’ve deployed a single VM. By undertaking this testing we can catch issues and errors before we deploy costly resources and before problems get into production where they are much more difficult to fix.

Test Types

Testing of infrastructure code can take many forms, we’re going to focus on these key areas:

File Validation
Syntax/Linting Tests
Content Validation
Template Validation
Deployment Testing

Testing Framework

You can really use any testing framework you like, but as using PowerShell is one of the easiest ways to interact with Azure and ARM I recommend using the Pester testing framework, which allows you to test using PowerShell. We’re not going to cover the intricacies of Pester in this article, so I recommend reading this introduction to Pester. To be able to run the samples in this article you will need to have pester installed, the easiest way is to just install from the PowerShell Gallery:

Find-Module pester -Repository psgallery | Install-Module

Test Setup

Before we run any tests we need to do some setup. We’re going to run tests 1-4 in this example in a single Pester File, in real life you might want to split this into separate files to make it easier to organise. A full example of this file and a sample ARM template to test against can be found on Github.

In tests 1-4 we’re not going to create any Azure resources, but some of the tests do actually require a resource group to exists. We’re going to uses Pesters BeforeAll and AfterAll statments to do this.

So first things in our test file, we are going to setup some variables for the locations of the files we want to test and the resource groups we want to test

$here = Split-Path -Parent $MyInvocation.MyCommand.Path 
$template = Split-Path -Leaf $here 
$ShortGUID = ([system.guid]::newguid().guid).Substring(0, 5) 
$TempValidationRG = "$ShortGUID-Pester-Validation-RG" 
$location = "West Europe"

Then we will start our Pester Describe statement (basically a container for your tests) and create the resource group, after all our tests we will destroy it.

Describe "Template: $template" { 
    BeforeAll { 
        New-AzureRmResourceGroup -Name $TempValidationRG -Location $Location 
    } 
    <tests go here> 
    AfterAll { 
        Remove-AzureRmResourceGroup $TempValidationRG -Force 
    } 
}

Testing

Now we have our test file setup, we’re ready to add some tests. I’m just going to show you the basic types of tests you can undertake here, there are hundreds of variations of these you could employ to test each individual resource in your template.

File Validation

The first test we are going to do is to validate that the files we expect to see are present. We’re doing an ARM template deployment so we expect a template file, a parameters file and maybe a metadata file. so we will test that these files exist. In this example we don’t have a metadata file, so we can see what a failure looks like.

We’re using Pester “It ” blocks to describe our tests and the “Should” command to validate whether they pass or fail. We are also going to wrap this section in a “Context” block, which is just another container for describing this section of the code

 Context "Template Syntax" { 
    It "Has a JSON template" { 
         "$here\azuredeploy.json" | Should Exist 
    } 

    It "Has a parameters file" { 
        "$here\azuredeploy.parameters.json" | Should Exist 
    } 

    It "Has a metadata file" { 
        "$here\metadata.json" | Should Exist 
    } 
}

If we run this, which you should be able to do by just pressing run in your PoweShell IDE of choice, we can quickly see the result of our tests.

As expected we see a failure for the metadata file, but pass the other two files.

Syntax/Linting

The first set of tests we are going to run are pretty simple, all they are going to do is check your code for syntax errors. This can save a pretty significant amount of time by quickly spotting that missing comma or brace before you try and deploy. We are also going to check that the ARM template has all the expected sections required for valid syntax.

The actual test we are running is very straight forward, we’re just using PowerShell to import the file and convert it from JSON into a PowerShell object, this will parse the JSON in the file and if it has any syntax errors it will generate an error, which will result in a failed test. We then check this parsed object for the required sections.

 It "Converts from JSON and has the expected properties" { 
    $expectedProperties = '$schema', 'contentVersion', 'parameters', 'variables', 'resources', 'outputs' 
    $templateProperties = (get-content "$here\azuredeploy.json" | ConvertFrom-Json -ErrorAction SilentlyContinue) | Get-Member -MemberType NoteProperty | % Name
    $templateProperties | Should Be $expectedProperties 

}

Content Validation

We now know we have the right files in place, and that there are no syntax errors in them. We’re now going to move onto tests which will be more specific to your ARM template. We now want to validate that the resources we expect to be configured in the ARM template are actually defined. To do this we are first going to check that the top level resources we expect to be defined are present, we we will parse the template and cross check the result against what we expect to see. In the example below we are check that the template does create a network, public IP, load balancer etc. If any one of these are missing it will fail.

 It "Creates the expected Azure resources" { 
     $expectedResources = 'Microsoft.Storage/storageAccounts', 'Microsoft.Network/virtualNetworks', 'Microsoft.Network/publicIPAddresses', 'Microsoft.Network/loadBalancers', 'Microsoft.Compute/virtualMachineScaleSets', 'Microsoft.Automation/automationAccounts', 'Microsoft.Insights/autoscaleSettings'
     $templateResources = (get-content "$here\azuredeploy.json" | ConvertFrom-Json -ErrorAction SilentlyContinue).Resources.type $templateResources | Should Be $expectedResources 
}

This is a fairly simple example, you could off course go more granular and deeper in your validation of resources and perhaps quantity of resources and even split these out into their own individual tests. You could go as far as validating every attribute in your ARM template if you wanted to.

As an example of this, we know from the test above that we are creating some virtual machines, but we want to get more detailed and check that these VM’s also get the DSC extension installed and that we are passing the correct parameters to the DSC file, so we’ll do a similar test to the one above, but select the DSC extension object and parse that.

 It "Contains the expected DSC extension properties" {
     $expectedDscExtensionProperties = 'RegistrationKey', 'RegistrationUrl', 'NodeConfigurationName', 'ConfigurationMode', 'ConfigurationModeFrequencyMins', 'RefreshFrequencyMins', 'RebootNodeIfNeeded', 'ActionAfterReboot', 'AllowModuleOverwrite', 'Timestamp' 
     $dscExtensionProperties = (get-content "$here\azuredeploy.json" | ConvertFrom-Json -ErrorAction SilentlyContinue).Resources | ? type -eq Microsoft.Compute/virtualMachineScaleSets | % properties | % virtualMachineProfile | % extensionProfile | % extensions | ? name -eq Microsoft.Powershell.DSC | % properties | % settings | % Properties | % Name $dscExtensionProperties | Should Be $expectedDscExtensionProperties 
 }

Template Validation

We’ve now tested our files, syntax and the content of our files, the next test before we deploy some resources is to check that the template as a whole is actually valid. We know there are no syntax errors from a JSON perspective, but we still could have issues with using the wrong names for attributes, incorrect version numbers etc. To test this we are going to use the Test-AzureRMResourceGroupDeployment PowerShell cmdlet. This command lets send your template to the ARM fabric and have it validation, as if you were doing a deployment, but not actually deploy anything. This is the reason why we need to create a resource group before running our tests, as it requires a resource group to run against. Our test is very simple, just running this command against our deployment and checking the result.

Context "Template Validation" { 
   It "Template $here\azuredeploy.json and parameter file passes validation" { 
       # Complete mode - will deploy everything in the template from scratch. If the resource group already contains things (or even items that are not in the template) they will be deleted first. 
       # If it passes validation no output is returned, hence we test for NullOrEmpty 
       $ValidationResult = Test-AzureRmResourceGroupDeployment -ResourceGroupName $TempValidationRG -Mode Complete -TemplateFile "$here\azuredeploy.json" -TemplateParameterFile "$here\azuredeploy.parameters.json" 
       $ValidationResult | Should BeNullOrEmpty 
   } 
}

That’s the last of our tests before deployment, so we’ll run it all the way through and check all our tests:

Deployment Testing

Our last test is something we are going to run after we deploy our resources. All the previous tests have been to check that our deployment template is valid and contains what we think it should, these tests are intended to check the other side of the coin, that we did actually deploy what we expected. We’re going to use Pester to validate that the resources now present in Azure are in fact what we expected.

These tests are going to validate resources in Azure against a set or Pester tests describing our expectations. In these examples we’re going to be looking to check that certain resources got created, however this testing could also be used in other ways. If your in IT security you could write similar tests that check for compliance, things like are there any external ports open in an NSG or is an AV extensions installed

The first test we are going to do is to check that the ResourceGroup who’s contents we plan to test actually exists, so all we are going to do is use regular Azure PowerShell to get the data we need then get Pester to validate it. Note I have already logged into Azure using login-azurermaccount.

$resourceGroup = "InfrastructureTesting" 
Describe "Resource Group tests" -tag "AzureInfrastructure" {
    Context "Resource Groups" { 
        It "Check Main Resource Group $resourceGroup Exists" { 
            Get-AzureRmResourceGroup -Name $resourceGroup -ErrorAction SilentlyContinue | Should Not be $null 
        } 
    } 
}

Pretty straightforward. Next we want to test our virtual network, so we will check it exists, check for a subnet and check that the subnet has the right IP range. Note that I am hard coding variables here, you could very easily pass these in as parameters, use a configuration file or even a database.

Describe "Networking Tests" -tag "AzureInfrastructure" { 
    Context "Networking" { 
        $vNet=Get-AzureRmVirtualNetwork -Name "$resourceGroup-vNet" -ResourceGroupName $resourceGroup -ErrorAction SilentlyContinue 
            
        it "Check Virtual Network $resourceGroup-vNet Exists" { 
            $vNet | Should Not be $null 
        } 

        it "Subnet $resourceGroup-subnet1 Should Exist" { 
            $subnet = Get-AzureRmVirtualNetworkSubnetConfig -Name "$resourceGroup-subnet1" -VirtualNetwork $vNet -ErrorAction SilentlyContinue 
            $subnet| Should Not be $null 
        } 

        it "Subnet $resourceGroup-subnet1 Should have Address Range 10.2.0.0/24" { 
            $subnet = Get-AzureRmVirtualNetworkSubnetConfig -Name "$resourceGroup-subnet1" -VirtualNetwork $vNet -ErrorAction SilentlyContinue 
            $subnet.AddressPrefix | Should be "10.2.0.0/24" 
        } 
    } 
}

Lastly we also want to check a VM, again we will check it exists and then I am going to check that it has been deployed with the size I expect and in the region I expect, again all using standard Azure Powershell with some Pester mixed in.

Describe "Virtual Machine Tests" -tag "AzureInfrastructure"{ 
    context "VM Tests"{ 
        $vmName="InfraTest-Vm1" $vm= Get-AzureRmVM -Name $vmName -ResourceGroupName $resourceGroup 

        it "Virtual Machine $vmName Should Exist" { 
            $vm| Should Not be $null 
        } 

        it "Virtual Machine $vmName Should Be Size Standard_DS1_v2" { 
            $vm.HardwareProfile.VmSize | should be "Standard_DS1_v2" 
        } 

        it "Virtual Machine $vmName Should Be Located in West Europe" { 
            $vm.Location | should be "westeurope" 
        } 
    } 
}

You can see the full test file here. We’ll go ahead and run that, and we should see it is all green.

Summary

Hopefully you’ve seen how powerful testing can be now that we are working with our infrastructure in code format. We’ve used some fairly basic examples here today, but they should be enough to demonstrate how this testing process should work, and hopefully give you ideas of how you can go off and build some of your own. If spending a few hours writing some tests saves you making mistakes in your deployments in the future it can become a real driver for improving the quality of of infrastructure code, spotting issues before they reach production and even preventing the addition of significant security issues if done properly.