Blog

Custom Azure RBAC Roles

Role Based Access Control is Azure’s method for setting permissions on resources to control who can manage and administer these resources. Each type of Azure resource has a number of permissions that can be set on it, and these permissions can be grouped into roles that can be applied to users or groups of users to grant rights to manage resources. Out of the box Azure comes with a large...

Building an Infrastructure Pipeline Part 1 Version Control

  Defining Infrastructure as Code is becoming prevalent in all areas of IT, but none more so than in the cloud. Be this Azure Resource Manager Templates, AWS Cloudformation or third party tools like Terraform. Once you’ve gone down this route, you open up the ability to treat your infrastructure code like any other code, and because it’s just code you can now leverage some of the tools...

Azure Free Accounts

Microsoft recently announced the introduction of Azure Free Accounts, these are credit and resources that are free for a certain time period. This is aimed at competing with AWS free tier and similar offerings. Before we take a look at what is on offer, there are a few caveats to check to see if you are actually eligible: To claim the free offer you need to be a new Azure customer, so that means...

Modularisation and Re-Use with Nested ARM Templates

Most example ARM templates use a single JSON file to contain all of the deployment details in a single file. This works fine for smaller deployments, but once you start doing larger deployments, working in teams, or wanting to re-use parts of your deployment templates then you really need to start looking at nested templates. Nested templates describes the process of calling an ARM template from...

Cross Subscription Deployments

Back in June Microsoft introduced the ability to have an ARM template deploy to more than one resource group, at Ignite this year they went a step further to allow cross subscription deployments. It is now possible to have a single deployment create resources in multiple resource groups, regions and subscriptions, which is really useful if you are trying to deploy environments that include...

Ignite Azure Update – Day 1

As expected, day 1 of Microsoft’s Ignite conference saw a large number of announcements across all products, including Azure. Below you’ll find my summary of Azure announcements I pulled in from the conference floor. There are a lot of quality of life announcements that didn’t necessarily make it into the Keynotes, but that could be a big deal if they impact you. Also a quick reminder, if your at...

Virtual Network Service Endpoints

It didn’t make it into todays Ignite Keynote, but today Microsoft released a preview of a new Azure service, Service Endpoints. It is now possible to take  Azure services that have previously only had public endpoints, and restrict these to only allow access from a specific virtual network (or multiple networks), or even specific subnets. This preview is starting out with Azure SQL and Storage...

Building Azure Scale Set Images with Packer and DSC

Virtual Machines in a Scale Set provide a great way to spin up many identical machines in parralell. Thes VMs are all based on a common VM image and so unless all you need on your VMs is Windows, you are going to need to load applicaitons and data into these VMs. Getting your applications into that image can be done one of two ways:

Use a Gallery image for your scale set and apply your applications on each VM when it is deployed using something like the PowerShell DSC extension, or Chef/Puppet etc.
Create a custom image that already includes your application and build your scale set from this

Option 1. is somewhat simpler, in that it doesn’t involve managing images. Future changes, upgrades etc. only require you to amend your deployment scripts, not bake a new image. However, this comes with a downside, you are taking a hit on deployment time, as each VM needs to install the application when it is created. If your scale set is fairly static that’s not a big deal, but if you are changing the size regularly this delay can be a problem. This was the case with a scale set I was using, which was being scaled on demand, with a fairly complex application and VMs needed to be available quickly.
With run time deployment not being feasible, we need to create a custom image. I wanted to create this image in an automated way and leverage the PowerShell DSC files we had already created to install the applications into our image. This is where Packer comes in.
 
Packer
Packer is a tool for automating the building of VM images, this includes building the machine to be imaged, apply configurations and applications, creating the required image and cleaning up. Packer is not Azure specific, but it does have capabilities to be able to use Azure virtual machines as the base for the image and to create images suitable for Azure. Packer also allows us to utilize DSC as the method of configuring our VM before creating the image. In this way, if we need to update our image all we need to do is update the DSC files (and any resources they use) and then run our image build script with Packer.
Installing Packer
The installer for Packer can be found here or can be installed via tools like Chocolaty. It’s a fairly simple install that just requires extracting the files and placing them where you want, then adding that location to the Windows Path. Full instructions can be found here.
I’m not going to go deep into the workings of Packer here, if you want to investigate that take a look at the Packer Getting Started, but Packer configurations are created using a JSON file which makes use of 3 types of resources:

Builders – These are the components that deal with the creation of the machine from which to take the image. There are builders for Azure, AWS,  VMWare, VirtualBox etc.
Provisioners – These are used to configure the machine prior to creating the image, this includes installing applications, Windows Features etc. There are various provisioners using different technologies like DSC, Chef, Puppet or just plain old PowerShell or CMD.
Post-Processors – These run after the Builder and Provisioner to conduct various operations. These are optional and we won’t make use of them here.

The default installation of Packer includes the Azure builder, so nothing to install there, howeve, we do need to install the DSC provisioner. This is a community project that can be downloaded here. Once downloaded this needs to be extracted to a location where Packer will find it, this can either be alongside your Packer JSON file, or in the %APPDATA%/packer.d/plugins folder.
There is a bug in the current release (at the time of writing) of the DSC  Provisioner, where it expects there to be a tmp folder in root of the drive where your packer file is, so make sure you create this.
DSC File
We are going to use PowerShell DSC as the method of configuring our VM once it is deployed, before the image is created, so we need a DSC configuration file to use. There is nothing Packer specific about this file, it’s plain old DSC. So if you already have DSC you are using to configure your VM’s then you can re-use this here and you can continue to use DSC for configuration of both VMs and images. All you need to do is place this somewhere accessible from your packer file. If you want to learn more about DSC you can start here.
You can of course use any of the other provisioning methods as an alternative to DSC.
Azure Resources
Packer needs to be able to authenticate to Azure and have the rights to create VMs and Images in Azure. It does this using a service principal, effectively a service account in Azure. This Service Principal needs the rights to create resource groups, VMs and Images, the easiest way to achieve this is to create a SP and grant it contributor rights on your subscription. Here is some PowerShell that will create this:

PowerShell

$sp = New-AzureRmADServicePrincipal -DisplayName "svcPackerBuilder" -Password "password"

Sleep 20

New-AzureRmRoleAssignment -RoleDefinitionName Contributor -ServicePrincipalName $sp.ApplicationId

write-output "Application ID: $($sp.ApplicationId)
write-output "ObjectID: $($sp.ObjectID)

12345678

$sp = New-AzureRmADServicePrincipal -DisplayName "svcPackerBuilder" -Password "password" Sleep 20 New-AzureRmRoleAssignment -RoleDefinitionName Contributor -ServicePrincipalName $sp.ApplicationId write-output "Application ID: $($sp.ApplicationId)write-output "ObjectID: $($sp.ObjectID)

Make a note of the App ID, Object ID and Password as you will need this later.
Packer also needs a storage account to place your images in, so go ahead and create a resource group, storage account and container to keep these.
Creating Packer File
The Packer file is what defines your Builders, Provisioners and Post-Processors. It’s a JSON file, so we will go ahead and create this file. The basic syntax of this file looks like this

JavaScript

{
"builders": [
{

}
],

"provisioners": [
{

}
]
}

12345678910111213

{  "builders": [    {          }  ],   "provisioners": [    {          }  ]}

Builder
The first thing we need is to add a builder that will create the Azure VM. This builder has a number of parameters which are documented here. These parameters tell the build all the information it needs to know to be able to connect to your Azure subscription. Below is a sample Builder

JavaScript

"builders": [
{
"type": "azure-arm",
"client_id": "<this is the ApplicationID from your service principle>",
"client_secret": "<this is the password from your service principle>",
"object_id": "<object ID for the service principle>",
"tenant_id": "<ID of the Azure AD tenant your SP is in>",
"subscription_id": "<subscription ID you want the VM and images created in>",
"resource_group_name": "<name of the resource group you created to store images>",
"storage_account": "<name of the storage account you created to store images>",
"capture_container_name": "<name of the storage container, in your storage account, you created to store images>",
"capture_name_prefix": "<prefix to use for your images, up to you>",
"os_type": "Windows",
"image_publisher": "MicrosoftWindowsServer",
"image_offer": "WindowsServer",
"image_sku": "2016-Datacenter",
"communicator": "winrm",
"winrm_use_ssl": "true",
"winrm_insecure": "true",
"winrm_timeout": "3m",
"winrm_username": "packer",
"location": "West Europe",
"vm_size": "Standard_DS2"
}
],

12345678910111213141516171819202122232425

"builders": [    {        "type": "azure-arm",        "client_id": "<this is the ApplicationID from your service principle>",        "client_secret": "<this is the password from your service principle>",        "object_id": "<object ID for the service principle>",        "tenant_id": "<ID of the Azure AD tenant your SP is in>",        "subscription_id": "<subscription ID you want the VM and images created in>",        "resource_group_name": "<name of the resource group you created to store images>",        "storage_account": "<name of the storage account  you created to store images>",        "capture_container_name": "<name of the storage container, in your storage account,  you created to store images>",        "capture_name_prefix": "<prefix to use for your images, up to you>",        "os_type": "Windows",        "image_publisher": "MicrosoftWindowsServer",        "image_offer": "WindowsServer",        "image_sku": "2016-Datacenter",        "communicator": "winrm",        "winrm_use_ssl": "true",        "winrm_insecure": "true",        "winrm_timeout": "3m",        "winrm_username": "packer",        "location": "West Europe",        "vm_size": "Standard_DS2"    }],

Replace the appropriate sections with your information, and change the OS as required. Once you have all the information added you have a builder setup to create an Azure VM. You could even run this now and it would create a VM with the gallery image, then taken an image of this, but it would be a bit pointless.
Provisioners
So now we have a Builder in place to create the VM, we need to configure it ready for imaging. For this we are going to use 3 provisioners

A PowerShell provisioner which will enable WINRM, this is not on by default in the gallery image but is required for DSC to run
The DSC provisioner to run our DSC file and configure the VM
A second PowerShell provisioner to clean the VM and run SysPrep so it is ready for imaging (this is a requirement for creating Azure Windows Images).

Our provisioner section is going to look like this:

JavaScript

"provisioners": [
{
"type": "powershell",
"inline": [
"WINRM QuickConfig -q"
]
},
{
"type": "dsc",
"manifest_file": "DSCConfiguration.ps1",
"configuration_name": "WebServer",
"install_package_management": true,
"install_modules": {
"xPSDesiredStateConfiguration": "6.4.0.0"
},
"configuration_params": {
"-PackageLocation": "",
"-AppliationName":"App1"

}
},
{
"type": "powershell",
"inline": [
"if( Test-Path $Env:SystemRoot\\windows\\system32\\Sysprep\\unattend.xml ){ rm $Env:SystemRoot\\windows\\system32\\Sysprep\\unattend.xml -Force}",
"& $Env:SystemRoot\\System32\\Sysprep\\Sysprep.exe /oobe /generalize /shutdown /quiet"
]
}
]

1234567891011121314151617181920212223242526272829

"provisioners": [    {      "type": "powershell",      "inline": [        "WINRM QuickConfig -q"      ]    },    {      "type": "dsc",      "manifest_file": "DSCConfiguration.ps1",      "configuration_name": "WebServer",      "install_package_management": true,      "install_modules": {        "xPSDesiredStateConfiguration": "6.4.0.0"      },      "configuration_params": {        "-PackageLocation": "",        "-AppliationName":"App1"       }    },    {      "type": "powershell",      "inline": [        "if( Test-Path $Env:SystemRoot\\windows\\system32\\Sysprep\\unattend.xml ){ rm $Env:SystemRoot\\windows\\system32\\Sysprep\\unattend.xml -Force}",        "& $Env:SystemRoot\\System32\\Sysprep\\Sysprep.exe /oobe /generalize /shutdown /quiet"      ]    }  ]

The two PowerShell provisioners are pretty straight foward, they just contain an “inline” section which contains the lines of PowerShell I want to run.
The DSC provisioner takes a few more parameters:

The Manifest_File – the path to my DSC file, in this example it is sat in the same folder as the packer file
Configuration_name – the name of the configuration inside my DSC file
Install_package_mangement  – installs OneGet on the server making it much easier to download any DSC modules you need for your configuration
Install_Modules – an array of DSC modules I need in my DSC file, these will get installed for me
configuration_params – these are any parameters you need to send to your DSC file

There are some other parameters that may be useful with the DSC provisioner, you can find them listed here .
Full Packer File
Now we have created a builder and provisioners, we have a working Packer file. We could add in some post-processors, but we don’t need them here. This is what the final file looks like.

JavaScript

{
"builders": [
{
"type": "azure-arm",
"client_id": "<this is the ApplicationID from your service principle>",
"client_secret": "<this is the password from your service principle>",
"object_id": "<object ID for the service principle>",
"tenant_id": "<ID of the Azure AD tenant your SP is in>",
"subscription_id": "<subscription ID you want the VM and images created in>",
"resource_group_name": "<name of the resource group you created to store images>",
"storage_account": "<name of the storage account you created to store images>",
"capture_container_name": "<name of the storage container, in your storage account, you created to store images>",
"capture_name_prefix": "<prefix to use for your images, up to you>",
"os_type": "Windows",
"image_publisher": "MicrosoftWindowsServer",
"image_offer": "WindowsServer",
"image_sku": "2016-Datacenter",
"communicator": "winrm",
"winrm_use_ssl": "true",
"winrm_insecure": "true",
"winrm_timeout": "3m",
"winrm_username": "packer",
"location": "West Europe",
"vm_size": "Standard_DS2"
}
],
"provisioners": [
{
"type": "powershell",
"inline": [
"WINRM QuickConfig -q"
]
},
{
"type": "dsc",
"manifest_file": "DSCConfiguration.ps1",
"configuration_name": "WebServer",
"install_package_management": true,
"install_modules": {
"xPSDesiredStateConfiguration": "6.4.0.0"
},
"configuration_params": {
"-PackageLocation": "",
"-AppliationName": "App1"
}
},
{
"type": "powershell",
"inline": [
"if( Test-Path $Env:SystemRoot\\windows\\system32\\Sysprep\\unattend.xml ){ rm $Env:SystemRoot\\windows\\system32\\Sysprep\\unattend.xml -Force}",
"& $Env:SystemRoot\\System32\\Sysprep\\Sysprep.exe /oobe /generalize /shutdown /quiet"
]
}
]
}

12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455

{    "builders": [        {            "type": "azure-arm",            "client_id": "<this is the ApplicationID from your service principle>",            "client_secret": "<this is the password from your service principle>",            "object_id": "<object ID for the service principle>",            "tenant_id": "<ID of the Azure AD tenant your SP is in>",            "subscription_id": "<subscription ID you want the VM and images created in>",            "resource_group_name": "<name of the resource group you created to store images>",            "storage_account": "<name of the storage account  you created to store images>",            "capture_container_name": "<name of the storage container, in your storage account,  you created to store images>",            "capture_name_prefix": "<prefix to use for your images, up to you>",            "os_type": "Windows",            "image_publisher": "MicrosoftWindowsServer",            "image_offer": "WindowsServer",            "image_sku": "2016-Datacenter",            "communicator": "winrm",            "winrm_use_ssl": "true",            "winrm_insecure": "true",            "winrm_timeout": "3m",            "winrm_username": "packer",            "location": "West Europe",            "vm_size": "Standard_DS2"        }    ],    "provisioners": [        {            "type": "powershell",            "inline": [                "WINRM QuickConfig -q"            ]        },        {            "type": "dsc",            "manifest_file": "DSCConfiguration.ps1",            "configuration_name": "WebServer",            "install_package_management": true,            "install_modules": {                "xPSDesiredStateConfiguration": "6.4.0.0"            },            "configuration_params": {                "-PackageLocation": "",                "-AppliationName": "App1"            }        },        {            "type": "powershell",            "inline": [                "if( Test-Path $Env:SystemRoot\\windows\\system32\\Sysprep\\unattend.xml ){ rm $Env:SystemRoot\\windows\\system32\\Sysprep\\unattend.xml -Force}",                "& $Env:SystemRoot\\System32\\Sysprep\\Sysprep.exe /oobe /generalize /shutdown /quiet"            ]        }    ]}

Running Packer
Now we have our Packer file we are ready to run this and create an image. Running this is a simple command from PowerShell or the command line.

PowerShell

Packer Build buildFile.json

1

Packer Build buildFile.json

This will start the build process and you should see logs showing you the process of your build. The build process will create a resource group, and the required resources inside this (VM, storage, Keyvault) once it is done it will clean all of these up.
Once it completes successfully you will see output that give you the URLs for you completed images, and you should see these in your designated storage account. You can then use images to create your scale sets, or just use as a standard VM.
If you need to update your images all you need to do is update your DSC file as required, then re-run your build. Packer will create a new image, with a different name.
Debugging
Creating images doesn’t always work as you expect and debugging can be quite hard when the VM you created gets deleted as soon as your build finishes. There are a couple of command line switches that may help.

PowerShell

Packer Build buildFile.json -on-error=ask

1

Packer Build buildFile.-on-error=ask

What this will do is pause when an error occurs, rather than immediately cleaning up the VM. You can then connect to the VM if require and debug any issues with DSC etc. You will find the IP and Username and Password for the VM in the build log.

PowerShell

Packer Build buildFile.json -debug

1

Packer Build buildFile.-debug

This will output additional debugging information, it will also pause after each section to allow you to review that information.
These commands can be used together if required.

PowerShell

Packer Build buildFile.json -on-error=ask -debug

1

Packer Build buildFile.-on-error=-debug

 
Next Steps
Now that we have a working Packer build my next step will be to hook this into my Continuous Integration system (in this case Visual Studio Team Services) so that any time a new version of my application is created an image is built. We’ll cover this in the next part of this article.
Further Reading
Introduction to Packer
Packer Azure Resource Manager Builder
Packer DSC Provisioner
 

Azure AD Connect and The Trouble With Expired Passwords

In an on premises world, with Active Directory, password expiry is easy. Set the required policy for your domain, make sure it’s applied and forget about it, AD will take care of enforcing password changes and compliance with your password rules. Moving your identity to Azure complicates things, and that’s what we are going to talk about today, and in particular password expiry and...

Dynamic ARM Templates with Inline Logic Operators

A while back I wrote an article talking about the new “Condition” option in Azure Resource Manger (ARM) templates. This was the first step into conditional logic in ARM templates and worked great where you needed to apply a  condition at the resource level. Where it fell down was where you needed a condition inside a resource, this resulted in you having to duplicate objects with...

Follow Me

Follow me on Twitter