Building Packer Images with Azure DevOps
We’ve looked previously at using the open source tool https://www.packer.io/intro/ from Hashicorp to automate the process of building Azure VM images, which can then be consumed using either single VMs or VM scale sets. By using this tool and some PowerShell DSC scripts, we have been able to automate the process of spinning up a VM in Azure, installing the required software and configuration, running Sysprep and capturing an image. So far we have done this by running our Packer script manually from the command line. If we are to move this process into production, we are going to need a way to regularly run this process in a consistent manner whenever we want to update the image (to apply updates, install new software etc.). We also need a way that other users can easily trigger this process without having to learn packer.
This is where a CI/CD solution comes in. We are going to use Azure DevOps to create a build that will allow people to create a new image with a few clicks. We’re going to use Azure DevOps because it is what I am familiar with, it is easy to get started and available for free; however, this process could also be done on a different CI/CD solution such as Jenkins.
For the rest of this article, I am going to assume you are already familiar with the process to write a Packer file and have been able to build an image manually. If you have not done this yet I recommend you read my two previous articles on how to create an image with packer:
I’m also going to assume that you have an Azure DevOps (formally VSTS) account to follow along with. If you don’t, you can get one for free here. You should also go ahead and create a project that your code and build will sit in, either a new one or add to an existing project.
All the files for this demo can be found on Github here - https://github.com/sam-cogan/Demos/tree/master/Packer
Creating the Build
Source Control
The first thing we need to do is get our Packer, DSC and any associated files into a place where Azure DevOps can access them, the best way to do this is to put them into version control. I’m going to use Github to store my files, so they are publicly accessible, but you can use private Repos or use Azure DevOps repository to store the data. You can find my demo files here.
Code Changes
Updating Packer Version
Coming up, we’re going to use an Azure DevOps task to build the image with Packer, and this task includes a part which supplies a version of the Packer executable. Unfortunately, the version of this executable can be pretty old, so we are going to supply our own Packer executable, which we can update as required. To do this, the first step is to add a copy of Packer to our Git repo alongside our code.
Later in this post we’ll add a build task to set Azure DevOps to use this exe.
Packer DSC Provisioner
We need to provide a copy of the DSC provisioner alongside our code; else the build will not be able to find it. However, there is a bug with DSC provisioner which can cause us some issues. The provisioner has been hardcoded to expect a “tmp” folder to exist in the root of the drive the provisioner is run on. If you’re running a custom build agent, then you can go ahead and create that directory and use the current release of the provisioner available here (you want the Windows x64 version). If you’re using a hosted build agent, however (which is going to be easier), then you can’t create this folder. Instead, I have updated the provisioner to remove this hard coding and allow it to run, you will need to download my version of the provisioner here, or if you would prefer to build it yourself, you can download the source here. I have submitted a pull request to get this back into the main project.
Wherever you get it from, download the packer-provisioner-dsc.exe and place it in the root of your project, alongside the packer.exe file. Your folder should now look like this:
Commit this back to your repository.
Create a Build
We need to create a build where we will run our scripts. To create this, log in to Azure DevOps, go to the Pipelines section and click on builds. Click on “New” and “New Build Pipeline”. The first thing we need to do is tell it what version control the code for this build will come from. Select the location of your code we added in the previous step; I’m using Github:
On the next page, you will be asked to select the type of build to create. The top option is to create a Pipeline defined in YAML; arguably this is the best approach to use because it allows you to store your build configuration as a file alongside your other code in your repository. However, writing the build as YAML does take some knowledge of Azure DevOps and the various build tasks, which can be challenging for those new to Azure DevOps. You can instead use the visual build tool, and this is what we will use here. I will also provide a YAML definition of the build when we complete it. We’re not going to use any of the pre-built templates, so click the option at the top to “Start with an empty job”.
This should then take us to a new visual build, the first thing to do is to change the Name of the build to something useful. We also to make sure that the build agent select is appropriate for the scripts we are going to run. For windows, use either the “hosted VS2017” agent, or if you have custom windows agents, you can use them.
Variables
When we ran Packer manually, we were storing our variables inside the Packer script. Going forward this is not a good idea for two reasons:
- We don’t want to store secrets (like service principal passwords) in the Packer script, as these will be stored in Git in plain text
- We want to make it easy for others to use and amend this build as required
Given this, we want to move the variables to be stored in Azure DevOps securely. Inside your build, you will see a variables tab. The first thing to do is to copy across any variables you think should either be secure, or you may want to change, into Azure DevOps. Be sure to set the sensitive ones to be secret in Azure DevOps.
In my setup, I have chosen to put the service principal details in here, to keep them secure, and then also add the name and resource group to use for the created image, and URL for our web app package download so that these can be easily changed.
Now that we have set up these variables in Azure DevOps, we need to set up the Packer Script to be able to accept them. We are going to pass these values in as variables, so there are two steps we need to do to our packer.json file:
We need to add a variables section to the top of the script that defines what variables the script will accept. We will set their value to an empty string for now. The last variable “workingDirectory” we set to use the system.defaultWorkingDirectory variable that Azure DevOps creates.
{ "variables": { "client_id": "", "client_secret": "", "tenant_id": "", "subscription_id": "", "managed_image_prefix": "", "managed_image_resource_group_name": "", "PackageLocation": "", "WorkingDirectory": "{{env `System_DefaultWorkingDirectory`}}" }, "builders": [ {...We need to then use the value of these variables in the rest of the packer.json file. User variables are referenced in the format of
{{user `variableName`}}"builders": [ { "type": "azure-arm", "client_id": "{{user `client_id`}}", "client_secret": "{{user `client_secret`}}", "tenant_id": "{{user `tenant_id`}}", "subscription_id": "{{user `subscription_id`}}", "managed_image_name": "{{user `managed_image_prefix`}}-{{isotime \"200601020304\"}}", "managed_image_resource_group_name": "{{user `managed_image_resource_group_name`}}", "os_type": "Windows", "image_publisher": "MicrosoftWindowsServer", "image_offer": "WindowsServer", "image_sku": "2016-Datacenter", "communicator": "winrm", "winrm_use_ssl": "true", "winrm_insecure": "true", "winrm_timeout": "3m", "winrm_username": "packer", "location": "West Europe", "vm_size": "Standard_DS2" } ], "provisioners": [ { "type": "powershell", "inline": [ "WINRM QuickConfig -q", "Install-PackageProvider Nuget -ForceBootstrap -Force", "Set-PSRepository -Name PSGallery -InstallationPolicy Trusted" ] }, { "type": "dsc", "manifest_file": "DSCConfiguration.ps1", "configuration_name": "webConfiguration", "install_package_management": true, "working_dir": "{{user `WorkingDirectory`}}", "install_modules": { "xPSDesiredStateConfiguration": "6.4.0.0" }, "configuration_params": { "-PackageLocation": "{{user `PackageLocation`}}" } }, { "type": "powershell", "inline": [ "if( Test-Path $Env:SystemRoot\\windows\\system32\\Sysprep\\unattend.xml ){ rm $Env:SystemRoot\\windows\\system32\\Sysprep\\unattend.xml -Force}", "& $Env:SystemRoot\\System32\\Sysprep\\Sysprep.exe /oobe /generalize /shutdown /quiet" ] } ]
We’ll commit these changes to our script back to Git, so we are ready to accept these variables.
Build Timeout
The build process to create the image is going to take a while, an hour or two. By default Azure DevOps will time-out the build, so we need to make a couple of changes to avoid this.
Adjust Timeout
We need to amend the timeouts in Azure DevOps to match our process. To do this, edit the build we created earlier and go to the options tab. On the right, you should see a “Build Job” section, with a “Build Job Timeout in Minutes” value, which will default to 60. Change this to 180.
Asynchronous Delete
By default, once your Packer task finishes it will delete the resource group it created, and it will wait until it completes deletion before ending. Given that this deletion can take a long time, and that packer can’t do anything if it fails, we can switch to deleting asynchronously, which will mean we don’t have to wait this time. To do this, we will add this line to the builder section in our Packer json file:
"async_resourcegroup_delete":true
So our builder section now looks like this:
"builders": [
{
"type": "azure-arm",
"client_id": "{{user `client_id`}}",
"client_secret": "{{user `client_secret`}}",
"tenant_id": "{{user `tenant_id`}}",
"subscription_id": "{{user `subscription_id`}}",
"managed_image_name": "{{user `managed_image_prefix`}}-{{isotime \"200601020304\"}}",
"managed_image_resource_group_name": "{{user `managed_image_resource_group_name`}}",
"os_type": "Windows",
"image_publisher": "MicrosoftWindowsServer",
"image_offer": "WindowsServer",
"image_sku": "2016-Datacenter",
"communicator": "winrm",
"winrm_use_ssl": "true",
"winrm_insecure": "true",
"winrm_timeout": "3m",
"winrm_username": "packer",
"location": "West Europe",
"vm_size": "Standard_DS2",
"async_resourcegroup_delete":true
}
],
Save this change and commit it back to the repository.
Build Steps
We are finally able to create our build steps. Open up the build we created in Azure DevOps and go to the Tasks tab.
Step 1 - Packer Env Variable
The first step is to set an environment variable so that Azure DevOps will use the version if Packer we provide. Click the plus button at the side of the agent phase and click on the PowerShell task.
Provide a sensible name for the task, then select the inline option, and enter the following code in the script box. Change the path of $tools to match the folder where packer sits in your repo.
$tools="$env:build_sourcesdirectory\Packer"
Write-Host $tools
Write-Host "##vso[task.setvariable variable=PATH;]${env:PATH};${tools}";
Your task should look like this:
Step 2 - Copy Files
For some reason, when Packer looks for the DSC files we want to run on the VM to upload them, it looks in the root of the working directory, not in the project root. I’m sure it’s possible to change this, but the easy solution is to add a task to copy these files to this location.
Click the plus button again and search for the “Copy Files” task. Add this and again give it a sensible name. For the source folder, select the root folder that your packer files sit in. In the contents section, add any DSC files you want to be copied to the server. Finally in the destination set it to “$(System.DefaultWorkingDirectory)”. Your task should look something like this:
Step 3 - Build the Image
It’s finally time to build the image. Again click the plus button and search for the “Build Machine Image” task, add this.
The first thing to do is change the version drop down to version 1. This is the only version that understands managed disks, so we need to use this.
Next, give the task a name, then we need to change the “Packer Template” option to “User Provided” to allow us to use or json file. In the template, location section, select the packer json file we created.
In the Template Parameters section, we need to list the the variables we want to pass into the template. If you have used the same variable names as me then you should be able to paste in this section:
{"client_id":"$(client_id)","client_secret":"$(client_secret)","tenant_id":"$(tenant_id)","subscription_id":"$(subscription_id)","managed_image_prefix":"$(managed_image_prefix)","managed_image_resource_group_name":"$(managed_image_resource_group_name)","PackageLocation":"$(PackageLocation)"}
Finally, in the output section, enter a name for the variable you want the path to the managed disk to be output into.
Your task should look like this:
Click Save, and we are done.
Yaml Definition
Skip this if you are not interested in YAML build definitions.
If you prefer using a YAML definition for the build rather than the graphical build, it looks like this:
resources:
- repo: self
queue:
name: Hosted VS2017
#Your build pipeline references a secret variable named ‘client_secret’. Create or edit the build pipeline for this YAML file, define the variable on the Variables tab, and then select the option to make it secret. See https://go.microsoft.com/fwlink/?linkid=865972
#Your build pipeline references an undefined variable named ‘managed_image_prefix’. Create or edit the build pipeline for this YAML file, define the variable on the Variables tab. See https://go.microsoft.com/fwlink/?linkid=865972
variables:
client_id: 'clientID'
tenant_id: 'tenantID'
subscription_id: 'subID'
managed_image_resource_group_name: 'app1ResourceGroup'
PackageLocation: 'https://storageaccount.blob.core.windows.net/webapp/SampleApps.zip'
steps:
- powershell: |
$tools="$env:build_sourcesdirectory\Packer"
Write-Host $tools
Write-Host "##vso[task.setvariable variable=PATH;]${env:PATH};${tools}";
displayName: 'Set Packer Env Variable'
- task: CopyFiles@2
displayName: 'Copy Files to: $(System.DefaultWorkingDirectory)'
inputs:
SourceFolder: Packer
Contents: DSCConfiguration.ps1
TargetFolder: '$(System.DefaultWorkingDirectory)'
- task: PackerBuild@1
displayName: 'Build immutable image'
inputs:
templateType: custom
customTemplateLocation: Packer/packer.json
customTemplateParameters: '{"client_id":"$(client_id)","client_secret":"$(client_secret)","tenant_id":"$(tenant_id)","subscription_id":"$(subscription_id)","managed_image_prefix":"$(managed_image_prefix)","managed_image_resource_group_name":"$(managed_image_resource_group_name)","PackageLocation":"$(PackageLocation)"}'
imageUri: '$imageURL'
Building the Image
Now that we have the build created, we can trigger a build to create an image. On the build page, you can go ahead and click the “Queue” button. This will open a window asking you to select a branch and a build agent to use. You should be able to accept the defaults if you configured things correctly earlier.
Click Queue and the build will be started. You are provided with a link that you can click on and watch the build progress if you wish. After 1-2 hours depending on how complicated your DSC is, you should see the build complete.
If we take a look at our destination resource group, we will see our managed image has been produced.
We can use this image to create new VMs or new VM scale sets as usual through the portal, CLI or ARM template. You can also use the output variable in the build to pass the reference to the image to other steps in build or release to automate the whole process of creating VMs.
Summary
I appreciate this is a reasonably complex article with lots of steps, but once you’ve done this once it should make sense. Please do comment with any questions or issues, and I’ll be happy to help.
You can find a full copy of all the files I used in this demo here - https://github.com/sam-cogan/Demos/tree/master/Packer.
