Using Terraform Modules from Git in Azure DevOps
If your working with Terraform you are eventually going to start writing your own modules. Modules allow for packaging your Terraform code and logic into a re-usable unit of work that you can then share with others, or just re-use yourself. If you are creating modules, then you should be version controlling them.
When it comes to using these modules in your Terraform configurations, ideally you want to avoid downloading and packaging these files manually, so Terraform has a nice feature where you can configure your modules to be pulled straight from your Git repo.
In this article we are going to take a look at how this works, in particular, we are going to look at how we can use this from within Azure DevOps pipelines. Using a pipeline to deploy your Terraform code is relatively standard, but it presents some challenges when trying to access code in protected Git repositories.
Referencing Modules in Git
To be able to use our modules directly from Git, the first thing we need to do is amend our Terraform configurations that call those modules to reference Git instead. This change is as simple as changing the “source” field in the module reference to a Git URL rather than a path.
HTTPS or SSH
Accessing a Git repo can be done using HTTPS or SSH. Generally, if you are accessing an unsecured Git repo, then using HTTPS is easiest. Where SSH can come in useful is if you are trying to access a secured repository, such as those hosted in Azure DevOps or Github private repos. Using an SSH key is often easier than trying to work with credentials over HTTP, especially when you are looking to run your Terraform code on a non-windows machine, like a Linux build agent. We will look at using SSH to authenticate later in this article.
If you are using https then your Git URL is going to look something like this:
https://github.com/sam-cogan/terraform-samcogan-aks.git
For SSH it will be something like:
git@github.com:sam-cogan/terraform-samcogan-aks.git
Note the username supplied in the URL.
Whichever option you wish to use, make a note of the URL.
Update Terraform File
Now we have our Git URL we can update the reference in the Terraform file. Let’s take a module that looks like this:
module "aks-module" {
source = "c:\\terraform\\modules\\aks"
cluster_name = "AKS001"
regions = ["WESTEUROPE", "NORTHEUROPE"]
}
To switch to using Git, we will swap out the source for our Git URL:
https
module "aks-module" {
source = "git::https://github.com/sam-cogan/terraform-samcogan-aks.git"
cluster_name = "AKS001"
regions = ["WESTEUROPE", "NORTHEUROPE"]
}
ssh
module "aks-module" {
source = "git::ssh://github.com:sam-cogan/terraform-samcogan-aks.git"
cluster_name = "AKS001"
regions = ["WESTEUROPE", "NORTHEUROPE"]
}
If you are using HTTPS against an unauthenticated repository, then that is all you need to do. If you run Terraform Init now, it should download your required modules for you ready to use. If you need to access an authenticated repository using SSH, then there are some extra steps to take.
Setup SSH Keys
To be able to access an authenticated Git repo using SSH, we need to set up an SSH key that will give us access. We will look at how to do this in Azure DevOps.
Generate Key
The first thing we need to do is generate a key pair that we will then use for authentication. We can use OpenSSH to do this which is available on most systems. Open a command prompt and run the command below, replacing the email with your email
ssh-keygen -C "user@email.com"
This command will ask you where to create the files, and then to supply a passphrase. This passphrase is just a password to protect your keys. Once this is done, you should find two keys are created in your requested location:
- id_rsa - your private key
- id_rsa.pub - your public key.
Add Key To Azure DevOps
Open up the Azure DevOps portal, click on the Settings icon on the top right, then go to “SSH Public Keys”.
Click the “New Key” button. In the page that opens enter a name for your key, then in the “Public Key Data” field paste in the content of the id_rsa.pub file, we created earlier. You are now set up to use SSH keys with your account.
Note that unlike PAT tokens, you cannot scope SSH keys. Using the SSH key will give you the same rights as the user who’s account this key belongs to, so use them carefully.
Accessing Modules in Azure DevOps Pipelines
Now we have our Terraform code referencing Git, and we have SSH keys setup we can look at getting our Azure DevOps pipeline to be able to run this Terraform.
Because we are using SSH to access the protected repository, we need to configure our pipeline with the information to connect using SSH.
Upload Private Key
The first thing we need is the private key we generated earlier, which will be used to authenticate. We need to store this securely, as this is the file that grants us access. To do this, we will use the secure file storage option in Azure DevOps.
In your Azure DevOps project go to the pipeline’s section, then library. In the top menu, click the “Secure Files” option.
Click the plus button, then browse and upload the id_rsa file we created earlier.
Known Host File
The next bit of information we need is the “known host entry”. This information identifies the server we want to connect to and tells Azure DevOps to trust it so that it does not try and prompt us to ask if this is ok.
To get this, we can run the following command:
ssh-keyscan <hostname of repo>
You should replace “
Create Pipeline
Now that we have all the values we need, we can create the pipeline. The first thing we need to do is create some variables to hold the data we collected above. Because these are sensitive variables, we will create them using the variable pane in the pipeline. We need to create the following:
- known_host - this contains the text from the ssh-keyscan we collected above
- ssh_public_key - paste into this the content of the id_rsa.pub file
- ssh_passphrase - if you created a passphrase when you generated your key, enter this here
Now we have the variables setup we can create our pipeline. The first task will be to set up the SSH key, so Terraform can use it, using the “Install SSH Key” task. The Yaml looks like this:
- task: InstallSSHKey@0
inputs:
knownHostsEntry: $(known_host)
sshPublicKey: $(ssh_public_key)
sshPassphrase: $(ssh_passphrase)
sshKeySecureFile: id_rsa
If the name of the secure file you uploaded earlier is not “id_rsa”, then replace this with the actual name.
Deploy Terraform
Now that we have the SSH keys set up, and our Terraform files are referencing the Git repo using SSH, that is all we need to do. Now when we run the Terraform file, using something like the Terraform or Bash task, it will use the installed keys automatically to check out the modules directly from Git.
Versions and Branches
The URL we used in the Terraform files above will checkout the module from the master branch of your repository. Sometimes you may want to check out the module from a specific branch or tag. For example:
- Your working on developing a new version of the module on a branch and want to test it
- You have tagged the release of your module with the version number so that you can run a specific version
If we want to use a specific version or tag, all we need to do is amend the URL used in our Terraform configuration to use the “ref” attribute to specify the branch or tag name. For example, the code below will get the module from the develop branch.
module "aks-module" {
source = "git::ssh://github.com:sam-cogan/terraform-samcogan-aks.git?ref=develop"
cluster_name = "AKS001"
regions = ["WESTEUROPE", "NORTHEUROPE"]
}
Similarly, the code below will use the v1.0.0 tag:
module "aks-module" {
source = "git::ssh://github.com:sam-cogan/terraform-samcogan-aks.git?ref=v1.0.0"
cluster_name = "AKS001"
regions = ["WESTEUROPE", "NORTHEUROPE"]
}
