Pulumi Development with GitHub Codespaces

Azure, Pulumi

Pulumi allows you to use real development languages to create your Infrastructure as Code. Because of this, you will usually need several prerequisites and libraries installed in your development environment, depending on which language you are using. You can set these up on your development PC, but one alternative approach is to use GitHub Codespaces. GitHub Codespaces allows you to quickly deploy a pre-configured development environment, along with your code running as a cloud service. You can find out more about Codespaces [here](GitHub Codespaces)

Codespaces can be particularly useful for getting new people started with a project, as they don’t need to spend time figuring out how to set up their development environment. Instead, they can go to the repo, click on Codespaces and get a working Dev environment quickly. Recently GitHub announced every GitHub user was getting free Codespace hours, with free users getting 120 core hours (60 hours of a two-core machine) and pro users getting 180 core hours, so it’s even easier to try this out.

You can create a Codespace in a few clicks by hitting the Codespace button on your Git repo, and you will quickly be put into a VS Code web view. However, this will use the default Codespace image (behind the scenes, Codespaces is running Docker images), which doesn’t have any Pulumi or language-specific resources installed. You can install them manually, but this will be a pain if you have to do it every time you launch a Codespace. Instead, with some work, we pre-configure our Codespace with everything we need to have a working Pulumi environment from the very start. Let’s see how to do that.

In this example, we’ll be writing our Pulumi code in Python, so we’ll be using the Python images and files, but you can substitute these for your language of choice.

VS Code

Codespace Repo

Codespaces are scoped to a GitHub repo, so each repo will need a separate Codespace configuration. Each instance of your Codespace will run with your specific repo checked out.

Docker File

GitHub Codespaces run Docker images. When you start a Codespace, you spin up a Docker container running VS Code and all the other resources you specify. You then connect with the VS Code web client, built into the container, or using the VS Code desktop client. When you launch a default Codespace you get the default Codespace image which is not customised. We want to configure Codespace to use an image we have customised with all the Pulumi resources we need installed.

This image will need both the Pulumi resources and the Codespace resources, so we have two options to create this:

  1. Take the existing Pulumi image for your language and install the Codespace components
  2. Take the existing Codespace image and install the Pulumi resources

There are not that many Pulumi resources we need to install, so it is easier to start with the Codespace image and add our Pulumi resources. To do this, we need to create a “.devcontainer” folder in the root of our repo. In this folder, we then create a file called “Dockerfile”. In this, we will specify how to build the Docker image. In this Docker file, we will use the multi-stage build functionality in Docker, which allows us to build something in one container and then take the outputs to use in another container. We will do the following:

  1. Use a standard Debian image
  2. Download and install Pulumi on the Debian image
  3. Use the VS Code devcontainer image for our language of choice (in this case, Python)
  4. Copy the Pulumi binaries from the Debian image to the devcontainer
  5. Set the environment variable to add Pulumi to the PATH

The result will be a VS Code dev container with all the components needed for Codespaces, which also has the Pulumi CLI installed. If you need any other binaries installed in this container, you can install them at this point too.

Our docker file looks like this:

ARG PYTHON_VERSION="3.9"
ARG PULUMI_VERSION=latest

# --------------------------------------------------------------------------------

FROM Debian:buster-slim AS builder
RUN apt-get update -y && \
    apt-get upgrade -y && \
    apt-get install -y \
    curl \
    build-essential \
    git


RUN if [ "$PULUMI_VERSION" = "latest" ]; then \
    curl -fsSL https://get.pulumi.com/ | bash; \
    else \
    curl -fsSL https://get.pulumi.com/ | bash -s -- --version $PULUMI_VERSION ; \
    fi

# --------------------------------------------------------------------------------

FROM mcr.microsoft.com/vscode/devcontainers/python:0-${PYTHON_VERSION}

COPY --from=builder /root/.pulumi/bin/pulumi /pulumi/bin/pulumi
COPY --from=builder /root/.pulumi/bin/*-python* /pulumi/bin/

ENV PATH "/pulumi/bin:${PATH}"

We also set some parameters on the file to allow us to override some of the versions when we build this.

You can build this container locally if you wish. You will find that you can exec into it and run Pulumi.

Devcontainer.json

The next file we need to create is the “devcontainer.json” file. This file configures our Codespace, and we will use it to define a few things:

  1. Tell GitHub to use the Dockerfile created above to run the Codespace
  2. Configure any features in VS Code
  3. Setup VS Code with any extensions we want installed
  4. Set any configuration settings in VS Code that we want to be pre-configured

We create the devcontainer.json file in the .devcontainer folder. You can find details on the format of this file here. Our file looks like this:

{
    "name": "Python",
    "build": {
        "dockerfile": "Dockerfile",
        "args": {
            "PYTHON_VERSION": "3.9",
            "PULUMI_VERSION": "latest"
  
        }
    },
    "features": {
        "azure-clip": "latest"
    },
    "settings": {
        "python.analysis.extraPaths":[],
        "python.analysis.diagnosticSeverityOverrides": {
            "reportShadowedImports": "none",
            "reportMissingImports": "none"
        },
        "python.linting.pylintArgs": [
            "--init-hook",
            "import sys; sys.path.append('venv/lib/python3.9/site-packages)"
        ]
    },
    "extensions": [
        "ms-python.python"
    ],
    "runArgs": [
        "--init"
    ],
    "remoteUser": "vscode"
}

Let’s break this down. First, we give the container a name and specify the docker file to use when running the container.

  "name": "Python",
    "build": {
        "dockerfile": "Dockerfile",
        "args": {
            "PYTHON_VERSION": "3.9",
            "PULUMI_VERSION": "latest"
  
        }
    },

The Dockerfile is a relative path, so we are pointing at the Dockerfile we created in the same folder. You recall we set a couple of parameters in that Dockerfile to allow us to vary the version of Python and Pulumi we use; here, we are populating these. This setting will get us a Codespace running our container.

Next up, we enable any features we want to be installed in our container. We are setting up the Azure CLI using the latest version in this case.

 "features": {
        "azure-clip": "latest"
    },

Next, we configure any VS Code settings we want in our environment so that VS Code is automatically configured the way you want. You could configure these manually when you run the code space, but remember, these containers are temporary, so if you stop the environment and rerun it, you will lose these changes. In this example, we are setting some pylint settings for the VS Code linter. When running Pulumi in Python in VS code, the default setup will show some errors and warnings, so we need to make these setting changes to avoid that.

    "settings": {
        "python.analysis.extraPaths":[],
        "python.analysis.diagnosticSeverityOverrides": {
            "reportShadowedImports": "none",
            "reportMissingImports": "none"
        },
        "python.linting.pylintArgs": [
            "--init-hook",
            "import sys; sys.path.append('venv/lib/python3.9/site-packages)"
        ]
    },

The next section enables any extensions in VS Code that you want to be installed. You can find the names used from their marketplace listing. I am just installing the Python extension.

   "extensions": [
        "ms-python.python"
    ],

Lastly, we tell the Codespace to run the init command when it launches the container and to run as the “vscode” user to avoid running as root.

    "runArgs": [
        "--init"
    ],
    "remoteUser": "vscode"

That’s all we need to configure our VS Code environment, but you can customise things as you see fit.

Secrets

At this point, we have a working environment. If we create a Codespace using these files now, it will work, and we’ll get a working environment with Pulumi installed. However, we will need to log in to Azure manually and wherever your Pulumi state will be stored. We would need to do this every time we spin up an environment as they are ephemeral and won’t persist the login data.

To avoid this and automate login, we can use secrets in Codespace. Secrets allow us to add protected values to our Codespace environment, which are injected into the container as environment variables using the secret’s name as the environment variable’s name. As Pulumi supports using environment variables for most configuration settings, we can create these secrets that will automatically login us into Azure and state storage. In my case, I am using the Pulumi service for state storage, but if you prefer to use your own backend you can do the same, just specify different secret names.

To create secrets, you need to do this at the account level, not the repo. In GitHub, click your profile picture in the top right, then go to settings. In the window that opens, go to “codespaces” on the left. This should open a window with “Codespace Secrets” showing. Here we can add all the secrets we want. The secret name needs to match the name of the environment variable you want to exist in the container. Below are the secret names I created for Azure and Pulumi service login.

Secrets

When you launch the Codespace, you should see these as environment variables.

Launch Codespace

Now everything is set up, we can create the Codespace. Back in the repo, click the green “Code” drop-down, and you should see a “codespaces” section. Click the ellipsis (…) button and then the “new with options” button.

New with options

In the window that opens, ensure that the “Dev Container Configuration” drop-down points to the devcontainer.json folder we created in the .devcontainer folder. You can set the rest of the options as you prefer for location and size. Remember that your free hours are core hours, so if you pick a four-core machine rather than a two-core, you will get half as much free time.

Create Codespace

Click the “Create Codesapce” button and wait for the Codespace to get created. The first time you start this, it can take a while.

Using the Codespace

By default, once you create the Codespace, it will launch the VS Code web editor with your repo open. You can edit and work with your code as required. When you are ready to run it, make sure you have the terminal open in VS Code by going to the hamburger menu on the left and then go to “terminal” and “new terminal.”

Terminal

From the terminal, you can run Pulumi Preview/Up/Destroy etc. commands as you would anywhere else. Pulumi will use the environment variables to connect to your backend services.

Terminal

That’s it. You’ve now got a working Codespace environment with all the settings configured in your repo, so anyone working on this repo can create a working development environment with all the required software installed in a matter of minutes.