WTH is Pulumi?

2020-08-30

A slight deviation for the WTH articles this week, as we look at a non-Microsoft product. I’ve mentioned I’ve been working with Pulumi to define infrastructure as code recently and have been enjoying working with it, so let’s have a look at what this tool is an why it might be useful.

What is Pulumi?

Pulumi is an Infrastructure as Code framework that you can use to deploy infrastructure in the cloud. Pulumi is relatively similar to Terraform in that it allows you to deploy infrastructure into many different cloud providers. Pulumi covers the 3 big cloud providers (Azure, AWS, GCP), but also has providers for many others, including:

Kubernetes
Alibaba Cloud
Digital Ocean
Linode
vSphere

A full list of providers can be found here.

The significant feature that differentiates Pulumi from something like Terraform is that it allows you to write your Infrastructure as Code in real programming languages. With something like Terraform or ARM Templates your writing you templates using a domain-specific language (DSL) explicitly created for that tool, HCL for Terraform or the JSON based language for ARM. These DSL’s provide a limited set of language features. Pulumi does not have a DSL; instead, it allows you to use an existing program language that you are already familiar with. Currently, Pulumi supports:

Node.js - including JavaScript, TypeScript or any other Node.js compatible language
Python
.NET Core, including C#, F#, and Visual Basic
Go

Because you’re using a full programming language, you get access to all the language features to help create your infrastructure deployments. You can also use your existing tools like IDE’s, test frameworks, build pipelines and so on.

Languages

How does Azure Pulumi Work?

Pulumi comprises several components:

An SDK for your chosen language
A command-line tool for running deployments
A state storage facility - either local, self-hosted, or using the Pulumi Cloud service

Pulumi uses a state file in the same way as Terraform, to record what it has done and allow for actions like plan and destroy. If you want to stick to using the free, open-source components, you can host your own state file and manage that yourself. The Pulumi Cloud service provides hosting for the state file for you, as well as offering additional services on top, including policy for deployments, webhooks for deployment actions and a web UI for deployment management and history. Pulumi’s cloud offering has a free tier for a single user, with various paid levels for teams.

Defining infrastructure using Pulumi requires using the Pulumi SDK in your IDE of choice. The SDK is available using the standard package management solutions such as NuGet, NPM and PIP. Once installed, you can use the SDK to define your infrastructure. I’ve been working with C#, so we’ll look at some examples using that.

The example below creates a resource group and storage account:

using Pulumi;
using Pulumi.Azure.Core;
using Pulumi.Azure.Storage;

class MyStack : Stack
{
    public MyStack()
    {
        // Create an Azure Resource Group
        var resourceGroup = new ResourceGroup("resourceGroup");

        // Create an Azure Storage Account
        var storageAccount = new Account("storage", new AccountArgs
        {
            ResourceGroupName = resourceGroup.Name,
            AccountReplicationType = "LRS",
            AccountTier = "Standard"
        });

        // Export the connection string for the storage account
        this.ConnectionString = storageAccount.PrimaryConnectionString;
    }

    [Output]
    public Output<string> ConnectionString { get; set; }
}

If you’re familiar with Terraform, you will notice that the resources look quite similar. This is because under the hood Pulumi actually uses the Terraform provider to communicate with Azure, and so any resources you can create in Terraform, you can create in Pulumi. Some providers in Pulumi use the Terraform provider, others (like Kubernetes) use a custom provider created by Pulumi.

A couple of other things to call out in this example. First, you will note that the resource group we create doesn’t actually have many values set, just a name which is solely used for identifying this resource group in Pulumi, not in Azure. The rest of the values are defined using some functionality in Pulumi:

The actual name of the resource group in Azure is auto-generated by Pulumi. You don’t have to do this, you can define an explicit name if you want.
The location for the resource group is coming from a configuration file rather than the code. Pulumi has the concept of “stacks” which allow you to define a configuration for each different environment you want to deploy, switching to a different environment can be done at the command line

Once you’re ready to deploy your infrastructure, you use the Pulumi command-line tool. This is very similar to the Terraform CLI and offers commands like:

Pulumi Preview - previews the changes you are going to make to let you know what actions it will take
Pulumi Up - deploy your infrastructure
Pulumi Destroy - remove your infrastructure

You can see the full list of commands here. The Pulumi CLI has a very nicely implemented auto-updating process that lets you view the status of the deployment and delve into preview data.

Pulumi Up

Why would I want to use Pulumi?

If your looking for an infrastructure as code tool one of the critical questions you want to answer early on is if you are just deploying to a single cloud, or multiple clouds and other locations (like Kubernetes). If you are looking to deploy to various different clouds and services, then you want to look at tools like Pulumi or Terraform.

Beyond the multi-cloud functionality, Pulumi has several features that differentiate it from things like Terraform. The biggest of these is obviously the support for real programming languages. If you are looking to get your developers involved in infrastructure deployments and you want to meet them with a language they know, then this could be a big selling point. Even if you don’t need this for your developers, the ability to use proper languages, with traditional loops and conditionals may make this an appealing tool. I know I have found this to be a pleasing experience!

Another benefit, which may seem small initially, but has a big win with the security folks, for me, is state encryption. Terraform storing secrets in plain text in the state file as been a big concern for a lot of people, but doesn’t seem to be getting resolved any time soon. Pulumi has resolved this by allowing you to encrypt individual secrets in the state file. This also includes the use of an external provider for storing the encryption key, including Azure Key Vault, AWS KMS, GCP KMS and Hashicorp Vault.

I mentioned earlier that for some providers, like Azure, Pulumi uses the Terraform Azure provider under the hood. For other providers though they have created their own. One of these is the Kubernetes provider, which in my view is much better than the Terraform one, as it claims to cover 100% of the Kubernetes API. Pulumi have also recently released a slew of new Kubernetes components such as a tool to convert YAML to Pulumi code, a Kubernetes operator for deploying Pulumi code and the ability to define custom resource definitions in Pulumi. If you are working with Kubernetes, then Pulumi has a lot of really nice features.

Because Pulumi is a state-based tool, it can provide features like plan and destroy to preview changes before deploying them, and removing everything when you are done with it. Pulumi also offers a choice of where to store your state, either with their cloud service (free for 1 user, otherwise paid), or your own location using Azure Blob Storage, AWS S3 or GCP Cloud Storage.

If you are interested in using the paid service then as well as getting managed state storage, you also get access to deployment history, audit, CI/CD and Webhook integrations and policy enforcement. The open-source version using your own state backend is free.

What issues does Pulumi have?

As with anything, there are some potential downsides or hurdles when using Pulumi. The first may or may not be an issue, but is the barrier to entry for non-developers. Developers familiar with one of the supported languages will pickup Pulumi very quickly, but if you are also looking to have your IT Pro’s or Ops people, who may not have experience with these languages, pick this up, then it can be daunting. To be fair to Pulumi the basics of deploying a cloud resource are relatively straight forward and aside from a few syntax changes, not that dissimilar to ARM or Terraform. However, once you want to start doing more advanced things, knowledge of the chosen programming language is required and so might be a barrier to some. From my personal experience, as a non-developer with some limited C# understanding, I was able to quickly get resources deployed, and build some custom modules. Still, there were some times I needed to talk to my developer colleagues about the best way to do things, or how to overcome an issue. If you have non-developers working with Pulumi, then it is a good idea to make sure they have access to developers who can assist.

Pulumi is a relatively new tool, around 3 years old, so you will find that the amount of examples, blog posts, support articles etc. is lower than something like Terraform. This is not unexpected but can be a bit of a pain when looking for examples or ways to solve a specific problem. The support for different languages can also add to this pain if you can only find a typescript example, and you wanted C#. However, hopefully, a bit of conversion can be done to come up with an answer.

Another potential issue is language parity. Not every feature is available in every language Pulumi supports. The standard features most people will use are present in all languages, but when you start trying to do some more advanced things, you may find that there are differences. Node/Typescript tends to have most features, whereas features like Customer Providers are missing from the .net languages. I am sure these will be added over time, but it is worth being aware of when you pick a language.

Another area that caught me out was some of the design decisions made by the Pulumi team. These were not necessarily wrong or bad, just different. I talked about one of this on more detail in this post where the decision made was to default to not refreshing state from the source when running. This is different from the way Terraform works and caused some confusion.

Finally, because some of the providers rely on the underlying Terraform providers then can be a slight delay in these getting updated when Terraform is updated (which itself can have a delay from when the cloud provider is updated). This delay is generally pretty low, often a few days, but something to be aware of.

What is Pulumi?

How does Azure Pulumi Work?

Why would I want to use Pulumi?

What issues does Pulumi have?

Further Reading