WTH is Pulumi ESC?

Pulumi, Configuration, WTH

This week Pulumi announced a brand new product for their Pulumi Cloud platform - Pulumi ESC. In the latest of our WTH series let’s look at what this service is, how it works and why you might want to use it.

What is Pulumi ESC?

ESC stands for Environments, Secrets and Configuration and is Pulumi’s answer to managing configuration in your Infrastructure as Code and more. If you’ve used Pulumi before you’ll know that there is built-in functionality for creating configuration in YAML files in your project, but this is limited to a single configuration file per deployment, it doesn’t provide functionality for having any sort of hierarchical configuration or sharing of configuration between projects.

Imagine you have a set of configuration data that is used in many different projects, for example, some IP addresses that you want to use in IP allow lists. With standard Pulumi configuration, you would need to copy this data into every configuration YAML file or introduce some other service or solution (like a configuration module. Pulumi ESC looks to fill this gap by providing a service for storing configuration data and allowing you to pull in data from other services. This service has a few key properties:

  • Environments that store configuration data that can be imported into other environments, providing a hierarchical setup
  • Providers that allow you to talk to external configuration stores such as Azure Key Vault, AWS or GCP secrets, Vault
  • Exporting configuration to Pulumi configuration files or Environment variables to be consumed
  • API and SDK for accessing configuration data from any service
  • Authentication with OIDC using Pulumi cloud

ESC Diagram

How Does Pulumi ESC Work?

Pulumi ESC is a component of the Pulumi cloud service. Within ESC you create environments, which then contain your configuration data. You can create as many environments as you want.

Environments

Environments

Environments and the configuration data within them can be managed either using the Pulumi Cloud UI, or via the CLI. The web UI is a simple YAML editor, which is a little limited at the moment. The UI allows you to edit the values and then get a preview of the data.

YAML UI

Configuration values can be created following the same approach as with a Pulumi config file, with a YAML approach for creating strings, lists or objects.

values:
	stringValue: string
	listValue:
	- item1
	- item2
	objectValue:
		property1: value
		property2: value

Other environments can be imported with the imports statement. If we wanted to create a configuration for production environments in Europe then we could an an imports statement that looks like this:

imports:
	- production
	- europe

Environments are merged in order, so in this list a value with the same name in both would take the value from the “europe” environment.

We can also use the CLI to set a value:

pulumi env set samcogan/development foo bar

and then preview the environment with:

pulumi env open samcogan/development

Open ENV

You can also open the environment if your editor of choice:

pulumi env edit --editor vim samcogan/development

Vim

Functions

ESC also includes functions, which allow us to perform transformations on values dynamically. For example, if you want to provide the base64 version of a string you can use a function to do this:

  passwordB64:
      fn::toBase64: ${app.password}

This means I’m only storing one value (app.password), and if someone asks for passwordB64, they get a dynamically created value. If app.password changes, I only need to update it in one place.

Other functions include:

  • Secret
  • Join
  • toJSON
  • toString

Providers

Providers allow you to connect to external data sources so that rather than making a copy of this data you can pull it directly from the source. Providers exist for a variety of sources. Let’s look at creating one for Azure Key Vault. We need to tell ESC how to login to Key Vault, and then what secrets to grab. The example below uses a service principle client/secret to connect, but OIDC with workload identity is also supported.

azure:
  login:
    fn::open::azure-login:
      clientId: xxxxxxxx
      clientSecret: xxxxxx
      tenantId: xxxxx
      subscriptionId: xxxxx
secrets:
  fn::open::azure-secrets:
    login: ${azure.login}
      vault: pulumistate01
        get:
          api-key:
            name: api-key
          app-secret:
            name: app-secret

This will then result in two configuration values being available from ESC- api-key and api-secret

Exports

Any configuration data in ESC can be obtained via the API/SDK, which means it is not limited to just Pulumi IaC. However, the exports section has some helpers that make consuming data in Pulumi easier.

First, there is the pulumiConfig section. Any configuration value listed here will be available in Pulumi IaC just like it was in the local configuration file. We need to create the value under the pulumiConfig heading, for example:

  pulumiConfig:
    samcogan:region: westeurope

then in the Pulumi config yaml file we just add an imports section

import:
  - development
config:
	# normal pulumi config

we can then consume that in our code:

config.require("region")

We can also export configuration data to environment variables using the environmentVariables section

environmentVariables:
	AZURE_CLIENT_ID: 123456

We can then use the pulumi env run command to run any executable with these environment variables configured.

pulumi env run /samcogan/development az vm list

Why Would I Want to Use Pulumi ESC

If you’re just getting started with Pulumi and all of your configuration data is unique for every deployment and you’re happy with the existing Pulumi configuration YAML file, then this service may not benefit you. However, as your configuration data gets more complex, or you try and standardise some of this data across your company then a centralised configuration service becomes a lot more appealing.

In addition to providing a centralised configuration store, ESC also adds the ability to use hierarchical configurations, having one configuration import another and another. In environments where you have multiple sources and owners of configuration data, with different lifecycles this can be a real benefit. Top-level organisational configuration data can be managed by one team and consumed by everyone else, rather than having to copy the data into every project and invariably getting it incorrect somewhere, or forgetting to update it.

Using providers means that if you have configuration data elsewhere, you don’t need to replicate it in two places and again struggle to keep them in sync. If you keep all your secret data in Key Vault then just connect a provider to that and bring it into your ESC environment. Then provide that environment to all your consuming teams, who can use it with very little work.

Functions provide a way to transform the data and provide it to your users without duplicating your data. A function is just a dynamic transformation of an existing configuration setting rather than a whole new value.

Finally, being part of Pulumi Cloud provides authentication and authorisation using your preferred OIDC provider.

You are also not limited to using ESC with just Pulumi, it can be used as a generic configuration store that any application can consume using the SDK.

What Issues Does Pulumi ESC Have

There are a few things to be aware of when looking at ESC. First, this is v1 of a brand-new product, so there are bound to be some bugs and issues, and there are a limited amount of providers and functions. These will grow over time, especially with the extensible model for adding new providers.

ESC is built into the Pulumi cloud service, there is not an open source alternative like there is with state or encryption management. This makes some sense given the integration with Pulumi cloud RBAC and OIDC, however, it may limit its use if you don’t currently have access to the cloud service. Pulumi Cloud is free for small teams, but if you are a larger enterprise, without a budget for this you may be out of luck. The pricing model for ESC has also not yet been announced (it’s free during the preview). I would love to see the ability to run your own ESC server outside of the Pulumi cloud.

Using Pulumi Cloud also means that you will need connectivity between where you run your IaC deployments and the Pulumi Cloud (or you can use the Pulumi Deployment service), for most people this is probably not an issue, but if you operate in a locked-down environment it may be.

I’d also add that at this time I’ve not had much chance to play with the SDK for accessing this configuration data. Consuming the data as Pulumi Config values, or Environment Variables is easy, but if you want to call the service directly from your code you will want to use the SDK. I will update you with details of the SDK once I have tested this fully.

Further Reading