Adding Security Contexts to Helm Charts with Pulumi Transformations

Azure, Pulumi, Helm

Previously I’ve talked about how you can use Azure Policy for AKS to replace Pod Security Policies now they are deprecated. One thing I mentioned is that for containers to pass these policies and be allowed to run. You need to make sure that you are declaring your security context settings in your container manifest YAML, something like this:

spec:
  securityContext:
    runAsUser: 1000
    runAsGroup: 3000
    fsGroup: 2000
  volumes:
  - name: sec-ctx-vol
    emptyDir: {}
  containers:
  - name: sec-ctx-demo
    image: busybox
    command: [ "sh", "-c", "sleep 1h" ]
    volumeMounts:
    - name: sec-ctx-vol
      mountPath: /data/demo
    securityContext:
      allowPrivilegeEscalation: false

Adding a SecurityContext is pretty easy to do if you build the containers and create the YAML. However, if you’re deploying Helm charts created by someone else, it can be a pain. Many popular Helm charts have a security context section in their values file, allowing you to pass in the values you want, which works great. However, so Helm charts don’t have this, or for some, they only apply it to some containers. A prime example of this is the Prometheus Helm chart which allows you to specify a security context for the primary containers, but some init containers don’t have one and aren’t allowed to run.

So what do you do in this situation? You could download the Helm chart, make the changes directly and then run it, but now you have to maintain your copy of the Helm chart. If you’re using Pulumi for your Infrastructure as Code solution to deploy your Helm chart, then there is a better way, using transformations. If you’d like to know more about Pulumi, what it is and how it works, take a look at this article.

What Are Transformations?

The way Pulumi deploys Helm charts is a bit different to how you might think. You use the Kubernetes provider to configure a Helm chart like this:

 var nginx = new Chart("Nginx-ingress", new ChartArgs
        {
            Chart = "nginx-ingress",
            Version = "1.24.4",
            FetchOptions = new ChartFetchArgs
            {
                Repo = "https://charts.helm.sh/stable//"
            }
        });

This code configures Pulumi to fetch the chart from the repo and deploy it. However, what happens here is Pulumi downloads the chart, and then extracts it and deploys each of the YAML files directly using the Kubernetes API. It does not use Helm. If you run “Helm List”, you will not see a Helm deployment in Kubernetes, and you cannot manage it directly through Helm.

This choice tends to divide people on whether it is a good or bad thing. By doing this, Pulumi can track the changes directly on what is in the YAML, so only updates the parts of the code that change, rather than deploying the whole Helm chart every time the version changes. However, you lose all the benefits of Helm on the cluster side because it was not deployed with Helm. I guess Pulumi would argue that you should be managing everything through Pulumi.

Regardless of your view on this, it does enable some extra functionality, including transformations. Because Pulumi is extracting the Helm chart and then running it, you can make changes to the Helm chart before it runs. These are called transformations. You can programmatically tell it to find a section in the chart and amend it before deploying it. So using this, we can have Pulumi insert security context sections into a Helm chart before running it.

We benefit from being able to amend the Helm chart, but it’s part of our deployment code rather than done manually. We have to manage our deployment code rather than fork the whole Helm chart and manage that.

Creating a Security Context Transformation

Let’s create a transformation that adds the “allowPrivilageEscalation” security context setting into any containers created by our Helm chart. The code below is in C#, but you can achieve this in any Pulumi language. I’d also point out that I am not a developer, so if there are better or more optimised ways the write this code, do let me know!

The code below creates a transform method which will then call from our Helm chart. This method looks at our Helm chart to find any containers and adds the security context. Let’s break it down and see how it works.

 ImmutableDictionary < string, object > setSecurityContext(ImmutableDictionary < string, object > obj,
   CustomResourceOptions opts) {

   if ((string) obj["kind"] == "Deployment" || (string) obj["kind"] == "DaemonSet") {
     var spec = (ImmutableDictionary < string, object > ) obj["spec"];
     var template = (ImmutableDictionary < string, object > ) spec["template"];
     var templateSpec = (ImmutableDictionary < string, object > ) template["spec"];
     var containers = (ImmutableArray < object > ) templateSpec["containers"];
     object[] newContainers = new object[containers.Length];
     int i = 0;
     foreach(ImmutableDictionary < string, object > container in containers) {
       var securityContext = new Dictionary < string,
         object > () {
           {
             "allowPrivilegeEscalation",
             false
           }
         };
       var tempContainer = container.SetItem("securityContext", securityContext);
       newContainers[i++] = tempContainer;
     }

     templateSpec = templateSpec.SetItem("containers", newContainers);
     template = template.SetItem("spec", templateSpec);
     spec = spec.SetItem("template", template);
     obj = obj.SetItem("spec", spec);
   }

   return obj;
 }

First, we have the method header:

 ImmutableDictionary < string, object > setSecurityContext(ImmutableDictionary < string, object > obj,
   CustomResourceOptions opts) {

This method needs to match what Pulumi expects for transformations, so the only thing to adjust here is the name, depending on what you want to call it. The ImmutableDictionary obj is the bit we are interested in, and this contains the content of our Helm chart. Once we have that, we first need to find the type of resources we want to edit. In my case, this is Deployments or Daemonsets. I know that this Helm chart only has these types. There are no StatefulSets or Jobs. I’m adding an if statement that looks in the content of “obj”, which is all our Helm YAML data. The code then looks inside that YAML for a section with the value “Kind” set to Deployment or DaemonSet.

if ((string) obj["kind"] == "Deployment" || (string) obj["kind"] == "DaemonSet") {

I now know that I have a resource of one of these types. Next, I need to find the section we want to edit. If you look at the Yaml example at the start of the article, you will know that the container security context section inside a Deployment or Daemon set is in the spec.template.spec.containers section:

spec:
  template:
    spec:
      containers:
        securityContext:
          allowPrivilegeEscalation: false

So, we need to work our way through the obj dictionary to get to this point. I’ve broken this out to make it easier to read, but you could concatenate a lot of this to make it short if you wanted.

var spec = (ImmutableDictionary < string, object > ) obj["spec"];
var template = (ImmutableDictionary < string, object > ) spec["template"];
var templateSpec = (ImmutableDictionary < string, object > ) template["spec"];
var containers = (ImmutableArray < object > ) templateSpec["containers"];

The containers variable now contains the bits we need to edit. Bear in mind that our Deployment/Daemonset may have multiple containers, so we are going to loop through them all. We are then going add in the security context section and store the changes in a new array:

object[] newContainers = new object[containers.Length];
int i = 0;
foreach(ImmutableDictionary < string, object > container in containers) {
    var securityContext = new Dictionary < string,
    object > () {
        {
            "allowPrivilegeEscalation",
            false
        }
    };
    var tempContainer = container.SetItem("securityContext", securityContext);
    newContainers[i++] = tempContainer;
}

Now the newContainers array contains the set of container objects with our added security context. We need to replace the containers section of the obj Helm data with this.

     templateSpec = templateSpec.SetItem("containers", newContainers);
     template = template.SetItem("spec", templateSpec);
     spec = spec.SetItem("template", template);
     obj = obj.SetItem("spec", spec);

Finally, we return “obj” as the output of the method. This is all the code we need to do the transformation. Now we need to amend our Helm resource to add a transformations section calling the method we created:

 var nginx = new Chart("Nginx-ingress", new ChartArgs
        {
            Chart = "nginx-ingress",
            Version = "1.24.4",
            FetchOptions = new ChartFetchArgs
            {
                Repo = "https://charts.helm.sh/stable//"
            },
            transformations = {
                setSecurityContext
            }
        });

Transformation is a list of methods, so you could have as many different transformations as you like, or you could have one big transformation method that does everything. In our example, we add the security context section at the container level, and we still need to do another transform to add it at the Deployment/DaemonSet level.

Summary

The way Pulumi deals with Helm charts is a bit controversial; however, the ability to undertake transformations of that Helm chart at deployment time is a real benefit. If you find yourself needing to amend a Helm chart that you didn’t create, it can be an excellent solution to avoid having to fork and manage that Helm chart yourself.