Using Host Process Containers to Interact with Kubernetes Windows Nodes

2022-09-10

Sometimes you need your Kubernetes workloads to interact with the underlying host OS, this can be for many reasons but a few common scenarios include:

Monitoring agents that need to read metrics from the host
Tools that need to access the underlying container runtime
Access to the host network or storage
Amending the configuration on the host
Installing additional software or agents on the host

For Linux hosts, this is fairly straightforward using a privileged daemonset on these nodes which can then access these host resources, but privileged containers aren’t an option for Windows nodes.

A recent addition to Windows containers can help us solve this problem - host process containers. Host process containers run directly on the host essentially running like an executable on the host, but coming from inside a container. You can package up whatever scripts, files, or processes you want to run on the host inside a container and then have this executed on the host. Because this is executing in the context of the host you have access to the host network, volumes, and any processes that are running in the host OS. Let’s look a bit deeper into how this works.

A Note on Security

This is probably obvious, but host process containers are running directly on the host as the user you specify. This means that they have significant rights on the host and so have access to a lot more privileged resources than your normal container workloads. Make sure that you only use host process containers where they are needed, and that you control access to these very privileged containers.

Pre-requisites

To use Host Process containers you need to meet a few pre-requisites:

Kubernetes 1.23 or higher
containerd 1.6 or higher

Creating the Container Image

You can make any windows container a host process container, so you could use any container image. However, there isn’t much point in using a large image. As mentioned, your host process container workloads execute on the host, not within the container image, so none of the container’s OS is actually used. The only point of the container is to store any of the scripts, files, etc. that you wish to copy to the host.

Given all this, you are best using the smallest image possible to decrease load times. Microsoft has facilitated this by producing a host process container base image that is only a few KB in size. You can use this as your base image and copy any files you need to it.

The docker file below uses this image to create the container, copies a PowerShell script into the image, and then runs it. This PowerShell script will run on the host.

FROM `mcr.microsoft.com/oss/kubernetes/windows-host-process-containers-base-image:v0.1.0`

ADD hello-world.ps1 .

ENV PATH="C:\Windows\system32;C:\Windows;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;"
ENTRYPOINT ["powershell.exe", "./hello-world.ps1"]

This script is running a PowerShell file, but because it is using a host process container, it uses PowerShell on the host. This means that PowerShell needs to already be installed on the host, or I need to install it from the image. Using a PowerShell base image won’t help, if PowerShell is not already installed on the node.

Here’s a simple PowerShell script that we can add to the hello-world.ps1 file to test that it is in fact running on the host.

$AdminRights = ([Security.Principal.WindowsPrincipal][Security.Principal.WindowsIdentity]::GetCurrent()).IsInRole([Security.Principal.WindowsBuiltInRole]"Administrator")
Write-Host "Process has admin rights: $AdminRights"
while ($true) { Start-Sleep -Seconds 2147483 }

Running the Host Process Container

Now that we have a container built for this workload we need to deploy it onto the Kubernetes cluster. Generally, where you want to run something on a node you will use a daemonset to ensure it runs on every node. You can do this with a deployment if you want, but you can’t easily control which node it goes on to.

Creating the daemonset is similar to a normal daemonset, but you need to tell it to use host process mode and provide privileged access where required.

    spec:
      ...
      containers:
          ...
          securityContext:
            privileged: true
            windowsOptions:
              hostProcess: true
              ...
      hostNetwork: true
      ...

In the YAML above we set the security context to allow the container to run in privileged mode, and then enable host process mode. If you just set it to privileged alone then it would not work. We also grant access to the host network.

The full YAML for our daemonset to run the docker file we created above is below:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: privileged-daemonset
  namespace: kube-system
  labels:
    app: privileged-daemonset
spec:
  selector:
    matchLabels:
      app: privileged-daemonset
  template:
    metadata:
      labels:
        app: privileged-daemonset
    spec:
      nodeSelector:
        kubernetes.io/os: windows
      containers:
        - name: powershell
          image: mcr.microsoft.com/powershell:lts-nanoserver-1809
          securityContext:
            privileged: true
            windowsOptions:
              hostProcess: true
              runAsUserName: "NT AUTHORITY\\SYSTEM"
      hostNetwork: true
      terminationGracePeriodSeconds: 0

User accounts

You can see in the YAML above that we are setting this container to run as the SYSTEM user. Again, this is using that user account on the host, not the container. With host process containers you have 3 options for users:

Each of these have different permissions, so choose the one best suited to your workload.

Limitations

There are some limitations with host process containers that you should be aware of:

Host process containers are currently in an alpha state in Kubernetes
You cannot mix host process and normal containers in the same pod
There is no isolation provided for host process containers, or between multiple host process containers on the same node
File System and Hyper V isolation are not supported