Windows Containers and Azure

Azure, Containers

I have a love/hate relationship with Windows containers. On the one hand, it is great that we have an option for containerising legacy applications that are dependant on things like the full .net framework or other Windows-only features. On the other hand, Windows containers have some many limitations and issues compared to Linux containers, that working with them becomes painful. These limitations include what options you have for hosting these contains when you want to run them in Azure. In this article, we’re going to take a look at what we can, and cannot do with Windows containers in Azure.

This article has been updated to include the preview of Windows container support in AKS

The Problem with Windows Containers

To be clear, it is my view (and a lot of other peoples), that if your application is written to be cross-platform (using .net core for example) and you want to containerise it, then using Linux containers is the way to go, at least at the moment. Windows containers are only for situations where you must run the application under Windows. This is down to some key limitations in Windows containers. To be clear, I am talking about the Server Core variant of Windows containers here. The nano version does not offer you the ability to run full .net applications, so requires .net core (in which case why not just use Linux).

Size

The smallest Linux containers are 50-100MB in size and can be downloaded in seconds, compare that to the smallest Windows container which is around 3GB. This image is significantly smaller than the original Windows Container release at over 10GB, but it is still massive. If your needing to download images on demand, this adds a significant amount of time to your deployment.

OS Version

Due to limitations in how Windows implements the container run time, you require that the host machine is running the same version of Windows as in your container. I don’t just mean the same family, but the same version. So if your host is running Server 2016 v1803, then your containers also need to run Server 2016 v1803, they cannot be running v1709. This is very problematic as it means rebuilding your images any time your host OS changes. It’s even more problematic for those trying to use cloud providers, as often you won’t know what OS is running on the machines you are hosted on.

You can combat this issue by running your containers as HyperV containers, which wraps your container in a thin virtual machine to abstract it from the OS, but this adds complexity.

Hosting Windows Containers in Azure

If you’ve decided to go down the route of running Windows containers, and dealing with their issues, then you need somewhere to run them. If your cloud platform of choice is Azure, it can be confusing as to what supports Windows containers and what does not, so let’s clear that up.

IaaS Virtual Machine

You can go ahead and deploy a Windows VM and install Docker on it. This will allow you to host Windows containers. You will also have the benefit of being able to run Server 2019 which has many container focussed improvements, which hasn’t made it into all of the PaaS platforms yet.

The downside to this approach is that your the one having to manage the VM, patch it, maintain it, and set up any high availability, orchestration etc. yourself.

AKS Engine

Windows containers are not supported in Azure Kubernetes Service, the PaaS Kubernetes offering in Azure. However, you can deploy a Kubernetes cluster that does support Windows into Azure using AKS engine. AKS engine is the replacement (or rename) or ACS Engine, which is a tool that helps you to generate ARM templates that can then be used to deploy a Kubernetes cluster into Azure. This cluster is still deployed using IaaS VM’s, so your still responsible for managing these machines, but it at least removes some of the complexity of deploying a Kubernetes cluster in the first place. You are responsible for managing the Kubernetes masters, which is the crucial difference between this and AKS, so it does require you to understand Kubernetes enough to manage it.

AKS engine is the tool on top of which AKS is built, so it is maintained and updated by Microsoft and other open source contributors.

Azure Container Instances (ACI)

ACI is Microsoft’s container as a service platform, which allows you to create a container as a top-level resource in Azure quickly. There is no need to manage underlying VMs or even an orchestrator.

ACI supports running Windows containers, and it’s a single command to deploy a Windows container and get it running. However, there are some limitations to be aware of:

  1. ACI is more expensive than running a VM of the same size. ACI is intended for burst workloads that are running for short periods. If you run an ACI instance 24/7, it will cost more than running a VM of the same size (but you don’t need to worry about the underlying VM management)
  2. When you launch an ACI you need to download the container image; there is no caching of images. That means the large size of Windows containers can become a problem, especially if you are regularly killing and recreating instances. Each time you do that the image is re-downloaded.
  3. Windows Containers on ACI do not currently support joining a virtual network. You can do this with Linux CI instances, but not Windows.

Azure Web Applications

Azure Web Applications have supported running Linux containers inside a web app for some time now. Earlier last year a preview was announced for running Windows containers in web apps as well. It follows the same process as the Linux version, allowing you to specify a container image to run in the web app. There are some limitations to be aware of here as well:

  1. Using Windows containers requires using the premium PC2, PC3 or P4 SKUs, which are pretty expensive
  2. The container base image version is fixed to Server Core 2016 Long Term Servicing, so version 1607. This is quite a way behind the current release This is no longer the case, see update below
  3. It’s currently still in public preview

Running containers in web apps does offer a straightforward way to host your containers if you don’t need orchestration, but at a cost.

Update

As of April 2019 Azure Web Apps added support for Server 2019 container images in public preview. As the hosts are now running Server 2019 this means that both 2019 and later versions of 2016 (1803 and 1709) are now supported, which is great. They also added a feature for caching certain common base images, which should result in quicker start-up times. More details can be found here.

Azure Kubernetes Service

As of May 2019, Windows Support for AKS is availbile in preview, you can find out more here - https://docs.microsoft.com/en-us/azure/aks/windows-container-cli

Yes I know, I did say that AKS doesn’t support Windows containers, and it doesn’t really. However, there is a somewhat hacky way to run Windows containers on AKS using the virtual Kubelet (note this is the virtual Kubelet, not the newer virtual nodes feature). With virtual Kubelet you can have AKS call out to ACI as a virtual worker node, and ACI supports Windows containers. You can read more about this approach here.

Before you try this, you should be aware that:

  1. Virtual Kubelet is in an “experimental” state, so likely, not suitable for production
  2. As ACI does not allow you to join Windows containers to a vNet, you will have the same limitation here

Service Fabric

If you want to run Windows containers in production today, with them joined to a virtual network, then the only option that ticks all these boxes is Service Fabric (SF). SF is Microsoft’s microservices platform which can host both executables and containers. It supports both Linux and Windows containers (but not in the same cluster).

Service Fabric is a “PaaS like” solution, like AKS, where it is still deploying IaaS VM’s but with easy ways to manage and update them. That said, you are still responsible for managing and administrating these machines. As well as running containers, SF also offers some programming models to help with microservice design, based around the Actor model.

Service Fabric does still suffer from the issue of size with Windows containers when initially deploying an application to the cluster; however, after initial deployment, the base images should be cached for quicker deployment.

The minimum supported cluster size for Service Fabric in production is 5 nodes, so this can get quite expensive.

Service Fabric Mesh

Service Fabric Mesh is the next step on from Service Fabric, which is designed to provide a completely serverless, PaaS, version of Service Fabric. It is in public preview currently and is seeing reasonably regular updates, but it is still missing some features it would need to be production ready. It again supports the running of both Windows and Linux VMs.

The big downside with this is the preview state. Once this is completed, and it goes GA this could be a very easy to use and cost-effective way to run Windows containers.

Summary

The state of Windows containers in general, and on Azure specifically is still a long way behind Linux. At present, I would always recommend writing your applications to be cross-platform and running them in Linux containers if possible.

However, if you’re looking at legacy applications, then this is not always going to be possible. If you have to run Windows containers in Azure today then my approach would be:

  • If you’re only running single containers and need to run them 24/7 look at Azure Web apps for containers
  • If your looking for multi-container orchestration and aren’t set on using Kubernetes, then look at Service Fabric
  • If you need Kubernetes, then look at using AKS engine

Hopefully over the next year or so we will see Windows containers come more in line with their Linux counterparts and have more options for hosting them.