Capturing Terraform Azure CLI Traffic with Fiddler

This topic is a bit specific, but it’s a problem I spent quite a long time on this week, so I’m documenting it for anyone who tries to do the same thing.

When working with Azure in Terraform, you occasionally find you need to something that isn’t supported by the Terraform Azure modules for whatever reason. On these occasions, I tend to fall back onto using the “local-exec” resource, which calls the Azure CLI.

resource "null_resource" "test" {
  provisioner "local-exec" {
    command = "az aks get-credentials -n name -g resourcegroup
  }
}

This method works reasonably well as the CLI can use the same credentials as your Terraform connection and now you have access to do anything the CLI can do. You have to be careful to keep your process idempotent, but it gets you out of a pinch when nothing else will do it.

This week I was trying to run a command through the CLI, and I was having some issues getting the syntax correct. What I needed to write in Terraform, using the correct string escaping and so on, was failing when it hit the Azure API. It seemed like what I thought I was generating wasn’t what was getting sent, so I had the bright idea of running Fiddler to capture the command being sent to the API. Using this, I could compare this to what was being sent when I ran this command in the CLI manually (which worked), simple right? Now ensued many hours that I’m not getting back.

Using Fiddler Proxy

The first issue I needed to contend with was the fact that the Python library behind the CLI doesn’t seem to use the system proxy, which Fiddler sets up automatically. So to make it work, you need to set some environment variables in the same command line window you are using to run Terraform, to force it to use Fiddler as the proxy. The commands below assume you are running Fiddler on the default port of 8888

PowerShell

$env:HTTP_PROXY="http://localhost:8888"
$env:HTTPS_PROXY="http://localhost:8888"

Bash

SET HTTP_PROXY="http://localhost:8888"
SET HTTPS_PROXY="http://localhost:8888"

Certificate

Now our traffic is routing via Fiddler. However, all our Terraform/CLI traffic is going over HTTPS, so to be able to read this traffic, we need to enable Fiddler to intercept HTTPS traffic. This is a built-in feature of Fiddler and is documented here. The issue is, to be able to intercept traffic Fiddler presents its HTTPS certificate to calls to its proxy, this causes the CLI to error as it does not trust this certificate. You’ll see an error like this:

Please ensure you have a network connection. Error detail: HTTPSConnectionPool(host='login.microsoftonline.com',
port=443): Max retries exceeded with url: /common/oauth2/devicecode?api-version=1.0 (Caused by
a ProxyError('Cannot connect to proxy.', NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection
object at 0x7f405b6eea58>: Failed to establish a new connection: [Errno 111] Connection refused',)))

Skipping Decryption

The first thing I found was that if Fiddler attempted to decrypt traffic to Azure AD when you logged in to the CLI, then nothing worked, so we need to disable that. Open Fiddler, go to the “Tools” menu and then the “HTTPS” tab. In this window enter the following URLs into the “skip decryption” box.

• login.microsoftonline.com
• graph.windows.net

Certificate Trust

Now that we can log in, we need to find a way to have the CLI not error when presented with the Fiddler certificate

First Attempt

The first thing I tried was having the CLI not check that it trusts the certificate. Note this is not something to ever do in production. Still, as a temporary measure running on my machine, I was happy to try it. You can do this by setting another environment variable, but don’t bother; you’ll see why shortly!

PowerShell

$env:AZURE_CLI_DISABLE_CONNECTION_VERIFICATION=1

Bash

SET AZURE_CLI_DISABLE_CONNECTION_VERIFICATION=1

Setting this variable did allow the CLI to ignore the validity of the certificate. Still, the problem now is that it outputs a warning indicating it is not checking the certificate validity. It seems that Terraform cannot cope with this warning, and so fails to work. Back to the drawing board.

Second Attempt

What did work was getting the CLI to trust the certificate Fiddler is presenting. This way it doesn’t output any warnings and Terraform is happy with the response.

To do this, we need to make sure we have already installed the Fiddler cert on our machine. If you have not, then follow the instructions in this article. You should only do this on a machine used for testing, and ideally, remove the trust when you are done.

We can now find this certificate in a local machine certificate store, search for “manage computer certificates” in the start menu. Once the certificate MMC opens, expand “trusted root certificates” and then “certificates” and find the cert called “DO_NOT_TRUST_FiddlerRoot”. Right-click on the certificate, go to tasks, then export. In the wizard that opens, export this to a CER file.

Open the file you exported in notepad, and copy the content, which should look like this:

Now we have the cert; we need to add it to the list of certs that the CLI trusts. To do this, you need to open the CA bundle used by the CLI.

On Windows this is located here - C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\Lib\site-packages\certifi\cacert.pem

On Linux it is located here - ``` /opt/az/lib/python3.6/site-packages/certifi/cacert.pem ``

Open this up in Notepad and scroll to the very end. Paste in the content of the CER file from earlier. Save the file and then close and re-open the CLI. If you need to, re-set the environment variables for the proxy, and traffic should now flow through fiddler.

Once I’d spent multiple hours getting this all working, I found the issue on the first request and fixed it in 5 minutes.