Scanning Containers during Builds with Azure Security Centre

Azure, Containers

If you are creating applications that run in containers then scanning your container images for vulnerabilities is important. This will make sure that your container images are up to date and not using an vulnerable software (at least at the time of scanning).

There are plenty of tools on the market that will do this for you. Microsoft have recently partnered with Qualys for scanning of Azure Container registries as part of Azure Security Centre. Scanning of ACR was announced at Ignite an went GA earlier this year. If you’d like to get more detail on how to setup ACR scanning with Security Centre then @Pixel_Robots has a great post on this here .

Image Scanning

The ACR integration with Security Centre will scan an image when it is pushed to the registry and you can then review the results through the security centre UI. This is fine for your security team who want to monitor a registry. However a very common approach is to scan your images during your build pipeline, so that you can detect any issues at build time and fail the build if they are vulnerable.

Many of the commercial tools for container scanning offer direct integration with your build pipeline, but Azure Security Centre does not. For it to scan the image it needs to be pushed to the registry. I hope this may change in the future, but for now we are going to look at a way you can replicate something similar using this tool.

Image Quarantine

As I mentioned, for Security Centre to scan the image it needs to be pushed to the registry. This means that a vulnerable image will end up in the registry and be available for use before you can scan it and then decide to remove it. Which is not ideal.

The solution to this is to quarantine images when you first push them to the registry, that way no one can download them. You can then scan the image, make a decision and either delete it or remove it from quarantine. I talked about using quarantine with ACR here. Unfortunately this solution is still in preview and the vulnerability scanning does not seem to integrate with it properly yet. Once it does, this will be the way to go.

Image Publishing Pipeline

For the rest of this article we will look at how you can use your CI/CD pipeline to scan images at the time they are created. For this demo I will use Azure DevOps, but this could be done with any CI/CD tool.

Build and Publish Images

The first step in the process is to actually build and publish your images to the Azure Container Registry. This is no different to how you would have done this previously. If you have not done this before you can see more details here.

Getting Scan Results

Once the image is published to ACR it is automatically scanned, we don’t need to do anything to make that happen. What we need is a way to retrieve the results of the scan in our pipeline. Fortunately all of the security centre assessments are accessible through the API. One of the easiest ways to query this is using the Azure Resource Graph. If you are not familiar with resource graph it provides a way to use the Kusto query language (also used by Log Analytics) to query your Azure Resources. If you want to find out more about Resource Graph see this article.

Using Resource Graph we can grab all the events relating to container registry vulnerabilities.

securityresources
| where type == "microsoft.security/assessments"
| summarize by assessmentKey=name //the ID of the assessment
| join kind=inner (
  securityresources
  | where type == "microsoft.security/assessments/subassessments"
  | extend assessmentKey = extract(".*assessments/(.+?)/.*",1, id)
) on assessmentKey
| where properties.additionalData.assessedResourceType == "ContainerRegistryVulnerability"
| extend status = properties.status.code
| extend severity = properties.status.severity

Query Scan Results in the Build Pipeline

To use this query as part of our build pipeline we are going to write a PowerShell script we can run as a task. You could do this in whatever scripting language you prefer.

You will notice that we are using the Azure CLI to talk to Azure rather than Azure PowerShell. This is due to the fact that when we get to the point of wanting the delete an image, there is no way to do this in Azure CLI currently. Azure DevOps doesn’t like mixing Azure CLI and Azure PowerShell in the same script, so we are doing everything in the CLI.

The Resource Graph query above will get us assessments for all scans in the registry, but we want to narrow this down to just the container(s) we are interested in as part of our build. To do this we need to filter on the specific digest of the containers, this is the SHA256 hash of the container image. This isn’t the easiest thing to obtain, so in our PowerShell the first thing we do is go to the registry and look this up, based on the image name and tag.

We have parameterised the PowerShell script to accept a search string for the different container image names (or repositories as ACR calls them) so we can look at multiple images (all with the same tag) if we need to.

 param(
   [Parameter(Mandatory = $true)]
   [string]$repositorySearchString,
   [Parameter(Mandatory = $true)]
   [string]$imageTag,
   [Parameter(Mandatory = $true)]
   [string]$registryName,
   [Parameter(Mandatory = $true)]
   [string]$registrysubscription,
   [int]$timeoutInSeconds= 600
 )

 az account set --subscription $registrysubscription
 $healthyCount = 0
 $unHealthyCount = 0


 $respositories = $(az acr repository list -n $destinationRegistryName | convertfrom-Json) | where-object { $_ -like $repositorySearchString }

 foreach ($repository in $respositories) {
   write-host $repository
   $digest = $($(az acr repository show-tags -n $destinationRegistryName --repository $repository --detail | convertfrom-json) | where-object { $_.name -eq "$imageTag" }).digest
   if ($null -ne $digest ) {
     write-host $digest
     $query = @"
 securityresources
 | where type == "microsoft.security/assessments"
 | summarize by assessmentKey=name //the ID of the assessment
 | join kind=inner (
   securityresources
   | where type == "microsoft.security/assessments/subassessments"
   | extend assessmentKey = extract(".**assessments/(.+?)/.**",1, id)
 ) on assessmentKey
 | where properties.additionalData.assessedResourceType == "ContainerRegistryVulnerability"
 | where properties.resourceDetails.id == "/repositories/$repository/images/$digest"
 | extend status = properties.status.code
 | extend severity = properties.status.severity
 "@
 
 $results = search-azgraph -query $query -Subscription $registrysubscription
 $timeout = 0
 while ($results.count -eq 0 -and $timeout -lt $timeoutInSeconds) {
   write-host "Waiting for scan"
   start-sleep 30
   $timeout = $timeout +30
   $results = search-azgraph -query $query -Subscription $registrysubscription
 }
 
 $status = "Healthy"
 
 foreach ($result in $results) {
   if ($result.status -eq "Unhealthy") {
     $unHealthyCount++
     $status = "Unhealthy"

   }
  
 }
 
 if($results.count -eq 0){
   throw "No scan results found"
 }

The process here is to:

  1. Get a list of the different repositories/images we want to query
  2. Loop through each one and get the digest of the specific tag we are looking for
  3. Use resource graph to query for the ACR status
  4. Loop until we get the status back (to account for delays in scanning). In theory this could take up to 10 minutes, but it has always been much quicker for me. We add a timeout to fail after 10 minutes

Now that we have the results, we can review them to check the status of the image. If the image is healthy there will be a single record with the status of “healthy”. If the image is unhealthy there will be a record for each vulnerability, each with a status of “unhealthy”. We check each record to get it’s status.

If the image is found to be unhealthy then we go ahead and delete the image from the repository.

   if ($status -eq "Unhealthy") {
      write-error "$repository`:$tagName is $status"
      az acr repository delete --name $destinationRegistryName --image "$repository`:$tagName" --yes
    }
    else {
      write-host "$repository`:$tagName is $status" -ForegroundColor Green
      $healthyCount++
    }

Finally, if any of the images we scanned are unhealthy, we throw an error to fail the build:

 if ($unHealthyCount -gt 0) {
   throw "At least one image with vulnerabilities detected"
 }
 else {
   if($healthyCount -gt 0){
   write-host "No vulnerabilities found" -ForegroundColor Green
   }
   else{
     Write-Warning "No images found"
   }
 }

The full script for this process is below, and on Github here.

param(
    [Parameter(Mandatory = $true)]
    [string]$repositorySearchString,
    [Parameter(Mandatory = $true)]
    [string]$imageTag,
    [Parameter(Mandatory = $true)]
    [string]$registryName,
    [Parameter(Mandatory = $true)]
    [string]$registrysubscription,
    [int]$timeoutInSeconds= 600
)

az account set --subscription $registrysubscription
$healthyCount = 0
$unHealthyCount = 0


$respositories = $(az acr repository list -n $destinationRegistryName | convertfrom-Json) | where-object { $_ -like $repositorySearchString }

foreach ($repository in $respositories) {
    write-host $repository
    $digest = $($(az acr repository show-tags -n $destinationRegistryName --repository $repository --detail | convertfrom-json) | where-object { $_.name -eq "$imageTag" }).digest
    if ($null -ne $digest ) {
        write-host $digest
        $query = @"
securityresources
| where type == "microsoft.security/assessments"
| summarize by assessmentKey=name //the ID of the assessment
 | join kind=inner (
    securityresources
     | where type == "microsoft.security/assessments/subassessments"
     | extend assessmentKey = extract(".*assessments/(.+?)/.*",1,  id)
 ) on assessmentKey
 | where properties.additionalData.assessedResourceType == "ContainerRegistryVulnerability"
 | where properties.resourceDetails.id == "/repositories/$repository/images/$digest"
 | extend status = properties.status.code
 | extend severity = properties.status.severity
"@

        $results = search-azgraph -query $query -Subscription $registrysubscription
        $timeout = 0
        while ($results.count -eq 0 -and $timeout -lt $timeoutInSeconds) {
            write-host "Waiting for scan"
            start-sleep 30
            $timeout = $timeout +30
            $results = search-azgraph -query $query -Subscription $registrysubscription
        }

        if($results.count -eq 0){
            throw "No scan results found"
        }

        $status = "Healthy"

        foreach ($result in $results) {
            if ($result.status -eq "Unhealthy") {
                $unHealthyCount++
                $status = "Unhealthy"
      
            }
  
        }
 
  
        if ($status -eq "Unhealthy") {
            write-error "$repository`:$tagName is $status"
            az acr repository delete --name $destinationRegistryName --image "$repository`:$tagName" --yes
        }
        else {
            write-host "$repository`:$tagName is $status" -ForegroundColor Green
            $healthyCount++
        }
    }
    else {
        write-warning "No image found for $repository`:$tagName"
    }

}

if ($unHealthyCount -gt 0) {
    throw "At least one image with vulnerabilities detected"
}
else {
    if($healthyCount -gt 0){
    write-host "No vulnerabilities found" -ForegroundColor Green
    }
    else{
        Write-Warning "No images found"
    }
}

Summary

This is a bit of a quick and dirty solution to get the basics of vulnerability scanning into our CI/CD pipelines if you are using Security Centre for scanning your images. Many of the commercial tools have much slicker processes, and I hope that eventually ACR will catch up, but for now this solution is what works.