Securing Container Builds

Securing Container Builds

About this codelab

subjectLast updated Mar 23, 2023
account_circleWritten by Christopher Grant

1. Introduction

ead1609267034bf7.png

Software vulnerabilities are weaknesses that can cause an accidental system failure or provide bad actors a means to compromise your software. Container Analysis provides two kinds of OS scanning to find vulnerabilities in containers:

  • The On-Demand Scanning API allows you to manually scan container images for OS vulnerabilities, either locally on your computer or remotely in Container Registry or Artifact Registry.
  • The Container Scanning API allows you to automate OS vulnerability detection, scanning each time you push an image to Container Registry or Artifact Registry. Enabling this API also enables language package scans for Go and Java vulnerabilities.

The On-Demand Scanning API allows you to scan images stored locally on your computer, or remotely in Container Registry or Artifact Registry. This gives you granular control over the containers you want to scan for vulnerabilities. You can use On-Demand Scanning to scan images in your CI/CD pipeline before deciding whether to store them in a registry.

What you'll learn

In this lab you'll:

  • Build Images with Cloud Build
  • Use Artifact Registry for Containers
  • Utilize automated vulnerability scanning
  • Configure On Demand Scanning
  • Add image scanning in CICD in Cloud Build

2. Setup and Requirements

Self-paced environment setup

  1. Sign-in to the Google Cloud Console and create a new project or reuse an existing one. If you don't already have a Gmail or Google Workspace account, you must create one.

b35bf95b8bf3d5d8.png

a99b7ace416376c4.png

bd84a6d3004737c5.png

  • The Project name is the display name for this project's participants. It is a character string not used by Google APIs. You can update it at any time.
  • The Project ID is unique across all Google Cloud projects and is immutable (cannot be changed after it has been set). The Cloud Console auto-generates a unique string; usually you don't care what it is. In most codelabs, you'll need to reference the Project ID (it is typically identified as PROJECT_ID). If you don't like the generated ID, you may generate another random one. Alternatively, you can try your own and see if it's available. It cannot be changed after this step and will remain for the duration of the project.
  • For your information, there is a third value, a Project Number which some APIs use. Learn more about all three of these values in the documentation.
  1. Next, you'll need to enable billing in the Cloud Console to use Cloud resources/APIs. Running through this codelab shouldn't cost much, if anything at all. To shut down resources so you don't incur billing beyond this tutorial, you can delete the resources you created or delete the whole project. New users of Google Cloud are eligible for the $300 USD Free Trial program.

Environment Setup

In Cloud Shell, set your project ID and the project number for your project. Save them as PROJECT_ID and PROJECT_ID variables.

export PROJECT_ID=$(gcloud config get-value project)
export PROJECT_NUMBER=$(gcloud projects describe $PROJECT_ID \
    --format='value(projectNumber)')

Enable services

Enable all necessary services:

gcloud services enable \
  cloudkms.googleapis.com \
  cloudbuild.googleapis.com \
  container.googleapis.com \
  containerregistry.googleapis.com \
  artifactregistry.googleapis.com \
  containerscanning.googleapis.com \
  ondemandscanning.googleapis.com \
  binaryauthorization.googleapis.com

3. Building Images with Cloud Build

In this section you will create an automated build pipeline that will build your container image, scan it then evaluate the results. If no CRITICAL vulnerabilities are found it will push the image to the repository. If CRITICAL vulnerabilities are found the build will fail and exit.

Provide access for Cloud Build Service Account

Cloud Build will need rights to access the on-demand scanning api. Provide access with the following commands.

gcloud projects add-iam-policy-binding ${PROJECT_ID} \
        --member="serviceAccount:${PROJECT_NUMBER}@cloudbuild.gserviceaccount.com" \
        --role="roles/iam.serviceAccountUser"
       
gcloud
projects add-iam-policy-binding ${PROJECT_ID} \
        --member="serviceAccount:${PROJECT_NUMBER}@cloudbuild.gserviceaccount.com" \
        --role="roles/ondemandscanning.admin"

Create and change into a work directory

mkdir vuln-scan && cd vuln-scan

Define a sample image

Create a file called Dockerfile with the following contents.

cat > ./Dockerfile << EOF
FROM gcr.io/google-appengine/debian9@sha256:ebffcf0df9aa33f342c4e1d4c8428b784fc571cdf6fbab0b31330347ca8af97a

# System
RUN apt update && apt install python3-pip -y

# App
WORKDIR /app
COPY . ./

RUN pip3 install Flask==1.1.4
RUN pip3 install gunicorn==20.1.0

CMD exec gunicorn --bind :$PORT --workers 1 --threads 8 --timeout 0 main:app

EOF

Create a file called main.py with the following contents

cat > ./main.py << EOF
import os
from flask import Flask

app = Flask(__name__)

@app.route("/")
def hello_world():
   
name = os.environ.get("NAME", "Worlds")
   
return "Hello {}!".format(name)

if __name__ == "__main__":
   
app.run(debug=True, host="0.0.0.0", port=int(os.environ.get("PORT", 8080)))
EOF

Create the Cloud Build pipeline

The following command will create a cloudbuild.yaml file in your directory that will be used for the automated process. For this example the steps are limited to the container build process. In practice however you would include application specific instructions and tests in addition to the container steps.

Create the file with the following command.

cat > ./cloudbuild.yaml << EOF
steps
:

# build
- id: "build"
  name: 'gcr.io/cloud-builders/docker'
  args: ['build', '-t', 'us-central1-docker.pkg.dev/${PROJECT_ID}/artifact-scanning-repo/sample-image', '.']
  waitFor: ['-']


EOF

Run the CI pipeline

Submit the build for processing

gcloud builds submit

Review Build Details

Once the build process has Started review progress in the Cloud Build dashboard.

  1. Open Cloud Build in the Cloud Console
  2. Click on the build to view the contents

4. Artifact Registry for Containers

Create Artifact Registry Repository

In this lab you will be using Artifact Registry to store and scan your images. Create the repository with the following command.

gcloud artifacts repositories create artifact-scanning-repo \
 
--repository-format=docker \
 
--location=us-central1 \
 
--description="Docker repository"

Configure docker to utilize your gcloud credentials when accessing Artifact Registry.

gcloud auth configure-docker us-central1-docker.pkg.dev

Update the Cloud Build pipeline

Modify your build pipeline to push the resulting image to Artifact Registry

cat > ./cloudbuild.yaml << EOF
steps
:

# build
- id: "build"
  name: 'gcr.io/cloud-builders/docker'
  args: ['build', '-t', 'us-central1-docker.pkg.dev/${PROJECT_ID}/artifact-scanning-repo/sample-image', '.']
  waitFor: ['-']

# push to artifact registry
- id: "push"
  name: 'gcr.io/cloud-builders/docker'
  args: ['push',  'us-central1-docker.pkg.dev/${PROJECT_ID}/artifact-scanning-repo/sample-image']

images
:
  - us-central1-docker.pkg.dev/${PROJECT_ID}/artifact-scanning-repo/sample-image
EOF

Run the CI pipeline

Submit the build for processing

gcloud builds submit

5. Automated vulnerability scanning

Artifact scanning triggers automatically every time you push a new image to Artifact Registry or Container Registry. Vulnerability information is continuously updated when new vulnerabilities are discovered. In this section you'll review the image you just built and pushed to the Artifact Registry and explore the vulnerability results.

Review Image Details

Once the previous build process has completed review the image and Vulnerability results in the Artifact Registry dashboard.

  1. Open Artifact Registry in the Cloud Console
  2. Click on the artifact-scanning-repo to view the contents
  3. Click into the image details
  4. Click into the latest digest of your image
  5. Once the scan has finished click on the vulnerabilities tab for the image

From the vulnerabilities tab you will see the results of the automatic scanning for the image you just built.

361be7b3bf293fca.png

Automating scanning is enabled by default. Explore the Artifact Registry Settings to see how you can turn off/on auto scanning.

6. On Demand Scanning

There are various scenarios where you may need to perform a scan before pushing the image to a repository. As an example, a container developer may scan an image and fix the issues, before pushing code to the source control. In the example below you will build and analyze the image locally before acting on the results.

Build an Image

In this step you will use local docker to build the image to your local cache.

docker build -t us-central1-docker.pkg.dev/${PROJECT_ID}/artifact-scanning-repo/sample-image .

Scan the image

Once the image has been built, request a scan of the image. The results of the scan are stored in a metadata server. The job completes with a location of the results in the metadata server.

gcloud artifacts docker images scan \
    us-central1-docker.pkg.dev/${PROJECT_ID}/artifact-scanning-repo/sample-image \
    --format="value(response.scan)" > scan_id.txt

Review Output File

Take a moment to review the output of the previous step which was stored in the scan_id.txt file. Notice the report location of the scan results in the metadata server.

cat scan_id.txt

Review detailed scan results

To view the actual results of the scan use the list-vulnerabilities command on the report location noted in the output file.

gcloud artifacts docker images list-vulnerabilities $(cat scan_id.txt) 

The output contains a significant amount of data about all the vulnerabilities in the image.

Flag Critical issues

Humans rarely use the data stored in the report directly. Typically the results are used by an automated process. Use the commands below to read the report details and log if any CRITICAL vulnerabilities were found

export SEVERITY=CRITICAL

gcloud artifacts docker images list-vulnerabilities $(cat scan_id.txt) --format="value(vulnerability.effectiveSeverity)" | if grep -Fxq ${SEVERITY}; then echo "Failed vulnerability check for ${SEVERITY} level"; else echo "No ${SEVERITY} Vulnerabilities found"; fi

The output from this command will be

Failed vulnerability check for CRITICAL level

7. Scanning in CICD with Cloud Build

Provide access for Cloud Build Service Account

Cloud Build will need rights to access the on-demand scanning api. Provide access with the following commands.

gcloud projects add-iam-policy-binding ${PROJECT_ID} \
        --member="serviceAccount:${PROJECT_NUMBER}@cloudbuild.gserviceaccount.com" \
        --role="roles/iam.serviceAccountUser"
       
gcloud
projects add-iam-policy-binding ${PROJECT_ID} \
        --member="serviceAccount:${PROJECT_NUMBER}@cloudbuild.gserviceaccount.com" \
        --role="roles/ondemandscanning.admin"

Update the Cloud Build pipeline

The following command will create a cloudbuild.yaml file in your directory that will be used for the automated process. For this example the steps are limited to the container build process. In practice however you would include application specific instructions and tests in addition to the container steps.

Create the file with the following command.

cat > ./cloudbuild.yaml << EOF
steps
:

# build
- id: "build"
  name: 'gcr.io/cloud-builders/docker'
  args: ['build', '-t', 'us-central1-docker.pkg.dev/${PROJECT_ID}/artifact-scanning-repo/sample-image', '.']
  waitFor: ['-']

#Run a vulnerability scan at _SECURITY level
- id: scan
  name: 'gcr.io/cloud-builders/gcloud'
  entrypoint: 'bash'
  args:
  - '-c'
  - |
    (gcloud artifacts docker images scan \
    us-central1-docker.pkg.dev/${PROJECT_ID}/artifact-scanning-repo/sample-image \
    --location us \
    --format="value(response.scan)") > /workspace/scan_id.txt

#Analyze the result of the scan
- id: severity check
  name: 'gcr.io/cloud-builders/gcloud'
  entrypoint: 'bash'
  args:
  - '-c'
  - |
      gcloud artifacts docker images list-vulnerabilities \$(cat /workspace/scan_id.txt) \
      --format="value(vulnerability.effectiveSeverity)" | if grep -Fxq CRITICAL; \
      then echo "Failed vulnerability check for CRITICAL level" && exit 1; else echo "No CRITICAL vulnerability found, congrats !" && exit 0; fi

#Retag
- id: "retag"
  name: 'gcr.io/cloud-builders/docker'
  args: ['tag',  'us-central1-docker.pkg.dev/${PROJECT_ID}/artifact-scanning-repo/sample-image', 'us-central1-docker.pkg.dev/${PROJECT_ID}/artifact-scanning-repo/sample-image:good']


#pushing to artifact registry
- id: "push"
  name: 'gcr.io/cloud-builders/docker'
  args: ['push',  'us-central1-docker.pkg.dev/${PROJECT_ID}/artifact-scanning-repo/sample-image:good']

images
:
  - us-central1-docker.pkg.dev/${PROJECT_ID}/artifact-scanning-repo/sample-image
EOF

Run the CI pipeline

Submit the build for processing to verify the build breaks when a CRITICAL severity vulnerability is found.

gcloud builds submit

Review Build Failure

The build you just submitted will fail because the image contains CRITICAL vulnerabilities.

Review the build failure in the Cloud Build History page

Fix the Vulnerability

Update the Dockerfile to use a base image that does not contain CRITICAL vulnerabilities.

Overwrite the Dockerfile to use the Debian 10 image with the following command

cat > ./Dockerfile << EOF
from python:3.8-slim  

# App
WORKDIR
/app
COPY
. ./

RUN pip3 install
Flask==2.1.0
RUN pip3 install gunicorn
==20.1.0

CMD
exec gunicorn --bind :\$PORT --workers 1 --threads 8 main:app

EOF

Run the CI process with the good image

Submit the build for processing to verify that build will succeed when no CRITICAL severity vulnerabilities are found.

gcloud builds submit

Review Build Success

The build you just submitted will succeed because the updated image contains no CRITICAL vulnerabilities.

Review the build success in the Cloud Build History page

Review Scan results

Review the good image in Artifact registry

  1. Open Artifact Registry in the Cloud Console
  2. Click on the artifact-scanning-repo to view the contents
  3. Click into the image details
  4. Click into the latest digest of your image
  5. Click on the vulnerabilities tab for the image

8. Congratulations!

Congratulations, you finished the codelab!

What we've covered:

  • Building Images with Cloud Build
  • Artifact Registry for Containers
  • Automated vulnerability scanning
  • On Demand Scanning
  • Scanning in CICD with Cloud Build

What's next:

Clean up

To avoid incurring charges to your Google Cloud account for the resources used in this tutorial, either delete the project that contains the resources, or keep the project and delete the individual resources.

Deleting the project

The easiest way to eliminate billing is to delete the project that you created for the tutorial.

Last update: 3/21/23