Signed container image codelab

1. Overview

This codelab is built on top of Confidential Space codelab. Signed container image support improves the usability for customers by giving them an option to authenticate a container using an attested public key instead of specifying image digest in the Workload Identity Pool (WIP) policy.

What changed with signed container image support in Confidential Space:

Improved usability: With the introduction of signed container image feature we can now shift from a workload image digest approach to container signature approach for collaborators/auditors authorizing an image.

• Under the image digest approach, resource owners must update their policies with an image digest every time they authorize a new image. With the container signature approach, the policy contains a public key fingerprint, whose corresponding private key is owned by the collaborator/auditor and used to sign the audited images.
• This new approach has several advantages over the image digest approach. First, it does not require the policy to be updated every time a new image is published. Second, it uses a trusted signing key, which greatly simplifies policy management.

Note: The new approach is NOT intended to replace the current image digest approach, which is better suited for infrequent workload image changes. Instead, the new approach is ideal for environments with high image change velocity and/or with a large number of resource owners.

No security regression: This container signature approach will not bring any security regression over the previous image digest approach because the trust boundaries remain the same. In the container signature approach, the resource owner authorizes a verification key by specifying the trusted public key fingerprint in the WIP policy, and the authorization check is performed by the Attestation Verifier Service and WIP; The Attestation Verifier Service verifies the signature is associated with the running workload, and the WIP policy checks that the public key asserted by the service is authorized by policy.

The only difference between these approaches is that the latter approach uses an extra layer of indirection where workload images are authorized with a signing key. This does not introduce any new security vulnerabilities because the trust boundaries remain the same.

What you'll learn

In this codelab, you'll learn how to utilize a container image signature to authorize access to protected resources:

• How to sign an audited container image using cosign
• How to upload container image signatures to OCI registries for signature discovery and storage
• How to configure the necessary cloud resources for running Confidential Space
• How to run the workload in a Confidential Space with the signed container image support

This codelab shows you how to use Confidential Space to remote attest to a container image signed by a trusted key running on Google Compute Engine.

What you'll need

• Complete Confidential Space codelab
• A Google Cloud Platform Project
• A Browser, such Chrome or Firefox
• Familiarity with standard Linux text editors such as Vim, EMACs or Nano
• Basic knowledge of Sigstore cosign
• Basic knowledge of Google Compute Engine ( codelab), Confidential VM, Containers and remote repositories
• Basic knowledge of Cloud KMS ( codelab)
• Basic knowledge of Service Accounts, Workload identity federation and attribute conditions Workload identity federation | IAM Documentation | Google Cloud
• Basic knowledge of Artifact Registry
• Basic knowledge of Digital Signatures

Before you begin

• Ensure you have completed the environment setup for Confidential Space codelab.
• Install cosign following instructions from here.

Configure resources:

Configure the following in Primus project:

• primus_customer_list.csv: the file that contains the customer data.
• $PRIMUS_PROJECT_ID-customer-storage: the bucket that stores the customer data file.
• trusted-workload-pool: the Workload Identity Pool (WIP) that validates claims.
• attestation-verifier: the Workload Identity Pool provider which includes the authorization condition to use for tokens signed by the Attestation Verifier Service.
• trusted-workload-account: the service account that trusted-workload-pool uses to access the protected resources - in this step it has permission to view the customer data that is stored in the $PRIMUS_PROJECT_ID-customer-storage bucket.
• primus-signatures: the artifact registry to store signatures. This is a new resource needed compared to Confidential Space codelab.

Configure the following in Secundus project:

• initial_workload.go: the workload that reads in the customer data owned by Primus Bank and counts the users in a given geographic location.
• secundus-workloads: the artifact registry.
• workload-container: the Docker container that stores the workload.
• run-confidential-vm: the service account that has permission to access the Confidential VM that runs the workload.
• $SECUNDUS_PROJECT_ID-results-storage: the bucket that stores the results of the workload.

How Confidential Space with Signed Container Image support works:

In this codelab, Primus Bank will be the auditor and the resource owner, which will be responsible for following:

1. Auditing the workload code.
2. Using cosign to sign the workload image.
3. Uploading the signature to a repository.
4. Configuring WIP policy to guard customer data.

Secundus Bank will be the workload author and operator, and responsible for:

1. Writing the workload code.
2. Publishing the workload image.
3. Running the workload in Confidential Space with signed container image support.

The Secundus Bank will develop and publish a workload that will query customer data owned by the Primus Bank, and the Primus Bank will audit the workload, sign the container image, and configure WIP policies.

Existing workflow

When you run the workload in Confidential Space, the following process takes place, using the configured resources:

1. The workload requests a general Google access token for the trusted-workload-account service account from the WIP. It offers an Attestation Verifier service token with workload and environment claims.
2. If the workload measurement claims in the Attestation Verifier service token match the attribute condition in the WIP, it returns the access token for trusted-workload-account.
3. The workload uses the service account access token to use a trusted-workload-account to access the customer data in the $PRIMUS_PROJECT_ID-customer-storage bucket.
4. The workload performs an operation on that data.
5. The workload uses the run-confidential-vm service account to write the results of that operation to the $SECUNDUS_PROJECT_ID-results-storage bucket.

New workflow with signed container support

The signed container support will be integrated into the existing workflow, as highlighted below. When you run the workload in Confidential Space with signed container image support, the following process takes place, using the configured resources:

1. Confidential Space discovers any container signatures related to the current running workload image and sends these to the attestation verifier. The attestation verifier verifies the signature, and includes any valid signatures in the attestation claims.
2. The workload requests a general Google access token for the trusted-workload-account service account from the WIP. It offers an Attestation Verifier service token with workload and environment claims.
3. If the container signature claims in the Attestation Verifier service token match the attribute condition in the WIP, it returns the access token for trusted-workload-account.
4. The workload uses the service account access token to use a trusted-workload-account to access the customer data in the $PRIMUS_PROJECT_ID-customer-storage bucket.
5. The workload performs an operation on that data.
6. The workload uses the run-confidential-vm service account to write the results of that operation to the $SECUNDUS_PROJECT_ID-results-storage bucket.

2. Step 1- Set up Secundus Bank

Set the default project:

Run the following command to set the default project to $SECUNDUS_PROJECT_ID for this section of the lab.

$ gcloud config set project $SECUNDUS_PROJECT_ID

Create workload

This step is the exact same as the section here where we create workload. We can just reuse the workload code.

Build and publish the new container:

Build the workload and create a Dockerfile. Then create a private registry where the Secundus run-confidential-vm service account can be granted access. Finally, build and publish the workload to a container.

1. Build the workload. Use CGO_ENABLED=0 so that the binary is statically linked.

$ go mod init initial_workload && go mod tidy
$ CGO_ENABLED=0 go build initial_workload.go

2. Create a Dockerfile.

$ cat <<EOF > Dockerfile
FROM alpine:latest

WORKDIR /test
COPY initial_workload /test

ENTRYPOINT ["/test/initial_workload"]

LABEL "tee.launch_policy.allow_cmd_override"="true"
LABEL "tee.launch_policy.log_redirect"="always"

CMD []
EOF

3. Create an Artifact Registry.

$ gcloud artifacts repositories create secundus-workloads \
  --repository-format=docker --location=us

4. Build and Publish container..

$ gcloud auth configure-docker us-docker.pkg.dev

$ docker build -t us-docker.pkg.dev/$SECUNDUS_PROJECT_ID/secundus-workloads/workload-container:latest .

$ docker push us-docker.pkg.dev/$SECUNDUS_PROJECT_ID/secundus-workloads/workload-container:latest

Create run-confidential-vm service account:

Complete the following steps:

1. Create the run-confidential-vm service account.

$ gcloud iam service-accounts create run-confidential-vm

2. Grant the Service Account User role on the run-confidential-vm service account to your user account. This allows your user account to impersonate the service account.

$ gcloud iam service-accounts add-iam-policy-binding \
run-confidential-vm@$SECUNDUS_PROJECT_ID.iam.gserviceaccount.com \
--member="user:$(gcloud config get-value account)" \
--role='roles/iam.serviceAccountUser'

3. Grant the service account the confidentialcomputing.workloadUser role. This will allow the user account to generate an attestation token.

$ gcloud projects add-iam-policy-binding $SECUNDUS_PROJECT_ID \
--member=serviceAccount:run-confidential-vm@$SECUNDUS_PROJECT_ID.iam.gserviceaccount.com \
--role=roles/confidentialcomputing.workloadUser

4. (Optional) Grant the service account the Log Writer permission. This allows the Confidential Space environment to write logs to Cloud Logging in addition to the Serial Console, so you can review logs after the VM is terminated (Requires Security Admin permission).

$ gcloud projects add-iam-policy-binding $SECUNDUS_PROJECT_ID \
--member=serviceAccount:run-confidential-vm@$SECUNDUS_PROJECT_ID.iam.gserviceaccount.com \
--role=roles/logging.logWriter

5. Grant the Viewer role on the secundus-workloads repository to the run-confidential-vm service account.

$ gcloud artifacts repositories add-iam-policy-binding \ secundus-workloads --project=$SECUNDUS_PROJECT_ID \ 
--role='roles/viewer' --location=us \
--member="serviceAccount:run-confidential-vm@$SECUNDUS_PROJECT_ID.iam.gserviceaccount.com"

Create a bucket for results:

Create the $SECUNDUS_PROJECT_ID-results-storage bucket. Then grant the run-confidential-vm service account permission to create files in the bucket, so it can store the workload results there.

1. Create the results-storage bucket.

$ gsutil mb gs://$SECUNDUS_PROJECT_ID-results-storage

2. Grant the Storage Object Creator role on the $SECUNDUS_PROJECT_ID-results-storage bucket to the run-confidential-vm service account. This permits the service account to store query results to the bucket.

$ gsutil iam ch \
    serviceAccount:run-confidential-vm@$SECUNDUS_PROJECT_ID.iam.gserviceaccount.com:objectCreator \
    gs://$SECUNDUS_PROJECT_ID-results-storage

3. Step 2 - Setup Primus Bank:

Set the default project:

Run the following command to set the default project to $PRIMUS_PROJECT_ID for this section of the lab.

$ gcloud config set project $PRIMUS_PROJECT_ID

Upload customer data to bucket:

1. To create the primus_customer_list.csv file, run the following at the command line.

$ cat <<EOF >> primus_customer_list.csv
15,Alice,Seattle
36,Bob,Everett
56,Eve,Shoreline
134,Ashley,Seattle
305,Clinton,Redmond
506,Stephen,Kirkland
788,Cooper,Tacoma
987,Eleanor,Bellevue
1052,April,Everett
1113,Lincoln,Bellevue
1990,Phillip,Tacoma
2048,Eric,Seattle
EOF

2. To create the bucket and upload the file, complete the following steps.

// Create a bucket
$ gsutil mb gs://$PRIMUS_PROJECT_ID-customer-storage

// Upload the CSV file to the bucket
$ gsutil cp primus_customer_list.csv \ gs://$PRIMUS_PROJECT_ID-customer-storage/primus_customer_list.csv

Create trusted-workload-account service account

Create the trusted-workload-account service account, and then grant it the Storage Object Viewer role on the $PRIMUS_PROJECT_ID-customer-storage bucket.

1. Create the trusted-workload-account service account

$ gcloud iam service-accounts create trusted-workload-account

2. Grant the Storage Object Viewer role on the $PRIMUS_PROJECT_ID-customer-storage bucket to the service account. This permits the service account to view the customer list stored in the bucket

$ gsutil iam ch \
serviceAccount:trusted-workload-account@$PRIMUS_PROJECT_ID.iam.gserviceaccount.com:objectViewer \
gs://$PRIMUS_PROJECT_ID-customer-storage

Specify a different repository than the workload image before signing:

Cosign will default to storing signatures in the same repo as the image it is signing. To specify a different repository for signatures, you can set the COSIGN_REPOSITORY environment variable.

Here we'll use Artifact Registry as an example. You can also choose other OCI-based registries such as Docker Hub, AWS CodeArtifact based on your preference.

Note: This step is highly recommended for use cases where the signatory does not have write permissions to the workload repository.

1. Create an Artifact Registry docker repository.

$ gcloud artifacts repositories create primus-signatures \
  --repository-format=docker --location=us

2. For Artifact Registry, a full image name such as $LOCATION/$PROJECT/$REPOSITORY/$IMAGE_NAME is expected. You can upload any container image to the repository for signature storage.

$ export COSIGN_REPOSITORY=us-docker.pkg.dev/$PRIMUS_PROJECT_ID/primus-signatures/demo

Note: different registries might expect different formats for the "repository."

• A GCR registry will expect format like:

$ export COSIGN_REPOSITORY=gcr.io/primus-signatures

• An Artifact Registry will expect a full image name like $LOCATION-docker.pkg.dev/$PROJECT/$REPO/$STORAGE_IMAGE, not just a repository. For example,

$ export COSIGN_REPOSITORY=us-docker.pkg.dev/$PRIMUS_PROJECT_ID/primus-signatures/demo

Specifying just a repo like us-docker.pkg.dev/$PRIMUS_PROJECT_ID/primus-signatures will not work in Artifact Registry.

3. Grant the Viewer role on the primus-signatures repository to the run-confidential-vm service account. This allows Confidential Space to discover any container image signatures uploaded to the primus-signatures.

$ gcloud artifacts repositories add-iam-policy-binding primus-signatures \
--project=$PRIMUS_PROJECT_ID --role='roles/viewer' --location=us \
--member="serviceAccount:run-confidential-vm@$SECUNDUS_PROJECT_ID.iam.gserviceaccount.com"

Use cosign to sign the workload:

Cosign is a powerful tool with multiple signing features. For our use case, we only require Cosign to sign with a key pair. Cosign keyless signing is not supported for this signed container image feature.

When signing with a key pair, there are two options:

1. Sign with a local key pair generated by Cosign.
2. Sign with a key pair stored elsewhere (for example, in a KMS).

Note: Confidential Space with signed container image support only supports Cosign sign with RSA and ECDSA keys.

1. Generate a key pair in Cosign if you don't have one. Refer to signing with self-managed keys for more details.

// Generate keys using Cosign.
$ cosign generate-key-pair

// Generate keys using a KMS provider.
$ cosign generate-key-pair --kms <some provider>://<some key>

In the above replace <some provider>://<some key> with gcpkms://projects/$PROJECT/locations/$LOCATION/keyRings/$KEYRING/cryptoKeys/$KEY/versions/$KEY_VERSION

• <some provider> : Refers to the KMS solution you are using
• <some key> : Refers to the key path in KMS

2. Retrieve the public key for verification.

// For KMS providers.
$ cosign public-key --key <some provider>://<some key> > pub.pem

// For local key pair signing.
$ cosign public-key --key cosign.key > pub.pem

3. Sign the workload using Cosign. Perform unpadded base64 encoding on the public key

$ PUB=$(cat pub.pem | openssl base64)

// Remove spaces and trailing "=" signs.
$ PUB=$(echo $PUB | tr -d '[:space:]' | sed 's/[=]*$//')

Note: We use the unpadded RFC4648 scheme to encode PEM public key as Cosign does not allow any extra "=" signs in the payload annotations.

4. Sign the workload using Cosign with the exported public key and signature algorithms attached.

$ IMAGE_REFERENCE=us-docker.pkg.dev/$SECUNDUS_PROJECT_ID/secundus-workloads/workload-container:latest

// Set Application Default Credentials.
$ gcloud auth application-default login 

// Sign with KMS support.
$ cosign sign --key <some provider>://<some key> $IMAGE_REFERENCE \
-a dev.cosignproject.cosign/sigalg=ECDSA_P256_SHA256 \
-a dev.cosignproject.cosign/pub=$PUB

// Sign with a local key pair.
$ cosign sign --key cosign.key $IMAGE_REFERENCE \
-a dev.cosignproject.cosign/sigalg=ECDSA_P256_SHA256 \
-a dev.cosignproject.cosign/pub=$PUB

• --key [REQUIRED] specifies which signing key to use. When referring to a key managed by a KMS provider, please follow specific URI format from Sigstore KMS support. When referring to a key generated by Cosign, use cosign.key instead.
• $IMAGE_REFERENCE [REQUIRED] specifies which container image to sign. The format of IMAGE_REFERENCE can be identified by tag or image digest. For example: us-docker.pkg.dev/$SECUNDUS_PROJECT_ID/secundus-workloads/workload-container:latest or us-docker.pkg.dev/$SECUNDUS_PROJECT_ID/secundus-workloads/workload-container[IMAGE-digest]
• -a [REQUIRED] specifies annotations attached to the signature payload. For Confidential Space signed container images, public key and signature algorithms are required to be attached to signature payload.

Warning: Do not change dev.cosignproject.cosign/pub and dev.cosignproject.cosign/sigalg as Confidential Space will only accept these key names.

• dev.cosignproject.cosign/sigalg ONLY accepts three values:
• RSASSA_PSS_SHA256: RSASSA algorithm with PSS padding with a SHA256 digest.
• RSASSA_PKCS1V15_SHA256: RSASSA algorithm with PKCS#1 v1.5 padding with a SHA256 digest.
• ECDSA_P256_SHA256: ECDSA on the P-256 Curve with a SHA256 digest. This is also the default signature algorithm for Cosign-generated key pairs.

Note: Please choose the corresponding signature algorithm when using Cosign to sign. Also for RSA keys, they should be at least 2048-bit length long.

5. Upload signatures to the docker repository

Cosign sign will automatically upload signatures to the specified COSIGN_REPOSITORY.

Create and configure Workload Identity Pool (WIP):

Primus Bank wants to authorize workloads to access their customer data based on attributes of the following resources.

• What: Code that is verified
• Where: An environment that is secure
• Who: An operator that is trusted

To create the WIP to enforce an access policy based on these requirements, complete the following steps:

1. Compute the fingerprint of a public key in hex format

$ openssl pkey -pubin -in pub.pem -outform DER | openssl sha256
SHA2-256(stdin)= <fingerprint of public key>

2. Create a WIP.

$ gcloud iam workload-identity-pools create trusted-workload-pool \
    --location="global"

3. Create a new OIDC workload identity pool provider.

$ gcloud iam workload-identity-pools providers create-oidc attestation-verifier \
    --location="global" \
    --workload-identity-pool="trusted-workload-pool" \
    --issuer-uri="https://confidentialcomputing.googleapis.com/" \
    --allowed-audiences="https://sts.googleapis.com" \
    --attribute-mapping="google.subject='assertion.sub'" \
    --attribute-condition="assertion.swname == 'CONFIDENTIAL_SPACE' &&
      'STABLE' in assertion.submods.confidential_space.support_attributes
      && 'run-confidential-vm@$SECUNDUS_PROJECT_ID.iam.gserviceaccount.com' in
      assertion.google_service_accounts"
      && [
'ECDSA_P256_SHA256:<fingerprint of the public key>'
].exists(fingerprint, fingerprint in assertions.submods.container.image_signatures.map(sig,sig.signature_algorithm+':'+sig.key_id))

Note: Update the <fingerprint of public key> in step three with the computed fingerprint in step 1 here.

The specified –attribute-condition authorizes access to the signed secundus-workloads container. It requires:

• What: The trusted public key fingerprint whose corresponding private key was used to sign the current running workload.
• Where: Confidential Space trusted execution environment.
• Who: Secundus Bank run-confidential-vm service account.

4. Grant the workloadIdentityUser role on the trusted-workload-account service account to the trusted-workload-pool WIP. This allows the WIP to impersonate the service account.

$ gcloud iam service-accounts add-iam-policy-binding \
trusted-workload-account@$PRIMUS_PROJECT_ID.iam.gserviceaccount.com \
--role=roles/iam.workloadIdentityUser \
--member="principalSet://iam.googleapis.com/projects/$PRIMUS_PROJECT_NUMBER/locations/global/workloadIdentityPools/trusted-workload-pool/*"

4. Step 3 - Secundus runs the workload

Set the default project:

Run the following command to set the default project to $SECUNDUS_PROJECT_ID for this section of the lab.

$ gcloud config set project $SECUNDUS_PROJECT_ID

Create the instance:

In the Secundus project, create a Confidential VM instance, and then view the results of the workload.

$ gcloud compute instances create signed-container-vm \ --confidential-compute \
--shielded-secure-boot \
--maintenance-policy=TERMINATE \
--scopes=cloud-platform --zone=us-west1-b \
--image-project=confidential-space-images \
--image-family=confidential-space \
--service-account=run-confidential-vm@$SECUNDUS_PROJECT_ID.iam.gserviceaccount.com \
--metadata ^~^tee-image-reference=us-docker.pkg.dev/$SECUNDUS_PROJECT_ID/secundus-workloads/workload-container:latest \
~tee-restart-policy=Never \
~tee-cmd="[\"list-common-customers\",\"gs://$SECUNDUS_PROJECT_ID-customer-storage/encrypted_secundus_customer_list.csv\",\"projects/$SECUNDUS_PROJECT_ID/locations/global/keyRings/secundus-data-keys/cryptoKeys/customer-data-key\",\"trusted-workload-account@$SECUNDUS_PROJECT_ID.iam.gserviceaccount.com\",\"projects/$SECUNDUS_PROJECT_NUMBER/locations/global/workloadIdentityPools/trusted-workload-pool/providers/attestation-verifier\",\"gs://$SECUNDUS_PROJECT_ID-results-storage/list-common-result\"]" \
~tee-signed-image-repos=us-docker.pkg.dev/$PRIMUS_PROJECT_ID/primus-signatures/demo

Note: Use "," to concatenate multiple signature repositories. Example: ~tee-signed-image-repos=gcr.io/signature_a,gcr.io/signature_b

View results:

In the Secundus project, view the results of the workload.

$ gsutil cat gs://$SECUNDUS_PROJECT_ID-results-storage/seattle-result

The result should be 3, as this is how many people from Seattle are listed in the primus_customer_list.csv file!

Cleanup:

// Set the project to the $SECUNDUS_PROJECT_ID project.
$ gcloud config set project $SECUNDUS_PROJECT_ID
// Delete the results file.
$ gsutil rm gs://$SECUNDUS_PROJECT_ID-results-storage/seattle-result

// Delete the Confidential VM instance.
$ gcloud compute instances delete signed-container-vm

Summary:

Now with the signed container image support for Confidential Space, every time Secundus Bank releases a new workload image, Primus Bank audits the workload code and signs the new workload image without needing to update the WIP policy with the image digest.

5. Congratulations

Congratulations, you've successfully completed the codelab!

You learned how to leverage the signed container image feature to improve usability of Confidential Space**.**

Clean up

If you are done exploring, please consider deleting your project.

• Go to the Cloud Platform Console
• Select the project you want to shut down, then click ‘Delete' at the top: this schedules the project for deletion

What's next?

Check out some of these codelabs...

• Encrypt and decrypt data with Cloud KMS
• Grant access to your project with IAM
• Developing Containers with Dockerfiles

Further reading

• Feeling isolated? Confidential computing to the rescue
• Confidential Computing at GCP
• Confidential Space: The future of privacy-preserving collaboration
• How Google and Intel make Confidential Computing more secure
• Privacy vs. Progress - Advancing Security with Google Cloud Confidential Computing