Signed container image codelab

1. Overview

This codelab builds on the Confidential Space codelab. Signed container image support given an option to authenticate a container using an attested public key instead of specifying image digest in the Workload Identity Pool (WIP) policy.

What changed with signed container image support in Confidential Space:

Improved usability: With the introduction of signed container image feature we can now shift from a workload image digest approach to container signature approach for collaborators/auditors authorizing an image.

When using image digests directly, resource owners must update their policies with an image digest every time they authorize a new image. By using image signatures,, the policy contains a public key fingerprint, whose corresponding private key is owned by the collaborator/auditor and used to sign the audited images.
For some security models, referencing a trusted image signing key is more convenient than updating a list of new image digest values.

No security regression: This container signature approach will not bring any security regression over the previous image digest approach because the trust boundaries remain the same. In the container signature approach, the resource owner authorizes a verification key by specifying the trusted public key fingerprint in the WIP policy, and the authorization check is performed by the Attestation Verifier Service and WIP; The Attestation Verifier Service verifies the signature is associated with the running workload, and the WIP policy checks that the public key asserted by the service is authorized by policy.

Strong security: Using container image signatures allows one to delegate some amount of trust to the image signer. By specifying a trusted signer's public key fingerprint in the attestation policy, the resource owner authorizes that signer to make endorsements on which container images meet a policy. The Attestation Verifier Service verifies the signature is associated with the running workload, and the policy checks that the public key that created the signature is authorized by policy. Through this, the added layer of indirection that image signing provides maintains the strong security guarantees of Confidential Space.

The only difference between these approaches is that the latter approach uses an extra layer of indirection where workload images are authorized with a signing key. This does not introduce any new security vulnerabilities because the trust boundaries remain the same.

What you'll learn

In this codelab, you'll learn how to utilize a container image signature to authorize access to protected resources:

How to sign an audited container image using cosign
How to upload container image signatures to OCI registries for signature discovery and storage
How to configure the necessary cloud resources for running Confidential Space
How to run the workload in a Confidential Space with the signed container image support

This codelab shows you how to use Confidential Space to remote attest to a container image signed by a trusted key running on Google Compute Engine.

What you'll need

Complete Confidential Space codelab
A Google Cloud Platform Project
A Browser, such Chrome or Firefox
Familiarity with standard Linux text editors such as Vim, Emacs or Nano
Basic knowledge of Sigstore cosign
Basic knowledge of Google Compute Engine ( codelab), Confidential VM, Containers and remote repositories
Basic knowledge of Cloud KMS ( codelab)
Basic knowledge of Service Accounts, Workload identity federation and attribute conditions.
Basic knowledge of Artifact Registry
Basic knowledge of Digital Signatures

Roles involved in a Confidential Space with Signed Container Image

In this codelab, Primus Bank will be the auditor and the resource owner, which will be responsible for the following:

Setting up required resources with sample data.
Auditing the workload code.
Using cosign to sign the workload image.
Uploading the signature to a repository.
Configuring WIP policy to protect customer data.

Secundus Bank will be the workload author and operator, and responsible for:

Setting up required resources to store the result.
Writing the workload code.
Publishing the workload image.
Running the workload in Confidential Space with signed container image support.

The Secundus Bank will develop and publish a workload that will query customer data stored in a cloud storage bucket and owned by the Primus Bank. The Primus Bank will audit the workload, sign the container image, and configure WIP policies to allow access to their data by approved workloads. The result of this workload execution will be stored in a cloud storage bucket owned by Secundus bank.

Resources involved in a Confidential Space setup

This codelab references a number of variables that you should set to appropriate values for your GCP project. The commands in this codelab assume that these variables have been set. (for example, export PRIMUS_INPUT_STORAGE_BUCKET='my-input-bucket' can be used to set the name of the input storage bucket of Primus bank.) If variables of the resource-names have not been set, then it will be generated based on GCP project-id.

Configure the following in Primus project:

$PRIMUS_INPUT_STORAGE_BUCKET: the bucket that stores the customer data file.
$PRIMUS_WORKLOAD_IDENTITY_POOL: the Workload Identity Pool (WIP) that validates claims.
$PRIMUS_WIP_PROVIDER: the Workload Identity Pool provider which includes the authorization condition to use for tokens signed by the Attestation Verifier Service.
$PRIMUS_SERVICEACCOUNT: the service account that $PRIMUS_WORKLOAD_IDENTITY_POOL uses to access the protected resources. In this step it has permission to view the customer data that is stored in the $PRIMUS_INPUT_STORAGE_BUCKET bucket.
$PRIMUS_ENC_KEY: the KMS key used to encrypt the data stored in $PRIMUS_INPUT_STORAGE_BUCKET.

Resources new to this codelab:

$PRIMUS_COSIGN_REPOSITORY: the Artifact Registry to store workload image signatures.
$PRIMUS_SIGNING_KEY: the KMS key used to sign the workload image by auditor/data-collaborators (e.g primus bank in this case).

Configure the following in Secundus project:

$SECUNDUS_ARTIFACT_REGISTRY: the artifact registry where workload docker image will be pushed.
$WORKLOAD_IMAGE_NAME: the name of workload docker image.
$WORKLOAD_IMAGE_TAG: the tag of workload docker image.
$WORKLOAD_SERVICEACCOUNT: the service account that has permission to access the Confidential VM that runs the workload.
$SECUNDUS_RESULT_BUCKET: the bucket that stores the results of the workload.

Other Resources:

primus_customer_list.csv contains the customer data. We will upload this data to $PRIMUS_INPUT_STORAGE_BUCKET and create a workload that will query this data.

Existing workflow

When you run the workload in Confidential Space, the following process takes place, using the configured resources:

The workload requests a general Google access token for the $PRIMUS_SERVICEACCOUNT from the WIP. It offers an Attestation Verifier service token with workload and environment claims.
If the workload measurement claims in the Attestation Verifier service token match the attribute condition in the WIP, it returns the access token for $PRIMUS_SERVICEACCOUNT.
The workload uses the service account access token associated with $PRIMUS_SERVICEACCOUNT to access the customer data in the $PRIMUS_INPUT_STORAGE_BUCKET bucket.
The workload performs an operation on that data.
The workload uses the $WORKLOAD_SERVICEACCOUNT service account to write the results of that operation to the $SECUNDUS_RESULT_STORAGE_BUCKET bucket.

New workflow with signed container support

The signed container support will be integrated into the existing workflow, as highlighted below. When you run the workload in Confidential Space with signed container image support, the following process takes place, using the configured resources:

Confidential Space discovers any container signatures related to the current running workload image and sends these to the attestation verifier. The attestation verifier verifies the signature, and includes any valid signatures in the attestation claims.
The workload requests a general Google access token for the $PRIMUS_SERVICEACCOUNT from the WIP. It offers an Attestation Verifier service token with workload and environment claims.
If the container signature claims in the Attestation Verifier service token match the attribute condition in the WIP, it returns the access token for $PRIMUS_SERVICEACCOUNT.
The workload uses the service account access token associated with $PRIMUS_SERVICEACCOUNT to access the customer data in the $PRIMUS_INPUT_STORAGE_BUCKET bucket.
The workload performs an operation on that data.
The workload uses the $WORKLOAD_SERVICEACCOUNT to write the results of that operation to the $SECUNDUS_RESULT_STORAGE_BUCKET bucket.

2. Set Up Cloud Resources

As part of Confidential Space setup, first you will create the required cloud resources under GCP projects of Primus and Secundus bank. These are the resources new to this codelab:

In the Primus project:

KMS signing key used to sign Secundus workloads, after auditing the code.
Artifact registry repository to store the Cosign signatures.

There are no new resources in the Secundus project. Once these resources are set up, you will create a service account for the workload with required roles and permissions. You will then create a workload image and the auditor, Primus bank, will sign the workload image. Workload will be then authorized by data-collaborators (Primus bank in this codelab) and workload operator (Secundus Bank in this case) will run the workload.

As part of Confidential Space setup, you will create the required cloud resources in the Primus and Secundus GCP projects.

Before you begin

Clone this repository using the below command to get required scripts that are used as part of this codelab.

$ git clone https://github.com/GoogleCloudPlatform/confidential-space

Ensure you have set the required projects as shown below.

$ export PRIMUS_PROJECT_ID=<GCP project id of primus bank>
$ export SECUNDUS_PROJECT_ID=<GCP project id of secundus bank>

Set the variables for resource names mentioned above using this command. You can override the resource names using these variables (e.g export PRIMUS_INPUT_STORAGE_BUCKET='my-input-bucket')
Run the following script to set the remaining variable names to values based on your project ID for resource names.

$ source config_env.sh

Install cosign following instructions from here.

Set up Primus bank resources

As part of this step, you will set up the required cloud resources for Primus bank. Run the following script to set up the resources for Primus bank. As part of these steps, below mentioned resources will be created:

Cloud storage bucket ($PRIMUS_INPUT_STORAGE_BUCKET) to store the encrypted customer data file of Primus bank.
Encryption key ($PRIMUS_ENC_KEY) and keyring ($PRIMUS_ENC_KEYRING) in KMS to encrypt the data file of Primus bank.
Workload identity pool ($PRIMUS_WORKLOAD_IDENTITY_POOL) to validate claims based on attributes conditions configured under its provider.
Service account ($PRIMUS_SERVICEACCOUNT) attached to above mentioned workload identity pool ($PRIMUS_WORKLOAD_IDENTITY_POOL) with with following IAM access:
roles/cloudkms.cryptoKeyDecrypter to decrypt the data using the KMS key.
objectViewer to read data from the cloud storage bucket.
roles/iam.workloadIdentityUser for connecting this service account to the workload identity pool.

$ ./setup_primus_bank_resources.sh

Set up Secundus bank resources

As part of this step, you will set up the required cloud resources for Secundus bank. Run the following script to set up the resources for Secundus bank. As part of this steps below mentioned resources will be created:

Cloud storage bucket ($SECUNDUS_RESULT_STORAGE_BUCKET) to store the result of workload execution by Secundus bank.

$ ./setup_secundus_bank_resources.sh

3. Create and Sign Workload

Create workload service account

Now, you will create a service account for the workload with required roles and permissions. Run the following script to create a workload service account in the Secundus bank project. This service-account would be used by the VM that runs the workload.

This workload service-account ($WORKLOAD_SERVICEACCOUNT) will have the following roles:
confidentialcomputing.workloadUser to get an attestation token
logging.logWriter to write logs to Cloud Logging.
objectViewer to read data from the $PRIMUS_INPUT_STORAGE_BUCKET cloud storage bucket.
objectAdmin to write the workload result to the $SECUNDUS_RESULT_STORAGE_BUCKET cloud storage bucket.

$ ./create_workload_serviceaccount.sh

Create workload

As part of this step, you will create a workload Docker image. The workload used in this Codelab is a simple CLI-based Go app which counts customers (from Primus bank customer data) from a provided geographical location in argument. Run the following script to create a workload in which the following steps are being performed:

Create Artifact Registry($SECUNDUS_ARTIFACT_REGISTRY) owned by Secundus bank.
Update the workload code with required resources names. Here is the workload code used for this codelab.
Build Go binary and create Dockerfile for building a Docker image of the workload code. Here is the Dockerfile used for this codelab.
Build and publish the Docker image to the Artifact Registry ($SECUNDUS_ARTIFACT_REGISTRY) owned by Secundus bank.
Grant $WORKLOAD_SERVICEACCOUNT read permission for $SECUNDUS_ARTIFACT_REGISTRY. This is needed for the workload container to pull the workload docker image from the Artifact Registry.

$ ./create_workload.sh

Sign Workload

We will be using Cosign to sign the workload image. Cosign will default to storing signatures in the same repo as the image it is signing. To specify a different repository for signatures, you can set the COSIGN_REPOSITORY environment variable.

Here we'll use Artifact Registry as an example. You can also choose other OCI-based registries such as Docker Hub, AWS CodeArtifact based on your preference.

Create an Artifact Registry docker repository.

$ gcloud config set project $PRIMUS_PROJECT_ID

$ gcloud artifacts repositories create $PRIMUS_COSIGN_REPOSITORY \
  --repository-format=docker --location=us

Create a keyring and key under KMS for signing a workload image.

$ gcloud config set project $PRIMUS_PROJECT_ID

$ gcloud kms keyrings create $PRIMUS_SIGNING_KEYRING \
  --location=global

$ gcloud kms keys create $PRIMUS_SIGNING_KEY \
  --keyring=$PRIMUS_SIGNING_KEYRING \
  --purpose=asymmetric-signing \
  --default-algorithm=ec-sign-p256-sha256
  --location=us

For Artifact Registry, a full image name such as $LOCATION/$PROJECT/$REPOSITORY/$IMAGE_NAME is expected. You can upload any container image to the repository for signature storage.

$ export COSIGN_REPOSITORY=us-docker.pkg.dev/${PRIMUS_PROJECT_ID}/${PRIMUS_COSIGN_REPOSITORY}/demo

Grant the Viewer role on the $PRIMUS_COSIGN_REPOSITORY repository to the $WORKLOAD_SERVICEACCOUNT service account. This allows Confidential Space to discover any container image signatures uploaded to the $PRIMUS_COSIGN_REPOSITORY.

$ gcloud artifacts repositories add-iam-policy-binding ${PRIMUS_COSIGN_REPOSITORY} \
--project=${PRIMUS_PROJECT_ID} --role='roles/viewer' --location=us \
--member="serviceAccount:${WORKLOAD_SERVICEACCOUNT}@${SECUNDUS_PROJECT_ID}.iam.gserviceaccount.com"

Cosign is a powerful tool with multiple signing features. For our use case, we only require Cosign to sign with a key pair. Cosign keyless signing is not supported for this signed container image feature.

When signing with a key pair, there are two options:

Sign with a local key pair generated by Cosign.
Sign with a key pair stored elsewhere (for example, in a KMS).

Generate a key pair in Cosign if you don't have one. Refer to signing with self-managed keys for more details.

// Set Application Default Credentials.
$ gcloud auth application-default login 

// Generate keys using a KMS provider.
$ cosign generate-key-pair --kms <provider>://<key>

// Generate keys using Cosign.
$ cosign generate-key-pair

In the above replace <provider>://<key> with gcpkms://projects/$PRIMUS_PROJECT_ID/locations/global/keyRings/$PRIMUS_SIGNING_KEYRING/cryptoKeys/$PRIMUS_SIGNING_KEY/cryptoKeyVersions/$PRIMUS_SIGNING_KEYVERSION

<provider> : Refers to the KMS solution you are using
<key> : Refers to the key path in KMS

Retrieve the public key for verification.

// For KMS providers.
$ cosign public-key --key <some provider>://<some key> > pub.pem

// For local key pair signing.
$ cosign public-key --key cosign.key > pub.pem

Sign the workload using Cosign. Perform unpadded base64 encoding on the public key

$ PUB=$(cat pub.pem | openssl base64)

// Remove spaces and trailing "=" signs.
$ PUB=$(echo $PUB | tr -d '[:space:]' | sed 's/[=]*$//')

Sign the workload using Cosign with the exported public key and signature algorithms attached.

$ IMAGE_REFERENCE=us-docker.pkg.dev/$SECUNDUS_PROJECT_ID/$SECUNDUS_ARTIFACT_REPOSITORY/$WORKLOAD_IMAGE_NAME:$WORKLOAD_IMAGE_TAG

// Sign with KMS support.
$ cosign sign --key <some provider>://<some key> $IMAGE_REFERENCE \
-a dev.cosignproject.cosign/sigalg=ECDSA_P256_SHA256 \
-a dev.cosignproject.cosign/pub=$PUB

// Sign with a local key pair.
$ cosign sign --key cosign.key $IMAGE_REFERENCE \
-a dev.cosignproject.cosign/sigalg=ECDSA_P256_SHA256 \
-a dev.cosignproject.cosign/pub=$PUB

--key [REQUIRED] specifies which signing key to use. When referring to a key managed by a KMS provider, please follow specific URI format from Sigstore KMS support. When referring to a key generated by Cosign, use cosign.key instead.
$IMAGE_REFERENCE [REQUIRED] specifies which container image to sign. The format of IMAGE_REFERENCE can be identified by tag or image digest. For example: us-docker.pkg.dev/$SECUNDUS_PROJECT_ID/secundus-workloads/workload-container:latest or us-docker.pkg.dev/$SECUNDUS_PROJECT_ID/secundus-workloads/workload-container[IMAGE-digest]
-a [REQUIRED] specifies annotations attached to the signature payload. For Confidential Space signed container images, public key and signature algorithms are required to be attached to signature payload.

dev.cosignproject.cosign/sigalg ONLY accepts three values:
RSASSA_PSS_SHA256: RSASSA algorithm with PSS padding with a SHA256 digest.
RSASSA_PKCS1V15_SHA256: RSASSA algorithm with PKCS#1 v1.5 padding with a SHA256 digest.
ECDSA_P256_SHA256: ECDSA on the P-256 Curve with a SHA256 digest. This is also the default signature algorithm for Cosign-generated key pairs.

Upload signatures to the docker repository

Cosign sign will automatically upload signatures to the specified COSIGN_REPOSITORY.

4. Authorize and Run Workload

Authorize Workload

As part of this step, we will be setting up the workload identity provider under the workload identity pool ($PRIMUS_WORKLOAD_IDENTITY_POOL). There are attribute-conditions configured for the workload identity as shown below. One of the conditions is to validate the workload image signature's fingerprint against signing public key's fingerprint. With this attribute condition, when Secundus Bank releases a new workload image, Primus Bank audits the workload code and signs the new workload image without needing to update the WIP policy with the image digest.

$ gcloud config set project $PRIMUS_PROJECT_ID

$ PUBLIC_KEY_FINGERPRINT=$(openssl pkey -pubin -in pub.pem -outform DER | openssl sha256 | cut -d' ' -f2)

$ gcloud iam workload-identity-pools providers create-oidc ${PRIMUS_WIP_PROVIDER} \
   --location="global" \
   --workload-identity-pool="${PRIMUS_WORKLOAD_IDENTITY_POOL}" \
   --issuer-uri="https://confidentialcomputing.googleapis.com/" \
   --allowed-audiences="https://sts.googleapis.com" \
   --attribute-mapping="google.subject='assertion.sub'" \
   --attribute-condition="assertion.swname == 'CONFIDENTIAL_SPACE' &&
  'STABLE' in assertion.submods.confidential_space.support_attributes
     && '${WORKLOAD_SERVICEACCOUNT}@${SECUNDUS_PROJECT_ID}.iam.gserviceaccount.com' in
     assertion.google_service_accounts
     && ['ECDSA_P256_SHA256:${PUBLIC_KEY_FINGERPRINT}']
       .exists(fingerprint, fingerprint in assertion.submods.container.image_signatures.map(sig,sig.signature_algorithm+':'+sig.key_id))"

Run Workload

As part of this step, we will be running the workload on Confidential VM. Required TEE arguments are passed using the metadata flag. Arguments for the workload container are passed using the "tee-cmd" portion of the flag. The workload is coded to publish its result to $SECUNDUS_RESULT_STORAGE_BUCKET.

$ gcloud config set project $SECUNDUS_PROJECT_ID

$ gcloud compute instances create signed-container-vm \
 --confidential-compute-type=SEV \
 --shielded-secure-boot \
 --maintenance-policy=TERMINATE \
 --scopes=cloud-platform --zone=us-west1-b \
 --image-project=confidential-space-images \
 --image-family=confidential-space \ --service-account=${WORKLOAD_SERVICEACCOUNT}@${SECUNDUS_PROJECT_ID}.iam.gserviceaccount.com \
 --metadata "^~^tee-image-reference=us-docker.pkg.dev/${SECUNDUS_PROJECT_ID}/${SECUNDUS_ARTIFACT_REPOSITORY}/${WORKLOAD_IMAGE_NAME}:${WORKLOAD_IMAGE_TAG}~tee-restart-policy=Never~tee-cmd="[\"count-location\",\"Seattle\",\"gs://${SECUNDUS_RESULT_STORAGE_BUCKET}/seattle-result\"]"~tee-signed-image-repos=us-docker.pkg.dev/${PRIMUS_PROJECT_ID}/${PRIMUS_COSIGN_REPOSITORY}/demo"

View results

In the Secundus project, view the results of the workload.

$ gcloud config set project $SECUNDUS_PROJECT_ID

$ gsutil cat gs://$SECUNDUS_RESULT_STORAGE_BUCKET/seattle-result

The result should be 3, as this is how many people from Seattle are listed in the primus_customer_list.csv file!

5. Clean Up

Here is the script that can be used to clean up the resources that we have created as part of this codelab. As part of this cleanup, the following resources will be deleted:

Input storage bucket of Primus bank ($PRIMUS_INPUT_STORAGE_BUCKET).
Primus bank service-account ($PRIMUS_SERVICEACCOUNT).
Primus Bank artifact registry which holds image signatures ($PRIMUS_COSIGN_REPOSITORY).
Primus Bank workload identity pool ($PRIMUS_WORKLOAD_IDENTITY_POOL).
Workload service account of Secundus Bank ($WORKLOAD_SERVICEACCOUNT).
Workload Compute Instance.
Result storage bucket of Secundus Bank ($SECUNDUS_RESULT_STORAGE_BUCKET).
Artifact registry of Secundus Bank ($SECUNDUS_ARTIFACT_REGISTRY).

// run the clean up script to delete the resources created as part of this codelab.
$ ./cleanup.sh

If you are done exploring, please consider deleting your project.

Go to the Cloud Platform Console
Select the project you want to shut down, then click ‘Delete' at the top: this schedules the project for deletion

Congratulations

Congratulations, you've successfully completed the codelab!

You learned how to leverage the signed container image feature to improve usability of Confidential Space.

What's next?

Check out some of these similar codelabs...