Use Confidential Space with protected resources that aren't stored with a cloud provider

1. Overview

Confidential Space offers secure multi-party data sharing and collaboration, while allowing organizations to preserve the confidentiality of their data. This means that organizations can collaborate with each other while still maintaining control over their data and protecting it from unauthorized access.

Confidential Space unlocks scenarios where you want to gain mutual value from aggregating and analyzing sensitive, often regulated, data, while retaining full control over it. With Confidential Space, organizations can gain mutual value from aggregating and analyzing sensitive data such as personally identifiable information (PII), protected health information (PHI), intellectual property, and cryptographic secrets — while retaining full control over it.

What you'll need

What you'll learn

  • How to configure the necessary Cloud resources for running Confidential Space
  • How to run a workload in a Confidential VM running the Confidential Space image
  • How to authorize access to protected resources based on the attributes of the workload code (what), the Confidential Space environment (where) and the account that is running the workload (who).

This codelab focuses on how to use Confidential Space with protected resources that are hosted somewhere other than Google Cloud. You will learn how to request a custom, self contained token from the Google Attestation Service by providing a nonce, audience and the PKI token type.

In this codelab, you will set up a Confidential Space between a fictional product - USleep, a containerized application, and a fictional product - UWear, a connected wearable device, to calculate your sleep quality. UWear will share protected health information (PHI) with USleep in a safe, secure and isolated environment (aka Trusted Execution Environment or TEE) such that the owners of the data retain complete confidentiality.

UWear is both the workload auditor and data owner. As workload auditor, it reviews the code in the workload that's being run and takes note of the image digest. As data owner, UWear writes the verification logic to check the validity of the token and its signature. It writes a validation policy, using the audited workloads image digest, that only allows the specific image digest, in a specific environment, to gain access to the sensitive data.

USleep, in this codelab, is deploying the containerized application. USleep does not have access to the sensitive data but runs the approved workload that is allowed access to the sensitive data.

The codelab involves the following steps:

  • Step 1: Set up necessary cloud resources for the codelab. Set up projects, billing, and permissions. Download the codelab source code and set environment variables.
  • Step 2: Download the root certificate and store it with your UWear source code.
  • Step 3: Create separate workload service accounts which will be used by the workload VM for USleep and UWear.
  • Step 4: Create the USleep workload which provides an attestation token.
  • Step 5: Create the UWear workload which validates the attestation token and sends the sensitive data if the token is approved.
  • Step 6: Run the USleep and UWear workloads. UWear will provide the sensitive data, and USleep will run a sleep algorithm on the data and output a result.
  • Step 7: (Optional) Run an unauthorized USleep workload and confirm sensitive data has not been received from UWear.
  • Step 8: Clean up all resources.

Understanding the workflow

USleep will be running the workload in Confidential Space. In order to run the workload it needs access to UWear's PHI. To gain access, the USleep workload first creates a secure TLS session. USleep will then also request an attestation token from the Google Attestation Service with a payload.

USleep will request an attestation token with a JSON payload that will contain three things:

  1. An attestation token that is bound to the TLS session. In order to bind the attestation token to the TLS session the nonce value will be the hash of the TLS Exported Keying Material. Binding the token to the TLS session ensures that there are no machine-in-the-middle attacks occurring since only the two parties involved in the TLS session will be able to generate the nonce value.
  2. An audience of "uwear" will be provided. UWear will be verifying that it is the intended audience for the attestation token.
  3. A token type of "PKI". A token type of "PKI" means USleep would like to request a self contained token. The self contained token can be verified that it is signed by Google by using the root downloaded from Confidential Space's well known PKI endpoint. This is in contrast to the default OIDC token type, whose signature is verified using a public key that rotates regularly.

bb013916a3222ce7.png

USleep workload receives the attestation token. UWear then joins the TLS connection with USleep and retrieves USleep's attestation token. UWear will validate the token by checking the x5c claim against the root certificate.

UWear will approve the USleep workload if:

  1. The token passes the PKI validation logic.
  2. UWear will validate the token by checking the x5c claim against the root certificate, checking the token is signed by the leaf certificate and lastly that the downloaded root certificate is the same root as in the x5c claim.
  3. The workload measurement claims in the token match the attribute conditions specified in the OPA policy. OPA is an open source, general purpose policy engine that unifies policy enforcement across the stack. OPA uses documents, with similar syntax to JSON, to set baseline values that the policy is validated against. See OPA baseline values for an example of what values the policy checks for.
  4. The nonce matches the expected nonce (TLS Exported Keying Material). This is verified in the OPA policy above.

Once all of those checks have been completed and have passed, UWear can confirm that the data will be sent and processed securely. UWear will then respond back with the sensitive PHI over the same TLS session and USleep will be able to use that data to calculate the customer's sleep quality.

2. Set Up Cloud Resources

Before you begin

  1. Set up two Google Cloud projects, one for USleep and one for UWear. For more information on creating a Google Cloud project, please refer to the "Set up and navigate your first Google project" codelab. You can refer to creating and managing projects to get details about how to retrieve the project ID and how it is different from project name and project number.
  2. Enable Billing for your projects.
  3. In one of your Google project's Cloud Shell, set the required project environment variables as shown below.
export UWEAR_PROJECT_ID=<Google Cloud project id of UWear>
export USLEEP_PROJECT_ID=<Google Cloud project id of USleep>
  1. Enable Confidential Computing API and following APIs for both the projects.
gcloud config set project $UWEAR_PROJECT_ID
gcloud services enable \
    cloudapis.googleapis.com \
    cloudshell.googleapis.com \
    container.googleapis.com \
    containerregistry.googleapis.com \
    confidentialcomputing.googleapis.com

gcloud config set project $USLEEP_PROJECT_ID
gcloud services enable \
    cloudapis.googleapis.com \
    cloudshell.googleapis.com \
    container.googleapis.com \
    containerregistry.googleapis.com \
    confidentialcomputing.googleapis.com
  1. Retrieve your Principal identifier using
gcloud auth list

# Output should contain
# ACCOUNT: <Principal Identifier>

# Set your member variable
export MEMBER='user:<Principal Identifier>'
  1. Add permissions for these two projects. Permissions can be added by following the details on the grant an IAM role webpage.
gcloud config set project $UWEAR_PROJECT_ID

# Add Artifact Registry Administrator role
gcloud projects add-iam-policy-binding $UWEAR_PROJECT_ID --member=$MEMBER --role='roles/iam.serviceAccountAdmin'

# Add Service Account Administrator role
gcloud projects add-iam-policy-binding $UWEAR_PROJECT_ID --member=$MEMBER --role='roles/artifactregistry.admin'

gcloud config set project $USLEEP_PROJECT_ID

# Add Service Account Administrator role
gcloud projects add-iam-policy-binding $USLEEP_PROJECT_ID --member=$MEMBER --role='roles/iam.serviceAccountAdmin'

# Add Artifact Registry Administrator role
gcloud projects add-iam-policy-binding $USLEEP_PROJECT_ID --member=$MEMBER --role='roles/artifactregistry.admin'

# Add Compute Administrator role
gcloud projects add-iam-policy-binding $USLEEP_PROJECT_ID --member=$MEMBER --role='roles/compute.admin'

# Add Storage Administrator role
gcloud projects add-iam-policy-binding $USLEEP_PROJECT_ID --member=$MEMBER --role='roles/compute.storageAdmin'
  1. In one of your Google Cloud projects Cloud Shell, clone the Confidential Space Codelab Github Repository using the below command to get the required scripts that are used as part of this codelab.
git clone https://github.com/GoogleCloudPlatform/confidential-space.git
  1. Change the directory to the scripts directory for the health data codelab.
cd confidential-space/codelabs/health_data_analysis_codelab/scripts
  1. Update these two lines in the config_env.sh script, located in codelabs/health_data_analysis_codelab/scripts directory. Update the project IDs with your project IDs for USleep and UWear. Be sure to remove the comment symbol "#" at the beginning of the line.
# TODO: Populate UWear and USleep Project IDs
export UWEAR_PROJECT_ID=your-uwear-project-id
export USLEEP_PROJECT_ID=your-usleep-project-id
  1. Optional: Set any pre-existing variables. You can override the resource names using these variables (e.g export UWEAR_ARTIFACT_REPOSITORY='my-artifact-repository')
  • You can set the following variables with existing cloud resource names. If the variable is set, then the corresponding existing cloud resource from the project would be used. If the variable is not set, cloud resource name would be generated from the values in the config_env.sh script.
  1. Run the config_env.sh script to set the remaining variable names to values based on your project ID for resource names.
# Navigate to the scripts folder
cd ~/confidential-space/codelabs/health_data_analysis_codelab/scripts

# Run the config_env script
source config_env.sh

# Verify the variables were set
# Expected output for default variable should be `workload-sa`
echo $USLEEP_WORKLOAD_SERVICE_ACCOUNT

3. Download the root certificate

  1. In order to validate the self contained token returned from the attestation service, UWear will need to validate the signature against the Confidential Space root certificate. UWear will need to download the root certificate and store it locally. In one of your Google Cloud project's console, run the following commands:
cd ~/confidential-space/codelabs/health_data_analysis_codelab/src/uwear

wget https://confidentialcomputing.googleapis.com/.well-known/confidential_space_root.crt -O confidential_space_root.pem
  1. Generate the fingerprint of the root certificate that was downloaded
openssl x509 -fingerprint -in confidential_space_root.pem -noout
  1. Verify the fingerprint matches the following SHA-1 digest:
B9:51:20:74:2C:24:E3:AA:34:04:2E:1C:3B:A3:AA:D2:8B:21:23:21

4. Create workload service account

Now, you will create two service accounts; one for the USleep and one for the UWear workloads. Run the create_service_accounts.sh script to create workload service accounts in the USleep and UWear projects. The VMs that run the workloads would use these service accounts.

# Navigate to the scripts folder
cd ~/confidential-space/codelabs/health_data_analysis_codelab/scripts

# Run the create_service_accounts script
./create_service_accounts.sh

The script:

  • Grants the iam.serviceAccountUser role which attaches the service account to the workload.
  • Grants the confidentialcomputing.workloadUser role to the workload service account . This will allow the user account to generate an attestation token.
  • Grants the logging.logWriter role to the workload service account permission. This allows the Confidential Space environment to write logs to Cloud Logging in addition to the Serial Console, so logs are available after the VM is terminated.Create workloads

5. Create USleep Workload

As part of this step, you will create Docker images for the workloads used in this codelab. The USleep workload is a simple Golang application which determines the quality of a customer's sleep using personal health information on a wearable device.

About the USleep Workload

The USleep workload is a simple Golang application which determines the quality of a customer's sleep, using personal health information on a wearable device. The USleep workload has three main pieces to it:

  1. Setting up a TLS Session and extracting the Exported Keying Material
func handleConnectionRequest(w http.ResponseWriter, r *http.Request) {
  // Upgrade HTTP Connection to a websocket.
  conn, err := upgrader.Upgrade(w, r, nil)
  if err != nil {
    fmt.Printf("failed to upgrade connection to a websocket with err: %v\n", err)
    return
  }
  defer conn.Close()

  // Get EKM
  hash, err := getEKMHashFromRequest(r)
  if err != nil {
    fmt.Printf("Failed to get EKM: %v", err)
  }
  ...
}

func getEKMHashFromRequest(r *http.Request) (string, error) {
  ekm, err := r.TLS.ExportKeyingMaterial("testing_nonce", nil, 32)
  if err != nil {
    err := fmt.Errorf("failed to get EKM from inbound http request: %w", err)
    return "", err
  }

  sha := sha256.New()
  sha.Write(ekm)
  hash := base64.StdEncoding.EncodeToString(sha.Sum(nil))

  fmt.Printf("EKM: %v\nSHA hash: %v", ekm, hash)
  return hash, nil
}
  1. Requesting a token from the Attestation Service with an audience, nonce and PKI token type.
func handleConnectionRequest(w http.ResponseWriter, r *http.Request) {
  ...

  // Request token with TLS Exported Keying Material (EKM) hashed.
  token, err := getCustomToken(hash)
  if err != nil {
    fmt.Printf("failed to get custom token from token endpoint: %v", err)
    return
  }

  // Respond to the client with the token.
  conn.WriteMessage(websocket.TextMessage, token)

  ...
}

var (
        socketPath    = "/run/container_launcher/teeserver.sock"
        tokenEndpoint = "http://localhost/v1/token"
        contentType   = "application/json"
)


func getCustomToken(nonce string) ([]byte, error) {
  httpClient := http.Client{
    Transport: &http.Transport{
      // Set the DialContext field to a function that creates
      // a new network connection to a Unix domain socket
      DialContext: func(_ context.Context, _, _ string) (net.Conn, error) {
        return net.Dial("unix", socketPath)
      },
    },
  }

  body := fmt.Sprintf(`{
    "audience": "uwear",
    "nonces": ["%s"],
    "token_type": "PKI"
  }`, nonce)

  resp, err := httpClient.Post(tokenEndpoint, contentType, strings.NewReader(body))
  if err != nil {
    return nil, err
  }

  fmt.Printf("Response from launcher: %v\n", resp)
  text, err := io.ReadAll(resp.Body)
  if err != nil {
    return nil, fmt.Errorf("Failed to read resp.Body: %w", err)
  }
  fmt.Printf("Token from the attestation service: %s\n", text)

  return text, nil
}
  1. Receiving the sensitive data and calculating the user's sleep quality
func handleConnectionRequest(w http.ResponseWriter, r *http.Request) {
  ...

  // Read the sensitive data
  _, content, err := conn.ReadMessage()
  if err != nil {
    fmt.Printf("failed to read message from the connection: %v\n", err)
  }
  fmt.Printf("Received content from other side, %v\n", string(content))

 // TODO: Handle sensitive data
  ...
}

Steps to create the USleep Workload

  1. Run the create_usleep_workload.sh script to create the USleep workload. This script:
  • Creates Artifact Registry ($USLEEP_ARTIFACT_REPOSITORY) owned by UWear where the workload would be published.
  • Builds the usleep/workload.go code and package it in a Docker image. See the Dockerfile configuration for USleep.
  • Publishes the Docker image to the Artifact Registry ($USLEEP_ARTIFACT_REPOSITORY) owned by UWear.
  • Grants the service account $USLEEP_WORKLOAD_SERVICE_ACCOUNT read permission for the Artifact Registry ($USLEEP_ARTIFACT_REPOSITORY).
./create_usleep_workload.sh
  1. Important: In the output logs, extract the image digest for USleep.
latest: digest: sha256:<USLEEP_IMAGE_DIGEST> size: 945
  1. Navigate to the UWear directory
cd ~/confidential-space/codelabs/health_data_analysis_codelab/src/uwear
  1. Replace the value under "allowed_submods_container_image_digest" in the opa_validation_values.json with the USLEEP_IMAGE_DIGEST.
# Replace the image digest
sed -i 's/sha256:bc4c32cb2ca046ba07dcd964b07a320b7d0ca88a5cf8e979da15cae68a2103ee/sha256:<USLEEP_IMAGE_DIGEST>/' ~/confidential-space/codelabs/health_data_analysis_codelab/src/uwear/opa_validation_values.json

6. Create UWear Workload

About the UWear Workload

The UWear workload has 4 main pieces to it:

  1. Joining the same TLS session that was created in USleep' workload and retrieving the attestation token from USleep over the secure TLS Session.
func main() {
  fmt.Println("Initializing client...")

  tlsconfig := &tls.Config{
    // Skipping client verification of the server's certificate chain and host name since we are
    // doing custom verification using the attestation token.
    InsecureSkipVerify: true,
  }

  dialer := websocket.Dialer{
    TLSClientConfig:  tlsconfig,
    HandshakeTimeout: 5 * time.Second,
  }

  ipAddress := os.Getenv(ipAddrEnvVar)
  url := fmt.Sprintf("wss://%s:8081/connection", ipAddress)

  fmt.Printf("Attempting to dial to url %v...\n", url)
  conn, _, err := dialer.Dial(url, nil)
  if err != nil {
    fmt.Printf("Failed to dial to url %s, err %v\n", url, err)
    return
  }

  defer conn.Close()

  tokenString, ekm, err := retrieveTokenAndEKMFromConn(conn)
  if err != nil {
    fmt.Printf("Failed to retrieve token and EKM from connection: %v\n", err)
    return
  }

  fmt.Printf("token: %v\n", tokenString)

  ...
}
  1. Validating the self contained token by:
  • Checking the x5c claim contains a certificate chain that chains correctly from the leaf certificate to the intermediate and finally to the root certificate.
  • Checking the token is signed by the leaf certificate contained in the x5c claim.
  • Checking the downloaded / stored root certificate is the same root as in the x5c claim.
func main() {
  ...

  token, err := validatePKIToken(tokenString)
  if err != nil {
    fmt.Printf("Failed to validate PKI token, err: %v\n.", err)
    return
  }
  fmt.Println("PKI token validated successfully")
 
  ...
}

// validatePKIToken validates the PKI token returned from the attestation service.
// It verifies the token the certificate chain and that the token is signed by Google
// Returns a jwt.Token or returns an error if invalid.
func validatePKIToken(attestationToken string) (jwt.Token, error) {
  // IMPORTANT: The attestation token should be considered untrusted until the certificate chain and
  // the signature is verified.
  rawRootCertificate, err := readFile(rootCertificateFile)
  if err != nil {
    return jwt.Token{}, fmt.Errorf("readFile(%v) - failed to read root certificate: %w", rootCertificateFile, err)
  }

  storedRootCert, err := decodeAndParseCertificate(string(rawRootCertificate))
  if err != nil {
    return jwt.Token{}, fmt.Errorf("DecodeAndParseCertificate(string) - failed to decode and parse root certificate: %w", err)
  }

  jwtHeaders, err := extractJWTHeaders(attestationToken)
  if err != nil {
    return jwt.Token{}, fmt.Errorf("ExtractJWTHeaders(token) - failed to extract JWT headers: %w", err)
  }

  if jwtHeaders["alg"] != "RS256" {
    return jwt.Token{}, fmt.Errorf("ValidatePKIToken(attestationToken, ekm) - got Alg: %v, want: %v", jwtHeaders["alg"], "RS256")
  }

  // Additional Check: Validate the ALG in the header matches the certificate SPKI.
  // https://datatracker.ietf.org/doc/html/rfc5280#section-4.1.2.7
  // This is included in Golang's jwt.Parse function

  x5cHeaders := jwtHeaders["x5c"].([]any)
  certificates, err := extractCertificatesFromX5CHeader(x5cHeaders)
  if err != nil {
    return jwt.Token{}, fmt.Errorf("ExtractCertificatesFromX5CHeader(x5cHeaders) returned error: %w", err)
  }

  // Verify the leaf certificate signature algorithm is an RSA key
  if certificates.LeafCert.SignatureAlgorithm != x509.SHA256WithRSA {
    return jwt.Token{}, fmt.Errorf("leaf certificate signature algorithm is not SHA256WithRSA")
  }

  // Verify the leaf certificate public key algorithm is RSA
  if certificates.LeafCert.PublicKeyAlgorithm != x509.RSA {
    return jwt.Token{}, fmt.Errorf("leaf certificate public key algorithm is not RSA")
  }

  // Verify the storedRootCertificate is the same as the root certificate returned in the token
  // storedRootCertificate is downloaded from the confidential computing well known endpoint
  // https://confidentialcomputing.googleapis.com/.well-known/attestation-pki-root
  err = compareCertificates(*storedRootCert, *certificates.RootCert)
  if err != nil {
    return jwt.Token{}, fmt.Errorf("failed to verify certificate chain: %w", err)
  }

  err = verifyCertificateChain(certificates)
  if err != nil {
    return jwt.Token{}, fmt.Errorf("VerifyCertificateChain(CertificateChain) - error verifying x5c chain: %v", err)
  }

  keyFunc := func(token *jwt.Token) (any, error) {
    return certificates.LeafCert.PublicKey, nil
  }

  verifiedJWT, err := jwt.Parse(attestationToken, keyFunc)
  return *verifiedJWT, err
}



// verifyCertificateChain verifies the certificate chain from leaf to root.
// It also checks that all certificate lifetimes are valid.
func verifyCertificateChain(certificates CertificateChain) error {
    // Additional check: Verify that all certificates in the cert chain are valid.
    // Note: The *x509.Certificate Verify method in Golang already validates this but for other coding
    // languages it is important to make sure the certificate lifetimes are checked.
    if isCertificateLifetimeValid(certificates.LeafCert) {
        return fmt.Errorf("leaf certificate is not valid")
    }

    if isCertificateLifetimeValid(certificates.IntermediateCert) {
        return fmt.Errorf("intermediate certificate is not valid")
    }
    interPool := x509.NewCertPool()
    interPool.AddCert(certificates.IntermediateCert)

    if isCertificateLifetimeValid(certificates.RootCert) {
        return fmt.Errorf("root certificate is not valid")
    }
    rootPool := x509.NewCertPool()
    rootPool.AddCert(certificates.RootCert)

    _, err := certificates.LeafCert.Verify(x509.VerifyOptions{
        Intermediates: interPool,
        Roots:         rootPool,
        KeyUsages:     []x509.ExtKeyUsage{x509.ExtKeyUsageAny},
    })

    if err != nil {
        return fmt.Errorf("failed to verify certificate chain: %v", err)
    }

    return nil
}
  1. The UWear workload will then check if the workload measurement claims in the token match the attribute conditions specified in the OPA policy. OPA is an open source, general purpose policy engine that unifies policy enforcement across the stack. OPA uses documents, with similar syntax to JSON, to set baseline values that the policy is validated against.
func main() {
  ...

  err = validateClaimsAgainstOPAPolicy(token, ekm)
  if err != nil {
    fmt.Printf("Failed to validate claims against OPA policy: %v\n", err)
  return
  }

  fmt.Println("Validated token and claims. Sending sensitive data")

  ...
}

// validateClaimsAgainstOPAPolicy validates the claims in the JWT token against the OPA policy.
func validateClaimsAgainstOPAPolicy(token jwt.Token, ekm string) error {
        data, err := os.ReadFile("opa_validation_values.json")
        authorized, err := evaluateOPAPolicy(context.Background(), token, ekm, string(data))
        if err != nil {
                fmt.Println("Error evaluating OPA policy:", err)
                return fmt.Errorf("failed to evaluate OPA policy: %w", err)
        }
        if !authorized {
                fmt.Println("Remote TEE's JWT failed policy check.")
                return fmt.Errorf("remote TEE's JWT failed policy check")
        }
        fmt.Println("JWT is authorized.")
        return nil
}


// evaluateOPAPolicy returns boolean indicating if OPA policy is satisfied or not, or error if occurred
func evaluateOPAPolicy(ctx context.Context, token jwt.Token, ekm string, policyData string) (bool, error) {
        var claims jwt.MapClaims
        var ok bool
        if claims, ok = token.Claims.(jwt.MapClaims); !ok {
                return false, fmt.Errorf("failed to get the claims from the JWT")
        }

        module := fmt.Sprintf(opaPolicy, ekm)

        var json map[string]any
        err := util.UnmarshalJSON([]byte(policyData), &json)
        store := inmem.NewFromObject(json)

        // Bind 'allow' to the value of the policy decision
        // Bind 'hw_verified', 'image_verified', 'audience_verified, 'nonce_verified' to their respective policy evaluations
        query, err := rego.New(
                rego.Query(regoQuery),                          // Argument 1 (Query string)
                rego.Store(store),                              // Argument 2 (Data store)
                rego.Module("confidential_space.rego", module), // Argument 3 (Policy module)
        ).PrepareForEval(ctx)

        if err != nil {
                fmt.Printf("Error creating query: %v\n", err)
                return false, err
        }

        fmt.Println("Performing OPA query evaluation...")
        results, err := query.Eval(ctx, rego.EvalInput(claims))

        if err != nil {
                fmt.Printf("Error evaluating OPA policy: %v\n", err)
                return false, err
        } else if len(results) == 0 {
                fmt.Println("Undefined result from evaluating OPA policy")
                return false, err
        } else if result, ok := results[0].Bindings["allow"].(bool); !ok {
                fmt.Printf("Unexpected result type: %v\n", ok)
                fmt.Printf("Result: %+v\n", result)
                return false, err
        }

        fmt.Println("OPA policy evaluation completed.")

        fmt.Println("OPA policy result values:")
        for key, value := range results[0].Bindings {
                fmt.Printf("[ %s ]: %v\n", key, value)
        }
        result := results[0].Bindings["allow"]
        if result == true {
                fmt.Println("Policy check PASSED")
                return true, nil
        }
        fmt.Println("Policy check FAILED")
        return false, nil
}

{
  "allowed_submods_container_image_digest": [
    "sha256:<USLEEP_IMAGE_DIGEST>"
  ],
  "allowed_hwmodel": [
    "GCP_INTEL_TDX",
    "GCP_SHIELDED_VM",
    "GCP_AMD_SEV_ES",
    "GCP_AMD_SEV"
  ],
  "allowed_aud": [
    "uwear"
  ],
  "allowed_issuer": [
    "https://confidentialcomputing.googleapis.com"
  ],
  "allowed_secboot": [
    true
  ],
  "allowed_sw_name": [
    "CONFIDENTIAL_SPACE"
  ]
}

package confidential_space

import rego.v1

default allow := false
default hw_verified := false
default image_digest_verified := false
default audience_verified := false
default nonce_verified := false
default issuer_verified := false
default secboot_verified := false
default sw_name_verified := false

allow if {
  hw_verified
  image_digest_verified
  audience_verified
  nonce_verified
  issuer_verified
  secboot_verified
  sw_name_verified
}

hw_verified if input.hwmodel in data.allowed_hwmodel
image_digest_verified if input.submods.container.image_digest in data.allowed_submods_container_image_digest
audience_verified if input.aud in data.allowed_aud
issuer_verified if input.iss in data.allowed_issuer
secboot_verified if input.secboot in data.allowed_secboot
sw_name_verified if input.swname in data.allowed_sw_name
nonce_verified if {
  input.eat_nonce == "%s"
}
  • Example Rego Query.
regoQuery = "
    allow = data.confidential_space.allow;
    hw_verified = data.confidential_space.hw_verified;
    image__digest_verified = data.confidential_space.image_digest_verified;
    audience_verified = data.confidential_space.audience_verified;
    nonce_verified = data.confidential_space.nonce_verified;
    issuer_verified = data.confidential_space.issuer_verified;
    secboot_verified = data.confidential_space.secboot_verified;
    sw_name_verified = data.confidential_space.sw_name_verified
"

Example code for getting EKM Hash:

func getEKMHashFromConn(c *websocket.Conn) (string, error) {
  conn, ok := c.NetConn().(*tls.Conn)
  if !ok {
    return "", fmt.Errorf("failed to cast NetConn to *tls.Conn")
  }

  state := conn.ConnectionState()
  ekm, err := state.ExportKeyingMaterial("testing_nonce", nil, 32)
  if err != nil {
    return "", fmt.Errorf("failed to get EKM from TLS connection: %w", err)
  }

  sha := sha256.New()
  sha.Write(ekm)
  hash := base64.StdEncoding.EncodeToString(sha.Sum(nil))

  return hash, nil
}
  1. Once all of those checks have been completed and have passed, UWear can confirm that the data will be sent and processed securely. UWear will then respond back with the sensitive PHI over the same TLS session and USleep will be able to use that data to calculate the customer's sleep quality.
func main() {
  ...

  fmt.Println("Validated token and claims. Sending sensitive data")

  data, err := readFile(mySensitiveDataFile)
  if err != nil {
    fmt.Printf("Failed to read data from the file: %v\n", err)
  }

  conn.WriteMessage(websocket.BinaryMessage, data)
  fmt.Println("Sent payload. Closing the connection")
  conn.Close()
  
  ...
}

Steps to create the USleep Workload

  1. Navigate to the scripts directory
cd ~/confidential-space/codelabs/health_data_analysis_codelab/scripts
  1. Run the create_uwear_workload.sh script to create the UWear workload:
  • Creates Artifact Registry ($UWEAR_ARTIFACT_REPOSITORY) owned by UWear where the workload would be published.
  • Builds the uwear/workload.go code and package it in a Docker image. See the Dockerfile configuration for USleep.
  • Publishes the Docker image to the Artifact Registry ($UWEAR_ARTIFACT_REPOSITORY) owned by UWear.
  • Grants the service account $UWEAR_WORKLOAD_SERVICE_ACCOUNT read permission for the Artifact Registry ($UWEAR_ARTIFACT_REPOSITORY).
./create_uwear_workload.sh

7. Run the USleep and UWear Workloads

Run USleep workload

gcloud config set project $USLEEP_PROJECT_ID


gcloud compute instances create \
 --confidential-compute-type=SEV \
 --shielded-secure-boot \
 --maintenance-policy=MIGRATE \
 --min-cpu-platform="AMD Milan" \
 --scopes=cloud-platform --zone=${USLEEP_PROJECT_ZONE} \
 --image-project=confidential-space-images \
 --image-family=confidential-space \
--service-account=${USLEEP_WORKLOAD_SERVICE_ACCOUNT}@${USLEEP_PROJECT_ID}.iam.gserviceaccount.com \
 --metadata ^~^tee-image-reference=${USLEEP_PROJECT_REPOSITORY_REGION}-docker.pkg.dev/${USLEEP_PROJECT_ID}/${USLEEP_ARTIFACT_REPOSITORY}/${USLEEP_WORKLOAD_IMAGE_NAME}:${USLEEP_WORKLOAD_IMAGE_TAG}~tee-restart-policy=Never~tee-container-log-redirect=true usleep

The response should return a STATUS: RUNNING and the EXTERNAL_IP should also be returned similar to this one:

NAME: usleep
ZONE: us-west1-b
MACHINE_TYPE: n2d-standard-2
PREEMPTIBLE:
INTERNAL_IP: 10.138.0.6
EXTERNAL_IP: 34.168.56.10
STATUS: RUNNING

Store the external IP in a variable

export USLEEP_EXTERNAL_IP=<add your external IP>

Verify the USleep Workload ran correctly

To verify the USleep workload is running correctly, navigate to the VM Instances page in the USleep project. Click on the "usleep" instance and press "Serial port 1(console)" under the Logs section. Once the server is up and running, at the bottom of the logs, the logs should show something similar to the following.

2024/09/13 17:00:00 workload task started
#####----- Local IP Address is <YOUR-LOCAL-IP> -----#####
Starting Server..

Run UWear workload

gcloud config set project $UWEAR_PROJECT_ID

gcloud compute instances create \
 --confidential-compute-type=SEV \
 --shielded-secure-boot \
 --maintenance-policy=MIGRATE \
 --min-cpu-platform="AMD Milan" \
 --scopes=cloud-platform --zone=${UWEAR_PROJECT_ZONE} \
 --image-project=confidential-space-images \
 --image-family=confidential-space \
--service-account=${UWEAR_WORKLOAD_SERVICE_ACCOUNT}@${UWEAR_PROJECT_ID}.iam.gserviceaccount.com \
 --metadata ^~^tee-image-reference=${UWEAR_PROJECT_REPOSITORY_REGION}-docker.pkg.dev/${UWEAR_PROJECT_ID}/${UWEAR_ARTIFACT_REPOSITORY}/${UWEAR_WORKLOAD_IMAGE_NAME}:${UWEAR_WORKLOAD_IMAGE_TAG}~tee-restart-policy=Never~tee-container-log-redirect=true~tee-env-remote_ip_addr=$USLEEP_EXTERNAL_IP uwear

Verify the UWear workload ran correctly

To view the logs of the UWear workload navigate to the VM Instances page in the UWear project. Click on the "uwear" instance and press "Serial port 1(console)" under the Logs section.

The log output once the instance has fully started up should look like this

In the UWear project, the serial logs should show something similar to

token: eyJ[...]MrXUg
PKI token validated successfully
Performing OPA query evaluation...
OPA policy evaluation completed.
OPA policy result values:
[ hw_verified ]: true
[ image__digest_verified ]: true
[ audience_verified ]: true
[ nonce_verified ]: true
[ issuer_verified ]: true
[ secboot_verified ]: true
[ sw_name_verified ]: true
[ allow ]: true
Policy check PASSED
JWT is authorized.
Validated token and claims. Sending sensitive data
Sent payload. Closing the connection

If your UWear workload does not look like this, see the notes below for instructions.

View the USleep results

To view the results navigate back to the VM Instances page in the USleep project. Click on the "usleep" instance and press "Serial port 1(console)" under the Logs section. View the results of the workload at the bottom of the logs. They should look similar to the sample below.

Token from the attestation service: eyJhbGci...Ii5A3CJBuDM2o5Q
Received content from other side, {
  "name": "Amy",
  "age": 29,
  "sleep": {
      "light": {
          "minutes": 270
      },
      "deep": {
          "minutes": 135
      },
      "rem": {
          "minutes": 105
      }
  }
}
Sleep quality result: total sleep time is less than 8 hours

The result should be "total sleep time is less than 8 hours".

Congratulations you have successfully created a Confidential Space between UWear and USleep to share sensitive information!

8. (Optional) Run Unauthorized Workload

In the next scenario USleep updates the code and runs a different workload on the sleep data provided by UWear. UWear hasn't agreed to this new workload and hasn't updated their OPA policy to allow the new image digest. We will verify that UWear will not send its sensitive data to the unauthorized workload.

USleep modifies their workload

  1. Set the project to $USLEEP_PROJECT_ID.
gcloud config set project $USLEEP_PROJECT_ID
  1. Delete the USleep VM instance.
gcloud compute instances delete usleep --zone $USLEEP_PROJECT_ZONE
  1. Navigate to the usleep/workload.go directory.
cd ~/confidential-space/codelabs/health_data_analysis_codelab/src/usleep
  1. In the usleep/workload.go file. Update the line "audience": "uwear". In this example, in order to change the image digest we will update the audience to a different value that UWear has not approved. So UWear should reject it for two reasons - unapproved image digest and incorrect audience.
"audience": "anotherCompany.com",
  1. Create new USleep workload
cd ~/confidential-space/codelabs/health_data_analysis_codelab/scripts

./create_usleep_workload.sh
  1. Create the new USleep VM Instance and run the workload
gcloud compute instances create \
 --confidential-compute-type=SEV \
 --shielded-secure-boot \
 --maintenance-policy=MIGRATE \
 --min-cpu-platform="AMD Milan" \
 --scopes=cloud-platform --zone=${USLEEP_PROJECT_ZONE} \
 --image-project=confidential-space-images \
 --image-family=confidential-space \
--service-account=${USLEEP_WORKLOAD_SERVICE_ACCOUNT}@${USLEEP_PROJECT_ID}.iam.gserviceaccount.com \
 --metadata ^~^tee-image-reference=${USLEEP_PROJECT_REPOSITORY_REGION}-docker.pkg.dev/${USLEEP_PROJECT_ID}/${USLEEP_ARTIFACT_REPOSITORY}/${USLEEP_WORKLOAD_IMAGE_NAME}:${USLEEP_WORKLOAD_IMAGE_TAG}~tee-restart-policy=Never~tee-container-log-redirect=true usleep
  1. Extract the new USleep external IP for later use
export USLEEP_EXTERNAL_IP=<add your external IP>

Re-run the workload

  1. Delete the UWear VM instance
gcloud config set project $UWEAR_PROJECT_ID

gcloud compute instances delete uwear --zone $UWEAR_PROJECT_ZONE
  1. Recreate the UWear VM instance using the new External IP
gcloud compute instances create \
 --confidential-compute-type=SEV \
 --shielded-secure-boot \
 --maintenance-policy=MIGRATE \
 --min-cpu-platform="AMD Milan" \
 --scopes=cloud-platform --zone=${UWEAR_PROJECT_ZONE} \
 --image-project=confidential-space-images \
 --image-family=confidential-space \
--service-account=${UWEAR_WORKLOAD_SERVICE_ACCOUNT}@${UWEAR_PROJECT_ID}.iam.gserviceaccount.com \
 --metadata ^~^tee-image-reference=${UWEAR_PROJECT_REPOSITORY_REGION}-docker.pkg.dev/${UWEAR_PROJECT_ID}/${UWEAR_ARTIFACT_REPOSITORY}/${UWEAR_WORKLOAD_IMAGE_NAME}:${UWEAR_WORKLOAD_IMAGE_TAG}~tee-restart-policy=Never~tee-container-log-redirect=true~tee-env-remote_ip_addr=$USLEEP_EXTERNAL_IP uwear
  1. In the UWear serial logs the following message should appear and the USleep VM should not receive any sensitive data
OPA policy result values:
[ nonce_verified ]: true
[ issuer_verified ]: true
[ secboot_verified ]: true
[ sw_name_verified ]: true
[ allow ]: false
[ hw_verified ]: true
[ image__digest_verified ]: false
[ audience_verified ]: false
Policy check FAILED
Remote TEE's JWT failed policy check.
Failed to validate claims against OPA policy: remote TEE's JWT failed policy check

9. Clean Up

The cleanup script can be used to clean up the resources that we have created as part of this codelab. As part of this cleanup, the following resources will be deleted:

  • The UWear service account ($UWEAR_SERVICE_ACCOUNT).
  • The UWear artifact registry ($UWEAR_ARTIFACT_REPOSITORY).
  • The UWear Compute Instance
  • The USleep service account ($USLEEP_SERVICE_ACCOUNT).
  • The USleep artifact registry ($USLEEP_ARTIFACT_REPOSITORY).
  • The USleep Compute Instance
./cleanup.sh

If you are done exploring, please consider deleting your project by following these instructions.

Congratulations

Congratulations, you've successfully completed the codelab!

You learned how to securely share data while retaining its confidentiality using Confidential Space.

What's next?

Check out some of these similar codelabs...

Further reading