1. Overview
Confidential Virtual Machines (CVM) are a type of Compute Engine Virtual machines that use hardware-based memory encryption. This helps ensure your data and applications can't be read or modified while in use. In this codelab, you are shown with remote attestation that allows a remote party to verify the CVM nodes. Besides, you will explore cloud logging for further auditing.
As depicted in the figure above, this codelab includes following steps:
- CVM Setup and Remote attestation
- Cloud Logging exploration on the CVM remote attestation
- Secret Release Web Server Setup
The components in the figure above include a go-tpm tool and Google Cloud Attestation:
- go-tpm tool: An open source tool to fetch attestation evidence from the vTPM (Virtual Trusted Platform Module) on the CVM and send it to Google Cloud Attestation for an attestation token.
- Google Cloud Attestation: Verify and validate the attestation evidence received and return an attestation token reflecting the facts about the requester's CVM.
What you'll learn
- How to perform remote attestation through Confidential Computing APIs on CVM
- How to use Cloud Logging to monitor CVM remote attestation
What you'll need
- A Google Cloud Platform Project
- A Browser, such Chrome or Firefox
- Basic knowledge of Google Compute Engine, Confidential VM
2. Setup and Requirements
To enable the necessary APIs, run the following command in cloud console or your local development environment:
gcloud auth login
gcloud services enable \
cloudapis.googleapis.com \
cloudshell.googleapis.com \
confidentialcomputing.googleapis.com \
compute.googleapis.com
3. Setting up CVM and vTPM remote attestation
In this step, you will create a CVM and perform a vTPM remote attestation to retrieve an Attestation Token (OIDC Token) on the CVM.
Go to the cloud console or your local development environment. Create a CVM as following (refer to Create a Confidential VM instance | Google Cloud). Scopes are needed to access Confidential Computing APIs.
gcloud config set project <project-id>
gcloud compute instances create cvm-attestation-codelab \
--machine-type=n2d-standard-2 \
--min-cpu-platform="AMD Milan" \
--zone=us-central1-c \
--confidential-compute \
--image=ubuntu-2204-jammy-v20240228 \
--image-project=ubuntu-os-cloud \
--scopes https://www.googleapis.com/auth/cloud-platform
Authorize CVM default service account to access Confidential Computing APIs (CVMs needs following permissions to fetch Attestation Token from Confidential Computing APIs):
1). Create a role for allowing access to Confidential Computing APIs.
gcloud iam roles create Confidential_Computing_User --project=<project-id> \
--title="CVM User" --description="Grants the ability to generate an attestation token in a CVM." \
--permissions="confidentialcomputing.challenges.create,confidentialcomputing.challenges.verify,confidentialcomputing.locations.get,confidentialcomputing.locations.list" --stage=GA
Replace the following:
project-idis the unique identifier of the project.
2). Add the VM default service account to the role.
gcloud projects add-iam-policy-binding <project-id> \
--member serviceAccount:$(gcloud iam service-accounts list --filter="email ~ compute@developer.gserviceaccount.com$" --format='value(email)'
) \
--role "projects/<project-id>/roles/Confidential_Computing_User"
Replace the following:
project-idis the unique identifier of the project.
3). Connect to CVM and set up the go-tpm tool binary for fetching Attestation Token from Confidential Computing APIs provided by Google Cloud Attestation.
- Connect to the CVM.
gcloud compute ssh --zone us-central1-c cvm-attestation-codelab
- Set up a Go environment:
wget https://go.dev/dl/go1.22.0.linux-amd64.tar.gz sudo tar -C /usr/local -xzf go1.22.0.linux-amd64.tar.gz export PATH=$PATH:/usr/local/go/bin
- Build the go-tpm tool binary. The go-tpm tool binary fetches attestation evidence from the vTPM on the CVM and sends it to Google Cloud Attestation for an attestation token.
git clone https://github.com/google/go-tpm-tools.git --depth 1 cd go-tpm-tools/cmd/gotpm/ go build
- The go-tpm tool command extracts attestation evidence of the CVM from vTPM and sends it to Google Cloud Attestation. Google Cloud Attestation verifies and validates the attestation evidence and returns an Attestation Token. The command creates an attestation_token file which contains your
attestation-token. You will use yourattestation-tokento fetch a secret later. You may decode the attestation token in jwt.io to view the claims.
sudo ./gotpm token > attestation_token
- (Optional) Alternatively to perform remote attestation with go-tpm tool and Google Cloud Attestation, we showcase the commands to fetch vTPM attestation evidence. This way, you could create a service like Google Cloud Attestation to verify and validate on the attestation evidence:
nonce=$(head -c 16 /dev/urandom | xxd -p) sudo ./gotpm attest --nonce $nonce --format textproto --output quote.dat sudo ./gotpm verify debug --nonce $nonce --format textproto --input quote.dat --output vtpm_report
vtpm_report contains the verified eventlog. You can use your preferred editor to look at it. Note that the verify command does not check the attestation key certificate of the quote.
4. Enable Cloud Logging and explore the remote attestation log
You may run the following command in your cvm-attestation-codelab CVM. This time, it logs the activity on cloud logging.
sudo ./gotpm token --cloud-log --audience "https://api.cvm-attestation-codelab.com"
Get cvm-attestation-codelab <instance-id> in your cloud console or local development environment.
gcloud compute instances describe cvm-attestation-codelab --zone us-central1-c --format='value(id)'
To explore Cloud Logging, open the following URL: https://console.cloud.google.com/logs. In the query field, enter the following:
resource.type="gce_instance" resource.labels.zone="us-central1-c" resource.labels.instance_id=<instance-id> log_name="projects/<project-id>/logs/gotpm" severity>=DEFAULT
Replace the following:
project-idis the unique identifier of the project.instance-idis the unique identifier of the instance.
You should be able to find the Attestation Token, its Claims, raw evidence and nonce sent to Google Cloud Attestation.
5. Setup a Secret Release Web Server
In this step, you exit your previous SSH session and set up another VM. On this VM, you set up a secret release web server. The web server validates the received Attestation Token and its claims. If the validations succeed, then it releases the secret to the requester.
1). Go to the cloud console or your local development environment. Create a virtual machine.
gcloud config set project <project-id>
gcloud compute instances create cvm-attestation-codelab-web-server \
--machine-type=n2d-standard-2 \
--zone=us-central1-c \
--image=ubuntu-2204-jammy-v20240228 \
--image-project=ubuntu-os-cloud
Replace the following:
project-idis the unique identifier of the project.
2). SSH to your new VM.
gcloud compute ssh --zone us-central1-c cvm-attestation-codelab-web-server
3). Set up the Go environment.
wget https://go.dev/dl/go1.22.0.linux-amd64.tar.gz sudo tar -C /usr/local -xzf go1.22.0.linux-amd64.tar.gz export PATH=$PATH:/usr/local/go/bin
4). Create the following two files to store the source code of the secret release web server (copy/paste with nano).
main.go
package main
import (
"fmt"
"net/http"
"strings"
"time"
"log"
"github.com/golang-jwt/jwt/v4"
)
const (
theSecret = "This is the super secret information!"
)
func homePage(w http.ResponseWriter, r *http.Request) {
tokenString := r.Header.Get("Authorization")
if tokenString != "" {
tokenString, err := extractToken(tokenString)
if err != nil {
http.Error(w, err.Error(), http.StatusUnauthorized)
}
tokenBytes := []byte(tokenString)
// A method to return a public key from the well-known endpoint
keyFunc := getRSAPublicKeyFromJWKsFile
token, err := decodeAndValidateToken(tokenBytes, keyFunc)
if err != nil {
http.Error(w, "Invalid JWT Token", http.StatusUnauthorized)
}
if ok, err := isValid(token.Claims.(jwt.MapClaims)); ok {
fmt.Fprintln(w, theSecret)
} else {
if err != nil {
http.Error(w, "Error validating JWT claims: "+err.Error(), http.StatusUnauthorized)
} else {
http.Error(w, "Invalid JWT token Claims", http.StatusUnauthorized)
}
}
} else {
http.Error(w, "Authorization token required", http.StatusUnauthorized)
}
}
func extractToken(tokenString string) (string, error) {
if strings.HasPrefix(tokenString, "Bearer ") {
return strings.TrimPrefix(tokenString, "Bearer "), nil
}
return "", fmt.Errorf("invalid token format")
}
func isValid(claims jwt.MapClaims) (bool, error) {
// 1. Evaluating Standard Claims:
subject, ok := claims["sub"].(string)
if !ok {
return false, fmt.Errorf("missing or invalid 'sub' claim")
}
fmt.Println("Subject:", subject)
// e.g. "sub":"https://www.googleapis.com/compute/v1/projects/<project_id>/zones/<project_zone>/instances/<instance_name>"
issuedAt, ok := claims["iat"].(float64)
if !ok {
return false, fmt.Errorf("missing or invalid 'iat' claim")
}
fmt.Println("Issued At:", time.Unix(int64(issuedAt), 0))
// 2. Evaluating Remote Attestation Claims:
hwModel, ok := claims["hwmodel"].(string)
if !ok || hwModel != "GCP_AMD_SEV" {
return false, fmt.Errorf("missing or invalid 'hwModel'")
}
fmt.Println("hwmodel:", hwModel)
swName, ok := claims["swname"].(string)
if !ok || swName != "GCE" {
return false, fmt.Errorf("missing or invalid 'hwModel'")
}
fmt.Println("swname:", swName)
return true, nil
}
func main() {
http.HandleFunc("/", homePage)
fmt.Println("Server listening on :8080")
err := http.ListenAndServe(":8080", nil)
if err != nil {
log.Fatalf("Server failed to start: %v", err)
}
}
helper.go
package main
import (
"crypto/rsa"
"encoding/base64"
"encoding/json"
"errors"
"fmt"
"io"
"math/big"
"net/http"
"github.com/golang-jwt/jwt/v4"
)
const (
socketPath = "/run/container_launcher/teeserver.sock"
expectedIssuer = "https://confidentialcomputing.googleapis.com"
wellKnownPath = "/.well-known/openid-configuration"
)
type jwksFile struct {
Keys []jwk `json:"keys"`
}
type jwk struct {
N string `json:"n"` // "nMMTBwJ7H6Id8zUCZd-L7uoNyz9b7lvoyse9izD9l2rtOhWLWbiG-7pKeYJyHeEpilHP4KdQMfUo8JCwhd-OMW0be_XtEu3jXEFjuq2YnPSPFk326eTfENtUc6qJohyMnfKkcOcY_kTE11jM81-fsqtBKjO_KiSkcmAO4wJJb8pHOjue3JCP09ZANL1uN4TuxbM2ibcyf25ODt3WQn54SRQTV0wn098Y5VDU-dzyeKYBNfL14iP0LiXBRfHd4YtEaGV9SBUuVhXdhx1eF0efztCNNz0GSLS2AEPLQduVuFoUImP4s51YdO9TPeeQ3hI8aGpOdC0syxmZ7LsL0rHE1Q",
E string `json:"e"` // "AQAB" or 65537 as an int
Kid string `json:"kid"` // "1f12fa916c3a0ef585894b4b420ad17dc9d6cdf5",
// Unused fields:
// Alg string `json:"alg"` // "RS256",
// Kty string `json:"kty"` // "RSA",
// Use string `json:"use"` // "sig",
}
type wellKnown struct {
JwksURI string `json:"jwks_uri"` // "https://www.googleapis.com/service_accounts/v1/metadata/jwk/signer@confidentialspace-sign.iam.gserviceaccount.com"
// Unused fields:
// Iss string `json:"issuer"` // "https://confidentialcomputing.googleapis.com"
// Subject_types_supported string `json:"subject_types_supported"` // [ "public" ]
// Response_types_supported string `json:"response_types_supported"` // [ "id_token" ]
// Claims_supported string `json:"claims_supported"` // [ "sub", "aud", "exp", "iat", "iss", "jti", "nbf", "dbgstat", "eat_nonce", "google_service_accounts", "hwmodel", "oemid", "secboot", "submods", "swname", "swversion" ]
// Id_token_signing_alg_values_supported string `json:"id_token_signing_alg_values_supported"` // [ "RS256" ]
// Scopes_supported string `json:"scopes_supported"` // [ "openid" ]
}
func getWellKnownFile() (wellKnown, error) {
httpClient := http.Client{}
resp, err := httpClient.Get(expectedIssuer + wellKnownPath)
if err != nil {
return wellKnown{}, fmt.Errorf("failed to get raw .well-known response: %w", err)
}
wellKnownJSON, err := io.ReadAll(resp.Body)
if err != nil {
return wellKnown{}, fmt.Errorf("failed to read .well-known response: %w", err)
}
wk := wellKnown{}
json.Unmarshal(wellKnownJSON, &wk)
return wk, nil
}
func getJWKFile() (jwksFile, error) {
wk, err := getWellKnownFile()
if err != nil {
return jwksFile{}, fmt.Errorf("failed to get .well-known json: %w", err)
}
// Get JWK URI from .wellknown
uri := wk.JwksURI
fmt.Printf("jwks URI: %v\n", uri)
httpClient := http.Client{}
resp, err := httpClient.Get(uri)
if err != nil {
return jwksFile{}, fmt.Errorf("failed to get raw JWK response: %w", err)
}
jwkbytes, err := io.ReadAll(resp.Body)
if err != nil {
return jwksFile{}, fmt.Errorf("failed to read JWK body: %w", err)
}
file := jwksFile{}
err = json.Unmarshal(jwkbytes, &file)
if err != nil {
return jwksFile{}, fmt.Errorf("failed to unmarshall JWK content: %w", err)
}
return file, nil
}
// N and E are 'base64urlUInt' encoded: https://www.rfc-editor.org/rfc/rfc7518#section-6.3
func base64urlUIntDecode(s string) (*big.Int, error) {
b, err := base64.RawURLEncoding.DecodeString(s)
if err != nil {
return nil, err
}
z := new(big.Int)
z.SetBytes(b)
return z, nil
}
func getRSAPublicKeyFromJWKsFile(t *jwt.Token) (any, error) {
keysfile, err := getJWKFile()
if err != nil {
return nil, fmt.Errorf("failed to fetch the JWK file: %w", err)
}
// Multiple keys are present in this endpoint to allow for key rotation.
// This method finds the key that was used for signing to pass to the validator.
kid := t.Header["kid"]
for _, key := range keysfile.Keys {
if key.Kid != kid {
continue // Select the key used for signing
}
n, err := base64urlUIntDecode(key.N)
if err != nil {
return nil, fmt.Errorf("failed to decode key.N %w", err)
}
e, err := base64urlUIntDecode(key.E)
if err != nil {
return nil, fmt.Errorf("failed to decode key.E %w", err)
}
// The parser expects an rsa.PublicKey: https://github.com/golang-jwt/jwt/blob/main/rsa.go#L53
// or an array of keys. We chose to show passing a single key in this example as its possible
// not all validators accept multiple keys for validation.
return &rsa.PublicKey{
N: n,
E: int(e.Int64()),
}, nil
}
return nil, fmt.Errorf("failed to find key with kid '%v' from well-known endpoint", kid)
}
func decodeAndValidateToken(tokenBytes []byte, keyFunc func(t *jwt.Token) (any, error)) (*jwt.Token, error) {
var err error
fmt.Println("Unmarshalling token and checking its validity...")
token, err := jwt.NewParser().Parse(string(tokenBytes), keyFunc)
fmt.Printf("Token valid: %v\n", token.Valid)
if token.Valid {
return token, nil
}
if ve, ok := err.(*jwt.ValidationError); ok {
if ve.Errors&jwt.ValidationErrorMalformed != 0 {
return nil, fmt.Errorf("token format invalid. Please contact the Confidential Space team for assistance")
}
if ve.Errors&(jwt.ValidationErrorNotValidYet) != 0 {
// If device time is not synchronized with the Attestation Service you may need to account for that here.
return nil, errors.New("token is not active yet")
}
if ve.Errors&(jwt.ValidationErrorExpired) != 0 {
return nil, fmt.Errorf("token is expired")
}
return nil, fmt.Errorf("unknown validation error: %v", err)
}
return nil, fmt.Errorf("couldn't handle this token or couldn't read a validation error: %v", err)
}
5). Run the following commands to build the web server and run it. This starts the secret release webserver at :8080 port.
go mod init google.com/codelab go mod tidy go get github.com/golang-jwt/jwt/v4 go build ./codelab
Troubleshooting: you might see the following warning which can be ignored when run go mod tidy:
go: finding module for package github.com/golang-jwt/jwt/v4 go: downloading github.com/golang-jwt/jwt v3.2.2+incompatible go: downloading github.com/golang-jwt/jwt/v4 v4.5.0 go: found github.com/golang-jwt/jwt/v4 in github.com/golang-jwt/jwt/v4 v4.5.0 go: google.com/codelab/go/pkg/mod/github.com/golang-jwt/jwt@v3.2.2+incompatible: import path "google.com/codelab/go/pkg/mod/github.com/golang-jwt/jwt@v3.2.2+incompatible" should not have @version go: google.com/codelab/go/pkg/mod/github.com/golang-jwt/jwt@v3.2.2+incompatible/cmd/jwt: import path "google.com/codelab/go/pkg/mod/github.com/golang-jwt/jwt@v3.2.2+incompatible/cmd/jwt" should not have @version go: google.com/codelab/go/pkg/mod/github.com/golang-jwt/jwt@v3.2.2+incompatible/request: import path "google.com/codelab/go/pkg/mod/github.com/golang-jwt/jwt@v3.2.2+incompatible/request" should not have @version go: google.com/codelab/go/pkg/mod/github.com/golang-jwt/jwt@v3.2.2+incompatible/test: import path "google.com/codelab/go/pkg/mod/github.com/golang-jwt/jwt@v3.2.2+incompatible/test" should not have @version
6). Now start another cloud console tab or local development environment session and run the following command. This will get you the <cvm-attestation-codelab-web-server-internal-ip>.
gcloud compute instances describe cvm-attestation-codelab-web-server --format='get(networkInterfaces[0].networkIP)' --zone=us-central1-c
7). SSH to your cvm-attestation-codelab VM instance.
gcloud compute ssh --zone "us-central1-c" "cvm-attestation-codelab"
8). The following command substitutes the attestation-token obtained previously (under ~/go-tpm-tools/cmd/gotpm/). This will fetch you the secret held by the secret release web server!
cd ~/go-tpm-tools/cmd/gotpm/ curl http://<cvm-attestation-codelab-web-server-internal-ip>:8080 -H "Authorization: Bearer $(cat ./attestation_token)"
Replace the following:
cvm-attestation-codelab-web-server-internal-ipis the internal ip of the cvm-attestation-codelab-web-server VM instance.
You will see "This is the super secret information!" on your screen.
If you put in an incorrect or expired attestation-token, you will see "curl: (52) Empty reply from server". You will also see "Token valid: false" on your secret release web server log in the cvm-attestation-codelab-web-server VM instance.
6. Cleanup
Run the following commands in cloud console or your local development environment:
# Delete the role binding
gcloud projects remove-iam-policy-binding <project-id> \
--member serviceAccount:$(gcloud iam service-accounts list --filter="email ~ compute@developer.gserviceaccount.com$" --format='value(email)'
) \
--role "projects/<project-id>/roles/Confidential_Computing_User"
# Delete the role
gcloud iam roles delete Confidential_Computing_User --project=<project-id>
# Delete the web server VM instance
gcloud compute instances delete cvm-attestation-codelab-web-server --zone=us-central1-c
# Delete the CVM instance
gcloud compute instances delete cvm-attestation-codelab --zone=us-central1-c
Replace the following:
project-idis the unique identifier of the project.
7. What's next
Learn more about Confidential VMs and Compute Engine.