Securing a Workplace Safety Detection System

1. Introduction

Within this Codelab, you will construct a basic demonstration application that illustrates how to integrate various Google Cloud services for a service located in a project with compliance restrictions. This project uses the following security capabilities:

• Data Boundaries to help address controls for regulated environments
• Customer Managed Encryption Keys ( CMEK) for safeguarding data-at-rest;
• GKE Shielded Nodes and GKE Confidential Nodes to help secure data during processing;
• Workload Identity for GKE to manage access to Vertex AI, which utilizes Gemini for image processing tasks.
• Generative AI on Vertex A to manage detection of hard hats from video stream

This Codelab is for developers of all levels, including beginners. You will use the command-line interface in Google Cloud Shell and Python code. You don't need to be a Python expert, but a basic understanding of how to read code will help you understand the concepts.

Note: This is a simplified proof-of-concept, not a production application. Apply additional precautions such as authenticated and secured external access to this service in a real-world scenario.

2. Before you begin

Project setup

../shared/_project-setup.md

Start Cloud Shell

Cloud Shell is a command-line environment running in Google Cloud that comes preloaded with necessary tools.

1. Click Activate Cloud Shell at the top of the Google Cloud console:

2. Once connected to Cloud Shell, run this command to verify your authentication in Cloud Shell:

gcloud auth list

3. Run the following command to confirm that your project is configured for use with gcloud:

gcloud config get project

4. Confirm the project is as expected, and then run the command below to set your project id:

export PROJECT_ID=$(gcloud config get project)

Required IAM permissions

The account you use for this codelab must have the following IAM roles. These permissions are necessary to create the required Google Cloud resources (projects, folders, GKE clusters, KMS keys, service accounts, etc.) and configure the Assured Workload.

At the Organization Level:

• roles/assuredworkloads.admin (Assured Workloads Administrator): To create and manage the Assured Workloads resource itself, ensuring compliance configuration.

At the Billing Account Level (on your Billing Account):

• roles/billing.accountUser (Billing Account User): To link the specified billing account.

Configuring Codelab variables

To create the needed infrastructure, you need to provide the following environment variables:

# The ID of the Billing Account in the format (XXXXXX-XXXXXX-XXXXXX).
# This value will be used to attach in the projects created using Assured Workloads
export BILLING_ACCOUNT=

# The ID of a Google Cloud Platform organization
# Run `gcloud organizations list` to check all your available organizations
export GCP_ORGANIZATION=

# The numeric ID of a folder where the Assured Workloads will create the resources.
export FOLDER_ID=

# Region where the application will be deployed.
# Since you are using Assured Workloads, you MUST use one of the valid locations as described here: <https://docs.cloud.google.com/assured-workloads/docs/locations>
export REGION="us-central1"

# The ID of an existing Google Cloud project to be used for API quota and billing purposes.
# This project will only be used to enable the Assured Workloads API and create an Assured Workload.
export QUOTA_PROJECT_ID=

# Random suffix used to avoid naming collisions when creating the GCP projects.
export RANDOM_SUFFIX=$(cat /dev/urandom | tr -dc 'a-z0-9' | head -c 5 ; echo)

# The ID of the projects that will be created using Assured Workloads.
# You can modify this value if you want a custom id.
export PROJECT_ID="il5-gemini-vision-aw-${RANDOM_SUFFIX}"
export KMS_PROJECT_ID="il5-gemini-vision-kms-${RANDOM_SUFFIX}"

3. Creating the Assured Workloads foundation

You can now lay the foundation for your application in a regulated environment. Use Assured Workloads to help enforce compliance requirements by creating a controlled environment for your resources.

Configure your quota project

Enable the Assured Workloads API in your quota project. This API is required for creating and managing Assured Workloads.

gcloud services enable assuredworkloads.googleapis.com \
  --project="${QUOTA_PROJECT_ID}"

Create the Assured Workload environment

The following command creates a secure ‘landing zone' for your demo. It creates two new Google Cloud projects under your specified folder and billing account.

• One project hosts your GKE cluster and application.
• The other project manages your Customer-Managed Encryption Keys (CMEK).

It automatically applies the specified IL5 compliance controls to both projects from the moment they are created.

Run the following to create the workload environment.

export ASSURED_WORKLOAD_ID=$(gcloud assured workloads create \
    --project="${QUOTA_PROJECT_ID}" \
    --display-name="DoD IL5 Gemini Vision Demo" \
    --compliance-regime="IL5" \
    --billing-account="billingAccounts/${BILLING_ACCOUNT}" \
    --location="${REGION}" \
    --organization="${GCP_ORGANIZATION}" \
    --provisioned-resources-parent="folders/${FOLDER_ID}" \
    --resource-settings="consumer-project-id=${PROJECT_ID},consumer-project-name=DoD IL5 Workloads,encryption-keys-project-id=${KMS_PROJECT_ID},encryption-keys-project-name=DoD IL5 KMS" \
    --labels="codelab=gemini-vision-demo" \
    --format="value(name)")

echo "Assured Workload created: ${ASSURED_WORKLOAD_ID}"

export WORKLOAD_FOLDER_ID=$(gcloud assured workloads describe ${ASSURED_WORKLOAD_ID} \
    --location="${REGION}" \
    --project="${QUOTA_PROJECT_ID}" \
    --format="json" | grep -B 1 "CONSUMER_FOLDER" | grep -oE "[0-9]{10,}")

echo "Assured Workload folder created: ${WORKLOAD_FOLDER_ID}"

gcloud projects create "${PROJECT_ID}" \
    --folder="${WORKLOAD_FOLDER_ID}" \
    --name="DoD IL5 Workloads"

gcloud billing projects link "${PROJECT_ID}" \
    --billing-account="${BILLING_ACCOUNT}"

export PROJECT_NUMBER=$(gcloud projects describe "${PROJECT_ID}" --format="value(projectNumber)")

Enabling required Google Cloud APIs

Before building, enable the required APIs for the needed services.

The command below activates all the necessary services for this Codelab within your primary workload project.

gcloud services enable \
    aiplatform.googleapis.com \
    artifactregistry.googleapis.com \
    cloudkms.googleapis.com \
    compute.googleapis.com \
    container.googleapis.com \
    iam.googleapis.com \
    logging.googleapis.com \
    monitoring.googleapis.com \
    --project="${PROJECT_ID}"

4. GKE infrastructure setup

Build the core infrastructure that supports your GKE cluster. This involves setting up a dedicated network for your cluster and configuring your own encryption keys to protect the data on your nodes.

Configuring the VPC network

Create a custom Virtual Private Cloud (VPC) and subnetwork. This approach gives you full control over the IP address range and ensures your cluster is isolated as intended.

Run the following commands to create the VPC network and a subnetwork within your specified region.

export GKE_NETWORK_NAME="il5-gke-network"
export GKE_SUBNETWORK_NAME="il5-gke-subnet"

gcloud compute networks create "${GKE_NETWORK_NAME}" \
    --description="VPC network for GKE cluster in DoD IL5 Assured Workload" \
    --subnet-mode="custom" \
    --project="${PROJECT_ID}"

gcloud compute networks subnets create "${GKE_SUBNETWORK_NAME}" \
    --network="${GKE_NETWORK_NAME}" \
    --range="10.10.0.0/20" \
    --region="${REGION}" \
    --description="Subnet for GKE cluster nodes in DoD IL5 Assured Workload" \
    --project="${PROJECT_ID}"

Configuring encryption with Cloud KMS

To meet strict compliance requirements for data-at-rest, use Customer-Managed Encryption Keys (CMEK). This gives you direct control over the keys used to encrypt the boot disks of your GKE nodes.

export KMS_KEYRING_NAME="il5_gke_key_ring"
export KMS_KEY_NAME="il5_gke_key"

gcloud kms keyrings create "${KMS_KEYRING_NAME}" \
    --location="$REGION" \
    --project="${KMS_PROJECT_ID}"

gcloud kms keys create "${KMS_KEY_NAME}" \
    --keyring="${KMS_KEYRING_NAME}" \
    --location="${REGION}" \
    --purpose="encryption" \
    --project="${KMS_PROJECT_ID}"

Because the encryption key exists in a separate KMS project from the service that needs to use it, the Google Compute Engine service agent in the workload project needs access to the key.

You must explicitly grant the GCE service agent from the workload project the permission to use this key. The command below adds an IAM policy binding to the key, giving the necessary role to the service agent.

gcloud kms keys add-iam-policy-binding "${KMS_KEY_NAME}" \
    --location="${REGION}" \
    --keyring="${KMS_KEYRING_NAME}" \
    --member="serviceAccount:service-${PROJECT_NUMBER}@compute-system.iam.gserviceaccount.com" \
    --role="roles/cloudkms.cryptoKeyEncrypterDecrypter" \
    --project="${KMS_PROJECT_ID}"

Configuring a GKE node service account

export GKE_NODE_SA=gke-node-sa

gcloud iam service-accounts create "${GKE_NODE_SA}" \
    --display-name="GKE Node Service Account" \
    --project="${PROJECT_ID}"

gcloud projects add-iam-policy-binding "${PROJECT_ID}" \
    --member="serviceAccount:${GKE_NODE_SA}@${PROJECT_ID}.iam.gserviceaccount.com" \
    --role="roles/logging.logWriter"

gcloud projects add-iam-policy-binding "${PROJECT_ID}" \
    --member="serviceAccount:${GKE_NODE_SA}@${PROJECT_ID}.iam.gserviceaccount.com" \
    --role="roles/monitoring.metricWriter"

5. GKE cluster creation and configuration

Now, you can create the GKE cluster. The following command provisions a GKE cluster with several security features enabled.

This takes several minutes to complete as Google Cloud provisions the nodes and control plane.

export GKE_CLUSTER=ppe-app

gcloud beta container clusters create "${GKE_CLUSTER}" \
    --project="$PROJECT_ID" \
    --region="$REGION" \
    --service-account="${GKE_NODE_SA}@${PROJECT_ID}.iam.gserviceaccount.com" \
    --release-channel="regular" \
    --machine-type="n2d-standard-4" \
    --image-type="COS_CONTAINERD" \
    --disk-type="pd-ssd" \
    --disk-size="50" \
    --boot-disk-kms-key="projects/${KMS_PROJECT_ID}/locations/${REGION}/keyRings/${KMS_KEYRING_NAME}/cryptoKeys/${KMS_KEY_NAME}" \
    --metadata disable-legacy-endpoints=true \
    --num-nodes="1" \
    --network="projects/${PROJECT_ID}/global/networks/${GKE_NETWORK_NAME}" \
    --subnetwork="projects/${PROJECT_ID}/regions/${REGION}/subnetworks/${GKE_SUBNETWORK_NAME}" \
    --security-posture="standard" \
    --workload-vulnerability-scanning="disabled" \
    --workload-pool="${PROJECT_ID}.svc.id.goog" \
    --workload-metadata=GKE_METADATA \
    --addons="HorizontalPodAutoscaling,HttpLoadBalancing,NodeLocalDNS,GcePersistentDiskCsiDriver" \
    --max-surge-upgrade=1 \
    --max-unavailable-upgrade=0 \
    --binauthz-evaluation-mode="DISABLED" \
    --no-enable-basic-auth \
    --enable-autoupgrade \
    --enable-autorepair \
    --enable-confidential-nodes \
    --confidential-node-type=sev \
    --enable-ip-access \
    --enable-ip-alias \
    --enable-managed-prometheus \
    --enable-dns-access \
    --enable-shielded-nodes \
    --shielded-integrity-monitoring \
    --shielded-secure-boot

Connecting to the new cluster

To interact with your new cluster, configure your local kubectl command-line tool.

This command fetches the cluster's credentials and endpoint, and automatically configures your local kubeconfig file. After running this, any kubectl commands you run will be directed at your new GKE cluster.

gcloud container clusters get-credentials "${GKE_CLUSTER}" \
    --region="${REGION}" \
    --project="${PROJECT_ID}" \
    --dns-endpoint

6. Application identity configuration

To connect a Kubernetes service account to a Google Cloud IAM service account, configure Workload Identity.

Creating service accounts

Create a dedicated Kubernetes Service Account (KSA) inside your cluster.

export GKE_NAMESPACE=default
export GKE_SA=ppe-sa

kubectl create sa "${GKE_SA}" --namespace="${GKE_NAMESPACE}"

Next, your application needs an identity within Google Cloud. Create a Google Cloud IAM service account for the application. After creating the service account, grant it the necessary roles.

# Create GCP service account
gcloud iam service-accounts create "${GKE_SA}" \
    --project="${PROJECT_ID}"
# Grant necessary roles
gcloud projects add-iam-policy-binding "${PROJECT_ID}" \
    --member="serviceAccount:${GKE_SA}@${PROJECT_ID}.iam.gserviceaccount.com" \
    --role="roles/aiplatform.user"

Allowing the Kubernetes Service Account to impersonate the IAM one

With both service accounts created, the final step is to create the link between them. This is a two-part process. First, add an IAM policy to your Google Cloud Service Account.

# Allow the Kubernetes service account to act as GCP service account by using Workload Identity
gcloud iam service-accounts add-iam-policy-binding "${GKE_SA}@${PROJECT_ID}.iam.gserviceaccount.com" \
    --project="${PROJECT_ID}" \
    --role="roles/iam.workloadIdentityUser" \
    --member="serviceAccount:${PROJECT_ID}.svc.id.goog[${GKE_NAMESPACE}/${GKE_SA}]"

Second, annotate your Kubernetes Service Account.

kubectl annotate --namespace="${GKE_NAMESPACE}" serviceaccount "$GKE_SA" \
    iam.gke.io/gcp-service-account="${GKE_SA}@${PROJECT_ID}.iam.gserviceaccount.com"

With this complete, any pod running in your cluster with the Kubernetes service account can now access the Vertex API.

Note: It's possible to eliminate service account impersonation to simplify this configuration. See here for details and limitations.

7. Application build and deployment

It's time to package your application, store it, and deploy it to your GKE cluster.

Creating the Artifact Registry repository

Before you can run the app, package it as a Docker container and store it inside an Artifact Registry repository. Use the command below to create this repository.

export REPOSITORY_ID=ppe-repo

gcloud artifacts repositories create "${REPOSITORY_ID}" \
  --repository-format=docker \
  --location="${REGION}" \
  --project="${PROJECT_ID}" \
  --description="Regional Docker repo for PPE App"

You must explicitly grant the service account used by the GKE nodes the ability to read from your new repository.

gcloud artifacts repositories add-iam-policy-binding "${REPOSITORY_ID}" \
    --location="${REGION}" \
    --role="roles/artifactregistry.reader" \
    --project="${PROJECT_ID}" \
    --member="serviceAccount:${GKE_NODE_SA}@${PROJECT_ID}.iam.gserviceaccount.com"

Building and pushing the Docker image

Build and push the Docker image to the repository. First, clone the source code and build the container image. Then, tag it with its full Artifact Registry path, and push it to the repository you previously created.

git clone https://github.com/GoogleCloudPlatform/next-26-sessions.git

cd BRK3-034-workplace-safety

cd ppe

export IMAGE_TAG="${REGION}-docker.pkg.dev/${PROJECT_ID}/${REPOSITORY_ID}/ppe-app:v15"
docker build -t "${IMAGE_TAG}" .
docker push "${IMAGE_TAG}"

Deploying to GKE

With the container image available in Artifact Registry, the final step is to instruct GKE to pull and run it. This is achieved by defining your application's resources in a Kubernetes manifest file and applying it to the cluster. This command creates the Deployment and Service objects defined in the file.

This command deploys the analyze end-point for access outside the GKE cluster. Ensure this endpoint is not accessible from untrusted networks such as the default VPC. Production applications should consider employing additional safeguards such as Model Armor.

export GKE_DEPLOY=ppe-detector

cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ${GKE_DEPLOY}
spec:
  replicas: 2
  selector:
    matchLabels:
      app: ${GKE_DEPLOY}
  template:
    metadata:
      labels:
        app: ${GKE_DEPLOY}
    spec:
      serviceAccountName: ${GKE_SA}
      containers:
      - name: ${GKE_DEPLOY}
        image: ${IMAGE_TAG}
        env:
        - name: PROJECT_ID
          value: ${PROJECT_ID}
        - name: LOCATION
          value: ${REGION}
        ports:
        - containerPort: 8080
        resources:
          requests:
            cpu: "250m"
            memory: "512Mi"
          limits:
            cpu: "500m"
            memory: "1Gi"
---
apiVersion: v1
kind: Service
metadata:
  name: ${GKE_DEPLOY}
spec:
  type: LoadBalancer
  selector:
    app: ${GKE_DEPLOY}
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8080
EOF

8. Testing the application

The final step is to access and test the demo application. This involves retrieving the external IP address assigned to the service and interacting with it via a simple frontend.

Retrieving the service's external IP

Get the external IP of the exposed service (takes ~30 seconds to provision)

export IP_ADDRESS=$(kubectl get service "${GKE_DEPLOY}" -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
echo $IP_ADDRESS

Running the frontend locally

For this demonstration, the frontend is a simple HTML/JavaScript page that is not part of the GKE deployment. It's designed to run on your local machine.

The front-end should be served from GCP in a production application. On local machine:

# Update the index.html file with the server IP address
cd frontend

# For Linux
sed -i "s#\(const BACKEND_URL = \"http://\)[^/]\+\(\/analyze\";\)#\1${IP_ADDRESS}\2#g" "index.html"

# For MacOS
#sed -i '' "s#\(const BACKEND_URL = #\"http://\)[^/]*\(\/analyze\";\)#\1${IP_ADDRESS}\2#g" "index.html"

python3 -m http.server 8001

Open http://localhost:8001/index.html in Chrome.

9. Clean Up

To avoid ongoing charges, delete the resources created in this codelab.

Destroying GKE cluster

To delete the entire application you just need to delete the GKE cluster. To do that, run:

gcloud container clusters delete "${GKE_CLUSTER}" \
    --region="$REGION" \
    --project="${PROJECT_ID}"

Deleting the Assured Workloads

Run the following commands to delete all Assured Workload related resources.

# Workload project deletion
gcloud billing projects unlink "${PROJECT_ID}"
gcloud projects delete "${PROJECT_ID}"

# KMS project deletion
gcloud billing projects unlink "${KMS_PROJECT_ID}"
gcloud projects delete "${KMS_PROJECT_ID}"

# Assured Workload folder deletion
gcloud resource-manager folders delete ${WORKLOAD_FOLDER_ID} --quiet

# Assured Workload deletion
gcloud assured workloads delete "${ASSURED_WORKLOAD_ID}" \
    --location="${REGION}" \
    --organization="${GCP_ORGANIZATION}" \
    --project="${QUOTA_PROJECT_ID}"

10. Congratulations

Mission Complete! You have successfully built a workplace safety detection system for regulated industries that uses Gemini to detect hard hats.

What you accomplished:

• Data protection and privacy: You provisioned a Confidential GKE node with CMEK
• Data Boundaries: You enabled platform compliance controls for a regulated environment

Reference docs

• Encrypt workload data in-use with Confidential GKE Nodes
• Use custom organization policies
• Overview of Data Boundaries via Assured Workloads