Autoscale Cloud Run worker pools based on the Pub/Sub queue volume using CREMA

1. Introduction

Overview

This tutorial shows you how to deploy a Cloud Run worker pool (consumer) to process Pub/Sub messages, and automatically scale your consumer instances based on queue depth using Cloud Run External Metrics Autoscaling (CREMA).

What you'll learn

In this codelab, you will:

  • Create a Pub/Sub topic and subscription and push messages to that topic.
  • Deploy a Cloud Run worker pool (consumer) that consumes messages from Pub/Sub.
  • Deploy the CREMA project on GitHub as a Cloud Run service to automatically scale your worker pool based on the number of messages in the Pub/Sub subscription.
  • Test your autoscaling configuration by generating load by running a python script locally.

2. Configure Environment Variables

Since many environment variables are used throughout this codelab, we recommend running

set -u

which will warn you if you try to use an env var that has not been set yet. To undo this setting, run set +u

First, change the following variable to your project id.

export PROJECT_ID=<YOUR_PROJECT_ID>

and then set it as the project for this codelab.

gcloud config set project $PROJECT_ID

Next, set the environment variables used by this codelab.

export REGION=us-central1
export TOPIC_ID=crema-pubsub-topic
export SUBSCRIPTION_ID=crema-pubsub-sub
export CREMA_SA_NAME=crema-service-account
export CONSUMER_SA_NAME=consumer-service-account
export CONSUMER_WORKER_POOL_NAME=worker-pool-consumer
export CREMA_SERVICE_NAME=my-crema-service

Create a directory for this codelab

mkdir crema-pubsub-codelab
cd crema-pubsub-codelab

Enable APIs

gcloud services enable \
        artifactregistry.googleapis.com \
        cloudbuild.googleapis.com \
        run.googleapis.com \
        parametermanager.googleapis.com

Lastly, make sure your gcloud is using the latest Version

gcloud components update

3. Pub/Sub Setup

Create the topic and the pull subscription that your worker pool will process. Bash

Create the topic.

gcloud pubsub topics create $TOPIC_ID

Create the subscription.

gcloud pubsub subscriptions create $SUBSCRIPTION_ID --topic=$TOPIC_ID

4. IAM & Service Accounts

It is recommended to create a service account for each Cloud Run resource. In this codelab, you'll create the following:

  • Consumer SA: Identity for the worker pool processing Pub/Sub messages.
  • CREMA SA: Identity for the CREMA autoscaler service.

Create Service Accounts

Create the worker pool consumer SA:

gcloud iam service-accounts create $CONSUMER_SA_NAME \
  --display-name="PubSub Consumer Service Account"

Create the worker pool CREMA service SA:

gcloud iam service-accounts create $CREMA_SA_NAME \
  --display-name="CREMA Autoscaler Service Account"

Grant Permissions to Consumer SA

Grant permissions to the worker pool consumer SA to pull messages from the subscription.

gcloud pubsub subscriptions add-iam-policy-binding $SUBSCRIPTION_ID \
  --member="serviceAccount:$CONSUMER_SA_NAME@$PROJECT_ID.iam.gserviceaccount.com" \
  --role="roles/pubsub.subscriber"

Grant Permissions to CREMA SA

CREMA needs permissions to read parameters, scale the worker pool, and monitor Pub/Sub metrics.

  1. Access Parameter Manager (Config Reader):
gcloud projects add-iam-policy-binding $PROJECT_ID \
  --member="serviceAccount:$CREMA_SA_NAME@$PROJECT_ID.iam.gserviceaccount.com" \
  --role="roles/parametermanager.parameterViewer"
  1. Scale the Worker Pool (Cloud Run Developer):
gcloud projects add-iam-policy-binding $PROJECT_ID \
  --member="serviceAccount:$CREMA_SA_NAME@$PROJECT_ID.iam.gserviceaccount.com" \
  --role="roles/run.developer"
  1. Monitor Pub/Sub:

Grant the monitoring viewer role.

gcloud projects add-iam-policy-binding $PROJECT_ID \
--member="serviceAccount:$CREMA_SA_NAME@$PROJECT_ID.iam.gserviceaccount.com" \
--role="roles/monitoring.viewer"

Add a policy to the subscription for the CREMA service SA to view it

gcloud pubsub subscriptions add-iam-policy-binding $SUBSCRIPTION_ID \
  --member="serviceAccount:$CREMA_SA_NAME@$PROJECT_ID.iam.gserviceaccount.com" \
  --role="roles/pubsub.viewer"

The CREMA SA also needs Service Account User, which is required to change instance counts:

gcloud iam service-accounts add-iam-policy-binding \
    $CONSUMER_SA_NAME@$PROJECT_ID.iam.gserviceaccount.com \
    --member="serviceAccount:$CREMA_SA_NAME@$PROJECT_ID.iam.gserviceaccount.com" \
    --role="roles/iam.serviceAccountUser"

5. Verify SA permissions

Before proceeding with the codelab, verify that the CREMA service SA has the correct project-level roles.

gcloud projects get-iam-policy $PROJECT_ID \
  --flatten="bindings[].members" \
  --format="table(bindings.role)" \
  --filter="bindings.members:serviceAccount:$CREMA_SA_NAME@$PROJECT_ID.iam.gserviceaccount.com"

Should result in the following:

roles/monitoring.viewer
roles/parametermanager.parameterViewer
roles/run.developer

Verify the Pub/Sub subscription has a policy that allows the CREMA service SA to view it.

gcloud pubsub subscriptions get-iam-policy $SUBSCRIPTION_ID \
  --flatten="bindings[].members" \
  --filter="bindings.members:serviceAccount:$CREMA_SA_NAME@$PROJECT_ID.iam.gserviceaccount.com" \
  --format="table(bindings.role)"

Should result in

roles/pubsub.viewer

and verify the CREMA SA has the Service Account User role

gcloud iam service-accounts get-iam-policy \
  $CONSUMER_SA_NAME@$PROJECT_ID.iam.gserviceaccount.com \
  --flatten="bindings[].members" \
  --filter="bindings.members:serviceAccount:$CREMA_SA_NAME@$PROJECT_ID.iam.gserviceaccount.com"

Should result in the following

bindings:
  members: serviceAccount:crema-service-account@<PROJECT_ID>.iam.gserviceaccount.com
  role: roles/iam.serviceAccountUser

And the Worker Pool Consumer SA has the Pub/Sub subscriber role

gcloud pubsub subscriptions get-iam-policy $SUBSCRIPTION_ID \
  --flatten="bindings[].members" \
  --filter="bindings.members:serviceAccount:$CONSUMER_SA_NAME@$PROJECT_ID.iam.gserviceaccount.com" \
  --format="table(bindings.role)"

Should result in

ROLE
roles/pubsub.subscriber

6. Build and Deploy the Consumer Worker Pool

Create a directory for your consumer code and enter it.

mkdir consumer
cd consumer
  1. Create a consumer.py file
import os
import time
from google.cloud import pubsub_v1
from concurrent.futures import TimeoutError

# Configuration
PROJECT_ID = os.environ.get('PROJECT_ID')
SUBSCRIPTION_ID = os.environ.get('SUBSCRIPTION_ID')

subscription_path = f"projects/{PROJECT_ID}/subscriptions/{SUBSCRIPTION_ID}"

print(f"Worker Pool instance starting. Watching {subscription_path}...")

subscriber = pubsub_v1.SubscriberClient()

def callback(message):
    try:
        data = message.data.decode("utf-8")
        print(f"Processing job: {data}")
        time.sleep(5)  # Simulate work
        print(f"Done {data}")
        message.ack()
    except Exception as e:
        print(f"Error processing message: {e}")
        message.nack()

streaming_pull_future = subscriber.subscribe(subscription_path, callback=callback)
print(f"Listening for messages on {subscription_path}...")

# Wrap subscriber in a 'with' block to automatically call close() when done.
with subscriber:
    try:
        # When `timeout` is not set, result() will block indefinitely,
        # unless an exception is encountered first.
        streaming_pull_future.result()
    except TimeoutError:
        streaming_pull_future.cancel()  # Trigger the shutdown.
        streaming_pull_future.result()  # Block until the shutdown is complete.
    except Exception as e:
        print(f"Streaming pull failed: {e}")
  1. Create a Dockerfile
FROM python:3.12-slim
RUN pip install google-cloud-pubsub
COPY consumer.py .
CMD ["python", "-u", "consumer.py"]
  1. Deploy Consumer Worker Pool

This codelab recommends deploying the worker pool with 0 instances to start, so you can watch CREMA scale the worker pool when it detects the Pub/Sub messages in the subscription.

gcloud beta run worker-pools deploy $CONSUMER_WORKER_POOL_NAME \
  --source . \
  --region $REGION \
  --service-account="$CONSUMER_SA_NAME@$PROJECT_ID.iam.gserviceaccount.com" \
  --instances=0 \
  --set-env-vars PROJECT_ID=$PROJECT_ID,SUBSCRIPTION_ID=$SUBSCRIPTION_ID

7. Configure CREMA

  1. Navigate back to the root directory for your project.
cd ..
  1. Create the Config File Create a file named crema-config.yaml
apiVersion: crema/v1
kind: CremaConfig
spec:
  pollingInterval: 30
  triggerAuthentications:
    - metadata:
        name: adc-trigger-auth
      spec:
        podIdentity:
          provider: gcp
  scaledObjects:
    - spec:
        scaleTargetRef:
          name: projects/PROJECT_ID_PLACEHOLDER/locations/REGION_PLACEHOLDER/workerpools/CONSUMER_WORKER_POOL_NAME_PLACEHOLDER
        triggers:
          - type: gcp-pubsub
            metadata:
              subscriptionName: "SUBSCRIPTION_ID_PLACEHOLDER"
              # Target number of undelivered messages per worker instance
              value: "10"
              mode: "SubscriptionSize"
            authenticationRef:
              name: adc-trigger-auth
  1. Substitute Variables
sed -i "s/PROJECT_ID_PLACEHOLDER/$PROJECT_ID/g" crema-config.yaml
sed -i "s/REGION_PLACEHOLDER/$REGION/g" crema-config.yaml
sed -i "s/CONSUMER_WORKER_POOL_NAME_PLACEHOLDER/$CONSUMER_WORKER_POOL_NAME/g" crema-config.yaml
sed -i "s/SUBSCRIPTION_ID_PLACEHOLDER/$SUBSCRIPTION_ID/g" crema-config.yaml
  1. Verify your crema-config.yaml is correct
if grep -q "_PLACEHOLDER" crema-config.yaml; then
  echo "❌ ERROR: Validations failed. '_PLACEHOLDER' was found in crema-config.yaml."
  echo "Please check your environment variables and run the 'sed' commands again."
else
  echo "✅ Config check passed: No placeholders found."
fi
  1. Upload to Parameter Manager

Set additional environment variables for Parameter Manager

export PARAMETER_ID=crema-config
export PARAMETER_REGION=global
export PARAMETER_VERSION=1

Create the Parameter resource

gcloud parametermanager parameters create $PARAMETER_ID \
  --location=$PARAMETER_REGION \
  --parameter-format=YAML

Create Parameter Version 1

gcloud parametermanager parameters versions create $PARAMETER_VERSION \
  --parameter=crema-config \
  --project=$PROJECT_ID \
  --location=$PARAMETER_REGION \
  --payload-data-from-file=crema-config.yaml

Verify the parameter is successfully added

gcloud parametermanager parameters versions list \
  --parameter=$PARAMETER_ID \
  --location=$PARAMETER_REGION

You should see something like

projects/<YOUR_PROJECT_ID>/locations/global/parameters/crema-config/versions/1

8. Deploy CREMA Service

In this section, you'll deploy the CREMA autoscaler service. You'll use the image that is publicly available.

  1. Set environment variables needed for CREMA
CREMA_CONFIG_PARAM_VERSION=projects/$PROJECT_ID/locations/$PARAMETER_REGION/parameters/$PARAMETER_ID/versions/$PARAMETER_VERSION
  1. Verify the version name path
echo $CREMA_CONFIG_PARAM_VERSION

It should look like

projects/<YOUR_PROJECT>/locations/global/parameters/crema-config/versions/1
  1. Set the env var for the CREMA image
IMAGE=us-central1-docker.pkg.dev/cloud-run-oss-images/crema-v1/autoscaler:1.0
  1. and deploy the CREMA service

Note that the base image is required.

gcloud beta run deploy $CREMA_SERVICE_NAME \
  --image=$IMAGE \
  --region=${REGION} \
  --service-account="${CREMA_SA_NAME}@${PROJECT_ID}.iam.gserviceaccount.com" \
  --no-allow-unauthenticated \
  --no-cpu-throttling \
  --labels=created-by=crema \
  --base-image=us-central1-docker.pkg.dev/serverless-runtimes/google-24/runtimes/java25 \
  --set-env-vars="CREMA_CONFIG=${CREMA_CONFIG_PARAM_VERSION},OUTPUT_SCALER_METRICS=True,ENABLE_CLOUD_LOGGING=True"

9. Load Testing

  1. Create a script that will publish messages to the Pub/Sub topic
touch load-pubsub.sh
  1. Add the following code to the load-pubsub.sh file
#!/bin/bash
TOPIC_ID=${TOPIC_ID} 
PROJECT_ID=${PROJECT_ID}
NUM_MESSAGES=100

echo "Publishing $NUM_MESSAGES messages to topic $TOPIC_ID..."

for i in $(seq 1 $NUM_MESSAGES); do
  gcloud pubsub topics publish $TOPIC_ID --message="job-$i" --project=$PROJECT_ID &
  if (( $i % 10 == 0 )); then
    wait
    echo "Published $i messages..."
  fi
done
wait
echo "Done. All messages published."
  1. Run Load Test
chmod +x load-pubsub.sh
./load-pubsub.sh
  1. Monitor Scaling Wait 3-4 minutes. View CREMA logs to see it recommending instances based on the new authenticationRef configuration.
gcloud logging read "resource.type=cloud_run_revision AND resource.labels.service_name=$CREMA_SERVICE_NAME AND textPayload:SCALER" \
  --limit=20 \
  --format="value(textPayload)" \
  --freshness=5m
  1. Monitor Processing View the Consumer logs to see it spinning up.
gcloud beta run worker-pools logs tail $CONSUMER_WORKER_POOL_NAME --region=$REGION

You should see logs like

Done job-100

10. Troubleshooting

First, you'll want to determine whether the issue is with the CREMA service configuration or with the PubSub consumer configuration.

Set the PubSub consumer autoscaler to 1, instead of 0. If it immediately begins to process pubsub messages, it's an issue with the CREMA. If it does not process the pubsub messages, there is an issue with the pubsub consumer.

11. Congratulations!

Congratulations for completing the codelab!

We recommend reviewing the Cloud Run documentation.

What we've covered

  • How to create a Pub/Sub topic and subscription and push messages to that topic.
  • How to deploy a Cloud Run worker pool (consumer) that consumes messages from Pub/Sub.
  • How to deploy the CREMA project on GitHub as a Cloud Run service to automatically scale your worker pool based on the number of messages in the Pub/Sub subscription.
  • How to test your autoscaling configuration by generating load by running a python script locally.

12. Clean up

To avoid incurring charges to your Google Cloud account for the resources used in this tutorial, you can either delete the resources you created in this codelab or you can delete the entire project.

Delete resources used in this codelab

  1. Delete the Cloud Run CREMA service
gcloud run services delete $CREMA_SERVICE_NAME --region=$REGION --quiet
  1. Delete the Cloud Run worker pool consumer
gcloud beta run worker-pools delete $CONSUMER_WORKER_POOL_NAME --region=$REGION --quiet
  1. Delete the Pub/Sub subscription and topic
gcloud pubsub subscriptions delete $SUBSCRIPTION_ID --quiet
gcloud pubsub topics delete $TOPIC_ID --quiet
  1. Delete the Parameter Manager config

Delete the version inside the parameter

gcloud parametermanager parameters versions delete $PARAMETER_VERSION \
  --parameter=$PARAMETER_ID \
  --location=$PARAMETER_REGION \
  --quiet

Now delete the empty parameter

gcloud parametermanager parameters delete $PARAMETER_ID \
  --location=$PARAMETER_REGION \
  --quiet
  1. Delete the Service Accounts
gcloud iam service-accounts delete "$CREMA_SA_NAME@$PROJECT_ID.iam.gserviceaccount.com" --quiet
gcloud iam service-accounts delete "$CONSUMER_SA_NAME@$PROJECT_ID.iam.gserviceaccount.com" --quiet

Or delete the entire project

To delete the entire project, go to Manage Resources, select the project you created in Step 2, and choose Delete. If you delete the project, you'll need to change projects in your Cloud SDK. You can view the list of all available projects by running gcloud projects list.