Create a Cloud Run worker pool for Pull-based Pub/Sub Subscriptions

1. Introduction

Overview

Worker pools are a Cloud Run resource specifically designed for non-request workloads, such as pull queues. Note that worker pools don't have the following features:

  • No endpoint/URL
  • No requirement for the deployed container to listen for requests at a port
  • No automatic scaling

In this codelab, you will create a worker pool that will continuously retrieve messages from a Pub/Sub pull subscription. You can learn more about Pub/Sub pull subscriptions in the documentation and in this code example.

What you'll learn

  • How to create and deploy to a Cloud Run worker pool
  • How to retrieve messages from a Pull-based Pub/Sub subscription

2. Setup and Requirements

First, set environment variables for this codelab:

export PROJECT_ID=<your_project_id>
export REGION=<your_region>

export WORKER_POOL_NAME=codelab-workers-pubsub
export SERVICE_ACCOUNT=worker-pools-sa
export SERVICE_ACCOUNT_EMAIL=${SERVICE_ACCOUNT}@${PROJECT_ID}.iam.gserviceaccount.com
export TOPIC=pull-pubsub-topic

Next, configure gcloud to use your project ID

gcloud config set project $PROJECT_ID

Before you can start using this codelab, enable the following APIs by running:

gcloud services enable run.googleapis.com \
    cloudbuild.googleapis.com \
    artifactregistry.googleapis.com \
    pubsub.googleapis.com

Create the service account by running:

gcloud iam service-accounts create ${SERVICE_ACCOUNT} \
  --display-name="Service account for worker pool codelab"

Lastly, grant your Cloud Run service account access to PubSub:

gcloud projects add-iam-policy-binding $PROJECT_ID \
    --member serviceAccount:$SERVICE_ACCOUNT_EMAIL \
    --role='roles/pubsub.admin'

3. Create the Cloud Run worker pool

First, create a directory for your worker pool code:

mkdir codelab-worker-pools
cd codelab-worker-pools

Next, create a file called package.json

{
    "name": "codelab-worker-pools",
    "version": "1.0.0",
    "description": "A codelab example of a Cloud Run worker pool retrieving messages from a Pull-based PubSub subscription",
    "main": "index.js",
    "scripts": {
        "start": "node index.js"
    },
    "engines": {
        "node": ">=22.0.0"
    },
    "keywords": [],
    "author": "",
    "license": "ISC",
    "dependencies": {
        "@google-cloud/pubsub": "^5.1.0"
    }
}

Now, create a file called index.js:

'use strict';

const subscriptionNameOrId = 'pull-pubsub-topic';

const { PubSub } = require('@google-cloud/pubsub');

// Creates a Pub/Sub client; cache this for further use.
const pubSubClient = new PubSub();

// References an existing subscription.
const subscription = pubSubClient.subscription(subscriptionNameOrId);

// This function is called when a shutdown signal is received.
const handleShutdown = async (signal) => {
  console.log(`\n${signal} signal caught. Shutting down gracefully...`);

  try {
    // 1. Stop listening for new messages. The `close()` method returns a Promise.
    console.log('Closing Pub/Sub subscription...');
    await subscription.close();
    console.log('Pub/Sub subscription closed.');

    // 2. Add any other cleanup logic here, like closing database connections.

  } catch (err) {
    console.error('Error during graceful shutdown:', err);
  } finally {
    console.log('Worker Pool exited.');
    process.exit(0);
  }
};

// Listen for termination signals.
// SIGINT handles Ctrl+C locally.
// SIGTERM handles signals from services like Cloud Run.
process.on('SIGINT', () => handleShutdown('SIGINT'));
process.on('SIGTERM', () => handleShutdown('SIGTERM'));

// ------------------ Pub/Sub Message Handling ------------------

// Create an event handler to process incoming messages.
const messageHandler = message => {
  console.log(`Received message ${message.id}:`);
  console.log(`\tData: ${message.data}`);
  console.log(`\tAttributes: ${JSON.stringify(message.attributes)}`);

  // Ack the message so it is not sent again.
  message.ack();
};

// Register the message handler and listen for messages.
subscription.on('message', messageHandler);

console.log(
  `Worker started. Listening for messages on "${subscriptionNameOrId}".`
);
console.log('If running locally, press Ctrl+C to quit.');

// The application will now listen for messages indefinitely until a shutdown
// signal is received.

4. Deploy the Worker Pool

Create the Cloud Run worker pool by running the following command:

gcloud beta run worker-pools deploy $WORKER_POOL_NAME --region=$REGION --source .

This command builds the image from source and deploys the job. It will take a few minutes to complete.

5. Publish a message to PubSub

Create a PubSub topic

gcloud pubsub topics create $TOPIC

Create a PubSub pull subscription

gcloud pubsub subscriptions create codelab-subscription --topic=$TOPIC

Run the following command to publish a message to your PubSub topic.

gcloud pubsub topics publish $TOPIC --message "Hello Worker Pools"

Check the logs for your worker pool

gcloud logging read 'resource.type="cloud_run_worker_pool" AND resource.labels.worker_pool_name="'$WORKER_POOL_NAME'" AND resource.labels.location="'$REGION'"' --limit 10

You should see Hello Worker Pools in the logs.

6. Delete the worker pool

Since worker pools run continuously, you should delete the worker pool.

gcloud beta run worker-pools delete $WORKER_POOL_NAME --region $REGION

7. Congratulations!

Congratulations for completing the codelab!

We recommend reviewing the Cloud Run documentation.

What we've covered

  • How to create and deploy to a Cloud Run worker pool
  • How to retrieve messages from a Pull-based Pub/Sub subscription

8. Clean up

To delete the Cloud Run worker pool, go to the Cloud Run Cloud Console at https://console.cloud.google.com/run and delete the codelab-workers-pubsub worker pool.

To delete the entire project, go to Manage Resources, select the project you created in Step 2, and choose Delete. If you delete the project, you'll need to change projects in your Cloud SDK. You can view the list of all available projects by running gcloud projects list.