Cloud Run jobs

1. Introduction

Last Updated: 2022-05-16

Cloud Run jobs

Cloud Run jobs allow you to run long-running jobs on Cloud Run.

This codelab is strictly informational, providing you knowledge about when and how to use Cloud Run jobs. It prepares you for the codelab Getting started with Cloud Run jobs, where you get hands-on experience using the Google Cloud Console to build a container, create and run a Cloud Run job, and update and schedule the job.

What you'll learn

  • What Cloud Run jobs are
  • When to use Cloud Run jobs
  • The Cloud Run jobs workflow
  • How to create a Cloud Run job
  • How to execute a Cloud Run job
  • How to execute a job on a schedule
  • How to view execution status
  • How to manage the job and job executions
  • How to detect job completion
  • How to update a job

Initially, Cloud Run jobs are available only in the europe-west9 region.

2. Cloud Run jobs overview

There are two ways to run your code on Cloud Run. You can run it continuously as a service,

c9a6f816087108b9.png

or, if your code performs work and then stops, you can use a Cloud Run job.

b24cdd6dd7e60e84.png

While Cloud Run services are a good fit for containers that run indefinitely listening for HTTP requests, Cloud Run jobs are a good fit for containers that run to completion and do not serve requests, such as data processing and administrative jobs.

Other use cases for Cloud Run jobs include:

  • Running a script to perform database migrations or other operational tasks.
  • Highly parallelized processing of all files in a Cloud Storage bucket.
  • Creating and sending invoices every month.
  • Every four hours, saving the results of a database query as XML and uploading the file.

Relative to other platforms that support long-running jobs, Cloud Run jobs start quickly after creation. Simple containers can start in as little as 10 seconds, depending on container complexity. This makes Cloud Run jobs well suited for situations where waiting a few minutes for a VM to start is problematic.

3. Cloud Run jobs workflow

The Cloud Run jobs workflow consists of two simple steps:

  1. Create the job. The job encapsulates all the configuration needed to execute the job, such as the container image.
  2. Execute the job. Use either the Cloud Console or the gcloud CLI and execute the job from the command line.

A job can start one container instance to run your code, or it can start many identical, independent container instances in parallel by specifying the number of tasks to run. For example, if you are reading 1,000 images from Cloud Storage to resize and crop them, processing them consecutively will be slower than processing them all at the same time with many container instances.

4. Create the job

To create the job, you can use either the Cloud Console or the gcloud CLI using the following command:

gcloud beta run jobs create JOB_NAME --image IMAGE_URL --region europe-west9 OPTIONS

Replace:

  • JOB_NAME with the name of the job you want to create.
  • IMAGE_URL with a reference to the container image.
  • Optionally, replace OPTIONS with any of the available flags. For a complete list of flags, run gcloud beta run jobs create --help.

Example flags include:

  • --tasks for the number of tasks to run.
  • -–max retries for the number of times a failed task is retried.
  • --parallelism for the maximum number of tasks that can run in parallel.
  • --execute-now to execute the job immediately after it is created.
  • --async to exit the job immediately after creating a new execution.

You can also use the usual Cloud Run features to secure your Cloud Run job and connect it to the rest of your Google Cloud Platform (GCP) environment.

5. Execute the job

To execute the job, you can use either the Cloud Console or the gcloud CLI using the following command:

  • In the Cloud Console, click on the job name, and click Execute near the top of the page.
  • In the gcloud CLI, use the following command:
gcloud beta run jobs execute JOB_NAME --region europe-west9 EXECUTION_OPTIONS

Replace JOB_NAME with the name of the job. Optionally, replace EXECUTION_OPTIONS to specify:

  • Immediate job execution after you create the job.
gcloud beta run jobs create JOB_NAME --region europe-west9 --execute-now
  • If you want to wait until the execution is complete before exiting.
gcloud beta run jobs create JOB_NAME --region europe-west9 --wait
  • If you want to exit immediately after creating a new execution.
gcloud beta run jobs create JOB_NAME --region europe-west9 --async

6. Execute the job on a schedule

If you want to execute your job on a schedule, use Cloud Scheduler.

For example, you may want to create and send invoices at regular intervals, or save the results of a database query as XML and upload the file every few hours.

With Cloud Scheduler, you can schedule the job and manage all your automation tasks from one place.

  • Run your batch and big data jobs at the same time each week, day, or hour with guaranteed execution and retries in case of failures.
  • Automate many of the tedious tasks associated with running cloud infrastructure in a reliable and fully managed manner.
  • Automate virtually anything.
  • View and manage all your jobs from a single UI or command-line interface.

Once you set up your environment to enable your project to use Cloud Scheduler, you create a Cloud Run job, then define a schedule by entering the name, region, description, frequency, and timezone. Cloud Scheduler then executes the Cloud Run job at the frequency you specify.

7. View the job execution status

Once your job executes, you can view logs in Cloud Logging logs and monitoring data in Cloud Monitoring.

To view logs, you can:

  • Use the Cloud Run page in the Cloud Console.
  • Use Cloud Logging Logs Explorer in the Cloud Console.

Both of these viewing methods examine the same logs stored in Cloud Logging, but the Cloud Logging Logs Explorer provides more details and more filtering capabilities.

Cloud Monitoring provides Cloud Run performance monitoring and metrics, along with alerts to send notifications when certain metric thresholds are exceeded. Cloud Run is automatically integrated with Cloud Monitoring with no setup or configuration required. This means that metrics of your Cloud Run jobs are captured automatically when they are running.

You can view metrics either in Cloud Monitoring or in the Cloud Run page in the console. Cloud Monitoring provides more charting and filtering options.

8. Manage the job and job executions

To manage your job, use the Cloud Console or the gcloud CLI to view a list of:

You can also:

  • Stop a running job execution by deleting it.
  • Delete job executions that are not running.
  • Delete a job.

Deleting a job cancels all pending job executions of that job, but the job logs and monitoring data continue to be available for the remainder of the log retention period, which is usually 30 days. Data monitoring continues to be available in Cloud Monitoring.

9. Detect the job execution completion

There are several ways to detect that an execution of your job finished running:

  • Run your job with the --wait flag and the gcloud CLI will exit when the job is done.
gcloud beta run jobs execute JOB_NAME --region europe-west9 --wait
  • Poll the REST API's executions.get endpoint, looking for the Completed status condition to be True.
  • Look for the written log entry when an execution completes which looks like this:

Completed condition status changed to True for Execution sleepy-l8zjp.

To look for this log message, run the following query in Cloud Logging:

resource.type="cloud_run_revision"
resource.labels.revision_name="EXECUTION_NAME" log_name="projects/PROJECT_ID/logs/cloudaudit.googleapis.com%2Fsystem_event"
protoPayload.response.status.conditions.type="Completed"
protoPayload.response.status.conditions.status="True"

You can also poll the location where you are writing output to detect when your output appears. However, this only works correctly if writing the output is the last thing your job does. For example, if your job does cleanup after writing output, the job could still fail after writing output.

10. Update the job

When you create a job, you can specify a container image with either a tag or an exact digest:

  • Tag example:
gcr.io/my-project/my-image:tag
  • Digest example:
gcr.io/my-project/my-image@sha256:41f34ab970ee...

If neither are specified, the tag :latest is used.

How you update your job to run the latest version of your code depends on whether you specified a container image with a tag or a digest.

If you specified a container image with a tag, Cloud Run resolves your image to a specific digest when you execute your job. This means that you don't need to explicitly update your job to pick up the latest version of your container the next time you run your job.

If you prefer to use a specific container digest every time your job runs, you can specify a container image with an exact digest. In this case, you have to explicitly update your job to use a different digest if you want your job to pick up the code changes you made:

gcloud beta run jobs update JOB_NAME --image NEW-IMAGE --region europe-west9

No matter how you specify your container image, all tasks of an execution will always use the same digest.

While an update is being applied, you cannot start new executions. You need to wait for the update to complete, which typically takes a few seconds. If the update fails, no new executions can be created. You will need to bring the job back to a good state first by making another update.

11. Congratulations

Congratulations, you learned when and how to use Cloud Run jobs!

You learned:

  • What Cloud Run jobs are
  • When to use Cloud Run jobs
  • The Cloud Run jobs workflow
  • How to create a Cloud Run job
  • How to execute a Cloud Run job
  • How to execute a job on a schedule
  • How to view execution status
  • How to manage the job and job executions
  • How to detect job completion
  • How to update a job

What's next?

Getting started with Cloud Run jobs

Help

For a full gcloud reference, use the help command:

gcloud beta run jobs --help

gcloud beta run jobs executions --help

Reference docs

Deploy jobs

Execute jobs

Execute jobs on a schedule

Setting up your environment

Logging and viewing logs

Monitoring Health and Performance

Cloud Logging documentation

Cloud Monitoring documentation

Google Cloud metrics

Introduction to alerting

Manage job executions

Manage jobs

Method: namespaces.executions.get

Jobs retries and checkpoints best practices