About this codelab
1. Introduction
Overview
If your GCP project is inside a VPC SC perimeter, you'll need to take additional steps in order to set up scheduled jobs. Because Cloud Scheduler cannot directly trigger jobs within VPC SC perimeters, you will need to proxy the request through another component. We recommend using a Cloud Run service as this proxy.
The architecture looks like this:
What you'll learn
- How to run a Cloud Run Job on a schedule inside a VPC SC perimeter
- How to create a Cloud Run service that triggers a Cloud Run job using the Cloud Run client libraries
- How to configure Cloud Scheduler to invoke a Cloud Run service on a schedule
2. Before you begin
First, make sure you have followed the steps to set up Cloud Run for VPC Service Controls.
Next, set environment variables that will be used throughout this codelab.
PROJECT_ID=<YOUR_PROJECT_ID> REGION=<YOUR_REGION> AR_REPO=sample-job-repo CLOUD_RUN_SERVICE=job-runner-service CLOUD_RUN_JOB=sample-job CLOUD_SCHEDULER=job-scheduler SERVICE_ACCOUNT="cloud-run-invoker-sa" SERVICE_ACCOUNT_ADDRESS=$SERVICE_ACCOUNT@$PROJECT_ID.iam.gserviceaccount.com NETWORK=default SUBNET=default
3. Create the Cloud Run job
This codelab uses the sample Cloud Run jobs container.
First, create an Artifact Registry repo for your Cloud Run job container.
gcloud artifacts repositories create $AR_REPO --repository-format=docker --location=$REGION --description="codelab for Cloud Run jobs on schedule within VPC SC"
Next, copy the sample Cloud Run jobs container to an Artifact Registry within your VPC SC-configured project. You can use the gcrane tool to do this by following these installation instructions. You can learn more about gcrane in the documentation for copying images between repositories.
gcrane cp us-docker.pkg.dev/cloudrun/container/job:latest $REGION-docker.pkg.dev/$PROJECT_ID/$AR_REPO/$CLOUD_RUN_JOB:latest
Second, deploy your Cloud Run job that is complaint for VPC Service Controls.
gcloud run jobs create $CLOUD_RUN_JOB --region $REGION \ --image $REGION-docker.pkg.dev/$PROJECT_ID/$AR_REPO/$CLOUD_RUN_JOB:latest \ --network=$NETWORK \ --subnet=$SUBNET \ --vpc-egress=all-traffic
For more information on creating Jobs, please follow the steps listed here in the Cloud Run jobs documentation.
4. Create a Service account
This service account will be used by Cloud Run to call the Cloud Run job.
First, create the service account by running this command:
gcloud iam service-accounts create $SERVICE_ACCOUNT \ --display-name="Cloud Run to run a Cloud Run job"
Second, grant the Cloud Run Invoker role and the Cloud Run Viewer role to the service account.
gcloud projects add-iam-policy-binding $PROJECT_ID \ --member serviceAccount:$SERVICE_ACCOUNT_ADDRESS \ --role=roles/run.invoker gcloud projects add-iam-policy-binding $PROJECT_ID \ --member serviceAccount:$SERVICE_ACCOUNT_ADDRESS \ --role=roles/run.viewer
5. Create the Cloud Run service
In this step, you'll deploy a Cloud Run service that acts as a proxy.
mkdir job-runner-service && cd $_
Create a file called main.py
with the following code.
import os from flask import Flask app = Flask(__name__) # pip install google-cloud-run from google.cloud import run_v2 @app.route('/') def hello(): client = run_v2.JobsClient() # UPDATE TO YOUR JOB NAME, REGION, AND PROJECT ID job_name = 'projects/YOUR_PROJECT_ID/locations/YOUR_JOB_REGION/jobs/YOUR_JOB_NAME' print("Triggering job...") request = run_v2.RunJobRequest(name=job_name) operation = client.run_job(request=request) response = operation.result() print(response) return "Done!" if __name__ == '__main__': app.run(debug=True, host="0.0.0.0", port=int(os.environ.get("PORT", 8080)))
Create a file named requirements.txt
with following code.
google-cloud-run flask
Lastly, create a Dockerfile
.
FROM python:3.9-slim-buster # for logging purposes ENV PYTHONUNBUFFERED=True WORKDIR /app COPY requirements.txt requirements.txt RUN pip install -r requirements.txt COPY . . CMD ["python3", "main.py"]
Next, build the container using the following docker commands. Note that Source-based deployments can be challenging to set up in a VPC SC environment. If you have an existing build and deploy pipeline, use it to build the source code into a container and deploy the container as a Cloud Run service.
docker build -t $CLOUD_RUN_SERVICE . docker tag $CLOUD_RUN_SERVICE $REGION-docker.pkg.dev/$PROJECT_ID/$AR_REPO/$CLOUD_RUN_SERVICE docker push $REGION-docker.pkg.dev/$PROJECT_ID/$AR_REPO/$CLOUD_RUN_SERVICE
Now deploy your Cloud Run service that is complaint for VPC Service Controls.
gcloud run deploy $CLOUD_RUN_SERVICE --region $REGION \ --image $REGION-docker.pkg.dev/$PROJECT_ID/$AR_REPO/$CLOUD_RUN_SERVICE \ --service-account $SERVICE_ACCOUNT_ADDRESS \ --network=$NETWORK \ --subnet=$SUBNET \ --vpc-egress=all-traffic \ --no-allow-unauthenticated \ --ingress internal
Save the service endpoint URL by running this command:
SERVICE_URL=$(gcloud run services describe $CLOUD_RUN_SERVICE --region $REGION --format 'value(status.url)')
6. Create a Cloud Schedule job to trigger the service
# create the Cloud Scheduler job gcloud scheduler jobs create http $CLOUD_SCHEDULER \ --location=$REGION \ --schedule="0 0 1 * *" \ --uri=$SERVICE_URL \ --http-method=GET \ --oidc-service-account-email=$SERVICE_ACCOUNT_ADDRESS
Once the Cloud Scheduler Job is created, you can run the following command to run the Cloud Scheduler Job immediately for testing purposes:
gcloud scheduler jobs run $CLOUD_SCHEDULER --location=$REGION
NOTE:
You may need to wait a couple of minutes for the job execution to complete. You can track its status on the Cloud Run Scheduler page.
Verify the Cloud Run job ran successfully by running the following commands:
EXECUTION_NAME=$(gcloud run jobs describe $CLOUD_RUN_JOB --region $REGION --format 'value(status.latestCreatedExecution.name)') gcloud run jobs executions describe $EXECUTION_NAME --region $REGION
and you should see something similar to the following:
✔ Execution sample-job-w6hrj in region us-central1 1 task completed successfully Elapsed time: 28 seconds
7. Congratulations!
Congratulations for completing the codelab!
What we've covered
- How to run a Cloud Run Job on a schedule inside a VPC SC perimeter
- How to create a Cloud Run service that triggers a Cloud Run job using the Cloud Run client libraries
- How to configure Cloud Scheduler to invoke a Cloud Run service on a schedule
8. Clean up
To avoid inadvertent charges, (for example, if the Cloud Run services are inadvertently invoked more times than your monthly Cloud Run invokement allocation in the free tier), you can either delete the GCP services or delete the project you created in Step 2.
To delete the Cloud Run service and Cloud Run job, go to the Cloud Run Cloud Console at https://console.cloud.google.com/run and delete the service.
If you choose to delete the entire project, you can go to https://console.cloud.google.com/cloud-resource-manager, select the project you created in Step 2, and choose Delete. If you delete the project, you'll need to change projects in your Cloud SDK. You can view the list of all available projects by running gcloud projects list
.