1. Introduction
Cloud Run allows you to run stateless containers in a fully managed environment. It is built from open-source Knative, letting you choose to run your containers either fully managed with Cloud Run, or in your Google Kubernetes Engine cluster with Cloud Run for Anthos.
Eventarc makes it easy to connect Cloud Run services with events from a variety of sources. It allows you to build event-driven architectures in which microservices are loosely coupled and distributed. It also takes care of event ingestion, delivery, security, authorization, and error-handling for you which improves developer agility and application resilience.
In this codelab, you will learn about Eventarc. More specifically, you will listen to events from Cloud Pub/Sub and Cloud Audit Logs with Eventarc and pass them to Cloud Run (fully managed).
What you'll learn
- Long term vision of Eventarc
- Current state of Eventarc
- Create a Cloud Run sink
- Create a trigger for Cloud Pub/Sub
- Create a trigger for Cloud Storage
- Create a trigger for Cloud Audit Logs
2. Long Term Vision
Long-term Vision is to be able to deliver events from various sources to Google Cloud sinks and Custom sinks.
Google Cloud sources | Event sources that are Google Cloud owned products |
Google sources | Event sources that are Google-owned products such as Gmail, Hangouts, Android Management and more |
Custom sources | Event sources that are not Google-owned products and are created by end-users themselves |
3rd party sources | Event sources that are neither Google-owned nor customer-produced. This includes popular event sources such as Github, SAP, Datadog, Pagerduty, etc that are owned and maintained by 3rd party providers, partners, or OSS communities. |
Events are normalized to CloudEvents v1.0 format for cross-service interoperability. CloudEvents is a vendor-neutral open spec describing event data in common formats, enabling interoperability across services, platforms and systems.
3. Current state
This is the current version which delivers an initial set of the long-term functionality.
You can draw events from Google Cloud sources and Custom applications publishing to Cloud Pub/Sub and deliver them to Google Cloud Run sinks.
Events from a breadth of Google Cloud sources are delivered by way of Cloud Audit Logs. The latency and availability of event delivery from these sources are tied to those of Cloud Audit Logs. Whenever an event from a Google Cloud source is fired, a corresponding Cloud Audit Log entry is created.
Custom applications publishing to Cloud Pub/Sub can publish messages to a Pub/Sub topic they specify in any format.
The underlying delivery mechanism in Eventarc is Cloud Pub/Sub and topics and subscriptions created to facilitate event delivery are visible in customers' Pub/Sub instance for their project.
Event triggers are the filtering mechanism to specify which events to deliver to which sink.
All events are delivered in the CloudEvents v1.0 format for cross service interoperability.
4. Setup and Requirements
Self-paced environment setup
- Sign in to Cloud Console and create a new project or reuse an existing one. If you don't already have a Gmail or Google Workspace account, you must create one.
- The Project Name is your display name for this project. As long as you follow its naming conventions, you can use anything you want and can update it at any time.
- The Project ID must be unique across all Google Cloud projects and is immutable (cannot be changed once set). The Cloud console auto-generates a unique string; usually you don't care what it is. In most codelabs, you'll need to reference the Project ID (and it is typically identified as
PROJECT_ID), so if you don't like it, generate another random one, or, you can try your own and see if it's available. Then it's "frozen" once the project is created.
- Next, you'll need to enable billing in Cloud Console in order to use Google Cloud resources.
Running through this codelab shouldn't cost much, if anything at all. Be sure to to follow any instructions in the "Cleaning up" section which advises you how to shut down resources so you don't incur billing beyond this tutorial. New users of Google Cloud are eligible for the $300 USD Free Trial program.
Start Cloud Shell
While Google Cloud can be operated remotely from your laptop, in this codelab you will be using Google Cloud Shell, a command line environment running in the Cloud.
From the GCP Console click the Cloud Shell icon on the top right toolbar:
It should only take a few moments to provision and connect to the environment. When it is finished, you should see something like this:
This virtual machine is loaded with all the development tools you'll need. It offers a persistent 5GB home directory, and runs on Google Cloud, greatly enhancing network performance and authentication. All of your work in this lab can be done with simply a browser.
5. Before you begin
Before creating an event sink and event triggers, go through some setup steps to enable APIs, set regions, create service accounts and so on.
Enable APIs
Inside Cloud Shell, make sure that you project id is setup:
gcloud config set project [YOUR-PROJECT-ID] PROJECT_ID=$(gcloud config get-value project)
Enable all necessary services:
gcloud services enable run.googleapis.com gcloud services enable eventarc.googleapis.com gcloud services enable logging.googleapis.com gcloud services enable cloudbuild.googleapis.com
You also need to enable Data Access Audit Logs (Admin read, Data read, Data write) for all Google Cloud services from which you intend to receive events. In a later step, we will show you how to enable audit logs for Google Cloud Storage.
Set region and platform
Eventarc is available in the multiple Google Cloud regions and globally. You can see the list of regions with this command:
gcloud eventarc locations list
In Cloud Shell, set the Cloud Run region to one of the supported regions, platform to managed and the location of Eventarc trigger:
REGION=europe-west1 gcloud config set run/region $REGION gcloud config set run/platform managed gcloud config set eventarc/location $REGION
You can check that the configuration is set:
gcloud config list ... [eventarc] location = europe-west1 ... [run] platform = managed region = europe-west1
Configure service accounts
The default compute service account will be used in triggers. Grant the eventarc.eventReceiver role to the default compute service account:
PROJECT_NUMBER=$(gcloud projects describe $PROJECT_ID --format='value(projectNumber)')
gcloud projects add-iam-policy-binding $PROJECT_ID \
--member serviceAccount:$PROJECT_NUMBER-compute@developer.gserviceaccount.com \
--role roles/eventarc.eventReceiver
Grant the pubsub.publisher role to the Cloud Storage service account. This is needed for the Eventarc Cloud Storage trigger:
SERVICE_ACCOUNT=$(gsutil kms serviceaccount -p $PROJECT_NUMBER)
gcloud projects add-iam-policy-binding $PROJECT_NUMBER \
--member serviceAccount:$SERVICE_ACCOUNT \
--role roles/pubsub.publisher
If you enabled the Pub/Sub service account on or before April 8, 2021, grant the iam.serviceAccountTokenCreator role to the Pub/Sub service account:
gcloud projects add-iam-policy-binding $PROJECT_ID \ --member serviceAccount:service-$PROJECT_NUMBER@gcp-sa-pubsub.iam.gserviceaccount.com \ --role roles/iam.serviceAccountTokenCreator
6. Event Discovery
You can discover what the registered sources are, the types of events they can emit, and how to configure triggers in order to consume them.
To see the list of different types of events:
gcloud beta eventarc attributes types list NAME DESCRIPTION google.cloud.audit.log.v1.written Cloud Audit Log written google.cloud.pubsub.topic.v1.messagePublished Cloud Pub/Sub message published
To get more information about each event type:
gcloud beta eventarc attributes types describe google.cloud.pubsub.topic.v1.messagePublished attributes: type description: Cloud Pub/Sub message published name: google.cloud.pubsub.topic.v1.messagePublished
7. Create a Cloud Run Sink
As an event sink, you can deploy Cloud Run's Hello container that already logs the contents of CloudEvents.
Deploy to Cloud Run
Deploy your containerized application to Cloud Run:
SERVICE_NAME=hello gcloud run deploy $SERVICE_NAME \ --image=gcr.io/cloudrun/hello \ --allow-unauthenticated
On success, the command line displays the service URL. You can now visit your deployed container by opening the service URL in any browser window.
8. Create a trigger for Cloud Pub/Sub
One way of receiving events is through Cloud Pub/Sub. Any application can publish messages to Pub/Sub and these messages can be delivered to Cloud Run sinks via Eventarc.
Create a trigger
Before creating the trigger, get more details on the parameters you'll need to construct a trigger for events from Cloud Pub/Sub:
gcloud beta eventarc attributes types describe google.cloud.pubsub.topic.v1.messagePublished
Create a trigger to filter events published to the Pub/Sub topic to our deployed Cloud Run service:
TRIGGER_NAME=trigger-pubsub gcloud eventarc triggers create $TRIGGER_NAME \ --destination-run-service=$SERVICE_NAME \ --destination-run-region=$REGION \ --event-filters="type=google.cloud.pubsub.topic.v1.messagePublished"
Find the topic
Pub/Sub trigger creates a Pub/Sub topic under the covers. Let's find it out and assign to a variable:
TOPIC_ID=$(gcloud eventarc triggers describe $TRIGGER_NAME --format='value(transport.pubsub.topic)')
Test the trigger
You can check that the trigger is created by listing all triggers:
gcloud eventarc triggers list
In order to simulate a custom application sending message, you can use gcloud to to fire an event:
gcloud pubsub topics publish $TOPIC_ID --message="Hello World"
The Cloud Run sink we created logs the body of the incoming message. You can view this in the Logs section of your Cloud Run instance:
Delete the trigger
Optionally, you can delete the trigger once done testing.
gcloud eventarc triggers delete $TRIGGER_NAME
Bring your own Pub/Sub topic
By default, when you create a Pub/Sub trigger, Eventarc creates a Pub/Sub topic under the covers for you to use as a transport topic between your application and a Cloud Run service. This is useful to easily and quickly create a Pub/Sub backed trigger but it is also limiting. For example, you cannot set up a fanout from a single Pub/Sub topic to multiple Cloud Run services.
There is a way to create triggers from an existing Pub/Sub topic. Eventarc allows you to specify an existing Pub/Sub topic in the same project with --transport-topic gcloud flag.
To see how this works, create a Pub/Sub topic to use as transport topic:
TOPIC_ID=my-topic gcloud pubsub topics create $TOPIC_ID
Create a trigger:
TRIGGER_NAME=trigger-pubsub-existing gcloud eventarc triggers create $TRIGGER_NAME \ --destination-run-service=$SERVICE_NAME \ --destination-run-region=$REGION \ --event-filters="type=google.cloud.pubsub.topic.v1.messagePublished" \ --transport-topic=projects/$PROJECT_ID/topics/$TOPIC_ID
You can test the trigger but sending a message to the topic:
gcloud pubsub topics publish $TOPIC_ID --message="Hello again"
9. Create a trigger for Cloud Storage
In this step, you will create a trigger to listen for events from Cloud Storage.
Create a bucket
First, create a Cloud Storage bucket in the same region as the deployed Cloud Run service:
BUCKET_NAME=eventarc-gcs-$PROJECT_ID gsutil mb -l $REGION gs://$BUCKET_NAME
Create a trigger
You are now ready to create an event trigger for Cloud Storage.
Create the trigger with the right event filters and the service account:
TRIGGER_NAME=trigger-gcs gcloud eventarc triggers create $TRIGGER_NAME\ --destination-run-service=$SERVICE_NAME \ --destination-run-region=$REGION \ --event-filters="type=google.cloud.storage.object.v1.finalized" \ --event-filters="bucket=$BUCKET_NAME" \ --service-account=$PROJECT_NUMBER-compute@developer.gserviceaccount.com
Test the trigger
List all triggers to confirm that trigger was successfully created:
gcloud eventarc triggers list
Upload a file to the Cloud Storage bucket:
echo "Hello World" > random.txt gsutil cp random.txt gs://$BUCKET_NAME/random.txt
If you check the logs of the Cloud Run service in Cloud Console, you should see the received event:
Delete the trigger
Optionally, you can delete the trigger once done testing:
gcloud eventarc triggers delete $TRIGGER_NAME
10. Create a trigger for Cloud Audit Logs
You can also create a trigger to listen for events from Cloud Storage via Cloud Audit Logs.
Create a bucket
First, create a Cloud Storage bucket in the same region as the deployed Cloud Run service:
BUCKET_NAME=eventarc-auditlog-$PROJECT_ID gsutil mb -l $REGION gs://$BUCKET_NAME
Enable Cloud Audit Logs
In order to receive events from a service, you need to enable Cloud Audit Logs. From the Cloud Console, select IAM & Admin and Audit Logs from the upper left-hand menu. In the list of services, check Google Cloud Storage:
On the right hand side, make sure Admin, Read and Write are selected. Click save:
Test Cloud Audit Logs
To learn how to identify the parameters you'll need to set up an actual trigger, perform an actual operation.
For example, create a random text file and upload it to the bucket:
echo "Hello World" > random.txt gsutil cp random.txt gs://$BUCKET_NAME/random.txt
Now, let's see what kind of audit log this update generated. From the Cloud Console, select Logging and Logs Viewer from the upper left-hand menu.
Under Query Builder, choose GCS Bucket and choose your bucket and its location. Click Add.
Once you run the query, you'll see logs for the storage bucket and one of those should be storage.objects.create:
Note the serviceName, methodName and resourceName. We'll use these in creating the trigger.
Create a trigger
You are now ready to create an event trigger for Cloud Audit Logs.
You can get more details on the parameters you'll need to construct the trigger:
gcloud beta eventarc attributes types describe google.cloud.audit.log.v1.written
Create the trigger with the right filters and the service account:
TRIGGER_NAME=trigger-auditlog gcloud eventarc triggers create $TRIGGER_NAME\ --destination-run-service=$SERVICE_NAME \ --destination-run-region=$REGION \ --event-filters="type=google.cloud.audit.log.v1.written" \ --event-filters="serviceName=storage.googleapis.com" \ --event-filters="methodName=storage.objects.create" \ --service-account=$PROJECT_NUMBER-compute@developer.gserviceaccount.com
Test the trigger
List all triggers to confirm that trigger was successfully created:
gcloud eventarc triggers list
Wait for up to 10 minutes for the trigger creation to be propagated and for it to begin filtering events. Once ready, it will filter create events and send them to the service.
You're now ready to fire an event.
Upload the same file to the Cloud Storage bucket as you did earlier:
gsutil cp random.txt gs://$BUCKET_NAME/random.txt
If you check the logs of the Cloud Run service in Cloud Console, you should see the received event:
Delete the trigger
Optionally, you can delete the trigger once done testing:
gcloud eventarc triggers delete $TRIGGER_NAME
11. Congratulations!
Congratulations for completing the codelab.
What we've covered
- Long term vision of Eventarc
- Current state of Eventarc
- Create a Cloud Run sink
- Create a trigger for Cloud Pub/Sub
- Create a trigger for Cloud Storage
- Create a trigger for Cloud Audit Logs