1. Introduction
Eventarc makes it easy to connect various services (Cloud Run, Kubernetes, Workflows) with events from a variety of sources. It allows you to build event-driven architectures in which microservices are loosely coupled and distributed. It also takes care of event ingestion, delivery, security, authorization, and error-handling for you which improves developer agility and application resilience. Check out Trigger Cloud Run with events from Eventarc codelab for an introduction to Eventarc.
In this codelab, you will use Eventarc to read events from Pub/Sub, Cloud Storage and Cloud Audit Logs and pass them to a Kubernetes service running on Google Kubernetes Engine (GKE).
What you'll learn
- Create a GKE cluster.
- Create a GKE service as an event sink.
- Create a Pub/Sub trigger.
- Create a Cloud Storage trigger
- Create a Cloud Audit Logs trigger.
2. Setup and requirements
Self-paced environment setup
- Sign-in to the Google Cloud Console and create a new project or reuse an existing one. If you don't already have a Gmail or Google Workspace account, you must create one.
- The Project name is the display name for this project's participants. It is a character string not used by Google APIs. You can update it at any time.
- The Project ID is unique across all Google Cloud projects and is immutable (cannot be changed after it has been set). The Cloud Console auto-generates a unique string; usually you don't care what it is. In most codelabs, you'll need to reference the Project ID (it is typically identified as
PROJECT_ID). If you don't like the generated ID, you may generate another random one. Alternatively, you can try your own and see if it's available. It cannot be changed after this step and will remain for the duration of the project. - For your information, there is a third value, a Project Number which some APIs use. Learn more about all three of these values in the documentation.
- Next, you'll need to enable billing in the Cloud Console to use Cloud resources/APIs. Running through this codelab shouldn't cost much, if anything at all. To shut down resources so you don't incur billing beyond this tutorial, you can delete the resources you created or delete the whole project. New users of Google Cloud are eligible for the $300 USD Free Trial program.
Start Cloud Shell
While Google Cloud can be operated remotely from your laptop, in this codelab you will be using Google Cloud Shell, a command line environment running in the Cloud.
From the Google Cloud Console, click the Cloud Shell icon on the top right toolbar:
It should only take a few moments to provision and connect to the environment. When it is finished, you should see something like this:
This virtual machine is loaded with all the development tools you'll need. It offers a persistent 5GB home directory, and runs on Google Cloud, greatly enhancing network performance and authentication. All of your work in this codelab can be done within a browser. You do not need to install anything.
Before you begin
Inside Cloud Shell, make sure that your project ID is setup:
PROJECT_ID=your-project-id gcloud config set project $PROJECT_ID
3. Create a GKE cluster
First, enable required services for GKE:
gcloud services enable container.googleapis.com
Create an Autopilot GKE cluster:
CLUSTER_NAME=eventarc-cluster REGION=us-central1 gcloud container clusters create-auto $CLUSTER_NAME --region $REGION
Make sure the cluster creation is completed before moving onto the next step.
4. Deploy a GKE service
Before deploying a service, get authentication credentials to interact with the cluster with kubectl:
gcloud container clusters get-credentials $CLUSTER_NAME \
--region $REGION
Next, deploy Cloud Run's hello container as a Kubernetes deployment on GKE. This service logs received HTTP requests and CloudEvents:
kubectl create deployment $SERVICE_NAME \
--image=gcr.io/cloudrun/hello
Expose the deployment as an internal Kubernetes service. This creates a service with a stable IP accessible within the cluster:
kubectl expose deployment $SERVICE_NAME \ --type ClusterIP --port 80 --target-port 8080
Before moving onto the next step, make sure the pod is running:
kubectl get pods NAME READY STATUS hello-gke-df6469d4b-5vv22 1/1 Running
You can also see the service:
kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP hello-gke LoadBalancer 10.51.1.26 <none>
5. Setup Eventarc
In this step, you will run steps to setup Eventarc and initialize Eventarc GKE destinations.
First, enable required services for Eventarc and Eventarc GKE destinations:
gcloud services enable eventarc.googleapis.com \ cloudresourcemanager.googleapis.com
Next, enable Eventarc to manage GKE clusters:
gcloud eventarc gke-destinations init
Eventarc creates a separate Event Forwarder pod for each trigger targeting a GKE service, and requires explicit permissions to make changes to the cluster. This is done by granting permissions to a special service account to manage resources in the cluster. This needs to be done once per Google Cloud project.
6. Event discovery
Before creating triggers, you can discover what the event sources are, the types of events they can emit, and how to configure triggers in order to consume them.
You can check the documentation page on events supported by Eventarc.You can also explore the events using gcloud.
To see the list of different types of events:
gcloud beta eventarc attributes types list NAME DESCRIPTION google.cloud.audit.log.v1.written Cloud Audit Log written google.cloud.pubsub.topic.v1.messagePublished Cloud Pub/Sub message published google.cloud.storage.object.v1.archived Cloud Storage: Sent when a live version of an (object versioned) object is archived or deleted. google.cloud.storage.object.v1.deleted Cloud Storage: Sent when an object has been permanently deleted. google.cloud.storage.object.v1.finalized Cloud Storage: Sent when a new object (or a new generation of an existing object). google.cloud.storage.object.v1.metadataUpdated Cloud Storage: Sent when the metadata of an existing object changes.
To get more information about each event type:
gcloud beta eventarc attributes types describe google.cloud.audit.log.v1.written attributes: type,serviceName,methodName,resourceName description: 'Cloud Audit Log: Sent when a log is written.' name: google.cloud.audit.log.v1.written
To see the list of services that emit a certain event type:
gcloud beta eventarc attributes service-names list --type=google.cloud.audit.log.v1.written SERVICE_NAME DISPLAY_NAME accessapproval.googleapis.com Access Approval accesscontextmanager.googleapis.com Access Context Manager admin.googleapis.com Google Workspace Admin aiplatform.googleapis.com AI Platform (under Vertex AI) apigee.googleapis.com Apigee apigeeconnect.googleapis.com Apigee Connect ... workflows.googleapis.com Workflows
To see the list of method names (sub-events) that each service can emit:
gcloud beta eventarc attributes method-names list --type=google.cloud.audit.log.v1.written --service-name=workflows.googleapis.com METHOD_NAME google.cloud.workflows.v1.Workflows.CreateWorkflow google.cloud.workflows.v1.Workflows.DeleteWorkflow google.cloud.workflows.v1.Workflows.GetWorkflow google.cloud.workflows.v1.Workflows.ListWorkflows google.cloud.workflows.v1.Workflows.UpdateWorkflow google.cloud.workflows.v1beta.Workflows.CreateWorkflow google.cloud.workflows.v1beta.Workflows.DeleteWorkflow google.cloud.workflows.v1beta.Workflows.GetWorkflow google.cloud.workflows.v1beta.Workflows.ListWorkflows google.cloud.workflows.v1beta.Workflows.UpdateWorkflow
7. Create a Pub/Sub trigger
One way of receiving events is through Pub/Sub. Any application can publish messages to Pub/Sub and these messages can be delivered to services via Eventarc.
Setup
Before creating any triggers, you need a service account to be used by triggers.
Create a service account:
SERVICE_ACCOUNT=eventarc-gke-trigger-sa gcloud iam service-accounts create $SERVICE_ACCOUNT
The service account must be granted the following roles for triggers with GKE destinations:
roles/pubsub.subscriberroles/monitoring.metricWriterroles/eventarc.eventReceiver(Only for AuditLog triggers. This will be added in a later step)
Grant the roles:
gcloud projects add-iam-policy-binding $PROJECT_ID \ --member serviceAccount:$SERVICE_ACCOUNT@$PROJECT_ID.iam.gserviceaccount.com \ --role roles/pubsub.subscriber gcloud projects add-iam-policy-binding $PROJECT_ID \ --member serviceAccount:$SERVICE_ACCOUNT@$PROJECT_ID.iam.gserviceaccount.com \ --role roles/monitoring.metricWriter
Create
Create a trigger to route Pub/Sub messages to your service:
TRIGGER_NAME=trigger-pubsub-gke gcloud eventarc triggers create $TRIGGER_NAME \ --destination-gke-cluster=$CLUSTER_NAME \ --destination-gke-location=$REGION \ --destination-gke-namespace=default \ --destination-gke-service=$SERVICE_NAME \ --destination-gke-path=/ \ --event-filters="type=google.cloud.pubsub.topic.v1.messagePublished" \ --location=$REGION \ --service-account=$SERVICE_ACCOUNT@$PROJECT_ID.iam.gserviceaccount.com
Test
You can check that the trigger is created by listing all triggers:
gcloud eventarc triggers list
Pub/Sub trigger creates a Pub/Sub topic under the covers. Let's find it out and assign to a variable:
TOPIC_ID=$(gcloud eventarc triggers describe $TRIGGER_NAME --location $REGION --format='value(transport.pubsub.topic)')
Use gcloud to publish a message to the topic:
gcloud pubsub topics publish $TOPIC_ID --message="Hello World"
To check if the event is received, first, find the pod name:
kubectl get pods NAME READY STATUS hello-gke-df6469d4b-5vv22 1/1 Running
And store it to a variable:
POD_NAME=hello-gke-df6469d4b-5vv22
Check the logs of the pod to verify the received event:
kubectl logs $POD_NAME
{
"severity": "INFO",
"eventType": "google.cloud.pubsub.topic.v1.messagePublished",
"message": "Received event of type google.cloud.pubsub.topic.v1.messagePublished. Event data: Hello World",
"event": {
"data": {
"subscription": "projects/atamel-eventarc-gke/subscriptions/eventarc-us-central1-trigger-pubsub-gke-sub-270",
"message": {
"data": "SGVsbG8gV29ybGQ=",
"messageId": "6031025573654834",
"publishTime": "2022-10-19T14:13:07.990Z"
}
},
"datacontenttype": "application/json",
"id": "6031025573654834",
"source": "//pubsub.googleapis.com/projects/atamel-eventarc-gke/topics/eventarc-us-central1-trigger-pubsub-gke-729",
"specversion": "1.0",
"time": "2022-10-19T14:13:07.99Z",
"type": "google.cloud.pubsub.topic.v1.messagePublished"
}
}
8. Create a Cloud Storage trigger
Another way of receiving events is through Cloud Storage. For example, an application can upload a file to a bucket and that event can be delivered to services via Eventarc.
Setup
Before creating a Cloud Storage trigger, create a bucket to receive events from:
BUCKET_NAME=eventarc-gcs-$PROJECT_ID gsutil mb -l $REGION gs://$BUCKET_NAME
You also need to add the pubsub.publisher role to the Cloud Storage service account for Cloud Storage triggers:
SERVICE_ACCOUNT_STORAGE=$(gsutil kms serviceaccount -p $PROJECT_ID)
gcloud projects add-iam-policy-binding $PROJECT_ID \
--member serviceAccount:$SERVICE_ACCOUNT_STORAGE \
--role roles/pubsub.publisher
Create
Create a trigger to route new file creation events from the bucket to your service:
TRIGGER_NAME=trigger-storage-gke gcloud eventarc triggers create $TRIGGER_NAME \ --destination-gke-cluster=$CLUSTER_NAME \ --destination-gke-location=$REGION \ --destination-gke-namespace=default \ --destination-gke-service=$SERVICE_NAME \ --destination-gke-path=/ \ --event-filters="type=google.cloud.storage.object.v1.finalized" \ --event-filters="bucket=$BUCKET_NAME" \ --location=$REGION \ --service-account=$SERVICE_ACCOUNT@$PROJECT_ID.iam.gserviceaccount.com
Test
You can check that the trigger is created by listing all triggers:
gcloud eventarc triggers list
Create a file and use gsutil to upload the file to the bucket:
echo "Hello World" > random.txt gsutil cp random.txt gs://$BUCKET_NAME/random.txt
Check the logs of the pod to verify the received event:
kubectl logs $POD_NAME
{
"severity": "INFO",
"eventType": "google.cloud.storage.object.v1.finalized",
"message": "Received event of type google.cloud.storage.object.v1.finalized. Event data: {\n \"kind\": \"storage#object\",\n \"id\": \"eventarc-gcs-atamel-eventarc-gke/random.txt/1666190425669022\",\n \"selfLink\": \"https://www.googleapis.com/storage/v1/b/eventarc-gcs-atamel-eventarc-gke/o/random.txt\",\n \"name\": \"random.txt\",\n \"bucket\": \"eventarc-gcs-atamel-eventarc-gke\",\n \"generation\": \"1666190425669022\",\n \"metageneration\": \"1\",\n \"contentType\": \"text/plain\",\n \"timeCreated\": \"2022-10-19T14:40:25.678Z\",\n \"updated\": \"2022-10-19T14:40:25.678Z\",\n \"storageClass\": \"STANDARD\",\n \"timeStorageClassUpdated\": \"2022-10-19T14:40:25.678Z\",\n \"size\": \"12\",\n \"md5Hash\": \"5Z/5eUEET4XfUpfhwwLSYA==\",\n \"mediaLink\": \"https://storage.googleapis.com/download/storage/v1/b/eventarc-gcs-atamel-eventarc-gke/o/random.txt?generation=1666190425669022\u0026alt=media\",\n \"contentLanguage\": \"en\",\n \"crc32c\": \"R1jUOQ==\",\n \"etag\": \"CJ77zIPD7PoCEAE=\"\n}\n",
"event": {
"bucket": "eventarc-gcs-atamel-eventarc-gke",
"data": {
"kind": "storage#object",
"id": "eventarc-gcs-atamel-eventarc-gke/random.txt/1666190425669022",
"selfLink": "https://www.googleapis.com/storage/v1/b/eventarc-gcs-atamel-eventarc-gke/o/random.txt",
"name": "random.txt",
"bucket": "eventarc-gcs-atamel-eventarc-gke",
"generation": "1666190425669022",
"metageneration": "1",
"contentType": "text/plain",
"timeCreated": "2022-10-19T14:40:25.678Z",
"updated": "2022-10-19T14:40:25.678Z",
"storageClass": "STANDARD",
"timeStorageClassUpdated": "2022-10-19T14:40:25.678Z",
"size": "12",
"md5Hash": "5Z/5eUEET4XfUpfhwwLSYA==",
"mediaLink": "https://storage.googleapis.com/download/storage/v1/b/eventarc-gcs-atamel-eventarc-gke/o/random.txt?generation=1666190425669022\u0026alt=media",
"contentLanguage": "en",
"crc32c": "R1jUOQ==",
"etag": "CJ77zIPD7PoCEAE="
},
"datacontenttype": "application/json",
"id": "6031255652220627",
"source": "//storage.googleapis.com/projects/_/buckets/eventarc-gcs-atamel-eventarc-gke",
"specversion": "1.0",
"subject": "objects/random.txt",
"time": "2022-10-19T14:40:25.678152Z",
"type": "google.cloud.storage.object.v1.finalized"
}
}
9. Create a Cloud Audit Logs trigger
Although Cloud Storage trigger is the better way to listen for Cloud Storage events, in this step, you create a Cloud Audit Log trigger to do the same.
Setup
To receive events from a service, you need to enable Audit Logs. From the Google Cloud Console, select IAM & Admin and Audit Logs from the upper left-hand menu. In the list of services, check Google Cloud Storage:
On the right hand side, make sure Admin, Read and Write are selected and click Save:
You also need to add eventarc.eventReceiver role to the trigger service account for Cloud Audit Logs triggers:
gcloud projects add-iam-policy-binding $PROJECT_ID \ --member serviceAccount:$SERVICE_ACCOUNT@$PROJECT_ID.iam.gserviceaccount.com \ --role roles/eventarc.eventReceiver
Create
Create a trigger to route new file creation events from the bucket to your service:
TRIGGER_NAME=trigger-auditlog-storage-gke gcloud eventarc triggers create $TRIGGER_NAME \ --destination-gke-cluster=$CLUSTER_NAME \ --destination-gke-location=$REGION \ --destination-gke-namespace=default \ --destination-gke-service=$SERVICE_NAME \ --destination-gke-path=/ \ --event-filters="type=google.cloud.audit.log.v1.written" \ --event-filters="serviceName=storage.googleapis.com" \ --event-filters="methodName=storage.objects.create" \ --event-filters-path-pattern="resourceName=/projects/_/buckets/$BUCKET_NAME/objects/*" \ --location=$REGION \ --service-account=$SERVICE_ACCOUNT@$PROJECT_ID.iam.gserviceaccount.com
Test
Audit Logs triggers take a little bit of time to initialize. You can check that the trigger is created by listing all triggers:
gcloud eventarc triggers list
You should see that the Active field is Yes:
NAME TYPE DESTINATION ACTIVE trigger-auditlog-storage-gke google.cloud.audit.log.v1.written GKE: hello-gke Yes
Create a file and use gsutil to upload the file to the bucket:
echo "Hello World" > random.txt gsutil cp random.txt gs://$BUCKET_NAME/random.txt
Check the logs of the pod to verify the received event:
kubectl logs $POD_NAME
{
"severity": "INFO",
"eventType": "google.cloud.audit.log.v1.written",
"message": "Received event of type google.cloud.audit.log.v1.written. Event data: {\"protoPayload\":{\"status\":{},\"authenticationInfo\":{\"principalEmail\":\"atameldev@gmail.com\"},\"requestMetadata\":{\"callerIp\":\"149.71.143.227\",\"callerSuppliedUserAgent\":\"apitools Python/3.10.4 gsutil/5.14 (darwin) analytics/disabled interactive/True command/cp google-cloud-sdk/405.0.1,gzip(gfe)\",\"requestAttributes\":{\"time\":\"2022-10-19T15:05:54.144615670Z\",\"auth\":{}},\"destinationAttributes\":{}},\"serviceName\":\"storage.googleapis.com\",\"methodName\":\"storage.objects.create\",\"authorizationInfo\":[{\"resource\":\"projects/_/buckets/eventarc-gcs-atamel-eventarc-gke/objects/random.txt\",\"permission\":\"storage.objects.delete\",\"granted\":true,\"resourceAttributes\":{}},{\"resource\":\"projects/_/buckets/eventarc-gcs-atamel-eventarc-gke/objects/random.txt\",\"permission\":\"storage.objects.create\",\"granted\":true,\"resourceAttributes\":{}}],\"resourceName\":\"projects/_/buckets/eventarc-gcs-atamel-eventarc-gke/objects/random.txt\",\"serviceData\":{\"@type\":\"type.googleapis.com/google.iam.v1.logging.AuditData\",\"policyDelta\":{\"bindingDeltas\":[{\"action\":\"ADD\",\"role\":\"roles/storage.legacyObjectOwner\",\"member\":\"projectOwner:atamel-eventarc-gke\"},{\"action\":\"ADD\",\"role\":\"roles/storage.legacyObjectOwner\",\"member\":\"projectEditor:atamel-eventarc-gke\"},{\"action\":\"ADD\",\"role\":\"roles/storage.legacyObjectOwner\",\"member\":\"user:atameldev@gmail.com\"},{\"action\":\"ADD\",\"role\":\"roles/storage.legacyObjectReader\",\"member\":\"projectViewer:atamel-eventarc-gke\"}]}},\"resourceLocation\":{\"currentLocations\":[\"us-central1\"]}},\"insertId\":\"-8vmrbve7pol2\",\"resource\":{\"type\":\"gcs_bucket\",\"labels\":{\"project_id\":\"atamel-eventarc-gke\",\"bucket_name\":\"eventarc-gcs-atamel-eventarc-gke\",\"location\":\"us-central1\"}},\"timestamp\":\"2022-10-19T15:05:54.138732321Z\",\"severity\":\"INFO\",\"logName\":\"projects/atamel-eventarc-gke/logs/cloudaudit.googleapis.com%2Fdata_access\",\"receiveTimestamp\":\"2022-10-19T15:05:54.839604461Z\"}",
"event": {
"data": {
"protoPayload": {
"status": {
},
"authenticationInfo": {
"principalEmail": "atameldev@gmail.com"
},
"requestMetadata": {
"callerIp": "149.71.143.227",
"callerSuppliedUserAgent": "apitools Python/3.10.4 gsutil/5.14 (darwin) analytics/disabled interactive/True command/cp google-cloud-sdk/405.0.1,gzip(gfe)",
"requestAttributes": {
"time": "2022-10-19T15:05:54.144615670Z",
"auth": {
}
},
"destinationAttributes": {
}
},
"serviceName": "storage.googleapis.com",
"methodName": "storage.objects.create",
"authorizationInfo": [
{
"resource": "projects/_/buckets/eventarc-gcs-atamel-eventarc-gke/objects/random.txt",
"permission": "storage.objects.delete",
"granted": true,
"resourceAttributes": {
}
},
{
"resource": "projects/_/buckets/eventarc-gcs-atamel-eventarc-gke/objects/random.txt",
"permission": "storage.objects.create",
"granted": true,
"resourceAttributes": {
}
}
],
"resourceName": "projects/_/buckets/eventarc-gcs-atamel-eventarc-gke/objects/random.txt",
"serviceData": {
"@type": "type.googleapis.com/google.iam.v1.logging.AuditData",
"policyDelta": {
"bindingDeltas": [
{
"action": "ADD",
"role": "roles/storage.legacyObjectOwner",
"member": "projectOwner:atamel-eventarc-gke"
},
{
"action": "ADD",
"role": "roles/storage.legacyObjectOwner",
"member": "projectEditor:atamel-eventarc-gke"
},
{
"action": "ADD",
"role": "roles/storage.legacyObjectOwner",
"member": "user:atameldev@gmail.com"
},
{
"action": "ADD",
"role": "roles/storage.legacyObjectReader",
"member": "projectViewer:atamel-eventarc-gke"
}
]
}
},
"resourceLocation": {
"currentLocations": [
"us-central1"
]
}
},
"insertId": "-8vmrbve7pol2",
"resource": {
"type": "gcs_bucket",
"labels": {
"project_id": "atamel-eventarc-gke",
"bucket_name": "eventarc-gcs-atamel-eventarc-gke",
"location": "us-central1"
}
},
"timestamp": "2022-10-19T15:05:54.138732321Z",
"severity": "INFO",
"logName": "projects/atamel-eventarc-gke/logs/cloudaudit.googleapis.com%2Fdata_access",
"receiveTimestamp": "2022-10-19T15:05:54.839604461Z"
},
"datacontenttype": "application/json; charset=utf-8",
"dataschema": "https://googleapis.github.io/google-cloudevents/jsonschema/google/events/cloud/audit/v1/LogEntryData.json",
"id": "projects/atamel-eventarc-gke/logs/cloudaudit.googleapis.com%2Fdata_access-8vmrbve7pol21666191954138732",
"methodname": "storage.objects.create",
"recordedtime": "2022-10-19T15:05:54.138732321Z",
"resourcename": "projects/_/buckets/eventarc-gcs-atamel-eventarc-gke/objects/random.txt",
"servicename": "storage.googleapis.com",
"source": "//cloudaudit.googleapis.com/projects/atamel-eventarc-gke/logs/data_access",
"specversion": "1.0",
"subject": "storage.googleapis.com/projects/_/buckets/eventarc-gcs-atamel-eventarc-gke/objects/random.txt",
"time": "2022-10-19T15:05:54.839604461Z",
"type": "google.cloud.audit.log.v1.written"
}
}
10. Congratulations!
Congratulations for completing the codelab.
What we've covered
- Create a GKE cluster.
- Create a GKE service as an event sink.
- Create a Pub/Sub trigger.
- Create a Cloud Storage trigger.
- Create a Cloud Audit Logs trigger.