Route Datadog monitoring alerts to Google Cloud with Eventarc (Part 1)

1. Introduction

Eventarc integrations feature is covered by the Pre-GA Offerings Terms of the Google Cloud Terms of Service. Pre-GA features might have limited support, and changes to pre-GA features might not be compatible with other pre-GA versions. For more information, see the launch stage descriptions.

You need to have a Datadog account to complete this codelab. You can sign-up for a trial account at Datadog's website.

Eventarc makes it easy to connect Google Cloud services with events from a variety of sources. It allows you to build event-driven architectures in which microservices are loosely coupled and distributed. It also takes care of event ingestion, delivery, security, authorization, and error-handling for you which improves developer agility and application resilience.

Datadog is a monitoring and security platform for cloud applications. It brings together end-to-end traces, metrics, and logs to make your applications, infrastructure, and third-party services observable.

In this first codelab, you will learn how to route a simple Datadog monitoring alert to Google Cloud with Eventarc.

What you'll learn

• How to discover the Datadog provider.
• How to setup a channel to the Datadog provider.
• How to create a workflow to log events.
• How to create an Eventarc trigger with the channel.
• How to create a Datadog monitor.
• How to test the Datadog monitor, Eventarc trigger and the workflow.

2. Setup and Requirements

Self-paced environment setup

1. Sign-in to the Google Cloud Console and create a new project or reuse an existing one. If you don't already have a Gmail or Google Workspace account, you must create one.

• The Project name is the display name for this project's participants. It is a character string not used by Google APIs. You can update it at any time.
• The Project ID must be unique across all Google Cloud projects and is immutable (cannot be changed after it has been set). The Cloud Console auto-generates a unique string; usually you don't care what it is. In most codelabs, you'll need to reference the Project ID (it is typically identified as PROJECT_ID). If you don't like the generated ID, you may generate another random one. Alternatively, you can try your own and see if it's available. It cannot be changed after this step and will remain for the duration of the project.
• For your information, there is a third value, a Project Number which some APIs use. Learn more about all three of these values in the documentation.

Caution: A project ID must be globally unique and cannot be used by anyone else after you've selected it. You are the only user of that ID. Even if a project is deleted, the ID can never be used again

Note: If you're using a Gmail account, you can leave the default location set to No organization. If you're using a Google Workspace account, then choose a location that makes sense for your organization.

2. Next, you'll need to enable billing in the Cloud Console to use Cloud resources/APIs. Running through this codelab shouldn't cost much, if anything at all. To shut down resources so you don't incur billing beyond this tutorial, you can delete the resources you created or delete the whole project. New users of Google Cloud are eligible for the $300 USD Free Trial program.

Start Cloud Shell

While Google Cloud can be operated remotely from your laptop, in this codelab you will be using Google Cloud Shell, a command line environment running in the Cloud.

From the Google Cloud Console, click the Cloud Shell icon on the top right toolbar:

It should only take a few moments to provision and connect to the environment. When it is finished, you should see something like this:

This virtual machine is loaded with all the development tools you'll need. It offers a persistent 5GB home directory, and runs on Google Cloud, greatly enhancing network performance and authentication. All of your work in this codelab can be done within a browser. You do not need to install anything.

Set up gcloud

In Cloud Shell, set your project ID and save it as the PROJECT_ID variable.

Also, set a REGION variable to us-central1. This is the region you will create resources in later.

PROJECT_ID=[YOUR-PROJECT-ID]
REGION=us-central1
gcloud config set core/project $PROJECT_ID

Enable APIs

Enable all necessary services:

gcloud services enable \
  eventarc.googleapis.com \
  eventarcpublishing.googleapis.com \
  workflows.googleapis.com \
  workflowexecutions.googleapis.com

3. Discover the Datadog provider

An Eventarc provider is a service or entity that can emit events directly to Google Cloud which are then routed to your project. Third-party providers, such as Datadog, are non-Google Cloud providers that are integrated with Google Cloud through Eventarc.

In Cloud Shell, run the following command to see the list of Google Cloud and third-party providers:

gcloud eventarc providers list

This lists Google Cloud and third-party providers and the locations they are available in:

NAME: storage.googleapis.com
LOCATION: asia

NAME: cloudaudit.googleapis.com
LOCATION: asia

NAME: pubsub.googleapis.com
LOCATION: asia

...

You can narrow down the list to third-party providers with this command:

gcloud eventarc providers list --filter='eventTypes.type!~^google*'

You should see Datadog in the list:

NAME: datadog
LOCATION: us-central1

You can also describe the Datadog provider to see the events it supports:

gcloud eventarc providers describe datadog --location $REGION

displayName: Datadog
eventTypes:
- type: datadog.v1.alert
name: projects/project-id/locations/us-central1/providers/datadog

4. Setup a channel

You need to set up a channel to integrate your project with a provider. This involves creating a channel, retrieving channel details and sending those details to the provider. Once the provider has initialized the connection to the channel, the provider can start sending events to your project.

Create a channel

You can create a channel for the Datadog provider using gcloud:

CHANNEL_NAME=datadog-channel
gcloud eventarc channels create $CHANNEL_NAME \
   --provider datadog \
   --location $REGION

You can also create it from the Channels section of the Eventarc page in Google Cloud Console:

Retrieve channel details

Once the channel is created, retrieve the details of the channel from gcloud:

gcloud eventarc channels describe $CHANNEL_NAME --location $REGION

The output should be similar to the following:

activationToken: so5g4Kdasda7y2MSasdaGn8njB2
createTime: '2022-03-09T09:53:42.428978603Z'
name: projects/project-id/locations/us-central1/channels/datadog-channel
provider: projects/project-id/locations/us-central1/providers/datadog
pubsubTopic: projects/project-id/topics/eventarc-channel-us-central1-datadog-channel-077
state: PENDING
uid: 183d3323-8cas-4e95-8d72-7d8c8b27cf9e
updateTime: '2022-03-09T09:53:48.290217299Z'

Similarly, you can see the channel from Google Cloud Console:

Note the channel full name and the activation token. You will need to send them to Datadog next.

The channel state indicates the channel's status. It can be one of the following:

• PENDING—The channel has been created successfully and there is an activation token available to create a connection with the provider. To change the state of the channel from PENDING to ACTIVE, the token must be given to the provider and used to connect the channel within 24 hours of the channel's creation.
• ACTIVE—The channel has been successfully connected with the provider. An ACTIVE channel is ready to receive and route events from the provider.
• INACTIVE—The channel cannot receive events nor be reactivated. The provider is either disconnected from this channel or the channel activation token has expired and the provider isn't connected. To re-establish a connection with a provider, you must create a new channel for the provider.

An activation token is a single-use, time-restricted token, used to create a connection between a provider and a subscriber's project. Only a specific provider, selected during the channel creation process can use the token. The token is valid for 24 hours after the channel's creation. After 24 hours, the channel becomes INACTIVE.

Send channel details to the provider

You need to send the following channel details to the Datadog provider:

• Channel name (eg. projects/project-id/locations/us-central1/channels/datadog-channel)
• Activation token (eg. so5g4Kdasda7y2MSasdaGn8njB2)

Login to Datadog, go to integrations page and make sure Google Eventarc integration is installed:

In the configuration section of Google Eventarc, enter the full channel name and the activation token:

You should now see the channel in the list of channels and after a few seconds, you should also see the channel become active in Google Cloud Console:

Now, you're ready to use the channel!

5. Create a workflow

You need a destination in Google Cloud to receive events from the provider. Eventarc supports a number of event destinations such as Cloud Run, Workflows, Kubernetes services. In this case, deploy a workflow to simply log the received events.

Create a workflow-datadog1.yaml file with the following contents:

main:
  params: [event]
  steps:
  - logStep:
      call: sys.log
      args:
        data: ${event}

Note that the workflow is receiving an event as a parameter. This event will come from Datadog monitoring via Eventarc. Once the event is received, the workflow simply logs the received event.

Deploy the workflow:

WORKFLOW_NAME=workflow-datadog1
gcloud workflows deploy $WORKFLOW_NAME \
  --source workflow-datadog1.yaml \
  --location $REGION

The workflow is deployed but it's not running yet. It will be executed by an Eventarc trigger when a Datadog alert is received.

6. Create an Eventarc trigger

You are now ready to connect events from the Datadog provider to Workflows with an Eventarc trigger.

Configure service account

You need a service account with the eventarc.eventReceiver role when creating a trigger. You can either create a dedicated service account or use the default compute service account.

For simplicity, use the default compute service account and grant the eventarc.eventReceiver role:

PROJECT_NUMBER=$(gcloud projects describe $PROJECT_ID --format='value(projectNumber)')

gcloud projects add-iam-policy-binding $PROJECT_ID \
    --member serviceAccount:$[e-mail] \
    --role roles/eventarc.eventReceiver

Create a trigger

Create a trigger with the Datadog channel, event type and also a workflow destination:

gcloud eventarc triggers create datadog-trigger1 \
  --location $REGION \
  --destination-workflow $WORKFLOW_NAME \
  --destination-workflow-location $REGION \
  --channel $CHANNEL_NAME \
  --event-filters type=datadog.v1.alert \
  --service-account $[e-mail]

You can list the triggers to see that the newly created trigger is active:

gcloud eventarc triggers list --location $REGION

NAME: datadog-trigger1
TYPE: datadog.v1.alert
DESTINATION: Workflows: workflow-datadog1
ACTIVE: Yes

7. Create a Datadog monitor

You will now create a Datadog monitor and connect it to Eventarc.

It will be a Hello World type monitor with default values. You will manually trigger it to generate the monitoring alerts which in turn will generate an Eventarc event in Google Cloud.

To create a monitor in Datadog, log in to Datadog. Hover over Monitors in the main menu and click New Monitor in the sub-menu. There are many monitor types. Choose the Metric monitor type.

In the New Monitor page, leave the defaults for steps 1 and 2.

• In step 3, set Alert threshold to 1
• In step 4, set Test monitor for Eventarc as the monitor name and set Notify your team to @eventarc_<your-project-id>_<your-region>_<your-channel-name>

Keep the monitor page open for the next step where you will test the monitor.

8. Test monitor and trigger

To test the Datadog monitor and the Eventarc trigger, you will manually trigger the monitor.

At the bottom of the monitor creation page, click on the Test Notifications button:

Then, click on the Run Test button:

This should simulate the state transition in the monitor and trigger an Eventarc event.

Check the workflow-datadog1 workflow. You should see that there's a new execution:

Check the details of the execution. You should see the Datadog event type datadog.v1.alert generated from the monitoring alert in the input of the workflow and also in the logs:

9. Congratulations

Congratulations, you finished the codelab! You can continue to the second codelab to learn how to respond to Datadog monitoring alerts with Workflows.

What we've covered

• How to discover the Datadog provider.
• How to setup a channel to the Datadog provider.
• How to create a workflow to log events.
• How to create an Eventarc trigger with the channel.
• How to create a Datadog monitor.
• How to test the Datadog monitor, Eventarc trigger and the workflow.