Empower Your Gmail Inbox with Google Cloud Functions

1. Introduction

Billions of businesses and individuals use Gmail and other G Suite services to communicate and process data. Google offers G Suite APIs to help you access information in these services programmatically, and you can use the APIs to easily automate your daily workflow. In this lab, you will build a powerful Gmail extension that automatically categorizes emails in incoming messages and saves those categories in a Google sheet. This extension will use G Suite's RESTful APIs, Google Cloud Functions, and other Google Cloud Platform services.

What you will build

In this lab, you will build and deploy a few Cloud Functions connected to G Suite APIs and other Google Cloud Platform services. These functions will:

  • Authorize secure access to your Gmail and Google Sheets data
  • Extract images attached to any incoming mail
  • Categorize those images using the Cloud Vision API
  • Write those categories, the sender's address, and the name of the attachment to a Google Sheet

What you will learn

  • Basics of G Suite RESTful APIs
  • Basics of Google Cloud Functions and other Google Cloud Platform services
  • How to access Gmail programmatically using Google Cloud Functions

What you will need

  • A Google account with Gmail and Google Sheets access. If you do not have one, create one here.
  • Basic knowledge of Javascript/Node.js.

2. First things first

Enable the APIs

In this lab you will use the following Google products/services:

  • Google Cloud Functions
  • Google Cloud Pub/Sub
  • Google Cloud Vision API
  • Google Cloud Datastore
  • Gmail API
  • Google Sheets API

Google Cloud Functions

Google Cloud Functions is Google's Serverless Functions-as-a-Service platform that allows you to run individual snippets of code (‘functions') in a simple, scalable manner.

To enable Google Cloud Functions, click the hamburger menu on the top left of your screen to open the left navigation sidebar:

f457988e33594bb6.png

Find Cloud Functions in the navigation menu and click it. Click Enable API to enable Google Cloud Functions in your project.

Google Cloud Pub/Sub

Google Cloud Pub/Sub is a simple and scalable foundation for data streaming and event delivery. In this lab it serves as the courier between Gmail and Google Cloud Functions.

To enable Google Cloud Pub/Sub, open the left navigation sidebar, find Pub/Sub, and click it. Click Enable API to enable Google Cloud Pub/Sub in your project.

Google Cloud Datastore

Google Cloud Datastore is a Serverless database that's scalable and distributed.

To enable Google Cloud Datastore, in the left navigation sidebar, find Datastore and click it. Click Select Datastore Mode in the new page.

98012c91fd4080d4.png

You can use any database location for this lab. Click Create Database to enable Google Cloud Datastore; it may take a few minutes to complete.

Google Cloud Vision

Google Cloud Vision API is a powerful machine learning service using pre-trained models to derive insights from your images.

See the instructions below for information on how to enable the Google Cloud Vision API.

Enabling the Gmail API, Google Sheets API, and Google Cloud Vision API

Once again, open the left navigation sidebar, and find APIs & Services. Click Library. In the Search for APIs & Services field, type in Gmail. In the search results, select Gmail API and click Enable.

Go back to the API Library page. Search for Google Sheets API and enable it.

Repeat the process. Search for Cloud Vision API and enable it.

Open Google Cloud Shell

In this lab, you will use Google Cloud Shell to perform most operations. Cloud Shell provides you command-line access to your Google Cloud Platform resources directly from your browser, allowing you to manage them without using a local machine.

To open Google Cloud Shell, click the Activate Cloud Shell button on the top blue horizontal bar:

fd5c2925ca9cdfdd.png

A new panel will appear at the bottom of the screen:

34f498402e910802.png

Click the Launch code editor button to start the Cloud Shell Code Editor:

10f8631ef48bed22.png

Cloud Shell Code Editor will open in a new window.

Download the code

Run the command below in your Cloud Shell to clone the project:

git clone https://github.com/googlecodelabs/gcf-gmail-codelab.git

cd gcf-gmail-codelab

You should see a new folder, gcf-gmail-codelab, appearing in the Cloud Shell Code Editor.

3. Architectural overview

Below is the workflow of this lab:

79c5d3e43f674b33.png

  1. User sets up Gmail push notifications: every time a new message arrives at inbox, Gmail will send a notification to Cloud Pub/Sub.
  2. Cloud Pub/Sub delivers the new message notification to Google Cloud Functions.
  3. Upon arrival of the new message notification, a Cloud Functions instance connects to Gmail and retrieves the new message.
  4. If the mail has an image as an attachment, the Cloud Functions instance calls Cloud Vision API to analyze the attachment.
  5. The Cloud Functions instance updates a Google Sheet of your choice, specifying who sends the message and where to download the attachment.

4. Authorize access to Gmail

Before setting up a Cloud Function to automatically read your emails, you must authorize its access to Gmail. You will need to register an OAuth client with Google and create an associated client ID.

Register an OAuth client

In the left navigation menu of Google Cloud Console, find APIs & Services. Click OAuth consent screen.

91b2a3bac30bb2c5.png

Type in a name in the Application name field, such as GCF + Gmail Codelab. Leave other settings untouched, scroll down the page and click Save.

Create an associated client ID

Switch to the Credentials tab. Click Create Credentials and choose OAuth client ID. Pick the Web application type, give it a name (you may use GCF + Gmail Codelab again here), and click Create. Leave the Restrictions fields empty for now.

Write down the client ID and the client secret returned in the pop-up window. You can click the name of your client on the page to view these values again:

1160d8027ea52d90.png

Perform the authorization process

In the sample code, auth/index.js specifies two Cloud Functions, auth_init and auth_callback, that work together to perform the authorization process, using the client ID and the client secret you just created.

To inspect the code, open auth/index.js in Cloud Shell Code Editor.

The authorization process returns two types of tokens: access tokens and refresh tokens.

  • Access tokens are short-lived proofs of identity that grant anyone in possession of them scoped access to your data; auth_callback saves them in Cloud Datastore.
  • Refresh tokens are used to obtain new access tokens, and are significantly longer-lived.

Typically, they are either encrypted and/or stored separately from access tokens.

Edit auth/env_vars.yaml in the Cloud Shell Code Editor. Replace YOUR-GOOGLE-CLIENT-ID, and YOUR-GOOGLE-CLIENT-SECRET with values of your own. See the earlier step for more information. Leave the values of YOUR-GOOGLE-CLIENT-CALLBACK-URL and YOUR-PUBSUB-TOPIC unchanged for now.

a2b4853c39a78bc6.png

After editing auth/env_vars.yaml, run the following command in Cloud Shell to deploy the Cloud Functions:

cd ~
cd gcf-gmail-codelab/auth

# Deploy Cloud Function auth_init
gcloud functions deploy auth_init --runtime=nodejs8 --trigger-http --env-vars-file=env_vars.yaml

# Deploy Cloud Function auth_callback
gcloud functions deploy auth_callback --runtime=nodejs8 --trigger-http --env-vars-file=env_vars.yaml

It may take a few minutes to deploy the Cloud Functions. If prompted, give Cloud SDK the permission to install beta commands.

Next, go to Google Cloud Console, and click Cloud Functions in the left navigation menu. Click auth_callback in the list of Cloud Functions, and switch to the Trigger tab.

cb094bd341f9b299.png

45678a327c80e0f1.png

Copy the URL on the page. Go back to the Cloud Functions page, click auth_init in the list of Cloud Functions. In the General tab, click Edit. Click Environmental variables, networking, timeouts and more, and replace the value of GOOGLE_CALLBACK_URL with the URL you just copied.

939ca3bd38047282.png

Click Deploy to apply the changes. Repeat the process and update auth_callback as well.

Lastly, open the left navigation menu and click APIs & Services > Domain verification. To add an authorized domain, click Add domain. For example, If the URL you copied earlier looks like

https://us-central1-my-project.cloudfunctions.net/auth_callback

You should add the following as an authorized domain:

us-central1-my-project.cloudfunctions.net

Press Add domain to confirm.

4348748f232ceb87.png

Go back to the Credentials page. Click the name of your OAuth client and add the URL you copied as an Authorized redirect URI. Press Enter to confirm.

Remove the /auth_callback part from the URL and add the rest as an Authorized Javascript origin. For example, if your URL looks like

https://us-central1-my-project.cloudfunctions.net/auth_callback

You should add the following as the origin:

https://us-central1-my-project.cloudfunctions.net/

159bad719432582c.png

Press Enter to confirm and click Save to apply the changes.

5. Set up Gmail push notifications

If the authorization process succeeds, auth_callback will automatically call the Gmail API to set up push notifications.

In order to receive Gmail push notifications, you must create a Pub/Sub topic. Any subscriber to the topic will receive incoming message notifications automatically as they arrive from Gmail.

To create a Pub/Sub topic, go to Google Cloud Console and click Pub/Sub > Topics in the left navigation menu. Click Create Topic. Type in the name of the topic, such as gmail-watch, and click Create. Additionally, you must give Gmail permission to send messages to your Pub/Sub topic: click the context menu of the topic you just create (three vertical dots), and choose Permissions; click Add members, specify gmail-api-push@system.gserviceaccount.com as a new member, and give it the role of Pub/Sub > Pub/Sub Publisher; lastly, click Save to apply the changes.

Update Cloud Function auth_callback to specify which Pub/Sub topic to use. Click Cloud Functions in the left navigation menu, and select auth_callback in the list of Cloud Functions. In the General tab, click Edit. Click More, and replace the value of PUBSUB_TOPIC with the name of the Pub/Sub topic you just created. Click Save to apply the changes.

Now you are ready to authorize and set up Gmail push notifications. Wait until new changes finalize, then go back to the Cloud Functions page, select auth_init in the list of Cloud Functions, and switch to the Trigger tab. Click the URL, and you will be redirected to the Sign-in with Google page:

348ab0a7e0c9cd03.png

Sign in with a Gmail account you own. Any new message arriving in the inbox of the account will trigger a push notification. After signing-in, you will see the page below:

cfdad62fd02de004.png

Click Allow to authorize access. auth_callback will complete the authorization process, save the access tokens, and set up Gmail push notifications for you. You should see the message Successfully set up Gmail push notifications in your browser when this process completes.

This codelab uses the @google-cloud/express-oauth2-handlers package to automate the authorization workflow for you. For more information, see its repository on GitHub.

6. Process incoming messages

As we mentioned earlier, any subscriber to the Pub/Sub topic you created will receive notifications when new messages arrive in your inbox. pubsub/index.js specifies a Cloud Function, watchGmailMessages, that, once deployed as a subscriber to the topic, will read the new messages, categorize attached images, and export those categories to a Google Sheet.

To inspect the code, open pubsub/index.js in Cloud Shell Code Editor.

Retrieving messages

A Gmail push notification includes the email address the notification is associated with, and a history ID. For simplicity reasons, in this codelab you will simply ask Gmail API for the latest message when a push notification arrives; for a better result, use history ID to look up messages instead.

// Look up the most recent message.
const listMessagesRes = await gmail.users.messages.list({
  userId: email,
  maxResults: 1
});
const messageId = listMessagesRes.messages[0].id;

// Get the message using the message ID.
const message = await gmail.users.messages.get({
  userId: email,
  id: messageId
});

return message;

Analyze image attachments

If the message has an image attachment, watchGmailMessages will call the Cloud Vision API to annotate the image. In this codelab, you will ask Cloud Vision API to classify the image and return a number of image tags; for example, if provided with an image of a blue sky, Cloud Vision API may return the tags blue, sky,and nature.

watchGmailMessages uses the Cloud Vision API Library for Node.js to call Cloud Vision API:

// Tag the attachment using Cloud Vision API
const analyzeAttachment = async (data, filename) => {
  var topLabels = ['', '', ''];
  if (filename.endsWith('.png') || filename.endsWith('.jpg')) {
    const [analysis] = await visionClient.labelDetection({
      image: {
        content: Buffer.from(data, 'base64')
      }
    });
    const labels = analysis.labelAnnotations;
    topLabels = labels.map(x => x.description).slice(0, 3);
  }

  return topLabels;
};

Update Google Sheet

watchGmailMessages exports the results of this analysis to a Google Sheet. It includes the name of the sender, the name of the attachment, and tags of image attachments (if any).

First, create a Google Sheet. Open Google Sheets and click the Blank template under Start a new spreadsheet. Copy the ID of your sheet. For example, if the address in your browser looks like this:

https://docs.google.com/spreadsheets/d/abcdefghij01234567890/edit#gid=0

The ID of your spreadsheet is abcdefghij01234567890. In Cloud Shell Code Editor, update gcf-gmail-codelab/pubsub/env_vars.yaml, and replace YOUR-GOOGLE-SHEET-ID with the value of your own.

watchGmailMessages connects with Google Sheets API to append information:

const updateReferenceSheet = async (from, filename, topLabels) => {
  await googleSheets.spreadsheets.values.append({
    spreadsheetId: SHEET,
    range: SHEET_RANGE,
    valueInputOption: 'USER_ENTERED',
    requestBody: {
      range: SHEET_RANGE,
      majorDimension: 'ROWS',
      values: [
        [from, filename].concat(topLabels)
      ]
    }
  });
};

One last step

In Cloud Shell Code Editor, open gcf-gmail-codelab/pubsub/env_vars.yaml and replace YOUR-GOOGLE-CLIENT-ID, YOUR-GOOGLE-CLIENT-SECRET, and YOUR-GOOGLE-CALLBACK-URL with values of your own. You can find these values in the Google Cloud Console: open Cloud Functions in the left navigation menu, select auth_init in the list of Cloud Functions, and look up for the Environment variables section.

Deploy the code

Run the command below to deploy the Cloud Function:

cd ~

cd gcf-gmail-codelab/pubsub

gcloud functions deploy watchGmailMessages --runtime=nodejs8 --trigger-topic=gmail-watch --env-vars-file=env_vars.yaml

If you named your Cloud Pub/Sub topic something other than gmail-watch, replace gmail-watch in the above command with the name of your topic. It may take a few seconds to deploy the Cloud Function.

7. Try it out

Congratulations, you're done! Send yourself an email with an image attachment. In a few seconds you will see the Google Sheet you created update automatically with the information you provide.