In this code lab, you build on the previous lab and add a thumbnail service. The thumbnail service is a web container that takes big pictures and creates thumbnails out of them.

As the picture is uploaded to Cloud Storage, a notification is sent via Cloud Pub/Sub to a Cloud Run web container, which then resizes images and saves them back in another bucket in Cloud Storage.

What you'll learn

Self-paced environment setup

If you don't already have a Google Account (Gmail or G Suite), you must create one. Sign-in to Google Cloud Platform console (console.cloud.google.com) and create a new project (or reuse an existing one):

Screenshot from 2016-02-10 12:45:26.png

Remember the project ID, a unique name across all Google Cloud projects (the name above has already been taken and will not work for you, sorry!). It will be referred to later in this codelab as PROJECT_ID.

Next, you'll need to enable billing in the Cloud Console in order to use Google Cloud resources.

Running through this codelab shouldn't cost you more than a few dollars, but it could be more if you decide to use more resources or if you leave them running (see "cleanup" section at the end of this document).

New users of Google Cloud Platform are eligible for a $300 free trial.

Start Cloud Shell

While Google Cloud can be operated remotely from your laptop, in this codelab you will be using Google Cloud Shell, a command line environment running in the Cloud.

From the GCP Console click the Cloud Shell icon on the top right toolbar:

It should only take a few moments to provision and connect to the environment. When it is finished, you should see something like this:

This virtual machine is loaded with all the development tools you'll need. It offers a persistent 5GB home directory, and runs on Google Cloud, greatly enhancing network performance and authentication. All of your work in this lab can be done with simply a browser.

In this lab, you will need Cloud Build to build container images and Cloud Run to deploy the container.

Enable both APIs from Cloud Shell:

gcloud services enable cloudbuild.googleapis.com run.googleapis.com

You should see the operation to finish successfully:

Operation "operations/acf.5c5ef4f6-f734-455d-b2f0-ee70b5a17322" finished successfully.

You will store thumbnails of the uploaded pictures in another bucket. Let's use gsutil to create the second bucket.

Inside Cloud Shell, set a variable for the unique bucket name. Cloud Shell already has GOOGLE_CLOUD_PROJECT set to your unique project id. You can append that to the bucket name.

For example:

BUCKET_THUMBNAILS=thumbnails-${GOOGLE_CLOUD_PROJECT}

Create a standard multi-region zone in Europe:

gsutil mb -l EU gs://${BUCKET_THUMBNAILS}

Ensure uniform bucket level access:

gsutil uniformbucketlevelaccess set on gs://${BUCKET_THUMBNAILS}

Make the bucket public:

gsutil iam ch allUsers:objectViewer gs://${BUCKET_THUMBNAILS}

In the end, you should have a public bucket:

Clone the code, if you haven't already:

git clone https://github.com/GoogleCloudPlatform/serverless-photosharing-workshop

You will have the following file layout for the service:

services
 |
 ├── thumbnails
      |
      ├── Dockerfile
      ├── index.js
      ├── package.json

Inside the thumbnails folder,, you have 3 files:

Dependencies

The package.json file defines the needed library dependencies:

{
  "name": "thumbnail_service",
  "version": "0.0.1",
  "main": "index.js",
  "scripts": {
    "start": "node index.js"
  },
  "dependencies": {
    "@google-cloud/storage": "^4.0.0",
    "body-parser": "^1.19.0",
    "express": "^4.16.4",
    "bluebird": "^3.5.0",
    "imagemagick": "^0.1.3"
  }
}

Cloud Storage library is used to read and save image files within Cloud Storage. Express is a JavaScript / Node web framework. The body-parser module is used to parse incoming requests easily. Bluebird is used for handling promises, and imagemagick is a library for manipulating images.

Dockerfile

Dockerfile defines the container image for the application:

FROM node:10-slim

# installing Imagemagick
RUN set -ex; \
  apt-get -y update; \
  apt-get -y install imagemagick; \
  rm -rf /var/lib/apt/lists/*; \
  mkdir /tmp/original; \
  mkdir /tmp/thumbnail;


WORKDIR /picadaily/services/thumbnails
COPY package*.json ./
RUN npm install --production
COPY . ./
CMD [ "npm", "start" ]

The base image is Node 10 and the imagemagick library is used for image manipulation. Some temporary directories are created for holding original and thumbnail picture files. Then NPM modules needed by our code are installed before starting the code with npm start.

index.js

Let's explore the code in pieces, so that we can better understand what this program is doing.

const express = require('express');
const bodyParser = require('body-parser');
const im = require('imagemagick');
const Promise = require("bluebird");
const path = require('path');
const {Storage} = require('@google-cloud/storage');
const storage = new Storage();

const app = express();
app.use(bodyParser.json());

We are first requiring the needed dependencies, and create our Express web application, as well as indicating that we want to use the JSON body parser, as incoming requests are actually just JSON payloads sent via a POST request to our application.

app.post('/', async (req, res) => {
    try {
        // ...
    } catch (err) {
        console.log(`Error: creating the thumbnail: ${err}`);
        console.error(err);
        res.status(500).send(err);
    }
});

We are receiving those incoming payloads on the / base URL, and we are wrapping our code with some error logic handling, to have better information of why something may be failing in our code by looking at the logs that will be visible from the Stackdriver Logging interface in the Google Cloud web console.

const pubSubMessage = req.body;
console.log(`PubSub message: ${JSON.stringify(pubSubMessage)}`);

const fileEvent = JSON.parse(Buffer.from(pubSubMessage.message.data, 'base64').toString().trim());
console.log(`Received thumbnail request for file ${fileEvent.name} from bucket ${fileEvent.bucket}`);

On the Cloud Run platform, Pub/Sub messages are sent via HTTP POST requests, as JSON payloads of the form:

{
  "message": {
    "attributes": {
      "bucketId": "uploaded-pictures",
      "eventTime": "2020-02-27T09:22:43.255225Z",
      "eventType": "OBJECT_FINALIZE",
      "notificationConfig": "projects/_/buckets/uploaded-pictures/notificationConfigs/28",
      "objectGeneration": "1582795363255481",
      "objectId": "IMG_20200213_181159.jpg",
      "payloadFormat": "JSON_API_V1"
    },
    "data": "ewogICJraW5kIjogInN0b3JhZ2Ujb2JqZWN...FQUU9Igp9Cg==",
    "messageId": "1014308302773399",
    "message_id": "1014308302773399",
    "publishTime": "2020-02-27T09:22:43.973Z",
    "publish_time": "2020-02-27T09:22:43.973Z"
  },
  "subscription": "projects/serverless-picadaily/subscriptions/gcs-events-subscription"
}

But what is really interesting in this JSON document is actually what is contained in the message.data attribute, which is just a string but that encodes the actual payload into Base 64. That's why our code above is decoded the Base 64 content of this attribute. That data attribute once decoded contains another JSON document that represent the Cloud Storage event details, which, among other metadata, indicates the file name and the bucket name.

{
  "kind": "storage#object",
  "id": "uploaded-pictures/IMG_20200213_181159.jpg/1582795363255481",
  "selfLink": "https://www.googleapis.com/storage/v1/b/uploaded-pictures/o/IMG_20200213_181159.jpg",
  "name": "IMG_20200213_181159.jpg",
  "bucket": "uploaded-pictures",
  "generation": "1582795363255481",
  "metageneration": "1",
  "contentType": "image/jpeg",
  "timeCreated": "2020-02-27T09:22:43.255Z",
  "updated": "2020-02-27T09:22:43.255Z",
  "storageClass": "STANDARD",
  "timeStorageClassUpdated": "2020-02-27T09:22:43.255Z",
  "size": "4944335",
  "md5Hash": "QzBIoPJBV2EvqB1EVk1riw==",
  "mediaLink": "https://www.googleapis.com/download/storage/v1/b/uploaded-pictures/o/IMG_20200213_181159.jpg?generation=1582795363255481&alt=media",
  "crc32c": "hQ3uHg==",
  "etag": "CLmJhJu08ecCEAE="
}

We're interested in the image and bucket names, as our code is going to fetch that image from the bucket for its thumbnail treatment:

const bucket = storage.bucket(fileEvent.bucket);
const thumbBucket = storage.bucket(process.env.BUCKET_THUMBNAILS);

const originalFile = path.resolve('/tmp/original', fileEvent.name);
const thumbFile = path.resolve('/tmp/thumbnail', fileEvent.name);

await bucket.file(fileEvent.name).download({
    destination: originalFile
});
console.log(`Downloaded picture into ${originalFile}`);

Be sure to update the name of the bucket with the one you have chosen before.

We have the origin bucket whose file creation triggered our Cloud Run service, and the destination bucket where we'll store the resulting image. We are using the path built-in API to do local file handling, as the imagemagick library will be creating the thumbnail locally in the /tmp temporary directory. We await for an asynchronous call to download the uploaded image file.

const resizeCrop = Promise.promisify(im.crop);
await resizeCrop({
        srcPath: originalFile,
        dstPath: thumbFile,
        width: 400,
        height: 400         
});
console.log(`Created local thumbnail in ${thumbFile}`);

The imagemagick module is not very async / await friendly, so we are wrapping it up within a Javascript promise (provided by the Bluebird module). Then we're calling the asynchronous resizing / cropping function we created with the parameters for the source and destination files, as well as the dimensions of the thumbnail we want to create.

await thumbBucket.upload(thumbFile);
console.log(`Uploaded thumbnail to Cloud Storage bucket ${process.env.BUCKET_THUMBNAILS}`);

res.status(204).send(`${fileEvent.name} processed`);

Once the thumbnail is created, we can upload it back to the destination Cloud Storage bucket, and then reply to the HTTP POST request that the file was properly processed.

const PORT = process.env.PORT || 8080;

app.listen(PORT, () => {
    console.log(`Started thumbnail generator on port ${PORT}`);
});

At the end of our source file, we have the instructions to have Express actually start our web application on the 8080 default port.

Test the code locally to make sure it works before deploying to cloud.

Inside thumbnails folder, install npm dependencies and start the server:

npm install; npm start

If everything went well, it should start the server on port 8080:

Started thumbnail generator on port 8080

Cloud Run runs containers but you first need to build the container image (defined in Dockerfile). Google Cloud Build can be used to build container images and then host to Google Container Registry.

Inside thumbnails folder where Dockerfile is, issue the following command to build the container image:

gcloud builds submit --tag gcr.io/${GOOGLE_CLOUD_PROJECT}/thumbnail_service

After a minute or two, the build should succeed:

The Cloud Build "history" section should show the successful build as well:

Clicking on the build ID to get the details view, in the "build artifacts" tab you should see that the container image has been uploaded to the Cloud Registry (GCR):

If you wish, you can double check that the container image runs locally in Cloud Shell:

docker run -p 8080:8080 gcr.io/${GOOGLE_CLOUD_PROJECT}/thumbnail_service

It should start the server on port 8080 in the container:

Started thumbnail generator on port 8080

Run the following command to deploy the container image on Cloud Run:

gcloud run deploy thumbnail-service \
    --image gcr.io/${GOOGLE_CLOUD_PROJECT}/thumbnail_service \
    --platform=managed \
    --region=europe-west1 \
    --no-allow-unauthenticated \
    --memory=1Gi \
    --update-env-vars BUCKET_THUMBNAILS=${BUCKET_THUMBNAILS}

Note the --no-allow-unauthenticated flag. This makes the Cloud Run service an internal service that will only be triggered by specific service accounts.

If the deployment is successful, you should see the following output:

The service is ready, but you still need to make Cloud Storage push events to the newly created Cloud Run service. Cloud Storage can send file creation events via Cloud Pub/Sub but there are a few steps to get this working.

First, set some variables we'll need in the next steps. TOPIC_NAME is for the Pub/Sub topic as the communication pipeline and the rest are variables we need along the way:

TOPIC_NAME=gcs-events
PROJECT_NUMBER="$(gcloud projects list --filter=${GOOGLE_CLOUD_PROJECT} --format='value(PROJECT_NUMBER)')"
SERVICE_NAME=thumbnail-service
SERVICE_URL="$(gcloud run services list --platform managed --filter=${SERVICE_NAME} --format='value(URL)')"
SERVICE_ACCOUNT=${TOPIC_NAME}-sa
BUCKET_PICTURES=uploaded-pictures-${GOOGLE_CLOUD_PROJECT}

Create a Pub/Sub topic as the communication pipeline:

gcloud pubsub topics create ${TOPIC_NAME}

Create Pub/Sub notifications when files are stored in the bucket:

gsutil notification create -t ${TOPIC_NAME} -f json gs://${BUCKET_PICTURES}

Create a service account to represent the Pub/Sub subscription identity:

gcloud iam service-accounts create ${SERVICE_ACCOUNT} \
     --display-name "Cloud Run Pub/Sub Invoker"

Give the service account permission to invoke the service:

gcloud run services add-iam-policy-binding ${SERVICE_NAME} \
   --member=serviceAccount:${SERVICE_ACCOUNT}@${GOOGLE_CLOUD_PROJECT}.iam.gserviceaccount.com \
   --role=roles/run.invoker \
   --platform managed

Enable Pub/Sub to create authentication tokens in our project:

gcloud projects add-iam-policy-binding ${GOOGLE_CLOUD_PROJECT} \
     --member=serviceAccount:service-${PROJECT_NUMBER}@gcp-sa-pubsub.iam.gserviceaccount.com \
     --role=roles/iam.serviceAccountTokenCreator

It can take a few minutes for the IAM changes to propagate.

Finally, create a Pub/Sub subscription with the service account:

gcloud pubsub subscriptions create ${TOPIC_NAME}-subscription --topic ${TOPIC_NAME} \
   --push-endpoint=${SERVICE_URL} \
   --push-auth-service-account=${SERVICE_ACCOUNT}@${GOOGLE_CLOUD_PROJECT}.iam.gserviceaccount.com

To test the setup is working, upload a new picture to the uploaded-pictures bucket and check in the thumbnails bucket that new resized pictures appear as expected.

You can also double check the logs to see the logging messages appear, as the various steps of the Cloud Run service are going through:

Everything is now in place:

What we've covered

Next Steps