Pic-a-daily: Lab 3—Create a collage of most recent pictures

1. Overview

In this code lab, you create a new Cloud Run service, collage service, that will be triggered by Cloud Scheduler at a regular interval of time. The service fetches the latest pictures uploaded and creates a collage of those pictures: it finds the list of recent pictures in Cloud Firestore, and then downloads the actual picture files from Cloud Storage.

What you'll learn

• Cloud Run
• Cloud Scheduler
• Cloud Storage
• Cloud Firestore

2. Setup and Requirements

Self-paced environment setup

1. Sign-in to the Google Cloud Console and create a new project or reuse an existing one. If you don't already have a Gmail or Google Workspace account, you must create one.

• The Project name is the display name for this project's participants. It is a character string not used by Google APIs, and you can update it at any time.
• The Project ID must be unique across all Google Cloud projects and is immutable (cannot be changed after it has been set). The Cloud Console auto-generates a unique string; usually you don't care what it is. In most codelabs, you'll need to reference the Project ID (and it is typically identified as PROJECT_ID), so if you don't like it, generate another random one, or, you can try your own and see if it's available. Then it's "frozen" after the project is created.
• There is a third value, a Project Number which some APIs use. Learn more about all three of these values in the documentation.

Caution: A project ID must be globally unique and cannot be used by anyone else after you've selected it. You are the only user of that ID. If a project is deleted, that ID can never be used again.

Note: If you're using a Gmail account, you can leave the default location set to No organization. If you're using a Google Workspace account, then choose a location that makes sense for your organization.

2. Next, you'll need to enable billing in the Cloud Console in order to use Cloud resources/APIs. Running through this codelab shouldn't cost much, if anything at all. To shut down resources so you don't incur billing beyond this tutorial, follow any "clean-up" instructions found at the end of the codelab. New users of Google Cloud are eligible for the $300 USD Free Trial program.

Start Cloud Shell

While Google Cloud can be operated remotely from your laptop, in this codelab you will be using Google Cloud Shell, a command line environment running in the Cloud.

From the GCP Console click the Cloud Shell icon on the top right toolbar:

It should only take a few moments to provision and connect to the environment. When it is finished, you should see something like this:

This virtual machine is loaded with all the development tools you'll need. It offers a persistent 5GB home directory, and runs on Google Cloud, greatly enhancing network performance and authentication. All of your work in this lab can be done with simply a browser.

3. Enable APIs

You will need a Cloud Scheduler to trigger the Cloud Run service on a regular interval. Make sure it is enabled:

gcloud services enable cloudscheduler.googleapis.com

You should see the operation to finish successfully:

Operation "operations/acf.5c5ef4f6-f734-455d-b2f0-ee70b5a17322" finished successfully.

4. Clone the code

Clone the code, if you haven't already in the previous code lab:

git clone https://github.com/GoogleCloudPlatform/serverless-photosharing-workshop

You can then go to the directory containing the service:

cd serverless-photosharing-workshop/services/collage/nodejs

You will have the following file layout for the service:

services
 |
 ├── collage
      |
      ├── nodejs
           |
           ├── Dockerfile
           ├── index.js
           ├── package.json

Inside the folder,, you have 3 files:

• index.js contains the Node.js code
• package.json defines the library dependencies
• Dockerfile defines the container image

5. Explore the code

Dependencies

The package.json file defines the needed library dependencies:

{
  "name": "collage_service",
  "version": "0.0.1",
  "main": "index.js",
  "scripts": {
    "start": "node index.js"
  },
  "dependencies": {
    "bluebird": "^3.7.2",
    "express": "^4.17.1",
    "imagemagick": "^0.1.3",
    "@google-cloud/firestore": "^4.9.9",
    "@google-cloud/storage": "^5.8.3"
  }
}

We depend on the Cloud Storage library to read and save image files within Cloud Storage. We declare a dependency on Cloud Firestore, to fetch picture metadata that we stored previously. Express is a JavaScript / Node web framework. Bluebird is used for handling promises, and imagemagick is a library for manipulating images.

Dockerfile

Dockerfile defines the container image for the application:

FROM node:14-slim

# installing Imagemagick
RUN set -ex; \
  apt-get -y update; \
  apt-get -y install imagemagick; \
  rm -rf /var/lib/apt/lists/*

WORKDIR /picadaily/services/collage
COPY package*.json ./
RUN npm install --production
COPY . .
CMD [ "npm", "start" ]

We're using a light Node 14 base image. We're installing the imagemagick library. Then we're installing the NPM modules needed by our code, and we run our node code with npm start.

index.js

Let's have a closer look at our index.js code:

const express = require('express');
const imageMagick = require('imagemagick');
const Promise = require("bluebird");
const path = require('path');
const {Storage} = require('@google-cloud/storage');
const Firestore = require('@google-cloud/firestore');

We require the various dependencies needed for our program to run: Express is the Node web framework we will be using, ImageMagick the library for doing image manipulation, Bluebird is a library for handling JavaScript promises, Path is used for dealing with files and directories paths, and then Storage and Firestore are for working respectively with Google Cloud Storage (our buckets of images), and the Cloud Firestore datastore.

const app = express();

app.get('/', async (req, res) => {
    try {
        console.log('Collage request');

        /* ... */

    } catch (err) {
        console.log(`Error: creating the collage: ${err}`);
        console.error(err);
        res.status(500).send(err);
    }
});

Above, we have the structure of our Node handler: our app responds to HTTP GET requests. And we're doing a bit of error handling in case something goes wrong. Let's now have a look at what is inside this structure.

const thumbnailFiles = [];
const pictureStore = new Firestore().collection('pictures');
const snapshot = await pictureStore
    .where('thumbnail', '==', true)
    .orderBy('created', 'desc')
    .limit(4).get();

if (snapshot.empty) {
    console.log('Empty collection, no collage to make');
    res.status(204).send("No collage created.");
} else {

    /* ... */

}

Our collage service will need at least four pictures (whose thumbnails have been generated), so be sure to upload 4 pictures first.

We retrieve the 4 latest pictures uploaded by our users, from the metadata stored in Cloud Firerstore. We check if the resulting collection is empty or not, and then proceed further in the else branch of our code.

Let's collect the list of file names:

snapshot.forEach(doc => {
    thumbnailFiles.push(doc.id);
});
console.log(`Picture file names: ${JSON.stringify(thumbnailFiles)}`);

We are going to download each of those files from the thumbnail bucket, whose name is coming from an environment variable that we set at deployment time:

const thumbBucket = storage.bucket(process.env.BUCKET_THUMBNAILS);

await Promise.all(thumbnailFiles.map(async fileName => {
    const filePath = path.resolve('/tmp', fileName);
    await thumbBucket.file(fileName).download({
        destination: filePath
    });
}));
console.log('Downloaded all thumbnails');

Once the latest thumbnails are uploaded, we're going to use the ImageMagick library to create a 4x4 grid of those thumbnail pictures. We use the Bluebird library and its Promise implementation to transform the callback-driven code into async / await friendly code, then we await on the promise that is making the image collage:

const collagePath = path.resolve('/tmp', 'collage.png');

const thumbnailPaths = thumbnailFiles.map(f => path.resolve('/tmp', f));
const convert = Promise.promisify(im.convert);
await convert([
    '(', ...thumbnailPaths.slice(0, 2), '+append', ')',
    '(', ...thumbnailPaths.slice(2), '+append', ')',
    '-size', '400x400', 'xc:none', '-background', 'none',  '-append',
    collagePath]);
console.log("Created local collage picture");

As the collage picture has been saved to disk locally in the temporary folder, we now need to upload it to Cloud Storage, and then return a successful response (status code 2xx):

await thumbBucket.upload(collagePath);
console.log("Uploaded collage to Cloud Storage bucket ${process.env.BUCKET_THUMBNAILS}");

res.status(204).send("Collage created.");

Now time to make our Node script listen to incoming requests:

const PORT = process.env.PORT || 8080;

app.listen(PORT, () => {
    console.log(`Started collage service on port ${PORT}`);
});

At the end of our source file, we have the instructions to have Express actually start our web application on the 8080 default port.

6. Test locally

Test the code locally to make sure it works before deploying to the cloud.

Inside collage/nodejs folder, install npm dependencies and start the server:

npm install; npm start

If everything went well, it should start the server on port 8080:

Started collage service on port 8080

Use CTRL-C to exit.

7. Build and deploy to Cloud Run

Before deploying to Cloud Run, set the Cloud Run region to one of the supported regions and platform to managed:

gcloud config set run/region europe-west1
gcloud config set run/platform managed

You can check that the configuration is set:

gcloud config list

...
[run]
platform = managed
region = europe-west1

Instead of building and publishing the container image using Cloud Build manually, you can also rely on Cloud Run to build the container image for you using Google Cloud Buildpacks.

Run the following command to build the container image:

BUCKET_THUMBNAILS=thumbnails-$GOOGLE_CLOUD_PROJECT
SERVICE_NAME=collage-service
gcloud run deploy $SERVICE_NAME \
    --source . \
    --no-allow-unauthenticated \
    --update-env-vars BUCKET_THUMBNAILS=$BUCKET_THUMBNAILS

Note the –-source flag. This is the source based deployment in Cloud Run. If a Dockerfile is present in the source code directory, the uploaded source code is built using that Dockerfile. If no Dockerfile is present in the source code directory, Google Cloud buildpacks automatically detects the language you are using and fetches the dependencies of the code to make a production-ready container image, using a secure base image managed by Google. This flags Cloud Run to use Google Cloud Buildpacks to build the container image defined in Dockerfile.

Note also that source based deployment uses Artifact Registry to store built containers. Artifact Registry is a modern version of Google Container Registry. The CLI will prompt to enable the API if it's not enabled already in the project and will create a repository with the name cloud-run-source-deploy in the region you are deploying to.

The --no-allow-unauthenticated flag makes the Cloud Run service an internal service that will only be triggered by specific service accounts.

8. Set up Cloud Scheduler

Now that the Cloud Run service is ready and deployed, it's time to create the regular schedule, to invoke the service every minute.

Create a service account:

SERVICE_ACCOUNT=collage-scheduler-sa
gcloud iam service-accounts create $SERVICE_ACCOUNT \
   --display-name "Collage Scheduler Service Account"

Give service account permission to invoke the Cloud Run service:

gcloud run services add-iam-policy-binding $SERVICE_NAME \
   --member=serviceAccount:$SERVICE_ACCOUNT@$GOOGLE_CLOUD_PROJECT.iam.gserviceaccount.com \
   --role=roles/run.invoker

Create a Cloud Scheduler job to execute every 1 minute:

SERVICE_URL=$(gcloud run services describe $SERVICE_NAME --format 'value(status.url)')
gcloud scheduler jobs create http $SERVICE_NAME-job --schedule "* * * * *" \
   --http-method=GET \
   --location=europe-west1 \
   --uri=$SERVICE_URL \
   --oidc-service-account-email=$SERVICE_ACCOUNT@$GOOGLE_CLOUD_PROJECT.iam.gserviceaccount.com \
   --oidc-token-audience=$SERVICE_URL

You can go to Cloud Scheduler section in Cloud Console to see that it is setup and pointing to Cloud Run service url:

9. Test the service

To test if the setup is working, check in the thumbnails bucket for the collage image (called collage.png). You can also check the logs of the service:

10. Clean up (Optional)

Note: If you intend to continue with other labs in the series, you can skip this step.

If you don't intend to continue with the other labs in the series, you can clean up resources to save costs and to be an overall good cloud citizen. You can clean up resources individually as follows.

Delete the service:

gcloud run services delete $SERVICE_NAME -q

Delete the Cloud Scheduler job:

gcloud scheduler jobs delete $SERVICE_NAME-job -q

Alternatively, you can delete the whole project:

gcloud projects delete $GOOGLE_CLOUD_PROJECT

11. Congratulations!

Congratulations! You created a scheduled service: thanks to Cloud Scheduler, which pushes a message every minute on a Pub/Sub topic, your Cloud Run collage service is invoked and is able to append pictures together to create the resulting picture.

What we've covered

• Cloud Run
• Cloud Scheduler
• Cloud Storage
• Cloud Firestore

Next Steps

• Pic-a-daily: Lab 4—Creating a web frontend