Pic-a-daily: Lab 1—Store and analyse pictures

In the first code lab, you will store pictures in a bucket. This will generate a file creation event that will be handled by a function. The function will make a call to Vision API to do image analysis and save results in a datastore.

6b7b4a1288add11a.png

What you'll learn

  • Cloud Storage
  • Cloud Functions
  • Cloud Vision API
  • Cloud Firestore

Self-paced environment setup

  1. Sign in to Cloud Console and create a new project or reuse an existing one. (If you don't already have a Gmail or G Suite account, you must create one.)

dMbN6g9RawQj_VXCSYpdYncY-DbaRzr2GbnwoV7jFf1u3avxJtmGPmKpMYgiaMH-qu80a_NJ9p2IIXFppYk8x3wyymZXavjglNLJJhuXieCem56H30hwXtd8PvXGpXJO9gEUDu3cZw

ci9Oe6PgnbNuSYlMyvbXF1JdQyiHoEgnhl4PlV_MFagm2ppzhueRkqX4eLjJllZco_2zCp0V0bpTupUSKji9KkQyWqj11pqit1K1faS1V6aFxLGQdkuzGp4rsQTan7F01iePL5DtqQ

8-tA_Lheyo8SscAVKrGii2coplQp2_D1Iosb2ViABY0UUO1A8cimXUu6Wf1R9zJIRExL5OB2j946aIiFtyKTzxDcNnuznmR45vZ2HMoK3o67jxuoUJCAnqvEX6NgPGFjCVNgASc-lg

Remember the project ID, a unique name across all Google Cloud projects (the name above has already been taken and will not work for you, sorry!). It will be referred to later in this codelab as PROJECT_ID.

  1. Next, you'll need to enable billing in Cloud Console in order to use Google Cloud resources.

Running through this codelab shouldn't cost much, if anything at all. Be sure to to follow any instructions in the "Cleaning up" section which advises you how to shut down resources so you don't incur billing beyond this tutorial. New users of Google Cloud are eligible for the $300USD Free Trial program.

Start Cloud Shell

While Google Cloud can be operated remotely from your laptop, in this codelab you will be using Google Cloud Shell, a command line environment running in the Cloud.

From the GCP Console click the Cloud Shell icon on the top right toolbar:

E0b6xMEnCN6XCtm5OITZ-CHPnhUsO3WrGGJFu0Yr587eWRPZG2xj4U9wHbTxF8d1LTHnk5yzgMxEbhAmTCwbNH8rMoQV70pEkLkz54gtUHD7kRtiSI_2EqrighTDFbuoO0Z146CC3Q

It should only take a few moments to provision and connect to the environment. When it is finished, you should see something like this:

kXnInzErAjsyeUStcIiTdNi179GwXpgp-2YTay2z0DW_7PoZ7uPWiKlaYk0LXNwv2kvkqUZEjhWjAgwNsgkX4Kpkhu8duXo5FTsog9bM405TSmdC_BUIX4ywkMV-tEc1VHtUzdTykg

This virtual machine is loaded with all the development tools you'll need. It offers a persistent 5GB home directory, and runs on Google Cloud, greatly enhancing network performance and authentication. All of your work in this lab can be done with simply a browser.

For this lab, you will be using Cloud Functions and Vision API but first they need to be enabled either in Cloud Console or with gcloud.

To enable Vision API in Cloud Console, search for Cloud Vision API in the search bar:

cf48b1747ba6a6fb.png

You will land on the Cloud Vision API page:

d13b3a36c0d69071.png

Click the ENABLE button.

Alternatively, you can also enable it Cloud Shell using the gcloud command line tool.

Inside Cloud Shell, run the following command:

gcloud services enable vision.googleapis.com

You should see the operation to finish successfully:

Operation "operations/acf.12dba18b-106f-4fd2-942d-fea80ecc5c1c" finished successfully.

Enable Cloud Functions as well:

gcloud services enable cloudfunctions.googleapis.com

Create a storage bucket for the pictures. You can do this from Google Cloud Platform console ( console.cloud.google.com) or with gsutil command line tool from Cloud Shell or your local development environment.

From the "hamburger" (☰) menu, navigate to the Storage page.

d7b79927a62e1de3.png

Name your bucket

Click on the CREATE BUCKET button.

4e66e53ee519a774.png

Click CONTINUE.

Choose Location

c7ffb7f656d25f32.png

Create a multi-regional bucket in the region of your choice (here Europe).

Click CONTINUE.

Choose default storage class

95239fd0872a45dc.png

Choose the Standard storage class for your data.

Click CONTINUE.

Set Access Control

eb662feb03de68be.png

As you will be working with publicly accessible images, you want all our pictures stored in this bucket to have the same uniform access control.

Choose the Uniform access control option.

Click CONTINUE.

Set Encryption

84cebcf7c9811f23.png

Select Google-managed key, as you won't use your own encryption keys.

Click CREATE, to eventually finalize our bucket creation.

Add allUsers as storage viewer

Go to the Permissions tab:

8d207d4e0fc795a.png

Add an allUsers member to the bucket, with a role of Storage > Storage Object Viewer, as follows:

7ccb44e5fb63afef.png

Click SAVE.

You can also use the gsutil command line tool in Cloud Shell to create buckets.

In Cloud Shell, set a variable for the unique bucket name. Cloud Shell already has GOOGLE_CLOUD_PROJECT set to your unique project id. You can append that to the bucket name.

For example:

export BUCKET_PICTURES=uploaded-pictures-${GOOGLE_CLOUD_PROJECT}

Create a standard multi-region zone in Europe:

gsutil mb -l EU gs://${BUCKET_PICTURES}

Ensure uniform bucket level access:

gsutil uniformbucketlevelaccess set on gs://${BUCKET_PICTURES}

Make the bucket public:

gsutil iam ch allUsers:objectViewer gs://${BUCKET_PICTURES}

If you go to Cloud Storage section of the console, you should have a public uploaded-pictures bucket:

a98ed4ba17873e40.png

Test that you can upload pictures to the bucket and the uploaded pictures are publicly available, as explained in the previous step.

Going back to the storage browser, you'll see your bucket in the list, with "Public" access (including a warning sign reminding you that anyone has access to the content of that bucket).

89e7a4d2c80a0319.png

Your bucket is now ready to receive pictures.

If you click on the bucket name, you'll see the bucket details.

8c698c13acf8d163.png

There, you can try the Upload files button, to test that you can add a picture to the bucket. A file chooser popup will ask you to select a file. Once selected, it'll be uploaded to your bucket, and you will see again the public access that has been automatically attributed to this new file.

2f65ba5d7654cfad.png

Along the Public access label, you will also see a little link icon. When clicking on it, your browser will navigate to the public URL of that image, which will be of the form:

https://storage.googleapis.com/BUCKET_NAME/PICTURE_FILE.png

With BUCKET_NAME being the globally unique name you have chosen for your bucket, and then the file name of your picture.

By clicking on the check box along the picture name, the DELETE button will be enabled, and you can delete this first image.

In this step, you create a function that reacts to picture upload events.

Visit the Cloud Functions section of the Google Cloud console. By visiting it, the Cloud Functions service will be automatically enabled.

4eb3242d1c82953.png

Click on Create function.

Choose a name (eg. picture-uploaded) and the amount of memory (here 256 MB):

1fe65150207f6253.png

There are two kinds of functions:

  • HTTP functions which can be invoked via a URL (ie. a web API),
  • Background functions which can be triggered by some event.

You want to create a background function that is triggered when a new file is uploaded to our Cloud Storage bucket:

1a662782c3d25572.png

You are interested in the Finalize/Create event type, which is the event that is triggered when a file is created or updated in the bucket:

2d92a9e0eb979a53.png

Select the bucket created before, to tell Cloud Functions to be notified when a file is created / updated in this particular bucket:

6eda8cd8e01e9118.png

Click Select to choose the bucket you created earlier.

Keep the Inline editor for this function:

15e649027af86784.png

Select the default Node.js 8 runtime:

310ee711fecf5217.png

The source code consists of an index.js JavaScript file, and a package.json file that provides various metadata and dependencies.

Leave the default snippet of code: it logs the file name of the uploaded picture:

40d4fbc15ac16c7a.png

For now, keep the name of the function to execute to helloGCS, for testing purposes.

At the bottom of the page, you'll notice the following folded section:

a1f7ceb9f50ed02.png

Click on Environment variables, networking, timeouts and more

Select the same region as other resources, for example europe-west1:

6281fb3a41bd55b9.png

No need to further change any other advanced setting.

Click on Create to deploy the function. Once the deployment succeeded, you should see a green-circled check mark in the list of functions:

2a254f75b2abce3.png

In this step, test that the function responds to storage events.

From the "hamburger" (☰) menu, navigate back to the Storage page.

Click on the images bucket, and then on Upload files to upload an image.

Navigate again within the cloud console to go to the Logging > Logs viewer page.

In the audited resource selector, select Cloud Function to see the logs dedicated to your functions. You can even select a specific function to have a finer-grained view of the functions related logs.

e3944e605471a58f.png

You should see the log items mentioning the creation of the function, the start and end times of the function, and our actual log statement:

2b48e5437d1370e4.png

Our log statement reads: Processing file: my-picture.png, meaning that the event related to the creation and storage of this picture has indeed been triggered as expected.

You will store information about the picture given by the Vision API into the Cloud Firestore database, a fast, fully managed, serverless, cloud-native NoSQL document database. Prepare your database by going to the Firestore section of the Cloud Console:

40351a87ec3e030d.png

Two options are offered: Native mode or Datastore mode. Use the native mode, which offers extra features like offline support and real-time synchronization.

Click on SELECT NATIVE MODE.

125e7b399ebc879b.png

Pick a multi-region (here in Europe, but ideally at least the same region your function and storage bucket are).

Click the CREATE DATABASE button.

Once the database is created, you should see the following:

afada6b463a2569e.png

Create a new collection by clicking the + START COLLECTION button.

Name collection pictures.

73df401b52167ceb.png

You don't need to create a document. You'll add them programmatically as new pictures are stored in Cloud Storage and analysed by the Vision API.

Click Save.

Firestore creates a first default document in the newly created collection, you can safely delete that document as it doesn't contain any useful information:

14866b199773626.png

The documents that will be created programmatically in our collection will contain 4 fields:

  • name (string): the file name of the uploaded picture, which is also he key of the document
  • labels (array of strings): the labels of recognised items by the Vision API
  • color (string): the hexadecimal color code of the dominant color (ie. #ab12ef)
  • created (date): the timestamp of when this image's metadata was stored
  • thumbnail (boolean): an optional field that will be present and be true if a thumbnail image has been generated for this picture

As we will be searching in Firestore to find pictures that have thumbnails available, and sorting along the creation date, we'll need to create a search index. You can create the index with the following command in Cloud Shell:

gcloud alpha firestore indexes composite create \ 
  --collection-group=pictures \
  --field-config field-path=thumbnail,order=descending \
  --field-config field-path=created,order=descending

Or you can also do it from the Cloud Console, by clicking on Indexes, in the navigation column on the left, and then creating a composite index as shown below:

fb2bc891ff1d47f8.png

Click Create.

Move back to the Functions page, to update the function to invoke the Vision API to analyze our pictures, and to store the metadata in Firestore.

From the "hamburger" (☰) menu, navigate to the Cloud Functions section, click on the function name, select the Source tab, and then click the EDIT button.

You can click the four-corner square icon in the top right hand corner to maximize the source code view:

da95f95a7f7cb253.png

First, edit the package.json file which lists the dependencies of our Node.JS function. Update the code to add the Cloud Vision API NPM dependency:

{
  "name": "picture-analysis-function",
  "version": "0.0.1",
  "dependencies": {
    "@google-cloud/storage": "^1.6.0",
    "@google-cloud/vision": "^1.8.0",
    "@google-cloud/firestore": "^3.4.1"
  }
}

Now that the dependencies are up-to-date, you are going to work on the code of our function, by updating the index.js file.

Replace the code in index.js with the code below. It will be explained in the next step.

const vision = require('@google-cloud/vision');
const Storage = require('@google-cloud/storage');
const Firestore = require('@google-cloud/firestore');

const client = new vision.ImageAnnotatorClient();

exports.vision_analysis = async (event, context) => {
    console.log(`Event: ${JSON.stringify(event)}`);

    const filename = event.name;
    const filebucket = event.bucket;

    console.log(`New picture uploaded ${filename} in ${filebucket}`);

    const request = {
        image: { source: { imageUri: `gs://${filebucket}/${filename}` } },
        features: [
            { type: 'LABEL_DETECTION' },
            { type: 'IMAGE_PROPERTIES' },
            { type: 'SAFE_SEARCH_DETECTION' }
        ]
    };

    // invoking the Vision API
    const [response] = await client.annotateImage(request);
    console.log(`Raw vision output for: ${filename}: ${JSON.stringify(response)}`);

    if (response.error === null) {
        // listing the labels found in the picture
        const labels = response.labelAnnotations
            .sort((ann1, ann2) => ann2.score - ann1.score)
            .map(ann => ann.description)
        console.log(`Labels: ${labels.join(', ')}`);

        // retrieving the dominant color of the picture
        const color = response.imagePropertiesAnnotation.dominantColors.colors
            .sort((c1, c2) => c2.score - c1.score)[0].color;
        const colorHex = decColorToHex(color.red, color.green, color.blue);
        console.log(`Colors: ${colorHex}`);

        // determining if the picture is safe to show
        const safeSearch = response.safeSearchAnnotation;
        const isSafe = ["adult", "spoof", "medical", "violence", "racy"].every(k => 
            !['LIKELY', 'VERY_LIKELY'].includes(safeSearch[k]));
        console.log(`Safe? ${isSafe}`);

        // if the picture is safe to display, store it in Firestore
        if (isSafe) {
            const pictureStore = new Firestore().collection('pictures');
            
            const doc = pictureStore.doc(filename);
            await doc.set({
                labels: labels,
                color: colorHex,
                created: Firestore.Timestamp.now()
            }, {merge: true});

            console.log("Stored metadata in Firestore");
        }
    } else {
        throw new Error(`Vision API error: code ${response.error.code}, message: "${response.error.message}"`);
    }
};

function decColorToHex(r, g, b) {
    return '#' + Number(r).toString(16).padStart(2, '0') + 
                 Number(g).toString(16).padStart(2, '0') + 
                 Number(b).toString(16).padStart(2, '0');
}

Let's have a closer look at the various interesting parts.

First, we're require-ing the needed modules, for Vision, Storage and Firestore:

const vision = require('@google-cloud/vision');
const Storage = require('@google-cloud/storage');
const Firestore = require('@google-cloud/firestore');

Then, we prepare a client for the Vision API:

const client = new vision.ImageAnnotatorClient();

Now comes the structure of our function. We make it an async function, as we are using the async / await capabilities introduced in Node.js 8:

exports.vision_analysis = async (event, context) => {
    ...
    const filename = event.name;
    const filebucket = event.bucket;
    ...
}

Notice the signature, but also how we retrieve the name of the file and bucket which triggered the Cloud Function.

For reference, here's what the event payload looks like:

{
  "bucket":"uploaded-pictures",
  "contentType":"image/png",
  "crc32c":"efhgyA==",
  "etag":"CKqB956MmucCEAE=",
  "generation":"1579795336773802",
  "id":"uploaded-pictures/Screenshot.png/1579795336773802",
  "kind":"storage#object",
  "md5Hash":"PN8Hukfrt6C7IyhZ8d3gfQ==",
  "mediaLink":"https://www.googleapis.com/download/storage/v1/b/uploaded-pictures/o/Screenshot.png?generation=1579795336773802&alt=media",
  "metageneration":"1",
  "name":"Screenshot.png",
  "selfLink":"https://www.googleapis.com/storage/v1/b/uploaded-pictures/o/Screenshot.png",
  "size":"173557",
  "storageClass":"STANDARD",
  "timeCreated":"2020-01-23T16:02:16.773Z",
  "timeStorageClassUpdated":"2020-01-23T16:02:16.773Z",
  "updated":"2020-01-23T16:02:16.773Z"
}

We prepare a request to send via the Vision client:

const request = {
    image: { source: { imageUri: `gs://${filebucket}/${filename}` } },
    features: [
        { type: 'LABEL_DETECTION' },
        { type: 'IMAGE_PROPERTIES' },
        { type: 'SAFE_SEARCH_DETECTION' }
    ]
};

We're asking for 3 key capabilities of the Vision API:

  • Label detection: to understand what's in those pictures
  • Image properties: to give interesting attributes of the picture (we're interested in the dominant color of the picture)
  • Safe search: to know if the image is safe to show (it shouldn't contain adult / medical / racy / violent content)

At this point, we can make the call to the Vision API:

const [response] = await client.annotateImage(request);

For reference, here's what the response from the Vision API looks like:

{
  "faceAnnotations": [],
  "landmarkAnnotations": [],
  "logoAnnotations": [],
  "labelAnnotations": [
    {
      "locations": [],
      "properties": [],
      "mid": "/m/01yrx",
      "locale": "",
      "description": "Cat",
      "score": 0.9959855675697327,
      "confidence": 0,
      "topicality": 0.9959855675697327,
      "boundingPoly": null
    },
    ✄ - - - ✄
  ],
  "textAnnotations": [],
  "localizedObjectAnnotations": [],
  "safeSearchAnnotation": {
    "adult": "VERY_UNLIKELY",
    "spoof": "UNLIKELY",
    "medical": "VERY_UNLIKELY",
    "violence": "VERY_UNLIKELY",
    "racy": "VERY_UNLIKELY",
    "adultConfidence": 0,
    "spoofConfidence": 0,
    "medicalConfidence": 0,
    "violenceConfidence": 0,
    "racyConfidence": 0,
    "nsfwConfidence": 0
  },
  "imagePropertiesAnnotation": {
    "dominantColors": {
      "colors": [
        {
          "color": {
            "red": 203,
            "green": 201,
            "blue": 201,
            "alpha": null
          },
          "score": 0.4175916016101837,
          "pixelFraction": 0.44456374645233154
        },
        ✄ - - - ✄
      ]
    }
  },
  "error": null,
  "cropHintsAnnotation": {
    "cropHints": [
      {
        "boundingPoly": {
          "vertices": [
            { "x": 0, "y": 118 },
            { "x": 1177, "y": 118 },
            { "x": 1177, "y": 783 },
            { "x": 0, "y": 783 }
          ],
          "normalizedVertices": []
        },
        "confidence": 0.41695669293403625,
        "importanceFraction": 1
      }
    ]
  },
  "fullTextAnnotation": null,
  "webDetection": null,
  "productSearchResults": null,
  "context": null
}

If there's no error returned, we can move on, hence why we have this if block:

if (response.error === null) {
    ...
} else {
    throw new Error(`Vision API error: code ${response.error.code},  
                     message: "${response.error.message}"`);
}

We are going to get the labels of the things, categories or themes recognised in the picture:

const labels = response.labelAnnotations
    .sort((ann1, ann2) => ann2.score - ann1.score)
    .map(ann => ann.description)

We're sorting the labels by highest score first.

We're interested in knowing the dominant color of the picture:

const color = response.imagePropertiesAnnotation.dominantColors.colors
    .sort((c1, c2) => c2.score - c1.score)[0].color;
const colorHex = decColorToHex(color.red, color.green, color.blue);

We're again sorting colors by score and take the first one.

We're also using a utility function to transform the red / green / blue values into an hexadecimal color code that we can use in CSS stylesheets.

Let's check if the picture is safe to show:

const safeSearch = response.safeSearchAnnotation;
const isSafe = ["adult", "spoof", "medical", "violence", "racy"]
    .every(k => !['LIKELY', 'VERY_LIKELY'].includes(safeSearch[k]));

We're checking the adult / spoof / medical / violence / racy attributes to see if they are not likely or very likely to.

If the result of the safe search is okay, we can store metadata in Firestore:

if (isSafe) {
    const pictureStore = new Firestore().collection('pictures');
            
    const doc = pictureStore.doc(filename);
    await doc.set({
        labels: labels,
        color: colorHex,
        created: Firestore.Timestamp.now()
    }, {merge: true});
}

Time to deploy the function.

cafc44da948816ab.png

Hit DEPLOY button and the new version will be deployed, you can see the progress:

2d0b60bd30b02a93.png

Once the function is successfully deployed, you will post a picture to Cloud Storage, see if our function is invoked, what the Vision API returns, and if metadata is stored in Firestore.

Navigate back to Cloud Storage, and click on the bucket we created at the beginning of the lab:

c6338c7a9161913b.png

Once in the bucket details page:

43e5ed8783550250.png

Click on the Upload files button to upload a picture.

From the "hamburger" (☰) menu, navigate to the Logging > Logs viewer:

8be09424af3fee5c.png

Notice that you can select the logs for particular resources. In our case, we're especially interested in seeing logs coming from our functions:

95d8d6170bf577e3.png

And indeed, in the list of logs, I can see that our function was invoked:

cec370811232a68f.png

The logs indicate the start and end of the function execution. And in between, we can see the logs we put in our function with the console.log() statements. We see:

  • The details of the event triggering our function,
  • The raw results from the Vision API call,
  • The labels that were found in the picture we uploaded,
  • The dominant colors information,
  • Whether the picture is safe to show,
  • And eventually those metadata about the picture have been stored in Firestore.

Again from the "hamburger" (☰) menu, go to the Firestore section. In the Data subsection (shown by default), you should see the pictures collection with a new document added, corresponding to the picture you just uploaded:

891c8603dd2892e7.png

If you don't intend to continue with the other labs in the series, you can clean up resources to save costs and to be an overall good cloud citizen. You can clean up resources individually as follows.

Delete the bucket:

gsutil rb gs://${BUCKET_PICTURES}

Delete the function:

gcloud functions delete picture-uploaded --region europe-west1 -q

Delete the Firestore collection by selecting Delete collection from the collection:

e75e8200704800ba.png

Alternatively, you can delete the whole project:

gcloud projects delete ${GOOGLE_CLOUD_PROJECT} 

Congratulations! You've successfully implemented the first key service of the project!

What we've covered

  • Cloud Storage
  • Cloud Functions
  • Cloud Vision API
  • Cloud Firestore

Next Steps