How to use App Engine blobstore (Module 15)

1. Overview

This series of codelabs (self-paced, hands-on tutorials) aims to help Google App Engine (Standard) developers modernize their apps by guiding them through a series of migrations. The most significant step is to move away from original runtime bundled services because the next generation runtimes are more flexible, giving users a greater variety of service options. Moving to the newer generation runtime enables you to integrate with Google Cloud products more easily, use a wider range of supported services, and support current language releases.

This codelab teaches you how to use App Engine blobstore and add its use to the baseline sample app from Module 0.

You'll learn how to

  • Add use of the App Engine blobstore API/library
  • Store user uploads to the blobstore service
  • Prepare for next step to migrate to Cloud Storage

What you'll need

Survey

How will you use this tutorial?

Read it through only Read it and complete the exercises

How would you rate your experience with Python?

Novice Intermediate Proficient

How would you rate your experience with using Google Cloud services?

Novice Intermediate Proficient

2. Background

In order to migrate from the App Engine Blobstore API, we need to add its usage to the existing baseline App Engine ndb app from Module 0. The sample app displays the ten most recent visits to the user. We are modifying the app to prompt the end-user to upload an artifact (a file) that corresponds to their "visit." If the user doesn't wish to do so, there is a "skip" option. Regardless of the user's decision, the next page renders the same output as the app from Module 0 (and many of the other modules in this series). With this App Engine blobstore integration implemented, we can migrate it to Cloud Storage in the next (Module 16) codelab.

App Engine provides access to the Django and Jinja2 templating systems, and one thing that makes this example different (besides adding Blobstore access) is that it switches from using Django in Module 0 to Jinja2 here in Module 15. A key step in modernizing App Engine apps is to migrate web frameworks from webapp2 to Flask. The latter uses Jinja2 as its default templating system, so we start moving in that direction by implementing Jinja2 while staying on webapp2 for Blobstore access. Since Flask uses Jinja2 by default, this means no changes to the template will be required ahead in Module 16.

3. Setup/Prework

Before we get to the main part of the tutorial, set up your project, get the code, and deploy the baseline app to start with working code.

1. Setup project

If you deployed the Module 0 app already, we recommend reusing the same project (and code). Alternatively, you can create a brand new project or reuse another existing project. Ensure the project has an active billing account and App Engine is enabled.

2. Get baseline sample app

One of the prerequisites to this codelab is to have a working Module 0 sample app. If you don't have it, you can get it from the Module 0 "START" folder (link below). This codelab walks you through each step, concluding with code that resembles what's in the Module 15 "FINISH" folder.

The directory of Module 0 STARTing files should look like this:

$ ls
README.md               index.html
app.yaml                main.py

3. (Re)Deploy baseline app

Your remaining prework steps to execute now:

  1. Re-familiarize yourself with the gcloud command-line tool
  2. Re-deploy the sample app with gcloud app deploy
  3. Confirm the app runs on App Engine without issue

Once you've successfully executed those steps and see your web app works (with output similar to the below), you're ready to add use of caching to your app.

a7a9d2b80d706a2b.png

4. Update configuration files

app.yaml

There are no material changes to the application configuration, however as mentioned earlier, we are moving from Django templating (default) to Jinja2, so in order to switch, users should specify the latest version of Jinja2 available on App Engine servers, and you do it by adding it to the built-in 3rd-party libraries section of app.yaml.

BEFORE:

runtime: python27
threadsafe: yes
api_version: 1

handlers:
- url: /.*
  script: main.app

Edit your app.yaml file by adding a new libraries section like you see here:

AFTER:

runtime: python27
threadsafe: yes
api_version: 1

handlers:
- url: /.*
  script: main.app

libraries:
- name: jinja2
  version: latest

No other configuration files need updating, so let's move ahead to application files.

5. Modify application files

Imports and Jinja2 support

The first set of changes for main.py include adding the use of the Blobstore API and replacing Django templating with Jinja2. Here is what is changing:

  1. The purpose of the os module is to create a file pathname to a Django template. Since we're switching to Jinja2 where this is handled, use of os as well as the Django template renderer, google.appengine.ext.webapp.template, are no longer needed, so they're being removed.
  2. Import the Blobstore API: google.appengine.ext.blobstore
  3. Import the Blobstore handlers found in the original webapp framework—they're not available in webapp2: google.appengine.ext.webapp.blobstore_handlers
  4. Import Jinja2 support from the webapp2_extras package

BEFORE:

import os
import webapp2
from google.appengine.ext import ndb
from google.appengine.ext.webapp import template

Implement the changes in the list above by replacing the current import section in main.py with the below code snippet.

AFTER:

import webapp2
from webapp2_extras import jinja2
from google.appengine.ext import blobstore, ndb
from google.appengine.ext.webapp import blobstore_handlers

After the imports, we need to add some boilerplate code to support use of Jinja2 as defined in the webapp2_extras docs. The following code snippet wraps the standard webapp2 request handler class with Jinja2 functionality, so add this code block to main.py just after the imports:

class BaseHandler(webapp2.RequestHandler):
    'Derived request handler mixing-in Jinja2 support'
    @webapp2.cached_property
    def jinja2(self):
        return jinja2.get_jinja2(app=self.app)

    def render_response(self, _template, **context):
        self.response.write(self.jinja2.render_template(_template, **context))

Add Blobstore support

Unlike other migrations in this series where we keep the sample app's functionality or output identical (or nearly the same) without (much) change to the UX, this example takes a more radical departure from the norm. Rather than immediately registering a new visit then displaying the most recent ten, we are updating the app to ask the user for a file artifact to register their visit with. End-users can then either upload a corresponding file or select "Skip" to not upload anything at all. Once this step is complete, the "most recent visits" page is displayed.

This change allows our app to use the Blobstore service to store (and possibly later render) that image or other file type on the most recent visits page.

Update data model and implement its use

We're storing more data, specifically updating the data model to store the ID (called a "BlobKey") of the file uploaded to Blobstore and adding a reference to save that in store_visit(). Since this extra data is returned along with everything else upon query, fetch_visits() stays the same.

Here are the before and after with these updates featuring file_blob, an ndb.BlobKeyProperty:

BEFORE:

class Visit(ndb.Model):
    'Visit entity registers visitor IP address & timestamp'
    visitor   = ndb.StringProperty()
    timestamp = ndb.DateTimeProperty(auto_now_add=True)

def store_visit(remote_addr, user_agent):
    'create new Visit entity in Datastore'
    Visit(visitor='{}: {}'.format(remote_addr, user_agent)).put()

def fetch_visits(limit):
    'get most recent visits'
    return Visit.query().order(-Visit.timestamp).fetch(limit)

AFTER:

class Visit(ndb.Model):
    'Visit entity registers visitor IP address & timestamp'
    visitor   = ndb.StringProperty()
    timestamp = ndb.DateTimeProperty(auto_now_add=True)
    file_blob = ndb.BlobKeyProperty()

def store_visit(remote_addr, user_agent, upload_key):
    'create new Visit entity in Datastore'
    Visit(visitor='{}: {}'.format(remote_addr, user_agent),
            file_blob=upload_key).put()

def fetch_visits(limit):
    'get most recent visits'
    return Visit.query().order(-Visit.timestamp).fetch(limit)

Here is a pictorial representation of the changes that have been made so far:

2270783776759f7f.png

Supporting file uploads

The most significant change in functionality is supporting file uploads, whether prompting the user for a file, supporting the "skip" feature, or rendering a file corresponding to a visit. All of it is part of the picture. These are the changes required to support file uploads:

  1. The main handler GET request no longer fetches the most recent visits for display. Instead, it prompts the user for an upload.
  2. When an end-user submits a file to upload or skips that process, a POST from the form passes control to the new UploadHandler, derived from google.appengine.ext.webapp.blobstore_handlers.BlobstoreUploadHandler.
  3. UploadHandler's POST method performs the upload, calls store_visit() to register the visit, and triggers an HTTP 307 redirect to send the user back to "/", where...
  4. The main handler's POST method queries for (via fetch_visits()) and displays the most recent visits. If the user selects "skip," no file is uploaded, but the visit is still registered followed by the same redirect.
  5. The most recent visits display includes a new field displayed to the user, either a hyperlinked "view" if an upload file is available or "none" otherwise. These changes are realized in the HTML template along with the addition of an upload form (more on this coming soon).
  6. If an end-user clicks the "view" link for any visit with an uploaded video, it makes a GET request to a new ViewBlobHandler, derived from google.appengine.ext.webapp.blobstore_handlers.BlobstoreDownloadHandler , either rendering the file if an image (in the browser if supported), prompting to download if not, or returning an HTTP 404 error if not found.
  7. In addition to the new pair of handler classes as well as a new pair of routes to send traffic to them, the main handler needs a new POST method to receive the 307 redirect described above.

Before these updates, the Module 0 app only featured a main handler with a GET method and a single route:

BEFORE:

class MainHandler(webapp2.RequestHandler):
    'main application (GET) handler'
    def get(self):
        store_visit(self.request.remote_addr, self.request.user_agent)
        visits = fetch_visits(10)
        tmpl = os.path.join(os.path.dirname(__file__), 'index.html')
        self.response.out.write(template.render(tmpl, {'visits': visits}))

app = webapp2.WSGIApplication([
    ('/', MainHandler),
], debug=True)

With those updates implemented, there are now three handlers: 1) upload handler with a POST method, 2) a "view blob" download handler with a GET method, and 3) the main handler with GET and POST methods. Make these changes so that the rest of your app now looks like the below.

AFTER:

class UploadHandler(blobstore_handlers.BlobstoreUploadHandler):
    'Upload blob (POST) handler'
    def post(self):
        uploads = self.get_uploads()
        blob_id = uploads[0].key() if uploads else None
        store_visit(self.request.remote_addr, self.request.user_agent, blob_id)
        self.redirect('/', code=307)

class ViewBlobHandler(blobstore_handlers.BlobstoreDownloadHandler):
    'view uploaded blob (GET) handler'
    def get(self, blob_key):
        self.send_blob(blob_key) if blobstore.get(blob_key) else self.error(404)

class MainHandler(BaseHandler):
    'main application (GET/POST) handler'
    def get(self):
        self.render_response('index.html',
                upload_url=blobstore.create_upload_url('/upload'))

    def post(self):
        visits = fetch_visits(10)
        self.render_response('index.html', visits=visits)

app = webapp2.WSGIApplication([
    ('/', MainHandler),
    ('/upload', UploadHandler),
    ('/view/([^/]+)?', ViewBlobHandler),
], debug=True)

There are several key calls in this code we just added:

  • In MainHandler.get, there's a call to blobstore.create_upload_url. This call generates the URL the form POSTs to, calling the upload handler to send the file to Blobstore.
  • In UploadHandler.post, there's a call to blobstore_handlers.BlobstoreUploadHandler.get_uploads. This is the real magic that puts the file into Blobstore and returns a unique and persistent ID for that file, its BlobKey.
  • In ViewBlobHandler.get, calling blobstore_handlers.BlobstoreDownloadHandler.send with a file's BlobKey results in fetching of the file and forwarding it to the end-user's browser

These calls represent the bulk of accessing the features added to the app. Here is a pictorial representation of this second and final set of changes to main.py:

da2960525ac1b90d.png

Update HTML template

Some of the updates to the main application affect the app's user interface (UI), so corresponding changes are required in the web template, two in fact:

  1. A file upload form is required with 3 input elements: a file and a pair of submit buttons for file upload and skip, respectively.
  2. Update the most recent visits output by adding a "view" link for visits with a corresponding file upload or "none" otherwise.

BEFORE:

<!doctype html>
<html>
<head>
<title>VisitMe Example</title>
<body>

<h1>VisitMe example</h1>
<h3>Last 10 visits</h3>
<ul>
{% for visit in visits %}
    <li>{{ visit.timestamp.ctime }} from {{ visit.visitor }}</li>
{% endfor %}
</ul>

</body>
</html>

Implement the changes in the list above to comprise the updated template:

AFTER:

<!doctype html>
<html>
<head>
<title>VisitMe Example</title>
<body>

<h1>VisitMe example</h1>
{% if upload_url %}

<h3>Welcome... upload a file? (optional)</h3>
<form action="{{ upload_url }}" method="POST" enctype="multipart/form-data">
    <input type="file" name="file"><p></p>
    <input type="submit"> <input type="submit" value="Skip">
</form>

{% else %}

<h3>Last 10 visits</h3>
<ul>
{% for visit in visits %}
<li>{{ visit.timestamp.ctime() }}
    <i><code>
    {% if visit.file_blob %}
        (<a href="/view/{{ visit.file_blob }}" target="_blank">view</a>)
    {% else %}
        (none)
    {% endif %}
    </code></i>
    from {{ visit.visitor }}
</li>
{% endfor %}
</ul>

{% endif %}

</body>
</html>

This image illustrates the required updates to index.html:

8583e975f25aa9e7.png

One final change is that Jinja2 prefers its templates in a templates folder, so create that folder and move index.html inside it. With this final move, you're now done with all of the necessary changes for adding the use of Blobstore to the Module 0 sample app.

(optional) Cloud Storage "enhancement"

Blobstore storage eventually evolved into Cloud Storage itself. This means that Blobstore uploads are visible in the Cloud console, specifically the Cloud Storage browser. The question is where. The answer is your App Engine app's default Cloud Storage bucket. It's name is the name of your App Engine app's full domain name, PROJECT_ID.appspot.com. It's so convenient because all project IDs are unique, right?

The updates made to the sample application drops uploaded files into that bucket, but developers have an option to choose a more specific location. The default bucket is programmatically accessible via google.appengine.api.app_identity.get_default_gcs_bucket_name(), requiring a new import if you want to access this value, say to use as a prefix for organizing uploaded files. For example, sorting by file type:

f61f7a23a1518705.png

To implement something like that for images, for example, you'll have code like this along with some code that checked file types to pick the desired bucket name:

ROOT_BUCKET = app_identity.get_default_gcs_bucket_name()
IMAGE_BUCKET = '%s/%s' % (ROOT_BUCKET, 'images')

You'll also validate the images uploaded using a tool like the Python Standard Library imghdr module to confirm image type. Finally, you'll probably want to limit the size of uploads in case of bad actors.

Let's say all that has been done. How can we update our app to support specifying where to store the uploaded files? The key is to tweak the call to blobstore.create_upload_url in MainHandler.get to specify the desired location in Cloud Storage for the upload by adding the gs_bucket_name parameter like this:

blobstore.create_upload_url('/upload', gs_bucket_name=IMAGE_BUCKET))

As this is an optional update if you want to specify where uploads should go, it is not part of the main.py file in the repo. Instead, an alternative named main-gcs.py is available for your review in the repo. Rather than using a separate bucket "folder," the code in main-gcs.py stores uploads in the "root" bucket (PROJECT_ID.appspot.com) just like main.py but provides the scaffolding you need if you were to derive the sample into something more as hinted in this section. Below is an illustration of the "diffs" between main.py and main-gcs.py.

256e1ea68241a501.png

6. Summary/Cleanup

Deploy application

Re-deploy your app with gcloud app deploy, and confirm the app works as advertised, differing in user experience (UX) from the Module 0 app. There are two different screens in your app now, the first being the visit file upload form prompt:

f5b5f9f19d8ae978.pngFrom there, end-users either upload a file and click "Submit" or click on "Skip" to not upload anything. In either case, the result is the most recent visit screen, now augmented with "view" links or "none" between visit timestamps and visitor information:

f5ac6b98ee8a34cb.png

Congratulations for completing this codelab adding use of App Engine Blobstore to the Module 0 sample app. Your code should now match what's in the FINISH (Module 15) folder. The alternative main-gcs.py is also present in that folder.

Clean up

If you are done for now, we recommend you disable your App Engine app to avoid incurring billing. However, if you want to play around with it a bit more, that is fine too. The App Engine platform has a free quota, and as long as you don't exceed that usage tier, you shouldn't be charged. That's for compute, however each App Engine service has its own billing schedule as well:

  • The App Engine Blobstore service falls under Stored Data quotas and limits, so review that.
  • The App Engine Datastore service has a free tier as well, but will charge you if you go beyond those limits. See its pricing page for more information.

For full disclosure, deploying to a Google Cloud serverless compute platform like App Engine incurs minor build and storage costs. Cloud Build has its own free quota as does Cloud Storage. Storage of that image uses up some of that quota. However, you might live in a region that does not have such a free tier, so be aware of your storage usage to minimize potential costs.

On the other hand, if you're not going to continue with this application or other related migration codelabs and want to delete everything completely, you can shut down your project.

Next steps

The next logical migration to consider is covered in Module 16, showing developers how to migrate from App Engine Blobstore service to using the Cloud Storage client library. Benefits to upgrading include being able to access more Cloud Storage features, becoming familiar with a client library that works for apps outside of App Engine, whether in Google Cloud, other clouds, or even on-premise. If you don't feel like you need all the features available from Cloud Storage or are concerned about its effects on cost, you are free to stay on App Engine Blobstore.

Beyond Module 16 are a whole slew of other possible migrations such as Cloud NDB and Cloud Datastore, Cloud Tasks, or Cloud Memorystore. There are also cross-product migrations to Cloud Run and Cloud Functions. The migration repo features all the code samples, links you to all the codelabs and videos available, and also provides guidance on which migrations to consider and any relevant "order" of migrations.

7. Additional resources

Codelab issues/feedback

If you find any issues with this codelab, please search for your issue first before filing. Links to search and create new issues:

Migration resources

Links to the repo folders for Module 0 (START) and Module 15 (FINISH) can be found in the table below. They can also be accessed from the repo for all App Engine codelab migrations which you can clone or download a ZIP file.

Codelab

Python 2

Python 3

Module 0

code

N/A

Module 15 (this codelab)

code

N/A

Online resources

Below are online resources which may be relevant for this tutorial:

App Engine

Google Cloud

Python

Videos

License

This work is licensed under a Creative Commons Attribution 2.0 Generic License.