Module 12: How to use App Engine memcache in Flask apps

1. Overview

This series of codelabs (self-paced, hands-on tutorials) aims to help Google App Engine (standard environment) developers modernize their apps. The codelabs guide users through a series of migrations, primarily moving away from legacy bundled services. The net effect is to make apps more portable, giving developers more options and flexibility. Some of these options include migrating to a standalone Cloud service, updating apps to the latest App Engine runtimes, and switching to sister severless platforms like Cloud Functions or Cloud Run, or to other compute products.

This codelab teaches you how to include and use App Engine memcache to the sample app from the Module 1 codelab. It also repeats the Module 2 migration moving from App Engine's ndb library to the Cloud NDB client library.

You'll learn how to

  • Add use of the App Engine Memcache API/library
  • Add caching to a Python 2 Flask and Cloud NDB app
  • Prepare for next step to migrate to Cloud Memorystore

What you'll need

Survey

How will you use this tutorial?

Read it through only Read it and complete the exercises

How would you rate your experience with Python?

Novice Intermediate Proficient

How would you rate your experience with using Google Cloud services?

Novice Intermediate Proficient

2. Background

In order to migrate from App Engine Memcache, we need to add its usage to the existing Flask and App Engine ndb app resulting from the Module 1 codelab. The sample app displays the ten most recent visits to the user. If the same user refreshes their browser, it's not optimal to continually create new Visit entities and fetch the most recent visits from Datastore, so we're going to cache those most recent visits.

If the same visitor hits the page, those visits are returned from the cache. If a new user visits the site or an hour has passed, the cache is flushed and replaced with the most recent entries (not to mention a new Visit registered). With this App Engine Memcache integration implemented, we can migrate it to Cloud Memorystore in the next (Module 13) codelab.

3. Setup/Prework

Before we get to the main part of the tutorial, let's set up our project, get the code, then deploy the baseline app so we know we started with working code.

1. Setup project

If you completed the Module 1 codelab, we recommend reusing that same project (and code). Alternatively, you can create a brand new project or reuse another existing project. Ensure the project has an active billing account and App Engine is enabled.

2. Get baseline sample app

One of the prerequisites to this codelab is to have a working Module 1 sample app. If you don't have one, go complete either tutorial (links above) before moving ahead here. Otherwise if you're already familiar with its contents, you can just start with the Module 1 code below.

Whether you use yours or ours, the Module 1 code is where we'll START. This codelab walks you through each step, concluding with code that resembles what's in the Module 11 repo folder (FINISH).

The directory of Module 1 STARTing files (yours or ours) should look like this:

$ ls
README.md               main.py                 templates
app.yaml                requirements.txt

3. (Re)Deploy baseline app

Your remaining prework steps to execute now:

  1. Re-familiarize yourself with the gcloud command-line tool
  2. Re-deploy the sample app with gcloud app deploy
  3. Confirm the app runs on App Engine without issue

Once you've successfully executed those steps and see your web app works (with output similar to the below), you're ready to add use of caching to your app.

a7a9d2b80d706a2b.png

4. Modify application files

Because we're only adding an App Engine API, there are no external packages involved, meaning no configuration files (app.yaml, requirements.txt, appengine_config.py) need to be updated. There is only one application file, main.py, so all changes in this section affect just that file.

Imports

The most important step is to import the Memcache library, google.appengine.api.memcache. Since we're going to cache the most recent visits for an hour, let's also add a constant for the number of seconds in an hour. Below is what your code looks like before and this change:

BEFORE:

from flask import Flask, render_template, request
from google.appengine.ext import ndb

app = Flask(__name__)

AFTER:

from flask import Flask, render_template, request
from google.appengine.api import memcache
from google.appengine.ext import ndb

app = Flask(__name__)
HOUR = 3600

Add caching

The most significant change is to add use of caching in our application. More specifically, we should cache the most recent visits, check to see if cached visits are available, and attempt to use the cached results as much as possible given our plan. These are the steps the app will take to accomplish our goal:

  1. Set current visit and call it visitor
  2. Attempt to fetch the most recent visits from cache
  3. If the cache is empty or the most recent visitor (visits[0]['visitor']) differs from the current visitor: store this newest visit, fetch the most recent visits, and cache them for an hour.
  4. Display visits to user through the web template

Here are the before and after with these updates:

BEFORE:

@app.route('/')
def root():
    'main application (GET) handler'
    store_visit(request.remote_addr, request.user_agent)
    visits = fetch_visits(10)
    return render_template('index.html', visits=visits)

AFTER:

@app.route('/')
def root():
    'main application (GET) handler'
    # check for (hour-)cached visits
    ip_addr, usr_agt = request.remote_addr, request.user_agent
    visitor = '{}: {}'.format(ip_addr, usr_agt)
    visits = memcache.get('visits')

    # register visit & run DB query if cache empty or new visitor
    if not visits or visits[0]['visitor'] != visitor:
        store_visit(ip_addr, usr_agt)
        visits = list(fetch_visits(10))
        memcache.set('visits', visits, HOUR)  # set() not add()

    return render_template('index.html', visits=visits)

Here is a pictorial representation of the changes that were made:

b1242503602f7bf0.png

This concludes all of the necessary changes for adding the use of App Engine memcache to the Module 1 sample app. Let's build and deploy this app to see it work!

5. Summary/Cleanup

Deploy application

Re-deploy your app with gcloud app deploy, and confirm the app works. Your code should now match what's in FINISH, the Module 12 folder. The output should be identical to the Module 1 app you deployed earlier:

a7a9d2b80d706a2b.png

All we did was speed up the user experience for the same user. When they refresh, you should get the results directly from the cache, which neither creates a new visit nor makes a Datastore fetch.

Congratulations for completing the Module 12 codelab for adding the use of App Engine memcache service to our sample application. You now have the option to port this Python 2 app to Python 3 in the bonus step.

Clean up

If you are done for now, we recommend you disable your App Engine app to avoid incurring billing. However, if you want to play around with it a bit more, that is fine too. The App Engine platform has a free quota, and as long as you don't exceed that usage tier, you shouldn't be charged. That's for compute, however each App Engine service has its own billing schedule as well:

  • The App Engine Memcache service comes in two different flavors, each with their own pricing structure, so you have to track that usage as it relates to billing.
  • The App Engine Datastore service has a free tier as well, but will charge you if you go beyond those limits. See its pricing page for more information.

For full disclosure, deploying to a Google Cloud serverless compute platform like App Engine incurs minor build and storage costs. Cloud Build has its own free quota as does Cloud Storage. Storage of that image uses up some of that quota. However, you might live in a region that does not have such a free tier, so be aware of your storage usage to minimize potential costs.

On the other hand, if you're not going to continue with this application or other related migration codelabs and want to delete everything completely, you can shut down your project.

Next steps

The next logical migration to consider is covered in Module 13, showing developers how to migrate from App Engine memcache service to Cloud Memorystore. These migrations are all optional and available to users who wish to take various steps to modernize their applications. The Cloud Memorystore service is a significant upgrade to App Engine's memcache for many reasons:

  • Cloud Memorystore is not serverless. This means that you have to allocate a server for the cache. Cloud Memorystore also does not have a free tier. Both of these factors can have a significant impact on cost.
  • Cloud Memorystore supports a pair of different underlying storage mechanisms (caching engines), Redis and Memcached.
  • Cloud Memorystore (for Redis) has a much richer and more in-depth feature set than App Engine Memcache.
  • To use Cloud Memorystore, you must set up a Cloud Memorystore server, add it to a Google Cloud VPC network, then have your App Engine app use that network to communicate with your Memorystore server.

If you don't feel like you need all the features available from Cloud Memorystore or are concerned about its effects on cost, you are free to stay on App Engine Memcache.

Beyond Module 13 are a whole slew of other possible migrations such as Cloud NDB and Cloud Datastore, or Cloud Tasks. There are also cross-product migrations to Cloud Run and Cloud Functions. You'll find them all at the migration repo.

Another possible next step is porting to Python 3, which is covered in the next section as an optional step.

6. BONUS: Migration to Python 3

Overview

This section comprises optional bonus content migrating the Module 12 application we just finished with above to Python 3. We start with configuration followed by the application.

Simplify app.yaml

One of the benefits of the Python 3 runtime is that the app.yaml can be significantly simplified.

BEFORE:

Below is what is in app.yaml at the conclusion of Module 12:

runtime: python27
threadsafe: yes
api_version: 1
 
handlers:
- url: /.*
  script: main.app

Because the Python 3 runtime requires web frameworks to do their own routing, all route handlers in app.yaml must be changed to auto. If there are no static files served, users can just outright remove the entire handlers: section. Also, both threadsafe and api_version have been deprecated.

AFTER:

With the required changes just described, this is the replacement app.yaml for Python 3:

runtime: python39
app_engine_apis: true

The only line that needs an explanation is app_engine_apis: true. When the legacy App Engine services became available to second generation runtimes in 2021, some runtimes, including Python 3, require additional bootstrapping to access those APIs like ndb, taskqueue, and memcache. This line in the configuration serves that purpose.

Update requirements.txt

Another bootstrapping of the original APIs is required in requirements.txt: access to the new App Engine SDK must be included.

BEFORE:

Below is what is in app.yaml at the conclusion of Module 12:

flask

AFTER:

Simply add the App Engine Python SDK, and you should have the following:

flask
appengine-python-standard

Delete appengine_config.py and lib

Delete the appengine_config.py file and the lib folder. In migrating to Python 3, App Engine automatically installs third-party packages listed in requirements.txt. Likewise, the appengine_config.py config file is deprecated for Python 3.

Both of those resources played a big role in accessing third-party packages for Python 2 App Engine, but this is significantly simplified in Python 3 (and other second generation App Engine runtimes). Here is a summary of those changes:

  1. No bundling of copied third-party libraries; now they are listed in requirements.txt
  2. No pip install into a lib folder, meaning no lib folder period
  3. No listing built-in third-party libraries in app.yaml
  4. No need to reference app to third-party libraries, so no appengine_config.py file

Listing all required third-party libraries in requirements.txt is all that's needed with App Engine now.

Update application to use App Engine SDK

As mentioned above, Python 3 apps require some modification to access App Engine bundled services:

  1. Bundle App Engine SDK (in requirements.txt)
  2. Activate App Engine SDK (in app.yaml)
  3. Wrap WSGI object (in main.py)

The first pair were completed above, so the last requirement is to update main.py.

BEFORE:

Below is the Python 2 main.py at the conclusion of Module 12:

from flask import Flask, render_template, request
from google.appengine.api import memcache
from google.appengine.ext import ndb

app = Flask(__name__)
HOUR = 3600

AFTER:

For the Python 3 port, import the SDK and wrap the Flask app object with it (the SDK wrapper), resulting in the following:

from flask import Flask, render_template, request
from google.appengine.api import memcache, wrap_wsgi_app
from google.appengine.ext import ndb

app = Flask(__name__)
app.wsgi_app = wrap_wsgi_app(app.wsgi_app)
HOUR = 3600

Developers need to make these changes to their Python apps when porting from 2.x to 3.x to access the bundled services. If you're not using Flask, there are also Django and Pyramid samples in the docs. If your Python 2 code isn't a web app, just including the SDK package should suffice when porting to Python 3. Our application code was originally crafted to work under Python 2 and 3, so no additional compatibility changes are needed.

Deploy application

After completing the above changes, you can deploy the updated sample app. (There is no issue when deploying a Python 3 version of your app over an original Python 2 version in the same GCP project.) App behavior should remain the same. If you need to compare your updated app to ours, see the Module 12b folder in the migration repo. To learn more about support of App Engine bundled services in the latest runtimes like Python 3, see the feature launch announcement as well as the Module 17 codelab.

Congrats on finishing the bonus step in Module 12! Also see the documentation on preparing configuration files for the Python 3 runtime. Review the Summary/Cleanup section above for next steps and cleanup.

7. Additional resources

Codelab issues/feedback

If you find any issues with this codelab, please search for your issue first before filing. Links to search and create new issues:

Migration resources

Links to the repo folders for Module 2 (START) and Module 12 (FINISH) can be found in the table below. They can also be accessed from the repo for all App Engine codelab migrations which you can clone or download a ZIP file.

Codelab

Python 2

Python 3

Module 1

code

code (not featured in this tutorial)

Module 12 (this codelab)

code

code

Online resources

Below are online resources which may be relevant for this tutorial:

App Engine

Cloud Memorystore and Cloud Datastore

Other Cloud information

Videos

License

This work is licensed under a Creative Commons Attribution 2.0 Generic License.