How to use App Engine Memcache in Flask apps (Module 12)

1. Overview

The Serverless Migration Station series of codelabs (self-paced, hands-on tutorials) and related videos aim to help Google Cloud serverless developers modernize their appications by guiding them through one or more migrations, primarily moving away from legacy services. Doing so makes your apps more portable and gives you more options and flexibility, enabling you to integrate with and access a wider range of Cloud products and more easily upgrade to newer language releases. While initially focusing on the earliest Cloud users, primarily App Engine (standard environment) developers, this series is broad enough to include other serverless platforms like Cloud Functions and Cloud Run, or elsewhere if applicable.

This codelab teaches you how to include and use App Engine Memcache to the sample app from the Module 1 codelab. We add usage of Memcache in this Module 12 tutorial then migrate to Cloud Memorystore next in Module 13.

You'll learn how to

  • Use the App Engine Memcache API/library
  • Add caching to a basic Python 2 Flask App Engine NDB app

What you'll need

Survey

How will you use this tutorial?

Read it through only Read it and complete the exercises

How would you rate your experience with Python?

Novice Intermediate Proficient

How would you rate your experience with using Google Cloud services?

Novice Intermediate Proficient

2. Background

In order to migrate from App Engine Memcache, add its usage to the existing Flask and App Engine NDB app resulting from the Module 1 codelab. The sample app displays the ten most recent visits to the user. If the same user refreshes their browser, it's not optimal to continually create new Visit entities and fetch the most recent visits from Datastore, so we're going to cache those most recent visits.

If the same visitor hits the page, those visits are returned from the cache. If a new user visits the site or an hour has passed, the cache is flushed and replaced with the most recent entries (not to mention a new Visit registered). With this App Engine Memcache integration implemented, we can migrate it to Cloud Memorystore in the next (Module 13) codelab.

This tutorial features the following steps:

  1. Setup/Prework
  2. Update configuration
  3. Modify application code

3. Setup/Prework

Before we get to the main part of the tutorial, let's set up our project, get the code, then deploy the baseline app so we know we started with working code.

1. Setup project

If you completed the Module 1 codelab, we recommend reusing that same project (and code). Alternatively, you can create a brand new project or reuse another existing project. Ensure the project has an active billing account and App Engine is enabled.

2. Get baseline sample app

One of the prerequisites to this codelab is to have a working Module 1 sample app. If you don't have one, go complete either tutorial (links above) before moving ahead here. Otherwise if you're already familiar with its contents, you can just start with the Module 1 code below.

Whether you use yours or ours, the Module 1 code is where we'll START. This codelab walks you through each step, concluding with code that resembles what's in the Module 11 repo folder (FINISH).

The directory of Module 1 STARTing files (yours or ours) should look like this:

$ ls
README.md               main.py                 templates
app.yaml                requirements.txt

3. (Re)Deploy baseline app

Your remaining prework steps to execute now:

  1. Re-familiarize yourself with the gcloud command-line tool
  2. Re-deploy the sample app with gcloud app deploy
  3. Confirm the app runs on App Engine without issue

Once you've successfully executed those steps and see your web app works (with output similar to the below), you're ready to add use of caching to your app.

a7a9d2b80d706a2b.png

4. Update configuration

No changes are necessary to the standard App Engine configuration files (app.yaml, requirements.txt, appengine_config.py).

5. Modify application files

Because we're only adding an App Engine API, there are no external packages involved, meaning no configuration files (app.yaml, requirements.txt, appengine_config.py) need to be updated. There is only one application file, main.py, so all changes in this section affect just that file.

Imports

The most important step is to import the Memcache library, google.appengine.api.memcache. Since we're going to cache the most recent visits for an hour, let's also add a constant for the number of seconds in an hour. Below is what your code looks like before and this change:

BEFORE:

from flask import Flask, render_template, request
from google.appengine.ext import ndb

app = Flask(__name__)

AFTER:

from flask import Flask, render_template, request
from google.appengine.api import memcache
from google.appengine.ext import ndb

app = Flask(__name__)
HOUR = 3600

Add caching with Memcache support

The most significant change is to add use of caching in our application. More specifically, we should cache the most recent visits, check to see if cached visits are available, and attempt to use the cached results as much as possible given our plan. These are the steps the app will take to accomplish our goal:

  1. Set current visit and call it visitor
  2. Attempt to fetch the most recent visits from cache
  3. If the cache is empty or the most recent visitor (visits[0]['visitor']) differs from the current visitor: store this newest visit, fetch the most recent visits, and cache them for an hour.
  4. Display visits to user through the web template

Here are the before and after with these updates:

BEFORE:

@app.route('/')
def root():
    'main application (GET) handler'
    store_visit(request.remote_addr, request.user_agent)
    visits = fetch_visits(10)
    return render_template('index.html', visits=visits)

AFTER:

@app.route('/')
def root():
    'main application (GET) handler'
    # check for (hour-)cached visits
    ip_addr, usr_agt = request.remote_addr, request.user_agent
    visitor = '{}: {}'.format(ip_addr, usr_agt)
    visits = memcache.get('visits')

    # register visit & run DB query if cache empty or new visitor
    if not visits or visits[0]['visitor'] != visitor:
        store_visit(ip_addr, usr_agt)
        visits = list(fetch_visits(10))
        memcache.set('visits', visits, HOUR)  # set() not add()

    return render_template('index.html', visits=visits)

Here is a pictorial representation of the changes that were made:

b1242503602f7bf0.png

This concludes all of the necessary changes for adding the use of App Engine memcache to the Module 1 sample app. Let's build and deploy this app to see it work!

6. Summary/Cleanup

This section wraps up this codelab by deploying the app, verifying it works as intended and in any reflected output. After app validation, perform any clean-up steps and consider next steps.

Deploy and verify application

Re-deploy your app with gcloud app deploy, and confirm the app works. Your code should now match what's in FINISH, the Module 12 folder. The output should be identical to the Module 1 app you deployed earlier:

a7a9d2b80d706a2b.png

All we did was speed up the user experience for the same user. When they refresh, you should get the results directly from the cache, which neither creates a new visit nor makes a Datastore fetch.

Congratulations for completing the Module 12 codelab for adding the use of App Engine memcache service to our sample application. You now have the option to port this Python 2 app to Python 3 in the bonus step.

Clean up

General

If you are done for now, we recommend you disable your App Engine app to avoid incurring billing. However if you wish to test or experiment some more, the App Engine platform has a free quota, and so as long as you don't exceed that usage tier, you shouldn't be charged. That's for compute, but there may also be charges for relevant App Engine services, so check its pricing page for more information. If this migration involves other Cloud services, those are billed separately. In either case, if applicable, see the "Specific to this codelab" section below.

For full disclosure, deploying to a Google Cloud serverless compute platform like App Engine incurs minor build and storage costs. Cloud Build has its own free quota as does Cloud Storage. Storage of that image uses up some of that quota. However, you might live in a region that does not have such a free tier, so be aware of your storage usage to minimize potential costs. Specific Cloud Storage "folders" you should review include:

  • console.cloud.google.com/storage/browser/LOC.artifacts.PROJECT_ID.appspot.com/containers/images
  • console.cloud.google.com/storage/browser/staging.PROJECT_ID.appspot.com
  • The storage links above depend on your PROJECT_ID and *LOC*ation, for example, "us" if your app is hosted in the USA.

On the other hand, if you're not going to continue with this application or other related migration codelabs and want to delete everything completely, shut down your project.

Specific to this codelab

The services listed below are unique to this codelab. Refer to each product's documentation for more information:

Next steps

The next logical migration to consider is covered in Module 13, showing developers how to migrate from App Engine memcache service to Cloud Memorystore. These migrations are all optional and available to users who wish to take various steps to modernize their applications. The Cloud Memorystore service is a significant upgrade to App Engine's memcache for many reasons:

  • Cloud Memorystore is not serverless. This means that you have to allocate a server for the cache. Cloud Memorystore also does not have a free tier. Both of these factors can have a significant impact on cost.
  • Cloud Memorystore supports a pair of different underlying storage mechanisms (caching engines), Redis and Memcached.
  • Cloud Memorystore (for Redis) has a much richer and more in-depth feature set than App Engine Memcache.
  • To use Cloud Memorystore, you must set up a Cloud Memorystore server, add it to a Google Cloud VPC network, then have your App Engine app use that network to communicate with your Memorystore server.

If you don't feel like you need all the features available from Cloud Memorystore or are concerned about its effects on cost, you are free to stay on App Engine Memcache.

Beyond Module 13 are a whole slew of other possible migrations such as Cloud NDB and Cloud Datastore, or Cloud Tasks. There are also cross-product migrations to Cloud Run and Cloud Functions. You'll find them all at the migration repo.

Another possible next step is porting to Python 3, which is covered in the next section as an optional step.

7. BONUS: Migration to Python 3

Overview

This section comprises optional bonus content migrating the Module 12 application we just finished with above to Python 3. We start with configuration followed by the application.

Simplify app.yaml

One of the benefits of the Python 3 runtime is that the app.yaml can be significantly simplified.

BEFORE:

Below is what is in app.yaml at the conclusion of Module 12:

runtime: python27
threadsafe: yes
api_version: 1

handlers:
- url: /.*
  script: main.app

Because the Python 3 runtime requires web frameworks to do their own routing, all route handlers in app.yaml must be changed to auto. If there are no static files served, users can just outright remove the entire handlers: section. Also, both threadsafe and api_version have been deprecated.

AFTER:

With the required changes just described, this is the replacement app.yaml for Python 3:

runtime: python39
app_engine_apis: true

The only line that needs an explanation is app_engine_apis: true. When the legacy App Engine services became available to second generation runtimes in 2021, some runtimes, including Python 3, require additional bootstrapping to access those APIs like ndb, taskqueue, and memcache. This line in the configuration serves that purpose.

Update requirements.txt

Another bootstrapping of the original APIs is required in requirements.txt: access to the new App Engine SDK must be included.

BEFORE:

Below is what is in app.yaml at the conclusion of Module 12:

flask

AFTER:

Simply add the App Engine Python SDK, and you should have the following:

flask
appengine-python-standard

Delete appengine_config.py and lib

Next generation App Engine runtimes revamp 3rd-party package usage:

  • Built-in libraries are those vetted by Google and made available on App Engine servers, likely because they contain C/C++ code which developers aren't allowed to deploy to the cloud—these are no longer available in the 2nd generation runtimes.
  • Copying non-built-in libraries (sometimes called "vendoring" or "self-bundling") is no longer needed in 2nd generation runtimes. Instead, they should be listed in requirements.txt where the build system automatically installs them on your behalf at deploy time.

As a result of those changes to 3rd-party package management, neither the appengine_config.py file nor lib folder are needed, so delete them. In 2nd generation runtimes, App Engine automatically installs third-party packages listed in requirements.txt. Summarizing:

  1. No self-bundled or copied 3rd-party libraries; list them in requirements.txt
  2. No pip install into a lib folder, meaning no lib folder period
  3. No listing built-in 3rd-party libraries (thus no libraries section) in app.yaml; list them in requirements.txt
  4. No 3rd-party libraries to reference from your app means no appengine_config.py file

Listing all desired 3rd-party libraries in requirements.txt is the only developer requirement.

Update application to use App Engine SDK

As mentioned above, Python 3 apps require some modification to access App Engine bundled services:

  1. Bundle App Engine SDK (in requirements.txt)
  2. Activate App Engine SDK (in app.yaml)
  3. Wrap WSGI object (in main.py)

The first pair were completed above, so the last requirement is to update main.py.

BEFORE:

Below is the Python 2 main.py at the conclusion of Module 12:

from flask import Flask, render_template, request
from google.appengine.api import memcache
from google.appengine.ext import ndb

app = Flask(__name__)
HOUR = 3600

AFTER:

For the Python 3 port, import the SDK and wrap the Flask app object with it (the SDK wrapper), resulting in the following:

from flask import Flask, render_template, request
from google.appengine.api import memcache, wrap_wsgi_app
from google.appengine.ext import ndb

app = Flask(__name__)
app.wsgi_app = wrap_wsgi_app(app.wsgi_app)
HOUR = 3600

Developers need to make these changes to their Python apps when porting from 2.x to 3.x to access the bundled services. If you're not using Flask, there are also Django and Pyramid samples in the docs. If your Python 2 code isn't a web app, just including the SDK package should suffice when porting to Python 3. Our application code was originally crafted to work under Python 2 and 3, so no additional compatibility changes are needed.

Deploy application

After completing the above changes, you can deploy the updated sample app. (There is no issue when deploying a Python 3 version of your app over an original Python 2 version in the same GCP project.) App behavior should remain the same. If you need to compare your updated app to ours, see the Module 12b folder in the migration repo. To learn more about support of App Engine bundled services in the latest runtimes like Python 3, see the feature launch announcement as well as the Module 17 codelab.

Congrats on finishing the bonus step in Module 12! Also see the documentation on preparing configuration files for the Python 3 runtime. Review the Summary/Cleanup section above for next steps and cleanup.

8. Additional resources

Listed below are additional resources for developers further exploring this or related Migration Module as well as related products. This includes places to provide feedback on this content, links to the code, and various pieces of documentation you may find useful.

Codelab issues/feedback

If you find any issues with this codelab, please search for your issue first before filing. Links to search and create new issues:

Migration resources

Links to the repo folders for Module 2 (START) and Module 12 (FINISH) can be found in the table below. They can also be accessed from the repo for all App Engine codelab migrations which you can clone or download a ZIP file.

Codelab

Python 2

Python 3

Module 1

code

code (not featured in this tutorial)

Module 12 (this codelab)

code

code

Online references

Below are online resources which may be relevant for this tutorial:

App Engine

Cloud Memorystore and Cloud Datastore

Other Cloud information

Videos

License

This work is licensed under a Creative Commons Attribution 2.0 Generic License.