Module 17: Extending support for App Engine bundled services (Part 1)

1. Overview

This series of codelabs (self-paced, hands-on tutorials) aims to help a Cloud serverless developer, modernize apps by guiding them through one or more migrations, primarily moving away from legacy services. Doing so make apps more portable and gives developers a greater variety of service options and flexibility, enabling you to integrate with Google Cloud products more easily, use a wider range of supported services such as leveraging other Cloud compute platforms, and support newer language releases. While initially focusing on our earliest users, primarily Google App Engine (standard environment) developers, this series will broaden to include modernizing apps running on other serverless platforms.

Previously, developers were required to migrate from App Engine's legacy "bundled services" like Datastore and Memcache before they could upgrade language versions, two potentially challenging endeavors back-to-back. By making many of the key bundled services available in the 2nd-generation App Engine service, developers can now port their apps to the latest runtimes while continuing to use (most of) the bundled services. This codelab walks you through upgrading a sample app from Python 2 to 3 while maintaining use of the Datastore bundled service (via the App Engine NDB library). Use of most bundled services only require a minor update to the code as will be covered in this tutorial, but there are others requiring more extensive changes; these will be covered by "Part 2," a follow-up module and codelab.

You'll learn how to

  • Add use of the App Engine memcache API/library
  • Add caching to a Python 2 Flask and Cloud NDB app
  • Prepare for next step to migrate to Cloud Memorystore

What you'll need

Survey

How will you use this tutorial?

Read it through only Read it and complete the exercises

How would you rate your experience with Python?

Novice Intermediate Proficient

How would you rate your experience with using Google Cloud services?

Novice Intermediate Proficient

2. Background

The original App Engine service launched in 2008 and came with a set of legacy APIs (now known as bundled services) to make it convenient for developers to build and deploy applications globally. Those services include Datastore, Memcache, and Task Queue. While convenient, users became concerned about the portability of their apps when using proprietary APIs tying them to App Engine and wanted their apps to be more portable. This coupled with the fact that many of these bundled services maturing to become their own standalone Cloud products led the App Engine team to launch the next-generation platform in 2018 without them.

Fast-forward to today with Python 2 developers eager to upgrade to Python 3. A 2.x app using bundled services required migrating away from those services before their apps could be ported to 3.x, representing a pair of forced migrations back-to-back, potentially challenging ones too. To aid in this transition, the App Engine team introduced in Fall 2021 a "wormhole" to the past, allowing apps running on next-generation runtimes to access many of those bundled services. While this release does not include all the services available in the original runtimes, major players such as Datastore, Task Queue, and Memcache, are available.

This codelab demonstrates the necessary changes for you to upgrade your app to Python 3 while preserving use of bundled services. The goal is to get your apps running on the latest runtimes, allowing you to then migrate from bundled services to Cloud standalone equivalents or 3rd-party alternatives on your own timelines, rather than having it be a blocker to a 3.x upgrade. While migrating away from bundled services isn't required any more, doing so gives you more portability and flexibility in terms of where your apps can be hosted, including shifting to platforms which may better serve your workloads, or simply staying on App Engine while upgrading to a more modern language release as just described.

The Module 1 Python 2 sample app utilizes the Datastore bundled service via App Engine NDB. The app has already migrated frameworks from webapp2 to Flask—completed in the Module 1 codelab—but with its Datastore usage intact.

3. Setup/Prework

Before we get to the main part of the tutorial, let's set up our project, get the code, then deploy the baseline app so we know we started with something that works.

1. Setup project

If you completed the Module 1 codelab, we recommend reusing that same project (and code). Alternatively, create a brand new Cloud project or reuse another existing project. Ensure the project has an active billing account where the App Engine service has been enabled.

2. Get baseline sample app

One prerequisite is that you must have a working Module 1 sample app. Complete the Module 1 codelab or grab the Module 1 code from the repo (link below). Whether you use yours or ours, this is where we'll begin ("START"). This codelab walks you through each step, concluding with code that resembles what's in the Module 1b repo folder ("FINISH").

The contents of the Module 1 folder should have the following contents:

$ ls
README.md               appengine_config.py     requirements.txt
app.yaml                main.py                 templates

3. (Re)Deploy baseline app

Follow the remaining prework steps below for your Python 2 code:

  1. Re-familiarize yourself with the gcloud command-line tool
  2. Point to the GCP project with gcloud config set project PROJECT_ID then confirm it has been set with gcloud config list
  3. Delete the lib folder if you have one then run pip install -r requirements.txt -t lib to copy 3rd-party libraries locally (NOTE: once converted to Python 3, you'll no longer perform this step)
  4. Deploy the app to the cloud with gcloud app deploy
  5. Confirm the app runs on App Engine without issues

Once you've successfully executed those steps and see your web app works (with output similar to the below), you're ready to port this app to Python 3, starting with config.

a7a9d2b80d706a2b.png

4. Modify configuration files

Add SDK to requirements.txt

The App Engine Python 3 runtime significantly reduces the overhead for using 3rd-party libraries. All that's necessary is listing them in requirements.txt. To use the bundled services in Python 3, add the App Engine SDK package, appengine-python-standard to it. The SDK package joins Flask from Module 1:

flask
appengine-python-standard

Update app.yaml

Follow the steps below to apply config changes to your app.yaml file:

  1. Replace the runtime directive with the supported Python 3 release; for example, specify python310 for Python 3.10.
  2. Delete both the threadsafe and api_version directives as neither are used in Python 3.
  3. Delete the handlers section entirely since this app only has script handlers. If your app has static file handlers, leave them intact in handlers.
  4. The Python 3 runtime doesn't support built-in 3rd-party libraries like the Python 2 runtime does. If your app has a libraries section in app.yaml, delete the entire section. (Required packages only need to be listed in requirements.txt like non-built-in libraries.) Our sample app does not have a libraries section, so move to the next step.
  5. Create an app_engine_apis directive set to true to use it—this corresponds with adding the App Engine SDK package to requirements.txt above.

Summary of the required changes to make to app.yaml:

BEFORE:

runtime: python27
threadsafe: yes
api_version: 1

handlers:
- url: /.*
  script: main.app

AFTER:

runtime: python310
app_engine_apis: true

Other configuration files

Because all 3rd-party packages only need to be listed in requirements.txt, unless you have something special in appengine_config.py, it isn't needed, so delete it. Similarly, because all 3rd-party libraries are automatically installed during the build process, there's no need to copy or vendor them, meaning no more pip install command nor lib folder, so delete it. Summarizing:

  • Delete appengine_config.py file
  • Delete lib folder

This concludes all necessary configuration changes.

5. Modify application file(s)

Accessing the majority of the available bundled services in the Python 3 runtime environment requires a short piece of code wrapping the Web Server Gateway Interface (WSGI) application object in main.py. The wrapper function is google.appengine.api.wrap_wsgi_app(), and you use it by importing it and wrapping your WSGI object with it. Make the changes below to reflect the required update for Flask in main.py:

BEFORE:

from flask import Flask, render_template, request
from google.appengine.ext import ndb

app = Flask(__name__)

AFTER:

from flask import Flask, render_template, request
from google.appengine.api import wrap_wsgi_app
from google.appengine.ext import ndb

app = Flask(__name__)
app.wsgi_app = wrap_wsgi_app(app.wsgi_app)

See the documentation for WSGI-wrapping examples for other Python frameworks.

While this example works to give your app access to most of the bundled services in Python 3, others like Blobstore and Mail require additional code. We'll cover those samples in another migration module.

This concludes all of the necessary changes for adding the use of App Engine bundled services to the Module 1 sample app. The application is already Python 2 and 3 compatible, so there are no additional changes to port it to Python 3 other than what you've already done in configuration. The final step: deploy this modified app to the next-generation App Engine Python 3 runtime and confirm the updates succeeded.

6. Summary/Cleanup

Deploy application

Deploy the Python 3 app with gcloud app deploy, and confirm the app works as it did in Python 2. None of the functionality changes, so the output should be identical to the Module 1 app:

a7a9d2b80d706a2b.png

Final notes

Congratulations for taking the first step to porting your Python 2 App Engine apps to Python 3 while retaining their use of the bundled services at this time.

Clean up

After completing this codelab, you can disable your App Engine app (temporarily or permanently) to avoid incurring billing. The App Engine platform has a free quota, so you won't be billed as long as you stay within its usage tier. The same applies to Datastore; see the Cloud Datastore pricing page for more details.

Deploying to platforms like App Engine incurs minor build and storage costs. In some regions, Cloud Build has its own free quota as does Cloud Storage. Builds consume some of that quota. Be aware of your storage usage to minimize potential costs, especially if your region doesn't have such a free tier.

However, if you're not going to continue with any other migration codelabs and want to delete everything completely, shut down your Cloud projects.

Next steps

There are several directions to go from here:

  1. Update code using bundled services requiring more code changes
  2. Migrate from bundled services to Cloud standalone products
  3. Migrate from App Engine to another Cloud serverless platform

Accessing other bundled services like Blobstore, Mail, and Deferred require more code changes and covered by these module(s):

  • Module 20 (when available): Extending support for App Engine bundled services (Part 2)

The following modules show developers how to migrate away from App Engine bundled services to Cloud standalone replacements:

  • Module 2: App Engine NDB to Cloud NDB
  • Modules 7-9: App Engine Task Queues (push tasks) to Cloud Tasks
  • Modules 12-13: App Engine Memcache to Cloud Memorystore
  • Modules 15-16: App Engine Blobstore to Cloud Storage
  • Modules 18-19 (coming soon): App Engine Task Queues (pull tasks) to Cloud Pub/Sub

Switching platforms shouldn't be taken lightly, so you should have good reasons for doing so. If you have a small App Engine app or one that has limited functionality, wish to turn it into a standalone microservice, or wish to break-up a monolithic app into multiple reusable components, all of those are good reasons to consider moving to Cloud Functions. If containerization has become part of your software development workflow, especially if it consists of a CI/CD pipeline, you may wish to consider migrating to Cloud Run. These scenarios are covered by the following modules:

7. Additional resources

Codelab issues/feedback

If you find any issues with this codelab, please search for your issue first before filing. Links to search and create new issues:

Migration resources

Links to the repo folders for Module 1 (START) and Module 1b (FINISH) can be found in the table below. They can also be accessed from the repo for all App Engine codelab migrations.

Codelab

Python 2

Python 3

Module 1

code

N/A

Module 1b (this codelab)

N/A

code

Online resources

Below are online resources which may be relevant for this tutorial:

App Engine bundled services

App Engine general docs

Other Cloud information

Videos

License

This work is licensed under a Creative Commons Attribution 2.0 Generic License.