Extending support for App Engine bundled services: Part 1 (Module 17)

1. Overview

The Serverless Migration Station series of codelabs (self-paced, hands-on tutorials) and related videos aim to help Google Cloud serverless developers modernize their appications by guiding them through one or more migrations, primarily moving away from legacy services. Doing so makes your apps more portable and gives you more options and flexibility, enabling you to integrate with and access a wider range of Cloud products and more easily upgrade to newer language releases. While initially focusing on the earliest Cloud users, primarily App Engine (standard environment) developers, this series is broad enough to include other serverless platforms like Cloud Functions and Cloud Run, or elsewhere if applicable.

Previously, developers were required to migrate from App Engine's legacy "bundled services" like Datastore and Memcache before they could upgrade language versions, two potentially challenging endeavors back-to-back. By making many of the key bundled services available in the 2nd-generation App Engine service, developers can now port their apps to the latest runtimes while continuing to use (most of) the bundled services. This codelab walks you through upgrading a sample app from Python 2 to 3 while maintaining use of the Datastore bundled service (via the App Engine NDB library). Use of most bundled services only require a minor update to the code as will be covered in this tutorial, but there are others requiring more extensive changes; these will be covered by "Part 2," a follow-up module and codelab.

You'll learn how to

  • Port sample App Engine app from Python 2 to 3
  • Update app configuration to include the App Engine SDK
  • Add SDK code to the app supporting bundled services in 2nd generation runtimes like Python 3

What you'll need

Survey

How will you use this tutorial?

Read it through only Read it and complete the exercises

How would you rate your experience with Python?

Novice Intermediate Proficient

How would you rate your experience with using Google Cloud services?

Novice Intermediate Proficient

2. Background

The original App Engine service launched in 2008 and came with a set of legacy APIs (now known as bundled services) to make it convenient for developers to build and deploy applications globally. Those services include Datastore, Memcache, and Task Queue. While convenient, users became concerned about the portability of their apps when using proprietary APIs tying them to App Engine and wanted their apps to be more portable. This coupled with the fact that many of these bundled services maturing to become their own standalone Cloud products led the App Engine team to launch the next-generation platform in 2018 without them.

Fast-forward to today with Python 2 developers eager to upgrade to Python 3. A 2.x app using bundled services required migrating away from those services before their apps could be ported to 3.x, representing a pair of forced migrations back-to-back, potentially challenging ones too. To aid in this transition, the App Engine team introduced in Fall 2021 a "wormhole" to the past, allowing apps running on next-generation runtimes to access many of those bundled services. While this release does not include all the services available in the original runtimes, major players such as Datastore, Task Queue, and Memcache, are available.

This codelab demonstrates the necessary changes for you to upgrade your app to Python 3 while preserving use of bundled services. The goal is to get your apps running on the latest runtimes, allowing you to then migrate from bundled services to Cloud standalone equivalents or 3rd-party alternatives on your own timelines, rather than having it be a blocker to a 3.x upgrade. While migrating away from bundled services isn't required any more, doing so gives you more portability and flexibility in terms of where your apps can be hosted, including shifting to platforms which may better serve your workloads, or simply staying on App Engine while upgrading to a more modern language release as just described.

The Module 1 Python 2 sample app utilizes the Datastore bundled service via App Engine NDB. The app has already migrated frameworks from webapp2 to Flask—completed in the Module 1 codelab—but with its Datastore usage intact.

This tutorial features the following steps:

  1. Setup/Prework
  2. Update configuration
  3. Modify application code

3. Setup/Prework

This section explains how to:

  1. Set up your Cloud project
  2. Get baseline sample app
  3. (Re)Deploy and validate baseline app

These steps ensure you're starting with working code.

1. Setup project

If you completed the Module 1 codelab, we recommend reusing that same project (and code). Alternatively, create a brand new Cloud project or reuse another existing project. Ensure the project has an active billing account where the App Engine service has been enabled.

2. Get baseline sample app

One of the prerequisites to this codelab is to have a working Module 1 App Engine app: complete the Module 1 codelab (recommended) or copy the Module 1 app from the repo. Whether you use yours or ours, the Module 1 code is where we'll "START." This codelab walks you through each step, concluding with code that resembles what's in the Module 7 repo folder "FINISH".

Regardless which Module 1 app you use, the folder should look like the below, possibly with a lib folder as well:

$ ls
README.md               appengine_config.py     requirements.txt
app.yaml                main.py                 templates

3. (Re)Deploy baseline app

Execute the following steps to (re)deploy the Module 1 app:

  1. Delete the lib folder if there is one and run: pip install -t lib -r requirements.txt to repopulate lib. You may need to use the pip2 command instead if you have both Python 2 and 3 installed.
  2. Ensure you've installed and initialized the gcloud command-line tool and reviewed its usage.
  3. Set your Cloud project with gcloud config set project PROJECT_ID if you don't want to enter your PROJECT_ID with each gcloud command issued.
  4. Deploy the sample app with gcloud app deploy
  5. Confirm the Module 1 app runs as expected without issue displaying the most recent visits (illustrated below)

a7a9d2b80d706a2b.png

4. Update configuration

Once you've successfully executed those steps and see your web app works, you're ready to port this app to Python 3, starting with config.

Add SDK to requirements.txt

The App Engine Python 3 runtime significantly reduces the overhead for using 3rd-party libraries. All that's necessary is listing them in requirements.txt. To use the bundled services in Python 3, add the App Engine SDK package, appengine-python-standard to it. The SDK package joins Flask from Module 1:

flask
appengine-python-standard

Update app.yaml

Follow the steps below to apply config changes to your app.yaml file:

  1. Replace the runtime directive with the supported Python 3 release; for example, specify python310 for Python 3.10.
  2. Delete both the threadsafe and api_version directives as neither are used in Python 3.
  3. Delete the handlers section entirely since this app only has script handlers. If your app has static file handlers, leave them intact in handlers.
  4. The Python 3 runtime doesn't support built-in 3rd-party libraries like the Python 2 runtime does. If your app has a libraries section in app.yaml, delete the entire section. (Required packages only need to be listed in requirements.txt like non-built-in libraries.) Our sample app does not have a libraries section, so move to the next step.
  5. Create an app_engine_apis directive set to true to use it—this corresponds with adding the App Engine SDK package to requirements.txt above.

Summary of the required changes to make to app.yaml:

BEFORE:

runtime: python27
threadsafe: yes
api_version: 1

handlers:
- url: /.*
  script: main.app

AFTER:

runtime: python310
app_engine_apis: true

Other configuration files

Because all 3rd-party packages only need to be listed in requirements.txt, unless you have something special in appengine_config.py, it isn't needed, so delete it. Similarly, because all 3rd-party libraries are automatically installed during the build process, there's no need to copy or vendor them, meaning no more pip install command nor lib folder, so delete it. Summarizing:

  • Delete appengine_config.py file
  • Delete lib folder

This concludes all necessary configuration changes.

5. Modify application code

Accessing the majority of the available bundled services in the Python 3 runtime environment requires a short piece of code wrapping the Web Server Gateway Interface (WSGI) application object in main.py. The wrapper function is google.appengine.api.wrap_wsgi_app(), and you use it by importing it and wrapping your WSGI object with it. Make the changes below to reflect the required update for Flask in main.py:

BEFORE:

from flask import Flask, render_template, request
from google.appengine.ext import ndb

app = Flask(__name__)

AFTER:

from flask import Flask, render_template, request
from google.appengine.api import wrap_wsgi_app
from google.appengine.ext import ndb

app = Flask(__name__)
app.wsgi_app = wrap_wsgi_app(app.wsgi_app)

See the documentation for WSGI-wrapping examples for other Python frameworks.

While this example works to give your app access to most of the bundled services in Python 3, others like Blobstore and Mail require additional code. We'll cover those samples in another migration module.

This concludes all of the necessary changes for adding the use of App Engine bundled services to the Module 1 sample app. The application is already Python 2 and 3 compatible, so there are no additional changes to port it to Python 3 other than what you've already done in configuration. The final step: deploy this modified app to the next-generation App Engine Python 3 runtime and confirm the updates succeeded.

6. Summary/Cleanup

This section wraps up this codelab by deploying the app, verifying it works as intended and in any reflected output. After app validation, perform any clean-up and consider next steps.

Deploy and verify application

Deploy the Python 3 app with gcloud app deploy, and confirm the app works as it did in Python 2. None of the functionality changes, so the output should be identical to the Module 1 app:

a7a9d2b80d706a2b.png

Final notes

Congratulations for taking the first step to porting your Python 2 App Engine apps to Python 3 while retaining their use of the bundled services at this time.

Clean up

General

If you are done for now, we recommend you disable your App Engine app to avoid incurring billing. However if you wish to test or experiment some more, the App Engine platform has a free quota, and so as long as you don't exceed that usage tier, you shouldn't be charged. That's for compute, but there may also be charges for relevant App Engine services, so check its pricing page for more information. If this migration involves other Cloud services, those are billed separately. In either case, if applicable, see the "Specific to this codelab" section below.

For full disclosure, deploying to a Google Cloud serverless compute platform like App Engine incurs minor build and storage costs. Cloud Build has its own free quota as does Cloud Storage. Storage of that image uses up some of that quota. However, you might live in a region that does not have such a free tier, so be aware of your storage usage to minimize potential costs. Specific Cloud Storage "folders" you should review include:

  • console.cloud.google.com/storage/browser/LOC.artifacts.PROJECT_ID.appspot.com/containers/images
  • console.cloud.google.com/storage/browser/staging.PROJECT_ID.appspot.com
  • The storage links above depend on your PROJECT_ID and *LOC*ation, for example, "us" if your app is hosted in the USA.

On the other hand, if you're not going to continue with this application or other related migration codelabs and want to delete everything completely, shut down your project.

Specific to this codelab

The services listed below are unique to this codelab. Refer to each product's documentation for more information:

Next steps

There are several directions to go from here:

  1. Update code using bundled services requiring more code changes
  2. Migrate from bundled services to Cloud standalone products
  3. Migrate from App Engine to another Cloud serverless platform

Accessing other bundled services like Blobstore, Mail, and Deferred require more code changes. Migration modules focusing on moving away from App Engine legacy bundled services to consider include:

App Engine is no longer the only serverless platform in Google Cloud. If you have a small App Engine app or one that has limited functionality and wish to turn it into a standalone microservice, or you want to break-up a monolithic app into multiple reusable components, these are good reasons to consider moving to Cloud Functions. If containerization has become part of your application development workflow, particularly if it consists of a CI/CD (continuous integration/continuous delivery or deployment) pipeline, consider migrating to Cloud Run. These scenarios are covered by the following modules:

  • Migrate from App Engine to Cloud Functions: see Module 11
  • Migrate from App Engine to Cloud Run: see Module 4 to containerize your app with Docker, or Module 5 to do it without containers, Docker knowledge, or Dockerfiles

Switching to another serverless platform is optional, and we recommend considering the best options for your apps and use cases before making any changes.

Regardless of which migration module you consider next, all Serverless Migration Station content (codelabs, videos, source code [when available]) can be accessed at its open source repo. The repo's README also provides guidance on which migrations to consider and any relevant "order" of Migration Modules.

7. Additional resources

Listed below are additional resources for developers further exploring this or related Migration Module as well as related products. This includes places to provide feedback on this content, links to the code, and various pieces of documentation you may find useful.

Codelab issues/feedback

If you find any issues with this codelab, please search for your issue first before filing. Links to search and create new issues:

Migration resources

Links to the repo folders for Module 1 (START) and Module 1b (FINISH) can be found in the table below. They can also be accessed from the repo for all App Engine codelab migrations.

Codelab

Python 2

Python 3

Module 1

code

N/A

Module 17 (this codelab)

N/A

code (mod1b-flask)

Online resources

Below are online resources which may be relevant for this tutorial:

App Engine bundled services

App Engine general docs

Other Cloud information

Videos

License

This work is licensed under a Creative Commons Attribution 2.0 Generic License.