The Serverless Migration Station series of codelabs (self-paced, hands-on tutorials) and related videos aim to help Google Cloud serverless developers modernize their appications by guiding them through one or more migrations, primarily moving away from legacy services. Doing so makes your apps more portable and gives you more options and flexibility, enabling you to integrate with and access a wider range of Cloud products and more easily upgrade to newer language releases. While initially focusing on the earliest Cloud users, primarily App Engine (standard environment) developers, this series is broad enough to include other serverless platforms like Cloud Functions and Cloud Run, or elsewhere if applicable.
Previously, developers were required to migrate from App Engine's legacy "bundled services" like Datastore and Memcache before they could upgrade language versions, two potentially challenging endeavors back-to-back. By making many of the key bundled services available in the 2nd-generation App Engine service, developers can now port their apps to the latest runtimes while continuing to use (most of) the bundled services. This codelab walks you through upgrading a sample app from Python 2 to 3 while maintaining use of the Datastore bundled service (via the App Engine NDB library). Use of most bundled services only require a minor update to the code as will be covered in this tutorial, but there are others requiring more extensive changes; these will be covered by "Part 2," a follow-up module and codelab.
You'll learn how to
- Port sample App Engine app from Python 2 to 3
- Update app configuration to include the App Engine SDK
- Add SDK code to the app supporting bundled services in 2nd generation runtimes like Python 3
What you'll need
- A Google Cloud project with an active GCP billing account
- Basic Python skills
- Working knowledge of common Linux commands
- Basic knowledge of developing and deploying App Engine apps
- A working Module 1 App Engine app (complete its codelab [recommended] or copy the app from the repo)
How will you use this tutorial?
How would you rate your experience with Python?
How would you rate your experience with using Google Cloud services?
The original App Engine service launched in 2008 and came with a set of legacy APIs (now known as bundled services) to make it convenient for developers to build and deploy applications globally. Those services include Datastore, Memcache, and Task Queue. While convenient, users became concerned about the portability of their apps when using proprietary APIs tying them to App Engine and wanted their apps to be more portable. This coupled with the fact that many of these bundled services maturing to become their own standalone Cloud products led the App Engine team to launch the next-generation platform in 2018 without them.
Fast-forward to today with Python 2 developers eager to upgrade to Python 3. A 2.x app using bundled services required migrating away from those services before their apps could be ported to 3.x, representing a pair of forced migrations back-to-back, potentially challenging ones too. To aid in this transition, the App Engine team introduced in Fall 2021 a "wormhole" to the past, allowing apps running on next-generation runtimes to access many of those bundled services. While this release does not include all the services available in the original runtimes, major players such as Datastore, Task Queue, and Memcache, are available.
This codelab demonstrates the necessary changes for you to upgrade your app to Python 3 while preserving use of bundled services. The goal is to get your apps running on the latest runtimes, allowing you to then migrate from bundled services to Cloud standalone equivalents or 3rd-party alternatives on your own timelines, rather than having it be a blocker to a 3.x upgrade. While migrating away from bundled services isn't required any more, doing so gives you more portability and flexibility in terms of where your apps can be hosted, including shifting to platforms which may better serve your workloads, or simply staying on App Engine while upgrading to a more modern language release as just described.
The Module 1 Python 2 sample app utilizes the Datastore bundled service via App Engine NDB. The app has already migrated frameworks from webapp2 to Flask—completed in the Module 1 codelab—but with its Datastore usage intact.
This tutorial features the following steps:
- Update configuration
- Modify application code
This section explains how to:
- Set up your Cloud project
- Get baseline sample app
- (Re)Deploy and validate baseline app
These steps ensure you're starting with working code.
1. Setup project
If you completed the Module 1 codelab, we recommend reusing that same project (and code). Alternatively, create a brand new Cloud project or reuse another existing project. Ensure the project has an active billing account where the App Engine service has been enabled.
2. Get baseline sample app
One of the prerequisites to this codelab is to have a working Module 1 App Engine app: complete the Module 1 codelab (recommended) or copy the Module 1 app from the repo. Whether you use yours or ours, the Module 1 code is where we'll "START." This codelab walks you through each step, concluding with code that resembles what's in the Module 7 repo folder "FINISH".
- START: Module 1 folder (Python 2)
- FINISH: Module 1b folder (Python 3)
- Entire repo (to clone or download ZIP file)
Regardless which Module 1 app you use, the folder should look like the below, possibly with a
lib folder as well:
$ ls README.md appengine_config.py requirements.txt app.yaml main.py templates
3. (Re)Deploy baseline app
Execute the following steps to (re)deploy the Module 1 app:
- Delete the
libfolder if there is one and run:
pip install -t lib -r requirements.txtto repopulate
lib. You may need to use the
pip2command instead if you have both Python 2 and 3 installed.
- Ensure you've installed and initialized the
gcloudcommand-line tool and reviewed its usage.
- Set your Cloud project with
gcloud config set project
PROJECT_IDif you don't want to enter your
- Deploy the sample app with
gcloud app deploy
- Confirm the Module 1 app runs as expected without issue displaying the most recent visits (illustrated below)
4. Update configuration
Once you've successfully executed those steps and see your web app works, you're ready to port this app to Python 3, starting with config.
Add SDK to requirements.txt
The App Engine Python 3 runtime significantly reduces the overhead for using 3rd-party libraries. All that's necessary is listing them in
requirements.txt. To use the bundled services in Python 3, add the App Engine SDK package,
appengine-python-standard to it. The SDK package joins Flask from Module 1:
Follow the steps below to apply config changes to your
- Replace the
runtimedirective with the supported Python 3 release; for example, specify
python310for Python 3.10.
- Delete both the
api_versiondirectives as neither are used in Python 3.
- Delete the
handlerssection entirely since this app only has script handlers. If your app has static file handlers, leave them intact in
- The Python 3 runtime doesn't support built-in 3rd-party libraries like the Python 2 runtime does. If your app has a
app.yaml, delete the entire section. (Required packages only need to be listed in
requirements.txtlike non-built-in libraries.) Our sample app does not have a
librariessection, so move to the next step.
- Create an
app_engine_apisdirective set to
trueto use it—this corresponds with adding the App Engine SDK package to
Summary of the required changes to make to
runtime: python27 threadsafe: yes api_version: 1 handlers: - url: /.* script: main.app
runtime: python310 app_engine_apis: true
Other configuration files
Because all 3rd-party packages only need to be listed in
requirements.txt, unless you have something special in
appengine_config.py, it isn't needed, so delete it. Similarly, because all 3rd-party libraries are automatically installed during the build process, there's no need to copy or vendor them, meaning no more
pip install command nor
lib folder, so delete it. Summarizing:
This concludes all necessary configuration changes.
5. Modify application code
Accessing the majority of the available bundled services in the Python 3 runtime environment requires a short piece of code wrapping the Web Server Gateway Interface (WSGI) application object in
main.py. The wrapper function is
google.appengine.api.wrap_wsgi_app(), and you use it by importing it and wrapping your WSGI object with it. Make the changes below to reflect the required update for Flask in
from flask import Flask, render_template, request from google.appengine.ext import ndb app = Flask(__name__)
from flask import Flask, render_template, request from google.appengine.api import wrap_wsgi_app from google.appengine.ext import ndb app = Flask(__name__) app.wsgi_app = wrap_wsgi_app(app.wsgi_app)
See the documentation for WSGI-wrapping examples for other Python frameworks.
While this example works to give your app access to most of the bundled services in Python 3, others like Blobstore and Mail require additional code. We'll cover those samples in another migration module.
This concludes all of the necessary changes for adding the use of App Engine bundled services to the Module 1 sample app. The application is already Python 2 and 3 compatible, so there are no additional changes to port it to Python 3 other than what you've already done in configuration. The final step: deploy this modified app to the next-generation App Engine Python 3 runtime and confirm the updates succeeded.
This section wraps up this codelab by deploying the app, verifying it works as intended and in any reflected output. After app validation, perform any clean-up and consider next steps.
Deploy and verify application
Deploy the Python 3 app with
gcloud app deploy, and confirm the app works as it did in Python 2. None of the functionality changes, so the output should be identical to the Module 1 app:
- Compare what you have with what's in the Module 1b folder (FINISH) If you took a misstep along the way and adjust as necessary.
- Compare the Module 0
main.pyside-by-side with the Module 1b
main.pyon this page if your app still uses
webapp2, then do the Module 1 codelab to learn how to migrate from
Congratulations for taking the first step to porting your Python 2 App Engine apps to Python 3 while retaining their use of the bundled services at this time.
If you are done for now, we recommend you disable your App Engine app to avoid incurring billing. However if you wish to test or experiment some more, the App Engine platform has a free quota, and so as long as you don't exceed that usage tier, you shouldn't be charged. That's for compute, but there may also be charges for relevant App Engine services, so check its pricing page for more information. If this migration involves other Cloud services, those are billed separately. In either case, if applicable, see the "Specific to this codelab" section below.
For full disclosure, deploying to a Google Cloud serverless compute platform like App Engine incurs minor build and storage costs. Cloud Build has its own free quota as does Cloud Storage. Storage of that image uses up some of that quota. However, you might live in a region that does not have such a free tier, so be aware of your storage usage to minimize potential costs. Specific Cloud Storage "folders" you should review include:
- The storage links above depend on your
LOC*ation, for example, "
us" if your app is hosted in the USA.
On the other hand, if you're not going to continue with this application or other related migration codelabs and want to delete everything completely, shut down your project.
Specific to this codelab
The services listed below are unique to this codelab. Refer to each product's documentation for more information:
- The App Engine Datastore service is provided by Cloud Datastore (Cloud Firestore in Datastore mode) which also has a free tier; see its pricing page for more information.
There are several directions to go from here:
- Update code using bundled services requiring more code changes
- Migrate from bundled services to Cloud standalone products
- Migrate from App Engine to another Cloud serverless platform
- Module 2: App Engine NDB to Cloud NDB
- Modules 7-9: App Engine TaskQueue (push tasks) to Cloud Tasks
- Modules 12-13: App Engine Memcache to Cloud Memorystore
- Modules 15-16: App Engine Blobstore to Cloud Storage
- Modules 18-19: App Engine TaskQueue (pull tasks) to Cloud Pub/Sub
App Engine is no longer the only serverless platform in Google Cloud. If you have a small App Engine app or one that has limited functionality and wish to turn it into a standalone microservice, or you want to break-up a monolithic app into multiple reusable components, these are good reasons to consider moving to Cloud Functions. If containerization has become part of your application development workflow, particularly if it consists of a CI/CD (continuous integration/continuous delivery or deployment) pipeline, consider migrating to Cloud Run. These scenarios are covered by the following modules:
- Migrate from App Engine to Cloud Functions: see Module 11
- Migrate from App Engine to Cloud Run: see Module 4 to containerize your app with Docker, or Module 5 to do it without containers, Docker knowledge, or
Switching to another serverless platform is optional, and we recommend considering the best options for your apps and use cases before making any changes.
Regardless of which migration module you consider next, all Serverless Migration Station content (codelabs, videos, source code [when available]) can be accessed at its open source repo. The repo's
README also provides guidance on which migrations to consider and any relevant "order" of Migration Modules.
7. Additional resources
Listed below are additional resources for developers further exploring this or related Migration Module as well as related products. This includes places to provide feedback on this content, links to the code, and various pieces of documentation you may find useful.
If you find any issues with this codelab, please search for your issue first before filing. Links to search and create new issues:
Links to the repo folders for Module 1 (START) and Module 1b (FINISH) can be found in the table below. They can also be accessed from the repo for all App Engine codelab migrations.
Module 17 (this codelab)
Below are online resources which may be relevant for this tutorial:
App Engine bundled services
- Accessing bundled services in the Python 3 next-generation runtime
- Module 0 app (Python 2) vs. Module 1b app (Python 3) side-by-side comparison
- App Engine SDK web framework WSGI object wrapper samples
- Support for App Engine bundled services in next-generation runtimes launch (2021)
App Engine general docs
- App Engine documentation
- Python 2 App Engine (standard environment) runtime
- Using App Engine built-in libraries on Python 2 App Engine
- Python 3 App Engine (standard environment) runtime
- Differences between Python 2 & 3 App Engine (standard environment) runtimes
- Python 2 to 3 App Engine (standard environment) migration guide
- App Engine pricing and quotas information
- Second generation App Engine platform launch (2018)
- Long-term support for legacy runtimes
- Documentation migration samples repo
- Community-contributed migration samples repo
Other Cloud information
- Python on Google Cloud Platform
- Google Cloud Python client libraries
- Google Cloud "Always Free" tier
- Google Cloud SDK (
- All Google Cloud documentation
- Serverless Migration Station
- Serverless Expeditions
- Subscribe to Google Cloud Tech
- Subscribe to Google Developers
This work is licensed under a Creative Commons Attribution 2.0 Generic License.