1. Overview
This series of codelabs (self-paced, hands-on tutorials) aims to help a Cloud serverless developer, modernize apps by guiding them through one or more migrations, primarily moving away from legacy services. Doing so make apps more portable and gives developers a greater variety of service options and flexibility, enabling you to integrate with Google Cloud products more easily, use a wider range of supported services such as leveraging other Cloud compute platforms, and support newer language releases. While initially focusing on our earliest users, primarily Google App Engine (standard environment) developers, this series will broaden to include modernizing apps running on other serverless platforms.
Previously, developers were required to migrate from App Engine's legacy "bundled services" like Datastore and Memcache before they could upgrade language versions, two potentially challenging endeavors back-to-back. By making many of the key bundled services available in the 2nd-generation App Engine service, developers can now port their apps to the latest runtimes while continuing to use (most of) the bundled services. This codelab walks you through upgrading a sample app from Python 2 to 3 while maintaining use of the Datastore bundled service (via the App Engine NDB library). Use of most bundled services only require a minor update to the code as will be covered in this tutorial, but there are others requiring more extensive changes; these will be covered by "Part 2," a follow-up module and codelab.
You'll learn how to
- Add use of the App Engine
memcache
API/library - Add caching to a Python 2 Flask and Cloud NDB app
- Prepare for next step to migrate to Cloud Memorystore
What you'll need
- A Google Cloud Platform project with an active GCP billing account
- Basic Python skills
- Working knowledge of common Linux commands
- Basic knowledge of developing and deploying App Engine apps
- A working Module 1 App Engine app (your own or copy the repo)
- Recommended: complete the Module 1 codelab
Survey
How will you use this tutorial?
How would you rate your experience with Python?
How would you rate your experience with using Google Cloud services?
2. Background
The original App Engine service launched in 2008 and came with a set of legacy APIs (now known as bundled services) to make it convenient for developers to build and deploy applications globally. Those services include Datastore, Memcache, and Task Queue. While convenient, users became concerned about the portability of their apps when using proprietary APIs tying them to App Engine and wanted their apps to be more portable. This coupled with the fact that many of these bundled services maturing to become their own standalone Cloud products led the App Engine team to launch the next-generation platform in 2018 without them.
Fast-forward to today with Python 2 developers eager to upgrade to Python 3. A 2.x app using bundled services required migrating away from those services before their apps could be ported to 3.x, representing a pair of forced migrations back-to-back, potentially challenging ones too. To aid in this transition, the App Engine team introduced in Fall 2021 a "wormhole" to the past, allowing apps running on next-generation runtimes to access many of those bundled services. While this release does not include all the services available in the original runtimes, major players such as Datastore, Task Queue, and Memcache, are available.
This codelab demonstrates the necessary changes for you to upgrade your app to Python 3 while preserving use of bundled services. The goal is to get your apps running on the latest runtimes, allowing you to then migrate from bundled services to Cloud standalone equivalents or 3rd-party alternatives on your own timelines, rather than having it be a blocker to a 3.x upgrade. While migrating away from bundled services isn't required any more, doing so gives you more portability and flexibility in terms of where your apps can be hosted, including shifting to platforms which may better serve your workloads, or simply staying on App Engine while upgrading to a more modern language release as just described.
The Module 1 Python 2 sample app utilizes the Datastore bundled service via App Engine NDB. The app has already migrated frameworks from webapp2 to Flask—completed in the Module 1 codelab—but with its Datastore usage intact.
3. Setup/Prework
Before we get to the main part of the tutorial, let's set up our project, get the code, then deploy the baseline app so we know we started with something that works.
1. Setup project
If you completed the Module 1 codelab, we recommend reusing that same project (and code). Alternatively, create a brand new Cloud project or reuse another existing project. Ensure the project has an active billing account where the App Engine service has been enabled.
2. Get baseline sample app
One prerequisite is that you must have a working Module 1 sample app. Complete the Module 1 codelab or grab the Module 1 code from the repo (link below). Whether you use yours or ours, this is where we'll begin ("START"). This codelab walks you through each step, concluding with code that resembles what's in the Module 1b repo folder ("FINISH").
- START: Module 1 folder (Python 2)
- FINISH: Module 1b folder (Python 3)
- Entire repo (to clone or download ZIP file)
The contents of the Module 1 folder should have the following contents:
$ ls README.md appengine_config.py requirements.txt app.yaml main.py templates
3. (Re)Deploy baseline app
Follow the remaining prework steps below for your Python 2 code:
- Re-familiarize yourself with the
gcloud
command-line tool - Point to the GCP project with
gcloud config set project
PROJECT_ID
then confirm it has been set withgcloud config list
- Delete the
lib
folder if you have one then runpip install -r requirements.txt -t lib
to copy 3rd-party libraries locally (NOTE: once converted to Python 3, you'll no longer perform this step) - Deploy the app to the cloud with
gcloud app deploy
- Confirm the app runs on App Engine without issues
Once you've successfully executed those steps and see your web app works (with output similar to the below), you're ready to port this app to Python 3, starting with config.
4. Modify configuration files
Add SDK to requirements.txt
The App Engine Python 3 runtime significantly reduces the overhead for using 3rd-party libraries. All that's necessary is listing them in requirements.txt
. To use the bundled services in Python 3, add the App Engine SDK package, appengine-python-standard
to it. The SDK package joins Flask from Module 1:
flask
appengine-python-standard
Update app.yaml
Follow the steps below to apply config changes to your app.yaml
file:
- Replace the
runtime
directive with the supported Python 3 release; for example, specifypython310
for Python 3.10. - Delete both the
threadsafe
andapi_version
directives as neither are used in Python 3. - Delete the
handlers
section entirely since this app only has script handlers. If your app has static file handlers, leave them intact inhandlers
. - The Python 3 runtime doesn't support built-in 3rd-party libraries like the Python 2 runtime does. If your app has a
libraries
section inapp.yaml
, delete the entire section. (Required packages only need to be listed inrequirements.txt
like non-built-in libraries.) Our sample app does not have alibraries
section, so move to the next step. - Create an
app_engine_apis
directive set totrue
to use it—this corresponds with adding the App Engine SDK package torequirements.txt
above.
Summary of the required changes to make to app.yaml
:
BEFORE:
runtime: python27
threadsafe: yes
api_version: 1
handlers:
- url: /.*
script: main.app
AFTER:
runtime: python310
app_engine_apis: true
Other configuration files
Because all 3rd-party packages only need to be listed in requirements.txt
, unless you have something special in appengine_config.py
, it isn't needed, so delete it. Similarly, because all 3rd-party libraries are automatically installed during the build process, there's no need to copy or vendor them, meaning no more pip install
command nor lib
folder, so delete it. Summarizing:
- Delete
appengine_config.py
file - Delete
lib
folder
This concludes all necessary configuration changes.
5. Modify application file(s)
Accessing the majority of the available bundled services in the Python 3 runtime environment requires a short piece of code wrapping the Web Server Gateway Interface (WSGI) application object in main.py
. The wrapper function is google.appengine.api.wrap_wsgi_app()
, and you use it by importing it and wrapping your WSGI object with it. Make the changes below to reflect the required update for Flask in main.py
:
BEFORE:
from flask import Flask, render_template, request
from google.appengine.ext import ndb
app = Flask(__name__)
AFTER:
from flask import Flask, render_template, request
from google.appengine.api import wrap_wsgi_app
from google.appengine.ext import ndb
app = Flask(__name__)
app.wsgi_app = wrap_wsgi_app(app.wsgi_app)
See the documentation for WSGI-wrapping examples for other Python frameworks.
While this example works to give your app access to most of the bundled services in Python 3, others like Blobstore and Mail require additional code. We'll cover those samples in another migration module.
This concludes all of the necessary changes for adding the use of App Engine bundled services to the Module 1 sample app. The application is already Python 2 and 3 compatible, so there are no additional changes to port it to Python 3 other than what you've already done in configuration. The final step: deploy this modified app to the next-generation App Engine Python 3 runtime and confirm the updates succeeded.
6. Summary/Cleanup
Deploy application
Deploy the Python 3 app with gcloud app deploy
, and confirm the app works as it did in Python 2. None of the functionality changes, so the output should be identical to the Module 1 app:
Final notes
- Compare what you have with what's in the Module 1b folder (FINISH) If you took a misstep along the way and adjust as necessary.
- Compare the Module 0
main.py
side-by-side with the Module 1bmain.py
on this page if your app still useswebapp2
, then do the Module 1 codelab to learn how to migrate fromwebapp2
to Flask.
Congratulations for taking the first step to porting your Python 2 App Engine apps to Python 3 while retaining their use of the bundled services at this time.
Clean up
After completing this codelab, you can disable your App Engine app (temporarily or permanently) to avoid incurring billing. The App Engine platform has a free quota, so you won't be billed as long as you stay within its usage tier. The same applies to Datastore; see the Cloud Datastore pricing page for more details.
Deploying to platforms like App Engine incurs minor build and storage costs. In some regions, Cloud Build has its own free quota as does Cloud Storage. Builds consume some of that quota. Be aware of your storage usage to minimize potential costs, especially if your region doesn't have such a free tier.
However, if you're not going to continue with any other migration codelabs and want to delete everything completely, shut down your Cloud projects.
Next steps
There are several directions to go from here:
- Update code using bundled services requiring more code changes
- Migrate from bundled services to Cloud standalone products
- Migrate from App Engine to another Cloud serverless platform
Accessing other bundled services like Blobstore, Mail, and Deferred require more code changes and covered by these module(s):
- Module 20 (when available): Extending support for App Engine bundled services (Part 2)
The following modules show developers how to migrate away from App Engine bundled services to Cloud standalone replacements:
- Module 2: App Engine NDB to Cloud NDB
- Modules 7-9: App Engine Task Queues (push tasks) to Cloud Tasks
- Modules 12-13: App Engine Memcache to Cloud Memorystore
- Modules 15-16: App Engine Blobstore to Cloud Storage
- Modules 18-19 (coming soon): App Engine Task Queues (pull tasks) to Cloud Pub/Sub
Switching platforms shouldn't be taken lightly, so you should have good reasons for doing so. If you have a small App Engine app or one that has limited functionality, wish to turn it into a standalone microservice, or wish to break-up a monolithic app into multiple reusable components, all of those are good reasons to consider moving to Cloud Functions. If containerization has become part of your software development workflow, especially if it consists of a CI/CD pipeline, you may wish to consider migrating to Cloud Run. These scenarios are covered by the following modules:
- Module 11: Migrate from App Engine to Cloud Functions
- Modules 4-5: Containerize App Engine apps for Cloud Run with or without Docker
7. Additional resources
Codelab issues/feedback
If you find any issues with this codelab, please search for your issue first before filing. Links to search and create new issues:
Migration resources
Links to the repo folders for Module 1 (START) and Module 1b (FINISH) can be found in the table below. They can also be accessed from the repo for all App Engine codelab migrations.
Codelab | Python 2 | Python 3 |
N/A | ||
Module 1b (this codelab) | N/A |
Online resources
Below are online resources which may be relevant for this tutorial:
App Engine bundled services
- Accessing bundled services in the Python 3 next-generation runtime
- Module 0 app (Python 2) vs. Module 1b app (Python 3) side-by-side comparison
- App Engine SDK web framework WSGI object wrapper samples
- Support for App Engine bundled services in next-generation runtimes launch (2021)
App Engine general docs
- App Engine documentation
- Python 2 App Engine (standard environment) runtime
- Python 3 App Engine (standard environment) runtime
- Differences between Python 2 & 3 App Engine (standard environment) runtimes
- Python 2 to 3 App Engine (standard environment) migration guide
- App Engine pricing and quotas information
- Second generation App Engine platform launch (2018)
- Long-term support for legacy runtimes
- Documentation migration samples repo
- Community-contributed migration samples repo
Other Cloud information
- Python on Google Cloud Platform
- Google Cloud Python client libraries
- Google Cloud "Always Free" tier
- Google Cloud SDK (
gcloud
command-line tool) - All Google Cloud documentation
Videos
- Serverless Migration Station
- Serverless Expeditions
- Subscribe to Google Cloud Tech
- Subscribe to Google Developers
License
This work is licensed under a Creative Commons Attribution 2.0 Generic License.