Migrate from Google App Engine to Cloud Run with Docker (Module 4)

The Google App Engine (GAE) migration modules show App Engine (Standard) developers how to modernize their apps running on the original legacy runtimes to the next generation platform. Once that's accomplished, they can make their apps more portable by explicitly containerizing them for Cloud Run, Google Cloud's container-hosting sister service to App Engine, and other container-hosting services.

This tutorial teaches you how to containerize App Engine apps for deploying to the Cloud Run fully-managed service using Docker, a well-known platform in industry for developing, shipping, and running applications in containers. For Python 2 developers, this tutorial STARTs with the Module 2 Cloud NDB App Engine sample app while Python 3 developers START with the Module 3 Cloud Datastore sample.

You'll learn how to

  • Containerize your app using Docker
  • Deploy container images to Cloud Run

What you'll need

Survey

How will you use this codelab?

Only read through it Read it and complete the exercises

PaaS systems like App Engine and Cloud Functions provide conveniences of serverless platforms like focus away from SysAdmin/DevOps towards building solutions, autoscaling up as needed, scaling down to 0 and pay-per-use billing to control costs, and variety of common development languages. However, the flexibility of containers is compelling as well, the ability to choose any language, any library, any binary. Giving users the best of both worlds, the convenience of serverless along with the flexibility of containers, is what Google Cloud Run is all about.

Learning how to use Cloud Run is not within the scope of this tutorial; that's covered by the Cloud Run documentation. By the end of this tutorial, you'll know how to containerize your App Engine app for Cloud Run (or other services). There are a few things you should know before moving forward, primarily that your user experience will be slightly different, a bit lower-level as you will no longer be taking application code and deploying it.

Instead, you'll need to learn something about containers like how to build and deploy them. You also get to decide what you want to put in the container image, including a web server as you won't be using App Engine's web server any more. If your prefer not to follow this path, keeping your apps on App Engine is not a bad option.

In this tutorial, you'll learn how to containerize your app, replacing App Engine config files with container configuration, determine what goes into the container, then specify how to start your app — many of these things are automatically handled by App Engine.

This migration features these steps:

  1. Setup/Prework
  2. Containerize application
    • Replace configuration files
    • Modify application files

Before we get going with the main part of the tutorial, let's setup our project, get the code, then deploy the baseline app so we know we started with working code.

1. Setup project

If you completed either the Module 2 or Module 3 codelabs, we recommend reusing that same project (and code). Alternatively, you can create a brand new project or reuse another existing project. Ensure the project has an active billing account and App Engine (app) is enabled.

2. "Get" baseline sample app

One of the prerequisites to this codelab is to have a working Module 2 or Module 3 sample app. If you don't have one, go complete either tutorial (links above) before moving ahead here. Otherwise if you're already familiar with its contents, you can just start by grabbing the Module 2 or 3 code below.

Whether you use yours or ours, the Module 2 code is where we'll START for the Python 2 version of this tutorial, and similarly, the Module 3 code for Python 3. This Module 4 codelab walks you through each step, and depending on your options, you should end up with code that resembles one of the Module 4 repo folders (FINISH) when complete.

The directory of Python 2 Module 2 STARTing files (yours or ours) should look like this:

$ ls
README.md               appengine_config.py     requirements.txt
app.yaml                main.py                 templates

If you're using your own Module 2 (2.x) code, you'll also have a lib folder. Neither lib nor appengine_config.py are used for Python 3, where the Module 3 (3.x) STARTing code should look like this:

$ ls
README.md               main.py                 templates
app.yaml                requirements.txt

3. (Re)Deploy baseline app

Your remaining prework steps to execute now:

  1. Re-familiarize yourself with the gcloud command-line tool
  2. Re-deploy sample app with gcloud app deploy
  3. Confirm app runs on App Engine without issue

Once you've successfully executed those steps, you're ready to containerize it.

Docker is the standard containerization platform in industry today. One challenge in using it, as mentioned earlier, is that it requires effort to curating an efficient Dockerfile, the configuration file that determines how your container images are built. The Buildpacks open specification lift this burden off the developer, using introspection to determine your app's dependencies, and builds the most efficient container for your app as possible.

You're in the right place if you already know about containers, Docker, and want to learn more about containerizing your App Engine app for Cloud Run. Feel free to also do the Module 5 codelab (identical to this one but with Cloud Buildpacks) afterwards. Our barebones sample app is lightweight enough to avoid some of those aforementioned Dockerfile issues.

The migration steps include replacing the App Engine configuration files and specifying how your app should start. Below is a table summarizing the configuration files to expect for each platform type:

Description

App Engine

Docker

Buildpacks

General config

app.yaml

Dockerfile

service.yaml

3rd-party libraries

requirements.txt

requirements.txt

requirements.txt

3rd-party config

app.yaml (plus appengine_config.py & lib [2.x-only])

(n/a)

(n/a)

Startup

(n/a) or app.yaml (if entrypoint used)

Dockerfile

Procfile

Ignore files

.gcloudignore & .gitignore

.gcloudignore, .gitignore, & .dockerignore

.gcloudignore & .gitignore

Once your app is containerized, it can be deployed to Cloud Run, as we will do in this tutorial. Other Google Cloud container platform options include Compute Engine, GKE, and Anthos.

General config

Migrating from App Engine means replacing app.yaml with a Dockerfile which outlines how to build & run the container. App Engine automatically starts your application, but Cloud Run doesn't. This is what the Dockerfile ENTRYPOINT and CMD commands are for. Learn more about Dockerfile from this Cloud Run docs page as well as see an example Dockerfile that spawns gunicorn.

An alternative to using ENTRYPOINT or CMD in a Dockerfile is to use a Procfile. Finally, a .dockerignore helps filter out non-app files to keep your container size down. More on these coming up!

Delete app.yaml and create Dockerfile

The

app.yaml

file is not used in containers so delete it now.

The container configuration file is Dockerfile, and our sample app only requires a minimal one. Create your Dockerfile by copying the following into yours, replacing NNN with "2" or "3", depending on which Python version you're using.

FROM python:NNN-slim
WORKDIR /app
COPY . .
RUN pip install -r requirements.txt
ENTRYPOINT ["python", "main.py"]

Most of the Dockerfile specifies how to create the container while the ENTRYPOINT command starts the container; in this case calling python main.py to start the Flask development server. If you're new to Docker, the FROM directive indicates the base image to start from, and "slim" refers to a minimal Python distribution. Learn more from the Docker Python images page.

The middle set of commands creates the working directory (/app), copies in the application files, then runs pip install to bring third-party libraries into the container. WORKDIR combines the Linux mkdir and cd commands together; read more about it in the WORKDIR documentation . The COPY and RUN directives are self-explanatory.

3rd-party libraries

The requirements.txt file can stay the same; Flask should be there along with your Datastore client library (Cloud Datastore or Cloud NDB). If you wish to use another WSGI-compliant HTTP server like Gunicorn — current version at the time of this writing is 20.0.4 — then add gunicorn==20.0.4 to requirements.txt.

3rd-party config

Python 2 App Engine developers know that third-party libraries are either copied into the lib folder, referenced in requirements.txt, itemized in app.yaml, and supported by appengine_config.py. Containers, like Python 3 App Engine apps, only use requirements.txt, so all the other stuff can be dropped, meaning if you have a 2.x App Engine app, delete

appengine_config.py

and any

lib

folder now.

Startup

Python 2 users do not startup App Engine's web server, but Python 3 users have the option of converting their app.yaml files to have an entrypoint instead of script: auto directives in their handlers section. If you use entrypoint in your Python 3 app.yaml, it would look something like this:

runtime: python38
entrypoint: python main.py

The entrypoint directive tells App Engine how to start your server. You can move it almost directly into your Dockerfile; and similarly, such a line would go into Procfile if you used Buildpacks:

  • Docker: This line in Dockerfile: ENTRYPOINT ["python", "main.py"]
  • Buildpacks: A Procfile consisting of just this line: web: python main.py

Using the Flask development server is fine for testing, but if using a production server like gunicorn for your application, be sure to point your ENTRYPOINT or CMD directive at it like in the Cloud Run Quickstart sample.

Ignore files

We recommend creating a .dockerignore file to trim the size of your container and not clutter your container image with superfluous files like these:

*.md
*.pyc
*.pyo
.git/
.gitignore
__pycache__

Application files

Whether you have a Module 2 or Module 3 app, all are fully 2.x & 3.x compatible, meaning there are no changes to the core components of main.py; we will only be adding a few lines of startup code. Most required changes are in configuration.

Add a pair of lines at the bottom of main.py to start the application. Cloud Run automatically "injects" 8080 as the PORT environment variable, so you don't need to set it in Dockerfile:

Cloud Run starts its web server on port 8080, automatically injected into the PORT environment variable. Add an import os at the top of main.py as well as a "main" at the bottom to start the server (if executed directly which our Dockerfile does):

if __name__ == '__main__':
    import os
    app.run(debug=True, threaded=True, host='0.0.0.0',
            port=int(os.environ.get('PORT', 8080)))

With your App Engine configuration replaced by either Docker's, you're ready to deploy.

1. Build Docker image & send to Cloud Container Registry

Execute this command to build your container image and archive to the Cloud Registry:

$ gcloud builds submit --tag gcr.io/PROJECT_ID/IMG_NAME
Creating temporary tarball archive of 5 file(s) totalling 1.9 KiB before compression.
Uploading tarball of [.] to [gs://PROJECT_ID_cloudbuild/source/1605604100.565976-8b4e2297826343d3bc21c36fbd3736b8.tgz]
Created [https://cloudbuild.googleapis.com/v1/projects/PROJECT_ID/builds/8c3900e2-ca90-441f-bf29-b1cfd5cab309].
Logs are available at [https://console.cloud.google.com/cloud-build/builds/8c3900e2-ca90-441f-bf29-b1cfd5cab309?project=1046901301037].
------------------------------------ REMOTE BUILD OUTPUT ------------------------------------
starting build "8c3900e2-ca90-441f-bf29-b1cfd5cab309"

FETCHSOURCE
Fetching storage object: gs://PROJECT_ID_cloudbuild/source/1605604100.565976-8b4e2297826343d3bc21c36fbd3736b8.tgz#1605604101281977
Copying gs://PROJECT_ID_cloudbuild/source/1605604100.565976-8b4e2297826343d3bc21c36fbd3736b8.tgz#1605604101281977...
/ [1 files][  1.3 KiB/  1.3 KiB]
Operation completed over 1 objects/1.3 KiB.
BUILD
Already have image (with digest): gcr.io/cloud-builders/docker
Sending build context to Docker daemon   7.68kB
Module 1/5 : FROM python:3-slim
3-slim: Pulling from library/python
bb79b6b2107f: Already exists
35e30c3f3e2b: Pulling fs layer
b13c2c0e2577: Pulling fs layer
263be93302fa: Pulling fs layer
30e7021a7001: Pulling fs layer
30e7021a7001: Waiting
263be93302fa: Verifying Checksum
263be93302fa: Download complete
35e30c3f3e2b: Verifying Checksum
35e30c3f3e2b: Download complete
b13c2c0e2577: Verifying Checksum
b13c2c0e2577: Download complete
35e30c3f3e2b: Pull complete
30e7021a7001: Verifying Checksum
30e7021a7001: Download complete
b13c2c0e2577: Pull complete
263be93302fa: Pull complete
30e7021a7001: Pull complete
Digest: sha256:c13fda093489a1b699ee84240df4f5d0880112b9e09ac21c5d6875003d1aa927
Status: Downloaded newer image for python:3-slim
 ---> a90139e6bc2f
Module 2/5 : WORKDIR /app
 ---> Running in 17fa98a79d4a
Removing intermediate container 17fa98a79d4a
 ---> cf1f7de92bb0
Module 3/5 : COPY . .
 ---> a4703449acd6
Module 4/5 : RUN pip install -r requirements.txt
 ---> Running in e23df87afa98
Collecting Flask==1.1.2
  Downloading Flask-1.1.2-py2.py3-none-any.whl (94 kB)
Collecting google-cloud-datastore==2.0.0
  Downloading google_cloud_datastore-2.0.0-py2.py3-none-any.whl (144 kB)
Collecting itsdangerous>=0.24
  Downloading itsdangerous-1.1.0-py2.py3-none-any.whl (16 kB)
Collecting Jinja2>=2.10.1
  Downloading Jinja2-2.11.2-py2.py3-none-any.whl (125 kB)
Collecting Werkzeug>=0.15
  Downloading Werkzeug-1.0.1-py2.py3-none-any.whl (298 kB)
Collecting click>=5.1
  Downloading click-7.1.2-py2.py3-none-any.whl (82 kB)
Collecting libcst>=0.2.5
  Downloading libcst-0.3.13-py3-none-any.whl (502 kB)
Collecting google-api-core[grpc]<2.0.0dev,>=1.22.2
  Downloading google_api_core-1.23.0-py2.py3-none-any.whl (91 kB)
Collecting google-cloud-core<2.0dev,>=1.4.0
  Downloading google_cloud_core-1.4.3-py2.py3-none-any.whl (27 kB)
Collecting proto-plus>=1.4.0
  Downloading proto-plus-1.11.0.tar.gz (44 kB)
Collecting MarkupSafe>=0.23
  Downloading MarkupSafe-1.1.1.tar.gz (19 kB)
Collecting pyyaml>=5.2
  Downloading PyYAML-5.3.1.tar.gz (269 kB)
Collecting typing-inspect>=0.4.0
  Downloading typing_inspect-0.6.0-py3-none-any.whl (8.1 kB)
Collecting typing-extensions>=3.7.4.2
  Downloading typing_extensions-3.7.4.3-py3-none-any.whl (22 kB)
Collecting requests<3.0.0dev,>=2.18.0
  Downloading requests-2.25.0-py2.py3-none-any.whl (61 kB)
Collecting googleapis-common-protos<2.0dev,>=1.6.0
  Downloading googleapis_common_protos-1.52.0-py2.py3-none-any.whl (100 kB)
Requirement already satisfied: setuptools>=34.0.0 in /usr/local/lib/python3.9/site-packages (from google-api-core[grpc]<2.0.0dev,>=1.22.2->google-cloud-datastore==2.0.0->-r requirements.txt (line 2)) (50.3.2)
Collecting six>=1.13.0
  Downloading six-1.15.0-py2.py3-none-any.whl (10 kB)
Collecting pytz
  Downloading pytz-2020.4-py2.py3-none-any.whl (509 kB)
Collecting google-auth<2.0dev,>=1.21.1
  Downloading google_auth-1.23.0-py2.py3-none-any.whl (114 kB)
Collecting protobuf>=3.12.0
  Downloading protobuf-3.14.0-py2.py3-none-any.whl (173 kB)
Collecting grpcio<2.0dev,>=1.29.0; extra == "grpc"
  Downloading grpcio-1.33.2-cp39-cp39-manylinux2014_x86_64.whl (3.8 MB)
Collecting mypy-extensions>=0.3.0
  Downloading mypy_extensions-0.4.3-py2.py3-none-any.whl (4.5 kB)
Collecting chardet<4,>=3.0.2
  Downloading chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Collecting certifi>=2017.4.17
  Downloading certifi-2020.11.8-py2.py3-none-any.whl (155 kB)
Collecting idna<3,>=2.5
  Downloading idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting urllib3<1.27,>=1.21.1
  Downloading urllib3-1.26.2-py2.py3-none-any.whl (136 kB)
Collecting pyasn1-modules>=0.2.1
  Downloading pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting cachetools<5.0,>=2.0.0
  Downloading cachetools-4.1.1-py3-none-any.whl (10 kB)
Collecting rsa<5,>=3.1.4; python_version >= "3.5"
  Downloading rsa-4.6-py3-none-any.whl (47 kB)
Collecting pyasn1<0.5.0,>=0.4.6
  Downloading pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Building wheels for collected packages: proto-plus, MarkupSafe, pyyaml
  Building wheel for proto-plus (setup.py): started
  Building wheel for proto-plus (setup.py): finished with status 'done'
  Created wheel for proto-plus: filename=proto_plus-1.11.0-py3-none-any.whl size=41569 sha256=54b12f4c68d426bdd21a73ec422e5df7351f09cd45ad673b93ec13b0a8b8a1ba
  Stored in directory: /root/.cache/pip/wheels/d8/b4/72/1293fbaaa4e6deb58b262d1d0b7aa9713b5a580784d373be0d
  Building wheel for MarkupSafe (setup.py): started
  Building wheel for MarkupSafe (setup.py): finished with status 'done'
  Created wheel for MarkupSafe: filename=MarkupSafe-1.1.1-py3-none-any.whl size=12627 sha256=ddc8a1e4a44df58258de3f31dc13ae3ad179afbf97eb264c8c190d734498766d
  Stored in directory: /root/.cache/pip/wheels/e0/19/6f/6ba857621f50dc08e084312746ed3ebc14211ba30037d5e44e
  Building wheel for pyyaml (setup.py): started
  Building wheel for pyyaml (setup.py): finished with status 'done'
  Created wheel for pyyaml: filename=PyYAML-5.3.1-cp39-cp39-linux_x86_64.whl size=44617 sha256=8e2901e9bb4fc3b9b505bf69cecf826401a3e7ecf7525bb57d63b30804721144
  Stored in directory: /root/.cache/pip/wheels/69/60/81/5cd74b8ee068fbe9e04ca0d53148f28f5c6e2c5b177d5dd622
Successfully built proto-plus MarkupSafe pyyaml
Installing collected packages: itsdangerous, MarkupSafe, Jinja2, Werkzeug, click, Flask, pyyaml, mypy-extensions, typing-extensions, typing-inspect, libcst, chardet, certifi, idna, urllib3, requests, six, protobuf, googleapis-common-protos, pytz, pyasn1, pyasn1-modules, cachetools, rsa, google-auth, grpcio, google-api-core, google-cloud-core, proto-plus, google-cloud-datastore
Successfully installed Flask-1.1.2 Jinja2-2.11.2 MarkupSafe-1.1.1 Werkzeug-1.0.1 cachetools-4.1.1 certifi-2020.11.8 chardet-3.0.4 click-7.1.2 google-api-core-1.23.0 google-auth-1.23.0 google-cloud-core-1.4.3 google-cloud-datastore-2.0.0 googleapis-common-protos-1.52.0 grpcio-1.33.2 idna-2.10 itsdangerous-1.1.0 libcst-0.3.13 mypy-extensions-0.4.3 proto-plus-1.11.0 protobuf-3.14.0 pyasn1-0.4.8 pyasn1-modules-0.2.8 pytz-2020.4 pyyaml-5.3.1 requests-2.25.0 rsa-4.6 six-1.15.0 typing-extensions-3.7.4.3 typing-inspect-0.6.0 urllib3-1.26.2
Removing intermediate container e23df87afa98
 ---> a94cb28e33e9
Module 5/5 : ENTRYPOINT ["python", "main.py"]
 ---> Running in fd5bd0bfa781
Removing intermediate container fd5bd0bfa781
 ---> f8dc8ff328c1
Successfully built f8dc8ff328c1
Successfully tagged gcr.io/PROJECT_ID/IMG_NAME:latest
PUSH
Pushing gcr.io/PROJECT_ID/IMG_NAME
The push refers to repository [gcr.io/PROJECT_ID/IMG_NAME]
cd9b381d5582: Preparing
    . . .
d0fe97fa8b8c: Preparing
83dcc420d3e6: Waiting
225ef82ca30a: Waiting
d0fe97fa8b8c: Waiting
8414f9cd7a8a: Pushed
c5deba9fb608: Pushed
d93c07024f51: Pushed
225ef82ca30a: Layer already exists
d0fe97fa8b8c: Layer already exists
b362843246c4: Pushed
cd9b381d5582: Pushed
83dcc420d3e6: Pushed
latest: digest: sha256:532c3ba8f6f8747ff43437483a92aa46080a960627f0ac413e5cea2524797ce4 size: 1997
DONE
---------------------------------------------------------------------------------------------

ID                                    CREATE_TIME                DURATION  SOURCE                                                                                      IMAGES                                 STATUS
8c3900e2-ca90-441f-bf29-b1cfd5cab309  2020-11-17T09:08:21+00:00  34S       gs://PROJECT_ID_cloudbuild/source/1605604100.565976-8b4e2297826343d3bc21c36fbd3736b8.tgz  gcr.io/PROJECT_ID/IMG_NAME (+1 more)  SUCCESS

2. Deploy Docker image from Container Registry

Execute this command to deploy that image to run on Cloud Run:

$ gcloud run deploy SVC_NAME --platform managed --image gcr.io/PROJECT_ID/IMG_NAME
Please specify a region:
 [1] asia-east1
 [2] asia-east2
 [3] asia-northeast1
 [4] asia-northeast2
 [5] asia-northeast3
 [6] asia-south1
 [7] asia-southeast1
 [8] asia-southeast2
 [9] australia-southeast1
 [10] europe-north1
 [11] europe-west1
 [12] europe-west2
 [13] europe-west3
 [14] europe-west4
 [15] europe-west6
 [16] northamerica-northeast1
 [17] southamerica-east1
 [18] us-central1
 [19] us-east1
 [20] us-east4
 [21] us-west1
 [22] cancel
Please enter your numeric choice:  21

To make this the default region, run `gcloud config set run/region us-west1`.

Deploying container to Cloud Run service [SVC_NAME] in project [PROJECT_ID] region [us-west1]
✓ Deploying... Done.
  ✓ Creating Revision...
  ✓ Routing traffic...
Done.
Service [SVC_NAME] revision [SVC_NAME-00012-waj] has been deployed and is serving 100 percent of traffic.
Service URL: https://SVC_NAME-REGION_HASH.a.run.app

Choose your region, and allow unauthenticated connections for easier testing.

Confirm the app works on Cloud Run identically to that on App Engine. Your code should now match what's in the Module 4 repo folder, whether it's 2.x or 3.x. Congrats for completing this Module 4 codelab.

Optional: Clean up

What about cleaning up to avoid being billed until you're ready to move onto the next migration codelab? Since you're now using a different product, be sure to review the Cloud Run pricing guide.

Optional: Disable service

If you're not ready to go to the next tutorial yet, disable your service to avoid incurring additional charges. When you're ready to move onto the next codelab, you can re-enable it. While your app is disabled, it won't get any traffic to incur charges, however another thing you can get billed for is your Datastore usage if it exceeds the free quota, so delete enough to fall under that limit.

On the other hand, if you're not going to continue with migrations and want to delete everything completely, you can either delete your service or shutdown your project entirely.

Next steps

Congratulations... your app is now containrized, concluding this tutorial. From here, the next step is to learn about how to do the same thing with Cloud Buildpacks in Module 5 codelab (link below) or other App Engine migrations to consider:

  • Migrate to Python 3 if you haven't already. The sample app is already 2.x & 3.x compatible, so the only change is for Docker users: update Dockerfile to use a Python 3 image; Buildpacks auto-detects language and selects appropriate base iamge.
  • Module 5: Migrate to Cloud Run with Cloud Buildpacks
    • Containerize your app to run on Cloud Run with Cloud Buildpacks
    • Do not need to know anything about Docker, containers, or Dockerfiles
    • Requires you to have already migrated your app to Python 3
  • Module 7: App Engine Push Task Queues (required if you use [push] Task Queues)
    • Adds App Engine taskqueue push tasks to Module 1 app
    • Prepares users for migrating to Cloud Tasks in Module 8
  • Module 3:
    • Modernize Datastore access from Cloud NDB to Cloud Datastore
    • This is the library used for Python 3 App Engine apps and non-App Engine apps
  • Module 6: Migrate to Cloud Firestore
    • Migrate to Cloud Firestore to access Firebase features
    • While Cloud Firestore supports Python 2, this codelab is available only in Python 3.

App Engine migration module codelabs issues/feedback

If you find any issues with this codelab, please search for your issue first before filing. Links to search and create new issues:

Migration resources

Links to the repo folders for Module 2 & 3 (START) and Module 4 (FINISH) can be found in the table below. They can also be accessed from the repo for all App Engine codelab migrations.

Codelab

Python 2

Python 3

Module 2

repo

(repo)

Module 3

(repo)

repo

Module 4

repo

repo

App Engine & Cloud Run resources

Below are additional resources regarding this specific migration: