1. Overview
This series of codelabs (self-paced, hands-on tutorials) aims to help Google App Engine (Standard) developers modernize their apps by guiding them through a series of migrations. The most significant step is to move away from original runtime bundled services because the next generation runtimes are more flexible, giving users a greater variety of service options. Moving to the newer generation runtime enables you to integrate with Google Cloud products more easily, use a wider range of supported services, and support current language releases.
This codelab helps users add usage of App Engine push tasks to the sample app from the Module 1 codelab. In addition, this tutorial repeats the Module 2 migration moving from App Engine's ndb
library to the Cloud NDB client library.
You'll learn how to
- Add use of the App Engine
taskqueue
API/library - Add push tasks to a Python 2 Flask and Cloud NDB app
- Prepare for next step to migrate to Cloud Tasks
What you'll need
- A Google Cloud Platform project with:
- Basic Python skills
- Working knowledge of common Linux commands
- Basic knowledge of developing and deploying App Engine apps
- You're recommended to complete the Module 1 codelab before starting this one (Module 7).
- A working Module 1 App Engine app (your own or copy the repo)
Survey
How will you use this codelab?
2. Background
In order to migrate from App Engine (push) Task Queues, we need to add push tasks to the existing Flask and App Engine ndb
app resulting from the Module 1 codelab. Then we can migrate it to Cloud Tasks in the next (Module 8) codelab.
This tutorial's migration features these primary steps:
- Setup/Prework
- Update application files
- Update imports
- Add push task
- Add task handler
- Update UI
3. Setup/Prework
Before we get going with the main part of the tutorial, let's set up our project, get the code, then deploy the baseline app so we know we started with working code.
1. Setup project
We recommend reusing the same project as the one you used for completing the Module 1 codelab. Alternatively, you can create a brand new project or reuse another existing project.
2. Get baseline sample app
One of the prerequisites to this codelab is to have a working Module 1 sample app. If you don't have one, we recommend completing the Module 1 tutorial (link above) before moving ahead here. Otherwise if you're already familiar with its contents, you can just start by grabbing the Module 1 code below.
Whether you use yours or ours, the Module 1 code is where we'll START. This Module 2 codelab walks you through each step, and when complete, it should resemble code at the FINISH point (including an optional port from Python 2 to 3).
- START: Module 1 repo
- FINISH: Module 7 repo
- Entire repo (clone or download ZIP)
The directory of Module 1 files (yours or ours) should look like this:
$ ls
README.md appengine_config.py requirements.txt
app.yaml main.py templates
If you completed the Module 1 tutorial, you'll also have a lib
folder with Flask and its dependencies.
3. (Re)Deploy Module 1 app
Your remaining prework steps to execute now:
- Re-familiarize yourself with the
gcloud
command-line tool (if nec.) - (Re)deploy the Module 1 code to App Engine (if nec.)
Once you've successfully executed those steps and confirm it's operational, we'll move ahead in this tutorial, starting with the configuration files.
4. Update application files
Because we're only adding an App Engine API, there are no external packages involved, meaning no configuration files (app.yaml
, requirements.txt
, appengine_config.py
) need to be updated. There is only one application file, main.py
, so all changes in this section affects just that file.
1. Imports
- BEFORE:
from flask import Flask, render_template, request
from google.appengine.ext import ndb
It's useful to add logging to applications to give the developer (and the user) more information (as long as it's useful). For Python 2 App Engine, this is done by using the Python standard library logging
module. For date and time functionality, add usage of the datetime.datetime
class as well as the time
module. The most important one, of course, is the Task Queue library, google.appengine.api.taskqueue
.
The Python best practices of alphabetized group listing order:
- Standard library modules first
- Third-party globally-installed packages
- Locally-installed packages
- Application imports
Following that recommendation, your imports should look like this when done:
- AFTER:
from datetime import datetime
import logging
import time
from flask import Flask, render_template, request
from google.appengine.api import taskqueue
from google.appengine.ext import ndb
2. Add a push task (gather data for task, create and spawn task)
Taking a step back, we know we're adding push tasks to this app, but what will the task perform? As you recall, the sample app registers each visit (GET
request to /
) by creating a new Visit
Entity for it then fetches and displays the 10 most recent Visit
s in the web UI. None of the oldest visits will ever be used again, so the push task will delete all Visit
s older than the oldest displayed entry.
To do this, we need to extract the timestamp of the oldest displayed visit then pass that to the task handler. The fetch_visits()
function queries for the most recent visits, so add code to save the timestamp of the last Visit
:
- BEFORE:
def fetch_visits(limit):
return (v.to_dict() for v in Visit.query().order(
-Visit.timestamp).fetch(limit))
Instead of immediately returning all Visit
s, we need to save the results, grab the last Visit
and save its timestamp, both as a str
ing (to display) and float
(to send to the task).
- AFTER:
def fetch_visits(limit):
'get most recent visits and add task to delete older visits'
data = Visit.query().order(-Visit.timestamp).fetch(limit)
oldest = time.mktime(data[-1].timestamp.timetuple())
oldest_str = time.ctime(oldest)
logging.info('Delete entities older than %s' % oldest_str)
taskqueue.add(url='/trim', params={'oldest': oldest})
return (v.to_dict() for v in data), oldest_str
The data
variable holds the Visit
s we used to return immediately, so we're just caching it for now. The oldest
variable represents the timestamp of the oldest displayed Visit
in seconds (as a float
) since the epoch, retrieved by (extracting datetime
object, morphed to Python time 9-tuple normalized form, then converted to float
with time.mktime()
). A (human-readable) string version is also created by time.ctime()
for display and logging purposes.
A new push task is added, calling the handler (/trim
) with oldest
as its only parameter. The same payload as the Module 1 fetch_visits()
is returned to the caller in addition to oldest
as a string. Following good practices, an application log at the INFO
level via logging.info()
.
3. Add a push task handler (code called when task runs)
While deletion of old Visit
s could've easily been accomplished in fetch_visits()
, this was a great excuse to make it a task which is handled asynchronously after fetch_user()
returns, and the data is presented to the user. This improves the user experience because there is no delay in waiting for the deletion of the older Datastore entities to complete.
@app.route('/trim', methods=['POST'])
def trim():
'(push) task queue handler to delete oldest visits'
oldest = request.form.get('oldest', type=float)
keys = Visit.query(
Visit.timestamp < datetime.fromtimestamp(oldest)
).fetch(keys_only=True)
nkeys = len(keys)
if nkeys:
logging.info('Deleting %d entities: %s' % (
nkeys, ', '.join(str(k.id()) for k in keys)))
ndb.delete_multi(keys)
else:
logging.info('No entities older than: %s' % time.ctime(oldest))
return '' # need to return SOME string w/200
Push tasks are POST
ed to the handler, so that must be specified (default: GET
). Once the timestamp of the oldest
visit is decoded, a Datastore query to find all entities strictly older than its timestamp is created. None of the actual data is needed, so a faster "keys-only" query is used. The number of entities to delete is logged, and the deletion command (ndb.delete_multi()
) given. Logging also occurs if there are no entities to delete. A return value is necessary to go along with the HTTP 200 return code, so use an empty string to be efficient.
4. Update UI (display deletion message in web template)
Update templates/index.html
by adding the following snippet after the unnumbered list of Visit
s but before the closing body
tag:
{% if oldest %}
<b>Deleting visits older than:</b> {{ oldest }}</p>
{% endif %}
5. Summary/Cleanup
Deploy application
Double-check all your config and app updates, re-deploy, and confirm the app (still) works. With the UI change, the output will be slightly different because it will also tell you that it has removed all visits older than the tenth:
That concludes this codelab. Your code should now match what's in the Module 7 repo. Congrats for completing the first part of the push tasks migration.
Optional: Clean up
What about cleaning up to avoid being billed until you're ready to move onto the next migration codelab? As existing developers, you're likely already up-to-speed on App Engine's pricing information.
Optional: Disable app
If you're not ready to go to the next tutorial yet, disable your app to avoid incurring charges. When you're ready to move onto the next codelab, you can re-enable it. While your app is disabled, it won't get any traffic to incur charges, however another thing you can get billed for is your Datastore usage if it exceeds the free quota, so delete enough to fall under that limit.
On the other hand, if you're not going to continue with migrations and want to delete everything completely, you can shutdown your project.
Next steps
Beyond this tutorial, the next step is the Module 8 codelab (link below) as well as these other migraion modules to consider:
- Module 8 Migrate to Cloud Tasks
- Migrate from App Engine
taskqueue
to Cloud Tasks - Prepares users for migrating to Python 3, Cloud Datastore, and Cloud Tasks v2 in Module 9
- Migrate from App Engine
- Module 4: Migrate to Cloud Run with Docker
- Containerize your app to run on Cloud Run with Docker
- This migration allows you to stay on Python 2.
- Module 5: Migrate to Cloud Run with Cloud Buildpacks
- Containerize your app to run on Cloud Run with Cloud Buildpacks
- You do not need to know anything about Docker, containers, or
Dockerfile
s. - Requires your app to have already migrated to Python 3 (Buildpacks doesn't support Python 2)
- Module 6: Migrate to Cloud Firestore
- Migrate to Cloud Firestore to access Firebase features
- While Cloud Firestore supports Python 2, this codelab is available only in Python 3.
6. Additional resources
App Engine migration module codelabs issues/feedback
If you find any issues with this codelab, please search for your issue first before filing. Links to search and create new issues:
Migration resources
Links to the repo folders for Module 1 (START) and Module 7 (FINISH) can be found in the table below. They can also be accessed from the repo for all App Engine codelab migrations which you can clone or download a ZIP file.
Codelab | Python 2 | Python 3 |
(n/a) | ||
Module 7 | (n/a) |
App Engine resources
Below are additional resources regarding this specific migration:
- App Engine
ndb
references - App Engine
taskqueue
references - Migrating to Python 3 and GAE next-generation runtime
- General