This series of codelab tutorials aims to help App Engine developers modernize their apps. The most significant step is to move away from original runtime bundled services because the next generation runtimes are more flexible in giving users more options. This codelab helps users add use of App Engine push tasks to the sample app from the Module 1 codelab.
In addition, this tutorial repeats the Module 2 migration moving from App Engine's ndb
library to the Cloud NDB client library.
You'll learn how to
- Add use of the App Engine
taskqueue
API/library - Add push tasks to a Python 2 Flask & Cloud NDB app
- Prepare for next step to migrate to Cloud Tasks
What you'll need
- A Google Cloud Platform project with:
- Basic Python skills
- Working knowledge of common Linux commands
- Basic knowledge of developing & deploying App Engine apps
- A working Module 1 App Engine app (your own or copy the repo)
Survey
How will you use this codelab?
In order to migration from App Engine (push) Task Queues, we need to add push tasks to the existing Flask and App Engine ndb
app resulting from the Module 1 codelab. Then we can migrate it to Cloud Tasks in the following tutorial.
This tutorial's migration features these primary steps:
- Setup/Prework
- Update application files
- Update imports
- Add push task
- Add task handler
- Update UI
There are 3 objectives in this part of the tutorial:
1. Setup project
We recommend reusing the same project as the one you used for completing the Module 1 codelab. Alternatively, you can create a brand new project or reuse another existing project.
2. "Get" baseline sample app
One of the prerequisites to this codelab is to have a working Module 1 sample app. If you don't have one, we recommend completing the Module 1 tutorial (link above) before moving ahead here. Otherwise if you're already familiar with its contents, you can just start by grabbing the Module 1 code below.
Whether you use yours or ours, the Module 1 code is where we'll START. This Module 2 codelab walks you through each step, and when complete, it should resemble code at the FINISH point (including an optional port from Python 2 to 3).
- START: Module 1 repo
- FINISH: Module 7 repo
- Entire repo (clone or download ZIP)
The directory of Module 1 files (yours or ours) should look like this:
$ ls
README.md appengine_config.py requirements.txt
app.yaml main.py templates
If you completed the Module 1 tutorial, you'll also have a lib
folder with Flask and its dependencies.
3. (Re)Deploy Module 1 app
Your remaining prework steps to execute now:
- Re-familiarize yourself with the
gcloud
command-line tool (if nec.) - (Re)deploy the Module 1 code to App Engine (if nec.)
Once you've successfully executed those steps and confirm it's operational, we'll move ahead in this tutorial, starting with the configuration files.
Because we're only adding an App Engine API, there are no external packages involved, meaning no configuration files (app.yaml
, requirements.txt
, appengine_config.py
) need to be updated.
1. Imports
- BEFORE:
from flask import Flask, render_template, request
from google.appengine.ext import ndb
It's useful to add logging to applications to give the developer (and the user) more information (as long as it's useful). For Python 2 App Engine, this is done by using the Python standard library logging
module. For date & time functionality, add use of the datetime.datetime
class as well as the time
module. The most important one, of course, is the Task Queue library, google.appengine.api.taskqueue
.
The Python best practices of alphabetized group listing order:
- Standard library modules first
- Third-party globally-installed packages
- Locally-installed packages
- Application imports
Following that recommendation, your imports should look like this when done:
- AFTER:
from datetime import datetime
import logging
import time
from flask import Flask, render_template, request
from google.appengine.api import taskqueue
from google.appengine.ext import ndb
2. Add a push task (gather data for task, create & spawn task)
Taking a step back, we know we're adding push tasks to this app, but what will the task perform? As you recall, the sample app registers each visit (GET
request to /
) by creating a new Visit
Entity for it then fetches and displays the 10 most recent Visit
s in the web UI. None of the oldest visits will ever be used again, so the push task will delete all Visit
s older than the oldest displayed entry.
To do this, we need to extract the timestamp of the oldest displayed visit then pass that to the task handler. The fetch_visits()
function queries for the most recent visits, so add code to save the timestamp of the last Visit
:
- BEFORE:
def fetch_visits(limit):
return (v.to_dict() for v in Visit.query().order(
-Visit.timestamp).fetch(limit))
Instead of immediately returning all Visit
s, we need to save the results, grab the last Visit
and save its timestamp, both as a str
ing (to display) and float
(to send to the task).
- AFTER:
def fetch_visits(limit):
'get most recent visits & add task to delete older visits'
data = Visit.query().order(-Visit.timestamp).fetch(limit)
oldest = time.mktime(data[-1].timestamp.timetuple())
oldest_str = time.ctime(oldest)
logging.info('Delete entities older than %s' % oldest_str)
taskqueue.add(url='/trim', params={'oldest': oldest})
return (v.to_dict() for v in data), oldest_str
The data
variable holds the Visit
s we used to return immediately, so we're just caching it for now. The oldest
variable represents the timestamp of the oldest displayed Visit
in seconds (as a float
) since the epoch, retrieved by (extracting datetime
object, morphed to Python time 9-tuple normalized form, then converted to float
with time.mktime()
). A (human-readable) string version is also created (by time.ctime()
for display and logging purposes.
A new push task is added, calling the handler (/trim
) with oldest
as its only parameter. The same payload as the Module 1 fetch_visits()
is returned to the caller in addition to oldest
as a string. Following good practices, an application log at the INFO
level via logging.info()
.
3. Add a push task handler (code called when task runs)
While deletion of old Visit
s could've easily been accomplished in fetch_visits()
, this was a great excuse to make it a task which is handled asynchronously after fetch_user()
returns, and the data is presented to the user. This improves the user experience because there is no delay in waiting for the deletion of the older Datastore entities to complete.
@app.route('/trim', methods=['POST'])
def trim():
'(push) task queue handler to delete oldest visits'
oldest = request.form.get('oldest', type=float)
keys = Visit.query(
Visit.timestamp < datetime.fromtimestamp(oldest)
).fetch(keys_only=True)
nkeys = len(keys)
if nkeys:
logging.info('Deleting %d entities: %s' % (
nkeys, ', '.join(str(k.id()) for k in keys)))
ndb.delete_multi(keys)
else:
logging.info('No entities older than: %s' % time.ctime(oldest))
return '' # need to return SOME string w/200
Push tasks are POST
ed to the handler, so that must be specified (default: GET
). Once the timestamp of the oldest
visit is decoded, a Datastore query to find all entities strictly older than its timestamp is created. None of the actual data is needed, so a faster "keys-only" query is used. The number of entities to delete is logged, and the deletion command (ndb.delete_multi()
) given. Logging also occurs if there are no entities to delete. A return value is necessary to go along with the HTTP 200 return code, so use an empty string to be efficient.
4. Update UI (display deletion message in web template)
Update templates/index.html
by adding the following snippet after the unnumbered list of Visit
s but before the closing body
tag:
{% if oldest %}
<b>Deleting visits older than:</b> {{ oldest }}</p>
{% endif %}
Deploy application
Doublecheck your updates per the tutorial, re-deploy your app with gcloud app deploy
, and confirm the app works. With the UI change, the output will be slightly different:
That concludes this codelab. Your code should now match what's in the Module 7 repo. Congrats for completing the first part of the push tasks migration.
Optional: Clean up
What about cleaning up to avoid being billed until you're ready to move onto the next migration codelab? As existing developers, you're likely already up-to-speed on App Engine's pricing information.
Optional: Disable app
If you're not ready to go to the next tutorial yet, disable your app to avoid incurring charges. When you're ready to move onto the next codelab, you can re-enable it. While your app is disabled, it won't get any traffic to incur charges, however another thing you can get billed for is your Datastore usage if it exceeds the free quota, so delete enough to fall under that limit.
On the other hand, if you're not going to continue with migrations and want to delete everything completely, you can shutdown your project.
Next steps
Beyond this tutorial, the next step is the Module 8 codelab (link below) as well as these other migraion modules to consider:
- Module 8 Migrate to Cloud Tasks
- Migrate from App Engine
taskqueue
to Cloud Tasks - Prepares users for migrating to Python 3, Cloud Datastore, and Cloud Tasks v2 in Module 9
- Migrate from App Engine
- Module 4: Migrate to Cloud Run with Docker
- Containerize your app to run on Cloud Run with Docker
- Allows you to stay on Python 2
- Module 5: Migrate to Cloud Run with Cloud Buildpacks
- Containerize your app to run on Cloud Run with Cloud Buildpacks
- Do not need to know anything about Docker, containers, or
Dockerfile
s - Requires you to have already migrated your app to Python 3
- Module 6: Migrate to Cloud Firestore
- Migrate to Cloud Firestore to access Firebase features
- While Cloud Firestore supports Python 2, this codelab is available only in Python 3.
App Engine migration module codelabs issues/feedback
If you find any issues with this codelab, please search for your issue first before filing. Links to search and create new issues:
Migration resources
Links to the repo folders for Module 1 (START) and Module 7 (FINISH) can be found in the table below. They can also be accessed from the repo for all App Engine codelab migrations.
Codelab | Python 2 | Python 3 |
(n/a) | ||
Module 7 | (n/a) |
App Engine resources
Below are additional resources regarding this specific migration:
- App Engine
ndb
references - App Engine
taskqueue
references - Migrating to Python 3 & GAE next-generation runtime
- General