Module 7: Add Push Tasks to Python 2 Flask App Engine ndb sample app

1. Overview

This series of codelabs (self-paced, hands-on tutorials) aims to help Google App Engine (Standard) developers modernize their apps by guiding them through a series of migrations. The most significant step is to move away from original runtime bundled services because the next generation runtimes are more flexible, giving users a greater variety of service options. Moving to the newer generation runtime enables you to integrate with Google Cloud products more easily, use a wider range of supported services, and support current language releases.

This codelab helps users add usage of App Engine push tasks to the sample app from the Module 1 codelab. In addition, this tutorial repeats the Module 2 migration moving from App Engine's ndb library to the Cloud NDB client library.

You'll learn how to

  • Add use of the App Engine taskqueue API/library
  • Add push tasks to a Python 2 Flask and Cloud NDB app
  • Prepare for next step to migrate to Cloud Tasks

What you'll need

Survey

How will you use this codelab?

Only read through it Read it and complete the exercises

2. Background

In order to migrate from App Engine (push) Task Queues, we need to add push tasks to the existing Flask and App Engine ndb app resulting from the Module 1 codelab. Then we can migrate it to Cloud Tasks in the next (Module 8) codelab.

This tutorial's migration features these primary steps:

  1. Setup/Prework
  2. Update application files
    • Update imports
    • Add push task
    • Add task handler
    • Update UI

3. Setup/Prework

Before we get going with the main part of the tutorial, let's set up our project, get the code, then deploy the baseline app so we know we started with working code.

1. Setup project

We recommend reusing the same project as the one you used for completing the Module 1 codelab. Alternatively, you can create a brand new project or reuse another existing project.

2. Get baseline sample app

One of the prerequisites to this codelab is to have a working Module 1 sample app. If you don't have one, we recommend completing the Module 1 tutorial (link above) before moving ahead here. Otherwise if you're already familiar with its contents, you can just start by grabbing the Module 1 code below.

Whether you use yours or ours, the Module 1 code is where we'll START. This Module 2 codelab walks you through each step, and when complete, it should resemble code at the FINISH point (including an optional port from Python 2 to 3).

The directory of Module 1 files (yours or ours) should look like this:

$ ls
README.md               appengine_config.py     requirements.txt
app.yaml                main.py                 templates

If you completed the Module 1 tutorial, you'll also have a lib folder with Flask and its dependencies.

3. (Re)Deploy Module 1 app

Your remaining prework steps to execute now:

  1. Re-familiarize yourself with the gcloud command-line tool (if nec.)
  2. (Re)deploy the Module 1 code to App Engine (if nec.)

Once you've successfully executed those steps and confirm it's operational, we'll move ahead in this tutorial, starting with the configuration files.

4. Update application files

Because we're only adding an App Engine API, there are no external packages involved, meaning no configuration files (app.yaml, requirements.txt, appengine_config.py) need to be updated. There is only one application file, main.py, so all changes in this section affects just that file.

1. Imports

  • BEFORE:
from flask import Flask, render_template, request
from google.appengine.ext import ndb

It's useful to add logging to applications to give the developer (and the user) more information (as long as it's useful). For Python 2 App Engine, this is done by using the Python standard library logging module. For date and time functionality, add usage of the datetime.datetime class as well as the time module. The most important one, of course, is the Task Queue library, google.appengine.api.taskqueue.

The Python best practices of alphabetized group listing order:

  1. Standard library modules first
  2. Third-party globally-installed packages
  3. Locally-installed packages
  4. Application imports

Following that recommendation, your imports should look like this when done:

  • AFTER:
from datetime import datetime
import logging
import time
from flask import Flask, render_template, request
from google.appengine.api import taskqueue
from google.appengine.ext import ndb

2. Add a push task (gather data for task, create and spawn task)

Taking a step back, we know we're adding push tasks to this app, but what will the task perform? As you recall, the sample app registers each visit (GET request to /) by creating a new Visit Entity for it then fetches and displays the 10 most recent Visits in the web UI. None of the oldest visits will ever be used again, so the push task will delete all Visits older than the oldest displayed entry.

To do this, we need to extract the timestamp of the oldest displayed visit then pass that to the task handler. The fetch_visits() function queries for the most recent visits, so add code to save the timestamp of the last Visit:

  • BEFORE:
def fetch_visits(limit):
    return (v.to_dict() for v in Visit.query().order(
            -Visit.timestamp).fetch(limit))

Instead of immediately returning all Visits, we need to save the results, grab the last Visit and save its timestamp, both as a string (to display) and float (to send to the task).

  • AFTER:
def fetch_visits(limit):
    'get most recent visits and add task to delete older visits'
    data = Visit.query().order(-Visit.timestamp).fetch(limit)
    oldest = time.mktime(data[-1].timestamp.timetuple())
    oldest_str = time.ctime(oldest)
    logging.info('Delete entities older than %s' % oldest_str)
    taskqueue.add(url='/trim', params={'oldest': oldest})
    return (v.to_dict() for v in data), oldest_str

The data variable holds the Visits we used to return immediately, so we're just caching it for now. The oldest variable represents the timestamp of the oldest displayed Visit in seconds (as a float) since the epoch, retrieved by (extracting datetime object, morphed to Python time 9-tuple normalized form, then converted to float with time.mktime()). A (human-readable) string version is also created by time.ctime() for display and logging purposes.

A new push task is added, calling the handler (/trim) with oldest as its only parameter. The same payload as the Module 1 fetch_visits() is returned to the caller in addition to oldest as a string. Following good practices, an application log at the INFO level via logging.info().

3. Add a push task handler (code called when task runs)

While deletion of old Visits could've easily been accomplished in fetch_visits(), this was a great excuse to make it a task which is handled asynchronously after fetch_user() returns, and the data is presented to the user. This improves the user experience because there is no delay in waiting for the deletion of the older Datastore entities to complete.

@app.route('/trim', methods=['POST'])
def trim():
    '(push) task queue handler to delete oldest visits'
    oldest = request.form.get('oldest', type=float)
    keys = Visit.query(
            Visit.timestamp < datetime.fromtimestamp(oldest)
    ).fetch(keys_only=True)
    nkeys = len(keys)
    if nkeys:
        logging.info('Deleting %d entities: %s' % (
                nkeys, ', '.join(str(k.id()) for k in keys)))
        ndb.delete_multi(keys)
    else:
        logging.info('No entities older than: %s' % time.ctime(oldest))
    return ''   # need to return SOME string w/200

Push tasks are POSTed to the handler, so that must be specified (default: GET). Once the timestamp of the oldest visit is decoded, a Datastore query to find all entities strictly older than its timestamp is created. None of the actual data is needed, so a faster "keys-only" query is used. The number of entities to delete is logged, and the deletion command (ndb.delete_multi()) given. Logging also occurs if there are no entities to delete. A return value is necessary to go along with the HTTP 200 return code, so use an empty string to be efficient.

4. Update UI (display deletion message in web template)

Update templates/index.html by adding the following snippet after the unnumbered list of Visits but before the closing body tag:

{% if oldest %}
    <b>Deleting visits older than:</b> {{ oldest }}</p>
{% endif %}

5. Summary/Cleanup

Deploy application

Double-check all your config and app updates, re-deploy, and confirm the app (still) works. With the UI change, the output will be slightly different because it will also tell you that it has removed all visits older than the tenth:

Module 7 visitme app

That concludes this codelab. Your code should now match what's in the Module 7 repo. Congrats for completing the first part of the push tasks migration.

Optional: Clean up

What about cleaning up to avoid being billed until you're ready to move onto the next migration codelab? As existing developers, you're likely already up-to-speed on App Engine's pricing information.

Optional: Disable app

If you're not ready to go to the next tutorial yet, disable your app to avoid incurring charges. When you're ready to move onto the next codelab, you can re-enable it. While your app is disabled, it won't get any traffic to incur charges, however another thing you can get billed for is your Datastore usage if it exceeds the free quota, so delete enough to fall under that limit.

On the other hand, if you're not going to continue with migrations and want to delete everything completely, you can shutdown your project.

Next steps

Beyond this tutorial, the next step is the Module 8 codelab (link below) as well as these other migraion modules to consider:

  • Module 8 Migrate to Cloud Tasks
    • Migrate from App Engine taskqueue to Cloud Tasks
    • Prepares users for migrating to Python 3, Cloud Datastore, and Cloud Tasks v2 in Module 9
  • Module 4: Migrate to Cloud Run with Docker
    • Containerize your app to run on Cloud Run with Docker
    • This migration allows you to stay on Python 2.
  • Module 5: Migrate to Cloud Run with Cloud Buildpacks
    • Containerize your app to run on Cloud Run with Cloud Buildpacks
    • You do not need to know anything about Docker, containers, or Dockerfiles.
    • Requires your app to have already migrated to Python 3 (Buildpacks doesn't support Python 2)
  • Module 6: Migrate to Cloud Firestore
    • Migrate to Cloud Firestore to access Firebase features
    • While Cloud Firestore supports Python 2, this codelab is available only in Python 3.

6. Additional resources

App Engine migration module codelabs issues/feedback

If you find any issues with this codelab, please search for your issue first before filing. Links to search and create new issues:

Migration resources

Links to the repo folders for Module 1 (START) and Module 7 (FINISH) can be found in the table below. They can also be accessed from the repo for all App Engine codelab migrations which you can clone or download a ZIP file.

Codelab

Python 2

Python 3

Module 1

code

(n/a)

Module 7

code

(n/a)

App Engine resources

Below are additional resources regarding this specific migration: