How to use App Engine Task Queue (push tasks) in Flask apps (Module 7)

1. Overview

The Serverless Migration Station series of codelabs (self-paced, hands-on tutorials) and related videos aim to help Google Cloud serverless developers modernize their appications by guiding them through one or more migrations, primarily moving away from legacy services. Doing so makes your apps more portable and gives you more options and flexibility, enabling you to integrate with and access a wider range of Cloud products and more easily upgrade to newer language releases. While initially focusing on the earliest Cloud users, primarily App Engine (standard environment) developers, this series is broad enough to include other serverless platforms like Cloud Functions and Cloud Run, or elsewhere if applicable.

This codelab teaches you how to use App Engine Task Queue push tasks in the sample app from the Module 1 codelab. The Module 7 blog post and video complement this tutorial, providing a brief overview of the content in this tutorial.

In this module, we will add the use of push tasks, then migrate that usage to Cloud Tasks in Module 8 and later to Python 3 and Cloud Datastore in Module 9. Those using Task Queues for pull tasks will migrate to Cloud Pub/Sub and should refer to Modules 18-19 instead.

You'll learn how to

  • Use the App Engine Task Queue API/bundled service
  • Add push task usage to a basic Python 2 Flask App Engine NDB app

What you'll need

Survey

How will you use this tutorial?

Read it through only Read it and complete the exercises

How would you rate your experience with Python?

Novice Intermediate Proficient

How would you rate your experience with using Google Cloud services?

Novice Intermediate Proficient

2. Background

App Engine Task Queue supports both push and pull tasks. To improve application portability, the Google Cloud team recommends migrating from legacy bundled services like Task Queue to other Cloud standalone or 3rd-party equivalent services.

Pull task migration is covered in Migration Modules 18-19 while Modules 7-9 focus on push task migration. In order to migrate from App Engine Task Queue push tasks, add its usage to the existing Flask and App Engine NDB app resulting from the Module 1 codelab. In that app, a new page view registers a new Visit and displays the most recent visits to the user. Since older visits are never shown again and take up space in Datastore, we're going to create a push task to automatically delete the oldest visits. Ahead in Module 8, we'll migrate that app from Task Queue to Cloud Tasks.

This tutorial features the following steps:

  1. Setup/Prework
  2. Update configuration
  3. Modify application code

3. Setup/Prework

This section explains how to:

  1. Set up your Cloud project
  2. Get baseline sample app
  3. (Re)Deploy and validate baseline app

These steps ensure you're starting with working code.

1. Setup project

If you completed the Module 1 codelab, we recommend reusing that same project (and code). Alternatively, you can create a brand new project or reuse another existing project. Ensure the project has an active billing account and App Engine is enabled.

2. Get baseline sample app

One of the prerequisites to this codelab is to have a working Module 1 App Engine app: complete the Module 1 codelab (recommended) or copy the Module 1 app from the repo. Whether you use yours or ours, the Module 1 code is where we'll "START." This codelab walks you through each step, concluding with code that resembles what's in the Module 7 repo folder "FINISH".

Regardless which Module 1 app you use, the folder should look like the below, possibly with a lib folder as well:

$ ls
README.md               main.py                 templates
app.yaml                requirements.txt

3. (Re)Deploy baseline app

Execute the following steps to (re)deploy the Module 1 app:

  1. Delete the lib folder if there is one and run: pip install -t lib -r requirements.txt to repopulate lib. You may need to use the pip2 command instead if you have both Python 2 and 3 installed.
  2. Ensure you've installed and initialized the gcloud command-line tool and reviewed its usage.
  3. Set your Cloud project with gcloud config set project PROJECT_ID if you don't want to enter your PROJECT_ID with each gcloud command issued.
  4. Deploy the sample app with gcloud app deploy
  5. Confirm the Module 1 app runs as expected without issue displaying the most recent visits (illustrated below)

a7a9d2b80d706a2b.png

4. Update configuration

No changes are necessary to the standard App Engine configuration files (app.yaml, requirements.txt, appengine_config.py).

5. Modify application files

The primary application file is main.py, and all updates in this section pertain to that file. There is also a minor update to the web template, templates/index.html. These are the changes to implement in this section:

  1. Update imports
  2. Add push task
  3. Add task handler
  4. Update web template

1. Update imports

An import of google.appengine.api.taskqueue brings in Task Queue functionality. Some Python standard library packages are also required:

  • Because we're adding a task to delete the oldest visits, the app will need to deal with timestamps, meaning use of time and datetime.
  • To log useful information regarding task execution, we need logging.

Adding all of these imports, below is what your code looks like before and after these changes:

BEFORE:

from flask import Flask, render_template, request
from google.appengine.ext import ndb

AFTER:

from datetime import datetime
import logging
import time
from flask import Flask, render_template, request
from google.appengine.api import taskqueue
from google.appengine.ext import ndb

2. Add push task (collate data for task, queue new task)

The push queue documentation states: "To process a task, you must add it to a push queue. App Engine provides a default push queue, named default, which is configured and ready to use with default settings. If you want, you can just add all your tasks to the default queue, without having to create and configure other queues." This codelab uses the default queue for brevity. To learn more about defining your own push queues, with the same or differing characteristics, see the Creating Push Queues documentation.

The primary goal of this codelab is to add a task (to the default push queue) whose job it is to delete old visits from Datastore that are no longer displayed. The baseline app registers each visit (GET request to /) by creating a new Visit entity, then fetches and displays the most recent visits. None of the oldest visits will ever be displayed or used again, so the push task deletes all visits older than the oldest displayed. To accomplish this, the app's behavior needs to change a bit:

  1. When querying the most recent visits, instead of returning those visits immediately, modify the app to save the timestamp of the last Visit, the oldest displayed—it is safe to delete all visits older than this.
  2. Create a push task with this timestamp as its payload and direct it to the task handler, accessible via an HTTP POST to /trim. Specifically, use standard Python utilities to convert the Datastore timestamp and send it (as a float) to the task but also log it (as a string) and return that string as a sentinel value to display to the user.

All of this takes place in fetch_visits(), and this is what it looks like before and after making these updates:

BEFORE:

def fetch_visits(limit):
    return (v.to_dict() for v in Visit.query().order(
            -Visit.timestamp).fetch(limit))

AFTER:

def fetch_visits(limit):
    'get most recent visits and add task to delete older visits'
    data = Visit.query().order(-Visit.timestamp).fetch(limit)
    oldest = time.mktime(data[-1].timestamp.timetuple())
    oldest_str = time.ctime(oldest)
    logging.info('Delete entities older than %s' % oldest_str)
    taskqueue.add(url='/trim', params={'oldest': oldest})
    return (v.to_dict() for v in data), oldest_str

3. Add task handler (code called when task runs)

While the deletion of old visits could have easily been accomplished in fetch_visits(), recognize that this functionality doesn't have much to do with the end-user. It's auxiliary functionality and a good candidate to process asynchronously outside of standard app requests. The end-user will reap the benefit of faster queries because there will be less information in Datastore. Create a new function trim(), called via a Task Queue POST request to /trim, which does the following:

  1. Extracts the "oldest visit" timestamp payload
  2. Issues a Datastore query to find all entities older than that timestamp.
  3. Opts for a faster "keys-only" query because no actual user data is needed.
  4. Logs the number of entities to delete (including zero).
  5. Calls ndb.delete_multi() to delete any entities (skipped if not).
  6. Returns an empty string (along with an implicit HTTP 200 return code).

You can see all of that in trim() below. Add it to main.py just after fetch_visits():

@app.route('/trim', methods=['POST'])
def trim():
    '(push) task queue handler to delete oldest visits'
    oldest = request.form.get('oldest', type=float)
    keys = Visit.query(
            Visit.timestamp < datetime.fromtimestamp(oldest)
    ).fetch(keys_only=True)
    nkeys = len(keys)
    if nkeys:
        logging.info('Deleting %d entities: %s' % (
                nkeys, ', '.join(str(k.id()) for k in keys)))
        ndb.delete_multi(keys)
    else:
        logging.info('No entities older than: %s' % time.ctime(oldest))
    return ''   # need to return SOME string w/200

4. Update web template

Update the web template, templates/index.html, with this Jinja2 conditional to display the oldest timestamp if that variable exists:

{% if oldest is defined %}
    <b>Deleting visits older than:</b> {{ oldest }}</p>
{% endif %}

Add this snippet after the displayed visits list but before closing out body so that your template looks like this:

<!doctype html>
<html>
<head>
<title>VisitMe Example</title>
<body>

<h1>VisitMe example</h1>
<h3>Last 10 visits</h3>
<ul>
{% for visit in visits %}
    <li>{{ visit.timestamp.ctime() }} from {{ visit.visitor }}</li>
{% endfor %}
</ul>

{% if oldest is defined %}
    <b>Deleting visits older than:</b> {{ oldest }}</p>
{% endif %}
</body>
</html>

6. Summary/Cleanup

This section wraps up this codelab by deploying the app, verifying it works as intended and in any reflected output. After app validation, perform any clean-up and consider next steps.

Deploy and verify application

Deploy the app with gcloud app deploy. The output should be identical to the Module 1 app except for a new line at the bottom displaying which visits will be deleted:

4aa8a2cb5f527079.png

Congratulations for completing the codelab. Your code should now match what's in the Module 7 repo folder. It is now ready to migrate to Cloud Tasks in Module 8.

Clean up

General

If you are done for now, we recommend you disable your App Engine app to avoid incurring billing. However if you wish to test or experiment some more, the App Engine platform has a free quota, and so as long as you don't exceed that usage tier, you shouldn't be charged. That's for compute, but there may also be charges for relevant App Engine services, so check its pricing page for more information. If this migration involves other Cloud services, those are billed separately. In either case, if applicable, see the "Specific to this codelab" section below.

For full disclosure, deploying to a Google Cloud serverless compute platform like App Engine incurs minor build and storage costs. Cloud Build has its own free quota as does Cloud Storage. Storage of that image uses up some of that quota. However, you might live in a region that does not have such a free tier, so be aware of your storage usage to minimize potential costs. Specific Cloud Storage "folders" you should review include:

  • console.cloud.google.com/storage/browser/LOC.artifacts.PROJECT_ID.appspot.com/containers/images
  • console.cloud.google.com/storage/browser/staging.PROJECT_ID.appspot.com
  • The storage links above depend on your PROJECT_ID and *LOC*ation, for example, "us" if your app is hosted in the USA.

On the other hand, if you're not going to continue with this application or other related migration codelabs and want to delete everything completely, shut down your project.

Specific to this codelab

The services listed below are unique to this codelab. Refer to each product's documentation for more information:

Next steps

In this "migration," you added Task Queue push queue usage to the Module 1 sample app, adding support for tracking visitors, resulting in the Module 7 sample app. The next migration teaches you how to upgrade from App Engine push tasks to Cloud Tasks should you choose to do so. As of Fall 2021, users no longer have to migrate to Cloud Tasks when upgrading to Python 3. Read more about this in the next section.

If you do want to move to Cloud Tasks, the Module 8 codelab is next. Beyond that are additional migrations to consider, such as Cloud Datastore, Cloud Memorystore, Cloud Storage, or Cloud Pub/Sub (pull queues). There are also cross-product migrations to Cloud Run and Cloud Functions. All Serverless Migration Station content (codelabs, videos, source code [when available]) can be accessed at its open source repo.

7. Migration to Python 3

In Fall 2021, the App Engine team extended support of many of the bundled services to 2nd generation runtimes (originally available only in 1st generation runtimes), meaning you are no longer required to migrate from bundled services like App Engine Task Queue to standalone Cloud or 3rd-party equivalents like Cloud Tasks when porting your app to Python 3. In other words, you can continue using Task Queue in Python 3 App Engine apps so long as you retrofit the code to access bundled services from next-generation runtimes.

You can learn more about how to migrate bundled services usage to Python 3 in the Module 17 codelab and its corresponding video. While that topic is out-of-scope for Module 7, linked below are Python 3 versions of both the Module 1 and 7 apps ported to Python 3 and still using App Engine NDB and Task Queue.

8. Additional resources

Listed below are additional resources for developers further exploring this or related Migration Module as well as related products. This includes places to provide feedback on this content, links to the code, and various pieces of documentation you may find useful.

Codelab issues/feedback

If you find any issues with this codelab, please search for your issue first before filing. Links to search and create new issues:

Migration resources

Links to the repo folders for Module 2 (START) and Module 7 (FINISH) can be found in the table below.

Codelab

Python 2

Python 3

Module 1

code

code (not featured in this tutorial)

Module 7 (this codelab)

code

code (not featured in this tutorial)

Online resources

Below are online resources which may be relevant for this tutorial:

App Engine Task Queue

App Engine platform

Other Cloud information

Videos

License

This work is licensed under a Creative Commons Attribution 2.0 Generic License.