1. Overview
The Serverless Migration Station series of codelabs (self-paced, hands-on tutorials) and related videos aim to help Google Cloud serverless developers modernize their appications by guiding them through one or more migrations, primarily moving away from legacy services. Doing so makes your apps more portable and gives you more options and flexibility, enabling you to integrate with and access a wider range of Cloud products and more easily upgrade to newer language releases. While initially focusing on the earliest Cloud users, primarily App Engine (standard environment) developers, this series is broad enough to include other serverless platforms like Cloud Functions and Cloud Run, or elsewhere if applicable.
This codelab teaches you how to use App Engine Task Queue push tasks in the sample app from the Module 1 codelab. The Module 7 blog post and video complement this tutorial, providing a brief overview of the content in this tutorial.
In this module, we will add the use of push tasks, then migrate that usage to Cloud Tasks in Module 8 and later to Python 3 and Cloud Datastore in Module 9. Those using Task Queues for pull tasks will migrate to Cloud Pub/Sub and should refer to Modules 18-19 instead.
You'll learn how to
- Use the App Engine Task Queue API/bundled service
- Add push task usage to a basic Python 2 Flask App Engine NDB app
What you'll need
- A Google Cloud project with an active GCP billing account
- Basic Python skills
- Working knowledge of common Linux commands
- Basic knowledge of developing and deploying App Engine apps
- A working Module 1 App Engine app (complete its codelab [recommended] or copy the app from the repo)
Survey
How will you use this tutorial?
How would you rate your experience with Python?
How would you rate your experience with using Google Cloud services?
2. Background
App Engine Task Queue supports both push and pull tasks. To improve application portability, the Google Cloud team recommends migrating from legacy bundled services like Task Queue to other Cloud standalone or 3rd-party equivalent services.
- Task Queue push task users should migrate to Cloud Tasks.
- Task Queue pull task users should migrate to Cloud Pub/Sub.
Pull task migration is covered in Migration Modules 18-19 while Modules 7-9 focus on push task migration. In order to migrate from App Engine Task Queue push tasks, add its usage to the existing Flask and App Engine NDB app resulting from the Module 1 codelab. In that app, a new page view registers a new Visit and displays the most recent visits to the user. Since older visits are never shown again and take up space in Datastore, we're going to create a push task to automatically delete the oldest visits. Ahead in Module 8, we'll migrate that app from Task Queue to Cloud Tasks.
This tutorial features the following steps:
- Setup/Prework
- Update configuration
- Modify application code
3. Setup/Prework
This section explains how to:
- Set up your Cloud project
- Get baseline sample app
- (Re)Deploy and validate baseline app
These steps ensure you're starting with working code.
1. Setup project
If you completed the Module 1 codelab, we recommend reusing that same project (and code). Alternatively, you can create a brand new project or reuse another existing project. Ensure the project has an active billing account and App Engine is enabled.
2. Get baseline sample app
One of the prerequisites to this codelab is to have a working Module 1 App Engine app: complete the Module 1 codelab (recommended) or copy the Module 1 app from the repo. Whether you use yours or ours, the Module 1 code is where we'll "START." This codelab walks you through each step, concluding with code that resembles what's in the Module 7 repo folder "FINISH".
- START: Module 1 folder (Python 2)
- FINISH: Module 7 folder (Python 2)
- Entire repo (to clone or download ZIP file)
Regardless which Module 1 app you use, the folder should look like the below, possibly with a lib
folder as well:
$ ls README.md main.py templates app.yaml requirements.txt
3. (Re)Deploy baseline app
Execute the following steps to (re)deploy the Module 1 app:
- Delete the
lib
folder if there is one and run:pip install -t lib -r requirements.txt
to repopulatelib
. You may need to use thepip2
command instead if you have both Python 2 and 3 installed. - Ensure you've installed and initialized the
gcloud
command-line tool and reviewed its usage. - Set your Cloud project with
gcloud config set project
PROJECT_ID
if you don't want to enter yourPROJECT_ID
with eachgcloud
command issued. - Deploy the sample app with
gcloud app deploy
- Confirm the Module 1 app runs as expected without issue displaying the most recent visits (illustrated below)
4. Update configuration
No changes are necessary to the standard App Engine configuration files (app.yaml
, requirements.txt
, appengine_config.py
).
5. Modify application files
The primary application file is main.py
, and all updates in this section pertain to that file. There is also a minor update to the web template, templates/index.html
. These are the changes to implement in this section:
- Update imports
- Add push task
- Add task handler
- Update web template
1. Update imports
An import of google.appengine.api.taskqueue
brings in Task Queue functionality. Some Python standard library packages are also required:
- Because we're adding a task to delete the oldest visits, the app will need to deal with timestamps, meaning use of
time
anddatetime
. - To log useful information regarding task execution, we need
logging
.
Adding all of these imports, below is what your code looks like before and after these changes:
BEFORE:
from flask import Flask, render_template, request
from google.appengine.ext import ndb
AFTER:
from datetime import datetime
import logging
import time
from flask import Flask, render_template, request
from google.appengine.api import taskqueue
from google.appengine.ext import ndb
2. Add push task (collate data for task, queue new task)
The push queue documentation states: "To process a task, you must add it to a push queue. App Engine provides a default push queue, named default
, which is configured and ready to use with default settings. If you want, you can just add all your tasks to the default queue, without having to create and configure other queues." This codelab uses the default
queue for brevity. To learn more about defining your own push queues, with the same or differing characteristics, see the Creating Push Queues documentation.
The primary goal of this codelab is to add a task (to the default
push queue) whose job it is to delete old visits from Datastore that are no longer displayed. The baseline app registers each visit (GET
request to /
) by creating a new Visit
entity, then fetches and displays the most recent visits. None of the oldest visits will ever be displayed or used again, so the push task deletes all visits older than the oldest displayed. To accomplish this, the app's behavior needs to change a bit:
- When querying the most recent visits, instead of returning those visits immediately, modify the app to save the timestamp of the last
Visit
, the oldest displayed—it is safe to delete all visits older than this. - Create a push task with this timestamp as its payload and direct it to the task handler, accessible via an HTTP
POST
to/trim
. Specifically, use standard Python utilities to convert the Datastore timestamp and send it (as a float) to the task but also log it (as a string) and return that string as a sentinel value to display to the user.
All of this takes place in fetch_visits()
, and this is what it looks like before and after making these updates:
BEFORE:
def fetch_visits(limit):
return (v.to_dict() for v in Visit.query().order(
-Visit.timestamp).fetch(limit))
AFTER:
def fetch_visits(limit):
'get most recent visits and add task to delete older visits'
data = Visit.query().order(-Visit.timestamp).fetch(limit)
oldest = time.mktime(data[-1].timestamp.timetuple())
oldest_str = time.ctime(oldest)
logging.info('Delete entities older than %s' % oldest_str)
taskqueue.add(url='/trim', params={'oldest': oldest})
return (v.to_dict() for v in data), oldest_str
3. Add task handler (code called when task runs)
While the deletion of old visits could have easily been accomplished in fetch_visits()
, recognize that this functionality doesn't have much to do with the end-user. It's auxiliary functionality and a good candidate to process asynchronously outside of standard app requests. The end-user will reap the benefit of faster queries because there will be less information in Datastore. Create a new function trim()
, called via a Task Queue POST
request to /trim
, which does the following:
- Extracts the "oldest visit" timestamp payload
- Issues a Datastore query to find all entities older than that timestamp.
- Opts for a faster "keys-only" query because no actual user data is needed.
- Logs the number of entities to delete (including zero).
- Calls
ndb.delete_multi()
to delete any entities (skipped if not). - Returns an empty string (along with an implicit HTTP 200 return code).
You can see all of that in trim()
below. Add it to main.py
just after fetch_visits()
:
@app.route('/trim', methods=['POST'])
def trim():
'(push) task queue handler to delete oldest visits'
oldest = request.form.get('oldest', type=float)
keys = Visit.query(
Visit.timestamp < datetime.fromtimestamp(oldest)
).fetch(keys_only=True)
nkeys = len(keys)
if nkeys:
logging.info('Deleting %d entities: %s' % (
nkeys, ', '.join(str(k.id()) for k in keys)))
ndb.delete_multi(keys)
else:
logging.info('No entities older than: %s' % time.ctime(oldest))
return '' # need to return SOME string w/200
4. Update web template
Update the web template, templates/index.html
, with this Jinja2 conditional to display the oldest timestamp if that variable exists:
{% if oldest is defined %}
<b>Deleting visits older than:</b> {{ oldest }}</p>
{% endif %}
Add this snippet after the displayed visits list but before closing out body so that your template looks like this:
<!doctype html>
<html>
<head>
<title>VisitMe Example</title>
<body>
<h1>VisitMe example</h1>
<h3>Last 10 visits</h3>
<ul>
{% for visit in visits %}
<li>{{ visit.timestamp.ctime() }} from {{ visit.visitor }}</li>
{% endfor %}
</ul>
{% if oldest is defined %}
<b>Deleting visits older than:</b> {{ oldest }}</p>
{% endif %}
</body>
</html>
6. Summary/Cleanup
This section wraps up this codelab by deploying the app, verifying it works as intended and in any reflected output. After app validation, perform any clean-up and consider next steps.
Deploy and verify application
Deploy the app with gcloud app deploy
. The output should be identical to the Module 1 app except for a new line at the bottom displaying which visits will be deleted:
Congratulations for completing the codelab. Your code should now match what's in the Module 7 repo folder. It is now ready to migrate to Cloud Tasks in Module 8.
Clean up
General
If you are done for now, we recommend you disable your App Engine app to avoid incurring billing. However if you wish to test or experiment some more, the App Engine platform has a free quota, and so as long as you don't exceed that usage tier, you shouldn't be charged. That's for compute, but there may also be charges for relevant App Engine services, so check its pricing page for more information. If this migration involves other Cloud services, those are billed separately. In either case, if applicable, see the "Specific to this codelab" section below.
For full disclosure, deploying to a Google Cloud serverless compute platform like App Engine incurs minor build and storage costs. Cloud Build has its own free quota as does Cloud Storage. Storage of that image uses up some of that quota. However, you might live in a region that does not have such a free tier, so be aware of your storage usage to minimize potential costs. Specific Cloud Storage "folders" you should review include:
console.cloud.google.com/storage/browser/LOC.artifacts.PROJECT_ID.appspot.com/containers/images
console.cloud.google.com/storage/browser/staging.PROJECT_ID.appspot.com
- The storage links above depend on your
PROJECT_ID
and *LOC
*ation, for example, "us
" if your app is hosted in the USA.
On the other hand, if you're not going to continue with this application or other related migration codelabs and want to delete everything completely, shut down your project.
Specific to this codelab
The services listed below are unique to this codelab. Refer to each product's documentation for more information:
- The App Engine Task Queue service doesn't incur any additional billing per the pricing page for legacy bundled services like Task Queue.
- The App Engine Datastore service is provided by Cloud Datastore (Cloud Firestore in Datastore mode) which also has a free tier; see its pricing page for more information.
Next steps
In this "migration," you added Task Queue push queue usage to the Module 1 sample app, adding support for tracking visitors, resulting in the Module 7 sample app. The next migration teaches you how to upgrade from App Engine push tasks to Cloud Tasks should you choose to do so. As of Fall 2021, users no longer have to migrate to Cloud Tasks when upgrading to Python 3. Read more about this in the next section.
If you do want to move to Cloud Tasks, the Module 8 codelab is next. Beyond that are additional migrations to consider, such as Cloud Datastore, Cloud Memorystore, Cloud Storage, or Cloud Pub/Sub (pull queues). There are also cross-product migrations to Cloud Run and Cloud Functions. All Serverless Migration Station content (codelabs, videos, source code [when available]) can be accessed at its open source repo.
7. Migration to Python 3
In Fall 2021, the App Engine team extended support of many of the bundled services to 2nd generation runtimes (originally available only in 1st generation runtimes), meaning you are no longer required to migrate from bundled services like App Engine Task Queue to standalone Cloud or 3rd-party equivalents like Cloud Tasks when porting your app to Python 3. In other words, you can continue using Task Queue in Python 3 App Engine apps so long as you retrofit the code to access bundled services from next-generation runtimes.
You can learn more about how to migrate bundled services usage to Python 3 in the Module 17 codelab and its corresponding video. While that topic is out-of-scope for Module 7, linked below are Python 3 versions of both the Module 1 and 7 apps ported to Python 3 and still using App Engine NDB and Task Queue.
8. Additional resources
Listed below are additional resources for developers further exploring this or related Migration Module as well as related products. This includes places to provide feedback on this content, links to the code, and various pieces of documentation you may find useful.
Codelab issues/feedback
If you find any issues with this codelab, please search for your issue first before filing. Links to search and create new issues:
Migration resources
Links to the repo folders for Module 2 (START) and Module 7 (FINISH) can be found in the table below.
Codelab | Python 2 | Python 3 |
code (not featured in this tutorial) | ||
Module 7 (this codelab) | code (not featured in this tutorial) |
Online resources
Below are online resources which may be relevant for this tutorial:
App Engine Task Queue
- App Engine Task Queue overview
- App Engine Task Queue push queues overview
- Creating Task Queue push queues
queue.yaml
referencequeue.yaml
vs. Cloud Tasks- Push queues to Cloud Tasks migration guide
- App Engine Task Queue push queues to Cloud Tasks documentation sample
App Engine platform
- App Engine documentation
- Python 2 App Engine (standard environment) runtime
- Using App Engine built-in libraries on Python 2 App Engine
- Python 3 App Engine (standard environment) runtime
- Differences between Python 2 & 3 App Engine (standard environment) runtimes
- Python 2 to 3 App Engine (standard environment) migration guide
- App Engine pricing and quotas information
- Second generation App Engine platform launch (2018)
- Comparing first & second generation platforms
- Long-term support for legacy runtimes
- Documentation migration samples
- Community-contributed migration samples
Other Cloud information
- Python on Google Cloud Platform
- Google Cloud Python client libraries
- Google Cloud "Always Free" tier
- Google Cloud SDK (
gcloud
command-line tool) - All Google Cloud documentation
Videos
- Serverless Migration Station
- Serverless Expeditions
- Subscribe to Google Cloud Tech
- Subscribe to Google Developers
License
This work is licensed under a Creative Commons Attribution 2.0 Generic License.