Extend an Android app to Google Assistant with App Actions (Level 2)

In the first App Actions codelab, you learned how to extend Google Assistant to a sample fitness app by implementing built-in intents (BII) from the Health and Fitness BII category. BIIs are organized into categories representing the kinds of tasks users often ask Assistant to perform.

In this codelab, you learn how to add App Actions to an app using BIIs from the [Common BII category], which represents common app tasks that almost any Android app can fulfill.

This codelab covers intermediate-level concepts for developing with App Actions. You should have prior experience developing Android apps, handling deep links, creating an actions.xml resource, and using the App Actions Test Tool plugin. Developers entirely new to App Actions may want to start with the previous codelab.

What you'll build

In this codelab, you add two Common BII to a sample To-do list app, enabling users to ask Assistant to:

Three progressive screens where Google Assistant displays active tasks in an app.

What you'll learn

  • How Common category BII can be used to extend Assistant to most Android apps.
  • How to test Common BII with the App Actions Test Tool plugin.

Prerequisites

  • A terminal to run shell commands with git installed.
  • The latest stable release of Android Studio.
  • A Google account with access to the Google Play Console.
  • An Android device or emulator with Internet access to the Google Play Store, to test your actions.

In this codelab, you use an Android device (physical or emulator) to test your actions. If using a physical device, make sure it's connected to your local development machine. You must also be signed in to the Google app on the device (or emulator), and signed in to Android Studio, using the same Google account. The device must also have the Google Assistant app installed.

This codelab starts with the To-do list sample app for Android. Users of this sample app can add items to to-do lists, search for items by category, and view information about completed tasks.

Download your base files

Run the following command to clone the sample app's GitHub repository:

git clone --branch codelab-start https://github.com/actions-on-google/appactions-common-biis-kotlin.git

Once you've cloned the repository, open it in Android Studio:

  • In the Welcome to Android Studio dialog, click Import project.
  • Find and select the folder where you cloned the repository.

To see a version of the Todo app representing the completed codelab, clone the sample app repo, using the --branch master flag.

Change Android application ID to enable testing

Later in this codelab, you use the App Actions Test Tool to test your actions on an Android device or emulator. To run, the test tool requires your app to first be uploaded to a project in the Google Play Console. To avoid a "Duplicate package name" error when uploading your app to the Play Console, change the sample app's applicationId to something unique to you.

In app/build.gradle, update the applicationId "com.example.myapp" value to a unique ID, like "com.codelabs.myname". If you need help determining an application ID, visit the Android reference page.

build.gradle

android {
...
    defaultConfig {
        // This ID uniquely identifies your app on the device and in Google Play Store
        applicationId "com.example.myapp"
    ...
    }
}

After updating applicationId, you may see a yellow warning box in the IDE labeled "Gradle files have changed since last project sync..." If so, click Sync Now in the box to avoid issues that may occur when uploading your app to Play Console.

Test the app on your device

Before making more changes to the app, it's helpful to get an idea of what the sample app can do. Try running the app on an emulator:

  • In Android Studio, select Run > Run app or click Run in the toolbar.
  • In the Select Deployment Target dialog, select a virtual device and click OK. The recommended OS version is Android 10 (API level 30) or higher, although Actions run on devices back to Android 5 (API level 21).

Set up and verify Assistant is working on your device by long-pressing on the Home button. You will need to sign in to Assistant on your device if you haven't already.

For more information on Android virtual devices, see: Create and manage virtual devices.

Animation demonstrating the Todo list sample app

Briefly explore the app to see what it can do. Tapping the Plus icon creates a new task item, and the menu items on the top right allow you to search for and filter task items by completion status.

Upload to Play Console

Uploading the app to a project in Google Play Console is a prerequisite for using the App Actions Test Tool in Android Studio. Build your app in Android Studio and upload it to the Play Console as an internal release draft.

To build your app in Android Studio, do the following:

  1. Go to Build > Generate Signed Bundle / APK.
  2. Select "Android App Bundle" and click Next.
  3. Enter details to sign your app and click Next.
  4. Select the "prodRelease" build variant and click Finish.

In the Google Play Console, upload the app bundle you just created as a new app:

  1. On the All apps page, click Create app.
  2. Give the app any name you want and click Create app. For this codelab, you won't need to fill out any of the app information once the app is created.
  3. From the sidebar menu, go to Testing and find the Internal testing page.
  4. Click Create new release on the Internal testing page.
  5. Click Continue to agree to App signing by Google Play.
  6. In the App bundles and APKs panel, upload the AAB file you generated earlier (it's likely in the app/prod/release directory). Click Save.

Install the test plugin

The App Actions Test Tool plugin allows you to test your App Actions, for a single user, on a test device. If you do not already have the test tool, install it by following these steps:

  1. Go to File > Settings (Android Studio > Preferences on MacOS).
  2. In the Plugins section, go to Marketplace and search for "App Actions test tool". You can also manually download and install the test tool.
  3. Install the tool and restart Android Studio.

In this step, you implement the Open app feature BII, enabling users to view their active, and completed tasks, using Assistant. To do this, you extend existing deep links for the task filtering feature to Assistant, by defining the BII in actions.xml. You complete this section by testing the action on your device.

Before implementing your BII, verify that the existing deep links in your app satisfy your identified App Actions behavior.

To test deep links with the Android Debug Bridge (adb) command line tool, follow these steps:

  1. Update the app on your device.
  2. Populate the to-do list with sample items by tapping the refresh button located in the app bar overflow:

Todo list app toolbar overflow menu

If given the option to open with Google, select "Always" to allow the Todo app to open supported links. You can change preference this later in your app settings.

In your terminal, enter each of the following adb commands to test the app's task filtering deep links:

View active tasks

adb shell am start -a android.intent.action.VIEW -d "https://todo.androidappactions.com/active-tasks"

View completed tasks

adb shell am start -a android.intent.action.VIEW -d "https://todo.androidappactions.com/completed-tasks"

View all tasks

adb shell am start -a android.intent.action.VIEW -d "https://todo.androidappactions.com/all-tasks"

The above commands open the list view feature of the app, displaying tasks filtered by the completion status specified in each deep link's trailing path.

Add the BII

Extend the task filtering deep links to Google Assistant, by defining an Open app feature BII, configured to open these links, in actions.xml.

Define the BII by updating actions.xml, located in the app/src/main/res/xml sample project directory, with the following code:

actions.xml

<?xml version="1.0" encoding="utf-8"?>
<actions>
    <action intentName="actions.intent.OPEN_APP_FEATURE">

        //TODO: Handle intent parameters

        //TODO: Handle action fulfillment

    </action>
</actions>

Handle intent parameters using inline inventory

Intent parameters represent the elements extracted from a user query. For example, if a user says something like, "Hey Google, order a pizza from ExampleApp," Assistant extracts "pizza" into a food.item schema.org intent parameter, and passes the parameter to your action to handle.

The Open app feature BII supports one parameter, feature, representing the app feature extracted from the user query. An inline inventory is required for this parameter, providing Assistant a set of supported app feature names to match the parameter value to.

To handle the feature intent parameter, update actions.xml with the following code:

actions.xml

<?xml version="1.0" encoding="utf-8"?>

<actions>
    <action intentName="actions.intent.OPEN_APP_FEATURE">

        <parameter name="feature">
            <entity-set-reference entitySetId="FeatureEntitySet" />
        </parameter>

        //TODO: Handle action fulfillment

    </action>
</actions>

With the above sample code, you instructed Assistant to match feature parameter values to an inline inventory, FeatureEntitySet, defined as an entity-set-reference in the parameter tag.

Define the inline inventory for your BII by adding the following entity-set to actions.xml, beneath the action element:

actions.xml

<?xml version="1.0" encoding="utf-8"?>
<actions>
    <action>
       ...
    </action>

    <entity-set entitySetId="FeatureEntitySet">
        <entity  url="https://todo.androidappactions.com/completed-tasks"
            name="completed tasks" alternateName="@array/completed_tasks_synonyms"/>
        <entity url="https://todo.androidappactions.com/active-tasks"
            name="active tasks" alternateName="@array/active_tasks_synonyms"/>
        <entity url="https://todo.androidappactions.com/all-tasks"
            name="all tasks" alternateName="@array/all_tasks_synonyms"/>
        <entity url="https://todo.androidappactions.com/statistics"
            name="statistics" alternateName="@array/statistics_synonyms"/>
    </entity-set>
</actions>

In the above code, you defined an inline inventory, represented as an entity-set element containing four entity elements. When a user activates this BII with a request, Assistant matches the feature parameter value to the name attribute of an entity, passing the url of a matching entity to your fulfillment.

For example, if a user says something like, "Hey Google, show my completed tasks in ExampleApp," Assistant matches the feature intent parameter value, "completed tasks", to the corresponding inventory entity. Assistant then passes the associated url, https://todo.androidappactions.com/completed-tasks, to your fulfillment.

Define the action fulfillment

To fulfill the action, you define a fulfillment element in actions.xml to handle the URL passed by Assistant. Add a fulfillment to your action using the following code:

<?xml version="1.0" encoding="utf-8"?>
<actions>
    <action intentName="actions.intent.OPEN_APP_FEATURE">

        ... //Parameter definition

        <fulfillment urlTemplate="{@url}"/>
        <fulfillment urlTemplate="https://todo.androidappactions.com/all-tasks"/>

    </action>
   ...
</actions>

With the above sample, you defined two fulfillment elements. The first triggers the deep link Assistant generated from the feature parameter, represented as a {@url} placeholder in the urlTemplate attribute. The second fulfillment is a fallback that opens a task view, displaying an unfiltered task lists, for situations where the feature parameter is missing, or if a matching inline inventory element could not be found.

Test your App Action

During development and testing, you use the App Actions Test Tool to preview your App Action with Assistant on a test device. You can also use the tool to adjust intent parameters for an App Action, to test how your action handles the various ways a user might ask Assistant to run it.

Test your app action with the test tool by following these steps:

  1. Connect your physical Android test device, or start your Android emulator
  2. Go to Tools > App Actions > App Actions Test Tool. You may be asked to sign in to Android Studio. Use the same account used earlier with the Google Play Console.
  3. In the Invocation Name field, enter "Task List".
  4. If the language of your Assistant is not "English (en)", enter the locale that matches the language of your Assistant in the Locale field.
  5. Click Create Preview.
  6. Using the Configure drop-down list, select the actions.intent.OPEN_APP_FEATURE intent.
  7. In the feature box, update the default "History" value with "Completed tasks".
  8. Click Run.

As an alternative, you can use the invocation name directly in the Assistant app on your device to try out your App Action. For example, you could say "Hey Google, show completed tasks in Task List".

The Get thing BII extends your in-app search functionality to Google Assistant. In this step, you implement and test the Get thing BII, enabling users to search for specific tasks in the sample app.

The sample app has an existing deep link for in-app search functionality. Before implementing your BII, use adb to verify that this deep link satisfies your identified App Actions behavior.

Test the search deep link with the following adb command:

# GET_THING deep link: in-app search
adb shell am start -a android.intent.action.VIEW -d "https://todo.androidappactions.com/search?q=milk"

The above command triggers a deep link which loads the item list activity filtered to items containing search query, "home".

Add the BII

During a search-related App Action, Assistant extracts search terms from the user query into a thing.name intent parameter. Assistant then generates a URL for fulfillment by mapping the parameter into a URL template you define in a fulfillment element.

To define the Get thing BII in your app, update actions.xml with this new action:

Actions.xml

<?xml version="1.0" encoding="utf-8"?>
<actions>
    <action intentName="actions.intent.OPEN_APP_FEATURE">...</action>

    <action intentName="actions.intent.GET_THING">
        <fulfillment urlTemplate="https://todo.androidappactions.com/search?q={q}">
            <parameter-mapping intentParameter="thing.name" urlParameter="q"/>
        </fulfillment>
    </action>
   ...
</actions>

In the above sample, the parameter-mapping element assigns the thing.name intent parameter to the "q" value of the fulfillment element's urlParameter attribute. Assistant then generates the deep link by mapping the urlParameter value to the "{q}" placeholder in the fulfillment urlTemplate.

Test your App Action

To test in-app search for your app from Assistant, follow these steps:

  1. Connect your physical Android test device, or start your Android emulator
  2. Go to Tools > App Actions > App Actions Test Tool.
  3. Click Update Preview.
  4. Using the Configure drop-down list, select the actions.intent.GET_THING intent. Change the "name" value from "running shoes" to "milk".
  5. Click Run.

In this test, the Get thing BII uses the name attribute to search the app for tasks containing "home". Like the previous App Action, you can test your action using the test tool, or simply say "Hey Google, search for cake mix in Task List", or other search phrases, on your test device.

Congratulations!

You now have the power to enable almost any Android app to work with Google Assistant, using common BIIs.

What we've covered

In this codelab, you learned:

  • How to let users deep dive to specific app features, using Assistant.
  • How users can access in-app search, from Assistant.
  • How to test common BIIs using the App Actions Test Tool.

What's next

From here, you can try making further refinements to your Task List app. To reference the finished project, see the repo –master branch on GitHub.

Here are some suggestions for further learning about extending this app with App Actions:

To continue your Actions on Google journey, explore these resources:

Follow us on Twitter @ActionsOnGoogle to stay tuned to our latest announcements, and tweet to #AoGDevs to share what you have built!

Feedback survey

Finally, please fill out this survey to give feedback about your experience with this codelab.