Extend an Android app to Google Assistant with App Actions (Level 2)

1. Overview

App Actions lets users launch directly into specific app features from Google Assistant to help you expand the reach of your Android app. As an Android developer, you can implement capabilities, which lets Google Assistant know the type of functionality available to users and how you want to fulfill these requests.

In the first App Actions codelab, you learned how to extend Google Assistant to a sample fitness app by implementing built-in intents (BII) from the Health and Fitness BII category. BIIs are organized into categories representing the kinds of tasks users often ask Assistant to perform.

Key term: An Assistant built-in intent, or BII for short, is a predefined trigger for a specific kind of functionality that the user wants to do. Google builds and maintains the language models for BIIs to understand a large variation of queries that relate to that particular intent.

In this codelab, you learn how to add App Actions to an app using BIIs from the "Common" BII category, which represents common app tasks that almost any Android app can fulfill.

This codelab covers intermediate-level concepts for developing with App Actions. You should have prior experience developing Android apps and implementing Android intents.

What you'll build

In this codelab, you add two Common BIIs to a sample To-do list app, enabling users to ask Assistant to:

• Navigate to features within the app with the actions.intent.OPEN_APP_FEATURE BII.
• Search for content using in-app search with the actions.intent.GET_THING BII.

Note: Apps with search functionality are required to implement the actions.intent.GET_THING BII. Learn more about App Actions deployment requirements.

Figure 1. Three progressive screens where Google Assistant displays active tasks in an app.

What you'll learn

You'll learn how to use Common category BIIs to extend Assistant to most Android apps. You will also learn how to test Common BIIs with the Google Assistant plugin for Android Studio.

Prerequisites

• A terminal to run shell commands with git installed.
• The latest stable release of Android Studio.
• A physical or virtual Android device with Internet access to the Google Play Store to test your actions.
• Your same Google account must be signed in to Android Studio, and both the Google app and Google Assistant app on your test device.

Note: You can test the App Actions described in this codelab with Google Assistant in a limited number of locales. Refer to the Common BII reference docs to find out what intents are available, based on locale.

In this codelab, you use an Android device (physical or virtual) to test your actions. If using a physical device, make sure it's connected to your local development machine. You must also be signed in to the Google app on the device, and signed in to Android Studio, using the same Google account. The device must also have the Google Assistant app installed.

2. Understand how it works

App Actions connect users from Google Assistant to your Android app. But how do they work?

When a user indicates to Assistant that they want to use your app, Assistant looks for App Actions registered to your app from a shortcuts.xml file. This file contains the app's capabilities, which links an Assistant built-in intent or custom intent to an Android intent or deep link.

Key term: Capabilities are templates you define in shortcuts.xml to declare the types of actions users can take to launch your app and jump directly a specific task. For details, see Android documentation for capabilities.

When a user says a query to Assistant, Assistant parses the user's input and matches it to an App Actions intent (in this codelab, it will be a BII). Assistant knows which capabilities you support from your shortcuts.xml file in your Android App. With the intent match, the capability with that BII contains how you want to fulfill that request. In this codelab, the fulfillment is an Android intent that launches an activity in your app.

The following diagram shows this Assistant flow:

Figure 2. A flow describing how Google Assistant processes a voice query.

The shortcuts.xml project file contains the following information for each App Action:

• What built-in intent or custom intent the App Action uses
• What Android activity or deep link should be provided to the user
• How parameters for the built-in intent map to information provided to Assistant by the user

Your Android activity then filters for and handles the provided Android intent or deep link to provide the user with their desired functionality. The result is a user experience where Assistant invokes your app functionality in response to a user's query.

3. Prepare your development environment

This codelab uses the To-do list sample app for Android. This sample app can add items to to-do lists, search for items by category, and view information about completed tasks.

Download your base files

Run the following command to clone the sample app's GitHub repository:

git clone --branch codelab-start https://github.com/actions-on-google/appactions-common-biis-kotlin.git

Once you've cloned the repository, follow these steps to open it in Android Studio:

1. In the Welcome to Android Studio dialog, click Import project.
2. Find and select the folder where you cloned the repository.

To see a version of the app representing the completed codelab, clone the sample app repo using the --branch master flag.

Update the Android application ID

Updating the app's application ID uniquely identifies the app on your test device and avoids a "Duplicate package name" error if the app is uploaded to the Play Console. To update the application ID, open app/build.gradle:

android {
...
  defaultConfig {
    applicationId "com.MYUNIQUENAME.android.fitactions"
    ...
  }
}

Replace "MYUNIQUENAME" in the applicationId field to something unique to you.

Note: After updating applicationId, you will receive a "Gradle files have changed since last project sync..." warning in Android Studio. Click Sync Now to avoid issues when uploading your app.

Test the app on your device

Before making more changes to the app, it's helpful to get an idea of what the sample app can do. To run the app on your emulator, follow these steps:

1. In Android Studio, select Run > Run app or click Run in the toolbar.
2. In the Select Deployment Target dialog, select a device and click OK. The recommended OS version is Android 10 (API level 30) or higher, although Actions run on devices back to Android 5 (API level 21).
3. Long-press on the Home button to set up Assistant and verify its working. You will need to sign in to Assistant on your device, if you haven't already.

For more information on Android virtual devices, see Create and manage virtual devices.

Figure 3. An animation demonstrating the To-do list sample app.

Briefly explore the app to see what it can do. Tapping the Plus icon creates a new task item, and the menu items on the top right allow you to search for and filter task items by completion status.

Install the test plugin

The Google Assistant plugin allows you to test your App Actions on a test device. If you do not already have the test tool, install it by following these steps:

1. Go to File > Settings (Android Studio > Preferences on MacOS).
2. In the Plugins section, go to Marketplace and search for "Google Assistant".You can also manually download and install the test tool.
   • If you cannot find the plugin on the Marketplace, download the plugin manually and follow the instructions on Install the plugin from disk.
3. Install the tool and restart Android Studio.

Note: The test plugin sends information to Assistant through the Google app on your Android device.

4. Add the Get Thing BII capability

The actions.intent.GET_THING BII extends your in-app search functionality to Google Assistant. In this step, you will implement and test the GET_THING BII, enabling users to search for specific tasks in the sample app.

Tip: The GET_THING BII is used to search for content or entities using the default in-app search feature in an app. Populate the search criteria with the content or entity name using the thing.name intent parameter, and present the most relevant search results to the user.

Define Get Thing capability

During a search-related App Action, Assistant extracts search terms from the user query into the thing.name BII parameter, then passes that value to the Android application.

To add the GET_THING BII to your app, update shortcuts.xml, located in the app/src/main/res/xml sample project directory, with a <capability> element within the top level <shortcuts> tag:

shortcuts.xml

<shortcuts ....>
    <capability android:name="actions.intent.GET_THING">
        <intent
            android:action="android.intent.action.VIEW"
            android:targetPackage="com.yourApplicationId.appaction"
            android:targetClass="com.example.android.architecture.blueprints.todoapp.tasks.TasksActivity">
            <parameter
                android:name="thing.name"
                android:key="q"/>
        </intent>
    </capability>
</shortcuts>

The configuration above:

1. Declares that the app responds to the GET_THING BII.
2. Specifies how to build an Android intent that launches the app in response to that BII.
   • The Activity is identified using targetPackage and targetClass.
   • The BII parameter thing.name is mapped into the Intent Extra named q.

The named activity must be present in the app's manifest and exported.

The provided Android application contains metadata so the AndroidManifest is aware of the shortcuts.xml file:

<meta-data
   android:name="android.app.shortcuts"
   android:resource="@xml/shortcuts" />

Test your App Action

To test in-app search for your app from Assistant, follow these steps:

1. Make sure your Android device is connected.
2. Go to Tools > Google Assistant > App Actions Test Tool.
3. Click Create Preview to accept the default values for App name and locale. If asked, review and accept the App Actions policies and terms of service.
4. In the first step where the tool asks you to select and configure a BII, select actions.intent.GET_THING. Change the name value from running shoes to milk.
5. Click Run App Action.

In this test, the GET_THING BII uses the name attribute to search the app for tasks containing "milk". Like the previous App Action, you can test your action using the test tool, or simply say "Hey Google, search for cake mix in Task List", or other search phrases, on your test device.

Tip: Check the troubleshooting guide if you run into issues running the test tool.

5. Add the Open app feature BII capability

In this step, you implement the Open app feature BII, enabling users to view their active and completed tasks using Assistant. To do this, complete the capability in shortcuts.xml which contains information on how the capability is triggered, how parameters are passed, and which Android intents to invoke. For this codelab, you'll be using OPEN_APP_FEATURE BII. After you implement this BII, you test the Action on your device.

Note: This BII launches a feature of the app. This intent is especially useful to help users navigate to features within your app which may be otherwise be difficult to find, or require multiple taps to get to. You must specify inline inventory for the feature parameter.

Add the Open app feature capability

Add a second capability for Open app feature in shortcuts.xml underneath the Get Thing Capability element:

shortcuts.xml

<capability android:name="actions.intent.OPEN_APP_FEATURE">
   <intent
       android:action="android.intent.action.VIEW"
       android:targetPackage="yourApplicationId"
       android:targetClass="com.example.android.architecture.blueprints.todoapp.tasks.TasksActivity">
       <parameter
           android:name="feature"
           android:key="feature"/>
   </intent>
</capability>

This capability maps the Open app feature BII and Android intent together, so when the Open app feature is triggered, the Android intent will trigger.

Before triggering the Android intent, supported parameters are pulled from the user's input. The OPEN_APP_FEATURE BII supports one parameter, feature, representing the app feature extracted from the user query. There are two types of features that this application will support: Active Task and Completed Task. These features allow users to open the app with a filtered view of their list of tasks. You need to use inline inventory to support these features.

Key term: An inline inventory guides entity extraction for BIIs by providing a set of entities an app understands to a BII parameter that supports inline inventory. For details, see Inline inventory for App Actions.

Handle intent parameters using inline inventory

Intent parameters represent the elements extracted from a user query. For example, if a user says something like, "Hey Google, order a pizza from ExampleApp," Assistant extracts "pizza" into a food.item schema.org intent parameter, and passes the parameter to your action to handle.

The Open app feature BII supports one parameter, feature, representing the app feature extracted from the user query. An inline inventory is required for this parameter, providing Assistant a set of supported app feature names to match the parameter value to.

To handle the feature intent parameter, add a shortcut to your shortcuts.xml with the following code above the Open app feature capability:

shortcuts.xml

<shortcut
   android:shortcutId="active_tasks"
   android:shortcutShortLabel="@string/label_active"
   android:enabled="false">
   <capability-binding
       android:key="actions.intent.OPEN_APP_FEATURE">
       <parameter-binding
           android:key="feature"
           android:value="@array/active_tasks_synonyms" />
   </capability-binding>
</shortcut>

<shortcut
   android:shortcutId="completed_tasks"
   android:shortcutShortLabel="@string/label_completed"
   android:enabled="false">
   <capability-binding
       android:key="actions.intent.OPEN_APP_FEATURE">
       <parameter-binding
           android:key="feature"
           android:value="@array/completed_tasks_synonyms" />
   </capability-binding>
</shortcut>

In the above code you defined an inline inventory, represented as shortcut elements with capability and parameter bindings. Shortcuts can be used as inventory for BII parameters. Google Assistant matches the user query with the values in the parameter-binding of Shortcuts. For a matched parameter value, the shortcutId will be added to the fulfillment intent. When a user activates the OPEN_APP_FEATURE BII with a request, Assistant matches the feature parameter value to the value attribute of a shortcut, passing the value to the targetClass as a parameter in Extra.

For example, if a user says something like, "Hey Google, show my completed tasks in ExampleApp," Assistant matches the feature intent parameter value, "completed tasks", to the corresponding inventory shortcut, which passes that value to the OPEN_APP_FEATURE capability. Assistant then triggers the Android intent.

Test your App Action

During development and testing, you use the Google Assistant plugin to preview your App Action with Assistant on a test device. You can also use the tool to adjust intent parameters for an App Action, to test how your action handles the various ways a user might ask Assistant to run it.

To test your App Action with the test tool, follow these steps:

1. Connect your physical Android test device, or start your Android emulator
2. If the test tool is not visible, open it by going to Tools > Google Assistant > App Actions Test Tool.
3. Click the Update button to refresh your preview.
4. In the first step where the tool asks you to select and configure a BII, select actions.intent.OPEN_APP_FEATURE.
5. In the feature box, update the default History value with Completed tasks.
6. Click Run App Action.

Tip: Check the troubleshooting guide if you run into issues running the test tool.

As an alternative, you can use the invocation name directly in the Assistant app on your device to try out your App Action. For example, you could say "Hey Google, show completed tasks in Task List".

6. Next steps

Congratulations!

You now have the power to enable almost any Android app to work with Google Assistant, using common BIIs.

What we've covered

In this codelab, you learned:

• How to let users deep dive to specific app features using Assistant.
• How users can access in-app search from Assistant.
• How to test common BIIs using the Google Assistant Plugin.

What's next

From here, you can try making further refinements to your To-do list app. To reference the finished project, see the repo –master branch on GitHub.

Here are some suggestions for further learning about extending this app with App Actions:

• Check out the To-do list sample with Google Analytics for Firebase to learn how to track the performance of your App Actions.
• Visit the App Actions built-in intents reference to discover more ways to extend your apps to Assistant.

To continue your Actions on Google journey, explore these resources:

• developers.google.com/assistant: Official documentation site for Actions on Google.
• App Actions sample index: Sample apps and code for exploring App Actions capabilities.
• Actions on Google GitHub repo: Sample code and libraries.
• r/GoogleAssistantDev: Official Reddit community for developers working with Google Assistant.

Follow us on Twitter @ActionsOnGoogle to stay tuned to our latest announcements, and tweet to #AoGDevs to share what you have built!

Feedback survey

Finally, please fill out this survey to give feedback about your experience with this codelab.