Build Actions for Google Assistant using the Actions SDK (Level 1)

1. Overview

The Google Assistant developer platform lets you create software to extend the functionality of Google Assistant, a virtual personal assistant, across more than 1 billion devices, including smart speakers, phones, cars, TVs, headphones, and more. Users engage Assistant in conversation to get things done, such as buying groceries or booking a ride. As a developer, you can use the Assistant developer platform to easily create and manage delightful and effective conversational experiences between users and your own third-party fulfillment service.

This codelab covers beginner-level concepts for developing with the Actions SDK for Google Assistant; you do not need any prior experience with the platform to complete it. In this codelab, you build a simple Action for Google Assistant that tells users their fortune as they begin their adventure in the mythical land of Gryffinberg. In the Actions SDK level 2 codelab, you build out this Action further to customize the user's fortune based on their input.

What you'll build

In this codelab, you build a simple Action with the following functions:

  • Responds to users with a greeting message
  • Asks users a question and responds appropriately to the user's selection
  • Provides suggestion chips users can click to provide input
  • Modifies the greeting message to the user based on whether they are a returning user

When you finish this codelab, your completed Action will have the following conversational flow (the text next to the microphone represents the user's input, while the text next to the speaker represents the Action's response):

1c1e79902bed7230.png

18ef55647b4cb52c.png

What you'll learn

  • How to create a project in the Actions console
  • How to use the gactions tool to push and pull your Action project between the Actions console and your local file system
  • How to send a prompt to the user after they invoke your Action
  • How to process a user's input and return a response
  • How to test your Action in the Actions simulator
  • How to implement fulfillment using the Cloud Functions editor

What you'll need

The following tools must be in your environment:

  • An IDE or text editor of your choice
  • A terminal to run shell commands for Node.js and npm
  • A web browser, such as Google Chrome

2. Set up

The following sections describe how to set up your development environment and create your Actions project.

Check your Google permission settings

To test the Action you build in this codelab, you need to enable the necessary permissions so the simulator can access your Action.

To enable permissions, follow these steps:

  1. Go to Activity Controls.
  2. If you haven't already done so, sign in with your Google Account.
  3. Enable the following permissions:
  • Web & App Activity
  • Under Web & App Activity, select the Include Chrome history and activity from sites, apps, and devices that use Google services checkbox**.**

Create an Actions project

Your Actions project is a container for your Action.

To create your Actions project for this codelab, follow these steps:

  1. Open the Actions console.
  2. Click New project.
  3. Enter a Project name, such as actions-codelab. (The name is for your internal reference. Later, you can set an external name for your project.)

8cd05a84c1c0a32f.png

  1. Click Create project.
  2. In the What kind of Action do you want to build? screen, select the Custom card.
  3. Click Next.
  4. In the How do you want to build it? screen, select the Blank project card.
  5. Click Start building.

Save the Project ID for your Action

The Project ID is a unique identifier for your Action. You need your Project ID for several steps in this codelab.

To retrieve your Project ID, follow these steps:

  1. In the Actions console, click the three vertical dots (insert icon here).
  2. Click Project settings.

6f59050b85943073.png

  1. Copy the project ID**.**

Associate a billing account

If you don't already have a billing account, you need to create a billing account and associate it with your project in Google Cloud to deploy your fulfillment later with Cloud Functions.

To associate a billing account with your project, follow these steps:

  1. Go to the Google Cloud Platform billing page.
  2. Click Add billing account or Create Account.
  3. Enter your payment information.
  4. Click Start my free trial or Submit and enable billing.
  5. Go to the Google Cloud Platform billing page.
  6. Click the My Projects tab.
  7. Click the three dots under Actions next to the Actions project for the codelab.
  8. Click Change billing.
  9. In the drop-down menu, select the billing account you configured.
  10. Click Set account.

To avoid incurring charges, follow the steps in the Clean up your project section at the end of this codelab.

Install the gactions command-line interface

In this codelab, you use the gactions command-line interface (CLI) tool to synchronize your Actions project between the Actions console and your local file system.

To install the gactions CLI, follow the instructions in Install the gactions command-line tool.

Download your Actions project

Start developing your Action by downloading your Actions project from the Actions console.

To download your Actions project, follow these steps:

  1. To make a new directory and change into that directory, run the following commands:
mkdir myproject
cd myproject
  1. To copy the configuration of your Actions project to your local file system, run the following command:
gactions pull --project-id <projectID>

Understand the file structure

The Actions project you download from the Actions console is represented in a YAML file structure. The following image shows a high-level representation of the file structure:

2aefeeab7c8eb32f.png

The file structure consists of the following:

  • actions/: Represents your Actions project. The system calls actions.yaml when your Action is invoked, which then calls the custom/global/actions.intent.MAIN.yaml file.
  • custom/: The directory you'll work in to modify your Action.
  • global/: This directory contains system intents that the platform adds to your project automatically. You'll learn more about system intents later in this codelab.
  • manifest.yaml: A file that contains information that's transportable, or not specific to any given project and can be moved between projects.
  • settings/: Represents the settings of an Actions project, such as the display name, default locale, and category.

3. Start a conversation

Users start the conversation with your Action through invocation. For example, if you have an Action named MovieTime, users can invoke your Action by saying a phrase like "Hey Google, talk to MovieTime", where MovieTime is the display name. Your Action must have a display name if you want to deploy it to production; however, to test your Action, you don't need to define the display name. Instead, you can use the phrase "Talk to my test app" in the simulator to invoke your Action. You learn more about the simulator later in this section.

You must edit the main invocation to define what happens after a user invokes your Action.

By default, your Action provides a generic prompt when your invocation is triggered ("Start building your Action by defining main invocation.").

In the next section, you customize the prompt for your main invocation in the custom/global/actions.intent.MAIN.yaml file.

Set up main invocation

You can edit your main invocation prompt in the actions.intent.MAIN.yaml file.

To modify the prompt your Action sends back to the user when they invoke your Action, follow these steps:

  1. Open custom/global/actions.intent.MAIN.yaml in your text editor.
  2. Replace the text in the speech field (Start building your action...) with the following welcome message: A wondrous greeting, adventurer! Welcome to the mythical land of Gryffinberg! Based on your clothes, you are not from around these lands. It looks like you're on your way to an epic journey.

actions.intent.MAIN.yaml

handler:
  staticPrompt:
    candidates:
    - promptResponse:
        firstSimple:
          variants:
          - speech: A wondrous greeting, adventurer! Welcome to the mythical land of 
                Gryffinberg! Based on your clothes, you are not from around these lands. 
                It looks like you're on your way to an epic journey.
transitionToScene: actions.scene.END_CONVERSATION
  1. Save the file.

Test the main invocation in the simulator

The Actions console provides a web tool for testing your Action called the simulator. The interface simulates hardware devices and their settings, so you can converse with your Action as if it were running on a smart display, phone, speaker, or KaiOS.

When you invoke your Action, it should now respond with the customized prompt you added ("A wondrous greeting, adventurer!...").

You can use the gactions deploy preview command to test your Action in the console without updating the version of your Actions project. When you run this command, none of the changes you make in your local file system are propagated to deployed versions of your Actions project, but you can test them on a preview version.

To test your Action's main invocation in the simulator, follow these steps:

  1. To deploy your project to the Actions console for testing, run the following command in the terminal:
gactions deploy preview

You should receive output that looks like the following:

✔ Done. You can now test your changes in Simulator with this URL: http://console.actions.google.com/project/{project-id}/simulator?disableAutoPreview
  1. Copy the provided URL and paste it into a browser.
  2. To invoke your Action in the simulator, type Talk to my test app in the Input field and press Enter.

656f5736af6a5a07.png

When you trigger your Action's main invocation, the Assistant responds with your customized welcome message. At this point, the conversation ends after the Assistant responds with a greeting. In the next section, you modify your Action so that the conversation continues.

View event logs

When you are in the Test tab in the Actions console, the panel shows the event logs, which display the conversation history as event logs. Each event log displays the events that happen during that turn of the conversation.

Your Action currently has one event log, which shows both the user's input ("Talk to my test app") and your Action's response. The following screenshot shows your Action's event log:

a1b748d1fcebca80.png

If you click on the downwards arrow next to Talk to my test app in the event log, you can see the events, arranged chronologically, that occurred in that turn of the conversation:

  • userInput: Corresponds to the user's input ("Talk to my test app")
  • interactionMatch: Corresponds to your Action's main invocation response, which was triggered by the user's input. If you expand this row by clicking the arrow, you can see the prompt you added for the main invocation (A wondrous greeting, adventurer!...)
  • endConversation: Corresponds to the selected transition in the Main invocation intent, which currently ends the conversation. You'll learn more about transitions in the next section of this codelab.

Event logs provide visibility into how your Action is working and are useful tools for debugging your Action if you have any issues. To see the details of an event, click the arrow next to the event name, as shown in the following screenshot:

fcc389b59af5bef1.png

4. Create your Action's conversation

Now that you've defined what happens after a user invokes your Action, you can build out the rest of your Action's conversation. Before continuing on with this codelab, familiarize yourself with the following terms to understand how your Action's conversation works:

Your Action can have one or many scenes, and you must activate each scene before it can run. The Action you build in this codelab only has one scene titled Start. The most common way to activate a scene is to configure your Action so that, when the user input matches an intent within a scene, that intent triggers the transition to another scene and activates it.

For example, imagine a hypothetical Action that provides the user with animal facts. When the user invokes this Action, the Main invocation intent is matched and triggers the transition to a scene named Facts. This transition activates the Facts scene, which sends the following prompt to the user: Would you like to hear a fact about cats or dogs? Within the Facts scene is a custom intent called Cat, which contains training phrases that the user might say to hear a cat fact, like "I want to hear a cat fact" or "cat". When the user asks to hear a cat fact, the Cat intent is matched, and triggers a transition to a scene called Cat fact. The Cat fact scene activates and sends a prompt to the user that includes a cat fact.

a78f549c90c3bff6.png

Figure 1. The flow of a typical conversational turn in an Action built with the Actions SDK.

Together, scenes, intents, and transitions make up the logic for your conversation and define the various paths your user can take through your Action's conversation. In the following section, you create a scene and define how that scene is activated after a user invokes your Action.

Transition from main invocation to scene

In this section, you create a new scene called Start, which sends a prompt to the user asking if they would like their fortune told. You also add a transition from the main invocation to the new Start scene.

To create this scene and add a transition to it, follow these steps:

  1. Open custom/global/actions.intent.MAIN.yaml in your text editor.
  2. Replace the text in the transitionToScene field (actions.scene.END_CONVERSATION) with the following: transitionToScene: Start

actions.intent.MAIN.yaml

handler:
  staticPrompt:
    candidates:
    - promptResponse:
        firstSimple:
          variants:
          - speech: Welcome to the mythical land of  Gryffinberg! Based on your clothes,
              you are not from around these lands. It looks like you're on your way
              to an epic journey.
transitionToScene: Start

This tells your Action to transition from the main invocation to the Start scene.

  1. Save the file.
  2. In the terminal, create a new scenes directory in the custom directory:
mkdir custom/scenes 
  1. Create a new file called Start.yaml in the scenes directory, which represents the start scene in your Action:
touch custom/scenes/Start.yaml 
  1. Open Start.yaml in your text editor.
  2. Paste the following code into the Start.yaml file:

Start.yaml

onEnter:
  staticPrompt:
    candidates:
    - promptResponse:
        firstSimple:
          variants:
          - speech: Before you continue on your quest, would you like your fortune
              told?

In the Start.yaml file code, there is a field called onEnter, which is the first stage that runs in a scene's lifecycle.

In this case, the prompt (Before you continue on your quest...) is added to the prompt queue when the user first enters the Start scene.

Add suggestion chips

Suggestion chips offer clickable suggestions for the user that your Action processes as user input. In this section, you add Yes and No suggestion chips that appear below the prompt you just configured (Before you continue on your quest, would you like your fortune told?) to support users on devices with screens.

To add suggestion chips to the Start scene's prompt, follow these steps:

  1. Update the code in Start.yaml to match the following code snippet, which includes the code for configuring suggestion chips:

Start.yaml

onEnter:
  staticPrompt:
    candidates:
    - promptResponse:
        firstSimple:
          variants:
          - speech: Before you continue on your quest, would you like your fortune
              told?
        suggestions:
        - title: "Yes"
        - title: "No"
  1. Save the file.

Test your Action in the simulator

At this point, your Action should transition from the main invocation to the Start scene and ask the user if they'd like their fortune told. Suggestion chips should also appear in the simulated display.

To test your Action in the simulator, follow these steps:

  1. In the terminal, run the following command:
gactions deploy preview

You should receive output that looks like the following:

✔ Done. You can now test your changes in Simulator with this URL: http://console.actions.google.com/project/{project-id}/simulator?disableAutoPreview
  1. Copy the provided URL and paste it into a browser.
  2. Click Test to take you to the simulator.
  3. Type Talk to my test app in the Input field and press Enter. Your Action should respond with the Main invocation prompt and the added Start scene prompt, "Before you continue on your quest, would you like your fortune told?", with the suggestion chips displayed.

The following screenshot shows this interaction:

3c2013ebb2da886a.png

  1. Click the Yes or No suggestion chip to respond to the prompt. (You can also say "Yes" or "No" or enter Yes or No in the Input field.)

When you respond to the prompt, your Action responds with a message indicating that it can't understand your input: "Sorry, I didn't catch that. Can you try again?" Since you haven't yet configured your Action to understand and respond to "Yes" or "No" input, your Action matches your input to a NO_MATCH intent.

By default, the NO_MATCH system intent provides generic responses, but you can customize these responses to indicate to the user that you didn't understand their input. Assistant ends the user's conversation with your Action after it can't match user input three times.

Add yes and no intents

Now that users can respond to the question your Action poses, you can configure your Action to understand the users' responses ("Yes" or "No"). In the following sections, you create custom intents that are matched when the user says "Yes" or "No", and add these intents to the Start scene.

Create yes intent

To create the yes intent, follow these steps:

  1. In the terminal, create a new directory called intents in the custom directory:
mkdir custom/intents 
  1. Create a new file called yes.yaml in the intents directory:
touch custom/intents/yes.yaml
  1. Open yes.yaml in your text editor.
  2. Paste the following code snippet containing training phrases into yes.yaml:

yes.yaml

trainingPhrases:
- of course
- let's do it
- ok
- sure
- "y"
- "yes"
  1. Save the file.

Add yes intent to Start scene

Now, the Action can understand when a user is expressing a "yes" intent. You can add the yes custom intent to the Start scene, since the user is responding to the Start prompt ("Before you continue on your quest, would you like your fortune told?").

To add this custom intent to the Start scene, follow these steps:

  1. Open custom/scenes/Start.yaml in your text editor.
  2. Add the intentEvents and yes handler to the end of the Start.yaml file:

Start.yaml

intentEvents:
- handler:
    staticPrompt:
      candidates:
      - promptResponse:
          firstSimple:
            variants:
            - speech: Your future depends on the item you choose to use for your quest. Choose wisely! Farewell, stranger.
  intent: "yes"
  transitionToScene: actions.scene.END_CONVERSATION

When the yes intent is matched, the prompt "Your future depends on the item you choose to use for your quest..." is added to the prompt queue. The Start scene then transitions to the actions.scene.END_CONVERSATION system scene, which delivers the prompts in the prompt queue and ends the conversation.

Test yes intent in simulator

At this point, your Action understands when the user wants to hear their fortune and returns the appropriate response.

To test this intent in the simulator, follow these steps:

  1. In the terminal, run the following command:
gactions deploy preview

You should receive output that looks like the following:

✔ Done. You can now test your changes in Simulator with this URL: http://console.actions.google.com/project/{project-id}/simulator?disableAutoPreview
  1. Copy the provided URL and paste it into a browser.
  2. Click Test to take you to the simulator.
  3. To test your Action in the simulator, type Talk to my test app in the Input field and press Enter.
  4. Type Yes in the Input field and press Enter. Alternatively, click the Yes suggestion chip.

f131998710d8ffd8.png

Your Action responds to the user and tells them their fortune depends on the aid they choose. Your Action then ends the session because you configured the End conversation transition for the yes intent.

Create no intent

Now, you can create the no intent so your Action can understand and respond to the user when they don't want to hear their fortune.

To create this intent, follow these steps:

  1. In the terminal, create a new file called no.yaml in the intents directory:
touch custom/intents/no.yaml
  1. Open no.yaml in your text editor.
  2. Paste the following training phrases into the no.yaml file:

no.yaml

trainingPhrases:
- nope
- I don't want
- "n"
- "no"
- nah
- no thanks
  1. Save the file.

Add no intent to Start scene

Now, the Action can understand when a user is expressing "no" or something similar to "no", like "nope". You need to add the no custom intent to the Start scene because the user is responding to the Start prompt ("Before you continue on your quest, would you like your fortune told?").

To add this intent for the Start scene, follow these steps:

  1. Open custom/scenes/Start.yaml in your text editor.
  2. Add the following no handler below the yes handler in Start.yaml:

Start.yaml

- handler:
    staticPrompt:
      candidates:
      - promptResponse:
          firstSimple:
            variants:
            - speech: I understand, stranger. Best of luck on your quest! Farewell.
  intent: "no"
  transitionToScene: actions.scene.END_CONVERSATION
  1. Save the file.

Test no intent in simulator

At this point, your Action understands when the user does not want to hear their fortune and returns the appropriate response.

To test this intent in the simulator, follow these steps:

  1. In the terminal, run the following command:
gactions deploy preview

You should receive output that looks like the following:

✔ Done. You can now test your changes in Simulator with this URL: http://console.actions.google.com/project/{project-id}/simulator?disableAutoPreview
  1. Copy the provided URL and paste it into a browser.
  2. Click Test to take you to the simulator.
  3. Type Talk to my test app in the Input field and press Enter.
  4. Type No in the Input field and press Enter. Alternatively, click the No suggestion chip.

c0c8b04066577eb2.png

Instead of giving the user their fortune, your Action wishes them luck on their journey. Your Action then ends the session because you configured the End conversation transition for the no intent.

5. Implement fulfillment

Currently, your Action's responses are static; when a scene containing a prompt is activated, your Action sends the same prompt each time. In this section, you implement fulfillment that contains the logic to construct a dynamic conversational response.

Your fulfillment identifies whether the user is a returning user or a new user and modifies the greeting message of the Action for returning users. The greeting message is shortened for returning users and acknowledges the user's return: "A wondrous greeting, adventurer! Welcome back to the mythical land of Gryffinberg!"

For this codelab, you use the Cloud Functions editor to edit and deploy your fulfillment code.

Your Action can trigger webhooks that notify your fulfillment of an event that occurs during an invocation or specific parts of a scene's execution. When a webhook is triggered, your Action sends a request with a JSON payload to your fulfillment along with the name of the handler to use to process the event. This handler carries out some logic and returns a corresponding JSON response.

Build your fulfillment

In this section, you modify your fulfillment to generate different prompts for returning users and new users when they invoke your Action.

To add this logic to your fulfillment, follow these steps:

  1. In the terminal, make sure you're in the root directory of your project and create a new webhooks directory:
mkdir webhooks 
  1. Create a new file called ActionsOnGoogleFulfillment.yaml in the webhooks directory:
touch webhooks/ActionsOnGoogleFulfillment.yaml
  1. Open ActionsOnGoogleFulfillment.yaml in your text editor.
  2. Add the greeting handler and inlineCloudFunction content to the ActionsOnGoogleFulfillment.yaml file:

ActionsOnGoogleFulfillment.yaml

handlers:
- name: greeting
inlineCloudFunction:
  executeFunction: ActionsOnGoogleFulfillment

The ActionsOnGoogleFulfillment.yaml file defines your webhook handlers (like the greeting handler) and tells your Action to use Cloud Functions as the webhook endpoint.

  1. Create a new ActionsOnGoogleFulfillment directory in the webhooks directory:
mkdir webhooks/ActionsOnGoogleFulfillment
  1. Create a new file called index.js in the ActionsOnGoogleFulfillment directory:
touch webhooks/ActionsOnGoogleFulfillment/index.js
  1. Open index.js in your text editor.
  2. Add the following code to index.js:

index.js

const { conversation } = require('@assistant/conversation');
const functions = require('firebase-functions');

const app = conversation({debug: true});

app.handle('greeting', conv => {
 let message = 'A wondrous greeting, adventurer! Welcome back to the mythical land of Gryffinberg!';
 if (!conv.user.lastSeenTime) {
   message = 'Welcome to the mythical land of  Gryffinberg! Based on your clothes, you are not from around these lands. It looks like you\'re on your way to an epic journey.';
 }
 conv.add(message);
});


exports.ActionsOnGoogleFulfillment = functions.https.onRequest(app);

This code defines the greeting handler, which sends the appropriate greeting to the

user.

  1. Save the file.
  2. Create a new file called package.json in the ActionsOnGoogleFulfillment directory:
touch webhooks/ActionsOnGoogleFulfillment/package.json

The package.json file specifies dependencies and other metadata for your webhook.

  1. Open package.json in your text editor.
  2. Copy the code from this GitHub repository and paste it into the package.json file.
  3. Save the file.

Understand the code

Your fulfillment, which uses the Actions on Google Fulfillment library for Node.js, responds to HTTP requests from Google Assistant.

In the previous code snippet, you define the greeting handler, which checks if the user has previously visited the Action with the lastSeenTime property. If the lastSeenTime property is not defined, the user is new and receives the greeting intended for new users. Otherwise, the message acknowledges the user's return and generates a modified greeting.

Update main invocation to trigger a webhook

Now that you've defined the greeting function, you can configure the greeting event handler in your main invocation intent so your Action knows to call this function when the user invokes your Action.

To configure your Action to call the new greeting handler, follow these steps:

  1. Open custom/global/actions.intent.MAIN.yaml in your text editor.
  2. Replace the code in actions.intent.MAIN.yaml with the following code:

actions.intent.MAIN.yaml

handler:
  webhookHandler: greeting
transitionToScene: Start
  1. Save the file.

Now, when your main invocation intent is matched, the greeting webhook handler is called.

Test updated main invocation in simulator

To test your Action in the simulator, follow these steps:

  1. In the terminal, run the following command:
gactions deploy preview

You should receive output that looks like the following:

✔ Done. You can now test your changes in Simulator with this URL: http://console.actions.google.com/project/{project-id}/simulator?disableAutoPreview
  1. Copy the provided URL and paste it into a browser.
  2. To test your Action in the simulator, type Talk to my test app in the Input field and press Enter.

Because you tested your Action earlier in this codelab, you are not a new user, so you receive the following shortened greeting: "A wondrous greeting, adventurer! Welcome back to the mythical land of Gryffinberg!..."

6. Visualize Action with Actions Builder

The Actions SDK has interoperability with a web-based IDE called Actions Builder that is integrated into the Actions console. You can push your local file system to the draft of your Action in the console with the gactions push command. Unlike gactions deploy preview, which only allows you to test your Action in the simulator, gactions push moves all the content from your local files to Actions Builder.

The Actions console provides a visual representation of your Action's configuration. Seeing your Action mapped out visually can be useful during development, and doesn't affect the version of your Action that's served for testing.

To push your Actions project and view it in the Actions console, follow these steps:

  1. In the terminal, run the following command to push your project to the Actions console:
gactions push

You should receive output that looks like the following:

✔ Done. Files were pushed to Actions Console, and you can now view your project with this URL: https://console.actions.google.com/project/{project-id}/overview. If you want to test your changes, run "gactions deploy preview", or navigate to the Test section in the Console.
  1. Copy the provided URL and paste it into a browser.
  2. In the Actions console, click Develop in the top navigation bar.
  3. Click the drop-down arrow next to Scenes and click Start. You should see a visual representation of your Action's Start scene, as shown in the following screenshot:

332404b148609e96.png

Clean up your project [recommended]

To avoid incurring possible charges, it is recommended to remove projects that you don't intend to use. To delete the projects you created in this codelab, follow these steps:

  1. To delete the Cloud Project and resources, complete the steps listed in the Shutting down (deleting) projects section.
  1. Optional: To immediately remove your project from the Actions console, complete the steps listed in the Delete a project section. If you don't complete this step, your project is automatically removed after approximately 30 days.

7. Congratulations!

You know the basics of building Actions for Google Assistant with the Actions SDK.

What you covered

  • How to set up an Actions project in the Actions console
  • How to use the Actions SDK to build your Actions project on your local file system
  • How to add a prompt to the main invocation so that users can start a conversation with your Action
  • How to create a conversational interface with scenes, intents, transitions, suggestion chips, and fulfillment
  • How to test your Action with the Actions simulator

Learn more

Explore the following resources to learn more about building Actions for Google Assistant:

Follow @ActionsOnGoogle on Twitter for the latest announcements and tweet with #AoGDevs to share what you build!

Feedback survey

Before you go, please fill out a brief survey about your experience.