1. Overview
The Google Assistant developer platform lets you create software to extend the functionality of Google Assistant, a virtual personal assistant, across more than 1 billion devices, including smart speakers, phones, cars, TVs, headphones, and more. Users engage Assistant in conversation to get things done, such as buying groceries or booking a ride. As a developer, you can use the Assistant developer platform to easily create and manage delightful and effective conversational experiences between users and your own third-party fulfillment service.
This codelab covers beginner-level concepts for developing with the Actions SDK for Google Assistant; you do not need any prior experience with the platform to complete it. In this codelab, you build a simple Action for Google Assistant that tells users their fortune as they begin their adventure in the mythical land of Gryffinberg. In the Actions SDK level 2 codelab, you build out this Action further to customize the user's fortune based on their input.
What you'll build
In this codelab, you build a simple Action with the following functions:
- Responds to users with a greeting message
- Asks users a question and responds appropriately to the user's selection
- Provides suggestion chips users can click to provide input
- Modifies the greeting message to the user based on whether they are a returning user
When you finish this codelab, your completed Action will have the following conversational flow (the text next to the microphone represents the user's input, while the text next to the speaker represents the Action's response):
What you'll learn
- How to create a project in the Actions console
- How to use the gactions tool to push and pull your Action project between the Actions console and your local file system
- How to send a prompt to the user after they invoke your Action
- How to process a user's input and return a response
- How to test your Action in the Actions simulator
- How to implement fulfillment using the Cloud Functions editor
What you'll need
The following tools must be in your environment:
- An IDE or text editor of your choice
- A terminal to run shell commands for Node.js and npm
- A web browser, such as Google Chrome
2. Set up
The following sections describe how to set up your development environment and create your Actions project.
Check your Google permission settings
To test the Action you build in this codelab, you need to enable the necessary permissions so the simulator can access your Action.
To enable permissions, follow these steps:
- Go to Activity Controls.
- If you haven't already done so, sign in with your Google Account.
- Enable the following permissions:
- Web & App Activity
- Under Web & App Activity, select the Include Chrome history and activity from sites, apps, and devices that use Google services checkbox**.**
Create an Actions project
Your Actions project is a container for your Action.
To create your Actions project for this codelab, follow these steps:
- Open the Actions console.
- Click New project.
- Enter a Project name, such as
actions-codelab
. (The name is for your internal reference. Later, you can set an external name for your project.)
- Click Create project.
- In the What kind of Action do you want to build? screen, select the Custom card.
- Click Next.
- In the How do you want to build it? screen, select the Blank project card.
- Click Start building.
Save the Project ID for your Action
The Project ID is a unique identifier for your Action. You need your Project ID for several steps in this codelab.
To retrieve your Project ID, follow these steps:
- In the Actions console, click the three vertical dots (insert icon here).
- Click Project settings.
- Copy the project ID**.**
Associate a billing account
If you don't already have a billing account, you need to create a billing account and associate it with your project in Google Cloud to deploy your fulfillment later with Cloud Functions.
To associate a billing account with your project, follow these steps:
- Go to the Google Cloud Platform billing page.
- Click Add billing account or Create Account.
- Enter your payment information.
- Click Start my free trial or Submit and enable billing.
- Go to the Google Cloud Platform billing page.
- Click the My Projects tab.
- Click the three dots under Actions next to the Actions project for the codelab.
- Click Change billing.
- In the drop-down menu, select the billing account you configured.
- Click Set account.
To avoid incurring charges, follow the steps in the Clean up your project section at the end of this codelab.
Install the gactions command-line interface
In this codelab, you use the gactions command-line interface (CLI) tool to synchronize your Actions project between the Actions console and your local file system.
To install the gactions CLI, follow the instructions in Install the gactions command-line tool.
Download your Actions project
Start developing your Action by downloading your Actions project from the Actions console.
To download your Actions project, follow these steps:
- To make a new directory and change into that directory, run the following commands:
mkdir myproject cd myproject
- To copy the configuration of your Actions project to your local file system, run the following command:
gactions pull --project-id <projectID>
Understand the file structure
The Actions project you download from the Actions console is represented in a YAML file structure. The following image shows a high-level representation of the file structure:
The file structure consists of the following:
actions/
: Represents your Actions project. The system callsactions.yaml
when your Action is invoked, which then calls thecustom/global/actions.intent.MAIN.yaml
file.custom/
: The directory you'll work in to modify your Action.global/
: This directory contains system intents that the platform adds to your project automatically. You'll learn more about system intents later in this codelab.manifest.yaml
: A file that contains information that's transportable, or not specific to any given project and can be moved between projects.settings/
: Represents the settings of an Actions project, such as the display name, default locale, and category.
3. Start a conversation
Users start the conversation with your Action through invocation. For example, if you have an Action named MovieTime, users can invoke your Action by saying a phrase like "Hey Google, talk to MovieTime", where MovieTime is the display name. Your Action must have a display name if you want to deploy it to production; however, to test your Action, you don't need to define the display name. Instead, you can use the phrase "Talk to my test app" in the simulator to invoke your Action. You learn more about the simulator later in this section.
You must edit the main invocation to define what happens after a user invokes your Action.
By default, your Action provides a generic prompt when your invocation is triggered ("Start building your Action by defining main invocation.").
In the next section, you customize the prompt for your main invocation in the custom/global/actions.intent.MAIN.yaml
file.
Set up main invocation
You can edit your main invocation prompt in the actions.intent.MAIN.yaml
file.
To modify the prompt your Action sends back to the user when they invoke your Action, follow these steps:
- Open
custom/global/actions.intent.MAIN.yaml
in your text editor. - Replace the text in the
speech
field (Start building your action...
) with the following welcome message:A wondrous greeting, adventurer! Welcome to the mythical land of Gryffinberg! Based on your clothes, you are not from around these lands. It looks like you're on your way to an epic journey.
actions.intent.MAIN.yaml
handler:
staticPrompt:
candidates:
- promptResponse:
firstSimple:
variants:
- speech: A wondrous greeting, adventurer! Welcome to the mythical land of
Gryffinberg! Based on your clothes, you are not from around these lands.
It looks like you're on your way to an epic journey.
transitionToScene: actions.scene.END_CONVERSATION
- Save the file.
Test the main invocation in the simulator
The Actions console provides a web tool for testing your Action called the simulator. The interface simulates hardware devices and their settings, so you can converse with your Action as if it were running on a smart display, phone, speaker, or KaiOS.
When you invoke your Action, it should now respond with the customized prompt you added ("A wondrous greeting, adventurer!...").
You can use the gactions deploy preview
command to test your Action in the console without updating the version of your Actions project. When you run this command, none of the changes you make in your local file system are propagated to deployed versions of your Actions project, but you can test them on a preview version.
To test your Action's main invocation in the simulator, follow these steps:
- To deploy your project to the Actions console for testing, run the following command in the terminal:
gactions deploy preview
You should receive output that looks like the following:
✔ Done. You can now test your changes in Simulator with this URL: http://console.actions.google.com/project/{project-id}/simulator?disableAutoPreview
- Copy the provided URL and paste it into a browser.
- To invoke your Action in the simulator, type
Talk to my test app
in the Input field and pressEnter
.
When you trigger your Action's main invocation, the Assistant responds with your customized welcome message. At this point, the conversation ends after the Assistant responds with a greeting. In the next section, you modify your Action so that the conversation continues.
View event logs
When you are in the Test tab in the Actions console, the panel shows the event logs, which display the conversation history as event logs. Each event log displays the events that happen during that turn of the conversation.
Your Action currently has one event log, which shows both the user's input ("Talk to my test app") and your Action's response. The following screenshot shows your Action's event log:
If you click on the downwards arrow next to Talk to my test app
in the event log, you can see the events, arranged chronologically, that occurred in that turn of the conversation:
userInput
: Corresponds to the user's input ("Talk to my test app")interactionMatch
: Corresponds to your Action's main invocation response, which was triggered by the user's input. If you expand this row by clicking the arrow, you can see the prompt you added for the main invocation (A wondrous greeting, adventurer!...
)endConversation
: Corresponds to the selected transition in theMain invocation
intent, which currently ends the conversation. You'll learn more about transitions in the next section of this codelab.
Event logs provide visibility into how your Action is working and are useful tools for debugging your Action if you have any issues. To see the details of an event, click the arrow next to the event name, as shown in the following screenshot:
4. Create your Action's conversation
Now that you've defined what happens after a user invokes your Action, you can build out the rest of your Action's conversation. Before continuing on with this codelab, familiarize yourself with the following terms to understand how your Action's conversation works:
Your Action can have one or many scenes, and you must activate each scene before it can run. The Action you build in this codelab only has one scene titled Start
. The most common way to activate a scene is to configure your Action so that, when the user input matches an intent within a scene, that intent triggers the transition to another scene and activates it.
For example, imagine a hypothetical Action that provides the user with animal facts. When the user invokes this Action, the Main invocation
intent is matched and triggers the transition to a scene named Facts.
This transition activates the Facts
scene, which sends the following prompt to the user: Would you like to hear a fact about cats or dogs?
Within the Facts
scene is a custom intent called Cat
, which contains training phrases that the user might say to hear a cat fact, like "I want to hear a cat fact" or "cat". When the user asks to hear a cat fact, the Cat
intent is matched, and triggers a transition to a scene called Cat fact.
The Cat fact
scene activates and sends a prompt to the user that includes a cat fact.
Figure 1. The flow of a typical conversational turn in an Action built with the Actions SDK.
Together, scenes, intents, and transitions make up the logic for your conversation and define the various paths your user can take through your Action's conversation. In the following section, you create a scene and define how that scene is activated after a user invokes your Action.
Transition from main invocation to scene
In this section, you create a new scene called Start
, which sends a prompt to the user asking if they would like their fortune told. You also add a transition from the main invocation to the new Start
scene.
To create this scene and add a transition to it, follow these steps:
- Open
custom/global/actions.intent.MAIN.yaml
in your text editor. - Replace the text in the
transitionToScene
field (actions.scene.END_CONVERSATION
) with the following:transitionToScene: Start
actions.intent.MAIN.yaml
handler:
staticPrompt:
candidates:
- promptResponse:
firstSimple:
variants:
- speech: Welcome to the mythical land of Gryffinberg! Based on your clothes,
you are not from around these lands. It looks like you're on your way
to an epic journey.
transitionToScene: Start
This tells your Action to transition from the main invocation to the Start
scene.
- Save the file.
- In the terminal, create a new
scenes
directory in thecustom
directory:
mkdir custom/scenes
- Create a new file called
Start.yaml
in thescenes
directory, which represents thestart
scene in your Action:
touch custom/scenes/Start.yaml
- Open
Start.yaml
in your text editor. - Paste the following code into the
Start.yaml
file:
Start.yaml
onEnter:
staticPrompt:
candidates:
- promptResponse:
firstSimple:
variants:
- speech: Before you continue on your quest, would you like your fortune
told?
In the Start.yaml
file code, there is a field called onEnter
, which is the first stage that runs in a scene's lifecycle.
In this case, the prompt (Before you continue on your quest...
) is added to the prompt queue when the user first enters the Start
scene.
Add suggestion chips
Suggestion chips offer clickable suggestions for the user that your Action processes as user input. In this section, you add Yes
and No
suggestion chips that appear below the prompt you just configured (Before you continue on your quest, would you like your fortune told?
) to support users on devices with screens.
To add suggestion chips to the Start
scene's prompt, follow these steps:
- Update the code in
Start.yaml
to match the following code snippet, which includes the code for configuring suggestion chips:
Start.yaml
onEnter:
staticPrompt:
candidates:
- promptResponse:
firstSimple:
variants:
- speech: Before you continue on your quest, would you like your fortune
told?
suggestions:
- title: "Yes"
- title: "No"
- Save the file.
Test your Action in the simulator
At this point, your Action should transition from the main invocation to the Start
scene and ask the user if they'd like their fortune told. Suggestion chips should also appear in the simulated display.
To test your Action in the simulator, follow these steps:
- In the terminal, run the following command:
gactions deploy preview
You should receive output that looks like the following:
✔ Done. You can now test your changes in Simulator with this URL: http://console.actions.google.com/project/{project-id}/simulator?disableAutoPreview
- Copy the provided URL and paste it into a browser.
- Click Test to take you to the simulator.
- Type
Talk to my test app
in the Input field and pressEnter
. Your Action should respond with theMain invocation
prompt and the addedStart
scene prompt, "Before you continue on your quest, would you like your fortune told?", with the suggestion chips displayed.
The following screenshot shows this interaction:
- Click the Yes or No suggestion chip to respond to the prompt. (You can also say "Yes" or "No" or enter
Yes
orNo
in the Input field.)
When you respond to the prompt, your Action responds with a message indicating that it can't understand your input: "Sorry, I didn't catch that. Can you try again?" Since you haven't yet configured your Action to understand and respond to "Yes" or "No" input, your Action matches your input to a NO_MATCH
intent.
By default, the NO_MATCH
system intent provides generic responses, but you can customize these responses to indicate to the user that you didn't understand their input. Assistant ends the user's conversation with your Action after it can't match user input three times.
Add yes
and no
intents
Now that users can respond to the question your Action poses, you can configure your Action to understand the users' responses ("Yes" or "No"). In the following sections, you create custom intents that are matched when the user says "Yes" or "No", and add these intents to the Start
scene.
Create yes
intent
To create the yes
intent, follow these steps:
- In the terminal, create a new directory called
intents
in thecustom
directory:
mkdir custom/intents
- Create a new file called
yes.yaml
in theintents
directory:
touch custom/intents/yes.yaml
- Open
yes.yaml
in your text editor. - Paste the following code snippet containing training phrases into
yes.yaml
:
yes.yaml
trainingPhrases:
- of course
- let's do it
- ok
- sure
- "y"
- "yes"
- Save the file.
Add yes
intent to Start
scene
Now, the Action can understand when a user is expressing a "yes" intent. You can add the yes
custom intent to the Start
scene, since the user is responding to the Start
prompt ("Before you continue on your quest, would you like your fortune told?").
To add this custom intent to the Start
scene, follow these steps:
- Open
custom/scenes/Start.yaml
in your text editor. - Add the
intentEvents
andyes
handler to the end of theStart.yaml
file:
Start.yaml
intentEvents:
- handler:
staticPrompt:
candidates:
- promptResponse:
firstSimple:
variants:
- speech: Your future depends on the item you choose to use for your quest. Choose wisely! Farewell, stranger.
intent: "yes"
transitionToScene: actions.scene.END_CONVERSATION
When the yes
intent is matched, the prompt "Your future depends on the item you choose to use for your quest..." is added to the prompt queue. The Start
scene then transitions to the actions.scene.END_CONVERSATION
system scene, which delivers the prompts in the prompt queue and ends the conversation.
Test yes
intent in simulator
At this point, your Action understands when the user wants to hear their fortune and returns the appropriate response.
To test this intent in the simulator, follow these steps:
- In the terminal, run the following command:
gactions deploy preview
You should receive output that looks like the following:
✔ Done. You can now test your changes in Simulator with this URL: http://console.actions.google.com/project/{project-id}/simulator?disableAutoPreview
- Copy the provided URL and paste it into a browser.
- Click Test to take you to the simulator.
- To test your Action in the simulator, type
Talk to my test app
in the Input field and pressEnter
. - Type
Yes
in the Input field and pressEnter
. Alternatively, click the Yes suggestion chip.
Your Action responds to the user and tells them their fortune depends on the aid they choose. Your Action then ends the session because you configured the End conversation
transition for the yes
intent.
Create no
intent
Now, you can create the no
intent so your Action can understand and respond to the user when they don't want to hear their fortune.
To create this intent, follow these steps:
- In the terminal, create a new file called
no.yaml
in theintents
directory:
touch custom/intents/no.yaml
- Open
no.yaml
in your text editor. - Paste the following training phrases into the
no.yaml
file:
no.yaml
trainingPhrases:
- nope
- I don't want
- "n"
- "no"
- nah
- no thanks
- Save the file.
Add no
intent to Start
scene
Now, the Action can understand when a user is expressing "no" or something similar to "no", like "nope". You need to add the no
custom intent to the Start
scene because the user is responding to the Start
prompt ("Before you continue on your quest, would you like your fortune told?").
To add this intent for the Start
scene, follow these steps:
- Open
custom/scenes/Start.yaml
in your text editor. - Add the following
no
handler below theyes
handler inStart.yaml
:
Start.yaml
- handler:
staticPrompt:
candidates:
- promptResponse:
firstSimple:
variants:
- speech: I understand, stranger. Best of luck on your quest! Farewell.
intent: "no"
transitionToScene: actions.scene.END_CONVERSATION
- Save the file.
Test no
intent in simulator
At this point, your Action understands when the user does not want to hear their fortune and returns the appropriate response.
To test this intent in the simulator, follow these steps:
- In the terminal, run the following command:
gactions deploy preview
You should receive output that looks like the following:
✔ Done. You can now test your changes in Simulator with this URL: http://console.actions.google.com/project/{project-id}/simulator?disableAutoPreview
- Copy the provided URL and paste it into a browser.
- Click Test to take you to the simulator.
- Type
Talk to my test app
in the Input field and pressEnter
. - Type
No
in the Input field and pressEnter
. Alternatively, click theNo
suggestion chip.
Instead of giving the user their fortune, your Action wishes them luck on their journey. Your Action then ends the session because you configured the End conversation
transition for the no
intent.
5. Implement fulfillment
Currently, your Action's responses are static; when a scene containing a prompt is activated, your Action sends the same prompt each time. In this section, you implement fulfillment that contains the logic to construct a dynamic conversational response.
Your fulfillment identifies whether the user is a returning user or a new user and modifies the greeting message of the Action for returning users. The greeting message is shortened for returning users and acknowledges the user's return: "A wondrous greeting, adventurer! Welcome back to the mythical land of Gryffinberg!"
For this codelab, you use the Cloud Functions editor to edit and deploy your fulfillment code.
Your Action can trigger webhooks that notify your fulfillment of an event that occurs during an invocation or specific parts of a scene's execution. When a webhook is triggered, your Action sends a request with a JSON payload to your fulfillment along with the name of the handler to use to process the event. This handler carries out some logic and returns a corresponding JSON response.
Build your fulfillment
In this section, you modify your fulfillment to generate different prompts for returning users and new users when they invoke your Action.
To add this logic to your fulfillment, follow these steps:
- In the terminal, make sure you're in the root directory of your project and create a new
webhooks
directory:
mkdir webhooks
- Create a new file called
ActionsOnGoogleFulfillment.yaml
in thewebhooks
directory:
touch webhooks/ActionsOnGoogleFulfillment.yaml
- Open
ActionsOnGoogleFulfillment.yaml
in your text editor. - Add the
greeting
handler andinlineCloudFunction
content to theActionsOnGoogleFulfillment.yaml
file:
ActionsOnGoogleFulfillment.yaml
handlers:
- name: greeting
inlineCloudFunction:
executeFunction: ActionsOnGoogleFulfillment
The ActionsOnGoogleFulfillment.yaml
file defines your webhook handlers (like the greeting
handler) and tells your Action to use Cloud Functions as the webhook endpoint.
- Create a new
ActionsOnGoogleFulfillment
directory in thewebhooks
directory:
mkdir webhooks/ActionsOnGoogleFulfillment
- Create a new file called
index.js
in theActionsOnGoogleFulfillment
directory:
touch webhooks/ActionsOnGoogleFulfillment/index.js
- Open
index.js
in your text editor. - Add the following code to
index.js
:
index.js
const { conversation } = require('@assistant/conversation');
const functions = require('firebase-functions');
const app = conversation({debug: true});
app.handle('greeting', conv => {
let message = 'A wondrous greeting, adventurer! Welcome back to the mythical land of Gryffinberg!';
if (!conv.user.lastSeenTime) {
message = 'Welcome to the mythical land of Gryffinberg! Based on your clothes, you are not from around these lands. It looks like you\'re on your way to an epic journey.';
}
conv.add(message);
});
exports.ActionsOnGoogleFulfillment = functions.https.onRequest(app);
This code defines the greeting
handler, which sends the appropriate greeting to the
user.
- Save the file.
- Create a new file called
package.json
in theActionsOnGoogleFulfillment
directory:
touch webhooks/ActionsOnGoogleFulfillment/package.json
The package.json
file specifies dependencies and other metadata for your webhook.
- Open
package.json
in your text editor. - Copy the code from this GitHub repository and paste it into the
package.json
file. - Save the file.
Understand the code
Your fulfillment, which uses the Actions on Google Fulfillment library for Node.js, responds to HTTP requests from Google Assistant.
In the previous code snippet, you define the greeting
handler, which checks if the user has previously visited the Action with the lastSeenTime
property. If the lastSeenTime
property is not defined, the user is new and receives the greeting intended for new users. Otherwise, the message acknowledges the user's return and generates a modified greeting.
Update main invocation to trigger a webhook
Now that you've defined the greeting
function, you can configure the greeting
event handler in your main invocation intent so your Action knows to call this function when the user invokes your Action.
To configure your Action to call the new greeting
handler, follow these steps:
- Open
custom/global/actions.intent.MAIN.yaml
in your text editor. - Replace the code in
actions.intent.MAIN.yaml
with the following code:
actions.intent.MAIN.yaml
handler:
webhookHandler: greeting
transitionToScene: Start
- Save the file.
Now, when your main invocation intent is matched, the greeting
webhook handler is called.
Test updated main invocation in simulator
To test your Action in the simulator, follow these steps:
- In the terminal, run the following command:
gactions deploy preview
You should receive output that looks like the following:
✔ Done. You can now test your changes in Simulator with this URL: http://console.actions.google.com/project/{project-id}/simulator?disableAutoPreview
- Copy the provided URL and paste it into a browser.
- To test your Action in the simulator, type
Talk to my test app
in the Input field and pressEnter
.
Because you tested your Action earlier in this codelab, you are not a new user, so you receive the following shortened greeting: "A wondrous greeting, adventurer! Welcome back to the mythical land of Gryffinberg!..."
6. Visualize Action with Actions Builder
The Actions SDK has interoperability with a web-based IDE called Actions Builder that is integrated into the Actions console. You can push your local file system to the draft of your Action in the console with the gactions push
command. Unlike gactions deploy preview
, which only allows you to test your Action in the simulator, gactions push
moves all the content from your local files to Actions Builder.
The Actions console provides a visual representation of your Action's configuration. Seeing your Action mapped out visually can be useful during development, and doesn't affect the version of your Action that's served for testing.
To push your Actions project and view it in the Actions console, follow these steps:
- In the terminal, run the following command to push your project to the Actions console:
gactions push
You should receive output that looks like the following:
✔ Done. Files were pushed to Actions Console, and you can now view your project with this URL: https://console.actions.google.com/project/{project-id}/overview. If you want to test your changes, run "gactions deploy preview", or navigate to the Test section in the Console.
- Copy the provided URL and paste it into a browser.
- In the Actions console, click Develop in the top navigation bar.
- Click the drop-down arrow next to Scenes and click Start. You should see a visual representation of your Action's
Start
scene, as shown in the following screenshot:
Clean up your project [recommended]
To avoid incurring possible charges, it is recommended to remove projects that you don't intend to use. To delete the projects you created in this codelab, follow these steps:
- To delete the Cloud Project and resources, complete the steps listed in the Shutting down (deleting) projects section.
- Optional: To immediately remove your project from the Actions console, complete the steps listed in the Delete a project section. If you don't complete this step, your project is automatically removed after approximately 30 days.
7. Congratulations!
You know the basics of building Actions for Google Assistant with the Actions SDK.
What you covered
- How to set up an Actions project in the Actions console
- How to use the Actions SDK to build your Actions project on your local file system
- How to add a prompt to the main invocation so that users can start a conversation with your Action
- How to create a conversational interface with scenes, intents, transitions, suggestion chips, and fulfillment
- How to test your Action with the Actions simulator
Learn more
Explore the following resources to learn more about building Actions for Google Assistant:
- The official documentation site for developing Actions for Google Assistant
- Actions on Google GitHub page for sample code and libraries
- The official Reddit community for developers working with the Assistant
Follow @ActionsOnGoogle on Twitter for the latest announcements and tweet with #AoGDevs to share what you build!
Feedback survey
Before you go, please fill out a brief survey about your experience.