1. Overview
The Google Assistant developer platform lets you create software to extend the functionality of Google Assistant, a virtual personal assistant, across more than 1 billion devices, including smart speakers, phones, cars, TVs, headphones, and more. Users engage Assistant in conversation to get things done, such as buying groceries or booking a ride. As a developer, you can use the Assistant developer platform to easily create and manage delightful and effective conversational experiences between users and your own third-party fulfillment service.
This codelab covers intermediate-level concepts for developing with Google Assistant and builds off of the Action created in the Build Actions for Google Assistant using Actions SDK (Level 1) codelab. We strongly recommend that you complete the Level 1 codelab before starting this one.
The Action you build in this codelab tells users their fortune for their quest in a mythical land, Gryffinberg, based on the aid they choose.
What you'll build
In this codelab, you build a sophisticated Conversational Action with the following functions:
- Collects data from the user and, depending on the value, modifies the conversational prompts.
- Responds with follow-up questions to further the conversation.
- Creates a game loop so a user can interact with the Action again after receiving a fortune.
Before you start building, you can interact with the live Action on your Google Assistant-enabled device by saying "Hey Google, talk to Fate and Fortune". The default path through this Action for a returning user looks like the following interaction:
What you'll learn
- How to use slots to gather data from the user
- How to use conditions to add logic to a scene
- How to add a game loop
- How to add a supportive path
What you'll need
The prerequisites for this codelab include the following:
- An IDE/text editor of your choice.
- A terminal to run shell commands with NodeJS, npm, and git installed.
- A web browser, such as Google Chrome.
- A completed codelab Level 1 Actions project.
Familiarity with JavaScript (ES6) is strongly recommended, although not required, to understand the fulfillment code for this codelab.
Optional: Get the sample code
You can optionally get the complete level 1 project code from the Actions Builder Codelab Level 1 GitHub repository so you can follow along with this codelab. You can also view the complete level 2 project code in this Github repository.
2. Continue building conversational interface
In the first codelab, you created a simple Conversational Action with a single scene, Start
.
In this codelab, you extend your Action's conversation. In the following sections, you configure your Action to do the following:
- Transition to a new
Fortune
scene when the user wants to hear their fortune - Ask the user which aid they want to choose for their journey
- Deliver a customized fortune based on the user's choice
Transition to and create Fortune
scene
In this section, you do the following:
- Remove the existing prompt from the
Start
scene, which responds to the user and ends the conversation - Define the transition from the
Start
scene to theFortune
scene - Create the
Fortune
scene
To modify the Start
scene and add a transition to the Fortune
scene, follow these steps:
- Open your Actions project from the Level 1 codelab in your text editor.
- Open the
custom/scenes/Start.yaml
file. - Update the
handler
for theyes
intent so your code matches the snippet below:
Start.yaml
intentEvents: - intent: "yes" transitionToScene: Fortune - handler: staticPrompt: candidates: - promptResponse: firstSimple: variants: - speech: I understand, stranger. Best of luck on your quest! Farewell. intent: "no" transitionToScene: actions.scene.END_CONVERSATION
- Save the file.
To create a new scene called Fortune
, follow these steps:
- Navigate to your codelab Level 1 Actions project in the terminal.
- Create a new file called
Fortune.yaml
in thescenes
directory:
touch custom/scenes/Fortune.yaml
You'll edit this file in the next section.
Define conversational logic for Fortune
scene
In this codelab, you configure your Fortune
scene to ask the user, "What do you choose to help you on your quest, a dragon, a translator, or a compass?" You can use a capability called slot filling to gather the necessary information from the user before proceeding.
Your Action provides fortunes for three aids: a dragon, translator, and compass. To configure your Action to identify these three options in a user's input, you must create a new type.
You can use types within a scene's slot filling stage to define the information you want from the user. When the NLU engine detects a slot match in user input, it extracts the slot as a typed parameter, so you can carry out logic with it in a scene.
Create available_options
type
In this section, you create a new type called available_options
, which specifies the three options the users can choose (dragon, translator, and compass) in response to the prompt. You also define a few synonyms for these options in case a user says something similar. In a later section, you'll add the available_options
type to a slot to specify that you want to obtain the user's choice.
To create the available_options
type, follow these steps:
- Create a new directory called
types
:
mkdir custom/types
- Create a new file called
available_options.yaml
in thetypes
directory:
touch custom/types/available_options.yaml
- Open
custom/types/available_options.yaml
in your text editor.
Types are configured as key-value pairs of information, where the key is the name of the type and the values are synonyms for that key. When you define the key, it is automatically added as a value. With the Actions SDK, you represent keys as entities
, and values as synonyms
.
To add the three options the user can choose, follow these steps:
- Add the following
entities
andsynonyms
in theavailable_options.yaml
file:
available_options.yaml
synonym: entities: dragon: synonyms: - dragon - hydra - lizard translator: synonyms: - translator - communicator - machine - decoder - translate compass: synonyms: - compass - direction - guide - navigator matchType: EXACT_MATCH
- Save the file.
Your Action now understands that the available_options
are dragon, translator, and compass, and can also recognize a few corresponding synonyms.
Configure slot filling
Next, you need to configure slot filling in the Fortune
scene. To configure the slot filling logic, follow these steps:
- Open
custom/scenes/Fortune.yaml
in your text editor. - Add the following
slots
data to theFortune.yaml
file:
Fortune.yaml
slots: - commitBehavior: writeSessionParam: chosenOptions name: chosenOptions promptSettings: initialPrompt: staticPrompt: candidates: - promptResponse: firstSimple: variants: - speech: What do you choose to help you on your quest, a dragon, a translator, or a compass? suggestions: - title: Dragon - title: Translator - title: Compass required: true type: name: available_options
- Save the file.
You've now added the available_options
type to the slot, which tells your Action the information you need to gather from the user (their choice of aid) before proceeding. You also configured a prompt within the slot, which is added to the prompt queue when the user reaches the slot filling stage of the scene.
When you name the slot chosenOptions
, the writeSessionsParam
field is updated with the same name ($session.params.chosenOptions
). You can access this parameter by that name in the prompt and in your fulfillment through the client library.
Add condition
Now that you've added a slot that requires the user's choice of aid, you can add a condition to check that the slot data has been obtained before the user can continue the conversation.
In this section, you add the condition scene.slots.status == "FINAL"
, which checks for slot filling to be complete. Once all the slots are filled, the condition adds a prompt (You picked $session.params.chosenOptions.
) to the prompt queue.
To configure the scene.slots.status == "FINAL"
condition, follow these steps:
- Open
custom/scenes/Fortune.yaml
in your text editor. - Add
conditionalEvents
data to the top of theFortune.yaml
file:
Fortune.yaml
conditionalEvents: - condition: scene.slots.status == "FINAL" handler: staticPrompt: candidates: - promptResponse: firstSimple: variants: - speech: You picked $session.params.chosenOptions.
- Save the file.
Test your Action in the simulator
At this point, you have defined which options the user should select to fill the slot. After obtaining this information from the user, your Action should provide a prompt referencing the specific option they've chosen.
To test your Action, follow these steps:
- In the terminal, run the following command:
gactions deploy preview
You should receive output that looks like the following:
✔ Done. You can now test your changes in Simulator with this URL: http://console.actions.google.com/project/{project-id}/simulator?disableAutoPreview
- Copy the provided URL and paste it into a browser.
- Click or type
Talk to my test app
in the Input field and press Enter. - Type
Yes
in the Input field and press Enter. Alternatively, you can click the Yes suggestion chip.
- Click, type, or say
dragon
. You should receive the prompt "You picked dragon."
In the next section, you customize the prompts for each aid the user could select.
Customize prompts using conditions
In this section, you add conditions for each option the user can choose and add a custom prompt for each condition.
Customize the dragon
fortune
To update the condition and customize the prompt for when a user chooses "dragon", follow these steps:
- Open
custom/scenes/Fortune.yaml
in your text editor. - Replace the
conditionalEvents
data with the following snippet in theFortune.yaml
file:
Fortune.yaml
conditionalEvents: - condition: scene.slots.status == "FINAL" && session.params.chosenOptions == "dragon" handler: staticPrompt: candidates: - promptResponse: firstSimple: variants: - speech: The people of Gryffinberg will be awestruck by the beauty and power of the ancient dragon. Much to your dismay, the townspeople fall into dispute over who will receive the honor of riding the dragon first. You return home from your quest without everlasting glory or a dragon.
- Save the file.
Now, when a user says "dragon" or something that sounds similar, your Action provides a fortune based on that selection. Next, you'll add the remaining two selections.
Customize the translator
and compass
fortunes
To add the conditions and customize the prompts for when a user says "translator" or "compass", follow these steps:
- In the
custom/scenes/Fortune.yaml
file, add the other two conditions under thedragon
condition:
Fortune.yaml
- condition: scene.slots.status == "FINAL" && session.params.chosenOptions == "translator" handler: staticPrompt: candidates: - promptResponse: firstSimple: variants: - speech: With the help of the translator, the rival factions in Gryffinberg are finally able to communicate with each other and resolve their disputes. You will complete your quest to restore peace in the town. The translator will be used on many other journeys across the earth. After its work is done, it retires honorably to a premier location in the Gryffinberg History Museum. - condition: scene.slots.status == "FINAL" && session.params.chosenOptions == "compass" handler: staticPrompt: candidates: - promptResponse: firstSimple: variants: - speech: The compass will help you find the mystical and ancient Library of Gryffinberg. Among its infinite stacks of dusty books, you find one entitled "Wisdom of the Ages". By the time you've read the 50,000-page tome, the townspeople have forgotten their problems. You will write a second edition of "Wisdom of the Ages", but have limited commercial success.
- Save the file.
Test your Action in the simulator
At this point, your Action should provide a customized fortune for the user based on the option they select.
To test your Action, follow these steps:
- In the terminal, run the following command:
gactions deploy preview
You should receive output that looks like the following:
✔ Done. You can now test your changes in Simulator with this URL: http://console.actions.google.com/project/{project-id}/simulator?disableAutoPreview
- Copy the provided URL and paste it into a browser.
- Type
Talk to my test app
in the Input field and press Enter. - Type Yes in the Input field and press Enter. Alternatively, click the Yes suggestion chip.
- Click, type, or say
Translator
.
You should receive the appropriate fortune for the "translator" option.
3. Add game loop
In this section, you configure your Action so that the user can select another option and hear a different fortune after making a selection. This change is similar to a "Do you want to play again?" message at the end of a game. To build this loop, you can re-use the previously created yes
and no
intents and add them to a new scene called Again
.
Create Again
scene
In this section, you create a new Again
scene and add a prompt that asks the user if they'd like to select a different option.
To create and configure the Again
scene, follow these steps:
- Create a new file called
Again.yaml
in thescenes
directory:
touch custom/scenes/Again.yaml
- Open
custom/scenes/Again.yaml
in your text editor. - Add the following
onEnter
data toAgain.yaml
:
Again.yaml
onEnter: staticPrompt: candidates: - promptResponse: firstSimple: variants: - speech: That is what I see for you. Would you like to choose a different option and explore another future? suggestions: - title: "Yes" - title: "No"
- Save the file.
Add transition from Fortune
to Again
scene
After the user receives their fortune, the conversation needs to transition to the new Again
scene.
To add a transition from the Fortune
scene to the Again
scene, follow these steps:
- Open
custom/scenes/Fortune.yaml
in your text editor. - Add
transitionToScene: Again
to each condition, as shown in the following snippet:
Fortune.yaml
conditionalEvents: - condition: scene.slots.status == "FINAL" && session.params.chosenOptions == "dragon" handler: staticPrompt: candidates: - promptResponse: firstSimple: variants: - speech: The people of Gryffinberg will be awestruck by the beauty and power of the ancient dragon. Much to your dismay, the townspeople fall into dispute over who will receive the honor of riding the dragon first. You return home from your quest without everlasting glory or a dragon. transitionToScene: Again - condition: scene.slots.status == "FINAL" && session.params.chosenOptions == "translator" handler: staticPrompt: candidates: - promptResponse: firstSimple: variants: - speech: With the help of the translator, the rival factions in Gryffinberg are finally able to communicate with each other and resolve their disputes. You will complete your quest to restore peace in the town. The translator will be used on many other journeys across the earth. After its work is done, it retires honorably to a premier location in the Gryffinberg History Museum. transitionToScene: Again - condition: scene.slots.status == "FINAL" && session.params.chosenOptions == "compass" handler: staticPrompt: candidates: - promptResponse: firstSimple: variants: - speech: The compass will help you find the mystical and ancient Library of Gryffinberg. Among its infinite stacks of dusty books, you find one entitled "Wisdom of the Ages". By the time you've read the 50,000-page tome, the townspeople have forgotten their problems. You will write a second edition of "Wisdom of the Ages", but have limited commercial success. transitionToScene: Again
- Save the file.
Test your Action in the simulator
At this point, your Action should provide the following prompt to the user after they receive their fortune: "That is what I see for you. Would you like to choose a different option and explore another future?"
To test your Action, follow these steps:
- In the terminal, run the following command:
gactions deploy preview
You should receive output that looks like the following:
✔ Done. You can now test your changes in Simulator with this URL: http://console.actions.google.com/project/{project-id}/simulator?disableAutoPreview
- Copy the provided URL and paste it into a browser.
- Type
Talk to my test app
in the Input field and press Enter. - Type
Yes
in the Input field and press Enter. Alternatively, click the Yes suggestion chip. - Click, type, or say
dragon
.
You should receive the fortune for the dragon option and the Again
prompt.
Add intents and transition to Again
scene
In this section, you add yes
and no
intents to the Again
scene so your Action understands if the user wants to choose a new option or not. You also add the appropriate transitions for the yes
and no
intents. The yes
intent transitions to the Fortune
scene, while the no
intent transitions to the system scene End conversation
.
To add intents and transitions to the Again
scene, follow these steps:
- Open
custom/scenes/Again.yaml
in your text editor. - Add
intentEvents
data at the top of theAgain.yaml
file, aboveOnEnter
:
Again.yaml
intentEvents: - intent: "yes" transitionToScene: Fortune - handler: staticPrompt: candidates: - promptResponse: firstSimple: variants: - speech: It pleases me that you are satisfied with your choice. Best of luck on your quest. Farewell. intent: "no" transitionToScene: actions.scene.END_CONVERSATION
- Save the file.
Test your Action in the simulator
Your Action should now understand if the user wants to choose a new option or end the conversation.
To test the yes
intent, follow these steps:
- In the terminal, run the following command:
gactions deploy preview
You should receive output that looks like the following:
✔ Done. You can now test your changes in Simulator with this URL: http://console.actions.google.com/project/{project-id}/simulator?disableAutoPreview
- Copy the provided URL and paste it into a browser.
- Type
Talk to my test app
in the Input field and press Enter. - Type
Yes
in the Input field and press Enter. Alternatively, click the Yes suggestion chip. - Click, type, or say one of the options.
- Type
Yes
in the Input field and press Enter.
You should receive the prompt, "What do you choose to help you on your quest, a dragon, a translator, or a compass?"
To test the no
intent, follow these steps:
- Click, type, or say one of the options.
- Type
No
in the input field and press Enter.
You should receive the End conversation
prompt: "It pleases me that you are satisfied with your choice. Best of luck on your quest. Farewell."
4. Add a supportive path
You've now built the main path that most users take in your Action. However, the user could respond to the prompt from the Fortune
scene, "What do you choose to help you on your quest, a dragon, a translator, or a compass?", with a choice that is not one of the provided options.
In this section, you configure your Action to understand when a user says "magic", "money", "horse", or "phone", and to re-prompt the user to select from one of the original three choices when they choose one of these options. To configure this logic, you need to create a new type
that contains these other choices and a new intent, other_option
, that is matched when a user says one of these options. You also need to annotate training phrases within the other_option
intent to identify and extract intent parameters.
When the Assistant's Natural Language Processing engine detects a parameter match in user input, it extracts the value as a typed parameter so you can carry out logic with it in a scene. In this codelab, you configure your Action to extract the aid the user chooses and refer to that choice in a prompt.
Create unavailable_options
type
You can now create an unavailable_options
type that contains a variety of different options so your Action can identify that data in a user's input.
To create the unavailable_options
type, follow these steps:
- Create a new file called
unavailable_options.yaml
in thetypes
directory:
touch custom/types/unavailable_options.yaml
- Open
custom/types/unavailable_options.yaml
in your text editor. - Add the following
synonyms
data to theunavailable_options.yaml
file:
unavailable_options.yaml
synonym: entities: money: synonyms: - money - cash - gold horse: synonyms: - horse - stallion - steed magic: synonyms: - magic - enchanted - spells phone: synonyms: - phone - cell - apps matchType: EXACT_MATCH
- Save the file.
Create other_option
intent
Next, you create an intent called other_option
and add training phrases that include the options in the unavailable_options
type. This intent is matched when the user selects a choice contained within the unavailable_options
type.
To create and configure the other_option
intent, follow these steps:
- Create a new file called
other_option.yaml
in theintents
directory:
touch custom/intents/other_option.yaml
- Open
custom/intents/other_option.yaml
in your text editor. - Add the following
parameters
data andtrainingPhrases
data to theother_option.yaml
file:
other_option.yaml
parameters: - name: chosenUnavailableOption type: name: unavailable_options trainingPhrases: - I want to use ($chosenUnavailableOption 'spells' auto=true) - I really really want to use a ($chosenUnavailableOption 'phone' auto=true) - ($chosenUnavailableOption 'magic' auto=true)! - ($chosenUnavailableOption 'cash' auto=true) - I want to ride a ($chosenUnavailableOption 'horse' auto=true)
Here, you manually annotate the training phrases with the unavailable options you specified in the previous section. The intent parameter, chosenUnavailableOption
, allows you to extract the name of the option and use that option in a prompt, which you'll do in the following section.
- Save the file.
Add other_option
intent to Fortune
scene
You now have an intent, other_option
, that can handle a user specifying an option that isn't one of the original options. In this section, you add the other_option
intent to the Fortune
scene. You use the intent parameter to customize the prompt based on the user's input.
To add the other_option
intent to the Fortune
scene, follow these steps:
- Open
custom/scenes/Fortune.yaml
in your text editor. - Add the following
intentEvents
data between theconditionalEvents
data andslots
data:
Fortune.yaml
intentEvents: - handler: staticPrompt: candidates: - promptResponse: firstSimple: variants: - speech: I have seen the future and a $intent.params.chosenUnavailableOption.original will not aid you on your journey. intent: other_option
- Save the file.
The expression $intent.params.chosenUnavailableOption
refers to the intent parameter object, and $intent.params.chosenUnavailableOption.original
refers to that object's value. The original
property refers to the raw input the user specifies.
When a user says an option listed in the unavailable_options
type during the Fortune
scene, the other_option
intent is matched and adds a prompt to the prompt queue. Since there is no transition specified, the scene execution loop continues by re-evaluating the conditions stage. The chosenOptions
slot then adds its prompt to the prompt queue, and the prompt queue is delivered to the user.
Test your Action in the simulator
Your Action should now respond appropriately when a user selects one of the options listed in the unavailable_options
type and specify which aid the user selected. Your Action should then re-prompt the user to pick one of the original choices (a dragon, translator, or compass).
To test your Action in the simulator, follow these steps:
- In the terminal, run the following command:
gactions deploy preview
You should receive output that looks like the following:
✔ Done. You can now test your changes in Simulator with this URL: http://console.actions.google.com/project/{project-id}/simulator?disableAutoPreview
- Copy the provided URL and paste it into a browser.
- Type
Talk to my test app
in the Input field and press Enter. - Type
Yes
in the Input field and press Enter. Alternatively, click the Yes suggestion chip. - Type
magic
in the Input field and press Enter.
You may notice that the prompt doesn't sound correct when the user chooses "magic" because of the "a" article placed before it. You address this issue in the following sections.
Add unavailable_options
handler
To place the "a" article before the appropriate choices from the unavailable_options
type, you can configure an event handler in your fulfillment logic to check if the option the user chooses needs an "a" before it. First, you need to configure your Action to call the handler from the Fortune
scene.
To add the unavailable_options
handler to the Fortune
scene, follow these steps:
- Open
custom/scenes/Fortune.yaml
in your text editor. - Update the
Fortune.yaml
file with the followingintentEvents
data:
Fortune.yaml
intentEvents: - handler: webhookHandler: unavailable_options intent: other_option
- Save the file.
Update and deploy fulfillment
Now that you've configured your Action to call the unavailable_options
event handler, you can update the handler in your fulfillment and deploy it.
To update your fulfillment, follow these steps:
- Open
webhooks/ActionsOnGoogleFulfillment/index.js
in your text editor. - Add the following code to
index.js
under thegreeting
handler:
index.js
app.handle('unavailable_options', conv => {
const option = conv.intent.params.chosenUnavailableOption.original;
const optionKey = conv.intent.params.chosenUnavailableOption.resolved;
let message = 'I have seen the future and ';
if(optionsNeedA.has(optionKey)){
message = message + 'a ';
}
message = message + `${option} will not aid you on your journey. `;
conv.add(message);
});
- Add the following code under
const app = conversation({debug:true});
:
index.js
const optionsNeedA = new Set();
optionsNeedA.add('horse').add('phone');
- Save the file.
Understand the code
The unavailable_options
handler does the following:
- Gets
option
data from theconv
object and assignsoption
to theoriginal
property, which is the raw input from the user - Assigns
optionKey
to theresolved
property, which is the key for theunavailable_options
type - Checks if
optionKey
is one of the options that needs an "a"; if it is, constructs the message with an added "a" - Adds the message via
conv.add(message)
Update handlers
To allow the Action to use unavailable_options
, add the unavailable_options
handler to webhooks/ActionsOnGoogleFulfillment.yaml
.
- Add the
unavailable_options
handler name toActionsOnGoogleFulfillment.yaml
:
ActionsOnGoogleFulfillment.yaml
handlers: - name: greeting - name: unavailable_options inlineCloudFunction: executeFunction: ActionsOnGoogleFulfillment
- Save the file.
Test your Action in the simulator
Your Action should now adjust the prompt based on whether the user's choice from the unavailable_options
type requires an "a" article before it.
To test your Action, follow these steps:
- In the terminal, run the following command:
gactions deploy preview
You should receive output that looks like the following:
✔ Done. You can now test your changes in Simulator with this URL: http://console.actions.google.com/project/{project-id}/simulator?disableAutoPreview
- Copy the provided URL and paste it into a browser.
- Click or type
Talk to my test app
in the Input field and press Enter. - Type
Yes
in the Input field and press Enter. Alternatively, click the Yes suggestion chip. - Type
magic
in the Input field and press Enter. Then, typehorse
in the Input field and press Enter.
Your Action should add the "a" article before the "horse" choice, while constructing the prompt without the "a" article for the "magic" choice.
5. Visualize Action in Actions console
The Actions SDK has interoperability with a web-based IDE called Actions Builder that is integrated into the Actions console. You can push your local file system to the draft of your Action in the console with the gactions push
command. The Actions console provides a visual representation of your Action's configuration. Seeing your Action mapped out visually can be useful during development, and doesn't affect the version of your Action that's served for testing.
To push your Actions project and view it in the Actions console, follow these steps:
- In the terminal, run the following command to push your project to the Actions console:
gactions push
You should receive output that looks like the following:
✔ Done. Files were pushed to Actions Console, and you can now view your project with this URL: https://console.actions.google.com/project/{project-id}/overview. If you want to test your changes, run "gactions deploy preview", or navigate to the Test section in the Console.
- Copy the provided URL and paste it into a browser.
- In the Actions console, click Develop in the top navigation bar.
- Click the drop-down arrow next to Scenes and click Start. You should see a visual representation of your Action's
Start
scene, as shown in the following screenshot:
Clean up your project [recommended]
To avoid incurring possible charges, It is recommended to remove projects that you don't intend to use. To delete the projects you created in this codelab, follow these steps:
- To delete the Cloud Project and resources, complete the steps listed in the Shutting down (deleting) projects section.
- Optional: To immediately remove your project from the Actions console, complete the steps listed in the Delete a project section. If you don't complete this step, your project will automatically be removed after approximately 30 days.
6. Congratulations!
You now know the intermediate skills necessary to build Actions for Google Assistant with the Actions SDK.
What you covered
- How to develop Conversational Actions using the Node.js fulfillment library
- How to use slots to gather data from the user
- How to use conditions to add logic to the scene
- How to add a game loop
- How to add a supportive path
Learn more
You can explore these resources for learning about building Actions for Google Assistant:
- Documentation for developing Actions for Google Assistant
- Actions on Google GitHub page for sample code and libraries
- The official Reddit community for developers working with the Google Assistant
- Conversation design guidelines for best practices and guidelines regarding Conversational Actions
Follow us on Twitter @ActionsOnGoogle to stay tuned to our latest announcements, and tweet to #AoGDevs to share what you have built!
Feedback survey
Before you go, please fill out a brief survey about your experience.