Enhance and secure your smart home Actions

1. Before you begin

Smart home Actions use device types to let the Google Assistant know what grammar should be used with a device. Device traits define the capabilities of a device type. A device inherits the states of each device trait added to an Action.

dc8dce0dea87cd5c.png

You can connect any supported traits to your chosen device type to customize the functionality of your users' devices. If you want to implement custom traits in your Actions that aren't currently available in the device schema, then the Modes and Toggles traits allow specific settings' control with a custom name that you define.

Beyond the basic control capability provided by types and traits, the Smart Home API has additional features to enhance the user experience. Error responses provide detailed user feedback when intents don't succeed. Secondary user verification extends those responses and adds additional security to the device trait of your choice. By sending specific error responses to challenge blocks issued from the Assistant, your smart home Action can require additional authorization to complete a command.

Prerequisites

What you'll build

In this codelab, you'll deploy a prebuilt smart home integration with Firebase, then learn how to add nonstandard traits to the smart home washer for load size and turbo mode. You'll also implement error and exception reporting, and learn to enforce a verbal acknowledgement to turn on the washer using secondary user verification.

What you'll learn

  • How to add the Modes and Toggles traits to your Action
  • How to report errors and exceptions
  • How to apply secondary user verification

What you'll need

2. Getting started

Enable Activity controls

In order to use the Google Assistant, you must share certain activity data with Google. The Google Assistant needs this data to function properly; however, the requirement to share data is not specific to the SDK. To share this data, create a Google account if you don't already have one. You can use any Google account—it does not need to be your developer account.

Open the Activity Controls page for the Google account that you want to use with the Assistant.

Ensure the following toggle switches are enabled:

  • Web & App Activity - In addition, be sure to select the Include Chrome history and activity from sites, apps, and devices that use Google services checkbox.
  • Device Information
  • Voice & Audio Activity

Create an Actions project

  1. Go to the Actions on Google Developer Console.
  2. Click New Project, enter a name for the project, and click CREATE PROJECT.

3d6b68ca79afd54c.png

Select the Smart Home App

On the Overview screen in the Actions console, select Smart home.

2fa4988f44f8914b.png

Choose the Smart home experience card, click Start Building, and you will then be directed to your project console.

Install the Firebase CLI

The Firebase Command Line Interface (CLI) will allow you to serve your web apps locally and deploy your web app to Firebase hosting.

To install the CLI, run the following npm command from the terminal:

npm install -g firebase-tools

To verify that the CLI has been installed correctly, run:

firebase --version

Authorize the Firebase CLI with your Google account by running:

firebase login

Enable the HomeGraph API

The HomeGraph API enables the storage and querying of devices and their states within a user's Home Graph. To use this API, you must first open the Google Cloud console and enable the HomeGraph API.

In the Google Cloud console, make sure to select the project that matches your Actions <project-id>. Then, in the API Library screen for the HomeGraph API, click Enable.

ee198858a6eac112.png

3. Run the starter app

Now that you set up your development environment, you can deploy the starter project to verify everything is configured properly.

Get the source code

Click the following link to download the sample for this codelab on your development machine:

...or you can clone the GitHub repository from the command line:

git clone https://github.com/google-home/smarthome-traits.git

Unpack the downloaded zip file.

About the project

The starter project contains the following subdirectories:

  • public: A frontend UI to easily control and monitor the state of the smart washer.
  • functions: A fully implemented cloud service that manages the smart washer with Cloud Functions for Firebase and Firebase Realtime Database.

The provided cloud fulfillment includes the following functions in index.js:

  • fakeauth: Authorization endpoint for account linking
  • faketoken: Token endpoint for account linking
  • smarthome: Smart home intent fulfillment endpoint
  • reportstate: Invokes the Home Graph API on device state changes
  • requestsync: Enables user device updates without requiring account relinking

Connect to Firebase

Navigate to the washer-start directory, then set up the Firebase CLI with your Actions project:

cd washer-start
firebase use <project-id>

Configure Firebase project

Initialize a Firebase project.

firebase init

Select the CLI features, Realtime Database, Functions, and the Hosting feature that includes Firebase Hosting.

? Which Firebase CLI features do you want to set up for this directory? Press Space to select features, then
 Enter to confirm your choices.
❯◉ Realtime Database: Configure a security rules file for Realtime Database and (optionally) provision default instance
 ◯ Firestore: Configure security rules and indexes files for Firestore
 ◉ Functions: Configure a Cloud Functions directory and its files
 ◉ Hosting: Configure files for Firebase Hosting and (optionally) set up GitHub Action deploys
 ◯ Hosting: Set up GitHub Action deploys
 ◯ Storage: Configure a security rules file for Cloud Storage
 ◯ Emulators: Set up local emulators for Firebase products
 ◯ Remote Config: Configure a template file for Remote Config
 ◯ Extensions: Set up an empty Extensions manifest

This will initialize the necessary APIs and features for your project.

When prompted, initialize Realtime Database. You can use the default location for the database instance.

? It seems like you haven't initialized Realtime Database in your project yet. Do you want to set it up?
Yes

? Please choose the location for your default Realtime Database instance:
us-central1

Since you are using the starter project code, choose the default file for the Security rules, and ensure you don't overwrite the existing database rules file.

? File database.rules.json already exists. Do you want to overwrite it with the Realtime Database Security Rules for <project-ID>-default-rtdb from the Firebase Console?
No

If you are reinitializing your project, select Overwrite when asked if you want to initialize or overwrite a codebase.

? Would you like to initialize a new codebase, or overwrite an existing one?
Overwrite

When configuring your Functions, you should use the default files, and ensure you don't overwrite the existing index.js and package.json files in the project sample.

? What language would you like to use to write Cloud Functions?
JavaScript

? Do you want to use ESLint to catch probable bugs and enforce style?
No

? File functions/package.json already exists. Overwrite?
No

? File functions/index.js already exists. Overwrite?
No

If you are reinitializing your project, select No when asked if you want to initialize or overwrite functions/.gitignore.

? File functions/.gitignore already exists. Overwrite?
No
? Do you want to install dependencies with npm now?
Yes

Finally, configure your Hosting setup to use the public directory in the project code, and use the existing index.html file. Select No when asked to use ESLint.

? What do you want to use as your public directory?
public

? Configure as a single-page app (rewrite all urls to /index.html)?
Yes

? Set up automatic builds and deploys with GitHub?
No

? File public/index.html already exists. Overwrite?
 No

If ESLint was accidentally enabled, there are two methods available to disable it:

  1. Using the GUI, go to the ../functions folder under the project, select the hidden file .eslintrc.js and delete it. Do not mistake it for the similarly named .eslintrc.json.
  2. Using the command line:
    cd functions
    rm .eslintrc.js
    

Deploy to Firebase

Now that you have installed the dependencies and configured your project, you are ready to run the app for the first time.

firebase deploy

This is the console output you should see:

...

✔ Deploy complete!

Project Console: https://console.firebase.google.com/project/<project-id>/overview
Hosting URL: https://<project-id>.web.app

This command deploys a web app, along with several Cloud Functions for Firebase.

Open the Hosting URL in your browser (https://<project-id>.web.app) to view the web app. You will see the following interface:

5845443e94705557.png

This web UI represents a third-party platform to view or modify device states. To begin populating your database with device information, click UPDATE. You won't see any changes on the page, but the current state of your washer will be stored in the database.

Now it's time to connect the cloud service you've deployed to the Google Assistant using the Actions console.

Configure your Actions console project

Under Overview > Build your Action, select Add Action(s). Enter the URL for your cloud function that provides fulfillment for the smart home intents and click Save.

https://us-central1-<project-id>.cloudfunctions.net/smarthome

9d7b223427f587ca.png

On the Develop > Invocation tab, add a Display Name for your Action, and click Save. This name will appear in the Google Home app.

774d0c40c351c7da.png

a8c4673eb11d76ee.png

To enable Account linking, select the Develop > Account linking option in the left navigation. Use these account linking settings:

Client ID

ABC123

Client secret

DEF456

Authorization URL

https://us-central1-<project-id>.cloudfunctions.net/fakeauth

Token URL

https://us-central1-<project-id>.cloudfunctions.net/faketoken

9730d20b90bcc038.png

Click Save to save your account linking configuration, then click Test to enable testing on your project.

ee0547f05b5efd98.png

You will be redirected to the Simulator. If you do not see "Test now enabled", click Reset Test to verify testing is enabled.

d0495810dbadf059.png

In order to test your smart home Action, you need to link your project with a Google account. This enables testing through Google Assistant surfaces and the Google Home app that are signed in to the same account.

  1. On your phone, open the Google Assistant settings. Note that you should be logged in as the same account as in the console.
  2. Navigate to Google Assistant > Settings > Home Control (under Assistant).
  3. Click the search icon in the upper right.
  4. Search for your test app using the [test] prefix to find your specific test app.
  5. Select that item. The Google Assistant will then authenticate with your service and send a SYNC request, asking your service to provide a list of devices for the user.

Open the Google Home app and verify that you can see your washer device.

ae252220753726f6.png

Verify that you can control the washer by using voice commands in the Google Home app. You should also see the device state change in the frontend web UI of your cloud fulfillment.

Now that you have a basic washer deployed, you can customize the modes available on your device.

4. Add modes

The action.devices.traits.Modes trait enables a device to have an arbitrary number of settings for a mode, of which only one can be set at a time. You'll add a mode to the washer to define the size of the laundry load: small, medium, or large.

Update SYNC response

You need to add information about the new trait to your SYNC response in functions/index.js. This data appears in the traits array and attributes object as shown in the following code snippet.

index.js

app.onSync(body => {
  return {
    requestId: 'ff36a3cc-ec34-11e6-b1a0-64510650abcf',
    payload: {
      agentUserId: USER_ID,
      devices: [{
        id: 'washer',
        type: 'action.devices.types.WASHER',
        traits: [
          'action.devices.traits.OnOff',
          'action.devices.traits.StartStop',
          'action.devices.traits.RunCycle',
          // Add Modes trait
          'action.devices.traits.Modes',
        ],
        name: { ... },
        deviceInfo: { ... },
        attributes: {
          pausable: true,
          //Add availableModes
          availableModes: [{
            name: 'load',
            name_values: [{
              name_synonym: ['load'],
              lang: 'en',
            }],
            settings: [{
              setting_name: 'small',
              setting_values: [{
                setting_synonym: ['small'],
                lang: 'en',
              }]
            }, {
              setting_name: 'medium',
              setting_values: [{
                setting_synonym: ['medium'],
                lang: 'en',
              }]
            }, {
              setting_name: 'large',
              setting_values: [{
                setting_synonym: ['large'],
                lang: 'en',
              }]
            }],
            ordered: true,
          }],
        },
      }],
    },
  };
});

Add new EXECUTE intent commands

In your EXECUTE intent, add the action.devices.commands.SetModes command as shown in the following code snippet.

index.js

const updateDevice = async (execution,deviceId) => {
  const {params,command} = execution;
  let state, ref;
  switch (command) {
    case 'action.devices.commands.OnOff':
      state = {on: params.on};
      ref = firebaseRef.child(deviceId).child('OnOff');
      break;
    case 'action.devices.commands.StartStop':
      state = {isRunning: params.start};
      ref = firebaseRef.child(deviceId).child('StartStop');
      break;
    case 'action.devices.commands.PauseUnpause':
      state = {isPaused: params.pause};
      ref = firebaseRef.child(deviceId).child('StartStop');
      Break;
    // Add SetModes command
    case 'action.devices.commands.SetModes':
      state = {load: params.updateModeSettings.load};
      ref = firebaseRef.child(deviceId).child('Modes');
      break;
}

Update QUERY response

Next, update your QUERY response to report the washer's current state.

Add the updated changes to the queryFirebase and queryDevice functions to obtain the state as stored in the Realtime Database.

index.js

const queryFirebase = async (deviceId) => {
  const snapshot = await firebaseRef.child(deviceId).once('value');
  const snapshotVal = snapshot.val();
  return {
    on: snapshotVal.OnOff.on,
    isPaused: snapshotVal.StartStop.isPaused,
    isRunning: snapshotVal.StartStop.isRunning,
    // Add Modes snapshot
    load: snapshotVal.Modes.load,
  };
}

const queryDevice = async (deviceId) => {
  const data = await queryFirebase(deviceId);
  return {
    on: data.on,
    isPaused: data.isPaused,
    isRunning: data.isRunning,
    currentRunCycle: [{ ... }],
    currentTotalRemainingTime: 1212,
    currentCycleRemainingTime: 301,
    // Add currentModeSettings
    currentModeSettings: {
      load: data.load,
    },
  };
};

Update Report State

Finally, update your reportstate function to report the washer's current load setting to Home Graph.

index.js

const requestBody = {
  requestId: 'ff36a3cc', /* Any unique ID */
  agentUserId: USER_ID,
  payload: {
    devices: {
      states: {
        /* Report the current state of your washer */
        [context.params.deviceId]: {
          on: snapshot.OnOff.on,
          isPaused: snapshot.StartStop.isPaused,
          isRunning: snapshot.StartStop.isRunning,
          // Add currentModeSettings
          currentModeSettings: {
            load: snapshot.Modes.load,
          },
        },
      },
    },
  },
};

Deploy to Firebase

Run the following command to deploy the updated action:

firebase deploy --only functions

After deployment completes, navigate to the web UI and click the Refresh ae8d3b25777a5e30.png button in the toolbar. That triggers a Request Sync so that the Assistant receives the updated SYNC response data.

bf4f6a866160a982.png

Now you can give a command to set the mode of the washer, such as:

"Hey Google, set the washer load to large."

Additionally, you can ask questions about your washer, such as:

"Hey Google, what is the washer load?"

5. Add toggles

The action.devices.traits.Toggles trait represents named aspects of a device that have a true or false state, such as whether the washer is in turbo mode.

Update SYNC response

In your SYNC response, you need to add information about the new device trait. It will appear in the traits array and attributes object as shown in the following code snippet.

index.js

app.onSync(body => {
  return {
    requestId: 'ff36a3cc-ec34-11e6-b1a0-64510650abcf',
    payload: {
      agentUserId: USER_ID,
      devices: [{
        id: 'washer',
        type: 'action.devices.types.WASHER',
        traits: [
          'action.devices.traits.OnOff',
          'action.devices.traits.StartStop',
          'action.devices.traits.RunCycle',
          'action.devices.traits.Modes',
          // Add Toggles trait
          'action.devices.traits.Toggles',
        ],
        name: { ... },
        deviceInfo: { ... },
        attributes: {
          pausable: true,
          availableModes: [{
            name: 'load',
            name_values: [{
              name_synonym: ['load'],
              lang: 'en'
            }],
            settings: [{ ... }],
            ordered: true,
          }],
          //Add availableToggles
          availableToggles: [{
            name: 'Turbo',
            name_values: [{
              name_synonym: ['turbo'],
              lang: 'en',
            }],
          }],
        },
      }],
    },
  };
});

Add new EXECUTE intent commands

In your EXECUTE intent, add the action.devices.commands.SetToggles command as shown in the following code snippet.

index.js

const updateDevice = async (execution,deviceId) => {
  const {params,command} = execution;
  let state, ref;
  switch (command) {
    case 'action.devices.commands.OnOff':
      state = {on: params.on};
      ref = firebaseRef.child(deviceId).child('OnOff');
      break;
    case 'action.devices.commands.StartStop':
      state = {isRunning: params.start};
      ref = firebaseRef.child(deviceId).child('StartStop');
      break;
    case 'action.devices.commands.PauseUnpause':
      state = {isPaused: params.pause};
      ref = firebaseRef.child(deviceId).child('StartStop');
      break;
    case 'action.devices.commands.SetModes':
      state = {load: params.updateModeSettings.load};
      ref = firebaseRef.child(deviceId).child('Modes');
      break;
    // Add SetToggles command
    case 'action.devices.commands.SetToggles':
      state = {Turbo: params.updateToggleSettings.Turbo};
      ref = firebaseRef.child(deviceId).child('Toggles');
      break;
  }

Update QUERY response

Finally, you need to update your QUERY response to report the washer's turbo mode. Add the updated changes to the queryFirebase and queryDevice functions to obtain the toggle state as stored in the Realtime Database.

index.js

const queryFirebase = async (deviceId) => {
  const snapshot = await firebaseRef.child(deviceId).once('value');
  const snapshotVal = snapshot.val();
  return {
    on: snapshotVal.OnOff.on,
    isPaused: snapshotVal.StartStop.isPaused,
    isRunning: snapshotVal.StartStop.isRunning,
    load: snapshotVal.Modes.load,
    // Add Toggles snapshot
    Turbo: snapshotVal.Toggles.Turbo,
  };
}

const queryDevice = async (deviceId) => {
  const data = queryFirebase(deviceId);
  return {
    on: data.on,
    isPaused: data.isPaused,
    isRunning: data.isRunning,
    currentRunCycle: [{ ... }],
    currentTotalRemainingTime: 1212,
    currentCycleRemainingTime: 301,
    currentModeSettings: {
      load: data.load,
    },
    // Add currentToggleSettings
    currentToggleSettings: {
      Turbo: data.Turbo,
    },
  };
};

Update Report State

Finally, update your reportstate function to report to Home Graph whether the washer is set to turbo.

index.js

const requestBody = {
  requestId: 'ff36a3cc', /* Any unique ID */
  agentUserId: USER_ID,
  payload: {
    devices: {
      states: {
        /* Report the current state of your washer */
        [context.params.deviceId]: {
          on: snapshot.OnOff.on,
          isPaused: snapshot.StartStop.isPaused,
          isRunning: snapshot.StartStop.isRunning,
          currentModeSettings: {
            load: snapshot.Modes.load,
          },
          // Add currentToggleSettings
          currentToggleSettings: {
            Turbo: snapshot.Toggles.Turbo,
          },
        },
      },
    },
  },
};

Deploy to Firebase

Run the following command to deploy the updated functions:

firebase deploy --only functions

Click the Refresh ae8d3b25777a5e30.png button in the web UI to trigger a Request Sync after deployment completes.

You can now give a command to set the washer to turbo mode by saying:

"Hey Google, turn on turbo for the washer."

You can also check if your washer is already in turbo mode by asking:

"Hey Google, is my washer in turbo mode?"

6. Reporting errors and exceptions

Error handling in your smart home Action enables you to report to users when issues cause EXECUTE and QUERY responses to fail. The notifications create a more positive user experience for your users when they interact with your smart device and Action.

Any time that an EXECUTE or QUERY request fails, your Action should return an error code. If, for example, you wanted to throw an error when a user attempts to start the washer with the lid open, then your EXECUTE response would look like the following code snippet:

{
  "requestId": "ff36a3cc-ec34-11e6-b1a0-64510650abcf",
  "payload": {
    "commands": [
      {
        "ids": [
          "456"
        ],
        "status": "ERROR",
        "errorCode": "deviceLidOpen"
      }
    ]
  }
}

Now, when a user asks to start the washer, the Assistant responds by saying:

"The lid is open on the washer. Please close it and try again."

Exceptions are similar to errors, but indicate when an alert is associated with a command, which may or may not block successful execution. An exception can provide related information using the StatusReport trait, such as battery level or recent state change. Non-blocking exception codes are returned along with a SUCCESS status, while blocking exception codes are returned with an EXCEPTIONS status.

An example response with an exception is in the following code snippet:

{
  "requestId": "ff36a3cc-ec34-11e6-b1a0-64510650abcf",
  "payload": {
    "commands": [{
      "ids": ["123"],
      "status": "SUCCESS",
      "states": {
        "online": true,
        "isPaused": false,
        "isRunning": false,
        "exceptionCode": "runCycleFinished"
      }
    }]
  }
}

The Assistant responds by saying:

"The washer has finished running."

To add error reporting for your washer, open functions/index.js and add the error class definition as seen in the following code snippet:

index.js

app.onQuery(async (body) => {...});

// Add SmartHome error handling
class SmartHomeError extends Error {
  constructor(errorCode, message) {
    super(message);
    this.name = this.constructor.name;
    this.errorCode = errorCode;
  }
}

Update the execute response to return the error code and the error status:

index.js

const executePromises = [];
const intent = body.inputs[0];
for (const command of intent.payload.commands) {
  for (const device of command.devices) {
    for (const execution of command.execution) {
      executePromises.push( ... )
          //Add error response handling
          .catch((error) => {
            functions.logger.error('EXECUTE', device.id, error);
            result.ids.push(device.id);
            if(error instanceof SmartHomeError) {
              result.status = 'ERROR';
              result.errorCode = error.errorCode;
            }
          })
      );
    }
  }
}

The Assistant can now tell your users about any error code that you report. You'll see a specific example in the next section.

7. Add secondary user verification

You should implement secondary user verification in your Action if your device has any modes that need to be secured or should be limited to a particular group of authorized users, such as a software update or lock disengage.

You can implement secondary user verification on all device types and traits, customizing whether the security challenge occurs every time or specific criteria needs to be met.

There are three supported challenge types:

  • No challenge—A request and response that doesn't use an authentication challenge (this is the default behavior)
  • ackNeeded—A secondary user verification that requires explicit acknowledgement (yes or no)
  • pinNeeded—A secondary user verification that requires a personal identification number (PIN)

For this codelab, add an ackNeeded challenge to the command for turning on the washer and the functionality to return an error if the secondary verification challenge fails.

Open functions/index.js, and add an error class definition that returns the error code and challenge type as seen in the following code snippet:

index.js

class SmartHomeError extends Error { ... }

// Add secondary user verification error handling
class ChallengeNeededError extends SmartHomeError {
  /**
   * Create a new ChallengeNeededError
   * @param {string} suvType secondary user verification challenge type
   */
  constructor(suvType) {
    super('challengeNeeded', suvType);
    this.suvType = suvType;
  }
}

You also need to update the execution response to return the challengeNeeded error as follows:

index.js

const executePromises = [];
const intent = body.inputs[0];
for (const command of intent.payload.commands) {
  for (const device of command.devices) {
    for (const execution of command.execution) {
      executePromises.push( ... )
          .catch((error) => {
            functions.logger.error('EXECUTE', device.id, error);
            result.ids.push(device.id);
            if(error instanceof SmartHomeError) {
              result.status = 'ERROR';
              result.errorCode = error.errorCode;
              //Add error response handling
              if(error instanceof ChallengeNeededError) {
                result.challengeNeeded = {
                  type: error.suvType
                };
              }
            }
          })
      );
    }
  }
}

Finally, modify updateDevice to require the explicit acknowledgment to turn the washer on or off.

index.js

const updateDevice = async (execution,deviceId) => {
  const {challenge,params,command} = execution; //Add secondary user challenge
  let state, ref;
  switch (command) {
    case 'action.devices.commands.OnOff':
      //Add secondary user verification challenge
      if (!challenge || !challenge.ack) {
        throw new ChallengeNeededError('ackNeeded');
      }
      state = {on: params.on};
      ref = firebaseRef.child(deviceId).child('OnOff');
      break;
    ...
  }

  return ref.update(state)
      .then(() => state);
};

Deploy to Firebase

Run the following command to deploy the updated function:

firebase deploy --only functions

After deploying the updated code, you must verbally acknowledge the action when you ask the Assistant to turn your washer on or off, like this:

You: "Hey Google, turn on the washer."

The Assistant: "Are you sure you want to turn on the washer?"

You: "Yes."

You can also see a detailed response for each step of the secondary user verification flow by opening your Firebase logs.

289dbe48f4bb8106.png

8. Congratulations

674c4f4392e98c1.png

Congratulations! You extended the features of smart home Actions through the Modes and Toggles traits, and secured their execution through secondary user verification.

Learn more

Here are some ideas you can implement to go deeper:

  • Add local execution capabilities to your devices.
  • Use a different secondary user verification challenge type to modify your device state.
  • Update the RunCycle trait QUERY response to update dynamically.
  • Explore this GitHub sample.