Connect smart home devices to the Google Assistant

1. Before you begin

As an Internet of Things (IoT) developer, you can build smart home Actions that give your users the ability to control their devices through touch controls in the Google Home app and voice commands with the Assistant.

79266e5f45e6ae20.gif

Smart home Actions rely on Home Graph to provide contextual data about the home and its devices, creating a logical map of the home. That context gives the Assistant a more natural understanding of the user's requests relative to his or her location in the home. For example, Home Graph can store the concept of a living room that contains multiple types of devices from different manufacturers, such as a thermostat, lamp, fan, and vacuum.

d009cef0f903d284.jpeg

Prerequisites

What you'll build

In this codelab, you'll publish a cloud service that manages a virtual smart washing machine, then build a smart home Action and connect it to the Assistant.

What you'll learn

  • How to deploy a smart home cloud service
  • How to connect your service to the Assistant
  • How to publish device state changes to Google

What you'll need

2. Getting started

Enable Activity controls

In order to use the Google Assistant, you must share certain activity data with Google. The Google Assistant needs this data to function properly; however, the requirement to share data is not specific to the SDK. To share this data, create a Google Account if you don't already have one. You can use any Google Account—it does not need to be your developer account.

Open the Activity Controls page for the Google Account that you want to use with the Assistant.

Ensure the following toggle switches are enabled:

  • Web & App Activity - In addition, be sure to select the Include Chrome history and activity from sites, apps, and devices that use Google services checkbox.
  • Device Information
  • Voice & Audio Activity

Create an Actions project

  1. Go to the Actions on Google Developer Console.
  2. Click New Project, enter a name for the project, and click CREATE PROJECT.

3d6b68ca79afd54c.png

Select the Smart Home App

On the Overview screen in the Actions console, select Smart home.

2fa4988f44f8914b.png

Choose the Smart home experience card, click Start Building, and you will then be directed to your project console.

Install the Firebase CLI

The Firebase Command Line Interface (CLI) will allow you to serve your web apps locally and deploy your web app to Firebase hosting.

To install the CLI, run the following npm command from the terminal:

npm install -g firebase-tools

To verify that the CLI has been installed correctly, run:

firebase --version

Authorize the Firebase CLI with your Google Account by running:

firebase login

3. Run the starter app

Now that you set up your development environment, you can deploy the starter project to verify everything is configured properly.

Get the source code

Click the following link to download the sample for this codelab on your development machine:

...or you can clone the GitHub repository from the command line:

git clone https://github.com/google-home/smarthome-washer.git

About the project

The starter project contains the following subdirectories:

  • public: A frontend UI to easily control and monitor the state of the smart washer.
  • functions: A fully implemented cloud service that manages the smart washer with Cloud Functions for Firebase and Firebase Realtime Database.

Connect to Firebase

Navigate to the washer-start directory, then set up the Firebase CLI with your Actions project:

cd washer-start
firebase use <project-id>

Configure Firebase project

Initialize a Firebase project.

firebase init

Select the CLI features, Realtime Database, Functions, and the Hosting feature that includes Firebase Hosting.

? Which Firebase CLI features do you want to set up for this directory? Press Space to select features, then
 Enter to confirm your choices.
❯◉ Realtime Database: Configure a security rules file for Realtime Database and (optionally) provision default instance
 ◯ Firestore: Configure security rules and indexes files for Firestore
 ◉ Functions: Configure a Cloud Functions directory and its files
 ◉ Hosting: Configure files for Firebase Hosting and (optionally) set up GitHub Action deploys
 ◯ Hosting: Set up GitHub Action deploys
 ◯ Storage: Configure a security rules file for Cloud Storage
 ◯ Emulators: Set up local emulators for Firebase products
 ◯ Remote Config: Configure a template file for Remote Config
 ◯ Extensions: Set up an empty Extensions manifest

This will initialize the necessary APIs and features for your project.

When prompted, initialize Realtime Database. You can use the default location for the database instance.

? It seems like you haven't initialized Realtime Database in your project yet. Do you want to set it up?
Yes

? Please choose the location for your default Realtime Database instance:
us-central1

Since you are using the starter project code, choose the default file for the Security rules, and ensure you don't overwrite the existing database rules file.

? File database.rules.json already exists. Do you want to overwrite it with the Realtime Database Security Rules for <project-ID>-default-rtdb from the Firebase Console?
No

If you are reinitializing your project, select Overwrite when asked if you want to initialize or overwrite a codebase.

? Would you like to initialize a new codebase, or overwrite an existing one?
Overwrite

When configuring your Functions, you should use the default files, and ensure you don't overwrite the existing index.js and package.json files in the project sample.

? What language would you like to use to write Cloud Functions?
JavaScript

? Do you want to use ESLint to catch probable bugs and enforce style?
No

? File functions/package.json already exists. Overwrite?
No

? File functions/index.js already exists. Overwrite?
No

If you are reinitializing your project, select No when asked if you want to initialize or overwrite functions/.gitignore.

? File functions/.gitignore already exists. Overwrite?
No
? Do you want to install dependencies with npm now?
Yes

Finally, configure your Hosting setup to use the public directory in the project code, and use the existing index.html file. Select No when asked to use ESLint.

? What do you want to use as your public directory?
public

? Configure as a single-page app (rewrite all urls to /index.html)?
Yes

? Set up automatic builds and deploys with GitHub?
No

? File public/index.html already exists. Overwrite?
 No

If ESLint was accidentally enabled, there are two methods available to disable it:

  1. Using the GUI, go to the ../functions folder under the project, select the hidden file .eslintrc.js and delete it. Do not mistake it for the similarly named .eslintrc.json.
  2. Using the command line:
    cd functions
    rm .eslintrc.js
    

To ensure that you have a correct and complete Firebase configuration, copy the firebase.json file from the app-done directory to the app-start directory, overwriting the one in app-start.

In the app-start directory:

cp -vp ../app-done/firebase.json .

Deploy to Firebase

Now that you have installed the dependencies and configured your project, you are ready to run the app for the first time.

firebase deploy

This is the console output you should see:

...

✔ Deploy complete!

Project Console: https://console.firebase.google.com/project/<project-id>/overview
Hosting URL: https://<project-id>.web.app

This command deploys a web app, along with several Cloud Functions for Firebase.

Open the Hosting URL in your browser (https://<project-id>.web.app) to view the web app. You will see the following interface:

5845443e94705557.png

This web UI represents a third-party platform to view or modify device states. To begin populating your database with device information, click UPDATE. You won't see any changes on the page, but the current state of your washer will be stored in the database.

Now it's time to connect the cloud service you've deployed to the Google Assistant using the Actions console.

Configure your Actions console project

Under Overview > Build your Action, select Add Action(s). Enter the URL for your cloud function that provides fulfillment for the smart home intents and click Save.

https://us-central1-<project-id>.cloudfunctions.net/smarthome

9d7b223427f587ca.png

On the Develop > Invocation tab, add a Display Name for your Action, and click Save. This name will appear in the Google Home app.

774d0c40c351c7da.png

a8c4673eb11d76ee.png

To enable Account linking, select the Develop > Account linking option in the left navigation. Use these account linking settings:

Client ID

ABC123

Client secret

DEF456

Authorization URL

https://us-central1-<project-id>.cloudfunctions.net/fakeauth

Token URL

https://us-central1-<project-id>.cloudfunctions.net/faketoken

9730d20b90bcc038.png

Click Save to save your account linking configuration, then click Test to enable testing on your project.

ee0547f05b5efd98.png

You will be redirected to the Simulator. If you don't see "Test now enabled", click Reset Test to verify testing is enabled.

d0495810dbadf059.png

Now you can begin implementing the webhooks necessary to connect the device state with the Assistant.

4. Create a washer

Now that you configured your Action, you can add devices and send data. Your cloud service needs to handle the following intents:

  • A SYNC intent occurs when the Assistant wants to know what devices the user has connected. This is sent to your service when the user links an account. You should respond with a JSON payload of all the user's devices and their capabilities.
  • A QUERY intent occurs when the Assistant wants to know the current state or status of a device. You should respond with a JSON payload with the state of each requested device.
  • An EXECUTE intent occurs when the Assistant wants to control a device on a user's behalf. You should respond with a JSON payload with the execution status of each requested device.
  • A DISCONNECT intent occurs when the user unlinks their account from the Assistant. You should stop sending events for this user's devices to the Assistant.

You will update the functions that you previously deployed to handle these intents in the following sections.

Update SYNC response

Open functions/index.js, which contains the code to respond to requests from the Assistant.

You will need to handle a SYNC intent by returning the device metadata and capabilities. Update the JSON in the onSync array to include the device information and recommended traits for a clothes washer.

index.js

app.onSync((body) => {
  return {
    requestId: body.requestId,
    payload: {
      agentUserId: USER_ID,
      devices: [{
        id: 'washer',
        type: 'action.devices.types.WASHER',
        traits: [
          'action.devices.traits.OnOff',
          'action.devices.traits.StartStop',
          'action.devices.traits.RunCycle',
        ],
        name: {
          defaultNames: ['My Washer'],
          name: 'Washer',
          nicknames: ['Washer'],
        },
        deviceInfo: {
          manufacturer: 'Acme Co',
          model: 'acme-washer',
          hwVersion: '1.0',
          swVersion: '1.0.1',
        },
        willReportState: true,
        attributes: {
          pausable: true,
        },
      }],
    },
  };
});

Deploy to Firebase

Deploy the updated cloud fulfillment using the Firebase CLI:

firebase deploy --only functions

In order to test your smart home Action, you need to link your project with a Google Account. This enables testing through Google Assistant surfaces and the Google Home app that are signed in to the same account.

  1. On your phone, open the Google Assistant settings. Note that you should be logged in as the same account as in the console.
  2. Navigate to Google Assistant > Settings > Home Control (under Assistant).
  3. Click the search icon in the upper right.
  4. Search for your test app using the [test] prefix to find your specific test app.
  5. Select that item. The Google Assistant will then authenticate with your service and send a SYNC request, asking your service to provide a list of devices for the user.

Open the Google Home app and verify that you can see your washer device.

ae252220753726f6.png

5. Handle commands and queries

Now that your cloud service properly reports the washer device to Google, you need to add the ability to request the device state and send commands.

Handle QUERY intent

A QUERY intent includes a set of devices. For each device, you should respond with its current state.

In functions/index.js, edit the QUERY handler to process the list of target devices contained in the intent request.

index.js

app.onQuery(async (body) => {
  const {requestId} = body;
  const payload = {
    devices: {},
  };
  const queryPromises = [];
  const intent = body.inputs[0];
  for (const device of intent.payload.devices) {
    const deviceId = device.id;
    queryPromises.push(queryDevice(deviceId)
        .then((data) => {
        // Add response to device payload
          payload.devices[deviceId] = data;
        }
        ));
  }
  // Wait for all promises to resolve
  await Promise.all(queryPromises);
  return {
    requestId: requestId,
    payload: payload,
  };
});

For each device contained in the request, return the current state stored in the Realtime Database. Update the queryFirebase and queryDevice functions to return the state data of the washer.

index.js

const queryFirebase = async (deviceId) => {
  const snapshot = await firebaseRef.child(deviceId).once('value');
  const snapshotVal = snapshot.val();
  return {
    on: snapshotVal.OnOff.on,
    isPaused: snapshotVal.StartStop.isPaused,
    isRunning: snapshotVal.StartStop.isRunning,
  };
};

const queryDevice = async (deviceId) => {
  const data = await queryFirebase(deviceId);
  return {
    on: data.on,
    isPaused: data.isPaused,
    isRunning: data.isRunning,
    currentRunCycle: [{
      currentCycle: 'rinse',
      nextCycle: 'spin',
      lang: 'en',
    }],
    currentTotalRemainingTime: 1212,
    currentCycleRemainingTime: 301,
  };
};

Handle EXECUTE intent

The EXECUTE intent handles commands to update device state. The response returns the status of each command—for example, SUCCESS, ERROR, or PENDING—and the new device state.

In functions/index.js, edit the EXECUTE handler to process the list of traits that need updates and the set of target devices for each command:

index.js

app.onExecute(async (body) => {
  const {requestId} = body;
  // Execution results are grouped by status
  const result = {
    ids: [],
    status: 'SUCCESS',
    states: {
      online: true,
    },
  };

  const executePromises = [];
  const intent = body.inputs[0];
  for (const command of intent.payload.commands) {
    for (const device of command.devices) {
      for (const execution of command.execution) {
        executePromises.push(
            updateDevice(execution, device.id)
                .then((data) => {
                  result.ids.push(device.id);
                  Object.assign(result.states, data);
                })
                .catch(() => functions.logger.error('EXECUTE', device.id)));
      }
    }
  }

  await Promise.all(executePromises);
  return {
    requestId: requestId,
    payload: {
      commands: [result],
    },
  };
});

For each command and target device, update the values in the Realtime Database that correspond to the requested trait. Modify the updateDevice function to update the appropriate Firebase reference and return the updated device state.

index.js

const updateDevice = async (execution, deviceId) => {
  const {params, command} = execution;
  let state; let ref;
  switch (command) {
    case 'action.devices.commands.OnOff':
      state = {on: params.on};
      ref = firebaseRef.child(deviceId).child('OnOff');
      break;
    case 'action.devices.commands.StartStop':
      state = {isRunning: params.start};
      ref = firebaseRef.child(deviceId).child('StartStop');
      break;
    case 'action.devices.commands.PauseUnpause':
      state = {isPaused: params.pause};
      ref = firebaseRef.child(deviceId).child('StartStop');
      break;
  }

  return ref.update(state)
      .then(() => state);
};

6. Test your Action

After you implement all three intents, you can test that your Action controls the washer.

Deploy to Firebase

Deploy the updated cloud fulfillment using the Firebase CLI:

firebase deploy --only functions

Test the washer

Now you can see the value change when you try any of the following voice commands through your phone:

"Hey Google, turn on my washer."

"Hey Google, pause my washer."

"Hey Google, stop my washer."

You can also see the current state of your washer by asking questions.

"Hey Google, is my washer on?"

"Hey Google, is my washer running?"

"Hey Google, what cycle is my washer on?"

You can view these queries and commands in the logs that appear under your function in the Functions section of the Firebase Console. Learn more about Firebase logs in Write and view logs.

You can also find these queries and commands in the Google Cloud Console by navigating to Logging > Logs Explorer. Learn more about Google Cloud logging in Access event logs with Cloud Logging.

7. Report updates to Google

You have fully integrated your cloud service with the smart home intents, enabling users to control and query the current state of their devices. However, the implementation still lacks a way for your service to proactively send event information—such as changes to device presence or state—to the Assistant.

With Request Sync, you can trigger a new sync request when users add or remove devices, or when their device capabilities change. With Report State, your cloud service can proactively send a device's state to Home Graph when users physically change a device state—for example, turning on a light switch—or change the state using another service.

In this section, you will add code to call these methods from the frontend web app.

Enable the HomeGraph API

The HomeGraph API enables the storage and querying of devices and their states within a user's Home Graph. To use this API, you must first open the Google Cloud console and enable the HomeGraph API.

In the Google Cloud console, make sure to select the project that matches your Actions <project-id>. Then, in the API Library screen for the HomeGraph API, click Enable.

ee198858a6eac112.png

Enable Report State

Writes to the Realtime Database trigger the reportstate function in the starter project. Update the reportstate function in functions/index.js to capture the data written to the database and post it to Home Graph via Report State.

index.js

exports.reportstate = functions.database.ref('{deviceId}').onWrite(
    async (change, context) => {
      functions.logger.info('Firebase write event triggered Report State');
      const snapshot = change.after.val();

      const requestBody = {
        requestId: 'ff36a3cc', /* Any unique ID */
        agentUserId: USER_ID,
        payload: {
          devices: {
            states: {
              /* Report the current state of our washer */
              [context.params.deviceId]: {
                on: snapshot.OnOff.on,
                isPaused: snapshot.StartStop.isPaused,
                isRunning: snapshot.StartStop.isRunning,
              },
            },
          },
        },
      };

      const res = await homegraph.devices.reportStateAndNotification({
        requestBody,
      });
      functions.logger.info('Report state response:', res.status, res.data);
    });

Enable Request Sync

Refreshing the icon in the frontend web UI triggers the requestsync function in the starter project. Implement the requestsync function in functions/index.js to call the HomeGraph API.

index.js

exports.requestsync = functions.https.onRequest(async (request, response) => {
  response.set('Access-Control-Allow-Origin', '*');
  functions.logger.info(`Request SYNC for user ${USER_ID}`);
  try {
    const res = await homegraph.devices.requestSync({
      requestBody: {
        agentUserId: USER_ID,
      },
    });
    functions.logger.info('Request sync response:', res.status, res.data);
    response.json(res.data);
  } catch (err) {
    functions.logger.error(err);
    response.status(500).send(`Error requesting sync: ${err}`);
  }
});

Deploy to Firebase

Deploy the updated code using the Firebase CLI:

firebase deploy --only functions

Test your implementation

Click the Refresh ae8d3b25777a5e30.png button in the web UI and verify that you see a sync request in the Firebase console log.

Next, adjust the attributes of the washer device in the frontend web UI and click Update. Verify that you can see the state change reported to Google in your Firebase console logs.

8. Congratulations

674c4f4392e98c1.png

Congratulations! You successfully integrated the Assistant with a device cloud service using smart home Actions.

Learn more

Here are some ideas you can implement to go deeper:

You can also learn more about testing and submitting an Action for review, including the certification process to publish your Action to users.